clan/clan-core
12
47
Fork 113
Code Issues 153 Pull Requests 35 Actions 1 Packages Projects 3 Releases Wiki Activity

WIP: working document about decoupling vars backends from clan #3196

Closed
lopter wants to merge 6759 commits from lopter/clan-core:lo-vars-brainstorm into main
pull from: lopter/clan-core:lo-vars-brainstorm
merge into: :main
Conversation 9 Commits 6759 Files Changed 1361
+99178 -42471
lopter commented 2025-04-03 20:29:31 +00:00
Contributor
Copy Link
Copy Source

Opening a PR so that everyone can comment.

Rendered version in my branch.

Opening a PR so that everyone can comment. [Rendered version](https://git.clan.lol/lopter/clan-core/src/branch/lo-vars-brainstorm/decisions/07-vars-backends.md) in my branch.
hsjobeki commented 2025-04-04 12:14:28 +00:00
Owner
Copy Link
Copy Source

Reposting from what i said in matrix, and maybe could structure this document

Into these headings:

  • Interactions: How to create and depend on impure values. Without going back and forth between evaluation and python calls. What approach do we take, and what are the limitations for higher layers. (i.e. cannot interpolate vars with vars as nix values, even if they specify to depend on each other)
  • API: how do we need to shape it such that it can connect to multiple encryption, storage, deployment backends? ...
Reposting from what i said in matrix, and maybe could structure this document Into these headings: - Interactions: How to create and depend on impure values. Without going back and forth between evaluation and python calls. What approach do we take, and what are the limitations for higher layers. (i.e. cannot interpolate vars with vars as nix values, even if they specify to depend on each other) - API: how do we need to shape it such that it can connect to multiple encryption, storage, deployment backends? ...
decisions/03-vars-backends.md Outdated
@@ -0,0 +10,4 @@
I[^1], have been thinking about how we could decouple the sops vars implementation from `clan-cli`, and the goal of this longer form post, is to help me / us converge towards some kind of spec of what that means. A few things are spinning in my head, points are numbered for easy referencing:
1. The sops-nix backend stores the secrets in the same repository as the clan which is problematic because it leaks metadata, e.g. the name of all the users in your directory as soon as you need to create per-users secrets;
Mic92 commented 2025-04-04 14:59:34 +00:00
Owner
Copy Link
Copy Source

We could actually support storing secrets in a different repository/location.
At the moment we require uploading secrets through flakes/sops, but sops-nix actually doesn't requires secrets in the nix store. It also supports a mode where secrets can be on the filesystem.

We could actually support storing secrets in a different repository/location. At the moment we require uploading secrets through flakes/sops, but sops-nix actually doesn't requires secrets in the nix store. It also supports a mode where secrets can be on the filesystem.
👍 1
lopter commented 2025-04-04 18:14:47 +00:00
Author
Contributor
Copy Link
Copy Source
  • Interactions: How to create and depend on impure values. Without going back and forth between evaluation and python calls. What approach do we take, and what are the limitations for higher layers. (i.e. cannot interpolate vars with vars as nix values, even if they specify to depend on each other)

I looked into nixops4 a little bit, and will look further to see how, vars and their backends, can be implemented as, resources and providers.

  • API: how do we need to shape it such that it can connect to multiple encryption, storage, deployment backends? ...

Top of mind there for me, is being able to use multiple vars backend on a single host.

Otherwise, in my mind, the API is basically the exchange we already have:

  1. vars declarations1 are passed to vars backends;
  2. vars backends generate vars definitions;
  3. vars definitions are referenced in the config (e.g. in activationScripts).

  1. I am using the work declaration here to refer to the data used to know how to generate the vars; as opposed to vars definitions which is the part that comes after the vars are generated. ↩︎

> - Interactions: How to create and depend on impure values. Without going back and forth between evaluation and python calls. What approach do we take, and what are the limitations for higher layers. (i.e. cannot interpolate vars with vars as nix values, even if they specify to depend on each other) I looked into nixops4 a little bit, and will look further to see how, vars and their backends, can be implemented as, resources and providers. > - API: how do we need to shape it such that it can connect to multiple encryption, storage, deployment backends? ... Top of mind there for me, is being able to use multiple vars backend on a single host. Otherwise, in my mind, the API is basically the exchange we already have: 1. vars declarations[^1] are passed to vars backends; 1. vars backends generate vars definitions; 1. vars definitions are referenced in the config (e.g. in `activationScripts`). [^1]: I am using the work declaration here to refer to the data used to know how to generate the vars; as opposed to vars definitions which is the part that comes after the vars are generated.
decisions/03-vars-backends.md Outdated
@@ -0,0 +11,4 @@
I[^1], have been thinking about how we could decouple the sops vars implementation from `clan-cli`, and the goal of this longer form post, is to help me / us converge towards some kind of spec of what that means. A few things are spinning in my head, points are numbered for easy referencing:
1. The sops-nix backend stores the secrets in the same repository as the clan which is problematic because it leaks metadata, e.g. the name of all the users in your directory as soon as you need to create per-users secrets;
1. I understand the [pass]\([-age]) backends are not subject to this issue, but otoh you can't specify recipients on a per-secret basis, the entire store is encrypted with the same set of PGP or AGE keys;
lassulus commented 2025-04-05 17:35:43 +00:00
Owner
Copy Link
Copy Source

you can override recipients (with pass at least) with a per folder .gpg-id file

you can override recipients (with pass at least) with a per folder .gpg-id file
👍 1
lopter commented 2025-04-05 17:38:39 +00:00
Author
Contributor
Copy Link
Copy Source

I had no idea, good to know!

I had no idea, good to know!
lassulus commented 2025-04-07 17:16:03 +00:00
Owner
Copy Link
Copy Source

About the secrets CLI. I had the rough idea to deprecate the secrets CLI (or move to another repo) and instead have a lower level cars CLI that is backend agnostic and can do most of the things we need. Like setting up users/groups. The CLI would get the configured back end from nix and use the python classes afterwards (the the vars upload code)

About the secrets CLI. I had the rough idea to deprecate the secrets CLI (or move to another repo) and instead have a lower level cars CLI that is backend agnostic and can do most of the things we need. Like setting up users/groups. The CLI would get the configured back end from nix and use the python classes afterwards (the the vars upload code)
lopter commented 2025-04-07 18:29:33 +00:00
Author
Contributor
Copy Link
Copy Source

About the secrets CLI. I had the rough idea to deprecate the secrets CLI (or move to another repo) and instead have a lower level cars CLI that is backend agnostic and can do most of the things we need. Like setting up users/groups. The CLI would get the configured back end from nix and use the python classes afterwards (the the vars upload code)

I agree with the sentiment that the secrets CLI does not belong in the clan CLI. I agree the users/group abstraction is nice to have inside clan, and could help have a more consistent/DRY interface across backends; we can leave it to clan users to setups multiple users and/or use groups only with backends that support it.

> About the secrets CLI. I had the rough idea to deprecate the secrets CLI (or move to another repo) and instead have a lower level cars CLI that is backend agnostic and can do most of the things we need. Like setting up users/groups. The CLI would get the configured back end from nix and use the python classes afterwards (the the vars upload code) I agree with the sentiment that the secrets CLI does not belong in the clan CLI. I agree the users/group abstraction is nice to have inside clan, and could help have a more consistent/DRY interface across backends; we can leave it to clan users to setups multiple users and/or use groups only with backends that support it.
lopter force-pushed lo-vars-brainstorm from b41e7cda9d to 421512bc82 2025-04-19 04:20:28 +00:00 Compare
lopter force-pushed lo-vars-brainstorm from 421512bc82 to bf36b08596 2025-04-30 17:01:20 +00:00 Compare
lopter force-pushed lo-vars-brainstorm from bf36b08596 to 2558328fb0 2025-04-30 17:19:42 +00:00 Compare
hsjobeki commented 2025-05-06 08:07:39 +00:00
Owner
Copy Link
Copy Source

I have attached my architecture vision of vars.

Vars-architecture.drawio.svg

Right now generate directly depends on the store and on the machine class, we should use dependency injection and a narrow contract for the interface between a generator and the vars generate.

Should i make my own adr ?

I have attached my architecture vision of vars. ![Vars-architecture.drawio.svg](/attachments/745c11ad-2926-40bc-ac75-26547b9a73a5) Right now generate directly depends on the store and on the machine class, we should use dependency injection and a narrow contract for the interface between a generator and the vars generate. Should i make my own adr ?
Vars-architecture.drawio.svg
214 KiB
lopter force-pushed lo-vars-brainstorm from 2558328fb0 to a28e0144fb 2025-05-14 02:59:12 +00:00 Compare
lopter force-pushed lo-vars-brainstorm from a28e0144fb to 1acd917c6e 2025-05-15 18:08:01 +00:00 Compare
jfly left a comment
Contributor
Copy Link
Copy Source

I read through this. I don't know clan's codebase at all, so some of the juiciest parts in the "Decision" section sort of went in one ear and out the other.

I left a lot of readability nits. Sorry for the noise!

I read through this. I don't know clan's codebase at all, so some of the juiciest parts in the "Decision" section sort of went in one ear and out the other. I left a lot of readability nits. Sorry for the noise!
decisions/07-vars-backends.md Outdated
jfly commented 2025-05-15 18:08:04 +00:00
Contributor
Copy Link
Copy Source

The talk about clan being coupled with sops is confusing to me, as Clan already provides multiple vars backends: both sops-nix and pass. I feel like this needs more words:

  1. Is clan really coupled with sops/sops-nix if it also supports pass?
  2. This calls for deleteing all sops-related code from clan_cli. Where would it go instead? Somewhere else in clan-core?
  3. What's the plan for the pass backend? Does it need similar treatment?

As a nit, I found the last sentence to be very confusing, perhaps it's missing a word?

Sops/sops-nix can be one of multiple (e.g. pass) ways, plugins, to choose from to store secrets.

The talk about clan being coupled with sops is confusing to me, as Clan already provides multiple vars backends: both sops-nix and pass. I feel like this needs more words: 1. Is clan really coupled with `sops/sops-nix` if it also supports `pass`? 2. This calls for deleteing all sops-related code from `clan_cli`. Where would it go instead? Somewhere else in `clan-core`? 3. What's the plan for the `pass` backend? Does it need similar treatment? As a nit, I found the last sentence to be very confusing, perhaps it's missing a word? > Sops/`sops-nix` can be one of multiple (e.g. `pass`) ways, plugins, to choose from to store secrets.
decisions/07-vars-backends.md Outdated
jfly commented 2025-05-15 18:10:09 +00:00
Contributor
Copy Link
Copy Source

nit: "is" -> "may be", right? Some generators just roll dice without human input.

nit: "is" -> "may be", right? Some generators just roll dice without human input.
decisions/07-vars-backends.md Outdated
jfly commented 2025-05-15 18:10:55 +00:00
Contributor
Copy Link
Copy Source

nit: Could use rephrasing, as this assumes there is user input (see my previous comment).

nit: Could use rephrasing, as this assumes there is user input (see my previous comment).
decisions/07-vars-backends.md Outdated
jfly commented 2025-05-15 18:12:15 +00:00
Contributor
Copy Link
Copy Source

nit: "declaration" -> "declarations"

nit: "declaration" -> "declarations"
decisions/07-vars-backends.md Outdated
jfly commented 2025-05-15 18:13:19 +00:00
Contributor
Copy Link
Copy Source

nit: For consistency with the previous entry ("Exporting ...", either changing this to "Prompting", or change the previous one to "Export ". I think the "*ing" reads a little nicer with the intro to this list.

nit: For consistency with the previous entry ("Exporting ...", either changing this to "Prompting", or change the previous one to "Export ". I think the "*ing" reads a little nicer with the intro to this list.
decisions/07-vars-backends.md Outdated
jfly commented 2025-05-15 18:18:11 +00:00
Contributor
Copy Link
Copy Source

I have some familiarity with vars, but basically none with clan. I'm having trouble understanding this, especially the "(vars backend)" parenthetical. Perhaps this needs more words, or just needs the parenthetical removed?

Getting secrets on the target machine is a key part of implementing a vars-based "system", and it's left out of the vars nix code because (as you call out above), it cannot be implemented in Nix. Calling this StoreBase CRUD interface the "vars backend" doesn't sound right to me, though.

I have some familiarity with vars, but basically none with clan. I'm having trouble understanding this, especially the "(vars backend)" parenthetical. Perhaps this needs more words, or just needs the parenthetical removed? Getting secrets on the target machine is a key part of implementing a vars-based "system", and it's left out of the vars nix code because (as you call out above), it cannot be implemented in Nix. Calling this `StoreBase` CRUD interface the "vars backend" doesn't sound right to me, though.
decisions/07-vars-backends.md Outdated
jfly commented 2025-05-15 18:31:52 +00:00
Contributor
Copy Link
Copy Source

CRUD interface ("vars backends")

I commented on this above: I'm confused by calling StoreBase (a python class) equivalent to a "vars backend". Vars backends require some nix code, right?

nit: I haven't written a lot of ADRs, but the phrasing here seems odd. You're proposing this ADR. The ADR has a decision, and to me, shouldn't use language like "propose".

I'd rephrase this from:

I propose that we move the CRUD interface

to:

Move the CRUD interface

(I see at least one more use of the word "propose" below, I won't comment on all of them)

> CRUD interface ("vars backends") I commented on this above: I'm confused by calling `StoreBase` (a python class) equivalent to a "vars backend". Vars backends require some nix code, right? <hr> nit: I haven't written a lot of ADRs, but the phrasing here seems odd. You're *proposing* this ADR. The ADR has a decision, and to me, shouldn't use language like "propose". I'd rephrase this from: > I propose that we move the CRUD interface to: > Move the CRUD interface (I see at least one more use of the word "propose" below, I won't comment on all of them)
lopter force-pushed lo-vars-brainstorm from 1acd917c6e to 53c5d33b53 2025-05-27 02:03:17 +00:00 Compare
lopter force-pushed lo-vars-brainstorm from 53c5d33b53 to ab65b6f6e0 2025-05-29 16:17:39 +00:00 Compare
lopter force-pushed lo-vars-brainstorm from ab65b6f6e0 to 3225f6ed5f 2025-06-18 16:20:27 +00:00 Compare
Qubasa commented 2025-09-09 10:18:29 +00:00
Owner
Copy Link
Copy Source

@lopter Closing for now, as commit history is broken. Please reopen if it is still relevant.

@lopter Closing for now, as commit history is broken. Please reopen if it is still relevant.
Qubasa closed this pull request 2025-09-09 10:18:29 +00:00

Pull request closed

Please reopen this pull request to perform a merge.
This pull request is marked as a work in progress.
Sign in to join this conversation.
Reviewers
No Reviewers
Mic92
lassulus
jfly
Labels
Clear labels
Type/Regression
Breaking change in behavior
Architecture
Change
Fundamental change spanning across components
Component
API
Issues regarding the clan api
Component
App
Issues regarding the clan app
Component
Backup
Component
CICD
Component
CLI
Component
Clan Services
Component
Clan module system
Issues regarding the clan module system including inventory.
Component
Docs
Component
Flake
Component
Install & Update
Component
Networking
Component
Select
Issues regarding select - the caching layer
Component
Templates
Issues regarding templates.
Component
Testing
Issues regarding testing
Component
VARS
Issues regarding vars.
Component
Virtualisation
Issues regarding virtual machines.
Good first issue
Help wanted
OS
Darwin
OS
Linux
Priority
Critical
1
Priority
High
2
Priority
Medium
3
Priority
Low
4
Status
Blocked
1
Status
On Hold
2
Status
Needs More Info
3
Status
Needs Review
4
Status
In Progress
5
Status
Won't Fix
6
Status
Available
Status
Duplicate
Type
Community Contribution
1
contributed by members of the Clan community
Type
Discussion
1
Type
Bug
2
Type
Question
3
Type
Epic
4
Type
Feature Request
5
Type
User Experience
5
Type
Enhancement
6
Type
Maintenance
7
Type
Documentation
Type
Question
Question to be resolved
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
admin brianmcgee clan-bot DavHau Enzime (Michael Hoang) hsjobeki kenji (Kenji Berthold) lassulus Mic92 nixbot pinpox Qubasa (Luis Hebendanz) renovate
No Assignees
6 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: clan/clan-core#3196