Add dynamic completion scaffolding to the clan `cli`.
Also add a dynamic completion mechanism for machines for commands that
have machines as their sole argument.
More intricate dynamic completions will be implemented in follow up
PR's.
Injecting nixos configuration and potentially overriding settings a user
made and can cause surprises.
In most cases, users want to just make these option part of their NixOS
configuration and by having the rest in the command line
we make it more explicit what other configuration is being applied.
When a user runs --help on a subcommand they don't see some options such
as --options or --flake. To fix this we now register all common flags
also in subcommands.
Since we already walk the user through creating a secret in an earlier step, it makes more sense explain first how to add machines/users to an existing secret instead of creating a new one
Add the `run_no_stdout` function suppressing stdout by default.This
keeps the noise down on most commands, while still
stayingdebuggable.Stdout will be active when the `--debug` flag is
passed to the cli.
Fixes#1443
pre-commit hook break git commits and are disruptive.
Therefore people that want to enable this feature, should enable it locally instead.
I.e. treefmt will also check untracked files that are not meant for the current commit.
Add `--regenerate` flag to `clan facts generate` which allows forcing
the generation of facts, regardless of their current existence.
Examples:
```
clan facts generate [MACHINE] --regenerate
```
or
```
clan facts generate [MACHINE] --service [SERVICE] --regenerate
```
Add `--service` flag to the `clan` cli which allows specifying a certain
service to be generated.
Example:
```
clan facts generate [MACHINE] --service [SERVICE]
```
Fixes#1395
make sure things are sane before they hit CI, re-purposing the existing
treefmt configuration.
this adds a custom installer for pre-commit hooks, which is inspired by
pre-commit.nix[0], but is much more minimal than the underlying
pre-commit[1] and builds on a historic idea[2] from this repository.
[0]: https://github.com/cachix/git-hooks.nix
[1]: https://github.com/pre-commit/pre-commit
[2]: 930923512c
Init zerotertier-static-peers module.
This module automatically configures the networkId.
It will automatically accept peers based on their zerotier-ips in the
clan flake.
This fixes `clan ssh` with the `--json` and `--png` flags.
It will now correctly use the actual fields that are present in the
generated json.
- probes if the ports are accessible
- if accessible will attempt a single ssh connection with the provided
password, in order to not spam ssh attempts
Fixes#1177
The guide instructs the reader to look for the PTUUID (partition ID) to fill
disk ID in the disklayout. This leads to an error as the partition gets deleted
and the UUID is no longer valid. The ID-LINK field is a unique ID provided by
the hardware manufacturer.
This integrates the generated options docs part of our website into the clan-core project. This is better than having it in a separate repos because we want to lear about breakages as early as possible.
Changes which break the documentation should be blocked by this early on
The devShell depended on clan-cli due to it being included as a dependency in the treefmt config. This is not optimal because this makes the devshell rebuild unnecessary often and also lead to build failures of the dev-shell if the clan-cli code is in a broken state (git rebasing, or during development etc.)
- add python modules for qemu protocols: QMP (hardware interactions) and QGA (guest service interaction)
- refactor state directory: remove name from path (already contains url)
- add impure vm test for basic qmp interaction
- simplify existing vm persistance test (factor out shared code)
- integrate graceful shutdown into GUI
the GUI integration still needs to be improved later:
- add fallback in case system doesn't react to powerdown button
- shutdown GUI switch fails if VM hasn't been started yet, and then remains in a wrong position
Done:
- move vm inspect attrs from system.clan.vm.config to clanCore.vm.inspect. This gives us proper name and type checking. everything in `system` is basically freeform, so the previous option definitions were never enforced
- when running VMs, mount state directory from ~/.config/clan/vmstate/{...} from the host to /var/vmstate inside the vm
- create bind mount inside the VM from /var/vmstate/{folder} to / for all folders defined in clanCore.state.<name>.folders
TODOs:
- make sure directories in ~/.config/clan/vmstate never collide (include hash of clan-url, etc.)
- port impure test to python
The icon is confusing. It distracts from the actual "Join" label. Also when a user
copy in an URL we can assume they trust the content, which might be not the case
when they just clicked on an URL.
Welcome to the cLAN Core Repository, the heart of the [clan.lol](https://clan.lol/) project! This monorepo houses all the essential packages, NixOS modules, CLI tools, and tests you need to contribute and work with the cLAN project.
Welcome to the Clan Core Repository, the heart of the [clan.lol](https://clan.lol/) project! This monorepo is the foundation of Clan, a revolutionary open-source project aimed at restoring fun, freedom, and functionality to computing. Here, you'll find all the essential packages, NixOS modules, CLI tools, and tests needed to contribute to and work with the Clan project. Clan leverages the Nix system to ensure reliability, security, and seamless management of digital environments, putting the power back into the hands of users.
## Getting Started
## Why Clan?
If you're new to cLAN and eager to dive in, start with our quickstart guide:
Our mission is simple: to democratize computing by providing tools that empower users, foster innovation, and challenge outdated paradigms. Clan represents our contribution to a future where technology serves humanity, not the other way around. By participating in Clan, you're joining a movement dedicated to creating a secure, user-empowered digital future.
- **Quickstart Guide**: Check out [quickstart.md](docs/quickstart.md) to get up and running with cLAN in no time.
## Features of Clan
## Managing Secrets
- **Full-Stack System Deployment:** Utilize Clan’s toolkit alongside Nix's reliability to build and manage systems effortlessly.
- **Overlay Networks:** Secure, private communication channels between devices.
- **Virtual Machine Integration:** Seamless operation of VM applications within the main operating system.
- **Robust Backup Management:** Long-term, self-hosted data preservation.
- **Intuitive Secret Management:** Simplified encryption and password management processes.
Security is paramount, and cLAN provides guidelines for handling secrets effectively:
## Getting Started with Clan
- **Secrets Management**: Learn how to manage secrets securely by reading [secrets-management.md](docs/secrets-management.md).
If you're new to Clan and eager to dive in, start with our quickstart guide and explore the core functionalities that Clan offers:
## Contributing to cLAN
- **Quickstart Guide**: Check out [getting started](https://docs.clan.lol/#starting-with-a-new-clan-project)<!-- [docs/site/index.md](docs/site/index.md) --> to get up and running with Clan in no time.
We welcome contributions from the community, and we've prepared a comprehensive guide to help you get started:
### Managing Secrets
- **Contribution Guidelines**: Find out how to contribute and make a meaningful impact on the cLAN project by reading [contributing.md](docs/contributing.md).
In the Clan ecosystem, security is paramount. Learn how to handle secrets effectively:
The Clan project thrives on community contributions. We welcome everyone to contribute and collaborate:
- **Contribution Guidelines**: Make a meaningful impact by following the steps in [contributing](https://docs.clan.lol/contributing/contributing/)<!-- [contributing.md](docs/CONTRIBUTING.md) -->.
## Join the Revolution
Clan is more than a tool; it's a movement towards a better digital future. By contributing to the Clan project, you're part of changing technology for the better, together.
### Community and Support
Connect with us and the Clan community for support and discussion:
- [Matrix channel](https://matrix.to/#/#clan:lassul.us) for live discussions.
- IRC bridges (coming soon) for real-time chat support.
Whether you're a newcomer or a seasoned developer, we look forward to your contributions and collaboration on the cLAN project. Let's build amazing things together!
A secure, file synchronization app for devices over networks, offering a private alternative to cloud services.
---
## Usage
We recommend configuring this module as an sync-service through the provided options. Although it provides a Web GUI through which more usage scenarios are supported.
## Features
- **Private and Secure**: Syncthing uses TLS encryption to secure data transfer between devices, ensuring that only the intended devices can read your data.
- **Decentralized**: No central server is involved in the data transfer. Each device communicates directly with others.
- **Open Source**: The source code is openly available for audit and contribution, fostering trust and continuous improvement.
- **Cross-Platform**: Syncthing supports multiple platforms including Windows, macOS, Linux, BSD, and Android.
- **Real-time Synchronization**: Changes made to files are synchronized in real-time across all connected devices.
- **Web GUI**: It includes a user-friendly web interface for managing devices and configurations. (`127.0.0.1:8384`)
## Configuration
- **Share Folders**: Select folders to share with connected devices and configure permissions and synchronization parameters.
!!! info
Clan automatically discovers other devices. Automatic discovery requires one machine to be an [introducer](#clan.syncthing.introducer)
If that is not the case you can add the other device by its Device ID manually.
You can find and share Device IDs under the "Add Device" button in the Web GUI. (`127.0.0.1:8384`)
## Troubleshooting
- **Sync Conflicts**: Resolve synchronization conflicts manually by reviewing file versions and modification times in the Web GUI (`127.0.0.1:8384`).
## Support
- **Documentation**: Extensive documentation is available on the [Syncthing website](https://docs.syncthing.net/).
**Continuous Integration (CI)**: Each pull request gets automatically tested by gitea. If any errors are detected, it will block pull requests until they're resolved.
**Dependency Management**: We use the [Nix package manager](https://nixos.org/) to manage dependencies and ensure reproducibility, making your development process more robust.
## Supported Operating Systems
- Linux
- macOS
# Getting Started with the Development Environment
Let's get your development environment up and running:
1. **Install Nix Package Manager**:
- You can install the Nix package manager by either [downloading the Nix installer](https://github.com/DeterminateSystems/nix-installer/releases) or running this command:
You can place `breakpoint()` in your Python code where you want to trigger a breakpoint for debugging.
## Test Locally in a Nix Sandbox
To run tests in a Nix sandbox, you have two options depending on whether your test functions have been marked as impure or not:
### Running Tests Marked as Impure
If your test functions need to execute `nix build` and have been marked as impure because you can't execute `nix build` inside a Nix sandbox, use the following command:
```bash
nix run .#impure-checks
```
This command will run the impure test functions.
### Running Pure Tests
For test functions that have not been marked as impure and don't require executing `nix build`, you can use the following command:
@ -8,13 +8,13 @@ Joining a self-hosted infrastructure involves connecting to a network, server, o
### Story 1: Joining a private network
Alice' son Bob has never heard of cLAN, but receives an invitation URL from Alice who already set up private cLAN network for her family.
Alice' son Bob has never heard of Clan, but receives an invitation URL from Alice who already set up private Clan network for her family.
Bob opens the invitation link and lands on the cLAN website. He quickly learns about what cLAN is and can see that the invitation is for a private network of his family that hosts a number of services, like a private voice chat and a photo sharing platform.
Bob opens the invitation link and lands on the Clan website. He quickly learns about what Clan is and can see that the invitation is for a private network of his family that hosts a number of services, like a private voice chat and a photo sharing platform.
Bob decides to join the network and follows the instructions to install the cLAN tool on his computer.
Bob decides to join the network and follows the instructions to install the Clan tool on his computer.
Feeding the invitation link to the cLAN tool, bob registers his machine with the network.
Feeding the invitation link to the Clan tool, bob registers his machine with the network.
All programs required to interact with the network will be installed and configured automatically and securely.
@ -22,7 +22,7 @@ Optionally, bob can customize the configuration of these programs through a simp
### Story 2: Receiving breaking changes
The cLAN family network which Bob is part of received an update.
The Clan family network which Bob is part of received an update.
The existing photo sharing service has been removed and replaced with another alternative service. The new photo sharing service requires a different client app to view and upload photos.
@ -30,7 +30,7 @@ Bob accepts the update. Now his environment will be updated. The old client soft
Because Bob has customized the previous photo viewing app, he is notified that this customization is no longer valid, as the software has been removed (deprecation message).l
Optionally, Bob can now customize the new photo viewing software through his cLAN configuration app or via a config file.
Optionally, Bob can now customize the new photo viewing software through his Clan configuration app or via a config file.
This issue serves to collect our common understanding how to design our API so that it is extensible and usable and understandable.
## Resource oriented
A resource-oriented API is generally modeled as a resource hierarchy, where each node is either a simple resource or a collection resource. For convenience, they are often called a resource and a collection, respectively.
Examples of Resource Nouns:
`machine`
`user`
`flake`
Often resources have sub-resources. Even if it is not foreseen, it is recommended to use plural (trailing `s`) on resources to allow them to be collections of sub-resources.
e.g,
`users`
->
`users/*/profile`
## Verbs
Verbs should not be part of the URL
Bad:
`/api/create-products`
Good:
`/api/products`
Only resources are part of the URL, verbs are described via the HTTP Method.
Exception:
If a different HTTP Method must be used for technical reasons it is okay to terminate the path with a (short) verb / action.
Okay ish:
`/api/products/create`
## Usually the following HTTP Methods exist to interact with a resource
- POST (create an order for a resource)
- GET (retrieve the information)
- PUT (update and replace information)
- PATCH (update and modify information) **(Not used yet)**
- DELETE (delete the item)
## Every resource should be CRUD compatible
All API resources MUST be designed in a way that allows the typical CRUD operations.
Where crud stands for:
C - Create
R - Read
U - Update
D - Delete
Resources should implement at least a "Read" operation.
## Body
Use JSON as an exchange format.
All responses MUST be JSON parseable.
Bad:
`bare string`
Better:
`"quoted string"`
Best: (Enveloped see next section)
`{ name: "quoted string"}`
Errors should have a consistent JSON format, such that it is clear in which field to look at for displaying error messages.
## Envelop all Data collections
Response data should be wrapped into an JSON Object `{}`
Lists `[]` should also contain Objects `{}`.
This allows everything, to be extensible, without breaking backwards compatibility. (Adding fields is trivial, since the schema doesn't change)
Example:
```
{
"users": [{
first_name: "John",
last_name: "Doe",
…
}, {
first_name: "Jane",
last_name: "Doe",
…
}
....
],
"skip": 0,
"limit": 20,
....
}
```
Bad Example of a breaking change:
`GET /api/flakes`
`old`
```
[
"dream2nix"
"disko"
]
```
`new`
```
[
{
name: "dream2nix",
url: "github/...."
},
{
name: "disko",
url: "github/...."
}
]
```
Those kind of breaking changes can be avoided by using an object from the beginning.
Even if the object only contains one key, it is extensible, without breaking.
Welcome to our website template repository! This template is designed to help you and your team build high-quality websites efficiently. We've carefully chosen the technologies to make development smooth and enjoyable. Here's what you can expect from this template:
**Frontend**: Our frontend is powered by [React NextJS](https://nextjs.org/), a popular and versatile framework for building web applications.
**Backend**: For the backend, we use Python along with the [FastAPI framework](https://fastapi.tiangolo.com/). To ensure seamless communication between the frontend and backend, we generate an `openapi.json` file from the Python code, which defines the REST API. This file is then used with [Orval](https://orval.dev/) to generate TypeScript bindings for the REST API. We're committed to code correctness, so we use [mypy](https://mypy-lang.org/) to ensure that our Python code is statically typed correctly. For backend testing, we rely on [pytest](https://docs.pytest.org/en/7.4.x/).
**Continuous Integration (CI)**: We've set up a CI bot that rigorously checks your code using the quality assurance (QA) tools mentioned above. If any errors are detected, it will block pull requests until they're resolved.
**Dependency Management**: We use the [Nix package manager](https://nixos.org/) to manage dependencies and ensure reproducibility, making your development process more robust.
## Supported Operating Systems
- Linux
- macOS
# Getting Started with the Development Environment
Let's get your development environment up and running:
1. **Install Nix Package Manager**:
- You can install the Nix package manager by either [downloading the Nix installer](https://github.com/DeterminateSystems/nix-installer/releases) or running this command:
- When you enter the directory, you'll receive an error message like this:
```bash
direnv: error .envrc is blocked. Run `direnv allow` to approve its content
```
- Execute `direnv allow` to automatically execute the shell script `.envrc` when entering the directory.
6. **Build the Backend**:
- Go to the `pkgs/clan-cli` directory and execute:
```bash
direnv allow
```
- Wait for the backend to build.
7. **Start the Backend Server**:
- To start the backend server, execute:
```bash
clan webui --reload --no-open --log-level debug
```
- The server will automatically restart if any Python files change.
8. **Build the Frontend**:
- In a different shell, navigate to the `pkgs/ui` directory and execute:
```bash
direnv allow
```
- Wait for the frontend to build.
NOTE: If you have the error "@clan/colors.json" you executed `npm install`, please do not do that. `direnv reload` will handle dependency management. Please delete node_modules with `rm -rf node_modules`.
9. **Start the Frontend**:
- To start the frontend, execute:
```bash
npm run dev
```
- Access the website by going to [http://localhost:3000](http://localhost:3000).
# Setting Up Your Git Workflow
Let's set up your Git workflow to collaborate effectively:
1. **Register Your Gitea Account Locally**:
- Execute the following command to add your Gitea account locally:
```bash
tea login add
```
- Fill out the prompt as follows:
- URL of Gitea instance: `https://gitea.gchq.icu`
- Name of new Login [gitea.gchq.icu]: `gitea.gchq.icu:7171`
- Do you have an access token? No
- Username: YourUsername
- Password: YourPassword
- Set Optional settings: No
2. **Git Workflow**:
1. Add your changes to Git using `git add <file1> <file2>`.
2. Run `nix fmt` to lint your files.
3. Commit your changes with a descriptive message: `git commit -a -m "My descriptive commit message"`.
4. Make sure your branch has the latest changes from upstream by executing:
```bash
git fetch && git rebase origin/main --autostash
```
5. Use `git status` to check for merge conflicts.
6. If conflicts exist, resolve them. Here's a tutorial for resolving conflicts in [VSCode](https://code.visualstudio.com/docs/sourcecontrol/overview#_merge-conflicts).
7. After resolving conflicts, execute `git merge --continue` and repeat step 5 until there are no conflicts.
3. **Create a Pull Request**:
- To automatically open a pull request that gets merged if all tests pass, execute:
```bash
merge-after-ci
```
4. **Review Your Pull Request**:
- Visit https://gitea.gchq.icu and go to the project page. Check under "Pull Requests" for any issues with your pull request.
5. **Push Your Changes**:
- If there are issues, fix them and redo step 2. Afterward, execute:
```bash
git push origin HEAD:YourUsername-main
```
- This will directly push to your open pull request.
# Debugging
When working on the backend of your project, debugging is an essential part of the development process. Here are some methods for debugging and testing the backend of your application:
## Test Backend Locally in Devshell with Breakpoints
To test the backend locally in a development environment and set breakpoints for debugging, follow these steps:
1. Run the following command to execute your tests and allow for debugging with breakpoints:
```bash
pytest -n0 -s --maxfail=1
```
You can place `breakpoint()` in your Python code where you want to trigger a breakpoint for debugging.
## Test Backend Locally in a Nix Sandbox
To run your backend tests in a Nix sandbox, you have two options depending on whether your test functions have been marked as impure or not:
### Running Tests Marked as Impure
If your test functions need to execute `nix build` and have been marked as impure because you can't execute `nix build` inside a Nix sandbox, use the following command:
```bash
nix run .#impure-checks
```
This command will run the impure test functions.
### Running Pure Tests
For test functions that have not been marked as impure and don't require executing `nix build`, you can use the following command:
If you want to test more request types edit the file `checks/impure/flake-module.nix`
### Inspecting the Nix Sandbox
If you need to inspect the Nix sandbox while running tests, follow these steps:
1. Insert an endless sleep into your test code where you want to pause the execution. For example:
```python
import time
time.sleep(3600) # Sleep for one hour
```
2. Use `cntr` and `psgrep` to attach to the Nix sandbox. This allows you to interactively debug your code while it's paused. For example:
```bash
cntr exec -w your_sandbox_name
psgrep -a -x your_python_process_name
```
These debugging and testing methods will help you identify and fix issues in your backend code efficiently, ensuring the reliability and robustness of your application.
For more information on testing read [property and contract based testing](testing.md)
# Using this Template
To make the most of this template:
1. Set up a new Gitea account named `ui-asset-bot`. Generate an access token with all access permissions and set it under `settings/actions/secrets` as a secret called `BOT_ACCESS_TOKEN`.
- Also, edit the file `.gitea/workflows/ui_assets.yaml` and change the `BOT_EMAIL` variable to match the email you set for that account. Gitea matches commits to accounts by their email address, so this step is essential.
2. Create a second Gitea account named `merge-bot`. Edit the file `pkgs/merge-after-ci/default.nix` if the name should be different. Under "Branches," set the main branch to be protected and add `merge-bot` to the whitelisted users for pushing. Set the unprotected file pattern to `**/ui-assets.nix`.
- Enable the status check for "build / test (pull_request)."
3. Add both `merge-bot` and `ui-asset-bot` as collaborators.
- Set the option to "Delete pull request branch after merge by default."
- Also, set the default merge style to "Rebase then create merge commit."
With this template, you're well-equipped to build and collaborate on high-quality websites efficiently. Happy coding!.
To start managing a new machine, use the following commands to create and then list your machines:
```shellSession
$ clan machines create my-machine
$ clan machines list
my-machine
```
## Configure Your Machine
In the example below, we demonstrate how to add a new user named `my-user` and set a password. This user will be configured to log in to the machine `my-machine`.
_Note: The `$(mkpasswd)` command generates a hashed password. Ensure you have the `mkpasswd` utility installed or use an alternative method to generate a secure hashed password._
## Test Your Machine Configuration Inside a VM
Before deploying your configuration to a live environment, you can run a virtual machine (VM) to test the settings:
```shellSession
$ clan vms run my-machine
```
This command run a VM based on the configuration of `my-machine`, allowing you to verify changes in a controlled environment.
## Installing a New Machine
Clan CLI, in conjunction with [nixos-anywhere](https://github.com/nix-community/nixos-anywhere), provides a seamless method for installing NixOS on various machines.
This process involves preparing a suitable hardware and disk partitioning configuration and ensuring the target machine is accessible via SSH.
### Prerequisites
- A running Linux system with SSH on the target machine is required. This is typically pre-configured for many server providers.
- For installations on physical hardware, create a NixOS installer image and transfer it to a bootable USB drive as described below.
## Creating a Bootable USB Drive on Linux
To create a bootable USB flash drive with the NixOS installer:
- After writing the installer to the USB drive, use it to boot the target machine.
- The installer will display an IP address and a root password, which you can use to connect via SSH.
### Finishing the installation
With the target machine running Linux and accessible via SSH, execute the following command to install NixOS on the target machine, replacing `<target_host>` with the machine's hostname or IP address:
```shellSession
$ clan machines install my-machine <target_host>
```
## Update Your Machines
Clan CLI enables you to remotely update your machines over SSH. This requires setting up a deployment address for each target machine.
### Setting the Deployment Address
Replace `host_or_ip` with the actual hostname or IP address of your target machine:
_Note: The use of `root@` in the deployment address implies SSH access as the root user. Ensure that the root login is secured and only used when necessary._
### Updating Machine Configurations
Execute the following command to update the specified machine:
```shellSession
$ clan machines update my-machine
```
You can also update all configured machines simultaneously by omitting the machine name:
2. Then use the following commands to initialize a new clan-flake:
```shellSession
$ clan flake create my-clan
```
This action will generate two primary files: `flake.nix` and `.clan-flake`.
```shellSession
$ ls -la
drwx------ joerg users 5 B a minute ago ./
drwxrwxrwt root root 139 B 12 seconds ago ../
.rw-r--r-- joerg users 77 B a minute ago .clan-flake
.rw-r--r-- joerg users 4.8 KB a minute ago flake.lock
.rw-r--r-- joerg users 242 B a minute ago flake.nix
```
### Understanding the .clan-flake Marker File
The `.clan-flake` marker file serves an optional purpose: it helps the `clan-cli` utility locate the project's root directory.
If `.clan-flake` is missing, `clan-cli` will instead search for other indicators like `.git`, `.hg`, `.svn`, or `flake.nix` to identify the project root.
## What's next
After creating your flake, you can check out how to add [new machines](./machines.md)
---
# Migrating Existing NixOS Configuration Flake
Absolutely, let's break down the migration step by step, explaining each action in detail:
#### Before You Begin
1. **Backup Your Current Configuration**: Always start by making a backup of your current NixOS configuration to ensure you can revert if needed.
```shellSession
$ cp -r /etc/nixos ~/nixos-backup
```
2. **Update Flake Inputs**: Add a new input for the `clan-core` dependency:
```nix
inputs.clan-core = {
url = "git+https://git.clan.lol/clan/clan-core";
# Don't do this if your machines are on nixpkgs stable.
inputs.nixpkgs.follows = "nixpkgs";
};
```
- `url`: Specifies the Git repository URL for Clan Core.
- `inputs.nixpkgs.follows`: Tells Nix to use the same `nixpkgs` input as your main input (in this case, it follows `nixpkgs`).
3. **Update Outputs**: Then modify the `outputs` section of your `flake.nix` to adapt to Clan Core's new provisioning method. The key changes are as follows:
in { inherit (clan) nixosConfigurations clanInternals; }
```
- `nixosConfigurations`: Defines NixOS configurations, using Clan Core’s `buildClan` function to manage the machines.
- Inside `machines`, a new machine configuration is defined (in this case, `example-desktop`).
- Inside `example-desktop` which is the target machine hostname, `nixpkgs.hostPlatform` specifies the host platform as `x86_64-linux`.
- `clanInternals`: Is required to enable evaluation of the secret generation/upload script on every architecture
- `clanName`: Is required and needs to be globally unique, as else we have a cLAN name clash
4. **Rebuild and Switch**: Rebuild your NixOS configuration using the updated flake:
```shellSession
$ sudo nixos-rebuild switch --flake .
```
- This command rebuilds and switches to the new configuration. Make sure to include the `--flake .` argument to use the current directory as the flake source.
5. **Test Configuration**: Before rebooting, verify that your new configuration builds without errors or warnings.
6. **Reboot**: If everything is fine, you can reboot your system to apply the changes:
```shellSession
$ sudo reboot
```
7. **Verify**: After the reboot, confirm that your system is running with the new configuration, and all services and applications are functioning as expected.
By following these steps, you've successfully migrated your NixOS Flake configuration to include the `clan-core` input and adapted the `outputs` section to work with Clan Core's new machine provisioning method.
## What's next
After creating your flake, you can check out how to add [new machines](./machines.md)
Clan enables encryption of secrets within a Clan flake, ensuring secure sharing among users.
This documentation will guide you through managing secrets with the Clan CLI,
which utilizes the [sops](https://github.com/getsops/sops) format and
integrates with [sops-nix](https://github.com/Mic92/sops-nix) on NixOS machines.
## 1. Generating Keys and Creating Secrets
To begin, generate a key pair:
```shellSession
$ clan secrets key generate
```
**Output**:
```
Public key: age1wkth7uhpkl555g40t8hjsysr20drq286netu8zptw50lmqz7j95sw2t3l7
Generated age private key at '/home/joerg/.config/sops/age/keys.txt' for your user.
Generated age private key at '/home/joerg/.config/sops/age/keys.txt' for your user. Please back it up on a secure location or you will lose access to your secrets.
Also add your age public key to the repository with 'clan secrets users add youruser age1wkth7uhpkl555g40t8hjsysr20drq286netu8zptw50lmqz7j95sw2t3l7' (replace you
user with your user name)
```
⚠️ **Important**: Backup the generated private key securely, or risk losing access to your secrets.
Next, add your public key to the Clan flake repository:
description: "Introducing Clan, a new model for a decentralized network, designed to provide families, smaller groups, and small businesses a platform that’s private, secure, and user-friendly."
authors:
- W
- Qubasa
date: 2024-03-19
---
In a digital age where users are guided increasingly toward submission and dependence, Clan reclaims computing and networking from the ground up.
Clan enables users to build any system from a git repository, automate secret handling, and join devices in a secure darknet. This control extends beyond applications to communication protocols and the operating system itself, putting you fully in charge of your own digital environment.
## Why We're Building Clan
Our mission is simple: to restore fun, freedom, and functionality to computing as an open source project. We believe in building tools that empower users, foster innovation, and challenge the limitations imposed by outdated paradigms. Clan, in its essence, is an open source endeavor; it's our contribution to a future where technology serves humanity, not the other way around.
## How Clan Changes the Game
Clan embodies a new philosophy in system, application, and network design. It enables seamless, secure communication across devices, simplifies software distribution and updates, and offers both public and private network configurations. Here are some of the ways it accomplishes this:
- **Nix as a Foundation:** Imagine a safety net for your computer's operating system, one that lets you make changes or updates without the fear of causing a crash or losing data. Nix simplifies the complexities of system design, ensuring that updates are safe and systems are more reliable.
- **Simplified System Deployment:** Building and managing a computer system, from the operating system to the software you use, often feels like putting together a complex puzzle. With Clan, the puzzle pieces are replaced by a set of building blocks. Leveraging the power of Nix and Clan's innovative toolkit, anyone from tech-savvy administrators to everyday users can create and maintain what we call "full-stack systems" (everything your computer needs to run smoothly).
- **A Leap in Connectivity:** Imagine if you could create private, secure pathways between your devices, bypassing the noisy and often insecure internet. Clan makes this possible through something called "overlay networks." These networks are like private tunnels, allowing your devices to talk to each other securely and directly. With Clan's built-in overlay networks and automatically configured services, connecting your devices becomes seamless, secure, and hassle-free.
- **Security Through Separation:** Clan employs sandboxing and virtual machines, a technology that runs code in isolated environments - so even if you explore new Clans, your system remains protected from potential threats.
- **Reliable:** With Clan, your data and services are preserved for the long haul. We focus on self-hosted backups and integration with the [Fediverse](https://de.wikipedia.org/wiki/Fediverse), a network of interconnected, independent online communities, so your digital life remains uninterrupted and under your control.
## A Glimpse at Clan's Features
- **Social Scaling:** Choose between creating a private sanctuary for your closest contacts, a dynamic space for a self-contained community, or embracing the open web with public Clans anyone can join.
{{ video(name="show_join.webm")}}
- **Seamless VM Integration:** Applications running in virtual machines can appear and behave as if they're part of your main operating system — a blend of power and simplicity.
{{ video(name="show_run.webm")}}
- **Robust Backup Management:** Keep your data safe _forever_ - never worry about cloud services disappearing in 10 years.
{{ asciinema(name="backups.cast") }}
- **Intuitive Secret Management:** Clan simplifies digital security by automating the creation and management of encryption keys and passwords for your services.
{{ asciinema(name="secrets.cast") }}
- **Remote Install:** Set up and manage Clan systems anywhere in the world with just a QR scan or SSH access, making remote installations as easy as snapping a photo or sharing a link.
{{ asciinema(name="nixos-install.cast") }}
## Who Stands to Benefit?
Clan is for anyone and everyone who believes in the power of open source technology to connect, empower, and protect. From system administrators to less tech-savvy individuals, small business owners to privacy-conscious users, Clan offers something for everyone — a way to reclaim control and redefine how we interact with technology.
## Join the Revolution
Ready to control your digital world? Clan is more than a tool—it's a movement. Secure your data, manage your systems easily, or connect with others how you like. Start with Clan for a better digital future.
Connect with us on our [Matrix channel at clan.lol](https://matrix.to/#/#clan:lassul.us) or through our IRC bridges (coming soon).
Want to see the code? Check it out [on our Gitea](https://git.clan.lol/clan/clan-core) or [on GitHub](https://github.com/clan-lol/clan-core).
Or follow our [RSS feed](https://docs.clan.lol/feed_rss_created.xml)!
Join us and be part of changing technology for the better, together.
title: "Dev Report: Introducing the NixOS to JSON Schema Converter"
description: "Discover our new library designed to extract JSON schema interfaces from NixOS modules, streamlining frontend development"
authors:
- DavHau
date: 2024-05-25
slug: jsonschema-converter
---
## Overview
We’ve developed a new library designed to extract interfaces from NixOS modules and convert them into JSON schemas, paving the way for effortless GUI generation. This blog post outlines the motivations behind this development, demonstrates the capabilities of the library, and guides you through leveraging it to create GUIs seamlessly.
## Motivation
In recent months, our team has been exploring various graphical user interfaces (GUIs) to streamline NixOS machine configuration. While our opinionated Clan modules simplify NixOS configurations, there's a need to configure these modules from diverse frontends, such as:
- Command-line interfaces (CLIs)
- Web-based UIs
- Desktop applications
- Mobile applications
- Large Language Models (LLMs)
Given this need, a universal format like JSON is a natural choice. It is already possible as of now, to import json based NixOS configurations, as illustrated below:
`configuration.json`:
```json
{ "networking": { "hostName": "my-machine" } }
```
This configuration can be then imported inside a classic NixOS config:
```nix
{config, lib, pkgs, ...}: {
imports = [
(lib.importJSON ./configuration.json)
];
}
```
This straightforward approach allows us to build a frontend that generates JSON, enabling the configuration of NixOS machines. But, two critical questions arise:
1. How does the frontend learn about existing configuration options?
2. How can it verify user input without running Nix?
Introducing [JSON schema](https://json-schema.org/), a widely supported standard that defines interfaces in JSON and validates input against them.
Validating input against JSON schemas is both efficient and well-supported across numerous programming languages. Using JSON schema validators, you can accurately check configurations like our `configuration.json`.
Certain libraries facilitate straightforward GUI generation from JSON schemas. For instance, the [react-jsonschema-form playground](https://rjsf-team.github.io/react-jsonschema-form/) auto-generates a form for any given schema.
## NixOS Module to JSON Schema Converter
To enable the development of responsive frontends, our library allows the extraction of interfaces from NixOS modules to JSON schemas. Open-sourced for community collaboration, this library supports building sophisticated user interfaces for NixOS.
Here’s a preview of our library's functions exposed through the [clan-core](https://git.clan.lol/clan/clan-core) flake:
- `lib.jsonschema.parseModule` - Generates a schema for a NixOS module.
- `lib.jsonschema.parseOption` - Generates a schema for a single NixOS option.
- `lib.jsonschema.parseOptions` - Generates a schema from an attrset of NixOS options.
Example:
`module.nix`:
```nix
{lib, config, pkgs, ...}: {
# a simple service with two options
options.services.example-web-service = {
enable = lib.mkEnableOption "Example web service";
port = lib.mkOption {
type = lib.types.int;
description = "Port used to serve the content";
};
};
}
```
Converted, using the `parseModule` function:
```shell
$ cd clan-core
$ nix eval --json --impure --expr \
'(import ./lib/jsonschema {}).parseModule ./module.nix' | jq | head
{
"properties": {
"services": {
"properties": {
"example-web-service": {
"properties": {
"enable": {
"default": false,
"description": "Whether to enable Example web service.",
"examples": [
...
```
This utility can also generate interfaces for existing NixOS modules or options.
## GUI for NGINX in Under a Minute
Creating a prototype GUI for the NGINX module using our library and [react-jsonschema-form playground](https://rjsf-team.github.io/react-jsonschema-form/) can be done quickly:
1. Export all NGINX options into a JSON schema using a Nix expression:
3. Open the [react-jsonschema-form playground](https://rjsf-team.github.io/react-jsonschema-form/), select `Blank` and paste the `nginx.json` contents.
This provides a quick look at a potential GUI (screenshot is cropped).
JSON schema mandates the declaration of all required fields upfront, which might be configured implicitly or remain unused. For instance, `services.nginx.virtualHosts.<name>.sslCertificate` must be specified even if SSL isn’t enabled.
### Limited Types
Certain NixOS module types, like `types.functionTo` and `types.package`, do not map straightforwardly to JSON. For full compatibility, adjustments to NixOS modules might be necessary, such as substituting `listOf package` with `listOf str`.
### Parsing NixOS Modules
Currently, our converter relies on the `options` attribute of evaluated NixOS modules, extracting information from the `type.name` attribute, which is suboptimal. Enhanced introspection capabilities within the NixOS module system would be beneficial.
## Future Prospects
We hope these experiments inspire the community, encourage contributions and further development in this space. Share your ideas and contributions through our issue tracker or matrix channel!
## Links
- [Comments on NixOS Discourse](https://discourse.nixos.org/t/introducing-the-nixos-to-json-schema-converter/45948)
- [Source Code of the JSON Schema Library](https://git.clan.lol/clan/clan-core/src/branch/main/lib/jsonschema)
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
