This repository contains nixos modules and terraform code that powers clan.lol. The website and git are currently hosted on Hetzner. The Jitsi server is hosted on Vultr.

Adding a New Admin User

To add a new admin user, follow these steps:

User generates an age key:

The new user runs:
```
$ clan secrets key generate
```
This creates an age key pair, which is used for secret management.
User provides credentials to an existing admin:

The user shares both of the following with a current admin:
- Their SSH public key
- Their age public key (found in ~/.config/sops/age/keys.txt or ~/Library/Application Support/sops/age/keys.txt on macOS)

Admin adds the user:

The admin runs:

$ clan secrets users add <username> <age-key>
$ clan secrets groups add-user admins <username>

Replace <username> and <age-key> with the actual values.

Admin updates configuration:

Add the new user to the modules/admins.nix file.

The new admin user will now have access according to the configuration.

Joining the clan-infra Zerotier Network

To connect your device to the clan-infra Zerotier network:

Get the Zerotier network ID:

$ clan vars get web01 zerotier-network-zerotier-claninfra/network-id
9581a9b77e4e8101

Configure your device to join the network:

Add the following to your NixOS configuration:

services.zerotierone.joinNetworks = [
  "9581a9b77e4e8101" # clan-infra network
];

Find your device's Zerotier ID:

After starting Zerotier, run:
```
sudo zerotier-cli info
```
The output will look like:
```
200 info <myid> 1.14.2 ONLINE
```
Note your <myid>.
Authorize your device on the network:

Add your <myid> to the controller's allow list in machines/flake-module.nix:
```
roles.controller.machines.web01.settings.allowedIds = [
  "<myid>" # your device
];
```
Once that change is deployed to web01, the controller admits your device and it will be connected to the clan-infra Zerotier network.

web01

Instance type: ax162-r
CPU: AMD EPYC™ 9454P
RAM: 256 GB DDR5 ECC
Storage: 2 x 1.92 TB NVMe

Initial setup

To install the system, you can run the following command:

$ clan machines install web01 --update-hardware-config nixos-facter --no-reboot

Then you can run the following script to reboot the machine and unlock the encrypted root filesystem:

$ ./machines/web01/reboot.sh

Deploy new configuration

$ clan machines update web01

web02

Instance type: vc2-2c-4gb
CPU: 2 Intel vCPU cores
RAM: 4 GB
Storage: 80 GB SSD

Initial setup

$ nix run .#terraform

Deploy new configuration

$ clan machines update web02

Redeploy server

To redeploy the server without running terraform destroy which will take down the clan.lol DNS:

# Run `apply` script first to ensure `terraform init` gets run
$ nix run .#terraform
$ nix run .#terraform.terraform -- apply -replace "vultr_instance.web02"

Destroy server

To destroy just the server without taking down the clan.lol DNS:

# Run `apply` script first to ensure `terraform init` gets run
$ nix run .#terraform
$ nix run .#terraform.terraform -- destroy -target "vultr_instance.web02"

monitoring01

Instance type: cx43
CPU: 8 Intel/AMD vCPU cores
RAM: 16 GB
Storage: 160 GB NVMe

Initial setup

$ nix run .#terraform

Deploy new configuration

$ clan machines update monitoring01

Redeploy server

To redeploy the server without running terraform destroy which will take down the clan.lol DNS:

# Run `apply` script first to ensure `terraform init` gets run
$ nix run .#terraform
$ nix run .#terraform.terraform -- apply -replace "hcloud_server.monitoring01"

Destroy server

To destroy just the server without taking down the clan.lol DNS:

# Run `apply` script first to ensure `terraform init` gets run
$ nix run .#terraform
$ nix run .#terraform.terraform -- destroy -target "hcloud_server.monitoring01"

build01

Instance type: rx170
CPU: Ampere® Altra® Q80-30
RAM: 128 GB DDR4 ECC
Storage: 2 x 960 GB NVMe

Initial setup

To install the system, you can run the following command:

$ nix run .#terraform

Deploy new configuration

To access this machine, you'll need to add this to your SSH config:

{
  programs.ssh.extraConfig = ''
    Host build01
      ProxyJump tunnel@clan.lol
      Hostname build01.vpn.clan.lol
  '';
}

$ clan machines update build01

build-x86-01

Instance type: Hetzner dedicated server (AMD)
Platform: x86_64-linux
Max parallel jobs: 32
Features: big-parallel, kvm, nixos-test, uid-range, recursive-nix

Initial setup

To install the system, you can run the following command:

$ clan machines install build-x86-01 --update-hardware-config nixos-facter --no-reboot

Deploy new configuration

$ clan machines update build-x86-01

build02

Instance type: Apple Mac mini (2024) (Mac16,10)
CPU: Apple M4 chip with 10-core CPU, 10-core GPU, 16-core Neural Engine
RAM: 24 GB unified memory
Storage: 512 GB SSD

Initial setup

Install Nix using the experimental Nix installer

curl --proto '=https' --tlsv1.2 -sSf -L https://artifacts.nixos.org/nix-installer | sh -s -- install

Enable Screen Sharing in System Settings > General > Sharing

You can leave both Anyone may request permission to control screen and VNC viewers may control screen with password disabled as macOS will allow you to control the screen by connecting with your macOS username and password.

Clone this repo into a temporary directory

nix run nixpkgs#git -- clone https://git.clan.lol/clan/clan-infra.git temp-bootstrap

Install nix-darwin from the temporary directory

nix run nix-darwin -- switch --flake ./temp-bootsrap

Log in to Tailscale
```
sudo tailscale up
```
Enable Allow full disk access for remote users and Allow access for all users in System Settings > General > Sharing > Remote Login
Delete the temporary directory
```
rm -rf ./temp-bootstrap
```

Deploy new configuration

To access this machine, you'll need to add this to your SSH config:

{
  programs.ssh.extraConfig = ''
    Host build02
      ProxyJump <clanuser>@clan.lol
      Hostname build02.vpn.clan.lol
  '';
}

build04

Instance type: Apple Mac mini (2024) (Mac16,10)
CPU: Apple M4 chip with 10-core CPU, 10-core GPU, 16-core Neural Engine
RAM: 32 GB unified memory
Storage: 1 TB SSD

Initial setup

Change initial password

ssh customer@build04.clan.lol passwd
ssh customer@build04.clan.lol security set-keychain-password

Install Nix using the experimental Nix installer

curl --proto '=https' --tlsv1.2 -sSf -L https://artifacts.nixos.org/nix-installer | sh -s -- install

Deploy initial nix-darwin configuration
```
clan machines update build04
```

Deploy new configuration

$ clan machines update build04

Adding new users

Add them to the configuration.

The user can create an age key:

$ clan secrets key generate

The private key (identity in age terms) and public key (recipient in age terms) are stored in ~/.config/sops/age/keys.txt (~/Library/Application Support/sops/age/keys.txt on macOS).

Add the new user's age key:

$ clan secrets users add <user> <age-key>

Add the new user as an admin:

$ clan secrets groups add-user admins <user>

Mailboxes that filter nothing and keep nothing

w@clan.lol carries noJunkFilter = true and purgeDownloaded = true in modules/mailserver.nix. It is a POP3-and-SMTP-only mailbox, by request: every message reaches INBOX, and nothing stays on the server once it has been downloaded.

noJunkFilter takes two separate changes because two separate things file junk. rspamd gets a local.d/settings.conf rule matching that recipient with want_spam = yes, a full bypass: no reject at 15, no greylisting, no X-Spam/X-Spamd-Result header, no history row. Nulling the individual actions is not enough, because upstream sets extended_spam_headers = true and every scanned message would still be tagged. The account also gets a sieve script of fileinto "INBOX"; stop;, because dovecot's junk filter is a server-wide after script; fileinto plus stop cancels the implicit keep, which is the only thing that stops the sequence before that script runs.

Two consequences. The rspamd rule matches per SMTP transaction, so a message addressed to this mailbox and a filtered one bypasses filtering for both. And want_spam must never be keyed on the authenticated sender instead: it would skip dkim_signing too and outbound mail would leave unsigned.

One message type still bounces: anything containing the literal GTUBE test pattern. rspamd matches GTUBE while parsing and sets a reject pre-result, before settings apply, so want_spam cannot undo it. GTUBE is therefore useless for probing this setup.

purgeDownloaded adds the account to mail-purge.service, a oneshot that runs on activation and every five minutes. It drains Junk back into INBOX (clearing \Seen first, so mail nobody downloaded is not destroyed in the same pass), expunges every \Seen message (POP3 RETR sets \Seen, so a downloaded message is a read message), and empties Sent, Drafts and Trash unconditionally. Dovecot has no delete-after-RETR setting, which is why this is a timer. It also makes these single-device mailboxes: a second client, or a reinstall, finds an empty INBOX.

Sent mail is never copied server-side - there is no always_bcc - but rspamd's redis history used to keep the subject, sender and recipients of every scanned message, submission included, in /var/lib/redis-rspamd. It is now off (local.d/history_redis.conf), so the rspamd web UI history is empty and verdicts have to be read from the journal:

$ journalctl -u rspamd --since -10min | grep rspamd_task_write_log

Disabling it stops new rows but leaves whatever redis already held. Purge that once by hand on web01:

$ redis-cli -s /run/redis-rspamd/redis.sock --scan --pattern 'rs_history*' \
    | xargs -r redis-cli -s /run/redis-rspamd/redis.sock del

The maildirs of purged accounts are excluded from the borg backup (modules/web01/borgbackup.nix), so mail that was expunged does not live on in the archives - and a restore will not bring those mailboxes back. That is the intent, not a bug.

Postfix still keeps queue files until a message is delivered or bounced, and journal lines (envelope sender, recipients, message-id, verdict - no bodies, no subjects) are kept as usual.

The behaviour is a flake check, checks/mail-unfiltered.nix. It takes the rspamd rule, the sieve script and the purge script straight out of web01's evaluated config, so it cannot drift from what ships. Three nodes - the mailserver, a client, and an SMTP sink everything non-local relays to - and thirteen subtests covering the three promises: nothing is filtered, nothing downloaded remains on disk, and nothing submitted is kept. It needs KVM:

$ nix build .#checks.x86_64-linux.mail-unfiltered -L

To probe the live server instead, from a residential connection:

$ nix run .#spam-test

It sends pkgs/spam-test/spam-test.eml - a forged paypal.com sender with no Date and no Message-ID - to infra@clan.lol, which must answer 554, and then to every mailbox with filtering off, which must answer 250, and exits non-zero if either answer is wrong. The recipient list comes out of web01's evaluated config. A 554 on the control is a pass: it means the reject action is live. Do not send from a normal mail provider (SPF and DKIM pass, the message scores near zero) and do not probe with GTUBE.

Update DNS

Currently DNS can't be updated separately to the machines, so you'll need to deploy the entire Terraform configuration:

$ nix run .#terraform

storinator01

Instance type: Storinator Q30
CPU: Intel Xeon Silver 4216 (16C/32T)
RAM: 128 GB DDR5 ECC
Storage:
- OS: 2 x 500GB SATA SSD
- Data: 18 HDDs in zraid2 + 1 Spare == 200TB

Deploy new configuration

To access this machine, you'll need to add this to your SSH config:

{
  programs.ssh.extraConfig = ''
    Host storinator01
      ProxyJump <clan-user>@clan.lol
      Hostname storinator01.vpn.clan.lol
  '';
}

$ clan machines update storinator01

Adding a new machine

Copy an existing machine
Run clan vars generate <machine>
If you aren't using Terraform to provision the server, make sure to add the Terraform deployment SSH key to your server which you can find by running:
```
$ nix run .#terraform.terraform -- init
$ nix run .#terraform.terraform -- state show tls_private_key.ssh_deploy_key
```
nix run .#terraform to run the initial deploy

To add a new project to CI

CI is nixbot, running at https://nixbot.clan.lol. It builds the .#checks flake output of the default branch and of every pull request.

Log in at https://nixbot.clan.lol via Gitea.
If the repository is missing from the list, press "refresh repos".
Press "enable" next to the repository. Requires instance admin (wheel user on web01) or admin permission on the repository in Gitea; write access only allows restarting and cancelling builds.

nixbot registers a webhook and builds the default branch. Anything else you want built (packages, NixOS machines) must be re-exported under .#checks; test locally with nix flake check -L or nix-fast-build.

Commit statuses use the buildbot/ prefix (buildbot/nix-eval, buildbot/nix-build), so existing branch protection rules keep working.