first draft of the backup article
All checks were successful
buildbot/nix-build .#checks.x86_64-linux.clan-app-pytest Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-bash Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-app-no-breakpoints Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-fakeroot Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-e2fsprogs Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-age Build done.
buildbot/nix-build .#checks.x86_64-linux.check-for-breakpoints Build done.
buildbot/nix-build .#checks.aarch64-darwin.nixos-test-backup Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-installer-apk Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-installer-archlinux Build done.
buildbot/nix-build .#checks.aarch64-linux.nixos-iso-installer Build done.
buildbot/nix-build .#checks.aarch64-linux.nixos-flash-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-nix Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-openssh Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-docs Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-pytest-without-core Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-git Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-webview-ui Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-clan-cli Build done.
buildbot/nix-build .#checks.x86_64-linux.inventory-schema-checks Build done.
buildbot/nix-build .#checks.x86_64-linux.lib-jsonschema-example-valid Build done.
buildbot/nix-build .#checks.x86_64-linux.container Build done.
buildbot/nix-build .#checks.x86_64-linux.deltachat Build done.
buildbot/nix-build .#checks.x86_64-linux.package-clan-cli-docs Build done.
buildbot/nix-build .#checks.aarch64-darwin.nixos-test_install_machine Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-installer-rpm Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-installer-deb Build done.
buildbot/nix-build .#checks.x86_64-linux.borgbackup Build done.
buildbot/nix-build .#checks.aarch64-darwin.nixos-flash-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.nixos-test-backup Build done.
buildbot/nix-build .#checks.x86_64-linux.package-module-docs Build done.
buildbot/nix-build .#checks.aarch64-darwin.nixos-iso-installer Build done.
buildbot/nix-build .#checks.aarch64-linux.nixos-test_install_machine Build done.
buildbot/nix-build .#checks.x86_64-linux.renderClanOptions Build done.
buildbot/nix-build .#checks.x86_64-linux."clan-dep-python3.11-qemu" Build done.
buildbot/nix-build .#checks.aarch64-linux.nixos-test-backup Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-rsync Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-sops Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-sshpass Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-tor Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-zbar Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-clan-app Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-default Build done.
buildbot/nix-build .#checks.x86_64-linux."clan-dep-python3.11-mypy" Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-inventory-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.matrix-synapse Build done.
buildbot/nix-build .#checks.x86_64-linux.package-clan-cli Build done.
buildbot/nix-build .#checks.x86_64-linux.package-clan-ts-api Build done.
buildbot/nix-build .#checks.x86_64-linux.package-default Build done.
buildbot/nix-build .#checks.x86_64-linux.package-editor Build done.
buildbot/nix-build .#checks.x86_64-linux.lib-jsonschema-nix-unit-tests Build done.
buildbot/nix-build .#checks.x86_64-linux.treefmt Build done.
buildbot/nix-build .#checks.x86_64-linux.module-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.package-clan-app Build done.
buildbot/nix-build .#checks.x86_64-linux.package-moonlight-sunshine-accept Build done.
buildbot/nix-build .#checks.x86_64-linux.package-pending-reviews Build done.
buildbot/nix-build .#checks.x86_64-linux.package-tea-create-pr Build done.
buildbot/nix-build .#checks.x86_64-linux.package-docs Build done.
buildbot/nix-build .#checks.x86_64-linux.postgresql Build done.
buildbot/nix-build .#checks.x86_64-linux.nixos-flash-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.package-module-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-install-test-ubuntu-22-04 Build done.
buildbot/nix-build .#checks.x86_64-linux.test-backups Build done.
buildbot/nix-build .#checks.x86_64-linux.package-impure-checks Build done.
buildbot/nix-build .#checks.x86_64-linux.package-inventory-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.package-webview-ui Build done.
buildbot/nix-build .#checks.x86_64-linux.package-zerotier-members Build done.
buildbot/nix-build .#checks.x86_64-linux.package-zerotierone Build done.
buildbot/nix-build .#checks.x86_64-linux.package-zt-tcp-relay Build done.
buildbot/nix-build .#checks.x86_64-linux.nixos-test_install_machine Build done.
buildbot/nix-build .#checks.x86_64-linux.package-function-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.package-merge-after-ci Build done.
buildbot/nix-build .#checks.x86_64-linux.nixos-iso-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.secrets Build done.
buildbot/nix-build .#checks.x86_64-linux.wayland-proxy-virtwl Build done.
buildbot/nix-build .#checks.x86_64-linux.template-minimal Build done.
buildbot/nix-build .#checks.x86_64-linux.package-iso-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.zt-tcp-relay Build done.
buildbot/nix-build .#checks.x86_64-linux.syncthing Build done.
buildbot/nix-build .#checks.x86_64-linux.package-deploy-docs Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-pytest-with-core Build done.
checks / checks-impure (pull_request) Successful in 2m18s
buildbot/nix-build .#checks.x86_64-linux.flash Build done.
buildbot/nix-build .#checks.x86_64-linux.test-installation Build done.
buildbot/nix-eval Build done.
All checks were successful
buildbot/nix-build .#checks.x86_64-linux.clan-app-pytest Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-bash Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-app-no-breakpoints Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-fakeroot Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-e2fsprogs Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-age Build done.
buildbot/nix-build .#checks.x86_64-linux.check-for-breakpoints Build done.
buildbot/nix-build .#checks.aarch64-darwin.nixos-test-backup Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-installer-apk Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-installer-archlinux Build done.
buildbot/nix-build .#checks.aarch64-linux.nixos-iso-installer Build done.
buildbot/nix-build .#checks.aarch64-linux.nixos-flash-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-nix Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-openssh Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-docs Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-pytest-without-core Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-git Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-webview-ui Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-clan-cli Build done.
buildbot/nix-build .#checks.x86_64-linux.inventory-schema-checks Build done.
buildbot/nix-build .#checks.x86_64-linux.lib-jsonschema-example-valid Build done.
buildbot/nix-build .#checks.x86_64-linux.container Build done.
buildbot/nix-build .#checks.x86_64-linux.deltachat Build done.
buildbot/nix-build .#checks.x86_64-linux.package-clan-cli-docs Build done.
buildbot/nix-build .#checks.aarch64-darwin.nixos-test_install_machine Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-installer-rpm Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-installer-deb Build done.
buildbot/nix-build .#checks.x86_64-linux.borgbackup Build done.
buildbot/nix-build .#checks.aarch64-darwin.nixos-flash-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.nixos-test-backup Build done.
buildbot/nix-build .#checks.x86_64-linux.package-module-docs Build done.
buildbot/nix-build .#checks.aarch64-darwin.nixos-iso-installer Build done.
buildbot/nix-build .#checks.aarch64-linux.nixos-test_install_machine Build done.
buildbot/nix-build .#checks.x86_64-linux.renderClanOptions Build done.
buildbot/nix-build .#checks.x86_64-linux."clan-dep-python3.11-qemu" Build done.
buildbot/nix-build .#checks.aarch64-linux.nixos-test-backup Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-rsync Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-sops Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-sshpass Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-tor Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-dep-zbar Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-clan-app Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-default Build done.
buildbot/nix-build .#checks.x86_64-linux."clan-dep-python3.11-mypy" Build done.
buildbot/nix-build .#checks.x86_64-linux.devShell-inventory-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.matrix-synapse Build done.
buildbot/nix-build .#checks.x86_64-linux.package-clan-cli Build done.
buildbot/nix-build .#checks.x86_64-linux.package-clan-ts-api Build done.
buildbot/nix-build .#checks.x86_64-linux.package-default Build done.
buildbot/nix-build .#checks.x86_64-linux.package-editor Build done.
buildbot/nix-build .#checks.x86_64-linux.lib-jsonschema-nix-unit-tests Build done.
buildbot/nix-build .#checks.x86_64-linux.treefmt Build done.
buildbot/nix-build .#checks.x86_64-linux.module-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.package-clan-app Build done.
buildbot/nix-build .#checks.x86_64-linux.package-moonlight-sunshine-accept Build done.
buildbot/nix-build .#checks.x86_64-linux.package-pending-reviews Build done.
buildbot/nix-build .#checks.x86_64-linux.package-tea-create-pr Build done.
buildbot/nix-build .#checks.x86_64-linux.package-docs Build done.
buildbot/nix-build .#checks.x86_64-linux.postgresql Build done.
buildbot/nix-build .#checks.x86_64-linux.nixos-flash-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.package-module-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.package-gui-install-test-ubuntu-22-04 Build done.
buildbot/nix-build .#checks.x86_64-linux.test-backups Build done.
buildbot/nix-build .#checks.x86_64-linux.package-impure-checks Build done.
buildbot/nix-build .#checks.x86_64-linux.package-inventory-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.package-webview-ui Build done.
buildbot/nix-build .#checks.x86_64-linux.package-zerotier-members Build done.
buildbot/nix-build .#checks.x86_64-linux.package-zerotierone Build done.
buildbot/nix-build .#checks.x86_64-linux.package-zt-tcp-relay Build done.
buildbot/nix-build .#checks.x86_64-linux.nixos-test_install_machine Build done.
buildbot/nix-build .#checks.x86_64-linux.package-function-schema Build done.
buildbot/nix-build .#checks.x86_64-linux.package-merge-after-ci Build done.
buildbot/nix-build .#checks.x86_64-linux.nixos-iso-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.secrets Build done.
buildbot/nix-build .#checks.x86_64-linux.wayland-proxy-virtwl Build done.
buildbot/nix-build .#checks.x86_64-linux.template-minimal Build done.
buildbot/nix-build .#checks.x86_64-linux.package-iso-installer Build done.
buildbot/nix-build .#checks.x86_64-linux.zt-tcp-relay Build done.
buildbot/nix-build .#checks.x86_64-linux.syncthing Build done.
buildbot/nix-build .#checks.x86_64-linux.package-deploy-docs Build done.
buildbot/nix-build .#checks.x86_64-linux.clan-pytest-with-core Build done.
checks / checks-impure (pull_request) Successful in 2m18s
buildbot/nix-build .#checks.x86_64-linux.flash Build done.
buildbot/nix-build .#checks.x86_64-linux.test-installation Build done.
buildbot/nix-eval Build done.
This commit is contained in:
parent
0dfa1d969f
commit
dcf4619ab0
130
docs/site/blog/posts/backups.md
Normal file
130
docs/site/blog/posts/backups.md
Normal file
|
@ -0,0 +1,130 @@
|
|||
---
|
||||
title: "Dev Report: Declarative Backups and Restore"
|
||||
description: "An extension to the NixOS module system to declaratively describe how application state is backed up and restored."
|
||||
authors:
|
||||
- Mic92
|
||||
date: 2024-06-24
|
||||
slug: backups
|
||||
---
|
||||
|
||||
Our goal with [Clan](https://clan.lol/) is to give users control over their data.
|
||||
However, with great power comes great responsibility, and owning your data means you also need to take care of backups yourself.
|
||||
|
||||
In our experience, setting up automatic backups is often a tedious process as it requires custom integration of the backup software and
|
||||
the services that produce the state. More importantly than the backup is the restore.
|
||||
Restores are often not well tested or documented, and if not working correctly, they can render the backup useless.
|
||||
|
||||
In Clan, we want to make backup and restore a first-class citizen.
|
||||
Every service should describe what state it produces and how it can be backed up and restored.
|
||||
|
||||
In this article, we will discuss how our backup interface in Clan works.
|
||||
The interface allows different backup software to be used interchangeably and
|
||||
allows module authors to define custom backup and restore logic for their services.
|
||||
|
||||
|
||||
## First Comes the State
|
||||
|
||||
Our services are built from Clan modules. Clan modules are essentially [NixOS modules](https://wiki.nixos.org/wiki/NixOS_modules), the basic configuration components of NixOS.
|
||||
However, we have enhanced them with additional features provided by Clan and restricted certain option types to enable configuration through a [graphical interface](https://docs.clan.lol/blog/2024/05/25/jsonschema-converter/).
|
||||
|
||||
In a simple case, this can be just a bunch of directories, such as what we define for our [ZeroTier](https://www.zerotier.com/) VPN service.
|
||||
|
||||
```nix
|
||||
{
|
||||
clan.core.state.zerotier.folders = [ "/var/lib/zerotier-one" ];
|
||||
}
|
||||
```
|
||||
|
||||
For other systems, we need more complex backup and restore logic.
|
||||
For each state, we can provide custom command hooks for backing up and restoring.
|
||||
|
||||
In our PostgreSQL module, for example, we define `preBackupCommand` and `postRestoreCommand` to use `pg_dump` and `pg_restore` to backup and restore individual databases:
|
||||
|
||||
```nix
|
||||
preBackupCommand = ''
|
||||
# ...
|
||||
runuser -u postgres -- pg_dump ${compression} --dbname=${db.name} -Fc -c > "${current}.tmp"
|
||||
# ...
|
||||
'';
|
||||
postRestoreCommand = ''
|
||||
# ...
|
||||
runuser -u postgres -- dropdb "${db.name}"
|
||||
runuser -u postgres -- pg_restore -C -d postgres "${current}"
|
||||
# ...
|
||||
'';
|
||||
```
|
||||
|
||||
## Then the Backup
|
||||
|
||||
Our CLI unifies the different backup providers in one [interface](https://docs.clan.lol/reference/cli/backups/).
|
||||
|
||||
As of now, we support backups using [BorgBackup](https://www.borgbackup.org/) and
|
||||
a backup module called "localbackup" based on [rsnapshot](https://rsnapshot.org/), optimized for backup on locally-attached storage media.
|
||||
|
||||
To use different backup software, a module needs to set the options provided by our backup interface.
|
||||
The following is a toy example that uses the `tar` program to perform backup and restore to illustrate how the backup interface works:
|
||||
|
||||
```nix
|
||||
clan.core.backups.providers.tar = {
|
||||
list = ''
|
||||
echo /var/lib/system-back.tar
|
||||
'';
|
||||
create = let
|
||||
uniqueFolders = lib.unique (
|
||||
lib.flatten (lib.mapAttrsToList (_name: state: state.folders) config.clan.core.state)
|
||||
);
|
||||
in ''
|
||||
# FIXME: a proper implementation should also run `state.preBackupCommand` of each state
|
||||
if [ -f /var/lib/system-back.tar ]; then
|
||||
tar -uvpf /var/lib/system-back.tar ${builtins.toString uniqueFolders}
|
||||
else
|
||||
tar -cvpf /var/lib/system-back.tar ${builtins.toString uniqueFolders}
|
||||
fi
|
||||
'';
|
||||
restore = ''
|
||||
IFS=':' read -ra FOLDER <<< "''$FOLDERS"
|
||||
echo "${FOLDER[@]}" > /run/folders-to-restore.txt
|
||||
tar -xvpf /var/lib/system-back.tar -C / -T /run/folders-to-restore.txt
|
||||
'';
|
||||
};
|
||||
```
|
||||
|
||||
For better real-world implementations, check out the implementations for [BorgBackup](https://git.clan.lol/clan/clan-core/src/branch/main/clanModules/borgbackup/default.nix)
|
||||
and [localbackup](https://git.clan.lol/clan/clan-core/src/branch/main/clanModules/localbackup/default.nix).
|
||||
|
||||
## What It Looks Like to the End User
|
||||
|
||||
After following the guide for configuring a [backup](https://docs.clan.lol/getting-started/backups/),
|
||||
users can use the CLI to create backups, list them, and restore them.
|
||||
|
||||
Backups can be created through the CLI like this:
|
||||
|
||||
```
|
||||
clan backups create web01
|
||||
```
|
||||
|
||||
BorgBackup will also create backups itself every day by default.
|
||||
|
||||
Completed backups can be listed like this:
|
||||
|
||||
```
|
||||
clan backups list web01
|
||||
...
|
||||
web01::u366395@u366395.your-storagebox.de:/./borgbackup::web01-web01-2024-06-18T01:00:00
|
||||
web03::u366395@u366395.your-storagebox.de:/./borgbackup::web01-web01-2024-06-19T01:00:00
|
||||
```
|
||||
One cool feature of our backup system is that it is aware of individual services/applications.
|
||||
Let's say we want to restore the state of our [Matrix](https://matrix.org/) chat server; we can just specify it like this:
|
||||
|
||||
```
|
||||
clan backups restore --service matrix-synapse web01 borgbackup web03::u366395@u366395.your-storagebox.de:/./borgbackup::web01-web01-2024-06-19T01:00:00
|
||||
```
|
||||
|
||||
In this case, it will first stop the matrix-synapse systemd service, then delete the [PostgreSQL](https://www.postgresql.org/) database, restore the database from the backup, and then start the matrix-synapse service again.
|
||||
|
||||
## Future work
|
||||
|
||||
Currently, our backup implementation backs up filesystem state from running services.
|
||||
This can lead to inconsistencies if applications change the state while the backup is running.
|
||||
In the future, we hope to make backups more atomic by backing up a filesystem snapshot instead of normal directories.
|
||||
This, however, requires the use of modern filesystems that support these features.
|
Loading…
Reference in New Issue
Block a user