public/docs/disaster-recovery.md

🧯 Disaster Recovery & Upgrades

Note

Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose. Both halves of this doc are about the same underlying discipline: assuming things will eventually fail, and designing so one failure doesn't cascade into a second.

Three Independent Recovery Layers

Why Three Layers Instead of One Comprehensive Backup System

A single backup tool covering everything was the simpler design, and it was rejected because "a host" and "a database's data" are fundamentally different kinds of loss with different recovery needs. Rebuilding a lost host from docs/iac-pipeline.md's pipeline is fast and doesn't need a backup at all — the code is the backup. But that same pipeline can't recover yesterday's database contents, which is what Layer 2 and 3 exist for. Trying to make one tool do both jobs well tends to produce a tool that's mediocre at each — full-host imaging is slow and coarse-grained for restoring one app's data, and application-level backup can't reconstruct a host's OS and configuration from scratch efficiently. Three narrow, purpose-fit layers beat one general-purpose one here.

Layer	Covers	Explicitly does not cover
Infrastructure as Code	A host lost entirely, VM or LXC alike — rebuilt from the same pipeline that built it (docs/iac-pipeline.md), against its existing tag-driven config	Secrets — recovered separately via Vault, see docs/secrets-and-pki.md, not part of this layer
Proxmox Backup Server	Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in docs/secrets-and-pki.md	One host's Docker volume data, a large media storage pool, personal data
Kopia	The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in stacks/	Anything outside that one host

This is the real Kopia repository, not a description of it — a dedicated set of directories on its own separate disk, confirmed live:

$ ls /mnt/samsungssd
kopia-cache  kopia-config  kopia-repo  kopia-tls
$ df -h /mnt/samsungssd
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p1  234G   65G  167G  28% /mnt/samsungssd

Layer 1's "rebuilt against its existing tag-driven config" claim is a real, checkable mechanism too — every role in the pipeline declares which of the three rebuild stages it belongs to:

$ head -12 roles_metadata.yml
roles_metadata:
  base_users:
    initialization: true
    provisioning: false
    compliance: false
    description: "Base user creation tasks"

  disk_resize:
    initialization: true
    provisioning: false
    compliance: true
    description: "Disk resizing tasks"

Important

The volume-level backup layer (Kopia) runs with read-only access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. A backup tool with write access to the data it protects, or one whose own storage lives on the same disk as that data, defeats a meaningful fraction of its own purpose the moment that disk fails.

Warning

A known, honestly-documented gap: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them. A three-layer recovery model is only as good as its actual coverage, and claiming full coverage when a real gap exists would undermine the credibility of the two layers that do work correctly.

Upgrades: two systems, deliberately not coupled

Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again via a real ping check before moving on — not just that the reboot command itself succeeded, which would miss a host that reboots but fails to come back up cleanly.

Tip

Why app-level updates don't use batch/rolling deployment logic: each host's update check in docs/registry-and-build.md produces an independent result — a pinned version correctly reporting "nothing to do" isn't a failure, it's the expected outcome for that host. A rolling-batch mechanism (the default in most orchestration tools) doesn't know that: it reads a non-change as a stalled batch and halts the rest of the run to protect against what it assumes is a bad deployment. Every other host's update gets silently skipped for the rest of that pass, and the only visible symptom is "the job stopped partway through" — no error, nothing pointing at which host or which assumption caused it. App-level updates route around this entirely by never batching in the first place.

---

Related: docs/iac-pipeline.md, docs/secrets-and-pki.md, docs/orchestration.md, docs/registry-and-build.md, docs/cron-scheduler.md