public/docs/disaster-recovery.md

🧯 Disaster Recovery & Upgrades

Note

Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose.

Three independent recovery layers

flowchart TD
    Loss["Something is lost"] --> Type{"What kind?"}
    Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same pipeline<br/>that built it originally"]
    Type -->|"VM/LXC-level data"| PBS["Layer 2: VM/LXC backups<br/>full-host snapshots"]
    Type -->|"App data on the main host"| Kopia["Layer 3: Volume-level backups<br/>per-stack, isolated repository"]

Layer	Covers	Explicitly does not cover
Infrastructure as Code	A host lost entirely — rebuilt from the same pipeline that built it, against its existing config	Secrets (recovered separately, not part of this layer)
VM/LXC-level backup	Full VM/LXC backups, including the secrets platform's own seal state and data	One host's Docker volume data, a large media storage pool, personal data
Volume-level backup	The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host	Anything outside that one host

Important

The volume-level backup layer runs with read-only access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up.

Warning

A known, honestly-documented gap: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them.

Upgrades: two systems, deliberately not coupled

flowchart LR
    subgraph OS["OS-level"]
        direction TB
        T1["VMs — 3 at a time<br/>+ network-critical host check"]
        T2["Hypervisor nodes — 1 at a time<br/>aborts if anything's mid-migration"]
        T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
    end

    subgraph App["App-level"]
        direction TB
        Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
    end

    OS -.completely separate concern.-> App

Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again before moving on — not just that the reboot command succeeded.

Tip

Why app-level updates deliberately don't use batch/rolling deployment logic: each host's update check is an independent, unrelated result — a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism would treat that non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures.

---

Related: docs/orchestration.md, docs/secrets-and-pki.md