public/docs/disaster-recovery.md

🧯 Disaster Recovery & Upgrades

Note

Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose. Both halves of this doc are about the same underlying discipline: assuming things will eventually fail, and designing so one failure doesn't cascade into a second.

Three independent recovery layers

flowchart TD
    Loss["Something is lost"] --> Type{"What kind?"}
    Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same Packer→Terraform→Ansible<br/>pipeline that built it originally"]
    Type -->|"VM/LXC-level data"| PBS["Layer 2: Proxmox Backup Server<br/>full-host snapshots"]
    Type -->|"App data on the main host"| Kopia["Layer 3: Kopia, volume-level<br/>per-stack, isolated repository"]

Why three layers instead of one comprehensive backup system

A single backup tool covering everything was the simpler design, and it was rejected because "a host" and "a database's data" are fundamentally different kinds of loss with different recovery needs. Rebuilding a lost host from docs/iac-pipeline.md's pipeline is fast and doesn't need a backup at all — the code is the backup. But that same pipeline can't recover yesterday's database contents, which is what Layer 2 and 3 exist for. Trying to make one tool do both jobs well tends to produce a tool that's mediocre at each — full-host imaging is slow and coarse-grained for restoring one app's data, and application-level backup can't reconstruct a host's OS and configuration from scratch efficiently. Three narrow, purpose-fit layers beat one general-purpose one here.

Layer	Covers	Explicitly does not cover
Infrastructure as Code	A host lost entirely — rebuilt from the same pipeline that built it (docs/iac-pipeline.md), against its existing tag-driven config	Secrets — recovered separately via Vault, see docs/secrets-and-pki.md, not part of this layer
Proxmox Backup Server	Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in docs/secrets-and-pki.md	One host's Docker volume data, a large media storage pool, personal data
Kopia	The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in stacks/	Anything outside that one host

Important

The volume-level backup layer (Kopia) runs with read-only access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. A backup tool with write access to the data it protects, or one whose own storage lives on the same disk as that data, defeats a meaningful fraction of its own purpose the moment that disk fails.

Warning

A known, honestly-documented gap: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them. A three-layer recovery model is only as good as its actual coverage, and claiming full coverage when a real gap exists would undermine the credibility of the two layers that do work correctly.

Upgrades: two systems, deliberately not coupled

flowchart LR
    subgraph OS["OS-level (os_update role)"]
        direction TB
        T1["VMs — 3 at a time<br/>+ network-critical host check"]
        T2["Proxmox nodes — 1 at a time<br/>aborts if anything's mid-migration"]
        T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
    end

    subgraph App["App-level (container_update / app_update)"]
        direction TB
        Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
    end

    OS -.completely separate concern.-> App

Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again via a real ping check before moving on — not just that the reboot command itself succeeded, which would miss a host that reboots but fails to come back up cleanly.

Tip

Why app-level updates deliberately don't use batch/rolling deployment logic: each host's update check in docs/registry-and-build.md is an independent, unrelated result — a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism (the kind built into most orchestration tools by default) would treat that expected non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment in the traditional sense, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures that would have been genuinely difficult to diagnose — the symptom would look like "the update job stopped partway through" with no obvious error pointing at the actual cause.

---

Related: docs/iac-pipeline.md, docs/secrets-and-pki.md, docs/orchestration.md, docs/registry-and-build.md, docs/cron-scheduler.md