diff --git a/assets/disaster-recovery-layers.svg b/assets/disaster-recovery-layers.svg new file mode 100644 index 0000000..c17406a --- /dev/null +++ b/assets/disaster-recovery-layers.svg @@ -0,0 +1,4 @@ + + + +
Something is lost
What kind?
Layer 1: Infrastructure as Code
rebuilt via the same Packer -> Terraform -> Ansible pipeline that built it originally
Layer 2: Proxmox Backup Server
full-host snapshots
Layer 3: Kopia, volume-level
per-stack, isolated repository
a host, entirely
VM/LXC-level data
app data, main host
\ No newline at end of file diff --git a/assets/disaster-recovery-upgrades.svg b/assets/disaster-recovery-upgrades.svg new file mode 100644 index 0000000..130c6e2 --- /dev/null +++ b/assets/disaster-recovery-upgrades.svg @@ -0,0 +1,4 @@ + + + +
OS-LEVEL (os_update role)
VMs
3 at a time + network-critical host check
Proxmox nodes
1 at a time, aborts if anything's mid-migration
Backup server
1 at a time, aborts if a backup job is running
APP-LEVEL (container_update / app_update)
Per-stack update strategy
independent, unrelated checks
completely separate concern
\ No newline at end of file diff --git a/docs/disaster-recovery.md b/docs/disaster-recovery.md index ea2f89a..89d1d0b 100644 --- a/docs/disaster-recovery.md +++ b/docs/disaster-recovery.md @@ -5,13 +5,7 @@ ## Three independent recovery layers -```mermaid -flowchart TD - Loss["Something is lost"] --> Type{"What kind?"} - Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code
rebuilt via the same Packer→Terraform→Ansible
pipeline that built it originally"] - Type -->|"VM/LXC-level data"| PBS["Layer 2: Proxmox Backup Server
full-host snapshots"] - Type -->|"App data on the main host"| Kopia["Layer 3: Kopia, volume-level
per-stack, isolated repository"] -``` +![Something lost branches three ways depending on what kind of loss it is: an entire host rebuilds through the same Packer, Terraform, and Ansible pipeline that built it originally; VM/LXC-level data recovers through Proxmox Backup Server's full-host snapshots; app data on the main Docker host recovers through Kopia's volume-level, per-stack isolated repository](../assets/disaster-recovery-layers.svg) ## Why three layers instead of one comprehensive backup system @@ -23,6 +17,34 @@ A single backup tool covering everything was the simpler design, and it was reje | Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data | | Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host | +This is the real Kopia repository, not a description of it — a dedicated set of directories on its own separate disk, confirmed live: + +``` +$ ls /mnt/samsungssd +kopia-cache kopia-config kopia-repo kopia-tls +$ df -h /mnt/samsungssd +Filesystem Size Used Avail Use% Mounted on +/dev/nvme0n1p1 234G 65G 167G 28% /mnt/samsungssd +``` + +Layer 1's "rebuilt against its existing tag-driven config" claim is a real, checkable mechanism too — every role in the pipeline declares which of the three rebuild stages it belongs to: + +``` +$ head -12 roles_metadata.yml +roles_metadata: + base_users: + initialization: true + provisioning: false + compliance: false + description: "Base user creation tasks" + + disk_resize: + initialization: true + provisioning: false + compliance: true + description: "Disk resizing tasks" +``` + > [!IMPORTANT] > The volume-level backup layer (Kopia) runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. A backup tool with write access to the data it protects, or one whose own storage lives on the same disk as that data, defeats a meaningful fraction of its own purpose the moment that disk fails. @@ -31,22 +53,7 @@ A single backup tool covering everything was the simpler design, and it was reje ## Upgrades: two systems, deliberately not coupled -```mermaid -flowchart LR - subgraph OS["OS-level (os_update role)"] - direction TB - T1["VMs — 3 at a time
+ network-critical host check"] - T2["Proxmox nodes — 1 at a time
aborts if anything's mid-migration"] - T3["Backup server — 1 at a time
aborts if a backup job is running"] - end - - subgraph App["App-level (container_update / app_update)"] - direction TB - Strategy["Per-stack update strategy
independent, unrelated checks"] - end - - OS -.completely separate concern.-> App -``` +![OS-level updates (VMs 3 at a time with a network-critical host check, Proxmox nodes 1 at a time aborting on mid-migration, the backup server 1 at a time aborting on a running backup job) are a completely separate concern from app-level updates, where every stack's own update strategy runs its own independent, unrelated check](../assets/disaster-recovery-upgrades.svg) Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again via a real ping check before moving on — not just that the reboot command itself succeeded, which would miss a host that reboots but fails to come back up cleanly.