# 🧯 Disaster Recovery & Upgrades

> [!NOTE]
> Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose. Both halves of this doc are about the same underlying discipline: assuming things will eventually fail, and designing so one failure doesn't cascade into a second.

## Three independent recovery layers

```mermaid
flowchart TD
    Loss["Something is lost"] --> Type{"What kind?"}
    Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same Packer→Terraform→Ansible<br/>pipeline that built it originally"]
    Type -->|"VM/LXC-level data"| PBS["Layer 2: Proxmox Backup Server<br/>full-host snapshots"]
    Type -->|"App data on the main host"| Kopia["Layer 3: Kopia, volume-level<br/>per-stack, isolated repository"]
```

## Why three layers instead of one comprehensive backup system

A single backup tool covering everything was the simpler design, and it was rejected because "a host" and "a database's data" are fundamentally different kinds of loss with different recovery needs. Rebuilding a lost host from [`docs/iac-pipeline.md`](iac-pipeline.md)'s pipeline is fast and doesn't need a backup at all — the *code* is the backup. But that same pipeline can't recover yesterday's database contents, which is what Layer 2 and 3 exist for. Trying to make one tool do both jobs well tends to produce a tool that's mediocre at each — full-host imaging is slow and coarse-grained for restoring one app's data, and application-level backup can't reconstruct a host's OS and configuration from scratch efficiently. Three narrow, purpose-fit layers beat one general-purpose one here.

| Layer | Covers | Explicitly does not cover |
|---|---|---|
| Infrastructure as Code | A host lost entirely — rebuilt from the same pipeline that built it ([`docs/iac-pipeline.md`](iac-pipeline.md)), against its existing tag-driven config | Secrets — recovered separately via Vault, see [`docs/secrets-and-pki.md`](secrets-and-pki.md), not part of this layer |
| Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data |
| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host |

> [!IMPORTANT]
> The volume-level backup layer (Kopia) runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. A backup tool with write access to the data it protects, or one whose own storage lives on the same disk as that data, defeats a meaningful fraction of its own purpose the moment that disk fails.

> [!WARNING]
> **A known, honestly-documented gap**: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them. A three-layer recovery model is only as good as its actual coverage, and claiming full coverage when a real gap exists would undermine the credibility of the two layers that do work correctly.

## Upgrades: two systems, deliberately not coupled

```mermaid
flowchart LR
    subgraph OS["OS-level (os_update role)"]
        direction TB
        T1["VMs — 3 at a time<br/>+ network-critical host check"]
        T2["Proxmox nodes — 1 at a time<br/>aborts if anything's mid-migration"]
        T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
    end

    subgraph App["App-level (container_update / app_update)"]
        direction TB
        Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
    end

    OS -.completely separate concern.-> App
```

Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again via a real ping check before moving on — not just that the reboot command itself succeeded, which would miss a host that reboots but fails to come back up cleanly.

> [!TIP]
> **Why app-level updates deliberately don't use batch/rolling deployment logic**: each host's update check in [`docs/registry-and-build.md`](registry-and-build.md) is an independent, unrelated result — a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism (the kind built into most orchestration tools by default) would treat that expected non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment in the traditional sense, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures that would have been genuinely difficult to diagnose — the symptom would look like "the update job stopped partway through" with no obvious error pointing at the actual cause.

---
Related: [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/orchestration.md`](orchestration.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/cron-scheduler.md`](cron-scheduler.md)