# 🧯 Disaster Recovery & Upgrades

> [!NOTE]
> Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose.

## Three independent recovery layers

```mermaid
flowchart TD
    Loss["Something is lost"] --> Type{"What kind?"}
    Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same pipeline<br/>that built it originally"]
    Type -->|"VM/LXC-level data"| PBS["Layer 2: VM/LXC backups<br/>full-host snapshots"]
    Type -->|"App data on the main host"| Kopia["Layer 3: Volume-level backups<br/>per-stack, isolated repository"]
```

| Layer | Covers | Explicitly does not cover |
|---|---|---|
| Infrastructure as Code | A host lost entirely — rebuilt from the same pipeline that built it, against its existing config | Secrets (recovered separately, not part of this layer) |
| VM/LXC-level backup | Full VM/LXC backups, including the secrets platform's own seal state and data | One host's Docker volume data, a large media storage pool, personal data |
| Volume-level backup | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host | Anything outside that one host |

> [!IMPORTANT]
> The volume-level backup layer runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up.

> [!WARNING]
> **A known, honestly-documented gap**: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them.

## Upgrades: two systems, deliberately not coupled

```mermaid
flowchart LR
    subgraph OS["OS-level"]
        direction TB
        T1["VMs — 3 at a time<br/>+ network-critical host check"]
        T2["Hypervisor nodes — 1 at a time<br/>aborts if anything's mid-migration"]
        T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
    end

    subgraph App["App-level"]
        direction TB
        Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
    end

    OS -.completely separate concern.-> App
```

Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again before moving on — not just that the reboot command succeeded.

> [!TIP]
> **Why app-level updates deliberately don't use batch/rolling deployment logic**: each host's update check is an independent, unrelated result — a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism would treat that non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures.

---
Related: [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md)