thesabear
20f39f3b6d
Completes the docs/ set - all 16 systems now have a full deep-dive page in the diagram+callout+table style. Architecture overview covers the 4 trust boundaries and 2 traffic planes; IaC pipeline is a fresh Packer->Terraform->Ansible writeup replacing the stale old-content/ snapshot; disaster recovery covers the 3 independent backup layers and the deliberately decoupled OS/app upgrade systems.
54 lines
3.2 KiB
Markdown
54 lines
3.2 KiB
Markdown
# 🧯 Disaster Recovery & Upgrades
|
|
|
|
> [!NOTE]
|
|
> Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose.
|
|
|
|
## Three independent recovery layers
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
Loss["Something is lost"] --> Type{"What kind?"}
|
|
Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same pipeline<br/>that built it originally"]
|
|
Type -->|"VM/LXC-level data"| PBS["Layer 2: VM/LXC backups<br/>full-host snapshots"]
|
|
Type -->|"App data on the main host"| Kopia["Layer 3: Volume-level backups<br/>per-stack, isolated repository"]
|
|
```
|
|
|
|
| Layer | Covers | Explicitly does not cover |
|
|
|---|---|---|
|
|
| Infrastructure as Code | A host lost entirely — rebuilt from the same pipeline that built it, against its existing config | Secrets (recovered separately, not part of this layer) |
|
|
| VM/LXC-level backup | Full VM/LXC backups, including the secrets platform's own seal state and data | One host's Docker volume data, a large media storage pool, personal data |
|
|
| Volume-level backup | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host | Anything outside that one host |
|
|
|
|
> [!IMPORTANT]
|
|
> The volume-level backup layer runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up.
|
|
|
|
> [!WARNING]
|
|
> **A known, honestly-documented gap**: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them.
|
|
|
|
## Upgrades: two systems, deliberately not coupled
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph OS["OS-level"]
|
|
direction TB
|
|
T1["VMs — 3 at a time<br/>+ network-critical host check"]
|
|
T2["Hypervisor nodes — 1 at a time<br/>aborts if anything's mid-migration"]
|
|
T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
|
|
end
|
|
|
|
subgraph App["App-level"]
|
|
direction TB
|
|
Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
|
|
end
|
|
|
|
OS -.completely separate concern.-> App
|
|
```
|
|
|
|
Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again before moving on — not just that the reboot command succeeded.
|
|
|
|
> [!TIP]
|
|
> **Why app-level updates deliberately don't use batch/rolling deployment logic**: each host's update check is an independent, unrelated result — a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism would treat that non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures.
|
|
|
|
---
|
|
Related: [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md)
|