docs: add architecture, IaC pipeline, and disaster recovery write-ups
Completes the docs/ set - all 16 systems now have a full deep-dive page in the diagram+callout+table style. Architecture overview covers the 4 trust boundaries and 2 traffic planes; IaC pipeline is a fresh Packer->Terraform->Ansible writeup replacing the stale old-content/ snapshot; disaster recovery covers the 3 independent backup layers and the deliberately decoupled OS/app upgrade systems.
This commit is contained in:
3 files changed
+182
No files matched your search
@@ -0,0 +1,53 @@
|
||||
# 🧯 Disaster Recovery & Upgrades
|
||||
|
||||
> [!NOTE]
|
||||
> Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose.
|
||||
|
||||
## Three independent recovery layers
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Loss["Something is lost"] --> Type{"What kind?"}
|
||||
Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same pipeline<br/>that built it originally"]
|
||||
Type -->|"VM/LXC-level data"| PBS["Layer 2: VM/LXC backups<br/>full-host snapshots"]
|
||||
Type -->|"App data on the main host"| Kopia["Layer 3: Volume-level backups<br/>per-stack, isolated repository"]
|
||||
```
|
||||
|
||||
| Layer | Covers | Explicitly does not cover |
|
||||
|---|---|---|
|
||||
| Infrastructure as Code | A host lost entirely — rebuilt from the same pipeline that built it, against its existing config | Secrets (recovered separately, not part of this layer) |
|
||||
| VM/LXC-level backup | Full VM/LXC backups, including the secrets platform's own seal state and data | One host's Docker volume data, a large media storage pool, personal data |
|
||||
| Volume-level backup | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host | Anything outside that one host |
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The volume-level backup layer runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up.
|
||||
|
||||
> [!WARNING]
|
||||
> **A known, honestly-documented gap**: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them.
|
||||
|
||||
## Upgrades: two systems, deliberately not coupled
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph OS["OS-level"]
|
||||
direction TB
|
||||
T1["VMs — 3 at a time<br/>+ network-critical host check"]
|
||||
T2["Hypervisor nodes — 1 at a time<br/>aborts if anything's mid-migration"]
|
||||
T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
|
||||
end
|
||||
|
||||
subgraph App["App-level"]
|
||||
direction TB
|
||||
Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
|
||||
end
|
||||
|
||||
OS -.completely separate concern.-> App
|
||||
```
|
||||
|
||||
Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again before moving on — not just that the reboot command succeeded.
|
||||
|
||||
> [!TIP]
|
||||
> **Why app-level updates deliberately don't use batch/rolling deployment logic**: each host's update check is an independent, unrelated result — a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism would treat that non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures.
|
||||
|
||||
---
|
||||
Related: [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md)
|
||||
Reference in new issue
Block a user