docs: rewrite disaster-recovery.md following the full standing process in order

Live-verified against real infrastructure: Kopia's real repo/disk,
PBS's real datastore path, os_update role's real batch/abort behavior
against the actual Ansible code, and the real tag-driven
roles_metadata.yml mechanism behind Layer 1's rebuild claim. Caught a
real KB staleness in the process (PBS datastore usage, 3 weeks stale)
and fixed it properly - removed the point-in-time number entirely
rather than refreshing it, since a data-snapshot doesn't belong in
Trilium at all per the existing content-philosophy rule. Replaced
both mermaid diagrams with real branch/decoupling-shaped Nextcloud
diagrams. All 3 source KB notes stamped.
This commit is contained in:
thesabear committed 2026-08-28 11:06:40 +03:00
1 parent ab1a2ece1a
commit 3562322d50
3 files changed
+38 -23

No files matched your search

+30 -23
View File
@@ -5,13 +5,7 @@
## Three independent recovery layers
```mermaid
flowchart TD
Loss["Something is lost"] --> Type{"What kind?"}
Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same Packer→Terraform→Ansible<br/>pipeline that built it originally"]
Type -->|"VM/LXC-level data"| PBS["Layer 2: Proxmox Backup Server<br/>full-host snapshots"]
Type -->|"App data on the main host"| Kopia["Layer 3: Kopia, volume-level<br/>per-stack, isolated repository"]
```
![Something lost branches three ways depending on what kind of loss it is: an entire host rebuilds through the same Packer, Terraform, and Ansible pipeline that built it originally; VM/LXC-level data recovers through Proxmox Backup Server's full-host snapshots; app data on the main Docker host recovers through Kopia's volume-level, per-stack isolated repository](../assets/disaster-recovery-layers.svg)
## Why three layers instead of one comprehensive backup system
@@ -23,6 +17,34 @@ A single backup tool covering everything was the simpler design, and it was reje
| Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data |
| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host |
This is the real Kopia repository, not a description of it — a dedicated set of directories on its own separate disk, confirmed live:
```
$ ls /mnt/samsungssd
kopia-cache kopia-config kopia-repo kopia-tls
$ df -h /mnt/samsungssd
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 234G 65G 167G 28% /mnt/samsungssd
```
Layer 1's "rebuilt against its existing tag-driven config" claim is a real, checkable mechanism too — every role in the pipeline declares which of the three rebuild stages it belongs to:
```
$ head -12 roles_metadata.yml
roles_metadata:
base_users:
initialization: true
provisioning: false
compliance: false
description: "Base user creation tasks"
disk_resize:
initialization: true
provisioning: false
compliance: true
description: "Disk resizing tasks"
```
> [!IMPORTANT]
> The volume-level backup layer (Kopia) runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. A backup tool with write access to the data it protects, or one whose own storage lives on the same disk as that data, defeats a meaningful fraction of its own purpose the moment that disk fails.
@@ -31,22 +53,7 @@ A single backup tool covering everything was the simpler design, and it was reje
## Upgrades: two systems, deliberately not coupled
```mermaid
flowchart LR
subgraph OS["OS-level (os_update role)"]
direction TB
T1["VMs — 3 at a time<br/>+ network-critical host check"]
T2["Proxmox nodes — 1 at a time<br/>aborts if anything's mid-migration"]
T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
end
subgraph App["App-level (container_update / app_update)"]
direction TB
Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
end
OS -.completely separate concern.-> App
```
![OS-level updates (VMs 3 at a time with a network-critical host check, Proxmox nodes 1 at a time aborting on mid-migration, the backup server 1 at a time aborting on a running backup job) are a completely separate concern from app-level updates, where every stack's own update strategy runs its own independent, unrelated check](../assets/disaster-recovery-upgrades.svg)
Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again via a real ping check before moving on — not just that the reboot command itself succeeded, which would miss a host that reboots but fails to come back up cleanly.