docs: rewrite disaster-recovery.md following the full standing process in order
Live-verified against real infrastructure: Kopia's real repo/disk, PBS's real datastore path, os_update role's real batch/abort behavior against the actual Ansible code, and the real tag-driven roles_metadata.yml mechanism behind Layer 1's rebuild claim. Caught a real KB staleness in the process (PBS datastore usage, 3 weeks stale) and fixed it properly - removed the point-in-time number entirely rather than refreshing it, since a data-snapshot doesn't belong in Trilium at all per the existing content-philosophy rule. Replaced both mermaid diagrams with real branch/decoupling-shaped Nextcloud diagrams. All 3 source KB notes stamped.
This commit is contained in:
3 files changed
+38
-23
No files matched your search
+30
-23
@@ -5,13 +5,7 @@
|
||||
|
||||
## Three independent recovery layers
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Loss["Something is lost"] --> Type{"What kind?"}
|
||||
Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same Packer→Terraform→Ansible<br/>pipeline that built it originally"]
|
||||
Type -->|"VM/LXC-level data"| PBS["Layer 2: Proxmox Backup Server<br/>full-host snapshots"]
|
||||
Type -->|"App data on the main host"| Kopia["Layer 3: Kopia, volume-level<br/>per-stack, isolated repository"]
|
||||
```
|
||||

|
||||
|
||||
## Why three layers instead of one comprehensive backup system
|
||||
|
||||
@@ -23,6 +17,34 @@ A single backup tool covering everything was the simpler design, and it was reje
|
||||
| Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data |
|
||||
| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host |
|
||||
|
||||
This is the real Kopia repository, not a description of it — a dedicated set of directories on its own separate disk, confirmed live:
|
||||
|
||||
```
|
||||
$ ls /mnt/samsungssd
|
||||
kopia-cache kopia-config kopia-repo kopia-tls
|
||||
$ df -h /mnt/samsungssd
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/nvme0n1p1 234G 65G 167G 28% /mnt/samsungssd
|
||||
```
|
||||
|
||||
Layer 1's "rebuilt against its existing tag-driven config" claim is a real, checkable mechanism too — every role in the pipeline declares which of the three rebuild stages it belongs to:
|
||||
|
||||
```
|
||||
$ head -12 roles_metadata.yml
|
||||
roles_metadata:
|
||||
base_users:
|
||||
initialization: true
|
||||
provisioning: false
|
||||
compliance: false
|
||||
description: "Base user creation tasks"
|
||||
|
||||
disk_resize:
|
||||
initialization: true
|
||||
provisioning: false
|
||||
compliance: true
|
||||
description: "Disk resizing tasks"
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The volume-level backup layer (Kopia) runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. A backup tool with write access to the data it protects, or one whose own storage lives on the same disk as that data, defeats a meaningful fraction of its own purpose the moment that disk fails.
|
||||
|
||||
@@ -31,22 +53,7 @@ A single backup tool covering everything was the simpler design, and it was reje
|
||||
|
||||
## Upgrades: two systems, deliberately not coupled
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph OS["OS-level (os_update role)"]
|
||||
direction TB
|
||||
T1["VMs — 3 at a time<br/>+ network-critical host check"]
|
||||
T2["Proxmox nodes — 1 at a time<br/>aborts if anything's mid-migration"]
|
||||
T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
|
||||
end
|
||||
|
||||
subgraph App["App-level (container_update / app_update)"]
|
||||
direction TB
|
||||
Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
|
||||
end
|
||||
|
||||
OS -.completely separate concern.-> App
|
||||
```
|
||||

|
||||
|
||||
Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again via a real ping check before moving on — not just that the reboot command itself succeeded, which would miss a host that reboots but fails to come back up cleanly.
|
||||
|
||||
|
||||
Reference in new issue
Block a user