docs: rewrite disaster-recovery.md following the full standing process in order
Live-verified against real infrastructure: Kopia's real repo/disk, PBS's real datastore path, os_update role's real batch/abort behavior against the actual Ansible code, and the real tag-driven roles_metadata.yml mechanism behind Layer 1's rebuild claim. Caught a real KB staleness in the process (PBS datastore usage, 3 weeks stale) and fixed it properly - removed the point-in-time number entirely rather than refreshing it, since a data-snapshot doesn't belong in Trilium at all per the existing content-philosophy rule. Replaced both mermaid diagrams with real branch/decoupling-shaped Nextcloud diagrams. All 3 source KB notes stamped.
This commit is contained in:
3 files changed
+38
-23
No files matched your search
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Do not edit this file with editors other than draw.io -->
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="932px" height="383px" viewBox="0 0 932 383" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Recovery Layers" id="recovery-layers-1"> <mxGraphModel dx="1178" dy="659" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1000" pageHeight="440" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="loss" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#F5F5F5;strokeColor=#666666;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Something is lost" vertex="1"> <mxGeometry height="60" width="180" x="410" y="20" as="geometry" /> </mxCell> <mxCell id="gate" parent="1" style="rhombus;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="What kind?" vertex="1"> <mxGeometry height="110" width="180" x="410" y="130" as="geometry" /> </mxCell> <mxCell id="iac" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Layer 1: Infrastructure as Code&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;rebuilt via the same Packer -&amp;gt; Terraform -&amp;gt; Ansible pipeline that built it originally&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="260" x="40" y="310" as="geometry" /> </mxCell> <mxCell id="pbs" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Layer 2: Proxmox Backup Server&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;full-host snapshots&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="260" x="370" y="310" as="geometry" /> </mxCell> <mxCell id="kopia" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Layer 3: Kopia, volume-level&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;per-stack, isolated repository&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="260" x="700" y="310" as="geometry" /> </mxCell> <mxCell id="e1" edge="1" parent="1" source="loss" style="html=1;endArrow=block;strokeColor=#666666;strokeWidth=2;rounded=1;" target="gate"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="e2" edge="1" parent="1" source="gate" style="html=1;labelBackgroundColor=#1C1D1E;endArrow=block;strokeColor=#6c8ebf;fontColor=#6c8ebf;strokeWidth=2;rounded=1;fontSize=11;edgeStyle=orthogonalEdgeStyle;exitX=0.2;exitY=1;entryX=0.5;entryY=0;" target="iac" value="&lt;span style=&quot;background-color: light-dark(rgb(255, 255, 255), rgb(28, 29, 30));&quot;&gt;a host, entirely&lt;/span&gt;"> <mxGeometry relative="1" x="0.118" y="-19" as="geometry"> <mxPoint as="offset" /> </mxGeometry> </mxCell> <mxCell id="e3" edge="1" parent="1" source="gate" style="html=1;labelBackgroundColor=#1C1D1E;endArrow=block;strokeColor=#82b366;fontColor=#82b366;strokeWidth=2;rounded=1;fontSize=11;" target="pbs" value="&lt;span style=&quot;background-color: light-dark(rgb(255, Line truncated
|
||||
|
After Width: | Height: | Size: 208 KiB |
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Do not edit this file with editors other than draw.io -->
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1132px" height="272px" viewBox="0 0 1132 272" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Upgrade Systems" id="upgrade-systems-1"> <mxGraphModel dx="1178" dy="659" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1100" pageHeight="360" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="osgroup" parent="1" style="rounded=1;whiteSpace=wrap;html=1;startSize=30;swimlane;fillColor=none;strokeColor=#6c8ebf;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=center;" value="OS-LEVEL (os_update role)" vertex="1"> <mxGeometry height="270" width="620" x="40" y="40" as="geometry" /> </mxCell> <mxCell id="t1" parent="osgroup" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="VMs&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;3 at a time + network-critical host check&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="560" x="30" y="45" as="geometry" /> </mxCell> <mxCell id="t2" parent="osgroup" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Proxmox nodes&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;1 at a time, aborts if anything&#39;s mid-migration&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="560" x="30" y="120" as="geometry" /> </mxCell> <mxCell id="t3" parent="osgroup" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Backup server&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;1 at a time, aborts if a backup job is running&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="560" x="30" y="195" as="geometry" /> </mxCell> <mxCell id="appgroup" parent="1" style="rounded=1;whiteSpace=wrap;html=1;startSize=30;swimlane;fillColor=none;strokeColor=#d79b00;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=center;" value="APP-LEVEL (container_update / app_update)" vertex="1"> <mxGeometry height="240" width="360" x="810" y="55" as="geometry" /> </mxCell> <mxCell id="strategy" parent="appgroup" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Per-stack update strategy&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;independent, unrelated checks&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="300" x="30" y="90" as="geometry" /> </mxCell> <mxCell id="e1" edge="1" parent="1" source="osgroup" style="html=1;labelBackgroundColor=#1C1D1E;endArrow=block;dashed=1;strokeColor=#b85450;fontColor=#b85450;strokeWidth=2;rounded=1;fontSize=12;" target="appgroup" value="&lt;span style=&quot;background-color: light-dark(rgb(255, 255, 255), rgb(28, 29, 30));&quot;&gt;completely separate concern&lt;/span&gt;"> <mxGeometry relative="1" y="20" as="geometry" /> </mxCell> </root> </mxGraphModel> </diagram> </mxfile> "><defs/><g><g data-cell-id="0"><g data-cell-id="1"><g data-cell-id="osgroup"><g transform="transLine truncated
|
||||
|
After Width: | Height: | Size: 242 KiB |
+30
-23
@@ -5,13 +5,7 @@
|
||||
|
||||
## Three independent recovery layers
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Loss["Something is lost"] --> Type{"What kind?"}
|
||||
Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same Packer→Terraform→Ansible<br/>pipeline that built it originally"]
|
||||
Type -->|"VM/LXC-level data"| PBS["Layer 2: Proxmox Backup Server<br/>full-host snapshots"]
|
||||
Type -->|"App data on the main host"| Kopia["Layer 3: Kopia, volume-level<br/>per-stack, isolated repository"]
|
||||
```
|
||||
|
||||
|
||||
## Why three layers instead of one comprehensive backup system
|
||||
|
||||
@@ -23,6 +17,34 @@ A single backup tool covering everything was the simpler design, and it was reje
|
||||
| Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data |
|
||||
| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host |
|
||||
|
||||
This is the real Kopia repository, not a description of it — a dedicated set of directories on its own separate disk, confirmed live:
|
||||
|
||||
```
|
||||
$ ls /mnt/samsungssd
|
||||
kopia-cache kopia-config kopia-repo kopia-tls
|
||||
$ df -h /mnt/samsungssd
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/nvme0n1p1 234G 65G 167G 28% /mnt/samsungssd
|
||||
```
|
||||
|
||||
Layer 1's "rebuilt against its existing tag-driven config" claim is a real, checkable mechanism too — every role in the pipeline declares which of the three rebuild stages it belongs to:
|
||||
|
||||
```
|
||||
$ head -12 roles_metadata.yml
|
||||
roles_metadata:
|
||||
base_users:
|
||||
initialization: true
|
||||
provisioning: false
|
||||
compliance: false
|
||||
description: "Base user creation tasks"
|
||||
|
||||
disk_resize:
|
||||
initialization: true
|
||||
provisioning: false
|
||||
compliance: true
|
||||
description: "Disk resizing tasks"
|
||||
```
|
||||
|
||||
> [!IMPORTANT]
|
||||
> The volume-level backup layer (Kopia) runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. A backup tool with write access to the data it protects, or one whose own storage lives on the same disk as that data, defeats a meaningful fraction of its own purpose the moment that disk fails.
|
||||
|
||||
@@ -31,22 +53,7 @@ A single backup tool covering everything was the simpler design, and it was reje
|
||||
|
||||
## Upgrades: two systems, deliberately not coupled
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph OS["OS-level (os_update role)"]
|
||||
direction TB
|
||||
T1["VMs — 3 at a time<br/>+ network-critical host check"]
|
||||
T2["Proxmox nodes — 1 at a time<br/>aborts if anything's mid-migration"]
|
||||
T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
|
||||
end
|
||||
|
||||
subgraph App["App-level (container_update / app_update)"]
|
||||
direction TB
|
||||
Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
|
||||
end
|
||||
|
||||
OS -.completely separate concern.-> App
|
||||
```
|
||||
|
||||
|
||||
Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again via a real ping check before moving on — not just that the reboot command itself succeeded, which would miss a host that reboots but fails to come back up cleanly.
|
||||
|
||||
|
||||
Reference in new issue
Block a user