docs: rewrite disaster-recovery.md following the full standing process in order

Live-verified against real infrastructure: Kopia's real repo/disk,
PBS's real datastore path, os_update role's real batch/abort behavior
against the actual Ansible code, and the real tag-driven
roles_metadata.yml mechanism behind Layer 1's rebuild claim. Caught a
real KB staleness in the process (PBS datastore usage, 3 weeks stale)
and fixed it properly - removed the point-in-time number entirely
rather than refreshing it, since a data-snapshot doesn't belong in
Trilium at all per the existing content-philosophy rule. Replaced
both mermaid diagrams with real branch/decoupling-shaped Nextcloud
diagrams. All 3 source KB notes stamped.

assets/disaster-recovery-layers.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Do not edit this file with editors other than draw.io -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="932px" height="383px" viewBox="0 0 932 383" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Recovery Layers&quot; id=&quot;recovery-layers-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1178&quot; dy=&quot;659&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;440&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;loss&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#F5F5F5;strokeColor=#666666;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Something is lost&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;180&quot; x=&quot;410&quot; y=&quot;20&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;gate&quot; parent=&quot;1&quot; style=&quot;rhombus;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;What kind?&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;110&quot; width=&quot;180&quot; x=&quot;410&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;iac&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Layer 1: Infrastructure as Code&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;rebuilt via the same Packer -&amp;amp;gt; Terraform -&amp;amp;gt; Ansible pipeline that built it originally&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;80&quot; width=&quot;260&quot; x=&quot;40&quot; y=&quot;310&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;pbs&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Layer 2: Proxmox Backup Server&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;full-host snapshots&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;80&quot; width=&quot;260&quot; x=&quot;370&quot; y=&quot;310&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;kopia&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Layer 3: Kopia, volume-level&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;per-stack, isolated repository&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;80&quot; width=&quot;260&quot; x=&quot;700&quot; y=&quot;310&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;loss&quot; style=&quot;html=1;endArrow=block;strokeColor=#666666;strokeWidth=2;rounded=1;&quot; target=&quot;gate&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;gate&quot; style=&quot;html=1;labelBackgroundColor=#1C1D1E;endArrow=block;strokeColor=#6c8ebf;fontColor=#6c8ebf;strokeWidth=2;rounded=1;fontSize=11;edgeStyle=orthogonalEdgeStyle;exitX=0.2;exitY=1;entryX=0.5;entryY=0;&quot; target=&quot;iac&quot; value=&quot;&amp;lt;span style=&amp;quot;background-color: light-dark(rgb(255, 255, 255), rgb(28, 29, 30));&amp;quot;&amp;gt;a host, entirely&amp;lt;/span&amp;gt;&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; x=&quot;0.118&quot; y=&quot;-19&quot; as=&quot;geometry&quot;&gt;&#10;            &lt;mxPoint as=&quot;offset&quot; /&gt;&#10;          &lt;/mxGeometry&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;gate&quot; style=&quot;html=1;labelBackgroundColor=#1C1D1E;endArrow=block;strokeColor=#82b366;fontColor=#82b366;strokeWidth=2;rounded=1;fontSize=11;&quot; target=&quot;pbs&quot; value=&quot;&amp;lt;span style=&amp;quot;background-color: light-dark(rgb(255, Line truncated

assets/disaster-recovery-upgrades.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Do not edit this file with editors other than draw.io -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1132px" height="272px" viewBox="0 0 1132 272" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Upgrade Systems&quot; id=&quot;upgrade-systems-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1178&quot; dy=&quot;659&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1100&quot; pageHeight=&quot;360&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;osgroup&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;startSize=30;swimlane;fillColor=none;strokeColor=#6c8ebf;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=center;&quot; value=&quot;OS-LEVEL (os_update role)&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;270&quot; width=&quot;620&quot; x=&quot;40&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;t1&quot; parent=&quot;osgroup&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;VMs&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;3 at a time + network-critical host check&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;560&quot; x=&quot;30&quot; y=&quot;45&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;t2&quot; parent=&quot;osgroup&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Proxmox nodes&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;1 at a time, aborts if anything&amp;#39;s mid-migration&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;560&quot; x=&quot;30&quot; y=&quot;120&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;t3&quot; parent=&quot;osgroup&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Backup server&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;1 at a time, aborts if a backup job is running&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;560&quot; x=&quot;30&quot; y=&quot;195&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;appgroup&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;startSize=30;swimlane;fillColor=none;strokeColor=#d79b00;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=center;&quot; value=&quot;APP-LEVEL (container_update / app_update)&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;240&quot; width=&quot;360&quot; x=&quot;810&quot; y=&quot;55&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;strategy&quot; parent=&quot;appgroup&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Per-stack update strategy&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;independent, unrelated checks&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;80&quot; width=&quot;300&quot; x=&quot;30&quot; y=&quot;90&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;osgroup&quot; style=&quot;html=1;labelBackgroundColor=#1C1D1E;endArrow=block;dashed=1;strokeColor=#b85450;fontColor=#b85450;strokeWidth=2;rounded=1;fontSize=12;&quot; target=&quot;appgroup&quot; value=&quot;&amp;lt;span style=&amp;quot;background-color: light-dark(rgb(255, 255, 255), rgb(28, 29, 30));&amp;quot;&amp;gt;completely separate concern&amp;lt;/span&amp;gt;&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; y=&quot;20&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;      &lt;/root&gt;&#10;    &lt;/mxGraphModel&gt;&#10;  &lt;/diagram&gt;&#10;&lt;/mxfile&gt;&#10;"><defs/><g><g data-cell-id="0"><g data-cell-id="1"><g data-cell-id="osgroup"><g transform="transLine truncated

docs/disaster-recovery.md

@@ -5,13 +5,7 @@
 
 ## Three independent recovery layers
 
-```mermaid
+![Something lost branches three ways depending on what kind of loss it is: an entire host rebuilds through the same Packer, Terraform, and Ansible pipeline that built it originally; VM/LXC-level data recovers through Proxmox Backup Server's full-host snapshots; app data on the main Docker host recovers through Kopia's volume-level, per-stack isolated repository](../assets/disaster-recovery-layers.svg)
-flowchart TD
-    Loss["Something is lost"] --> Type{"What kind?"}
-    Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same Packer→Terraform→Ansible<br/>pipeline that built it originally"]
-    Type -->|"VM/LXC-level data"| PBS["Layer 2: Proxmox Backup Server<br/>full-host snapshots"]
-    Type -->|"App data on the main host"| Kopia["Layer 3: Kopia, volume-level<br/>per-stack, isolated repository"]
-```
 
 ## Why three layers instead of one comprehensive backup system
 
@@ -23,6 +17,34 @@ A single backup tool covering everything was the simpler design, and it was reje
 | Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data |
 | Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host |
 
+This is the real Kopia repository, not a description of it — a dedicated set of directories on its own separate disk, confirmed live:
+
+```
+$ ls /mnt/samsungssd
+kopia-cache  kopia-config  kopia-repo  kopia-tls
+$ df -h /mnt/samsungssd
+Filesystem      Size  Used Avail Use% Mounted on
+/dev/nvme0n1p1  234G   65G  167G  28% /mnt/samsungssd
+```
+
+Layer 1's "rebuilt against its existing tag-driven config" claim is a real, checkable mechanism too — every role in the pipeline declares which of the three rebuild stages it belongs to:
+
+```
+$ head -12 roles_metadata.yml
+roles_metadata:
+  base_users:
+    initialization: true
+    provisioning: false
+    compliance: false
+    description: "Base user creation tasks"
+
+  disk_resize:
+    initialization: true
+    provisioning: false
+    compliance: true
+    description: "Disk resizing tasks"
+```
+
 > [!IMPORTANT]
 > The volume-level backup layer (Kopia) runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. A backup tool with write access to the data it protects, or one whose own storage lives on the same disk as that data, defeats a meaningful fraction of its own purpose the moment that disk fails.
 
@@ -31,22 +53,7 @@ A single backup tool covering everything was the simpler design, and it was reje
 
 ## Upgrades: two systems, deliberately not coupled
 
-```mermaid
+![OS-level updates (VMs 3 at a time with a network-critical host check, Proxmox nodes 1 at a time aborting on mid-migration, the backup server 1 at a time aborting on a running backup job) are a completely separate concern from app-level updates, where every stack's own update strategy runs its own independent, unrelated check](../assets/disaster-recovery-upgrades.svg)
-flowchart LR
-    subgraph OS["OS-level (os_update role)"]
-        direction TB
-        T1["VMs — 3 at a time<br/>+ network-critical host check"]
-        T2["Proxmox nodes — 1 at a time<br/>aborts if anything's mid-migration"]
-        T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
-    end
-
-    subgraph App["App-level (container_update / app_update)"]
-        direction TB
-        Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
-    end
-
-    OS -.completely separate concern.-> App
-```
 
 Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again via a real ping check before moving on — not just that the reboot command itself succeeded, which would miss a host that reboots but fails to come back up cleanly.