docs: fix disaster-recovery.md drift (Kopia stack count, stale evidence)

Kopia's coverage claim conflated the public repo's curated stacks/
folder (a documentation subset) with the real live stack count on
rocky-docker - fixed to the actual live figure (41, cross-checked
against docker_stacks.yml and the real repo directory count), same
fix applied to the source KB note which had its own stale ~38 figure
and an incorrect hyper1 backup mode (suspend -> stop, confirmed
against live /etc/pve/jobs.cfg). Refreshed the ls/df and
roles_metadata.yml command-output evidence to match current state.

docs/architecture-overview.md

@@ -1,5 +1,5 @@
-<!-- verified: 2026-09-06
-     rocky-docker-lxc@4f43dcc : caddy/Caddyfile
+<!-- verified: 2026-09-18
+     rocky-docker-lxc@7dd5025 : caddy/Caddyfile
      caddy-vm@9369f8b : caddy/Caddyfile
      trilium : 2wCPV6NXQa3l, 0mu75aAXGH8X, fscYadnUtmD5, QMTh5G7ldlLV, TIbCwYGitboy, Cl4TWBZ7ytYr, 6cJurv76PNDR, nAPJq9A1iuX8, DvKxK0gHXsng
 -->
@@ -31,7 +31,7 @@ The actual answer here is **four boundaries, each doing a different job**:
 | 🔴 WAN → edge | Anyone on the internet | Public reverse proxy, firewall-filtered | The only boundary facing genuinely untrusted traffic — gets the strictest, most isolated treatment |
 | 🔵 Browsers → user-facing proxy | A human, on a device that could be compromised, phished, or simply lost | Mutual TLS (client cert required) + SSO | The highest-value target for credential theft — a stolen password alone isn't enough without the client cert too |
 | 🟢 App-to-app | A container that's already inside the network, already provisioned by the same pipeline as everything else | IP-allowlist + SSO fallback, no mTLS | Full mTLS here would mean managing certificate lifecycle for every internal service-to-service call — real overhead for traffic that's already constrained to known, provisioned sources. The risk profile doesn't justify the same control as human-operated browsers. |
-| 🟡 LAN infra-to-infra | Non-HTTP traffic — metrics, logs, NTP, agent reporting | Explicit per-flow firewall rule, no proxy at all | Can't go through a reverse proxy — it isn't HTTP. The firewall itself is the only layer that *can* enforce this one. |
+| 🟡 LAN infra-to-infra | Non-HTTP traffic — metrics, logs, NTP, agent reporting | Explicit per-flow rule at each host's own firewalld — the real gate, since same-subnet traffic never transits a router | Can't go through a reverse proxy — it isn't HTTP. Each host's own firewall is the layer that actually sees this traffic; pfSense carries a mirrored ruleset for anything that does route through it, as defense-in-depth. |
 
 Every boundary above only means something if a service can't just skip it by reading the wrong header: **backend services trust proxy headers, never the raw client IP.** If that weren't true, the mTLS/SSO enforcement happening at the proxy layer would be theater — anything downstream could just claim to be a trusted IP and walk straight through. The proxy is the only thing allowed an opinion on who a client really is.
 
@@ -56,11 +56,11 @@ A single-router design fails this test completely — one outage, one misconfigu
 >
 > The second fix was conceptual, and the bigger one: for a long time the app-to-app boundary simply didn't exist. Internal service traffic crossed a flat, unauthenticated mesh network — anything on it could reach anything else on it, no proxy, no identity check, nothing. That's exactly the "you're on the network, you're trusted" pattern this document opens by calling the easy, wrong answer. Replacing it with the two-zone model in [`docs/dns-naming.md`](dns-naming.md) is what gave app-to-app traffic a real boundary for the first time.
 >
-> The third fix found the same flaw hiding somewhere less obvious: plain LAN-to-LAN traffic — metrics, logs, NTP, agent reporting — still had a blanket allow at the firewall, even after the app-to-app zone had its own identity-checked proxy. Any host could reach any other host; the only thing standing between "one compromised container" and "everything else on the LAN" was whatever that individual host's own firewall happened to be configured to do, which is a coin flip, not a boundary. Closing it meant giving up "it just works" for LAN infra traffic and writing an explicit rule for every flow that's actually supposed to exist — the same trade this document makes everywhere else: a boundary only counts if crossing it takes a decision, not an accident of network topology.
+> The third fix found the same flaw hiding somewhere less obvious: plain LAN-to-LAN traffic — metrics, logs, NTP, agent reporting — still had a blanket allow, even after the app-to-app zone had its own identity-checked proxy. Any host could reach any other host; the only thing standing between "one compromised container" and "everything else on the LAN" was whatever that individual host's own firewall happened to be configured to do, which is a coin flip, not a boundary. Closing it meant writing an explicit per-flow rule everywhere that boundary is actually enforced: each host's own firewalld (the real gate — this LAN is one flat /24, so traffic between two hosts on it never transits a router at all) and, for anything that does route through pfSense itself, a matching explicit-allow-plus-catch-all-deny ruleset there too, as defense-in-depth and a stated record of intent.
 >
 > Three different places, the same failure mode, the same fix each time — that's not a coincidence, it's what happens when you actually go looking for where "trusted by default" is still hiding.
 
-That third fix isn't just a claim — it's the actual live ruleset:
+That third fix isn't just a claim. Here's pfSense's half of it — even though genuine host-to-host traffic on this flat LAN never actually reaches it:
 
 ![pfSense LAN catch-all deny rule: block+log east-west traffic not explicitly allowed](../assets/pfsense-rules-lan-catchall-deny.png)
 
@@ -69,9 +69,9 @@ That third fix isn't just a claim — it's the actual live ruleset:
 <details>
 <summary>See the rest of the actual pfSense configuration</summary>
 
-**WAN — only three things pass inbound, everything else default-deny:**
+**WAN — only four things pass inbound, everything else default-deny:** SSH management, HTTPS via HAProxy, an HTTP→HTTPS redirect, and a STUN port-forward for the NetBird mesh VPN's NAT traversal.
 
-![pfSense WAN rules: SSH management, HTTPS via HAProxy, HTTP redirect, bogon block](../assets/pfsense-rules-wan.png)
+![pfSense WAN rules: SSH management, HTTPS via HAProxy, HTTP redirect, NetBird STUN, bogon block](../assets/pfsense-rules-wan.png)
 
 **LAN — fleet observability and security-agent reporting, one explicit rule per flow:**
 
@@ -95,16 +95,16 @@ That third fix isn't just a claim — it's the actual live ruleset:
 
 ## The Other Six Traffic Laws
 
-Each of these gets its own full mechanism doc elsewhere in this repo — this table is the map, not the territory:
+Laws 1–2 are covered above (the trust-boundary table); these are 3–8. Each of these gets its own full mechanism doc elsewhere in this repo — this table is the map, not the territory:
 
 | # | Law | Failure behavior |
 |---|---|---|
-| 3. Secrets | Applications never talk to Vault directly — one hop through a local agent only, with one narrow exception: a dedicated SSH-certificate refresh reuses that agent's token to call Vault directly. See [`docs/secrets-and-pki.md`](secrets-and-pki.md) and [`docs/secret-distribution.md`](secret-distribution.md). | Vault down → secret/certificate renewal stops, services run until their current token or cert expires — no immediate disruption. The one flow that doesn't degrade this gently: the SSH-certificate exception above has no static-key fallback, so a sustained outage eventually locks the automation identity out of the fleet entirely, not just "stale but running." |
-| 4. DNS | Which internal zone a hostname resolves in determines which trust boundary (above) a request enters. See [`docs/dns-naming.md`](dns-naming.md). | DNS down → app-to-app resolution fails; a direct-IP fallback exists, and it bypasses the SSO gate — but only for traffic already coming from an IP on the trusted allowlist. Anything else still hits Authentik regardless of DNS. |
-| 5. Observability | Logs and metrics push asynchronously — never inline with application traffic, full stop. See [`docs/observability.md`](observability.md) and [`docs/security-monitoring.md`](security-monitoring.md). | Monitoring down → collection stops; every application service keeps running, completely unaffected. |
-| 6. Notifications | Two independent branches (message-queue and HTTP webhook) converge on one formatter; neither is part of any service's runtime dependency chain. See [`docs/notifications.md`](notifications.md). | Notification delivery is event-driven, not persistent inline execution — a failure here doesn't touch application traffic either way. |
-| 7. CI/CD | Build/deploy traffic never sits inline with runtime; schedule-triggered, not push-triggered. See [`docs/registry-and-build.md`](registry-and-build.md). | A failed deploy doesn't affect already-running services — enforced by the redeploy step's own rollback logic. |
-| 8. Management | Administrative access is its own separate plane — one dedicated identity, non-standard ports, explicit per-host firewall rules from one source only on the LAN, plus a separate zero-trust mesh giving that same access a scoped path in from off-LAN too. See [`docs/netbird-zero-trust.md`](netbird-zero-trust.md). | On-LAN admin access never depended on the mesh and is unaffected if it goes down; off-LAN access specifically loses its only path until the mesh recovers. Every running service is unaffected either way. |
+| 3 | **Secrets.** Applications never talk to Vault directly — one hop through a local agent only, with one narrow exception: a dedicated SSH-certificate refresh reuses that agent's token to call Vault directly. See [`docs/secrets-and-pki.md`](secrets-and-pki.md) and [`docs/secret-distribution.md`](secret-distribution.md). | Vault down → secret/certificate renewal stops, services run until their current token or cert expires — no immediate disruption. The one flow that doesn't degrade this gently: the SSH-certificate exception above has no static-key fallback, so a sustained outage eventually locks the automation identity out of the fleet entirely, not just "stale but running." |
+| 4 | **DNS.** Which internal zone a hostname resolves in determines which trust boundary (above) a request enters. See [`docs/dns-naming.md`](dns-naming.md). | DNS down → app-to-app resolution fails; a direct-IP fallback exists, and it bypasses the SSO gate — but only for traffic already coming from an IP on the trusted allowlist. Anything else still hits Authentik regardless of DNS. |
+| 5 | **Observability.** Logs and metrics push asynchronously — never inline with application traffic, full stop. See [`docs/observability.md`](observability.md) and [`docs/security-monitoring.md`](security-monitoring.md). | Monitoring down → collection stops; every application service keeps running, completely unaffected. |
+| 6 | **Notifications.** Two independent branches (message-queue and HTTP webhook) converge on one formatter; neither is part of any service's runtime dependency chain. See [`docs/notifications.md`](notifications.md). | Notification delivery is event-driven, not persistent inline execution — a failure here doesn't touch application traffic either way. |
+| 7 | **CI/CD.** Build/deploy traffic never sits inline with runtime; schedule-triggered, not push-triggered. See [`docs/registry-and-build.md`](registry-and-build.md). | A failed deploy doesn't affect already-running services — enforced by the redeploy step's own rollback logic. |
+| 8 | **Management.** Administrative access is its own separate plane — one dedicated identity, non-standard ports, explicit per-host firewall rules from one source only on the LAN, plus a separate zero-trust mesh giving that same access a scoped path in from off-LAN too. See [`docs/netbird-zero-trust.md`](netbird-zero-trust.md). | On-LAN admin access never depended on the mesh and is unaffected if it goes down; off-LAN access specifically loses its only path until the mesh recovers. Every running service is unaffected either way. |
 
 The pattern repeats across all eight laws, not just the trust-boundary story above: **control-plane failure never takes down the data plane.** Every "failure behavior" row above ends the same way — the thing that broke stops working, and everything else keeps running regardless. That's not a coincidence; it's the same design principle applied consistently everywhere, not just to network trust.
 

docs/cron-scheduler.md

@@ -1,4 +1,4 @@
-<!-- verified: 2026-09-06
+<!-- verified: 2026-09-18
      trilium : EJGewkYGStHf
 -->
 
@@ -11,7 +11,7 @@
 
 A Node-RED tab fetches the inventory straight from NocoDB's REST API at request time — no cache — and renders it as a 7×24 grid (Mon–Sun × 00:00–23:00). Each cell's color is a **computed load weight**, not a raw job count: base `1` per job, plus a duration-bucket weight, plus a resource-tag weight (`disk-intensive`=2, `cpu-intensive`=1.5, `network-intensive`=0.5, summed if a job carries more than one), plus a flat `+1` if the job's category is `backup` — disk I/O contention from an overlapping backup is judged a worse collision than two lightweight jobs sharing an hour, and the weight formula says so explicitly rather than pretending every job costs the same. Weights bucket into 4 color bands (green → yellow → orange → red) relative to the current busiest cell. Jobs whose time can't be resolved to a specific day+hour are excluded from the grid and listed separately rather than forced into a misleading cell.
 
-This is the real heatmap, not a description of it — 50 known jobs, 15 without a fixed time:
+This is the real heatmap, not a description of it — 49 known jobs, 14 without a fixed time:
 
 ![The real Fleet Schedule Heatmap widget in Dashy, with Tue 00:00 clicked open showing its 8 real jobs — name, host, category, and FIXED/APP UI badges for jobs that can't simply be rescheduled](../assets/cron-scheduler-heatmap.png)
 
@@ -20,11 +20,11 @@ This is the real heatmap, not a description of it — 50 known jobs, 15 without
 
 Jobs with no fixed time get their own list instead of a misleading grid cell:
 
-![The heatmap's "no fixed time" section expanded — 15 real jobs excluded from the grid, including the manually-triggered-only Kestra flows and the PBS prune job with its mismatched datastore name](../assets/cron-scheduler-no-fixed-time.png)
+![The heatmap's "no fixed time" section expanded — 14 real jobs excluded from the grid, including the manually-triggered-only Kestra flows and Hawser's continuous 30-second heartbeat agent](../assets/cron-scheduler-no-fixed-time.png)
 
 Both of those are Dashy's *rendered* view. This is the raw NocoDB table underneath it — the actual data structure, not the heatmap built from it:
 
-![The real "Cron Scheduler" NocoDB table in grid view, showing all 50 records with their Job Name, Category, Schedule Expr, Est Start, Duration, Resource Tags, and Source columns](../assets/cron-scheduler-table.png)
+![The real "Cron Scheduler" NocoDB table in grid view, showing all 49 records with their Job Name, Category, Schedule Expr, Est Start, Duration, Resource Tags, and Source columns](../assets/cron-scheduler-table.png)
 
 ## Why A Manual Inventory Instead Of An Auto-discovered One
 

docs/disaster-recovery.md

@@ -1,6 +1,6 @@
-<!-- verified: 2026-09-06
-     iac@49ff427 : ansible/roles_metadata.yml
-     ansible@ccbda37 : roles/os_update, playbooks/os_update_vms.yml, playbooks/os_update_pve.yml, playbooks/os_update_pbs.yml, playbooks/app_update.yml
+<!-- verified: 2026-09-18
+     iac@f770e1c : ansible/roles_metadata.yml
+     ansible@24e1bf0 : roles/os_update, playbooks/os_update_vms.yml, playbooks/os_update_pve.yml, playbooks/os_update_pbs.yml, playbooks/app_update.yml
      trilium : xJRSQovMncQr, 6AbdS9cUE45T, bW84ccRoAJ5i, p45qNP2sIuhA, 6oESvJYzj1at
 -->
 
@@ -21,23 +21,24 @@ A single backup tool covering everything was the simpler design, and it was reje
 |---|---|---|
 | Infrastructure as Code | A host lost entirely, VM or LXC alike — rebuilt from the same pipeline that built it ([`docs/iac-pipeline.md`](iac-pipeline.md)), against its existing tag-driven config | Secrets — recovered separately via Vault, see [`docs/secrets-and-pki.md`](secrets-and-pki.md), not part of this layer |
 | Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data |
-| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host |
+| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering all 41 stacks running there | Anything outside that one host |
 
 This is the real Kopia repository, not a description of it — a dedicated set of directories on its own separate disk, confirmed live:
 
 ```
 $ ls /mnt/samsungssd
-kopia-cache  kopia-config  kopia-repo  kopia-tls
+kopia-cache  kopia-config  kopia-repo  kopia-tls  recovery
 $ df -h /mnt/samsungssd
 Filesystem      Size  Used Avail Use% Mounted on
-/dev/nvme0n1p1  234G   65G  167G  28% /mnt/samsungssd
+/dev/nvme0n1p1  234G   75G  157G  33% /mnt/samsungssd
 ```
 
 Layer 1's "rebuilt against its existing tag-driven config" claim is a real, checkable mechanism too — every role in the pipeline declares which of the three rebuild stages it belongs to:
 
 ```
-$ head -12 roles_metadata.yml
+$ head -14 roles_metadata.yml
 roles_metadata:
+  # Bootstrap roles
   base_users:
     initialization: true
     provisioning: false