Merge pull request 'docs: iac commit IDs after its history rewrite (ISO removed, content unchanged)' (#3) from iac-history-ids into main

Covers the design rationale, the refill-cycle mechanism, three real
issues found and fixed via live testing against a real Kodi session
(a silent result-key mismatch that meant the daemon never once
resolved a track, a starved similarity candidate pool, and a
4-minute poll-interval blind spot), and live proof of a working
auto-queued run of similar tracks.

.gitea/workflows/ci.yml

@@ -0,0 +1,15 @@
+# CI for this repo - the actual checks live in the shared ci-workflows repo.
+name: ci
+
+"on":
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  gitleaks:
+    # Short owner/repo form: Gitea resolves reusable workflows server-side and
+    # rejects absolute URLs that don't match its own ROOT_URL.
+    uses: thesabear/ci-workflows/.gitea/workflows/gitleaks.yml@main
+    secrets: inherit

.gitleaksignore

@@ -0,0 +1,7 @@
+# Findings checked by hand and confirmed dead - each line ignores exactly one
+# finding (commit:file:rule:line). Anything else gitleaks finds still fails CI.
+
+# Paperless OIDC client secret in old history (removed from the files 2026-07-06).
+# Checked 2026-10-11: that client_id no longer exists in Authentik (all 104
+# OAuth2 providers), and Paperless uses a different client and secret.
+973546647df3fe505e913c42af9ed3547083593c:Docker/Paperless-NGX/docker-compose.yaml:generic-api-key:50

README.md

@@ -12,15 +12,15 @@
 
 <table style="margin: 0 auto;">
 <tr>
-<td align="center">🖥️<br/><strong style="font-size:1.6em;">11</strong><br/>fleet hosts</td>
+<td align="center">🖥️<br/><strong style="font-size:1.6em;">12</strong><br/>fleet hosts</td>
 <td align="center">🐳<br/><strong style="font-size:1.6em;">44</strong><br/>compose stacks</td>
-<td align="center">📚<br/><strong style="font-size:1.6em;">15</strong><br/>documented systems</td>
+<td align="center">📚<br/><strong style="font-size:1.6em;">16</strong><br/>documented systems</td>
 <td align="center">🧱<br/><strong style="font-size:1.6em;">4</strong><br/>trust boundaries</td>
 <td align="center">🔁<br/><strong style="font-size:1.6em;">13</strong><br/>automated maintenance flows</td>
 </tr>
 </table>
 
-I'm a curious, self-taught technologist who runs a real, multi-host homelab as a production-grade playground. Most homelabs are a media server with some extra apps bolted on. This one is a platform: The same categories of systems a company's platform/security team owns, run at homelab scale, for real, 24/7, by one person.
+I'm a curious, self-taught technologist who runs a real, multi-host homelab as a production-grade playground. Most homelabs are a media server with some extra apps bolted on. This one is a platform: the same categories of systems a company's platform/security team owns, run at homelab scale, for real, 24/7, by one person.
 
 My long-term goal is to build a career in IT — specifically in environments where learning never stops and innovation is part of the culture.
 What started as a hobby project has grown into a full-on journey into DevOps, automation, infrastructure, and security;
@@ -30,6 +30,7 @@ What started as a hobby project has grown into a full-on journey into DevOps, au
 - <strong style="font-size:1.4em;">HashiCorp Vault</strong> — issuing PKI certs and managing every application secret
 - <strong style="font-size:1.4em;">Caddy + Authentik</strong> — a two-zone reverse-proxy design fronted by SSO, behind pfSense and firewalld for real network segmentation
 - <strong style="font-size:1.4em;">Kestra</strong> — fleet orchestration with scheduled flows backed by Ansible roles and playbooks
+- <strong style="font-size:1.4em;">NetBird</strong> — a self-hosted zero-trust WireGuard mesh, every host and device with its own identity, deny-by-default policy instead of trusting LAN position
 - <strong style="font-size:1.4em;">Gitea</strong> — a self-hosted registry building and shipping container images for every stack that's opted in
 - <strong style="font-size:1.4em;">Wazuh + Dashy</strong> — a self-built SOC watching all of it
 
@@ -47,7 +48,17 @@ The three routing authorities are also split across two separate physical nodes,
 
 Full breakdown — the boundary-by-boundary reasoning, the routing-authority failure table, all 8 traffic laws: [docs/architecture-overview.md](docs/architecture-overview.md).
 
-## Architecture at a Glance
+## Zero Trust, Actually Implemented
+
+Zero trust is where enterprise security has been heading for years now — NIST formalized it as a reference architecture, and "trusted because it's on the LAN" has been steadily losing ground everywhere as the default posture. The core idea is simple to state and hard to actually build: nothing gets access because of *where* it's connecting from, only because of *what it's proven itself to be*. Network position stops being a substitute for identity, and every single request gets checked against explicit policy instead of inheriting trust from whatever segment it happened to arrive on.
+
+This fleet doesn't just talk about that model — the four trust boundaries above are one expression of it, and remote/roaming access is a second, separate one, since "how do you reach the fleet from outside it" is a genuinely different problem from "how do services inside it talk to each other." That second problem is solved with a self-hosted zero-trust mesh (**NetBird**, WireGuard-based): every host and every device gets its own cryptographic identity, access is granted per-identity through explicit policy, and the mesh's own built-in allow-everything default is disabled — nothing crosses it unless a rule says so.
+
+![Perimeter model versus zero-trust model: on the left, anything inside one shared boundary can reach anything else inside it; on the right, every device has its own identity and only an explicit policy grants access, everything else blocked regardless of network position](assets/readme-zero-trust-model.svg)
+
+Full breakdown — the mesh-vs-VPN reasoning, the complete policy model (groups, scoped keys, peer approval, posture checks), the real bring-up story, and live evidence straight from the running system: [docs/netbird-zero-trust.md](docs/netbird-zero-trust.md).
+
+## Architecture At a Glance
 
 ![Architecture Overview](assets/architecture-overview.svg)
 Solid arrows are hard dependencies — the target has to be reachable for the source to function. Dashed arrows are secondary relationships (request routing like "proxies"/"auth", or async telemetry reporting) that don't block the source if the target's unavailable. Full breakdown of all 8 traffic laws (ingress, east-west, secrets, DNS, observability, notifications, CI/CD, management) with the actual trust-boundary reasoning: [docs/architecture-overview.md](docs/architecture-overview.md).
@@ -74,10 +85,18 @@ Two internal DNS zones split browser-facing traffic from service-to-service traf
 No blanket internal trust anywhere on the network — every real service-to-service flow gets its own explicit rule, and everything not explicitly allowed is logged and dropped by default. Enforced at two layers: a firewall at the network edge, and firewalld locally on every host, each opening only what that specific host actually needs. A few rules go narrower still — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet.
 → [docs/firewall-and-network.md](docs/firewall-and-network.md)
 
+### 📇 Homelab Inventory
+Every physical host, VM/LXC, and running service in the fleet, re-derived from the machines themselves once a week rather than hand-maintained — so the reference is structurally unable to drift out of sync with reality. A three-tier Ansible collection (hardware off the hypervisors, guests from the Proxmox APIs, listening sockets off every host) writes three linked tables, surfaced on one dashboard page. The one hand-held detail — a NAS that reports a placeholder string for its own model — is protected by a merge-only write that structurally can't overwrite it.
+→ [docs/homelab-inventory.md](docs/homelab-inventory.md)
+
 ### 🏗️ Infrastructure as Code
 Packer builds a hardened base image, Terraform provisions each VM from it, and Ansible configures and continuously re-hardens every host after that — three tools, three separate responsibilities, connected by one tag that threads through all three stages instead of one tool trying to do every job. Terraform's part in that link ends at creation — it writes the tags file once, and that file then drives Ansible independently for the rest of that host's life.
 → [docs/iac-pipeline.md](docs/iac-pipeline.md)
 
+### 🎧 Kodi <-> Music Assistant Smart Queue Bridge
+Kodi stays the only music player, UI, and ratings authority; Music Assistant runs invisibly behind it as a similarity engine, never a player, never writing anything back to the library. A small daemon watches Kodi's live queue and auto-refills it with similar tracks via MA's recommendation API when it runs low. Three real issues surfaced only once it was tested against a live Kodi session — not exercised in isolation — including one where the daemon had silently never worked at all.
+→ [docs/kodi-music-assistant-bridge.md](docs/kodi-music-assistant-bridge.md)
+
 ### 🔔 Notifications
 Every alert in the fleet — from apps, from hosts, from security events — funnels through one Node-RED instance and one shared formatter before reaching Slack, so a notification reads the same regardless of which of a dozen-plus sources triggered it. A dedicated role puts every host onto the same pipeline for OS-level health, independent of any single application. One shared endpoint handles every app's webhook regardless of shape — a source field where an app provides one, structural fingerprinting of the payload where it doesn't — instead of standing up a dedicated integration per app.
 → [docs/notifications.md](docs/notifications.md)
@@ -110,12 +129,22 @@ Docker images get built and hosted entirely in-fleet on the Git server's own bui
 Authentik, self-hosted, fronts single sign-on fleet-wide — native OIDC where an app supports it, proxy-based forward-auth covering everything else, so apps that were never built with SSO in mind still get it. Every OIDC login crosses reverse-proxy zones mid-handshake, and Git automation runs on a scoped bot account rather than an admin identity.
 → [docs/sso-and-identity.md](docs/sso-and-identity.md)
 
+### 🕸️ Zero-Trust Mesh Overlay
+A self-hosted WireGuard mesh gives every fleet host and personal device its own stable identity, independent of where it physically sits — access is granted per-identity through mesh policy, deny-by-default, not inferred from LAN position. Rejected a classic VPN concentrator on purpose: the mesh's control plane only ever coordinates, real traffic flows peer-to-peer and never routes through it.
+→ [docs/netbird-zero-trust.md](docs/netbird-zero-trust.md)
+
 ## What's Running
 
 44 Docker Compose stacks, grouped by purpose rather than by which host happens to run them — see [`stacks/`](stacks/): identity & security, platform/infra, media, productivity, and dev tools.
 
 ---
 
+## Cheatsheets
+
+Quick-reference material for tools I use day-to-day — command references and config notes, not full write-ups. See [`cheatsheets/`](cheatsheets/).
+
+---
+
 ## Scripts & Tooling
 
 A few standalone scripts live in [`projects/`](projects/) alongside the Docker stacks — real tooling, not demo code:

assets/homelab-inventory-pipeline.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Do not edit this file with editors other than draw.io -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1251px" height="486px" viewBox="0 0 1251 486" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;homelab-inventory-pipeline&quot; id=&quot;hip1&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;874&quot; dy=&quot;519&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1320&quot; pageHeight=&quot;560&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;title&quot; parent=&quot;1&quot; style=&quot;text;html=1;align=left;verticalAlign=middle;fontSize=15;fontStyle=1;fontColor=#333333;&quot; value=&quot;Homelab Inventory Pipeline&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;26&quot; width=&quot;360&quot; x=&quot;40&quot; y=&quot;20&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;kestra&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=11;fontColor=#333333;&quot; value=&quot;Kestra&amp;#xa;homelab-inventory-sync&amp;#xa;weekly · Thu 12:00&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;64&quot; width=&quot;200&quot; x=&quot;60&quot; y=&quot;60&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;t1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=11;fontColor=#333333;&quot; value=&quot;Tier 1 · Hardware&amp;lt;br&amp;gt;node1 / node2 + pfSense&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;56&quot; width=&quot;200&quot; x=&quot;60&quot; y=&quot;190&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;t2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=11;fontColor=#333333;&quot; value=&quot;Tier 2 · VMs / LXCs&amp;#xa;Proxmox API — no SSH&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;56&quot; width=&quot;200&quot; x=&quot;60&quot; y=&quot;286&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;t3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=11;fontColor=#333333;&quot; value=&quot;Tier 3 · Services&amp;#xa;ss -tlnp + docker ps&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;56&quot; width=&quot;200&quot; x=&quot;60&quot; y=&quot;382&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;gate&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;fontSize=10;fontColor=#333333;dashed=1;&quot; value=&quot;pfSense Manufacturer / Model — entered by hand.&amp;#xa;Tier 1 never sends these keys, so the weekly&amp;#xa;push cannot overwrite them.&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;66&quot; width=&quot;360&quot; x=&quot;420&quot; y=&quot;50&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;noco&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#F5F5F5;strokeColor=#666666;fontSize=11;fontStyle=1;fontColor=#333333;verticalAlign=top;spacingTop=6;&quot; value=&quot;NocoDB — 3 linked tables&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;300&quot; width=&quot;290&quot; x=&quot;420&quot; y=&quot;150&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;hw&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=11;fontColor=#333333;&quot; value=&quot;Hardware Inventory&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;48&quot; width=&quot;230&quot; x=&quot;450&quot; y=&quot;190&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;vm&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=11;fontColor=#333333;&quot; value=&quot;VM / LXC Inventory&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;48&quot; width=&quot;230&quot; x=&quot;450&quot; y=&quot;286&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;svc&quot; parent=&quot;1&Line truncated

assets/iac-pipeline-ansible-stages.svg

@@ -1,4 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!-- Do not edit this file with editors other than draw.io -->
 <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="932px" height="318px" viewBox="0 0 932 318" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Ansible Stages&quot; id=&quot;iac-ansiblestages-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1080&quot; pageHeight=&quot;360&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;tags&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Per-host tag list&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;from Terraform&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;70&quot; width=&quot;200&quot; x=&quot;20&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;init&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Initialization stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;runs once, at creation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;20&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;prov&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Provisioning stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;every run — 1 role today, one host&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;comp&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Compliance stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;re-applied every run&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;sub&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;9 independently-tagged&amp;lt;br&amp;gt;hardening sub-roles&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;720&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;init&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;prov&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;comp&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e4&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;comp&quot; style=&quot;html=1;endArrow=block;strokeColor=#82b366;strokeWidth=2;rounded=1;&quot; target=&quot;sub&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quoLine truncated
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="932px" height="318px" viewBox="0 0 932 318" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;Ansible Stages&quot; id=&quot;iac-ansiblestages-1&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1080&quot; pageHeight=&quot;360&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;tags&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Per-host tag list&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;from Terraform&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;70&quot; width=&quot;200&quot; x=&quot;20&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;init&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Initialization stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;runs once, at creation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;20&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;prov&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Provisioning stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;every run — 2 roles today&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;comp&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Compliance stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;re-applied every run&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;sub&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;9 independently-tagged&amp;lt;br&amp;gt;hardening sub-roles&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;720&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;init&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;prov&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;comp&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e4&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;comp&quot; style=&quot;html=1;endArrow=block;strokeColor=#82b366;strokeWidth=2;rounded=1;&quot; target=&quot;sub&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometrLine truncated

assets/kodi-music-assistant-bridge-refill-cycle.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Do not edit this file with editors other than draw.io -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="624px" height="632px" viewBox="0 0 624 632" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;Refill Cycle&quot; id=&quot;0&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;0&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;900&quot; pageHeight=&quot;700&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;poll&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;&quot; value=&quot;Poll each Kodi instance&amp;lt;br&amp;gt;(every 240s)&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;active&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;&quot; value=&quot;Active player + queue&amp;lt;br&amp;gt;length on this instance&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;150&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;gate&quot; parent=&quot;1&quot; style=&quot;rhombus;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=13;&quot; value=&quot;&amp;lt;font style=&amp;quot;font-size: 11px;&amp;quot;&amp;gt;Remaining &amp;amp;gt;&amp;lt;br&amp;gt;REFILL_THRESHOLD (5)?&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;90&quot; width=&quot;190&quot; x=&quot;355&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;resolve&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=13;&quot; value=&quot;Resolve current track&amp;lt;br&amp;gt;in MA (music/search)&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;400&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;similar&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=13;&quot; value=&quot;Query similar_tracks&amp;lt;br&amp;gt;(large candidate pool)&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;500&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;inject&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;&quot; value=&quot;Filter + translate paths,&amp;lt;br&amp;gt;Playlist.Add up to 10&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;600&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;poll&quot; style=&quot;edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#333333;&quot; target=&quot;active&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;active&quot; style=&quot;edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#333333;&quot; target=&quot;gate&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;gate&quot; style=&quot;edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;exitX=1;exitY=0.5;labelBackgroundColor=#1C1D1E;exitDx=0;exitDy=0;&quot; target=&quot;poll&quot; value=&quot;&amp;lt;span style=&amp;quot;background-color: light-dark(rgb(255, 255, 255), rgb(28, 29, 30));&amp;quot;&amp;gt;yes &amp;amp;mdash; skip&amp;lt;/span&amp;gt;&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; x=&quot;0.0698&quot; y=&quot;-33&quot; as=&quot;geometry&quot;&gt;&#xA;            &lt;mxPoint as=&quot;offset&quot; /&gt;&#xA;            &lt;Array as=&quot;points&quot;&gt;&#xA;              &lt;mxPoint x=&quot;60Line truncated

assets/observability-log-fanout.svg

@@ -1,4 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!-- Do not edit this file with editors other than draw.io -->
 <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="273px" viewBox="0 0 872 273" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Log Fanout&quot; id=&quot;obs-logfanout-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;320&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;a1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;system logs&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=system&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a1&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b1&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;a2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker API&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;container logs, survives recreation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=docker&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a2&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b2&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;a3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;DNS query log&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;daily rotation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=dns, real event time&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a3&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b3Line truncated
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="373px" viewBox="0 0 872 373" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;Log Fanout&quot; id=&quot;obs-logfanout-1&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;420&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;a1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;system logs&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;b1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=system&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a1&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b1&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;a2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker API&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;container logs, survives recreation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;b2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=docker&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a2&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b2&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;a3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;DNS query log&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;daily rotation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;b3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=technitium-dns, real event time&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a3&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; targLine truncated

assets/orchestration-flow-patterns.svg

@@ -1,4 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!-- Do not edit this file with editors other than draw.io -->
 <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1142px" height="642px" viewBox="0 0 1142 642" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Flow Patterns&quot; id=&quot;orch-patterns-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1200&quot; pageHeight=&quot;640&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;lane1&quot; parent=&quot;1&quot; style=&quot;swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#6c8ebf;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=left;spacingLeft=8;verticalAlign=middle;&quot; value=&quot;12 OF 13 FLOWS — no human gate&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;140&quot; width=&quot;1140&quot; x=&quot;30&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;l1s&quot; parent=&quot;lane1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Weekly / scheduled trigger&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;40&quot; y=&quot;50&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;l1r&quot; parent=&quot;lane1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Runs the playbook directly&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;os-update-vms, container-update, disk-cleanup, os-update-pbs, dashy-fleet-audit, and 7 more&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;70&quot; width=&quot;470&quot; x=&quot;340&quot; y=&quot;45&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;l1n&quot; parent=&quot;lane1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Notify: succeeded / failed&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;880&quot; y=&quot;50&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;lane1&quot; source=&quot;l1s&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;l1r&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;lane1&quot; source=&quot;l1r&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;l1n&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;lane2&quot; parent=&quot;1&quot; style=&quot;swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#d79b00;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=left;spacingLeft=8;verticalAlign=middle;&quot; value=&quot;1 OF 13 FLOWS (app-update) — check → pause → human approval → apply&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;480&quot; width=&quot;1140&quot; x=&quot;30&quot; y=&quot;190&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;l2s&quot; parent=&quot;lane2&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Weekly schedule&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;180&quot; x=&quot;40&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;l2c&quot; parent=&quot;lane2&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Check mode run&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;detect pending Vault/Technitium/Alloy updates&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10Line truncated
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1142px" height="642px" viewBox="0 0 1142 642" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;Flow Patterns&quot; id=&quot;orch-patterns-1&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1200&quot; pageHeight=&quot;640&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;lane1&quot; parent=&quot;1&quot; style=&quot;swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#6c8ebf;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=left;spacingLeft=8;verticalAlign=middle;&quot; value=&quot;14 OF 15 FLOWS — no human gate&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;140&quot; width=&quot;1140&quot; x=&quot;30&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;l1s&quot; parent=&quot;lane1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Weekly / scheduled trigger&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;40&quot; y=&quot;50&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;l1r&quot; parent=&quot;lane1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Runs the playbook directly&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;os-update-vms, container-update, disk-cleanup, os-update-pbs, dashy-fleet-audit, and 9 more&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;70&quot; width=&quot;470&quot; x=&quot;340&quot; y=&quot;45&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;l1n&quot; parent=&quot;lane1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Notify: succeeded / failed&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;880&quot; y=&quot;50&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;lane1&quot; source=&quot;l1s&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;l1r&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;lane1&quot; source=&quot;l1r&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;l1n&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;lane2&quot; parent=&quot;1&quot; style=&quot;swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#d79b00;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=left;spacingLeft=8;verticalAlign=middle;&quot; value=&quot;1 OF 15 FLOWS (app-update) — check → pause → human approval → apply&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;480&quot; width=&quot;1140&quot; x=&quot;30&quot; y=&quot;190&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;l2s&quot; parent=&quot;lane2&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Weekly schedule&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;180&quot; x=&quot;40&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;l2c&quot; parent=&quot;lane2&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Check mode run&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;detect pending Vault/Technitium/Alloy updates&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xALine truncated

assets/readme-zero-trust-model.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Do not edit this file with editors other than draw.io -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1062px" height="342px" viewBox="0 0 1062 342" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Zero Trust Model&quot; id=&quot;readme-zt-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1178&quot; dy=&quot;659&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1080&quot; pageHeight=&quot;420&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;lPerim&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;dashed=1;fillColor=none;strokeColor=#b85450;fontSize=13;fontStyle=1;fontColor=#333333;verticalAlign=top;align=center;spacingTop=10;&quot; value=&quot;Perimeter model — trust by position&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;340&quot; width=&quot;460&quot; x=&quot;30&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;lBoundary&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;fontColor=#333333;fontSize=11;fontStyle=2;verticalAlign=top;align=center;spacingTop=8;&quot; value=&quot;&amp;quot;inside the network&amp;quot;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;240&quot; width=&quot;380&quot; x=&quot;60&quot; y=&quot;90&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;lA&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=11;shadow=1;&quot; value=&quot;Device A&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;50&quot; width=&quot;120&quot; x=&quot;90&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;lB&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=11;shadow=1;&quot; value=&quot;Device B&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;50&quot; width=&quot;120&quot; x=&quot;270&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;lC&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=11;shadow=1;&quot; value=&quot;Device C&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;50&quot; width=&quot;120&quot; x=&quot;180&quot; y=&quot;240&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;le1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;lA&quot; style=&quot;html=1;endArrow=block;startArrow=block;strokeColor=#b85450;strokeWidth=2;&quot; target=&quot;lB&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;le2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;lA&quot; style=&quot;html=1;endArrow=block;startArrow=block;strokeColor=#b85450;strokeWidth=2;&quot; target=&quot;lC&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;le3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;lB&quot; style=&quot;html=1;endArrow=block;startArrow=block;strokeColor=#b85450;strokeWidth=2;&quot; target=&quot;lC&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;lNote&quot; parent=&quot;1&quot; style=&quot;text;html=1;align=center;fontSize=10;fontColor=#b85450;fontStyle=2;&quot; value=&quot;&amp;lt;font style=&amp;quot;font-size: 12px;&amp;quot;&amp;gt;anything in here reaches anything else in here&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;20&quot; width=&quot;380&quot; x=&quot;50&quot; y=&quot;340&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;rZT&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;dashed=1;fillColor=none;strokeColor=#82b366;fontSize=13;fontStyle=1;fontColor=#333333;verticalAlign=top;align=center;spacingTop=10;&quot; value=&quot;Zero-trust model — trust by identity&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;340&quot; Line truncated

assets/secret-distribution-flows.svg

@@ -1,4 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!-- Do not edit this file with editors other than draw.io -->
 <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1179px" height="545px" viewBox="0 0 1179 545" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Secret Distribution Flows&quot; id=&quot;secret-dist-flows-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1200&quot; pageHeight=&quot;620&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;repo&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;&quot; value=&quot;Provisioning Repo&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;template inventory&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;80&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;60&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;agentconfig&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;&quot; value=&quot;Vault Agent Config&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;on the target host&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;80&quot; width=&quot;200&quot; x=&quot;410&quot; y=&quot;60&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;bgfill_dataplane&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=none;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;290&quot; width=&quot;440&quot; x=&quot;380&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;dataplane&quot; parent=&quot;1&quot; style=&quot;swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#9673a6;dashed=1;fontSize=12;fontStyle=1;fontColor=#333333;align=center;spacingLeft=8;verticalAlign=middle;&quot; value=&quot;DATA PLANE — what Vault Agent actually renders&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;290&quot; width=&quot;440&quot; x=&quot;380&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;vaultagent&quot; parent=&quot;dataplane&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Vault Agent&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;140&quot; x=&quot;150&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;certflow&quot; parent=&quot;dataplane&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Certificate Flow&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;PKI engine, short-lived&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;170&quot; x=&quot;30&quot; y=&quot;180&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;secflow&quot; parent=&quot;dataplane&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;App Secret Flow&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;KV, rendered 0600&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;170&quot; x=&quot;240&quot; y=&quot;180&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;container&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;&quot; value=&quot;Container&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;reads local disk only&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;80&quot; width=&quot;200&quot; x=&quot;900&quotLine truncated
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1262px" height="625px" viewBox="0 0 1262 625" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;Secret Distribution Flows&quot; id=&quot;secret-dist-flows-1&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1450&quot; pageHeight=&quot;700&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;repo&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;&quot; value=&quot;Provisioning Repo&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;template inventory&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;80&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;60&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;agentconfig&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;&quot; value=&quot;Vault Agent Config&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;on the target host&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;80&quot; width=&quot;200&quot; x=&quot;410&quot; y=&quot;60&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;bgfill_dataplane&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=none;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;290&quot; width=&quot;620&quot; x=&quot;380&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;dataplane&quot; parent=&quot;1&quot; style=&quot;swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#9673a6;dashed=1;fontSize=12;fontStyle=1;fontColor=#333333;align=center;spacingLeft=8;verticalAlign=middle;&quot; value=&quot;DATA PLANE — Vault-issued, installed locally&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;290&quot; width=&quot;620&quot; x=&quot;380&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;vaultagent&quot; parent=&quot;dataplane&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Vault Agent&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;140&quot; x=&quot;240&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;certflow&quot; parent=&quot;dataplane&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Certificate Flow&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;PKI engine, short-lived&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;180&quot; x=&quot;20&quot; y=&quot;180&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;secflow&quot; parent=&quot;dataplane&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;App Secret Flow&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;KV, rendered 0600&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;180&quot; x=&quot;220&quot; y=&quot;180&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;sshcertflow&quot; parent=&quot;dataplane&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;SSH Certificate Flow&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size:11px&amp;quot;&amp;gt;own timer, signs via Vault&amp;lt;br/&amp;gt;Agent&amp;#39;s token — not its&amp;lt;br/&amp;gt;render cycle&amp;lt;/font&aLine truncated

cheatsheets/README.md

@@ -0,0 +1,13 @@
+# Cheatsheets
+
+Quick-reference material, not deep dives. The [`docs/`](../docs/) directory is
+where the real engineering write-ups live — verified claims, real evidence,
+the actual design decisions behind each system. This directory is lower-key:
+command references and config notes for tools I use day-to-day, kept here so
+they're easy to find and easy to add to.
+
+- **[proxmox/](proxmox/)** — CLI cheat sheet for the hypervisor cluster this
+  fleet runs on (`qm`/`pct`/`pvesm`/`pveam`), plus API token auth and
+  certificate management notes.
+- **[zsh/](zsh/)** — the aliases, functions, and keybindings from my actual
+  zsh config, not a generic zsh tutorial.

cheatsheets/proxmox/proxmox-api-authentication.md

@@ -0,0 +1,18 @@
+## Create an API Token on Proxmox
+
+To create a new API Token for your `user` in Proxmox, follow these steps:
+
+1. Open the Proxmox Web UI and navigate to the **'Datacenter'** in the **'Server View'** menu.
+2. Select **'API Token'** in the **'Permissions'** menu.
+3. Click on **'Add'**.
+4. Select the `user`, and add a **'Token ID'**.
+5. (Optional) Disable `Privilege Separation`, and set an `Expire` date.
+6. Click on **'Add'**.
+
+## Check the API Token
+
+To test the API Token, you can use the following command:
+
+```sh
+curl -H "Authorization: PVEAPIToken=root@pam!monitoring=aaaaaaaaa-bbb-cccc-dddd-ef0123456789" https://your-proxmox-url:8006/api2/json/
+```

cheatsheets/proxmox/proxmox-certificate-management.md

@@ -0,0 +1,324 @@
+## Certificates for Intra-Cluster Communication
+
+Each Proxmox VE cluster creates by default its own (self-signed) Certificate
+Authority (CA) and generates a certificate for each node which gets signed by
+the aforementioned CA. These certificates are used for encrypted communication
+with the cluster's pveproxy service and the Shell/Console feature if SPICE is used.
+
+The CA certificate and key are stored in the Proxmox Cluster File System (pmxcfs).
+
+## Certificates for API and Web GUI
+
+The REST API and web GUI are provided by the pveproxy service, which runs on each node.
+You have the following options for the certificate used by pveproxy:
+
+* By default the node-specific certificate in `/etc/pve/nodes/NODENAME/pve-ssl.pem` is used. This certificate is signed by the cluster CA and therefore not automatically trusted by browsers and operating systems.
+* Use an externally provided certificate (e.g. signed by a commercial CA).
+* Use ACME (Let's Encrypt) to get a trusted certificate with automatic renewal, this is also integrated in the Proxmox VE API and web interface.
+
+For options 2 and 3 the file `/etc/pve/local/pveproxy-ssl.pem` (and
+`/etc/pve/local/pveproxy-ssl.key`, which needs to be without password) is used.
+Keep in mind that `/etc/pve/local` is a node specific symlink to
+`/etc/pve/nodes/NODENAME`.
+
+Certificates are managed with the Proxmox VE Node management command
+(see the `pvenode(1)` manpage). The current node certificate's details can
+be checked directly from the CLI too:
+
+```shell
+pvenode cert info [FORMAT_OPTIONS]
+```
+
+Do not replace or manually modify the automatically generated node
+certificate files in `/etc/pve/local/pve-ssl.pem` and
+`/etc/pve/local/pve-ssl.key` or the cluster CA files in
+`/etc/pve/pve-root-ca.pem` and `/etc/pve/priv/pve-root-ca.key`.
+
+## Upload Custom Certificate
+
+If you already have a certificate which you want to use for a Proxmox VE node
+you can upload that certificate simply over the web interface. Note that the
+certificates key file, if provided, mustn't be password protected.
+
+## Trusted certificates via Let's Encrypt (ACME)
+
+Proxmox VE includes an implementation of the Automatic Certificate
+Management Environment ACME protocol, allowing Proxmox VE admins to
+use an ACME provider like Let's Encrypt for easy setup of TLS certificates
+which are accepted and trusted on modern operating systems and web browsers
+out of the box.
+
+Currently, the two ACME endpoints implemented are the
+Let's Encrypt (LE) production and its staging
+environment. Our ACME client supports validation of http-01 challenges using
+a built-in web server and validation of dns-01 challenges using a DNS plugin
+supporting all the DNS API endpoints `acme.sh` does.
+
+### ACME Account
+
+You need to register an ACME account per cluster with the endpoint you want to
+use. The email address used for that account will serve as contact point for
+renewal-due or similar notifications from the ACME endpoint.
+You can register and deactivate ACME accounts over the web interface
+`Datacenter -> ACME` or using the `pvenode` command line tool.
+
+```shell
+pvenode acme account register account-name [email protected]
+```
+
+Because of rate-limits you should use LE staging for experiments or if you use
+ACME for the first time.
+
+### ACME Plugins
+
+The ACME plugins task is to provide automatic verification that you, and thus
+the Proxmox VE cluster under your operation, are the real owner of a domain. This is
+the basis building block for automatic certificate management.
+
+The ACME protocol specifies different types of challenges, for example the
+http-01 where a web server provides a file with a certain content to prove
+that it controls a domain. Sometimes this isn't possible, either because of
+technical limitations or if the address of a record to is not reachable from
+the public internet. The dns-01 challenge can be used in these cases. This
+challenge is fulfilled by creating a certain DNS record in the domain's zone.
+
+Proxmox VE supports both of those challenge types out of the box, you can configure
+plugins either over the web interface under `Datacenter -> ACME`, or using the
+`pvenode acme plugin add` command.
+
+ACME Plugin configurations are stored in `/etc/pve/priv/acme/plugins.cfg`.
+A plugin is available for all nodes in the cluster.
+
+### Node Domains
+
+Each domain is node specific. You can add new or manage existing domain entries
+under Node -> Certificates, or using the `pvenode config` command.
+
+After configuring the desired domain(s) for a node and ensuring that the
+desired ACME account is selected, you can order your new certificate over the
+web-interface. On success the interface will reload after 10 seconds.
+
+Renewal will happen automatically.
+
+## ACME HTTP Challenge Plugin
+
+There is always an implicitly configured standalone plugin for validating
+http-01 challenges via the built-in webserver spawned on port 80.
+
+The name `standalone` means that it can provide the validation on it's
+own, without any third party service. So, this plugin works also for cluster
+nodes.
+
+There are a few prerequisites to use it for certificate management with Let's
+Encrypts ACME.
+
+* You have to accept the ToS of Let's Encrypt to register an account.
+* Port 80 of the node needs to be reachable from the internet.
+* There must be no other listener on port 80.
+* The requested (sub)domain needs to resolve to a public IP of the Node.
+
+## ACME DNS API Challenge Plugin
+
+On systems where external access for validation via the http-01 method is
+not possible or desired, it is possible to use the dns-01 validation method.
+This validation method requires a DNS server that allows provisioning of TXT
+records via an API.
+
+### Configuring ACME DNS APIs for validation
+
+Proxmox VE re-uses the DNS plugins developed for the
+[acme.sh project](https://github.com/acmesh-official/acme.sh). Please
+refer to its documentation for details on configuration of specific APIs.
+
+The easiest way to configure a new plugin with the DNS API is using the web
+interface (`Datacenter -> ACME`).
+
+Choose DNS as challenge type. Then you can select your API provider, enter
+the credential data to access your account over their API.
+
+See the `acme.sh`
+[How to use DNS API](https://github.com/acmesh-official/acme.sh/wiki/dnsapi#how-to-use-dns-api)
+wiki for more detailed information about getting API credentials for your
+provider.
+
+As there are many DNS providers and API endpoints Proxmox VE automatically generates
+the form for the credentials for some providers. For the others you will see a
+bigger text area, simply copy all the credentials KEY=VALUE pairs in there.
+
+## DNS Validation through CNAME Alias
+
+A special alias mode can be used to handle the validation on a different
+domain/DNS server, in case your primary/real DNS does not support provisioning
+via an API. Manually set up a permanent CNAME record for
+`_acme-challenge.domain1.example` pointing to `_acme-challenge.domain2.example`
+and set the alias property in the Proxmox VE node configuration file to
+`domain2.example` to allow the DNS server of `domain2.example` to validate all
+challenges for `domain1.example`.
+
+### Combination of Plugins
+
+Combining http-01 and dns-01 validation is possible in case your node is
+reachable via multiple domains with different requirements / DNS provisioning
+capabilities. Mixing DNS APIs from multiple providers or instances is also
+possible by specifying different plugin instances per domain.
+Accessing the same service over multiple domains increases complexity and
+should be avoided if possible.
+
+## Automatic renewal of ACME certificates
+
+If a node has been successfully configured with an ACME-provided certificate
+(either via `pvenode` or via the GUI), the certificate will be automatically
+renewed by the `pve-daily-update.service`. Currently, renewal will be attempted
+if the certificate has expired already, or will expire in the next 30 days.
+A certificate can also be renewed directly, on demand:
+
+```shell
+pvenode acme cert renew [OPTIONS]
+```
+
+## ACME Examples with pvenode
+
+*Example*: Sample `pvenode` invocation for using Let's Encrypt certificates
+
+```sh
+root@proxmox:~# pvenode acme account register default [email protected]
+Directory endpoints:
+0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
+1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
+2) Custom
+Enter selection: 1
+Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
+Do you agree to the above terms? [y|N]y
+...
+Task OK
+root@proxmox:~# pvenode config set --acme domains=example.invalid
+root@proxmox:~# pvenode acme cert order
+Loading ACME account details
+Placing ACME order
+...
+Status is 'valid'!
+All domains validated!
+...
+Downloading certificate
+Setting pveproxy certificate and key
+Restarting pveproxy
+Task OK
+```
+
+*Example*: Setting up the OVH API for validating a domain
+
+The account registration steps are the same no matter which plugins are
+used, and are not repeated here.
+
+OVH_AK and OVH_AS need to be obtained from OVH according to the OVH
+API documentation
+
+First you need to get all information so you and Proxmox VE can access the API.
+
+```sh
+root@proxmox:~# cat /path/to/api-token
+OVH_AK=XXXXXXXXXXXXXXXX
+OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
+root@proxmox:~# source /path/to/api-token
+root@proxmox:~# curl -XPOST -H"X-Ovh-Application: $OVH_AK" -H "Content-type: application/json" \
+https://eu.api.ovh.com/1.0/auth/credential  -d '{
+  "accessRules": [
+    {"method": "GET","path": "/auth/time"},
+    {"method": "GET","path": "/domain"},
+    {"method": "GET","path": "/domain/zone/*"},
+    {"method": "GET","path": "/domain/zone/*/record"},
+    {"method": "POST","path": "/domain/zone/*/record"},
+    {"method": "POST","path": "/domain/zone/*/refresh"},
+    {"method": "PUT","path": "/domain/zone/*/record/"},
+    {"method": "DELETE","path": "/domain/zone/*/record/*"}
+]
+}'
+{"consumerKey":"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ","state":"pendingValidation","validationUrl":"https://eu.api.ovh.com/auth/?credentialToken=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}
+(open validation URL and follow instructions to link Application Key with account/Consumer Key)
+root@proxmox:~# echo "OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" >> /path/to/api-token
+```
+
+Now you can setup the ACME plugin:
+
+```sh
+root@proxmox:~# pvenode acme plugin add dns example_plugin --api ovh --data /path/to/api_token
+root@proxmox:~# pvenode acme plugin config example_plugin
+```
+
+```
+┌────────┬──────────────────────────────────────────┐
+│ key    │ value                                    │
+╞════════╪══════════════════════════════════════════╡
+│ api    │ ovh                                      │
+├────────┼──────────────────────────────────────────┤
+│ data   │ OVH_AK=XXXXXXXXXXXXXXXX                  │
+│        │ OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY  │
+│        │ OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ  │
+├────────┼──────────────────────────────────────────┤
+│ digest │ 867fcf556363ca1bea866863093fcab83edf47a1 │
+├────────┼──────────────────────────────────────────┤
+│ plugin │ example_plugin                           │
+├────────┼──────────────────────────────────────────┤
+│ type   │ dns                                      │
+└────────┴──────────────────────────────────────────┘
+```
+
+At last you can configure the domain you want to get certificates for and
+place the certificate order for it:
+
+```sh
+root@proxmox:~# pvenode config set --acmedomain0 example.proxmox.com,plugin=example_plugin
+root@proxmox:~# pvenode acme cert order
+Loading ACME account details
+Placing ACME order
+Order URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/11111111/22222222
+Getting authorization details from 'https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/33333333'
+The validation for example.proxmox.com is pending!
+[Wed Apr 22 09:25:30 CEST 2020] Using OVH endpoint: ovh-eu
+[Wed Apr 22 09:25:30 CEST 2020] Checking authentication
+[Wed Apr 22 09:25:30 CEST 2020] Consumer key is ok.
+[Wed Apr 22 09:25:31 CEST 2020] Adding record
+[Wed Apr 22 09:25:32 CEST 2020] Added, sleep 10 seconds.
+Add TXT record: _acme-challenge.example.proxmox.com
+Triggering validation
+Sleeping for 5 seconds
+Status is 'valid'!
+[Wed Apr 22 09:25:48 CEST 2020] Using OVH endpoint: ovh-eu
+[Wed Apr 22 09:25:48 CEST 2020] Checking authentication
+[Wed Apr 22 09:25:48 CEST 2020] Consumer key is ok.
+Remove TXT record: _acme-challenge.example.proxmox.com
+All domains validated!
+Creating CSR
+Checking order status
+Order is ready, finalizing order
+valid!
+Downloading certificate
+Setting pveproxy certificate and key
+Restarting pveproxy
+Task OK
+```
+
+### Example: Switching from the staging to the regular ACME directory
+
+Changing the ACME directory for an account is unsupported, but as Proxmox VE
+supports more than one account you can just create a new one with the
+production (trusted) ACME directory as endpoint. You can also deactivate the
+staging account and recreate it.
+
+*Example*: Changing the default ACME account from staging to directory using `pvenode`
+
+```sh
+root@proxmox:~# pvenode acme account deactivate default
+Renaming account file from '/etc/pve/priv/acme/default' to '/etc/pve/priv/acme/_deactivated_default_4'
+Task OK
+root@proxmox:~# pvenode acme account register default [email protected]
+Directory endpoints:
+0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
+1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
+2) Custom
+Enter selection: 0
+Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
+Do you agree to the above terms? [y|N]y
+...
+Task OK
+```

cheatsheets/proxmox/proxmox-cheat-sheet.md

@@ -0,0 +1,426 @@
+Proxmox Virtual Environment (Proxmox VE or PVE) is a hyper-converged infrastructure open-source software. It is a hosted hypervisor that can run operating systems including Linux and Windows on x64 hardware. It is a Debian-based Linux distribution with a modified Ubuntu LTS kernel and allows deployment and management of virtual machines and containers. Proxmox VE includes a web console and command-line tools, and provides a REST API for third-party tools. Two types of virtualization are supported: container-based with LXC (starting from version 4.0 replacing OpenVZ used in version up to 3.4, included), and full virtualization with KVM. It includes a web-based management interface.
+
+Proxmox VE is licensed under the GNU Affero General Public License, version 3.
+
+Repository: [https://git.proxmox.com](https://git.proxmox.com)
+Website: [https://pve.proxmox.com](https://pve.proxmox.com)
+
+## VM Management
+
+```shell
+# list VMs
+qm list
+
+# Create or restore a virtual machine.
+qm create your-vm-id
+
+# start a VM
+qm start your-vm-id
+
+# Suspend virtual machine.
+qm suspend your-vm-id
+
+# shutdown a VM
+qm shutdown your-vm-id
+
+# reboot a VM
+qm reboot your-vm-id
+
+# reset a VM
+qm reset your-vm-id
+
+# stop a VM
+qm stop your-vm-id
+
+# Destroy the VM and all used/owned volumes.
+# Removes any VM specific permissions and firewall rules
+qm destroy your-vm-id
+
+# Enter Qemu Monitor interface.
+qm monitor your-vm-id
+
+# Get the virtual machine configuration with both current and pending values.
+qm pending your-vm-id
+
+# Get the virtual machine configuration with pending changes applied.
+qm config your-vm-id [OPTIONS]
+
+# Send key event to virtual machine.
+qm sendkey your-vm-id your-key-event [OPTIONS]
+
+# Show command line which is used to start the VM (debug info).
+qm showcmd your-vm-id [OPTIONS]
+
+# Unlock the VM.
+qm unlock your-vm-id
+
+# Clone a VM
+qm clone your-vm-id new-vm-id
+
+# Migrate a VM
+qm migrate your-vm-id target-node
+
+# Migrate a VM to a remote cluster. EXPERIMENTAL.
+qm remote-migrate your-vm-id target-endpoint --target-bridge <string> --target-storage <string> [OPTIONS]
+
+# Create a new VM from an OVF manifest.
+qm importovf your-vm-id <manifest> <storage> [OPTIONS]
+
+# Enroll updated UEFI certificates to the EFI disk.
+qm enroll-efi-keys your-vm-id
+
+# Show VM status
+qm status your-vm-id
+
+# Wait until the VM is stopped.
+qm wait your-vm-id [OPTIONS]
+
+# Clean up resources for a VM
+qm cleanup your-vm-id your-clean-shutdown your-guest-requested
+
+# Create a Template.
+qm template your-vm-id [OPTIONS]
+
+# Set virtual machine options (synchrounous API)
+qm set your-vm-id [OPTIONS]
+```
+
+### Cloudinit
+
+```shell
+# Get automatically generated cloudinit config.
+qm cloudinit dump your-vm-id your-vm-type
+
+# Get the cloudinit configuration with both current and pending values.
+qm cloudinit pending your-vm-id
+
+# Regenerate and change cloudinit config drive.
+qm cloudinit update your-vm-id
+```
+
+### Disk
+
+```shell
+# Import an external disk image as an unused disk in a VM.
+# The image format has to be supported by qemu-img(1).
+qm disk import your-vm-id your-target-source your-target-storage
+
+# Move volume to different storage or to a different VM.
+qm disk move your-vm-id your-vm-disk [<storage>] [OPTIONS]
+
+# Rescan all storages and update disk sizes and unused disk images.
+qm disk rescan [OPTIONS]
+
+# Extend volume size.
+qm disk resize your-vm-id your-vm-disk <size> [OPTIONS]
+
+# Unlink/delete disk images.
+qm disk unlink your-vm-id --idlist <string> [OPTIONS]
+
+# rescan volumes
+qm rescan
+```
+
+### Snapshot
+
+```shell
+# List all snapshots.
+qm listsnapshot your-vm-id
+
+# Snapshot a VM
+qm snapshot your-vm-id <snapname>
+
+# Delete a snapshot.
+qm delsnapshot your-vm-id <snapname>
+
+# Rollback a snapshot
+qm rollback your-vm-id <snapname>
+
+# Open a terminal using a serial device
+# (The VM need to have a serial device configured, for example serial0: socket)
+qm terminal your-vm-id [OPTIONS]
+
+# Proxy VM VNC traffic to stdin/stdout
+qm vncproxy your-vm-id
+```
+
+### Misc
+
+```shell
+# Execute Qemu Guest Agent commands.
+qm guest cmd your-vm-id <command>
+
+# Executes the given command via the guest agent
+qm guest exec your-vm-id [<extra-args>] [OPTIONS]
+
+# Gets the status of the given pid started by the guest-agent
+qm guest exec-status your-vm-id <pid>
+
+# Sets the password for the given user to the given password
+qm guest passwd your-vm-id <username> [OPTIONS]
+```
+
+### PV, VG, LV Management
+
+```shell
+# Create a PV
+pvcreate <disk-device-name>
+
+# Remove a PV
+pvremove <disk-device-name>
+
+# List all PVs
+pvs
+
+# Create a VG
+vgcreate <vg-name> <disk-device-name>
+
+# Remove a VG
+vgremove <vg-name>
+
+# List all VGs
+vgs
+
+# Create a LV
+lvcreate -L <lv-size> -n <lv-name> <vg-name>
+
+# Remove a LV
+lvremove <vg-name>/<lv-name>
+
+# List all LVs
+lvs
+```
+
+### Storage Management
+
+```shell
+# Create a new storage.
+pvesm add <type> <storage> [OPTIONS]
+
+# Allocate disk images.
+pvesm alloc <storage> your-vm-id <filename> <size> [OPTIONS]
+
+# Delete volume
+pvesm free <volume> [OPTIONS]
+
+# Delete storage configuration.
+pvesm remove <storage>
+
+# Update storage configuration.
+pvesm set <storage> [OPTIONS]
+
+# List storage content.
+pvesm list <storage> [OPTIONS]
+
+# Get the filesystem path for a specified volume.
+pvesm path <volume>
+
+# Prune backups on a storage (standard naming scheme only).
+pvesm prune-backups <storage> [OPTIONS]
+
+# Extract configuration from a vzdump backup archive.
+pvesm extractconfig <volume>
+
+# Returns the storage API version and age.
+pvesm apiinfo
+
+# An alias for pvesm scan lvm.
+pvesm lvmscan
+
+# An alias for pvesm scan lvmthin.
+pvesm lvmthinscan
+
+# List local LVM volume groups.
+pvesm scan lvm
+
+# List local LVM Thin Pools.
+pvesm scan lvmthin <vg>
+
+# Get status for all datastores.
+pvesm status [OPTIONS]
+```
+
+### Template Management
+
+```shell
+# list all templates
+pveam available
+
+# list all templates
+pveam list <storage>
+
+# Download appliance templates
+pveam download <storage> <template>
+
+# Remove a template.
+pveam remove <template-path>
+
+# Update Container Template Database.
+pveam update
+```
+
+## Certificate Management
+
+See the [Proxmox Certificate Management](proxmox-certificate-management.md) cheat sheet.
+
+## Container Management
+
+```shell
+# List containers
+pct list
+
+# Create or restore a container.
+pct create your-vm-id <ostemplate> [OPTIONS]
+
+# Start the container.
+pct start your-vm-id [OPTIONS]
+
+# Create a container clone/copy
+pct clone your-vm-id new-vm-id [OPTIONS]
+
+# Suspend the container. This is experimental.
+pct suspend your-vm-id
+
+# Resume the container.
+pct resume your-vm-id
+
+# Stop the container.
+# This will abruptly stop all processes running in the container.
+pct stop your-vm-id [OPTIONS]
+
+# Shutdown the container.
+# This will trigger a clean shutdown of the container, see lxc-stop(1) for details.
+pct shutdown your-vm-id [OPTIONS]
+
+# Destroy the container (also delete all uses files).
+pct destroy your-vm-id [OPTIONS]
+
+# Show CT status.
+pct status your-vm-id [OPTIONS]
+
+# Migrate the container to another node. Creates a new migration task.
+pct migrate your-vm-id <target> [OPTIONS]
+
+# Migrate the container to a remote cluster. Creates a new migration task. EXPERIMENTAL.
+pct remote-migrate your-vm-id target-endpoint --target-bridge <string> --target-storage <string> [OPTIONS]
+
+# Get container configuration.
+pct config your-vm-id [OPTIONS]
+
+# Print the list of assigned CPU sets.
+pct cpusets
+
+# Get container configuration, including pending changes.
+pct pending your-vm-id
+
+# Reboot the container by shutting it down, and starting it again. Applies pending changes.
+pct reboot your-vm-id [OPTIONS]
+
+# Create or restore a container.
+pct restore your-vm-id <ostemplate> [OPTIONS]
+
+# Set container options.
+pct set your-vm-id [OPTIONS]
+
+# Create a Template.
+pct template your-vm-id
+
+# Unlock the VM.
+pct unlock your-vm-id
+```
+
+### Container Disks
+
+```shell
+# Get the container's current disk usage.
+pct df your-vm-id
+
+# Run a filesystem check (fsck) on a container volume.
+pct fsck your-vm-id [OPTIONS]
+
+# Run fstrim on a chosen CT and its mountpoints.
+pct fstrim your-vm-id [OPTIONS]
+
+# Mount the container's filesystem on the host.
+# This will hold a lock on the container and is meant for emergency maintenance only
+# as it will prevent further operations on the container other than start and stop.
+pct mount your-vm-id
+
+# Move a rootfs-/mp-volume to a different storage or to a different container.
+pct move-volume your-vm-id <volume> [<storage>] [<target-vmid>] [<target-volume>] [OPTIONS]
+
+# Unmount the container's filesystem.
+pct unmount your-vm-id
+
+# Resize a container mount point.
+pct resize your-vm-id your-vm-disk <size> [OPTIONS]
+
+# Rescan all storages and update disk sizes and unused disk images.
+pct rescan [OPTIONS]
+
+# Connect to container
+pct enter your-vm-id
+
+# Launch a console for the specified container.
+pct console your-vm-id [OPTIONS]
+
+# Launch a shell for the specified container.
+pct enter your-vm-id
+
+# Launch a command inside the specified container.
+pct exec your-vm-id [<extra-args>]
+
+# Copy a file from the container to the local system.
+pct pull your-vm-id <path> <destination> [OPTIONS]
+
+# Copy a local file to the container.
+pct push your-vm-id <file> <destination> [OPTIONS]
+```
+
+### Container Snapshot
+
+```shell
+# Snapshot a container.
+pct snapshot your-vm-id <snapname> [OPTIONS]
+
+# List all snapshots.
+pct listsnapshot your-vm-id
+
+# Rollback LXC state to specified snapshot.
+pct rollback your-vm-id <snapname> [OPTIONS]
+
+# Delete a LXC snapshot.
+pct delsnapshot your-vm-id <snapname> [OPTIONS]
+```
+
+## Web GUI
+
+```shell
+# Restart web GUI
+service pveproxy restart
+```
+
+## Resize Disk
+
+### Increase disk size
+
+Increase disk size in the GUI or with the following command
+
+```shell
+qm resize 100 virtio0 +5G
+```
+
+### Decrease disk size
+
+> Before decreasing disk sizes in Proxmox, you should take a backup!
+
+1. Convert qcow2 to raw: `qemu-img convert vm-100.qcow2 vm-100.raw`
+2. Shrink the disk `qemu-img resize -f raw vm-100.raw 10G`
+3. Convert back to qcow2 `qemu-img convert -p -O qcow2 vm-100.raw vm-100.qcow2`
+
+## Further information
+
+More examples and tutorials regarding Proxmox can be found in the link list below:
+
+- Ansible playbook that automates Linux VM updates running on Proxmox (including snapshots): [TheDatabaseMe - update_proxmox_vm](https://github.com/thedatabaseme/update_proxmox_vm)
+- Manage Proxmox VM templates with Packer: [Use Packer to build Proxmox images](https://thedatabaseme.de/2022/10/16/what-a-golden-boy-use-packer-to-build-proxmox-images/)

cheatsheets/zsh/zsh-cheatsheet.md

@@ -0,0 +1,169 @@
+# Zsh Cheatsheet
+
+Reference for the new ZDOTDIR-based zsh setup (oh-my-zsh removed,
+config lives in `~/.config/zsh/`).
+
+## Aliases
+
+### Docker Compose
+| Alias | Expands to | What it does |
+|---|---|---|
+| `dcu` | `docker compose up -d` | Start containers in detached mode |
+| `dcd` | `docker compose down` | Stop and remove containers |
+| `dcr` | `docker compose restart` | Restart containers |
+| `dcp` | `docker compose pull` | Pull latest images |
+
+### Navigation
+| Alias | Expands to | What it does |
+|---|---|---|
+| `..` | `cd ..` | Up one directory |
+| `...` | `cd ../..` | Up two directories |
+| `....` | `cd ../../..` | Up three directories |
+
+### File listing
+| Alias | Expands to | What it does |
+|---|---|---|
+| `ls` | `eza -lahF --group-directories-first --time-style="+%d/%m/%Y %H:%M" --git` (falls back to plain `ls -lahF ...` if eza isn't installed) | Long listing, all files, classify symbols (`/`, `*`, `@`), directories first, custom date format, git status column |
+| `tree` | `eza --tree` (only if eza is installed) | Tree view of directory structure |
+| `cat` | `bat` (only if bat is installed) | Syntax-highlighted file viewer |
+
+### System
+| Alias | Expands to | What it does |
+|---|---|---|
+| `df` | `df -h` | Disk usage, human-readable sizes |
+| `free` | `free -h` | Memory usage, human-readable sizes |
+| `c` | `clear` | Clear the terminal |
+
+## Functions
+
+| Function | Usage | What it does |
+|---|---|---|
+| `cd` (overridden) | `cd <dir>` | Changes directory, then automatically runs `ls` (so you always see the contents of where you land). `cd` with no argument goes to `$HOME`. |
+| `mkcd` | `mkcd <dir>` | Creates a directory and `cd`s into it in one step |
+| `extract` | `extract <file>` | Extracts an archive based on its extension. Supports `.tar.gz`/`.tgz`, `.tar.bz2`, `.tar.xz`, `.zip`, `.gz` |
+| `ports` | `ports` | Shows listening ports and the processes using them (`sudo ss -tulpen`) |
+| `psg` | `psg <name>` | Searches running processes by name (case-insensitive `ps aux | grep`) |
+| `killport` | `killport <port>` | Kills whatever process is listening on the given port |
+| `logtail` | `logtail <file> [lines]` | Tails a log file (`-f`), default 50 lines, or specify a custom number |
+| `reload` | `reload` | Reloads the zsh config (`source $ZDOTDIR/.zshrc`) without restarting the shell |
+
+## Keybindings
+
+| Key | Action | Notes |
+|---|---|---|
+| `Ctrl + →` | Move cursor forward one word | |
+| `Ctrl + ←` | Move cursor backward one word | |
+| `Ctrl + \` | Toggle autosuggestions on/off | Echoes "Autosuggestions: on/off" so you know the current state |
+| `↑` (Up arrow) | Search history backward by prefix | Type a few characters first, then press Up to find matching previous commands |
+| `↓` (Down arrow) | Search history forward by prefix | Same as above, opposite direction |
+
+## Line editing (built-in zsh defaults)
+
+These come from zsh's default "emacs" editing mode (the standard unless
+something switches it to vi mode — this setup doesn't). Most of these
+work in any emacs-style shell (bash, zsh) and are worth building into
+muscle memory since they work everywhere.
+
+### Cursor movement
+
+| Key | Action |
+|---|---|
+| `←` / `→` | Move cursor one character left/right |
+| `Ctrl + ←` / `Ctrl + →` | Move cursor one **word** left/right (custom binding in this setup) |
+| `Ctrl + A` | Jump to the **start** of the line |
+| `Ctrl + E` | Jump to the **end** of the line |
+| `Alt + B` | Move backward one word (alternative to Ctrl+←) |
+| `Alt + F` | Move forward one word (alternative to Ctrl+→) |
+
+### Deleting / cutting text
+
+| Key | Action |
+|---|---|
+| `Backspace` | Delete the character **before** the cursor |
+| `Ctrl + D` / `Delete` | Delete the character **under/after** the cursor (Ctrl+D on an empty line exits the shell — careful) |
+| `Ctrl + W` | Delete the **word before** the cursor (cuts to kill-ring) |
+| `Alt + D` | Delete the **word after** the cursor (cuts to kill-ring) |
+| `Alt + Backspace` | Same as Ctrl+W — delete word before cursor |
+| `Ctrl + U` | Delete **everything before** the cursor (whole line back to start) |
+| `Ctrl + K` | Delete **everything after** the cursor (to end of line) |
+
+### Pasting (yanking) deleted text
+
+zsh doesn't have "select text" like a GUI — instead, anything you cut
+with `Ctrl+W`, `Alt+D`, `Ctrl+U`, or `Ctrl+K` goes into a clipboard-like
+buffer (the "kill ring"), which you can paste back with:
+
+| Key | Action |
+|---|---|
+| `Ctrl + Y` | Paste ("yank") the last cut text at the cursor |
+
+**Practical example:** to move a word from the end of a command to the
+start — put cursor at end of word, `Alt + Backspace` to cut it, `Ctrl + A`
+to jump to the start of the line, `Ctrl + Y` to paste it there.
+
+### Undo / retype
+
+| Key | Action |
+|---|---|
+| `Ctrl + _` (Ctrl + Shift + -) | Undo the last edit |
+| `Ctrl + L` | Clear the screen (same as the `c` alias, but doesn't lose scrollback the same way) |
+
+### Transposing / fixing typos
+
+| Key | Action |
+|---|---|
+| `Ctrl + T` | Swap the two characters around the cursor (fixes a typo like `hte` → `the`) — **note:** this is overridden by fzf's Ctrl+T in this setup if fzf is installed |
+| `Alt + T` | Swap the two **words** around the cursor |
+
+### Case changes
+
+| Key | Action |
+|---|---|
+| `Alt + U` | Uppercase the word from the cursor onward |
+| `Alt + L` | Lowercase the word from the cursor onward |
+| `Alt + C` | Capitalize the word from the cursor onward |
+
+### Re-running / editing previous commands
+
+| Key | Action |
+|---|---|
+| `!!` | Re-run the last command (e.g. `sudo !!` re-runs the last command with sudo) |
+| `!$` | The last **argument** of the previous command (e.g. `cat !$` opens whatever file you last referenced) |
+| `Ctrl + R` (no fzf) | Reverse incremental search through history (built-in, before fzf takes over — see fzf section below for the upgraded version) |
+| `fc` | Opens the last command in `$EDITOR` (nano) for editing, then runs it |
+
+## Fuzzy search (fzf)
+
+Only active if `fzf` is installed.
+
+| Key | Action | Notes |
+|---|---|---|
+| `Ctrl + R` | Fuzzy search command history | Type to filter, select a command, it's inserted at the cursor. This overrides the built-in reverse search mentioned above |
+| `Ctrl + F` | Fuzzy file search (includes hidden files, excludes `.git`) | Selected file path is inserted at the cursor. Uses `fd` if installed, otherwise falls back to `find` |
+| `Ctrl + T` | Fuzzy file search (fzf's built-in, non-hidden files) | Only active if fzf's own shell integration is found on the system; uses `fd` for speed if installed. This overrides the built-in character-transpose shortcut mentioned above |
+
+## Plugins (auto-loaded on shell startup)
+
+These are cloned automatically into `$ZDOTDIR/plugins/` the first time
+you start a shell after the role runs — no manual setup needed.
+
+| Plugin | What it does |
+|---|---|
+| `zsh-autosuggestions` | Shows a greyed-out suggestion as you type, based on history. Press `→` (right arrow) or `End` to accept it |
+| `fast-syntax-highlighting` | Colors commands as you type — valid commands, paths, strings, etc. get different colors so you can spot typos before pressing Enter |
+
+## Environment / history
+
+- History file: `~/.local/state/zsh/history`
+- History size: 10,000 commands
+- History behavior: shared across sessions, duplicates ignored, commands
+  starting with a space aren't recorded (`HIST_IGNORE_SPACE`)
+- Default editor: `nano` (`$EDITOR` / `$VISUAL`)
+- Config location: `$ZDOTDIR` = `~/.config/zsh/`
+- Per-host extra env vars (e.g. `VAULT_ADDR`) are set via `zsh_extra_env`
+  in Ansible host_vars, appended to `env.zsh`
+
+## Startup
+
+`fastfetch` runs automatically every time a new shell opens, showing
+system info (OS, kernel, uptime, packages, etc.) above the prompt.

docs/architecture-overview.md

@@ -1,3 +1,9 @@
+<!-- verified: 2026-09-18
+     rocky-docker-lxc@d225776 : caddy/conf/Caddyfile
+     caddy-vm@c5c3420 : caddy/conf/Caddyfile
+     trilium : 2wCPV6NXQa3l, 0mu75aAXGH8X, fscYadnUtmD5, QMTh5G7ldlLV, TIbCwYGitboy, Cl4TWBZ7ytYr, 6cJurv76PNDR, nAPJq9A1iuX8, DvKxK0gHXsng
+-->
+
 # 📐 Architecture Overview
 
 > [!NOTE]
@@ -5,7 +11,7 @@
 
 <table>
 <tr>
-<td align="center"><strong>11</strong><br/>fleet hosts</td>
+<td align="center"><strong>12</strong><br/>fleet hosts</td>
 <td align="center"><strong>4</strong><br/>trust boundaries</td>
 <td align="center"><strong>3</strong><br/>routing authorities</td>
 <td align="center"><strong>8</strong><br/>traffic laws</td>
@@ -25,13 +31,13 @@ The actual answer here is **four boundaries, each doing a different job**:
 | 🔴 WAN → edge | Anyone on the internet | Public reverse proxy, firewall-filtered | The only boundary facing genuinely untrusted traffic — gets the strictest, most isolated treatment |
 | 🔵 Browsers → user-facing proxy | A human, on a device that could be compromised, phished, or simply lost | Mutual TLS (client cert required) + SSO | The highest-value target for credential theft — a stolen password alone isn't enough without the client cert too |
 | 🟢 App-to-app | A container that's already inside the network, already provisioned by the same pipeline as everything else | IP-allowlist + SSO fallback, no mTLS | Full mTLS here would mean managing certificate lifecycle for every internal service-to-service call — real overhead for traffic that's already constrained to known, provisioned sources. The risk profile doesn't justify the same control as human-operated browsers. |
-| 🟡 LAN infra-to-infra | Non-HTTP traffic — metrics, logs, NTP, agent reporting | Explicit per-flow firewall rule, no proxy at all | Can't go through a reverse proxy — it isn't HTTP. The firewall itself is the only layer that *can* enforce this one. |
+| 🟡 LAN infra-to-infra | Non-HTTP traffic — metrics, logs, NTP, agent reporting | Explicit per-flow rule at each host's own firewalld — the real gate, since same-subnet traffic never transits a router | Can't go through a reverse proxy — it isn't HTTP. Each host's own firewall is the layer that actually sees this traffic; pfSense carries a mirrored ruleset for anything that does route through it, as defense-in-depth. |
 
 Every boundary above only means something if a service can't just skip it by reading the wrong header: **backend services trust proxy headers, never the raw client IP.** If that weren't true, the mTLS/SSO enforcement happening at the proxy layer would be theater — anything downstream could just claim to be a trusted IP and walk straight through. The proxy is the only thing allowed an opinion on who a client really is.
 
 **Three separate routing authorities enforce these** — the edge firewall, the user-facing proxy, and the internal proxy zone — rather than one central router handling everything. That's a deliberate rejection of the simpler single-router design: one router is one thing to hardening-focus on and one thing that takes everything down if it's compromised or misconfigured.
 
-### What actually happens when one fails
+### What Actually Happens When One Fails
 
 Not a hypothetical — this is the real, deliberate payoff of splitting routing authority three ways instead of one:
 
@@ -50,11 +56,11 @@ A single-router design fails this test completely — one outage, one misconfigu
 >
 > The second fix was conceptual, and the bigger one: for a long time the app-to-app boundary simply didn't exist. Internal service traffic crossed a flat, unauthenticated mesh network — anything on it could reach anything else on it, no proxy, no identity check, nothing. That's exactly the "you're on the network, you're trusted" pattern this document opens by calling the easy, wrong answer. Replacing it with the two-zone model in [`docs/dns-naming.md`](dns-naming.md) is what gave app-to-app traffic a real boundary for the first time.
 >
-> The third fix found the same flaw hiding somewhere less obvious: plain LAN-to-LAN traffic — metrics, logs, NTP, agent reporting — still had a blanket allow at the firewall, even after the app-to-app zone had its own identity-checked proxy. Any host could reach any other host; the only thing standing between "one compromised container" and "everything else on the LAN" was whatever that individual host's own firewall happened to be configured to do, which is a coin flip, not a boundary. Closing it meant giving up "it just works" for LAN infra traffic and writing an explicit rule for every flow that's actually supposed to exist — the same trade this document makes everywhere else: a boundary only counts if crossing it takes a decision, not an accident of network topology.
+> The third fix found the same flaw hiding somewhere less obvious: plain LAN-to-LAN traffic — metrics, logs, NTP, agent reporting — still had a blanket allow, even after the app-to-app zone had its own identity-checked proxy. Any host could reach any other host; the only thing standing between "one compromised container" and "everything else on the LAN" was whatever that individual host's own firewall happened to be configured to do, which is a coin flip, not a boundary. Closing it meant writing an explicit per-flow rule everywhere that boundary is actually enforced: each host's own firewalld (the real gate — this LAN is one flat /24, so traffic between two hosts on it never transits a router at all) and, for anything that does route through pfSense itself, a matching explicit-allow-plus-catch-all-deny ruleset there too, as defense-in-depth and a stated record of intent.
 >
 > Three different places, the same failure mode, the same fix each time — that's not a coincidence, it's what happens when you actually go looking for where "trusted by default" is still hiding.
 
-That third fix isn't just a claim — it's the actual live ruleset:
+That third fix isn't just a claim. Here's pfSense's half of it — even though genuine host-to-host traffic on this flat LAN never actually reaches it:
 
 ![pfSense LAN catch-all deny rule: block+log east-west traffic not explicitly allowed](../assets/pfsense-rules-lan-catchall-deny.png)
 
@@ -63,9 +69,9 @@ That third fix isn't just a claim — it's the actual live ruleset:
 <details>
 <summary>See the rest of the actual pfSense configuration</summary>
 
-**WAN — only three things pass inbound, everything else default-deny:**
+**WAN — only four things pass inbound, everything else default-deny:** SSH management, HTTPS via HAProxy, an HTTP→HTTPS redirect, and a STUN port-forward for the NetBird mesh VPN's NAT traversal.
 
-![pfSense WAN rules: SSH management, HTTPS via HAProxy, HTTP redirect, bogon block](../assets/pfsense-rules-wan.png)
+![pfSense WAN rules: SSH management, HTTPS via HAProxy, HTTP redirect, NetBird STUN, bogon block](../assets/pfsense-rules-wan.png)
 
 **LAN — fleet observability and security-agent reporting, one explicit rule per flow:**
 
@@ -87,24 +93,24 @@ That third fix isn't just a claim — it's the actual live ruleset:
 
 </details>
 
-## The other six traffic laws
+## The Other Six Traffic Laws
 
-Each of these gets its own full mechanism doc elsewhere in this repo — this table is the map, not the territory:
+Laws 1–2 are covered above (the trust-boundary table); these are 3–8. Each of these gets its own full mechanism doc elsewhere in this repo — this table is the map, not the territory:
 
 | # | Law | Failure behavior |
 |---|---|---|
-| 3. Secrets | Applications never talk to Vault directly — one hop through a local agent only. See [`docs/secrets-and-pki.md`](secrets-and-pki.md). | Vault down → secret renewal stops, services run until their current token expires. No immediate disruption. |
+| 3 | **Secrets.** Applications never talk to Vault directly — one hop through a local agent only, with one narrow exception: a dedicated SSH-certificate refresh reuses that agent's token to call Vault directly. See [`docs/secrets-and-pki.md`](secrets-and-pki.md) and [`docs/secret-distribution.md`](secret-distribution.md). | Vault down → secret/certificate renewal stops, services run until their current token or cert expires — no immediate disruption. The one flow that doesn't degrade this gently: the SSH-certificate exception above has no static-key fallback, so a sustained outage eventually locks the automation identity out of the fleet entirely, not just "stale but running." |
-| 4. DNS | Which internal zone a hostname resolves in determines which trust boundary (above) a request enters. See [`docs/dns-naming.md`](dns-naming.md). | DNS down → app-to-app resolution fails; a direct-IP fallback exists, and it bypasses the SSO gate — but only for traffic already coming from an IP on the trusted allowlist. Anything else still hits Authentik regardless of DNS. |
+| 4 | **DNS.** Which internal zone a hostname resolves in determines which trust boundary (above) a request enters. See [`docs/dns-naming.md`](dns-naming.md). | DNS down → app-to-app resolution fails; a direct-IP fallback exists, and it bypasses the SSO gate — but only for traffic already coming from an IP on the trusted allowlist. Anything else still hits Authentik regardless of DNS. |
-| 5. Observability | Logs and metrics push asynchronously — never inline with application traffic, full stop. See [`docs/observability.md`](observability.md) and [`docs/security-monitoring.md`](security-monitoring.md). | Monitoring down → collection stops; every application service keeps running, completely unaffected. |
+| 5 | **Observability.** Logs and metrics push asynchronously — never inline with application traffic, full stop. See [`docs/observability.md`](observability.md) and [`docs/security-monitoring.md`](security-monitoring.md). | Monitoring down → collection stops; every application service keeps running, completely unaffected. |
-| 6. Notifications | Two independent branches (message-queue and HTTP webhook) converge on one formatter; neither is part of any service's runtime dependency chain. See [`docs/notifications.md`](notifications.md). | Notification delivery is event-driven, not persistent inline execution — a failure here doesn't touch application traffic either way. |
+| 6 | **Notifications.** Two independent branches (message-queue and HTTP webhook) converge on one formatter; neither is part of any service's runtime dependency chain. See [`docs/notifications.md`](notifications.md). | Notification delivery is event-driven, not persistent inline execution — a failure here doesn't touch application traffic either way. |
-| 7. CI/CD | Build/deploy traffic never sits inline with runtime; schedule-triggered, not push-triggered. See [`docs/registry-and-build.md`](registry-and-build.md). | A failed deploy doesn't affect already-running services — enforced by the redeploy step's own rollback logic. |
+| 7 | **CI/CD.** Build/deploy traffic never sits inline with runtime; schedule-triggered, not push-triggered. See [`docs/registry-and-build.md`](registry-and-build.md). | A failed deploy doesn't affect already-running services — enforced by the redeploy step's own rollback logic. |
-| 8. Management | Administrative access is its own separate plane — one dedicated identity, non-standard ports, explicit per-host firewall rules from one source only. | Management access down → administration unavailable; every running service is completely unaffected. |
+| 8 | **Management.** Administrative access is its own separate plane — one dedicated identity, non-standard ports, explicit per-host firewall rules from one source only on the LAN, plus a separate zero-trust mesh giving that same access a scoped path in from off-LAN too. See [`docs/netbird-zero-trust.md`](netbird-zero-trust.md). | On-LAN admin access never depended on the mesh and is unaffected if it goes down; off-LAN access specifically loses its only path until the mesh recovers. Every running service is unaffected either way. |
 
 The pattern repeats across all eight laws, not just the trust-boundary story above: **control-plane failure never takes down the data plane.** Every "failure behavior" row above ends the same way — the thing that broke stops working, and everything else keeps running regardless. That's not a coincidence; it's the same design principle applied consistently everywhere, not just to network trust.
 
-## What this map deliberately excludes
+## What This Map Deliberately Excludes
 
 Not a firewall rule listing, not a port inventory, not a service configuration reference — those live in [`docs/firewall-and-network.md`](firewall-and-network.md) and the docs linked in the table above. This is strictly the stable, directional shape of how trust and traffic actually move — meant to still be true even after the specific tools implementing it change again, the way it already has three times.
 
 ---
-Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md)
+Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/netbird-zero-trust.md`](netbird-zero-trust.md)

docs/cron-scheduler.md

@@ -1,13 +1,17 @@
+<!-- verified: 2026-09-18
+     trilium : EJGewkYGStHf
+-->
+
 # ⏱️ Cron Scheduler
 
 > [!NOTE]
 > A NocoDB-backed inventory of every known scheduled job in the fleet — Kestra flows, systemd timers, Ansible-managed cron, app-internal schedules — visualized as a Dashy heatmap so a new job's time can be picked without colliding with existing load. Small in scope compared to the rest of this repo, but it's a real answer to a problem every one of the other systems here creates: once you have 50 independently-scheduled jobs across [`docs/orchestration.md`](orchestration.md), [`docs/registry-and-build.md`](registry-and-build.md), and [`docs/disaster-recovery.md`](disaster-recovery.md), "when does anything run" stops being something you can hold in your head.
 
-## The heatmap
+## The Heatmap
 
 A Node-RED tab fetches the inventory straight from NocoDB's REST API at request time — no cache — and renders it as a 7×24 grid (Mon–Sun × 00:00–23:00). Each cell's color is a **computed load weight**, not a raw job count: base `1` per job, plus a duration-bucket weight, plus a resource-tag weight (`disk-intensive`=2, `cpu-intensive`=1.5, `network-intensive`=0.5, summed if a job carries more than one), plus a flat `+1` if the job's category is `backup` — disk I/O contention from an overlapping backup is judged a worse collision than two lightweight jobs sharing an hour, and the weight formula says so explicitly rather than pretending every job costs the same. Weights bucket into 4 color bands (green → yellow → orange → red) relative to the current busiest cell. Jobs whose time can't be resolved to a specific day+hour are excluded from the grid and listed separately rather than forced into a misleading cell.
 
-This is the real heatmap, not a description of it — 50 known jobs, 15 without a fixed time:
+This is the real heatmap, not a description of it — 49 known jobs, 14 without a fixed time:
 
 ![The real Fleet Schedule Heatmap widget in Dashy, with Tue 00:00 clicked open showing its 8 real jobs — name, host, category, and FIXED/APP UI badges for jobs that can't simply be rescheduled](../assets/cron-scheduler-heatmap.png)
 
@@ -16,13 +20,13 @@ This is the real heatmap, not a description of it — 50 known jobs, 15 without
 
 Jobs with no fixed time get their own list instead of a misleading grid cell:
 
-![The heatmap's "no fixed time" section expanded — 15 real jobs excluded from the grid, including the manually-triggered-only Kestra flows and the PBS prune job with its mismatched datastore name](../assets/cron-scheduler-no-fixed-time.png)
+![The heatmap's "no fixed time" section expanded — 14 real jobs excluded from the grid, including the manually-triggered-only Kestra flows and Hawser's continuous 30-second heartbeat agent](../assets/cron-scheduler-no-fixed-time.png)
 
 Both of those are Dashy's *rendered* view. This is the raw NocoDB table underneath it — the actual data structure, not the heatmap built from it:
 
-![The real "Cron Scheduler" NocoDB table in grid view, showing all 50 records with their Job Name, Category, Schedule Expr, Est Start, Duration, Resource Tags, and Source columns](../assets/cron-scheduler-table.png)
+![The real "Cron Scheduler" NocoDB table in grid view, showing all 49 records with their Job Name, Category, Schedule Expr, Est Start, Duration, Resource Tags, and Source columns](../assets/cron-scheduler-table.png)
 
-## Why a manual inventory instead of an auto-discovered one
+## Why A Manual Inventory Instead Of An Auto-discovered One
 
 The harder, more "correct" version of this tool would auto-discover every scheduled job by querying Kestra's API, parsing every host's crontab and systemd timers, and re-syncing continuously. That was scoped down deliberately: auto-discovery across three genuinely different scheduling mechanisms (Kestra's API, systemd's timer units, cron's own format) is real engineering effort for a tool whose entire job is picking a good time slot for the next new job — a task that happens occasionally, not continuously. A manually-maintained table, kept honest by being checked at the one moment it actually matters (adding a new job), was judged the better return on effort than building and maintaining three different scanners for a problem that doesn't need real-time accuracy.
 

docs/dashy-soc-board.md

@@ -1,3 +1,7 @@
+<!-- verified: 2026-09-19
+     trilium : 5XjzoVTH1r3B, cVFNSn4CuuEN
+-->
+
 # 🖥️ Dashy SOC Board
 
 > [!NOTE]
@@ -10,10 +14,10 @@ This board didn't start from nothing — it absorbed several formerly-standalone
 | Source | What it actually contributes |
 |---|---|
 | Wazuh indexer | 7-day findings, categorized: ClamAV, vulnerabilities, rootkit/anomaly, authentication, other |
-| Vault | Seal state, active lease/token counts, KV engine counts, token TTL distribution, and root/intermediate CA cert expiry — parsed live from Vault's own `pki/cert/ca` and `pki_int/cert/ca` endpoints on every load, not a cached value |
+| Vault | Seal state, active lease/token counts, KV engine counts, and token TTL distribution — read from Prometheus metrics, not a live Vault call, so all of them go stale together if the metrics-reader token lapses. Root/intermediate CA cert expiry is the one exception, parsed live from Vault's own `pki/cert/ca` and `pki_int/cert/ca` endpoints on every load regardless of metrics health |
 | Technitium DNS | Query stats, top clients/domains, blocklist state, plus a dedicated "DNS Security Signals" section: forwarder failures (read from the Loki-shipped query log), query-log ingestion lag, next blocklist update time |
 | Fleet config-drift audit | *Not* computed by this board's own function — read from Node-RED's shared `fleetAudit` global, populated by a separate ingestion process, covering 4 finding types: untracked stacks, missing declared directories, containers stuck in `Created`, and TZ drift |
-| Slack | 12 app-specific channels (the *arr suite, Jellyfin/Emby/Seerr, Kopia, Authentik, Gitea, Netbox, Grafana) mapped to their owning host by a plain lookup table, plus each host's own alert channel |
+| Slack | 12 app-specific channels (the *arr suite, Jellyfin/Emby/Seerr, Kopia, Authentik, Gitea, Proxmox, Grafana) mapped to their owning host by a plain lookup table, plus each host's own alert channel |
 | Proxmox `cluster/resources` API | The two hypervisor nodes' own CPU/RAM/disk (no Prometheus coverage on bare metal) plus the full running VM/LXC tree per node |
 
 The fourteen, enumerated directly from the function's own upstream-URL constants rather than counted by hand — this is the actual grep and its real output, not a reconstruction:
@@ -44,10 +48,10 @@ for n in data:
     if n.get('type') == 'function' and n.get('name') == 'Build SOC Board HTML':
         print(n['name'], '->', len(n['func']), 'chars')
 "
-Build SOC Board HTML -> 104443 chars
+Build SOC Board HTML -> 104635 chars
 ```
 
-## Why one function instead of native Dashy widgets
+## Why One Function Instead of Native Dashy Widgets
 
 Dashy's own built-in widget types cover simple cases well, but none of them can join data across sources the way this board needs to — a per-host tile that combines Wazuh findings, Prometheus resource usage, and Slack alert history into one expandable panel isn't expressible as a single native widget config. Building it as one Node-RED function that does the aggregation server-side, then hands the dashboard pre-joined data, was the tradeoff made: a large, custom function to maintain, in exchange for a board that actually reflects cross-system state instead of fourteen disconnected tiles a human has to mentally correlate themselves.
 
@@ -66,7 +70,7 @@ Clicking through to one of those "folded-in specialty widgets" — the Vault sea
 > [!TIP]
 > **Deliberately scoped**: security/error/warning signal only. Per-app uptime is handled separately by Dashy's own native status checks on individual tiles — the only up/down signal inside this board is *infrastructure-tier* reachability from Prometheus, not per-app availability. Keeping the scope narrow — resisting the temptation to also make this the uptime dashboard — is what keeps the board actually scannable instead of turning into a second copy of Grafana.
 
-## Every finding ships with a copy-ready incident summary
+## Every Finding Ships With a Copy-ready Incident Summary
 
 Each finding's expanded row has a "Copy Debug Prompt" button, not just an acknowledge control. Clicking it builds a fully-formed summary from that specific finding's own host, severity, title, and detail, and copies it straight to the clipboard — a complete, structured starting point for triage without manually re-gathering context from the board itself. This is the real template:
 
@@ -80,7 +84,7 @@ Detail: {meta}
 ```
 
 
-## Acknowledge mechanism: client-side, no backend
+## Acknowledge Mechanism: client-side, no backend
 
 ![A finding's stable key (hostId::group::title) gets acknowledged into the browser's localStorage as a flat map, which live re-derives the host tile badge, the Active Issues strip, and any open detail panel's gauges — and never touches Wazuh, Slack, or any other upstream source](../assets/dashy-soc-board-acknowledge.svg)
 
@@ -95,7 +99,7 @@ localStorage.getItem("socAckedV2")
 > [!IMPORTANT]
 > Acknowledging a finding is a **display filter only** — it never writes back to Wazuh, Slack, or any upstream source described elsewhere in this repo, is scoped per-browser rather than fleet-wide, and a full data set for every host is always kept in memory (in inert template elements, not just whichever panel is currently open) so the recompute has everything it needs regardless of which panel happens to be open. A reload in a different browser shows the finding again. This was a deliberate tradeoff favoring simplicity — no backend state to build, secure, or back up — over persistence, made because the board's job is triage visibility, not an audit trail of who acknowledged what (that's what Wazuh's own alert log in [`docs/security-monitoring.md`](security-monitoring.md) is for).
 
-## Real integration gotchas hit building this
+## Real Integration Gotchas Hit Building This
 
 > [!WARNING]
 > A few non-obvious lessons from actually building this, in case they save someone else the debugging time:
@@ -105,7 +109,7 @@ localStorage.getItem("socAckedV2")
 > - Node-RED's Mustache templating in a core node's URL field is scoped to `msg` directly, not `msg.msg` — the correct reference is `{{{url}}}`, not `{{{msg.url}}}`, and it needs the triple-brace form specifically for values containing characters like `&` that the double-brace form would HTML-escape. Get either wrong and the request silently goes to an empty or mangled URL instead of erroring.
 > - Replacing a bind-mounted config file's content doesn't get picked up by a plain container restart — the container keeps watching the old inode; a full recreate is required, the same gotcha that shows up in [`docs/registry-and-build.md`](registry-and-build.md)'s redeploy path.
 
-## Widget design: shape follows data, not one template everywhere
+## Widget Design: shape follows data, not one template everywhere
 
 Every widget's visual style is chosen to match the *shape* of the data it shows, not a single card/badge component reused everywhere — the same design instinct behind this repo's own docs, honestly:
 
@@ -115,6 +119,7 @@ Every widget's visual style is chosen to match the *shape* of the data it shows,
 | Hierarchy (host containing guests) | Real visual nesting — indentation, a tree — not a flat grid |
 | Time-ordered audit/status log | A dense, sortable list ranked worst-first, recency as the dominant signal |
 | Named media/content items | A poster/cover-art grid, never a text list |
+| Large flat set of homogeneous records (every listening service, every VM/LXC) | One searchable list with click-to-expand rows — a live text filter, mandatory past ~100 rows, not a card wall or an unbroken scroll |
 
 > [!TIP]
 > The media-poster widgets route through one shared image-proxy endpoint rather than each hitting its own upstream API directly — since these are private, authenticated homelab APIs, not public image CDNs, and a browser can't hold the credentials each one needs. One exception: a public poster CDN gets hit directly, since no auth or proxying is needed there.

docs/disaster-recovery.md

@@ -1,37 +1,44 @@
+<!-- verified: 2026-09-18
+     iac@ae315de : ansible/roles_metadata.yml
+     ansible@24e1bf0 : roles/os_update, playbooks/os_update_vms.yml, playbooks/os_update_pve.yml, playbooks/os_update_pbs.yml, playbooks/app_update.yml
+     trilium : xJRSQovMncQr, 6AbdS9cUE45T, bW84ccRoAJ5i, p45qNP2sIuhA, 6oESvJYzj1at
+-->
+
 # 🧯 Disaster Recovery & Upgrades
 
 > [!NOTE]
 > Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose. Both halves of this doc are about the same underlying discipline: assuming things will eventually fail, and designing so one failure doesn't cascade into a second.
 
-## Three independent recovery layers
+## Three Independent Recovery Layers
 
 ![Something lost branches three ways depending on what kind of loss it is: an entire host rebuilds through the same Packer, Terraform, and Ansible pipeline that built it originally; VM/LXC-level data recovers through Proxmox Backup Server's full-host snapshots; app data on the main Docker host recovers through Kopia's volume-level, per-stack isolated repository](../assets/disaster-recovery-layers.svg)
 
-## Why three layers instead of one comprehensive backup system
+## Why Three Layers Instead of One Comprehensive Backup System
 
 A single backup tool covering everything was the simpler design, and it was rejected because "a host" and "a database's data" are fundamentally different kinds of loss with different recovery needs. Rebuilding a lost host from [`docs/iac-pipeline.md`](iac-pipeline.md)'s pipeline is fast and doesn't need a backup at all — the *code* is the backup. But that same pipeline can't recover yesterday's database contents, which is what Layer 2 and 3 exist for. Trying to make one tool do both jobs well tends to produce a tool that's mediocre at each — full-host imaging is slow and coarse-grained for restoring one app's data, and application-level backup can't reconstruct a host's OS and configuration from scratch efficiently. Three narrow, purpose-fit layers beat one general-purpose one here.
 
 | Layer | Covers | Explicitly does not cover |
 |---|---|---|
-| Infrastructure as Code | A host lost entirely — rebuilt from the same pipeline that built it ([`docs/iac-pipeline.md`](iac-pipeline.md)), against its existing tag-driven config | Secrets — recovered separately via Vault, see [`docs/secrets-and-pki.md`](secrets-and-pki.md), not part of this layer |
+| Infrastructure as Code | A host lost entirely, VM or LXC alike — rebuilt from the same pipeline that built it ([`docs/iac-pipeline.md`](iac-pipeline.md)), against its existing tag-driven config | Secrets — recovered separately via Vault, see [`docs/secrets-and-pki.md`](secrets-and-pki.md), not part of this layer |
 | Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data |
-| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host |
+| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering all 41 stacks running there | Anything outside that one host |
 
 This is the real Kopia repository, not a description of it — a dedicated set of directories on its own separate disk, confirmed live:
 
 ```
 $ ls /mnt/samsungssd
-kopia-cache  kopia-config  kopia-repo  kopia-tls
+kopia-cache  kopia-config  kopia-repo  kopia-tls  recovery
 $ df -h /mnt/samsungssd
 Filesystem      Size  Used Avail Use% Mounted on
-/dev/nvme0n1p1  234G   65G  167G  28% /mnt/samsungssd
+/dev/nvme0n1p1  234G   75G  157G  33% /mnt/samsungssd
 ```
 
 Layer 1's "rebuilt against its existing tag-driven config" claim is a real, checkable mechanism too — every role in the pipeline declares which of the three rebuild stages it belongs to:
 
 ```
-$ head -12 roles_metadata.yml
+$ head -14 roles_metadata.yml
 roles_metadata:
+  # Bootstrap roles
   base_users:
     initialization: true
     provisioning: false

docs/dns-naming.md

@@ -1,3 +1,7 @@
+<!-- verified: 2026-09-18
+     trilium : nAPJq9A1iuX8, G6NWXtdWX9ap, 0mu75aAXGH8X, fscYadnUtmD5
+-->
+
 # 🌐 DNS Naming Model
 
 > [!NOTE]
@@ -11,14 +15,14 @@
 | **Depends on** | Nothing upstream for internal resolution — it's the root of trust for internal naming |
 | **Depended on by** | Every system in this repo that resolves a hostname — which is effectively all of them |
 
-## Why self-hosted DNS at all
+## Why Self-hosted DNS At All
 
 A homelab this size could get away with a shared hosts file or a single flat zone. Two things specifically ruled that out:
 
 1. **The two-zone trust model needs a real answer to "which front door am I behind."** Splitting traffic into a mutual-TLS-enforced browser zone and a lighter trusted-IP internal zone (see [`docs/firewall-and-network.md`](firewall-and-network.md)) only works if something authoritative decides, per-hostname, which zone a request lands in. A hosts file can't do that at scale across a dozen-plus hosts; a real DNS server with two real zones can.
 2. **Every other system needs one source of truth for internal names.** The secrets platform issues a certificate to the DNS server's own admin interface (see below); the orchestration engine resolves every fleet host by name before it can SSH anywhere; the observability pipeline ships DNS query logs as their own dedicated stream. Hand-rolling internal name resolution per-app would mean N inconsistent answers instead of one.
 
-## The two zones
+## The Two Zones
 
 ![Anything on the LAN queries Technitium, which either answers from one of its two authoritative internal zones — browser-facing, routing to the user-facing reverse proxy with mTLS+SSO, or app-to-app, routing to the internal reverse proxy with a trusted-IP bypass — or forwards unresolved queries upstream to the edge firewall and a public resolver](../assets/dns-naming-zones.svg)
 
@@ -46,12 +50,12 @@ Same result for `loki` and `wazuh-indexer` (caddy-zone only), and the reverse fo
 | App-to-app | `NXDOMAIN` |
 | Browser-facing | `NXDOMAIN` |
 
-## DNS as an enforced boundary, not just a convenience
+## DNS As An Enforced Boundary, Not Just a Convenience
 
 > [!IMPORTANT]
 > Every host on the LAN has its resolver pointed at this one internal DNS server — and that's enforced, not just configured. Direct outbound DNS queries to anywhere else (a public resolver, another host acting as its own resolver) are blocked at the firewall level, covered in [`docs/firewall-and-network.md`](firewall-and-network.md). This closes a real bypass: without it, any host could sidestep the internal zones entirely by just asking a public resolver instead, which would quietly undermine the whole two-zone trust model above. DNS enforcement is what makes the rest of this architecture actually load-bearing instead of advisory.
 
-## Two things live-checking found the documentation had wrong
+## Two Things Live-checking Found The Documentation Had Wrong
 
 Trusting the KB without re-checking it against the running server would have kept both of these wrong indefinitely — so both got checked directly against the real config and the real zone data, not taken on the KB's word:
 
@@ -68,7 +72,7 @@ $ strings /etc/dns/dns.config | grep -E '^\d+\.\d+\.\d+\.\d+$'
 
 Neither is a security problem — the reverse zone being broader than described doesn't leak anything, and the missing forwarder just means one less redundant upstream. What they have in common is more useful than either fact alone: documentation describing what a system does isn't the same as verifying it, and the gap between the two doesn't announce itself until something actually goes and checks.
 
-## Where DNS shows up in every other system
+## Where DNS Shows Up In Every Other System
 
 DNS isn't a standalone concern in this fleet — it's a dependency almost everything else sits on top of:
 

docs/firewall-and-network.md

@@ -1,13 +1,18 @@
+<!-- verified: 2026-09-06
+     iac@49d9773 : ansible/roles/bootstrap_roles/firewall_ssh, ansible/roles/secure_vault/firewall_rich_rules_per_vm, ansible/roles/cis_hardening/password_policy
+     trilium : 0mu75aAXGH8X, o3DDWo1v54vI, oEbKGleDKuvb, EpssgVXxqIbE, 0R8GeBmH7boV, TIbCwYGitboy, fscYadnUtmD5, yUJqswizuET2, QMTh5G7ldlLV, jBPOx07B1Y2Z, Cl4TWBZ7ytYr, 6cJurv76PNDR, kwbbLjeI6ody, MSVD5o5HiJ69, kA00FQiQWVem
+-->
+
 # 🧱 Firewall & Network Segmentation
 
 > [!NOTE]
 > pfSense at the edge, firewalld on every individual host — two layers, two different jobs, and a network model with no blanket "anything on the internal network can talk to anything else on it" rule anywhere. Every real flow is explicit; everything else is logged and dropped. This is the enforcement layer underneath the trust boundaries described in [`docs/architecture-overview.md`](architecture-overview.md).
 
-## Why two layers, not one
+## Why Two Layers, Not One
 
 A single firewall layer was rejected early: the edge firewall (pfSense) is the fleet's only genuinely public-facing component, so it carries a different threat model — internet-scale scanning and abuse — than internal host-to-host traffic ever will. Collapsing both into one ruleset would mean either the internal segmentation rules get diluted by public-facing noise, or the public-facing rules get relaxed to match internal assumptions. Keeping them separate means each layer's rules can be as strict as that layer's actual threat model demands, independently.
 
-## The one rule that matters most
+## The One Rule That Matters Most
 
 This wasn't always this tight. The LAN interface used to run pfSense's own standard default — a blanket allow between hosts — before it was deliberately narrowed to LAN→WAN/internet only, closing off unrestricted lateral movement that had nothing to do with any application actually needing it. What replaced it is the model below.
 
@@ -24,7 +29,7 @@ That narrowing and the catch-all it produced aren't just claims — here's the a
 
 DNS enforcement specifically doesn't rely on just one mechanism. Every static-lease device gets the internal DNS server pushed via a per-lease DHCP override — every current static lease carries it — but that's a soft guarantee: a brand-new device on a plain dynamic lease would fall back to the router's own upstream resolvers instead, at least until it gets a static reservation of its own. The firewall closes that gap independently, at the network level: an explicit rule blocks any LAN client from reaching external DNS outright, full stop. So a device that somehow ended up pointed at the wrong resolver still can't actually use it — DNS bypass isn't just discouraged by configuration, it's structurally blocked by a second, unrelated layer that doesn't depend on the first one being right.
 
-## Named groups, not raw addresses
+## Named Groups, Not Raw Addresses
 
 pfSense aliases (named groups) stand in for hardcoded IP lists throughout — adding a host to a group is a single membership edit, not a hunt through every rule that needs to reach it. Groups nest: a "whole managed fleet" alias is composed of several narrower per-role aliases, so a rule against the broad group automatically covers new members of any narrower one without the rule itself ever changing.
 
@@ -34,7 +39,7 @@ The same discipline applies to the other axis of every rule, and it's still acti
 
 ![pfSense firewall port aliases, one per service/flow](../assets/pfsense-aliases-ports.png)
 
-## Per-host firewalls: minimal, not blanket
+## Per-host Firewalls: minimal, not blanket
 
 Each host runs firewalld locally, opening only the specific ports it actually needs — not a blanket "allow everything, block nothing" default. Vault's host goes further than the rest: instead of one open port for its most sensitive service, individual source IPs get their own explicit rich-rule — five named clients, confirmed directly against the live ruleset, and zero blanket ports open at all — so a compromised unrelated service on the same network still can't reach it even if it somehow got past pfSense. This is the actual live output, not a paraphrase:
 
@@ -64,7 +69,26 @@ rule family="ipv4" source address="[automation host]" port port="[this host's SS
 
 ![Two enforcement layers per packet — pfSense always applies, firewalld only catches native services, Docker-published ports skip straight past it](../assets/firewall-and-network-docker-bypass.svg)
 
-## Edge firewall: three jobs, one device
+A fresh, real example of exactly this — the mesh overlay's own host in [`docs/netbird-zero-trust.md`](netbird-zero-trust.md), firewalld genuinely locked down to just its explicit rules, and three container-published ports sitting completely outside them:
+
+```
+$ firewall-cmd --zone=public --list-ports
+[no output — zero blanket-opened ports]
+
+$ firewall-cmd --zone=public --list-rich-rules
+rule family="ipv4" source address="[admin workstation]" port port="[this host's SSH port]" protocol="tcp" accept
+rule family="ipv4" source address="[automation host]" port port="[this host's SSH port]" protocol="tcp" accept
+rule family="ipv4" destination address="[vault host]" port port="8200" protocol="tcp" accept
+
+$ docker ps --format "{{.Names}}: {{.Ports}}"
+netbird-proxy: 0.0.0.0:33000->33000/tcp, [::]:33000->33000/tcp
+netbird-server: 0.0.0.0:34780->34780/udp, [::]:34780->34780/udp
+hawser: 0.0.0.0:2376->2376/tcp, [::]:2376->2376/tcp
+```
+
+Three ports, zero matching firewalld rules, all three genuinely reachable LAN-wide via Docker's own NAT — not a hypothetical, the exact gap above on a host provisioned this same week.
+
+## Edge Firewall: three jobs, one device
 
 One device, three distinct responsibilities — kept together deliberately, not split into separate boxes, since none of the three is heavy enough on its own to justify another VM to patch and monitor:
 
@@ -75,7 +99,7 @@ One device, three distinct responsibilities — kept together deliberately, not
 > [!IMPORTANT]
 > **Failure impact, thought through in advance, not discovered during an actual outage**: if pfSense goes down, public access and outbound internet both go with it — but every internal service that doesn't depend on WAN reachability keeps running normally, including the orchestration flows in [`docs/orchestration.md`](orchestration.md), which only ever reach internal hosts. The blast radius is deliberately bounded to "the edge," not "the fleet" — a design property worth stating explicitly rather than leaving implicit, since it's exactly the kind of thing that should be known *before* an incident, not discovered during one. One thing does quietly degrade even though nothing breaks outright, and it's worth being honest about rather than rounding up to "everything's fine": Vault Agent's certificate renewal and Technitium's own PKI issuance both route through pfSense-mediated LAN paths, so an extended outage would eventually stall renewals fleet-wide — not an immediate failure, but a real one if the outage runs long enough to matter.
 
-## Defense-in-depth below the network layer: PAM
+## Defense-in-depth Below The Network Layer: PAM
 
 Firewalls aren't the only layer that matters. On this fleet's Debian-family hosts, the login stack itself got hardened after a real, *recurring* incident — not a one-off. `pam_unix.so`'s default auth line uses a numbered jump ("on success, skip forward N modules") to reach approval, which is fragile by construction: any future package that inserts its own line anywhere in that stack silently shifts what the number actually lands on. This broke real, correct-password logins **twice**: once during initial rollout, fixed by bumping the number up by one — and again five days later, on a different host, when an unrelated package pulled in its own PAM line and broke that same numbered jump a second time, landing on a deny instead of an allow. A correct password got treated as a failure. The permanent fix replaced the number with a keyword that terminates the stack on success immediately, regardless of how many lines ever get added after it later — closing the entire class of bug, not just the one instance that got caught. Rocky/RHEL hosts never carried this risk in the first place; they manage their PAM stack through `authselect`, a declarative system not exposed to the same ordering hazard.
 
@@ -84,4 +108,4 @@ Firewalls aren't the only layer that matters. On this fleet's Debian-family host
 Proxmox has its own built-in firewall capability, left with an empty ruleset on purpose. pfSense and firewalld already cover the two layers that matter — public/internal segmentation and per-host minimal exposure — and a third overlapping layer at the hypervisor level would add complexity (a third ruleset to audit, a third place a rule could silently be wrong) without meaningfully improving posture. Recognizing when *not* to add a redundant control is as much a design decision as adding one, and it's the same reasoning that kept the secrets model in [`docs/secrets-and-pki.md`](secrets-and-pki.md) to one trust root instead of two.
 
 ---
-Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/architecture-overview.md`](architecture-overview.md), [`docs/security-monitoring.md`](security-monitoring.md) · Stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/), [`stacks/platform/docker-socket-proxy/`](../stacks/platform/docker-socket-proxy/)
+Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/architecture-overview.md`](architecture-overview.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/netbird-zero-trust.md`](netbird-zero-trust.md) · Stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/), [`stacks/platform/docker-socket-proxy/`](../stacks/platform/docker-socket-proxy/)

docs/homelab-inventory.md

@@ -0,0 +1,117 @@
+<!-- verified: 2026-09-07
+     ansible@9a4ba6d : roles/homelab_inventory_hardware, roles/homelab_inventory_vms, roles/homelab_inventory_services, playbooks/homelab_inventory_hardware.yml, playbooks/homelab_inventory_vms.yml, playbooks/homelab_inventory_services.yml
+     rocky-docker-lxc@d225776 : caddy/conf/Caddyfile, dashy/docker-compose.yaml
+     trilium : O0d4fqw0mr05
+-->
+
+# 📇 Homelab Inventory
+
+> [!NOTE]
+> A live reference for every physical host, VM/LXC, and running service in the fleet — **re-derived from the machines themselves once a week**, not hand-maintained. The problem it solves is inventory rot: a wiki table of "what hardware do I have, what's running where" is accurate the day it's written and drifting by the next change. This one is structurally unable to drift, because nothing attests it — a scheduled job re-reads the real systems and overwrites the store.
+
+The knowledge base (Trilium) stays the hand-authored layer — design rationale, gotchas, why things are the way they are. This is the **machine-derived layer** underneath it: three linked tables in a self-hosted NocoDB instance, refreshed by a three-tier Ansible collection on a weekly schedule, surfaced on a dedicated dashboard page.
+
+![The collection pipeline: a weekly Kestra trigger runs three Ansible tiers in sequence — hardware off the hypervisors and firewall, VMs/LXCs from the Proxmox APIs, listening services off every running guest — each writing its own NocoDB table, chained by NocoDB link fields, and read live per request by three Node-RED flows behind one Dashy page](../assets/homelab-inventory-pipeline.svg)
+
+## Three Collection Tiers, Three Different Access Models
+
+The three tiers don't share a collection method, because the three things being inventoried don't live in the same place or answer to the same protocol.
+
+| Tier | Where it runs | How it collects | Writes | Row write mode |
+|---|---|---|---|---|
+| **Hardware** | hypervisor + firewall hosts, as root | `dmidecode` / `ip -j link` / `/proc` on the Linux hosts; `kenv \| grep -i smbios` + `ifconfig -a` over the `raw` module on the firewall (no Python on that box) | Hardware Inventory | **Merge-only** — only collected fields are sent |
+| **VMs / LXCs** | control node only — no SSH anywhere | each hypervisor node's own Proxmox API (`/qemu` + `/lxc`); guest IP from the LXC `net0` config or the VM's QEMU guest agent | VM/LXC Inventory, linked → Hardware | Full field-set replace every run |
+| **Services** | every running guest, as root | `ss -tlnp` for listening sockets, plus `docker ps` on the Docker hosts | Services Inventory, linked → VM/LXC | Full field-set replace every run |
+
+A few of those choices are worth pulling out, because the obvious approach was rejected in each case.
+
+### Why `ss -tlnp` and not `systemctl`
+
+"What services does this host run" sounds like a `systemctl list-units` question. It isn't — that answer is 40–80 units of `dbus`, `journald`, udev timers and mount units per host, none of which is a *service the host provides*. Listening sockets are a far closer match to the real question, so that's what the tier collects:
+
+```
+$ sed -n '2,6p' roles/homelab_inventory_services/tasks/main.yml
+# Deliberately NOT collecting `systemctl list-units --state=running` here -
+# that dumps 40-80 units per host (dbus, journald, udev timers, ...) that
+# have nothing to do with "what service is this host providing." `ss -tlnp`
+# only shows things with an actual open listening socket, which is a much
+# closer match to what "service" means for a reference doc - see card [77].
+```
+
+### Merge-only vs full replace: protecting the one field a machine can't collect
+
+Tiers 2 and 3 overwrite every column on a row every run — every value there is meant to always reflect current state, and there's nothing to protect. Tier 1 is different, and the difference exists for exactly one reason: the firewall appliance is a QNAP NAS whose BIOS reports the literal string `Default string` for manufacturer, model, and serial, even with full root access. Those three fields are entered by hand in NocoDB once (QNAP / TS-251A / left blank) — and the collector has to be structurally incapable of clobbering them on the next run.
+
+The guarantee isn't "write a blank" or "skip if unchanged" — it's that a field the collector has no data for is **never included in the API payload at all**:
+
+```
+$ grep -A3 'def normalize' roles/homelab_inventory_hardware/files/push_hardware_to_nocodb.py
+def normalize(record):
+    """Build the NocoDB field dict - only include keys we actually have data
+    for. This is the merge-only guarantee: a key not added here is a key
+    never sent in the API call, so it can never overwrite anything."""
+```
+
+The firewall's collection path deliberately never sets `Manufacturer`/`Model`/`Serial` keys; the Linux path sets them from `dmidecode`. Same table, two code paths, one protected by omission. Here's the result — the CPU and RAM on the QNAP row *are* machine-collected (via `kenv`), only the three identity fields are hand-held:
+
+![The NocoDB Hardware Inventory table: three physical hosts. The two Intel NUCs carry machine-read manufacturer/model/serial/CPU/RAM; the QNAP row has hand-entered manufacturer and model, a blank serial, but machine-collected CPU and RAM, plus a link out to its guests](../assets/homelab-inventory-nocodb-hardware.png)
+
+### Tier 2 is pure API-to-API
+
+The VM/LXC tier touches no host over SSH — it runs entirely on the control node and pulls from each hypervisor node's own Proxmox API with one dedicated read-only `PVEAuditor` token. The two nodes aren't clustered, so each API only ever returns its own guests; the tier queries both and merges. Guest IPs come from two different places depending on type — an LXC's is static in its `net0` config, a VM's has to come from the QEMU guest agent — and a stopped VM or one with no agent simply gets no IP recorded, not an error. Build templates are skipped.
+
+## The Linked Store
+
+Three NocoDB tables, related **Hardware ← VM/LXC ← Services**: each guest row links to its host's hardware row, each service row to its guest's VM/LXC row, set through NocoDB's Links API after each upsert. Service rows are named `<host>: <service>` — native names collide across hosts (every host has an `sshd`), and the prefix is what keeps two rows distinct and lets the viewer derive a service's host straight from its name.
+
+Live counts, straight off the API — matching the screenshots below:
+
+```
+$ # GET /api/v2/tables/<id>/records?limit=1  →  .pageInfo.totalRows
+Hardware   : 3 rows
+VM/LXC     : 12 rows
+Services   : 111 rows
+```
+
+![The NocoDB VM/LXC Inventory table: twelve guest rows, VM and LXC types mixed, each with VMID, status, RAM/CPU/disk, a link back to its Proxmox host and a link forward to its services](../assets/homelab-inventory-nocodb-vms.png)
+
+![The NocoDB Services Inventory table: 111 rows. Socket-derived rows show status "listening" with no image; container-derived rows show status "running" with an image/version — the `ss` and `docker ps` sources stay distinguishable in the data. Every row links to exactly one VM/LXC](../assets/homelab-inventory-nocodb-services.png)
+
+The push scripts use nothing but the Python standard library — they run under the control node's system `python3`, invoked straight from Ansible's `command` module, not a virtualenv with `requests` available:
+
+```
+$ grep -hE '^import |^from ' roles/homelab_inventory_*/files/*.py | sort -u
+from datetime import datetime, timezone
+import glob
+import json
+import os
+import re
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+```
+
+## The Viewer
+
+A dedicated Dashy page, **Homelab Inventory** — three collapsible sections (Hardware, VMs + LXCs, Services), each backed by its own Node-RED flow on the automation host. Each flow is a single `http-in → function → http-response` that reads its NocoDB table **live on every request with no caching** and renders a searchable click-to-expand list. Past ~100 rows a flat scroll is unusable, so the Services section carries a live text filter. Each flow gets its own unauthenticated path exception in the reverse-proxy config, the same pattern every Dashy widget on this fleet uses.
+
+![The Dashy Homelab Inventory page: Hardware expanded to three hosts (two NUC10i3FNH, one QNAP TS-251A), VMs/LXCs collapsed, Services expanded showing 111 rows with a "filter by host or service name" box and NATIVE / CONTAINER badges](../assets/homelab-inventory-dashy-page.png)
+
+## Scheduled, Not Triggered
+
+A single Kestra flow, `homelab-inventory-sync`, cron `0 12 * * 4` — Thursday noon. There's no webhook: nothing about live hardware or a running-process list is an event the orchestrator would otherwise see, so a fixed weekly cadence is the whole trigger.
+
+```
+  - id: schedule
+    type: io.kestra.plugin.core.trigger.Schedule
+    cron: "0 12 * * 4"
+```
+
+One SSH task hops to the automation host and runs the three playbooks in sequence through the pinned virtualenv, capturing a per-tier exit code. A non-zero tier sends a critical Slack alert through the shared [`notifications`](notifications.md) pipeline; all-clear sends an info line. The dedicated-key SSH hop and the check→notify shape are the same model every flow in [`orchestration.md`](orchestration.md) uses.
+
+> [!IMPORTANT]
+> **This layer never writes to the knowledge base, and the knowledge base never sources from it.** They're deliberately separate: the KB is hand-authored design intent, this is a machine-derived snapshot of current reality, and wiring either one to feed the other would collapse that distinction. A separate mechanism does push a handful of per-host CPU/RAM figures into the KB for at-a-glance context, but it's a different job with its own source and its own destination — it doesn't read or write anything here.
+
+---
+Related: [`docs/orchestration.md`](orchestration.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md), [`docs/cron-scheduler.md`](cron-scheduler.md), [`docs/notifications.md`](notifications.md) · Stacks: [`stacks/platform/nocodb/`](../stacks/platform/nocodb/), [`stacks/platform/node-red/`](../stacks/platform/node-red/), [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/kestra/`](../stacks/platform/kestra/)

docs/iac-pipeline.md

@@ -1,3 +1,8 @@
+<!-- verified: 2026-09-19
+     iac@ae315de : packer/rocky10-proxmox-template, terraform, ansible/roles_metadata.yml, ansible/playbooks/provisioning.yml, ansible/host_vars/technitium/host_tags.yml, ansible/roles/cis_hardening/password_policy
+     trilium : c5JADrRVvjIn, f1uTcvXvq0u3, 7anU4VULpp7n, sovEBDmn16hh, sqVGFejKPOFn, kA00FQiQWVem, MSVD5o5HiJ69, NwwrSew7mP2z, QgviHP8r14t2
+-->
+
 # 🏗️ Infrastructure as Code
 
 > [!NOTE]
@@ -7,7 +12,7 @@
 
 ![Packer builds the base image, Terraform provisions the VM and generates host_tags.yml, which Ansible consumes to configure and harden the host, re-applying its compliance stage on every run](../assets/iac-pipeline-tools-flow.svg)
 
-## Why three separate tools instead of one
+## Why Three Separate Tools Instead Of One
 
 A single tool covering image-build-through-configuration was a real option — several products exist specifically for that — and it was rejected because the three jobs have genuinely different rhythms: an image gets rebuilt rarely (a base OS refresh), a VM gets provisioned once, and configuration gets re-applied continuously (the compliance stage below runs on every pass, not just at creation). Collapsing all three into one tool's model would mean forcing the rarely-changing and the continuously-changing onto the same execution model. Keeping them separate, connected by one deliberate handoff (the tag file below) rather than a tightly-coupled integration, means each tool does the one job it's actually good at.
 
@@ -48,7 +53,7 @@ Terraform manages both VMs and LXCs now — two separate resource types, one per
 The LXC path is genuinely newer and mirrors the VM flow step-for-step, but has to close a gap the VM path never has: the stock container template has no `sshd` and no fleet users at all, since it never went through Packer. A `pct exec` bootstrap step runs first — installing `sshd` and creating both fleet identities with their keys, the same baseline Packer bakes into the VM image at build time instead of at provision time — before the same wait-for-SSH and tag-file-render steps every VM already gets. This is the real resource definition, not a paraphrase:
 
 ```
-$ head -20 containers.tf
+$ sed -n '20,39p' containers.tf
 resource "proxmox_virtual_environment_container" "ct" {
   provider = proxmox-bpg
   for_each = var.containers
@@ -59,6 +64,8 @@ resource "proxmox_virtual_environment_container" "ct" {
   start_on_boot = each.value.onboot
   started       = true
 
+  # PVE-native tags — cosmetic/searchable in the PVE UI, mirrors main.tf's
+  # `tags = join(",", each.value.tags)` on the VM resource.
   tags = each.value.tags
 
   cpu {
@@ -77,10 +84,12 @@ This is the actual template that renders into every host's tag file — the lite
 
 ```
 $ cat host_tags.yml.tpl
+# Host metadata from Terraform provisioning
 vmid: ${vmid}
 vm_name: ${name}
 target_node: ${target_node}
 template: ${template}
+inventory_group: ${inventory_group}
 cores: ${cores}
 sockets: ${sockets}
 memory: ${memory}
@@ -97,7 +106,12 @@ vm_tags:
 %{ endfor ~}
 ```
 
-Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes. And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, `ip`/`macaddr` redacted:
+Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes.
+
+> [!NOTE]
+> **A field that exists in the template but not yet in any VM's real file.** `inventory_group` was added to this template alongside the 2026-08 LXC rollout — new LXCs render it correctly (`rocky_lxc`, confirmed live on netbird), but no existing VM has been re-applied since, so `inventory_group: rocky_vm` isn't actually in any of their rendered files yet, technitium's included below. A real, current gap, not a hypothetical one — worth naming rather than quietly patching the example to look more finished than the fleet actually is.
+
+And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, `ip`/`macaddr` redacted:
 
 ```
 $ cat host_vars/technitium/host_tags.yml
@@ -137,9 +151,10 @@ vm_tags:
   - observability_agent
   - mqtt_client
   - wazuh_agent_tuning
+  - netbird_agent
 ```
 
-21 tags on this one host alone, spanning both the initialization roles (`packages`, `firewall_ssh`, …) and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads.
+22 tags on this one host alone, spanning the initialization roles (`packages`, `firewall_ssh`, …), the provisioning-stage `netbird_agent` role, and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads.
 
 ## Ansible: tag-driven, re-applied continuously
 
@@ -154,15 +169,15 @@ d = yaml.safe_load(open('roles_metadata.yml'))['roles_metadata']
 print('provisioning:', [k for k,v in d.items() if v.get('provisioning') is True])
 print('compliance:', sorted(k for k,v in d.items() if v.get('compliance') is True))
 "
-provisioning: ['clamav']
+provisioning: ['clamav', 'netbird_agent']
-compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom']
+compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_ca_client', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom']
 ```
 
-`clamav` is the only provisioning-stage role, and only one host carries it in its `vm_tags`. An under-used pipeline stage isn't the same as an unused one — it exists on purpose for exactly this shape of role, not as leftover scaffolding.
+Two roles are tagged for the provisioning stage: `clamav` (rocky-docker only) and `netbird_agent` (enrols a host into the NetBird mesh overlay covered in [`docs/netbird-zero-trust.md`](netbird-zero-trust.md) — monitoring was the first host onboarded through it). Neither fits "once at init" or "every compliance run" cleanly, which is exactly the shape this stage exists for — an under-used pipeline stage isn't the same as an unused one.
 
-Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 15 total compliance-stage roles above — the other 6 handle observability, notifications, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
+Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 16 total compliance-stage roles above — the other 7 handle observability, notifications, SSH certificate refresh, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
 
-## The honest exception: not everything fits the clean model
+## The Honest Exception: not everything fits the clean model
 
 > [!WARNING]
 > Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into LXC containers outside Terraform's management — a resource-type mismatch that Terraform's VM-only tooling genuinely can't represent (LXCs aren't `proxmox_vm_qemu` resources). Rather than deleting their old Terraform entries when this happened, they were commented out, preserving the historical record of how these hosts actually came to exist rather than presenting a cleaner story than what's true. Ansible still configures both today through the same tag-driven mechanism as every VM — only the initial provisioning step is now manual instead of automated. This is the kind of real-world messiness that a from-scratch design document never mentions but that every infrastructure actually accumulates over time — worth showing, not hiding.

docs/kodi-music-assistant-bridge.md

@@ -0,0 +1,134 @@
+<!-- verified: 2026-10-16
+     rocky-docker-lxc@1ede96f : kodi/bridge.py, kodi/docker-compose.yaml, kodi/.env, music-assistant/docker-compose.yaml
+     trilium : Y2VzE58wL8w1
+-->
+
+# 🎧 Kodi <-> Music Assistant Smart Queue Bridge
+
+> [!NOTE]
+> Kodi stays the only music player, UI, and ratings authority on this fleet — ratings and library metadata live in a shared MariaDB, untouched. Music Assistant runs invisibly behind it as a pure similarity engine, never opened day-to-day, never a player, never writing anything back to the library. A small daemon watches Kodi's live playback queue and auto-refills it with musically similar tracks when it runs low, using MA purely as a recommendation backend.
+
+## Why This Shape, Not The Obvious One
+
+Two more obvious designs were deliberately rejected. The first: let MA write ratings and tags back into the audio files directly, so it could double as a second library manager. MA doesn't support this upstream, and even if it did, it would create two competing sources of truth for the same library — exactly the failure mode this design avoids. The second: make Kodi a playback target *for* Music Assistant, so MA could drive playback directly. The one community project for this integration (`MAserver-Kodi-Provider`) is archived and abandoned — betting a working setup on dead third-party code wasn't worth it.
+
+What's left is the actual design: Kodi remains the single source of truth for everything a listener touches (library, ratings, playback), and MA is reduced to a disposable, stateless recommendation service — it could be deleted and rebuilt from an empty config tomorrow without losing a single rating or play count.
+
+## How A Refill Cycle Actually Works
+
+`kodi-ma-bridge` is a small asyncio daemon, deployed as part of the same `kodi` stack as Kodi's MariaDB (not a separate stack — the two were merged once the bridge was built, since the bridge only exists to serve Kodi):
+
+```
+$ docker ps --filter name=kodi --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"
+NAMES            STATUS                  IMAGE
+kodi-ma-bridge   Up 18 minutes           python:3.13-slim
+kodi-mariadb     Up 38 hours (healthy)   <fleet registry>/mariadb:latest
+```
+
+Every 60 seconds (`POLL_INTERVAL_SECONDS=60`), it checks each configured Kodi instance:
+
+```
+$ docker exec kodi-ma-bridge env | grep -E "POLL_INTERVAL|REFILL|MA_BASE_URL|KODI_INSTANCES"
+KODI_INSTANCES=[personal-device-1]:<host>:8115,[personal-device-2]:<host>:8115
+MA_BASE_URL=http://music-assistant:8095
+POLL_INTERVAL_SECONDS=60
+REFILL_COUNT=10
+REFILL_THRESHOLD=5
+```
+
+1. Ask Kodi's JSON-RPC API (`Player.GetActivePlayers`) whether anything's playing. If the instance is unreachable or idle, skip silently — these are personal desktops, not always-on servers, so "nothing there right now" is normal, not an error condition.
+2. If something's playing, check how many tracks remain after the current position (`Playlist.GetItems` + `Player.GetProperties`). Above `REFILL_THRESHOLD` (5 remaining), stop here — nothing to do yet.
+3. Below the threshold: read the current track's artist/title and resolve it to an MA library item via `music/search`.
+4. Ask MA's `music/tracks/similar_tracks` for a large candidate pool, filter out anything already queued or recently added, translate MA's library-relative path to Kodi's SMB path, and push up to `REFILL_COUNT` (10) new tracks straight into the live playlist via `Playlist.Add`.
+
+That queue-length gate is the one real branch in the whole design — everything else is a straight pipeline — so it's worth seeing as a diagram rather than another paragraph of prose:
+
+![kodi-ma-bridge refill cycle: poll each Kodi instance, check remaining queue length, and only resolve+inject new tracks when the queue actually drops below threshold](../assets/kodi-music-assistant-bridge-refill-cycle.svg)
+
+## Three Real Issues, Found By Actually Testing It
+
+The bridge sat deployed and silently non-functional for weeks before this. All three of the following were found by doing the one thing that actually proves a system works: playing real music on a real machine and watching what happened, not just exercising the API calls in isolation.
+
+**1 — the daemon never once successfully resolved a track.** Kodi's own JSON-RPC API wraps every response in a `result` key, so the bridge's generic API-calling helper was written to unwrap it the same way for *every* call, including the ones to MA. MA's REST API doesn't use that envelope at all — it returns the payload flat, at the top level:
+
+```
+$ curl -s -X POST http://music-assistant:8095/api -H "Content-Type: application/json" \
+  -d '{"message_id":"t","command":"music/search","args":{"search_query":"Oasis - Champagne Supernova","media_types":["track"],"limit":1,"library_only":true}}' \
+  | python3 -c "import json,sys; d=json.load(sys.stdin); print('top-level keys:', list(d.keys())); print('has result key:', 'result' in d)"
+top-level keys: ['artists', 'albums', 'genres', 'tracks', 'playlists', 'radio', 'audiobooks', 'podcasts', 'sound_effects']
+has result key: False
+```
+
+`data.get("result")` against that shape silently returns `None` every time — no exception, no log line, just a daemon that looked alive and never did anything. Confirmed live against a real Kodi playback session before touching a line of code; fixed by returning MA's response body directly instead of unwrapping a key that was never there.
+
+**2 — even after the fix, MA only ever offered one real match for a mainstream track.** The bridge asked MA for a small candidate pool (15 tracks) and assumed that was the full ranked result. It isn't — MA runs its own internal similarity-distance filter over whatever raw pool size you request, so most of a small batch gets discarded before it ever reaches the caller:
+
+```
+limit=10  -> 1 match
+limit=15  -> 1 match   (the bridge's original request size)
+limit=20  -> 1 match
+limit=30  -> 2 matches
+limit=50  -> 5 matches
+limit=100 -> 6 matches
+limit=150 -> 8 matches
+limit=200 -> 10 matches
+```
+
+The real candidate pool for that seed track was there the whole time — a solid run of mid-90s Britpop and alt-rock (The Verve, The Cranberries, Ocean Colour Scene, Beck, Manic Street Preachers) — it just never got a large enough raw pool to survive MA's own filtering. Fixed by requesting 200 candidates up front; the bridge's own `REFILL_COUNT` cap still bounds how many actually get queued.
+
+**3 — a real blind-spot window between the queue running dry and the bridge noticing.** With both of the above fixed and the daemon genuinely working, the original 4-minute poll interval turned out to be too coarse on its own: during one active listening session the live queue hit 0 remaining tracks twice before the next scheduled poll caught it and refilled —
+
+```
+[personal-device] queue low (0 left), added 10 tracks seeded from 'Primal Scream - Shine Like Stars'
+[personal-device] queue low (0 left), added 10 tracks seeded from 'Kula Shaker - Govinda'
+```
+
+`REFILL_THRESHOLD=5` only guarantees a refill *attempt* starts with 5 tracks of runway left — it says nothing about how long that runway lasts in wall-clock time, and a 4-minute gap between checks was long enough, on short tracks, to burn through that runway before the next poll ever ran. Playback itself never actually stopped in practice, but the margin was thinner than the design assumed. Fixed by shortening the poll interval to 60 seconds rather than widening the threshold — each idle poll is a handful of cheap JSON-RPC calls, so checking 4x more often costs effectively nothing.
+
+## Proof It Actually Works
+
+Confirmed end-to-end against a live Kodi session, mid-album, queue genuinely running low:
+
+```
+[personal-device] queue low (2 left), added 8 tracks seeded from 'Oasis - Champagne Supernova'
+```
+
+And the live playlist, read directly from Kodi right after:
+
+```
+$ curl -s -u <user>:*** -X POST -H "Content-Type: application/json" \
+  -d '{"jsonrpc":"2.0","method":"Playlist.GetItems","params":{"playlistid":0,"properties":["title","artist"]},"id":1}' \
+  http://<host>:8115/jsonrpc
+...
+02. The Verve - Sonnet
+The Day We Caught the Train
+Dreams
+Loser
+You Stole the Sun From My Heart
+Movin' On Up
+Blue Orchid
+Buck Rogers
+Reptilia
+Use Somebody
+```
+
+A real, coherent 90s/00s indie-and-alt-rock run, auto-queued with zero manual input, directly behind the Oasis album that was already playing:
+
+![Kodi's playlist view, showing a long auto-queued run of similar indie/alt-rock tracks the bridge added on its own once the queue ran low](../assets/kodi-music-assistant-bridge-live-queue.png)
+
+## Similarity Engine
+
+Matching is powered by MA's `sonic_analysis` provider (on-device CLAP audio-embedding analysis — no cloud service, no internet dependency for recommendations) feeding the `sonic_similarity` plugin, confirmed enabled against the running library:
+
+```
+$ docker exec music-assistant python3 -c "..."
+sonic_analysis -> enabled: True
+sonic_similarity -> enabled: True
+filesystem_local -> enabled: True
+```
+
+The library syncs every 12 hours and diffs the file list, so a replaced file (an upgraded rip, say) is picked up as a new file and the stale entry pruned automatically — no manual re-indexing required.
+
+## Secrets
+
+Both the Kodi JSON-RPC password and the MA API token are Vault-sourced via the fleet's standard 3-file docker-secrets pattern (see [`docs/secret-distribution.md`](secret-distribution.md)) — nothing hardcoded in the compose file or the bridge script itself.

docs/netbird-zero-trust.md

@@ -1,9 +1,15 @@
+<!-- verified: 2026-09-19
+     netbird-lxc@ea8967e : netbird/conf/Caddyfile, netbird/docker-compose.yaml, netbird/config.yaml.ctmpl
+     iac@ae315de : ansible/roles/secure_vault/netbird_agent, terraform/containers.tf
+     trilium : qWLU4xqYLdJ3, oYuIjaczDrZA, nAPJq9A1iuX8, Cl4TWBZ7ytYr
+-->
+
 # 🕸️ Zero-Trust Mesh Overlay
 
 > [!NOTE]
 > A self-hosted **NetBird** WireGuard mesh gives every fleet host — and every device that needs to reach it — its own stable overlay identity, independent of where that device physically sits. Access is granted per-identity through mesh policy, not inferred from LAN position. The mesh is **deny-by-default**: nothing crosses it unless an explicit policy allows it, the same instinct behind every other boundary in [`docs/architecture-overview.md`](architecture-overview.md), applied to remote and roaming access specifically.
 
-## Why a mesh, not a classic VPN
+## Why A Mesh, Not a Classic VPN
 
 A traditional VPN concentrator was the obvious alternative, and it was rejected for the same reason a flat LAN got rejected everywhere else in this fleet: it's one more device sitting in a path, and reaching it means punching a hole to a single choke point rather than establishing an identity-scoped connection to exactly the host that's actually needed. A mesh instead gives every peer — fleet host or personal device — its own stable identity (`<host>.netbird.selfhosted`, a `100.91.0.0/16` overlay address) and lets WireGuard tunnels form directly, peer-to-peer, falling back to a relay only when direct connectivity isn't possible. Policy is written once, per identity or group, and travels with the peer regardless of which network it's physically on.
 
@@ -13,11 +19,7 @@ This is the actual shape of it — the control plane coordinates, but it's delib
 
 That's the actual difference from a VPN concentrator, drawn rather than just asserted: a concentrator puts itself in every data path by design; here the control plane's spokes only ever carry coordination traffic, and every thick line is two peers talking directly to each other.
 
-## Built on a capability that didn't exist yet
+## Control Plane
-
-The mesh's own control plane is the first host in this fleet ever provisioned through a genuinely new capability: Terraform gaining native LXC support (a second provider, `bpg/proxmox`, alongside the existing VM-only one — full story in [`docs/iac-pipeline.md`](iac-pipeline.md)). Every other LXC in this fleet either predates Terraform entirely or was hand-converted after the fact; this one went through the pipeline start to finish with no manual provisioning step anywhere. That's not a coincidence — a lightweight mesh control-plane container was exactly the kind of workload that made closing the LXC gap worth doing.
-
-## Control plane
 
 One `netbirdio/netbird-server` container runs Management, Signal, the embedded relay, and NetBird's own embedded IdP together — combined-server mode, not four separate services. A digest-pinned `netbirdio/dashboard` container serves the admin SPA. This is the real container list, not a claimed one:
 
@@ -31,7 +33,18 @@ netbird-dashboard   f3eb26c93ca9                                        Up 38 ho
 
 Configuration is Vault-Agent-rendered, never committed — secrets live in Vault under a dedicated AppRole and policy scoped to this one host, the same pattern as every other secret in [`docs/secrets-and-pki.md`](secrets-and-pki.md). The identity provider is NetBird's own embedded one, not Authentik — Authentik is deliberately LAN-only in this fleet and publishing it wasn't in scope, so there's no OIDC migration path here; the first-login owner account is the permanent identity for this system specifically.
 
-## The edge routing problem, and what actually broke
+Self-hosted rather than the vendor's own managed cloud offering, for the same reason every other security-critical system in this fleet is self-hosted rather than SaaS: every peer's identity, every access policy, every session record stays inside the same trust boundary Vault protects in [`docs/secrets-and-pki.md`](secrets-and-pki.md), instead of a third party holding the map of exactly which devices exist and what they're allowed to reach. For a mesh whose entire job is deciding who gets in, that map is itself sensitive — running the control plane anywhere else would mean trusting someone else's infrastructure with the fleet's own access model.
+
+**Addressing:**
+
+| Item | Value |
+|---|---|
+| Overlay v4 range | `100.91.0.0/16` |
+| Overlay v6 range | a ULA `/64` |
+| Peer identity | dashboard name = short hostname; mesh DNS label = `<host>.netbird.selfhosted` |
+| Peer WireGuard | port `51820`, kernel interface `wt0` |
+
+## The Edge Routing Problem, and What Actually Broke
 
 > [!WARNING]
 > **Learned the hard way during bring-up, not designed in from the start.** NetBird's Management and Signal services need native gRPC over HTTP/2-cleartext (h2c) end-to-end — any downgrade to HTTP/1.1 anywhere in the chain kills the bidirectional stream with an `unexpected EOF`, and the client agent just wedges with no useful error. The edge firewall's own reverse proxy can carry h2c to a single plain-HTTP backend, but it can't perform NetBird's own content-type/path split between the management API, the relay, and the dashboard SPA — so a small in-stack proxy does exactly that split, one hop further in. A second, unrelated thing broke the same week: the public DNS record for this service has to stay **DNS-only**, not proxied through a CDN — a proxying CDN returns an outright 403 to the gRPC-speaking agent, no explanation given, and the fix was disabling that proxying entirely for this one record rather than trying to work around it.
@@ -43,9 +56,10 @@ The real path a request takes:
 And this is the actual splitter doing that work — the real config, not a description of the logic:
 
 ```
-$ cat Caddyfile
+$ cat conf/Caddyfile
 {
-	admin off
+	# Container loopback only (port not published) - `caddy reload` needs it.
+	admin localhost:2019
 	auto_https off
 	servers {
 		protocols h1 h2 h2c
@@ -72,17 +86,19 @@ $ cat Caddyfile
 }
 ```
 
-## Fleet enrolment
+## Fleet Enrolment
 
 A dedicated Ansible role installs the mesh agent and enrols a host, opt-in per host via the same tag-driven mechanism as every other role in [`docs/iac-pipeline.md`](iac-pipeline.md) — nothing joins the mesh by default just because it exists. Enrolment uses a reusable, class-scoped setup key fetched live from the secrets platform at play time, never stored in the repo, and read only on a host's first join. Because peer approval is required, a brand-new peer pauses for an operator to approve it in the dashboard before it gets any mesh access at all — there's no silent auto-join path.
 
-All fleet hosts are enrolled today, plus two personal devices (a laptop with full mesh access, a home device with none by default — being *on* the mesh and being *allowed* anywhere on it are two separate, independently-granted things). Fleet servers run with local DNS resolution disabled on the mesh interface deliberately — they're not roaming, and the fleet's own internal DNS already handles them; only the one roaming device gets split-DNS behavior, resolving the fleet's internal zones over the mesh specifically because it's the one peer that actually needs to.
+All fleet hosts are enrolled today, plus a small number of personal devices — being *on* the mesh and being *allowed* anywhere on it are two separate, independently-granted things, and not every enrolled device gets the same access. Fleet servers run with local DNS resolution disabled on the mesh interface deliberately — they're not roaming, and the fleet's own internal DNS already handles them.
 
-This is the real, live peer roster, not a claimed enrolment count — every fleet host plus both personal devices, each with its own overlay identity:
+**Split-DNS for the one device that actually roams.** The mesh distributes a nameserver group scoping the fleet's own internal DNS zones (see [`docs/dns-naming.md`](dns-naming.md)) to a dedicated internal resolver, reachable over the mesh itself — but only to the admin group, and marked non-primary so it never competes with a peer's normal DNS for everything else. On the one peer that runs *with* mesh DNS enabled, those specific internal zones resolve through that mesh path; every other name still goes through whatever resolver the peer would normally use. On-LAN, the directly-connected network resolver still wins for those same zones regardless — this only matters, and only activates, off-LAN, which is exactly the one condition it exists for.
 
-![Real NetBird dashboard: 13 connected peers, a mix of fleet hosts and personal devices, each showing its own group membership, agent version, and OS](../assets/netbird-zero-trust-peers.png)
+This is the real, live peer roster, not a claimed enrolment count:
 
-## Zero-trust policy: deny-by-default, not just present
+![Real NetBird dashboard: connected peers, a mix of fleet hosts and personal devices, each showing its own group membership, agent version, and OS](../assets/netbird-zero-trust-peers.png)
+
+## Zero-trust Policy: deny-by-default, not just present
 
 The mesh ships with a built-in allow-everything default policy — the first real security decision here was turning it off. With it disabled, no traffic crosses the mesh interface at all unless a specific policy grants it; LAN traffic is completely untouched by this, since the policy only governs the overlay interface, not the physical network.
 
@@ -106,7 +122,7 @@ This is the mesh actually enforcing that model, not a claimed state — a real e
 ```
 $ netbird status --detail
 Peers detail:
- [laptop]:
+ [peer]:
   NetBird IP: [overlay address]
   Status: Idle
   -- detail --
@@ -120,20 +136,5 @@ Events:
 
 The network map updating on its own, unprompted, on a peer that isn't even the control plane, is the mesh's policy engine actually pushing live state to every connected peer — not a dashboard claiming a policy exists.
 
-## Rollout, phased and honestly incomplete
-
-Not everything is finished, and that's stated plainly rather than rounded up:
-
-| Phase | Scope | State |
-|---|---|---|
-| Control plane + dashboard, embedded identity | Stand up the mesh itself | Done |
-| First peers | Laptop + one fleet host, verified on-LAN | Done |
-| Fleet enrolment | Every fleet host via the dedicated role | Done |
-| Roaming | Split-DNS (done) + full-LAN route + off-LAN verification | Partly done |
-| Zero-trust policy | Groups, keys, deny-by-default, approval, posture, login expiry | Done |
-| Enforcement | Per-host firewall rejects non-mesh admin access | Planned |
-
-The honest gaps: off-LAN connectivity (STUN/relay through NAT) hasn't been verified end-to-end yet, the full internal-network route for agent-less devices is still open, and per-host firewall enforcement — actually *requiring* the mesh for admin access rather than just offering it as one more path — is a deliberate later phase, not an oversight. A phased rollout that's honest about which phases aren't done yet is worth more than a status page that rounds every in-progress item up to "complete."
-
 ---
 Related: [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/architecture-overview.md`](architecture-overview.md), [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/dns-naming.md`](dns-naming.md)

docs/notifications.md

@@ -1,18 +1,24 @@
+<!-- verified: 2026-09-19
+     automation-vm@7c057de : mosquitto
+     iac@ae315de : ansible/roles/monitoring/mqtt_client
+     trilium : tt4idCVJPpZM, EAU5LUwFFQ0E, rGfAb4CddKMi, otSPaU6IfjlP, klrmnlh3Wrnd
+-->
+
 # 🔔 Notifications
 
 > [!NOTE]
 > Mosquitto (MQTT) and a single Node-RED instance are the backbone here — two independent fan-in branches, app webhooks and OS-level host alerts, converging on one flow and one shared formatter before reaching Slack. The point isn't the individual pieces; it's that every alert reads the same regardless of where it came from.
 
-## Fan-in, fan-out
+## Fan-in, Fan-out
 
-![Three independent fan-in sources converging on one shared formatting subflow: 15 apps over HTTP webhook go through the App Router and Switch to a per-app mapper, Kopia/Bazarr post to their own dedicated endpoints and bypass the router entirely with their own mapper, and every host publishes over MQTT into a topic-agnostic Inventory Parse function — all three land on the same subflow before reaching Slack](../assets/notifications-fan-in-fan-out.svg)
+![Three independent fan-in sources converging on one shared formatting subflow: 14 apps over HTTP webhook go through the App Router and Switch to a per-app mapper, Kopia/Bazarr post to their own dedicated endpoints and bypass the router entirely with their own mapper, and every host publishes over MQTT into a topic-agnostic Inventory Parse function — all three land on the same subflow before reaching Slack](../assets/notifications-fan-in-fan-out.svg)
 
-## Why one Node-RED instance instead of per-app scripts
+## Why One Node-RED Instance Instead Of Per-App Scripts
 
 The alternative — each app posting directly to a per-app Slack webhook, or a small script per integration — was rejected because it would mean N different message formats, N different places to add a new integration's logic, and no single point to add fleet-wide behavior (like the OS-level checks below) without touching every app individually. One Node-RED instance with a shared router and a shared formatting subflow means a new app's alerts get the same visual treatment as every existing one automatically, and a fleet-wide change (say, adding a severity-based routing rule) happens in one place.
 
 > [!WARNING]
-> **Real debt in the flow**: of the apps wired into the router, two no longer resolve to anything — NetBox was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) but its detection code was never swept out, and Kestra's alerts actually travel a completely different path than its own webhook route suggests: Kestra publishes its own alerts over MQTT, not HTTP, which makes its dedicated webhook route effectively dead code. A couple of newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own onboarding pattern for a new app's alerts.
+> **Real debt in the flow, one instance already cleaned up**: NetBox's dead detection code — left behind after the app was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) — has since been swept out of the flow entirely. What's still live: Kestra's alerts actually travel a completely different path than its own webhook route suggests — Kestra publishes its own alerts over MQTT, not HTTP, which makes its dedicated webhook route effectively dead code. A couple of newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own onboarding pattern for a new app's alerts.
 
 This is the real routing table, not a claimed count — counted directly from the flow export, not restated from memory:
 
@@ -25,11 +31,11 @@ print('Switch rules:', len(switch['rules']))
 dedicated = [n for n in flows if n.get('type')=='http in' and n.get('url','').endswith('-webhook') and n.get('url') != '/app-webhook']
 print('Dedicated non-router endpoints:', len(dedicated))
 "
-Switch rules: 16
+Switch rules: 15
 Dedicated non-router endpoints: 2
 ```
 
-16 rules = the 15 routed apps plus one `unknown` fallback that just drops the payload. And this is the real function behind the "completely different path" line above — the MQTT branch doesn't parse `msg.topic` looking for a known set of fleet hosts, it just takes whatever the second path segment is:
+15 rules = the 14 routed apps plus one `unknown` fallback that just drops the payload. And this is the real function behind the "completely different path" line above — the MQTT branch doesn't parse `msg.topic` looking for a known set of fleet hosts, it just takes whatever the second path segment is:
 
 ```
 $ grep -A4 '"Inventory Parse Function' flows.json
@@ -42,11 +48,11 @@ return msg;
 
 Topic-agnostic by construction — so a publisher that isn't a fleet host at all gets handled identically to one that is, which is exactly the mechanism that makes an app's *own* MQTT-published alerts real and its dedicated webhook route genuinely dead, not just theoretically bypassable.
 
-This is the real flow, not a redrawn version of it — the App Router feeding a 16-way Switch, NetBox's and Kestra's mapper functions still sitting there wired in and unreachable, Kopia and Bazarr's dedicated endpoints bypassing all of it entirely on their way to the same shared subflow:
+This is the real flow, not a redrawn version of it — the App Router feeding a 15-way Switch, Kestra's mapper function still sitting there wired in and unreachable, Kopia and Bazarr's dedicated endpoints bypassing all of it entirely on their way to the same shared subflow:
 
-![Real Node-RED canvas: HTTP-in webhook through App Router into a 16-rule Switch function feeding per-app mapper functions (including netbox mapper and kestra mapper, both still wired but dead), MQTT-in through the Inventory Parse Function, and Kopia/Bazarr's own dedicated HTTP-in nodes bypassing the router entirely — all converging on the shared MQTT → Slack subflow](../assets/notifications-node-red-flow.png)
+![Real Node-RED canvas: HTTP-in webhook through App Router into a 15-rule Switch function feeding per-app mapper functions (including kestra mapper, still wired but dead since its real alerts travel over MQTT instead), MQTT-in through the Inventory Parse Function, and Kopia/Bazarr's own dedicated HTTP-in nodes bypassing the router entirely — all converging on the shared MQTT → Slack subflow](../assets/notifications-node-red-flow.png)
 
-## Message broker
+## Message Broker
 
 | | |
 |---|---|
@@ -85,11 +91,11 @@ And this is the broker actually being used, not just configured — real connect
 > [!IMPORTANT]
 > **A real, tracked fragility, worth naming precisely**: the publish identity's password has to stay in sync across three separate places — the Vault-rendered broker credential file, the source value used to render every host's own publish script (via the same tag-driven Ansible model in [`docs/iac-pipeline.md`](iac-pipeline.md)), and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from Vault at all, unlike everything else in [`docs/secret-distribution.md`](secret-distribution.md)). Rotating it means updating all three by hand — a documented, real exception to the "one source of truth" rule that governs every other secret in this fleet, called out explicitly rather than left for someone to discover mid-rotation.
 
-## Detection logic
+## Detection Logic
 
 The Node-RED App Router figures out which app sent a webhook by inspecting the payload's shape — an explicit field for some apps, structural pattern-matching for others (Emby, for instance, is identified purely by a nested `data.Server` field only that vendor's payload happens to include, since it never sends an explicit app identifier). Kopia and Bazarr skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON and the shared router's shape-detection logic simply doesn't apply to them.
 
-## Every host, not just apps
+## Every Host, Not Just Apps
 
 > [!TIP]
 > A dedicated Ansible role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert — the same fleet-wide, tag-driven rollout pattern used for the observability agent in [`docs/observability.md`](observability.md) and the compliance hardening in [`docs/security-monitoring.md`](security-monitoring.md). It runs over 20 distinct checks per host — disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (Vault audit log anomalies, Vault Agent authentication failures) — deliberately complementary to Wazuh in [`docs/security-monitoring.md`](security-monitoring.md), not duplicating its file-integrity or account monitoring. Splitting OS-health checks from the SOC's own detection engine keeps each system's role narrow enough to actually reason about, instead of one monolithic monitoring tool trying to do both jobs at once.
@@ -98,12 +104,12 @@ The Node-RED App Router figures out which app sent a webhook by inspecting the p
 >
 > ```
 > $ grep -oE '\$STATE_DIR/[a-zA-Z_]+' mqtt-monitor.sh.j2 | sed -E 's/(_alert|_last_count|_since|_report)$//' | sort -u | wc -l
-> 22
+> 23
 > ```
 >
-> 22 from that count, plus disk-usage thresholds — which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name — makes 23. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts.
+> Of that 23, one (`failed_units_current`) is a companion state file for the same "failed systemd units" check as `failed_units`, not a distinct 23rd check — so 22 real checks, plus disk-usage thresholds, which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name, still makes 23 total. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts.
 
-## One formatter, every source
+## One Formatter, Every Source
 
 Every branch — app webhooks, host-level alerts, and Kopia/Bazarr's own dedicated non-JSON endpoints — converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it — a reader in Slack never has to learn a different alert format depending on which system fired it.
 

docs/observability.md

@@ -1,3 +1,8 @@
+<!-- verified: 2026-09-19
+     iac@ae315de : ansible/roles/monitoring/observability_agent
+     trilium : eeiUz4iTEgPL, zquji4XOpw5M, s5tWZshDEpQb, oWr1jJhvqitv
+-->
+
 # 📈 Observability
 
 > [!NOTE]
@@ -10,7 +15,7 @@
 | **Metrics** | Push-based `remote_write`, not pull-based scraping |
 | **Coverage** | All fleet hosts, confirmed live |
 
-## Why push instead of pull — the actual reasoning
+## Why Push Instead Of Pull — The Actual Eeasoning
 
 A pull-based model (Prometheus scraping every host directly) is the more common default, and it was rejected for a specific reason tied directly to [`docs/firewall-and-network.md`](firewall-and-network.md): scraping means the monitoring host needs inbound network access to every single host it monitors, which means punching a hole through the zero-blanket-allow firewall model for every host, every metrics port. A push model inverts that — only the monitoring host needs to accept inbound writes, and every other host only ever *initiates* outbound connections, which fits the existing "explicit outbound rule, default-deny inbound" posture instead of fighting it. The tradeoff is real: push-based metrics are slightly harder to reason about for ad-hoc debugging (no simple `curl` against a target to see what it'd report), but it was the right call given the network model already in place.
 
@@ -18,7 +23,7 @@ A pull-based model (Prometheus scraping every host directly) is the more common
 
 ![Four per-host sources (journald, Docker containers, node metrics, native app metrics) all feed into Grafana Alloy, which ships logs to Loki and metrics to Prometheus via remote_write every 30s](../assets/observability-pipeline.svg)
 
-## Agent install: deliberate, not default
+## Agent Install: deliberate, not default
 
 > [!TIP]
 > The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. That's a real maintenance cost (manual version bumps instead of `apt upgrade` handling it), accepted because it keeps the exact version pinned and reproducible across the fleet through the same tag-driven Ansible model described in [`docs/iac-pipeline.md`](iac-pipeline.md), rather than whatever version a package repo happens to serve on a given day. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).
@@ -43,7 +48,9 @@ SystemKeepFree=1G
 MaxRetentionSec=14day
 ```
 
-![Three independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, and the DNS query log as job=dns with real event time instead of ingestion time](../assets/observability-log-fanout.svg)
+![Four independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, the DNS query log as job=technitium-dns with real event time instead of ingestion time, and auditd's own log as job=auditd on the 6 hosts that carry it](../assets/observability-log-fanout.svg)
+
+A fourth, newer branch ships raw auditd logs to Loki too, on the 6 hosts that carry the `auditd_hardening` compliance role (automation, caddy, monitoring, technitium, vault, wazuh) — a real gap-fill after one host's own `/var/log/audit` once filled to 100% with zero external visibility to catch it. It doesn't replace the local cleanup timer that actually bounds that disk usage; it just means the data isn't only ever local anymore.
 
 This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance:
 

docs/orchestration.md

@@ -1,19 +1,24 @@
+<!-- verified: 2026-09-19
+     ansible@24e1bf0 : host_vars/hyper2.yml, roles/os_update/tasks/reboot.yml, playbooks/os_update_pve_verify.yml, playbooks/automation_self_update.yml
+     trilium : bW84ccRoAJ5i, tt4idCVJPpZM
+-->
+
 # 🔁 Orchestration
 
 > [!NOTE]
-> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint.
+> Kestra runs every scheduled maintenance flow across the fleet — 15 flows covering OS updates, container updates, app updates, disk cleanup, fleet inventory, KB self-maintenance, and public-doc drift checks. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint.
 
 This is the real flow list, not a claimed count:
 
-![Kestra's real flow list, homelab namespace: 13 flows total, every one showing a real last-execution date and SUCCESS status](../assets/orchestration-flow-list.png)
+![Kestra's flow list for the homelab namespace — every flow showing a real last-execution date and SUCCESS status](../assets/orchestration-flow-list.png)
 
-## Why only one flow gets a human gate
+## Why Only One Flow Gets a Human Gate
 
 A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in [`docs/registry-and-build.md`](registry-and-build.md)), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't.
 
-## Two patterns, not one
+## Two Patterns, Not One
 
-![12 of 13 flows run straight through on schedule with no human gate, relying on the automated health-check/rollback path instead; the one flow applying Vault/Technitium/Alloy updates runs a check-mode pass, pauses indefinitely on anything pending, and only applies after explicit human approval](../assets/orchestration-flow-patterns.svg)
+![14 of 15 flows run straight through on schedule with no human gate, relying on the automated health-check/rollback path instead; the one flow applying Vault/Technitium/Alloy updates runs a check-mode pass, pauses indefinitely on anything pending, and only applies after explicit human approval](../assets/orchestration-flow-patterns.svg)
 
 That's not just a diagram of the pattern — it's the actual flow's own topology in Kestra, `wait_for_approval` sitting right there as a real node between the pending-check and the apply step:
 
@@ -25,7 +30,8 @@ This is the real check, not an assumption — every flow's own definition, queri
 
 ```
 $ for flow in os-update-vms os-update-pve os-update-pve-verify os-update-pbs \
-              container-update disk-cleanup app-update; do
+              container-update disk-cleanup app-update homelab-inventory-sync \
+              public-docs-drift; do
     echo "$flow: $(curl -s .../flows/homelab/$flow | grep -c 'core.flow.Pause')"
   done
 os-update-vms: 0
@@ -35,14 +41,30 @@ os-update-pbs: 0
 container-update: 0
 disk-cleanup: 0
 app-update: 1
+homelab-inventory-sync: 0
+public-docs-drift: 0
 ```
 
-## Identity split, per flow
+## One Dedicated Identity, Not a Shared One
 
 > [!IMPORTANT]
-> Most flows connect using one operator identity over SSH. One flow — the one that needs to run Ansible itself on the control node, not just reach out over SSH to a target — deliberately logs in as a separate, narrower service identity instead, since it needs local execution rights on the control node that the SSH-only identity doesn't have. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally, following the same "scope credentials to exactly what's needed" instinct as the least-privilege Git access model in [`docs/sso-and-identity.md`](sso-and-identity.md). Worth checking which one a given flow uses before assuming — it's not always the same answer, and assuming wrong is a fast way to get a confusing permission error mid-debugging.
+> Every flow connects to automation using the exact same credential — a dedicated keypair that exists for this one purpose and nothing else, unrelated to the `ansible` automation identity used everywhere else in this repo. That separation is deliberate, not incidental: Kestra's SSH task plugin is a plain Java SSH client with no certificate support at all, so it can only ever authenticate with a static key — it has no way to participate in the certificate-based model the rest of the fleet's `ansible` identity uses (see [`docs/secrets-and-pki.md`](secrets-and-pki.md)). Giving Kestra its own key instead of reusing `ansible`'s means a leak of one credential can never expose the other — the blast radius of "Kestra's automation-host hop" and "the fleet-wide `ansible` identity" stay fully independent. This is the real config, not a paraphrase — every SSH task in every flow points at the same two KV entries:
+>
+> ```
+> $ curl -s .../flows/homelab/app-update?source=true | grep -A2 'type: io.kestra.plugin.fs.ssh.Command'
+> type: io.kestra.plugin.fs.ssh.Command
+> ...
+> username: "{{ kv('kestra_local_user') }}"
+> privateKey: "{{ kv('kestra_local_key') }}"
+> ```
+>
+> The SSH hop itself is only ever a means to escape Kestra's own container and reach the control node's local filesystem and tooling — it's never used to reach the wider fleet. `ansible-playbook`, once it's actually running (as a local process, invoked over that same SSH hop), makes its own outbound connections to every other fleet host, using the fleet-wide `ansible` identity and its own certificate.
 
-## A real special case, thought through in advance
+This is the actual KV Store, not a description of it — exactly two entries, nothing else:
+
+![Kestra's homelab-namespace KV Store: exactly two entries, kestra_local_key and kestra_local_user, the dedicated credential every flow's SSH task uses](../assets/orchestration-kestra-kv-store.png)
+
+## A Real Special Case, Thought Through In Advance
 
 > [!WARNING]
 > One of the two Proxmox hypervisor nodes also happens to host the automation VM that Kestra itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death, since the flow doing the waiting dies with the VM it's waiting on. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM, including the notification pipeline itself from [`docs/notifications.md`](notifications.md), which also runs there. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. This is the kind of edge case that only shows up once you actually think through "what happens when the thing doing the orchestrating is itself one of the things being orchestrated" — a real, non-obvious problem in any self-hosted automation system. This is the real config comment, not a summary of the reasoning:
@@ -57,9 +79,9 @@ app-update: 1
 > reboot_wait_for_reconnect: false
 > ```
 
-## Trilium KB self-maintenance
+## Trilium KB Self-maintenance
 
-Two flows keep this fleet's own internal knowledge base structure current — the *only* two things allowed to write to it, everything else (including this repo's own generation) only ever reads from it:
+Two flows keep this fleet's own internal knowledge base structure current. Those two, plus one more that only ever tags notes for review without altering their content, are the only things that write to the KB at all — everything else, including this repo's own generation, strictly reads from it:
 
 | Flow | Trigger | What it does |
 |---|---|---|
@@ -69,14 +91,14 @@ Two flows keep this fleet's own internal knowledge base structure current — th
 > [!TIP]
 > The push-triggered flow also runs a lightweight completeness check for the fleet's own host-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel via [`docs/notifications.md`](notifications.md) — a small, automatic nudge against exactly the kind of documentation drift that this whole portfolio repo exists to avoid in the first place.
 
-## Outside the orchestration engine entirely
+## Outside The Orchestration Engine Entirely
 
 The automation host's own OS patches and Kestra's own container updates run via a standalone host-level systemd timer, not through Kestra itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run without risking killing the process doing the patching. Being outside the normal flow means these updates don't get the same visibility as everything else in this doc, which is a deliberate, accepted tradeoff — worth a periodic manual check rather than fully trusting the dashboard to surface it. This is the real timer, not an assumption about its schedule:
 
 ```
 $ systemctl list-timers automation-self-update.timer
-NEXT                          LEFT  LAST                          PASSED  UNIT
+NEXT                          LEFT     LAST                          PASSED   UNIT
-Sat 2026-08-29 04:00:00 EEST  ...   Sat 2026-08-22 04:00:26 EEST   -      automation-self-update.timer
+Sat 2026-09-26 04:00:00 EEST  6 days   Sat 2026-09-19 04:00:09 EEST  5h ago   automation-self-update.timer
 ```
 
 ---

docs/registry-and-build.md

@@ -1,3 +1,9 @@
+<!-- verified: 2026-09-19
+     ansible@24e1bf0 : host_vars/rocky-docker/docker_stacks.yml, host_vars/netbird/docker_stacks.yml, roles/container_update, playbooks/container_update.yml
+     iac@ae315de : ansible/roles/secure_vault/vault_agent/defaults/main.yml
+     trilium : dcRUArk5dWH8, bW84ccRoAJ5i, hWiIGZIlvQiN
+-->
+
 # 📦 Self-Hosted Registry & Build Pipeline
 
 > [!NOTE]
@@ -7,9 +13,9 @@
 |---|---|
 | **Registry** | Gitea's own built-in container registry |
 | **Trigger** | Weekly schedule via Kestra, not push-triggered |
-| **Rollout** | All 5 Docker hosts converted; the largest has 36 of 41 stacks on this pattern |
+| **Rollout** | All 6 Docker hosts converted; the largest has 35 of 41 stacks on this pattern |
 
-## Why self-host the registry instead of just using Docker Hub
+## Why Self-host The Registry Instead Of Just Using Docker Hub
 
 Pulling directly from Docker Hub/ghcr at runtime was the status quo before this existed, and it was replaced for a concrete reason: every container pull became a runtime dependency on an external service staying up and staying reachable through the firewall model in [`docs/firewall-and-network.md`](firewall-and-network.md). Building images in-fleet and pushing to a registry that's already inside the trust boundary means a redeploy never depends on an external registry's availability, and the exact image contents are fully within this fleet's own control rather than whatever an upstream maintainer pushed as `:latest` today.
 
@@ -19,28 +25,29 @@ This is the real registry, not a claim about it — every one of these is a genu
 
 ![Gitea's own package registry, filtered to container type — real self-built images including this fleet's own automation tooling, all private](../assets/registry-and-build-packages.png)
 
-## The cheap-check-first build decision
+## The Cheap-check-first Build Decision
 
 ![Every build starts with docker manifest inspect — metadata only, no pull — comparing the upstream image's digest against what's already in the fleet registry. If they match, nothing runs at all. If they differ, only then does it pull, build, compare image IDs, push what changed, and redeploy through the shared health-check/rollback path](../assets/registry-and-build-cheap-check.svg)
 
 > [!TIP]
 > Most stacks use a minimal `FROM <upstream-image>:${VERSION}` wrapper Dockerfile rather than a full clone-and-build-from-source setup — simpler, builds near-instantly, can't break from upstream build tooling changing underneath it, while still achieving the real goal: an in-fleet registry with no direct Docker Hub/ghcr dependency at runtime. A handful of multi-build-image stacks (the ones with more than one real upstream component, like Paperless-ngx, Ghost, Ollama, and the Grafana stack) skip the cheap check entirely, since a single "upstream digest" comparison doesn't make sense when there's more than one upstream involved.
 
-## Deliberate holdouts
+## Deliberate Holdouts
 
 Not every stack is on this pattern, and that's a decision, not an oversight — knowing when *not* to apply a pattern uniformly is as much a design skill as building the pattern in the first place. This is the real breakdown, not a rounded estimate:
 
 ```
 $ python3 -c "import yaml; d=yaml.safe_load(open('docker_stacks.yml'))['stacks']; ..."
 total: 41
-dockerfile     36
+dockerfile     35
-pinned_manual   3
+pinned_manual   4
 latest          2
 ```
 
 | Stack | Why it's held out |
 |---|---|
 | Authentik, Immich, tdarr | Complex multi-service upstream builds — not worth replicating a build process the upstream project already maintains well; the maintenance cost of tracking their own build changes would exceed the benefit |
+| Music-assistant | Was on this pattern, deliberately reverted — auto-updating `:latest` kept bumping its `smart_fades`/`loudness_analysis` `analysis_version`, mass-invalidating ~86k tracks' audio-analysis backlog each time it moved; now pinned by hand, bumped only after checking release notes |
 | Kodi's own app image | Genuinely still a direct Docker Hub dependency — low enough stakes that the extra build step isn't worth it |
 | Nextcloud's own `nextcloud`/`euro-office` services | Pre-existing custom Dockerfiles, left as-is beyond adding `image:` references — its own db/redis still float on Docker Hub tags directly, a conscious "don't touch what already works" call |
 
@@ -57,7 +64,7 @@ mariadb-latest:
     firefly iii, kodi) pulls from our registry instead of Docker Hub.
 ```
 
-## Trigger discipline
+## Trigger Discipline
 
 > [!WARNING]
 > This is schedule-triggered, not push-triggered — a deliberate choice, not a missing feature. Editing a stack's own Dockerfile doesn't by itself trigger a rebuild; it waits for the next weekly sweep to find the upstream base image has actually changed, or for someone to force a manual run via [`docs/orchestration.md`](orchestration.md). The tradeoff: a simpler mental model and no rebuild storm on every unrelated commit, at the real cost of same-day propagation for an urgent base-image CVE fix — accepted because a weekly cadence matches this fleet's actual risk tolerance, not because push-triggering would have been harder to build.

docs/secret-distribution.md

@@ -1,13 +1,22 @@
+<!-- verified: 2026-09-19
+     iac@ae315de : ansible/roles/secure_vault/ssh_ca_client, ansible/roles/secure_vault/vault_agent/defaults/main.yml, ansible/roles/secure_vault/SecureVault – Vault Contract.md, ansible/roles/secure_vault/securevault-vault-facts.md
+     ansible@24e1bf0 : roles/app_update/defaults/main.yml
+     trilium : DvKxK0gHXsng, wovVrgvoDVy2, GW8BMoTKtRBe, 1seDR2nBmxKO
+-->
+
 # 🔑 Secret Distribution Model
 
 > [!NOTE]
-> [`docs/secrets-and-pki.md`](secrets-and-pki.md) covers *what* Vault is and *why* it exists. This doc covers something different and, arguably, harder to get right: the actual contract for how a secret value moves from being written into Vault to being consumed by a running container — and why that contract is deliberately split into four separate flows instead of one universal one.
+> [`docs/secrets-and-pki.md`](secrets-and-pki.md) covers *what* Vault is and *why* it exists. This doc covers something different and, arguably, harder to get right: the actual contract for how a secret value moves from being written into Vault to being consumed by a running container — and why that contract is deliberately split into five separate flows instead of one universal one.
 
-![Secret distribution flows: Configuration Distribution Flow (orange) is the control plane that gates what Vault Agent renders; Certificate Flow and App Secret Flow (purple) are the data plane running through that same agent; Administrative Secret Flow (gray) is a genuinely separate system with no Vault involvement](../assets/secret-distribution-flows.svg)
+![Secret distribution flows: Configuration Distribution Flow (orange) is the control plane that gates what Vault Agent renders; Certificate Flow and App Secret Flow (purple) are rendered by that same agent; SSH Certificate Flow (also purple, dashed line) reuses the agent's own token but bypasses its render cycle entirely, feeding fleet-wide sshd trust instead of a container; Administrative Secret Flow (gray) is a genuinely separate system with no Vault involvement](../assets/secret-distribution-flows.svg)
 
-## Why four flows instead of one
+## Why Five Flows Instead Of One
 
-The obvious simpler design is one universal "secret goes in, secret comes out" pipeline. It was rejected because the four things being moved genuinely have different risk profiles and different failure tolerances: a certificate expiring is a slow-building problem measured in days; a missing application secret is often an immediate outage; a broken config-distribution push can silently stop *all* of a host's secrets from updating, not just one. Treating them identically would mean either over-engineering the simple cases or under-protecting the risky one. Four flows, each with its own explicit Law/Flow/Failure-behavior contract, is more code to maintain but far easier to reason about when something breaks — anyone debugging an incident knows immediately which contract governs the thing that's wrong.
+The obvious simpler design is one universal "secret goes in, secret comes out" pipeline. It was rejected because the five things being moved genuinely have different risk profiles and different failure tolerances: a certificate expiring is a slow-building problem measured in days; a missing application secret is often an immediate outage; a broken config-distribution push can silently stop *all* of a host's secrets from updating, not just one; losing the ability to refresh an SSH certificate doesn't break anything immediately, but eventually locks the automation identity out of the fleet entirely. Treating them identically would mean either over-engineering the simple cases or under-protecting the risky ones. Five flows, each with its own explicit Law/Flow/Failure-behavior contract, is more code to maintain but far easier to reason about when something breaks — anyone debugging an incident knows immediately which contract governs the thing that's wrong.
+
+> [!NOTE]
+> **Why the SSH Certificate Flow isn't just folded into Certificate Flow** — both issue short-lived Vault-signed credentials, so treating them as one flow looks tempting. They're kept separate because the actual mechanism is genuinely different, not just the credential type: Certificate Flow is triggered and rendered by Vault Agent itself, on Vault Agent's own schedule. The SSH Certificate Flow runs on its own independent systemd timer, calls Vault directly, and only *reuses* Vault Agent's already-authenticated token as a convenience — Vault Agent never renders it and doesn't know it exists. Collapsing them into one contract would hide that difference from exactly the person who needs it most: whoever is debugging why an SSH certificate didn't refresh, who needs to know immediately that the answer isn't "check Vault Agent's logs."
 
 ## 1️⃣ Certificate Flow
 
@@ -19,7 +28,30 @@ The obvious simpler design is one universal "secret goes in, secret comes out" p
 
 **Invariants:** certificates are short-lived and re-issued on Vault Agent's own schedule, never on-demand by the consumer. A host only ever receives certs tagged for its own identity. The private key never traverses the network after issuance.
 
-## 2️⃣ Application Secret Flow
+## 2️⃣ SSH Certificate Flow
+
+| | |
+|---|---|
+| **Law** | The automation identity never holds a static, standing-access SSH key. Only a Vault-signed certificate, valid for hours, ever grants it access to a fleet host. |
+| **Flow** | Its own systemd timer (2h interval) → reuses Vault Agent's already-authenticated token → `vault write ssh-client-signer/sign/[role name]` → certificate installed next to the identity's own key on the one host that needs it → every fleet host's sshd trusts the CA directly, no per-host secret involved |
+| **Failure behavior** | A single missed refresh is harmless — the timer fires every 2 hours against a 6-hour certificate, so there's hours of margin. A sustained outage spanning several missed cycles eventually locks the automation identity out of certificate-based SSH entirely, with no static key to fall back to — an accepted tradeoff, not an oversight (see [`docs/secrets-and-pki.md`](secrets-and-pki.md)) |
+
+This is the real shape of a signed certificate from this flow, not a description of one — host path, CA fingerprint, and validity window redacted since they identify live infrastructure, everything else genuine:
+
+```
+$ ssh-keygen -Lf [cert path]
+Type: ssh-ed25519-cert-v01@openssh.com user certificate
+Signing CA: RSA SHA256:[fleet's CA fingerprint] (using rsa-sha2-512)
+Valid: from [issued] to [issued + 6h]
+Principals:
+        [automation identity]
+Extensions:
+        permit-port-forwarding
+```
+
+**Invariants:** exactly one identity is ever signed for. Vault's role config enforces this independently of the script that calls it — a compromised or buggy script still can't get a certificate for any other principal. Every host trusts the CA's public key directly; none of them store the automation identity's actual key material at all.
+
+## 3️⃣ Application Secret Flow
 
 | | |
 |---|---|
@@ -30,7 +62,7 @@ The obvious simpler design is one universal "secret goes in, secret comes out" p
 > [!WARNING]
 > The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it. This is the single most common way this model gets violated by accident: someone SSHes in to fix a value under time pressure, and the fix silently disappears on the next render.
 
-## 3️⃣ Configuration Distribution Flow
+## 4️⃣ Configuration Distribution Flow
 
 | | |
 |---|---|
@@ -48,14 +80,14 @@ That redeploy-and-restart sequence isn't just a claim — here's a real one, sta
 **Shared-secret variant:** not every template is host-scoped — one credential needed by every docker host for its own registry pushes (see [`docs/registry-and-build.md`](registry-and-build.md)) is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one — and it relies on every host's policy already granting read access to that shared path, the same "shared + private" pattern described in [`docs/secrets-and-pki.md`](secrets-and-pki.md)'s access model. This is the real template list entry, repeated per host:
 
 ```
-$ grep -A4 "service: automation" vault_agent/defaults/main.yml
+$ grep -B1 -A3 "registry-token.ctmpl" vault_agent/defaults/main.yml
 - service: automation
   src_path: .../automation-vm/registry-token/registry-token.ctmpl
   dest_path: /opt/docker/docker_volumes/registry-token/registry-token
   reload_command: "chmod 600 .../registry-token/registry-token"
 ```
 
-## 4️⃣ Administrative Secret Flow — a deliberately separate, simpler system
+## 5️⃣ Administrative Secret Flow — a deliberately separate, simpler system
 
 | | |
 |---|---|

docs/secrets-and-pki.md

@@ -1,3 +1,9 @@
+<!-- verified: 2026-09-19
+     iac@ae315de : ansible/roles/secure_vault, ansible/group_vars/all/vault_provisioner.yml, ansible/roles/monitoring/mqtt_client
+     rocky-docker-lxc@d225776 : caddy/conf/Caddyfile
+     trilium : Eu0wns00QOBG, wovVrgvoDVy2, 1seDR2nBmxKO, GW8BMoTKtRBe, JbKe9CHzOP9D, Tu7FX48fxSQZ, DvKxK0gHXsng, otSPaU6IfjlP
+-->
+
 # 🔐 Secrets & PKI
 
 > [!NOTE]
@@ -6,21 +12,39 @@
 | | |
 |---|---|
 | **Software** | HashiCorp Vault, self-hosted, native systemd service (not containerized) |
-| **Role** | Certificate authority (2 PKI mounts) + secret store (KV v2) |
+| **Role** | Certificate authority (2 PKI mounts + 1 SSH CA mount) + secret store (KV v2) |
 | **Distribution** | Vault Agent on every consuming host, AppRole auth, renders to local disk |
 | **Replaces** | A prior plaintext-`.env`-in-git secrets model |
 
-## Why Vault, and why self-hosted
+## Why Vault, and Why Self-hosted
 
 A managed secrets service (a cloud provider's KMS/Secrets Manager) was a real alternative and was rejected for a specific reason: this fleet's PKI needs to issue certificates to purely internal hosts that have no business talking to a cloud API, and the two-zone reverse-proxy model in [`docs/dns-naming.md`](dns-naming.md) depends on an internal CA that's reachable without any external dependency. Vault does both certificate issuance and secret storage under one trust model instead of stitching together a separate CA tool and a separate secrets tool — fewer moving parts, one audit log, one policy language to reason about.
 
-## What it does
+## What It Does
 
 - **Certificate authority** — two PKI mounts: one for public-facing wildcard certs (consumed by the edge in [`docs/firewall-and-network.md`](firewall-and-network.md)), one for internal mTLS certs (consumed by the browser-facing reverse-proxy zone). Both issue short-lived certificates, not long-lived ones — a deliberate choice that shrinks the window a leaked cert is useful for.
 - **Secret store** — a KV v2 mount holds every migrated application's real secret values, one namespace per host, enforced by policy (below).
 - **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly — this single rule is what makes the rest of the model tractable to reason about.
+- **SSH certificate authority** — a dedicated mount issues short-lived SSH certificates for the fleet's automation identity, instead of that identity holding a single static key with standing, permanent access to every host. A signed certificate is valid for 6 hours (12-hour hard cap), scoped to exactly one principal, and every host trusts the CA's public key directly rather than storing that identity's key at all — the same short-lived-credential philosophy as the PKI mounts above, applied to SSH instead of TLS. A leaked certificate is worthless in hours; a leaked static key used to be useful forever. The one thing that can't use this model at all is Kestra's own SSH-based automation flows — its SSH plugin is a plain client with no certificate support, full stop — see [`docs/orchestration.md`](orchestration.md) for what it uses instead, and why that's a completely separate, unrelated credential rather than a workaround bolted onto this one.
 
-## Why it's built this way
+This is the real role definition, not a paraphrase — role and principal names redacted since they identify a live credential, everything else genuine:
+
+```
+$ vault read ssh-client-signer/roles/[role name]
+key_type              ca
+allowed_users         [automation identity]
+default_user          [automation identity]
+ttl                   6h    (21600s)
+max_ttl               12h   (43200s)
+algorithm_signer      rsa-sha2-512
+default_extensions    permit-pty, permit-port-forwarding
+```
+
+And this is the real mount list, not an assumption of what's enabled — the SSH CA sitting right alongside the two PKI mounts, not a separate system bolted on beside Vault:
+
+![Vault's live Secrets engines list: cubbyhole/, kv/, pki/ (public wildcard), pki_int/ (internal mTLS), and ssh-client-signer/ — the SSH certificate authority is a real mount, not a description of an aspiration](../assets/vault-secrets-engines.png)
+
+## Why It's Built This Way
 
 Before this system existed, every stack's secrets sat in plaintext inside git-tracked `.env` files — readable by anyone with repo access, and permanently baked into git history even after being "fixed." SecureVault replaces that:
 
@@ -44,17 +68,19 @@ Every Vault Agent reaches Vault through the internal app-to-app proxy zone descr
 | User-Facing Proxy | Server + client certs for the mTLS-enforced browser zone, plus a shared registry-token render and Hawser's application secret |
 | Internal DNS | Server cert for its own admin UI/API only |
 | Security Monitoring | Its manager and dashboard services' shared secrets (indexer, API, dashboard passwords), plus Hawser's application secret, plus a shared registry-token render |
+| NetBird | Its entire server config rendered as one file, embedded secrets included — the same file-templating pattern used for Kestra's `application.yml` — plus Hawser's application secret, plus a shared registry-token render |
 
 None of these ever call the Vault API themselves — every one of them goes through the same local-render sequence:
 
 ![Secret lifecycle: Vault Agent authenticates via AppRole, gets a token, renders the secret to a local file, the application reads that file and never talks to Vault directly — rotation happens asynchronously, updated once in Vault and picked up on the agent's next render](../assets/secrets-and-pki-lifecycle.svg)
 
-**Two authentication flows, not one:**
+**Three authentication flows, not one:**
 
 | Flow | When | Path |
 |---|---|---|
 | AppRole auth | Every host, ongoing | Vault Agent → role_id + secret_id → Vault → short-lived token, cached, auto-renewed |
-| Bootstrap auth | Rare, human-initiated | Operator uses a root token to mint new AppRole credentials or update policies — never part of any workload's runtime path |
+| Provisioner auth | Occasional, playbook-triggered | A narrowly-scoped AppRole mints new host AppRole credentials and reads two specific KV paths — cannot create policies or AppRole roles itself |
+| Break-glass auth | Rare, human-initiated, no standing token | Operator + a quorum of unseal-key shares → a freshly-minted root token, used once, revoked immediately — the only path that can create policies or onboard a genuinely new host |
 
 > [!IMPORTANT]
 > **The one deliberately-asymmetric trust decision**: every other backend connection inside the internal zone described in [`docs/dns-naming.md`](dns-naming.md) skips certificate verification on the reverse-proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic between containers that already share a trusted network. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert. Given that Vault is the root of trust for every other secret in the fleet, treating it identically to a media-server backend would have been the actual mistake — the asymmetry here is the correct call, not an inconsistency. This is the real config contrast, not a description of it — every other backend uses this:
@@ -80,21 +106,21 @@ None of these ever call the Vault API themselves — every one of them goes thro
 > }
 > ```
 
-## What's protected
+## What's Protected
 
 | Layer | Protection |
 |---|---|
 | Transport | Every Vault Agent ↔ Vault connection is TLS, certificate-verified |
 | Secrets at rest | KV v2, encrypted by Vault's storage layer; rendered files are `0600`, root-owned, gitignored |
 | Audit trail | Every request/response logged to a file audit device; secret values HMAC-hashed, not plaintext — what's visible is metadata: which token, which path, when. Rotated weekly, 8 generations kept. |
-| Alerting | Root-token use or permission-denied bursts trigger an alert through [`docs/notifications.md`](notifications.md)'s pipeline |
+| Alerting | Any root-policy token being used, or permission-denied bursts, trigger an alert through [`docs/notifications.md`](notifications.md)'s pipeline — the check matches generically on the token's policy, so it fires identically whether that's a routine mistake or a genuine break-glass event |
 | Cert renewal reliability | Technitium's DNS-over-TLS/DoH cert bundle auto-rebuilds whenever Vault Agent re-renders its certs, guarded to skip unless under 7 days from expiry. This closed a real prior gap: the rebuild never used to happen automatically, and an expired cert was silently served for 19 days before anyone caught it. |
 
 This is the real hook script, not a description of it — the full guard logic, the actual PKCS12 rebuild, and the service restart it triggers (password redacted, everything else genuine):
 
 ![The real technitium-rebuild-p12.sh script: the 19-day-expired-cert incident documented in the script's own comments, the 7-day expiry guard, and the actual PKCS12 rebuild + dns.service restart](../assets/secrets-and-pki-cert-rebuild-hook.png)
 
-## Access model
+## Access Model
 
 One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does *not* automatically imply the private one, so both get verified before assuming a host is fully provisioned. This same "shared + private" pattern shows up again in [`docs/iac-pipeline.md`](iac-pipeline.md)'s tag-driven Ansible model — a deliberate consistency, not a coincidence.
 

docs/security-monitoring.md

@@ -1,3 +1,9 @@
+<!-- verified: 2026-09-19
+     iac@ae315de : ansible/roles/monitoring/wazuh_agent_tuning, ansible/roles/monitoring/docker_bench, ansible/roles/cis_hardening
+     wazuh-vm@07ed482 : wazuh/docker-compose.yml
+     trilium : KpvtX1USDhkY, TstjWVV5Q1TS, OtIgVyQC5lJg, MSVD5o5HiJ69, kA00FQiQWVem, oY5CbyuxZRyS
+-->
+
 # 🛡️ Security Monitoring
 
 > [!NOTE]
@@ -6,7 +12,7 @@
 | | |
 |---|---|
 | **Software** | Wazuh (manager/indexer/dashboard), self-hosted |
-| **Coverage** | 11 active agents, one per fleet host, no gaps |
+| **Coverage** | 12 active agents, one per fleet host, no gaps |
 | **Detection layers** | Compliance scoring (SCA) · file integrity monitoring (syscheck) · rootkit/anomaly heuristics (rootcheck) |
 | **Runbook** | 10+ recurring alert types root-caused once, documented, never re-triaged from scratch |
 | **Query path** | Direct index queries, bypassing the SOC dashboard entirely when needed |
@@ -27,16 +33,17 @@ ID: 009, Name: vault, IP: any, Active
 ID: 010, Name: caddy, IP: any, Active
 ID: 011, Name: technitium, IP: any, Active
 ID: 012, Name: file-server, IP: any, Active
+ID: 013, Name: netbird, IP: any, Active
 ```
 
-## Why agent-based, and why self-hosted
+## Why Agent-based, and Why Self-hosted
 
 The alternative to an agent on every host is a purely network-level monitoring approach — watching traffic at the firewall in [`docs/firewall-and-network.md`](firewall-and-network.md) and inferring what's happening inside each host. That approach can't see file integrity changes, local privilege escalation, or rootkit behavior — the things that matter most once an attacker is already past the network perimeter. An agent on every host closes that gap at the cost of real per-host overhead and a rollout that has to cover 100% of the fleet to be trustworthy — a partial rollout would create a false sense of coverage that's arguably worse than no agent at all. Self-hosting the manager/indexer/dashboard stack, rather than using a SaaS SIEM, keeps every alert's raw data (hostnames, internal paths, credential-adjacent metadata) inside the same trust boundary Vault protects in [`docs/secrets-and-pki.md`](secrets-and-pki.md), instead of shipping it to a third party.
 
 > [!TIP]
 > **The manager watches itself, as two independent things sharing one VM.** The manager/indexer/dashboard Docker stack and the native agent monitoring that same host are unrelated — the manager isn't self-aware, it's just also enrolled as agent `000`. And the two halves of "how Wazuh is configured" have different discipline levels on purpose: the compose stack itself is tracked in its own Gitea repo, but the actual detection rules and decoders live in a Docker volume, edited directly through the dashboard or manager API — not version-controlled. If the manager goes down entirely, agents don't lose events: they buffer locally and replay once it's back, so an outage costs visibility, not data.
 
-## A fleet-wide bug this system caught in itself
+## A Fleet-wide Bug This System Caught In Itself
 
 > [!IMPORTANT]
 > Wazuh's own default agent log-queue buffer (1024 entries) overflowed fleet-wide during Ansible orchestration bursts — enough hosts logging enough events at once that the queue filled before the agent could forward them, silently dropping data rather than erroring loudly. Found, not assumed: root-caused to the default queue size specifically, then fixed with a dedicated tuning role that raises `logcollector.queue_size` on every agent. That role is deliberately narrow — it only touches queue size and enabled/started state; syscheck ignores, rootcheck ignores, and other per-agent tuning are still one-off manual edits, honestly left uncovered rather than implied as handled by a role that doesn't actually reach them. This is the real root-cause comment from the fix itself, not a summary of it:
@@ -50,12 +57,14 @@ The alternative to an agent on every host is a purely network-level monitoring a
 > # during that night's role redeploys, confirmed via the indexer across
 > # 9 of the fleet's agents.
 > ```
+>
+> The same role found a second gap a month later, in itself this time: it silently skipped tuning the wazuh host's own agent, because that agent runs inside the manager's Docker container rather than natively on the VM — the role's own "is the agent installed" check only ever looked for a native install. That let the wazuh host's own queue sit at the untuned default until a real queue-full alert exposed it, the same class of bug the role was originally written to fix, this time hiding in the one host built differently from the rest. Fixed by giving that host its own branch instead of assuming every agent looks the same.
 
-## How an alert gets handled
+## How An Alert Gets Handled
 
 ![An alert either matches a known, already-root-caused pattern and goes straight to the runbook with no re-investigation, or it doesn't and gets investigated fresh via wazuh-logtest and a direct index query — a fresh investigation that becomes routine gets folded into the runbook for next time](../assets/security-monitoring-alert-flow.svg)
 
-## The curated "don't re-investigate this" runbook
+## The Curated "don't re-investigate this" Runbook
 
 Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded — this is the difference between a SOC that actually scales past a handful of hosts and one that drowns in its own alert volume. A representative slice of what's actually in it, real rule IDs and all:
 
@@ -71,7 +80,7 @@ Rather than re-triaging the same recurring alert from scratch every time it fire
 > [!TIP]
 > **One catch worth calling out in detail**: rule `19011`, "nftables default deny not detected," turned out to be a false positive *in the check itself*, not a real gap. The check greps for a raw nftables `policy drop` string that firewalld — the actual firewall in use, per [`docs/firewall-and-network.md`](firewall-and-network.md) — simply never produces, because firewalld implements default-deny through its own zone/chain structure instead. The fix was verifying the real firewall posture directly against firewalld's own state, not touching the firewall itself. This is a useful pattern in general: a security tool's own check can be wrong about a system it wasn't originally written for, and the fix is understanding the actual system, not chasing the tool's assumption.
 
-## Querying and testing directly
+## Querying And Testing Directly
 
 Alert data lives in a searchable OpenSearch-based index, reachable directly without going through the [Dashy SOC Board](dashy-soc-board.md) — a terms-aggregation on `rule.id`, with sub-aggregations for `rule.description`, max `rule.level`, `agent.name`, and max `timestamp`, returns a fleet-wide "what's actually firing and where" summary in one call — useful when the dashboard's own pre-aggregated view isn't granular enough for a specific question.
 
@@ -100,7 +109,7 @@ $ echo 'Aug 28 12:00:00 caddy sshd[12345]: Failed password for invalid user test
 
 One rule-writing gotcha that's easy to get backwards: a rule's `<decoded_as>` field has to reference the *parent* decoder's name, never a child regex-decoder further down the same chain — `wazuh-logtest` always reports the parent's name in its output regardless of which child actually matched, so debugging by what the tool prints back leads to the wrong name. And `wazuh-logtest` re-reads the rule/decoder files fresh on every invocation, independent of whether the live daemon has actually reloaded them — a rule that tests correctly can still be inert in production until `wazuh-control restart` actually loads it.
 
-## Compliance hardening: justified exceptions, not blanket enforcement
+## Compliance Hardening: justified exceptions, not blanket enforcement
 
 ![A CIS-oriented hardening framework of 9 tagged Ansible sub-roles applies cleanly to most hosts and gets enforced fleet-wide, re-applied every compliance run — but two controls are deliberately skipped with the reason documented: kernel forwarding flipped for Docker hosts since container networking needs it, and SSH TCP forwarding kept permissive fleet-wide since real remote access depends on it](../assets/security-monitoring-compliance-gate.svg)
 
@@ -110,10 +119,10 @@ This is the same tag-driven selection model described in [`docs/iac-pipeline.md`
 
 The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards under pressure while writing the config.
 
-## Automated benchmarking: native, not containerized
+## Automated Benchmarking: native, not containerized
 
 > [!IMPORTANT]
 > The fleet's automated security-benchmark tool (Docker Bench for Security) runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations (glibc vs. musl) that made it fail to run reliably. Running it natively instead of debugging a broken official container was the pragmatic call: the tool's job is auditing Docker, not testing container-runtime compatibility edge cases. It runs weekly, deliberately timed against real observed cron load from the other scheduled jobs described in [`docs/cron-scheduler.md`](cron-scheduler.md), rather than an arbitrary default schedule that might collide with heavier jobs.
 
 ---
-Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md), [`docs/notifications.md`](notifications.md), [`docs/iac-pipeline.md`](iac-pipeline.md) · Stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)
+Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md), [`docs/notifications.md`](notifications.md), [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/netbird-zero-trust.md`](netbird-zero-trust.md) · Stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)

docs/sso-and-identity.md

@@ -1,9 +1,16 @@
+<!-- verified: 2026-09-18
+     rocky-docker-lxc@d225776 : caddy/conf/Caddyfile
+     caddy-vm@c5c3420 : caddy/conf/Caddyfile
+     iac@ae315de : ansible/roles/bootstrap_roles/docker_git_setup
+     trilium : baWhWTUejvY6, hWiIGZIlvQiN, nAPJq9A1iuX8
+-->
+
 # 🪪 SSO & Identity
 
 > [!NOTE]
 > Authentik is the identity provider for this fleet — self-hosted rather than an external IdP, for the same reason Vault is self-hosted in [`docs/secrets-and-pki.md`](secrets-and-pki.md): internal-only apps shouldn't need to trust a third party just to authenticate a login, and the two-zone reverse-proxy model in [`docs/dns-naming.md`](dns-naming.md) only works if the identity provider itself lives inside that same trust boundary.
 
-## Why centralize authentication at all
+## Why Centralize Authentication At All
 
 A homelab with dozens of self-hosted apps means dozens of different auth implementations if every app handles its own login — wildly inconsistent quality, no shared MFA, and revoking access means hunting down credentials app by app instead of doing it in one place. Centralizing on one identity provider fixes all three: every app's login gets the same enforcement regardless of how good or bad that app's own native auth actually is, and killing access anywhere starts with one action, not a checklist.
 
@@ -54,7 +61,7 @@ This is the actual live query against Authentik's API, not a paraphrase of what'
 ```
 $ curl -s -H "Authorization: Bearer [REDACTED]" \
   "https://authentik.[service-to-service zone]/api/v3/providers/proxy/?page_size=1" | jq '.pagination.count'
-96
+94
 $ curl -s -H "Authorization: Bearer [REDACTED]" \
   "https://authentik.[service-to-service zone]/api/v3/providers/proxy/?page_size=200" | jq -r '.results[].name' | grep -E '^proxmox'
 [no output — Proxmox has no Proxy Provider at all]
@@ -63,11 +70,11 @@ $ curl -s -H "Authorization: Bearer [REDACTED]" \
 > [!TIP]
 > **Removal is a real, deliberate step, not an afterthought.** Decommissioning an app means deleting the Application (by slug), then the Proxy Provider, then detaching that provider from the outpost's own provider list, then removing its icon file — in that order. Skipping the outpost-detach step is the easy mistake: the Application and Provider both look gone, but the outpost still references a provider that no longer resolves to anything, and that orphaned reference doesn't surface as an error anywhere obvious.
 
-## Dual-zone registration
+## Dual-zone Registration
 
 Every new app gets a Proxy Provider + Application pair registered per reverse-proxy zone it's actually exposed on — up to two registrations for the same app, one per zone from [`docs/dns-naming.md`](dns-naming.md), driven by actual need rather than registering every app on both zones by default.
 
-## OIDC login: the redirect crosses zones, every time
+## OIDC Login: the redirect crosses zones, every time
 
 For an app using native OIDC, the login always follows the same path:
 
@@ -83,13 +90,25 @@ So an app's OIDC discovery/authority setting always points at the service-to-ser
 > [!NOTE]
 > **Not every app can take the OIDC path above.** Some — Jellyfin's auth plugin among them — only support LDAP: search the directory, then bind as whatever matched. Authentik's own LDAP outpost hides superuser accounts from that search by design, so the account that matters most can't authenticate this way for an app that requires it. The fix wasn't a second account or demoting the superuser — it was skipping LDAP for that app and reusing the same header-based forward-auth pattern the proxy-provider apps already use.
 
-## Git access: least privilege, not the admin account
+## Git Access: least privilege, not the admin account
 
-Automation should never run with more privilege than it needs. CI and orchestration pushes (the flows described in [`docs/orchestration.md`](orchestration.md)) run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. This wasn't the original setup: automated pushes used to run through the personal admin account used for interactive access, the same one with full admin rights over the Git server. That got fixed by standing up a separate account with its own SSH keypair and adding it as a write collaborator on every repo it touches — routine automated commits no longer run under an admin credential at all. Hosts that only need to *read* get separate read-only deploy keys instead of any shared credential. The practical payoff: if an automation credential ever leaks, the blast radius is "can push to specific repos," not "has full admin control of the Git server."
+Automation should never run with more privilege than it needs. CI and orchestration pushes (the flows described in [`docs/orchestration.md`](orchestration.md)) run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. This wasn't the original setup: automated pushes used to run through the personal admin account used for interactive access, the same one with full admin rights over the Git server. That got fixed by standing up a separate account with its own SSH keypair and adding it as a write collaborator on every repo it touches — routine automated commits no longer run under an admin credential at all. The 5 docker hosts that only ever pull (never push) use that same bot account's key too — copied identically to each host at first provisioning, not a separate credential per host. The practical payoff: if an automation credential ever leaks, the blast radius is "can push to the repos this bot account touches," not "has full admin control of the Git server."
 
 ![Gitea repo collaborators: automation-bot with Write access, not an admin account](../assets/gitea-collaborators-automation-bot.png)
 
-![Gitea repo deploy keys: a host-specific key scoped to Read only](../assets/gitea-deploy-keys.png)
+This is the real provisioning task, not a description of it — the automation-bot key gets copied to every docker host identically:
+
+```
+# The automation-bot private key - authenticates the push to Gitea AND signs
+# commits (ssh_key_path == ssh_signing_key_path). Copied from the control node.
+- name: Install the automation-bot git key
+  copy:
+    src: "{{ ssh_key_path }}"
+    dest: "{{ ssh_key_path }}"
+    owner: "{{ git_repo_owner }}"
+    group: "{{ git_repo_group }}"
+    mode: "0600"
+```
 
 ---
 Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md) · Stacks: [`stacks/identity-and-security/authentik/`](../stacks/identity-and-security/authentik/), [`stacks/platform/gitea/`](../stacks/platform/gitea/)

stacks/platform/caddy-internal/docker-compose.yaml

@@ -6,11 +6,11 @@ services:
     ports:
       - "${HTTPS_PORT}:443"
     volumes:
-      - ./Caddyfile:/etc/caddy/Caddyfile
+      - ./conf:/etc/caddy:ro
       - /path/to/caddy-internal/caddy_data:/data
       - /path/to/caddy-internal/caddy_config:/config
       - /path/to/caddy-internal/logs:/var/log/caddy
-      - ./certs/pki_int_ca.crt:/etc/caddy/certs/pki_int_ca.crt:ro
+      - ./certs:/etc/caddy/certs:ro
     networks:
       - caddy-net
 

stacks/platform/caddy/docker-compose.yaml

@@ -6,7 +6,7 @@ services:
     ports:
       - "${HTTPS_PORT}:443"
     volumes:
-      - ./Caddyfile:/etc/caddy/Caddyfile
+      - ./conf:/etc/caddy:ro
       - /path/to/caddy/caddy_data:/data
       - /path/to/caddy/caddy_config:/config
       - /etc/caddy/certs:/etc/caddy/certs

stacks/platform/node-red/.env.secrets.ctmpl

@@ -15,7 +15,6 @@ SEERR_KEY={{ .Data.data.seerr_key }}
 SLACK_TOKEN={{ .Data.data.slack_token }}
 SONARR_KEY={{ .Data.data.sonarr_key }}
 TECHNITIUM_TOKEN={{ .Data.data.technitium_token }}
-VAULT_TOKEN={{ .Data.data.vault_token }}
 WAZUH_API_PASS={{ .Data.data.wazuh_api_pass }}
 WAZUH_INDEXER_AUTH={{ .Data.data.wazuh_indexer_auth }}
 {{ end }}