README: ground every Systems bullet in verified, sourced detail

Fixed one real inaccuracy (Gitea registry scope, Secret Distribution
path count) and replaced generic descriptions with true, specific
detail across all 14 Systems bullets - real incidents, real design
decisions, concrete numbers - each checked against Trilium and, where
relevant, the live Ansible roles rather than trusted at face value.

Also stages a verified-but-unintegrated PAM hardening finding in
docs/firewall-and-network.md for that file's own upcoming audit pass.

README.md

@@ -27,7 +27,7 @@ I'm a curious, self-taught technologist who runs a real, multi-host homelab as a
 - <strong style="font-size:1.4em;">HashiCorp Vault</strong> — issuing PKI certs and managing every application secret
 - <strong style="font-size:1.4em;">Caddy + Authentik</strong> — a two-zone reverse-proxy design fronted by SSO, behind pfSense and firewalld for real network segmentation
 - <strong style="font-size:1.4em;">Kestra</strong> — fleet orchestration with scheduled flows backed by Ansible roles and playbooks
-- <strong style="font-size:1.4em;">Gitea</strong> — a self-hosted registry building and shipping every container image in-fleet
+- <strong style="font-size:1.4em;">Gitea</strong> — a self-hosted registry building and shipping container images for every stack that's opted in
 - <strong style="font-size:1.4em;">Wazuh + Dashy</strong> — a self-built SOC watching all of it
 
 ## The Decision That Shows The Most
@@ -86,59 +86,59 @@ Solid arrows are hard dependencies — the target has to be reachable for the so
 ## Systems
 
 ### 🔐 Secrets & PKI
-A self-hosted Vault cluster is both the certificate authority and the secret store for every host in the fleet — every TLS certificate, every database password, every API key traces back to this one system. Nothing reaches Vault directly; a local agent on each host authenticates itself and renders what that host needs to disk. Replaced an earlier setup where secrets sat in plaintext inside git.
+A self-hosted Vault cluster is both the certificate authority and the secret store for every host in the fleet — every TLS certificate, every database password, every API key traces back to this one system. Nothing reaches Vault directly; a local agent on each host authenticates itself and renders what that host needs to disk. Replaced an earlier setup where secrets sat in plaintext inside git. A DNS resolver's cert once renewed out of sync and ran expired for 19 days before anyone noticed — the fix wasn't tighter monitoring, it was making that renewal step structurally unable to skip itself again.
 → [docs/secrets-and-pki.md](docs/secrets-and-pki.md)
 
 ### 🔑 Secret Distribution Model
-Four separate delivery paths move material out of Vault, purpose-built for what's actually being delivered — a short-lived certificate, an application secret, a full config file, or an admin-only credential — each with its own contract for what the rest of the fleet does if that specific path fails.
+Three separate delivery paths move material out of Vault, purpose-built for what's actually being delivered — a short-lived certificate, an application secret, or a full config file — each with its own contract for what the rest of the fleet does if that specific path fails. Admin and tooling credentials deliberately live in a fourth, completely separate system instead — encrypted directly in git — since they're used by playbooks and operators, not consumed by a running service.
 → [docs/secret-distribution.md](docs/secret-distribution.md)
 
 ### 🪪 SSO & Identity
-Authentik, self-hosted, fronts single sign-on fleet-wide — native OIDC where an app supports it, proxy-based forward-auth in front of everything else, so apps that were never built with SSO in mind still get it. Every OIDC login crosses reverse-proxy zones mid-handshake, and Git automation runs on a scoped bot account rather than an admin identity.
+Authentik, self-hosted, fronts single sign-on fleet-wide — native OIDC where an app supports it, proxy-based forward-auth covering everything else, so apps that were never built with SSO in mind still get it. Every OIDC login crosses reverse-proxy zones mid-handshake, and Git automation runs on a scoped bot account rather than an admin identity.
 → [docs/sso-and-identity.md](docs/sso-and-identity.md)
 
 ### 🧱 Firewall & Network Segmentation
-No blanket internal trust anywhere on the network — every real service-to-service flow gets its own explicit rule, and everything not explicitly allowed is logged and dropped by default. Enforced at two layers: a firewall at the network edge, and firewalld locally on every host, each opening only what that specific host actually needs.
+No blanket internal trust anywhere on the network — every real service-to-service flow gets its own explicit rule, and everything not explicitly allowed is logged and dropped by default. Enforced at two layers: a firewall at the network edge, and firewalld locally on every host, each opening only what that specific host actually needs. A few rules go narrower still — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet.
 → [docs/firewall-and-network.md](docs/firewall-and-network.md)
 
 ### 🌐 DNS
-Two internal DNS zones split browser-facing traffic from service-to-service traffic — which zone a hostname resolves in is what decides which trust boundary a request enters, not just where it's found. Technitium runs both zones, self-hosted, with direct DNS bypass blocked at the firewall so the split is actually enforced, not just a naming convention.
+Two internal DNS zones split browser-facing traffic from service-to-service traffic — which zone a hostname resolves in is what decides which trust boundary a request enters, not just where it's found. Technitium runs both zones, self-hosted, with direct DNS bypass blocked at the firewall so the split is actually enforced, not just a naming convention. Not every app gets a record in both zones, either: something that's purely a backend service other apps talk to only needs the app-to-app zone, and something purely browser-facing only needs the user-facing one — each app gets exactly the DNS records its real traffic pattern uses.
 → [docs/dns-naming.md](docs/dns-naming.md)
 
 ### 📈 Observability
-Every fleet host runs a native metrics-and-logs agent that pushes outward to a central Loki/Prometheus stack, rather than waiting to be scraped — chosen specifically so monitoring never needs inbound access opened through the firewall model to every host it watches. Logs, container output, and DNS query history all land in one searchable place.
+Every fleet host runs a native metrics-and-logs agent that pushes outward to a central Loki/Prometheus stack, rather than waiting to be scraped — chosen specifically so monitoring never needs inbound access opened through the firewall model to every host it watches. Logs, container output, and DNS query history all land in one searchable place, and a handful of apps go further still, exposing their own internal metrics directly into the same pipeline instead of relying on host-level visibility alone.
 → [docs/observability.md](docs/observability.md)
 
 ### 🛡️ Security Monitoring
-A Wazuh agent runs on every host in the fleet, continuously watching file integrity, rootkit behavior, and compliance drift against a CIS-based benchmark. A curated runbook root-causes routine alerts once so they're never re-triaged from scratch, and hardening controls apply fleet-wide with specific, reasoned exceptions rather than uniform enforcement regardless of fit.
+A Wazuh agent runs on every host in the fleet, continuously watching file integrity, rootkit behavior, and compliance drift against a CIS-based benchmark. A curated runbook root-causes routine alerts once so they're never re-triaged from scratch, and hardening controls apply fleet-wide with specific, reasoned exceptions rather than uniform enforcement regardless of fit — audit logging, for one, is skipped on every LXC container in the fleet because the kernel capability it needs simply isn't available inside a container, not because it was overlooked.
 → [docs/security-monitoring.md](docs/security-monitoring.md)
 
 ### 🔁 Orchestration
-Kestra is the orchestration engine that runs the fleet's scheduled maintenance — OS patching, container updates, app updates, disk cleanup, and internal KB upkeep.
+Kestra is the orchestration engine that runs the fleet's scheduled maintenance — OS patching, container updates, app updates, disk cleanup, and internal KB upkeep. Most of it runs fully unattended, but updates to the secrets platform, DNS, and monitoring agent — the systems everything else depends on — wait for an explicit human approval before applying.
 → [docs/orchestration.md](docs/orchestration.md)
 
 ### 🔔 Notifications
-Every alert in the fleet — from apps, from hosts, from security events — funnels through one Node-RED instance and one shared formatter before reaching Slack, so a notification reads the same regardless of which of a dozen-plus sources triggered it. A dedicated role puts every host onto the same pipeline for OS-level health, independent of any single application.
+Every alert in the fleet — from apps, from hosts, from security events — funnels through one Node-RED instance and one shared formatter before reaching Slack, so a notification reads the same regardless of which of a dozen-plus sources triggered it. A dedicated role puts every host onto the same pipeline for OS-level health, independent of any single application. One shared endpoint handles every app's webhook regardless of shape — a source field where an app provides one, structural fingerprinting of the payload where it doesn't — instead of standing up a dedicated integration per app.
 → [docs/notifications.md](docs/notifications.md)
 
 ### 📦 Self-Hosted Registry & Build Pipeline
-Docker images get built and hosted entirely in-fleet on the Git server's own built-in registry — no runtime dependency on Docker Hub or ghcr for the stacks that have opted in. A weekly scheduled pass checks upstream image digests before committing to a full rebuild, so nothing gets rebuilt unless something upstream actually changed.
+Docker images get built and hosted entirely in-fleet on the Git server's own built-in registry — no runtime dependency on Docker Hub or ghcr for the majority of stacks, which have opted in. A weekly scheduled pass checks upstream image digests before committing to a full rebuild, so nothing gets rebuilt unless something upstream actually changed.
 → [docs/registry-and-build.md](docs/registry-and-build.md)
 
 ### 🖥️ Dashy SOC Board
-One custom-built dashboard aggregates fourteen different data sources — security alerts, logs, metrics, CI health, secrets state, DNS, backups, firewall stats — into a single glanceable board, replacing what used to be fourteen separate tabs to check. Acknowledgement state lives entirely in the browser, no backend to host or secure.
+One custom-built dashboard aggregates fourteen different data sources — security alerts, logs, metrics, CI health, secrets state, DNS, backups, firewall stats — into a single glanceable board, replacing what used to be fourteen separate tabs to check. Acknowledgement state lives entirely in the browser, no backend to host or secure. Every widget's visual shape matches the shape of its own data — a gauge for a proportion, a nested tree for parent/child hardware, a poster grid for media — instead of one generic card template stretched to fit everything.
 → [docs/dashy-soc-board.md](docs/dashy-soc-board.md)
 
 ### ⏱️ Cron Scheduler
-Every scheduled job across the fleet — orchestration flows, system timers, app-internal schedules — lives in one inventory, visualized as a load heatmap weighted by actual resource impact, so a new job's time slot can be picked without colliding with existing load.
+Every scheduled job across the fleet — orchestration flows, system timers, app-internal schedules — lives in one inventory, visualized as a load heatmap weighted by actual resource impact — a disk-heavy backup job counts for more than a quick health check, not just raw job count — so a new job's time slot can be picked without colliding with existing load.
 → [docs/cron-scheduler.md](docs/cron-scheduler.md)
 
 ### 🏗️ Infrastructure as Code
-Packer builds a hardened base image, Terraform provisions each VM from it, and Ansible configures and continuously re-hardens every host after that — three tools, three separate responsibilities, connected by one tag that threads through all three stages instead of one tool trying to do every job.
+Packer builds a hardened base image, Terraform provisions each VM from it, and Ansible configures and continuously re-hardens every host after that — three tools, three separate responsibilities, connected by one tag that threads through all three stages instead of one tool trying to do every job. Terraform's part in that link ends at creation — it writes the tags file once, and that file then drives Ansible independently for the rest of that host's life.
 → [docs/iac-pipeline.md](docs/iac-pipeline.md)
 
 ### 🧯 Disaster Recovery & Upgrades
-Three independent recovery layers cover three different kinds of loss — a lost host rebuilds from code, VM-level backups cover full-host snapshots, and a separate volume-level backup covers application data specifically. OS-level and application-level upgrades run on entirely separate, deliberately decoupled schedules and safety checks.
+Three independent recovery layers cover three different kinds of loss — a lost host rebuilds from code, VM-level backups cover full-host snapshots, and a separate volume-level backup covers application data specifically. Failure-domain mapping goes down to the disk: one specific volume, if lost, would take out the boot images for two hosts at once — identified and factored into backup priority rather than discovered the hard way. OS-level and application-level upgrades run on entirely separate, deliberately decoupled schedules and safety checks.
 → [docs/disaster-recovery.md](docs/disaster-recovery.md)
 
 ## What's running

docs/firewall-and-network.md

@@ -61,5 +61,8 @@ The second reverse-proxy layer here is genuinely separate from the internal two-
 
 Proxmox has its own built-in firewall capability, left with an empty ruleset on purpose. pfSense and firewalld already cover the two layers that matter — public/internal segmentation and per-host minimal exposure — and a third overlapping layer at the hypervisor level would add complexity (a third ruleset to audit, a third place a rule could silently be wrong) without meaningfully improving posture. Recognizing when *not* to add a redundant control is as much a design decision as adding one, and it's the same reasoning that kept the secrets model in [`docs/secrets-and-pki.md`](secrets-and-pki.md) to one trust root instead of two.
 
+> [!NOTE]
+> **Staged for this file's full audit pass — verified true, not yet integrated into the prose above.** Confirmed directly against the live Ansible role, not just against documentation: on the fleet's Debian-family hosts, the PAM login stack is deliberately configured to stop immediately on a successful authentication instead of jumping to a numbered step — a defense against a future package silently inserting its own PAM line and shifting that jump target, which would misread a correct password as a failed one. Not theoretical: exactly that bug was caught live on one of those hosts before this fix went in. Rocky/RHEL-family hosts don't need the same fix — they manage their PAM stack through `authselect`, a different, declarative system not exposed to the same class of ordering bug. Fold this in wherever it fits once this file gets its real pass — likely near "Per-host firewalls" above, as another example of defense-in-depth below the network layer.
+
 ---
 Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/architecture-overview.md`](architecture-overview.md), [`docs/security-monitoring.md`](security-monitoring.md) · Stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/), [`stacks/platform/docker-socket-proxy/`](../stacks/platform/docker-socket-proxy/)