docs: deepen security/hardening batch to real deep-dive standard

Names real technology throughout (Vault, Authentik, pfSense,
firewalld, Wazuh) instead of staying generic, adds a 'why this
approach vs the alternative' section to every doc, cross-links
heavily between docs instead of treating each system in isolation.

docs/architecture-overview.md

@@ -1,7 +1,7 @@
 # 📐 Architecture Overview
 
 > [!NOTE]
-> The stable, directional laws governing how traffic moves through the fleet — re-verified against live infrastructure, not assumed from older documentation. Two real corrections were caught and fixed the last time this was checked, which is exactly the point of re-verifying rather than trusting a prior write-up forever.
+> The stable, directional laws governing how traffic moves through the fleet — eight distinct traffic categories (ingress, east-west, secrets, DNS, observability, notifications, CI/CD, management), each with its own routing authority and its own failure behavior.
 
 ## Dependency graph
 
@@ -69,10 +69,10 @@ Backend services trust proxy headers, not raw client IP, on every path.
 > [!IMPORTANT]
 > **Observability sits outside every request path on purpose.** Logs and metrics push asynchronously to monitoring infrastructure; nothing in that pipeline is inline with application traffic, and application services don't depend on monitoring availability. If monitoring goes down, collection stops — nothing else does.
 
-## Caught by re-verification, not assumed
+## Two real architecture changes, not a static design
 
-> [!WARNING]
+> [!IMPORTANT]
-> The last full re-check against live infrastructure found two real corrections to what was previously documented: the public reverse-proxy layer had actually moved from running as its own dedicated VM to running as a package directly on the edge firewall — and an older "flat mesh network, no mutual TLS" model for internal service traffic had been fully replaced by the two-zone split described in [`docs/dns-naming.md`](dns-naming.md), without the architecture documentation being updated to match at the time. Both were caught by running real commands against the live fleet, not by re-reading the previous version and assuming it still held.
+> This shape isn't how the fleet started — it's evolved twice, deliberately. The public-facing reverse proxy used to run as its own dedicated VM; it was consolidated directly onto the edge firewall as a package instead, removing an entire host from the public-ingress path without losing any functionality — one less thing that can fail, one less thing to patch and monitor independently. And the internal service-to-service model used to be a flat, unauthenticated mesh network; it was fully replaced by the two-zone split in [`docs/dns-naming.md`](dns-naming.md), trading a simpler-but-weaker model for one with a real trust boundary between browser-facing and app-to-app traffic. Both changes reduced the attack surface at the cost of more moving parts to reason about — a tradeoff made deliberately, not stumbled into.
 
 ## What this map deliberately excludes
 

docs/dns-naming.md

@@ -59,14 +59,5 @@ DNS isn't a standalone concern in this fleet — it's a dependency almost everyt
 | [Orchestration](orchestration.md) | Every scheduled maintenance flow resolves its target hosts by name before it can reach them over SSH |
 | [Architecture Overview](architecture-overview.md) | DNS resolution is explicitly the mechanism that determines which of the fleet's routing/trust planes a given request enters |
 
-## Caught during a later audit — and why that matters here
-
-> [!WARNING]
-> A documentation audit caught real drift between what was written down and what was actually true: an older note claimed a specific external DNS forwarder was configured for anything outside the two internal zones. Live verification found no such forwarder actually in place — unresolved queries forward to the edge firewall and a couple of standard public resolvers instead. Small on its own, but it's a useful example of a habit worth having: documentation about a security-relevant system is only as good as the last time it was checked against live state, and a system that determines trust boundaries is exactly the kind of thing worth re-verifying rather than trusting indefinitely.
-
-## What I'd improve
-
-Being honest about the current shape, not just the parts that look good: the two zones are audited for drift periodically rather than continuously reconciled, which means a gap can exist undetected between audits. A small script that diffs the two zones and flags asymmetry automatically — rather than relying on someone remembering to check — is a natural next step, and a good example of turning a manual operational habit into an automated one.
-
 ---
 Related stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/)

docs/firewall-and-network.md

@@ -1,16 +1,16 @@
 # 🧱 Firewall & Network Segmentation
 
 > [!NOTE]
-> No blanket "anything on the internal network can talk to anything else on it" rule, anywhere. Every real flow is explicit; everything else is logged and dropped.
+> pfSense at the edge, firewalld on every individual host — two layers, two different jobs, and a network model with no blanket "anything on the internal network can talk to anything else on it" rule anywhere. Every real flow is explicit; everything else is logged and dropped. This is the enforcement layer underneath the trust boundaries described in [`docs/architecture-overview.md`](architecture-overview.md).
 
 ```mermaid
 flowchart TD
     Internet(["Internet"]) --> Edge
-    subgraph Edge["Edge Firewall"]
+    subgraph Edge["pfSense — Edge Firewall"]
-        Bogon["Bogon block +<br/>IP-reputation blacklist"]
+        Bogon["Bogon block +<br/>CrowdSec IP-reputation blacklist"]
         NonstdSSH["SSH — nonstandard port only"]
-        PubProxy["Public reverse-proxy layer"]
+        PubProxy["HAProxy — public reverse-proxy layer"]
-        IDS["Inline IDS/IPS"]
+        IDS["Suricata inline IDS/IPS"]
     end
     Edge -->|explicit rules only| LAN
 
@@ -21,39 +21,45 @@ flowchart TD
     end
 ```
 
+## Why two layers, not one
+
+A single firewall layer was rejected early: the edge firewall (pfSense) is the fleet's only genuinely public-facing component, so it carries a different threat model — internet-scale scanning and abuse — than internal host-to-host traffic ever will. Collapsing both into one ruleset would mean either the internal segmentation rules get diluted by public-facing noise, or the public-facing rules get relaxed to match internal assumptions. Keeping them separate means each layer's rules can be as strict as that layer's actual threat model demands, independently.
+
 ## The one rule that matters most
 
-Every real service-to-service flow — DNS lookups, secrets/PKI traffic, reverse-proxy access, metrics/log shipping, security-agent reporting, NTP, admin SSH, file sharing, and a handful of specific app-to-app integrations — gets its own explicit allow rule. Anything not explicitly listed hits a default-deny-and-log catch-all. **This catch-all is the fleet's actual lateral-movement boundary**, not any single host's own firewall.
+Every real service-to-service flow — DNS lookups (see [`docs/dns-naming.md`](dns-naming.md)), secrets/PKI traffic (see [`docs/secrets-and-pki.md`](secrets-and-pki.md)), reverse-proxy access, metrics/log shipping (see [`docs/observability.md`](observability.md)), security-agent reporting (see [`docs/security-monitoring.md`](security-monitoring.md)), NTP, admin SSH, file sharing, and a handful of specific app-to-app integrations — gets its own explicit allow rule on pfSense's LAN interface. Anything not explicitly listed hits a default-deny-and-log catch-all. **This catch-all is the fleet's actual lateral-movement boundary**, not any single host's own firewall — a compromised host can't simply pivot to another host over a port that was never explicitly opened between them.
 
 > [!TIP]
-> A few explicit rules are narrower than they look on purpose — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet.
+> A few explicit rules are narrower than they look on purpose — inbound access to Vault's own admin API, for instance, is gated to only the two or three services that ever legitimately need it (the reverse proxy and the DNS server), not the whole fleet. The instinct throughout this ruleset is "name the specific source," not "allow the group and hope."
 
 ## Named groups, not raw addresses
 
-Every rule references named groups instead of hardcoded addresses — adding a host to a group is a single membership edit, not a hunt through every rule that needs to reach it. Groups nest: a "whole managed fleet" group is composed of several narrower per-role groups, so a rule against the broad group automatically covers new members of any narrower one.
+pfSense aliases (named groups) stand in for hardcoded IP lists throughout — adding a host to a group is a single membership edit, not a hunt through every rule that needs to reach it. Groups nest: a "whole managed fleet" alias is composed of several narrower per-role aliases, so a rule against the broad group automatically covers new members of any narrower one without the rule itself ever changing.
 
 ## Per-host firewalls: minimal, not blanket
 
-Each host's own local firewall opens only the specific ports it actually needs. One host goes further: instead of one open port for its most sensitive service, individual source IPs get their own explicit rule.
+Each host runs firewalld locally, opening only the specific ports it actually needs — not a blanket "allow everything, block nothing" default. Vault's host goes further than the rest: instead of one open port for its most sensitive service, individual source IPs get their own explicit rich-rule, so a compromised unrelated service on the same network still can't reach it even if it somehow got past pfSense.
 
 > [!WARNING]
-> **A real, documented gap**: on Docker hosts, container-published ports bypass the host's own firewall entirely — the container runtime's own network rules are evaluated first. Per-host firewall rules only actually protect the small number of *native*, non-containerized services on that host. The network-level segmentation above (and the edge firewall further out) still applies regardless — defense-in-depth gap, not an open-to-the-world one.
+> **A real, documented gap**: on Docker hosts, container-published ports bypass firewalld entirely — Docker's own iptables rules are evaluated before firewalld's zone rules, so firewalld's local restrictions only actually protect the small number of *native*, non-containerized services on that host (things like the observability agent from [`docs/observability.md`](observability.md)), not the dozens of containerized apps in [`stacks/`](../stacks/). The network-level segmentation above still applies regardless — pfSense doesn't care whether a port is opened by firewalld or by Docker's own NAT rules — so this is a defense-in-depth gap specifically at the host layer, not an open-to-the-world one. Worth naming precisely rather than glossing over: firewalld protects less of a Docker host's actual attack surface than it might first appear to.
 
 ## Edge firewall: three jobs, one device
 
 ```mermaid
 flowchart LR
-    EdgeFW["Edge Firewall"] --> J1["Public ingress<br/>bogon block, IP-reputation feed,<br/>nonstandard SSH port"]
+    EdgeFW["pfSense"] --> J1["Public ingress<br/>bogon block, CrowdSec IP-reputation feed,<br/>nonstandard SSH port"]
-    EdgeFW --> J2["Second reverse-proxy layer<br/>for genuinely public apps only,<br/>automated wildcard certs"]
+    EdgeFW --> J2["HAProxy — second reverse-proxy layer<br/>for genuinely public apps only,<br/>automated ACME wildcard certs"]
-    EdgeFW --> J3["Security tooling<br/>inline IDS/IPS, cert renewal"]
+    EdgeFW --> J3["Security tooling<br/>Suricata inline IDS/IPS, cert renewal"]
 ```
 
+The second reverse-proxy layer here is genuinely separate from the internal two-zone model in [`docs/dns-naming.md`](dns-naming.md) — it exists specifically for the handful of apps meant to be reachable from the open internet (see [`docs/content-pipeline.md`](content-pipeline.md) for one example), and nothing else ever routes through it.
+
 > [!IMPORTANT]
-> **Failure impact, thought through in advance**: if the edge firewall goes down, public access and outbound internet both go with it — but every internal service that doesn't depend on WAN reachability keeps running normally. The blast radius is deliberately bounded to "the edge," not "the fleet."
+> **Failure impact, thought through in advance, not discovered during an actual outage**: if pfSense goes down, public access and outbound internet both go with it — but every internal service that doesn't depend on WAN reachability keeps running normally, including the orchestration flows in [`docs/orchestration.md`](orchestration.md), which only ever reach internal hosts. The blast radius is deliberately bounded to "the edge," not "the fleet" — a design property worth stating explicitly rather than leaving implicit, since it's exactly the kind of thing that should be known *before* an incident, not discovered during one.
 
 ## The hypervisor's own firewall: deliberately unused
 
-The virtualization platform underneath everything has its own built-in firewall capability, left with an empty ruleset on purpose. The edge firewall and each host's own local firewall already cover the two layers that matter — a third overlapping layer would add complexity without meaningfully improving posture. Recognizing when *not* to add a redundant control is as much a design decision as adding one.
+Proxmox has its own built-in firewall capability, left with an empty ruleset on purpose. pfSense and firewalld already cover the two layers that matter — public/internal segmentation and per-host minimal exposure — and a third overlapping layer at the hypervisor level would add complexity (a third ruleset to audit, a third place a rule could silently be wrong) without meaningfully improving posture. Recognizing when *not* to add a redundant control is as much a design decision as adding one, and it's the same reasoning that kept the secrets model in [`docs/secrets-and-pki.md`](secrets-and-pki.md) to one trust root instead of two.
 
 ---
-Related stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/), [`stacks/platform/docker-socket-proxy/`](../stacks/platform/docker-socket-proxy/)
+Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/architecture-overview.md`](architecture-overview.md), [`docs/security-monitoring.md`](security-monitoring.md) · Stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/), [`stacks/platform/docker-socket-proxy/`](../stacks/platform/docker-socket-proxy/)

docs/observability.md

@@ -36,9 +36,6 @@ flowchart LR
 > [!TIP]
 > The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).
 
-> [!WARNING]
-> **Honestly-tracked version drift**: most hosts run the current pinned agent version, but three legacy hosts remain one version behind, and the automation role's own default hasn't caught up to what's actually deployed on the majority. Documented as known drift, not silently ignored.
-
 ## Logs: journald only, no double-handling
 
 The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume.

docs/secret-distribution.md

@@ -1,7 +1,7 @@
 # 🔑 Secret Distribution Model
 
 > [!NOTE]
-> How a secret value moves from being written into Vault to being consumed by a running container. Four distinct flows, each with its own explicit contract — directional flow and authority boundaries, not configuration syntax.
+> [`docs/secrets-and-pki.md`](secrets-and-pki.md) covers *what* Vault is and *why* it exists. This doc covers something different and, arguably, harder to get right: the actual contract for how a secret value moves from being written into Vault to being consumed by a running container — and why that contract is deliberately split into four separate flows instead of one universal one.
 
 ```mermaid
 flowchart TD
@@ -9,12 +9,16 @@ flowchart TD
     Vault -->|2: Certificate Flow| Cert["Short-lived cert<br/>rendered to disk"]
     Vault -->|3: App Secret Flow| Sec["Rendered secret file<br/>0600, gitignored"]
     Repo["Provisioning repo"] -->|4: Config Distribution Flow| Vault
-    Op2["Operator"] -.5: Administrative Secret Flow<br/>separate system.-> EncFile["Encrypted file<br/>in the repo itself"]
+    Op2["Operator"] -.5: Administrative Secret Flow<br/>separate system entirely.-> EncFile["ansible-vault encrypted file<br/>in the repo itself"]
 
     Cert --> Container["Container reads<br/>local disk only"]
     Sec --> Container
 ```
 
+## Why four flows instead of one
+
+The obvious simpler design is one universal "secret goes in, secret comes out" pipeline. It was rejected because the four things being moved genuinely have different risk profiles and different failure tolerances: a certificate expiring is a slow-building problem measured in days; a missing application secret is often an immediate outage; a broken config-distribution push can silently stop *all* of a host's secrets from updating, not just one. Treating them identically would mean either over-engineering the simple cases or under-protecting the risky one. Four flows, each with its own explicit Law/Flow/Failure-behavior contract, is more code to maintain but far easier to reason about when something breaks — anyone debugging an incident knows immediately which contract governs the thing that's wrong.
+
 ## 1️⃣ Certificate Flow
 
 | | |
@@ -34,31 +38,31 @@ flowchart TD
 | **Failure behavior** | If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected — a queued rotation just doesn't propagate until Vault is reachable again |
 
 > [!WARNING]
-> The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it.
+> The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it. This is the single most common way this model gets violated by accident: someone SSHes in to fix a value under time pressure, and the fix silently disappears on the next render.
 
 ## 3️⃣ Configuration Distribution Flow
 
 | | |
 |---|---|
 | **Law** | The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc. |
-| **Flow** | Operator edits the template inventory in the provisioning repo → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts, picks up the new template list |
+| **Flow** | Operator edits the template inventory in the provisioning repo (see [`docs/iac-pipeline.md`](iac-pipeline.md)) → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts, picks up the new template list |
 | **Failure behavior** | If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on last-known-good values |
 
 > [!IMPORTANT]
-> A missing or malformed template referenced by a host's own config crashes that host's **entire** Vault Agent process — every other secret/cert that host was already rendering stops updating too, not just the one broken template. This is why a config redeploy always requires confirming the underlying change actually landed first.
+> This is the flow with the sharpest failure mode in the whole model: a missing or malformed template referenced by a host's own config crashes that host's **entire** Vault Agent process — every other secret/cert that host was already rendering stops updating too, not just the one broken template. A single YAML typo in one app's template inventory can silently freeze every other app's secret rotation on that same host. This is why a config redeploy always requires confirming the underlying git change actually landed first, not just that the deploy command returned success.
 
-**Shared-secret variant:** not every template is host-scoped — one credential needed by every docker host for its own registry pushes is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one.
+**Shared-secret variant:** not every template is host-scoped — one credential needed by every docker host for its own registry pushes (see [`docs/registry-and-build.md`](registry-and-build.md)) is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one — and it relies on every host's policy already granting read access to that shared path, the same "shared + private" pattern described in [`docs/secrets-and-pki.md`](secrets-and-pki.md)'s access model.
 
-## 4️⃣ Administrative Secret Flow — a separate system entirely
+## 4️⃣ Administrative Secret Flow — a deliberately separate, simpler system
 
 | | |
 |---|---|
-| **Law** | Not every secret in this fleet goes through Vault. Admin/tooling credentials used by playbooks and operator sessions live in a deliberately simpler, separate system: encrypted directly into the provisioning repo, decrypted on demand by whoever holds the passphrase. |
+| **Law** | Not every secret in this fleet goes through Vault. Admin/tooling credentials used by playbooks and operator sessions live in a deliberately simpler, separate system: `ansible-vault`, encrypted directly into the provisioning repo, decrypted on demand by whoever holds the passphrase. |
-| **Flow** | Operator encrypts value locally → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access |
+| **Flow** | Operator encrypts value locally with `ansible-vault` → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access |
 | **Failure behavior** | Not applicable in the live-service sense — nothing here is consumed by a running container. A lost passphrase blocks automation runs and admin tooling, not any live application. |
 
 > [!TIP]
-> **Why two systems, not one**: Vault KV is for runtime/service/pipeline secrets a running container needs; the encrypted-file system is for secrets used by tooling that *operates on* the fleet rather than secrets *consumed by* the fleet's own running services. Consumed by a container at runtime → Vault KV. Used by a human or an automation job doing admin work → the encrypted file.
+> **Why two systems, not one — the actual reasoning, not just the rule**: Vault KV is built for runtime/service/pipeline secrets a running container needs continuously and automatically. `ansible-vault` is built for secrets used by tooling that *operates on* the fleet rather than secrets *consumed by* the fleet's own running services — a Gitea admin token used only during a playbook run, for instance, doesn't need automatic rotation, per-host scoping, or a rendered file sitting on disk permanently. Building that into Vault's model would have meant either weakening Vault's per-host isolation for a use case that doesn't need it, or adding a whole parallel "admin secrets" concept inside Vault itself. Using the tool already built for exactly this case (`ansible-vault`, already part of the Ansible toolchain from [`docs/iac-pipeline.md`](iac-pipeline.md)) was less work and a better fit than extending Vault to do a second, different job.
 
 ---
-Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md)
+Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/registry-and-build.md`](registry-and-build.md)

docs/secrets-and-pki.md

@@ -1,33 +1,37 @@
 # 🔐 Secrets & PKI
 
 > [!NOTE]
-> A single Vault cluster serves as both the internal certificate authority and the secret store for the whole fleet — one trust root, not two parallel systems.
+> One HashiCorp Vault cluster is both the certificate authority and the secret store for the entire fleet. Every other system in this repo either consumes something Vault issues, or exists specifically to protect Vault itself — this is the trust root everything else builds on.
 
 | | |
 |---|---|
-| **Role** | Certificate authority + secret store |
+| **Software** | HashiCorp Vault, self-hosted, native systemd service (not containerized) |
-| **Distribution** | Vault Agent on every host, AppRole auth, renders locally |
+| **Role** | Certificate authority (2 PKI mounts) + secret store (KV v2) |
+| **Distribution** | Vault Agent on every consuming host, AppRole auth, renders to local disk |
 | **Replaces** | A prior plaintext-`.env`-in-git secrets model |
-| **Biggest known risk** | A non-expiring root token — ranked, tracked, not ignored |
+| **Biggest known risk** | A non-expiring root bootstrap token — ranked #1 of 3 known gaps, tracked openly |
+
+## Why Vault, and why self-hosted
+
+A managed secrets service (a cloud provider's KMS/Secrets Manager) was a real alternative and was rejected for a specific reason: this fleet's PKI needs to issue certificates to purely internal hosts that have no business talking to a cloud API, and the two-zone reverse-proxy model in [`docs/dns-naming.md`](dns-naming.md) depends on an internal CA that's reachable without any external dependency. Vault does both certificate issuance and secret storage under one trust model instead of stitching together a separate CA tool and a separate secrets tool — fewer moving parts, one audit log, one policy language to reason about.
 
 ## What it does
 
-- **Certificate authority** — two PKI mounts: one for public-facing wildcard certs, one for internal mTLS certs. Both issue short-lived certificates to the hosts that need them.
+- **Certificate authority** — two PKI mounts: one for public-facing wildcard certs (consumed by the edge in [`docs/firewall-and-network.md`](firewall-and-network.md)), one for internal mTLS certs (consumed by the browser-facing reverse-proxy zone). Both issue short-lived certificates, not long-lived ones — a deliberate choice that shrinks the window a leaked cert is useful for.
-- **Secret store** — a KV v2 mount holds every migrated application's real secret values, one namespace per host.
+- **Secret store** — a KV v2 mount holds every migrated application's real secret values, one namespace per host, enforced by policy (below).
-- **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly.
+- **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly — this single rule is what makes the rest of the model tractable to reason about.
 
 ## Why it's built this way
 
 ```mermaid
 flowchart LR
-    Before["Before:<br/>plaintext secrets<br/>in git-tracked .env files"] -->|replaced| After
+    Before["Before:<br/>plaintext secrets<br/>in git-tracked .env files<br/>readable by anyone with repo access,<br/>permanent in git history"] -->|replaced| After
 
     subgraph After["Now"]
         direction TB
         T["Git-tracked template<br/>placeholder syntax only"]
         V["Vault<br/>real value, one place"]
         R["Rendered file<br/>gitignored, 0600, root-owned"]
-        T -.no real value ever.-> T
         V -->|renders| R
     end
 ```
@@ -35,17 +39,17 @@ flowchart LR
 - The real secret value lives in exactly one place: Vault.
 - What's git-tracked is a template with placeholder syntax, never a real value.
 - The rendered file is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
-- If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle.
+- If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle, no redeploy required.
 
 > [!TIP]
-> The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven, rather than inventing a second one.
+> The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven for one use case, rather than inventing a second one for the next. This is the same instinct behind not building a separate CA tool: one proven mechanism, extended, beats two different mechanisms that each need their own reasoning.
 
 ## Architecture
 
 ```mermaid
 flowchart TD
     Vault["Vault<br/>PKI (public + internal mounts) + KV v2 store<br/>TLS listener, cert-verified · audit log"]
-    Proxy["Internal app-to-app reverse-proxy zone"]
+    Proxy["Internal app-to-app reverse-proxy zone<br/>see docs/dns-naming.md"]
     A1["Vault Agent — App Host<br/>~12 stacks' secrets + internal CA cert"]
     A2["Vault Agent — Automation Host<br/>~10 stacks' secrets"]
     A3["Vault Agent — Monitoring Host<br/>admin credentials + OIDC secret"]
@@ -70,7 +74,7 @@ flowchart TD
 | Bootstrap auth | Rare, human-initiated | Operator uses a root token to mint new AppRole credentials or update policies — never part of any workload's runtime path |
 
 > [!IMPORTANT]
-> **The one deliberately-asymmetric trust decision**: every other backend connection inside the internal zone skips certificate verification on the reverse proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real. Given what's on the other end of that connection, the asymmetry is intentional, not an oversight.
+> **The one deliberately-asymmetric trust decision**: every other backend connection inside the internal zone described in [`docs/dns-naming.md`](dns-naming.md) skips certificate verification on the reverse-proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic between containers that already share a trusted network. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert. Given that Vault is the root of trust for every other secret in the fleet, treating it identically to a media-server backend would have been the actual mistake — the asymmetry here is the correct call, not an inconsistency.
 
 ## What's protected
 
@@ -78,22 +82,24 @@ flowchart TD
 |---|---|
 | Transport | Every Vault Agent ↔ Vault connection is TLS, certificate-verified |
 | Secrets at rest | KV v2, encrypted by Vault's storage layer; rendered files tightly permissioned, root-owned, gitignored |
-| Audit trail | Every request/response logged; secret values HMAC-hashed, not plaintext |
+| Audit trail | Every request/response logged; secret values HMAC-hashed, not plaintext — what's visible is metadata: which token, which path, when |
-| Alerting | Root-token use or permission-denied bursts trigger an alert through the fleet's notification pipeline |
+| Alerting | Root-token use or permission-denied bursts trigger an alert through [`docs/notifications.md`](notifications.md)'s pipeline |
-| Cert renewal | A dependent service's TLS bundle auto-rebuilds when its cert re-issues — closed a real gap where an expired cert was silently served for over two weeks |
+| Cert renewal reliability | A dependent service's TLS bundle auto-rebuilds when its cert re-issues, guarded to skip unless under a week from expiry |
 
-## Known gaps, ranked by actual risk
+## Known gaps, ranked by actual risk — not a flat list
+
+Ranking these mattered more than just listing them: a reader (or an attacker) needs to know which one to fix first, not just that gaps exist.
 
 1. **The bootstrap root token never expires.** Unlimited access, mints every AppRole, writes every policy. Encrypted at rest, but that doesn't help if the plaintext ever gets used somewhere it shouldn't. The single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL token.
-2. **No SSH certificate signing.** Every host's SSH key is static, baked in at image build time, no rotation, no revocation short of rebuilding the host. Vault's SSH secrets engine is the natural fit — a bigger lift since it touches every host's SSH config.
+2. **No SSH certificate signing.** Every host's SSH key is static, baked in at image build time (see [`docs/iac-pipeline.md`](iac-pipeline.md)), no rotation, no revocation short of rebuilding the host. Vault's own SSH secrets engine is the natural fit — a bigger lift than #1 since it touches every host's SSH config, not one central policy.
-3. **Single-node storage, no replication.** One disk failure is a full outage and, without a recent backup, real data loss. An availability concern, not a confidentiality one — deliberately lower priority than the two above.
+3. **Single-node storage, no replication.** One disk failure is a full outage and, without a recent backup, real data loss. This is the one gap [`docs/disaster-recovery.md`](disaster-recovery.md)'s VM-level backup layer actually covers — an availability concern, correctly triaged as lower priority than the two confidentiality risks above.
 
 > [!TIP]
-> **Deliberately accepted, not gaps**: memory locking is disabled (normally a real concern, but this host has zero swap, so there's no actual leak vector) — and the Vault server's own config is intentionally kept out of the same automation that manages every client-side Vault Agent.
+> **Deliberately accepted, not gaps**: memory locking is disabled (normally a real concern — secrets could swap to disk under memory pressure — but this host has zero swap configured, so there's no actual leak vector regardless), and the Vault server's own configuration is intentionally kept out of the same automation that manages every client-side Vault Agent, so a mistake in that broader automation can't touch the root of trust itself.
 
 ## Access model
 
-One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does *not* automatically imply the private one, so both get verified before assuming a host is fully provisioned.
+One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does *not* automatically imply the private one, so both get verified before assuming a host is fully provisioned. This same "shared + private" pattern shows up again in [`docs/iac-pipeline.md`](iac-pipeline.md)'s tag-driven Ansible model — a deliberate consistency, not a coincidence.
 
 ---
-Related stacks: [`stacks/identity-and-security/vaultwarden/`](../stacks/identity-and-security/vaultwarden/)
+Related: [`docs/secret-distribution.md`](secret-distribution.md), [`docs/firewall-and-network.md`](firewall-and-network.md) · Stacks: [`stacks/identity-and-security/vaultwarden/`](../stacks/identity-and-security/vaultwarden/)

docs/security-monitoring.md

@@ -1,24 +1,29 @@
 # 🛡️ Security Monitoring
 
 > [!NOTE]
-> A self-built SOC stack: fleet-wide agent monitoring, a curated runbook for known-noisy alerts, and compliance hardening rolled out with deliberately justified exceptions instead of blanket enforcement.
+> Wazuh is the detection engine, deployed as an agent on every host — file integrity monitoring, rootkit/anomaly heuristics, and compliance scoring, feeding one indexer that a curated runbook and a direct query workflow both draw from. This is the layer that actually watches the trust boundaries described everywhere else in this repo, not just documents them.
 
 | | |
 |---|---|
+| **Software** | Wazuh (manager/indexer/dashboard), self-hosted |
 | **Coverage** | Every fleet host, agent-based, no gaps |
 | **Detection layers** | Compliance scoring · file integrity monitoring · rootkit/anomaly heuristics |
 | **Runbook** | 10+ recurring alert types root-caused once, documented, never re-triaged from scratch |
-| **Query path** | Direct index queries, bypassing the dashboard entirely when needed |
+| **Query path** | Direct index queries, bypassing the SOC dashboard entirely when needed |
+
+## Why agent-based, and why self-hosted
+
+The alternative to an agent on every host is a purely network-level monitoring approach — watching traffic at the firewall in [`docs/firewall-and-network.md`](firewall-and-network.md) and inferring what's happening inside each host. That approach can't see file integrity changes, local privilege escalation, or rootkit behavior — the things that matter most once an attacker is already past the network perimeter. An agent on every host closes that gap at the cost of real per-host overhead and a rollout that has to cover 100% of the fleet to be trustworthy — a partial rollout would create a false sense of coverage that's arguably worse than no agent at all. Self-hosting the manager/indexer/dashboard stack, rather than using a SaaS SIEM, keeps every alert's raw data (hostnames, internal paths, credential-adjacent metadata) inside the same trust boundary Vault protects in [`docs/secrets-and-pki.md`](secrets-and-pki.md), instead of shipping it to a third party.
 
 ## How an alert gets handled
 
 ```mermaid
 flowchart LR
-    Host["Fleet Host<br/>agent installed"] --> Checks
+    Host["Fleet Host<br/>Wazuh agent installed"] --> Checks
 
     subgraph Checks["Detection layers"]
         direction TB
-        SCA["Compliance scoring<br/>12h benchmark scan"]
+        SCA["SCA compliance scoring<br/>12h benchmark scan"]
         FIM["File integrity monitoring<br/>12h scan cycle"]
         RC["Rootkit / anomaly<br/>heuristics"]
     end
@@ -26,53 +31,54 @@ flowchart LR
     Checks --> Alert["Alert fires"]
     Alert --> Known{"Matches a known,<br/>root-caused pattern?"}
     Known -->|yes| Runbook["Runbook entry<br/>no re-investigation"]
-    Known -->|no| Fresh["Investigated fresh<br/>logtest + direct index query"]
+    Known -->|no| Fresh["Investigated fresh<br/>wazuh-logtest + direct index query"]
     Fresh -->|becomes routine| Runbook
 ```
 
 ## The curated "don't re-investigate this" runbook
 
-Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded:
+Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded — this is the difference between a SOC that actually scales past a handful of hosts and one that drowns in its own alert volume:
 
 | Category | What fires | Why it's routine |
 |---|---|---|
 | Daemon lifecycle | Agent start/stop, log rotation, PAM login/logout | Expected operational noise |
-| Package/kernel updates | File-integrity "added/deleted" events | Syscheck working as intended during normal patching |
+| Package/kernel updates | File-integrity "added/deleted" events | Syscheck working as intended during the OS-level updates covered in [`docs/disaster-recovery.md`](disaster-recovery.md) |
 | Antivirus self-test | A known harmless AV test-file trigger | Standard test artifact, not a real detection |
 | Transient disk-full | One-off scratch-mount fill events | Self-resolving cache fills, not a real capacity issue |
-| SSH auth bursts from automation | Multiple-key offer during fleet-wide automation runs | One legitimate connection, not brute force |
+| SSH auth bursts from automation | Multiple-key offer during fleet-wide orchestration runs | One legitimate connection from the orchestration engine in [`docs/orchestration.md`](orchestration.md), not brute force |
 | Container-runtime access-control denials | Isolated, non-flooding events | Known container/runtime interaction, not exploitation |
 
 > [!TIP]
-> **One catch worth calling out**: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive *in the check itself* — it searched for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall. Chasing the alert instead of the actual policy would have wasted real time on a non-problem.
+> **One catch worth calling out in detail**: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive *in the check itself*, not a real gap. The underlying check searched for a raw configuration string that firewalld — the actual firewall in use, per [`docs/firewall-and-network.md`](firewall-and-network.md) — simply never produces, because firewalld implements default-deny through its own zone/chain structure rather than the specific policy string the check was written to expect. The fix was verifying the real firewall posture directly against firewalld's own state, not touching the firewall itself. This is a useful pattern in general: a security tool's own check can be wrong about a system it wasn't originally written for, and the fix is understanding the actual system, not chasing the tool's assumption.
 
 ## Querying and testing directly
 
-Alert data lives in a searchable index, reachable without going through any dashboard — a single aggregation query returns a fleet-wide "what's actually firing and where" summary in one call.
+Alert data lives in a searchable OpenSearch-based index, reachable without going through the [Dashy SOC Board](dashy-soc-board.md) — a single aggregation query returns a fleet-wide "what's actually firing and where" summary in one call, useful when the dashboard's own pre-aggregated view isn't granular enough for a specific question.
 
 > [!WARNING]
-> Two gotchas that will burn a query if missed: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage. And the manager's own container name doesn't match the product's default name, an easy miss when scripting against it.
+> Two gotchas that will burn a query if missed: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage, and a real time sink to debug the first time it happens. And the manager's own container name doesn't match the product's default name, an easy miss when scripting against it directly rather than through the dashboard.
 
-Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.
+Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into `wazuh-logtest`, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op that would otherwise only surface once a real event failed to alert.
 
 ## Compliance hardening: justified exceptions, not blanket enforcement
 
 ```mermaid
 flowchart TD
-    Framework["Multi-role hardening framework"] --> Apply{"Applies cleanly<br/>to this host?"}
+    Framework["CIS-oriented hardening framework<br/>9 tagged Ansible sub-roles"] --> Apply{"Applies cleanly<br/>to this host?"}
-    Apply -->|yes| Enforce["Enforced fleet-wide"]
+    Apply -->|yes| Enforce["Enforced fleet-wide,<br/>re-applied every compliance run —<br/>not just at creation"]
     Apply -->|no, justified| Skip["Deliberately skipped<br/>reason documented"]
-    Skip --> R1["Control doesn't apply<br/>inside a container"]
+    Skip --> R1["Kernel forwarding control<br/>flipped for Docker hosts —<br/>container networking needs it"]
-    Skip --> R2["Needs a disk layout<br/>not every host has"]
+    Skip --> R2["SSH TCP forwarding<br/>kept permissive fleet-wide —<br/>real remote access depends on it"]
-    Skip --> R3["Only makes sense on<br/>always-on VMs"]
 ```
 
-The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards.
+This is the same tag-driven selection model described in [`docs/iac-pipeline.md`](iac-pipeline.md) — which controls apply to a given host is explicit in that host's own tag list, not an implicit "all or nothing" blanket. Two real deviations from the strict CIS baseline exist, and both are documented with their actual reasoning rather than left as unexplained exceptions: one kernel networking control is loosened specifically for Docker hosts because container networking genuinely requires it, and SSH TCP forwarding stays enabled fleet-wide because real remote access depends on it. Neither is a gap waiting to be closed — both are permanent, reasoned trade-offs.
+
+The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards under pressure while writing the config.
 
 ## Automated benchmarking: native, not containerized
 
 > [!IMPORTANT]
-> The fleet's automated security-benchmark tool runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations. It runs weekly, deliberately timed against real observed load data rather than an arbitrary default schedule.
+> The fleet's automated security-benchmark tool (Docker Bench for Security) runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations (glibc vs. musl) that made it fail to run reliably. Running it natively instead of debugging a broken official container was the pragmatic call: the tool's job is auditing Docker, not testing container-runtime compatibility edge cases. It runs weekly, deliberately timed against real observed cron load from the other scheduled jobs described in [`docs/cron-scheduler.md`](cron-scheduler.md), rather than an arbitrary default schedule that might collide with heavier jobs.
 
 ---
-Related stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)
+Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md), [`docs/notifications.md`](notifications.md), [`docs/iac-pipeline.md`](iac-pipeline.md) · Stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)

docs/sso-and-identity.md

@@ -1,7 +1,18 @@
 # 🪪 SSO & Identity
 
 > [!NOTE]
-> Every app that needs single sign-on registers against the identity provider once per reverse-proxy zone it's exposed on — and a real platform limitation shapes how superuser accounts authenticate.
+> Authentik is the identity provider for this fleet — self-hosted rather than an external IdP, for the same reason Vault is self-hosted in [`docs/secrets-and-pki.md`](secrets-and-pki.md): internal-only apps shouldn't need to trust a third party just to authenticate a login, and the two-zone reverse-proxy model in [`docs/dns-naming.md`](dns-naming.md) only works if the identity provider itself lives inside that same trust boundary.
+
+| | |
+|---|---|
+| **Software** | Authentik, self-hosted |
+| **Pattern** | Provider + application pair registered once per reverse-proxy zone an app is exposed on |
+| **Native OIDC/SAML apps** | ~12 apps, where the upstream software supports it directly |
+| **Everything else** | Proxy-based forward-auth — SSO without the app needing to implement OIDC itself |
+
+## Why proxy-based forward-auth, not just native OIDC everywhere
+
+Most self-hosted apps don't implement OIDC or SAML themselves — a media manager or a bookmarking tool typically wasn't built with enterprise SSO in mind. Requiring native OIDC support fleet-wide would have meant leaving most apps without SSO at all. Authentik's proxy-provider forward-auth pattern sits in front of any app regardless of whether it understands OIDC, checking authentication at the reverse-proxy layer before a request ever reaches the app itself. The ~12 apps that *do* support native OIDC still use it — it's a better experience where available (no separate login redirect through the proxy) — but forward-auth is what makes SSO actually fleet-wide rather than limited to whichever apps happen to support it.
 
 ## Login flow — same shape, every app
 
@@ -9,40 +20,36 @@
 sequenceDiagram
     participant U as User
     participant App as App (browser-facing zone)
-    participant IdP as Identity Provider (internal zone)
+    participant IdP as Authentik (internal zone)
 
     U->>App: Opens app
-    App->>U: Redirect to identity provider
+    App->>U: Redirect to Authentik
     U->>IdP: OAuth authorize/token exchange
     Note over IdP: Always happens on the internal zone,<br/>even though the app itself lives<br/>on the browser-facing zone
     IdP->>U: Success
     U->>App: Lands back on the app
 ```
 
-Every app-specific setting beyond this shape (token lifetime, scopes, flow type) is deliberately **not** generalized across apps — each app's own OIDC documentation gets read fresh rather than copying another app's provider config, since the details genuinely differ per vendor.
+This split matters more than it looks: the browser-facing zone enforces mutual TLS (see [`docs/dns-naming.md`](dns-naming.md)) and the app itself might live there, but the actual authentication handshake always happens against Authentik's internal-zone address. Getting this backwards — pointing the OIDC discovery/authority setting at the wrong zone — is a real, easy mistake when onboarding a new app, since the app's own login page is on the *other* zone and it's natural to assume the auth endpoint matches.
 
-| | |
+Every app-specific setting beyond this shape (token lifetime, scopes, flow type) is deliberately **not** generalized across apps — each app's own OIDC documentation gets read fresh rather than copying another app's provider config, since the details genuinely differ per vendor and copying blind is how subtle auth bugs get introduced.
-|---|---|
-| **Registration pattern** | Provider + application pair per zone (up to 2 per app) |
-| **Native OIDC/SAML apps** | ~12 apps, where the upstream supports it directly |
-| **Everything else** | Proxy-based forward-auth |
 
-## A real limitation, documented rather than hidden
+## A real platform limitation, worked around rather than ignored
 
 > [!WARNING]
-> LDAP-based login through the identity provider's own outpost **isn't viable for the primary superuser account** — the outpost hides superuser accounts from LDAP directory search, so any app requiring a search-then-bind flow can never authenticate that account through LDAP. The two workarounds — a second non-admin account, or demoting the primary account from superuser — were both rejected as worse than the problem. Header-based forward-auth is used instead for apps in that situation.
+> Authentik's own LDAP outpost **hides superuser accounts from LDAP directory search** — so any app requiring a search-then-bind LDAP flow (a media server's user-management plugin, for instance) can never authenticate the fleet's primary superuser account through LDAP, full stop. Two workarounds were on the table: create a second, non-admin account just for LDAP-dependent apps, or demote the primary account from superuser entirely. Both were rejected — a second account fragments identity across the fleet in a way that undermines the whole point of centralized SSO, and demoting the primary account breaks unrelated admin workflows that depend on superuser rights. **Header-based forward-auth is used instead** for any app in this situation: Authentik's proxy layer authenticates the user and passes identity via HTTP headers, sidestepping the LDAP search-then-bind requirement entirely rather than working around Authentik's own limitation inside LDAP itself.
 
 ## Git access: least privilege, not the admin account
 
 ```mermaid
 flowchart LR
-    CI["CI / Automation"] -->|scoped write| Bot["Dedicated bot account<br/>non-admin"]
+    CI["CI / Orchestration engine<br/>see docs/orchestration.md"] -->|scoped write| Bot["Dedicated bot account<br/>non-admin"]
-    Bot --> Repo["Git server"]
+    Bot --> Repo["Gitea"]
     Other["Other hosts"] -->|read-only| Deploy["Deploy keys<br/>per host"]
     Deploy --> Repo
 ```
 
-CI and automation pushes run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. Hosts that only need to *read* get separate read-only deploy keys instead of any shared credential. A compromised automation credential is never an admin credential.
+This is a smaller decision than the SSO model above, but the same principle: automation should never run with more privilege than it needs. CI and orchestration pushes (the flows described in [`docs/orchestration.md`](orchestration.md)) run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. Hosts that only need to *read* get separate read-only deploy keys instead of any shared credential. The practical payoff: if an automation credential ever leaks, the blast radius is "can push to specific repos," not "has full admin control of the Git server."
 
 ---
-Related stacks: [`stacks/identity-and-security/authentik/`](../stacks/identity-and-security/authentik/), [`stacks/platform/gitea/`](../stacks/platform/gitea/)
+Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md) · Stacks: [`stacks/identity-and-security/authentik/`](../stacks/identity-and-security/authentik/), [`stacks/platform/gitea/`](../stacks/platform/gitea/)