docs: deepen security/hardening batch to real deep-dive standard

Names real technology throughout (Vault, Authentik, pfSense,
firewalld, Wazuh) instead of staying generic, adds a 'why this
approach vs the alternative' section to every doc, cross-links
heavily between docs instead of treating each system in isolation.

docs/security-monitoring.md

@@ -1,24 +1,29 @@
 # 🛡️ Security Monitoring
 
 > [!NOTE]
-> A self-built SOC stack: fleet-wide agent monitoring, a curated runbook for known-noisy alerts, and compliance hardening rolled out with deliberately justified exceptions instead of blanket enforcement.
+> Wazuh is the detection engine, deployed as an agent on every host — file integrity monitoring, rootkit/anomaly heuristics, and compliance scoring, feeding one indexer that a curated runbook and a direct query workflow both draw from. This is the layer that actually watches the trust boundaries described everywhere else in this repo, not just documents them.
 
 | | |
 |---|---|
+| **Software** | Wazuh (manager/indexer/dashboard), self-hosted |
 | **Coverage** | Every fleet host, agent-based, no gaps |
 | **Detection layers** | Compliance scoring · file integrity monitoring · rootkit/anomaly heuristics |
 | **Runbook** | 10+ recurring alert types root-caused once, documented, never re-triaged from scratch |
-| **Query path** | Direct index queries, bypassing the dashboard entirely when needed |
+| **Query path** | Direct index queries, bypassing the SOC dashboard entirely when needed |
+
+## Why agent-based, and why self-hosted
+
+The alternative to an agent on every host is a purely network-level monitoring approach — watching traffic at the firewall in [`docs/firewall-and-network.md`](firewall-and-network.md) and inferring what's happening inside each host. That approach can't see file integrity changes, local privilege escalation, or rootkit behavior — the things that matter most once an attacker is already past the network perimeter. An agent on every host closes that gap at the cost of real per-host overhead and a rollout that has to cover 100% of the fleet to be trustworthy — a partial rollout would create a false sense of coverage that's arguably worse than no agent at all. Self-hosting the manager/indexer/dashboard stack, rather than using a SaaS SIEM, keeps every alert's raw data (hostnames, internal paths, credential-adjacent metadata) inside the same trust boundary Vault protects in [`docs/secrets-and-pki.md`](secrets-and-pki.md), instead of shipping it to a third party.
 
 ## How an alert gets handled
 
 ```mermaid
 flowchart LR
-    Host["Fleet Host<br/>agent installed"] --> Checks
+    Host["Fleet Host<br/>Wazuh agent installed"] --> Checks
 
     subgraph Checks["Detection layers"]
         direction TB
-        SCA["Compliance scoring<br/>12h benchmark scan"]
+        SCA["SCA compliance scoring<br/>12h benchmark scan"]
         FIM["File integrity monitoring<br/>12h scan cycle"]
         RC["Rootkit / anomaly<br/>heuristics"]
     end
@@ -26,53 +31,54 @@ flowchart LR
     Checks --> Alert["Alert fires"]
     Alert --> Known{"Matches a known,<br/>root-caused pattern?"}
     Known -->|yes| Runbook["Runbook entry<br/>no re-investigation"]
-    Known -->|no| Fresh["Investigated fresh<br/>logtest + direct index query"]
+    Known -->|no| Fresh["Investigated fresh<br/>wazuh-logtest + direct index query"]
     Fresh -->|becomes routine| Runbook
 ```
 
 ## The curated "don't re-investigate this" runbook
 
-Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded:
+Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded — this is the difference between a SOC that actually scales past a handful of hosts and one that drowns in its own alert volume:
 
 | Category | What fires | Why it's routine |
 |---|---|---|
 | Daemon lifecycle | Agent start/stop, log rotation, PAM login/logout | Expected operational noise |
-| Package/kernel updates | File-integrity "added/deleted" events | Syscheck working as intended during normal patching |
+| Package/kernel updates | File-integrity "added/deleted" events | Syscheck working as intended during the OS-level updates covered in [`docs/disaster-recovery.md`](disaster-recovery.md) |
 | Antivirus self-test | A known harmless AV test-file trigger | Standard test artifact, not a real detection |
 | Transient disk-full | One-off scratch-mount fill events | Self-resolving cache fills, not a real capacity issue |
-| SSH auth bursts from automation | Multiple-key offer during fleet-wide automation runs | One legitimate connection, not brute force |
+| SSH auth bursts from automation | Multiple-key offer during fleet-wide orchestration runs | One legitimate connection from the orchestration engine in [`docs/orchestration.md`](orchestration.md), not brute force |
 | Container-runtime access-control denials | Isolated, non-flooding events | Known container/runtime interaction, not exploitation |
 
 > [!TIP]
-> **One catch worth calling out**: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive *in the check itself* — it searched for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall. Chasing the alert instead of the actual policy would have wasted real time on a non-problem.
+> **One catch worth calling out in detail**: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive *in the check itself*, not a real gap. The underlying check searched for a raw configuration string that firewalld — the actual firewall in use, per [`docs/firewall-and-network.md`](firewall-and-network.md) — simply never produces, because firewalld implements default-deny through its own zone/chain structure rather than the specific policy string the check was written to expect. The fix was verifying the real firewall posture directly against firewalld's own state, not touching the firewall itself. This is a useful pattern in general: a security tool's own check can be wrong about a system it wasn't originally written for, and the fix is understanding the actual system, not chasing the tool's assumption.
 
 ## Querying and testing directly
 
-Alert data lives in a searchable index, reachable without going through any dashboard — a single aggregation query returns a fleet-wide "what's actually firing and where" summary in one call.
+Alert data lives in a searchable OpenSearch-based index, reachable without going through the [Dashy SOC Board](dashy-soc-board.md) — a single aggregation query returns a fleet-wide "what's actually firing and where" summary in one call, useful when the dashboard's own pre-aggregated view isn't granular enough for a specific question.
 
 > [!WARNING]
-> Two gotchas that will burn a query if missed: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage. And the manager's own container name doesn't match the product's default name, an easy miss when scripting against it.
+> Two gotchas that will burn a query if missed: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage, and a real time sink to debug the first time it happens. And the manager's own container name doesn't match the product's default name, an easy miss when scripting against it directly rather than through the dashboard.
 
-Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.
+Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into `wazuh-logtest`, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op that would otherwise only surface once a real event failed to alert.
 
 ## Compliance hardening: justified exceptions, not blanket enforcement
 
 ```mermaid
 flowchart TD
-    Framework["Multi-role hardening framework"] --> Apply{"Applies cleanly<br/>to this host?"}
-    Apply -->|yes| Enforce["Enforced fleet-wide"]
+    Framework["CIS-oriented hardening framework<br/>9 tagged Ansible sub-roles"] --> Apply{"Applies cleanly<br/>to this host?"}
+    Apply -->|yes| Enforce["Enforced fleet-wide,<br/>re-applied every compliance run —<br/>not just at creation"]
     Apply -->|no, justified| Skip["Deliberately skipped<br/>reason documented"]
-    Skip --> R1["Control doesn't apply<br/>inside a container"]
-    Skip --> R2["Needs a disk layout<br/>not every host has"]
-    Skip --> R3["Only makes sense on<br/>always-on VMs"]
+    Skip --> R1["Kernel forwarding control<br/>flipped for Docker hosts —<br/>container networking needs it"]
+    Skip --> R2["SSH TCP forwarding<br/>kept permissive fleet-wide —<br/>real remote access depends on it"]
 ```
 
-The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards.
+This is the same tag-driven selection model described in [`docs/iac-pipeline.md`](iac-pipeline.md) — which controls apply to a given host is explicit in that host's own tag list, not an implicit "all or nothing" blanket. Two real deviations from the strict CIS baseline exist, and both are documented with their actual reasoning rather than left as unexplained exceptions: one kernel networking control is loosened specifically for Docker hosts because container networking genuinely requires it, and SSH TCP forwarding stays enabled fleet-wide because real remote access depends on it. Neither is a gap waiting to be closed — both are permanent, reasoned trade-offs.
+
+The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards under pressure while writing the config.
 
 ## Automated benchmarking: native, not containerized
 
 > [!IMPORTANT]
-> The fleet's automated security-benchmark tool runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations. It runs weekly, deliberately timed against real observed load data rather than an arbitrary default schedule.
+> The fleet's automated security-benchmark tool (Docker Bench for Security) runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations (glibc vs. musl) that made it fail to run reliably. Running it natively instead of debugging a broken official container was the pragmatic call: the tool's job is auditing Docker, not testing container-runtime compatibility edge cases. It runs weekly, deliberately timed against real observed cron load from the other scheduled jobs described in [`docs/cron-scheduler.md`](cron-scheduler.md), rather than an arbitrary default schedule that might collide with heavier jobs.
 
 ---
-Related stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)
+Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md), [`docs/notifications.md`](notifications.md), [`docs/iac-pipeline.md`](iac-pipeline.md) · Stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)