docs: rework security/hardening batch with diagrams and callouts

Mermaid diagrams up front, GFM alert callouts for key decisions and
gotchas, tables instead of bullet walls - replaces plain prose with
the visual template established for security-monitoring.md.

docs/security-monitoring.md

@@ -1,30 +1,78 @@
-# Security Monitoring
+# 🛡️ Security Monitoring
 
-A self-built SOC stack: fleet-wide agent-based monitoring, a curated runbook for known-noisy alerts, and a compliance-hardening rollout with deliberately justified exceptions rather than blanket enforcement.
+> [!NOTE]
+> A self-built SOC stack: fleet-wide agent monitoring, a curated runbook for known-noisy alerts, and compliance hardening rolled out with deliberately justified exceptions instead of blanket enforcement.
 
-## Detection mechanisms, fleet-wide
+| | |
+|---|---|
+| **Coverage** | Every fleet host, agent-based, no gaps |
+| **Detection layers** | Compliance scoring · file integrity monitoring · rootkit/anomaly heuristics |
+| **Runbook** | 10+ recurring alert types root-caused once, documented, never re-triaged from scratch |
+| **Query path** | Direct index queries, bypassing the dashboard entirely when needed |
 
-- **Security Configuration Assessment** — CIS-benchmark compliance scoring on a 12-hour interval, with a distinct benchmark policy per OS family. Scores below 100% are treated as an accepted, tracked tradeoff rather than an open bug to chase — re-litigating each individual control isn't the goal; the hardening rollout itself (below) is the actual control.
-- **File integrity monitoring** — enabled fleet-wide on a 12-hour scan cycle, the fleet's only FIM mechanism. A small number of hosts have specific paths excluded for known high-churn or false-positive-prone locations (e.g. a hypervisor's own live cluster-state files).
-- **Rootkit/anomaly heuristics** — a handful of container-hosted machines have specific exclusions for container pseudo-filesystem paths that the heuristic otherwise misreads — a known false-positive class for containerized hosts, not a suppressed real finding. Signature-file suppressions are per-agent, not centrally synced, so a suppression on one host deliberately doesn't apply anywhere else.
+## How an alert gets handled
 
-## A curated "don't re-investigate this" runbook
+```mermaid
+flowchart LR
+    Host["Fleet Host<br/>agent installed"] --> Checks
 
-Rather than re-triaging the same recurring alert from scratch every time it fires, routine alerts get root-caused once and documented — daemon lifecycle noise, expected package-manager/kernel-update file changes, a standard antivirus test-file trigger, one-off transient full-disk events on scratch mounts, and several others each with their actual cause on record rather than a blanket dismissal. One example worth calling out: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive in the *check itself* — it was grepping for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall.
+    subgraph Checks["Detection layers"]
+        direction TB
+        SCA["Compliance scoring<br/>12h benchmark scan"]
+        FIM["File integrity monitoring<br/>12h scan cycle"]
+        RC["Rootkit / anomaly<br/>heuristics"]
+    end
+
+    Checks --> Alert["Alert fires"]
+    Alert --> Known{"Matches a known,<br/>root-caused pattern?"}
+    Known -->|yes| Runbook["Runbook entry<br/>no re-investigation"]
+    Known -->|no| Fresh["Investigated fresh<br/>logtest + direct index query"]
+    Fresh -->|becomes routine| Runbook
+```
+
+## The curated "don't re-investigate this" runbook
+
+Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded:
+
+| Category | What fires | Why it's routine |
+|---|---|---|
+| Daemon lifecycle | Agent start/stop, log rotation, PAM login/logout | Expected operational noise |
+| Package/kernel updates | File-integrity "added/deleted" events | Syscheck working as intended during normal patching |
+| Antivirus self-test | A known harmless AV test-file trigger | Standard test artifact, not a real detection |
+| Transient disk-full | One-off scratch-mount fill events | Self-resolving cache fills, not a real capacity issue |
+| SSH auth bursts from automation | Multiple-key offer during fleet-wide automation runs | One legitimate connection, not brute force |
+| Container-runtime access-control denials | Isolated, non-flooding events | Known container/runtime interaction, not exploitation |
+
+> [!TIP]
+> **One catch worth calling out**: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive *in the check itself* — it searched for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall. Chasing the alert instead of the actual policy would have wasted real time on a non-problem.
 
 ## Querying and testing directly
 
-Alert data lives in a searchable index, queryable directly without going through the dashboard — a single aggregation query gives a fleet-wide "what's actually firing and where" summary in one call. Two gotchas worth knowing before querying: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query needs converting first or it silently returns zero hits (indistinguishable at a glance from an actual ingestion outage); and the manager's container name doesn't match the product's own default name, easy to get wrong when scripting against it.
+Alert data lives in a searchable index, reachable without going through any dashboard — a single aggregation query returns a fleet-wide "what's actually firing and where" summary in one call.
 
-Custom detection rules get tested before rollout by piping a synthetic log line into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.
+> [!WARNING]
+> Two gotchas that will burn a query if missed: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage. And the manager's own container name doesn't match the product's default name, an easy miss when scripting against it.
+
+Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.
 
 ## Compliance hardening: justified exceptions, not blanket enforcement
 
-A multi-role hardening framework is rolled out fleet-wide, with a small number of deliberate, reasoned exceptions rather than force-applying every control regardless of fit — one control that simply doesn't apply inside a container gets skipped there; another needs a disk layout feature not every host has; a third only makes sense on always-on virtual machines. The authentication stack is configured to fail safe under a specific failure mode rather than fail open, a subtler design choice than it sounds.
+```mermaid
+flowchart TD
+    Framework["Multi-role hardening framework"] --> Apply{"Applies cleanly<br/>to this host?"}
+    Apply -->|yes| Enforce["Enforced fleet-wide"]
+    Apply -->|no, justified| Skip["Deliberately skipped<br/>reason documented"]
+    Skip --> R1["Control doesn't apply<br/>inside a container"]
+    Skip --> R2["Needs a disk layout<br/>not every host has"]
+    Skip --> R3["Only makes sense on<br/>always-on VMs"]
+```
+
+The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards.
 
 ## Automated benchmarking: native, not containerized
 
-The fleet's automated security-benchmark tool runs as a native binary rather than through its own official container image, after that image hit a real binary-compatibility wall between two different C library implementations. It runs on a weekly schedule, deliberately timed against real observed cron-load data rather than an arbitrary "every night at midnight" default.
+> [!IMPORTANT]
+> The fleet's automated security-benchmark tool runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations. It runs weekly, deliberately timed against real observed load data rather than an arbitrary default schedule.
 
 ---
 Related stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)