# Security Monitoring

A self-built SOC stack: fleet-wide agent-based monitoring, a curated runbook for known-noisy alerts, and a compliance-hardening rollout with deliberately justified exceptions rather than blanket enforcement.

## Detection mechanisms, fleet-wide

- **Security Configuration Assessment** — CIS-benchmark compliance scoring on a 12-hour interval, with a distinct benchmark policy per OS family. Scores below 100% are treated as an accepted, tracked tradeoff rather than an open bug to chase — re-litigating each individual control isn't the goal; the hardening rollout itself (below) is the actual control.
- **File integrity monitoring** — enabled fleet-wide on a 12-hour scan cycle, the fleet's only FIM mechanism. A small number of hosts have specific paths excluded for known high-churn or false-positive-prone locations (e.g. a hypervisor's own live cluster-state files).
- **Rootkit/anomaly heuristics** — a handful of container-hosted machines have specific exclusions for container pseudo-filesystem paths that the heuristic otherwise misreads — a known false-positive class for containerized hosts, not a suppressed real finding. Signature-file suppressions are per-agent, not centrally synced, so a suppression on one host deliberately doesn't apply anywhere else.

## A curated "don't re-investigate this" runbook

Rather than re-triaging the same recurring alert from scratch every time it fires, routine alerts get root-caused once and documented — daemon lifecycle noise, expected package-manager/kernel-update file changes, a standard antivirus test-file trigger, one-off transient full-disk events on scratch mounts, and several others each with their actual cause on record rather than a blanket dismissal. One example worth calling out: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive in the *check itself* — it was grepping for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall.

## Querying and testing directly

Alert data lives in a searchable index, queryable directly without going through the dashboard — a single aggregation query gives a fleet-wide "what's actually firing and where" summary in one call. Two gotchas worth knowing before querying: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query needs converting first or it silently returns zero hits (indistinguishable at a glance from an actual ingestion outage); and the manager's container name doesn't match the product's own default name, easy to get wrong when scripting against it.

Custom detection rules get tested before rollout by piping a synthetic log line into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.

## Compliance hardening: justified exceptions, not blanket enforcement

A multi-role hardening framework is rolled out fleet-wide, with a small number of deliberate, reasoned exceptions rather than force-applying every control regardless of fit — one control that simply doesn't apply inside a container gets skipped there; another needs a disk layout feature not every host has; a third only makes sense on always-on virtual machines. The authentication stack is configured to fail safe under a specific failure mode rather than fail open, a subtler design choice than it sounds.

## Automated benchmarking: native, not containerized

The fleet's automated security-benchmark tool runs as a native binary rather than through its own official container image, after that image hit a real binary-compatibility wall between two different C library implementations. It runs on a weekly schedule, deliberately timed against real observed cron-load data rather than an arbitrary "every night at midnight" default.

---
Related stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)