public/docs/security-monitoring.md

🛡️ Security Monitoring

Note

A self-built SOC stack: fleet-wide agent monitoring, a curated runbook for known-noisy alerts, and compliance hardening rolled out with deliberately justified exceptions instead of blanket enforcement.

Coverage	Every fleet host, agent-based, no gaps
Detection layers	Compliance scoring · file integrity monitoring · rootkit/anomaly heuristics
Runbook	10+ recurring alert types root-caused once, documented, never re-triaged from scratch
Query path	Direct index queries, bypassing the dashboard entirely when needed

How an alert gets handled

flowchart LR
    Host["Fleet Host<br/>agent installed"] --> Checks

    subgraph Checks["Detection layers"]
        direction TB
        SCA["Compliance scoring<br/>12h benchmark scan"]
        FIM["File integrity monitoring<br/>12h scan cycle"]
        RC["Rootkit / anomaly<br/>heuristics"]
    end

    Checks --> Alert["Alert fires"]
    Alert --> Known{"Matches a known,<br/>root-caused pattern?"}
    Known -->|yes| Runbook["Runbook entry<br/>no re-investigation"]
    Known -->|no| Fresh["Investigated fresh<br/>logtest + direct index query"]
    Fresh -->|becomes routine| Runbook

The curated "don't re-investigate this" runbook

Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded:

Category	What fires	Why it's routine
Daemon lifecycle	Agent start/stop, log rotation, PAM login/logout	Expected operational noise
Package/kernel updates	File-integrity "added/deleted" events	Syscheck working as intended during normal patching
Antivirus self-test	A known harmless AV test-file trigger	Standard test artifact, not a real detection
Transient disk-full	One-off scratch-mount fill events	Self-resolving cache fills, not a real capacity issue
SSH auth bursts from automation	Multiple-key offer during fleet-wide automation runs	One legitimate connection, not brute force
Container-runtime access-control denials	Isolated, non-flooding events	Known container/runtime interaction, not exploitation

Tip

One catch worth calling out: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive in the check itself — it searched for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall. Chasing the alert instead of the actual policy would have wasted real time on a non-problem.

Querying and testing directly

Alert data lives in a searchable index, reachable without going through any dashboard — a single aggregation query returns a fleet-wide "what's actually firing and where" summary in one call.

Warning

Two gotchas that will burn a query if missed: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage. And the manager's own container name doesn't match the product's default name, an easy miss when scripting against it.

Custom detection rules get tested before rollout, not after: a synthetic log line gets piped into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.

Compliance hardening: justified exceptions, not blanket enforcement

flowchart TD
    Framework["Multi-role hardening framework"] --> Apply{"Applies cleanly<br/>to this host?"}
    Apply -->|yes| Enforce["Enforced fleet-wide"]
    Apply -->|no, justified| Skip["Deliberately skipped<br/>reason documented"]
    Skip --> R1["Control doesn't apply<br/>inside a container"]
    Skip --> R2["Needs a disk layout<br/>not every host has"]
    Skip --> R3["Only makes sense on<br/>always-on VMs"]

The authentication stack is configured to fail safe under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards.

Automated benchmarking: native, not containerized

Important

The fleet's automated security-benchmark tool runs as a native binary, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations. It runs weekly, deliberately timed against real observed load data rather than an arbitrary default schedule.

---

Related stacks: stacks/identity-and-security/wazuh/