public/docs/security-monitoring.md

🛡️ Security Monitoring

Note

Wazuh is the detection engine, deployed as an agent on every host — file integrity monitoring, rootkit/anomaly heuristics, and compliance scoring, feeding one indexer that a curated runbook and a direct query workflow both draw from. This is the layer that actually watches the trust boundaries described everywhere else in this repo, not just documents them.

Software	Wazuh (manager/indexer/dashboard), self-hosted
Coverage	11 active agents, one per fleet host, no gaps
Detection layers	Compliance scoring (SCA) · file integrity monitoring (syscheck) · rootkit/anomaly heuristics (rootcheck)
Runbook	10+ recurring alert types root-caused once, documented, never re-triaged from scratch
Query path	Direct index queries, bypassing the SOC dashboard entirely when needed

This is the actual live agent roster, not a claimed count:

$ agent_control -l
ID: 000, Name: wazuh.manager (server), IP: 127.0.0.1, Active/Local
ID: 001, Name: hyper1, IP: any, Active
ID: 002, Name: hyper2, IP: any, Active
ID: 003, Name: pbs, IP: any, Active
ID: 004, Name: automation, IP: any, Active
ID: 006, Name: monitoring, IP: any, Active
ID: 007, Name: wazuh, IP: any, Active
ID: 008, Name: rocky-docker, IP: any, Active
ID: 009, Name: vault, IP: any, Active
ID: 010, Name: caddy, IP: any, Active
ID: 011, Name: technitium, IP: any, Active
ID: 012, Name: file-server, IP: any, Active

Why agent-based, and why self-hosted

The alternative to an agent on every host is a purely network-level monitoring approach — watching traffic at the firewall in docs/firewall-and-network.md and inferring what's happening inside each host. That approach can't see file integrity changes, local privilege escalation, or rootkit behavior — the things that matter most once an attacker is already past the network perimeter. An agent on every host closes that gap at the cost of real per-host overhead and a rollout that has to cover 100% of the fleet to be trustworthy — a partial rollout would create a false sense of coverage that's arguably worse than no agent at all. Self-hosting the manager/indexer/dashboard stack, rather than using a SaaS SIEM, keeps every alert's raw data (hostnames, internal paths, credential-adjacent metadata) inside the same trust boundary Vault protects in docs/secrets-and-pki.md, instead of shipping it to a third party.

Tip

The manager watches itself, as two independent things sharing one VM. The manager/indexer/dashboard Docker stack and the native agent monitoring that same host are unrelated — the manager isn't self-aware, it's just also enrolled as agent 000. And the two halves of "how Wazuh is configured" have different discipline levels on purpose: the compose stack itself is tracked in its own Gitea repo, but the actual detection rules and decoders live in a Docker volume, edited directly through the dashboard or manager API — not version-controlled. If the manager goes down entirely, agents don't lose events: they buffer locally and replay once it's back, so an outage costs visibility, not data.

A fleet-wide bug this system caught in itself

Important

Wazuh's own default agent log-queue buffer (1024 entries) overflowed fleet-wide during Ansible orchestration bursts — enough hosts logging enough events at once that the queue filled before the agent could forward them, silently dropping data rather than erroring loudly. Found, not assumed: root-caused to the default queue size specifically, then fixed with a dedicated tuning role that raises logcollector.queue_size on every agent. That role is deliberately narrow — it only touches queue size and enabled/started state; syscheck ignores, rootcheck ignores, and other per-agent tuning are still one-off manual edits, honestly left uncovered rather than implied as handled by a role that doesn't actually reach them.

How an alert gets handled

The curated "don't re-investigate this" runbook

Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded — this is the difference between a SOC that actually scales past a handful of hosts and one that drowns in its own alert volume. A representative slice of what's actually in it, real rule IDs and all:

Rule(s)	What fires	Why it's routine
5758 + cascade (2501/5503/5557/5716/5760)	Multiple-key offer during Ansible fan-out	One legitimate connection from the orchestration engine in docs/orchestration.md, not brute force
553/554	File-integrity "added/deleted"	Syscheck working as intended during the OS-level updates covered in docs/disaster-recovery.md
52502/52503	ClamAV's own harmless AV test-file trigger	Standard test artifact, not a real detection
531	One-off scratch-mount fill events	Self-resolving cache fills, not a real capacity issue
80732	SELinux/container_runtime_t interaction, isolated	Known container-runtime behavior, not exploitation
Daemon lifecycle noise	Agent start/stop, log rotation, PAM login/logout	Expected operational noise

Tip

One catch worth calling out in detail: rule 19011, "nftables default deny not detected," turned out to be a false positive in the check itself, not a real gap. The check greps for a raw nftables policy drop string that firewalld — the actual firewall in use, per docs/firewall-and-network.md — simply never produces, because firewalld implements default-deny through its own zone/chain structure instead. The fix was verifying the real firewall posture directly against firewalld's own state, not touching the firewall itself. This is a useful pattern in general: a security tool's own check can be wrong about a system it wasn't originally written for, and the fix is understanding the actual system, not chasing the tool's assumption.

Querying and testing directly

Alert data lives in a searchable OpenSearch-based index, reachable directly without going through the Dashy SOC Board — a terms-aggregation on rule.id, with sub-aggregations for rule.description, max rule.level, agent.name, and max timestamp, returns a fleet-wide "what's actually firing and where" summary in one call — useful when the dashboard's own pre-aggregated view isn't granular enough for a specific question.

Warning

Two gotchas that will burn a query if missed. Alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage, and a real time sink the first time it happens. And the manager's own container is named single-node-wazuh.manager-1, not the product's default wazuh.manager — an easy miss when scripting against it directly rather than through the dashboard.

Custom detection rules get tested before rollout, not after: a synthetic log line gets piped into wazuh-logtest, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op that would otherwise only surface once a real event failed to alert. This is a real synthetic SSH failure piped through the actual tool, not a description of what it would show:

$ echo 'Aug 28 12:00:00 caddy sshd[12345]: Failed password for invalid user testuser from 203.0.113.5 port 51234 ssh2' \
  | docker exec -i single-node-wazuh.manager-1 /var/ossec/bin/wazuh-logtest

	name: 'sshd'
	parent: 'sshd'
	srcip: '203.0.113.5'
	srcuser: 'testuser'

	id: '5710'
	level: '5'
	description: 'sshd: Attempt to login using a non-existent user'
	groups: '['syslog', 'sshd', 'authentication_failed', 'invalid_login']'
	mitre.id: '['T1110.001', 'T1021.004']'
	mitre.tactic: '['Credential Access', 'Lateral Movement']'
	pci_dss: '['10.2.4', '10.2.5', '10.6.1']'

One rule-writing gotcha that's easy to get backwards: a rule's <decoded_as> field has to reference the parent decoder's name, never a child regex-decoder further down the same chain — wazuh-logtest always reports the parent's name in its output regardless of which child actually matched, so debugging by what the tool prints back leads to the wrong name. And wazuh-logtest re-reads the rule/decoder files fresh on every invocation, independent of whether the live daemon has actually reloaded them — a rule that tests correctly can still be inert in production until wazuh-control restart actually loads it.

Compliance hardening: justified exceptions, not blanket enforcement

This is the same tag-driven selection model described in docs/iac-pipeline.md — which controls apply to a given host is explicit in that host's own tag list, not an implicit "all or nothing" blanket. SCA scores below 100% on any given host are an accepted tradeoff, not tracked as open bugs to close one by one — the hardening rollout itself is the actual control, not a perfect per-control score. Two real deviations from the strict CIS baseline exist fleet-wide, and both are documented with their actual reasoning rather than left as unexplained exceptions — neither is a gap waiting to be closed, both are permanent, reasoned trade-offs.

The authentication stack is configured to fail safe under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards under pressure while writing the config.

Automated benchmarking: native, not containerized

Important

The fleet's automated security-benchmark tool (Docker Bench for Security) runs as a native binary, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations (glibc vs. musl) that made it fail to run reliably. Running it natively instead of debugging a broken official container was the pragmatic call: the tool's job is auditing Docker, not testing container-runtime compatibility edge cases. It runs weekly, deliberately timed against real observed cron load from the other scheduled jobs described in docs/cron-scheduler.md, rather than an arbitrary default schedule that might collide with heavier jobs.

---

Related: docs/firewall-and-network.md, docs/dashy-soc-board.md, docs/notifications.md, docs/iac-pipeline.md · Stacks: stacks/identity-and-security/wazuh/