🛡️ Security Monitoring

Note

Wazuh is the detection engine, deployed as an agent on every host — file integrity monitoring, rootkit/anomaly heuristics, and compliance scoring, feeding one indexer that a curated runbook and a direct query workflow both draw from. This is the layer that actually watches the trust boundaries described everywhere else in this repo, not just documents them.


Software	Wazuh (manager/indexer/dashboard), self-hosted
Coverage	12 active agents, one per fleet host, no gaps
Detection layers	Compliance scoring (SCA) · file integrity monitoring (syscheck) · rootkit/anomaly heuristics (rootcheck)
Runbook	10+ recurring alert types root-caused once, documented, never re-triaged from scratch
Query path	Direct index queries, bypassing the SOC dashboard entirely when needed

This is the actual live agent roster, not a claimed count:

$ agent_control -l
ID: 000, Name: wazuh.manager (server), IP: 127.0.0.1, Active/Local
ID: 001, Name: hyper1, IP: any, Active
ID: 002, Name: hyper2, IP: any, Active
ID: 003, Name: pbs, IP: any, Active
ID: 004, Name: automation, IP: any, Active
ID: 006, Name: monitoring, IP: any, Active
ID: 007, Name: wazuh, IP: any, Active
ID: 008, Name: rocky-docker, IP: any, Active
ID: 009, Name: vault, IP: any, Active
ID: 010, Name: caddy, IP: any, Active
ID: 011, Name: technitium, IP: any, Active
ID: 012, Name: file-server, IP: any, Active
ID: 013, Name: netbird, IP: any, Active

Why Agent-based, and Why Self-hosted

The alternative to an agent on every host is a purely network-level monitoring approach — watching traffic at the firewall in docs/firewall-and-network.md and inferring what's happening inside each host. That approach can't see file integrity changes, local privilege escalation, or rootkit behavior — the things that matter most once an attacker is already past the network perimeter. An agent on every host closes that gap at the cost of real per-host overhead and a rollout that has to cover 100% of the fleet to be trustworthy — a partial rollout would create a false sense of coverage that's arguably worse than no agent at all. Self-hosting the manager/indexer/dashboard stack, rather than using a SaaS SIEM, keeps every alert's raw data (hostnames, internal paths, credential-adjacent metadata) inside the same trust boundary Vault protects in docs/secrets-and-pki.md, instead of shipping it to a third party.

Tip

The manager watches itself, as two independent things sharing one VM. The manager/indexer/dashboard Docker stack and the native agent monitoring that same host are unrelated — the manager isn't self-aware, it's just also enrolled as agent 000. And the two halves of "how Wazuh is configured" have different discipline levels on purpose: the compose stack itself is tracked in its own Gitea repo, but the actual detection rules and decoders live in a Docker volume, edited directly through the dashboard or manager API — not version-controlled. If the manager goes down entirely, agents don't lose events: they buffer locally and replay once it's back, so an outage costs visibility, not data.

A Fleet-wide Bug This System Caught In Itself

Important

Wazuh's own default agent log-queue buffer (1024 entries) overflowed fleet-wide during Ansible orchestration bursts — enough hosts logging enough events at once that the queue filled before the agent could forward them, silently dropping data rather than erroring loudly. Found, not assumed: root-caused to the default queue size specifically, then fixed with a dedicated tuning role that raises logcollector.queue_size on every agent. That role is deliberately narrow — it only touches queue size and enabled/started state; syscheck ignores, rootcheck ignores, and other per-agent tuning are still one-off manual edits, honestly left uncovered rather than implied as handled by a role that doesn't actually reach them. This is the real root-cause comment from the fix itself, not a summary of it:
$ cat wazuh_agent_tuning/tasks/main.yml
# Root-caused 2026-07-28: the default 1024-entry logcollector->agentd queue
# overflows during bursts of heavy journal activity (ansible fan-out runs
# touching many hosts at once, container restarts, etc.) - Wazuh's own
# "Agent event queue is full/90% full" rules (202/203) fired fleet-wide
# during that night's role redeploys, confirmed via the indexer across
# 9 of the fleet's agents.
The same role found a second gap a month later, in itself this time: it silently skipped tuning the wazuh host's own agent, because that agent runs inside the manager's Docker container rather than natively on the VM — the role's own "is the agent installed" check only ever looked for a native install. That let the wazuh host's own queue sit at the untuned default until a real queue-full alert exposed it, the same class of bug the role was originally written to fix, this time hiding in the one host built differently from the rest. Fixed by giving that host its own branch instead of assuming every agent looks the same.

How An Alert Gets Handled

The Curated "don't re-investigate this" Runbook

Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded — this is the difference between a SOC that actually scales past a handful of hosts and one that drowns in its own alert volume. A representative slice of what's actually in it, real rule IDs and all:

Rule(s)	What fires	Why it's routine
`5758` + cascade (`2501`/`5503`/`5557`/`5716`/`5760`)	Multiple-key offer during Ansible fan-out	One legitimate connection from the orchestration engine in `docs/orchestration.md`, not brute force
`553`/`554`	File-integrity "added/deleted"	Syscheck working as intended during the OS-level updates covered in `docs/disaster-recovery.md`
`52502`/`52503`	ClamAV's own harmless AV test-file trigger	Standard test artifact, not a real detection
`531`	One-off scratch-mount fill events	Self-resolving cache fills, not a real capacity issue
`80732`	SELinux/`container_runtime_t` interaction, isolated	Known container-runtime behavior, not exploitation
Daemon lifecycle noise	Agent start/stop, log rotation, PAM login/logout	Expected operational noise

Tip

One catch worth calling out in detail: rule 19011, "nftables default deny not detected," turned out to be a false positive in the check itself, not a real gap. The check greps for a raw nftables policy drop string that firewalld — the actual firewall in use, per docs/firewall-and-network.md — simply never produces, because firewalld implements default-deny through its own zone/chain structure instead. The fix was verifying the real firewall posture directly against firewalld's own state, not touching the firewall itself. This is a useful pattern in general: a security tool's own check can be wrong about a system it wasn't originally written for, and the fix is understanding the actual system, not chasing the tool's assumption.

Querying And Testing Directly

Alert data lives in a searchable OpenSearch-based index, reachable directly without going through the Dashy SOC Board — a terms-aggregation on rule.id, with sub-aggregations for rule.description, max rule.level, agent.name, and max timestamp, returns a fleet-wide "what's actually firing and where" summary in one call — useful when the dashboard's own pre-aggregated view isn't granular enough for a specific question.

Warning

Two gotchas that will burn a query if missed. Alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage, and a real time sink the first time it happens. And the manager's own container is named single-node-wazuh.manager-1, not the product's default wazuh.manager — an easy miss when scripting against it directly rather than through the dashboard.

Custom detection rules get tested before rollout, not after: a synthetic log line gets piped into wazuh-logtest, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op that would otherwise only surface once a real event failed to alert. This is a real synthetic SSH failure piped through the actual tool, not a description of what it would show:

$ echo 'Aug 28 12:00:00 caddy sshd[12345]: Failed password for invalid user testuser from 203.0.113.5 port 51234 ssh2' \
  | docker exec -i single-node-wazuh.manager-1 /var/ossec/bin/wazuh-logtest

	name: 'sshd'
	parent: 'sshd'
	srcip: '203.0.113.5'
	srcuser: 'testuser'

	id: '5710'
	level: '5'
	description: 'sshd: Attempt to login using a non-existent user'
	groups: '['syslog', 'sshd', 'authentication_failed', 'invalid_login']'
	mitre.id: '['T1110.001', 'T1021.004']'
	mitre.tactic: '['Credential Access', 'Lateral Movement']'
	pci_dss: '['10.2.4', '10.2.5', '10.6.1']'

One rule-writing gotcha that's easy to get backwards: a rule's <decoded_as> field has to reference the parent decoder's name, never a child regex-decoder further down the same chain — wazuh-logtest always reports the parent's name in its output regardless of which child actually matched, so debugging by what the tool prints back leads to the wrong name. And wazuh-logtest re-reads the rule/decoder files fresh on every invocation, independent of whether the live daemon has actually reloaded them — a rule that tests correctly can still be inert in production until wazuh-control restart actually loads it.

Compliance Hardening: justified exceptions, not blanket enforcement

This is the same tag-driven selection model described in docs/iac-pipeline.md — which controls apply to a given host is explicit in that host's own tag list, not an implicit "all or nothing" blanket. SCA scores below 100% on any given host are an accepted tradeoff, not tracked as open bugs to close one by one — the hardening rollout itself is the actual control, not a perfect per-control score. Two real deviations from the strict CIS baseline exist fleet-wide, and both are documented with their actual reasoning rather than left as unexplained exceptions — neither is a gap waiting to be closed, both are permanent, reasoned trade-offs. This is a real score, not a hypothetical one — one host's actual compliance dashboard:

The authentication stack is configured to fail safe under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards under pressure while writing the config.

Automated Benchmarking: native, not containerized

Important

The fleet's automated security-benchmark tool (Docker Bench for Security) runs as a native binary, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations (glibc vs. musl) that made it fail to run reliably. Running it natively instead of debugging a broken official container was the pragmatic call: the tool's job is auditing Docker, not testing container-runtime compatibility edge cases. It runs weekly, deliberately timed against real observed cron load from the other scheduled jobs described in docs/cron-scheduler.md, rather than an arbitrary default schedule that might collide with heavier jobs.

13 KiB Raw Permalink Blame History