public/docs/security-monitoring.md

# 🛡️ Security Monitoring

> [!NOTE]
> Wazuh is the detection engine, deployed as an agent on every host — file integrity monitoring, rootkit/anomaly heuristics, and compliance scoring, feeding one indexer that a curated runbook and a direct query workflow both draw from. This is the layer that actually watches the trust boundaries described everywhere else in this repo, not just documents them.

| | |
|---|---|
| **Software** | Wazuh (manager/indexer/dashboard), self-hosted |
| **Coverage** | 12 active agents, one per fleet host, no gaps |
| **Detection layers** | Compliance scoring (SCA) · file integrity monitoring (syscheck) · rootkit/anomaly heuristics (rootcheck) |
| **Runbook** | 10+ recurring alert types root-caused once, documented, never re-triaged from scratch |
| **Query path** | Direct index queries, bypassing the SOC dashboard entirely when needed |

This is the actual live agent roster, not a claimed count:

```
$ agent_control -l
ID: 000, Name: wazuh.manager (server), IP: 127.0.0.1, Active/Local
ID: 001, Name: hyper1, IP: any, Active
ID: 002, Name: hyper2, IP: any, Active
ID: 003, Name: pbs, IP: any, Active
ID: 004, Name: automation, IP: any, Active
ID: 006, Name: monitoring, IP: any, Active
ID: 007, Name: wazuh, IP: any, Active
ID: 008, Name: rocky-docker, IP: any, Active
ID: 009, Name: vault, IP: any, Active
ID: 010, Name: caddy, IP: any, Active
ID: 011, Name: technitium, IP: any, Active
ID: 012, Name: file-server, IP: any, Active
ID: 013, Name: netbird, IP: any, Active
```

## Why Agent-based, and Why Self-hosted

The alternative to an agent on every host is a purely network-level monitoring approach — watching traffic at the firewall in [`docs/firewall-and-network.md`](firewall-and-network.md) and inferring what's happening inside each host. That approach can't see file integrity changes, local privilege escalation, or rootkit behavior — the things that matter most once an attacker is already past the network perimeter. An agent on every host closes that gap at the cost of real per-host overhead and a rollout that has to cover 100% of the fleet to be trustworthy — a partial rollout would create a false sense of coverage that's arguably worse than no agent at all. Self-hosting the manager/indexer/dashboard stack, rather than using a SaaS SIEM, keeps every alert's raw data (hostnames, internal paths, credential-adjacent metadata) inside the same trust boundary Vault protects in [`docs/secrets-and-pki.md`](secrets-and-pki.md), instead of shipping it to a third party.

> [!TIP]
> **The manager watches itself, as two independent things sharing one VM.** The manager/indexer/dashboard Docker stack and the native agent monitoring that same host are unrelated — the manager isn't self-aware, it's just also enrolled as agent `000`. And the two halves of "how Wazuh is configured" have different discipline levels on purpose: the compose stack itself is tracked in its own Gitea repo, but the actual detection rules and decoders live in a Docker volume, edited directly through the dashboard or manager API — not version-controlled. If the manager goes down entirely, agents don't lose events: they buffer locally and replay once it's back, so an outage costs visibility, not data.

## A Fleet-wide Bug This System Caught In Itself

> [!IMPORTANT]
> Wazuh's own default agent log-queue buffer (1024 entries) overflowed fleet-wide during Ansible orchestration bursts — enough hosts logging enough events at once that the queue filled before the agent could forward them, silently dropping data rather than erroring loudly. Found, not assumed: root-caused to the default queue size specifically, then fixed with a dedicated tuning role that raises `logcollector.queue_size` on every agent. That role is deliberately narrow — it only touches queue size and enabled/started state; syscheck ignores, rootcheck ignores, and other per-agent tuning are still one-off manual edits, honestly left uncovered rather than implied as handled by a role that doesn't actually reach them. This is the real root-cause comment from the fix itself, not a summary of it:
>
> ```
> $ cat wazuh_agent_tuning/tasks/main.yml
> # Root-caused 2026-07-28: the default 1024-entry logcollector->agentd queue
> # overflows during bursts of heavy journal activity (ansible fan-out runs
> # touching many hosts at once, container restarts, etc.) - Wazuh's own
> # "Agent event queue is full/90% full" rules (202/203) fired fleet-wide
> # during that night's role redeploys, confirmed via the indexer across
> # 9 of the fleet's agents.
> ```

## How An Alert Gets Handled

![An alert either matches a known, already-root-caused pattern and goes straight to the runbook with no re-investigation, or it doesn't and gets investigated fresh via wazuh-logtest and a direct index query — a fresh investigation that becomes routine gets folded into the runbook for next time](../assets/security-monitoring-alert-flow.svg)

## The Curated "don't re-investigate this" Runbook

Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded — this is the difference between a SOC that actually scales past a handful of hosts and one that drowns in its own alert volume. A representative slice of what's actually in it, real rule IDs and all:

| Rule(s) | What fires | Why it's routine |
|---|---|---|
| `5758` + cascade (`2501`/`5503`/`5557`/`5716`/`5760`) | Multiple-key offer during Ansible fan-out | One legitimate connection from the orchestration engine in [`docs/orchestration.md`](orchestration.md), not brute force |
| `553`/`554` | File-integrity "added/deleted" | Syscheck working as intended during the OS-level updates covered in [`docs/disaster-recovery.md`](disaster-recovery.md) |
| `52502`/`52503` | ClamAV's own harmless AV test-file trigger | Standard test artifact, not a real detection |
| `531` | One-off scratch-mount fill events | Self-resolving cache fills, not a real capacity issue |
| `80732` | SELinux/`container_runtime_t` interaction, isolated | Known container-runtime behavior, not exploitation |
| Daemon lifecycle noise | Agent start/stop, log rotation, PAM login/logout | Expected operational noise |

> [!TIP]
> **One catch worth calling out in detail**: rule `19011`, "nftables default deny not detected," turned out to be a false positive *in the check itself*, not a real gap. The check greps for a raw nftables `policy drop` string that firewalld — the actual firewall in use, per [`docs/firewall-and-network.md`](firewall-and-network.md) — simply never produces, because firewalld implements default-deny through its own zone/chain structure instead. The fix was verifying the real firewall posture directly against firewalld's own state, not touching the firewall itself. This is a useful pattern in general: a security tool's own check can be wrong about a system it wasn't originally written for, and the fix is understanding the actual system, not chasing the tool's assumption.

## Querying And Testing Directly

Alert data lives in a searchable OpenSearch-based index, reachable directly without going through the [Dashy SOC Board](dashy-soc-board.md) — a terms-aggregation on `rule.id`, with sub-aggregations for `rule.description`, max `rule.level`, `agent.name`, and max `timestamp`, returns a fleet-wide "what's actually firing and where" summary in one call — useful when the dashboard's own pre-aggregated view isn't granular enough for a specific question.

> [!WARNING]
> Two gotchas that will burn a query if missed. Alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage, and a real time sink the first time it happens. And the manager's own container is named `single-node-wazuh.manager-1`, not the product's default `wazuh.manager` — an easy miss when scripting against it directly rather than through the dashboard.

Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into `wazuh-logtest`, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op that would otherwise only surface once a real event failed to alert. This is a real synthetic SSH failure piped through the actual tool, not a description of what it would show:

```
$ echo 'Aug 28 12:00:00 caddy sshd[12345]: Failed password for invalid user testuser from 203.0.113.5 port 51234 ssh2' \
  | docker exec -i single-node-wazuh.manager-1 /var/ossec/bin/wazuh-logtest

	name: 'sshd'
	parent: 'sshd'
	srcip: '203.0.113.5'
	srcuser: 'testuser'

	id: '5710'
	level: '5'
	description: 'sshd: Attempt to login using a non-existent user'
	groups: '['syslog', 'sshd', 'authentication_failed', 'invalid_login']'
	mitre.id: '['T1110.001', 'T1021.004']'
	mitre.tactic: '['Credential Access', 'Lateral Movement']'
	pci_dss: '['10.2.4', '10.2.5', '10.6.1']'
```

One rule-writing gotcha that's easy to get backwards: a rule's `<decoded_as>` field has to reference the *parent* decoder's name, never a child regex-decoder further down the same chain — `wazuh-logtest` always reports the parent's name in its output regardless of which child actually matched, so debugging by what the tool prints back leads to the wrong name. And `wazuh-logtest` re-reads the rule/decoder files fresh on every invocation, independent of whether the live daemon has actually reloaded them — a rule that tests correctly can still be inert in production until `wazuh-control restart` actually loads it.

## Compliance Hardening: justified exceptions, not blanket enforcement

![A CIS-oriented hardening framework of 9 tagged Ansible sub-roles applies cleanly to most hosts and gets enforced fleet-wide, re-applied every compliance run — but two controls are deliberately skipped with the reason documented: kernel forwarding flipped for Docker hosts since container networking needs it, and SSH TCP forwarding kept permissive fleet-wide since real remote access depends on it](../assets/security-monitoring-compliance-gate.svg)

This is the same tag-driven selection model described in [`docs/iac-pipeline.md`](iac-pipeline.md) — which controls apply to a given host is explicit in that host's own tag list, not an implicit "all or nothing" blanket. SCA scores below 100% on any given host are an accepted tradeoff, not tracked as open bugs to close one by one — the hardening rollout itself is the actual control, not a perfect per-control score. Two real deviations from the strict CIS baseline exist fleet-wide, and both are documented with their actual reasoning rather than left as unexplained exceptions — neither is a gap waiting to be closed, both are permanent, reasoned trade-offs. This is a real score, not a hypothetical one — one host's actual compliance dashboard:

![A real Wazuh endpoint dashboard: CIS Rocky Linux 10 Benchmark compliance at 87% (139 passed, 19 failed, 8 not applicable) — below 100%, exactly the accepted-tradeoff case described above, not a perfect score being claimed](../assets/security-monitoring-sca-dashboard.png)

The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards under pressure while writing the config.

## Automated Benchmarking: native, not containerized

> [!IMPORTANT]
> The fleet's automated security-benchmark tool (Docker Bench for Security) runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations (glibc vs. musl) that made it fail to run reliably. Running it natively instead of debugging a broken official container was the pragmatic call: the tool's job is auditing Docker, not testing container-runtime compatibility edge cases. It runs weekly, deliberately timed against real observed cron load from the other scheduled jobs described in [`docs/cron-scheduler.md`](cron-scheduler.md), rather than an arbitrary default schedule that might collide with heavier jobs.

---
Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md), [`docs/notifications.md`](notifications.md), [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/netbird-zero-trust.md`](netbird-zero-trust.md) · Stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)