docs: rewrite security-monitoring.md with real evidence and diagrams
Live-verified against the real Wazuh manager (agent_control -l: 11 active agents matching exactly; docker ps: real manager container name) and stamped both source KB notes. Replaced both mermaid flowcharts with real branch-shaped Nextcloud diagrams (alert-handling gate, compliance-hardening gate). Added real evidence throughout: the live agent roster, a real synthetic SSH-failure line piped through wazuh-logtest showing actual decoding/rule-match output (MITRE ATT&CK + compliance tags included), and real rule IDs in the runbook table instead of vague categories. New content: the fleet-wide agent log-queue-overflow bug found and fixed via a dedicated tuning role, and the compose-stack-vs-rules version-control split.
This commit is contained in:
3 files changed
+71
-40
No files matched your search
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Do not edit this file with editors other than draw.io -->
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="947px" height="612px" viewBox="0 0 947 612" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Alert Handling Flow" id="alert-flow-1"> <mxGraphModel dx="1414" dy="791" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1000" pageHeight="640" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="host" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#F5F5F5;strokeColor=#666666;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;" value="Fleet Host&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;Wazuh agent installed&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="200" x="400" y="20" as="geometry" /> </mxCell> <mxCell id="checks" parent="1" style="rounded=1;whiteSpace=wrap;html=1;startSize=26;swimlane;fillColor=none;strokeColor=#666666;dashed=1;fontSize=12;fontStyle=1;fontColor=#333333;align=center;" value="DETECTION LAYERS" vertex="1"> <mxGeometry height="110" width="620" x="190" y="130" as="geometry" /> </mxCell> <mxCell id="sca" parent="checks" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="SCA compliance scoring&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;12h benchmark scan&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="180" x="20" y="35" as="geometry" /> </mxCell> <mxCell id="fim" parent="checks" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="File integrity monitoring&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;12h scan cycle&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="180" x="220" y="35" as="geometry" /> </mxCell> <mxCell id="rootcheck" parent="checks" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Rootkit / anomaly&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;heuristics&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="180" x="420" y="35" as="geometry" /> </mxCell> <mxCell id="alert" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;" value="Alert fires" vertex="1"> <mxGeometry height="60" width="180" x="410" y="290" as="geometry" /> </mxCell> <mxCell id="gate" parent="1" style="rhombus;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Matches a known,&lt;br&gt;root-caused pattern?" vertex="1"> <mxGeometry height="120" width="220" x="390" y="390" as="geometry" /> </mxCell> <mxCell id="runbook" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Runbook entry&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;no re-investigation&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="200" x="120" y="410" asLine truncated
|
||||
|
After Width: | Height: | Size: 336 KiB |
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Do not edit this file with editors other than draw.io -->
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1072px" height="477px" viewBox="0 0 1072 477" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Compliance Gate" id="compliance-gate-1"> <mxGraphModel dx="1414" dy="791" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1000" pageHeight="560" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="framework" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="CIS-oriented hardening framework&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;9 tagged Ansible sub-roles&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="260" x="370" y="30" as="geometry" /> </mxCell> <mxCell id="gate" parent="1" style="rhombus;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Applies cleanly&lt;br&gt;to this host?" vertex="1"> <mxGeometry height="120" width="220" x="390" y="150" as="geometry" /> </mxCell> <mxCell id="enforce" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Enforced fleet-wide&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;re-applied every compliance run —&lt;br&gt;not just at creation&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="260" x="60" y="170" as="geometry" /> </mxCell> <mxCell id="skip" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Deliberately skipped&lt;br&gt;&lt;font style=&quot;font-size: 11px;&quot;&gt;reason documented&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="240" x="710" y="175" as="geometry" /> </mxCell> <mxCell id="r1" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F1E);strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Kernel forwarding control&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;flipped for Docker hosts —&lt;br&gt;container networking needs it&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="260" x="580" y="330" as="geometry" /> </mxCell> <mxCell id="r2" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F1E);strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="SSH TCP forwarding&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;kept permissive fleet-wide —&lt;br&gt;real remote access depends on it&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="260" x="860" y="330" as="geometry" /> </mxCell> <mxCell id="e1" edge="1" parent="1" source="framework" style="html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;" target="gate"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="e2" edge="1" parent="1" source="gate" style="html=1;labelBackgroundColor=#1C1D1E;endArrow=block;strokeColor=#82b366;fontColor=#82b366;strokeWidth=2;rounded=1;fontSize=12;" target="enforceLine truncated
|
||||
|
After Width: | Height: | Size: 320 KiB |
+63
-40
@@ -6,72 +6,95 @@
|
||||
| | |
|
||||
|---|---|
|
||||
| **Software** | Wazuh (manager/indexer/dashboard), self-hosted |
|
||||
| **Coverage** | Every fleet host, agent-based, no gaps |
|
||||
| **Detection layers** | Compliance scoring · file integrity monitoring · rootkit/anomaly heuristics |
|
||||
| **Coverage** | 11 active agents, one per fleet host, no gaps |
|
||||
| **Detection layers** | Compliance scoring (SCA) · file integrity monitoring (syscheck) · rootkit/anomaly heuristics (rootcheck) |
|
||||
| **Runbook** | 10+ recurring alert types root-caused once, documented, never re-triaged from scratch |
|
||||
| **Query path** | Direct index queries, bypassing the SOC dashboard entirely when needed |
|
||||
|
||||
This is the actual live agent roster, not a claimed count:
|
||||
|
||||
```
|
||||
$ agent_control -l
|
||||
ID: 000, Name: wazuh.manager (server), IP: 127.0.0.1, Active/Local
|
||||
ID: 001, Name: hyper1, IP: any, Active
|
||||
ID: 002, Name: hyper2, IP: any, Active
|
||||
ID: 003, Name: pbs, IP: any, Active
|
||||
ID: 004, Name: automation, IP: any, Active
|
||||
ID: 006, Name: monitoring, IP: any, Active
|
||||
ID: 007, Name: wazuh, IP: any, Active
|
||||
ID: 008, Name: rocky-docker, IP: any, Active
|
||||
ID: 009, Name: vault, IP: any, Active
|
||||
ID: 010, Name: caddy, IP: any, Active
|
||||
ID: 011, Name: technitium, IP: any, Active
|
||||
ID: 012, Name: file-server, IP: any, Active
|
||||
```
|
||||
|
||||
## Why agent-based, and why self-hosted
|
||||
|
||||
The alternative to an agent on every host is a purely network-level monitoring approach — watching traffic at the firewall in [`docs/firewall-and-network.md`](firewall-and-network.md) and inferring what's happening inside each host. That approach can't see file integrity changes, local privilege escalation, or rootkit behavior — the things that matter most once an attacker is already past the network perimeter. An agent on every host closes that gap at the cost of real per-host overhead and a rollout that has to cover 100% of the fleet to be trustworthy — a partial rollout would create a false sense of coverage that's arguably worse than no agent at all. Self-hosting the manager/indexer/dashboard stack, rather than using a SaaS SIEM, keeps every alert's raw data (hostnames, internal paths, credential-adjacent metadata) inside the same trust boundary Vault protects in [`docs/secrets-and-pki.md`](secrets-and-pki.md), instead of shipping it to a third party.
|
||||
|
||||
> [!TIP]
|
||||
> **The manager watches itself, as two independent things sharing one VM.** The manager/indexer/dashboard Docker stack and the native agent monitoring that same host are unrelated — the manager isn't self-aware, it's just also enrolled as agent `000`. And the two halves of "how Wazuh is configured" have different discipline levels on purpose: the compose stack itself is tracked in its own Gitea repo, but the actual detection rules and decoders live in a Docker volume, edited directly through the dashboard or manager API — not version-controlled. If the manager goes down entirely, agents don't lose events: they buffer locally and replay once it's back, so an outage costs visibility, not data.
|
||||
|
||||
## A fleet-wide bug this system caught in itself
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Wazuh's own default agent log-queue buffer (1024 entries) overflowed fleet-wide during Ansible orchestration bursts — enough hosts logging enough events at once that the queue filled before the agent could forward them, silently dropping data rather than erroring loudly. Found, not assumed: root-caused to the default queue size specifically, then fixed with a dedicated tuning role that raises `logcollector.queue_size` on every agent. That role is deliberately narrow — it only touches queue size and enabled/started state; syscheck ignores, rootcheck ignores, and other per-agent tuning are still one-off manual edits, honestly left uncovered rather than implied as handled by a role that doesn't actually reach them.
|
||||
|
||||
## How an alert gets handled
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Host["Fleet Host<br/>Wazuh agent installed"] --> Checks
|
||||
|
||||
subgraph Checks["Detection layers"]
|
||||
direction TB
|
||||
SCA["SCA compliance scoring<br/>12h benchmark scan"]
|
||||
FIM["File integrity monitoring<br/>12h scan cycle"]
|
||||
RC["Rootkit / anomaly<br/>heuristics"]
|
||||
end
|
||||
|
||||
Checks --> Alert["Alert fires"]
|
||||
Alert --> Known{"Matches a known,<br/>root-caused pattern?"}
|
||||
Known -->|yes| Runbook["Runbook entry<br/>no re-investigation"]
|
||||
Known -->|no| Fresh["Investigated fresh<br/>wazuh-logtest + direct index query"]
|
||||
Fresh -->|becomes routine| Runbook
|
||||
```
|
||||
|
||||
|
||||
## The curated "don't re-investigate this" runbook
|
||||
|
||||
Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded — this is the difference between a SOC that actually scales past a handful of hosts and one that drowns in its own alert volume:
|
||||
Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded — this is the difference between a SOC that actually scales past a handful of hosts and one that drowns in its own alert volume. A representative slice of what's actually in it, real rule IDs and all:
|
||||
|
||||
| Category | What fires | Why it's routine |
|
||||
| Rule(s) | What fires | Why it's routine |
|
||||
|---|---|---|
|
||||
| Daemon lifecycle | Agent start/stop, log rotation, PAM login/logout | Expected operational noise |
|
||||
| Package/kernel updates | File-integrity "added/deleted" events | Syscheck working as intended during the OS-level updates covered in [`docs/disaster-recovery.md`](disaster-recovery.md) |
|
||||
| Antivirus self-test | A known harmless AV test-file trigger | Standard test artifact, not a real detection |
|
||||
| Transient disk-full | One-off scratch-mount fill events | Self-resolving cache fills, not a real capacity issue |
|
||||
| SSH auth bursts from automation | Multiple-key offer during fleet-wide orchestration runs | One legitimate connection from the orchestration engine in [`docs/orchestration.md`](orchestration.md), not brute force |
|
||||
| Container-runtime access-control denials | Isolated, non-flooding events | Known container/runtime interaction, not exploitation |
|
||||
| `5758` + cascade (`2501`/`5503`/`5557`/`5716`/`5760`) | Multiple-key offer during Ansible fan-out | One legitimate connection from the orchestration engine in [`docs/orchestration.md`](orchestration.md), not brute force |
|
||||
| `553`/`554` | File-integrity "added/deleted" | Syscheck working as intended during the OS-level updates covered in [`docs/disaster-recovery.md`](disaster-recovery.md) |
|
||||
| `52502`/`52503` | ClamAV's own harmless AV test-file trigger | Standard test artifact, not a real detection |
|
||||
| `531` | One-off scratch-mount fill events | Self-resolving cache fills, not a real capacity issue |
|
||||
| `80732` | SELinux/`container_runtime_t` interaction, isolated | Known container-runtime behavior, not exploitation |
|
||||
| Daemon lifecycle noise | Agent start/stop, log rotation, PAM login/logout | Expected operational noise |
|
||||
|
||||
> [!TIP]
|
||||
> **One catch worth calling out in detail**: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive *in the check itself*, not a real gap. The underlying check searched for a raw configuration string that firewalld — the actual firewall in use, per [`docs/firewall-and-network.md`](firewall-and-network.md) — simply never produces, because firewalld implements default-deny through its own zone/chain structure rather than the specific policy string the check was written to expect. The fix was verifying the real firewall posture directly against firewalld's own state, not touching the firewall itself. This is a useful pattern in general: a security tool's own check can be wrong about a system it wasn't originally written for, and the fix is understanding the actual system, not chasing the tool's assumption.
|
||||
> **One catch worth calling out in detail**: rule `19011`, "nftables default deny not detected," turned out to be a false positive *in the check itself*, not a real gap. The check greps for a raw nftables `policy drop` string that firewalld — the actual firewall in use, per [`docs/firewall-and-network.md`](firewall-and-network.md) — simply never produces, because firewalld implements default-deny through its own zone/chain structure instead. The fix was verifying the real firewall posture directly against firewalld's own state, not touching the firewall itself. This is a useful pattern in general: a security tool's own check can be wrong about a system it wasn't originally written for, and the fix is understanding the actual system, not chasing the tool's assumption.
|
||||
|
||||
## Querying and testing directly
|
||||
|
||||
Alert data lives in a searchable OpenSearch-based index, reachable without going through the [Dashy SOC Board](dashy-soc-board.md) — a single aggregation query returns a fleet-wide "what's actually firing and where" summary in one call, useful when the dashboard's own pre-aggregated view isn't granular enough for a specific question.
|
||||
Alert data lives in a searchable OpenSearch-based index, reachable directly without going through the [Dashy SOC Board](dashy-soc-board.md) — a terms-aggregation on `rule.id`, with sub-aggregations for `rule.description`, max `rule.level`, `agent.name`, and max `timestamp`, returns a fleet-wide "what's actually firing and where" summary in one call — useful when the dashboard's own pre-aggregated view isn't granular enough for a specific question.
|
||||
|
||||
> [!WARNING]
|
||||
> Two gotchas that will burn a query if missed: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage, and a real time sink to debug the first time it happens. And the manager's own container name doesn't match the product's default name, an easy miss when scripting against it directly rather than through the dashboard.
|
||||
> Two gotchas that will burn a query if missed. Alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage, and a real time sink the first time it happens. And the manager's own container is named `single-node-wazuh.manager-1`, not the product's default `wazuh.manager` — an easy miss when scripting against it directly rather than through the dashboard.
|
||||
|
||||
Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into `wazuh-logtest`, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op that would otherwise only surface once a real event failed to alert.
|
||||
Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into `wazuh-logtest`, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op that would otherwise only surface once a real event failed to alert. This is a real synthetic SSH failure piped through the actual tool, not a description of what it would show:
|
||||
|
||||
```
|
||||
$ echo 'Aug 28 12:00:00 caddy sshd[12345]: Failed password for invalid user testuser from 203.0.113.5 port 51234 ssh2' \
|
||||
| docker exec -i single-node-wazuh.manager-1 /var/ossec/bin/wazuh-logtest
|
||||
|
||||
name: 'sshd'
|
||||
parent: 'sshd'
|
||||
srcip: '203.0.113.5'
|
||||
srcuser: 'testuser'
|
||||
|
||||
id: '5710'
|
||||
level: '5'
|
||||
description: 'sshd: Attempt to login using a non-existent user'
|
||||
groups: '['syslog', 'sshd', 'authentication_failed', 'invalid_login']'
|
||||
mitre.id: '['T1110.001', 'T1021.004']'
|
||||
mitre.tactic: '['Credential Access', 'Lateral Movement']'
|
||||
pci_dss: '['10.2.4', '10.2.5', '10.6.1']'
|
||||
```
|
||||
|
||||
One rule-writing gotcha that's easy to get backwards: a rule's `<decoded_as>` field has to reference the *parent* decoder's name, never a child regex-decoder further down the same chain — `wazuh-logtest` always reports the parent's name in its output regardless of which child actually matched, so debugging by what the tool prints back leads to the wrong name. And `wazuh-logtest` re-reads the rule/decoder files fresh on every invocation, independent of whether the live daemon has actually reloaded them — a rule that tests correctly can still be inert in production until `wazuh-control restart` actually loads it.
|
||||
|
||||
## Compliance hardening: justified exceptions, not blanket enforcement
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Framework["CIS-oriented hardening framework<br/>9 tagged Ansible sub-roles"] --> Apply{"Applies cleanly<br/>to this host?"}
|
||||
Apply -->|yes| Enforce["Enforced fleet-wide,<br/>re-applied every compliance run —<br/>not just at creation"]
|
||||
Apply -->|no, justified| Skip["Deliberately skipped<br/>reason documented"]
|
||||
Skip --> R1["Kernel forwarding control<br/>flipped for Docker hosts —<br/>container networking needs it"]
|
||||
Skip --> R2["SSH TCP forwarding<br/>kept permissive fleet-wide —<br/>real remote access depends on it"]
|
||||
```
|
||||
|
||||
|
||||
This is the same tag-driven selection model described in [`docs/iac-pipeline.md`](iac-pipeline.md) — which controls apply to a given host is explicit in that host's own tag list, not an implicit "all or nothing" blanket. Two real deviations from the strict CIS baseline exist, and both are documented with their actual reasoning rather than left as unexplained exceptions: one kernel networking control is loosened specifically for Docker hosts because container networking genuinely requires it, and SSH TCP forwarding stays enabled fleet-wide because real remote access depends on it. Neither is a gap waiting to be closed — both are permanent, reasoned trade-offs.
|
||||
This is the same tag-driven selection model described in [`docs/iac-pipeline.md`](iac-pipeline.md) — which controls apply to a given host is explicit in that host's own tag list, not an implicit "all or nothing" blanket. SCA scores below 100% on any given host are an accepted tradeoff, not tracked as open bugs to close one by one — the hardening rollout itself is the actual control, not a perfect per-control score. Two real deviations from the strict CIS baseline exist fleet-wide, and both are documented with their actual reasoning rather than left as unexplained exceptions — neither is a gap waiting to be closed, both are permanent, reasoned trade-offs.
|
||||
|
||||
The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards under pressure while writing the config.
|
||||
|
||||
|
||||
Reference in new issue
Block a user