# 🛡️ Security Monitoring

> [!NOTE]
> A self-built SOC stack: fleet-wide agent monitoring, a curated runbook for known-noisy alerts, and compliance hardening rolled out with deliberately justified exceptions instead of blanket enforcement.

| | |
|---|---|
| **Coverage** | Every fleet host, agent-based, no gaps |
| **Detection layers** | Compliance scoring · file integrity monitoring · rootkit/anomaly heuristics |
| **Runbook** | 10+ recurring alert types root-caused once, documented, never re-triaged from scratch |
| **Query path** | Direct index queries, bypassing the dashboard entirely when needed |

## How an alert gets handled

```mermaid
flowchart LR
    Host["Fleet Host<br/>agent installed"] --> Checks

    subgraph Checks["Detection layers"]
        direction TB
        SCA["Compliance scoring<br/>12h benchmark scan"]
        FIM["File integrity monitoring<br/>12h scan cycle"]
        RC["Rootkit / anomaly<br/>heuristics"]
    end

    Checks --> Alert["Alert fires"]
    Alert --> Known{"Matches a known,<br/>root-caused pattern?"}
    Known -->|yes| Runbook["Runbook entry<br/>no re-investigation"]
    Known -->|no| Fresh["Investigated fresh<br/>logtest + direct index query"]
    Fresh -->|becomes routine| Runbook
```

## The curated "don't re-investigate this" runbook

Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded:

| Category | What fires | Why it's routine |
|---|---|---|
| Daemon lifecycle | Agent start/stop, log rotation, PAM login/logout | Expected operational noise |
| Package/kernel updates | File-integrity "added/deleted" events | Syscheck working as intended during normal patching |
| Antivirus self-test | A known harmless AV test-file trigger | Standard test artifact, not a real detection |
| Transient disk-full | One-off scratch-mount fill events | Self-resolving cache fills, not a real capacity issue |
| SSH auth bursts from automation | Multiple-key offer during fleet-wide automation runs | One legitimate connection, not brute force |
| Container-runtime access-control denials | Isolated, non-flooding events | Known container/runtime interaction, not exploitation |

> [!TIP]
> **One catch worth calling out**: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive *in the check itself* — it searched for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall. Chasing the alert instead of the actual policy would have wasted real time on a non-problem.

## Querying and testing directly

Alert data lives in a searchable index, reachable without going through any dashboard — a single aggregation query returns a fleet-wide "what's actually firing and where" summary in one call.

> [!WARNING]
> Two gotchas that will burn a query if missed: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage. And the manager's own container name doesn't match the product's default name, an easy miss when scripting against it.

Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.

## Compliance hardening: justified exceptions, not blanket enforcement

```mermaid
flowchart TD
    Framework["Multi-role hardening framework"] --> Apply{"Applies cleanly<br/>to this host?"}
    Apply -->|yes| Enforce["Enforced fleet-wide"]
    Apply -->|no, justified| Skip["Deliberately skipped<br/>reason documented"]
    Skip --> R1["Control doesn't apply<br/>inside a container"]
    Skip --> R2["Needs a disk layout<br/>not every host has"]
    Skip --> R3["Only makes sense on<br/>always-on VMs"]
```

The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards.

## Automated benchmarking: native, not containerized

> [!IMPORTANT]
> The fleet's automated security-benchmark tool runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations. It runs weekly, deliberately timed against real observed load data rather than an arbitrary default schedule.

---
Related stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)