# 🔔 Notifications

> [!NOTE]
> Mosquitto (MQTT) and a single Node-RED instance are the backbone here — two independent fan-in branches, app webhooks and OS-level host alerts, converging on one flow and one shared formatter before reaching Slack. The point isn't the individual pieces; it's that every alert reads the same regardless of where it came from.

## Fan-in, fan-out

```mermaid
flowchart LR
    subgraph FanIn["Fan-in"]
        Apps["15 apps<br/>HTTP webhook"]
        Dedicated["2 apps<br/>dedicated endpoints<br/>non-JSON payloads"]
        MQTT["Every host<br/>MQTT publish"]
    end

    Apps --> Router["Node-RED App Router<br/>payload-shape detection"]
    Dedicated --> Router
    Router --> Mapper["Per-app mapper"]
    MQTT --> Parser["Inventory parser"]

    Mapper --> Subflow["Shared formatting subflow"]
    Parser --> Subflow
    Subflow --> Slack["Slack"]
```

## Why one Node-RED instance instead of per-app scripts

The alternative — each app posting directly to a per-app Slack webhook, or a small script per integration — was rejected because it would mean N different message formats, N different places to add a new integration's logic, and no single point to add fleet-wide behavior (like the OS-level checks below) without touching every app individually. One Node-RED instance with a shared router and a shared formatting subflow means a new app's alerts get the same visual treatment as every existing one automatically, and a fleet-wide change (say, adding a severity-based routing rule) happens in one place.

> [!WARNING]
> **Debt, documented rather than hidden**: of the apps wired into the router, a couple no longer resolve to anything — one was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) but its detection code was never swept out of the flow, and another's alerts actually travel a completely different path than its own webhook route suggests, making that route effectively dead code. Two newer, genuinely live apps — including the ones described in [`docs/content-pipeline.md`](content-pipeline.md) — aren't wired into either branch yet. Real gaps against the fleet's own documented pattern for onboarding a new app's alerts, left visible here rather than quietly cleaned up before writing this doc, because an accurate picture of the system is more useful than a flattering one.

## Message broker

| | |
|---|---|
| **Broker** | Mosquitto, self-hosted, runs as its own container on the automation host |
| **Protocol** | MQTT v3.1.1, auth required, anonymous disabled |
| **Topic structure** | One topic per host, wildcard-subscribed centrally |
| **Users** | Two — a publish identity used fleet-wide, and Node-RED's own read-only subscriber |

Both broker users are credential-managed through the same Vault-rendered pipeline as everything else in [`docs/secrets-and-pki.md`](secrets-and-pki.md) — not a bare manually-edited password file.

> [!IMPORTANT]
> **A real, tracked fragility, worth naming precisely**: the publish identity's password has to stay in sync across three separate places — the Vault-rendered broker credential file, the source value used to render every host's own publish script (via the same tag-driven Ansible model in [`docs/iac-pipeline.md`](iac-pipeline.md)), and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from Vault at all, unlike everything else in [`docs/secret-distribution.md`](secret-distribution.md)). Rotating it means updating all three by hand — a documented, real exception to the "one source of truth" rule that governs every other secret in this fleet, called out explicitly rather than left for someone to discover mid-rotation.

## Detection logic

The Node-RED App Router figures out which app sent a webhook by inspecting the payload's shape — an explicit field for some apps, structural pattern-matching for others (one app, for instance, is identified purely by a nested field only that vendor's payload happens to include, since it never sends an explicit app identifier). Two apps skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON and the shared router's shape-detection logic simply doesn't apply to them.

## Every host, not just apps

> [!TIP]
> A dedicated Ansible role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert — the same fleet-wide, tag-driven rollout pattern used for the observability agent in [`docs/observability.md`](observability.md) and the compliance hardening in [`docs/security-monitoring.md`](security-monitoring.md). It runs over 20 distinct checks per host — disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (Vault audit log anomalies, Vault Agent authentication failures) — deliberately complementary to Wazuh in [`docs/security-monitoring.md`](security-monitoring.md), not duplicating its file-integrity or account monitoring. Splitting OS-health checks from the SOC's own detection engine keeps each system's role narrow enough to actually reason about, instead of one monolithic monitoring tool trying to do both jobs at once.

## One formatter, every source

Every branch — app webhooks, host-level alerts, and the two dedicated non-JSON endpoints — converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it — a reader in Slack never has to learn a different alert format depending on which system fired it.

---
Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/observability.md`](observability.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/mosquitto/`](../stacks/platform/mosquitto/), [`stacks/platform/node-red/`](../stacks/platform/node-red/)