# 🔔 Notifications

> [!NOTE]
> Two independent branches — app webhooks and OS-level host alerts — converge on a single automation flow and one shared formatter before reaching Slack, so every alert reads the same regardless of where it came from.

## Fan-in, fan-out

```mermaid
flowchart LR
    subgraph FanIn["Fan-in"]
        Apps["15 apps<br/>HTTP webhook"]
        Dedicated["2 apps<br/>dedicated endpoints<br/>non-JSON payloads"]
        MQTT["Every host<br/>MQTT publish"]
    end

    Apps --> Router["App Router<br/>payload-shape detection"]
    Dedicated --> Router
    Router --> Mapper["Per-app mapper"]
    MQTT --> Parser["Inventory parser"]

    Mapper --> Subflow["Shared formatting subflow"]
    Parser --> Subflow
    Subflow --> Slack["Slack"]
```

> [!WARNING]
> **Debt, documented rather than hidden**: of the apps wired into the router, a couple no longer resolve to anything — one was fully decommissioned but its detection code was never swept out, and another's alerts actually travel a completely different path than its own webhook route suggests, making that route effectively dead code. Two newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own documented pattern for onboarding a new app's alerts.

## Message broker

| | |
|---|---|
| **Broker** | Runs as its own container on the automation host |
| **Protocol** | MQTT v3.1.1, auth required, anonymous disabled |
| **Topic structure** | One topic per host, wildcard-subscribed centrally |
| **Users** | Two — a publish identity used fleet-wide, and the automation flow's own read-only subscriber |

Both broker users are credential-managed through the same secret-rendering pipeline as everything else in [`docs/secrets-and-pki.md`](secrets-and-pki.md) — not a bare manually-edited password file.

> [!IMPORTANT]
> **A real, tracked fragility**: the publish identity's password has to stay in sync across three separate places — the rendered broker credential file, the source value used to render every host's own publish script, and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from the secret store at all). Rotating it means updating all three by hand — a documented gap, not a silent one.

## Detection logic

The router figures out which app sent a webhook by inspecting the payload's shape — an explicit field for some apps, structural pattern-matching for others (one app, for instance, is identified purely by a nested field only that vendor's payload happens to include). Two apps skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON.

## Every host, not just apps

> [!TIP]
> A dedicated automation role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert. It runs over 20 distinct checks per host — disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (audit log anomalies, authentication failures) — deliberately complementary to the SOC stack in [`docs/security-monitoring.md`](security-monitoring.md), not duplicating its file-integrity or account monitoring.

## One formatter, every source

Every branch — app webhooks, host-level alerts, and the two dedicated non-JSON endpoints — converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it.

---
Related stacks: [`stacks/platform/mosquitto/`](../stacks/platform/mosquitto/), [`stacks/platform/node-red/`](../stacks/platform/node-red/)