public/docs/notifications.md

🔔 Notifications

Note

Two independent branches — app webhooks and OS-level host alerts — converge on a single automation flow and one shared formatter before reaching Slack, so every alert reads the same regardless of where it came from.

Fan-in, fan-out

flowchart LR
    subgraph FanIn["Fan-in"]
        Apps["15 apps<br/>HTTP webhook"]
        Dedicated["2 apps<br/>dedicated endpoints<br/>non-JSON payloads"]
        MQTT["Every host<br/>MQTT publish"]
    end

    Apps --> Router["App Router<br/>payload-shape detection"]
    Dedicated --> Router
    Router --> Mapper["Per-app mapper"]
    MQTT --> Parser["Inventory parser"]

    Mapper --> Subflow["Shared formatting subflow"]
    Parser --> Subflow
    Subflow --> Slack["Slack"]

Warning

Debt, documented rather than hidden: of the apps wired into the router, a couple no longer resolve to anything — one was fully decommissioned but its detection code was never swept out, and another's alerts actually travel a completely different path than its own webhook route suggests, making that route effectively dead code. Two newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own documented pattern for onboarding a new app's alerts.

Message broker

Broker	Runs as its own container on the automation host
Protocol	MQTT v3.1.1, auth required, anonymous disabled
Topic structure	One topic per host, wildcard-subscribed centrally
Users	Two — a publish identity used fleet-wide, and the automation flow's own read-only subscriber

Both broker users are credential-managed through the same secret-rendering pipeline as everything else in docs/secrets-and-pki.md — not a bare manually-edited password file.

Important

A real, tracked fragility: the publish identity's password has to stay in sync across three separate places — the rendered broker credential file, the source value used to render every host's own publish script, and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from the secret store at all). Rotating it means updating all three by hand — a documented gap, not a silent one.

Detection logic

The router figures out which app sent a webhook by inspecting the payload's shape — an explicit field for some apps, structural pattern-matching for others (one app, for instance, is identified purely by a nested field only that vendor's payload happens to include). Two apps skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON.

Every host, not just apps

Tip

A dedicated automation role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert. It runs over 20 distinct checks per host — disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (audit log anomalies, authentication failures) — deliberately complementary to the SOC stack in docs/security-monitoring.md, not duplicating its file-integrity or account monitoring.

One formatter, every source

Every branch — app webhooks, host-level alerts, and the two dedicated non-JSON endpoints — converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it.

---

Related stacks: stacks/platform/mosquitto/, stacks/platform/node-red/