118 lines
10 KiB
Markdown
118 lines
10 KiB
Markdown
<!-- verified: 2026-09-06
|
|
automation-vm@7c057de : mosquitto
|
|
iac@49ff427 : ansible/roles/monitoring/mqtt_client
|
|
trilium : tt4idCVJPpZM, EAU5LUwFFQ0E, rGfAb4CddKMi, otSPaU6IfjlP, klrmnlh3Wrnd
|
|
-->
|
|
|
|
# 🔔 Notifications
|
|
|
|
> [!NOTE]
|
|
> Mosquitto (MQTT) and a single Node-RED instance are the backbone here — two independent fan-in branches, app webhooks and OS-level host alerts, converging on one flow and one shared formatter before reaching Slack. The point isn't the individual pieces; it's that every alert reads the same regardless of where it came from.
|
|
|
|
## Fan-in, Fan-out
|
|
|
|
|
|
|
|
## Why One Node-RED Instance Instead Of Per-App Scripts
|
|
|
|
The alternative — each app posting directly to a per-app Slack webhook, or a small script per integration — was rejected because it would mean N different message formats, N different places to add a new integration's logic, and no single point to add fleet-wide behavior (like the OS-level checks below) without touching every app individually. One Node-RED instance with a shared router and a shared formatting subflow means a new app's alerts get the same visual treatment as every existing one automatically, and a fleet-wide change (say, adding a severity-based routing rule) happens in one place.
|
|
|
|
> [!WARNING]
|
|
> **Real debt in the flow**: of the apps wired into the router, two no longer resolve to anything — NetBox was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) but its detection code was never swept out, and Kestra's alerts actually travel a completely different path than its own webhook route suggests: Kestra publishes its own alerts over MQTT, not HTTP, which makes its dedicated webhook route effectively dead code. A couple of newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own onboarding pattern for a new app's alerts.
|
|
|
|
This is the real routing table, not a claimed count — counted directly from the flow export, not restated from memory:
|
|
|
|
```
|
|
$ python3 -c "
|
|
import json
|
|
flows = json.load(open('flows.json'))
|
|
switch = next(n for n in flows if n.get('type')=='switch')
|
|
print('Switch rules:', len(switch['rules']))
|
|
dedicated = [n for n in flows if n.get('type')=='http in' and n.get('url','').endswith('-webhook') and n.get('url') != '/app-webhook']
|
|
print('Dedicated non-router endpoints:', len(dedicated))
|
|
"
|
|
Switch rules: 16
|
|
Dedicated non-router endpoints: 2
|
|
```
|
|
|
|
16 rules = the 15 routed apps plus one `unknown` fallback that just drops the payload. And this is the real function behind the "completely different path" line above — the MQTT branch doesn't parse `msg.topic` looking for a known set of fleet hosts, it just takes whatever the second path segment is:
|
|
|
|
```
|
|
$ grep -A4 '"Inventory Parse Function' flows.json
|
|
const parts = msg.topic.split("/");
|
|
msg.inventory = parts[1];
|
|
msg.channel = `#${msg.inventory}`;
|
|
msg.slackData = msg.payload;
|
|
return msg;
|
|
```
|
|
|
|
Topic-agnostic by construction — so a publisher that isn't a fleet host at all gets handled identically to one that is, which is exactly the mechanism that makes an app's *own* MQTT-published alerts real and its dedicated webhook route genuinely dead, not just theoretically bypassable.
|
|
|
|
This is the real flow, not a redrawn version of it — the App Router feeding a 16-way Switch, NetBox's and Kestra's mapper functions still sitting there wired in and unreachable, Kopia and Bazarr's dedicated endpoints bypassing all of it entirely on their way to the same shared subflow:
|
|
|
|
|
|
|
|
## Message Broker
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Broker** | Mosquitto, self-hosted, runs as its own container on the automation host |
|
|
| **Protocol** | MQTT v3.1.1, auth required, anonymous disabled |
|
|
| **Topic structure** | One topic per host, wildcard-subscribed centrally |
|
|
| **Users** | Two — a publish identity used fleet-wide, and Node-RED's own read-only subscriber |
|
|
|
|
Both broker users are credential-managed through the same Vault-rendered pipeline as everything else in [`docs/secrets-and-pki.md`](secrets-and-pki.md) — not a bare manually-edited password file. This is the real broker config and ACL, not a description of the intended setup:
|
|
|
|
```
|
|
$ cut -d: -f1 mosquitto/config/passwords
|
|
notifier
|
|
node-red
|
|
|
|
$ cat mosquitto/config/acl
|
|
user notifier
|
|
topic write homelab/+/notify
|
|
topic read #
|
|
|
|
user node-red
|
|
topic read #
|
|
|
|
$ grep -E 'listener|allow_anonymous|protocol' mosquitto/config/mosquitto.conf
|
|
listener 1883
|
|
protocol mqtt
|
|
allow_anonymous false
|
|
```
|
|
|
|
The ACL is the concrete version of "publish identity" and "read-only subscriber" above: `notifier` can write only to its own topic pattern (plus read access to the full wildcard, not write-only despite the name), `node-red` can only ever read.
|
|
|
|
And this is the broker actually being used, not just configured — real connection log, including Kestra's own `kestra-notify` client publishing under the `notifier` identity, the same live evidence behind the "Kestra's alerts travel over MQTT" claim above:
|
|
|
|
|
|
|
|
> [!IMPORTANT]
|
|
> **A real, tracked fragility, worth naming precisely**: the publish identity's password has to stay in sync across three separate places — the Vault-rendered broker credential file, the source value used to render every host's own publish script (via the same tag-driven Ansible model in [`docs/iac-pipeline.md`](iac-pipeline.md)), and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from Vault at all, unlike everything else in [`docs/secret-distribution.md`](secret-distribution.md)). Rotating it means updating all three by hand — a documented, real exception to the "one source of truth" rule that governs every other secret in this fleet, called out explicitly rather than left for someone to discover mid-rotation.
|
|
|
|
## Detection Logic
|
|
|
|
The Node-RED App Router figures out which app sent a webhook by inspecting the payload's shape — an explicit field for some apps, structural pattern-matching for others (Emby, for instance, is identified purely by a nested `data.Server` field only that vendor's payload happens to include, since it never sends an explicit app identifier). Kopia and Bazarr skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON and the shared router's shape-detection logic simply doesn't apply to them.
|
|
|
|
## Every Host, Not Just Apps
|
|
|
|
> [!TIP]
|
|
> A dedicated Ansible role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert — the same fleet-wide, tag-driven rollout pattern used for the observability agent in [`docs/observability.md`](observability.md) and the compliance hardening in [`docs/security-monitoring.md`](security-monitoring.md). It runs over 20 distinct checks per host — disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (Vault audit log anomalies, Vault Agent authentication failures) — deliberately complementary to Wazuh in [`docs/security-monitoring.md`](security-monitoring.md), not duplicating its file-integrity or account monitoring. Splitting OS-health checks from the SOC's own detection engine keeps each system's role narrow enough to actually reason about, instead of one monolithic monitoring tool trying to do both jobs at once.
|
|
>
|
|
> This is a real count against the role's own script, not the doc's own claim restated:
|
|
>
|
|
> ```
|
|
> $ grep -oE '\$STATE_DIR/[a-zA-Z_]+' mqtt-monitor.sh.j2 | sed -E 's/(_alert|_last_count|_since|_report)$//' | sort -u | wc -l
|
|
> 22
|
|
> ```
|
|
>
|
|
> 22 from that count, plus disk-usage thresholds — which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name — makes 23. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts.
|
|
|
|
## One Formatter, Every Source
|
|
|
|
Every branch — app webhooks, host-level alerts, and Kopia/Bazarr's own dedicated non-JSON endpoints — converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it — a reader in Slack never has to learn a different alert format depending on which system fired it.
|
|
|
|
---
|
|
Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/observability.md`](observability.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/mosquitto/`](../stacks/platform/mosquitto/), [`stacks/platform/node-red/`](../stacks/platform/node-red/)
|