diff --git a/docs/notifications.md b/docs/notifications.md index 700e58f..6b62d73 100644 --- a/docs/notifications.md +++ b/docs/notifications.md @@ -1,6 +1,6 @@ - @@ -11,14 +11,14 @@ ## Fan-in, Fan-out -![Three independent fan-in sources converging on one shared formatting subflow: 15 apps over HTTP webhook go through the App Router and Switch to a per-app mapper, Kopia/Bazarr post to their own dedicated endpoints and bypass the router entirely with their own mapper, and every host publishes over MQTT into a topic-agnostic Inventory Parse function — all three land on the same subflow before reaching Slack](../assets/notifications-fan-in-fan-out.svg) +![Three independent fan-in sources converging on one shared formatting subflow: 14 apps over HTTP webhook go through the App Router and Switch to a per-app mapper, Kopia/Bazarr post to their own dedicated endpoints and bypass the router entirely with their own mapper, and every host publishes over MQTT into a topic-agnostic Inventory Parse function — all three land on the same subflow before reaching Slack](../assets/notifications-fan-in-fan-out.svg) ## Why One Node-RED Instance Instead Of Per-App Scripts The alternative — each app posting directly to a per-app Slack webhook, or a small script per integration — was rejected because it would mean N different message formats, N different places to add a new integration's logic, and no single point to add fleet-wide behavior (like the OS-level checks below) without touching every app individually. One Node-RED instance with a shared router and a shared formatting subflow means a new app's alerts get the same visual treatment as every existing one automatically, and a fleet-wide change (say, adding a severity-based routing rule) happens in one place. > [!WARNING] -> **Real debt in the flow**: of the apps wired into the router, two no longer resolve to anything — NetBox was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) but its detection code was never swept out, and Kestra's alerts actually travel a completely different path than its own webhook route suggests: Kestra publishes its own alerts over MQTT, not HTTP, which makes its dedicated webhook route effectively dead code. A couple of newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own onboarding pattern for a new app's alerts. +> **Real debt in the flow, one instance already cleaned up**: NetBox's dead detection code — left behind after the app was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) — has since been swept out of the flow entirely. What's still live: Kestra's alerts actually travel a completely different path than its own webhook route suggests — Kestra publishes its own alerts over MQTT, not HTTP, which makes its dedicated webhook route effectively dead code. A couple of newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own onboarding pattern for a new app's alerts. This is the real routing table, not a claimed count — counted directly from the flow export, not restated from memory: @@ -31,11 +31,11 @@ print('Switch rules:', len(switch['rules'])) dedicated = [n for n in flows if n.get('type')=='http in' and n.get('url','').endswith('-webhook') and n.get('url') != '/app-webhook'] print('Dedicated non-router endpoints:', len(dedicated)) " -Switch rules: 16 +Switch rules: 15 Dedicated non-router endpoints: 2 ``` -16 rules = the 15 routed apps plus one `unknown` fallback that just drops the payload. And this is the real function behind the "completely different path" line above — the MQTT branch doesn't parse `msg.topic` looking for a known set of fleet hosts, it just takes whatever the second path segment is: +15 rules = the 14 routed apps plus one `unknown` fallback that just drops the payload. And this is the real function behind the "completely different path" line above — the MQTT branch doesn't parse `msg.topic` looking for a known set of fleet hosts, it just takes whatever the second path segment is: ``` $ grep -A4 '"Inventory Parse Function' flows.json @@ -48,9 +48,9 @@ return msg; Topic-agnostic by construction — so a publisher that isn't a fleet host at all gets handled identically to one that is, which is exactly the mechanism that makes an app's *own* MQTT-published alerts real and its dedicated webhook route genuinely dead, not just theoretically bypassable. -This is the real flow, not a redrawn version of it — the App Router feeding a 16-way Switch, NetBox's and Kestra's mapper functions still sitting there wired in and unreachable, Kopia and Bazarr's dedicated endpoints bypassing all of it entirely on their way to the same shared subflow: +This is the real flow, not a redrawn version of it — the App Router feeding a 15-way Switch, Kestra's mapper function still sitting there wired in and unreachable, Kopia and Bazarr's dedicated endpoints bypassing all of it entirely on their way to the same shared subflow: -![Real Node-RED canvas: HTTP-in webhook through App Router into a 16-rule Switch function feeding per-app mapper functions (including netbox mapper and kestra mapper, both still wired but dead), MQTT-in through the Inventory Parse Function, and Kopia/Bazarr's own dedicated HTTP-in nodes bypassing the router entirely — all converging on the shared MQTT → Slack subflow](../assets/notifications-node-red-flow.png) +![Real Node-RED canvas: HTTP-in webhook through App Router into a 15-rule Switch function feeding per-app mapper functions (including kestra mapper, still wired but dead since its real alerts travel over MQTT instead), MQTT-in through the Inventory Parse Function, and Kopia/Bazarr's own dedicated HTTP-in nodes bypassing the router entirely — all converging on the shared MQTT → Slack subflow](../assets/notifications-node-red-flow.png) ## Message Broker @@ -104,10 +104,10 @@ The Node-RED App Router figures out which app sent a webhook by inspecting the p > > ``` > $ grep -oE '\$STATE_DIR/[a-zA-Z_]+' mqtt-monitor.sh.j2 | sed -E 's/(_alert|_last_count|_since|_report)$//' | sort -u | wc -l -> 22 +> 23 > ``` > -> 22 from that count, plus disk-usage thresholds — which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name — makes 23. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts. +> Of that 23, one (`failed_units_current`) is a companion state file for the same "failed systemd units" check as `failed_units`, not a distinct 23rd check — so 22 real checks, plus disk-usage thresholds, which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name, still makes 23 total. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts. ## One Formatter, Every Source