docs: deepen observability, notifications, orchestration, registry, and dashy batch

Same treatment as the security batch - real tech names, 'why this vs
the alternative' reasoning, heavy cross-linking, ranked/specific
tradeoffs instead of generic hedges.
This commit is contained in:
thesabear committed 2026-08-22 08:06:47 +03:00
1 parent 12e69acde7
commit 1382a63dae
5 files changed
+88 -71

No files matched your search

+26 -22
View File
@@ -1,7 +1,7 @@
# 🖥️ Dashy SOC Board
> [!NOTE]
> The homelab's "one pane of glass" — a single dashboard page backed by one large custom function, aggregating roughly a dozen distinct data sources that used to be scattered across separate pages into one glanceable board.
> The homelab's "one pane of glass" — a single Dashy page backed by one large custom Node-RED function, aggregating fourteen distinct data sources that used to be scattered across separate pages into one glanceable board. This is where every other doc in this repo actually converges visually — if a system produces a signal worth watching, it shows up here.
## Fourteen data sources, one board
@@ -9,30 +9,34 @@
flowchart TD
subgraph Sources["Data sources"]
direction LR
Wazuh["SOC indexer<br/>+ compliance API"]
Loki["Log aggregator<br/>24h error counts"]
Prom["Metrics<br/>CPU/RAM/disk"]
Slack["Alert channels<br/>per app + per host"]
Orch["Orchestration APIs<br/>execution health"]
Wazuh["Wazuh indexer<br/>+ compliance API"]
Loki["Loki<br/>24h error counts"]
Prom["Prometheus<br/>CPU/RAM/disk"]
Slack["Slack channels<br/>per app + per host"]
Orch["Kestra / n8n APIs<br/>execution health"]
Audit["Fleet config-drift<br/>audit feed"]
Hyper["Hypervisor API<br/>VM/LXC tree"]
Vault["Secrets platform<br/>seal state, lease counts"]
SSO["SSO provider<br/>login/failure stats"]
DNS["DNS server<br/>query stats, blocklist"]
Hyper["Proxmox API<br/>VM/LXC tree"]
Vault["Vault<br/>seal state, lease counts"]
SSO["Authentik<br/>login/failure stats"]
DNS["Technitium<br/>query stats, blocklist"]
Backup["Backup server<br/>job age/status"]
FW["Edge firewall<br/>state table, top rules"]
FW["pfSense<br/>state table, top rules"]
end
Sources --> Function["One aggregation function"]
Sources --> Function["One Node-RED function<br/>~104KB source"]
Function --> Board["SOC Board<br/>client-rendered"]
```
## Why one function instead of native Dashy widgets
Dashy's own built-in widget types cover simple cases well, but none of them can join data across sources the way this board needs to — a per-host tile that combines Wazuh findings, Prometheus resource usage, and Slack alert history into one expandable panel isn't expressible as a single native widget config. Building it as one Node-RED function that does the aggregation server-side, then hands the dashboard pre-joined data, was the tradeoff made: a large, custom function to maintain, in exchange for a board that actually reflects cross-system state instead of fourteen disconnected tiles a human has to mentally correlate themselves.
## Layout
A top row shows a 6-tier infrastructure dependency ladder alongside fleet-wide 24-hour KPIs. Below that, an "Active Issues" strip surfaces the worst findings fleet-wide, followed by a grid of per-host tiles — worst-severity badge, error sparkline, sorted consistently. Clicking a tile expands a full detail panel: compliance and resource gauges, that host's folded-in specialty widget where relevant, and findings grouped by category.
A top row shows a 6-tier infrastructure dependency ladder (mirroring the trust-boundary structure in [`docs/architecture-overview.md`](architecture-overview.md)) alongside fleet-wide 24-hour KPIs. Below that, an "Active Issues" strip surfaces the worst findings fleet-wide, followed by a grid of per-host tiles — worst-severity badge, error sparkline, sorted consistently. Clicking a tile expands a full detail panel: compliance and resource gauges, that host's folded-in specialty widget where relevant (a Vault seal-state panel on the secrets host, a Wazuh compliance ring on any host, and so on), and findings grouped by category.
> [!TIP]
> **Deliberately scoped**: security/error/warning signal only. Per-app uptime is handled separately by the dashboard's own native status checks on individual tiles — the only up/down signal inside this board is *infrastructure-tier* reachability, not per-app availability. Keeping the scope narrow is what keeps the board actually scannable.
> **Deliberately scoped**: security/error/warning signal only. Per-app uptime is handled separately by Dashy's own native status checks on individual tiles — the only up/down signal inside this board is *infrastructure-tier* reachability from Prometheus, not per-app availability. Keeping the scope narrow — resisting the temptation to also make this the uptime dashboard — is what keeps the board actually scannable instead of turning into a second copy of Grafana.
## Acknowledge mechanism: client-side, no backend
@@ -45,21 +49,21 @@ flowchart LR
```
> [!IMPORTANT]
> Acknowledging a finding is a **display filter only** — it never writes back to any upstream source, is scoped per-browser rather than fleet-wide, and a full data set for every host is always kept in memory so the recompute has everything it needs regardless of which panel happens to be open. A reload in a different browser shows the finding again — a deliberate tradeoff favoring simplicity (no backend state to manage) over persistence.
> Acknowledging a finding is a **display filter only** — it never writes back to Wazuh, Slack, or any upstream source described elsewhere in this repo, is scoped per-browser rather than fleet-wide, and a full data set for every host is always kept in memory so the recompute has everything it needs regardless of which panel happens to be open. A reload in a different browser shows the finding again. This was a deliberate tradeoff favoring simplicity — no backend state to build, secure, or back up — over persistence, made because the board's job is triage visibility, not an audit trail of who acknowledged what (that's what Wazuh's own alert log in [`docs/security-monitoring.md`](security-monitoring.md) is for).
## Real integration gotchas hit building this
> [!WARNING]
> A few non-obvious lessons from actually building this, in case they save someone else the debugging time:
> - Embedding via `fetch` + `innerHTML` means injected `<script>` tags never execute — each one has to be explicitly re-created on every refresh to keep interactive elements working.
> - A background poll that could clobber an open detail panel needs to check "is a panel open" **both** before *and* after the fetch, since the aggregation endpoint itself can take tens of seconds to respond.
> - One backing API hard-caps results per page regardless of a higher requested limit — real pagination has to be implemented, not assumed away.
> - Templating engines scoped to the wrong object silently produce an empty string instead of erroring — an easy, quiet failure mode.
> - Replacing a bind-mounted config file's content doesn't get picked up by a plain container restart — the container keeps watching the old inode; a full recreate is required.
> - A background poll that could clobber an open detail panel needs to check "is a panel open" **both** before *and* after the fetch, since the aggregation endpoint itself can take tens of seconds to respond given fourteen upstream calls.
> - Gitea's own package-list API hard-caps results per page regardless of a higher requested limit — real pagination has to be implemented, not assumed away, or the registry data from [`docs/registry-and-build.md`](registry-and-build.md) silently truncates.
> - Node-RED's Mustache templating scoped to the wrong object silently produces an empty string instead of erroring — an easy, quiet failure mode that looks like an upstream API problem until you check the template scoping.
> - Replacing a bind-mounted config file's content doesn't get picked up by a plain container restart — the container keeps watching the old inode; a full recreate is required, the same gotcha that shows up in [`docs/registry-and-build.md`](registry-and-build.md)'s redeploy path.
## Widget design: shape follows data, not one template everywhere
Every widget's visual style is chosen to match the *shape* of the data it shows, not a single card/badge component reused everywhere:
Every widget's visual style is chosen to match the *shape* of the data it shows, not a single card/badge component reused everywhere — the same design instinct behind this repo's own docs, honestly:
| Data shape | Visual treatment |
|---|---|
@@ -69,7 +73,7 @@ Every widget's visual style is chosen to match the *shape* of the data it shows,
| Named media/content items | A poster/cover-art grid, never a text list |
> [!TIP]
> The media-poster widgets route through one shared image-proxy endpoint rather than each hitting its own upstream API directly — since these are private, authenticated homelab APIs, not public image CDNs. One exception: a public poster CDN gets hit directly, since no auth or proxying is needed there.
> The media-poster widgets route through one shared image-proxy endpoint rather than each hitting its own upstream API directly — since these are private, authenticated homelab APIs, not public image CDNs, and a browser can't hold the credentials each one needs. One exception: a public poster CDN gets hit directly, since no auth or proxying is needed there.
---
Related stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/node-red/`](../stacks/platform/node-red/)
Related: [`docs/architecture-overview.md`](architecture-overview.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/cron-scheduler.md`](cron-scheduler.md) · Stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/node-red/`](../stacks/platform/node-red/)