thesabear/public
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: deepen observability, notifications, orchestration, registry, and dashy batch

Browse Source 
Same treatment as the security batch - real tech names, 'why this vs
the alternative' reasoning, heavy cross-linking, ranked/specific
tradeoffs instead of generic hedges.
This commit is contained in:
thesabear committed 2026-08-22 08:06:47 +03:00
1 parent 12e69acde7
commit 1382a63dae
5 files changed
+88 -71
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/dashy-soc-board.md
+26 -22
View File
@@ -1,7 +1,7 @@
# 🖥️ Dashy SOC Board # 🖥️ Dashy SOC Board
> [!NOTE] > [!NOTE]
> The homelab's "one pane of glass" — a single dashboard page backed by one large custom function, aggregating roughly a dozen distinct data sources that used to be scattered across separate pages into one glanceable board. > The homelab's "one pane of glass" — a single Dashy page backed by one large custom Node-RED function, aggregating fourteen distinct data sources that used to be scattered across separate pages into one glanceable board. This is where every other doc in this repo actually converges visually — if a system produces a signal worth watching, it shows up here.
## Fourteen data sources, one board ## Fourteen data sources, one board
@@ -9,30 +9,34 @@
flowchart TD flowchart TD
subgraph Sources["Data sources"] subgraph Sources["Data sources"]
direction LR direction LR
Wazuh["SOC indexer<br/>+ compliance API"] Wazuh["Wazuh indexer<br/>+ compliance API"]
Loki["Log aggregator<br/>24h error counts"] Loki["Loki<br/>24h error counts"]
Prom["Metrics<br/>CPU/RAM/disk"] Prom["Prometheus<br/>CPU/RAM/disk"]
Slack["Alert channels<br/>per app + per host"] Slack["Slack channels<br/>per app + per host"]
Orch["Orchestration APIs<br/>execution health"] Orch["Kestra / n8n APIs<br/>execution health"]
Audit["Fleet config-drift<br/>audit feed"] Audit["Fleet config-drift<br/>audit feed"]
Hyper["Hypervisor API<br/>VM/LXC tree"] Hyper["Proxmox API<br/>VM/LXC tree"]
Vault["Secrets platform<br/>seal state, lease counts"] Vault["Vault<br/>seal state, lease counts"]
SSO["SSO provider<br/>login/failure stats"] SSO["Authentik<br/>login/failure stats"]
DNS["DNS server<br/>query stats, blocklist"] DNS["Technitium<br/>query stats, blocklist"]
Backup["Backup server<br/>job age/status"] Backup["Backup server<br/>job age/status"]
FW["Edge firewall<br/>state table, top rules"] FW["pfSense<br/>state table, top rules"]
end end
Sources --> Function["One aggregation function"] Sources --> Function["One Node-RED function<br/>~104KB source"]
Function --> Board["SOC Board<br/>client-rendered"] Function --> Board["SOC Board<br/>client-rendered"]
``` ```
## Why one function instead of native Dashy widgets
Dashy's own built-in widget types cover simple cases well, but none of them can join data across sources the way this board needs to — a per-host tile that combines Wazuh findings, Prometheus resource usage, and Slack alert history into one expandable panel isn't expressible as a single native widget config. Building it as one Node-RED function that does the aggregation server-side, then hands the dashboard pre-joined data, was the tradeoff made: a large, custom function to maintain, in exchange for a board that actually reflects cross-system state instead of fourteen disconnected tiles a human has to mentally correlate themselves.
## Layout ## Layout
A top row shows a 6-tier infrastructure dependency ladder alongside fleet-wide 24-hour KPIs. Below that, an "Active Issues" strip surfaces the worst findings fleet-wide, followed by a grid of per-host tiles — worst-severity badge, error sparkline, sorted consistently. Clicking a tile expands a full detail panel: compliance and resource gauges, that host's folded-in specialty widget where relevant, and findings grouped by category. A top row shows a 6-tier infrastructure dependency ladder (mirroring the trust-boundary structure in [`docs/architecture-overview.md`](architecture-overview.md)) alongside fleet-wide 24-hour KPIs. Below that, an "Active Issues" strip surfaces the worst findings fleet-wide, followed by a grid of per-host tiles — worst-severity badge, error sparkline, sorted consistently. Clicking a tile expands a full detail panel: compliance and resource gauges, that host's folded-in specialty widget where relevant (a Vault seal-state panel on the secrets host, a Wazuh compliance ring on any host, and so on), and findings grouped by category.
> [!TIP] > [!TIP]
> **Deliberately scoped**: security/error/warning signal only. Per-app uptime is handled separately by the dashboard's own native status checks on individual tiles — the only up/down signal inside this board is *infrastructure-tier* reachability, not per-app availability. Keeping the scope narrow is what keeps the board actually scannable. > **Deliberately scoped**: security/error/warning signal only. Per-app uptime is handled separately by Dashy's own native status checks on individual tiles — the only up/down signal inside this board is *infrastructure-tier* reachability from Prometheus, not per-app availability. Keeping the scope narrow — resisting the temptation to also make this the uptime dashboard — is what keeps the board actually scannable instead of turning into a second copy of Grafana.
## Acknowledge mechanism: client-side, no backend ## Acknowledge mechanism: client-side, no backend
@@ -45,21 +49,21 @@ flowchart LR
``` ```
> [!IMPORTANT] > [!IMPORTANT]
> Acknowledging a finding is a **display filter only** — it never writes back to any upstream source, is scoped per-browser rather than fleet-wide, and a full data set for every host is always kept in memory so the recompute has everything it needs regardless of which panel happens to be open. A reload in a different browser shows the finding again — a deliberate tradeoff favoring simplicity (no backend state to manage) over persistence. > Acknowledging a finding is a **display filter only** — it never writes back to Wazuh, Slack, or any upstream source described elsewhere in this repo, is scoped per-browser rather than fleet-wide, and a full data set for every host is always kept in memory so the recompute has everything it needs regardless of which panel happens to be open. A reload in a different browser shows the finding again. This was a deliberate tradeoff favoring simplicity — no backend state to build, secure, or back up — over persistence, made because the board's job is triage visibility, not an audit trail of who acknowledged what (that's what Wazuh's own alert log in [`docs/security-monitoring.md`](security-monitoring.md) is for).
## Real integration gotchas hit building this ## Real integration gotchas hit building this
> [!WARNING] > [!WARNING]
> A few non-obvious lessons from actually building this, in case they save someone else the debugging time: > A few non-obvious lessons from actually building this, in case they save someone else the debugging time:
> - Embedding via `fetch` + `innerHTML` means injected `<script>` tags never execute — each one has to be explicitly re-created on every refresh to keep interactive elements working. > - Embedding via `fetch` + `innerHTML` means injected `<script>` tags never execute — each one has to be explicitly re-created on every refresh to keep interactive elements working.
> - A background poll that could clobber an open detail panel needs to check "is a panel open" **both** before *and* after the fetch, since the aggregation endpoint itself can take tens of seconds to respond. > - A background poll that could clobber an open detail panel needs to check "is a panel open" **both** before *and* after the fetch, since the aggregation endpoint itself can take tens of seconds to respond given fourteen upstream calls.
> - One backing API hard-caps results per page regardless of a higher requested limit — real pagination has to be implemented, not assumed away. > - Gitea's own package-list API hard-caps results per page regardless of a higher requested limit — real pagination has to be implemented, not assumed away, or the registry data from [`docs/registry-and-build.md`](registry-and-build.md) silently truncates.
> - Templating engines scoped to the wrong object silently produce an empty string instead of erroring — an easy, quiet failure mode. > - Node-RED's Mustache templating scoped to the wrong object silently produces an empty string instead of erroring — an easy, quiet failure mode that looks like an upstream API problem until you check the template scoping.
> - Replacing a bind-mounted config file's content doesn't get picked up by a plain container restart — the container keeps watching the old inode; a full recreate is required. > - Replacing a bind-mounted config file's content doesn't get picked up by a plain container restart — the container keeps watching the old inode; a full recreate is required, the same gotcha that shows up in [`docs/registry-and-build.md`](registry-and-build.md)'s redeploy path.
## Widget design: shape follows data, not one template everywhere ## Widget design: shape follows data, not one template everywhere
Every widget's visual style is chosen to match the *shape* of the data it shows, not a single card/badge component reused everywhere: Every widget's visual style is chosen to match the *shape* of the data it shows, not a single card/badge component reused everywhere — the same design instinct behind this repo's own docs, honestly:
| Data shape | Visual treatment | | Data shape | Visual treatment |
|---|---| |---|---|
@@ -69,7 +73,7 @@ Every widget's visual style is chosen to match the *shape* of the data it shows,
| Named media/content items | A poster/cover-art grid, never a text list | | Named media/content items | A poster/cover-art grid, never a text list |
> [!TIP] > [!TIP]
> The media-poster widgets route through one shared image-proxy endpoint rather than each hitting its own upstream API directly — since these are private, authenticated homelab APIs, not public image CDNs. One exception: a public poster CDN gets hit directly, since no auth or proxying is needed there. > The media-poster widgets route through one shared image-proxy endpoint rather than each hitting its own upstream API directly — since these are private, authenticated homelab APIs, not public image CDNs, and a browser can't hold the credentials each one needs. One exception: a public poster CDN gets hit directly, since no auth or proxying is needed there.
--- ---
Related stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/node-red/`](../stacks/platform/node-red/) Related: [`docs/architecture-overview.md`](architecture-overview.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/cron-scheduler.md`](cron-scheduler.md) · Stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/node-red/`](../stacks/platform/node-red/)
docs/notifications.md
+15 -11
View File
@@ -1,7 +1,7 @@
# 🔔 Notifications # 🔔 Notifications
> [!NOTE] > [!NOTE]
> Two independent branches — app webhooks and OS-level host alerts — converge on a single automation flow and one shared formatter before reaching Slack, so every alert reads the same regardless of where it came from. > Mosquitto (MQTT) and a single Node-RED instance are the backbone here — two independent fan-in branches, app webhooks and OS-level host alerts, converging on one flow and one shared formatter before reaching Slack. The point isn't the individual pieces; it's that every alert reads the same regardless of where it came from.
## Fan-in, fan-out ## Fan-in, fan-out
@@ -13,7 +13,7 @@ flowchart LR
MQTT["Every host<br/>MQTT publish"] MQTT["Every host<br/>MQTT publish"]
end end
Apps --> Router["App Router<br/>payload-shape detection"] Apps --> Router["Node-RED App Router<br/>payload-shape detection"]
Dedicated --> Router Dedicated --> Router
Router --> Mapper["Per-app mapper"] Router --> Mapper["Per-app mapper"]
MQTT --> Parser["Inventory parser"] MQTT --> Parser["Inventory parser"]
@@ -23,35 +23,39 @@ flowchart LR
Subflow --> Slack["Slack"] Subflow --> Slack["Slack"]
``` ```
## Why one Node-RED instance instead of per-app scripts
The alternative — each app posting directly to a per-app Slack webhook, or a small script per integration — was rejected because it would mean N different message formats, N different places to add a new integration's logic, and no single point to add fleet-wide behavior (like the OS-level checks below) without touching every app individually. One Node-RED instance with a shared router and a shared formatting subflow means a new app's alerts get the same visual treatment as every existing one automatically, and a fleet-wide change (say, adding a severity-based routing rule) happens in one place.
> [!WARNING] > [!WARNING]
> **Debt, documented rather than hidden**: of the apps wired into the router, a couple no longer resolve to anything — one was fully decommissioned but its detection code was never swept out, and another's alerts actually travel a completely different path than its own webhook route suggests, making that route effectively dead code. Two newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own documented pattern for onboarding a new app's alerts. > **Debt, documented rather than hidden**: of the apps wired into the router, a couple no longer resolve to anything — one was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) but its detection code was never swept out of the flow, and another's alerts actually travel a completely different path than its own webhook route suggests, making that route effectively dead code. Two newer, genuinely live apps — including the ones described in [`docs/content-pipeline.md`](content-pipeline.md) — aren't wired into either branch yet. Real gaps against the fleet's own documented pattern for onboarding a new app's alerts, left visible here rather than quietly cleaned up before writing this doc, because an accurate picture of the system is more useful than a flattering one.
## Message broker ## Message broker
| | | | | |
|---|---| |---|---|
| **Broker** | Runs as its own container on the automation host | | **Broker** | Mosquitto, self-hosted, runs as its own container on the automation host |
| **Protocol** | MQTT v3.1.1, auth required, anonymous disabled | | **Protocol** | MQTT v3.1.1, auth required, anonymous disabled |
| **Topic structure** | One topic per host, wildcard-subscribed centrally | | **Topic structure** | One topic per host, wildcard-subscribed centrally |
| **Users** | Two — a publish identity used fleet-wide, and the automation flow's own read-only subscriber | | **Users** | Two — a publish identity used fleet-wide, and Node-RED's own read-only subscriber |
Both broker users are credential-managed through the same secret-rendering pipeline as everything else in [`docs/secrets-and-pki.md`](secrets-and-pki.md) — not a bare manually-edited password file. Both broker users are credential-managed through the same Vault-rendered pipeline as everything else in [`docs/secrets-and-pki.md`](secrets-and-pki.md) — not a bare manually-edited password file.
> [!IMPORTANT] > [!IMPORTANT]
> **A real, tracked fragility**: the publish identity's password has to stay in sync across three separate places — the rendered broker credential file, the source value used to render every host's own publish script, and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from the secret store at all). Rotating it means updating all three by hand — a documented gap, not a silent one. > **A real, tracked fragility, worth naming precisely**: the publish identity's password has to stay in sync across three separate places — the Vault-rendered broker credential file, the source value used to render every host's own publish script (via the same tag-driven Ansible model in [`docs/iac-pipeline.md`](iac-pipeline.md)), and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from Vault at all, unlike everything else in [`docs/secret-distribution.md`](secret-distribution.md)). Rotating it means updating all three by hand — a documented, real exception to the "one source of truth" rule that governs every other secret in this fleet, called out explicitly rather than left for someone to discover mid-rotation.
## Detection logic ## Detection logic
The router figures out which app sent a webhook by inspecting the payload's shape — an explicit field for some apps, structural pattern-matching for others (one app, for instance, is identified purely by a nested field only that vendor's payload happens to include). Two apps skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON. The Node-RED App Router figures out which app sent a webhook by inspecting the payload's shape — an explicit field for some apps, structural pattern-matching for others (one app, for instance, is identified purely by a nested field only that vendor's payload happens to include, since it never sends an explicit app identifier). Two apps skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON and the shared router's shape-detection logic simply doesn't apply to them.
## Every host, not just apps ## Every host, not just apps
> [!TIP] > [!TIP]
> A dedicated automation role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert. It runs over 20 distinct checks per host — disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (audit log anomalies, authentication failures) — deliberately complementary to the SOC stack in [`docs/security-monitoring.md`](security-monitoring.md), not duplicating its file-integrity or account monitoring. > A dedicated Ansible role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert — the same fleet-wide, tag-driven rollout pattern used for the observability agent in [`docs/observability.md`](observability.md) and the compliance hardening in [`docs/security-monitoring.md`](security-monitoring.md). It runs over 20 distinct checks per host — disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (Vault audit log anomalies, Vault Agent authentication failures) — deliberately complementary to Wazuh in [`docs/security-monitoring.md`](security-monitoring.md), not duplicating its file-integrity or account monitoring. Splitting OS-health checks from the SOC's own detection engine keeps each system's role narrow enough to actually reason about, instead of one monolithic monitoring tool trying to do both jobs at once.
## One formatter, every source ## One formatter, every source
Every branch — app webhooks, host-level alerts, and the two dedicated non-JSON endpoints — converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it. Every branch — app webhooks, host-level alerts, and the two dedicated non-JSON endpoints — converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it — a reader in Slack never has to learn a different alert format depending on which system fired it.
--- ---
Related stacks: [`stacks/platform/mosquitto/`](../stacks/platform/mosquitto/), [`stacks/platform/node-red/`](../stacks/platform/node-red/) Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/observability.md`](observability.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/mosquitto/`](../stacks/platform/mosquitto/), [`stacks/platform/node-red/`](../stacks/platform/node-red/)
docs/observability.md
+17 -16
View File
@@ -1,15 +1,19 @@
# 📈 Observability # 📈 Observability
> [!NOTE] > [!NOTE]
> Every fleet host runs the same collection agent natively (not as a container), pushing logs and metrics outward rather than waiting to be scraped — a deliberate choice that keeps monitoring off the firewall's critical path. > Grafana Alloy runs natively on every fleet host, shipping logs to Loki and metrics to Prometheus via `remote_write` — pushed outward rather than scraped, a choice made specifically because of the firewall model in [`docs/firewall-and-network.md`](firewall-and-network.md), not despite it.
| | | | | |
|---|---| |---|---|
| **Agent** | Native systemd service on every host, not Docker | | **Agent** | Grafana Alloy, native systemd service on every host, not Docker |
| **Logs** | journald → Loki | | **Logs** | journald → Loki |
| **Metrics** | Push-based `remote_write`, not pull-based scraping | | **Metrics** | Push-based `remote_write`, not pull-based scraping |
| **Coverage** | All fleet hosts, confirmed live | | **Coverage** | All fleet hosts, confirmed live |
## Why push instead of pull — the actual reasoning
A pull-based model (Prometheus scraping every host directly) is the more common default, and it was rejected for a specific reason tied directly to [`docs/firewall-and-network.md`](firewall-and-network.md): scraping means the monitoring host needs inbound network access to every single host it monitors, which means punching a hole through the zero-blanket-allow firewall model for every host, every metrics port. A push model inverts that — only the monitoring host needs to accept inbound writes, and every other host only ever *initiates* outbound connections, which fits the existing "explicit outbound rule, default-deny inbound" posture instead of fighting it. The tradeoff is real: push-based metrics are slightly harder to reason about for ad-hoc debugging (no simple `curl` against a target to see what it'd report), but it was the right call given the network model already in place.
## Pipeline ## Pipeline
```mermaid ```mermaid
@@ -21,7 +25,7 @@ flowchart LR
App["Native app metrics<br/>5 apps expose endpoints"] App["Native app metrics<br/>5 apps expose endpoints"]
end end
Agent["Collection Agent<br/>native systemd service"] Agent["Grafana Alloy<br/>native systemd service"]
J --> Agent J --> Agent
D -->|docker_hosts only| Agent D -->|docker_hosts only| Agent
N --> Agent N --> Agent
@@ -34,11 +38,11 @@ flowchart LR
## Agent install: deliberate, not default ## Agent install: deliberate, not default
> [!TIP] > [!TIP]
> The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway). > The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. That's a real maintenance cost (manual version bumps instead of `apt upgrade` handling it), accepted because it keeps the exact version pinned and reproducible across the fleet through the same tag-driven Ansible model described in [`docs/iac-pipeline.md`](iac-pipeline.md), rather than whatever version a package repo happens to serve on a given day. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).
## Logs: journald only, no double-handling ## Logs: journald only, no double-handling
The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume. The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet.
```mermaid ```mermaid
flowchart LR flowchart LR
@@ -47,22 +51,19 @@ flowchart LR
C["DNS query log<br/>daily rotation"] --> L3["Loki<br/>job=dns · real event time"] C["DNS query log<br/>daily rotation"] --> L3["Loki<br/>job=dns · real event time"]
``` ```
Every host's logs land in the same log store under its own host label, filterable in the dashboard by host or job. Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated. The DNS query log pipeline is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time. Every host's logs land in the same Loki instance under its own host label, filterable in Grafana by host or job — the same "one source of truth, not N inconsistent ones" instinct behind the DNS model in [`docs/dns-naming.md`](dns-naming.md). Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated during the update flows in [`docs/registry-and-build.md`](registry-and-build.md) — tailing a file path would silently stop working the moment that container got rebuilt. The DNS query log pipeline (feeding into [`docs/dns-naming.md`](dns-naming.md)'s enforcement story) is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time — the same UTC-vs-wall-clock discipline that matters when querying Wazuh's alert data in [`docs/security-monitoring.md`](security-monitoring.md).
## Metrics: pushed, not pulled ## Metrics: pushed, not pulled
> [!IMPORTANT]
> **Why push instead of pull**: Prometheus doesn't need network access to every host — only the monitoring host needs to accept inbound writes. This matters directly for hosts sitting behind the tighter firewall rules described in [`firewall-and-network.md`](firewall-and-network.md) — a pull model would mean punching a scrape-target hole through that segmentation for every single host.
| App | What's exposed | | App | What's exposed |
|---|---| |---|---|
| Secrets platform | Dedicated read-only metrics token, separate from its normal agent token | | Vault | Dedicated read-only metrics token, separate from its normal agent token — see [`docs/secrets-and-pki.md`](secrets-and-pki.md) |
| Orchestration engine | Built-in metrics endpoint | | Kestra | Built-in metrics endpoint |
| Git server | Built-in metrics endpoint | | Gitea | Built-in metrics endpoint |
| Workflow automation | Built-in metrics endpoint | | n8n | Built-in metrics endpoint |
| Identity provider | Built-in metrics endpoint, localhost-only | | Authentik | Built-in metrics endpoint, localhost-only |
Five apps expose their own metrics endpoints directly, scraped by that host's own agent and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility. Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open.
--- ---
Related stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/) Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)
docs/orchestration.md
+13 -9
View File
@@ -1,7 +1,11 @@
# 🔁 Orchestration # 🔁 Orchestration
> [!NOTE] > [!NOTE]
> Thirteen scheduled flows handle fleet maintenance end to end — OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes. > Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes. This is the layer that actually *acts* on the fleet, which is why it's the most tightly gated system in this repo.
## Why a human-gated pattern instead of full automation
Fully automatic updates were the obvious simpler design, and they were rejected specifically because the cost of a bad automatic change (a broken update applied fleet-wide at 2am with nobody watching) is higher than the cost of a delayed one. The check → pause → approve pattern below trades speed for a human checkpoint at exactly the moment something is about to actually change — cheap insurance against the failure mode that matters most in a system with no on-call rotation behind it.
## The core pattern ## The core pattern
@@ -18,21 +22,21 @@ flowchart LR
Apply --> NotifyResult["Notify: applied / failed"] Apply --> NotifyResult["Notify: applied / failed"]
``` ```
Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it. Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert.
## Identity split, per flow ## Identity split, per flow
> [!IMPORTANT] > [!IMPORTANT]
> Most flows connect using one operator identity. One flow — the one that needs to run the automation tool itself on the control node, not just reach out over SSH — deliberately logs in as a separate, narrower service identity instead. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally. Worth checking which one a given flow uses before assuming — it's not always the same answer. > Most flows connect using one operator identity over SSH. One flow — the one that needs to run Ansible itself on the control node, not just reach out over SSH to a target — deliberately logs in as a separate, narrower service identity instead, since it needs local execution rights on the control node that the SSH-only identity doesn't have. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally, following the same "scope credentials to exactly what's needed" instinct as the least-privilege Git access model in [`docs/sso-and-identity.md`](sso-and-identity.md). Worth checking which one a given flow uses before assuming — it's not always the same answer, and assuming wrong is a fast way to get a confusing permission error mid-debugging.
## A real special case, thought through ## A real special case, thought through in advance
> [!WARNING] > [!WARNING]
> One of the two hypervisor nodes also happens to host the automation VM that the orchestration engine itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. > One of the two Proxmox hypervisor nodes also happens to host the automation VM that Kestra itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death, since the flow doing the waiting dies with the VM it's waiting on. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM, including the notification pipeline itself from [`docs/notifications.md`](notifications.md), which also runs there. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. This is the kind of edge case that only shows up once you actually think through "what happens when the thing doing the orchestrating is itself one of the things being orchestrated" — a real, non-obvious problem in any self-hosted automation system.
## Trilium KB self-maintenance ## Trilium KB self-maintenance
Two flows keep the internal knowledge base's own structure current — the *only* two things allowed to write to it: Two flows keep this fleet's own internal knowledge base structure current — the *only* two things allowed to write to it, everything else (including this repo's own generation) only ever reads from it:
| Flow | Trigger | What it does | | Flow | Trigger | What it does |
|---|---|---| |---|---|---|
@@ -40,11 +44,11 @@ Two flows keep the internal knowledge base's own structure current — the *only
| Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review | | Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review |
> [!TIP] > [!TIP]
> The push-triggered flow also runs a lightweight completeness check for the fleet's own VM-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel — a small nudge against exactly the kind of documentation drift this whole repo exists to avoid. > The push-triggered flow also runs a lightweight completeness check for the fleet's own host-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel via [`docs/notifications.md`](notifications.md) — a small, automatic nudge against exactly the kind of documentation drift that this whole portfolio repo exists to avoid in the first place.
## Outside the orchestration engine entirely ## Outside the orchestration engine entirely
The automation host's own OS patches and the orchestration engine's own container updates run via a standalone host-level timer, not through the orchestration engine itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run. Being outside the normal flow means these updates don't get the same visibility as everything else, worth a periodic manual check rather than fully trusting the dashboard. The automation host's own OS patches and Kestra's own container updates run via a standalone host-level systemd timer, not through Kestra itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run without risking killing the process doing the patching. Being outside the normal flow means these updates don't get the same visibility as everything else in this doc, which is a deliberate, accepted tradeoff — worth a periodic manual check rather than fully trusting the dashboard to surface it.
--- ---
Related stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/) Related: [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/disaster-recovery.md`](disaster-recovery.md) · Stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)
docs/registry-and-build.md
+17 -13
View File
@@ -1,19 +1,23 @@
# 📦 Self-Hosted Registry & Build Pipeline # 📦 Self-Hosted Registry & Build Pipeline
> [!NOTE] > [!NOTE]
> Docker images are built and hosted in-fleet rather than pulled from Docker Hub/ghcr directly — for every stack opted into this pattern, at runtime. > Gitea's own built-in container registry hosts every image this fleet builds — no separate registry service to operate, no dependency on Docker Hub or ghcr at runtime for the stacks that have opted in. This is the mechanism that makes [`docs/iac-pipeline.md`](iac-pipeline.md)'s "own everything in the stack" philosophy actually true at the container-image layer, not just the OS layer.
| | | | | |
|---|---| |---|---|
| **Registry** | The Git server's own built-in container registry — no separate registry service to run | | **Registry** | Gitea's own built-in container registry |
| **Trigger** | Weekly schedule, not push-triggered | | **Trigger** | Weekly schedule via Kestra, not push-triggered |
| **Rollout** | All 5 Docker hosts converted; the largest has 36 of 41 stacks on this pattern | | **Rollout** | All 5 Docker hosts converted; the largest has 36 of 41 stacks on this pattern |
## Why self-host the registry instead of just using Docker Hub
Pulling directly from Docker Hub/ghcr at runtime was the status quo before this existed, and it was replaced for a concrete reason: every container pull became a runtime dependency on an external service staying up and staying reachable through the firewall model in [`docs/firewall-and-network.md`](firewall-and-network.md). Building images in-fleet and pushing to a registry that's already inside the trust boundary means a redeploy never depends on an external registry's availability, and the exact image contents are fully within this fleet's own control rather than whatever an upstream maintainer pushed as `:latest` today.
## The cheap-check-first build decision ## The cheap-check-first build decision
```mermaid ```mermaid
flowchart TD flowchart TD
Start["Weekly trigger"] --> Check["Cheap digest check<br/>metadata only, no pull"] Start["Weekly Kestra trigger<br/>see docs/orchestration.md"] --> Check["Cheap digest check<br/>metadata only, no pull"]
Check --> Match{"Upstream digest ==<br/>fleet registry digest?"} Check --> Match{"Upstream digest ==<br/>fleet registry digest?"}
Match -->|yes| Nothing["Nothing runs<br/>no pull, no build, no push"] Match -->|yes| Nothing["Nothing runs<br/>no pull, no build, no push"]
Match -->|no| Pull["Pull base images"] Match -->|no| Pull["Pull base images"]
@@ -24,27 +28,27 @@ flowchart TD
``` ```
> [!TIP] > [!TIP]
> Most stacks use a minimal wrapper Dockerfile around the upstream image rather than a full clone-and-build-from-source setup — simpler, builds near-instantly, can't break from upstream build tooling, while still achieving the real goal: an in-fleet registry with no direct Docker Hub/ghcr dependency at runtime. A handful of multi-build-image stacks skip the cheap check entirely, since a single "upstream digest" comparison doesn't make sense when there's more than one real upstream involved. > Most stacks use a minimal wrapper Dockerfile around the upstream image rather than a full clone-and-build-from-source setup — simpler, builds near-instantly, can't break from upstream build tooling changing underneath it, while still achieving the real goal: an in-fleet registry with no direct Docker Hub/ghcr dependency at runtime. A handful of multi-build-image stacks (the ones with more than one real upstream component, like the identity provider in [`docs/sso-and-identity.md`](sso-and-identity.md)) skip the cheap check entirely, since a single "upstream digest" comparison doesn't make sense when there's more than one upstream involved.
## Deliberate holdouts ## Deliberate holdouts
Not every stack is on this pattern, and that's a decision, not an oversight: Not every stack is on this pattern, and that's a decision, not an oversight — knowing when *not* to apply a pattern uniformly is as much a design skill as building the pattern in the first place:
| Stack type | Why it's held out | | Stack type | Why it's held out |
|---|---| |---|---|
| Complex multi-service upstream builds | Not worth replicating a build process someone else already maintains well | | Complex multi-service upstream builds (e.g. the identity provider, a photo-management app, a media transcoding tool) | Not worth replicating a build process someone else already maintains well — the maintenance cost of tracking upstream's own build changes would exceed the benefit |
| A docker-socket-proxy style stack | Simple enough that a custom build adds no value | | A docker-socket-proxy style stack | Simple enough that a custom build adds no real value over the upstream image directly |
| One database service still on `latest` | Deliberately not pinned to this pattern | | One database service still on `latest` | Deliberately not pinned to this pattern — low enough stakes that the extra build step isn't worth it |
| One stack with pre-existing custom Dockerfiles | Left as-is beyond adding image references — its own DB/cache still float on upstream tags directly | | A productivity suite with pre-existing custom Dockerfiles | Left as-is beyond adding image references — its own DB/cache still float on Docker Hub tags directly, a conscious "don't touch what already works" call |
## Trigger discipline ## Trigger discipline
> [!WARNING] > [!WARNING]
> This is schedule-triggered, not push-triggered. Editing a stack's own Dockerfile doesn't by itself trigger a rebuild — it waits for the next weekly sweep to find the upstream base image has actually changed, or for someone to force a manual run. A deliberate tradeoff: simpler mental model, avoids a rebuild storm on every unrelated commit, at the cost of same-day propagation for an urgent base-image CVE fix. > This is schedule-triggered, not push-triggered — a deliberate choice, not a missing feature. Editing a stack's own Dockerfile doesn't by itself trigger a rebuild; it waits for the next weekly sweep to find the upstream base image has actually changed, or for someone to force a manual run via [`docs/orchestration.md`](orchestration.md). The tradeoff: a simpler mental model and no rebuild storm on every unrelated commit, at the real cost of same-day propagation for an urgent base-image CVE fix — accepted because a weekly cadence matches this fleet's actual risk tolerance, not because push-triggering would have been harder to build.
## Housekeeping ## Housekeeping
Unused local images get pruned automatically at the end of every build pass — for build-and-push-only stacks, the registry is the source of truth, not whatever happens to be sitting on local disk. Unused local images get pruned automatically at the end of every build pass — for build-and-push-only stacks, the registry is the source of truth, not whatever happens to be sitting on local disk, so nothing is lost by aggressively reclaiming local space immediately.
--- ---
Related stacks: [`stacks/platform/gitea/`](../stacks/platform/gitea/), [`stacks/platform/kestra/`](../stacks/platform/kestra/) Related: [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/orchestration.md`](orchestration.md), [`docs/sso-and-identity.md`](sso-and-identity.md) · Stacks: [`stacks/platform/gitea/`](../stacks/platform/gitea/), [`stacks/platform/kestra/`](../stacks/platform/kestra/)