thesabear
c10ff81d47
Completes the orchestration/registry/content batch - Kestra's check-pause-approve-apply pattern, the self-hosted build pipeline, the SOC dashboard's 14-source aggregation, the cron heatmap, and the in-progress content pipeline, all in the diagram+callout+table style.
3.7 KiB
🔁 Orchestration
Note
Thirteen scheduled flows handle fleet maintenance end to end — OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes.
The core pattern
flowchart LR
Schedule["Weekly schedule"] --> Check["Check mode run<br/>detect pending updates"]
Check --> Found{"Anything<br/>pending?"}
Found -->|no| NotifyClean["Notify: up to date"]
Found -->|yes| NotifyPending["Notify: pending"]
NotifyPending --> Pause["Pause<br/>indefinite, no timeout"]
Pause --> Decide{"Human<br/>resumes"}
Decide -->|approve| Apply["Apply mode run"]
Decide -->|decline| Skip["Skipped this cycle"]
Apply --> NotifyResult["Notify: applied / failed"]
Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it.
Identity split, per flow
Important
Most flows connect using one operator identity. One flow — the one that needs to run the automation tool itself on the control node, not just reach out over SSH — deliberately logs in as a separate, narrower service identity instead. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally. Worth checking which one a given flow uses before assuming — it's not always the same answer.
A real special case, thought through
Warning
One of the two hypervisor nodes also happens to host the automation VM that the orchestration engine itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification.
Trilium KB self-maintenance
Two flows keep the internal knowledge base's own structure current — the only two things allowed to write to it:
| Flow | Trigger | What it does |
|---|---|---|
| KB sync | Webhook, on every push | Mirrors exact file changes from any tracked repo into the KB, upserts/deletes only what changed — never re-walks a whole repo |
| Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review |
Tip
The push-triggered flow also runs a lightweight completeness check for the fleet's own VM-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel — a small nudge against exactly the kind of documentation drift this whole repo exists to avoid.
Outside the orchestration engine entirely
The automation host's own OS patches and the orchestration engine's own container updates run via a standalone host-level timer, not through the orchestration engine itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run. Being outside the normal flow means these updates don't get the same visibility as everything else, worth a periodic manual check rather than fully trusting the dashboard.
Related stacks: stacks/platform/kestra/