thesabear
c10ff81d47
Completes the orchestration/registry/content batch - Kestra's check-pause-approve-apply pattern, the self-hosted build pipeline, the SOC dashboard's 14-source aggregation, the cron heatmap, and the in-progress content pipeline, all in the diagram+callout+table style.
51 lines
3.7 KiB
Markdown
51 lines
3.7 KiB
Markdown
# 🔁 Orchestration
|
|
|
|
> [!NOTE]
|
|
> Thirteen scheduled flows handle fleet maintenance end to end — OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes.
|
|
|
|
## The core pattern
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Schedule["Weekly schedule"] --> Check["Check mode run<br/>detect pending updates"]
|
|
Check --> Found{"Anything<br/>pending?"}
|
|
Found -->|no| NotifyClean["Notify: up to date"]
|
|
Found -->|yes| NotifyPending["Notify: pending"]
|
|
NotifyPending --> Pause["Pause<br/>indefinite, no timeout"]
|
|
Pause --> Decide{"Human<br/>resumes"}
|
|
Decide -->|approve| Apply["Apply mode run"]
|
|
Decide -->|decline| Skip["Skipped this cycle"]
|
|
Apply --> NotifyResult["Notify: applied / failed"]
|
|
```
|
|
|
|
Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it.
|
|
|
|
## Identity split, per flow
|
|
|
|
> [!IMPORTANT]
|
|
> Most flows connect using one operator identity. One flow — the one that needs to run the automation tool itself on the control node, not just reach out over SSH — deliberately logs in as a separate, narrower service identity instead. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally. Worth checking which one a given flow uses before assuming — it's not always the same answer.
|
|
|
|
## A real special case, thought through
|
|
|
|
> [!WARNING]
|
|
> One of the two hypervisor nodes also happens to host the automation VM that the orchestration engine itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification.
|
|
|
|
## Trilium KB self-maintenance
|
|
|
|
Two flows keep the internal knowledge base's own structure current — the *only* two things allowed to write to it:
|
|
|
|
| Flow | Trigger | What it does |
|
|
|---|---|---|
|
|
| KB sync | Webhook, on every push | Mirrors exact file changes from any tracked repo into the KB, upserts/deletes only what changed — never re-walks a whole repo |
|
|
| Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review |
|
|
|
|
> [!TIP]
|
|
> The push-triggered flow also runs a lightweight completeness check for the fleet's own VM-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel — a small nudge against exactly the kind of documentation drift this whole repo exists to avoid.
|
|
|
|
## Outside the orchestration engine entirely
|
|
|
|
The automation host's own OS patches and the orchestration engine's own container updates run via a standalone host-level timer, not through the orchestration engine itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run. Being outside the normal flow means these updates don't get the same visibility as everything else, worth a periodic manual check rather than fully trusting the dashboard.
|
|
|
|
---
|
|
Related stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)
|