docs: add orchestration, registry, dashy, cron, and content pipeline write-ups
Completes the orchestration/registry/content batch - Kestra's check-pause-approve-apply pattern, the self-hosted build pipeline, the SOC dashboard's 14-source aggregation, the cron heatmap, and the in-progress content pipeline, all in the diagram+callout+table style.
This commit is contained in:
5 files changed
+248
No files matched your search
@@ -0,0 +1,50 @@
|
||||
# 🔁 Orchestration
|
||||
|
||||
> [!NOTE]
|
||||
> Thirteen scheduled flows handle fleet maintenance end to end — OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes.
|
||||
|
||||
## The core pattern
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Schedule["Weekly schedule"] --> Check["Check mode run<br/>detect pending updates"]
|
||||
Check --> Found{"Anything<br/>pending?"}
|
||||
Found -->|no| NotifyClean["Notify: up to date"]
|
||||
Found -->|yes| NotifyPending["Notify: pending"]
|
||||
NotifyPending --> Pause["Pause<br/>indefinite, no timeout"]
|
||||
Pause --> Decide{"Human<br/>resumes"}
|
||||
Decide -->|approve| Apply["Apply mode run"]
|
||||
Decide -->|decline| Skip["Skipped this cycle"]
|
||||
Apply --> NotifyResult["Notify: applied / failed"]
|
||||
```
|
||||
|
||||
Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it.
|
||||
|
||||
## Identity split, per flow
|
||||
|
||||
> [!IMPORTANT]
|
||||
> Most flows connect using one operator identity. One flow — the one that needs to run the automation tool itself on the control node, not just reach out over SSH — deliberately logs in as a separate, narrower service identity instead. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally. Worth checking which one a given flow uses before assuming — it's not always the same answer.
|
||||
|
||||
## A real special case, thought through
|
||||
|
||||
> [!WARNING]
|
||||
> One of the two hypervisor nodes also happens to host the automation VM that the orchestration engine itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification.
|
||||
|
||||
## Trilium KB self-maintenance
|
||||
|
||||
Two flows keep the internal knowledge base's own structure current — the *only* two things allowed to write to it:
|
||||
|
||||
| Flow | Trigger | What it does |
|
||||
|---|---|---|
|
||||
| KB sync | Webhook, on every push | Mirrors exact file changes from any tracked repo into the KB, upserts/deletes only what changed — never re-walks a whole repo |
|
||||
| Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review |
|
||||
|
||||
> [!TIP]
|
||||
> The push-triggered flow also runs a lightweight completeness check for the fleet's own VM-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel — a small nudge against exactly the kind of documentation drift this whole repo exists to avoid.
|
||||
|
||||
## Outside the orchestration engine entirely
|
||||
|
||||
The automation host's own OS patches and the orchestration engine's own container updates run via a standalone host-level timer, not through the orchestration engine itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run. Being outside the normal flow means these updates don't get the same visibility as everything else, worth a periodic manual check rather than fully trusting the dashboard.
|
||||
|
||||
---
|
||||
Related stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)
|
||||
Reference in new issue
Block a user