thesabear
1382a63dae
Same treatment as the security batch - real tech names, 'why this vs the alternative' reasoning, heavy cross-linking, ranked/specific tradeoffs instead of generic hedges.
55 lines
5.8 KiB
Markdown
55 lines
5.8 KiB
Markdown
# 🔁 Orchestration
|
|
|
|
> [!NOTE]
|
|
> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes. This is the layer that actually *acts* on the fleet, which is why it's the most tightly gated system in this repo.
|
|
|
|
## Why a human-gated pattern instead of full automation
|
|
|
|
Fully automatic updates were the obvious simpler design, and they were rejected specifically because the cost of a bad automatic change (a broken update applied fleet-wide at 2am with nobody watching) is higher than the cost of a delayed one. The check → pause → approve pattern below trades speed for a human checkpoint at exactly the moment something is about to actually change — cheap insurance against the failure mode that matters most in a system with no on-call rotation behind it.
|
|
|
|
## The core pattern
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
Schedule["Weekly schedule"] --> Check["Check mode run<br/>detect pending updates"]
|
|
Check --> Found{"Anything<br/>pending?"}
|
|
Found -->|no| NotifyClean["Notify: up to date"]
|
|
Found -->|yes| NotifyPending["Notify: pending"]
|
|
NotifyPending --> Pause["Pause<br/>indefinite, no timeout"]
|
|
Pause --> Decide{"Human<br/>resumes"}
|
|
Decide -->|approve| Apply["Apply mode run"]
|
|
Decide -->|decline| Skip["Skipped this cycle"]
|
|
Apply --> NotifyResult["Notify: applied / failed"]
|
|
```
|
|
|
|
Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert.
|
|
|
|
## Identity split, per flow
|
|
|
|
> [!IMPORTANT]
|
|
> Most flows connect using one operator identity over SSH. One flow — the one that needs to run Ansible itself on the control node, not just reach out over SSH to a target — deliberately logs in as a separate, narrower service identity instead, since it needs local execution rights on the control node that the SSH-only identity doesn't have. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally, following the same "scope credentials to exactly what's needed" instinct as the least-privilege Git access model in [`docs/sso-and-identity.md`](sso-and-identity.md). Worth checking which one a given flow uses before assuming — it's not always the same answer, and assuming wrong is a fast way to get a confusing permission error mid-debugging.
|
|
|
|
## A real special case, thought through in advance
|
|
|
|
> [!WARNING]
|
|
> One of the two Proxmox hypervisor nodes also happens to host the automation VM that Kestra itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death, since the flow doing the waiting dies with the VM it's waiting on. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM, including the notification pipeline itself from [`docs/notifications.md`](notifications.md), which also runs there. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. This is the kind of edge case that only shows up once you actually think through "what happens when the thing doing the orchestrating is itself one of the things being orchestrated" — a real, non-obvious problem in any self-hosted automation system.
|
|
|
|
## Trilium KB self-maintenance
|
|
|
|
Two flows keep this fleet's own internal knowledge base structure current — the *only* two things allowed to write to it, everything else (including this repo's own generation) only ever reads from it:
|
|
|
|
| Flow | Trigger | What it does |
|
|
|---|---|---|
|
|
| KB sync | Webhook, on every push | Mirrors exact file changes from any tracked repo into the KB, upserts/deletes only what changed — never re-walks a whole repo |
|
|
| Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review |
|
|
|
|
> [!TIP]
|
|
> The push-triggered flow also runs a lightweight completeness check for the fleet's own host-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel via [`docs/notifications.md`](notifications.md) — a small, automatic nudge against exactly the kind of documentation drift that this whole portfolio repo exists to avoid in the first place.
|
|
|
|
## Outside the orchestration engine entirely
|
|
|
|
The automation host's own OS patches and Kestra's own container updates run via a standalone host-level systemd timer, not through Kestra itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run without risking killing the process doing the patching. Being outside the normal flow means these updates don't get the same visibility as everything else in this doc, which is a deliberate, accepted tradeoff — worth a periodic manual check rather than fully trusting the dashboard to surface it.
|
|
|
|
---
|
|
Related: [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/disaster-recovery.md`](disaster-recovery.md) · Stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)
|