diff --git a/assets/orchestration-flow-patterns.svg b/assets/orchestration-flow-patterns.svg new file mode 100644 index 0000000..0c48165 --- /dev/null +++ b/assets/orchestration-flow-patterns.svg @@ -0,0 +1,4 @@ + + + +
12 OF 13 FLOWS — no human gate
Weekly / scheduled trigger
Runs the playbook directly
os-update-vms, container-update, disk-cleanup, os-update-pbs, dashy-fleet-audit, and 7 more
Notify: succeeded / failed
1 OF 13 FLOWS (app-update) — check → pause → human approval → apply
Weekly schedule
Check mode run
detect pending Vault/Technitium/Alloy updates
Anything pending?
Notify: up to date
Notify: pending
PAUSE — indefinite, no timeout
Human resumes
Apply mode run
Skipped this cycle
Notify: applied / failed
no
yes
approve
decline
\ No newline at end of file diff --git a/docs/orchestration.md b/docs/orchestration.md index 784e653..0da08a3 100644 --- a/docs/orchestration.md +++ b/docs/orchestration.md @@ -1,28 +1,33 @@ # 🔁 Orchestration > [!NOTE] -> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes. This is the layer that actually *acts* on the fleet, which is why it's the most tightly gated system in this repo. +> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint. -## Why a human-gated pattern instead of full automation +## Why only one flow gets a human gate -Fully automatic updates were the obvious simpler design, and they were rejected specifically because the cost of a bad automatic change (a broken update applied fleet-wide at 2am with nobody watching) is higher than the cost of a delayed one. The check → pause → approve pattern below trades speed for a human checkpoint at exactly the moment something is about to actually change — cheap insurance against the failure mode that matters most in a system with no on-call rotation behind it. +A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in [`docs/registry-and-build.md`](registry-and-build.md)), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't. -## The core pattern +## Two patterns, not one + +![12 of 13 flows run straight through on schedule with no human gate, relying on the automated health-check/rollback path instead; the one flow applying Vault/Technitium/Alloy updates runs a check-mode pass, pauses indefinitely on anything pending, and only applies after explicit human approval](../assets/orchestration-flow-patterns.svg) + +Nothing changes on Vault, Technitium, or Alloy without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every other flow's safety net is automated rollback, not a person watching. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert. + +This is the real check, not an assumption — every flow's own definition, queried directly for a pause step: -```mermaid -flowchart LR - Schedule["Weekly schedule"] --> Check["Check mode run
detect pending updates"] - Check --> Found{"Anything
pending?"} - Found -->|no| NotifyClean["Notify: up to date"] - Found -->|yes| NotifyPending["Notify: pending"] - NotifyPending --> Pause["Pause
indefinite, no timeout"] - Pause --> Decide{"Human
resumes"} - Decide -->|approve| Apply["Apply mode run"] - Decide -->|decline| Skip["Skipped this cycle"] - Apply --> NotifyResult["Notify: applied / failed"] ``` - -Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert. +$ for flow in os-update-vms os-update-pve os-update-pve-verify os-update-pbs \ + container-update disk-cleanup app-update; do + echo "$flow: $(curl -s .../flows/homelab/$flow | grep -c 'core.flow.Pause')" + done +os-update-vms: 0 +os-update-pve: 0 +os-update-pve-verify: 0 +os-update-pbs: 0 +container-update: 0 +disk-cleanup: 0 +app-update: 1 +``` ## Identity split, per flow