From dd05ecf0e861811f841d46f2543e6adde0930119 Mon Sep 17 00:00:00 2001 From: thesabear Date: Fri, 28 Aug 2026 13:11:37 +0300 Subject: [PATCH] orchestration: fix major mischaracterization - only 1 of 13 flows has a human gate Live-checked every flow's actual Kestra definition for a Pause task. Only app-update has one; the other 6 main flows (os-update-vms, container-update, disk-cleanup, os-update-pbs, os-update-pve, os-update-pve-verify) run fully automatically with no human checkpoint at all - their safety net is the automated health-check/rollback path, not human approval. The doc previously presented check->pause->approve as the universal pattern for all 13 flows and stated outright that 'nothing gets changed on the fleet without a human explicitly resuming a paused flow', which was false for the flows that actually change the fleet. The KB was already correct on this point. Also replaced the mermaid diagram (broke this repo's established Nextcloud-SVG convention) with a real diagram showing both patterns, plus live evidence of the Pause-task check itself. --- assets/orchestration-flow-patterns.svg | 4 +++ docs/orchestration.md | 39 +++++++++++++++----------- 2 files changed, 26 insertions(+), 17 deletions(-) create mode 100644 assets/orchestration-flow-patterns.svg diff --git a/assets/orchestration-flow-patterns.svg b/assets/orchestration-flow-patterns.svg new file mode 100644 index 0000000..0c48165 --- /dev/null +++ b/assets/orchestration-flow-patterns.svg @@ -0,0 +1,4 @@ + + + +
12 OF 13 FLOWS — no human gate
Weekly / scheduled trigger
Runs the playbook directly
os-update-vms, container-update, disk-cleanup, os-update-pbs, dashy-fleet-audit, and 7 more
Notify: succeeded / failed
1 OF 13 FLOWS (app-update) — check → pause → human approval → apply
Weekly schedule
Check mode run
detect pending Vault/Technitium/Alloy updates
Anything pending?
Notify: up to date
Notify: pending
PAUSE — indefinite, no timeout
Human resumes
Apply mode run
Skipped this cycle
Notify: applied / failed
no
yes
approve
decline
 \ No newline at end of file diff --git a/docs/orchestration.md b/docs/orchestration.md index 784e653..0da08a3 100644 --- a/docs/orchestration.md +++ b/docs/orchestration.md @@ -1,28 +1,33 @@ # 🔁 Orchestration > [!NOTE] -> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes. This is the layer that actually *acts* on the fleet, which is why it's the most tightly gated system in this repo. +> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint. -## Why a human-gated pattern instead of full automation +## Why only one flow gets a human gate -Fully automatic updates were the obvious simpler design, and they were rejected specifically because the cost of a bad automatic change (a broken update applied fleet-wide at 2am with nobody watching) is higher than the cost of a delayed one. The check → pause → approve pattern below trades speed for a human checkpoint at exactly the moment something is about to actually change — cheap insurance against the failure mode that matters most in a system with no on-call rotation behind it. +A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in [`docs/registry-and-build.md`](registry-and-build.md)), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't. -## The core pattern +## Two patterns, not one + +![12 of 13 flows run straight through on schedule with no human gate, relying on the automated health-check/rollback path instead; the one flow applying Vault/Technitium/Alloy updates runs a check-mode pass, pauses indefinitely on anything pending, and only applies after explicit human approval](../assets/orchestration-flow-patterns.svg) + +Nothing changes on Vault, Technitium, or Alloy without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every other flow's safety net is automated rollback, not a person watching. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert. + +This is the real check, not an assumption — every flow's own definition, queried directly for a pause step: -```mermaid -flowchart LR - Schedule["Weekly schedule"] --> Check["Check mode run
detect pending updates"] - Check --> Found{"Anything
pending?"} - Found -->|no| NotifyClean["Notify: up to date"] - Found -->|yes| NotifyPending["Notify: pending"] - NotifyPending --> Pause["Pause
indefinite, no timeout"] - Pause --> Decide{"Human
resumes"} - Decide -->|approve| Apply["Apply mode run"] - Decide -->|decline| Skip["Skipped this cycle"] - Apply --> NotifyResult["Notify: applied / failed"] ``` - -Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert. +$ for flow in os-update-vms os-update-pve os-update-pve-verify os-update-pbs \ + container-update disk-cleanup app-update; do + echo "$flow: $(curl -s .../flows/homelab/$flow | grep -c 'core.flow.Pause')" + done +os-update-vms: 0 +os-update-pve: 0 +os-update-pve-verify: 0 +os-update-pbs: 0 +container-update: 0 +disk-cleanup: 0 +app-update: 1 +``` ## Identity split, per flow