orchestration: fix major mischaracterization - only 1 of 13 flows has a human gate
Live-checked every flow's actual Kestra definition for a Pause task. Only app-update has one; the other 6 main flows (os-update-vms, container-update, disk-cleanup, os-update-pbs, os-update-pve, os-update-pve-verify) run fully automatically with no human checkpoint at all - their safety net is the automated health-check/rollback path, not human approval. The doc previously presented check->pause->approve as the universal pattern for all 13 flows and stated outright that 'nothing gets changed on the fleet without a human explicitly resuming a paused flow', which was false for the flows that actually change the fleet. The KB was already correct on this point. Also replaced the mermaid diagram (broke this repo's established Nextcloud-SVG convention) with a real diagram showing both patterns, plus live evidence of the Pause-task check itself.
This commit is contained in:
2 files changed
+26
-17
No files matched your search
@@ -0,0 +1,4 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!-- Do not edit this file with editors other than draw.io -->
|
||||||
|
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1142px" height="642px" viewBox="0 0 1142 642" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Flow Patterns" id="orch-patterns-1"> <mxGraphModel dx="1414" dy="791" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1200" pageHeight="640" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="lane1" parent="1" style="swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#6c8ebf;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=left;spacingLeft=8;verticalAlign=middle;" value="12 OF 13 FLOWS — no human gate" vertex="1"> <mxGeometry height="140" width="1140" x="30" y="30" as="geometry" /> </mxCell> <mxCell id="l1s" parent="lane1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Weekly / scheduled trigger" vertex="1"> <mxGeometry height="60" width="220" x="40" y="50" as="geometry" /> </mxCell> <mxCell id="l1r" parent="lane1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Runs the playbook directly&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;os-update-vms, container-update, disk-cleanup, os-update-pbs, dashy-fleet-audit, and 7 more&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="420" x="360" y="45" as="geometry" /> </mxCell> <mxCell id="l1n" parent="lane1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Notify: succeeded / failed" vertex="1"> <mxGeometry height="60" width="220" x="880" y="50" as="geometry" /> </mxCell> <mxCell id="e1" edge="1" parent="lane1" source="l1s" style="html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;" target="l1r"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="e2" edge="1" parent="lane1" source="l1r" style="html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;" target="l1n"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="lane2" parent="1" style="swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#d79b00;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=left;spacingLeft=8;verticalAlign=middle;" value="1 OF 13 FLOWS (app-update) — check → pause → human approval → apply" vertex="1"> <mxGeometry height="480" width="1140" x="30" y="190" as="geometry" /> </mxCell> <mxCell id="l2s" parent="lane2" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Weekly schedule" vertex="1"> <mxGeometry height="60" width="180" x="40" y="40" as="geometry" /> </mxCell> <mxCell id="l2c" parent="lane2" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Check mode run&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;detect pending Vault/Technitium/Alloy updates&lt;/font&gt;" vertex="1">
Line truncated
|
||||||
|
After Width: | Height: | Size: 400 KiB |
+22
-17
@@ -1,28 +1,33 @@
|
|||||||
# 🔁 Orchestration
|
# 🔁 Orchestration
|
||||||
|
|
||||||
> [!NOTE]
|
> [!NOTE]
|
||||||
> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes. This is the layer that actually *acts* on the fleet, which is why it's the most tightly gated system in this repo.
|
> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint.
|
||||||
|
|
||||||
## Why a human-gated pattern instead of full automation
|
## Why only one flow gets a human gate
|
||||||
|
|
||||||
Fully automatic updates were the obvious simpler design, and they were rejected specifically because the cost of a bad automatic change (a broken update applied fleet-wide at 2am with nobody watching) is higher than the cost of a delayed one. The check → pause → approve pattern below trades speed for a human checkpoint at exactly the moment something is about to actually change — cheap insurance against the failure mode that matters most in a system with no on-call rotation behind it.
|
A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in [`docs/registry-and-build.md`](registry-and-build.md)), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't.
|
||||||
|
|
||||||
## The core pattern
|
## Two patterns, not one
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
|
Nothing changes on Vault, Technitium, or Alloy without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every other flow's safety net is automated rollback, not a person watching. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert.
|
||||||
|
|
||||||
|
This is the real check, not an assumption — every flow's own definition, queried directly for a pause step:
|
||||||
|
|
||||||
```mermaid
|
|
||||||
flowchart LR
|
|
||||||
Schedule["Weekly schedule"] --> Check["Check mode run<br/>detect pending updates"]
|
|
||||||
Check --> Found{"Anything<br/>pending?"}
|
|
||||||
Found -->|no| NotifyClean["Notify: up to date"]
|
|
||||||
Found -->|yes| NotifyPending["Notify: pending"]
|
|
||||||
NotifyPending --> Pause["Pause<br/>indefinite, no timeout"]
|
|
||||||
Pause --> Decide{"Human<br/>resumes"}
|
|
||||||
Decide -->|approve| Apply["Apply mode run"]
|
|
||||||
Decide -->|decline| Skip["Skipped this cycle"]
|
|
||||||
Apply --> NotifyResult["Notify: applied / failed"]
|
|
||||||
```
|
```
|
||||||
|
$ for flow in os-update-vms os-update-pve os-update-pve-verify os-update-pbs \
|
||||||
Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert.
|
container-update disk-cleanup app-update; do
|
||||||
|
echo "$flow: $(curl -s .../flows/homelab/$flow | grep -c 'core.flow.Pause')"
|
||||||
|
done
|
||||||
|
os-update-vms: 0
|
||||||
|
os-update-pve: 0
|
||||||
|
os-update-pve-verify: 0
|
||||||
|
os-update-pbs: 0
|
||||||
|
container-update: 0
|
||||||
|
disk-cleanup: 0
|
||||||
|
app-update: 1
|
||||||
|
```
|
||||||
|
|
||||||
## Identity split, per flow
|
## Identity split, per flow
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user