orchestration: add real evidence for hyper2 special case, self-update timer, and 2 real screenshots
Step 5 re-audit, completing the checklist gap that started this whole detour. Added: the real hyper2 host_vars comment (matches the WARNING callout's reasoning almost word for word), the real automation-self- update.timer schedule, the real Kestra flow list (visually confirms 13 total flows), and app-update's real execution topology (visually confirms wait_for_approval as an actual node, not just a diagram concept).
This commit is contained in:
3 files changed
+26
-2
No files matched your search
Binary file not shown.
|
After Width: | Height: | Size: 148 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 238 KiB |
+26
-2
@@ -3,6 +3,10 @@
|
|||||||
> [!NOTE]
|
> [!NOTE]
|
||||||
> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint.
|
> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint.
|
||||||
|
|
||||||
|
This is the real flow list, not a claimed count:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
## Why only one flow gets a human gate
|
## Why only one flow gets a human gate
|
||||||
|
|
||||||
A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in [`docs/registry-and-build.md`](registry-and-build.md)), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't.
|
A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in [`docs/registry-and-build.md`](registry-and-build.md)), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't.
|
||||||
@@ -11,6 +15,10 @@ A blanket human-gate policy was the obvious simpler design, and it was rejected:
|
|||||||
|
|
||||||

|

|
||||||
|
|
||||||
|
That's not just a diagram of the pattern — it's the actual flow's own topology in Kestra, `wait_for_approval` sitting right there as a real node between the pending-check and the apply step:
|
||||||
|
|
||||||
|

|
||||||
|
|
||||||
Nothing changes on Vault, Technitium, or Alloy without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every other flow's safety net is automated rollback, not a person watching. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert.
|
Nothing changes on Vault, Technitium, or Alloy without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every other flow's safety net is automated rollback, not a person watching. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert.
|
||||||
|
|
||||||
This is the real check, not an assumption — every flow's own definition, queried directly for a pause step:
|
This is the real check, not an assumption — every flow's own definition, queried directly for a pause step:
|
||||||
@@ -37,7 +45,17 @@ app-update: 1
|
|||||||
## A real special case, thought through in advance
|
## A real special case, thought through in advance
|
||||||
|
|
||||||
> [!WARNING]
|
> [!WARNING]
|
||||||
> One of the two Proxmox hypervisor nodes also happens to host the automation VM that Kestra itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death, since the flow doing the waiting dies with the VM it's waiting on. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM, including the notification pipeline itself from [`docs/notifications.md`](notifications.md), which also runs there. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. This is the kind of edge case that only shows up once you actually think through "what happens when the thing doing the orchestrating is itself one of the things being orchestrated" — a real, non-obvious problem in any self-hosted automation system.
|
> One of the two Proxmox hypervisor nodes also happens to host the automation VM that Kestra itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death, since the flow doing the waiting dies with the VM it's waiting on. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM, including the notification pipeline itself from [`docs/notifications.md`](notifications.md), which also runs there. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. This is the kind of edge case that only shows up once you actually think through "what happens when the thing doing the orchestrating is itself one of the things being orchestrated" — a real, non-obvious problem in any self-hosted automation system. This is the real config comment, not a summary of the reasoning:
|
||||||
|
>
|
||||||
|
> ```
|
||||||
|
> $ grep -B5 reboot_wait_for_reconnect host_vars/hyper2.yml
|
||||||
|
> # hyper2 hosts the Automation VM, which runs Kestra/ansible-playbook itself.
|
||||||
|
> # A reboot of hyper2 takes down the control node mid-play, so os_update's
|
||||||
|
> # reboot.yml must not block waiting to reconnect - see roles/os_update/tasks/
|
||||||
|
> # reboot.yml and playbooks/os_update_pve_verify.yml (confirms recovery
|
||||||
|
> # separately, via a follow-up Kestra flow).
|
||||||
|
> reboot_wait_for_reconnect: false
|
||||||
|
> ```
|
||||||
|
|
||||||
## Trilium KB self-maintenance
|
## Trilium KB self-maintenance
|
||||||
|
|
||||||
@@ -53,7 +71,13 @@ Two flows keep this fleet's own internal knowledge base structure current — th
|
|||||||
|
|
||||||
## Outside the orchestration engine entirely
|
## Outside the orchestration engine entirely
|
||||||
|
|
||||||
The automation host's own OS patches and Kestra's own container updates run via a standalone host-level systemd timer, not through Kestra itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run without risking killing the process doing the patching. Being outside the normal flow means these updates don't get the same visibility as everything else in this doc, which is a deliberate, accepted tradeoff — worth a periodic manual check rather than fully trusting the dashboard to surface it.
|
The automation host's own OS patches and Kestra's own container updates run via a standalone host-level systemd timer, not through Kestra itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run without risking killing the process doing the patching. Being outside the normal flow means these updates don't get the same visibility as everything else in this doc, which is a deliberate, accepted tradeoff — worth a periodic manual check rather than fully trusting the dashboard to surface it. This is the real timer, not an assumption about its schedule:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ systemctl list-timers automation-self-update.timer
|
||||||
|
NEXT LEFT LAST PASSED UNIT
|
||||||
|
Sat 2026-08-29 04:00:00 EEST ... Sat 2026-08-22 04:00:26 EEST - automation-self-update.timer
|
||||||
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
Related: [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/disaster-recovery.md`](disaster-recovery.md) · Stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)
|
Related: [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/disaster-recovery.md`](disaster-recovery.md) · Stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)
|
||||||
Reference in new issue
Block a user