public/docs/orchestration.md

# 🔁 Orchestration

> [!NOTE]
> Kestra runs every scheduled maintenance flow across the fleet — 13 flows covering OS updates, container updates, app updates, disk cleanup, and KB self-maintenance. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint.

This is the real flow list, not a claimed count:

![Kestra's real flow list, homelab namespace: 13 flows total, every one showing a real last-execution date and SUCCESS status](../assets/orchestration-flow-list.png)

## Why Only One Flow Gets a Human Gate

A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in [`docs/registry-and-build.md`](registry-and-build.md)), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't.

## Two Patterns, Not One

![12 of 13 flows run straight through on schedule with no human gate, relying on the automated health-check/rollback path instead; the one flow applying Vault/Technitium/Alloy updates runs a check-mode pass, pauses indefinitely on anything pending, and only applies after explicit human approval](../assets/orchestration-flow-patterns.svg)

That's not just a diagram of the pattern — it's the actual flow's own topology in Kestra, `wait_for_approval` sitting right there as a real node between the pending-check and the apply step:

![app-update's real execution topology in Kestra: check → has_pending → notify_pending → wait_for_approval → approved/notify_declined, then apply → apply_result → notify_applied/notify_apply_failed](../assets/orchestration-app-update-topology.png)

Nothing changes on Vault, Technitium, or Alloy without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every other flow's safety net is automated rollback, not a person watching. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert.

This is the real check, not an assumption — every flow's own definition, queried directly for a pause step:

```
$ for flow in os-update-vms os-update-pve os-update-pve-verify os-update-pbs \
              container-update disk-cleanup app-update; do
    echo "$flow: $(curl -s .../flows/homelab/$flow | grep -c 'core.flow.Pause')"
  done
os-update-vms: 0
os-update-pve: 0
os-update-pve-verify: 0
os-update-pbs: 0
container-update: 0
disk-cleanup: 0
app-update: 1
```

## Identity Split, Per Flow

> [!IMPORTANT]
> Most flows connect using one operator identity over SSH. One flow — the one that needs to run Ansible itself on the control node, not just reach out over SSH to a target — deliberately logs in as a separate, narrower service identity instead, since it needs local execution rights on the control node that the SSH-only identity doesn't have. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally, following the same "scope credentials to exactly what's needed" instinct as the least-privilege Git access model in [`docs/sso-and-identity.md`](sso-and-identity.md). Worth checking which one a given flow uses before assuming — it's not always the same answer, and assuming wrong is a fast way to get a confusing permission error mid-debugging.

## A Real Special Case, Thought Through In Advance

> [!WARNING]
> One of the two Proxmox hypervisor nodes also happens to host the automation VM that Kestra itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death, since the flow doing the waiting dies with the VM it's waiting on. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM, including the notification pipeline itself from [`docs/notifications.md`](notifications.md), which also runs there. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. This is the kind of edge case that only shows up once you actually think through "what happens when the thing doing the orchestrating is itself one of the things being orchestrated" — a real, non-obvious problem in any self-hosted automation system. This is the real config comment, not a summary of the reasoning:
>
> ```
> $ grep -B5 reboot_wait_for_reconnect host_vars/hyper2.yml
> # hyper2 hosts the Automation VM, which runs Kestra/ansible-playbook itself.
> # A reboot of hyper2 takes down the control node mid-play, so os_update's
> # reboot.yml must not block waiting to reconnect - see roles/os_update/tasks/
> # reboot.yml and playbooks/os_update_pve_verify.yml (confirms recovery
> # separately, via a follow-up Kestra flow).
> reboot_wait_for_reconnect: false
> ```

## Trilium KB Self-maintenance

Two flows keep this fleet's own internal knowledge base structure current — the *only* two things allowed to write to it, everything else (including this repo's own generation) only ever reads from it:

| Flow | Trigger | What it does |
|---|---|---|
| KB sync | Webhook, on every push | Mirrors exact file changes from any tracked repo into the KB, upserts/deletes only what changed — never re-walks a whole repo |
| Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review |

> [!TIP]
> The push-triggered flow also runs a lightweight completeness check for the fleet's own host-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel via [`docs/notifications.md`](notifications.md) — a small, automatic nudge against exactly the kind of documentation drift that this whole portfolio repo exists to avoid in the first place.

## Outside The Orchestration Engine Entirely

The automation host's own OS patches and Kestra's own container updates run via a standalone host-level systemd timer, not through Kestra itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run without risking killing the process doing the patching. Being outside the normal flow means these updates don't get the same visibility as everything else in this doc, which is a deliberate, accepted tradeoff — worth a periodic manual check rather than fully trusting the dashboard to surface it. This is the real timer, not an assumption about its schedule:

```
$ systemctl list-timers automation-self-update.timer
NEXT                          LEFT  LAST                          PASSED  UNIT
Sat 2026-08-29 04:00:00 EEST  ...   Sat 2026-08-22 04:00:26 EEST   -      automation-self-update.timer
```

---
Related: [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/disaster-recovery.md`](disaster-recovery.md) · Stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)