public/docs/orchestration.md

🔁 Orchestration

Note

Kestra runs every scheduled maintenance flow across the fleet — 15 flows covering OS updates, container updates, app updates, disk cleanup, fleet inventory, KB self-maintenance, and public-doc drift checks. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint.

This is the real flow list, not a claimed count:

Why Only One Flow Gets a Human Gate

A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in docs/registry-and-build.md), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't.

Two Patterns, Not One

That's not just a diagram of the pattern — it's the actual flow's own topology in Kestra, wait_for_approval sitting right there as a real node between the pending-check and the apply step:

Nothing changes on Vault, Technitium, or Alloy without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every other flow's safety net is automated rollback, not a person watching. Every notification in this pattern routes through the shared pipeline in docs/notifications.md, so a pending-approval flow shows up in the same place as every other alert.

This is the real check, not an assumption — every flow's own definition, queried directly for a pause step:

$ for flow in os-update-vms os-update-pve os-update-pve-verify os-update-pbs \
              container-update disk-cleanup app-update homelab-inventory-sync \
              public-docs-drift; do
    echo "$flow: $(curl -s .../flows/homelab/$flow | grep -c 'core.flow.Pause')"
  done
os-update-vms: 0
os-update-pve: 0
os-update-pve-verify: 0
os-update-pbs: 0
container-update: 0
disk-cleanup: 0
app-update: 1
homelab-inventory-sync: 0
public-docs-drift: 0

One Dedicated Identity, Not a Shared One

Important

Every flow connects to automation using the exact same credential — a dedicated keypair that exists for this one purpose and nothing else, unrelated to the ansible automation identity used everywhere else in this repo. That separation is deliberate, not incidental: Kestra's SSH task plugin is a plain Java SSH client with no certificate support at all, so it can only ever authenticate with a static key — it has no way to participate in the certificate-based model the rest of the fleet's ansible identity uses (see docs/secrets-and-pki.md). Giving Kestra its own key instead of reusing ansible's means a leak of one credential can never expose the other — the blast radius of "Kestra's automation-host hop" and "the fleet-wide ansible identity" stay fully independent. This is the real config, not a paraphrase — every SSH task in every flow points at the same two KV entries:

$ curl -s .../flows/homelab/app-update?source=true | grep -A2 'type: io.kestra.plugin.fs.ssh.Command'
type: io.kestra.plugin.fs.ssh.Command
...
username: "{{ kv('kestra_local_user') }}"
privateKey: "{{ kv('kestra_local_key') }}"

The SSH hop itself is only ever a means to escape Kestra's own container and reach the control node's local filesystem and tooling — it's never used to reach the wider fleet. ansible-playbook, once it's actually running (as a local process, invoked over that same SSH hop), makes its own outbound connections to every other fleet host, using the fleet-wide ansible identity and its own certificate.

This is the actual KV Store, not a description of it — exactly two entries, nothing else:

A Real Special Case, Thought Through In Advance

Warning

One of the two Proxmox hypervisor nodes also happens to host the automation VM that Kestra itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death, since the flow doing the waiting dies with the VM it's waiting on. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM, including the notification pipeline itself from docs/notifications.md, which also runs there. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. This is the kind of edge case that only shows up once you actually think through "what happens when the thing doing the orchestrating is itself one of the things being orchestrated" — a real, non-obvious problem in any self-hosted automation system. This is the real config comment, not a summary of the reasoning:

$ grep -B5 reboot_wait_for_reconnect host_vars/hyper2.yml
# hyper2 hosts the Automation VM, which runs Kestra/ansible-playbook itself.
# A reboot of hyper2 takes down the control node mid-play, so os_update's
# reboot.yml must not block waiting to reconnect - see roles/os_update/tasks/
# reboot.yml and playbooks/os_update_pve_verify.yml (confirms recovery
# separately, via a follow-up Kestra flow).
reboot_wait_for_reconnect: false

Trilium KB Self-maintenance

Two flows keep this fleet's own internal knowledge base structure current. Those two, plus one more that only ever tags notes for review without altering their content, are the only things that write to the KB at all — everything else, including this repo's own generation, strictly reads from it:

Flow	Trigger	What it does
KB sync	Webhook, on every push	Mirrors exact file changes from any tracked repo into the KB, upserts/deletes only what changed — never re-walks a whole repo
Structure index refresh	Every 6 hours	Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review

Tip

The push-triggered flow also runs a lightweight completeness check for the fleet's own host-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel via docs/notifications.md — a small, automatic nudge against exactly the kind of documentation drift that this whole portfolio repo exists to avoid in the first place.

Outside The Orchestration Engine Entirely

The automation host's own OS patches and Kestra's own container updates run via a standalone host-level systemd timer, not through Kestra itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run without risking killing the process doing the patching. Being outside the normal flow means these updates don't get the same visibility as everything else in this doc, which is a deliberate, accepted tradeoff — worth a periodic manual check rather than fully trusting the dashboard to surface it. This is the real timer, not an assumption about its schedule:

$ systemctl list-timers automation-self-update.timer
NEXT                          LEFT     LAST                          PASSED   UNIT
Sat 2026-09-12 04:00:00 EEST  6 days   Sat 2026-09-05 04:00:32 EEST  4h 17min automation-self-update.timer

---

Related: docs/notifications.md, docs/registry-and-build.md, docs/disaster-recovery.md · Stacks: stacks/platform/kestra/