public/docs/orchestration.md

<!-- verified: 2026-09-19
     ansible@24e1bf0 : host_vars/hyper2.yml, roles/os_update/tasks/reboot.yml, playbooks/os_update_pve_verify.yml, playbooks/automation_self_update.yml
     trilium : bW84ccRoAJ5i, tt4idCVJPpZM
-->

# 🔁 Orchestration

> [!NOTE]
> Kestra runs every scheduled maintenance flow across the fleet — 15 flows covering OS updates, container updates, app updates, disk cleanup, fleet inventory, KB self-maintenance, and public-doc drift checks. Most run straight through on their own schedule with nobody watching; exactly one is built around a check → pause → human approval → apply pattern instead, reserved specifically for the one category of change judged worth a checkpoint.

This is the real flow list, not a claimed count:

![Kestra's flow list for the homelab namespace — every flow showing a real last-execution date and SUCCESS status](../assets/orchestration-flow-list.png)

## Why Only One Flow Gets a Human Gate

A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in [`docs/registry-and-build.md`](registry-and-build.md)), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't.

## Two Patterns, Not One

![14 of 15 flows run straight through on schedule with no human gate, relying on the automated health-check/rollback path instead; the one flow applying Vault/Technitium/Alloy updates runs a check-mode pass, pauses indefinitely on anything pending, and only applies after explicit human approval](../assets/orchestration-flow-patterns.svg)

That's not just a diagram of the pattern — it's the actual flow's own topology in Kestra, `wait_for_approval` sitting right there as a real node between the pending-check and the apply step:

![app-update's real execution topology in Kestra: check → has_pending → notify_pending → wait_for_approval → approved/notify_declined, then apply → apply_result → notify_applied/notify_apply_failed](../assets/orchestration-app-update-topology.png)

Nothing changes on Vault, Technitium, or Alloy without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it, and that's treated as acceptable, not as a workflow to be optimized away. Every other flow's safety net is automated rollback, not a person watching. Every notification in this pattern routes through the shared pipeline in [`docs/notifications.md`](notifications.md), so a pending-approval flow shows up in the same place as every other alert.

This is the real check, not an assumption — every flow's own definition, queried directly for a pause step:

```
$ for flow in os-update-vms os-update-pve os-update-pve-verify os-update-pbs \
              container-update disk-cleanup app-update homelab-inventory-sync \
              public-docs-drift; do
    echo "$flow: $(curl -s .../flows/homelab/$flow | grep -c 'core.flow.Pause')"
  done
os-update-vms: 0
os-update-pve: 0
os-update-pve-verify: 0
os-update-pbs: 0
container-update: 0
disk-cleanup: 0
app-update: 1
homelab-inventory-sync: 0
public-docs-drift: 0
```

## One Dedicated Identity, Not a Shared One

> [!IMPORTANT]
> Every flow connects to automation using the exact same credential — a dedicated keypair that exists for this one purpose and nothing else, unrelated to the `ansible` automation identity used everywhere else in this repo. That separation is deliberate, not incidental: Kestra's SSH task plugin is a plain Java SSH client with no certificate support at all, so it can only ever authenticate with a static key — it has no way to participate in the certificate-based model the rest of the fleet's `ansible` identity uses (see [`docs/secrets-and-pki.md`](secrets-and-pki.md)). Giving Kestra its own key instead of reusing `ansible`'s means a leak of one credential can never expose the other — the blast radius of "Kestra's automation-host hop" and "the fleet-wide `ansible` identity" stay fully independent. This is the real config, not a paraphrase — every SSH task in every flow points at the same two KV entries:
>
> ```
> $ curl -s .../flows/homelab/app-update?source=true | grep -A2 'type: io.kestra.plugin.fs.ssh.Command'
> type: io.kestra.plugin.fs.ssh.Command
> ...
> username: "{{ kv('kestra_local_user') }}"
> privateKey: "{{ kv('kestra_local_key') }}"
> ```
>
> The SSH hop itself is only ever a means to escape Kestra's own container and reach the control node's local filesystem and tooling — it's never used to reach the wider fleet. `ansible-playbook`, once it's actually running (as a local process, invoked over that same SSH hop), makes its own outbound connections to every other fleet host, using the fleet-wide `ansible` identity and its own certificate.

This is the actual KV Store, not a description of it — exactly two entries, nothing else:

![Kestra's homelab-namespace KV Store: exactly two entries, kestra_local_key and kestra_local_user, the dedicated credential every flow's SSH task uses](../assets/orchestration-kestra-kv-store.png)

## A Real Special Case, Thought Through In Advance

> [!WARNING]
> One of the two Proxmox hypervisor nodes also happens to host the automation VM that Kestra itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death, since the flow doing the waiting dies with the VM it's waiting on. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM, including the notification pipeline itself from [`docs/notifications.md`](notifications.md), which also runs there. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. This is the kind of edge case that only shows up once you actually think through "what happens when the thing doing the orchestrating is itself one of the things being orchestrated" — a real, non-obvious problem in any self-hosted automation system. This is the real config comment, not a summary of the reasoning:
>
> ```
> $ grep -B5 reboot_wait_for_reconnect host_vars/hyper2.yml
> # hyper2 hosts the Automation VM, which runs Kestra/ansible-playbook itself.
> # A reboot of hyper2 takes down the control node mid-play, so os_update's
> # reboot.yml must not block waiting to reconnect - see roles/os_update/tasks/
> # reboot.yml and playbooks/os_update_pve_verify.yml (confirms recovery
> # separately, via a follow-up Kestra flow).
> reboot_wait_for_reconnect: false
> ```

## Trilium KB Self-maintenance

Two flows keep this fleet's own internal knowledge base structure current. Those two, plus one more that only ever tags notes for review without altering their content, are the only things that write to the KB at all — everything else, including this repo's own generation, strictly reads from it:

| Flow | Trigger | What it does |
|---|---|---|
| KB sync | Webhook, on every push | Mirrors exact file changes from any tracked repo into the KB, upserts/deletes only what changed — never re-walks a whole repo |
| Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review |

> [!TIP]
> The push-triggered flow also runs a lightweight completeness check for the fleet's own host-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel via [`docs/notifications.md`](notifications.md) — a small, automatic nudge against exactly the kind of documentation drift that this whole portfolio repo exists to avoid in the first place.

## Outside The Orchestration Engine Entirely

The automation host's own OS patches and Kestra's own container updates run via a standalone host-level systemd timer, not through Kestra itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run without risking killing the process doing the patching. Being outside the normal flow means these updates don't get the same visibility as everything else in this doc, which is a deliberate, accepted tradeoff — worth a periodic manual check rather than fully trusting the dashboard to surface it. This is the real timer, not an assumption about its schedule:

```
$ systemctl list-timers automation-self-update.timer
NEXT                          LEFT     LAST                          PASSED   UNIT
Sat 2026-09-26 04:00:00 EEST  6 days   Sat 2026-09-19 04:00:09 EEST  5h ago   automation-self-update.timer
```

---
Related: [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/disaster-recovery.md`](disaster-recovery.md) · Stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)