# 🔁 Orchestration

> [!NOTE]
> Thirteen scheduled flows handle fleet maintenance end to end — OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes.

## The core pattern

```mermaid
flowchart LR
    Schedule["Weekly schedule"] --> Check["Check mode run<br/>detect pending updates"]
    Check --> Found{"Anything<br/>pending?"}
    Found -->|no| NotifyClean["Notify: up to date"]
    Found -->|yes| NotifyPending["Notify: pending"]
    NotifyPending --> Pause["Pause<br/>indefinite, no timeout"]
    Pause --> Decide{"Human<br/>resumes"}
    Decide -->|approve| Apply["Apply mode run"]
    Decide -->|decline| Skip["Skipped this cycle"]
    Apply --> NotifyResult["Notify: applied / failed"]
```

Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it.

## Identity split, per flow

> [!IMPORTANT]
> Most flows connect using one operator identity. One flow — the one that needs to run the automation tool itself on the control node, not just reach out over SSH — deliberately logs in as a separate, narrower service identity instead. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally. Worth checking which one a given flow uses before assuming — it's not always the same answer.

## A real special case, thought through

> [!WARNING]
> One of the two hypervisor nodes also happens to host the automation VM that the orchestration engine itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification.

## Trilium KB self-maintenance

Two flows keep the internal knowledge base's own structure current — the *only* two things allowed to write to it:

| Flow | Trigger | What it does |
|---|---|---|
| KB sync | Webhook, on every push | Mirrors exact file changes from any tracked repo into the KB, upserts/deletes only what changed — never re-walks a whole repo |
| Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review |

> [!TIP]
> The push-triggered flow also runs a lightweight completeness check for the fleet's own VM-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel — a small nudge against exactly the kind of documentation drift this whole repo exists to avoid.

## Outside the orchestration engine entirely

The automation host's own OS patches and the orchestration engine's own container updates run via a standalone host-level timer, not through the orchestration engine itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run. Being outside the normal flow means these updates don't get the same visibility as everything else, worth a periodic manual check rather than fully trusting the dashboard.

---
Related stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)