public/docs/cron-scheduler.md

⏱️ Cron Scheduler

Note

A NocoDB-backed inventory of every known scheduled job in the fleet — Kestra flows, systemd timers, Ansible-managed cron, app-internal schedules — visualized as a Dashy heatmap so a new job's time can be picked without colliding with existing load. Small in scope compared to the rest of this repo, but it's a real answer to a problem every one of the other systems here creates: once you have a dozen-plus independently-scheduled jobs across docs/orchestration.md, docs/registry-and-build.md, and docs/disaster-recovery.md, "when does anything run" stops being something you can hold in your head.

The heatmap

flowchart LR
    Sources["Job sources<br/>Kestra flows · systemd timers ·<br/>Ansible cron · app schedules"] --> Table["NocoDB table<br/>name, host, category,<br/>schedule, duration, resource tags"]
    Table --> Weight["Weighted load calculation"]
    Weight --> Grid["7×24 grid<br/>day × hour"]
    Grid --> Bands["4 color bands<br/>quieter → busier"]

Each cell's color reflects a computed load weight, not a raw job count: a base weight per job, plus a duration-based weight, plus resource-intensity weight (disk/CPU/network, summed if a job carries multiple tags), plus an extra weight for anything tagged as a backup job — backup jobs get weighted higher deliberately, since disk I/O contention from an overlapping backup is a worse collision than two lightweight jobs running at the same hour. Jobs whose time can't be resolved to a specific day and hour are excluded from the grid and listed separately rather than forced into a misleading cell.

Tip

Clicking a populated cell expands the full job list for that slot — name, host, category, and whether that job's schedule can actually be moved or is fixed by whatever app owns it. That last distinction matters in practice: there's no point suggesting a new job avoid a time slot occupied by something that can't be rescheduled anyway versus one that easily could be.

Why a manual inventory instead of an auto-discovered one

The harder, more "correct" version of this tool would auto-discover every scheduled job by querying Kestra's API, parsing every host's crontab and systemd timers, and re-syncing continuously. That was scoped down deliberately: auto-discovery across three genuinely different scheduling mechanisms (Kestra's API, systemd's timer units, cron's own format) is real engineering effort for a tool whose entire job is picking a good time slot for the next new job — a task that happens occasionally, not continuously. A manually-maintained table, kept honest by being checked at the one moment it actually matters (adding a new job), was judged the better return on effort than building and maintaining three different scanners for a problem that doesn't need real-time accuracy.

Warning

This is a manually-maintained inventory, and that's stated plainly, not glossed over. The schema has a "last synced" column, but nothing currently populates it — no automated process re-scans the fleet to keep this current. Entries get added or edited by hand as jobs are created or changed, which means the tool is only as accurate as the discipline of updating it — a real limitation of the scoping decision above, not hidden behind a column that implies more automation than actually exists.

A real, unresolved bug this system surfaced

Important

Building this inventory surfaced something that had been silently broken: one backup job's prune step targets a datastore name that doesn't actually exist on the backup server described in docs/disaster-recovery.md — not a scheduling collision, an orphaned config that's been silently failing every run. Two fixes are on the table (drop the dead job and let a different retention mechanism be authoritative, or repoint it at the real datastore name and let its own richer retention rules take over) — deliberately left unresolved here rather than silently patched over with a guess, since the right fix depends on a retention-policy decision that's genuinely still open, not a technical one that just needs picking.

---

Related: docs/orchestration.md, docs/disaster-recovery.md, docs/dashy-soc-board.md · Stacks: stacks/platform/dashy/, stacks/platform/nocodb/