public/docs/cron-scheduler.md

⏱️ Cron Scheduler

Note

A NocoDB-backed inventory of every known scheduled job in the fleet — Kestra flows, systemd timers, Ansible-managed cron, app-internal schedules — visualized as a Dashy heatmap so a new job's time can be picked without colliding with existing load. Small in scope compared to the rest of this repo, but it's a real answer to a problem every one of the other systems here creates: once you have 50 independently-scheduled jobs across docs/orchestration.md, docs/registry-and-build.md, and docs/disaster-recovery.md, "when does anything run" stops being something you can hold in your head.

The heatmap

A Node-RED tab fetches the inventory straight from NocoDB's REST API at request time — no cache — and renders it as a 7×24 grid (Mon–Sun × 00:00–23:00). Each cell's color is a computed load weight, not a raw job count: base 1 per job, plus a duration-bucket weight, plus a resource-tag weight (disk-intensive=2, cpu-intensive=1.5, network-intensive=0.5, summed if a job carries more than one), plus a flat +1 if the job's category is backup — disk I/O contention from an overlapping backup is judged a worse collision than two lightweight jobs sharing an hour, and the weight formula says so explicitly rather than pretending every job costs the same. Weights bucket into 4 color bands (green → yellow → orange → red) relative to the current busiest cell. Jobs whose time can't be resolved to a specific day+hour are excluded from the grid and listed separately rather than forced into a misleading cell.

This is the real heatmap, not a description of it — 50 known jobs, 15 without a fixed time:

Tip

Clicking a populated cell expands the full job list for that slot — name, host, category, and whether that job's schedule can actually be moved or is fixed by whatever app owns it. That last distinction matters in practice: there's no point suggesting a new job avoid a time slot occupied by something that can't be rescheduled anyway versus one that easily could be.

Jobs with no fixed time get their own list instead of a misleading grid cell:

Why a manual inventory instead of an auto-discovered one

The harder, more "correct" version of this tool would auto-discover every scheduled job by querying Kestra's API, parsing every host's crontab and systemd timers, and re-syncing continuously. That was scoped down deliberately: auto-discovery across three genuinely different scheduling mechanisms (Kestra's API, systemd's timer units, cron's own format) is real engineering effort for a tool whose entire job is picking a good time slot for the next new job — a task that happens occasionally, not continuously. A manually-maintained table, kept honest by being checked at the one moment it actually matters (adding a new job), was judged the better return on effort than building and maintaining three different scanners for a problem that doesn't need real-time accuracy.

Warning

This is a manually-maintained inventory, and that's stated plainly, not glossed over. The schema has a "Last Synced" column, but nothing currently populates it on any of the 50 rows — no automated process re-scans the fleet to keep this current. Entries get added or edited by hand as jobs are created or changed, which means the tool is only as accurate as the discipline of updating it — a real limitation of the scoping decision above, not hidden behind a column that implies more automation than actually exists.

What building the inventory actually found

Populating 50 rows by hand across a dozen-plus different systems wasn't just data entry — it surfaced real scheduling conflicts and real bad assumptions:

• A three-way monthly collision, resolved by moving one job. PBS's own datastore garbage-collection job used to run first-Saturday-of-month 02:00 — the same slot as the weekly os-update-vms Kestra flow and the dashy-fleet-audit flow, both of which also land on Saturday. Moved to first-Tuesday-of-month 10:00 instead, clearing all three out of each other's way in one change rather than shuffling the other two around it.
• A cascading reschedule, not an isolated fix. Kopia's backup snapshot used to run 07:00 — directly on top of Proxmox hyper2's own fixed VM-backup slot (disk-intensive, kept fixed on purpose even while currently disabled). Moving Kopia to 08:00 then put it on top of ClamAV's scan, so ClamAV moved again to 09:00 — one collision fix triggering a second, tracked and resolved rather than left half-done.
• Don't trust a duration number until it comes from a real completed run. app-update's first-logged duration was ~2-3 seconds — which turned out to be sampled from killed, failed, and paused runs, not a real completion. The actual number, from a genuine successful run, is ~16 minutes: an 8x difference between "what the data said" and "what actually happened," caught only by checking.
• The heaviest job in the entire fleet isn't a backup. Radarr's own Refresh Movie task runs ~1h15min daily — heavier than any single PVE backup job, and the heaviest scheduled task found anywhere in the *arr media suite. Radarr's own backup job, by contrast, finishes in about 41 seconds.
• A 404 that looked like an auth problem wasn't one. Pulling Authentik's 18 internal task schedules initially 404'd against the endpoint documented for older versions (/admin/system_tasks/) — the real one had moved to /api/v3/tasks/schedules/ in the installed version. A wrong assumption about why something failed costs more time than the actual fix would have.

---

Related: docs/orchestration.md, docs/disaster-recovery.md, docs/dashy-soc-board.md · Stacks: stacks/platform/dashy/, stacks/platform/nocodb/