public/docs/cron-scheduler.md

⏱️ Cron Scheduler

Note

A NocoDB-backed inventory of every known scheduled job in the fleet — Kestra flows, systemd timers, Ansible-managed cron, app-internal schedules — visualized as a Dashy heatmap so a new job's time can be picked without colliding with existing load. Small in scope compared to the rest of this repo, but it's a real answer to a problem every one of the other systems here creates: once you have 50 independently-scheduled jobs across docs/orchestration.md, docs/registry-and-build.md, and docs/disaster-recovery.md, "when does anything run" stops being something you can hold in your head.

The heatmap

A Node-RED tab fetches the inventory straight from NocoDB's REST API at request time — no cache — and renders it as a 7×24 grid (Mon–Sun × 00:00–23:00). Each cell's color is a computed load weight, not a raw job count: base 1 per job, plus a duration-bucket weight, plus a resource-tag weight (disk-intensive=2, cpu-intensive=1.5, network-intensive=0.5, summed if a job carries more than one), plus a flat +1 if the job's category is backup — disk I/O contention from an overlapping backup is judged a worse collision than two lightweight jobs sharing an hour, and the weight formula says so explicitly rather than pretending every job costs the same. Weights bucket into 4 color bands (green → yellow → orange → red) relative to the current busiest cell. Jobs whose time can't be resolved to a specific day+hour are excluded from the grid and listed separately rather than forced into a misleading cell.

This is the real table, not a description of it:

$ curl -s -H "xc-token: [REDACTED]" \
  "https://nocodb.[internal zone]/api/v2/tables/[table-id]/records?limit=100" | jq '.list | length'
50
$ ... | jq '.list | group_by(.Category) | map({category: .[0].Category, count: length})'
[{"category":"backup","count":6},{"category":"maintenance","count":23},
 {"category":"media","count":11},{"category":"monitoring","count":5},
 {"category":"security","count":5}]

Tip

Clicking a populated cell expands the full job list for that slot — name, host, category, and whether that job's schedule can actually be moved or is fixed by whatever app owns it. That last distinction matters in practice: there's no point suggesting a new job avoid a time slot occupied by something that can't be rescheduled anyway versus one that easily could be.

Why a manual inventory instead of an auto-discovered one

The harder, more "correct" version of this tool would auto-discover every scheduled job by querying Kestra's API, parsing every host's crontab and systemd timers, and re-syncing continuously. That was scoped down deliberately: auto-discovery across three genuinely different scheduling mechanisms (Kestra's API, systemd's timer units, cron's own format) is real engineering effort for a tool whose entire job is picking a good time slot for the next new job — a task that happens occasionally, not continuously. A manually-maintained table, kept honest by being checked at the one moment it actually matters (adding a new job), was judged the better return on effort than building and maintaining three different scanners for a problem that doesn't need real-time accuracy.

Warning

This is a manually-maintained inventory, and that's stated plainly, not glossed over. The schema has a "Last Synced" column, but nothing currently populates it on any of the 50 rows — no automated process re-scans the fleet to keep this current. Entries get added or edited by hand as jobs are created or changed, which means the tool is only as accurate as the discipline of updating it — a real limitation of the scoping decision above, not hidden behind a column that implies more automation than actually exists.

What building the inventory actually found

Populating 50 rows by hand across a dozen-plus different systems wasn't just data entry — it surfaced real scheduling conflicts and real bad assumptions:

• A three-way monthly collision, resolved by moving one job. PBS's own datastore garbage-collection job used to run first-Saturday-of-month 02:00 — the same slot as the weekly os-update-vms Kestra flow and the dashy-fleet-audit flow, both of which also land on Saturday. Moved to first-Tuesday-of-month 10:00 instead, clearing all three out of each other's way in one change rather than shuffling the other two around it.
• A cascading reschedule, not an isolated fix. Kopia's backup snapshot used to run 07:00 — directly on top of Proxmox hyper2's own fixed VM-backup slot (disk-intensive, kept fixed on purpose even while currently disabled). Moving Kopia to 08:00 then put it on top of ClamAV's scan, so ClamAV moved again to 09:00 — one collision fix triggering a second, tracked and resolved rather than left half-done.
• Don't trust a duration number until it comes from a real completed run. app-update's first-logged duration was ~2-3 seconds — which turned out to be sampled from killed, failed, and paused runs, not a real completion. The actual number, from a genuine successful run, is ~16 minutes: an 8x difference between "what the data said" and "what actually happened," caught only by checking.
• The heaviest job in the entire fleet isn't a backup. Radarr's own Refresh Movie task runs ~1h15min daily — heavier than any single PVE backup job, and the heaviest scheduled task found anywhere in the *arr media suite. Radarr's own backup job, by contrast, finishes in about 41 seconds.
• A 404 that looked like an auth problem wasn't one. Pulling Authentik's 18 internal task schedules initially 404'd against the endpoint documented for older versions (/admin/system_tasks/) — the real one had moved to /api/v3/tasks/schedules/ in the installed version. A wrong assumption about why something failed costs more time than the actual fix would have.

Two real, unresolved gaps this system surfaced

Important

A silently broken prune job. PBS's own prune job targets a datastore name that doesn't actually exist on the backup server — only one real datastore name is in use elsewhere on the same server. Not a scheduling collision, an orphaned config that's been failing every run without anyone noticing. Two fixes are on the table, deliberately left unresolved rather than guessed at: drop the dead job and let Proxmox VE's own keep-last retention be the sole authority, or repoint it at the real datastore name and let PBS's own richer daily/weekly/monthly retention rules take over instead. Moot for now — the PVE backup jobs it would matter for are currently disabled — but worth resolving before they're ever turned back on, not after.

Warning

A config that looks active but isn't. Technitium's blocklist auto-update is configured with a 24-hour interval — but no actual blocklist URLs are set, so it's a dormant no-op: the schedule exists, nothing downloads. Same category of gap as the PBS prune job — a setting that reads as "handled" until someone actually checks whether it does anything.

---

Related: docs/orchestration.md, docs/disaster-recovery.md, docs/dashy-soc-board.md · Stacks: stacks/platform/dashy/, stacks/platform/nocodb/