From f2d52234373589c3b3a8804a337b3e472214176d Mon Sep 17 00:00:00 2001 From: thesabear Date: Fri, 28 Aug 2026 00:41:34 +0300 Subject: [PATCH] docs: rewrite cron-scheduler.md with real evidence from the live table Live-pulled the actual 50-row NocoDB inventory (also caught and fixed a stale KB fact along the way - the note pointed at the wrong NocoDB deployment entirely, nocodb-content/automation instead of the real nocodb/rocky-docker instance). Replaced the vague weight-formula prose with the real numbers, cut the mermaid diagram that only restated the prose above it, and added five real stories mined from populating the table by hand: a three-way monthly collision fix, a cascading Kopia->ClamAV reschedule, a duration figure corrected 8x after checking real completed-run data, the heaviest job in the whole fleet (not a backup), and a 404 that looked like an auth problem and wasn't. Enriched the pending PBS datastore bug with its real resolution options and added a second real gap: Technitium's blocklist auto-update is configured but silently a no-op. --- docs/cron-scheduler.md | 41 +++++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/docs/cron-scheduler.md b/docs/cron-scheduler.md index 62ba6b8..9d7e27c 100644 --- a/docs/cron-scheduler.md +++ b/docs/cron-scheduler.md @@ -1,19 +1,23 @@ # ⏱️ Cron Scheduler > [!NOTE] -> A NocoDB-backed inventory of every known scheduled job in the fleet — Kestra flows, systemd timers, Ansible-managed cron, app-internal schedules — visualized as a Dashy heatmap so a new job's time can be picked without colliding with existing load. Small in scope compared to the rest of this repo, but it's a real answer to a problem every one of the other systems here creates: once you have a dozen-plus independently-scheduled jobs across [`docs/orchestration.md`](orchestration.md), [`docs/registry-and-build.md`](registry-and-build.md), and [`docs/disaster-recovery.md`](disaster-recovery.md), "when does anything run" stops being something you can hold in your head. +> A NocoDB-backed inventory of every known scheduled job in the fleet — Kestra flows, systemd timers, Ansible-managed cron, app-internal schedules — visualized as a Dashy heatmap so a new job's time can be picked without colliding with existing load. Small in scope compared to the rest of this repo, but it's a real answer to a problem every one of the other systems here creates: once you have 50 independently-scheduled jobs across [`docs/orchestration.md`](orchestration.md), [`docs/registry-and-build.md`](registry-and-build.md), and [`docs/disaster-recovery.md`](disaster-recovery.md), "when does anything run" stops being something you can hold in your head. ## The heatmap -```mermaid -flowchart LR - Sources["Job sources
Kestra flows · systemd timers ·
Ansible cron · app schedules"] --> Table["NocoDB table
name, host, category,
schedule, duration, resource tags"] - Table --> Weight["Weighted load calculation"] - Weight --> Grid["7×24 grid
day × hour"] - Grid --> Bands["4 color bands
quieter → busier"] -``` +A Node-RED tab fetches the inventory straight from NocoDB's REST API at request time — no cache — and renders it as a 7×24 grid (Mon–Sun × 00:00–23:00). Each cell's color is a **computed load weight**, not a raw job count: base `1` per job, plus a duration-bucket weight, plus a resource-tag weight (`disk-intensive`=2, `cpu-intensive`=1.5, `network-intensive`=0.5, summed if a job carries more than one), plus a flat `+1` if the job's category is `backup` — disk I/O contention from an overlapping backup is judged a worse collision than two lightweight jobs sharing an hour, and the weight formula says so explicitly rather than pretending every job costs the same. Weights bucket into 4 color bands (green → yellow → orange → red) relative to the current busiest cell. Jobs whose time can't be resolved to a specific day+hour are excluded from the grid and listed separately rather than forced into a misleading cell. -Each cell's color reflects a **computed load weight**, not a raw job count: a base weight per job, plus a duration-based weight, plus resource-intensity weight (disk/CPU/network, summed if a job carries multiple tags), plus an extra weight for anything tagged as a backup job — backup jobs get weighted higher deliberately, since disk I/O contention from an overlapping backup is a worse collision than two lightweight jobs running at the same hour. Jobs whose time can't be resolved to a specific day and hour are excluded from the grid and listed separately rather than forced into a misleading cell. +This is the real table, not a description of it: + +``` +$ curl -s -H "xc-token: [REDACTED]" \ + "https://nocodb.[internal zone]/api/v2/tables/[table-id]/records?limit=100" | jq '.list | length' +50 +$ ... | jq '.list | group_by(.Category) | map({category: .[0].Category, count: length})' +[{"category":"backup","count":6},{"category":"maintenance","count":23}, + {"category":"media","count":11},{"category":"monitoring","count":5}, + {"category":"security","count":5}] +``` > [!TIP] > Clicking a populated cell expands the full job list for that slot — name, host, category, and whether that job's schedule can actually be moved or is fixed by whatever app owns it. That last distinction matters in practice: there's no point suggesting a new job avoid a time slot occupied by something that can't be rescheduled anyway versus one that easily could be. @@ -23,12 +27,25 @@ Each cell's color reflects a **computed load weight**, not a raw job count: a ba The harder, more "correct" version of this tool would auto-discover every scheduled job by querying Kestra's API, parsing every host's crontab and systemd timers, and re-syncing continuously. That was scoped down deliberately: auto-discovery across three genuinely different scheduling mechanisms (Kestra's API, systemd's timer units, cron's own format) is real engineering effort for a tool whose entire job is picking a good time slot for the next new job — a task that happens occasionally, not continuously. A manually-maintained table, kept honest by being checked at the one moment it actually matters (adding a new job), was judged the better return on effort than building and maintaining three different scanners for a problem that doesn't need real-time accuracy. > [!WARNING] -> **This is a manually-maintained inventory, and that's stated plainly, not glossed over.** The schema has a "last synced" column, but nothing currently populates it — no automated process re-scans the fleet to keep this current. Entries get added or edited by hand as jobs are created or changed, which means the tool is only as accurate as the discipline of updating it — a real limitation of the scoping decision above, not hidden behind a column that implies more automation than actually exists. +> **This is a manually-maintained inventory, and that's stated plainly, not glossed over.** The schema has a "Last Synced" column, but nothing currently populates it on any of the 50 rows — no automated process re-scans the fleet to keep this current. Entries get added or edited by hand as jobs are created or changed, which means the tool is only as accurate as the discipline of updating it — a real limitation of the scoping decision above, not hidden behind a column that implies more automation than actually exists. -## A real, unresolved bug this system surfaced +## What building the inventory actually found + +Populating 50 rows by hand across a dozen-plus different systems wasn't just data entry — it surfaced real scheduling conflicts and real bad assumptions: + +- **A three-way monthly collision, resolved by moving one job.** PBS's own datastore garbage-collection job used to run first-Saturday-of-month 02:00 — the same slot as the weekly `os-update-vms` Kestra flow *and* the `dashy-fleet-audit` flow, both of which also land on Saturday. Moved to first-Tuesday-of-month 10:00 instead, clearing all three out of each other's way in one change rather than shuffling the other two around it. +- **A cascading reschedule, not an isolated fix.** Kopia's backup snapshot used to run 07:00 — directly on top of Proxmox hyper2's own fixed VM-backup slot (disk-intensive, kept fixed on purpose even while currently disabled). Moving Kopia to 08:00 then put it on top of ClamAV's scan, so ClamAV moved again to 09:00 — one collision fix triggering a second, tracked and resolved rather than left half-done. +- **Don't trust a duration number until it comes from a real completed run.** `app-update`'s first-logged duration was ~2-3 seconds — which turned out to be sampled from killed, failed, and paused runs, not a real completion. The actual number, from a genuine successful run, is ~16 minutes: an 8x difference between "what the data said" and "what actually happened," caught only by checking. +- **The heaviest job in the entire fleet isn't a backup.** Radarr's own `Refresh Movie` task runs ~1h15min daily — heavier than any single PVE backup job, and the heaviest scheduled task found anywhere in the *arr media suite. Radarr's own backup job, by contrast, finishes in about 41 seconds. +- **A 404 that looked like an auth problem wasn't one.** Pulling Authentik's 18 internal task schedules initially 404'd against the endpoint documented for older versions (`/admin/system_tasks/`) — the real one had moved to `/api/v3/tasks/schedules/` in the installed version. A wrong assumption about *why* something failed costs more time than the actual fix would have. + +## Two real, unresolved gaps this system surfaced > [!IMPORTANT] -> Building this inventory surfaced something that had been silently broken: one backup job's prune step targets a datastore name that doesn't actually exist on the backup server described in [`docs/disaster-recovery.md`](disaster-recovery.md) — not a scheduling collision, an orphaned config that's been silently failing every run. Two fixes are on the table (drop the dead job and let a different retention mechanism be authoritative, or repoint it at the real datastore name and let its own richer retention rules take over) — deliberately left unresolved here rather than silently patched over with a guess, since the right fix depends on a retention-policy decision that's genuinely still open, not a technical one that just needs picking. +> **A silently broken prune job.** PBS's own prune job targets a datastore name that doesn't actually exist on the backup server — only one real datastore name is in use elsewhere on the same server. Not a scheduling collision, an orphaned config that's been failing every run without anyone noticing. Two fixes are on the table, deliberately left unresolved rather than guessed at: drop the dead job and let Proxmox VE's own `keep-last` retention be the sole authority, or repoint it at the real datastore name and let PBS's own richer daily/weekly/monthly retention rules take over instead. Moot for now — the PVE backup jobs it would matter for are currently disabled — but worth resolving before they're ever turned back on, not after. + +> [!WARNING] +> **A config that looks active but isn't.** Technitium's blocklist auto-update is configured with a 24-hour interval — but no actual blocklist URLs are set, so it's a dormant no-op: the schedule exists, nothing downloads. Same category of gap as the PBS prune job — a setting that reads as "handled" until someone actually checks whether it does anything. --- Related: [`docs/orchestration.md`](orchestration.md), [`docs/disaster-recovery.md`](disaster-recovery.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/nocodb/`](../stacks/platform/nocodb/)