automation-bot
9be8e25c3b
Live NocoDB now has 49 jobs/14 without a fixed time, not 50/15 - traced to a removed inventory row for a "PBS prune job" that never actually existed (confirmed against live data and the KB note, which states PBS runs no prune job at all). Updated counts and refreshed all three screenshots to match current state.
77 lines
7.8 KiB
Markdown
77 lines
7.8 KiB
Markdown
<!-- verified: 2026-09-18
|
||
trilium : EJGewkYGStHf
|
||
-->
|
||
|
||
# ⏱️ Cron Scheduler
|
||
|
||
> [!NOTE]
|
||
> A NocoDB-backed inventory of every known scheduled job in the fleet — Kestra flows, systemd timers, Ansible-managed cron, app-internal schedules — visualized as a Dashy heatmap so a new job's time can be picked without colliding with existing load. Small in scope compared to the rest of this repo, but it's a real answer to a problem every one of the other systems here creates: once you have 50 independently-scheduled jobs across [`docs/orchestration.md`](orchestration.md), [`docs/registry-and-build.md`](registry-and-build.md), and [`docs/disaster-recovery.md`](disaster-recovery.md), "when does anything run" stops being something you can hold in your head.
|
||
|
||
## The Heatmap
|
||
|
||
A Node-RED tab fetches the inventory straight from NocoDB's REST API at request time — no cache — and renders it as a 7×24 grid (Mon–Sun × 00:00–23:00). Each cell's color is a **computed load weight**, not a raw job count: base `1` per job, plus a duration-bucket weight, plus a resource-tag weight (`disk-intensive`=2, `cpu-intensive`=1.5, `network-intensive`=0.5, summed if a job carries more than one), plus a flat `+1` if the job's category is `backup` — disk I/O contention from an overlapping backup is judged a worse collision than two lightweight jobs sharing an hour, and the weight formula says so explicitly rather than pretending every job costs the same. Weights bucket into 4 color bands (green → yellow → orange → red) relative to the current busiest cell. Jobs whose time can't be resolved to a specific day+hour are excluded from the grid and listed separately rather than forced into a misleading cell.
|
||
|
||
This is the real heatmap, not a description of it — 49 known jobs, 14 without a fixed time:
|
||
|
||
|
||
|
||
> [!TIP]
|
||
> Clicking a populated cell expands the full job list for that slot — name, host, category, and whether that job's schedule can actually be moved or is fixed by whatever app owns it. That last distinction matters in practice: there's no point suggesting a new job avoid a time slot occupied by something that can't be rescheduled anyway versus one that easily could be.
|
||
|
||
Jobs with no fixed time get their own list instead of a misleading grid cell:
|
||
|
||
|
||
|
||
Both of those are Dashy's *rendered* view. This is the raw NocoDB table underneath it — the actual data structure, not the heatmap built from it:
|
||
|
||
|
||
|
||
## Why A Manual Inventory Instead Of An Auto-discovered One
|
||
|
||
The harder, more "correct" version of this tool would auto-discover every scheduled job by querying Kestra's API, parsing every host's crontab and systemd timers, and re-syncing continuously. That was scoped down deliberately: auto-discovery across three genuinely different scheduling mechanisms (Kestra's API, systemd's timer units, cron's own format) is real engineering effort for a tool whose entire job is picking a good time slot for the next new job — a task that happens occasionally, not continuously. A manually-maintained table, kept honest by being checked at the one moment it actually matters (adding a new job), was judged the better return on effort than building and maintaining three different scanners for a problem that doesn't need real-time accuracy.
|
||
|
||
> [!WARNING]
|
||
> **This is a manually-maintained inventory, and that's stated plainly, not glossed over.** The schema has a "Last Synced" column, but nothing currently populates it on any of the 50 rows — no automated process re-scans the fleet to keep this current. Entries get added or edited by hand as jobs are created or changed, which means the tool is only as accurate as the discipline of updating it — a real limitation of the scoping decision above, not hidden behind a column that implies more automation than actually exists.
|
||
|
||
## What building the inventory actually found
|
||
|
||
Populating 50 rows by hand across a dozen-plus different systems wasn't just data entry — it surfaced real scheduling conflicts and real bad assumptions:
|
||
|
||
- **A three-way monthly collision, resolved by moving one job.** PBS's own datastore garbage-collection job used to run first-Saturday-of-month 02:00 — the same slot as the weekly `os-update-vms` Kestra flow *and* the `dashy-fleet-audit` flow, both of which also land on Saturday. Moved to first-Tuesday-of-month 10:00 instead, clearing all three out of each other's way in one change rather than shuffling the other two around it.
|
||
- **A cascading reschedule, not an isolated fix.** Kopia's backup snapshot used to run 07:00 — directly on top of Proxmox hyper2's own fixed VM-backup slot (disk-intensive, kept fixed on purpose). Moving Kopia to 08:00 then put it on top of ClamAV's scan, so ClamAV moved again to 09:00 — one collision fix triggering a second, tracked and resolved rather than left half-done.
|
||
- **Don't trust a duration number until it comes from a real completed run.** `app-update`'s first-logged duration was ~2-3 seconds — which turned out to be sampled from killed, failed, and paused runs, not a real completion. The actual number, from a genuine successful run, is ~16 minutes: an 8x difference between "what the data said" and "what actually happened," caught only by checking. This is the actual row note, not a paraphrase:
|
||
|
||
```
|
||
$ curl -sk -H "xc-token: [REDACTED]" "https://nocodb.[app-to-app zone]/api/v2/tables/<table-id>/records?limit=100" \
|
||
| python3 -c "
|
||
import json,sys
|
||
d = json.load(sys.stdin)
|
||
for r in d['list']:
|
||
if r.get('Job Name') == 'app-update (Kestra)':
|
||
print(r.get('Notes'))
|
||
"
|
||
Kestra flow homelab/app-update. Checks non-Docker app updates + ansible
|
||
venv. 1h after os-update-vms same night. Duration corrected 2026-07-31:
|
||
real successful completion took ~16min - the earlier ~2-3s figure came
|
||
from KILLED/FAILED/PAUSED runs, not a real completion.
|
||
```
|
||
- **The heaviest job in the entire fleet isn't a backup.** Radarr's own `Refresh Movie` task runs ~1h15min daily — heavier than any single PVE backup job, and the heaviest scheduled task found anywhere in the *arr media suite. Radarr's own backup job, by contrast, finishes in about 41 seconds.
|
||
- **A 404 that looked like an auth problem wasn't one.** Pulling Authentik's 18 internal task schedules initially 404'd against the endpoint documented for older versions (`/admin/system_tasks/`) — the real one had moved to `/api/v3/tasks/schedules/` in the installed version. A wrong assumption about *why* something failed costs more time than the actual fix would have. Same live query, same real note:
|
||
|
||
```
|
||
$ curl -sk -H "xc-token: [REDACTED]" "https://nocodb.[app-to-app zone]/api/v2/tables/<table-id>/records?limit=100" \
|
||
| python3 -c "
|
||
import json,sys
|
||
d = json.load(sys.stdin)
|
||
for r in d['list']:
|
||
if 'Authentik' in (r.get('Job Name') or ''):
|
||
print(r.get('Notes'))
|
||
"
|
||
Resolved via GET /api/v3/tasks/schedules/ 2026-07-29 (endpoint moved
|
||
from the old /admin/system_tasks/ path in this Authentik version -
|
||
that's why the first attempt 404'd, not an auth problem). [...]
|
||
```
|
||
|
||
---
|
||
Related: [`docs/orchestration.md`](orchestration.md), [`docs/disaster-recovery.md`](disaster-recovery.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/nocodb/`](../stacks/platform/nocodb/)
|