docs: add orchestration, registry, dashy, cron, and content pipeline write-ups

Completes the orchestration/registry/content batch - Kestra's
check-pause-approve-apply pattern, the self-hosted build pipeline,
the SOC dashboard's 14-source aggregation, the cron heatmap, and the
in-progress content pipeline, all in the diagram+callout+table style.
This commit is contained in:
thesabear committed 2026-08-22 07:36:47 +03:00
1 parent 2bc4ee831d
commit c10ff81d47
5 files changed
+248

No files matched your search

+41
View File
@@ -0,0 +1,41 @@
# 📝 Content Pipeline
> [!WARNING]
> Still under active development — this covers the current shape, not a finished architecture. Included deliberately rather than only showcasing finished work.
## The intended shape
```mermaid
flowchart LR
Idea["Idea"] --> Draft["Drafting"]
Draft --> Review["Review"]
Review --> Approved["Approved"]
Approved --> Published["Published"]
Approved -.glue via automation tool.-> Storage["Draft/asset staging"]
Approved -.glue via automation tool.-> Blog["Published blog"]
Approved -.planned, not yet wired.-> Social["Social platforms"]
```
A structured content tracker moves ideas through a status pipeline, with a workflow-automation tool gluing the stages together: approved content flows through to a file-storage staging area and out to a public blog. A separate, still-incomplete leg is meant to fan approved posts out to social platforms automatically.
## A deliberate constraint, not a limitation stumbled into
> [!TIP]
> The social-publishing tool is **single-origin by design** — its frontend/backend URLs are baked to one specific public hostname at configuration time, so it deliberately has no presence in either internal DNS zone described in [`docs/dns-naming.md`](dns-naming.md); any other hostname hitting it fails CORS. This was a conscious choice, not a workaround: the tool is meant to be reached exactly one way.
## A real surprise in scale
> [!IMPORTANT]
> The social-publishing tool's real deployment footprint is 8 containers, not the app+database+cache most people would guess — it bundles its own workflow engine (mandatory for all of its background jobs, including scheduled posting itself) plus that engine's own database and search-based visibility store. A useful reminder that a tool's advertised simplicity and its actual operational footprint can be very different things.
## What's actually built vs. what's still wiring up
| Piece | Status |
|---|---|
| Content tracker → automation → draft storage → blog | Most built-out, working end to end |
| AI-assisted drafting step | Still being wired up |
| Social-publishing fan-out | Still being wired up |
---
Related stacks: [`stacks/productivity/postiz/`](../stacks/productivity/postiz/), [`stacks/productivity/nocodb-content/`](../stacks/productivity/nocodb-content/), [`stacks/productivity/ghost/`](../stacks/productivity/ghost/)
+32
View File
@@ -0,0 +1,32 @@
# ⏱️ Cron Scheduler
> [!NOTE]
> A fleet-wide inventory of every known scheduled job — orchestration flows, system timers, automation-managed cron, app-internal schedules — visualized as a heatmap so a new job's time can be picked without colliding with existing load.
## The heatmap
```mermaid
flowchart LR
Sources["Job sources<br/>flows · timers · cron · app schedules"] --> Table["Structured inventory<br/>name, host, category,<br/>schedule, duration, resource tags"]
Table --> Weight["Weighted load calculation"]
Weight --> Grid["7×24 grid<br/>day × hour"]
Grid --> Bands["4 color bands<br/>quieter → busier"]
```
Each cell's color reflects a **computed load weight**, not a raw job count: a base weight per job, plus a duration-based weight, plus resource-intensity weight (disk/CPU/network, summed if a job carries multiple tags), plus an extra weight for anything tagged as a backup job. Jobs whose time can't be resolved to a specific day and hour are excluded from the grid and listed separately rather than forced into a misleading cell.
> [!TIP]
> Clicking a populated cell expands the full job list for that slot — name, host, category, and whether that job's schedule can actually be moved or is fixed by whatever app owns it.
## Honest about its own limitation
> [!WARNING]
> **This is a manually-maintained inventory, not an auto-refreshed one.** The schema has a "last synced" column, but nothing currently populates it — no automated process re-scans the fleet to keep this current. Entries get added or edited by hand as jobs are created or changed. Documented as a real limitation rather than presented as more automated than it actually is.
## A real, unresolved bug this system surfaced
> [!IMPORTANT]
> One backup job's prune step targets a datastore name that doesn't actually exist — not a scheduling collision, an orphaned config that's been silently failing. Two fixes are on the table (drop the dead job and let a different retention mechanism be authoritative, or repoint it at the real datastore name and let its richer retention rules take over) — deliberately not resolved yet, tracked openly rather than silently patched over with a guess.
---
Related stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/nocodb/`](../stacks/platform/nocodb/)
+75
View File
@@ -0,0 +1,75 @@
# 🖥️ Dashy SOC Board
> [!NOTE]
> The homelab's "one pane of glass" — a single dashboard page backed by one large custom function, aggregating roughly a dozen distinct data sources that used to be scattered across separate pages into one glanceable board.
## Fourteen data sources, one board
```mermaid
flowchart TD
subgraph Sources["Data sources"]
direction LR
Wazuh["SOC indexer<br/>+ compliance API"]
Loki["Log aggregator<br/>24h error counts"]
Prom["Metrics<br/>CPU/RAM/disk"]
Slack["Alert channels<br/>per app + per host"]
Orch["Orchestration APIs<br/>execution health"]
Audit["Fleet config-drift<br/>audit feed"]
Hyper["Hypervisor API<br/>VM/LXC tree"]
Vault["Secrets platform<br/>seal state, lease counts"]
SSO["SSO provider<br/>login/failure stats"]
DNS["DNS server<br/>query stats, blocklist"]
Backup["Backup server<br/>job age/status"]
FW["Edge firewall<br/>state table, top rules"]
end
Sources --> Function["One aggregation function"]
Function --> Board["SOC Board<br/>client-rendered"]
```
## Layout
A top row shows a 6-tier infrastructure dependency ladder alongside fleet-wide 24-hour KPIs. Below that, an "Active Issues" strip surfaces the worst findings fleet-wide, followed by a grid of per-host tiles — worst-severity badge, error sparkline, sorted consistently. Clicking a tile expands a full detail panel: compliance and resource gauges, that host's folded-in specialty widget where relevant, and findings grouped by category.
> [!TIP]
> **Deliberately scoped**: security/error/warning signal only. Per-app uptime is handled separately by the dashboard's own native status checks on individual tiles — the only up/down signal inside this board is *infrastructure-tier* reachability, not per-app availability. Keeping the scope narrow is what keeps the board actually scannable.
## Acknowledge mechanism: client-side, no backend
```mermaid
flowchart LR
Finding["Finding<br/>stable content-derived key"] --> Ack["User acknowledges"]
Ack --> Storage["Browser localStorage<br/>flat map, per-browser"]
Storage --> Recompute["Live re-derive:<br/>badges, issue strip, gauges"]
Recompute -.never touches.-> Upstream["Wazuh / Slack / any<br/>upstream source"]
```
> [!IMPORTANT]
> Acknowledging a finding is a **display filter only** — it never writes back to any upstream source, is scoped per-browser rather than fleet-wide, and a full data set for every host is always kept in memory so the recompute has everything it needs regardless of which panel happens to be open. A reload in a different browser shows the finding again — a deliberate tradeoff favoring simplicity (no backend state to manage) over persistence.
## Real integration gotchas hit building this
> [!WARNING]
> A few non-obvious lessons from actually building this, in case they save someone else the debugging time:
> - Embedding via `fetch` + `innerHTML` means injected `<script>` tags never execute — each one has to be explicitly re-created on every refresh to keep interactive elements working.
> - A background poll that could clobber an open detail panel needs to check "is a panel open" **both** before *and* after the fetch, since the aggregation endpoint itself can take tens of seconds to respond.
> - One backing API hard-caps results per page regardless of a higher requested limit — real pagination has to be implemented, not assumed away.
> - Templating engines scoped to the wrong object silently produce an empty string instead of erroring — an easy, quiet failure mode.
> - Replacing a bind-mounted config file's content doesn't get picked up by a plain container restart — the container keeps watching the old inode; a full recreate is required.
## Widget design: shape follows data, not one template everywhere
Every widget's visual style is chosen to match the *shape* of the data it shows, not a single card/badge component reused everywhere:
| Data shape | Visual treatment |
|---|---|
| Proportion of a whole (disk %, capacity) | Gauge or bar, not a badge |
| Hierarchy (host containing guests) | Real visual nesting — indentation, a tree — not a flat grid |
| Time-ordered audit/status log | A dense, sortable list ranked worst-first, recency as the dominant signal |
| Named media/content items | A poster/cover-art grid, never a text list |
> [!TIP]
> The media-poster widgets route through one shared image-proxy endpoint rather than each hitting its own upstream API directly — since these are private, authenticated homelab APIs, not public image CDNs. One exception: a public poster CDN gets hit directly, since no auth or proxying is needed there.
---
Related stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/node-red/`](../stacks/platform/node-red/)
+50
View File
@@ -0,0 +1,50 @@
# 🔁 Orchestration
> [!NOTE]
> Thirteen scheduled flows handle fleet maintenance end to end — OS updates, container updates, app updates, disk cleanup, and KB self-maintenance — built around a check → pause → human approval → apply pattern rather than blind automatic changes.
## The core pattern
```mermaid
flowchart LR
Schedule["Weekly schedule"] --> Check["Check mode run<br/>detect pending updates"]
Check --> Found{"Anything<br/>pending?"}
Found -->|no| NotifyClean["Notify: up to date"]
Found -->|yes| NotifyPending["Notify: pending"]
NotifyPending --> Pause["Pause<br/>indefinite, no timeout"]
Pause --> Decide{"Human<br/>resumes"}
Decide -->|approve| Apply["Apply mode run"]
Decide -->|decline| Skip["Skipped this cycle"]
Apply --> NotifyResult["Notify: applied / failed"]
```
Nothing gets changed on the fleet without a human explicitly resuming a paused flow — a check run can sit paused for days until someone's actually available to review it.
## Identity split, per flow
> [!IMPORTANT]
> Most flows connect using one operator identity. One flow — the one that needs to run the automation tool itself on the control node, not just reach out over SSH — deliberately logs in as a separate, narrower service identity instead. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally. Worth checking which one a given flow uses before assuming — it's not always the same answer.
## A real special case, thought through
> [!WARNING]
> One of the two hypervisor nodes also happens to host the automation VM that the orchestration engine itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification.
## Trilium KB self-maintenance
Two flows keep the internal knowledge base's own structure current — the *only* two things allowed to write to it:
| Flow | Trigger | What it does |
|---|---|---|
| KB sync | Webhook, on every push | Mirrors exact file changes from any tracked repo into the KB, upserts/deletes only what changed — never re-walks a whole repo |
| Structure index refresh | Every 6 hours | Refreshes a live crawl-based index, flags drift, tags genuinely new content as needing review |
> [!TIP]
> The push-triggered flow also runs a lightweight completeness check for the fleet's own VM-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel — a small nudge against exactly the kind of documentation drift this whole repo exists to avoid.
## Outside the orchestration engine entirely
The automation host's own OS patches and the orchestration engine's own container updates run via a standalone host-level timer, not through the orchestration engine itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run. Being outside the normal flow means these updates don't get the same visibility as everything else, worth a periodic manual check rather than fully trusting the dashboard.
---
Related stacks: [`stacks/platform/kestra/`](../stacks/platform/kestra/)
+50
View File
@@ -0,0 +1,50 @@
# 📦 Self-Hosted Registry & Build Pipeline
> [!NOTE]
> Docker images are built and hosted in-fleet rather than pulled from Docker Hub/ghcr directly — for every stack opted into this pattern, at runtime.
| | |
|---|---|
| **Registry** | The Git server's own built-in container registry — no separate registry service to run |
| **Trigger** | Weekly schedule, not push-triggered |
| **Rollout** | All 5 Docker hosts converted; the largest has 36 of 41 stacks on this pattern |
## The cheap-check-first build decision
```mermaid
flowchart TD
Start["Weekly trigger"] --> Check["Cheap digest check<br/>metadata only, no pull"]
Check --> Match{"Upstream digest ==<br/>fleet registry digest?"}
Match -->|yes| Nothing["Nothing runs<br/>no pull, no build, no push"]
Match -->|no| Pull["Pull base images"]
Pull --> Build["docker compose build --no-cache"]
Build --> Compare["Compare pre/post image IDs"]
Compare --> Push["Push what actually changed"]
Push --> Redeploy["Redeploy via shared<br/>health-check/rollback path"]
```
> [!TIP]
> Most stacks use a minimal wrapper Dockerfile around the upstream image rather than a full clone-and-build-from-source setup — simpler, builds near-instantly, can't break from upstream build tooling, while still achieving the real goal: an in-fleet registry with no direct Docker Hub/ghcr dependency at runtime. A handful of multi-build-image stacks skip the cheap check entirely, since a single "upstream digest" comparison doesn't make sense when there's more than one real upstream involved.
## Deliberate holdouts
Not every stack is on this pattern, and that's a decision, not an oversight:
| Stack type | Why it's held out |
|---|---|
| Complex multi-service upstream builds | Not worth replicating a build process someone else already maintains well |
| A docker-socket-proxy style stack | Simple enough that a custom build adds no value |
| One database service still on `latest` | Deliberately not pinned to this pattern |
| One stack with pre-existing custom Dockerfiles | Left as-is beyond adding image references — its own DB/cache still float on upstream tags directly |
## Trigger discipline
> [!WARNING]
> This is schedule-triggered, not push-triggered. Editing a stack's own Dockerfile doesn't by itself trigger a rebuild — it waits for the next weekly sweep to find the upstream base image has actually changed, or for someone to force a manual run. A deliberate tradeoff: simpler mental model, avoids a rebuild storm on every unrelated commit, at the cost of same-day propagation for an urgent base-image CVE fix.
## Housekeeping
Unused local images get pruned automatically at the end of every build pass — for build-and-push-only stacks, the registry is the source of truth, not whatever happens to be sitting on local disk.
---
Related stacks: [`stacks/platform/gitea/`](../stacks/platform/gitea/), [`stacks/platform/kestra/`](../stacks/platform/kestra/)