automation-bot
e5b6e8a77d
New doc for the weekly self-deriving inventory pipeline: a three-tier Ansible collection (hardware off the hypervisors/firewall, VMs/LXCs from the Proxmox APIs, listening sockets off every guest) writing three linked NocoDB tables, read live behind a Dashy page. Covers the design decisions worth defending: ss -tlnp over systemctl for "what does this host serve", merge-only-by-omission on Tier 1 to protect the one hand-entered field a lying SMBIOS can't provide, API-only Tier 2 because the two Proxmox nodes aren't clustered, stdlib-only push scripts, and the deliberate KB-vs-machine-layer separation. Pipeline diagram + four screenshots (Dashy page, all three NocoDB tables). Provenance comment wired to the ansible roles/playbooks, the rocky Caddy/ dashy config, and KB note O0d4fqw0mr05. Indexed in README under Systems.
118 lines
10 KiB
Markdown
118 lines
10 KiB
Markdown
<!-- verified: 2026-09-07
|
||
ansible@9a4ba6d : roles/homelab_inventory_hardware, roles/homelab_inventory_vms, roles/homelab_inventory_services, playbooks/homelab_inventory_hardware.yml, playbooks/homelab_inventory_vms.yml, playbooks/homelab_inventory_services.yml
|
||
rocky-docker-lxc@145d3df : caddy/Caddyfile, dashy/docker-compose.yaml
|
||
trilium : O0d4fqw0mr05
|
||
-->
|
||
|
||
# 📇 Homelab Inventory
|
||
|
||
> [!NOTE]
|
||
> A live reference for every physical host, VM/LXC, and running service in the fleet — **re-derived from the machines themselves once a week**, not hand-maintained. The problem it solves is inventory rot: a wiki table of "what hardware do I have, what's running where" is accurate the day it's written and drifting by the next change. This one is structurally unable to drift, because nothing attests it — a scheduled job re-reads the real systems and overwrites the store.
|
||
|
||
The knowledge base (Trilium) stays the hand-authored layer — design rationale, gotchas, why things are the way they are. This is the **machine-derived layer** underneath it: three linked tables in a self-hosted NocoDB instance, refreshed by a three-tier Ansible collection on a weekly schedule, surfaced on a dedicated dashboard page.
|
||
|
||
|
||
|
||
## Three Collection Tiers, Three Different Access Models
|
||
|
||
The three tiers don't share a collection method, because the three things being inventoried don't live in the same place or answer to the same protocol.
|
||
|
||
| Tier | Where it runs | How it collects | Writes | Row write mode |
|
||
|---|---|---|---|---|
|
||
| **Hardware** | hypervisor + firewall hosts, as root | `dmidecode` / `ip -j link` / `/proc` on the Linux hosts; `kenv \| grep -i smbios` + `ifconfig -a` over the `raw` module on the firewall (no Python on that box) | Hardware Inventory | **Merge-only** — only collected fields are sent |
|
||
| **VMs / LXCs** | control node only — no SSH anywhere | each hypervisor node's own Proxmox API (`/qemu` + `/lxc`); guest IP from the LXC `net0` config or the VM's QEMU guest agent | VM/LXC Inventory, linked → Hardware | Full field-set replace every run |
|
||
| **Services** | every running guest, as root | `ss -tlnp` for listening sockets, plus `docker ps` on the Docker hosts | Services Inventory, linked → VM/LXC | Full field-set replace every run |
|
||
|
||
A few of those choices are worth pulling out, because the obvious approach was rejected in each case.
|
||
|
||
### Why `ss -tlnp` and not `systemctl`
|
||
|
||
"What services does this host run" sounds like a `systemctl list-units` question. It isn't — that answer is 40–80 units of `dbus`, `journald`, udev timers and mount units per host, none of which is a *service the host provides*. Listening sockets are a far closer match to the real question, so that's what the tier collects:
|
||
|
||
```
|
||
$ sed -n '2,6p' roles/homelab_inventory_services/tasks/main.yml
|
||
# Deliberately NOT collecting `systemctl list-units --state=running` here -
|
||
# that dumps 40-80 units per host (dbus, journald, udev timers, ...) that
|
||
# have nothing to do with "what service is this host providing." `ss -tlnp`
|
||
# only shows things with an actual open listening socket, which is a much
|
||
# closer match to what "service" means for a reference doc - see card [77].
|
||
```
|
||
|
||
### Merge-only vs full replace: protecting the one field a machine can't collect
|
||
|
||
Tiers 2 and 3 overwrite every column on a row every run — every value there is meant to always reflect current state, and there's nothing to protect. Tier 1 is different, and the difference exists for exactly one reason: the firewall appliance is a QNAP NAS whose BIOS reports the literal string `Default string` for manufacturer, model, and serial, even with full root access. Those three fields are entered by hand in NocoDB once (QNAP / TS-251A / left blank) — and the collector has to be structurally incapable of clobbering them on the next run.
|
||
|
||
The guarantee isn't "write a blank" or "skip if unchanged" — it's that a field the collector has no data for is **never included in the API payload at all**:
|
||
|
||
```
|
||
$ grep -A3 'def normalize' roles/homelab_inventory_hardware/files/push_hardware_to_nocodb.py
|
||
def normalize(record):
|
||
"""Build the NocoDB field dict - only include keys we actually have data
|
||
for. This is the merge-only guarantee: a key not added here is a key
|
||
never sent in the API call, so it can never overwrite anything."""
|
||
```
|
||
|
||
The firewall's collection path deliberately never sets `Manufacturer`/`Model`/`Serial` keys; the Linux path sets them from `dmidecode`. Same table, two code paths, one protected by omission. Here's the result — the CPU and RAM on the QNAP row *are* machine-collected (via `kenv`), only the three identity fields are hand-held:
|
||
|
||
|
||
|
||
### Tier 2 is pure API-to-API
|
||
|
||
The VM/LXC tier touches no host over SSH — it runs entirely on the control node and pulls from each hypervisor node's own Proxmox API with one dedicated read-only `PVEAuditor` token. The two nodes aren't clustered, so each API only ever returns its own guests; the tier queries both and merges. Guest IPs come from two different places depending on type — an LXC's is static in its `net0` config, a VM's has to come from the QEMU guest agent — and a stopped VM or one with no agent simply gets no IP recorded, not an error. Build templates are skipped.
|
||
|
||
## The Linked Store
|
||
|
||
Three NocoDB tables, related **Hardware ← VM/LXC ← Services**: each guest row links to its host's hardware row, each service row to its guest's VM/LXC row, set through NocoDB's Links API after each upsert. Service rows are named `<host>: <service>` — native names collide across hosts (every host has an `sshd`), and the prefix is what keeps two rows distinct and lets the viewer derive a service's host straight from its name.
|
||
|
||
Live counts, straight off the API — matching the screenshots below:
|
||
|
||
```
|
||
$ # GET /api/v2/tables/<id>/records?limit=1 → .pageInfo.totalRows
|
||
Hardware : 3 rows
|
||
VM/LXC : 12 rows
|
||
Services : 111 rows
|
||
```
|
||
|
||
|
||
|
||
|
||
|
||
The push scripts use nothing but the Python standard library — they run under the control node's system `python3`, invoked straight from Ansible's `command` module, not a virtualenv with `requests` available:
|
||
|
||
```
|
||
$ grep -hE '^import |^from ' roles/homelab_inventory_*/files/*.py | sort -u
|
||
from datetime import datetime, timezone
|
||
import glob
|
||
import json
|
||
import os
|
||
import re
|
||
import sys
|
||
import urllib.error
|
||
import urllib.parse
|
||
import urllib.request
|
||
```
|
||
|
||
## The Viewer
|
||
|
||
A dedicated Dashy page, **Homelab Inventory** — three collapsible sections (Hardware, VMs + LXCs, Services), each backed by its own Node-RED flow on the automation host. Each flow is a single `http-in → function → http-response` that reads its NocoDB table **live on every request with no caching** and renders a searchable click-to-expand list. Past ~100 rows a flat scroll is unusable, so the Services section carries a live text filter. Each flow gets its own unauthenticated path exception in the reverse-proxy config, the same pattern every Dashy widget on this fleet uses.
|
||
|
||
|
||
|
||
## Scheduled, Not Triggered
|
||
|
||
A single Kestra flow, `homelab-inventory-sync`, cron `0 12 * * 4` — Thursday noon. There's no webhook: nothing about live hardware or a running-process list is an event the orchestrator would otherwise see, so a fixed weekly cadence is the whole trigger.
|
||
|
||
```
|
||
- id: schedule
|
||
type: io.kestra.plugin.core.trigger.Schedule
|
||
cron: "0 12 * * 4"
|
||
```
|
||
|
||
One SSH task hops to the automation host and runs the three playbooks in sequence through the pinned virtualenv, capturing a per-tier exit code. A non-zero tier sends a critical Slack alert through the shared [`notifications`](notifications.md) pipeline; all-clear sends an info line. The dedicated-key SSH hop and the check→notify shape are the same model every flow in [`orchestration.md`](orchestration.md) uses.
|
||
|
||
> [!IMPORTANT]
|
||
> **This layer never writes to the knowledge base, and the knowledge base never sources from it.** They're deliberately separate: the KB is hand-authored design intent, this is a machine-derived snapshot of current reality, and wiring either one to feed the other would collapse that distinction. A separate mechanism does push a handful of per-host CPU/RAM figures into the KB for at-a-glance context, but it's a different job with its own source and its own destination — it doesn't read or write anything here.
|
||
|
||
---
|
||
Related: [`docs/orchestration.md`](orchestration.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md), [`docs/cron-scheduler.md`](cron-scheduler.md), [`docs/notifications.md`](notifications.md) · Stacks: [`stacks/platform/nocodb/`](../stacks/platform/nocodb/), [`stacks/platform/node-red/`](../stacks/platform/node-red/), [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/kestra/`](../stacks/platform/kestra/)
|