docs: add homelab-inventory — the machine-derived fleet inventory layer
New doc for the weekly self-deriving inventory pipeline: a three-tier Ansible collection (hardware off the hypervisors/firewall, VMs/LXCs from the Proxmox APIs, listening sockets off every guest) writing three linked NocoDB tables, read live behind a Dashy page. Covers the design decisions worth defending: ss -tlnp over systemctl for "what does this host serve", merge-only-by-omission on Tier 1 to protect the one hand-entered field a lying SMBIOS can't provide, API-only Tier 2 because the two Proxmox nodes aren't clustered, stdlib-only push scripts, and the deliberate KB-vs-machine-layer separation. Pipeline diagram + four screenshots (Dashy page, all three NocoDB tables). Provenance comment wired to the ansible roles/playbooks, the rocky Caddy/ dashy config, and KB note O0d4fqw0mr05. Indexed in README under Systems.
This commit is contained in:
7 files changed
+125
No files matched your search
@@ -85,6 +85,10 @@ Two internal DNS zones split browser-facing traffic from service-to-service traf
|
||||
No blanket internal trust anywhere on the network — every real service-to-service flow gets its own explicit rule, and everything not explicitly allowed is logged and dropped by default. Enforced at two layers: a firewall at the network edge, and firewalld locally on every host, each opening only what that specific host actually needs. A few rules go narrower still — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet.
|
||||
→ [docs/firewall-and-network.md](docs/firewall-and-network.md)
|
||||
|
||||
### 📇 Homelab Inventory
|
||||
Every physical host, VM/LXC, and running service in the fleet, re-derived from the machines themselves once a week rather than hand-maintained — so the reference is structurally unable to drift out of sync with reality. A three-tier Ansible collection (hardware off the hypervisors, guests from the Proxmox APIs, listening sockets off every host) writes three linked tables, surfaced on one dashboard page. The one hand-held detail — a NAS that reports a placeholder string for its own model — is protected by a merge-only write that structurally can't overwrite it.
|
||||
→ [docs/homelab-inventory.md](docs/homelab-inventory.md)
|
||||
|
||||
### 🏗️ Infrastructure as Code
|
||||
Packer builds a hardened base image, Terraform provisions each VM from it, and Ansible configures and continuously re-hardens every host after that — three tools, three separate responsibilities, connected by one tag that threads through all three stages instead of one tool trying to do every job. Terraform's part in that link ends at creation — it writes the tags file once, and that file then drives Ansible independently for the rest of that host's life.
|
||||
→ [docs/iac-pipeline.md](docs/iac-pipeline.md)
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 134 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 99 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 227 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 181 KiB |
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Do not edit this file with editors other than draw.io -->
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1251px" height="486px" viewBox="0 0 1251 486" content="<mxfile host="embed.diagrams.net" scale="1" border="0">
 <diagram name="homelab-inventory-pipeline" id="hip1">
 <mxGraphModel dx="874" dy="519" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1320" pageHeight="560" math="0" shadow="0">
 <root>
 <mxCell id="0" />
 <mxCell id="1" parent="0" />
 <mxCell id="title" parent="1" style="text;html=1;align=left;verticalAlign=middle;fontSize=15;fontStyle=1;fontColor=#333333;" value="Homelab Inventory Pipeline" vertex="1">
 <mxGeometry height="26" width="360" x="40" y="20" as="geometry" />
 </mxCell>
 <mxCell id="kestra" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=11;fontColor=#333333;" value="Kestra&#xa;homelab-inventory-sync&#xa;weekly · Thu 12:00" vertex="1">
 <mxGeometry height="64" width="200" x="60" y="60" as="geometry" />
 </mxCell>
 <mxCell id="t1" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=11;fontColor=#333333;" value="Tier 1 · Hardware&lt;br&gt;node1 / node2 + pfSense" vertex="1">
 <mxGeometry height="56" width="200" x="60" y="190" as="geometry" />
 </mxCell>
 <mxCell id="t2" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=11;fontColor=#333333;" value="Tier 2 · VMs / LXCs&#xa;Proxmox API — no SSH" vertex="1">
 <mxGeometry height="56" width="200" x="60" y="286" as="geometry" />
 </mxCell>
 <mxCell id="t3" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=11;fontColor=#333333;" value="Tier 3 · Services&#xa;ss -tlnp + docker ps" vertex="1">
 <mxGeometry height="56" width="200" x="60" y="382" as="geometry" />
 </mxCell>
 <mxCell id="gate" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#f8cecc;strokeColor=#b85450;fontSize=10;fontColor=#333333;dashed=1;" value="pfSense Manufacturer / Model — entered by hand.&#xa;Tier 1 never sends these keys, so the weekly&#xa;push cannot overwrite them." vertex="1">
 <mxGeometry height="66" width="360" x="420" y="50" as="geometry" />
 </mxCell>
 <mxCell id="noco" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#F5F5F5;strokeColor=#666666;fontSize=11;fontStyle=1;fontColor=#333333;verticalAlign=top;spacingTop=6;" value="NocoDB — 3 linked tables" vertex="1">
 <mxGeometry height="300" width="290" x="420" y="150" as="geometry" />
 </mxCell>
 <mxCell id="hw" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=11;fontColor=#333333;" value="Hardware Inventory" vertex="1">
 <mxGeometry height="48" width="230" x="450" y="190" as="geometry" />
 </mxCell>
 <mxCell id="vm" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontSize=11;fontColor=#333333;" value="VM / LXC Inventory" vertex="1">
 <mxGeometry height="48" width="230" x="450" y="286" as="geometry" />
 </mxCell>
 <mxCell id="svc" parent="1&Line truncated
|
||||
|
After Width: | Height: | Size: 369 KiB |
@@ -0,0 +1,117 @@
|
||||
<!-- verified: 2026-09-07
|
||||
ansible@9a4ba6d : roles/homelab_inventory_hardware, roles/homelab_inventory_vms, roles/homelab_inventory_services, playbooks/homelab_inventory_hardware.yml, playbooks/homelab_inventory_vms.yml, playbooks/homelab_inventory_services.yml
|
||||
rocky-docker-lxc@145d3df : caddy/Caddyfile, dashy/docker-compose.yaml
|
||||
trilium : O0d4fqw0mr05
|
||||
-->
|
||||
|
||||
# 📇 Homelab Inventory
|
||||
|
||||
> [!NOTE]
|
||||
> A live reference for every physical host, VM/LXC, and running service in the fleet — **re-derived from the machines themselves once a week**, not hand-maintained. The problem it solves is inventory rot: a wiki table of "what hardware do I have, what's running where" is accurate the day it's written and drifting by the next change. This one is structurally unable to drift, because nothing attests it — a scheduled job re-reads the real systems and overwrites the store.
|
||||
|
||||
The knowledge base (Trilium) stays the hand-authored layer — design rationale, gotchas, why things are the way they are. This is the **machine-derived layer** underneath it: three linked tables in a self-hosted NocoDB instance, refreshed by a three-tier Ansible collection on a weekly schedule, surfaced on a dedicated dashboard page.
|
||||
|
||||
|
||||
|
||||
## Three Collection Tiers, Three Different Access Models
|
||||
|
||||
The three tiers don't share a collection method, because the three things being inventoried don't live in the same place or answer to the same protocol.
|
||||
|
||||
| Tier | Where it runs | How it collects | Writes | Row write mode |
|
||||
|---|---|---|---|---|
|
||||
| **Hardware** | hypervisor + firewall hosts, as root | `dmidecode` / `ip -j link` / `/proc` on the Linux hosts; `kenv \| grep -i smbios` + `ifconfig -a` over the `raw` module on the firewall (no Python on that box) | Hardware Inventory | **Merge-only** — only collected fields are sent |
|
||||
| **VMs / LXCs** | control node only — no SSH anywhere | each hypervisor node's own Proxmox API (`/qemu` + `/lxc`); guest IP from the LXC `net0` config or the VM's QEMU guest agent | VM/LXC Inventory, linked → Hardware | Full field-set replace every run |
|
||||
| **Services** | every running guest, as root | `ss -tlnp` for listening sockets, plus `docker ps` on the Docker hosts | Services Inventory, linked → VM/LXC | Full field-set replace every run |
|
||||
|
||||
A few of those choices are worth pulling out, because the obvious approach was rejected in each case.
|
||||
|
||||
### Why `ss -tlnp` and not `systemctl`
|
||||
|
||||
"What services does this host run" sounds like a `systemctl list-units` question. It isn't — that answer is 40–80 units of `dbus`, `journald`, udev timers and mount units per host, none of which is a *service the host provides*. Listening sockets are a far closer match to the real question, so that's what the tier collects:
|
||||
|
||||
```
|
||||
$ sed -n '2,6p' roles/homelab_inventory_services/tasks/main.yml
|
||||
# Deliberately NOT collecting `systemctl list-units --state=running` here -
|
||||
# that dumps 40-80 units per host (dbus, journald, udev timers, ...) that
|
||||
# have nothing to do with "what service is this host providing." `ss -tlnp`
|
||||
# only shows things with an actual open listening socket, which is a much
|
||||
# closer match to what "service" means for a reference doc - see card [77].
|
||||
```
|
||||
|
||||
### Merge-only vs full replace: protecting the one field a machine can't collect
|
||||
|
||||
Tiers 2 and 3 overwrite every column on a row every run — every value there is meant to always reflect current state, and there's nothing to protect. Tier 1 is different, and the difference exists for exactly one reason: the firewall appliance is a QNAP NAS whose BIOS reports the literal string `Default string` for manufacturer, model, and serial, even with full root access. Those three fields are entered by hand in NocoDB once (QNAP / TS-251A / left blank) — and the collector has to be structurally incapable of clobbering them on the next run.
|
||||
|
||||
The guarantee isn't "write a blank" or "skip if unchanged" — it's that a field the collector has no data for is **never included in the API payload at all**:
|
||||
|
||||
```
|
||||
$ grep -A3 'def normalize' roles/homelab_inventory_hardware/files/push_hardware_to_nocodb.py
|
||||
def normalize(record):
|
||||
"""Build the NocoDB field dict - only include keys we actually have data
|
||||
for. This is the merge-only guarantee: a key not added here is a key
|
||||
never sent in the API call, so it can never overwrite anything."""
|
||||
```
|
||||
|
||||
The firewall's collection path deliberately never sets `Manufacturer`/`Model`/`Serial` keys; the Linux path sets them from `dmidecode`. Same table, two code paths, one protected by omission. Here's the result — the CPU and RAM on the QNAP row *are* machine-collected (via `kenv`), only the three identity fields are hand-held:
|
||||
|
||||
|
||||
|
||||
### Tier 2 is pure API-to-API
|
||||
|
||||
The VM/LXC tier touches no host over SSH — it runs entirely on the control node and pulls from each hypervisor node's own Proxmox API with one dedicated read-only `PVEAuditor` token. The two nodes aren't clustered, so each API only ever returns its own guests; the tier queries both and merges. Guest IPs come from two different places depending on type — an LXC's is static in its `net0` config, a VM's has to come from the QEMU guest agent — and a stopped VM or one with no agent simply gets no IP recorded, not an error. Build templates are skipped.
|
||||
|
||||
## The Linked Store
|
||||
|
||||
Three NocoDB tables, related **Hardware ← VM/LXC ← Services**: each guest row links to its host's hardware row, each service row to its guest's VM/LXC row, set through NocoDB's Links API after each upsert. Service rows are named `<host>: <service>` — native names collide across hosts (every host has an `sshd`), and the prefix is what keeps two rows distinct and lets the viewer derive a service's host straight from its name.
|
||||
|
||||
Live counts, straight off the API — matching the screenshots below:
|
||||
|
||||
```
|
||||
$ # GET /api/v2/tables/<id>/records?limit=1 → .pageInfo.totalRows
|
||||
Hardware : 3 rows
|
||||
VM/LXC : 12 rows
|
||||
Services : 111 rows
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
The push scripts use nothing but the Python standard library — they run under the control node's system `python3`, invoked straight from Ansible's `command` module, not a virtualenv with `requests` available:
|
||||
|
||||
```
|
||||
$ grep -hE '^import |^from ' roles/homelab_inventory_*/files/*.py | sort -u
|
||||
from datetime import datetime, timezone
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
```
|
||||
|
||||
## The Viewer
|
||||
|
||||
A dedicated Dashy page, **Homelab Inventory** — three collapsible sections (Hardware, VMs + LXCs, Services), each backed by its own Node-RED flow on the automation host. Each flow is a single `http-in → function → http-response` that reads its NocoDB table **live on every request with no caching** and renders a searchable click-to-expand list. Past ~100 rows a flat scroll is unusable, so the Services section carries a live text filter. Each flow gets its own unauthenticated path exception in the reverse-proxy config, the same pattern every Dashy widget on this fleet uses.
|
||||
|
||||
|
||||
|
||||
## Scheduled, Not Triggered
|
||||
|
||||
A single Kestra flow, `homelab-inventory-sync`, cron `0 12 * * 4` — Thursday noon. There's no webhook: nothing about live hardware or a running-process list is an event the orchestrator would otherwise see, so a fixed weekly cadence is the whole trigger.
|
||||
|
||||
```
|
||||
- id: schedule
|
||||
type: io.kestra.plugin.core.trigger.Schedule
|
||||
cron: "0 12 * * 4"
|
||||
```
|
||||
|
||||
One SSH task hops to the automation host and runs the three playbooks in sequence through the pinned virtualenv, capturing a per-tier exit code. A non-zero tier sends a critical Slack alert through the shared [`notifications`](notifications.md) pipeline; all-clear sends an info line. The dedicated-key SSH hop and the check→notify shape are the same model every flow in [`orchestration.md`](orchestration.md) uses.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **This layer never writes to the knowledge base, and the knowledge base never sources from it.** They're deliberately separate: the KB is hand-authored design intent, this is a machine-derived snapshot of current reality, and wiring either one to feed the other would collapse that distinction. A separate mechanism does push a handful of per-host CPU/RAM figures into the KB for at-a-glance context, but it's a different job with its own source and its own destination — it doesn't read or write anything here.
|
||||
|
||||
---
|
||||
Related: [`docs/orchestration.md`](orchestration.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md), [`docs/cron-scheduler.md`](cron-scheduler.md), [`docs/notifications.md`](notifications.md) · Stacks: [`stacks/platform/nocodb/`](../stacks/platform/nocodb/), [`stacks/platform/node-red/`](../stacks/platform/node-red/), [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/kestra/`](../stacks/platform/kestra/)
|
||||
Reference in new issue
Block a user