diff --git a/README.md b/README.md index 8238af5..6346996 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,10 @@ Two internal DNS zones split browser-facing traffic from service-to-service traf No blanket internal trust anywhere on the network β€” every real service-to-service flow gets its own explicit rule, and everything not explicitly allowed is logged and dropped by default. Enforced at two layers: a firewall at the network edge, and firewalld locally on every host, each opening only what that specific host actually needs. A few rules go narrower still β€” inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet. β†’ [docs/firewall-and-network.md](docs/firewall-and-network.md) +### πŸ“‡ Homelab Inventory +Every physical host, VM/LXC, and running service in the fleet, re-derived from the machines themselves once a week rather than hand-maintained β€” so the reference is structurally unable to drift out of sync with reality. A three-tier Ansible collection (hardware off the hypervisors, guests from the Proxmox APIs, listening sockets off every host) writes three linked tables, surfaced on one dashboard page. The one hand-held detail β€” a NAS that reports a placeholder string for its own model β€” is protected by a merge-only write that structurally can't overwrite it. +β†’ [docs/homelab-inventory.md](docs/homelab-inventory.md) + ### πŸ—οΈ Infrastructure as Code Packer builds a hardened base image, Terraform provisions each VM from it, and Ansible configures and continuously re-hardens every host after that β€” three tools, three separate responsibilities, connected by one tag that threads through all three stages instead of one tool trying to do every job. Terraform's part in that link ends at creation β€” it writes the tags file once, and that file then drives Ansible independently for the rest of that host's life. β†’ [docs/iac-pipeline.md](docs/iac-pipeline.md) diff --git a/assets/homelab-inventory-dashy-page.png b/assets/homelab-inventory-dashy-page.png new file mode 100644 index 0000000..b844eed Binary files /dev/null and b/assets/homelab-inventory-dashy-page.png differ diff --git a/assets/homelab-inventory-nocodb-hardware.png b/assets/homelab-inventory-nocodb-hardware.png new file mode 100644 index 0000000..2c690a7 Binary files /dev/null and b/assets/homelab-inventory-nocodb-hardware.png differ diff --git a/assets/homelab-inventory-nocodb-services.png b/assets/homelab-inventory-nocodb-services.png new file mode 100644 index 0000000..7202942 Binary files /dev/null and b/assets/homelab-inventory-nocodb-services.png differ diff --git a/assets/homelab-inventory-nocodb-vms.png b/assets/homelab-inventory-nocodb-vms.png new file mode 100644 index 0000000..e10397a Binary files /dev/null and b/assets/homelab-inventory-nocodb-vms.png differ diff --git a/assets/homelab-inventory-pipeline.svg b/assets/homelab-inventory-pipeline.svg new file mode 100644 index 0000000..c3f76c1 --- /dev/null +++ b/assets/homelab-inventory-pipeline.svg @@ -0,0 +1,4 @@ + + + +
Homelab Inventory Pipeline
Kestra
homelab-inventory-sync
weekly Β· Thu 12:00
Tier 1 Β· Hardware
node1 / node2 + pfSense
Tier 2 Β· VMs / LXCs
Proxmox API β€” no SSH
Tier 3 Β· Services
ss -tlnp + docker ps
pfSense Manufacturer / Model β€” entered by hand.
Tier 1 never sends these keys, so the weekly
push cannot overwrite them.
NocoDB β€” 3 linked tables
Hardware Inventory
VM / LXC Inventory
Services Inventory
Node-RED β€” 3 widgets
live read per request, no cache
Dashy page
"Homelab Inventory"
3 sections
runs the 3 playbooks
merge-only
full replace
Links API
Legend
scheduled trigger
write to NocoDB
table link (Links API)
live render
full replace
\ No newline at end of file diff --git a/docs/homelab-inventory.md b/docs/homelab-inventory.md new file mode 100644 index 0000000..6896f1c --- /dev/null +++ b/docs/homelab-inventory.md @@ -0,0 +1,117 @@ + + +# πŸ“‡ Homelab Inventory + +> [!NOTE] +> A live reference for every physical host, VM/LXC, and running service in the fleet β€” **re-derived from the machines themselves once a week**, not hand-maintained. The problem it solves is inventory rot: a wiki table of "what hardware do I have, what's running where" is accurate the day it's written and drifting by the next change. This one is structurally unable to drift, because nothing attests it β€” a scheduled job re-reads the real systems and overwrites the store. + +The knowledge base (Trilium) stays the hand-authored layer β€” design rationale, gotchas, why things are the way they are. This is the **machine-derived layer** underneath it: three linked tables in a self-hosted NocoDB instance, refreshed by a three-tier Ansible collection on a weekly schedule, surfaced on a dedicated dashboard page. + +![The collection pipeline: a weekly Kestra trigger runs three Ansible tiers in sequence β€” hardware off the hypervisors and firewall, VMs/LXCs from the Proxmox APIs, listening services off every running guest β€” each writing its own NocoDB table, chained by NocoDB link fields, and read live per request by three Node-RED flows behind one Dashy page](../assets/homelab-inventory-pipeline.svg) + +## Three Collection Tiers, Three Different Access Models + +The three tiers don't share a collection method, because the three things being inventoried don't live in the same place or answer to the same protocol. + +| Tier | Where it runs | How it collects | Writes | Row write mode | +|---|---|---|---|---| +| **Hardware** | hypervisor + firewall hosts, as root | `dmidecode` / `ip -j link` / `/proc` on the Linux hosts; `kenv \| grep -i smbios` + `ifconfig -a` over the `raw` module on the firewall (no Python on that box) | Hardware Inventory | **Merge-only** β€” only collected fields are sent | +| **VMs / LXCs** | control node only β€” no SSH anywhere | each hypervisor node's own Proxmox API (`/qemu` + `/lxc`); guest IP from the LXC `net0` config or the VM's QEMU guest agent | VM/LXC Inventory, linked β†’ Hardware | Full field-set replace every run | +| **Services** | every running guest, as root | `ss -tlnp` for listening sockets, plus `docker ps` on the Docker hosts | Services Inventory, linked β†’ VM/LXC | Full field-set replace every run | + +A few of those choices are worth pulling out, because the obvious approach was rejected in each case. + +### Why `ss -tlnp` and not `systemctl` + +"What services does this host run" sounds like a `systemctl list-units` question. It isn't β€” that answer is 40–80 units of `dbus`, `journald`, udev timers and mount units per host, none of which is a *service the host provides*. Listening sockets are a far closer match to the real question, so that's what the tier collects: + +``` +$ sed -n '2,6p' roles/homelab_inventory_services/tasks/main.yml +# Deliberately NOT collecting `systemctl list-units --state=running` here - +# that dumps 40-80 units per host (dbus, journald, udev timers, ...) that +# have nothing to do with "what service is this host providing." `ss -tlnp` +# only shows things with an actual open listening socket, which is a much +# closer match to what "service" means for a reference doc - see card [77]. +``` + +### Merge-only vs full replace: protecting the one field a machine can't collect + +Tiers 2 and 3 overwrite every column on a row every run β€” every value there is meant to always reflect current state, and there's nothing to protect. Tier 1 is different, and the difference exists for exactly one reason: the firewall appliance is a QNAP NAS whose BIOS reports the literal string `Default string` for manufacturer, model, and serial, even with full root access. Those three fields are entered by hand in NocoDB once (QNAP / TS-251A / left blank) β€” and the collector has to be structurally incapable of clobbering them on the next run. + +The guarantee isn't "write a blank" or "skip if unchanged" β€” it's that a field the collector has no data for is **never included in the API payload at all**: + +``` +$ grep -A3 'def normalize' roles/homelab_inventory_hardware/files/push_hardware_to_nocodb.py +def normalize(record): + """Build the NocoDB field dict - only include keys we actually have data + for. This is the merge-only guarantee: a key not added here is a key + never sent in the API call, so it can never overwrite anything.""" +``` + +The firewall's collection path deliberately never sets `Manufacturer`/`Model`/`Serial` keys; the Linux path sets them from `dmidecode`. Same table, two code paths, one protected by omission. Here's the result β€” the CPU and RAM on the QNAP row *are* machine-collected (via `kenv`), only the three identity fields are hand-held: + +![The NocoDB Hardware Inventory table: three physical hosts. The two Intel NUCs carry machine-read manufacturer/model/serial/CPU/RAM; the QNAP row has hand-entered manufacturer and model, a blank serial, but machine-collected CPU and RAM, plus a link out to its guests](../assets/homelab-inventory-nocodb-hardware.png) + +### Tier 2 is pure API-to-API + +The VM/LXC tier touches no host over SSH β€” it runs entirely on the control node and pulls from each hypervisor node's own Proxmox API with one dedicated read-only `PVEAuditor` token. The two nodes aren't clustered, so each API only ever returns its own guests; the tier queries both and merges. Guest IPs come from two different places depending on type β€” an LXC's is static in its `net0` config, a VM's has to come from the QEMU guest agent β€” and a stopped VM or one with no agent simply gets no IP recorded, not an error. Build templates are skipped. + +## The Linked Store + +Three NocoDB tables, related **Hardware ← VM/LXC ← Services**: each guest row links to its host's hardware row, each service row to its guest's VM/LXC row, set through NocoDB's Links API after each upsert. Service rows are named `: ` β€” native names collide across hosts (every host has an `sshd`), and the prefix is what keeps two rows distinct and lets the viewer derive a service's host straight from its name. + +Live counts, straight off the API β€” matching the screenshots below: + +``` +$ # GET /api/v2/tables//records?limit=1 β†’ .pageInfo.totalRows +Hardware : 3 rows +VM/LXC : 12 rows +Services : 111 rows +``` + +![The NocoDB VM/LXC Inventory table: twelve guest rows, VM and LXC types mixed, each with VMID, status, RAM/CPU/disk, a link back to its Proxmox host and a link forward to its services](../assets/homelab-inventory-nocodb-vms.png) + +![The NocoDB Services Inventory table: 111 rows. Socket-derived rows show status "listening" with no image; container-derived rows show status "running" with an image/version β€” the `ss` and `docker ps` sources stay distinguishable in the data. Every row links to exactly one VM/LXC](../assets/homelab-inventory-nocodb-services.png) + +The push scripts use nothing but the Python standard library β€” they run under the control node's system `python3`, invoked straight from Ansible's `command` module, not a virtualenv with `requests` available: + +``` +$ grep -hE '^import |^from ' roles/homelab_inventory_*/files/*.py | sort -u +from datetime import datetime, timezone +import glob +import json +import os +import re +import sys +import urllib.error +import urllib.parse +import urllib.request +``` + +## The Viewer + +A dedicated Dashy page, **Homelab Inventory** β€” three collapsible sections (Hardware, VMs + LXCs, Services), each backed by its own Node-RED flow on the automation host. Each flow is a single `http-in β†’ function β†’ http-response` that reads its NocoDB table **live on every request with no caching** and renders a searchable click-to-expand list. Past ~100 rows a flat scroll is unusable, so the Services section carries a live text filter. Each flow gets its own unauthenticated path exception in the reverse-proxy config, the same pattern every Dashy widget on this fleet uses. + +![The Dashy Homelab Inventory page: Hardware expanded to three hosts (two NUC10i3FNH, one QNAP TS-251A), VMs/LXCs collapsed, Services expanded showing 111 rows with a "filter by host or service name" box and NATIVE / CONTAINER badges](../assets/homelab-inventory-dashy-page.png) + +## Scheduled, Not Triggered + +A single Kestra flow, `homelab-inventory-sync`, cron `0 12 * * 4` β€” Thursday noon. There's no webhook: nothing about live hardware or a running-process list is an event the orchestrator would otherwise see, so a fixed weekly cadence is the whole trigger. + +``` + - id: schedule + type: io.kestra.plugin.core.trigger.Schedule + cron: "0 12 * * 4" +``` + +One SSH task hops to the automation host and runs the three playbooks in sequence through the pinned virtualenv, capturing a per-tier exit code. A non-zero tier sends a critical Slack alert through the shared [`notifications`](notifications.md) pipeline; all-clear sends an info line. The dedicated-key SSH hop and the checkβ†’notify shape are the same model every flow in [`orchestration.md`](orchestration.md) uses. + +> [!IMPORTANT] +> **This layer never writes to the knowledge base, and the knowledge base never sources from it.** They're deliberately separate: the KB is hand-authored design intent, this is a machine-derived snapshot of current reality, and wiring either one to feed the other would collapse that distinction. A separate mechanism does push a handful of per-host CPU/RAM figures into the KB for at-a-glance context, but it's a different job with its own source and its own destination β€” it doesn't read or write anything here. + +--- +Related: [`docs/orchestration.md`](orchestration.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md), [`docs/cron-scheduler.md`](cron-scheduler.md), [`docs/notifications.md`](notifications.md) Β· Stacks: [`stacks/platform/nocodb/`](../stacks/platform/nocodb/), [`stacks/platform/node-red/`](../stacks/platform/node-red/), [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/kestra/`](../stacks/platform/kestra/)