Covers the design rationale, the refill-cycle mechanism, three real
issues found and fixed via live testing against a real Kodi session
(a silent result-key mismatch that meant the daemon never once
resolved a track, a starved similarity candidate pool, and a
4-minute poll-interval blind spot), and live proof of a working
auto-queued run of similar tracks.
Vault panel data was overstated as fully live; only CA cert expiry
actually is - the rest comes from a cached Prometheus query. Slack
channel list still named the decommissioned Netbox app instead of
the real Proxmox channel.
Completes the fix started in e74e049 - the Ansible Stages diagram's
Provisioning box now reads '2 roles today' instead of the stale
'1 role today, one host'. Provenance bumped to today's verify pass.
Partial fix - diagram update still pending (Nextcloud unavailable).
Live-verified against roles_metadata.yml: provisioning stage now has
2 roles (clamav, netbird_agent - added for NetBird mesh enrollment),
compliance stage now has 16 (added ssh_ca_client). Technitium's shown
tag-file example was stale (21->22 tags, missing netbird_agent).
containers.tf gained an 18-line header comment that pushed its actual
resource block past line 20, so the doc's 'head -20' example no longer
reproduced the shown output - fixed to a targeted sed range. Also
found and documented a genuine live inconsistency: host_tags.yml.tpl
gained an inventory_group field with the 2026-08 LXC rollout, but no
existing VM host has been re-applied since, so none of their rendered
files have it yet (only new LXCs do) - called out explicitly rather
than papered over.
A real log source was added the day after this doc's last verify
(2026-09-07): auditd logs now ship to Loki too, on the 6 hosts
carrying the auditd_hardening role, after a real incident where one
host's /var/log/audit filled to 100% with zero external visibility.
Updated the diagram to add the 4th branch (rebuilt via Nextcloud
Diagramming, re-exported) and added prose describing it. Also
corrected the diagram's DNS branch label (job=dns -> job=technitium-dns,
matching the doc's own already-correct evidence-screenshot caption).
Journald sizing config (1G rocky-docker override, 500M fleet default)
confirmed byte-for-byte accurate live on both.
Live-checked the flow export directly: NetBox's dead switch rule and
mapper function have actually been removed since this doc was written
(16->15 switch rules, 15->14 routed apps) - the doc's own 'debt' claim
was half-stale, since only Kestra's dead webhook route remains.
Corrected all four places this count appeared (routing-table example,
two screenshot captions, one diagram's alt-text). Also fixed the
mqtt-monitor.sh check-count example: raw grep now returns 23 due to a
companion state file (failed_units_current) for an existing check, not
a genuinely new 23rd check - reworded so the total (23) stays accurate
without implying a new check was silently added. Mosquitto broker
config/passwords/ACL and the Inventory Parse Function code both
confirmed unchanged.
Live-checked everything: 15 total Kestra flows (exact match), the
14-straight-through/1-gated split (API + pause-step grep + diagram all
agree), the KV store's exactly 2 entries, and the hyper2 config comment
(byte-for-byte). Only the automation-self-update.timer example had
gone stale (dates); refreshed to current live output.
Same music-assistant strategy change fixed in the KB earlier this
session (dockerfile -> pinned_manual, avoiding auto-update-triggered
audio-analysis invalidation): stack count corrected 36->35, added to
the Deliberate Holdouts table with its own real reason. Also found a
second gap: 'All 5 Docker hosts converted' should be 6 - NetBird now
has its own dockerfile-strategy stack (hawser), confirmed live in its
docker_stacks.yml, and was missing from this count.
Live-checked the container roster (exact version match), the Caddyfile
splitter config (byte-for-byte match), and mesh connectivity
(Management/Signal both confirmed connected). Peer-roster/policy-list
screenshots not independently re-verified (no accessible NetBird API
token - it's a manual dashboard PAT, not Vault/ansible-managed), but
nothing else indicates drift. Both diagrams still accurate. Bumping
verified date/SHAs only.
Found while auditing secret-distribution.md: NetBird is one of the 6
real docker_secret_templates-consuming hosts (confirmed via Agent
Configurations KB note and the live template list - config.yaml,
Hawser's secret, shared registry-token), but was missing from this
doc's consumer table entirely.
The registry-token example's shown grep command didn't actually
reproduce the shown output (it matched the first 'service: automation'
entry, gitea, not registry-token). Fixed to grep on the registry-token
path directly, so the command is genuinely reproducible. Re-verified
everything else live: SSH cert flow's 2h timer, the technitium API
token lookup snippet, and the flow diagram - all accurate.
Live-checked every claim: Vault mount list, SSH CA role TTLs/algorithm,
the Caddyfile's asymmetric-trust config for Vault (exact match), all
per-host consumer stack counts (12 rocky-docker / 10 automation,
verified against the authoritative docker_secret_templates source, not
just Vault KV path listings which include unrelated Kestra-flow
secrets), the cert-rebuild script's 19-day-incident story, and the
lifecycle diagram - all still accurate. Bumping verified date/SHAs only.
Re-verified every claim live: 12-agent roster, manager container name,
9 CIS sub-roles + 2 documented deviations, Docker Bench native-binary
story, and the SCA compliance screenshot (139/19/8/87%, confirmed exact
match against automation's live score) all still accurate - no drift.
Added a second real incident found in the same wazuh_agent_tuning file:
a 2026-08-30 fix for the role silently skipping the wazuh host's own
containerized agent.
Proxy provider count is 94 live, not 96. Bigger fix: the doc claimed the
5 pull-only docker hosts each have a separate read-only Gitea Deploy Key
- live investigation found no Deploy Key object exists anywhere in Gitea
at all. The real mechanism (docker_git_setup role, iac repo) copies the
same write-capable automation-bot account key identically to every host
at first provisioning. Corrected the claim and swapped the now-inaccurate
screenshot for the real provisioning task as evidence. Same fix applied
to the source KB note (Gitea Access Model).
Live-checked every specific claim (all 6 per-app zone-presence
examples, forwarders, corrected reverse-zone PTR, DNS-bypass firewall
rule) against Technitium and the firewall directly - all still
accurate. Bumping verified date only.
Kopia's coverage claim conflated the public repo's curated stacks/
folder (a documentation subset) with the real live stack count on
rocky-docker - fixed to the actual live figure (41, cross-checked
against docker_stacks.yml and the real repo directory count), same
fix applied to the source KB note which had its own stale ~38 figure
and an incorrect hyper1 backup mode (suspend -> stop, confirmed
against live /etc/pve/jobs.cfg). Refreshed the ls/df and
roles_metadata.yml command-output evidence to match current state.
Live NocoDB now has 49 jobs/14 without a fixed time, not 50/15 - traced
to a removed inventory row for a "PBS prune job" that never actually
existed (confirmed against live data and the KB note, which states PBS
runs no prune job at all). Updated counts and refreshed all three
screenshots to match current state.
Live-verification found two real inaccuracies: WAN now passes 4 things
inbound (NetBird's STUN forward was added, undocumented), and the "third
fix" LAN-to-LAN narrative credited pfSense's ruleset when the actual
enforcement for host-to-host traffic is each host's own firewalld (pfSense
never sees same-subnet traffic on this flat LAN). Refreshed the WAN rules
screenshot and reworded the boundary table + narrative to match. Also
cleaned up the "Other Six Traffic Laws" table formatting per feedback.
The Kestra namespace gained public-docs-drift since this doc was last
verified. New flow-list screenshot (Total: 15), flow-patterns diagram
re-exported to 14 of 15 no-gate, and the pause-step snippet now includes
it. Also corrects the "only two things write to Trilium" claim — that
flow stamps #pendingPublicDoc labels, making it a third writer (labels
only, never content). Same correction applied to the KB source note.
New doc for the weekly self-deriving inventory pipeline: a three-tier
Ansible collection (hardware off the hypervisors/firewall, VMs/LXCs from
the Proxmox APIs, listening sockets off every guest) writing three linked
NocoDB tables, read live behind a Dashy page.
Covers the design decisions worth defending: ss -tlnp over systemctl for
"what does this host serve", merge-only-by-omission on Tier 1 to protect
the one hand-entered field a lying SMBIOS can't provide, API-only Tier 2
because the two Proxmox nodes aren't clustered, stdlib-only push scripts,
and the deliberate KB-vs-machine-layer separation.
Pipeline diagram + four screenshots (Dashy page, all three NocoDB tables).
Provenance comment wired to the ansible roles/playbooks, the rocky Caddy/
dashy config, and KB note O0d4fqw0mr05. Indexed in README under Systems.
The standing root token this doc described is retired. Replaced the
2-flow table with the real 3-flow model now in place: routine AppRole
auth, a narrowly-scoped provisioner AppRole for occasional bootstrap
work, and a break-glass generate-root path for the rare cases the
provisioner deliberately can't cover. Alerting row corrected to note
the audit-log check fires on any root-policy token generically, not
a specific standing one.
Two command-block "evidence" snippets added earlier today pasted real,
live-captured values verbatim: the exact cert file path, the fleet's
actual CA fingerprint, today's real issuance timestamp, and the live
Vault role/principal name. All of it identified real infrastructure
in a way the rest of this repo's established redaction convention
(see docs/firewall-and-network.md's [admin workstation]-style
placeholders) already exists specifically to avoid. Redacted to
bracket placeholders; the structural/format evidence (cert type,
extensions, TTLs) stays genuine.
"No immediate disruption" was true for TLS certs and app secrets but not
for the SSH certificate refresh, which has no static-key fallback and
eventually locks the automation identity out entirely during a sustained
Vault outage. The "always talks through a local agent" law also had an
unstated exception worth naming rather than leaving implicit.
Vault's ssh-client-signer mount was undocumented here entirely. Adds it
as flow 2/5 rather than merging into Certificate Flow, since the actual
mechanism differs (own systemd timer, bypasses Vault Agent's render
cycle, only reuses its cached token) - and explains why that distinction
matters for debugging. Diagram updated to match: a third data-plane box
feeding fleet-wide sshd trust instead of a container.
The SSH CA mount (ssh-client-signer/) existed live in Vault but was
missing from this doc's capability list entirely. Adds it as a fourth
"What It Does" bullet with real role config and a live screenshot of
the secrets engines list, and refreshes the stale ACL policies
screenshot (was showing 15 policies, live is 18).
The old claim described two flows sharing one keypair under different
usernames as a deliberate scoping decision. That was actually a stale,
half-migrated credential mismatch, not a design choice - the real fix was
giving Kestra a dedicated keypair with zero relation to the ansible
automation identity's own key, so a leak of one can never expose the other.
Kestra's SSH plugin has no certificate support at all (a plain Java client,
confirmed via its own plugin schema), which is the actual reason it needs
its own static key regardless of what the fleet's ansible identity does.
Added the real current flow config (kestra_local_user/kestra_local_key) and
a screenshot of the live KV Store showing exactly those two entries. Also
refreshed the automation-self-update.timer example, which had gone stale.
Small, precise clarification - the table row was always worded
generically ('a host', not 'a VM') so it was never technically wrong,
just aspirational for LXC hosts before Terraform could provision them
natively. Now genuinely true for both, so it says so.
Live re-ran agent_control -l against the real Wazuh manager - the
netbird host (ID 013) is now enrolled and Active, matching the fleet's
real current host count.
Live-verified against the new NetBird host: firewalld genuinely locked
to 3 explicit rich-rules (zero blanket ports), yet 3 container-published
ports (netbird-proxy, netbird-server, hawser) sit completely outside
those rules and are reachable LAN-wide via Docker's own NAT - the exact
gap the doc already documented, now with a second real example on a
host provisioned this same week instead of just the one from before.
Fleet host count was stale (11, now 12 with the NetBird control plane).
Traffic law 8 (Management) now reflects the real new off-LAN admin path
via the zero-trust mesh, alongside the existing LAN-scoped firewall
rules - accurate about what's affected if the mesh goes down (on-LAN
access unaffected, off-LAN access specifically loses its only path).
README: real dedicated 'Zero Trust, Actually Implemented' section (general
concept -> NetBird as the concrete implementation -> link out), host/doc
count bumped to 12/16, NetBird tool bullet.
netbird-zero-trust.md: dropped the LXC-provisioning tangent (not relevant
to a doc about NetBird itself), added self-hosted-vs-SaaS rationale, a
real Addressing table, and expanded split-DNS mechanism detail. Also
genericized personal-device specifics (device types/exact split of
access) that were too specific for the public repo.
automation-bot
thesabear