Covers the design rationale, the refill-cycle mechanism, three real
issues found and fixed via live testing against a real Kodi session
(a silent result-key mismatch that meant the daemon never once
resolved a track, a starved similarity candidate pool, and a
4-minute poll-interval blind spot), and live proof of a working
auto-queued run of similar tracks.
Completes the fix started in e74e049 - the Ansible Stages diagram's
Provisioning box now reads '2 roles today' instead of the stale
'1 role today, one host'. Provenance bumped to today's verify pass.
A real log source was added the day after this doc's last verify
(2026-09-07): auditd logs now ship to Loki too, on the 6 hosts
carrying the auditd_hardening role, after a real incident where one
host's /var/log/audit filled to 100% with zero external visibility.
Updated the diagram to add the 4th branch (rebuilt via Nextcloud
Diagramming, re-exported) and added prose describing it. Also
corrected the diagram's DNS branch label (job=dns -> job=technitium-dns,
matching the doc's own already-correct evidence-screenshot caption).
Journald sizing config (1G rocky-docker override, 500M fleet default)
confirmed byte-for-byte accurate live on both.
Proxy provider count is 94 live, not 96. Bigger fix: the doc claimed the
5 pull-only docker hosts each have a separate read-only Gitea Deploy Key
- live investigation found no Deploy Key object exists anywhere in Gitea
at all. The real mechanism (docker_git_setup role, iac repo) copies the
same write-capable automation-bot account key identically to every host
at first provisioning. Corrected the claim and swapped the now-inaccurate
screenshot for the real provisioning task as evidence. Same fix applied
to the source KB note (Gitea Access Model).
Live NocoDB now has 49 jobs/14 without a fixed time, not 50/15 - traced
to a removed inventory row for a "PBS prune job" that never actually
existed (confirmed against live data and the KB note, which states PBS
runs no prune job at all). Updated counts and refreshed all three
screenshots to match current state.
Live-verification found two real inaccuracies: WAN now passes 4 things
inbound (NetBird's STUN forward was added, undocumented), and the "third
fix" LAN-to-LAN narrative credited pfSense's ruleset when the actual
enforcement for host-to-host traffic is each host's own firewalld (pfSense
never sees same-subnet traffic on this flat LAN). Refreshed the WAN rules
screenshot and reworded the boundary table + narrative to match. Also
cleaned up the "Other Six Traffic Laws" table formatting per feedback.
The Kestra namespace gained public-docs-drift since this doc was last
verified. New flow-list screenshot (Total: 15), flow-patterns diagram
re-exported to 14 of 15 no-gate, and the pause-step snippet now includes
it. Also corrects the "only two things write to Trilium" claim — that
flow stamps #pendingPublicDoc labels, making it a third writer (labels
only, never content). Same correction applied to the KB source note.
New doc for the weekly self-deriving inventory pipeline: a three-tier
Ansible collection (hardware off the hypervisors/firewall, VMs/LXCs from
the Proxmox APIs, listening sockets off every guest) writing three linked
NocoDB tables, read live behind a Dashy page.
Covers the design decisions worth defending: ss -tlnp over systemctl for
"what does this host serve", merge-only-by-omission on Tier 1 to protect
the one hand-entered field a lying SMBIOS can't provide, API-only Tier 2
because the two Proxmox nodes aren't clustered, stdlib-only push scripts,
and the deliberate KB-vs-machine-layer separation.
Pipeline diagram + four screenshots (Dashy page, all three NocoDB tables).
Provenance comment wired to the ansible roles/playbooks, the rocky Caddy/
dashy config, and KB note O0d4fqw0mr05. Indexed in README under Systems.
Previous screenshot's token counts/TTL distribution predated card 59
(11 active tokens, 20m/1h/no-expiry split including the standing root
token). Current state: 14 active tokens, 1h/no-expiry split - no root
token in the mix anymore.
Vault's ssh-client-signer mount was undocumented here entirely. Adds it
as flow 2/5 rather than merging into Certificate Flow, since the actual
mechanism differs (own systemd timer, bypasses Vault Agent's render
cycle, only reuses its cached token) - and explains why that distinction
matters for debugging. Diagram updated to match: a third data-plane box
feeding fleet-wide sshd trust instead of a container.
The SSH CA mount (ssh-client-signer/) existed live in Vault but was
missing from this doc's capability list entirely. Adds it as a fourth
"What It Does" bullet with real role config and a live screenshot of
the secrets engines list, and refreshes the stale ACL policies
screenshot (was showing 15 policies, live is 18).
The old claim described two flows sharing one keypair under different
usernames as a deliberate scoping decision. That was actually a stale,
half-migrated credential mismatch, not a design choice - the real fix was
giving Kestra a dedicated keypair with zero relation to the ansible
automation identity's own key, so a leak of one can never expose the other.
Kestra's SSH plugin has no certificate support at all (a plain Java client,
confirmed via its own plugin schema), which is the actual reason it needs
its own static key regardless of what the fleet's ansible identity does.
Added the real current flow config (kestra_local_user/kestra_local_key) and
a screenshot of the live KV Store showing exactly those two entries. Also
refreshed the automation-self-update.timer example, which had gone stale.
Was reusing netbird-zero-trust-mesh-topology.svg (the doc's detailed
peer-topology diagram) in the README too - breaks this repo's own
established convention of the README getting its own dedicated asset
even when covering related material (see readme-trust-boundaries-
blast-radius.svg vs architecture-overview-trust-depth.svg). New diagram
is a simple perimeter-vs-zero-trust conceptual contrast, matching the
README section's generalized framing rather than NetBird's specific
implementation topology.
New file covering NetBird - a self-hosted WireGuard mesh giving every
fleet host and personal device a stable overlay identity, with a
genuinely-implemented zero-trust policy layer (deny-by-default, groups,
scoped setup keys, peer approval, posture checks).
- Why a mesh over a classic VPN concentrator, illustrated with a real
topology diagram (control plane handles coordination only; actual
data flows peer-to-peer, bypassing it entirely - the literal
distinction from a concentrator, drawn not just asserted)
- Ties back to today's iac-pipeline.md update: this LXC is the first
host ever built through the new Terraform LXC-native path, start to
finish, no manual provisioning step
- Real bring-up incident: gRPC/h2c requirements broke through a naive
proxy chain, and a proxying CDN 403s the agent outright - both root
causes documented, not just the fixes
- Evidence: 3 real command blocks (docker ps, live Caddyfile splitter
config, netbird status --detail from an enrolled peer), 2 real
screenshots (dashboard Peers list, Access Control Policies - port
counts in the screenshot match the real policy detail in the text)
- Second diagram (edge routing chain) for the specific gRPC/h2c/path
split detail, kept separate from the topology diagram since they show
genuinely different shapes of information
- Honest, non-rounded-up rollout status table - several phases still
explicitly open
- Sanitize-checked: no internal IPs/domains/personal email
Also: fixes iac-pipeline.md's forward-reference to this doc to use the
repo's actual link convention ([docs/x.md](x.md)), not descriptive
prose link text.
New export reflecting the user's latest diagram edit. Same white-line
strip as aae1ff2 applied fresh to this export (variant this time: flat
#1C1D1E plus inverted light-dark(#1C1D1E, #d3d4d5)) - verified 0 broken
patterns remain, valid XML, all 8 correct span backgrounds preserved.
Same fix as orchestration-flow-patterns.svg (8772564), applied after a
comprehensive re-check that found the bug wasn't limited to the 6 files
originally caught by a narrower pattern match - architecture-overview.svg
and firewall-and-network-docker-bypass.svg were also affected.
Confirmed via a control test on orchestration-flow-patterns.svg that the
.drawio source's labelBackgroundColor property was never the actual
cause (reverted it, re-exported, still broken, still fixed cleanly by
the same strip) - this is a draw.io export-time quirk, not a content or
convention problem, so no CLAUDE.md changes needed.
Each file: stripped the same broken wrapper-div background-color
declarations (flat #1C1D1E / #ffffff, or the equivalent light-dark()/
var(--ge-adaptive-bg) variants - not consistent across exports of the
same file), verified 0 remain, valid XML, and the correct adaptive span
background count unchanged before/after in every file.
Files: architecture-overview.svg, firewall-and-network-docker-bypass.svg,
iac-pipeline-tools-flow.svg, observability-pipeline.svg,
readme-trust-boundaries-blast-radius.svg, secret-distribution-flows.svg,
secrets-and-pki-lifecycle.svg
The exported SVG's 4 edge labels (no/yes/approve/decline) had two extra,
non-adaptive background-color declarations stamped onto their wrapper
divs by the draw.io exporter (flat #ffffff, and an undefined
var(--ge-adaptive-bg) CSS variable that falls back to #ffffff) -
independent of both the .drawio source content and the export dialog's
Appearance setting (tested Automatic and Light, both showed it).
Surgically stripped just those two declarations from the exported SVG,
verified count (4+4 removed, 0 remaining), confirmed the correct
adaptive span background (light-dark(white, #1C1D1E)) and every other
color in the diagram were untouched, confirmed still valid XML.
- Replaced both mermaid diagrams with real Nextcloud diagrams; the
Ansible-stages rebuild caught a real omission - the old diagram showed
only 2 stages (init/compliance) when there are genuinely 3
(provisioning.yml exists too, currently just clamav/rocky-docker)
- Live-verified against the real iac repo: roles_metadata.yml (9
cis_hardening + 6 other compliance roles, 1 provisioning role),
kickstart partitioning/packages/SSH config, both Packer bootstrap
scripts, terraform.tfvars, pct list on both hypervisors, a real
rendered host_tags.yml
- KB fix: LXC Strategy note's PBS row was wrong, not just stale - PBS
genuinely has a populated vm_tags list and live compliance roles
(confirmed via real artifacts on the host itself), not 'never
Ansible-managed' as it said; corrected and re-verified via re-fetch
- Added real content not previously in the doc: the deliberately
permissive build-time SSH window closed by ssh_hardening, the
admin/ansible identity split baked into the image at build time, and a
third pipeline exception (PBS) that's genuinely different in kind from
the other two, including a real Rocky-vs-Debian PAM ordering bug found
onboarding it
- Considered two screenshot candidates (Proxmox template list, VM/LXC
mix) - declined as low-value; command evidence covers this
source-driven topic well on its own
- Sanitize-checked: no internal IPs/MACs/domains/personal info/forbidden
app names
- Replaced mermaid diagram with real Nextcloud diagram; rebuild caught a
real bug in the old one (showed Kopia/Bazarr flowing through the App
Router, which flows.json disproves - they bypass it entirely)
- Live-verified Mosquitto broker config/ACL, Node-RED flow routing (16
switch rules = 15 apps + unknown fallback, 2 dedicated endpoints),
mqtt-monitor.sh check count against the real script
- Named NetBox and Kestra directly (previously generic) after the real
Node-RED screenshot made them visible anyway; Postiz/nocodb-content
stay generic per standing instruction
- Added real evidence: routing-table command output, the real
topic-agnostic Inventory Parse function source, real Mosquitto
ACL/config, real check-count grep, a real Node-RED flow canvas
screenshot, and a real Mosquitto connection log screenshot (the
kestra-notify client is live proof of the MQTT-not-HTTP claim)
- KB fix: Ansible mqtt_client Role note was missing a 23rd real check
(crash-looping systemd units, distinct from the failed-units check,
added to the script after a real vault-agent crash-loop incident)
- Sanitize-checked: no internal IPs/domains/personal info/forbidden app
names
1. Live-verify: journald persistent-storage config confirmed on two
hosts (1G vs 500M cap, matches the doc's claim exactly), Alloy
active confirmed across all 11 fleet hosts.
2. Fix KB: Alloy Agent Model was stale - version had moved from
v1.18.0 to v1.18.1 on all 8 'newer' hosts since last checked,
role's own default still unreconciled at 1.17.1. Fixed and
confirmed via re-fetch.
3. Pulled the full Observability Pipeline tree: Alloy Agent Model,
Log Flow, Metrics Flow.
4. Replaced both mermaid diagrams (broke this repo's Nextcloud-SVG
convention) with real diagrams built through the proper round-trip.
5. Evidence: real journald config dumps, plus 3 real Grafana
screenshots (host metrics dashboard, detailed Loki log stream
showing job=technitium-dns/docker/system, and the fleet-wide log
volume dashboard) - one screenshot round caught and fixed an
unredacted internal IP before being used.
6/7. Compelling-proof test passes; sanitize-check clean.
Step 5 re-audit, completing the checklist gap that started this whole
detour. Added: the real hyper2 host_vars comment (matches the WARNING
callout's reasoning almost word for word), the real automation-self-
update.timer schedule, the real Kestra flow list (visually confirms
13 total flows), and app-update's real execution topology (visually
confirms wait_for_approval as an actual node, not just a diagram
concept).
Step 5 re-audit. The wazuh_agent_tuning root-cause comment adds a
specific fact not previously in the doc (9 of the fleet's agents
affected). The SCA dashboard screenshot is the doc's first visual
evidence - everything else was CLI/text - and shows a real 87% score,
directly proving the 'below 100% is an accepted tradeoff' claim with
an actual number instead of just asserting the policy.
Step 5 re-audit. Caddyfile contrast (skip-verify default vs Vault's
real cert verification block) shown as text, IP/domain genericized.
The cert-rebuild hook script's real 19-day-expiry story is shown as a
screenshot instead of a text excerpt - the script has a hardcoded P12
password in its body, so a screenshot with that one line blurred shows
the actual guard logic in full without a text-copy risk of the
password ever landing in a file.
Two fixes from the step 5 re-audit:
1. The joplin/public-tool evidence blocks showed bracketed placeholders
inside function arguments ('joplin.[app-to-app zone]') - not valid
syntax, wouldn't produce the shown NXDOMAIN for the claimed reason.
Replaced with result tables showing the same real data honestly.
2. Added the real Technitium screenshot of the cleaned-up reverse zone.
Getting this screenshot caught a real methodology error: an earlier
claim of '2 real PTR records (automation, caddy)' was wrong - the
automation entry only resolved because I tested it by SSHing into
automation itself, which satisfied the lookup via its own local
/etc/hosts, never actually querying Technitium. Querying from a
different host returns NXDOMAIN, matching the screenshot's real
3-record zone (NS, SOA, one PTR) exactly. Only 1 real PTR record
exists. Same error existed in the KB, fixed there too.
Step 5 re-audit - real screenshot of the Vault tile's expanded detail
panel (UNSEALED, real lease/token/KV/secret-engine counts, TTL
distribution, CA cert expiry), showing the specialty-widget behavior
the Layout section describes but the existing screenshot doesn't show.
Captured after fixing the metrics-reader token issue found while
requesting this screenshot - first attempt genuinely showed a false
SEALED state, now shows the real healthy panel.
Step 5 re-audit - real screenshot showing the underlying table data
(50 records, all columns) that the Dashy heatmap is rendered from,
distinct from the two existing rendered-widget screenshots.
Live-checked every flow's actual Kestra definition for a Pause task.
Only app-update has one; the other 6 main flows (os-update-vms,
container-update, disk-cleanup, os-update-pbs, os-update-pve,
os-update-pve-verify) run fully automatically with no human checkpoint
at all - their safety net is the automated health-check/rollback path,
not human approval. The doc previously presented check->pause->approve
as the universal pattern for all 13 flows and stated outright that
'nothing gets changed on the fleet without a human explicitly resuming
a paused flow', which was false for the flows that actually change
the fleet. The KB was already correct on this point.
Also replaced the mermaid diagram (broke this repo's established
Nextcloud-SVG convention) with a real diagram showing both patterns,
plus live evidence of the Pause-task check itself.
Live-verified against real infrastructure: Kopia's real repo/disk,
PBS's real datastore path, os_update role's real batch/abort behavior
against the actual Ansible code, and the real tag-driven
roles_metadata.yml mechanism behind Layer 1's rebuild claim. Caught a
real KB staleness in the process (PBS datastore usage, 3 weeks stale)
and fixed it properly - removed the point-in-time number entirely
rather than refreshing it, since a data-snapshot doesn't belong in
Trilium at all per the existing content-philosophy rule. Replaced
both mermaid diagrams with real branch/decoupling-shaped Nextcloud
diagrams. All 3 source KB notes stamped.
Found by actually re-checking the running function instead of stopping
at what was already written: each finding has a "Copy Debug Prompt"
button generating a ready-to-paste AI-triage prompt, with a built-in
"don't act without approval" instruction baked into the template
itself. Embedded the real board screenshot (redacted) - dependency
ladder, Wazuh KPIs, Active Issues strip, per-host tile grid - since a
doc about a visual dashboard needs to actually show it, not just
describe it in command output.
Live-verified every specific claim directly against the real Node-RED
function source (not just trusted from the KB): the 104,443-char
function size, all 5 Wazuh finding categories, Vault's live PKI cert
endpoints, the real DNS Security Signals implementation, the
fleetAudit global pattern, the 12-app Slack channel map, the
acknowledge key format and localStorage key name, and the Mustache
triple-brace URL gotcha. Replaced both mermaid diagrams: the flat
data-sources fan-in became a real specifics table (redundant diagram
cut per the diagramming standing process), and the acknowledge-flow
diagram was rebuilt as a real Nextcloud diagram. Stamped both source
KB notes.
Live-verified against the real Technitium server: spot-checked
caddy-only/local-only hosts and Postiz's zero-presence claim via
direct DNS resolution, confirmed the forwarder list directly from the
real dns.config (pfSense + Quad9 only, no Cloudflare), and confirmed
the reverse zone's real /24 scope plus its sparse PTR content via
live reverse lookups - fixed the KB stamp accordingly after an
earlier version of it claimed these weren't re-checked. Replaced the
mermaid zone-routing diagram with a real branch-shaped Nextcloud
diagram. Real command output embedded throughout instead of
paraphrased claims.
Live-verified the container_update role's strategy_dockerfile.yml
task file exists at the exact path the KB names, stamped the source
note. Replaced the mermaid cheap-check flowchart with a real
branch-shaped Nextcloud diagram. Added: the real Vault-backed push
auth mechanism (Gitea PAT -> Vault Agent -> docker login
--password-stdin, same pattern as everywhere else in the fleet), the
real docker manifest inspect command instead of a vague "digest
check", real app names in the holdouts table instead of generic
descriptions (matching how the rest of this repo already names real
apps), the anonymous-volume-pruning detail, and a real (redacted)
screenshot of the actual Gitea package registry showing genuine
self-built container images.
Live-verified against the real Wazuh manager (agent_control -l: 11
active agents matching exactly; docker ps: real manager container
name) and stamped both source KB notes. Replaced both mermaid
flowcharts with real branch-shaped Nextcloud diagrams (alert-handling
gate, compliance-hardening gate). Added real evidence throughout: the
live agent roster, a real synthetic SSH-failure line piped through
wazuh-logtest showing actual decoding/rule-match output (MITRE ATT&CK
+ compliance tags included), and real rule IDs in the runbook table
instead of vague categories. New content: the fleet-wide agent
log-queue-overflow bug found and fixed via a dedicated tuning role,
and the compose-stack-vs-rules version-control split.
Swapped the curl/jq text block for what the doc actually needed:
real screenshots of the live Dashy Fleet Schedule Heatmap widget -
one showing a populated cell clicked open (proving the TIP's
click-to-expand claim is real, not just described), one showing the
"no fixed time" section for the 15 jobs excluded from the grid.
The zone-band labels spelled out *.local.thesabear.net and
*.caddy.thesabear.net literally - the same internal zone naming
pattern already sanitized out of this doc's prose, missed on the
diagram itself. Replaced with generic zone names, no domain.
Sequence diagram showing the browser carries every hop itself -
Authentik and the app never talk directly - with only the OAuth
authorize/token exchange crossing from the browser-facing zone into
the service-to-service zone. Built to the public-repo diagram
conventions in CLAUDE.md (labelBackgroundColor + matching rich-text
span, off-line label positioning, line-color-matched labels, legend
instead of a color-explaining caption).
Asset renames (matching the new Nextcloud source naming convention —
doc name first, then description):
- architecture-trust-depth.svg -> architecture-overview-trust-depth.svg
- firewall-docker-bypass.svg -> firewall-and-network-docker-bypass.svg
- secrets-lifecycle.svg -> secrets-and-pki-lifecycle.svg
- trust-boundaries-blast-radius.svg -> readme-trust-boundaries-blast-radius.svg
Updated all 4 markdown references accordingly.
Also refreshed diagram exports (secret-distribution-flows.svg, and the
4 renamed ones) with today's fixes, and folded in the earlier pending
sso-and-identity.md work: precise live-verified native-OIDC app list
(12 apps, not ~9), the Proxmox no-Proxy-Provider exception, real API
evidence for it, and the automation-bot migration story.
Root cause was the edge's own labelBackgroundColor style property,
left enabled with no explicit value in the app, distinct from the
label's HTML text styling entirely. Set it explicitly to #1C1D1E,
then stripped a competing plain-white background layer draw.io's
export was still injecting on top of it.
draw.io's export was auto-injecting its own adaptive-background CSS
onto edge labels regardless of source content, with a fallback that
didn't match the page. Stripped background-color entirely so labels
sit directly on the page background with nothing to mismatch.
Nextcloud's draw.io export was inconsistently applying its own
page-level adaptive-background variable to some edge labels instead
of the authored value, making them render darker than the rest.
Post-processed the exported SVG to normalize every label to the same
background value.
automation-bot
thesabear