Small, precise clarification - the table row was always worded
generically ('a host', not 'a VM') so it was never technically wrong,
just aspirational for LXC hosts before Terraform could provision them
natively. Now genuinely true for both, so it says so.
Live re-ran agent_control -l against the real Wazuh manager - the
netbird host (ID 013) is now enrolled and Active, matching the fleet's
real current host count.
Live-verified against the new NetBird host: firewalld genuinely locked
to 3 explicit rich-rules (zero blanket ports), yet 3 container-published
ports (netbird-proxy, netbird-server, hawser) sit completely outside
those rules and are reachable LAN-wide via Docker's own NAT - the exact
gap the doc already documented, now with a second real example on a
host provisioned this same week instead of just the one from before.
Fleet host count was stale (11, now 12 with the NetBird control plane).
Traffic law 8 (Management) now reflects the real new off-LAN admin path
via the zero-trust mesh, alongside the existing LAN-scoped firewall
rules - accurate about what's affected if the mesh goes down (on-LAN
access unaffected, off-LAN access specifically loses its only path).
README: real dedicated 'Zero Trust, Actually Implemented' section (general
concept -> NetBird as the concrete implementation -> link out), host/doc
count bumped to 12/16, NetBird tool bullet.
netbird-zero-trust.md: dropped the LXC-provisioning tangent (not relevant
to a doc about NetBird itself), added self-hosted-vs-SaaS rationale, a
real Addressing table, and expanded split-DNS mechanism detail. Also
genericized personal-device specifics (device types/exact split of
access) that were too specific for the public repo.
New file covering NetBird - a self-hosted WireGuard mesh giving every
fleet host and personal device a stable overlay identity, with a
genuinely-implemented zero-trust policy layer (deny-by-default, groups,
scoped setup keys, peer approval, posture checks).
- Why a mesh over a classic VPN concentrator, illustrated with a real
topology diagram (control plane handles coordination only; actual
data flows peer-to-peer, bypassing it entirely - the literal
distinction from a concentrator, drawn not just asserted)
- Ties back to today's iac-pipeline.md update: this LXC is the first
host ever built through the new Terraform LXC-native path, start to
finish, no manual provisioning step
- Real bring-up incident: gRPC/h2c requirements broke through a naive
proxy chain, and a proxying CDN 403s the agent outright - both root
causes documented, not just the fixes
- Evidence: 3 real command blocks (docker ps, live Caddyfile splitter
config, netbird status --detail from an enrolled peer), 2 real
screenshots (dashboard Peers list, Access Control Policies - port
counts in the screenshot match the real policy detail in the text)
- Second diagram (edge routing chain) for the specific gRPC/h2c/path
split detail, kept separate from the topology diagram since they show
genuinely different shapes of information
- Honest, non-rounded-up rollout status table - several phases still
explicitly open
- Sanitize-checked: no internal IPs/domains/personal email
Also: fixes iac-pipeline.md's forward-reference to this doc to use the
repo's actual link convention ([docs/x.md](x.md)), not descriptive
prose link text.
Terraform gained a second provider (bpg/proxmox) alongside the existing
telmate/proxmox, via a new containers.tf / proxmox_virtual_environment_container
resource - real, live-verified against the actual repo files. Corrects
the doc's previous "Terraform only manages virtual machines... cannot
create or manage LXC containers" claim, which is no longer true.
- Terraform section: VM/LXC comparison table, real containers.tf
excerpt, explains why LXCs need an extra pct-exec bootstrap step
(stock template has no sshd, since it never went through Packer)
- WARNING section: new paragraph noting the LXC-provisioning gap is
closed for new hosts going forward (the three existing exceptions
stay as preserved history, not retroactively fixed) - forward-links
to docs/netbird-zero-trust.md (next commit)
- Fixed now-stale section heading ("one resource type" -> "two
resource types")
- Sanitize-checked: no internal IPs/domains
- Replaced both mermaid diagrams with real Nextcloud diagrams; the
Ansible-stages rebuild caught a real omission - the old diagram showed
only 2 stages (init/compliance) when there are genuinely 3
(provisioning.yml exists too, currently just clamav/rocky-docker)
- Live-verified against the real iac repo: roles_metadata.yml (9
cis_hardening + 6 other compliance roles, 1 provisioning role),
kickstart partitioning/packages/SSH config, both Packer bootstrap
scripts, terraform.tfvars, pct list on both hypervisors, a real
rendered host_tags.yml
- KB fix: LXC Strategy note's PBS row was wrong, not just stale - PBS
genuinely has a populated vm_tags list and live compliance roles
(confirmed via real artifacts on the host itself), not 'never
Ansible-managed' as it said; corrected and re-verified via re-fetch
- Added real content not previously in the doc: the deliberately
permissive build-time SSH window closed by ssh_hardening, the
admin/ansible identity split baked into the image at build time, and a
third pipeline exception (PBS) that's genuinely different in kind from
the other two, including a real Rocky-vs-Debian PAM ordering bug found
onboarding it
- Considered two screenshot candidates (Proxmox template list, VM/LXC
mix) - declined as low-value; command evidence covers this
source-driven topic well on its own
- Sanitize-checked: no internal IPs/MACs/domains/personal info/forbidden
app names
- Replaced mermaid diagram with real Nextcloud diagram; rebuild caught a
real bug in the old one (showed Kopia/Bazarr flowing through the App
Router, which flows.json disproves - they bypass it entirely)
- Live-verified Mosquitto broker config/ACL, Node-RED flow routing (16
switch rules = 15 apps + unknown fallback, 2 dedicated endpoints),
mqtt-monitor.sh check count against the real script
- Named NetBox and Kestra directly (previously generic) after the real
Node-RED screenshot made them visible anyway; Postiz/nocodb-content
stay generic per standing instruction
- Added real evidence: routing-table command output, the real
topic-agnostic Inventory Parse function source, real Mosquitto
ACL/config, real check-count grep, a real Node-RED flow canvas
screenshot, and a real Mosquitto connection log screenshot (the
kestra-notify client is live proof of the MQTT-not-HTTP claim)
- KB fix: Ansible mqtt_client Role note was missing a 23rd real check
(crash-looping systemd units, distinct from the failed-units check,
added to the script after a real vault-agent crash-loop incident)
- Sanitize-checked: no internal IPs/domains/personal info/forbidden app
names
1. Live-verify: journald persistent-storage config confirmed on two
hosts (1G vs 500M cap, matches the doc's claim exactly), Alloy
active confirmed across all 11 fleet hosts.
2. Fix KB: Alloy Agent Model was stale - version had moved from
v1.18.0 to v1.18.1 on all 8 'newer' hosts since last checked,
role's own default still unreconciled at 1.17.1. Fixed and
confirmed via re-fetch.
3. Pulled the full Observability Pipeline tree: Alloy Agent Model,
Log Flow, Metrics Flow.
4. Replaced both mermaid diagrams (broke this repo's Nextcloud-SVG
convention) with real diagrams built through the proper round-trip.
5. Evidence: real journald config dumps, plus 3 real Grafana
screenshots (host metrics dashboard, detailed Loki log stream
showing job=technitium-dns/docker/system, and the fleet-wide log
volume dashboard) - one screenshot round caught and fixed an
unredacted internal IP before being used.
6/7. Compelling-proof test passes; sanitize-check clean.
Step 5 re-audit, completing the checklist gap that started this whole
detour. Added: the real hyper2 host_vars comment (matches the WARNING
callout's reasoning almost word for word), the real automation-self-
update.timer schedule, the real Kestra flow list (visually confirms
13 total flows), and app-update's real execution topology (visually
confirms wait_for_approval as an actual node, not just a diagram
concept).
Step 5 re-audit. The table itself (the file's central correction from
earlier this session) had no embedded evidence - only the corrected
count was asserted. Added the real, freshly-repulled provider list
confirming all 10 by actual model type, not just name.
Step 5 re-audit. The wazuh_agent_tuning root-cause comment adds a
specific fact not previously in the doc (9 of the fleet's agents
affected). The SCA dashboard screenshot is the doc's first visual
evidence - everything else was CLI/text - and shows a real 87% score,
directly proving the 'below 100% is an accepted tradeoff' claim with
an actual number instead of just asserting the policy.
Step 5 re-audit. Caddyfile contrast (skip-verify default vs Vault's
real cert verification block) shown as text, IP/domain genericized.
The cert-rebuild hook script's real 19-day-expiry story is shown as a
screenshot instead of a text excerpt - the script has a hardcoded P12
password in its body, so a screenshot with that one line blurred shows
the actual guard logic in full without a text-copy risk of the
password ever landing in a file.
Step 5 re-audit. Both verified live earlier this session but never
shown: the registry-token shared template entry (proves the shared-
path mechanism), and the technitium_api_token lookup (proves the
flagged admin/runtime-split exception with the actual Jinja syntax).
Step 5 re-audit. Both were verified live earlier this session but
never shown - the strategy breakdown backs the doc's own headline
stat directly, and the mariadb-latest entry proves the specific
shared-mirror mechanism rather than just describing it.
Two fixes from the step 5 re-audit:
1. The joplin/public-tool evidence blocks showed bracketed placeholders
inside function arguments ('joplin.[app-to-app zone]') - not valid
syntax, wouldn't produce the shown NXDOMAIN for the claimed reason.
Replaced with result tables showing the same real data honestly.
2. Added the real Technitium screenshot of the cleaned-up reverse zone.
Getting this screenshot caught a real methodology error: an earlier
claim of '2 real PTR records (automation, caddy)' was wrong - the
automation entry only resolved because I tested it by SSHing into
automation itself, which satisfied the lookup via its own local
/etc/hosts, never actually querying Technitium. Querying from a
different host returns NXDOMAIN, matching the screenshot's real
3-record zone (NS, SOA, one PTR) exactly. Only 1 real PTR record
exists. Same error existed in the KB, fixed there too.
Step 5 re-audit - real screenshot of the Vault tile's expanded detail
panel (UNSEALED, real lease/token/KV/secret-engine counts, TTL
distribution, CA cert expiry), showing the specialty-widget behavior
the Layout section describes but the existing screenshot doesn't show.
Captured after fixing the metrics-reader token issue found while
requesting this screenshot - first attempt genuinely showed a false
SEALED state, now shows the real healthy panel.
The block added earlier this pass showed a grep pattern with a
bracketed placeholder embedded inside the regex itself
('\.[service-to-service zone]') - not valid regex syntax, would never
produce the shown output if actually run. Replaced with the real
command (matching the _URL constant-naming pattern, which doesn't
need the domain at all) and its real output, domain suffix genericized
in output only. Separated the Authentik/Slack/fleetAudit/pfSense
reasoning into prose, not disguised as more grep output.
Step 5 re-audit - real screenshot showing the underlying table data
(50 records, all columns) that the Dashy heatmap is rendered from,
distinct from the two existing rendered-widget screenshots.
The two blocks added earlier this pass showed 'curl | jq' syntax that
was never actually run - the real command was 'curl | python3 -c'
via SSH. The output text was genuinely real either way, but the
command shown wasn't what produced it. Replaced with the literal
command actually executed, re-run fresh to confirm the output still
matches.
Step 5 re-pass under the strengthened evidence gate. The 14-source
count was the doc's own opening framing claim but had no evidence
attached - I'd verified it live earlier this session (grepping the
real function for every upstream host it calls) but never actually
showed the work. Added it.
Step 5 re-pass under the strengthened evidence gate. Both the
app-update duration correction and the Authentik 404 finding were
narrative-only, backed by data I'd already verified live but never
actually shown. Added the real NocoDB row notes as evidence blocks.
Left the other three bullets as-is - the file already has 2 real
screenshots plus these 2 data blocks, judged sufficient rather than
padding every bullet for its own sake.
Live-checked every flow's actual Kestra definition for a Pause task.
Only app-update has one; the other 6 main flows (os-update-vms,
container-update, disk-cleanup, os-update-pbs, os-update-pve,
os-update-pve-verify) run fully automatically with no human checkpoint
at all - their safety net is the automated health-check/rollback path,
not human approval. The doc previously presented check->pause->approve
as the universal pattern for all 13 flows and stated outright that
'nothing gets changed on the fleet without a human explicitly resuming
a paused flow', which was false for the flows that actually change
the fleet. The KB was already correct on this point.
Also replaced the mermaid diagram (broke this repo's established
Nextcloud-SVG convention) with a real diagram showing both patterns,
plus live evidence of the Pause-task check itself.
Live-queried Authentik's API directly: firefly (and emby, not in this
doc's table anyway) are both Proxy Provider only, no native OIDC/OAuth2
provider exists for either. Real count is 10 apps with native OIDC,
not the 11/12 both this doc and the KB previously implied. Same gap
existed in the Authentik/SSO Integration KB note, fixed there too.
Live-checked docker_stacks.yml directly. Total/dockerfile-strategy
counts (41/36) and the pinned_manual holdouts were already accurate.
But docker-socket-proxy and kodi's mariadb service were wrongly listed
as still hitting Docker Hub directly - both actually float a :latest
pull against a dedicated in-fleet mirror stack (docker-socket-proxy-
mirror, mariadb-latest) built via the normal dockerfile pattern. Fixed
the table and documented the real mechanism. Same gap existed in the
KB note, fixed there too.
The rest of the doc consistently says 'browser-facing zone'/'app-to-app
zone' and never states the literal domain suffixes. One evidence code
block leaked both real domains as part of the FQDNs being resolved -
fixed to match the doc's own established pattern. Caught because I'd
only sanitize-checked my own diff, not the whole file - full-file grep
for thesabear.net now confirms zero literal domain mentions.
Live audit found 4 records in the reverse zone, all wrong: 2 malformed
PTR entries (name/value swapped, never actually resolved), 1 orphaned
record from a decommissioned host, 1 correct-looking but misleading
record (proxy host's PTR pointing at one proxied app's name instead of
its own identity). All fixed live on Technitium today, not just
documented. Doc updated to describe the real defect and the fix.
The 6-tier dependency ladder doesn't mirror that doc's trust-boundary
structure (4 boundaries, 3 routing authorities - a different taxonomy).
The 6-tier list only exists in the live code; no public doc documents
it, so the doc now describes it standalone instead of pointing at a
file that doesn't actually contain it.
Same status the disaster-recovery.md conversation already ruled out of
the public repo - operationally true, not employer-facing. KB and
NocoDB job notes keep the real detail; only the public prose changes.
Live-verified against real infrastructure: Kopia's real repo/disk,
PBS's real datastore path, os_update role's real batch/abort behavior
against the actual Ansible code, and the real tag-driven
roles_metadata.yml mechanism behind Layer 1's rebuild claim. Caught a
real KB staleness in the process (PBS datastore usage, 3 weeks stale)
and fixed it properly - removed the point-in-time number entirely
rather than refreshing it, since a data-snapshot doesn't belong in
Trilium at all per the existing content-philosophy rule. Replaced
both mermaid diagrams with real branch/decoupling-shaped Nextcloud
diagrams. All 3 source KB notes stamped.
Found by actually re-checking the running function instead of stopping
at what was already written: each finding has a "Copy Debug Prompt"
button generating a ready-to-paste AI-triage prompt, with a built-in
"don't act without approval" instruction baked into the template
itself. Embedded the real board screenshot (redacted) - dependency
ladder, Wazuh KPIs, Active Issues strip, per-host tile grid - since a
doc about a visual dashboard needs to actually show it, not just
describe it in command output.
Live-verified every specific claim directly against the real Node-RED
function source (not just trusted from the KB): the 104,443-char
function size, all 5 Wazuh finding categories, Vault's live PKI cert
endpoints, the real DNS Security Signals implementation, the
fleetAudit global pattern, the 12-app Slack channel map, the
acknowledge key format and localStorage key name, and the Mustache
triple-brace URL gotcha. Replaced both mermaid diagrams: the flat
data-sources fan-in became a real specifics table (redundant diagram
cut per the diagramming standing process), and the acknowledge-flow
diagram was rebuilt as a real Nextcloud diagram. Stamped both source
KB notes.
Live-verified against the real Technitium server: spot-checked
caddy-only/local-only hosts and Postiz's zero-presence claim via
direct DNS resolution, confirmed the forwarder list directly from the
real dns.config (pfSense + Quad9 only, no Cloudflare), and confirmed
the reverse zone's real /24 scope plus its sparse PTR content via
live reverse lookups - fixed the KB stamp accordingly after an
earlier version of it claimed these weren't re-checked. Replaced the
mermaid zone-routing diagram with a real branch-shaped Nextcloud
diagram. Real command output embedded throughout instead of
paraphrased claims.
Live-verified the container_update role's strategy_dockerfile.yml
task file exists at the exact path the KB names, stamped the source
note. Replaced the mermaid cheap-check flowchart with a real
branch-shaped Nextcloud diagram. Added: the real Vault-backed push
auth mechanism (Gitea PAT -> Vault Agent -> docker login
--password-stdin, same pattern as everywhere else in the fleet), the
real docker manifest inspect command instead of a vague "digest
check", real app names in the holdouts table instead of generic
descriptions (matching how the rest of this repo already names real
apps), the anonymous-volume-pruning detail, and a real (redacted)
screenshot of the actual Gitea package registry showing genuine
self-built container images.
Live-verified against the real Wazuh manager (agent_control -l: 11
active agents matching exactly; docker ps: real manager container
name) and stamped both source KB notes. Replaced both mermaid
flowcharts with real branch-shaped Nextcloud diagrams (alert-handling
gate, compliance-hardening gate). Added real evidence throughout: the
live agent roster, a real synthetic SSH-failure line piped through
wazuh-logtest showing actual decoding/rule-match output (MITRE ATT&CK
+ compliance tags included), and real rule IDs in the runbook table
instead of vague categories. New content: the fleet-wide agent
log-queue-overflow bug found and fixed via a dedicated tuning role,
and the compose-stack-vs-rules version-control split.
Swapped the curl/jq text block for what the doc actually needed:
real screenshots of the live Dashy Fleet Schedule Heatmap widget -
one showing a populated cell clicked open (proving the TIP's
click-to-expand claim is real, not just described), one showing the
"no fixed time" section for the 15 jobs excluded from the grid.
Live-pulled the actual 50-row NocoDB inventory (also caught and fixed
a stale KB fact along the way - the note pointed at the wrong NocoDB
deployment entirely, nocodb-content/automation instead of the real
nocodb/rocky-docker instance). Replaced the vague weight-formula prose
with the real numbers, cut the mermaid diagram that only restated the
prose above it, and added five real stories mined from populating the
table by hand: a three-way monthly collision fix, a cascading
Kopia->ClamAV reschedule, a duration figure corrected 8x after
checking real completed-run data, the heaviest job in the whole fleet
(not a backup), and a 404 that looked like an auth problem and
wasn't. Enriched the pending PBS datastore bug with its real
resolution options and added a second real gap: Technitium's
blocklist auto-update is configured but silently a no-op.
Sequence diagram showing the browser carries every hop itself -
Authentik and the app never talk directly - with only the OAuth
authorize/token exchange crossing from the browser-facing zone into
the service-to-service zone. Built to the public-repo diagram
conventions in CLAUDE.md (labelBackgroundColor + matching rich-text
span, off-line label positioning, line-color-matched labels, legend
instead of a color-explaining caption).
Asset renames (matching the new Nextcloud source naming convention —
doc name first, then description):
- architecture-trust-depth.svg -> architecture-overview-trust-depth.svg
- firewall-docker-bypass.svg -> firewall-and-network-docker-bypass.svg
- secrets-lifecycle.svg -> secrets-and-pki-lifecycle.svg
- trust-boundaries-blast-radius.svg -> readme-trust-boundaries-blast-radius.svg
Updated all 4 markdown references accordingly.
Also refreshed diagram exports (secret-distribution-flows.svg, and the
4 renamed ones) with today's fixes, and folded in the earlier pending
sso-and-identity.md work: precise live-verified native-OIDC app list
(12 apps, not ~9), the Proxmox no-Proxy-Provider exception, real API
evidence for it, and the automation-bot migration story.
Add the Suricata dual-layer detail (DNS lockdown, Vault protection
each enforced independently at two layers) and honest live-backend
churn instead of a flat inventory list. Drop the disabled-LAN-frontend
TIP entirely — not worth keeping in the public doc.
Pulled every pfSense child note (Default Policy, Proxmox Firewall Usage,
VM-Level Firewalld Strategy, WAN Rules, Security Tooling, Failure Impact,
HAProxy, Aliases, DNS & Resolver Config, LAN Rules, NAT, VPN), not just
the originally-scoped subset. Live-verified firewalld defaults, SSH
rich-rule restrictions, and Vault's per-client rules directly against
the running hosts - all matched exactly.
Caught and fixed a real staleness bug in the KB itself: WAN Rules and
Security Tooling (both dated a week old) still described CrowdSec as
active, but the more recent Aliases note (updated yesterday, checked
against pfSense's actual config.xml) confirmed the package has been
fully purged. Fixed both stale notes and the public doc, which had
already inherited the wrong claim.
Folded in the staged PAM defense-in-depth note properly - found the
fuller, better story directly in the Ansible role's source comments
(the bug hit twice, on two different hosts, before the permanent fix).
Added the LAN lockdown narrative this doc was missing, the DNS
bypass-block defense-in-depth story, and the this-week port-alias
conversion. Cut two redundant Mermaid diagrams. Added real evidence:
four pfSense screenshots already validated for architecture-overview.md
reused here since they directly prove this doc's own claims, plus two
real firewall-cmd command outputs from live verification this session.
Source Trilium notes (Authentik/SSO Integration, Gitea Access Model)
live-verified and stamped this session - authentik_api_token confirmed
in vault.yml, automation-bot confirmed as a real non-admin Gitea account
via the API.
Cut the redundant Mermaid diagram (prose right below it said the same
thing). Restored the LDAP/superuser-outpost exception from the source
note as a compact callout rather than a full section, proportionate to
its actual weight. Added the real reasoning for centralizing auth and
the proxy-layer defense-in-depth argument, a table of the ~9 apps
running real native OIDC/SAML on top of the universal Proxy Provider
baseline (corrected after initially misstating it as either/or), the
provider-removal discipline, and two screenshots proving the Write/
Read-only access split instead of just asserting it.
Source Trilium note (Secret Distribution Model) was live-verified and
stamped this session - Certificate Flow, App Secret Flow, Configuration
Distribution Flow, and the technitium_api_token Vault-KV exception all
confirmed against the real hosts, not just trusted from the KB.
Replaced the Mermaid diagram, which incorrectly showed the Configuration
Distribution Flow feeding into Vault directly, with a corrected version:
that flow is a control-plane gate on what Vault Agent renders, not a
peer data flow. Added a real Vault Agent log excerpt proving the full
redeploy chain (restart -> render -> reload_command -> container
recreated) actually happens, and restored the technitium_api_token
exception that was in the source note but missing from this doc.
thesabear