Step 5 re-pass under the strengthened evidence gate. The 14-source
count was the doc's own opening framing claim but had no evidence
attached - I'd verified it live earlier this session (grepping the
real function for every upstream host it calls) but never actually
showed the work. Added it.
Step 5 re-pass under the strengthened evidence gate. Both the
app-update duration correction and the Authentik 404 finding were
narrative-only, backed by data I'd already verified live but never
actually shown. Added the real NocoDB row notes as evidence blocks.
Left the other three bullets as-is - the file already has 2 real
screenshots plus these 2 data blocks, judged sufficient rather than
padding every bullet for its own sake.
Live-checked every flow's actual Kestra definition for a Pause task.
Only app-update has one; the other 6 main flows (os-update-vms,
container-update, disk-cleanup, os-update-pbs, os-update-pve,
os-update-pve-verify) run fully automatically with no human checkpoint
at all - their safety net is the automated health-check/rollback path,
not human approval. The doc previously presented check->pause->approve
as the universal pattern for all 13 flows and stated outright that
'nothing gets changed on the fleet without a human explicitly resuming
a paused flow', which was false for the flows that actually change
the fleet. The KB was already correct on this point.
Also replaced the mermaid diagram (broke this repo's established
Nextcloud-SVG convention) with a real diagram showing both patterns,
plus live evidence of the Pause-task check itself.
Live-queried Authentik's API directly: firefly (and emby, not in this
doc's table anyway) are both Proxy Provider only, no native OIDC/OAuth2
provider exists for either. Real count is 10 apps with native OIDC,
not the 11/12 both this doc and the KB previously implied. Same gap
existed in the Authentik/SSO Integration KB note, fixed there too.
Live-checked docker_stacks.yml directly. Total/dockerfile-strategy
counts (41/36) and the pinned_manual holdouts were already accurate.
But docker-socket-proxy and kodi's mariadb service were wrongly listed
as still hitting Docker Hub directly - both actually float a :latest
pull against a dedicated in-fleet mirror stack (docker-socket-proxy-
mirror, mariadb-latest) built via the normal dockerfile pattern. Fixed
the table and documented the real mechanism. Same gap existed in the
KB note, fixed there too.
The rest of the doc consistently says 'browser-facing zone'/'app-to-app
zone' and never states the literal domain suffixes. One evidence code
block leaked both real domains as part of the FQDNs being resolved -
fixed to match the doc's own established pattern. Caught because I'd
only sanitize-checked my own diff, not the whole file - full-file grep
for thesabear.net now confirms zero literal domain mentions.
Live audit found 4 records in the reverse zone, all wrong: 2 malformed
PTR entries (name/value swapped, never actually resolved), 1 orphaned
record from a decommissioned host, 1 correct-looking but misleading
record (proxy host's PTR pointing at one proxied app's name instead of
its own identity). All fixed live on Technitium today, not just
documented. Doc updated to describe the real defect and the fix.
The 6-tier dependency ladder doesn't mirror that doc's trust-boundary
structure (4 boundaries, 3 routing authorities - a different taxonomy).
The 6-tier list only exists in the live code; no public doc documents
it, so the doc now describes it standalone instead of pointing at a
file that doesn't actually contain it.
Same status the disaster-recovery.md conversation already ruled out of
the public repo - operationally true, not employer-facing. KB and
NocoDB job notes keep the real detail; only the public prose changes.
Live-verified against real infrastructure: Kopia's real repo/disk,
PBS's real datastore path, os_update role's real batch/abort behavior
against the actual Ansible code, and the real tag-driven
roles_metadata.yml mechanism behind Layer 1's rebuild claim. Caught a
real KB staleness in the process (PBS datastore usage, 3 weeks stale)
and fixed it properly - removed the point-in-time number entirely
rather than refreshing it, since a data-snapshot doesn't belong in
Trilium at all per the existing content-philosophy rule. Replaced
both mermaid diagrams with real branch/decoupling-shaped Nextcloud
diagrams. All 3 source KB notes stamped.
Found by actually re-checking the running function instead of stopping
at what was already written: each finding has a "Copy Debug Prompt"
button generating a ready-to-paste AI-triage prompt, with a built-in
"don't act without approval" instruction baked into the template
itself. Embedded the real board screenshot (redacted) - dependency
ladder, Wazuh KPIs, Active Issues strip, per-host tile grid - since a
doc about a visual dashboard needs to actually show it, not just
describe it in command output.
Live-verified every specific claim directly against the real Node-RED
function source (not just trusted from the KB): the 104,443-char
function size, all 5 Wazuh finding categories, Vault's live PKI cert
endpoints, the real DNS Security Signals implementation, the
fleetAudit global pattern, the 12-app Slack channel map, the
acknowledge key format and localStorage key name, and the Mustache
triple-brace URL gotcha. Replaced both mermaid diagrams: the flat
data-sources fan-in became a real specifics table (redundant diagram
cut per the diagramming standing process), and the acknowledge-flow
diagram was rebuilt as a real Nextcloud diagram. Stamped both source
KB notes.
Live-verified against the real Technitium server: spot-checked
caddy-only/local-only hosts and Postiz's zero-presence claim via
direct DNS resolution, confirmed the forwarder list directly from the
real dns.config (pfSense + Quad9 only, no Cloudflare), and confirmed
the reverse zone's real /24 scope plus its sparse PTR content via
live reverse lookups - fixed the KB stamp accordingly after an
earlier version of it claimed these weren't re-checked. Replaced the
mermaid zone-routing diagram with a real branch-shaped Nextcloud
diagram. Real command output embedded throughout instead of
paraphrased claims.
Live-verified the container_update role's strategy_dockerfile.yml
task file exists at the exact path the KB names, stamped the source
note. Replaced the mermaid cheap-check flowchart with a real
branch-shaped Nextcloud diagram. Added: the real Vault-backed push
auth mechanism (Gitea PAT -> Vault Agent -> docker login
--password-stdin, same pattern as everywhere else in the fleet), the
real docker manifest inspect command instead of a vague "digest
check", real app names in the holdouts table instead of generic
descriptions (matching how the rest of this repo already names real
apps), the anonymous-volume-pruning detail, and a real (redacted)
screenshot of the actual Gitea package registry showing genuine
self-built container images.
Live-verified against the real Wazuh manager (agent_control -l: 11
active agents matching exactly; docker ps: real manager container
name) and stamped both source KB notes. Replaced both mermaid
flowcharts with real branch-shaped Nextcloud diagrams (alert-handling
gate, compliance-hardening gate). Added real evidence throughout: the
live agent roster, a real synthetic SSH-failure line piped through
wazuh-logtest showing actual decoding/rule-match output (MITRE ATT&CK
+ compliance tags included), and real rule IDs in the runbook table
instead of vague categories. New content: the fleet-wide agent
log-queue-overflow bug found and fixed via a dedicated tuning role,
and the compose-stack-vs-rules version-control split.
Swapped the curl/jq text block for what the doc actually needed:
real screenshots of the live Dashy Fleet Schedule Heatmap widget -
one showing a populated cell clicked open (proving the TIP's
click-to-expand claim is real, not just described), one showing the
"no fixed time" section for the 15 jobs excluded from the grid.
Live-pulled the actual 50-row NocoDB inventory (also caught and fixed
a stale KB fact along the way - the note pointed at the wrong NocoDB
deployment entirely, nocodb-content/automation instead of the real
nocodb/rocky-docker instance). Replaced the vague weight-formula prose
with the real numbers, cut the mermaid diagram that only restated the
prose above it, and added five real stories mined from populating the
table by hand: a three-way monthly collision fix, a cascading
Kopia->ClamAV reschedule, a duration figure corrected 8x after
checking real completed-run data, the heaviest job in the whole fleet
(not a backup), and a 404 that looked like an auth problem and
wasn't. Enriched the pending PBS datastore bug with its real
resolution options and added a second real gap: Technitium's
blocklist auto-update is configured but silently a no-op.
The zone-band labels spelled out *.local.thesabear.net and
*.caddy.thesabear.net literally - the same internal zone naming
pattern already sanitized out of this doc's prose, missed on the
diagram itself. Replaced with generic zone names, no domain.
Sequence diagram showing the browser carries every hop itself -
Authentik and the app never talk directly - with only the OAuth
authorize/token exchange crossing from the browser-facing zone into
the service-to-service zone. Built to the public-repo diagram
conventions in CLAUDE.md (labelBackgroundColor + matching rich-text
span, off-line label positioning, line-color-matched labels, legend
instead of a color-explaining caption).
Asset renames (matching the new Nextcloud source naming convention —
doc name first, then description):
- architecture-trust-depth.svg -> architecture-overview-trust-depth.svg
- firewall-docker-bypass.svg -> firewall-and-network-docker-bypass.svg
- secrets-lifecycle.svg -> secrets-and-pki-lifecycle.svg
- trust-boundaries-blast-radius.svg -> readme-trust-boundaries-blast-radius.svg
Updated all 4 markdown references accordingly.
Also refreshed diagram exports (secret-distribution-flows.svg, and the
4 renamed ones) with today's fixes, and folded in the earlier pending
sso-and-identity.md work: precise live-verified native-OIDC app list
(12 apps, not ~9), the Proxmox no-Proxy-Provider exception, real API
evidence for it, and the automation-bot migration story.
Root cause was the edge's own labelBackgroundColor style property,
left enabled with no explicit value in the app, distinct from the
label's HTML text styling entirely. Set it explicitly to #1C1D1E,
then stripped a competing plain-white background layer draw.io's
export was still injecting on top of it.
draw.io's export was auto-injecting its own adaptive-background CSS
onto edge labels regardless of source content, with a fallback that
didn't match the page. Stripped background-color entirely so labels
sit directly on the page background with nothing to mismatch.
Nextcloud's draw.io export was inconsistently applying its own
page-level adaptive-background variable to some edge labels instead
of the authored value, making them render darker than the rest.
Post-processed the exported SVG to normalize every label to the same
background value.
Add the Suricata dual-layer detail (DNS lockdown, Vault protection
each enforced independently at two layers) and honest live-backend
churn instead of a flat inventory list. Drop the disabled-LAN-frontend
TIP entirely — not worth keeping in the public doc.
Pulled every pfSense child note (Default Policy, Proxmox Firewall Usage,
VM-Level Firewalld Strategy, WAN Rules, Security Tooling, Failure Impact,
HAProxy, Aliases, DNS & Resolver Config, LAN Rules, NAT, VPN), not just
the originally-scoped subset. Live-verified firewalld defaults, SSH
rich-rule restrictions, and Vault's per-client rules directly against
the running hosts - all matched exactly.
Caught and fixed a real staleness bug in the KB itself: WAN Rules and
Security Tooling (both dated a week old) still described CrowdSec as
active, but the more recent Aliases note (updated yesterday, checked
against pfSense's actual config.xml) confirmed the package has been
fully purged. Fixed both stale notes and the public doc, which had
already inherited the wrong claim.
Folded in the staged PAM defense-in-depth note properly - found the
fuller, better story directly in the Ansible role's source comments
(the bug hit twice, on two different hosts, before the permanent fix).
Added the LAN lockdown narrative this doc was missing, the DNS
bypass-block defense-in-depth story, and the this-week port-alias
conversion. Cut two redundant Mermaid diagrams. Added real evidence:
four pfSense screenshots already validated for architecture-overview.md
reused here since they directly prove this doc's own claims, plus two
real firewall-cmd command outputs from live verification this session.
Source Trilium notes (Authentik/SSO Integration, Gitea Access Model)
live-verified and stamped this session - authentik_api_token confirmed
in vault.yml, automation-bot confirmed as a real non-admin Gitea account
via the API.
Cut the redundant Mermaid diagram (prose right below it said the same
thing). Restored the LDAP/superuser-outpost exception from the source
note as a compact callout rather than a full section, proportionate to
its actual weight. Added the real reasoning for centralizing auth and
the proxy-layer defense-in-depth argument, a table of the ~9 apps
running real native OIDC/SAML on top of the universal Proxy Provider
baseline (corrected after initially misstating it as either/or), the
provider-removal discipline, and two screenshots proving the Write/
Read-only access split instead of just asserting it.
Edge-label captions were using an ambient light-dark() background that
didn't match Gitea's actual dark theme. Fixed to a plain rgb(13,17,23),
matching the corrected CLAUDE.md diagram convention.
Source Trilium note (Secret Distribution Model) was live-verified and
stamped this session - Certificate Flow, App Secret Flow, Configuration
Distribution Flow, and the technitium_api_token Vault-KV exception all
confirmed against the real hosts, not just trusted from the KB.
Replaced the Mermaid diagram, which incorrectly showed the Configuration
Distribution Flow feeding into Vault directly, with a corrected version:
that flow is a control-plane gate on what Vault Agent renders, not a
peer data flow. Added a real Vault Agent log excerpt proving the full
redeploy chain (restart -> render -> reload_command -> container
recreated) actually happens, and restored the technitium_api_token
exception that was in the source note but missing from this doc.
Publishing a ranked list of a live system's own weak points - including
explicitly naming the "single highest-value target" - is a real security
problem for a public repo, not a portfolio concern. Removed the section
and the matching stat-table row. Left the fixed cert-renewal incident in
place since there's nothing exploitable in a resolved issue.
Access model now covers the narrowly-scoped read-only utility policies
(ansible sync roles, metrics-reader, pki-reader) alongside the
host-scoped svc-* pattern - the fuller picture confirmed live against
Vault's actual ACL policy list, not just the original narrower claim.
Added two real screenshots: the live policy list, and an actual audit
log entry showing a KV read with every secret value as hmac-sha256:...
instead of plaintext - proof for the audit-trail claim instead of just
an assertion of it.
Removed the before/after plaintext-secrets Mermaid diagram (the bullets
right below it already said the same thing more clearly) and the
Architecture flowchart (its per-host labels were silently missing
registry-token and Hawser-secret details that didn't fit in the boxes).
Replaced the flowchart with an accurate, complete "who gets what" table
sourced directly from Trilium, one sentence for the proxy-routing fact
it was trying to show, and a new sequence diagram for the actual secret
lifecycle (auth -> render -> read -> async rotation) - a genuinely
different kind of diagram since that's process/time-ordered information
a table can't represent well.
Also restored real specificity Trilium had that the doc had flattened
into vague language: the cert-renewal row now names Technitium
specifically and states the actual incident (an expired cert silently
served for 19 days before the auto-rebuild hook existed), and the audit
trail row now states the real rotation policy (weekly, 8 generations).
Removed the dependency-graph Mermaid diagram (had backwards Vault
dependency arrows, contradicted the documented ingress model) and the
four-boundary Mermaid diagram (fully redundant with the table below it).
Verified every remaining claim against Trilium's Traffic Flow Model and
fixed two that didn't hold up: the DNS-bypass failure behavior overstated
an unconditional SSO bypass, and the notifications failure behavior had
an unsourced claim about backlog retry.
Added the real story behind the trust-boundary hardening (three stages
of the same "trusted by default" flaw getting found and closed, not a
changelog of dated facts), a new trust-depth diagram showing that being
"inside the network" still enforces its own internal boundaries, and
real pfSense screenshots proving the LAN lockdown rules actually exist
in production, not just in the doc.
Two real, working scripts: an Ansible playbook run-menu (the same
pattern used identically across both the fleet-config and infra-
provisioning repos), and cuesplit, a frame-accurate FLAC/.cue
splitting utility. Sanitized paths only, logic untouched.
README gets a new "Scripts & Tooling" section pointing at it, placed
right after "What's Running" alongside the other repo-structure
pointer.
Adds the two Dockerfiles nextcloud's compose build: contexts were
already referencing but never had (custom smbclient support for the
main container, a documentserver base for euro-office) - both had
their real internal-CA-trust mechanism replaced with a comment
explaining what's there and why, per the standing rule against
publishing that detail.
Also fixes a compose bug: kodi-ma-bridge's `environment:` key had
been dropped, leaving KEY=value pairs nested under `env_file:` (which
expects file paths, not inline vars) - would have made docker-compose
try to open a file literally named `TZ=${TZ}`.
Rest of this commit is manual cleanup (comment tidying, dropped
personal media-library volume mounts, a couple of .env comment
trims).
Replaced 29 host-port values across 27 stacks with the app's own
well-known default (read directly from each compose file's own
container-side port mapping) instead of the arbitrary port numbers
this fleet actually assigns to avoid collisions across ~44 services
on one host. More recognizable to a reader, and removes the last
fleet-specific detail these files carried.
Also removes a stale adminer service from n8n's stack entirely
(compose service + .env var) - it's no longer actually deployed.
Adds cloudcommander and it-tools .env files. it-tools' private .env
had two real domains (it-tools.local.thesabear.net,
it-tools.thesabear.net) that the public compose file doesn't
reference at all, so dropped along with the unused DOCKER var.
This completes .env/.env.secrets.ctmpl coverage across every
category in stacks/ (identity-and-security, platform, media,
productivity, dev-tools) - every var cross-checked against its
actual public compose file throughout.
Adds real Vault Agent secret templates and non-secret env vars for
all 9 productivity stacks. Sanitized several real leaks: a real
residential public IP (ghost's UMAMI_IGNORE_IP), real personal
domains/email used across joplin/n8n/ghost/paperless-ngx, and a
real embedded OAuth client ID in paperless-ngx's private OIDC config
(dropped entirely - the public compose file doesn't wire up OIDC at
all, so none of that config was needed).
Also dropped n8n secrets referencing the out-of-scope Content
Pipeline system (nocodb-content, Postiz), and did not carry over a
real Redis password found sitting in plaintext in nextcloud's
private .env (unused by the public compose anyway).
Every var cross-checked against what its actual public compose file
references before writing.
Adds real Vault Agent secret templates and non-secret env vars for
all 15 media stacks. Sanitized real leaks found in the source data:
kodi's real username and real hostnames+IPs in KODI_INSTANCES,
tdarr's real IP and hostname reference, immich's real username -
replaced with generic values (IPs use the RFC 5737 documentation
range, 192.0.2.0/24, rather than another real-looking address).
Every var cross-checked against what its actual public compose file
references before writing, learned from the platform batch.
Adds real Vault Agent secret templates and non-secret env vars for
14 platform stacks (caddy, caddy-internal, dashy, docker-socket-
proxy, dockhand, gitea, grafana-stack, hawser, kestra, kopia,
mosquitto, nocodb, node-red, speedtest-tracker), sanitized per the
kv/host/... convention established for identity-and-security.
Also fixes 6 variable-name mismatches discovered while cross-checking
every new .env/.ctmpl against what its actual public compose file
references (the public compose files were already adapted from the
private originals during an earlier reorg pass, so names don't always
match 1:1): authentik (PG_USER/PG_DB/PG_PASS), gitea (added
GITEA_DOMAIN/GITEA_ROOT_URL, dropped an unused duplicate-instance pair
of vars), grafana-stack (GRAFANA_ADMIN_PASSWORD), kestra (added
USERNAME/PASSWORD), kopia (SECRET_PASSWORD), nocodb (JWT_SECRET).
Also dropped an unused DOCKER var across most stacks where the
compose file hardcodes its path instead of referencing it.
Adds the real Vault Agent secret-rendering templates and non-secret
env vars for authentik, vaultwarden, and wazuh - sanitized (real
domain/email/SMTP provider genericized, kv/<host>/... paths use a
generic 'host' placeholder instead of the real hostname). This is
the standing pattern for the rest of stacks/ going forward.
Removed: ollama (AI-related concerns), emby/qbittorrent/prowlarr
(media-download concerns), bookstack (being deprecated in the
homelab), and mariadb-latest/mysql-8/postgres-18-alpine/valkey-9.0-
alpine (build-only base-image factories - nothing connects to them,
they add no portfolio value on their own; real per-app DB usage,
e.g. Kodi's own mariadb service, already lives in that app's stack
folder and is unaffected).
Stack count updated 53 -> 44 in both README references.
- Heading capitalization consistent Title Case throughout
- Moved the three personal sections (What I Have Explored, What I'm
Focused On, Fun Fact) from between the two strongest technical
sections to after Systems/What's Running, as a closing note -
keeps momentum through the trust-boundary story into the diagram
and full Systems breakdown, and gives the page a real closing beat
instead of ending on a bare stack-count line
- Split the dense opening bio paragraph into two at its natural seam
Adds a real drawio-exported diagram to "Why Four Trust Boundaries,
Not One" showing the physical blast-radius split behind the routing-
authority claim in the text (previously prose-only). Sanitized -
verified no real hostnames/IPs made it into the export.
Also reorders all 14 Systems bullets alphabetically for easier
scanning.
Fixed one real inaccuracy (Gitea registry scope, Secret Distribution
path count) and replaced generic descriptions with true, specific
detail across all 14 Systems bullets - real incidents, real design
decisions, concrete numbers - each checked against Trilium and, where
relevant, the live Ansible roles rather than trusted at face value.
Also stages a verified-but-unintegrated PAM hardening finding in
docs/firewall-and-network.md for that file's own upcoming audit pass.
1.15em rendered fine (sanitizer allow-list is working, confirmed by
the stat-table numbers) but was too subtle a jump to actually look
bigger next to body text. 1.4em matches the visual weight of the
stat-strip numbers.
Gitea's default sanitizer strips style="" from every element; added a
markup.sanitizer allowlist for style on common tags (app.ini, this
Gitea instance) and restarted the service. Now that it's actually
respected: table centered via style="margin:0 auto" (the deprecated
align attribute never reliably did this), and the app-name font-size
bump from the last commit should finally show up too.
Plain markdown bold has no size control. Switched each app name to
<strong style="font-size:1.15em;"> so it stands out from the
description text instead of just being the same size in bold.
align="center" and a div wrapper both failed to actually center the
table under Gitea's markdown CSS. Rather than keep guessing, dropped
the centering attempt - table stays left-aligned with its per-cell
alignment intact.
align="center" on <table> itself gets ignored - Gitea's markdown CSS
forces tables to display:block for horizontal scroll, which breaks
that attribute. Wrapping in <div align="center"> actually centers it.
Table wasn't centered on the page (only text inside cells was) - added
align=\"center\" to the table itself. Icons were arbitrary; replaced
with ones that actually trace back to the related system's own icon
elsewhere in this doc (Docker whale for compose stacks, the brick wall
from Firewall & Network Segmentation for trust boundaries, the same
cycle icon Orchestration uses for automated maintenance flows).
Both paragraphs made the same 'this isn't a hobby, it's a platform'
point and duplicated tech mentions. Merged into one section: a
bulleted list naming the real technology behind every category
(Vault for secrets, Kestra for orchestration, Wazuh+Dashy for the
SOC, etc.) instead of one dense paragraph or abstract category names.
Caption claimed all dashed arrows were telemetry reporting, but two
of them ('proxies', 'auth') are traffic-routing relationships, not
telemetry - the old caption was written for the full 20-edge version
and never updated when the diagram got simplified.
No more Mermaid, per standing rule - the architecture diagram is now
a real image (assets/architecture-overview.svg), built and exported
from the Nextcloud Diagramming app, fetched via WebDAV and validated
as well-formed XML before committing. Ends the rendering-gap chase
entirely since there's no live renderer involved anymore, just a
static image.
Restructured so the edge proxy feeds forward into the app host
(EdgeProxy -.proxies.-> Apps) instead of the app host and SSO
pointing backward into the proxy - matches the real traffic
direction (browser hits the proxy first, which routes to the app)
and removes the long rank-spanning edges that were likely behind the
rendering gap. Full node set restored, colors restored.
7 nodes, strictly tree-shaped, no diamond merges (removed DNS-->Vault
and the EdgeProxy/SSO/InternalProxy branch entirely). If gap is gone,
the rejoining-path topology is the trigger, not raw node count.
Diagnostic bisection - if gap reappears here, it's a size/complexity
threshold. If not, the trigger is specifically in the observability
fan-in portion I removed for this test.
Diagnostic only - not a real content change. If the gap disappears
with this minimal diagram, it's genuinely about the real diagram's
content and I can bisect from here. If the gap is still there, it's
not about the diagram at all.
None of edge reduction, classDef removal, or multi-line label removal
fixed the gap - restoring the full diagram content since there's no
reason to keep it degraded for a fix that didn't work.
Compared node labels between this diagram and the non-gapped
architecture-overview.md one - this had 5 two-line labels, that one
only 2. Reduced to 1 remaining (App Host) to test whether multi-line
label height is what's breaking the container size calculation.
The docs/architecture-overview.md diagram has no custom colors and
renders with no gap; this one had classDef/class styling and did.
Real comparative evidence, not a blind guess this time - testing
whether custom fill/stroke styling is what's breaking Gitea's
container height calculation.
8 dotted 'reports to' edges converging on 2 nodes was likely forcing
Mermaid's layout engine to reserve extra vertical space for edge
routing in the LR diagram, leaving a large blank area below the
visible nodes. Reduced to 2 representative edges - can't verify the
actual render myself, this is a targeted guess to test.
thesabear