From a63a5dc5891b62a3d3282187593d7446ec0f8d2d Mon Sep 17 00:00:00 2001 From: thesabear Date: Mon, 24 Aug 2026 19:53:34 +0300 Subject: [PATCH] README: add trust-boundary diagram, alphabetize Systems section Adds a real drawio-exported diagram to "Why Four Trust Boundaries, Not One" showing the physical blast-radius split behind the routing- authority claim in the text (previously prose-only). Sanitized - verified no real hostnames/IPs made it into the export. Also reorders all 14 Systems bullets alphabetically for easier scanning. --- README.md | 93 ++++++++++++------------ assets/trust-boundaries-blast-radius.svg | 4 + 2 files changed, 52 insertions(+), 45 deletions(-) create mode 100644 assets/trust-boundaries-blast-radius.svg diff --git a/README.md b/README.md index bf346e3..a4bd409 100644 --- a/README.md +++ b/README.md @@ -39,6 +39,9 @@ Most self-hosted setups have one trust boundary: "you're on the LAN, you're trus This fleet runs **four trust boundaries, each calibrated to who's actually on the other end** — public internet traffic, human browsers on personal devices, machine-provisioned containers, and non-HTTP infra traffic each get a different control, matched to their real risk, not a blanket policy applied everywhere or nowhere. Enforced by **three independent routing authorities** instead of one central router, specifically so a failure or misconfiguration in any single one only ever takes out its own third of the traffic — never all of it at once. It didn't start this way, either: the public proxy used to be its own VM, and the internal service-to-service model used to be a flat, unauthenticated mesh with no boundary at all. Both got replaced deliberately, trading operational simplicity for a smaller blast radius. +![Trust Boundaries and Blast Radius](assets/trust-boundaries-blast-radius.svg) +The three routing authorities are also split across two separate physical nodes, so a hardware failure is contained the same way a routing failure is — never all three at once. + Full breakdown — the boundary-by-boundary reasoning, the routing-authority failure table, all 8 traffic laws: [docs/architecture-overview.md](docs/architecture-overview.md). --- @@ -76,61 +79,61 @@ Solid arrows are hard dependencies — the target has to be reachable for the so ## Systems -### 🔐 Secrets & PKI -A self-hosted Vault cluster is both the certificate authority and the secret store for every host in the fleet — every TLS certificate, every database password, every API key traces back to this one system. Nothing reaches Vault directly; a local agent on each host authenticates itself and renders what that host needs to disk. Replaced an earlier setup where secrets sat in plaintext inside git. A DNS resolver's cert once renewed out of sync and ran expired for 19 days before anyone noticed — the fix wasn't tighter monitoring, it was making that renewal step structurally unable to skip itself again. -→ [docs/secrets-and-pki.md](docs/secrets-and-pki.md) - -### 🔑 Secret Distribution Model -Three separate delivery paths move material out of Vault, purpose-built for what's actually being delivered — a short-lived certificate, an application secret, or a full config file — each with its own contract for what the rest of the fleet does if that specific path fails. Admin and tooling credentials deliberately live in a fourth, completely separate system instead — encrypted directly in git — since they're used by playbooks and operators, not consumed by a running service. -→ [docs/secret-distribution.md](docs/secret-distribution.md) - -### 🪪 SSO & Identity -Authentik, self-hosted, fronts single sign-on fleet-wide — native OIDC where an app supports it, proxy-based forward-auth covering everything else, so apps that were never built with SSO in mind still get it. Every OIDC login crosses reverse-proxy zones mid-handshake, and Git automation runs on a scoped bot account rather than an admin identity. -→ [docs/sso-and-identity.md](docs/sso-and-identity.md) - -### 🧱 Firewall & Network Segmentation -No blanket internal trust anywhere on the network — every real service-to-service flow gets its own explicit rule, and everything not explicitly allowed is logged and dropped by default. Enforced at two layers: a firewall at the network edge, and firewalld locally on every host, each opening only what that specific host actually needs. A few rules go narrower still — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet. -→ [docs/firewall-and-network.md](docs/firewall-and-network.md) - -### 🌐 DNS -Two internal DNS zones split browser-facing traffic from service-to-service traffic — which zone a hostname resolves in is what decides which trust boundary a request enters, not just where it's found. Technitium runs both zones, self-hosted, with direct DNS bypass blocked at the firewall so the split is actually enforced, not just a naming convention. Not every app gets a record in both zones, either: something that's purely a backend service other apps talk to only needs the app-to-app zone, and something purely browser-facing only needs the user-facing one — each app gets exactly the DNS records its real traffic pattern uses. -→ [docs/dns-naming.md](docs/dns-naming.md) - -### 📈 Observability -Every fleet host runs a native metrics-and-logs agent that pushes outward to a central Loki/Prometheus stack, rather than waiting to be scraped — chosen specifically so monitoring never needs inbound access opened through the firewall model to every host it watches. Logs, container output, and DNS query history all land in one searchable place, and a handful of apps go further still, exposing their own internal metrics directly into the same pipeline instead of relying on host-level visibility alone. -→ [docs/observability.md](docs/observability.md) - -### 🛡️ Security Monitoring -A Wazuh agent runs on every host in the fleet, continuously watching file integrity, rootkit behavior, and compliance drift against a CIS-based benchmark. A curated runbook root-causes routine alerts once so they're never re-triaged from scratch, and hardening controls apply fleet-wide with specific, reasoned exceptions rather than uniform enforcement regardless of fit — audit logging, for one, is skipped on every LXC container in the fleet because the kernel capability it needs simply isn't available inside a container, not because it was overlooked. -→ [docs/security-monitoring.md](docs/security-monitoring.md) - -### 🔁 Orchestration -Kestra is the orchestration engine that runs the fleet's scheduled maintenance — OS patching, container updates, app updates, disk cleanup, and internal KB upkeep. Most of it runs fully unattended, but updates to the secrets platform, DNS, and monitoring agent — the systems everything else depends on — wait for an explicit human approval before applying. -→ [docs/orchestration.md](docs/orchestration.md) - -### 🔔 Notifications -Every alert in the fleet — from apps, from hosts, from security events — funnels through one Node-RED instance and one shared formatter before reaching Slack, so a notification reads the same regardless of which of a dozen-plus sources triggered it. A dedicated role puts every host onto the same pipeline for OS-level health, independent of any single application. One shared endpoint handles every app's webhook regardless of shape — a source field where an app provides one, structural fingerprinting of the payload where it doesn't — instead of standing up a dedicated integration per app. -→ [docs/notifications.md](docs/notifications.md) - -### 📦 Self-Hosted Registry & Build Pipeline -Docker images get built and hosted entirely in-fleet on the Git server's own built-in registry — no runtime dependency on Docker Hub or ghcr for the majority of stacks, which have opted in. A weekly scheduled pass checks upstream image digests before committing to a full rebuild, so nothing gets rebuilt unless something upstream actually changed. -→ [docs/registry-and-build.md](docs/registry-and-build.md) +### ⏱️ Cron Scheduler +Every scheduled job across the fleet — orchestration flows, system timers, app-internal schedules — lives in one inventory, visualized as a load heatmap weighted by actual resource impact — a disk-heavy backup job counts for more than a quick health check, not just raw job count — so a new job's time slot can be picked without colliding with existing load. +→ [docs/cron-scheduler.md](docs/cron-scheduler.md) ### 🖥️ Dashy SOC Board One custom-built dashboard aggregates fourteen different data sources — security alerts, logs, metrics, CI health, secrets state, DNS, backups, firewall stats — into a single glanceable board, replacing what used to be fourteen separate tabs to check. Acknowledgement state lives entirely in the browser, no backend to host or secure. Every widget's visual shape matches the shape of its own data — a gauge for a proportion, a nested tree for parent/child hardware, a poster grid for media — instead of one generic card template stretched to fit everything. → [docs/dashy-soc-board.md](docs/dashy-soc-board.md) -### ⏱️ Cron Scheduler -Every scheduled job across the fleet — orchestration flows, system timers, app-internal schedules — lives in one inventory, visualized as a load heatmap weighted by actual resource impact — a disk-heavy backup job counts for more than a quick health check, not just raw job count — so a new job's time slot can be picked without colliding with existing load. -→ [docs/cron-scheduler.md](docs/cron-scheduler.md) +### 🧯 Disaster Recovery & Upgrades +Three independent recovery layers cover three different kinds of loss — a lost host rebuilds from code, VM-level backups cover full-host snapshots, and a separate volume-level backup covers application data specifically. Failure-domain mapping goes down to the disk: one specific volume, if lost, would take out the boot images for two hosts at once — identified and factored into backup priority rather than discovered the hard way. OS-level and application-level upgrades run on entirely separate, deliberately decoupled schedules and safety checks. +→ [docs/disaster-recovery.md](docs/disaster-recovery.md) + +### 🌐 DNS +Two internal DNS zones split browser-facing traffic from service-to-service traffic — which zone a hostname resolves in is what decides which trust boundary a request enters, not just where it's found. Technitium runs both zones, self-hosted, with direct DNS bypass blocked at the firewall so the split is actually enforced, not just a naming convention. Not every app gets a record in both zones, either: something that's purely a backend service other apps talk to only needs the app-to-app zone, and something purely browser-facing only needs the user-facing one — each app gets exactly the DNS records its real traffic pattern uses. +→ [docs/dns-naming.md](docs/dns-naming.md) + +### 🧱 Firewall & Network Segmentation +No blanket internal trust anywhere on the network — every real service-to-service flow gets its own explicit rule, and everything not explicitly allowed is logged and dropped by default. Enforced at two layers: a firewall at the network edge, and firewalld locally on every host, each opening only what that specific host actually needs. A few rules go narrower still — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet. +→ [docs/firewall-and-network.md](docs/firewall-and-network.md) ### 🏗️ Infrastructure as Code Packer builds a hardened base image, Terraform provisions each VM from it, and Ansible configures and continuously re-hardens every host after that — three tools, three separate responsibilities, connected by one tag that threads through all three stages instead of one tool trying to do every job. Terraform's part in that link ends at creation — it writes the tags file once, and that file then drives Ansible independently for the rest of that host's life. → [docs/iac-pipeline.md](docs/iac-pipeline.md) -### 🧯 Disaster Recovery & Upgrades -Three independent recovery layers cover three different kinds of loss — a lost host rebuilds from code, VM-level backups cover full-host snapshots, and a separate volume-level backup covers application data specifically. Failure-domain mapping goes down to the disk: one specific volume, if lost, would take out the boot images for two hosts at once — identified and factored into backup priority rather than discovered the hard way. OS-level and application-level upgrades run on entirely separate, deliberately decoupled schedules and safety checks. -→ [docs/disaster-recovery.md](docs/disaster-recovery.md) +### 🔔 Notifications +Every alert in the fleet — from apps, from hosts, from security events — funnels through one Node-RED instance and one shared formatter before reaching Slack, so a notification reads the same regardless of which of a dozen-plus sources triggered it. A dedicated role puts every host onto the same pipeline for OS-level health, independent of any single application. One shared endpoint handles every app's webhook regardless of shape — a source field where an app provides one, structural fingerprinting of the payload where it doesn't — instead of standing up a dedicated integration per app. +→ [docs/notifications.md](docs/notifications.md) + +### 📈 Observability +Every fleet host runs a native metrics-and-logs agent that pushes outward to a central Loki/Prometheus stack, rather than waiting to be scraped — chosen specifically so monitoring never needs inbound access opened through the firewall model to every host it watches. Logs, container output, and DNS query history all land in one searchable place, and a handful of apps go further still, exposing their own internal metrics directly into the same pipeline instead of relying on host-level visibility alone. +→ [docs/observability.md](docs/observability.md) + +### 🔁 Orchestration +Kestra is the orchestration engine that runs the fleet's scheduled maintenance — OS patching, container updates, app updates, disk cleanup, and internal KB upkeep. Most of it runs fully unattended, but updates to the secrets platform, DNS, and monitoring agent — the systems everything else depends on — wait for an explicit human approval before applying. +→ [docs/orchestration.md](docs/orchestration.md) + +### 🔑 Secret Distribution Model +Three separate delivery paths move material out of Vault, purpose-built for what's actually being delivered — a short-lived certificate, an application secret, or a full config file — each with its own contract for what the rest of the fleet does if that specific path fails. Admin and tooling credentials deliberately live in a fourth, completely separate system instead — encrypted directly in git — since they're used by playbooks and operators, not consumed by a running service. +→ [docs/secret-distribution.md](docs/secret-distribution.md) + +### 🔐 Secrets & PKI +A self-hosted Vault cluster is both the certificate authority and the secret store for every host in the fleet — every TLS certificate, every database password, every API key traces back to this one system. Nothing reaches Vault directly; a local agent on each host authenticates itself and renders what that host needs to disk. Replaced an earlier setup where secrets sat in plaintext inside git. A DNS resolver's cert once renewed out of sync and ran expired for 19 days before anyone noticed — the fix wasn't tighter monitoring, it was making that renewal step structurally unable to skip itself again. +→ [docs/secrets-and-pki.md](docs/secrets-and-pki.md) + +### 🛡️ Security Monitoring +A Wazuh agent runs on every host in the fleet, continuously watching file integrity, rootkit behavior, and compliance drift against a CIS-based benchmark. A curated runbook root-causes routine alerts once so they're never re-triaged from scratch, and hardening controls apply fleet-wide with specific, reasoned exceptions rather than uniform enforcement regardless of fit — audit logging, for one, is skipped on every LXC container in the fleet because the kernel capability it needs simply isn't available inside a container, not because it was overlooked. +→ [docs/security-monitoring.md](docs/security-monitoring.md) + +### 📦 Self-Hosted Registry & Build Pipeline +Docker images get built and hosted entirely in-fleet on the Git server's own built-in registry — no runtime dependency on Docker Hub or ghcr for the majority of stacks, which have opted in. A weekly scheduled pass checks upstream image digests before committing to a full rebuild, so nothing gets rebuilt unless something upstream actually changed. +→ [docs/registry-and-build.md](docs/registry-and-build.md) + +### 🪪 SSO & Identity +Authentik, self-hosted, fronts single sign-on fleet-wide — native OIDC where an app supports it, proxy-based forward-auth covering everything else, so apps that were never built with SSO in mind still get it. Every OIDC login crosses reverse-proxy zones mid-handshake, and Git automation runs on a scoped bot account rather than an admin identity. +→ [docs/sso-and-identity.md](docs/sso-and-identity.md) ## What's running diff --git a/assets/trust-boundaries-blast-radius.svg b/assets/trust-boundaries-blast-radius.svg new file mode 100644 index 0000000..8ce9c2d --- /dev/null +++ b/assets/trust-boundaries-blast-radius.svg @@ -0,0 +1,4 @@ + + + +
pfSense
Edge Firewall + Public HAProxy
Proxmox Node A
day-to-day applications
Internal Caddy
app-to-app zone
Docker Apps
LXC host, most of the fleet's stacks
Proxmox Node B
trust & sensitive infrastructure
Caddy VM
browser zone, mTLS
Vault + Technitium DNS
trust backbone
public traffic (direct, bypasses everything else)
app-to-app traffic
browser traffic
issues certs
issues certs
Three independent routing authorities, split across two physical nodes — a failure in any one only takes out its own third of the traffic.
 \ No newline at end of file