From a77e3075bb63f5b84936008a4490de7e3e70492d Mon Sep 17 00:00:00 2001 From: thesabear Date: Sat, 22 Aug 2026 12:50:34 +0300 Subject: [PATCH] docs: restructure architecture-overview.md around one central story Leads with the hardest actual decision (4 trust boundaries, each calibrated to a different risk level, enforced by 3 separate routing authorities) given full narrative depth instead of flat template treatment. The other 6 traffic laws move to a compact reference table linking to their own dedicated docs, avoiding redundancy with content that already lives elsewhere in more depth. --- docs/architecture-overview.md | 126 ++++++++-------------------------- 1 file changed, 30 insertions(+), 96 deletions(-) diff --git a/docs/architecture-overview.md b/docs/architecture-overview.md index 618d5b0..ec17e4d 100644 --- a/docs/architecture-overview.md +++ b/docs/architecture-overview.md @@ -1,7 +1,7 @@ # πŸ“ Architecture Overview > [!NOTE] -> This document defines the invariant movement of traffic between infrastructure components β€” directional flow and authority boundaries, not configuration syntax, ports, or IP addresses. Eight distinct traffic categories, each stated as a Law (what must always be true), a Flow (the actual path), and a Failure Behavior (what happens when the target is unreachable). +> Most homelabs have one trust boundary: "inside the LAN = trusted." This one has four, each calibrated to a different actual risk level rather than applying maximum security everywhere or none anywhere β€” that calibration is the hardest and most interesting decision in the whole architecture, and the rest of this doc builds out from it. ## Dependency graph @@ -45,101 +45,13 @@ flowchart LR ``` > [!TIP] -> Solid arrows are real dependencies β€” the target has to be reachable and healthy for the source to function. Dotted arrows are asynchronous reporting, not a blocking dependency β€” the distinction Law 5 below makes explicit. +> Solid arrows are real dependencies β€” the target has to be reachable and healthy for the source to function. Dotted arrows are asynchronous reporting, not a blocking dependency. -## The eight traffic laws +## The hardest decision: how many trust boundaries is enough? -### 1. Ingress β€” two independent authorities +The easy answers are both wrong. **One boundary** ("you're on the LAN, you're trusted") is what most self-hosted setups actually run, and it means a single compromised container can reach everything else with no further resistance. **Maximum boundaries everywhere** (mTLS on every single connection, no exceptions) sounds safer but isn't free β€” every hop gets slower, every service needs its own client certificate lifecycle, and the operational burden of *that* is exactly the kind of complexity that gets quietly disabled under deadline pressure, which is worse than never having built it. -**Law**: Public traffic and LAN-facing user traffic terminate at different places and are never conflated. - -| | Public | LAN-facing (user apps) | -|---|---|---| -| **Path** | Client β†’ WAN β†’ Edge Firewall β†’ Backend | Client β†’ LAN β†’ User-Facing Proxy (mTLS + SSO) β†’ Backend | -| **TLS terminates at** | The edge firewall's own reverse-proxy package | The user-facing proxy | -| **Enforcement** | No backend is ever directly exposed to WAN β€” only HTTPS/HTTP and one admin SSH port pass the WAN interface | Mutual TLS (client cert required) + SSO | - -**Failure behavior**: if the edge firewall fails, public access is unavailable and everything else keeps running. If the user-facing proxy fails, all browser-facing access is unavailable and everything else keeps running. The two failures are structurally independent β€” one can't cascade into the other. - -### 2. East-West β€” two genuinely distinct patterns - -**Law**: App-to-app HTTP/S traffic and non-HTTP infra-to-infra traffic are not interchangeable, and neither uses a flat unauthenticated mesh (see "what changed," below). - -| Pattern | Path | Trust | -|---|---|---| -| **(a) App-to-app HTTP/S** | Service β†’ internal proxy zone β†’ trusted-IP bypass *or* SSO forward-auth β†’ Service | IP-allowlist based (docker bridge subnets + every fleet host), not mTLS β€” mTLS is reserved exclusively for Law 1's LAN-facing ingress | -| **(b) Infra-to-infra, non-HTTP** | Service β†’ firewall rule (explicit per-flow allow) β†’ Service, direct, no proxy involved | Metrics/logs, MQTT, security-agent reporting, NTP, Vault Agent auth all take this path | - -**Failure behavior**: if internal DNS fails, app-to-app resolution fails and that path breaks with it. Direct IP communication may still work for pattern (a) but bypasses the SSO gate entirely if the source IP happens to be trusted β€” a real, accepted risk surface, not an oversight. Core service runtime doesn't depend on the public-facing edge (Law 1) at all β€” completely separate plane. - -### 3. Secrets β€” one hop, never direct - -**Law**: Applications do not communicate directly with Vault. - -**Flow**: Service β†’ local Vault Agent β†’ Vault Server. Vault Agent performs AppRole authentication and token renewal; in practice it renders local files (an env file consumed via Compose, or a full rendered config file where multi-line config is needed) that the application reads. Applications never call the Vault API themselves, and Vault is never exposed directly to application containers. - -**Failure behavior**: if Vault becomes unavailable, secret renewal stops but services continue operating until their current token expires β€” no immediate disruption, a slow-building one only if the outage outlasts every live token's TTL. - -### 4. DNS β€” resolution determines the routing plane - -**Law**: Which internal DNS zone a hostname resolves in determines which traffic plane (Law 1 or Law 2) a request enters. - -Full model β€” the two zones, the enforcement that makes this a real boundary rather than a convention, and the audit that caught real drift in how it was documented: **[`docs/dns-naming.md`](dns-naming.md)**. - -### 5. Observability β€” asynchronous, never inline - -**Law**: Logs and metrics push asynchronously to monitoring infrastructure. Monitoring systems are not inline with application traffic. - -**Flow**: Service β†’ observability agent β†’ Monitoring (logs, metrics). Service β†’ security agent β†’ Security Monitoring. Every managed host runs both agents, enforced by automation that self-heals on drift rather than being applied once and left alone. - -> [!IMPORTANT] -> Monitoring infrastructure does not proxy or mediate application traffic, and service runtime does not depend on monitoring availability β€” this is why these are the dotted, not solid, arrows in the dependency graph above. **Failure behavior**: if monitoring fails, log/metric collection stops; application services continue operating normally, completely unaffected. - -Full detail on what's actually collected and why push beats pull here specifically: **[`docs/observability.md`](observability.md)** and **[`docs/security-monitoring.md`](security-monitoring.md)**. - -### 6. Notifications β€” two branches, one non-critical destination - -**Law**: Notifications converge on one flow and are non-critical to service operation β€” neither branch is part of any service's runtime dependency chain. - -| Branch | Path | -|---|---| -| **MQTT** | Service β†’ local notify script β†’ message broker (per-host topic) β†’ central subscriber | -| **HTTP webhook** | App β†’ shared webhook endpoint (App Router + per-app mapper) *or* a dedicated endpoint for non-JSON payloads β†’ same downstream formatter | - -Both branches converge on the same formatting step before delivery, but reach it differently β€” the MQTT branch derives its destination channel from the topic name, the webhook branch's per-app mapper functions set it explicitly. **Failure behavior**: notification failures never impact core workloads β€” this is event-driven, not persistent inline execution, so a dropped notification is lost, not queued against a retry that could back up into the sender. - -Full mechanics: **[`docs/notifications.md`](notifications.md)**. - -### 7. CI/CD β€” deployment traffic is separate from runtime traffic - -**Law**: Build and deploy activity never sits inline with application runtime. - -**Flow**: Scheduler (weekly) β†’ build automation β†’ self-hosted registry β†’ target host redeploy. - -**Invariants**: CI/CD components always initiate connections outward, never accept inbound triggers from the apps they build. Runtime services do not depend on CI/CD availability. Deployment failure doesn't affect already-running services β€” enforced by the redeploy step's own health-check/rollback logic, not just incidentally true. This is schedule- and upstream-digest-triggered, not push-triggered β€” editing a stack's own build file doesn't by itself trigger a rebuild until the next scheduled sweep finds the upstream image has actually changed. - -Full mechanics and the reasoning behind schedule-triggered over push-triggered: **[`docs/registry-and-build.md`](registry-and-build.md)**. - -### 8. Management β€” administrative access is its own separate plane - -**Law**: Administrative access is explicitly controlled and restricted to approved management interfaces, separate from every other plane above. - -**Flow**: Administrator β†’ automation host β†’ a dedicated, key-based service identity β†’ target host, on a non-standard port. - -**Invariants**: management interfaces are restricted to approved, non-standard ports fleet-wide, never the service's default port. Access is limited by explicit per-host-pair firewall rules sourced from the automation host only β€” not a blanket LAN allow. Administrative access originates only from the automation host, via one dedicated identity, never from arbitrary internal sources. Application services do not depend on management interfaces being reachable. - -**Failure behavior**: if management access fails, administration becomes unavailable but every running service continues operating normally β€” the same "control plane failure doesn't touch data plane" property that holds across every law above. - -## Global architectural laws - -These apply across every category above, not just one: - -| Law | Statement | -|---|---| -| **Routing authority** | Three routing authorities exist, each scoped to a distinct plane β€” the edge firewall (public ingress), the user-facing proxy (LAN ingress), the internal proxy zone (app-to-app east-west). No internal service ever acts as its own router. | -| **Trust boundary** | Trust is segmented across four boundaries: WANβ†’edge (public), LAN browsersβ†’user-facing proxy (mTLS+SSO), app-to-appβ†’internal proxy (allowlist+SSO), LAN infra-to-infraβ†’firewall rule (explicit per-flow). Backend services trust proxy headers, never raw client IP, on any path. | -| **Plane separation** | Two structurally independent traffic planes β€” Ingress (public and LAN-facing sub-paths) and East-West (app-to-app and infra-to-infra sub-paths) β€” with no crossover between them. | -| **Observability independence** | Observability never sits inline in any request path; it's asynchronous and push-based, full stop (Law 5). | +The actual answer here is **four boundaries, each doing a different job**: ```mermaid flowchart LR @@ -149,14 +61,36 @@ flowchart LR Infra["LAN infra-to-infra"] -->|per-flow rule| HostFW["Host firewall"] ``` -## Two real architecture changes, not a static design +| Boundary | Who's on the other end | Control | Why this control, not another | +|---|---|---|---| +| WAN β†’ edge | Anyone on the internet | Public reverse proxy, firewall-filtered | The only boundary facing genuinely untrusted traffic β€” gets the strictest, most isolated treatment | +| Browsers β†’ user-facing proxy | A human, on a device that could be compromised, phished, or simply lost | Mutual TLS (client cert required) + SSO | The highest-value target for credential theft β€” a stolen password alone isn't enough without the client cert too | +| App-to-app | A container that's already inside the network, already provisioned by the same pipeline as everything else | IP-allowlist + SSO fallback, no mTLS | Full mTLS here would mean managing certificate lifecycle for every internal service-to-service call β€” real overhead for traffic that's already constrained to known, provisioned sources. The risk profile doesn't justify the same control as human-operated browsers. | +| LAN infra-to-infra | Non-HTTP traffic β€” metrics, logs, NTP, agent reporting | Explicit per-flow firewall rule, no proxy at all | Can't go through a reverse proxy β€” it isn't HTTP. The firewall itself is the only layer that *can* enforce this one. | + +**Three separate routing authorities enforce these** β€” the edge firewall, the user-facing proxy, and the internal proxy zone β€” rather than one central router handling everything. That's a deliberate rejection of the simpler single-router design: one router is one thing to hardening-focus on and one thing that takes everything down if it's compromised or misconfigured. Three means a mistake in one router's config, or a full outage of one of them, has a blast radius bounded to the traffic it alone handles β€” the failure-behavior tables throughout this doc all trace back to this one decision. > [!IMPORTANT] -> This shape isn't how the fleet started β€” it's evolved twice, deliberately. The public-facing reverse proxy used to run as its own dedicated VM; it was consolidated directly onto the edge firewall as a package instead, removing an entire host from the public-ingress path without losing any functionality β€” one less thing that can fail, one less thing to patch and monitor independently. And Law 2's internal service-to-service model used to be a flat, unauthenticated mesh network with no trust boundary at all; it was fully replaced by the two-zone split now described in [`docs/dns-naming.md`](dns-naming.md), trading a simpler-but-weaker model for one with a real trust boundary between browser-facing and app-to-app traffic. Both changes reduced the attack surface at the cost of more moving parts to reason about β€” a tradeoff made deliberately, not stumbled into. +> **This wasn't the starting design β€” it was built up over two real, deliberate changes.** The public-facing reverse proxy used to run as its own dedicated VM; it was consolidated directly onto the edge firewall as a package instead, removing an entire host from the public-ingress path without losing any functionality. And the app-to-app boundary used to not exist at all β€” internal service traffic ran over a flat, unauthenticated mesh network with no trust boundary. It was replaced by the two-zone model now described in [`docs/dns-naming.md`](dns-naming.md). Both changes added real operational complexity in exchange for a smaller blast radius β€” accepted deliberately, not stumbled into, and the kind of decision that only looks obvious in hindsight. + +## The other six traffic laws + +Each of these gets its own full mechanism doc elsewhere in this repo β€” this table is the map, not the territory: + +| # | Law | Failure behavior | +|---|---|---| +| 3. Secrets | Applications never talk to Vault directly β€” one hop through a local agent only. See [`docs/secrets-and-pki.md`](secrets-and-pki.md). | Vault down β†’ secret renewal stops, services run until their current token expires. No immediate disruption. | +| 4. DNS | Which internal zone a hostname resolves in determines which trust boundary (above) a request enters. See [`docs/dns-naming.md`](dns-naming.md). | DNS down β†’ app-to-app resolution fails; direct-IP fallback exists but silently bypasses the SSO gate. | +| 5. Observability | Logs and metrics push asynchronously β€” never inline with application traffic, full stop. See [`docs/observability.md`](observability.md) and [`docs/security-monitoring.md`](security-monitoring.md). | Monitoring down β†’ collection stops; every application service keeps running, completely unaffected. | +| 6. Notifications | Two independent branches (message-queue and HTTP webhook) converge on one formatter; neither is part of any service's runtime dependency chain. See [`docs/notifications.md`](notifications.md). | A dropped notification is lost, not retried into a backlog that could pressure the sender. | +| 7. CI/CD | Build/deploy traffic never sits inline with runtime; schedule-triggered, not push-triggered. See [`docs/registry-and-build.md`](registry-and-build.md). | A failed deploy doesn't affect already-running services β€” enforced by the redeploy step's own rollback logic. | +| 8. Management | Administrative access is its own separate plane β€” one dedicated identity, non-standard ports, explicit per-host firewall rules from one source only. | Management access down β†’ administration unavailable; every running service is completely unaffected. | + +The pattern repeats across all eight laws, not just the trust-boundary story above: **control-plane failure never takes down the data plane.** Every "failure behavior" row above ends the same way β€” the thing that broke stops working, and everything else keeps running regardless. That's not a coincidence; it's the same design principle applied consistently everywhere, not just to network trust. ## What this map deliberately excludes -Not a firewall rule listing, not a port inventory, not a service configuration reference β€” those live in [`docs/firewall-and-network.md`](firewall-and-network.md) and the other docs linked throughout each law above. This is strictly the stable, directional shape of how trust and traffic actually move β€” the eight laws above are meant to still be true even after the specific tools implementing them change. +Not a firewall rule listing, not a port inventory, not a service configuration reference β€” those live in [`docs/firewall-and-network.md`](firewall-and-network.md) and the docs linked in the table above. This is strictly the stable, directional shape of how trust and traffic actually move β€” meant to still be true even after the specific tools implementing it change again, the way it already has twice. --- Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md)