architecture-overview: verify against Trilium, fix inaccurate dependency graph, add real evidence
Removed the dependency-graph Mermaid diagram (had backwards Vault dependency arrows, contradicted the documented ingress model) and the four-boundary Mermaid diagram (fully redundant with the table below it). Verified every remaining claim against Trilium's Traffic Flow Model and fixed two that didn't hold up: the DNS-bypass failure behavior overstated an unconditional SSO bypass, and the notifications failure behavior had an unsourced claim about backlog retry. Added the real story behind the trust-boundary hardening (three stages of the same "trusted by default" flaw getting found and closed, not a changelog of dated facts), a new trust-depth diagram showing that being "inside the network" still enforces its own internal boundaries, and real pfSense screenshots proving the LAN lockdown rules actually exist in production, not just in the doc.
This commit is contained in:
10 files changed
+52
-65
No files matched your search
@@ -6,79 +6,19 @@
|
||||
<table>
|
||||
<tr>
|
||||
<td align="center"><strong>11</strong><br/>fleet hosts</td>
|
||||
<td align="center"><strong>55</strong><br/>compose stacks</td>
|
||||
<td align="center"><strong>4</strong><br/>trust boundaries</td>
|
||||
<td align="center"><strong>3</strong><br/>routing authorities</td>
|
||||
<td align="center"><strong>8</strong><br/>traffic laws</td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
## Dependency graph
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Internet(["Internet"]) --> Edge["Edge Firewall"]
|
||||
Edge --> LAN["LAN"]
|
||||
|
||||
LAN --> DNS["Internal DNS"]
|
||||
LAN --> Vault["Vault"]
|
||||
DNS --> Vault
|
||||
|
||||
Vault --> Apps["App Host<br/>41 stacks"]
|
||||
Vault --> Automation["Automation Host"]
|
||||
LAN --> FileServer["File Server"]
|
||||
DNS --> FileServer
|
||||
LAN --> Automation
|
||||
|
||||
Apps --> InternalProxy["Internal Proxy Zone"]
|
||||
Apps --> SSO["SSO"]
|
||||
Automation --> GitHost["Git Hosting"]
|
||||
|
||||
Vault --> EdgeProxy["User-Facing Proxy<br/>mTLS"]
|
||||
DNS --> EdgeProxy
|
||||
Apps -.proxied.-> EdgeProxy
|
||||
SSO -.auth.-> EdgeProxy
|
||||
|
||||
Apps -.-> Monitoring["Monitoring"]
|
||||
Automation -.-> Monitoring
|
||||
FileServer -.-> Monitoring
|
||||
Vault -.-> Monitoring
|
||||
DNS -.-> Monitoring
|
||||
EdgeProxy -.-> Monitoring
|
||||
|
||||
Apps -.-> SecMon["Security Monitoring"]
|
||||
Automation -.-> SecMon
|
||||
FileServer -.-> SecMon
|
||||
Vault -.-> SecMon
|
||||
DNS -.-> SecMon
|
||||
EdgeProxy -.-> SecMon
|
||||
```
|
||||
|
||||
> [!TIP]
|
||||
> Solid arrows are real dependencies — the target has to be reachable and healthy for the source to function. Dotted arrows are asynchronous reporting, not a blocking dependency.
|
||||
|
||||
## The hardest decision: how many trust boundaries is enough?
|
||||
|
||||
The easy answers are both wrong. **One boundary** ("you're on the LAN, you're trusted") is what most self-hosted setups actually run, and it means a single compromised container can reach everything else with no further resistance. **Maximum boundaries everywhere** (mTLS on every single connection, no exceptions) sounds safer but isn't free — every hop gets slower, every service needs its own client certificate lifecycle, and the operational burden of *that* is exactly the kind of complexity that gets quietly disabled under deadline pressure, which is worse than never having built it.
|
||||
|
||||
The actual answer here is **four boundaries, each doing a different job**:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
WAN["🌐 WAN<br/>untrusted"] -->|public| EdgeFW["Edge Firewall<br/>public proxy"]
|
||||
Browser["🧑 LAN browsers<br/>human-operated"] -->|mTLS + SSO| UserProxy["User-facing proxy"]
|
||||
Service["📦 App-to-app<br/>machine-provisioned"] -->|allowlist + SSO| InternalProxy["Internal proxy"]
|
||||
Infra["⚙️ LAN infra-to-infra<br/>non-HTTP"] -->|per-flow rule| HostFW["Host firewall"]
|
||||
|
||||
classDef untrusted fill:#f8cecc,stroke:#b85450,color:#333
|
||||
classDef human fill:#dae8fc,stroke:#6c8ebf,color:#333
|
||||
classDef machine fill:#d5e8d4,stroke:#82b366,color:#333
|
||||
classDef infra fill:#fff2cc,stroke:#d6b656,color:#333
|
||||
class WAN,EdgeFW untrusted
|
||||
class Browser,UserProxy human
|
||||
class Service,InternalProxy machine
|
||||
class Infra,HostFW infra
|
||||
```
|
||||
|
||||
|
||||
| Boundary | Who's on the other end | Control | Why this control, not another |
|
||||
|---|---|---|---|
|
||||
@@ -87,6 +27,8 @@ flowchart LR
|
||||
| 🟢 App-to-app | A container that's already inside the network, already provisioned by the same pipeline as everything else | IP-allowlist + SSO fallback, no mTLS | Full mTLS here would mean managing certificate lifecycle for every internal service-to-service call — real overhead for traffic that's already constrained to known, provisioned sources. The risk profile doesn't justify the same control as human-operated browsers. |
|
||||
| 🟡 LAN infra-to-infra | Non-HTTP traffic — metrics, logs, NTP, agent reporting | Explicit per-flow firewall rule, no proxy at all | Can't go through a reverse proxy — it isn't HTTP. The firewall itself is the only layer that *can* enforce this one. |
|
||||
|
||||
Every boundary above only means something if a service can't just skip it by reading the wrong header: **backend services trust proxy headers, never the raw client IP.** If that weren't true, the mTLS/SSO enforcement happening at the proxy layer would be theater — anything downstream could just claim to be a trusted IP and walk straight through. The proxy is the only thing allowed an opinion on who a client really is.
|
||||
|
||||
**Three separate routing authorities enforce these** — the edge firewall, the user-facing proxy, and the internal proxy zone — rather than one central router handling everything. That's a deliberate rejection of the simpler single-router design: one router is one thing to hardening-focus on and one thing that takes everything down if it's compromised or misconfigured.
|
||||
|
||||
### What actually happens when one fails
|
||||
@@ -102,7 +44,48 @@ Not a hypothetical — this is the real, deliberate payoff of splitting routing
|
||||
A single-router design fails this test completely — one outage, one misconfiguration, everything goes dark at once. Three independent authorities mean the worst case for any one failure is "a third of the traffic types stop," never all of them.
|
||||
|
||||
> [!IMPORTANT]
|
||||
> **This wasn't the starting design — it was built up over two real, deliberate changes.** The public-facing reverse proxy used to run as its own dedicated VM; it was consolidated directly onto the edge firewall as a package instead, removing an entire host from the public-ingress path without losing any functionality. And the app-to-app boundary used to not exist at all — internal service traffic ran over a flat, unauthenticated mesh network with no trust boundary. It was replaced by the two-zone model now described in [`docs/dns-naming.md`](dns-naming.md). Both changes added real operational complexity in exchange for a smaller blast radius — accepted deliberately, not stumbled into, and the kind of decision that only looks obvious in hindsight.
|
||||
> **None of this existed at once — it hardened in stages, each one closing whatever soft spot the last stage left standing.**
|
||||
>
|
||||
> The first fix was structural: the public reverse proxy used to run as its own dedicated VM, a whole extra host sitting in the public-ingress path for no reason beyond history. Folding HAProxy directly into the edge firewall as a package collapsed that hop away entirely — one less host that could be the reason the public path went down.
|
||||
>
|
||||
> The second fix was conceptual, and the bigger one: for a long time the app-to-app boundary simply didn't exist. Internal service traffic crossed a flat, unauthenticated mesh network — anything on it could reach anything else on it, no proxy, no identity check, nothing. That's exactly the "you're on the network, you're trusted" pattern this document opens by calling the easy, wrong answer. Replacing it with the two-zone model in [`docs/dns-naming.md`](dns-naming.md) is what gave app-to-app traffic a real boundary for the first time.
|
||||
>
|
||||
> The third fix found the same flaw hiding somewhere less obvious: plain LAN-to-LAN traffic — metrics, logs, NTP, agent reporting — still had a blanket allow at the firewall, even after the app-to-app zone had its own identity-checked proxy. Any host could reach any other host; the only thing standing between "one compromised container" and "everything else on the LAN" was whatever that individual host's own firewall happened to be configured to do, which is a coin flip, not a boundary. Closing it meant giving up "it just works" for LAN infra traffic and writing an explicit rule for every flow that's actually supposed to exist — the same trade this document makes everywhere else: a boundary only counts if crossing it takes a decision, not an accident of network topology.
|
||||
>
|
||||
> Three different places, the same failure mode, the same fix each time — that's not a coincidence, it's what happens when you actually go looking for where "trusted by default" is still hiding.
|
||||
|
||||
That third fix isn't just a claim — it's the actual live ruleset:
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<details>
|
||||
<summary>See the rest of the actual pfSense configuration</summary>
|
||||
|
||||
**WAN — only three things pass inbound, everything else default-deny:**
|
||||
|
||||
|
||||
|
||||
**LAN — fleet observability and security-agent reporting, one explicit rule per flow:**
|
||||
|
||||
|
||||
|
||||
**LAN — the Ansible SSH fan-out, one explicit rule per host, sourced from the automation host only:**
|
||||
|
||||
|
||||
|
||||
**LAN — the built-in anti-lockout rule, at the top of the list by necessity:**
|
||||
|
||||
|
||||
|
||||
**The aliases behind all of the above — named per host/role, not raw IPs scattered through every rule:**
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
## The other six traffic laws
|
||||
|
||||
@@ -111,9 +94,9 @@ Each of these gets its own full mechanism doc elsewhere in this repo — this ta
|
||||
| # | Law | Failure behavior |
|
||||
|---|---|---|
|
||||
| 3. Secrets | Applications never talk to Vault directly — one hop through a local agent only. See [`docs/secrets-and-pki.md`](secrets-and-pki.md). | Vault down → secret renewal stops, services run until their current token expires. No immediate disruption. |
|
||||
| 4. DNS | Which internal zone a hostname resolves in determines which trust boundary (above) a request enters. See [`docs/dns-naming.md`](dns-naming.md). | DNS down → app-to-app resolution fails; direct-IP fallback exists but silently bypasses the SSO gate. |
|
||||
| 4. DNS | Which internal zone a hostname resolves in determines which trust boundary (above) a request enters. See [`docs/dns-naming.md`](dns-naming.md). | DNS down → app-to-app resolution fails; a direct-IP fallback exists, and it bypasses the SSO gate — but only for traffic already coming from an IP on the trusted allowlist. Anything else still hits Authentik regardless of DNS. |
|
||||
| 5. Observability | Logs and metrics push asynchronously — never inline with application traffic, full stop. See [`docs/observability.md`](observability.md) and [`docs/security-monitoring.md`](security-monitoring.md). | Monitoring down → collection stops; every application service keeps running, completely unaffected. |
|
||||
| 6. Notifications | Two independent branches (message-queue and HTTP webhook) converge on one formatter; neither is part of any service's runtime dependency chain. See [`docs/notifications.md`](notifications.md). | A dropped notification is lost, not retried into a backlog that could pressure the sender. |
|
||||
| 6. Notifications | Two independent branches (message-queue and HTTP webhook) converge on one formatter; neither is part of any service's runtime dependency chain. See [`docs/notifications.md`](notifications.md). | Notification delivery is event-driven, not persistent inline execution — a failure here doesn't touch application traffic either way. |
|
||||
| 7. CI/CD | Build/deploy traffic never sits inline with runtime; schedule-triggered, not push-triggered. See [`docs/registry-and-build.md`](registry-and-build.md). | A failed deploy doesn't affect already-running services — enforced by the redeploy step's own rollback logic. |
|
||||
| 8. Management | Administrative access is its own separate plane — one dedicated identity, non-standard ports, explicit per-host firewall rules from one source only. | Management access down → administration unavailable; every running service is completely unaffected. |
|
||||
|
||||
@@ -121,7 +104,7 @@ The pattern repeats across all eight laws, not just the trust-boundary story abo
|
||||
|
||||
## What this map deliberately excludes
|
||||
|
||||
Not a firewall rule listing, not a port inventory, not a service configuration reference — those live in [`docs/firewall-and-network.md`](firewall-and-network.md) and the docs linked in the table above. This is strictly the stable, directional shape of how trust and traffic actually move — meant to still be true even after the specific tools implementing it change again, the way it already has twice.
|
||||
Not a firewall rule listing, not a port inventory, not a service configuration reference — those live in [`docs/firewall-and-network.md`](firewall-and-network.md) and the docs linked in the table above. This is strictly the stable, directional shape of how trust and traffic actually move — meant to still be true even after the specific tools implementing it change again, the way it already has three times.
|
||||
|
||||
---
|
||||
Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/notifications.md`](notifications.md), [`docs/registry-and-build.md`](registry-and-build.md)
|
||||
Reference in new issue
Block a user