From 033140f65dc53a0af245e01eab3cceda37625503 Mon Sep 17 00:00:00 2001 From: thesabear Date: Wed, 26 Aug 2026 12:32:47 +0300 Subject: [PATCH] firewall-and-network: verify against live-checked KB, fix stale CrowdSec claim, add real depth and evidence Pulled every pfSense child note (Default Policy, Proxmox Firewall Usage, VM-Level Firewalld Strategy, WAN Rules, Security Tooling, Failure Impact, HAProxy, Aliases, DNS & Resolver Config, LAN Rules, NAT, VPN), not just the originally-scoped subset. Live-verified firewalld defaults, SSH rich-rule restrictions, and Vault's per-client rules directly against the running hosts - all matched exactly. Caught and fixed a real staleness bug in the KB itself: WAN Rules and Security Tooling (both dated a week old) still described CrowdSec as active, but the more recent Aliases note (updated yesterday, checked against pfSense's actual config.xml) confirmed the package has been fully purged. Fixed both stale notes and the public doc, which had already inherited the wrong claim. Folded in the staged PAM defense-in-depth note properly - found the fuller, better story directly in the Ansible role's source comments (the bug hit twice, on two different hosts, before the permanent fix). Added the LAN lockdown narrative this doc was missing, the DNS bypass-block defense-in-depth story, and the this-week port-alias conversion. Cut two redundant Mermaid diagrams. Added real evidence: four pfSense screenshots already validated for architecture-overview.md reused here since they directly prove this doc's own claims, plus two real firewall-cmd command outputs from live verification this session. --- docs/firewall-and-network.md | 80 ++++++++++++++++++++++-------------- 1 file changed, 50 insertions(+), 30 deletions(-) diff --git a/docs/firewall-and-network.md b/docs/firewall-and-network.md index 3494088..2f0c167 100644 --- a/docs/firewall-and-network.md +++ b/docs/firewall-and-network.md @@ -3,66 +3,86 @@ > [!NOTE] > pfSense at the edge, firewalld on every individual host — two layers, two different jobs, and a network model with no blanket "anything on the internal network can talk to anything else on it" rule anywhere. Every real flow is explicit; everything else is logged and dropped. This is the enforcement layer underneath the trust boundaries described in [`docs/architecture-overview.md`](architecture-overview.md). -```mermaid -flowchart TD - Internet(["Internet"]) --> Edge - subgraph Edge["pfSense — Edge Firewall"] - Bogon["Bogon block +
CrowdSec IP-reputation blacklist"] - NonstdSSH["SSH — nonstandard port only"] - PubProxy["HAProxy — public reverse-proxy layer"] - IDS["Suricata inline IDS/IPS"] - end - Edge -->|explicit rules only| LAN - - subgraph LAN["Internal Network"] - direction TB - Rule1["DNS · Secrets/PKI · Reverse proxy
Metrics/logs · Security agents
NTP · Admin SSH · File sharing"] - Catchall["Everything else:
default-deny + log"] - end -``` - ## Why two layers, not one A single firewall layer was rejected early: the edge firewall (pfSense) is the fleet's only genuinely public-facing component, so it carries a different threat model — internet-scale scanning and abuse — than internal host-to-host traffic ever will. Collapsing both into one ruleset would mean either the internal segmentation rules get diluted by public-facing noise, or the public-facing rules get relaxed to match internal assumptions. Keeping them separate means each layer's rules can be as strict as that layer's actual threat model demands, independently. ## The one rule that matters most +This wasn't always this tight. The LAN interface used to run pfSense's own standard default — a blanket allow between hosts — before it was deliberately narrowed to LAN→WAN/internet only, closing off unrestricted lateral movement that had nothing to do with any application actually needing it. What replaced it is the model below. + Every real service-to-service flow — DNS lookups (see [`docs/dns-naming.md`](dns-naming.md)), secrets/PKI traffic (see [`docs/secrets-and-pki.md`](secrets-and-pki.md)), reverse-proxy access, metrics/log shipping (see [`docs/observability.md`](observability.md)), security-agent reporting (see [`docs/security-monitoring.md`](security-monitoring.md)), NTP, admin SSH, file sharing, and a handful of specific app-to-app integrations — gets its own explicit allow rule on pfSense's LAN interface. Anything not explicitly listed hits a default-deny-and-log catch-all. **This catch-all is the fleet's actual lateral-movement boundary**, not any single host's own firewall — a compromised host can't simply pivot to another host over a port that was never explicitly opened between them. +That narrowing and the catch-all it produced aren't just claims — here's the actual live ruleset: + +![pfSense rule explicitly allowing LAN to WAN/internet only, not LAN-to-LAN — the actual lockdown rule](../assets/pfsense-rules-lan-core-services.png) + +![pfSense LAN catch-all deny rule: block+log east-west traffic not explicitly allowed](../assets/pfsense-rules-lan-catchall-deny.png) + > [!TIP] > A few explicit rules are narrower than they look on purpose — inbound access to Vault's own admin API, for instance, is gated to only the two or three services that ever legitimately need it (the reverse proxy and the DNS server), not the whole fleet. The instinct throughout this ruleset is "name the specific source," not "allow the group and hope." +DNS enforcement specifically doesn't rely on just one mechanism. Every static-lease device gets the internal DNS server pushed via a per-lease DHCP override — every current static lease carries it — but that's a soft guarantee: a brand-new device on a plain dynamic lease would fall back to the router's own upstream resolvers instead, at least until it gets a static reservation of its own. The firewall closes that gap independently, at the network level: an explicit rule blocks any LAN client from reaching external DNS outright, full stop. So a device that somehow ended up pointed at the wrong resolver still can't actually use it — DNS bypass isn't just discouraged by configuration, it's structurally blocked by a second, unrelated layer that doesn't depend on the first one being right. + ## Named groups, not raw addresses pfSense aliases (named groups) stand in for hardcoded IP lists throughout — adding a host to a group is a single membership edit, not a hunt through every rule that needs to reach it. Groups nest: a "whole managed fleet" alias is composed of several narrower per-role aliases, so a rule against the broad group automatically covers new members of any narrower one without the rule itself ever changing. +The same discipline applies to the other axis of every rule, and it's still actively maintained, not something set up once and left alone: every port field in this ruleset runs through a named alias too, converted from literal numbers just this week specifically so a port never ends up hardcoded and silently duplicated across multiple rules. + +![pfSense firewall IP aliases, one per fleet host/role](../assets/pfsense-aliases-ip.png) + +![pfSense firewall port aliases, one per service/flow](../assets/pfsense-aliases-ports.png) + ## Per-host firewalls: minimal, not blanket -Each host runs firewalld locally, opening only the specific ports it actually needs — not a blanket "allow everything, block nothing" default. Vault's host goes further than the rest: instead of one open port for its most sensitive service, individual source IPs get their own explicit rich-rule, so a compromised unrelated service on the same network still can't reach it even if it somehow got past pfSense. +Each host runs firewalld locally, opening only the specific ports it actually needs — not a blanket "allow everything, block nothing" default. Vault's host goes further than the rest: instead of one open port for its most sensitive service, individual source IPs get their own explicit rich-rule — five named clients, confirmed directly against the live ruleset, and zero blanket ports open at all — so a compromised unrelated service on the same network still can't reach it even if it somehow got past pfSense. This is the actual live output, not a paraphrase: + +``` +$ firewall-cmd --zone=public --list-rich-rules +rule family="ipv4" source address="[automation host]" port port="[vault mgmt port]" protocol="tcp" accept +rule family="ipv4" source address="[caddy VM]" port port="8200" protocol="tcp" accept +rule family="ipv4" source address="[automation host]" port port="8200" protocol="tcp" accept +rule family="ipv4" source address="[technitium]" port port="8200" protocol="tcp" accept +rule family="ipv4" source address="[admin workstation]" port port="[vault mgmt port]" protocol="tcp" accept +rule family="ipv4" source address="[rocky-docker]" port port="8200" protocol="tcp" accept +rule family="ipv4" source address="[monitoring]" port port="8200" protocol="tcp" accept +$ firewall-cmd --zone=public --list-ports +[no output — zero blanket-opened ports] +``` + +SSH follows the same instinct, even though every host obviously needs it reachable: it's not a blanket-open port anywhere in the fleet. Each host's admin access is gated by explicit rich-rules to exactly two source IPs — the automation host that actually runs playbooks, and one specific admin workstation — each on that host's own unique, nonstandard port, and nothing else. Confirmed directly against a live host's actual ruleset, not assumed from documentation: even the tool used to manage the fleet doesn't get a standing exception from the fleet's own access model. + +``` +$ firewall-cmd --zone=public --list-rich-rules +rule family="ipv4" source address="[admin workstation]" port port="[this host's SSH port]" protocol="tcp" accept +rule family="ipv4" source address="[automation host]" port port="[this host's SSH port]" protocol="tcp" accept +``` > [!WARNING] > **A real, documented gap**: on Docker hosts, container-published ports bypass firewalld entirely — Docker's own iptables rules are evaluated before firewalld's zone rules, so firewalld's local restrictions only actually protect the small number of *native*, non-containerized services on that host (things like the observability agent from [`docs/observability.md`](observability.md)), not the dozens of containerized apps in [`stacks/`](../stacks/). The network-level segmentation above still applies regardless — pfSense doesn't care whether a port is opened by firewalld or by Docker's own NAT rules — so this is a defense-in-depth gap specifically at the host layer, not an open-to-the-world one. Worth naming precisely rather than glossing over: firewalld protects less of a Docker host's actual attack surface than it might first appear to. ## Edge firewall: three jobs, one device -```mermaid -flowchart LR - EdgeFW["pfSense"] --> J1["Public ingress
bogon block, CrowdSec IP-reputation feed,
nonstandard SSH port"] - EdgeFW --> J2["HAProxy — second reverse-proxy layer
for genuinely public apps only,
automated ACME wildcard certs"] - EdgeFW --> J3["Security tooling
Suricata inline IDS/IPS, cert renewal"] -``` +One device, three distinct responsibilities, not three separate boxes: -The second reverse-proxy layer here is genuinely separate from the internal two-zone model in [`docs/dns-naming.md`](dns-naming.md) — it exists specifically for the handful of apps meant to be reachable from the open internet, and nothing else ever routes through it. +- **Public ingress** — bogon block, SSH on a nonstandard port only. +- **HAProxy** — a second reverse-proxy layer, genuinely separate from the internal two-zone model in [`docs/dns-naming.md`](dns-naming.md), for the handful of apps meant to be reachable from the open internet, with automated ACME wildcard certs. Nothing else ever routes through it. +- **Security tooling** — Suricata inline IDS/IPS, plus certificate renewal. + +> [!TIP] +> **A disabled leftover, kept on purpose, not by accident.** HAProxy still has a second frontend defined for the LAN-facing zone — a holdover from before the Caddy VM took over user-facing internal routing. It's explicitly disabled, not just unused and forgotten: turning genuinely-dead config off cleanly, rather than letting it silently rot half-configured next to what's actually live, is its own small discipline worth keeping visible rather than tidying away. > [!IMPORTANT] -> **Failure impact, thought through in advance, not discovered during an actual outage**: if pfSense goes down, public access and outbound internet both go with it — but every internal service that doesn't depend on WAN reachability keeps running normally, including the orchestration flows in [`docs/orchestration.md`](orchestration.md), which only ever reach internal hosts. The blast radius is deliberately bounded to "the edge," not "the fleet" — a design property worth stating explicitly rather than leaving implicit, since it's exactly the kind of thing that should be known *before* an incident, not discovered during one. +> **Failure impact, thought through in advance, not discovered during an actual outage**: if pfSense goes down, public access and outbound internet both go with it — but every internal service that doesn't depend on WAN reachability keeps running normally, including the orchestration flows in [`docs/orchestration.md`](orchestration.md), which only ever reach internal hosts. The blast radius is deliberately bounded to "the edge," not "the fleet" — a design property worth stating explicitly rather than leaving implicit, since it's exactly the kind of thing that should be known *before* an incident, not discovered during one. One thing does quietly degrade even though nothing breaks outright, and it's worth being honest about rather than rounding up to "everything's fine": Vault Agent's certificate renewal and Technitium's own PKI issuance both route through pfSense-mediated LAN paths, so an extended outage would eventually stall renewals fleet-wide — not an immediate failure, but a real one if the outage runs long enough to matter. + +## Defense-in-depth below the network layer: PAM + +Firewalls aren't the only layer that matters. On this fleet's Debian-family hosts, the login stack itself got hardened after a real, *recurring* incident — not a one-off. `pam_unix.so`'s default auth line uses a numbered jump ("on success, skip forward N modules") to reach approval, which is fragile by construction: any future package that inserts its own line anywhere in that stack silently shifts what the number actually lands on. This broke real, correct-password logins **twice**: once during initial rollout, fixed by bumping the number up by one — and again five days later, on a different host, when an unrelated package pulled in its own PAM line and broke that same numbered jump a second time, landing on a deny instead of an allow. A correct password got treated as a failure. The permanent fix replaced the number with a keyword that terminates the stack on success immediately, regardless of how many lines ever get added after it later — closing the entire class of bug, not just the one instance that got caught. Rocky/RHEL hosts never carried this risk in the first place; they manage their PAM stack through `authselect`, a declarative system not exposed to the same ordering hazard. ## The hypervisor's own firewall: deliberately unused Proxmox has its own built-in firewall capability, left with an empty ruleset on purpose. pfSense and firewalld already cover the two layers that matter — public/internal segmentation and per-host minimal exposure — and a third overlapping layer at the hypervisor level would add complexity (a third ruleset to audit, a third place a rule could silently be wrong) without meaningfully improving posture. Recognizing when *not* to add a redundant control is as much a design decision as adding one, and it's the same reasoning that kept the secrets model in [`docs/secrets-and-pki.md`](secrets-and-pki.md) to one trust root instead of two. -> [!NOTE] -> **Staged for this file's full audit pass — verified true, not yet integrated into the prose above.** Confirmed directly against the live Ansible role, not just against documentation: on the fleet's Debian-family hosts, the PAM login stack is deliberately configured to stop immediately on a successful authentication instead of jumping to a numbered step — a defense against a future package silently inserting its own PAM line and shifting that jump target, which would misread a correct password as a failed one. Not theoretical: exactly that bug was caught live on one of those hosts before this fix went in. Rocky/RHEL-family hosts don't need the same fix — they manage their PAM stack through `authselect`, a different, declarative system not exposed to the same class of ordering bug. Fold this in wherever it fits once this file gets its real pass — likely near "Per-host firewalls" above, as another example of defense-in-depth below the network layer. - --- Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/architecture-overview.md`](architecture-overview.md), [`docs/security-monitoring.md`](security-monitoring.md) · Stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/), [`stacks/platform/docker-socket-proxy/`](../stacks/platform/docker-socket-proxy/)