docs: add security/hardening deep-dive write-ups

Secrets & PKI, Secret Distribution Model, SSO & Identity, Firewall & Network Segmentation, DNS Naming Model, and Security Monitoring - the first batch of per-system write-ups linked from the new README, sourced and sanitized from the internal KB.
2026-08-21 22:48:50 +03:00
1 parent f37efc2654
commit d5f37b5787
@@ -0,0 +1,23 @@
 # DNS Naming Model
 ## Two zones, deliberately never conflated
 Internal DNS is authoritative for two zones that mirror each other structurally but serve different traffic:
 | Zone | Serves | Enforcement |
 |---|---|---|
 | Browser-facing zone | Users/browsers, via the user-facing reverse proxy | Mutual TLS + SSO enforced at the edge |
 | Service-to-service zone | App-to-app traffic (webhooks, API integrations), via the internal reverse proxy | Trusted-IP allowlist, no mTLS |
 Most apps get one hostname per zone, resolving to a different front door depending on who's asking — a browser gets the mTLS-enforced path, another service gets the lighter-weight internal path. The two zones aren't perfectly identical at any given moment; a handful of apps exist in only one zone rather than both, which is expected — not every stack needs both zones, driven by actual need rather than blanket symmetry.
 ## The public exception
 One app is a deliberate exception to the whole two-zone model: it's genuinely public-facing, routed directly through the edge firewall's own reverse-proxy package, and has zero presence in either internal zone. Apps meant for the public internet skip the internal DNS model entirely rather than getting a third, redundant record.
 ## Caught during a later audit
 A later documentation audit caught real drift from what had been assumed: an older note claimed an external DNS forwarder (a third-party provider) was configured for anything outside the two internal zones. Live verification found no such forwarder actually configured — unresolved queries forward to the edge firewall and a couple of standard public resolvers instead, and internal clients are blocked from using any other resolver directly. The takeaway kept from that audit: documentation drift is caught by checking live state, not by assuming a prior note was still accurate.
 ---
 Related stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/)
@@ -0,0 +1,34 @@
 # Firewall & Network Segmentation
 ## No blanket LAN-to-LAN allow
 The most important rule in the whole model is a negative one: there is no default "anything on the internal network can talk to anything else on it" rule. Every real service-to-service flow — DNS lookups, secrets/PKI traffic, reverse-proxy access, metrics/log shipping, security-agent reporting, NTP sync, admin SSH, file sharing, and a handful of specific app-to-app API integrations — gets its own explicit allow rule. Anything not explicitly listed hits a default-deny-and-log catch-all at the bottom of the ruleset. This catch-all is the fleet's actual lateral-movement boundary, not any single host's own firewall (see below).
 A few of the explicit rules are narrower than they look at first glance, on purpose — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet.
 ## Firewall objects are named groups, not raw addresses
 Every rule references named groups instead of hardcoded addresses — adding a host to a group is a single membership edit rather than hunting down and updating every rule that needs to reach it. Groups nest: a "the whole managed fleet" group is composed of several more specific per-role groups, so a rule written against the broad group automatically covers new members added to any of the narrower ones. One group is deliberately scoped *narrower* than its name might suggest — it only covers the fleet's actively-managed tier, and anything outside that tier has to be added on purpose if it needs the same access.
 ## Per-host firewalls: minimal, not blanket
 On top of the network-level segmentation above, each host's own local firewall opens only the specific ports that host actually needs — not a blanket "allow everything, block nothing" default. One host takes this further: instead of one open port for its most sensitive service, individual source IPs get their own explicit rule, so a compromised unrelated service on the same network still can't reach it.
 **A real, documented gap, not glossed over:** on Docker hosts, container-published ports bypass the host's own firewall rules entirely — the container runtime's own network rules are evaluated first, so the host firewall only actually protects the small number of *native* (non-containerized) services running there, not the dozens of containerized apps. The network-level segmentation above (and the edge firewall further out) still applies regardless — this is a defense-in-depth gap, not an open-to-the-world one.
 ## Edge firewall
 The edge firewall is the fleet's outermost boundary and does three separate jobs:
 - **Public ingress** — a bogon-address block, an IP-reputation blacklist feed, SSH exposed only on a nonstandard port, and HTTPS/HTTP forwarded to the fleet's public-facing reverse-proxy layer. Nothing else reaches the edge firewall's public interface at all.
 - **A second, independent reverse-proxy layer** — separate from the internal two-zone model covered in [DNS Naming Model](dns-naming.md), the edge firewall runs its own reverse-proxy package specifically for the small number of apps that are genuinely meant to be public. Each public app gets its own backend rule mapping its public hostname to the correct internal service, with automated wildcard certificate issuance.
 - **Security tooling** — an inline intrusion detection/prevention engine, the same IP-reputation service driving the WAN blacklist, and automated certificate renewal all run at this layer.
 **Failure impact, thought through in advance:** if the edge firewall goes down, public access and outbound internet access both go with it — but every internal service that doesn't depend on WAN reachability keeps running normally. The blast radius is deliberately bounded to "the edge," not "the fleet."
 ## The hypervisor's own firewall: deliberately unused
 The virtualization platform underneath everything has its own built-in firewall capability. It's left with an empty ruleset on purpose — the edge firewall and each host's own local firewall already cover the two layers that actually matter, and a third overlapping layer would add complexity without meaningfully improving the security posture. Recognizing when *not* to add a redundant control is as much a design decision as adding one.
 ---
 Related stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/), [`stacks/platform/docker-socket-proxy/`](../stacks/platform/docker-socket-proxy/)
@@ -0,0 +1,60 @@
 # Secret Distribution Model
 How a secret value moves from being written into Vault to being consumed by a running container. Four distinct flows, each with its own explicit contract — directional flow and authority boundaries, not configuration syntax.
 ## 1. Certificate Flow
 **Law** — Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines.
 **Flow** — Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host.
 **Invariants**
 - Certificates are short-lived and re-issued on Vault Agent's own schedule, not on-demand by the consuming service.
 - A host only ever receives certificates tagged for its own service identity.
 - The private key never traverses the network after issuance — written directly to local disk by the agent that requested it.
 **Failure behavior** — If Vault is unreachable, existing certificates keep working until they expire. No immediate outage; a slow-building one if the outage persists past the shortest cert's TTL.
 ## 2. Application Secret Flow
 **Law** — A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored).
 **Flow** — Operator writes value → Vault KV → Vault Agent (polling/watching) → rendered file on local disk → container reads it via an env file or a direct bind-mount.
 **Invariants**
 - The git-tracked template contains placeholder syntax only — a real secret value anywhere in this flow's git history is a bug, not a variant.
 - The rendered file is owned entirely by Vault Agent. Hand-editing it gets overwritten on the next render cycle and hides the actual source of truth.
 - A container consumes the rendered file from its own host's disk — it never reaches across the network to fetch a secret itself.
 **Failure behavior** — If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected. A secret rotation queued during an outage simply doesn't propagate until Vault Agent can reach Vault again — no service disruption, just a delayed rotation.
 ## 3. Configuration Distribution Flow
 **Law** — The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc.
 **Flow** — Operator edits the template inventory in the provisioning repo → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts and picks up the new template list.
 **Invariants**
 - A host's Vault Agent only ever renders templates tagged for its own identity — the full template list is visible to every host, but each host filters to its own subset.
 - A missing or malformed template referenced by a host's own config crashes that host's *entire* Vault Agent process, not just the one broken template — every other secret/cert that host was already rendering stops updating too. This is why a config redeploy always requires confirming the underlying change actually landed before assuming it worked.
 **Failure behavior** — If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on their last-known-good values until the config is fixed and Vault Agent restarts successfully.
 **Shared-secret variant** — not every template is host-scoped. One credential needed by every docker host for its own registry pushes is templated from a single shared path rather than a per-host one — same mechanics as the rest of this flow, just a shared source instead of a private one.
 ## 4. Administrative Secret Flow (a separate system entirely)
 **Law** — Not every secret in this fleet goes through Vault. Admin and tooling credentials used by playbooks and operator sessions themselves (third-party API tokens, login credentials for internal tools) live in a completely separate, deliberately simpler system: encrypted directly into the provisioning repo using a standard vault-encryption tool, decrypted on demand by whoever holds the passphrase.
 **Flow** — Operator encrypts value locally → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access (an automation run, or a human operator).
 **Why two systems, not one** — Vault KV is for runtime/service/pipeline secrets a running container or live service needs; the simpler encrypted-file system is for secrets used by tooling that *operates on* the fleet rather than secrets *consumed by* the fleet's own running services. A reader who only knows the Vault-KV flow above would have no way to tell which system a given secret belongs to — consumed by a container at runtime means Vault KV; used by a human or an automation job doing administrative work means the encrypted-file system.
 **Invariants**
 - The encrypted blob is a single opaque ciphertext per file — searching for a variable name won't find it; only a full decrypt reveals what's stored.
 - Unlike Vault KV, there's no per-host scoping or automatic rendering — whoever has the passphrase can read every secret in the file.
 **Failure behavior** — Not applicable in the live-service sense; this isn't consumed by running containers. A lost passphrase would block automation runs and admin tooling needing those credentials, not any live application.
 ---
 Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md)
@@ -0,0 +1,77 @@
 # Secrets & PKI
 A single Vault cluster serves as both the internal certificate authority and the secret store for the whole fleet — one trust root, not two parallel systems.
 ## What it does
 - **Certificate authority** — two PKI mounts: one for public-facing wildcard certs, one for internal mTLS certs. Both issue short-lived certificates to the hosts that need them.
 - **Secret store** — a KV v2 mount holds every migrated application's real secret values (database passwords, API tokens, signing keys), one namespace per host.
 - **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly.
 ## Why it's built this way
 Before this existed, every stack's secrets sat in plaintext inside git-tracked `.env` files — readable by anyone with repo access, and permanently baked into git history even after being "fixed." The replacement model:
 - The real secret value lives in exactly one place: Vault.
 - What's git-tracked is a template with placeholder syntax, never a real value.
 - The rendered file with the real value is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
 - If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle.
 The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before this was extended to cover application secrets — reusing a trust model that was already proven, rather than inventing a second one.
 ## Architecture
 ```mermaid
 flowchart TD
    Vault["Vault<br/>PKI (public + internal mounts) + KV v2 store<br/>TLS listener, cert-verified · audit log"]
    Proxy["Internal app-to-app reverse-proxy zone"]
    A1["Vault Agent — App Host<br/>~12 stacks' secrets + internal CA cert"]
    A2["Vault Agent — Automation Host<br/>~10 stacks' secrets"]
    A3["Vault Agent — Monitoring Host<br/>admin credentials + OIDC secret"]
    A4["Vault Agent — Edge Proxy Host<br/>mTLS zone certs"]
    A5["Vault Agent — DNS Host<br/>admin UI cert"]
    A6["Vault Agent — Security Monitoring Host<br/>manager/indexer/dashboard secrets"]
    Vault -->|HTTPS, cert-verified| Proxy
    Proxy --> A1
    Proxy --> A2
    Proxy --> A3
    Proxy --> A4
    Proxy --> A5
    Proxy --> A6
 ```
 ### Two authentication flows, not one
 - **AppRole auth** (every host, ongoing): Vault Agent → role_id + secret_id → Vault → short-lived token, cached locally, renewed automatically.
 - **Bootstrap auth** (rare, human-initiated): an operator uses a root token to mint new AppRole credentials when onboarding a host, or to create/update policies. Not part of any workload's runtime path.
 ### The one deliberately-asymmetric trust decision
 Every other backend connection inside the internal app-to-app zone skips certificate verification on the reverse proxy hop (it still encrypts the connection, just doesn't check whose cert it's talking to) — a pragmatic tradeoff for low-stakes internal traffic. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert rendered specifically for that purpose. Given what's on the other end of that connection, the asymmetry is intentional, not an oversight.
 ## What's protected
 - **Transport** — every Vault Agent ↔ Vault connection is TLS, certificate-verified.
 - **Secrets at rest** — KV v2, encrypted by Vault's own storage layer; rendered files on disk are tightly permissioned, root-owned, gitignored.
 - **Audit trail** — a file audit device logs every request/response. Secret values are HMAC-hashed in the log, not plaintext — what's visible is metadata: which token, which path, when.
 - **Alerting** — a monitoring check tails new audit log lines and pushes an alert through the fleet's existing notification pipeline when the root token gets used, or when permission-denied responses come in bursts.
 - **Certificate renewal reliability** — a dependent service's own TLS bundle is rebuilt automatically whenever its cert gets re-issued, guarded to skip unless the cert has under a week left. This closed a real prior gap: the rebuild never used to happen automatically, and an expired certificate was silently served for over two weeks before it was caught.
 ## Known gaps, ranked by actual risk
 1. **The bootstrap root token never expires.** A single long-lived token with unlimited access mints every AppRole and writes every policy. It's encrypted at rest, but encryption doesn't help if the plaintext ever gets used somewhere it shouldn't. This is the single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL provisioning token.
 2. **No SSH certificate signing.** Every host's SSH access key is static, baked in once at image build time, never rotated, with no revocation path short of rebuilding the host. Vault's SSH secrets engine (short-lived signed certs instead of static keys) is the natural fit — a bigger lift since it touches SSH config on every host, not just one config file.
 3. **Single-node storage, no replication.** One disk failure on the Vault host is a full outage and, without a recent backup, real data loss. An availability/durability concern, not a confidentiality one — deliberately lower priority than the two above.
 ## Deliberately accepted, not gaps
 - **Memory locking is disabled** — normally a real concern (secrets could swap to disk under memory pressure), but this host has zero swap configured, so there's no actual leak vector regardless.
 - **The Vault server's own config isn't managed by the same automation that manages every client-side Vault Agent** — set up manually, stays manual by choice.
 ## Access model
 One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration data, plus a private grant giving full read/write access only to that host's own secret namespace. When onboarding a new host, this shape gets copied rather than reinvented — and the shared grant does *not* automatically imply the private one, so both need verifying before assuming a host is fully provisioned.
 ---
 Related stacks: [`stacks/identity-and-security/vaultwarden/`](../stacks/identity-and-security/vaultwarden/)
@@ -0,0 +1,30 @@
 # Security Monitoring
 A self-built SOC stack: fleet-wide agent-based monitoring, a curated runbook for known-noisy alerts, and a compliance-hardening rollout with deliberately justified exceptions rather than blanket enforcement.
 ## Detection mechanisms, fleet-wide
 - **Security Configuration Assessment** — CIS-benchmark compliance scoring on a 12-hour interval, with a distinct benchmark policy per OS family. Scores below 100% are treated as an accepted, tracked tradeoff rather than an open bug to chase — re-litigating each individual control isn't the goal; the hardening rollout itself (below) is the actual control.
 - **File integrity monitoring** — enabled fleet-wide on a 12-hour scan cycle, the fleet's only FIM mechanism. A small number of hosts have specific paths excluded for known high-churn or false-positive-prone locations (e.g. a hypervisor's own live cluster-state files).
 - **Rootkit/anomaly heuristics** — a handful of container-hosted machines have specific exclusions for container pseudo-filesystem paths that the heuristic otherwise misreads — a known false-positive class for containerized hosts, not a suppressed real finding. Signature-file suppressions are per-agent, not centrally synced, so a suppression on one host deliberately doesn't apply anywhere else.
 ## A curated "don't re-investigate this" runbook
 Rather than re-triaging the same recurring alert from scratch every time it fires, routine alerts get root-caused once and documented — daemon lifecycle noise, expected package-manager/kernel-update file changes, a standard antivirus test-file trigger, one-off transient full-disk events on scratch mounts, and several others each with their actual cause on record rather than a blanket dismissal. One example worth calling out: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive in the *check itself* — it was grepping for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall.
 ## Querying and testing directly
 Alert data lives in a searchable index, queryable directly without going through the dashboard — a single aggregation query gives a fleet-wide "what's actually firing and where" summary in one call. Two gotchas worth knowing before querying: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query needs converting first or it silently returns zero hits (indistinguishable at a glance from an actual ingestion outage); and the manager's container name doesn't match the product's own default name, easy to get wrong when scripting against it.
 Custom detection rules get tested before rollout by piping a synthetic log line into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.
 ## Compliance hardening: justified exceptions, not blanket enforcement
 A multi-role hardening framework is rolled out fleet-wide, with a small number of deliberate, reasoned exceptions rather than force-applying every control regardless of fit — one control that simply doesn't apply inside a container gets skipped there; another needs a disk layout feature not every host has; a third only makes sense on always-on virtual machines. The authentication stack is configured to fail safe under a specific failure mode rather than fail open, a subtler design choice than it sounds.
 ## Automated benchmarking: native, not containerized
 The fleet's automated security-benchmark tool runs as a native binary rather than through its own official container image, after that image hit a real binary-compatibility wall between two different C library implementations. It runs on a weekly schedule, deliberately timed against real observed cron-load data rather than an arbitrary "every night at midnight" default.
 ---
 Related stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)
@@ -0,0 +1,24 @@
 # SSO & Identity
 ## Dual-zone SSO registration
 Every app that needs single sign-on gets a provider + application pair registered *per reverse-proxy zone it's exposed on* — up to two registrations for the same app, one for the browser-facing zone and one for the internal service-to-service zone. About a dozen apps also use native OIDC/SAML instead of (or alongside) proxy-based forward-auth, where the upstream app supports it directly.
 The login flow is the same shape regardless of which app or which zone it's normally accessed through:
 1. User opens the app on the browser-facing zone.
 2. The app redirects the browser to the identity provider for the actual OAuth authorize/token exchange — and that exchange *always* happens against the internal zone's identity-provider address, even though the app itself is normally reached via the browser-facing zone.
 3. On success, the browser lands back on the app's browser-facing address.
 Every app-specific setting beyond this shape (token lifetime, scopes, flow type) is deliberately *not* generalized across apps — each app's own OIDC documentation gets read fresh rather than copying another app's provider config, since the details genuinely differ per vendor.
 ## A real platform limitation, documented rather than hidden
 LDAP-based login through the identity provider's own outpost isn't viable for the primary superuser account: the outpost hides superuser accounts from LDAP directory search, so any app that requires a search-then-bind flow can never authenticate that account through LDAP. The two ways around it — creating a second non-admin account, or demoting the primary account from superuser — were both rejected as worse than the problem. Header-based forward-auth is used instead for apps in that situation, which sidesteps the LDAP search-then-bind requirement entirely.
 ## Git access: least privilege, not the admin account
 CI and automation pushes to the internal Git server run through a dedicated, non-admin bot account with scoped write access — not through an administrator identity. Other hosts that only need to *read* (pulling code, not pushing) get separate read-only deploy keys instead of any shared credential. This keeps a compromised automation credential from ever being an admin credential.
 ---
 Related stacks: [`stacks/identity-and-security/authentik/`](../stacks/identity-and-security/authentik/), [`stacks/platform/gitea/`](../stacks/platform/gitea/)