thesabear/public
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: rework security/hardening batch with diagrams and callouts

Browse Source 
Mermaid diagrams up front, GFM alert callouts for key decisions and
gotchas, tables instead of bullet walls - replaces plain prose with
the visual template established for security-monitoring.md.
This commit is contained in:
thesabear committed 2026-08-21 23:17:10 +03:00
1 parent a4c334b3d8
commit 41a6f569b4
6 files changed
+247 -114
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/dns-naming.md
+19 -9
View File
@@ -1,23 +1,33 @@
# DNS Naming Model
# 🌐 DNS Naming Model
## Two zones, deliberately never conflated
> [!NOTE]
> Two internal DNS zones, mirroring each other structurally but serving completely different traffic — deliberately never conflated.
Internal DNS is authoritative for two zones that mirror each other structurally but serve different traffic:
```mermaid
flowchart LR
Browser["Browser / User"] -->|Zone A| Edge["User-facing reverse proxy<br/>mTLS + SSO enforced"]
Service["Another service"] -->|Zone B| Internal["Internal reverse proxy<br/>trusted-IP allowlist"]
Public["Public internet"] -.bypasses both zones.-> EdgeFW["Edge firewall's own<br/>public reverse proxy"]
Edge --> App["App"]
Internal --> App
EdgeFW --> PublicApp["The one genuinely<br/>public-facing app"]
```
| Zone | Serves | Enforcement |
|---|---|---|
| Browser-facing zone | Users/browsers, via the user-facing reverse proxy | Mutual TLS + SSO enforced at the edge |
| Service-to-service zone | App-to-app traffic (webhooks, API integrations), via the internal reverse proxy | Trusted-IP allowlist, no mTLS |
| Service-to-service zone | App-to-app traffic (webhooks, API integrations) | Trusted-IP allowlist, no mTLS |
Most apps get one hostname per zone, resolving to a different front door depending on who's asking — a browser gets the mTLS-enforced path, another service gets the lighter-weight internal path. The two zones aren't perfectly identical at any given moment; a handful of apps exist in only one zone rather than both, which is expected — not every stack needs both zones, driven by actual need rather than blanket symmetry.
Most apps get one hostname per zone, resolving to a different front door depending on who's asking. The two zones aren't perfectly symmetric at any given moment — a handful of apps exist in only one, which is expected: not every stack needs both zones, driven by actual need rather than blanket symmetry.
## The public exception
One app is a deliberate exception to the whole two-zone model: it's genuinely public-facing, routed directly through the edge firewall's own reverse-proxy package, and has zero presence in either internal zone. Apps meant for the public internet skip the internal DNS model entirely rather than getting a third, redundant record.
> [!TIP]
> **The public exception**: one app is genuinely public-facing, routed directly through the edge firewall's own reverse-proxy package, with zero presence in either internal zone. Apps meant for the public internet skip the internal DNS model entirely rather than getting a redundant third record.
## Caught during a later audit
A later documentation audit caught real drift from what had been assumed: an older note claimed an external DNS forwarder (a third-party provider) was configured for anything outside the two internal zones. Live verification found no such forwarder actually configured — unresolved queries forward to the edge firewall and a couple of standard public resolvers instead, and internal clients are blocked from using any other resolver directly. The takeaway kept from that audit: documentation drift is caught by checking live state, not by assuming a prior note was still accurate.
> [!IMPORTANT]
> A documentation audit caught real drift: an older note claimed an external DNS forwarder was configured for anything outside the two internal zones. Live verification found no such forwarder actually in place — unresolved queries forward to the edge firewall and a couple of standard public resolvers instead, and internal clients are blocked from using any other resolver directly. The takeaway: documentation drift gets caught by checking live state, not by trusting a prior note was still accurate.
---
Related stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/)
docs/firewall-and-network.md
+41 -16
View File
@@ -1,34 +1,59 @@
# Firewall & Network Segmentation
# 🧱 Firewall & Network Segmentation
## No blanket LAN-to-LAN allow
> [!NOTE]
> No blanket "anything on the internal network can talk to anything else on it" rule, anywhere. Every real flow is explicit; everything else is logged and dropped.
The most important rule in the whole model is a negative one: there is no default "anything on the internal network can talk to anything else on it" rule. Every real service-to-service flow — DNS lookups, secrets/PKI traffic, reverse-proxy access, metrics/log shipping, security-agent reporting, NTP sync, admin SSH, file sharing, and a handful of specific app-to-app API integrations — gets its own explicit allow rule. Anything not explicitly listed hits a default-deny-and-log catch-all at the bottom of the ruleset. This catch-all is the fleet's actual lateral-movement boundary, not any single host's own firewall (see below).
```mermaid
flowchart TD
Internet(["Internet"]) --> Edge
subgraph Edge["Edge Firewall"]
Bogon["Bogon block +<br/>IP-reputation blacklist"]
NonstdSSH["SSH — nonstandard port only"]
PubProxy["Public reverse-proxy layer"]
IDS["Inline IDS/IPS"]
end
Edge -->|explicit rules only| LAN
A few of the explicit rules are narrower than they look at first glance, on purpose — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet.
subgraph LAN["Internal Network"]
direction TB
Rule1["DNS · Secrets/PKI · Reverse proxy<br/>Metrics/logs · Security agents<br/>NTP · Admin SSH · File sharing"]
Catchall["Everything else:<br/>default-deny + log"]
end
```
## Firewall objects are named groups, not raw addresses
## The one rule that matters most
Every rule references named groups instead of hardcoded addresses — adding a host to a group is a single membership edit rather than hunting down and updating every rule that needs to reach it. Groups nest: a "the whole managed fleet" group is composed of several more specific per-role groups, so a rule written against the broad group automatically covers new members added to any of the narrower ones. One group is deliberately scoped *narrower* than its name might suggest — it only covers the fleet's actively-managed tier, and anything outside that tier has to be added on purpose if it needs the same access.
Every real service-to-service flow — DNS lookups, secrets/PKI traffic, reverse-proxy access, metrics/log shipping, security-agent reporting, NTP, admin SSH, file sharing, and a handful of specific app-to-app integrations — gets its own explicit allow rule. Anything not explicitly listed hits a default-deny-and-log catch-all. **This catch-all is the fleet's actual lateral-movement boundary**, not any single host's own firewall.
> [!TIP]
> A few explicit rules are narrower than they look on purpose — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet.
## Named groups, not raw addresses
Every rule references named groups instead of hardcoded addresses — adding a host to a group is a single membership edit, not a hunt through every rule that needs to reach it. Groups nest: a "whole managed fleet" group is composed of several narrower per-role groups, so a rule against the broad group automatically covers new members of any narrower one.
## Per-host firewalls: minimal, not blanket
On top of the network-level segmentation above, each host's own local firewall opens only the specific ports that host actually needs — not a blanket "allow everything, block nothing" default. One host takes this further: instead of one open port for its most sensitive service, individual source IPs get their own explicit rule, so a compromised unrelated service on the same network still can't reach it.
Each host's own local firewall opens only the specific ports it actually needs. One host goes further: instead of one open port for its most sensitive service, individual source IPs get their own explicit rule.
**A real, documented gap, not glossed over:** on Docker hosts, container-published ports bypass the host's own firewall rules entirely — the container runtime's own network rules are evaluated first, so the host firewall only actually protects the small number of *native* (non-containerized) services running there, not the dozens of containerized apps. The network-level segmentation above (and the edge firewall further out) still applies regardless — this is a defense-in-depth gap, not an open-to-the-world one.
> [!WARNING]
> **A real, documented gap**: on Docker hosts, container-published ports bypass the host's own firewall entirely — the container runtime's own network rules are evaluated first. Per-host firewall rules only actually protect the small number of *native*, non-containerized services on that host. The network-level segmentation above (and the edge firewall further out) still applies regardless — defense-in-depth gap, not an open-to-the-world one.
## Edge firewall
## Edge firewall: three jobs, one device
The edge firewall is the fleet's outermost boundary and does three separate jobs:
```mermaid
flowchart LR
EdgeFW["Edge Firewall"] --> J1["Public ingress<br/>bogon block, IP-reputation feed,<br/>nonstandard SSH port"]
EdgeFW --> J2["Second reverse-proxy layer<br/>for genuinely public apps only,<br/>automated wildcard certs"]
EdgeFW --> J3["Security tooling<br/>inline IDS/IPS, cert renewal"]
```
- **Public ingress** — a bogon-address block, an IP-reputation blacklist feed, SSH exposed only on a nonstandard port, and HTTPS/HTTP forwarded to the fleet's public-facing reverse-proxy layer. Nothing else reaches the edge firewall's public interface at all.
- **A second, independent reverse-proxy layer** — separate from the internal two-zone model covered in [DNS Naming Model](dns-naming.md), the edge firewall runs its own reverse-proxy package specifically for the small number of apps that are genuinely meant to be public. Each public app gets its own backend rule mapping its public hostname to the correct internal service, with automated wildcard certificate issuance.
- **Security tooling** — an inline intrusion detection/prevention engine, the same IP-reputation service driving the WAN blacklist, and automated certificate renewal all run at this layer.
**Failure impact, thought through in advance:** if the edge firewall goes down, public access and outbound internet access both go with it — but every internal service that doesn't depend on WAN reachability keeps running normally. The blast radius is deliberately bounded to "the edge," not "the fleet."
> [!IMPORTANT]
> **Failure impact, thought through in advance**: if the edge firewall goes down, public access and outbound internet both go with it — but every internal service that doesn't depend on WAN reachability keeps running normally. The blast radius is deliberately bounded to "the edge," not "the fleet."
## The hypervisor's own firewall: deliberately unused
The virtualization platform underneath everything has its own built-in firewall capability. It's left with an empty ruleset on purpose — the edge firewall and each host's own local firewall already cover the two layers that actually matter, and a third overlapping layer would add complexity without meaningfully improving the security posture. Recognizing when *not* to add a redundant control is as much a design decision as adding one.
The virtualization platform underneath everything has its own built-in firewall capability, left with an empty ruleset on purpose. The edge firewall and each host's own local firewall already cover the two layers that matter — a third overlapping layer would add complexity without meaningfully improving posture. Recognizing when *not* to add a redundant control is as much a design decision as adding one.
---
Related stacks: [`stacks/platform/caddy/`](../stacks/platform/caddy/), [`stacks/platform/caddy-internal/`](../stacks/platform/caddy-internal/), [`stacks/platform/docker-socket-proxy/`](../stacks/platform/docker-socket-proxy/)
docs/secret-distribution.md
+45 -41
View File
@@ -1,60 +1,64 @@
# Secret Distribution Model
# 🔑 Secret Distribution Model
How a secret value moves from being written into Vault to being consumed by a running container. Four distinct flows, each with its own explicit contract — directional flow and authority boundaries, not configuration syntax.
> [!NOTE]
> How a secret value moves from being written into Vault to being consumed by a running container. Four distinct flows, each with its own explicit contract — directional flow and authority boundaries, not configuration syntax.
## 1. Certificate Flow
```mermaid
flowchart TD
Op["Operator"] -->|1: writes cert/secret request| Vault["Vault"]
Vault -->|2: Certificate Flow| Cert["Short-lived cert<br/>rendered to disk"]
Vault -->|3: App Secret Flow| Sec["Rendered secret file<br/>0600, gitignored"]
Repo["Provisioning repo"] -->|4: Config Distribution Flow| Vault
Op2["Operator"] -.5: Administrative Secret Flow<br/>separate system.-> EncFile["Encrypted file<br/>in the repo itself"]
**Law** — Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines.
Cert --> Container["Container reads<br/>local disk only"]
Sec --> Container
```
**Flow** — Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host.
## 1️⃣ Certificate Flow
**Invariants**
- Certificates are short-lived and re-issued on Vault Agent's own schedule, not on-demand by the consuming service.
- A host only ever receives certificates tagged for its own service identity.
- The private key never traverses the network after issuance — written directly to local disk by the agent that requested it.
| | |
|---|---|
| **Law** | Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines. |
| **Flow** | Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host |
| **Failure behavior** | If Vault is unreachable, existing certificates keep working until they expire — no immediate outage, a slow-building one if the outage outlasts the shortest cert's TTL |
**Failure behavior** — If Vault is unreachable, existing certificates keep working until they expire. No immediate outage; a slow-building one if the outage persists past the shortest cert's TTL.
**Invariants:** certificates are short-lived and re-issued on Vault Agent's own schedule, never on-demand by the consumer. A host only ever receives certs tagged for its own identity. The private key never traverses the network after issuance.
## 2. Application Secret Flow
## 2️⃣ Application Secret Flow
**Law** — A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored).
| | |
|---|---|
| **Law** | A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored). |
| **Flow** | Operator writes value → Vault KV → Vault Agent → rendered file, `0600` → container reads via env file or bind-mount |
| **Failure behavior** | If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected — a queued rotation just doesn't propagate until Vault is reachable again |
**Flow** — Operator writes value → Vault KV → Vault Agent (polling/watching) → rendered file on local disk → container reads it via an env file or a direct bind-mount.
> [!WARNING]
> The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it.
**Invariants**
- The git-tracked template contains placeholder syntax only — a real secret value anywhere in this flow's git history is a bug, not a variant.
- The rendered file is owned entirely by Vault Agent. Hand-editing it gets overwritten on the next render cycle and hides the actual source of truth.
- A container consumes the rendered file from its own host's disk — it never reaches across the network to fetch a secret itself.
## 3️⃣ Configuration Distribution Flow
**Failure behavior** — If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected. A secret rotation queued during an outage simply doesn't propagate until Vault Agent can reach Vault again — no service disruption, just a delayed rotation.
| | |
|---|---|
| **Law** | The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc. |
| **Flow** | Operator edits the template inventory in the provisioning repo → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts, picks up the new template list |
| **Failure behavior** | If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on last-known-good values |
## 3. Configuration Distribution Flow
> [!IMPORTANT]
> A missing or malformed template referenced by a host's own config crashes that host's **entire** Vault Agent process — every other secret/cert that host was already rendering stops updating too, not just the one broken template. This is why a config redeploy always requires confirming the underlying change actually landed first.
**Law** — The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc.
**Shared-secret variant:** not every template is host-scoped — one credential needed by every docker host for its own registry pushes is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one.
**Flow** — Operator edits the template inventory in the provisioning repo → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts and picks up the new template list.
## 4️⃣ Administrative Secret Flow — a separate system entirely
**Invariants**
- A host's Vault Agent only ever renders templates tagged for its own identity — the full template list is visible to every host, but each host filters to its own subset.
- A missing or malformed template referenced by a host's own config crashes that host's *entire* Vault Agent process, not just the one broken template — every other secret/cert that host was already rendering stops updating too. This is why a config redeploy always requires confirming the underlying change actually landed before assuming it worked.
| | |
|---|---|
| **Law** | Not every secret in this fleet goes through Vault. Admin/tooling credentials used by playbooks and operator sessions live in a deliberately simpler, separate system: encrypted directly into the provisioning repo, decrypted on demand by whoever holds the passphrase. |
| **Flow** | Operator encrypts value locally → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access |
| **Failure behavior** | Not applicable in the live-service sense — nothing here is consumed by a running container. A lost passphrase blocks automation runs and admin tooling, not any live application. |
**Failure behavior** — If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on their last-known-good values until the config is fixed and Vault Agent restarts successfully.
**Shared-secret variant** — not every template is host-scoped. One credential needed by every docker host for its own registry pushes is templated from a single shared path rather than a per-host one — same mechanics as the rest of this flow, just a shared source instead of a private one.
## 4. Administrative Secret Flow (a separate system entirely)
**Law** — Not every secret in this fleet goes through Vault. Admin and tooling credentials used by playbooks and operator sessions themselves (third-party API tokens, login credentials for internal tools) live in a completely separate, deliberately simpler system: encrypted directly into the provisioning repo using a standard vault-encryption tool, decrypted on demand by whoever holds the passphrase.
**Flow** — Operator encrypts value locally → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access (an automation run, or a human operator).
**Why two systems, not one** — Vault KV is for runtime/service/pipeline secrets a running container or live service needs; the simpler encrypted-file system is for secrets used by tooling that *operates on* the fleet rather than secrets *consumed by* the fleet's own running services. A reader who only knows the Vault-KV flow above would have no way to tell which system a given secret belongs to — consumed by a container at runtime means Vault KV; used by a human or an automation job doing administrative work means the encrypted-file system.
**Invariants**
- The encrypted blob is a single opaque ciphertext per file — searching for a variable name won't find it; only a full decrypt reveals what's stored.
- Unlike Vault KV, there's no per-host scoping or automatic rendering — whoever has the passphrase can read every secret in the file.
**Failure behavior** — Not applicable in the live-service sense; this isn't consumed by running containers. A lost passphrase would block automation runs and admin tooling needing those credentials, not any live application.
> [!TIP]
> **Why two systems, not one**: Vault KV is for runtime/service/pipeline secrets a running container needs; the encrypted-file system is for secrets used by tooling that *operates on* the fleet rather than secrets *consumed by* the fleet's own running services. Consumed by a container at runtime → Vault KV. Used by a human or an automation job doing admin work → the encrypted file.
---
Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md)
docs/secrets-and-pki.md
+47 -25
View File
@@ -1,23 +1,44 @@
# Secrets & PKI
# 🔐 Secrets & PKI
A single Vault cluster serves as both the internal certificate authority and the secret store for the whole fleet — one trust root, not two parallel systems.
> [!NOTE]
> A single Vault cluster serves as both the internal certificate authority and the secret store for the whole fleet — one trust root, not two parallel systems.
| | |
|---|---|
| **Role** | Certificate authority + secret store |
| **Distribution** | Vault Agent on every host, AppRole auth, renders locally |
| **Replaces** | A prior plaintext-`.env`-in-git secrets model |
| **Biggest known risk** | A non-expiring root token — ranked, tracked, not ignored |
## What it does
- **Certificate authority** — two PKI mounts: one for public-facing wildcard certs, one for internal mTLS certs. Both issue short-lived certificates to the hosts that need them.
- **Secret store** — a KV v2 mount holds every migrated application's real secret values (database passwords, API tokens, signing keys), one namespace per host.
- **Secret store** — a KV v2 mount holds every migrated application's real secret values, one namespace per host.
- **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly.
## Why it's built this way
Before this existed, every stack's secrets sat in plaintext inside git-tracked `.env` files — readable by anyone with repo access, and permanently baked into git history even after being "fixed." The replacement model:
```mermaid
flowchart LR
Before["Before:<br/>plaintext secrets<br/>in git-tracked .env files"] -->|replaced| After
subgraph After["Now"]
direction TB
T["Git-tracked template<br/>placeholder syntax only"]
V["Vault<br/>real value, one place"]
R["Rendered file<br/>gitignored, 0600, root-owned"]
T -.no real value ever.-> T
V -->|renders| R
end
```
- The real secret value lives in exactly one place: Vault.
- What's git-tracked is a template with placeholder syntax, never a real value.
- The rendered file with the real value is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
- The rendered file is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
- If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle.
The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before this was extended to cover application secrets — reusing a trust model that was already proven, rather than inventing a second one.
> [!TIP]
> The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven, rather than inventing a second one.
## Architecture
@@ -41,37 +62,38 @@ flowchart TD
Proxy --> A6
```
### Two authentication flows, not one
**Two authentication flows, not one:**
- **AppRole auth** (every host, ongoing): Vault Agent → role_id + secret_id → Vault → short-lived token, cached locally, renewed automatically.
- **Bootstrap auth** (rare, human-initiated): an operator uses a root token to mint new AppRole credentials when onboarding a host, or to create/update policies. Not part of any workload's runtime path.
| Flow | When | Path |
|---|---|---|
| AppRole auth | Every host, ongoing | Vault Agent → role_id + secret_id → Vault → short-lived token, cached, auto-renewed |
| Bootstrap auth | Rare, human-initiated | Operator uses a root token to mint new AppRole credentials or update policies — never part of any workload's runtime path |
### The one deliberately-asymmetric trust decision
Every other backend connection inside the internal app-to-app zone skips certificate verification on the reverse proxy hop (it still encrypts the connection, just doesn't check whose cert it's talking to) — a pragmatic tradeoff for low-stakes internal traffic. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert rendered specifically for that purpose. Given what's on the other end of that connection, the asymmetry is intentional, not an oversight.
> [!IMPORTANT]
> **The one deliberately-asymmetric trust decision**: every other backend connection inside the internal zone skips certificate verification on the reverse proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real. Given what's on the other end of that connection, the asymmetry is intentional, not an oversight.
## What's protected
- **Transport** — every Vault Agent ↔ Vault connection is TLS, certificate-verified.
- **Secrets at rest** — KV v2, encrypted by Vault's own storage layer; rendered files on disk are tightly permissioned, root-owned, gitignored.
- **Audit trail** — a file audit device logs every request/response. Secret values are HMAC-hashed in the log, not plaintext — what's visible is metadata: which token, which path, when.
- **Alerting** — a monitoring check tails new audit log lines and pushes an alert through the fleet's existing notification pipeline when the root token gets used, or when permission-denied responses come in bursts.
- **Certificate renewal reliability** — a dependent service's own TLS bundle is rebuilt automatically whenever its cert gets re-issued, guarded to skip unless the cert has under a week left. This closed a real prior gap: the rebuild never used to happen automatically, and an expired certificate was silently served for over two weeks before it was caught.
| Layer | Protection |
|---|---|
| Transport | Every Vault Agent ↔ Vault connection is TLS, certificate-verified |
| Secrets at rest | KV v2, encrypted by Vault's storage layer; rendered files tightly permissioned, root-owned, gitignored |
| Audit trail | Every request/response logged; secret values HMAC-hashed, not plaintext |
| Alerting | Root-token use or permission-denied bursts trigger an alert through the fleet's notification pipeline |
| Cert renewal | A dependent service's TLS bundle auto-rebuilds when its cert re-issues — closed a real gap where an expired cert was silently served for over two weeks |
## Known gaps, ranked by actual risk
1. **The bootstrap root token never expires.** A single long-lived token with unlimited access mints every AppRole and writes every policy. It's encrypted at rest, but encryption doesn't help if the plaintext ever gets used somewhere it shouldn't. This is the single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL provisioning token.
2. **No SSH certificate signing.** Every host's SSH access key is static, baked in once at image build time, never rotated, with no revocation path short of rebuilding the host. Vault's SSH secrets engine (short-lived signed certs instead of static keys) is the natural fit — a bigger lift since it touches SSH config on every host, not just one config file.
3. **Single-node storage, no replication.** One disk failure on the Vault host is a full outage and, without a recent backup, real data loss. An availability/durability concern, not a confidentiality one — deliberately lower priority than the two above.
1. **The bootstrap root token never expires.** Unlimited access, mints every AppRole, writes every policy. Encrypted at rest, but that doesn't help if the plaintext ever gets used somewhere it shouldn't. The single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL token.
2. **No SSH certificate signing.** Every host's SSH key is static, baked in at image build time, no rotation, no revocation short of rebuilding the host. Vault's SSH secrets engine is the natural fit — a bigger lift since it touches every host's SSH config.
3. **Single-node storage, no replication.** One disk failure is a full outage and, without a recent backup, real data loss. An availability concern, not a confidentiality one — deliberately lower priority than the two above.
## Deliberately accepted, not gaps
- **Memory locking is disabled** — normally a real concern (secrets could swap to disk under memory pressure), but this host has zero swap configured, so there's no actual leak vector regardless.
- **The Vault server's own config isn't managed by the same automation that manages every client-side Vault Agent** — set up manually, stays manual by choice.
> [!TIP]
> **Deliberately accepted, not gaps**: memory locking is disabled (normally a real concern, but this host has zero swap, so there's no actual leak vector) — and the Vault server's own config is intentionally kept out of the same automation that manages every client-side Vault Agent.
## Access model
One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration data, plus a private grant giving full read/write access only to that host's own secret namespace. When onboarding a new host, this shape gets copied rather than reinvented — and the shared grant does *not* automatically imply the private one, so both need verifying before assuming a host is fully provisioned.
One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does *not* automatically imply the private one, so both get verified before assuming a host is fully provisioned.
---
Related stacks: [`stacks/identity-and-security/vaultwarden/`](../stacks/identity-and-security/vaultwarden/)
docs/security-monitoring.md
+60 -12
View File
@@ -1,30 +1,78 @@
# Security Monitoring
# 🛡️ Security Monitoring
A self-built SOC stack: fleet-wide agent-based monitoring, a curated runbook for known-noisy alerts, and a compliance-hardening rollout with deliberately justified exceptions rather than blanket enforcement.
> [!NOTE]
> A self-built SOC stack: fleet-wide agent monitoring, a curated runbook for known-noisy alerts, and compliance hardening rolled out with deliberately justified exceptions instead of blanket enforcement.
## Detection mechanisms, fleet-wide
| | |
|---|---|
| **Coverage** | Every fleet host, agent-based, no gaps |
| **Detection layers** | Compliance scoring · file integrity monitoring · rootkit/anomaly heuristics |
| **Runbook** | 10+ recurring alert types root-caused once, documented, never re-triaged from scratch |
| **Query path** | Direct index queries, bypassing the dashboard entirely when needed |
- **Security Configuration Assessment** — CIS-benchmark compliance scoring on a 12-hour interval, with a distinct benchmark policy per OS family. Scores below 100% are treated as an accepted, tracked tradeoff rather than an open bug to chase — re-litigating each individual control isn't the goal; the hardening rollout itself (below) is the actual control.
- **File integrity monitoring** — enabled fleet-wide on a 12-hour scan cycle, the fleet's only FIM mechanism. A small number of hosts have specific paths excluded for known high-churn or false-positive-prone locations (e.g. a hypervisor's own live cluster-state files).
- **Rootkit/anomaly heuristics** — a handful of container-hosted machines have specific exclusions for container pseudo-filesystem paths that the heuristic otherwise misreads — a known false-positive class for containerized hosts, not a suppressed real finding. Signature-file suppressions are per-agent, not centrally synced, so a suppression on one host deliberately doesn't apply anywhere else.
## How an alert gets handled
## A curated "don't re-investigate this" runbook
```mermaid
flowchart LR
Host["Fleet Host<br/>agent installed"] --> Checks
Rather than re-triaging the same recurring alert from scratch every time it fires, routine alerts get root-caused once and documented — daemon lifecycle noise, expected package-manager/kernel-update file changes, a standard antivirus test-file trigger, one-off transient full-disk events on scratch mounts, and several others each with their actual cause on record rather than a blanket dismissal. One example worth calling out: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive in the *check itself* — it was grepping for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall.
subgraph Checks["Detection layers"]
direction TB
SCA["Compliance scoring<br/>12h benchmark scan"]
FIM["File integrity monitoring<br/>12h scan cycle"]
RC["Rootkit / anomaly<br/>heuristics"]
end
Checks --> Alert["Alert fires"]
Alert --> Known{"Matches a known,<br/>root-caused pattern?"}
Known -->|yes| Runbook["Runbook entry<br/>no re-investigation"]
Known -->|no| Fresh["Investigated fresh<br/>logtest + direct index query"]
Fresh -->|becomes routine| Runbook
```
## The curated "don't re-investigate this" runbook
Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded:
| Category | What fires | Why it's routine |
|---|---|---|
| Daemon lifecycle | Agent start/stop, log rotation, PAM login/logout | Expected operational noise |
| Package/kernel updates | File-integrity "added/deleted" events | Syscheck working as intended during normal patching |
| Antivirus self-test | A known harmless AV test-file trigger | Standard test artifact, not a real detection |
| Transient disk-full | One-off scratch-mount fill events | Self-resolving cache fills, not a real capacity issue |
| SSH auth bursts from automation | Multiple-key offer during fleet-wide automation runs | One legitimate connection, not brute force |
| Container-runtime access-control denials | Isolated, non-flooding events | Known container/runtime interaction, not exploitation |
> [!TIP]
> **One catch worth calling out**: an alert claiming the firewall's default-deny policy wasn't detected turned out to be a false positive *in the check itself* — it searched for a raw config string that a different (but equally correct) firewall implementation simply never produces. The fix was verifying the real firewall posture directly, not touching the firewall. Chasing the alert instead of the actual policy would have wasted real time on a non-problem.
## Querying and testing directly
Alert data lives in a searchable index, queryable directly without going through the dashboard — a single aggregation query gives a fleet-wide "what's actually firing and where" summary in one call. Two gotchas worth knowing before querying: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query needs converting first or it silently returns zero hits (indistinguishable at a glance from an actual ingestion outage); and the manager's container name doesn't match the product's own default name, easy to get wrong when scripting against it.
Alert data lives in a searchable index, reachable without going through any dashboard — a single aggregation query returns a fleet-wide "what's actually firing and where" summary in one call.
Custom detection rules get tested before rollout by piping a synthetic log line into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.
> [!WARNING]
> Two gotchas that will burn a query if missed: alert timestamps are stored in UTC while the fleet runs in a different timezone, so a wall-clock-derived query silently returns zero hits instead of erroring — indistinguishable at a glance from an actual ingestion outage. And the manager's own container name doesn't match the product's default name, an easy miss when scripting against it.
Custom detection rules get tested *before* rollout, not after: a synthetic log line gets piped into the manager's own log-testing tool, which shows the pre-decoding, decoding, and rule-filtering phases separately — so a decoder match with no rule firing is immediately visible rather than a silent no-op.
## Compliance hardening: justified exceptions, not blanket enforcement
A multi-role hardening framework is rolled out fleet-wide, with a small number of deliberate, reasoned exceptions rather than force-applying every control regardless of fit — one control that simply doesn't apply inside a container gets skipped there; another needs a disk layout feature not every host has; a third only makes sense on always-on virtual machines. The authentication stack is configured to fail safe under a specific failure mode rather than fail open, a subtler design choice than it sounds.
```mermaid
flowchart TD
Framework["Multi-role hardening framework"] --> Apply{"Applies cleanly<br/>to this host?"}
Apply -->|yes| Enforce["Enforced fleet-wide"]
Apply -->|no, justified| Skip["Deliberately skipped<br/>reason documented"]
Skip --> R1["Control doesn't apply<br/>inside a container"]
Skip --> R2["Needs a disk layout<br/>not every host has"]
Skip --> R3["Only makes sense on<br/>always-on VMs"]
```
The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards.
## Automated benchmarking: native, not containerized
The fleet's automated security-benchmark tool runs as a native binary rather than through its own official container image, after that image hit a real binary-compatibility wall between two different C library implementations. It runs on a weekly schedule, deliberately timed against real observed cron-load data rather than an arbitrary "every night at midnight" default.
> [!IMPORTANT]
> The fleet's automated security-benchmark tool runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations. It runs weekly, deliberately timed against real observed load data rather than an arbitrary default schedule.
---
Related stacks: [`stacks/identity-and-security/wazuh/`](../stacks/identity-and-security/wazuh/)
docs/sso-and-identity.md
+35 -11
View File
@@ -1,24 +1,48 @@
# SSO & Identity
# 🪪 SSO & Identity
## Dual-zone SSO registration
> [!NOTE]
> Every app that needs single sign-on registers against the identity provider once per reverse-proxy zone it's exposed on — and a real platform limitation shapes how superuser accounts authenticate.
Every app that needs single sign-on gets a provider + application pair registered *per reverse-proxy zone it's exposed on* — up to two registrations for the same app, one for the browser-facing zone and one for the internal service-to-service zone. About a dozen apps also use native OIDC/SAML instead of (or alongside) proxy-based forward-auth, where the upstream app supports it directly.
## Login flow — same shape, every app
The login flow is the same shape regardless of which app or which zone it's normally accessed through:
```mermaid
sequenceDiagram
participant U as User
participant App as App (browser-facing zone)
participant IdP as Identity Provider (internal zone)
1. User opens the app on the browser-facing zone.
2. The app redirects the browser to the identity provider for the actual OAuth authorize/token exchange — and that exchange *always* happens against the internal zone's identity-provider address, even though the app itself is normally reached via the browser-facing zone.
3. On success, the browser lands back on the app's browser-facing address.
U->>App: Opens app
App->>U: Redirect to identity provider
U->>IdP: OAuth authorize/token exchange
Note over IdP: Always happens on the internal zone,<br/>even though the app itself lives<br/>on the browser-facing zone
IdP->>U: Success
U->>App: Lands back on the app
```
Every app-specific setting beyond this shape (token lifetime, scopes, flow type) is deliberately *not* generalized across apps — each app's own OIDC documentation gets read fresh rather than copying another app's provider config, since the details genuinely differ per vendor.
Every app-specific setting beyond this shape (token lifetime, scopes, flow type) is deliberately **not** generalized across apps — each app's own OIDC documentation gets read fresh rather than copying another app's provider config, since the details genuinely differ per vendor.
## A real platform limitation, documented rather than hidden
| | |
|---|---|
| **Registration pattern** | Provider + application pair per zone (up to 2 per app) |
| **Native OIDC/SAML apps** | ~12 apps, where the upstream supports it directly |
| **Everything else** | Proxy-based forward-auth |
LDAP-based login through the identity provider's own outpost isn't viable for the primary superuser account: the outpost hides superuser accounts from LDAP directory search, so any app that requires a search-then-bind flow can never authenticate that account through LDAP. The two ways around it — creating a second non-admin account, or demoting the primary account from superuser — were both rejected as worse than the problem. Header-based forward-auth is used instead for apps in that situation, which sidesteps the LDAP search-then-bind requirement entirely.
## A real limitation, documented rather than hidden
> [!WARNING]
> LDAP-based login through the identity provider's own outpost **isn't viable for the primary superuser account** — the outpost hides superuser accounts from LDAP directory search, so any app requiring a search-then-bind flow can never authenticate that account through LDAP. The two workarounds — a second non-admin account, or demoting the primary account from superuser — were both rejected as worse than the problem. Header-based forward-auth is used instead for apps in that situation.
## Git access: least privilege, not the admin account
CI and automation pushes to the internal Git server run through a dedicated, non-admin bot account with scoped write access — not through an administrator identity. Other hosts that only need to *read* (pulling code, not pushing) get separate read-only deploy keys instead of any shared credential. This keeps a compromised automation credential from ever being an admin credential.
```mermaid
flowchart LR
CI["CI / Automation"] -->|scoped write| Bot["Dedicated bot account<br/>non-admin"]
Bot --> Repo["Git server"]
Other["Other hosts"] -->|read-only| Deploy["Deploy keys<br/>per host"]
Deploy --> Repo
```
CI and automation pushes run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. Hosts that only need to *read* get separate read-only deploy keys instead of any shared credential. A compromised automation credential is never an admin credential.
---
Related stacks: [`stacks/identity-and-security/authentik/`](../stacks/identity-and-security/authentik/), [`stacks/platform/gitea/`](../stacks/platform/gitea/)