thesabear
d5f37b5787
Secrets & PKI, Secret Distribution Model, SSO & Identity, Firewall & Network Segmentation, DNS Naming Model, and Security Monitoring - the first batch of per-system write-ups linked from the new README, sourced and sanitized from the internal KB.
5.3 KiB
Secret Distribution Model
How a secret value moves from being written into Vault to being consumed by a running container. Four distinct flows, each with its own explicit contract — directional flow and authority boundaries, not configuration syntax.
1. Certificate Flow
Law — Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines.
Flow — Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host.
Invariants
- Certificates are short-lived and re-issued on Vault Agent's own schedule, not on-demand by the consuming service.
- A host only ever receives certificates tagged for its own service identity.
- The private key never traverses the network after issuance — written directly to local disk by the agent that requested it.
Failure behavior — If Vault is unreachable, existing certificates keep working until they expire. No immediate outage; a slow-building one if the outage persists past the shortest cert's TTL.
2. Application Secret Flow
Law — A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored).
Flow — Operator writes value → Vault KV → Vault Agent (polling/watching) → rendered file on local disk → container reads it via an env file or a direct bind-mount.
Invariants
- The git-tracked template contains placeholder syntax only — a real secret value anywhere in this flow's git history is a bug, not a variant.
- The rendered file is owned entirely by Vault Agent. Hand-editing it gets overwritten on the next render cycle and hides the actual source of truth.
- A container consumes the rendered file from its own host's disk — it never reaches across the network to fetch a secret itself.
Failure behavior — If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected. A secret rotation queued during an outage simply doesn't propagate until Vault Agent can reach Vault again — no service disruption, just a delayed rotation.
3. Configuration Distribution Flow
Law — The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc.
Flow — Operator edits the template inventory in the provisioning repo → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts and picks up the new template list.
Invariants
- A host's Vault Agent only ever renders templates tagged for its own identity — the full template list is visible to every host, but each host filters to its own subset.
- A missing or malformed template referenced by a host's own config crashes that host's entire Vault Agent process, not just the one broken template — every other secret/cert that host was already rendering stops updating too. This is why a config redeploy always requires confirming the underlying change actually landed before assuming it worked.
Failure behavior — If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on their last-known-good values until the config is fixed and Vault Agent restarts successfully.
Shared-secret variant — not every template is host-scoped. One credential needed by every docker host for its own registry pushes is templated from a single shared path rather than a per-host one — same mechanics as the rest of this flow, just a shared source instead of a private one.
4. Administrative Secret Flow (a separate system entirely)
Law — Not every secret in this fleet goes through Vault. Admin and tooling credentials used by playbooks and operator sessions themselves (third-party API tokens, login credentials for internal tools) live in a completely separate, deliberately simpler system: encrypted directly into the provisioning repo using a standard vault-encryption tool, decrypted on demand by whoever holds the passphrase.
Flow — Operator encrypts value locally → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access (an automation run, or a human operator).
Why two systems, not one — Vault KV is for runtime/service/pipeline secrets a running container or live service needs; the simpler encrypted-file system is for secrets used by tooling that operates on the fleet rather than secrets consumed by the fleet's own running services. A reader who only knows the Vault-KV flow above would have no way to tell which system a given secret belongs to — consumed by a container at runtime means Vault KV; used by a human or an automation job doing administrative work means the encrypted-file system.
Invariants
- The encrypted blob is a single opaque ciphertext per file — searching for a variable name won't find it; only a full decrypt reveals what's stored.
- Unlike Vault KV, there's no per-host scoping or automatic rendering — whoever has the passphrase can read every secret in the file.
Failure behavior — Not applicable in the live-service sense; this isn't consumed by running containers. A lost passphrase would block automation runs and admin tooling needing those credentials, not any live application.
Related: docs/secrets-and-pki.md