thesabear
41a6f569b4
Mermaid diagrams up front, GFM alert callouts for key decisions and gotchas, tables instead of bullet walls - replaces plain prose with the visual template established for security-monitoring.md.
4.6 KiB
🔑 Secret Distribution Model
Note
How a secret value moves from being written into Vault to being consumed by a running container. Four distinct flows, each with its own explicit contract — directional flow and authority boundaries, not configuration syntax.
flowchart TD
Op["Operator"] -->|1: writes cert/secret request| Vault["Vault"]
Vault -->|2: Certificate Flow| Cert["Short-lived cert<br/>rendered to disk"]
Vault -->|3: App Secret Flow| Sec["Rendered secret file<br/>0600, gitignored"]
Repo["Provisioning repo"] -->|4: Config Distribution Flow| Vault
Op2["Operator"] -.5: Administrative Secret Flow<br/>separate system.-> EncFile["Encrypted file<br/>in the repo itself"]
Cert --> Container["Container reads<br/>local disk only"]
Sec --> Container
1️⃣ Certificate Flow
| Law | Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines. |
| Flow | Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host |
| Failure behavior | If Vault is unreachable, existing certificates keep working until they expire — no immediate outage, a slow-building one if the outage outlasts the shortest cert's TTL |
Invariants: certificates are short-lived and re-issued on Vault Agent's own schedule, never on-demand by the consumer. A host only ever receives certs tagged for its own identity. The private key never traverses the network after issuance.
2️⃣ Application Secret Flow
| Law | A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored). |
| Flow | Operator writes value → Vault KV → Vault Agent → rendered file, 0600 → container reads via env file or bind-mount |
| Failure behavior | If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected — a queued rotation just doesn't propagate until Vault is reachable again |
Warning
The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it.
3️⃣ Configuration Distribution Flow
| Law | The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc. |
| Flow | Operator edits the template inventory in the provisioning repo → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts, picks up the new template list |
| Failure behavior | If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on last-known-good values |
Important
A missing or malformed template referenced by a host's own config crashes that host's entire Vault Agent process — every other secret/cert that host was already rendering stops updating too, not just the one broken template. This is why a config redeploy always requires confirming the underlying change actually landed first.
Shared-secret variant: not every template is host-scoped — one credential needed by every docker host for its own registry pushes is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one.
4️⃣ Administrative Secret Flow — a separate system entirely
| Law | Not every secret in this fleet goes through Vault. Admin/tooling credentials used by playbooks and operator sessions live in a deliberately simpler, separate system: encrypted directly into the provisioning repo, decrypted on demand by whoever holds the passphrase. |
| Flow | Operator encrypts value locally → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access |
| Failure behavior | Not applicable in the live-service sense — nothing here is consumed by a running container. A lost passphrase blocks automation runs and admin tooling, not any live application. |
Tip
Why two systems, not one: Vault KV is for runtime/service/pipeline secrets a running container needs; the encrypted-file system is for secrets used by tooling that operates on the fleet rather than secrets consumed by the fleet's own running services. Consumed by a container at runtime → Vault KV. Used by a human or an automation job doing admin work → the encrypted file.
Related: docs/secrets-and-pki.md