public/docs/secret-distribution.md

🔑 Secret Distribution Model

Note

docs/secrets-and-pki.md covers what Vault is and why it exists. This doc covers something different and, arguably, harder to get right: the actual contract for how a secret value moves from being written into Vault to being consumed by a running container — and why that contract is deliberately split into four separate flows instead of one universal one.

Why four flows instead of one

The obvious simpler design is one universal "secret goes in, secret comes out" pipeline. It was rejected because the four things being moved genuinely have different risk profiles and different failure tolerances: a certificate expiring is a slow-building problem measured in days; a missing application secret is often an immediate outage; a broken config-distribution push can silently stop all of a host's secrets from updating, not just one. Treating them identically would mean either over-engineering the simple cases or under-protecting the risky one. Four flows, each with its own explicit Law/Flow/Failure-behavior contract, is more code to maintain but far easier to reason about when something breaks — anyone debugging an incident knows immediately which contract governs the thing that's wrong.

1️⃣ Certificate Flow

Law	Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines.
Flow	Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host
Failure behavior	If Vault is unreachable, existing certificates keep working until they expire — no immediate outage, a slow-building one if the outage outlasts the shortest cert's TTL

Invariants: certificates are short-lived and re-issued on Vault Agent's own schedule, never on-demand by the consumer. A host only ever receives certs tagged for its own identity. The private key never traverses the network after issuance.

2️⃣ Application Secret Flow

Law	A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored).
Flow	Operator writes value → Vault KV → Vault Agent → rendered file, 0600 → container reads via env file or bind-mount
Failure behavior	If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected — a queued rotation just doesn't propagate until Vault is reachable again

Warning

The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it. This is the single most common way this model gets violated by accident: someone SSHes in to fix a value under time pressure, and the fix silently disappears on the next render.

3️⃣ Configuration Distribution Flow

Law	The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc.
Flow	Operator edits the template inventory in the provisioning repo (see docs/iac-pipeline.md) → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts, picks up the new template list
Failure behavior	If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on last-known-good values

Important

This is the flow with the sharpest failure mode in the whole model: a missing or malformed template referenced by a host's own config crashes that host's entire Vault Agent process — every other secret/cert that host was already rendering stops updating too, not just the one broken template. A single YAML typo in one app's template inventory can silently freeze every other app's secret rotation on that same host. This is why a config redeploy always requires confirming the underlying git change actually landed first, not just that the deploy command returned success.

That redeploy-and-restart sequence isn't just a claim — here's a real one, start to finish: Vault Agent restarts, authenticates, renders a fresh .env.secrets, and its reload_command fires docker compose up -d, which recreates and restarts the actual container:

Shared-secret variant: not every template is host-scoped — one credential needed by every docker host for its own registry pushes (see docs/registry-and-build.md) is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one — and it relies on every host's policy already granting read access to that shared path, the same "shared + private" pattern described in docs/secrets-and-pki.md's access model.

4️⃣ Administrative Secret Flow — a deliberately separate, simpler system

Law	Not every secret in this fleet goes through Vault. Admin/tooling credentials used by playbooks and operator sessions live in a deliberately simpler, separate system: ansible-vault, encrypted directly into the provisioning repo, decrypted on demand by whoever holds the passphrase.
Flow	Operator encrypts value locally with ansible-vault → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access
Failure behavior	Not applicable in the live-service sense — nothing here is consumed by a running container. A lost passphrase blocks automation runs and admin tooling, not any live application.

Tip

Why two systems, not one — the actual reasoning, not just the rule: Vault KV is built for runtime/service/pipeline secrets a running container needs continuously and automatically. ansible-vault is built for secrets used by tooling that operates on the fleet rather than secrets consumed by the fleet's own running services — a Gitea admin token used only during a playbook run, for instance, doesn't need automatic rotation, per-host scoping, or a rendered file sitting on disk permanently. Building that into Vault's model would have meant either weakening Vault's per-host isolation for a use case that doesn't need it, or adding a whole parallel "admin secrets" concept inside Vault itself. Using the tool already built for exactly this case (ansible-vault, already part of the Ansible toolchain from docs/iac-pipeline.md) was less work and a better fit than extending Vault to do a second, different job.

Note

One exception to this split, flagged rather than smoothed over: Technitium's API token moved out of ansible-vault into live Vault KV (fetched via a community.hashi_vault.vault_kv2_get lookup at playbook-run time) even though it's consumed by an ansible-playbook, not a running service — exactly the case the law above says belongs in ansible-vault instead. The admin/runtime split is the general rule here, not an absolute one, and this is the one place it doesn't cleanly apply.

---

Related: docs/secrets-and-pki.md, docs/iac-pipeline.md, docs/registry-and-build.md