thesabear
12e69acde7
Names real technology throughout (Vault, Authentik, pfSense, firewalld, Wazuh) instead of staying generic, adds a 'why this approach vs the alternative' section to every doc, cross-links heavily between docs instead of treating each system in isolation.
69 lines
7.0 KiB
Markdown
69 lines
7.0 KiB
Markdown
# 🔑 Secret Distribution Model
|
|
|
|
> [!NOTE]
|
|
> [`docs/secrets-and-pki.md`](secrets-and-pki.md) covers *what* Vault is and *why* it exists. This doc covers something different and, arguably, harder to get right: the actual contract for how a secret value moves from being written into Vault to being consumed by a running container — and why that contract is deliberately split into four separate flows instead of one universal one.
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
Op["Operator"] -->|1: writes cert/secret request| Vault["Vault"]
|
|
Vault -->|2: Certificate Flow| Cert["Short-lived cert<br/>rendered to disk"]
|
|
Vault -->|3: App Secret Flow| Sec["Rendered secret file<br/>0600, gitignored"]
|
|
Repo["Provisioning repo"] -->|4: Config Distribution Flow| Vault
|
|
Op2["Operator"] -.5: Administrative Secret Flow<br/>separate system entirely.-> EncFile["ansible-vault encrypted file<br/>in the repo itself"]
|
|
|
|
Cert --> Container["Container reads<br/>local disk only"]
|
|
Sec --> Container
|
|
```
|
|
|
|
## Why four flows instead of one
|
|
|
|
The obvious simpler design is one universal "secret goes in, secret comes out" pipeline. It was rejected because the four things being moved genuinely have different risk profiles and different failure tolerances: a certificate expiring is a slow-building problem measured in days; a missing application secret is often an immediate outage; a broken config-distribution push can silently stop *all* of a host's secrets from updating, not just one. Treating them identically would mean either over-engineering the simple cases or under-protecting the risky one. Four flows, each with its own explicit Law/Flow/Failure-behavior contract, is more code to maintain but far easier to reason about when something breaks — anyone debugging an incident knows immediately which contract governs the thing that's wrong.
|
|
|
|
## 1️⃣ Certificate Flow
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Law** | Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines. |
|
|
| **Flow** | Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host |
|
|
| **Failure behavior** | If Vault is unreachable, existing certificates keep working until they expire — no immediate outage, a slow-building one if the outage outlasts the shortest cert's TTL |
|
|
|
|
**Invariants:** certificates are short-lived and re-issued on Vault Agent's own schedule, never on-demand by the consumer. A host only ever receives certs tagged for its own identity. The private key never traverses the network after issuance.
|
|
|
|
## 2️⃣ Application Secret Flow
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Law** | A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored). |
|
|
| **Flow** | Operator writes value → Vault KV → Vault Agent → rendered file, `0600` → container reads via env file or bind-mount |
|
|
| **Failure behavior** | If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected — a queued rotation just doesn't propagate until Vault is reachable again |
|
|
|
|
> [!WARNING]
|
|
> The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it. This is the single most common way this model gets violated by accident: someone SSHes in to fix a value under time pressure, and the fix silently disappears on the next render.
|
|
|
|
## 3️⃣ Configuration Distribution Flow
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Law** | The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc. |
|
|
| **Flow** | Operator edits the template inventory in the provisioning repo (see [`docs/iac-pipeline.md`](iac-pipeline.md)) → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts, picks up the new template list |
|
|
| **Failure behavior** | If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on last-known-good values |
|
|
|
|
> [!IMPORTANT]
|
|
> This is the flow with the sharpest failure mode in the whole model: a missing or malformed template referenced by a host's own config crashes that host's **entire** Vault Agent process — every other secret/cert that host was already rendering stops updating too, not just the one broken template. A single YAML typo in one app's template inventory can silently freeze every other app's secret rotation on that same host. This is why a config redeploy always requires confirming the underlying git change actually landed first, not just that the deploy command returned success.
|
|
|
|
**Shared-secret variant:** not every template is host-scoped — one credential needed by every docker host for its own registry pushes (see [`docs/registry-and-build.md`](registry-and-build.md)) is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one — and it relies on every host's policy already granting read access to that shared path, the same "shared + private" pattern described in [`docs/secrets-and-pki.md`](secrets-and-pki.md)'s access model.
|
|
|
|
## 4️⃣ Administrative Secret Flow — a deliberately separate, simpler system
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Law** | Not every secret in this fleet goes through Vault. Admin/tooling credentials used by playbooks and operator sessions live in a deliberately simpler, separate system: `ansible-vault`, encrypted directly into the provisioning repo, decrypted on demand by whoever holds the passphrase. |
|
|
| **Flow** | Operator encrypts value locally with `ansible-vault` → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access |
|
|
| **Failure behavior** | Not applicable in the live-service sense — nothing here is consumed by a running container. A lost passphrase blocks automation runs and admin tooling, not any live application. |
|
|
|
|
> [!TIP]
|
|
> **Why two systems, not one — the actual reasoning, not just the rule**: Vault KV is built for runtime/service/pipeline secrets a running container needs continuously and automatically. `ansible-vault` is built for secrets used by tooling that *operates on* the fleet rather than secrets *consumed by* the fleet's own running services — a Gitea admin token used only during a playbook run, for instance, doesn't need automatic rotation, per-host scoping, or a rendered file sitting on disk permanently. Building that into Vault's model would have meant either weakening Vault's per-host isolation for a use case that doesn't need it, or adding a whole parallel "admin secrets" concept inside Vault itself. Using the tool already built for exactly this case (`ansible-vault`, already part of the Ansible toolchain from [`docs/iac-pipeline.md`](iac-pipeline.md)) was less work and a better fit than extending Vault to do a second, different job.
|
|
|
|
---
|
|
Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/registry-and-build.md`](registry-and-build.md)
|