# 🔑 Secret Distribution Model > [!NOTE] > [`docs/secrets-and-pki.md`](secrets-and-pki.md) covers *what* Vault is and *why* it exists. This doc covers something different and, arguably, harder to get right: the actual contract for how a secret value moves from being written into Vault to being consumed by a running container — and why that contract is deliberately split into five separate flows instead of one universal one.  ## Why Five Flows Instead Of One The obvious simpler design is one universal "secret goes in, secret comes out" pipeline. It was rejected because the five things being moved genuinely have different risk profiles and different failure tolerances: a certificate expiring is a slow-building problem measured in days; a missing application secret is often an immediate outage; a broken config-distribution push can silently stop *all* of a host's secrets from updating, not just one; losing the ability to refresh an SSH certificate doesn't break anything immediately, but eventually locks the automation identity out of the fleet entirely. Treating them identically would mean either over-engineering the simple cases or under-protecting the risky ones. Five flows, each with its own explicit Law/Flow/Failure-behavior contract, is more code to maintain but far easier to reason about when something breaks — anyone debugging an incident knows immediately which contract governs the thing that's wrong. > [!NOTE] > **Why the SSH Certificate Flow isn't just folded into Certificate Flow** — both issue short-lived Vault-signed credentials, so treating them as one flow looks tempting. They're kept separate because the actual mechanism is genuinely different, not just the credential type: Certificate Flow is triggered and rendered by Vault Agent itself, on Vault Agent's own schedule. The SSH Certificate Flow runs on its own independent systemd timer, calls Vault directly, and only *reuses* Vault Agent's already-authenticated token as a convenience — Vault Agent never renders it and doesn't know it exists. Collapsing them into one contract would hide that difference from exactly the person who needs it most: whoever is debugging why an SSH certificate didn't refresh, who needs to know immediately that the answer isn't "check Vault Agent's logs." ## 1️⃣ Certificate Flow | | | |---|---| | **Law** | Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines. | | **Flow** | Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host | | **Failure behavior** | If Vault is unreachable, existing certificates keep working until they expire — no immediate outage, a slow-building one if the outage outlasts the shortest cert's TTL | **Invariants:** certificates are short-lived and re-issued on Vault Agent's own schedule, never on-demand by the consumer. A host only ever receives certs tagged for its own identity. The private key never traverses the network after issuance. ## 2️⃣ SSH Certificate Flow | | | |---|---| | **Law** | The automation identity never holds a static, standing-access SSH key. Only a Vault-signed certificate, valid for hours, ever grants it access to a fleet host. | | **Flow** | Its own systemd timer (2h interval) → reuses Vault Agent's already-authenticated token → `vault write ssh-client-signer/sign/[role name]` → certificate installed next to the identity's own key on the one host that needs it → every fleet host's sshd trusts the CA directly, no per-host secret involved | | **Failure behavior** | A single missed refresh is harmless — the timer fires every 2 hours against a 6-hour certificate, so there's hours of margin. A sustained outage spanning several missed cycles eventually locks the automation identity out of certificate-based SSH entirely, with no static key to fall back to — an accepted tradeoff, not an oversight (see [`docs/secrets-and-pki.md`](secrets-and-pki.md)) | This is the real shape of a signed certificate from this flow, not a description of one — host path, CA fingerprint, and validity window redacted since they identify live infrastructure, everything else genuine: ``` $ ssh-keygen -Lf [cert path] Type: ssh-ed25519-cert-v01@openssh.com user certificate Signing CA: RSA SHA256:[fleet's CA fingerprint] (using rsa-sha2-512) Valid: from [issued] to [issued + 6h] Principals: [automation identity] Extensions: permit-port-forwarding ``` **Invariants:** exactly one identity is ever signed for. Vault's role config enforces this independently of the script that calls it — a compromised or buggy script still can't get a certificate for any other principal. Every host trusts the CA's public key directly; none of them store the automation identity's actual key material at all. ## 3️⃣ Application Secret Flow | | | |---|---| | **Law** | A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored). | | **Flow** | Operator writes value → Vault KV → Vault Agent → rendered file, `0600` → container reads via env file or bind-mount | | **Failure behavior** | If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected — a queued rotation just doesn't propagate until Vault is reachable again | > [!WARNING] > The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it. This is the single most common way this model gets violated by accident: someone SSHes in to fix a value under time pressure, and the fix silently disappears on the next render. ## 4️⃣ Configuration Distribution Flow | | | |---|---| | **Law** | The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc. | | **Flow** | Operator edits the template inventory in the provisioning repo (see [`docs/iac-pipeline.md`](iac-pipeline.md)) → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts, picks up the new template list | | **Failure behavior** | If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on last-known-good values | > [!IMPORTANT] > This is the flow with the sharpest failure mode in the whole model: a missing or malformed template referenced by a host's own config crashes that host's **entire** Vault Agent process — every other secret/cert that host was already rendering stops updating too, not just the one broken template. A single YAML typo in one app's template inventory can silently freeze every other app's secret rotation on that same host. This is why a config redeploy always requires confirming the underlying git change actually landed first, not just that the deploy command returned success. That redeploy-and-restart sequence isn't just a claim — here's a real one, start to finish: Vault Agent restarts, authenticates, renders a fresh `.env.secrets`, and its `reload_command` fires `docker compose up -d`, which recreates and restarts the actual container:  **Shared-secret variant:** not every template is host-scoped — one credential needed by every docker host for its own registry pushes (see [`docs/registry-and-build.md`](registry-and-build.md)) is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one — and it relies on every host's policy already granting read access to that shared path, the same "shared + private" pattern described in [`docs/secrets-and-pki.md`](secrets-and-pki.md)'s access model. This is the real template list entry, repeated per host: ``` $ grep -B1 -A3 "registry-token.ctmpl" vault_agent/defaults/main.yml - service: automation src_path: .../automation-vm/registry-token/registry-token.ctmpl dest_path: /opt/docker/docker_volumes/registry-token/registry-token reload_command: "chmod 600 .../registry-token/registry-token" ``` ## 5️⃣ Administrative Secret Flow — a deliberately separate, simpler system | | | |---|---| | **Law** | Not every secret in this fleet goes through Vault. Admin/tooling credentials used by playbooks and operator sessions live in a deliberately simpler, separate system: `ansible-vault`, encrypted directly into the provisioning repo, decrypted on demand by whoever holds the passphrase. | | **Flow** | Operator encrypts value locally with `ansible-vault` → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access | | **Failure behavior** | Not applicable in the live-service sense — nothing here is consumed by a running container. A lost passphrase blocks automation runs and admin tooling, not any live application. | > [!TIP] > **Why two systems, not one — the actual reasoning, not just the rule**: Vault KV is built for runtime/service/pipeline secrets a running container needs continuously and automatically. `ansible-vault` is built for secrets used by tooling that *operates on* the fleet rather than secrets *consumed by* the fleet's own running services — a Gitea admin token used only during a playbook run, for instance, doesn't need automatic rotation, per-host scoping, or a rendered file sitting on disk permanently. Building that into Vault's model would have meant either weakening Vault's per-host isolation for a use case that doesn't need it, or adding a whole parallel "admin secrets" concept inside Vault itself. Using the tool already built for exactly this case (`ansible-vault`, already part of the Ansible toolchain from [`docs/iac-pipeline.md`](iac-pipeline.md)) was less work and a better fit than extending Vault to do a second, different job. > [!NOTE] > **One exception to this split, flagged rather than smoothed over**: Technitium's API token moved out of `ansible-vault` into live Vault KV (fetched via a `community.hashi_vault.vault_kv2_get` lookup at playbook-run time) even though it's consumed by an ansible-playbook, not a running service — exactly the case the law above says belongs in `ansible-vault` instead. The admin/runtime split is the general rule here, not an absolute one, and this is the one place it doesn't cleanly apply. This is the actual lookup, not a description of it: > > ``` > $ grep -A2 "technitium_api_token:" app_update/defaults/main.yml > technitium_api_token: >- > {{ lookup('community.hashi_vault.vault_kv2_get', 'technitium/secrets', > engine_mount_point='kv', url=vault_addr, token=technitium_control_vault_token).secret.api_token }} > ``` --- Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/registry-and-build.md`](registry-and-build.md)