docs: rework security/hardening batch with diagrams and callouts

Mermaid diagrams up front, GFM alert callouts for key decisions and
gotchas, tables instead of bullet walls - replaces plain prose with
the visual template established for security-monitoring.md.

docs/secret-distribution.md

@@ -1,60 +1,64 @@
-# Secret Distribution Model
+# 🔑 Secret Distribution Model
 
-How a secret value moves from being written into Vault to being consumed by a running container. Four distinct flows, each with its own explicit contract — directional flow and authority boundaries, not configuration syntax.
+> [!NOTE]
+> How a secret value moves from being written into Vault to being consumed by a running container. Four distinct flows, each with its own explicit contract — directional flow and authority boundaries, not configuration syntax.
 
-## 1. Certificate Flow
+```mermaid
+flowchart TD
+    Op["Operator"] -->|1: writes cert/secret request| Vault["Vault"]
+    Vault -->|2: Certificate Flow| Cert["Short-lived cert<br/>rendered to disk"]
+    Vault -->|3: App Secret Flow| Sec["Rendered secret file<br/>0600, gitignored"]
+    Repo["Provisioning repo"] -->|4: Config Distribution Flow| Vault
+    Op2["Operator"] -.5: Administrative Secret Flow<br/>separate system.-> EncFile["Encrypted file<br/>in the repo itself"]
 
-**Law** — Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines.
+    Cert --> Container["Container reads<br/>local disk only"]
+    Sec --> Container
+```
 
-**Flow** — Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host.
+## 1️⃣ Certificate Flow
 
-**Invariants**
-- Certificates are short-lived and re-issued on Vault Agent's own schedule, not on-demand by the consuming service.
-- A host only ever receives certificates tagged for its own service identity.
-- The private key never traverses the network after issuance — written directly to local disk by the agent that requested it.
+| | |
+|---|---|
+| **Law** | Applications never request their own certificates. Only Vault Agent talks to Vault's PKI engines. |
+| **Flow** | Vault Agent → Vault PKI engine (issue) → rendered to local disk → consumed by the service on that same host |
+| **Failure behavior** | If Vault is unreachable, existing certificates keep working until they expire — no immediate outage, a slow-building one if the outage outlasts the shortest cert's TTL |
 
-**Failure behavior** — If Vault is unreachable, existing certificates keep working until they expire. No immediate outage; a slow-building one if the outage persists past the shortest cert's TTL.
+**Invariants:** certificates are short-lived and re-issued on Vault Agent's own schedule, never on-demand by the consumer. A host only ever receives certs tagged for its own identity. The private key never traverses the network after issuance.
 
-## 2. Application Secret Flow
+## 2️⃣ Application Secret Flow
 
-**Law** — A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored).
+| | |
+|---|---|
+| **Law** | A secret's real value exists in exactly one place: Vault's KV store. Every other copy is either a placeholder (git-tracked) or a local render (gitignored). |
+| **Flow** | Operator writes value → Vault KV → Vault Agent → rendered file, `0600` → container reads via env file or bind-mount |
+| **Failure behavior** | If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected — a queued rotation just doesn't propagate until Vault is reachable again |
 
-**Flow** — Operator writes value → Vault KV → Vault Agent (polling/watching) → rendered file on local disk → container reads it via an env file or a direct bind-mount.
+> [!WARNING]
+> The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it.
 
-**Invariants**
-- The git-tracked template contains placeholder syntax only — a real secret value anywhere in this flow's git history is a bug, not a variant.
-- The rendered file is owned entirely by Vault Agent. Hand-editing it gets overwritten on the next render cycle and hides the actual source of truth.
-- A container consumes the rendered file from its own host's disk — it never reaches across the network to fetch a secret itself.
+## 3️⃣ Configuration Distribution Flow
 
-**Failure behavior** — If Vault is unreachable, the last successfully rendered file stays on disk and the container keeps running unaffected. A secret rotation queued during an outage simply doesn't propagate until Vault Agent can reach Vault again — no service disruption, just a delayed rotation.
+| | |
+|---|---|
+| **Law** | The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc. |
+| **Flow** | Operator edits the template inventory in the provisioning repo → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts, picks up the new template list |
+| **Failure behavior** | If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on last-known-good values |
 
-## 3. Configuration Distribution Flow
+> [!IMPORTANT]
+> A missing or malformed template referenced by a host's own config crashes that host's **entire** Vault Agent process — every other secret/cert that host was already rendering stops updating too, not just the one broken template. This is why a config redeploy always requires confirming the underlying change actually landed first.
 
-**Law** — The mapping of "which host renders which template to which path" is itself version-controlled, not ad-hoc.
+**Shared-secret variant:** not every template is host-scoped — one credential needed by every docker host for its own registry pushes is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one.
 
-**Flow** — Operator edits the template inventory in the provisioning repo → push → target host pulls → Vault Agent config redeployed → Vault Agent restarts and picks up the new template list.
+## 4️⃣ Administrative Secret Flow — a separate system entirely
 
-**Invariants**
-- A host's Vault Agent only ever renders templates tagged for its own identity — the full template list is visible to every host, but each host filters to its own subset.
-- A missing or malformed template referenced by a host's own config crashes that host's *entire* Vault Agent process, not just the one broken template — every other secret/cert that host was already rendering stops updating too. This is why a config redeploy always requires confirming the underlying change actually landed before assuming it worked.
+| | |
+|---|---|
+| **Law** | Not every secret in this fleet goes through Vault. Admin/tooling credentials used by playbooks and operator sessions live in a deliberately simpler, separate system: encrypted directly into the provisioning repo, decrypted on demand by whoever holds the passphrase. |
+| **Flow** | Operator encrypts value locally → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access |
+| **Failure behavior** | Not applicable in the live-service sense — nothing here is consumed by a running container. A lost passphrase blocks automation runs and admin tooling, not any live application. |
 
-**Failure behavior** — If Vault Agent crashes on a bad config, previously-rendered files stay on disk (stale but present) and affected containers keep running on their last-known-good values until the config is fixed and Vault Agent restarts successfully.
-
-**Shared-secret variant** — not every template is host-scoped. One credential needed by every docker host for its own registry pushes is templated from a single shared path rather than a per-host one — same mechanics as the rest of this flow, just a shared source instead of a private one.
-
-## 4. Administrative Secret Flow (a separate system entirely)
-
-**Law** — Not every secret in this fleet goes through Vault. Admin and tooling credentials used by playbooks and operator sessions themselves (third-party API tokens, login credentials for internal tools) live in a completely separate, deliberately simpler system: encrypted directly into the provisioning repo using a standard vault-encryption tool, decrypted on demand by whoever holds the passphrase.
-
-**Flow** — Operator encrypts value locally → committed to git as ciphertext → decrypted on demand by whoever has repo + passphrase access (an automation run, or a human operator).
-
-**Why two systems, not one** — Vault KV is for runtime/service/pipeline secrets a running container or live service needs; the simpler encrypted-file system is for secrets used by tooling that *operates on* the fleet rather than secrets *consumed by* the fleet's own running services. A reader who only knows the Vault-KV flow above would have no way to tell which system a given secret belongs to — consumed by a container at runtime means Vault KV; used by a human or an automation job doing administrative work means the encrypted-file system.
-
-**Invariants**
-- The encrypted blob is a single opaque ciphertext per file — searching for a variable name won't find it; only a full decrypt reveals what's stored.
-- Unlike Vault KV, there's no per-host scoping or automatic rendering — whoever has the passphrase can read every secret in the file.
-
-**Failure behavior** — Not applicable in the live-service sense; this isn't consumed by running containers. A lost passphrase would block automation runs and admin tooling needing those credentials, not any live application.
+> [!TIP]
+> **Why two systems, not one**: Vault KV is for runtime/service/pipeline secrets a running container needs; the encrypted-file system is for secrets used by tooling that *operates on* the fleet rather than secrets *consumed by* the fleet's own running services. Consumed by a container at runtime → Vault KV. Used by a human or an automation job doing admin work → the encrypted file.
 
 ---
 Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md)