docs: rework security/hardening batch with diagrams and callouts

Mermaid diagrams up front, GFM alert callouts for key decisions and
gotchas, tables instead of bullet walls - replaces plain prose with
the visual template established for security-monitoring.md.

docs/secrets-and-pki.md

@@ -1,23 +1,44 @@
-# Secrets & PKI
+# 🔐 Secrets & PKI
 
-A single Vault cluster serves as both the internal certificate authority and the secret store for the whole fleet — one trust root, not two parallel systems.
+> [!NOTE]
+> A single Vault cluster serves as both the internal certificate authority and the secret store for the whole fleet — one trust root, not two parallel systems.
+
+| | |
+|---|---|
+| **Role** | Certificate authority + secret store |
+| **Distribution** | Vault Agent on every host, AppRole auth, renders locally |
+| **Replaces** | A prior plaintext-`.env`-in-git secrets model |
+| **Biggest known risk** | A non-expiring root token — ranked, tracked, not ignored |
 
 ## What it does
 
 - **Certificate authority** — two PKI mounts: one for public-facing wildcard certs, one for internal mTLS certs. Both issue short-lived certificates to the hosts that need them.
-- **Secret store** — a KV v2 mount holds every migrated application's real secret values (database passwords, API tokens, signing keys), one namespace per host.
+- **Secret store** — a KV v2 mount holds every migrated application's real secret values, one namespace per host.
 - **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly.
 
 ## Why it's built this way
 
-Before this existed, every stack's secrets sat in plaintext inside git-tracked `.env` files — readable by anyone with repo access, and permanently baked into git history even after being "fixed." The replacement model:
+```mermaid
+flowchart LR
+    Before["Before:<br/>plaintext secrets<br/>in git-tracked .env files"] -->|replaced| After
+
+    subgraph After["Now"]
+        direction TB
+        T["Git-tracked template<br/>placeholder syntax only"]
+        V["Vault<br/>real value, one place"]
+        R["Rendered file<br/>gitignored, 0600, root-owned"]
+        T -.no real value ever.-> T
+        V -->|renders| R
+    end
+```
 
 - The real secret value lives in exactly one place: Vault.
 - What's git-tracked is a template with placeholder syntax, never a real value.
-- The rendered file with the real value is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
+- The rendered file is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
 - If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle.
 
-The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before this was extended to cover application secrets — reusing a trust model that was already proven, rather than inventing a second one.
+> [!TIP]
+> The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven, rather than inventing a second one.
 
 ## Architecture
 
@@ -41,37 +62,38 @@ flowchart TD
     Proxy --> A6
 ```
 
-### Two authentication flows, not one
+**Two authentication flows, not one:**
 
-- **AppRole auth** (every host, ongoing): Vault Agent → role_id + secret_id → Vault → short-lived token, cached locally, renewed automatically.
-- **Bootstrap auth** (rare, human-initiated): an operator uses a root token to mint new AppRole credentials when onboarding a host, or to create/update policies. Not part of any workload's runtime path.
+| Flow | When | Path |
+|---|---|---|
+| AppRole auth | Every host, ongoing | Vault Agent → role_id + secret_id → Vault → short-lived token, cached, auto-renewed |
+| Bootstrap auth | Rare, human-initiated | Operator uses a root token to mint new AppRole credentials or update policies — never part of any workload's runtime path |
 
-### The one deliberately-asymmetric trust decision
-
-Every other backend connection inside the internal app-to-app zone skips certificate verification on the reverse proxy hop (it still encrypts the connection, just doesn't check whose cert it's talking to) — a pragmatic tradeoff for low-stakes internal traffic. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert rendered specifically for that purpose. Given what's on the other end of that connection, the asymmetry is intentional, not an oversight.
+> [!IMPORTANT]
+> **The one deliberately-asymmetric trust decision**: every other backend connection inside the internal zone skips certificate verification on the reverse proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real. Given what's on the other end of that connection, the asymmetry is intentional, not an oversight.
 
 ## What's protected
 
-- **Transport** — every Vault Agent ↔ Vault connection is TLS, certificate-verified.
-- **Secrets at rest** — KV v2, encrypted by Vault's own storage layer; rendered files on disk are tightly permissioned, root-owned, gitignored.
-- **Audit trail** — a file audit device logs every request/response. Secret values are HMAC-hashed in the log, not plaintext — what's visible is metadata: which token, which path, when.
-- **Alerting** — a monitoring check tails new audit log lines and pushes an alert through the fleet's existing notification pipeline when the root token gets used, or when permission-denied responses come in bursts.
-- **Certificate renewal reliability** — a dependent service's own TLS bundle is rebuilt automatically whenever its cert gets re-issued, guarded to skip unless the cert has under a week left. This closed a real prior gap: the rebuild never used to happen automatically, and an expired certificate was silently served for over two weeks before it was caught.
+| Layer | Protection |
+|---|---|
+| Transport | Every Vault Agent ↔ Vault connection is TLS, certificate-verified |
+| Secrets at rest | KV v2, encrypted by Vault's storage layer; rendered files tightly permissioned, root-owned, gitignored |
+| Audit trail | Every request/response logged; secret values HMAC-hashed, not plaintext |
+| Alerting | Root-token use or permission-denied bursts trigger an alert through the fleet's notification pipeline |
+| Cert renewal | A dependent service's TLS bundle auto-rebuilds when its cert re-issues — closed a real gap where an expired cert was silently served for over two weeks |
 
 ## Known gaps, ranked by actual risk
 
-1. **The bootstrap root token never expires.** A single long-lived token with unlimited access mints every AppRole and writes every policy. It's encrypted at rest, but encryption doesn't help if the plaintext ever gets used somewhere it shouldn't. This is the single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL provisioning token.
-2. **No SSH certificate signing.** Every host's SSH access key is static, baked in once at image build time, never rotated, with no revocation path short of rebuilding the host. Vault's SSH secrets engine (short-lived signed certs instead of static keys) is the natural fit — a bigger lift since it touches SSH config on every host, not just one config file.
-3. **Single-node storage, no replication.** One disk failure on the Vault host is a full outage and, without a recent backup, real data loss. An availability/durability concern, not a confidentiality one — deliberately lower priority than the two above.
+1. **The bootstrap root token never expires.** Unlimited access, mints every AppRole, writes every policy. Encrypted at rest, but that doesn't help if the plaintext ever gets used somewhere it shouldn't. The single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL token.
+2. **No SSH certificate signing.** Every host's SSH key is static, baked in at image build time, no rotation, no revocation short of rebuilding the host. Vault's SSH secrets engine is the natural fit — a bigger lift since it touches every host's SSH config.
+3. **Single-node storage, no replication.** One disk failure is a full outage and, without a recent backup, real data loss. An availability concern, not a confidentiality one — deliberately lower priority than the two above.
 
-## Deliberately accepted, not gaps
-
-- **Memory locking is disabled** — normally a real concern (secrets could swap to disk under memory pressure), but this host has zero swap configured, so there's no actual leak vector regardless.
-- **The Vault server's own config isn't managed by the same automation that manages every client-side Vault Agent** — set up manually, stays manual by choice.
+> [!TIP]
+> **Deliberately accepted, not gaps**: memory locking is disabled (normally a real concern, but this host has zero swap, so there's no actual leak vector) — and the Vault server's own config is intentionally kept out of the same automation that manages every client-side Vault Agent.
 
 ## Access model
 
-One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration data, plus a private grant giving full read/write access only to that host's own secret namespace. When onboarding a new host, this shape gets copied rather than reinvented — and the shared grant does *not* automatically imply the private one, so both need verifying before assuming a host is fully provisioned.
+One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does *not* automatically imply the private one, so both get verified before assuming a host is fully provisioned.
 
 ---
 Related stacks: [`stacks/identity-and-security/vaultwarden/`](../stacks/identity-and-security/vaultwarden/)