public/docs/secrets-and-pki.md

# 🔐 Secrets & PKI

> [!NOTE]
> A single Vault cluster serves as both the internal certificate authority and the secret store for the whole fleet — one trust root, not two parallel systems.

| | |
|---|---|
| **Role** | Certificate authority + secret store |
| **Distribution** | Vault Agent on every host, AppRole auth, renders locally |
| **Replaces** | A prior plaintext-`.env`-in-git secrets model |
| **Biggest known risk** | A non-expiring root token — ranked, tracked, not ignored |

## What it does

- **Certificate authority** — two PKI mounts: one for public-facing wildcard certs, one for internal mTLS certs. Both issue short-lived certificates to the hosts that need them.
- **Secret store** — a KV v2 mount holds every migrated application's real secret values, one namespace per host.
- **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly.

## Why it's built this way

```mermaid
flowchart LR
    Before["Before:<br/>plaintext secrets<br/>in git-tracked .env files"] -->|replaced| After

    subgraph After["Now"]
        direction TB
        T["Git-tracked template<br/>placeholder syntax only"]
        V["Vault<br/>real value, one place"]
        R["Rendered file<br/>gitignored, 0600, root-owned"]
        T -.no real value ever.-> T
        V -->|renders| R
    end
```

- The real secret value lives in exactly one place: Vault.
- What's git-tracked is a template with placeholder syntax, never a real value.
- The rendered file is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
- If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle.

> [!TIP]
> The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven, rather than inventing a second one.

## Architecture

```mermaid
flowchart TD
    Vault["Vault<br/>PKI (public + internal mounts) + KV v2 store<br/>TLS listener, cert-verified · audit log"]
    Proxy["Internal app-to-app reverse-proxy zone"]
    A1["Vault Agent — App Host<br/>~12 stacks' secrets + internal CA cert"]
    A2["Vault Agent — Automation Host<br/>~10 stacks' secrets"]
    A3["Vault Agent — Monitoring Host<br/>admin credentials + OIDC secret"]
    A4["Vault Agent — Edge Proxy Host<br/>mTLS zone certs"]
    A5["Vault Agent — DNS Host<br/>admin UI cert"]
    A6["Vault Agent — Security Monitoring Host<br/>manager/indexer/dashboard secrets"]

    Vault -->|HTTPS, cert-verified| Proxy
    Proxy --> A1
    Proxy --> A2
    Proxy --> A3
    Proxy --> A4
    Proxy --> A5
    Proxy --> A6
```

**Two authentication flows, not one:**

| Flow | When | Path |
|---|---|---|
| AppRole auth | Every host, ongoing | Vault Agent → role_id + secret_id → Vault → short-lived token, cached, auto-renewed |
| Bootstrap auth | Rare, human-initiated | Operator uses a root token to mint new AppRole credentials or update policies — never part of any workload's runtime path |

> [!IMPORTANT]
> **The one deliberately-asymmetric trust decision**: every other backend connection inside the internal zone skips certificate verification on the reverse proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real. Given what's on the other end of that connection, the asymmetry is intentional, not an oversight.

## What's protected

| Layer | Protection |
|---|---|
| Transport | Every Vault Agent ↔ Vault connection is TLS, certificate-verified |
| Secrets at rest | KV v2, encrypted by Vault's storage layer; rendered files tightly permissioned, root-owned, gitignored |
| Audit trail | Every request/response logged; secret values HMAC-hashed, not plaintext |
| Alerting | Root-token use or permission-denied bursts trigger an alert through the fleet's notification pipeline |
| Cert renewal | A dependent service's TLS bundle auto-rebuilds when its cert re-issues — closed a real gap where an expired cert was silently served for over two weeks |

## Known gaps, ranked by actual risk

1. **The bootstrap root token never expires.** Unlimited access, mints every AppRole, writes every policy. Encrypted at rest, but that doesn't help if the plaintext ever gets used somewhere it shouldn't. The single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL token.
2. **No SSH certificate signing.** Every host's SSH key is static, baked in at image build time, no rotation, no revocation short of rebuilding the host. Vault's SSH secrets engine is the natural fit — a bigger lift since it touches every host's SSH config.
3. **Single-node storage, no replication.** One disk failure is a full outage and, without a recent backup, real data loss. An availability concern, not a confidentiality one — deliberately lower priority than the two above.

> [!TIP]
> **Deliberately accepted, not gaps**: memory locking is disabled (normally a real concern, but this host has zero swap, so there's no actual leak vector) — and the Vault server's own config is intentionally kept out of the same automation that manages every client-side Vault Agent.

## Access model

One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does *not* automatically imply the private one, so both get verified before assuming a host is fully provisioned.

---
Related stacks: [`stacks/identity-and-security/vaultwarden/`](../stacks/identity-and-security/vaultwarden/)