# Secrets & PKI

A single Vault cluster serves as both the internal certificate authority and the secret store for the whole fleet — one trust root, not two parallel systems.

## What it does

- **Certificate authority** — two PKI mounts: one for public-facing wildcard certs, one for internal mTLS certs. Both issue short-lived certificates to the hosts that need them.
- **Secret store** — a KV v2 mount holds every migrated application's real secret values (database passwords, API tokens, signing keys), one namespace per host.
- **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly.

## Why it's built this way

Before this existed, every stack's secrets sat in plaintext inside git-tracked `.env` files — readable by anyone with repo access, and permanently baked into git history even after being "fixed." The replacement model:

- The real secret value lives in exactly one place: Vault.
- What's git-tracked is a template with placeholder syntax, never a real value.
- The rendered file with the real value is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
- If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle.

The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before this was extended to cover application secrets — reusing a trust model that was already proven, rather than inventing a second one.

## Architecture

```mermaid
flowchart TD
    Vault["Vault<br/>PKI (public + internal mounts) + KV v2 store<br/>TLS listener, cert-verified · audit log"]
    Proxy["Internal app-to-app reverse-proxy zone"]
    A1["Vault Agent — App Host<br/>~12 stacks' secrets + internal CA cert"]
    A2["Vault Agent — Automation Host<br/>~10 stacks' secrets"]
    A3["Vault Agent — Monitoring Host<br/>admin credentials + OIDC secret"]
    A4["Vault Agent — Edge Proxy Host<br/>mTLS zone certs"]
    A5["Vault Agent — DNS Host<br/>admin UI cert"]
    A6["Vault Agent — Security Monitoring Host<br/>manager/indexer/dashboard secrets"]

    Vault -->|HTTPS, cert-verified| Proxy
    Proxy --> A1
    Proxy --> A2
    Proxy --> A3
    Proxy --> A4
    Proxy --> A5
    Proxy --> A6
```

### Two authentication flows, not one

- **AppRole auth** (every host, ongoing): Vault Agent → role_id + secret_id → Vault → short-lived token, cached locally, renewed automatically.
- **Bootstrap auth** (rare, human-initiated): an operator uses a root token to mint new AppRole credentials when onboarding a host, or to create/update policies. Not part of any workload's runtime path.

### The one deliberately-asymmetric trust decision

Every other backend connection inside the internal app-to-app zone skips certificate verification on the reverse proxy hop (it still encrypts the connection, just doesn't check whose cert it's talking to) — a pragmatic tradeoff for low-stakes internal traffic. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert rendered specifically for that purpose. Given what's on the other end of that connection, the asymmetry is intentional, not an oversight.

## What's protected

- **Transport** — every Vault Agent ↔ Vault connection is TLS, certificate-verified.
- **Secrets at rest** — KV v2, encrypted by Vault's own storage layer; rendered files on disk are tightly permissioned, root-owned, gitignored.
- **Audit trail** — a file audit device logs every request/response. Secret values are HMAC-hashed in the log, not plaintext — what's visible is metadata: which token, which path, when.
- **Alerting** — a monitoring check tails new audit log lines and pushes an alert through the fleet's existing notification pipeline when the root token gets used, or when permission-denied responses come in bursts.
- **Certificate renewal reliability** — a dependent service's own TLS bundle is rebuilt automatically whenever its cert gets re-issued, guarded to skip unless the cert has under a week left. This closed a real prior gap: the rebuild never used to happen automatically, and an expired certificate was silently served for over two weeks before it was caught.

## Known gaps, ranked by actual risk

1. **The bootstrap root token never expires.** A single long-lived token with unlimited access mints every AppRole and writes every policy. It's encrypted at rest, but encryption doesn't help if the plaintext ever gets used somewhere it shouldn't. This is the single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL provisioning token.
2. **No SSH certificate signing.** Every host's SSH access key is static, baked in once at image build time, never rotated, with no revocation path short of rebuilding the host. Vault's SSH secrets engine (short-lived signed certs instead of static keys) is the natural fit — a bigger lift since it touches SSH config on every host, not just one config file.
3. **Single-node storage, no replication.** One disk failure on the Vault host is a full outage and, without a recent backup, real data loss. An availability/durability concern, not a confidentiality one — deliberately lower priority than the two above.

## Deliberately accepted, not gaps

- **Memory locking is disabled** — normally a real concern (secrets could swap to disk under memory pressure), but this host has zero swap configured, so there's no actual leak vector regardless.
- **The Vault server's own config isn't managed by the same automation that manages every client-side Vault Agent** — set up manually, stays manual by choice.

## Access model

One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration data, plus a private grant giving full read/write access only to that host's own secret namespace. When onboarding a new host, this shape gets copied rather than reinvented — and the shared grant does *not* automatically imply the private one, so both need verifying before assuming a host is fully provisioned.

---
Related stacks: [`stacks/identity-and-security/vaultwarden/`](../stacks/identity-and-security/vaultwarden/)