public/docs/secrets-and-pki.md

🔐 Secrets & PKI

Note

A single Vault cluster serves as both the internal certificate authority and the secret store for the whole fleet — one trust root, not two parallel systems.

Role	Certificate authority + secret store
Distribution	Vault Agent on every host, AppRole auth, renders locally
Replaces	A prior plaintext-.env-in-git secrets model
Biggest known risk	A non-expiring root token — ranked, tracked, not ignored

What it does

• Certificate authority — two PKI mounts: one for public-facing wildcard certs, one for internal mTLS certs. Both issue short-lived certificates to the hosts that need them.
• Secret store — a KV v2 mount holds every migrated application's real secret values, one namespace per host.
• Distribution — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly.

Why it's built this way

flowchart LR
    Before["Before:<br/>plaintext secrets<br/>in git-tracked .env files"] -->|replaced| After

    subgraph After["Now"]
        direction TB
        T["Git-tracked template<br/>placeholder syntax only"]
        V["Vault<br/>real value, one place"]
        R["Rendered file<br/>gitignored, 0600, root-owned"]
        T -.no real value ever.-> T
        V -->|renders| R
    end

• The real secret value lives in exactly one place: Vault.
• What's git-tracked is a template with placeholder syntax, never a real value.
• The rendered file is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
• If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle.

Tip

The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven, rather than inventing a second one.

Architecture

flowchart TD
    Vault["Vault<br/>PKI (public + internal mounts) + KV v2 store<br/>TLS listener, cert-verified · audit log"]
    Proxy["Internal app-to-app reverse-proxy zone"]
    A1["Vault Agent — App Host<br/>~12 stacks' secrets + internal CA cert"]
    A2["Vault Agent — Automation Host<br/>~10 stacks' secrets"]
    A3["Vault Agent — Monitoring Host<br/>admin credentials + OIDC secret"]
    A4["Vault Agent — Edge Proxy Host<br/>mTLS zone certs"]
    A5["Vault Agent — DNS Host<br/>admin UI cert"]
    A6["Vault Agent — Security Monitoring Host<br/>manager/indexer/dashboard secrets"]

    Vault -->|HTTPS, cert-verified| Proxy
    Proxy --> A1
    Proxy --> A2
    Proxy --> A3
    Proxy --> A4
    Proxy --> A5
    Proxy --> A6

Two authentication flows, not one:

Flow	When	Path
AppRole auth	Every host, ongoing	Vault Agent → role_id + secret_id → Vault → short-lived token, cached, auto-renewed
Bootstrap auth	Rare, human-initiated	Operator uses a root token to mint new AppRole credentials or update policies — never part of any workload's runtime path

Important

The one deliberately-asymmetric trust decision: every other backend connection inside the internal zone skips certificate verification on the reverse proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real. Given what's on the other end of that connection, the asymmetry is intentional, not an oversight.

What's protected

Layer	Protection
Transport	Every Vault Agent ↔ Vault connection is TLS, certificate-verified
Secrets at rest	KV v2, encrypted by Vault's storage layer; rendered files tightly permissioned, root-owned, gitignored
Audit trail	Every request/response logged; secret values HMAC-hashed, not plaintext
Alerting	Root-token use or permission-denied bursts trigger an alert through the fleet's notification pipeline
Cert renewal	A dependent service's TLS bundle auto-rebuilds when its cert re-issues — closed a real gap where an expired cert was silently served for over two weeks

Known gaps, ranked by actual risk

1. The bootstrap root token never expires. Unlimited access, mints every AppRole, writes every policy. Encrypted at rest, but that doesn't help if the plaintext ever gets used somewhere it shouldn't. The single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL token.
2. No SSH certificate signing. Every host's SSH key is static, baked in at image build time, no rotation, no revocation short of rebuilding the host. Vault's SSH secrets engine is the natural fit — a bigger lift since it touches every host's SSH config.
3. Single-node storage, no replication. One disk failure is a full outage and, without a recent backup, real data loss. An availability concern, not a confidentiality one — deliberately lower priority than the two above.

Tip

Deliberately accepted, not gaps: memory locking is disabled (normally a real concern, but this host has zero swap, so there's no actual leak vector) — and the Vault server's own config is intentionally kept out of the same automation that manages every client-side Vault Agent.

Access model

One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does not automatically imply the private one, so both get verified before assuming a host is fully provisioned.

---

Related stacks: stacks/identity-and-security/vaultwarden/