public/docs/secrets-and-pki.md

🔐 Secrets & PKI

Note

One HashiCorp Vault cluster is both the certificate authority and the secret store for the entire fleet. Every other system in this repo either consumes something Vault issues, or exists specifically to protect Vault itself — this is the trust root everything else builds on.

Software	HashiCorp Vault, self-hosted, native systemd service (not containerized)
Role	Certificate authority (2 PKI mounts) + secret store (KV v2)
Distribution	Vault Agent on every consuming host, AppRole auth, renders to local disk
Replaces	A prior plaintext-.env-in-git secrets model
Biggest known risk	A non-expiring root bootstrap token — ranked #1 of 3 known gaps, tracked openly

Why Vault, and why self-hosted

A managed secrets service (a cloud provider's KMS/Secrets Manager) was a real alternative and was rejected for a specific reason: this fleet's PKI needs to issue certificates to purely internal hosts that have no business talking to a cloud API, and the two-zone reverse-proxy model in docs/dns-naming.md depends on an internal CA that's reachable without any external dependency. Vault does both certificate issuance and secret storage under one trust model instead of stitching together a separate CA tool and a separate secrets tool — fewer moving parts, one audit log, one policy language to reason about.

What it does

• Certificate authority — two PKI mounts: one for public-facing wildcard certs (consumed by the edge in docs/firewall-and-network.md), one for internal mTLS certs (consumed by the browser-facing reverse-proxy zone). Both issue short-lived certificates, not long-lived ones — a deliberate choice that shrinks the window a leaked cert is useful for.
• Secret store — a KV v2 mount holds every migrated application's real secret values, one namespace per host, enforced by policy (below).
• Distribution — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly — this single rule is what makes the rest of the model tractable to reason about.

Why it's built this way

Before this system existed, every stack's secrets sat in plaintext inside git-tracked .env files — readable by anyone with repo access, and permanently baked into git history even after being "fixed." SecureVault replaces that:

• The real secret value lives in exactly one place: Vault.
• What's git-tracked is a template with placeholder syntax, never a real value.
• The rendered file is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
• If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle, no redeploy required.

Tip

The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven for one use case, rather than inventing a second one for the next. This is the same instinct behind not building a separate CA tool: one proven mechanism, extended, beats two different mechanisms that each need their own reasoning.

Architecture

Every Vault Agent reaches Vault through the internal app-to-app proxy zone described in docs/dns-naming.md — never directly. It's the one hop in that entire zone where the proxy actually verifies the backend's certificate instead of skipping it, which is exactly the asymmetric-trust decision explained below.

Consumer	What it gets from Vault
App Host	12 stacks' worth of application secrets, an internal CA cert to verify Vault's own listener, plus a shared registry-token render
Automation Host	10 stacks' worth of application secrets, plus a shared registry-token render
Monitoring Host	Grafana's admin password and the Authentik OIDC secret, plus Hawser's application secret, plus a shared registry-token render
User-Facing Proxy	Server + client certs for the mTLS-enforced browser zone, plus a shared registry-token render and Hawser's application secret
Internal DNS	Server cert for its own admin UI/API only
Security Monitoring	Its manager and dashboard services' shared secrets (indexer, API, dashboard passwords), plus Hawser's application secret, plus a shared registry-token render

None of these ever call the Vault API themselves — every one of them goes through the same local-render sequence:

Two authentication flows, not one:

Flow	When	Path
AppRole auth	Every host, ongoing	Vault Agent → role_id + secret_id → Vault → short-lived token, cached, auto-renewed
Bootstrap auth	Rare, human-initiated	Operator uses a root token to mint new AppRole credentials or update policies — never part of any workload's runtime path

Important

The one deliberately-asymmetric trust decision: every other backend connection inside the internal zone described in docs/dns-naming.md skips certificate verification on the reverse-proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic between containers that already share a trusted network. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert. Given that Vault is the root of trust for every other secret in the fleet, treating it identically to a media-server backend would have been the actual mistake — the asymmetry here is the correct call, not an inconsistency.

What's protected

Layer	Protection
Transport	Every Vault Agent ↔ Vault connection is TLS, certificate-verified
Secrets at rest	KV v2, encrypted by Vault's storage layer; rendered files are 0600, root-owned, gitignored
Audit trail	Every request/response logged to a file audit device; secret values HMAC-hashed, not plaintext — what's visible is metadata: which token, which path, when. Rotated weekly, 8 generations kept.
Alerting	Root-token use or permission-denied bursts trigger an alert through docs/notifications.md's pipeline
Cert renewal reliability	Technitium's DNS-over-TLS/DoH cert bundle auto-rebuilds whenever Vault Agent re-renders its certs, guarded to skip unless under 7 days from expiry. This closed a real prior gap: the rebuild never used to happen automatically, and an expired cert was silently served for 19 days before anyone caught it.

Known gaps, ranked by actual risk — not a flat list

Ranking these mattered more than just listing them: a reader (or an attacker) needs to know which one to fix first, not just that gaps exist.

1. The bootstrap root token never expires. Unlimited access, mints every AppRole, writes every policy. Encrypted at rest, but that doesn't help if the plaintext ever gets used somewhere it shouldn't. The single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL token.
2. No SSH certificate signing. Every host's SSH key is static, baked in at image build time (see docs/iac-pipeline.md), no rotation, no revocation short of rebuilding the host. Vault's own SSH secrets engine is the natural fit — a bigger lift than #1 since it touches every host's SSH config, not one central policy.
3. Single-node storage, no replication. One disk failure is a full outage and, without a recent backup, real data loss. This is the one gap docs/disaster-recovery.md's VM-level backup layer actually covers — an availability concern, correctly triaged as lower priority than the two confidentiality risks above.

Tip

Deliberately accepted, not gaps: memory locking is disabled (normally a real concern — secrets could swap to disk under memory pressure — but this host has zero swap configured, so there's no actual leak vector regardless), and the Vault server's own configuration is intentionally kept out of the same automation that manages every client-side Vault Agent, so a mistake in that broader automation can't touch the root of trust itself.

Access model

One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does not automatically imply the private one, so both get verified before assuming a host is fully provisioned. This same "shared + private" pattern shows up again in docs/iac-pipeline.md's tag-driven Ansible model — a deliberate consistency, not a coincidence.

---

Related: docs/secret-distribution.md, docs/firewall-and-network.md · Stacks: stacks/identity-and-security/vaultwarden/