public/docs/secrets-and-pki.md

🔐 Secrets & PKI

Note

One HashiCorp Vault cluster is both the certificate authority and the secret store for the entire fleet. Every other system in this repo either consumes something Vault issues, or exists specifically to protect Vault itself — this is the trust root everything else builds on.

Software	HashiCorp Vault, self-hosted, native systemd service (not containerized)
Role	Certificate authority (2 PKI mounts + 1 SSH CA mount) + secret store (KV v2)
Distribution	Vault Agent on every consuming host, AppRole auth, renders to local disk
Replaces	A prior plaintext-.env-in-git secrets model

Why Vault, and Why Self-hosted

A managed secrets service (a cloud provider's KMS/Secrets Manager) was a real alternative and was rejected for a specific reason: this fleet's PKI needs to issue certificates to purely internal hosts that have no business talking to a cloud API, and the two-zone reverse-proxy model in docs/dns-naming.md depends on an internal CA that's reachable without any external dependency. Vault does both certificate issuance and secret storage under one trust model instead of stitching together a separate CA tool and a separate secrets tool — fewer moving parts, one audit log, one policy language to reason about.

What It Does

• Certificate authority — two PKI mounts: one for public-facing wildcard certs (consumed by the edge in docs/firewall-and-network.md), one for internal mTLS certs (consumed by the browser-facing reverse-proxy zone). Both issue short-lived certificates, not long-lived ones — a deliberate choice that shrinks the window a leaked cert is useful for.
• Secret store — a KV v2 mount holds every migrated application's real secret values, one namespace per host, enforced by policy (below).
• Distribution — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly — this single rule is what makes the rest of the model tractable to reason about.
• SSH certificate authority — a dedicated mount issues short-lived SSH certificates for the fleet's automation identity, instead of that identity holding a single static key with standing, permanent access to every host. A signed certificate is valid for 6 hours (12-hour hard cap), scoped to exactly one principal, and every host trusts the CA's public key directly rather than storing that identity's key at all — the same short-lived-credential philosophy as the PKI mounts above, applied to SSH instead of TLS. A leaked certificate is worthless in hours; a leaked static key used to be useful forever. The one thing that can't use this model at all is Kestra's own SSH-based automation flows — its SSH plugin is a plain client with no certificate support, full stop — see docs/orchestration.md for what it uses instead, and why that's a completely separate, unrelated credential rather than a workaround bolted onto this one.

This is the real role definition, not a paraphrase — role and principal names redacted since they identify a live credential, everything else genuine:

$ vault read ssh-client-signer/roles/[role name]
key_type              ca
allowed_users         [automation identity]
default_user          [automation identity]
ttl                   6h    (21600s)
max_ttl               12h   (43200s)
algorithm_signer      rsa-sha2-512
default_extensions    permit-pty, permit-port-forwarding

And this is the real mount list, not an assumption of what's enabled — the SSH CA sitting right alongside the two PKI mounts, not a separate system bolted on beside Vault:

Why It's Built This Way

Before this system existed, every stack's secrets sat in plaintext inside git-tracked .env files — readable by anyone with repo access, and permanently baked into git history even after being "fixed." SecureVault replaces that:

• The real secret value lives in exactly one place: Vault.
• What's git-tracked is a template with placeholder syntax, never a real value.
• The rendered file is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
• If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle, no redeploy required.

Tip

The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven for one use case, rather than inventing a second one for the next. This is the same instinct behind not building a separate CA tool: one proven mechanism, extended, beats two different mechanisms that each need their own reasoning.

Architecture

Every Vault Agent reaches Vault through the internal app-to-app proxy zone described in docs/dns-naming.md — never directly. It's the one hop in that entire zone where the proxy actually verifies the backend's certificate instead of skipping it, which is exactly the asymmetric-trust decision explained below.

Consumer	What it gets from Vault
App Host	12 stacks' worth of application secrets, an internal CA cert to verify Vault's own listener, plus a shared registry-token render
Automation Host	10 stacks' worth of application secrets, plus a shared registry-token render
Monitoring Host	Grafana's admin password and the Authentik OIDC secret, plus Hawser's application secret, plus a shared registry-token render
User-Facing Proxy	Server + client certs for the mTLS-enforced browser zone, plus a shared registry-token render and Hawser's application secret
Internal DNS	Server cert for its own admin UI/API only
Security Monitoring	Its manager and dashboard services' shared secrets (indexer, API, dashboard passwords), plus Hawser's application secret, plus a shared registry-token render
NetBird	Its entire server config rendered as one file, embedded secrets included — the same file-templating pattern used for Kestra's application.yml — plus Hawser's application secret, plus a shared registry-token render

None of these ever call the Vault API themselves — every one of them goes through the same local-render sequence:

Three authentication flows, not one:

Flow	When	Path
AppRole auth	Every host, ongoing	Vault Agent → role_id + secret_id → Vault → short-lived token, cached, auto-renewed
Provisioner auth	Occasional, playbook-triggered	A narrowly-scoped AppRole mints new host AppRole credentials and reads two specific KV paths — cannot create policies or AppRole roles itself
Break-glass auth	Rare, human-initiated, no standing token	Operator + a quorum of unseal-key shares → a freshly-minted root token, used once, revoked immediately — the only path that can create policies or onboard a genuinely new host

Important

The one deliberately-asymmetric trust decision: every other backend connection inside the internal zone described in docs/dns-naming.md skips certificate verification on the reverse-proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic between containers that already share a trusted network. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert. Given that Vault is the root of trust for every other secret in the fleet, treating it identically to a media-server backend would have been the actual mistake — the asymmetry here is the correct call, not an inconsistency. This is the real config contrast, not a description of it — every other backend uses this:

handle @trusted {
    reverse_proxy https://{args.0} {
        transport http { tls_insecure_skip_verify }
    }
}

Vault's own block, specifically, does not:

handle @trusted {
    reverse_proxy https://[Vault's IP]:8200 {
        transport http {
            tls_trust_pool file { pem_file /etc/caddy/certs/pki_int_ca.crt }
            tls_server_name vault.[app-to-app zone]
        }
    }
}

What's Protected

Layer	Protection
Transport	Every Vault Agent ↔ Vault connection is TLS, certificate-verified
Secrets at rest	KV v2, encrypted by Vault's storage layer; rendered files are 0600, root-owned, gitignored
Audit trail	Every request/response logged to a file audit device; secret values HMAC-hashed, not plaintext — what's visible is metadata: which token, which path, when. Rotated weekly, 8 generations kept.
Alerting	Any root-policy token being used, or permission-denied bursts, trigger an alert through docs/notifications.md's pipeline — the check matches generically on the token's policy, so it fires identically whether that's a routine mistake or a genuine break-glass event
Cert renewal reliability	Technitium's DNS-over-TLS/DoH cert bundle auto-rebuilds whenever Vault Agent re-renders its certs, guarded to skip unless under 7 days from expiry. This closed a real prior gap: the rebuild never used to happen automatically, and an expired cert was silently served for 19 days before anyone caught it.

This is the real hook script, not a description of it — the full guard logic, the actual PKCS12 rebuild, and the service restart it triggers (password redacted, everything else genuine):

Access Model

One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does not automatically imply the private one, so both get verified before assuming a host is fully provisioned. This same "shared + private" pattern shows up again in docs/iac-pipeline.md's tag-driven Ansible model — a deliberate consistency, not a coincidence.

Beyond the host-scoped policies, a handful of narrowly-scoped read-only policies exist for specific, single-purpose needs — an Ansible role reading one sync secret at playbook-run time, Prometheus/Alloy reading Vault's own metrics endpoint, an auditor listing issued certificates across both PKI mounts. Each one grants exactly one path and nothing else — the same least-privilege instinct as the host policies, applied at a finer grain.

That "HMAC-hashed, not plaintext" claim in the table above isn't just a description — it's what a real audit log entry actually looks like:

---

Related: docs/secret-distribution.md, docs/firewall-and-network.md · Stacks: stacks/identity-and-security/vaultwarden/