# 🔐 Secrets & PKI

> [!NOTE]
> One HashiCorp Vault cluster is both the certificate authority and the secret store for the entire fleet. Every other system in this repo either consumes something Vault issues, or exists specifically to protect Vault itself — this is the trust root everything else builds on.

| | |
|---|---|
| **Software** | HashiCorp Vault, self-hosted, native systemd service (not containerized) |
| **Role** | Certificate authority (2 PKI mounts) + secret store (KV v2) |
| **Distribution** | Vault Agent on every consuming host, AppRole auth, renders to local disk |
| **Replaces** | A prior plaintext-`.env`-in-git secrets model |

## Why Vault, and Why Self-hosted

A managed secrets service (a cloud provider's KMS/Secrets Manager) was a real alternative and was rejected for a specific reason: this fleet's PKI needs to issue certificates to purely internal hosts that have no business talking to a cloud API, and the two-zone reverse-proxy model in [`docs/dns-naming.md`](dns-naming.md) depends on an internal CA that's reachable without any external dependency. Vault does both certificate issuance and secret storage under one trust model instead of stitching together a separate CA tool and a separate secrets tool — fewer moving parts, one audit log, one policy language to reason about.

## What It Does

- **Certificate authority** — two PKI mounts: one for public-facing wildcard certs (consumed by the edge in [`docs/firewall-and-network.md`](firewall-and-network.md)), one for internal mTLS certs (consumed by the browser-facing reverse-proxy zone). Both issue short-lived certificates, not long-lived ones — a deliberate choice that shrinks the window a leaked cert is useful for.
- **Secret store** — a KV v2 mount holds every migrated application's real secret values, one namespace per host, enforced by policy (below).
- **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly — this single rule is what makes the rest of the model tractable to reason about.

## Why It's Built This Way

Before this system existed, every stack's secrets sat in plaintext inside git-tracked `.env` files — readable by anyone with repo access, and permanently baked into git history even after being "fixed." SecureVault replaces that:

- The real secret value lives in exactly one place: Vault.
- What's git-tracked is a template with placeholder syntax, never a real value.
- The rendered file is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
- If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle, no redeploy required.

> [!TIP]
> The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven for one use case, rather than inventing a second one for the next. This is the same instinct behind not building a separate CA tool: one proven mechanism, extended, beats two different mechanisms that each need their own reasoning.

## Architecture

Every Vault Agent reaches Vault through the internal app-to-app proxy zone described in [`docs/dns-naming.md`](dns-naming.md) — never directly. It's the one hop in that entire zone where the proxy actually verifies the backend's certificate instead of skipping it, which is exactly the asymmetric-trust decision explained below.

| Consumer | What it gets from Vault |
|---|---|
| App Host | 12 stacks' worth of application secrets, an internal CA cert to verify Vault's own listener, plus a shared registry-token render |
| Automation Host | 10 stacks' worth of application secrets, plus a shared registry-token render |
| Monitoring Host | Grafana's admin password and the Authentik OIDC secret, plus Hawser's application secret, plus a shared registry-token render |
| User-Facing Proxy | Server + client certs for the mTLS-enforced browser zone, plus a shared registry-token render and Hawser's application secret |
| Internal DNS | Server cert for its own admin UI/API only |
| Security Monitoring | Its manager and dashboard services' shared secrets (indexer, API, dashboard passwords), plus Hawser's application secret, plus a shared registry-token render |

None of these ever call the Vault API themselves — every one of them goes through the same local-render sequence:

![Secret lifecycle: Vault Agent authenticates via AppRole, gets a token, renders the secret to a local file, the application reads that file and never talks to Vault directly — rotation happens asynchronously, updated once in Vault and picked up on the agent's next render](../assets/secrets-and-pki-lifecycle.svg)

**Two authentication flows, not one:**

| Flow | When | Path |
|---|---|---|
| AppRole auth | Every host, ongoing | Vault Agent → role_id + secret_id → Vault → short-lived token, cached, auto-renewed |
| Bootstrap auth | Rare, human-initiated | Operator uses a root token to mint new AppRole credentials or update policies — never part of any workload's runtime path |

> [!IMPORTANT]
> **The one deliberately-asymmetric trust decision**: every other backend connection inside the internal zone described in [`docs/dns-naming.md`](dns-naming.md) skips certificate verification on the reverse-proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic between containers that already share a trusted network. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert. Given that Vault is the root of trust for every other secret in the fleet, treating it identically to a media-server backend would have been the actual mistake — the asymmetry here is the correct call, not an inconsistency. This is the real config contrast, not a description of it — every other backend uses this:
>
> ```
> handle @trusted {
>     reverse_proxy https://{args.0} {
>         transport http { tls_insecure_skip_verify }
>     }
> }
> ```
>
> Vault's own block, specifically, does not:
>
> ```
> handle @trusted {
>     reverse_proxy https://[Vault's IP]:8200 {
>         transport http {
>             tls_trust_pool file { pem_file /etc/caddy/certs/pki_int_ca.crt }
>             tls_server_name vault.[app-to-app zone]
>         }
>     }
> }
> ```

## What's Protected

| Layer | Protection |
|---|---|
| Transport | Every Vault Agent ↔ Vault connection is TLS, certificate-verified |
| Secrets at rest | KV v2, encrypted by Vault's storage layer; rendered files are `0600`, root-owned, gitignored |
| Audit trail | Every request/response logged to a file audit device; secret values HMAC-hashed, not plaintext — what's visible is metadata: which token, which path, when. Rotated weekly, 8 generations kept. |
| Alerting | Root-token use or permission-denied bursts trigger an alert through [`docs/notifications.md`](notifications.md)'s pipeline |
| Cert renewal reliability | Technitium's DNS-over-TLS/DoH cert bundle auto-rebuilds whenever Vault Agent re-renders its certs, guarded to skip unless under 7 days from expiry. This closed a real prior gap: the rebuild never used to happen automatically, and an expired cert was silently served for 19 days before anyone caught it. |

This is the real hook script, not a description of it — the full guard logic, the actual PKCS12 rebuild, and the service restart it triggers (password redacted, everything else genuine):

![The real technitium-rebuild-p12.sh script: the 19-day-expired-cert incident documented in the script's own comments, the 7-day expiry guard, and the actual PKCS12 rebuild + dns.service restart](../assets/secrets-and-pki-cert-rebuild-hook.png)

## Access Model

One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does *not* automatically imply the private one, so both get verified before assuming a host is fully provisioned. This same "shared + private" pattern shows up again in [`docs/iac-pipeline.md`](iac-pipeline.md)'s tag-driven Ansible model — a deliberate consistency, not a coincidence.

Beyond the host-scoped policies, a handful of narrowly-scoped read-only policies exist for specific, single-purpose needs — an Ansible role reading one sync secret at playbook-run time, Prometheus/Alloy reading Vault's own metrics endpoint, an auditor listing issued certificates across both PKI mounts. Each one grants exactly one path and nothing else — the same least-privilege instinct as the host policies, applied at a finer grain.

![Vault's live ACL policy list — one policy per host, plus the narrowly-scoped read-only utility policies](../assets/vault-acl-policies.png)

That "HMAC-hashed, not plaintext" claim in the table above isn't just a description — it's what a real audit log entry actually looks like:

![A real Vault audit log entry: a KV read of the paperless-ngx secret, every value shown as hmac-sha256:... instead of plaintext](../assets/vault-audit-log-hashed.png)

---
Related: [`docs/secret-distribution.md`](secret-distribution.md), [`docs/firewall-and-network.md`](firewall-and-network.md) · Stacks: [`stacks/identity-and-security/vaultwarden/`](../stacks/identity-and-security/vaultwarden/)