# 🔐 Secrets & PKI

> [!NOTE]
> One HashiCorp Vault cluster is both the certificate authority and the secret store for the entire fleet. Every other system in this repo either consumes something Vault issues, or exists specifically to protect Vault itself — this is the trust root everything else builds on.

| | |
|---|---|
| **Software** | HashiCorp Vault, self-hosted, native systemd service (not containerized) |
| **Role** | Certificate authority (2 PKI mounts) + secret store (KV v2) |
| **Distribution** | Vault Agent on every consuming host, AppRole auth, renders to local disk |
| **Replaces** | A prior plaintext-`.env`-in-git secrets model |
| **Biggest known risk** | A non-expiring root bootstrap token — ranked #1 of 3 known gaps, tracked openly |

## Why Vault, and why self-hosted

A managed secrets service (a cloud provider's KMS/Secrets Manager) was a real alternative and was rejected for a specific reason: this fleet's PKI needs to issue certificates to purely internal hosts that have no business talking to a cloud API, and the two-zone reverse-proxy model in [`docs/dns-naming.md`](dns-naming.md) depends on an internal CA that's reachable without any external dependency. Vault does both certificate issuance and secret storage under one trust model instead of stitching together a separate CA tool and a separate secrets tool — fewer moving parts, one audit log, one policy language to reason about.

## What it does

- **Certificate authority** — two PKI mounts: one for public-facing wildcard certs (consumed by the edge in [`docs/firewall-and-network.md`](firewall-and-network.md)), one for internal mTLS certs (consumed by the browser-facing reverse-proxy zone). Both issue short-lived certificates, not long-lived ones — a deliberate choice that shrinks the window a leaked cert is useful for.
- **Secret store** — a KV v2 mount holds every migrated application's real secret values, one namespace per host, enforced by policy (below).
- **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly — this single rule is what makes the rest of the model tractable to reason about.

## Why it's built this way

```mermaid
flowchart LR
    Before["Before:<br/>plaintext secrets<br/>in git-tracked .env files<br/>readable by anyone with repo access,<br/>permanent in git history"] -->|replaced| After

    subgraph After["Now"]
        direction TB
        T["Git-tracked template<br/>placeholder syntax only"]
        V["Vault<br/>real value, one place"]
        R["Rendered file<br/>gitignored, 0600, root-owned"]
        V -->|renders| R
    end
```

- The real secret value lives in exactly one place: Vault.
- What's git-tracked is a template with placeholder syntax, never a real value.
- The rendered file is gitignored, tightly permissioned, and owned entirely by Vault Agent — never hand-edited.
- If a secret rotates, updating it once in Vault is enough; every consumer picks it up on its next render cycle, no redeploy required.

> [!TIP]
> The mechanism (Vault Agent authenticating via AppRole, rendering a template to disk) already existed for TLS certificates before it was extended to cover application secrets — reusing a trust model that was already proven for one use case, rather than inventing a second one for the next. This is the same instinct behind not building a separate CA tool: one proven mechanism, extended, beats two different mechanisms that each need their own reasoning.

## Architecture

```mermaid
flowchart TD
    Vault["Vault<br/>PKI (public + internal mounts) + KV v2 store<br/>TLS listener, cert-verified · audit log"]
    Proxy["Internal app-to-app reverse-proxy zone<br/>see docs/dns-naming.md"]
    A1["Vault Agent — App Host<br/>~12 stacks' secrets + internal CA cert"]
    A2["Vault Agent — Automation Host<br/>~10 stacks' secrets"]
    A3["Vault Agent — Monitoring Host<br/>admin credentials + OIDC secret"]
    A4["Vault Agent — Edge Proxy Host<br/>mTLS zone certs"]
    A5["Vault Agent — DNS Host<br/>admin UI cert"]
    A6["Vault Agent — Security Monitoring Host<br/>manager/indexer/dashboard secrets"]

    Vault -->|HTTPS, cert-verified| Proxy
    Proxy --> A1
    Proxy --> A2
    Proxy --> A3
    Proxy --> A4
    Proxy --> A5
    Proxy --> A6
```

**Two authentication flows, not one:**

| Flow | When | Path |
|---|---|---|
| AppRole auth | Every host, ongoing | Vault Agent → role_id + secret_id → Vault → short-lived token, cached, auto-renewed |
| Bootstrap auth | Rare, human-initiated | Operator uses a root token to mint new AppRole credentials or update policies — never part of any workload's runtime path |

> [!IMPORTANT]
> **The one deliberately-asymmetric trust decision**: every other backend connection inside the internal zone described in [`docs/dns-naming.md`](dns-naming.md) skips certificate verification on the reverse-proxy hop (still encrypted, just doesn't check identity) — a pragmatic tradeoff for low-stakes internal traffic between containers that already share a trusted network. The connection to Vault is the single exception: the proxy verifies Vault's certificate for real, against a dedicated internal CA cert. Given that Vault is the root of trust for every other secret in the fleet, treating it identically to a media-server backend would have been the actual mistake — the asymmetry here is the correct call, not an inconsistency.

## What's protected

| Layer | Protection |
|---|---|
| Transport | Every Vault Agent ↔ Vault connection is TLS, certificate-verified |
| Secrets at rest | KV v2, encrypted by Vault's storage layer; rendered files tightly permissioned, root-owned, gitignored |
| Audit trail | Every request/response logged; secret values HMAC-hashed, not plaintext — what's visible is metadata: which token, which path, when |
| Alerting | Root-token use or permission-denied bursts trigger an alert through [`docs/notifications.md`](notifications.md)'s pipeline |
| Cert renewal reliability | A dependent service's TLS bundle auto-rebuilds when its cert re-issues, guarded to skip unless under a week from expiry |

## Known gaps, ranked by actual risk — not a flat list

Ranking these mattered more than just listing them: a reader (or an attacker) needs to know which one to fix first, not just that gaps exist.

1. **The bootstrap root token never expires.** Unlimited access, mints every AppRole, writes every policy. Encrypted at rest, but that doesn't help if the plaintext ever gets used somewhere it shouldn't. The single highest-value target in the whole system — tracked for replacement with a narrowly-scoped, short-TTL token.
2. **No SSH certificate signing.** Every host's SSH key is static, baked in at image build time (see [`docs/iac-pipeline.md`](iac-pipeline.md)), no rotation, no revocation short of rebuilding the host. Vault's own SSH secrets engine is the natural fit — a bigger lift than #1 since it touches every host's SSH config, not one central policy.
3. **Single-node storage, no replication.** One disk failure is a full outage and, without a recent backup, real data loss. This is the one gap [`docs/disaster-recovery.md`](disaster-recovery.md)'s VM-level backup layer actually covers — an availability concern, correctly triaged as lower priority than the two confidentiality risks above.

> [!TIP]
> **Deliberately accepted, not gaps**: memory locking is disabled (normally a real concern — secrets could swap to disk under memory pressure — but this host has zero swap configured, so there's no actual leak vector regardless), and the Vault server's own configuration is intentionally kept out of the same automation that manages every client-side Vault Agent, so a mistake in that broader automation can't touch the root of trust itself.

## Access model

One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does *not* automatically imply the private one, so both get verified before assuming a host is fully provisioned. This same "shared + private" pattern shows up again in [`docs/iac-pipeline.md`](iac-pipeline.md)'s tag-driven Ansible model — a deliberate consistency, not a coincidence.

---
Related: [`docs/secret-distribution.md`](secret-distribution.md), [`docs/firewall-and-network.md`](firewall-and-network.md) · Stacks: [`stacks/identity-and-security/vaultwarden/`](../stacks/identity-and-security/vaultwarden/)