docs: add the SSH Certificate Flow as its own contract, not folded into #1
Vault's ssh-client-signer mount was undocumented here entirely. Adds it as flow 2/5 rather than merging into Certificate Flow, since the actual mechanism differs (own systemd timer, bypasses Vault Agent's render cycle, only reuses its cached token) - and explains why that distinction matters for debugging. Diagram updated to match: a third data-plane box feeding fleet-wide sshd trust instead of a container.
This commit is contained in:
2 files changed
+34
-8
No files matched your search
@@ -1,4 +1,4 @@
|
|||||||
<?xml version="1.0" encoding="UTF-8"?>
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
<!-- Do not edit this file with editors other than draw.io -->
|
<!-- Do not edit this file with editors other than draw.io -->
|
||||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1179px" height="545px" viewBox="0 0 1179 545" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Secret Distribution Flows" id="secret-dist-flows-1"> <mxGraphModel dx="1414" dy="791" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1200" pageHeight="620" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="repo" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;" value="Provisioning Repo&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;template inventory&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="200" x="40" y="60" as="geometry" /> </mxCell> <mxCell id="agentconfig" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;" value="Vault Agent Config&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;on the target host&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="200" x="410" y="60" as="geometry" /> </mxCell> <mxCell id="bgfill_dataplane" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=none;" vertex="1"> <mxGeometry height="290" width="440" x="380" y="230" as="geometry" /> </mxCell> <mxCell id="dataplane" parent="1" style="swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#9673a6;dashed=1;fontSize=12;fontStyle=1;fontColor=#333333;align=center;spacingLeft=8;verticalAlign=middle;" value="DATA PLANE — what Vault Agent actually renders" vertex="1"> <mxGeometry height="290" width="440" x="380" y="230" as="geometry" /> </mxCell> <mxCell id="vaultagent" parent="dataplane" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Vault Agent" vertex="1"> <mxGeometry height="60" width="140" x="150" y="40" as="geometry" /> </mxCell> <mxCell id="certflow" parent="dataplane" style="rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Certificate Flow&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;PKI engine, short-lived&lt;/font&gt;" vertex="1"> <mxGeometry height="65" width="170" x="30" y="180" as="geometry" /> </mxCell> <mxCell id="secflow" parent="dataplane" style="rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="App Secret Flow&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;KV, rendered 0600&lt;/font&gt;" vertex="1"> <mxGeometry height="65" width="170" x="240" y="180" as="geometry" /> </mxCell> <mxCell id="container" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;" value="Container&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;reads local disk only&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="200" x="900"Line truncated
|
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1262px" height="625px" viewBox="0 0 1262 625" content="<mxfile host="embed.diagrams.net" scale="1" border="0">
 <diagram name="Secret Distribution Flows" id="secret-dist-flows-1">
 <mxGraphModel dx="1398" dy="830" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1450" pageHeight="700" math="0" shadow="0">
 <root>
 <mxCell id="0" />
 <mxCell id="1" parent="0" />
 <mxCell id="repo" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;" value="Provisioning Repo&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;template inventory&lt;/font&gt;" vertex="1">
 <mxGeometry height="80" width="200" x="40" y="60" as="geometry" />
 </mxCell>
 <mxCell id="agentconfig" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=14;fontStyle=1;shadow=1;" value="Vault Agent Config&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;on the target host&lt;/font&gt;" vertex="1">
 <mxGeometry height="80" width="200" x="410" y="60" as="geometry" />
 </mxCell>
 <mxCell id="bgfill_dataplane" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=none;" vertex="1">
 <mxGeometry height="290" width="620" x="380" y="230" as="geometry" />
 </mxCell>
 <mxCell id="dataplane" parent="1" style="swimlane;whiteSpace=wrap;html=1;startSize=30;fillColor=none;strokeColor=#9673a6;dashed=1;fontSize=12;fontStyle=1;fontColor=#333333;align=center;spacingLeft=8;verticalAlign=middle;" value="DATA PLANE — Vault-issued, installed locally" vertex="1">
 <mxGeometry height="290" width="620" x="380" y="230" as="geometry" />
 </mxCell>
 <mxCell id="vaultagent" parent="dataplane" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#e1d5e7;strokeColor=#9673a6;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Vault Agent" vertex="1">
 <mxGeometry height="60" width="140" x="240" y="40" as="geometry" />
 </mxCell>
 <mxCell id="certflow" parent="dataplane" style="rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Certificate Flow&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;PKI engine, short-lived&lt;/font&gt;" vertex="1">
 <mxGeometry height="65" width="180" x="20" y="180" as="geometry" />
 </mxCell>
 <mxCell id="secflow" parent="dataplane" style="rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="App Secret Flow&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;KV, rendered 0600&lt;/font&gt;" vertex="1">
 <mxGeometry height="65" width="180" x="220" y="180" as="geometry" />
 </mxCell>
 <mxCell id="sshcertflow" parent="dataplane" style="rounded=1;whiteSpace=wrap;html=1;fillColor=light-dark(#FFFFFF,#392F3F);strokeColor=#9673a6;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="SSH Certificate Flow&lt;br&gt;&lt;font style=&quot;font-size:11px&quot;&gt;own timer, signs via Vault&lt;br/&gt;Agent&#39;s token — not its&lt;br/&gt;render cycle&lt;/font&aLine truncated
|
||||||
|
Before Width: | Height: | Size: 422 KiB After Width: | Height: | Size: 420 KiB |
@@ -1,13 +1,16 @@
|
|||||||
# 🔑 Secret Distribution Model
|
# 🔑 Secret Distribution Model
|
||||||
|
|
||||||
> [!NOTE]
|
> [!NOTE]
|
||||||
> [`docs/secrets-and-pki.md`](secrets-and-pki.md) covers *what* Vault is and *why* it exists. This doc covers something different and, arguably, harder to get right: the actual contract for how a secret value moves from being written into Vault to being consumed by a running container — and why that contract is deliberately split into four separate flows instead of one universal one.
|
> [`docs/secrets-and-pki.md`](secrets-and-pki.md) covers *what* Vault is and *why* it exists. This doc covers something different and, arguably, harder to get right: the actual contract for how a secret value moves from being written into Vault to being consumed by a running container — and why that contract is deliberately split into five separate flows instead of one universal one.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
## Why Four Flows Instead Of One
|
## Why Five Flows Instead Of One
|
||||||
|
|
||||||
The obvious simpler design is one universal "secret goes in, secret comes out" pipeline. It was rejected because the four things being moved genuinely have different risk profiles and different failure tolerances: a certificate expiring is a slow-building problem measured in days; a missing application secret is often an immediate outage; a broken config-distribution push can silently stop *all* of a host's secrets from updating, not just one. Treating them identically would mean either over-engineering the simple cases or under-protecting the risky one. Four flows, each with its own explicit Law/Flow/Failure-behavior contract, is more code to maintain but far easier to reason about when something breaks — anyone debugging an incident knows immediately which contract governs the thing that's wrong.
|
The obvious simpler design is one universal "secret goes in, secret comes out" pipeline. It was rejected because the five things being moved genuinely have different risk profiles and different failure tolerances: a certificate expiring is a slow-building problem measured in days; a missing application secret is often an immediate outage; a broken config-distribution push can silently stop *all* of a host's secrets from updating, not just one; losing the ability to refresh an SSH certificate doesn't break anything immediately, but eventually locks the automation identity out of the fleet entirely. Treating them identically would mean either over-engineering the simple cases or under-protecting the risky ones. Five flows, each with its own explicit Law/Flow/Failure-behavior contract, is more code to maintain but far easier to reason about when something breaks — anyone debugging an incident knows immediately which contract governs the thing that's wrong.
|
||||||
|
|
||||||
|
> [!NOTE]
|
||||||
|
> **Why the SSH Certificate Flow isn't just folded into Certificate Flow** — both issue short-lived Vault-signed credentials, so treating them as one flow looks tempting. They're kept separate because the actual mechanism is genuinely different, not just the credential type: Certificate Flow is triggered and rendered by Vault Agent itself, on Vault Agent's own schedule. The SSH Certificate Flow runs on its own independent systemd timer, calls Vault directly, and only *reuses* Vault Agent's already-authenticated token as a convenience — Vault Agent never renders it and doesn't know it exists. Collapsing them into one contract would hide that difference from exactly the person who needs it most: whoever is debugging why an SSH certificate didn't refresh, who needs to know immediately that the answer isn't "check Vault Agent's logs."
|
||||||
|
|
||||||
## 1️⃣ Certificate Flow
|
## 1️⃣ Certificate Flow
|
||||||
|
|
||||||
@@ -19,7 +22,30 @@ The obvious simpler design is one universal "secret goes in, secret comes out" p
|
|||||||
|
|
||||||
**Invariants:** certificates are short-lived and re-issued on Vault Agent's own schedule, never on-demand by the consumer. A host only ever receives certs tagged for its own identity. The private key never traverses the network after issuance.
|
**Invariants:** certificates are short-lived and re-issued on Vault Agent's own schedule, never on-demand by the consumer. A host only ever receives certs tagged for its own identity. The private key never traverses the network after issuance.
|
||||||
|
|
||||||
## 2️⃣ Application Secret Flow
|
## 2️⃣ SSH Certificate Flow
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| **Law** | The automation identity never holds a static, standing-access SSH key. Only a Vault-signed certificate, valid for hours, ever grants it access to a fleet host. |
|
||||||
|
| **Flow** | Its own systemd timer (2h interval) → reuses Vault Agent's already-authenticated token → `vault write ssh-client-signer/sign/sign-ansible` → certificate installed next to the identity's own key on the one host that needs it → every fleet host's sshd trusts the CA directly, no per-host secret involved |
|
||||||
|
| **Failure behavior** | A single missed refresh is harmless — the timer fires every 2 hours against a 6-hour certificate, so there's hours of margin. A sustained outage spanning several missed cycles eventually locks the automation identity out of certificate-based SSH entirely, with no static key to fall back to — an accepted tradeoff, not an oversight (see [`docs/secrets-and-pki.md`](secrets-and-pki.md)) |
|
||||||
|
|
||||||
|
This is the real signed certificate, not a description of one — installed and ready to use, principal and validity window both visible:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ ssh-keygen -Lf /home/ansible/.ssh/auto-cert.pub
|
||||||
|
Type: [email protected] user certificate
|
||||||
|
Signing CA: RSA SHA256:vcGM3I3hc5AYm9ji1lXYBx1yhVuxR3Gj/3CFdRRkjOo (using rsa-sha2-512)
|
||||||
|
Valid: from 2026-09-05T08:55:53 to 2026-09-05T14:56:23
|
||||||
|
Principals:
|
||||||
|
ansible
|
||||||
|
Extensions:
|
||||||
|
permit-port-forwarding
|
||||||
|
```
|
||||||
|
|
||||||
|
**Invariants:** exactly one identity (`ansible`) is ever signed for. Vault's role config enforces this independently of the script that calls it — a compromised or buggy script still can't get a certificate for any other principal. Every host trusts the CA's public key directly; none of them store the automation identity's actual key material at all.
|
||||||
|
|
||||||
|
## 3️⃣ Application Secret Flow
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -30,7 +56,7 @@ The obvious simpler design is one universal "secret goes in, secret comes out" p
|
|||||||
> [!WARNING]
|
> [!WARNING]
|
||||||
> The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it. This is the single most common way this model gets violated by accident: someone SSHes in to fix a value under time pressure, and the fix silently disappears on the next render.
|
> The rendered file is owned entirely by Vault Agent. Hand-editing it gets silently overwritten on the next render cycle — and hides the actual source of truth from whoever edited it. This is the single most common way this model gets violated by accident: someone SSHes in to fix a value under time pressure, and the fix silently disappears on the next render.
|
||||||
|
|
||||||
## 3️⃣ Configuration Distribution Flow
|
## 4️⃣ Configuration Distribution Flow
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -55,7 +81,7 @@ $ grep -A4 "service: automation" vault_agent/defaults/main.yml
|
|||||||
reload_command: "chmod 600 .../registry-token/registry-token"
|
reload_command: "chmod 600 .../registry-token/registry-token"
|
||||||
```
|
```
|
||||||
|
|
||||||
## 4️⃣ Administrative Secret Flow — a deliberately separate, simpler system
|
## 5️⃣ Administrative Secret Flow — a deliberately separate, simpler system
|
||||||
|
|
||||||
| | |
|
| | |
|
||||||
|---|---|
|
|---|---|
|
||||||
|
|||||||
Reference in new issue
Block a user