iac-pipeline.md: full 7-step audit
- Replaced both mermaid diagrams with real Nextcloud diagrams; the Ansible-stages rebuild caught a real omission - the old diagram showed only 2 stages (init/compliance) when there are genuinely 3 (provisioning.yml exists too, currently just clamav/rocky-docker) - Live-verified against the real iac repo: roles_metadata.yml (9 cis_hardening + 6 other compliance roles, 1 provisioning role), kickstart partitioning/packages/SSH config, both Packer bootstrap scripts, terraform.tfvars, pct list on both hypervisors, a real rendered host_tags.yml - KB fix: LXC Strategy note's PBS row was wrong, not just stale - PBS genuinely has a populated vm_tags list and live compliance roles (confirmed via real artifacts on the host itself), not 'never Ansible-managed' as it said; corrected and re-verified via re-fetch - Added real content not previously in the doc: the deliberately permissive build-time SSH window closed by ssh_hardening, the admin/ansible identity split baked into the image at build time, and a third pipeline exception (PBS) that's genuinely different in kind from the other two, including a real Rocky-vs-Debian PAM ordering bug found onboarding it - Considered two screenshot candidates (Proxmox template list, VM/LXC mix) - declined as low-value; command evidence covers this source-driven topic well on its own - Sanitize-checked: no internal IPs/MACs/domains/personal info/forbidden app names 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 5 <[email protected]>
This commit is contained in:
thesabearandClaude Sonnet 5 committed
1 parent
d7a472d85f
commit
c6266c0081
3 files changed
+113
-15
No files matched your search
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Do not edit this file with editors other than draw.io -->
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="932px" height="318px" viewBox="0 0 932 318" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Ansible Stages" id="iac-ansiblestages-1"> <mxGraphModel dx="1414" dy="791" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1080" pageHeight="360" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="tags" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Per-host tag list&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;from Terraform&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="200" x="20" y="140" as="geometry" /> </mxCell> <mxCell id="init" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Initialization stage&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;runs once, at creation&lt;/font&gt;" vertex="1"> <mxGeometry height="65" width="220" x="360" y="20" as="geometry" /> </mxCell> <mxCell id="prov" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Provisioning stage&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;every run — 1 role today, one host&lt;/font&gt;" vertex="1"> <mxGeometry height="65" width="220" x="360" y="140" as="geometry" /> </mxCell> <mxCell id="comp" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Compliance stage&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;re-applied every run&lt;/font&gt;" vertex="1"> <mxGeometry height="65" width="220" x="360" y="260" as="geometry" /> </mxCell> <mxCell id="sub" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="9 independently-tagged&lt;br&gt;hardening sub-roles" vertex="1"> <mxGeometry height="65" width="220" x="720" y="260" as="geometry" /> </mxCell> <mxCell id="e1" edge="1" parent="1" source="tags" style="html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;" target="init"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="e2" edge="1" parent="1" source="tags" style="html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;" target="prov"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="e3" edge="1" parent="1" source="tags" style="html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;" target="comp"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="e4" edge="1" parent="1" source="comp" style="html=1;endArrow=block;strokeColor=#82b366;strokeWidth=2;rounded=1;" target="sub"> <mxGeometry relative="1" as=&quoLine truncated
|
||||
|
After Width: | Height: | Size: 153 KiB |
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Do not edit this file with editors other than draw.io -->
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="1052px" height="132px" viewBox="0 0 1052 132" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Tools Flow" id="iac-toolsflow-1"> <mxGraphModel dx="1414" dy="791" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1080" pageHeight="240" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="packer" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Packer&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;builds the base image&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="200" x="20" y="80" as="geometry" /> </mxCell> <mxCell id="terraform" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Terraform&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;provisions the VM&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="200" x="300" y="80" as="geometry" /> </mxCell> <mxCell id="tags" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="host_tags.yml&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;the tag list&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="200" x="580" y="80" as="geometry" /> </mxCell> <mxCell id="ansible" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Ansible&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;configures &amp;amp; hardens&lt;/font&gt;" vertex="1"> <mxGeometry height="70" width="200" x="860" y="80" as="geometry" /> </mxCell> <mxCell id="e1" edge="1" parent="1" source="packer" style="html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;" target="terraform"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="e2" edge="1" parent="1" source="terraform" style="html=1;labelBackgroundColor=#1C1D1E;endArrow=block;strokeColor=#d79b00;fontColor=#d79b00;strokeWidth=2;rounded=1;fontSize=11;" target="tags" value="&lt;span style=&quot;background-color: light-dark(rgb(255, 255, 255), rgb(28, 29, 30));&quot;&gt;generates&lt;/span&gt;"> <mxGeometry relative="1" y="15" as="geometry" /> </mxCell> <mxCell id="e3" edge="1" parent="1" source="tags" style="html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;" target="ansible"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="loop" edge="1" parent="1" source="ansible" style="html=1;endArrow=block;strokeColor=#82b366;strokeWidth=2;rounded=1;curved=1;labelBackgroundColor=#1C1D1E;fontColor=#82b366;fontSize=11;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;" target="ansible" value="&lt;span style=&quot;background-color: light-dark(rgb(255, 255, 255), rgb(28, 29, 30));&quot;&gt;&lt;font style=&quot;font-size: 12px;&quot;&gt;re-applied every run&lt;/font&gt;&lt;/span&gt;"> <mxGeometry Line truncated
|
||||
|
After Width: | Height: | Size: 125 KiB |
+105
-15
@@ -5,13 +5,7 @@
|
||||
|
||||
## The pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Packer["Packer<br/>builds the base image"] --> Terraform["Terraform<br/>provisions the VM"]
|
||||
Terraform -->|generates| Tags["host_tags.yml<br/>the tag list"]
|
||||
Tags --> Ansible["Ansible<br/>configures & hardens"]
|
||||
Ansible -->|re-applied every run| Compliance["Compliance stage<br/>not just once at creation"]
|
||||
```
|
||||
|
||||
|
||||
## Why three separate tools instead of one
|
||||
|
||||
@@ -19,7 +13,23 @@ A single tool covering image-build-through-configuration was a real option — s
|
||||
|
||||
## Packer: hardening baked in at install time, not bolted on after
|
||||
|
||||
A single base image (`rocky10-cis`) is built from a minimal Rocky Linux 10 install with CIS-oriented hardening baked into the kickstart itself — disk partitioning (mount options like `nodev`/`nosuid`/`noexec` on specific partitions) and package selection decided at install time, not patched in afterward by Ansible. This was a deliberate split with the compliance work in [`docs/security-monitoring.md`](security-monitoring.md): some controls are genuinely easier and more reliable to get right once, at image-build time, than to retrofit repeatedly with a configuration tool after the fact — partition layout in particular can't be changed after boot without real risk, so it has to be correct in the image, not corrected later.
|
||||
A single base image (`rocky10-cis`) is built from a minimal Rocky Linux 10 install with CIS-oriented hardening baked into the kickstart itself — disk partitioning (mount options like `nodev`/`nosuid`/`noexec` on specific partitions) and package selection decided at install time, not patched in afterward by Ansible. This was a deliberate split with the compliance work in [`docs/security-monitoring.md`](security-monitoring.md): some controls are genuinely easier and more reliable to get right once, at image-build time, than to retrofit repeatedly with a configuration tool after the fact — partition layout in particular can't be changed after boot without real risk, so it has to be correct in the image, not corrected later. This is the real kickstart partitioning, not a paraphrase of the intent:
|
||||
|
||||
```
|
||||
$ grep -A8 'Logical volumes' http/ks.cfg
|
||||
logvol /tmp --fstype=xfs --size=2048 --name=lv_tmp --vgname=rocky --fsoptions="defaults,nodev,nosuid"
|
||||
logvol /home --fstype=xfs --size=2048 --name=lv_home --vgname=rocky --fsoptions="defaults,nodev"
|
||||
logvol /var --fstype=xfs --size=4096 --name=lv_var --vgname=rocky --fsoptions="defaults,nodev"
|
||||
logvol /var/tmp --fstype=xfs --size=1024 --name=lv_vartmp --vgname=rocky --fsoptions="defaults,nodev,nosuid,noexec"
|
||||
logvol /var/log --fstype=xfs --size=2048 --name=lv_varlog --vgname=rocky --fsoptions="defaults,nodev,noexec,nosuid"
|
||||
logvol /var/log/audit --fstype=xfs --size=2048 --name=lv_audit --vgname=rocky --fsoptions="defaults,nodev,noexec,nosuid"
|
||||
logvol / --fstype=xfs --name=lv_root --vgname=rocky --grow --size=512 --fsoptions="defaults"
|
||||
```
|
||||
|
||||
`/var/tmp` and `/var/log` both get the full `nodev,nosuid,noexec` treatment; `/home` and `/var` only get `nodev` — a reader can see exactly which mount points earned the stricter posture and which didn't, rather than one blanket policy applied everywhere.
|
||||
|
||||
> [!TIP]
|
||||
> Two things happen at image-build time that aren't part of the pipeline's "steady state" and are easy to miss reading only the diagram above. First, the kickstart deliberately leaves SSH in a **permissive** state during the build — `PermitRootLogin yes` and `PasswordAuthentication yes` — because the build environment isn't reachable from anywhere except the hypervisor doing the build, and that window closes for good once the compliance stage's `ssh_hardening` role runs on a provisioned host. Second, Packer's own post-install provisioners create both fleet identities right there in the image — a personal admin account, and the dedicated `ansible` service account used for all automation — and the admin account's setup script ends by writing a sudoers rule that explicitly **denies it permission to run `ansible`/`ansible-playbook` at all**. Automation only ever runs as the service account; that's not a convention someone has to remember, it's baked into every host from before it's even provisioned.
|
||||
|
||||
## Terraform: one resource type, tag generation as the real output
|
||||
|
||||
@@ -28,21 +38,101 @@ A single base image (`rocky10-cis`) is built from a minimal Rocky Linux 10 insta
|
||||
|
||||
Terraform only manages virtual machines, deliberately — containers are handled differently (below), a scope boundary that avoids fighting Terraform's own resource model for something it wasn't built to represent well.
|
||||
|
||||
## Ansible: tag-driven, re-applied continuously
|
||||
This is the actual template that renders into every host's tag file — the literal source, not a description of its fields:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Tags["Per-host tag list<br/>from Terraform"] --> Init["Initialization stage<br/>runs once, at creation"]
|
||||
Tags --> Compliance["Compliance stage<br/>re-applied every run"]
|
||||
Compliance --> Sub["9 independently-tagged<br/>hardening sub-roles"]
|
||||
```
|
||||
$ cat host_tags.yml.tpl
|
||||
vmid: ${vmid}
|
||||
vm_name: ${name}
|
||||
target_node: ${target_node}
|
||||
template: ${template}
|
||||
cores: ${cores}
|
||||
sockets: ${sockets}
|
||||
memory: ${memory}
|
||||
balloon: ${balloon}
|
||||
bridge: ${bridge}
|
||||
macaddr: ${macaddr}
|
||||
disk_size: ${disk_size}GB
|
||||
disk_storage: ${disk_storage}
|
||||
ip: ${ip}
|
||||
vlan: ${vlan}
|
||||
vm_tags:
|
||||
%{ for tag in tags ~}
|
||||
- ${tag}
|
||||
%{ endfor ~}
|
||||
```
|
||||
|
||||
Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
|
||||
Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes. And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, `ip`/`macaddr` redacted:
|
||||
|
||||
```
|
||||
$ cat host_vars/technitium/host_tags.yml
|
||||
vmid: 118
|
||||
vm_name: technitium
|
||||
target_node: hyper2
|
||||
template: rocky10-cis
|
||||
cores: 2
|
||||
sockets: 1
|
||||
memory: 3072
|
||||
balloon: 1536
|
||||
bridge: vmbr0
|
||||
macaddr: [technitium's MAC]
|
||||
disk_size: 50GB
|
||||
disk_storage: local
|
||||
ip: [technitium's IP]
|
||||
vlan: 0
|
||||
vm_tags:
|
||||
- packages
|
||||
- hostname_network
|
||||
- root_password
|
||||
- disk_resize
|
||||
- base_users
|
||||
- firewall_ssh
|
||||
- zsh_custom
|
||||
- vault_agent
|
||||
- technitium
|
||||
- ssh_hardening
|
||||
- password_policy
|
||||
- banners
|
||||
- file_permissions
|
||||
- service_disable
|
||||
- kernel_hardening
|
||||
- filesystem_hardening
|
||||
- chrony_hardening
|
||||
- auditd_hardening
|
||||
- observability_agent
|
||||
- mqtt_client
|
||||
- wazuh_agent_tuning
|
||||
```
|
||||
|
||||
21 tags on this one host alone, spanning both the initialization roles (`packages`, `firewall_ssh`, …) and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads.
|
||||
|
||||
## Ansible: tag-driven, re-applied continuously
|
||||
|
||||
|
||||
|
||||
There are genuinely three stages, not two — the third one is easy to miss because it's currently almost empty. `provisioning.yml` exists for roles that fit neither "once at init" nor "every compliance run," and as of this writing exactly one role is tagged for it:
|
||||
|
||||
```
|
||||
$ python3 -c "
|
||||
import yaml
|
||||
d = yaml.safe_load(open('roles_metadata.yml'))['roles_metadata']
|
||||
print('provisioning:', [k for k,v in d.items() if v.get('provisioning') is True])
|
||||
print('compliance:', sorted(k for k,v in d.items() if v.get('compliance') is True))
|
||||
"
|
||||
provisioning: ['clamav']
|
||||
compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom']
|
||||
```
|
||||
|
||||
`clamav` is the only provisioning-stage role, and only one host carries it in its `vm_tags`. An under-used pipeline stage isn't the same as an unused one — it exists on purpose for exactly this shape of role, not as leftover scaffolding.
|
||||
|
||||
Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 15 total compliance-stage roles above — the other 6 handle observability, notifications, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
|
||||
|
||||
## The honest exception: not everything fits the clean model
|
||||
|
||||
> [!WARNING]
|
||||
> Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into LXC containers outside Terraform's management — a resource-type mismatch that Terraform's VM-only tooling genuinely can't represent (LXCs aren't `proxmox_vm_qemu` resources). Rather than deleting their old Terraform entries when this happened, they were commented out, preserving the historical record of how these hosts actually came to exist rather than presenting a cleaner story than what's true. Ansible still configures both today through the same tag-driven mechanism as every VM — only the initial provisioning step is now manual instead of automated. This is the kind of real-world messiness that a from-scratch design document never mentions but that every infrastructure actually accumulates over time — worth showing, not hiding.
|
||||
>
|
||||
> A third host breaks the model in a genuinely different way — not a variation on the same story, a different one entirely. Proxmox Backup Server runs as the vendor's own official Debian-based appliance image, deployed by hand and never a Terraform resource at all — there's no commented-out entry to preserve, because there was never a Terraform-managed version of it to convert away from. It still ended up on the tag-driven Ansible side of the pipeline: wired into its own inventory group and given a hand-written tag list after the fact, since compliance only ever checks whether a tag is present, not how a host arrived in the inventory. It's also the fleet's only non-Rocky host, which surfaced a real gap in one of the 9 hardening sub-roles above: `password_policy` configures PAM lockout behavior, and the ordering it assumes is Rocky's own — safe on every other host, but wrong on this one until it was fixed at the role level and explicitly re-validated.
|
||||
|
||||
---
|
||||
Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/disaster-recovery.md`](disaster-recovery.md)
|
||||
Reference in new issue
Block a user