From c6266c0081a784e2d4c906b6772fb47edda93e6e Mon Sep 17 00:00:00 2001 From: thesabear Date: Sat, 29 Aug 2026 08:21:38 +0300 Subject: [PATCH] iac-pipeline.md: full 7-step audit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replaced both mermaid diagrams with real Nextcloud diagrams; the Ansible-stages rebuild caught a real omission - the old diagram showed only 2 stages (init/compliance) when there are genuinely 3 (provisioning.yml exists too, currently just clamav/rocky-docker) - Live-verified against the real iac repo: roles_metadata.yml (9 cis_hardening + 6 other compliance roles, 1 provisioning role), kickstart partitioning/packages/SSH config, both Packer bootstrap scripts, terraform.tfvars, pct list on both hypervisors, a real rendered host_tags.yml - KB fix: LXC Strategy note's PBS row was wrong, not just stale - PBS genuinely has a populated vm_tags list and live compliance roles (confirmed via real artifacts on the host itself), not 'never Ansible-managed' as it said; corrected and re-verified via re-fetch - Added real content not previously in the doc: the deliberately permissive build-time SSH window closed by ssh_hardening, the admin/ansible identity split baked into the image at build time, and a third pipeline exception (PBS) that's genuinely different in kind from the other two, including a real Rocky-vs-Debian PAM ordering bug found onboarding it - Considered two screenshot candidates (Proxmox template list, VM/LXC mix) - declined as low-value; command evidence covers this source-driven topic well on its own - Sanitize-checked: no internal IPs/MACs/domains/personal info/forbidden app names 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 5 --- assets/iac-pipeline-ansible-stages.svg | 4 + assets/iac-pipeline-tools-flow.svg | 4 + docs/iac-pipeline.md | 120 +++++++++++++++++++++---- 3 files changed, 113 insertions(+), 15 deletions(-) create mode 100644 assets/iac-pipeline-ansible-stages.svg create mode 100644 assets/iac-pipeline-tools-flow.svg diff --git a/assets/iac-pipeline-ansible-stages.svg b/assets/iac-pipeline-ansible-stages.svg new file mode 100644 index 0000000..da3dc7a --- /dev/null +++ b/assets/iac-pipeline-ansible-stages.svg @@ -0,0 +1,4 @@ + + + +
Per-host tag list
from Terraform
Initialization stage
runs once, at creation
Provisioning stage
every run — 1 role today, one host
Compliance stage
re-applied every run
9 independently-tagged
hardening sub-roles
 \ No newline at end of file diff --git a/assets/iac-pipeline-tools-flow.svg b/assets/iac-pipeline-tools-flow.svg new file mode 100644 index 0000000..475e427 --- /dev/null +++ b/assets/iac-pipeline-tools-flow.svg @@ -0,0 +1,4 @@ + + + +
Packer
builds the base image
Terraform
provisions the VM
host_tags.yml
the tag list
Ansible
configures & hardens
generates
re-applied every run
 \ No newline at end of file diff --git a/docs/iac-pipeline.md b/docs/iac-pipeline.md index d77d959..56d2779 100644 --- a/docs/iac-pipeline.md +++ b/docs/iac-pipeline.md @@ -5,13 +5,7 @@ ## The pipeline -```mermaid -flowchart LR - Packer["Packer
builds the base image"] --> Terraform["Terraform
provisions the VM"] - Terraform -->|generates| Tags["host_tags.yml
the tag list"] - Tags --> Ansible["Ansible
configures & hardens"] - Ansible -->|re-applied every run| Compliance["Compliance stage
not just once at creation"] -``` +![Packer builds the base image, Terraform provisions the VM and generates host_tags.yml, which Ansible consumes to configure and harden the host, re-applying its compliance stage on every run](../assets/iac-pipeline-tools-flow.svg) ## Why three separate tools instead of one @@ -19,7 +13,23 @@ A single tool covering image-build-through-configuration was a real option — s ## Packer: hardening baked in at install time, not bolted on after -A single base image (`rocky10-cis`) is built from a minimal Rocky Linux 10 install with CIS-oriented hardening baked into the kickstart itself — disk partitioning (mount options like `nodev`/`nosuid`/`noexec` on specific partitions) and package selection decided at install time, not patched in afterward by Ansible. This was a deliberate split with the compliance work in [`docs/security-monitoring.md`](security-monitoring.md): some controls are genuinely easier and more reliable to get right once, at image-build time, than to retrofit repeatedly with a configuration tool after the fact — partition layout in particular can't be changed after boot without real risk, so it has to be correct in the image, not corrected later. +A single base image (`rocky10-cis`) is built from a minimal Rocky Linux 10 install with CIS-oriented hardening baked into the kickstart itself — disk partitioning (mount options like `nodev`/`nosuid`/`noexec` on specific partitions) and package selection decided at install time, not patched in afterward by Ansible. This was a deliberate split with the compliance work in [`docs/security-monitoring.md`](security-monitoring.md): some controls are genuinely easier and more reliable to get right once, at image-build time, than to retrofit repeatedly with a configuration tool after the fact — partition layout in particular can't be changed after boot without real risk, so it has to be correct in the image, not corrected later. This is the real kickstart partitioning, not a paraphrase of the intent: + +``` +$ grep -A8 'Logical volumes' http/ks.cfg +logvol /tmp --fstype=xfs --size=2048 --name=lv_tmp --vgname=rocky --fsoptions="defaults,nodev,nosuid" +logvol /home --fstype=xfs --size=2048 --name=lv_home --vgname=rocky --fsoptions="defaults,nodev" +logvol /var --fstype=xfs --size=4096 --name=lv_var --vgname=rocky --fsoptions="defaults,nodev" +logvol /var/tmp --fstype=xfs --size=1024 --name=lv_vartmp --vgname=rocky --fsoptions="defaults,nodev,nosuid,noexec" +logvol /var/log --fstype=xfs --size=2048 --name=lv_varlog --vgname=rocky --fsoptions="defaults,nodev,noexec,nosuid" +logvol /var/log/audit --fstype=xfs --size=2048 --name=lv_audit --vgname=rocky --fsoptions="defaults,nodev,noexec,nosuid" +logvol / --fstype=xfs --name=lv_root --vgname=rocky --grow --size=512 --fsoptions="defaults" +``` + +`/var/tmp` and `/var/log` both get the full `nodev,nosuid,noexec` treatment; `/home` and `/var` only get `nodev` — a reader can see exactly which mount points earned the stricter posture and which didn't, rather than one blanket policy applied everywhere. + +> [!TIP] +> Two things happen at image-build time that aren't part of the pipeline's "steady state" and are easy to miss reading only the diagram above. First, the kickstart deliberately leaves SSH in a **permissive** state during the build — `PermitRootLogin yes` and `PasswordAuthentication yes` — because the build environment isn't reachable from anywhere except the hypervisor doing the build, and that window closes for good once the compliance stage's `ssh_hardening` role runs on a provisioned host. Second, Packer's own post-install provisioners create both fleet identities right there in the image — a personal admin account, and the dedicated `ansible` service account used for all automation — and the admin account's setup script ends by writing a sudoers rule that explicitly **denies it permission to run `ansible`/`ansible-playbook` at all**. Automation only ever runs as the service account; that's not a convention someone has to remember, it's baked into every host from before it's even provisioned. ## Terraform: one resource type, tag generation as the real output @@ -28,21 +38,101 @@ A single base image (`rocky10-cis`) is built from a minimal Rocky Linux 10 insta Terraform only manages virtual machines, deliberately — containers are handled differently (below), a scope boundary that avoids fighting Terraform's own resource model for something it wasn't built to represent well. -## Ansible: tag-driven, re-applied continuously +This is the actual template that renders into every host's tag file — the literal source, not a description of its fields: -```mermaid -flowchart TD - Tags["Per-host tag list
from Terraform"] --> Init["Initialization stage
runs once, at creation"] - Tags --> Compliance["Compliance stage
re-applied every run"] - Compliance --> Sub["9 independently-tagged
hardening sub-roles"] +``` +$ cat host_tags.yml.tpl +vmid: ${vmid} +vm_name: ${name} +target_node: ${target_node} +template: ${template} +cores: ${cores} +sockets: ${sockets} +memory: ${memory} +balloon: ${balloon} +bridge: ${bridge} +macaddr: ${macaddr} +disk_size: ${disk_size}GB +disk_storage: ${disk_storage} +ip: ${ip} +vlan: ${vlan} +vm_tags: +%{ for tag in tags ~} + - ${tag} +%{ endfor ~} ``` -Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior. +Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes. And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, `ip`/`macaddr` redacted: + +``` +$ cat host_vars/technitium/host_tags.yml +vmid: 118 +vm_name: technitium +target_node: hyper2 +template: rocky10-cis +cores: 2 +sockets: 1 +memory: 3072 +balloon: 1536 +bridge: vmbr0 +macaddr: [technitium's MAC] +disk_size: 50GB +disk_storage: local +ip: [technitium's IP] +vlan: 0 +vm_tags: + - packages + - hostname_network + - root_password + - disk_resize + - base_users + - firewall_ssh + - zsh_custom + - vault_agent + - technitium + - ssh_hardening + - password_policy + - banners + - file_permissions + - service_disable + - kernel_hardening + - filesystem_hardening + - chrony_hardening + - auditd_hardening + - observability_agent + - mqtt_client + - wazuh_agent_tuning +``` + +21 tags on this one host alone, spanning both the initialization roles (`packages`, `firewall_ssh`, …) and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads. + +## Ansible: tag-driven, re-applied continuously + +![The per-host tag list from Terraform drives three independent Ansible stages: initialization (runs once, at creation), provisioning (every run, one role today on one host), and compliance (re-applied every run), which fans out into 9 independently-tagged hardening sub-roles](../assets/iac-pipeline-ansible-stages.svg) + +There are genuinely three stages, not two — the third one is easy to miss because it's currently almost empty. `provisioning.yml` exists for roles that fit neither "once at init" nor "every compliance run," and as of this writing exactly one role is tagged for it: + +``` +$ python3 -c " +import yaml +d = yaml.safe_load(open('roles_metadata.yml'))['roles_metadata'] +print('provisioning:', [k for k,v in d.items() if v.get('provisioning') is True]) +print('compliance:', sorted(k for k,v in d.items() if v.get('compliance') is True)) +" +provisioning: ['clamav'] +compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom'] +``` + +`clamav` is the only provisioning-stage role, and only one host carries it in its `vm_tags`. An under-used pipeline stage isn't the same as an unused one — it exists on purpose for exactly this shape of role, not as leftover scaffolding. + +Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 15 total compliance-stage roles above — the other 6 handle observability, notifications, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior. ## The honest exception: not everything fits the clean model > [!WARNING] > Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into LXC containers outside Terraform's management — a resource-type mismatch that Terraform's VM-only tooling genuinely can't represent (LXCs aren't `proxmox_vm_qemu` resources). Rather than deleting their old Terraform entries when this happened, they were commented out, preserving the historical record of how these hosts actually came to exist rather than presenting a cleaner story than what's true. Ansible still configures both today through the same tag-driven mechanism as every VM — only the initial provisioning step is now manual instead of automated. This is the kind of real-world messiness that a from-scratch design document never mentions but that every infrastructure actually accumulates over time — worth showing, not hiding. +> +> A third host breaks the model in a genuinely different way — not a variation on the same story, a different one entirely. Proxmox Backup Server runs as the vendor's own official Debian-based appliance image, deployed by hand and never a Terraform resource at all — there's no commented-out entry to preserve, because there was never a Terraform-managed version of it to convert away from. It still ended up on the tag-driven Ansible side of the pipeline: wired into its own inventory group and given a hand-written tag list after the fact, since compliance only ever checks whether a tag is present, not how a host arrived in the inventory. It's also the fleet's only non-Rocky host, which surfaced a real gap in one of the 9 hardening sub-roles above: `password_policy` configures PAM lockout behavior, and the ordering it assumes is Rocky's own — safe on every other host, but wrong on this one until it was fixed at the role level and explicitly re-validated. --- Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/disaster-recovery.md`](disaster-recovery.md)