public/docs/iac-pipeline.md

🏗️ Infrastructure as Code

Note

Packer, Terraform, and Ansible — three tools, three responsibilities, and one thread running through all of them: a host's tag list, generated once by Terraform and consumed by every Ansible stage after it. This is the foundation everything else in this repo actually runs on top of.

The pipeline

Why three separate tools instead of one

A single tool covering image-build-through-configuration was a real option — several products exist specifically for that — and it was rejected because the three jobs have genuinely different rhythms: an image gets rebuilt rarely (a base OS refresh), a VM gets provisioned once, and configuration gets re-applied continuously (the compliance stage below runs on every pass, not just at creation). Collapsing all three into one tool's model would mean forcing the rarely-changing and the continuously-changing onto the same execution model. Keeping them separate, connected by one deliberate handoff (the tag file below) rather than a tightly-coupled integration, means each tool does the one job it's actually good at.

Packer: hardening baked in at install time, not bolted on after

A single base image (rocky10-cis) is built from a minimal Rocky Linux 10 install with CIS-oriented hardening baked into the kickstart itself — disk partitioning (mount options like nodev/nosuid/noexec on specific partitions) and package selection decided at install time, not patched in afterward by Ansible. This was a deliberate split with the compliance work in docs/security-monitoring.md: some controls are genuinely easier and more reliable to get right once, at image-build time, than to retrofit repeatedly with a configuration tool after the fact — partition layout in particular can't be changed after boot without real risk, so it has to be correct in the image, not corrected later. This is the real kickstart partitioning, not a paraphrase of the intent:

$ grep -A8 'Logical volumes' http/ks.cfg
logvol /tmp              --fstype=xfs --size=2048  --name=lv_tmp     --vgname=rocky --fsoptions="defaults,nodev,nosuid"
logvol /home             --fstype=xfs --size=2048  --name=lv_home    --vgname=rocky --fsoptions="defaults,nodev"
logvol /var              --fstype=xfs --size=4096  --name=lv_var     --vgname=rocky --fsoptions="defaults,nodev"
logvol /var/tmp          --fstype=xfs --size=1024  --name=lv_vartmp  --vgname=rocky --fsoptions="defaults,nodev,nosuid,noexec"
logvol /var/log          --fstype=xfs --size=2048  --name=lv_varlog  --vgname=rocky --fsoptions="defaults,nodev,noexec,nosuid"
logvol /var/log/audit    --fstype=xfs --size=2048  --name=lv_audit   --vgname=rocky --fsoptions="defaults,nodev,noexec,nosuid"
logvol /                 --fstype=xfs --name=lv_root --vgname=rocky --grow --size=512 --fsoptions="defaults"

/var/tmp and /var/log both get the full nodev,nosuid,noexec treatment; /home and /var only get nodev — a reader can see exactly which mount points earned the stricter posture and which didn't, rather than one blanket policy applied everywhere.

Tip

Two things happen at image-build time that aren't part of the pipeline's "steady state" and are easy to miss reading only the diagram above. First, the kickstart deliberately leaves SSH in a permissive state during the build — PermitRootLogin yes and PasswordAuthentication yes — because the build environment isn't reachable from anywhere except the hypervisor doing the build, and that window closes for good once the compliance stage's ssh_hardening role runs on a provisioned host. Second, Packer's own post-install provisioners create both fleet identities right there in the image — a personal admin account, and the dedicated ansible service account used for all automation — and the admin account's setup script ends by writing a sudoers rule that explicitly denies it permission to run ansible/ansible-playbook at all. Automation only ever runs as the service account; that's not a convention someone has to remember, it's baked into every host from before it's even provisioned.

Terraform: two resource types, tag generation as the real output

Tip

Every VM is one entry in a single map — VM ID, target node, template, resources, network, disk, and its own tag list. Two supporting pieces do the real connective work: a wait step that blocks completion until the new VM actually answers SSH as the automation identity (so nothing downstream races a host that isn't up yet), and a template render that writes that VM's tag list directly into Ansible's own host-variable tree. This tag file is the actual link between provisioning and configuration — the tags Terraform assigns to a VM are the exact same tags every later Ansible stage (including the security hardening in docs/security-monitoring.md and the observability rollout in docs/observability.md) matches against. Get this file wrong and every downstream stage silently configures the wrong thing for that host — which is why it's generated, not hand-written.

Terraform manages both VMs and LXCs now — two separate resource types, one per guest kind, coexisting in one config with no shared state between them:

	VMs	LXCs
Provider	telmate/proxmox	bpg/proxmox
Resource	proxmox_vm_qemu	proxmox_virtual_environment_container
Source image	the rocky10-cis Packer template — hardening, sshd, both fleet identities pre-baked	stock Rocky 10 template — root-only, no sshd
Ansible group	rocky_vm	rocky_lxc

The LXC path is genuinely newer and mirrors the VM flow step-for-step, but has to close a gap the VM path never has: the stock container template has no sshd and no fleet users at all, since it never went through Packer. A pct exec bootstrap step runs first — installing sshd and creating both fleet identities with their keys, the same baseline Packer bakes into the VM image at build time instead of at provision time — before the same wait-for-SSH and tag-file-render steps every VM already gets. This is the real resource definition, not a paraphrase:

$ head -20 containers.tf
resource "proxmox_virtual_environment_container" "ct" {
  provider = proxmox-bpg
  for_each = var.containers

  node_name     = each.value.target_node
  vm_id         = each.value.vmid
  unprivileged  = each.value.unprivileged
  start_on_boot = each.value.onboot
  started       = true

  tags = each.value.tags

  cpu {
    cores = each.value.cores
  }

  memory {
    dedicated = each.value.memory
    swap      = each.value.swap
  }

An empty var.containers makes this file completely inert — nothing created, no effect — the same "opt-in, not a parallel system that has to be kept in sync" pattern as every other tag-driven piece of this pipeline.

This is the actual template that renders into every host's tag file — the literal source, not a description of its fields:

$ cat host_tags.yml.tpl
vmid: ${vmid}
vm_name: ${name}
target_node: ${target_node}
template: ${template}
cores: ${cores}
sockets: ${sockets}
memory: ${memory}
balloon: ${balloon}
bridge: ${bridge}
macaddr: ${macaddr}
disk_size: ${disk_size}GB
disk_storage: ${disk_storage}
ip: ${ip}
vlan: ${vlan}
vm_tags:
%{ for tag in tags ~}
  - ${tag}
%{ endfor ~}

Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes. And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, ip/macaddr redacted:

$ cat host_vars/technitium/host_tags.yml
vmid: 118
vm_name: technitium
target_node: hyper2
template: rocky10-cis
cores: 2
sockets: 1
memory: 3072
balloon: 1536
bridge: vmbr0
macaddr: [technitium's MAC]
disk_size: 50GB
disk_storage: local
ip: [technitium's IP]
vlan: 0
vm_tags:
  - packages
  - hostname_network
  - root_password
  - disk_resize
  - base_users
  - firewall_ssh
  - zsh_custom
  - vault_agent
  - technitium
  - ssh_hardening
  - password_policy
  - banners
  - file_permissions
  - service_disable
  - kernel_hardening
  - filesystem_hardening
  - chrony_hardening
  - auditd_hardening
  - observability_agent
  - mqtt_client
  - wazuh_agent_tuning

21 tags on this one host alone, spanning both the initialization roles (packages, firewall_ssh, …) and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads.

Ansible: tag-driven, re-applied continuously

There are genuinely three stages, not two — the third one is easy to miss because it's currently almost empty. provisioning.yml exists for roles that fit neither "once at init" nor "every compliance run," and as of this writing exactly one role is tagged for it:

$ python3 -c "
import yaml
d = yaml.safe_load(open('roles_metadata.yml'))['roles_metadata']
print('provisioning:', [k for k,v in d.items() if v.get('provisioning') is True])
print('compliance:', sorted(k for k,v in d.items() if v.get('compliance') is True))
"
provisioning: ['clamav']
compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom']

clamav is the only provisioning-stage role, and only one host carries it in its vm_tags. An under-used pipeline stage isn't the same as an unused one — it exists on purpose for exactly this shape of role, not as leftover scaffolding.

Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 15 total compliance-stage roles above — the other 6 handle observability, notifications, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in docs/security-monitoring.md auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.

The honest exception: not everything fits the clean model

Warning

Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into LXC containers outside Terraform's management — a resource-type mismatch that Terraform's VM-only tooling genuinely can't represent (LXCs aren't proxmox_vm_qemu resources). Rather than deleting their old Terraform entries when this happened, they were commented out, preserving the historical record of how these hosts actually came to exist rather than presenting a cleaner story than what's true. Ansible still configures both today through the same tag-driven mechanism as every VM — only the initial provisioning step is now manual instead of automated. This is the kind of real-world messiness that a from-scratch design document never mentions but that every infrastructure actually accumulates over time — worth showing, not hiding.

A third host breaks the model in a genuinely different way — not a variation on the same story, a different one entirely. Proxmox Backup Server runs as the vendor's own official Debian-based appliance image, deployed by hand and never a Terraform resource at all — there's no commented-out entry to preserve, because there was never a Terraform-managed version of it to convert away from. It still ended up on the tag-driven Ansible side of the pipeline: wired into its own inventory group and given a hand-written tag list after the fact, since compliance only ever checks whether a tag is present, not how a host arrived in the inventory. It's also the fleet's only non-Rocky host, which surfaced a real gap in one of the 9 hardening sub-roles above: password_policy configures PAM lockout behavior, and the ordering it assumes is Rocky's own — safe on every other host, but wrong on this one until it was fixed at the role level and explicitly re-validated.

None of these three exceptions were retroactively fixed once the LXC-provider gap closed — the point of preserving history is that it stays history, not that it gets quietly cleaned up the moment it becomes possible to. But the gap itself is real and worth naming honestly too: for any new LXC going forward, this entire honest-exception story no longer applies. The fleet's zero-trust mesh overlay is the first host actually built through the LXC-native path above, start to finish, with no manual step anywhere in its provisioning.

---

Related: docs/secrets-and-pki.md, docs/security-monitoring.md, docs/disaster-recovery.md, docs/netbird-zero-trust.md