public/docs/iac-pipeline.md

🏗️ Infrastructure as Code

Note

Packer, Terraform, and Ansible — three tools, three responsibilities, and one thread running through all of them: a host's tag list, generated once by Terraform and consumed by every Ansible stage after it. This is the foundation everything else in this repo actually runs on top of.

The pipeline

flowchart LR
    Packer["Packer<br/>builds the base image"] --> Terraform["Terraform<br/>provisions the VM"]
    Terraform -->|generates| Tags["host_tags.yml<br/>the tag list"]
    Tags --> Ansible["Ansible<br/>configures & hardens"]
    Ansible -->|re-applied every run| Compliance["Compliance stage<br/>not just once at creation"]

Why three separate tools instead of one

A single tool covering image-build-through-configuration was a real option — several products exist specifically for that — and it was rejected because the three jobs have genuinely different rhythms: an image gets rebuilt rarely (a base OS refresh), a VM gets provisioned once, and configuration gets re-applied continuously (the compliance stage below runs on every pass, not just at creation). Collapsing all three into one tool's model would mean forcing the rarely-changing and the continuously-changing onto the same execution model. Keeping them separate, connected by one deliberate handoff (the tag file below) rather than a tightly-coupled integration, means each tool does the one job it's actually good at.

Packer: hardening baked in at install time, not bolted on after

A single base image (rocky10-cis) is built from a minimal Rocky Linux 10 install with CIS-oriented hardening baked into the kickstart itself — disk partitioning (mount options like nodev/nosuid/noexec on specific partitions) and package selection decided at install time, not patched in afterward by Ansible. This was a deliberate split with the compliance work in docs/security-monitoring.md: some controls are genuinely easier and more reliable to get right once, at image-build time, than to retrofit repeatedly with a configuration tool after the fact — partition layout in particular can't be changed after boot without real risk, so it has to be correct in the image, not corrected later.

Terraform: one resource type, tag generation as the real output

Tip

Every VM is one entry in a single map — VM ID, target node, template, resources, network, disk, and its own tag list. Two supporting pieces do the real connective work: a wait step that blocks completion until the new VM actually answers SSH as the automation identity (so nothing downstream races a host that isn't up yet), and a template render that writes that VM's tag list directly into Ansible's own host-variable tree. This tag file is the actual link between provisioning and configuration — the tags Terraform assigns to a VM are the exact same tags every later Ansible stage (including the security hardening in docs/security-monitoring.md and the observability rollout in docs/observability.md) matches against. Get this file wrong and every downstream stage silently configures the wrong thing for that host — which is why it's generated, not hand-written.

Terraform only manages virtual machines, deliberately — containers are handled differently (below), a scope boundary that avoids fighting Terraform's own resource model for something it wasn't built to represent well.

Ansible: tag-driven, re-applied continuously

flowchart TD
    Tags["Per-host tag list<br/>from Terraform"] --> Init["Initialization stage<br/>runs once, at creation"]
    Tags --> Compliance["Compliance stage<br/>re-applied every run"]
    Compliance --> Sub["9 independently-tagged<br/>hardening sub-roles"]

Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in docs/security-monitoring.md auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.

The honest exception: not everything fits the clean model

Warning

Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into LXC containers outside Terraform's management — a resource-type mismatch that Terraform's VM-only tooling genuinely can't represent (LXCs aren't proxmox_vm_qemu resources). Rather than deleting their old Terraform entries when this happened, they were commented out, preserving the historical record of how these hosts actually came to exist rather than presenting a cleaner story than what's true. Ansible still configures both today through the same tag-driven mechanism as every VM — only the initial provisioning step is now manual instead of automated. This is the kind of real-world messiness that a from-scratch design document never mentions but that every infrastructure actually accumulates over time — worth showing, not hiding.

---

Related: docs/secrets-and-pki.md, docs/security-monitoring.md, docs/disaster-recovery.md