public/docs/iac-pipeline.md

# 🏗️ Infrastructure as Code

> [!NOTE]
> Packer, Terraform, and Ansible — three tools, three responsibilities, and one thread running through all of them: a host's tag list, generated once by Terraform and consumed by every Ansible stage after it. This is the foundation everything else in this repo actually runs on top of.

## The pipeline

![Packer builds the base image, Terraform provisions the VM and generates host_tags.yml, which Ansible consumes to configure and harden the host, re-applying its compliance stage on every run](../assets/iac-pipeline-tools-flow.svg)

## Why three separate tools instead of one

A single tool covering image-build-through-configuration was a real option — several products exist specifically for that — and it was rejected because the three jobs have genuinely different rhythms: an image gets rebuilt rarely (a base OS refresh), a VM gets provisioned once, and configuration gets re-applied continuously (the compliance stage below runs on every pass, not just at creation). Collapsing all three into one tool's model would mean forcing the rarely-changing and the continuously-changing onto the same execution model. Keeping them separate, connected by one deliberate handoff (the tag file below) rather than a tightly-coupled integration, means each tool does the one job it's actually good at.

## Packer: hardening baked in at install time, not bolted on after

A single base image (`rocky10-cis`) is built from a minimal Rocky Linux 10 install with CIS-oriented hardening baked into the kickstart itself — disk partitioning (mount options like `nodev`/`nosuid`/`noexec` on specific partitions) and package selection decided at install time, not patched in afterward by Ansible. This was a deliberate split with the compliance work in [`docs/security-monitoring.md`](security-monitoring.md): some controls are genuinely easier and more reliable to get right once, at image-build time, than to retrofit repeatedly with a configuration tool after the fact — partition layout in particular can't be changed after boot without real risk, so it has to be correct in the image, not corrected later. This is the real kickstart partitioning, not a paraphrase of the intent:

```
$ grep -A8 'Logical volumes' http/ks.cfg
logvol /tmp              --fstype=xfs --size=2048  --name=lv_tmp     --vgname=rocky --fsoptions="defaults,nodev,nosuid"
logvol /home             --fstype=xfs --size=2048  --name=lv_home    --vgname=rocky --fsoptions="defaults,nodev"
logvol /var              --fstype=xfs --size=4096  --name=lv_var     --vgname=rocky --fsoptions="defaults,nodev"
logvol /var/tmp          --fstype=xfs --size=1024  --name=lv_vartmp  --vgname=rocky --fsoptions="defaults,nodev,nosuid,noexec"
logvol /var/log          --fstype=xfs --size=2048  --name=lv_varlog  --vgname=rocky --fsoptions="defaults,nodev,noexec,nosuid"
logvol /var/log/audit    --fstype=xfs --size=2048  --name=lv_audit   --vgname=rocky --fsoptions="defaults,nodev,noexec,nosuid"
logvol /                 --fstype=xfs --name=lv_root --vgname=rocky --grow --size=512 --fsoptions="defaults"
```

`/var/tmp` and `/var/log` both get the full `nodev,nosuid,noexec` treatment; `/home` and `/var` only get `nodev` — a reader can see exactly which mount points earned the stricter posture and which didn't, rather than one blanket policy applied everywhere.

> [!TIP]
> Two things happen at image-build time that aren't part of the pipeline's "steady state" and are easy to miss reading only the diagram above. First, the kickstart deliberately leaves SSH in a **permissive** state during the build — `PermitRootLogin yes` and `PasswordAuthentication yes` — because the build environment isn't reachable from anywhere except the hypervisor doing the build, and that window closes for good once the compliance stage's `ssh_hardening` role runs on a provisioned host. Second, Packer's own post-install provisioners create both fleet identities right there in the image — a personal admin account, and the dedicated `ansible` service account used for all automation — and the admin account's setup script ends by writing a sudoers rule that explicitly **denies it permission to run `ansible`/`ansible-playbook` at all**. Automation only ever runs as the service account; that's not a convention someone has to remember, it's baked into every host from before it's even provisioned.

## Terraform: two resource types, tag generation as the real output

> [!TIP]
> Every VM is one entry in a single map — VM ID, target node, template, resources, network, disk, and its own tag list. Two supporting pieces do the real connective work: a wait step that blocks completion until the new VM actually answers SSH as the automation identity (so nothing downstream races a host that isn't up yet), and a template render that writes that VM's tag list directly into Ansible's own host-variable tree. **This tag file is the actual link between provisioning and configuration** — the tags Terraform assigns to a VM are the exact same tags every later Ansible stage (including the security hardening in [`docs/security-monitoring.md`](security-monitoring.md) and the observability rollout in [`docs/observability.md`](observability.md)) matches against. Get this file wrong and every downstream stage silently configures the wrong thing for that host — which is why it's generated, not hand-written.

Terraform manages both VMs and LXCs now — two separate resource types, one per guest kind, coexisting in one config with no shared state between them:

| | VMs | LXCs |
|---|---|---|
| Provider | `telmate/proxmox` | `bpg/proxmox` |
| Resource | `proxmox_vm_qemu` | `proxmox_virtual_environment_container` |
| Source image | the `rocky10-cis` Packer template — hardening, `sshd`, both fleet identities pre-baked | stock Rocky 10 template — root-only, no `sshd` |
| Ansible group | `rocky_vm` | `rocky_lxc` |

The LXC path is genuinely newer and mirrors the VM flow step-for-step, but has to close a gap the VM path never has: the stock container template has no `sshd` and no fleet users at all, since it never went through Packer. A `pct exec` bootstrap step runs first — installing `sshd` and creating both fleet identities with their keys, the same baseline Packer bakes into the VM image at build time instead of at provision time — before the same wait-for-SSH and tag-file-render steps every VM already gets. This is the real resource definition, not a paraphrase:

```
$ head -20 containers.tf
resource "proxmox_virtual_environment_container" "ct" {
  provider = proxmox-bpg
  for_each = var.containers

  node_name     = each.value.target_node
  vm_id         = each.value.vmid
  unprivileged  = each.value.unprivileged
  start_on_boot = each.value.onboot
  started       = true

  tags = each.value.tags

  cpu {
    cores = each.value.cores
  }

  memory {
    dedicated = each.value.memory
    swap      = each.value.swap
  }
```

An empty `var.containers` makes this file completely inert — nothing created, no effect — the same "opt-in, not a parallel system that has to be kept in sync" pattern as every other tag-driven piece of this pipeline.

This is the actual template that renders into every host's tag file — the literal source, not a description of its fields:

```
$ cat host_tags.yml.tpl
vmid: ${vmid}
vm_name: ${name}
target_node: ${target_node}
template: ${template}
cores: ${cores}
sockets: ${sockets}
memory: ${memory}
balloon: ${balloon}
bridge: ${bridge}
macaddr: ${macaddr}
disk_size: ${disk_size}GB
disk_storage: ${disk_storage}
ip: ${ip}
vlan: ${vlan}
vm_tags:
%{ for tag in tags ~}
  - ${tag}
%{ endfor ~}
```

Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes. And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, `ip`/`macaddr` redacted:

```
$ cat host_vars/technitium/host_tags.yml
vmid: 118
vm_name: technitium
target_node: hyper2
template: rocky10-cis
cores: 2
sockets: 1
memory: 3072
balloon: 1536
bridge: vmbr0
macaddr: [technitium's MAC]
disk_size: 50GB
disk_storage: local
ip: [technitium's IP]
vlan: 0
vm_tags:
  - packages
  - hostname_network
  - root_password
  - disk_resize
  - base_users
  - firewall_ssh
  - zsh_custom
  - vault_agent
  - technitium
  - ssh_hardening
  - password_policy
  - banners
  - file_permissions
  - service_disable
  - kernel_hardening
  - filesystem_hardening
  - chrony_hardening
  - auditd_hardening
  - observability_agent
  - mqtt_client
  - wazuh_agent_tuning
```

21 tags on this one host alone, spanning both the initialization roles (`packages`, `firewall_ssh`, …) and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads.

## Ansible: tag-driven, re-applied continuously

![The per-host tag list from Terraform drives three independent Ansible stages: initialization (runs once, at creation), provisioning (every run, one role today on one host), and compliance (re-applied every run), which fans out into 9 independently-tagged hardening sub-roles](../assets/iac-pipeline-ansible-stages.svg)

There are genuinely three stages, not two — the third one is easy to miss because it's currently almost empty. `provisioning.yml` exists for roles that fit neither "once at init" nor "every compliance run," and as of this writing exactly one role is tagged for it:

```
$ python3 -c "
import yaml
d = yaml.safe_load(open('roles_metadata.yml'))['roles_metadata']
print('provisioning:', [k for k,v in d.items() if v.get('provisioning') is True])
print('compliance:', sorted(k for k,v in d.items() if v.get('compliance') is True))
"
provisioning: ['clamav']
compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom']
```

`clamav` is the only provisioning-stage role, and only one host carries it in its `vm_tags`. An under-used pipeline stage isn't the same as an unused one — it exists on purpose for exactly this shape of role, not as leftover scaffolding.

Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 15 total compliance-stage roles above — the other 6 handle observability, notifications, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.

## The honest exception: not everything fits the clean model

> [!WARNING]
> Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into LXC containers outside Terraform's management — a resource-type mismatch that Terraform's VM-only tooling genuinely can't represent (LXCs aren't `proxmox_vm_qemu` resources). Rather than deleting their old Terraform entries when this happened, they were commented out, preserving the historical record of how these hosts actually came to exist rather than presenting a cleaner story than what's true. Ansible still configures both today through the same tag-driven mechanism as every VM — only the initial provisioning step is now manual instead of automated. This is the kind of real-world messiness that a from-scratch design document never mentions but that every infrastructure actually accumulates over time — worth showing, not hiding.
>
> A third host breaks the model in a genuinely different way — not a variation on the same story, a different one entirely. Proxmox Backup Server runs as the vendor's own official Debian-based appliance image, deployed by hand and never a Terraform resource at all — there's no commented-out entry to preserve, because there was never a Terraform-managed version of it to convert away from. It still ended up on the tag-driven Ansible side of the pipeline: wired into its own inventory group and given a hand-written tag list after the fact, since compliance only ever checks whether a tag is present, not how a host arrived in the inventory. It's also the fleet's only non-Rocky host, which surfaced a real gap in one of the 9 hardening sub-roles above: `password_policy` configures PAM lockout behavior, and the ordering it assumes is Rocky's own — safe on every other host, but wrong on this one until it was fixed at the role level and explicitly re-validated.
>
> None of these three exceptions were retroactively fixed once the LXC-provider gap closed — the point of preserving history is that it stays history, not that it gets quietly cleaned up the moment it becomes possible to. But the gap itself is real and worth naming honestly too: for any *new* LXC going forward, this entire honest-exception story no longer applies. The fleet's [zero-trust mesh overlay](netbird-zero-trust.md) is the first host actually built through the LXC-native path above, start to finish, with no manual step anywhere in its provisioning.

---
Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/disaster-recovery.md`](disaster-recovery.md), [`docs/netbird-zero-trust.md`](netbird-zero-trust.md)