docs: deepen cron-scheduler, content-pipeline, iac-pipeline, and disaster-recovery

Completes the full depth pass across all 16 docs - real tech names,
why-this-vs-the-alternative reasoning, heavy cross-linking between
every doc, ranked/specific tradeoffs instead of generic hedges.
This commit is contained in:
thesabear committed 2026-08-22 08:13:49 +03:00
1 parent 1382a63dae
commit febbf2e44c
4 files changed
+51 -44

No files matched your search

+12 -11
View File
@@ -1,7 +1,7 @@
# 🏗️ Infrastructure as Code
> [!NOTE]
> Three tools, three responsibilities, one thread running through all of them: a host's tag list, generated once and consumed by every stage after it.
> Packer, Terraform, and Ansible — three tools, three responsibilities, and one thread running through all of them: a host's tag list, generated once by Terraform and consumed by every Ansible stage after it. This is the foundation everything else in this repo actually runs on top of.
## The pipeline
@@ -13,35 +13,36 @@ flowchart LR
Ansible -->|re-applied every run| Compliance["Compliance stage<br/>not just once at creation"]
```
## Why three separate tools instead of one
A single tool covering image-build-through-configuration was a real option — several products exist specifically for that — and it was rejected because the three jobs have genuinely different rhythms: an image gets rebuilt rarely (a base OS refresh), a VM gets provisioned once, and configuration gets re-applied continuously (the compliance stage below runs on every pass, not just at creation). Collapsing all three into one tool's model would mean forcing the rarely-changing and the continuously-changing onto the same execution model. Keeping them separate, connected by one deliberate handoff (the tag file below) rather than a tightly-coupled integration, means each tool does the one job it's actually good at.
## Packer: hardening baked in at install time, not bolted on after
A single base image is built from a minimal Rocky Linux install with CIS-oriented hardening baked into the kickstart itself — disk partitioning and package selection decided at install time, not patched in afterward. Some controls (like mount-option restrictions on specific partitions) are genuinely easier and more reliable to get right at image-build time than to retrofit with a configuration tool after the fact.
A single base image (`rocky10-cis`) is built from a minimal Rocky Linux 10 install with CIS-oriented hardening baked into the kickstart itself — disk partitioning (mount options like `nodev`/`nosuid`/`noexec` on specific partitions) and package selection decided at install time, not patched in afterward by Ansible. This was a deliberate split with the compliance work in [`docs/security-monitoring.md`](security-monitoring.md): some controls are genuinely easier and more reliable to get right once, at image-build time, than to retrofit repeatedly with a configuration tool after the fact — partition layout in particular can't be changed after boot without real risk, so it has to be correct in the image, not corrected later.
## Terraform: one resource type, tag generation as the real output
> [!TIP]
> Every VM is one entry in a single map — VM ID, target node, template, resources, network, disk, and its own tag list. Two supporting pieces do the real connective work: a wait step that blocks completion until the new VM actually answers SSH (so nothing downstream races a host that isn't up yet), and a template render that writes that VM's tag list directly into Ansible's own host-variable tree. **This is the actual link between provisioning and configuration** — the tags Terraform assigns to a VM are the exact same tags every later Ansible stage matches against.
> Every VM is one entry in a single map — VM ID, target node, template, resources, network, disk, and its own tag list. Two supporting pieces do the real connective work: a wait step that blocks completion until the new VM actually answers SSH as the automation identity (so nothing downstream races a host that isn't up yet), and a template render that writes that VM's tag list directly into Ansible's own host-variable tree. **This tag file is the actual link between provisioning and configuration** — the tags Terraform assigns to a VM are the exact same tags every later Ansible stage (including the security hardening in [`docs/security-monitoring.md`](security-monitoring.md) and the observability rollout in [`docs/observability.md`](observability.md)) matches against. Get this file wrong and every downstream stage silently configures the wrong thing for that host — which is why it's generated, not hand-written.
Terraform only manages virtual machines, deliberately — containers are handled differently (below).
Terraform only manages virtual machines, deliberately — containers are handled differently (below), a scope boundary that avoids fighting Terraform's own resource model for something it wasn't built to represent well.
## Ansible: tag-driven, re-applied continuously
```mermaid
flowchart TD
Tags["Per-host tag list"] --> Init["Initialization stage<br/>runs once, at creation"]
Tags["Per-host tag list<br/>from Terraform"] --> Init["Initialization stage<br/>runs once, at creation"]
Tags --> Compliance["Compliance stage<br/>re-applied every run"]
Compliance --> Sub["9 independently-tagged<br/>hardening sub-roles"]
```
Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles are active fleet-wide; none are disabled or pending.
> [!IMPORTANT]
> **Deliberate deviations from the hardening baseline, with real reasoning attached**: one kernel-level control is flipped from the stricter default on container-hosting machines specifically, because the container platform's own networking genuinely requires it. One SSH control stays permissive fleet-wide because real remote access depends on it. Both are documented as accepted, permanent trade-offs — not gaps waiting to be closed, and not silently ignored either.
Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
## The honest exception: not everything fits the clean model
> [!WARNING]
> Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into containers outside Terraform's management — a resource type mismatch that Terraform's VM-only tooling can't represent. Rather than deleting their old Terraform entries, they were commented out, preserving the historical record. Ansible still configures both today through the same tag-driven mechanism as everything else — only the initial provisioning step is now manual instead of automated.
> Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into LXC containers outside Terraform's management — a resource-type mismatch that Terraform's VM-only tooling genuinely can't represent (LXCs aren't `proxmox_vm_qemu` resources). Rather than deleting their old Terraform entries when this happened, they were commented out, preserving the historical record of how these hosts actually came to exist rather than presenting a cleaner story than what's true. Ansible still configures both today through the same tag-driven mechanism as every VM — only the initial provisioning step is now manual instead of automated. This is the kind of real-world messiness that a from-scratch design document never mentions but that every infrastructure actually accumulates over time — worth showing, not hiding.
---
Related: [`docs/disaster-recovery.md`](disaster-recovery.md)
Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/disaster-recovery.md`](disaster-recovery.md)