docs: deepen cron-scheduler, content-pipeline, iac-pipeline, and disaster-recovery

Completes the full depth pass across all 16 docs - real tech names,
why-this-vs-the-alternative reasoning, heavy cross-linking between
every doc, ranked/specific tradeoffs instead of generic hedges.
This commit is contained in:
thesabear committed 2026-08-22 08:13:49 +03:00
1 parent 1382a63dae
commit febbf2e44c
4 files changed
+51 -44

No files matched your search

+10 -10
View File
@@ -1,7 +1,7 @@
# 📝 Content Pipeline # 📝 Content Pipeline
> [!WARNING] > [!WARNING]
> Still under active development — this covers the current shape, not a finished architecture. Included deliberately rather than only showcasing finished work. > Still under active development — this covers the current shape, not a finished architecture. Included deliberately: a portfolio that only shows finished, polished systems doesn't show how something gets built, only how it ends up. This one's mid-build.
## The intended shape ## The intended shape
@@ -12,30 +12,30 @@ flowchart LR
Review --> Approved["Approved"] Review --> Approved["Approved"]
Approved --> Published["Published"] Approved --> Published["Published"]
Approved -.glue via automation tool.-> Storage["Draft/asset staging"] Approved -.glue via n8n.-> Storage["Nextcloud<br/>draft/asset staging"]
Approved -.glue via automation tool.-> Blog["Published blog"] Approved -.glue via n8n.-> Blog["Ghost<br/>published blog"]
Approved -.planned, not yet wired.-> Social["Social platforms"] Approved -.planned, not yet wired.-> Social["Social platforms<br/>via Postiz"]
``` ```
A structured content tracker moves ideas through a status pipeline, with a workflow-automation tool gluing the stages together: approved content flows through to a file-storage staging area and out to a public blog. A separate, still-incomplete leg is meant to fan approved posts out to social platforms automatically. A NocoDB table tracks content ideas through a status pipeline, with n8n gluing the stages together — the same n8n instance mentioned in [`docs/orchestration.md`](orchestration.md)'s ecosystem, here doing content workflow instead of infrastructure maintenance. Approved content flows through to Nextcloud (draft/asset staging) and out to Ghost, a self-hosted blog. A separate, still-incomplete leg is meant to fan approved posts out to social platforms automatically via Postiz.
## A deliberate constraint, not a limitation stumbled into ## A deliberate constraint, not a limitation stumbled into
> [!TIP] > [!TIP]
> The social-publishing tool is **single-origin by design** — its frontend/backend URLs are baked to one specific public hostname at configuration time, so it deliberately has no presence in either internal DNS zone described in [`docs/dns-naming.md`](dns-naming.md); any other hostname hitting it fails CORS. This was a conscious choice, not a workaround: the tool is meant to be reached exactly one way. > Postiz is **single-origin by design** — its frontend/backend URLs are baked to one specific public hostname at configuration time, so it deliberately has no presence in either internal DNS zone described in [`docs/dns-naming.md`](dns-naming.md); any other hostname hitting it fails CORS. This mirrors the exact reasoning in [`docs/dns-naming.md`](dns-naming.md) for why genuinely public apps skip the internal zone model entirely: a social-publishing tool has no legitimate internal-only caller, so giving it internal DNS presence would just be attack surface with no corresponding benefit. This was a conscious choice, verified against Postiz's own architecture, not a workaround discovered after something broke.
## A real surprise in scale ## A real surprise in scale
> [!IMPORTANT] > [!IMPORTANT]
> The social-publishing tool's real deployment footprint is 8 containers, not the app+database+cache most people would guess — it bundles its own workflow engine (mandatory for all of its background jobs, including scheduled posting itself) plus that engine's own database and search-based visibility store. A useful reminder that a tool's advertised simplicity and its actual operational footprint can be very different things. > Postiz's real deployment footprint is 8 containers, not the app+database+cache most people would guess from its marketing page — it bundles its own Temporal workflow engine (mandatory as of Postiz v2.12.0 for all background jobs, including scheduled posting itself, not an optional add-on) plus that engine's own Postgres database and Elasticsearch-based visibility store. Provisioning this stack meant sizing for 8 containers' worth of resource overhead, not the 3 the app's own documentation implies at a glance — a useful reminder, worth stating plainly, that a tool's advertised simplicity and its actual operational footprint on the host running [`docs/iac-pipeline.md`](iac-pipeline.md)'s hardened images can be very different things, and that finding out matters before capacity planning, not after.
## What's actually built vs. what's still wiring up ## What's actually built vs. what's still wiring up
| Piece | Status | | Piece | Status |
|---|---| |---|---|
| Content tracker → automation → draft storage → blog | Most built-out, working end to end | | NocoDB → n8n → Nextcloud/Ghost | Most built-out, working end to end |
| AI-assisted drafting step | Still being wired up | | AI-assisted drafting step | Still being wired up |
| Social-publishing fan-out | Still being wired up | | Social-publishing fan-out via Postiz | Still being wired up — see the notification-routing gap in [`docs/notifications.md`](notifications.md), which also hasn't caught up to this app yet |
--- ---
Related stacks: [`stacks/productivity/postiz/`](../stacks/productivity/postiz/), [`stacks/productivity/nocodb-content/`](../stacks/productivity/nocodb-content/), [`stacks/productivity/ghost/`](../stacks/productivity/ghost/) Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/orchestration.md`](orchestration.md), [`docs/notifications.md`](notifications.md) · Stacks: [`stacks/productivity/postiz/`](../stacks/productivity/postiz/), [`stacks/productivity/nocodb-content/`](../stacks/productivity/nocodb-content/), [`stacks/productivity/ghost/`](../stacks/productivity/ghost/)
+10 -8
View File
@@ -1,32 +1,34 @@
# ⏱️ Cron Scheduler # ⏱️ Cron Scheduler
> [!NOTE] > [!NOTE]
> A fleet-wide inventory of every known scheduled job — orchestration flows, system timers, automation-managed cron, app-internal schedules — visualized as a heatmap so a new job's time can be picked without colliding with existing load. > A NocoDB-backed inventory of every known scheduled job in the fleet — Kestra flows, systemd timers, Ansible-managed cron, app-internal schedules — visualized as a Dashy heatmap so a new job's time can be picked without colliding with existing load. Small in scope compared to the rest of this repo, but it's a real answer to a problem every one of the other systems here creates: once you have a dozen-plus independently-scheduled jobs across [`docs/orchestration.md`](orchestration.md), [`docs/registry-and-build.md`](registry-and-build.md), and [`docs/disaster-recovery.md`](disaster-recovery.md), "when does anything run" stops being something you can hold in your head.
## The heatmap ## The heatmap
```mermaid ```mermaid
flowchart LR flowchart LR
Sources["Job sources<br/>flows · timers · cron · app schedules"] --> Table["Structured inventory<br/>name, host, category,<br/>schedule, duration, resource tags"] Sources["Job sources<br/>Kestra flows · systemd timers ·<br/>Ansible cron · app schedules"] --> Table["NocoDB table<br/>name, host, category,<br/>schedule, duration, resource tags"]
Table --> Weight["Weighted load calculation"] Table --> Weight["Weighted load calculation"]
Weight --> Grid["7×24 grid<br/>day × hour"] Weight --> Grid["7×24 grid<br/>day × hour"]
Grid --> Bands["4 color bands<br/>quieter → busier"] Grid --> Bands["4 color bands<br/>quieter → busier"]
``` ```
Each cell's color reflects a **computed load weight**, not a raw job count: a base weight per job, plus a duration-based weight, plus resource-intensity weight (disk/CPU/network, summed if a job carries multiple tags), plus an extra weight for anything tagged as a backup job. Jobs whose time can't be resolved to a specific day and hour are excluded from the grid and listed separately rather than forced into a misleading cell. Each cell's color reflects a **computed load weight**, not a raw job count: a base weight per job, plus a duration-based weight, plus resource-intensity weight (disk/CPU/network, summed if a job carries multiple tags), plus an extra weight for anything tagged as a backup job — backup jobs get weighted higher deliberately, since disk I/O contention from an overlapping backup is a worse collision than two lightweight jobs running at the same hour. Jobs whose time can't be resolved to a specific day and hour are excluded from the grid and listed separately rather than forced into a misleading cell.
> [!TIP] > [!TIP]
> Clicking a populated cell expands the full job list for that slot — name, host, category, and whether that job's schedule can actually be moved or is fixed by whatever app owns it. > Clicking a populated cell expands the full job list for that slot — name, host, category, and whether that job's schedule can actually be moved or is fixed by whatever app owns it. That last distinction matters in practice: there's no point suggesting a new job avoid a time slot occupied by something that can't be rescheduled anyway versus one that easily could be.
## Honest about its own limitation ## Why a manual inventory instead of an auto-discovered one
The harder, more "correct" version of this tool would auto-discover every scheduled job by querying Kestra's API, parsing every host's crontab and systemd timers, and re-syncing continuously. That was scoped down deliberately: auto-discovery across three genuinely different scheduling mechanisms (Kestra's API, systemd's timer units, cron's own format) is real engineering effort for a tool whose entire job is picking a good time slot for the next new job — a task that happens occasionally, not continuously. A manually-maintained table, kept honest by being checked at the one moment it actually matters (adding a new job), was judged the better return on effort than building and maintaining three different scanners for a problem that doesn't need real-time accuracy.
> [!WARNING] > [!WARNING]
> **This is a manually-maintained inventory, not an auto-refreshed one.** The schema has a "last synced" column, but nothing currently populates it — no automated process re-scans the fleet to keep this current. Entries get added or edited by hand as jobs are created or changed. Documented as a real limitation rather than presented as more automated than it actually is. > **This is a manually-maintained inventory, and that's stated plainly, not glossed over.** The schema has a "last synced" column, but nothing currently populates it — no automated process re-scans the fleet to keep this current. Entries get added or edited by hand as jobs are created or changed, which means the tool is only as accurate as the discipline of updating it — a real limitation of the scoping decision above, not hidden behind a column that implies more automation than actually exists.
## A real, unresolved bug this system surfaced ## A real, unresolved bug this system surfaced
> [!IMPORTANT] > [!IMPORTANT]
> One backup job's prune step targets a datastore name that doesn't actually exist — not a scheduling collision, an orphaned config that's been silently failing. Two fixes are on the table (drop the dead job and let a different retention mechanism be authoritative, or repoint it at the real datastore name and let its richer retention rules take over) — deliberately not resolved yet, tracked openly rather than silently patched over with a guess. > Building this inventory surfaced something that had been silently broken: one backup job's prune step targets a datastore name that doesn't actually exist on the backup server described in [`docs/disaster-recovery.md`](disaster-recovery.md) — not a scheduling collision, an orphaned config that's been silently failing every run. Two fixes are on the table (drop the dead job and let a different retention mechanism be authoritative, or repoint it at the real datastore name and let its own richer retention rules take over) — deliberately left unresolved here rather than silently patched over with a guess, since the right fix depends on a retention-policy decision that's genuinely still open, not a technical one that just needs picking.
--- ---
Related stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/nocodb/`](../stacks/platform/nocodb/) Related: [`docs/orchestration.md`](orchestration.md), [`docs/disaster-recovery.md`](disaster-recovery.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/dashy/`](../stacks/platform/dashy/), [`stacks/platform/nocodb/`](../stacks/platform/nocodb/)
+19 -15
View File
@@ -1,42 +1,46 @@
# 🧯 Disaster Recovery & Upgrades # 🧯 Disaster Recovery & Upgrades
> [!NOTE] > [!NOTE]
> Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose. > Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose. Both halves of this doc are about the same underlying discipline: assuming things will eventually fail, and designing so one failure doesn't cascade into a second.
## Three independent recovery layers ## Three independent recovery layers
```mermaid ```mermaid
flowchart TD flowchart TD
Loss["Something is lost"] --> Type{"What kind?"} Loss["Something is lost"] --> Type{"What kind?"}
Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same pipeline<br/>that built it originally"] Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same Packer→Terraform→Ansible<br/>pipeline that built it originally"]
Type -->|"VM/LXC-level data"| PBS["Layer 2: VM/LXC backups<br/>full-host snapshots"] Type -->|"VM/LXC-level data"| PBS["Layer 2: Proxmox Backup Server<br/>full-host snapshots"]
Type -->|"App data on the main host"| Kopia["Layer 3: Volume-level backups<br/>per-stack, isolated repository"] Type -->|"App data on the main host"| Kopia["Layer 3: Kopia, volume-level<br/>per-stack, isolated repository"]
``` ```
## Why three layers instead of one comprehensive backup system
A single backup tool covering everything was the simpler design, and it was rejected because "a host" and "a database's data" are fundamentally different kinds of loss with different recovery needs. Rebuilding a lost host from [`docs/iac-pipeline.md`](iac-pipeline.md)'s pipeline is fast and doesn't need a backup at all — the *code* is the backup. But that same pipeline can't recover yesterday's database contents, which is what Layer 2 and 3 exist for. Trying to make one tool do both jobs well tends to produce a tool that's mediocre at each — full-host imaging is slow and coarse-grained for restoring one app's data, and application-level backup can't reconstruct a host's OS and configuration from scratch efficiently. Three narrow, purpose-fit layers beat one general-purpose one here.
| Layer | Covers | Explicitly does not cover | | Layer | Covers | Explicitly does not cover |
|---|---|---| |---|---|---|
| Infrastructure as Code | A host lost entirely — rebuilt from the same pipeline that built it, against its existing config | Secrets (recovered separately, not part of this layer) | | Infrastructure as Code | A host lost entirely — rebuilt from the same pipeline that built it ([`docs/iac-pipeline.md`](iac-pipeline.md)), against its existing tag-driven config | Secrets — recovered separately via Vault, see [`docs/secrets-and-pki.md`](secrets-and-pki.md), not part of this layer |
| VM/LXC-level backup | Full VM/LXC backups, including the secrets platform's own seal state and data | One host's Docker volume data, a large media storage pool, personal data | | Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data |
| Volume-level backup | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host | Anything outside that one host | | Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host |
> [!IMPORTANT] > [!IMPORTANT]
> The volume-level backup layer runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. > The volume-level backup layer (Kopia) runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. A backup tool with write access to the data it protects, or one whose own storage lives on the same disk as that data, defeats a meaningful fraction of its own purpose the moment that disk fails.
> [!WARNING] > [!WARNING]
> **A known, honestly-documented gap**: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them. > **A known, honestly-documented gap**: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them. A three-layer recovery model is only as good as its actual coverage, and claiming full coverage when a real gap exists would undermine the credibility of the two layers that do work correctly.
## Upgrades: two systems, deliberately not coupled ## Upgrades: two systems, deliberately not coupled
```mermaid ```mermaid
flowchart LR flowchart LR
subgraph OS["OS-level"] subgraph OS["OS-level (os_update role)"]
direction TB direction TB
T1["VMs — 3 at a time<br/>+ network-critical host check"] T1["VMs — 3 at a time<br/>+ network-critical host check"]
T2["Hypervisor nodes — 1 at a time<br/>aborts if anything's mid-migration"] T2["Proxmox nodes — 1 at a time<br/>aborts if anything's mid-migration"]
T3["Backup server — 1 at a time<br/>aborts if a backup job is running"] T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
end end
subgraph App["App-level"] subgraph App["App-level (container_update / app_update)"]
direction TB direction TB
Strategy["Per-stack update strategy<br/>independent, unrelated checks"] Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
end end
@@ -44,10 +48,10 @@ flowchart LR
OS -.completely separate concern.-> App OS -.completely separate concern.-> App
``` ```
Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again before moving on — not just that the reboot command succeeded. Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again via a real ping check before moving on — not just that the reboot command itself succeeded, which would miss a host that reboots but fails to come back up cleanly.
> [!TIP] > [!TIP]
> **Why app-level updates deliberately don't use batch/rolling deployment logic**: each host's update check is an independent, unrelated result — a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism would treat that non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures. > **Why app-level updates deliberately don't use batch/rolling deployment logic**: each host's update check in [`docs/registry-and-build.md`](registry-and-build.md) is an independent, unrelated result — a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism (the kind built into most orchestration tools by default) would treat that expected non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment in the traditional sense, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures that would have been genuinely difficult to diagnose — the symptom would look like "the update job stopped partway through" with no obvious error pointing at the actual cause.
--- ---
Related: [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md) Related: [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/orchestration.md`](orchestration.md), [`docs/registry-and-build.md`](registry-and-build.md), [`docs/cron-scheduler.md`](cron-scheduler.md)
+12 -11
View File
@@ -1,7 +1,7 @@
# 🏗️ Infrastructure as Code # 🏗️ Infrastructure as Code
> [!NOTE] > [!NOTE]
> Three tools, three responsibilities, one thread running through all of them: a host's tag list, generated once and consumed by every stage after it. > Packer, Terraform, and Ansible — three tools, three responsibilities, and one thread running through all of them: a host's tag list, generated once by Terraform and consumed by every Ansible stage after it. This is the foundation everything else in this repo actually runs on top of.
## The pipeline ## The pipeline
@@ -13,35 +13,36 @@ flowchart LR
Ansible -->|re-applied every run| Compliance["Compliance stage<br/>not just once at creation"] Ansible -->|re-applied every run| Compliance["Compliance stage<br/>not just once at creation"]
``` ```
## Why three separate tools instead of one
A single tool covering image-build-through-configuration was a real option — several products exist specifically for that — and it was rejected because the three jobs have genuinely different rhythms: an image gets rebuilt rarely (a base OS refresh), a VM gets provisioned once, and configuration gets re-applied continuously (the compliance stage below runs on every pass, not just at creation). Collapsing all three into one tool's model would mean forcing the rarely-changing and the continuously-changing onto the same execution model. Keeping them separate, connected by one deliberate handoff (the tag file below) rather than a tightly-coupled integration, means each tool does the one job it's actually good at.
## Packer: hardening baked in at install time, not bolted on after ## Packer: hardening baked in at install time, not bolted on after
A single base image is built from a minimal Rocky Linux install with CIS-oriented hardening baked into the kickstart itself — disk partitioning and package selection decided at install time, not patched in afterward. Some controls (like mount-option restrictions on specific partitions) are genuinely easier and more reliable to get right at image-build time than to retrofit with a configuration tool after the fact. A single base image (`rocky10-cis`) is built from a minimal Rocky Linux 10 install with CIS-oriented hardening baked into the kickstart itself — disk partitioning (mount options like `nodev`/`nosuid`/`noexec` on specific partitions) and package selection decided at install time, not patched in afterward by Ansible. This was a deliberate split with the compliance work in [`docs/security-monitoring.md`](security-monitoring.md): some controls are genuinely easier and more reliable to get right once, at image-build time, than to retrofit repeatedly with a configuration tool after the fact — partition layout in particular can't be changed after boot without real risk, so it has to be correct in the image, not corrected later.
## Terraform: one resource type, tag generation as the real output ## Terraform: one resource type, tag generation as the real output
> [!TIP] > [!TIP]
> Every VM is one entry in a single map — VM ID, target node, template, resources, network, disk, and its own tag list. Two supporting pieces do the real connective work: a wait step that blocks completion until the new VM actually answers SSH (so nothing downstream races a host that isn't up yet), and a template render that writes that VM's tag list directly into Ansible's own host-variable tree. **This is the actual link between provisioning and configuration** — the tags Terraform assigns to a VM are the exact same tags every later Ansible stage matches against. > Every VM is one entry in a single map — VM ID, target node, template, resources, network, disk, and its own tag list. Two supporting pieces do the real connective work: a wait step that blocks completion until the new VM actually answers SSH as the automation identity (so nothing downstream races a host that isn't up yet), and a template render that writes that VM's tag list directly into Ansible's own host-variable tree. **This tag file is the actual link between provisioning and configuration** — the tags Terraform assigns to a VM are the exact same tags every later Ansible stage (including the security hardening in [`docs/security-monitoring.md`](security-monitoring.md) and the observability rollout in [`docs/observability.md`](observability.md)) matches against. Get this file wrong and every downstream stage silently configures the wrong thing for that host — which is why it's generated, not hand-written.
Terraform only manages virtual machines, deliberately — containers are handled differently (below). Terraform only manages virtual machines, deliberately — containers are handled differently (below), a scope boundary that avoids fighting Terraform's own resource model for something it wasn't built to represent well.
## Ansible: tag-driven, re-applied continuously ## Ansible: tag-driven, re-applied continuously
```mermaid ```mermaid
flowchart TD flowchart TD
Tags["Per-host tag list"] --> Init["Initialization stage<br/>runs once, at creation"] Tags["Per-host tag list<br/>from Terraform"] --> Init["Initialization stage<br/>runs once, at creation"]
Tags --> Compliance["Compliance stage<br/>re-applied every run"] Tags --> Compliance["Compliance stage<br/>re-applied every run"]
Compliance --> Sub["9 independently-tagged<br/>hardening sub-roles"] Compliance --> Sub["9 independently-tagged<br/>hardening sub-roles"]
``` ```
Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles are active fleet-wide; none are disabled or pending. Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
> [!IMPORTANT]
> **Deliberate deviations from the hardening baseline, with real reasoning attached**: one kernel-level control is flipped from the stricter default on container-hosting machines specifically, because the container platform's own networking genuinely requires it. One SSH control stays permissive fleet-wide because real remote access depends on it. Both are documented as accepted, permanent trade-offs — not gaps waiting to be closed, and not silently ignored either.
## The honest exception: not everything fits the clean model ## The honest exception: not everything fits the clean model
> [!WARNING] > [!WARNING]
> Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into containers outside Terraform's management — a resource type mismatch that Terraform's VM-only tooling can't represent. Rather than deleting their old Terraform entries, they were commented out, preserving the historical record. Ansible still configures both today through the same tag-driven mechanism as everything else — only the initial provisioning step is now manual instead of automated. > Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into LXC containers outside Terraform's management — a resource-type mismatch that Terraform's VM-only tooling genuinely can't represent (LXCs aren't `proxmox_vm_qemu` resources). Rather than deleting their old Terraform entries when this happened, they were commented out, preserving the historical record of how these hosts actually came to exist rather than presenting a cleaner story than what's true. Ansible still configures both today through the same tag-driven mechanism as every VM — only the initial provisioning step is now manual instead of automated. This is the kind of real-world messiness that a from-scratch design document never mentions but that every infrastructure actually accumulates over time — worth showing, not hiding.
--- ---
Related: [`docs/disaster-recovery.md`](disaster-recovery.md) Related: [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/disaster-recovery.md`](disaster-recovery.md)