docs: add architecture, IaC pipeline, and disaster recovery write-ups

Completes the docs/ set - all 16 systems now have a full deep-dive
page in the diagram+callout+table style. Architecture overview covers
the 4 trust boundaries and 2 traffic planes; IaC pipeline is a fresh
Packer->Terraform->Ansible writeup replacing the stale old-content/
snapshot; disaster recovery covers the 3 independent backup layers
and the deliberately decoupled OS/app upgrade systems.

docs/architecture-overview.md

@@ -0,0 +1,82 @@
+# 📐 Architecture Overview
+
+> [!NOTE]
+> The stable, directional laws governing how traffic moves through the fleet — re-verified against live infrastructure, not assumed from older documentation. Two real corrections were caught and fixed the last time this was checked, which is exactly the point of re-verifying rather than trusting a prior write-up forever.
+
+## Dependency graph
+
+```mermaid
+flowchart TD
+    Internet(["Internet"]) --> Edge["Edge Firewall<br/>NAT · public reverse proxy · NTP · DHCP"]
+    Edge --> LAN["LAN"]
+
+    LAN --> DNS["Internal DNS"]
+    LAN --> Vault["Vault"]
+    DNS --> Vault
+
+    Vault --> Apps["App Host<br/>~40 stacks"]
+    LAN --> FileServer["File Server"]
+    DNS --> FileServer
+    Vault --> Automation["Automation Host"]
+    LAN --> Automation
+
+    Apps --> InternalProxy["Internal reverse-proxy zone"]
+    Apps --> SSO["SSO"]
+
+    Automation --> Gitea["Git hosting<br/>internal + public mirror"]
+
+    Vault --> EdgeProxy["User-facing reverse proxy<br/>mTLS"]
+    DNS --> EdgeProxy
+    Apps -.proxied by.-> EdgeProxy
+    SSO -.auth gate via.-> EdgeProxy
+
+    Apps -.reports to.-> Monitoring["Monitoring"]
+    Automation -.reports to.-> Monitoring
+    FileServer -.reports to.-> Monitoring
+    EdgeProxy -.reports to.-> Monitoring
+    Vault -.reports to.-> Monitoring
+    DNS -.reports to.-> Monitoring
+
+    Apps -.reports to.-> Wazuh["Security Monitoring"]
+    Automation -.reports to.-> Wazuh
+    FileServer -.reports to.-> Wazuh
+    EdgeProxy -.reports to.-> Wazuh
+    Vault -.reports to.-> Wazuh
+    DNS -.reports to.-> Wazuh
+```
+
+## Four trust boundaries, not two
+
+```mermaid
+flowchart LR
+    WAN["WAN"] -->|public application boundary| EdgeFW["Edge Firewall / public proxy"]
+    Browser["LAN browsers"] -->|mTLS + SSO| UserProxy["User-facing proxy"]
+    Service["App-to-app"] -->|IP-allowlist + SSO| InternalProxy["Internal proxy"]
+    Infra["LAN infra-to-infra"] -->|explicit per-flow rules| HostFW["Host firewall rules"]
+```
+
+Backend services trust proxy headers, not raw client IP, on every path.
+
+## Two structurally independent traffic planes
+
+| Plane | Path |
+|---|---|
+| **Ingress — public** | Client → edge firewall/proxy → backend |
+| **Ingress — LAN-facing** | Client → user-facing proxy (mTLS) → backend |
+| **East-west — app-to-app** | Service → internal proxy zone → service |
+| **East-west — infra-to-infra** | Service → firewall rule → service, direct, no proxy |
+
+> [!IMPORTANT]
+> **Observability sits outside every request path on purpose.** Logs and metrics push asynchronously to monitoring infrastructure; nothing in that pipeline is inline with application traffic, and application services don't depend on monitoring availability. If monitoring goes down, collection stops — nothing else does.
+
+## Caught by re-verification, not assumed
+
+> [!WARNING]
+> The last full re-check against live infrastructure found two real corrections to what was previously documented: the public reverse-proxy layer had actually moved from running as its own dedicated VM to running as a package directly on the edge firewall — and an older "flat mesh network, no mutual TLS" model for internal service traffic had been fully replaced by the two-zone split described in [`docs/dns-naming.md`](dns-naming.md), without the architecture documentation being updated to match at the time. Both were caught by running real commands against the live fleet, not by re-reading the previous version and assuming it still held.
+
+## What this map deliberately excludes
+
+Not a firewall rule listing, not a port inventory, not a service configuration reference — those live in their own docs ([`docs/firewall-and-network.md`](firewall-and-network.md) and elsewhere). This is strictly the stable, directional shape of how trust and traffic actually move.
+
+---
+Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md)

docs/disaster-recovery.md

@@ -0,0 +1,53 @@
+# 🧯 Disaster Recovery & Upgrades
+
+> [!NOTE]
+> Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose.
+
+## Three independent recovery layers
+
+```mermaid
+flowchart TD
+    Loss["Something is lost"] --> Type{"What kind?"}
+    Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code<br/>rebuilt via the same pipeline<br/>that built it originally"]
+    Type -->|"VM/LXC-level data"| PBS["Layer 2: VM/LXC backups<br/>full-host snapshots"]
+    Type -->|"App data on the main host"| Kopia["Layer 3: Volume-level backups<br/>per-stack, isolated repository"]
+```
+
+| Layer | Covers | Explicitly does not cover |
+|---|---|---|
+| Infrastructure as Code | A host lost entirely — rebuilt from the same pipeline that built it, against its existing config | Secrets (recovered separately, not part of this layer) |
+| VM/LXC-level backup | Full VM/LXC backups, including the secrets platform's own seal state and data | One host's Docker volume data, a large media storage pool, personal data |
+| Volume-level backup | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host | Anything outside that one host |
+
+> [!IMPORTANT]
+> The volume-level backup layer runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting — so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up.
+
+> [!WARNING]
+> **A known, honestly-documented gap**: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe — explicitly flagged as unprotected until a decision is made to cover them.
+
+## Upgrades: two systems, deliberately not coupled
+
+```mermaid
+flowchart LR
+    subgraph OS["OS-level"]
+        direction TB
+        T1["VMs — 3 at a time<br/>+ network-critical host check"]
+        T2["Hypervisor nodes — 1 at a time<br/>aborts if anything's mid-migration"]
+        T3["Backup server — 1 at a time<br/>aborts if a backup job is running"]
+    end
+
+    subgraph App["App-level"]
+        direction TB
+        Strategy["Per-stack update strategy<br/>independent, unrelated checks"]
+    end
+
+    OS -.completely separate concern.-> App
+```
+
+Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again before moving on — not just that the reboot command succeeded.
+
+> [!TIP]
+> **Why app-level updates deliberately don't use batch/rolling deployment logic**: each host's update check is an independent, unrelated result — a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism would treat that non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures.
+
+---
+Related: [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md)

docs/iac-pipeline.md

@@ -0,0 +1,47 @@
+# 🏗️ Infrastructure as Code
+
+> [!NOTE]
+> Three tools, three responsibilities, one thread running through all of them: a host's tag list, generated once and consumed by every stage after it.
+
+## The pipeline
+
+```mermaid
+flowchart LR
+    Packer["Packer<br/>builds the base image"] --> Terraform["Terraform<br/>provisions the VM"]
+    Terraform -->|generates| Tags["host_tags.yml<br/>the tag list"]
+    Tags --> Ansible["Ansible<br/>configures & hardens"]
+    Ansible -->|re-applied every run| Compliance["Compliance stage<br/>not just once at creation"]
+```
+
+## Packer: hardening baked in at install time, not bolted on after
+
+A single base image is built from a minimal Rocky Linux install with CIS-oriented hardening baked into the kickstart itself — disk partitioning and package selection decided at install time, not patched in afterward. Some controls (like mount-option restrictions on specific partitions) are genuinely easier and more reliable to get right at image-build time than to retrofit with a configuration tool after the fact.
+
+## Terraform: one resource type, tag generation as the real output
+
+> [!TIP]
+> Every VM is one entry in a single map — VM ID, target node, template, resources, network, disk, and its own tag list. Two supporting pieces do the real connective work: a wait step that blocks completion until the new VM actually answers SSH (so nothing downstream races a host that isn't up yet), and a template render that writes that VM's tag list directly into Ansible's own host-variable tree. **This is the actual link between provisioning and configuration** — the tags Terraform assigns to a VM are the exact same tags every later Ansible stage matches against.
+
+Terraform only manages virtual machines, deliberately — containers are handled differently (below).
+
+## Ansible: tag-driven, re-applied continuously
+
+```mermaid
+flowchart TD
+    Tags["Per-host tag list"] --> Init["Initialization stage<br/>runs once, at creation"]
+    Tags --> Compliance["Compliance stage<br/>re-applied every run"]
+    Compliance --> Sub["9 independently-tagged<br/>hardening sub-roles"]
+```
+
+Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles are active fleet-wide; none are disabled or pending.
+
+> [!IMPORTANT]
+> **Deliberate deviations from the hardening baseline, with real reasoning attached**: one kernel-level control is flipped from the stricter default on container-hosting machines specifically, because the container platform's own networking genuinely requires it. One SSH control stays permissive fleet-wide because real remote access depends on it. Both are documented as accepted, permanent trade-offs — not gaps waiting to be closed, and not silently ignored either.
+
+## The honest exception: not everything fits the clean model
+
+> [!WARNING]
+> Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into containers outside Terraform's management — a resource type mismatch that Terraform's VM-only tooling can't represent. Rather than deleting their old Terraform entries, they were commented out, preserving the historical record. Ansible still configures both today through the same tag-driven mechanism as everything else — only the initial provisioning step is now manual instead of automated.
+
+---
+Related: [`docs/disaster-recovery.md`](disaster-recovery.md)