diff --git a/docs/architecture-overview.md b/docs/architecture-overview.md new file mode 100644 index 0000000..ac92d93 --- /dev/null +++ b/docs/architecture-overview.md @@ -0,0 +1,82 @@ +# ๐Ÿ“ Architecture Overview + +> [!NOTE] +> The stable, directional laws governing how traffic moves through the fleet โ€” re-verified against live infrastructure, not assumed from older documentation. Two real corrections were caught and fixed the last time this was checked, which is exactly the point of re-verifying rather than trusting a prior write-up forever. + +## Dependency graph + +```mermaid +flowchart TD + Internet(["Internet"]) --> Edge["Edge Firewall
NAT ยท public reverse proxy ยท NTP ยท DHCP"] + Edge --> LAN["LAN"] + + LAN --> DNS["Internal DNS"] + LAN --> Vault["Vault"] + DNS --> Vault + + Vault --> Apps["App Host
~40 stacks"] + LAN --> FileServer["File Server"] + DNS --> FileServer + Vault --> Automation["Automation Host"] + LAN --> Automation + + Apps --> InternalProxy["Internal reverse-proxy zone"] + Apps --> SSO["SSO"] + + Automation --> Gitea["Git hosting
internal + public mirror"] + + Vault --> EdgeProxy["User-facing reverse proxy
mTLS"] + DNS --> EdgeProxy + Apps -.proxied by.-> EdgeProxy + SSO -.auth gate via.-> EdgeProxy + + Apps -.reports to.-> Monitoring["Monitoring"] + Automation -.reports to.-> Monitoring + FileServer -.reports to.-> Monitoring + EdgeProxy -.reports to.-> Monitoring + Vault -.reports to.-> Monitoring + DNS -.reports to.-> Monitoring + + Apps -.reports to.-> Wazuh["Security Monitoring"] + Automation -.reports to.-> Wazuh + FileServer -.reports to.-> Wazuh + EdgeProxy -.reports to.-> Wazuh + Vault -.reports to.-> Wazuh + DNS -.reports to.-> Wazuh +``` + +## Four trust boundaries, not two + +```mermaid +flowchart LR + WAN["WAN"] -->|public application boundary| EdgeFW["Edge Firewall / public proxy"] + Browser["LAN browsers"] -->|mTLS + SSO| UserProxy["User-facing proxy"] + Service["App-to-app"] -->|IP-allowlist + SSO| InternalProxy["Internal proxy"] + Infra["LAN infra-to-infra"] -->|explicit per-flow rules| HostFW["Host firewall rules"] +``` + +Backend services trust proxy headers, not raw client IP, on every path. + +## Two structurally independent traffic planes + +| Plane | Path | +|---|---| +| **Ingress โ€” public** | Client โ†’ edge firewall/proxy โ†’ backend | +| **Ingress โ€” LAN-facing** | Client โ†’ user-facing proxy (mTLS) โ†’ backend | +| **East-west โ€” app-to-app** | Service โ†’ internal proxy zone โ†’ service | +| **East-west โ€” infra-to-infra** | Service โ†’ firewall rule โ†’ service, direct, no proxy | + +> [!IMPORTANT] +> **Observability sits outside every request path on purpose.** Logs and metrics push asynchronously to monitoring infrastructure; nothing in that pipeline is inline with application traffic, and application services don't depend on monitoring availability. If monitoring goes down, collection stops โ€” nothing else does. + +## Caught by re-verification, not assumed + +> [!WARNING] +> The last full re-check against live infrastructure found two real corrections to what was previously documented: the public reverse-proxy layer had actually moved from running as its own dedicated VM to running as a package directly on the edge firewall โ€” and an older "flat mesh network, no mutual TLS" model for internal service traffic had been fully replaced by the two-zone split described in [`docs/dns-naming.md`](dns-naming.md), without the architecture documentation being updated to match at the time. Both were caught by running real commands against the live fleet, not by re-reading the previous version and assuming it still held. + +## What this map deliberately excludes + +Not a firewall rule listing, not a port inventory, not a service configuration reference โ€” those live in their own docs ([`docs/firewall-and-network.md`](firewall-and-network.md) and elsewhere). This is strictly the stable, directional shape of how trust and traffic actually move. + +--- +Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md) diff --git a/docs/disaster-recovery.md b/docs/disaster-recovery.md new file mode 100644 index 0000000..fd84478 --- /dev/null +++ b/docs/disaster-recovery.md @@ -0,0 +1,53 @@ +# ๐Ÿงฏ Disaster Recovery & Upgrades + +> [!NOTE] +> Three independent recovery layers, each covering a different kind of loss โ€” losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose. + +## Three independent recovery layers + +```mermaid +flowchart TD + Loss["Something is lost"] --> Type{"What kind?"} + Type -->|"A host, entirely"| IaC["Layer 1: Infrastructure as Code
rebuilt via the same pipeline
that built it originally"] + Type -->|"VM/LXC-level data"| PBS["Layer 2: VM/LXC backups
full-host snapshots"] + Type -->|"App data on the main host"| Kopia["Layer 3: Volume-level backups
per-stack, isolated repository"] +``` + +| Layer | Covers | Explicitly does not cover | +|---|---|---| +| Infrastructure as Code | A host lost entirely โ€” rebuilt from the same pipeline that built it, against its existing config | Secrets (recovered separately, not part of this layer) | +| VM/LXC-level backup | Full VM/LXC backups, including the secrets platform's own seal state and data | One host's Docker volume data, a large media storage pool, personal data | +| Volume-level backup | The gap the layer above leaves โ€” every stack's databases, uploaded files, and app config on the fleet's largest host | Anything outside that one host | + +> [!IMPORTANT] +> The volume-level backup layer runs with **read-only** access to every other stack's data, and its own repository lives on a dedicated, physically separate disk from the stacks it's protecting โ€” so the thing doing the backing-up isn't itself a single point of failure shared with what it backs up. + +> [!WARNING] +> **A known, honestly-documented gap**: one large media storage pool and one personal-data volume sit outside both the VM-level and volume-level backup layers. Not silently assumed safe โ€” explicitly flagged as unprotected until a decision is made to cover them. + +## Upgrades: two systems, deliberately not coupled + +```mermaid +flowchart LR + subgraph OS["OS-level"] + direction TB + T1["VMs โ€” 3 at a time
+ network-critical host check"] + T2["Hypervisor nodes โ€” 1 at a time
aborts if anything's mid-migration"] + T3["Backup server โ€” 1 at a time
aborts if a backup job is running"] + end + + subgraph App["App-level"] + direction TB + Strategy["Per-stack update strategy
independent, unrelated checks"] + end + + OS -.completely separate concern.-> App +``` + +Every OS-level tier reboots automatically when required, then waits and confirms the host is actually responsive again before moving on โ€” not just that the reboot command succeeded. + +> [!TIP] +> **Why app-level updates deliberately don't use batch/rolling deployment logic**: each host's update check is an independent, unrelated result โ€” a pinned version correctly reporting "nothing to do" isn't a failure, but a naive batching mechanism would treat that non-change as blocking progress to the next host, silently aborting every other host's run for the rest of the pass. Recognizing that app updates aren't a rolling deployment, and shouldn't be forced through tooling built for one, avoided a real class of silent partial failures. + +--- +Related: [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md) diff --git a/docs/iac-pipeline.md b/docs/iac-pipeline.md new file mode 100644 index 0000000..ffe1521 --- /dev/null +++ b/docs/iac-pipeline.md @@ -0,0 +1,47 @@ +# ๐Ÿ—๏ธ Infrastructure as Code + +> [!NOTE] +> Three tools, three responsibilities, one thread running through all of them: a host's tag list, generated once and consumed by every stage after it. + +## The pipeline + +```mermaid +flowchart LR + Packer["Packer
builds the base image"] --> Terraform["Terraform
provisions the VM"] + Terraform -->|generates| Tags["host_tags.yml
the tag list"] + Tags --> Ansible["Ansible
configures & hardens"] + Ansible -->|re-applied every run| Compliance["Compliance stage
not just once at creation"] +``` + +## Packer: hardening baked in at install time, not bolted on after + +A single base image is built from a minimal Rocky Linux install with CIS-oriented hardening baked into the kickstart itself โ€” disk partitioning and package selection decided at install time, not patched in afterward. Some controls (like mount-option restrictions on specific partitions) are genuinely easier and more reliable to get right at image-build time than to retrofit with a configuration tool after the fact. + +## Terraform: one resource type, tag generation as the real output + +> [!TIP] +> Every VM is one entry in a single map โ€” VM ID, target node, template, resources, network, disk, and its own tag list. Two supporting pieces do the real connective work: a wait step that blocks completion until the new VM actually answers SSH (so nothing downstream races a host that isn't up yet), and a template render that writes that VM's tag list directly into Ansible's own host-variable tree. **This is the actual link between provisioning and configuration** โ€” the tags Terraform assigns to a VM are the exact same tags every later Ansible stage matches against. + +Terraform only manages virtual machines, deliberately โ€” containers are handled differently (below). + +## Ansible: tag-driven, re-applied continuously + +```mermaid +flowchart TD + Tags["Per-host tag list"] --> Init["Initialization stage
runs once, at creation"] + Tags --> Compliance["Compliance stage
re-applied every run"] + Compliance --> Sub["9 independently-tagged
hardening sub-roles"] +``` + +Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list โ€” which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles are active fleet-wide; none are disabled or pending. + +> [!IMPORTANT] +> **Deliberate deviations from the hardening baseline, with real reasoning attached**: one kernel-level control is flipped from the stricter default on container-hosting machines specifically, because the container platform's own networking genuinely requires it. One SSH control stays permissive fleet-wide because real remote access depends on it. Both are documented as accepted, permanent trade-offs โ€” not gaps waiting to be closed, and not silently ignored either. + +## The honest exception: not everything fits the clean model + +> [!WARNING] +> Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into containers outside Terraform's management โ€” a resource type mismatch that Terraform's VM-only tooling can't represent. Rather than deleting their old Terraform entries, they were commented out, preserving the historical record. Ansible still configures both today through the same tag-driven mechanism as everything else โ€” only the initial provisioning step is now manual instead of automated. + +--- +Related: [`docs/disaster-recovery.md`](disaster-recovery.md)