diff --git a/docs/architecture-overview.md b/docs/architecture-overview.md index 4591b32..c486670 100644 --- a/docs/architecture-overview.md +++ b/docs/architecture-overview.md @@ -31,7 +31,7 @@ Every boundary above only means something if a service can't just skip it by rea **Three separate routing authorities enforce these** — the edge firewall, the user-facing proxy, and the internal proxy zone — rather than one central router handling everything. That's a deliberate rejection of the simpler single-router design: one router is one thing to hardening-focus on and one thing that takes everything down if it's compromised or misconfigured. -### What actually happens when one fails +### What Actually Happens When One Fails Not a hypothetical — this is the real, deliberate payoff of splitting routing authority three ways instead of one: @@ -87,7 +87,7 @@ That third fix isn't just a claim — it's the actual live ruleset: -## The other six traffic laws +## The Other Six Traffic Laws Each of these gets its own full mechanism doc elsewhere in this repo — this table is the map, not the territory: @@ -102,7 +102,7 @@ Each of these gets its own full mechanism doc elsewhere in this repo — this ta The pattern repeats across all eight laws, not just the trust-boundary story above: **control-plane failure never takes down the data plane.** Every "failure behavior" row above ends the same way — the thing that broke stops working, and everything else keeps running regardless. That's not a coincidence; it's the same design principle applied consistently everywhere, not just to network trust. -## What this map deliberately excludes +## What This Map Deliberately Excludes Not a firewall rule listing, not a port inventory, not a service configuration reference — those live in [`docs/firewall-and-network.md`](firewall-and-network.md) and the docs linked in the table above. This is strictly the stable, directional shape of how trust and traffic actually move — meant to still be true even after the specific tools implementing it change again, the way it already has three times. diff --git a/docs/cron-scheduler.md b/docs/cron-scheduler.md index b86e7a1..603e132 100644 --- a/docs/cron-scheduler.md +++ b/docs/cron-scheduler.md @@ -3,7 +3,7 @@ > [!NOTE] > A NocoDB-backed inventory of every known scheduled job in the fleet — Kestra flows, systemd timers, Ansible-managed cron, app-internal schedules — visualized as a Dashy heatmap so a new job's time can be picked without colliding with existing load. Small in scope compared to the rest of this repo, but it's a real answer to a problem every one of the other systems here creates: once you have 50 independently-scheduled jobs across [`docs/orchestration.md`](orchestration.md), [`docs/registry-and-build.md`](registry-and-build.md), and [`docs/disaster-recovery.md`](disaster-recovery.md), "when does anything run" stops being something you can hold in your head. -## The heatmap +## The Heatmap A Node-RED tab fetches the inventory straight from NocoDB's REST API at request time — no cache — and renders it as a 7×24 grid (Mon–Sun × 00:00–23:00). Each cell's color is a **computed load weight**, not a raw job count: base `1` per job, plus a duration-bucket weight, plus a resource-tag weight (`disk-intensive`=2, `cpu-intensive`=1.5, `network-intensive`=0.5, summed if a job carries more than one), plus a flat `+1` if the job's category is `backup` — disk I/O contention from an overlapping backup is judged a worse collision than two lightweight jobs sharing an hour, and the weight formula says so explicitly rather than pretending every job costs the same. Weights bucket into 4 color bands (green → yellow → orange → red) relative to the current busiest cell. Jobs whose time can't be resolved to a specific day+hour are excluded from the grid and listed separately rather than forced into a misleading cell. @@ -22,7 +22,7 @@ Both of those are Dashy's *rendered* view. This is the raw NocoDB table undernea ![The real "Cron Scheduler" NocoDB table in grid view, showing all 50 records with their Job Name, Category, Schedule Expr, Est Start, Duration, Resource Tags, and Source columns](../assets/cron-scheduler-table.png) -## Why a manual inventory instead of an auto-discovered one +## Why A Manual Inventory Instead Of An Auto-discovered One The harder, more "correct" version of this tool would auto-discover every scheduled job by querying Kestra's API, parsing every host's crontab and systemd timers, and re-syncing continuously. That was scoped down deliberately: auto-discovery across three genuinely different scheduling mechanisms (Kestra's API, systemd's timer units, cron's own format) is real engineering effort for a tool whose entire job is picking a good time slot for the next new job — a task that happens occasionally, not continuously. A manually-maintained table, kept honest by being checked at the one moment it actually matters (adding a new job), was judged the better return on effort than building and maintaining three different scanners for a problem that doesn't need real-time accuracy. diff --git a/docs/dashy-soc-board.md b/docs/dashy-soc-board.md index 0a56414..2905e3e 100644 --- a/docs/dashy-soc-board.md +++ b/docs/dashy-soc-board.md @@ -47,7 +47,7 @@ for n in data: Build SOC Board HTML -> 104443 chars ``` -## Why one function instead of native Dashy widgets +## Why One Function Instead of Native Dashy Widgets Dashy's own built-in widget types cover simple cases well, but none of them can join data across sources the way this board needs to — a per-host tile that combines Wazuh findings, Prometheus resource usage, and Slack alert history into one expandable panel isn't expressible as a single native widget config. Building it as one Node-RED function that does the aggregation server-side, then hands the dashboard pre-joined data, was the tradeoff made: a large, custom function to maintain, in exchange for a board that actually reflects cross-system state instead of fourteen disconnected tiles a human has to mentally correlate themselves. @@ -66,7 +66,7 @@ Clicking through to one of those "folded-in specialty widgets" — the Vault sea > [!TIP] > **Deliberately scoped**: security/error/warning signal only. Per-app uptime is handled separately by Dashy's own native status checks on individual tiles — the only up/down signal inside this board is *infrastructure-tier* reachability from Prometheus, not per-app availability. Keeping the scope narrow — resisting the temptation to also make this the uptime dashboard — is what keeps the board actually scannable instead of turning into a second copy of Grafana. -## Every finding ships with a copy-ready incident summary +## Every Finding Ships With a Copy-ready Incident Summary Each finding's expanded row has a "Copy Debug Prompt" button, not just an acknowledge control. Clicking it builds a fully-formed summary from that specific finding's own host, severity, title, and detail, and copies it straight to the clipboard — a complete, structured starting point for triage without manually re-gathering context from the board itself. This is the real template: @@ -80,7 +80,7 @@ Detail: {meta} ``` -## Acknowledge mechanism: client-side, no backend +## Acknowledge Mechanism: client-side, no backend ![A finding's stable key (hostId::group::title) gets acknowledged into the browser's localStorage as a flat map, which live re-derives the host tile badge, the Active Issues strip, and any open detail panel's gauges — and never touches Wazuh, Slack, or any other upstream source](../assets/dashy-soc-board-acknowledge.svg) @@ -95,7 +95,7 @@ localStorage.getItem("socAckedV2") > [!IMPORTANT] > Acknowledging a finding is a **display filter only** — it never writes back to Wazuh, Slack, or any upstream source described elsewhere in this repo, is scoped per-browser rather than fleet-wide, and a full data set for every host is always kept in memory (in inert template elements, not just whichever panel is currently open) so the recompute has everything it needs regardless of which panel happens to be open. A reload in a different browser shows the finding again. This was a deliberate tradeoff favoring simplicity — no backend state to build, secure, or back up — over persistence, made because the board's job is triage visibility, not an audit trail of who acknowledged what (that's what Wazuh's own alert log in [`docs/security-monitoring.md`](security-monitoring.md) is for). -## Real integration gotchas hit building this +## Real Integration Gotchas Hit Building This > [!WARNING] > A few non-obvious lessons from actually building this, in case they save someone else the debugging time: @@ -105,7 +105,7 @@ localStorage.getItem("socAckedV2") > - Node-RED's Mustache templating in a core node's URL field is scoped to `msg` directly, not `msg.msg` — the correct reference is `{{{url}}}`, not `{{{msg.url}}}`, and it needs the triple-brace form specifically for values containing characters like `&` that the double-brace form would HTML-escape. Get either wrong and the request silently goes to an empty or mangled URL instead of erroring. > - Replacing a bind-mounted config file's content doesn't get picked up by a plain container restart — the container keeps watching the old inode; a full recreate is required, the same gotcha that shows up in [`docs/registry-and-build.md`](registry-and-build.md)'s redeploy path. -## Widget design: shape follows data, not one template everywhere +## Widget Design: shape follows data, not one template everywhere Every widget's visual style is chosen to match the *shape* of the data it shows, not a single card/badge component reused everywhere — the same design instinct behind this repo's own docs, honestly: diff --git a/docs/disaster-recovery.md b/docs/disaster-recovery.md index d35614e..0548a09 100644 --- a/docs/disaster-recovery.md +++ b/docs/disaster-recovery.md @@ -3,11 +3,11 @@ > [!NOTE] > Three independent recovery layers, each covering a different kind of loss — losing one doesn't mean the others are unavailable. And two deliberately decoupled upgrade systems, kept separate on purpose. Both halves of this doc are about the same underlying discipline: assuming things will eventually fail, and designing so one failure doesn't cascade into a second. -## Three independent recovery layers +## Three Independent Recovery Layers ![Something lost branches three ways depending on what kind of loss it is: an entire host rebuilds through the same Packer, Terraform, and Ansible pipeline that built it originally; VM/LXC-level data recovers through Proxmox Backup Server's full-host snapshots; app data on the main Docker host recovers through Kopia's volume-level, per-stack isolated repository](../assets/disaster-recovery-layers.svg) -## Why three layers instead of one comprehensive backup system +## Why Three Layers Instead of One Comprehensive Backup System A single backup tool covering everything was the simpler design, and it was rejected because "a host" and "a database's data" are fundamentally different kinds of loss with different recovery needs. Rebuilding a lost host from [`docs/iac-pipeline.md`](iac-pipeline.md)'s pipeline is fast and doesn't need a backup at all — the *code* is the backup. But that same pipeline can't recover yesterday's database contents, which is what Layer 2 and 3 exist for. Trying to make one tool do both jobs well tends to produce a tool that's mediocre at each — full-host imaging is slow and coarse-grained for restoring one app's data, and application-level backup can't reconstruct a host's OS and configuration from scratch efficiently. Three narrow, purpose-fit layers beat one general-purpose one here. diff --git a/docs/dns-naming.md b/docs/dns-naming.md index 3dadfdc..c44dbd5 100644 --- a/docs/dns-naming.md +++ b/docs/dns-naming.md @@ -11,14 +11,14 @@ | **Depends on** | Nothing upstream for internal resolution — it's the root of trust for internal naming | | **Depended on by** | Every system in this repo that resolves a hostname — which is effectively all of them | -## Why self-hosted DNS at all +## Why Self-hosted DNS At All A homelab this size could get away with a shared hosts file or a single flat zone. Two things specifically ruled that out: 1. **The two-zone trust model needs a real answer to "which front door am I behind."** Splitting traffic into a mutual-TLS-enforced browser zone and a lighter trusted-IP internal zone (see [`docs/firewall-and-network.md`](firewall-and-network.md)) only works if something authoritative decides, per-hostname, which zone a request lands in. A hosts file can't do that at scale across a dozen-plus hosts; a real DNS server with two real zones can. 2. **Every other system needs one source of truth for internal names.** The secrets platform issues a certificate to the DNS server's own admin interface (see below); the orchestration engine resolves every fleet host by name before it can SSH anywhere; the observability pipeline ships DNS query logs as their own dedicated stream. Hand-rolling internal name resolution per-app would mean N inconsistent answers instead of one. -## The two zones +## The Two Zones ![Anything on the LAN queries Technitium, which either answers from one of its two authoritative internal zones — browser-facing, routing to the user-facing reverse proxy with mTLS+SSO, or app-to-app, routing to the internal reverse proxy with a trusted-IP bypass — or forwards unresolved queries upstream to the edge firewall and a public resolver](../assets/dns-naming-zones.svg) @@ -46,12 +46,12 @@ Same result for `loki` and `wazuh-indexer` (caddy-zone only), and the reverse fo | App-to-app | `NXDOMAIN` | | Browser-facing | `NXDOMAIN` | -## DNS as an enforced boundary, not just a convenience +## DNS As An Enforced Boundary, Not Just a Convenience > [!IMPORTANT] > Every host on the LAN has its resolver pointed at this one internal DNS server — and that's enforced, not just configured. Direct outbound DNS queries to anywhere else (a public resolver, another host acting as its own resolver) are blocked at the firewall level, covered in [`docs/firewall-and-network.md`](firewall-and-network.md). This closes a real bypass: without it, any host could sidestep the internal zones entirely by just asking a public resolver instead, which would quietly undermine the whole two-zone trust model above. DNS enforcement is what makes the rest of this architecture actually load-bearing instead of advisory. -## Two things live-checking found the documentation had wrong +## Two Things Live-checking Found The Documentation Had Wrong Trusting the KB without re-checking it against the running server would have kept both of these wrong indefinitely — so both got checked directly against the real config and the real zone data, not taken on the KB's word: @@ -68,7 +68,7 @@ $ strings /etc/dns/dns.config | grep -E '^\d+\.\d+\.\d+\.\d+$' Neither is a security problem — the reverse zone being broader than described doesn't leak anything, and the missing forwarder just means one less redundant upstream. What they have in common is more useful than either fact alone: documentation describing what a system does isn't the same as verifying it, and the gap between the two doesn't announce itself until something actually goes and checks. -## Where DNS shows up in every other system +## Where DNS Shows Up In Every Other System DNS isn't a standalone concern in this fleet — it's a dependency almost everything else sits on top of: diff --git a/docs/firewall-and-network.md b/docs/firewall-and-network.md index 952fa39..2e9072e 100644 --- a/docs/firewall-and-network.md +++ b/docs/firewall-and-network.md @@ -3,11 +3,11 @@ > [!NOTE] > pfSense at the edge, firewalld on every individual host — two layers, two different jobs, and a network model with no blanket "anything on the internal network can talk to anything else on it" rule anywhere. Every real flow is explicit; everything else is logged and dropped. This is the enforcement layer underneath the trust boundaries described in [`docs/architecture-overview.md`](architecture-overview.md). -## Why two layers, not one +## Why Two Layers, Not One A single firewall layer was rejected early: the edge firewall (pfSense) is the fleet's only genuinely public-facing component, so it carries a different threat model — internet-scale scanning and abuse — than internal host-to-host traffic ever will. Collapsing both into one ruleset would mean either the internal segmentation rules get diluted by public-facing noise, or the public-facing rules get relaxed to match internal assumptions. Keeping them separate means each layer's rules can be as strict as that layer's actual threat model demands, independently. -## The one rule that matters most +## The One Rule That Matters Most This wasn't always this tight. The LAN interface used to run pfSense's own standard default — a blanket allow between hosts — before it was deliberately narrowed to LAN→WAN/internet only, closing off unrestricted lateral movement that had nothing to do with any application actually needing it. What replaced it is the model below. @@ -24,7 +24,7 @@ That narrowing and the catch-all it produced aren't just claims — here's the a DNS enforcement specifically doesn't rely on just one mechanism. Every static-lease device gets the internal DNS server pushed via a per-lease DHCP override — every current static lease carries it — but that's a soft guarantee: a brand-new device on a plain dynamic lease would fall back to the router's own upstream resolvers instead, at least until it gets a static reservation of its own. The firewall closes that gap independently, at the network level: an explicit rule blocks any LAN client from reaching external DNS outright, full stop. So a device that somehow ended up pointed at the wrong resolver still can't actually use it — DNS bypass isn't just discouraged by configuration, it's structurally blocked by a second, unrelated layer that doesn't depend on the first one being right. -## Named groups, not raw addresses +## Named Groups, Not Raw Addresses pfSense aliases (named groups) stand in for hardcoded IP lists throughout — adding a host to a group is a single membership edit, not a hunt through every rule that needs to reach it. Groups nest: a "whole managed fleet" alias is composed of several narrower per-role aliases, so a rule against the broad group automatically covers new members of any narrower one without the rule itself ever changing. @@ -34,7 +34,7 @@ The same discipline applies to the other axis of every rule, and it's still acti ![pfSense firewall port aliases, one per service/flow](../assets/pfsense-aliases-ports.png) -## Per-host firewalls: minimal, not blanket +## Per-host Firewalls: minimal, not blanket Each host runs firewalld locally, opening only the specific ports it actually needs — not a blanket "allow everything, block nothing" default. Vault's host goes further than the rest: instead of one open port for its most sensitive service, individual source IPs get their own explicit rich-rule — five named clients, confirmed directly against the live ruleset, and zero blanket ports open at all — so a compromised unrelated service on the same network still can't reach it even if it somehow got past pfSense. This is the actual live output, not a paraphrase: @@ -83,7 +83,7 @@ hawser: 0.0.0.0:2376->2376/tcp, [::]:2376->2376/tcp Three ports, zero matching firewalld rules, all three genuinely reachable LAN-wide via Docker's own NAT — not a hypothetical, the exact gap above on a host provisioned this same week. -## Edge firewall: three jobs, one device +## Edge Firewall: three jobs, one device One device, three distinct responsibilities — kept together deliberately, not split into separate boxes, since none of the three is heavy enough on its own to justify another VM to patch and monitor: @@ -94,7 +94,7 @@ One device, three distinct responsibilities — kept together deliberately, not > [!IMPORTANT] > **Failure impact, thought through in advance, not discovered during an actual outage**: if pfSense goes down, public access and outbound internet both go with it — but every internal service that doesn't depend on WAN reachability keeps running normally, including the orchestration flows in [`docs/orchestration.md`](orchestration.md), which only ever reach internal hosts. The blast radius is deliberately bounded to "the edge," not "the fleet" — a design property worth stating explicitly rather than leaving implicit, since it's exactly the kind of thing that should be known *before* an incident, not discovered during one. One thing does quietly degrade even though nothing breaks outright, and it's worth being honest about rather than rounding up to "everything's fine": Vault Agent's certificate renewal and Technitium's own PKI issuance both route through pfSense-mediated LAN paths, so an extended outage would eventually stall renewals fleet-wide — not an immediate failure, but a real one if the outage runs long enough to matter. -## Defense-in-depth below the network layer: PAM +## Defense-in-depth Below The Network Layer: PAM Firewalls aren't the only layer that matters. On this fleet's Debian-family hosts, the login stack itself got hardened after a real, *recurring* incident — not a one-off. `pam_unix.so`'s default auth line uses a numbered jump ("on success, skip forward N modules") to reach approval, which is fragile by construction: any future package that inserts its own line anywhere in that stack silently shifts what the number actually lands on. This broke real, correct-password logins **twice**: once during initial rollout, fixed by bumping the number up by one — and again five days later, on a different host, when an unrelated package pulled in its own PAM line and broke that same numbered jump a second time, landing on a deny instead of an allow. A correct password got treated as a failure. The permanent fix replaced the number with a keyword that terminates the stack on success immediately, regardless of how many lines ever get added after it later — closing the entire class of bug, not just the one instance that got caught. Rocky/RHEL hosts never carried this risk in the first place; they manage their PAM stack through `authselect`, a declarative system not exposed to the same ordering hazard. diff --git a/docs/iac-pipeline.md b/docs/iac-pipeline.md index 7fbd2ac..6069638 100644 --- a/docs/iac-pipeline.md +++ b/docs/iac-pipeline.md @@ -7,7 +7,7 @@ ![Packer builds the base image, Terraform provisions the VM and generates host_tags.yml, which Ansible consumes to configure and harden the host, re-applying its compliance stage on every run](../assets/iac-pipeline-tools-flow.svg) -## Why three separate tools instead of one +## Why Three Separate Tools Instead Of One A single tool covering image-build-through-configuration was a real option — several products exist specifically for that — and it was rejected because the three jobs have genuinely different rhythms: an image gets rebuilt rarely (a base OS refresh), a VM gets provisioned once, and configuration gets re-applied continuously (the compliance stage below runs on every pass, not just at creation). Collapsing all three into one tool's model would mean forcing the rarely-changing and the continuously-changing onto the same execution model. Keeping them separate, connected by one deliberate handoff (the tag file below) rather than a tightly-coupled integration, means each tool does the one job it's actually good at. @@ -162,7 +162,7 @@ compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', ' Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 15 total compliance-stage roles above — the other 6 handle observability, notifications, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior. -## The honest exception: not everything fits the clean model +## The Honest Exception: not everything fits the clean model > [!WARNING] > Two hosts don't follow the pipeline above cleanly, and that history is preserved rather than erased. Both were originally provisioned as Terraform-managed virtual machines, then at some point converted directly into LXC containers outside Terraform's management — a resource-type mismatch that Terraform's VM-only tooling genuinely can't represent (LXCs aren't `proxmox_vm_qemu` resources). Rather than deleting their old Terraform entries when this happened, they were commented out, preserving the historical record of how these hosts actually came to exist rather than presenting a cleaner story than what's true. Ansible still configures both today through the same tag-driven mechanism as every VM — only the initial provisioning step is now manual instead of automated. This is the kind of real-world messiness that a from-scratch design document never mentions but that every infrastructure actually accumulates over time — worth showing, not hiding. diff --git a/docs/netbird-zero-trust.md b/docs/netbird-zero-trust.md index 1c80632..7daa1a8 100644 --- a/docs/netbird-zero-trust.md +++ b/docs/netbird-zero-trust.md @@ -3,7 +3,7 @@ > [!NOTE] > A self-hosted **NetBird** WireGuard mesh gives every fleet host — and every device that needs to reach it — its own stable overlay identity, independent of where that device physically sits. Access is granted per-identity through mesh policy, not inferred from LAN position. The mesh is **deny-by-default**: nothing crosses it unless an explicit policy allows it, the same instinct behind every other boundary in [`docs/architecture-overview.md`](architecture-overview.md), applied to remote and roaming access specifically. -## Why a mesh, not a classic VPN +## Why A Mesh, Not a Classic VPN A traditional VPN concentrator was the obvious alternative, and it was rejected for the same reason a flat LAN got rejected everywhere else in this fleet: it's one more device sitting in a path, and reaching it means punching a hole to a single choke point rather than establishing an identity-scoped connection to exactly the host that's actually needed. A mesh instead gives every peer — fleet host or personal device — its own stable identity (`.netbird.selfhosted`, a `100.91.0.0/16` overlay address) and lets WireGuard tunnels form directly, peer-to-peer, falling back to a relay only when direct connectivity isn't possible. Policy is written once, per identity or group, and travels with the peer regardless of which network it's physically on. @@ -13,7 +13,7 @@ This is the actual shape of it — the control plane coordinates, but it's delib That's the actual difference from a VPN concentrator, drawn rather than just asserted: a concentrator puts itself in every data path by design; here the control plane's spokes only ever carry coordination traffic, and every thick line is two peers talking directly to each other. -## Control plane +## Control Plane One `netbirdio/netbird-server` container runs Management, Signal, the embedded relay, and NetBird's own embedded IdP together — combined-server mode, not four separate services. A digest-pinned `netbirdio/dashboard` container serves the admin SPA. This is the real container list, not a claimed one: @@ -38,7 +38,7 @@ Self-hosted rather than the vendor's own managed cloud offering, for the same re | Peer identity | dashboard name = short hostname; mesh DNS label = `.netbird.selfhosted` | | Peer WireGuard | port `51820`, kernel interface `wt0` | -## The edge routing problem, and what actually broke +## The Edge Routing Problem, and What Actually Broke > [!WARNING] > **Learned the hard way during bring-up, not designed in from the start.** NetBird's Management and Signal services need native gRPC over HTTP/2-cleartext (h2c) end-to-end — any downgrade to HTTP/1.1 anywhere in the chain kills the bidirectional stream with an `unexpected EOF`, and the client agent just wedges with no useful error. The edge firewall's own reverse proxy can carry h2c to a single plain-HTTP backend, but it can't perform NetBird's own content-type/path split between the management API, the relay, and the dashboard SPA — so a small in-stack proxy does exactly that split, one hop further in. A second, unrelated thing broke the same week: the public DNS record for this service has to stay **DNS-only**, not proxied through a CDN — a proxying CDN returns an outright 403 to the gRPC-speaking agent, no explanation given, and the fix was disabling that proxying entirely for this one record rather than trying to work around it. @@ -79,7 +79,7 @@ $ cat Caddyfile } ``` -## Fleet enrolment +## Fleet Enrolment A dedicated Ansible role installs the mesh agent and enrols a host, opt-in per host via the same tag-driven mechanism as every other role in [`docs/iac-pipeline.md`](iac-pipeline.md) — nothing joins the mesh by default just because it exists. Enrolment uses a reusable, class-scoped setup key fetched live from the secrets platform at play time, never stored in the repo, and read only on a host's first join. Because peer approval is required, a brand-new peer pauses for an operator to approve it in the dashboard before it gets any mesh access at all — there's no silent auto-join path. @@ -91,7 +91,7 @@ This is the real, live peer roster, not a claimed enrolment count: ![Real NetBird dashboard: connected peers, a mix of fleet hosts and personal devices, each showing its own group membership, agent version, and OS](../assets/netbird-zero-trust-peers.png) -## Zero-trust policy: deny-by-default, not just present +## Zero-trust Policy: deny-by-default, not just present The mesh ships with a built-in allow-everything default policy — the first real security decision here was turning it off. With it disabled, no traffic crosses the mesh interface at all unless a specific policy grants it; LAN traffic is completely untouched by this, since the policy only governs the overlay interface, not the physical network. diff --git a/docs/notifications.md b/docs/notifications.md index eebd870..65c1d2a 100644 --- a/docs/notifications.md +++ b/docs/notifications.md @@ -3,11 +3,11 @@ > [!NOTE] > Mosquitto (MQTT) and a single Node-RED instance are the backbone here — two independent fan-in branches, app webhooks and OS-level host alerts, converging on one flow and one shared formatter before reaching Slack. The point isn't the individual pieces; it's that every alert reads the same regardless of where it came from. -## Fan-in, fan-out +## Fan-in, Fan-out ![Three independent fan-in sources converging on one shared formatting subflow: 15 apps over HTTP webhook go through the App Router and Switch to a per-app mapper, Kopia/Bazarr post to their own dedicated endpoints and bypass the router entirely with their own mapper, and every host publishes over MQTT into a topic-agnostic Inventory Parse function — all three land on the same subflow before reaching Slack](../assets/notifications-fan-in-fan-out.svg) -## Why one Node-RED instance instead of per-app scripts +## Why One Node-RED Instance Instead Of Per-App Scripts The alternative — each app posting directly to a per-app Slack webhook, or a small script per integration — was rejected because it would mean N different message formats, N different places to add a new integration's logic, and no single point to add fleet-wide behavior (like the OS-level checks below) without touching every app individually. One Node-RED instance with a shared router and a shared formatting subflow means a new app's alerts get the same visual treatment as every existing one automatically, and a fleet-wide change (say, adding a severity-based routing rule) happens in one place. @@ -46,7 +46,7 @@ This is the real flow, not a redrawn version of it — the App Router feeding a ![Real Node-RED canvas: HTTP-in webhook through App Router into a 16-rule Switch function feeding per-app mapper functions (including netbox mapper and kestra mapper, both still wired but dead), MQTT-in through the Inventory Parse Function, and Kopia/Bazarr's own dedicated HTTP-in nodes bypassing the router entirely — all converging on the shared MQTT → Slack subflow](../assets/notifications-node-red-flow.png) -## Message broker +## Message Broker | | | |---|---| @@ -85,11 +85,11 @@ And this is the broker actually being used, not just configured — real connect > [!IMPORTANT] > **A real, tracked fragility, worth naming precisely**: the publish identity's password has to stay in sync across three separate places — the Vault-rendered broker credential file, the source value used to render every host's own publish script (via the same tag-driven Ansible model in [`docs/iac-pipeline.md`](iac-pipeline.md)), and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from Vault at all, unlike everything else in [`docs/secret-distribution.md`](secret-distribution.md)). Rotating it means updating all three by hand — a documented, real exception to the "one source of truth" rule that governs every other secret in this fleet, called out explicitly rather than left for someone to discover mid-rotation. -## Detection logic +## Detection Logic The Node-RED App Router figures out which app sent a webhook by inspecting the payload's shape — an explicit field for some apps, structural pattern-matching for others (Emby, for instance, is identified purely by a nested `data.Server` field only that vendor's payload happens to include, since it never sends an explicit app identifier). Kopia and Bazarr skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON and the shared router's shape-detection logic simply doesn't apply to them. -## Every host, not just apps +## Every Host, Not Just Apps > [!TIP] > A dedicated Ansible role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert — the same fleet-wide, tag-driven rollout pattern used for the observability agent in [`docs/observability.md`](observability.md) and the compliance hardening in [`docs/security-monitoring.md`](security-monitoring.md). It runs over 20 distinct checks per host — disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (Vault audit log anomalies, Vault Agent authentication failures) — deliberately complementary to Wazuh in [`docs/security-monitoring.md`](security-monitoring.md), not duplicating its file-integrity or account monitoring. Splitting OS-health checks from the SOC's own detection engine keeps each system's role narrow enough to actually reason about, instead of one monolithic monitoring tool trying to do both jobs at once. @@ -103,7 +103,7 @@ The Node-RED App Router figures out which app sent a webhook by inspecting the p > > 22 from that count, plus disk-usage thresholds — which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name — makes 23. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts. -## One formatter, every source +## One Formatter, Every Source Every branch — app webhooks, host-level alerts, and Kopia/Bazarr's own dedicated non-JSON endpoints — converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it — a reader in Slack never has to learn a different alert format depending on which system fired it. diff --git a/docs/observability.md b/docs/observability.md index b6a6995..cf827a3 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -10,7 +10,7 @@ | **Metrics** | Push-based `remote_write`, not pull-based scraping | | **Coverage** | All fleet hosts, confirmed live | -## Why push instead of pull — the actual reasoning +## Why Push Instead Of Pull — The Actual Eeasoning A pull-based model (Prometheus scraping every host directly) is the more common default, and it was rejected for a specific reason tied directly to [`docs/firewall-and-network.md`](firewall-and-network.md): scraping means the monitoring host needs inbound network access to every single host it monitors, which means punching a hole through the zero-blanket-allow firewall model for every host, every metrics port. A push model inverts that — only the monitoring host needs to accept inbound writes, and every other host only ever *initiates* outbound connections, which fits the existing "explicit outbound rule, default-deny inbound" posture instead of fighting it. The tradeoff is real: push-based metrics are slightly harder to reason about for ad-hoc debugging (no simple `curl` against a target to see what it'd report), but it was the right call given the network model already in place. @@ -18,7 +18,7 @@ A pull-based model (Prometheus scraping every host directly) is the more common ![Four per-host sources (journald, Docker containers, node metrics, native app metrics) all feed into Grafana Alloy, which ships logs to Loki and metrics to Prometheus via remote_write every 30s](../assets/observability-pipeline.svg) -## Agent install: deliberate, not default +## Agent Install: deliberate, not default > [!TIP] > The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. That's a real maintenance cost (manual version bumps instead of `apt upgrade` handling it), accepted because it keeps the exact version pinned and reproducible across the fleet through the same tag-driven Ansible model described in [`docs/iac-pipeline.md`](iac-pipeline.md), rather than whatever version a package repo happens to serve on a given day. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway). diff --git a/docs/orchestration.md b/docs/orchestration.md index 232ee93..b9519ac 100644 --- a/docs/orchestration.md +++ b/docs/orchestration.md @@ -7,11 +7,11 @@ This is the real flow list, not a claimed count: ![Kestra's real flow list, homelab namespace: 13 flows total, every one showing a real last-execution date and SUCCESS status](../assets/orchestration-flow-list.png) -## Why only one flow gets a human gate +## Why Only One Flow Gets a Human Gate A blanket human-gate policy was the obvious simpler design, and it was rejected: most of what these flows do — OS patching, container rebuilds, disk cleanup — already has its own automated safety net (the health-check/rollback path in [`docs/registry-and-build.md`](registry-and-build.md)), so a checkpoint on top of that would just be friction with nothing real behind it. The one flow that applies non-Docker infrastructure updates — Vault, Technitium, Alloy — is different: it draws the line at exactly the same place internally, auto-applying its own low-risk, self-contained piece without a gate and pausing only for the higher-stakes core-infrastructure changes. The pattern isn't "automation is risky, add a human" everywhere — it's a human checkpoint placed specifically where the risk profile actually warrants one, and left out everywhere it doesn't. -## Two patterns, not one +## Two Patterns, Not One ![12 of 13 flows run straight through on schedule with no human gate, relying on the automated health-check/rollback path instead; the one flow applying Vault/Technitium/Alloy updates runs a check-mode pass, pauses indefinitely on anything pending, and only applies after explicit human approval](../assets/orchestration-flow-patterns.svg) @@ -37,12 +37,12 @@ disk-cleanup: 0 app-update: 1 ``` -## Identity split, per flow +## Identity Split, Per Flow > [!IMPORTANT] > Most flows connect using one operator identity over SSH. One flow — the one that needs to run Ansible itself on the control node, not just reach out over SSH to a target — deliberately logs in as a separate, narrower service identity instead, since it needs local execution rights on the control node that the SSH-only identity doesn't have. Same underlying key material either way, just presented as two different accounts depending on what the flow actually needs to do locally, following the same "scope credentials to exactly what's needed" instinct as the least-privilege Git access model in [`docs/sso-and-identity.md`](sso-and-identity.md). Worth checking which one a given flow uses before assuming — it's not always the same answer, and assuming wrong is a fast way to get a confusing permission error mid-debugging. -## A real special case, thought through in advance +## A Real Special Case, Thought Through In Advance > [!WARNING] > One of the two Proxmox hypervisor nodes also happens to host the automation VM that Kestra itself runs on. Updating and rebooting that node the normal way — waiting for it to reconnect before declaring success — would mean waiting on its own death, since the flow doing the waiting dies with the VM it's waiting on. Its reboot path is deliberately fire-and-forget instead, which means that flow's own success/failure notification isn't reliable for that specific node: the execution can die mid-notify along with the VM, including the notification pipeline itself from [`docs/notifications.md`](notifications.md), which also runs there. A separate, dedicated verification flow (pinging both nodes fresh afterward) is the actual source of truth when the two are triggered in sequence — not the original flow's own notification. This is the kind of edge case that only shows up once you actually think through "what happens when the thing doing the orchestrating is itself one of the things being orchestrated" — a real, non-obvious problem in any self-hosted automation system. This is the real config comment, not a summary of the reasoning: @@ -57,7 +57,7 @@ app-update: 1 > reboot_wait_for_reconnect: false > ``` -## Trilium KB self-maintenance +## Trilium KB Self-maintenance Two flows keep this fleet's own internal knowledge base structure current — the *only* two things allowed to write to it, everything else (including this repo's own generation) only ever reads from it: @@ -69,7 +69,7 @@ Two flows keep this fleet's own internal knowledge base structure current — th > [!TIP] > The push-triggered flow also runs a lightweight completeness check for the fleet's own host-mapped repos: if a push adds a new top-level stack directory that's never mentioned in that repo's own README, and the README wasn't touched in the same push, it posts a warning directly to that host's own alert channel via [`docs/notifications.md`](notifications.md) — a small, automatic nudge against exactly the kind of documentation drift that this whole portfolio repo exists to avoid in the first place. -## Outside the orchestration engine entirely +## Outside The Orchestration Engine Entirely The automation host's own OS patches and Kestra's own container updates run via a standalone host-level systemd timer, not through Kestra itself — for the same reason as the hypervisor special case above: a system can't reliably orchestrate patching itself mid-run without risking killing the process doing the patching. Being outside the normal flow means these updates don't get the same visibility as everything else in this doc, which is a deliberate, accepted tradeoff — worth a periodic manual check rather than fully trusting the dashboard to surface it. This is the real timer, not an assumption about its schedule: diff --git a/docs/registry-and-build.md b/docs/registry-and-build.md index 8b018a3..7f6c97a 100644 --- a/docs/registry-and-build.md +++ b/docs/registry-and-build.md @@ -9,7 +9,7 @@ | **Trigger** | Weekly schedule via Kestra, not push-triggered | | **Rollout** | All 5 Docker hosts converted; the largest has 36 of 41 stacks on this pattern | -## Why self-host the registry instead of just using Docker Hub +## Why Self-host The Registry Instead Of Just Using Docker Hub Pulling directly from Docker Hub/ghcr at runtime was the status quo before this existed, and it was replaced for a concrete reason: every container pull became a runtime dependency on an external service staying up and staying reachable through the firewall model in [`docs/firewall-and-network.md`](firewall-and-network.md). Building images in-fleet and pushing to a registry that's already inside the trust boundary means a redeploy never depends on an external registry's availability, and the exact image contents are fully within this fleet's own control rather than whatever an upstream maintainer pushed as `:latest` today. @@ -19,14 +19,14 @@ This is the real registry, not a claim about it — every one of these is a genu ![Gitea's own package registry, filtered to container type — real self-built images including this fleet's own automation tooling, all private](../assets/registry-and-build-packages.png) -## The cheap-check-first build decision +## The Cheap-check-first Build Decision ![Every build starts with docker manifest inspect — metadata only, no pull — comparing the upstream image's digest against what's already in the fleet registry. If they match, nothing runs at all. If they differ, only then does it pull, build, compare image IDs, push what changed, and redeploy through the shared health-check/rollback path](../assets/registry-and-build-cheap-check.svg) > [!TIP] > Most stacks use a minimal `FROM :${VERSION}` wrapper Dockerfile rather than a full clone-and-build-from-source setup — simpler, builds near-instantly, can't break from upstream build tooling changing underneath it, while still achieving the real goal: an in-fleet registry with no direct Docker Hub/ghcr dependency at runtime. A handful of multi-build-image stacks (the ones with more than one real upstream component, like Paperless-ngx, Ghost, Ollama, and the Grafana stack) skip the cheap check entirely, since a single "upstream digest" comparison doesn't make sense when there's more than one upstream involved. -## Deliberate holdouts +## Deliberate Holdouts Not every stack is on this pattern, and that's a decision, not an oversight — knowing when *not* to apply a pattern uniformly is as much a design skill as building the pattern in the first place. This is the real breakdown, not a rounded estimate: @@ -57,7 +57,7 @@ mariadb-latest: firefly iii, kodi) pulls from our registry instead of Docker Hub. ``` -## Trigger discipline +## Trigger Discipline > [!WARNING] > This is schedule-triggered, not push-triggered — a deliberate choice, not a missing feature. Editing a stack's own Dockerfile doesn't by itself trigger a rebuild; it waits for the next weekly sweep to find the upstream base image has actually changed, or for someone to force a manual run via [`docs/orchestration.md`](orchestration.md). The tradeoff: a simpler mental model and no rebuild storm on every unrelated commit, at the real cost of same-day propagation for an urgent base-image CVE fix — accepted because a weekly cadence matches this fleet's actual risk tolerance, not because push-triggering would have been harder to build. diff --git a/docs/secret-distribution.md b/docs/secret-distribution.md index 948d46b..a498dc3 100644 --- a/docs/secret-distribution.md +++ b/docs/secret-distribution.md @@ -5,7 +5,7 @@ ![Secret distribution flows: Configuration Distribution Flow (orange) is the control plane that gates what Vault Agent renders; Certificate Flow and App Secret Flow (purple) are the data plane running through that same agent; Administrative Secret Flow (gray) is a genuinely separate system with no Vault involvement](../assets/secret-distribution-flows.svg) -## Why four flows instead of one +## Why Four Flows Instead Of One The obvious simpler design is one universal "secret goes in, secret comes out" pipeline. It was rejected because the four things being moved genuinely have different risk profiles and different failure tolerances: a certificate expiring is a slow-building problem measured in days; a missing application secret is often an immediate outage; a broken config-distribution push can silently stop *all* of a host's secrets from updating, not just one. Treating them identically would mean either over-engineering the simple cases or under-protecting the risky one. Four flows, each with its own explicit Law/Flow/Failure-behavior contract, is more code to maintain but far easier to reason about when something breaks — anyone debugging an incident knows immediately which contract governs the thing that's wrong. diff --git a/docs/secrets-and-pki.md b/docs/secrets-and-pki.md index 77e6322..824ac6f 100644 --- a/docs/secrets-and-pki.md +++ b/docs/secrets-and-pki.md @@ -10,17 +10,17 @@ | **Distribution** | Vault Agent on every consuming host, AppRole auth, renders to local disk | | **Replaces** | A prior plaintext-`.env`-in-git secrets model | -## Why Vault, and why self-hosted +## Why Vault, and Why Self-hosted A managed secrets service (a cloud provider's KMS/Secrets Manager) was a real alternative and was rejected for a specific reason: this fleet's PKI needs to issue certificates to purely internal hosts that have no business talking to a cloud API, and the two-zone reverse-proxy model in [`docs/dns-naming.md`](dns-naming.md) depends on an internal CA that's reachable without any external dependency. Vault does both certificate issuance and secret storage under one trust model instead of stitching together a separate CA tool and a separate secrets tool — fewer moving parts, one audit log, one policy language to reason about. -## What it does +## What It Does - **Certificate authority** — two PKI mounts: one for public-facing wildcard certs (consumed by the edge in [`docs/firewall-and-network.md`](firewall-and-network.md)), one for internal mTLS certs (consumed by the browser-facing reverse-proxy zone). Both issue short-lived certificates, not long-lived ones — a deliberate choice that shrinks the window a leaked cert is useful for. - **Secret store** — a KV v2 mount holds every migrated application's real secret values, one namespace per host, enforced by policy (below). - **Distribution** — Vault Agent runs on every fleet host, authenticates via AppRole, and renders both certificates and secrets to local disk. No application or service ever talks to Vault directly — this single rule is what makes the rest of the model tractable to reason about. -## Why it's built this way +## Why It's Built This Way Before this system existed, every stack's secrets sat in plaintext inside git-tracked `.env` files — readable by anyone with repo access, and permanently baked into git history even after being "fixed." SecureVault replaces that: @@ -80,7 +80,7 @@ None of these ever call the Vault API themselves — every one of them goes thro > } > ``` -## What's protected +## What's Protected | Layer | Protection | |---|---| @@ -94,7 +94,7 @@ This is the real hook script, not a description of it — the full guard logic, ![The real technitium-rebuild-p12.sh script: the 19-day-expired-cert incident documented in the script's own comments, the 7-day expiry guard, and the actual PKCS12 rebuild + dns.service restart](../assets/secrets-and-pki-cert-rebuild-hook.png) -## Access model +## Access Model One policy per host, scoped to exactly what that host needs — no host's credentials can read another host's secrets. Every host-scoped policy follows the same shape: a shared grant for lightweight self-registration, plus a private grant with full read/write access only to that host's own namespace. Onboarding a new host copies this shape rather than reinventing it — and the shared grant does *not* automatically imply the private one, so both get verified before assuming a host is fully provisioned. This same "shared + private" pattern shows up again in [`docs/iac-pipeline.md`](iac-pipeline.md)'s tag-driven Ansible model — a deliberate consistency, not a coincidence. diff --git a/docs/security-monitoring.md b/docs/security-monitoring.md index 714751e..627455b 100644 --- a/docs/security-monitoring.md +++ b/docs/security-monitoring.md @@ -30,14 +30,14 @@ ID: 012, Name: file-server, IP: any, Active ID: 013, Name: netbird, IP: any, Active ``` -## Why agent-based, and why self-hosted +## Why Agent-based, and Why Self-hosted The alternative to an agent on every host is a purely network-level monitoring approach — watching traffic at the firewall in [`docs/firewall-and-network.md`](firewall-and-network.md) and inferring what's happening inside each host. That approach can't see file integrity changes, local privilege escalation, or rootkit behavior — the things that matter most once an attacker is already past the network perimeter. An agent on every host closes that gap at the cost of real per-host overhead and a rollout that has to cover 100% of the fleet to be trustworthy — a partial rollout would create a false sense of coverage that's arguably worse than no agent at all. Self-hosting the manager/indexer/dashboard stack, rather than using a SaaS SIEM, keeps every alert's raw data (hostnames, internal paths, credential-adjacent metadata) inside the same trust boundary Vault protects in [`docs/secrets-and-pki.md`](secrets-and-pki.md), instead of shipping it to a third party. > [!TIP] > **The manager watches itself, as two independent things sharing one VM.** The manager/indexer/dashboard Docker stack and the native agent monitoring that same host are unrelated — the manager isn't self-aware, it's just also enrolled as agent `000`. And the two halves of "how Wazuh is configured" have different discipline levels on purpose: the compose stack itself is tracked in its own Gitea repo, but the actual detection rules and decoders live in a Docker volume, edited directly through the dashboard or manager API — not version-controlled. If the manager goes down entirely, agents don't lose events: they buffer locally and replay once it's back, so an outage costs visibility, not data. -## A fleet-wide bug this system caught in itself +## A Fleet-wide Bug This System Caught In Itself > [!IMPORTANT] > Wazuh's own default agent log-queue buffer (1024 entries) overflowed fleet-wide during Ansible orchestration bursts — enough hosts logging enough events at once that the queue filled before the agent could forward them, silently dropping data rather than erroring loudly. Found, not assumed: root-caused to the default queue size specifically, then fixed with a dedicated tuning role that raises `logcollector.queue_size` on every agent. That role is deliberately narrow — it only touches queue size and enabled/started state; syscheck ignores, rootcheck ignores, and other per-agent tuning are still one-off manual edits, honestly left uncovered rather than implied as handled by a role that doesn't actually reach them. This is the real root-cause comment from the fix itself, not a summary of it: @@ -52,11 +52,11 @@ The alternative to an agent on every host is a purely network-level monitoring a > # 9 of the fleet's agents. > ``` -## How an alert gets handled +## How An Alert Gets Handled ![An alert either matches a known, already-root-caused pattern and goes straight to the runbook with no re-investigation, or it doesn't and gets investigated fresh via wazuh-logtest and a direct index query — a fresh investigation that becomes routine gets folded into the runbook for next time](../assets/security-monitoring-alert-flow.svg) -## The curated "don't re-investigate this" runbook +## The Curated "don't re-investigate this" Runbook Rather than re-triaging the same recurring alert from scratch every time it fires, routine noise gets root-caused once and recorded — this is the difference between a SOC that actually scales past a handful of hosts and one that drowns in its own alert volume. A representative slice of what's actually in it, real rule IDs and all: @@ -72,7 +72,7 @@ Rather than re-triaging the same recurring alert from scratch every time it fire > [!TIP] > **One catch worth calling out in detail**: rule `19011`, "nftables default deny not detected," turned out to be a false positive *in the check itself*, not a real gap. The check greps for a raw nftables `policy drop` string that firewalld — the actual firewall in use, per [`docs/firewall-and-network.md`](firewall-and-network.md) — simply never produces, because firewalld implements default-deny through its own zone/chain structure instead. The fix was verifying the real firewall posture directly against firewalld's own state, not touching the firewall itself. This is a useful pattern in general: a security tool's own check can be wrong about a system it wasn't originally written for, and the fix is understanding the actual system, not chasing the tool's assumption. -## Querying and testing directly +## Querying And Testing Directly Alert data lives in a searchable OpenSearch-based index, reachable directly without going through the [Dashy SOC Board](dashy-soc-board.md) — a terms-aggregation on `rule.id`, with sub-aggregations for `rule.description`, max `rule.level`, `agent.name`, and max `timestamp`, returns a fleet-wide "what's actually firing and where" summary in one call — useful when the dashboard's own pre-aggregated view isn't granular enough for a specific question. @@ -101,7 +101,7 @@ $ echo 'Aug 28 12:00:00 caddy sshd[12345]: Failed password for invalid user test One rule-writing gotcha that's easy to get backwards: a rule's `` field has to reference the *parent* decoder's name, never a child regex-decoder further down the same chain — `wazuh-logtest` always reports the parent's name in its output regardless of which child actually matched, so debugging by what the tool prints back leads to the wrong name. And `wazuh-logtest` re-reads the rule/decoder files fresh on every invocation, independent of whether the live daemon has actually reloaded them — a rule that tests correctly can still be inert in production until `wazuh-control restart` actually loads it. -## Compliance hardening: justified exceptions, not blanket enforcement +## Compliance Hardening: justified exceptions, not blanket enforcement ![A CIS-oriented hardening framework of 9 tagged Ansible sub-roles applies cleanly to most hosts and gets enforced fleet-wide, re-applied every compliance run — but two controls are deliberately skipped with the reason documented: kernel forwarding flipped for Docker hosts since container networking needs it, and SSH TCP forwarding kept permissive fleet-wide since real remote access depends on it](../assets/security-monitoring-compliance-gate.svg) @@ -111,7 +111,7 @@ This is the same tag-driven selection model described in [`docs/iac-pipeline.md` The authentication stack is configured to fail **safe** under a specific failure mode rather than fail open — a subtler design choice than it sounds, and the kind of detail that's easy to get backwards under pressure while writing the config. -## Automated benchmarking: native, not containerized +## Automated Benchmarking: native, not containerized > [!IMPORTANT] > The fleet's automated security-benchmark tool (Docker Bench for Security) runs as a **native binary**, not through its own official container image — that image hit a real binary-compatibility wall between two different C library implementations (glibc vs. musl) that made it fail to run reliably. Running it natively instead of debugging a broken official container was the pragmatic call: the tool's job is auditing Docker, not testing container-runtime compatibility edge cases. It runs weekly, deliberately timed against real observed cron load from the other scheduled jobs described in [`docs/cron-scheduler.md`](cron-scheduler.md), rather than an arbitrary default schedule that might collide with heavier jobs. diff --git a/docs/sso-and-identity.md b/docs/sso-and-identity.md index edbd19e..500fac0 100644 --- a/docs/sso-and-identity.md +++ b/docs/sso-and-identity.md @@ -3,7 +3,7 @@ > [!NOTE] > Authentik is the identity provider for this fleet — self-hosted rather than an external IdP, for the same reason Vault is self-hosted in [`docs/secrets-and-pki.md`](secrets-and-pki.md): internal-only apps shouldn't need to trust a third party just to authenticate a login, and the two-zone reverse-proxy model in [`docs/dns-naming.md`](dns-naming.md) only works if the identity provider itself lives inside that same trust boundary. -## Why centralize authentication at all +## Why Centralize Authentication At All A homelab with dozens of self-hosted apps means dozens of different auth implementations if every app handles its own login — wildly inconsistent quality, no shared MFA, and revoking access means hunting down credentials app by app instead of doing it in one place. Centralizing on one identity provider fixes all three: every app's login gets the same enforcement regardless of how good or bad that app's own native auth actually is, and killing access anywhere starts with one action, not a checklist. @@ -63,11 +63,11 @@ $ curl -s -H "Authorization: Bearer [REDACTED]" \ > [!TIP] > **Removal is a real, deliberate step, not an afterthought.** Decommissioning an app means deleting the Application (by slug), then the Proxy Provider, then detaching that provider from the outpost's own provider list, then removing its icon file — in that order. Skipping the outpost-detach step is the easy mistake: the Application and Provider both look gone, but the outpost still references a provider that no longer resolves to anything, and that orphaned reference doesn't surface as an error anywhere obvious. -## Dual-zone registration +## Dual-zone Registration Every new app gets a Proxy Provider + Application pair registered per reverse-proxy zone it's actually exposed on — up to two registrations for the same app, one per zone from [`docs/dns-naming.md`](dns-naming.md), driven by actual need rather than registering every app on both zones by default. -## OIDC login: the redirect crosses zones, every time +## OIDC Login: the redirect crosses zones, every time For an app using native OIDC, the login always follows the same path: @@ -83,7 +83,7 @@ So an app's OIDC discovery/authority setting always points at the service-to-ser > [!NOTE] > **Not every app can take the OIDC path above.** Some — Jellyfin's auth plugin among them — only support LDAP: search the directory, then bind as whatever matched. Authentik's own LDAP outpost hides superuser accounts from that search by design, so the account that matters most can't authenticate this way for an app that requires it. The fix wasn't a second account or demoting the superuser — it was skipping LDAP for that app and reusing the same header-based forward-auth pattern the proxy-provider apps already use. -## Git access: least privilege, not the admin account +## Git Access: least privilege, not the admin account Automation should never run with more privilege than it needs. CI and orchestration pushes (the flows described in [`docs/orchestration.md`](orchestration.md)) run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. This wasn't the original setup: automated pushes used to run through the personal admin account used for interactive access, the same one with full admin rights over the Git server. That got fixed by standing up a separate account with its own SSH keypair and adding it as a write collaborator on every repo it touches — routine automated commits no longer run under an admin credential at all. Hosts that only need to *read* get separate read-only deploy keys instead of any shared credential. The practical payoff: if an automation credential ever leaks, the blast radius is "can push to specific repos," not "has full admin control of the Git server."