thesabear's Homelab

A real, multi-host homelab run as a production-grade platform — not a pile of docker run commands. A Proxmox hypervisor cluster hosts a fleet of hardened VMs/LXCs; a self-hosted Vault cluster issues PKI certs and distributes secrets to every consumer; a dual-zone reverse-proxy design (one zone for browsers, one for service-to-service traffic) sits behind SSO; Kestra-driven automation handles fleet maintenance on a schedule with a human approval gate; and a self-built SOC stack (Wazuh + Grafana + a custom dashboard) watches all of it.

This repo is a sanitized showcase of the systems behind that platform — the actual engineering decisions and tradeoffs, not just a list of installed apps. Every real IP, hostname, and internal domain has been stripped or genericized throughout.

11 fleet hosts

55 compose stacks

16 documented systems

4 trust boundaries

13 automated maintenance flows

Architecture at a glance

flowchart LR
    Internet(["Internet"]) --> Edge["Edge Firewall"]
    Edge --> LAN["LAN"]

    LAN --> DNS["Internal DNS<br/>two zones"]
    LAN --> Vault["Vault<br/>PKI + secrets"]
    DNS --> Vault

    Vault --> Apps["App Host<br/>41 stacks"]
    Vault --> Automation["Automation VM<br/>Git + orchestration"]
    LAN --> FileServer["File Server"]
    DNS --> FileServer
    LAN --> Automation

    Apps --> InternalProxy["Internal Proxy Zone"]
    Apps --> SSO["SSO"]

    Vault --> EdgeProxy["User-Facing Proxy<br/>mTLS"]
    DNS --> EdgeProxy
    Apps -.proxied.-> EdgeProxy
    SSO -.auth.-> EdgeProxy

    Apps -.-> Monitoring["Monitoring"]
    Automation -.-> Monitoring
    FileServer -.-> Monitoring
    EdgeProxy -.-> Monitoring

    Apps -.-> SecMon["Security Monitoring"]
    Automation -.-> SecMon
    FileServer -.-> SecMon
    EdgeProxy -.-> SecMon

    classDef edge fill:#f8cecc,stroke:#b85450,color:#333
    classDef core fill:#dae8fc,stroke:#6c8ebf,color:#333
    classDef obs fill:#fff2cc,stroke:#d6b656,color:#333
    class Internet,Edge edge
    class LAN,DNS,Vault,Apps,Automation,FileServer,InternalProxy,SSO,EdgeProxy core
    class Monitoring,SecMon obs

Solid arrows are real dependencies; dotted arrows are asynchronous telemetry reporting, not a blocking dependency. Full breakdown of all 8 traffic laws (ingress, east-west, secrets, DNS, observability, notifications, CI/CD, management) with the actual trust-boundary reasoning: docs/architecture-overview.md.

Every system, at a glance

System	The core decision	Doc
🔐 Secrets & PKI	One Vault cluster as CA + secret store, zero plaintext secrets in git	→
🔑 Secret Distribution	Four distinct flows, each with its own failure contract	→
🪪 SSO & Identity	Dual-zone SSO registration, header-based fallback for a real LDAP limitation	→
🧱 Firewall & Network	Zero blanket-allow rules — every flow explicit, everything else dropped	→
🌐 DNS	Two zones decide which trust boundary a request enters	→
📈 Observability	Push-based metrics, chosen specifically to fit the firewall model	→
🛡️ Security Monitoring	Curated alert runbook + justified compliance exceptions	→
🔁 Orchestration	13 flows, check → pause → human approval → apply	→
🔔 Notifications	Two fan-in branches, one shared formatter, zero runtime coupling	→
📦 Registry & Build	Self-hosted registry, digest-checked before any rebuild	→
🖥️ Dashy SOC Board	14 data sources aggregated into one glanceable board	→
⏱️ Cron Scheduler	Fleet-wide job inventory, weighted load heatmap	→
📝 Content Pipeline	In-progress, shown honestly rather than hidden	→
🏗️ Infrastructure as Code	Packer → Terraform → Ansible, one tag threading all three	→
🧯 Disaster Recovery	3 independent recovery layers, one real documented gap	→
📐 Architecture Overview	4 trust boundaries, 3 routing authorities, calibrated not maxed	→

Systems

🔐 Secrets & PKI

One Vault cluster serves as both the internal certificate authority and the secret store — no separate PKI tool bolted on. It replaced an earlier plaintext-.env-in-git secrets model with AppRole-based Vault Agent distribution: every consuming host authenticates itself and renders secrets locally, nothing calls the Vault API directly. A non-expiring root token is tracked as the single biggest known risk, ranked and reasoned about rather than ignored. → docs/secrets-and-pki.md

🔑 Secret Distribution Model

Four distinct flows — certificate issuance, per-app secret rendering, config-file distribution, and admin/provisioning-time secrets — each with an explicit contract for what happens if it fails. Two genuinely different secret mechanisms exist side by side on purpose, rather than forcing every use case through one system. → docs/secret-distribution.md

🪪 SSO & Identity

Every app registers against the SSO provider twice — once per reverse-proxy zone — because authentication always happens on the user-facing zone even when the app itself lives on the internal one. A real platform limitation is documented rather than papered over: the SSO provider's LDAP outpost can't do search-then-bind auth for superuser accounts, so that path uses header-based forward-auth instead. Git automation follows least privilege — a scoped bot account with write access replaces routing CI through an admin identity. → docs/sso-and-identity.md

🧱 Firewall & Network Segmentation

No blanket "anything on the LAN can talk to anything else on the LAN" rule. Every real service-to-service flow gets its own explicit allow rule; everything else is logged and dropped. Firewall objects are managed through named group aliases, not by hand-editing raw addresses into every rule. A real, documented gap: container traffic bypasses the host firewall entirely, so per-host firewall rules only actually protect native, non-containerized services. → docs/firewall-and-network.md

🌐 DNS

Two separate internal DNS zones by design — one for browser-facing apps (mutual TLS enforced), one for service-to-service traffic — deliberately never conflated. A later audit caught and corrected real documentation drift: an old note claimed an external DNS forwarder was in use that, in practice, wasn't. → docs/dns-naming.md

📈 Observability

Metrics are pushed, not scraped — a deliberate choice made to avoid punching scrape-target holes through the firewall model above. The collection agent runs as a native systemd service on every host (not a container) and is installed by a compliance-tagged automation role that self-heals on drift, not just applied once at setup. → docs/observability.md

🛡️ Security Monitoring

A curated "known-alert" runbook exists specifically so routine noise doesn't get re-investigated from scratch every time it fires — every entry documents its actual root cause. Compliance hardening is rolled out fleet-wide with deliberate, justified exceptions (a control that simply can't apply inside a container gets skipped there, not blanket-forced), and the automated security benchmark tool runs natively after hitting a real binary-compatibility wall with its own official container image. → docs/security-monitoring.md

🔁 Orchestration

Thirteen scheduled flows handle fleet maintenance end to end, built around a check → pause → require human approval → apply pattern instead of blind automatic changes. Different flows deliberately authenticate as different service identities depending on what they need to touch locally on the control node. → docs/orchestration.md

🔔 Notifications

Two independent paths — a message-queue branch and an HTTP webhook branch — both converge on one automation flow and a shared formatter, so every app's alert reads the same regardless of source. Documented honestly rather than hidden: a decommissioned app's routing code is still present, and a credential that's supposed to live in one place is known to actually be synced across three. → docs/notifications.md

📦 Self-Hosted Registry & Build Pipeline

Runs on the self-hosted Git server's own built-in container registry — no separate registry service to operate. Rebuilds trigger on a weekly schedule against upstream image digests rather than on every push, with a cheap digest check run first before committing to a full rebuild. → docs/registry-and-build.md

🖥️ Dashy SOC Board

One dashboard aggregates roughly a dozen distinct data sources — security alerts, logs, metrics, CI status, secrets health, DNS, backups, and more — into a single glanceable board. Acknowledgement state lives entirely client-side, no backend required. → docs/dashy-soc-board.md

⏱️ Cron Scheduler

Every scheduled job across the fleet is tracked in one inventory and visualized as a load heatmap, weighted by actual resource impact rather than raw job count. → docs/cron-scheduler.md

📝 Content Pipeline

An in-progress pipeline connecting a structured content tracker through an automation tool out to a blog and file storage — documented honestly as unfinished rather than presented as done. → docs/content-pipeline.md

🏗️ Infrastructure as Code

Three tools, three responsibilities: image builds, provisioning, and configuration are kept as separate layers instead of one monolithic script. → docs/iac-pipeline.md

🧯 Disaster Recovery & Upgrades

Three independent recovery layers — infrastructure-as-code, VM/LXC-level backups, and application-data backups — each cover a different kind of loss, and a real known gap (one data pool covered by neither) is documented rather than assumed handled. OS-level and application-level upgrades are deliberately decoupled systems; they don't share a rolling-batch mechanism, because doing so would let one host's expected no-op result silently abort every other host's run. → docs/disaster-recovery.md

What's running

~55 Docker Compose stacks, grouped by purpose rather than by which host happens to run them — see stacks/: identity & security, platform/infra, media, productivity, and dev tools.

---

About me

Self-taught, building toward a career in IT/DevOps — everything in this repo is something I learned by doing, from reading docs to breaking my own lab and fixing it again.