thesabear
a4c334b3d8
Keep LinkedIn and Website visible; the rest stay in the file, just hidden, not deleted.
139 lines
10 KiB
Markdown
139 lines
10 KiB
Markdown
# thesabear's Homelab
|
|
|
|
A real, multi-host homelab run as a production-grade platform — not a pile of `docker run` commands. A Proxmox hypervisor cluster hosts a fleet of hardened VMs/LXCs; a self-hosted Vault cluster issues PKI certs and distributes secrets to every consumer; a dual-zone reverse-proxy design (one zone for browsers, one for service-to-service traffic) sits behind SSO; Kestra-driven automation handles fleet maintenance on a schedule with a human approval gate; and a self-built SOC stack (Wazuh + Grafana + a custom dashboard) watches all of it.
|
|
|
|
This repo is a sanitized showcase of the systems behind that platform — the actual engineering decisions and tradeoffs, not just a list of installed apps. Every real IP, hostname, and internal domain has been stripped or genericized throughout.
|
|
|
|
## Architecture at a glance
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
Internet(["Internet"])
|
|
Edge["Edge Firewall<br/>NAT · public reverse proxy · NTP · DHCP"]
|
|
LAN["LAN"]
|
|
DNS["Internal DNS<br/>two zones: browser-facing / service-to-service"]
|
|
Vault["Vault<br/>PKI + secret store"]
|
|
|
|
Apps["App Host (LXC)<br/>~40 Docker Compose stacks"]
|
|
InternalProxy["Internal reverse-proxy zone<br/>app-to-app HTTPS"]
|
|
SSO["SSO / forward-auth"]
|
|
|
|
FileServer["File Server<br/>NFS · SMB"]
|
|
Automation["Automation VM<br/>Git hosting · orchestration"]
|
|
EdgeProxy["User-facing reverse proxy<br/>mTLS enforced"]
|
|
Monitoring["Monitoring<br/>metrics · logs"]
|
|
SecMonitoring["Security Monitoring<br/>SOC stack"]
|
|
|
|
Internet --> Edge --> LAN
|
|
LAN --> DNS
|
|
LAN --> Vault
|
|
DNS --> Vault
|
|
|
|
Vault --> Apps
|
|
LAN --> FileServer
|
|
DNS --> FileServer
|
|
Vault --> Automation
|
|
LAN --> Automation
|
|
|
|
Apps --> InternalProxy
|
|
Apps --> SSO
|
|
|
|
Vault --> EdgeProxy
|
|
DNS --> EdgeProxy
|
|
Apps -. proxied by .-> EdgeProxy
|
|
SSO -. auth gate via .-> EdgeProxy
|
|
|
|
Apps -. reports to .-> Monitoring
|
|
Automation -. reports to .-> Monitoring
|
|
FileServer -. reports to .-> Monitoring
|
|
EdgeProxy -. reports to .-> Monitoring
|
|
|
|
Apps -. reports to .-> SecMonitoring
|
|
Automation -. reports to .-> SecMonitoring
|
|
FileServer -. reports to .-> SecMonitoring
|
|
EdgeProxy -. reports to .-> SecMonitoring
|
|
```
|
|
|
|
Full traffic-law breakdown (ingress, east-west, secrets, DNS, observability, notifications, CI/CD, and management planes, each with its own failure behavior): [docs/architecture-overview.md](docs/architecture-overview.md).
|
|
|
|
## Systems
|
|
|
|
### 🔐 Secrets & PKI
|
|
One Vault cluster serves as both the internal certificate authority and the secret store — no separate PKI tool bolted on. It replaced an earlier plaintext-`.env`-in-git secrets model with AppRole-based Vault Agent distribution: every consuming host authenticates itself and renders secrets locally, nothing calls the Vault API directly. A non-expiring root token is tracked as the single biggest known risk, ranked and reasoned about rather than ignored.
|
|
→ [docs/secrets-and-pki.md](docs/secrets-and-pki.md)
|
|
|
|
### 🔑 Secret Distribution Model
|
|
Four distinct flows — certificate issuance, per-app secret rendering, config-file distribution, and admin/provisioning-time secrets — each with an explicit contract for what happens if it fails. Two genuinely different secret mechanisms exist side by side on purpose, rather than forcing every use case through one system.
|
|
→ [docs/secret-distribution.md](docs/secret-distribution.md)
|
|
|
|
### 🪪 SSO & Identity
|
|
Every app registers against the SSO provider twice — once per reverse-proxy zone — because authentication always happens on the user-facing zone even when the app itself lives on the internal one. A real platform limitation is documented rather than papered over: the SSO provider's LDAP outpost can't do search-then-bind auth for superuser accounts, so that path uses header-based forward-auth instead. Git automation follows least privilege — a scoped bot account with write access replaces routing CI through an admin identity.
|
|
→ [docs/sso-and-identity.md](docs/sso-and-identity.md)
|
|
|
|
### 🧱 Firewall & Network Segmentation
|
|
No blanket "anything on the LAN can talk to anything else on the LAN" rule. Every real service-to-service flow gets its own explicit allow rule; everything else is logged and dropped. Firewall objects are managed through named group aliases, not by hand-editing raw addresses into every rule. A real, documented gap: container traffic bypasses the host firewall entirely, so per-host firewall rules only actually protect native, non-containerized services.
|
|
→ [docs/firewall-and-network.md](docs/firewall-and-network.md)
|
|
|
|
### 🌐 DNS
|
|
Two separate internal DNS zones by design — one for browser-facing apps (mutual TLS enforced), one for service-to-service traffic — deliberately never conflated. A later audit caught and corrected real documentation drift: an old note claimed an external DNS forwarder was in use that, in practice, wasn't.
|
|
→ [docs/dns-naming.md](docs/dns-naming.md)
|
|
|
|
### 📈 Observability
|
|
Metrics are pushed, not scraped — a deliberate choice made to avoid punching scrape-target holes through the firewall model above. The collection agent runs as a native systemd service on every host (not a container) and is installed by a compliance-tagged automation role that self-heals on drift, not just applied once at setup.
|
|
→ [docs/observability.md](docs/observability.md)
|
|
|
|
### 🛡️ Security Monitoring
|
|
A curated "known-alert" runbook exists specifically so routine noise doesn't get re-investigated from scratch every time it fires — every entry documents its actual root cause. Compliance hardening is rolled out fleet-wide with deliberate, justified exceptions (a control that simply can't apply inside a container gets skipped there, not blanket-forced), and the automated security benchmark tool runs natively after hitting a real binary-compatibility wall with its own official container image.
|
|
→ [docs/security-monitoring.md](docs/security-monitoring.md)
|
|
|
|
### 🔁 Orchestration
|
|
Thirteen scheduled flows handle fleet maintenance end to end, built around a check → pause → require human approval → apply pattern instead of blind automatic changes. Different flows deliberately authenticate as different service identities depending on what they need to touch locally on the control node.
|
|
→ [docs/orchestration.md](docs/orchestration.md)
|
|
|
|
### 🔔 Notifications
|
|
Two independent paths — a message-queue branch and an HTTP webhook branch — both converge on one automation flow and a shared formatter, so every app's alert reads the same regardless of source. Documented honestly rather than hidden: a decommissioned app's routing code is still present, and a credential that's supposed to live in one place is known to actually be synced across three.
|
|
→ [docs/notifications.md](docs/notifications.md)
|
|
|
|
### 📦 Self-Hosted Registry & Build Pipeline
|
|
Runs on the self-hosted Git server's own built-in container registry — no separate registry service to operate. Rebuilds trigger on a weekly schedule against upstream image digests rather than on every push, with a cheap digest check run first before committing to a full rebuild.
|
|
→ [docs/registry-and-build.md](docs/registry-and-build.md)
|
|
|
|
### 🖥️ Dashy SOC Board
|
|
One dashboard aggregates roughly a dozen distinct data sources — security alerts, logs, metrics, CI status, secrets health, DNS, backups, and more — into a single glanceable board. Acknowledgement state lives entirely client-side, no backend required.
|
|
→ [docs/dashy-soc-board.md](docs/dashy-soc-board.md)
|
|
|
|
### ⏱️ Cron Scheduler
|
|
Every scheduled job across the fleet is tracked in one inventory and visualized as a load heatmap, weighted by actual resource impact rather than raw job count.
|
|
→ [docs/cron-scheduler.md](docs/cron-scheduler.md)
|
|
|
|
### 📝 Content Pipeline
|
|
An in-progress pipeline connecting a structured content tracker through an automation tool out to a blog and file storage — documented honestly as unfinished rather than presented as done.
|
|
→ [docs/content-pipeline.md](docs/content-pipeline.md)
|
|
|
|
### 🏗️ Infrastructure as Code
|
|
Three tools, three responsibilities: image builds, provisioning, and configuration are kept as separate layers instead of one monolithic script.
|
|
→ [docs/iac-pipeline.md](docs/iac-pipeline.md)
|
|
|
|
### 🧯 Disaster Recovery & Upgrades
|
|
Three independent recovery layers — infrastructure-as-code, VM/LXC-level backups, and application-data backups — each cover a different kind of loss, and a real known gap (one data pool covered by neither) is documented rather than assumed handled. OS-level and application-level upgrades are deliberately decoupled systems; they don't share a rolling-batch mechanism, because doing so would let one host's expected no-op result silently abort every other host's run.
|
|
→ [docs/disaster-recovery.md](docs/disaster-recovery.md)
|
|
|
|
## What's running
|
|
|
|
~55 Docker Compose stacks, grouped by purpose rather than by which host happens to run them — see [`stacks/`](stacks/): identity & security, platform/infra, media, productivity, and dev tools.
|
|
|
|
---
|
|
|
|
## About me
|
|
|
|
Self-taught, building toward a career in IT/DevOps — everything in this repo is something I learned by doing, from reading docs to breaking my own lab and fixing it again.
|
|
|
|
[](https://www.linkedin.com/in/michael-surgey)
|
|
<!--
|
|
[](https://x.com/nomadianation)
|
|
[](https://www.instagram.com/nomadianation)
|
|
[](https://www.tiktok.com/@nomadianation)
|
|
[](https://www.youtube.com/@nomadianation)
|
|
-->
|
|
[](https://nomadianation.thesabear.net/)
|