thesabear
62d1a508eb
Fixed one real inaccuracy (Gitea registry scope, Secret Distribution path count) and replaced generic descriptions with true, specific detail across all 14 Systems bullets - real incidents, real design decisions, concrete numbers - each checked against Trilium and, where relevant, the live Ansible roles rather than trusted at face value. Also stages a verified-but-unintegrated PAM hardening finding in docs/firewall-and-network.md for that file's own upcoming audit pass.
148 lines
15 KiB
Markdown
148 lines
15 KiB
Markdown
## About me
|
|
|
|
# 👋 Hi there — I'm @thesabear
|
|
[](https://www.linkedin.com/in/michael-surgey)
|
|
[](https://nomadianation.thesabear.net/)
|
|
<!--
|
|
[](https://x.com/nomadianation)
|
|
[](https://www.instagram.com/nomadianation)
|
|
[](https://www.tiktok.com/@nomadianation)
|
|
[](https://www.youtube.com/@nomadianation)
|
|
-->
|
|
|
|
<table style="margin: 0 auto;">
|
|
<tr>
|
|
<td align="center">🖥️<br/><strong style="font-size:1.6em;">11</strong><br/>fleet hosts</td>
|
|
<td align="center">🐳<br/><strong style="font-size:1.6em;">53</strong><br/>compose stacks</td>
|
|
<td align="center">📚<br/><strong style="font-size:1.6em;">15</strong><br/>documented systems</td>
|
|
<td align="center">🧱<br/><strong style="font-size:1.6em;">4</strong><br/>trust boundaries</td>
|
|
<td align="center">🔁<br/><strong style="font-size:1.6em;">13</strong><br/>automated maintenance flows</td>
|
|
</tr>
|
|
</table>
|
|
|
|
I'm a curious, self-taught technologist who runs a real, multi-host homelab as a production-grade playground. Most homelabs are a media server with some extra apps bolted on. This one is a platform: The same categories of systems a company's platform/security team owns, run at homelab scale, for real, 24/7, by one person. What started as a hobby project has grown into a full-on journey into DevOps, automation, infrastructure, and security;
|
|
|
|
- <strong style="font-size:1.4em;">Proxmox</strong> — a hypervisor cluster running a fleet of hardened VMs/LXCs provisioned via Terraform
|
|
- <strong style="font-size:1.4em;">Terraform + Packer</strong> — for bootstrapping new hosts for the fleet, from hardened, minimal images and continuously re-hardened by Ansible
|
|
- <strong style="font-size:1.4em;">HashiCorp Vault</strong> — issuing PKI certs and managing every application secret
|
|
- <strong style="font-size:1.4em;">Caddy + Authentik</strong> — a two-zone reverse-proxy design fronted by SSO, behind pfSense and firewalld for real network segmentation
|
|
- <strong style="font-size:1.4em;">Kestra</strong> — fleet orchestration with scheduled flows backed by Ansible roles and playbooks
|
|
- <strong style="font-size:1.4em;">Gitea</strong> — a self-hosted registry building and shipping container images for every stack that's opted in
|
|
- <strong style="font-size:1.4em;">Wazuh + Dashy</strong> — a self-built SOC watching all of it
|
|
|
|
## The Decision That Shows The Most
|
|
|
|
Most self-hosted setups have one trust boundary: "you're on the LAN, you're trusted." That's the easy answer, and it's wrong — a single compromised container can reach everything else with no further resistance. The other easy answer, mutual TLS on every single connection with no exceptions, is also wrong: real overhead, real certificate-lifecycle burden, on traffic that doesn't need it — exactly the kind of complexity that gets quietly disabled under pressure, which is worse than never building it.
|
|
|
|
This fleet runs **four trust boundaries, each calibrated to who's actually on the other end** — public internet traffic, human browsers on personal devices, machine-provisioned containers, and non-HTTP infra traffic each get a different control, matched to their real risk, not a blanket policy applied everywhere or nowhere. Enforced by **three independent routing authorities** instead of one central router, specifically so a failure or misconfiguration in any single one only ever takes out its own third of the traffic — never all of it at once. It didn't start this way, either: the public proxy used to be its own VM, and the internal service-to-service model used to be a flat, unauthenticated mesh with no boundary at all. Both got replaced deliberately, trading operational simplicity for a smaller blast radius.
|
|
|
|
Full breakdown — the boundary-by-boundary reasoning, the routing-authority failure table, all 8 traffic laws: [docs/architecture-overview.md](docs/architecture-overview.md).
|
|
|
|
This repo is a sanitized showcase of the systems behind that platform — the actual engineering decisions and tradeoffs, not just a list of installed apps. Every real IP, hostname, and internal domain has been stripped or genericized throughout.
|
|
|
|
My long-term goal is to build a career in IT — specifically in environments where learning never stops and innovation is part of the culture.
|
|
|
|
---
|
|
|
|
## 👀 What I'm Exploring
|
|
I use my homelab as a real-world playground to deepen my understanding of core IT concepts. Some of the areas I'm actively exploring include:
|
|
|
|
- Infrastructure-as-Code: provisioning and hardening hosts from scratch with Terraform + Packer + Ansible
|
|
- Reverse-proxy architecture and identity: a dual-zone Caddy setup (internal service-to-service vs. user-facing) fronted by Authentik forward-auth
|
|
- Workflow automation: Ansible for scheduled fleet maintenance, Kestra and n8n for everything else
|
|
- Monitoring, observability, and resilient service design
|
|
|
|
Each project teaches me something new, and everything I learn pushes me further toward becoming a well-rounded IT/DevOps engineer.
|
|
|
|
---
|
|
|
|
## 🌱 What I'm Learning Right Now
|
|
I'm currently focused on:
|
|
|
|
- Refining the Ansible roles and playbooks that keep the fleet patched and consistent
|
|
- Tightening the Caddy/Authentik reverse-proxy and SSO design as new services get added
|
|
- Running my own self-hosted Gitea instead of relying on external platforms
|
|
- Improving scripting skills — Python and Bash are next on my list
|
|
|
|
I enjoy understanding not just how something works, but how to automate it, secure it, and integrate it cleanly into a larger system.
|
|
|
|
---
|
|
|
|
## ⚡ Fun Fact
|
|
I'm entirely self-taught. Every tool, trick, concept, and workflow in my repos is something I learned by doing — from reading docs to breaking my lab at 2 a.m. and fixing it again.
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
## Architecture at a glance
|
|
|
|
|
|
Solid arrows are hard dependencies — the target has to be reachable for the source to function. Dashed arrows are secondary relationships (request routing like "proxies"/"auth", or async telemetry reporting) that don't block the source if the target's unavailable. Full breakdown of all 8 traffic laws (ingress, east-west, secrets, DNS, observability, notifications, CI/CD, management) with the actual trust-boundary reasoning: [docs/architecture-overview.md](docs/architecture-overview.md).
|
|
|
|
## Systems
|
|
|
|
### 🔐 Secrets & PKI
|
|
A self-hosted Vault cluster is both the certificate authority and the secret store for every host in the fleet — every TLS certificate, every database password, every API key traces back to this one system. Nothing reaches Vault directly; a local agent on each host authenticates itself and renders what that host needs to disk. Replaced an earlier setup where secrets sat in plaintext inside git. A DNS resolver's cert once renewed out of sync and ran expired for 19 days before anyone noticed — the fix wasn't tighter monitoring, it was making that renewal step structurally unable to skip itself again.
|
|
→ [docs/secrets-and-pki.md](docs/secrets-and-pki.md)
|
|
|
|
### 🔑 Secret Distribution Model
|
|
Three separate delivery paths move material out of Vault, purpose-built for what's actually being delivered — a short-lived certificate, an application secret, or a full config file — each with its own contract for what the rest of the fleet does if that specific path fails. Admin and tooling credentials deliberately live in a fourth, completely separate system instead — encrypted directly in git — since they're used by playbooks and operators, not consumed by a running service.
|
|
→ [docs/secret-distribution.md](docs/secret-distribution.md)
|
|
|
|
### 🪪 SSO & Identity
|
|
Authentik, self-hosted, fronts single sign-on fleet-wide — native OIDC where an app supports it, proxy-based forward-auth covering everything else, so apps that were never built with SSO in mind still get it. Every OIDC login crosses reverse-proxy zones mid-handshake, and Git automation runs on a scoped bot account rather than an admin identity.
|
|
→ [docs/sso-and-identity.md](docs/sso-and-identity.md)
|
|
|
|
### 🧱 Firewall & Network Segmentation
|
|
No blanket internal trust anywhere on the network — every real service-to-service flow gets its own explicit rule, and everything not explicitly allowed is logged and dropped by default. Enforced at two layers: a firewall at the network edge, and firewalld locally on every host, each opening only what that specific host actually needs. A few rules go narrower still — inbound access to the secrets platform's own admin API, for instance, is gated to only the two or three services that ever legitimately need it, not the whole fleet.
|
|
→ [docs/firewall-and-network.md](docs/firewall-and-network.md)
|
|
|
|
### 🌐 DNS
|
|
Two internal DNS zones split browser-facing traffic from service-to-service traffic — which zone a hostname resolves in is what decides which trust boundary a request enters, not just where it's found. Technitium runs both zones, self-hosted, with direct DNS bypass blocked at the firewall so the split is actually enforced, not just a naming convention. Not every app gets a record in both zones, either: something that's purely a backend service other apps talk to only needs the app-to-app zone, and something purely browser-facing only needs the user-facing one — each app gets exactly the DNS records its real traffic pattern uses.
|
|
→ [docs/dns-naming.md](docs/dns-naming.md)
|
|
|
|
### 📈 Observability
|
|
Every fleet host runs a native metrics-and-logs agent that pushes outward to a central Loki/Prometheus stack, rather than waiting to be scraped — chosen specifically so monitoring never needs inbound access opened through the firewall model to every host it watches. Logs, container output, and DNS query history all land in one searchable place, and a handful of apps go further still, exposing their own internal metrics directly into the same pipeline instead of relying on host-level visibility alone.
|
|
→ [docs/observability.md](docs/observability.md)
|
|
|
|
### 🛡️ Security Monitoring
|
|
A Wazuh agent runs on every host in the fleet, continuously watching file integrity, rootkit behavior, and compliance drift against a CIS-based benchmark. A curated runbook root-causes routine alerts once so they're never re-triaged from scratch, and hardening controls apply fleet-wide with specific, reasoned exceptions rather than uniform enforcement regardless of fit — audit logging, for one, is skipped on every LXC container in the fleet because the kernel capability it needs simply isn't available inside a container, not because it was overlooked.
|
|
→ [docs/security-monitoring.md](docs/security-monitoring.md)
|
|
|
|
### 🔁 Orchestration
|
|
Kestra is the orchestration engine that runs the fleet's scheduled maintenance — OS patching, container updates, app updates, disk cleanup, and internal KB upkeep. Most of it runs fully unattended, but updates to the secrets platform, DNS, and monitoring agent — the systems everything else depends on — wait for an explicit human approval before applying.
|
|
→ [docs/orchestration.md](docs/orchestration.md)
|
|
|
|
### 🔔 Notifications
|
|
Every alert in the fleet — from apps, from hosts, from security events — funnels through one Node-RED instance and one shared formatter before reaching Slack, so a notification reads the same regardless of which of a dozen-plus sources triggered it. A dedicated role puts every host onto the same pipeline for OS-level health, independent of any single application. One shared endpoint handles every app's webhook regardless of shape — a source field where an app provides one, structural fingerprinting of the payload where it doesn't — instead of standing up a dedicated integration per app.
|
|
→ [docs/notifications.md](docs/notifications.md)
|
|
|
|
### 📦 Self-Hosted Registry & Build Pipeline
|
|
Docker images get built and hosted entirely in-fleet on the Git server's own built-in registry — no runtime dependency on Docker Hub or ghcr for the majority of stacks, which have opted in. A weekly scheduled pass checks upstream image digests before committing to a full rebuild, so nothing gets rebuilt unless something upstream actually changed.
|
|
→ [docs/registry-and-build.md](docs/registry-and-build.md)
|
|
|
|
### 🖥️ Dashy SOC Board
|
|
One custom-built dashboard aggregates fourteen different data sources — security alerts, logs, metrics, CI health, secrets state, DNS, backups, firewall stats — into a single glanceable board, replacing what used to be fourteen separate tabs to check. Acknowledgement state lives entirely in the browser, no backend to host or secure. Every widget's visual shape matches the shape of its own data — a gauge for a proportion, a nested tree for parent/child hardware, a poster grid for media — instead of one generic card template stretched to fit everything.
|
|
→ [docs/dashy-soc-board.md](docs/dashy-soc-board.md)
|
|
|
|
### ⏱️ Cron Scheduler
|
|
Every scheduled job across the fleet — orchestration flows, system timers, app-internal schedules — lives in one inventory, visualized as a load heatmap weighted by actual resource impact — a disk-heavy backup job counts for more than a quick health check, not just raw job count — so a new job's time slot can be picked without colliding with existing load.
|
|
→ [docs/cron-scheduler.md](docs/cron-scheduler.md)
|
|
|
|
### 🏗️ Infrastructure as Code
|
|
Packer builds a hardened base image, Terraform provisions each VM from it, and Ansible configures and continuously re-hardens every host after that — three tools, three separate responsibilities, connected by one tag that threads through all three stages instead of one tool trying to do every job. Terraform's part in that link ends at creation — it writes the tags file once, and that file then drives Ansible independently for the rest of that host's life.
|
|
→ [docs/iac-pipeline.md](docs/iac-pipeline.md)
|
|
|
|
### 🧯 Disaster Recovery & Upgrades
|
|
Three independent recovery layers cover three different kinds of loss — a lost host rebuilds from code, VM-level backups cover full-host snapshots, and a separate volume-level backup covers application data specifically. Failure-domain mapping goes down to the disk: one specific volume, if lost, would take out the boot images for two hosts at once — identified and factored into backup priority rather than discovered the hard way. OS-level and application-level upgrades run on entirely separate, deliberately decoupled schedules and safety checks.
|
|
→ [docs/disaster-recovery.md](docs/disaster-recovery.md)
|
|
|
|
## What's running
|
|
|
|
53 Docker Compose stacks, grouped by purpose rather than by which host happens to run them — see [`stacks/`](stacks/): identity & security, platform/infra, media, productivity, and dev tools.
|
|
|
|
--- |