thesabear/public
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: thesabear/public:main
thesabear/public:main thesabear/public:iac-history-ids
..
compare: thesabear/public:77ffc18b1a4de3d640986d873c5b1832315b8f48
thesabear/public:main thesabear/public:iac-history-ids
3 Commits
main .. 77ffc18b1a
Author SHA1 Message Date
automation-bot 77ffc18b1a docs: fix disaster-recovery.md drift (Kopia stack count, stale evidence) 
Kopia's coverage claim conflated the public repo's curated stacks/
folder (a documentation subset) with the real live stack count on
rocky-docker - fixed to the actual live figure (41, cross-checked
against docker_stacks.yml and the real repo directory count), same
fix applied to the source KB note which had its own stale ~38 figure
and an incorrect hyper1 backup mode (suspend -> stop, confirmed
against live /etc/pve/jobs.cfg). Refreshed the ls/df and
roles_metadata.yml command-output evidence to match current state.
 2026-09-18 23:26:42 +03:00
automation-botandClaude Sonnet 5 79b295d0f1 docs: fix cron-scheduler.md job-count drift (50->49, phantom PBS prune entry) 
Live NocoDB now has 49 jobs/14 without a fixed time, not 50/15 - traced
to a removed inventory row for a "PBS prune job" that never actually
existed (confirmed against live data and the KB note, which states PBS
runs no prune job at all). Updated counts and refreshed all three
screenshots to match current state.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
 2026-09-18 23:20:34 +03:00
automation-botandClaude Sonnet 5 bb834ca217 docs: fix architecture-overview.md drift (WAN rule count, LAN enforcement mechanism) 
Live-verification found two real inaccuracies: WAN now passes 4 things
inbound (NetBird's STUN forward was added, undocumented), and the "third
fix" LAN-to-LAN narrative credited pfSense's ruleset when the actual
enforcement for host-to-host traffic is each host's own firewalld (pfSense
never sees same-subnet traffic on this flat LAN). Refreshed the WAN rules
screenshot and reworded the boundary table + narrative to match. Also
cleaned up the "Other Six Traffic Laws" table formatting per feedback.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
 2026-09-18 11:48:37 +03:00
32 changed files with 66 additions and 1216 deletions
Expand all files Collapse all files

No files matched your search

.gitea/workflows/ci.yml
-15
View File
@@ -1,15 +0,0 @@
# CI for this repo - the actual checks live in the shared ci-workflows repo.
name: ci
"on":
push:
branches: [main]
pull_request:
workflow_dispatch:
jobs:
gitleaks:
# Short owner/repo form: Gitea resolves reusable workflows server-side and
# rejects absolute URLs that don't match its own ROOT_URL.
uses: thesabear/ci-workflows/.gitea/workflows/gitleaks.yml@main
secrets: inherit
.gitleaksignore
-7
View File
@@ -1,7 +0,0 @@
# Findings checked by hand and confirmed dead - each line ignores exactly one
# finding (commit:file:rule:line). Anything else gitleaks finds still fails CI.
# Paperless OIDC client secret in old history (removed from the files 2026-07-06).
# Checked 2026-10-11: that client_id no longer exists in Authentik (all 104
# OAuth2 providers), and Paperless uses a different client and secret.
973546647df3fe505e913c42af9ed3547083593c:Docker/Paperless-NGX/docker-compose.yaml:generic-api-key:50
README.md
-10
View File
@@ -93,10 +93,6 @@ Every physical host, VM/LXC, and running service in the fleet, re-derived from t
Packer builds a hardened base image, Terraform provisions each VM from it, and Ansible configures and continuously re-hardens every host after that — three tools, three separate responsibilities, connected by one tag that threads through all three stages instead of one tool trying to do every job. Terraform's part in that link ends at creation — it writes the tags file once, and that file then drives Ansible independently for the rest of that host's life.
→ [docs/iac-pipeline.md](docs/iac-pipeline.md)
### 🎧 Kodi <-> Music Assistant Smart Queue Bridge
Kodi stays the only music player, UI, and ratings authority; Music Assistant runs invisibly behind it as a similarity engine, never a player, never writing anything back to the library. A small daemon watches Kodi's live queue and auto-refills it with similar tracks via MA's recommendation API when it runs low. Three real issues surfaced only once it was tested against a live Kodi session — not exercised in isolation — including one where the daemon had silently never worked at all.
→ [docs/kodi-music-assistant-bridge.md](docs/kodi-music-assistant-bridge.md)
### 🔔 Notifications
Every alert in the fleet — from apps, from hosts, from security events — funnels through one Node-RED instance and one shared formatter before reaching Slack, so a notification reads the same regardless of which of a dozen-plus sources triggered it. A dedicated role puts every host onto the same pipeline for OS-level health, independent of any single application. One shared endpoint handles every app's webhook regardless of shape — a source field where an app provides one, structural fingerprinting of the payload where it doesn't — instead of standing up a dedicated integration per app.
→ [docs/notifications.md](docs/notifications.md)
@@ -139,12 +135,6 @@ A self-hosted WireGuard mesh gives every fleet host and personal device its own
---
## Cheatsheets
Quick-reference material for tools I use day-to-day — command references and config notes, not full write-ups. See [`cheatsheets/`](cheatsheets/).
---
## Scripts & Tooling
A few standalone scripts live in [`projects/`](projects/) alongside the Docker stacks — real tooling, not demo code:
assets/gitea-deploy-keys.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 80 KiB

assets/iac-pipeline-ansible-stages.svg
+1 -1
View File
@@ -1,4 +1,4 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- Do not edit this file with editors other than draw.io -->
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="932px" height="318px" viewBox="0 0 932 318" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA; &lt;diagram name=&quot;Ansible Stages&quot; id=&quot;iac-ansiblestages-1&quot;&gt;&#xA; &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1080&quot; pageHeight=&quot;360&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA; &lt;root&gt;&#xA; &lt;mxCell id=&quot;0&quot; /&gt;&#xA; &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA; &lt;mxCell id=&quot;tags&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Per-host tag list&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;from Terraform&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;70&quot; width=&quot;200&quot; x=&quot;20&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;init&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Initialization stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;runs once, at creation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;20&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;prov&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Provisioning stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;every run — 2 roles today&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;comp&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Compliance stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;re-applied every run&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;sub&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;9 independently-tagged&amp;lt;br&amp;gt;hardening sub-roles&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;720&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;init&quot;&gt;&#xA; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;prov&quot;&gt;&#xA; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;comp&quot;&gt;&#xA; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e4&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;comp&quot; style=&quot;html=1;endArrow=block;strokeColor=#82b366;strokeWidth=2;rounded=1;&quot; target=&quot;sub&quot;&gt;&#xA; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometrLine truncated
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="932px" height="318px" viewBox="0 0 932 318" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10; &lt;diagram name=&quot;Ansible Stages&quot; id=&quot;iac-ansiblestages-1&quot;&gt;&#10; &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1080&quot; pageHeight=&quot;360&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10; &lt;root&gt;&#10; &lt;mxCell id=&quot;0&quot; /&gt;&#10; &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10; &lt;mxCell id=&quot;tags&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Per-host tag list&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;from Terraform&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;70&quot; width=&quot;200&quot; x=&quot;20&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;init&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Initialization stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;runs once, at creation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;20&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;prov&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Provisioning stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;every run — 1 role today, one host&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;comp&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Compliance stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;re-applied every run&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;sub&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;9 independently-tagged&amp;lt;br&amp;gt;hardening sub-roles&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;720&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;init&quot;&gt;&#10; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;prov&quot;&gt;&#10; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;comp&quot;&gt;&#10; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;e4&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;comp&quot; style=&quot;html=1;endArrow=block;strokeColor=#82b366;strokeWidth=2;rounded=1;&quot; target=&quot;sub&quot;&gt;&#10; &lt;mxGeometry relative=&quot;1&quot; as=&quoLine truncated
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 118 KiB

After

Width:  |  Height:  |  Size: 153 KiB

assets/kodi-music-assistant-bridge-live-queue.png
BIN
View File
Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 430 KiB

assets/kodi-music-assistant-bridge-refill-cycle.svg
-4
View File
@@ -1,4 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- Do not edit this file with editors other than draw.io -->
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="624px" height="632px" viewBox="0 0 624 632" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA; &lt;diagram name=&quot;Refill Cycle&quot; id=&quot;0&quot;&gt;&#xA; &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;0&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;900&quot; pageHeight=&quot;700&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA; &lt;root&gt;&#xA; &lt;mxCell id=&quot;0&quot; /&gt;&#xA; &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA; &lt;mxCell id=&quot;poll&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;&quot; value=&quot;Poll each Kodi instance&amp;lt;br&amp;gt;(every 240s)&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;active&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;&quot; value=&quot;Active player + queue&amp;lt;br&amp;gt;length on this instance&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;150&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;gate&quot; parent=&quot;1&quot; style=&quot;rhombus;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=13;&quot; value=&quot;&amp;lt;font style=&amp;quot;font-size: 11px;&amp;quot;&amp;gt;Remaining &amp;amp;gt;&amp;lt;br&amp;gt;REFILL_THRESHOLD (5)?&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;90&quot; width=&quot;190&quot; x=&quot;355&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;resolve&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=13;&quot; value=&quot;Resolve current track&amp;lt;br&amp;gt;in MA (music/search)&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;400&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;similar&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=13;&quot; value=&quot;Query similar_tracks&amp;lt;br&amp;gt;(large candidate pool)&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;500&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;inject&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;&quot; value=&quot;Filter + translate paths,&amp;lt;br&amp;gt;Playlist.Add up to 10&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;600&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;poll&quot; style=&quot;edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#333333;&quot; target=&quot;active&quot;&gt;&#xA; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;active&quot; style=&quot;edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#333333;&quot; target=&quot;gate&quot;&gt;&#xA; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;gate&quot; style=&quot;edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;exitX=1;exitY=0.5;labelBackgroundColor=#1C1D1E;exitDx=0;exitDy=0;&quot; target=&quot;poll&quot; value=&quot;&amp;lt;span style=&amp;quot;background-color: light-dark(rgb(255, 255, 255), rgb(28, 29, 30));&amp;quot;&amp;gt;yes &amp;amp;mdash; skip&amp;lt;/span&amp;gt;&quot;&gt;&#xA; &lt;mxGeometry relative=&quot;1&quot; x=&quot;0.0698&quot; y=&quot;-33&quot; as=&quot;geometry&quot;&gt;&#xA; &lt;mxPoint as=&quot;offset&quot; /&gt;&#xA; &lt;Array as=&quot;points&quot;&gt;&#xA; &lt;mxPoint x=&quot;60Line truncated
Side by Side

Before

Width:  |  Height:  |  Size: 162 KiB

assets/observability-log-fanout.svg
+1 -1
View File
@@ -1,4 +1,4 @@
<?xml version="1.0" encoding="UTF-8"?>
<!-- Do not edit this file with editors other than draw.io -->
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="373px" viewBox="0 0 872 373" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA; &lt;diagram name=&quot;Log Fanout&quot; id=&quot;obs-logfanout-1&quot;&gt;&#xA; &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;420&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA; &lt;root&gt;&#xA; &lt;mxCell id=&quot;0&quot; /&gt;&#xA; &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA; &lt;mxCell id=&quot;a1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;system logs&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;b1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=system&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a1&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b1&quot;&gt;&#xA; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;a2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker API&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;container logs, survives recreation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;b2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=docker&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a2&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b2&quot;&gt;&#xA; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;a3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;DNS query log&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;daily rotation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;b3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=technitium-dns, real event time&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA; &lt;/mxCell&gt;&#xA; &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a3&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; targLine truncated
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="273px" viewBox="0 0 872 273" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10; &lt;diagram name=&quot;Log Fanout&quot; id=&quot;obs-logfanout-1&quot;&gt;&#10; &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;320&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10; &lt;root&gt;&#10; &lt;mxCell id=&quot;0&quot; /&gt;&#10; &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10; &lt;mxCell id=&quot;a1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;system logs&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;b1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=system&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a1&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b1&quot;&gt;&#10; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;a2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker API&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;container logs, survives recreation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;b2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=docker&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a2&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b2&quot;&gt;&#10; &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;a3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;DNS query log&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;daily rotation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;b3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=dns, real event time&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10; &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10; &lt;/mxCell&gt;&#10; &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a3&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b3Line truncated
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 136 KiB

After

Width:  |  Height:  |  Size: 135 KiB

cheatsheets/README.md
-13
View File
@@ -1,13 +0,0 @@
# Cheatsheets
Quick-reference material, not deep dives. The [`docs/`](../docs/) directory is
where the real engineering write-ups live — verified claims, real evidence,
the actual design decisions behind each system. This directory is lower-key:
command references and config notes for tools I use day-to-day, kept here so
they're easy to find and easy to add to.
- **[proxmox/](proxmox/)** — CLI cheat sheet for the hypervisor cluster this
fleet runs on (`qm`/`pct`/`pvesm`/`pveam`), plus API token auth and
certificate management notes.
- **[zsh/](zsh/)** — the aliases, functions, and keybindings from my actual
zsh config, not a generic zsh tutorial.
cheatsheets/proxmox/proxmox-api-authentication.md
-18
View File
@@ -1,18 +0,0 @@
## Create an API Token on Proxmox
To create a new API Token for your `user` in Proxmox, follow these steps:
1. Open the Proxmox Web UI and navigate to the **'Datacenter'** in the **'Server View'** menu.
2. Select **'API Token'** in the **'Permissions'** menu.
3. Click on **'Add'**.
4. Select the `user`, and add a **'Token ID'**.
5. (Optional) Disable `Privilege Separation`, and set an `Expire` date.
6. Click on **'Add'**.
## Check the API Token
To test the API Token, you can use the following command:
```sh
curl -H "Authorization: PVEAPIToken=root@pam!monitoring=aaaaaaaaa-bbb-cccc-dddd-ef0123456789" https://your-proxmox-url:8006/api2/json/
```
cheatsheets/proxmox/proxmox-certificate-management.md
-324
View File
@@ -1,324 +0,0 @@
## Certificates for Intra-Cluster Communication
Each Proxmox VE cluster creates by default its own (self-signed) Certificate
Authority (CA) and generates a certificate for each node which gets signed by
the aforementioned CA. These certificates are used for encrypted communication
with the cluster's pveproxy service and the Shell/Console feature if SPICE is used.
The CA certificate and key are stored in the Proxmox Cluster File System (pmxcfs).
## Certificates for API and Web GUI
The REST API and web GUI are provided by the pveproxy service, which runs on each node.
You have the following options for the certificate used by pveproxy:
* By default the node-specific certificate in `/etc/pve/nodes/NODENAME/pve-ssl.pem` is used. This certificate is signed by the cluster CA and therefore not automatically trusted by browsers and operating systems.
* Use an externally provided certificate (e.g. signed by a commercial CA).
* Use ACME (Let's Encrypt) to get a trusted certificate with automatic renewal, this is also integrated in the Proxmox VE API and web interface.
For options 2 and 3 the file `/etc/pve/local/pveproxy-ssl.pem` (and
`/etc/pve/local/pveproxy-ssl.key`, which needs to be without password) is used.
Keep in mind that `/etc/pve/local` is a node specific symlink to
`/etc/pve/nodes/NODENAME`.
Certificates are managed with the Proxmox VE Node management command
(see the `pvenode(1)` manpage). The current node certificate's details can
be checked directly from the CLI too:
```shell
pvenode cert info [FORMAT_OPTIONS]
```
Do not replace or manually modify the automatically generated node
certificate files in `/etc/pve/local/pve-ssl.pem` and
`/etc/pve/local/pve-ssl.key` or the cluster CA files in
`/etc/pve/pve-root-ca.pem` and `/etc/pve/priv/pve-root-ca.key`.
## Upload Custom Certificate
If you already have a certificate which you want to use for a Proxmox VE node
you can upload that certificate simply over the web interface. Note that the
certificates key file, if provided, mustn't be password protected.
## Trusted certificates via Let's Encrypt (ACME)
Proxmox VE includes an implementation of the Automatic Certificate
Management Environment ACME protocol, allowing Proxmox VE admins to
use an ACME provider like Let's Encrypt for easy setup of TLS certificates
which are accepted and trusted on modern operating systems and web browsers
out of the box.
Currently, the two ACME endpoints implemented are the
Let's Encrypt (LE) production and its staging
environment. Our ACME client supports validation of http-01 challenges using
a built-in web server and validation of dns-01 challenges using a DNS plugin
supporting all the DNS API endpoints `acme.sh` does.
### ACME Account
You need to register an ACME account per cluster with the endpoint you want to
use. The email address used for that account will serve as contact point for
renewal-due or similar notifications from the ACME endpoint.
You can register and deactivate ACME accounts over the web interface
`Datacenter -> ACME` or using the `pvenode` command line tool.
```shell
pvenode acme account register account-name mail@example.com
```
Because of rate-limits you should use LE staging for experiments or if you use
ACME for the first time.
### ACME Plugins
The ACME plugins task is to provide automatic verification that you, and thus
the Proxmox VE cluster under your operation, are the real owner of a domain. This is
the basis building block for automatic certificate management.
The ACME protocol specifies different types of challenges, for example the
http-01 where a web server provides a file with a certain content to prove
that it controls a domain. Sometimes this isn't possible, either because of
technical limitations or if the address of a record to is not reachable from
the public internet. The dns-01 challenge can be used in these cases. This
challenge is fulfilled by creating a certain DNS record in the domain's zone.
Proxmox VE supports both of those challenge types out of the box, you can configure
plugins either over the web interface under `Datacenter -> ACME`, or using the
`pvenode acme plugin add` command.
ACME Plugin configurations are stored in `/etc/pve/priv/acme/plugins.cfg`.
A plugin is available for all nodes in the cluster.
### Node Domains
Each domain is node specific. You can add new or manage existing domain entries
under Node -> Certificates, or using the `pvenode config` command.
After configuring the desired domain(s) for a node and ensuring that the
desired ACME account is selected, you can order your new certificate over the
web-interface. On success the interface will reload after 10 seconds.
Renewal will happen automatically.
## ACME HTTP Challenge Plugin
There is always an implicitly configured standalone plugin for validating
http-01 challenges via the built-in webserver spawned on port 80.
The name `standalone` means that it can provide the validation on it's
own, without any third party service. So, this plugin works also for cluster
nodes.
There are a few prerequisites to use it for certificate management with Let's
Encrypts ACME.
* You have to accept the ToS of Let's Encrypt to register an account.
* Port 80 of the node needs to be reachable from the internet.
* There must be no other listener on port 80.
* The requested (sub)domain needs to resolve to a public IP of the Node.
## ACME DNS API Challenge Plugin
On systems where external access for validation via the http-01 method is
not possible or desired, it is possible to use the dns-01 validation method.
This validation method requires a DNS server that allows provisioning of TXT
records via an API.
### Configuring ACME DNS APIs for validation
Proxmox VE re-uses the DNS plugins developed for the
[acme.sh project](https://github.com/acmesh-official/acme.sh). Please
refer to its documentation for details on configuration of specific APIs.
The easiest way to configure a new plugin with the DNS API is using the web
interface (`Datacenter -> ACME`).
Choose DNS as challenge type. Then you can select your API provider, enter
the credential data to access your account over their API.
See the `acme.sh`
[How to use DNS API](https://github.com/acmesh-official/acme.sh/wiki/dnsapi#how-to-use-dns-api)
wiki for more detailed information about getting API credentials for your
provider.
As there are many DNS providers and API endpoints Proxmox VE automatically generates
the form for the credentials for some providers. For the others you will see a
bigger text area, simply copy all the credentials KEY=VALUE pairs in there.
## DNS Validation through CNAME Alias
A special alias mode can be used to handle the validation on a different
domain/DNS server, in case your primary/real DNS does not support provisioning
via an API. Manually set up a permanent CNAME record for
`_acme-challenge.domain1.example` pointing to `_acme-challenge.domain2.example`
and set the alias property in the Proxmox VE node configuration file to
`domain2.example` to allow the DNS server of `domain2.example` to validate all
challenges for `domain1.example`.
### Combination of Plugins
Combining http-01 and dns-01 validation is possible in case your node is
reachable via multiple domains with different requirements / DNS provisioning
capabilities. Mixing DNS APIs from multiple providers or instances is also
possible by specifying different plugin instances per domain.
Accessing the same service over multiple domains increases complexity and
should be avoided if possible.
## Automatic renewal of ACME certificates
If a node has been successfully configured with an ACME-provided certificate
(either via `pvenode` or via the GUI), the certificate will be automatically
renewed by the `pve-daily-update.service`. Currently, renewal will be attempted
if the certificate has expired already, or will expire in the next 30 days.
A certificate can also be renewed directly, on demand:
```shell
pvenode acme cert renew [OPTIONS]
```
## ACME Examples with pvenode
*Example*: Sample `pvenode` invocation for using Let's Encrypt certificates
```sh
root@proxmox:~# pvenode acme account register default mail@example.invalid
Directory endpoints:
0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
2) Custom
Enter selection: 1
Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Do you agree to the above terms? [y|N]y
...
Task OK
root@proxmox:~# pvenode config set --acme domains=example.invalid
root@proxmox:~# pvenode acme cert order
Loading ACME account details
Placing ACME order
...
Status is 'valid'!
All domains validated!
...
Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
Task OK
```
*Example*: Setting up the OVH API for validating a domain
The account registration steps are the same no matter which plugins are
used, and are not repeated here.
OVH_AK and OVH_AS need to be obtained from OVH according to the OVH
API documentation
First you need to get all information so you and Proxmox VE can access the API.
```sh
root@proxmox:~# cat /path/to/api-token
OVH_AK=XXXXXXXXXXXXXXXX
OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
root@proxmox:~# source /path/to/api-token
root@proxmox:~# curl -XPOST -H"X-Ovh-Application: $OVH_AK" -H "Content-type: application/json" \
https://eu.api.ovh.com/1.0/auth/credential -d '{
"accessRules": [
{"method": "GET","path": "/auth/time"},
{"method": "GET","path": "/domain"},
{"method": "GET","path": "/domain/zone/*"},
{"method": "GET","path": "/domain/zone/*/record"},
{"method": "POST","path": "/domain/zone/*/record"},
{"method": "POST","path": "/domain/zone/*/refresh"},
{"method": "PUT","path": "/domain/zone/*/record/"},
{"method": "DELETE","path": "/domain/zone/*/record/*"}
]
}'
{"consumerKey":"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ","state":"pendingValidation","validationUrl":"https://eu.api.ovh.com/auth/?credentialToken=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}
(open validation URL and follow instructions to link Application Key with account/Consumer Key)
root@proxmox:~# echo "OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" >> /path/to/api-token
```
Now you can setup the ACME plugin:
```sh
root@proxmox:~# pvenode acme plugin add dns example_plugin --api ovh --data /path/to/api_token
root@proxmox:~# pvenode acme plugin config example_plugin
```
```
┌────────┬──────────────────────────────────────────┐
│ key │ value │
╞════════╪══════════════════════════════════════════╡
│ api │ ovh │
├────────┼──────────────────────────────────────────┤
│ data │ OVH_AK=XXXXXXXXXXXXXXXX │
│ │ OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY │
│ │ OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ │
├────────┼──────────────────────────────────────────┤
│ digest │ 867fcf556363ca1bea866863093fcab83edf47a1 │
├────────┼──────────────────────────────────────────┤
│ plugin │ example_plugin │
├────────┼──────────────────────────────────────────┤
│ type │ dns │
└────────┴──────────────────────────────────────────┘
```
At last you can configure the domain you want to get certificates for and
place the certificate order for it:
```sh
root@proxmox:~# pvenode config set --acmedomain0 example.proxmox.com,plugin=example_plugin
root@proxmox:~# pvenode acme cert order
Loading ACME account details
Placing ACME order
Order URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/11111111/22222222
Getting authorization details from 'https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/33333333'
The validation for example.proxmox.com is pending!
[Wed Apr 22 09:25:30 CEST 2020] Using OVH endpoint: ovh-eu
[Wed Apr 22 09:25:30 CEST 2020] Checking authentication
[Wed Apr 22 09:25:30 CEST 2020] Consumer key is ok.
[Wed Apr 22 09:25:31 CEST 2020] Adding record
[Wed Apr 22 09:25:32 CEST 2020] Added, sleep 10 seconds.
Add TXT record: _acme-challenge.example.proxmox.com
Triggering validation
Sleeping for 5 seconds
Status is 'valid'!
[Wed Apr 22 09:25:48 CEST 2020] Using OVH endpoint: ovh-eu
[Wed Apr 22 09:25:48 CEST 2020] Checking authentication
[Wed Apr 22 09:25:48 CEST 2020] Consumer key is ok.
Remove TXT record: _acme-challenge.example.proxmox.com
All domains validated!
Creating CSR
Checking order status
Order is ready, finalizing order
valid!
Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
Task OK
```
### Example: Switching from the staging to the regular ACME directory
Changing the ACME directory for an account is unsupported, but as Proxmox VE
supports more than one account you can just create a new one with the
production (trusted) ACME directory as endpoint. You can also deactivate the
staging account and recreate it.
*Example*: Changing the default ACME account from staging to directory using `pvenode`
```sh
root@proxmox:~# pvenode acme account deactivate default
Renaming account file from '/etc/pve/priv/acme/default' to '/etc/pve/priv/acme/_deactivated_default_4'
Task OK
root@proxmox:~# pvenode acme account register default example@proxmox.com
Directory endpoints:
0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
2) Custom
Enter selection: 0
Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Do you agree to the above terms? [y|N]y
...
Task OK
```
cheatsheets/proxmox/proxmox-cheat-sheet.md
-426
View File
@@ -1,426 +0,0 @@
Proxmox Virtual Environment (Proxmox VE or PVE) is a hyper-converged infrastructure open-source software. It is a hosted hypervisor that can run operating systems including Linux and Windows on x64 hardware. It is a Debian-based Linux distribution with a modified Ubuntu LTS kernel and allows deployment and management of virtual machines and containers. Proxmox VE includes a web console and command-line tools, and provides a REST API for third-party tools. Two types of virtualization are supported: container-based with LXC (starting from version 4.0 replacing OpenVZ used in version up to 3.4, included), and full virtualization with KVM. It includes a web-based management interface.
Proxmox VE is licensed under the GNU Affero General Public License, version 3.
Repository: [https://git.proxmox.com](https://git.proxmox.com)
Website: [https://pve.proxmox.com](https://pve.proxmox.com)
## VM Management
```shell
# list VMs
qm list
# Create or restore a virtual machine.
qm create your-vm-id
# start a VM
qm start your-vm-id
# Suspend virtual machine.
qm suspend your-vm-id
# shutdown a VM
qm shutdown your-vm-id
# reboot a VM
qm reboot your-vm-id
# reset a VM
qm reset your-vm-id
# stop a VM
qm stop your-vm-id
# Destroy the VM and all used/owned volumes.
# Removes any VM specific permissions and firewall rules
qm destroy your-vm-id
# Enter Qemu Monitor interface.
qm monitor your-vm-id
# Get the virtual machine configuration with both current and pending values.
qm pending your-vm-id
# Get the virtual machine configuration with pending changes applied.
qm config your-vm-id [OPTIONS]
# Send key event to virtual machine.
qm sendkey your-vm-id your-key-event [OPTIONS]
# Show command line which is used to start the VM (debug info).
qm showcmd your-vm-id [OPTIONS]
# Unlock the VM.
qm unlock your-vm-id
# Clone a VM
qm clone your-vm-id new-vm-id
# Migrate a VM
qm migrate your-vm-id target-node
# Migrate a VM to a remote cluster. EXPERIMENTAL.
qm remote-migrate your-vm-id target-endpoint --target-bridge <string> --target-storage <string> [OPTIONS]
# Create a new VM from an OVF manifest.
qm importovf your-vm-id <manifest> <storage> [OPTIONS]
# Enroll updated UEFI certificates to the EFI disk.
qm enroll-efi-keys your-vm-id
# Show VM status
qm status your-vm-id
# Wait until the VM is stopped.
qm wait your-vm-id [OPTIONS]
# Clean up resources for a VM
qm cleanup your-vm-id your-clean-shutdown your-guest-requested
# Create a Template.
qm template your-vm-id [OPTIONS]
# Set virtual machine options (synchrounous API)
qm set your-vm-id [OPTIONS]
```
### Cloudinit
```shell
# Get automatically generated cloudinit config.
qm cloudinit dump your-vm-id your-vm-type
# Get the cloudinit configuration with both current and pending values.
qm cloudinit pending your-vm-id
# Regenerate and change cloudinit config drive.
qm cloudinit update your-vm-id
```
### Disk
```shell
# Import an external disk image as an unused disk in a VM.
# The image format has to be supported by qemu-img(1).
qm disk import your-vm-id your-target-source your-target-storage
# Move volume to different storage or to a different VM.
qm disk move your-vm-id your-vm-disk [<storage>] [OPTIONS]
# Rescan all storages and update disk sizes and unused disk images.
qm disk rescan [OPTIONS]
# Extend volume size.
qm disk resize your-vm-id your-vm-disk <size> [OPTIONS]
# Unlink/delete disk images.
qm disk unlink your-vm-id --idlist <string> [OPTIONS]
# rescan volumes
qm rescan
```
### Snapshot
```shell
# List all snapshots.
qm listsnapshot your-vm-id
# Snapshot a VM
qm snapshot your-vm-id <snapname>
# Delete a snapshot.
qm delsnapshot your-vm-id <snapname>
# Rollback a snapshot
qm rollback your-vm-id <snapname>
# Open a terminal using a serial device
# (The VM need to have a serial device configured, for example serial0: socket)
qm terminal your-vm-id [OPTIONS]
# Proxy VM VNC traffic to stdin/stdout
qm vncproxy your-vm-id
```
### Misc
```shell
# Execute Qemu Guest Agent commands.
qm guest cmd your-vm-id <command>
# Executes the given command via the guest agent
qm guest exec your-vm-id [<extra-args>] [OPTIONS]
# Gets the status of the given pid started by the guest-agent
qm guest exec-status your-vm-id <pid>
# Sets the password for the given user to the given password
qm guest passwd your-vm-id <username> [OPTIONS]
```
### PV, VG, LV Management
```shell
# Create a PV
pvcreate <disk-device-name>
# Remove a PV
pvremove <disk-device-name>
# List all PVs
pvs
# Create a VG
vgcreate <vg-name> <disk-device-name>
# Remove a VG
vgremove <vg-name>
# List all VGs
vgs
# Create a LV
lvcreate -L <lv-size> -n <lv-name> <vg-name>
# Remove a LV
lvremove <vg-name>/<lv-name>
# List all LVs
lvs
```
### Storage Management
```shell
# Create a new storage.
pvesm add <type> <storage> [OPTIONS]
# Allocate disk images.
pvesm alloc <storage> your-vm-id <filename> <size> [OPTIONS]
# Delete volume
pvesm free <volume> [OPTIONS]
# Delete storage configuration.
pvesm remove <storage>
# Update storage configuration.
pvesm set <storage> [OPTIONS]
# List storage content.
pvesm list <storage> [OPTIONS]
# Get the filesystem path for a specified volume.
pvesm path <volume>
# Prune backups on a storage (standard naming scheme only).
pvesm prune-backups <storage> [OPTIONS]
# Extract configuration from a vzdump backup archive.
pvesm extractconfig <volume>
# Returns the storage API version and age.
pvesm apiinfo
# An alias for pvesm scan lvm.
pvesm lvmscan
# An alias for pvesm scan lvmthin.
pvesm lvmthinscan
# List local LVM volume groups.
pvesm scan lvm
# List local LVM Thin Pools.
pvesm scan lvmthin <vg>
# Get status for all datastores.
pvesm status [OPTIONS]
```
### Template Management
```shell
# list all templates
pveam available
# list all templates
pveam list <storage>
# Download appliance templates
pveam download <storage> <template>
# Remove a template.
pveam remove <template-path>
# Update Container Template Database.
pveam update
```
## Certificate Management
See the [Proxmox Certificate Management](proxmox-certificate-management.md) cheat sheet.
## Container Management
```shell
# List containers
pct list
# Create or restore a container.
pct create your-vm-id <ostemplate> [OPTIONS]
# Start the container.
pct start your-vm-id [OPTIONS]
# Create a container clone/copy
pct clone your-vm-id new-vm-id [OPTIONS]
# Suspend the container. This is experimental.
pct suspend your-vm-id
# Resume the container.
pct resume your-vm-id
# Stop the container.
# This will abruptly stop all processes running in the container.
pct stop your-vm-id [OPTIONS]
# Shutdown the container.
# This will trigger a clean shutdown of the container, see lxc-stop(1) for details.
pct shutdown your-vm-id [OPTIONS]
# Destroy the container (also delete all uses files).
pct destroy your-vm-id [OPTIONS]
# Show CT status.
pct status your-vm-id [OPTIONS]
# Migrate the container to another node. Creates a new migration task.
pct migrate your-vm-id <target> [OPTIONS]
# Migrate the container to a remote cluster. Creates a new migration task. EXPERIMENTAL.
pct remote-migrate your-vm-id target-endpoint --target-bridge <string> --target-storage <string> [OPTIONS]
# Get container configuration.
pct config your-vm-id [OPTIONS]
# Print the list of assigned CPU sets.
pct cpusets
# Get container configuration, including pending changes.
pct pending your-vm-id
# Reboot the container by shutting it down, and starting it again. Applies pending changes.
pct reboot your-vm-id [OPTIONS]
# Create or restore a container.
pct restore your-vm-id <ostemplate> [OPTIONS]
# Set container options.
pct set your-vm-id [OPTIONS]
# Create a Template.
pct template your-vm-id
# Unlock the VM.
pct unlock your-vm-id
```
### Container Disks
```shell
# Get the container's current disk usage.
pct df your-vm-id
# Run a filesystem check (fsck) on a container volume.
pct fsck your-vm-id [OPTIONS]
# Run fstrim on a chosen CT and its mountpoints.
pct fstrim your-vm-id [OPTIONS]
# Mount the container's filesystem on the host.
# This will hold a lock on the container and is meant for emergency maintenance only
# as it will prevent further operations on the container other than start and stop.
pct mount your-vm-id
# Move a rootfs-/mp-volume to a different storage or to a different container.
pct move-volume your-vm-id <volume> [<storage>] [<target-vmid>] [<target-volume>] [OPTIONS]
# Unmount the container's filesystem.
pct unmount your-vm-id
# Resize a container mount point.
pct resize your-vm-id your-vm-disk <size> [OPTIONS]
# Rescan all storages and update disk sizes and unused disk images.
pct rescan [OPTIONS]
# Connect to container
pct enter your-vm-id
# Launch a console for the specified container.
pct console your-vm-id [OPTIONS]
# Launch a shell for the specified container.
pct enter your-vm-id
# Launch a command inside the specified container.
pct exec your-vm-id [<extra-args>]
# Copy a file from the container to the local system.
pct pull your-vm-id <path> <destination> [OPTIONS]
# Copy a local file to the container.
pct push your-vm-id <file> <destination> [OPTIONS]
```
### Container Snapshot
```shell
# Snapshot a container.
pct snapshot your-vm-id <snapname> [OPTIONS]
# List all snapshots.
pct listsnapshot your-vm-id
# Rollback LXC state to specified snapshot.
pct rollback your-vm-id <snapname> [OPTIONS]
# Delete a LXC snapshot.
pct delsnapshot your-vm-id <snapname> [OPTIONS]
```
## Web GUI
```shell
# Restart web GUI
service pveproxy restart
```
## Resize Disk
### Increase disk size
Increase disk size in the GUI or with the following command
```shell
qm resize 100 virtio0 +5G
```
### Decrease disk size
> Before decreasing disk sizes in Proxmox, you should take a backup!
1. Convert qcow2 to raw: `qemu-img convert vm-100.qcow2 vm-100.raw`
2. Shrink the disk `qemu-img resize -f raw vm-100.raw 10G`
3. Convert back to qcow2 `qemu-img convert -p -O qcow2 vm-100.raw vm-100.qcow2`
## Further information
More examples and tutorials regarding Proxmox can be found in the link list below:
- Ansible playbook that automates Linux VM updates running on Proxmox (including snapshots): [TheDatabaseMe - update_proxmox_vm](https://github.com/thedatabaseme/update_proxmox_vm)
- Manage Proxmox VM templates with Packer: [Use Packer to build Proxmox images](https://thedatabaseme.de/2022/10/16/what-a-golden-boy-use-packer-to-build-proxmox-images/)
cheatsheets/zsh/zsh-cheatsheet.md
-169
View File
@@ -1,169 +0,0 @@
# Zsh Cheatsheet
Reference for the new ZDOTDIR-based zsh setup (oh-my-zsh removed,
config lives in `~/.config/zsh/`).
## Aliases
### Docker Compose
| Alias | Expands to | What it does |
|---|---|---|
| `dcu` | `docker compose up -d` | Start containers in detached mode |
| `dcd` | `docker compose down` | Stop and remove containers |
| `dcr` | `docker compose restart` | Restart containers |
| `dcp` | `docker compose pull` | Pull latest images |
### Navigation
| Alias | Expands to | What it does |
|---|---|---|
| `..` | `cd ..` | Up one directory |
| `...` | `cd ../..` | Up two directories |
| `....` | `cd ../../..` | Up three directories |
### File listing
| Alias | Expands to | What it does |
|---|---|---|
| `ls` | `eza -lahF --group-directories-first --time-style="+%d/%m/%Y %H:%M" --git` (falls back to plain `ls -lahF ...` if eza isn't installed) | Long listing, all files, classify symbols (`/`, `*`, `@`), directories first, custom date format, git status column |
| `tree` | `eza --tree` (only if eza is installed) | Tree view of directory structure |
| `cat` | `bat` (only if bat is installed) | Syntax-highlighted file viewer |
### System
| Alias | Expands to | What it does |
|---|---|---|
| `df` | `df -h` | Disk usage, human-readable sizes |
| `free` | `free -h` | Memory usage, human-readable sizes |
| `c` | `clear` | Clear the terminal |
## Functions
| Function | Usage | What it does |
|---|---|---|
| `cd` (overridden) | `cd <dir>` | Changes directory, then automatically runs `ls` (so you always see the contents of where you land). `cd` with no argument goes to `$HOME`. |
| `mkcd` | `mkcd <dir>` | Creates a directory and `cd`s into it in one step |
| `extract` | `extract <file>` | Extracts an archive based on its extension. Supports `.tar.gz`/`.tgz`, `.tar.bz2`, `.tar.xz`, `.zip`, `.gz` |
| `ports` | `ports` | Shows listening ports and the processes using them (`sudo ss -tulpen`) |
| `psg` | `psg <name>` | Searches running processes by name (case-insensitive `ps aux | grep`) |
| `killport` | `killport <port>` | Kills whatever process is listening on the given port |
| `logtail` | `logtail <file> [lines]` | Tails a log file (`-f`), default 50 lines, or specify a custom number |
| `reload` | `reload` | Reloads the zsh config (`source $ZDOTDIR/.zshrc`) without restarting the shell |
## Keybindings
| Key | Action | Notes |
|---|---|---|
| `Ctrl + →` | Move cursor forward one word | |
| `Ctrl + ←` | Move cursor backward one word | |
| `Ctrl + \` | Toggle autosuggestions on/off | Echoes "Autosuggestions: on/off" so you know the current state |
| `↑` (Up arrow) | Search history backward by prefix | Type a few characters first, then press Up to find matching previous commands |
| `↓` (Down arrow) | Search history forward by prefix | Same as above, opposite direction |
## Line editing (built-in zsh defaults)
These come from zsh's default "emacs" editing mode (the standard unless
something switches it to vi mode — this setup doesn't). Most of these
work in any emacs-style shell (bash, zsh) and are worth building into
muscle memory since they work everywhere.
### Cursor movement
| Key | Action |
|---|---|
| `←` / `→` | Move cursor one character left/right |
| `Ctrl + ←` / `Ctrl + →` | Move cursor one **word** left/right (custom binding in this setup) |
| `Ctrl + A` | Jump to the **start** of the line |
| `Ctrl + E` | Jump to the **end** of the line |
| `Alt + B` | Move backward one word (alternative to Ctrl+←) |
| `Alt + F` | Move forward one word (alternative to Ctrl+→) |
### Deleting / cutting text
| Key | Action |
|---|---|
| `Backspace` | Delete the character **before** the cursor |
| `Ctrl + D` / `Delete` | Delete the character **under/after** the cursor (Ctrl+D on an empty line exits the shell — careful) |
| `Ctrl + W` | Delete the **word before** the cursor (cuts to kill-ring) |
| `Alt + D` | Delete the **word after** the cursor (cuts to kill-ring) |
| `Alt + Backspace` | Same as Ctrl+W — delete word before cursor |
| `Ctrl + U` | Delete **everything before** the cursor (whole line back to start) |
| `Ctrl + K` | Delete **everything after** the cursor (to end of line) |
### Pasting (yanking) deleted text
zsh doesn't have "select text" like a GUI — instead, anything you cut
with `Ctrl+W`, `Alt+D`, `Ctrl+U`, or `Ctrl+K` goes into a clipboard-like
buffer (the "kill ring"), which you can paste back with:
| Key | Action |
|---|---|
| `Ctrl + Y` | Paste ("yank") the last cut text at the cursor |
**Practical example:** to move a word from the end of a command to the
start — put cursor at end of word, `Alt + Backspace` to cut it, `Ctrl + A`
to jump to the start of the line, `Ctrl + Y` to paste it there.
### Undo / retype
| Key | Action |
|---|---|
| `Ctrl + _` (Ctrl + Shift + -) | Undo the last edit |
| `Ctrl + L` | Clear the screen (same as the `c` alias, but doesn't lose scrollback the same way) |
### Transposing / fixing typos
| Key | Action |
|---|---|
| `Ctrl + T` | Swap the two characters around the cursor (fixes a typo like `hte` → `the`) — **note:** this is overridden by fzf's Ctrl+T in this setup if fzf is installed |
| `Alt + T` | Swap the two **words** around the cursor |
### Case changes
| Key | Action |
|---|---|
| `Alt + U` | Uppercase the word from the cursor onward |
| `Alt + L` | Lowercase the word from the cursor onward |
| `Alt + C` | Capitalize the word from the cursor onward |
### Re-running / editing previous commands
| Key | Action |
|---|---|
| `!!` | Re-run the last command (e.g. `sudo !!` re-runs the last command with sudo) |
| `!$` | The last **argument** of the previous command (e.g. `cat !$` opens whatever file you last referenced) |
| `Ctrl + R` (no fzf) | Reverse incremental search through history (built-in, before fzf takes over — see fzf section below for the upgraded version) |
| `fc` | Opens the last command in `$EDITOR` (nano) for editing, then runs it |
## Fuzzy search (fzf)
Only active if `fzf` is installed.
| Key | Action | Notes |
|---|---|---|
| `Ctrl + R` | Fuzzy search command history | Type to filter, select a command, it's inserted at the cursor. This overrides the built-in reverse search mentioned above |
| `Ctrl + F` | Fuzzy file search (includes hidden files, excludes `.git`) | Selected file path is inserted at the cursor. Uses `fd` if installed, otherwise falls back to `find` |
| `Ctrl + T` | Fuzzy file search (fzf's built-in, non-hidden files) | Only active if fzf's own shell integration is found on the system; uses `fd` for speed if installed. This overrides the built-in character-transpose shortcut mentioned above |
## Plugins (auto-loaded on shell startup)
These are cloned automatically into `$ZDOTDIR/plugins/` the first time
you start a shell after the role runs — no manual setup needed.
| Plugin | What it does |
|---|---|
| `zsh-autosuggestions` | Shows a greyed-out suggestion as you type, based on history. Press `→` (right arrow) or `End` to accept it |
| `fast-syntax-highlighting` | Colors commands as you type — valid commands, paths, strings, etc. get different colors so you can spot typos before pressing Enter |
## Environment / history
- History file: `~/.local/state/zsh/history`
- History size: 10,000 commands
- History behavior: shared across sessions, duplicates ignored, commands
starting with a space aren't recorded (`HIST_IGNORE_SPACE`)
- Default editor: `nano` (`$EDITOR` / `$VISUAL`)
- Config location: `$ZDOTDIR` = `~/.config/zsh/`
- Per-host extra env vars (e.g. `VAULT_ADDR`) are set via `zsh_extra_env`
in Ansible host_vars, appended to `env.zsh`
## Startup
`fastfetch` runs automatically every time a new shell opens, showing
system info (OS, kernel, uptime, packages, etc.) above the prompt.
docs/architecture-overview.md
+2 -2
View File
@@ -1,6 +1,6 @@
<!-- verified: 2026-09-18
rocky-docker-lxc@d225776 : caddy/conf/Caddyfile
caddy-vm@c5c3420 : caddy/conf/Caddyfile
rocky-docker-lxc@7dd5025 : caddy/Caddyfile
caddy-vm@9369f8b : caddy/Caddyfile
trilium : 2wCPV6NXQa3l, 0mu75aAXGH8X, fscYadnUtmD5, QMTh5G7ldlLV, TIbCwYGitboy, Cl4TWBZ7ytYr, 6cJurv76PNDR, nAPJq9A1iuX8, DvKxK0gHXsng
-->
docs/dashy-soc-board.md
+4 -4
View File
@@ -1,4 +1,4 @@
<!-- verified: 2026-09-19
<!-- verified: 2026-09-06
trilium : 5XjzoVTH1r3B, cVFNSn4CuuEN
-->
@@ -14,10 +14,10 @@ This board didn't start from nothing — it absorbed several formerly-standalone
| Source | What it actually contributes |
|---|---|
| Wazuh indexer | 7-day findings, categorized: ClamAV, vulnerabilities, rootkit/anomaly, authentication, other |
| Vault | Seal state, active lease/token counts, KV engine counts, and token TTL distribution — read from Prometheus metrics, not a live Vault call, so all of them go stale together if the metrics-reader token lapses. Root/intermediate CA cert expiry is the one exception, parsed live from Vault's own `pki/cert/ca` and `pki_int/cert/ca` endpoints on every load regardless of metrics health |
| Vault | Seal state, active lease/token counts, KV engine counts, token TTL distribution, and root/intermediate CA cert expiry — parsed live from Vault's own `pki/cert/ca` and `pki_int/cert/ca` endpoints on every load, not a cached value |
| Technitium DNS | Query stats, top clients/domains, blocklist state, plus a dedicated "DNS Security Signals" section: forwarder failures (read from the Loki-shipped query log), query-log ingestion lag, next blocklist update time |
| Fleet config-drift audit | *Not* computed by this board's own function — read from Node-RED's shared `fleetAudit` global, populated by a separate ingestion process, covering 4 finding types: untracked stacks, missing declared directories, containers stuck in `Created`, and TZ drift |
| Slack | 12 app-specific channels (the *arr suite, Jellyfin/Emby/Seerr, Kopia, Authentik, Gitea, Proxmox, Grafana) mapped to their owning host by a plain lookup table, plus each host's own alert channel |
| Slack | 12 app-specific channels (the *arr suite, Jellyfin/Emby/Seerr, Kopia, Authentik, Gitea, Netbox, Grafana) mapped to their owning host by a plain lookup table, plus each host's own alert channel |
| Proxmox `cluster/resources` API | The two hypervisor nodes' own CPU/RAM/disk (no Prometheus coverage on bare metal) plus the full running VM/LXC tree per node |
The fourteen, enumerated directly from the function's own upstream-URL constants rather than counted by hand — this is the actual grep and its real output, not a reconstruction:
@@ -48,7 +48,7 @@ for n in data:
if n.get('type') == 'function' and n.get('name') == 'Build SOC Board HTML':
print(n['name'], '->', len(n['func']), 'chars')
"
Build SOC Board HTML -> 104635 chars
Build SOC Board HTML -> 104443 chars
```
## Why One Function Instead of Native Dashy Widgets
docs/disaster-recovery.md
+1 -1
View File
@@ -1,5 +1,5 @@
<!-- verified: 2026-09-18
iac@ae315de : ansible/roles_metadata.yml
iac@f770e1c : ansible/roles_metadata.yml
ansible@24e1bf0 : roles/os_update, playbooks/os_update_vms.yml, playbooks/os_update_pve.yml, playbooks/os_update_pbs.yml, playbooks/app_update.yml
trilium : xJRSQovMncQr, 6AbdS9cUE45T, bW84ccRoAJ5i, p45qNP2sIuhA, 6oESvJYzj1at
-->
docs/dns-naming.md
+1 -1
View File
@@ -1,4 +1,4 @@
<!-- verified: 2026-09-18
<!-- verified: 2026-09-06
trilium : nAPJq9A1iuX8, G6NWXtdWX9ap, 0mu75aAXGH8X, fscYadnUtmD5
-->
docs/firewall-and-network.md
+1 -1
View File
@@ -1,5 +1,5 @@
<!-- verified: 2026-09-06
iac@49d9773 : ansible/roles/bootstrap_roles/firewall_ssh, ansible/roles/secure_vault/firewall_rich_rules_per_vm, ansible/roles/cis_hardening/password_policy
iac@49ff427 : ansible/roles/bootstrap_roles/firewall_ssh, ansible/roles/secure_vault/firewall_rich_rules_per_vm, ansible/roles/cis_hardening/password_policy
trilium : 0mu75aAXGH8X, o3DDWo1v54vI, oEbKGleDKuvb, EpssgVXxqIbE, 0R8GeBmH7boV, TIbCwYGitboy, fscYadnUtmD5, yUJqswizuET2, QMTh5G7ldlLV, jBPOx07B1Y2Z, Cl4TWBZ7ytYr, 6cJurv76PNDR, kwbbLjeI6ody, MSVD5o5HiJ69, kA00FQiQWVem
-->
docs/homelab-inventory.md
+1 -1
View File
@@ -1,6 +1,6 @@
<!-- verified: 2026-09-07
ansible@9a4ba6d : roles/homelab_inventory_hardware, roles/homelab_inventory_vms, roles/homelab_inventory_services, playbooks/homelab_inventory_hardware.yml, playbooks/homelab_inventory_vms.yml, playbooks/homelab_inventory_services.yml
rocky-docker-lxc@d225776 : caddy/conf/Caddyfile, dashy/docker-compose.yaml
rocky-docker-lxc@145d3df : caddy/Caddyfile, dashy/docker-compose.yaml
trilium : O0d4fqw0mr05
-->
docs/iac-pipeline.md
+9 -19
View File
@@ -1,5 +1,5 @@
<!-- verified: 2026-09-19
iac@ae315de : packer/rocky10-proxmox-template, terraform, ansible/roles_metadata.yml, ansible/playbooks/provisioning.yml, ansible/host_vars/technitium/host_tags.yml, ansible/roles/cis_hardening/password_policy
<!-- verified: 2026-09-06
iac@49ff427 : packer/rocky10-proxmox-template, terraform, ansible/roles_metadata.yml, ansible/playbooks/provisioning.yml, ansible/host_vars/technitium/host_tags.yml, ansible/roles/cis_hardening/password_policy
trilium : c5JADrRVvjIn, f1uTcvXvq0u3, 7anU4VULpp7n, sovEBDmn16hh, sqVGFejKPOFn, kA00FQiQWVem, MSVD5o5HiJ69, NwwrSew7mP2z, QgviHP8r14t2
-->
@@ -53,7 +53,7 @@ Terraform manages both VMs and LXCs now — two separate resource types, one per
The LXC path is genuinely newer and mirrors the VM flow step-for-step, but has to close a gap the VM path never has: the stock container template has no `sshd` and no fleet users at all, since it never went through Packer. A `pct exec` bootstrap step runs first — installing `sshd` and creating both fleet identities with their keys, the same baseline Packer bakes into the VM image at build time instead of at provision time — before the same wait-for-SSH and tag-file-render steps every VM already gets. This is the real resource definition, not a paraphrase:
```
$ sed -n '20,39p' containers.tf
$ head -20 containers.tf
resource "proxmox_virtual_environment_container" "ct" {
provider = proxmox-bpg
for_each = var.containers
@@ -64,8 +64,6 @@ resource "proxmox_virtual_environment_container" "ct" {
start_on_boot = each.value.onboot
started = true
# PVE-native tags — cosmetic/searchable in the PVE UI, mirrors main.tf's
# `tags = join(",", each.value.tags)` on the VM resource.
tags = each.value.tags
cpu {
@@ -84,12 +82,10 @@ This is the actual template that renders into every host's tag file — the lite
```
$ cat host_tags.yml.tpl
# Host metadata from Terraform provisioning
vmid: ${vmid}
vm_name: ${name}
target_node: ${target_node}
template: ${template}
inventory_group: ${inventory_group}
cores: ${cores}
sockets: ${sockets}
memory: ${memory}
@@ -106,12 +102,7 @@ vm_tags:
%{ endfor ~}
```
Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes.
> [!NOTE]
> **A field that exists in the template but not yet in any VM's real file.** `inventory_group` was added to this template alongside the 2026-08 LXC rollout — new LXCs render it correctly (`rocky_lxc`, confirmed live on netbird), but no existing VM has been re-applied since, so `inventory_group: rocky_vm` isn't actually in any of their rendered files yet, technitium's included below. A real, current gap, not a hypothetical one — worth naming rather than quietly patching the example to look more finished than the fleet actually is.
And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, `ip`/`macaddr` redacted:
Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes. And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, `ip`/`macaddr` redacted:
```
$ cat host_vars/technitium/host_tags.yml
@@ -151,10 +142,9 @@ vm_tags:
- observability_agent
- mqtt_client
- wazuh_agent_tuning
- netbird_agent
```
22 tags on this one host alone, spanning the initialization roles (`packages`, `firewall_ssh`, …), the provisioning-stage `netbird_agent` role, and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads.
21 tags on this one host alone, spanning both the initialization roles (`packages`, `firewall_ssh`, …) and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads.
## Ansible: tag-driven, re-applied continuously
@@ -169,13 +159,13 @@ d = yaml.safe_load(open('roles_metadata.yml'))['roles_metadata']
print('provisioning:', [k for k,v in d.items() if v.get('provisioning') is True])
print('compliance:', sorted(k for k,v in d.items() if v.get('compliance') is True))
"
provisioning: ['clamav', 'netbird_agent']
compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_ca_client', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom']
provisioning: ['clamav']
compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom']
```
Two roles are tagged for the provisioning stage: `clamav` (rocky-docker only) and `netbird_agent` (enrols a host into the NetBird mesh overlay covered in [`docs/netbird-zero-trust.md`](netbird-zero-trust.md) — monitoring was the first host onboarded through it). Neither fits "once at init" or "every compliance run" cleanly, which is exactly the shape this stage exists for — an under-used pipeline stage isn't the same as an unused one.
`clamav` is the only provisioning-stage role, and only one host carries it in its `vm_tags`. An under-used pipeline stage isn't the same as an unused one — it exists on purpose for exactly this shape of role, not as leftover scaffolding.
Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 16 total compliance-stage roles above — the other 7 handle observability, notifications, SSH certificate refresh, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 15 total compliance-stage roles above — the other 6 handle observability, notifications, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
## The Honest Exception: not everything fits the clean model
docs/kodi-music-assistant-bridge.md
-134
View File
@@ -1,134 +0,0 @@
<!-- verified: 2026-10-16
rocky-docker-lxc@1ede96f : kodi/bridge.py, kodi/docker-compose.yaml, kodi/.env, music-assistant/docker-compose.yaml
trilium : Y2VzE58wL8w1
-->
# 🎧 Kodi <-> Music Assistant Smart Queue Bridge
> [!NOTE]
> Kodi stays the only music player, UI, and ratings authority on this fleet — ratings and library metadata live in a shared MariaDB, untouched. Music Assistant runs invisibly behind it as a pure similarity engine, never opened day-to-day, never a player, never writing anything back to the library. A small daemon watches Kodi's live playback queue and auto-refills it with musically similar tracks when it runs low, using MA purely as a recommendation backend.
## Why This Shape, Not The Obvious One
Two more obvious designs were deliberately rejected. The first: let MA write ratings and tags back into the audio files directly, so it could double as a second library manager. MA doesn't support this upstream, and even if it did, it would create two competing sources of truth for the same library — exactly the failure mode this design avoids. The second: make Kodi a playback target *for* Music Assistant, so MA could drive playback directly. The one community project for this integration (`MAserver-Kodi-Provider`) is archived and abandoned — betting a working setup on dead third-party code wasn't worth it.
What's left is the actual design: Kodi remains the single source of truth for everything a listener touches (library, ratings, playback), and MA is reduced to a disposable, stateless recommendation service — it could be deleted and rebuilt from an empty config tomorrow without losing a single rating or play count.
## How A Refill Cycle Actually Works
`kodi-ma-bridge` is a small asyncio daemon, deployed as part of the same `kodi` stack as Kodi's MariaDB (not a separate stack — the two were merged once the bridge was built, since the bridge only exists to serve Kodi):
```
$ docker ps --filter name=kodi --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"
NAMES STATUS IMAGE
kodi-ma-bridge Up 18 minutes python:3.13-slim
kodi-mariadb Up 38 hours (healthy) <fleet registry>/mariadb:latest
```
Every 60 seconds (`POLL_INTERVAL_SECONDS=60`), it checks each configured Kodi instance:
```
$ docker exec kodi-ma-bridge env | grep -E "POLL_INTERVAL|REFILL|MA_BASE_URL|KODI_INSTANCES"
KODI_INSTANCES=[personal-device-1]:<host>:8115,[personal-device-2]:<host>:8115
MA_BASE_URL=http://music-assistant:8095
POLL_INTERVAL_SECONDS=60
REFILL_COUNT=10
REFILL_THRESHOLD=5
```
1. Ask Kodi's JSON-RPC API (`Player.GetActivePlayers`) whether anything's playing. If the instance is unreachable or idle, skip silently — these are personal desktops, not always-on servers, so "nothing there right now" is normal, not an error condition.
2. If something's playing, check how many tracks remain after the current position (`Playlist.GetItems` + `Player.GetProperties`). Above `REFILL_THRESHOLD` (5 remaining), stop here — nothing to do yet.
3. Below the threshold: read the current track's artist/title and resolve it to an MA library item via `music/search`.
4. Ask MA's `music/tracks/similar_tracks` for a large candidate pool, filter out anything already queued or recently added, translate MA's library-relative path to Kodi's SMB path, and push up to `REFILL_COUNT` (10) new tracks straight into the live playlist via `Playlist.Add`.
That queue-length gate is the one real branch in the whole design — everything else is a straight pipeline — so it's worth seeing as a diagram rather than another paragraph of prose:
![kodi-ma-bridge refill cycle: poll each Kodi instance, check remaining queue length, and only resolve+inject new tracks when the queue actually drops below threshold](../assets/kodi-music-assistant-bridge-refill-cycle.svg)
## Three Real Issues, Found By Actually Testing It
The bridge sat deployed and silently non-functional for weeks before this. All three of the following were found by doing the one thing that actually proves a system works: playing real music on a real machine and watching what happened, not just exercising the API calls in isolation.
**1 — the daemon never once successfully resolved a track.** Kodi's own JSON-RPC API wraps every response in a `result` key, so the bridge's generic API-calling helper was written to unwrap it the same way for *every* call, including the ones to MA. MA's REST API doesn't use that envelope at all — it returns the payload flat, at the top level:
```
$ curl -s -X POST http://music-assistant:8095/api -H "Content-Type: application/json" \
-d '{"message_id":"t","command":"music/search","args":{"search_query":"Oasis - Champagne Supernova","media_types":["track"],"limit":1,"library_only":true}}' \
| python3 -c "import json,sys; d=json.load(sys.stdin); print('top-level keys:', list(d.keys())); print('has result key:', 'result' in d)"
top-level keys: ['artists', 'albums', 'genres', 'tracks', 'playlists', 'radio', 'audiobooks', 'podcasts', 'sound_effects']
has result key: False
```
`data.get("result")` against that shape silently returns `None` every time — no exception, no log line, just a daemon that looked alive and never did anything. Confirmed live against a real Kodi playback session before touching a line of code; fixed by returning MA's response body directly instead of unwrapping a key that was never there.
**2 — even after the fix, MA only ever offered one real match for a mainstream track.** The bridge asked MA for a small candidate pool (15 tracks) and assumed that was the full ranked result. It isn't — MA runs its own internal similarity-distance filter over whatever raw pool size you request, so most of a small batch gets discarded before it ever reaches the caller:
```
limit=10 -> 1 match
limit=15 -> 1 match (the bridge's original request size)
limit=20 -> 1 match
limit=30 -> 2 matches
limit=50 -> 5 matches
limit=100 -> 6 matches
limit=150 -> 8 matches
limit=200 -> 10 matches
```
The real candidate pool for that seed track was there the whole time — a solid run of mid-90s Britpop and alt-rock (The Verve, The Cranberries, Ocean Colour Scene, Beck, Manic Street Preachers) — it just never got a large enough raw pool to survive MA's own filtering. Fixed by requesting 200 candidates up front; the bridge's own `REFILL_COUNT` cap still bounds how many actually get queued.
**3 — a real blind-spot window between the queue running dry and the bridge noticing.** With both of the above fixed and the daemon genuinely working, the original 4-minute poll interval turned out to be too coarse on its own: during one active listening session the live queue hit 0 remaining tracks twice before the next scheduled poll caught it and refilled —
```
[personal-device] queue low (0 left), added 10 tracks seeded from 'Primal Scream - Shine Like Stars'
[personal-device] queue low (0 left), added 10 tracks seeded from 'Kula Shaker - Govinda'
```
`REFILL_THRESHOLD=5` only guarantees a refill *attempt* starts with 5 tracks of runway left — it says nothing about how long that runway lasts in wall-clock time, and a 4-minute gap between checks was long enough, on short tracks, to burn through that runway before the next poll ever ran. Playback itself never actually stopped in practice, but the margin was thinner than the design assumed. Fixed by shortening the poll interval to 60 seconds rather than widening the threshold — each idle poll is a handful of cheap JSON-RPC calls, so checking 4x more often costs effectively nothing.
## Proof It Actually Works
Confirmed end-to-end against a live Kodi session, mid-album, queue genuinely running low:
```
[personal-device] queue low (2 left), added 8 tracks seeded from 'Oasis - Champagne Supernova'
```
And the live playlist, read directly from Kodi right after:
```
$ curl -s -u <user>:*** -X POST -H "Content-Type: application/json" \
-d '{"jsonrpc":"2.0","method":"Playlist.GetItems","params":{"playlistid":0,"properties":["title","artist"]},"id":1}' \
http://<host>:8115/jsonrpc
...
02. The Verve - Sonnet
The Day We Caught the Train
Dreams
Loser
You Stole the Sun From My Heart
Movin' On Up
Blue Orchid
Buck Rogers
Reptilia
Use Somebody
```
A real, coherent 90s/00s indie-and-alt-rock run, auto-queued with zero manual input, directly behind the Oasis album that was already playing:
![Kodi's playlist view, showing a long auto-queued run of similar indie/alt-rock tracks the bridge added on its own once the queue ran low](../assets/kodi-music-assistant-bridge-live-queue.png)
## Similarity Engine
Matching is powered by MA's `sonic_analysis` provider (on-device CLAP audio-embedding analysis — no cloud service, no internet dependency for recommendations) feeding the `sonic_similarity` plugin, confirmed enabled against the running library:
```
$ docker exec music-assistant python3 -c "..."
sonic_analysis -> enabled: True
sonic_similarity -> enabled: True
filesystem_local -> enabled: True
```
The library syncs every 12 hours and diffs the file list, so a replaced file (an upgraded rip, say) is picked up as a new file and the stale entry pruned automatically — no manual re-indexing required.
## Secrets
Both the Kodi JSON-RPC password and the MA API token are Vault-sourced via the fleet's standard 3-file docker-secrets pattern (see [`docs/secret-distribution.md`](secret-distribution.md)) — nothing hardcoded in the compose file or the bridge script itself.
docs/netbird-zero-trust.md
+5 -6
View File
@@ -1,6 +1,6 @@
<!-- verified: 2026-09-19
netbird-lxc@ea8967e : netbird/conf/Caddyfile, netbird/docker-compose.yaml, netbird/config.yaml.ctmpl
iac@ae315de : ansible/roles/secure_vault/netbird_agent, terraform/containers.tf
<!-- verified: 2026-09-06
netbird-lxc@13b5d80 : netbird/Caddyfile, netbird/docker-compose.yaml, netbird/config.yaml.ctmpl
iac@49ff427 : ansible/roles/secure_vault/netbird_agent, terraform/containers.tf
trilium : qWLU4xqYLdJ3, oYuIjaczDrZA, nAPJq9A1iuX8, Cl4TWBZ7ytYr
-->
@@ -56,10 +56,9 @@ The real path a request takes:
And this is the actual splitter doing that work — the real config, not a description of the logic:
```
$ cat conf/Caddyfile
$ cat Caddyfile
{
# Container loopback only (port not published) - `caddy reload` needs it.
admin localhost:2019
admin off
auto_https off
servers {
protocols h1 h2 h2c
docs/notifications.md
+10 -10
View File
@@ -1,6 +1,6 @@
<!-- verified: 2026-09-19
<!-- verified: 2026-09-06
automation-vm@7c057de : mosquitto
iac@ae315de : ansible/roles/monitoring/mqtt_client
iac@49ff427 : ansible/roles/monitoring/mqtt_client
trilium : tt4idCVJPpZM, EAU5LUwFFQ0E, rGfAb4CddKMi, otSPaU6IfjlP, klrmnlh3Wrnd
-->
@@ -11,14 +11,14 @@
## Fan-in, Fan-out
![Three independent fan-in sources converging on one shared formatting subflow: 14 apps over HTTP webhook go through the App Router and Switch to a per-app mapper, Kopia/Bazarr post to their own dedicated endpoints and bypass the router entirely with their own mapper, and every host publishes over MQTT into a topic-agnostic Inventory Parse function — all three land on the same subflow before reaching Slack](../assets/notifications-fan-in-fan-out.svg)
![Three independent fan-in sources converging on one shared formatting subflow: 15 apps over HTTP webhook go through the App Router and Switch to a per-app mapper, Kopia/Bazarr post to their own dedicated endpoints and bypass the router entirely with their own mapper, and every host publishes over MQTT into a topic-agnostic Inventory Parse function — all three land on the same subflow before reaching Slack](../assets/notifications-fan-in-fan-out.svg)
## Why One Node-RED Instance Instead Of Per-App Scripts
The alternative — each app posting directly to a per-app Slack webhook, or a small script per integration — was rejected because it would mean N different message formats, N different places to add a new integration's logic, and no single point to add fleet-wide behavior (like the OS-level checks below) without touching every app individually. One Node-RED instance with a shared router and a shared formatting subflow means a new app's alerts get the same visual treatment as every existing one automatically, and a fleet-wide change (say, adding a severity-based routing rule) happens in one place.
> [!WARNING]
> **Real debt in the flow, one instance already cleaned up**: NetBox's dead detection code — left behind after the app was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) — has since been swept out of the flow entirely. What's still live: Kestra's alerts actually travel a completely different path than its own webhook route suggests — Kestra publishes its own alerts over MQTT, not HTTP, which makes its dedicated webhook route effectively dead code. A couple of newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own onboarding pattern for a new app's alerts.
> **Real debt in the flow**: of the apps wired into the router, two no longer resolve to anything — NetBox was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) but its detection code was never swept out, and Kestra's alerts actually travel a completely different path than its own webhook route suggests: Kestra publishes its own alerts over MQTT, not HTTP, which makes its dedicated webhook route effectively dead code. A couple of newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own onboarding pattern for a new app's alerts.
This is the real routing table, not a claimed count — counted directly from the flow export, not restated from memory:
@@ -31,11 +31,11 @@ print('Switch rules:', len(switch['rules']))
dedicated = [n for n in flows if n.get('type')=='http in' and n.get('url','').endswith('-webhook') and n.get('url') != '/app-webhook']
print('Dedicated non-router endpoints:', len(dedicated))
"
Switch rules: 15
Switch rules: 16
Dedicated non-router endpoints: 2
```
15 rules = the 14 routed apps plus one `unknown` fallback that just drops the payload. And this is the real function behind the "completely different path" line above — the MQTT branch doesn't parse `msg.topic` looking for a known set of fleet hosts, it just takes whatever the second path segment is:
16 rules = the 15 routed apps plus one `unknown` fallback that just drops the payload. And this is the real function behind the "completely different path" line above — the MQTT branch doesn't parse `msg.topic` looking for a known set of fleet hosts, it just takes whatever the second path segment is:
```
$ grep -A4 '"Inventory Parse Function' flows.json
@@ -48,9 +48,9 @@ return msg;
Topic-agnostic by construction — so a publisher that isn't a fleet host at all gets handled identically to one that is, which is exactly the mechanism that makes an app's *own* MQTT-published alerts real and its dedicated webhook route genuinely dead, not just theoretically bypassable.
This is the real flow, not a redrawn version of it — the App Router feeding a 15-way Switch, Kestra's mapper function still sitting there wired in and unreachable, Kopia and Bazarr's dedicated endpoints bypassing all of it entirely on their way to the same shared subflow:
This is the real flow, not a redrawn version of it — the App Router feeding a 16-way Switch, NetBox's and Kestra's mapper functions still sitting there wired in and unreachable, Kopia and Bazarr's dedicated endpoints bypassing all of it entirely on their way to the same shared subflow:
![Real Node-RED canvas: HTTP-in webhook through App Router into a 15-rule Switch function feeding per-app mapper functions (including kestra mapper, still wired but dead since its real alerts travel over MQTT instead), MQTT-in through the Inventory Parse Function, and Kopia/Bazarr's own dedicated HTTP-in nodes bypassing the router entirely — all converging on the shared MQTT → Slack subflow](../assets/notifications-node-red-flow.png)
![Real Node-RED canvas: HTTP-in webhook through App Router into a 16-rule Switch function feeding per-app mapper functions (including netbox mapper and kestra mapper, both still wired but dead), MQTT-in through the Inventory Parse Function, and Kopia/Bazarr's own dedicated HTTP-in nodes bypassing the router entirely — all converging on the shared MQTT → Slack subflow](../assets/notifications-node-red-flow.png)
## Message Broker
@@ -104,10 +104,10 @@ The Node-RED App Router figures out which app sent a webhook by inspecting the p
>
> ```
> $ grep -oE '\$STATE_DIR/[a-zA-Z_]+' mqtt-monitor.sh.j2 | sed -E 's/(_alert|_last_count|_since|_report)$//' | sort -u | wc -l
> 23
> 22
> ```
>
> Of that 23, one (`failed_units_current`) is a companion state file for the same "failed systemd units" check as `failed_units`, not a distinct 23rd check — so 22 real checks, plus disk-usage thresholds, which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name, still makes 23 total. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts.
> 22 from that count, plus disk-usage thresholds — which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name — makes 23. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts.
## One Formatter, Every Source
docs/observability.md
+3 -5
View File
@@ -1,5 +1,5 @@
<!-- verified: 2026-09-19
iac@ae315de : ansible/roles/monitoring/observability_agent
<!-- verified: 2026-09-06
iac@49ff427 : ansible/roles/monitoring/observability_agent
trilium : eeiUz4iTEgPL, zquji4XOpw5M, s5tWZshDEpQb, oWr1jJhvqitv
-->
@@ -48,9 +48,7 @@ SystemKeepFree=1G
MaxRetentionSec=14day
```
![Four independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, the DNS query log as job=technitium-dns with real event time instead of ingestion time, and auditd's own log as job=auditd on the 6 hosts that carry it](../assets/observability-log-fanout.svg)
A fourth, newer branch ships raw auditd logs to Loki too, on the 6 hosts that carry the `auditd_hardening` compliance role (automation, caddy, monitoring, technitium, vault, wazuh) — a real gap-fill after one host's own `/var/log/audit` once filled to 100% with zero external visibility to catch it. It doesn't replace the local cleanup timer that actually bounds that disk usage; it just means the data isn't only ever local anymore.
![Three independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, and the DNS query log as job=dns with real event time instead of ingestion time](../assets/observability-log-fanout.svg)
This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance:
docs/orchestration.md
+3 -3
View File
@@ -1,5 +1,5 @@
<!-- verified: 2026-09-19
ansible@24e1bf0 : host_vars/hyper2.yml, roles/os_update/tasks/reboot.yml, playbooks/os_update_pve_verify.yml, playbooks/automation_self_update.yml
<!-- verified: 2026-09-07
ansible@9a4ba6d : host_vars/hyper2.yml, roles/os_update/tasks/reboot.yml, playbooks/os_update_pve_verify.yml, playbooks/automation_self_update.yml
trilium : bW84ccRoAJ5i, tt4idCVJPpZM
-->
@@ -98,7 +98,7 @@ The automation host's own OS patches and Kestra's own container updates run via
```
$ systemctl list-timers automation-self-update.timer
NEXT LEFT LAST PASSED UNIT
Sat 2026-09-26 04:00:00 EEST 6 days Sat 2026-09-19 04:00:09 EEST 5h ago automation-self-update.timer
Sat 2026-09-12 04:00:00 EEST 6 days Sat 2026-09-05 04:00:32 EEST 4h 17min automation-self-update.timer
```
---
docs/registry-and-build.md
+6 -7
View File
@@ -1,6 +1,6 @@
<!-- verified: 2026-09-19
ansible@24e1bf0 : host_vars/rocky-docker/docker_stacks.yml, host_vars/netbird/docker_stacks.yml, roles/container_update, playbooks/container_update.yml
iac@ae315de : ansible/roles/secure_vault/vault_agent/defaults/main.yml
<!-- verified: 2026-09-06
ansible@ccbda37 : host_vars/rocky-docker/docker_stacks.yml, roles/container_update, playbooks/container_update.yml
iac@49ff427 : ansible/roles/secure_vault/vault_agent/defaults/main.yml
trilium : dcRUArk5dWH8, bW84ccRoAJ5i, hWiIGZIlvQiN
-->
@@ -13,7 +13,7 @@
|---|---|
| **Registry** | Gitea's own built-in container registry |
| **Trigger** | Weekly schedule via Kestra, not push-triggered |
| **Rollout** | All 6 Docker hosts converted; the largest has 35 of 41 stacks on this pattern |
| **Rollout** | All 5 Docker hosts converted; the largest has 36 of 41 stacks on this pattern |
## Why Self-host The Registry Instead Of Just Using Docker Hub
@@ -39,15 +39,14 @@ Not every stack is on this pattern, and that's a decision, not an oversight —
```
$ python3 -c "import yaml; d=yaml.safe_load(open('docker_stacks.yml'))['stacks']; ..."
total: 41
dockerfile 35
pinned_manual 4
dockerfile 36
pinned_manual 3
latest 2
```
| Stack | Why it's held out |
|---|---|
| Authentik, Immich, tdarr | Complex multi-service upstream builds — not worth replicating a build process the upstream project already maintains well; the maintenance cost of tracking their own build changes would exceed the benefit |
| Music-assistant | Was on this pattern, deliberately reverted — auto-updating `:latest` kept bumping its `smart_fades`/`loudness_analysis` `analysis_version`, mass-invalidating ~86k tracks' audio-analysis backlog each time it moved; now pinned by hand, bumped only after checking release notes |
| Kodi's own app image | Genuinely still a direct Docker Hub dependency — low enough stakes that the extra build step isn't worth it |
| Nextcloud's own `nextcloud`/`euro-office` services | Pre-existing custom Dockerfiles, left as-is beyond adding `image:` references — its own db/redis still float on Docker Hub tags directly, a conscious "don't touch what already works" call |
docs/secret-distribution.md
+4 -4
View File
@@ -1,6 +1,6 @@
<!-- verified: 2026-09-19
iac@ae315de : ansible/roles/secure_vault/ssh_ca_client, ansible/roles/secure_vault/vault_agent/defaults/main.yml, ansible/roles/secure_vault/SecureVault – Vault Contract.md, ansible/roles/secure_vault/securevault-vault-facts.md
ansible@24e1bf0 : roles/app_update/defaults/main.yml
<!-- verified: 2026-09-06
iac@49ff427 : ansible/roles/secure_vault/ssh_ca_client, ansible/roles/secure_vault/vault_agent/defaults/main.yml, ansible/roles/secure_vault/SecureVault – Vault Contract.md, ansible/roles/secure_vault/securevault-vault-facts.md
ansible@ccbda37 : roles/app_update/defaults/main.yml
trilium : DvKxK0gHXsng, wovVrgvoDVy2, GW8BMoTKtRBe, 1seDR2nBmxKO
-->
@@ -80,7 +80,7 @@ That redeploy-and-restart sequence isn't just a claim — here's a real one, sta
**Shared-secret variant:** not every template is host-scoped — one credential needed by every docker host for its own registry pushes (see [`docs/registry-and-build.md`](registry-and-build.md)) is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one — and it relies on every host's policy already granting read access to that shared path, the same "shared + private" pattern described in [`docs/secrets-and-pki.md`](secrets-and-pki.md)'s access model. This is the real template list entry, repeated per host:
```
$ grep -B1 -A3 "registry-token.ctmpl" vault_agent/defaults/main.yml
$ grep -A4 "service: automation" vault_agent/defaults/main.yml
- service: automation
src_path: .../automation-vm/registry-token/registry-token.ctmpl
dest_path: /opt/docker/docker_volumes/registry-token/registry-token
docs/secrets-and-pki.md
+3 -4
View File
@@ -1,6 +1,6 @@
<!-- verified: 2026-09-19
iac@ae315de : ansible/roles/secure_vault, ansible/group_vars/all/vault_provisioner.yml, ansible/roles/monitoring/mqtt_client
rocky-docker-lxc@d225776 : caddy/conf/Caddyfile
<!-- verified: 2026-09-06
iac@49ff427 : ansible/roles/secure_vault, ansible/group_vars/all/vault_provisioner.yml, ansible/roles/monitoring/mqtt_client
rocky-docker-lxc@4f43dcc : caddy/Caddyfile
trilium : Eu0wns00QOBG, wovVrgvoDVy2, 1seDR2nBmxKO, GW8BMoTKtRBe, JbKe9CHzOP9D, Tu7FX48fxSQZ, DvKxK0gHXsng, otSPaU6IfjlP
-->
@@ -68,7 +68,6 @@ Every Vault Agent reaches Vault through the internal app-to-app proxy zone descr
| User-Facing Proxy | Server + client certs for the mTLS-enforced browser zone, plus a shared registry-token render and Hawser's application secret |
| Internal DNS | Server cert for its own admin UI/API only |
| Security Monitoring | Its manager and dashboard services' shared secrets (indexer, API, dashboard passwords), plus Hawser's application secret, plus a shared registry-token render |
| NetBird | Its entire server config rendered as one file, embedded secrets included — the same file-templating pattern used for Kestra's `application.yml` — plus Hawser's application secret, plus a shared registry-token render |
None of these ever call the Vault API themselves — every one of them goes through the same local-render sequence:
docs/security-monitoring.md
+2 -4
View File
@@ -1,5 +1,5 @@
<!-- verified: 2026-09-19
iac@ae315de : ansible/roles/monitoring/wazuh_agent_tuning, ansible/roles/monitoring/docker_bench, ansible/roles/cis_hardening
<!-- verified: 2026-09-06
iac@49ff427 : ansible/roles/monitoring/wazuh_agent_tuning, ansible/roles/monitoring/docker_bench, ansible/roles/cis_hardening
wazuh-vm@07ed482 : wazuh/docker-compose.yml
trilium : KpvtX1USDhkY, TstjWVV5Q1TS, OtIgVyQC5lJg, MSVD5o5HiJ69, kA00FQiQWVem, oY5CbyuxZRyS
-->
@@ -57,8 +57,6 @@ The alternative to an agent on every host is a purely network-level monitoring a
> # during that night's role redeploys, confirmed via the indexer across
> # 9 of the fleet's agents.
> ```
>
> The same role found a second gap a month later, in itself this time: it silently skipped tuning the wazuh host's own agent, because that agent runs inside the manager's Docker container rather than natively on the VM — the role's own "is the agent installed" check only ever looked for a native install. That let the wazuh host's own queue sit at the untuned default until a real queue-full alert exposed it, the same class of bug the role was originally written to fix, this time hiding in the one host built differently from the rest. Fixed by giving that host its own branch instead of assuming every agent looks the same.
## How An Alert Gets Handled
docs/sso-and-identity.md
+6 -19
View File
@@ -1,7 +1,6 @@
<!-- verified: 2026-09-18
rocky-docker-lxc@d225776 : caddy/conf/Caddyfile
caddy-vm@c5c3420 : caddy/conf/Caddyfile
iac@ae315de : ansible/roles/bootstrap_roles/docker_git_setup
<!-- verified: 2026-09-06
rocky-docker-lxc@4f43dcc : caddy/Caddyfile
caddy-vm@9369f8b : caddy/Caddyfile
trilium : baWhWTUejvY6, hWiIGZIlvQiN, nAPJq9A1iuX8
-->
@@ -61,7 +60,7 @@ This is the actual live query against Authentik's API, not a paraphrase of what'
```
$ curl -s -H "Authorization: Bearer [REDACTED]" \
"https://authentik.[service-to-service zone]/api/v3/providers/proxy/?page_size=1" | jq '.pagination.count'
94
96
$ curl -s -H "Authorization: Bearer [REDACTED]" \
"https://authentik.[service-to-service zone]/api/v3/providers/proxy/?page_size=200" | jq -r '.results[].name' | grep -E '^proxmox'
[no output — Proxmox has no Proxy Provider at all]
@@ -92,23 +91,11 @@ So an app's OIDC discovery/authority setting always points at the service-to-ser
## Git Access: least privilege, not the admin account
Automation should never run with more privilege than it needs. CI and orchestration pushes (the flows described in [`docs/orchestration.md`](orchestration.md)) run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. This wasn't the original setup: automated pushes used to run through the personal admin account used for interactive access, the same one with full admin rights over the Git server. That got fixed by standing up a separate account with its own SSH keypair and adding it as a write collaborator on every repo it touches — routine automated commits no longer run under an admin credential at all. The 5 docker hosts that only ever pull (never push) use that same bot account's key too — copied identically to each host at first provisioning, not a separate credential per host. The practical payoff: if an automation credential ever leaks, the blast radius is "can push to the repos this bot account touches," not "has full admin control of the Git server."
Automation should never run with more privilege than it needs. CI and orchestration pushes (the flows described in [`docs/orchestration.md`](orchestration.md)) run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. This wasn't the original setup: automated pushes used to run through the personal admin account used for interactive access, the same one with full admin rights over the Git server. That got fixed by standing up a separate account with its own SSH keypair and adding it as a write collaborator on every repo it touches — routine automated commits no longer run under an admin credential at all. Hosts that only need to *read* get separate read-only deploy keys instead of any shared credential. The practical payoff: if an automation credential ever leaks, the blast radius is "can push to specific repos," not "has full admin control of the Git server."
![Gitea repo collaborators: automation-bot with Write access, not an admin account](../assets/gitea-collaborators-automation-bot.png)
This is the real provisioning task, not a description of it — the automation-bot key gets copied to every docker host identically:
```
# The automation-bot private key - authenticates the push to Gitea AND signs
# commits (ssh_key_path == ssh_signing_key_path). Copied from the control node.
- name: Install the automation-bot git key
copy:
src: "{{ ssh_key_path }}"
dest: "{{ ssh_key_path }}"
owner: "{{ git_repo_owner }}"
group: "{{ git_repo_group }}"
mode: "0600"
```
![Gitea repo deploy keys: a host-specific key scoped to Read only](../assets/gitea-deploy-keys.png)
---
Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md) · Stacks: [`stacks/identity-and-security/authentik/`](../stacks/identity-and-security/authentik/), [`stacks/platform/gitea/`](../stacks/platform/gitea/)
stacks/platform/caddy-internal/docker-compose.yaml
+2 -2
View File
@@ -6,11 +6,11 @@ services:
ports:
- "${HTTPS_PORT}:443"
volumes:
- ./conf:/etc/caddy:ro
- ./Caddyfile:/etc/caddy/Caddyfile
- /path/to/caddy-internal/caddy_data:/data
- /path/to/caddy-internal/caddy_config:/config
- /path/to/caddy-internal/logs:/var/log/caddy
- ./certs:/etc/caddy/certs:ro
- ./certs/pki_int_ca.crt:/etc/caddy/certs/pki_int_ca.crt:ro
networks:
- caddy-net
stacks/platform/caddy/docker-compose.yaml
+1 -1
View File
@@ -6,7 +6,7 @@ services:
ports:
- "${HTTPS_PORT}:443"
volumes:
- ./conf:/etc/caddy:ro
- ./Caddyfile:/etc/caddy/Caddyfile
- /path/to/caddy/caddy_data:/data
- /path/to/caddy/caddy_config:/config
- /etc/caddy/certs:/etc/caddy/certs