Merge pull request 'docs: iac commit IDs after its history rewrite (ISO removed, content unchanged)' (#3) from iac-history-ids into main

Covers the design rationale, the refill-cycle mechanism, three real
issues found and fixed via live testing against a real Kodi session
(a silent result-key mismatch that meant the daemon never once
resolved a track, a starved similarity candidate pool, and a
4-minute poll-interval blind spot), and live proof of a working
auto-queued run of similar tracks.

.gitea/workflows/ci.yml

@@ -0,0 +1,15 @@
+# CI for this repo - the actual checks live in the shared ci-workflows repo.
+name: ci
+
+"on":
+  push:
+    branches: [main]
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  gitleaks:
+    # Short owner/repo form: Gitea resolves reusable workflows server-side and
+    # rejects absolute URLs that don't match its own ROOT_URL.
+    uses: thesabear/ci-workflows/.gitea/workflows/gitleaks.yml@main
+    secrets: inherit

.gitleaksignore

@@ -0,0 +1,7 @@
+# Findings checked by hand and confirmed dead - each line ignores exactly one
+# finding (commit:file:rule:line). Anything else gitleaks finds still fails CI.
+
+# Paperless OIDC client secret in old history (removed from the files 2026-07-06).
+# Checked 2026-10-11: that client_id no longer exists in Authentik (all 104
+# OAuth2 providers), and Paperless uses a different client and secret.
+973546647df3fe505e913c42af9ed3547083593c:Docker/Paperless-NGX/docker-compose.yaml:generic-api-key:50

README.md

@@ -93,6 +93,10 @@ Every physical host, VM/LXC, and running service in the fleet, re-derived from t
 Packer builds a hardened base image, Terraform provisions each VM from it, and Ansible configures and continuously re-hardens every host after that — three tools, three separate responsibilities, connected by one tag that threads through all three stages instead of one tool trying to do every job. Terraform's part in that link ends at creation — it writes the tags file once, and that file then drives Ansible independently for the rest of that host's life.
 → [docs/iac-pipeline.md](docs/iac-pipeline.md)
 
+### 🎧 Kodi <-> Music Assistant Smart Queue Bridge
+Kodi stays the only music player, UI, and ratings authority; Music Assistant runs invisibly behind it as a similarity engine, never a player, never writing anything back to the library. A small daemon watches Kodi's live queue and auto-refills it with similar tracks via MA's recommendation API when it runs low. Three real issues surfaced only once it was tested against a live Kodi session — not exercised in isolation — including one where the daemon had silently never worked at all.
+→ [docs/kodi-music-assistant-bridge.md](docs/kodi-music-assistant-bridge.md)
+
 ### 🔔 Notifications
 Every alert in the fleet — from apps, from hosts, from security events — funnels through one Node-RED instance and one shared formatter before reaching Slack, so a notification reads the same regardless of which of a dozen-plus sources triggered it. A dedicated role puts every host onto the same pipeline for OS-level health, independent of any single application. One shared endpoint handles every app's webhook regardless of shape — a source field where an app provides one, structural fingerprinting of the payload where it doesn't — instead of standing up a dedicated integration per app.
 → [docs/notifications.md](docs/notifications.md)
@@ -135,6 +139,12 @@ A self-hosted WireGuard mesh gives every fleet host and personal device its own
 
 ---
 
+## Cheatsheets
+
+Quick-reference material for tools I use day-to-day — command references and config notes, not full write-ups. See [`cheatsheets/`](cheatsheets/).
+
+---
+
 ## Scripts & Tooling
 
 A few standalone scripts live in [`projects/`](projects/) alongside the Docker stacks — real tooling, not demo code:

assets/iac-pipeline-ansible-stages.svg

@@ -1,4 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!-- Do not edit this file with editors other than draw.io -->
 <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="932px" height="318px" viewBox="0 0 932 318" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Ansible Stages&quot; id=&quot;iac-ansiblestages-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1080&quot; pageHeight=&quot;360&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;tags&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Per-host tag list&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;from Terraform&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;70&quot; width=&quot;200&quot; x=&quot;20&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;init&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Initialization stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;runs once, at creation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;20&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;prov&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Provisioning stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;every run — 1 role today, one host&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;comp&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Compliance stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;re-applied every run&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;sub&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;9 independently-tagged&amp;lt;br&amp;gt;hardening sub-roles&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;720&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;init&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;prov&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;comp&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e4&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;comp&quot; style=&quot;html=1;endArrow=block;strokeColor=#82b366;strokeWidth=2;rounded=1;&quot; target=&quot;sub&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quoLine truncated
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="932px" height="318px" viewBox="0 0 932 318" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;Ansible Stages&quot; id=&quot;iac-ansiblestages-1&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1080&quot; pageHeight=&quot;360&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;tags&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Per-host tag list&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;from Terraform&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;70&quot; width=&quot;200&quot; x=&quot;20&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;init&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Initialization stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;runs once, at creation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;20&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;prov&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Provisioning stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;every run — 2 roles today&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;140&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;comp&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Compliance stage&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;re-applied every run&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;360&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;sub&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;9 independently-tagged&amp;lt;br&amp;gt;hardening sub-roles&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;65&quot; width=&quot;220&quot; x=&quot;720&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;init&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;prov&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;tags&quot; style=&quot;html=1;endArrow=block;strokeColor=#d6b656;strokeWidth=2;rounded=1;exitX=1;exitY=0.5;entryX=0;entryY=0.5;&quot; target=&quot;comp&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e4&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;comp&quot; style=&quot;html=1;endArrow=block;strokeColor=#82b366;strokeWidth=2;rounded=1;&quot; target=&quot;sub&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometrLine truncated

assets/kodi-music-assistant-bridge-refill-cycle.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Do not edit this file with editors other than draw.io -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="624px" height="632px" viewBox="0 0 624 632" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;Refill Cycle&quot; id=&quot;0&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;0&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;900&quot; pageHeight=&quot;700&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;poll&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;&quot; value=&quot;Poll each Kodi instance&amp;lt;br&amp;gt;(every 240s)&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;active&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;&quot; value=&quot;Active player + queue&amp;lt;br&amp;gt;length on this instance&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;150&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;gate&quot; parent=&quot;1&quot; style=&quot;rhombus;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontSize=13;&quot; value=&quot;&amp;lt;font style=&amp;quot;font-size: 11px;&amp;quot;&amp;gt;Remaining &amp;amp;gt;&amp;lt;br&amp;gt;REFILL_THRESHOLD (5)?&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;90&quot; width=&quot;190&quot; x=&quot;355&quot; y=&quot;260&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;resolve&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=13;&quot; value=&quot;Resolve current track&amp;lt;br&amp;gt;in MA (music/search)&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;400&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;similar&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#ffe6cc;strokeColor=#d79b00;fontSize=13;&quot; value=&quot;Query similar_tracks&amp;lt;br&amp;gt;(large candidate pool)&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;500&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;inject&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontSize=13;&quot; value=&quot;Filter + translate paths,&amp;lt;br&amp;gt;Playlist.Add up to 10&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;220&quot; x=&quot;340&quot; y=&quot;600&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;poll&quot; style=&quot;edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#333333;&quot; target=&quot;active&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;active&quot; style=&quot;edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#333333;&quot; target=&quot;gate&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;gate&quot; style=&quot;edgeStyle=orthogonalEdgeStyle;rounded=0;html=1;strokeColor=#b85450;exitX=1;exitY=0.5;labelBackgroundColor=#1C1D1E;exitDx=0;exitDy=0;&quot; target=&quot;poll&quot; value=&quot;&amp;lt;span style=&amp;quot;background-color: light-dark(rgb(255, 255, 255), rgb(28, 29, 30));&amp;quot;&amp;gt;yes &amp;amp;mdash; skip&amp;lt;/span&amp;gt;&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; x=&quot;0.0698&quot; y=&quot;-33&quot; as=&quot;geometry&quot;&gt;&#xA;            &lt;mxPoint as=&quot;offset&quot; /&gt;&#xA;            &lt;Array as=&quot;points&quot;&gt;&#xA;              &lt;mxPoint x=&quot;60Line truncated

assets/observability-log-fanout.svg

@@ -1,4 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!-- Do not edit this file with editors other than draw.io -->
 <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="273px" viewBox="0 0 872 273" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Log Fanout&quot; id=&quot;obs-logfanout-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;320&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;a1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;system logs&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=system&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a1&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b1&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;a2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker API&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;container logs, survives recreation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=docker&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a2&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b2&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;a3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;DNS query log&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;daily rotation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=dns, real event time&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a3&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b3Line truncated
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="373px" viewBox="0 0 872 373" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;Log Fanout&quot; id=&quot;obs-logfanout-1&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;420&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;a1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;system logs&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;b1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=system&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a1&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b1&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;a2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker API&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;container logs, survives recreation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;b2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=docker&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a2&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b2&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;a3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;DNS query log&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;daily rotation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;b3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=technitium-dns, real event time&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a3&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; targLine truncated

cheatsheets/README.md

@@ -0,0 +1,13 @@
+# Cheatsheets
+
+Quick-reference material, not deep dives. The [`docs/`](../docs/) directory is
+where the real engineering write-ups live — verified claims, real evidence,
+the actual design decisions behind each system. This directory is lower-key:
+command references and config notes for tools I use day-to-day, kept here so
+they're easy to find and easy to add to.
+
+- **[proxmox/](proxmox/)** — CLI cheat sheet for the hypervisor cluster this
+  fleet runs on (`qm`/`pct`/`pvesm`/`pveam`), plus API token auth and
+  certificate management notes.
+- **[zsh/](zsh/)** — the aliases, functions, and keybindings from my actual
+  zsh config, not a generic zsh tutorial.

cheatsheets/proxmox/proxmox-api-authentication.md

@@ -0,0 +1,18 @@
+## Create an API Token on Proxmox
+
+To create a new API Token for your `user` in Proxmox, follow these steps:
+
+1. Open the Proxmox Web UI and navigate to the **'Datacenter'** in the **'Server View'** menu.
+2. Select **'API Token'** in the **'Permissions'** menu.
+3. Click on **'Add'**.
+4. Select the `user`, and add a **'Token ID'**.
+5. (Optional) Disable `Privilege Separation`, and set an `Expire` date.
+6. Click on **'Add'**.
+
+## Check the API Token
+
+To test the API Token, you can use the following command:
+
+```sh
+curl -H "Authorization: PVEAPIToken=root@pam!monitoring=aaaaaaaaa-bbb-cccc-dddd-ef0123456789" https://your-proxmox-url:8006/api2/json/
+```

cheatsheets/proxmox/proxmox-certificate-management.md

@@ -0,0 +1,324 @@
+## Certificates for Intra-Cluster Communication
+
+Each Proxmox VE cluster creates by default its own (self-signed) Certificate
+Authority (CA) and generates a certificate for each node which gets signed by
+the aforementioned CA. These certificates are used for encrypted communication
+with the cluster's pveproxy service and the Shell/Console feature if SPICE is used.
+
+The CA certificate and key are stored in the Proxmox Cluster File System (pmxcfs).
+
+## Certificates for API and Web GUI
+
+The REST API and web GUI are provided by the pveproxy service, which runs on each node.
+You have the following options for the certificate used by pveproxy:
+
+* By default the node-specific certificate in `/etc/pve/nodes/NODENAME/pve-ssl.pem` is used. This certificate is signed by the cluster CA and therefore not automatically trusted by browsers and operating systems.
+* Use an externally provided certificate (e.g. signed by a commercial CA).
+* Use ACME (Let's Encrypt) to get a trusted certificate with automatic renewal, this is also integrated in the Proxmox VE API and web interface.
+
+For options 2 and 3 the file `/etc/pve/local/pveproxy-ssl.pem` (and
+`/etc/pve/local/pveproxy-ssl.key`, which needs to be without password) is used.
+Keep in mind that `/etc/pve/local` is a node specific symlink to
+`/etc/pve/nodes/NODENAME`.
+
+Certificates are managed with the Proxmox VE Node management command
+(see the `pvenode(1)` manpage). The current node certificate's details can
+be checked directly from the CLI too:
+
+```shell
+pvenode cert info [FORMAT_OPTIONS]
+```
+
+Do not replace or manually modify the automatically generated node
+certificate files in `/etc/pve/local/pve-ssl.pem` and
+`/etc/pve/local/pve-ssl.key` or the cluster CA files in
+`/etc/pve/pve-root-ca.pem` and `/etc/pve/priv/pve-root-ca.key`.
+
+## Upload Custom Certificate
+
+If you already have a certificate which you want to use for a Proxmox VE node
+you can upload that certificate simply over the web interface. Note that the
+certificates key file, if provided, mustn't be password protected.
+
+## Trusted certificates via Let's Encrypt (ACME)
+
+Proxmox VE includes an implementation of the Automatic Certificate
+Management Environment ACME protocol, allowing Proxmox VE admins to
+use an ACME provider like Let's Encrypt for easy setup of TLS certificates
+which are accepted and trusted on modern operating systems and web browsers
+out of the box.
+
+Currently, the two ACME endpoints implemented are the
+Let's Encrypt (LE) production and its staging
+environment. Our ACME client supports validation of http-01 challenges using
+a built-in web server and validation of dns-01 challenges using a DNS plugin
+supporting all the DNS API endpoints `acme.sh` does.
+
+### ACME Account
+
+You need to register an ACME account per cluster with the endpoint you want to
+use. The email address used for that account will serve as contact point for
+renewal-due or similar notifications from the ACME endpoint.
+You can register and deactivate ACME accounts over the web interface
+`Datacenter -> ACME` or using the `pvenode` command line tool.
+
+```shell
+pvenode acme account register account-name [email protected]
+```
+
+Because of rate-limits you should use LE staging for experiments or if you use
+ACME for the first time.
+
+### ACME Plugins
+
+The ACME plugins task is to provide automatic verification that you, and thus
+the Proxmox VE cluster under your operation, are the real owner of a domain. This is
+the basis building block for automatic certificate management.
+
+The ACME protocol specifies different types of challenges, for example the
+http-01 where a web server provides a file with a certain content to prove
+that it controls a domain. Sometimes this isn't possible, either because of
+technical limitations or if the address of a record to is not reachable from
+the public internet. The dns-01 challenge can be used in these cases. This
+challenge is fulfilled by creating a certain DNS record in the domain's zone.
+
+Proxmox VE supports both of those challenge types out of the box, you can configure
+plugins either over the web interface under `Datacenter -> ACME`, or using the
+`pvenode acme plugin add` command.
+
+ACME Plugin configurations are stored in `/etc/pve/priv/acme/plugins.cfg`.
+A plugin is available for all nodes in the cluster.
+
+### Node Domains
+
+Each domain is node specific. You can add new or manage existing domain entries
+under Node -> Certificates, or using the `pvenode config` command.
+
+After configuring the desired domain(s) for a node and ensuring that the
+desired ACME account is selected, you can order your new certificate over the
+web-interface. On success the interface will reload after 10 seconds.
+
+Renewal will happen automatically.
+
+## ACME HTTP Challenge Plugin
+
+There is always an implicitly configured standalone plugin for validating
+http-01 challenges via the built-in webserver spawned on port 80.
+
+The name `standalone` means that it can provide the validation on it's
+own, without any third party service. So, this plugin works also for cluster
+nodes.
+
+There are a few prerequisites to use it for certificate management with Let's
+Encrypts ACME.
+
+* You have to accept the ToS of Let's Encrypt to register an account.
+* Port 80 of the node needs to be reachable from the internet.
+* There must be no other listener on port 80.
+* The requested (sub)domain needs to resolve to a public IP of the Node.
+
+## ACME DNS API Challenge Plugin
+
+On systems where external access for validation via the http-01 method is
+not possible or desired, it is possible to use the dns-01 validation method.
+This validation method requires a DNS server that allows provisioning of TXT
+records via an API.
+
+### Configuring ACME DNS APIs for validation
+
+Proxmox VE re-uses the DNS plugins developed for the
+[acme.sh project](https://github.com/acmesh-official/acme.sh). Please
+refer to its documentation for details on configuration of specific APIs.
+
+The easiest way to configure a new plugin with the DNS API is using the web
+interface (`Datacenter -> ACME`).
+
+Choose DNS as challenge type. Then you can select your API provider, enter
+the credential data to access your account over their API.
+
+See the `acme.sh`
+[How to use DNS API](https://github.com/acmesh-official/acme.sh/wiki/dnsapi#how-to-use-dns-api)
+wiki for more detailed information about getting API credentials for your
+provider.
+
+As there are many DNS providers and API endpoints Proxmox VE automatically generates
+the form for the credentials for some providers. For the others you will see a
+bigger text area, simply copy all the credentials KEY=VALUE pairs in there.
+
+## DNS Validation through CNAME Alias
+
+A special alias mode can be used to handle the validation on a different
+domain/DNS server, in case your primary/real DNS does not support provisioning
+via an API. Manually set up a permanent CNAME record for
+`_acme-challenge.domain1.example` pointing to `_acme-challenge.domain2.example`
+and set the alias property in the Proxmox VE node configuration file to
+`domain2.example` to allow the DNS server of `domain2.example` to validate all
+challenges for `domain1.example`.
+
+### Combination of Plugins
+
+Combining http-01 and dns-01 validation is possible in case your node is
+reachable via multiple domains with different requirements / DNS provisioning
+capabilities. Mixing DNS APIs from multiple providers or instances is also
+possible by specifying different plugin instances per domain.
+Accessing the same service over multiple domains increases complexity and
+should be avoided if possible.
+
+## Automatic renewal of ACME certificates
+
+If a node has been successfully configured with an ACME-provided certificate
+(either via `pvenode` or via the GUI), the certificate will be automatically
+renewed by the `pve-daily-update.service`. Currently, renewal will be attempted
+if the certificate has expired already, or will expire in the next 30 days.
+A certificate can also be renewed directly, on demand:
+
+```shell
+pvenode acme cert renew [OPTIONS]
+```
+
+## ACME Examples with pvenode
+
+*Example*: Sample `pvenode` invocation for using Let's Encrypt certificates
+
+```sh
+root@proxmox:~# pvenode acme account register default [email protected]
+Directory endpoints:
+0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
+1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
+2) Custom
+Enter selection: 1
+Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
+Do you agree to the above terms? [y|N]y
+...
+Task OK
+root@proxmox:~# pvenode config set --acme domains=example.invalid
+root@proxmox:~# pvenode acme cert order
+Loading ACME account details
+Placing ACME order
+...
+Status is 'valid'!
+All domains validated!
+...
+Downloading certificate
+Setting pveproxy certificate and key
+Restarting pveproxy
+Task OK
+```
+
+*Example*: Setting up the OVH API for validating a domain
+
+The account registration steps are the same no matter which plugins are
+used, and are not repeated here.
+
+OVH_AK and OVH_AS need to be obtained from OVH according to the OVH
+API documentation
+
+First you need to get all information so you and Proxmox VE can access the API.
+
+```sh
+root@proxmox:~# cat /path/to/api-token
+OVH_AK=XXXXXXXXXXXXXXXX
+OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
+root@proxmox:~# source /path/to/api-token
+root@proxmox:~# curl -XPOST -H"X-Ovh-Application: $OVH_AK" -H "Content-type: application/json" \
+https://eu.api.ovh.com/1.0/auth/credential  -d '{
+  "accessRules": [
+    {"method": "GET","path": "/auth/time"},
+    {"method": "GET","path": "/domain"},
+    {"method": "GET","path": "/domain/zone/*"},
+    {"method": "GET","path": "/domain/zone/*/record"},
+    {"method": "POST","path": "/domain/zone/*/record"},
+    {"method": "POST","path": "/domain/zone/*/refresh"},
+    {"method": "PUT","path": "/domain/zone/*/record/"},
+    {"method": "DELETE","path": "/domain/zone/*/record/*"}
+]
+}'
+{"consumerKey":"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ","state":"pendingValidation","validationUrl":"https://eu.api.ovh.com/auth/?credentialToken=AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}
+(open validation URL and follow instructions to link Application Key with account/Consumer Key)
+root@proxmox:~# echo "OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ" >> /path/to/api-token
+```
+
+Now you can setup the ACME plugin:
+
+```sh
+root@proxmox:~# pvenode acme plugin add dns example_plugin --api ovh --data /path/to/api_token
+root@proxmox:~# pvenode acme plugin config example_plugin
+```
+
+```
+┌────────┬──────────────────────────────────────────┐
+│ key    │ value                                    │
+╞════════╪══════════════════════════════════════════╡
+│ api    │ ovh                                      │
+├────────┼──────────────────────────────────────────┤
+│ data   │ OVH_AK=XXXXXXXXXXXXXXXX                  │
+│        │ OVH_AS=YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY  │
+│        │ OVH_CK=ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ  │
+├────────┼──────────────────────────────────────────┤
+│ digest │ 867fcf556363ca1bea866863093fcab83edf47a1 │
+├────────┼──────────────────────────────────────────┤
+│ plugin │ example_plugin                           │
+├────────┼──────────────────────────────────────────┤
+│ type   │ dns                                      │
+└────────┴──────────────────────────────────────────┘
+```
+
+At last you can configure the domain you want to get certificates for and
+place the certificate order for it:
+
+```sh
+root@proxmox:~# pvenode config set --acmedomain0 example.proxmox.com,plugin=example_plugin
+root@proxmox:~# pvenode acme cert order
+Loading ACME account details
+Placing ACME order
+Order URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/11111111/22222222
+Getting authorization details from 'https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/33333333'
+The validation for example.proxmox.com is pending!
+[Wed Apr 22 09:25:30 CEST 2020] Using OVH endpoint: ovh-eu
+[Wed Apr 22 09:25:30 CEST 2020] Checking authentication
+[Wed Apr 22 09:25:30 CEST 2020] Consumer key is ok.
+[Wed Apr 22 09:25:31 CEST 2020] Adding record
+[Wed Apr 22 09:25:32 CEST 2020] Added, sleep 10 seconds.
+Add TXT record: _acme-challenge.example.proxmox.com
+Triggering validation
+Sleeping for 5 seconds
+Status is 'valid'!
+[Wed Apr 22 09:25:48 CEST 2020] Using OVH endpoint: ovh-eu
+[Wed Apr 22 09:25:48 CEST 2020] Checking authentication
+[Wed Apr 22 09:25:48 CEST 2020] Consumer key is ok.
+Remove TXT record: _acme-challenge.example.proxmox.com
+All domains validated!
+Creating CSR
+Checking order status
+Order is ready, finalizing order
+valid!
+Downloading certificate
+Setting pveproxy certificate and key
+Restarting pveproxy
+Task OK
+```
+
+### Example: Switching from the staging to the regular ACME directory
+
+Changing the ACME directory for an account is unsupported, but as Proxmox VE
+supports more than one account you can just create a new one with the
+production (trusted) ACME directory as endpoint. You can also deactivate the
+staging account and recreate it.
+
+*Example*: Changing the default ACME account from staging to directory using `pvenode`
+
+```sh
+root@proxmox:~# pvenode acme account deactivate default
+Renaming account file from '/etc/pve/priv/acme/default' to '/etc/pve/priv/acme/_deactivated_default_4'
+Task OK
+root@proxmox:~# pvenode acme account register default [email protected]
+Directory endpoints:
+0) Let's Encrypt V2 (https://acme-v02.api.letsencrypt.org/directory)
+1) Let's Encrypt V2 Staging (https://acme-staging-v02.api.letsencrypt.org/directory)
+2) Custom
+Enter selection: 0
+Terms of Service: https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
+Do you agree to the above terms? [y|N]y
+...
+Task OK
+```

cheatsheets/proxmox/proxmox-cheat-sheet.md

@@ -0,0 +1,426 @@
+Proxmox Virtual Environment (Proxmox VE or PVE) is a hyper-converged infrastructure open-source software. It is a hosted hypervisor that can run operating systems including Linux and Windows on x64 hardware. It is a Debian-based Linux distribution with a modified Ubuntu LTS kernel and allows deployment and management of virtual machines and containers. Proxmox VE includes a web console and command-line tools, and provides a REST API for third-party tools. Two types of virtualization are supported: container-based with LXC (starting from version 4.0 replacing OpenVZ used in version up to 3.4, included), and full virtualization with KVM. It includes a web-based management interface.
+
+Proxmox VE is licensed under the GNU Affero General Public License, version 3.
+
+Repository: [https://git.proxmox.com](https://git.proxmox.com)
+Website: [https://pve.proxmox.com](https://pve.proxmox.com)
+
+## VM Management
+
+```shell
+# list VMs
+qm list
+
+# Create or restore a virtual machine.
+qm create your-vm-id
+
+# start a VM
+qm start your-vm-id
+
+# Suspend virtual machine.
+qm suspend your-vm-id
+
+# shutdown a VM
+qm shutdown your-vm-id
+
+# reboot a VM
+qm reboot your-vm-id
+
+# reset a VM
+qm reset your-vm-id
+
+# stop a VM
+qm stop your-vm-id
+
+# Destroy the VM and all used/owned volumes.
+# Removes any VM specific permissions and firewall rules
+qm destroy your-vm-id
+
+# Enter Qemu Monitor interface.
+qm monitor your-vm-id
+
+# Get the virtual machine configuration with both current and pending values.
+qm pending your-vm-id
+
+# Get the virtual machine configuration with pending changes applied.
+qm config your-vm-id [OPTIONS]
+
+# Send key event to virtual machine.
+qm sendkey your-vm-id your-key-event [OPTIONS]
+
+# Show command line which is used to start the VM (debug info).
+qm showcmd your-vm-id [OPTIONS]
+
+# Unlock the VM.
+qm unlock your-vm-id
+
+# Clone a VM
+qm clone your-vm-id new-vm-id
+
+# Migrate a VM
+qm migrate your-vm-id target-node
+
+# Migrate a VM to a remote cluster. EXPERIMENTAL.
+qm remote-migrate your-vm-id target-endpoint --target-bridge <string> --target-storage <string> [OPTIONS]
+
+# Create a new VM from an OVF manifest.
+qm importovf your-vm-id <manifest> <storage> [OPTIONS]
+
+# Enroll updated UEFI certificates to the EFI disk.
+qm enroll-efi-keys your-vm-id
+
+# Show VM status
+qm status your-vm-id
+
+# Wait until the VM is stopped.
+qm wait your-vm-id [OPTIONS]
+
+# Clean up resources for a VM
+qm cleanup your-vm-id your-clean-shutdown your-guest-requested
+
+# Create a Template.
+qm template your-vm-id [OPTIONS]
+
+# Set virtual machine options (synchrounous API)
+qm set your-vm-id [OPTIONS]
+```
+
+### Cloudinit
+
+```shell
+# Get automatically generated cloudinit config.
+qm cloudinit dump your-vm-id your-vm-type
+
+# Get the cloudinit configuration with both current and pending values.
+qm cloudinit pending your-vm-id
+
+# Regenerate and change cloudinit config drive.
+qm cloudinit update your-vm-id
+```
+
+### Disk
+
+```shell
+# Import an external disk image as an unused disk in a VM.
+# The image format has to be supported by qemu-img(1).
+qm disk import your-vm-id your-target-source your-target-storage
+
+# Move volume to different storage or to a different VM.
+qm disk move your-vm-id your-vm-disk [<storage>] [OPTIONS]
+
+# Rescan all storages and update disk sizes and unused disk images.
+qm disk rescan [OPTIONS]
+
+# Extend volume size.
+qm disk resize your-vm-id your-vm-disk <size> [OPTIONS]
+
+# Unlink/delete disk images.
+qm disk unlink your-vm-id --idlist <string> [OPTIONS]
+
+# rescan volumes
+qm rescan
+```
+
+### Snapshot
+
+```shell
+# List all snapshots.
+qm listsnapshot your-vm-id
+
+# Snapshot a VM
+qm snapshot your-vm-id <snapname>
+
+# Delete a snapshot.
+qm delsnapshot your-vm-id <snapname>
+
+# Rollback a snapshot
+qm rollback your-vm-id <snapname>
+
+# Open a terminal using a serial device
+# (The VM need to have a serial device configured, for example serial0: socket)
+qm terminal your-vm-id [OPTIONS]
+
+# Proxy VM VNC traffic to stdin/stdout
+qm vncproxy your-vm-id
+```
+
+### Misc
+
+```shell
+# Execute Qemu Guest Agent commands.
+qm guest cmd your-vm-id <command>
+
+# Executes the given command via the guest agent
+qm guest exec your-vm-id [<extra-args>] [OPTIONS]
+
+# Gets the status of the given pid started by the guest-agent
+qm guest exec-status your-vm-id <pid>
+
+# Sets the password for the given user to the given password
+qm guest passwd your-vm-id <username> [OPTIONS]
+```
+
+### PV, VG, LV Management
+
+```shell
+# Create a PV
+pvcreate <disk-device-name>
+
+# Remove a PV
+pvremove <disk-device-name>
+
+# List all PVs
+pvs
+
+# Create a VG
+vgcreate <vg-name> <disk-device-name>
+
+# Remove a VG
+vgremove <vg-name>
+
+# List all VGs
+vgs
+
+# Create a LV
+lvcreate -L <lv-size> -n <lv-name> <vg-name>
+
+# Remove a LV
+lvremove <vg-name>/<lv-name>
+
+# List all LVs
+lvs
+```
+
+### Storage Management
+
+```shell
+# Create a new storage.
+pvesm add <type> <storage> [OPTIONS]
+
+# Allocate disk images.
+pvesm alloc <storage> your-vm-id <filename> <size> [OPTIONS]
+
+# Delete volume
+pvesm free <volume> [OPTIONS]
+
+# Delete storage configuration.
+pvesm remove <storage>
+
+# Update storage configuration.
+pvesm set <storage> [OPTIONS]
+
+# List storage content.
+pvesm list <storage> [OPTIONS]
+
+# Get the filesystem path for a specified volume.
+pvesm path <volume>
+
+# Prune backups on a storage (standard naming scheme only).
+pvesm prune-backups <storage> [OPTIONS]
+
+# Extract configuration from a vzdump backup archive.
+pvesm extractconfig <volume>
+
+# Returns the storage API version and age.
+pvesm apiinfo
+
+# An alias for pvesm scan lvm.
+pvesm lvmscan
+
+# An alias for pvesm scan lvmthin.
+pvesm lvmthinscan
+
+# List local LVM volume groups.
+pvesm scan lvm
+
+# List local LVM Thin Pools.
+pvesm scan lvmthin <vg>
+
+# Get status for all datastores.
+pvesm status [OPTIONS]
+```
+
+### Template Management
+
+```shell
+# list all templates
+pveam available
+
+# list all templates
+pveam list <storage>
+
+# Download appliance templates
+pveam download <storage> <template>
+
+# Remove a template.
+pveam remove <template-path>
+
+# Update Container Template Database.
+pveam update
+```
+
+## Certificate Management
+
+See the [Proxmox Certificate Management](proxmox-certificate-management.md) cheat sheet.
+
+## Container Management
+
+```shell
+# List containers
+pct list
+
+# Create or restore a container.
+pct create your-vm-id <ostemplate> [OPTIONS]
+
+# Start the container.
+pct start your-vm-id [OPTIONS]
+
+# Create a container clone/copy
+pct clone your-vm-id new-vm-id [OPTIONS]
+
+# Suspend the container. This is experimental.
+pct suspend your-vm-id
+
+# Resume the container.
+pct resume your-vm-id
+
+# Stop the container.
+# This will abruptly stop all processes running in the container.
+pct stop your-vm-id [OPTIONS]
+
+# Shutdown the container.
+# This will trigger a clean shutdown of the container, see lxc-stop(1) for details.
+pct shutdown your-vm-id [OPTIONS]
+
+# Destroy the container (also delete all uses files).
+pct destroy your-vm-id [OPTIONS]
+
+# Show CT status.
+pct status your-vm-id [OPTIONS]
+
+# Migrate the container to another node. Creates a new migration task.
+pct migrate your-vm-id <target> [OPTIONS]
+
+# Migrate the container to a remote cluster. Creates a new migration task. EXPERIMENTAL.
+pct remote-migrate your-vm-id target-endpoint --target-bridge <string> --target-storage <string> [OPTIONS]
+
+# Get container configuration.
+pct config your-vm-id [OPTIONS]
+
+# Print the list of assigned CPU sets.
+pct cpusets
+
+# Get container configuration, including pending changes.
+pct pending your-vm-id
+
+# Reboot the container by shutting it down, and starting it again. Applies pending changes.
+pct reboot your-vm-id [OPTIONS]
+
+# Create or restore a container.
+pct restore your-vm-id <ostemplate> [OPTIONS]
+
+# Set container options.
+pct set your-vm-id [OPTIONS]
+
+# Create a Template.
+pct template your-vm-id
+
+# Unlock the VM.
+pct unlock your-vm-id
+```
+
+### Container Disks
+
+```shell
+# Get the container's current disk usage.
+pct df your-vm-id
+
+# Run a filesystem check (fsck) on a container volume.
+pct fsck your-vm-id [OPTIONS]
+
+# Run fstrim on a chosen CT and its mountpoints.
+pct fstrim your-vm-id [OPTIONS]
+
+# Mount the container's filesystem on the host.
+# This will hold a lock on the container and is meant for emergency maintenance only
+# as it will prevent further operations on the container other than start and stop.
+pct mount your-vm-id
+
+# Move a rootfs-/mp-volume to a different storage or to a different container.
+pct move-volume your-vm-id <volume> [<storage>] [<target-vmid>] [<target-volume>] [OPTIONS]
+
+# Unmount the container's filesystem.
+pct unmount your-vm-id
+
+# Resize a container mount point.
+pct resize your-vm-id your-vm-disk <size> [OPTIONS]
+
+# Rescan all storages and update disk sizes and unused disk images.
+pct rescan [OPTIONS]
+
+# Connect to container
+pct enter your-vm-id
+
+# Launch a console for the specified container.
+pct console your-vm-id [OPTIONS]
+
+# Launch a shell for the specified container.
+pct enter your-vm-id
+
+# Launch a command inside the specified container.
+pct exec your-vm-id [<extra-args>]
+
+# Copy a file from the container to the local system.
+pct pull your-vm-id <path> <destination> [OPTIONS]
+
+# Copy a local file to the container.
+pct push your-vm-id <file> <destination> [OPTIONS]
+```
+
+### Container Snapshot
+
+```shell
+# Snapshot a container.
+pct snapshot your-vm-id <snapname> [OPTIONS]
+
+# List all snapshots.
+pct listsnapshot your-vm-id
+
+# Rollback LXC state to specified snapshot.
+pct rollback your-vm-id <snapname> [OPTIONS]
+
+# Delete a LXC snapshot.
+pct delsnapshot your-vm-id <snapname> [OPTIONS]
+```
+
+## Web GUI
+
+```shell
+# Restart web GUI
+service pveproxy restart
+```
+
+## Resize Disk
+
+### Increase disk size
+
+Increase disk size in the GUI or with the following command
+
+```shell
+qm resize 100 virtio0 +5G
+```
+
+### Decrease disk size
+
+> Before decreasing disk sizes in Proxmox, you should take a backup!
+
+1. Convert qcow2 to raw: `qemu-img convert vm-100.qcow2 vm-100.raw`
+2. Shrink the disk `qemu-img resize -f raw vm-100.raw 10G`
+3. Convert back to qcow2 `qemu-img convert -p -O qcow2 vm-100.raw vm-100.qcow2`
+
+## Further information
+
+More examples and tutorials regarding Proxmox can be found in the link list below:
+
+- Ansible playbook that automates Linux VM updates running on Proxmox (including snapshots): [TheDatabaseMe - update_proxmox_vm](https://github.com/thedatabaseme/update_proxmox_vm)
+- Manage Proxmox VM templates with Packer: [Use Packer to build Proxmox images](https://thedatabaseme.de/2022/10/16/what-a-golden-boy-use-packer-to-build-proxmox-images/)

cheatsheets/zsh/zsh-cheatsheet.md

@@ -0,0 +1,169 @@
+# Zsh Cheatsheet
+
+Reference for the new ZDOTDIR-based zsh setup (oh-my-zsh removed,
+config lives in `~/.config/zsh/`).
+
+## Aliases
+
+### Docker Compose
+| Alias | Expands to | What it does |
+|---|---|---|
+| `dcu` | `docker compose up -d` | Start containers in detached mode |
+| `dcd` | `docker compose down` | Stop and remove containers |
+| `dcr` | `docker compose restart` | Restart containers |
+| `dcp` | `docker compose pull` | Pull latest images |
+
+### Navigation
+| Alias | Expands to | What it does |
+|---|---|---|
+| `..` | `cd ..` | Up one directory |
+| `...` | `cd ../..` | Up two directories |
+| `....` | `cd ../../..` | Up three directories |
+
+### File listing
+| Alias | Expands to | What it does |
+|---|---|---|
+| `ls` | `eza -lahF --group-directories-first --time-style="+%d/%m/%Y %H:%M" --git` (falls back to plain `ls -lahF ...` if eza isn't installed) | Long listing, all files, classify symbols (`/`, `*`, `@`), directories first, custom date format, git status column |
+| `tree` | `eza --tree` (only if eza is installed) | Tree view of directory structure |
+| `cat` | `bat` (only if bat is installed) | Syntax-highlighted file viewer |
+
+### System
+| Alias | Expands to | What it does |
+|---|---|---|
+| `df` | `df -h` | Disk usage, human-readable sizes |
+| `free` | `free -h` | Memory usage, human-readable sizes |
+| `c` | `clear` | Clear the terminal |
+
+## Functions
+
+| Function | Usage | What it does |
+|---|---|---|
+| `cd` (overridden) | `cd <dir>` | Changes directory, then automatically runs `ls` (so you always see the contents of where you land). `cd` with no argument goes to `$HOME`. |
+| `mkcd` | `mkcd <dir>` | Creates a directory and `cd`s into it in one step |
+| `extract` | `extract <file>` | Extracts an archive based on its extension. Supports `.tar.gz`/`.tgz`, `.tar.bz2`, `.tar.xz`, `.zip`, `.gz` |
+| `ports` | `ports` | Shows listening ports and the processes using them (`sudo ss -tulpen`) |
+| `psg` | `psg <name>` | Searches running processes by name (case-insensitive `ps aux | grep`) |
+| `killport` | `killport <port>` | Kills whatever process is listening on the given port |
+| `logtail` | `logtail <file> [lines]` | Tails a log file (`-f`), default 50 lines, or specify a custom number |
+| `reload` | `reload` | Reloads the zsh config (`source $ZDOTDIR/.zshrc`) without restarting the shell |
+
+## Keybindings
+
+| Key | Action | Notes |
+|---|---|---|
+| `Ctrl + →` | Move cursor forward one word | |
+| `Ctrl + ←` | Move cursor backward one word | |
+| `Ctrl + \` | Toggle autosuggestions on/off | Echoes "Autosuggestions: on/off" so you know the current state |
+| `↑` (Up arrow) | Search history backward by prefix | Type a few characters first, then press Up to find matching previous commands |
+| `↓` (Down arrow) | Search history forward by prefix | Same as above, opposite direction |
+
+## Line editing (built-in zsh defaults)
+
+These come from zsh's default "emacs" editing mode (the standard unless
+something switches it to vi mode — this setup doesn't). Most of these
+work in any emacs-style shell (bash, zsh) and are worth building into
+muscle memory since they work everywhere.
+
+### Cursor movement
+
+| Key | Action |
+|---|---|
+| `←` / `→` | Move cursor one character left/right |
+| `Ctrl + ←` / `Ctrl + →` | Move cursor one **word** left/right (custom binding in this setup) |
+| `Ctrl + A` | Jump to the **start** of the line |
+| `Ctrl + E` | Jump to the **end** of the line |
+| `Alt + B` | Move backward one word (alternative to Ctrl+←) |
+| `Alt + F` | Move forward one word (alternative to Ctrl+→) |
+
+### Deleting / cutting text
+
+| Key | Action |
+|---|---|
+| `Backspace` | Delete the character **before** the cursor |
+| `Ctrl + D` / `Delete` | Delete the character **under/after** the cursor (Ctrl+D on an empty line exits the shell — careful) |
+| `Ctrl + W` | Delete the **word before** the cursor (cuts to kill-ring) |
+| `Alt + D` | Delete the **word after** the cursor (cuts to kill-ring) |
+| `Alt + Backspace` | Same as Ctrl+W — delete word before cursor |
+| `Ctrl + U` | Delete **everything before** the cursor (whole line back to start) |
+| `Ctrl + K` | Delete **everything after** the cursor (to end of line) |
+
+### Pasting (yanking) deleted text
+
+zsh doesn't have "select text" like a GUI — instead, anything you cut
+with `Ctrl+W`, `Alt+D`, `Ctrl+U`, or `Ctrl+K` goes into a clipboard-like
+buffer (the "kill ring"), which you can paste back with:
+
+| Key | Action |
+|---|---|
+| `Ctrl + Y` | Paste ("yank") the last cut text at the cursor |
+
+**Practical example:** to move a word from the end of a command to the
+start — put cursor at end of word, `Alt + Backspace` to cut it, `Ctrl + A`
+to jump to the start of the line, `Ctrl + Y` to paste it there.
+
+### Undo / retype
+
+| Key | Action |
+|---|---|
+| `Ctrl + _` (Ctrl + Shift + -) | Undo the last edit |
+| `Ctrl + L` | Clear the screen (same as the `c` alias, but doesn't lose scrollback the same way) |
+
+### Transposing / fixing typos
+
+| Key | Action |
+|---|---|
+| `Ctrl + T` | Swap the two characters around the cursor (fixes a typo like `hte` → `the`) — **note:** this is overridden by fzf's Ctrl+T in this setup if fzf is installed |
+| `Alt + T` | Swap the two **words** around the cursor |
+
+### Case changes
+
+| Key | Action |
+|---|---|
+| `Alt + U` | Uppercase the word from the cursor onward |
+| `Alt + L` | Lowercase the word from the cursor onward |
+| `Alt + C` | Capitalize the word from the cursor onward |
+
+### Re-running / editing previous commands
+
+| Key | Action |
+|---|---|
+| `!!` | Re-run the last command (e.g. `sudo !!` re-runs the last command with sudo) |
+| `!$` | The last **argument** of the previous command (e.g. `cat !$` opens whatever file you last referenced) |
+| `Ctrl + R` (no fzf) | Reverse incremental search through history (built-in, before fzf takes over — see fzf section below for the upgraded version) |
+| `fc` | Opens the last command in `$EDITOR` (nano) for editing, then runs it |
+
+## Fuzzy search (fzf)
+
+Only active if `fzf` is installed.
+
+| Key | Action | Notes |
+|---|---|---|
+| `Ctrl + R` | Fuzzy search command history | Type to filter, select a command, it's inserted at the cursor. This overrides the built-in reverse search mentioned above |
+| `Ctrl + F` | Fuzzy file search (includes hidden files, excludes `.git`) | Selected file path is inserted at the cursor. Uses `fd` if installed, otherwise falls back to `find` |
+| `Ctrl + T` | Fuzzy file search (fzf's built-in, non-hidden files) | Only active if fzf's own shell integration is found on the system; uses `fd` for speed if installed. This overrides the built-in character-transpose shortcut mentioned above |
+
+## Plugins (auto-loaded on shell startup)
+
+These are cloned automatically into `$ZDOTDIR/plugins/` the first time
+you start a shell after the role runs — no manual setup needed.
+
+| Plugin | What it does |
+|---|---|
+| `zsh-autosuggestions` | Shows a greyed-out suggestion as you type, based on history. Press `→` (right arrow) or `End` to accept it |
+| `fast-syntax-highlighting` | Colors commands as you type — valid commands, paths, strings, etc. get different colors so you can spot typos before pressing Enter |
+
+## Environment / history
+
+- History file: `~/.local/state/zsh/history`
+- History size: 10,000 commands
+- History behavior: shared across sessions, duplicates ignored, commands
+  starting with a space aren't recorded (`HIST_IGNORE_SPACE`)
+- Default editor: `nano` (`$EDITOR` / `$VISUAL`)
+- Config location: `$ZDOTDIR` = `~/.config/zsh/`
+- Per-host extra env vars (e.g. `VAULT_ADDR`) are set via `zsh_extra_env`
+  in Ansible host_vars, appended to `env.zsh`
+
+## Startup
+
+`fastfetch` runs automatically every time a new shell opens, showing
+system info (OS, kernel, uptime, packages, etc.) above the prompt.

docs/architecture-overview.md

@@ -1,6 +1,6 @@
-<!-- verified: 2026-09-06
-     rocky-docker-lxc@4f43dcc : caddy/Caddyfile
-     caddy-vm@9369f8b : caddy/Caddyfile
+<!-- verified: 2026-09-18
+     rocky-docker-lxc@d225776 : caddy/conf/Caddyfile
+     caddy-vm@c5c3420 : caddy/conf/Caddyfile
      trilium : 2wCPV6NXQa3l, 0mu75aAXGH8X, fscYadnUtmD5, QMTh5G7ldlLV, TIbCwYGitboy, Cl4TWBZ7ytYr, 6cJurv76PNDR, nAPJq9A1iuX8, DvKxK0gHXsng
 -->
 
@@ -31,7 +31,7 @@ The actual answer here is **four boundaries, each doing a different job**:
 | 🔴 WAN → edge | Anyone on the internet | Public reverse proxy, firewall-filtered | The only boundary facing genuinely untrusted traffic — gets the strictest, most isolated treatment |
 | 🔵 Browsers → user-facing proxy | A human, on a device that could be compromised, phished, or simply lost | Mutual TLS (client cert required) + SSO | The highest-value target for credential theft — a stolen password alone isn't enough without the client cert too |
 | 🟢 App-to-app | A container that's already inside the network, already provisioned by the same pipeline as everything else | IP-allowlist + SSO fallback, no mTLS | Full mTLS here would mean managing certificate lifecycle for every internal service-to-service call — real overhead for traffic that's already constrained to known, provisioned sources. The risk profile doesn't justify the same control as human-operated browsers. |
-| 🟡 LAN infra-to-infra | Non-HTTP traffic — metrics, logs, NTP, agent reporting | Explicit per-flow firewall rule, no proxy at all | Can't go through a reverse proxy — it isn't HTTP. The firewall itself is the only layer that *can* enforce this one. |
+| 🟡 LAN infra-to-infra | Non-HTTP traffic — metrics, logs, NTP, agent reporting | Explicit per-flow rule at each host's own firewalld — the real gate, since same-subnet traffic never transits a router | Can't go through a reverse proxy — it isn't HTTP. Each host's own firewall is the layer that actually sees this traffic; pfSense carries a mirrored ruleset for anything that does route through it, as defense-in-depth. |
 
 Every boundary above only means something if a service can't just skip it by reading the wrong header: **backend services trust proxy headers, never the raw client IP.** If that weren't true, the mTLS/SSO enforcement happening at the proxy layer would be theater — anything downstream could just claim to be a trusted IP and walk straight through. The proxy is the only thing allowed an opinion on who a client really is.
 
@@ -56,11 +56,11 @@ A single-router design fails this test completely — one outage, one misconfigu
 >
 > The second fix was conceptual, and the bigger one: for a long time the app-to-app boundary simply didn't exist. Internal service traffic crossed a flat, unauthenticated mesh network — anything on it could reach anything else on it, no proxy, no identity check, nothing. That's exactly the "you're on the network, you're trusted" pattern this document opens by calling the easy, wrong answer. Replacing it with the two-zone model in [`docs/dns-naming.md`](dns-naming.md) is what gave app-to-app traffic a real boundary for the first time.
 >
-> The third fix found the same flaw hiding somewhere less obvious: plain LAN-to-LAN traffic — metrics, logs, NTP, agent reporting — still had a blanket allow at the firewall, even after the app-to-app zone had its own identity-checked proxy. Any host could reach any other host; the only thing standing between "one compromised container" and "everything else on the LAN" was whatever that individual host's own firewall happened to be configured to do, which is a coin flip, not a boundary. Closing it meant giving up "it just works" for LAN infra traffic and writing an explicit rule for every flow that's actually supposed to exist — the same trade this document makes everywhere else: a boundary only counts if crossing it takes a decision, not an accident of network topology.
+> The third fix found the same flaw hiding somewhere less obvious: plain LAN-to-LAN traffic — metrics, logs, NTP, agent reporting — still had a blanket allow, even after the app-to-app zone had its own identity-checked proxy. Any host could reach any other host; the only thing standing between "one compromised container" and "everything else on the LAN" was whatever that individual host's own firewall happened to be configured to do, which is a coin flip, not a boundary. Closing it meant writing an explicit per-flow rule everywhere that boundary is actually enforced: each host's own firewalld (the real gate — this LAN is one flat /24, so traffic between two hosts on it never transits a router at all) and, for anything that does route through pfSense itself, a matching explicit-allow-plus-catch-all-deny ruleset there too, as defense-in-depth and a stated record of intent.
 >
 > Three different places, the same failure mode, the same fix each time — that's not a coincidence, it's what happens when you actually go looking for where "trusted by default" is still hiding.
 
-That third fix isn't just a claim — it's the actual live ruleset:
+That third fix isn't just a claim. Here's pfSense's half of it — even though genuine host-to-host traffic on this flat LAN never actually reaches it:
 
 ![pfSense LAN catch-all deny rule: block+log east-west traffic not explicitly allowed](../assets/pfsense-rules-lan-catchall-deny.png)
 
@@ -69,9 +69,9 @@ That third fix isn't just a claim — it's the actual live ruleset:
 <details>
 <summary>See the rest of the actual pfSense configuration</summary>
 
-**WAN — only three things pass inbound, everything else default-deny:**
+**WAN — only four things pass inbound, everything else default-deny:** SSH management, HTTPS via HAProxy, an HTTP→HTTPS redirect, and a STUN port-forward for the NetBird mesh VPN's NAT traversal.
 
-![pfSense WAN rules: SSH management, HTTPS via HAProxy, HTTP redirect, bogon block](../assets/pfsense-rules-wan.png)
+![pfSense WAN rules: SSH management, HTTPS via HAProxy, HTTP redirect, NetBird STUN, bogon block](../assets/pfsense-rules-wan.png)
 
 **LAN — fleet observability and security-agent reporting, one explicit rule per flow:**
 
@@ -95,16 +95,16 @@ That third fix isn't just a claim — it's the actual live ruleset:
 
 ## The Other Six Traffic Laws
 
-Each of these gets its own full mechanism doc elsewhere in this repo — this table is the map, not the territory:
+Laws 1–2 are covered above (the trust-boundary table); these are 3–8. Each of these gets its own full mechanism doc elsewhere in this repo — this table is the map, not the territory:
 
 | # | Law | Failure behavior |
 |---|---|---|
-| 3. Secrets | Applications never talk to Vault directly — one hop through a local agent only, with one narrow exception: a dedicated SSH-certificate refresh reuses that agent's token to call Vault directly. See [`docs/secrets-and-pki.md`](secrets-and-pki.md) and [`docs/secret-distribution.md`](secret-distribution.md). | Vault down → secret/certificate renewal stops, services run until their current token or cert expires — no immediate disruption. The one flow that doesn't degrade this gently: the SSH-certificate exception above has no static-key fallback, so a sustained outage eventually locks the automation identity out of the fleet entirely, not just "stale but running." |
-| 4. DNS | Which internal zone a hostname resolves in determines which trust boundary (above) a request enters. See [`docs/dns-naming.md`](dns-naming.md). | DNS down → app-to-app resolution fails; a direct-IP fallback exists, and it bypasses the SSO gate — but only for traffic already coming from an IP on the trusted allowlist. Anything else still hits Authentik regardless of DNS. |
-| 5. Observability | Logs and metrics push asynchronously — never inline with application traffic, full stop. See [`docs/observability.md`](observability.md) and [`docs/security-monitoring.md`](security-monitoring.md). | Monitoring down → collection stops; every application service keeps running, completely unaffected. |
-| 6. Notifications | Two independent branches (message-queue and HTTP webhook) converge on one formatter; neither is part of any service's runtime dependency chain. See [`docs/notifications.md`](notifications.md). | Notification delivery is event-driven, not persistent inline execution — a failure here doesn't touch application traffic either way. |
-| 7. CI/CD | Build/deploy traffic never sits inline with runtime; schedule-triggered, not push-triggered. See [`docs/registry-and-build.md`](registry-and-build.md). | A failed deploy doesn't affect already-running services — enforced by the redeploy step's own rollback logic. |
-| 8. Management | Administrative access is its own separate plane — one dedicated identity, non-standard ports, explicit per-host firewall rules from one source only on the LAN, plus a separate zero-trust mesh giving that same access a scoped path in from off-LAN too. See [`docs/netbird-zero-trust.md`](netbird-zero-trust.md). | On-LAN admin access never depended on the mesh and is unaffected if it goes down; off-LAN access specifically loses its only path until the mesh recovers. Every running service is unaffected either way. |
+| 3 | **Secrets.** Applications never talk to Vault directly — one hop through a local agent only, with one narrow exception: a dedicated SSH-certificate refresh reuses that agent's token to call Vault directly. See [`docs/secrets-and-pki.md`](secrets-and-pki.md) and [`docs/secret-distribution.md`](secret-distribution.md). | Vault down → secret/certificate renewal stops, services run until their current token or cert expires — no immediate disruption. The one flow that doesn't degrade this gently: the SSH-certificate exception above has no static-key fallback, so a sustained outage eventually locks the automation identity out of the fleet entirely, not just "stale but running." |
+| 4 | **DNS.** Which internal zone a hostname resolves in determines which trust boundary (above) a request enters. See [`docs/dns-naming.md`](dns-naming.md). | DNS down → app-to-app resolution fails; a direct-IP fallback exists, and it bypasses the SSO gate — but only for traffic already coming from an IP on the trusted allowlist. Anything else still hits Authentik regardless of DNS. |
+| 5 | **Observability.** Logs and metrics push asynchronously — never inline with application traffic, full stop. See [`docs/observability.md`](observability.md) and [`docs/security-monitoring.md`](security-monitoring.md). | Monitoring down → collection stops; every application service keeps running, completely unaffected. |
+| 6 | **Notifications.** Two independent branches (message-queue and HTTP webhook) converge on one formatter; neither is part of any service's runtime dependency chain. See [`docs/notifications.md`](notifications.md). | Notification delivery is event-driven, not persistent inline execution — a failure here doesn't touch application traffic either way. |
+| 7 | **CI/CD.** Build/deploy traffic never sits inline with runtime; schedule-triggered, not push-triggered. See [`docs/registry-and-build.md`](registry-and-build.md). | A failed deploy doesn't affect already-running services — enforced by the redeploy step's own rollback logic. |
+| 8 | **Management.** Administrative access is its own separate plane — one dedicated identity, non-standard ports, explicit per-host firewall rules from one source only on the LAN, plus a separate zero-trust mesh giving that same access a scoped path in from off-LAN too. See [`docs/netbird-zero-trust.md`](netbird-zero-trust.md). | On-LAN admin access never depended on the mesh and is unaffected if it goes down; off-LAN access specifically loses its only path until the mesh recovers. Every running service is unaffected either way. |
 
 The pattern repeats across all eight laws, not just the trust-boundary story above: **control-plane failure never takes down the data plane.** Every "failure behavior" row above ends the same way — the thing that broke stops working, and everything else keeps running regardless. That's not a coincidence; it's the same design principle applied consistently everywhere, not just to network trust.
 

docs/cron-scheduler.md

@@ -1,4 +1,4 @@
-<!-- verified: 2026-09-06
+<!-- verified: 2026-09-18
      trilium : EJGewkYGStHf
 -->
 
@@ -11,7 +11,7 @@
 
 A Node-RED tab fetches the inventory straight from NocoDB's REST API at request time — no cache — and renders it as a 7×24 grid (Mon–Sun × 00:00–23:00). Each cell's color is a **computed load weight**, not a raw job count: base `1` per job, plus a duration-bucket weight, plus a resource-tag weight (`disk-intensive`=2, `cpu-intensive`=1.5, `network-intensive`=0.5, summed if a job carries more than one), plus a flat `+1` if the job's category is `backup` — disk I/O contention from an overlapping backup is judged a worse collision than two lightweight jobs sharing an hour, and the weight formula says so explicitly rather than pretending every job costs the same. Weights bucket into 4 color bands (green → yellow → orange → red) relative to the current busiest cell. Jobs whose time can't be resolved to a specific day+hour are excluded from the grid and listed separately rather than forced into a misleading cell.
 
-This is the real heatmap, not a description of it — 50 known jobs, 15 without a fixed time:
+This is the real heatmap, not a description of it — 49 known jobs, 14 without a fixed time:
 
 ![The real Fleet Schedule Heatmap widget in Dashy, with Tue 00:00 clicked open showing its 8 real jobs — name, host, category, and FIXED/APP UI badges for jobs that can't simply be rescheduled](../assets/cron-scheduler-heatmap.png)
 
@@ -20,11 +20,11 @@ This is the real heatmap, not a description of it — 50 known jobs, 15 without
 
 Jobs with no fixed time get their own list instead of a misleading grid cell:
 
-![The heatmap's "no fixed time" section expanded — 15 real jobs excluded from the grid, including the manually-triggered-only Kestra flows and the PBS prune job with its mismatched datastore name](../assets/cron-scheduler-no-fixed-time.png)
+![The heatmap's "no fixed time" section expanded — 14 real jobs excluded from the grid, including the manually-triggered-only Kestra flows and Hawser's continuous 30-second heartbeat agent](../assets/cron-scheduler-no-fixed-time.png)
 
 Both of those are Dashy's *rendered* view. This is the raw NocoDB table underneath it — the actual data structure, not the heatmap built from it:
 
-![The real "Cron Scheduler" NocoDB table in grid view, showing all 50 records with their Job Name, Category, Schedule Expr, Est Start, Duration, Resource Tags, and Source columns](../assets/cron-scheduler-table.png)
+![The real "Cron Scheduler" NocoDB table in grid view, showing all 49 records with their Job Name, Category, Schedule Expr, Est Start, Duration, Resource Tags, and Source columns](../assets/cron-scheduler-table.png)
 
 ## Why A Manual Inventory Instead Of An Auto-discovered One
 

docs/dashy-soc-board.md

@@ -1,4 +1,4 @@
-<!-- verified: 2026-09-06
+<!-- verified: 2026-09-19
      trilium : 5XjzoVTH1r3B, cVFNSn4CuuEN
 -->
 
@@ -14,10 +14,10 @@ This board didn't start from nothing — it absorbed several formerly-standalone
 | Source | What it actually contributes |
 |---|---|
 | Wazuh indexer | 7-day findings, categorized: ClamAV, vulnerabilities, rootkit/anomaly, authentication, other |
-| Vault | Seal state, active lease/token counts, KV engine counts, token TTL distribution, and root/intermediate CA cert expiry — parsed live from Vault's own `pki/cert/ca` and `pki_int/cert/ca` endpoints on every load, not a cached value |
+| Vault | Seal state, active lease/token counts, KV engine counts, and token TTL distribution — read from Prometheus metrics, not a live Vault call, so all of them go stale together if the metrics-reader token lapses. Root/intermediate CA cert expiry is the one exception, parsed live from Vault's own `pki/cert/ca` and `pki_int/cert/ca` endpoints on every load regardless of metrics health |
 | Technitium DNS | Query stats, top clients/domains, blocklist state, plus a dedicated "DNS Security Signals" section: forwarder failures (read from the Loki-shipped query log), query-log ingestion lag, next blocklist update time |
 | Fleet config-drift audit | *Not* computed by this board's own function — read from Node-RED's shared `fleetAudit` global, populated by a separate ingestion process, covering 4 finding types: untracked stacks, missing declared directories, containers stuck in `Created`, and TZ drift |
-| Slack | 12 app-specific channels (the *arr suite, Jellyfin/Emby/Seerr, Kopia, Authentik, Gitea, Netbox, Grafana) mapped to their owning host by a plain lookup table, plus each host's own alert channel |
+| Slack | 12 app-specific channels (the *arr suite, Jellyfin/Emby/Seerr, Kopia, Authentik, Gitea, Proxmox, Grafana) mapped to their owning host by a plain lookup table, plus each host's own alert channel |
 | Proxmox `cluster/resources` API | The two hypervisor nodes' own CPU/RAM/disk (no Prometheus coverage on bare metal) plus the full running VM/LXC tree per node |
 
 The fourteen, enumerated directly from the function's own upstream-URL constants rather than counted by hand — this is the actual grep and its real output, not a reconstruction:
@@ -48,7 +48,7 @@ for n in data:
     if n.get('type') == 'function' and n.get('name') == 'Build SOC Board HTML':
         print(n['name'], '->', len(n['func']), 'chars')
 "
-Build SOC Board HTML -> 104443 chars
+Build SOC Board HTML -> 104635 chars
 ```
 
 ## Why One Function Instead of Native Dashy Widgets

docs/disaster-recovery.md

@@ -1,6 +1,6 @@
-<!-- verified: 2026-09-06
-     iac@49ff427 : ansible/roles_metadata.yml
-     ansible@ccbda37 : roles/os_update, playbooks/os_update_vms.yml, playbooks/os_update_pve.yml, playbooks/os_update_pbs.yml, playbooks/app_update.yml
+<!-- verified: 2026-09-18
+     iac@ae315de : ansible/roles_metadata.yml
+     ansible@24e1bf0 : roles/os_update, playbooks/os_update_vms.yml, playbooks/os_update_pve.yml, playbooks/os_update_pbs.yml, playbooks/app_update.yml
      trilium : xJRSQovMncQr, 6AbdS9cUE45T, bW84ccRoAJ5i, p45qNP2sIuhA, 6oESvJYzj1at
 -->
 
@@ -21,23 +21,24 @@ A single backup tool covering everything was the simpler design, and it was reje
 |---|---|---|
 | Infrastructure as Code | A host lost entirely, VM or LXC alike — rebuilt from the same pipeline that built it ([`docs/iac-pipeline.md`](iac-pipeline.md)), against its existing tag-driven config | Secrets — recovered separately via Vault, see [`docs/secrets-and-pki.md`](secrets-and-pki.md), not part of this layer |
 | Proxmox Backup Server | Full VM/LXC backups, including the secrets platform's own seal state and Vault's own data described in [`docs/secrets-and-pki.md`](secrets-and-pki.md) | One host's Docker volume data, a large media storage pool, personal data |
-| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering the ~55 stacks in [`stacks/`](../stacks/) | Anything outside that one host |
+| Kopia | The gap the layer above leaves — every stack's databases, uploaded files, and app config on the fleet's largest host, covering all 41 stacks running there | Anything outside that one host |
 
 This is the real Kopia repository, not a description of it — a dedicated set of directories on its own separate disk, confirmed live:
 
 ```
 $ ls /mnt/samsungssd
-kopia-cache  kopia-config  kopia-repo  kopia-tls
+kopia-cache  kopia-config  kopia-repo  kopia-tls  recovery
 $ df -h /mnt/samsungssd
 Filesystem      Size  Used Avail Use% Mounted on
-/dev/nvme0n1p1  234G   65G  167G  28% /mnt/samsungssd
+/dev/nvme0n1p1  234G   75G  157G  33% /mnt/samsungssd
 ```
 
 Layer 1's "rebuilt against its existing tag-driven config" claim is a real, checkable mechanism too — every role in the pipeline declares which of the three rebuild stages it belongs to:
 
 ```
-$ head -12 roles_metadata.yml
+$ head -14 roles_metadata.yml
 roles_metadata:
+  # Bootstrap roles
   base_users:
     initialization: true
     provisioning: false

docs/dns-naming.md

@@ -1,4 +1,4 @@
-<!-- verified: 2026-09-06
+<!-- verified: 2026-09-18
      trilium : nAPJq9A1iuX8, G6NWXtdWX9ap, 0mu75aAXGH8X, fscYadnUtmD5
 -->
 

docs/firewall-and-network.md

@@ -1,5 +1,5 @@
 <!-- verified: 2026-09-06
-     iac@49ff427 : ansible/roles/bootstrap_roles/firewall_ssh, ansible/roles/secure_vault/firewall_rich_rules_per_vm, ansible/roles/cis_hardening/password_policy
+     iac@49d9773 : ansible/roles/bootstrap_roles/firewall_ssh, ansible/roles/secure_vault/firewall_rich_rules_per_vm, ansible/roles/cis_hardening/password_policy
      trilium : 0mu75aAXGH8X, o3DDWo1v54vI, oEbKGleDKuvb, EpssgVXxqIbE, 0R8GeBmH7boV, TIbCwYGitboy, fscYadnUtmD5, yUJqswizuET2, QMTh5G7ldlLV, jBPOx07B1Y2Z, Cl4TWBZ7ytYr, 6cJurv76PNDR, kwbbLjeI6ody, MSVD5o5HiJ69, kA00FQiQWVem
 -->
 

docs/homelab-inventory.md

@@ -1,6 +1,6 @@
 <!-- verified: 2026-09-07
      ansible@9a4ba6d : roles/homelab_inventory_hardware, roles/homelab_inventory_vms, roles/homelab_inventory_services, playbooks/homelab_inventory_hardware.yml, playbooks/homelab_inventory_vms.yml, playbooks/homelab_inventory_services.yml
-     rocky-docker-lxc@145d3df : caddy/Caddyfile, dashy/docker-compose.yaml
+     rocky-docker-lxc@d225776 : caddy/conf/Caddyfile, dashy/docker-compose.yaml
      trilium : O0d4fqw0mr05
 -->
 

docs/iac-pipeline.md

@@ -1,5 +1,5 @@
-<!-- verified: 2026-09-06
-     iac@49ff427 : packer/rocky10-proxmox-template, terraform, ansible/roles_metadata.yml, ansible/playbooks/provisioning.yml, ansible/host_vars/technitium/host_tags.yml, ansible/roles/cis_hardening/password_policy
+<!-- verified: 2026-09-19
+     iac@ae315de : packer/rocky10-proxmox-template, terraform, ansible/roles_metadata.yml, ansible/playbooks/provisioning.yml, ansible/host_vars/technitium/host_tags.yml, ansible/roles/cis_hardening/password_policy
      trilium : c5JADrRVvjIn, f1uTcvXvq0u3, 7anU4VULpp7n, sovEBDmn16hh, sqVGFejKPOFn, kA00FQiQWVem, MSVD5o5HiJ69, NwwrSew7mP2z, QgviHP8r14t2
 -->
 
@@ -53,7 +53,7 @@ Terraform manages both VMs and LXCs now — two separate resource types, one per
 The LXC path is genuinely newer and mirrors the VM flow step-for-step, but has to close a gap the VM path never has: the stock container template has no `sshd` and no fleet users at all, since it never went through Packer. A `pct exec` bootstrap step runs first — installing `sshd` and creating both fleet identities with their keys, the same baseline Packer bakes into the VM image at build time instead of at provision time — before the same wait-for-SSH and tag-file-render steps every VM already gets. This is the real resource definition, not a paraphrase:
 
 ```
-$ head -20 containers.tf
+$ sed -n '20,39p' containers.tf
 resource "proxmox_virtual_environment_container" "ct" {
   provider = proxmox-bpg
   for_each = var.containers
@@ -64,6 +64,8 @@ resource "proxmox_virtual_environment_container" "ct" {
   start_on_boot = each.value.onboot
   started       = true
 
+  # PVE-native tags — cosmetic/searchable in the PVE UI, mirrors main.tf's
+  # `tags = join(",", each.value.tags)` on the VM resource.
   tags = each.value.tags
 
   cpu {
@@ -82,10 +84,12 @@ This is the actual template that renders into every host's tag file — the lite
 
 ```
 $ cat host_tags.yml.tpl
+# Host metadata from Terraform provisioning
 vmid: ${vmid}
 vm_name: ${name}
 target_node: ${target_node}
 template: ${template}
+inventory_group: ${inventory_group}
 cores: ${cores}
 sockets: ${sockets}
 memory: ${memory}
@@ -102,7 +106,12 @@ vm_tags:
 %{ endfor ~}
 ```
 
-Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes. And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, `ip`/`macaddr` redacted:
+Every field in this file traces back to one map entry in Terraform's own VM definitions — nothing here is hand-maintained separately from the VM it describes.
+
+> [!NOTE]
+> **A field that exists in the template but not yet in any VM's real file.** `inventory_group` was added to this template alongside the 2026-08 LXC rollout — new LXCs render it correctly (`rocky_lxc`, confirmed live on netbird), but no existing VM has been re-applied since, so `inventory_group: rocky_vm` isn't actually in any of their rendered files yet, technitium's included below. A real, current gap, not a hypothetical one — worth naming rather than quietly patching the example to look more finished than the fleet actually is.
+
+And this is a real rendered instance of it, not a hypothetical fill-in — one fleet host's actual generated tag file, `ip`/`macaddr` redacted:
 
 ```
 $ cat host_vars/technitium/host_tags.yml
@@ -142,9 +151,10 @@ vm_tags:
   - observability_agent
   - mqtt_client
   - wazuh_agent_tuning
+  - netbird_agent
 ```
 
-21 tags on this one host alone, spanning both the initialization roles (`packages`, `firewall_ssh`, …) and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads.
+22 tags on this one host alone, spanning the initialization roles (`packages`, `firewall_ssh`, …), the provisioning-stage `netbird_agent` role, and every one of the 9 compliance sub-roles — the same file Terraform generated is the exact input every Ansible stage after it reads.
 
 ## Ansible: tag-driven, re-applied continuously
 
@@ -159,13 +169,13 @@ d = yaml.safe_load(open('roles_metadata.yml'))['roles_metadata']
 print('provisioning:', [k for k,v in d.items() if v.get('provisioning') is True])
 print('compliance:', sorted(k for k,v in d.items() if v.get('compliance') is True))
 "
-provisioning: ['clamav']
-compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom']
+provisioning: ['clamav', 'netbird_agent']
+compliance: ['auditd_hardening', 'banners', 'chrony_hardening', 'disk_resize', 'docker_bench', 'file_permissions', 'filesystem_hardening', 'kernel_hardening', 'mqtt_client', 'observability_agent', 'password_policy', 'service_disable', 'ssh_ca_client', 'ssh_hardening', 'wazuh_agent_tuning', 'zsh_custom']
 ```
 
-`clamav` is the only provisioning-stage role, and only one host carries it in its `vm_tags`. An under-used pipeline stage isn't the same as an unused one — it exists on purpose for exactly this shape of role, not as leftover scaffolding.
+Two roles are tagged for the provisioning stage: `clamav` (rocky-docker only) and `netbird_agent` (enrols a host into the NetBird mesh overlay covered in [`docs/netbird-zero-trust.md`](netbird-zero-trust.md) — monitoring was the first host onboarded through it). Neither fits "once at init" or "every compliance run" cleanly, which is exactly the shape this stage exists for — an under-used pipeline stage isn't the same as an unused one.
 
-Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 15 total compliance-stage roles above — the other 6 handle observability, notifications, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
+Compliance hardening runs as a set of independently-tagged sub-roles, selected per host via that same tag list — which controls apply to a given host is explicit and auditable in that host's own config, not an implicit "all or nothing." All 9 current sub-roles (of the 16 total compliance-stage roles above — the other 7 handle observability, notifications, SSH certificate refresh, and disk/shell setup, not CIS hardening specifically) are active fleet-wide; none are disabled or pending. This same tag-driven selection mechanism is what makes the deliberate CIS deviations in [`docs/security-monitoring.md`](security-monitoring.md) auditable rather than hidden — a reader can see exactly which controls a given host runs by reading its tags, not by reverse-engineering behavior.
 
 ## The Honest Exception: not everything fits the clean model
 

docs/kodi-music-assistant-bridge.md

@@ -0,0 +1,134 @@
+<!-- verified: 2026-10-16
+     rocky-docker-lxc@1ede96f : kodi/bridge.py, kodi/docker-compose.yaml, kodi/.env, music-assistant/docker-compose.yaml
+     trilium : Y2VzE58wL8w1
+-->
+
+# 🎧 Kodi <-> Music Assistant Smart Queue Bridge
+
+> [!NOTE]
+> Kodi stays the only music player, UI, and ratings authority on this fleet — ratings and library metadata live in a shared MariaDB, untouched. Music Assistant runs invisibly behind it as a pure similarity engine, never opened day-to-day, never a player, never writing anything back to the library. A small daemon watches Kodi's live playback queue and auto-refills it with musically similar tracks when it runs low, using MA purely as a recommendation backend.
+
+## Why This Shape, Not The Obvious One
+
+Two more obvious designs were deliberately rejected. The first: let MA write ratings and tags back into the audio files directly, so it could double as a second library manager. MA doesn't support this upstream, and even if it did, it would create two competing sources of truth for the same library — exactly the failure mode this design avoids. The second: make Kodi a playback target *for* Music Assistant, so MA could drive playback directly. The one community project for this integration (`MAserver-Kodi-Provider`) is archived and abandoned — betting a working setup on dead third-party code wasn't worth it.
+
+What's left is the actual design: Kodi remains the single source of truth for everything a listener touches (library, ratings, playback), and MA is reduced to a disposable, stateless recommendation service — it could be deleted and rebuilt from an empty config tomorrow without losing a single rating or play count.
+
+## How A Refill Cycle Actually Works
+
+`kodi-ma-bridge` is a small asyncio daemon, deployed as part of the same `kodi` stack as Kodi's MariaDB (not a separate stack — the two were merged once the bridge was built, since the bridge only exists to serve Kodi):
+
+```
+$ docker ps --filter name=kodi --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"
+NAMES            STATUS                  IMAGE
+kodi-ma-bridge   Up 18 minutes           python:3.13-slim
+kodi-mariadb     Up 38 hours (healthy)   <fleet registry>/mariadb:latest
+```
+
+Every 60 seconds (`POLL_INTERVAL_SECONDS=60`), it checks each configured Kodi instance:
+
+```
+$ docker exec kodi-ma-bridge env | grep -E "POLL_INTERVAL|REFILL|MA_BASE_URL|KODI_INSTANCES"
+KODI_INSTANCES=[personal-device-1]:<host>:8115,[personal-device-2]:<host>:8115
+MA_BASE_URL=http://music-assistant:8095
+POLL_INTERVAL_SECONDS=60
+REFILL_COUNT=10
+REFILL_THRESHOLD=5
+```
+
+1. Ask Kodi's JSON-RPC API (`Player.GetActivePlayers`) whether anything's playing. If the instance is unreachable or idle, skip silently — these are personal desktops, not always-on servers, so "nothing there right now" is normal, not an error condition.
+2. If something's playing, check how many tracks remain after the current position (`Playlist.GetItems` + `Player.GetProperties`). Above `REFILL_THRESHOLD` (5 remaining), stop here — nothing to do yet.
+3. Below the threshold: read the current track's artist/title and resolve it to an MA library item via `music/search`.
+4. Ask MA's `music/tracks/similar_tracks` for a large candidate pool, filter out anything already queued or recently added, translate MA's library-relative path to Kodi's SMB path, and push up to `REFILL_COUNT` (10) new tracks straight into the live playlist via `Playlist.Add`.
+
+That queue-length gate is the one real branch in the whole design — everything else is a straight pipeline — so it's worth seeing as a diagram rather than another paragraph of prose:
+
+![kodi-ma-bridge refill cycle: poll each Kodi instance, check remaining queue length, and only resolve+inject new tracks when the queue actually drops below threshold](../assets/kodi-music-assistant-bridge-refill-cycle.svg)
+
+## Three Real Issues, Found By Actually Testing It
+
+The bridge sat deployed and silently non-functional for weeks before this. All three of the following were found by doing the one thing that actually proves a system works: playing real music on a real machine and watching what happened, not just exercising the API calls in isolation.
+
+**1 — the daemon never once successfully resolved a track.** Kodi's own JSON-RPC API wraps every response in a `result` key, so the bridge's generic API-calling helper was written to unwrap it the same way for *every* call, including the ones to MA. MA's REST API doesn't use that envelope at all — it returns the payload flat, at the top level:
+
+```
+$ curl -s -X POST http://music-assistant:8095/api -H "Content-Type: application/json" \
+  -d '{"message_id":"t","command":"music/search","args":{"search_query":"Oasis - Champagne Supernova","media_types":["track"],"limit":1,"library_only":true}}' \
+  | python3 -c "import json,sys; d=json.load(sys.stdin); print('top-level keys:', list(d.keys())); print('has result key:', 'result' in d)"
+top-level keys: ['artists', 'albums', 'genres', 'tracks', 'playlists', 'radio', 'audiobooks', 'podcasts', 'sound_effects']
+has result key: False
+```
+
+`data.get("result")` against that shape silently returns `None` every time — no exception, no log line, just a daemon that looked alive and never did anything. Confirmed live against a real Kodi playback session before touching a line of code; fixed by returning MA's response body directly instead of unwrapping a key that was never there.
+
+**2 — even after the fix, MA only ever offered one real match for a mainstream track.** The bridge asked MA for a small candidate pool (15 tracks) and assumed that was the full ranked result. It isn't — MA runs its own internal similarity-distance filter over whatever raw pool size you request, so most of a small batch gets discarded before it ever reaches the caller:
+
+```
+limit=10  -> 1 match
+limit=15  -> 1 match   (the bridge's original request size)
+limit=20  -> 1 match
+limit=30  -> 2 matches
+limit=50  -> 5 matches
+limit=100 -> 6 matches
+limit=150 -> 8 matches
+limit=200 -> 10 matches
+```
+
+The real candidate pool for that seed track was there the whole time — a solid run of mid-90s Britpop and alt-rock (The Verve, The Cranberries, Ocean Colour Scene, Beck, Manic Street Preachers) — it just never got a large enough raw pool to survive MA's own filtering. Fixed by requesting 200 candidates up front; the bridge's own `REFILL_COUNT` cap still bounds how many actually get queued.
+
+**3 — a real blind-spot window between the queue running dry and the bridge noticing.** With both of the above fixed and the daemon genuinely working, the original 4-minute poll interval turned out to be too coarse on its own: during one active listening session the live queue hit 0 remaining tracks twice before the next scheduled poll caught it and refilled —
+
+```
+[personal-device] queue low (0 left), added 10 tracks seeded from 'Primal Scream - Shine Like Stars'
+[personal-device] queue low (0 left), added 10 tracks seeded from 'Kula Shaker - Govinda'
+```
+
+`REFILL_THRESHOLD=5` only guarantees a refill *attempt* starts with 5 tracks of runway left — it says nothing about how long that runway lasts in wall-clock time, and a 4-minute gap between checks was long enough, on short tracks, to burn through that runway before the next poll ever ran. Playback itself never actually stopped in practice, but the margin was thinner than the design assumed. Fixed by shortening the poll interval to 60 seconds rather than widening the threshold — each idle poll is a handful of cheap JSON-RPC calls, so checking 4x more often costs effectively nothing.
+
+## Proof It Actually Works
+
+Confirmed end-to-end against a live Kodi session, mid-album, queue genuinely running low:
+
+```
+[personal-device] queue low (2 left), added 8 tracks seeded from 'Oasis - Champagne Supernova'
+```
+
+And the live playlist, read directly from Kodi right after:
+
+```
+$ curl -s -u <user>:*** -X POST -H "Content-Type: application/json" \
+  -d '{"jsonrpc":"2.0","method":"Playlist.GetItems","params":{"playlistid":0,"properties":["title","artist"]},"id":1}' \
+  http://<host>:8115/jsonrpc
+...
+02. The Verve - Sonnet
+The Day We Caught the Train
+Dreams
+Loser
+You Stole the Sun From My Heart
+Movin' On Up
+Blue Orchid
+Buck Rogers
+Reptilia
+Use Somebody
+```
+
+A real, coherent 90s/00s indie-and-alt-rock run, auto-queued with zero manual input, directly behind the Oasis album that was already playing:
+
+![Kodi's playlist view, showing a long auto-queued run of similar indie/alt-rock tracks the bridge added on its own once the queue ran low](../assets/kodi-music-assistant-bridge-live-queue.png)
+
+## Similarity Engine
+
+Matching is powered by MA's `sonic_analysis` provider (on-device CLAP audio-embedding analysis — no cloud service, no internet dependency for recommendations) feeding the `sonic_similarity` plugin, confirmed enabled against the running library:
+
+```
+$ docker exec music-assistant python3 -c "..."
+sonic_analysis -> enabled: True
+sonic_similarity -> enabled: True
+filesystem_local -> enabled: True
+```
+
+The library syncs every 12 hours and diffs the file list, so a replaced file (an upgraded rip, say) is picked up as a new file and the stale entry pruned automatically — no manual re-indexing required.
+
+## Secrets
+
+Both the Kodi JSON-RPC password and the MA API token are Vault-sourced via the fleet's standard 3-file docker-secrets pattern (see [`docs/secret-distribution.md`](secret-distribution.md)) — nothing hardcoded in the compose file or the bridge script itself.

docs/netbird-zero-trust.md

@@ -1,6 +1,6 @@
-<!-- verified: 2026-09-06
-     netbird-lxc@13b5d80 : netbird/Caddyfile, netbird/docker-compose.yaml, netbird/config.yaml.ctmpl
-     iac@49ff427 : ansible/roles/secure_vault/netbird_agent, terraform/containers.tf
+<!-- verified: 2026-09-19
+     netbird-lxc@ea8967e : netbird/conf/Caddyfile, netbird/docker-compose.yaml, netbird/config.yaml.ctmpl
+     iac@ae315de : ansible/roles/secure_vault/netbird_agent, terraform/containers.tf
      trilium : qWLU4xqYLdJ3, oYuIjaczDrZA, nAPJq9A1iuX8, Cl4TWBZ7ytYr
 -->
 
@@ -56,9 +56,10 @@ The real path a request takes:
 And this is the actual splitter doing that work — the real config, not a description of the logic:
 
 ```
-$ cat Caddyfile
+$ cat conf/Caddyfile
 {
-	admin off
+	# Container loopback only (port not published) - `caddy reload` needs it.
+	admin localhost:2019
 	auto_https off
 	servers {
 		protocols h1 h2 h2c

docs/notifications.md

@@ -1,6 +1,6 @@
-<!-- verified: 2026-09-06
+<!-- verified: 2026-09-19
      automation-vm@7c057de : mosquitto
-     iac@49ff427 : ansible/roles/monitoring/mqtt_client
+     iac@ae315de : ansible/roles/monitoring/mqtt_client
      trilium : tt4idCVJPpZM, EAU5LUwFFQ0E, rGfAb4CddKMi, otSPaU6IfjlP, klrmnlh3Wrnd
 -->
 
@@ -11,14 +11,14 @@
 
 ## Fan-in, Fan-out
 
-![Three independent fan-in sources converging on one shared formatting subflow: 15 apps over HTTP webhook go through the App Router and Switch to a per-app mapper, Kopia/Bazarr post to their own dedicated endpoints and bypass the router entirely with their own mapper, and every host publishes over MQTT into a topic-agnostic Inventory Parse function — all three land on the same subflow before reaching Slack](../assets/notifications-fan-in-fan-out.svg)
+![Three independent fan-in sources converging on one shared formatting subflow: 14 apps over HTTP webhook go through the App Router and Switch to a per-app mapper, Kopia/Bazarr post to their own dedicated endpoints and bypass the router entirely with their own mapper, and every host publishes over MQTT into a topic-agnostic Inventory Parse function — all three land on the same subflow before reaching Slack](../assets/notifications-fan-in-fan-out.svg)
 
 ## Why One Node-RED Instance Instead Of Per-App Scripts
 
 The alternative — each app posting directly to a per-app Slack webhook, or a small script per integration — was rejected because it would mean N different message formats, N different places to add a new integration's logic, and no single point to add fleet-wide behavior (like the OS-level checks below) without touching every app individually. One Node-RED instance with a shared router and a shared formatting subflow means a new app's alerts get the same visual treatment as every existing one automatically, and a fleet-wide change (say, adding a severity-based routing rule) happens in one place.
 
 > [!WARNING]
-> **Real debt in the flow**: of the apps wired into the router, two no longer resolve to anything — NetBox was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) but its detection code was never swept out, and Kestra's alerts actually travel a completely different path than its own webhook route suggests: Kestra publishes its own alerts over MQTT, not HTTP, which makes its dedicated webhook route effectively dead code. A couple of newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own onboarding pattern for a new app's alerts.
+> **Real debt in the flow, one instance already cleaned up**: NetBox's dead detection code — left behind after the app was fully decommissioned (see [`docs/registry-and-build.md`](registry-and-build.md) for how stacks get retired) — has since been swept out of the flow entirely. What's still live: Kestra's alerts actually travel a completely different path than its own webhook route suggests — Kestra publishes its own alerts over MQTT, not HTTP, which makes its dedicated webhook route effectively dead code. A couple of newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own onboarding pattern for a new app's alerts.
 
 This is the real routing table, not a claimed count — counted directly from the flow export, not restated from memory:
 
@@ -31,11 +31,11 @@ print('Switch rules:', len(switch['rules']))
 dedicated = [n for n in flows if n.get('type')=='http in' and n.get('url','').endswith('-webhook') and n.get('url') != '/app-webhook']
 print('Dedicated non-router endpoints:', len(dedicated))
 "
-Switch rules: 16
+Switch rules: 15
 Dedicated non-router endpoints: 2
 ```
 
-16 rules = the 15 routed apps plus one `unknown` fallback that just drops the payload. And this is the real function behind the "completely different path" line above — the MQTT branch doesn't parse `msg.topic` looking for a known set of fleet hosts, it just takes whatever the second path segment is:
+15 rules = the 14 routed apps plus one `unknown` fallback that just drops the payload. And this is the real function behind the "completely different path" line above — the MQTT branch doesn't parse `msg.topic` looking for a known set of fleet hosts, it just takes whatever the second path segment is:
 
 ```
 $ grep -A4 '"Inventory Parse Function' flows.json
@@ -48,9 +48,9 @@ return msg;
 
 Topic-agnostic by construction — so a publisher that isn't a fleet host at all gets handled identically to one that is, which is exactly the mechanism that makes an app's *own* MQTT-published alerts real and its dedicated webhook route genuinely dead, not just theoretically bypassable.
 
-This is the real flow, not a redrawn version of it — the App Router feeding a 16-way Switch, NetBox's and Kestra's mapper functions still sitting there wired in and unreachable, Kopia and Bazarr's dedicated endpoints bypassing all of it entirely on their way to the same shared subflow:
+This is the real flow, not a redrawn version of it — the App Router feeding a 15-way Switch, Kestra's mapper function still sitting there wired in and unreachable, Kopia and Bazarr's dedicated endpoints bypassing all of it entirely on their way to the same shared subflow:
 
-![Real Node-RED canvas: HTTP-in webhook through App Router into a 16-rule Switch function feeding per-app mapper functions (including netbox mapper and kestra mapper, both still wired but dead), MQTT-in through the Inventory Parse Function, and Kopia/Bazarr's own dedicated HTTP-in nodes bypassing the router entirely — all converging on the shared MQTT → Slack subflow](../assets/notifications-node-red-flow.png)
+![Real Node-RED canvas: HTTP-in webhook through App Router into a 15-rule Switch function feeding per-app mapper functions (including kestra mapper, still wired but dead since its real alerts travel over MQTT instead), MQTT-in through the Inventory Parse Function, and Kopia/Bazarr's own dedicated HTTP-in nodes bypassing the router entirely — all converging on the shared MQTT → Slack subflow](../assets/notifications-node-red-flow.png)
 
 ## Message Broker
 
@@ -104,10 +104,10 @@ The Node-RED App Router figures out which app sent a webhook by inspecting the p
 >
 > ```
 > $ grep -oE '\$STATE_DIR/[a-zA-Z_]+' mqtt-monitor.sh.j2 | sed -E 's/(_alert|_last_count|_since|_report)$//' | sort -u | wc -l
-> 22
+> 23
 > ```
 >
-> 22 from that count, plus disk-usage thresholds — which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name — makes 23. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts.
+> Of that 23, one (`failed_units_current`) is a companion state file for the same "failed systemd units" check as `failed_units`, not a distinct 23rd check — so 22 real checks, plus disk-usage thresholds, which don't show up in that grep because its state key is built dynamically per mountpoint, not a fixed name, still makes 23 total. One of those 22 was a genuine KB gap this doc caught mid-audit: crash-looping systemd units (`activating`/`auto-restart` stuck in a loop) never trips the plain failed-units check above it, because a unit that keeps getting restarted never actually reaches systemd's "failed" state. That distinction wasn't theoretical — it's in the script because of a real incident where exactly that happened and produced zero alerts.
 
 ## One Formatter, Every Source
 

docs/observability.md

@@ -1,5 +1,5 @@
-<!-- verified: 2026-09-06
-     iac@49ff427 : ansible/roles/monitoring/observability_agent
+<!-- verified: 2026-09-19
+     iac@ae315de : ansible/roles/monitoring/observability_agent
      trilium : eeiUz4iTEgPL, zquji4XOpw5M, s5tWZshDEpQb, oWr1jJhvqitv
 -->
 
@@ -48,7 +48,9 @@ SystemKeepFree=1G
 MaxRetentionSec=14day
 ```
 
-![Three independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, and the DNS query log as job=dns with real event time instead of ingestion time](../assets/observability-log-fanout.svg)
+![Four independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, the DNS query log as job=technitium-dns with real event time instead of ingestion time, and auditd's own log as job=auditd on the 6 hosts that carry it](../assets/observability-log-fanout.svg)
+
+A fourth, newer branch ships raw auditd logs to Loki too, on the 6 hosts that carry the `auditd_hardening` compliance role (automation, caddy, monitoring, technitium, vault, wazuh) — a real gap-fill after one host's own `/var/log/audit` once filled to 100% with zero external visibility to catch it. It doesn't replace the local cleanup timer that actually bounds that disk usage; it just means the data isn't only ever local anymore.
 
 This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance:
 

docs/orchestration.md

@@ -1,5 +1,5 @@
-<!-- verified: 2026-09-07
-     ansible@9a4ba6d : host_vars/hyper2.yml, roles/os_update/tasks/reboot.yml, playbooks/os_update_pve_verify.yml, playbooks/automation_self_update.yml
+<!-- verified: 2026-09-19
+     ansible@24e1bf0 : host_vars/hyper2.yml, roles/os_update/tasks/reboot.yml, playbooks/os_update_pve_verify.yml, playbooks/automation_self_update.yml
      trilium : bW84ccRoAJ5i, tt4idCVJPpZM
 -->
 
@@ -98,7 +98,7 @@ The automation host's own OS patches and Kestra's own container updates run via
 ```
 $ systemctl list-timers automation-self-update.timer
 NEXT                          LEFT     LAST                          PASSED   UNIT
-Sat 2026-09-12 04:00:00 EEST  6 days   Sat 2026-09-05 04:00:32 EEST  4h 17min automation-self-update.timer
+Sat 2026-09-26 04:00:00 EEST  6 days   Sat 2026-09-19 04:00:09 EEST  5h ago   automation-self-update.timer
 ```
 
 ---

docs/registry-and-build.md

@@ -1,6 +1,6 @@
-<!-- verified: 2026-09-06
-     ansible@ccbda37 : host_vars/rocky-docker/docker_stacks.yml, roles/container_update, playbooks/container_update.yml
-     iac@49ff427 : ansible/roles/secure_vault/vault_agent/defaults/main.yml
+<!-- verified: 2026-09-19
+     ansible@24e1bf0 : host_vars/rocky-docker/docker_stacks.yml, host_vars/netbird/docker_stacks.yml, roles/container_update, playbooks/container_update.yml
+     iac@ae315de : ansible/roles/secure_vault/vault_agent/defaults/main.yml
      trilium : dcRUArk5dWH8, bW84ccRoAJ5i, hWiIGZIlvQiN
 -->
 
@@ -13,7 +13,7 @@
 |---|---|
 | **Registry** | Gitea's own built-in container registry |
 | **Trigger** | Weekly schedule via Kestra, not push-triggered |
-| **Rollout** | All 5 Docker hosts converted; the largest has 36 of 41 stacks on this pattern |
+| **Rollout** | All 6 Docker hosts converted; the largest has 35 of 41 stacks on this pattern |
 
 ## Why Self-host The Registry Instead Of Just Using Docker Hub
 
@@ -39,14 +39,15 @@ Not every stack is on this pattern, and that's a decision, not an oversight —
 ```
 $ python3 -c "import yaml; d=yaml.safe_load(open('docker_stacks.yml'))['stacks']; ..."
 total: 41
-dockerfile     36
-pinned_manual   3
+dockerfile     35
+pinned_manual   4
 latest          2
 ```
 
 | Stack | Why it's held out |
 |---|---|
 | Authentik, Immich, tdarr | Complex multi-service upstream builds — not worth replicating a build process the upstream project already maintains well; the maintenance cost of tracking their own build changes would exceed the benefit |
+| Music-assistant | Was on this pattern, deliberately reverted — auto-updating `:latest` kept bumping its `smart_fades`/`loudness_analysis` `analysis_version`, mass-invalidating ~86k tracks' audio-analysis backlog each time it moved; now pinned by hand, bumped only after checking release notes |
 | Kodi's own app image | Genuinely still a direct Docker Hub dependency — low enough stakes that the extra build step isn't worth it |
 | Nextcloud's own `nextcloud`/`euro-office` services | Pre-existing custom Dockerfiles, left as-is beyond adding `image:` references — its own db/redis still float on Docker Hub tags directly, a conscious "don't touch what already works" call |
 

docs/secret-distribution.md

@@ -1,6 +1,6 @@
-<!-- verified: 2026-09-06
-     iac@49ff427 : ansible/roles/secure_vault/ssh_ca_client, ansible/roles/secure_vault/vault_agent/defaults/main.yml, ansible/roles/secure_vault/SecureVault – Vault Contract.md, ansible/roles/secure_vault/securevault-vault-facts.md
-     ansible@ccbda37 : roles/app_update/defaults/main.yml
+<!-- verified: 2026-09-19
+     iac@ae315de : ansible/roles/secure_vault/ssh_ca_client, ansible/roles/secure_vault/vault_agent/defaults/main.yml, ansible/roles/secure_vault/SecureVault – Vault Contract.md, ansible/roles/secure_vault/securevault-vault-facts.md
+     ansible@24e1bf0 : roles/app_update/defaults/main.yml
      trilium : DvKxK0gHXsng, wovVrgvoDVy2, GW8BMoTKtRBe, 1seDR2nBmxKO
 -->
 
@@ -80,7 +80,7 @@ That redeploy-and-restart sequence isn't just a claim — here's a real one, sta
 **Shared-secret variant:** not every template is host-scoped — one credential needed by every docker host for its own registry pushes (see [`docs/registry-and-build.md`](registry-and-build.md)) is templated from a single shared path rather than a per-host one. Same mechanics, just a shared source instead of a private one — and it relies on every host's policy already granting read access to that shared path, the same "shared + private" pattern described in [`docs/secrets-and-pki.md`](secrets-and-pki.md)'s access model. This is the real template list entry, repeated per host:
 
 ```
-$ grep -A4 "service: automation" vault_agent/defaults/main.yml
+$ grep -B1 -A3 "registry-token.ctmpl" vault_agent/defaults/main.yml
 - service: automation
   src_path: .../automation-vm/registry-token/registry-token.ctmpl
   dest_path: /opt/docker/docker_volumes/registry-token/registry-token

docs/secrets-and-pki.md

@@ -1,6 +1,6 @@
-<!-- verified: 2026-09-06
-     iac@49ff427 : ansible/roles/secure_vault, ansible/group_vars/all/vault_provisioner.yml, ansible/roles/monitoring/mqtt_client
-     rocky-docker-lxc@4f43dcc : caddy/Caddyfile
+<!-- verified: 2026-09-19
+     iac@ae315de : ansible/roles/secure_vault, ansible/group_vars/all/vault_provisioner.yml, ansible/roles/monitoring/mqtt_client
+     rocky-docker-lxc@d225776 : caddy/conf/Caddyfile
      trilium : Eu0wns00QOBG, wovVrgvoDVy2, 1seDR2nBmxKO, GW8BMoTKtRBe, JbKe9CHzOP9D, Tu7FX48fxSQZ, DvKxK0gHXsng, otSPaU6IfjlP
 -->
 
@@ -68,6 +68,7 @@ Every Vault Agent reaches Vault through the internal app-to-app proxy zone descr
 | User-Facing Proxy | Server + client certs for the mTLS-enforced browser zone, plus a shared registry-token render and Hawser's application secret |
 | Internal DNS | Server cert for its own admin UI/API only |
 | Security Monitoring | Its manager and dashboard services' shared secrets (indexer, API, dashboard passwords), plus Hawser's application secret, plus a shared registry-token render |
+| NetBird | Its entire server config rendered as one file, embedded secrets included — the same file-templating pattern used for Kestra's `application.yml` — plus Hawser's application secret, plus a shared registry-token render |
 
 None of these ever call the Vault API themselves — every one of them goes through the same local-render sequence:
 

docs/security-monitoring.md

@@ -1,5 +1,5 @@
-<!-- verified: 2026-09-06
-     iac@49ff427 : ansible/roles/monitoring/wazuh_agent_tuning, ansible/roles/monitoring/docker_bench, ansible/roles/cis_hardening
+<!-- verified: 2026-09-19
+     iac@ae315de : ansible/roles/monitoring/wazuh_agent_tuning, ansible/roles/monitoring/docker_bench, ansible/roles/cis_hardening
      wazuh-vm@07ed482 : wazuh/docker-compose.yml
      trilium : KpvtX1USDhkY, TstjWVV5Q1TS, OtIgVyQC5lJg, MSVD5o5HiJ69, kA00FQiQWVem, oY5CbyuxZRyS
 -->
@@ -57,6 +57,8 @@ The alternative to an agent on every host is a purely network-level monitoring a
 > # during that night's role redeploys, confirmed via the indexer across
 > # 9 of the fleet's agents.
 > ```
+>
+> The same role found a second gap a month later, in itself this time: it silently skipped tuning the wazuh host's own agent, because that agent runs inside the manager's Docker container rather than natively on the VM — the role's own "is the agent installed" check only ever looked for a native install. That let the wazuh host's own queue sit at the untuned default until a real queue-full alert exposed it, the same class of bug the role was originally written to fix, this time hiding in the one host built differently from the rest. Fixed by giving that host its own branch instead of assuming every agent looks the same.
 
 ## How An Alert Gets Handled
 

docs/sso-and-identity.md

@@ -1,6 +1,7 @@
-<!-- verified: 2026-09-06
-     rocky-docker-lxc@4f43dcc : caddy/Caddyfile
-     caddy-vm@9369f8b : caddy/Caddyfile
+<!-- verified: 2026-09-18
+     rocky-docker-lxc@d225776 : caddy/conf/Caddyfile
+     caddy-vm@c5c3420 : caddy/conf/Caddyfile
+     iac@ae315de : ansible/roles/bootstrap_roles/docker_git_setup
      trilium : baWhWTUejvY6, hWiIGZIlvQiN, nAPJq9A1iuX8
 -->
 
@@ -60,7 +61,7 @@ This is the actual live query against Authentik's API, not a paraphrase of what'
 ```
 $ curl -s -H "Authorization: Bearer [REDACTED]" \
   "https://authentik.[service-to-service zone]/api/v3/providers/proxy/?page_size=1" | jq '.pagination.count'
-96
+94
 $ curl -s -H "Authorization: Bearer [REDACTED]" \
   "https://authentik.[service-to-service zone]/api/v3/providers/proxy/?page_size=200" | jq -r '.results[].name' | grep -E '^proxmox'
 [no output — Proxmox has no Proxy Provider at all]
@@ -91,11 +92,23 @@ So an app's OIDC discovery/authority setting always points at the service-to-ser
 
 ## Git Access: least privilege, not the admin account
 
-Automation should never run with more privilege than it needs. CI and orchestration pushes (the flows described in [`docs/orchestration.md`](orchestration.md)) run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. This wasn't the original setup: automated pushes used to run through the personal admin account used for interactive access, the same one with full admin rights over the Git server. That got fixed by standing up a separate account with its own SSH keypair and adding it as a write collaborator on every repo it touches — routine automated commits no longer run under an admin credential at all. Hosts that only need to *read* get separate read-only deploy keys instead of any shared credential. The practical payoff: if an automation credential ever leaks, the blast radius is "can push to specific repos," not "has full admin control of the Git server."
+Automation should never run with more privilege than it needs. CI and orchestration pushes (the flows described in [`docs/orchestration.md`](orchestration.md)) run through a dedicated, non-admin bot account with scoped write access — never through an administrator identity. This wasn't the original setup: automated pushes used to run through the personal admin account used for interactive access, the same one with full admin rights over the Git server. That got fixed by standing up a separate account with its own SSH keypair and adding it as a write collaborator on every repo it touches — routine automated commits no longer run under an admin credential at all. The 5 docker hosts that only ever pull (never push) use that same bot account's key too — copied identically to each host at first provisioning, not a separate credential per host. The practical payoff: if an automation credential ever leaks, the blast radius is "can push to the repos this bot account touches," not "has full admin control of the Git server."
 
 ![Gitea repo collaborators: automation-bot with Write access, not an admin account](../assets/gitea-collaborators-automation-bot.png)
 
-![Gitea repo deploy keys: a host-specific key scoped to Read only](../assets/gitea-deploy-keys.png)
+This is the real provisioning task, not a description of it — the automation-bot key gets copied to every docker host identically:
+
+```
+# The automation-bot private key - authenticates the push to Gitea AND signs
+# commits (ssh_key_path == ssh_signing_key_path). Copied from the control node.
+- name: Install the automation-bot git key
+  copy:
+    src: "{{ ssh_key_path }}"
+    dest: "{{ ssh_key_path }}"
+    owner: "{{ git_repo_owner }}"
+    group: "{{ git_repo_group }}"
+    mode: "0600"
+```
 
 ---
 Related: [`docs/dns-naming.md`](dns-naming.md), [`docs/orchestration.md`](orchestration.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md) · Stacks: [`stacks/identity-and-security/authentik/`](../stacks/identity-and-security/authentik/), [`stacks/platform/gitea/`](../stacks/platform/gitea/)

stacks/platform/caddy-internal/docker-compose.yaml

@@ -6,11 +6,11 @@ services:
     ports:
       - "${HTTPS_PORT}:443"
     volumes:
-      - ./Caddyfile:/etc/caddy/Caddyfile
+      - ./conf:/etc/caddy:ro
       - /path/to/caddy-internal/caddy_data:/data
       - /path/to/caddy-internal/caddy_config:/config
       - /path/to/caddy-internal/logs:/var/log/caddy
-      - ./certs/pki_int_ca.crt:/etc/caddy/certs/pki_int_ca.crt:ro
+      - ./certs:/etc/caddy/certs:ro
     networks:
       - caddy-net
 

stacks/platform/caddy/docker-compose.yaml

@@ -6,7 +6,7 @@ services:
     ports:
       - "${HTTPS_PORT}:443"
     volumes:
-      - ./Caddyfile:/etc/caddy/Caddyfile
+      - ./conf:/etc/caddy:ro
       - /path/to/caddy/caddy_data:/data
       - /path/to/caddy/caddy_config:/config
       - /etc/caddy/certs:/etc/caddy/certs