observability.md: full 7-step audit, first pass this session

1. Live-verify: journald persistent-storage config confirmed on two
   hosts (1G vs 500M cap, matches the doc's claim exactly), Alloy
   active confirmed across all 11 fleet hosts.
2. Fix KB: Alloy Agent Model was stale - version had moved from
   v1.18.0 to v1.18.1 on all 8 'newer' hosts since last checked,
   role's own default still unreconciled at 1.17.1. Fixed and
   confirmed via re-fetch.
3. Pulled the full Observability Pipeline tree: Alloy Agent Model,
   Log Flow, Metrics Flow.
4. Replaced both mermaid diagrams (broke this repo's Nextcloud-SVG
   convention) with real diagrams built through the proper round-trip.
5. Evidence: real journald config dumps, plus 3 real Grafana
   screenshots (host metrics dashboard, detailed Loki log stream
   showing job=technitium-dns/docker/system, and the fleet-wide log
   volume dashboard) - one screenshot round caught and fixed an
   unredacted internal IP before being used.
6/7. Compelling-proof test passes; sanitize-check clean.

assets/observability-log-fanout.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Do not edit this file with editors other than draw.io -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="273px" viewBox="0 0 872 273" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Log Fanout&quot; id=&quot;obs-logfanout-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;320&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;a1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;system logs&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=system&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a1&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b1&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;a2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker API&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;container logs, survives recreation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=docker&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a2&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b2&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;a3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;DNS query log&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;daily rotation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=dns, real event time&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a3&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b3Line truncated

assets/observability-pipeline.svg

@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- Do not edit this file with editors other than draw.io -->
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="838px" height="324px" viewBox="0 0 838 324" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Pipeline&quot; id=&quot;obs-pipeline-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1178&quot; dy=&quot;659&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;420&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;srcgroup&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;startSize=30;swimlane;fillColor=none;strokeColor=#666666;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=center;&quot; value=&quot;PER-HOST SOURCES&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;320&quot; width=&quot;220&quot; x=&quot;30&quot; y=&quot;40&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;s1&quot; parent=&quot;srcgroup&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;persistent, 14-day retention&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;180&quot; x=&quot;20&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;s2&quot; parent=&quot;srcgroup&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker container&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;stdout/stderr, docker_hosts only&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;180&quot; x=&quot;20&quot; y=&quot;105&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;s3&quot; parent=&quot;srcgroup&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Node metrics&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;CPU / mem / disk / net&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;180&quot; x=&quot;20&quot; y=&quot;180&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;s4&quot; parent=&quot;srcgroup&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Native app metrics&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;5 apps expose endpoints&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;180&quot; x=&quot;20&quot; y=&quot;255&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;agent&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Grafana Alloy&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;native systemd service&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;80&quot; width=&quot;200&quot; x=&quot;360&quot; y=&quot;160&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;loki&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Loki&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;160&quot; x=&quot;700&quot; y=&quot;70&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;prom&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;&quot; value=&quot;Prometheus&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;160&quot; x=&quot;700&quot; y=&quot;270&quot; as=&quot;geometry&quot; /&gt;&#10;    Line truncated

docs/observability.md

@@ -16,24 +16,7 @@ A pull-based model (Prometheus scraping every host directly) is the more common
 
 ## Pipeline
 
-```mermaid
+![Four per-host sources (journald, Docker containers, node metrics, native app metrics) all feed into Grafana Alloy, which ships logs to Loki and metrics to Prometheus via remote_write every 30s](../assets/observability-pipeline.svg)
-flowchart LR
-    subgraph Sources["Per-host sources"]
-        J["journald<br/>persistent, 14-day retention"]
-        D["Docker container<br/>stdout/stderr"]
-        N["node metrics<br/>CPU/mem/disk/net"]
-        App["Native app metrics<br/>5 apps expose endpoints"]
-    end
-
-    Agent["Grafana Alloy<br/>native systemd service"]
-    J --> Agent
-    D -->|docker_hosts only| Agent
-    N --> Agent
-    App --> Agent
-
-    Agent -->|logs| Loki["Loki"]
-    Agent -->|remote_write, 30s| Prom["Prometheus"]
-```
 
 ## Agent install: deliberate, not default
 
@@ -42,14 +25,33 @@ flowchart LR
 
 ## Logs: journald only, no double-handling
 
-The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet.
+The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet. This is the real config on two different hosts, not a described default:
 
-```mermaid
-flowchart LR
-    A["journald<br/>system logs"] --> L1["Loki<br/>job=system"]
-    B["Docker API<br/>container logs<br/>survives recreation"] --> L2["Loki<br/>job=docker"]
-    C["DNS query log<br/>daily rotation"] --> L3["Loki<br/>job=dns · real event time"]
 ```
+$ cat /etc/systemd/journald.conf.d/*.conf   # the higher-volume host
+[Journal]
+Storage=persistent
+SystemMaxUse=1G
+SystemKeepFree=1G
+MaxRetentionSec=14day
+
+$ cat /etc/systemd/journald.conf.d/*.conf   # every other host
+[Journal]
+Storage=persistent
+SystemMaxUse=500M
+SystemKeepFree=1G
+MaxRetentionSec=14day
+```
+
+![Three independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, and the DNS query log as job=dns with real event time instead of ingestion time](../assets/observability-log-fanout.svg)
+
+This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance:
+
+![Real Grafana logs view: job=technitium-dns DNS query/answer lines, job=docker container log lines, and a job=system journald line, all in the same log stream](../assets/observability-logs-detail.png)
+
+And the aggregate view across the whole fleet — real log volume, not an estimate:
+
+![A real Grafana dashboard: total log line count, log volume broken down by host, and log rate over time — actual ingested data, not a described capacity](../assets/observability-logs-overview.png)
 
 Every host's logs land in the same Loki instance under its own host label, filterable in Grafana by host or job — the same "one source of truth, not N inconsistent ones" instinct behind the DNS model in [`docs/dns-naming.md`](dns-naming.md). Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated during the update flows in [`docs/registry-and-build.md`](registry-and-build.md) — tailing a file path would silently stop working the moment that container got rebuilt. The DNS query log pipeline (feeding into [`docs/dns-naming.md`](dns-naming.md)'s enforcement story) is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time — the same UTC-vs-wall-clock discipline that matters when querying Wazuh's alert data in [`docs/security-monitoring.md`](security-monitoring.md).
 
@@ -63,7 +65,9 @@ Every host's logs land in the same Loki instance under its own host label, filte
 | n8n | Built-in metrics endpoint |
 | Authentik | Built-in metrics endpoint, localhost-only |
 
-Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open.
+Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open. This is that same push path landing in Grafana, for a real host's own metrics — not one of the 5 app endpoints specifically, but proof the pipeline actually delivers real numbers, not placeholder gauges:
+
+![A real Grafana dashboard built entirely from remote_write metrics: live CPU/RAM/swap/disk gauges and real 2-day CPU, memory, network, and disk-usage graphs for an actual fleet host](../assets/observability-metrics-dashboard.png)
 
 ---
 Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)