docs: fix observability.md - add missing 4th log fan-out (auditd)

A real log source was added the day after this doc's last verify
(2026-09-07): auditd logs now ship to Loki too, on the 6 hosts
carrying the auditd_hardening role, after a real incident where one
host's /var/log/audit filled to 100% with zero external visibility.
Updated the diagram to add the 4th branch (rebuilt via Nextcloud
Diagramming, re-exported) and added prose describing it. Also
corrected the diagram's DNS branch label (job=dns -> job=technitium-dns,
matching the doc's own already-correct evidence-screenshot caption).
Journald sizing config (1G rocky-docker override, 500M fleet default)
confirmed byte-for-byte accurate live on both.

assets/observability-log-fanout.svg

@@ -1,4 +1,4 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <!-- Do not edit this file with editors other than draw.io -->
 <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
-<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="273px" viewBox="0 0 872 273" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#10;  &lt;diagram name=&quot;Log Fanout&quot; id=&quot;obs-logfanout-1&quot;&gt;&#10;    &lt;mxGraphModel dx=&quot;1414&quot; dy=&quot;791&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;320&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#10;      &lt;root&gt;&#10;        &lt;mxCell id=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#10;        &lt;mxCell id=&quot;a1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;system logs&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=system&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a1&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b1&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;a2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker API&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;container logs, survives recreation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=docker&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a2&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b2&quot;&gt;&#10;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;a3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;DNS query log&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;daily rotation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;b3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=dns, real event time&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#10;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#10;        &lt;/mxCell&gt;&#10;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a3&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b3Line truncated
+<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="373px" viewBox="0 0 872 373" content="&lt;mxfile host=&quot;embed.diagrams.net&quot; scale=&quot;1&quot; border=&quot;0&quot;&gt;&#xA;  &lt;diagram name=&quot;Log Fanout&quot; id=&quot;obs-logfanout-1&quot;&gt;&#xA;    &lt;mxGraphModel dx=&quot;1398&quot; dy=&quot;830&quot; grid=&quot;1&quot; gridSize=&quot;10&quot; guides=&quot;1&quot; tooltips=&quot;1&quot; connect=&quot;1&quot; arrows=&quot;1&quot; fold=&quot;1&quot; page=&quot;1&quot; pageScale=&quot;1&quot; pageWidth=&quot;1000&quot; pageHeight=&quot;420&quot; math=&quot;0&quot; shadow=&quot;0&quot;&gt;&#xA;      &lt;root&gt;&#xA;        &lt;mxCell id=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;1&quot; parent=&quot;0&quot; /&gt;&#xA;        &lt;mxCell id=&quot;a1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;journald&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;system logs&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;b1&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=system&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;30&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e1&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a1&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b1&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;a2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Docker API&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;container logs, survives recreation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;b2&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=docker&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;130&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e2&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a2&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; target=&quot;b2&quot;&gt;&#xA;          &lt;mxGeometry relative=&quot;1&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;a3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;DNS query log&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;daily rotation&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;40&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;b3&quot; parent=&quot;1&quot; style=&quot;rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;&quot; value=&quot;Loki&amp;lt;br&amp;gt;&amp;lt;font style=&amp;quot;font-size: 10px;&amp;quot;&amp;gt;job=technitium-dns, real event time&amp;lt;/font&amp;gt;&quot; vertex=&quot;1&quot;&gt;&#xA;          &lt;mxGeometry height=&quot;60&quot; width=&quot;200&quot; x=&quot;700&quot; y=&quot;230&quot; as=&quot;geometry&quot; /&gt;&#xA;        &lt;/mxCell&gt;&#xA;        &lt;mxCell id=&quot;e3&quot; edge=&quot;1&quot; parent=&quot;1&quot; source=&quot;a3&quot; style=&quot;html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;&quot; targLine truncated

docs/observability.md

@@ -1,5 +1,5 @@
-<!-- verified: 2026-09-06
-     iac@49ff427 : ansible/roles/monitoring/observability_agent
+<!-- verified: 2026-09-19
+     iac@f770e1c : ansible/roles/monitoring/observability_agent
      trilium : eeiUz4iTEgPL, zquji4XOpw5M, s5tWZshDEpQb, oWr1jJhvqitv
 -->
 
@@ -48,7 +48,9 @@ SystemKeepFree=1G
 MaxRetentionSec=14day
 ```
 
-![Three independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, and the DNS query log as job=dns with real event time instead of ingestion time](../assets/observability-log-fanout.svg)
+![Four independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, the DNS query log as job=technitium-dns with real event time instead of ingestion time, and auditd's own log as job=auditd on the 6 hosts that carry it](../assets/observability-log-fanout.svg)
+
+A fourth, newer branch ships raw auditd logs to Loki too, on the 6 hosts that carry the `auditd_hardening` compliance role (automation, caddy, monitoring, technitium, vault, wazuh) — a real gap-fill after one host's own `/var/log/audit` once filled to 100% with zero external visibility to catch it. It doesn't replace the local cleanup timer that actually bounds that disk usage; it just means the data isn't only ever local anymore.
 
 This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance: