docs: fix observability.md - add missing 4th log fan-out (auditd)

A real log source was added the day after this doc's last verify
(2026-09-07): auditd logs now ship to Loki too, on the 6 hosts
carrying the auditd_hardening role, after a real incident where one
host's /var/log/audit filled to 100% with zero external visibility.
Updated the diagram to add the 4th branch (rebuilt via Nextcloud
Diagramming, re-exported) and added prose describing it. Also
corrected the diagram's DNS branch label (job=dns -> job=technitium-dns,
matching the doc's own already-correct evidence-screenshot caption).
Journald sizing config (1G rocky-docker override, 500M fleet default)
confirmed byte-for-byte accurate live on both.
This commit is contained in:
automation-bot committed 2026-09-19 09:28:11 +03:00
1 parent 8258339daf
commit a5254a95ac
2 files changed
+6 -4

No files matched your search

+5 -3
View File
@@ -1,5 +1,5 @@
<!-- verified: 2026-09-06
iac@49ff427 : ansible/roles/monitoring/observability_agent
<!-- verified: 2026-09-19
iac@f770e1c : ansible/roles/monitoring/observability_agent
trilium : eeiUz4iTEgPL, zquji4XOpw5M, s5tWZshDEpQb, oWr1jJhvqitv
-->
@@ -48,7 +48,9 @@ SystemKeepFree=1G
MaxRetentionSec=14day
```
![Three independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, and the DNS query log as job=dns with real event time instead of ingestion time](../assets/observability-log-fanout.svg)
![Four independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, the DNS query log as job=technitium-dns with real event time instead of ingestion time, and auditd's own log as job=auditd on the 6 hosts that carry it](../assets/observability-log-fanout.svg)
A fourth, newer branch ships raw auditd logs to Loki too, on the 6 hosts that carry the `auditd_hardening` compliance role (automation, caddy, monitoring, technitium, vault, wazuh) — a real gap-fill after one host's own `/var/log/audit` once filled to 100% with zero external visibility to catch it. It doesn't replace the local cleanup timer that actually bounds that disk usage; it just means the data isn't only ever local anymore.
This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance: