From a5254a95ac6dea53259c94aad05d23d9dfa387a4 Mon Sep 17 00:00:00 2001 From: automation-bot Date: Sat, 19 Sep 2026 09:28:11 +0300 Subject: [PATCH] docs: fix observability.md - add missing 4th log fan-out (auditd) A real log source was added the day after this doc's last verify (2026-09-07): auditd logs now ship to Loki too, on the 6 hosts carrying the auditd_hardening role, after a real incident where one host's /var/log/audit filled to 100% with zero external visibility. Updated the diagram to add the 4th branch (rebuilt via Nextcloud Diagramming, re-exported) and added prose describing it. Also corrected the diagram's DNS branch label (job=dns -> job=technitium-dns, matching the doc's own already-correct evidence-screenshot caption). Journald sizing config (1G rocky-docker override, 500M fleet default) confirmed byte-for-byte accurate live on both. --- assets/observability-log-fanout.svg | 2 +- docs/observability.md | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/assets/observability-log-fanout.svg b/assets/observability-log-fanout.svg index 382f26c..05254b6 100644 --- a/assets/observability-log-fanout.svg +++ b/assets/observability-log-fanout.svg @@ -1,4 +1,4 @@ -
journald
system logs
Loki
job=system
Docker API
container logs, survives recreation
Loki
job=docker
DNS query log
daily rotation
Loki
job=dns, real event time
 \ No newline at end of file +
journald
system logs
Loki
job=system
Docker API
container logs, survives recreation
Loki
job=docker
DNS query log
daily rotation
Loki
job=technitium-dns, real event time
auditd log
6 hosts, gap-fill
Loki
job=auditd, real event time
 \ No newline at end of file diff --git a/docs/observability.md b/docs/observability.md index 5b52486..32b2fbe 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -1,5 +1,5 @@ - @@ -48,7 +48,9 @@ SystemKeepFree=1G MaxRetentionSec=14day ``` -![Three independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, and the DNS query log as job=dns with real event time instead of ingestion time](../assets/observability-log-fanout.svg) +![Four independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, the DNS query log as job=technitium-dns with real event time instead of ingestion time, and auditd's own log as job=auditd on the 6 hosts that carry it](../assets/observability-log-fanout.svg) + +A fourth, newer branch ships raw auditd logs to Loki too, on the 6 hosts that carry the `auditd_hardening` compliance role (automation, caddy, monitoring, technitium, vault, wazuh) — a real gap-fill after one host's own `/var/log/audit` once filled to 100% with zero external visibility to catch it. It doesn't replace the local cleanup timer that actually bounds that disk usage; it just means the data isn't only ever local anymore. This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance: