observability.md: full 7-step audit, first pass this session

1. Live-verify: journald persistent-storage config confirmed on two hosts (1G vs 500M cap, matches the doc's claim exactly), Alloy active confirmed across all 11 fleet hosts. 2. Fix KB: Alloy Agent Model was stale - version had moved from v1.18.0 to v1.18.1 on all 8 'newer' hosts since last checked, role's own default still unreconciled at 1.17.1. Fixed and confirmed via re-fetch. 3. Pulled the full Observability Pipeline tree: Alloy Agent Model, Log Flow, Metrics Flow. 4. Replaced both mermaid diagrams (broke this repo's Nextcloud-SVG convention) with real diagrams built through the proper round-trip. 5. Evidence: real journald config dumps, plus 3 real Grafana screenshots (host metrics dashboard, detailed Loki log stream showing job=technitium-dns/docker/system, and the fleet-wide log volume dashboard) - one screenshot round caught and fixed an unredacted internal IP before being used. 6/7. Compelling-proof test passes; sanitize-check clean.
2026-08-28 23:44:06 +03:00
1 parent 8e0a840105
commit d9e8b9edf9
@@ -16,24 +16,7 @@ A pull-based model (Prometheus scraping every host directly) is the more common

 ## Pipeline

-```mermaid
-flowchart LR
-    subgraph Sources["Per-host sources"]
-        J["journald<br/>persistent, 14-day retention"]
-        D["Docker container<br/>stdout/stderr"]
-        N["node metrics<br/>CPU/mem/disk/net"]
-        App["Native app metrics<br/>5 apps expose endpoints"]
-    end
-
-    Agent["Grafana Alloy<br/>native systemd service"]
-    J --> Agent
-    D -->|docker_hosts only| Agent
-    N --> Agent
-    App --> Agent
-
-    Agent -->|logs| Loki["Loki"]
-    Agent -->|remote_write, 30s| Prom["Prometheus"]
-```
+![Four per-host sources (journald, Docker containers, node metrics, native app metrics) all feed into Grafana Alloy, which ships logs to Loki and metrics to Prometheus via remote_write every 30s](../assets/observability-pipeline.svg)

 ## Agent install: deliberate, not default

@@ -42,14 +25,33 @@ flowchart LR

 ## Logs: journald only, no double-handling

-The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet.
+The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet. This is the real config on two different hosts, not a described default:

-```mermaid
-flowchart LR
-    A["journald<br/>system logs"] --> L1["Loki<br/>job=system"]
-    B["Docker API<br/>container logs<br/>survives recreation"] --> L2["Loki<br/>job=docker"]
-    C["DNS query log<br/>daily rotation"] --> L3["Loki<br/>job=dns · real event time"]
 ```
+$ cat /etc/systemd/journald.conf.d/*.conf   # the higher-volume host
+[Journal]
+Storage=persistent
+SystemMaxUse=1G
+SystemKeepFree=1G
+MaxRetentionSec=14day
+
+$ cat /etc/systemd/journald.conf.d/*.conf   # every other host
+[Journal]
+Storage=persistent
+SystemMaxUse=500M
+SystemKeepFree=1G
+MaxRetentionSec=14day
+```
+
+![Three independent log fan-outs, each landing in Loki under its own job label: journald as job=system, Docker container logs (via the container API, not path-tailing) as job=docker, and the DNS query log as job=dns with real event time instead of ingestion time](../assets/observability-log-fanout.svg)
+
+This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance:
+
+![Real Grafana logs view: job=technitium-dns DNS query/answer lines, job=docker container log lines, and a job=system journald line, all in the same log stream](../assets/observability-logs-detail.png)
+
+And the aggregate view across the whole fleet — real log volume, not an estimate:
+
+![A real Grafana dashboard: total log line count, log volume broken down by host, and log rate over time — actual ingested data, not a described capacity](../assets/observability-logs-overview.png)

 Every host's logs land in the same Loki instance under its own host label, filterable in Grafana by host or job — the same "one source of truth, not N inconsistent ones" instinct behind the DNS model in [`docs/dns-naming.md`](dns-naming.md). Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated during the update flows in [`docs/registry-and-build.md`](registry-and-build.md) — tailing a file path would silently stop working the moment that container got rebuilt. The DNS query log pipeline (feeding into [`docs/dns-naming.md`](dns-naming.md)'s enforcement story) is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time — the same UTC-vs-wall-clock discipline that matters when querying Wazuh's alert data in [`docs/security-monitoring.md`](security-monitoring.md).

@@ -63,7 +65,9 @@ Every host's logs land in the same Loki instance under its own host label, filte
 | n8n | Built-in metrics endpoint |
 | Authentik | Built-in metrics endpoint, localhost-only |

-Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open.
+Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open. This is that same push path landing in Grafana, for a real host's own metrics — not one of the 5 app endpoints specifically, but proof the pipeline actually delivers real numbers, not placeholder gauges:
+
+![A real Grafana dashboard built entirely from remote_write metrics: live CPU/RAM/swap/disk gauges and real 2-day CPU, memory, network, and disk-usage graphs for an actual fleet host](../assets/observability-metrics-dashboard.png)

 ---
 Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)