diff --git a/assets/observability-log-fanout.svg b/assets/observability-log-fanout.svg
new file mode 100644
index 0000000..382f26c
--- /dev/null
+++ b/assets/observability-log-fanout.svg
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
diff --git a/assets/observability-logs-detail.png b/assets/observability-logs-detail.png
new file mode 100644
index 0000000..a14a413
Binary files /dev/null and b/assets/observability-logs-detail.png differ
diff --git a/assets/observability-logs-overview.png b/assets/observability-logs-overview.png
new file mode 100644
index 0000000..56f6400
Binary files /dev/null and b/assets/observability-logs-overview.png differ
diff --git a/assets/observability-metrics-dashboard.png b/assets/observability-metrics-dashboard.png
new file mode 100644
index 0000000..eefb06b
Binary files /dev/null and b/assets/observability-metrics-dashboard.png differ
diff --git a/assets/observability-pipeline.svg b/assets/observability-pipeline.svg
new file mode 100644
index 0000000..87684bd
--- /dev/null
+++ b/assets/observability-pipeline.svg
@@ -0,0 +1,4 @@
+
+
+
+
\ No newline at end of file
diff --git a/docs/observability.md b/docs/observability.md
index 63c9e70..b6a6995 100644
--- a/docs/observability.md
+++ b/docs/observability.md
@@ -16,24 +16,7 @@ A pull-based model (Prometheus scraping every host directly) is the more common
## Pipeline
-```mermaid
-flowchart LR
- subgraph Sources["Per-host sources"]
- J["journald persistent, 14-day retention"]
- D["Docker container stdout/stderr"]
- N["node metrics CPU/mem/disk/net"]
- App["Native app metrics 5 apps expose endpoints"]
- end
-
- Agent["Grafana Alloy native systemd service"]
- J --> Agent
- D -->|docker_hosts only| Agent
- N --> Agent
- App --> Agent
-
- Agent -->|logs| Loki["Loki"]
- Agent -->|remote_write, 30s| Prom["Prometheus"]
-```
+
## Agent install: deliberate, not default
@@ -42,14 +25,33 @@ flowchart LR
## Logs: journald only, no double-handling
-The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet.
+The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet. This is the real config on two different hosts, not a described default:
-```mermaid
-flowchart LR
- A["journald system logs"] --> L1["Loki job=system"]
- B["Docker API container logs survives recreation"] --> L2["Loki job=docker"]
- C["DNS query log daily rotation"] --> L3["Loki job=dns · real event time"]
```
+$ cat /etc/systemd/journald.conf.d/*.conf # the higher-volume host
+[Journal]
+Storage=persistent
+SystemMaxUse=1G
+SystemKeepFree=1G
+MaxRetentionSec=14day
+
+$ cat /etc/systemd/journald.conf.d/*.conf # every other host
+[Journal]
+Storage=persistent
+SystemMaxUse=500M
+SystemKeepFree=1G
+MaxRetentionSec=14day
+```
+
+
+
+This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance:
+
+
+
+And the aggregate view across the whole fleet — real log volume, not an estimate:
+
+
Every host's logs land in the same Loki instance under its own host label, filterable in Grafana by host or job — the same "one source of truth, not N inconsistent ones" instinct behind the DNS model in [`docs/dns-naming.md`](dns-naming.md). Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated during the update flows in [`docs/registry-and-build.md`](registry-and-build.md) — tailing a file path would silently stop working the moment that container got rebuilt. The DNS query log pipeline (feeding into [`docs/dns-naming.md`](dns-naming.md)'s enforcement story) is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time — the same UTC-vs-wall-clock discipline that matters when querying Wazuh's alert data in [`docs/security-monitoring.md`](security-monitoring.md).
@@ -63,7 +65,9 @@ Every host's logs land in the same Loki instance under its own host label, filte
| n8n | Built-in metrics endpoint |
| Authentik | Built-in metrics endpoint, localhost-only |
-Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open.
+Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open. This is that same push path landing in Grafana, for a real host's own metrics — not one of the 5 app endpoints specifically, but proof the pipeline actually delivers real numbers, not placeholder gauges:
+
+
---
Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)