public/docs/observability.md

📈 Observability

Note

Grafana Alloy runs natively on every fleet host, shipping logs to Loki and metrics to Prometheus via remote_write — pushed outward rather than scraped, a choice made specifically because of the firewall model in docs/firewall-and-network.md, not despite it.

Agent	Grafana Alloy, native systemd service on every host, not Docker
Logs	journald → Loki
Metrics	Push-based remote_write, not pull-based scraping
Coverage	All fleet hosts, confirmed live

Why push instead of pull — the actual reasoning

A pull-based model (Prometheus scraping every host directly) is the more common default, and it was rejected for a specific reason tied directly to docs/firewall-and-network.md: scraping means the monitoring host needs inbound network access to every single host it monitors, which means punching a hole through the zero-blanket-allow firewall model for every host, every metrics port. A push model inverts that — only the monitoring host needs to accept inbound writes, and every other host only ever initiates outbound connections, which fits the existing "explicit outbound rule, default-deny inbound" posture instead of fighting it. The tradeoff is real: push-based metrics are slightly harder to reason about for ad-hoc debugging (no simple curl against a target to see what it'd report), but it was the right call given the network model already in place.

Pipeline

Agent install: deliberate, not default

Tip

The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. That's a real maintenance cost (manual version bumps instead of apt upgrade handling it), accepted because it keeps the exact version pinned and reproducible across the fleet through the same tag-driven Ansible model described in docs/iac-pipeline.md, rather than whatever version a package repo happens to serve on a given day. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).

Logs: journald only, no double-handling

The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet. This is the real config on two different hosts, not a described default:

$ cat /etc/systemd/journald.conf.d/*.conf   # the higher-volume host
[Journal]
Storage=persistent
SystemMaxUse=1G
SystemKeepFree=1G
MaxRetentionSec=14day

$ cat /etc/systemd/journald.conf.d/*.conf   # every other host
[Journal]
Storage=persistent
SystemMaxUse=500M
SystemKeepFree=1G
MaxRetentionSec=14day

This is real log data flowing through that pipeline, not a mockup — job=technitium-dns entries showing real DNS query/answer content, job=docker entries from real containers, and job=system from journald itself, all landing in the same Loki instance:

And the aggregate view across the whole fleet — real log volume, not an estimate:

Every host's logs land in the same Loki instance under its own host label, filterable in Grafana by host or job — the same "one source of truth, not N inconsistent ones" instinct behind the DNS model in docs/dns-naming.md. Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated during the update flows in docs/registry-and-build.md — tailing a file path would silently stop working the moment that container got rebuilt. The DNS query log pipeline (feeding into docs/dns-naming.md's enforcement story) is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time — the same UTC-vs-wall-clock discipline that matters when querying Wazuh's alert data in docs/security-monitoring.md.

Metrics: pushed, not pulled

App	What's exposed
Vault	Dedicated read-only metrics token, separate from its normal agent token — see docs/secrets-and-pki.md
Kestra	Built-in metrics endpoint
Gitea	Built-in metrics endpoint
n8n	Built-in metrics endpoint
Authentik	Built-in metrics endpoint, localhost-only

Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound remote_write connection every host already has open. This is that same push path landing in Grafana, for a real host's own metrics — not one of the 5 app endpoints specifically, but proof the pipeline actually delivers real numbers, not placeholder gauges:

---

Related: docs/firewall-and-network.md, docs/dns-naming.md, docs/security-monitoring.md, docs/dashy-soc-board.md · Stacks: stacks/platform/grafana-stack/