automation-bot
a5254a95ac
A real log source was added the day after this doc's last verify (2026-09-07): auditd logs now ship to Loki too, on the 6 hosts carrying the auditd_hardening role, after a real incident where one host's /var/log/audit filled to 100% with zero external visibility. Updated the diagram to add the 4th branch (rebuilt via Nextcloud Diagramming, re-exported) and added prose describing it. Also corrected the diagram's DNS branch label (job=dns -> job=technitium-dns, matching the doc's own already-correct evidence-screenshot caption). Journald sizing config (1G rocky-docker override, 500M fleet default) confirmed byte-for-byte accurate live on both.
81 lines
7.5 KiB
Markdown
81 lines
7.5 KiB
Markdown
<!-- verified: 2026-09-19
|
|
iac@f770e1c : ansible/roles/monitoring/observability_agent
|
|
trilium : eeiUz4iTEgPL, zquji4XOpw5M, s5tWZshDEpQb, oWr1jJhvqitv
|
|
-->
|
|
|
|
# 📈 Observability
|
|
|
|
> [!NOTE]
|
|
> Grafana Alloy runs natively on every fleet host, shipping logs to Loki and metrics to Prometheus via `remote_write` — pushed outward rather than scraped, a choice made specifically because of the firewall model in [`docs/firewall-and-network.md`](firewall-and-network.md), not despite it.
|
|
|
|
| | |
|
|
|---|---|
|
|
| **Agent** | Grafana Alloy, native systemd service on every host, not Docker |
|
|
| **Logs** | journald → Loki |
|
|
| **Metrics** | Push-based `remote_write`, not pull-based scraping |
|
|
| **Coverage** | All fleet hosts, confirmed live |
|
|
|
|
## Why Push Instead Of Pull — The Actual Eeasoning
|
|
|
|
A pull-based model (Prometheus scraping every host directly) is the more common default, and it was rejected for a specific reason tied directly to [`docs/firewall-and-network.md`](firewall-and-network.md): scraping means the monitoring host needs inbound network access to every single host it monitors, which means punching a hole through the zero-blanket-allow firewall model for every host, every metrics port. A push model inverts that — only the monitoring host needs to accept inbound writes, and every other host only ever *initiates* outbound connections, which fits the existing "explicit outbound rule, default-deny inbound" posture instead of fighting it. The tradeoff is real: push-based metrics are slightly harder to reason about for ad-hoc debugging (no simple `curl` against a target to see what it'd report), but it was the right call given the network model already in place.
|
|
|
|
## Pipeline
|
|
|
|
|
|
|
|
## Agent Install: deliberate, not default
|
|
|
|
> [!TIP]
|
|
> The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. That's a real maintenance cost (manual version bumps instead of `apt upgrade` handling it), accepted because it keeps the exact version pinned and reproducible across the fleet through the same tag-driven Ansible model described in [`docs/iac-pipeline.md`](iac-pipeline.md), rather than whatever version a package repo happens to serve on a given day. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).
|
|
|
|
## Logs: journald only, no double-handling
|
|
|
|
The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet. This is the real config on two different hosts, not a described default:
|
|
|
|
```
|
|
$ cat /etc/systemd/journald.conf.d/*.conf # the higher-volume host
|
|
[Journal]
|
|
Storage=persistent
|
|
SystemMaxUse=1G
|
|
SystemKeepFree=1G
|
|
MaxRetentionSec=14day
|
|
|
|
$ cat /etc/systemd/journald.conf.d/*.conf # every other host
|
|
[Journal]
|
|
Storage=persistent
|
|
SystemMaxUse=500M
|
|
SystemKeepFree=1G
|
|
MaxRetentionSec=14day
|
|
```
|
|
|
|
|
|
|
|
A fourth, newer branch ships raw auditd logs to Loki too, on the 6 hosts that carry the `auditd_hardening` compliance role (automation, caddy, monitoring, technitium, vault, wazuh) — a real gap-fill after one host's own `/var/log/audit` once filled to 100% with zero external visibility to catch it. It doesn't replace the local cleanup timer that actually bounds that disk usage; it just means the data isn't only ever local anymore.
|
|
|
|
This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance:
|
|
|
|
|
|
|
|
And the aggregate view across the whole fleet — real log volume, not an estimate:
|
|
|
|
|
|
|
|
Every host's logs land in the same Loki instance under its own host label, filterable in Grafana by host or job — the same "one source of truth, not N inconsistent ones" instinct behind the DNS model in [`docs/dns-naming.md`](dns-naming.md). Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated during the update flows in [`docs/registry-and-build.md`](registry-and-build.md) — tailing a file path would silently stop working the moment that container got rebuilt. The DNS query log pipeline (feeding into [`docs/dns-naming.md`](dns-naming.md)'s enforcement story) is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time — the same UTC-vs-wall-clock discipline that matters when querying Wazuh's alert data in [`docs/security-monitoring.md`](security-monitoring.md).
|
|
|
|
## Metrics: pushed, not pulled
|
|
|
|
| App | What's exposed |
|
|
|---|---|
|
|
| Vault | Dedicated read-only metrics token, separate from its normal agent token — see [`docs/secrets-and-pki.md`](secrets-and-pki.md) |
|
|
| Kestra | Built-in metrics endpoint |
|
|
| Gitea | Built-in metrics endpoint |
|
|
| n8n | Built-in metrics endpoint |
|
|
| Authentik | Built-in metrics endpoint, localhost-only |
|
|
|
|
Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open. This is that same push path landing in Grafana, for a real host's own metrics — not one of the 5 app endpoints specifically, but proof the pipeline actually delivers real numbers, not placeholder gauges:
|
|
|
|
|
|
|
|
---
|
|
Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)
|