public/docs/observability.md

📈 Observability

Note

Every fleet host runs the same collection agent natively (not as a container), pushing logs and metrics outward rather than waiting to be scraped — a deliberate choice that keeps monitoring off the firewall's critical path.

Agent	Native systemd service on every host, not Docker
Logs	journald → Loki
Metrics	Push-based remote_write, not pull-based scraping
Coverage	All fleet hosts, confirmed live

Pipeline

flowchart LR
    subgraph Sources["Per-host sources"]
        J["journald<br/>persistent, 14-day retention"]
        D["Docker container<br/>stdout/stderr"]
        N["node metrics<br/>CPU/mem/disk/net"]
        App["Native app metrics<br/>5 apps expose endpoints"]
    end

    Agent["Collection Agent<br/>native systemd service"]
    J --> Agent
    D -->|docker_hosts only| Agent
    N --> Agent
    App --> Agent

    Agent -->|logs| Loki["Loki"]
    Agent -->|remote_write, 30s| Prom["Prometheus"]

Agent install: deliberate, not default

Tip

The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).

Warning

Honestly-tracked version drift: most hosts run the current pinned agent version, but three legacy hosts remain one version behind, and the automation role's own default hasn't caught up to what's actually deployed on the majority. Documented as known drift, not silently ignored.

Logs: journald only, no double-handling

The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume.

flowchart LR
    A["journald<br/>system logs"] --> L1["Loki<br/>job=system"]
    B["Docker API<br/>container logs<br/>survives recreation"] --> L2["Loki<br/>job=docker"]
    C["DNS query log<br/>daily rotation"] --> L3["Loki<br/>job=dns · real event time"]

Every host's logs land in the same log store under its own host label, filterable in the dashboard by host or job. Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated. The DNS query log pipeline is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time.

Metrics: pushed, not pulled

Important

Why push instead of pull: Prometheus doesn't need network access to every host — only the monitoring host needs to accept inbound writes. This matters directly for hosts sitting behind the tighter firewall rules described in firewall-and-network.md — a pull model would mean punching a scrape-target hole through that segmentation for every single host.

App	What's exposed
Secrets platform	Dedicated read-only metrics token, separate from its normal agent token
Orchestration engine	Built-in metrics endpoint
Git server	Built-in metrics endpoint
Workflow automation	Built-in metrics endpoint
Identity provider	Built-in metrics endpoint, localhost-only

Five apps expose their own metrics endpoints directly, scraped by that host's own agent and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility.

---

Related stacks: stacks/platform/grafana-stack/