# 📈 Observability

> [!NOTE]
> Every fleet host runs the same collection agent natively (not as a container), pushing logs and metrics outward rather than waiting to be scraped — a deliberate choice that keeps monitoring off the firewall's critical path.

| | |
|---|---|
| **Agent** | Native systemd service on every host, not Docker |
| **Logs** | journald → Loki |
| **Metrics** | Push-based `remote_write`, not pull-based scraping |
| **Coverage** | All fleet hosts, confirmed live |

## Pipeline

```mermaid
flowchart LR
    subgraph Sources["Per-host sources"]
        J["journald<br/>persistent, 14-day retention"]
        D["Docker container<br/>stdout/stderr"]
        N["node metrics<br/>CPU/mem/disk/net"]
        App["Native app metrics<br/>5 apps expose endpoints"]
    end

    Agent["Collection Agent<br/>native systemd service"]
    J --> Agent
    D -->|docker_hosts only| Agent
    N --> Agent
    App --> Agent

    Agent -->|logs| Loki["Loki"]
    Agent -->|remote_write, 30s| Prom["Prometheus"]
```

## Agent install: deliberate, not default

> [!TIP]
> The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).

## Logs: journald only, no double-handling

The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume.

```mermaid
flowchart LR
    A["journald<br/>system logs"] --> L1["Loki<br/>job=system"]
    B["Docker API<br/>container logs<br/>survives recreation"] --> L2["Loki<br/>job=docker"]
    C["DNS query log<br/>daily rotation"] --> L3["Loki<br/>job=dns · real event time"]
```

Every host's logs land in the same log store under its own host label, filterable in the dashboard by host or job. Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated. The DNS query log pipeline is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time.

## Metrics: pushed, not pulled

> [!IMPORTANT]
> **Why push instead of pull**: Prometheus doesn't need network access to every host — only the monitoring host needs to accept inbound writes. This matters directly for hosts sitting behind the tighter firewall rules described in [`firewall-and-network.md`](firewall-and-network.md) — a pull model would mean punching a scrape-target hole through that segmentation for every single host.

| App | What's exposed |
|---|---|
| Secrets platform | Dedicated read-only metrics token, separate from its normal agent token |
| Orchestration engine | Built-in metrics endpoint |
| Git server | Built-in metrics endpoint |
| Workflow automation | Built-in metrics endpoint |
| Identity provider | Built-in metrics endpoint, localhost-only |

Five apps expose their own metrics endpoints directly, scraped by that host's own agent and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility.

---
Related stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)