diff --git a/docs/notifications.md b/docs/notifications.md new file mode 100644 index 0000000..1c86806 --- /dev/null +++ b/docs/notifications.md @@ -0,0 +1,57 @@ +# ๐Ÿ”” Notifications + +> [!NOTE] +> Two independent branches โ€” app webhooks and OS-level host alerts โ€” converge on a single automation flow and one shared formatter before reaching Slack, so every alert reads the same regardless of where it came from. + +## Fan-in, fan-out + +```mermaid +flowchart LR + subgraph FanIn["Fan-in"] + Apps["15 apps
HTTP webhook"] + Dedicated["2 apps
dedicated endpoints
non-JSON payloads"] + MQTT["Every host
MQTT publish"] + end + + Apps --> Router["App Router
payload-shape detection"] + Dedicated --> Router + Router --> Mapper["Per-app mapper"] + MQTT --> Parser["Inventory parser"] + + Mapper --> Subflow["Shared formatting subflow"] + Parser --> Subflow + Subflow --> Slack["Slack"] +``` + +> [!WARNING] +> **Debt, documented rather than hidden**: of the apps wired into the router, a couple no longer resolve to anything โ€” one was fully decommissioned but its detection code was never swept out, and another's alerts actually travel a completely different path than its own webhook route suggests, making that route effectively dead code. Two newer, genuinely live apps aren't wired into either branch yet โ€” a real gap against the fleet's own documented pattern for onboarding a new app's alerts. + +## Message broker + +| | | +|---|---| +| **Broker** | Runs as its own container on the automation host | +| **Protocol** | MQTT v3.1.1, auth required, anonymous disabled | +| **Topic structure** | One topic per host, wildcard-subscribed centrally | +| **Users** | Two โ€” a publish identity used fleet-wide, and the automation flow's own read-only subscriber | + +Both broker users are credential-managed through the same secret-rendering pipeline as everything else in [`docs/secrets-and-pki.md`](secrets-and-pki.md) โ€” not a bare manually-edited password file. + +> [!IMPORTANT] +> **A real, tracked fragility**: the publish identity's password has to stay in sync across three separate places โ€” the rendered broker credential file, the source value used to render every host's own publish script, and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from the secret store at all). Rotating it means updating all three by hand โ€” a documented gap, not a silent one. + +## Detection logic + +The router figures out which app sent a webhook by inspecting the payload's shape โ€” an explicit field for some apps, structural pattern-matching for others (one app, for instance, is identified purely by a nested field only that vendor's payload happens to include). Two apps skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON. + +## Every host, not just apps + +> [!TIP] +> A dedicated automation role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert. It runs over 20 distinct checks per host โ€” disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (audit log anomalies, authentication failures) โ€” deliberately complementary to the SOC stack in [`docs/security-monitoring.md`](security-monitoring.md), not duplicating its file-integrity or account monitoring. + +## One formatter, every source + +Every branch โ€” app webhooks, host-level alerts, and the two dedicated non-JSON endpoints โ€” converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it. + +--- +Related stacks: [`stacks/platform/mosquitto/`](../stacks/platform/mosquitto/), [`stacks/platform/node-red/`](../stacks/platform/node-red/) diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 0000000..7a1fc7e --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,71 @@ +# ๐Ÿ“ˆ Observability + +> [!NOTE] +> Every fleet host runs the same collection agent natively (not as a container), pushing logs and metrics outward rather than waiting to be scraped โ€” a deliberate choice that keeps monitoring off the firewall's critical path. + +| | | +|---|---| +| **Agent** | Native systemd service on every host, not Docker | +| **Logs** | journald โ†’ Loki | +| **Metrics** | Push-based `remote_write`, not pull-based scraping | +| **Coverage** | All fleet hosts, confirmed live | + +## Pipeline + +```mermaid +flowchart LR + subgraph Sources["Per-host sources"] + J["journald
persistent, 14-day retention"] + D["Docker container
stdout/stderr"] + N["node metrics
CPU/mem/disk/net"] + App["Native app metrics
5 apps expose endpoints"] + end + + Agent["Collection Agent
native systemd service"] + J --> Agent + D -->|docker_hosts only| Agent + N --> Agent + App --> Agent + + Agent -->|logs| Loki["Loki"] + Agent -->|remote_write, 30s| Prom["Prometheus"] +``` + +## Agent install: deliberate, not default + +> [!TIP] +> The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit โ€” not installed from a package repo or the vendor's own packaged service file. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway). + +> [!WARNING] +> **Honestly-tracked version drift**: most hosts run the current pinned agent version, but three legacy hosts remain one version behind, and the automation role's own default hasn't caught up to what's actually deployed on the majority. Documented as known drift, not silently ignored. + +## Logs: journald only, no double-handling + +The system's built-in syslog daemon is explicitly disabled once journald takes over โ€” avoids the same log stream getting handled twice. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume. + +```mermaid +flowchart LR + A["journald
system logs"] --> L1["Loki
job=system"] + B["Docker API
container logs
survives recreation"] --> L2["Loki
job=docker"] + C["DNS query log
daily rotation"] --> L3["Loki
job=dns ยท real event time"] +``` + +Every host's logs land in the same log store under its own host label, filterable in the dashboard by host or job. Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated. The DNS query log pipeline is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time. + +## Metrics: pushed, not pulled + +> [!IMPORTANT] +> **Why push instead of pull**: Prometheus doesn't need network access to every host โ€” only the monitoring host needs to accept inbound writes. This matters directly for hosts sitting behind the tighter firewall rules described in [`firewall-and-network.md`](firewall-and-network.md) โ€” a pull model would mean punching a scrape-target hole through that segmentation for every single host. + +| App | What's exposed | +|---|---| +| Secrets platform | Dedicated read-only metrics token, separate from its normal agent token | +| Orchestration engine | Built-in metrics endpoint | +| Git server | Built-in metrics endpoint | +| Workflow automation | Built-in metrics endpoint | +| Identity provider | Built-in metrics endpoint, localhost-only | + +Five apps expose their own metrics endpoints directly, scraped by that host's own agent and pushed through the same path as host-level metrics โ€” no separate pipeline needed for app-level visibility. + +--- +Related stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)