docs: add observability and notifications write-ups

Same visual template - diagrams, callouts, tables - covering the
Alloy/Loki/Prometheus pipeline and the MQTT+webhook fan-in to Slack.

docs/notifications.md

@@ -0,0 +1,57 @@
+# 🔔 Notifications
+
+> [!NOTE]
+> Two independent branches — app webhooks and OS-level host alerts — converge on a single automation flow and one shared formatter before reaching Slack, so every alert reads the same regardless of where it came from.
+
+## Fan-in, fan-out
+
+```mermaid
+flowchart LR
+    subgraph FanIn["Fan-in"]
+        Apps["15 apps<br/>HTTP webhook"]
+        Dedicated["2 apps<br/>dedicated endpoints<br/>non-JSON payloads"]
+        MQTT["Every host<br/>MQTT publish"]
+    end
+
+    Apps --> Router["App Router<br/>payload-shape detection"]
+    Dedicated --> Router
+    Router --> Mapper["Per-app mapper"]
+    MQTT --> Parser["Inventory parser"]
+
+    Mapper --> Subflow["Shared formatting subflow"]
+    Parser --> Subflow
+    Subflow --> Slack["Slack"]
+```
+
+> [!WARNING]
+> **Debt, documented rather than hidden**: of the apps wired into the router, a couple no longer resolve to anything — one was fully decommissioned but its detection code was never swept out, and another's alerts actually travel a completely different path than its own webhook route suggests, making that route effectively dead code. Two newer, genuinely live apps aren't wired into either branch yet — a real gap against the fleet's own documented pattern for onboarding a new app's alerts.
+
+## Message broker
+
+| | |
+|---|---|
+| **Broker** | Runs as its own container on the automation host |
+| **Protocol** | MQTT v3.1.1, auth required, anonymous disabled |
+| **Topic structure** | One topic per host, wildcard-subscribed centrally |
+| **Users** | Two — a publish identity used fleet-wide, and the automation flow's own read-only subscriber |
+
+Both broker users are credential-managed through the same secret-rendering pipeline as everything else in [`docs/secrets-and-pki.md`](secrets-and-pki.md) — not a bare manually-edited password file.
+
+> [!IMPORTANT]
+> **A real, tracked fragility**: the publish identity's password has to stay in sync across three separate places — the rendered broker credential file, the source value used to render every host's own publish script, and a hardcoded copy inside the orchestration engine's own alert-publishing flow definition (not sourced from the secret store at all). Rotating it means updating all three by hand — a documented gap, not a silent one.
+
+## Detection logic
+
+The router figures out which app sent a webhook by inspecting the payload's shape — an explicit field for some apps, structural pattern-matching for others (one app, for instance, is identified purely by a nested field only that vendor's payload happens to include). Two apps skip the shared router entirely with their own dedicated endpoints, since they post plain text rather than JSON.
+
+## Every host, not just apps
+
+> [!TIP]
+> A dedicated automation role puts every fleet host onto this same pipeline for OS-level health, independent of any application-level alert. It runs over 20 distinct checks per host — disk thresholds, password/certificate expiry, failed logins (both local and from outside the trusted network), cron/systemd failures, container health, pending reboots, time-sync drift, OOM events, read-only filesystem remounts, and several security-platform-specific checks (audit log anomalies, authentication failures) — deliberately complementary to the SOC stack in [`docs/security-monitoring.md`](security-monitoring.md), not duplicating its file-integrity or account monitoring.
+
+## One formatter, every source
+
+Every branch — app webhooks, host-level alerts, and the two dedicated non-JSON endpoints — converges on one shared subflow before anything reaches Slack. This is the single point where message formatting is standardized, regardless of which of the dozen-plus sources triggered it.
+
+---
+Related stacks: [`stacks/platform/mosquitto/`](../stacks/platform/mosquitto/), [`stacks/platform/node-red/`](../stacks/platform/node-red/)

docs/observability.md

@@ -0,0 +1,71 @@
+# 📈 Observability
+
+> [!NOTE]
+> Every fleet host runs the same collection agent natively (not as a container), pushing logs and metrics outward rather than waiting to be scraped — a deliberate choice that keeps monitoring off the firewall's critical path.
+
+| | |
+|---|---|
+| **Agent** | Native systemd service on every host, not Docker |
+| **Logs** | journald → Loki |
+| **Metrics** | Push-based `remote_write`, not pull-based scraping |
+| **Coverage** | All fleet hosts, confirmed live |
+
+## Pipeline
+
+```mermaid
+flowchart LR
+    subgraph Sources["Per-host sources"]
+        J["journald<br/>persistent, 14-day retention"]
+        D["Docker container<br/>stdout/stderr"]
+        N["node metrics<br/>CPU/mem/disk/net"]
+        App["Native app metrics<br/>5 apps expose endpoints"]
+    end
+
+    Agent["Collection Agent<br/>native systemd service"]
+    J --> Agent
+    D -->|docker_hosts only| Agent
+    N --> Agent
+    App --> Agent
+
+    Agent -->|logs| Loki["Loki"]
+    Agent -->|remote_write, 30s| Prom["Prometheus"]
+```
+
+## Agent install: deliberate, not default
+
+> [!TIP]
+> The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).
+
+> [!WARNING]
+> **Honestly-tracked version drift**: most hosts run the current pinned agent version, but three legacy hosts remain one version behind, and the automation role's own default hasn't caught up to what's actually deployed on the majority. Documented as known drift, not silently ignored.
+
+## Logs: journald only, no double-handling
+
+The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume.
+
+```mermaid
+flowchart LR
+    A["journald<br/>system logs"] --> L1["Loki<br/>job=system"]
+    B["Docker API<br/>container logs<br/>survives recreation"] --> L2["Loki<br/>job=docker"]
+    C["DNS query log<br/>daily rotation"] --> L3["Loki<br/>job=dns · real event time"]
+```
+
+Every host's logs land in the same log store under its own host label, filterable in the dashboard by host or job. Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated. The DNS query log pipeline is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time.
+
+## Metrics: pushed, not pulled
+
+> [!IMPORTANT]
+> **Why push instead of pull**: Prometheus doesn't need network access to every host — only the monitoring host needs to accept inbound writes. This matters directly for hosts sitting behind the tighter firewall rules described in [`firewall-and-network.md`](firewall-and-network.md) — a pull model would mean punching a scrape-target hole through that segmentation for every single host.
+
+| App | What's exposed |
+|---|---|
+| Secrets platform | Dedicated read-only metrics token, separate from its normal agent token |
+| Orchestration engine | Built-in metrics endpoint |
+| Git server | Built-in metrics endpoint |
+| Workflow automation | Built-in metrics endpoint |
+| Identity provider | Built-in metrics endpoint, localhost-only |
+
+Five apps expose their own metrics endpoints directly, scraped by that host's own agent and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility.
+
+---
+Related stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)