docs: deepen observability, notifications, orchestration, registry, and dashy batch

Same treatment as the security batch - real tech names, 'why this vs
the alternative' reasoning, heavy cross-linking, ranked/specific
tradeoffs instead of generic hedges.

docs/observability.md

@@ -1,15 +1,19 @@
 # 📈 Observability
 
 > [!NOTE]
-> Every fleet host runs the same collection agent natively (not as a container), pushing logs and metrics outward rather than waiting to be scraped — a deliberate choice that keeps monitoring off the firewall's critical path.
+> Grafana Alloy runs natively on every fleet host, shipping logs to Loki and metrics to Prometheus via `remote_write` — pushed outward rather than scraped, a choice made specifically because of the firewall model in [`docs/firewall-and-network.md`](firewall-and-network.md), not despite it.
 
 | | |
 |---|---|
-| **Agent** | Native systemd service on every host, not Docker |
+| **Agent** | Grafana Alloy, native systemd service on every host, not Docker |
 | **Logs** | journald → Loki |
 | **Metrics** | Push-based `remote_write`, not pull-based scraping |
 | **Coverage** | All fleet hosts, confirmed live |
 
+## Why push instead of pull — the actual reasoning
+
+A pull-based model (Prometheus scraping every host directly) is the more common default, and it was rejected for a specific reason tied directly to [`docs/firewall-and-network.md`](firewall-and-network.md): scraping means the monitoring host needs inbound network access to every single host it monitors, which means punching a hole through the zero-blanket-allow firewall model for every host, every metrics port. A push model inverts that — only the monitoring host needs to accept inbound writes, and every other host only ever *initiates* outbound connections, which fits the existing "explicit outbound rule, default-deny inbound" posture instead of fighting it. The tradeoff is real: push-based metrics are slightly harder to reason about for ad-hoc debugging (no simple `curl` against a target to see what it'd report), but it was the right call given the network model already in place.
+
 ## Pipeline
 
 ```mermaid
@@ -21,7 +25,7 @@ flowchart LR
         App["Native app metrics<br/>5 apps expose endpoints"]
     end
 
-    Agent["Collection Agent<br/>native systemd service"]
+    Agent["Grafana Alloy<br/>native systemd service"]
     J --> Agent
     D -->|docker_hosts only| Agent
     N --> Agent
@@ -34,11 +38,11 @@ flowchart LR
 ## Agent install: deliberate, not default
 
 > [!TIP]
-> The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).
+> The agent binary is downloaded directly from upstream releases and run under a hand-written systemd unit — not installed from a package repo or the vendor's own packaged service file. That's a real maintenance cost (manual version bumps instead of `apt upgrade` handling it), accepted because it keeps the exact version pinned and reproducible across the fleet through the same tag-driven Ansible model described in [`docs/iac-pipeline.md`](iac-pipeline.md), rather than whatever version a package repo happens to serve on a given day. Before install, a preflight check fails the run if the host doesn't run systemd, or if the root filesystem has too little free space for journald's own headroom, and configures/waits for time-sync confirmation (skipped on container-based hosts, where the clock can't be locally adjusted anyway).
 
 ## Logs: journald only, no double-handling
 
-The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume.
+The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet.
 
 ```mermaid
 flowchart LR
@@ -47,22 +51,19 @@ flowchart LR
     C["DNS query log<br/>daily rotation"] --> L3["Loki<br/>job=dns · real event time"]
 ```
 
-Every host's logs land in the same log store under its own host label, filterable in the dashboard by host or job. Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated. The DNS query log pipeline is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time.
+Every host's logs land in the same Loki instance under its own host label, filterable in Grafana by host or job — the same "one source of truth, not N inconsistent ones" instinct behind the DNS model in [`docs/dns-naming.md`](dns-naming.md). Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated during the update flows in [`docs/registry-and-build.md`](registry-and-build.md) — tailing a file path would silently stop working the moment that container got rebuilt. The DNS query log pipeline (feeding into [`docs/dns-naming.md`](dns-naming.md)'s enforcement story) is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time — the same UTC-vs-wall-clock discipline that matters when querying Wazuh's alert data in [`docs/security-monitoring.md`](security-monitoring.md).
 
 ## Metrics: pushed, not pulled
 
-> [!IMPORTANT]
-> **Why push instead of pull**: Prometheus doesn't need network access to every host — only the monitoring host needs to accept inbound writes. This matters directly for hosts sitting behind the tighter firewall rules described in [`firewall-and-network.md`](firewall-and-network.md) — a pull model would mean punching a scrape-target hole through that segmentation for every single host.
-
 | App | What's exposed |
 |---|---|
-| Secrets platform | Dedicated read-only metrics token, separate from its normal agent token |
-| Orchestration engine | Built-in metrics endpoint |
-| Git server | Built-in metrics endpoint |
-| Workflow automation | Built-in metrics endpoint |
-| Identity provider | Built-in metrics endpoint, localhost-only |
+| Vault | Dedicated read-only metrics token, separate from its normal agent token — see [`docs/secrets-and-pki.md`](secrets-and-pki.md) |
+| Kestra | Built-in metrics endpoint |
+| Gitea | Built-in metrics endpoint |
+| n8n | Built-in metrics endpoint |
+| Authentik | Built-in metrics endpoint, localhost-only |
 
-Five apps expose their own metrics endpoints directly, scraped by that host's own agent and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility.
+Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open.
 
 ---
-Related stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)
+Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)