observability.md: full 7-step audit, first pass this session
1. Live-verify: journald persistent-storage config confirmed on two hosts (1G vs 500M cap, matches the doc's claim exactly), Alloy active confirmed across all 11 fleet hosts. 2. Fix KB: Alloy Agent Model was stale - version had moved from v1.18.0 to v1.18.1 on all 8 'newer' hosts since last checked, role's own default still unreconciled at 1.17.1. Fixed and confirmed via re-fetch. 3. Pulled the full Observability Pipeline tree: Alloy Agent Model, Log Flow, Metrics Flow. 4. Replaced both mermaid diagrams (broke this repo's Nextcloud-SVG convention) with real diagrams built through the proper round-trip. 5. Evidence: real journald config dumps, plus 3 real Grafana screenshots (host metrics dashboard, detailed Loki log stream showing job=technitium-dns/docker/system, and the fleet-wide log volume dashboard) - one screenshot round caught and fixed an unredacted internal IP before being used. 6/7. Compelling-proof test passes; sanitize-check clean.
This commit is contained in:
6 files changed
+37
-25
No files matched your search
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Do not edit this file with editors other than draw.io -->
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="872px" height="273px" viewBox="0 0 872 273" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Log Fanout" id="obs-logfanout-1"> <mxGraphModel dx="1414" dy="791" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1000" pageHeight="320" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="a1" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="journald&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;system logs&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="200" x="40" y="30" as="geometry" /> </mxCell> <mxCell id="b1" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Loki&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;job=system&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="200" x="700" y="30" as="geometry" /> </mxCell> <mxCell id="e1" edge="1" parent="1" source="a1" style="html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;" target="b1"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="a2" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Docker API&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;container logs, survives recreation&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="200" x="40" y="130" as="geometry" /> </mxCell> <mxCell id="b2" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Loki&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;job=docker&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="200" x="700" y="130" as="geometry" /> </mxCell> <mxCell id="e2" edge="1" parent="1" source="a2" style="html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;" target="b2"> <mxGeometry relative="1" as="geometry" /> </mxCell> <mxCell id="a3" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="DNS query log&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;daily rotation&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="200" x="40" y="230" as="geometry" /> </mxCell> <mxCell id="b3" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Loki&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;job=dns, real event time&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="200" x="700" y="230" as="geometry" /> </mxCell> <mxCell id="e3" edge="1" parent="1" source="a3" style="html=1;endArrow=block;strokeColor=#6c8ebf;strokeWidth=2;rounded=1;" target="b3Line truncated
|
||||
|
After Width: | Height: | Size: 135 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 366 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 182 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 234 KiB |
@@ -0,0 +1,4 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!-- Do not edit this file with editors other than draw.io -->
|
||||
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
|
||||
<svg xmlns="http://www.w3.org/2000/svg" style="background: transparent; background-color: transparent; color-scheme: light dark;" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="838px" height="324px" viewBox="0 0 838 324" content="<mxfile host="embed.diagrams.net" scale="1" border="0"> <diagram name="Pipeline" id="obs-pipeline-1"> <mxGraphModel dx="1178" dy="659" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1000" pageHeight="420" math="0" shadow="0"> <root> <mxCell id="0" /> <mxCell id="1" parent="0" /> <mxCell id="srcgroup" parent="1" style="rounded=1;whiteSpace=wrap;html=1;startSize=30;swimlane;fillColor=none;strokeColor=#666666;dashed=1;fontSize=13;fontStyle=1;fontColor=#333333;align=center;" value="PER-HOST SOURCES" vertex="1"> <mxGeometry height="320" width="220" x="30" y="40" as="geometry" /> </mxCell> <mxCell id="s1" parent="srcgroup" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="journald&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;persistent, 14-day retention&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="180" x="20" y="30" as="geometry" /> </mxCell> <mxCell id="s2" parent="srcgroup" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Docker container&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;stdout/stderr, docker_hosts only&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="180" x="20" y="105" as="geometry" /> </mxCell> <mxCell id="s3" parent="srcgroup" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Node metrics&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;CPU / mem / disk / net&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="180" x="20" y="180" as="geometry" /> </mxCell> <mxCell id="s4" parent="srcgroup" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=12;fontStyle=1;shadow=1;" value="Native app metrics&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;5 apps expose endpoints&lt;/font&gt;" vertex="1"> <mxGeometry height="60" width="180" x="20" y="255" as="geometry" /> </mxCell> <mxCell id="agent" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#fff2cc;strokeColor=#d6b656;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Grafana Alloy&lt;br&gt;&lt;font style=&quot;font-size: 10px;&quot;&gt;native systemd service&lt;/font&gt;" vertex="1"> <mxGeometry height="80" width="200" x="360" y="160" as="geometry" /> </mxCell> <mxCell id="loki" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#dae8fc;strokeColor=#6c8ebf;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Loki" vertex="1"> <mxGeometry height="60" width="160" x="700" y="70" as="geometry" /> </mxCell> <mxCell id="prom" parent="1" style="rounded=1;whiteSpace=wrap;html=1;fillColor=#d5e8d4;strokeColor=#82b366;fontColor=#333333;fontSize=13;fontStyle=1;shadow=1;" value="Prometheus" vertex="1"> <mxGeometry height="60" width="160" x="700" y="270" as="geometry" /> Line truncated
|
||||
|
After Width: | Height: | Size: 211 KiB |
+29
-25
@@ -16,24 +16,7 @@ A pull-based model (Prometheus scraping every host directly) is the more common
|
||||
|
||||
## Pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Sources["Per-host sources"]
|
||||
J["journald<br/>persistent, 14-day retention"]
|
||||
D["Docker container<br/>stdout/stderr"]
|
||||
N["node metrics<br/>CPU/mem/disk/net"]
|
||||
App["Native app metrics<br/>5 apps expose endpoints"]
|
||||
end
|
||||
|
||||
Agent["Grafana Alloy<br/>native systemd service"]
|
||||
J --> Agent
|
||||
D -->|docker_hosts only| Agent
|
||||
N --> Agent
|
||||
App --> Agent
|
||||
|
||||
Agent -->|logs| Loki["Loki"]
|
||||
Agent -->|remote_write, 30s| Prom["Prometheus"]
|
||||
```
|
||||
|
||||
|
||||
## Agent install: deliberate, not default
|
||||
|
||||
@@ -42,14 +25,33 @@ flowchart LR
|
||||
|
||||
## Logs: journald only, no double-handling
|
||||
|
||||
The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet.
|
||||
The system's built-in syslog daemon is explicitly disabled once journald takes over — avoids the same log stream getting handled twice, which would double both storage cost and noise in every downstream query. journald itself is switched from the default in-memory ring buffer to persistent on-disk storage, sized larger on the one host with much higher container-log volume than the rest of the fleet. This is the real config on two different hosts, not a described default:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["journald<br/>system logs"] --> L1["Loki<br/>job=system"]
|
||||
B["Docker API<br/>container logs<br/>survives recreation"] --> L2["Loki<br/>job=docker"]
|
||||
C["DNS query log<br/>daily rotation"] --> L3["Loki<br/>job=dns · real event time"]
|
||||
```
|
||||
$ cat /etc/systemd/journald.conf.d/*.conf # the higher-volume host
|
||||
[Journal]
|
||||
Storage=persistent
|
||||
SystemMaxUse=1G
|
||||
SystemKeepFree=1G
|
||||
MaxRetentionSec=14day
|
||||
|
||||
$ cat /etc/systemd/journald.conf.d/*.conf # every other host
|
||||
[Journal]
|
||||
Storage=persistent
|
||||
SystemMaxUse=500M
|
||||
SystemKeepFree=1G
|
||||
MaxRetentionSec=14day
|
||||
```
|
||||
|
||||
|
||||
|
||||
This is real log data flowing through that pipeline, not a mockup — `job=technitium-dns` entries showing real DNS query/answer content, `job=docker` entries from real containers, and `job=system` from journald itself, all landing in the same Loki instance:
|
||||
|
||||
|
||||
|
||||
And the aggregate view across the whole fleet — real log volume, not an estimate:
|
||||
|
||||
|
||||
|
||||
Every host's logs land in the same Loki instance under its own host label, filterable in Grafana by host or job — the same "one source of truth, not N inconsistent ones" instinct behind the DNS model in [`docs/dns-naming.md`](dns-naming.md). Docker logs are scraped via the container runtime's own API (not by tailing log files on disk) specifically so log collection survives a container being recreated during the update flows in [`docs/registry-and-build.md`](registry-and-build.md) — tailing a file path would silently stop working the moment that container got rebuilt. The DNS query log pipeline (feeding into [`docs/dns-naming.md`](dns-naming.md)'s enforcement story) is kept deliberately separate from general system/container error tracking, with its timestamps extracted from the log line itself rather than ingestion time — the same UTC-vs-wall-clock discipline that matters when querying Wazuh's alert data in [`docs/security-monitoring.md`](security-monitoring.md).
|
||||
|
||||
@@ -63,7 +65,9 @@ Every host's logs land in the same Loki instance under its own host label, filte
|
||||
| n8n | Built-in metrics endpoint |
|
||||
| Authentik | Built-in metrics endpoint, localhost-only |
|
||||
|
||||
Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open.
|
||||
Five apps expose their own Prometheus-format metrics endpoints directly, scraped by that host's own Alloy instance and pushed through the same path as host-level metrics — no separate pipeline needed for app-level visibility, and no separate firewall exception needed either, since it rides the same outbound `remote_write` connection every host already has open. This is that same push path landing in Grafana, for a real host's own metrics — not one of the 5 app endpoints specifically, but proof the pipeline actually delivers real numbers, not placeholder gauges:
|
||||
|
||||
|
||||
|
||||
---
|
||||
Related: [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/dns-naming.md`](dns-naming.md), [`docs/security-monitoring.md`](security-monitoring.md), [`docs/dashy-soc-board.md`](dashy-soc-board.md) · Stacks: [`stacks/platform/grafana-stack/`](../stacks/platform/grafana-stack/)
|
||||
Reference in new issue
Block a user