public/docs/netbird-zero-trust.md

# 🕸️ Zero-Trust Mesh Overlay

> [!NOTE]
> A self-hosted **NetBird** WireGuard mesh gives every fleet host — and every device that needs to reach it — its own stable overlay identity, independent of where that device physically sits. Access is granted per-identity through mesh policy, not inferred from LAN position. The mesh is **deny-by-default**: nothing crosses it unless an explicit policy allows it, the same instinct behind every other boundary in [`docs/architecture-overview.md`](architecture-overview.md), applied to remote and roaming access specifically.

## Why a mesh, not a classic VPN

A traditional VPN concentrator was the obvious alternative, and it was rejected for the same reason a flat LAN got rejected everywhere else in this fleet: it's one more device sitting in a path, and reaching it means punching a hole to a single choke point rather than establishing an identity-scoped connection to exactly the host that's actually needed. A mesh instead gives every peer — fleet host or personal device — its own stable identity (`<host>.netbird.selfhosted`, a `100.91.0.0/16` overlay address) and lets WireGuard tunnels form directly, peer-to-peer, falling back to a relay only when direct connectivity isn't possible. Policy is written once, per identity or group, and travels with the peer regardless of which network it's physically on.

This is the actual shape of it — the control plane coordinates, but it's deliberately not where the traffic goes:

![The NetBird control plane sends thin coordination-only signals to every peer, while real data flows over direct peer-to-peer WireGuard tunnels that bypass the control plane entirely, with relay fallback only when a direct path isn't possible](../assets/netbird-zero-trust-mesh-topology.svg)

That's the actual difference from a VPN concentrator, drawn rather than just asserted: a concentrator puts itself in every data path by design; here the control plane's spokes only ever carry coordination traffic, and every thick line is two peers talking directly to each other.

## Built on a capability that didn't exist yet

The mesh's own control plane is the first host in this fleet ever provisioned through a genuinely new capability: Terraform gaining native LXC support (a second provider, `bpg/proxmox`, alongside the existing VM-only one — full story in [`docs/iac-pipeline.md`](iac-pipeline.md)). Every other LXC in this fleet either predates Terraform entirely or was hand-converted after the fact; this one went through the pipeline start to finish with no manual provisioning step anywhere. That's not a coincidence — a lightweight mesh control-plane container was exactly the kind of workload that made closing the LXC gap worth doing.

## Control plane

One `netbirdio/netbird-server` container runs Management, Signal, the embedded relay, and NetBird's own embedded IdP together — combined-server mode, not four separate services. A digest-pinned `netbirdio/dashboard` container serves the admin SPA. This is the real container list, not a claimed one:

```
$ docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
NAMES               IMAGE                                               STATUS
netbird-proxy       caddy:2-alpine                                      Up 38 hours
netbird-server      netbirdio/netbird-server:0.77.1                     Up 38 hours
netbird-dashboard   f3eb26c93ca9                                        Up 38 hours
```

Configuration is Vault-Agent-rendered, never committed — secrets live in Vault under a dedicated AppRole and policy scoped to this one host, the same pattern as every other secret in [`docs/secrets-and-pki.md`](secrets-and-pki.md). The identity provider is NetBird's own embedded one, not Authentik — Authentik is deliberately LAN-only in this fleet and publishing it wasn't in scope, so there's no OIDC migration path here; the first-login owner account is the permanent identity for this system specifically.

## The edge routing problem, and what actually broke

> [!WARNING]
> **Learned the hard way during bring-up, not designed in from the start.** NetBird's Management and Signal services need native gRPC over HTTP/2-cleartext (h2c) end-to-end — any downgrade to HTTP/1.1 anywhere in the chain kills the bidirectional stream with an `unexpected EOF`, and the client agent just wedges with no useful error. The edge firewall's own reverse proxy can carry h2c to a single plain-HTTP backend, but it can't perform NetBird's own content-type/path split between the management API, the relay, and the dashboard SPA — so a small in-stack proxy does exactly that split, one hop further in. A second, unrelated thing broke the same week: the public DNS record for this service has to stay **DNS-only**, not proxied through a CDN — a proxying CDN returns an outright 403 to the gRPC-speaking agent, no explanation given, and the fix was disabling that proxying entirely for this one record rather than trying to work around it.

The real path a request takes:

![A client's gRPC request passes through the edge firewall's HTTPS termination, into an in-stack proxy that splits traffic by content-type and path, then either to the management/signal server or the dashboard SPA depending on that split](../assets/netbird-zero-trust-edge-path.svg)

And this is the actual splitter doing that work — the real config, not a description of the logic:

```
$ cat Caddyfile
{
	admin off
	auto_https off
	servers {
		protocols h1 h2 h2c
	}
}

:33000 {
	# native gRPC + gRPC-web (any application/grpc* content-type) -> server over h2c
	@grpc header Content-Type application/grpc*
	handle @grpc {
		reverse_proxy h2c://netbird-server:80
	}

	# REST, OAuth, relay/signal WebSockets, and the gRPC service paths -> server
	@srv path /api/* /oauth2/* /relay* /ws-proxy/* /signalexchange* /management*
	handle @srv {
		reverse_proxy netbird-server:80
	}

	# everything else -> dashboard SPA
	handle {
		reverse_proxy netbird-dashboard:80
	}
}
```

## Fleet enrolment

A dedicated Ansible role installs the mesh agent and enrols a host, opt-in per host via the same tag-driven mechanism as every other role in [`docs/iac-pipeline.md`](iac-pipeline.md) — nothing joins the mesh by default just because it exists. Enrolment uses a reusable, class-scoped setup key fetched live from the secrets platform at play time, never stored in the repo, and read only on a host's first join. Because peer approval is required, a brand-new peer pauses for an operator to approve it in the dashboard before it gets any mesh access at all — there's no silent auto-join path.

All fleet hosts are enrolled today, plus two personal devices (a laptop with full mesh access, a home device with none by default — being *on* the mesh and being *allowed* anywhere on it are two separate, independently-granted things). Fleet servers run with local DNS resolution disabled on the mesh interface deliberately — they're not roaming, and the fleet's own internal DNS already handles them; only the one roaming device gets split-DNS behavior, resolving the fleet's internal zones over the mesh specifically because it's the one peer that actually needs to.

This is the real, live peer roster, not a claimed enrolment count — every fleet host plus both personal devices, each with its own overlay identity:

![Real NetBird dashboard: 13 connected peers, a mix of fleet hosts and personal devices, each showing its own group membership, agent version, and OS](../assets/netbird-zero-trust-peers.png)

## Zero-trust policy: deny-by-default, not just present

The mesh ships with a built-in allow-everything default policy — the first real security decision here was turning it off. With it disabled, no traffic crosses the mesh interface at all unless a specific policy grants it; LAN traffic is completely untouched by this, since the policy only governs the overlay interface, not the physical network.

**Groups** carry the actual access model:

| Group | Role |
|---|---|
| Admin | Full fleet access — the policy source for everything below |
| Servers | Every fleet host — coarse policy destination |
| Endpoints | Personal devices with no fleet access by default — on the mesh for connectivity, not for reach |
| Functional sub-groups | Hypervisors / infra / Docker hosts / storage — each server is in exactly one, no policy on them yet, there so future rules can be scoped precisely instead of broadly |

**Policies**, all sourced from the admin group and one-directional: SSH and hypervisor-management ports to servers (an explicit port list — the platform's own API rejects ranges, so nothing here is a lazy wildcard), ICMP for diagnostics, and DNS specifically to the infra sub-group to support split-DNS for the roaming device. A posture check gates admin-level access behind a minimum agent version and kernel version — an out-of-date or unpatched peer simply doesn't qualify for the policy regardless of which group it's in. Login expiration applies only to the human-authenticated peer; the fleet's own service-key peers don't expire on a timer, since there's no human session behind them to expire.

This is the real policy list, not a description of the intent — the built-in allow-all sitting disabled at the top, every real rule beneath it scoped to a specific source, direction, and an explicit port list rather than "allow the group and hope":

![Real NetBird Access Control page: 5 policies, the built-in Default allow-all shown disabled, and four explicit admin-sourced rules each scoped to a specific protocol and port list](../assets/netbird-zero-trust-policies.png)

This is the mesh actually enforcing that model, not a claimed state — a real enrolled peer's own live status:

```
$ netbird status --detail
Peers detail:
 [laptop]:
  NetBird IP: [overlay address]
  Status: Idle
  -- detail --
  Last WireGuard handshake: -
  ...
Events:
  [INFO] SYSTEM
    Message: Network map updated
    Time: 5 minutes, 7 seconds ago
```

The network map updating on its own, unprompted, on a peer that isn't even the control plane, is the mesh's policy engine actually pushing live state to every connected peer — not a dashboard claiming a policy exists.

## Rollout, phased and honestly incomplete

Not everything is finished, and that's stated plainly rather than rounded up:

| Phase | Scope | State |
|---|---|---|
| Control plane + dashboard, embedded identity | Stand up the mesh itself | Done |
| First peers | Laptop + one fleet host, verified on-LAN | Done |
| Fleet enrolment | Every fleet host via the dedicated role | Done |
| Roaming | Split-DNS (done) + full-LAN route + off-LAN verification | Partly done |
| Zero-trust policy | Groups, keys, deny-by-default, approval, posture, login expiry | Done |
| Enforcement | Per-host firewall rejects non-mesh admin access | Planned |

The honest gaps: off-LAN connectivity (STUN/relay through NAT) hasn't been verified end-to-end yet, the full internal-network route for agent-less devices is still open, and per-host firewall enforcement — actually *requiring* the mesh for admin access rather than just offering it as one more path — is a deliberate later phase, not an oversight. A phased rollout that's honest about which phases aren't done yet is worth more than a status page that rounds every in-progress item up to "complete."

---
Related: [`docs/iac-pipeline.md`](iac-pipeline.md), [`docs/architecture-overview.md`](architecture-overview.md), [`docs/firewall-and-network.md`](firewall-and-network.md), [`docs/secrets-and-pki.md`](secrets-and-pki.md), [`docs/dns-naming.md`](dns-naming.md)