gns3-server/docs/features/docker-kernel-datapath.md

This documentation is AI-generated with reference to actual code and verified against real-kernel testing. AI can make mistakes — please verify against the source code when in doubt.

Docker Kernel Datapath (veth + per-link Linux bridge)

Overview

Docker node links historically flowed through a uBridge userspace relay: each adapter was a TAP whose file descriptor uBridge pumped between the container and UDP tunnels. This implementation replaces that model with kernel-native forwarding: every adapter is born as a veth pair, and each link is a dedicated Linux kernel bridge enslaving the two endpoints' veth host ends. Frames never leave the kernel.

	Old (relay)	New (kernel datapath)
Adapter interface	TAP moved into the container netns	veth pair (guest end in container, host end in root ns)
Link	uBridge bridge, TAP fd ↔ UDP, userspace copy	per-link kernel bridge gns3{link_id[:11]}, zero-copy
Relay latency	~0.3 ms RTT	~0.05 ms RTT (measured)
Link attach at runtime	yes (relay attaches the pre-existing TAP)	yes (brctl addif / add_nio_ethernet — interface untouched)
Packet filters	uBridge userspace filters	tc netem on the veth host end (delay / packet_loss / corrupt / duplicate + the netem extensions rate, reorder, gemodel, seed, limit, jitter distributions, loss correlation); bpf as cls_bpf match-drop classifiers; frequency_drop stays relay-only (eBPF classifier pending)
Capture / markers	relay bridge	uBridge AF_PACKET modules on the veth host end

The work landed in seven stages on branch stack feat/docker-kernel-datapath → feat/docker-kernel-capture → feat/docker-kernel-markers → feat/docker-veth-everywhere → feat/docker-kernel-filters → feat/docker-kernel-bpf-drop → feat/docker-kernel-netem-ext:

1. Kernel datapath — NIOBridge NIO type, per-link bridges, carrier-based suspend, crash-safe reconciliation.
2. Capture — capture start_kernel (AF_PACKET) on the veth host end.
3. Markers — marker add_kernel (AF_PACKET + BPF) with the same signals, pcaps and fine-grained REST operations as the relay mark filter.
4. veth-everywhere — the TAP path deleted; every adapter is a veth and the datapath is a runtime decision (kernel or relay) that never touches a running container's interfaces.
5. Filters — impairment filters (delay, packet_loss, corrupt) become one tc netem qdisc per veth host end, pushed through uBridge's netlink tc module. This also removes the relay delay filter's nanosleep bottleneck (upstream #2827) from the kernel path.
6. bpf match-drop — the bpf filter (any line matching drops) runs as cls_bpf classifiers on the veth host end's clsact egress (tc bpf_drop: uBridge pcap-compiles the line to classic BPF, the kernel migrates it to eBPF internally — no CAP_BPF needed). Requires a uBridge reporting cbpf=1 via tc capabilities; older builds get a clear upgrade error.
7. netem extensions — the netem-extension surface delivered in uBridge feature/tc-netem-ext is exposed as GNS3 filter types: rate, reorder, gemodel, duplicate, seed, limit, plus the delay distribution and loss/dup correlation parameters. Kernel-datapath links only (the relay has no equivalent); capability-gated per uBridge process. Reconcile became reset-before-set: the kernel's netem replace merges optional attributes, so removing a parameter needs the explicit reset.

Adapter interface types

Which interface a Docker node's adapters use is decided by the node class (Docker._select_node_class, from console_type + GNS3_* environment markers):

Node class	Selected by	Adapter interface	Kernel links
DockerVM	default	veth pair	yes
VendorDockerVM (non-unix-socket)	console_type=docker_exec, GNS3_SKIP_INIT, … (XRd, SR Linux, …)	veth pair (same path via super())	yes
IOLDockerVM	GNS3_IOL_RUNNER=1	AF_UNIX datagram socket pairs (sNN/cNN) — the container netns is unused	rejected (no host-side interface)
VendorDockerVM (unix-socket)	GNS3_UNIX_SOCKET_NIO=1	same socket contract, generic capability for vendor NOS images	rejected

Unix-socket containers are detected both in the controller (_is_unix_socket_docker, link eligibility) and on the compute side (NIO rejection) — a missed detection surfaces as a clearer-late error.

Architecture

Kernel link between two containers on the same compute:

 container A netns                 root namespace                  container B netns
 ┌────────────┐   veth pair   ┌───────────────────┐   veth pair   ┌────────────┐
 │ eth0 ◄─────┼───────────────┼─► gv{id}e0p0 ──────┼──►  gns3{lid} │──────────► │ eth0
 └────────────┘  gc{id}e0p0   │  (bridge port)     │  (bridge port)│            └────────────┘
                               │  per-link kernel  │   gv{id}e0p0
                               │  bridge = "cable" │
                               └───────────────────┘

Relay link on the same unified veth (fallback for filtered links or when enable_kernel_datapath is off — e.g. to reach another compute):

 ┌────────────┐              ┌────────────────────────────┐              ┌────────────┐
 │ eth0       │    veth      │ uBridge relay bridge       │    UDP       │ peer node  │
 │ (gc end)   │◄────────────►│ add_nio_ethernet (AF_      │◄────────────►│ (any type, │
 └────────────┘  gv host end │ PACKET) ↔ add_nio_udp      │  tunnel     │  any host) │
                             └────────────────────────────┘              └────────────┘

Both datapaths anchor on the same veth host end; switching between them (link delete + recreate) never touches the container-side interface.

Datapath selection

UDPLink._kernel_datapath_eligible decides in the controller at NIO prepare time; the compute side only reacts to the NIO type:

• both endpoints Docker (any class except unix-socket containers), same compute
• no active packet filters (filters live in the relay; kernelization via tc netem is planned)
• Server.enable_kernel_datapath enabled (default)

There is no stopped-node requirement. Since every adapter is a veth, links attach to running containers: brctl addif (kernel) or bridge add_nio_ethernet (relay) are runtime-safe operations on the host end. On project (re)open every link re-runs _prepare, so relay links whose endpoints became eligible are upgraded to the kernel datapath automatically.

Adapter lifecycle

• Birth (container start) — _create_veth: stale sweep of any leftover pair (crash residue), docker create_veth, host end admin-down, MAC from the adapter base, docker move_to_ns of the guest end (renamed eth{N}, or the GNS3_INTERFACE_NAMES mapping). Names are deterministic: gv|gc{node_id[:8]}e{adapter}p{port} (≤ 15 chars). Unconnected adapters are born too (carrier off) so the interface is visible inside the container.
• Life (running) — the veth is never created/deleted/moved again. Link create/delete/switch, suspend, capture and markers only change what is attached to the host end.
• Death (container stop) — _remove_kernel_veths deletes the host ends explicitly: unlike a TAP (which died with the container netns), a veth host end outlives the container. Deleting either end destroys the pair.

Link operations

Operation	Kernel-datapath action
Create / attach	brctl create (EEXIST tolerated via brctl show verify) + link set up + brctl addif both ends (concurrent, race-tolerant) + carrier up + markers + tc netem set (filters)
Delete / detach	marker teardown → tc reset (netem teardown — the veth survives the link) → carrier off → brctl delif → brctl delete (last endpoint wins; EBUSY/ENOENT suppressed). The veth survives — unlike a relay bridge, whose death dropped its filters, an orphaned AF_PACKET marker would keep sniffing, hence the explicit teardown
Reset (POST /links/{id}/reset)	delete + create (re-evaluates eligibility)
Suspend (PUT /links/{id} {"suspend": true})	veth host end admin-state down both ends — 100 % loss, no synthetic filter needed; resume restores (netem qdisc survives the carrier flap)

Capture

Kernel links capture via uBridge's AF_PACKET module bound to the veth host end: capture start_kernel <if> "<pcap>" [dlt] / capture stop_kernel. Single capture per uBridge process (second concurrent → EALREADY). Start/stop key on the NIO type, not on veth presence — a relay NIO riding a veth captures at its relay bridge.

Markers

Markers ride the NIO like on the relay datapath, routed by capture_node_id; the anchor is the veth host end instead of a relay bridge (_ubridge_apply_markers(anchor, nio) is datapath-agnostic; DockerVM overrides the add/delete/enable primitives to translate the commands):

marker add_kernel    <name> <if> "<bpf>" [tag <id>] [link <id>] [dir <tx|rx>] [linktype <name>] [pcap "<path>"]
marker delete_kernel <if> <name>                    # idempotent
marker enable_kernel <if> <name> <on|off>           # off = installed but silent

• Per-deployment convention: only the capture node's end runs add_kernel (symmetric with capture) — signals form clean send/return pairs (PACKET_OUTGOING on the host end = container receiving = rx).
• Same BPF compilation, MARK UDP signals, pcap files and fine-grained REST toggle/rebuild/delete endpoints as the relay mark filter.
• Node restart reinstalls markers from the NIO; link deletion tears them down.

Packet filters

Impairment filters run in the kernel as tc netem qdiscs on the veth host ends (uBridge tc netem set, raw netlink — no tc binary needed):

GNS3 filter	kernel implementation	Notes
delay [ms, jitter, dist]	netem delay X jitter Y [distribution D]	jitter 0 omitted; dist ∈ uniform/normal/pareto/paretonormal (kernel-only parameter)
packet_loss [%, correl]	netem loss P [correl C]	per direction (see below); correlation is a kernel-only parameter
corrupt [%]	netem corrupt P
duplicate [%, correl]	netem dup P [correl C]	kernel-only type
rate ["512kbit"]	netem rate B	tc-style integer+unit (bit/kbit/mbit/gbit/bps/kbps/mbps, ≤100gbit); kernel-only type
reorder [%, correl, gap]	netem reorder P [correl C] [gap G]	requires delay; kernel-only type
gemodel [p, r, 1-h]	netem loss gemodel P R H	Gilbert-Elliot bursty loss; mutually exclusive with packet_loss; kernel-only type
seed [u32]	netem seed N	reproducible random draws; kernel-only type
limit [pkts]	netem limit N	queue depth above the 1000 default (rate+delay BDP); kernel-only type
bpf (one line per expression, OR)	tc bpf_drop add <if> <prio> "<expr>" — cls_bpf on clsact egress + gact drop	needs uBridge cbpf=1 (tc capabilities, probed once per uBridge process); a line that fails to compile on the compute is skipped with a warning, mirroring the relay
frequency_drop	— not yet (eBPF stateful classifier spec'd: ubridge-kernel-impairment-spec, part B pending)	409 on kernel links; relay fallback

Semantics:

• Both endpoints receive the filter dict and attach one qdisc to their veth host end. The qdisc's egress covers traffic entering that container, so every direction of the link is impaired exactly once — the same net effect as the relay, where both directions cross the single filtered bridge. A delay 100 link measures ≈200 ms RTT; packet_loss 30 measures ≈51 % round-trip (1 − 0.7²); rate 512kbit adds ≈2× the per-packet serialization time to the RTT.
• Reconcile = reset + full re-apply. The kernel's netem replace MERGES optional attributes (rate, correlation, reorder, corrupt, gemodel, distribution: an absent attr keeps its previous value), so re-applying a filter set with a parameter removed would silently keep the old value. Every apply therefore starts with tc reset (also drops clsact and its bpf_drop filters — re-added right after in the same flow) followed by one netem set built from the current filters; no per-parameter diffing. An empty filter set is the reset alone (ENOENT-tolerated).
• Extension gating. The netem-extension keywords (rate, reorder, gemodel, dist, seed, limit, and the correl suffixes) probe tc capabilities once per uBridge process and require the tokens; a plain delay/loss/corrupt/dup filter never probes — an old uBridge serves the original surface untouched.
• kernel-only vs relay-only. The extension types have no uBridge relay equivalent: available_filters hides them on relay links, setting one on a created relay link returns 409, and a project loaded with such filters on a link that cannot be kernel-wired drops them with a warning (invalid filters are dropped the same way at load). Symmetrically, frequency_drop stays relay-only.
• Validation mirrors the tc grammar: reorder requires delay, gemodel and packet_loss are mutually exclusive (both map to the netem loss keyword), distribution requires jitter > 0, rate must be integer + unit ≤ 100gbit.
• Node restart restores the qdisc from the NIO (like capture and markers); link deletion detaches it explicitly — the veth survives the link and an orphaned qdisc would keep impairing the next one.
• Suspend keeps the qdisc (carrier-driven loss); the first packet after resume takes one extra delay interval (known netem idle-baseline behaviour) and steady state is exact.
• bpf reconcile = flush + re-add (tc bpf_drop flush, then one add per line at priorities 10, 11, …). tc reset (link delete / filter clear) is the full restore in uBridge: filters → clsact → root qdisc, idempotent.
• Observability caveat: clsact drops happen before the AF_PACKET tap points — markers/captures on the same veth do not observe cls_bpf-dropped frames (on the relay, filter-vs-mark ordering determined visibility instead). Also note tc qdisc show cannot print a distribution's name (the kernel stores only the sampled table).

uBridge command surface

No uBridge changes were needed beyond the marker module's *_kernel commands (the AF_PACKET capture module was pre-existing):

docker create_veth / delete_veth / move_to_ns / set_mac_addr
brctl create / delete / addif / delif / show
link set <if> up|down
capture start_kernel / stop_kernel
marker add_kernel / delete_kernel / enable_kernel
tc netem set <if> [delay <ms>] [jitter <ms>] [loss <%> [correl <%>] | loss gemodel <p> <r> <1-h>]
                                  [dup <%> [correl <%>]] [corrupt <%>] [reorder <%> [correl <%>] [gap <n>]]
                                  [rate <bw>] [limit <pkts>] [distribution uniform|normal|pareto|paretonormal] [seed <u32>]
tc bpf_drop add <if> <prio 10-99> "<expr>" / tc bpf_drop flush <if>
tc reset <if>                     # full restore: filters -> clsact -> root qdisc, idempotent
tc capabilities                   # "netem=<kw,...>;ebpf=0|1;cbpf=0|1", probed once per process
bridge add_nio_ethernet / add_nio_udp / start / stop / start_capture / stop_capture

Configuration

[Server]
# Wire eligible Docker-to-Docker links through kernel veth/bridge interfaces
enable_kernel_datapath = True

Intended to grow into the global datapath switch as QEMU / IOU / Dynamips migrate to kernel bridges (QEMU: tap enslaved per-link bridge, zero relay).

Verification

End-to-end on a five-container FRR topology (mixed runtime-drawn and reloaded links): 8/8 links on the kernel datapath, ping RTT ≈ 0.05 ms, FDB learning, suspend = 100 % loss / resume restores, project reopen upgrades relay links, runtime link creation on running containers, marker pcaps exact (tx/rx pairs), capture freeze on stop, server restart reconciliation.

Filters (two-container kernel link, 27/27 checks): baseline 0.06 ms → delay 100 = 200.2 ms RTT (2× per direction) → delay 50 = 100.2 ms (atomic replace) → clear = baseline (qdisc detached); packet_loss 30 = 48 % round-trip loss (expected 51 % = 1 − 0.7²); frequency_drop = 409; suspend with delay active = 100 % loss, resume keeps the qdisc (200.1 ms); node restart restores it (200.1 ms); link delete detaches it (/usr/sbin/tc qdisc show — no netem left); re-created link has no residual impairment.

bpf match-drop (same setup, 22/22 checks): bpf "icmp" = 100 % loss on both ends' clsact; size-discriminating expression (greater 150) = small pings pass / 300-byte pings dropped (real byte-level BPF match); two-line OR; coexistence with netem (delay 50 → small pings 100.2 ms while big dropped); clear removes clsact entirely; suspend/resume keeps the drops; link delete + recreate leaves no residual filters; frequency_drop still 409; available_filters shows bpf and hides frequency_drop. Unit tests: tests/compute/docker/test_docker_kernel_datapath.py, tests/controller/test_kernel_datapath_link.py.

netem extensions (same setup, 24/24 checks): rate 512kbit on both veths, 1400-byte pings measure 45.5 ms RTT = baseline + 2× the 22.2 ms serialization (exact); delay 100 20 paretonormal applies delay+jitter and clears the previous rate (leak detector for the kernel's merge-on-replace); reorder 25 gap 5 visible in the qdisc dump and traffic passes; gemodel 100/0/30 = 50 % bursty round-trip loss with the previous reorder cleared; duplicate 50 link alive; packet_loss 30 correl 50 shows loss 30% 50%; seed 42 + limit 5000 both visible in the dump, RTT exactly 2×50 ms; suspend/resume and node restart keep the filters; a docker↔ethernet-switch relay link rejects rate with 409 and its available_filters hides the kernel-only types while the kernel link shows them all (minus frequency_drop); reorder without delay and gemodel+packet_loss are 409s; teardown leaves no netem on either veth. Also verified live: a relay attach (bridge add_nio_ethernet) on the admin-down veth host end fails in libpcap — the relay path now brings the interface up before attaching. Unit tests: tests/utils/test_packet_filter_validation.py (P6a class), tests/compute/docker/test_docker_kernel_datapath.py, tests/controller/test_kernel_datapath_link.py.