gns3-server/docs/features/qemu-kernel-datapath.md

QEMU kernel datapath

QEMU adapters anchor on a persistent TAP, and links between two QEMU nodes — or a QEMU node and a Docker container — on the same compute are wired in the kernel (per-link Linux bridge) instead of the uBridge UDP relay. The compute side is the same machinery Docker already uses: KernelDatapathMixin (gns3server/compute/kernel_datapath.py) takes an anchor interface and does not care whether it is a veth host end or a TAP.

Architecture

Kernel link between two QEMU nodes on one compute:

 QEMU A (fd)                root namespace                 QEMU B (fd)
 ┌──────────┐   -netdev   ┌──────────────────┐  -netdev   ┌──────────┐
 │ e1000    │◄───tap──────┤  gq{idA}e0p0 ────┼──► gns3{link_id_prefix}
 └──────────┘             │  (bridge port)   │    (bridge port)  │
                          │  per-link kernel │   gq{idB}e0p0     │
                          │  bridge = "cable"│                   │
                          └──────────────────┘

Relay link on the same anchor (cross-compute, mixed node types, or enable_kernel_datapath off):

 ┌──────────┐   tap    ┌────────────────────────────┐   UDP   ┌──────────┐
 │ QEMU A   │◄────────►│ uBridge relay bridge       │◄───────►│ peer     │
 │ (fd)     │ gq…e0p0  │ add_nio_ethernet ↔ add_    │ tunnel  │ (any)    │
 └──────────┘          │ nio_udp                    │         └──────────┘
                       └────────────────────────────┘

The relay attaches the TAP with bridge add_nio_ethernet (AF_PACKET, which does not take the device over) — never bridge add_nio_tap, which would claim the fd QEMU itself holds.

Why a TAP works like a veth host end

Event	Docker anchor (veth host end)	QEMU anchor (TAP)
Frames the node sends	arrive on the veth as PACKET_HOST	written to the tap fd, arrive as RX (PACKET_HOST)
Frames the peer sends	injected by uBridge, seen as PACKET_OUTGOING	injected by uBridge, seen as PACKET_OUTGOING
tc egress qdisc impairs	traffic entering the container	traffic entering the VM
Capture/marker anchor	the interface	the same interface

So every filter, capture and marker primitive carries over verbatim, and delay 100 on both anchors still measures ≈200 ms round trip while each direction is impaired exactly once.

Datapath selection

UDPLink._kernel_datapath_eligible needs both endpoints on one compute and both able to anchor:

• Docker always can (adapters are born as veth pairs);
• QEMU can when that compute's uBridge reports the tap module — asked per compute through /capabilities (ubridge_tap, probed by creating and deleting one throwaway TAP, cached by binary identity), because an old uBridge leaves QEMU on the legacy socket-netdev datapath and cannot anchor a kernel link. An unknown/failed probe keeps the link on the relay.

Because the -netdev type is fixed when QEMU is launched, this is all-or-nothing per VM: on the TAP datapath every link can attach to (or detach from) a running VM, including switching a link between the two datapaths (delete + re-create, or a project reopen).

Adapter lifecycle

• Birth (node start, before QEMU is launched) — _prepare_tap_datapath probes the tap module, then _create_taps sweeps any leftover (a persistent TAP outlives a crash) and creates one TAP per adapter: tap create gq{node_id[:8]}e{adapter}p{port} → tap set_owner <uid> (so the unprivileged QEMU process can open it) → link set <tap> down (carrier off until a link attaches). QEMU then opens it as -netdev tap,id=gns3-{adapter},ifname=<tap>,script=no,downscript=no.
• Life (running) — the TAP is never created or destroyed again. Link create/delete/switch, suspend, capture and markers only change what is attached to it.
• Death (node stop) — _remove_taps un-persists the TAPs (tap delete; QEMU's open fd keeps each device alive until it exits) and deletes the per-link kernel bridges the node still holds. Deleting a node never runs the link-teardown path, so those bridges would otherwise stay behind as empty orphans; a restart rebuilds them from the NIO.

Link operations

Operation	Kernel-datapath action
Create / attach	brctl create (EEXIST tolerated via brctl show) + link set <bridge> up + brctl addif <bridge> <tap> + carrier up + markers + tc netem set (filters)
Delete / detach	marker teardown → tc reset <tap> → carrier off → brctl delif → brctl delete (last endpoint wins; EBUSY/ENOENT suppressed). The TAP survives — the adapter keeps it for the next link
Reset (POST /links/{id}/reset)	delete + create (re-evaluates eligibility)
Suspend (PUT /links/{id} {"suspend": true})	tap admin-down (writing to a down TAP fd fails with EIO, so the link is genuinely dead) + QMP set_link gns3-N off so the guest notices; resume restores both, and the netem qdisc survives the flap

Capture, markers, filters

Identical to Docker's kernel links, on the TAP: capture start_kernel <tap> (driven by the NIO type, not by a link flag), marker add_kernel <tap> … (only the capture node's end), and one tc netem set per anchor so each direction is impaired once. frequency_drop/quota/window_drop run as the eBPF classifier on the anchor's clsact egress.

Legacy fallback

A uBridge build without the tap module (or without CAP_NET_ADMIN) keeps the previous datapath untouched: local UDP tunnels into uBridge, QEMU launched with -netdev socket, and the node logs a warning that it cannot carry kernel links. Such a node is never offered a kernel link, so nothing fails.

Verified

End-to-end through a real server (isolated instance, two QEMU nodes with a stub binary, the test playing the guest by holding the TAP fds and pushing frames): 32/32 checks on the kernel datapath and 29/29 on the relay variant (enable_kernel_datapath = false). Kernel run highlights:

• anchors created persistent and owned by the server user, QEMU's netdev is the TAP, no local UDP tunnel left; link_list reports kernel_datapath: true and one per-link bridge holds both taps;
• frames cross in both directions (~0.1 ms); delay 100 ⇒ netem on both anchors and a measured 100.3 ms one-way;
• suspend ⇒ anchor DOWN, no traffic, resume ⇒ 100.3 ms again (filter kept);
• capture ⇒ a 252-byte pcap written by capture start_kernel;
• link delete ⇒ bridge gone, qdisc clean, anchor alive; re-create ⇒ 0.1 ms with no residual impairment; node stop ⇒ no taps, no bridges left.

Relay run the same script with the flag off: same lifecycle, filter measured at 109.8 ms (applied by uBridge's bridge — no netem on the anchor, as expected), no kernel bridge holding the tap.

Configuration

[Server]
# Wire eligible links (both endpoints on one compute, both able to anchor)
# through kernel interfaces instead of the uBridge UDP relay.
enable_kernel_datapath = True

Known caveat

Anchors are host-side netdevs, so the kernel gives them an IPv6 link-local address and emits MLD/DAD from them; those frames flood into the emulated segment. The hardening requirement (anchors must be pure L2, no L3 identity) is specified for uBridge in docs/design/ubridge-l2-anchor-spec.md and is delivered: anchors and per-link bridges come up with addrgenmode none, no addresses, and an idle anchor is silent (asserted in the e2e after a 2.5 s settle).

One residual is accepted and documented there: enslaving a port makes the kernel announce the bridge's multicast memberships once — an IGMPv3 plus one or two MLDv2 reports in the first second — which floods into the segment like any L2 control frame. It stops; it is outside the L2-only command's reach (no addresses involved, IPv4 has no per-device IGMP switch).

Roadmap

IOU (its fabric terminator in uBridge needs a TAP-terminated port — iol_bridge add_nio_tap, frozen in docs/design/ubridge-iol-tap-anchor-spec.md; unlike QEMU, one userspace hop on the IOU leg is irreducible, the fabric is Unix sockets), Dynamips (hypervisor-created taps, enslavable as they are), and the Ethernet switch / cloud paths (their anchors are ubridge-owned by design) are still on the relay; cross-compute kernel links need VXLAN/GENEVE encapsulation, which is a separate project (it would serve Docker links the same way).

Update: IOU has landed on the same mixin — see docs/features/iou-kernel-datapath.md (Ethernet bays anchor on persistent TAPs bound to the IOL fabric; serial links stay relay). Dynamips has landed too — see docs/features/dynamips-kernel-datapath.md (the hypervisor opens uBridge-created TAPs with nio create_tap: the same external-fd-holder shape QEMU uses; serial/ATM/POS ports stay relay).