gns3-server/docs/features/iou-kernel-datapath.md

IOU kernel datapath

IOU's only physical layer is the Unix-datagram IOL fabric (/tmp/netio<uid>/, 8-byte IOL headers, NETMAP), so it has no host netdev to enslave — a kernel link instead gives each Ethernet bay/unit a persistent TAP anchor (gi{node_id[:8]}e{bay}p{unit}), and uBridge's iol_bridge (the fake IOL instance the NETMAP maps to) binds the port to that TAP with iol_bridge add_nio_tap. Links then attach to the anchor exactly as they do for Docker (veth) and QEMU (TAP): a per-link Linux bridge enslaves it, tc impairments run on it, capture and markers anchor on it. The uBridge capability is specified in docs/design/ubridge-iol-tap-anchor-spec.md and delivered by the feature/iol-tap-anchor branch of the ubridge fork.

Serial bays never anchor (a TAP carries Ethernet frames): serial links stay on the relay datapath whatever the compute's capabilities.

Architecture

 IOU process                     root namespace
 ┌──────────┐  unix datagram     ┌──────────────────────────────┐
 │ IOL      │◄──── /tmp/netio ──►│ uBridge iol_bridge (listener)│
 │ fabric   │    (8-byte hdr)    │        │ holds the TAP fd    │
 └──────────┘                    │        ▼                      │
                                 │   gi{id}e{bay}p{unit}  ──────┼──► gns3{link_id}
                                 │   (persistent TAP anchor)    │    (per-link kernel
                                 └──────────────────────────────┘     bridge port)

One userspace hop on the IOU leg is irreducible — the fabric is Unix sockets, so frames always cross fabric → listener thread → TAP fd. What the kernel datapath buys is the link segment: anchor → per-link bridge → peer anchor crosses entirely in the kernel (including IOU ↔ Docker/QEMU links), impairments run as kernel tc instead of the userspace delay line (the known blocking-nanosleep RTT defect), and suspend maps to anchor admin-down (writes to a down TAP fail EIO and are dropped by the delivered §B hardening). Direction semantics on the anchor match QEMU's exactly: uBridge holds the fd like the QEMU process does.

Datapath selection

UDPLink._kernel_datapath_eligible extends the Docker/QEMU rules with:

• the compute must report ubridge_iol_tap (its own capability: the tap module and iol_bridge add_nio_tap land independently, and either can be missing on an old build). Probed by running one add/delete cycle on a scratch bridge + scratch TAP (probe_iol_tap_support, cached by binary identity, scratch bridge id 1050 — above the per-node id space 513..1024 the fabric locks); any failure means unknown → relay;
• both linked ports must be Ethernet — an IOU serial port disqualifies the kernel path per-port, not per-node, so serial links between capable IOU nodes silently stay on the relay instead of failing at the compute.

The NIO schema on the compute NIO routes accepts nio_bridge alongside the UDP/TAP/Ethernet NIOs (the same widening QEMU needed — without it the controller's NIO POST is 422'd before the node ever sees it).

Anchor lifecycle — and why the stop order is QEMU's reverse

• Birth (node start) — _prepare_tap_datapath probes the capability and creates one persistent TAP per Ethernet bay/unit (4 units per bay, swept for stale leftovers first, born DOWN). uBridge holds a device's fd only while a kernel link binds the port; until then the TAP has no holder.
• Life — the TAP is never recreated. A kernel link adds iol_bridge add_nio_tap <bridge> <iol_id> <bay> <unit> <tap> then enslaves the anchor (_kernel_attach); a relay link keeps iol_bridge add_nio_udp and never touches the anchor. Switching a port between datapaths is just an NIO swap — the IOL port frees its previous NIO (fd or socket) on attach.
• Death (node stop) — _stop_ubridge runs the cleanup in the reverse order of QEMU's, because here uBridge itself holds the anchor fds: iol_bridge delete (releases every port's TAP fd) → tap delete per anchor (a held TAP answers EBADFD/207) → brctl delete for per-link bridges still held → stop the hypervisor. Removing a single kernel link does the same in miniature: _remove_kernel_nio (markers, tc reset, delif, brctl delete) then iol_bridge delete_nio_tap — releasing the fd while the control channel lives.

Link operations

Operation	Kernel-datapath action
Create / attach	iol_bridge add_nio_tap (bind port ↔ anchor, uBridge holds the fd) + brctl create (EEXIST tolerated) + link set <bridge> up + brctl addif + carrier up + markers + tc netem set
Delete / detach	marker teardown → tc reset <tap> → carrier off → brctl delif/brctl delete (last endpoint wins) → iol_bridge delete_nio_tap (release the fd; the device survives)
Update (filters/markers changed)	reconcile on the anchor only — no re-binding, no re-enslaving
Suspend	anchor admin-down (IOL relay writes fail EIO → dropped; reads block) + resume restores

Capture, markers, filters

Capture on a kernel link is capture start_kernel <tap> (driven by the NIO type); markers ride the base reconcile with the anchor as the location — IOU's marker primitives are polymorphic (marker add_kernel/enable_kernel/ delete_kernel on an anchor, iol_bridge … mark on an IOL location), which let the old IOU-specific reconcile override be deleted outright. Filters are one tc netem qdisc per anchor plus cls_bpf/eBPF drops — the port's userspace iol_bridge add_packet_filter filters carry relay links only and are never combined with tc on the same port.

Relay fallback

A uBridge without iol_bridge add_nio_tap (old build) keeps the node on the relay datapath unchanged: every link rides iol_bridge add_nio_udp, the node starts fine, and the warning says it cannot carry kernel links. The controller never offers such a node a kernel link (capability unreported).

Verified

Unit level: anchor lifecycle and naming, kernel/relay binding on add, update and remove, the stop order (iol_bridge delete before tap delete before brctl delete before hypervisor stop), _networking on restart with mixed kernel/relay ports, capture and marker command shapes on both datapaths, the capability probe (cycle, old-build fallback, cleanup, caching), capability plumbing, and controller eligibility (per-compute capability, its independence from ubridge_tap, serial-port exclusion).

End-to-end on a live server (isolated instance, two IOU nodes behind a fake ELF image, the test playing the IOL fabric on /tmp/netio<uid>/ and pushing frames through it): 64/64 checks on the kernel datapath and 62/62 on the relay (enable_kernel_datapath = false). Kernel highlights:

• 8 anchors per node (2 bays × 4 units), persistent, DOWN, address-free; link_list reports kernel_datapath: true and one per-link bridge holds both anchors with brport/state = 3;
• fabric → anchor → bridge → anchor → fabric in both directions at ~0.2 ms; delay 100 ⇒ netem on both anchors and a measured 100.5 ms one-way;
• suspend ⇒ anchor DOWN, no traffic, resume ⇒ 100.4 ms with the qdisc intact;
• capture ⇒ a 328-byte pcap written by capture start_kernel;
• link delete ⇒ bridge gone, qdisc clean, anchor alive, re-create at 0.6 ms;
• a serial link between the same two capable nodes stays on the relay (kernel_datapath: false) — per-port exclusion, not per-node;
• single-sided restart ⇒ stopped node's anchors removed, peer's kept, link re-attached from the NIO (0.3 ms); stop ⇒ no stray anchors, bridges or fabric sockets.

The relay run exercises the same lifecycle with the IOL UDP NIOs (no tc qdiscs, userspace delay at 100.7 ms, suspend via the synthetic frequency_drop, iol_bridge start_capture), which is also the regression net for the port-coordinate capture restore this work touched.

The repository's live e2e suite carries the same scenario with real IOU routers (tests/e2e/test_iou_kernel_datapath.py, pytest marker e2e): a real L3 IOU image provides the fabric and the real IOS CLI on the server's telnet console, driving anchors-born-with-the-node, real ICMP through the port bridge, netem delay, suspend, capture, link delete/re-create, node stop/start rewiring, the serial link staying on the relay, and a relay negative control.

The uBridge side passed its own 30/30 (tests/iol/test_tap_anchor.py in the fork), including the DOWN-anchor resilience and kernel-bridge interop this design depends on.

One pre-existing bug surfaced here and is fixed: the IOU NIO update route never copied the body's suspend onto the NIO (Docker and QEMU routes do). Harmless on the relay, where suspend is emulated by the synthetic frequency_drop filter, but fatal for a kernel link, whose suspend is the anchor's admin state — a "suspended" link kept forwarding.

Measuring seeded impairments on IOU: IOL emits CDP/keepalive frames of its own, and they draw from the netem RNG stream — a seed-reproducible run never matches while they are on (measured: three rounds of mismatches, then bit-identical pcaps after no cdp run). Turn the platform chatter off before comparing seeded runs; the extra frames are visible in the capture.

Configuration

[Server]
# Wire eligible links (same compute, both endpoints Ethernet and able to
# anchor) through kernel interfaces instead of the uBridge UDP relay.
enable_kernel_datapath = True

Roadmap

Dynamips has landed (see dynamips-kernel-datapath.md — the hypervisor opens uBridge-created TAPs, the same shape QEMU uses), and so has the Ethernet switch (see ethernet-switch-kernel-datapath.md — it absorbs the peer's anchor into its own kernel bridge). Switch-to-switch cascades (a veth pair between the two kernel bridges) have landed, and so has the IOL runner container — see iol-docker-kernel-datapath.md (same persistent-TAP shape, through the generic bridge module's swappable TAP leg instead of the IOL bridge). Cross-compute kernel links need VXLAN/GENEVE encapsulation — deferred until every node type is kernelized.