From 02340f2ed330481a5a23088f57ec781ef224c1ae Mon Sep 17 00:00:00 2001 From: YueGuobin Date: Fri, 2 Oct 2026 13:30:59 +0800 Subject: [PATCH] docs(iol): record the IOL runner container kernel datapath MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The feature doc covers the architecture (one irreducible userspace hop on the unix-socket leg; the link segment kernelized through the port bridge's swappable TAP leg), the capability gate, the anchor lifecycle, the swap flow, the inherited capture/markers/filters, the relay fallback and the live verification — including the ensure-then-add lesson (tap create is strictly create-only). The other datapath docs' roadmaps catch up: the IOL runner container has landed (and the ethernet-switch doc's stale 'cascade deferred' exclusion from d2f8aba58 is corrected while there); the docker doc's adapter table no longer rejects IOLDockerVM. --- docs/features/docker-kernel-datapath.md | 2 +- docs/features/dynamips-kernel-datapath.md | 4 +- .../ethernet-switch-kernel-datapath.md | 20 +- docs/features/iol-docker-kernel-datapath.md | 172 ++++++++++++++++++ docs/features/iou-kernel-datapath.md | 11 +- docs/features/qemu-kernel-datapath.md | 7 +- 6 files changed, 199 insertions(+), 17 deletions(-) create mode 100644 docs/features/iol-docker-kernel-datapath.md diff --git a/docs/features/docker-kernel-datapath.md b/docs/features/docker-kernel-datapath.md index 42ebb763f..7de504a59 100644 --- a/docs/features/docker-kernel-datapath.md +++ b/docs/features/docker-kernel-datapath.md @@ -85,7 +85,7 @@ markers): |---|---|---|---| | `DockerVM` | default | veth pair | yes | | `VendorDockerVM` (non-unix-socket) | `console_type=docker_exec`, `GNS3_SKIP_INIT`, … (XRd, SR Linux, …) | veth pair (same path via `super()`) | yes | -| `IOLDockerVM` | `GNS3_IOL_RUNNER=1` | AF_UNIX datagram socket pairs (`sNN`/`cNN`) — the container netns is unused | rejected (no host-side interface) | +| `IOLDockerVM` | `GNS3_IOL_RUNNER=1` | AF_UNIX datagram socket pairs (`sNN`/`cNN`) — the container netns is unused; kernel links ride persistent `gx` TAP anchors through the port bridge's swappable TAP leg (see `iol-docker-kernel-datapath.md`) | yes (gated on `ubridge_bridge_tap` + `ubridge_tap`) | | `VendorDockerVM` (unix-socket) | `GNS3_UNIX_SOCKET_NIO=1` | same socket contract, generic capability for vendor NOS images | rejected | Unix-socket containers are detected both in the controller diff --git a/docs/features/dynamips-kernel-datapath.md b/docs/features/dynamips-kernel-datapath.md index 64e18ad02..4d731c7b9 100644 --- a/docs/features/dynamips-kernel-datapath.md +++ b/docs/features/dynamips-kernel-datapath.md @@ -151,5 +151,7 @@ enable_kernel_datapath = True The Ethernet switch has landed (see `ethernet-switch-kernel-datapath.md` — it absorbs the peer's anchor into its own kernel bridge; switch-to-switch -cascades and the IOL Docker node are next). Cross-compute kernel links need +cascades and the IOL runner container have landed too — see +`ethernet-switch-kernel-datapath.md` and +`iol-docker-kernel-datapath.md`). Cross-compute kernel links need VXLAN/GENEVE encapsulation — deferred until every node type is kernelized. diff --git a/docs/features/ethernet-switch-kernel-datapath.md b/docs/features/ethernet-switch-kernel-datapath.md index 2c309707a..7808129ee 100644 --- a/docs/features/ethernet-switch-kernel-datapath.md +++ b/docs/features/ethernet-switch-kernel-datapath.md @@ -28,15 +28,17 @@ end / TAP becomes the switch port; frames cross ## Datapath selection A link with an `ethernet_switch` endpoint rides the fast path when the -**peer** can anchor (same compute; Docker always, QEMU/IOU/Dynamips per +**peer** can anchor (same compute; Docker always — including IOL runner +containers, per their own capability gate — QEMU/IOU/Dynamips per their capabilities) — the switch side needs no anchoring capability of its own, its bridge and brctl exist by construction. Exclusions, all staying on the relay: -* **switch ↔ switch**: a kernel interface can belong to exactly one - bridge, so cascading two switch bridges needs a veth pair between them — - a separate mechanism, deferred; -* anchor-less peers (VPCS, cloud, the Dynamips-hosted hub); +* **switch ↔ switch** cascades ride the kernel too — through a veth pair + joining the two bridges (one end per switch, each side applying its own + port mode to its own end); see the wire format below; +* anchor-less peers (VPCS, cloud, the Dynamips-hosted hub, generic + `GNS3_UNIX_SOCKET_NIO` containers); * cross-compute links, non-Ethernet ports: the common rules. ## The wire format: one NIO names the other end's interface @@ -153,6 +155,8 @@ enable_kernel_datapath = True ## Roadmap Switch-to-switch links (a veth pair joining the two kernel bridges, with -each side's port mode applied at its own end) and the IOL Docker node -(`bridge add_nio_tap`); cross-compute kernel links need VXLAN/GENEVE -encapsulation, deferred until every node type is kernelized. +each side's port mode applied at its own end) have landed (see above), and +so has the IOL runner container (`iol-docker-kernel-datapath.md` — its +`iol_docker` anchors absorb like any peer's). Cross-compute kernel links +need VXLAN/GENEVE encapsulation, deferred until every node type is +kernelized. diff --git a/docs/features/iol-docker-kernel-datapath.md b/docs/features/iol-docker-kernel-datapath.md new file mode 100644 index 000000000..91d3a4b08 --- /dev/null +++ b/docs/features/iol-docker-kernel-datapath.md @@ -0,0 +1,172 @@ + + +> This documentation is AI-generated with reference to actual code and verified +> against real-kernel testing. AI can make mistakes — please verify against +> the source code when in doubt. + +# IOL runner container kernel datapath + +## Overview + +IOL runner containers (`IOLDockerVM`, selected by the `GNS3_IOL_RUNNER` +environment marker — the Cisco CML `iol-runner` images such as +`iol-xe/iol-xe:17-18-02`) have no container-namespace networking at all: +their guest leg is the runner's per-interface AF_UNIX datagram socket pairs. +A unix socket is not a kernel interface, so until now every link of such a +container rode the uBridge userspace relay (unix ↔ UDP). + +The kernel datapath gives every Ethernet bay/unit a **persistent TAP anchor** +(`gx` names, born at node start exactly like IOU's) and wires the *link +segment* through the kernel. One userspace hop on the container leg is +irreducible — the socket pair is the only physical layer the runner exposes — +but the anchor → per-link bridge → peer anchor crossing, the kernel filters, +the suspend carrier and the AF_PACKET capture/markers are all native now. + +``` + container netns (iol-runner) root namespace + ┌────────────┐ unix datagram ┌────────────────────────────────┐ + │ netiomux │◄─ sNN/cNN.sock ──►│ uBridge port bridge │ + │ (sNN/cNN) │ (host-side dir) │ [nio_unix, nio_tap] relay │ + └────────────┘ │ │ holds the TAP fd │ + │ ▼ │ + │ gx{id}e{bay}p{unit} ─────────┼─► gns3{link_id} + │ (persistent TAP anchor) │ (per-link kernel + └────────────────────────────────┘ bridge port) +``` + +The port bridge is the deployment shape ubridge's +`doc/gns3server-integration.md` § *bridge — generic NIO relay with a +swappable TAP leg* froze: a per-node bridge created at start, the unix NIO in +the first slot for the node's whole life, the topology leg in the second — +`add_nio_udp` (relay) or `add_nio_tap` (kernel link) — swapped as links are +attached, suspended, deleted or the project reopens, through +stop → delete → add → start. The c-socket binding never re-binds across a +swap: frames the container sends while the bridge is stopped queue on the +socket and relay after `start` (regression-tested in ubridge's bridge suite). + +## Datapath selection + +`UDPLink._kernel_datapath_eligible` treats an IOL runner container like any +Docker node (same compute, Ethernet port) with one gate of its own in +`_kernel_endpoint_ready`: the compute must report both + +* `ubridge_tap` — the tap module (`tap create`/`tap delete`), the same + capability QEMU and Dynamips gate on, and +* `ubridge_bridge_tap` — `bridge delete_nio_tap`, the swappable-leg command + (probed with one command against a bridge that cannot exist: 202 = old + build, 214 = new; no scratch objects, no capabilities, cached by binary + identity like every other probe). + +Both strict `True`; anything unreported (old uBridge, failed probe) keeps the +link on the relay, where it always works. The generic `GNS3_UNIX_SOCKET_NIO` +containers stay excluded as before — that marker's semantics are +image-specific, while `GNS3_IOL_RUNNER` selects a class whose anchors this +server owns. + +## Anchor lifecycle + +* **Birth (node start, before any link)** — `_start_ubridge` runs + `_prepare_tap_datapath` right after the hypervisor connects: probe both + capabilities, then for every (bay, unit 0-3) sweep a leftover + (`tap delete`, suppressed), `tap create` (hardened, starts DOWN) and + `link set down` — carrier off until a link attaches. The anchors exist + whether or not any link ever uses them: an anchor born with its link would + leave the Ethernet switch fast path's deferred join waiting forever. + No `tap set_owner`: uBridge itself holds these fds. +* **Life** — the anchor is never recreated. Link create/delete/switch, + suspend, capture and markers only change what is attached to it, and the + ensure-then-add contract guards the one window where the device could have + been swept (`/sys/class/net` existence check — `tap create` is strictly + create-only, `IFF_TUN_EXCL`, and refuses an existing device with EBUSY). +* **Death (node stop)** — the port bridges hold the anchor fds + (`bridge add_nio_tap`), so they are deleted first (`bridge delete` stops + their relay threads and frees the NIOs), then the anchors themselves + (`tap delete` answers EBADFD on a device another fd still holds), then the + per-link kernel bridges — all while the control channel lives, before the + hypervisor stops. A restart recreates everything from the NIO. + +## Link operations + +* **Kernel link create** (`_attach_kernel_link`): ensure the port bridge and + its unix NIO → ensure the anchor → `bridge add_nio_tap` (uBridge opens and + holds the fd) → the shared mixin flow (`brctl create`/`addif` the per-link + bridge, capture, markers, tc netem/cls_bpf/eBPF filters on the anchor) → + `bridge start` (the [unix ↔ tap] relay) → the carrier pass (which is also + what brings a born-down anchor up; a suspended NIO sets it back down). +* **Link delete** (`_release_port_tap`): `bridge stop` (delete_nio_tap + refuses while running — the relay threads hold the NIO pointers for their + whole life; freeing under them is a use-after-free) → + `bridge delete_nio_tap ""` (matched on the kernel-resolved + name) → the mixin teardown (markers, tc reset, `brctl delif`, per-link + bridge deletion). The anchor survives — the node owns it, not the link. +* **Relay link** — unchanged: the port bridge's second leg is `add_nio_udp`, + userspace filters at the port bridge. The unix NIO stays in its slot + across every datapath switch. + +## Capture, markers, filters + +Inherited from `KernelDatapathMixin` unchanged — everything is keyed on the +anchor interface name, which is why a TAP and a veth host end are +interchangeable there: `capture start_kernel` / `marker add_kernel` +(AF_PACKET), one tc netem qdisc (the full netem surface plus the +extensions), `bpf` as cls_bpf match-drop, `frequency_drop`/`quota`/ +`window_drop` as the eBPF stateful classifier — all capability-gated per +compute exactly like the Docker veth datapath. + +Suspend semantics: anchor admin-down. The port bridge's TAP writes fail EIO +(tolerated by the delivered uBridge hardening) and its reads fall silent — +traffic stops both ways. The runner itself is unaware of the link state +(netiomux has no carrier signal), exactly as on the relay datapath where +suspend rode the synthetic frequency_drop filter instead. + +## Relay fallback + +Missing either capability, the node runs relay-only: links ride +unix ↔ UDP, `kernel_datapath` stays false, and anchors are simply never +created (or, with `enable_kernel_datapath` off server-wide, created but +never attached — the same semantics as QEMU's and Dynamips' anchors). + +## Verified + +Live on the real server (`tests/e2e/test_iol_docker_kernel_datapath.py`, +isolated instance, real `iol-xe/iol-xe:17-18-02` images, real uBridge with +`bridge delete_nio_tap`): + +* kernel link reports `kernel_datapath`, anchors exist from node start, the + per-link bridge enslaves exactly the two of them; +* real ICMP crosses the [unix ↔ tap] port-bridge relay (100 % ping); +* `delay 100` lands as netem on both anchors, measured RTT grows ≥ 150 ms + and returns < 50 ms when cleared; +* suspend admin-downs the anchor and kills the traffic; resume restores; +* AF_PACKET capture writes a real pcap of the ICMP exchange; +* link delete/re-create swaps the port bridge's TAP leg out and back in + (unix binding never re-binds) with the ping returning; +* node stop tears the anchors down (no `gx` devices left), a restart + recreates the whole wiring from the NIO; +* project delete leaves no anchors and no bridges; the relay control on a + relay-configured instance keeps the link off the kernel and still pings. + +Also caught and fixed live: `tap create` is strictly create-only +(`IFF_TUN_EXCL`) — the ensure-then-add guard must check existence first +(`/sys/class/net`), not re-create unconditionally (EBUSY on the device the +start loop just made). + +## Configuration + +``` +[Server] +enable_kernel_datapath = True # the global switch (link attachment) +``` + +The anchor lifecycle needs no configuration of its own — the capability +probes decide per compute, and an IOL runner container without them simply +never anchors. + +## Roadmap + +Cross-compute kernel links need VXLAN/GENEVE encapsulation — deferred until +every node type is kernelized; with the IOL runner container anchored, that +condition is one project away from met. diff --git a/docs/features/iou-kernel-datapath.md b/docs/features/iou-kernel-datapath.md index 17be06e3c..c1b531a98 100644 --- a/docs/features/iou-kernel-datapath.md +++ b/docs/features/iou-kernel-datapath.md @@ -172,8 +172,9 @@ enable_kernel_datapath = True Dynamips has landed (see `dynamips-kernel-datapath.md` — the hypervisor opens uBridge-created TAPs, the same shape QEMU uses), and so has the Ethernet switch (see `ethernet-switch-kernel-datapath.md` — it absorbs the -peer's anchor into its own kernel bridge). What remains: switch-to-switch -cascades (a veth pair between the two kernel bridges) and the IOL Docker -node (same `bridge add_nio_tap` shape as this one). Cross-compute kernel -links need VXLAN/GENEVE encapsulation — deferred until every node type is -kernelized. +peer's anchor into its own kernel bridge). Switch-to-switch cascades (a veth +pair between the two kernel bridges) have landed, and so has the IOL runner +container — see `iol-docker-kernel-datapath.md` (same persistent-TAP shape, +through the generic bridge module's swappable TAP leg instead of the IOL +bridge). Cross-compute kernel links need VXLAN/GENEVE encapsulation — +deferred until every node type is kernelized. diff --git a/docs/features/qemu-kernel-datapath.md b/docs/features/qemu-kernel-datapath.md index 91ee4fc8d..0c9441209 100644 --- a/docs/features/qemu-kernel-datapath.md +++ b/docs/features/qemu-kernel-datapath.md @@ -182,5 +182,8 @@ hypervisor opens uBridge-created TAPs with `nio create_tap`: the same external-fd-holder shape QEMU uses; serial/ATM/POS ports stay relay). **The Ethernet switch has landed** — see `docs/features/ethernet-switch-kernel-datapath.md` (the switch absorbs a -peer's anchor into its own kernel bridge; switch-to-switch cascades stay -relay for now). +peer's anchor into its own kernel bridge; switch-to-switch cascades too). +**The IOL runner container has landed** — see +`docs/features/iol-docker-kernel-datapath.md` (persistent TAP anchors +through the generic bridge module's swappable TAP leg; the unix-socket guest +leg keeps one userspace hop, like IOU's fabric).