From 3910a6b58597587df6459e0cabcb5d8f0d2cf99f Mon Sep 17 00:00:00 2001 From: YueGuobin Date: Thu, 1 Oct 2026 11:55:33 +0800 Subject: [PATCH] docs(dynamips): record the kernel datapath and its live verification New features/dynamips-kernel-datapath.md: the anchor model (persistent TAP per Ethernet port, the hypervisor opens it with nio create_tap), port selection via the adapter registry, the anchor lifecycle with QEMU's stop order and the stop-is-not-close semantics, link operations, capture/ markers/filters, the relay fallback, the live e2e results, configuration and the roadmap (Ethernet switch next). The IOU and QEMU roadmaps note that Dynamips has landed. --- docs/features/dynamips-kernel-datapath.md | 155 ++++++++++++++++++++++ docs/features/iou-kernel-datapath.md | 12 +- docs/features/qemu-kernel-datapath.md | 5 +- 3 files changed, 166 insertions(+), 6 deletions(-) create mode 100644 docs/features/dynamips-kernel-datapath.md diff --git a/docs/features/dynamips-kernel-datapath.md b/docs/features/dynamips-kernel-datapath.md new file mode 100644 index 000000000..474af3936 --- /dev/null +++ b/docs/features/dynamips-kernel-datapath.md @@ -0,0 +1,155 @@ + + +# Dynamips kernel datapath + +Dynamips routers emulate every port inside the Dynamips process; today a link +reaches a router through a UDP tunnel (a Dynamips `nio create_udp`, a +loopback port pair, the node's uBridge relay, and the link's UDP NIOs). The +kernel datapath replaces that segment with the same anchor model Docker, +QEMU and IOU use: every **Ethernet slot/port** owns a persistent TAP +(`gd{node_id[:8]}e{slot}p{port}`), created by the server through uBridge's +tap module, and links attach to the anchor — a per-link Linux bridge +enslaves it, tc impairments run on it, capture and markers anchor on it. +The Dynamips **hypervisor** opens the TAP with `nio create_tap` (QEMU's +move: an external process holds the fd), unprivileged because the server +handed the device's ownership to its own user. + +``` + Dynamips hypervisor root namespace + ┌──────────────┐ nio create_tap ┌────────────────────────────────┐ + │ IOS router │◄────────────────►│ gd{id}e{slot}p{port} (TAP) │ + │ (emulated │ holds the fd │ │ per-link kernel bridge│ + │ ports) │ │ ▼ gns3{link_id} │ + └──────────────┘ └────────────────────────────────┘ +``` + +The legacy relay is untouched: a port without a kernel link keeps its +Dynamips UDP NIO and the node's uBridge relay tunnel exactly as before, and +a kernel link binds the same port to its anchor instead — per port, no +node-wide switch. + +## Datapath selection + +`UDPLink._kernel_datapath_eligible` extends the common rules with: + +* the compute must report **`ubridge_tap`** — the anchors are uBridge-created + TAPs the hypervisor merely opens, the same capability QEMU gates on; + there is no Dynamips-specific one; +* **both ports must be Ethernet**. The controller's port matrix decides + per port; serial, ATM and POS ports keep the relay whatever the + capabilities (the adapter registry in + `compute/dynamips/adapters/adapter.py` names the Ethernet models — + `ETHERNET_ADAPTERS` / `ETHERNET_WICS`, WIC-1ENET included, at its + Dynamips port number `16 * (wic_slot + 1)`). + +## Anchor lifecycle + +* **Birth (node start)** — `_prepare_tap_datapath` probes the tap module + (one create/delete cycle) and creates one persistent TAP per Ethernet + slot/port, ownership handed to the server user, born DOWN. A stale + leftover from a previous run is swept first; an anchor that already + exists (restart) is kept — an anchor is never recreated under a live + link. Serial/ATM/POS ports simply have no anchor. +* **Life** — a kernel link opens the anchor in the hypervisor + (`nio create_tap`), binds it to the slot/port and enslaves it into the + per-link bridge; removing the link unbinds the port, deletes the + hypervisor's TAP NIO (releasing the fd) and tears the anchor out of the + bridge. A link bound while the router is **stopped** is stored and wired + by the next start (deferred wiring — the NIO simply waits in the slot + adapter). +* **Death (node close)** — `_stop_ubridge` runs in **QEMU's order**, not + IOU's: here the Dynamips hypervisor holds the anchor fds, so its TAP + NIOs go first (`nio delete` closes the fd), then `tap delete` per anchor, + then the per-link kernel bridges, then uBridge itself. + +**A stop is not a close.** `stop` only halts the emulated router: the +hypervisor process — and with it the TAP fds, the port bindings and the +per-link bridges — survives `vm stop` (the same reason a relay link's +tunnel always survived a stop). Links therefore keep working through a +stop/start with no re-wiring at all; the restart's attach pass skips every +port that still has its TAP NIO. If the hypervisor died mid-flight (crash +or kill), the persistent TAPs remain and the next start re-attaches from +the stored NIOs. + +## Link operations + +| Operation | Kernel-datapath action | +|---|---| +| Create / attach | `nio create_tap` (hypervisor holds the fd) + `vm slot_add_nio_binding` + `brctl create` + `brctl addif` + carrier up + markers + `tc netem set` | +| Delete / detach | marker teardown → `tc reset` → carrier off → `brctl delif`/`brctl delete` → unbind + `nio delete` (release the fd) | +| Update (filters/markers/suspend changed) | reconcile on the anchor only — no re-binding, no re-enslaving | +| Suspend | anchor admin-down (traffic dead both ways) + resume restores | +| Adapter hot-add (OIR) | anchors for the new adapter's Ethernet ports created at once, so a kernel link can attach immediately | + +## Capture, markers, filters + +Capture on a kernel link is `capture start_kernel `; markers ride +the base reconcile with the anchor as the location (`marker add_kernel` — +the shared KernelDatapathMixin machinery, no Dynamips-specific code); +filters are one tc netem qdisc per anchor plus cls_bpf/eBPF drops. A relay +port keeps its Dynamips NIO and the node's uBridge relay bridge +(`DYNAMIPS-*`) with userspace filters, exactly as before. + +## Relay fallback + +A uBridge without the tap module keeps the node relay-only: `_prepare_tap_datapath` +warns, no anchors are created, the warning says the node cannot carry +kernel links, and every link keeps its `nio create_udp` tunnel. The +controller never offers such a node a kernel link (capability unreported). + +## Verified + +Unit level (`tests/compute/dynamips/test_dynamips_kernel_datapath.py`): +anchor topology (Ethernet models and WIC-1ENET port numbering only), the +lifecycle (create/sweep/skip-existing), deferred wiring of NIOs bound +while stopped, attach/update/remove command shapes on both datapaths, the +stop order (hypervisor NIO delete before tap delete before brctl delete), +capture and marker shapes, hot-added adapters, and the manager's +`nio_bridge` construction. Plus the controller-eligibility cases (mixable +with docker/qemu/iou, tap capability required, serial excluded) in +`tests/controller/test_kernel_datapath_link.py`. + +End-to-end (`tests/e2e/test_dynamips_kernel_datapath.py`, pytest marker +`e2e`) on a live server with **two real c7200 routers on a real IOS image**, +driven through the REST API and the IOS consoles with real ICMP: + +* a link created while both routers are stopped selects the kernel + datapath and is wired by node start (deferred wiring); +* the per-link kernel bridge on the host has exactly the two anchor TAPs + enslaved; `show ip int brief` is up/up and ping R1→R2 succeeds; +* `delay 100` ⇒ netem qdisc visible on both anchors and the ping RTT + grows by ~200 ms (one-way 100 per direction, each anchor impairing its + own ingress side); clearing resets the qdiscs and the RTT; +* suspend ⇒ anchor administratively down, ping 0%; resume restores both; +* capture writes a pcap full of ICMP-over-Ethernet records; +* a **serial** link between the same two routers stays on the relay + (per-port exclusion) and pings over it — the relay regression net; +* link delete/re-create: the bridge goes and comes back, the anchors + survive, traffic resumes; +* router stop/start keeps the wiring intact (bridge membership unchanged, + no duplicate NIOs) and traffic resumes after the reboot; +* deleting the project removes every anchor and bridge. + +`test_dynamips_relay_control` is the negative control: the same topology on +an isolated relay-configured instance (`enable_kernel_datapath = false`) +has no per-link bridge and nothing enslaved, and still pings — so the +kernel objects above can only come from the kernel datapath. + +## Configuration + +```ini +[Server] +# Wire eligible links (same compute, both endpoints Ethernet and able to +# anchor) through kernel interfaces instead of the uBridge UDP relay. +enable_kernel_datapath = True +``` + +## Roadmap + +The Ethernet switch is next (its ports are already uBridge-owned TAPs; the +kernel link should attach the peer's anchor straight into the switch's +per-node bridge), then the IOL Docker node. Cross-compute kernel links need +VXLAN/GENEVE encapsulation — deferred until every node type is kernelized. diff --git a/docs/features/iou-kernel-datapath.md b/docs/features/iou-kernel-datapath.md index 8396f3998..4b9444433 100644 --- a/docs/features/iou-kernel-datapath.md +++ b/docs/features/iou-kernel-datapath.md @@ -169,8 +169,10 @@ enable_kernel_datapath = True ## Roadmap -Dynamips next (hypervisor-created taps, enslavable as they are), then the -Ethernet switch (its anchors are ubridge-owned by design) and the IOL Docker -node (same `bridge add_nio_tap` shape as this one). Cross-compute kernel -links need VXLAN/GENEVE encapsulation — deferred until every node type is -kernelized. +Dynamips has landed (see `dynamips-kernel-datapath.md` — the hypervisor +opens uBridge-created TAPs, the same shape QEMU uses). Next: the Ethernet +switch (its ports are already uBridge-owned TAPs; the kernel link should +attach the peer's anchor straight into the switch's per-node bridge), then +the IOL Docker node (same `bridge add_nio_tap` shape as this one). +Cross-compute kernel links need VXLAN/GENEVE encapsulation — deferred +until every node type is kernelized. diff --git a/docs/features/qemu-kernel-datapath.md b/docs/features/qemu-kernel-datapath.md index 90e336665..9ee177e77 100644 --- a/docs/features/qemu-kernel-datapath.md +++ b/docs/features/qemu-kernel-datapath.md @@ -172,4 +172,7 @@ a separate project (it would serve Docker links the same way). Update: **IOU has landed** on the same mixin — see `docs/features/iou-kernel-datapath.md` (Ethernet bays anchor on persistent -TAPs bound to the IOL fabric; serial links stay relay). +TAPs bound to the IOL fabric; serial links stay relay). **Dynamips has +landed** too — see `docs/features/dynamips-kernel-datapath.md` (the +hypervisor opens uBridge-created TAPs with `nio create_tap`: the same +external-fd-holder shape QEMU uses; serial/ATM/POS ports stay relay).