diff --git a/.claude/skills/gns3-documentation/SKILL.md b/.claude/skills/gns3-documentation/SKILL.md
index 8772e1f07..485de4945 100644
--- a/.claude/skills/gns3-documentation/SKILL.md
+++ b/.claude/skills/gns3-documentation/SKILL.md
@@ -1,7 +1,7 @@
---
name: documentation
description: Use this skill when creating or updating technical documentation under docs/ directory for GNS3 server
-version: 3.0.0
+version: 3.1.0
---
# GNS3 Server Technical Documentation Standard
@@ -39,7 +39,7 @@ Code details are left to the codebase — readers can use AI to find implementat
## What to Include
-- Mermaid diagrams for architecture and flow (GitHub renders natively)
+- Mermaid diagrams for architecture and flow (GitHub renders natively) — see Mermaid Diagram Rules below
- `graph` for component relationships
- `sequenceDiagram` for request/response flows
- `flowchart` for data processing steps
@@ -47,6 +47,26 @@ Code details are left to the codebase — readers can use AI to find implementat
- Request/response JSON examples (for interfaces)
- Performance data (only measured numbers, no estimates)
+## Mermaid Diagram Rules
+
+Draw only the diagram kinds the story needs — each kind answers exactly one reader question, in reading order:
+
+- **Anatomy — "what exists, and who owns it?"** — `flowchart LR` with the fewest nodes that show ownership: one subgraph per ownership boundary (a process, a namespace, userspace vs kernel), anything shared between owners a bare node inside the enclosing boundary. The steady-state path is bidirectional solid (`<-->`) edges labeled with the carrying technology, never with a direction; each observer/configurator gets one dashed sideband edge per role.
+- **Comparison — "how does X differ from Y?"** — the mirror of its counterpart diagram: same scenario, same node ids, same concrete names, one variable changed. Draw one direction with unidirectional `-->` and state in prose that the reverse is the mirror image.
+- **Control sequence — "how is it built / torn down?"** — `sequenceDiagram` spanning every layer (API → coordinator → per-node handlers → the managed process): local steps are self-messages, concurrent fan-out is a `par / and / end` block, each message to the bottom layer is one real command in real order, and a closing `Note` names the steady state.
+- **Walkthrough — "how does one item traverse it?"** — `flowchart LR` as a linear pipeline: one subgraph per stage, titled with that stage's role in this trace (`X — ingress`), sideband dashed edges (reverse-direction role, observation point) anchored at the stage they belong to.
+
+Whatever the kind:
+
+- **One step per node, one short line per label.** If a label needs `
` lines to explain itself, split the node into smaller steps instead of packing text — needing many words means the flow needs further decomposition.
+- **Details belong to prose.** Parentheticals, exceptions, and parameter lists go to the surrounding text or the section that owns the topic; a diagram never duplicates what prose already says.
+- **Define jargon before the first diagram.** A Term | Meaning table placed above the diagrams covers the domain terms they use (interface names, kernel mechanisms), so readers don't have to scan forward.
+- **Edge style is semantics.** Solid arrows carry the main flow (data plane); dashed `-.->` edges are sideband relationships (taps, control installs). A node with multiple sideband roles gets one dashed edge per role, each labeled with a single function — and edge *direction* matters too: a config edge points from the configurere to its target, an observation edge from the target to the observer.
+- **Decision points are diamonds with labeled outcomes**, not text-packed rectangles spelling out every branch inside one node.
+- **A comparison diagram changes exactly one variable.** When two diagrams contrast approaches (kernel vs relay, old vs new), everything else — scenario, topology, node names — stays identical between them; a difference that isn't the point belongs in a prose aside, not in the picture.
+- **Numbered crossings stay consistent.** When kernel/user or process-boundary copies are the point of a diagram, number the crossing edges (`copy N: user→kernel`) and keep one numbering scheme across every diagram in the document.
+- **Concrete names instantiate stated rules.** Every interface/bridge/instance name appearing in a diagram instantiates a deterministic naming rule stated in prose before the first diagram — never invent ad-hoc names.
+
## What to Skip
- Code snippets — let readers search the codebase
diff --git a/docs/features/docker-kernel-datapath.md b/docs/features/docker-kernel-datapath.md
index 423189a2c..711d856c9 100644
--- a/docs/features/docker-kernel-datapath.md
+++ b/docs/features/docker-kernel-datapath.md
@@ -155,8 +155,9 @@ flowchart LR
subgraph GVB["gvB — egress chain (clsact, then netem)"]
S5["eBPF prio 1"]
S6["bpf_drop prio 10-99"]
- S7["netem"]
+ S7["netem qdisc"]
end
+ T["capture / markers"]
end
subgraph BNS["container B netns"]
S9["eth0 B receives"]
@@ -168,9 +169,11 @@ flowchart LR
S4 --> S5
S5 --> S6 --> S7
S7 -->|"veth pair"| S9
+ S3 -. "AF_PACKET tap" .-> T
+ S7 -. "AF_PACKET tap" .-> T
```
-What each stage drops: the reserved-range gate passes LACP, LLDP, 802.1X and STP per the port mask but never PAUSE/PFC (Link-local frames below); the clsact/netem stages drop or impair per the installed filters (Packet filters below). The AF_PACKET taps observe both host ends but never a clsact-dropped frame.
+What each stage drops: the reserved-range gate passes LACP, LLDP, 802.1X and STP per the port mask but never PAUSE/PFC (Link-local frames below); the clsact/netem stages drop or impair per the installed filters (Packet filters below). Whichever end hosts capture / markers, its AF_PACKET tap — the dashed sideband above — clones the frame at that anchor: a tap on gvA sees the frame on arrival, a tap on gvB only if the clsact chain let it through.
All dozen-plus GNS3 filter types ride these three kernel objects — nine of them (delay, packet_loss, corrupt, duplicate, rate, reorder, gemodel, seed, limit) are parameters of the single netem qdisc, `bpf` expressions are the cls_bpf filters, and frequency_drop / quota / window_drop are the three modes inside the one eBPF program (evaluated nth → quota → window). The complete type-to-object mapping is the Packet filters table below.
diff --git a/docs/features/dynamips-kernel-datapath.md b/docs/features/dynamips-kernel-datapath.md
index baed487c7..5f844017d 100644
--- a/docs/features/dynamips-kernel-datapath.md
+++ b/docs/features/dynamips-kernel-datapath.md
@@ -5,74 +5,185 @@ See LICENSE file for licensing information.
# Dynamips kernel datapath
-Dynamips routers emulate every port inside the Dynamips process; today a link
-reaches a router through a UDP tunnel (a Dynamips `nio create_udp`, a
-loopback port pair, the node's uBridge relay, and the link's UDP NIOs). The
-kernel datapath replaces that segment with the same anchor model Docker,
-QEMU and IOU use: every **Ethernet slot/port** owns a persistent TAP
-(`gd{node_id[:8]}e{slot}p{port}`), created by the server through uBridge's
-tap module, and links attach to the anchor — a per-link Linux bridge
-enslaves it, tc impairments run on it, capture and markers anchor on it.
-The Dynamips **hypervisor** opens the TAP with `nio create_tap` (QEMU's
-move: an external process holds the fd), unprivileged because the server
-handed the device's ownership to its own user.
+Dynamips routers emulate every port inside the Dynamips process; on the relay datapath a link reaches a router through a UDP tunnel (a Dynamips `nio create_udp`, a loopback port pair, the node's uBridge relay, and the link's UDP NIOs). The kernel datapath replaces that segment with the same anchor model Docker, QEMU and IOU use: every **Ethernet slot/port** owns a persistent TAP (`gd{node_id[:8]}e{slot}p{port}`), created by the server through uBridge's tap module, and links attach to the anchor — a per-link Linux bridge enslaves it, tc impairments run on it, capture and markers anchor on it. The Dynamips **hypervisor** opens the TAP with `nio create_tap` (QEMU's move: an external process holds the fd), unprivileged because the server handed the device's ownership to its own user.
-```
- Dynamips hypervisor root namespace
- ┌──────────────┐ nio create_tap ┌────────────────────────────────┐
- │ IOS router │◄────────────────►│ gd{id}e{slot}p{port} (TAP) │
- │ (emulated │ holds the fd │ │ per-link kernel bridge│
- │ ports) │ │ ▼ gns3{link_id} │
- └──────────────┘ └────────────────────────────────┘
+## Architecture
+
+The diagram shows the e2e shape — two c7200 routers on one link (slot 0 port 0 each), the topology `tests/e2e/test_dynamips_kernel_datapath.py` drives. A Dynamips↔Docker/QEMU/IOU link is the same shape with the peer's own anchor on the other side; a routed transit is two such links with IOS forwarding between them. Interface names follow the deterministic rules `gd{node_id[:8]}e{slot}p{port}` for the anchor TAPs and `gns3{link_id[:11]}` for the bridge; the concrete names in the diagrams use node ids `7c1d4e2f` / `9e8b6a3f` and link id prefix `5d3c2b1a4e6` (`gdA`/`gdB` shorthand in the sequence below). Terms the diagrams use:
+
+| Term | Meaning |
+|---|---|
+| hypervisor | the Dynamips process emulating the router; it holds every anchor TAP's fd |
+| anchor | the host-side interface a link attaches to — for Dynamips the `gd` persistent TAP, for Docker the `gv` veth host end |
+| persistent TAP | the TAP created per Ethernet slot/port at node start (`tap create` + `tap set_owner`, ownership handed to the server user so the unprivileged hypervisor can open it); born DOWN, it outlives `vm stop` and hypervisor crashes |
+| per-link bridge | the Linux bridge `gns3{link_id[:11]}` that stands in for the cable; the two anchors are its only ports |
+| tap fd | a TAP's file descriptor: reads return frames the kernel received on the device, writes inject frames into it — the hypervisor opens it with `nio create_tap` and holds it for the port's lifetime |
+| FDB | the bridge's forwarding database (learned MAC → port); unknown, broadcast and multicast destinations are flooded |
+| `01:80:c2` / `group_fwd_mask` | the IEEE 802.1D reserved multicast range (STP, LACP, LLDP, 802.1X, PAUSE); the bridge forwards it only per the port's `group_fwd_mask` — `0xfffd` passes everything except PAUSE/PFC |
+| `clsact` | the tc classifier carrier on the anchor; its egress chain is where the drop classifiers run |
+| eBPF classifier | the stateful drop modes (every-Nth, quota, time window) at clsact egress prio 1 |
+| `bpf_drop` | match-drop BPF expressions at clsact egress prio 10–99 |
+| netem | the impairment qdisc behind clsact: delay, loss, rate, corrupt, duplicate, … |
+| `AF_PACKET tap` | uBridge's packet socket on the anchor — how capture and markers observe frames off the forwarding path |
+| carrier | the anchor's admin state (`link set up/down`); suspend is carrier down — traffic dead both ways |
+| `nio_bridge` | the NIO record the controller posts to each endpoint node: bridge name, filters, markers, suspend flag |
+| uBridge | the per-node helper process — on a kernel link it created the anchor, enslaves it and taps it, but never carries frames |
+
+```mermaid
+flowchart LR
+ subgraph H1["Dynamips hypervisor R1"]
+ ios1["IOS router 1"]
+ end
+ subgraph RootNS["root namespace"]
+ gdA["gd7c1d4e2fe0p0"]
+ BR["gns35d3c2b1a4e6"]
+ gdB["gd9e8b6a3fe0p0"]
+ ub1["uBridge R1"]
+ ub2["uBridge R2"]
+ end
+ subgraph H2["Dynamips hypervisor R2"]
+ ios2["IOS router 2"]
+ end
+ ios1 <-->|"TAP fd"| gdA
+ gdA <-->|"bridge port"| BR
+ BR <-->|"bridge port"| gdB
+ gdB <-->|"TAP fd"| ios2
+ gdA -.->|"AF_PACKET tap"| ub1
+ ub1 -.->|"tc qdisc"| gdA
+ gdB -.->|"AF_PACKET tap"| ub2
+ ub2 -.->|"tc qdisc"| gdB
```
-The legacy relay is untouched: a port without a kernel link keeps its
-Dynamips UDP NIO and the node's uBridge relay tunnel exactly as before, and
-a kernel link binds the same port to its anchor instead — per port, no
-node-wide switch.
+The only userspace crossings left are the Dynamips processes themselves — the routers *are* the emulation, and the hypervisor reads and writes the TAP directly (QEMU's external-fd-holder shape, no extra relay hop on the node leg unlike IOU's fabric). What the kernel datapath buys is the link segment: anchor → per-link bridge → peer anchor crosses entirely in the kernel, impairments run as kernel tc instead of userspace filters, and suspend maps to anchor admin-down. Each side's uBridge does its own tap-and-qdisc work on its node's anchor — but neither dashed edge is a steady fixture: the qdisc exists only while the link carries filters (deletion `tc reset`s it), the tap only while capture or markers run, and by convention only on the capture node's end, so an unfiltered, uncaptured link shows neither. The anchors themselves were created through uBridge at node start (Anchor lifecycle below).
+
+The legacy relay is untouched: a port without a kernel link keeps its Dynamips UDP NIO and the node's uBridge relay tunnel exactly as before, and a kernel link binds the same port to its anchor instead — per port, no node-wide switch. The same R1 ↔ R2 link on the relay — one direction drawn, the reverse is the mirror image; this is the exact shape the negative control keeps when `enable_kernel_datapath = false` (Verified below). Each relay bridge is named `DYNAMIPS-{lport}-{rport}` after the loopback port pair its NIO allocates at link time:
+
+```mermaid
+flowchart LR
+ subgraph H1["Dynamips hypervisor R1"]
+ ios1["IOS router 1"]
+ nio1["UDP NIO"]
+ end
+ subgraph ub1["uBridge R1"]
+ rl1["relay bridge"]
+ end
+ subgraph KERN["kernel"]
+ lp1["loopback UDP"]
+ tun["UDP tunnel"]
+ lp2["loopback UDP"]
+ end
+ subgraph ub2["uBridge R2"]
+ rl2["relay bridge"]
+ end
+ subgraph H2["Dynamips hypervisor R2"]
+ nio2["UDP NIO"]
+ ios2["IOS router 2"]
+ end
+ ios1 --> nio1
+ nio1 -->|"copy 1: user→kernel"| lp1
+ lp1 -->|"copy 2: kernel→user"| rl1
+ rl1 -->|"copy 3: user→kernel"| tun
+ tun -->|"copy 4: kernel→user"| rl2
+ rl2 -->|"copy 5: user→kernel"| lp2
+ lp2 -->|"copy 6: kernel→user"| nio2
+ nio2 --> ios2
+```
+
+Six kernel/user crossings each way, against the kernel datapath's two — and those two are the TAP-fd legs that are the emulation itself, so the relay's four extra copies are pure overhead. Userspace filters run inside the relay bridges, between copies 2-3 and 4-5. On a same-compute link the UDP tunnel rides loopback as drawn; cross-compute it rides the real network with the copy count unchanged.
+
+Frame path for one R1 → R2 frame. The impairment chain fires on the egress of the *destination* end's anchor, in order clsact → netem (a classifier drop means netem never sees the frame); each direction is impaired exactly once and R2 → R1 is symmetric on gdA:
+
+```mermaid
+flowchart LR
+ subgraph H1["Dynamips hypervisor R1"]
+ S1["IOS port sends"]
+ end
+ subgraph ROOT["root namespace"]
+ subgraph GA["gd7c1d4e2fe0p0 — ingress (egress chain serves R2→R1)"]
+ S3["ingress"]
+ end
+ subgraph BRP["per-link bridge"]
+ G1{"reserved 01:80:c2?"}
+ D["dropped"]
+ S4["FDB lookup / flood"]
+ end
+ subgraph GB["gd9e8b6a3fe0p0 — egress chain (clsact, then netem)"]
+ S5["eBPF prio 1"]
+ S6["bpf_drop prio 10-99"]
+ S7["netem qdisc"]
+ end
+ T["capture / markers"]
+ end
+ subgraph H2["Dynamips hypervisor R2"]
+ S9["IOS port receives"]
+ end
+ S1 -->|"tap fd, copy 1: user→kernel"| S3
+ S3 --> G1
+ G1 -->|"PAUSE 0x01"| D
+ G1 -->|"0xfffd passes"| S4
+ S4 --> S5
+ S5 --> S6 --> S7
+ S7 -->|"tap fd, copy 2: kernel→user"| S9
+ S3 -. "AF_PACKET tap" .-> T
+ S7 -. "AF_PACKET tap" .-> T
+```
+
+What each stage drops: the reserved-range gate passes LACP, LLDP, 802.1X and STP per the port mask but never PAUSE/PFC (Link-local frames below); the clsact/netem stages drop or impair per the installed filters (Capture, markers, filters below). The R2 → R1 reply is symmetric — its impairment points are `gd9e8b6a3fe0p0`'s egress (entering R2) and `gd7c1d4e2fe0p0`'s egress (entering R1) — so every direction is impaired exactly once, and `delay 100` measures ≈200 ms round trip (the e2e-verified number). Copies 1-2 are the whole kernel/user bill — both are the TAP-fd legs of the emulation itself; the same frame on the relay datapath crosses six times (diagram above). The dashed sideband is uBridge's observation: capture and markers run as AF_PACKET taps on whichever end hosts them, cloning this frame at that anchor's stage — gdA's rx or gdB's tx — and a clsact-dropped frame never reaches the tap.
## Datapath selection
`UDPLink._kernel_datapath_eligible` extends the common rules with:
-* the compute must report **`ubridge_tap`** — the anchors are uBridge-created
- TAPs the hypervisor merely opens, the same capability QEMU gates on;
- there is no Dynamips-specific one;
-* **both ports must be Ethernet**. The controller's port matrix decides
- per port; serial, ATM and POS ports keep the relay whatever the
- capabilities (the adapter registry in
- `compute/dynamips/adapters/adapter.py` names the Ethernet models —
- `ETHERNET_ADAPTERS` / `ETHERNET_WICS`, WIC-1ENET included, at its
- Dynamips port number `16 * (wic_slot + 1)`).
+* the compute must report **`ubridge_tap`** — the anchors are uBridge-created TAPs the hypervisor merely opens, the same capability QEMU gates on; there is no Dynamips-specific one;
+* **both ports must be Ethernet**. The controller's port matrix decides per port; serial, ATM and POS ports keep the relay whatever the capabilities (the adapter registry in `compute/dynamips/adapters/adapter.py` names the Ethernet models — `ETHERNET_ADAPTERS` / `ETHERNET_WICS`, WIC-1ENET included, at its Dynamips port number `16 * (wic_slot + 1)`).
+
+## Link creation sequence
+
+Both routers sit on the same compute (a kernel-link precondition) and each drives its own uBridge process; the hypervisor is one process per router. The two node handlers attach concurrently and compute the same bridge name independently (a pure function of the link id), so either may win the `brctl create` race — the loser verifies the bridge exists (`brctl show`) and continues:
+
+```mermaid
+sequenceDiagram
+ participant API as REST API
+ participant C as Controller
+ participant R1 as Router R1
+ participant HY1 as Hypervisor R1
+ participant UB1 as uBridge R1
+ participant R2 as Router R2
+ participant HY2 as Hypervisor R2
+ participant UB2 as uBridge R2
+
+ API->>C: POST /links
+ C->>C: eligibility check
+ C->>C: bridge = gns3 + link_id[:11]
+ C--)R1: nio_bridge
+ C--)R2: nio_bridge (same bridge)
+ par Router R1
+ R1->>HY1: nio create_tap
+ R1->>HY1: vm slot_add_nio_binding
+ R1->>HY1: vm slot_enable_nio
+ R1->>UB1: brctl create
+ R1->>UB1: link set gns35d3c2b1a4e6 up
+ R1->>UB1: brctl addif gns35d3c2b1a4e6 gdA
+ R1->>UB1: capture start_kernel gdA
+ R1->>UB1: marker add_kernel gdA
+ R1->>UB1: tc netem set gdA
+ R1->>UB1: link set gdA up
+ and Router R2
+ R2->>HY2: nio create_tap
+ R2->>UB2: same on gdB
+ end
+ Note over HY1,UB2: steady state — kernel forwards between the anchors, the hypervisors hold the tap fds
+```
+
+`nio create_tap`, `vm slot_add_nio_binding` and `vm slot_enable_nio` are the Dynamips-specific steps: the hypervisor opens the anchor and binds it to the slot/port before any bridge work, so a half-wired port never appears on the link. Capture, markers and `tc netem set` run only when the NIO carries them — a marker's real form is `marker add_kernel ""`, `bpf` expressions arrive as `tc bpf_drop add`, the stateful modes as `tc nth/quota/window_drop` — and `brctl addif` is itself what opens the port's `group_fwd_mask` to `0xfffd` (no separate command exists). The anchor's carrier stays down while the link is suspended. A link created while the router is **stopped** is stored and wired by the next start — the NIO simply waits in the slot adapter (deferred wiring, Anchor lifecycle below).
## Anchor lifecycle
-* **Birth (node start)** — `_prepare_tap_datapath` probes the tap module
- (one create/delete cycle) and creates one persistent TAP per Ethernet
- slot/port, ownership handed to the server user, born DOWN. A stale
- leftover from a previous run is swept first; an anchor that already
- exists (restart) is kept — an anchor is never recreated under a live
- link. Serial/ATM/POS ports simply have no anchor.
-* **Life** — a kernel link opens the anchor in the hypervisor
- (`nio create_tap`), binds it to the slot/port and enslaves it into the
- per-link bridge; removing the link unbinds the port, deletes the
- hypervisor's TAP NIO (releasing the fd) and tears the anchor out of the
- bridge. A link bound while the router is **stopped** is stored and wired
- by the next start (deferred wiring — the NIO simply waits in the slot
- adapter).
-* **Death (node close)** — `_stop_ubridge` runs in **QEMU's order**, not
- IOU's: here the Dynamips hypervisor holds the anchor fds, so its TAP
- NIOs go first (`nio delete` closes the fd), then `tap delete` per anchor,
- then the per-link kernel bridges, then uBridge itself.
+* **Birth (node start)** — `_prepare_tap_datapath` probes the tap module (one create/delete cycle) and creates one persistent TAP per Ethernet slot/port, ownership handed to the server user, born DOWN. A stale leftover from a previous run is swept first; an anchor that already exists (restart) is kept — an anchor is never recreated under a live link. Serial/ATM/POS ports simply have no anchor.
+* **Life** — a kernel link opens the anchor in the hypervisor (`nio create_tap`), binds it to the slot/port and enslaves it into the per-link bridge; removing the link unbinds the port, deletes the hypervisor's TAP NIO (releasing the fd) and tears the anchor out of the bridge. A link bound while the router is **stopped** is stored and wired by the next start (deferred wiring — the NIO simply waits in the slot adapter).
+* **Death (node close)** — `_stop_ubridge` runs in **QEMU's order**, not IOU's: here the Dynamips hypervisor holds the anchor fds, so its TAP NIOs go first (`nio delete` closes the fd), then `tap delete` per anchor, then the per-link kernel bridges, then uBridge itself.
-**A stop is not a close.** `stop` only halts the emulated router: the
-hypervisor process — and with it the TAP fds, the port bindings and the
-per-link bridges — survives `vm stop` (the same reason a relay link's
-tunnel always survived a stop). Links therefore keep working through a
-stop/start with no re-wiring at all; the restart's attach pass skips every
-port that still has its TAP NIO. If the hypervisor died mid-flight (crash
-or kill), the persistent TAPs remain and the next start re-attaches from
-the stored NIOs.
+**A stop is not a close.** `stop` only halts the emulated router: the hypervisor process — and with it the TAP fds, the port bindings and the per-link bridges — survives `vm stop` (the same reason a relay link's tunnel always survived a stop). Links therefore keep working through a stop/start with no re-wiring at all; the restart's attach pass skips every port that still has its TAP NIO. If the hypervisor died mid-flight (crash or kill), the persistent TAPs remain and the next start re-attaches from the stored NIOs.
## Link operations
@@ -86,78 +197,35 @@ the stored NIOs.
## Capture, markers, filters
-Capture on a kernel link is `capture start_kernel `; markers ride
-the base reconcile with the anchor as the location (`marker add_kernel` —
-the shared KernelDatapathMixin machinery, no Dynamips-specific code);
-filters are one tc netem qdisc per anchor plus cls_bpf/eBPF drops. A relay
-port keeps its Dynamips NIO and the node's uBridge relay bridge
-(`DYNAMIPS-*`) with userspace filters, exactly as before.
+Capture on a kernel link is `capture start_kernel `; markers ride the base reconcile with the anchor as the location (`marker add_kernel` — the shared KernelDatapathMixin machinery, no Dynamips-specific code); filters are one tc netem qdisc per anchor plus cls_bpf/eBPF drops. A relay port keeps its Dynamips NIO and the node's uBridge relay bridge (`DYNAMIPS-*`) with userspace filters, exactly as before.
## Link-local frames (LACP, LLDP, 802.1X, STP)
-The bridge stands in for a cable, but the kernel's `br_handle_frame()` does
-not forward the IEEE 802.1D reserved range (`01:80:c2:00:00:00`-`0f`) by
-default — LACP, LLDP/DCBX and 802.1X would never cross a kernel link while
-the UDP relay carries them fine. uBridge therefore opens every port's
-per-port `group_fwd_mask` (`IFLA_BRPORT_GROUP_FWD_MASK`) to `0xfffd` at
-`brctl addif` time (Linux 4.15+, best-effort on older kernels). The
-bridge-level knob cannot do this: `BR_GROUPFWD_RESTRICTED` rejects bits 0-2,
-so LACP is reachable per port only.
+The bridge stands in for a cable, but the kernel's `br_handle_frame()` does not forward the IEEE 802.1D reserved range (`01:80:c2:00:00:00`-`0f`) by default — LACP, LLDP/DCBX and 802.1X would never cross a kernel link while the UDP relay carries them fine. uBridge therefore opens every port's per-port `group_fwd_mask` (`IFLA_BRPORT_GROUP_FWD_MASK`) to `0xfffd` at `brctl addif` time (Linux 4.15+, best-effort on older kernels). The bridge-level knob cannot do this: `BR_GROUPFWD_RESTRICTED` rejects bits 0-2, so LACP is reachable per port only.
-Result on kernel links: ordinary multicast, STP/RSTP, LACP, 802.1X and
-LLDP/DCBX all cross; **802.3x PAUSE / PFC does not** — `case 0x01` in
-`br_handle_frame()` drops it unconditionally and no mask can enable it.
-That is a documented limit, not a regression (PFC's hardware semantics are
-out of emulation's reach anyway); a lab that needs PAUSE frames on the wire
-must use a relay link. Verified live by
-`tests/e2e/test_docker_link_local_frames.py` (per-MAC guest-to-guest
-matrix); the contract and kernel references live in
-`docs/design/ubridge-link-local-frame-forwarding-spec.md`.
+Result on kernel links: ordinary multicast, STP/RSTP, LACP, 802.1X and LLDP/DCBX all cross; **802.3x PAUSE / PFC does not** — `case 0x01` in `br_handle_frame()` drops it unconditionally and no mask can enable it. That is a documented limit, not a regression (PFC's hardware semantics are out of emulation's reach anyway); a lab that needs PAUSE frames on the wire must use a relay link. Verified live by `tests/e2e/test_docker_link_local_frames.py` (per-MAC guest-to-guest matrix); the contract and kernel references live in `docs/design/ubridge-link-local-frame-forwarding-spec.md`.
## Relay fallback
-A uBridge without the tap module keeps the node relay-only: `_prepare_tap_datapath`
-warns, no anchors are created, the warning says the node cannot carry
-kernel links, and every link keeps its `nio create_udp` tunnel. The
-controller never offers such a node a kernel link (capability unreported).
+A uBridge without the tap module keeps the node relay-only: `_prepare_tap_datapath` warns, no anchors are created, the warning says the node cannot carry kernel links, and every link keeps its `nio create_udp` tunnel. The controller never offers such a node a kernel link (capability unreported).
## Verified
-Unit level (`tests/compute/dynamips/test_dynamips_kernel_datapath.py`):
-anchor topology (Ethernet models and WIC-1ENET port numbering only), the
-lifecycle (create/sweep/skip-existing), deferred wiring of NIOs bound
-while stopped, attach/update/remove command shapes on both datapaths, the
-stop order (hypervisor NIO delete before tap delete before brctl delete),
-capture and marker shapes, hot-added adapters, and the manager's
-`nio_bridge` construction. Plus the controller-eligibility cases (mixable
-with docker/qemu/iou, tap capability required, serial excluded) in
-`tests/controller/test_kernel_datapath_link.py`.
+Unit level (`tests/compute/dynamips/test_dynamips_kernel_datapath.py`): anchor topology (Ethernet models and WIC-1ENET port numbering only), the lifecycle (create/sweep/skip-existing), deferred wiring of NIOs bound while stopped, attach/update/remove command shapes on both datapaths, the stop order (hypervisor NIO delete before tap delete before brctl delete), capture and marker shapes, hot-added adapters, and the manager's `nio_bridge` construction. Plus the controller-eligibility cases (mixable with docker/qemu/iou, tap capability required, serial excluded) in `tests/controller/test_kernel_datapath_link.py`.
-End-to-end (`tests/e2e/test_dynamips_kernel_datapath.py`, pytest marker
-`e2e`) on a live server with **two real c7200 routers on a real IOS image**,
-driven through the REST API and the IOS consoles with real ICMP:
+End-to-end (`tests/e2e/test_dynamips_kernel_datapath.py`, pytest marker `e2e`) on a live server with **two real c7200 routers on a real IOS image**, driven through the REST API and the IOS consoles with real ICMP:
-* a link created while both routers are stopped selects the kernel
- datapath and is wired by node start (deferred wiring);
-* the per-link kernel bridge on the host has exactly the two anchor TAPs
- enslaved; `show ip int brief` is up/up and ping R1→R2 succeeds;
-* `delay 100` ⇒ netem qdisc visible on both anchors and the ping RTT
- grows by ~200 ms (one-way 100 per direction, each anchor impairing its
- own ingress side); clearing resets the qdiscs and the RTT;
+* a link created while both routers are stopped selects the kernel datapath and is wired by node start (deferred wiring);
+* the per-link kernel bridge on the host has exactly the two anchor TAPs enslaved; `show ip int brief` is up/up and ping R1→R2 succeeds;
+* `delay 100` ⇒ netem qdisc visible on both anchors and the ping RTT grows by ~200 ms (one-way 100 per direction, each anchor impairing its own ingress side); clearing resets the qdiscs and the RTT;
* suspend ⇒ anchor administratively down, ping 0%; resume restores both;
* capture writes a pcap full of ICMP-over-Ethernet records;
-* a **serial** link between the same two routers stays on the relay
- (per-port exclusion) and pings over it — the relay regression net;
-* link delete/re-create: the bridge goes and comes back, the anchors
- survive, traffic resumes;
-* router stop/start keeps the wiring intact (bridge membership unchanged,
- no duplicate NIOs) and traffic resumes after the reboot;
+* a **serial** link between the same two routers stays on the relay (per-port exclusion) and pings over it — the relay regression net;
+* link delete/re-create: the bridge goes and comes back, the anchors survive, traffic resumes;
+* router stop/start keeps the wiring intact (bridge membership unchanged, no duplicate NIOs) and traffic resumes after the reboot;
* deleting the project removes every anchor and bridge.
-`test_dynamips_relay_control` is the negative control: the same topology on
-an isolated relay-configured instance (`enable_kernel_datapath = false`)
-has no per-link bridge and nothing enslaved, and still pings — so the
-kernel objects above can only come from the kernel datapath.
+`test_dynamips_relay_control` is the negative control: the same topology on an isolated relay-configured instance (`enable_kernel_datapath = false`) has no per-link bridge and nothing enslaved, and still pings — so the kernel objects above can only come from the kernel datapath.
## Configuration
@@ -170,9 +238,4 @@ enable_kernel_datapath = True
## Roadmap
-The Ethernet switch has landed (see `ethernet-switch-kernel-datapath.md` —
-it absorbs the peer's anchor into its own kernel bridge; switch-to-switch
-cascades and the IOL runner container have landed too — see
-`ethernet-switch-kernel-datapath.md` and
-`iol-docker-kernel-datapath.md`). Cross-compute kernel links need
-VXLAN/GENEVE encapsulation — deferred until every node type is kernelized.
+The Ethernet switch has landed (see `ethernet-switch-kernel-datapath.md` — it absorbs the peer's anchor into its own kernel bridge; switch-to-switch cascades and the IOL runner container have landed too — see `ethernet-switch-kernel-datapath.md` and `iol-docker-kernel-datapath.md`). Cross-compute kernel links need VXLAN/GENEVE encapsulation — deferred until every node type is kernelized.
diff --git a/docs/features/iol-docker-kernel-datapath.md b/docs/features/iol-docker-kernel-datapath.md
index 4df8974a6..2a544d3f5 100644
--- a/docs/features/iol-docker-kernel-datapath.md
+++ b/docs/features/iol-docker-kernel-datapath.md
@@ -121,22 +121,53 @@ Drawn one-way; the reverse direction is the mirror image. The IOL relay crosses
Two further contrasts with Docker's relay: the IOL relay's local leg is the unix NIO itself, so the anchors never appear on the path (idle from birth — a Docker relay attaches to the *same* veth host end the kernel link enslaves); and the port bridge's first slot is shared with the kernel datapath — the unix NIO and its c-socket binding survive every leg swap, so frames the container sends while the bridge is stopped for a swap queue on the socket and relay after `start`. On this datapath suspend rides the synthetic frequency_drop filter (the kernel datapath's suspend is the anchor's admin state).
-Frame path for one A → B frame. The impairment chain fires on the egress of the *destination* anchor, in order clsact → netem (a classifier drop means netem never sees the frame); each direction is impaired exactly once and B → A is symmetric on gxA — hence `delay 100` doubles the RTT (measured ≥ 150 ms):
+Frame path for one A → B frame, with the relay diagram's copy numbering: copies 1, 2, 5 and 6 are the socket guest legs and are paid identically on both datapaths — only the middle two swap (TAP fd write/read here, loopback UDP there). The impairment chain fires on the egress of the *destination* anchor, in order clsact → netem (a classifier drop means netem never sees the frame); each direction is impaired exactly once and B → A is symmetric on gxA's egress chain — hence `delay 100` doubles the RTT (measured ≥ 150 ms):
```mermaid
flowchart LR
- S1["IOL A sends on Eth0/0"] --> S2["netiomux writes c00.sock"]
- S2 --> S3["port bridge A relay"]
- S3 --> S4["gxA ingress"]
- S4 --> G1{"reserved 01:80:c2?"}
- G1 -->|"PAUSE 0x01"| D["dropped"]
- G1 -->|"0xfffd passes"| S5["FDB lookup / flood"]
- S5 --> S6["clsact: eBPF prio 1"]
- S6 --> S7["clsact: bpf_drop prio 10-99"]
- S7 --> S8["netem qdisc"]
- S8 --> S9["port bridge B relay"]
- S9 --> S10["netiomux B via s00.sock"]
- S10 --> S11["IOL B receives on Eth0/0"]
+ subgraph CTA["IOL A container — userspace"]
+ S1["IOL A sends on Eth0/0"]
+ end
+ subgraph KERN["kernel"]
+ SOCKA["c00.sock"]
+ SOCKB["s00.sock"]
+ end
+ subgraph UBA["uBridge port bridge A — userspace"]
+ S2["relay"]
+ end
+ subgraph GXA["gxA — ingress (egress chain serves B→A)"]
+ S4["ingress"]
+ end
+ subgraph BRP["per-link bridge"]
+ G1{"reserved 01:80:c2?"}
+ D["dropped"]
+ S5["FDB lookup / flood"]
+ end
+ subgraph GXB["gxB — egress chain (clsact, then netem)"]
+ S6["eBPF prio 1"]
+ S7["bpf_drop prio 10-99"]
+ S8["netem qdisc"]
+ end
+ subgraph UBB["uBridge port bridge B — userspace"]
+ S9["relay"]
+ end
+ subgraph CTB["IOL B container — userspace"]
+ S11["IOL B receives on Eth0/0"]
+ end
+ tap["capture / markers"]
+ S1 -->|"copy 1: user→kernel"| SOCKA
+ SOCKA -->|"copy 2: kernel→user"| S2
+ S2 -->|"copy 3: user→kernel"| S4
+ S4 --> G1
+ G1 -->|"PAUSE 0x01"| D
+ G1 -->|"0xfffd passes"| S5
+ S5 --> S6
+ S6 --> S7 --> S8
+ S8 -->|"copy 4: kernel→user"| S9
+ S9 -->|"copy 5: user→kernel"| SOCKB
+ SOCKB -->|"copy 6: kernel→user"| S11
+ S4 -.->|"AF_PACKET tap"| tap
+ S8 -.->|"AF_PACKET tap"| tap
```
The reserved-range gate passes LACP, LLDP, 802.1X and STP per the port mask but never PAUSE/PFC (Link-local frames below); the clsact/netem stages drop or impair per the installed filters (Capture, markers, filters below). The AF_PACKET taps observe both anchors but never a clsact-dropped frame.
diff --git a/docs/features/iou-kernel-datapath.md b/docs/features/iou-kernel-datapath.md
index 60844968b..5dbeeaf41 100644
--- a/docs/features/iou-kernel-datapath.md
+++ b/docs/features/iou-kernel-datapath.md
@@ -115,10 +115,8 @@ flowchart LR
subgraph UBA["uBridge A — userspace"]
S2["relay"]
end
- subgraph GIA["giA — ingress"]
+ subgraph GIA["giA — ingress (its egress chain serves B→A)"]
S4["ingress"]
- chA["clsact + netem chain"]
- tapA["AF_PACKET tap: capture + markers"]
end
subgraph BRP["per-link bridge"]
G1{"reserved 01:80:c2?"}
@@ -128,7 +126,7 @@ flowchart LR
subgraph GIB["giB — egress chain (clsact, then netem)"]
S6["eBPF prio 1"]
S7["bpf_drop prio 10-99"]
- S8["netem"]
+ S8["netem qdisc"]
end
subgraph UBB["uBridge B — userspace"]
S9["relay"]
@@ -147,8 +145,9 @@ flowchart LR
S8 -->|"copy 4: kernel→user"| S9
S9 -->|"copy 5: user→kernel"| mbxB
mbxB -->|"copy 6: kernel→user"| S11
- S4 -.->|"serves B→A"| chA
- S4 -.->|"observes tx + rx"| tapA
+ tap["capture / markers"]
+ S4 -.->|"AF_PACKET tap"| tap
+ S8 -.->|"AF_PACKET tap"| tap
```
What each stage drops: the reserved-range gate passes LACP, LLDP, 802.1X and STP per the port mask but never PAUSE/PFC (Link-local frames below); the clsact/netem stages drop or impair per the installed filters (Capture, markers, filters below); the AF_PACKET tap — capture and markers ride it, installed at whichever end captures — observes both directions of the link through its anchor (tx/rx signal pairs) but never a clsact-dropped frame. The crossings do not disappear on this datapath: each leg still crosses the kernel/user boundary three times (`sendto`/`recvfrom` on the fabric, then the TAP write — the B leg mirrors them, six per one-way frame, the same count as the relay); what moves into the kernel is the link segment — no UDP tunnel, no protocol-stack pass — and with it the impairment chain. A fabric → anchor → bridge → anchor → fabric round trip measures ≈0.2 ms (Verified below).
diff --git a/docs/features/qemu-kernel-datapath.md b/docs/features/qemu-kernel-datapath.md
index 84dc8bf5f..703bd1164 100644
--- a/docs/features/qemu-kernel-datapath.md
+++ b/docs/features/qemu-kernel-datapath.md
@@ -139,8 +139,9 @@ flowchart LR
subgraph GQB["gqB — egress chain (clsact, then netem)"]
S5["eBPF prio 1"]
S6["bpf_drop prio 10-99"]
- S7["netem"]
+ S7["netem qdisc"]
end
+ T["capture / markers"]
end
subgraph QPB["QEMU B — userspace"]
S8["tap fd to QEMU"]
@@ -149,17 +150,19 @@ flowchart LR
S9["NIC receives"]
end
S1 -->|"virtio"| S2
- S2 --> S3
+ S2 -->|"tap fd, copy 1: user→kernel"| S3
S3 --> G1
G1 -->|"PAUSE 0x01"| D
G1 -->|"0xfffd passes"| S4
S4 --> S5
S5 --> S6 --> S7
- S7 --> S8
+ S7 -->|"tap fd, copy 2: kernel→user"| S8
S8 -->|"virtio IRQ"| S9
+ S3 -. "AF_PACKET tap" .-> T
+ S7 -. "AF_PACKET tap" .-> T
```
-What each stage drops: the reserved-range gate passes LACP, LLDP, 802.1X and STP per the port mask but never PAUSE/PFC (Link-local frames below); the clsact/netem stages drop or impair per the installed filters (Capture, markers, filters below). The AF_PACKET taps observe both anchors but never a clsact-dropped frame.
+What each stage drops: the reserved-range gate passes LACP, LLDP, 802.1X and STP per the port mask but never PAUSE/PFC (Link-local frames below); the clsact/netem stages drop or impair per the installed filters (Capture, markers, filters below). The AF_PACKET taps observe both anchors but never a clsact-dropped frame. Copies 1–2 are the whole path's only kernel/user crossings — both halves of the emulation's own TAP-fd leg (QEMU A's write, QEMU B's read); every stage between them runs in the kernel.
## Why a TAP works like a veth host end