diff --git a/docs/design/ubridge-iol-tap-anchor-spec.md b/docs/design/ubridge-iol-tap-anchor-spec.md new file mode 100644 index 000000000..172a656b9 --- /dev/null +++ b/docs/design/ubridge-iol-tap-anchor-spec.md @@ -0,0 +1,268 @@ + + +> Frozen requirements spec for the **uBridge** project. Status: **delivered +> and verified from the server side** — implemented on the +> `feature/iol-tap-anchor` branch of the ubridge fork (both commands, the §B +> hardening, and one fix beyond the spec: `iol_bridge delete` now releases +> every port NIO even when the bridge was not running, so a stopped node's +> anchor fds never leak). The server side landed on +> `feat/iou-kernel-datapath` and passed a live e2e (64/64 kernel, 62/62 +> relay; see `docs/features/iou-kernel-datapath.md`). +> +> Notes from the implementation: the two new commands answer `208` for a +> missing bridge where the older `add_nio_udp` answered `214` (no client +> branches on the code), and a write to an admin-DOWN anchor logs one +> `perror` line per dropped frame. +> +> Scope guard: no wire format change, no change to existing commands or +> replies. This adds one NIO type to the existing `iol_bridge` module (two +> commands) plus one error-tolerance fix in the two listener threads. The +> `tap`, `brctl`, `link`, `tc`, `marker` and `capture` modules are reused +> as-is. + +# IOL bridge TAP anchor ports (`iol_bridge add_nio_tap`) + +## 0. Background & scope + +### What IOU's physical layer is + +An IOU instance has no host netdev. Its only wire is the Unix-datagram +fabric in `/tmp/netio/`: frames carry an 8-byte IOL header (dst id, +src id, dst port, src port, message type, channel), and the NETMAP file +maps every `bay/unit` of the instance to a second, fake instance — which +today is uBridge itself (`iol_bridge create `, +binding `/tmp/netio/`). Each port of that fake +instance terminates on a **destination NIO**, and the only destination +NIO type is UDP (`iol_bridge add_nio_udp`): one listener thread per port +relays fabric → NIO, and one bridge listener relays fabric-side reads → +NIO sends. + +### The problem + +With UDP as the only termination, every IOU link is a uBridge userspace +relay leg: + +- an IOU port cannot join a per-link kernel bridge, so IOU ↔ Docker/QEMU + traffic always crosses through a UDP relay even when both ends sit on + the same compute; +- kernel-class impairments (`tc netem`, the clsact eBPF classifiers) and + the `*_kernel` capture/marker commands cannot anchor on an IOU port — + only the userspace packet filters apply, whose delay implementation is + the known blocking-nanosleep defect that inflates RTT under load; +- suspend of a single link has no clean mapping. + +### What this change is — and what it cannot be + +Give the port a second destination NIO type: a **persistent TAP, +created by gns3-server with the existing `tap create` command and opened +(held) by uBridge**. uBridge's iol_bridge then plays, on that TAP, the +same role the QEMU process plays on its TAP: it is the device's +userspace endpoint. Everything the anchor model already delivers — +`brctl addif` enslavement, `tc` on the anchor, `capture start_kernel`, +`marker add_kernel`, `link set up/down` for carrier/suspend — is keyed +on the interface name and does not care who holds the fd. + +One userspace hop on the IOU leg is irreducible: the Unix fabric is +IOU's only physical layer, so frames always pass +fabric ↔ listener thread ↔ TAP fd. What kernelization buys is that the +**link segment itself** is the kernel: anchor → per-link bridge → peer +anchor, with kernel filters, markers and capture on the anchors, and +suspend mapped to anchor admin-down. That is the accepted design, not a +deficiency of this spec. + +### Goal + +`iol_bridge` ports can terminate on a pre-existing persistent TAP +anchor, with the same lifecycle semantics as `add_nio_udp` (attach +before or after `iol_bridge start`, replace a previous NIO, delete and +re-add), and the relay survives — without exiting — the anchor being +administratively DOWN, which is a steady state of this design (port +anchored but no link attached; link suspended). + +Explicitly **not** in scope: + +- creating or deleting the TAP device itself — the server owns the + anchor lifecycle through the existing `tap` module (`tap create`, + `tap delete`; the device is created hardened per the L2-only spec); +- serial bays — a TAP carries Ethernet frames; serial links stay on + `add_nio_udp`; +- an MTU command (`link set mtu `) — does not exist today and is + not required by default-MTU IOU traffic; noted as future work if + jumbo frames ever matter; +- any change to the NETMAP, the IOL header encoding, or the L1 + keepalive protocol. + +## A. New commands + +Belong to the existing `iol_bridge` module, next to their UDP +counterparts. `bay`/`unit` and the IOL header handling are identical to +`add_nio_udp`. + +### `iol_bridge add_nio_tap ` (6 args) + +| Arg | Description | +|-----|-------------| +| `` | An existing IOL bridge (`208` if not) | +| `` | The real instance's application id, encoded into the IOL header as with `add_nio_udp`; the existing "`iol_id` == bridge id" refusal in `create_iol_port_entry` applies unchanged | +| ``, `` | Port coordinates, same encoding and MAX_PORTS check as `add_nio_udp` | +| `` | A **pre-existing** network device (`208` if not, see below); ≤ IFNAMSIZ−1 chars (`204` otherwise) | + +Semantics: + +1. **Require an existing device.** Resolve `` with + `if_nametoindex` first and refuse with `208` when absent. This is the + same reasoning as the `tap` module's `tap_require_existing`: without + the check, `TUNSETIFF` silently *creates a transient* TAP that dies + with the fd — an anchor that vanishes on detach is worse than a + clear error. +2. Open the device with the existing `create_nio_tap()` (by-name + `TUNSETIFF`, `IFF_TAP | IFF_NO_PI`, default carrier). No owner or + persistence ioctl — the device is persistent and owned already. +3. Install it through the existing `create_iol_port_entry()`, which is + generic over `nio_t *`: it frees whatever NIO the port held before + (listener stop guarded on `tid`, delay lines torn down under + `iol_delay_lock`, capture/filters freed), pre-computes the IOL + header, and starts the port listener if the bridge is running. A + port may therefore swap `add_nio_udp` ↔ `add_nio_tap` repeatedly — + that is the datapath-switch path and must keep working. +4. If the fd cannot be opened (`create_nio_tap` fails), reply `206` and + free the nio as the UDP handler does. + +No `carrier` argument: administrative state is the server's lever, via +the existing `link set up|down`. + +### `iol_bridge delete_nio_tap ` (3 args) + +Same body as `cmd_delete_nio_udp`, including its thread-safety guards: +stop the port listener (cancel guarded on `tid != 0`, join), destroy +both delay lines under `iol_delay_lock`, free capture, filters and the +NIO (which closes the TAP fd). **The persistent device itself survives** +— the server deletes it with `tap delete` when the node stops, exactly +as it does for QEMU anchors. Deleting a port that holds no NIO is a +no-op `100` (mirrors `delete_nio_udp`). + +Replies: + +``` +iol_bridge add_nio_tap IOL-BRIDGE-513 1 0 0 gi0badfe9e0p0 +100-NIO TAP added to IOL bridge 'IOL-BRIDGE-513' +iol_bridge delete_nio_tap IOL-BRIDGE-513 0 0 +100-NIO TAP deleted from IOL bridge 'IOL-BRIDGE-513' +``` + +## B. Required hardening: the anchor can be administratively DOWN + +A TAP anchor is expected to sit admin-DOWN for long periods (port +anchored, no link attached yet; or the link suspended). The listeners +must treat that as normal: + +1. **`iol_bridge_listener`, NIO send path** (IOL → TAP, `nio->send` = a + `write` on the fd): a write to an admin-DOWN TAP fails with **EIO**. + The current tolerated set is `ECONNREFUSED | ENETDOWN | EINVAL`, + anything else is `exit(EXIT_FAILURE)` — which would take the whole + uBridge down the first time a suspended link carries traffic. Add + `EIO` to the tolerated set: drop the frame (optionally a debug log) + and continue. +2. **`iol_nio_listener`, NIO receive path** (TAP → IOL, `nio_recv` = a + `read` on the fd): while the device is DOWN the read blocks (no + carrier), which is fine — but a return of `0` must not be counted or + forwarded (it would fabricate header-only packets for the instance). + Treat `recv <= 0` as continue; treat `-1` with `EIO` as continue + alongside the existing `ECONNREFUSED | ENETDOWN`. + +No other listener behavior changes: the fabric-side `sendto` already +tolerates `ENOENT`/`ECONNREFUSED` for the window where the IOU process +has not created its endpoint yet, and that stays. + +## C. Implementation notes + +- The command handlers are thin: existence check + `create_nio_tap` + + `create_iol_port_entry` (add), and the `cmd_delete_nio_udp` body + (delete). The three copies of that teardown block in the module are a + pre-existing pattern — a shared helper is welcome but not required by + this spec. +- Do **not** route the TAP through `nio_tap_open`'s `/dev/net/tun` + path-name variant; the anchor is addressed by interface name. +- Statistics (`iol_bridge get_stats`) work unchanged: the counters hang + off the generic nio and both listeners already update them. +- Port filters, delay lines, capture (`iol_bridge add_packet_filter`, + `start_capture`, …) keep working on a TAP-terminated port; gns3-server + simply will not combine userspace filters with `tc` on the same port + (see the alignment section). + +## D. Reply contract & error codes + +| Code | Meaning | +|------|---------| +| `100` | OK — NIO added/deleted (delete of an empty port is also `100`) | +| `203` | Bad number of parameters | +| `204` | Invalid parameter (tap name longer than IFNAMSIZ−1) | +| `206` | Unable to create (bridge exists but the TAP fd cannot be opened) | +| `208` | Unknown object — bridge or **tap device** does not exist | + +Old builds answer `202-Unknown command` for both commands; gns3-server +uses that to fall back to the UDP datapath, mirroring the `tap`-probe +degradation. + +## E. Test requirements + +1. **Both-direction relay.** On a running IOL bridge with one + TAP-terminated port: a frame sent on the fabric for that bay/unit + appears on the TAP netdev (stripped of the IOL header); a frame + injected on the netdev reaches the fabric with the port's correct + 8-byte header. (A scratch Unix-datagram endpoint can play the + instance; no IOU binary is needed.) +2. **No transient device.** `add_nio_tap … nosuchtap0` → `208`, and no + interface is created (`ip -o link` unchanged). +3. **Unknowns.** Unknown bridge → `208`; name ≥ IFNAMSIZ → `204`; a + bridge-id == iol_id → the existing refusal. +4. **Delete keeps the device.** After `delete_nio_tap`, the port has no + NIO (stats/list reflect that), the TAP netdev still exists, and the + port accepts `add_nio_udp` (and re-`add_nio_tap`) afterwards. +5. **Swap semantics.** A port holding a UDP NIO that gets `add_nio_tap` + releases the UDP socket (old listener joined, port usable), and vice + versa — no fd or thread leak after several swaps. +6. **DOWN-anchor resilience.** With the TAP admin-DOWN, drive traffic + from the fabric side for ≥ 100 frames: uBridge stays alive, frames + are dropped, nothing is written to the netdev. After `link set up`, + traffic flows again. This is the regression test for §B.1. +7. **Idle DOWN port is quiet.** A DOWN anchored port produces no + header-only or zero-length fabric sends (§B.2). +8. **Kernel-bridge interop.** Enslave the anchor into a kernel bridge + (`brctl create` + `addif`) while the port relays: frames cross + bridge ↔ fabric through the port, and a peer port on the same bridge + receives them. This is the actual deployment shape. + +## gns3-server alignment (informational — not uBridge scope) + +- **Anchors**: one TAP per Ethernet bay/unit, `gi{node_id[:8]}e{bay}p{unit}`, + created persistent (hardened by `tap create` per the L2-only spec) at + node start, swept for stale leftovers first, `tap delete`d at node + stop — the exact lifecycle QEMU anchors use. All Ethernet ports are + anchored at start ("tap-everywhere"); serial bays never are. +- **Links**: kernel links enslave the anchor via the shared + `KernelDatapathMixin` (`brctl create`/`addif`, `tc`, markers, + `capture start_kernel`, suspend = anchor admin-down). Relay links keep + `add_nio_udp` and never touch the anchor. A port switching datapaths + is just an NIO swap (§A.3). +- **Filters are never double-applied**: on a kernel link only the anchor + `tc` path is used; the port's userspace filters carry relay links + only. +- **Capability gating**: the eligibility check needs to know the command + exists before creating a link. Probe by attempting the operation on a + scratch bridge + scratch TAP (create, `add_nio_tap`, delete, clean + up), cached by binary identity like `probe_tap_support`, reported + through `/capabilities`; an old build answering `202` keeps IOU on + the relay datapath. +- **L1 keepalives**: unchanged — an anchored-but-unconnected port holds + no NIO from the server's perspective, so the responder keeps ignoring + it and IOU sees the port down, which matches the anchor being DOWN. + +## Roadmap note + +`link set mtu ` does not exist today. Default-MTU IOU traffic +does not need it; if jumbo IOU frames ever matter it becomes a small, +separate requirement on the `link` module. diff --git a/docs/features/docker-kernel-datapath.md b/docs/features/docker-kernel-datapath.md index 0be8facef..42ebb763f 100644 --- a/docs/features/docker-kernel-datapath.md +++ b/docs/features/docker-kernel-datapath.md @@ -367,8 +367,12 @@ enable_kernel_datapath = True Intended to grow into the global datapath switch as IOU / Dynamips migrate to kernel bridges. **QEMU has landed** on the same pattern (persistent TAP anchor enslaved into the per-link bridge, zero relay) — see -`docs/features/qemu-kernel-datapath.md`; the shared half of this document's -compute machinery now lives in `gns3server/compute/kernel_datapath.py`. +`docs/features/qemu-kernel-datapath.md` — and **IOU after it** +(`docs/features/iou-kernel-datapath.md`; the fabric terminator in uBridge +gained a TAP-terminated port, `iol_bridge add_nio_tap`, specified in +`docs/design/ubridge-iol-tap-anchor-spec.md`). The shared half of this +document's compute machinery now lives in +`gns3server/compute/kernel_datapath.py`. ## Verification diff --git a/docs/features/iou-kernel-datapath.md b/docs/features/iou-kernel-datapath.md new file mode 100644 index 000000000..b90efc673 --- /dev/null +++ b/docs/features/iou-kernel-datapath.md @@ -0,0 +1,170 @@ + + +# IOU kernel datapath + +IOU's only physical layer is the Unix-datagram IOL fabric (`/tmp/netio/`, +8-byte IOL headers, NETMAP), so it has no host netdev to enslave — a kernel +link instead gives each **Ethernet bay/unit** a persistent TAP anchor +(`gi{node_id[:8]}e{bay}p{unit}`), and uBridge's `iol_bridge` (the fake IOL +instance the NETMAP maps to) binds the port to that TAP with +`iol_bridge add_nio_tap`. Links then attach to the anchor exactly as they do +for Docker (veth) and QEMU (TAP): a per-link Linux bridge enslaves it, tc +impairments run on it, capture and markers anchor on it. The uBridge +capability is specified in `docs/design/ubridge-iol-tap-anchor-spec.md` and +delivered by the `feature/iol-tap-anchor` branch of the ubridge fork. + +Serial bays never anchor (a TAP carries Ethernet frames): serial links stay +on the relay datapath whatever the compute's capabilities. + +## Architecture + +``` + IOU process root namespace + ┌──────────┐ unix datagram ┌──────────────────────────────┐ + │ IOL │◄──── /tmp/netio ──►│ uBridge iol_bridge (listener)│ + │ fabric │ (8-byte hdr) │ │ holds the TAP fd │ + └──────────┘ │ ▼ │ + │ gi{id}e{bay}p{unit} ──────┼──► gns3{link_id} + │ (persistent TAP anchor) │ (per-link kernel + └──────────────────────────────┘ bridge port) +``` + +One userspace hop on the IOU leg is **irreducible** — the fabric is Unix +sockets, so frames always cross fabric → listener thread → TAP fd. What the +kernel datapath buys is the **link segment**: anchor → per-link bridge → +peer anchor crosses entirely in the kernel (including IOU ↔ Docker/QEMU +links), impairments run as kernel tc instead of the userspace delay line +(the known blocking-nanosleep RTT defect), and suspend maps to anchor +admin-down (writes to a down TAP fail EIO and are dropped by the delivered +§B hardening). Direction semantics on the anchor match QEMU's exactly: +uBridge holds the fd like the QEMU process does. + +## Datapath selection + +`UDPLink._kernel_datapath_eligible` extends the Docker/QEMU rules with: + +* the compute must report **`ubridge_iol_tap`** (its own capability: the tap + module and `iol_bridge add_nio_tap` land independently, and either can be + missing on an old build). Probed by running one add/delete cycle on a + scratch bridge + scratch TAP (`probe_iol_tap_support`, cached by binary + identity, scratch bridge id 1050 — above the per-node id space 513..1024 + the fabric locks); any failure means unknown → relay; +* **both linked ports must be Ethernet** — an IOU serial port disqualifies + the kernel path per-port, not per-node, so serial links between capable + IOU nodes silently stay on the relay instead of failing at the compute. + +The NIO schema on the compute NIO routes accepts `nio_bridge` alongside the +UDP/TAP/Ethernet NIOs (the same widening QEMU needed — without it the +controller's NIO POST is 422'd before the node ever sees it). + +## Anchor lifecycle — and why the stop order is QEMU's reverse + +* **Birth (node start)** — `_prepare_tap_datapath` probes the capability and + creates one persistent TAP per Ethernet bay/unit (4 units per bay, swept + for stale leftovers first, born DOWN). uBridge holds a device's fd only + while a kernel link binds the port; until then the TAP has no holder. +* **Life** — the TAP is never recreated. A kernel link adds + `iol_bridge add_nio_tap ` then + enslaves the anchor (`_kernel_attach`); a relay link keeps + `iol_bridge add_nio_udp` and never touches the anchor. Switching a port + between datapaths is just an NIO swap — the IOL port frees its previous + NIO (fd or socket) on attach. +* **Death (node stop)** — `_stop_ubridge` runs the cleanup **in the reverse + order of QEMU's**, because here uBridge itself holds the anchor fds: + `iol_bridge delete` (releases every port's TAP fd) → `tap delete` per + anchor (a held TAP answers EBADFD/207) → `brctl delete` for per-link + bridges still held → stop the hypervisor. Removing a single kernel link + does the same in miniature: `_remove_kernel_nio` (markers, tc reset, + delif, brctl delete) then `iol_bridge delete_nio_tap` — releasing the fd + while the control channel lives. + +## Link operations + +| Operation | Kernel-datapath action | +|---|---| +| Create / attach | `iol_bridge add_nio_tap` (bind port ↔ anchor, uBridge holds the fd) + `brctl create` (EEXIST tolerated) + `link set up` + `brctl addif` + carrier up + markers + `tc netem set` | +| Delete / detach | marker teardown → `tc reset ` → carrier off → `brctl delif`/`brctl delete` (last endpoint wins) → `iol_bridge delete_nio_tap` (release the fd; the device survives) | +| Update (filters/markers changed) | reconcile on the anchor only — no re-binding, no re-enslaving | +| Suspend | anchor admin-down (IOL relay writes fail EIO → dropped; reads block) + resume restores | + +## Capture, markers, filters + +Capture on a kernel link is `capture start_kernel ` (driven by the NIO +type); markers ride the base reconcile with the anchor as the location — +IOU's marker primitives are polymorphic (`marker add_kernel/enable_kernel/ +delete_kernel` on an anchor, `iol_bridge … mark` on an IOL location), which +let the old IOU-specific reconcile override be deleted outright. Filters are +one tc netem qdisc per anchor plus cls_bpf/eBPF drops — the port's +userspace `iol_bridge add_packet_filter` filters carry relay links only and +are never combined with tc on the same port. + +## Relay fallback + +A uBridge without `iol_bridge add_nio_tap` (old build) keeps the node on the +relay datapath unchanged: every link rides `iol_bridge add_nio_udp`, the +node starts fine, and the warning says it cannot carry kernel links. The +controller never offers such a node a kernel link (capability unreported). + +## Verified + +Unit level: anchor lifecycle and naming, kernel/relay binding on add, +update and remove, the stop order (iol_bridge delete before tap delete +before brctl delete before hypervisor stop), `_networking` on restart with +mixed kernel/relay ports, capture and marker command shapes on both +datapaths, the capability probe (cycle, old-build fallback, cleanup, +caching), capability plumbing, and controller eligibility (per-compute +capability, its independence from `ubridge_tap`, serial-port exclusion). + +End-to-end on a live server (isolated instance, two IOU nodes behind a fake +ELF image, the test playing the IOL fabric on `/tmp/netio/` and pushing +frames through it): **64/64 checks** on the kernel datapath and **62/62** on +the relay (`enable_kernel_datapath = false`). Kernel highlights: + +* 8 anchors per node (2 bays × 4 units), persistent, DOWN, address-free; + `link_list` reports `kernel_datapath: true` and one per-link bridge holds + both anchors with `brport/state = 3`; +* fabric → anchor → bridge → anchor → fabric in both directions at ~0.2 ms; + `delay 100` ⇒ netem on both anchors and a measured **100.5 ms** one-way; +* suspend ⇒ anchor DOWN, no traffic, resume ⇒ 100.4 ms with the qdisc intact; +* capture ⇒ a 328-byte pcap written by `capture start_kernel`; +* link delete ⇒ bridge gone, qdisc clean, anchor alive, re-create at 0.6 ms; +* a **serial** link between the same two capable nodes stays on the relay + (`kernel_datapath: false`) — per-port exclusion, not per-node; +* single-sided restart ⇒ stopped node's anchors removed, peer's kept, link + re-attached from the NIO (0.3 ms); stop ⇒ no stray anchors, bridges or + fabric sockets. + +The relay run exercises the same lifecycle with the IOL UDP NIOs (no tc +qdiscs, userspace `delay` at 100.7 ms, suspend via the synthetic +frequency_drop, `iol_bridge start_capture`), which is also the regression +net for the port-coordinate capture restore this work touched. + +The uBridge side passed its own 30/30 (`tests/iol/test_tap_anchor.py` in the +fork), including the DOWN-anchor resilience and kernel-bridge interop this +design depends on. + +One pre-existing bug surfaced here and is fixed: the IOU NIO **update** route +never copied the body's `suspend` onto the NIO (Docker and QEMU routes do). +Harmless on the relay, where suspend is emulated by the synthetic +frequency_drop filter, but fatal for a kernel link, whose suspend *is* the +anchor's admin state — a "suspended" link kept forwarding. + +## Configuration + +```ini +[Server] +# Wire eligible links (same compute, both endpoints Ethernet and able to +# anchor) through kernel interfaces instead of the uBridge UDP relay. +enable_kernel_datapath = True +``` + +## Roadmap + +Dynamips next (hypervisor-created taps, enslavable as they are), then the +Ethernet switch (its anchors are ubridge-owned by design) and the IOL Docker +node (same `bridge add_nio_tap` shape as this one). Cross-compute kernel +links need VXLAN/GENEVE encapsulation — deferred until every node type is +kernelized. diff --git a/docs/features/qemu-kernel-datapath.md b/docs/features/qemu-kernel-datapath.md index 1797ca4c6..90e336665 100644 --- a/docs/features/qemu-kernel-datapath.md +++ b/docs/features/qemu-kernel-datapath.md @@ -161,8 +161,15 @@ addresses involved, IPv4 has no per-device IGMP switch). ## Roadmap -IOU (its fabric would need a kernel-side port in `iol_bridge`), Dynamips +IOU (its fabric terminator in uBridge needs a TAP-terminated port — +`iol_bridge add_nio_tap`, frozen in +`docs/design/ubridge-iol-tap-anchor-spec.md`; unlike QEMU, one userspace hop +on the IOU leg is irreducible, the fabric is Unix sockets), Dynamips (hypervisor-created taps, enslavable as they are), and the Ethernet switch / cloud paths (their anchors are ubridge-owned by design) are still on the relay; cross-compute kernel links need VXLAN/GENEVE encapsulation, which is a separate project (it would serve Docker links the same way). + +Update: **IOU has landed** on the same mixin — see +`docs/features/iou-kernel-datapath.md` (Ethernet bays anchor on persistent +TAPs bound to the IOL fabric; serial links stay relay).