From 94b5495683969a3c426893811a5dec677e80ef7d Mon Sep 17 00:00:00 2001 From: YueGuobin Date: Mon, 28 Sep 2026 15:09:03 +0800 Subject: [PATCH] docs: Docker kernel datapath implementation document Cover the four-stage stack (kernel datapath, AF_PACKET capture, markers, veth-everywhere): adapter interface types per node class, datapath selection, veth lifecycle, link operations, suspend/capture/markers semantics, uBridge command surface and the tc netem filters roadmap. Cross-link from the traffic-insight marker doc for the kernel variant. --- docs/features/docker-kernel-datapath.md | 198 ++++++++++++++++++++++++ docs/features/marker-traffic-insight.md | 8 + 2 files changed, 206 insertions(+) create mode 100644 docs/features/docker-kernel-datapath.md diff --git a/docs/features/docker-kernel-datapath.md b/docs/features/docker-kernel-datapath.md new file mode 100644 index 000000000..f3aa181e0 --- /dev/null +++ b/docs/features/docker-kernel-datapath.md @@ -0,0 +1,198 @@ + + +> This documentation is AI-generated with reference to actual code and verified +> against real-kernel testing. AI can make mistakes — please verify against +> the source code when in doubt. + +# Docker Kernel Datapath (veth + per-link Linux bridge) + +## Overview + +Docker node links historically flowed through a uBridge userspace relay: each +adapter was a TAP whose file descriptor uBridge pumped between the container +and UDP tunnels. This implementation replaces that model with **kernel-native +forwarding**: every adapter is born as a **veth pair**, and each link is a +**dedicated Linux kernel bridge** enslaving the two endpoints' veth host ends. +Frames never leave the kernel. + +| | Old (relay) | New (kernel datapath) | +|---|---|---| +| Adapter interface | TAP moved into the container netns | veth pair (guest end in container, host end in root ns) | +| Link | uBridge bridge, TAP fd ↔ UDP, userspace copy | per-link kernel bridge `gns3{link_id[:11]}`, zero-copy | +| Relay latency | ~0.3 ms RTT | ~0.05 ms RTT (measured) | +| Link attach at runtime | yes (relay attaches the pre-existing TAP) | yes (brctl addif / add_nio_ethernet — interface untouched) | +| Packet filters | uBridge userspace filters | not supported yet (tc netem roadmap) → relay fallback | +| Capture / markers | relay bridge | uBridge AF_PACKET modules on the veth host end | + +The work landed in four stages on branch stack `feat/docker-kernel-datapath` → +`feat/docker-kernel-capture` → `feat/docker-kernel-markers` → +`feat/docker-veth-everywhere`: + +1. **Kernel datapath** — NIOBridge NIO type, per-link bridges, carrier-based + suspend, crash-safe reconciliation. +2. **Capture** — `capture start_kernel` (AF_PACKET) on the veth host end. +3. **Markers** — `marker add_kernel` (AF_PACKET + BPF) with the same signals, + pcaps and fine-grained REST operations as the relay `mark` filter. +4. **veth-everywhere** — the TAP path deleted; every adapter is a veth and the + datapath is a *runtime* decision (kernel or relay) that never touches a + running container's interfaces. + +## Adapter interface types + +Which interface a Docker node's adapters use is decided by the node class +(`Docker._select_node_class`, from `console_type` + `GNS3_*` environment +markers): + +| Node class | Selected by | Adapter interface | Kernel links | +|---|---|---|---| +| `DockerVM` | default | veth pair | yes | +| `VendorDockerVM` (non-unix-socket) | `console_type=docker_exec`, `GNS3_SKIP_INIT`, … (XRd, SR Linux, …) | veth pair (same path via `super()`) | yes | +| `IOLDockerVM` | `GNS3_IOL_RUNNER=1` | AF_UNIX datagram socket pairs (`sNN`/`cNN`) — the container netns is unused | rejected (no host-side interface) | +| `VendorDockerVM` (unix-socket) | `GNS3_UNIX_SOCKET_NIO=1` | same socket contract, generic capability for vendor NOS images | rejected | + +Unix-socket containers are detected both in the controller +(`_is_unix_socket_docker`, link eligibility) and on the compute side (NIO +rejection) — a missed detection surfaces as a clearer-late error. + +## Architecture + +Kernel link between two containers on the same compute: + +``` + container A netns root namespace container B netns + ┌────────────┐ veth pair ┌───────────────────┐ veth pair ┌────────────┐ + │ eth0 ◄─────┼───────────────┼─► gv{id}e0p0 ──────┼──► gns3{lid} │──────────► │ eth0 + └────────────┘ gc{id}e0p0 │ (bridge port) │ (bridge port)│ └────────────┘ + │ per-link kernel │ gv{id}e0p0 + │ bridge = "cable" │ + └───────────────────┘ +``` + +Relay link on the same unified veth (fallback for filtered links or when +`enable_kernel_datapath` is off — e.g. to reach another compute): + +``` + ┌────────────┐ ┌────────────────────────────┐ ┌────────────┐ + │ eth0 │ veth │ uBridge relay bridge │ UDP │ peer node │ + │ (gc end) │◄────────────►│ add_nio_ethernet (AF_ │◄────────────►│ (any type, │ + └────────────┘ gv host end │ PACKET) ↔ add_nio_udp │ tunnel │ any host) │ + └────────────────────────────┘ └────────────┘ +``` + +Both datapaths anchor on the *same* veth host end; switching between them +(link delete + recreate) never touches the container-side interface. + +## Datapath selection + +`UDPLink._kernel_datapath_eligible` decides in the controller at NIO prepare +time; the compute side only reacts to the NIO type: + +* both endpoints Docker (any class except unix-socket containers), same + compute +* no active packet filters (filters live in the relay; kernelization via tc + netem is planned) +* `Server.enable_kernel_datapath` enabled (default) + +There is **no stopped-node requirement**. Since every adapter is a veth, links +attach to running containers: `brctl addif` (kernel) or +`bridge add_nio_ethernet` (relay) are runtime-safe operations on the host end. +On project (re)open every link re-runs `_prepare`, so relay links whose +endpoints became eligible are upgraded to the kernel datapath automatically. + +## Adapter lifecycle + +* **Birth (container start)** — `_create_veth`: stale sweep of any leftover + pair (crash residue), `docker create_veth`, host end admin-down, MAC from + the adapter base, `docker move_to_ns` of the guest end (renamed `eth{N}`, or + the `GNS3_INTERFACE_NAMES` mapping). Names are deterministic: + `gv|gc{node_id[:8]}e{adapter}p{port}` (≤ 15 chars). Unconnected adapters are + born too (carrier off) so the interface is visible inside the container. +* **Life (running)** — the veth is never created/deleted/moved again. Link + create/delete/switch, suspend, capture and markers only change *what is + attached to the host end*. +* **Death (container stop)** — `_remove_kernel_veths` deletes the host ends + explicitly: unlike a TAP (which died with the container netns), a veth host + end outlives the container. Deleting either end destroys the pair. + +## Link operations + +| Operation | Kernel-datapath action | +|---|---| +| Create / attach | `brctl create` (EEXIST tolerated via `brctl show` verify) + `link set up` + `brctl addif` both ends (concurrent, race-tolerant) + carrier up | +| Delete / detach | marker teardown → carrier off → `brctl delif` → `brctl delete` (last endpoint wins; EBUSY/ENOENT suppressed). The veth survives — unlike a relay bridge, whose death dropped its filters, an orphaned AF_PACKET marker would keep sniffing, hence the explicit teardown | +| Reset (`POST /links/{id}/reset`) | delete + create (re-evaluates eligibility) | +| Suspend (`PUT /links/{id}` `{"suspend": true}`) | veth host end admin-state down both ends — 100 % loss, no synthetic filter needed; resume restores | + +## Capture + +Kernel links capture via uBridge's AF_PACKET module bound to the veth host +end: `capture start_kernel "" [dlt]` / `capture stop_kernel`. +Single capture per uBridge process (second concurrent → EALREADY). Start/stop +key on the **NIO type**, not on veth presence — a relay NIO riding a veth +captures at its relay bridge. + +## Markers + +Markers ride the NIO like on the relay datapath, routed by `capture_node_id`; +the anchor is the veth host end instead of a relay bridge +(`_ubridge_apply_markers(anchor, nio)` is datapath-agnostic; DockerVM +overrides the add/delete/enable primitives to translate the commands): + +``` +marker add_kernel "" [tag ] [link ] [dir ] [linktype ] [pcap ""] +marker delete_kernel # idempotent +marker enable_kernel # off = installed but silent +``` + +* Per-deployment convention: **only the capture node's end** runs + `add_kernel` (symmetric with capture) — signals form clean send/return + pairs (`PACKET_OUTGOING` on the host end = container receiving = `rx`). +* Same BPF compilation, MARK UDP signals, pcap files and fine-grained REST + toggle/rebuild/delete endpoints as the relay `mark` filter. +* Node restart reinstalls markers from the NIO; link deletion tears them down. + +## Packet filters + +Not supported on the kernel datapath yet: a link with active filters is wired +on the relay (where uBridge's userspace filters apply), and adding filters to +an existing kernel link returns 409. Kernelization via `tc netem` on the veth +is the planned next stage — it also removes the relay's `delay` filter +nanosleep bottleneck under high packet rates. + +## uBridge command surface + +No uBridge changes were needed beyond the marker module's `*_kernel` commands +(the AF_PACKET capture module was pre-existing): + +``` +docker create_veth / delete_veth / move_to_ns / set_mac_addr +brctl create / delete / addif / delif / show +link set up|down +capture start_kernel / stop_kernel +marker add_kernel / delete_kernel / enable_kernel +bridge add_nio_ethernet / add_nio_udp / start / stop / start_capture / stop_capture +``` + +## Configuration + +```ini +[Server] +# Wire eligible Docker-to-Docker links through kernel veth/bridge interfaces +enable_kernel_datapath = True +``` + +Intended to grow into the global datapath switch as QEMU / IOU / Dynamips +migrate to kernel bridges (QEMU: tap enslaved per-link bridge, zero relay). + +## Verification + +End-to-end on a five-container FRR topology (mixed runtime-drawn and reloaded +links): 8/8 links on the kernel datapath, ping RTT ≈ 0.05 ms, FDB learning, +suspend = 100 % loss / resume restores, project reopen upgrades relay links, +runtime link creation on running containers, marker pcaps exact (tx/rx +pairs), capture freeze on stop, server restart reconciliation. Unit tests: +`tests/compute/docker/test_docker_kernel_datapath.py`, +`tests/controller/test_kernel_datapath_link.py`. diff --git a/docs/features/marker-traffic-insight.md b/docs/features/marker-traffic-insight.md index ab71cbf65..4beb725c2 100644 --- a/docs/features/marker-traffic-insight.md +++ b/docs/features/marker-traffic-insight.md @@ -46,6 +46,14 @@ unchanged. Each compute process runs one UDP listener serving every uBridge on t `node` and `link` fields in each signal together identify the source link (see [Per-link attribution](#per-link-attribution)). +On **kernel-datapath** links (see +[docker-kernel-datapath.md](docker-kernel-datapath.md)) there is no relay +bridge: the marker attaches directly to the veth host end via uBridge's +AF_PACKET marker module (`marker add_kernel`), with identical BPF matching, +signals, pcaps and REST operations. Deployment convention: only the capture +node's end carries the marker, so each exchange produces one clean +`tx`/`rx` signal pair. + ## Business Process ```mermaid