mirror of
https://github.com/GNS3/gns3-server.git
synced 2026-10-13 05:55:35 +03:00
docs(iol): record the IOL runner container kernel datapath
The feature doc covers the architecture (one irreducible userspace hop on the unix-socket leg; the link segment kernelized through the port bridge's swappable TAP leg), the capability gate, the anchor lifecycle, the swap flow, the inherited capture/markers/filters, the relay fallback and the live verification — including the ensure-then-add lesson (tap create is strictly create-only). The other datapath docs' roadmaps catch up: the IOL runner container has landed (and the ethernet-switch doc's stale 'cascade deferred' exclusion from d2f8aba58 is corrected while there); the docker doc's adapter table no longer rejects IOLDockerVM.
This commit is contained in:
parent
72aa379e87
commit
02340f2ed3
@ -85,7 +85,7 @@ markers):
|
||||
|---|---|---|---|
|
||||
| `DockerVM` | default | veth pair | yes |
|
||||
| `VendorDockerVM` (non-unix-socket) | `console_type=docker_exec`, `GNS3_SKIP_INIT`, … (XRd, SR Linux, …) | veth pair (same path via `super()`) | yes |
|
||||
| `IOLDockerVM` | `GNS3_IOL_RUNNER=1` | AF_UNIX datagram socket pairs (`sNN`/`cNN`) — the container netns is unused | rejected (no host-side interface) |
|
||||
| `IOLDockerVM` | `GNS3_IOL_RUNNER=1` | AF_UNIX datagram socket pairs (`sNN`/`cNN`) — the container netns is unused; kernel links ride persistent `gx` TAP anchors through the port bridge's swappable TAP leg (see `iol-docker-kernel-datapath.md`) | yes (gated on `ubridge_bridge_tap` + `ubridge_tap`) |
|
||||
| `VendorDockerVM` (unix-socket) | `GNS3_UNIX_SOCKET_NIO=1` | same socket contract, generic capability for vendor NOS images | rejected |
|
||||
|
||||
Unix-socket containers are detected both in the controller
|
||||
|
||||
@ -151,5 +151,7 @@ enable_kernel_datapath = True
|
||||
|
||||
The Ethernet switch has landed (see `ethernet-switch-kernel-datapath.md` —
|
||||
it absorbs the peer's anchor into its own kernel bridge; switch-to-switch
|
||||
cascades and the IOL Docker node are next). Cross-compute kernel links need
|
||||
cascades and the IOL runner container have landed too — see
|
||||
`ethernet-switch-kernel-datapath.md` and
|
||||
`iol-docker-kernel-datapath.md`). Cross-compute kernel links need
|
||||
VXLAN/GENEVE encapsulation — deferred until every node type is kernelized.
|
||||
|
||||
@ -28,15 +28,17 @@ end / TAP becomes the switch port; frames cross
|
||||
## Datapath selection
|
||||
|
||||
A link with an `ethernet_switch` endpoint rides the fast path when the
|
||||
**peer** can anchor (same compute; Docker always, QEMU/IOU/Dynamips per
|
||||
**peer** can anchor (same compute; Docker always — including IOL runner
|
||||
containers, per their own capability gate — QEMU/IOU/Dynamips per
|
||||
their capabilities) — the switch side needs no anchoring capability of its
|
||||
own, its bridge and brctl exist by construction. Exclusions, all staying
|
||||
on the relay:
|
||||
|
||||
* **switch ↔ switch**: a kernel interface can belong to exactly one
|
||||
bridge, so cascading two switch bridges needs a veth pair between them —
|
||||
a separate mechanism, deferred;
|
||||
* anchor-less peers (VPCS, cloud, the Dynamips-hosted hub);
|
||||
* **switch ↔ switch** cascades ride the kernel too — through a veth pair
|
||||
joining the two bridges (one end per switch, each side applying its own
|
||||
port mode to its own end); see the wire format below;
|
||||
* anchor-less peers (VPCS, cloud, the Dynamips-hosted hub, generic
|
||||
`GNS3_UNIX_SOCKET_NIO` containers);
|
||||
* cross-compute links, non-Ethernet ports: the common rules.
|
||||
|
||||
## The wire format: one NIO names the other end's interface
|
||||
@ -153,6 +155,8 @@ enable_kernel_datapath = True
|
||||
## Roadmap
|
||||
|
||||
Switch-to-switch links (a veth pair joining the two kernel bridges, with
|
||||
each side's port mode applied at its own end) and the IOL Docker node
|
||||
(`bridge add_nio_tap`); cross-compute kernel links need VXLAN/GENEVE
|
||||
encapsulation, deferred until every node type is kernelized.
|
||||
each side's port mode applied at its own end) have landed (see above), and
|
||||
so has the IOL runner container (`iol-docker-kernel-datapath.md` — its
|
||||
`iol_docker` anchors absorb like any peer's). Cross-compute kernel links
|
||||
need VXLAN/GENEVE encapsulation, deferred until every node type is
|
||||
kernelized.
|
||||
|
||||
172
docs/features/iol-docker-kernel-datapath.md
Normal file
172
docs/features/iol-docker-kernel-datapath.md
Normal file
@ -0,0 +1,172 @@
|
||||
<!--
|
||||
SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
See LICENSE file for licensing information.
|
||||
-->
|
||||
|
||||
> This documentation is AI-generated with reference to actual code and verified
|
||||
> against real-kernel testing. AI can make mistakes — please verify against
|
||||
> the source code when in doubt.
|
||||
|
||||
# IOL runner container kernel datapath
|
||||
|
||||
## Overview
|
||||
|
||||
IOL runner containers (`IOLDockerVM`, selected by the `GNS3_IOL_RUNNER`
|
||||
environment marker — the Cisco CML `iol-runner` images such as
|
||||
`iol-xe/iol-xe:17-18-02`) have no container-namespace networking at all:
|
||||
their guest leg is the runner's per-interface AF_UNIX datagram socket pairs.
|
||||
A unix socket is not a kernel interface, so until now every link of such a
|
||||
container rode the uBridge userspace relay (unix ↔ UDP).
|
||||
|
||||
The kernel datapath gives every Ethernet bay/unit a **persistent TAP anchor**
|
||||
(`gx` names, born at node start exactly like IOU's) and wires the *link
|
||||
segment* through the kernel. One userspace hop on the container leg is
|
||||
irreducible — the socket pair is the only physical layer the runner exposes —
|
||||
but the anchor → per-link bridge → peer anchor crossing, the kernel filters,
|
||||
the suspend carrier and the AF_PACKET capture/markers are all native now.
|
||||
|
||||
```
|
||||
container netns (iol-runner) root namespace
|
||||
┌────────────┐ unix datagram ┌────────────────────────────────┐
|
||||
│ netiomux │◄─ sNN/cNN.sock ──►│ uBridge port bridge │
|
||||
│ (sNN/cNN) │ (host-side dir) │ [nio_unix, nio_tap] relay │
|
||||
└────────────┘ │ │ holds the TAP fd │
|
||||
│ ▼ │
|
||||
│ gx{id}e{bay}p{unit} ─────────┼─► gns3{link_id}
|
||||
│ (persistent TAP anchor) │ (per-link kernel
|
||||
└────────────────────────────────┘ bridge port)
|
||||
```
|
||||
|
||||
The port bridge is the deployment shape ubridge's
|
||||
`doc/gns3server-integration.md` § *bridge — generic NIO relay with a
|
||||
swappable TAP leg* froze: a per-node bridge created at start, the unix NIO in
|
||||
the first slot for the node's whole life, the topology leg in the second —
|
||||
`add_nio_udp` (relay) or `add_nio_tap` (kernel link) — swapped as links are
|
||||
attached, suspended, deleted or the project reopens, through
|
||||
stop → delete → add → start. The c-socket binding never re-binds across a
|
||||
swap: frames the container sends while the bridge is stopped queue on the
|
||||
socket and relay after `start` (regression-tested in ubridge's bridge suite).
|
||||
|
||||
## Datapath selection
|
||||
|
||||
`UDPLink._kernel_datapath_eligible` treats an IOL runner container like any
|
||||
Docker node (same compute, Ethernet port) with one gate of its own in
|
||||
`_kernel_endpoint_ready`: the compute must report both
|
||||
|
||||
* `ubridge_tap` — the tap module (`tap create`/`tap delete`), the same
|
||||
capability QEMU and Dynamips gate on, and
|
||||
* `ubridge_bridge_tap` — `bridge delete_nio_tap`, the swappable-leg command
|
||||
(probed with one command against a bridge that cannot exist: 202 = old
|
||||
build, 214 = new; no scratch objects, no capabilities, cached by binary
|
||||
identity like every other probe).
|
||||
|
||||
Both strict `True`; anything unreported (old uBridge, failed probe) keeps the
|
||||
link on the relay, where it always works. The generic `GNS3_UNIX_SOCKET_NIO`
|
||||
containers stay excluded as before — that marker's semantics are
|
||||
image-specific, while `GNS3_IOL_RUNNER` selects a class whose anchors this
|
||||
server owns.
|
||||
|
||||
## Anchor lifecycle
|
||||
|
||||
* **Birth (node start, before any link)** — `_start_ubridge` runs
|
||||
`_prepare_tap_datapath` right after the hypervisor connects: probe both
|
||||
capabilities, then for every (bay, unit 0-3) sweep a leftover
|
||||
(`tap delete`, suppressed), `tap create` (hardened, starts DOWN) and
|
||||
`link set down` — carrier off until a link attaches. The anchors exist
|
||||
whether or not any link ever uses them: an anchor born with its link would
|
||||
leave the Ethernet switch fast path's deferred join waiting forever.
|
||||
No `tap set_owner`: uBridge itself holds these fds.
|
||||
* **Life** — the anchor is never recreated. Link create/delete/switch,
|
||||
suspend, capture and markers only change what is attached to it, and the
|
||||
ensure-then-add contract guards the one window where the device could have
|
||||
been swept (`/sys/class/net` existence check — `tap create` is strictly
|
||||
create-only, `IFF_TUN_EXCL`, and refuses an existing device with EBUSY).
|
||||
* **Death (node stop)** — the port bridges hold the anchor fds
|
||||
(`bridge add_nio_tap`), so they are deleted first (`bridge delete` stops
|
||||
their relay threads and frees the NIOs), then the anchors themselves
|
||||
(`tap delete` answers EBADFD on a device another fd still holds), then the
|
||||
per-link kernel bridges — all while the control channel lives, before the
|
||||
hypervisor stops. A restart recreates everything from the NIO.
|
||||
|
||||
## Link operations
|
||||
|
||||
* **Kernel link create** (`_attach_kernel_link`): ensure the port bridge and
|
||||
its unix NIO → ensure the anchor → `bridge add_nio_tap` (uBridge opens and
|
||||
holds the fd) → the shared mixin flow (`brctl create`/`addif` the per-link
|
||||
bridge, capture, markers, tc netem/cls_bpf/eBPF filters on the anchor) →
|
||||
`bridge start` (the [unix ↔ tap] relay) → the carrier pass (which is also
|
||||
what brings a born-down anchor up; a suspended NIO sets it back down).
|
||||
* **Link delete** (`_release_port_tap`): `bridge stop` (delete_nio_tap
|
||||
refuses while running — the relay threads hold the NIO pointers for their
|
||||
whole life; freeing under them is a use-after-free) →
|
||||
`bridge delete_nio_tap <bridge> "<tap>"` (matched on the kernel-resolved
|
||||
name) → the mixin teardown (markers, tc reset, `brctl delif`, per-link
|
||||
bridge deletion). The anchor survives — the node owns it, not the link.
|
||||
* **Relay link** — unchanged: the port bridge's second leg is `add_nio_udp`,
|
||||
userspace filters at the port bridge. The unix NIO stays in its slot
|
||||
across every datapath switch.
|
||||
|
||||
## Capture, markers, filters
|
||||
|
||||
Inherited from `KernelDatapathMixin` unchanged — everything is keyed on the
|
||||
anchor interface name, which is why a TAP and a veth host end are
|
||||
interchangeable there: `capture start_kernel` / `marker add_kernel`
|
||||
(AF_PACKET), one tc netem qdisc (the full netem surface plus the
|
||||
extensions), `bpf` as cls_bpf match-drop, `frequency_drop`/`quota`/
|
||||
`window_drop` as the eBPF stateful classifier — all capability-gated per
|
||||
compute exactly like the Docker veth datapath.
|
||||
|
||||
Suspend semantics: anchor admin-down. The port bridge's TAP writes fail EIO
|
||||
(tolerated by the delivered uBridge hardening) and its reads fall silent —
|
||||
traffic stops both ways. The runner itself is unaware of the link state
|
||||
(netiomux has no carrier signal), exactly as on the relay datapath where
|
||||
suspend rode the synthetic frequency_drop filter instead.
|
||||
|
||||
## Relay fallback
|
||||
|
||||
Missing either capability, the node runs relay-only: links ride
|
||||
unix ↔ UDP, `kernel_datapath` stays false, and anchors are simply never
|
||||
created (or, with `enable_kernel_datapath` off server-wide, created but
|
||||
never attached — the same semantics as QEMU's and Dynamips' anchors).
|
||||
|
||||
## Verified
|
||||
|
||||
Live on the real server (`tests/e2e/test_iol_docker_kernel_datapath.py`,
|
||||
isolated instance, real `iol-xe/iol-xe:17-18-02` images, real uBridge with
|
||||
`bridge delete_nio_tap`):
|
||||
|
||||
* kernel link reports `kernel_datapath`, anchors exist from node start, the
|
||||
per-link bridge enslaves exactly the two of them;
|
||||
* real ICMP crosses the [unix ↔ tap] port-bridge relay (100 % ping);
|
||||
* `delay 100` lands as netem on both anchors, measured RTT grows ≥ 150 ms
|
||||
and returns < 50 ms when cleared;
|
||||
* suspend admin-downs the anchor and kills the traffic; resume restores;
|
||||
* AF_PACKET capture writes a real pcap of the ICMP exchange;
|
||||
* link delete/re-create swaps the port bridge's TAP leg out and back in
|
||||
(unix binding never re-binds) with the ping returning;
|
||||
* node stop tears the anchors down (no `gx` devices left), a restart
|
||||
recreates the whole wiring from the NIO;
|
||||
* project delete leaves no anchors and no bridges; the relay control on a
|
||||
relay-configured instance keeps the link off the kernel and still pings.
|
||||
|
||||
Also caught and fixed live: `tap create` is strictly create-only
|
||||
(`IFF_TUN_EXCL`) — the ensure-then-add guard must check existence first
|
||||
(`/sys/class/net`), not re-create unconditionally (EBUSY on the device the
|
||||
start loop just made).
|
||||
|
||||
## Configuration
|
||||
|
||||
```
|
||||
[Server]
|
||||
enable_kernel_datapath = True # the global switch (link attachment)
|
||||
```
|
||||
|
||||
The anchor lifecycle needs no configuration of its own — the capability
|
||||
probes decide per compute, and an IOL runner container without them simply
|
||||
never anchors.
|
||||
|
||||
## Roadmap
|
||||
|
||||
Cross-compute kernel links need VXLAN/GENEVE encapsulation — deferred until
|
||||
every node type is kernelized; with the IOL runner container anchored, that
|
||||
condition is one project away from met.
|
||||
@ -172,8 +172,9 @@ enable_kernel_datapath = True
|
||||
Dynamips has landed (see `dynamips-kernel-datapath.md` — the hypervisor
|
||||
opens uBridge-created TAPs, the same shape QEMU uses), and so has the
|
||||
Ethernet switch (see `ethernet-switch-kernel-datapath.md` — it absorbs the
|
||||
peer's anchor into its own kernel bridge). What remains: switch-to-switch
|
||||
cascades (a veth pair between the two kernel bridges) and the IOL Docker
|
||||
node (same `bridge add_nio_tap` shape as this one). Cross-compute kernel
|
||||
links need VXLAN/GENEVE encapsulation — deferred until every node type is
|
||||
kernelized.
|
||||
peer's anchor into its own kernel bridge). Switch-to-switch cascades (a veth
|
||||
pair between the two kernel bridges) have landed, and so has the IOL runner
|
||||
container — see `iol-docker-kernel-datapath.md` (same persistent-TAP shape,
|
||||
through the generic bridge module's swappable TAP leg instead of the IOL
|
||||
bridge). Cross-compute kernel links need VXLAN/GENEVE encapsulation —
|
||||
deferred until every node type is kernelized.
|
||||
|
||||
@ -182,5 +182,8 @@ hypervisor opens uBridge-created TAPs with `nio create_tap`: the same
|
||||
external-fd-holder shape QEMU uses; serial/ATM/POS ports stay relay).
|
||||
**The Ethernet switch has landed** — see
|
||||
`docs/features/ethernet-switch-kernel-datapath.md` (the switch absorbs a
|
||||
peer's anchor into its own kernel bridge; switch-to-switch cascades stay
|
||||
relay for now).
|
||||
peer's anchor into its own kernel bridge; switch-to-switch cascades too).
|
||||
**The IOL runner container has landed** — see
|
||||
`docs/features/iol-docker-kernel-datapath.md` (persistent TAP anchors
|
||||
through the generic bridge module's swappable TAP leg; the unix-socket guest
|
||||
leg keeps one userspace hop, like IOU's fabric).
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user