mirror of
https://github.com/GNS3/gns3-server.git
synced 2026-10-22 02:42:22 +03:00
docs(iou): record the IOU kernel datapath and its spec
* docs/features/iou-kernel-datapath.md: architecture (one irreducible userspace hop on the fabric leg, kernel link segment), datapath selection, the anchor lifecycle with the reverse-of-QEMU stop order, capture/markers/filters, relay fallback, and the verified numbers (an isolated server with two IOU nodes behind a fake ELF image, the test playing the IOL fabric: 64/64 kernel, 62/62 relay). * docs/design/ubridge-iol-tap-anchor-spec.md: delivered — implemented on the ubridge fork's feature/iol-tap-anchor branch (both commands, the listener hardening, plus releasing every port NIO on iol_bridge delete even when the bridge was not running), with the implementation notes that came back (208 vs the older 214 for a missing bridge; one perror line per frame dropped on a down anchor). * The Docker/QEMU feature docs point at it from their roadmaps.
This commit is contained in:
parent
f23c3bb6e6
commit
3b8172cbc8
268
docs/design/ubridge-iol-tap-anchor-spec.md
Normal file
268
docs/design/ubridge-iol-tap-anchor-spec.md
Normal file
@ -0,0 +1,268 @@
|
|||||||
|
<!--
|
||||||
|
SPDX-License-Identifier: CC-BY-SA-4.0
|
||||||
|
See LICENSE file for licensing information.
|
||||||
|
-->
|
||||||
|
|
||||||
|
> Frozen requirements spec for the **uBridge** project. Status: **delivered
|
||||||
|
> and verified from the server side** — implemented on the
|
||||||
|
> `feature/iol-tap-anchor` branch of the ubridge fork (both commands, the §B
|
||||||
|
> hardening, and one fix beyond the spec: `iol_bridge delete` now releases
|
||||||
|
> every port NIO even when the bridge was not running, so a stopped node's
|
||||||
|
> anchor fds never leak). The server side landed on
|
||||||
|
> `feat/iou-kernel-datapath` and passed a live e2e (64/64 kernel, 62/62
|
||||||
|
> relay; see `docs/features/iou-kernel-datapath.md`).
|
||||||
|
>
|
||||||
|
> Notes from the implementation: the two new commands answer `208` for a
|
||||||
|
> missing bridge where the older `add_nio_udp` answered `214` (no client
|
||||||
|
> branches on the code), and a write to an admin-DOWN anchor logs one
|
||||||
|
> `perror` line per dropped frame.
|
||||||
|
>
|
||||||
|
> Scope guard: no wire format change, no change to existing commands or
|
||||||
|
> replies. This adds one NIO type to the existing `iol_bridge` module (two
|
||||||
|
> commands) plus one error-tolerance fix in the two listener threads. The
|
||||||
|
> `tap`, `brctl`, `link`, `tc`, `marker` and `capture` modules are reused
|
||||||
|
> as-is.
|
||||||
|
|
||||||
|
# IOL bridge TAP anchor ports (`iol_bridge add_nio_tap`)
|
||||||
|
|
||||||
|
## 0. Background & scope
|
||||||
|
|
||||||
|
### What IOU's physical layer is
|
||||||
|
|
||||||
|
An IOU instance has no host netdev. Its only wire is the Unix-datagram
|
||||||
|
fabric in `/tmp/netio<uid>/`: frames carry an 8-byte IOL header (dst id,
|
||||||
|
src id, dst port, src port, message type, channel), and the NETMAP file
|
||||||
|
maps every `bay/unit` of the instance to a second, fake instance — which
|
||||||
|
today is uBridge itself (`iol_bridge create <name> <app_id + 512>`,
|
||||||
|
binding `/tmp/netio<uid>/<app_id + 512>`). Each port of that fake
|
||||||
|
instance terminates on a **destination NIO**, and the only destination
|
||||||
|
NIO type is UDP (`iol_bridge add_nio_udp`): one listener thread per port
|
||||||
|
relays fabric → NIO, and one bridge listener relays fabric-side reads →
|
||||||
|
NIO sends.
|
||||||
|
|
||||||
|
### The problem
|
||||||
|
|
||||||
|
With UDP as the only termination, every IOU link is a uBridge userspace
|
||||||
|
relay leg:
|
||||||
|
|
||||||
|
- an IOU port cannot join a per-link kernel bridge, so IOU ↔ Docker/QEMU
|
||||||
|
traffic always crosses through a UDP relay even when both ends sit on
|
||||||
|
the same compute;
|
||||||
|
- kernel-class impairments (`tc netem`, the clsact eBPF classifiers) and
|
||||||
|
the `*_kernel` capture/marker commands cannot anchor on an IOU port —
|
||||||
|
only the userspace packet filters apply, whose delay implementation is
|
||||||
|
the known blocking-nanosleep defect that inflates RTT under load;
|
||||||
|
- suspend of a single link has no clean mapping.
|
||||||
|
|
||||||
|
### What this change is — and what it cannot be
|
||||||
|
|
||||||
|
Give the port a second destination NIO type: a **persistent TAP,
|
||||||
|
created by gns3-server with the existing `tap create` command and opened
|
||||||
|
(held) by uBridge**. uBridge's iol_bridge then plays, on that TAP, the
|
||||||
|
same role the QEMU process plays on its TAP: it is the device's
|
||||||
|
userspace endpoint. Everything the anchor model already delivers —
|
||||||
|
`brctl addif` enslavement, `tc` on the anchor, `capture start_kernel`,
|
||||||
|
`marker add_kernel`, `link set up/down` for carrier/suspend — is keyed
|
||||||
|
on the interface name and does not care who holds the fd.
|
||||||
|
|
||||||
|
One userspace hop on the IOU leg is irreducible: the Unix fabric is
|
||||||
|
IOU's only physical layer, so frames always pass
|
||||||
|
fabric ↔ listener thread ↔ TAP fd. What kernelization buys is that the
|
||||||
|
**link segment itself** is the kernel: anchor → per-link bridge → peer
|
||||||
|
anchor, with kernel filters, markers and capture on the anchors, and
|
||||||
|
suspend mapped to anchor admin-down. That is the accepted design, not a
|
||||||
|
deficiency of this spec.
|
||||||
|
|
||||||
|
### Goal
|
||||||
|
|
||||||
|
`iol_bridge` ports can terminate on a pre-existing persistent TAP
|
||||||
|
anchor, with the same lifecycle semantics as `add_nio_udp` (attach
|
||||||
|
before or after `iol_bridge start`, replace a previous NIO, delete and
|
||||||
|
re-add), and the relay survives — without exiting — the anchor being
|
||||||
|
administratively DOWN, which is a steady state of this design (port
|
||||||
|
anchored but no link attached; link suspended).
|
||||||
|
|
||||||
|
Explicitly **not** in scope:
|
||||||
|
|
||||||
|
- creating or deleting the TAP device itself — the server owns the
|
||||||
|
anchor lifecycle through the existing `tap` module (`tap create`,
|
||||||
|
`tap delete`; the device is created hardened per the L2-only spec);
|
||||||
|
- serial bays — a TAP carries Ethernet frames; serial links stay on
|
||||||
|
`add_nio_udp`;
|
||||||
|
- an MTU command (`link set <if> mtu <n>`) — does not exist today and is
|
||||||
|
not required by default-MTU IOU traffic; noted as future work if
|
||||||
|
jumbo frames ever matter;
|
||||||
|
- any change to the NETMAP, the IOL header encoding, or the L1
|
||||||
|
keepalive protocol.
|
||||||
|
|
||||||
|
## A. New commands
|
||||||
|
|
||||||
|
Belong to the existing `iol_bridge` module, next to their UDP
|
||||||
|
counterparts. `bay`/`unit` and the IOL header handling are identical to
|
||||||
|
`add_nio_udp`.
|
||||||
|
|
||||||
|
### `iol_bridge add_nio_tap <bridge> <iol_id> <bay> <unit> <tap_name>` (6 args)
|
||||||
|
|
||||||
|
| Arg | Description |
|
||||||
|
|-----|-------------|
|
||||||
|
| `<bridge>` | An existing IOL bridge (`208` if not) |
|
||||||
|
| `<iol_id>` | The real instance's application id, encoded into the IOL header as with `add_nio_udp`; the existing "`iol_id` == bridge id" refusal in `create_iol_port_entry` applies unchanged |
|
||||||
|
| `<bay>`, `<unit>` | Port coordinates, same encoding and MAX_PORTS check as `add_nio_udp` |
|
||||||
|
| `<tap_name>` | A **pre-existing** network device (`208` if not, see below); ≤ IFNAMSIZ−1 chars (`204` otherwise) |
|
||||||
|
|
||||||
|
Semantics:
|
||||||
|
|
||||||
|
1. **Require an existing device.** Resolve `<tap_name>` with
|
||||||
|
`if_nametoindex` first and refuse with `208` when absent. This is the
|
||||||
|
same reasoning as the `tap` module's `tap_require_existing`: without
|
||||||
|
the check, `TUNSETIFF` silently *creates a transient* TAP that dies
|
||||||
|
with the fd — an anchor that vanishes on detach is worse than a
|
||||||
|
clear error.
|
||||||
|
2. Open the device with the existing `create_nio_tap()` (by-name
|
||||||
|
`TUNSETIFF`, `IFF_TAP | IFF_NO_PI`, default carrier). No owner or
|
||||||
|
persistence ioctl — the device is persistent and owned already.
|
||||||
|
3. Install it through the existing `create_iol_port_entry()`, which is
|
||||||
|
generic over `nio_t *`: it frees whatever NIO the port held before
|
||||||
|
(listener stop guarded on `tid`, delay lines torn down under
|
||||||
|
`iol_delay_lock`, capture/filters freed), pre-computes the IOL
|
||||||
|
header, and starts the port listener if the bridge is running. A
|
||||||
|
port may therefore swap `add_nio_udp` ↔ `add_nio_tap` repeatedly —
|
||||||
|
that is the datapath-switch path and must keep working.
|
||||||
|
4. If the fd cannot be opened (`create_nio_tap` fails), reply `206` and
|
||||||
|
free the nio as the UDP handler does.
|
||||||
|
|
||||||
|
No `carrier` argument: administrative state is the server's lever, via
|
||||||
|
the existing `link set <tap> up|down`.
|
||||||
|
|
||||||
|
### `iol_bridge delete_nio_tap <bridge> <bay> <unit>` (3 args)
|
||||||
|
|
||||||
|
Same body as `cmd_delete_nio_udp`, including its thread-safety guards:
|
||||||
|
stop the port listener (cancel guarded on `tid != 0`, join), destroy
|
||||||
|
both delay lines under `iol_delay_lock`, free capture, filters and the
|
||||||
|
NIO (which closes the TAP fd). **The persistent device itself survives**
|
||||||
|
— the server deletes it with `tap delete` when the node stops, exactly
|
||||||
|
as it does for QEMU anchors. Deleting a port that holds no NIO is a
|
||||||
|
no-op `100` (mirrors `delete_nio_udp`).
|
||||||
|
|
||||||
|
Replies:
|
||||||
|
|
||||||
|
```
|
||||||
|
iol_bridge add_nio_tap IOL-BRIDGE-513 1 0 0 gi0badfe9e0p0
|
||||||
|
100-NIO TAP added to IOL bridge 'IOL-BRIDGE-513'
|
||||||
|
iol_bridge delete_nio_tap IOL-BRIDGE-513 0 0
|
||||||
|
100-NIO TAP deleted from IOL bridge 'IOL-BRIDGE-513'
|
||||||
|
```
|
||||||
|
|
||||||
|
## B. Required hardening: the anchor can be administratively DOWN
|
||||||
|
|
||||||
|
A TAP anchor is expected to sit admin-DOWN for long periods (port
|
||||||
|
anchored, no link attached yet; or the link suspended). The listeners
|
||||||
|
must treat that as normal:
|
||||||
|
|
||||||
|
1. **`iol_bridge_listener`, NIO send path** (IOL → TAP, `nio->send` = a
|
||||||
|
`write` on the fd): a write to an admin-DOWN TAP fails with **EIO**.
|
||||||
|
The current tolerated set is `ECONNREFUSED | ENETDOWN | EINVAL`,
|
||||||
|
anything else is `exit(EXIT_FAILURE)` — which would take the whole
|
||||||
|
uBridge down the first time a suspended link carries traffic. Add
|
||||||
|
`EIO` to the tolerated set: drop the frame (optionally a debug log)
|
||||||
|
and continue.
|
||||||
|
2. **`iol_nio_listener`, NIO receive path** (TAP → IOL, `nio_recv` = a
|
||||||
|
`read` on the fd): while the device is DOWN the read blocks (no
|
||||||
|
carrier), which is fine — but a return of `0` must not be counted or
|
||||||
|
forwarded (it would fabricate header-only packets for the instance).
|
||||||
|
Treat `recv <= 0` as continue; treat `-1` with `EIO` as continue
|
||||||
|
alongside the existing `ECONNREFUSED | ENETDOWN`.
|
||||||
|
|
||||||
|
No other listener behavior changes: the fabric-side `sendto` already
|
||||||
|
tolerates `ENOENT`/`ECONNREFUSED` for the window where the IOU process
|
||||||
|
has not created its endpoint yet, and that stays.
|
||||||
|
|
||||||
|
## C. Implementation notes
|
||||||
|
|
||||||
|
- The command handlers are thin: existence check + `create_nio_tap` +
|
||||||
|
`create_iol_port_entry` (add), and the `cmd_delete_nio_udp` body
|
||||||
|
(delete). The three copies of that teardown block in the module are a
|
||||||
|
pre-existing pattern — a shared helper is welcome but not required by
|
||||||
|
this spec.
|
||||||
|
- Do **not** route the TAP through `nio_tap_open`'s `/dev/net/tun`
|
||||||
|
path-name variant; the anchor is addressed by interface name.
|
||||||
|
- Statistics (`iol_bridge get_stats`) work unchanged: the counters hang
|
||||||
|
off the generic nio and both listeners already update them.
|
||||||
|
- Port filters, delay lines, capture (`iol_bridge add_packet_filter`,
|
||||||
|
`start_capture`, …) keep working on a TAP-terminated port; gns3-server
|
||||||
|
simply will not combine userspace filters with `tc` on the same port
|
||||||
|
(see the alignment section).
|
||||||
|
|
||||||
|
## D. Reply contract & error codes
|
||||||
|
|
||||||
|
| Code | Meaning |
|
||||||
|
|------|---------|
|
||||||
|
| `100` | OK — NIO added/deleted (delete of an empty port is also `100`) |
|
||||||
|
| `203` | Bad number of parameters |
|
||||||
|
| `204` | Invalid parameter (tap name longer than IFNAMSIZ−1) |
|
||||||
|
| `206` | Unable to create (bridge exists but the TAP fd cannot be opened) |
|
||||||
|
| `208` | Unknown object — bridge or **tap device** does not exist |
|
||||||
|
|
||||||
|
Old builds answer `202-Unknown command` for both commands; gns3-server
|
||||||
|
uses that to fall back to the UDP datapath, mirroring the `tap`-probe
|
||||||
|
degradation.
|
||||||
|
|
||||||
|
## E. Test requirements
|
||||||
|
|
||||||
|
1. **Both-direction relay.** On a running IOL bridge with one
|
||||||
|
TAP-terminated port: a frame sent on the fabric for that bay/unit
|
||||||
|
appears on the TAP netdev (stripped of the IOL header); a frame
|
||||||
|
injected on the netdev reaches the fabric with the port's correct
|
||||||
|
8-byte header. (A scratch Unix-datagram endpoint can play the
|
||||||
|
instance; no IOU binary is needed.)
|
||||||
|
2. **No transient device.** `add_nio_tap … nosuchtap0` → `208`, and no
|
||||||
|
interface is created (`ip -o link` unchanged).
|
||||||
|
3. **Unknowns.** Unknown bridge → `208`; name ≥ IFNAMSIZ → `204`; a
|
||||||
|
bridge-id == iol_id → the existing refusal.
|
||||||
|
4. **Delete keeps the device.** After `delete_nio_tap`, the port has no
|
||||||
|
NIO (stats/list reflect that), the TAP netdev still exists, and the
|
||||||
|
port accepts `add_nio_udp` (and re-`add_nio_tap`) afterwards.
|
||||||
|
5. **Swap semantics.** A port holding a UDP NIO that gets `add_nio_tap`
|
||||||
|
releases the UDP socket (old listener joined, port usable), and vice
|
||||||
|
versa — no fd or thread leak after several swaps.
|
||||||
|
6. **DOWN-anchor resilience.** With the TAP admin-DOWN, drive traffic
|
||||||
|
from the fabric side for ≥ 100 frames: uBridge stays alive, frames
|
||||||
|
are dropped, nothing is written to the netdev. After `link set up`,
|
||||||
|
traffic flows again. This is the regression test for §B.1.
|
||||||
|
7. **Idle DOWN port is quiet.** A DOWN anchored port produces no
|
||||||
|
header-only or zero-length fabric sends (§B.2).
|
||||||
|
8. **Kernel-bridge interop.** Enslave the anchor into a kernel bridge
|
||||||
|
(`brctl create` + `addif`) while the port relays: frames cross
|
||||||
|
bridge ↔ fabric through the port, and a peer port on the same bridge
|
||||||
|
receives them. This is the actual deployment shape.
|
||||||
|
|
||||||
|
## gns3-server alignment (informational — not uBridge scope)
|
||||||
|
|
||||||
|
- **Anchors**: one TAP per Ethernet bay/unit, `gi{node_id[:8]}e{bay}p{unit}`,
|
||||||
|
created persistent (hardened by `tap create` per the L2-only spec) at
|
||||||
|
node start, swept for stale leftovers first, `tap delete`d at node
|
||||||
|
stop — the exact lifecycle QEMU anchors use. All Ethernet ports are
|
||||||
|
anchored at start ("tap-everywhere"); serial bays never are.
|
||||||
|
- **Links**: kernel links enslave the anchor via the shared
|
||||||
|
`KernelDatapathMixin` (`brctl create`/`addif`, `tc`, markers,
|
||||||
|
`capture start_kernel`, suspend = anchor admin-down). Relay links keep
|
||||||
|
`add_nio_udp` and never touch the anchor. A port switching datapaths
|
||||||
|
is just an NIO swap (§A.3).
|
||||||
|
- **Filters are never double-applied**: on a kernel link only the anchor
|
||||||
|
`tc` path is used; the port's userspace filters carry relay links
|
||||||
|
only.
|
||||||
|
- **Capability gating**: the eligibility check needs to know the command
|
||||||
|
exists before creating a link. Probe by attempting the operation on a
|
||||||
|
scratch bridge + scratch TAP (create, `add_nio_tap`, delete, clean
|
||||||
|
up), cached by binary identity like `probe_tap_support`, reported
|
||||||
|
through `/capabilities`; an old build answering `202` keeps IOU on
|
||||||
|
the relay datapath.
|
||||||
|
- **L1 keepalives**: unchanged — an anchored-but-unconnected port holds
|
||||||
|
no NIO from the server's perspective, so the responder keeps ignoring
|
||||||
|
it and IOU sees the port down, which matches the anchor being DOWN.
|
||||||
|
|
||||||
|
## Roadmap note
|
||||||
|
|
||||||
|
`link set <if> mtu <n>` does not exist today. Default-MTU IOU traffic
|
||||||
|
does not need it; if jumbo IOU frames ever matter it becomes a small,
|
||||||
|
separate requirement on the `link` module.
|
||||||
@ -367,8 +367,12 @@ enable_kernel_datapath = True
|
|||||||
Intended to grow into the global datapath switch as IOU / Dynamips migrate to
|
Intended to grow into the global datapath switch as IOU / Dynamips migrate to
|
||||||
kernel bridges. **QEMU has landed** on the same pattern (persistent TAP anchor
|
kernel bridges. **QEMU has landed** on the same pattern (persistent TAP anchor
|
||||||
enslaved into the per-link bridge, zero relay) — see
|
enslaved into the per-link bridge, zero relay) — see
|
||||||
`docs/features/qemu-kernel-datapath.md`; the shared half of this document's
|
`docs/features/qemu-kernel-datapath.md` — and **IOU after it**
|
||||||
compute machinery now lives in `gns3server/compute/kernel_datapath.py`.
|
(`docs/features/iou-kernel-datapath.md`; the fabric terminator in uBridge
|
||||||
|
gained a TAP-terminated port, `iol_bridge add_nio_tap`, specified in
|
||||||
|
`docs/design/ubridge-iol-tap-anchor-spec.md`). The shared half of this
|
||||||
|
document's compute machinery now lives in
|
||||||
|
`gns3server/compute/kernel_datapath.py`.
|
||||||
|
|
||||||
## Verification
|
## Verification
|
||||||
|
|
||||||
|
|||||||
170
docs/features/iou-kernel-datapath.md
Normal file
170
docs/features/iou-kernel-datapath.md
Normal file
@ -0,0 +1,170 @@
|
|||||||
|
<!--
|
||||||
|
SPDX-License-Identifier: CC-BY-SA-4.0
|
||||||
|
See LICENSE file for licensing information.
|
||||||
|
-->
|
||||||
|
|
||||||
|
# IOU kernel datapath
|
||||||
|
|
||||||
|
IOU's only physical layer is the Unix-datagram IOL fabric (`/tmp/netio<uid>/`,
|
||||||
|
8-byte IOL headers, NETMAP), so it has no host netdev to enslave — a kernel
|
||||||
|
link instead gives each **Ethernet bay/unit** a persistent TAP anchor
|
||||||
|
(`gi{node_id[:8]}e{bay}p{unit}`), and uBridge's `iol_bridge` (the fake IOL
|
||||||
|
instance the NETMAP maps to) binds the port to that TAP with
|
||||||
|
`iol_bridge add_nio_tap`. Links then attach to the anchor exactly as they do
|
||||||
|
for Docker (veth) and QEMU (TAP): a per-link Linux bridge enslaves it, tc
|
||||||
|
impairments run on it, capture and markers anchor on it. The uBridge
|
||||||
|
capability is specified in `docs/design/ubridge-iol-tap-anchor-spec.md` and
|
||||||
|
delivered by the `feature/iol-tap-anchor` branch of the ubridge fork.
|
||||||
|
|
||||||
|
Serial bays never anchor (a TAP carries Ethernet frames): serial links stay
|
||||||
|
on the relay datapath whatever the compute's capabilities.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
IOU process root namespace
|
||||||
|
┌──────────┐ unix datagram ┌──────────────────────────────┐
|
||||||
|
│ IOL │◄──── /tmp/netio ──►│ uBridge iol_bridge (listener)│
|
||||||
|
│ fabric │ (8-byte hdr) │ │ holds the TAP fd │
|
||||||
|
└──────────┘ │ ▼ │
|
||||||
|
│ gi{id}e{bay}p{unit} ──────┼──► gns3{link_id}
|
||||||
|
│ (persistent TAP anchor) │ (per-link kernel
|
||||||
|
└──────────────────────────────┘ bridge port)
|
||||||
|
```
|
||||||
|
|
||||||
|
One userspace hop on the IOU leg is **irreducible** — the fabric is Unix
|
||||||
|
sockets, so frames always cross fabric → listener thread → TAP fd. What the
|
||||||
|
kernel datapath buys is the **link segment**: anchor → per-link bridge →
|
||||||
|
peer anchor crosses entirely in the kernel (including IOU ↔ Docker/QEMU
|
||||||
|
links), impairments run as kernel tc instead of the userspace delay line
|
||||||
|
(the known blocking-nanosleep RTT defect), and suspend maps to anchor
|
||||||
|
admin-down (writes to a down TAP fail EIO and are dropped by the delivered
|
||||||
|
§B hardening). Direction semantics on the anchor match QEMU's exactly:
|
||||||
|
uBridge holds the fd like the QEMU process does.
|
||||||
|
|
||||||
|
## Datapath selection
|
||||||
|
|
||||||
|
`UDPLink._kernel_datapath_eligible` extends the Docker/QEMU rules with:
|
||||||
|
|
||||||
|
* the compute must report **`ubridge_iol_tap`** (its own capability: the tap
|
||||||
|
module and `iol_bridge add_nio_tap` land independently, and either can be
|
||||||
|
missing on an old build). Probed by running one add/delete cycle on a
|
||||||
|
scratch bridge + scratch TAP (`probe_iol_tap_support`, cached by binary
|
||||||
|
identity, scratch bridge id 1050 — above the per-node id space 513..1024
|
||||||
|
the fabric locks); any failure means unknown → relay;
|
||||||
|
* **both linked ports must be Ethernet** — an IOU serial port disqualifies
|
||||||
|
the kernel path per-port, not per-node, so serial links between capable
|
||||||
|
IOU nodes silently stay on the relay instead of failing at the compute.
|
||||||
|
|
||||||
|
The NIO schema on the compute NIO routes accepts `nio_bridge` alongside the
|
||||||
|
UDP/TAP/Ethernet NIOs (the same widening QEMU needed — without it the
|
||||||
|
controller's NIO POST is 422'd before the node ever sees it).
|
||||||
|
|
||||||
|
## Anchor lifecycle — and why the stop order is QEMU's reverse
|
||||||
|
|
||||||
|
* **Birth (node start)** — `_prepare_tap_datapath` probes the capability and
|
||||||
|
creates one persistent TAP per Ethernet bay/unit (4 units per bay, swept
|
||||||
|
for stale leftovers first, born DOWN). uBridge holds a device's fd only
|
||||||
|
while a kernel link binds the port; until then the TAP has no holder.
|
||||||
|
* **Life** — the TAP is never recreated. A kernel link adds
|
||||||
|
`iol_bridge add_nio_tap <bridge> <iol_id> <bay> <unit> <tap>` then
|
||||||
|
enslaves the anchor (`_kernel_attach`); a relay link keeps
|
||||||
|
`iol_bridge add_nio_udp` and never touches the anchor. Switching a port
|
||||||
|
between datapaths is just an NIO swap — the IOL port frees its previous
|
||||||
|
NIO (fd or socket) on attach.
|
||||||
|
* **Death (node stop)** — `_stop_ubridge` runs the cleanup **in the reverse
|
||||||
|
order of QEMU's**, because here uBridge itself holds the anchor fds:
|
||||||
|
`iol_bridge delete` (releases every port's TAP fd) → `tap delete` per
|
||||||
|
anchor (a held TAP answers EBADFD/207) → `brctl delete` for per-link
|
||||||
|
bridges still held → stop the hypervisor. Removing a single kernel link
|
||||||
|
does the same in miniature: `_remove_kernel_nio` (markers, tc reset,
|
||||||
|
delif, brctl delete) then `iol_bridge delete_nio_tap` — releasing the fd
|
||||||
|
while the control channel lives.
|
||||||
|
|
||||||
|
## Link operations
|
||||||
|
|
||||||
|
| Operation | Kernel-datapath action |
|
||||||
|
|---|---|
|
||||||
|
| Create / attach | `iol_bridge add_nio_tap` (bind port ↔ anchor, uBridge holds the fd) + `brctl create` (EEXIST tolerated) + `link set <bridge> up` + `brctl addif` + carrier up + markers + `tc netem set` |
|
||||||
|
| Delete / detach | marker teardown → `tc reset <tap>` → carrier off → `brctl delif`/`brctl delete` (last endpoint wins) → `iol_bridge delete_nio_tap` (release the fd; the device survives) |
|
||||||
|
| Update (filters/markers changed) | reconcile on the anchor only — no re-binding, no re-enslaving |
|
||||||
|
| Suspend | anchor admin-down (IOL relay writes fail EIO → dropped; reads block) + resume restores |
|
||||||
|
|
||||||
|
## Capture, markers, filters
|
||||||
|
|
||||||
|
Capture on a kernel link is `capture start_kernel <tap>` (driven by the NIO
|
||||||
|
type); markers ride the base reconcile with the anchor as the location —
|
||||||
|
IOU's marker primitives are polymorphic (`marker add_kernel/enable_kernel/
|
||||||
|
delete_kernel` on an anchor, `iol_bridge … mark` on an IOL location), which
|
||||||
|
let the old IOU-specific reconcile override be deleted outright. Filters are
|
||||||
|
one tc netem qdisc per anchor plus cls_bpf/eBPF drops — the port's
|
||||||
|
userspace `iol_bridge add_packet_filter` filters carry relay links only and
|
||||||
|
are never combined with tc on the same port.
|
||||||
|
|
||||||
|
## Relay fallback
|
||||||
|
|
||||||
|
A uBridge without `iol_bridge add_nio_tap` (old build) keeps the node on the
|
||||||
|
relay datapath unchanged: every link rides `iol_bridge add_nio_udp`, the
|
||||||
|
node starts fine, and the warning says it cannot carry kernel links. The
|
||||||
|
controller never offers such a node a kernel link (capability unreported).
|
||||||
|
|
||||||
|
## Verified
|
||||||
|
|
||||||
|
Unit level: anchor lifecycle and naming, kernel/relay binding on add,
|
||||||
|
update and remove, the stop order (iol_bridge delete before tap delete
|
||||||
|
before brctl delete before hypervisor stop), `_networking` on restart with
|
||||||
|
mixed kernel/relay ports, capture and marker command shapes on both
|
||||||
|
datapaths, the capability probe (cycle, old-build fallback, cleanup,
|
||||||
|
caching), capability plumbing, and controller eligibility (per-compute
|
||||||
|
capability, its independence from `ubridge_tap`, serial-port exclusion).
|
||||||
|
|
||||||
|
End-to-end on a live server (isolated instance, two IOU nodes behind a fake
|
||||||
|
ELF image, the test playing the IOL fabric on `/tmp/netio<uid>/` and pushing
|
||||||
|
frames through it): **64/64 checks** on the kernel datapath and **62/62** on
|
||||||
|
the relay (`enable_kernel_datapath = false`). Kernel highlights:
|
||||||
|
|
||||||
|
* 8 anchors per node (2 bays × 4 units), persistent, DOWN, address-free;
|
||||||
|
`link_list` reports `kernel_datapath: true` and one per-link bridge holds
|
||||||
|
both anchors with `brport/state = 3`;
|
||||||
|
* fabric → anchor → bridge → anchor → fabric in both directions at ~0.2 ms;
|
||||||
|
`delay 100` ⇒ netem on both anchors and a measured **100.5 ms** one-way;
|
||||||
|
* suspend ⇒ anchor DOWN, no traffic, resume ⇒ 100.4 ms with the qdisc intact;
|
||||||
|
* capture ⇒ a 328-byte pcap written by `capture start_kernel`;
|
||||||
|
* link delete ⇒ bridge gone, qdisc clean, anchor alive, re-create at 0.6 ms;
|
||||||
|
* a **serial** link between the same two capable nodes stays on the relay
|
||||||
|
(`kernel_datapath: false`) — per-port exclusion, not per-node;
|
||||||
|
* single-sided restart ⇒ stopped node's anchors removed, peer's kept, link
|
||||||
|
re-attached from the NIO (0.3 ms); stop ⇒ no stray anchors, bridges or
|
||||||
|
fabric sockets.
|
||||||
|
|
||||||
|
The relay run exercises the same lifecycle with the IOL UDP NIOs (no tc
|
||||||
|
qdiscs, userspace `delay` at 100.7 ms, suspend via the synthetic
|
||||||
|
frequency_drop, `iol_bridge start_capture`), which is also the regression
|
||||||
|
net for the port-coordinate capture restore this work touched.
|
||||||
|
|
||||||
|
The uBridge side passed its own 30/30 (`tests/iol/test_tap_anchor.py` in the
|
||||||
|
fork), including the DOWN-anchor resilience and kernel-bridge interop this
|
||||||
|
design depends on.
|
||||||
|
|
||||||
|
One pre-existing bug surfaced here and is fixed: the IOU NIO **update** route
|
||||||
|
never copied the body's `suspend` onto the NIO (Docker and QEMU routes do).
|
||||||
|
Harmless on the relay, where suspend is emulated by the synthetic
|
||||||
|
frequency_drop filter, but fatal for a kernel link, whose suspend *is* the
|
||||||
|
anchor's admin state — a "suspended" link kept forwarding.
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Server]
|
||||||
|
# Wire eligible links (same compute, both endpoints Ethernet and able to
|
||||||
|
# anchor) through kernel interfaces instead of the uBridge UDP relay.
|
||||||
|
enable_kernel_datapath = True
|
||||||
|
```
|
||||||
|
|
||||||
|
## Roadmap
|
||||||
|
|
||||||
|
Dynamips next (hypervisor-created taps, enslavable as they are), then the
|
||||||
|
Ethernet switch (its anchors are ubridge-owned by design) and the IOL Docker
|
||||||
|
node (same `bridge add_nio_tap` shape as this one). Cross-compute kernel
|
||||||
|
links need VXLAN/GENEVE encapsulation — deferred until every node type is
|
||||||
|
kernelized.
|
||||||
@ -161,8 +161,15 @@ addresses involved, IPv4 has no per-device IGMP switch).
|
|||||||
|
|
||||||
## Roadmap
|
## Roadmap
|
||||||
|
|
||||||
IOU (its fabric would need a kernel-side port in `iol_bridge`), Dynamips
|
IOU (its fabric terminator in uBridge needs a TAP-terminated port —
|
||||||
|
`iol_bridge add_nio_tap`, frozen in
|
||||||
|
`docs/design/ubridge-iol-tap-anchor-spec.md`; unlike QEMU, one userspace hop
|
||||||
|
on the IOU leg is irreducible, the fabric is Unix sockets), Dynamips
|
||||||
(hypervisor-created taps, enslavable as they are), and the Ethernet switch /
|
(hypervisor-created taps, enslavable as they are), and the Ethernet switch /
|
||||||
cloud paths (their anchors are ubridge-owned by design) are still on the
|
cloud paths (their anchors are ubridge-owned by design) are still on the
|
||||||
relay; cross-compute kernel links need VXLAN/GENEVE encapsulation, which is
|
relay; cross-compute kernel links need VXLAN/GENEVE encapsulation, which is
|
||||||
a separate project (it would serve Docker links the same way).
|
a separate project (it would serve Docker links the same way).
|
||||||
|
|
||||||
|
Update: **IOU has landed** on the same mixin — see
|
||||||
|
`docs/features/iou-kernel-datapath.md` (Ethernet bays anchor on persistent
|
||||||
|
TAPs bound to the IOL fabric; serial links stay relay).
|
||||||
|
|||||||
Loading…
x
Reference in New Issue
Block a user