A correctness read-through of the five delivered datapath documents, plus
the two code comments that carried the same stale vocabulary.
Datapath selection: the VendorDockerVM selector is `console_type=docker_exec`
alone (`GNS3_SKIP_INIT` and friends are behavior knobs of the chosen class);
the cross-compute and anchor-less fallbacks are named for what they are; the
Ethernet-switch endpoint is the documented exception to the anchoring rule,
and the IOL runner container is called out as the Docker node that does not
anchor on a veth.
Filter vocabulary: the stateful eBPF program evaluates four modes
(nth → quota → window → flow), GNS3 exposes three of them, and window_drop
joins the kernel-only list; the correl suffixes gate on the `rate` token as
the extension-build marker.
group_fwd_mask: the 0xfffd port mask is applied inside uBridge's `brctl addif`
itself, not as a separate command — the link-creation sequences and the
operation tables no longer show an arrow the server never sends. Same class
of fix for `link set up` on the IOL anchor (addif already brought it up; the
command only pushes a suspended anchor back down) and for the QEMU carrier
replication (QMP `set_link`, gated by `replicate_network_connection_state`).
Lifecycles and stop orders re-worded around the invariant rather than the
node type: the fd holder must release before the devices are deleted, which
is QEMU's process-first order, IOU's uBridge-first order and Dynamips'
hypervisor-NIO-first order. Deferred wiring for a stopped endpoint is
documented as the common path it is, and the Dynamips "stop is not a close"
note now says what recovery from a dead hypervisor actually requires.
Dynamips specifics: a c7200 needs an adapter in slot 1 (slot 0 is the fixed
I/O slot), the tap-fd direction is stated correctly (writes inject into the
kernel, reads return what the kernel transmits), and the anchor registry is
name-keyed against the controller's port matrix by convention, not derivation.
IOL port bridge naming: `bridge{bay}` on unit 0, `bridge{bay}_{unit}`
elsewhere; the leg swap happens on link delete/re-create or a datapath
switch, while suspend is an in-place NIO update and a project reopen is a
fresh build.
Verified and Roadmap sections: the in-repo e2e suite is distinguished from
the isolated-instance development harness (IOU), assertions are stated with
their bounds and capability gates, and the roadmap entries collapse into the
one remaining project — VXLAN/GENEVE for cross-compute links.
Code comments: `flow` added to the documented mode order in the eBPF apply
path, and the `nio_bridge` module docstring corrected on which filter types
run on the kernel datapath (bpf via cls_bpf, frequency_drop/quota/window_drop
via the eBPF classifier — neither is relay-only any more).
Switch-to-switch links were the last shape left on the relay: the switch
fast path works by absorbing the *peer's* anchor, and a switch has none —
its ports are its own bridge, and a kernel interface belongs to exactly
one bridge, so two bridges cannot be joined by enslaving anything.
Join them with a veth pair instead: one end per bridge, each side
enslaving its own end and applying its own port mode to it — an access
port on one side and a trunk on the other behaves like the real cable it
models. uBridge's docker create_veth/delete_veth (the primitive every
Docker adapter is already born from) creates and destroys the pair; the
names derive from the link id (utils.kernel_anchor.kernel_cascade_names),
so each side computes its own end and the names survive a restart.
The pair belongs to the link, and the two ends are wired by two
independent uBridge processes — the project-open batch drives different
nodes concurrently — so the protocol is order-free: both sides create the
pair when their own end is missing (one create wins; the loser's failure
is re-checked against the kernel and "the pair is there" is not an
error), and either side's teardown destroys it (deleting one veth end
takes the other with it, so both converge on the same gone pair and the
second delif answers ENOENT). A pre-existing end — this switch restarted,
or a leftover from a previous life of the link id — is reused rather than
recreated, because the other end may still be a live port of the peer
switch; a stale tc qdisc is all a reused end can carry.
No new wire format was needed: a cascade is an anchor NIO on both ends,
the absorbed-anchor case plus the peer end's name. That also corrects the
filter routing — the single-sided rule exists because an absorbed anchor
is one interface, while a cascade has one per switch, so impairments go
to both ends like any other kernel link.
Tests: the wire format, the eligibility flip and the both-ends re-push on
node start (controller); the naming contract (utils); the switch-side
lifecycle — create, reuse, lost create race, delete destroys the pair
(compute routes).
Drive-by: the relay filter test's compute mock carried no awaitable
`put`, so tests/controller/test_kernel_datapath_link.py failed when run
alone while an argument-less full run passed; callers now get one.
The Ethernet switch's kernel fast path: instead of a link crossing the
switch's per-port uBridge relay, the switch absorbs the peer's anchor into
its own kernel bridge. The peer's interface (Docker veth host end, QEMU
TAP, IOU port TAP, Dynamips port TAP) becomes the switch port, and frames
cross anchor -> switch bridge -> switch ports entirely in the kernel.
Wire format: the switch end receives nio_anchor ('join this interface to
my own bridge' — the mirror of nio_bridge), the peer end nio_bridge with a
null bridge ('your anchor is bridged elsewhere': carrier, markers and
impairments only — never bridge membership, and never a bridge delete, on
an interface this node does not own). Anchor names come from the shared
naming contract (utils.kernel_anchor), a pure function of node type, id,
adapter and port — the controller computes the peer's anchor while the
node may be stopped, which is what makes links to stopped nodes wireable.
Stopped peers defer the join; after any node of a link starts
(UDPLink.node_started) the controller re-pushes the switch NIO, and the
switch completes — or repairs, when a peer restart replaced its interface
under the same name — the join against the kernel's actual membership
instead of assuming it.
VLAN changes now edit membership in place: only the ports whose mode/VLAN
actually changed are touched, each losing exactly the previous mode's
entries the new mode does not want (the dot1q admits-all entry is one
range operation in both directions), and no port ever leaves the bridge to
change its VLANs. This replaces the old 'reset by re-enslaving' shortcut
(delif+addif on every wired port, added when the ports were the switch's
own TAPs): on absorbed anchors, re-enslaving an interface owned by another
node is where live testing watched links die — anchors occasionally lost
their bridge membership right after a ports_mapping update (later updates
then failed with delif EINVAL), reproducibly with qemu, IOU and Dynamips
peers and never with Docker's veths.
Eligibility: the switch side needs no capability of its own (its bridge
exists by construction); the peer must be able to anchor. Switch-to-switch
cascades stay on the relay — a kernel interface belongs to one bridge, and
joining two bridges needs a veth pair, deferred.
Verified live on a real server: qemu peers through four VLAN cycles with
both anchors enslaved throughout; a mixed switch (2 IOU + 2 Dynamips, four
kernel ports); Docker peers with real ICMP isolating and recovering (with
both anchors still enslaved); VPCS relay ports likewise in place.
Impairment filters with a netem equivalent (delay, packet_loss, corrupt)
now run on the kernel datapath as one tc netem qdisc per veth host end,
pushed through uBridge's netlink tc module (tc netem set / tc reset).
This removes the relay delay filter's nanosleep bottleneck (upstream
Both link endpoints carry the filters: each end's qdisc egress covers
the traffic entering that container, so every direction is impaired
exactly once -- the same net effect as the relay, where both directions
cross the single filtered bridge.
- controller: eligibility keys on relay-only types (frequency_drop, bpf)
instead of any filter; kernel links push the real filters (the
synthetic suspend frequency_drop no longer flips a suspended link to
relay on reopen/reset); update_filters 409s only relay-only types;
available_filters hides them on kernel links
- compute: create_nio whitelists netem-capable filter types; the NIO
update path keeps its own guard (PUT mutates the NIO in place);
_ubridge_apply_netem translates the filter dict into one atomic
netem-set (full re-apply IS the reconcile); node restart restores the
qdisc, link deletion detaches it (the veth survives the link)
- e2e (27/27): delay 100 -> 200.2 ms RTT, atomic replace 50 -> 100.2 ms,
clear -> baseline, packet_loss 30 -> 48 % round-trip (expected 51 %),
suspend/resume and node restart keep the qdisc, link delete leaves no
residual impairment
Markers no longer disqualify the kernel datapath. They ride the NIO like
on the relay path and attach to the veth host end through uBridge's
marker add_kernel/delete_kernel/enable_kernel commands (AF_PACKET taps,
same BPF/direction/signal semantics as the relay mark filter).
- base_node: marker anchors are now polymorphic (relay bridge name or
kernel veth host interface); the reconcile loop is datapath-agnostic
and DockerVM overrides the add/delete/enable primitives to translate
the commands. Marker-name validation extracted for both add paths.
- docker_vm: kernel NIO connect/update apply markers on the veth anchor;
link deletion tears markers down explicitly (the veth survives the
link, unlike a relay bridge whose death drops its filters).
- controller: eligibility and start_marker no longer exclude markers or
project-level marker definitions; the kernel NIO carries the capture
node's marker specs.
- NIO creation accepts markers on kernel NIOs (filters still rejected).
Eligible links (both endpoints Docker, same compute, no filters/markers,
stopped nodes, no unix-socket NIO containers) are wired as veth pairs
enslaved into a per-link Linux kernel bridge (gns3 + 11 hex of the link
id, filling IFNAMSIZ) instead of the uBridge UDP relay, so frames flow
entirely in the kernel.
- NIOBridge NIO type: the controller decides the datapath in
UDPLink._prepare and both computes enslave their deterministic gv*/gc*
veth pair via uBridge's docker/brctl modules
- carrier/suspend = host-end admin state; both endpoints race-safely
create/delete the bridge (EEXIST/EBUSY/ENOENT tolerated)
- veth adapters can still attach relay (UDP) links on a running node:
lazy relay bridge over AF_PACKET on the host end (add_nio_ethernet)
- filters/markers/capture are rejected with explicit errors on this
datapath (they live in the uBridge relay); vendor unix-socket
containers are excluded; Server.enable_kernel_datapath is the
kill-switch (default on)
- host-end veths survive container death and are removed on stop plus
swept by name on the next start (crash/stale recovery)
Mirror the packet-filter lifecycle: marker specs now live on the NIO
(next to filters), ride in link_data from controller to compute on
every NIO create/update, and are reapplied by _ubridge_apply_markers
in add_ubridge_udp_connection (bridge creation / node restart) and
update_ubridge_udp_connection (NIO update — following the preceding
reset_packet_filters so markers survive filter changes).
Changes:
- NIO / NIOUDP: _markers property + asdict
- schemas/compute/nios.py: UDPNIO.markers field
- base_manager.create_nio: nio.markers from settings
- PUT /nio routes (vpcs/qemu/docker): nio.markers update
- base_node: _ubridge_apply_markers(bridge_name, nio) iterates
nio.markers, computes pcap path, calls _ubridge_add_marker_filter
+ MarkerManager.register; called after _ubridge_apply_filters
- controller udp_link: _get_node_markers + _markers_for_node (route
by capture_node_id); markers in create() and update() link_data
- /markers/start,stop endpoints: mirror spec onto nio.markers so
the marker survives a subsequent node stop/start without a PUT
- tests: add markers field to NIO data expectations
This covers:
- Node restart: NIO persists, add_ubridge_udp_connection re-applies
- Filter update: reset wipes markers, _ubridge_apply_markers re-adds
- Project reload: create() carries markers in link_data → create_nio
- Immediate create: endpoint sets nio.markers immediately
