Two Alpine containers on a kernel link, a static raw-Ethernet probe built
at run time and docker-cp'd in (the image has no raw-frame tool; DockerDaemon
gains the cp), injecting per-destination-MAC frames from guest 1 while
guest 2 counts arrivals: ordinary multicast, STP, LACP, 802.1X and LLDP must
cross, PAUSE must not (the documented kernel limit). Asserts the anchors'
sysfs brport group_fwd_mask reads 0xfffd through the server's own wiring
(int-parsed: newer kernels print it hex), that the bridge level stays 0, and
that a container stop/start replays the mask with LACP surviving. Negative
control is the uBridge suite's setportgroupfwd 0 case.
The relay engines' observable paths had no live coverage: every
capture/start call in the suite sat on a kernel anchor (AF_PACKET
capture start_kernel) and markers were only asserted on marker
add_kernel. The two relay engines' own command shapes were unit-tested
only:
* bridge engine - bridge start_capture / bridge add_packet_filter ... mark
* iol_bridge engine - iol_bridge start_capture / iol_bridge add_packet_filter ... mark
Docker relay control (bridge engine, 1 passed 33s): the capture start
REST call writes the ICMP exchange to a pcap off the relay bridge; a
mark filter attached with capture_node_id n1 signals exactly 10
marker.match events over the dedicated marker WS for 5 pings (5
requests + 5 replies, both directions), carrying node_id/link_id/tag and
the busybox frame length, writes exactly those 10 frames to the marker
pcap, and deleting the marker takes the pcap with it.
IOU relay control (iol_bridge engine, 1 passed 173s): the same two
checks on iol_bridge start_capture and the port's mark filter, driven
by the real IOS ping through the fabric socket relay.
Not duplicated in the ESW/VPCS/cloud relay controls: they issue the same
uBridge bridge commands through their own call sites, which stay covered
by unit tests; the IOL container relay goes through the same iol_bridge
engine already exercised here.
The L2-anchor spec's behavioral guarantee — an idle link stays silent for
5 s once the enslavement burst settles — only ran in the Dynamips and
Docker suites. It now also runs in QEMU, IOU, the IOL container, the
switch fast path and the switch cascade, which covers every host-device
creation path:
* tap create (Dynamips/QEMU/IOU anchors), docker create_veth (Docker host
ends; the cascade's race-loser end is hardened after create by the
server), bridge add_nio_tap (the IOL container anchor, the fifth
creator), and the brctl switch bridges, whose multicast snooping is off
precisely so the bridge role can reach silence;
* each scenario silences its guests first (IOS interfaces shut through
the console; Docker guests' links down) and re-verifies the ping after;
* the cascade measures 8 devices at once (four absorbed router anchors,
both cascade ends, both switch bridges).
IOL 158 s, cascade 293 s, ESW fast path 378 s, QEMU 428 s, IOU 308 s —
all green.
The filter matrix (cls_bpf, the eBPF classifier) only ran on Docker veth
host ends; the IOU and IOL-container scenarios now spot-check it on their
TAP anchors and prove a filter survives a node restart:
* bpf "icmp" drops everything with clsact on the anchor, both ends, and
clears back to an empty qdisc;
* frequency_drop 3 (the eBPF every-nth mode) measured 56 % round-trip
loss (1 - (2/3)^2 = 55.6 % expected) in both scenarios;
* the IOU test applies delay 100 before a router stop/start: the fresh
anchor gets netem back (tc level) and the restarted router's pings
measure RTT >= 150 ms until the filter is cleared (a re-attach that
dropped the filter would show ~100 ms).
IOU file 492 s, IOL file 182 s, both green.
Extend the docker and IOU relay negative controls with the filters the
relay itself supports (delay, frequency_drop, bpf), asserted through real
ICMP — the first live coverage of the relay filter path:
* bridge engine (docker): delay 100 grows the RTT by ~200 ms and comes
back on clear; frequency_drop 2 kills the round trips (the filter counts
every packet crossing the bridge, both directions sharing one counter,
so on an alternating ping stream one whole direction dies; verified
against uBridge directly), frequency_drop -1 blackholes, and bpf drops
by libpcap expression ('icmp', then 'greater 150' discriminating by
frame size).
* iol_bridge engine (IOU): the same delay and frequency_drop effects
through the port's iol_bridge filter list, plus the available_filters
spot check.
Docker 33 s, IOU 168 s, both green.
Two Alpine containers on a kernel-only link with traffic-insight markers
attached to a veth anchor (marker add_kernel), observed on the dedicated
marker WebSocket via a new harness piece (WebSocketCollector, a sync
facade over websockets' sync client). Real ICMP proves: the marker.match
identity is complete (filter/tag/link_id/capture node/ts/len and the
tx/rx direction set of a ping), the main project channel carries no
matches, the BPF and direction are enforced (a non-matching sibling and
a tx-only marker each see exactly their traffic), the host pcap holds
exactly the matched frames, pausing silences one marker while a sibling
keeps counting (silence is never a dead channel), tag replay serves the
paused tag's timeline with display-filter/link/window narrowing and
refuses with 409 while capturing, and deleting a marker takes its pcap
and signals with it.
Two switches cascaded through the link-owned veth pair, two c7200
routers per switch, real ICMP crossing both bridges: the pair is born
UP with exactly one end per switch bridge, a dot1q trunk cascade
carries two VLANs at once, the symmetric access/trunk mode flip on the
cascade ends isolates and revives a VLAN in place (asymmetric flips are
semantically broken — the access side egresses untagged into the far
trunk's native PVID), a delay filter lands as netem on both ends (one
per direction, RTT ~= 2x delay), suspend admin-downs both ends, and a
link delete/re-create retires the whole pair and mints a new one under
the new link id — zero host residue after the project goes.
docker create_veth hardens only its FIRST end (the Docker shape: host
anchor plus a container leg that keeps its L3 life). A cascade end is
that second, unhardened end whenever the peer switch's create won the
race, and the kernel gives it an IPv6 link-local — a silent L3 identity
on the switch fabric (ubridge-l2-anchor-spec §E). Each switch now
hardens its OWN end with 'link l2only <end> on' once the pair exists,
whatever branch brought it there (fresh create, lost race, reuse);
idempotent, and best-effort with a warning on a uBridge without the
command.
The compute-side backstop of the controller's duplicate-port guard
(keep-port-links + live-port bind made the controller guard reliable;
this is the layer under it). IOU and Dynamips bound the adapter NIO
before the uBridge/hypervisor attach: a failed attach left the port's
binding replaced — the link that still owned the port lost its handle
and its host state leaked (the IOU EBUSY incident's 8 orphan TAPs).
Dynamips was worse: its wiring idempotence check silently skipped the
attach on a second binding and rebound only the bookkeeping. A port
that already carries an NIO is now refused, and both types bind the
NIO only after the wiring succeeded.
Link.delete cleared the port link through the port object stored at
add_node time; a node update since then rebuilt the port list, so that
object was stale and the live port kept its link — the next link on the
port was refused with "Port is already used". Masked until now by the
older rebuild bug (c2d68a65d): links never survived a rebuild, so the
guard never fired and the stale clear never mattered.
Node._list_ports() rebuilds every Port object, and it runs on each node
update — including every compute node.updated notification (a node
reporting started is one). The links are unaffected by that rebuild, but
the fresh port objects started out with link=None, so after any node
update the "Port is already used" guard in Link.add_node stopped matching:
a second link on an occupied port reached the compute, where the port's
NIO was replaced before the attach failed on the already-bound anchor,
leaving the link half-torn-down (on IOU, deleting the project afterwards
left the node's anchor TAPs and a per-link bridge on the host).
Carry the link back-references over by (adapter_number, port_number),
first match wins, mirroring get_port; a port that still exists keeps its
link. Regression tests pin both the reference (test_node) and the guard
(test_link: the duplicate link is refused after a node update).
Two real IOU routers on a real L3 image (the image's own IOL fabric, the
real IOS CLI on the server's telnet console), driving the anchor TAPs the
kernel datapath builds on: anchors born with the node (one per Ethernet
bay/unit), real ICMP through the [fabric <-> TAP] port bridge, netem
delay, suspend, capture, link delete/re-create, node stop/start rewiring,
the serial link staying on the relay (serial bays never anchor) and a
relay negative control on an isolated instance. Every scenario ends with
no anchors, bridges or tmpdirs left behind.
harness: create_iou_node/iou_images/iou_anchor_name, and the IOS console
helper is now ios_console (dynamips, IOU and the IOL containers share it).
Two real Alpine containers on an isolated instance, addressed the way a
GNS3 user does it (the node's persistent /etc/network/interfaces, applied
by init.sh) and driven through the Docker daemon socket the server itself
uses. Four scenarios:
* fast path: the per-link kernel bridge enslaves exactly the two veth host
ends, real ICMP crosses, the L2-anchor spec E.1/E.2 assertions hold
(idle silence measured with the guests' links down -- a container's own
boot-time IGMP report re-fires ~10 s later, so silencing is not just
IPv6), netem delay, suspend/resume, capture, link delete/re-create,
container stop/start (the veth host ends die with the adapter and are
rebuilt from the NIO), zero residue on project delete;
* filter matrix, one type at a time, each asserted through real traffic
and its tc state: netem core (delay, jitter+distribution, loss, corrupt,
duplicate), the extensions (reorder+correl+gap, seed+limit, gemodel,
rate), cls_bpf match-drop including a byte-size discriminator, the eBPF
classifier (frequency_drop every-Nth and -1, quota byte-cap cutoff,
window_drop single outage with per-seq pass/drop/pass evidence), plus
available_filters offering the kernel-only types;
* relay control: no per-link bridge, the kernel-only types hidden;
* relay->kernel reopen upgrade: the datapath choice is flipped across a
server restart and the project reopened -- the link is rebuilt on the
kernel datapath and forwards.
The harness gains a Docker guest driver (a stdlib Engine API client over
the daemon socket: exec with stream demux, container id by deterministic
name -- the controller's node payload does not carry it, image inspect),
a digest-pinned image helper (cache hit = no network, miss = pull through
the server's own route, neither = skip) and Server.restart() for the
reopen scenario (which also needs the local compute's capabilities to be
up before a project open is accepted).
Live: 4/4 scenarios, 142 s for the file; full e2e suite 10/10 (1503 s);
unit suite 2149 passed; ruff clean. The matrix measures loss with 20
packets at 0.2 s spacing (143 s -> 58 s) and the jitter RTT assertion
keeps statistical margin (per-direction N(50,20); 10 packets, >= 65 ms).
Without one every c7200 busy-waits a full CPU core for the whole scenario
-- the tests create raw nodes and no template supplies a value (measured:
constant ~100 % per router, up to ~6 minutes per dynamips scenario).
harness.dynamips_idlepc() detects a value through the controller's
auto_idlepc endpoint (the GUI idle-PC finder path: a throwaway project
boots the image, candidate values are validated by measuring actual CPU
usage) and caches it in ~/.cache/gns3-e2e/idlepc.json keyed by the image
checksum, so the cost (~100 s here) is paid once per image, not per run.
GNS3_E2E_IDLEPC overrides detection; any failure returns None and the
scenarios run without one rather than fail on a CPU optimisation.
All four scenarios that boot c7200 routers (dynamips kernel/relay,
ethernet switch kernel/relay) pass the value at creation. Live: detected
0x60189234 for the M11 image (238 s first run including detection); the
routers drop from ~100 % during boot to 3-8 % steady (sampled per
process), and a re-run reports the cached value with no detection.
The spec's gns3-server alignment section promised three assertions the
suites never carried; they land in harness.py and run in every kernel
scenario (plus the relay controls' anchor checks):
- E.1 assert_pure_l2: anchors and bridges carry no L3 identity (addrgenmode
none, no address in /proc/net/if_inet6, no ip -o -4 addr) -- the kernel
scenarios' anchors and per-link/switch bridges, relay-mode anchors and
the switch's relay TAPs alike (hardening is a creation-time property,
not a datapath one);
- E.2 assert_idle_silence (Dynamips suite): guest interfaces shut to
silence IOS's own CDP/keepalives, settle 2.5 s, then a 5 s window of
zero traffic over both anchors and the per-link bridge;
- assert_forwarding: every attached port reports brport/state == 3 -- a
bridge left DOWN keeps its ports disabled and forwards nothing, with no
error anywhere; the assertion is the only signal.
harness.l2only_supported() probes a throwaway uBridge once (link l2only
against a nonexistent device: 208 = command present, 202-Unknown command
= old build); without the command the assertions skip -- the spec's
degradation contract, mirroring the tc-capability skip.
The first live run caught a fifth-creator gap: the switch's relay port
TAPs (minted implicitly by bridge add_nio_tap) carried a live fe80::.
Delivered uBridge-side per the spec (fb72758: the transient TAP is
hardened at creation, attaches untouched); the server needed no change,
and the relay-control E.1 check is that creator's end-to-end coverage.
Two real iol-xe containers (Cisco CML iol-runner images, real IOS-XE on
the telnet console) against an isolated instance, the real uBridge with
the swappable bridge TAP leg:
* kernel link reports kernel_datapath; anchors exist from node start
and the per-link bridge enslaves exactly the two of them;
* real ICMP crosses the [unix <-> tap] port-bridge relay; delay 100
lands as netem on both anchors with the measured RTT growing >= 150 ms
and returning < 50 ms when cleared;
* suspend admin-downs the anchor and kills the traffic; resume
restores; AF_PACKET capture writes a real pcap;
* link delete/re-create swaps the port leg out and back in with the
ping returning; node stop leaves no gx devices and a start recreates
the whole wiring from the NIO; project delete cleans everything;
* the relay control on a relay-configured instance keeps the link off
the kernel and still pings (anchors may exist idle — the same
semantics as QEMU's and Dynamips' anchors: the server flag decides
attachment, not anchor birth).
Harness: create_iol_router (the appliance's template properties),
docker_images, iol_anchor_name, stage_iol_base_config (the shipped
iol-xe-base.txt copied into the isolated instance's configs dir — the
GNS3_IOL_STARTUP_CONFIG lookup path), boot_wait(straight_prompt=True)
for images that boot from NVRAM straight into the CLI with no 'Press
RETURN' sentinel, and GNS3_E2E_DEBUG=1 to run the isolated instance
with debug logging (the uBridge command stream lands in server.log —
how the ensure-anchor EBUSY was pinned down).
_is_unix_socket_docker no longer excludes the IOL runner marker: those
containers anchor on persistent TAPs now (compute side), so they
graduate through _kernel_endpoint_ready's gate instead — both
ubridge_bridge_tap (the swappable leg) and ubridge_tap (the tap module)
strict True, an unreported capability keeping the link on the relay
where it always works. The generic GNS3_UNIX_SOCKET_NIO exclusion keeps
its shape (an IOL runner environment wins the compute's class
selection, so the exclusion must not fire when both markers are set).
The switch fast path names an IOL runner peer's anchor through
kernel_anchor_type — the iol_docker key, never the gv veth namespace.
The 307d676cb regression tests move with the semantics: the capability
matrix replaces the unconditional relay, and the switch-path test also
asserts the gx naming. Its original concern stands — a compute that
cannot serve the anchor lifecycle must never be handed a kernel link
(the switch's deferred join would make it a silent dead cable).
IOL runner containers (IOLDockerVM, GNS3_IOL_RUNNER) had no kernel
datapath: their guest leg is the runner's AF_UNIX socket pairs, so every
link rode the uBridge relay. Their Ethernet bays now own persistent TAP
anchors (gx names, the iol_docker key of the shared naming contract —
born at node start like IOU's, before any link, so the switch fast
path's deferred join can never miss them), and the per-port bridge's
topology leg swaps between UDP (relay) and the anchor (kernel link)
through the frozen stop -> delete -> add -> start contract — the unix
binding never re-binds across a swap.
- iol_docker_vm: KernelDatapathMixin hooks, the anchor lifecycle
(_prepare_tap_datapath at _start_ubridge: probe ubridge_bridge_tap +
ubridge_tap, sweep leftover, tap create born down), the kernel attach
(ensure-then-add — tap create is strictly create-only, IFF_TUN_EXCL,
so the ensure checks /sys/class/net first: an unconditional create
answers EBUSY on the device the start loop just made), the release
(port bridge stop + delete_nio_tap before the shared teardown) and the
stop order the fd holders dictate (port bridges first, then tap
delete, then the per-link bridges — all before the hypervisor stops).
- vendor_docker_vm: the port-bridge creation factored into
_ensure_unix_port_bridge (idempotent on the registered name), keeping
the NIOBridge rejection for the generic unix-socket containers.
- docker_vm: the three _kernel_veths lookups go through the
_kernel_host_ifc hook — behavior-identical for DockerVM, and it is
what lets the subclass route without overriding the binding methods.
- kernel_anchor: the iol_docker key (gx) and kernel_anchor_type, the
controller-side mapping for naming an IOL runner's anchor on the
switch fast path (the marker check stays inline: this module sits
below everything and must not import the controller package).
The bridge module gained delete_nio_tap (uBridge's gns3server-integration
contract, section 'bridge — generic NIO relay with a swappable TAP leg'):
a per-port bridge can swap its topology leg between UDP and a persistent
TAP through stop -> delete -> add -> start, which is what lets an IOL
runner container's port bridge carry a kernel-link anchor.
The probe needs no scratch objects and no capabilities: one command
against a bridge that cannot exist — 202 (unknown command) is an old
build, 214 (bridge doesn't exist) is a new one. Cached by binary identity
like every other probe; any failure reports unknown.
GET /v3/compute/capabilities gains ubridge_bridge_tap; the controller's
default capability dict carries it too (the single schema definition, per
the 1e67fbf12 lesson).
Switch-to-switch links were the last shape left on the relay: the switch
fast path works by absorbing the *peer's* anchor, and a switch has none —
its ports are its own bridge, and a kernel interface belongs to exactly
one bridge, so two bridges cannot be joined by enslaving anything.
Join them with a veth pair instead: one end per bridge, each side
enslaving its own end and applying its own port mode to it — an access
port on one side and a trunk on the other behaves like the real cable it
models. uBridge's docker create_veth/delete_veth (the primitive every
Docker adapter is already born from) creates and destroys the pair; the
names derive from the link id (utils.kernel_anchor.kernel_cascade_names),
so each side computes its own end and the names survive a restart.
The pair belongs to the link, and the two ends are wired by two
independent uBridge processes — the project-open batch drives different
nodes concurrently — so the protocol is order-free: both sides create the
pair when their own end is missing (one create wins; the loser's failure
is re-checked against the kernel and "the pair is there" is not an
error), and either side's teardown destroys it (deleting one veth end
takes the other with it, so both converge on the same gone pair and the
second delif answers ENOENT). A pre-existing end — this switch restarted,
or a leftover from a previous life of the link id — is reused rather than
recreated, because the other end may still be a live port of the peer
switch; a stale tc qdisc is all a reused end can carry.
No new wire format was needed: a cascade is an anchor NIO on both ends,
the absorbed-anchor case plus the peer end's name. That also corrects the
filter routing — the single-sided rule exists because an absorbed anchor
is one interface, while a cascade has one per switch, so impairments go
to both ends like any other kernel link.
Tests: the wire format, the eligibility flip and the both-ends re-push on
node start (controller); the naming contract (utils); the switch-side
lifecycle — create, reuse, lost create race, delete destroys the pair
(compute routes).
Drive-by: the relay filter test's compute mock carried no awaitable
`put`, so tests/controller/test_kernel_datapath_link.py failed when run
alone while an argument-less full run passed; callers now get one.
The unix-socket container gate matched GNS3_UNIX_SOCKET_NIO in the node
environment only, but the IOL runner reaches the same wiring through the
GNS3_IOL_RUNNER marker: the compute selects IOLDockerVM on it, and that
class forces the AF_UNIX socket pairs whatever the generic knob says —
and the documented environment for those nodes carries only the marker.
Such a container was therefore handed a kernel link its compute cannot
serve. The plain case failed at link creation (no veth end to enslave);
on the switch fast path it was worse — the switch defers a join against a
missing anchor by design (a link to a stopped node must not fail), so the
cable was silently dead instead of merely slow.
Match both markers, through the shared is_iol_runner_environment helper so
the controller and the compute select the same way. Verified live on a
real server: docker↔IOL-XE and switch↔IOL-XE links stay relay while the
docker↔switch control link still takes the kernel datapath.
QEMU's process monitor calls stop() by itself when the process dies, so an
API stop and a process-death stop run concurrently in the normal case — and
the datapath release sat outside _execute_lock, letting both stops sweep the
adapter TAP map at the same time. Whichever finished first cleared
_kernel_taps under the other's iterator, which died with "dictionary changed
size during iteration" (the monitor's task, so it surfaced as "Task exception
was never retrieved") with the rest of its sweep, the orphan-bridge removal
and uBridge's shutdown skipped.
Keep the previous fix's ordering (process side first, then the devices) and
hold the lock across the release: serialized, the second stop finds uBridge
gone and sweeps nothing. Reproduced deterministically with two concurrent
stop() calls, the first parked mid-sweep — it raised at qemu_vm.py:1659
before the change, and every TAP is swept exactly once after it.
QEMU is the guest side of every adapter TAP, and uBridge's tap delete
refuses a device another process holds open ('Device or resource busy') —
a best-effort delete whose failure is suppressed. QEMU's stop did the
delete first and stopped the process after, so every node stop or project
delete leaked the persistent TAPs: found by inspecting the host after a
project delete (16 taps left behind, carrier down, no holder), and the
held-tap behaviour reproduced against uBridge directly.
Stop the QEMU process first, then remove the TAPs and the per-link kernel
bridges — process side first, then the devices, the same lesson as IOU's
reverse stop order. Verified live: node stop and project delete now leave
no taps and no processes behind, while a restart recreates the TAPs.
The adapter anchor naming also goes through the shared
utils.kernel_anchor helper, like the other node types.
The Ethernet switch's kernel fast path: instead of a link crossing the
switch's per-port uBridge relay, the switch absorbs the peer's anchor into
its own kernel bridge. The peer's interface (Docker veth host end, QEMU
TAP, IOU port TAP, Dynamips port TAP) becomes the switch port, and frames
cross anchor -> switch bridge -> switch ports entirely in the kernel.
Wire format: the switch end receives nio_anchor ('join this interface to
my own bridge' — the mirror of nio_bridge), the peer end nio_bridge with a
null bridge ('your anchor is bridged elsewhere': carrier, markers and
impairments only — never bridge membership, and never a bridge delete, on
an interface this node does not own). Anchor names come from the shared
naming contract (utils.kernel_anchor), a pure function of node type, id,
adapter and port — the controller computes the peer's anchor while the
node may be stopped, which is what makes links to stopped nodes wireable.
Stopped peers defer the join; after any node of a link starts
(UDPLink.node_started) the controller re-pushes the switch NIO, and the
switch completes — or repairs, when a peer restart replaced its interface
under the same name — the join against the kernel's actual membership
instead of assuming it.
VLAN changes now edit membership in place: only the ports whose mode/VLAN
actually changed are touched, each losing exactly the previous mode's
entries the new mode does not want (the dot1q admits-all entry is one
range operation in both directions), and no port ever leaves the bridge to
change its VLANs. This replaces the old 'reset by re-enslaving' shortcut
(delif+addif on every wired port, added when the ports were the switch's
own TAPs): on absorbed anchors, re-enslaving an interface owned by another
node is where live testing watched links die — anchors occasionally lost
their bridge membership right after a ports_mapping update (later updates
then failed with delif EINVAL), reproducibly with qemu, IOU and Dynamips
peers and never with Docker's veths.
Eligibility: the switch side needs no capability of its own (its bridge
exists by construction); the peer must be able to anchor. Switch-to-switch
cascades stay on the relay — a kernel interface belongs to one bridge, and
joining two bridges needs a veth pair, deferred.
Verified live on a real server: qemu peers through four VLAN cycles with
both anchors enslaved throughout; a mixed switch (2 IOU + 2 Dynamips, four
kernel ports); Docker peers with real ICMP isolating and recovering (with
both anchors still enslaved); VPCS relay ports likewise in place.
Live e2e verification moves into the repository instead of being rewritten
as a throwaway script every time: tests/e2e/ carries a shared harness (REST
client, isolated-instance lifecycle, a deliberately dumb IOS console
driver, host-side kernel inspection: bridge membership, tap admin state,
tc qdiscs) and each node type adds only its scenario module. The e2e marker
plus 'addopts = -m "not e2e"' keeps these privileged, image-dependent
tests out of normal runs; they are opt-in: pytest -m e2e tests/e2e -s.
The harness targets an existing server via GNS3_E2E_URL or starts an
isolated instance (own config dir — DB and paths follow it — on a free
port) and deletes its project afterwards (GNS3_E2E_KEEP=1 keeps it on
failure).
The first scenario drives two real c7200 routers on a real IOS image
through the REST API and the consoles: the kernel link is created while the
routers are stopped (deferred wiring), the per-link kernel bridge enslaves
exactly the two anchors, real ICMP crosses it, delay 100 shows netem on
both anchors and grows the RTT (~200 ms), suspend kills the traffic on an
admin-down anchor, capture writes a pcap of ICMP records, a serial link
between the same routers stays on the relay and still pings, delete/
re-create and stop/start leave the wiring consistent, and deleting the
project removes every anchor and bridge. A negative control runs the same
topology on an isolated relay-configured instance (no per-link bridge,
nothing enslaved, still pings).
A Dynamips Ethernet port can anchor a kernel link whenever its compute's
uBridge has the tap module — the anchors are uBridge-created TAPs the
hypervisor merely opens, the same capability QEMU gates on, so no
Dynamips-specific capability is needed. Serial, ATM and POS ports are
excluded per port by the existing link-type check (the controller's port
matrix decides), keeping them on the relay whatever the capabilities.
Eligibility tests cover the mixed pairs (docker/qemu/iou endpoints on the
same compute), the strict tap-capability requirement, and the serial
exclusion.
Dynamips links can now ride the kernel datapath like Docker, QEMU and IOU:
each Ethernet slot/port gets a persistent TAP anchor (gd{id}e{slot}p{port}),
created through uBridge's tap module, and the Dynamips hypervisor opens it
with nio create_tap — the hypervisor holds the fd, exactly QEMU's shape.
Links attach the anchor to their per-link kernel bridge; the legacy UDP
tunnel remains the relay datapath for every other port.
Anchor lifecycle: born at node start (tap module probed with one
create/delete cycle, sweep of stale devices, ownership handed to the server
user, born down); a kernel link opens the anchor in the hypervisor, binds
the slot/port and enslaves it; removing the link releases the fd again
(nio delete) and detaches. A link bound while the router is stopped is
wired by the next start (deferred wiring). Node close follows QEMU's stop
order — the hypervisor's TAP NIOs go before tap delete, since it is
Dynamips, not uBridge, that holds the fds.
Stop is not close: vm stop only halts the emulated router, so the
hypervisor, the TAP fds, the bindings and the per-link bridges all survive
a stop/start untouched (attach skips ports that still hold their TAP NIO).
Port selection follows the adapter registry: ETHERNET_ADAPTERS and
ETHERNET_WICS name the models whose ports carry Ethernet frames (WIC-1ENET
included, at its Dynamips port number); serial, ATM and POS ports never
anchor, and a kernel link there errors with the relay-datapath message
instead of half-wiring. Adapters hot-added to a running router (OIR) get
their anchors at once.
The compute NIO routes accept nio_bridge alongside nio_udp (the same
widening QEMU needed), carry the endpoint's suspend flag on update (it is
the anchor's admin state on a kernel link), skip the hypervisor delete for
a kernel NIO (its TAP NIO was already released by the slot removal), and
the close path leaves NIOBridge instances to _stop_ubridge.
The tool descriptions have always advertised kernel_datapath as a
requestable field, but the handler's whitelist did not contain it: asking
for it was rejected as an invalid field, so the MCP clients could not see
which links run on kernel interfaces at all.
Also record what seeding a netem experiment on IOU costs: IOL emits its own
CDP/keepalive frames and they draw from the netem RNG stream, so a
seeded run only reproduces after `no cdp run` (verified bit-identical pcaps).
GET /computes/{id} answered with only part of the capabilities the compute
reports: the response schema kept a second, private Capabilities class of
its own (it knew ubridge_tc and nothing else), and FastAPI's response model
silently drops fields it does not define — so ubridge_tap and
ubridge_iol_tap never reached clients even though the compute had sent
them and links were already being built on those datapaths.
Import the one definition instead of duplicating it, so the next capability
cannot drift the same way.
The IOU NIO update route never copied the body's suspend flag onto the
NIO; the Docker and QEMU routes always did. Harmless on the relay, where
suspend is emulated by the synthetic frequency_drop filter the controller
injects, but fatal on a kernel link, whose suspend *is* the anchor's
admin state: a suspended link kept forwarding traffic, and the resume
looked like it worked because nothing had ever stopped.
Eligibility gains an IOU branch gated on the compute's ubridge_iol_tap
(strict True: an unreported capability keeps the link on the relay), and
a port-level rule: a kernel link is an Ethernet segment, so any endpoint
whose port is not Ethernet (IOU serial) disqualifies the link before it
is dispatched. Without that, a serial link between two capable IOU nodes
would be sent as nio_bridge and fail at the compute.
The IOU NIO routes accept nio_bridge alongside the UDP/TAP/Ethernet NIOs
— without the widening the controller's POST is 422'd before the node
ever sees it, the same trap the QEMU routes hit.
Whether an IOU port can terminate on a persistent TAP (iol_bridge
add_nio_tap) is a capability of its own: the tap module and that command
land independently, so a build may have one without the other. Probed on
a scratch IOL bridge with a scratch TAP — one add/delete cycle, no
iol_bridge start needed (attaching a NIO works on a stopped bridge) —
cached by binary identity, with the scratch bridge id above the per-node
id space the fabric locks (513..1024) so a probe cannot collide with a
running node. Any failure, including an old build's "Unknown command",
means unknown: IOU stays on the relay datapath.
Reported through /capabilities as ubridge_iol_tap, next to ubridge_tap.
IOU's only wire is the IOL fabric (Unix datagrams, 8-byte headers,
NETMAP), so it has no host netdev to enslave — the kernel datapath gives
each Ethernet bay/unit a persistent TAP anchor the uBridge IOL bridge
binds the port to (iol_bridge add_nio_tap), which links then enslave into
their per-link kernel bridge exactly like Docker's veth and QEMU's TAP.
* IOUVM joins KernelDatapathMixin: one TAP per Ethernet bay/unit
(gi{id8}e{bay}p{unit}), created at start after a capability probe and a
stale sweep, born down and address-free; serial bays never anchor.
* _networking dispatches per NIO type: a relay link keeps iol_bridge
add_nio_udp, a kernel link binds the anchor then runs _kernel_attach.
* Stop order is QEMU's reverse, because uBridge holds the anchor fds:
iol_bridge delete (releases every port NIO) -> tap delete (a held TAP
answers EBADFD) -> brctl delete -> hypervisor stop. Removing a single
link releases its fd the same way (delete_nio_tap).
* The IOU-specific marker reconcile override is gone: marker primitives
are polymorphic on the anchor (marker *_kernel on a TAP, iol_bridge
commands on an IOL location), so the base reconcile serves both.
Capture on a kernel link anchors on the TAP (capture start_kernel);
filters are tc on the anchor, never combined with the userspace
iol_bridge filters a relay link uses.
uBridge delivered `link l2only` (e2e3155) and the brctl delete refusal
(83a84fb); verified from the server side -- anchors and per-link bridges
report addrgenmode none, carry no addresses, and an idle anchor is silent
once the one-shot membership burst has settled.
That burst is the one thing the command cannot remove, so it is now recorded
where the acceptance criteria live, with the measurement: enslaving a port
makes the kernel announce the bridge's multicast memberships (90 B MLDv2 at
+0.00 s, 54 B IGMPv3 at +0.09 s, 90 B MLDv2 at +0.73 s, then silence; no
addresses involved, and IPv4 has no per-device IGMP switch). Decision:
accepted and documented, not suppressed -- §E.2 measures idle silence after a
settle window instead.
Pins the other delivered behaviour with a test: a bridge now survives a
single-sided node stop, so brctl create answers EEXIST on the common path
rather than only in a create race -- the attach must not short-circuit there,
because the restarted side has a brand-new anchor whose filters and markers
still have to be re-applied.
_console_options() fell through the "none" branch and returned None, so
_build_command() blew up on `command.extend(None)`: a QEMU node with
console_type "none" -- offered by the GUI, and what API/MCP-driven labs use
to skip the console entirely -- failed to start with a 500 and a traceback.
Return an empty list, exactly like _aux_options() already does, and cover it
with a test that builds the command line for a headless node. Verified on a
live instance: such a node now starts (204) with `console: null` and no
-serial device in the QEMU command line.
Kernel eligibility becomes "both endpoints on one compute, each able to
anchor": Docker always (adapters are born as veth pairs), QEMU when that
compute's uBridge can create the persistent TAPs its adapters anchor on. The
question is asked per compute, because the link needs both ends anchored --
so a mixed or half-upgraded deployment degrades to the relay instead of
failing an attach.
probe_tap_support() creates and deletes one throwaway persistent TAP through a
scratch uBridge, cached by binary identity like the tc report; /capabilities
exposes `ubridge_tap` (absent = unknown, which keeps links on the relay).
The compute route for a QEMU adapter NIO took UDPNIO only, so a kernel link
was rejected with 422: its body and response now accept
Union[UDPNIO, BridgeNIO] exactly like Docker's.
Stale "Docker-to-Docker" wording in the config, link and MCP descriptions now
talks about anchors, and docs/features/qemu-kernel-datapath.md documents the
datapath -- verified end-to-end on a real instance: 32/32 kernel-datapath
checks and 29/29 on the relay variant.
QEMU had no host-side interface at all: every adapter was a local UDP tunnel
into uBridge (-netdev socket plus two UDP ports per adapter), so a link could
only ever be a userspace relay. Each adapter now owns a persistent TAP
(gq{node_id[:8]}e{adapter}p{port}), created by uBridge before QEMU is launched
and handed to this user, and the netdev is that TAP -- the anchor role Docker's
veth host end plays. Links attach to it:
* kernel link (NIOBridge): brctl create/up/addif enslaves the TAP into the
per-link bridge and capture, markers and tc filters are installed on it,
verbatim through the shared KernelDatapathMixin -- a TAP and a veth host end
are interchangeable there (an egress qdisc on either impairs traffic entering
the node, and PACKET_OUTGOING on either is the node receiving);
* relay link: bridge add_nio_ethernet plus add_nio_udp -- AF_PACKET, never the
TAP NIO, which would take the fd QEMU itself holds.
The netdev type is fixed at launch, so this is all-or-nothing per VM: on the
TAP datapath every link type attaches to (or detaches from) a running VM,
including switching a link between the two datapaths. Stop un-persists the
TAPs (QEMU's open fd keeps each device alive until it exits) and removes the
per-link bridges; start sweeps leftovers first. A uBridge without the tap
module keeps the previous socket-netdev datapath, so an unupgraded install is
unaffected -- such a node simply cannot carry kernel links.
Move the datapath-agnostic half of DockerKernelDatapathMixin into a new
KernelDatapathMixin (compute/kernel_datapath.py) so a second node type can
reuse it verbatim: the three marker primitives, the tc netem / cls_bpf / eBPF
appliers, the uBridge tc-capability probe, kernel marker teardown, link
detach, the per-link bridge cleanup and the adapter carrier -- plus
_kernel_attach(), lifted from DockerVM._connect_nio's kernel branch.
The mixin talks about "the anchor" (the host-side interface an adapter owns in
the root namespace) instead of "the veth host end": a node type provides
_kernel_host_ifc(), _kernel_anchors() and, when needed, _relay_carrier_bridge()
and _kernel_error(). Docker keeps only the veth lifecycle and those hooks;
behaviour is unchanged.
The C901 baseline entry moves with _netem_command_parts (the docker module is
no longer over the limit) and the bpf-compile-failure test patches the logger
of the module that now owns the code -- net zero new ignores.
Three fixes from live-calibrating the kernel impairment filters on an
isolated instance (Docker-to-Docker veth link, per-sequence evidence):
- update_filters skipped an unchanged filters dict as a no-op, so a
one-shot window_drop outage that had already passed could never be
re-armed: re-applying the same values produced no datapath command at
all. A PUT still carrying window_drop now reconciles, restarting the
schedule from the apply as documented. Measured: identical re-apply
moves the outage window from seq 50-70 to 70-91 (arm at +8s of the
new apply).
- gemodel parameter semantics were documented backwards in the
validation rules, available_filters, the MCP/copilot descriptions and
the feature docs. Measured against the kernel (per-direction counters,
offered = sent + dropped): p is the good-to-bad transition chance, r
the bad-to-good transition, 1-h the loss chance in the bad state (the
good state loses nothing) - mean one-way loss p/(p+r)*(1-h), e.g.
[10,30,100] -> 24.8%/25.7% measured vs 25% predicted, [5,45,100] ->
10.8% both ways vs 10%.
- kernel_datapath was silently stripped from every REST/MCP link payload
by the response model (LinkBase carried no such field), so clients
could not tell a kernel link from a relay one.
Also verified on the same setup: the 3-argument window_drop form works
exactly as specified ([3000,5000,100] -> a single seq 31-55 outage,
[20000,6000,100] -> seq 115-144), and both generations of the command
builder emit the parameters unpadded.
Adds an API test for the field and controller tests for the no-op skip
and the window_drop re-arm exemption.
LLM agents transcribe UUIDs unreliably — a 2-character slip invents an
id that never existed and is invisible to the eye. Tool parameters now
accept names wherever the server guarantees uniqueness: project names
among live projects, node names within a project. Links, drawings,
appliances and snapshots have no names and stay UUID-only.
Resolution sits in the single dispatch choke point (_run_handler_sync):
UUID-shaped values pass through untouched, names resolve with one list
call, and an unknown name fails fast listing the known names instead of
a confusing downstream 404. As a backstop for residual UUID typos,
error replies carrying "<UUID> doesn't exist" are matched against the
in-scope ids; a single candidate within edit distance 2 earns a
"Did you mean ...?" hint.
Runtime fact recomputed from the NIO wiring on every load (not
persisted). Without it clients cannot tell a relay link from a kernel
link whose compute lacks a filter capability — both show up as
kernel-only types missing from available_filters.
Consume uBridge's ebpf_modes contract (feature/tc-window 40c36ae): a
stateful mode is usable iff ebpf=1 and its token is listed; builds
predating the field keep their shipped modes (nth/quota/flow), so the
pre-correction window semantics can never be installed silently.
- gns3server/utils/tc_capabilities.py: shared reply parser, mode
policy and filter-type mappings (compute and controller gate on the
same rules)
- gns3server/compute/ubridge/tc_probe.py: standalone probe spawning a
throwaway uBridge as the server's own user, cached by binary
identity (path/mtime/size - the version string cannot distinguish
rebuilds); failures cache as unknown
- GET /v3/compute/capabilities gains ubridge_tc {netem, ebpf,
ebpf_modes, cbpf}; the controller forwards it (Compute.capabilities)
and available_filters hides kernel-datapath filter types an endpoint
compute cannot run (both ends apply their own filters, so one
incapable compute hides the type); computes without a report keep
the full list and apply-time 409 stays the guard
- docker: _ubridge_apply_ebpf_drops gates per mode token; the cleanup
path only sends "off" for modes the build declares
- split DockerVM's kernel-datapath machinery (veth lifecycle, carrier,
marker primitives, tc filter translation, kernel NIO teardown) into
compute/docker/docker_kernel_datapath.py (DockerKernelDatapathMixin)
The bridge teardown only hung off the link-removal path, but deleting a
node or closing a project never issues a per-link delete — the veths
disappeared with their containers and left the bridge behind as an empty
orphan (reproduced: create a kernel link, DELETE the project, the
gns3{link_id} bridge survives). `_remove_kernel_veths` now deletes the
node's kernel-NIO bridges alongside the veth host ends: both endpoints do
it, the peer's still-enslaved port answers EBUSY (suppressed, last one
wins) exactly like link deletion, a relay NIO's uBridge bridge is left
alone (it dies with the process), and a node restart rebuilds the bridge
from the NIO in _connect_nio.
Found while cleaning up the window_drop e2e: seven such bridges had
accumulated on this host. Re-verified live: the bridge is gone after the
project delete, and the window_drop e2e (which includes a node restart
with the kernel link attached) still passes 15/15.
uBridge's window mode gained the semantics the B.2 spec asked for —
`tc window_drop <if> <start> <outage> <pct> [<period> [<jitter>]]`: a
single outage that traffic passes before *and* after, recurring flaps
with period >= outage, and a per-cycle jitter — so the kernel-only
`window_drop` filter type is now exposed:
- validation mirrors the tc grammar (3-5 params, ranges, period >=
outage, jitter cap) and adds window_drop to KERNEL_ONLY_FILTERS;
filter_inactive_filters keeps start=0 — unlike every other filter a
zero first parameter means "the outage starts now", not "disabled".
- `_ubridge_apply_ebpf_drops` sends `tc window_drop` (positional
pass-through) and turns the mode off explicitly when absent, under the
same ebpf=1 gate.
- FILTERS entry (Start/Outage/Chance/Period/Jitter) + available_filters.
The schedule is relative to the apply: reconcile re-sets the classifier
on every apply, so any filter change, node restart or link reset restarts
the window.
Verified on a two-container kernel link, 15/15: a single outage
[1000, 1500, 100] arrives per-seq as [0..4, 13..19] (passing before the
window, dropped inside, passing after); recurring 1000/2000 ms = 52 %
round trip (duty cycle); period == outage = permanent outage; period <
outage = 409 before uBridge; jittered flap keeps the ~50 % duty cycle;
30 % chance on a permanent window = 53 % round trip against the
per-direction theory 1-0.7^2 = 51 %; coexist with netem; node restart
restores; available_filters kernel vs relay (409 + hidden); teardown
clean. P6b re-run as a regression on the same uBridge build: 13/13.
Worth recording for the next rebuild: the first delivered window program
loaded as root but reported ebpf=0 through gns3-server — the verifier's
non-root path loses the loop counter's bound, treats the catch-up loop as
unbounded and rejects the *whole* program (E2BIG, 8193 pending states),
which also took frequency_drop and quota down, with ebpf=0 as the only
symptom. Smaller trip caps did not help; the loop had to go (uBridge
572ca0c), on top of a Makefile header-dependency fix (53e34bb) that had
been shipping a stale program object.
Consume the eBPF impairment classifier delivered in uBridge
feature/tc-precision: frequency_drop becomes the exact every-Nth mode
(tc nth_drop; -1 = drop everything maps to every 1st) and the new
kernel-only quota type is the byte-cap mode (tc quota_drop, 100 = hard
cutoff). KERNEL_UNSUPPORTED_FILTERS is gone entirely — kernel/relay
eligibility is now purely topological. Both modes gate on ebpf=1 from
the per-process tc capabilities probe (error names the setcap fix); the
cleanup path (modes off) only talks to the eBPF commands when the probe
reported them. Reconcile is a plain re-set: the netem reset-before-set
already detaches the program and clears uBridge's per-interface
registry, so a mode set after it is a fresh load+attach.
Documented semantic difference: the relay counts both directions
through one bridge counter, the kernel counts per direction — N=3
round-trip loss is 1-(2/3)^2 = 55.6% on kernel vs ~1/N on relay
(measured 57%). window_drop is NOT exposed: the delivered
implementation advances the window with no gap, making 'outside the
window packets pass' unreachable (needs outage/period fields —
recorded in the spec); flow_drop awaits a parameter-shape decision.
Also fixes a pre-existing restart bug the e2e exposed: _bridges (relay
bridge names) was never cleared on stop, so restarting a node with a
relay link skipped 'bridge create' and failed on add_nio_udp.
e2e (isolated instance): 13/13 — pref-1 tc_impair classifier on both
veths, 57% round-trip for N=3, -1 = exact 100%, quota hard cutoff,
delay coexistence, clean teardown, suspend/resume, node restart,
relay-leg frequency_drop regression through an Ethernet switch. Full
suite: 1925 passed.
Surface the netem extensions delivered in uBridge feature/tc-netem-ext
as GNS3 filter types: rate (tc-style bandwidth string), reorder (needs
delay), gemodel (Gilbert-Elliot bursty loss, exclusive with
packet_loss), duplicate, seed and limit, plus the delay jitter
distribution and the loss/dup correlation parameters. All are
kernel-datapath only — the relay's packet-filter registry has no
equivalent, so available_filters hides them on relay links, setting one
on a created relay link returns 409, and the project-load path drops
them with a warning (update_filters runs before the datapath is
decided, so it cannot reject there). Validation mirrors the tc grammar:
reorder requires delay, gemodel and packet_loss are mutually exclusive,
a distribution requires jitter > 0, rate is integer+unit capped at
100gbit.
The translation in DockerVM._netem_command_parts emits the frozen
grammar order and gates extension keywords on the 'tc capabilities'
netem token list (probed once per uBridge process; a plain
delay/loss/corrupt/dup filter never probes, keeping old uBridge builds
untouched; the correl suffix has no token and gates on 'rate' as the
extension-build marker).
Reconcile became reset-before-set: the kernel's netem change merges
optional attributes (rate, correlation, reorder, corrupt, gemodel,
distribution, seed keep their previous value when the new message omits
them), so removing a parameter needs an explicit 'tc reset' — the
bpf_drop filters are re-added right after in the same flow, so the
clsact churn is invisible. The capability check runs before the reset
so a failed update leaves the previous qdisc in place.
e2e (isolated instance, kernel link): 24/24 — rate 512kbit adds exactly
2x the per-packet serialization to the RTT (45.5 ms for 1400-byte
pings), paretonormal jitter + rate-cleared leak check, reorder gap 5,
gemodel 50% bursty loss, duplicate, loss 30% correl 50%, seed 42 +
limit 5000 visible in the qdisc dump, suspend/resume and node restart
keep the filters, relay link rejects rate with 409, teardown leaves no
netem. Full unit suite: 1916 passed.
The unified veth host end is born admin-down (carrier off until a link
attaches), and libpcap cannot open a packet socket on an admin-down
interface — its netlink promiscuous-mode transaction returns ENOENT, so
'bridge add_nio_ethernet' failed on every runtime relay link (and would
on any relay link at container start). Bring the interface up for the
attach; the carrier pass in the caller refines the state afterwards (a
suspended NIO sets it back down). Verified live with a direct uBridge
probe: attach on a down veth -> ENOENT, after 'link set up' -> OK.
The GNS3 bpf filter (one expression per line, any line matching drops the
packet) now runs on the kernel datapath: uBridge's tc bpf_drop compiles
each line with libpcap (same compiler as the relay filter) into classic
BPF, attaches it to the veth host end's clsact egress as a cls_bpf
classifier with a gact TC_ACT_SHOT action, and the kernel migrates the
cBPF to eBPF internally -- no CAP_BPF needed.
- reconcile is flush + one add per line at priorities 10, 11, ... (OR
semantics, same as the relay's per-line filters); tc reset (link
delete / clear) is the full restore in uBridge and clears them
- capability-gated: the compute probes "tc capabilities" once per uBridge
process (cached, invalidated on stop); without cbpf=1 the filter apply
fails with an actionable upgrade error, and old uBridge builds without
the tc module are never touched when no bpf filter is present
- a line that compiled for the controller's tcpdump but not for the
compute's uBridge libpcap is skipped with a warning, mirroring the
relay path
- KERNEL_UNSUPPORTED_FILTERS shrinks to frequency_drop (its eBPF stateful
classifier is still pending, spec part B)
e2e (22/22): bpf "icmp" = 100% loss; size-discriminating expression
("greater 150") passes 98-byte and drops 300-byte pings (real byte-level
match); two-line OR; coexistence with netem (delay 50 -> small pings
100.2 ms while big dropped); clear removes clsact; suspend/resume keeps
the drops; link delete + recreate leaves no residual filters.
Impairment filters with a netem equivalent (delay, packet_loss, corrupt)
now run on the kernel datapath as one tc netem qdisc per veth host end,
pushed through uBridge's netlink tc module (tc netem set / tc reset).
This removes the relay delay filter's nanosleep bottleneck (upstream
Both link endpoints carry the filters: each end's qdisc egress covers
the traffic entering that container, so every direction is impaired
exactly once -- the same net effect as the relay, where both directions
cross the single filtered bridge.
- controller: eligibility keys on relay-only types (frequency_drop, bpf)
instead of any filter; kernel links push the real filters (the
synthetic suspend frequency_drop no longer flips a suspended link to
relay on reopen/reset); update_filters 409s only relay-only types;
available_filters hides them on kernel links
- compute: create_nio whitelists netem-capable filter types; the NIO
update path keeps its own guard (PUT mutates the NIO in place);
_ubridge_apply_netem translates the filter dict into one atomic
netem-set (full re-apply IS the reconcile); node restart restores the
qdisc, link deletion detaches it (the veth survives the link)
- e2e (27/27): delay 100 -> 200.2 ms RTT, atomic replace 50 -> 100.2 ms,
clear -> baseline, packet_loss 30 -> 48 % round-trip (expected 51 %),
suspend/resume and node restart keep the qdisc, link delete leaves no
residual impairment
