Resolve conflicts against the upstream wave (PRs #2952-#2965, incl. the
typed-response/schema hardening and the drop of Python 3.10):
- controller/node.py: keep upstream's _transitioning/_confirm_status
lifecycle guard; the kernel-anchor link wiring (node_started) runs
after the transition completes.
- controller/link.py: keep upstream's add_node rollback (release_ports
on create failure) and delete via release_ports; release_ports now
resolves ports live (same staleness reason as _bind_members) and
discards the node link unconditionally.
- schemas/controller/links.py: extend LinkFilterType with the eight
kernel-datapath filter types and LinkFilterParameter.type with "str"
so the newly typed /available_filters response validates; Link.filters
stays a free-form dict — the vocabulary is capability-gated per link
and validated by the controller (409), not by a strict request schema.
- schemas/controller/topology.py: drop the type: ignore made redundant
by the dict filters field.
- tests: route-level filter validation now expects the controller's 409
(unknown types, out-of-range values); packet_loss correlation is a
valid kernel parameter (positive test added); OpenAPI assertions match
the free-form filters property.
The persistent-TAP lifecycle — probe, sweep, create, register, delete —
was implemented four times, once per node type, and the copies had drifted
(probe mechanism, _tap_datapath reset, keep-vs-recreate policy). The
mechanical half moves into KernelDatapathMixin:
- _probe_persistent_taps(probe_name): the scratch-TAP probe, returning the
refusal for the caller's own warning (QEMU, Dynamips);
- _create_anchor_taps(ports, set_owner, keep_existing): sweep + create +
optional ownership handover + carrier down + registry, with the ports
enumerated by the node type and named through the new _tap_name hook;
- _delete_anchor_taps(): the suppressed tap sweep + per-link bridge
cleanup + registry clear.
Each node type keeps what its fd holder dictates: the release step before
the tap sweep (the QEMU process, the IOL bridge, the hypervisor's NIO
bindings, the container port bridges), its probe policy, and its registry
lifetime. QEMU's now-unconditional registry clear aligns it with the other
three; everything else is behavior-preserving.
Compute:
- Track the anchor that owns uBridge's single AF_PACKET capture slot: a
second kernel port's capture is refused with a clear error (its flag
rolled back), a failed start no longer wedges the port, and only the
owning port's stop issues the process-wide `capture stop_kernel`
(docker/qemu/iou/dynamips/switch, via the mixin's slot helpers).
- Unwind failed starts instead of leaking uBridge and the anchor TAPs:
QEMU's stop body runs under try/finally, QEMU and IOU start the datapath
inside the launch try, Dynamips undoes a half-attached port (the retry
used to be short-circuited into a silent black hole), and Docker deletes
the veth pair when the namespace move fails.
- Restore the relay filter guard on IOU (kernel-only filters raise
NodeError, an uncompilable BPF line degrades to a warning), validate the
eBPF/quota/window/bpf filter shapes before touching the bridge (short or
non-numeric values surfaced as 500s after partial state), and refuse a
second NIO on an occupied Docker/QEMU port (the guard IOU and Dynamips
already had).
- Probe fixes: `_probe_bridge_tap` survives a failing uBridge start, and
the probe caches key on ctime too, so a chmod/setcap remedy re-probes.
Controller:
- Parse GNS3_UNIX_SOCKET_NIO like the compute does (line-anchored, truthy
value) and mirror the compute's class selection in kernel_anchor_type.
- Read partial capability reports defensively; make BridgeNIO.bridge
required-but-nullable so an omitted key is a 422 rather than a KeyError.
- Strip the packet_loss correlation slot even when it is zero: the relay
filter takes exactly one argument and rejected `packet_loss 10 0`.
Ethernet switch: skip ports whose absorbed anchor does not exist yet (the
join is deferred by design) and roll the ports_mapping back when the VLAN
reconcile fails, so a retry still diffs against the old mapping.
Agent: reject device config entries without a device_name instead of
silently dropping them (a malformed batch used to execute partially while
reporting success).
Also split the netem builder to clear C901 and drop the two per-file ruff
ignores the PR had added, keeping the remove-only policy.
The rebase onto 65c858d45 brought upstream's UP032 enforcement; the
conflict resolution kept this branch's semantics on the overlapping
lines, so their .format() calls needed the conversion afterwards
(ruff check --fix, 41 sites) plus the formatter's rewrap (7 files).
Replicates the CI recipe locally: pip check, ruff check/format, mypy
2.3.1 (clean, 434 files), unit suite 2216 passed.
The rebase onto upstream 3.1 (mypy + ruff I001/F401 waves) left the tree
short of the newly enforced baselines in three ways:
* contextlib / UbridgeError imports dropped as false duplicates during
conflict resolution left NameErrors in iou_vm and qemu_vm (F821);
the Dynamips router grabbed the compute-level NIOTAP instead of the
hypervisor one, breaking TAP anchoring (test failure).
* unused imports (KERNEL_UNSUPPORTED_FILTERS, UbridgeError, List) and
unsorted blocks left behind where the stack's later commits moved the
last usage elsewhere — ruff I001/F401/RUF022 now enforce these.
* mypy, now baseline-free: cache dict annotations in tc_probe, the
locators candidate list, the resolved-params None branch in the MCP
runner, and validate_bpf_syntax keeps the upstream Dict[str, Any]
return (the strict Optional[str] was wrong for the "valid" bool entry).
The e2e raw-frame probe resolves its compiler via shutil.which (S607)
and zips with strict= (B905); the remaining subprocess call is S603 by
nature and gets a per-file ignore like other test files.
ruff check/format clean, mypy at upstream parity (no new errors),
full unit suite 2167 passed.
The peer end of an Ethernet switch fast-path link is a NIOBridge with no
bridge of its own (the switch absorbed its anchor into the switch bridge).
Its update reconcile still ran the impairment pass with the empty filter
set the controller routes to that end — and an empty set is a plain
`tc reset` on the shared anchor, silently wiping the netem/cls_bpf/eBPF
the switch had just applied. Any link update did it (filters, markers,
suspend, capture start/stop), and the outcome depended on the link's
endpoint order: UdpLink.update() PUTs node1 then node2, so with the
switch first its apply ran first and the peer's reset ran last (API
green, data plane unimpaired), while with the peer first the reset ran
first and the filter survived. That order dependence is also why the
shipped e2e never caught it — it only created peer-first links.
Ownership is now explicit: only the end that owns the anchor's tc state
applies the impairment reconcile (_owns_anchor_impairments — a NIOBridge
with `bridge is None` is a passive carrier, the same idiom the
bridge-delete path already used). Markers, capture and the suspend
carrier still apply on both ends: markers reconcile per process, so
neither end can strip the other's, and the carrier converges by
agreement.
Red/green: two new docker unit tests fail on the unfixed tree with
`tc reset "<anchor>"` and pass after; the switch e2e also creates one
link switch-first and asserts the netem qdisc and the RTT hold under
that order (failed with the wipe, passes with the fix). Full unit suite
2110 passed; the switch e2e file passed end to end (378s).
A correctness read-through of the five delivered datapath documents, plus
the two code comments that carried the same stale vocabulary.
Datapath selection: the VendorDockerVM selector is `console_type=docker_exec`
alone (`GNS3_SKIP_INIT` and friends are behavior knobs of the chosen class);
the cross-compute and anchor-less fallbacks are named for what they are; the
Ethernet-switch endpoint is the documented exception to the anchoring rule,
and the IOL runner container is called out as the Docker node that does not
anchor on a veth.
Filter vocabulary: the stateful eBPF program evaluates four modes
(nth → quota → window → flow), GNS3 exposes three of them, and window_drop
joins the kernel-only list; the correl suffixes gate on the `rate` token as
the extension-build marker.
group_fwd_mask: the 0xfffd port mask is applied inside uBridge's `brctl addif`
itself, not as a separate command — the link-creation sequences and the
operation tables no longer show an arrow the server never sends. Same class
of fix for `link set up` on the IOL anchor (addif already brought it up; the
command only pushes a suspended anchor back down) and for the QEMU carrier
replication (QMP `set_link`, gated by `replicate_network_connection_state`).
Lifecycles and stop orders re-worded around the invariant rather than the
node type: the fd holder must release before the devices are deleted, which
is QEMU's process-first order, IOU's uBridge-first order and Dynamips'
hypervisor-NIO-first order. Deferred wiring for a stopped endpoint is
documented as the common path it is, and the Dynamips "stop is not a close"
note now says what recovery from a dead hypervisor actually requires.
Dynamips specifics: a c7200 needs an adapter in slot 1 (slot 0 is the fixed
I/O slot), the tap-fd direction is stated correctly (writes inject into the
kernel, reads return what the kernel transmits), and the anchor registry is
name-keyed against the controller's port matrix by convention, not derivation.
IOL port bridge naming: `bridge{bay}` on unit 0, `bridge{bay}_{unit}`
elsewhere; the leg swap happens on link delete/re-create or a datapath
switch, while suspend is an in-place NIO update and a project reopen is a
fresh build.
Verified and Roadmap sections: the in-repo e2e suite is distinguished from
the isolated-instance development harness (IOU), assertions are stated with
their bounds and capability gates, and the roadmap entries collapse into the
one remaining project — VXLAN/GENEVE for cross-compute links.
Code comments: `flow` added to the documented mode order in the eBPF apply
path, and the `nio_bridge` module docstring corrected on which filter types
run on the kernel datapath (bpf via cls_bpf, frequency_drop/quota/window_drop
via the eBPF classifier — neither is relay-only any more).
docker create_veth hardens only its FIRST end (the Docker shape: host
anchor plus a container leg that keeps its L3 life). A cascade end is
that second, unhardened end whenever the peer switch's create won the
race, and the kernel gives it an IPv6 link-local — a silent L3 identity
on the switch fabric (ubridge-l2-anchor-spec §E). Each switch now
hardens its OWN end with 'link l2only <end> on' once the pair exists,
whatever branch brought it there (fresh create, lost race, reuse);
idempotent, and best-effort with a warning on a uBridge without the
command.
The compute-side backstop of the controller's duplicate-port guard
(keep-port-links + live-port bind made the controller guard reliable;
this is the layer under it). IOU and Dynamips bound the adapter NIO
before the uBridge/hypervisor attach: a failed attach left the port's
binding replaced — the link that still owned the port lost its handle
and its host state leaked (the IOU EBUSY incident's 8 orphan TAPs).
Dynamips was worse: its wiring idempotence check silently skipped the
attach on a second binding and rebound only the bookkeeping. A port
that already carries an NIO is now refused, and both types bind the
NIO only after the wiring succeeded.
The spec's B table grows its fifth row: bridge add_nio_tap mints a
transient TAP when the name is free and hardens it at creation, while an
attach to an existing device (possibly a user-owned TAP with addresses)
is left untouched. The rule paragraph spells out attach-is-not-a-creation.
Delivery: uBridge fb72758, closing the gap the new e2e assertions found
(live: the switch's gns3*-0 relay TAP carried fe80::). The server issues
no link l2only of its own -- the switch call site now documents that the
TAP is minted-and-hardened by add_nio_tap, and the ESW feature doc
describes the uBridge-side close instead of a server workaround.
IOL runner containers (IOLDockerVM, GNS3_IOL_RUNNER) had no kernel
datapath: their guest leg is the runner's AF_UNIX socket pairs, so every
link rode the uBridge relay. Their Ethernet bays now own persistent TAP
anchors (gx names, the iol_docker key of the shared naming contract —
born at node start like IOU's, before any link, so the switch fast
path's deferred join can never miss them), and the per-port bridge's
topology leg swaps between UDP (relay) and the anchor (kernel link)
through the frozen stop -> delete -> add -> start contract — the unix
binding never re-binds across a swap.
- iol_docker_vm: KernelDatapathMixin hooks, the anchor lifecycle
(_prepare_tap_datapath at _start_ubridge: probe ubridge_bridge_tap +
ubridge_tap, sweep leftover, tap create born down), the kernel attach
(ensure-then-add — tap create is strictly create-only, IFF_TUN_EXCL,
so the ensure checks /sys/class/net first: an unconditional create
answers EBUSY on the device the start loop just made), the release
(port bridge stop + delete_nio_tap before the shared teardown) and the
stop order the fd holders dictate (port bridges first, then tap
delete, then the per-link bridges — all before the hypervisor stops).
- vendor_docker_vm: the port-bridge creation factored into
_ensure_unix_port_bridge (idempotent on the registered name), keeping
the NIOBridge rejection for the generic unix-socket containers.
- docker_vm: the three _kernel_veths lookups go through the
_kernel_host_ifc hook — behavior-identical for DockerVM, and it is
what lets the subclass route without overriding the binding methods.
- kernel_anchor: the iol_docker key (gx) and kernel_anchor_type, the
controller-side mapping for naming an IOL runner's anchor on the
switch fast path (the marker check stays inline: this module sits
below everything and must not import the controller package).
The bridge module gained delete_nio_tap (uBridge's gns3server-integration
contract, section 'bridge — generic NIO relay with a swappable TAP leg'):
a per-port bridge can swap its topology leg between UDP and a persistent
TAP through stop -> delete -> add -> start, which is what lets an IOL
runner container's port bridge carry a kernel-link anchor.
The probe needs no scratch objects and no capabilities: one command
against a bridge that cannot exist — 202 (unknown command) is an old
build, 214 (bridge doesn't exist) is a new one. Cached by binary identity
like every other probe; any failure reports unknown.
GET /v3/compute/capabilities gains ubridge_bridge_tap; the controller's
default capability dict carries it too (the single schema definition, per
the 1e67fbf12 lesson).
Switch-to-switch links were the last shape left on the relay: the switch
fast path works by absorbing the *peer's* anchor, and a switch has none —
its ports are its own bridge, and a kernel interface belongs to exactly
one bridge, so two bridges cannot be joined by enslaving anything.
Join them with a veth pair instead: one end per bridge, each side
enslaving its own end and applying its own port mode to it — an access
port on one side and a trunk on the other behaves like the real cable it
models. uBridge's docker create_veth/delete_veth (the primitive every
Docker adapter is already born from) creates and destroys the pair; the
names derive from the link id (utils.kernel_anchor.kernel_cascade_names),
so each side computes its own end and the names survive a restart.
The pair belongs to the link, and the two ends are wired by two
independent uBridge processes — the project-open batch drives different
nodes concurrently — so the protocol is order-free: both sides create the
pair when their own end is missing (one create wins; the loser's failure
is re-checked against the kernel and "the pair is there" is not an
error), and either side's teardown destroys it (deleting one veth end
takes the other with it, so both converge on the same gone pair and the
second delif answers ENOENT). A pre-existing end — this switch restarted,
or a leftover from a previous life of the link id — is reused rather than
recreated, because the other end may still be a live port of the peer
switch; a stale tc qdisc is all a reused end can carry.
No new wire format was needed: a cascade is an anchor NIO on both ends,
the absorbed-anchor case plus the peer end's name. That also corrects the
filter routing — the single-sided rule exists because an absorbed anchor
is one interface, while a cascade has one per switch, so impairments go
to both ends like any other kernel link.
Tests: the wire format, the eligibility flip and the both-ends re-push on
node start (controller); the naming contract (utils); the switch-side
lifecycle — create, reuse, lost create race, delete destroys the pair
(compute routes).
Drive-by: the relay filter test's compute mock carried no awaitable
`put`, so tests/controller/test_kernel_datapath_link.py failed when run
alone while an argument-less full run passed; callers now get one.
QEMU's process monitor calls stop() by itself when the process dies, so an
API stop and a process-death stop run concurrently in the normal case — and
the datapath release sat outside _execute_lock, letting both stops sweep the
adapter TAP map at the same time. Whichever finished first cleared
_kernel_taps under the other's iterator, which died with "dictionary changed
size during iteration" (the monitor's task, so it surfaced as "Task exception
was never retrieved") with the rest of its sweep, the orphan-bridge removal
and uBridge's shutdown skipped.
Keep the previous fix's ordering (process side first, then the devices) and
hold the lock across the release: serialized, the second stop finds uBridge
gone and sweeps nothing. Reproduced deterministically with two concurrent
stop() calls, the first parked mid-sweep — it raised at qemu_vm.py:1659
before the change, and every TAP is swept exactly once after it.
QEMU is the guest side of every adapter TAP, and uBridge's tap delete
refuses a device another process holds open ('Device or resource busy') —
a best-effort delete whose failure is suppressed. QEMU's stop did the
delete first and stopped the process after, so every node stop or project
delete leaked the persistent TAPs: found by inspecting the host after a
project delete (16 taps left behind, carrier down, no holder), and the
held-tap behaviour reproduced against uBridge directly.
Stop the QEMU process first, then remove the TAPs and the per-link kernel
bridges — process side first, then the devices, the same lesson as IOU's
reverse stop order. Verified live: node stop and project delete now leave
no taps and no processes behind, while a restart recreates the TAPs.
The adapter anchor naming also goes through the shared
utils.kernel_anchor helper, like the other node types.
The Ethernet switch's kernel fast path: instead of a link crossing the
switch's per-port uBridge relay, the switch absorbs the peer's anchor into
its own kernel bridge. The peer's interface (Docker veth host end, QEMU
TAP, IOU port TAP, Dynamips port TAP) becomes the switch port, and frames
cross anchor -> switch bridge -> switch ports entirely in the kernel.
Wire format: the switch end receives nio_anchor ('join this interface to
my own bridge' — the mirror of nio_bridge), the peer end nio_bridge with a
null bridge ('your anchor is bridged elsewhere': carrier, markers and
impairments only — never bridge membership, and never a bridge delete, on
an interface this node does not own). Anchor names come from the shared
naming contract (utils.kernel_anchor), a pure function of node type, id,
adapter and port — the controller computes the peer's anchor while the
node may be stopped, which is what makes links to stopped nodes wireable.
Stopped peers defer the join; after any node of a link starts
(UDPLink.node_started) the controller re-pushes the switch NIO, and the
switch completes — or repairs, when a peer restart replaced its interface
under the same name — the join against the kernel's actual membership
instead of assuming it.
VLAN changes now edit membership in place: only the ports whose mode/VLAN
actually changed are touched, each losing exactly the previous mode's
entries the new mode does not want (the dot1q admits-all entry is one
range operation in both directions), and no port ever leaves the bridge to
change its VLANs. This replaces the old 'reset by re-enslaving' shortcut
(delif+addif on every wired port, added when the ports were the switch's
own TAPs): on absorbed anchors, re-enslaving an interface owned by another
node is where live testing watched links die — anchors occasionally lost
their bridge membership right after a ports_mapping update (later updates
then failed with delif EINVAL), reproducibly with qemu, IOU and Dynamips
peers and never with Docker's veths.
Eligibility: the switch side needs no capability of its own (its bridge
exists by construction); the peer must be able to anchor. Switch-to-switch
cascades stay on the relay — a kernel interface belongs to one bridge, and
joining two bridges needs a veth pair, deferred.
Verified live on a real server: qemu peers through four VLAN cycles with
both anchors enslaved throughout; a mixed switch (2 IOU + 2 Dynamips, four
kernel ports); Docker peers with real ICMP isolating and recovering (with
both anchors still enslaved); VPCS relay ports likewise in place.
Dynamips links can now ride the kernel datapath like Docker, QEMU and IOU:
each Ethernet slot/port gets a persistent TAP anchor (gd{id}e{slot}p{port}),
created through uBridge's tap module, and the Dynamips hypervisor opens it
with nio create_tap — the hypervisor holds the fd, exactly QEMU's shape.
Links attach the anchor to their per-link kernel bridge; the legacy UDP
tunnel remains the relay datapath for every other port.
Anchor lifecycle: born at node start (tap module probed with one
create/delete cycle, sweep of stale devices, ownership handed to the server
user, born down); a kernel link opens the anchor in the hypervisor, binds
the slot/port and enslaves it; removing the link releases the fd again
(nio delete) and detaches. A link bound while the router is stopped is
wired by the next start (deferred wiring). Node close follows QEMU's stop
order — the hypervisor's TAP NIOs go before tap delete, since it is
Dynamips, not uBridge, that holds the fds.
Stop is not close: vm stop only halts the emulated router, so the
hypervisor, the TAP fds, the bindings and the per-link bridges all survive
a stop/start untouched (attach skips ports that still hold their TAP NIO).
Port selection follows the adapter registry: ETHERNET_ADAPTERS and
ETHERNET_WICS name the models whose ports carry Ethernet frames (WIC-1ENET
included, at its Dynamips port number); serial, ATM and POS ports never
anchor, and a kernel link there errors with the relay-datapath message
instead of half-wiring. Adapters hot-added to a running router (OIR) get
their anchors at once.
The compute NIO routes accept nio_bridge alongside nio_udp (the same
widening QEMU needed), carry the endpoint's suspend flag on update (it is
the anchor's admin state on a kernel link), skip the hypervisor delete for
a kernel NIO (its TAP NIO was already released by the slot removal), and
the close path leaves NIOBridge instances to _stop_ubridge.
Whether an IOU port can terminate on a persistent TAP (iol_bridge
add_nio_tap) is a capability of its own: the tap module and that command
land independently, so a build may have one without the other. Probed on
a scratch IOL bridge with a scratch TAP — one add/delete cycle, no
iol_bridge start needed (attaching a NIO works on a stopped bridge) —
cached by binary identity, with the scratch bridge id above the per-node
id space the fabric locks (513..1024) so a probe cannot collide with a
running node. Any failure, including an old build's "Unknown command",
means unknown: IOU stays on the relay datapath.
Reported through /capabilities as ubridge_iol_tap, next to ubridge_tap.
IOU's only wire is the IOL fabric (Unix datagrams, 8-byte headers,
NETMAP), so it has no host netdev to enslave — the kernel datapath gives
each Ethernet bay/unit a persistent TAP anchor the uBridge IOL bridge
binds the port to (iol_bridge add_nio_tap), which links then enslave into
their per-link kernel bridge exactly like Docker's veth and QEMU's TAP.
* IOUVM joins KernelDatapathMixin: one TAP per Ethernet bay/unit
(gi{id8}e{bay}p{unit}), created at start after a capability probe and a
stale sweep, born down and address-free; serial bays never anchor.
* _networking dispatches per NIO type: a relay link keeps iol_bridge
add_nio_udp, a kernel link binds the anchor then runs _kernel_attach.
* Stop order is QEMU's reverse, because uBridge holds the anchor fds:
iol_bridge delete (releases every port NIO) -> tap delete (a held TAP
answers EBADFD) -> brctl delete -> hypervisor stop. Removing a single
link releases its fd the same way (delete_nio_tap).
* The IOU-specific marker reconcile override is gone: marker primitives
are polymorphic on the anchor (marker *_kernel on a TAP, iol_bridge
commands on an IOL location), so the base reconcile serves both.
Capture on a kernel link anchors on the TAP (capture start_kernel);
filters are tc on the anchor, never combined with the userspace
iol_bridge filters a relay link uses.
_console_options() fell through the "none" branch and returned None, so
_build_command() blew up on `command.extend(None)`: a QEMU node with
console_type "none" -- offered by the GUI, and what API/MCP-driven labs use
to skip the console entirely -- failed to start with a 500 and a traceback.
Return an empty list, exactly like _aux_options() already does, and cover it
with a test that builds the command line for a headless node. Verified on a
live instance: such a node now starts (204) with `console: null` and no
-serial device in the QEMU command line.
Kernel eligibility becomes "both endpoints on one compute, each able to
anchor": Docker always (adapters are born as veth pairs), QEMU when that
compute's uBridge can create the persistent TAPs its adapters anchor on. The
question is asked per compute, because the link needs both ends anchored --
so a mixed or half-upgraded deployment degrades to the relay instead of
failing an attach.
probe_tap_support() creates and deletes one throwaway persistent TAP through a
scratch uBridge, cached by binary identity like the tc report; /capabilities
exposes `ubridge_tap` (absent = unknown, which keeps links on the relay).
The compute route for a QEMU adapter NIO took UDPNIO only, so a kernel link
was rejected with 422: its body and response now accept
Union[UDPNIO, BridgeNIO] exactly like Docker's.
Stale "Docker-to-Docker" wording in the config, link and MCP descriptions now
talks about anchors, and docs/features/qemu-kernel-datapath.md documents the
datapath -- verified end-to-end on a real instance: 32/32 kernel-datapath
checks and 29/29 on the relay variant.
QEMU had no host-side interface at all: every adapter was a local UDP tunnel
into uBridge (-netdev socket plus two UDP ports per adapter), so a link could
only ever be a userspace relay. Each adapter now owns a persistent TAP
(gq{node_id[:8]}e{adapter}p{port}), created by uBridge before QEMU is launched
and handed to this user, and the netdev is that TAP -- the anchor role Docker's
veth host end plays. Links attach to it:
* kernel link (NIOBridge): brctl create/up/addif enslaves the TAP into the
per-link bridge and capture, markers and tc filters are installed on it,
verbatim through the shared KernelDatapathMixin -- a TAP and a veth host end
are interchangeable there (an egress qdisc on either impairs traffic entering
the node, and PACKET_OUTGOING on either is the node receiving);
* relay link: bridge add_nio_ethernet plus add_nio_udp -- AF_PACKET, never the
TAP NIO, which would take the fd QEMU itself holds.
The netdev type is fixed at launch, so this is all-or-nothing per VM: on the
TAP datapath every link type attaches to (or detaches from) a running VM,
including switching a link between the two datapaths. Stop un-persists the
TAPs (QEMU's open fd keeps each device alive until it exits) and removes the
per-link bridges; start sweeps leftovers first. A uBridge without the tap
module keeps the previous socket-netdev datapath, so an unupgraded install is
unaffected -- such a node simply cannot carry kernel links.
Move the datapath-agnostic half of DockerKernelDatapathMixin into a new
KernelDatapathMixin (compute/kernel_datapath.py) so a second node type can
reuse it verbatim: the three marker primitives, the tc netem / cls_bpf / eBPF
appliers, the uBridge tc-capability probe, kernel marker teardown, link
detach, the per-link bridge cleanup and the adapter carrier -- plus
_kernel_attach(), lifted from DockerVM._connect_nio's kernel branch.
The mixin talks about "the anchor" (the host-side interface an adapter owns in
the root namespace) instead of "the veth host end": a node type provides
_kernel_host_ifc(), _kernel_anchors() and, when needed, _relay_carrier_bridge()
and _kernel_error(). Docker keeps only the veth lifecycle and those hooks;
behaviour is unchanged.
The C901 baseline entry moves with _netem_command_parts (the docker module is
no longer over the limit) and the bpf-compile-failure test patches the logger
of the module that now owns the code -- net zero new ignores.
Consume uBridge's ebpf_modes contract (feature/tc-window 40c36ae): a
stateful mode is usable iff ebpf=1 and its token is listed; builds
predating the field keep their shipped modes (nth/quota/flow), so the
pre-correction window semantics can never be installed silently.
- gns3server/utils/tc_capabilities.py: shared reply parser, mode
policy and filter-type mappings (compute and controller gate on the
same rules)
- gns3server/compute/ubridge/tc_probe.py: standalone probe spawning a
throwaway uBridge as the server's own user, cached by binary
identity (path/mtime/size - the version string cannot distinguish
rebuilds); failures cache as unknown
- GET /v3/compute/capabilities gains ubridge_tc {netem, ebpf,
ebpf_modes, cbpf}; the controller forwards it (Compute.capabilities)
and available_filters hides kernel-datapath filter types an endpoint
compute cannot run (both ends apply their own filters, so one
incapable compute hides the type); computes without a report keep
the full list and apply-time 409 stays the guard
- docker: _ubridge_apply_ebpf_drops gates per mode token; the cleanup
path only sends "off" for modes the build declares
- split DockerVM's kernel-datapath machinery (veth lifecycle, carrier,
marker primitives, tc filter translation, kernel NIO teardown) into
compute/docker/docker_kernel_datapath.py (DockerKernelDatapathMixin)
The bridge teardown only hung off the link-removal path, but deleting a
node or closing a project never issues a per-link delete — the veths
disappeared with their containers and left the bridge behind as an empty
orphan (reproduced: create a kernel link, DELETE the project, the
gns3{link_id} bridge survives). `_remove_kernel_veths` now deletes the
node's kernel-NIO bridges alongside the veth host ends: both endpoints do
it, the peer's still-enslaved port answers EBUSY (suppressed, last one
wins) exactly like link deletion, a relay NIO's uBridge bridge is left
alone (it dies with the process), and a node restart rebuilds the bridge
from the NIO in _connect_nio.
Found while cleaning up the window_drop e2e: seven such bridges had
accumulated on this host. Re-verified live: the bridge is gone after the
project delete, and the window_drop e2e (which includes a node restart
with the kernel link attached) still passes 15/15.
uBridge's window mode gained the semantics the B.2 spec asked for —
`tc window_drop <if> <start> <outage> <pct> [<period> [<jitter>]]`: a
single outage that traffic passes before *and* after, recurring flaps
with period >= outage, and a per-cycle jitter — so the kernel-only
`window_drop` filter type is now exposed:
- validation mirrors the tc grammar (3-5 params, ranges, period >=
outage, jitter cap) and adds window_drop to KERNEL_ONLY_FILTERS;
filter_inactive_filters keeps start=0 — unlike every other filter a
zero first parameter means "the outage starts now", not "disabled".
- `_ubridge_apply_ebpf_drops` sends `tc window_drop` (positional
pass-through) and turns the mode off explicitly when absent, under the
same ebpf=1 gate.
- FILTERS entry (Start/Outage/Chance/Period/Jitter) + available_filters.
The schedule is relative to the apply: reconcile re-sets the classifier
on every apply, so any filter change, node restart or link reset restarts
the window.
Verified on a two-container kernel link, 15/15: a single outage
[1000, 1500, 100] arrives per-seq as [0..4, 13..19] (passing before the
window, dropped inside, passing after); recurring 1000/2000 ms = 52 %
round trip (duty cycle); period == outage = permanent outage; period <
outage = 409 before uBridge; jittered flap keeps the ~50 % duty cycle;
30 % chance on a permanent window = 53 % round trip against the
per-direction theory 1-0.7^2 = 51 %; coexist with netem; node restart
restores; available_filters kernel vs relay (409 + hidden); teardown
clean. P6b re-run as a regression on the same uBridge build: 13/13.
Worth recording for the next rebuild: the first delivered window program
loaded as root but reported ebpf=0 through gns3-server — the verifier's
non-root path loses the loop counter's bound, treats the catch-up loop as
unbounded and rejects the *whole* program (E2BIG, 8193 pending states),
which also took frequency_drop and quota down, with ebpf=0 as the only
symptom. Smaller trip caps did not help; the loop had to go (uBridge
572ca0c), on top of a Makefile header-dependency fix (53e34bb) that had
been shipping a stale program object.
Consume the eBPF impairment classifier delivered in uBridge
feature/tc-precision: frequency_drop becomes the exact every-Nth mode
(tc nth_drop; -1 = drop everything maps to every 1st) and the new
kernel-only quota type is the byte-cap mode (tc quota_drop, 100 = hard
cutoff). KERNEL_UNSUPPORTED_FILTERS is gone entirely — kernel/relay
eligibility is now purely topological. Both modes gate on ebpf=1 from
the per-process tc capabilities probe (error names the setcap fix); the
cleanup path (modes off) only talks to the eBPF commands when the probe
reported them. Reconcile is a plain re-set: the netem reset-before-set
already detaches the program and clears uBridge's per-interface
registry, so a mode set after it is a fresh load+attach.
Documented semantic difference: the relay counts both directions
through one bridge counter, the kernel counts per direction — N=3
round-trip loss is 1-(2/3)^2 = 55.6% on kernel vs ~1/N on relay
(measured 57%). window_drop is NOT exposed: the delivered
implementation advances the window with no gap, making 'outside the
window packets pass' unreachable (needs outage/period fields —
recorded in the spec); flow_drop awaits a parameter-shape decision.
Also fixes a pre-existing restart bug the e2e exposed: _bridges (relay
bridge names) was never cleared on stop, so restarting a node with a
relay link skipped 'bridge create' and failed on add_nio_udp.
e2e (isolated instance): 13/13 — pref-1 tc_impair classifier on both
veths, 57% round-trip for N=3, -1 = exact 100%, quota hard cutoff,
delay coexistence, clean teardown, suspend/resume, node restart,
relay-leg frequency_drop regression through an Ethernet switch. Full
suite: 1925 passed.
Surface the netem extensions delivered in uBridge feature/tc-netem-ext
as GNS3 filter types: rate (tc-style bandwidth string), reorder (needs
delay), gemodel (Gilbert-Elliot bursty loss, exclusive with
packet_loss), duplicate, seed and limit, plus the delay jitter
distribution and the loss/dup correlation parameters. All are
kernel-datapath only — the relay's packet-filter registry has no
equivalent, so available_filters hides them on relay links, setting one
on a created relay link returns 409, and the project-load path drops
them with a warning (update_filters runs before the datapath is
decided, so it cannot reject there). Validation mirrors the tc grammar:
reorder requires delay, gemodel and packet_loss are mutually exclusive,
a distribution requires jitter > 0, rate is integer+unit capped at
100gbit.
The translation in DockerVM._netem_command_parts emits the frozen
grammar order and gates extension keywords on the 'tc capabilities'
netem token list (probed once per uBridge process; a plain
delay/loss/corrupt/dup filter never probes, keeping old uBridge builds
untouched; the correl suffix has no token and gates on 'rate' as the
extension-build marker).
Reconcile became reset-before-set: the kernel's netem change merges
optional attributes (rate, correlation, reorder, corrupt, gemodel,
distribution, seed keep their previous value when the new message omits
them), so removing a parameter needs an explicit 'tc reset' — the
bpf_drop filters are re-added right after in the same flow, so the
clsact churn is invisible. The capability check runs before the reset
so a failed update leaves the previous qdisc in place.
e2e (isolated instance, kernel link): 24/24 — rate 512kbit adds exactly
2x the per-packet serialization to the RTT (45.5 ms for 1400-byte
pings), paretonormal jitter + rate-cleared leak check, reorder gap 5,
gemodel 50% bursty loss, duplicate, loss 30% correl 50%, seed 42 +
limit 5000 visible in the qdisc dump, suspend/resume and node restart
keep the filters, relay link rejects rate with 409, teardown leaves no
netem. Full unit suite: 1916 passed.
The unified veth host end is born admin-down (carrier off until a link
attaches), and libpcap cannot open a packet socket on an admin-down
interface — its netlink promiscuous-mode transaction returns ENOENT, so
'bridge add_nio_ethernet' failed on every runtime relay link (and would
on any relay link at container start). Bring the interface up for the
attach; the carrier pass in the caller refines the state afterwards (a
suspended NIO sets it back down). Verified live with a direct uBridge
probe: attach on a down veth -> ENOENT, after 'link set up' -> OK.
The GNS3 bpf filter (one expression per line, any line matching drops the
packet) now runs on the kernel datapath: uBridge's tc bpf_drop compiles
each line with libpcap (same compiler as the relay filter) into classic
BPF, attaches it to the veth host end's clsact egress as a cls_bpf
classifier with a gact TC_ACT_SHOT action, and the kernel migrates the
cBPF to eBPF internally -- no CAP_BPF needed.
- reconcile is flush + one add per line at priorities 10, 11, ... (OR
semantics, same as the relay's per-line filters); tc reset (link
delete / clear) is the full restore in uBridge and clears them
- capability-gated: the compute probes "tc capabilities" once per uBridge
process (cached, invalidated on stop); without cbpf=1 the filter apply
fails with an actionable upgrade error, and old uBridge builds without
the tc module are never touched when no bpf filter is present
- a line that compiled for the controller's tcpdump but not for the
compute's uBridge libpcap is skipped with a warning, mirroring the
relay path
- KERNEL_UNSUPPORTED_FILTERS shrinks to frequency_drop (its eBPF stateful
classifier is still pending, spec part B)
e2e (22/22): bpf "icmp" = 100% loss; size-discriminating expression
("greater 150") passes 98-byte and drops 300-byte pings (real byte-level
match); two-line OR; coexistence with netem (delay 50 -> small pings
100.2 ms while big dropped); clear removes clsact; suspend/resume keeps
the drops; link delete + recreate leaves no residual filters.
Impairment filters with a netem equivalent (delay, packet_loss, corrupt)
now run on the kernel datapath as one tc netem qdisc per veth host end,
pushed through uBridge's netlink tc module (tc netem set / tc reset).
This removes the relay delay filter's nanosleep bottleneck (upstream
Both link endpoints carry the filters: each end's qdisc egress covers
the traffic entering that container, so every direction is impaired
exactly once -- the same net effect as the relay, where both directions
cross the single filtered bridge.
- controller: eligibility keys on relay-only types (frequency_drop, bpf)
instead of any filter; kernel links push the real filters (the
synthetic suspend frequency_drop no longer flips a suspended link to
relay on reopen/reset); update_filters 409s only relay-only types;
available_filters hides them on kernel links
- compute: create_nio whitelists netem-capable filter types; the NIO
update path keeps its own guard (PUT mutates the NIO in place);
_ubridge_apply_netem translates the filter dict into one atomic
netem-set (full re-apply IS the reconcile); node restart restores the
qdisc, link deletion detaches it (the veth survives the link)
- e2e (27/27): delay 100 -> 200.2 ms RTT, atomic replace 50 -> 100.2 ms,
clear -> baseline, packet_loss 30 -> 48 % round-trip (expected 51 %),
suspend/resume and node restart keep the qdisc, link delete leaves no
residual impairment
Every Docker adapter is now born as a veth pair at container start
(unified interface) regardless of NIO type: guest end in the container
namespace, host end in the root namespace. Kernel links enslave the host
end into the per-link Linux bridge; relay links attach it to the uBridge
relay via AF_PACKET (bridge add_nio_ethernet) — the TAP path is gone.
This makes the datapath a runtime decision:
- links can be created on running containers (kernel or relay) without
touching container interfaces — the controller stopped-node
eligibility check is removed
- project reopen re-prepares relay links and upgrades eligible ones to
the kernel datapath
- packet capture start/stop now key on the NIO type (not veth presence)
so relay-over-veth captures use the relay capture, kernel ones the
AF_PACKET module
- vendor unix-socket containers keep their wiring: no veth is required
on the relay path
Markers no longer disqualify the kernel datapath. They ride the NIO like
on the relay path and attach to the veth host end through uBridge's
marker add_kernel/delete_kernel/enable_kernel commands (AF_PACKET taps,
same BPF/direction/signal semantics as the relay mark filter).
- base_node: marker anchors are now polymorphic (relay bridge name or
kernel veth host interface); the reconcile loop is datapath-agnostic
and DockerVM overrides the add/delete/enable primitives to translate
the commands. Marker-name validation extracted for both add paths.
- docker_vm: kernel NIO connect/update apply markers on the veth anchor;
link deletion tears markers down explicitly (the veth survives the
link, unlike a relay bridge whose death drops its filters).
- controller: eligibility and start_marker no longer exclude markers or
project-level marker definitions; the kernel NIO carries the capture
node's marker specs.
- NIO creation accepts markers on kernel NIOs (filters still rejected).
Replace the capture rejection with wiring to uBridge's capture module:
an AF_PACKET socket bound to the veth host end writes into the same pcap
file the relay path uses (capture start_kernel <if> "<pcap>" /
capture stop_kernel), so the pcap streaming to clients is unchanged.
- capture survives node restarts: _connect_nio re-arms it when the NIO
is already capturing, mirroring the relay path
- uBridge's kernel capture is a singleton per process: a concurrent
second capture on the same node surfaces uBridge's EALREADY error
- update the datapath docs: filters and markers remain relay-only,
capture is now served on both datapaths
Eligible links (both endpoints Docker, same compute, no filters/markers,
stopped nodes, no unix-socket NIO containers) are wired as veth pairs
enslaved into a per-link Linux kernel bridge (gns3 + 11 hex of the link
id, filling IFNAMSIZ) instead of the uBridge UDP relay, so frames flow
entirely in the kernel.
- NIOBridge NIO type: the controller decides the datapath in
UDPLink._prepare and both computes enslave their deterministic gv*/gc*
veth pair via uBridge's docker/brctl modules
- carrier/suspend = host-end admin state; both endpoints race-safely
create/delete the bridge (EEXIST/EBUSY/ENOENT tolerated)
- veth adapters can still attach relay (UDP) links on a running node:
lazy relay bridge over AF_PACKET on the host end (add_nio_ethernet)
- filters/markers/capture are rejected with explicit errors on this
datapath (they live in the uBridge relay); vendor unix-socket
containers are excluded; Server.enable_kernel_datapath is the
kill-switch (default on)
- host-end veths survive container death and are removed on stop plus
swept by name on the next start (crash/stale recovery)
