Kernel eligibility becomes "both endpoints on one compute, each able to
anchor": Docker always (adapters are born as veth pairs), QEMU when that
compute's uBridge can create the persistent TAPs its adapters anchor on. The
question is asked per compute, because the link needs both ends anchored --
so a mixed or half-upgraded deployment degrades to the relay instead of
failing an attach.
probe_tap_support() creates and deletes one throwaway persistent TAP through a
scratch uBridge, cached by binary identity like the tc report; /capabilities
exposes `ubridge_tap` (absent = unknown, which keeps links on the relay).
The compute route for a QEMU adapter NIO took UDPNIO only, so a kernel link
was rejected with 422: its body and response now accept
Union[UDPNIO, BridgeNIO] exactly like Docker's.
Stale "Docker-to-Docker" wording in the config, link and MCP descriptions now
talks about anchors, and docs/features/qemu-kernel-datapath.md documents the
datapath -- verified end-to-end on a real instance: 32/32 kernel-datapath
checks and 29/29 on the relay variant.
QEMU had no host-side interface at all: every adapter was a local UDP tunnel
into uBridge (-netdev socket plus two UDP ports per adapter), so a link could
only ever be a userspace relay. Each adapter now owns a persistent TAP
(gq{node_id[:8]}e{adapter}p{port}), created by uBridge before QEMU is launched
and handed to this user, and the netdev is that TAP -- the anchor role Docker's
veth host end plays. Links attach to it:
* kernel link (NIOBridge): brctl create/up/addif enslaves the TAP into the
per-link bridge and capture, markers and tc filters are installed on it,
verbatim through the shared KernelDatapathMixin -- a TAP and a veth host end
are interchangeable there (an egress qdisc on either impairs traffic entering
the node, and PACKET_OUTGOING on either is the node receiving);
* relay link: bridge add_nio_ethernet plus add_nio_udp -- AF_PACKET, never the
TAP NIO, which would take the fd QEMU itself holds.
The netdev type is fixed at launch, so this is all-or-nothing per VM: on the
TAP datapath every link type attaches to (or detaches from) a running VM,
including switching a link between the two datapaths. Stop un-persists the
TAPs (QEMU's open fd keeps each device alive until it exits) and removes the
per-link bridges; start sweeps leftovers first. A uBridge without the tap
module keeps the previous socket-netdev datapath, so an unupgraded install is
unaffected -- such a node simply cannot carry kernel links.
Move the datapath-agnostic half of DockerKernelDatapathMixin into a new
KernelDatapathMixin (compute/kernel_datapath.py) so a second node type can
reuse it verbatim: the three marker primitives, the tc netem / cls_bpf / eBPF
appliers, the uBridge tc-capability probe, kernel marker teardown, link
detach, the per-link bridge cleanup and the adapter carrier -- plus
_kernel_attach(), lifted from DockerVM._connect_nio's kernel branch.
The mixin talks about "the anchor" (the host-side interface an adapter owns in
the root namespace) instead of "the veth host end": a node type provides
_kernel_host_ifc(), _kernel_anchors() and, when needed, _relay_carrier_bridge()
and _kernel_error(). Docker keeps only the veth lifecycle and those hooks;
behaviour is unchanged.
The C901 baseline entry moves with _netem_command_parts (the docker module is
no longer over the limit) and the bpf-compile-failure test patches the logger
of the module that now owns the code -- net zero new ignores.
Consume uBridge's ebpf_modes contract (feature/tc-window 40c36ae): a
stateful mode is usable iff ebpf=1 and its token is listed; builds
predating the field keep their shipped modes (nth/quota/flow), so the
pre-correction window semantics can never be installed silently.
- gns3server/utils/tc_capabilities.py: shared reply parser, mode
policy and filter-type mappings (compute and controller gate on the
same rules)
- gns3server/compute/ubridge/tc_probe.py: standalone probe spawning a
throwaway uBridge as the server's own user, cached by binary
identity (path/mtime/size - the version string cannot distinguish
rebuilds); failures cache as unknown
- GET /v3/compute/capabilities gains ubridge_tc {netem, ebpf,
ebpf_modes, cbpf}; the controller forwards it (Compute.capabilities)
and available_filters hides kernel-datapath filter types an endpoint
compute cannot run (both ends apply their own filters, so one
incapable compute hides the type); computes without a report keep
the full list and apply-time 409 stays the guard
- docker: _ubridge_apply_ebpf_drops gates per mode token; the cleanup
path only sends "off" for modes the build declares
- split DockerVM's kernel-datapath machinery (veth lifecycle, carrier,
marker primitives, tc filter translation, kernel NIO teardown) into
compute/docker/docker_kernel_datapath.py (DockerKernelDatapathMixin)
The bridge teardown only hung off the link-removal path, but deleting a
node or closing a project never issues a per-link delete — the veths
disappeared with their containers and left the bridge behind as an empty
orphan (reproduced: create a kernel link, DELETE the project, the
gns3{link_id} bridge survives). `_remove_kernel_veths` now deletes the
node's kernel-NIO bridges alongside the veth host ends: both endpoints do
it, the peer's still-enslaved port answers EBUSY (suppressed, last one
wins) exactly like link deletion, a relay NIO's uBridge bridge is left
alone (it dies with the process), and a node restart rebuilds the bridge
from the NIO in _connect_nio.
Found while cleaning up the window_drop e2e: seven such bridges had
accumulated on this host. Re-verified live: the bridge is gone after the
project delete, and the window_drop e2e (which includes a node restart
with the kernel link attached) still passes 15/15.
uBridge's window mode gained the semantics the B.2 spec asked for —
`tc window_drop <if> <start> <outage> <pct> [<period> [<jitter>]]`: a
single outage that traffic passes before *and* after, recurring flaps
with period >= outage, and a per-cycle jitter — so the kernel-only
`window_drop` filter type is now exposed:
- validation mirrors the tc grammar (3-5 params, ranges, period >=
outage, jitter cap) and adds window_drop to KERNEL_ONLY_FILTERS;
filter_inactive_filters keeps start=0 — unlike every other filter a
zero first parameter means "the outage starts now", not "disabled".
- `_ubridge_apply_ebpf_drops` sends `tc window_drop` (positional
pass-through) and turns the mode off explicitly when absent, under the
same ebpf=1 gate.
- FILTERS entry (Start/Outage/Chance/Period/Jitter) + available_filters.
The schedule is relative to the apply: reconcile re-sets the classifier
on every apply, so any filter change, node restart or link reset restarts
the window.
Verified on a two-container kernel link, 15/15: a single outage
[1000, 1500, 100] arrives per-seq as [0..4, 13..19] (passing before the
window, dropped inside, passing after); recurring 1000/2000 ms = 52 %
round trip (duty cycle); period == outage = permanent outage; period <
outage = 409 before uBridge; jittered flap keeps the ~50 % duty cycle;
30 % chance on a permanent window = 53 % round trip against the
per-direction theory 1-0.7^2 = 51 %; coexist with netem; node restart
restores; available_filters kernel vs relay (409 + hidden); teardown
clean. P6b re-run as a regression on the same uBridge build: 13/13.
Worth recording for the next rebuild: the first delivered window program
loaded as root but reported ebpf=0 through gns3-server — the verifier's
non-root path loses the loop counter's bound, treats the catch-up loop as
unbounded and rejects the *whole* program (E2BIG, 8193 pending states),
which also took frequency_drop and quota down, with ebpf=0 as the only
symptom. Smaller trip caps did not help; the loop had to go (uBridge
572ca0c), on top of a Makefile header-dependency fix (53e34bb) that had
been shipping a stale program object.
Consume the eBPF impairment classifier delivered in uBridge
feature/tc-precision: frequency_drop becomes the exact every-Nth mode
(tc nth_drop; -1 = drop everything maps to every 1st) and the new
kernel-only quota type is the byte-cap mode (tc quota_drop, 100 = hard
cutoff). KERNEL_UNSUPPORTED_FILTERS is gone entirely — kernel/relay
eligibility is now purely topological. Both modes gate on ebpf=1 from
the per-process tc capabilities probe (error names the setcap fix); the
cleanup path (modes off) only talks to the eBPF commands when the probe
reported them. Reconcile is a plain re-set: the netem reset-before-set
already detaches the program and clears uBridge's per-interface
registry, so a mode set after it is a fresh load+attach.
Documented semantic difference: the relay counts both directions
through one bridge counter, the kernel counts per direction — N=3
round-trip loss is 1-(2/3)^2 = 55.6% on kernel vs ~1/N on relay
(measured 57%). window_drop is NOT exposed: the delivered
implementation advances the window with no gap, making 'outside the
window packets pass' unreachable (needs outage/period fields —
recorded in the spec); flow_drop awaits a parameter-shape decision.
Also fixes a pre-existing restart bug the e2e exposed: _bridges (relay
bridge names) was never cleared on stop, so restarting a node with a
relay link skipped 'bridge create' and failed on add_nio_udp.
e2e (isolated instance): 13/13 — pref-1 tc_impair classifier on both
veths, 57% round-trip for N=3, -1 = exact 100%, quota hard cutoff,
delay coexistence, clean teardown, suspend/resume, node restart,
relay-leg frequency_drop regression through an Ethernet switch. Full
suite: 1925 passed.
Surface the netem extensions delivered in uBridge feature/tc-netem-ext
as GNS3 filter types: rate (tc-style bandwidth string), reorder (needs
delay), gemodel (Gilbert-Elliot bursty loss, exclusive with
packet_loss), duplicate, seed and limit, plus the delay jitter
distribution and the loss/dup correlation parameters. All are
kernel-datapath only — the relay's packet-filter registry has no
equivalent, so available_filters hides them on relay links, setting one
on a created relay link returns 409, and the project-load path drops
them with a warning (update_filters runs before the datapath is
decided, so it cannot reject there). Validation mirrors the tc grammar:
reorder requires delay, gemodel and packet_loss are mutually exclusive,
a distribution requires jitter > 0, rate is integer+unit capped at
100gbit.
The translation in DockerVM._netem_command_parts emits the frozen
grammar order and gates extension keywords on the 'tc capabilities'
netem token list (probed once per uBridge process; a plain
delay/loss/corrupt/dup filter never probes, keeping old uBridge builds
untouched; the correl suffix has no token and gates on 'rate' as the
extension-build marker).
Reconcile became reset-before-set: the kernel's netem change merges
optional attributes (rate, correlation, reorder, corrupt, gemodel,
distribution, seed keep their previous value when the new message omits
them), so removing a parameter needs an explicit 'tc reset' — the
bpf_drop filters are re-added right after in the same flow, so the
clsact churn is invisible. The capability check runs before the reset
so a failed update leaves the previous qdisc in place.
e2e (isolated instance, kernel link): 24/24 — rate 512kbit adds exactly
2x the per-packet serialization to the RTT (45.5 ms for 1400-byte
pings), paretonormal jitter + rate-cleared leak check, reorder gap 5,
gemodel 50% bursty loss, duplicate, loss 30% correl 50%, seed 42 +
limit 5000 visible in the qdisc dump, suspend/resume and node restart
keep the filters, relay link rejects rate with 409, teardown leaves no
netem. Full unit suite: 1916 passed.
The unified veth host end is born admin-down (carrier off until a link
attaches), and libpcap cannot open a packet socket on an admin-down
interface — its netlink promiscuous-mode transaction returns ENOENT, so
'bridge add_nio_ethernet' failed on every runtime relay link (and would
on any relay link at container start). Bring the interface up for the
attach; the carrier pass in the caller refines the state afterwards (a
suspended NIO sets it back down). Verified live with a direct uBridge
probe: attach on a down veth -> ENOENT, after 'link set up' -> OK.
The GNS3 bpf filter (one expression per line, any line matching drops the
packet) now runs on the kernel datapath: uBridge's tc bpf_drop compiles
each line with libpcap (same compiler as the relay filter) into classic
BPF, attaches it to the veth host end's clsact egress as a cls_bpf
classifier with a gact TC_ACT_SHOT action, and the kernel migrates the
cBPF to eBPF internally -- no CAP_BPF needed.
- reconcile is flush + one add per line at priorities 10, 11, ... (OR
semantics, same as the relay's per-line filters); tc reset (link
delete / clear) is the full restore in uBridge and clears them
- capability-gated: the compute probes "tc capabilities" once per uBridge
process (cached, invalidated on stop); without cbpf=1 the filter apply
fails with an actionable upgrade error, and old uBridge builds without
the tc module are never touched when no bpf filter is present
- a line that compiled for the controller's tcpdump but not for the
compute's uBridge libpcap is skipped with a warning, mirroring the
relay path
- KERNEL_UNSUPPORTED_FILTERS shrinks to frequency_drop (its eBPF stateful
classifier is still pending, spec part B)
e2e (22/22): bpf "icmp" = 100% loss; size-discriminating expression
("greater 150") passes 98-byte and drops 300-byte pings (real byte-level
match); two-line OR; coexistence with netem (delay 50 -> small pings
100.2 ms while big dropped); clear removes clsact; suspend/resume keeps
the drops; link delete + recreate leaves no residual filters.
Impairment filters with a netem equivalent (delay, packet_loss, corrupt)
now run on the kernel datapath as one tc netem qdisc per veth host end,
pushed through uBridge's netlink tc module (tc netem set / tc reset).
This removes the relay delay filter's nanosleep bottleneck (upstream
Both link endpoints carry the filters: each end's qdisc egress covers
the traffic entering that container, so every direction is impaired
exactly once -- the same net effect as the relay, where both directions
cross the single filtered bridge.
- controller: eligibility keys on relay-only types (frequency_drop, bpf)
instead of any filter; kernel links push the real filters (the
synthetic suspend frequency_drop no longer flips a suspended link to
relay on reopen/reset); update_filters 409s only relay-only types;
available_filters hides them on kernel links
- compute: create_nio whitelists netem-capable filter types; the NIO
update path keeps its own guard (PUT mutates the NIO in place);
_ubridge_apply_netem translates the filter dict into one atomic
netem-set (full re-apply IS the reconcile); node restart restores the
qdisc, link deletion detaches it (the veth survives the link)
- e2e (27/27): delay 100 -> 200.2 ms RTT, atomic replace 50 -> 100.2 ms,
clear -> baseline, packet_loss 30 -> 48 % round-trip (expected 51 %),
suspend/resume and node restart keep the qdisc, link delete leaves no
residual impairment
Every Docker adapter is now born as a veth pair at container start
(unified interface) regardless of NIO type: guest end in the container
namespace, host end in the root namespace. Kernel links enslave the host
end into the per-link Linux bridge; relay links attach it to the uBridge
relay via AF_PACKET (bridge add_nio_ethernet) — the TAP path is gone.
This makes the datapath a runtime decision:
- links can be created on running containers (kernel or relay) without
touching container interfaces — the controller stopped-node
eligibility check is removed
- project reopen re-prepares relay links and upgrades eligible ones to
the kernel datapath
- packet capture start/stop now key on the NIO type (not veth presence)
so relay-over-veth captures use the relay capture, kernel ones the
AF_PACKET module
- vendor unix-socket containers keep their wiring: no veth is required
on the relay path
Markers no longer disqualify the kernel datapath. They ride the NIO like
on the relay path and attach to the veth host end through uBridge's
marker add_kernel/delete_kernel/enable_kernel commands (AF_PACKET taps,
same BPF/direction/signal semantics as the relay mark filter).
- base_node: marker anchors are now polymorphic (relay bridge name or
kernel veth host interface); the reconcile loop is datapath-agnostic
and DockerVM overrides the add/delete/enable primitives to translate
the commands. Marker-name validation extracted for both add paths.
- docker_vm: kernel NIO connect/update apply markers on the veth anchor;
link deletion tears markers down explicitly (the veth survives the
link, unlike a relay bridge whose death drops its filters).
- controller: eligibility and start_marker no longer exclude markers or
project-level marker definitions; the kernel NIO carries the capture
node's marker specs.
- NIO creation accepts markers on kernel NIOs (filters still rejected).
Replace the capture rejection with wiring to uBridge's capture module:
an AF_PACKET socket bound to the veth host end writes into the same pcap
file the relay path uses (capture start_kernel <if> "<pcap>" /
capture stop_kernel), so the pcap streaming to clients is unchanged.
- capture survives node restarts: _connect_nio re-arms it when the NIO
is already capturing, mirroring the relay path
- uBridge's kernel capture is a singleton per process: a concurrent
second capture on the same node surfaces uBridge's EALREADY error
- update the datapath docs: filters and markers remain relay-only,
capture is now served on both datapaths
Eligible links (both endpoints Docker, same compute, no filters/markers,
stopped nodes, no unix-socket NIO containers) are wired as veth pairs
enslaved into a per-link Linux kernel bridge (gns3 + 11 hex of the link
id, filling IFNAMSIZ) instead of the uBridge UDP relay, so frames flow
entirely in the kernel.
- NIOBridge NIO type: the controller decides the datapath in
UDPLink._prepare and both computes enslave their deterministic gv*/gc*
veth pair via uBridge's docker/brctl modules
- carrier/suspend = host-end admin state; both endpoints race-safely
create/delete the bridge (EEXIST/EBUSY/ENOENT tolerated)
- veth adapters can still attach relay (UDP) links on a running node:
lazy relay bridge over AF_PACKET on the host end (add_nio_ethernet)
- filters/markers/capture are rejected with explicit errors on this
datapath (they live in the uBridge relay); vendor unix-socket
containers are excluded; Server.enable_kernel_datapath is the
kill-switch (default on)
- host-end veths survive container death and are removed on stop plus
swept by name on the next start (crash/stale recovery)
