Review feedback: 3.1 is still in alpha and the GNS3 VM ships a matching
uBridge, so a hard minimum is simpler than a version-based degrade path,
and it also covers Docker endpoints and the builtin Ethernet switch,
which never went through the anchor probes. Raise minimum_required_version
to 1.3.1 and drop supports_kernel_datapath and its uses; the relay
fallback for capability problems (no CAP_NET_ADMIN) is unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The TAP netdev kept QEMU's default vnet_hdr, so virtio-net, e1000e, igb
and vmxnet3 guests offload TCP/UDP checksums (and TSO) to the TAP: their
frames carry only the pseudo-header sum in the checksum field. A kernel
bridge link finishes them, but the uBridge relay (cloud, NAT, hub, ...)
reads the TAP with libpcap and forwards the frames as they are, so DNS
and TCP through a cloud fail while ICMP and DHCP work.
With vnet_hdr=off QEMU stops offering the offloads and the guest (or the
emulated NIC) sends finished frames, as with the 2.x socket netdev.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A server killed before close() leaves the switch's kernel bridge behind
with its ports still enslaved: a peer's persistent anchor TAP and cascade
veth ends outlive uBridge. uBridge's brctl delete refuses a bridge that
still has ports (EBUSY), so the best-effort recovery delete in
_ensure_bridge() failed silently and brctl create hit EEXIST: the switch,
and the project holding it, could no longer be opened (HTTP 409 "Could
not create bridge gns3xxxxxx: File exists").
List the leftover bridge's ports from /sys/class/net/<bridge>/brif and
release each with brctl delif before the delete. uBridge's brctl show
reports no ports, and the switch already reads sysfs on the same host.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uBridge 1.2.3 already has the tap module, so the persistent-TAP probes
(_probe_persistent_taps on the node, probe_tap_support and siblings for
/capabilities) pass on it. QEMU, Dynamips and VPCS nodes then anchor on
TAPs and the controller builds kernel links that need 1.3.x commands:
kernel markers and the eBPF tc modes (1.3.0) and per-interface kernel
captures (`capture stop_kernel <ifc>`, 1.3.1). On 1.2.3 a MikroTik CHR
node sent frames but never got DHCP past DISCOVER/OFFER, and stopping
a capture failed with "Bad number of parameters".
Add Hypervisor.supports_kernel_datapath (version >= 1.3.1) and refuse
older builds in the node-side probe and the four anchoring capability
probes (tap, iol tap, bridge tap, vxlan). The existing fallback then
applies: nodes keep the legacy relay datapath with the usual warning,
which now names the installed version, and links stay on the relay.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When only a port is renamed in a template's custom adapters, the adapter is
saved with "adapter_type": null. The emulators read it with
custom_adapter.get("adapter_type", self._adapter_type), which returns None
for a key set to null: QEMU was then started with "-device None,..." and
died ("'None' is not a valid device model name"), GNS3 only reporting
"Could not start Telnet QEMU console". VirtualBox and VMware read the
adapter type the same way.
Drop the null settings in _get_custom_adapter_settings so the node's own
adapter type (and MAC address) applies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the compute is itself a Hyper-V guest without an invariant TSC (e.g.
the GNS3 VM on hosts whose CPU lacks TSC scaling), Linux falls back to the
hyperv_clocksource_tsc_page clocksource and the guest timekeeping KVM
derives from the TSC is broken with recent kernels:
- kvmclock runs thousands of times too fast: Linux guests (MikroTik
RouterOS, ...) panic with "watchdog: BUG: soft lockup - CPU#0 stuck for
29604s" and reboot in a loop right after boot;
- TSC deadline timers fire far too late: guests with -cpu host hang.
The only workaround so far was -machine accel=tcg, i.e. no KVM at all.
Add a kvmclock setting to the [Qemu] section (auto, enabled, disabled).
In auto mode, KVM guests started on a host whose clocksource is a Hyper-V
one get kvmclock and tsc-deadline turned off: on the -cpu given in the
additional options (unless it sets these features itself), otherwise on
QEMU's default CPU model.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Add per-drive IDE/SATA selection with backward-compatible IDE defaults
- Persist interfaces in nodes, templates, and appliance imports
- Add database migration support for versioned and unversioned databases
- Prevent partial updates when rejecting live interface changes
- Validate interfaces and support SATA media changes through the monitor
Cross-compute links can ride kernel VXLAN tunnels instead of the uBridge
UDP relay: each endpoint creates one VXLAN device (uBridge's rtnetlink
vxlan module) enslaved into the same per-link Linux bridge the same-host
datapath uses, so filters, markers, capture and suspend keep running on
the anchor unchanged. Opt-in (Server.enable_vxlan_cross_compute, default
off) because the tunnel caps the usable MTU at the underlay minus the
outer headers; the relay stays the default and the fallback.
The controller derives what the pair must agree on: one VNI per link
(upper VNI space, per-link random salt) and deterministic device names
(gvl{link}{node}) that make crashed-run leftovers claimable
(show -> delete -> create). Same-host compute pairs, switch endpoints
and serial/unix-socket containers stay on the relay.
A VNI the kernel refuses self-heals on every path it can surface:
interactive create retries a fresh salt then degrades to the relay;
project open re-dispatches refused batch entries the same way; node
start recognises the refused device, stops the half-started node,
re-wires both ends on a fresh VNI (or degrades the link) and retries
the start. The device MTU is derived from the lower device (dev=) and
the per-link bridge inherits it - no configuration. The capability
probe (ubridge_vxlan in /capabilities) runs one scratch create/delete
on a scratch port that never equals the configured dstport.
The e2e scenario runs two computes on one machine (a network namespace
joined by a veth pair as the fabric) and needs root.
Updating any Docker node property destroys and recreates the container
(the route calls DockerVM.update() as soon as the property diff is
non-empty), so it discards everything written inside it. On a running
node that happened silently, as a side effect of an unrelated edit: the
user only saw the node restart.
Refuse the update while the container is up and say what stopping it is
for, like QEMU, IOU, Dynamips, VPCS, VirtualBox and VMware already do
for their own restart-requiring settings. The check runs before anything
is applied — a refused update leaves the node untouched — and only when
the PUT really changes a property, since Web clients send the whole node
back on every edit.
is_running() is not reusable for the question: it reports a paused
container as not running and stops crashed containers as a side effect,
so DockerVM gains is_up() for "running or paused".
When a project is opened, Docker nodes are created in parallel. The free
display was found by checking whether /tmp/.X11-unix/X<n> exists, but that
socket only appears once Xvnc has started, so several nodes got the same
display. Only the first Xvnc started; the others exited with "Server is
already active for display N" ("The vnc process has stopped with return
code 1"), and their containers drew on the screen of another node.
- reserve the displays given to the nodes of this server, and also skip a
display whose lock file belongs to a live process
- restart a crashed Xvnc on the display the container was created with,
after removing its stale socket, instead of picking a new display the
container cannot reach (black screen)
- on close, only remove the X socket of our own Xvnc, never the socket of
the node that owns the display
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
uBridge's capture module now serves one AF_PACKET capture per interface,
any number per process, and `capture stop_kernel` takes the interface it
stops — this rides that (breaking, paired-binary) protocol change.
The server-side single-slot tracker (_kernel_capture_ifc with its
reserve/release/owned_by helpers) is gone: same-port duplicates were
already refused by the NIO capturing flag, uBridge answers EALREADY on a
same-interface duplicate, and per-interface stops removed the process-
wide stop the ownership gating guarded against.
The docker e2e drives ten concurrent captures on one node's uBridge and
asserts a per-interface stop freezes exactly the stopped half while the
live half keeps recording.
The cloud node offered the kernel datapath's host-side interfaces —
node anchors (gv/gq/gi/gd/gx/gp), switch cascade veth ends (gs) and
cross-compute VXLAN devices (gvl) — in its interface list and even
pre-filled them as ports on a freshly created cloud: none of them is
flagged special and only "gns3*" names were filtered. These interfaces
belong to the link or node that created them for the project's whole
lifetime, so bridging a cloud to one would race its owner.
Add is_kernel_datapath_interface() next to the naming contract in
utils/kernel_anchor.py: exact-shape matching (prefix plus its
fixed-length id-derived tail), so a user interface that merely starts
with "gv" is never swallowed. The cloud keeps managed names out of its
host-interface list and default ports mapping, while saved mappings
keep working (the existence checks still see the full interface list).
Docker ids are not comparable across image stores: for the same content a
classic store reports the config digest while a containerd store (or an
image loaded from an OCI archive) reports the manifest digest. The
controller pins the id of its own daemon, so a compute whose store reports
another id for that content answered ImageMissingError forever: every node
creation streamed the image again (save/load round trip) without ever
converging, then failed with "The image 'x' is missing" after the six
retries. Docker 29 makes this easy to hit, since the id a daemon reports
now depends on how the image entered it.
The load endpoint now takes the tag and the digest pinned by the
controller, and the compute records how its own daemon identifies the
image it just received. Node creation accepts the pin through that pair,
and only while the tag still resolves to the synced content: a tag moved
to some other image, or an image the controller never pushed here, still
counts as missing. The pair lives in the compute process, so a restarted
compute syncs once more.
Computes that predate the new endpoint parameters keep the previous
behaviour, as do controllers that do not send them.
A VPCS node's single adapter anchors on a persistent TAP the server
creates (tap create + set_owner via uBridge) and the vpcs process
attaches to by name: -e -d <gp{id8}e0p0> -n, the same
server-creates/guest-attaches shape as QEMU's netdev. -n (a vpcs fork
feature) keeps the guest's ip command from ifconfig'ing the anchor on
the host; the vpcs build is probed once per binary identity
(vpcs_tap capability, usage scan) and gates kernel links together
with ubridge_tap — anything else keeps the node on the relay (the
legacy local UDP tunnel on a stock binary, an AF_PACKET endpoint on
the anchor with a capable one).
VPCSVM takes KernelDatapathMixin: _connect_nio dispatches kernel
(enslave into the per-link bridge) / relay-on-anchor / legacy,
captures pick capture start_kernel vs the bridge engine, stop runs
under an execute lock and terminates the process before deleting the
anchor TAPs (a held device refuses deletion), and the duplicate-port
guard backstops the controller's. The NIO routes accept nio_bridge
and write suspend back on update.
_delete_anchor_taps verifies each tap is gone from sysfs and retries
a bounded three times: a holder can release its fd a moment after its
process was reaped (vpcs's daemon parent kills its console child and
exits first), so the unconditional delete raced EBUSY and leaked the
anchor intermittently.
The vpcs side needs IFNAMSIZ (16) instead of its own IFNAMESIZ (12)
for the -d name buffer — anchor names are 14 chars and truncated to a
nonexistent device, which TUNSETIFF then tries to create without
CAP_NET_ADMIN (fixed in the vpcs fork).
e2e (tests/e2e/test_vpcs_kernel_datapath.py): kernel and relay
scenarios green against the real binary — real ICMP both ways, §E.1/§
E.2 on the anchors, delay/suspend/capture, link delete/re-create,
node stop/restart self-heal, zero leftovers; the relay control passes
with both the fork binary and a stock one.
Resolve conflicts against the upstream wave (PRs #2952-#2965, incl. the
typed-response/schema hardening and the drop of Python 3.10):
- controller/node.py: keep upstream's _transitioning/_confirm_status
lifecycle guard; the kernel-anchor link wiring (node_started) runs
after the transition completes.
- controller/link.py: keep upstream's add_node rollback (release_ports
on create failure) and delete via release_ports; release_ports now
resolves ports live (same staleness reason as _bind_members) and
discards the node link unconditionally.
- schemas/controller/links.py: extend LinkFilterType with the eight
kernel-datapath filter types and LinkFilterParameter.type with "str"
so the newly typed /available_filters response validates; Link.filters
stays a free-form dict — the vocabulary is capability-gated per link
and validated by the controller (409), not by a strict request schema.
- schemas/controller/topology.py: drop the type: ignore made redundant
by the dict filters field.
- tests: route-level filter validation now expects the controller's 409
(unknown types, out-of-range values); packet_loss correlation is a
valid kernel parameter (positive test added); OpenAPI assertions match
the free-form filters property.
The persistent-TAP lifecycle — probe, sweep, create, register, delete —
was implemented four times, once per node type, and the copies had drifted
(probe mechanism, _tap_datapath reset, keep-vs-recreate policy). The
mechanical half moves into KernelDatapathMixin:
- _probe_persistent_taps(probe_name): the scratch-TAP probe, returning the
refusal for the caller's own warning (QEMU, Dynamips);
- _create_anchor_taps(ports, set_owner, keep_existing): sweep + create +
optional ownership handover + carrier down + registry, with the ports
enumerated by the node type and named through the new _tap_name hook;
- _delete_anchor_taps(): the suppressed tap sweep + per-link bridge
cleanup + registry clear.
Each node type keeps what its fd holder dictates: the release step before
the tap sweep (the QEMU process, the IOL bridge, the hypervisor's NIO
bindings, the container port bridges), its probe policy, and its registry
lifetime. QEMU's now-unconditional registry clear aligns it with the other
three; everything else is behavior-preserving.
Compute:
- Track the anchor that owns uBridge's single AF_PACKET capture slot: a
second kernel port's capture is refused with a clear error (its flag
rolled back), a failed start no longer wedges the port, and only the
owning port's stop issues the process-wide `capture stop_kernel`
(docker/qemu/iou/dynamips/switch, via the mixin's slot helpers).
- Unwind failed starts instead of leaking uBridge and the anchor TAPs:
QEMU's stop body runs under try/finally, QEMU and IOU start the datapath
inside the launch try, Dynamips undoes a half-attached port (the retry
used to be short-circuited into a silent black hole), and Docker deletes
the veth pair when the namespace move fails.
- Restore the relay filter guard on IOU (kernel-only filters raise
NodeError, an uncompilable BPF line degrades to a warning), validate the
eBPF/quota/window/bpf filter shapes before touching the bridge (short or
non-numeric values surfaced as 500s after partial state), and refuse a
second NIO on an occupied Docker/QEMU port (the guard IOU and Dynamips
already had).
- Probe fixes: `_probe_bridge_tap` survives a failing uBridge start, and
the probe caches key on ctime too, so a chmod/setcap remedy re-probes.
Controller:
- Parse GNS3_UNIX_SOCKET_NIO like the compute does (line-anchored, truthy
value) and mirror the compute's class selection in kernel_anchor_type.
- Read partial capability reports defensively; make BridgeNIO.bridge
required-but-nullable so an omitted key is a 422 rather than a KeyError.
- Strip the packet_loss correlation slot even when it is zero: the relay
filter takes exactly one argument and rejected `packet_loss 10 0`.
Ethernet switch: skip ports whose absorbed anchor does not exist yet (the
join is deferred by design) and roll the ports_mapping back when the VLAN
reconcile fails, so a retry still diffs against the old mapping.
Agent: reject device config entries without a device_name instead of
silently dropping them (a malformed batch used to execute partially while
reporting success).
Also split the netem builder to clear C901 and drop the two per-file ruff
ignores the PR had added, keeping the remove-only policy.
The rebase onto 65c858d45 brought upstream's UP032 enforcement; the
conflict resolution kept this branch's semantics on the overlapping
lines, so their .format() calls needed the conversion afterwards
(ruff check --fix, 41 sites) plus the formatter's rewrap (7 files).
Replicates the CI recipe locally: pip check, ruff check/format, mypy
2.3.1 (clean, 434 files), unit suite 2216 passed.
The rebase onto upstream 3.1 (mypy + ruff I001/F401 waves) left the tree
short of the newly enforced baselines in three ways:
* contextlib / UbridgeError imports dropped as false duplicates during
conflict resolution left NameErrors in iou_vm and qemu_vm (F821);
the Dynamips router grabbed the compute-level NIOTAP instead of the
hypervisor one, breaking TAP anchoring (test failure).
* unused imports (KERNEL_UNSUPPORTED_FILTERS, UbridgeError, List) and
unsorted blocks left behind where the stack's later commits moved the
last usage elsewhere — ruff I001/F401/RUF022 now enforce these.
* mypy, now baseline-free: cache dict annotations in tc_probe, the
locators candidate list, the resolved-params None branch in the MCP
runner, and validate_bpf_syntax keeps the upstream Dict[str, Any]
return (the strict Optional[str] was wrong for the "valid" bool entry).
The e2e raw-frame probe resolves its compiler via shutil.which (S607)
and zips with strict= (B905); the remaining subprocess call is S603 by
nature and gets a per-file ignore like other test files.
ruff check/format clean, mypy at upstream parity (no new errors),
full unit suite 2167 passed.
The peer end of an Ethernet switch fast-path link is a NIOBridge with no
bridge of its own (the switch absorbed its anchor into the switch bridge).
Its update reconcile still ran the impairment pass with the empty filter
set the controller routes to that end — and an empty set is a plain
`tc reset` on the shared anchor, silently wiping the netem/cls_bpf/eBPF
the switch had just applied. Any link update did it (filters, markers,
suspend, capture start/stop), and the outcome depended on the link's
endpoint order: UdpLink.update() PUTs node1 then node2, so with the
switch first its apply ran first and the peer's reset ran last (API
green, data plane unimpaired), while with the peer first the reset ran
first and the filter survived. That order dependence is also why the
shipped e2e never caught it — it only created peer-first links.
Ownership is now explicit: only the end that owns the anchor's tc state
applies the impairment reconcile (_owns_anchor_impairments — a NIOBridge
with `bridge is None` is a passive carrier, the same idiom the
bridge-delete path already used). Markers, capture and the suspend
carrier still apply on both ends: markers reconcile per process, so
neither end can strip the other's, and the carrier converges by
agreement.
Red/green: two new docker unit tests fail on the unfixed tree with
`tc reset "<anchor>"` and pass after; the switch e2e also creates one
link switch-first and asserts the netem qdisc and the RTT hold under
that order (failed with the wipe, passes with the fix). Full unit suite
2110 passed; the switch e2e file passed end to end (378s).
A correctness read-through of the five delivered datapath documents, plus
the two code comments that carried the same stale vocabulary.
Datapath selection: the VendorDockerVM selector is `console_type=docker_exec`
alone (`GNS3_SKIP_INIT` and friends are behavior knobs of the chosen class);
the cross-compute and anchor-less fallbacks are named for what they are; the
Ethernet-switch endpoint is the documented exception to the anchoring rule,
and the IOL runner container is called out as the Docker node that does not
anchor on a veth.
Filter vocabulary: the stateful eBPF program evaluates four modes
(nth → quota → window → flow), GNS3 exposes three of them, and window_drop
joins the kernel-only list; the correl suffixes gate on the `rate` token as
the extension-build marker.
group_fwd_mask: the 0xfffd port mask is applied inside uBridge's `brctl addif`
itself, not as a separate command — the link-creation sequences and the
operation tables no longer show an arrow the server never sends. Same class
of fix for `link set up` on the IOL anchor (addif already brought it up; the
command only pushes a suspended anchor back down) and for the QEMU carrier
replication (QMP `set_link`, gated by `replicate_network_connection_state`).
Lifecycles and stop orders re-worded around the invariant rather than the
node type: the fd holder must release before the devices are deleted, which
is QEMU's process-first order, IOU's uBridge-first order and Dynamips'
hypervisor-NIO-first order. Deferred wiring for a stopped endpoint is
documented as the common path it is, and the Dynamips "stop is not a close"
note now says what recovery from a dead hypervisor actually requires.
Dynamips specifics: a c7200 needs an adapter in slot 1 (slot 0 is the fixed
I/O slot), the tap-fd direction is stated correctly (writes inject into the
kernel, reads return what the kernel transmits), and the anchor registry is
name-keyed against the controller's port matrix by convention, not derivation.
IOL port bridge naming: `bridge{bay}` on unit 0, `bridge{bay}_{unit}`
elsewhere; the leg swap happens on link delete/re-create or a datapath
switch, while suspend is an in-place NIO update and a project reopen is a
fresh build.
Verified and Roadmap sections: the in-repo e2e suite is distinguished from
the isolated-instance development harness (IOU), assertions are stated with
their bounds and capability gates, and the roadmap entries collapse into the
one remaining project — VXLAN/GENEVE for cross-compute links.
Code comments: `flow` added to the documented mode order in the eBPF apply
path, and the `nio_bridge` module docstring corrected on which filter types
run on the kernel datapath (bpf via cls_bpf, frequency_drop/quota/window_drop
via the eBPF classifier — neither is relay-only any more).
docker create_veth hardens only its FIRST end (the Docker shape: host
anchor plus a container leg that keeps its L3 life). A cascade end is
that second, unhardened end whenever the peer switch's create won the
race, and the kernel gives it an IPv6 link-local — a silent L3 identity
on the switch fabric (ubridge-l2-anchor-spec §E). Each switch now
hardens its OWN end with 'link l2only <end> on' once the pair exists,
whatever branch brought it there (fresh create, lost race, reuse);
idempotent, and best-effort with a warning on a uBridge without the
command.
The compute-side backstop of the controller's duplicate-port guard
(keep-port-links + live-port bind made the controller guard reliable;
this is the layer under it). IOU and Dynamips bound the adapter NIO
before the uBridge/hypervisor attach: a failed attach left the port's
binding replaced — the link that still owned the port lost its handle
and its host state leaked (the IOU EBUSY incident's 8 orphan TAPs).
Dynamips was worse: its wiring idempotence check silently skipped the
attach on a second binding and rebound only the bookkeeping. A port
that already carries an NIO is now refused, and both types bind the
NIO only after the wiring succeeded.
The spec's B table grows its fifth row: bridge add_nio_tap mints a
transient TAP when the name is free and hardens it at creation, while an
attach to an existing device (possibly a user-owned TAP with addresses)
is left untouched. The rule paragraph spells out attach-is-not-a-creation.
Delivery: uBridge fb72758, closing the gap the new e2e assertions found
(live: the switch's gns3*-0 relay TAP carried fe80::). The server issues
no link l2only of its own -- the switch call site now documents that the
TAP is minted-and-hardened by add_nio_tap, and the ESW feature doc
describes the uBridge-side close instead of a server workaround.
IOL runner containers (IOLDockerVM, GNS3_IOL_RUNNER) had no kernel
datapath: their guest leg is the runner's AF_UNIX socket pairs, so every
link rode the uBridge relay. Their Ethernet bays now own persistent TAP
anchors (gx names, the iol_docker key of the shared naming contract —
born at node start like IOU's, before any link, so the switch fast
path's deferred join can never miss them), and the per-port bridge's
topology leg swaps between UDP (relay) and the anchor (kernel link)
through the frozen stop -> delete -> add -> start contract — the unix
binding never re-binds across a swap.
- iol_docker_vm: KernelDatapathMixin hooks, the anchor lifecycle
(_prepare_tap_datapath at _start_ubridge: probe ubridge_bridge_tap +
ubridge_tap, sweep leftover, tap create born down), the kernel attach
(ensure-then-add — tap create is strictly create-only, IFF_TUN_EXCL,
so the ensure checks /sys/class/net first: an unconditional create
answers EBUSY on the device the start loop just made), the release
(port bridge stop + delete_nio_tap before the shared teardown) and the
stop order the fd holders dictate (port bridges first, then tap
delete, then the per-link bridges — all before the hypervisor stops).
- vendor_docker_vm: the port-bridge creation factored into
_ensure_unix_port_bridge (idempotent on the registered name), keeping
the NIOBridge rejection for the generic unix-socket containers.
- docker_vm: the three _kernel_veths lookups go through the
_kernel_host_ifc hook — behavior-identical for DockerVM, and it is
what lets the subclass route without overriding the binding methods.
- kernel_anchor: the iol_docker key (gx) and kernel_anchor_type, the
controller-side mapping for naming an IOL runner's anchor on the
switch fast path (the marker check stays inline: this module sits
below everything and must not import the controller package).
The bridge module gained delete_nio_tap (uBridge's gns3server-integration
contract, section 'bridge — generic NIO relay with a swappable TAP leg'):
a per-port bridge can swap its topology leg between UDP and a persistent
TAP through stop -> delete -> add -> start, which is what lets an IOL
runner container's port bridge carry a kernel-link anchor.
The probe needs no scratch objects and no capabilities: one command
against a bridge that cannot exist — 202 (unknown command) is an old
build, 214 (bridge doesn't exist) is a new one. Cached by binary identity
like every other probe; any failure reports unknown.
GET /v3/compute/capabilities gains ubridge_bridge_tap; the controller's
default capability dict carries it too (the single schema definition, per
the 1e67fbf12 lesson).
Switch-to-switch links were the last shape left on the relay: the switch
fast path works by absorbing the *peer's* anchor, and a switch has none —
its ports are its own bridge, and a kernel interface belongs to exactly
one bridge, so two bridges cannot be joined by enslaving anything.
Join them with a veth pair instead: one end per bridge, each side
enslaving its own end and applying its own port mode to it — an access
port on one side and a trunk on the other behaves like the real cable it
models. uBridge's docker create_veth/delete_veth (the primitive every
Docker adapter is already born from) creates and destroys the pair; the
names derive from the link id (utils.kernel_anchor.kernel_cascade_names),
so each side computes its own end and the names survive a restart.
The pair belongs to the link, and the two ends are wired by two
independent uBridge processes — the project-open batch drives different
nodes concurrently — so the protocol is order-free: both sides create the
pair when their own end is missing (one create wins; the loser's failure
is re-checked against the kernel and "the pair is there" is not an
error), and either side's teardown destroys it (deleting one veth end
takes the other with it, so both converge on the same gone pair and the
second delif answers ENOENT). A pre-existing end — this switch restarted,
or a leftover from a previous life of the link id — is reused rather than
recreated, because the other end may still be a live port of the peer
switch; a stale tc qdisc is all a reused end can carry.
No new wire format was needed: a cascade is an anchor NIO on both ends,
the absorbed-anchor case plus the peer end's name. That also corrects the
filter routing — the single-sided rule exists because an absorbed anchor
is one interface, while a cascade has one per switch, so impairments go
to both ends like any other kernel link.
Tests: the wire format, the eligibility flip and the both-ends re-push on
node start (controller); the naming contract (utils); the switch-side
lifecycle — create, reuse, lost create race, delete destroys the pair
(compute routes).
Drive-by: the relay filter test's compute mock carried no awaitable
`put`, so tests/controller/test_kernel_datapath_link.py failed when run
alone while an argument-less full run passed; callers now get one.
QEMU's process monitor calls stop() by itself when the process dies, so an
API stop and a process-death stop run concurrently in the normal case — and
the datapath release sat outside _execute_lock, letting both stops sweep the
adapter TAP map at the same time. Whichever finished first cleared
_kernel_taps under the other's iterator, which died with "dictionary changed
size during iteration" (the monitor's task, so it surfaced as "Task exception
was never retrieved") with the rest of its sweep, the orphan-bridge removal
and uBridge's shutdown skipped.
Keep the previous fix's ordering (process side first, then the devices) and
hold the lock across the release: serialized, the second stop finds uBridge
gone and sweeps nothing. Reproduced deterministically with two concurrent
stop() calls, the first parked mid-sweep — it raised at qemu_vm.py:1659
before the change, and every TAP is swept exactly once after it.
QEMU is the guest side of every adapter TAP, and uBridge's tap delete
refuses a device another process holds open ('Device or resource busy') —
a best-effort delete whose failure is suppressed. QEMU's stop did the
delete first and stopped the process after, so every node stop or project
delete leaked the persistent TAPs: found by inspecting the host after a
project delete (16 taps left behind, carrier down, no holder), and the
held-tap behaviour reproduced against uBridge directly.
Stop the QEMU process first, then remove the TAPs and the per-link kernel
bridges — process side first, then the devices, the same lesson as IOU's
reverse stop order. Verified live: node stop and project delete now leave
no taps and no processes behind, while a restart recreates the TAPs.
The adapter anchor naming also goes through the shared
utils.kernel_anchor helper, like the other node types.