test(e2e): take the switch anchors out of NetworkManager's reach

The esw kernel fast-path scenario deletes and re-creates a link, which
re-enslaves the peer anchor — and on NetworkManager hosts NM's netdev
state machine can release an NM-managed TAP from its bridge some 60-70 ms
after enslavement, ~50% of re-enslavements, while the tap is in NM's
not-yet-assumed state. The freed port stays administratively up and no
GNS3 process logs anything: the membership assert sees the join, then the
ping measures a silent blackhole (this was the line-277 e2e failure).

Root-caused with a server-free harness (one uBridge, one fresh tap, one
bridge, an addif/delif loop): the same tap frees 20/40 while NM-managed,
0/40 after `nmcli device set <tap> managed no`, 20/40 again when
re-managed. Docker's veth host ends are not NM-managed, which is why only
TAP anchors ever showed it — including the "anchor lost its bridge
membership after a ports_mapping update (TAP peers only)" incident from
the switch's development, whose conclusion is corrected in the docs.

Unmanage the anchors once the routers have booted, before the scenario
re-enslaves them; document the failure signature and the host-side remedy
(per-device nmcli call, or NM unmanaged-devices configuration) in the
L2-anchor spec and the switch feature doc.
This commit is contained in:
YueGuobin 2026-10-05 13:39:30 +08:00
parent dc5e2d90ba
commit 1c85befbab
No known key found for this signature in database
4 changed files with 51 additions and 4 deletions

View File

@ -86,6 +86,12 @@ acceptance test in §E.2 therefore measures after a settle window. Suppressing
it (e.g. `disable_ipv6` on the bridge before enslaving, for the MLD half) would
be a new, separate requirement and has not been asked for.
### NetworkManager releases managed TAPs from bridges (measured)
On hosts where NetworkManager manages the data-plane TAPs — it claims uBridge-created tun/tap devices on many desktops (`nmcli device` shows them as `disconnected` before NM's state machine has assumed them, `connected (externally)` after) — NM can **release a tap from its bridge** some 60-70 ms after enslavement, while the port is in that not-yet-assumed state. Measured with a minimal harness (one uBridge, one fresh tap, one bridge, an `addif`/`delif` loop, no server): ~50 % of re-enslavements lose the port, the freed device stays administratively **up** with the bridge's promiscuity/allmulti cleared, and no GNS3 process sends anything — gns3-server and uBridge log nothing, and the server's attachment bookkeeping stays intact. The *first* enslavement of a newly created tap was stable in every run; the hazard is a **re-enslavement** (link delete + recreate, a port re-join after a peer restart) before NM assumes the device — on the same tap, `nmcli device set <tap> managed no` flips the failure rate from 20/40 to 0/40, and re-managing it restores 20/40.
Consequence for an affected link: the port silently leaves the bridge (frames drop) while the API stays green — the next NIO update re-heals it, because the switch's `_kernel_update` re-checks `brif` membership, but nothing forces one. Docker's veth host ends are not NM-managed and never showed the failure; every TAP anchor family is exposed (Dynamips `gd`, QEMU `gq`, IOU `gi`, IOL `gx`, and switch-absorbed anchors). Host-side remedy: `nmcli device set <tap> managed no` (runtime, per device), or NM's `unmanaged-devices` configuration keyed on the anchor name prefixes; the e2e harness takes the anchors it re-enslaves out of NM's hands (`harness.unmanage_from_networkmanager`).
### Goal
Every host-side device uBridge creates for the data plane is **pure L2**: no

View File

@ -113,10 +113,16 @@ on **absorbed anchors** re-enslaving an interface owned by another node
proved to be where links died: live testing showed an anchor occasionally
losing its bridge membership right after a `ports_mapping` update (later
updates then failed `delif` with EINVAL), reproducibly with qemu, IOU and
Dynamips peers, never with Docker's veths. Replaying the exact command
sequence by hand against uBridge (held TAPs, no server involved) did not
reproduce it, and the in-place reconcile removes the operation entirely —
the failure mode no longer exists by construction.
Dynamips peers, never with Docker's veths. That failure was root-caused
later: it is NetworkManager releasing NM-managed TAPs from their bridge
tens of milliseconds after (re)enslavement — veths are not NM-managed,
which is exactly why only TAP peers showed it (the mechanics and the
`nmcli device set <tap> managed no` remedy are in the L2-anchor spec). The
in-place reconcile removes this particular re-enslavement, but link delete
+ recreate and peer restarts still re-enslave the anchor, so the hazard
remains on NM hosts wherever NM manages the taps; the kernel e2e scenarios
take the anchors out of NM's hands
(`harness.unmanage_from_networkmanager`) before they exercise those paths.
## Link-local frames (LACP, LLDP, 802.1X, STP)

View File

@ -1053,6 +1053,32 @@ def bridge_members(bridge):
return sorted(os.listdir(f"/sys/class/net/{bridge}/brif"))
def unmanage_from_networkmanager(*names):
"""Best-effort ``nmcli device set <name> managed no`` for the given
interfaces.
NetworkManager (desktop hosts) manages the TAP devices uBridge creates.
While a tap is in NM's not-yet-assumed state, NM's netdev-event state
machine can *release* it from its bridge some 60-70 ms after
enslavement (measured: ~50% of re-enslavements on an NM host): the port
is left administratively up, out of the bridge, with no server-side log
or error — a silent blackhole. Scenarios that detach and re-enslave an
anchor (link delete + recreate, node restart) must call this once their
nodes have booted, or they are nondeterministically flaky on such
hosts. Silently a no-op where nmcli is unavailable.
"""
nmcli = shutil.which("nmcli")
if not nmcli:
return
for name in names:
# S603: resolved binary, fixed argv, no shell
subprocess.run( # noqa: S603
[nmcli, "device", "set", name, "managed", "no"],
capture_output=True,
check=False,
)
# ---------------------------------------------------------------------------
# L2-anchor spec assertions (docs/design/ubridge-l2-anchor-spec.md §E)
# ---------------------------------------------------------------------------

View File

@ -146,6 +146,15 @@ def test_ethernet_switch_kernel_fast_path():
c1 = _boot(server, pid, r1_id)
c2 = _boot(server, pid, r2_id)
# Keep NetworkManager's hands off the anchors: on NM hosts its
# netdev state machine can release a freshly re-enslaved tap from
# the bridge ~60 ms after the join (silently — the port stays up,
# the server keeps believing it is attached), which the link
# delete/recreate below triggers nondeterministically. The first
# join at node start is unaffected; this covers everything after
# it.
harness.unmanage_from_networkmanager(a1, a2)
# Deferred joins completed by the node-start re-push: the anchors
# ARE the switch ports now.
assert harness.wait_until(lambda: harness.bridge_members(sw_bridge) == sorted([a1, a2]), timeout=15), (