gns3-server/docs/bugs/networkmanager-tap-release.md

This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.

NetworkManager Releases Enslaved TAP Anchors from Their Bridge (Silent Link Death)

Bug Report

Date: 2026-10-05 (root-caused; the same signature first hit on 2026-10-01 during the Ethernet-switch kernel-datapath development) Severity: High on affected hosts (one link dies silently — API and UI stay green; ~50 % per re-enslavement while NetworkManager has not yet assumed the TAP; environment-dependent and intermittent) Status: Root-caused; host-side fix verified. The NetworkManager declaration (unmanaged-devices) takes the reproducing loop from 19-20/40 to 0/40 and lets the e2e scenario pass with all test-side workarounds disabled. Test-side best-effort hardening is shipped (harness.unmanage_from_networkmanager); a product-side guard remains an open decision. Component: Host environment — NetworkManager × uBridge-created TAP anchors. No GNS3 code path performs the detaching operation: gns3-server and uBridge log nothing and their bookkeeping stays intact.

Symptoms

On a kernel-datapath link, after a re-enslavement of the TAP anchor — a link delete + recreate, a port re-join after a peer restart — the link may die: ping 0 %, routing protocols never form — while every API/UI surface reports health (kernel_datapath: true, no error, no log). The node, the emulated router and the anchor device itself keep running; only the bridge membership is gone.

Self-heal exists but nothing forces it: any subsequent NIO update re-checks /sys/class/net/<bridge>/brif/<anchor> and re-attaches (EthernetSwitch.update_nio's "not joined" branch, and the equivalent attach path per node type), which is why the state heals on the next filter change, suspend toggle or peer restart. The first enslavement of a freshly created TAP is stable in every observation; the hazard is re-enslavement before NetworkManager has assumed the device (a window of minutes), at ~50 % per attempt.

Forensic signatures (what the corpse looks like)

• /sys/class/net/<bridge>/brif/<anchor> missing, /sys/class/net/<anchor>/brport/ gone — the port was released, not the device destroyed: ifindex unchanged, ip -d link show reports no master.
• Flags drop IFF_PROMISC|IFF_ALLMULTI (0x300, set by the kernel bridge on enslavement): as seen with a 50 Hz brif sampler, fl 0x1303 → 0x1003 while administrative UP is kept.
• The release lands 58-87 ms after the join — after the attach API has already returned success, so wait_until-style checks see the join and the next traffic sample sees the blackhole.
• No brctl delif (or any netlink write) from any GNS3 process; server-side registrations (_kernel_ports, NIO state) are untouched; uBridge logs stay clean.
• nmcli device status shows the anchor as managed by NetworkManager: disconnected before it assumes the device, connected (externally) after.

Root Cause (NetworkManager side, source-level)

On hosts where NetworkManager manages the anchors — it claims uBridge-created tun/tap devices by default (nmcli device then shows them managed) — NM's device state machine resolves the fresh enslavement in its own favour while the device is in its pre-assumption window:

1. An external enslavement triggers an assumption recheck — NetworkManager commit a6ceb382e9 ("device: connect slave assumption recheck on external enslavement") exists precisely for ip link set <dev> master <bridge> performed by someone else.
2. The recheck activates (assumes) the device, and activation enforces that a connection which does not declare a master must not keep its kernel master: the slave is force-released, reason NM_DEVICE_STATE_REASON_CONNECTION_ASSUMED — NetworkManager commit ec12912908 ("device: enforce the absence of a master during activation"). This is the detaching operation; it runs several asynchronous state-machine steps after the enslavement, which is the 60-80 ms latency.
3. Once the device is assumed (connected (externally)), later external enslavements take the opposite path: external changes must not touch the device — NetworkManager commit 3127fb0d17 ("device: don't let external changes cause a release of the slave", configure = FALSE, "let the device be"). The model is updated; the kernel is left alone. This is why assumed taps were stable across 14 re-enslavements in the field, and why the hazard window ends at assumption.

Both processes' worlds are mutually exclusive here: NM requires "no master during activation", the datapath requires "enslaved to the switch/per-link bridge" — there is no configuration in which both hold. The sanctioned resolution is a jurisdiction declaration (mark the devices unmanaged), not a race.

Docker veth host ends are not NM-managed (and the veth naming/type heuristics historically keep NM away — see History), so only the TAP anchor families are affected: gd (Dynamips), gq (QEMU), gi (IOU), gx (IOL runner containers), and switch-absorbed anchors.

Reproduction (minimal, gns3-server not involved)

A self-contained reproducer — one uBridge, one TAP, one kernel bridge, an enslavement loop — needs none of the server. It drives uBridge's control socket through the repository's own client, so no build step and no extra dependencies beyond the venv:

#!/usr/bin/env python3
"""Minimal reproducer: NetworkManager releases an enslaved TAP from its bridge.

Run from the gns3-server repo root on an affected host (Linux + NetworkManager
managing the TAPs; uBridge installed with cap_net_admin):

    venv/bin/python nm_tap_release_repro.py       # 40 re-enslavements
    nmcli device set gdlab-nm managed no          # the A/B flip (same tap!)
    venv/bin/python nm_tap_release_repro.py       # -> 0 releases

The tap and bridge are kept between runs (same device = same NM state).
Clean up afterwards:

    venv/bin/python nm_tap_release_repro.py --cleanup
"""
import asyncio
import os
import shutil
import subprocess
import sys
import time

sys.path.insert(0, os.getcwd())  # run from the repo root for gns3server imports
from gns3server.compute.ubridge.ubridge_hypervisor import UBridgeHypervisor

TAP = "gdlab-nm"
BRIDGE = "gns3lab-nm"
SOCK = "/tmp/nm_tap_release_repro.sock"
ITERATIONS = 40
SAMPLE_MS = 200  # membership sampled at ~1 kHz for this long after each addif


def in_bridge():
    return os.path.exists(f"/sys/class/net/{BRIDGE}/brif/{TAP}")


async def main():
    ubridge = shutil.which("ubridge")
    if not ubridge:
        sys.exit("ubridge not found in PATH")
    if os.path.exists(SOCK):
        os.unlink(SOCK)
    proc = subprocess.Popen(
        [ubridge, "-U", SOCK], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, cwd="/tmp"
    )
    try:
        h = UBridgeHypervisor(socket_path=SOCK)
        await h.connect()

        async def send(cmd):
            try:
                await h.send(cmd)
            except Exception:
                pass  # ENOENT during setup/teardown is expected

        if "--cleanup" in sys.argv:
            await send(f'tap delete "{TAP}"')
            await send(f'brctl delete "{BRIDGE}"')
            return
        await send(f'tap create "{TAP}"')  # no-op if the tap already exists
        await send(f'brctl create "{BRIDGE}"')
        await send(f'link set "{BRIDGE}" up')
        await send(f'brctl vlanfiltering "{BRIDGE}" on')

        releases, latencies = 0, []
        for _ in range(ITERATIONS):
            await send(f'link set "{TAP}" up')
            await send(f'brctl addif "{BRIDGE}" "{TAP}"')
            t0 = time.time()
            for _ in range(SAMPLE_MS):
                await asyncio.sleep(0.001)
                if not in_bridge():
                    releases += 1
                    latencies.append(round((time.time() - t0) * 1000, 1))
                    break
            await send(f'brctl delif "{BRIDGE}" "{TAP}"')
            await send(f'link set "{TAP}" down')
            await asyncio.sleep(0.005)

        print(f"releases: {releases}/{ITERATIONS} re-enslavements", flush=True)
        if latencies:
            print(f"release latency after join (ms): {latencies[:10]}")
            print(f"expect ~19-20/40 while NetworkManager manages {TAP}, "
                  f"0/40 after `nmcli device set {TAP} managed no`")
    finally:
        proc.kill()


asyncio.run(main())

How the numbers are produced:

• 40 iterations; each is one re-enslavement: link set <tap> up, brctl addif, then membership sampled at ~1 kHz for 200 ms, then brctl delif + link set down.
• A release is counted when /sys/class/net/<bridge>/brif/<tap> disappears inside that 200 ms window; the script prints the release latency after the join (the NM release lands at 58-87 ms).
• The tap is kept between runs, so the A/B flip is single-variable on the same device; a fresh device would reset NM's assumption state (the first enslavement of a fresh tap is always stable).
• The numbers below were measured on a host without the declaration from Fix. On a host that already carries it the tap is unmanaged from birth and the loop prints 0/40 by design; to re-observe the baseline there, temporarily remove the conf.d file (sudo rm /etc/NetworkManager/conf.d/99-gns3-datapath.conf && sudo systemctl reload NetworkManager) or run on an un-declared host. A runtime nmcli device set <tap> managed yes is not enough — the configuration match wins (the device stays unmanaged and the loop still prints 0/40; measured).

Measured on an affected host (same code, five baseline runs, then the A/B between runs):

Condition	Releases / 40 re-enslavements
NM-managed (default)	19, 20, 19, 20, 20
nmcli device set <tap> managed no, same tap	0
re-managed (nmcli device set <tap> managed yes), same tap	20

e2e-level reproduction: tests/e2e/test_ethernet_switch_kernel_datapath.py::test_ethernet_switch_kernel_fast_path fails at its link delete/recreate step (wait_ping after the anchor re-join) roughly half of fresh runs on an NM host — it failed twice consecutively on 2026-10-05 during the 3.1 rebase validation before the fix.

Fix

Three layers, of which the first is the verified root fix:

1. Host declaration — /etc/NetworkManager/conf.d/99-gns3-datapath.conf:

   [keyfile]
   unmanaged-devices=interface-name:gd*;interface-name:gq*;interface-name:gi*;interface-name:gx*;interface-name:gv*;interface-name:gns3*
   

   then systemctl reload NetworkManager. Devices matching are unmanaged from birth (also future ones and after re-creations); hosts without NM treat the file as inert. Verified on the affected host: minimal loop 0/40, and the e2e scenario passes with the test-side workaround disabled (pure host config); re-verify with the reproducer below — same tap, expect 0/40 after the reload. This belongs in installers/documentation, not in the server (no root; and the server must not reconfigure the host's NM).

2. Test-side best effort — harness.unmanage_from_networkmanager(*taps) runs nmcli device set <tap> managed no where nmcli exists (no-op otherwise); kernel-datapath scenarios that re-enslave anchors call it after their nodes have booted. This keeps the e2e deterministic on un-declared hosts.

3. Product-side guard — open decision: a bounded post-attach re-check (~250 ms) that re-attaches once and warns (naming NetworkManager), a warn-only variant, or nothing (documentation only). Until decided, an undeclared host relies on the existing heal-on-next-update path.

Scope

• Affected: Linux hosts where NetworkManager manages the TAPs — desktop-native gns3server deployments (the most common GNS3 form factor for Linux users). Every TAP anchor family, the switch fast path and per-link kernel links alike.
• Not affected: the GNS3 VM (its Ubuntu-server stack is netplan/systemd-networkd, no NM — verified against the gns3-vm repository), hosts without NM, Docker veth anchors, relay links, cross-compute links (relay by default).
• Blast radius of one occurrence: exactly one link, silently dead; no incorrect data, but it can mislead an experiment's conclusions (e.g. "this protocol never converges here") and it mimics the project's own "API green, data plane dead" bug class — during the rebase validation it was triaged as a regression until the timing signature (join, then +60 ms release with no command) identified it as environmental.

History

• 2026-10-01: the same signature first hit during the Ethernet-switch development, described then as "an anchor occasionally losing its bridge membership right after a ports_mapping update"; reproduced with QEMU, IOU and Dynamips peers (all TAPs) and never with Docker veths; the then-unknown cause was worked around by making VLAN reconfiguration in-place (no re-enslavement) and the failure mode was assumed gone "by construction". See docs/features/ethernet-switch-kernel-datapath.md.
• 2016: the same NM-vs-GNS3 jurisdiction class hit Docker veths — NM configured them like plain NICs and a container DHCP server hijacked the host's default route (issue #440, closed). Upstream worked around it by renaming gns3-veth… to veth-gns3-…, the commit message reading "the name create different behavior with network manager" (59c1e125d). NM's management heuristics even depend on interface names — the rename moved the veths into NM's ignored namespace.
• 2026-10-05: root-caused with the minimal harness and the single-variable A/B; host fix applied and verified; test-side hardening shipped; full write-up (this document).

Related

• NetworkManager commits: a6ceb382e9, ec12912908, 3127fb0d17.
• GNS3 issue #440 and commit 59c1e125d (veth rename).
• docs/design/ubridge-l2-anchor-spec.md — "NetworkManager releases managed TAPs from bridges" subsection (short form).
• docs/features/ethernet-switch-kernel-datapath.md — the corrected account of the 2026-10-01 incident.
• tests/e2e/harness.py — unmanage_from_networkmanager.