gns3-server/docs/features/ethernet-switch-kernel-datapath.md

Ethernet switch kernel datapath

The builtin Ethernet switch has been a real Linux kernel bridge since its brctl migration: one bridge per switch node, every port a persistent TAP enslaved to it (see ethernet-switch-ubridge-brctl-migration design notes in gns3server/compute/builtin/nodes/ethernet_switch.py). What remained on the userspace path was the cable: each port's TAP doubled as a uBridge relay endpoint, and a link to another node crossed peer anchor/UDP tunnel → uBridge relay ↔ port TAP → bridge — two extra copies of every frame.

The kernel datapath removes the cable's userspace leg: the switch absorbs the peer's anchor directly into its own bridge. The peer's veth host end / TAP becomes the switch port; frames cross peer anchor → switch bridge → other switch ports entirely in the kernel.

 relay (before):  peer ──UDP── ubridge[nio_udp↔nio_tap] ──tap── kernel bridge
 kernel (now):    peer anchor ──────────────────────────► kernel bridge
                  (the peer's own interface IS the switch port)

Datapath selection

A link with an ethernet_switch endpoint rides the fast path when the peer can anchor (same compute; Docker always — including IOL runner containers, per their own capability gate — QEMU/IOU/Dynamips per their capabilities) — the switch side needs no anchoring capability of its own, its bridge and brctl exist by construction. Exclusions, all staying on the relay:

• switch ↔ switch cascades ride the kernel too — through a veth pair joining the two bridges (one end per switch, each side applying its own port mode to its own end); see the wire format below;
• anchor-less peers (VPCS, cloud, the Dynamips-hosted hub, generic GNS3_UNIX_SOCKET_NIO containers);
• cross-compute links, non-Ethernet ports: the common rules.

Relay port TAPs. The switch's own relay-path port TAPs (the gns3{id}-N devices) are created implicitly by uBridge's bridge add_nio_tap — a name that is free is minted as a transient TAP, and uBridge hardens it L2-only at creation (the fifth creator of docs/design/ubridge-l2-anchor-spec.md §B, uBridge fb72758). The server issues nothing extra: without that hardening the device would take a kernel-default IPv6 identity whose DAD/MLD noise floods the segment, which is exactly what the e2e's relay-control §E.1 assertion guards against.

The wire format: one NIO names the other end's interface

 controller ──► switch compute:  nio_anchor   { anchor: "gv00010203e0p0", filters, markers }
 controller ──► peer compute:    nio_bridge   { bridge: null,             filters: {}, markers }

• nio_anchor (new, NIOAnchor) is the mirror of nio_bridge: "enslave this interface into my own bridge". The switch joins the anchor and applies the port's VLAN mode to it (brctl addif + the access/dot1q/ qinq composition), exactly as it programs its own port TAPs.
• nio_bridge with bridge: null on the peer end means "externally bridged": the peer keeps everything its node type does around the anchor (Dynamips opens the TAP in the hypervisor, IOU binds the fabric to it, QEMU's netdev, Docker's carrier) and applies markers/impairments, but never touches bridge membership — including on detach, where deleting a bridge it does not own would take the switch down.

The anchor's name is the shared naming contract (gns3server/utils/kernel_anchor.py): g<type>{node_id[:8]}e{adapter}p{port}, the same function the four node types create their anchors with. It is a pure function of (node type, node id, adapter, port) — valid whether or not the peer is running, which is what makes links to stopped nodes wireable at all.

Ownership. The switch owns everything about the anchor's bridge state (membership, VLANs). The link's link state — capture, markers, tc impairments, suspend — is attached to the same interface by exactly one end: filters and capture go to the switch (the controller assigns them single-sided; both ends would hit one interface twice), suspend rides both (same admin state either way), markers follow the usual capture-node routing and work from either end.

Stopped peers: deferred join and the re-push

The peer's anchor exists only while the peer runs. A link created against a stopped peer cannot be joined immediately, so the switch defers: it remembers the anchor and touches nothing (the link must not fail — this is the normal case when a project reopens while nodes are still starting).

The controller completes the wiring: after any node of a link reaches the started state (Node.start → UDPLink.node_started), a switch link's NIO is re-pushed to the switch, which joins the anchor now. The same re-join runs when a peer restart replaced its interface under the same name (QEMU recreates its TAPs at every start): the membership is checked against the kernel (/sys/class/net/<bridge>/brif/<anchor>), not assumed.

Port setting changes (ports_mapping update) edit VLAN membership in place: only the ports whose mode/VLAN actually changed are touched, and each change deletes the previous mode's entries the new mode does not want and adds the new mode's (the dot1q admits-all entry is one range operation in both directions). A port never leaves the bridge to change its VLANs — no traffic gap, no FDB flush, and nothing that detaches an interface this switch does not own.

This replaced an earlier "reset by re-enslaving" shortcut (delif + addif, so the kernel would hand the port a fresh default membership — used because a mode change must drop stale entries, e.g. dot1q's all-VIDs range). On the switch's own relay TAPs it merely cost a brief traffic gap; on absorbed anchors re-enslaving an interface owned by another node proved to be where links died: live testing showed an anchor occasionally losing its bridge membership right after a ports_mapping update (later updates then failed delif with EINVAL), reproducibly with qemu, IOU and Dynamips peers, never with Docker's veths. That failure was root-caused later: it is NetworkManager releasing NM-managed TAPs from their bridge tens of milliseconds after (re)enslavement — veths are not NM-managed, which is exactly why only TAP peers showed it (the mechanics and the nmcli device set <tap> managed no remedy are in the L2-anchor spec). The in-place reconcile removes this particular re-enslavement, but link delete

• recreate and peer restarts still re-enslave the anchor, so the hazard remains on NM hosts wherever NM manages the taps; the kernel e2e scenarios take the anchors out of NM's hands (harness.unmanage_from_networkmanager) before they exercise those paths. The full record — forensic signatures, the minimal server-free reproducer, the NetworkManager source-level mechanism and the verified host fix — is in docs/bugs/networkmanager-tap-release.md.

Link-local frames (LACP, LLDP, 802.1X, STP)

The bridge stands in for a cable, but the kernel's br_handle_frame() does not forward the IEEE 802.1D reserved range (01:80:c2:00:00:00-0f) by default — LACP, LLDP/DCBX and 802.1X would never cross a kernel link while the UDP relay carries them fine. uBridge therefore opens every port's per-port group_fwd_mask (IFLA_BRPORT_GROUP_FWD_MASK) to 0xfffd at brctl addif time (Linux 4.15+, best-effort on older kernels). The bridge-level knob cannot do this: BR_GROUPFWD_RESTRICTED rejects bits 0-2, so LACP is reachable per port only.

Result on kernel links: ordinary multicast, STP/RSTP, LACP, 802.1X and LLDP/DCBX all cross; 802.3x PAUSE / PFC does not — case 0x01 in br_handle_frame() drops it unconditionally and no mask can enable it. That is a documented limit, not a regression (PFC's hardware semantics are out of emulation's reach anyway); a lab that needs PAUSE frames on the wire must use a relay link. Verified live by tests/e2e/test_docker_link_local_frames.py (per-MAC guest-to-guest matrix); the contract and kernel references live in docs/design/ubridge-link-local-frame-forwarding-spec.md.

The switch node's own bridge ports carry the same mask: the builtin Ethernet switch is a transparent L2 segment in GNS3, not a protocol endpoint, so LACP/LLDP between two peers crosses it (a physical switch would terminate them itself).

Verified

Unit level (all under tests/): the switch's kernel-port lifecycle (join with VLAN composition, deferred join, re-join, detach with marker/tc/ carrier teardown, capture on the anchor, batch acceptance for project open); the in-place VLAN reconcile (every mode transition, untouched ports emit nothing, relay TAPs and absorbed anchors alike); the shared naming contract; the mixin's externally-bridged branch (no bridge membership work in attach/remove, the bridge sweep never deletes a bridge the node does not own); the controller's eligibility table, wire format (anchor name, single-sided filters, endpoint-order alignment), kernel_datapath reporting and the node-start re-push.

Live end-to-end on a real server (tests/e2e/test_ethernet_switch_kernel_datapath.py plus scripted reproduction rounds over the REST API):

• two qemu peers (the previously 2-out-of-2 failing scenario): deferred links → start → both anchors joined; four ports_mapping cycles (10/20 ↔ 10/10) with both anchors in the bridge throughout, no failed update, kernel VLAN state correct at every step;
• a mixed switch (2 × IOU + 2 × Dynamips on one bridge, four kernel ports): four anchors stable across updates that changed all ports and across updates that changed only two;
• Docker peers with real ICMP: isolating VLANs cuts the ping with both anchors still enslaved (the filter does the cutting, not a broken link); restoring the VLAN restores the ping;
• VPCS peers (the relay path, same switch-side mechanics as cloud): the relay TAPs likewise stay enslaved across VLAN changes, with a real ping proving isolation and recovery.

The cascade has its own in-repo e2e (tests/e2e/test_ethernet_switch_cascade.py): two switches joined by the link-owned veth pair, two c7200 routers per switch, real ICMP crossing both bridges — the pair is born UP with exactly one end per bridge, a dot1q trunk cascade carries two VLANs at once, the §E.2 idle-silence window with all four router interfaces shut (the absorbed anchors, both cascade ends and both switch bridges), the symmetric access/trunk mode flip on the cascade ends isolates and revives a VLAN in place, a delay filter lands as netem on both ends (one per direction, RTT ≈ 2×delay), suspend admin-downs both ends, and a link delete/re-create retires the whole pair and mints a new one under the new link id — with zero host residue after the project goes.

The host-side checks throughout are the kernel's own view: bridge membership under /sys/class/net/<bridge>/brif/, the master link attribute, bridge vlan show, and live pings where the peer allows it.

Configuration

Same switch as every node type:

[Server]
enable_kernel_datapath = True

Roadmap

Switch-to-switch links (a veth pair joining the two kernel bridges, with each side's port mode applied at its own end) have landed (see above), and so has the IOL runner container (iol-docker-kernel-datapath.md — its iol_docker anchors absorb like any peer's). Cross-compute kernel links need VXLAN/GENEVE encapsulation, deferred until every node type is kernelized.