The custom 14789 was picked to keep our demux socket off the ports other
VXLAN software listens on, but a private default surprises operators and
blocks interop with gear that pins the standard port. Default to the
IANA-assigned 4789 instead; the gns3_server.conf override is unchanged
for computes where swarm overlay/flannel-style VXLAN already owns 4789.
Mixed-version deployments must pin the port on every compute until they
all run the new default, or the two tunnel ends will disagree.
Per tunnel, not a host setting: a connected UDP socket runs the
kernel's route lookup for the peer, the chosen source address is
matched back to its owning interface, and that interface becomes
the dev= (and with it the derived MTU and the outer source
address). Different peers may ride different lower devices with
independently derived MTUs; an unresolvable peer is a wiring error
that degrades the link to the relay. Re-creation re-resolves too,
so route changes ride the same node-start / link-recreation paths
as MTU changes.
Verified against the running kernel's sources (7.2.6): the 50-byte
headroom is 20 outer IPv4 + 8 UDP + 8 VXLAN + 14 inner Ethernet
header (vxlan_headroom(), include/net/vxlan.h); the vxlan driver has
no NETDEV_CHANGEMTU handler so a live tunnel never follows its lower
device, while the bridge does follow its ports at runtime
(NETDEV_CHANGEMTU -> br_mtu_auto_adjust). Document the operational
consequence: the wiring path re-creates the device on every node
start and link (re)creation, so raising the underlay MTU on both
computes and stop/start-ing the nodes is the fix — no manual
ip link needed.
The scenario needed root for its network namespace and its fabric subnet
turned out host-specific: policy routing (a proxy's table consulted
before main) sends any non-RFC1918 destination out the default route, so
the peer compute was unreachable behind the veth — unfixable by picking
another subnet, and the failure paths leaked namespaces and processes.
Too environment-coupled to keep in the suite.
Unit coverage is unchanged (selection matrix, tunnel spec derivation,
VNI retry/degrade/self-heal, device lifecycle, capability probe); the
feature doc's Verification section now carries the manual two-compute
checklist instead of the e2e reference.
Cross-compute links can ride kernel VXLAN tunnels instead of the uBridge
UDP relay: each endpoint creates one VXLAN device (uBridge's rtnetlink
vxlan module) enslaved into the same per-link Linux bridge the same-host
datapath uses, so filters, markers, capture and suspend keep running on
the anchor unchanged. Opt-in (Server.enable_vxlan_cross_compute, default
off) because the tunnel caps the usable MTU at the underlay minus the
outer headers; the relay stays the default and the fallback.
The controller derives what the pair must agree on: one VNI per link
(upper VNI space, per-link random salt) and deterministic device names
(gvl{link}{node}) that make crashed-run leftovers claimable
(show -> delete -> create). Same-host compute pairs, switch endpoints
and serial/unix-socket containers stay on the relay.
A VNI the kernel refuses self-heals on every path it can surface:
interactive create retries a fresh salt then degrades to the relay;
project open re-dispatches refused batch entries the same way; node
start recognises the refused device, stops the half-started node,
re-wires both ends on a fresh VNI (or degrades the link) and retries
the start. The device MTU is derived from the lower device (dev=) and
the per-link bridge inherits it - no configuration. The capability
probe (ubridge_vxlan in /capabilities) runs one scratch create/delete
on a scratch port that never equals the configured dstport.
The e2e scenario runs two computes on one machine (a network namespace
joined by a veth pair as the fabric) and needs root.
