diff --git a/docs/features/vxlan-cross-compute.md b/docs/features/vxlan-cross-compute.md index 1e07e2332..e498432ea 100644 --- a/docs/features/vxlan-cross-compute.md +++ b/docs/features/vxlan-cross-compute.md @@ -102,9 +102,9 @@ The `vxlan show` step is the idempotency contract: a device found under the link ## Known caveats -* **MTU is the whole story.** The kernel derives the tunnel device's MTU from the lower device (`underlay − 50` for the outer headers; `− 70` for an IPv6 underlay) and the per-link bridge, which carries no MTU of its own, reports the minimum of its ports — i.e. the derived value. A frame past it is dropped at the bridge egress **silently** (no ICMP, no log), which is why the datapath is opt-in: on a 1500-byte fabric a tunnel caps guests at 1450 while the relay carries 1500 with no tax. Guests set their own interface MTU, exactly like a VM or container raising eth0 on the same-host datapath. +* **MTU is the whole story.** The kernel derives the tunnel device's MTU from the lower device (`underlay − 50` for an IPv4 underlay, `− 70` for IPv6; the 50 is 20 outer IPv4 + 8 UDP + 8 VXLAN + 14 inner Ethernet header — a VXLAN packet carries the guest's complete inner frame, per `vxlan_headroom()` in `include/net/vxlan.h`) and the per-link bridge, which carries no MTU of its own, reports the minimum of its ports — i.e. the derived value. A frame past it is dropped at the bridge egress **silently** (no ICMP, no log), which is why the datapath is opt-in: on a 1500-byte fabric a tunnel caps guests at 1450 while the relay carries 1500 with no tax. Guests set their own interface MTU, exactly like a VM or container raising eth0 on the same-host datapath. * **Asymmetric underlays give one-way jumbo.** The derivation is per machine: a 9000-MTU compute paired with a 1500-MTU one forwards large frames in one direction only. Keep the fabric's MTU uniform between computes. -* **The MTU is derived once, at device creation.** Changing the underlay's MTU does not follow; a link delete/re-create (or project reopen) re-derives. +* **The MTU is derived at device creation and never tracks the underlay afterwards** — the vxlan driver registers no `NETDEV_CHANGEMTU` handler. Re-creation re-derives, and the wiring path deletes and re-creates the device (`_ensure_vxlan_device`) on every node start and link (re)creation, so the operational fix is: change the underlay MTU on **both** computes, then stop/start the nodes (or re-create the link, or reopen the project). A manual `ip link set mtu` on a live deployment also sticks: the bridge, unlike the tunnel, follows its ports' MTU at runtime (`NETDEV_CHANGEMTU → br_mtu_auto_adjust`), up and down, as long as the bridge MTU was never set by hand — ours never is. * **VNI collisions self-heal, not never happen.** The VNI is drawn from the upper VNI space (1M–16.7M) but two links of one deployment can still collide on a host. Every path where the kernel's refusal can surface carries its own heal: link creation retries a fresh VNI then degrades to the relay, project open re-dispatches refused entries the same way, and node start (a link wired while its nodes were stopped) stops the half-started node, re-wires the link on a fresh VNI or degrades it on a second refusal, and retries the start. A reopen re-rolls every link's VNI, so an unlucky link is not permanently unlucky. * **Security is the relay's, unchanged.** VXLAN carries no authentication; the VNI and the shared port are the only demux, and anyone who can reach the port can inject into any VNI. Treat the compute fabric as trusted, exactly as for the relay's UDP tunnels. * **Same-host compute pairs stay on the relay**, as do switch links, serial ports and unix-socket vendor containers — the tunnel is a per-link-bridge shape, and those cases are not (or would collide with themselves on one host).