fix: UDP port allocation race causing link self-loop (one-way links)

PortManager.get_free_udp_port had an unguarded find-then-add sequence.
A link allocates both ends concurrently (asyncio.gather in
UDPLink._prepare -> two POST /ports/udp) and FastAPI runs the sync
route handler in a threadpool, so both threads could probe the same
'free' port before either recorded it — handing lport == rport to both
ends. uBridge sets SO_REUSEADDR on UDP NIO sockets, so the double bind
succeeds silently and the kernel delivers everything to the last-bound
socket: one node starves, the other echoes to itself.

Make every TCP/UDP allocate/reserve/release path atomic with an RLock,
and rebuild _link_data in UDPLink._prepare so reset() commits the fresh
port pair instead of re-sending the stale, already-released one.

Regression tests: threaded barrier allocation never returns duplicates
(red on the old code, UDP and TCP); reset() leaves exactly one mirrored
NIO pair per side with lport != rport (red on the old code).

docs/README.md

@@ -109,7 +109,7 @@ Cisco XRd as a GNS3 Docker router: vendor path + shm/device injection (`GNS3_SHM
 ## Known Issues (`bugs/`)
 
 - [Telnet Server Connection Race Condition](bugs/telnet-server-connection-race-condition.md) — `getpeername()` error when client disconnects during connection setup (High severity, Open)
-- [Docker Link UDP Self-Loop](bugs/link-udp-self-loop.md) — intermittent one-way link (one end's NIO points at itself); delete/recreate the link as workaround (Medium severity, Open)
+- [Docker Link UDP Self-Loop](bugs/link-udp-self-loop.md) — intermittent one-way link (both ends handed the same UDP port by an allocation race); **fixed** (Medium severity)
 
 ---
 

docs/bugs/link-udp-self-loop.md

@@ -5,25 +5,24 @@ See LICENSE file for licensing information.
 
 > This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
 
-
 # Docker Link UDP Self-Loop Bug (One-Way Link)
 
 ## Bug Report
 
 **Date**: 2026-08-14
 **Severity**: Medium (one-way connectivity, CPU burn from packet duplication; intermittent)
-**Status**: Open — root cause not yet pinpointed; workaround reliable
-**Component**: Link wiring — `gns3server/controller/udp_link.py` (`_prepare` /
-`pop_preallocated_udp_port` in `gns3server/controller/project.py`) interacting with
-node/uBridge restarts
+**Status**: **Fixed** — root cause found and unit-tested (same day)
+**Component**: UDP port allocation — `gns3server/compute/port_manager.py`
+(`get_free_udp_port` find-then-add race); secondary: `gns3server/controller/udp_link.py`
+(`_prepare` accumulated stale `_link_data` on reset)
 
 ## Symptoms
 
-Two Docker nodes (observed with Cisco XRd; likely node-type agnostic) linked on the
+Two Docker nodes (observed with Cisco XRd; node-type agnostic) linked on the
 same compute cannot ping each other. Packet capture on the link shows only **one**
 side sending ARP. The other side's traffic never appears on the link at all.
 
-## Evidence (from a live occurrence)
+## Evidence (from the live occurrence)
 
 Captured simultaneously on both nodes' uBridge `bridge1` (see diagnostics below):
 
@@ -34,23 +33,44 @@ Captured simultaneously on both nodes' uBridge `bridge1` (see diagnostics below)
 | B's `ethN` counters | RX ≈ TX ≈ 5000+ — B receives its own transmissions back |
 | A's `ethN` counters | TX > 0, RX = 0 — never receives anything |
 
-Conclusion: **B's `nio_udp` rport pointed at B's own lport** — a UDP self-loop. B's
-uBridge had restarted ~77 s after A's (node stop/start during the session),
-i.e. the link was re-wired at least once.
+## Root Cause (confirmed)
 
-Deleting and re-creating the link fixed it immediately (fresh port allocation).
+**`PortManager.get_free_udp_port` had an unguarded find-then-add sequence.**
+A link allocates the UDP port for **both ends concurrently**
+(`asyncio.gather` in `UDPLink._prepare` → two `POST /ports/udp`). The route
+handler is a sync `def`, so FastAPI executes the two requests **in parallel
+threads**. Both threads ran `find_unused_port` (socket-probing, GIL-releasing)
+before either reached `_used_udp_ports.add(port)` — the set add is idempotent,
+so no error was raised and **both ends were handed the same port number**:
+`lport == rport` on both NIOs, a literal self-loop.
 
-## Root Cause Analysis (narrowed, not final)
+Why it was *silent* and *asymmetric*:
 
-- `UDPLink._prepare()` builds mirrored NIO data correctly
-  (`node1: lport=P1/rport=P2`, `node2: lport=P2/rport=P1`) — the logic itself is sound.
-- Suspects for the corrupted runtime state:
-  1. `Project.pop_preallocated_udp_port()` — the batch project-open preallocation
-     (link-create performance work) racing with link re-creation;
-  2. link re-creation racing a node/uBridge restart (commit NIOs to a uBridge that is
-     being torn down/rebuilt), leaving a stale/self-pointing NIO on one side.
-- A deterministic reproduction is still needed: create two Docker nodes + link,
-  restart one node, then verify the UDP wiring (see diagnostics).
+- uBridge sets `SO_REUSEADDR` on UDP NIO sockets (`ubridge/src/nio_udp.c`), so
+  the second bind of the same port **succeeds** instead of failing with
+  `EADDRINUSE` — link creation returned success.
+- With two sockets bound to the same port, the kernel delivers to one of them
+  (last bound wins). The node that started later — in the live case B,
+  restarted ~77 s after A — received **everything**: A's packets *and* its own
+  transmissions echoed back. The 77 s restart did not cause the corruption; it
+  only decided which end starves.
+- The same-compute condition is part of the trigger: both allocations hit the
+  same `PortManager` instance (a cross-compute link races two processes and
+  cannot self-collide).
+
+A second, smaller defect was found while auditing: `UDPLink._prepare()`
+**appended** to `self._link_data` but the committed NIOs are always taken from
+indices 0/1 — after `reset()` (delete + create on the same object) the stale,
+already-released port pair was re-committed and the freshly allocated ports
+were leaked.
+
+## Fix
+
+| Change | Where |
+|---|---|
+| `threading.RLock` making find-then-add (and reserve/release) atomic for TCP and UDP | `gns3server/compute/port_manager.py` |
+| `_prepare()` rebuilds `_link_data` from scratch instead of appending | `gns3server/controller/udp_link.py` |
+| Regression tests: threaded allocation never returns duplicates (red on the old code); `reset()` commits the fresh mirrored pair with `lport != rport` | `tests/compute/test_port_manager.py`, `tests/controller/test_udp_link.py` |
 
 ## Diagnostics (uBridge console is the fast path)
 
@@ -63,8 +83,10 @@ Deleting and re-creating the link fixed it immediately (fresh port allocation).
 2. **Container counters** — `docker exec <cid> ip -s link show ethN`:
    TX>0/RX=0 → peer never returns; RX≈TX huge with µs-scale duplicates → self-loop.
 3. **UDP sockets** — `ss -uln` (no `-p`; uBridge runs setuid-root so process names
-   are hidden, ports are visible): a two-node link owns exactly two "orphan" UDP ports.
-4. **Workaround** — delete and re-create the link (or stop/start both nodes).
+   are hidden, ports are visible): a two-node link owns exactly two "orphan" UDP
+   ports. **One port instead of two = this bug.**
+4. **Recovery** — delete and re-create the link (or stop/start both nodes);
+   with the fix, the corruption no longer occurs in the first place.
 
 Note: a Docker node's in-container `ethN` is a **TAP device** whose file descriptor
 lives inside uBridge (the interface is created host-side, then moved into the
@@ -74,4 +96,6 @@ waste time hunting for one in the host namespace.
 ## Related
 
 - `docs/features/vendor-nos-xrd.md` — troubleshooting table entry pointing here.
-- Link-create batch optimization (`controller/udp_link.py`, project-open bulk path).
+- Link-create batch optimization (`controller/udp_link.py`, project-open bulk path)
+  was exonerated: the pool path allocates sequentially in one handler and cannot
+  self-collide.

docs/features/vendor-nos-xrd.md

@@ -158,7 +158,7 @@ sequenceDiagram
 | `Invalid interface entries ... XR_MGMT_INTERFACES` | use `xr_name=Mg0/RP0/CPU0/0` |
 | Host audio muted / USB reconnects when the node starts | set `GNS3_MASK_UDEV=1` |
 | Config lost across stop/start | `extra_volumes` must be `/xr-storage-shadow` |
-| Two nodes can't ping, only one side ARPs | GNS3 link wiring bug (UDP self-loop on one end), not XRd — delete and re-create the link; see [link-udp-self-loop](../bugs/link-udp-self-loop.md) |
+| Two nodes can't ping, only one side ARPs | GNS3 link wiring bug (UDP self-loop on one end), not XRd — fixed (port-allocation race); on older builds delete and re-create the link; see [link-udp-self-loop](../bugs/link-udp-self-loop.md) |
 | Compute log: busybox coredump storm | fixed by the container-chown change; verify gns3-server is current |
 
 ## Notes
@@ -191,5 +191,6 @@ sequenceDiagram
 
 | Version | Date | Changes |
 |---------|------|---------|
+| 1.2 | 2026-08-14 | Link UDP self-loop root-caused to a port-allocation race and fixed — see [link-udp-self-loop](../bugs/link-udp-self-loop.md) (now marked Fixed). |
 | 1.1 | 2026-08-14 | Datapath validated end-to-end (XRd brings its own interfaces up; ARP/ICMP bidirectional). Add troubleshooting entry for the one-way-link symptom (GNS3 link UDP self-loop bug, see bugs/link-udp-self-loop.md). |
 | 1.0 | 2026-08-14 | Initial documentation of the XRd control-plane adaptation: vendor path requirement, shm/devices/extra_configs/udev-mask mechanisms, host-disturbance root causes, appliance recipe. |