The 60 s SIGTERM grace was hardcoded, unlike every other vendor knob
(GNS3_SHM_SIZE, GNS3_DEVICES, GNS3_MASK_UDEV, ...) which rides the
environment line. Parse GNS3_STOP_TIMEOUT=<seconds> (default 60,
clamped to 1-600, invalid values keep the default) and use it in
VendorDockerVM._terminate_container().
DockerVM.stop() terminated containers with an immediate SIGKILL — fine
for init.sh-based containers whose state is persisted beforehand, but a
systemd NOS (Cisco XRd, SR Linux) needs a graceful shutdown and treats
the abrupt kill as an unclean shutdown (exit 137 on every stop).
Extract the final termination into _terminate_container() and override
it in VendorDockerVM: POST /containers/{id}/stop?t=60 sends SIGTERM and
waits for systemd to stop services; Docker itself SIGKILLs the
container once the grace period expires, so no fallback is needed.
Docker's 304 (already stopped) is swallowed.
PortManager.get_free_udp_port had an unguarded find-then-add sequence.
A link allocates both ends concurrently (asyncio.gather in
UDPLink._prepare -> two POST /ports/udp) and FastAPI runs the sync
route handler in a threadpool, so both threads could probe the same
'free' port before either recorded it — handing lport == rport to both
ends. uBridge sets SO_REUSEADDR on UDP NIO sockets, so the double bind
succeeds silently and the kernel delivers everything to the last-bound
socket: one node starves, the other echoes to itself.
Make every TCP/UDP allocate/reserve/release path atomic with an RLock,
and rebuild _link_data in UDPLink._prepare so reset() commits the fresh
port pair instead of re-sending the stale, already-released one.
Regression tests: threaded barrier allocation never returns duplicates
(red on the old code, UDP and TCP); reset() leaves exactly one mirrored
NIO pair per side with lport != rport (red on the old code).
Add bugs/link-udp-self-loop.md: the intermittent one-way docker link
observed during XRd validation (one end's nio_udp rport pointing at its
own lport after a uBridge restart), with the evidence table, the uBridge
console capture diagnostics, the delete/recreate workaround, and the
narrowed suspects (batch port preallocation / link re-creation race).
Also note that a docker node's in-container ethN is a TAP device held by
uBridge (no veth host end exists).
Update the XRd feature doc: datapath validated end-to-end (XRd brings
its own interfaces up, ARP/ICMP bidirectional), add a troubleshooting
row pointing at the bug doc. Index the bug in docs/README.md.
New feature doc covering why XRd takes the vendor docker_exec/SKIP_INIT
path (init.sh wrapper crashes its glibc loader), the four generic
mechanisms added for it (GNS3_SHM_SIZE/GNS3_DEVICES HostConfig injection,
extra_configs file injection, GNS3_MASK_UDEV + udevadm null-binding,
host-readiness check), the three host-disturbance root causes isolated by
plain docker-run A/B/C testing, the appliance recipe with XRd-specific
gotchas (Mg0/RP0/CPU0/0, /xr-storage-shadow persistence, first-boot
semantics), and troubleshooting. Indexed in docs/README.md alongside the
docker-exec-console base doc.
Masking the udev systemd units stopped the daemon's coldplug (host audio
resets), but host USB devices still reconnected on every XRd start. A/B
testing with plain `docker run` isolated the trigger: XRd's own
xr_startup.sh calls udevadm directly (USB license-dongle probing, e.g.
`udevadm trigger --action=add --parent-match=<usb device>`), which
synthesizes uevents into the host kernel from the privileged container --
no udevd required.
GNS3_MASK_UDEV=1 now also binds /dev/null over the udevadm binary
(/bin, /sbin, /usr/bin). Verified with a plain-run experiment: with the
bind, host udev monitor shows zero usb/input/hid/sound events during XRd
boot (only normal docker veth traffic), and XRd itself boots to running
state -- it does not need udevadm under GNS3 (interfaces are pre-created
veths).
The vendor skip-init path (_setup_skip_init_volumes, _fix_permissions) runs
`/gns3/bin/busybox chown` inside the container via docker exec. busybox is
statically linked, and its chown dlopens NSS modules (libnss_*) from the
container; on NOS images whose glibc differs from the host's (e.g. Cisco
XRd) that mismatches and aborts with the glibc assertion
`_dl_call_libc_early_init: sym != NULL` (SIGABRT). The per-file chown loop
then crash-loops, and the resulting core-dump storm -- processed by the
host's systemd-coredump -- cascades into host device rescans, reconnecting
USB / resetting audio / corrupting the journal on every XRd start.
cp/chmod/find/stat don't touch NSS and work fine on busybox, so only chown
is affected. Prefer the container's own coreutils chown
(`command -v chown && chown ...`), falling back to busybox chown only when
the container ships no chown (minimal images, where the glibc matches and
busybox is safe).
A privileged systemd-based NOS container (Cisco XRd boots /usr/sbin/init)
runs systemd-udevd, which on startup coldplugs every device it can reach.
In privileged mode that includes the HOST's USB/input/audio/disk devices,
so every XRd start reconnects USB, mutes audio, and disrupts the host
journal -- highly disruptive on Linux desktops (caught in the act: the
container's udevd was even rescanning the host BTRFS root device).
XRd doesn't need udev (its interfaces are pre-created by GNS3 veth and
mapped via XR_INTERFACES). Add two opt-in env vars, consumed host-side at
container create time in the inherited DockerVM.create (so VendorDockerVM
nodes get it too):
GNS3_MASK_UDEV=1 -> bind /dev/null over the udevd unit, its two
activation sockets, and the coldplug/settle
trigger services
GNS3_MASK_SYSTEMD=u1,u2 -> bind /dev/null over arbitrary units in
/etc/systemd/system/ (comma/semicolon list)
Only injected when set, so ordinary nodes are unaffected.
The extra_configs schema field needs a column on the docker_templates table
to actually round-trip through the controller DB (the schema alone is
accepted but dropped by the SQLAlchemy model mapping). Add the JSON column
and an Alembic migration so existing databases get it on upgrade.
Add an `extra_configs` field (list of {target, content}) to the docker
node/template/appliance schemas. For each entry GNS3 writes `content` to a
file in the node working directory and bind-mounts it read-only at `target`
inside the container.
This lets a NOS appliance seed its startup config without rebuilding the
image: XRd points XR_FIRST_BOOT_CONFIG at an injected /firstboot.cfg, FRR at
/etc/frr/frr.conf, etc. The bind is a single-file mount applied at create
time, so it works for both the generic init.sh path and vendor nodes that
skip init.sh (console_type=docker_exec). Entries are only injected when
present, so ordinary nodes are unaffected.
The content can't go through `environment` (it is line-delimited, one var per
line), hence a dedicated field -- the same plumbing shape as extra_volumes.
Add a read-only _check_host_readiness() that runs once after the Docker
daemon connection is established. It reads /proc/sys inotify/file-max
limits and /proc/filesystems (for FUSE), and logs a warning with the exact
commands to fix when they are too low for heavy containers -- XRd wants
~4000 inotify instances per node against a stock default of 128.
The server runs unprivileged (only the setuid ubridge helper has root), so
it can only check, not set; the warning tells the admin exactly what to
raise once. Stays silent when the limits are already sufficient.
Heavy NOS containers (e.g. Cisco XRd) need /dev/shm larger than Docker's
64 MB default and host device nodes such as /dev/fuse. Add two opt-in
environment variables, consumed host-side and applied as native Docker
HostConfig keys at create time:
GNS3_SHM_SIZE (MB) -> HostConfig.ShmSize (bytes)
GNS3_DEVICES -> HostConfig.Devices in `docker run --device` syntax
(host[:container[:perm]]; Docker resolves major/minor
from the host node itself)
Native HostConfig (rather than remount/mknod inside init.sh) is used so this
works for vendor NOS nodes that skip init.sh (console_type=docker_exec) --
the path XRd must take, since GNS3's init.sh wrapper crashes XRd's glibc
loader. It applies whether or not init.sh runs, needs no schema/API/UI
change (reuses the `environment` field), and only takes effect when the vars
are set, so ordinary nodes keep default Docker behaviour.
GNS3_-prefixed user env vars stay dropped from the container environment
(only consumed here host-side), keeping GNS3-injected vars safe.
Document the SR Linux gns3a (35-adapter full chassis, matching
GNS3_INTERFACE_NAMES + custom_adapters), the three server-side schema fixes
needed for it to load (DockerConsoleType.docker_exec,
ApplianceV1_6.custom_adapters, extra_volumes docker-block passthrough), and
the symbol-theme behaviour that rewrites any :/symbols/-prefixed symbol to
the category default at load time (so router_cloud.svg cannot be used from
an appliance; use a custom symbol under symbols_path instead).
The top-level ApplianceV1_6 model declared first_port_name /
port_name_format / port_segment_size but not custom_adapters, so the
GET /appliances endpoint (response_model=schemas.Appliance) stripped
custom_adapters from the response — the frontend never saw per-adapter
port names even though the appliance file and the server-side template
conversion (appliance_to_template reads it from the raw dict) handled it.
Add custom_adapters: Optional[List[CustomAdapterItem]] to ApplianceV1_6
so the field round-trips through the API. (ApplianceV8 already models it
inside its TemplateSetting.)
The Docker appliance Pydantic model (DockerConsoleType) rejected
console_type='docker_exec', so an appliance file using the vendor NOS
docker_exec console could not be loaded (validation error at import).
Add docker_exec to the enum — it is already a valid ConsoleType
(schemas/common.py) and is handled by VendorDockerVM.
The reconnect-blank-screen bug: when sr_cli exited (quit / idle timeout /
crash) the while-true wrapper restarted it mid-session with no client
attached, so its startup CPR probe (\e[6n) went unanswered and the TUI
degraded/blocked. On reconnect lazy_started=True skipped recreation, so the
client saw a blank screen.
Fix: drop the while-true wrapper. Now when the CLI exits, the exec pty
closes (EOF), the broadcast task ends, and the next client connection
detects the dead upstream via _upstream_alive() and recreates the exec —
with a terminal attached, so CPR is answered. A live exec is reused
(just a Ctrl-L redraw).
_LazyExecTelnetServer is extracted from a closure to module level so the
reconnect/recreate logic is unit-testable. Add 9 tests covering
_upstream_alive states and the recreate-on-death / reuse-if-live /
close-half-dead-writer / no-while-true behaviors.
Full Docker suite (120) passes.
25 tests covering:
- Docker.create_node factory: selects VendorDockerVM iff console_type ==
docker_exec, DockerVM otherwise (including telnet/ssh/vnc/http/none/spice)
- GNS3_* env parsing: SKIP_INIT, INTERFACE_NAMES, CONSOLE_CMD (single and
multiline), defaults
- create(): init.sh skipped under GNS3_SKIP_INIT, prepended otherwise;
GNS3_MAX_ETHERNET follows the interface rename; /etc/network mount dropped
under SKIP_INIT (and host skeleton dir removed) but kept without it
- _add_ubridge_connection: move_to_ns targets the renamed interface
(mgmt0) or falls back to eth{N}
- start(): docker_exec console dispatch + SKIP_INIT volume bridge + permission
fix; without SKIP_INIT the vendor passes are skipped
- _fix_permissions: skips dead/missing containers (no restart), targets
/gns3volumes bind-mount paths
- _setup_skip_init_volumes: runs the docker exec bridge script
- _cleanup_console_resources: closes the exec pty writer
Full Docker suite (111) and compute suite (395) pass — the four hook
extractions in DockerVM introduce no regressions.
Override _mount_binds in VendorDockerVM: for GNS3_SKIP_INIT containers the
/etc/network volume (GNS3's own network config consumed by init.sh's ifup)
is dead weight — init.sh never runs and the NOS manages its own
interfaces. The override removes the bind, filters /etc/network out of
self._volumes (keeping GNS3_VOLUMES and the bridge/fix passes consistent)
and deletes the host-side skeleton directory created by the base class.
Without GNS3_SKIP_INIT the mount is kept, matching base behaviour.
_persistent_volumes() is removed — the mount override is now the single
filter point.
GNS3_SKIP_INIT containers never run init.sh, so /etc/network (GNS3's own
network config consumed by init.sh's ifup) has no consumer — the NOS
manages its own interfaces. VendorDockerVM._persistent_volumes() filters it
out for both _setup_skip_init_volumes and _fix_permissions, saving one
docker exec per pass. The shared _mount_binds is untouched, and without
GNS3_SKIP_INIT the full volume list is returned so behaviour matches the
base class.
Document why host-user ownership of volume files during runtime is
harmless for SR Linux (root processes, self-healing daemons like aaamgr
rewriting its managed files, ACL-based access) and the deviation from the
standard init.sh model, with the escape hatch of dropping the start-time
fix pass for strict-ownership NOS images.
Also document the boot-ordering caveat: the volume bridge comes up after
the NOS boots, so early boot reads overlay defaults — verify the
save/stop/start closed loop, and add troubleshooting entry #10 for config
present on the host but not applied after restart.
Document the aaamgr_local_user.json case: SR Linux's aaamgr daemon rewrites
the file during boot as the image's srlinux user (uid 1002) after the
start-time permission pass, leaving it unreadable until the stop-time pass.
Trace the warning to the file-browser API chain (Show in file manager ->
list_node_files -> magic.from_file) and note the impact is limited to the
file_type field.
The host-side pass could not work for unprivileged GNS3 processes: the
.gns3_perms marker is created root-owned by the container-side touch, and
chowning root-owned files from the host requires root.
Rewrite VendorDockerVM._fix_permissions to run the busybox
record/chmod/chown script inside the container (as root) on the
/gns3volumes bind-mount targets — they exist for the container's whole
lifetime and do not depend on the mount --bind bridge, so a container
restart can no longer make the fix hit the overlay copy. A
stopped/exited container is skipped (logged) instead of restarted; the
next start's pass fixes ownership.
Replace the container-side _fix_permissions for vendor NOS containers with a
host-side pass that walks the node's project directories directly (they are
the Docker bind-mount sources): records mode:uid:gid into .gns3_perms and
chowns to the GNS3 user. No docker exec, no container restart — the base
implementation restarts an exited container just to chown, and after the
restart the mount --bind bridge is gone so it would fix the overlay copy
instead of the host files.
The pass runs at start (after _setup_skip_init_volumes seeds and bridges the
volumes) so the controller can read project files while the node runs, and
again at stop for files written during runtime.
Update docker-exec-console.md: VendorDockerVM architecture, hook points,
class-selection factory, volume-persistence lifecycle, and new
troubleshooting entries.
Extract the docker_exec console and GNS3_* prototype knobs (SKIP_INIT,
INTERFACE_NAMES, CONSOLE_CMD) from DockerVM into a VendorDockerVM subclass.
DockerVM is restored to its 3.1 baseline plus four small extension hooks
(_prepare_init_and_interface_env, _start_console_server,
_get_container_ifname, _cleanup_console_resources) that are pure
refactorings with zero behaviour change for existing nodes.
VendorDockerVM additionally replicates init.sh's volume persistence
(bind-mount /gns3volumes over the in-container path) via docker exec for
containers that skip init.sh, so vendor NOS config (e.g. /etc/opt/srlinux)
survives node stop/start.
The Docker manager selects VendorDockerVM when console_type == docker_exec;
all other nodes keep using DockerVM unchanged.
create_docker_node() passes console, aux etc. to create_node() via
.get() so those keys remain in node_data. The setattr fallback loop
then re-applies them — if reserve_tcp_port returned a different port
in __init__, the setter fires an INFO log and performs a wasted
release→reserve round-trip.
Pop the 15 keys already consumed by create_node() before the loop so
it only handles truly extra keys.
The per-commit force_close=not _local optimisation reused TCP connections
for the loopback compute, but _session() is also used for the controller's
WebSocket heartbeat connection (_connect_notification -> ws_connect).
The different connector behaviour prevented the compute from receiving
pings, so no compute.updated events reached the WebUI and the compute
cache stayed empty.
create_batch_nios bound NIOs in a serial for-loop, so during project open
every builtin L2 node (ethernet_switch/hub/cloud/nat) started its uBridge
one at a time (~0.5s each for fork + AF_UNIX connect). Group entries by
node and bind in parallel with asyncio.gather — mirroring update_batch_nios
— so independent uBridge processes start concurrently. Within a node,
entries stay serial to respect the per-node uBridge command lock.
The reconcile pass in _ubridge_apply_markers walked the node-wide
_marker_filter_bridges map but compared against `desired`, which only
carries the markers of the NIO being updated. Updating any one link
therefore deleted every other link's markers (and their pcaps) on that
node — a regression from the add-only→reconcile switch. IOU's override
had the same flaw across its ports.
Guard the delete pass with the current bridge (base_node) / IOL location
(IOU) so only markers on the NIO being reconciled can be removed. Added
a regression test that fails without the guard.
The print(node_data.chassis, platform in DEFAULT_CHASSIS) at
dynamips_nodes.py:67 (added in 8ad7b3f6, 2022) emitted "None False"
to stdout on every Dynamips router creation. Pure debug leftover;
the very next line tests the same condition.
The batch marker-def fan-out (PR #2848) routed create/update/delete
marker_definition through memory_only + a batch PUT /nios/batch that
re-applies markers via _ubridge_apply_markers. But _ubridge_apply_markers
was strictly add-only: it skipped any (name, link_id) already in
_marker_filter_bridges, and reset_packet_filters preserves mark filters
(contract). So:
* delete_marker_definition left the deleted marker's filter alive in
uBridge (still matching / signalling / writing pcap) until node restart.
* update_marker_definition (bpf/tag/direction change) never reached
uBridge — the live filter kept the old expression until node restart.
Make _ubridge_apply_markers a real reconcile against the desired
nio.markers:
- installed but no longer desired → delete_packet_filter + unlink pcap
+ unregister
- desired with changed filter field → rebuild (delete + re-add)
- desired with only enabled changed → instant toggle (pcap preserved)
- desired and unchanged → skip
- desired and new → add
Track installed specs in a parallel _marker_specs dict so changes can be
detected. Both base_node and the IOU iol_bridge override are updated.
Added tests for the delete-removed and rebuild-changed-bpf paths.
MarkerManager now logs every 10s how many marker datagrams the UDP sink
processed and the current throughput rate (match/s), so operators can
tell at a glance whether the single sink keeps up with the aggregated
uBridge traffic. Error count is also logged.
High-frequency marker.matches shared the single project notification queue with topology events (node.*/link.*), causing head-of-line blocking. Add a separate marker channel: Notification.project_marker_queue/marker_emit, dispatch routes marker.* off the main project queue, plus a new WS /{project_id}/notifications/markers/ws endpoint. Fully migrated (the main project WS no longer carries marker.match); marker listeners are independent of project auto_close. Compute side unchanged.
1000+ uBridge processes share a single UDP marker.sink endpoint. The
default kernel receive buffer (~208 KB) holds ~1000 datagrams — a
traffic burst can overflow it before the event loop drains them. Grow
it to 8 MB via setsockopt(SO_RCVBUF) so the kernel absorbs bursts
without silent packet loss. UDP is unordered — buffer size does not
affect per-datagram latency, only burst-loss resilience.
Previously apply_defs_to_new_link ran during finalize (after
link._created=True), issuing one PUT /nio round-trip per link end for
each inherited marker — 5000+ HTTP round-trips even for a single def.
Move it into _prepare_link_from_topology (memory_only) so the inherited
markers are already in _link_data when _prepare() constructs the NIO
specs, and create_batch_nios carries them in the single batch dispatch.
Finalize no longer calls apply_defs_to_new_link. Interactive link
creation (dragging a cable in the UI) still goes through the per-link
create() → apply_defs_to_new_link path — a single link is fast.
