The reconnect-blank-screen bug: when sr_cli exited (quit / idle timeout /
crash) the while-true wrapper restarted it mid-session with no client
attached, so its startup CPR probe (\e[6n) went unanswered and the TUI
degraded/blocked. On reconnect lazy_started=True skipped recreation, so the
client saw a blank screen.
Fix: drop the while-true wrapper. Now when the CLI exits, the exec pty
closes (EOF), the broadcast task ends, and the next client connection
detects the dead upstream via _upstream_alive() and recreates the exec —
with a terminal attached, so CPR is answered. A live exec is reused
(just a Ctrl-L redraw).
_LazyExecTelnetServer is extracted from a closure to module level so the
reconnect/recreate logic is unit-testable. Add 9 tests covering
_upstream_alive states and the recreate-on-death / reuse-if-live /
close-half-dead-writer / no-while-true behaviors.
Full Docker suite (120) passes.
Override _mount_binds in VendorDockerVM: for GNS3_SKIP_INIT containers the
/etc/network volume (GNS3's own network config consumed by init.sh's ifup)
is dead weight — init.sh never runs and the NOS manages its own
interfaces. The override removes the bind, filters /etc/network out of
self._volumes (keeping GNS3_VOLUMES and the bridge/fix passes consistent)
and deletes the host-side skeleton directory created by the base class.
Without GNS3_SKIP_INIT the mount is kept, matching base behaviour.
_persistent_volumes() is removed — the mount override is now the single
filter point.
GNS3_SKIP_INIT containers never run init.sh, so /etc/network (GNS3's own
network config consumed by init.sh's ifup) has no consumer — the NOS
manages its own interfaces. VendorDockerVM._persistent_volumes() filters it
out for both _setup_skip_init_volumes and _fix_permissions, saving one
docker exec per pass. The shared _mount_binds is untouched, and without
GNS3_SKIP_INIT the full volume list is returned so behaviour matches the
base class.
The host-side pass could not work for unprivileged GNS3 processes: the
.gns3_perms marker is created root-owned by the container-side touch, and
chowning root-owned files from the host requires root.
Rewrite VendorDockerVM._fix_permissions to run the busybox
record/chmod/chown script inside the container (as root) on the
/gns3volumes bind-mount targets — they exist for the container's whole
lifetime and do not depend on the mount --bind bridge, so a container
restart can no longer make the fix hit the overlay copy. A
stopped/exited container is skipped (logged) instead of restarted; the
next start's pass fixes ownership.
Replace the container-side _fix_permissions for vendor NOS containers with a
host-side pass that walks the node's project directories directly (they are
the Docker bind-mount sources): records mode:uid:gid into .gns3_perms and
chowns to the GNS3 user. No docker exec, no container restart — the base
implementation restarts an exited container just to chown, and after the
restart the mount --bind bridge is gone so it would fix the overlay copy
instead of the host files.
The pass runs at start (after _setup_skip_init_volumes seeds and bridges the
volumes) so the controller can read project files while the node runs, and
again at stop for files written during runtime.
Update docker-exec-console.md: VendorDockerVM architecture, hook points,
class-selection factory, volume-persistence lifecycle, and new
troubleshooting entries.
Extract the docker_exec console and GNS3_* prototype knobs (SKIP_INIT,
INTERFACE_NAMES, CONSOLE_CMD) from DockerVM into a VendorDockerVM subclass.
DockerVM is restored to its 3.1 baseline plus four small extension hooks
(_prepare_init_and_interface_env, _start_console_server,
_get_container_ifname, _cleanup_console_resources) that are pure
refactorings with zero behaviour change for existing nodes.
VendorDockerVM additionally replicates init.sh's volume persistence
(bind-mount /gns3volumes over the in-container path) via docker exec for
containers that skip init.sh, so vendor NOS config (e.g. /etc/opt/srlinux)
survives node stop/start.
The Docker manager selects VendorDockerVM when console_type == docker_exec;
all other nodes keep using DockerVM unchanged.
The _connect_nio thread-pool optimisation (send_batch_sync) targeted
node-start performance, but start_all already runs at concurrency=3
(by design, to avoid overwhelming the host). It also introduced a
Python 3.13 incompatibility (trsock.setblocking forbidden) that
prevented docker nodes from starting. Since node-start is not the
target of this branch (project-open link creation is), revert to the
simple per-command async _ubridge_send.
The project-open batch NIO dispatch (create_batch_nios) is unaffected —
it never called _connect_nio (nodes aren't started during open).
At 1000+ nodes the per-node INFO lines flood the log during open /
start-all / stop-all: MAC changed, adapters changed, created, started,
console listen, fix ownership, stopped, paused, removed, adapter created,
NIO removed, capture start/stop, CPU/memory limits, mount resources.
Demote all of these routine per-node/per-adapter lines to DEBUG. Keep
INFO only for genuinely rare/important events: image pull (missing image)
and stale-container cleanup. Warnings unchanged.
- Drop the diagnostic stage-timing logs added during link-create perf
work (nodes / preallocate / prepare / dispatch) now that bottlenecks
are resolved and verified.
- Lower the per-NIO 'added to adapter' log in docker_vm from INFO to
DEBUG — at 5000+ NIOs per project open it floods the log at INFO.
Replace the default asyncio executor (capped at ~32 threads) with a
dedicated ThreadPoolExecutor sized for large-topology parallelism.
When 500 nodes each call _connect_nio, up to 500 OS threads can now
send blocking ubridge commands in parallel — no longer serialised by
either the event loop or a small thread pool.
- ubridge_hypervisor: module-level _ubridge_sync_pool (max_workers=500)
- docker_vm._connect_nio: dispatches to the dedicated pool instead of
the default executor
Replace the 3-5 sequential await _ubridge_send calls in _connect_nio
with a single run_in_executor batch. The batch holds the node-level
asyncio Lock to prevent interleaving with async sends, then uses the
hypervisor's new send_batch_sync method which does blocking socket
sendall/recv inside the thread pool. Different nodes' batches now
run in true OS-thread parallelism rather than serialising through
the asyncio event loop between every command.
- ubridge_hypervisor.send_batch_sync: blocking batch send using
the underlying socket from the asyncio transport, protected by
threading.Lock.
- _connect_nio: builds command list (add_nio_udp, start_capture,
bridge start, reset_packet_filters, add_packet_filter) and
dispatches to the default executor.
Temporary diagnostic instrumentation to measure the wall-clock time of
each ubridge command during NIO addition (add_nio_udp, bridge start,
filters, markers). The logs will reveal whether the 12-NIO/s
throughput stems from ubridge command latency itself or from lock
contention / HTTP overhead outside _connect_nio.
Docker node stop took ~5s every time. The stop API grace period
(params t=5, unchanged since 2015) was always exhausted: the business
process (often an interactive shell) ignores SIGTERM, and GNS3 doesn't
depend on graceful shutdown — _fix_permissions and /gns3volumes already
persist container state before stop() is called.
Use POST /containers/{id}/kill (SIGKILL, zero delay) instead of stop.
The 409 (container already stopped) replaces the previous 304 handling
for the race where the container exits between the state check and the call.
t=5 traced to commit 33edbefa3 (2015-10-14) "Docker cleanup and
improvements" — introduced with no recorded rationale.
The condition 'state != "stopped" or state != "exited"' is a tautology,
so the state check was a no-op and a stop request was sent even for a
container that had already exited.
_get_container_state() never returns "stopped" (only "running",
"paused" or "exited"), so the intended negation of the condition used in
_fix_permissions() requires 'and', not 'or' (De Morgan's law).
Added a regression test asserting no stop query is issued for an
already-exited container.
Docker's _connect_nio and adapter_update_nio_binding applied packet
filters but never called _ubridge_apply_markers, so markers silently
did nothing on Docker links despite docker being in the allowlist.
Add the missing calls (same pattern as the IOU fix).
Also narrow _MARKER_CAPABLE_TYPES to the four types that actually
implement marker support — vpcs, qemu, docker, iou — removing
dynamips, virtualbox, vmware, and cloud which have no marker pathway
and would silently fail when selected as the capture side.
Markers now follow exactly the same apply pattern as packet filters:
state lives in Link._markers, application goes through NIO
(update() -> PUT /nio -> _ubridge_apply_markers). The former
immediate-apply REST endpoints (/markers/start, /markers/stop on
the compute side) and the per-node start_marker/stop_marker methods
are removed — they were a legacy of the original capture-inspired
design and have been superseded by the NIO flow.
Changes:
- controller/udp_link: start_marker/stop_marker/update_marker now
set _markers state + call self.update() (mirrors update_filters).
Removed _marker_capture_nodes runtime dict and its helpers.
- controller/project: _create_link_from_topology_data restores
_markers directly from persisted data (with BPF validation,
like filter reload). No long calls start_marker during load.
- compute: _ubridge_apply_markers swallows BPF compile errors
(warn+skip), matching _ubridge_apply_filters behaviour so a
single bad expression cannot break link creation / node restart.
- Removed: /markers/start,stop endpoints (6 handlers across
vpcs/qemu/docker route files), node start_marker/stop_marker
methods (3 VM files), _ubridge_delete_marker_filter,
_marker_capture_nodes, MarkerDelete schema.
Net: ~280 lines of dead code removed; marker and packet filter now
share a single, unified apply path via the NIO.
When a Docker container with the same name already exists (e.g., from a
previous crashed GNS3 session), Docker returns a 409 Conflict error
when trying to create a new container with that name. This causes the
project open operation to fail.
This fix adds automatic cleanup of stale containers when encountering
a name conflict:
- Added DockerHttp409Error exception class
- Updated http_query to detect 409 status codes
- Modified create() to remove conflicting containers and retry
Fixes the issue where opening a project fails with:
"Docker has returned an error: 409 Conflict. The container name
'/GNS3.xxx' is already in use by container 'xxx'"
Rootful Docker recreates volume mount points as root on start,
preventing the GNS3 process from writing files into node directories
while the container is running. self._fix_permissions() would resolve
this but is currently only called at container stop time.
- _fix_permissions: capture stderr, check returncode, only set
_permissions_fixed on success instead of silently marking as fixed
- list_node_files: wrap os.scandir in try-except to handle
PermissionError gracefully
When updating project variables while Docker containers are running, the
system now properly handles both dictionary-format variables and Pydantic
Variable objects. This prevents AttributeError when containers are recreated
after variable updates.
Changes:
- Modified DockerVM.create() to detect and handle Pydantic Variable objects
- Updated _format_env() method to support both variable formats
- Maintains backward compatibility with existing dictionary format
Fixes error: AttributeError: 'Variable' object has no attribute 'get'
When closing a Docker node, if container deletion fails, the error
is silently ignored. This can lead to stale containers remaining on
the system and causing 409 conflicts when reopening projects.
Changes:
- Distinguish between 404 (container already removed, normal) and
other DockerError (deletion failed, needs attention)
- Log warning when deletion fails with error details
- Add comment explaining stale containers will be cleaned up on
project open (via automatic 409 conflict resolution)
This improves observability without blocking project close operations.
The root cause of stale containers can now be diagnosed from logs.
Fixes#2708
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a hostname validation fails, the error message now includes
the allowed character set to help users provide valid names.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When a Docker container with the same name already exists (e.g., from a
previous crashed GNS3 session), Docker returns a 409 Conflict error
when trying to create a new container with that name. This causes the
project open operation to fail.
This fix adds automatic cleanup of stale containers when encountering
a name conflict:
- Added DockerHttp409Error exception class
- Updated http_query to detect 409 status codes
- Modified create() to remove conflicting containers and retry
Fixes the issue where opening a project fails with:
"Docker has returned an error: 409 Conflict. The container name
'/GNS3.xxx' is already in use by container 'xxx'"
On musl-based systems (Alpine), ldd returns exit code 0 for static
binaries, unlike glibc which returns 1. This causes install_busybox()
to reject all busybox binaries as "dynamically linked" on Alpine.
Fix by also accepting binaries whose executable name contains "static"
(i.e. busybox-static, busybox.static), which are the first two
candidates checked by the function. The generic "busybox" fallback
still relies on the ldd return code check.
Most VNC clients use the Desktop name in their window title.
Currently this defaults to user@host which means all docker vnc
connections have identical window names of e.g.
SSVNC: gns3@gns3server
This uses the node name instead.
