The SKIP_INIT volume bridge replicated init.sh's seed + mount --bind script
via docker exec *after* the container started. That copied the mechanism but
not the invariant that makes init.sh safe — the entrypoint position, which
guarantees the volume is in place before the application runs. The exec runs
concurrently with the NOS boot, so whether the NOS loaded its persisted
config or the overlay's factory copy was a timing race:
- single node stop/start on an idle system won it (exec ~1s, SR Linux reads
its startup config at ~2-4s) — the save/stop/start round-trip passed;
- a server restart + project reload lost it (concurrent node starts queue on
the Docker API, delaying the exec by seconds) — SR Linux booted factory
while the persisted config.json sat intact on the host;
- XRd was immune (systemd boots tens of seconds before XR touches
/xr-storage), which is why the race was never observed on it.
Replace the bridge entirely:
- new DockerVM._prepare_volumes hook (no-op in the base class) runs in
create() after the image is present, before the container is created;
VendorDockerVM overrides it to seed each volume's host directory from the
image (throwaway docker create container + docker cp -a, nothing
executes). The .gns3_perms marker gates the seeding: a volume that ever
started is never re-seeded, so saved configuration is never overwritten
with factory content (also the upgrade path for existing nodes).
- VendorDockerVM._mount_binds now binds the volumes directly at their real
in-container paths (/etc/opt/srlinux) instead of /gns3volumes aliases, so
the persisted config is visible to the NOS from the very first process.
- _setup_skip_init_volumes and its start() call are gone; the container-side
_fix_permissions targets the volume paths directly (the direct binds
exist for the whole container lifetime, unlike the old bridge).
The volume-list computation (validation + overlap de-duplication) moves
into DockerVM._persistent_volume_list so create-time seeding and _mount_binds
cannot drift apart.
A browser's terminal-size control frame (NAWS through the console telnet
server) can arrive while client_connected_hook is still creating the
exec; the resize is a no-op then, and the tall default applied after
creation would overwrite it, leaving the session at 511x10000 until the
user resizes.
Record sizes received before the exec exists and prefer them over the
tall default once creation finishes. The recorded size is cleared when
the last client disconnects, together with the restore-to-default.
The exec behind a docker_exec console is shared by every console client,
so a browser's terminal-size resize (WS control frames -> NAWS) also
changes the geometry concurrent netmiko sessions see. SR Linux doesn't
care (no pager, no hard wrapping), but CLIs that page on the PTY window
size (IOS-XR) would park at --More-- again the moment a browser is
connected.
Split the client-driven NAWS path (_on_naws) from the internal resize
(_resize_exec): GNS3_CONSOLE_RESIZE=0 makes the console ignore client
resizes entirely and keep the tall 511x10000 no-paging default, while
the creation-time default and the restore-on-last-disconnect still go
through the internal path. XRd appliance templates should set it.
The docker_exec console defaults its exec PTY to 511x10000 (the no-NAWS
default that keeps the IOS-XR pager quiet for netmiko). A CPR-answering
client (xterm.js) on top of that tall canvas makes prompt_toolkit-based
CLIs (SR Linux sr_cli) re-emit their accumulated output on every
incremental render: ~145 KB instead of ~60 KB per command, visible in
the WebUI as full-screen clear/redraw flicker.
Let WebSocket console clients propagate their real terminal geometry:
binary frames {"cols": N, "rows": M} alongside text frames carrying
terminal data. The controller forwards binary frames (previously only
text was forwarded), and the compute side turns them into a NAWS
subnegotiation for telnet-based consoles (docker_exec included) or an
asyncssh pty size change for SSH consoles.
The docker_exec console restores the tall 511x10000 default when its
last client disconnects, so a later non-NAWS client (netmiko, bare
telnet) connecting to the still-live exec doesn't inherit a browser
geometry and hit PTY-window paging again.
The docker_exec console resized its exec PTY to 80x24 until a client
sent NAWS. CLIs that page on the PTY window size instead of the
terminal length (the IOS-XR pager) therefore parked long output at
--More-- for clients that never negotiate NAWS — netmiko, bare telnet —
making copilot device commands time out on XRd.
Default the exec to 511x10000 instead (511 matches netmiko's own
'terminal width 511' convention): no paging and no hard wrapping for
non-NAWS clients, while real NAWS clients keep resizing to their actual
geometry as before.
Also updates the project memory record with the confirmed root cause
and the fix.
The 600 s clamp was unreachable in practice: the controller's stop
request times out at 240 s (controller/node.py) and the Docker stop
query gets the value +30 s as its HTTP timeout, so anything above 210
would abort upstream first and surface an error while the stop keeps
running server-side. Cap at the derived ceiling and document the chain
in the clamp and the docstring.
Nine fixes from a review of the docker-shm-devices diff:
* GNS3_STOP_TIMEOUT >300 s aborted at the manager's default HTTP timeout
before Docker finished the stop — the stop query now gets a timeout
with a margin over the grace period.
* Overlapping bind targets (GNS3_MASK_UDEV + GNS3_MASK_SYSTEMD on the
same unit, a unit named twice, an extra_configs target equal to a
masked unit) made Docker reject the create with 'Duplicate mount
point' — Mounts are deduplicated by target.
* ExtraConfig.target now carries a pydantic validator (absolute file
path, no '..'), so bad targets 422 at template-save time instead of
failing at node-create time after a multi-GB image pull; directory
forms ('/', '/etc/') are also rejected by the runtime guard instead
of raising IsADirectoryError (raw 500).
* _check_host_readiness skipped every remaining check when one
/proc/sys key was unreadable (mid-loop return) — now continues.
* The base-class GNS3_* env parser strips trailing commas like the
vendor parser, so 'GNS3_MASK_UDEV=1,' composed from a list still
activates.
* Vendor env knobs are re-parsed on every create(), so a PUT to the
node's environment takes effect on the next (re)create.
* The graceful SIGTERM stop is now limited to the explicit user stop
route; delete/update/close/crash-cleanup keep the immediate kill
(those paths force-delete or recreate the container right after).
* An extra_configs target beneath a persisted volume is shadowed by the
volume bind — warn at create time.
The 60 s SIGTERM grace was hardcoded, unlike every other vendor knob
(GNS3_SHM_SIZE, GNS3_DEVICES, GNS3_MASK_UDEV, ...) which rides the
environment line. Parse GNS3_STOP_TIMEOUT=<seconds> (default 60,
clamped to 1-600, invalid values keep the default) and use it in
VendorDockerVM._terminate_container().
DockerVM.stop() terminated containers with an immediate SIGKILL — fine
for init.sh-based containers whose state is persisted beforehand, but a
systemd NOS (Cisco XRd, SR Linux) needs a graceful shutdown and treats
the abrupt kill as an unclean shutdown (exit 137 on every stop).
Extract the final termination into _terminate_container() and override
it in VendorDockerVM: POST /containers/{id}/stop?t=60 sends SIGTERM and
waits for systemd to stop services; Docker itself SIGKILLs the
container once the grace period expires, so no fallback is needed.
Docker's 304 (already stopped) is swallowed.
The vendor skip-init path (_setup_skip_init_volumes, _fix_permissions) runs
`/gns3/bin/busybox chown` inside the container via docker exec. busybox is
statically linked, and its chown dlopens NSS modules (libnss_*) from the
container; on NOS images whose glibc differs from the host's (e.g. Cisco
XRd) that mismatches and aborts with the glibc assertion
`_dl_call_libc_early_init: sym != NULL` (SIGABRT). The per-file chown loop
then crash-loops, and the resulting core-dump storm -- processed by the
host's systemd-coredump -- cascades into host device rescans, reconnecting
USB / resetting audio / corrupting the journal on every XRd start.
cp/chmod/find/stat don't touch NSS and work fine on busybox, so only chown
is affected. Prefer the container's own coreutils chown
(`command -v chown && chown ...`), falling back to busybox chown only when
the container ships no chown (minimal images, where the glibc matches and
busybox is safe).
The reconnect-blank-screen bug: when sr_cli exited (quit / idle timeout /
crash) the while-true wrapper restarted it mid-session with no client
attached, so its startup CPR probe (\e[6n) went unanswered and the TUI
degraded/blocked. On reconnect lazy_started=True skipped recreation, so the
client saw a blank screen.
Fix: drop the while-true wrapper. Now when the CLI exits, the exec pty
closes (EOF), the broadcast task ends, and the next client connection
detects the dead upstream via _upstream_alive() and recreates the exec —
with a terminal attached, so CPR is answered. A live exec is reused
(just a Ctrl-L redraw).
_LazyExecTelnetServer is extracted from a closure to module level so the
reconnect/recreate logic is unit-testable. Add 9 tests covering
_upstream_alive states and the recreate-on-death / reuse-if-live /
close-half-dead-writer / no-while-true behaviors.
Full Docker suite (120) passes.
Override _mount_binds in VendorDockerVM: for GNS3_SKIP_INIT containers the
/etc/network volume (GNS3's own network config consumed by init.sh's ifup)
is dead weight — init.sh never runs and the NOS manages its own
interfaces. The override removes the bind, filters /etc/network out of
self._volumes (keeping GNS3_VOLUMES and the bridge/fix passes consistent)
and deletes the host-side skeleton directory created by the base class.
Without GNS3_SKIP_INIT the mount is kept, matching base behaviour.
_persistent_volumes() is removed — the mount override is now the single
filter point.
GNS3_SKIP_INIT containers never run init.sh, so /etc/network (GNS3's own
network config consumed by init.sh's ifup) has no consumer — the NOS
manages its own interfaces. VendorDockerVM._persistent_volumes() filters it
out for both _setup_skip_init_volumes and _fix_permissions, saving one
docker exec per pass. The shared _mount_binds is untouched, and without
GNS3_SKIP_INIT the full volume list is returned so behaviour matches the
base class.
The host-side pass could not work for unprivileged GNS3 processes: the
.gns3_perms marker is created root-owned by the container-side touch, and
chowning root-owned files from the host requires root.
Rewrite VendorDockerVM._fix_permissions to run the busybox
record/chmod/chown script inside the container (as root) on the
/gns3volumes bind-mount targets — they exist for the container's whole
lifetime and do not depend on the mount --bind bridge, so a container
restart can no longer make the fix hit the overlay copy. A
stopped/exited container is skipped (logged) instead of restarted; the
next start's pass fixes ownership.
Replace the container-side _fix_permissions for vendor NOS containers with a
host-side pass that walks the node's project directories directly (they are
the Docker bind-mount sources): records mode:uid:gid into .gns3_perms and
chowns to the GNS3 user. No docker exec, no container restart — the base
implementation restarts an exited container just to chown, and after the
restart the mount --bind bridge is gone so it would fix the overlay copy
instead of the host files.
The pass runs at start (after _setup_skip_init_volumes seeds and bridges the
volumes) so the controller can read project files while the node runs, and
again at stop for files written during runtime.
Update docker-exec-console.md: VendorDockerVM architecture, hook points,
class-selection factory, volume-persistence lifecycle, and new
troubleshooting entries.
Extract the docker_exec console and GNS3_* prototype knobs (SKIP_INIT,
INTERFACE_NAMES, CONSOLE_CMD) from DockerVM into a VendorDockerVM subclass.
DockerVM is restored to its 3.1 baseline plus four small extension hooks
(_prepare_init_and_interface_env, _start_console_server,
_get_container_ifname, _cleanup_console_resources) that are pure
refactorings with zero behaviour change for existing nodes.
VendorDockerVM additionally replicates init.sh's volume persistence
(bind-mount /gns3volumes over the in-container path) via docker exec for
containers that skip init.sh, so vendor NOS config (e.g. /etc/opt/srlinux)
survives node stop/start.
The Docker manager selects VendorDockerVM when console_type == docker_exec;
all other nodes keep using DockerVM unchanged.
