The peer end of an Ethernet switch fast-path link is a NIOBridge with no
bridge of its own (the switch absorbed its anchor into the switch bridge).
Its update reconcile still ran the impairment pass with the empty filter
set the controller routes to that end — and an empty set is a plain
`tc reset` on the shared anchor, silently wiping the netem/cls_bpf/eBPF
the switch had just applied. Any link update did it (filters, markers,
suspend, capture start/stop), and the outcome depended on the link's
endpoint order: UdpLink.update() PUTs node1 then node2, so with the
switch first its apply ran first and the peer's reset ran last (API
green, data plane unimpaired), while with the peer first the reset ran
first and the filter survived. That order dependence is also why the
shipped e2e never caught it — it only created peer-first links.
Ownership is now explicit: only the end that owns the anchor's tc state
applies the impairment reconcile (_owns_anchor_impairments — a NIOBridge
with `bridge is None` is a passive carrier, the same idiom the
bridge-delete path already used). Markers, capture and the suspend
carrier still apply on both ends: markers reconcile per process, so
neither end can strip the other's, and the carrier converges by
agreement.
Red/green: two new docker unit tests fail on the unfixed tree with
`tc reset "<anchor>"` and pass after; the switch e2e also creates one
link switch-first and asserts the netem qdisc and the RTT hold under
that order (failed with the wipe, passes with the fix). Full unit suite
2110 passed; the switch e2e file passed end to end (378s).
IOL runner containers (IOLDockerVM, GNS3_IOL_RUNNER) had no kernel
datapath: their guest leg is the runner's AF_UNIX socket pairs, so every
link rode the uBridge relay. Their Ethernet bays now own persistent TAP
anchors (gx names, the iol_docker key of the shared naming contract —
born at node start like IOU's, before any link, so the switch fast
path's deferred join can never miss them), and the per-port bridge's
topology leg swaps between UDP (relay) and the anchor (kernel link)
through the frozen stop -> delete -> add -> start contract — the unix
binding never re-binds across a swap.
- iol_docker_vm: KernelDatapathMixin hooks, the anchor lifecycle
(_prepare_tap_datapath at _start_ubridge: probe ubridge_bridge_tap +
ubridge_tap, sweep leftover, tap create born down), the kernel attach
(ensure-then-add — tap create is strictly create-only, IFF_TUN_EXCL,
so the ensure checks /sys/class/net first: an unconditional create
answers EBUSY on the device the start loop just made), the release
(port bridge stop + delete_nio_tap before the shared teardown) and the
stop order the fd holders dictate (port bridges first, then tap
delete, then the per-link bridges — all before the hypervisor stops).
- vendor_docker_vm: the port-bridge creation factored into
_ensure_unix_port_bridge (idempotent on the registered name), keeping
the NIOBridge rejection for the generic unix-socket containers.
- docker_vm: the three _kernel_veths lookups go through the
_kernel_host_ifc hook — behavior-identical for DockerVM, and it is
what lets the subclass route without overriding the binding methods.
- kernel_anchor: the iol_docker key (gx) and kernel_anchor_type, the
controller-side mapping for naming an IOL runner's anchor on the
switch fast path (the marker check stays inline: this module sits
below everything and must not import the controller package).
uBridge delivered `link l2only` (e2e3155) and the brctl delete refusal
(83a84fb); verified from the server side -- anchors and per-link bridges
report addrgenmode none, carry no addresses, and an idle anchor is silent
once the one-shot membership burst has settled.
That burst is the one thing the command cannot remove, so it is now recorded
where the acceptance criteria live, with the measurement: enslaving a port
makes the kernel announce the bridge's multicast memberships (90 B MLDv2 at
+0.00 s, 54 B IGMPv3 at +0.09 s, 90 B MLDv2 at +0.73 s, then silence; no
addresses involved, and IPv4 has no per-device IGMP switch). Decision:
accepted and documented, not suppressed -- §E.2 measures idle silence after a
settle window instead.
Pins the other delivered behaviour with a test: a bridge now survives a
single-sided node stop, so brctl create answers EEXIST on the common path
rather than only in a create race -- the attach must not short-circuit there,
because the restarted side has a brand-new anchor whose filters and markers
still have to be re-applied.
Move the datapath-agnostic half of DockerKernelDatapathMixin into a new
KernelDatapathMixin (compute/kernel_datapath.py) so a second node type can
reuse it verbatim: the three marker primitives, the tc netem / cls_bpf / eBPF
appliers, the uBridge tc-capability probe, kernel marker teardown, link
detach, the per-link bridge cleanup and the adapter carrier -- plus
_kernel_attach(), lifted from DockerVM._connect_nio's kernel branch.
The mixin talks about "the anchor" (the host-side interface an adapter owns in
the root namespace) instead of "the veth host end": a node type provides
_kernel_host_ifc(), _kernel_anchors() and, when needed, _relay_carrier_bridge()
and _kernel_error(). Docker keeps only the veth lifecycle and those hooks;
behaviour is unchanged.
The C901 baseline entry moves with _netem_command_parts (the docker module is
no longer over the limit) and the bpf-compile-failure test patches the logger
of the module that now owns the code -- net zero new ignores.
Consume uBridge's ebpf_modes contract (feature/tc-window 40c36ae): a
stateful mode is usable iff ebpf=1 and its token is listed; builds
predating the field keep their shipped modes (nth/quota/flow), so the
pre-correction window semantics can never be installed silently.
- gns3server/utils/tc_capabilities.py: shared reply parser, mode
policy and filter-type mappings (compute and controller gate on the
same rules)
- gns3server/compute/ubridge/tc_probe.py: standalone probe spawning a
throwaway uBridge as the server's own user, cached by binary
identity (path/mtime/size - the version string cannot distinguish
rebuilds); failures cache as unknown
- GET /v3/compute/capabilities gains ubridge_tc {netem, ebpf,
ebpf_modes, cbpf}; the controller forwards it (Compute.capabilities)
and available_filters hides kernel-datapath filter types an endpoint
compute cannot run (both ends apply their own filters, so one
incapable compute hides the type); computes without a report keep
the full list and apply-time 409 stays the guard
- docker: _ubridge_apply_ebpf_drops gates per mode token; the cleanup
path only sends "off" for modes the build declares
- split DockerVM's kernel-datapath machinery (veth lifecycle, carrier,
marker primitives, tc filter translation, kernel NIO teardown) into
compute/docker/docker_kernel_datapath.py (DockerKernelDatapathMixin)
The bridge teardown only hung off the link-removal path, but deleting a
node or closing a project never issues a per-link delete — the veths
disappeared with their containers and left the bridge behind as an empty
orphan (reproduced: create a kernel link, DELETE the project, the
gns3{link_id} bridge survives). `_remove_kernel_veths` now deletes the
node's kernel-NIO bridges alongside the veth host ends: both endpoints do
it, the peer's still-enslaved port answers EBUSY (suppressed, last one
wins) exactly like link deletion, a relay NIO's uBridge bridge is left
alone (it dies with the process), and a node restart rebuilds the bridge
from the NIO in _connect_nio.
Found while cleaning up the window_drop e2e: seven such bridges had
accumulated on this host. Re-verified live: the bridge is gone after the
project delete, and the window_drop e2e (which includes a node restart
with the kernel link attached) still passes 15/15.
uBridge's window mode gained the semantics the B.2 spec asked for —
`tc window_drop <if> <start> <outage> <pct> [<period> [<jitter>]]`: a
single outage that traffic passes before *and* after, recurring flaps
with period >= outage, and a per-cycle jitter — so the kernel-only
`window_drop` filter type is now exposed:
- validation mirrors the tc grammar (3-5 params, ranges, period >=
outage, jitter cap) and adds window_drop to KERNEL_ONLY_FILTERS;
filter_inactive_filters keeps start=0 — unlike every other filter a
zero first parameter means "the outage starts now", not "disabled".
- `_ubridge_apply_ebpf_drops` sends `tc window_drop` (positional
pass-through) and turns the mode off explicitly when absent, under the
same ebpf=1 gate.
- FILTERS entry (Start/Outage/Chance/Period/Jitter) + available_filters.
The schedule is relative to the apply: reconcile re-sets the classifier
on every apply, so any filter change, node restart or link reset restarts
the window.
Verified on a two-container kernel link, 15/15: a single outage
[1000, 1500, 100] arrives per-seq as [0..4, 13..19] (passing before the
window, dropped inside, passing after); recurring 1000/2000 ms = 52 %
round trip (duty cycle); period == outage = permanent outage; period <
outage = 409 before uBridge; jittered flap keeps the ~50 % duty cycle;
30 % chance on a permanent window = 53 % round trip against the
per-direction theory 1-0.7^2 = 51 %; coexist with netem; node restart
restores; available_filters kernel vs relay (409 + hidden); teardown
clean. P6b re-run as a regression on the same uBridge build: 13/13.
Worth recording for the next rebuild: the first delivered window program
loaded as root but reported ebpf=0 through gns3-server — the verifier's
non-root path loses the loop counter's bound, treats the catch-up loop as
unbounded and rejects the *whole* program (E2BIG, 8193 pending states),
which also took frequency_drop and quota down, with ebpf=0 as the only
symptom. Smaller trip caps did not help; the loop had to go (uBridge
572ca0c), on top of a Makefile header-dependency fix (53e34bb) that had
been shipping a stale program object.
Consume the eBPF impairment classifier delivered in uBridge
feature/tc-precision: frequency_drop becomes the exact every-Nth mode
(tc nth_drop; -1 = drop everything maps to every 1st) and the new
kernel-only quota type is the byte-cap mode (tc quota_drop, 100 = hard
cutoff). KERNEL_UNSUPPORTED_FILTERS is gone entirely — kernel/relay
eligibility is now purely topological. Both modes gate on ebpf=1 from
the per-process tc capabilities probe (error names the setcap fix); the
cleanup path (modes off) only talks to the eBPF commands when the probe
reported them. Reconcile is a plain re-set: the netem reset-before-set
already detaches the program and clears uBridge's per-interface
registry, so a mode set after it is a fresh load+attach.
Documented semantic difference: the relay counts both directions
through one bridge counter, the kernel counts per direction — N=3
round-trip loss is 1-(2/3)^2 = 55.6% on kernel vs ~1/N on relay
(measured 57%). window_drop is NOT exposed: the delivered
implementation advances the window with no gap, making 'outside the
window packets pass' unreachable (needs outage/period fields —
recorded in the spec); flow_drop awaits a parameter-shape decision.
Also fixes a pre-existing restart bug the e2e exposed: _bridges (relay
bridge names) was never cleared on stop, so restarting a node with a
relay link skipped 'bridge create' and failed on add_nio_udp.
e2e (isolated instance): 13/13 — pref-1 tc_impair classifier on both
veths, 57% round-trip for N=3, -1 = exact 100%, quota hard cutoff,
delay coexistence, clean teardown, suspend/resume, node restart,
relay-leg frequency_drop regression through an Ethernet switch. Full
suite: 1925 passed.
Surface the netem extensions delivered in uBridge feature/tc-netem-ext
as GNS3 filter types: rate (tc-style bandwidth string), reorder (needs
delay), gemodel (Gilbert-Elliot bursty loss, exclusive with
packet_loss), duplicate, seed and limit, plus the delay jitter
distribution and the loss/dup correlation parameters. All are
kernel-datapath only — the relay's packet-filter registry has no
equivalent, so available_filters hides them on relay links, setting one
on a created relay link returns 409, and the project-load path drops
them with a warning (update_filters runs before the datapath is
decided, so it cannot reject there). Validation mirrors the tc grammar:
reorder requires delay, gemodel and packet_loss are mutually exclusive,
a distribution requires jitter > 0, rate is integer+unit capped at
100gbit.
The translation in DockerVM._netem_command_parts emits the frozen
grammar order and gates extension keywords on the 'tc capabilities'
netem token list (probed once per uBridge process; a plain
delay/loss/corrupt/dup filter never probes, keeping old uBridge builds
untouched; the correl suffix has no token and gates on 'rate' as the
extension-build marker).
Reconcile became reset-before-set: the kernel's netem change merges
optional attributes (rate, correlation, reorder, corrupt, gemodel,
distribution, seed keep their previous value when the new message omits
them), so removing a parameter needs an explicit 'tc reset' — the
bpf_drop filters are re-added right after in the same flow, so the
clsact churn is invisible. The capability check runs before the reset
so a failed update leaves the previous qdisc in place.
e2e (isolated instance, kernel link): 24/24 — rate 512kbit adds exactly
2x the per-packet serialization to the RTT (45.5 ms for 1400-byte
pings), paretonormal jitter + rate-cleared leak check, reorder gap 5,
gemodel 50% bursty loss, duplicate, loss 30% correl 50%, seed 42 +
limit 5000 visible in the qdisc dump, suspend/resume and node restart
keep the filters, relay link rejects rate with 409, teardown leaves no
netem. Full unit suite: 1916 passed.
The unified veth host end is born admin-down (carrier off until a link
attaches), and libpcap cannot open a packet socket on an admin-down
interface — its netlink promiscuous-mode transaction returns ENOENT, so
'bridge add_nio_ethernet' failed on every runtime relay link (and would
on any relay link at container start). Bring the interface up for the
attach; the carrier pass in the caller refines the state afterwards (a
suspended NIO sets it back down). Verified live with a direct uBridge
probe: attach on a down veth -> ENOENT, after 'link set up' -> OK.
The GNS3 bpf filter (one expression per line, any line matching drops the
packet) now runs on the kernel datapath: uBridge's tc bpf_drop compiles
each line with libpcap (same compiler as the relay filter) into classic
BPF, attaches it to the veth host end's clsact egress as a cls_bpf
classifier with a gact TC_ACT_SHOT action, and the kernel migrates the
cBPF to eBPF internally -- no CAP_BPF needed.
- reconcile is flush + one add per line at priorities 10, 11, ... (OR
semantics, same as the relay's per-line filters); tc reset (link
delete / clear) is the full restore in uBridge and clears them
- capability-gated: the compute probes "tc capabilities" once per uBridge
process (cached, invalidated on stop); without cbpf=1 the filter apply
fails with an actionable upgrade error, and old uBridge builds without
the tc module are never touched when no bpf filter is present
- a line that compiled for the controller's tcpdump but not for the
compute's uBridge libpcap is skipped with a warning, mirroring the
relay path
- KERNEL_UNSUPPORTED_FILTERS shrinks to frequency_drop (its eBPF stateful
classifier is still pending, spec part B)
e2e (22/22): bpf "icmp" = 100% loss; size-discriminating expression
("greater 150") passes 98-byte and drops 300-byte pings (real byte-level
match); two-line OR; coexistence with netem (delay 50 -> small pings
100.2 ms while big dropped); clear removes clsact; suspend/resume keeps
the drops; link delete + recreate leaves no residual filters.
Impairment filters with a netem equivalent (delay, packet_loss, corrupt)
now run on the kernel datapath as one tc netem qdisc per veth host end,
pushed through uBridge's netlink tc module (tc netem set / tc reset).
This removes the relay delay filter's nanosleep bottleneck (upstream
Both link endpoints carry the filters: each end's qdisc egress covers
the traffic entering that container, so every direction is impaired
exactly once -- the same net effect as the relay, where both directions
cross the single filtered bridge.
- controller: eligibility keys on relay-only types (frequency_drop, bpf)
instead of any filter; kernel links push the real filters (the
synthetic suspend frequency_drop no longer flips a suspended link to
relay on reopen/reset); update_filters 409s only relay-only types;
available_filters hides them on kernel links
- compute: create_nio whitelists netem-capable filter types; the NIO
update path keeps its own guard (PUT mutates the NIO in place);
_ubridge_apply_netem translates the filter dict into one atomic
netem-set (full re-apply IS the reconcile); node restart restores the
qdisc, link deletion detaches it (the veth survives the link)
- e2e (27/27): delay 100 -> 200.2 ms RTT, atomic replace 50 -> 100.2 ms,
clear -> baseline, packet_loss 30 -> 48 % round-trip (expected 51 %),
suspend/resume and node restart keep the qdisc, link delete leaves no
residual impairment
Every Docker adapter is now born as a veth pair at container start
(unified interface) regardless of NIO type: guest end in the container
namespace, host end in the root namespace. Kernel links enslave the host
end into the per-link Linux bridge; relay links attach it to the uBridge
relay via AF_PACKET (bridge add_nio_ethernet) — the TAP path is gone.
This makes the datapath a runtime decision:
- links can be created on running containers (kernel or relay) without
touching container interfaces — the controller stopped-node
eligibility check is removed
- project reopen re-prepares relay links and upgrades eligible ones to
the kernel datapath
- packet capture start/stop now key on the NIO type (not veth presence)
so relay-over-veth captures use the relay capture, kernel ones the
AF_PACKET module
- vendor unix-socket containers keep their wiring: no veth is required
on the relay path
Markers no longer disqualify the kernel datapath. They ride the NIO like
on the relay path and attach to the veth host end through uBridge's
marker add_kernel/delete_kernel/enable_kernel commands (AF_PACKET taps,
same BPF/direction/signal semantics as the relay mark filter).
- base_node: marker anchors are now polymorphic (relay bridge name or
kernel veth host interface); the reconcile loop is datapath-agnostic
and DockerVM overrides the add/delete/enable primitives to translate
the commands. Marker-name validation extracted for both add paths.
- docker_vm: kernel NIO connect/update apply markers on the veth anchor;
link deletion tears markers down explicitly (the veth survives the
link, unlike a relay bridge whose death drops its filters).
- controller: eligibility and start_marker no longer exclude markers or
project-level marker definitions; the kernel NIO carries the capture
node's marker specs.
- NIO creation accepts markers on kernel NIOs (filters still rejected).
Replace the capture rejection with wiring to uBridge's capture module:
an AF_PACKET socket bound to the veth host end writes into the same pcap
file the relay path uses (capture start_kernel <if> "<pcap>" /
capture stop_kernel), so the pcap streaming to clients is unchanged.
- capture survives node restarts: _connect_nio re-arms it when the NIO
is already capturing, mirroring the relay path
- uBridge's kernel capture is a singleton per process: a concurrent
second capture on the same node surfaces uBridge's EALREADY error
- update the datapath docs: filters and markers remain relay-only,
capture is now served on both datapaths
Eligible links (both endpoints Docker, same compute, no filters/markers,
stopped nodes, no unix-socket NIO containers) are wired as veth pairs
enslaved into a per-link Linux kernel bridge (gns3 + 11 hex of the link
id, filling IFNAMSIZ) instead of the uBridge UDP relay, so frames flow
entirely in the kernel.
- NIOBridge NIO type: the controller decides the datapath in
UDPLink._prepare and both computes enslave their deterministic gv*/gc*
veth pair via uBridge's docker/brctl modules
- carrier/suspend = host-end admin state; both endpoints race-safely
create/delete the bridge (EEXIST/EBUSY/ENOENT tolerated)
- veth adapters can still attach relay (UDP) links on a running node:
lazy relay bridge over AF_PACKET on the host end (add_nio_ethernet)
- filters/markers/capture are rejected with explicit errors on this
datapath (they live in the uBridge relay); vendor unix-socket
containers are excluded; Server.enable_kernel_datapath is the
kill-switch (default on)
- host-end veths survive container death and are removed on stop plus
swept by name on the next start (crash/stale recovery)
The rmtree error handler in BaseNode.delete() was chmod'ing the failed
path to S_IWRITE (0o200). On POSIX this strips the search permission
from directories, turning a transient deletion failure into a directory
that can no longer be traversed or deleted. The node deletion itself
silently "succeeds" (rmtree gives up once its error handler returns)
and a later project deletion then fails with EACCES. The handler also
never retried the failed operation, so it did not help on Windows
either (the platform it was written for).
The transient failure exists in practice: a concurrent MD5 checksum
computation caching its result in the node directory (e.g. a properties
request racing the deletion) can recreate a file after rmtree has
listed the directory, making the final rmdir fail with ENOTEMPTY.
- add the missing user permissions instead of replacing the whole mode,
and retry the failed unlink/rmdir
- retry the whole deletion a few times to absorb files recreated while
the directory is being deleted
- raise a ComputeError when the directory cannot be fully deleted
instead of failing silently
Only relying on the image name lets a moved tag (e.g. a newer :latest)
silently serve stale content from a compute that already has an image
under the same name. When creating a Docker node, the controller now
pins the image id (Id from the Docker daemon on the controller host)
into the create payload. A compute holding a different image under the
same tag reports the image as missing, which routes it through the
image sync added by the previous commit and re-aligns the tag.
No new template fields or database changes: the controller host daemon
remains the source of truth and the pin is resolved per creation. When
the image is not available on the controller host the pin is omitted
and behavior is unchanged (the compute pulls from the repository).
When a Docker node is created on a remote compute whose Docker daemon
does not have the image, the compute now raises ImageMissingError
instead of blindly pulling from the Docker repository. The controller
exports the image from the Docker daemon on its host (docker save
stream) and streams it to the compute which loads it, so locally built
or docker-loaded images work across computes. When the image is not
available on the controller host either, the compute is asked to pull
it from the Docker repository as a fallback.
- add a POST /docker/images/load compute endpoint that streams a
docker save tar into the Docker daemon
- let Docker.http_query pass raw (non-dict) request bodies through so
the tar can be streamed to the daemon
- drop the inline pull from DockerVM.create() and the now unused
DockerVM.pull_image wrapper
The one-shot reclaim container inherited the image's baked-in USER:
ghcr.io/nokia/srlinux runs as "user:user", so the "privileged" helper
was exactly as unprivileged as the server itself — chmod/chown on files
written by other uids (srlinux writes as a large internal uid) failed
with EPERM and node/project deletion still broke, just with a different
error. Pass --user 0:0 explicitly so the helper is root no matter what
the image declares, and fix the manual reclaim hint the same way.
Validated live on two stuck srlinux node directories (257/258
foreign-owned entries reclaimed to 0 in ~0.5 s each).
The stop-time permission pass necessarily runs before the container's
processes exit, so files written during the shutdown window (syslog
archives, trace flushes) and after any SIGKILL path stay owned by root
on the host. An unprivileged server can neither chown nor delete them,
which broke node deletion and project deletion.
Reclaim them through the only privilege door a non-root server has:
a one-shot throwaway container of the node's own image, entrypoint
overridden to the GNS3 busybox (nothing of the guest boots), chowning
the node directory back to the server user. It runs at the end of
close() — project deletion rmtrees the directory right after the nodes
close, so close must leave a clean tree — and as a retry fallback in
delete(). The helper resolves the image by its create-time ID with
--pull=never, so a stale or retagged image name cannot turn into a
registry pull attempt.
Upstream #2870 added TAP carrier control (set_nio_tap_carrier) and a
busybox interface-status monitor to the Docker base class. Both assume
TAP wiring: unix-socket NIO bridges carry no TAP NIO, so uBridge rejects
the carrier command ('bridge has no TAP NIO') and every link create,
update or delete on a running iol-runner node would fail; the monitor
polls eth{adapter} interfaces that do not exist in the container's
network namespace (or misreports the Docker default eth0 as adapter 0).
VendorDockerVM now short-circuits _set_adapter_carrier and the interface
monitor under GNS3_UNIX_SOCKET_NIO; TAP-wired vendor nodes keep the base
behavior.
Upstream #2870's carrier tests assert the two-argument
_set_adapter_carrier call; the 4-port-unit commit threads port_number
through every carrier call site, so single-port adapters now pass 0.
create() re-runs _parse_vendor_environment on every (re)create — and a
stop removes the container, so every start recreates it. Resetting the
controller-allocated application id there silently flipped nodes to the
hash fallback after their first stop/start (MAC change, plus collision
risk with the allocation pool), and dropped any pending startup-config
delivered by a PUT.
The application id and startup-config state now default via class
attributes instead of being re-initialized by the parser, and the hash
fallback is removed outright: starting an IOL node without a coordinated
allocation raises an actionable error — an uncoordinated id could
collide with the pool and blackhole traffic as a MAC loop.
Found in the E2E run: node booted app id 512, came back as 566 (hash
fallback) after one stop/start; the PUT-ed config edit vanished the same
way.
Templates reference a config file with the GNS3_IOL_STARTUP_CONFIG
environment knob; the controller materializes the file content into
startup_config_content on node creation (sent once, knob consumed —
the same pattern as the IOU startup_config mapping). The compute builds
the content into the node's nvram_<app id> at the next start using the
IOU nvram_import utility (IOL and IOU share the nvram container format,
verified against iol-xe 17.18.02): valid config at boot, no setup
dialog, %h hostname substitution, hostname rewrite on rename.
Semantics verified against the runner: IOL boots from NVRAM whenever it
holds a config, so a plain stop/start never re-applies the startup
config and 'write memory' survives restarts; an explicit content edit
(PUT) is re-applied on the next start and wins over the saved config,
like IOU.
IOL interface MACs derive from the application ID (aabb.cc{app}{iface}),
so ids must be unique across opened projects sharing computes — the same
reason IOU has its allocator. IOL Docker nodes draw from the upper half
(512-1022, netiomux's fixed peer is 1023) so the two node types can
neither collide nor starve each other; IOU behavior is unchanged.
The controller sniffs the same GNS3_IOL_RUNNER environment marker the
compute uses to select IOLDockerVM (both the nested-properties and
template/top-level-kwarg shapes), stores the id in node properties like
IOU does, and passes it through the Docker create payload. Without an
allocation the compute falls back to a stable node-derived id in the
same upper range.
IOL derives interface MACs from its application ID. With a constant
local-app every node shared the same MACs, and linked routers dropped
each other's frames as MAC loops — ARP never resolved and pings were
100% lost even though the whole uBridge datapath was forwarding.
Derive local-app from the node UUID (stable across restarts, kept in
1..1022) and align remote-app with the netiomux convention CML uses
(1023). Verified end to end: two iol-xe nodes, Ethernet0/0 link,
ARP resolves both ways, ping 4/5 (first loss = ARP), graceful
stop/start keeps the NVRAM config.
IOL interfaces come in 4-port units (Ethernet0/0-3, Ethernet1/0-3, ...),
addressed like IOU as (adapter_number, port_number 0-3):
- IOLDockerVM builds EthernetAdapter(interfaces=4) per adapter and asks
the runner for adapters x 4 interfaces (num-eth).
- DockerVM threads port_number through the NIO/capture API (the compute
routes parsed it from the URL but dropped it); single-port adapters
keep the historical bridge{N} names and command sequence, multi-port
adapters get one bridge per port (bridge{a}_{p}).
- The unix-socket NIO wiring addresses sockets flat across adapters:
adapter x ports-per-adapter + port, so single-port images keep their
exact socket layout.
- The controller port list for GNS3_IOL_RUNNER docker nodes is generated
by StandardPortFactory with the IOU naming (Ethernet{segment0}/{port0},
segment size 4); plain docker nodes keep eth{N}.
_get_container_ifname only names the kernel interface inside the
container's network namespace (single call site: docker move_to_ns) -
IOL never uses that path, and the controller port list never reads it.
Name the ports through the template's custom_adapters instead (the
mechanism SR Linux appliances use for mgmt0/e1-1): adapter N shows as
Ethernet{N/4}/{N%4}, matching the IOS CLI, with no server-side changes.
Name ports after the IOL interface they map to (Ethernet0/0, one 4-port
unit per adapter range) via the _get_container_ifname override point, so
the GNS3 UI matches the IOS CLI. Display only — the flat adapter number
remains the socket index.
uBridge cannot reach sockets bind-mounted from the node's projects
directory: the path exceeds AF_UNIX's 107-byte sun_path cap. The /proc
detour from the previous commit is a dead end too - the runner drops
from root to the server uid, which makes the process non-dumpable and
its /proc/<pid>/root unreadable for the unprivileged server.
Instead, mirror how CML itself runs the image (source=<scratch>/tmp,
target=/tmp in its node definition): bind a per-node directory from the
runtime directory (<XDG_RUNTIME_DIR>/gns3/unixio/<node-id>, next to the
uBridge control sockets) at the image's socket directory. The path is
short, the directory is owned by the server user (whom the agent drops
its privileges to), and it is removed with the node. A socket directory
covered by a persisted volume keeps working as before.
IOLDockerVM persists only /tmp/run (nested bind at /tmp/run) and cleans
stale sockets/netio dirs from the socket directory on start, skipping
the cleanup when the container is already running. A failed wiring now
stops uBridge so the next start does not hit 'bridge already exist'.
The unix-socket NIO no longer requires the socket directory to be a
persisted volume: uBridge references the sockets through
/proc/<container-pid>/root<dir>/..., which is always well under the
107-byte sun_path cap (a project directory path alone exceeds it) and
leaves the sockets ephemeral in the container's own filesystem.
This replaces the runtime-dir symlink alias and its cleanup, and the
mount-time volume enforcement. IOLDockerVM now persists only /tmp/run
(the IOL working directory: startup-config + NVRAM) as a nested bind
instead of the whole /tmp; stale socket cleanup is gone with it, as
containers are recreated on every start.
AF_UNIX sun_path caps at 107 bytes and node volume directories
(projects/<uuid>/project-files/docker/<uuid>/tmp) exceed it — uBridge
rejects the NIO with 'invalid file path size'. When the wiring path is
too long, create <XDG_RUNTIME_DIR>/gns3/unixio-<node-id> as a symlink
to the real volume directory (same trick as the uBridge control
socket) and reference the alias in the uBridge commands; the symlink is
removed when uBridge stops. Found during E2E with iol-xe:17-18-02.
Run Cisco CML containerized IOL images (e.g. iol-xe/iol-xe:17-18-02,
driven by virl.lab/cmd/iol-runner) as GNS3 Docker router nodes:
- VendorDockerVM: generic GNS3_UNIX_SOCKET_NIO/GNS3_UNIX_SOCKET_DIR knobs
wiring adapters through AF_UNIX datagram socket pairs
(add_nio_unix cNN.sock sNN.sock) instead of TAP + move_to_ns; node
creation fails if the socket dir is not a persisted volume.
- IOLDockerVM (GNS3_IOL_RUNNER=1): forces skip-init + unix-socket NIO +
/config,/tmp volumes, writes iol-config.json on every start (num-eth
tracks adapters, runner drops to the server uid/gid so the sockets are
reachable), pre-creates tmp/run, cleans stale sockets after unclean
kills, and makes reload a graceful stop + full start (NVRAM flush +
rewiring). Console stays telnet on PID 1 stdio.
- Manager selects the node class from console_type or GNS3_* environment
markers (create-time, like console_type).
- Image-free tests (25) and feature documentation.
The SKIP_INIT volume bridge replicated init.sh's seed + mount --bind script
via docker exec *after* the container started. That copied the mechanism but
not the invariant that makes init.sh safe — the entrypoint position, which
guarantees the volume is in place before the application runs. The exec runs
concurrently with the NOS boot, so whether the NOS loaded its persisted
config or the overlay's factory copy was a timing race:
- single node stop/start on an idle system won it (exec ~1s, SR Linux reads
its startup config at ~2-4s) — the save/stop/start round-trip passed;
- a server restart + project reload lost it (concurrent node starts queue on
the Docker API, delaying the exec by seconds) — SR Linux booted factory
while the persisted config.json sat intact on the host;
- XRd was immune (systemd boots tens of seconds before XR touches
/xr-storage), which is why the race was never observed on it.
Replace the bridge entirely:
- new DockerVM._prepare_volumes hook (no-op in the base class) runs in
create() after the image is present, before the container is created;
VendorDockerVM overrides it to seed each volume's host directory from the
image (throwaway docker create container + docker cp -a, nothing
executes). The .gns3_perms marker gates the seeding: a volume that ever
started is never re-seeded, so saved configuration is never overwritten
with factory content (also the upgrade path for existing nodes).
- VendorDockerVM._mount_binds now binds the volumes directly at their real
in-container paths (/etc/opt/srlinux) instead of /gns3volumes aliases, so
the persisted config is visible to the NOS from the very first process.
- _setup_skip_init_volumes and its start() call are gone; the container-side
_fix_permissions targets the volume paths directly (the direct binds
exist for the whole container lifetime, unlike the old bridge).
The volume-list computation (validation + overlap de-duplication) moves
into DockerVM._persistent_volume_list so create-time seeding and _mount_binds
cannot drift apart.
A browser's terminal-size control frame (NAWS through the console telnet
server) can arrive while client_connected_hook is still creating the
exec; the resize is a no-op then, and the tall default applied after
creation would overwrite it, leaving the session at 511x10000 until the
user resizes.
Record sizes received before the exec exists and prefer them over the
tall default once creation finishes. The recorded size is cleared when
the last client disconnects, together with the restore-to-default.
The exec behind a docker_exec console is shared by every console client,
so a browser's terminal-size resize (WS control frames -> NAWS) also
changes the geometry concurrent netmiko sessions see. SR Linux doesn't
care (no pager, no hard wrapping), but CLIs that page on the PTY window
size (IOS-XR) would park at --More-- again the moment a browser is
connected.
Split the client-driven NAWS path (_on_naws) from the internal resize
(_resize_exec): GNS3_CONSOLE_RESIZE=0 makes the console ignore client
resizes entirely and keep the tall 511x10000 no-paging default, while
the creation-time default and the restore-on-last-disconnect still go
through the internal path. XRd appliance templates should set it.
The docker_exec console resized its exec PTY to 80x24 until a client
sent NAWS. CLIs that page on the PTY window size instead of the
terminal length (the IOS-XR pager) therefore parked long output at
--More-- for clients that never negotiate NAWS — netmiko, bare telnet —
making copilot device commands time out on XRd.
Default the exec to 511x10000 instead (511 matches netmiko's own
'terminal width 511' convention): no paging and no hard wrapping for
non-NAWS clients, while real NAWS clients keep resizing to their actual
geometry as before.
Also updates the project memory record with the confirmed root cause
and the fix.
The 600 s clamp was unreachable in practice: the controller's stop
request times out at 240 s (controller/node.py) and the Docker stop
query gets the value +30 s as its HTTP timeout, so anything above 210
would abort upstream first and surface an error while the stop keeps
running server-side. Cap at the derived ceiling and document the chain
in the clamp and the docstring.
Nine fixes from a review of the docker-shm-devices diff:
* GNS3_STOP_TIMEOUT >300 s aborted at the manager's default HTTP timeout
before Docker finished the stop — the stop query now gets a timeout
with a margin over the grace period.
* Overlapping bind targets (GNS3_MASK_UDEV + GNS3_MASK_SYSTEMD on the
same unit, a unit named twice, an extra_configs target equal to a
masked unit) made Docker reject the create with 'Duplicate mount
point' — Mounts are deduplicated by target.
* ExtraConfig.target now carries a pydantic validator (absolute file
path, no '..'), so bad targets 422 at template-save time instead of
failing at node-create time after a multi-GB image pull; directory
forms ('/', '/etc/') are also rejected by the runtime guard instead
of raising IsADirectoryError (raw 500).
* _check_host_readiness skipped every remaining check when one
/proc/sys key was unreadable (mid-loop return) — now continues.
* The base-class GNS3_* env parser strips trailing commas like the
vendor parser, so 'GNS3_MASK_UDEV=1,' composed from a list still
activates.
* Vendor env knobs are re-parsed on every create(), so a PUT to the
node's environment takes effect on the next (re)create.
* The graceful SIGTERM stop is now limited to the explicit user stop
route; delete/update/close/crash-cleanup keep the immediate kill
(those paths force-delete or recreate the container right after).
* An extra_configs target beneath a persisted volume is shadowed by the
volume bind — warn at create time.
The 60 s SIGTERM grace was hardcoded, unlike every other vendor knob
(GNS3_SHM_SIZE, GNS3_DEVICES, GNS3_MASK_UDEV, ...) which rides the
environment line. Parse GNS3_STOP_TIMEOUT=<seconds> (default 60,
clamped to 1-600, invalid values keep the default) and use it in
VendorDockerVM._terminate_container().
DockerVM.stop() terminated containers with an immediate SIGKILL — fine
for init.sh-based containers whose state is persisted beforehand, but a
systemd NOS (Cisco XRd, SR Linux) needs a graceful shutdown and treats
the abrupt kill as an unclean shutdown (exit 137 on every stop).
Extract the final termination into _terminate_container() and override
it in VendorDockerVM: POST /containers/{id}/stop?t=60 sends SIGTERM and
waits for systemd to stop services; Docker itself SIGKILLs the
container once the grace period expires, so no fallback is needed.
Docker's 304 (already stopped) is swallowed.
Masking the udev systemd units stopped the daemon's coldplug (host audio
resets), but host USB devices still reconnected on every XRd start. A/B
testing with plain `docker run` isolated the trigger: XRd's own
xr_startup.sh calls udevadm directly (USB license-dongle probing, e.g.
`udevadm trigger --action=add --parent-match=<usb device>`), which
synthesizes uevents into the host kernel from the privileged container --
no udevd required.
GNS3_MASK_UDEV=1 now also binds /dev/null over the udevadm binary
(/bin, /sbin, /usr/bin). Verified with a plain-run experiment: with the
bind, host udev monitor shows zero usb/input/hid/sound events during XRd
boot (only normal docker veth traffic), and XRd itself boots to running
state -- it does not need udevadm under GNS3 (interfaces are pre-created
veths).
