The batch marker-def fan-out (PR #2848) routed create/update/delete
marker_definition through memory_only + a batch PUT /nios/batch that
re-applies markers via _ubridge_apply_markers. But _ubridge_apply_markers
was strictly add-only: it skipped any (name, link_id) already in
_marker_filter_bridges, and reset_packet_filters preserves mark filters
(contract). So:
* delete_marker_definition left the deleted marker's filter alive in
uBridge (still matching / signalling / writing pcap) until node restart.
* update_marker_definition (bpf/tag/direction change) never reached
uBridge — the live filter kept the old expression until node restart.
Make _ubridge_apply_markers a real reconcile against the desired
nio.markers:
- installed but no longer desired → delete_packet_filter + unlink pcap
+ unregister
- desired with changed filter field → rebuild (delete + re-add)
- desired with only enabled changed → instant toggle (pcap preserved)
- desired and unchanged → skip
- desired and new → add
Track installed specs in a parallel _marker_specs dict so changes can be
detected. Both base_node and the IOU iol_bridge override are updated.
Added tests for the delete-removed and rebuild-changed-bpf paths.
High-frequency marker.matches shared the single project notification queue with topology events (node.*/link.*), causing head-of-line blocking. Add a separate marker channel: Notification.project_marker_queue/marker_emit, dispatch routes marker.* off the main project queue, plus a new WS /{project_id}/notifications/markers/ws endpoint. Fully migrated (the main project WS no longer carries marker.match); marker listeners are independent of project auto_close. Compute side unchanged.
The _connect_nio thread-pool optimisation (send_batch_sync) targeted
node-start performance, but start_all already runs at concurrency=3
(by design, to avoid overwhelming the host). It also introduced a
Python 3.13 incompatibility (trsock.setblocking forbidden) that
prevented docker nodes from starting. Since node-start is not the
target of this branch (project-open link creation is), revert to the
simple per-command async _ubridge_send.
The project-open batch NIO dispatch (create_batch_nios) is unaffected —
it never called _connect_nio (nodes aren't started during open).
Dynamips.create_nio is async def while BaseManager.create_nio is a sync
def. The previous fix only added the extra 'node' argument but did not
await the resulting coroutine, causing 'was never awaited' warnings and
passing a coroutine object instead of an NIO instance to the binding
dispatch. Add 'await' on the Dynamips branch. Test updated to verify
both the async nature and the parameter count.
The inspect check tested unbound function signatures (3 params unbound vs
2 unbound) but node.manager.create_nio is a bound method — inspect
excludes 'self'. Dynamips bound = 2 (node + nio_settings), standard
bound = 1 (nio_settings). The old '== 3' never matched, so the extra
'node' arg was never passed. Switch to '>= 2' and rewrite the test to
exercise the actual bound-method scenario.
Cover every dispatch branch in the batch NIO endpoint so that future
additions of node types with unusual NIO-binding signatures are caught
at test time.
Dynamips.create_nio requires the node as first positional argument
(unlike every other manager which takes only nio_settings). The batch
handler now detects this via parameter-count inspection (3 vs 2) and
passes node when needed.
Also add Dynamips to _add_nio_binding dispatch: routers use
slot_add_nio_binding(slot, port, nio), switches/hubs fall back to
add_nio(nio, port_number).
Add tests covering Dynamips router dispatch, switch dispatch, and the
create_nio signature detection to prevent regression.
Project open used to create each link by issuing two NIO POSTs from the
controller to the compute — ~5000 HTTP round-trips for a 2500-link
topology, all funnelling through the single shared controller/compute
event loop and capping throughput near 12 links/s.
Replace it with a bulk path:
- UDPLink split into _prepare() (local: ports, peer addrs, link_data)
and _commit_nios() (dispatch). create() = prepare + commit (interactive).
- Link.add_node gains batch=True: attach both nodes without triggering
per-link NIO HTTP.
- compute: new POST /projects/{id}/nios/batch endpoint with a unified
_add_nio_binding dispatch across node types (docker/qemu/iou/vpcs/
builtin differ in signature).
- project.open: prepare all links locally, group NIO entries by compute,
send each compute a single /nios/batch, then finalise (wire node/port
refs, mark created, notify, apply marker defs) in parallel.
Cuts controller->compute HTTP from O(links) to O(computes). Test added
for the batch endpoint.
Replace the 3-5 sequential await _ubridge_send calls in _connect_nio
with a single run_in_executor batch. The batch holds the node-level
asyncio Lock to prevent interleaving with async sends, then uses the
hypervisor's new send_batch_sync method which does blocking socket
sendall/recv inside the thread pool. Different nodes' batches now
run in true OS-thread parallelism rather than serialising through
the asyncio event loop between every command.
- ubridge_hypervisor.send_batch_sync: blocking batch send using
the underlying socket from the asyncio transport, protected by
threading.Lock.
- _connect_nio: builds command list (add_nio_udp, start_capture,
bridge start, reset_packet_filters, add_packet_filter) and
dispatches to the default executor.
Docker node stop took ~5s every time. The stop API grace period
(params t=5, unchanged since 2015) was always exhausted: the business
process (often an interactive shell) ignores SIGTERM, and GNS3 doesn't
depend on graceful shutdown — _fix_permissions and /gns3volumes already
persist container state before stop() is called.
Use POST /containers/{id}/kill (SIGKILL, zero delay) instead of stop.
The 409 (container already stopped) replaces the previous 304 handling
for the race where the container exits between the state check and the call.
t=5 traced to commit 33edbefa3 (2015-10-14) "Docker cleanup and
improvements" — introduced with no recorded rationale.
The per-definition fan-out applied markers to links in a serial loop -- one compute round-trip per link. On a 1000-link project that serializes N HTTP round-trips (minutes on remote computes). Fan out with asyncio.gather + Semaphore(32): links are independent (own _markers/_link_data), per-link ControllerError stays isolated, and Project.dump is synchronous + atomic (tmp + rename) so concurrent dumps cannot corrupt the topology file.
Converts the definition-create fan-out, the definition-update sync and re-fan-out loops, and the definition-delete cleanup to the shared _marker_apply_concurrently helper. apply_defs_to_new_link stays serial deliberately: all definitions share one link and each push carries the link's full marker set, so concurrent pushes would race and lose markers.
Markers now work on serial links (Cisco HDLC / PPP / Frame Relay / ATM), not just Ethernet. A marker carries a data_link_type (default DLT_EN10MB); at the uBridge boundary it becomes the 'mark ... linktype <dlt>' keyword so the BPF compiles and the pcap is written with the matching link-layer.
- MarkerCreate / MarkerDefinitionCreate gain data_link_type (default DLT_EN10MB). Per-link it is create-only; definitions are updatable (a change re-fans-out).
- base_node._marker_linktype() normalizes the GNS3 DLT name (strip DLT_, uppercase, None for EN10MB). Single source is SerialPort.data_link_types, so Cisco PPP -> PPP_SERIAL (50), matching the capture path -- no second mapping table.
- _ubridge_add_marker_filter (generic) and the IOU marker loop append 'linktype <dlt>'.
- Definition fan-out branches on link_type in inherit_marker: Ethernet is always EN10MB; a serial link uses the definition's WAN encapsulation, or is SKIPPED when none was chosen (an EN10MB pcap on serial is undecodable). One definition covers a mixed topology.
- MCP marker_definition exposes data_link_type (None = not forwarded).
- No uBridge rebuild on a data_link_type change -- only that one marker's filter is swapped (delete + re-add), mirroring a BPF change; reset_packet_filters preserves sibling mark filters.
Requires the uBridge build with 'mark ... linktype' support.
A marker definition fans out to every link and auto-selects its capture node
on each, so tx/rx is relative to a node that varies per link — the controller
already rejects it (409). Exposing direction on the MCP definition tool let an
agent ask for something that could only fail. Remove the parameter and the
handler's direction handling; the docstring now points to encoding direction in
the BPF (e.g. 'icmp and icmp[icmptype]==8'). Per-link link_marker keeps
direction, where the capture node is fixed.
128 characters was far beyond any realistic marker label (icmp, arp, tcp-syn)
and would have collided with the pcap filename budget once a tag prefix is
added later. Cap the user-facing name at 32 in both MarkerCreate and
MarkerDefinitionCreate; the compute-side name guard now also rejects names
longer than 48, which covers the `global-{def_name}` inherited form (≤ 39).
Deleting a marker while its node was stopped, then starting the node, recreated
an empty pcap. Root cause: delete_marker_capture removed the uBridge filter and
the pcap file but not the marker spec cached on the port NIO (nio.markers) —
the data source _ubridge_apply_markers reads on node start. The stale spec
reinstalled the marker when uBridge came up.
This was a regression from switching stop_marker off update() (which re-sent
the NIO and implicitly refreshed nio.markers) to the fine-grained
node.delete(/markers/{name}) path.
Fix: make the delete port-aware so the compute can locate the NIO. The DELETE
marker route becomes /adapters/{a}/ports/{p}/markers/{name} across all six
node types; the handler resolves the NIO via get_nio and passes it to
delete_marker_capture, which now pops the marker from nio.markers. get_nio
works regardless of uBridge state, so the stopped-node case is covered. The
controller's stop_marker targets the capture side's adapter/port.
A marker definition validated its BPF N times — once per link in the fan-out
(start_marker runs validate_bpf_syntax on every copy), spawning one tcpdump -d
subprocess per link for the same expression. Definitions did not validate BPF
at all; only direction was checked.
Move the single validation point to the definition layer (create/update), and
validate each definition's BPF on project load (dropping any that have gone
invalid, like private markers). The inherited fan-out (start_marker) and def
sync (update_marker) now skip validate_bpf_syntax for inherited copies, since
the BPF comes from an already-validated definition. Private per-link markers
still validate inline as before. uBridge still runs pcap_compile at install, so
an invalid expression can never slip through.
Creating a definition over N links now runs one tcpdump instead of N.
_ubridge_apply_markers now installs only markers not already on the bridge
(uBridge's reset_packet_filters preserves mark filters), so an NIO update no
longer re-adds — and reopens — sibling markers' pcaps. _stop_ubridge clears
_marker_filter_bridges so a node restart re-installs everything (the map would
otherwise keep stale entries pointing at a fresh, empty uBridge).
Deleting or updating a marker no longer triggers a full NIO reapply
(reset_packet_filters + re-add), which closed/reopened every sibling
marker's pcap via uBridge. Instead operate on single filters:
- stop_marker: bridge delete_packet_filter + unlink the pcap (works with
the node stopped; filter removal is skipped, the file is still deleted).
- update_marker: bpf/tag/direction → rebuild just that filter (delete + add);
enabled → instant toggle; color/highlight_duration → stored only.
- compute delete_marker_capture / rebuild_marker_filter + per-node routes
(DELETE /markers/{name}, PUT /markers/{name}/rebuild) + MarkerRebuild schema.
IOU overrides _ubridge_delete_marker_filter for iol_bridge; rebuild reuses
the already-overridden add/delete/set, so IOU needs no rebuild override.
Restore a private marker's direction (and highlight_duration) when a
project is reopened — _create_link_from_topology_data previously dropped
them, silently reverting rx/tx markers to "both".
Reject tx/rx direction on marker definitions with HTTP 409: a definition
auto-selects its capture node per link and direction is relative to that
node, so a fixed tx/rx has no stable project-wide meaning. Per-link
markers still support tx/rx; only the project-wide definition is restricted
to "both" (the default).
The _marker_filter_bridges dict was keyed by marker name alone, so when one
node hosted the same filter name on several links (IUOL-BRIDGE per node with
many bays/units, or a multi-interface router), successive apply calls
overwrote earlier entries. pause_marker_definition then toggled only the last
recorded bridge/location — other copies stayed active and kept emitting.
Key by (name, link_id) so each copy is independent, and iterate all matching
entries in _ubridge_set_marker_filter_state (both generic bridge and IOU
iol_bridge override). Toggle route existence checks also iterate matching
names. Tests updated.
The project-wide mute (POST /markers/pause|resume + _markers_paused) paused
every marker with one button. The actual need is per-rule control: pause one
definition and toggle only its inherited global-{name} copies across all links.
- Drop project-level: _markers_paused (init/asdict/load/start_all), the
pause_all/resume_all_markers methods, and the /markers/pause|resume routes.
- Add per-definition: a persisted `paused` flag on each definition;
pause/resume_marker_definition fan out update_marker(enabled) to every
global-{name} copy — instant, via the existing enable_packet_filter toggle
(no NIO rebuild, pcap/emitted preserved). New links inherit a paused
definition already off (inherit_marker passes enabled=not paused).
- start_marker takes an enabled kwarg; update_marker's enabled-only short-circuit
now also covers inherited copies so def pause/resume is instant.
- Routes: POST /marker-definitions/{name}/pause|resume.
- Docs + tests updated.
pause/resume was fire-and-forget: the controller sent marker pause/resume but
stored nothing, so the Web UI could only keep a local optimistic flag that was
lost on panel reopen. Treat the mute as a project-level config (like per-marker
enabled): record _markers_paused, persist it in the topology (asdict + load),
and echo it on the project object so the UI renders from server truth.
Because marker pause is a uBridge runtime flag that resets on node restart,
start_all re-applies the mute to freshly started uBridges after a project
reopen — a paused project stays paused across close/reopen.
- compute (test_base_node.py): set_marker_filter_state on/off command,
marker pause/resume command, and apply issues enable_packet_filter off
for a disabled marker (+ records the name->bridge map).
- controller (test_marker.py): _markers_for_node keeps disabled markers
and carries enabled; update_marker enabled-only hits the toggle route
(not NIO rebuild) while a bpf change still rebuilds; pause/resume fan
out to capture nodes.
A marker is single-sided — only the chosen capture node's uBridge installs the
mark filter — and dir=tx|rx is interpreted from that node's perspective. Until
now the observer was always auto-picked (_choose_marker_side), so dir=tx meant
"the auto-chosen endpoint is sending", which is unpredictable and makes the
direction filter hard to render meaningfully in the Web UI.
Add an optional create-only capture_node_id to MarkerCreate: when set, the
marker is pinned to that endpoint's uBridge (validated as a link endpoint and a
marker-capable type); when omitted, behavior is unchanged (auto-pick). The
chosen id is already echoed back as capture_node_id and in MARK signals, so the
UI can always render the observer regardless of who picked it.
capture_node_id is create-only (changing it would silently flip the meaning of
stored direction; recreate instead) and is not accepted on project-level
definitions — they are link-agnostic and have no endpoints, so inherited
markers keep auto-picking per link.
Plumbed through REST create_marker, the MCP link_marker tool, and base
Link.start_marker. update_marker does not forward it.
Direction field in marker.match events
=======================================
Read ubridge dir=<tx|rx> from MARK signal datagrams and forward it
as a "dir" key in the marker.match notification event. The field is
additive -- older ubridge builds omit it and the parser leaves it null,
so consumers fall back to undirected rendering with no version coupling.
Semantics are relative to the capture node (the signal's node=<id>):
tx = capture node is sending (ingressed device-side NIO), rx = it is
receiving (ingressed link-side NIO).
Per-marker direction filter (opt-in, server-side pipeline)
==========================================================
Add a direction field to MarkerCreate and MarkerDefinitionCreate
schemas ("tx" | "rx" | null). Plumb it through the full pipeline:
Schema -> controller (start_marker/update_marker, marker_entry,
_markers_for_node, update_marker_definition sync)
-> REST/MCP handlers -> compute _ubridge_add_marker_filter +
IOU _ubridge_apply_markers -> bridge add_packet_filter dir <tx|rx>
When set, ubridge only fires the mark handler (signal + pcap) for
packets matching the chosen direction. null (default/legacy) = both
directions -- zero behavioural change for existing markers.
Docs and tests
==============
- docs/features/marker-traffic-insight.md: signal format updated,
new Direction section with NIO mapping, arrow mapping, and additive
compatibility note.
- tests/compute/marker/test_marker_manager.py: 3 new parser tests
(dir tx/rx/absent) plus existing test extended to assert dir=None.
13 files, +123/-22, 72 tests pass (zero breakage)
Project.delete() calls open() to rebuild the internal data structures
needed for cleanup. open() schedules start_all() when the project has
auto_start enabled, so deleting an auto-start project actually launched
every node process, allocating ports and consuming resources, only for
close() to kill them moments later.
open() now takes an auto_start argument (default True, so normal opens
are unchanged) and delete() passes auto_start=False.
Fixes#2784
The condition 'state != "stopped" or state != "exited"' is a tautology,
so the state check was a no-op and a stop request was sent even for a
container that had already exited.
_get_container_state() never returns "stopped" (only "running",
"paused" or "exited"), so the intended negation of the condition used in
_fix_permissions() requires 'and', not 'or' (De Morgan's law).
Added a regression test asserting no stop query is issued for an
already-exited container.
NAT now includes an 'interfaces' field in its response, filtered to
contain only the mapped NAT interface (virbr0 on Linux, vmnet8 on
macOS/Windows). This lets connected nodes discover the NAT subnet
and gateway address without needing to list all host interfaces.
The field format mirrors Cloud.asdict(): name, type, special, and
ip_addresses list (both IPv4 and IPv6).
Replace psutil.net_if_addrs() scan with a deterministic name derived from
the switch's node UUID: gns3 + first 6 hex chars (10 chars, fits kernel
IFNAMSIZ limit of 15). Taps: <bridge>-<port> (12-13 chars).
Crash recovery: brctl delete the bridge first (best-effort), then create
fresh. Stale bridges from abnormal gns3server shutdown are automatically
reclaimed on the next start — no EEXIST or leaked interfaces.
Remove _free_iface helper and psutil import (no longer needed).
brctl create leaves the bridge administratively DOWN. Add link set up
so the bridge actually forwards frames between enslaved ports.
Also record Docker iptables FORWARD DROP pitfall in project memory.
Replace the builtin EthernetSwitch stub with a Linux kernel bridge backed
by uBridge's brctl module. Each switch node creates one kernel bridge
(gns3br{N}) with VLAN filtering; each port is a persistent TAP enslaved to
the bridge, relayed by a per-port uBridge bridge (nio_tap <-> nio_udp).
- Access/dot1q/qinq port modes translated to brctl vlan primitives
- Compute router repointed from Dynamips to Builtin manager
- Tests updated: 21 router-level tests + 215 surrounding tests pass
- Real-kernel e2e verified: access 100 PVID untagged, dot1q trunk
VIDs 1-4094 + native 1, qinq 802.1ad proto + 200 PVID
Ethertype 0x9100/0x9200 handling and port-count guard relaxation are
deferred pending resolution.
For Docker nodes, reload bottoms out as a raw POST /containers/{id}/restart
to the Docker daemon, bypassing GNS3's start/stop lifecycle (uBridge
re-attach, console servers, NIC setup). The container restarts at the
Docker level but GNS3's plumbing goes out of sync, and the daemon call can
block up to the controller's 240s timeout — manifesting as "reload hangs /
no response". stop+start runs the full lifecycle and is reliable.
Remove the node_reload and node_reload_all MCP tools (tool functions,
handlers, NODE_TOOLS entry, tests, docs). The underlying REST endpoints
(POST /nodes/{id}/reload, POST /nodes/reload) are kept for native API
users. MCP callers should use node_stop + node_start (partial) or
close/open project (full restart) instead.
A uBridge MARK signal carries only node= and filter= (no bridge/link field), so when one node is the capture side for several links that share a marker name (always the case for global-{name} definitions on a multi-interface node) the signals were indistinguishable and the (node, filter) registry collapsed them to a single link.
The mark filter is now stamped with its link id (mark <bpf> ... link <link_id>); uBridge echoes it verbatim (link=<link_id>) and the listener uses the signal's link= as the authoritative link_id of the marker.match event, falling back to the registry only for legacy signals without it. base_node and iou apply paths pass link_id; covered by two new listener tests.
19 controller-layer tests covering start/stop/update_marker (storage,
inheritance guards + bypasses, partial-update preservation of render
hints), project-def CRUD (fan-out, sync, delete-cleanup regression),
apply_defs_to_new_link, persist_markers/asdict, and aggregation.
15 API-route tests covering per-link create/update/delete (201/200/204),
global-prefix rejection on create (409 regression), bad-format rejection
(422), PUT/DELETE-on-inherited guard (409 regression), project-def CRUD
endpoints, and the aggregation view.
All 34 tests pass when run as part of the full suite.
Mirror the packet-filter lifecycle: marker specs now live on the NIO
(next to filters), ride in link_data from controller to compute on
every NIO create/update, and are reapplied by _ubridge_apply_markers
in add_ubridge_udp_connection (bridge creation / node restart) and
update_ubridge_udp_connection (NIO update — following the preceding
reset_packet_filters so markers survive filter changes).
Changes:
- NIO / NIOUDP: _markers property + asdict
- schemas/compute/nios.py: UDPNIO.markers field
- base_manager.create_nio: nio.markers from settings
- PUT /nio routes (vpcs/qemu/docker): nio.markers update
- base_node: _ubridge_apply_markers(bridge_name, nio) iterates
nio.markers, computes pcap path, calls _ubridge_add_marker_filter
+ MarkerManager.register; called after _ubridge_apply_filters
- controller udp_link: _get_node_markers + _markers_for_node (route
by capture_node_id); markers in create() and update() link_data
- /markers/start,stop endpoints: mirror spec onto nio.markers so
the marker survives a subsequent node stop/start without a PUT
- tests: add markers field to NIO data expectations
This covers:
- Node restart: NIO persists, add_ubridge_udp_connection re-applies
- Filter update: reset wipes markers, _ubridge_apply_markers re-adds
- Project reload: create() carries markers in link_data → create_nio
- Immediate create: endpoint sets nio.markers immediately
Each host interface surfaced by the cloud node now reports:
- ip_addresses: every IPv4 and IPv6 address (previously only a single
IPv4 was collected internally and then dropped before the response)
- status / speed / mtu / flags: operational state and link attributes
sourced from psutil.net_if_stats(), with flags normalized to a list
The legacy ip_address / netmask / mac_address fields are preserved so
existing callers (compute link detection, GNS3 VM, VMware, has_netmask)
keep working. The new fields travel through the existing interfaces
payload that the controller forwards verbatim, so no controller-side
change is required and the PUT / ports_mapping flow is unaffected.
- New config: Controller.jwt_refresh_token_expire_minutes (default 30 days)
- New endpoint: POST /v3/access/users/refresh (public, unauthenticated)
- Login/authenticate responses now include refresh_token
- AuthService: _create_token helper, create_refresh_token, get_token_data
now parses type claim (token_use) for token classification
- Security: refresh tokens rejected on HTTP + WebSocket access paths;
/refresh strictly requires type=='refresh'
- Logout works for free via existing token_version mechanism
- Tests: 9 new TestRefreshToken cases, all passing; 34 existing tests
still pass (no regressions)
