per-def operations are project-level — create, update AND delete all
modify the marker policy on every link — so all three must batch, not
just create.
Extend memory_only to update_marker and stop_marker (merge/delete into
_markers + refresh _link_data, no per-link HTTP), and route
update_marker_definition and delete_marker_definition through the same
two-phase path as create: memory-only per link, then one
PUT /projects/{id}/nios/batch per compute.
Trade-off: a full reapply resets every marker's pcap on the link (the
old precise update_marker/stop_marker preserved sibling pcaps). For
project-level policy changes this is acceptable — real-time insight
matters more than pcap continuity, and batching turns 5000+ round-trips
into one per compute.
When a marker definition is created (or re-applied on data_link_type
change), the fan-out used to call inherit_marker -> update() on every
link, issuing one PUT /nio per link end (5000+ round-trips on a 2500-link
project). On started nodes each round-trip also reconfigured uBridge.
Two-phase fan-out:
- inherit_marker/start_marker gain memory_only: writes the marker into
link._markers and refreshes _link_data without any HTTP/emit/dump.
- _apply_def_to_all_links applies memory-only to every link, then
_batch_update_link_nios groups the updated NIO specs by compute and
sends a single PUT /projects/{id}/nios/batch per compute.
compute: new PUT /projects/{id}/nios/batch endpoint with _get_existing_nio
+ _update_nio_binding dispatch (mirrors create_batch_nios), re-applies
filters+markers to uBridge on started nodes.
Precise per-marker operations (update_marker bpf change, stop_marker on
def delete) are untouched — they deliberately avoid a full reapply to
preserve sibling marker pcaps.
Distinguish the two marker-policy fan-out paths in the log so we can tell
which is slow:
- apply_defs_to_new_link (project-open finalize): logs def count + link
count + elapsed, only when marker_definitions is non-empty.
- _marker_apply_concurrently (interactive create/update/delete def): logs
link count + elapsed per fan-out.
3 is too conservative for modern hardware; 10 provides a moderate boost
without the risk of overwhelming the host (start is the heaviest
operation: ubridge process + docker start + network config for each
node).
The _connect_nio thread-pool optimisation (send_batch_sync) targeted
node-start performance, but start_all already runs at concurrency=3
(by design, to avoid overwhelming the host). It also introduced a
Python 3.13 incompatibility (trsock.setblocking forbidden) that
prevented docker nodes from starting. Since node-start is not the
target of this branch (project-open link creation is), revert to the
simple per-command async _ubridge_send.
The project-open batch NIO dispatch (create_batch_nios) is unaffected —
it never called _connect_nio (nodes aren't started during open).
Python 3.13's asyncio TransportSocket wrapper rejects setblocking(). Dup
the underlying fd via socket.fromfd() into a plain socket that the
executor thread can drive in blocking mode, then detach() after the
batch to avoid closing the transport's fd.
Dynamips.create_nio is async def while BaseManager.create_nio is a sync
def. The previous fix only added the extra 'node' argument but did not
await the resulting coroutine, causing 'was never awaited' warnings and
passing a coroutine object instead of an NIO instance to the binding
dispatch. Add 'await' on the Dynamips branch. Test updated to verify
both the async nature and the parameter count.
The inspect check tested unbound function signatures (3 params unbound vs
2 unbound) but node.manager.create_nio is a bound method — inspect
excludes 'self'. Dynamips bound = 2 (node + nio_settings), standard
bound = 1 (nio_settings). The old '== 3' never matched, so the extra
'node' arg was never passed. Switch to '>= 2' and rewrite the test to
exercise the actual bound-method scenario.
Cover every dispatch branch in the batch NIO endpoint so that future
additions of node types with unusual NIO-binding signatures are caught
at test time.
Dynamips.create_nio requires the node as first positional argument
(unlike every other manager which takes only nio_settings). The batch
handler now detects this via parameter-count inspection (3 vs 2) and
passes node when needed.
Also add Dynamips to _add_nio_binding dispatch: routers use
slot_add_nio_binding(slot, port, nio), switches/hubs fall back to
add_nio(nio, port_number).
Add tests covering Dynamips router dispatch, switch dispatch, and the
create_nio signature detection to prevent regression.
- start_all, stop_all: emit 'starting/stopping N nodes...' and complete
lines with project name+id, matching the open-node/link log style.
- close: emit 'project closing.../closed' bracketing the whole teardown.
- Revert start_all concurrency 20->3 and reset_console 20->3: start
(ubridge process + docker start + network) is heavy enough that high
concurrency risks overwhelming the server. stop remains 100 (kill is
light) and suspend remains 50 (pause is light).
The per-node created / is-closing / Starting-uBridge / Stopping-uBridge
INFO lines flood the log at 1000+ node scale during project open, close,
start-all and stop-all. Demote them to DEBUG (consistent with the
docker_vm lifecycle-log demotion).
When opening a project, nodes are created with Pool(concurrency=100).
Each _create_node did an unlocked check-then-act: 'if compute not in
_project_created_on_compute' -> await POST /projects -> add(compute).
The await let dozens of concurrent node creations race past the check
before any registered, each firing a redundant POST /projects at the
same compute. The compute-side sync handler then ran in a thread pool
and instantiated the Project N times (the repeated 'Project ... created'
INFO logs, ~16x).
Guard the check+POST+register with a project-level asyncio.Lock. The
first creation holds it for one POST; the rest acquire, see the compute
already registered, and return immediately — negligible serialization.
Also annotate the node/link progress logs with project name+id and add
completion lines, so the log clearly shows which project is loading and
when each phase finishes.
At 1000+ nodes the per-node INFO lines flood the log during open /
start-all / stop-all: MAC changed, adapters changed, created, started,
console listen, fix ownership, stopped, paused, removed, adapter created,
NIO removed, capture start/stop, CPU/memory limits, mount resources.
Demote all of these routine per-node/per-adapter lines to DEBUG. Keep
INFO only for genuinely rare/important events: image pull (missing image)
and stale-container cleanup. Warnings unchanged.
A large topology takes ~50s to open (dominated by docker daemon node
creation). Without any start marker the user sees no feedback that work
is in progress. Add two INFO lines: 'Loading N nodes...' before the node
pool and 'Creating N links...' before the bulk link prepare/dispatch.
- Drop the diagnostic stage-timing logs added during link-create perf
work (nodes / preallocate / prepare / dispatch) now that bottlenecks
are resolved and verified.
- Lower the per-NIO 'added to adapter' log in docker_vm from INFO to
DEBUG — at 5000+ NIOs per project open it floods the log at INFO.
Stage timing showed prepare taking 96s for 1275 links (vs 0.31s for the
batch dispatch itself). Root cause: _prepare_link_from_topology called
add_link (dump=True default) and update_link_style/update_show_filters_icon
(each unconditionally dump the full topology). 1275 links x serialize-
and-write-the-whole-topology = the entire 96s.
- add_link(..., dump=False): the project is dumped once at the end of open
- set link._link_style / _show_filters_icon directly instead of the
update_* helpers, which also avoids spurious 'link.updated' notifications
before the link is finalised
The final self.dump() at the end of project.open already persists everything.
Project open used to create each link by issuing two NIO POSTs from the
controller to the compute — ~5000 HTTP round-trips for a 2500-link
topology, all funnelling through the single shared controller/compute
event loop and capping throughput near 12 links/s.
Replace it with a bulk path:
- UDPLink split into _prepare() (local: ports, peer addrs, link_data)
and _commit_nios() (dispatch). create() = prepare + commit (interactive).
- Link.add_node gains batch=True: attach both nodes without triggering
per-link NIO HTTP.
- compute: new POST /projects/{id}/nios/batch endpoint with a unified
_add_nio_binding dispatch across node types (docker/qemu/iou/vpcs/
builtin differ in signature).
- project.open: prepare all links locally, group NIO entries by compute,
send each compute a single /nios/batch, then finalise (wire node/port
refs, mark created, notify, apply marker defs) in parallel.
Cuts controller->compute HTTP from O(links) to O(computes). Test added
for the batch endpoint.
host_ip resolved socket.gethostbyname on every access with no cache.
get_ip_on_same_subnet touches host_ip 2-4 times per link, so opening a
2500-link project issued thousands of blocking DNS calls inline on the
event loop — freezing all concurrent link coroutines each time. This is
the most likely cause of the 12-link/s throughput (1000x below what
Pool(concurrency=100) should deliver) and the burst+pause pattern.
- compute.host_ip: cache the resolution in _host_ip_cache, invalidate
on host setter change
- UDPLink.create: timing log splitting get_ip / ports / nio so the next
project-open confirms where time actually goes
Replace the default asyncio executor (capped at ~32 threads) with a
dedicated ThreadPoolExecutor sized for large-topology parallelism.
When 500 nodes each call _connect_nio, up to 500 OS threads can now
send blocking ubridge commands in parallel — no longer serialised by
either the event loop or a small thread pool.
- ubridge_hypervisor: module-level _ubridge_sync_pool (max_workers=500)
- docker_vm._connect_nio: dispatches to the dedicated pool instead of
the default executor
Replace the 3-5 sequential await _ubridge_send calls in _connect_nio
with a single run_in_executor batch. The batch holds the node-level
asyncio Lock to prevent interleaving with async sends, then uses the
hypervisor's new send_batch_sync method which does blocking socket
sendall/recv inside the thread pool. Different nodes' batches now
run in true OS-thread parallelism rather than serialising through
the asyncio event loop between every command.
- ubridge_hypervisor.send_batch_sync: blocking batch send using
the underlying socket from the asyncio transport, protected by
threading.Lock.
- _connect_nio: builds command list (add_nio_udp, start_capture,
bridge start, reset_packet_filters, add_packet_filter) and
dispatches to the default executor.
When controller and compute share the same process, every HTTP request
to localhost was paying a full TCP handshake (force_close=True forced
connection teardown after each request). With 2500+ links each sending
two NIO POSTs, that's 5000 SYN->SYN-ACK->ACK cycles even for sub-ms
in-memory handlers. Switch to keep-alive for loopback compute
(127.0.0.1 / ::1 / localhost) while keeping force_close for remote
computes that may sit behind NAT/firewalls that drop idle connections.
Temporary diagnostic instrumentation to measure the wall-clock time of
each ubridge command during NIO addition (add_nio_udp, bridge start,
filters, markers). The logs will reveal whether the 12-NIO/s
throughput stems from ubridge command latency itself or from lock
contention / HTTP overhead outside _connect_nio.
Previously the two sides' UDP port reservations and NIO tunnel
POSTs were issued sequentially, even though they are independent
once the peer addresses are known — each node talks to its own
compute/uBridge with no shared lock, so the HTTP round-trips can
overlap.
- Port allocation: gather _allocate_port for both computes
- NIO creation: gather both node.post calls together
- Error handling: if either side fails, roll back whichever side
succeeded (previously only node2-fails→cleanup-node1) before
re-raising the first error
Batch loading (open/import project) benefits most because links
are created at high concurrency and the per-link wall-clock
time is dominated by the sum of its two sequential NIO POSTs.
Docker node stop took ~5s every time. The stop API grace period
(params t=5, unchanged since 2015) was always exhausted: the business
process (often an interactive shell) ignores SIGTERM, and GNS3 doesn't
depend on graceful shutdown — _fix_permissions and /gns3volumes already
persist container state before stop() is called.
Use POST /containers/{id}/kill (SIGKILL, zero delay) instead of stop.
The 409 (container already stopped) replaces the previous 304 handling
for the race where the container exits between the state check and the call.
t=5 traced to commit 33edbefa3 (2015-10-14) "Docker cleanup and
improvements" — introduced with no recorded rationale.
Every started node holds ~3 file descriptors in the server's table (pidfd + stdout/stderr pipes per child process), so a few hundred started nodes exhaust the default 1024 soft limit and the uBridge version-check subprocess fails with EMFILE ('Too many open files: /dev/null').
At startup, best-effort raise RLIMIT_NOFILE to 65535 (capped by the hard limit); failures are logged, never fatal. Runs before daemonize() so the daemon inherits the raised limit. Linux-only, matching the platform target.
Every per-link marker operation (start_marker / stop_marker / update_marker) called Project.dump() -- a full topology serialization + file write. A definition fan-out over 500 links therefore wrote the whole topology 500+ times (each blocking the event loop), which dominated the observed ~1 minute; the NIO round-trips themselves were negligible.
Add a dump: bool = True parameter to the three per-link operations and inherit_marker (matching the existing dump param on Link.add_node). The bulk paths -- definition create fan-out, definition-update sync and re-fan-out, definition-delete cleanup, pause/resume, new-link inheritance -- pass dump=False and their caller dumps once after. apply_defs_to_new_link suppresses per-def dumps too: link create / project open dump once after, so opening a 500-link project with N definitions no longer does 500xN topology writes.
The per-definition fan-out applied markers to links in a serial loop -- one compute round-trip per link. On a 1000-link project that serializes N HTTP round-trips (minutes on remote computes). Fan out with asyncio.gather + Semaphore(32): links are independent (own _markers/_link_data), per-link ControllerError stays isolated, and Project.dump is synchronous + atomic (tmp + rename) so concurrent dumps cannot corrupt the topology file.
Converts the definition-create fan-out, the definition-update sync and re-fan-out loops, and the definition-delete cleanup to the shared _marker_apply_concurrently helper. apply_defs_to_new_link stays serial deliberately: all definitions share one link and each push carries the link's full marker set, so concurrent pushes would race and lose markers.
The per-link create_marker route (links.py) called start_marker without the data_link_type from the body, so it always defaulted to DLT_EN10MB and a serial encapsulation chosen by the caller was silently dropped. Same oversight the IOU capture and definition routes had; one-arg fix.
The IOU override of _ubridge_apply_markers lacked the incremental guard the generic path has, so on an NIO update it re-added every marker the port already carried. uBridge's add_packet_filter rejects a duplicate filter name (packet_filter.c find_packet_filter), so adding a private marker to an IOU link that already hosted an inherited global-* copy failed with 'Failed to add filter global-...' -- the NIO update re-sends ALL markers on the port (inherited + new private), and the pre-existing one collided.
Add the same (name, link_id) in self._marker_filter_bridges skip as the generic base_node path, so an update only installs markers not already on the port. Mirrors how Dynamips/vpcs/etc. stay idempotent across add + update.
Markers now work on serial links (Cisco HDLC / PPP / Frame Relay / ATM), not just Ethernet. A marker carries a data_link_type (default DLT_EN10MB); at the uBridge boundary it becomes the 'mark ... linktype <dlt>' keyword so the BPF compiles and the pcap is written with the matching link-layer.
- MarkerCreate / MarkerDefinitionCreate gain data_link_type (default DLT_EN10MB). Per-link it is create-only; definitions are updatable (a change re-fans-out).
- base_node._marker_linktype() normalizes the GNS3 DLT name (strip DLT_, uppercase, None for EN10MB). Single source is SerialPort.data_link_types, so Cisco PPP -> PPP_SERIAL (50), matching the capture path -- no second mapping table.
- _ubridge_add_marker_filter (generic) and the IOU marker loop append 'linktype <dlt>'.
- Definition fan-out branches on link_type in inherit_marker: Ethernet is always EN10MB; a serial link uses the definition's WAN encapsulation, or is SKIPPED when none was chosen (an EN10MB pcap on serial is undecodable). One definition covers a mixed topology.
- MCP marker_definition exposes data_link_type (None = not forwarded).
- No uBridge rebuild on a data_link_type change -- only that one marker's filter is swapped (delete + re-add), mirroring a BPF change; reset_packet_filters preserves sibling mark filters.
Requires the uBridge build with 'mark ... linktype' support.
The IOU compute capture/start route received data_link_type in the NodeCapture body but dropped it when calling node.start_capture(), so iou_vm.start_capture always defaulted to DLT_EN10MB -- every IOU serial capture (Cisco HDLC / PPP / Frame Relay) was written as an Ethernet pcap. Dynamips, cloud and the L2 switches already forward this value; IOU was the only serial-capable node that omitted it. One-argument fix: the node method already accepts and forwards data_link_type to 'iol_bridge start_capture', so only the route was missing it.
Verified: IOU serial captures now produce the expected linktype (C_HDLC / PPP_SERIAL / FRELAY) instead of Ethernet.
Switch ubridge_control_transport from tcp (-H) to unix (-U), the
AF_UNIX + SO_PEERCRED channel recommended on Linux for kernel-level
peer authentication. tcp is retained as an opt-in for backward
compatibility. Existing deployments that set the key explicitly are
unaffected; only fresh installs / unset keys pick up the new default.
Drop the platform-specific floor (0.9.12 darwin / 0.9.14 others) in
favour of a single 1.2.0 minimum. This server now relies on features
only present in recent uBridge builds: the AF_UNIX control channel
(-U / SO_PEERCRED), the marker (mark) filter, and the brctl-backed
builtin Ethernet Switch. Removes the now-unused sys import left behind
by the dropped darwin branch (target is Linux-only).
A marker definition fans out to every link and auto-selects its capture node
on each, so tx/rx is relative to a node that varies per link — the controller
already rejects it (409). Exposing direction on the MCP definition tool let an
agent ask for something that could only fail. Remove the parameter and the
handler's direction handling; the docstring now points to encoding direction in
the BPF (e.g. 'icmp and icmp[icmptype]==8'). Per-link link_marker keeps
direction, where the capture node is fixed.
128 characters was far beyond any realistic marker label (icmp, arp, tcp-syn)
and would have collided with the pcap filename budget once a tag prefix is
added later. Cap the user-facing name at 32 in both MarkerCreate and
MarkerDefinitionCreate; the compute-side name guard now also rejects names
longer than 48, which covers the `global-{def_name}` inherited form (≤ 39).
