1187 Commits

Author SHA1 Message Date
YueGuobin
3aa39a2da3
perf: raise start/stop/suspend/reset-console concurrency from 3
The historic concurrency=3 was far too conservative for 1000+ node
topologies. Raise them based on operation weight:

  stop_all:      3 -> 100  (docker kill + permissions cleanup, light)
  suspend_all:   3 ->  50  (docker pause/unpause, light)
  start_all:     3 ->  20  (ubridge startup + docker start + network, heavy)
  reset_console: 3 ->  20  (terminal reset, light)

Node creation (open) already uses concurrency=100 and approaches the
container-create daemon limit (~20/s); stop/kill is substantially faster
than create, so 100 is safe.
2026-08-11 00:33:49 +08:00
YueGuobin
7dc66d4836
fix: race in concurrent node creation sending duplicate POST /projects
When opening a project, nodes are created with Pool(concurrency=100).
Each _create_node did an unlocked check-then-act: 'if compute not in
_project_created_on_compute' -> await POST /projects -> add(compute).
The await let dozens of concurrent node creations race past the check
before any registered, each firing a redundant POST /projects at the
same compute. The compute-side sync handler then ran in a thread pool
and instantiated the Project N times (the repeated 'Project ... created'
INFO logs, ~16x).

Guard the check+POST+register with a project-level asyncio.Lock. The
first creation holds it for one POST; the rest acquire, see the compute
already registered, and return immediately — negligible serialization.

Also annotate the node/link progress logs with project name+id and add
completion lines, so the log clearly shows which project is loading and
when each phase finishes.
2026-08-11 00:30:33 +08:00
YueGuobin
4c6ef7752a
log: emit progress lines for node and link loading on project open
A large topology takes ~50s to open (dominated by docker daemon node
creation). Without any start marker the user sees no feedback that work
is in progress. Add two INFO lines: 'Loading N nodes...' before the node
pool and 'Creating N links...' before the bulk link prepare/dispatch.
2026-08-11 00:16:32 +08:00
YueGuobin
e2fd922922
cleanup: remove project-open stage timing logs, lower NIO-added log to debug
- Drop the diagnostic stage-timing logs added during link-create perf
  work (nodes / preallocate / prepare / dispatch) now that bottlenecks
  are resolved and verified.
- Lower the per-NIO 'added to adapter' log in docker_vm from INFO to
  DEBUG — at 5000+ NIOs per project open it floods the log at INFO.
2026-08-11 00:14:58 +08:00
YueGuobin
c9a93c0aeb
perf: skip per-link topology dump during project-open prepare
Stage timing showed prepare taking 96s for 1275 links (vs 0.31s for the
batch dispatch itself). Root cause: _prepare_link_from_topology called
add_link (dump=True default) and update_link_style/update_show_filters_icon
(each unconditionally dump the full topology). 1275 links x serialize-
and-write-the-whole-topology = the entire 96s.

- add_link(..., dump=False): the project is dumped once at the end of open
- set link._link_style / _show_filters_icon directly instead of the
  update_* helpers, which also avoids spurious 'link.updated' notifications
  before the link is finalised

The final self.dump() at the end of project.open already persists everything.
2026-08-10 23:54:26 +08:00
YueGuobin
38c49a655c
perf: batch NIO dispatch on project open (one HTTP per compute)
Project open used to create each link by issuing two NIO POSTs from the
controller to the compute — ~5000 HTTP round-trips for a 2500-link
topology, all funnelling through the single shared controller/compute
event loop and capping throughput near 12 links/s.

Replace it with a bulk path:
- UDPLink split into _prepare() (local: ports, peer addrs, link_data)
  and _commit_nios() (dispatch). create() = prepare + commit (interactive).
- Link.add_node gains batch=True: attach both nodes without triggering
  per-link NIO HTTP.
- compute: new POST /projects/{id}/nios/batch endpoint with a unified
  _add_nio_binding dispatch across node types (docker/qemu/iou/vpcs/
  builtin differ in signature).
- project.open: prepare all links locally, group NIO entries by compute,
  send each compute a single /nios/batch, then finalise (wire node/port
  refs, mark created, notify, apply marker defs) in parallel.

Cuts controller->compute HTTP from O(links) to O(computes). Test added
for the batch endpoint.
2026-08-10 23:42:50 +08:00
YueGuobin
96d4d82716
perf: cache compute.host_ip + add UDPLink.create timing log
host_ip resolved socket.gethostbyname on every access with no cache.
get_ip_on_same_subnet touches host_ip 2-4 times per link, so opening a
2500-link project issued thousands of blocking DNS calls inline on the
event loop — freezing all concurrent link coroutines each time. This is
the most likely cause of the 12-link/s throughput (1000x below what
Pool(concurrency=100) should deliver) and the burst+pause pattern.

- compute.host_ip: cache the resolution in _host_ip_cache, invalidate
  on host setter change
- UDPLink.create: timing log splitting get_ip / ports / nio so the next
  project-open confirms where time actually goes
2026-08-10 23:20:40 +08:00
YueGuobin
506b0b9f4a
perf: enable TCP keep-alive for local compute HTTP requests
When controller and compute share the same process, every HTTP request
to localhost was paying a full TCP handshake (force_close=True forced
connection teardown after each request).  With 2500+ links each sending
two NIO POSTs, that's 5000 SYN->SYN-ACK->ACK cycles even for sub-ms
in-memory handlers.  Switch to keep-alive for loopback compute
(127.0.0.1 / ::1 / localhost) while keeping force_close for remote
computes that may sit behind NAT/firewalls that drop idle connections.
2026-08-10 23:00:59 +08:00
YueGuobin
acc9e670af
perf: parallelize UDP port allocation and NIO creation in UDPLink.create
Previously the two sides' UDP port reservations and NIO tunnel
POSTs were issued sequentially, even though they are independent
once the peer addresses are known — each node talks to its own
compute/uBridge with no shared lock, so the HTTP round-trips can
overlap.

- Port allocation: gather _allocate_port for both computes
- NIO creation: gather both node.post calls together
- Error handling: if either side fails, roll back whichever side
  succeeded (previously only node2-fails→cleanup-node1) before
  re-raising the first error

Batch loading (open/import project) benefits most because links
are created at high concurrency and the per-link wall-clock
time is dominated by the sum of its two sequential NIO POSTs.
2026-08-10 22:36:41 +08:00
grossmj
bc8e43dfca
fix: add event handling for deleted projects 2026-08-08 17:27:52 +02:00
grossmj
dff79f65d3
Merge branch '2.2' into 3.1
# Conflicts:
#	gns3server/compute/qemu/qemu_vm.py
#	gns3server/controller/import_project.py
#	gns3server/controller/project.py
#	gns3server/crash_report.py
#	gns3server/version.py
#	tests/compute/docker/test_docker_vm.py
#	tests/controller/test_import_project.py
#	tests/controller/test_project.py
2026-08-08 16:27:14 +02:00
YueGuobin
ad8bac8328
marker: batch topology dumps in bulk fan-out (per-def on 500+ links was ~1 minute)
Every per-link marker operation (start_marker / stop_marker / update_marker) called Project.dump() -- a full topology serialization + file write. A definition fan-out over 500 links therefore wrote the whole topology 500+ times (each blocking the event loop), which dominated the observed ~1 minute; the NIO round-trips themselves were negligible.

Add a dump: bool = True parameter to the three per-link operations and inherit_marker (matching the existing dump param on Link.add_node). The bulk paths -- definition create fan-out, definition-update sync and re-fan-out, definition-delete cleanup, pause/resume, new-link inheritance -- pass dump=False and their caller dumps once after. apply_defs_to_new_link suppresses per-def dumps too: link create / project open dump once after, so opening a 500-link project with N definitions no longer does 500xN topology writes.
2026-08-08 00:03:30 +08:00
YueGuobin
078cf92aef
marker: concurrent (bounded) definition fan-out
The per-definition fan-out applied markers to links in a serial loop -- one compute round-trip per link. On a 1000-link project that serializes N HTTP round-trips (minutes on remote computes). Fan out with asyncio.gather + Semaphore(32): links are independent (own _markers/_link_data), per-link ControllerError stays isolated, and Project.dump is synchronous + atomic (tmp + rename) so concurrent dumps cannot corrupt the topology file.

Converts the definition-create fan-out, the definition-update sync and re-fan-out loops, and the definition-delete cleanup to the shared _marker_apply_concurrently helper. apply_defs_to_new_link stays serial deliberately: all definitions share one link and each push carries the link's full marker set, so concurrent pushes would race and lose markers.
2026-08-07 23:50:41 +08:00
YueGuobin
3322233658
marker: serial-link (WAN) support via data_link_type -> uBridge linktype
Markers now work on serial links (Cisco HDLC / PPP / Frame Relay / ATM), not just Ethernet. A marker carries a data_link_type (default DLT_EN10MB); at the uBridge boundary it becomes the 'mark ... linktype <dlt>' keyword so the BPF compiles and the pcap is written with the matching link-layer.

- MarkerCreate / MarkerDefinitionCreate gain data_link_type (default DLT_EN10MB). Per-link it is create-only; definitions are updatable (a change re-fans-out).
- base_node._marker_linktype() normalizes the GNS3 DLT name (strip DLT_, uppercase, None for EN10MB). Single source is SerialPort.data_link_types, so Cisco PPP -> PPP_SERIAL (50), matching the capture path -- no second mapping table.
- _ubridge_add_marker_filter (generic) and the IOU marker loop append 'linktype <dlt>'.
- Definition fan-out branches on link_type in inherit_marker: Ethernet is always EN10MB; a serial link uses the definition's WAN encapsulation, or is SKIPPED when none was chosen (an EN10MB pcap on serial is undecodable). One definition covers a mixed topology.
- MCP marker_definition exposes data_link_type (None = not forwarded).
- No uBridge rebuild on a data_link_type change -- only that one marker's filter is swapped (delete + re-add), mirroring a BPF change; reset_packet_filters preserves sibling mark filters.

Requires the uBridge build with 'mark ... linktype' support.
2026-08-07 01:53:33 +08:00
YueGuobin
75f278228b
marker: drop deleted marker from port NIO cache to stop empty pcap on restart
Deleting a marker while its node was stopped, then starting the node, recreated
an empty pcap. Root cause: delete_marker_capture removed the uBridge filter and
the pcap file but not the marker spec cached on the port NIO (nio.markers) —
the data source _ubridge_apply_markers reads on node start. The stale spec
reinstalled the marker when uBridge came up.

This was a regression from switching stop_marker off update() (which re-sent
the NIO and implicitly refreshed nio.markers) to the fine-grained
node.delete(/markers/{name}) path.

Fix: make the delete port-aware so the compute can locate the NIO. The DELETE
marker route becomes /adapters/{a}/ports/{p}/markers/{name} across all six
node types; the handler resolves the NIO via get_nio and passes it to
delete_marker_capture, which now pops the marker from nio.markers. get_nio
works regardless of uBridge state, so the stopped-node case is covered. The
controller's stop_marker targets the capture side's adapter/port.
2026-08-05 00:08:12 +08:00
YueGuobin
1a0ce51f38
marker: validate def BPF once, skip re-validation on inherited fan-out
A marker definition validated its BPF N times — once per link in the fan-out
(start_marker runs validate_bpf_syntax on every copy), spawning one tcpdump -d
subprocess per link for the same expression. Definitions did not validate BPF
at all; only direction was checked.

Move the single validation point to the definition layer (create/update), and
validate each definition's BPF on project load (dropping any that have gone
invalid, like private markers). The inherited fan-out (start_marker) and def
sync (update_marker) now skip validate_bpf_syntax for inherited copies, since
the BPF comes from an already-validated definition. Private per-link markers
still validate inline as before. uBridge still runs pcap_compile at install, so
an invalid expression can never slip through.

Creating a definition over N links now runs one tcpdump instead of N.
2026-08-04 23:20:58 +08:00
YueGuobin
60e2bbbbbb
marker: point directional defs at BPF, refresh implementation doc
The 409 on a tx/rx definition now recommends encoding direction in the BPF
(e.g. icmp[icmptype]==8) as the primary fix, with per-link markers as the
single-link fallback. Doc updated: per-def rejects tx/rx (why + BPF), the
pause section no longer claims bpf changes reset+reapply (they rebuild one
filter), and a new Capture files section covers pcap cleanup + reset-preserves-mark.
2026-08-04 22:59:12 +08:00
YueGuobin
caec71aa71
marker: fine-grained filter ops, clean pcap on remove
Deleting or updating a marker no longer triggers a full NIO reapply
(reset_packet_filters + re-add), which closed/reopened every sibling
marker's pcap via uBridge. Instead operate on single filters:

- stop_marker: bridge delete_packet_filter + unlink the pcap (works with
  the node stopped; filter removal is skipped, the file is still deleted).
- update_marker: bpf/tag/direction → rebuild just that filter (delete + add);
  enabled → instant toggle; color/highlight_duration → stored only.
- compute delete_marker_capture / rebuild_marker_filter + per-node routes
  (DELETE /markers/{name}, PUT /markers/{name}/rebuild) + MarkerRebuild schema.

IOU overrides _ubridge_delete_marker_filter for iol_bridge; rebuild reuses
the already-overridden add/delete/set, so IOU needs no rebuild override.
2026-08-04 21:35:31 +08:00
YueGuobin
19815f7a37
marker: restore direction on project load, reject tx/rx on definitions
Restore a private marker's direction (and highlight_duration) when a
project is reopened — _create_link_from_topology_data previously dropped
them, silently reverting rx/tx markers to "both".

Reject tx/rx direction on marker definitions with HTTP 409: a definition
auto-selects its capture node per link and direction is relative to that
node, so a fixed tx/rx has no stable project-wide meaning. Per-link
markers still support tx/rx; only the project-wide definition is restricted
to "both" (the default).
2026-08-04 10:14:02 +08:00
YueGuobin
1eeee024bd
marker: rework pause/resume from project-wide to per-definition
The project-wide mute (POST /markers/pause|resume + _markers_paused) paused
every marker with one button. The actual need is per-rule control: pause one
definition and toggle only its inherited global-{name} copies across all links.

- Drop project-level: _markers_paused (init/asdict/load/start_all), the
  pause_all/resume_all_markers methods, and the /markers/pause|resume routes.
- Add per-definition: a persisted `paused` flag on each definition;
  pause/resume_marker_definition fan out update_marker(enabled) to every
  global-{name} copy — instant, via the existing enable_packet_filter toggle
  (no NIO rebuild, pcap/emitted preserved). New links inherit a paused
  definition already off (inherit_marker passes enabled=not paused).
- start_marker takes an enabled kwarg; update_marker's enabled-only short-circuit
  now also covers inherited copies so def pause/resume is instant.
- Routes: POST /marker-definitions/{name}/pause|resume.
- Docs + tests updated.
2026-08-02 22:46:33 +08:00
YueGuobin
f7d7ba165a
marker: persist project-wide markers_paused to the .gns3 file
pause/resume was fire-and-forget: the controller sent marker pause/resume but
stored nothing, so the Web UI could only keep a local optimistic flag that was
lost on panel reopen. Treat the mute as a project-level config (like per-marker
enabled): record _markers_paused, persist it in the topology (asdict + load),
and echo it on the project object so the UI renders from server truth.

Because marker pause is a uBridge runtime flag that resets on node restart,
start_all re-applies the mute to freshly started uBridges after a project
reopen — a paused project stays paused across close/reopen.
2026-08-02 22:21:05 +08:00
YueGuobin
3d06c4e22f
marker: drive uBridge enabled/pause/resume in real time
Wire gns3-server to uBridge's real-time marker controls (contract
../ubridge/doc/gns3server-integration.md §3.2), in three layers:

A. enabled reaches uBridge — _markers_for_node no longer drops disabled
   markers controller-side; it carries `enabled` in the NIO spec. apply
   installs every marker then issues `enable_packet_filter … off` for the
   disabled ones (base_node bridge / iou iol_bridge; old-ubridge errors
   downgrade to a warning so a toggle can't break link create).

B. instant per-filter toggle — apply records name→bridge so a new
   `_ubridge_set_marker_filter_state` can flip a running filter with
   `enable_packet_filter on|off` (iou overrides for iol_bridge + bay/unit).
   Each marker-capable node type gains PUT /markers/{name}; update_marker
   short-circuits to it when only `enabled` changes (no NIO rebuild, no pcap
   flush), falling back to reset+reapply if the route is unavailable.

C. global pause/resume — `_ubridge_marker_pause/resume` send `marker pause`
   / `marker resume` direct to the hypervisor (pause stops signal+pcap,
   resume instant, sink retained). Six node-type routes add POST
   /markers/pause|resume; project.pause_all/resume_all_markers fan out to
   each capture node (deduped, best-effort); REST exposes
   POST /projects/{id}/markers/pause|resume.

Toggling enabled and pause/resume are now both instant — only marker create
or a bpf change still go through reset+reapply.
2026-08-02 16:19:20 +08:00
YueGuobin
37cb9f0a9c
marker: support clearing direction via explicit null / "both"
direction was settable but not clearable: once a marker or project-level
definition had direction=tx/rx, no update path could return it to "both
directions", and the definition-sync fan-out silently kept stale values on
every inherited copy.

Introduce a _UNSET sentinel (link.py) distinct from None so updaters can
tell "caller omitted direction" (preserve) from "caller passed None"
(clear). Thread it through UDPLink.update_marker and
Project.update_marker_definition; the two REST routes use Pydantic v2
model_fields_set to translate an explicit JSON null into the sentinel.

MCP follows with a "both" token: link_marker / marker_definition handlers
map direction="both" to a null in the REST body (tri-state: omit=preserve,
tx/rx=set, both=clear), and the tool descriptions/docstrings document it.

Backward compatible: omitting direction or passing tx/rx behaves exactly
as before; only an explicit null / "both" clears.
2026-08-02 11:37:49 +08:00
YueGuobin
71fa778d50
marker: let callers pin the capture node via capture_node_id
A marker is single-sided — only the chosen capture node's uBridge installs the
mark filter — and dir=tx|rx is interpreted from that node's perspective. Until
now the observer was always auto-picked (_choose_marker_side), so dir=tx meant
"the auto-chosen endpoint is sending", which is unpredictable and makes the
direction filter hard to render meaningfully in the Web UI.

Add an optional create-only capture_node_id to MarkerCreate: when set, the
marker is pinned to that endpoint's uBridge (validated as a link endpoint and a
marker-capable type); when omitted, behavior is unchanged (auto-pick). The
chosen id is already echoed back as capture_node_id and in MARK signals, so the
UI can always render the observer regardless of who picked it.

capture_node_id is create-only (changing it would silently flip the meaning of
stored direction; recreate instead) and is not accepted on project-level
definitions — they are link-agnostic and have no endpoints, so inherited
markers keep auto-picking per link.

Plumbed through REST create_marker, the MCP link_marker tool, and base
Link.start_marker. update_marker does not forward it.
2026-08-01 22:24:47 +08:00
YueGuobin
6749b872fa
marker: forward dir= from ubridge and add per-marker direction filter
Direction field in marker.match events
=======================================

Read ubridge dir=<tx|rx> from MARK signal datagrams and forward it
as a "dir" key in the marker.match notification event.  The field is
additive -- older ubridge builds omit it and the parser leaves it null,
so consumers fall back to undirected rendering with no version coupling.
Semantics are relative to the capture node (the signal's node=<id>):
tx = capture node is sending (ingressed device-side NIO), rx = it is
receiving (ingressed link-side NIO).

Per-marker direction filter (opt-in, server-side pipeline)
==========================================================

Add a direction field to MarkerCreate and MarkerDefinitionCreate
schemas ("tx" | "rx" | null).  Plumb it through the full pipeline:

  Schema -> controller (start_marker/update_marker, marker_entry,
  _markers_for_node, update_marker_definition sync)
  -> REST/MCP handlers -> compute _ubridge_add_marker_filter +
  IOU _ubridge_apply_markers -> bridge add_packet_filter dir <tx|rx>

When set, ubridge only fires the mark handler (signal + pcap) for
packets matching the chosen direction.  null (default/legacy) = both
directions -- zero behavioural change for existing markers.

Docs and tests
==============

- docs/features/marker-traffic-insight.md: signal format updated,
  new Direction section with NIO mapping, arrow mapping, and additive
  compatibility note.
- tests/compute/marker/test_marker_manager.py: 3 new parser tests
  (dir tx/rx/absent) plus existing test extended to assert dir=None.

13 files, +123/-22, 72 tests pass (zero breakage)
2026-08-01 14:52:07 +08:00
grossmj
b797981a65
test: add tests for unvalidated symlink creation in import_project 2026-07-29 18:48:31 +02:00
grossmj
a225622a6e
fix(import): unvalidated symlink creation in import_project 2026-07-29 18:10:52 +02:00
Sanjay Santhanam
00ac2c19bd Do not start nodes when deleting a project
Project.delete() calls open() to rebuild the internal data structures
needed for cleanup. open() schedules start_all() when the project has
auto_start enabled, so deleting an auto-start project actually launched
every node process, allocating ports and consuming resources, only for
close() to kill them moments later.

open() now takes an auto_start argument (default True, so normal opens
are unchanged) and delete() passes auto_start=False.

Fixes #2784
2026-07-25 20:00:43 -07:00
grossmj
0091f2c64a
Merge remote-tracking branch 'origin/3.1' into 3.1 2026-07-21 17:04:35 +02:00
grossmj
7487ec14e0
Merge branch '2.2' into 3.1
# Conflicts:
#	CHANGELOG
#	conf/gns3_server.conf
#	gns3server/compute/docker/__init__.py
#	gns3server/compute/docker/docker_vm.py
#	gns3server/compute/qemu/qemu_vm.py
#	gns3server/controller/import_project.py
#	gns3server/crash_report.py
#	gns3server/version.py
2026-07-21 17:03:00 +02:00
YueGuobin
ddf7d4ca60
fix(controller): refresh cloud/nat node interfaces from compute on GET
Cloud and NAT nodes need live host network interface data. Previously,
GET /projects/{project_id}/nodes/{node_id} returned cached properties
from creation time, so newly added host interfaces (e.g. kernel bridges
created by EthernetSwitch nodes) were invisible until the node was
deleted and recreated.

Now the controller fetches fresh data from the compute node before
returning the response, so host interface changes are reflected
immediately. Falls back to cached data if compute is unreachable.
2026-07-19 17:41:37 +08:00
YueGuobin
cdc27c37a7
fix(marker): clean inherited markers on def delete
delete_marker_definition removed the def but left the inherited global-*
copies on every link: it called stop_marker(), which rejects inherited
markers (409), and the ControllerError was swallowed as a warning. The
orphaned copies were in-memory only (_persist_markers filters inherited
markers) so a restart hid the symptom, but during a running session they
were undeletable via either the per-link or project API.

Add the same inherited=True bypass that update_marker already has, and
pass it from the def-delete fan-out so the copies are removed for real.
2026-07-16 00:39:52 +08:00
YueGuobin
98bcd5eddd
feat(marker): add highlight_duration render hint
Per-marker UI hint (milliseconds, ge=1) for how long the Web UI keeps a
marker highlighted after a match. Mirrors color: stored on the link,
persisted in the topology, inherited via project-level definitions, and
never sent to uBridge. Omitted = null = frontend uses its own default.

The default is intentionally NOT set in the schema: MarkerCreate also
backs PUT updates, so a schema default would make every partial update
silently reset the value. None means "not provided" (keep existing on
update / frontend decides on create).
2026-07-16 00:39:52 +08:00
YueGuobin
179f072c33
fix(marker): fix inheritance hook, topology load, update sync, and asdict
Bugs found via end-to-end testing of project-level marker definitions:

1. New links didn't inherit — apply_defs_to_new_link is async but was
   called without await in UDPLink.create().

2. Project load crashed — load_project passed marker_definitions to
   Project.__init__. Now popped in load_project and restored separately
   in Project.open() (it backs a read-only property).

3. PUT on a definition didn't sync to links — update_marker's guard
   rejected even the project-layer sync call. Added an `inherited`
   bypass flag used by update_marker_definition.

4. _ubridge_add_marker_filter raised re.PatternError — the name regex
   used (?i)(?!global) look-around, invalid in Python's re module.
   Dropped the prefix check there: "global-*" names are legitimate at
   the uBridge boundary (inherited definitions); forbidden only at the
   user-facing schema.

5. GET /links hid inherited markers — asdict()'s runtime branch used
   _persist_markers() (which filters inherited markers). Restored
   self._markers for the runtime branch; only the topology_dump branch
   filters (inherited markers are rebuilt from definitions on load).

6. Duplicate "already exists" warnings on project open — open() fanned
   out definitions to all links, but UDPLink.create() had already done
   so via its inheritance hook. Removed the redundant fan-out in open().
2026-07-16 00:39:52 +08:00
YueGuobin
31991fe359
feat(marker): add project-level marker definition inheritance
Project-level marker definitions fan out to every link (existing and new).
A definition is stored once on Project._marker_definitions; when applied to
a link the marker is named "global-{def_name}" — the "global" prefix was
pre-reserved in the schema, so inherited and per-link markers can never
collide, nor will their registry keys.

Key behavior:
- POST /projects/{pid}/marker-definitions → fan out to all existing links
- PUT  /projects/{pid}/marker-definitions/{name} → sync all inherited copies
- DELETE → remove every inherited copy from every link
- New links auto-inherit all active defs (hook in UDPLink.create)
- Per-link DELETE/PUT of a "global-*" marker is rejected (409)
- Inherited markers are NOT persisted in the topology; they are re-created
  from _marker_definitions on project load
- Compute side is untouched — the marker reaches uBridge via the existing
  start_marker→update→NIO→ubridge pipeline

Files:
- controller/project.py — _marker_definitions + CRUD + fanout + topology load
- controller/link.py — Link.inherit_marker() + asdict() filter
- controller/udp_link.py — guards on stop/update + create() inheritance hook
- controller/topology.py — persist marker_definitions in project topology
- schemas/controller/links.py — MarkerDefinitionCreate schema
- api/routes/controller/projects.py — REST endpoints (marker-definitions)
2026-07-16 00:39:52 +08:00
YueGuobin
d3527c00de
fix(marker): extend marker support to dynamips and cloud nodes
Dynamips needed its own NIO base class extended with a `_markers`
attribute and `markers` property (mirroring the existing `filters`
pattern), plus marker propagation from the controller-supplied NIO
settings through NIOUDP.create/update to the common destination
NIO that base_node._ubridge_apply_markers reads.

Cloud simply needed the missing _ubridge_apply_markers calls after
_ubridge_apply_filters in its two ubridge connection methods, same
pattern as the earlier docker fix.

Both types are now added back to _MARKER_CAPABLE_TYPES.
2026-07-16 00:39:51 +08:00
YueGuobin
e4be98984c
fix(marker): enable marker support for docker nodes
Docker's _connect_nio and adapter_update_nio_binding applied packet
filters but never called _ubridge_apply_markers, so markers silently
did nothing on Docker links despite docker being in the allowlist.
Add the missing calls (same pattern as the IOU fix).

Also narrow _MARKER_CAPABLE_TYPES to the four types that actually
implement marker support — vpcs, qemu, docker, iou — removing
dynamips, virtualbox, vmware, and cloud which have no marker pathway
and would silently fail when selected as the capture side.
2026-07-16 00:39:51 +08:00
YueGuobin
f2360f85fc
feat(marker): add project-level marker aggregation endpoint
Add a read-only `markers` property on Project that flattens every link's
markers into a single dict keyed by "{link_id}/{name}", each entry
carrying the parent link_id and capture-side node_id. Expose it via
GET /projects/{pid}/markers (Project.Audit) so the frontend can fetch all
markers in one round-trip instead of enumerating links first.

Also surface a marker count in project.stats().
2026-07-16 00:39:51 +08:00
YueGuobin
c62b9b0283
refactor(marker): converge to filter single-path model, remove dual-apply endpoints
Markers now follow exactly the same apply pattern as packet filters:
state lives in Link._markers, application goes through NIO
(update() -> PUT /nio -> _ubridge_apply_markers). The former
immediate-apply REST endpoints (/markers/start, /markers/stop on
the compute side) and the per-node start_marker/stop_marker methods
are removed — they were a legacy of the original capture-inspired
design and have been superseded by the NIO flow.

Changes:
- controller/udp_link: start_marker/stop_marker/update_marker now
  set _markers state + call self.update() (mirrors update_filters).
  Removed _marker_capture_nodes runtime dict and its helpers.
- controller/project: _create_link_from_topology_data restores
  _markers directly from persisted data (with BPF validation,
  like filter reload). No long calls start_marker during load.
- compute: _ubridge_apply_markers swallows BPF compile errors
  (warn+skip), matching _ubridge_apply_filters behaviour so a
  single bad expression cannot break link creation / node restart.
- Removed: /markers/start,stop endpoints (6 handlers across
  vpcs/qemu/docker route files), node start_marker/stop_marker
  methods (3 VM files), _ubridge_delete_marker_filter,
  _marker_capture_nodes, MarkerDelete schema.

Net: ~280 lines of dead code removed; marker and packet filter now
share a single, unified apply path via the NIO.
2026-07-16 00:39:51 +08:00
YueGuobin
88c03d1431
fix(marker): restore markers from topology on project load
_create_link_from_topology_data now restores traffic-insight markers
(mirroring how filters are restored via update_filters), so markers —
including their color — survive project close/reopen and server restart.

Guard start_marker's uBridge POST with 'if self._created' (exactly as
update_filters guards its update() call): during project load the link
is not yet created, so only _markers state is recorded and the marker
is applied once via the NIO flow in create()/_ubridge_apply_markers —
no double application.
2026-07-16 00:39:51 +08:00
YueGuobin
3f15ac38f6
feat(marker): add optional color field to marker data model
Add a user-chosen hex color (e.g. '#ff5722') to each marker, stored in
Link._markers alongside bpf/tag/enabled. The color:

- Is serialized with the link (asdict) and persisted in the topology, so
  it survives project reload and is consistent across devices.
- Never reaches uBridge (compute ignores it) — it is purely a Web UI
  concern.
- Can be updated independently of the BPF (color-only changes skip the
  ubridge round-trip).

MarkerCreate schema gains color: Optional[str]; start_marker and
update_marker carry it through; REST create/update endpoints pass it.

The marker.match WebSocket event stays lean (no color) — the Web UI maps
filter -> color from the link state it already holds.
2026-07-16 00:39:51 +08:00
YueGuobin
b8f49696cc
fix(marker): use allowlist in _choose_marker_side, align with _get_filter_node
Replace the complex tier-based fallback (which leaked into unsupported
switch types on the final tiers) with a simple allowlist of
_MARKER_CAPABLE_TYPES, mirroring _get_filter_node in link.py.

Also accept stopped nodes — like packet filters, markers are stored on
the NIO and applied when the node starts via _ubridge_apply_markers.
2026-07-16 00:39:51 +08:00
YueGuobin
239f42f5aa
fix(marker): stop auto-deleting markers on node stop in node_updated
Markers are persistent link-scoped features (like packet filters), not
transient runtime features (like capture).  Tearing them down on every
node stop breaks stop/start restore: the NIO-based re-application in
_ubridge_apply_markers works correctly on restart, but node_updated
was prematurely calling stop_marker which erased _markers state and
dumped an empty topology.

Now markers survive node stop transparently — the ubridge filter dies
with the process but is re-added from nio.markers in
add_ubridge_udp_connection when the node starts again, mirroring
exactly how packet filters behave.
2026-07-16 00:39:51 +08:00
YueGuobin
7c90fa9e64
fix(marker): ride markers on NIO so they survive node restart and project reload
Mirror the packet-filter lifecycle: marker specs now live on the NIO
(next to filters), ride in link_data from controller to compute on
every NIO create/update, and are reapplied by _ubridge_apply_markers
in add_ubridge_udp_connection (bridge creation / node restart) and
update_ubridge_udp_connection (NIO update — following the preceding
reset_packet_filters so markers survive filter changes).

Changes:
- NIO / NIOUDP: _markers property + asdict
- schemas/compute/nios.py: UDPNIO.markers field
- base_manager.create_nio: nio.markers from settings
- PUT /nio routes (vpcs/qemu/docker): nio.markers update
- base_node: _ubridge_apply_markers(bridge_name, nio) iterates
  nio.markers, computes pcap path, calls _ubridge_add_marker_filter
  + MarkerManager.register; called after _ubridge_apply_filters
- controller udp_link: _get_node_markers + _markers_for_node (route
  by capture_node_id); markers in create() and update() link_data
- /markers/start,stop endpoints: mirror spec onto nio.markers so
  the marker survives a subsequent node stop/start without a PUT
- tests: add markers field to NIO data expectations

This covers:
- Node restart: NIO persists, add_ubridge_udp_connection re-applies
- Filter update: reset wipes markers, _ubridge_apply_markers re-adds
- Project reload: create() carries markers in link_data → create_nio
- Immediate create: endpoint sets nio.markers immediately
2026-07-16 00:39:51 +08:00
YueGuobin
88465e872c
fix(marker): add _choose_marker_side to exclude non-uBridge node types
_choose_capture_side prefers BUILTIN_NODE_TYPES which includes
ethernet_switch/ethernet_hub/NAT etc. — nodes that do not use uBridge
and therefore have no bridge for a marker filter to attach to.

Add _choose_marker_side with a _MARKER_UNSUPPORTED_TYPES blocklist
so marker capture-side selection skips NIO-only switches and picks
the uBridge-backed node (VPCS/QEMU/Docker/etc.) instead.

Verified end-to-end: PC1↔Switch↔PC2 links with arp markers correctly
pick VPCS nodes; MARK signals flow through to WebSocket and pcaps
are written.
2026-07-16 00:39:51 +08:00
YueGuobin
d2e9823e6c
feat: implement traffic insight marker backend (ubridge mark filter)
Add compute-side marker subsystem that receives ubridge UDP MARK signals
and forwards them as project-scoped notifications to the web UI for
real-time traffic coloring. Matched packets are always saved to per-link
pcaps for future replay.

Key components:
- gns3server/compute/marker/: MarkerManager (singleton, UDP listener +
  O(1) registry keyed by (node_id, filter_name)) + MarkerListener
  (DatagramProtocol parsing MARK lines per ubridge integration contract)
- gns3server/compute/base_node.py: marker sink/node config at ubridge
  startup; shared _ubridge_add_marker_filter / _ubridge_delete_marker_filter
- Per-node start_marker/stop_marker: VPCS (VPCS-{id}), QEMU
  (QEMU-{id}-{adapter}), Docker (bridge{adapter})
- Compute REST /markers/start + /markers/stop (vpcs/qemu/docker route files)
- Controller Link._markers state + UDPLink.start_marker/stop_marker/
  update_marker (mirror capture pattern: BPF validation, _choose_capture_side,
  node.post forwarding, topology persistence)
- Controller REST GET/POST/DELETE/PUT /v3/projects/{p}/links/{l}/markers
- Config: marker_listen_host / marker_listen_port in ServerSettings
- Signal routing: creation-time registry O(1) lookup, no node-table scan;
  project-scoped WS stream (not global); event payload always carries
  project_id for frontend scoping

Tests: 14 unit tests (registry, listener parsing, UDP round-trip);
562 existing tests pass with zero regressions.
2026-07-16 00:39:51 +08:00
grossmj
1986bce538
Only set IOU images to be executable when importing images 2026-07-15 09:43:34 +02:00
grossmj
c528d2cd95
fix(import): project import does not move images symlinks into place 2026-07-13 18:19:52 +02:00
grossmj
ea09cc2724
fix: gns3-server crashes on startup if "Open this project in the background" is active but there is a problem with that project 2026-07-12 13:46:12 +02:00
grossmj
e3e0a511b6
Handle HTTPNotFound exception when retrieving compute status 2026-07-12 11:08:13 +02:00