36 Commits

Author SHA1 Message Date
YueGuobin
187201a08c
feat(controller): offer QEMU the kernel datapath, gated per compute
Kernel eligibility becomes "both endpoints on one compute, each able to
anchor": Docker always (adapters are born as veth pairs), QEMU when that
compute's uBridge can create the persistent TAPs its adapters anchor on. The
question is asked per compute, because the link needs both ends anchored --
so a mixed or half-upgraded deployment degrades to the relay instead of
failing an attach.

probe_tap_support() creates and deletes one throwaway persistent TAP through a
scratch uBridge, cached by binary identity like the tc report; /capabilities
exposes `ubridge_tap` (absent = unknown, which keeps links on the relay).

The compute route for a QEMU adapter NIO took UDPNIO only, so a kernel link
was rejected with 422: its body and response now accept
Union[UDPNIO, BridgeNIO] exactly like Docker's.

Stale "Docker-to-Docker" wording in the config, link and MCP descriptions now
talks about anchors, and docs/features/qemu-kernel-datapath.md documents the
datapath -- verified end-to-end on a real instance: 32/32 kernel-datapath
checks and 29/29 on the relay variant.
2026-10-06 00:21:08 +08:00
Mark Paronyan
306d8fbde4
refactor: remove unused imports 2026-10-03 19:43:58 +03:00
Mark Paronyan
634189d443 style: sort imports in api and utils modules 2026-10-02 12:43:26 +03:00
Mark Paronyan
b8a51fdefc
fix(typing): resolve mypy errors in api.routes.compute.qemu_nodes 2026-09-28 21:47:36 +03:00
Mark Paronyan
a0450b6bce
refactor: ruff autofixes 2026-09-27 15:55:10 +03:00
Cristi
da16dc9874 fix(qemu): Fixed missing image replacement for linked-clone nodes 2026-09-18 23:58:49 +03:00
YueGuobin
f9f1a4d4d8
log: demote all per-node lifecycle INFO logs to DEBUG
Extend the docker_vm / base_node demotion to the remaining node types
and supporting layers:

  qemu_vm: MAC, adapters, disk image, RAM, priority, NIO added, created
  iou_vm:  application ID, adapters, serial, image, RAM, NIO added
  dynamips router: created, adapter, RAM, NVRAM, IOS, idle-PC, disk,
    MAC, NIO bound; hypervisor create/start/connect; nio_udp created
  builtin: ethernet_switch/hub, cloud, nat — created, NIO bound
  ubridge: hypervisor start/connect

At multi-node scale these per-node lines flood the log. Only the
project-open progress summary (loaded N nodes / creating N links)
now remains at INFO alongside genuinely exceptional events.
2026-08-11 01:05:53 +08:00
grossmj
29c1b04078
fix: add missing HTTPException import 2026-08-05 21:55:23 +02:00
YueGuobin
75f278228b
marker: drop deleted marker from port NIO cache to stop empty pcap on restart
Deleting a marker while its node was stopped, then starting the node, recreated
an empty pcap. Root cause: delete_marker_capture removed the uBridge filter and
the pcap file but not the marker spec cached on the port NIO (nio.markers) —
the data source _ubridge_apply_markers reads on node start. The stale spec
reinstalled the marker when uBridge came up.

This was a regression from switching stop_marker off update() (which re-sent
the NIO and implicitly refreshed nio.markers) to the fine-grained
node.delete(/markers/{name}) path.

Fix: make the delete port-aware so the compute can locate the NIO. The DELETE
marker route becomes /adapters/{a}/ports/{p}/markers/{name} across all six
node types; the handler resolves the NIO via get_nio and passes it to
delete_marker_capture, which now pops the marker from nio.markers. get_nio
works regardless of uBridge state, so the stopped-node case is covered. The
controller's stop_marker targets the capture side's adapter/port.
2026-08-05 00:08:12 +08:00
YueGuobin
caec71aa71
marker: fine-grained filter ops, clean pcap on remove
Deleting or updating a marker no longer triggers a full NIO reapply
(reset_packet_filters + re-add), which closed/reopened every sibling
marker's pcap via uBridge. Instead operate on single filters:

- stop_marker: bridge delete_packet_filter + unlink the pcap (works with
  the node stopped; filter removal is skipped, the file is still deleted).
- update_marker: bpf/tag/direction → rebuild just that filter (delete + add);
  enabled → instant toggle; color/highlight_duration → stored only.
- compute delete_marker_capture / rebuild_marker_filter + per-node routes
  (DELETE /markers/{name}, PUT /markers/{name}/rebuild) + MarkerRebuild schema.

IOU overrides _ubridge_delete_marker_filter for iol_bridge; rebuild reuses
the already-overridden add/delete/set, so IOU needs no rebuild override.
2026-08-04 21:35:31 +08:00
YueGuobin
ff907da5f6
marker: key _marker_filter_bridges by (name, link_id) so multi-link nodes toggle every copy
The _marker_filter_bridges dict was keyed by marker name alone, so when one
node hosted the same filter name on several links (IUOL-BRIDGE per node with
many bays/units, or a multi-interface router), successive apply calls
overwrote earlier entries. pause_marker_definition then toggled only the last
recorded bridge/location — other copies stayed active and kept emitting.

Key by (name, link_id) so each copy is independent, and iterate all matching
entries in _ubridge_set_marker_filter_state (both generic bridge and IOU
iol_bridge override). Toggle route existence checks also iterate matching
names. Tests updated.
2026-08-02 23:48:24 +08:00
YueGuobin
34b644a548
marker: make per-filter toggle fall back to NIO rebuild when the marker isn't installed
Toggle routes silently no-opped when _marker_filter_bridges lacked the filter
name, so update_marker's enabled-only short-circuit succeeded without toggling
uBridge — the controller-layer enabled was set but the uBridge filter stayed on
and kept emitting signals. Now the toggle routes raise HTTPException 404
(FastAPI handles it directly, no ERROR log); the controller's except catches it
and falls back to self.update() (NIO rebuild, which applies the marker + off).
2026-08-02 23:41:43 +08:00
YueGuobin
3d06c4e22f
marker: drive uBridge enabled/pause/resume in real time
Wire gns3-server to uBridge's real-time marker controls (contract
../ubridge/doc/gns3server-integration.md §3.2), in three layers:

A. enabled reaches uBridge — _markers_for_node no longer drops disabled
   markers controller-side; it carries `enabled` in the NIO spec. apply
   installs every marker then issues `enable_packet_filter … off` for the
   disabled ones (base_node bridge / iou iol_bridge; old-ubridge errors
   downgrade to a warning so a toggle can't break link create).

B. instant per-filter toggle — apply records name→bridge so a new
   `_ubridge_set_marker_filter_state` can flip a running filter with
   `enable_packet_filter on|off` (iou overrides for iol_bridge + bay/unit).
   Each marker-capable node type gains PUT /markers/{name}; update_marker
   short-circuits to it when only `enabled` changes (no NIO rebuild, no pcap
   flush), falling back to reset+reapply if the route is unavailable.

C. global pause/resume — `_ubridge_marker_pause/resume` send `marker pause`
   / `marker resume` direct to the hypervisor (pause stops signal+pcap,
   resume instant, sink retained). Six node-type routes add POST
   /markers/pause|resume; project.pause_all/resume_all_markers fan out to
   each capture node (deduped, best-effort); REST exposes
   POST /projects/{id}/markers/pause|resume.

Toggling enabled and pause/resume are now both instant — only marker create
or a bpf change still go through reset+reapply.
2026-08-02 16:19:20 +08:00
YueGuobin
c62b9b0283
refactor(marker): converge to filter single-path model, remove dual-apply endpoints
Markers now follow exactly the same apply pattern as packet filters:
state lives in Link._markers, application goes through NIO
(update() -> PUT /nio -> _ubridge_apply_markers). The former
immediate-apply REST endpoints (/markers/start, /markers/stop on
the compute side) and the per-node start_marker/stop_marker methods
are removed — they were a legacy of the original capture-inspired
design and have been superseded by the NIO flow.

Changes:
- controller/udp_link: start_marker/stop_marker/update_marker now
  set _markers state + call self.update() (mirrors update_filters).
  Removed _marker_capture_nodes runtime dict and its helpers.
- controller/project: _create_link_from_topology_data restores
  _markers directly from persisted data (with BPF validation,
  like filter reload). No long calls start_marker during load.
- compute: _ubridge_apply_markers swallows BPF compile errors
  (warn+skip), matching _ubridge_apply_filters behaviour so a
  single bad expression cannot break link creation / node restart.
- Removed: /markers/start,stop endpoints (6 handlers across
  vpcs/qemu/docker route files), node start_marker/stop_marker
  methods (3 VM files), _ubridge_delete_marker_filter,
  _marker_capture_nodes, MarkerDelete schema.

Net: ~280 lines of dead code removed; marker and packet filter now
share a single, unified apply path via the NIO.
2026-07-16 00:39:51 +08:00
YueGuobin
7c90fa9e64
fix(marker): ride markers on NIO so they survive node restart and project reload
Mirror the packet-filter lifecycle: marker specs now live on the NIO
(next to filters), ride in link_data from controller to compute on
every NIO create/update, and are reapplied by _ubridge_apply_markers
in add_ubridge_udp_connection (bridge creation / node restart) and
update_ubridge_udp_connection (NIO update — following the preceding
reset_packet_filters so markers survive filter changes).

Changes:
- NIO / NIOUDP: _markers property + asdict
- schemas/compute/nios.py: UDPNIO.markers field
- base_manager.create_nio: nio.markers from settings
- PUT /nio routes (vpcs/qemu/docker): nio.markers update
- base_node: _ubridge_apply_markers(bridge_name, nio) iterates
  nio.markers, computes pcap path, calls _ubridge_add_marker_filter
  + MarkerManager.register; called after _ubridge_apply_filters
- controller udp_link: _get_node_markers + _markers_for_node (route
  by capture_node_id); markers in create() and update() link_data
- /markers/start,stop endpoints: mirror spec onto nio.markers so
  the marker survives a subsequent node stop/start without a PUT
- tests: add markers field to NIO data expectations

This covers:
- Node restart: NIO persists, add_ubridge_udp_connection re-applies
- Filter update: reset wipes markers, _ubridge_apply_markers re-adds
- Project reload: create() carries markers in link_data → create_nio
- Immediate create: endpoint sets nio.markers immediately
2026-07-16 00:39:51 +08:00
YueGuobin
d2e9823e6c
feat: implement traffic insight marker backend (ubridge mark filter)
Add compute-side marker subsystem that receives ubridge UDP MARK signals
and forwards them as project-scoped notifications to the web UI for
real-time traffic coloring. Matched packets are always saved to per-link
pcaps for future replay.

Key components:
- gns3server/compute/marker/: MarkerManager (singleton, UDP listener +
  O(1) registry keyed by (node_id, filter_name)) + MarkerListener
  (DatagramProtocol parsing MARK lines per ubridge integration contract)
- gns3server/compute/base_node.py: marker sink/node config at ubridge
  startup; shared _ubridge_add_marker_filter / _ubridge_delete_marker_filter
- Per-node start_marker/stop_marker: VPCS (VPCS-{id}), QEMU
  (QEMU-{id}-{adapter}), Docker (bridge{adapter})
- Compute REST /markers/start + /markers/stop (vpcs/qemu/docker route files)
- Controller Link._markers state + UDPLink.start_marker/stop_marker/
  update_marker (mirror capture pattern: BPF validation, _choose_capture_side,
  node.post forwarding, topology persistence)
- Controller REST GET/POST/DELETE/PUT /v3/projects/{p}/links/{l}/markers
- Config: marker_listen_host / marker_listen_port in ServerSettings
- Signal routing: creation-time registry O(1) lookup, no node-table scan;
  project-scoped WS stream (not global); event payload always carries
  project_id for frontend scoping

Tests: 14 unit tests (registry, listener parsing, UDP round-trip);
562 existing tests pass with zero regressions.
2026-07-16 00:39:51 +08:00
YueGuobin
64d3c02d42 refactor(compute): remove SPICE WebSocket console support
Remove all SPICE WebSocket-related code due to frontend dependency issues
with spice-html5 library (missing RSAKey/BigInteger implementations).

Changes:
- Remove start_spice_websocket_console() from BaseNode
- Remove SPICE WebSocket endpoints from QEMU and Docker compute APIs
- Remove SPICE WebSocket proxy endpoint from controller API
- Remove WebSocket subprotocol handling from authentication layers
- Remove SPICE documentation

The SPICE console type remains functional for direct connections,
but WebSocket proxy support has been removed.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-03-17 01:15:26 +08:00
YueGuobin
c98f802362 feat(compute): add SPICE WebSocket console support
Add WebSocket-based console support for SPICE protocol, enabling
   browser-based graphical console access with enhanced features
   like clipboard sharing, USB redirection, and audio streaming.

   Changes:
   - Add start_spice_websocket_console() method in BaseNode for SPICE
     WebSocket to TCP bridging with bidirectional binary forwarding
   - Add /console/spice WebSocket endpoints in QEMU and Docker compute APIs
   - Add /console/spice WebSocket proxy endpoint in controller API
   - Add comprehensive API documentation in docs/features/

   Supported console types:
   - spice: Basic SPICE protocol support
   - spice+agent: SPICE with spice-vdagent for enhanced features

   Architecture:
   - Browser WebSocket → Controller (JWT + RBAC) → Compute (Basic Auth)
   - No external websockify processes required
   - Consistent with existing VNC WebSocket implementation
2026-03-16 22:21:25 +08:00
YueGuobin
ba92b405da feat: add VNC WebSocket console support for Docker and QEMU nodes
Add VNC console WebSocket endpoints for Docker and QEMU nodes:
- Add /console/vnc WebSocket endpoint to compute API (Docker & QEMU)
- Add /console/vnc WebSocket endpoint to controller API
- Implement start_vnc_websocket_console() in BaseNode
- Forward VNC WebSocket traffic between controller and compute layers

The implementation provides bidirectional WebSocket to TCP forwarding
for VNC protocol (RFB) connections, allowing browser-based VNC console
access to containers and VMs.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-03-16 16:40:00 +08:00
grossmj
f41cc24383
Fix for duplicating Qemu nodes 2026-03-12 13:24:30 +08:00
grossmj
5e3ca29c58
Fix updating packet filters 2026-02-21 21:42:22 +08:00
grossmj
44c066ac68 Fix compute authentication for websocket endpoints 2023-10-22 12:44:39 +10:00
grossmj
fc6aeb715a Remove explicit Response for endpoints returning HTTP 204 status code 2022-07-16 00:12:18 +02:00
grossmj
e50bed5bee Support delete Qemu disk image from API
Return the real disk image name in the 'hdx_disk_image_backed' property for Qemu VMs
2022-04-14 17:01:54 +07:00
grossmj
fda2a37b98 Handle creating Qemu disk images and resizing 2022-04-07 16:21:47 +08:00
grossmj
bc36d95060 Finalize image management refactoring and auto install appliance if possible 2021-10-10 17:35:11 +10:30
grossmj
ce55ec73a4 Fix exception with endpoints returning HTTP status code 204. Fixes #1891 2021-08-10 21:53:21 +09:30
grossmj
3ae7783360 Fix "-machine accel=tcg" check 2021-08-10 15:41:05 +09:30
grossmj
9404c00411 Complete type annotations for API endpoints. 2021-04-18 15:40:38 +09:30
grossmj
cefab8d362 Rename __json__() to asdict() 2021-04-17 23:34:28 +09:30
grossmj
c021e21309 Use black with -l 120 param. 2021-04-13 18:46:50 +09:30
grossmj
f928738bd5 Use pyupgrade with --py36-plus param. 2021-04-13 18:37:58 +09:30
grossmj
a9eb035b7d Move error responses to API routers. 2021-04-13 16:19:56 +09:30
grossmj
30ebae207f Use Pydantic to validate the server config file. 2021-04-12 17:02:23 +09:30
grossmj
bf7cf862af Refactor tests and start work on database integration. 2020-12-02 18:39:08 +10:30
grossmj
c043830e3f Move endpoints to routes & preparations to use a database. 2020-11-19 15:21:03 +10:30