Review-driven session/transport fixes (each reproduced live against
sharkd 4.6.7 before fixing):
- raise the RPC stream limit to 16 MB: a full 1000-row frames page
measures ~190 KB against the 64 KB StreamReader default, which failed
the request with a 500 and desynchronized the resident session; a
line-over-limit ValueError is now treated as a transport failure
- verify JSON-RPC reply ids: a timed-out request's late reply was
served as the next request's answer; timeouts, dead pipes, malformed
and stale replies now kill the session for good instead
- make check-spawn atomic under one manager lock: concurrent requests
for the same pcap double-spawned sharkd and leaked the loser (process
plus /tmp scratch copy) forever
- refcount sessions and evict idle only (LRU, cap raised 8 -> 16): a
tag with more sources than the cap respawned every source on every
request, and concurrent requests could get their session killed
mid-RPC (spurious 502)
- map FilterError to sharkd's filter rejection (-13002) only; other
engine failures with a filter set are 502, not a client 400
- detail: accept an optional frame_number to disambiguate
same-microsecond frames (ts is not unique within a pcap); drop the
-8003 -> 404 mapping (the range is validated locally, engine errors
are real faults); a failed hex read is a 404 instead of "hex": null
- a pcap deleted mid-request is a 404, not a 500; the pcap-sized
scratch copy runs off the event loop; server shutdown kills every
resident session and drops its scratch directory
- pin the packet-list layout through scratch-HOME Wireshark
preferences: the column indexes are a contract the server owns
(protocol-level column negotiation is rejected by sharkd 4.6.x)
Range contract change (WebUI moved to an always-flat list): the merged
frame list is returned in full, deliberately uncapped - truncated and
per-second buckets are removed, frame_count always equals
len(frames), and rendering cost is the client's concern (the window
endpoint remains the incremental path).
Narrows the merged frame stream to one capture source BEFORE counting,
slicing and bucketing (frame_count / frames | buckets all recomputed on
the narrowed set), AND-composing with the display filter. A pure
identity filter applied before any engine work — only the selected
link's pcap gets a sharkd pass, so link+filter is cheaper than filter
alone.
Two boundaries by contract with the Web UI:
- sources[] stays the tag's stable inventory: every capture source
listed with engine-free TOTAL counts, unaffected by link/filter — a
source dropdown must not shrink when the view narrows (this also
settles sources[].count on total counts rather than post-filter
matches, which no spec ever required)
- an unknown link_id matches nothing: frame_count 0, start null, empty
frames/buckets — the same shape as a zero-match display filter,
deliberately not a 404; an empty link param is treated as absent
sharkd (the Wireshark daemon) is now the single decode engine — the
tshark/PDML path is gone, and without sharkd every replay endpoint
returns 501 (no degraded mode: one engine, one rendering shape for the
Web UI).
- Frame entries gain packet-list columns from sharkd's frames RPC:
src/dst/proto/info plus the Wireshark coloring hints bg/fg
- range and frames accept ?filter=<display filter>, applied before
counting and slicing; invalid expressions are 400 carrying sharkd's
original text; filters travel as single argv-style elements, capped
at 2000 chars; filtered frames keep their original pcap frame numbers
- frame detail returns sharkd's protocol tree with keys renamed into
the REST contract (element/label/name/filter_expr/pos+size/expert/
generated/children): a census-verified closed key set, values
untouched, unknown keys passed through verbatim, Wireshark-internal
hf ids dropped. filter_expr gives the UI click-to-filter; pos/size
drives hex highlighting (hex still read straight from the pcap)
- one resident 'sharkd -' session per source pcap: lazy spawn, /tmp
scratch copy + scratch HOME (hardened profiles), per-request
(mtime,size) validation with respawn, LRU bound, per-session lock,
per-RPC timeout, bounded close
Timeline backbone (gate, record-header scan, merge ordering, canonical
ts strings, hex reads) stays plain Python — identity and ordering never
depend on the engine.
Markers on different links sharing a tag form one distributed capture
session. Once every marker under the tag is paused (409 otherwise), three
read-only endpoints replay it:
- GET .../markers/tags/{tag}/replay/range merges the per-marker pcaps by
scanning 16-byte record headers only (no tshark) into a timestamp-
ordered frame list (per-second buckets above a 5000-frame cap)
- GET .../replay/frames?ts=&window_ms= returns frames in [T, T+window];
an empty window is a normal empty array
- GET .../replay/frame/detail lazily decodes one frame the user opened:
raw bytes for the hex view read straight from the pcap, protocol tree
from 'tshark -T pdml' mapped isomorphically to JSON (every attribute
survives, values stay strings). tshark reads a /tmp scratch copy with a
scratch HOME — hardened profiles deny it the project directory.
Sort key is (ts, source file, frame number): ts is not unique across a
merge. The ts parameter round-trips as the exact string from the frame
list. Round-trip tests pin the PDML→JSON fidelity (element count and
attribute coverage).
GET /v3/projects/{project_id}/gns3file returns the raw .gns3 file as
application/json so the WebUI can render topology thumbnails without
opening the project. Adds a public Project.topology_file property.
High-frequency marker.matches shared the single project notification queue with topology events (node.*/link.*), causing head-of-line blocking. Add a separate marker channel: Notification.project_marker_queue/marker_emit, dispatch routes marker.* off the main project queue, plus a new WS /{project_id}/notifications/markers/ws endpoint. Fully migrated (the main project WS no longer carries marker.match); marker listeners are independent of project auto_close. Compute side unchanged.
Markers now work on serial links (Cisco HDLC / PPP / Frame Relay / ATM), not just Ethernet. A marker carries a data_link_type (default DLT_EN10MB); at the uBridge boundary it becomes the 'mark ... linktype <dlt>' keyword so the BPF compiles and the pcap is written with the matching link-layer.
- MarkerCreate / MarkerDefinitionCreate gain data_link_type (default DLT_EN10MB). Per-link it is create-only; definitions are updatable (a change re-fans-out).
- base_node._marker_linktype() normalizes the GNS3 DLT name (strip DLT_, uppercase, None for EN10MB). Single source is SerialPort.data_link_types, so Cisco PPP -> PPP_SERIAL (50), matching the capture path -- no second mapping table.
- _ubridge_add_marker_filter (generic) and the IOU marker loop append 'linktype <dlt>'.
- Definition fan-out branches on link_type in inherit_marker: Ethernet is always EN10MB; a serial link uses the definition's WAN encapsulation, or is SKIPPED when none was chosen (an EN10MB pcap on serial is undecodable). One definition covers a mixed topology.
- MCP marker_definition exposes data_link_type (None = not forwarded).
- No uBridge rebuild on a data_link_type change -- only that one marker's filter is swapped (delete + re-add), mirroring a BPF change; reset_packet_filters preserves sibling mark filters.
Requires the uBridge build with 'mark ... linktype' support.
The project-wide mute (POST /markers/pause|resume + _markers_paused) paused
every marker with one button. The actual need is per-rule control: pause one
definition and toggle only its inherited global-{name} copies across all links.
- Drop project-level: _markers_paused (init/asdict/load/start_all), the
pause_all/resume_all_markers methods, and the /markers/pause|resume routes.
- Add per-definition: a persisted `paused` flag on each definition;
pause/resume_marker_definition fan out update_marker(enabled) to every
global-{name} copy — instant, via the existing enable_packet_filter toggle
(no NIO rebuild, pcap/emitted preserved). New links inherit a paused
definition already off (inherit_marker passes enabled=not paused).
- start_marker takes an enabled kwarg; update_marker's enabled-only short-circuit
now also covers inherited copies so def pause/resume is instant.
- Routes: POST /marker-definitions/{name}/pause|resume.
- Docs + tests updated.
Wire gns3-server to uBridge's real-time marker controls (contract
../ubridge/doc/gns3server-integration.md §3.2), in three layers:
A. enabled reaches uBridge — _markers_for_node no longer drops disabled
markers controller-side; it carries `enabled` in the NIO spec. apply
installs every marker then issues `enable_packet_filter … off` for the
disabled ones (base_node bridge / iou iol_bridge; old-ubridge errors
downgrade to a warning so a toggle can't break link create).
B. instant per-filter toggle — apply records name→bridge so a new
`_ubridge_set_marker_filter_state` can flip a running filter with
`enable_packet_filter on|off` (iou overrides for iol_bridge + bay/unit).
Each marker-capable node type gains PUT /markers/{name}; update_marker
short-circuits to it when only `enabled` changes (no NIO rebuild, no pcap
flush), falling back to reset+reapply if the route is unavailable.
C. global pause/resume — `_ubridge_marker_pause/resume` send `marker pause`
/ `marker resume` direct to the hypervisor (pause stops signal+pcap,
resume instant, sink retained). Six node-type routes add POST
/markers/pause|resume; project.pause_all/resume_all_markers fan out to
each capture node (deduped, best-effort); REST exposes
POST /projects/{id}/markers/pause|resume.
Toggling enabled and pause/resume are now both instant — only marker create
or a bpf change still go through reset+reapply.
direction was settable but not clearable: once a marker or project-level
definition had direction=tx/rx, no update path could return it to "both
directions", and the definition-sync fan-out silently kept stale values on
every inherited copy.
Introduce a _UNSET sentinel (link.py) distinct from None so updaters can
tell "caller omitted direction" (preserve) from "caller passed None"
(clear). Thread it through UDPLink.update_marker and
Project.update_marker_definition; the two REST routes use Pydantic v2
model_fields_set to translate an explicit JSON null into the sentinel.
MCP follows with a "both" token: link_marker / marker_definition handlers
map direction="both" to a null in the REST body (tri-state: omit=preserve,
tx/rx=set, both=clear), and the tool descriptions/docstrings document it.
Backward compatible: omitting direction or passing tx/rx behaves exactly
as before; only an explicit null / "both" clears.
Direction field in marker.match events
=======================================
Read ubridge dir=<tx|rx> from MARK signal datagrams and forward it
as a "dir" key in the marker.match notification event. The field is
additive -- older ubridge builds omit it and the parser leaves it null,
so consumers fall back to undirected rendering with no version coupling.
Semantics are relative to the capture node (the signal's node=<id>):
tx = capture node is sending (ingressed device-side NIO), rx = it is
receiving (ingressed link-side NIO).
Per-marker direction filter (opt-in, server-side pipeline)
==========================================================
Add a direction field to MarkerCreate and MarkerDefinitionCreate
schemas ("tx" | "rx" | null). Plumb it through the full pipeline:
Schema -> controller (start_marker/update_marker, marker_entry,
_markers_for_node, update_marker_definition sync)
-> REST/MCP handlers -> compute _ubridge_add_marker_filter +
IOU _ubridge_apply_markers -> bridge add_packet_filter dir <tx|rx>
When set, ubridge only fires the mark handler (signal + pcap) for
packets matching the chosen direction. null (default/legacy) = both
directions -- zero behavioural change for existing markers.
Docs and tests
==============
- docs/features/marker-traffic-insight.md: signal format updated,
new Direction section with NIO mapping, arrow mapping, and additive
compatibility note.
- tests/compute/marker/test_marker_manager.py: 3 new parser tests
(dir tx/rx/absent) plus existing test extended to assert dir=None.
13 files, +123/-22, 72 tests pass (zero breakage)
The global-prefix reservation belongs on the create path, where it keeps
user-chosen names disjoint from inherited global-{name} markers. It was
implemented as a field_validator on MarkerCreate / MarkerDefinitionCreate,
which also back the PUT update bodies — so updating an inherited marker
while echoing its name in the body tripped the validator with a generic
422 before the controller could return the actionable 409 ("inherited,
use the marker-definitions API").
Drop the schema validators and enforce the prefix in the two create routes
(ControllerError -> 409). PUT no longer validates the body name (ignored
anyway — the target is the {name} path param), so editing an inherited
marker now reaches the controller's inheritance guard and returns the
clear 409. The name-format regex stays on the schema (still 422).
Per-marker UI hint (milliseconds, ge=1) for how long the Web UI keeps a
marker highlighted after a match. Mirrors color: stored on the link,
persisted in the topology, inherited via project-level definitions, and
never sent to uBridge. Omitted = null = frontend uses its own default.
The default is intentionally NOT set in the schema: MarkerCreate also
backs PUT updates, so a schema default would make every partial update
silently reset the value. None means "not provided" (keep existing on
update / frontend decides on create).
Project-level marker definitions fan out to every link (existing and new).
A definition is stored once on Project._marker_definitions; when applied to
a link the marker is named "global-{def_name}" — the "global" prefix was
pre-reserved in the schema, so inherited and per-link markers can never
collide, nor will their registry keys.
Key behavior:
- POST /projects/{pid}/marker-definitions → fan out to all existing links
- PUT /projects/{pid}/marker-definitions/{name} → sync all inherited copies
- DELETE → remove every inherited copy from every link
- New links auto-inherit all active defs (hook in UDPLink.create)
- Per-link DELETE/PUT of a "global-*" marker is rejected (409)
- Inherited markers are NOT persisted in the topology; they are re-created
from _marker_definitions on project load
- Compute side is untouched — the marker reaches uBridge via the existing
start_marker→update→NIO→ubridge pipeline
Files:
- controller/project.py — _marker_definitions + CRUD + fanout + topology load
- controller/link.py — Link.inherit_marker() + asdict() filter
- controller/udp_link.py — guards on stop/update + create() inheritance hook
- controller/topology.py — persist marker_definitions in project topology
- schemas/controller/links.py — MarkerDefinitionCreate schema
- api/routes/controller/projects.py — REST endpoints (marker-definitions)
Add a read-only `markers` property on Project that flattens every link's
markers into a single dict keyed by "{link_id}/{name}", each entry
carrying the parent link_id and capture-side node_id. Expose it via
GET /projects/{pid}/markers (Project.Audit) so the frontend can fetch all
markers in one round-trip instead of enumerating links first.
Also surface a marker count in project.stats().
The controller endpoint received name via TemplateUsage but never
passed it to add_node_from_template, causing the name parameter to
be silently ignored. name is optional - if omitted the template's
default_name_format is used.
Replace per-project check_user_has_privilege calls with a single
batch method that performs 3 fixed DB queries regardless of project
count. Reduces GET /projects response time for 10000 projects from
~12s to ~290ms (40x improvement).
Fixed a bug where projects created by a user that are also in a resource pool
the user has access to would appear twice in the GET /projects response.
Changes:
- Add seen_project_ids set to track already added projects
- Check for duplicates before adding projects in Step 2 (user projects)
- Check for duplicates before adding projects in Step 3 (resource pool projects)
This ensures each project appears only once regardless of whether it's user-created
or shared via resource pool.
Implement the correct three-step permission check logic:
- Step 1: ACE check - basic access permission (get projects user has ACE for)
- Step 2: Filter ace_projects by created_by - user's own projects (project sharing only through resource pools)
- Step 3: Resource pool projects (projects shared through resource pools)
This fixes the design flaw where:
- ACE check could bypass user isolation with broad ACE configurations
- seen_project_ids mechanism prevented proper layered checking
- Project sharing was confused with direct ACE configuration
The new logic ensures:
- User isolation works even with broad ACE (path='/', propagate=True)
- Project sharing is only available through resource pools (clear design)
- Proper layered checking without seen blocking mechanism
Implement a three-layer permission system:
- Layer 1: ACE strategy check (explicitly authorized/shared projects)
- Layer 2: Ownership check (user's own projects based on created_by)
- Layer 3: Resource pools (team shared projects)
This approach:
- Resolves the conflict between ACE and user isolation
- Enables project sharing via ACE (other users can grant access)
- Maintains default user isolation via ownership
- Prevents duplicate projects in results
- Preserves resource pool functionality
