Preserve image reconciliation, locking, and regression coverage while integrating upstream release, dependency, typing, and formatting updates. Align feature code with the new Ruff checks.
When a Docker node is created on a remote compute whose Docker daemon
does not have the image, the compute now raises ImageMissingError
instead of blindly pulling from the Docker repository. The controller
exports the image from the Docker daemon on its host (docker save
stream) and streams it to the compute which loads it, so locally built
or docker-loaded images work across computes. When the image is not
available on the controller host either, the compute is asked to pull
it from the Docker repository as a fallback.
- add a POST /docker/images/load compute endpoint that streams a
docker save tar into the Docker daemon
- let Docker.http_query pass raw (non-dict) request bodies through so
the tar can be streamed to the daemon
- drop the inline pull from DockerVM.create() and the now unused
DockerVM.pull_image wrapper
Review-driven session/transport fixes (each reproduced live against
sharkd 4.6.7 before fixing):
- raise the RPC stream limit to 16 MB: a full 1000-row frames page
measures ~190 KB against the 64 KB StreamReader default, which failed
the request with a 500 and desynchronized the resident session; a
line-over-limit ValueError is now treated as a transport failure
- verify JSON-RPC reply ids: a timed-out request's late reply was
served as the next request's answer; timeouts, dead pipes, malformed
and stale replies now kill the session for good instead
- make check-spawn atomic under one manager lock: concurrent requests
for the same pcap double-spawned sharkd and leaked the loser (process
plus /tmp scratch copy) forever
- refcount sessions and evict idle only (LRU, cap raised 8 -> 16): a
tag with more sources than the cap respawned every source on every
request, and concurrent requests could get their session killed
mid-RPC (spurious 502)
- map FilterError to sharkd's filter rejection (-13002) only; other
engine failures with a filter set are 502, not a client 400
- detail: accept an optional frame_number to disambiguate
same-microsecond frames (ts is not unique within a pcap); drop the
-8003 -> 404 mapping (the range is validated locally, engine errors
are real faults); a failed hex read is a 404 instead of "hex": null
- a pcap deleted mid-request is a 404, not a 500; the pcap-sized
scratch copy runs off the event loop; server shutdown kills every
resident session and drops its scratch directory
- pin the packet-list layout through scratch-HOME Wireshark
preferences: the column indexes are a contract the server owns
(protocol-level column negotiation is rejected by sharkd 4.6.x)
Range contract change (WebUI moved to an always-flat list): the merged
frame list is returned in full, deliberately uncapped - truncated and
per-second buckets are removed, frame_count always equals
len(frames), and rendering cost is the client's concern (the window
endpoint remains the incremental path).
Narrows the merged frame stream to one capture source BEFORE counting,
slicing and bucketing (frame_count / frames | buckets all recomputed on
the narrowed set), AND-composing with the display filter. A pure
identity filter applied before any engine work — only the selected
link's pcap gets a sharkd pass, so link+filter is cheaper than filter
alone.
Two boundaries by contract with the Web UI:
- sources[] stays the tag's stable inventory: every capture source
listed with engine-free TOTAL counts, unaffected by link/filter — a
source dropdown must not shrink when the view narrows (this also
settles sources[].count on total counts rather than post-filter
matches, which no spec ever required)
- an unknown link_id matches nothing: frame_count 0, start null, empty
frames/buckets — the same shape as a zero-match display filter,
deliberately not a 404; an empty link param is treated as absent
sharkd (the Wireshark daemon) is now the single decode engine — the
tshark/PDML path is gone, and without sharkd every replay endpoint
returns 501 (no degraded mode: one engine, one rendering shape for the
Web UI).
- Frame entries gain packet-list columns from sharkd's frames RPC:
src/dst/proto/info plus the Wireshark coloring hints bg/fg
- range and frames accept ?filter=<display filter>, applied before
counting and slicing; invalid expressions are 400 carrying sharkd's
original text; filters travel as single argv-style elements, capped
at 2000 chars; filtered frames keep their original pcap frame numbers
- frame detail returns sharkd's protocol tree with keys renamed into
the REST contract (element/label/name/filter_expr/pos+size/expert/
generated/children): a census-verified closed key set, values
untouched, unknown keys passed through verbatim, Wireshark-internal
hf ids dropped. filter_expr gives the UI click-to-filter; pos/size
drives hex highlighting (hex still read straight from the pcap)
- one resident 'sharkd -' session per source pcap: lazy spawn, /tmp
scratch copy + scratch HOME (hardened profiles), per-request
(mtime,size) validation with respawn, LRU bound, per-session lock,
per-RPC timeout, bounded close
Timeline backbone (gate, record-header scan, merge ordering, canonical
ts strings, hex reads) stays plain Python — identity and ordering never
depend on the engine.
The bare 'os.kill = MagicMock()' was never restored, so every later test
in the same process ran with a no-op kill. Any test that kills a child
process and waits for it then hangs forever (resident sharkd sessions
waiting on an immortal process). Use monkeypatch so the patch is undone.
The project-open bulk path (_add_nio_binding / _get_existing_nio /
_update_nio_binding in routes/compute/projects.py) dropped port_number
for Docker nodes, unlike the per-node routes and the IOU branch. With
multi-port docker adapters (iol-runner nodes model 4 ports per adapter,
0ca9ccc63) every batched NIO landed on port 0 where Adapter.add_nio()
silently overwrites — the last entry per node won — so reopening a
project clobbered the port-0 links with the port-1 NIOs: links ended up
cross-wired between the wrong node pairs and the real links died
(IOL direct-link ping failures after close/reopen, EXCESSCOLL storms
from the phantom loops).
Templates reference a config file with the GNS3_IOL_STARTUP_CONFIG
environment knob; the controller materializes the file content into
startup_config_content on node creation (sent once, knob consumed —
the same pattern as the IOU startup_config mapping). The compute builds
the content into the node's nvram_<app id> at the next start using the
IOU nvram_import utility (IOL and IOU share the nvram container format,
verified against iol-xe 17.18.02): valid config at boot, no setup
dialog, %h hostname substitution, hostname rewrite on rename.
Semantics verified against the runner: IOL boots from NVRAM whenever it
holds a config, so a plain stop/start never re-applies the startup
config and 'write memory' survives restarts; an explicit content edit
(PUT) is re-applied on the next start and wins over the saved config,
like IOU.
Markers on different links sharing a tag form one distributed capture
session. Once every marker under the tag is paused (409 otherwise), three
read-only endpoints replay it:
- GET .../markers/tags/{tag}/replay/range merges the per-marker pcaps by
scanning 16-byte record headers only (no tshark) into a timestamp-
ordered frame list (per-second buckets above a 5000-frame cap)
- GET .../replay/frames?ts=&window_ms= returns frames in [T, T+window];
an empty window is a normal empty array
- GET .../replay/frame/detail lazily decodes one frame the user opened:
raw bytes for the hex view read straight from the pcap, protocol tree
from 'tshark -T pdml' mapped isomorphically to JSON (every attribute
survives, values stay strings). tshark reads a /tmp scratch copy with a
scratch HOME — hardened profiles deny it the project directory.
Sort key is (ts, source file, frame number): ts is not unique across a
merge. The ts parameter round-trips as the exact string from the frame
list. Round-trip tests pin the PDML→JSON fidelity (element count and
attribute coverage).
Jeremy Grossmann
Mark Paronyan
Cristi
YueGuobin