mirror of
https://github.com/GNS3/gns3-server.git
synced 2026-09-07 02:25:21 +03:00
Merge branch 'feat/marker-tag-replay' into local-test
This commit is contained in:
YueGuobin
commit
efed973107
@ -75,6 +75,9 @@ Web-based packet capture analysis using Docker + xpra HTML5 client. Zero-install
|
||||
### Marker (Traffic Insight) (`features/marker-traffic-insight.md`)
|
||||
Real-time traffic insight via per-link BPF markers and project-level inherited definitions. A marker taps a link in uBridge, emitting match notifications and pcap capture on BPF hit; definitions fan out to every capable link automatically.
|
||||
|
||||
### Marker Tag Replay (`features/marker-tag-replay.md`)
|
||||
Aggregate playback across links keyed by `tag`: once every marker under a tag is paused, their pcaps merge into one timestamp-ordered timeline; frames are decoded on demand via tshark into an isomorphic JSON protocol tree. The cross-link delta of the same packet measures the intermediate node's forwarding latency.
|
||||
|
||||
### Docker exec Console (Vendor NOS) (`features/docker-exec-console.md`)
|
||||
Console for vendor NOS containers (SR Linux, XRd, …) whose CLI is a TUI off PID 1: runs the vendor CLI via the Docker exec API, plus `GNS3_SKIP_INIT`/`GNS3_INTERFACE_NAMES` boot knobs and SKIP_INIT volume persistence.
|
||||
|
||||
@ -127,4 +130,4 @@ Quick-start guide for Ubuntu 24.04: install via PPA, set up dependencies, and ru
|
||||
|
||||
---
|
||||
|
||||
_Last updated: 2026-08-14_
|
||||
_Last updated: 2026-09-01_
|
||||
|
||||
257
docs/features/marker-tag-replay.md
Normal file
257
docs/features/marker-tag-replay.md
Normal file
@ -0,0 +1,257 @@
|
||||
<!--
|
||||
SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
See LICENSE file for licensing information.
|
||||
-->
|
||||
|
||||
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
|
||||
|
||||
# Marker Tag Replay (Aggregate Playback)
|
||||
|
||||
## Overview
|
||||
|
||||
Replays traffic captured by [markers](marker-traffic-insight.md) **across links**, keyed by
|
||||
`tag`. Markers on different links that share a tag form one *distributed capture session*;
|
||||
once every marker under the tag is paused, their per-marker pcaps are merged into a single
|
||||
timestamp-ordered timeline. The Web UI browses that timeline and fetches individual frames
|
||||
on demand — each fetch decodes exactly one frame via `tshark` into a self-describing JSON
|
||||
protocol tree.
|
||||
|
||||
The unique observable: the delta between the same packet hitting two consecutive links
|
||||
measures the **intermediate node's forwarding latency** (host view) — something a
|
||||
single-link capture can never show.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
UI["Web UI"]
|
||||
|
||||
subgraph Controller["Controller (replay endpoints)"]
|
||||
GATE["Tag gate<br/>(all markers under tag paused?)"]
|
||||
SCAN["Timeline scan<br/>(pcap record headers)"]
|
||||
MAP["PDML → JSON<br/>isomorphic mapper"]
|
||||
end
|
||||
|
||||
FS[("markers dir<br/>{node}_{link}_{marker}.pcap")]
|
||||
TS["tshark -T pdml<br/>(one frame at a time)"]
|
||||
TMP["/tmp scratch copy<br/>(hardened-profile workaround)"]
|
||||
|
||||
UI -->|"GET range / frames"| GATE
|
||||
GATE --> SCAN
|
||||
SCAN --> FS
|
||||
UI -->|"GET frame detail (lazy)"| MAP
|
||||
MAP --> TMP
|
||||
TMP --> TS
|
||||
MAP -->|"hex: raw bytes"| FS
|
||||
```
|
||||
|
||||
Two deliberately separated performance regimes:
|
||||
|
||||
| Path | Work | tshark |
|
||||
|------|------|--------|
|
||||
| Timeline | 16-byte record-header scan per frame, cross-file merge sort | **never invoked** — browsing works without tshark |
|
||||
| Frame detail | locate frame → `tshark -T pdml -Y "frame.number == N"` → map to JSON | forked once per frame the user opens |
|
||||
|
||||
tshark call count equals user clicks — no caching or rate limiting needed, and a tshark
|
||||
failure (501/502) affects only that one frame, never timeline browsing. pcap files are
|
||||
compute-side (`<project>/project-files/markers/`); the initial scope is single-server
|
||||
deployments (controller and compute in one process, direct file access) — remote computes
|
||||
will reuse the existing capture-file proxy pattern.
|
||||
|
||||
## Business Process
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant UI as Web UI
|
||||
participant C as Controller
|
||||
participant PC as markers dir (pcaps)
|
||||
participant TS as tshark
|
||||
|
||||
Note over UI,PC: ① configure — same tag on every link's marker
|
||||
UI->>C: POST markers (bpf, tag=666) on each link
|
||||
|
||||
Note over PC: ② capture — uBridge appends matches (flushed per packet), replay forbidden
|
||||
UI->>C: GET range
|
||||
C--xC: 409 (a marker under tag 666 is still enabled)
|
||||
|
||||
Note over UI: ③ pause — every marker under the tag
|
||||
UI->>C: PUT markers {"enabled": false} × each
|
||||
|
||||
Note over UI,TS: ④ replay
|
||||
UI->>C: GET /markers/tags/666/replay/range
|
||||
C->>PC: scan record headers, merge sort
|
||||
C-->>UI: {start, end, sources, frames}
|
||||
UI->>C: GET frames?ts=T&window_ms=W
|
||||
C-->>UI: frames in [T, T+W] — or {"frames": []}
|
||||
UI->>C: GET frame/detail?ts=…&node_id=…&link_id=…&marker=…
|
||||
C->>PC: locate frame, read raw bytes (hex)
|
||||
C->>TS: -T pdml (reads the /tmp copy)
|
||||
TS-->>C: PDML
|
||||
C-->>UI: protocol tree + hex
|
||||
```
|
||||
|
||||
## The tag gate
|
||||
|
||||
Replay reads append-only pcaps, so it is only available while the data is at rest. Every
|
||||
replay endpoint evaluates the same gate: walk every marker in the project carrying the
|
||||
requested tag; if any has `enabled: true` → 409 (the response names them); a tag with no
|
||||
markers at all → 404.
|
||||
|
||||
| Marker state under the tag | pcap file | Replay |
|
||||
|---------------------------|-----------|--------|
|
||||
| any `enabled: true` (capturing) | growing | denied — 409 |
|
||||
| all `enabled: false` (paused) | retained, frozen | **allowed** |
|
||||
| deleted | file unlinked | no data |
|
||||
| `bpf`/`tag`/`direction` changed (rebuild) | pcap reopened (truncated) — new session | prior history gone |
|
||||
| capture node (re)started | pcap reopened (truncated) — new session | prior history gone |
|
||||
|
||||
- **Pause, not delete.** Deleting a marker (or its definition) deletes its pcap — replay
|
||||
before deleting or the data is gone.
|
||||
- **Pause → resume → pause is fine.** The pcap accumulates the full history; replay covers
|
||||
everything up to the current pause point.
|
||||
- **The replay window ends when nodes restart.** A pcap's lifetime equals its uBridge's
|
||||
lifetime: a fresh uBridge reinstalls every desired marker — paused ones too (install
|
||||
first, then turn the filter off) — and uBridge opens the pcap with truncate semantics
|
||||
(`pcap_dump_open`, not `_append`). Server restart + project reopen **without starting
|
||||
nodes** is safe: nothing touches the files until a uBridge comes up (verified live).
|
||||
Docker nodes effectively restart on server restart as well (stale-container cleanup),
|
||||
so their window is shorter still.
|
||||
- uBridge flushes every matched packet to the pcap immediately (`pcap_dump_flush` per
|
||||
packet under a mutex — verified in the uBridge source), so a pause boundary never loses
|
||||
tail frames.
|
||||
|
||||
## API Endpoints
|
||||
|
||||
All read-only; JWT bearer token, privilege `Project.Audit`.
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/v3/projects/{pid}/markers/tags/{tag}/replay/range` | Timeline metadata + full frame list for the tag |
|
||||
| GET | `/v3/projects/{pid}/markers/tags/{tag}/replay/frames?ts=&window_ms=&limit=` | Frames with ts in `[T, T+window]`, merged across sources |
|
||||
| GET | `/v3/projects/{pid}/markers/tags/{tag}/replay/frame/detail?ts=&node_id=&link_id=&marker=` | Single frame: tshark protocol tree + raw hex |
|
||||
|
||||
### `range` — the timeline
|
||||
|
||||
```json
|
||||
{
|
||||
"tag": 666,
|
||||
"start": "1788196663.226372",
|
||||
"end": "1788196713.706634",
|
||||
"frame_count": 20,
|
||||
"truncated": false,
|
||||
"sources": [
|
||||
{ "node_id": "b764c434…", "link_id": "316ef8fd…", "marker": "global-def-…",
|
||||
"data_link_type": "DLT_EN10MB", "count": 10 }
|
||||
],
|
||||
"frames": [
|
||||
{ "ts": "1788196663.226372", "len": 98, "node_id": "b764c434…",
|
||||
"link_id": "316ef8fd…", "marker": "global-def-…", "frame_number": 1 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- `frames` is the **full, merged, time-ordered list** (cap 5000) — the Web UI lays out the
|
||||
whole timeline from one request. Over the cap, `frames` is omitted and per-second
|
||||
`buckets` are returned instead with `truncated: true`.
|
||||
- Each frame entry carries `(node_id, link_id, marker, frame_number)` — the locating
|
||||
tuple for the detail request.
|
||||
|
||||
### `frames` — point / window query
|
||||
|
||||
A time with no frames is a normal, successful answer — an empty array, no sentinel strings:
|
||||
|
||||
```json
|
||||
GET …/replay/frames?ts=1788196700.000&window_ms=500
|
||||
→ { "frames": [] }
|
||||
```
|
||||
|
||||
### `frame/detail` — lazy single-frame decode
|
||||
|
||||
Invoked only when the user opens a frame. The `ts` must be the **exact string received in
|
||||
the timeline/frame list** (round-tripped verbatim — never re-serialized through a float);
|
||||
`node_id + link_id + marker` identify the pcap. The server re-resolves the ts against the
|
||||
file, guarding against a capture rebuilt between the timeline view and this click.
|
||||
|
||||
```json
|
||||
{
|
||||
"ts": "1788196663.226372",
|
||||
"source": { "node_id": "b764c434…", "link_id": "316ef8fd…",
|
||||
"marker": "global-def-…", "frame_number": 1 },
|
||||
"tshark_version": "TShark (Wireshark) 4.6.7 …",
|
||||
"field_count": 85,
|
||||
"hex": "45000062…",
|
||||
"tree": [
|
||||
{ "element": "proto", "name": "ip",
|
||||
"showname": "Internet Protocol Version 4, Src: 10.1.10.101, Dst: 203.0.113.1",
|
||||
"children": [
|
||||
{ "element": "field", "name": "ip.ttl", "show": "64",
|
||||
"showname": "Time to Live: 64", "value": "40", "size": "1",
|
||||
"pos": "22", "children": [] }
|
||||
] }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
- `tree` mirrors PDML **isomorphically**: every `<proto>`/`<field>` becomes a node, every
|
||||
XML attribute (`name`, `show`, `showname`, `value`, `size`, `pos`, `hide`, `mask`,
|
||||
`unmaskedvalue`) becomes a JSON key, plus one structural key `element` (proto/field).
|
||||
Nothing is selected out, nothing interpreted, and **all values stay strings** — numeric
|
||||
conversion is the client's business.
|
||||
- `hex` is the raw frame bytes read straight from the pcap (not via tshark); keeping
|
||||
`pos`/`size` on every field enables Wireshark-style *click field → highlight bytes*.
|
||||
- `field_count` is the mapped node count — a client-side sanity check.
|
||||
|
||||
## Ordering and timestamps
|
||||
|
||||
- `ts` is the pcap record timestamp (µs) written by uBridge at match time — a userspace
|
||||
`gettimeofday()` instant measured after the packet has crossed the kernel twice. The
|
||||
last digit or two are scheduling noise; microseconds are sufficient in a simulated
|
||||
environment.
|
||||
- The sort key is `(ts, source file, frame_number)` — ts alone is **not** unique (two
|
||||
links can hit the same microsecond); the tiebreaker yields a stable, determined order
|
||||
instead of a fictional one. Index structures must never use ts as a dict key, or
|
||||
same-microsecond frames silently overwrite each other.
|
||||
- The cross-link delta is the intermediate node's end-to-end forwarding latency
|
||||
(veth/TAP → guest protocol stack → back to host), typically hundreds of microseconds to
|
||||
milliseconds. UI labels should read *node forwarding latency (host view)*, not link
|
||||
propagation delay. A live capture pair confirmed it end-to-end: same `ip.id`,
|
||||
TTL 64→63, 509 µs between two links.
|
||||
|
||||
## Fidelity guarantee (PDML → JSON)
|
||||
|
||||
The conversion is an isomorphic structure map, not a semantic transform, with two rules:
|
||||
**map every attribute** and **keep values as strings**. A round-trip test enforces both:
|
||||
PDML element count equals JSON node count, and every XML attribute survives with an
|
||||
identical JSON value (`tests/controller/test_marker_replay.py`). The raw frame bytes —
|
||||
the one thing PDML genuinely does not contain — are covered by `hex` read directly from
|
||||
the pcap.
|
||||
|
||||
## Error Responses
|
||||
|
||||
| Status | Description |
|
||||
|--------|-------------|
|
||||
| 401 | Not authenticated |
|
||||
| 404 | Tag has no markers in the project; detail source unknown, or ts does not match the file (the capture may have been rebuilt) |
|
||||
| 409 | Tag gate: a marker under the tag is still `enabled: true` (the response lists them) |
|
||||
| 501 | tshark not installed / unavailable — affects detail only; the timeline never needs tshark |
|
||||
| 502 | tshark failed or timed out (10 s); truncated output never reaches the mapper |
|
||||
|
||||
## Notes
|
||||
|
||||
- **Heterogeneous link types coexist.** Frames are never merged into a single pcap
|
||||
(mergecap is deliberately not used) — each frame carries its source and is decoded
|
||||
individually, so Ethernet and serial (cHDLC/PPP) markers can share one timeline.
|
||||
Malformed packets are tshark's problem: it emits `[Malformed Packet]` as regular PDML
|
||||
and carries on.
|
||||
- **Hardened tshark profiles.** openSUSE-style profiles (AppArmor &c.) can deny tshark
|
||||
access to the project directory and the user's home even though the server process can
|
||||
read both. The detail path therefore copies the pcap to a real file under `/tmp`
|
||||
(not a symlink — the profile resolves real paths) and gives tshark a scratch `HOME`;
|
||||
the copy is unlinked afterwards. The hex view still reads the original file.
|
||||
- **Tag type.** REST and the `marker.match` WS event both carry `tag` as `int` (the
|
||||
listener normalizes); replay keys on that int value.
|
||||
- **Follow-ups.** Remote-compute support via the existing capture-file proxy pattern;
|
||||
convenience APIs (`GET …/markers/tags` to list tags, `POST …/markers/tags/{tag}/pause`
|
||||
to batch-pause — a one-call path to the replayable state); uBridge-side
|
||||
`pcap_dump_open_append` (with a linktype-header check on the existing file) so capture
|
||||
history survives node restarts instead of being truncated on every reinstall.
|
||||
@ -44,6 +44,8 @@ from gns3server.controller.link import _UNSET
|
||||
from gns3server.controller.controller_error import ControllerError, ControllerBadRequestError
|
||||
from gns3server.controller.import_project import import_project as import_controller_project
|
||||
from gns3server.controller.export_project import export_project as export_controller_project
|
||||
from gns3server.controller import marker_replay
|
||||
from gns3server.controller.marker_replay import TsharkError, TsharkMissingError
|
||||
from gns3server.utils.asyncio import aiozipstream
|
||||
from gns3server.utils.path import is_safe_path
|
||||
from gns3server.db.repositories.templates import TemplatesRepository
|
||||
@ -219,6 +221,87 @@ def get_project_markers(project: Project = Depends(dep_project)) -> dict:
|
||||
return project.markers
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{project_id}/markers/tags/{tag}/replay/range",
|
||||
dependencies=[Depends(has_privilege("Project.Audit"))],
|
||||
)
|
||||
def replay_tag_range(tag: int, project: Project = Depends(dep_project)) -> dict:
|
||||
"""
|
||||
Aggregate replay timeline for a tag: merges the pcap of every marker
|
||||
carrying ``tag`` into one timestamp-ordered view (design reference:
|
||||
``marker_replay`` module docstring).
|
||||
|
||||
The tag gate applies: every marker under the tag must be paused
|
||||
(``enabled: false``) — 409 otherwise. The response carries the timeline
|
||||
bounds, per-source stats, and the full merged frame list while under the
|
||||
frame cap (5000); above it the list is replaced by per-second buckets.
|
||||
|
||||
Required privilege: Project.Audit
|
||||
"""
|
||||
|
||||
return marker_replay.build_timeline(project, tag)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{project_id}/markers/tags/{tag}/replay/frames",
|
||||
dependencies=[Depends(has_privilege("Project.Audit"))],
|
||||
)
|
||||
def replay_tag_frames(
|
||||
tag: int,
|
||||
ts: str,
|
||||
window_ms: int = 100,
|
||||
limit: int = 1000,
|
||||
project: Project = Depends(dep_project),
|
||||
) -> dict:
|
||||
"""
|
||||
Frames with ts in ``[ts, ts + window_ms]`` merged across every source of
|
||||
the tag. A time with no frames is a normal successful answer:
|
||||
``{"frames": []}``. The tag gate applies (409 while any marker captures).
|
||||
|
||||
``ts`` must be the exact string returned by the range response — never
|
||||
re-serialize it through a float.
|
||||
|
||||
Required privilege: Project.Audit
|
||||
"""
|
||||
|
||||
return marker_replay.query_frames(project, tag, ts, window_ms=window_ms, limit=limit)
|
||||
|
||||
|
||||
@router.get(
|
||||
"/{project_id}/markers/tags/{tag}/replay/frame/detail",
|
||||
dependencies=[Depends(has_privilege("Project.Audit"))],
|
||||
)
|
||||
async def replay_tag_frame_detail(
|
||||
tag: int,
|
||||
ts: str,
|
||||
node_id: str,
|
||||
link_id: str,
|
||||
marker: str,
|
||||
project: Project = Depends(dep_project),
|
||||
) -> dict:
|
||||
"""
|
||||
Decode exactly one frame (lazy — invoked when the user opens a frame,
|
||||
never by the timeline itself): raw bytes for the hex view read straight
|
||||
from the pcap, protocol tree from ``tshark -T pdml`` mapped isomorphically
|
||||
to JSON (every PDML attribute survives, values stay strings).
|
||||
|
||||
``ts`` must be the exact string from the frame list; ``node_id`` +
|
||||
``link_id`` + ``marker`` identify the source pcap. The tag gate applies.
|
||||
|
||||
Required privilege: Project.Audit
|
||||
"""
|
||||
|
||||
try:
|
||||
return await marker_replay.decode_frame(project, tag, ts, node_id, link_id, marker)
|
||||
except TsharkMissingError:
|
||||
raise HTTPException(
|
||||
status_code=status.HTTP_501_NOT_IMPLEMENTED,
|
||||
detail="tshark is not installed on this server — frame detail is unavailable",
|
||||
)
|
||||
except TsharkError as e:
|
||||
raise HTTPException(status_code=status.HTTP_502_BAD_GATEWAY, detail=str(e))
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Project-level marker definitions (global rules inherited by every link)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@ -114,13 +114,24 @@ class MarkerListener(asyncio.DatagramProtocol):
|
||||
# signals that carry no `link=`.
|
||||
signal_link = link if link and link != "-" else None
|
||||
|
||||
# Normalize the tag to int so the event matches the REST schema
|
||||
# (MarkerCreate.tag is Optional[int]): the signal merely echoes the
|
||||
# decimal we installed via `mark <bpf> tag <id>`, so parsing cannot
|
||||
# fail for well-formed signals; a malformed value keeps the registered
|
||||
# int, and the tag is None only when neither side carries one.
|
||||
event_tag = registered_tag
|
||||
if tag and tag != "-":
|
||||
try:
|
||||
event_tag = int(tag)
|
||||
except ValueError:
|
||||
pass # malformed signal tag: keep the registered value
|
||||
|
||||
event = {
|
||||
"project_id": project_id,
|
||||
"node_id": node_id,
|
||||
"link_id": signal_link or link_id,
|
||||
"filter": filter_name,
|
||||
# Prefer the value carried in the signal; fall back to the one we registered.
|
||||
"tag": tag if tag and tag != "-" else registered_tag,
|
||||
"tag": event_tag,
|
||||
"ts": ts,
|
||||
"len": int(length) if length and length.isdigit() else 0,
|
||||
# Travel direction relative to the capture node (node_id above);
|
||||
|
||||
428
gns3server/controller/marker_replay.py
Normal file
428
gns3server/controller/marker_replay.py
Normal file
@ -0,0 +1,428 @@
|
||||
#!/usr/bin/env python
|
||||
#
|
||||
# Copyright (C) 2024 GNS3 Technologies Inc.
|
||||
#
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
"""
|
||||
Tag-keyed aggregate replay over paused markers' pcap files.
|
||||
|
||||
Markers on different links sharing a ``tag`` form one distributed capture
|
||||
session. This module merges their per-marker pcaps
|
||||
(``<project>/project-files/markers/{node_id}_{link_id}_{name}.pcap``) into a
|
||||
single timestamp-ordered timeline and decodes individual frames on demand.
|
||||
|
||||
Two deliberately separated performance regimes:
|
||||
|
||||
* the timeline path reads only the 16-byte pcap record headers — tshark is
|
||||
never invoked, so browsing works even where tshark is not installed;
|
||||
* the detail path runs one ``tshark -T pdml`` per frame the caller asks about
|
||||
(call count = user clicks) and maps the XML to JSON isomorphically — every
|
||||
PDML attribute survives as a JSON key, values stay strings, nothing is
|
||||
selected out or interpreted.
|
||||
|
||||
Timestamps are uBridge's userspace ``gettimeofday`` at match time (µs, a
|
||||
value measured after the packet has crossed the kernel twice — the last
|
||||
digit or two are scheduling noise). A timestamp is NOT a unique key: the
|
||||
merge sorts by ``(ts, source file, frame number)`` and index structures must
|
||||
never use ts alone as a dict key, or same-microsecond frames silently
|
||||
overwrite each other.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import struct
|
||||
import tempfile
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
from .controller_error import ControllerError, ControllerNotFoundError, ControllerBadRequestError
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# Full frame list is embedded in the range response while under this cap;
|
||||
# above it the response degrades to start/end + per-second buckets so the
|
||||
# client is never flooded by a high-traffic BPF.
|
||||
FRAME_LIST_CAP = 5000
|
||||
|
||||
# One tshark decode per user click: a generous ceiling, not a rate limiter.
|
||||
TSHARK_TIMEOUT_SECONDS = 10.0
|
||||
|
||||
|
||||
class TsharkMissingError(ControllerError):
|
||||
"""tshark is not installed (or not on PATH) — frame detail unavailable."""
|
||||
|
||||
|
||||
class TsharkError(ControllerError):
|
||||
"""tshark exited non-zero / timed out / produced unusable output."""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# pcap record-header scanning (timeline path — no tshark)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
# magic → (byte order, timestamp unit). Both pcap families uBridge can write
|
||||
# (libpcap default µs; ns variant accepted defensively) and both endiannesses.
|
||||
_PCAP_MAGICS = {
|
||||
0xA1B2C3D4: ("<", 1), # little-endian, microseconds
|
||||
0xD4C3B2A1: (">", 1), # big-endian, microseconds
|
||||
0xA1B23C4D: ("<", 1000), # little-endian, nanoseconds
|
||||
0x4D3CB2A1: (">", 1000), # big-endian, nanoseconds
|
||||
}
|
||||
|
||||
|
||||
def _format_ts(sec: int, usec: int) -> str:
|
||||
"""Canonical ts string — the exact form clients must round-trip back."""
|
||||
|
||||
return f"{sec}.{usec:06d}"
|
||||
|
||||
|
||||
def _parse_ts(ts: str) -> int:
|
||||
"""Parse a round-tripped ts string to integer microseconds (exact, no floats)."""
|
||||
|
||||
try:
|
||||
sec, _, frac = ts.partition(".")
|
||||
usec = int(frac.ljust(6, "0")[:6]) if frac else 0
|
||||
return int(sec) * 1_000_000 + usec
|
||||
except ValueError:
|
||||
raise ControllerBadRequestError(f"Invalid timestamp: {ts!r}")
|
||||
|
||||
|
||||
def scan_pcap_frames(path):
|
||||
"""
|
||||
Walk a pcap file reading only record headers.
|
||||
|
||||
:returns: list of ``(ts_sec, ts_usec, incl_len)`` per frame, 1-based order.
|
||||
Tolerates a truncated tail (stops when a record header claims more bytes
|
||||
than the file holds) so a snapshot mid-write never raises.
|
||||
"""
|
||||
|
||||
frames = []
|
||||
try:
|
||||
with open(path, "rb") as f:
|
||||
data = f.read()
|
||||
except OSError as e:
|
||||
raise ControllerError(f"Cannot read marker pcap {path}: {e}")
|
||||
|
||||
if len(data) < 24:
|
||||
return frames # not even a global header — zero frames
|
||||
magic = struct.unpack("<I", data[:4])[0]
|
||||
spec = _PCAP_MAGICS.get(magic)
|
||||
if spec is None:
|
||||
raise ControllerError(f"Unrecognized pcap magic in {path}")
|
||||
byteorder, unit = spec
|
||||
|
||||
pos = 24
|
||||
while pos + 16 <= len(data):
|
||||
ts_sec, ts_frac, incl_len, _orig_len = struct.unpack(
|
||||
byteorder + "IIII", data[pos:pos + 16]
|
||||
)
|
||||
if incl_len > 0xFFFF or pos + 16 + incl_len > len(data):
|
||||
break # truncated tail (snapshot mid-write / torn final record)
|
||||
# Normalize ns pcaps to µs by truncation — uBridge writes µs anyway.
|
||||
frames.append((ts_sec, ts_frac // unit if unit > 1 else ts_frac, incl_len))
|
||||
pos += 16 + incl_len
|
||||
return frames
|
||||
|
||||
|
||||
def read_frame_bytes(path, frame_number):
|
||||
"""
|
||||
Read one frame's raw bytes (hex view) straight from the pcap — never via
|
||||
tshark. ``frame_number`` is 1-based (the same number tshark's
|
||||
``frame.number`` filter uses).
|
||||
"""
|
||||
|
||||
frames = scan_pcap_frames(path)
|
||||
if not 1 <= frame_number <= len(frames):
|
||||
return None
|
||||
# Offset arithmetic mirrors the header scan: global header + every full
|
||||
# record before the target + the target's own record header.
|
||||
offset = 24 + sum(16 + incl for _s, _u, incl in frames[:frame_number - 1]) + 16
|
||||
incl_len = frames[frame_number - 1][2]
|
||||
with open(path, "rb") as f:
|
||||
f.seek(offset)
|
||||
return f.read(incl_len).hex()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tag gate + timeline assembly
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _tag_markers(project, tag):
|
||||
"""
|
||||
Every marker entry in the project carrying ``tag`` (flat
|
||||
``project.markers`` values: link_id / node_id / name keys included).
|
||||
"""
|
||||
|
||||
entries = []
|
||||
for key, info in project.markers.items():
|
||||
if info.get("tag") == tag:
|
||||
link_id, _, name = key.partition("/")
|
||||
entries.append({
|
||||
"node_id": info["node_id"],
|
||||
"link_id": link_id,
|
||||
"marker": name,
|
||||
"enabled": info.get("enabled", True),
|
||||
"data_link_type": info.get("data_link_type", "DLT_EN10MB"),
|
||||
})
|
||||
return entries
|
||||
|
||||
|
||||
def gate_tag(project, tag):
|
||||
"""
|
||||
Replay reads append-only pcaps, so it is only available while the data is
|
||||
at rest: every marker under the tag must be paused. Raises 409 listing
|
||||
the still-running markers, 404 when the tag has no markers at all.
|
||||
"""
|
||||
|
||||
entries = _tag_markers(project, tag)
|
||||
if not entries:
|
||||
raise ControllerNotFoundError(f"No markers with tag {tag} in project")
|
||||
running = [f"{e['marker']} on link {e['link_id']}" for e in entries if e["enabled"]]
|
||||
if running:
|
||||
raise ControllerError(
|
||||
f"Cannot replay tag {tag} while markers are capturing: {', '.join(running)}. "
|
||||
"Pause every marker under the tag first."
|
||||
)
|
||||
return entries
|
||||
|
||||
|
||||
def _merged_frames(project, entries):
|
||||
"""
|
||||
Scan every source pcap and merge into one list sorted by
|
||||
``(ts, source file, frame number)`` — ts alone is not unique (two links
|
||||
can hit the same microsecond); the tiebreaker yields a stable, determined
|
||||
order instead of a fictional one.
|
||||
"""
|
||||
|
||||
markers_dir = project.markers_directory
|
||||
merged = []
|
||||
sources = []
|
||||
for entry in entries:
|
||||
pcap = os.path.join(
|
||||
markers_dir, f"{entry['node_id']}_{entry['link_id']}_{entry['marker']}.pcap"
|
||||
)
|
||||
frames = scan_pcap_frames(pcap) if os.path.exists(pcap) else []
|
||||
sources.append({**{k: entry[k] for k in ("node_id", "link_id", "marker", "data_link_type")},
|
||||
"count": len(frames)})
|
||||
for frame_number, (sec, usec, incl_len) in enumerate(frames, start=1):
|
||||
merged.append({
|
||||
"ts": _format_ts(sec, usec),
|
||||
"ts_us": sec * 1_000_000 + usec,
|
||||
"len": incl_len,
|
||||
"node_id": entry["node_id"],
|
||||
"link_id": entry["link_id"],
|
||||
"marker": entry["marker"],
|
||||
"frame_number": frame_number,
|
||||
"_source": f"{entry['node_id']}_{entry['link_id']}_{entry['marker']}",
|
||||
})
|
||||
merged.sort(key=lambda f: (f["ts_us"], f["_source"], f["frame_number"]))
|
||||
for frame in merged:
|
||||
del frame["ts_us"]
|
||||
del frame["_source"]
|
||||
return merged, sources
|
||||
|
||||
|
||||
def build_timeline(project, tag, frame_cap=FRAME_LIST_CAP):
|
||||
"""
|
||||
The ``range`` response: timeline bounds, per-source stats, and (under
|
||||
``frame_cap``) the full merged frame list for one-request timeline
|
||||
layout. Over the cap the list is replaced by per-second buckets.
|
||||
"""
|
||||
|
||||
entries = gate_tag(project, tag)
|
||||
frames, sources = _merged_frames(project, entries)
|
||||
|
||||
response = {
|
||||
"tag": tag,
|
||||
"start": frames[0]["ts"] if frames else None,
|
||||
"end": frames[-1]["ts"] if frames else None,
|
||||
"frame_count": len(frames),
|
||||
"truncated": len(frames) > frame_cap,
|
||||
"sources": sources,
|
||||
}
|
||||
if len(frames) <= frame_cap:
|
||||
response["frames"] = frames
|
||||
else:
|
||||
buckets = {}
|
||||
for frame in frames:
|
||||
second = _parse_ts(frame["ts"]) // 1_000_000
|
||||
buckets[second] = buckets.get(second, 0) + 1
|
||||
response["buckets"] = [
|
||||
{"ts": _format_ts(second, 0), "count": count}
|
||||
for second, count in sorted(buckets.items())
|
||||
]
|
||||
return response
|
||||
|
||||
|
||||
def query_frames(project, tag, ts, window_ms=100, limit=1000):
|
||||
"""
|
||||
Frames with ts in ``[T, T+window_ms]`` merged across sources. A time with
|
||||
no frames is a normal, successful answer — ``{"frames": []}``.
|
||||
"""
|
||||
|
||||
entries = gate_tag(project, tag)
|
||||
frames, _sources = _merged_frames(project, entries)
|
||||
|
||||
start_us = _parse_ts(ts)
|
||||
end_us = start_us + max(window_ms, 0) * 1000
|
||||
hits = [f for f in frames if start_us <= _parse_ts(f["ts"]) <= end_us]
|
||||
return {"frames": hits[:max(limit, 0)]}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frame detail (tshark path — lazy, one frame per call)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def _tshark_version():
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"tshark", "--version",
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL,
|
||||
)
|
||||
stdout, _ = await asyncio.wait_for(proc.communicate(), timeout=TSHARK_TIMEOUT_SECONDS)
|
||||
return stdout.decode(errors="replace").splitlines()[0].strip()
|
||||
except (OSError, asyncio.TimeoutError, IndexError):
|
||||
raise TsharkMissingError("tshark is not available on this server")
|
||||
|
||||
|
||||
def _tshark_scratch_copy(pcap):
|
||||
"""
|
||||
Copy the pcap to a scratch file under the system temp dir for tshark to
|
||||
read. Hardened tshark profiles (AppArmor &c.) can deny it access to the
|
||||
project directory / the user's home while still allowing /tmp — a real
|
||||
copy, deliberately not a symlink, since the profile resolves real paths.
|
||||
Caller must unlink the returned path.
|
||||
"""
|
||||
|
||||
fd, scratch = tempfile.mkstemp(suffix=".pcap", prefix="gns3-replay-")
|
||||
os.close(fd)
|
||||
shutil.copyfile(pcap, scratch)
|
||||
return scratch
|
||||
|
||||
|
||||
def _tshark_env():
|
||||
"""Scratch HOME so tshark never even tries to read the user's home."""
|
||||
|
||||
env = dict(os.environ)
|
||||
env["HOME"] = tempfile.gettempdir()
|
||||
return env
|
||||
|
||||
|
||||
def _pdml_to_nodes(element):
|
||||
"""
|
||||
Isomorphic PDML → JSON mapping: every XML attribute becomes a JSON key
|
||||
verbatim (values stay strings), children nest under ``children``. The
|
||||
element tag ("proto"/"field") is carried as ``element`` — the one
|
||||
structural key beyond the attributes, so a renderer can tell a protocol
|
||||
group from a leaf field (geninfo's tagless names make names unreliable).
|
||||
"""
|
||||
|
||||
return {
|
||||
"element": element.tag,
|
||||
**element.attrib,
|
||||
"children": [_pdml_to_nodes(child) for child in element],
|
||||
}
|
||||
|
||||
|
||||
def _count_nodes(nodes):
|
||||
return 1 + sum(_count_nodes(child) for child in nodes.get("children", []))
|
||||
|
||||
|
||||
async def decode_frame(project, tag, ts, node_id, link_id, marker):
|
||||
"""
|
||||
Decode exactly one frame: locate its pcap by source identity, verify the
|
||||
round-tripped ts still matches the file (guards a rebuild between the
|
||||
timeline view and this click), read the raw bytes for the hex view, and
|
||||
map tshark's PDML of that single frame to JSON.
|
||||
"""
|
||||
|
||||
entries = gate_tag(project, tag)
|
||||
entry = next(
|
||||
(e for e in entries
|
||||
if e["node_id"] == node_id and e["link_id"] == link_id and e["marker"] == marker),
|
||||
None,
|
||||
)
|
||||
if entry is None:
|
||||
raise ControllerNotFoundError(
|
||||
f"No marker '{marker}' with tag {tag} on link {link_id} captured by {node_id}"
|
||||
)
|
||||
|
||||
pcap = os.path.join(project.markers_directory, f"{node_id}_{link_id}_{marker}.pcap")
|
||||
if not os.path.exists(pcap):
|
||||
raise ControllerNotFoundError(f"No capture file for marker '{marker}' (nothing ever matched)")
|
||||
|
||||
frames = scan_pcap_frames(pcap)
|
||||
# The ts must be the exact string the timeline returned; find the frame
|
||||
# it identifies rather than trusting any position hint from the client.
|
||||
frame_number = next(
|
||||
(i for i, (sec, usec, _len) in enumerate(frames, start=1)
|
||||
if _format_ts(sec, usec) == ts),
|
||||
None,
|
||||
)
|
||||
if frame_number is None:
|
||||
raise ControllerNotFoundError(
|
||||
f"No frame at ts {ts} in marker '{marker}' (the capture may have been rebuilt)"
|
||||
)
|
||||
|
||||
raw_hex = read_frame_bytes(pcap, frame_number)
|
||||
|
||||
if shutil.which("tshark") is None:
|
||||
raise TsharkMissingError("tshark is not installed — frame detail is unavailable")
|
||||
version = await _tshark_version()
|
||||
|
||||
# Hand tshark a scratch copy under the temp dir: hardened profiles may
|
||||
# deny it the project directory even though this process can read it
|
||||
# (the hex view above reads the original directly).
|
||||
scratch = _tshark_scratch_copy(pcap)
|
||||
try:
|
||||
try:
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"tshark", "-r", scratch, "-T", "pdml", "-Y", f"frame.number == {frame_number}",
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE,
|
||||
env=_tshark_env(),
|
||||
)
|
||||
stdout, stderr = await asyncio.wait_for(
|
||||
proc.communicate(), timeout=TSHARK_TIMEOUT_SECONDS
|
||||
)
|
||||
except OSError as e:
|
||||
raise TsharkError(f"Could not run tshark: {e}")
|
||||
except asyncio.TimeoutError:
|
||||
raise TsharkError(f"tshark timed out after {TSHARK_TIMEOUT_SECONDS:.0f}s")
|
||||
if proc.returncode != 0 or not stdout.strip():
|
||||
# Never feed truncated/failed output to the mapper.
|
||||
raise TsharkError(f"tshark failed: {stderr.decode(errors='replace').strip()[:500]}")
|
||||
finally:
|
||||
try:
|
||||
os.unlink(scratch)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
try:
|
||||
root = ET.fromstring(stdout)
|
||||
except ET.ParseError as e:
|
||||
raise TsharkError(f"Malformed PDML from tshark: {e}")
|
||||
|
||||
packet = root.find("./packet")
|
||||
tree = [_pdml_to_nodes(child) for child in packet] if packet is not None else []
|
||||
return {
|
||||
"ts": ts,
|
||||
"source": {"node_id": node_id, "link_id": link_id, "marker": marker,
|
||||
"frame_number": frame_number},
|
||||
"tshark_version": version,
|
||||
"field_count": sum(_count_nodes(node) for node in tree),
|
||||
"hex": raw_hex,
|
||||
"tree": tree,
|
||||
}
|
||||
@ -504,6 +504,17 @@ class Project:
|
||||
os.makedirs(path, exist_ok=True)
|
||||
return path
|
||||
|
||||
@property
|
||||
def markers_directory(self):
|
||||
"""
|
||||
Location of the marker pcap files (same layout the compute side writes
|
||||
via ``markers_working_directory`` — single-server deployments share the
|
||||
project directory, which is what tag replay reads).
|
||||
"""
|
||||
path = os.path.join(self._path, "project-files", "markers")
|
||||
os.makedirs(path, exist_ok=True)
|
||||
return path
|
||||
|
||||
@property
|
||||
def pictures_directory(self):
|
||||
"""
|
||||
|
||||
190
tests/api/routes/controller/test_marker_replay.py
Normal file
190
tests/api/routes/controller/test_marker_replay.py
Normal file
@ -0,0 +1,190 @@
|
||||
#!/usr/bin/env python
|
||||
#
|
||||
# Copyright (C) 2025 GNS3 Technologies Inc.
|
||||
#
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
"""
|
||||
HTTP-route tests for the tag replay endpoints: the tag gate (409 while any
|
||||
marker captures, 404 unknown tag), the merged timeline, window queries
|
||||
(empty window = success), and the lazy frame detail (ts guard, isomorphic
|
||||
JSON, 501 when tshark is unavailable).
|
||||
"""
|
||||
|
||||
import shutil
|
||||
from unittest.mock import patch
|
||||
|
||||
import pytest
|
||||
from fastapi import FastAPI, status
|
||||
from httpx import AsyncClient
|
||||
|
||||
from gns3server.controller.project import Project
|
||||
from gns3server.controller.udp_link import UDPLink
|
||||
|
||||
from tests.controller.test_marker_replay import _write_pcap, _icmp_frame
|
||||
|
||||
pytestmark = pytest.mark.asyncio
|
||||
|
||||
tshark_present = pytest.mark.skipif(shutil.which("tshark") is None, reason="tshark not installed")
|
||||
|
||||
|
||||
def _add_marker(project, tag, enabled, node_id, frames=None):
|
||||
"""Create a paused/capturing link+marker and optionally its pcap."""
|
||||
|
||||
link = UDPLink(project)
|
||||
link._markers["icmp"] = {"bpf": "icmp", "tag": tag, "enabled": enabled, "color": None,
|
||||
"highlight_duration": None, "capture_node_id": node_id,
|
||||
"direction": None, "data_link_type": "DLT_EN10MB"}
|
||||
project._links[link.id] = link
|
||||
if frames is not None:
|
||||
_write_pcap(
|
||||
f"{project.markers_directory}/{node_id}_{link.id}_icmp.pcap", frames
|
||||
)
|
||||
return link
|
||||
|
||||
|
||||
class TestReplayRoutes:
|
||||
|
||||
async def test_range_409_while_capturing(self, app: FastAPI, client: AsyncClient, project: Project) -> None:
|
||||
|
||||
_add_marker(project, tag=7, enabled=False, node_id="n1")
|
||||
running = _add_marker(project, tag=7, enabled=True, node_id="n2")
|
||||
|
||||
response = await client.get(
|
||||
app.url_path_for("replay_tag_range", project_id=project.id, tag=7)
|
||||
)
|
||||
assert response.status_code == status.HTTP_409_CONFLICT
|
||||
assert "icmp" in response.json()["message"]
|
||||
assert running.id in response.json()["message"]
|
||||
|
||||
async def test_range_404_unknown_tag(self, app: FastAPI, client: AsyncClient, project: Project) -> None:
|
||||
|
||||
_add_marker(project, tag=7, enabled=False, node_id="n1")
|
||||
|
||||
response = await client.get(
|
||||
app.url_path_for("replay_tag_range", project_id=project.id, tag=99)
|
||||
)
|
||||
assert response.status_code == status.HTTP_404_NOT_FOUND
|
||||
|
||||
async def test_range_merges_sources_in_ts_order(self, app: FastAPI, client: AsyncClient, project: Project) -> None:
|
||||
|
||||
# r1→r2 captures at t1 and t3; r2→r3 captures at t2 and t3 (same µs
|
||||
# as source A's t3 — the tiebreak must keep both frames).
|
||||
_add_marker(project, tag=7, enabled=False, node_id="n1", frames=[
|
||||
(1693472000, 500000, b"a" * 60),
|
||||
(1693472002, 0, b"a" * 60),
|
||||
])
|
||||
_add_marker(project, tag=7, enabled=False, node_id="n2", frames=[
|
||||
(1693472001, 0, b"b" * 60),
|
||||
(1693472002, 0, b"b" * 60),
|
||||
])
|
||||
|
||||
response = await client.get(
|
||||
app.url_path_for("replay_tag_range", project_id=project.id, tag=7)
|
||||
)
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
body = response.json()
|
||||
assert body["tag"] == 7
|
||||
assert body["frame_count"] == 4
|
||||
assert body["start"] == "1693472000.500000"
|
||||
assert body["end"] == "1693472002.000000"
|
||||
assert body["truncated"] is False
|
||||
assert [f["node_id"] for f in body["frames"]] == ["n1", "n2", "n1", "n2"]
|
||||
assert [f["ts"] for f in body["frames"]] == [
|
||||
"1693472000.500000", "1693472001.000000",
|
||||
"1693472002.000000", "1693472002.000000",
|
||||
]
|
||||
assert len(body["sources"]) == 2
|
||||
|
||||
async def test_frames_window_miss_is_empty_success(
|
||||
self, app: FastAPI, client: AsyncClient, project: Project
|
||||
) -> None:
|
||||
|
||||
_add_marker(project, tag=7, enabled=False, node_id="n1", frames=[
|
||||
(1693472000, 0, b"a" * 60),
|
||||
])
|
||||
|
||||
response = await client.get(
|
||||
app.url_path_for("replay_tag_frames", project_id=project.id, tag=7),
|
||||
params={"ts": "1693472001.000000", "window_ms": 100},
|
||||
)
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
assert response.json() == {"frames": []}
|
||||
|
||||
async def test_frames_window_hit(self, app: FastAPI, client: AsyncClient, project: Project) -> None:
|
||||
|
||||
_add_marker(project, tag=7, enabled=False, node_id="n1", frames=[
|
||||
(1693472000, 0, b"a" * 60),
|
||||
(1693472000, 150000, b"a" * 60),
|
||||
])
|
||||
|
||||
response = await client.get(
|
||||
app.url_path_for("replay_tag_frames", project_id=project.id, tag=7),
|
||||
params={"ts": "1693472000.000000", "window_ms": 150},
|
||||
)
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
assert [f["ts"] for f in response.json()["frames"]] == [
|
||||
"1693472000.000000", "1693472000.150000"
|
||||
]
|
||||
|
||||
async def test_detail_404_on_ts_mismatch(self, app: FastAPI, client: AsyncClient, project: Project) -> None:
|
||||
|
||||
link = _add_marker(project, tag=7, enabled=False, node_id="n1", frames=[
|
||||
(1693472000, 123456, _icmp_frame()),
|
||||
])
|
||||
|
||||
response = await client.get(
|
||||
app.url_path_for("replay_tag_frame_detail", project_id=project.id, tag=7),
|
||||
params={"ts": "1.000000", "node_id": "n1", "link_id": link.id, "marker": "icmp"},
|
||||
)
|
||||
assert response.status_code == status.HTTP_404_NOT_FOUND
|
||||
assert "rebuilt" in response.json()["message"]
|
||||
|
||||
async def test_detail_501_without_tshark(self, app: FastAPI, client: AsyncClient, project: Project) -> None:
|
||||
|
||||
link = _add_marker(project, tag=7, enabled=False, node_id="n1", frames=[
|
||||
(1693472000, 123456, _icmp_frame()),
|
||||
])
|
||||
|
||||
with patch("gns3server.controller.marker_replay.shutil.which", return_value=None):
|
||||
response = await client.get(
|
||||
app.url_path_for("replay_tag_frame_detail", project_id=project.id, tag=7),
|
||||
params={"ts": "1693472000.123456", "node_id": "n1",
|
||||
"link_id": link.id, "marker": "icmp"},
|
||||
)
|
||||
assert response.status_code == status.HTTP_501_NOT_IMPLEMENTED
|
||||
|
||||
@tshark_present
|
||||
async def test_detail_decodes_single_frame(self, app: FastAPI, client: AsyncClient, project: Project) -> None:
|
||||
|
||||
link = _add_marker(project, tag=7, enabled=False, node_id="n1", frames=[
|
||||
(1693472000, 123456, _icmp_frame()),
|
||||
])
|
||||
|
||||
response = await client.get(
|
||||
app.url_path_for("replay_tag_frame_detail", project_id=project.id, tag=7),
|
||||
params={"ts": "1693472000.123456", "node_id": "n1",
|
||||
"link_id": link.id, "marker": "icmp"},
|
||||
)
|
||||
assert response.status_code == status.HTTP_200_OK
|
||||
body = response.json()
|
||||
assert body["source"]["frame_number"] == 1
|
||||
assert body["hex"] == _icmp_frame().hex()
|
||||
assert body["field_count"] > 10
|
||||
assert "tshark" in body["tshark_version"].lower()
|
||||
|
||||
ip = next(p for p in body["tree"] if p.get("name") == "ip")
|
||||
ttl = next(f for f in ip["children"] if f.get("name") == "ip.ttl")
|
||||
# Values arrive as strings, exactly as tshark emitted them.
|
||||
assert ttl["show"] == "64" and ttl["showname"] == "Time to Live: 64"
|
||||
@ -121,7 +121,11 @@ class TestMarkerListener:
|
||||
assert ev["node_id"] == "n1"
|
||||
assert ev["link_id"] == "l1"
|
||||
assert ev["filter"] == "f1"
|
||||
assert ev["tag"] == "7"
|
||||
# The event tag is normalized to int to match the REST schema — the
|
||||
# signal echoes the decimal we installed, so str and int variants of
|
||||
# the same tag must never reach consumers as different keys.
|
||||
assert ev["tag"] == 7
|
||||
assert isinstance(ev["tag"], int)
|
||||
assert ev["ts"] == pytest.approx(1700000000.123456)
|
||||
assert ev["len"] == 98
|
||||
# No dir= in the signal (legacy uBridge) → undirected.
|
||||
@ -163,6 +167,24 @@ class TestMarkerListener:
|
||||
lis.datagram_received(b"MARK 2.0 node=n filter=f tag=- len=20\n", None)
|
||||
assert fmgr.events[0][1]["tag"] == 42
|
||||
|
||||
def test_non_numeric_signal_tag_falls_back_to_registered(self):
|
||||
# A corrupt/unknown signal value must not leak a str tag into the
|
||||
# event stream — the registry's int wins.
|
||||
fmgr = FakeMarkerManager()
|
||||
fmgr.register("p", "n", "f", "l", tag=42)
|
||||
lis = MarkerListener(fmgr)
|
||||
lis.connection_made(None)
|
||||
lis.datagram_received(b"MARK 2.0 node=n filter=f tag=oops len=20\n", None)
|
||||
assert fmgr.events[0][1]["tag"] == 42
|
||||
|
||||
def test_no_tag_anywhere_is_none(self):
|
||||
fmgr = FakeMarkerManager()
|
||||
fmgr.register("p", "n", "f", "l", tag=None)
|
||||
lis = MarkerListener(fmgr)
|
||||
lis.connection_made(None)
|
||||
lis.datagram_received(b"MARK 2.0 node=n filter=f tag=- len=20\n", None)
|
||||
assert fmgr.events[0][1]["tag"] is None
|
||||
|
||||
def test_link_in_signal_overrides_registry_link(self):
|
||||
# Per-link attribution (contract §3.2/§3.3): the signal's `link=` is
|
||||
# authoritative and must disambiguate links sharing a node+filter —
|
||||
|
||||
346
tests/controller/test_marker_replay.py
Normal file
346
tests/controller/test_marker_replay.py
Normal file
@ -0,0 +1,346 @@
|
||||
#!/usr/bin/env python
|
||||
#
|
||||
# Copyright (C) 2025 GNS3 Technologies Inc.
|
||||
#
|
||||
# This program is free software: you can redistribute it and/or modify
|
||||
# it under the terms of the GNU General Public License as published by
|
||||
# the Free Software Foundation, either version 3 of the License, or
|
||||
# (at your option) any later version.
|
||||
#
|
||||
# This program is distributed in the hope that it will be useful,
|
||||
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
# GNU General Public License for more details.
|
||||
#
|
||||
# You should have received a copy of the GNU General Public License
|
||||
# along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
|
||||
"""
|
||||
Unit tests for the tag-keyed aggregate replay module (controller layer):
|
||||
|
||||
* pcap record-header scanning (ts extraction, truncated-tail tolerance,
|
||||
nanosecond-magic normalization) and raw-bytes reads for the hex view
|
||||
* the tag gate (404 unknown tag, 409 while any marker captures)
|
||||
* timeline assembly: cross-source merge ordered by (ts, source, frame number),
|
||||
same-microsecond tiebreak, missing-pcap sources, frame-cap degradation to
|
||||
per-second buckets
|
||||
* window queries (inclusive bounds, empty-window success)
|
||||
* the lazy frame detail: round-tripped-ts guard, and — where tshark is
|
||||
installed — the isomorphic PDML → JSON mapping with a round-trip check
|
||||
(element count and attribute coverage).
|
||||
"""
|
||||
|
||||
import os
|
||||
import shutil
|
||||
import struct
|
||||
|
||||
import pytest
|
||||
from types import SimpleNamespace
|
||||
|
||||
from gns3server.controller.controller_error import ControllerError, ControllerNotFoundError
|
||||
from gns3server.controller.marker_replay import (
|
||||
build_timeline,
|
||||
decode_frame,
|
||||
query_frames,
|
||||
read_frame_bytes,
|
||||
scan_pcap_frames,
|
||||
_format_ts,
|
||||
_parse_ts,
|
||||
)
|
||||
|
||||
pytestmark = pytest.mark.asyncio
|
||||
|
||||
PCAP_MAGIC_US = 0xA1B2C3D4
|
||||
PCAP_MAGIC_NS = 0xA1B23C4D
|
||||
|
||||
|
||||
def _write_pcap(path, frames, magic=PCAP_MAGIC_US, snaplen=65535):
|
||||
"""frames: list of (sec, frac, payload bytes); frac is µs (or ns for the ns magic)."""
|
||||
|
||||
with open(path, "wb") as f:
|
||||
f.write(struct.pack("<IHHiIII", magic, 2, 4, 0, 0, snaplen, 1))
|
||||
for sec, frac, payload in frames:
|
||||
f.write(struct.pack("<IIII", sec, frac, len(payload), len(payload)))
|
||||
f.write(payload)
|
||||
|
||||
|
||||
def _icmp_frame():
|
||||
"""A minimal well-formed ICMP echo request (10.0.0.1 → 10.0.0.3)."""
|
||||
|
||||
def cksum(data):
|
||||
if len(data) % 2:
|
||||
data = data + b"\x00" # RFC 1071 odd-length padding
|
||||
s = 0
|
||||
for i in range(0, len(data), 2):
|
||||
s += (data[i] << 8) + data[i + 1]
|
||||
while s >> 16:
|
||||
s = (s & 0xFFFF) + (s >> 16)
|
||||
return (~s) & 0xFFFF
|
||||
|
||||
icmp = bytes([8, 0, 0, 0]) + struct.pack(">HHH", 1, 1, 0) + b"payload12"
|
||||
icmp = icmp[:2] + struct.pack(">H", cksum(icmp)) + icmp[4:]
|
||||
ip0 = struct.pack(">BBHHHBBH4s4s", 0x45, 0, 20 + len(icmp), 1, 0, 64, 1, 0,
|
||||
bytes([10, 0, 0, 1]), bytes([10, 0, 0, 3]))
|
||||
ip = ip0[:10] + struct.pack(">H", cksum(ip0)) + ip0[12:]
|
||||
return bytes.fromhex("0200000000020200000000010800") + ip + icmp
|
||||
|
||||
|
||||
def _fake_project(tmp_path, markers, markers_dir=None):
|
||||
"""markers: the flat project.markers shape ({'link/name': {..., node_id}})."""
|
||||
|
||||
return SimpleNamespace(markers=markers, markers_directory=str(markers_dir or tmp_path))
|
||||
|
||||
|
||||
def _marker_entry(tag, enabled=True, node_id="node-1"):
|
||||
return {"bpf": "icmp", "tag": tag, "enabled": enabled, "color": None,
|
||||
"highlight_duration": None, "capture_node_id": node_id,
|
||||
"direction": None, "data_link_type": "DLT_EN10MB",
|
||||
"node_id": node_id}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# pcap scanning
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestScanPcap:
|
||||
|
||||
async def test_scans_frames_and_truncated_tail(self, tmp_path):
|
||||
pcap = tmp_path / "a.pcap"
|
||||
_write_pcap(pcap, [
|
||||
(1693472000, 123456, b"x" * 60),
|
||||
(1693472001, 654321, b"y" * 40),
|
||||
])
|
||||
# Tear the final record in half: a snapshot mid-write must not raise.
|
||||
data = bytearray(pcap.read_bytes())
|
||||
pcap.write_bytes(data[:len(data) - 20])
|
||||
|
||||
frames = scan_pcap_frames(str(pcap))
|
||||
assert frames == [(1693472000, 123456, 60)]
|
||||
|
||||
async def test_ns_magic_normalized_to_us(self, tmp_path):
|
||||
pcap = tmp_path / "ns.pcap"
|
||||
_write_pcap(pcap, [(1693472000, 1500000, b"z" * 10)], magic=PCAP_MAGIC_NS)
|
||||
assert scan_pcap_frames(str(pcap)) == [(1693472000, 1500, 10)] # 1.5 ms in µs
|
||||
|
||||
async def test_read_frame_bytes_offsets(self, tmp_path):
|
||||
pcap = tmp_path / "b.pcap"
|
||||
_write_pcap(pcap, [
|
||||
(100, 0, b"first" + b"0" * 55), # 60 bytes
|
||||
(200, 0, b"second"), # 6 bytes
|
||||
])
|
||||
assert read_frame_bytes(str(pcap), 2) == b"second".hex()
|
||||
assert read_frame_bytes(str(pcap), 1) == (b"first" + b"0" * 55).hex()
|
||||
assert read_frame_bytes(str(pcap), 3) is None
|
||||
|
||||
async def test_ts_string_round_trip_is_exact(self):
|
||||
ts = _format_ts(1693472000, 5)
|
||||
assert ts == "1693472000.000005"
|
||||
assert _parse_ts(ts) == 1693472000000005
|
||||
assert _parse_ts(_format_ts(1693472000, 123456)) == 1693472000123456
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Tag gate + timeline
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestGateAndTimeline:
|
||||
|
||||
async def test_unknown_tag_404(self, tmp_path):
|
||||
project = _fake_project(tmp_path, {"linkA/icmp": _marker_entry(tag=1)})
|
||||
with pytest.raises(ControllerNotFoundError):
|
||||
build_timeline(project, tag=7)
|
||||
|
||||
async def test_gate_409_while_capturing(self, tmp_path):
|
||||
project = _fake_project(tmp_path, {
|
||||
"linkA/icmp": _marker_entry(tag=7, enabled=False, node_id="n1"),
|
||||
"linkB/icmp": _marker_entry(tag=7, enabled=True, node_id="n2"),
|
||||
})
|
||||
with pytest.raises(ControllerError, match="linkB"):
|
||||
build_timeline(project, tag=7)
|
||||
|
||||
async def test_merge_orders_by_ts_with_stable_tiebreak(self, tmp_path):
|
||||
# Two sources, deliberately interleaved in time, colliding on one µs.
|
||||
_write_pcap(tmp_path / "n1_linkA_icmp.pcap", [
|
||||
(1693472000, 500000, b"a" * 60), # t1 sourceA
|
||||
(1693472002, 000000, b"a" * 60), # t3 sourceA
|
||||
])
|
||||
_write_pcap(tmp_path / "n2_linkB_icmp.pcap", [
|
||||
(1693472001, 000000, b"b" * 60), # t2 sourceB
|
||||
(1693472002, 000000, b"b" * 60), # t3 sourceB — same µs as t3 sourceA
|
||||
])
|
||||
project = _fake_project(tmp_path, {
|
||||
"linkA/icmp": _marker_entry(tag=7, enabled=False, node_id="n1"),
|
||||
"linkB/icmp": _marker_entry(tag=7, enabled=False, node_id="n2"),
|
||||
})
|
||||
|
||||
timeline = build_timeline(project, tag=7)
|
||||
assert timeline["frame_count"] == 4
|
||||
assert timeline["start"] == "1693472000.500000"
|
||||
assert timeline["end"] == "1693472002.000000"
|
||||
assert [f["node_id"] for f in timeline["frames"]] == ["n1", "n2", "n1", "n2"]
|
||||
# Same-microsecond pair keeps both frames (a ts dict key would drop one).
|
||||
assert [f["ts"] for f in timeline["frames"]][2:] == ["1693472002.000000"] * 2
|
||||
assert [f["frame_number"] for f in timeline["frames"]] == [1, 1, 2, 2]
|
||||
assert {s["count"] for s in timeline["sources"]} == {2}
|
||||
|
||||
async def test_missing_pcap_is_zero_count_source(self, tmp_path):
|
||||
project = _fake_project(tmp_path, {"linkA/icmp": _marker_entry(tag=7, enabled=False)})
|
||||
timeline = build_timeline(project, tag=7)
|
||||
assert timeline["frame_count"] == 0
|
||||
assert timeline["start"] is None and timeline["end"] is None
|
||||
assert timeline["frames"] == []
|
||||
assert timeline["sources"][0]["count"] == 0
|
||||
|
||||
async def test_over_cap_degrades_to_buckets(self, tmp_path):
|
||||
_write_pcap(tmp_path / "n1_linkA_icmp.pcap", [
|
||||
(1693472000, 0, b"a" * 60), (1693472000, 500000, b"a" * 60),
|
||||
(1693472001, 0, b"a" * 60),
|
||||
])
|
||||
project = _fake_project(tmp_path, {"linkA/icmp": _marker_entry(tag=7, enabled=False, node_id="n1")})
|
||||
|
||||
timeline = build_timeline(project, tag=7, frame_cap=2)
|
||||
assert timeline["truncated"] is True
|
||||
assert "frames" not in timeline
|
||||
assert timeline["buckets"] == [
|
||||
{"ts": "1693472000.000000", "count": 2},
|
||||
{"ts": "1693472001.000000", "count": 1},
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Window query
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestQueryFrames:
|
||||
|
||||
def _project(self, tmp_path):
|
||||
_write_pcap(tmp_path / "n1_linkA_icmp.pcap", [
|
||||
(1693472000, 0, b"a" * 60),
|
||||
(1693472000, 150000, b"a" * 60),
|
||||
(1693472005, 0, b"a" * 60),
|
||||
])
|
||||
return _fake_project(tmp_path, {"linkA/icmp": _marker_entry(tag=7, enabled=False, node_id="n1")})
|
||||
|
||||
async def test_window_inclusive_bounds(self, tmp_path):
|
||||
result = query_frames(self._project(tmp_path), tag=7, ts="1693472000.000000", window_ms=150)
|
||||
assert [f["ts"] for f in result["frames"]] == ["1693472000.000000", "1693472000.150000"]
|
||||
|
||||
async def test_window_miss_is_empty_success(self, tmp_path):
|
||||
result = query_frames(self._project(tmp_path), tag=7, ts="1693472001.000000", window_ms=100)
|
||||
assert result == {"frames": []}
|
||||
|
||||
async def test_limit_applies(self, tmp_path):
|
||||
result = query_frames(self._project(tmp_path), tag=7, ts="1693472000.000000",
|
||||
window_ms=150, limit=1)
|
||||
assert len(result["frames"]) == 1
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Frame detail (tshark path)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
tshark_present = pytest.mark.skipif(shutil.which("tshark") is None, reason="tshark not installed")
|
||||
|
||||
|
||||
class TestDecodeFrame:
|
||||
|
||||
def _project(self, tmp_path):
|
||||
_write_pcap(tmp_path / "n1_linkA_icmp.pcap", [
|
||||
(1693472000, 123456, _icmp_frame()),
|
||||
])
|
||||
return _fake_project(tmp_path, {
|
||||
"linkA/icmp": _marker_entry(tag=7, enabled=False, node_id="n1"),
|
||||
})
|
||||
|
||||
async def test_ts_mismatch_guard_404(self, tmp_path):
|
||||
project = self._project(tmp_path)
|
||||
with pytest.raises(ControllerNotFoundError, match="rebuilt"):
|
||||
await decode_frame(project, tag=7, ts="1.000000",
|
||||
node_id="n1", link_id="linkA", marker="icmp")
|
||||
|
||||
async def test_unknown_source_404(self, tmp_path):
|
||||
project = self._project(tmp_path)
|
||||
with pytest.raises(ControllerNotFoundError):
|
||||
await decode_frame(project, tag=7, ts="1693472000.123456",
|
||||
node_id="nobody", link_id="linkA", marker="icmp")
|
||||
|
||||
async def test_decode_feeds_tshark_a_scratch_copy(self, tmp_path):
|
||||
"""Hardened tshark profiles deny the project dir — tshark must read a
|
||||
/tmp copy (a real copy, not a symlink) that is unlinked afterwards."""
|
||||
|
||||
import tempfile
|
||||
from unittest.mock import patch, AsyncMock
|
||||
|
||||
observed = []
|
||||
PDML = (b'<pdml><packet><proto name="frame" showname="Frame 1: 60 bytes">'
|
||||
b'<field name="frame.len" show="60" showname="Frame Length: 60"/>'
|
||||
b'</proto></packet></pdml>')
|
||||
|
||||
class FakeProc:
|
||||
returncode = 0
|
||||
|
||||
async def communicate(self):
|
||||
return PDML, b""
|
||||
|
||||
async def fake_exec(*args, **kwargs):
|
||||
r_index = args.index("-r")
|
||||
observed.append((args[r_index + 1], kwargs.get("env")))
|
||||
return FakeProc()
|
||||
|
||||
project = self._project(tmp_path)
|
||||
with patch("gns3server.controller.marker_replay.shutil.which", return_value="tshark"), \
|
||||
patch("gns3server.controller.marker_replay._tshark_version", AsyncMock(return_value="tshark 4.6.7")), \
|
||||
patch("gns3server.controller.marker_replay.asyncio.create_subprocess_exec", side_effect=fake_exec):
|
||||
detail = await decode_frame(project, tag=7, ts="1693472000.123456",
|
||||
node_id="n1", link_id="linkA", marker="icmp")
|
||||
|
||||
assert detail["field_count"] == 2 # proto + field from the canned PDML
|
||||
(scratch, env), = observed
|
||||
original = str(tmp_path / "n1_linkA_icmp.pcap")
|
||||
assert scratch != original
|
||||
assert scratch.startswith(tempfile.gettempdir()) and scratch.endswith(".pcap")
|
||||
assert env["HOME"] == tempfile.gettempdir()
|
||||
assert not os.path.exists(scratch) # cleaned up after the decode
|
||||
|
||||
@tshark_present
|
||||
async def test_decode_isomorphic_mapping(self, tmp_path):
|
||||
import asyncio
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
project = self._project(tmp_path)
|
||||
detail = await decode_frame(project, tag=7, ts="1693472000.123456",
|
||||
node_id="n1", link_id="linkA", marker="icmp")
|
||||
|
||||
assert detail["source"]["frame_number"] == 1
|
||||
assert detail["hex"] == _icmp_frame().hex()
|
||||
assert detail["field_count"] > 0
|
||||
assert "tshark" in detail["tshark_version"].lower()
|
||||
|
||||
# Round-trip fidelity: node count equals the PDML element count
|
||||
# (protos + fields, excluding the <packet> container itself)…
|
||||
proc = await asyncio.create_subprocess_exec(
|
||||
"tshark", "-r", str(tmp_path / "n1_linkA_icmp.pcap"), "-T", "pdml",
|
||||
stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.DEVNULL,
|
||||
)
|
||||
stdout, _ = await proc.communicate()
|
||||
packet = ET.fromstring(stdout).find("./packet")
|
||||
xml_elements = [e for e in packet.iter() if e is not packet]
|
||||
assert detail["field_count"] == len(xml_elements)
|
||||
|
||||
# …and every XML attribute survives verbatim as a JSON string key.
|
||||
def walk(element, node):
|
||||
for key, value in element.attrib.items():
|
||||
assert node.get(key) == value
|
||||
assert all(isinstance(v, str) for k, v in node.items() if k != "children")
|
||||
for child, child_node in zip(element, node["children"]):
|
||||
walk(child, child_node)
|
||||
|
||||
for element, node in zip(packet, detail["tree"]):
|
||||
walk(element, node)
|
||||
|
||||
# Values stay strings (no numeric re-typing).
|
||||
ttl = next(
|
||||
f for p in detail["tree"] if p.get("name") == "ip"
|
||||
for f in p["children"] if f.get("name") == "ip.ttl"
|
||||
)
|
||||
assert ttl["show"] == "64" and ttl["showname"] == "Time to Live: 64"
|
||||
Loading…
x
Reference in New Issue
Block a user