382 Commits

Author SHA1 Message Date
YueGuobin
0c4ec8c38c
feat(docker): eBPF stateful packet filters on kernel links
Consume the eBPF impairment classifier delivered in uBridge
feature/tc-precision: frequency_drop becomes the exact every-Nth mode
(tc nth_drop; -1 = drop everything maps to every 1st) and the new
kernel-only quota type is the byte-cap mode (tc quota_drop, 100 = hard
cutoff). KERNEL_UNSUPPORTED_FILTERS is gone entirely — kernel/relay
eligibility is now purely topological. Both modes gate on ebpf=1 from
the per-process tc capabilities probe (error names the setcap fix); the
cleanup path (modes off) only talks to the eBPF commands when the probe
reported them. Reconcile is a plain re-set: the netem reset-before-set
already detaches the program and clears uBridge's per-interface
registry, so a mode set after it is a fresh load+attach.

Documented semantic difference: the relay counts both directions
through one bridge counter, the kernel counts per direction — N=3
round-trip loss is 1-(2/3)^2 = 55.6% on kernel vs ~1/N on relay
(measured 57%). window_drop is NOT exposed: the delivered
implementation advances the window with no gap, making 'outside the
window packets pass' unreachable (needs outage/period fields —
recorded in the spec); flow_drop awaits a parameter-shape decision.

Also fixes a pre-existing restart bug the e2e exposed: _bridges (relay
bridge names) was never cleared on stop, so restarting a node with a
relay link skipped 'bridge create' and failed on add_nio_udp.

e2e (isolated instance): 13/13 — pref-1 tc_impair classifier on both
veths, 57% round-trip for N=3, -1 = exact 100%, quota hard cutoff,
delay coexistence, clean teardown, suspend/resume, node restart,
relay-leg frequency_drop regression through an Ethernet switch. Full
suite: 1925 passed.
2026-10-06 00:21:08 +08:00
YueGuobin
d4b57d49fb
feat(docker): expose netem-extension packet filters on kernel links
Surface the netem extensions delivered in uBridge feature/tc-netem-ext
as GNS3 filter types: rate (tc-style bandwidth string), reorder (needs
delay), gemodel (Gilbert-Elliot bursty loss, exclusive with
packet_loss), duplicate, seed and limit, plus the delay jitter
distribution and the loss/dup correlation parameters. All are
kernel-datapath only — the relay's packet-filter registry has no
equivalent, so available_filters hides them on relay links, setting one
on a created relay link returns 409, and the project-load path drops
them with a warning (update_filters runs before the datapath is
decided, so it cannot reject there). Validation mirrors the tc grammar:
reorder requires delay, gemodel and packet_loss are mutually exclusive,
a distribution requires jitter > 0, rate is integer+unit capped at
100gbit.

The translation in DockerVM._netem_command_parts emits the frozen
grammar order and gates extension keywords on the 'tc capabilities'
netem token list (probed once per uBridge process; a plain
delay/loss/corrupt/dup filter never probes, keeping old uBridge builds
untouched; the correl suffix has no token and gates on 'rate' as the
extension-build marker).

Reconcile became reset-before-set: the kernel's netem change merges
optional attributes (rate, correlation, reorder, corrupt, gemodel,
distribution, seed keep their previous value when the new message omits
them), so removing a parameter needs an explicit 'tc reset' — the
bpf_drop filters are re-added right after in the same flow, so the
clsact churn is invisible. The capability check runs before the reset
so a failed update leaves the previous qdisc in place.

e2e (isolated instance, kernel link): 24/24 — rate 512kbit adds exactly
2x the per-packet serialization to the RTT (45.5 ms for 1400-byte
pings), paretonormal jitter + rate-cleared leak check, reorder gap 5,
gemodel 50% bursty loss, duplicate, loss 30% correl 50%, seed 42 +
limit 5000 visible in the qdisc dump, suspend/resume and node restart
keep the filters, relay link rejects rate with 409, teardown leaves no
netem. Full unit suite: 1916 passed.
2026-10-06 00:21:08 +08:00
YueGuobin
2b45f29b2c
feat(docker): bpf packet filter on kernel links via cls_bpf match-drop
The GNS3 bpf filter (one expression per line, any line matching drops the
packet) now runs on the kernel datapath: uBridge's tc bpf_drop compiles
each line with libpcap (same compiler as the relay filter) into classic
BPF, attaches it to the veth host end's clsact egress as a cls_bpf
classifier with a gact TC_ACT_SHOT action, and the kernel migrates the
cBPF to eBPF internally -- no CAP_BPF needed.

- reconcile is flush + one add per line at priorities 10, 11, ... (OR
  semantics, same as the relay's per-line filters); tc reset (link
  delete / clear) is the full restore in uBridge and clears them
- capability-gated: the compute probes "tc capabilities" once per uBridge
  process (cached, invalidated on stop); without cbpf=1 the filter apply
  fails with an actionable upgrade error, and old uBridge builds without
  the tc module are never touched when no bpf filter is present
- a line that compiled for the controller's tcpdump but not for the
  compute's uBridge libpcap is skipped with a warning, mirroring the
  relay path
- KERNEL_UNSUPPORTED_FILTERS shrinks to frequency_drop (its eBPF stateful
  classifier is still pending, spec part B)

e2e (22/22): bpf "icmp" = 100% loss; size-discriminating expression
("greater 150") passes 98-byte and drops 300-byte pings (real byte-level
match); two-line OR; coexistence with netem (delay 50 -> small pings
100.2 ms while big dropped); clear removes clsact; suspend/resume keeps
the drops; link delete + recreate leaves no residual filters.
2026-10-06 00:21:08 +08:00
YueGuobin
0a8d430d50
feat(docker): packet filters on kernel-datapath links via tc netem
Impairment filters with a netem equivalent (delay, packet_loss, corrupt)
now run on the kernel datapath as one tc netem qdisc per veth host end,
pushed through uBridge's netlink tc module (tc netem set / tc reset).
This removes the relay delay filter's nanosleep bottleneck (upstream

Both link endpoints carry the filters: each end's qdisc egress covers
the traffic entering that container, so every direction is impaired
exactly once -- the same net effect as the relay, where both directions
cross the single filtered bridge.

- controller: eligibility keys on relay-only types (frequency_drop, bpf)
  instead of any filter; kernel links push the real filters (the
  synthetic suspend frequency_drop no longer flips a suspended link to
  relay on reopen/reset); update_filters 409s only relay-only types;
  available_filters hides them on kernel links
- compute: create_nio whitelists netem-capable filter types; the NIO
  update path keeps its own guard (PUT mutates the NIO in place);
  _ubridge_apply_netem translates the filter dict into one atomic
  netem-set (full re-apply IS the reconcile); node restart restores the
  qdisc, link deletion detaches it (the veth survives the link)
- e2e (27/27): delay 100 -> 200.2 ms RTT, atomic replace 50 -> 100.2 ms,
  clear -> baseline, packet_loss 30 -> 48 % round-trip (expected 51 %),
  suspend/resume and node restart keep the qdisc, link delete leaves no
  residual impairment
2026-10-06 00:21:08 +08:00
Mark Paronyan
3135c8b7e3
refactor: use f-strings in remaining modules, scripts and tests 2026-10-03 19:44:02 +03:00
Mark Paronyan
b622ccae7b
refactor: declare explicit public API in package __init__ modules 2026-10-03 19:43:59 +03:00
Mark Paronyan
306d8fbde4
refactor: remove unused imports 2026-10-03 19:43:58 +03:00
Mark Paronyan
634189d443 style: sort imports in api and utils modules 2026-10-02 12:43:26 +03:00
Mark Paronyan
a77e06b907
Merge branch '3.1' into ruff/trivia-fixups 2026-10-01 19:58:02 +03:00
Mark Paronyan
74943b4c74
refactor: convert remaining percent formatting in log messages 2026-10-01 19:57:47 +03:00
Mark Paronyan
95ada23395
refactor: replace percent formatting with format specifiers 2026-10-01 19:57:47 +03:00
Mark Paronyan
2dc7479d11
style: remove f-strings without placeholders 2026-10-01 19:57:46 +03:00
Mark Paronyan
396407827e
style: drop redundant open mode arguments 2026-10-01 19:57:44 +03:00
Mark Paronyan
7a1e79a8a0
style: parenthesize chained boolean operators 2026-10-01 19:57:43 +03:00
Mark Paronyan
c5b5091f5e
refactor: use explicit conversion in f-strings 2026-10-01 19:57:41 +03:00
grossmj
1456d36bf0
chore(tests): fix mypy errors 2026-10-01 15:22:17 +02:00
Jeremy Grossmann
a2cdc14a25
Merge branch '3.1' into image-manager-out-of-sync 2026-10-01 14:44:24 +02:00
Cristi
87df09c00c fix(images): Fix symlinked image imports and add regression tests 2026-09-29 20:16:22 +03:00
Cristi
2e12775911 fix(images): Fix symlink support in image inventory 2026-09-29 15:42:49 +03:00
Jeremy Grossmann
5c87c0e1a4
Merge branch '3.1' into mypy-utils 2026-09-29 13:41:58 +02:00
Jeremy Grossmann
1838ad7f05
Merge branch '3.1' into mypy-schemas-plus 2026-09-29 13:41:11 +02:00
Jeremy Grossmann
ec4b04ba54
Merge pull request #2914 from markparonyan/mypy-utils-misc
fix(typing): resolve mypy errors in utils
2026-09-29 13:01:12 +02:00
Mark Paronyan
1284701253
fix(typing): resolve mypy errors in utils.zipfile_zstd._patcher 2026-09-29 09:23:15 +03:00
Mark Paronyan
df8b3c2903
fix(typing): resolve mypy errors in utils.asyncio.aiozipstream 2026-09-29 09:23:15 +03:00
Mark Paronyan
1772c7380e
fix(typing): resolve mypy errors in utils.websocket_to_websocket 2026-09-29 09:23:15 +03:00
Mark Paronyan
5bf3c8d203
fix(typing): resolve mypy errors in utils.http_client 2026-09-29 09:23:15 +03:00
Mark Paronyan
94bf56040b
fix(typing): resolve mypy errors in utils.packet_filter_validation 2026-09-29 08:48:06 +03:00
Mark Paronyan
ab5572a0c1
fix(typing): resolve mypy errors in utils.images 2026-09-29 08:48:06 +03:00
Mark Paronyan
943bd6732c
fix(typing): resolve mypy errors in utils.encryption 2026-09-29 08:48:06 +03:00
Cristi
42d30651da Merge origin/3.1 into image-manager-out-of-sync
Preserve image reconciliation, locking, and regression coverage while integrating upstream release, dependency, typing, and formatting updates. Align feature code with the new Ruff checks.
2026-09-28 23:16:24 +03:00
Jeremy Grossmann
71965e0e81
Merge pull request #2894 from markparonyan/mypy-utils-zipfile-zstd
fix(typing): resolve mypy errors in utils.zipfile_zstd
2026-09-28 20:50:20 +02:00
Mark Paronyan
3fc1d27937
fix(typing): resolve mypy errors in utils.zipfile_zstd 2026-09-28 21:47:05 +03:00
Cristi
046dff4d08 fix(images): reconcile the image inventory with filesystem changes 2026-09-28 16:01:36 +03:00
Mark Paronyan
d1df96e0a6
fix(typing): resolve mypy errors in utils.asyncio.embed_shell 2026-09-28 04:56:18 +00:00
Mark Paronyan
a0450b6bce
refactor: ruff autofixes 2026-09-27 15:55:10 +03:00
YueGuobin
0119373690
feat: allocate IOL Docker application IDs from a pool disjoint from IOU
IOL interface MACs derive from the application ID (aabb.cc{app}{iface}),
so ids must be unique across opened projects sharing computes — the same
reason IOU has its allocator. IOL Docker nodes draw from the upper half
(512-1022, netiomux's fixed peer is 1023) so the two node types can
neither collide nor starve each other; IOU behavior is unchanged.

The controller sniffs the same GNS3_IOL_RUNNER environment marker the
compute uses to select IOLDockerVM (both the nested-properties and
template/top-level-kwarg shapes), stores the id in node properties like
IOU does, and passes it through the Docker create payload. Without an
allocation the compute falls back to a stable node-derived id in the
same upper range.
2026-09-05 21:54:52 +08:00
YueGuobin
7d3cbf1023
feat: add config read-modify-write update and harden file watcher
Config.update_config() applies submitted options to the main
configuration file via configparser read-modify-write: unknown
options are preserved, null removes an option, the merged view
of all files is validated as ServerConfig before anything is
written (a bad file would kill the FileWatcher polling loop),
and the write is atomic (.tmp + os.replace, mode 0600). Options
whose effective value is owned by a later configuration file
raise ConfigConflictError instead of writing a no-op. The
reload logic is factored into reload_and_notify() and the file
watcher callback is exception-guarded so polling never dies.
2026-08-23 18:54:23 +08:00
YueGuobin
e98c51889e
fix: notification ping starved under sustained event load
NotificationQueue.get only generated a synthetic ping when the queue
was idle for the full timeout. Under sustained event load (e.g. a
project with markers matching at 15-260 events/s) the queue never
idled, so compute notification streams never carried a ping and the
controller stopped emitting compute.updated: clients lost compute
statistics until the event flow paused or the server restarted.

A ping is now guaranteed at least every timeout seconds regardless of
event flow: when the ping deadline is reached the next get() returns a
ping ahead of queued events (pings only carry statistics, so skipping
ahead of real events is harmless). Both the compute stream (compute
CPU/memory/disk stats -> compute.updated) and the controller stream
(idle keepalive) benefit.
2026-08-22 00:50:49 +08:00
grossmj
52de01523f
Merge branch '2.2' into 3.1
# Conflicts:
#	gns3server/compute/builtin/nodes/cloud.py
#	gns3server/utils/interfaces.py
2026-08-18 18:05:56 +02:00
chara
80da7da515 add support basic support for OpenBSD.
there remains some work on ubridge, but this seems to work so far.
2026-08-13 22:24:58 +03:30
YueGuobin
0ed1715fe6
Use typed InterfaceStatus enum and simplify stats fallback in interfaces()
- Change `status: str` to `status: InterfaceStatus` (str Enum) in the
  HostInterface schema, matching the existing IPAddressFamily pattern.
- Use default-then-override pattern in interfaces() to eliminate the
  `else` branch and deduplicate defaults.
2026-07-10 14:55:21 +08:00
YueGuobin
9721660cc3
Expose full interface address list and link state in cloud node API
Each host interface surfaced by the cloud node now reports:
- ip_addresses: every IPv4 and IPv6 address (previously only a single
  IPv4 was collected internally and then dropped before the response)
- status / speed / mtu / flags: operational state and link attributes
  sourced from psutil.net_if_stats(), with flags normalized to a list

The legacy ip_address / netmask / mac_address fields are preserved so
existing callers (compute link detection, GNS3 VM, VMware, has_netmask)
keep working. The new fields travel through the existing interfaces
payload that the controller forwards verbatim, so no controller-side
change is required and the PUT / ports_mapping flow is unaffected.
2026-07-10 13:56:07 +08:00
YueGuobin
292b60efaa
Make AI features (AI Copilot + MCP) optional via [ai-features] extra
- Move fastmcp from core requirements.txt to mcp-requirements.txt
- Add MCP_AVAILABLE feature flag in agent/__init__.py (graceful degradation)
- Guard MCP imports/registration in server.py and tasks.py
- Replace ai-copilot/mcp/ai-support extras with single ai-features extra
- Add stub MCP routes returning 501 when MCP is not installed
- Add gns3server-uninstall-ai-features CLI command
- Remove old gns3server-uninstall-ai-copilot command
- Update all error messages and docs to reference ai-features

Closes #2794
2026-06-27 10:14:43 +08:00
YueGuobin
3465c39eaa
Reduce md5sum cache write failure log level from error to warning
This is a non-fatal issue that occurs during concurrent project close
when multiple nodes try to write their md5sum cache while the project
directory is being cleaned up.
2026-06-15 13:27:14 +08:00
YueGuobin
e659b64bf0
Add async_iterable_to_stream utility to avoid aiohttp compatibility issues
Create async_iterable_to_stream() in gns3server.utils.asyncio that
converts an async iterable to an aiohttp StreamReader via a background
feeder task. This bypasses aiohttp's AsyncIterablePayload which can
cause 'Connection reset by peer' with certain HTTP servers.

Use it in _run_http_query for the __aiter__ data path.
2026-06-09 23:55:59 +08:00
YueGuobin
3958279b8a
feat: add client information logging to MCP connection rejection
Add reusable utility for extracting client information from ASGI scope:
- Create gns3server/utils/request_utils.py with extract_client_info()
- Extract client IP, port, path, method, and authenticated username
- Format comprehensive log messages with client context

Benefits:
- Better observability for connection rejection events
- Reusable utility for other modules
- Consistent client logging format across codebase
- Helps diagnose timing issues during server startup

Usage:
  from gns3server.utils.request_utils import extract_client_info
  client_info = extract_client_info(scope, auth_service)
  log.warning(f"Connection rejected - Client: {client_info['host']}:{client_info['port']} ({client_info['user_info']})")
2026-06-06 22:35:43 +08:00
YueGuobin
ced73574b4
Fix delay filter validation: ensure delay: [0, X] returns proper error message
This fix addresses the issue where delay: [0, X] configurations were being
silently dropped instead of returning validation errors.

Changes:
- Created new utility function filter_inactive_filters() in packet_filter_validation.py
- Implemented smart filtering logic for delay filter that checks both latency and jitter:
  * delay: [0, 0] → User wants to disable delay, filter out silently
  * delay: [0, X] where X > 0 → Invalid config, keep for validation error
  * delay: [X, X] where X > 0 → Normal configuration, validate normally
- Simplified link.py update_filters() method to use the new utility function
- Added comprehensive tests for the new filtering logic

Before this fix:
- delay: [0, 100] would be silently dropped with no error message
- Users wouldn't know their configuration was invalid

After this fix:
- delay: [0, 100] returns proper error: "delay parameter Latency must be between 1 and 32767 ms, got: 0"
- delay: [0, 0] is correctly handled as intentional disable
- Normal delay configurations continue to work as expected
2026-05-31 22:35:58 +08:00
YueGuobin
0aa0467b02
Fix delay latency minimum: ubridge rejects latency <= 0
Align validation rules with ubridge source: delay latency must be > 0
(packet_filter.c delay_setup line 182). Update FILTERS definition in
link.py and test cases accordingly.
2026-05-30 01:21:00 +08:00
YueGuobin
7bcb96a368
Improve packet filter validation: use tcpdump, handle multi-line BPF, safe project load
Changes:
- Replace tshark BPF validation with tcpdump -d (calls pcap_compile
  internally like ubridge, returns instantly without waiting for traffic)
- Support multi-line BPF expressions: split on newlines and validate
  each line individually
- Always validate, never save invalid filters on error
- Drop invalid filters during project load with warning (prevents
  old topologies with bad filters from failing to open)
- Simplify test cases (no longer depend on tshark availability)
2026-05-30 01:02:21 +08:00
YueGuobin
87f38b1507
Add packet filter parameter validation to prevent ubridge errors
This commit implements comprehensive parameter validation for GNS3 packet
filters at the API layer, preventing invalid parameters from reaching
ubridge and causing technical errors.

## Changes

### New Features
- **Add packet filter validation module** (`gns3server/utils/packet_filter_validation.py`)
  - Validate parameter ranges for all filter types:
    - frequency_drop: -1 to 32767
    - packet_loss: 0 to 100%
    - delay: 0 to 32767ms (latency + jitter)
    - corrupt: 0 to 100%
  - **BPF syntax validation** using tshark (method from gns3_copilot)
  - Parameter count and type validation
  - User-friendly error messages with parameter details

### Integration
- **Integrate validation into Link.update_filters()** (`gns3server/controller/link.py`)
  - Validate parameters before applying filters
  - Raise ControllerError with clear error messages for invalid input
  - Prevent invalid parameters from reaching ubridge

### Testing
- **Add comprehensive test suite** (`tests/utils/test_packet_filter_validation.py`)
  - 15 test cases covering all validation scenarios
  - Tests for valid/invalid parameters, edge cases, and BPF syntax
  - All tests passing

## Benefits
- **Better UX**: Clear, actionable error messages instead of ubridge technical errors
- **Prevents crashes**: Validate parameters at API layer before reaching ubridge
- **Consistent validation**: Centralized validation logic for all filter types
- **BPF safety**: Syntax validation using tshark prevents invalid BPF expressions
2026-05-29 23:39:44 +08:00