NgTech-LTD/gns3-server
2
0 Fork 0
mirror of https://github.com/GNS3/gns3-server.git synced 2026-11-03 22:11:14 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat(web-wireshark): add implementation plan for script-driven (#2666)

Browse Source 
* feat(web-wireshark): add implementation plan for script-driven integration

Add comprehensive implementation plan for Web Wireshark integration using script-driven approach. The plan outlines:

- Background and motivation for script-based Web Wireshark integration
- Core workflow from API request to WebSocket proxy connection
- Management script architecture with WebWiresharkManager class
- Docker container configuration and resource management
- Performance analysis and optimization recommendations
- Network setup and management procedures

Key features include:
- JWT token extraction from Authorization header
- Automatic GNS3 server URL detection
- Container lifecycle management per project
- Xpra session isolation per link
- WebSocket proxy integration through gns3server
- Resource monitoring and scaling guidelines

The implementation enables users to start Web Wireshark sessions via POST requests with `wireshark: true` parameter, providing a unified web-based packet capture interface.

* feat(web-wireshark): implement script-driven Web Wireshark integration

Add complete Web Wireshark container integration with script-driven architecture:

- Create manage_wireshark.py script for Docker container management
- Add LinkCapture schema with wireshark boolean field
- Update Link class with wireshark and jwt_token parameters
- Implement WebSocket proxy endpoint for xpra HTML5 client
- Add cleanup logic in Project class for container lifecycle

Key features:
- Script-driven container and xpra session management
- JWT token extraction from Authorization header
- GNS3 URL auto-detection with Controller/Config fallback
- WebSocket proxy for unified access through gns3server
- Proper cleanup on project close/delete

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(web-wireshark): use aiohttp instead of docker SDK

Replace docker-py SDK with direct Docker HTTP API calls using aiohttp,
following GNS3's existing architecture pattern in gns3server/compute/docker/.

Changes:
- Create DockerHTTPClient class using aiohttp + Unix socket
- Implement all Docker operations as async methods
- Remove dependency on docker Python SDK
- Use Docker API v1.44 via /var/run/docker.sock

Benefits:
- No additional dependencies required
- Consistent with GNS3's async architecture
- Better integration with existing codebase

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(web-wireshark): add dynamic Docker API version detection

Implement dynamic Docker API version detection following GNS3's pattern
in gns3server/compute/docker/__init__.py.

Changes:
- Add DOCKER_MINIMUM_API_VERSION and DOCKER_PREFERRED_API_VERSION constants
- Remove hardcoded "v1.44" prefix (now using "1.44" format)
- Add _check_connection() method to detect Docker daemon version
- Dynamically select API version based on daemon capabilities
- Add proper error handling for version mismatches

Behavior:
- Initialize with minimum API version (1.40)
- On first connection, detect Docker daemon API version
- Use preferred API version (1.44) if supported
- Fall back to daemon's minimum API version if needed
- Raise error if daemon version is too old

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(web-wireshark): add optional --verbose logging parameter

Remove hardcoded logging.basicConfig() and add --verbose parameter
to control log output, following GNS3's logging pattern.

Changes:
- Remove logging.basicConfig(level=logging.INFO) from module level
- Add --verbose/-v command line parameter
- Configure logging only when --verbose is specified
- Use structured log format with timestamp when verbose

Behavior:
- When called by GNS3: uses GNS3's logging configuration (default)
- When run standalone: no output unless --verbose is specified
- With --verbose: shows detailed logs with timestamps

This prevents the script from interfering with GNS3's logging
configuration while still allowing debug output when needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(web-wireshark): fix code quality issues from static analysis

Fix issues identified by flake8 and pylint static analysis:

Import and formatting fixes:
- Remove unused 'time' import
- Fix import order (stdlib before third-party)
- Remove unnecessary f-strings without placeholders
- Split long line to comply with flake8 (max line length: 127)

Code structure improvements:
- Remove unnecessary 'else' after 'return'
- Add 'from e' to exception re-raising for better tracebacks

Code quality metrics:
- pylint score: 8.11/10 → 9.97/10 (+1.86)
- flake8: 0 errors (max line length: 127, per CI/CD standard)
- All critical issues resolved

Note: R0914 (too-many-locals) warning is informational only,
code remains clear and maintainable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): export LinkCapture schema in schemas module

Add LinkCapture to the schemas module exports in __init__.py to fix
the AttributeError when starting the GNS3 server.

Error was:
  AttributeError: module 'gns3server.schemas' has no attribute 'LinkCapture'

This was caused by adding the LinkCapture class to controller/links.py
but forgetting to export it in the schemas __init__.py.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web-wireshark): add --image parameter for custom Docker images

Add optional --image parameter to allow using custom Docker images
for testing, instead of requiring gns3/web-wireshark:latest.

Changes:
- Add --image parameter to start command (default: gns3/web-wireshark:latest)
- Update get_or_create_container() to accept image parameter
- Update start_wireshark_session() to accept and pass image parameter
- Update TEST.md with examples of using custom images

Usage:
  # Using default image
  python3 manage_wireshark.py start --project-id x --link-id y --jwt-token z

  # Using custom image (for testing)
  python3 manage_wireshark.py start --project-id x --link-id y --jwt-token z --image ubuntu:latest

This makes it easier to test the script without having the official
gns3/web-wireshark image available.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web-wireshark): add configurable resource limits and increase session capacity

Add configurable resource parameters and increase session support from 10 to 100.

Resource parameters:
- --memory: Memory limit (default: 2g)
- --memory-swap: Memory swap limit (default: same as memory)
- --cpus: CPU cores (default: 1.0)
- --pids-limit: Process limit (default: 1000)

Other improvements:
- Fix CPU quota calculation: 1000000 microseconds = 1.0 CPU core
  (was incorrectly 100000 = 0.1 CPU core)
- Add health check: xpra list command
- Add log configuration: json-file, max-size=10m, max-file=3
- Increase session capacity: 10 → 100 concurrent sessions
- Port range: 12300-12399 (was 12300-12309)
- Display range: :100-:199 (was :100-:109)

This matches the docker run command parameters and provides
better resource management and monitoring capabilities.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): update UDPLink.start_capture() to accept new parameters

Update UDPLink.start_capture() to accept the new wireshark and jwt_token
parameters and pass them to the parent Link class.

This fixes the TypeError when starting capture on UDP links:
  TypeError: UDPLink.start_capture() got an unexpected keyword argument 'wireshark'

The UDPLink class overrides start_capture() but didn't include the new
parameters added for Web Wireshark support.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs: add project memory infrastructure and JWT token flow documentation

- Add memory skill for recording project knowledge
- Document JWT token flow in Web Wireshark integration
- Update .gitignore to track skills and memory directories

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): fix Docker API URL format and use docker exec CLI

- Add "v" prefix to Docker API URL (http://docker/v{version}/...)
- Remove unused exec_create/exec_start API methods
- Fix Healthcheck.Test format to ["CMD-SHELL", "command"]
- Use docker exec CLI instead of Docker API for command execution
- This aligns with GNS3 docker_vm pattern

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): improve xpra session management and error handling

- Check and clean up existing sessions before starting new xpra session
- Add verification that xpra session started successfully
- Use docker exec CLI consistently instead of Docker API
- Improve error logging and diagnostics

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web-wireshark): add timeout handling and container health checks

- Add REQUEST_TIMEOUT constant for Docker HTTP API requests
- Add asyncio.timeout wrapper for Docker API calls to prevent hanging
- Add _is_container_healthy() to check container responsiveness
- Add _exec_in_container() helper with timeout support
- Improve unhealthy container handling with force remove
- Add detailed logging for container health state

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(web-wireshark): split monolithic file into modules

Split manage_wireshark.py into:
- docker_client.py: Docker HTTP API client
- manager.py: WebWiresharkManager with session management
- manage_wireshark.py: CLI entry point

This improves code organization and maintainability.
Keep all timeout handling and health check logic.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): correct NanoCpus calculation to use nanoseconds

NanoCpus in Docker API requires nanoseconds (1 CPU = 1000000000),
not the previous incorrect multiplier of 100000.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): fix xpra command quoting and session verification

- Add quotes around --xvfb parameter value to preserve spaces
- Fix session verification to check display number instead of session name
  (xpra list shows "LIVE session at :185" not session name)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): run wireshark in background with &

Wireshark with curl pipeline runs continuously, so it must run
in background to avoid blocking. Also reduced timeout since we
don't wait for completion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web-wireshark): add deterministic hash for display/port allocation

Use MD5-based hash instead of Python's hash() which is randomized
across process restarts. This ensures display and port numbers are
stable when the GNS3 server restarts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: extract link_id_to_port to shared utils module

Move deterministic hash functions to gns3server/utils/port_allocator.py
to avoid code duplication and ensure consistent algorithm across
manager.py and links.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(links): move import to top of file

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(web-wireshark): use correct Config access pattern like gns3-copilot

The Server config is an object with .protocol.value, .host, .port
attributes, not a dict. Fixes URL detection failing and falling back
to 127.0.0.1:3080 which doesn't work from inside containers.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add UUID validation utility and use in manage_wireshark

Create gns3server/utils/uuid_validator.py with validate_uuid function
to validate UUID format and catch typos early with clear error messages.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: use argparse.ArgumentTypeError for proper error message display

Previously ValueError was used which only showed 'invalid validate_uuid value'.
Now shows the full helpful message about expected format.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web-wireshark): add --dpi=96 to xpra start parameters

Set standard DPI for better display scaling in web Wireshark.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: update TEST.md with correct project-id format

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web-wireshark): hide xpra shutdown menu via XPRA_CLIENT_CAN_SHUTDOWN

Prevents users from accidentally shutting down the xpra server
through the HTML5 client menu.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(memory): add xpra-html5-client configuration reference

Document xpra HTML5 client configuration including:
- URL parameters for toolbar menu control
- default-settings.txt parameters (Features, Connection, Advanced)
- Server-controlled submenu items
- XPRA_CLIENT_CAN_SHUTDOWN environment variable
- Background image customization

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web-wireshark): remove xpra stop cleanup to avoid zombie processes

xpra stop can leave zombie processes and timeout. Instead, let xpra start
reuse the display directly (it will overwrite existing session).

Also added --file-transfer=no, --printing=no, --sound=no to disable
unneeded features.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(web-wireshark): add fullscreen_button=false to default-settings.txt

Also remove --file-transfer/--printing/--sound from command line since
xpra doesn't support these options. Configure in default-settings.txt instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web-wireshark): update xpra background with GNS3 branding

- Replace default xpra background with GNS3 icon and modern gradient
- Update CSS to use SVG background image with cover sizing
- Change background color to gradient from #021d3a to linear gradient
- Improves visual integration with GNS3 web interface

* style: add GPLv3 headers with copyright to web_wireshark modules

Co-Authored-By: YueGuobin <yueguobin@gmail.com>

* feat(docker): use Alibaba Cloud Debian mirror for faster builds in China

Use mirrors.aliyun.com for Debian packages with correct paths for
both main repo (/debian) and security repo (/debian-security).

Co-Authored-By: YueGuobin <yueguobin@gmail.com>

* feat(web-wireshark): update xpra background styling and echo command

- Replace `echo -e` with `printf` for better POSIX compatibility in Dockerfile
- Update xpra HTML5 client background to use GNS3 icon with light gradient
- Change background color from dark blue to light gray for improved visibility

* fix(web-wireshark): use pkill to stop xpra sessions

Replace 'xpra stop :{display}' with 'pkill -f "xpra.*:{display}"' for more reliable session termination.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web-wireshark): start Wireshark in fullscreen mode

Add --fullscreen flag to Wireshark startup command to prevent window decorations and improve user experience.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): use carrier-grade NAT subnet to avoid conflicts

Change Docker network subnet from 172.28.0.0/16 to 100.64.1.0/24 to avoid conflicts with common private networks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(web-wireshark): update subnet to /22 for 1000+ projects

Change Docker network subnet from 100.64.1.0/24 to 100.64.0.0/22 to support 1000+ projects (1022 available IPs). Update all documentation to reflect the new subnet.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): replace CGNAT subnet with configurable private subnet

Replace the CGNAT address block (100.64.0.0/22) with a standard private
subnet (172.31.0.0/22) to avoid network access issues. Many networks and
ISPs block or cannot route CGNAT addresses.

Changes:
- Add WebWiresharkSettings to config schema with configurable subnet
- Read network_subnet from config, default to 172.31.0.0/22
- Add Web Wireshark configuration section to sample config

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): comment out dpi setting

Allow xpra to use default DPI settings for better display scaling.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): set DPI in Xvfb to prevent scaling warnings

Add -dpi 96 to Xvfb startup command to match xpra's expected DPI
and avoid "scaling problems" warnings.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): remove custom Xvfb to use default Xorg-dummy

Remove --xvfb parameter to let xpra use the default Xorg-dummy driver,
which properly handles DPI changes when resize-display is enabled and
prevents DPI mismatch warnings.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* revert(web-wireshark): restore Xvfb configuration

Restore the Xvfb display server configuration for xpra sessions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web-wireshark): enhance xpra HTML5 default settings

Update xpra HTML5 configuration to disable additional unused features:
- audio, keyboard, mediasource, aurora, http-stream
- Improve readability using heredoc format instead of printf

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(api): add RBAC authentication to web wireshark WebSocket endpoint

- Add `has_privilege_on_websocket` dependency to enforce Link.Capture privilege
- Include current_user parameter in endpoint for user identification and logging
- Update endpoint documentation to reflect token requirement and privilege
- Add test example WebSocket URL in TEST.md for reference

* refactor(web-wireshark): add generic WebSocket proxy and disable xpra HTML

- Add websocket_to_websocket.py utility for binary WebSocket proxy
- Disable xpra HTML server with --html=no flag
- Update manager return value: url -> ws_url
- Simplify WebSocket proxy implementation in links.py

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(web-wireshark): remove unnecessary sleep delays

Remove 2-second sleep delays after container start and xpra initialization.
Health checks and verification happen immediately, improving startup time.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(agent): prevent blocking during project close

- Add 5s timeout for AgentService.checkpointer_conn.close()
- Optimize stop_all_sessions: single command instead of 100 docker exec calls
- Prevents indefinite hang when closing projects

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web-wireshark): return errors to client on startup failure

- Throw ControllerError when Web Wireshark startup fails
- Include detailed error messages from script stderr
- Support both ws_url and url response formats
- Client now receives proper error response instead of success

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(web-wireshark): parallelize startup and remove unnecessary waits

- Parallel execution: container info query + xpra start
- Remove xpra list verification (unnecessary)
- Fire-and-forget Wireshark startup (no timeout wait)
- Startup time reduced from ~7s to ~1s

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web-wireshark): add stop-container and delete-container commands

- Add stop-container command to stop container on project close
- Add delete-container command to delete container on project delete
- Enables proper container lifecycle management
- Containers are now stopped (not deleted) when project closes
- Containers are deleted when project is deleted

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(project): simplify close flow by removing redundant xpra session cleanup

Since docker stop with timeout=0 already force-kills all processes in the
container, explicitly stopping xpra sessions before stopping the container
is unnecessary. Remove the _cleanup_web_wireshark_xpra_sessions() call
to reduce subprocess overhead and simplify the close flow.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* perf(docker): force kill containers on stop by default

Change docker stop timeout from 10 seconds to 0 (immediate SIGKILL).
Web Wireshark containers don't need graceful shutdown since they have
no persistent state.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(web-wireshark): enable WebSocket protocol for xpra connection

- Change xpra bind from --bind-tcp to --bind-ws for WebSocket support
- Enable HTML client with --html=on for xpra WebSocket server
- Pass binary subprotocol when connecting to xpra container

The xpra WebSocket server requires the 'binary' subprotocol during
handshake. Without it, the server returns 403 with message:
"client does not support 'binary' protocol".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(web-wireshark): add WebSocket subprotocol negotiation support

Fix WebSocket proxy for xpra container by implementing proper subprotocol
negotiation. The xpra client requires the server to respond with the
negotiated subprotocol (binary) in the WebSocket handshake response.

Changes:
- Modified get_current_active_user_from_websocket to extract client's
  requested subprotocols from Sec-WebSocket-Protocol header
- Updated websocket.accept() call to include negotiated subprotocol
- Enhanced websocket_proxy to support subprotocol parameter
- Improved logging for WebSocket connection debugging

This fixes the issue where xpra clients would immediately disconnect
after connection due to missing subprotocol in server response.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore(web-wireshark): remove verbose debug logging

Remove excessive debug logs from WebSocket proxy implementation while
keeping essential logs for troubleshooting:
- Keep: subprotocol negotiation, connection establishment, errors
- Remove: verbose step-by-step debugging information

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(web-wireshark): translate Chinese comments and docs to English

- Translate all Chinese comments in project.py, link.py, and links.py to English
- Convert TEST.md and maunal-test.md documentation to English
- Remove obsolete IMPLEMENTATION_PLAN.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore(docker): lock xpra version to 6.4.3 for reproducible builds

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore(docker): improve Dockerfile with TZ, no-install-recommends, and labels

- Add TZ=UTC timezone setting
- Use --no-install-recommends to reduce image size
- Add LABEL metadata for maintainer and description

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(docs): rename and consolidate documentation

- Rename TEST.md to WEB_WIRESHARK.md for clearer naming
- Merge maunal-test.md content into main document under "Manual Testing" section
- Remove obsolete gns3_icon_black.svg

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(docker): add xpra-x11 package and remove --no-install-recommends

- Add xpra-x11 package required for seamless mode
- Remove --no-install-recommends to ensure all recommended packages are installed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(api): add capture file download endpoint

Add GET /{link_id}/capture/file endpoint to download PCAP capture files.
Supports downloading while capture is active (streaming).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(web_wireshark): improve session cleanup and add known issues

- Add cleanup of existing processes before starting xpra session to prevent
  "another window manager seems to be running" errors
- Stop all associated processes (xpra, wireshark, Xvfb) when stopping sessions
- Document known issues in WEB_WIRESHARK.md including JWT token security,
  duplicate code, and other implementation concerns

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web_wireshark): add restart wireshark API endpoint

- Add POST /links/{link_id}/capture/wireshark/restart endpoint
- Add restart command to manage_wireshark.py CLI
- Add _restart_web_wireshark method in link.py controller
- Restart simply calls start_wireshark_session which handles cleanup

This allows users to recover after accidentally closing the Wireshark
window without having to stop and restart the entire capture.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add Web Wireshark business process documentation

Document the Web Wireshark feature including architecture diagrams,
business processes for capture start/stop, container lifecycle,
WebSocket connection flow, and session management.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(web_wireshark): fix container network access and improve session cleanup

- Fix GNS3 server URL detection to use container gateway IP (172.31.0.1)
  instead of localhost/127.0.0.1/0.0.0.0 which don't work from containers
- Add import socket module for gateway IP conversion
- Move urllib.parse import to file header (code style improvement)
- Add logging for Web Wireshark startup (capture stream URL, display, etc.)
- Fix X lock file cleanup to prevent "Server is already active" errors
- Use exec pkill to reduce zombie processes from docker exec bash
- Add _cleanup_x_lock() method to remove X lock files after stopping sessions
- Update link.py to always log subprocess stderr for debugging

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web_wireshark): avoid zombie pkill processes by using Docker API

Replace pkill commands with Docker API-based process killing to prevent
zombie process accumulation in Web Wireshark containers.

Changes:
- Add DockerHTTPClient.list_processes() method to query container processes
- Rewrite _kill_process_tree() to use Docker API instead of pkill
- Unify all process cleanup to use _kill_process_tree() method
- Add logging for killed processes (count and PIDs)
- Include fallback to pkill if Docker API fails

This prevents the accumulation of zombie pkill processes that occurred
when using bash -c "pkill -9 -f pattern" commands.

Testing: Started and stopped multiple capture sessions, verified no new
pkill zombie processes are created during cleanup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web_wireshark): use docker-init and cleanup socket files

Enable Docker init system (tini) as PID 1 and clean up xpra socket
files when stopping sessions to prevent zombie processes and leftover
sockets.

Changes:
- Add "Init": True to container host_config to use docker-init (tini)
- Extend _cleanup_x_lock() to remove xpra socket files
  * /run/user/1000/xpra/{display}/socket
  * /run/user/1000/xpra/*-{display}
  * /home/gns3/.xpra/*-{display}

Benefits:
- docker-init (tini) automatically reaps orphan processes, eliminating
  zombie process accumulation (tested: 53 zombies -> 0 zombies)
- Socket cleanup prevents xpra list from showing UNKNOWN sessions
- Cleaner container state after stopping sessions

Testing:
- Started and stopped 3 capture sessions
- Verified 0 zombie processes after stopping
- Verified xpra list shows "No xpra sessions found" (no UNKNOWN)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(web_wireshark): add comprehensive CLI script documentation

Add detailed module docstring to manage_wireshark.py explaining:
- Script purpose: CLI tool for manual management, debugging, and testing
- Important note: Use WebWiresharkManager directly for programmatic access
- Usage examples for all commands (start, stop, restart, stop-all, delete)
- Available commands list with descriptions
- Output format specification (JSON to stdout/stderr)
- Help information for getting command-specific usage

This clarifies that the script is intended as a CLI utility, not for
subprocess calls from within GNS3 server code.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(web_wireshark): use direct API calls instead of subprocess

Replace subprocess calls to manage_wireshark.py with direct API calls
to WebWiresharkManager, eliminating subprocess overhead and JSON parsing.

Changes:
- Add import for WebWiresharkManager
- Refactor _start_web_wireshark() to use direct API call
- Refactor _stop_web_wireshark() to use direct API call
- Refactor _restart_web_wireshark() to use direct API call

Benefits:
- Code reduction: 78 lines (-68%)
- Better performance: No subprocess creation overhead
- Better logging: Manager logs directly to GNS3 logging system
- Simpler error handling: Direct exceptions instead of return codes
- No JSON parsing: Direct Python objects
- Resource cleanup: Added finally blocks to ensure manager.close()

Testing:
- All Web Wireshark operations work correctly
- Logs now appear in GNS3 server logs instead of stderr
- Error handling improved with proper exception propagation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web_wireshark): resolve circular import with delayed imports

Fix circular import error by moving WebWiresharkManager import from
module level to function level (delayed import).

Circular dependency was:
  link.py → WebWiresharkManager → Controller → link.py

Solution:
  - Remove top-level import of WebWiresharkManager
  - Add delayed imports in each method that uses it:
    * _start_web_wireshark()
    * _stop_web_wireshark()
    * _restart_web_wireshark()

This allows the controller module to fully initialize before importing
WebWiresharkManager, breaking the circular dependency.

Error before fix:
  ImportError: cannot import name 'Controller' from partially initialized
  module 'gns3server.controller' (most likely due to a circular import)

Testing:
  - Successfully imports Link module
  - GNS3 server starts without import errors

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor: move imports to module level and remove Controller dependency

Remove circular dependency between link.py and manager.py by:

1. Remove Controller import from manager.py
   - Manager no longer imports Controller
   - URL detection now relies on Config or default only
   - link.py passes capture_stream_url to manager

2. Move WebWiresharkManager import to module level in link.py
   - No longer need delayed imports since no circular dependency
   - Clean, standard Python import pattern

3. Remove redundant Config import in ensure_network()
   - Config was already imported at module level

Changes:
- manager.py: -24 lines (removed _get_gns3_url_from_controller and redundant import)
- link.py: -30 lines net reduction after moving import to top

Testing:
- All imports work without circular dependency errors
- GNS3 server starts successfully

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(docker_client): use text response for list_processes API

Docker API /containers/{id}/top returns plain text, not JSON.
The generic _request method auto-parses JSON which caused:
  "'dict' object has no attribute 'strip'"

Fix by using session.get() directly with response.text()
instead of the generic _request method.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web_wireshark): use correct capture stream URL endpoint

Remove passing capture_stream_url from link.py to manager.py.
Manager.py auto-detects URL using correct endpoint:
  /v3/projects/{project_id}/links/{link_id}/capture/stream

This endpoint supports JWT authentication, unlike the compute URL
(/v3/compute/...) which requires compute credentials.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web_wireshark): use container-local PIDs for process killing

- Docker API returns host PIDs, not container PIDs - use pgrep inside
  container to get correct PIDs for kill
- Fix list_processes to parse Docker API JSON response correctly
- Remove unnecessary exec prefix in shell commands
- Add logging for debugging process killing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* perf(web_wireshark): optimize process killing with single pgrep call

Replace sequential _kill_process_tree calls with a new _kill_process_tree_batch
method that combines all patterns into a single regex. This reduces docker exec
calls from 8 to 1, improving performance from ~8 seconds to ~0.9 seconds.

Changes:
- Add _kill_process_tree_batch() method with combined regex pattern matching
- Update start_wireshark_session() to use batch cleanup
- Update stop_wireshark_session() to use batch cleanup
- Update stop_all_sessions() to use batch cleanup

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(web_wireshark): disable HTML5 client to improve xpra startup speed

Change --html=on to --html=off to disable the xpra HTML5 client interface.
This reduces xpra startup time from ~6.4s to ~2.9s (55% improvement) while
keeping the WebSocket server functional for browser connections.

The HTML5 client is not needed as we only require the WebSocket endpoint
for remote display forwarding.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(web_wireshark): optimize container health check using Docker built-in status

Replace manual docker exec ping with Docker's built-in health check status
to reduce container startup latency by ~1 second.

Changes:
- Use container["Health"]["Status"] instead of manual ping check
- Skip health check for containers with "healthy" or "starting" status
- Only verify containers with "unhealthy" status
- Keep manual health check for newly started containers

This reduces the container verification time from ~1s to near-zero
for healthy running containers while maintaining safety for unhealthy ones.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(web_wireshark): get gateway IP from Docker API instead of container exec

Replace slow docker exec method with fast Docker Network API call to get
gateway IP. This reduces gateway detection from ~850ms to ~1ms (1000x faster).

Changes:
- Use docker.get_network() API to get gateway from IPAM config
- Keep container exec methods as fallback if Docker API fails
- Remove container_id requirement (not needed for Docker API method)

The Docker network gateway is shared by all containers in the network,
so querying the network is more efficient than querying each container.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(web_wireshark): remove unreliable nameserver fallback for gateway detection

Remove the fallback method that reads /etc/resolv.conf nameserver as gateway.
This method is unreliable because:
- nameserver is not necessarily the gateway (could be upstream DNS)
- Many systems use 127.0.0.53 (systemd-resolved) or 127.0.0.1
- Even when not local, it could be public DNS (8.8.8.8, 1.1.1.1)

The current fallback strategy is sufficient:
1. Docker Network API (fast, reliable)
2. /proc/net/route (standard gateway detection)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(web_wireshark): use host perspective for process killing (37x faster)

Replace docker exec with host perspective process management for killing
container processes. This reduces process cleanup from ~850ms to ~23ms.

Key optimization:
- Get container init PID using docker inspect (fast)
- Use pgrep -P <init_pid> to find child processes from host perspective
- Kill processes directly using host PID (no docker exec needed)

Performance improvements:
- Process finding: 850ms → 23ms (37x faster)
- Process killing: 850ms → <1ms (850x faster)
- File checking: 850ms → 0.003ms (280,000x faster)

Fallback to docker exec method if host perspective fails.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(web_wireshark): use host perspective for file cleanup (891x faster)

Replace docker exec with direct filesystem access from host perspective
for cleaning up X lock files and xpra sockets. This reduces cleanup time
from ~890ms to ~1ms (891x faster).

Key optimization:
- Access /proc/<container_pid>/root/ directly from host
- Use os.remove() and glob.glob() instead of docker exec rm -f
- Fallback to docker exec if host perspective fails

This complements the earlier optimization for process killing, further
reducing startup time.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Revert "perf(web_wireshark): use host perspective for file cleanup (891x faster)"

* fix(web_wireshark): recursively kill all descendant processes to prevent orphans

Previous implementation only killed direct children of container init using
`pgrep -P <init_pid>`, but xpra spawns child processes (Xvfb, pulseaudio,
ibus-daemon) that become grandchildren and were not being terminated.

This fix walks the entire process tree recursively to find and kill all
descendant processes, ensuring complete cleanup without orphans.

Performance: ~20-40ms (only 10-20ms slower than previous 23ms, but ensures
thorough cleanup).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* docs(web_wireshark): add comprehensive startup/shutdown performance metrics

Add detailed performance characteristics section documenting:
- Startup performance breakdown (before/after optimization)
- Shutdown performance breakdown (before/after optimization)
- First startup vs subsequent startup comparison
- Measured production data
- Key optimization techniques

Performance improvements: 67% faster startup, 78% faster shutdown,
zero orphan processes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web_wireshark): ensure container stops on project close

Remove dependency on _web_wireshark_container_created flag which gets
reset when project is reloaded, causing container to not stop on project close.

Now directly attempts to stop container on every project close. If container
doesn't exist, the script handles it gracefully. This is simpler and more
reliable than maintaining a flag.

Also removed unnecessary _cleanup_web_wireshark_xpra_sessions() call since
stopping the container automatically terminates all xpra sessions inside.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web_wireshark): ensure container deletion on project delete

Remove dependency on _web_wireshark_container_created flag from
_cleanup_web_wireshark_container() to ensure container is deleted
when project is deleted, even if project was reloaded and flag was reset.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(web_wireshark): remove unused _web_wireshark_container_created flag

This flag was set but never read for any conditionals since we changed
the stop/delete methods to directly attempt the operation. It's purely
redundant code that adds confusion.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(web_wireshark): skip unnecessary cleanup using fast host perspective check

Add _check_residuals_exist() function that uses host perspective (~20ms)
to check if residual processes or socket files exist before cleanup.

Returns (has_process_residuals, has_socket_residuals) tuple to allow
selective cleanup - if only sockets need cleaning, skip the slower
process tree killing.

For new containers with no residuals, this saves ~3 seconds of
unnecessary docker exec calls.

Socket cleanup always uses docker exec for safety (avoiding accidental
host filesystem deletion from /proc/<pid>/root/ path errors).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(web_wireshark): increase container exec timeout from 5s to 10s

xpra start with Xvfb initialization can take longer than 5 seconds in Docker containers, causing timeout failures. Increasing timeout to 10 seconds to allow sufficient startup time.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* perf(web_wireshark): optimize cleanup and document docker exec limitations

- Combine X lock and xpra socket cleanup into single docker exec call
  Reduces exec calls from 2 to 1 per stop operation, improving performance
  when stopping multiple capture sessions.

- Document Docker exec performance limitations with test data showing
  parallel exec is 42% slower than serial due to Docker daemon's
  internal queuing mechanism.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(links): prevent AssertionError in stream_pcap during rapid stop_capture

Fix race condition where stream_pcap checks link.capturing (True) but
link.capture_node becomes None before accessing link.compute.

This happens when stop_capture() is called and rapidly cleans up
_capture_node while WebSocket requests are still processing.

Now check both capturing and capture_node to ensure capture is still active.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(links): prevent AssertionError in stream_pcap during rapid stop_capture

Fix race condition where stream_pcap checks link.capturing (True) but
link.capture_node becomes None before accessing link.compute.

This happens when stop_capture() is called and rapidly cleans up
_capture_node while WebSocket requests are still processing.

Now check both capturing and capture_node to ensure capture is still active,
and log at DEBUG level since this is expected behavior during rapid stop.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web_wireshark): load container config from settings

- Load memory, cpus, and pids_limit from WebWireshark config section
- Apply config only on container creation (_start_web_wireshark)
- Skip config on restart (_restart_web_wireshark) as container exists

Users can now configure container resources in gns3_server.conf:

[WebWireshark]
memory = 4g
cpus = 2.0
pids_limit = 2000

If not configured, defaults (2g, 1.0, 1000) are used.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(web_wireshark): add container statistics to /v3/statistics endpoint

Add Web Wireshark container monitoring to the statistics API for
better visibility into packet capture sessions.

Changes:
- Create new stats.py module with collect_webwireshark_stats()
  * Collects container info (status, project, active sessions)
  * Gets resource limits (memory, CPU, PIDs) from container config
  * Gets resource usage via docker stats (memory, CPU, PIDs)
  * Properly closes aiohttp connections to avoid leaks

- Update /v3/statistics endpoint to include webwireshark data
  * total_containers: Total number of Web Wireshark containers
  * running_containers: Number of containers currently running
  * active_sessions: Total active capture sessions
  * containers: Array with per-container details

- Update statistics-api.md documentation
  * Change URL from /v1/statistics to /v3/statistics
  * Add webwireshark field descriptions and examples
  * Add dashboard integration for Web Wireshark monitoring

Example response:
{
  "webwireshark": {
    "total_containers": 1,
    "running_containers": 1,
    "active_sessions": 2,
    "containers": [{
      "project_id": "...",
      "project_name": "test",
      "container_id": "6edc9029bac0",
      "status": "running",
      "memory_limit": "4.0 GB",
      "cpu_limit": "4.0",
      "pids_limit": 4000,
      "memory": "535.7MiB / 4GiB",
      "cpu": "0.29%",
      "pids": 124
    }]
  }
}

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor(project): use WebWiresharkManager directly instead of subprocess

Replace subprocess calls to manage_wireshark.py with direct
WebWiresharkManager method calls in project.py. This eliminates
unnecessary process overhead and maintains consistent usage pattern
with link.py.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(logging): use %-formatting instead of f-strings in WebWireshark methods

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(web_wireshark): use info level log when container not found

Change log level from warning to info when container doesn't exist
during stop/delete operations, and fix f-string without placeholders.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(logging): lower log level for client disconnection

Change log level from warning to debug when client possibly
disconnects, as this is normal behavior when users close
browser tabs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(linting): resolve ruff warnings and errors

- Remove unused variable proc (fire-and-forget subprocess)
- Remove duplicate nodes property definition
- Remove extraneous f-prefix from f-strings without placeholders
- Change bare except to except Exception
- Remove unused imports (sys, asyncio, json)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web_wireshark): improve error message when Docker image not found

When creating a container fails due to missing image, provide helpful
error message with docker pull command and local build instructions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(schema): add wireshark field to Link schema

Add wireshark boolean field to Link Pydantic schema so it is
included in API responses when Web Wireshark session is active.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(link): add wireshark state tracking

Track whether Web Wireshark is running on a link by adding
_wireshark boolean property, updated on start/stop operations.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(chat): add session abort functionality

Add ability to abort streaming chat sessions via REST API.

Changes:
- Add abort flag to MessagesState for tracking abort requests
- Add session_id to state for abort event correlation
- Add _abort_flags dict and check/set/clear functions in gns3_copilot.py
- Add abort_handler_node to generate aborted tool messages
- Modify should_continue to route to abort_handler_node when aborting
- Add abort_session() method in AgentService
- Add POST /sessions/{session_id}/abort API endpoint
- Clear abort flag at stream start

When abort is triggered during streaming:
- If LLM has tool_calls pending, abort_handler_node generates
  aborted tool messages to maintain checkpoint consistency
- Prevents "insufficient tool messages" error on resume

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(chat): send tool_end event when stream is aborted

When a stream is aborted during tool execution, yield proper tool_end
events for aborted tools instead of a generic abort event. This maintains
compatibility with the frontend's expected tool_start/tool_end flow.

Changes:
- Add stream_aborted tracking flag
- After stream loop, check abort flag and yield tool_end events
- Add "abort" to ChatResponse type enum (unused but available)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(web_wireshark): add gns3-wireshark-setup command for Docker image setup

Add a new entry point script that allows users to setup the Web
Wireshark Docker image with a single command:

    pip install gns3-server && gns3-wireshark-setup

The script will:
1. Try to pull gns3/web-wireshark:latest from Docker Hub
2. If pull fails, build the image locally using the included Dockerfile
3. Show raw docker pull/build output for full visibility

Files changed:
- Add setup_wireshark_image.py (new entry point script)
- Update pyproject.toml (add gns3-wireshark-setup entry point)
- Update documentation (README.md, WEB_WIRESHARK.md, web-wireshark-business-process.md)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(appliance): add tags field support for appliances and templates

- Add tags field to ApplianceV1_6 and ApplianceV8 schemas
- Add tags propagation from appliance config to template
- Simplify VPCS builtin template tags

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix test in test_link.py

* Fix link.py after failed tests

* feat(copilot): add device skills system for LLM context injection

- Add skills module with registry and DeviceSkillsTool
- Skills organized by vendor/series for easy extensibility
- Support device_type, category (device/protocol/feature), operation (config/diagnosis)
- First implementation: VPCS skill (gns3_vpcs_telnet)
- Skills tool integrated into teaching_assistant and lab_automation modes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(vpcs): change prompt detection log from warning to debug

The "VPCS prompt not clearly detected" message is not an error,
connection works fine. Change to debug level to reduce noise.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(memory): add uBridge permission issue documentation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(copilot): include link_id in links_summary output

Return link_id in the links_summary method so that the AI can
identify which link to analyze when using the packet capture tool.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(copilot): add packet capture analysis tool

Add PacketCaptureTool that allows AI to analyze packets from an active
GNS3 capture. The tool downloads capture files from the GNS3 server
and runs tshark analysis to help users understand network traffic.

Features:
- Download capture file from /capture/file endpoint
- Run tshark with custom arguments for flexible analysis
- Support analyzing specific packets (e.g., "explain packet #42")
- Support protocol statistics, traffic analysis, and expert info

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(copilot): use shlex.split to properly parse quoted tshark arguments

tshark_args may contain quoted filter expressions like "-Y \"frame.number == 24\"".
Using plain str.split() breaks these quoted arguments, causing tshark warnings
about conflicting display filters. Use shlex.split() to correctly parse quoted
arguments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(copilot): simplify PacketCaptureTool to accept only packet_number

Instead of exposing complex tshark arguments to the LLM, the tool now
accepts a simple packet_number parameter and internally constructs the
tshark command with verbose output. This prevents the LLM from generating
incorrect tshark parameters while still providing detailed packet analysis.

Changes:
- Remove tshark_args and max_lines parameters
- Accept only packet_number as input
- Internal command: tshark -r <file> -Y "frame.number == N" -V
- Returns complete packet structure without line limits

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: add CC BY-SA 4.0 license headers to documentation files

Add SPDX license headers referencing docs/LICENSE file to all markdown
and HTML documentation files in the docs directory.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(copilot): add topology planner skill

Add new skill for automatic network lab topology planning:
- IOU as default image, 10.0.0.0/8 IP range, max 10 nodes
- Node naming convention: R/S/PC + number
- 7-step workflow for create/rename/link/start/verify/config
- IP allocation rules and troubleshooting guide

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(copilot): add node positioning rules to topology planner skill

Add grid-based positioning for GNS3 nodes:
- Minimum 250px distance between nodes
- Grid layout: 4 columns, 300x250px spacing
- Formula: x = -400 + col * 300, y = -200 + row * 250
- Update workflow params_required to include x, y coordinates

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(copilot): remove unused helper functions from topology planner skill

Delete Python helper functions that cannot be serialized to JSON:
- calculate_node_positions()
- get_position_for_node()
- allocate_subnet()
- allocate_ip()
- TOPOLOGY_CREATION_STEPS (duplicate of workflow in skill dict)

LLM should use formula descriptions in skill to calculate positions/IPs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(copilot): add topology_planner workflow to lab automation prompt

Add device_skills tool to AVAILABLE TOOLS table.
Add TOPOLOGY PLANNING WORKFLOW section explaining:
- How to query topology_planner skill
- Default conventions (IOU, 10.0.0.0/8, naming rules, position formula)
- 7-step workflow for topology creation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(copilot): support name parameter in gns3_create_node tool

- Add optional 'name' field to node creation, allowing direct naming
- Update Node constructor to pass name parameter
- Remove separate rename step (step_3) from topology planner workflow
- Renumber workflow steps: 1-2-3-4-5-6 (skipping old step_3 rename)

This eliminates the need for a separate gns3_update_node_name_tool call
when creating nodes with predefined names.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(copilot): update topology workflow to 6 steps

Remove rename step since name can be set directly in create_gns3_node.
Workflow changed from 7 steps to 6 steps.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(copilot): replace grid positioning with topology-based layout

Redesign node positioning rules to follow classic network topologies:
- Star: hub at center, spokes radiating outward
- Ring: nodes in circular arrangement
- Bus: linear chain of nodes
- Mesh: grid pattern for interconnected nodes
- Hierarchical: three-tier (Core → Distribution → Access)
- Linear P2P: point-to-point WAN links in a line

Remove old grid formula. Update prompt and workflow to reflect new approach.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs(copilot): add note about combining topology types

LLM can mix topology types as needed, e.g., star + linear_p2p for WAN segments.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(copilot): remove language matching rules from prompts

Remove language matching rules that cause inconsistent responses:
- "User writes in Chinese → Respond in Chinese"
- "User writes in English → Respond in English"

These created ambiguity and led to inconsistent language behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(copilot): remove Chinese from response template

Replace mixed Chinese/English headers with English only:
- "操作总结 / Operation Summary" → "Operation Summary"
- "详细信息 / Details" → "Details"

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor(copilot): remove Chinese mentions from prompts

Remove references to "Chinese" language in title_prompt.py:
- "Generates concise Chinese or English titles" → "Generates concise titles"
- "If the content is predominantly in Chinese, generate a Chinese title" → removed

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(copilot): use short_name for link labels and support short port names

- Match ports by both name and short_name for flexibility
- Use short_name (e.g. "e0/0") instead of full name (e.g. "Ethernet0/0")
  for link labels, improving visual clarity in dense topologies

* feat(topology): replace hardcoded interface names with dynamic placeholders

- Update topology planner skill template to use {short_name} placeholder
- Allows dynamic interface name generation based on device type
- Improves template flexibility for different network device configurations

* refactor(topology): use node-pair IP format and hyphenated naming

- Use 10.0.{node_pair}.x format for P2P links (e.g., 10.0.12.x for R-1-R-2)
- Change node naming to hyphenated format: R-1, R-2, SW-1, PC-1
- Use {short_name} as placeholder for port names in output template
- Remove unused IP_SUBNET_POOL constant
- Simplify ip_planning rules to intuitive format description

* fix(web_wireshark): raise minimum Docker API version to 1.44

Docker daemon requires API version 1.44+, so the minimum
supported version should match rather than defaulting to 1.40.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(web_wireshark): call ensure_network in start_wireshark_session

Ensure Docker network exists before creating container, so that
Web Wireshark works when started via API without going through CLI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(agent): improve Docker image setup with network error detection

- Add `is_network_error()` function to detect network-related failures in Docker pull output
- Modify `pull_image()` to capture output and return success status along with output
- Enhance main logic to detect network errors and skip local build when Docker Hub is inaccessible
- Provide helpful suggestions for network issues (Docker mirror, VPN, manual image transfer)
- Improve error messages to differentiate between network failures and other errors

* feat(docs): add Ubuntu 24.04 development setup guide

Add a comprehensive development environment setup guide for Ubuntu 24.04. The guide includes steps to install dependencies via PPA, configure Docker with mirror accelerators for users in mainland China, set up user permissions, and run the server from source with a Python virtual environment. This provides a clear, step-by-step reference for new contributors and developers.

* feat(docs): add pip mirror note for China mainland users

Add a comment in the development setup documentation suggesting the use of the Aliyun PyPI mirror for users in China mainland to improve installation speed and reliability. This helps overcome network restrictions and slow downloads from the default PyPI repository.

* feat(docs): add LVM root partition expansion guide

Add optional section to development setup documentation with instructions for expanding the root partition when using LVM. This helps developers resolve low disk space issues by utilizing unallocated space in the volume group. The guide includes commands to check LVM status, extend the logical volume, and verify the changes.

* feat(docs): add tshark to development setup dependencies

Add tshark to the apt install command in the development setup documentation. Tshark is required for packet capture functionality in GNS3, ensuring the development environment has all necessary tools for network analysis and debugging.

* feat: add GNS3 documentation skill with standardized structure

Add new documentation skill file defining standards for GNS3 technical documentation. The skill establishes a structured approach focusing on architecture diagrams, flow diagrams, and text descriptions while prohibiting code examples and user scenarios. This ensures consistent, high-quality documentation across the GNS3 ecosystem.

* feat(docs): update documentation skill to focus on server technical docs

Update the GNS3 documentation skill to specifically target server technical documentation under the docs/ directory. The revised standard emphasizes high-level understanding over implementation details, using ASCII diagrams for architecture and business processes, API endpoint tables, and measured performance data. Code specifics are intentionally omitted, directing readers to the codebase for implementation details.

* feat(docs): rewrite documentation skill and statistics API doc

- Rewrite gns3-documentation skill to match actual server-side doc style:
  focus on architecture/flow diagrams, API data, skip code details
- Switch diagram standard from ASCII to Mermaid (GitHub native rendering)
- Rewrite statistics-api.md with Mermaid architecture and sequence diagrams,
  add missing webwireshark container fields (memory_limit, cpu_limit, pids_limit),
  remove speculative content

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(docs): update VNC WebSocket console documentation with Mermaid diagrams

- Replace ASCII diagrams with Mermaid flowcharts for better visualization
- Clarify connection flow between browser, controller, compute, and VNC server
- Update authentication details and endpoint descriptions
- Add missing documentation for packet capture workflow
- Improve overall readability and maintainability

* docs: add AI disclaimer to documentation files

Add a standardized disclaimer to multiple documentation files indicating that the content has been organized by AI with reference to actual code. The disclaimer warns users that AI can make mistakes and advises verification against the source code when in doubt. This improves transparency about the documentation's origin and encourages careful usage.

* feat(docs): update chat API documentation with new features and clarifications

- Update title from "Design Document" to reflect current implementation status
- Add new features: session abort, copilot modes (teaching_assistant, lab_automation_assistant), session pinning
- Enhance architecture diagram with LangGraph StateGraph details including abort_handler_node, title_generator_node, and conditional edges
- Clarify statistics tracking: add ai_response_counted flag, filter title_generator_node from LLM counts, explain incremental token counting
- Improve documentation accuracy for real-time statistics collection and token calculation methods

* feat(docs): restructure command security documentation with visual workflows

- Replace verbose implementation details with concise architecture overview
- Add Mermaid diagrams to visualize command filtering and multi-line expansion flows
- Simplify configuration instructions and remove redundant examples
- Consolidate file structure and function reference tables for clarity
- Maintain all security principles while improving readability and maintainability

* feat(context): refactor context window management documentation for clarity

- Reorganize documentation with improved structure and visual diagrams
- Add Mermaid diagrams to illustrate architecture and trimming process
- Simplify content while maintaining technical accuracy
- Update token counting and trimming strategy explanations
- Enhance readability with better formatting and component tables

* feat(docs): update LLM model configs documentation

- Change config column type from JSONB to JSON (JSONB on PostgreSQL)
- Update provider table to mark base_url as required with planned optional status
- Add note about base_url being currently required in API schema
- Clarify GET own configs endpoint returns plain array
- Add copilot_mode field to update request schema
- Document update limitations for context_strategy and copilot_mode fields

* feat(docs): add GNS3StartNodeQuickTool and update node creation examples

- Add documentation for new `GNS3StartNodeQuickTool` (`start_gns3_node_quick`) that starts nodes without waiting for boot completion
- Update node creation example to include optional `name` field in template placement
- Clarify dynamic wait time strategy for `GNS3StartNodeTool` and contrast with quick tool
- Update tool file structure to reflect new configuration, display, and packet capture tools
- Fix truncated line in suspend tool documentation

* feat(docs): add comprehensive documentation structure with CC BY-SA 4.0 license

Add initial documentation directory with detailed README.md that outlines:
- Dual license structure (CC BY-SA 4.0 for docs, GPLv3 for code)
- Complete directory structure for features, AI Copilot, and bugs
- Feature documentation covering Controller+Compute setup, Statistics API, VNC WebSocket console, and Web Wireshark
- AI Copilot implemented features including Chat API, LLM model configs, command security, and multi-vendor device support

This provides organized technical documentation for the GNS3 server project with proper licensing and feature coverage.

* docs: update multi-vendor device support documentation

- Simplify vendor support table by removing redundant platform column
- Add VPCS as a simulator in status column
- Improve VPCS driver diagram with Mermaid syntax and detailed components
- Document VPCS-specific Netmiko parameters (fast_cli, global_delay_factor)
- Update VPCS tool usage example with connection options structure
- Remove outdated Nornir configuration approaches, reference current implementation
- Add file reference for VPCS tools location
- Clean up documentation structure for better readability

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: YueGuobin <yueguobin@gmail.com>
Co-authored-by: Jeremy Grossmann <grossmj@gns3.net>
This commit is contained in:
Guobin YueClaude Sonnet 4.6YueGuobinJeremy Grossmann
 authored and GitHub committed 2026-04-20 22:56:55 +08:00
1 parent 80cc85e5d5
commit f07d21c511
77 files changed
+7544 -1767
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/gns3-copilot/README.md
-167
View File
@@ -1,167 +0,0 @@
# GNS3 AI Copilot Documentation
This directory contains design documentation, implementation guides, and future plans for the GNS3 AI Copilot feature.
## Directory Structure
```
docs/gns3-copilot/
├── README.md # This file
├── template-based-configuration-roadmap.md # Future: Template-based config with HITL
└── implemented/ # Implemented features and designs
├── chat-api.md # Chat API design (SSE, session management)
├── llm-model-configs.md # LLM model configuration system
├── command-security.md # Command security and filtering
├── context-window-management.md # Context window optimization
├── node-control-tools.md # Node start/stop/suspend tools for lab automation
└── multi-vendor-device-support.md # Multi-vendor device support (Cisco, Huawei)
```
## Implemented Features
### Chat API (`implemented/chat-api.md`)
The core Chat API that enables AI-powered conversations within GNS3 projects.
**Key Features:**
- Server-Sent Events (SSE) for streaming responses
- Project-level session isolation
- Session management (CRUD operations)
- Statistics tracking (messages, tokens, LLM calls)
- User-level LLM configuration
**Status:** ✅ Implemented
### LLM Model Configs (`implemented/llm-model-configs.md`)
Multi-level LLM model configuration system.
**Key Features:**
- System-wide defaults
- Group-level configurations
- User-level overrides
- Runtime parameter adjustment
- Model provider abstraction
**Status:** ✅ Implemented
### Command Security (`implemented/command-security.md`)
Security framework for AI-generated commands.
**Key Features:**
- Command filtering and validation
- Dangerous operation detection
- HITL (Human-in-the-Loop) confirmations
- Audit logging
**Status:** ✅ Implemented
### Context Window Management (`implemented/context-window-management.md`)
Optimization strategies for handling large project contexts.
**Key Features:**
- Intelligent content filtering
- Token usage optimization
- Summary generation
- Context compression
**Status:** ✅ Implemented
### Node Control Tools (`implemented/node-control-tools.md`)
Tools for controlling network device lifecycle in GNS3 projects.
**Key Features:**
- Start nodes with progress tracking
- Quick start for automated workflows
- Stop nodes for lab shutdown
- Batch operations support
- Mode-based access control
**Status:** ✅ Implemented
### Multi-Vendor Device Support (`implemented/multi-vendor-device-support.md`)
Multi-vendor network device support with custom Netmiko drivers for Huawei, Ruijie, and VPCS devices.
**Key Features:**
- Custom HuaweiTelnetCE driver for Huawei CloudEngine (no authentication)
- Custom RuijieTelnetEnhanced driver for Ruijie OS (interactive command handling)
- Custom VPCSTelnet driver for VPCS simulator (no authentication, ANSI code stripping)
- Cisco IOS Telnet support
- Dynamic device type detection from GNS3 tags
- Unified Nornir + Netmiko architecture
- Vendor-specific command handling (VRP system-view, Ruijie interactive prompts, VPCS simple prompts)
**Tested Vendors:**
- Cisco IOS (Telnet)
- Huawei CloudEngine (Telnet, custom driver)
- Ruijie (锐捷) OS (Telnet, custom enhanced driver)
- VPCS (Virtual PC Simulator, Telnet, custom driver)
**Status:** ✅ Implemented
## Future Enhancements
### Template-Based Configuration with HITL (`template-based-configuration-roadmap.md`)
**Status:** 💡 Proposed | **Target:** Next Release
A revolutionary approach to network device configuration using Jinja2 templates with Human-in-the-Loop confirmations.
**Key Features:**
- Three-step HITL workflow (Template → Parameters → Execute)
- 70-80% token savings for multi-device configurations
- Template reusability across projects
- Human review at every critical step
- Configuration preview before execution
**Benefits:**
- Massive token cost reduction
- Enhanced safety through human oversight
- Template library for common configurations
- Multi-vendor support (Cisco, Huawei, H3C, etc.)
**Implementation Timeline:**
- Phase 1: Core MVP (3-5 days) - Basic template workflow
- Phase 2: UX Enhancement (2-3 days) - Review interfaces, preview
- Phase 3: Template Library (2-3 days) - Pre-built templates, caching
- Phase 4: Advanced Features (3-4 days) - Multi-vendor, composition, analytics
See [`template-based-configuration-roadmap.md`](./template-based-configuration-roadmap.md) for complete details.
### Other Proposed Features
- Vision-based Topology Creation: Create network topologies from images/diagrams
- Enhanced HITL Workflows: Advanced confirmation patterns for other operations
- Web UI Enhancements: Improved management interfaces
- Configuration Diff & Comparison: Compare configurations across devices
- Rollback & Undo: Revert configuration changes
## Contributing
When adding new documentation:
1. **Implementation:** Add documentation to `implemented/` when feature is complete
2. **Naming:** Use concise names like `{feature}.md`
## Document Status Legend
| Status | Description |
|--------|-------------|
| ✅ Implemented | Feature is fully implemented and deployed |
| 📋 Design Complete | Design is done, awaiting implementation |
| 🚧 In Progress | Currently being implemented |
| 💡 Proposed | Initial idea or proposal |
## Related Documentation
- [GNS3 Server API Documentation](https://api.gns3.com/)
- [GNS3 Web UI Documentation](https://docs.gns3.com/)
- [LangGraph Documentation](https://langchain-ai.github.io/langgraph/)
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
## Quick Links
- **Current Feature Branch:** `feature/ai-copilot-bridge`
- **Main Branch:** `master`
- **Issue Tracker:** [GitHub Issues](https://github.com/GNS3/gns3-server/issues)
---
_Last updated: 2026-03-20_
docs/gns3-copilot/implemented/chat-api.md
+134 -39
View File
@@ -1,15 +1,25 @@
# GNS3 Copilot Agent Chat API Design Document
<!--
SPDX-License-Identifier: CC-BY-SA-4.0
See LICENSE file for licensing information.
-->
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
# GNS3 Copilot Agent Chat API
## Overview
This document describes the architectural design and implementation plan for the GNS3 Copilot Chat API. This API enables clients to interact with the GNS3 Copilot Agent through a RESTful interface, providing streaming conversations, session management, and project topology queries.
This document describes the implementation of the GNS3 Copilot Chat API. This API enables clients to interact with the GNS3 Copilot Agent through a RESTful interface, providing streaming conversations, session management, session abort, and project topology queries.
## Core Features
- **Project-level Isolation**: Each GNS3 project has its own Agent instance and session storage
- **Streaming Responses**: Uses Server-Sent Events (SSE) for real-time streaming output
- **Session Management**: Supports session listing, renaming, deletion, and history queries
- **Session Management**: Supports session listing, renaming, deletion, pinning, and history queries
- **Session Abort**: Supports aborting an ongoing streaming session mid-conversation
- **Statistics Tracking**: Automatically records message counts, LLM call counts, and token usage
- **Copilot Modes**: Supports `teaching_assistant` (diagnostic only) and `lab_automation_assistant` (full config) modes
- **User Isolation**: Each user has independent LLM configurations and session spaces
## Architecture Design
@@ -31,10 +41,19 @@ AgentService (per project)
│ ├─ checkpoints table (LangGraph state)
│ └─ chat_sessions table (session metadata)
│
└─ LangGraph Agent
└─ LangGraph Agent (StateGraph)
├─ llm_call node
├─ should_continue node
└─ tool_node (GNS3 tools)
├─ tool_node (GNS3 tools)
├─ title_generator_node (auto title)
└─ abort_handler_node (abort handling)
│
├─ Conditional Edges (routing functions):
│ ├─ should_continue (after llm_call)
│ └─ recursion_limit_continue (after tool_node)
│
└─ Copilot Modes:
├─ teaching_assistant (diagnostic tools only)
└─ lab_automation_assistant (full diagnostic + config tools)
```
### Project-level Checkpoint Design
@@ -184,38 +203,34 @@ Statistics are collected in real-time during conversation, and updated to `chat_
1. **message_count (number of messages)**
- Initial value: 1 (user message)
- `on_chat_model_end` event: +1 (AI complete reply, not each chunk)
- `on_chat_model_end` event: +1 (AI complete reply, only counted once per turn via `ai_response_counted` flag, not each streaming chunk)
- `on_tool_end` event: +1 (each tool execution result)
2. **llm_calls_count (number of LLM calls)**
- Listen to `on_chat_model_start` event
- +1 each time LLM starts generation
- **Filtered**: `title_generator_node` events are excluded from count (internal use only)
3. **input_tokens (input tokens)**
- Extracted from `usage_metadata` in `on_chat_model_end` event
- **Important**: input_tokens returned by LangGraph already includes conversation history, accumulates previous conversation content on each LLM call
- Example: 1st call input=8674, 2nd call input=9421 (includes 1st conversation 8674+675+system prompt increment)
- Uses incremental addition (`+=`): each event's token count is added to the running total
- **Filtered**: `title_generator_node` events are excluded from token counting
- Tries multiple extraction methods: `response.usage_metadata` → `output.usage_metadata` → direct data fields
4. **output_tokens (output tokens)**
- Extracted from `usage_metadata` in `on_chat_model_end` event
- **Important**: output_tokens returned by LangGraph is also accumulated value, includes output from all LLM calls
- Example: 1st actual output=675, 2nd actual output=9, accumulated output=684 (675+9)
- Uses incremental addition (`+=`): each event's token count is added to the running total
- **Filtered**: `title_generator_node` events are excluded from token counting
5. **total_tokens (total tokens)**
- Calculation formula: input_tokens + output_tokens
- Take the accumulated value from the last LLM call for calculation
**Statistics Example** (real data):
- 1st LLM call (AI reply): input=8674, output=675
- 2nd LLM call (generate title): input=9421, output=684 (accumulated value: 675+9)
- Final storage: input_tokens=9421, output_tokens=684, total_tokens=10105
- Note: LangGraph automatically accumulates, code can directly take the last value
- Calculation formula: `input_tokens + output_tokens` (computed at update time)
**Notes**:
- message_count counts **complete messages**, not streaming chunks
- `ai_response_counted` flag ensures AI responses are only counted once per turn, even if multiple `on_chat_model_end` events fire
- `title_generator_node` is completely excluded from all statistics (llm_calls, tokens, streaming output)
- Token data depends on LLM's returned `usage_metadata`, some models may not support
- Statistics are incrementally updated to database via `update_session` method after stream ends
- LangGraph automatically handles input and output history accumulation, code uses the last LLM call value
- Statistics are incrementally updated to database via `update_session` method after stream ends (using SQL `field = field + ?` syntax)
- **Message ID handling**: Assign ID when creating initial message (`HumanMessage(id=str(uuid4()))`), messages read from checkpoint without ID are also automatically generated
- **Format conversion**: Use `message_converters.py` module to handle conversion between LangChain and OpenAI formats, ensuring tool_calls format conforms to OpenAI specification
@@ -241,12 +256,14 @@ Chat API uses Server-Sent Events (SSE) for streaming transmission.
| type | Description | Included Fields |
|------|-------------|------------------|
| content | AI text content (streaming) | content, message_id (optional) |
| content | AI text content (streaming) | content, message_id (optional), session_id |
| tool_call | LLM decides to call tool (streaming, parameters accumulated gradually) | tool_call (object, includes id, type, function), session_id, message_id (optional) |
| tool_start | Tool starts execution | tool_name, tool_call_id, session_id |
| tool_end | Tool execution complete | tool_name, tool_output, session_id |
| error | Error message | error, session_id |
| abort | Stream aborted by user | session_id |
| done | Stream end | session_id |
| heartbeat | *(Planned)* Heartbeat keepalive | session_id |
**Tool Output Format** (`tool_output` field):
- If the tool returns a non-string type (dict, list), it is automatically serialized to JSON format using `json.dumps(obj, ensure_ascii=False, indent=2)`
@@ -254,8 +271,6 @@ Chat API uses Server-Sent Events (SSE) for streaming transmission.
- This ensures all structured data is in standard JSON format, making it easy for the frontend to parse with `JSON.parse()`
- Chinese and other non-ASCII characters are preserved (not escaped to `\uXXXX`)
| heartbeat | Heartbeat keepalive | session_id |
### Message Examples
```json
@@ -333,6 +348,9 @@ Chat API uses Server-Sent Events (SSE) for streaming transmission.
// Error
{"type": "error", "error": "Project not found", "session_id": "xxx"}
// Stream aborted by user
{"type": "abort", "session_id": "xxx"}
```
### Streaming Tool Call Mechanism
@@ -384,14 +402,16 @@ function handleToolCallEvent(chunk) {
}
```
### Heartbeat Mechanism
### Heartbeat Mechanism *(Planned)*
**Purpose**: Prevent proxy server/load balancer from disconnecting SSE connection due to timeout.
**Implementation**: Use `asyncio.wait` to set timeout, send `heartbeat` message after timeout, then continue waiting for next event.
**Planned Implementation**: Use `asyncio.wait` to set timeout, send `heartbeat` message after timeout, then continue waiting for next event.
**Frontend Handling**: When receiving `heartbeat` message, ignore it directly, don't render anything.
**Note**: The `heartbeat` type is defined in the `ChatResponse` schema but not yet implemented in the streaming path. Currently, long-running tool executions may cause proxy timeouts.
## API Endpoints
All endpoints are under `/v3/projects/{project_id}/chat/` path.
@@ -403,6 +423,7 @@ All endpoints are under `/v3/projects/{project_id}/chat/` path.
| GET | `/sessions/{session_id}/history` | Get session history |
| PATCH | `/sessions/{session_id}` | Rename session |
| DELETE | `/sessions/{session_id}` | Delete session |
| POST | `/sessions/{session_id}/abort` | Abort ongoing streaming session |
| PUT | `/sessions/{session_id}/pin` | Pin session |
| DELETE | `/sessions/{session_id}/pin` | Unpin session |
@@ -514,7 +535,7 @@ data: {"type": "done", "session_id": "d7e76375-6960-419a-9367-211ef64af877"}
{
"id": "f0247568-071d-412f-9e3e-4cbe815834ea",
"role": "user",
"content": "你能干点啥。",
"content": "What can you do?",
"metadata": {
"created_at": "2026-03-07T17:48:07.848519"
}
@@ -522,7 +543,7 @@ data: {"type": "done", "session_id": "d7e76375-6960-419a-9367-211ef64af877"}
{
"id": "lc_run--019cc969-eb81-7dd1-a894-e819daf81cd0",
"role": "assistant",
"content": "我可以作为GNS3网络实验的助教...",
"content": "I can serve as a teaching assistant for GNS3 network labs...",
"tool_calls": [
{
"id": "call_00_xxx",
@@ -566,6 +587,34 @@ data: {"type": "done", "session_id": "d7e76375-6960-419a-9367-211ef64af877"}
**Response**: 204 No Content
### POST /v3/projects/{project_id}/chat/sessions/{session_id}/abort
**Function**: Abort an ongoing streaming session
**Behavior**:
1. Sets an in-memory abort flag (`_abort_flags[session_id] = True`)
2. The LangGraph graph checks this flag at conditional edges (`should_continue`, `recursion_limit_continue`)
3. If abort is detected during a tool call, `abort_handler_node` generates placeholder `ToolMessage` results to maintain message history consistency
4. The stream ends gracefully, and any aborted tool messages are yielded as `tool_end` events with `{"status": "aborted"}` content
**Response Example**:
```json
{"status": "ok", "session_id": "d7e76375-6960-419a-9367-211ef64af877"}
```
**Abort Flow**:
```
POST /abort → set_abort_flag(session_id)
↓
Graph conditional edge checks flag
↓
┌─ Has pending tool_calls?
│ YES → abort_handler_node (generates placeholder ToolMessages)
│ NO → END directly
↓
Stream ends → yields aborted tool_end events → done
```
### PUT /v3/projects/{project_id}/chat/sessions/{session_id}/pin
**Function**: Pin session to top of list
@@ -684,9 +733,9 @@ OpenAI-compatible message model.
**Responsibility**: Convert between LangChain message format and OpenAI-compatible format
**Main Functions**:
- `convert_langchain_to_openai()`: LangChain → OpenAI format
- `convert_openai_to_langchain()`: OpenAI → LangChain format
- `convert_stream_event_to_openai()`: Stream event → OpenAI SSE format
- `convert_langchain_to_openai()`: LangChain → OpenAI format (used in `get_history` and message conversion)
- `convert_openai_to_langchain()`: OpenAI → LangChain format (utility, not used in main streaming path)
- `convert_stream_event_to_openai()`: *(Not used in main streaming path)* — streaming uses `ToolCallStreamAccumulator` for `on_chat_model_stream` and `AgentService._convert_event_to_chunk` for other events
**Key Conversion Logic**:
@@ -718,12 +767,13 @@ OpenAI-compatible message model.
**Responsibility**: LangGraph-based workflow orchestration for AI conversation
**Main Components**:
**Graph Nodes**:
1. **llm_call Node**: Invokes LLM with tools and conversation history
- Injects topology information into system prompt
- Handles message trimming for context window management
- Routes to tool execution or generates response
- Selects tools based on copilot mode (`teaching_assistant` vs `lab_automation_assistant`)
- Creates fresh model instance with tools for each call
2. **tool_node Function**: Executes tool calls and returns results
- **Critical**: Serializes tool output to JSON before creating ToolMessage
@@ -742,7 +792,47 @@ OpenAI-compatible message model.
)
```
3. **generate_title Node**: Auto-generates conversation title on first interaction
3. **title_generator_node** (`generate_title` function): Auto-generates conversation title on first interaction
- Uses a separate lightweight LLM (title_model) to generate a title from the first user message and assistant response
- Title is truncated to 40 characters max
- Fallback: uses first 30 chars of user's message if title generation fails
- This node is **filtered out** from statistics and SSE streaming (internal use only)
4. **abort_handler_node**: Handles abort when pending tool_calls exist
- Generates placeholder `ToolMessage` with `{"status": "aborted"}` content
- Ensures message history consistency and prevents checkpoint corruption
- Only triggered when abort flag is set and the last AI message has tool_calls
**Conditional Edges (Routing Functions)**:
- **should_continue** (after `llm_call`): Routes to `tool_node`, `title_generator_node`, `abort_handler_node`, or `END` based on:
1. Check abort flag → `abort_handler_node` (if pending tool_calls) or `END`
2. Has tool_calls → `tool_node`
3. First interaction without title → `title_generator_node`
4. Otherwise → `END`
- **recursion_limit_continue** (after `tool_node`): Routes to `llm_call` or `END` based on:
1. Check abort flag → `END`
2. Remaining steps < 4 → `END` (prevent infinite loops)
3. Otherwise → `llm_call`
**State** (`MessagesState`):
- `messages`: Conversation messages (cumulative with `operator.add`)
- `llm_calls`: LLM invocation counter
- `remaining_steps`: Recursion depth tracker (initial: 20)
- `conversation_title`: Auto-generated title
- `topology_info`: GNS3 project topology data
- `session_id`: Session identifier (for abort tracking)
- `abort`: Abort flag
**Copilot Modes**:
The agent supports two tool sets, selected by `copilot_mode` in the user's LLM configuration:
| Mode | Tools | Description |
|------|-------|-------------|
| `teaching_assistant` (default) | GNS3Template, GNS3CreateNode, GNS3Link, GNS3StartNode, GNS3UpdateNodeName, ExecuteMultipleDeviceCommands, PacketCapture, DeviceSkills | Read-only diagnostic + node creation |
| `lab_automation_assistant` | All teaching_assistant tools + GNS3StopNode, GNS3SuspendNode, ExecuteMultipleDeviceConfigCommands, VPCSCommands | Full diagnostic + configuration tools |
**Why Serialize in tool_node?**
@@ -780,6 +870,8 @@ ToolMessage(content=JSON_string)
- `list_sessions`: List sessions
- `delete_session`: Delete session
- `rename_session`: Rename session
- `pin_session`: Pin or unpin session
- `abort_session`: Signal abort for a running session (sets in-memory flag)
- `close`: Close database connection
**Core Flow** (stream_chat):
@@ -799,9 +891,11 @@ ToolMessage(content=JSON_string)
- Statistics logic doesn't depend on converted SSE chunk, gets directly from original events
**Key Event Handling**:
- `on_chat_model_start`: LLM call count +1
- `on_chat_model_end`: Extract token usage (from `output.usage_metadata`), AI message count +1
- `on_chat_model_start`: LLM call count +1 (excludes `title_generator_node`)
- `on_chat_model_end`: Extract token usage via `response.usage_metadata` → `output.usage_metadata` → data fields (excludes `title_generator_node`), AI message count +1 (once per turn via `ai_response_counted` flag)
- `on_tool_end`: Tool message count +1
- `on_chat_model_stream`: Processed by `ToolCallStreamAccumulator` for progressive tool call arguments (excludes `title_generator_node` from SSE output)
- Abort flag is cleared at stream start, checked during streaming for graceful termination
**Implementation Location**: `agent_service.py`
@@ -868,8 +962,9 @@ Handle different types based on SSE message's `type` field:
| tool_start | Optional: show tool start execution status |
| tool_end | Create tool_result type message, display tool execution result |
| error | Display error message |
| abort | Mark stream as aborted, stop loading state |
| done | Mark stream end, stop loading state |
| heartbeat | Ignore (keepalive signal) |
| heartbeat | *(Planned)* Ignore (keepalive signal) |
### Session ID Management (Important)
@@ -977,15 +1072,15 @@ const displayTime = timestamp ? new Date(timestamp).toLocaleString() : 'Unknown'
- Lower database lock contention
- Improve real-time performance of streaming response
**Implementation Location**: `agent_service.py` lines 283-294
**Implementation Location**: `agent_service.py` `stream_chat` method (statistics collection in event loop + batch update after stream)
## Dependencies
- `langchain` >= 0.3.0
- `langgraph` >= 0.2.0
- `langchain-core`
- `langgraph-checkpoint-sqlite` >= 3.0.1
- `aiosqlite`
- `fastapi`
## Extensibility
docs/gns3-copilot/implemented/command-security.md
+142 -419
View File
@@ -1,3 +1,11 @@
<!--
SPDX-License-Identifier: CC-BY-SA-4.0
See LICENSE file for licensing information.
-->
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
# Command Security Configuration
## Overview
@@ -5,483 +13,208 @@
GNS3-Copilot includes multiple security layers to prevent execution of commands that may cause issues in the lab environment:
- **Command Filtering**: Prevents commands that may timeout or lock up the console
- **Configuration Safety**: Prohibits dangerous configuration changes (AAA, passwords, etc.)
- **Configuration Safety**: Restricts dangerous configuration changes (AAA, passwords, etc.)
- **Multi-line Command Handling**: Properly processes commands with embedded newlines (banner, etc.)
This helps:
## Architecture
- **Prevent tool timeouts**: Commands like `traceroute` may run longer than the tool timeout
- **Maintain console availability**: Long-running commands can leave the device console unavailable for subsequent commands
- **Prevent device lockout**: AAA/password changes can lock users out of devices
- **Ensure reliable execution**: Filtering problematic commands ensures the remaining commands can execute properly
```mermaid
graph TD
A[Tool Request] --> B{Command Filter}
B -->|allowed| C[Nornir/Netmiko Execution]
B -->|blocked| D[blocked_commands_info]
C --> E[Process Results]
D --> E
E --> F[Return Result]
## Implementation Status
subgraph Command Filter
B1[Load forbidden_commands.txt]
B2[Prefix match per command]
B1 --> B2
end
**Status**: ✅ **Implemented and Verified**
The command filtering system is fully implemented and has been tested in a live GNS3 environment with actual network devices. Key features:
- ✅ Simple text-based configuration file
- ✅ Substring matching (case-insensitive)
- ✅ Non-blocking filtering (allowed commands execute normally)
- ✅ Detailed blocking feedback in tool results
- ✅ Multi-device support
- ✅ Verified with real Cisco IOS devices
See the [Implementation Verification](#implementation-verification) section for actual test results.
## Problem Context
### Why Filter Commands?
When GNS3-Copilot tools execute commands on network devices using Nornir/Netmiko, there is a timeout limit (typically 30-60 seconds). If a command exceeds this timeout:
1. The tool stops waiting and returns a timeout error
2. The device console may still be executing the command
3. Subsequent commands sent to the device fail or produce incorrect results
4. The user may need to manually interrupt the command on the device console
### Example Scenario
```
Time Agent Action Device Console Status
t0 Execute: traceroute 8.8.8.8 [Command starts]
t1 ...waiting... [Tracing...]
t2 ...waiting... [Tracing...]
t30 Timeout! Proceed to next tool [Still tracing!]
t31 Execute: show ip route [Ignored or corrupted]
t32 ❌ Command fails [Console still busy]
subgraph Security Layers
S1[Layer 1: Command Filter<br/>prefix-based blocking]
S2[Layer 2: AI Prompt Rules<br/>AAA/password guidance only]
S3[Layer 3: Multi-line Expansion<br/>banner/ACL handling]
S1 --> S2 --> S3
end
```
## Current Implementation
## Business Process
### 1. Command Filtering System
### Command Filtering Flow
#### Forbidden Commands List
Commands are listed in a simple text file at:
```mermaid
flowchart TD
Start([Tool receives commands]) --> Load[Load forbidden patterns<br/>from config file]
Load --> Loop{For each command}
Loop -->|next command| Prefix{Command starts with<br/>forbidden pattern?}
Prefix -->|yes| Block[Add to blocked_commands_info]
Prefix -->|no| Allow[Add to allowed list]
Block --> Loop
Allow --> Loop
Loop -->|all checked| Empty{Allowed list empty?}
Empty -->|yes| Return1[Return partial_success<br/>with blocked info only]
Empty -->|no| Exec[Execute allowed commands<br/>via Nornir/Netmiko]
Exec --> HasBlocked{Has blocked commands?}
HasBlocked -->|yes| Return2[Return partial_success<br/>with output + blocked info]
HasBlocked -->|no| Return3[Return success<br/>with output]
```
gns3server/agent/gns3_copilot/config/forbidden_commands.txt
### Multi-line Command Expansion Flow
```mermaid
flowchart TD
A[config_tools receives commands] --> B{Command contains \n?}
B -->|yes| C[Split by newline]
C --> D[Filter empty lines]
D --> E[Add expanded lines to command list]
B -->|no| F[Keep command as-is]
E --> G[Execute via netmiko_send_config]
F --> G
```
## Tools Using Command Security
| Tool | File | Filter | Multi-line Expansion | Result Fields |
|------|------|--------|---------------------|---------------|
| `ExecuteMultipleDeviceCommands` | `display_tools_nornir.py` | Yes | No | `diagnostic_commands` |
| `ExecuteMultipleDeviceConfigCommands` | `config_tools_nornir.py` | Yes | Yes | `config_commands` |
## Command Filtering System
### Forbidden Commands Configuration
**File:** `gns3server/agent/gns3_copilot/config/forbidden_commands.txt`
**Format:**
- One command pattern per line
- Simple substring matching (case-insensitive)
- **Prefix matching** (case-insensitive) — matches the beginning of each command
- Empty lines and lines starting with `#` are ignored
- Match is performed on the beginning of each command
**Example:**
```
# Network diagnostic commands that may timeout
traceroute
tracepath
tracert
**Default patterns (used when config file is missing):**
# Debug commands that may destabilize devices
debug
# Test commands that may affect device stability
test
```
### Filter Behavior
1. **Input Commands**: `["show version", "traceroute 8.8.8.8", "show ip int brief"]`
2. **Filtering**: `traceroute 8.8.8.8` is removed (matches `traceroute`)
3. **Executed**: `["show version", "show ip int brief"]`
4. **Result**: Returns successful output with blocked command information
| Pattern | Reason |
|---------|--------|
| `traceroute` | Can run 30+ seconds, exceeds tool timeout |
| `tracepath` | Similar to traceroute, long execution time |
| `tracert` | Windows traceroute, same timeout issues |
| `ping -f` | Flood ping can overwhelm lab devices |
| `debug` | Can produce overwhelming output and destabilize devices |
| `test` | May affect device stability |
### Result Format
When commands are filtered, the result includes additional fields:
**Status values:**
| Status | Condition |
|--------|-----------|
| `"success"` | All commands executed, none blocked |
| `"partial_success"` | Some commands blocked (including all blocked + execution succeeded) |
| `"failed"` | Execution failed (device not found, connection error, etc.) |
**Example — partial success (display tool):**
```json
{
"device_name": "R-1",
"status": "partial_success",
"output": "R-1#show version\nCisco IOS Software...\nR-1#show ip int brief\nInterface...",
"output": "...",
"diagnostic_commands": ["show version", "show ip int brief"],
"blocked_commands": ["traceroute 8.8.8.8"],
"blocked_info": {
"traceroute 8.8.8.8": "Command 'traceroute 8.8.8.8' is not allowed because it matches the forbidden pattern 'traceroute'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands."
"traceroute 8.8.8.8": "Command 'traceroute 8.8.8.8' is not allowed because it matches the forbidden pattern 'traceroute'. ..."
}
}
```
**Status values:**
- `"success"`: All commands executed successfully
- `"partial_success"`: Some commands were blocked, but remaining commands executed successfully
- `"failed"`: Command execution failed (device not found, connection error, etc.)
> **Note:** `diagnostic_commands` is specific to display tools. Config tools use `config_commands` instead.
## Module Structure
### Command Filter Module
**Command Filter:** `gns3server/agent/gns3_copilot/utils/command_filter.py`
**File:** `gns3server/agent/gns3_copilot/utils/command_filter.py`
| Function | Purpose |
|----------|---------|
| `filter_forbidden_commands(commands)` | Returns `(allowed_commands, blocked_commands_info)` |
| `is_command_forbidden(command)` | Check if a single command matches a forbidden pattern |
| `get_forbidden_commands()` | Get current forbidden patterns list |
| `reload_forbidden_commands()` | Clear cache, reload from file on next access |
**Functions:**
- `filter_forbidden_commands(commands: list[str]) -> tuple[list[str], dict[str, str]]`
- Returns allowed commands and blocked command information
- `is_command_forbidden(command: str) -> bool`
- Check if a single command is forbidden
- `get_forbidden_commands() -> list[str]`
- Get the current list of forbidden patterns
- `reload_forbidden_commands() -> None`
- Reload the forbidden commands list (useful after editing the file)
**Integration points:**
- `display_tools_nornir.py` — `_filter_forbidden_commands_from_device_configs()`
- `config_tools_nornir.py` — `_filter_forbidden_commands_from_device_configs()` + `_expand_multiline_commands()`
### Integration Points
## Configuration Safety (AI Prompt Level)
The filter is integrated into:
- **Display Tools** (`display_tools_nornir.py`): `ExecuteMultipleDeviceCommands`
- **Configuration Tools** (`config_tools_nornir.py`): `ExecuteMultipleDeviceConfigCommands`
In addition to the command filter, the AI agent is instructed via system prompt (`lab_automation_assistant_prompt.py`) to handle sensitive configuration commands with caution:
Both tools use the same filtering logic and return format.
**Prompt-enforced rules:**
- **FORBIDDEN (guidance only):** `enable secret`, `username`, `aaa new-model`, `service password-encryption`, `line vty` — AI provides configuration guidance instead of executing
- **Caution required:** `reload`, `erase`, `format` — AI warns user before executing destructive operations
> These are prompt-level rules, not code-level enforcement. Users can always execute commands directly via device console or SSH/Telnet.
## Multi-line Command Handling
Commands containing `\n` are automatically split before execution by `config_tools_nornir.py:_expand_multiline_commands()`.
**Example:** `["banner motd #\nWelcome\n#"]` becomes `["banner motd #", "Welcome", "#"]`
Applies to any command with embedded newlines: `banner`, multi-line ACLs, route-maps, etc.
## Configuration
### Default Forbidden Commands
### Customizing Forbidden Commands
If the configuration file is not found, these defaults are used:
- `traceroute`
- `tracepath`
- `tracert`
- `ping -f`
- `debug`
- `test`
### Customizing the List
To add or remove forbidden commands:
1. Edit the configuration file:
```bash
nano gns3server/agent/gns3_copilot/config/forbidden_commands.txt
```
2. Add your command patterns (one per line):
```
# My custom blocked commands
my_dangerous_command
another_pattern
```
3. Restart GNS3 server to apply changes
### Reloading Without Restart
To reload the forbidden commands list without restarting the server:
```python
from gns3server.agent.gns3_copilot.utils.command_filter import reload_forbidden_commands
reload_forbidden_commands()
```
## Usage Examples
### Example 1: All Commands Allowed
**Input:**
```json
{
"project_id": "abc-123-def",
"device_configs": [
{
"device_name": "R-1",
"commands": ["show version", "show ip route"]
}
]
}
```
**Output:**
```json
{
"device_name": "R-1",
"status": "success",
"output": "...",
"diagnostic_commands": ["show version", "show ip route"]
}
```
### Example 2: Some Commands Blocked
**Input:**
```json
{
"project_id": "abc-123-def",
"device_configs": [
{
"device_name": "R-1",
"commands": ["show version", "traceroute 8.8.8.8", "show ip route"]
}
]
}
```
**Output:**
```json
{
"device_name": "R-1",
"status": "partial_success",
"output": "...",
"diagnostic_commands": ["show version", "show ip route"],
"blocked_commands": ["traceroute 8.8.8.8"],
"blocked_info": {
"traceroute 8.8.8.8": "Command 'traceroute 8.8.8.8' is not allowed because it matches the forbidden pattern 'traceroute'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands."
}
}
```
### Example 3: All Commands Blocked
**Input:**
```json
{
"project_id": "abc-123-def",
"device_configs": [
{
"device_name": "R-1",
"commands": ["traceroute 8.8.8.8", "debug ip routing"]
}
]
}
```
**Output:**
```json
{
"device_name": "R-1",
"status": "success",
"output": "",
"diagnostic_commands": [],
"blocked_commands": ["traceroute 8.8.8.8", "debug ip routing"],
"blocked_info": {
"traceroute 8.8.8.8": "Command 'traceroute 8.8.8.8' is not allowed because it matches the forbidden pattern 'traceroute'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands.",
"debug ip routing": "Command 'debug ip routing' is not allowed because it matches the forbidden pattern 'debug'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands."
}
}
```
Edit `gns3server/agent/gns3_copilot/config/forbidden_commands.txt` and either restart the server or call `reload_forbidden_commands()` to apply changes without restart.
## Implementation Verification
### Real-World Test Results
### Test Results (Live GNS3 Environment)
The command filtering system has been tested in a live GNS3 environment with actual network devices. Below are actual execution results:
**Scenario:** IOU-L2-1 (Cisco IOS L3 switch), mixed allowed/forbidden commands.
**Test Scenario:**
- Devices: IOU-L2-1, IOU-L2-2 (Cisco IOS Layer 3 switches)
- Commands: Mixed allowed and forbidden commands
- Forbidden pattern: `traceroute`
**Actual Output:**
```json
{
"device_name": "IOU-L2-1",
"status": "partial_success",
"diagnostic_commands": [
"show ip route",
"show ip interface brief",
"ping 10.0.0.1",
"ping 10.0.0.2",
"ping 10.0.0.4"
"show ip route", "show ip interface brief",
"ping 10.0.0.1", "ping 10.0.0.2", "ping 10.0.0.4"
],
"blocked_commands": ["traceroute 10.0.0.2"],
"blocked_info": {
"traceroute 10.0.0.2": "Command 'traceroute 10.0.0.2' is not allowed because it matches the forbidden pattern 'traceroute'. This command may run longer than the tool timeout or leave the device console unavailable for subsequent commands."
"traceroute 10.0.0.2": "Command 'traceroute 10.0.0.2' is not allowed because it matches the forbidden pattern 'traceroute'. ..."
}
}
```
**Key Observations:**
1. ✅ `traceroute` command was successfully filtered
2. ✅ All other commands (`show`, `ping`) executed normally
3. ✅ Status correctly set to `partial_success`
4. ✅ Both `diagnostic_commands` (executed) and `blocked_commands` (filtered) are clearly listed
5. ✅ Detailed blocking reason provided in `blocked_info`
6. ✅ Tool execution continued without timeout or console lockup issues
### Functionality Verification Matrix
| Feature | Status | Notes |
|---------|--------|-------|
| Command filtering (substring match) | ✅ Verified | `traceroute` correctly matched and blocked |
| Partial execution | ✅ Verified | Other commands executed successfully |
| Return format consistency | ✅ Verified | Contains all expected fields |
| Multi-device support | ✅ Verified | Each device filtered independently |
| Error messages | ✅ Verified | Clear, informative blocking reasons |
| Status field accuracy | ✅ Verified | `partial_success` set correctly |
| Non-blocking behavior | ✅ Verified | No tool timeouts or console issues |
| Prefix match filtering | Verified | `traceroute` correctly matched and blocked |
| Partial execution | Verified | Other commands executed successfully |
| `partial_success` status | Verified | Set correctly when blocked + succeeded |
| Multi-device support | Verified | Each device filtered independently |
| Non-blocking behavior | Verified | No tool timeouts or console issues |
### Benefits Confirmed
## Future Enhancements
1. **Timeout Prevention**: The `traceroute` command that could have taken 30+ seconds was filtered, preventing tool timeout
2. **Console Availability**: Since `traceroute` was not executed, the device console remained available for subsequent commands
3. **Clear Feedback**: The LLM receives clear information about which commands were blocked and why
4. **Partial Execution**: Useful commands (`show`, `ping`) still executed, providing valuable diagnostic information
## Future Enhancements (TODO)
### Planned Improvements
1. **Regex Support**: Allow more sophisticated pattern matching
```python
# Current: simple substring match
"traceroute"
# Future: regex patterns
"^traceroute\\s+"
"ping\\s+.*\\s+-f"
```
2. **User Override File**: Allow per-project or user-specific overrides
```
/etc/gns3-server/forbidden_commands_override.txt
<project_dir>/forbidden_commands_override.txt
```
3. **Web UI Configuration**: Manage forbidden commands through GNS3 web interface
4. **Audit Logging**: Log blocked commands for security analysis
5. **Per-Command Timeouts**: Configure timeouts for specific commands instead of blocking
```python
"command_timeouts": {
"traceroute.*": 120,
"debug.*": 5
}
```
6. **Interrupt Mechanism**: Send Ctrl+C to interrupt long-running commands instead of blocking
```python
def execute_with_timeout(cmd, timeout=30):
try:
return device.execute(cmd, timeout=timeout)
except Timeout:
device.send_break() # Ctrl+C
return f"Command interrupted after {timeout}s"
```
7. **Command State Tracking**: Track device console state to ensure availability
```python
device_state = {
"console_available": True,
"current_command": None,
"last_prompt_seen": timestamp
}
```
### Advanced Features (Long-term)
- **Per-Device Filtering**: Different rules for different device types
- **Time-Based Restrictions**: Block certain commands during specific hours
- **Severity Levels**: Classify commands by severity (warn, block, allow)
- **ML-Based Detection**: Learn which commands cause problems and auto-block them
- **Regex support** for more sophisticated pattern matching
- **Per-project override files** for forbidden commands
- **Web UI configuration** for managing forbidden commands
- **Audit logging** for blocked command analysis
- **Per-command timeouts** instead of blocking
- **Interrupt mechanism** (Ctrl+C) for long-running commands
## Troubleshooting
### Commands Are Being Blocked Unexpectedly
**Problem:** A command you want to use is being blocked.
**Solution:**
1. Check the blocked command list in the result output
2. Identify which pattern is matching your command
3. Edit `forbidden_commands.txt` to remove or modify the pattern
4. Restart GNS3 server
### Forbidden Commands File Not Found
**Problem:** The system logs "Forbidden commands file not found. Using default list."
**Solution:**
1. Verify the file exists at the expected location
2. Check file permissions (should be readable by the GNS3 server process)
3. Ensure the file is not empty
### Changes Not Taking Effect
**Problem:** You edited the file but commands are still being blocked.
**Solution:**
1. Restart the GNS3 server (required to reload the configuration)
2. Or use the `reload_forbidden_commands()` function if available in your context
## Security Considerations
### 1. Why These Commands Are Blocked (Command Filtering)
| Command | Reason |
|---------|--------|
| `traceroute` | Can run for 30+ seconds, exceeds typical tool timeout |
| `tracepath` | Similar to traceroute, long execution time |
| `tracert` | Windows traceroute, same timeout issues |
| `ping -f` | Flood ping can overwhelm lab devices |
| `debug` | Debug commands can produce overwhelming output and destabilize devices |
| `test` | Test commands may affect device stability |
### 2. Configuration Safety (Prohibited Commands)
In addition to timeout-based filtering, GNS3-Copilot prohibits execution of sensitive configuration commands that could lock users out of devices or cause security issues. These restrictions are enforced at the **AI agent level** through system prompts.
**Prohibited Configuration Categories:**
| Category | Commands | Reason |
|----------|----------|--------|
| **AAA Configuration** | `aaa new-model`, `radius-server`, `tacacs-server` | May lock users out; requires manual configuration |
| **Login Passwords** | `enable secret`, `password`, `username ... password` | Can lock users out; security risk |
| **Console/VTY Authentication** | `line console 0`, `line vty 0 4`, `login local` | May block console access |
| **Password Encryption** | `service password-encryption` | Security-sensitive; manual setup required |
| **Access Control Lists** | `access-list ... deny ip any any` (on mgmt interfaces) | Can block management access |
| **Dangerous System Operations** | `reload`, `erase startup-config`, `format` | Destructive operations |
**Implementation:**
- **System Prompt**: Restrictions are defined in `lab_automation_assistant_prompt.py`
- **Behavior**: When AI detects these commands, it provides configuration guidance instead of execution
- **Example Response**:
```
"I cannot execute AAA/password commands directly as they may lock you out.
Here's how to configure them manually..."
```
**User Override:**
Users can manually execute these commands through:
1. Direct device console access
2. GNS3 device console
3. Manual SSH/Telnet connection
### 3. Multi-line Command Handling
**Problem:** Some configuration commands contain embedded newlines (e.g., `banner motd`), which cause Netmiko to fail when processed as single strings.
**Solution:** The system automatically expands multi-line commands before execution.
**Example:**
```python
# Input (single string with newlines)
["banner motd #\nWelcome\nUnauthorized access prohibited\n#"]
# After expansion
["banner motd #", "Welcome", "Unauthorized access prohibited", "#"]
```
**Implementation:**
- **Location**: `config_tools_nornir.py:_expand_multiline_commands()`
- **Detection**: Checks for `\n` newline character in commands
- **Processing**: Splits by `\n` and filters empty lines
- **Logging**: Records expansion for debugging
**Supported Commands:**
- `banner motd`, `banner login`, `banner exec`
- Multi-line ACLs
- Route-maps with continue statements
- Any command with embedded newlines
### Best Practices
1. **Education Environment**: Use the default filtering for safety
2. **Personal Lab**: Consider which commands you actually need
3. **Production-like Environment**: Keep restrictions enabled
4. **Always Understand**: Before allowing a command, understand why it was blocked
| Problem | Solution |
|---------|----------|
| Command blocked unexpectedly | Check `blocked_commands` in result, identify matching pattern, edit `forbidden_commands.txt` |
| "File not found, using defaults" | Verify `config/forbidden_commands.txt` exists and is readable |
| Changes not taking effect | Restart server or call `reload_forbidden_commands()` |
## Related Documentation
@@ -489,13 +222,3 @@ Users can manually execute these commands through:
- [GNS3-Copilot Documentation](../README.md)
- [Contributing Guide](../../CONTRIBUTING.md)
## Feedback and Issues
If you:
- Find commands that should be blocked by default
- Need to allow commands for legitimate use cases
- Have suggestions for improving the filtering system
Please submit an issue: https://github.com/yueguobin/gns3-copilot/issues
docs/gns3-copilot/implemented/context-window-management.md
+126 -251
View File
@@ -1,255 +1,161 @@
# LLM Context Window Management Implementation Document
<!--
SPDX-License-Identifier: CC-BY-SA-4.0
See LICENSE file for licensing information.
-->
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
# LLM Context Window Management
## Overview
This document explains the context window management implementation mechanism for GNS3 Copilot, including message trimming, token counting, and configuration validation.
GNS3 Copilot's context window management prevents LLM context overflow by trimming conversation history to fit within configured token limits. It uses tiktoken for accurate token counting, supports three context strategies (conservative/balanced/aggressive), and automatically injects GNS3 topology information into the system prompt before each LLM call.
## Implementation Architecture
## Architecture
### 1. Core Modules
```mermaid
graph TD
subgraph "context_manager.py"
A[create_pre_model_hook] --> B[_inject_topology_into_system]
A --> C[estimate_tool_tokens]
A --> D[trim_messages via LangChain]
E[count_tokens via tiktoken]
E --> C
E --> F[_count_tokens_for_message]
F --> D
end
**File Location**: `gns3server/agent/gns3_copilot/agent/context_manager.py`
subgraph "gns3_copilot.py"
G[llm_call node] --> H[Get topology via GNS3TopologyTool]
H --> A
A --> I[model_with_tools.invoke]
end
#### Token Counting Strategy
The system uses **tiktoken** for token counting (context_manager.py:60):
```python
_tiktoken_encoding = tiktoken.get_encoding("cl100k_base")
style A fill:#4a9eff,color:#fff
style D fill:#ff6b6b,color:#fff
style E fill:#51cf66,color:#fff
```
**Required Dependency**:
```bash
pip install tiktoken>=0.8.0
## Key Components
| Function | File | Purpose |
|----------|------|---------|
| `create_pre_model_hook()` | `context_manager.py` | Factory that creates the preprocessing hook |
| `_inject_topology_into_system()` | `context_manager.py` | Injects topology into system prompt via `{{topology_info}}` placeholder |
| `count_tokens()` | `context_manager.py` | Accurate token counting using tiktoken (`cl100k_base`) |
| `estimate_tool_tokens()` | `context_manager.py` | Estimates token cost of tool definitions (JSON serialization) |
| `_count_tokens_for_message()` | `context_manager.py` | Token counter adapter for LangChain's `trim_messages` |
| `llm_call()` | `gns3_copilot.py` | StateGraph node that orchestrates the full LLM call pipeline |
**Token Counting**: Uses tiktoken `cl100k_base` encoding (GPT-4 compatible, 95%+ accuracy for most modern LLMs).
**tiktoken Cache**: Encoding files are cached locally at `<agent_package>/cache/tiktoken/` to avoid repeated downloads. This is configured before tiktoken import via `TIKTOKEN_CACHE_DIR` environment variable.
**Required Dependency**: `tiktoken>=0.8.0` — if not installed, `ModuleNotFoundError` is raised at startup.
## Trimming Process
```mermaid
flowchart TD
A[llm_call node invoked] --> B[Get topology via GNS3TopologyTool]
B --> C[Create pre_model_hook]
C --> D[Inject topology into system prompt]
D --> E{topology data available?}
E -->|Yes| F[Replace placeholder with topology content]
E -->|No| G[Replace placeholder with 'No topology information available']
F --> H[Estimate tool definition tokens]
G --> H
H --> I[Calculate trim_messages budget]
I --> J[trim_messages with strategy='last']
J --> K[Return prepared messages]
K --> L[model.invoke with prepared messages]
```
If tiktoken is not installed, the system will throw a `ModuleNotFoundError` at startup.
### Token Budget Calculation
#### Key Functions
**`count_tokens(text: str) -> int`** (context_manager.py:84-100)
- Uses tiktoken to accurately count tokens in text
- Uses `cl100k_base` encoding
- Returns the exact token count
**`estimate_tool_tokens(tools: list) -> int`** (context_manager.py:103-169)
- Serializes tool schema to JSON
- Uses tiktoken to count token consumption of tool definitions
- Supports Pydantic v1/v2 compatibility
- Falls back to 1000 tokens on failure
**`create_pre_model_hook(...)`** (context_manager.py:195-402)
- Creates a preprocessing function (pre_model_hook)
- Automatically executes before each LLM call:
1. Injects topology information into system prompt
2. Estimates token consumption of tool definitions
3. Trims message history to fit context limits
- Returns a callable function for preparing messages
### 2. Detailed Trimming Logic
#### 2.1 Token Budget Allocation
When calling the LLM, the content sent consists of two parts:
```
Complete request sent to LLM:
┌─────────────────────────────────────────────────────────────┐
│ 1. Messages (managed by us) │
│ ├─ SystemMessage: system prompt + topology (template injection) │
│ └─ HumanMessage/AIMessage: user messages / history messages │
├─────────────────────────────────────────────────────────────┤
│ 2. Tool Definitions (LangChain adds automatically, not in messages) │
│ ├─ Tool 1 schema (name, description, parameters) │
│ ├─ Tool 2 schema │
│ └─ ... (about 500-1500 tokens per tool) │
└─────────────────────────────────────────────────────────────┘
```mermaid
flowchart LR
A["context_limit<br/>(e.g. 128K)"] -->|"× strategy_ratio"| B["Input Budget<br/>(e.g. 96K @ balanced)"]
B -->|"− tool_tokens"| C["trim_messages Budget<br/>(includes system message)"]
C --> D["trim_messages<br/>max_tokens parameter"]
```
**System Message Structure**:
- Uses template variable `{{topology_info}}` to dynamically inject topology
- System prompt contains placeholder: `"### CURRENT TOPOLOGY\n{{topology_info}}"`
- If topology exists, replaces with actual content
- If no topology, replaces with `"(No topology information available)"`
**Key**: `max_tokens` passed to `trim_messages` **includes** the system message. System tokens are NOT subtracted separately — `trim_messages` handles system message preservation via `include_system=True`.
#### 2.2 Trimming Process
### Budget Calculation Example
```
Step 1: Calculate Input Budget
┌─────────────────────────────────────────────────────────────┐
│ context_limit: 128,000 tokens (128K) │
│ strategy: balanced (75%) │
│ │
│ Input budget = 128 × 1000 × 0.75 = 96,000 tokens │
└─────────────────────────────────────────────────────────────┘
↓
Step 2: Subtract Tool Definitions
┌─────────────────────────────────────────────────────────────┐
│ Input budget: 96,000 tokens │
│ Tool definitions: 1,725 tokens │
│ │
│ Available for messages = 96,000 - 1,725 = 94,275 tokens │
└─────────────────────────────────────────────────────────────┘
↓
Step 3: trim_messages Processing
┌─────────────────────────────────────────────────────────────┐
│ Call LangChain's trim_messages: │
│ - max_tokens = 94,275 (includes system message) │
│ - strategy = "last" (keep latest messages) │
│ - token_counter = tiktoken counting function │
│ - include_system = True (always keep system) │
│ │
│ trim_messages will: │
│ 1. Keep SystemMessage (system + topology) │
│ 2. Starting from latest messages, keep as much history │
│ 3. When exceeding limit, discard oldest messages │
└─────────────────────────────────────────────────────────────┘
```
| Step | Calculation | Result |
|------|------------|--------|
| Context limit | 128 (configured) | 128K tokens |
| Strategy ratio | balanced = 0.75 | 128 × 1000 × 0.75 = 96,000 |
| Tool definitions | estimated from JSON schema | ~1,725 tokens |
| trim_messages budget | 96,000 − 1,725 | **94,275 tokens** (includes system) |
#### 2.3 Trimming Priority
### Trimming Priority
The system preserves content in the following priority order:
| Priority | Content | Behavior |
|----------|---------|----------|
| 1 | System Message (system prompt + topology) | Always kept (`include_system=True`) |
| 2 | Latest conversation messages | Kept from newest to oldest |
| 3 | Older conversation history | Discarded first when exceeding budget |
| Priority | Content | Description |
|----------|---------|-------------|
| 1️⃣ | System Message (system prompt + topology) | Never removed |
| 2️⃣ | Latest user message | Keep at least the last 1 |
| 3️⃣ | Old conversation history | Discarded in chronological order |
**Note**: System prompt and topology are merged into a single `SystemMessage` via the `{{topology_info}}` template variable and cannot be trimmed independently.
**Note**: System prompt and topology info are merged into one SystemMessage via template variable and cannot be separated.
### Edge Case Handling
#### 2.4 Edge Case Handling
| Scenario | Handling |
| Scenario | Behavior |
|----------|----------|
| System (including topology) > budget | Keep complete SystemMessage (cannot separate system and topology) |
| Tools > budget | ERROR log, suggest increasing context_limit or reducing tool count |
| All history trimmed | Keep last 1 user message |
| System + tools > input budget | ERROR log with recommendations (reduce prompt/tools, use larger model, switch to conservative). `trim_messages` is still called — `include_system=True` ensures system message is kept, but no history messages will fit. |
| Trim budget < system × 1.5 | WARNING log: minimal room for conversation history |
| Invalid context_strategy | Falls back to `balanced` with WARNING log |
| `trim_messages` throws exception | Returns untrimmed messages (graceful degradation) |
| Missing `{{topology_info}}` placeholder | Returns original messages without injection, WARNING log |
| Topology retrieval fails | Injects `"(No topology information available)"` placeholder |
**Important Notes**:
- When system + topology exceed available budget, **both are preserved**
- Cannot discard only topology while keeping system prompt (already merged)
## Integration with GNS3 Copilot
### 3. Integration with GNS3 Copilot
**File**: `gns3server/agent/gns3_copilot/agent/gns3_copilot.py`
**File Location**: `gns3server/agent/gns3_copilot/agent/gns3_copilot.py`
### Why Direct Call, Not Config-based?
#### Implementation Method
**Key Point**: The system uses a **custom StateGraph**, not LangGraph's pre-built agent.
Therefore, `pre_model_hook` cannot be passed via `model.invoke(config={"configurable": {"pre_model_hook": ...}})`.
**Correct Usage**: **Directly call** the `pre_hook` function to prepare messages.
```python
def llm_call(state: dict, config: RunnableConfig | None = None):
"""LLM decides whether to call a tool or not."""
# 1. Get topology information
project_id = config["configurable"].get("project_id")
topology_info = None
if project_id:
topology_tool = GNS3TopologyTool()
topology = topology_tool._run(project_id=project_id)
if topology and "error" not in topology:
topology_info = topology
# 2. Create pre_model_hook
system_prompt = load_system_prompt()
pre_hook = create_pre_model_hook(
system_prompt=system_prompt,
get_topology_func=lambda s: s.get("topology_info"),
get_llm_config_func=get_current_llm_config,
get_tools_func=lambda: tools,
)
# 3. Create model with tools
model_with_tools = create_base_model_with_tools(tools, llm_config=llm_config)
# 4. ⭐ Key: directly call pre_hook to prepare messages
logger.info("Calling pre_hook to prepare %d messages", len(messages))
prepared_state = pre_hook({"messages": messages, "topology_info": topology_info})
prepared_messages = prepared_state["messages"]
# 5. Use prepared messages to call LLM
response = model_with_tools.invoke(prepared_messages)
return {"messages": [response], ...}
```
#### Why Not Pass via Config?
LangGraph's `pre_model_hook` parameter only applies to **pre-built agents**, not custom StateGraphs.
The agent uses a **custom StateGraph** (`StateGraph(MessagesState)`), not LangGraph's pre-built agent. LangGraph's `pre_model_hook` config parameter only works with pre-built agents like `create_react_agent`.
| Agent Type | pre_model_hook Support |
|------------|------------------------|
| `create_react_agent` | ✅ Via `pre_model_hook` parameter |
| `chat_agent_executor` | ✅ Via `pre_model_hook` parameter |
| **Custom StateGraph** | ❌ **Not supported**, need to call directly |
| `create_react_agent` | Via `config={"configurable": {"pre_model_hook": ...}}` |
| `chat_agent_executor` | Via `config={"configurable": {"pre_model_hook": ...}}` |
| **Custom StateGraph** | **Not supported** — must call directly |
Our implementation uses a custom StateGraph (`agent_builder = StateGraph(MessagesState)`), so we must call `pre_hook` directly.
### Execution Flow
### 4. Execution Flow
In the `llm_call` node of `gns3_copilot.py`:
```
User sends message
↓
llm_call node is called
↓
Get project_id (from config["configurable"])
↓
Call GNS3TopologyTool._run(project_id) to get topology
↓
Store topology_info to state
↓
Create pre_model_hook (via create_pre_model_hook())
↓
[Key] Directly call pre_hook({"messages": messages, "topology_info": topology_info})
├─ 1. Inject topology into system prompt
├─ 2. Estimate tool definitions tokens
├─ 3. Call trim_messages() to trim messages
└─ 4. Return prepared message list
↓
Call model.invoke() with prepared messages
↓
Return LLM response
```
1. Get `llm_config` from request-scoped context variable
2. Get `project_id` from `config["configurable"]`
3. Retrieve topology via `GNS3TopologyTool._run(project_id)`
4. Select tools based on `copilot_mode` (`teaching_assistant` vs `lab_automation_assistant`)
5. Load system prompt and create `pre_model_hook`
6. **Directly call** `pre_hook({"messages": messages, "topology_info": topology_info})` to prepare messages
7. Invoke `model_with_tools.invoke(prepared_messages)`
---
> **Note**: The actual `llm_call` function also includes defensive checks for missing config, empty messages, and topology retrieval errors — see source code for full details.
## Strategy Implementation
### Context Strategy Ratios
**Definition** (context_manager.py:68-72):
Defined as `CONTEXT_STRATEGY_RATIOS` in `context_manager.py`, with default `DEFAULT_CONTEXT_STRATEGY = "balanced"`.
```python
CONTEXT_STRATEGY_RATIOS = {
"conservative": 0.60,
"balanced": 0.75,
"aggressive": 0.85,
}
```
**Default Value** (context_manager.py:74):
```python
DEFAULT_CONTEXT_STRATEGY = "balanced"
```
### Strategy Comparison
| Strategy | Input Ratio | Output Reserved | Calculation Formula |
|----------|-------------|-----------------|---------------------|
| Strategy | Input Ratio | Output Reserved | Calculation |
|----------|-------------|-----------------|-------------|
| Conservative | 60% | 40% | `context_limit × 1000 × 0.60` |
| Balanced | 75% | 25% | `context_limit × 1000 × 0.75` |
| Aggressive | 85% | 15% | `context_limit × 1000 × 0.85` |
---
## Log Output
### Normal Case (topology successfully injected)
### Normal Case (topology injected)
```
INFO: Calling pre_hook to prepare 1 messages
@@ -268,7 +174,7 @@ INFO: Messages trimmed: 50 → 25 msgs. Total: ~82000 tokens + 1725 tools = 8372
INFO: Messages prepared: 50 → 25
```
### When topology is None
### When Topology is None
```
INFO: Calling pre_hook to prepare 1 messages
@@ -276,53 +182,22 @@ WARNING: ✗ Topology data is None, injecting placeholder
INFO: Context ready: 2 msgs, ~800 tokens + 1725 tools = 2525 / 128K (2.0%), strategy=balanced
```
---
## Error Handling
### tiktoken Not Installed
| Error | Behavior | Resolution |
|-------|----------|------------|
| tiktoken not installed | `ModuleNotFoundError` at startup | `pip install tiktoken>=0.8.0` |
| `context_limit` missing | `ValueError` raised | Add `context_limit` to LLM config |
| Invalid `context_limit` | `ValueError` raised | Must be positive integer |
| Invalid `context_strategy` | WARNING, fallback to `balanced` | Use `conservative`/`balanced`/`aggressive` |
| `trim_messages` failure | ERROR log, returns untrimmed messages | Check message format compatibility |
If tiktoken is not installed, the system will throw an error at startup:
## Deprecated API
```python
ModuleNotFoundError: No module named 'tiktoken'
```
**Solution**:
```bash
pip install tiktoken>=0.8.0
```
### context_limit Missing or Invalid
If there is no `context_limit` in the LLM configuration or the value is invalid (context_manager.py:285-295):
```python
if "context_limit" not in llm_config:
raise ValueError("context_limit is required in LLM config")
limit = llm_config["context_limit"]
if not isinstance(limit, int) or limit <= 0:
raise ValueError(f"Invalid context_limit: {limit}")
```
### Trimming Failure
```python
try:
trimmed = trim_messages(...)
except Exception as e:
logger.error("Failed to trim messages: %s", e)
logger.warning("Returning original messages due to trimming error")
return {"messages": messages_with_system}
```
---
`prepare_context_messages()` is deprecated. It remains in `context_manager.py` for backward compatibility but emits `DeprecationWarning`. Use `create_pre_model_hook()` instead.
## Related Source Files
- `gns3server/agent/gns3_copilot/agent/context_manager.py` - Context management core logic
- `gns3server/agent/gns3_copilot/agent/gns3_copilot.py` - LLM call node (StateGraph)
- `gns3server/agent/gns3_copilot/agent/model_factory.py` - Model creation and tool binding
- `gns3server/agent/gns3_copilot/agent/context_manager.py` — Context management core logic
- `gns3server/agent/gns3_copilot/agent/gns3_copilot.py` — LLM call node (StateGraph)
- `gns3server/agent/gns3_copilot/agent/model_factory.py` — Model creation and tool binding
docs/gns3-copilot/implemented/llm-model-configs.md
+21 -7
View File
@@ -1,3 +1,11 @@
<!--
SPDX-License-Identifier: CC-BY-SA-4.0
See LICENSE file for licensing information.
-->
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
# LLM Model Configurations API
## Overview
@@ -40,7 +48,7 @@ User's own config > User's group config
| `config_id` | UUID | Primary key |
| `name` | VARCHAR(100) | Configuration name (table-level for indexing) |
| `model_type` | VARCHAR(50) | Model type (table-level for filtering) |
| `config` | JSONB | Configuration data (provider, base_url, model, temperature, api_key, etc.) |
| `config` | JSON (JSONB on PostgreSQL) | Configuration data (provider, base_url, model, temperature, api_key, etc.) |
| `user_id` | UUID (nullable) | Foreign key to users table |
| `group_id` | UUID (nullable) | Foreign key to user_groups table |
| `is_default` | BOOLEAN | Default configuration flag |
@@ -77,13 +85,15 @@ GNS3-Copilot uses LangChain's `init_chat_model` function, which supports the fol
| Provider | Provider Value | Default base_url | base_url Required? | Notes |
|----------|---------------|------------------|-------------------|-------|
| OpenAI | `openai` | `https://api.openai.com/v1` | ❌ No | Most popular, supports GPT-4, GPT-3.5 |
| Anthropic | `anthropic` | `https://api.anthropic.com` | ❌ No | Claude 3.5 Sonnet, Claude 3 Opus |
| Google | `google` | `https://generativelanguage.googleapis.com` | ❌ No | Gemini Pro, Gemini Flash |
| OpenAI | `openai` | `https://api.openai.com/v1` | ✅ Yes (planned optional) | Most popular, supports GPT-4, GPT-3.5 |
| Anthropic | `anthropic` | `https://api.anthropic.com` | ✅ Yes (planned optional) | Claude 3.5 Sonnet, Claude 3 Opus |
| Google | `google` | `https://generativelanguage.googleapis.com` | ✅ Yes (planned optional) | Gemini Pro, Gemini Flash |
| AWS Bedrock | `aws` | Varies by region | ✅ Yes | Requires AWS configuration |
| Ollama | `ollama` | `http://localhost:11434` | ✅ Yes | Local models, typically running on localhost |
| DeepSeek | `deepseek` | `https://api.deepseek.com` | ❌ No | DeepSeek Chat, DeepSeek Coder |
| xAI | `xai` | `https://api.x.ai` | ❌ No | Grok models |
| DeepSeek | `deepseek` | `https://api.deepseek.com` | ✅ Yes (planned optional) | DeepSeek Chat, DeepSeek Coder |
| xAI | `xai` | `https://api.x.ai` | ✅ Yes (planned optional) | Grok models |
> **Note:** The `base_url` field is currently **required** in the API schema for all providers. Entries marked "planned optional" indicate providers where `base_url` may become optional in a future release (see [Future Enhancements](#optional-base_url-field)). For now, use the default endpoint URL listed in the table.
### When to Specify `base_url`
@@ -169,7 +179,7 @@ For a complete list of LangChain-supported providers, see: https://python.langch
| Method | Path | Description | Privilege |
|--------|------|-------------|-----------|
| GET | `/v3/access/users/{user_id}/llm-model-configs` | Get user's effective configs (own + inherited) | User.Audit |
| GET | `/v3/access/users/{user_id}/llm-model-configs/own` | Get user's own configs only | User.Audit |
| GET | `/v3/access/users/{user_id}/llm-model-configs/own` | Get user's own configs only (returns `List[LLMModelConfigResponse]`, a plain array) | User.Audit |
| GET | `/v3/access/users/{user_id}/llm-model-configs/default` | Get user's default configuration | User.Audit |
| POST | `/v3/access/users/{user_id}/llm-model-configs` | Create a new configuration | User.Modify |
| PUT | `/v3/access/users/{user_id}/llm-model-configs/{config_id}` | Update a configuration | User.Modify |
@@ -242,6 +252,7 @@ For a complete list of LangChain-supported providers, see: https://python.langch
| `context_limit` | integer (optional) | Model context window limit in K tokens |
| `context_strategy` | string (optional) | Context trimming strategy |
| `is_default` | boolean (optional) | Default flag |
| `copilot_mode` | string (optional) | GNS3-Copilot mode: "teaching_assistant" or "lab_automation_assistant" |
| `expected_version` | integer (optional) | **Optimistic locking version** |
**Note:** When using `expected_version`, the API will verify the version hasn't changed since you read the data. If it has, you'll receive a 409 Conflict error.
@@ -249,6 +260,9 @@ For a complete list of LangChain-supported providers, see: https://python.langch
**Reserved Fields:**
- `max_tokens`: Reserved for future implementation. Currently not used by the system. The maximum output tokens are controlled automatically by the LLM provider based on the model and input size.
**Update Limitations:**
- `context_strategy` and `copilot_mode` can be set to new values but **cannot be cleared to null** via update. This is because the API filters out null values before processing.
### LLMModelConfigResponse
| Field | Type | Description |
docs/gns3-copilot/implemented/multi-vendor-device-support.md
+197 -467
View File
@@ -1,3 +1,11 @@
<!--
SPDX-License-Identifier: CC-BY-SA-4.0
See LICENSE file for licensing information.
-->
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
# Multi-Vendor Network Device Support
## Overview
@@ -6,12 +14,12 @@ GNS3-Copilot supports network devices from multiple vendors through Netmiko and
## Supported Vendors
| Vendor | Platform | Device Type | Protocol | Status |
|--------|----------|-------------|----------|--------|
| **Cisco** | `cisco_ios` | `cisco_ios_telnet` | Telnet | ✅ Tested |
| **Huawei** | `huawei` | `gns3_huawei_telnet_ce` | Telnet | ✅ Tested (Custom Driver) |
| **Ruijie (锐捷)** | `ruijie_os` | `gns3_ruijie_telnet` | Telnet | ✅ Tested (Custom Driver) |
| **VPCS** | `vpcs` | `gns3_vpcs_telnet` | Telnet | ✅ Tested (Custom Driver) |
| Vendor | Device Type | Protocol | Status |
|--------|-------------|----------|--------|
| **Cisco** | `cisco_ios_telnet` | Telnet | ✅ Tested |
| **Huawei** | `gns3_huawei_telnet_ce` | Telnet | ✅ Tested (Custom Driver) |
| **Ruijie (锐捷)** | `gns3_ruijie_telnet` | Telnet | ✅ Tested (Custom Driver) |
| **VPCS** | `gns3_vpcs_telnet` | Telnet | ✅ Tested (Custom Driver, Simulator) |
## Custom VPCS Driver (`VPCSTelnet`)
@@ -25,10 +33,13 @@ VPCS (Virtual PC Simulator) is a lightweight virtual PC simulator used in GNS3 l
### Solution: Lightweight Custom Driver
```
BaseConnection (Netmiko base class)
↓
VPCSTelnet (Custom GNS3 driver)
```mermaid
graph TD
A[BaseConnection<br/>Netmiko base class] --> B[VPCSTelnet<br/>Custom GNS3 driver]
B --> C[No authentication]
B --> D[Simple prompt: PC\d+]
B --> E[No config mode]
B --> F[ANSI stripping]
```
**Why Not Use Standard Telnet Driver?**
@@ -81,26 +92,38 @@ def disable_paging(self) -> str:
return "" # VPCS doesn't use paging
```
**5. Custom `send_command`**
VPCS overrides `send_command` with non-standard defaults:
- `strip_prompt=False`, `strip_command=False`, `normalize=False` — VPCS output doesn't need standard Netmiko processing
- Uses `_strip_ansi_codes()` to clean terminal escape sequences from VPCS output
**6. ANSI Escape Code Stripping**
VPCS output often contains ANSI terminal codes. The `_strip_ansi_codes()` method strips bold, underline, reset, and color codes before returning output to callers.
### VPCS Tool Usage
The VPCS driver is used by the `execute_vpcs_commands` tool:
```python
from gns3server.agent.gns3_copilot.tools_v2.vpcs_tools_netmiko import VPCSCommands
> File: `gns3server/agent/gns3_copilot/tools_v2/vpcs_tools_netmiko.py`
tool = VPCSCommands()
result = tool._run(json.dumps({
"project_id": "<PROJECT_UUID>",
"device_configs": [
{
"device_name": "PC1",
"commands": [
"ip 10.10.0.12/24 10.10.0.254",
"ping 10.10.0.254"
]
**VPCS Connection Parameters:**
VPCS devices use additional Netmiko parameters for reliable connections:
| Parameter | Value | Reason |
|-----------|-------|--------|
| `fast_cli` | `False` | VPCS is slow, disable fast CLI mode |
| `global_delay_factor` | `2.0` | Double the delay between commands |
```python
"connection_options": {
"netmiko": {
"extras": {
"device_type": "gns3_vpcs_telnet",
"fast_cli": False,
"global_delay_factor": 2.0,
}
]
}))
}
}
```
### VPCS Built-in Template Configuration
@@ -111,34 +134,23 @@ VPCS nodes created from the built-in template automatically include the necessar
| Tag | Value | Purpose |
|-----|-------|---------|
| `platform` | `vpcs` | Platform identification |
| `device_type` | `gns3_vpcs_telnet` | Netmiko driver selection |
**Built-in Template Definition:**
```python
# gns3server/services/templates.py
{
"template_id": uuid.uuid5(uuid.NAMESPACE_X500, "vpcs"),
"template_type": "vpcs",
"name": "VPCS",
"default_name_format": "PC{0}",
"category": "guest",
"symbol": "vpcs_guest",
"builtin": True,
"tags": ["platform:vpcs", "device_type:gns3_vpcs_telnet"], # ✅ Auto-applied
}
```
> File: `gns3server/services/templates.py`
The VPCS built-in template includes `"tags": ["device_type:gns3_vpcs_telnet"]`, which is automatically applied to all VPCS nodes created from the template.
**User Benefits:**
- ✅ **No manual tagging required** - Tags are applied automatically when creating VPCS nodes
- ✅ **Automatic driver selection** - Copilot tools automatically use the correct Netmiko driver
- ✅ **Consistent behavior** - All VPCS nodes from the built-in template work identically
- ✅ **Zero configuration** - Users don't need to understand device_type tags
- ✅ **No manual tagging required** — Tags are applied automatically when creating VPCS nodes
- ✅ **Automatic driver selection** — Copilot tools automatically use the correct Netmiko driver
- ✅ **Consistent behavior** — All VPCS nodes from the built-in template work identically
- ✅ **Zero configuration** — Users don't need to understand device_type tags
**How It Works:**
1. User creates a VPCS node from the built-in "VPCS" template
2. Node automatically inherits the tags: `platform:vpcs` and `device_type:gns3_vpcs_telnet`
3. Copilot tools read these tags and select the appropriate VPCS Netmiko driver
2. Node automatically inherits the tag: `device_type:gns3_vpcs_telnet`
3. Copilot tools read the tag and select the appropriate VPCS Netmiko driver
4. Commands execute using the VPCS-optimized driver (no authentication, simple prompts)
### Supported VPCS Commands
@@ -192,14 +204,14 @@ Telnet Connection → Direct access to command line (no login prompts)
### Solution: Custom Driver Architecture
```
BaseConnection (Netmiko base class)
↓
CiscoBaseConnection (Cisco-style base class)
↓
HuaweiBase (Huawei device base class) ← Inherits VRP support
↓
GNS3HuaweiTelnetCE (Custom GNS3 driver) ← Overrides telnet_login only
```mermaid
graph TD
A[BaseConnection] --> B[CiscoBaseConnection]
B --> C[HuaweiBase<br/>VRP support, system-view,<br/>prompt patterns, paging]
C --> D[GNS3HuaweiTelnetCE<br/>Overrides telnet_login only]
D --> E[Skip authentication]
D --> F[Auto-commit before exit]
D --> G[y/n auto-confirm]
```
**Why Inherit from HuaweiBase?**
@@ -258,20 +270,18 @@ custom_netmiko/
#### Limitations
**Authentication Requirement:**
- The `gns3_huawei_telnet_ce` driver is designed for GNS3 devices **without authentication**
- If your Huawei device has been configured with a username/password:
- **Option 1**: Use the standard `huawei_telnet` driver (requires username/password)
- **Option 2**: Remove authentication from the device for GNS3 testing
- The driver does **not** currently auto-detect authentication requirements
**Why Not Use the Standard `huawei_telnet` Driver?**
- The standard Netmiko Huawei driver (`huawei_telnet`) has known issues in GNS3 emulation environments
- This is the primary reason the custom `gns3_huawei_telnet_ce` driver was developed
- The custom driver is designed for GNS3 devices **without authentication**
- If your GNS3 Huawei device has been configured with authentication, remove it for GNS3 testing to use the custom driver
**When to Use Each Driver:**
**When to Use:**
| Scenario | Use Driver | Requires Credentials? |
|----------|-----------|----------------------|
| GNS3 Huawei (fresh, no auth) | `gns3_huawei_telnet_ce` | ❌ No |
| GNS3 Huawei (configured with username/password) | `huawei_telnet` | ✅ Yes |
| Real Huawei hardware | `huawei_telnet` | ✅ Yes |
| Scenario | Recommendation |
|----------|---------------|
| GNS3 Huawei device | Always use `gns3_huawei_telnet_ce` |
| GNS3 device has auth configured | Remove auth, then use `gns3_huawei_telnet_ce` |
#### Method Overrides
@@ -393,14 +403,14 @@ Standard Netmiko's `send_config_set()` waits for a prompt pattern, but the `[yes
### Solution: Hybrid Strategy with Interactive Prompt Handling
```
BaseConnection (Netmiko base class)
↓
CiscoBaseConnection (Cisco-style base class)
↓
RuijieOSBase (Netmiko's Ruijie implementation)
↓
RuijieTelnetEnhanced (Custom GNS3 driver) ← Adds interactive prompt handling
```mermaid
graph TD
A[BaseConnection] --> B[CiscoBaseConnection]
B --> C[RuijieOSBase<br/>Netmiko built-in]
C --> D[RuijieTelnetEnhanced<br/>Interactive prompt handling]
D --> E[Preprocessing<br/>Insert yes after known commands]
D --> F[Batch send<br/>Fast path 2-3s]
D --> G[Fallback one-by-one<br/>Slow but reliable ~10s]
```
**Why Hybrid Strategy?**
@@ -430,35 +440,22 @@ INTERACTIVE_PATTERNS = [
```
**2. Hybrid Send Strategy**
```
┌─────────────────────────────────────┐
│ Input Configuration Commands │
└──────────────┬──────────────────────┘
↓
┌─────────────────────────────────────┐
│ Step 1: Preprocessing │
│ - Detect interactive commands │
│ - Insert 'yes' after them │
└──────────────┬──────────────────────┘
↓
┌─────────────────────────────────────┐
│ Step 2: Try Batch Send (Fast) │
│ - Write all commands rapidly │
│ - Read output once │
│ - last_read=2.0s (Netmiko standard) │
└──────────────┬──────────────────────┘
↓
Success? ──Yes──→ Return output
│
No
↓
┌─────────────────────────────────────┐
│ Step 3: Fallback (Slow but Reliable)│
│ - Send commands one-by-one │
│ - Detect prompts after each command │
│ - Send 'yes' when needed │
│ - last_read=0.5s per command │
└─────────────────────────────────────┘
```mermaid
flowchart TD
A[Input Configuration Commands] --> B[Preprocessing]
B --> B1[Detect interactive commands]
B1 --> B2[Insert 'yes' after them]
B2 --> C[Try Batch Send - Fast]
C --> C1[Write all commands rapidly]
C1 --> C2[Read output once, last_read=2.0s]
C2 --> D{Success?}
D -->|Yes| E[Return output]
D -->|No| F[Fallback: One-by-One]
F --> F1[Send each command individually]
F1 --> F2[Detect prompts after each]
F2 --> F3[Send 'yes' when needed, last_read=0.5s]
F3 --> E
```
**3. Batch Send Performance**
@@ -482,7 +479,6 @@ if re.search(r"\[yes/no\]", new_output, re.IGNORECASE):
**For Ruijie devices in GNS3:**
```
device_type:gns3_ruijie_telnet
platform:ruijie_os
```
**Example Usage:**
@@ -514,38 +510,36 @@ with ConnectHandler(**device) as conn:
- **Not Covered**: Unknown or vendor-specific interactive prompts
- **Fallback**: If batch fails, falls back to one-by-one with real-time detection
**When to Use Each Driver:**
**When to Use:**
| Scenario | Use Driver |
|----------|------------|
| GNS3 Ruijie (known commands) | `gns3_ruijie_telnet` (batch works) |
| GNS3 Ruijie (unknown commands) | `gns3_ruijie_telnet` (auto-fallback) |
| Real Ruijie hardware | `ruijie_os_telnet` (standard) |
| GNS3 Ruijie (known interactive commands) | `gns3_ruijie_telnet` (batch + auto-fallback) |
| GNS3 Ruijie (unknown interactive prompts) | `gns3_ruijie_telnet` (auto-fallback to one-by-one) |
## Dynamic Device Type Detection
### GNS3 Node Tags
Device type and platform are extracted from GNS3 node tags:
Device type is extracted from GNS3 node tags:
```
device_type:gns3_huawei_telnet_ce → Netmiko device type (precise)
platform:huawei → Nornir platform (high-level)
device_type:gns3_huawei_telnet_ce → Netmiko driver selection
```
**Tag Examples:**
| Vendor | Device Type Tag | Platform Tag | Template Source |
|--------|----------------|--------------|------------------|
| Cisco IOS | `device_type:cisco_ios_telnet` | `platform:cisco_ios` | User appliance |
| Huawei CE | `device_type:gns3_huawei_telnet_ce` | `platform:huawei` | User appliance |
| Ruijie | `device_type:gns3_ruijie_telnet` | `platform:ruijie_os` | User appliance |
| **VPCS** | `device_type:gns3_vpcs_telnet` | `platform:vpcs` | **Built-in ✅** |
| Vendor | Device Type Tag | Template Source |
|--------|----------------|-----------------|
| Cisco IOS | `device_type:cisco_ios_telnet` | User appliance |
| Huawei CE | `device_type:gns3_huawei_telnet_ce` | User appliance |
| Ruijie | `device_type:gns3_ruijie_telnet` | User appliance |
| **VPCS** | `device_type:gns3_vpcs_telnet` | **Built-in ✅** |
**VPCS Built-in Template:**
- VPCS has a **built-in template** with pre-configured tags
- Tags are **automatically applied** when creating VPCS nodes
- No manual configuration required - works out of the box
- VPCS has a **built-in template** with pre-configured `device_type` tag
- The tag is **automatically applied** when creating VPCS nodes
- No manual configuration required — works out of the box
- Other devices require users to import appliances and configure tags manually
### Nornir Best Practice: Host-Level Connection Configuration
@@ -556,7 +550,6 @@ The system uses **Nornir's configuration priority** (host > group > defaults) to
# From get_gns3_device_port.py
hosts_data[device_name] = {
"port": console_port,
"platform": platform, # Reserved for future use (NAPALM, scrapli)
"groups": ["network_devices"], # All devices share one group
"connection_options": {
"netmiko": {
@@ -570,305 +563,77 @@ hosts_data[device_name] = {
- ✅ Each device has its own `device_type` (host-level config)
- ✅ All devices share common settings via group inheritance (`hostname`, `timeout`)
- ✅ No need to dynamically create multiple groups for each device type
- ✅ Cleaner code structure - single generic group for all devices
- ✅ Cleaner code structure — single generic group for all devices
- ✅ Follows Nornir best practice: "configuration proximity"
**Configuration Priority:**
```
Host Level (connection_options.device_type)
↓ OVERRIDES
Group Level (hostname, timeout, username, password)
↓ OVERRIDES
Defaults Level (data.location)
```mermaid
graph TD
A["Host Level<br/>connection_options.device_type<br/>(gns3_huawei_telnet_ce, cisco_ios_telnet, etc.)"]
B["Group Level<br/>hostname=127.0.0.1, timeout=120<br/>username, password"]
C["Defaults Level<br/>data.location=gns3"]
A -->|overrides| B
B -->|overrides| C
```
**Before (Old Approach - Dynamic Groups):**
```python
# Had to create multiple groups dynamically
groups = {
"cisco_ios_telnet": {"device_type": "cisco_ios_telnet", ...},
"huawei_telnet": {"device_type": "gns3_huawei_telnet_ce", ...},
"juniper_junos": {"device_type": "juniper_junos_telnet", ...},
}
# Each host assigned to its vendor-specific group
```
**After (Current Approach - Host-Level Config):**
```python
# Single group for shared settings
groups = {
"network_devices": {
"hostname": "127.0.0.1",
"timeout": 120,
"username": "",
"password": "",
}
}
# Each host has device-specific connection_options
# Host config overrides group config automatically
```
| Aspect | Old: Dynamic Groups | New: Host-Level Config |
|--------|--------------------|-----------------------|
| Group count | One per vendor type | Single `network_devices` group |
| Device type location | Group `connection_options` | Host `connection_options` |
| Complexity | Dynamic group creation logic | No dynamic group logic |
## Architecture Evolution
### Problem: Multi-Vendor Device Support
**Initial Challenge:**
```
Topology: Cisco R1 + Huawei SW1 + Juniper SRX
↓
Need: Different Netmiko drivers for each device
↓
Question: How to configure Nornir for multiple device types?
```mermaid
graph TD
A[Topology: Cisco R1 + Huawei SW1 + Juniper SRX] --> B[Need: Different Netmiko drivers per device]
B --> C[Question: How to configure Nornir for multiple device types?]
```
### Solution Evolution
#### ❌ Approach 1: Single Group with First Device's Type (Initial Implementation)
| Approach | Strategy | Issue |
|----------|----------|-------|
| ❌ Approach 1 | Single group with first device's `device_type` | All devices use wrong driver |
| ❌ Approach 2 | Dynamic groups per vendor (`huawei_telnet`, `cisco_telnet`, ...) | Complex logic, code duplication, not Nornir best practice |
| ✅ Approach 3 (Current) | **Host-level `connection_options`** + single generic group | Clean, follows Nornir host > group > defaults priority |
```python
# PROBLEM: Only uses first device's configuration
def _initialize_nornir(hosts_data):
first_device = next(iter(hosts_data.values()))
device_type = first_device["device_type"] # Only one type!
return InitNornir(
inventory={
"options": {
"hosts": hosts_data, # Has multiple device types
"groups": {
"network_devices": {
"connection_options": {
"netmiko": {"extras": {"device_type": device_type}}
}
}
}
}
}
)
```
**Issue:** All devices use the first device's driver!
- Cisco R1 → Uses Huawei driver (if Huawei is first) ❌
- Huawei SW1 → Uses Cisco driver (if Cisco is first) ❌
#### ❌ Approach 2: Dynamic Groups (Intermediate Solution)
```python
# COMPLEX: Create multiple groups dynamically
groups = {}
for host_data in hosts_data.values():
device_type = host_data["device_type"]
platform = host_data["platform"]
group_name = f"{platform}_telnet" # e.g., "huawei_telnet"
if group_name not in groups:
groups[group_name] = {
"platform": platform,
"connection_options": {
"netmiko": {"extras": {"device_type": device_type}}
}
}
host_data["groups"] = [group_name]
```
**Issues:**
- Complex logic to detect and create groups
- Code duplication in multiple files
- Had to delete helper functions (`_get_nornir_groups_config`, `_get_nornir_group`)
- Not following Nornir best practices
#### ✅ Approach 3: Host-Level Configuration (Current - Best Practice)
```python
# SIMPLE: Single group + host-level device_type
hosts_data[device_name] = {
"port": console_port,
"platform": platform, # Reserved for future use
"groups": ["network_devices"], # All devices in one group
"connection_options": { # Device-specific config
"netmiko": {
"extras": {"device_type": device_type}
}
}
}
# Single generic group for shared settings
groups = {
"network_devices": {
"hostname": "127.0.0.1",
"timeout": 120,
"username": "",
"password": "",
}
}
```
**Advantages:**
- ✅ Clean, simple code
- ✅ Follows Nornir best practice (host > group > defaults)
- ✅ No dynamic group creation logic
- ✅ Each device's `connection_options` overrides group settings automatically
- ✅ Easy to extend with new device types
> See `get_gns3_device_port.py:get_device_ports_from_topology()` for the current implementation.
### Configuration Priority Demonstration
```python
# Host level (highest priority)
host["connection_options"]["netmiko"]["extras"]["device_type"] = "gns3_huawei_telnet_ce"
↓ OVERRIDES
# Group level (middle priority)
group["hostname"] = "127.0.0.1"
group["timeout"] = 120
↓ OVERRIDES
# Defaults level (lowest priority)
defaults["data"]["location"] = "gns3"
```mermaid
graph TD
A["Host Level (highest)<br/>device_type: gns3_huawei_telnet_ce"]
B["Group Level (middle)<br/>hostname: 127.0.0.1, timeout: 120"]
C["Defaults Level (lowest)<br/>data.location: gns3"]
A -->|overrides| B -->|overrides| C
D[Result] --> D1["Each device uses its own device_type"]
D --> D2["All share hostname, timeout from group"]
D --> D3["All share data.location from defaults"]
```
**Result:**
- Each device uses its own `device_type` from host level
- All devices share `hostname`, `timeout` from group level
- All devices share `data.location` from defaults level
## Usage Examples
### Direct Netmiko Usage
**Huawei Device (Custom Driver):**
```python
from netmiko import ConnectHandler
from gns3server.agent.gns3_copilot.utils import custom_netmiko
**Custom driver auto-registers on import. No username/password needed for GNS3 devices.**
# Custom driver auto-registers on import
device = {
"device_type": "gns3_huawei_telnet_ce",
"host": "127.0.0.1",
"port": 5000,
# No username/password needed!
}
with ConnectHandler(**device) as conn:
# Execute display command
output = conn.send_command("display version")
# Execute configuration commands
config = [
"interface GE1/0/1",
"description Uplink-to-Core",
"undo shutdown"
]
output = conn.send_config_set(config)
```
**Cisco IOS Device (Standard Driver):**
```python
from netmiko import ConnectHandler
device = {
"device_type": "cisco_ios_telnet",
"host": "127.0.0.1",
"port": 5001,
"username": "cisco",
"password": "cisco",
}
with ConnectHandler(**device) as conn:
output = conn.send_command("show version")
config = ["interface GigabitEthernet0/0", "description Test"]
output = conn.send_config_set(config)
```
### Nornir Multi-Vendor Automation
```python
from nornir import InitNornir
from gns3server.agent.gns3_copilot.utils import custom_netmiko
# Auto-register custom driver (happens automatically on import)
from gns3server.agent.gns3_copilot.utils.custom_netmiko import huawei_ce
huawei_ce.register_custom_device_type()
# Initialize Nornir with mixed-vendor inventory
# Using host-level connection_options (best practice)
inventory = {
"plugin": "DictInventory",
"options": {
"hosts": {
"huawei-sw1": {
"hostname": "127.0.0.1",
"port": 5001,
"platform": "huawei", # Reserved for future use
"groups": ["network_devices"],
"connection_options": {
"netmiko": {
"extras": {"device_type": "gns3_huawei_telnet_ce"}
}
}
},
"cisco-r1": {
"hostname": "127.0.0.1",
"port": 5002,
"platform": "cisco_ios",
"groups": ["network_devices"],
"connection_options": {
"netmiko": {
"extras": {"device_type": "cisco_ios_telnet"}
}
}
}
},
"groups": {
"network_devices": {
"hostname": "127.0.0.1", # Shared by all devices
"timeout": 120,
"username": "",
"password": "",
}
},
"defaults": {
"data": {"location": "gns3"}
}
}
}
nr = InitNornir(inventory=inventory)
# Execute commands on all devices (multi-vendor)
result = nr.run(task=send_commands, commands=["display version"])
# Each device gets vendor-specific command handling
# huawei-sw1 uses gns3_huawei_telnet_ce driver
# cisco-r1 uses cisco_ios_telnet driver
```
| Device Type | `device_type` | Auth | Notes |
|-------------|---------------|------|-------|
| Huawei CE | `gns3_huawei_telnet_ce` | None | Custom driver, skip auth |
| Ruijie | `gns3_ruijie_telnet` | None | Custom driver, interactive prompts handled |
| VPCS | `gns3_vpcs_telnet` | None | Custom driver, no config mode |
| Cisco IOS | `cisco_ios_telnet` | Required | Standard Netmiko driver |
### GNS3 Copilot Tool Usage
```python
from gns3server.agent.gns3_copilot.tools_v2 import DisplayToolNornir
> Tools: `tools_v2/display_tools_nornir.py`, `tools_v2/config_tools_nornir.py`, `tools_v2/vpcs_tools_netmiko.py`
tool = DisplayToolNornir()
result = tool._run(json.dumps({
"device_names": ["huawei-sw1", "cisco-r1"],
"commands": ["display version", "show version"],
"project_id": "project-uuid"
}))
# Returns:
# {
# "huawei-sw1": {
# "display version": "<Huawei output>",
# "status": "success"
# },
# "cisco-r1": {
# "show version": "<Cisco output>",
# "status": "success"
# }
# }
```
Each tool reads node tags to determine `device_type`, builds host-level Nornir inventory, and dispatches commands to the appropriate Netmiko driver. Multi-vendor topologies are handled transparently — each device uses its own driver.
## Module Structure
@@ -876,29 +641,24 @@ result = tool._run(json.dumps({
gns3server/agent/gns3_copilot/
├── utils/
│ ├── custom_netmiko/ # Custom Netmiko drivers package
│ │ ├── __init__.py # Package initialization
│ │ ├── __init__.py # Auto-registers all drivers on import
│ │ ├── huawei_ce.py # Huawei CloudEngine driver
│ │ ├── ruijie_telnet.py # Ruijie enhanced driver
│ │ ├── vpcs_telnet.py # VPCS simulator driver (NEW)
│ │ ├── vpcs_telnet.py # VPCS simulator driver
│ │ ├── README.md # Driver development guide
│ │ └── tests/ # Unit tests
│ │ └── tests/ # Unit tests (52 total)
│ │ ├── __init__.py
│ │ └── test_huawei_ce.py # Huawei CE driver tests
│ └── get_gns3_device_port.py # Device port extraction with host-level config
│ ├── _expand_multiline_commands() # Expand banner commands
│ └── _error_handling() # device_type missing errors
│ │ ├── test_huawei_ce.py # Huawei CE driver tests (9)
│ │ ├── test_ruijie_telnet.py # Ruijie driver tests (15)
│ │ └── test_vpcs_telnet.py # VPCS driver tests (28)
│ └── get_gns3_device_port.py # Device port extraction
│ └── get_device_ports_from_topology() # Extract ports + device_type from tags
├── tools_v2/
│ ├── display_tools_nornir.py # Multi-vendor display commands
│ │ ├── _get_nornir_defaults() # Returns default Nornir config
│ │ └── _initialize_nornir() # Single generic group + host-level device_type
│ ├── config_tools_nornir.py # Multi-vendor config commands
│ │ ├── _get_nornir_defaults() # Returns default Nornir config
│ │ └── _initialize_nornir() # Single generic group + host-level device_type
│ │ ├── _expand_multiline_commands() # Expand banner commands
│ │ └── _error_handling() # device_type validation
│ └── vpcs_tools_netmiko.py # VPCS commands using Nornir + Netmiko (NEW)
│ ├── VPCSCommands # VPCS tool class
│ └── _initialize_nornir() # VPCS device inventory setup
│ │ └── _expand_multiline_commands() # Expand banner commands
│ └── vpcs_tools_netmiko.py # VPCS commands using Nornir + Netmiko
│ └── VPCSCommands # VPCS tool class
```
**Key Architectural Changes (2026-03-14):**
@@ -915,80 +675,51 @@ gns3server/agent/gns3_copilot/
- ✅ Updated: `get_gns3_device_port.py()` - Returns host-level `connection_options`
- ✅ Added: `ruijie_telnet.py` - Custom Ruijie driver with interactive prompt handling
- ✅ Added: `_expand_multiline_commands()` - Auto-expands banner and multi-line commands
- ✅ Added: `_error_handling()` - Validates device_type tags, returns error if missing
- ✅ Added: device_type tag validation with error feedback (inline in `_run`)
## Unit Testing
### Test Coverage
```python
# test_netmiko_custom.py
> Test files located at: `gns3server/agent/gns3_copilot/utils/custom_netmiko/tests/`
class TestGNS3HuaweiTelnetCEDriver(unittest.TestCase):
def test_device_type_registered(self):
"""Verify gns3_huawei_telnet_ce is in Netmiko CLASS_MAPPER"""
from netmiko.ssh_dispatcher import CLASS_MAPPER
self.assertIn("gns3_huawei_telnet_ce", CLASS_MAPPER)
**Total: 52 tests across 3 test files**
def test_inheritance_from_huawei_base(self):
"""Verify inherits from HuaweiBase"""
from netmiko.huawei.huawei import HuaweiBase
self.assertTrue(issubclass(GNS3HuaweiTelnetCE, HuaweiBase))
| Test File | Tests | Coverage Focus |
|-----------|-------|----------------|
| `test_huawei_ce.py` | 9 | Registration, inheritance, VRP methods, mock connection |
| `test_ruijie_telnet.py` | 15 | Registration, interactive patterns, preprocessing, batch/fallback |
| `test_vpcs_telnet.py` | 28 | Registration, login, command sending, ANSI stripping, config mode |
def test_vrp_methods_available(self):
"""Verify VRP-specific methods are available"""
methods = ["config_mode", "check_config_mode", "exit_config_mode"]
for method in methods:
self.assertTrue(hasattr(GNS3HuaweiTelnetCE, method))
```
#### Huawei CE Tests (`TestHuaweiTelnetCEDriver` + `TestHuaweiTelnetCEIntegration`)
- Device type registration in CLASS_MAPPER
- Inheritance from HuaweiBase
- VRP-specific methods (`config_mode`, `check_config_mode`, `exit_config_mode`)
- Prompt pattern constants
- Mock telnet connection
#### Ruijie Tests (`TestRuijieTelnetEnhancedDriver` + `TestRuijieTelnetEnhancedIntegration`)
- Device type registration
- Inheritance from RuijieOSBase
- Interactive pattern matching (`router-id`, `erase`, `delete`, etc.)
- Preprocessing of interactive commands
- Batch and fallback send strategies
#### VPCS Tests (`TestVPCSTelnetDriver` + 5 more test classes)
- Device type registration
- Inheritance from BaseConnection
- No-auth login with `PC\d+>` prompt
- `send_command` / `send_command_timing` overrides
- ANSI escape code stripping (8 dedicated tests)
- Config mode always returns empty/false
**Running Tests:**
```bash
source venv/bin/activate
python gns3server/agent/gns3_copilot/utils/custom_netmiko/tests/test_huawei_ce.py
python -m pytest gns3server/agent/gns3_copilot/utils/custom_netmiko/tests/
```
**Current Test Status:** ✅ All 9 tests passing
## Platform vs Device Type
### Key Concepts
**Platform (Nornir):**
- High-level vendor identifier
- **Reserved for future use** with plugins like NAPALM, scrapli
- Used for metadata and logging
- Examples: `huawei`, `cisco_ios`
- ⚠️ **Not used by nornir_netmiko** (only `device_type` matters)
**Device Type (Netmiko):**
- Precise driver type for Netmiko connection
- Includes protocol information
- **Actively used** to determine which Netmiko driver class to load
- Examples: `gns3_huawei_telnet_ce`, `cisco_ios_telnet`
### Why Keep `platform` Field?
| Purpose | Plugin | Uses `platform`? |
|---------|--------|------------------|
| Connection driver | nornir_netmiko | ❌ No (uses `device_type`) |
| Driver selection | NAPALM | ✅ Yes |
| Driver selection | scrapli | ✅ Yes |
| Metadata/Logging | General | ✅ Yes (future) |
**Conclusion:** The `platform` field is kept for:
1. **Future plugin support** (NAPALM, scrapli)
2. **Debugging and logging** (vendor identification)
3. **Data completeness** (industry standard practice)
### Mapping
| Platform | Device Type | Netmiko Usage | Notes |
|----------|-------------|---------------|-------|
| `huawei` | `gns3_huawei_telnet_ce` | ✅ Active | Custom driver for GNS3 |
| `cisco_ios` | `cisco_ios_telnet` | ✅ Active | Standard Netmiko driver |
**Important:** For nornir_netmiko, only `device_type` in `connection_options` matters. The `platform` field is informational only.
**Current Test Status:** ✅ All 52 tests passing
## Related Documentation
@@ -1007,13 +738,13 @@ python gns3server/agent/gns3_copilot/utils/custom_netmiko/tests/test_huawei_ce.p
_Implementation Date: 2026-03-12_
_Last Updated: 2026-03-14 (Added VPCS driver and unified tool architecture)_
_Last Updated: 2026-04-20 (Documentation review: fixed VPCS template tags, updated test coverage, added Mermaid diagrams, removed code bloat)_
_Status: ✅ Implemented - Custom drivers for Huawei, Ruijie, and VPCS; multi-vendor support with Cisco IOS, Huawei, Ruijie, and VPCS tested_
_Architecture: Nornir best practice - host-level connection_options with single generic group_
_Unit Tests: ✅ 9/9 passing_
_Unit Tests: ✅ 52/52 passing (9 Huawei + 15 Ruijie + 28 VPCS)_
_Changelog:_
- **2026-03-14**: Added VPCS support and unified tool architecture
@@ -1038,5 +769,4 @@ _Changelog:_
- Removed `_get_nornir_groups_config()` and `_get_nornir_group()` helper functions
- Simplified `_initialize_nornir()` to use single generic group
- Updated `get_gns3_device_port.py()` to return host-level configuration
- Reserved `platform` field for future NAPALM/scrapli plugin support
- **2026-03-12**: Initial implementation with custom Huawei driver
docs/gns3-copilot/implemented/node-control-tools.md
+98 -47
View File
@@ -1,3 +1,11 @@
<!--
SPDX-License-Identifier: CC-BY-SA-4.0
See LICENSE file for licensing information.
-->
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
# Node and Topology Management Tools
## Overview
@@ -80,7 +88,8 @@ The following built-in utility templates are excluded as they are not actual net
{
"template_id": "uuid-of-template",
"x": 100,
"y": -200
"y": -200,
"name": "R1"
},
{
"template_id": "uuid-of-template2",
@@ -98,7 +107,7 @@ The following built-in utility templates are excluded as they are not actual net
"created_nodes": [
{
"node_id": "uuid-of-node1",
"name": "NodeName1",
"name": "R1",
"status": "success"
},
{
@@ -117,6 +126,7 @@ The following built-in utility templates are excluded as they are not actual net
- Batch create multiple nodes
- Uses templates for consistent node configuration
- X/Y coordinate positioning for topology layout
- Optional `name` field to set node name directly (if omitted, GNS3 assigns default name)
- **Important**: Ensure distance between any two nodes is greater than 250px for clear interface labels
**Use Cases:**
@@ -307,6 +317,58 @@ The tool automatically detects node types and calculates optimal wait times:
- Logs device types and selected wait strategy
- Progress bar displays calculated wait time
### GNS3StartNodeQuickTool 🆕
**Tool Name:** `start_gns3_node_quick`
**Description:** Starts one or multiple nodes in a GNS3 project WITHOUT waiting for startup completion. Suitable for automated deployment workflows where long waits would cause HTTP timeouts.
**Input:**
```json
{
"project_id": "uuid-of-project",
"node_ids": ["uuid-of-node-1", "uuid-of-node-2"]
}
```
**Output:**
```json
{
"project_id": "...",
"total_nodes": 2,
"successful": 2,
"failed": 0,
"nodes": [
{"node_id": "...", "name": "...", "status": "started"},
{"node_id": "...", "name": "...", "status": "started"}
],
"note": "Start commands sent. Nodes are booting in background. Check node status later."
}
```
**Features:**
- Batch start multiple nodes
- No progress bar or wait time
- Immediate API response
- Nodes boot in background after tool returns
- Comprehensive error handling
**When to Use Quick vs Regular Start:**
| Scenario | Use `start_gns3_node` | Use `start_gns3_node_quick` |
|----------|----------------------|-----------------------------|
| Interactive lab startup | ✅ | ❌ |
| CI/CD pipeline | ❌ | ✅ |
| HTTP timeout risk | ❌ | ✅ |
| Need verified status | ✅ | ❌ |
| Automated bulk deployment | ❌ | ✅ |
**Implementation Details:**
- Calls `POST /projects/{project_id}/nodes/{node_id}/start` for each node
- Returns immediately after sending start commands (no progress bar)
- Status reflects command send result, not boot completion
- Source: `gns3_start_node.py` (same file as `GNS3StartNodeTool`)
### GNS3StopNodeTool
**Tool Name:** `stop_gns3_node`
@@ -444,9 +506,12 @@ gns3server/agent/gns3_copilot/tools_v2/
├── gns3_create_link.py # Link creation tool 🆕
├── gns3_get_node_temp.py # Template retrieval tool 🆕
├── gns3_update_node_name.py # Node rename tool 🆕
├── gns3_start_node.py # Start node tool
├── gns3_start_node.py # Start node tool (+ GNS3StartNodeQuickTool)
├── gns3_stop_node.py # Stop node tool
└── gns3_suspend_node.py # Suspend node tool
├── gns3_suspend_node.py # Suspend node tool
├── config_tools_nornir.py # Configuration command execution
├── display_tools_nornir.py # Display command execution
└── packet_capture_tools.py # Packet capture analysis
```
### API Integration
@@ -478,44 +543,40 @@ node.start() # or node.stop() / node.suspend()
### Progress Tracking
**GNS3StartNodeTool** includes visual progress bar:
**GNS3StartNodeTool** includes visual progress bar with dynamic wait time:
```
Starting 3 node(s), please wait...
[===========> ] 35.0%
```
**Progress Calculation:**
- Base duration: 140 seconds
- Extra duration: 10 seconds per additional node
- Formula: `total_duration = 140 + max(0, node_count - 1) * 10`
Wait time is calculated by `calculate_startup_time()` based on node types (see Dynamic Wait Time Strategy above). No hardcoded duration.
**GNS3StartNodeQuickTool** does NOT include progress tracking:
- Returns immediately after sending start commands
- Nodes boot in background
**GNS3StopNodeTool** does not include progress tracking:
- Stop operations are typically fast (< 5 seconds)
- Immediate API response provides status feedback
- No need for progress indication
**GNS3SuspendNodeTool** does not include progress tracking:
- Suspend operations are typically fast (< 10 seconds)
- Immediate API response provides status feedback
- No need for progress indication
## Node State Transitions
```mermaid
stateDiagram-v2
[*] --> Stopped
Stopped --> Started : start()
Started --> Stopped : stop()
Started --> Suspended : suspend()
Suspended --> Started : start() [resume]
Suspended --> Stopped : stop()
```
┌─────────┐
│ Stopped │◀─────── stop()
└────┬────┘
│
│ start()
▼
┌──────────┐
│ Started │───suspend()───▶ Suspended
└──────────┘ │
▲ │ │
│ │ stop() │ resume()
└──────┴──────────────────────────┘
```
> **Note:** Resuming a suspended node uses `start()` — there is no separate `resume()` method.
**State Change Allowed Operations:**
@@ -538,11 +599,15 @@ Note: These special node types are filtered out by GNS3TemplateTool and won't ap
- `GNS3LinkTool` - Create links between nodes 🆕
- `GNS3UpdateNodeNameTool` - Rename nodes 🆕
- `GNS3StartNodeTool` - For diagnostics requiring started nodes
- `ExecuteMultipleDeviceCommands` - Execute show/display/debug commands (READ-ONLY)
- `PacketCaptureTool` - Analyze packets from active capture
- `DeviceSkillsTool` - Get device-specific skills and command knowledge
**Capabilities:**
- READ-ONLY diagnostic tools
- Can create and manage topology (nodes, links, names)
- Cannot stop or suspend nodes (prevents disruption of active labs)
- Cannot execute configuration commands
### Lab Automation Assistant Mode
@@ -554,10 +619,16 @@ Note: These special node types are filtered out by GNS3TemplateTool and won't ap
- `GNS3StartNodeTool` - Full lab deployment
- `GNS3StopNodeTool` - Full lab shutdown
- `GNS3SuspendNodeTool` - Lab pause with state preservation
- `ExecuteMultipleDeviceCommands` - Execute show/display/debug commands (READ-ONLY)
- `ExecuteMultipleDeviceConfigCommands` - Execute configuration commands
- `VPCSCommands` - Execute VPCS commands using Netmiko
- `PacketCaptureTool` - Analyze packets from active capture
- `DeviceSkillsTool` - Get device-specific skills and command knowledge
**Capabilities:**
- Full diagnostic and configuration tools
- Complete topology and lifecycle management (create, connect, start/stop/suspend)
- Device configuration via Nornir/Netmiko
- Automated workflows with state preservation
- Lab snapshot capabilities for later resumption
@@ -898,7 +969,9 @@ logger.info("Suspend command sent for node %s (%s)", node_id, node.name)
| Create Node | < 1s per node | 0s | No | N/A |
| Create Link | < 1s per link | 0s | No | N/A |
| Update Name | < 1s per node | 0s | No | N/A |
| Start | 60-180s | ~140s base | Yes | N/A |
| Start (VPCS/IOU) | 15-37s | Dynamic | Yes | N/A |
| Start (QEMU/etc.) | 120-160s+ | Dynamic | Yes | N/A |
| Start Quick | < 2s | 0s | No | N/A |
| Stop | < 5s | 0s | No | ❌ No |
| Suspend | < 10s | 0s | No | ✅ Yes |
@@ -911,28 +984,6 @@ logger.info("Suspend command sent for node %s (%s)", node_id, node.name)
- Create node/link operations are fast and require no waiting
- Template retrieval is instant with no parameters needed
## Future Enhancements
### Planned Features
- [ ] **Quick Start Tool**: Start nodes without waiting for completion (for CI/CD)
- [ ] **Delete Node Tool**: Remove nodes from topology
- [ ] **Delete Link Tool**: Remove links from topology
- [ ] **Resume Tool**: Explicit resume operation for suspended nodes
- [ ] **Restart Tool**: Combined stop + start operation
- [ ] **Bulk Status Check**: Query multiple nodes without stopping
- [ ] **Conditional Stop/Suspend**: Operate only if node is in specific state
- [ ] **Graceful Shutdown**: Send halt commands before stopping
### Potential Improvements
- [ ] Auto-layout calculation (optimal node positioning)
- [ ] Progress tracking for long suspend operations (rare but possible)
- [ ] Concurrent create/link operations (parallel API calls)
- [ ] Suspend node groups by name pattern
- [ ] Dependency-aware suspend (suspend in dependency order)
- [ ] Auto-suspend after idle timeout
- [ ] State snapshots (save multiple suspend states)
## Related Documentation
docs/gns3-copilot/netmiko_devices.md
+8
View File
@@ -1,3 +1,11 @@
<!--
SPDX-License-Identifier: CC-BY-SA-4.0
See LICENSE file for licensing information.
-->
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
# Netmiko Supported Devices
**Netmiko Version:** 4.6.0
docs/gns3-copilot/template-based-configuration-roadmap.md
+8
View File
@@ -1,3 +1,11 @@
<!--
SPDX-License-Identifier: CC-BY-SA-4.0
See LICENSE file for licensing information.
-->
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
# Template-Based System with HITL - Future Roadmap
**Status:** 💡 Proposed