mirror of
https://github.com/GNS3/gns3-server.git
synced 2026-08-27 12:30:13 +03:00
Merge pull request #2738 from yueguobin/docs/skills-editor-api
Documentation improvements and Skills Editor API roadmap
This commit is contained in:
Jeremy Grossmann
GitHub
commit
ca2004e715
182
docs/gns3-copilot/implemented/packet-analysis.md
Normal file
182
docs/gns3-copilot/implemented/packet-analysis.md
Normal file
@ -0,0 +1,182 @@
|
||||
<!--
|
||||
SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
See LICENSE file for licensing information.
|
||||
-->
|
||||
|
||||
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
|
||||
|
||||
|
||||
# Protocol-Oriented Packet Analysis
|
||||
|
||||
## Overview
|
||||
|
||||
GNS3 Copilot provides protocol-oriented packet analysis that allows the AI assistant to diagnose network issues from live GNS3 captures. Protocol definitions (tshark fields, display filters, check rules) are stored as YAML files in the external GNS3-Skills repository and loaded into memory at startup.
|
||||
|
||||
The system exposes two LangChain tools: `PacketAnalysisSkillsTool` queries protocol definitions, and `PacketAnalysisTool` runs tshark against downloaded pcap files using the LLM-constructed arguments.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "GNS3-Skills Repository"
|
||||
YAML[packet_analysis/*.yaml<br/>40+ protocol files]
|
||||
end
|
||||
|
||||
subgraph "GNS3 Server"
|
||||
SM[SkillsManager]
|
||||
SL[SkillsLoader]
|
||||
REG[PACKET_ANALYSIS_REGISTRY<br/>in-memory dict]
|
||||
SKILL[PacketAnalysisSkillsTool<br/>query protocol definitions]
|
||||
TOOL[PacketAnalysisTool<br/>run tshark on captures]
|
||||
end
|
||||
|
||||
subgraph "tshark"
|
||||
FIELDS["tshark -G fields"<br/>live field registry]
|
||||
CAP["tshark -r pcap"<br/>packet capture analysis]
|
||||
end
|
||||
|
||||
YAML -->|load at startup / reload| SL
|
||||
SL --> REG
|
||||
SM --> SL
|
||||
|
||||
SKILL -->|read protocol fields/filters| REG
|
||||
TOOL -->|validate -e fields| FIELDS
|
||||
TOOL -->|download pcap + run tshark| CAP
|
||||
```
|
||||
|
||||
## Supported Protocols
|
||||
|
||||
Definitions are loaded from `GNS3-Skills/packet_analysis/*.yaml`. The following protocol families are covered:
|
||||
|
||||
| Category | Protocols |
|
||||
|----------|-----------|
|
||||
| Network Layer | ip, ipv6, arp, icmp, icmpv6 |
|
||||
| Transport Layer | tcp, udp |
|
||||
| Data Link | ethernet, ppp, hdlc, frame_relay, vlan, isl, llc |
|
||||
| Routing | ospf, eigrp, rip, bgp, isis, pim, dvmrp |
|
||||
| Link Protocols | l2tp, lacp, pagp, udld |
|
||||
| Infrastructure | cdp, lldp, stp, vtp, dtp |
|
||||
| Management | snmp, telnet, ssh, radius, tacacs |
|
||||
| Application | dns, http, bootp, dhcp |
|
||||
| Tunneling / Security | gre, esp, ah, mpls, eapol, ssl, isakmp |
|
||||
| Miscellaneous | nbns, slarp, ocsp, wccp, auto_rp, loop |
|
||||
|
||||
Each protocol YAML contains:
|
||||
|
||||
```yaml
|
||||
name: "OSPF Packet Analysis"
|
||||
description: "Analyze OSPF routing protocol packets"
|
||||
protocol_key: "ospf"
|
||||
display_filter: "ospf"
|
||||
fields:
|
||||
- label: "Source IP"
|
||||
tshark_field: "ip.src"
|
||||
description: "Source IPv4 address"
|
||||
- label: "OSPF Message Type"
|
||||
tshark_field: "ospf.msg"
|
||||
description: "1=Hello, 2=DBD, 3=LSR, 4=LSU, 5=LSAck"
|
||||
filter_examples:
|
||||
- description: "Show OSPF Hello packets"
|
||||
filter: "ospf.msg == 1"
|
||||
checks:
|
||||
- name: hello_dead_mismatch
|
||||
severity: critical
|
||||
message: "Hello/Dead Interval mismatch between neighbors"
|
||||
```
|
||||
|
||||
## Analysis Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant LLM as LLM Agent
|
||||
participant SKILL as PacketAnalysisSkillsTool
|
||||
participant TOOL as PacketAnalysisTool
|
||||
participant tshark as tshark
|
||||
participant GNS3 as GNS3 Server
|
||||
|
||||
LLM->>SKILL: {"action": "list"}
|
||||
SKILL-->>LLM: Available protocols
|
||||
|
||||
LLM->>SKILL: {"action": "get", "protocol": "ospf"}
|
||||
SKILL-->>LLM: Fields, filters, check rules
|
||||
|
||||
Note over LLM: LLM constructs tshark_args<br/>using protocol knowledge
|
||||
|
||||
LLM->>TOOL: {project_id, link_id,<br/>tshark_args: "-Y ospf -T fields -e ip.src -e ospf.msg"}
|
||||
TOOL->>TOOL: Validate -e field names<br/>against tshark -G fields
|
||||
TOOL->>GNS3: Download pcap for link_id
|
||||
GNS3-->>TOOL: Capture file
|
||||
TOOL->>tshark: tshark -r pcap -Y ospf -T fields -e ip.src -e ospf.msg
|
||||
tshark-->>TOOL: Tab-separated output
|
||||
TOOL-->>LLM: Raw tshark output
|
||||
```
|
||||
|
||||
## Tool Registration
|
||||
|
||||
The packet analysis tools are available in two copilot modes:
|
||||
|
||||
| Tool | Teaching Assistant | Lab Automation | Troubleshooting Injection |
|
||||
|------|:------------------:|:--------------:|:-------------------------:|
|
||||
| `PacketAnalysisTool` | Yes | Yes | No |
|
||||
| `PacketAnalysisSkillsTool` | Yes | Yes | No |
|
||||
|
||||
## Tool Interface
|
||||
|
||||
### PacketAnalysisSkillsTool (`packet_analysis_skills`)
|
||||
|
||||
Queries protocol definitions from the `PACKET_ANALYSIS_REGISTRY`. Used before running tshark to look up valid field names, display filters, and anomaly checks.
|
||||
|
||||
| Action | Input | Output |
|
||||
|--------|-------|--------|
|
||||
| List protocols | `{"action": "list"}` | `{count, protocols: [{protocol, name, description}]}` |
|
||||
| Get protocol | `{"action": "get", "protocol": "ospf"}` | Protocol definition with fields, filters, checks |
|
||||
|
||||
### PacketAnalysisTool (`packet_analysis`)
|
||||
|
||||
Runs tshark against a downloaded GNS3 capture file. Supports two modes:
|
||||
|
||||
**Capture analysis mode:**
|
||||
|
||||
| Parameter | Required | Description |
|
||||
|-----------|----------|-------------|
|
||||
| `project_id` | Yes | UUID of the GNS3 project |
|
||||
| `link_id` | Yes | UUID of the link to analyze |
|
||||
| `tshark_args` | Yes | tshark arguments (after `-r <pcap>`) |
|
||||
|
||||
**Field search mode:**
|
||||
|
||||
| Parameter | Required | Description |
|
||||
|-----------|----------|-------------|
|
||||
| `action` | Yes | `"search_fields"` |
|
||||
| `query` | Yes | Single keyword (e.g., `"ospf.lsa"`, `"bgp.open"`) |
|
||||
|
||||
### Validation and Error Handling
|
||||
|
||||
Before downloading the capture, the tool validates `-e` field names against the live tshark field registry (`tshark -G fields`). Invalid field names are rejected early with a hint to use `search_fields`.
|
||||
|
||||
| Scenario | Response |
|
||||
|----------|----------|
|
||||
| Invalid `-e` field name | `{error, hint, invalid_fields}` |
|
||||
| tshark filter/field error | `{error: "tshark argument error", hints}` |
|
||||
| Empty capture file | `{error: "Capture file is empty"}` |
|
||||
| No matching packets | `{result: "No matching packets found", hints}` |
|
||||
| tshark timeout (30s) | `{error: "tshark timeout after 30 seconds"}` |
|
||||
| tshark not installed | `{error: "tshark not installed"}` |
|
||||
|
||||
When `-c` is used and no results are found, the tool hints that `-c` limits total packets read (not matched count), and suggests removing it or piping to `head`.
|
||||
|
||||
## Hot Reload
|
||||
|
||||
Packet analysis protocols can be reloaded without restarting the server via the existing reload API:
|
||||
|
||||
```
|
||||
POST /v3/copilot/reload/skills
|
||||
```
|
||||
|
||||
This triggers `SkillsManager.reload_packet_analysis_protocols()`, which re-reads all YAML files from the `packet_analysis/` directory and updates `PACKET_ANALYSIS_REGISTRY` in place.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [External Skills Repository](skills-repository.md)
|
||||
- [Fault Injection](fault-injection.md)
|
||||
- [Chat API](chat-api.md)
|
||||
@ -1,92 +0,0 @@
|
||||
<!--
|
||||
SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
See LICENSE file for licensing information.
|
||||
-->
|
||||
|
||||
> This document is a roadmap/planning document. The described features have not been implemented yet.
|
||||
|
||||
|
||||
# Protocol-Oriented Packet Analysis — Roadmap
|
||||
|
||||
## Problem
|
||||
|
||||
The current `PacketCaptureTool` (`analyze_packets`) only accepts a single `packet_number` parameter and runs `tshark -V` on that one frame. This approach:
|
||||
|
||||
- Forces the LLM to guess packet numbers without any visibility into the capture
|
||||
- Returns raw verbose output instead of structured data
|
||||
- Has no protocol awareness — every protocol looks the same to the tool
|
||||
- Provides no built-in anomaly detection; the LLM must infer issues from raw output each time
|
||||
- Downloads the same pcap from the server on every call (no caching)
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
Move from number-based to **protocol-oriented** packet analysis. Protocol definitions (fields, display filters, anomaly checks) are stored as YAML files in the GNS3-Skills repository. The tool only needs `link_id` + `protocol` — no complex parameters for the LLM to get wrong.
|
||||
|
||||
### Data Flow
|
||||
|
||||
```
|
||||
GNS3-Skills/packet_analysis/<protocol>.yaml
|
||||
│
|
||||
▼ loaded at startup / reload
|
||||
PACKET_ANALYSIS_REGISTRY (in-memory dict)
|
||||
│
|
||||
▼
|
||||
PacketAnalysisTool(link_id, protocol)
|
||||
│
|
||||
├── download pcap (with link_id caching)
|
||||
├── tshark -T fields -e <predefined fields>
|
||||
├── run anomaly checks from YAML
|
||||
└── return structured JSON + check results
|
||||
│
|
||||
▼
|
||||
LLM produces natural language explanation
|
||||
```
|
||||
|
||||
### YAML Format
|
||||
|
||||
```yaml
|
||||
name: "OSPF Packet Analysis"
|
||||
protocol: "ospf"
|
||||
display_filter: "ospf"
|
||||
fields:
|
||||
- label: "Source IP"
|
||||
field: "ip.src"
|
||||
description: "Source IPv4 address"
|
||||
- label: "OSPF Message Type"
|
||||
field: "ospf.msg"
|
||||
description: "1=Hello, 2=DBD, 3=LSR, 4=LSU, 5=LSAck"
|
||||
checks:
|
||||
- name: hello_dead_mismatch
|
||||
severity: critical
|
||||
message: "Hello/Dead Interval mismatch between {src} and {dst}"
|
||||
condition: "Same broadcast domain has inconsistent hello/dead intervals"
|
||||
```
|
||||
|
||||
### Planned Protocols
|
||||
|
||||
| File | Protocols | Key Checks |
|
||||
|------|-----------|------------|
|
||||
| `arp.yaml` | ARP, NDP (ICMPv6 NS/NA) | Duplicate IP, no ARP reply, ARP flooding |
|
||||
| `icmp.yaml` | ICMPv4, ICMPv6 | Unreachable classification, ping loss, PMTUD issues |
|
||||
| `ospf.yaml` | OSPFv2, OSPFv3 | Hello/Dead mismatch, Area ID mismatch, Router ID conflict |
|
||||
| `bgp.yaml` | BGPv4, BGP+ | Hold timer mismatch, Notification analysis, AS_PATH loop |
|
||||
|
||||
### Tool Interface
|
||||
|
||||
```json
|
||||
{
|
||||
"link_id": "uuid (required)",
|
||||
"protocol": "arp | icmp | ospf | bgp (required)",
|
||||
"summary_only": "bool (optional, default: false)"
|
||||
}
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
- [ ] GNS3-Skills: create `packet_analysis/` directory and YAML definitions
|
||||
- [ ] gns3-server: add `PACKET_ANALYSIS_REGISTRY` loading from skills repo
|
||||
- [ ] gns3-server: implement `PacketAnalysisTool` with tshark field extraction
|
||||
- [ ] gns3-server: implement protocol-specific anomaly checks
|
||||
- [ ] gns3-server: add pcap caching by `link_id`
|
||||
- [ ] gns3-server: register tool in teaching assistant and lab automation modes
|
||||
- [ ] gns3-server: deprecate and remove old `PacketCaptureTool`
|
||||
318
docs/gns3-copilot/roadmap/skills-editor-api-roadmap.md
Normal file
318
docs/gns3-copilot/roadmap/skills-editor-api-roadmap.md
Normal file
@ -0,0 +1,318 @@
|
||||
<!--
|
||||
SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
See LICENSE file for licensing information.
|
||||
-->
|
||||
|
||||
> This documentation is organized by AI with reference to actual code. AI can make mistakes — please verify against the source code when in doubt.
|
||||
|
||||
|
||||
# Skills Editor API
|
||||
|
||||
## Overview
|
||||
|
||||
A management API that allows the Web UI to browse, edit, save, and contribute skill files (prompts, fault injection, device skills, packet analysis protocols) back to the upstream GNS3-Skills repository via Pull Requests.
|
||||
|
||||
Today, skills are read-only from the server's perspective — the only management endpoint is `POST /copilot/reload/skills` for hot-reloading. This API adds full CRUD operations on the local skills repository plus Git commit/push/PR workflows.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "Web UI"
|
||||
UI[Skills Editor Page]
|
||||
end
|
||||
|
||||
subgraph "GNS3 Server API"
|
||||
API[skills_editor.py<br/>/copilot/skills/*]
|
||||
FM[SkillsFileManager]
|
||||
SM[SkillsManager]
|
||||
end
|
||||
|
||||
subgraph "Local Git Repo"
|
||||
INJ[injection/*.yaml]
|
||||
DEV[device/*.yaml]
|
||||
PA[packet_analysis/*.yaml]
|
||||
PRM[prompts/*.md]
|
||||
CFG[config/*.txt]
|
||||
end
|
||||
|
||||
subgraph "Remote"
|
||||
GH[GitHub API<br/>Pull Requests]
|
||||
REPO[yueguobin/GNS3-Skills]
|
||||
end
|
||||
|
||||
UI -->|CRUD + PR| API
|
||||
API --> FM
|
||||
FM -->|file read/write| INJ
|
||||
FM -->|file read/write| DEV
|
||||
FM -->|file read/write| PA
|
||||
FM -->|file read/write| PRM
|
||||
FM -->|file read/write| CFG
|
||||
FM -->|git commit/push| REPO
|
||||
FM -->|create PR| GH
|
||||
SM -->|hot reload| INJ
|
||||
SM -->|hot reload| DEV
|
||||
SM -->|hot reload| PA
|
||||
SM -->|hot reload| PRM
|
||||
```
|
||||
|
||||
## Business Process
|
||||
|
||||
### Edit and Contribute Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant UI as Web UI
|
||||
participant API as Skills Editor API
|
||||
participant FM as SkillsFileManager
|
||||
participant Git as Local Git Repo
|
||||
participant GH as GitHub
|
||||
|
||||
UI->>API: GET /copilot/skills
|
||||
API-->>UI: List categories + file counts
|
||||
|
||||
UI->>API: GET /copilot/skills/injection
|
||||
API-->>UI: List YAML files
|
||||
|
||||
UI->>API: GET /copilot/skills/injection/ospf_issues
|
||||
API->>FM: read_file("injection", "ospf_issues")
|
||||
FM-->>API: YAML content
|
||||
API-->>UI: File content
|
||||
|
||||
UI->>API: PUT /copilot/skills/injection/ospf_issues<br/>{content: "..."}
|
||||
API->>FM: write_file("injection", "ospf_issues", content)
|
||||
FM->>Git: Write file to disk
|
||||
FM-->>API: Success
|
||||
API-->>UI: Updated
|
||||
|
||||
UI->>API: POST /copilot/skills/commit<br/>{message: "fix: update OSPF fault"}
|
||||
API->>FM: commit_changes(message, files)
|
||||
FM->>Git: git add + git commit
|
||||
FM-->>API: Commit hash
|
||||
API-->>UI: Committed
|
||||
|
||||
UI->>API: POST /copilot/skills/pull-request<br/>{title, body, branch}
|
||||
API->>FM: create_pull_request(...)
|
||||
FM->>Git: git push origin <branch>
|
||||
FM->>GH: POST /repos/{owner}/{repo}/pulls
|
||||
GH-->>FM: PR URL
|
||||
FM-->>API: PR created
|
||||
API-->>UI: PR URL
|
||||
|
||||
UI->>API: POST /copilot/skills/reload
|
||||
API->>FM: Hot reload all registries
|
||||
API-->>UI: Reloaded
|
||||
```
|
||||
|
||||
## Valid Categories
|
||||
|
||||
| Category | Directory | File Extension | Content |
|
||||
|----------|-----------|---------------|---------|
|
||||
| `prompts` | `prompts/` | `.md` | System prompts (teaching_assistant, lab_automation_assistant, etc.) |
|
||||
| `injection` | `injection/` | `.yaml` | Fault injection skill definitions (OSPF, BGP, VLAN, etc.) |
|
||||
| `device` | `device/` | `.yaml` | Device-specific command knowledge |
|
||||
| `packet_analysis` | `packet_analysis/` | `.yaml` | Protocol definitions for tshark-based analysis |
|
||||
| `config` | `config/` | `.txt` | Security and configuration files (forbidden_commands, etc.) |
|
||||
|
||||
## API Endpoints
|
||||
|
||||
All endpoints require **superadmin** authentication. Prefix: `/v3/copilot/skills`.
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/copilot/skills` | List all categories with file counts |
|
||||
| GET | `/copilot/skills/{category}` | List files in a category |
|
||||
| GET | `/copilot/skills/{category}/{filename}` | Read file content (without extension) |
|
||||
| POST | `/copilot/skills/{category}` | Create a new skill file |
|
||||
| PUT | `/copilot/skills/{category}/{filename}` | Update existing file content |
|
||||
| DELETE | `/copilot/skills/{category}/{filename}` | Delete a skill file |
|
||||
| GET | `/copilot/skills/status` | Git status (modified/untracked/deleted files) + repo info |
|
||||
| POST | `/copilot/skills/commit` | Stage and commit changes |
|
||||
| POST | `/copilot/skills/push` | Push a branch to remote |
|
||||
| POST | `/copilot/skills/pull-request` | Create a Pull Request (direct or via fork) |
|
||||
| POST | `/copilot/skills/reload` | Hot reload all skills into memory (replaces `/reload/skills`) |
|
||||
| POST | `/copilot/skills/rollback/{commit_hash}` | Rollback repository to a specific commit |
|
||||
|
||||
### Response Examples
|
||||
|
||||
**GET /copilot/skills** — List categories
|
||||
|
||||
```json
|
||||
{
|
||||
"categories": [
|
||||
{"category": "injection", "file_count": 39, "path": "injection/"},
|
||||
{"category": "device", "file_count": 2, "path": "device/"},
|
||||
{"category": "packet_analysis", "file_count": 8, "path": "packet_analysis/"},
|
||||
{"category": "prompts", "file_count": 4, "path": "prompts/"},
|
||||
{"category": "config", "file_count": 1, "path": "config/"}
|
||||
],
|
||||
"repository": {
|
||||
"repo_url": "https://github.com/yueguobin/GNS3-Skills.git",
|
||||
"branch": "main",
|
||||
"current_version": "abc123def456...",
|
||||
"is_dirty": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**GET /copilot/skills/injection** — List files in category
|
||||
|
||||
```json
|
||||
{
|
||||
"category": "injection",
|
||||
"files": [
|
||||
{"filename": "ospf_issues", "extension": ".yaml", "size": 4521, "last_modified": "2026-05-10T08:30:00Z"},
|
||||
{"filename": "bgp_issues", "extension": ".yaml", "size": 3820, "last_modified": "2026-05-09T14:00:00Z"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**GET /copilot/skills/injection/ospf_issues** — Read file content
|
||||
|
||||
```json
|
||||
{
|
||||
"category": "injection",
|
||||
"filename": "ospf_issues",
|
||||
"extension": ".yaml",
|
||||
"content": "name: OSPF Fault Injection\n...\n",
|
||||
"size": 4521,
|
||||
"last_modified": "2026-05-10T08:30:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
**PUT /copilot/skills/injection/ospf_issues** — Update file
|
||||
|
||||
```json
|
||||
{
|
||||
"content": "name: OSPF Fault Injection\n..."
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"category": "injection",
|
||||
"filename": "ospf_issues",
|
||||
"size": 4600,
|
||||
"last_modified": "2026-05-12T10:00:00Z",
|
||||
"status": "modified"
|
||||
}
|
||||
```
|
||||
|
||||
**POST /copilot/skills/commit** — Commit changes
|
||||
|
||||
Request:
|
||||
|
||||
```json
|
||||
{
|
||||
"message": "fix: update OSPF hello/dead interval fault descriptions",
|
||||
"files": ["injection/ospf_issues.yaml"]
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"commit_hash": "def456abc789...",
|
||||
"message": "fix: update OSPF hello/dead interval fault descriptions",
|
||||
"files_committed": 1
|
||||
}
|
||||
```
|
||||
|
||||
**POST /copilot/skills/pull-request** — Create PR
|
||||
|
||||
Request:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "Fix OSPF fault injection descriptions",
|
||||
"body": "Updated OSPF hello/dead interval fault descriptions for clarity.",
|
||||
"branch": "fix/ospf-descriptions",
|
||||
"target_branch": "main",
|
||||
"fork_url": "https://github.com/user/GNS3-Skills.git"
|
||||
}
|
||||
```
|
||||
|
||||
Response:
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"pr_url": "https://github.com/yueguobin/GNS3-Skills/pull/42",
|
||||
"pr_number": 42,
|
||||
"branch": "fix/ospf-descriptions"
|
||||
}
|
||||
```
|
||||
|
||||
**GET /copilot/skills/status** — Git status
|
||||
|
||||
```json
|
||||
{
|
||||
"current_version": "abc123def456...",
|
||||
"branch": "main",
|
||||
"is_dirty": true,
|
||||
"modified": ["injection/ospf_issues.yaml"],
|
||||
"untracked": [],
|
||||
"deleted": [],
|
||||
"staged": [],
|
||||
"available_versions": [
|
||||
{"hash": "abc123def456", "message": "Add MPLS fault scenarios", "author": "dev", "date": "2026-05-10T08:00:00Z"}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### New Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `gns3server/agent/gns3_copilot/skills/file_manager.py` | `SkillsFileManager` class — file CRUD, git commit/push, GitHub PR API |
|
||||
| `gns3server/api/routes/controller/skills_editor.py` | FastAPI router with all endpoints above |
|
||||
|
||||
### Modified Files
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `gns3server/agent/gns3_copilot/skills/manager.py` | Add `get_file_manager()` method returning a `SkillsFileManager` |
|
||||
| `gns3server/api/routes/controller/__init__.py` | Register `skills_editor` router under `/copilot/skills` prefix |
|
||||
| `gns3server/api/routes/controller/copilot.py` | Deprecate `/reload/skills` in favor of `/skills/reload` |
|
||||
|
||||
### SkillsFileManager Key Methods
|
||||
|
||||
| Method | Returns |
|
||||
|--------|---------|
|
||||
| `list_categories()` | `list[{category, file_count, path}]` |
|
||||
| `list_files(category)` | `list[{filename, size, last_modified}]` |
|
||||
| `read_file(category, filename)` | File content as string |
|
||||
| `write_file(category, filename, content)` | Write (create or update) |
|
||||
| `delete_file(category, filename)` | Delete file |
|
||||
| `get_git_status()` | Modified/untracked/deleted file lists |
|
||||
| `commit_changes(message, files)` | git add + commit |
|
||||
| `push_to_remote(branch)` | git push |
|
||||
| `create_pull_request(title, body, branch, target, fork_url)` | Push + GitHub PR API |
|
||||
|
||||
## Security
|
||||
|
||||
- **Authentication**: All endpoints require superadmin (`current_user.is_superadmin` check)
|
||||
- **Path traversal prevention**: Category validated against whitelist; filename sanitized (no `/`, `..`, or absolute paths)
|
||||
- **File extension enforcement**: `.yaml` for injection/device/packet_analysis, `.md` for prompts, `.txt` for config
|
||||
- **File size limit**: Reject files > 1MB
|
||||
- **YAML validation**: Validate with `yaml.safe_load` before saving
|
||||
- **GitHub token**: Required for PR creation, stored in GNS3 server config
|
||||
|
||||
### PR Creation Modes
|
||||
|
||||
1. **Direct push**: Push to a new branch on the main repo, create PR via GitHub API
|
||||
2. **Fork**: Push to user's fork, create PR against upstream
|
||||
|
||||
The `fork_url` parameter selects the mode. When omitted, the API pushes to the same repository and creates a PR directly.
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [External Skills Repository](../implemented/skills-repository.md)
|
||||
- [Fault Injection](../implemented/fault-injection.md)
|
||||
- [Chat API](../implemented/chat-api.md)
|
||||
@ -0,0 +1,372 @@
|
||||
<!--
|
||||
SPDX-License-Identifier: CC-BY-SA-4.0
|
||||
See LICENSE file for licensing information.
|
||||
-->
|
||||
|
||||
> This document is a roadmap/planning document. The described features have not been implemented yet.
|
||||
|
||||
|
||||
# Web Wireshark Docker Image Size Optimization — Roadmap
|
||||
|
||||
## Problem
|
||||
|
||||
The `gns3/web-wireshark` Docker image currently occupies **~2GB** of disk space, which impacts:
|
||||
- Initial pull/download time for users
|
||||
- Storage requirements on Docker Hub
|
||||
- Deployment flexibility in resource-constrained environments
|
||||
|
||||
### Current Size Breakdown
|
||||
|
||||
Based on `docker history gns3/web-wireshark:latest`:
|
||||
|
||||
| Component | Size | Percentage |
|
||||
|-----------|------|------------|
|
||||
| `debian:trixie` base image | 120 MB | 6% |
|
||||
| xpra + dependencies | 65.1 MB | 3.3% |
|
||||
| Wireshark + GUI stack (Qt, GTK, X11) | 1.82 GB | 91% |
|
||||
| **Total** | **~2 GB** | **100%** |
|
||||
|
||||
### Detailed File System Analysis
|
||||
|
||||
Analysis of container file system reveals significant cleanup opportunities:
|
||||
|
||||
| Directory | Size | Cleanup Potential |
|
||||
|-----------|------|-------------------|
|
||||
| `/usr/lib` | 1.3 GB | ~50MB (static libraries) |
|
||||
| `/usr/share/locale` | 151 MB | **~145MB** (192 locales → 1) |
|
||||
| `/usr/share/ibus` | 130 MB | **~130MB** (input framework) |
|
||||
| `/usr/share/doc` | 76 MB | **~76MB** (documentation) |
|
||||
| `/usr/share/backgrounds` | 37 MB | **~37MB** (desktop backgrounds) |
|
||||
| `/usr/share/man` | 27 MB | **~27MB** (man pages) |
|
||||
| `/usr/share/icons` | 16 MB | ~5MB (keep essential) |
|
||||
| Development packages | ~50 MB | **~50MB** (21 `-dev` packages) |
|
||||
| Static libraries (`*.a`, `*.la`) | 16 MB | **~16MB** |
|
||||
| Sounds/help/perl | ~20 MB | **~20MB** |
|
||||
| **Total Cleanable** | **~570 MB** | |
|
||||
|
||||
**Key Findings**:
|
||||
- 192 locale languages installed (only need en_US)
|
||||
- ibus input framework installed (not needed in headless container)
|
||||
- 21 development packages with headers/headers
|
||||
- 161 static library files (`.a`, `.la`)
|
||||
- Complete documentation and man pages
|
||||
- Desktop environment components (backgrounds, sounds)
|
||||
|
||||
## Proposed Optimizations
|
||||
|
||||
### Phase 1: Safe Optimizations (Estimated: -300~400MB)
|
||||
|
||||
Low-risk changes that maintain full compatibility.
|
||||
|
||||
#### 1. Use Slim Base Image (-50MB)
|
||||
|
||||
```dockerfile
|
||||
FROM debian:trixie-slim # Instead of debian:trixie
|
||||
```
|
||||
|
||||
**Impact**: Reduces base from 120MB to ~70MB
|
||||
**Risk**: Low - slim variant contains all essential runtime libraries
|
||||
**Testing Required**: Verify xpra and Wireshark launch without errors
|
||||
|
||||
#### 2. Install Without Recommended Packages (-150~200MB)
|
||||
|
||||
```dockerfile
|
||||
RUN apt-get install -y --no-install-recommends \
|
||||
wireshark-common \
|
||||
wireshark \
|
||||
xpra=6.4.3* \
|
||||
xpra-x11 \
|
||||
xvfb \
|
||||
curl \
|
||||
x11-utils
|
||||
```
|
||||
|
||||
**Impact**: Prevents installation of non-essential recommended packages
|
||||
**Risk**: Low - only excludes recommended packages, not required dependencies
|
||||
**Testing Required**: Full functionality test (capture, analysis, WebSocket)
|
||||
|
||||
#### 3. Cleanup Unnecessary Files (-400~500MB)
|
||||
|
||||
```dockerfile
|
||||
RUN apt-get install -y --no-install-recommends \
|
||||
wireshark-common wireshark xpra=6.4.3* xpra-x11 xvfb curl x11-utils \
|
||||
# Remove documentation and man pages
|
||||
&& rm -rf /var/lib/apt/lists/* \
|
||||
&& rm -rf /usr/share/doc/* /usr/share/man/* /usr/share/help/* \
|
||||
# Remove unnecessary locales (keep only en_US)
|
||||
&& rm -rf /usr/share/locale/* \
|
||||
&& localedef -i en_US -f UTF-8 en_US.UTF-8 \
|
||||
# Remove desktop environment components
|
||||
&& rm -rf /usr/share/backgrounds/* /usr/share/sounds/* \
|
||||
# Remove static libraries
|
||||
&& find /usr/lib -name '*.a' -delete \
|
||||
&& find /usr/lib -name '*.la' -delete \
|
||||
# Remove unnecessary packages
|
||||
&& apt-get purge -y -y \
|
||||
ibus ibus-data ibus-gtk* python3-ibus-1.0 \
|
||||
gnome-backgrounds \
|
||||
&& apt-get autoremove -y \
|
||||
&& apt-get clean
|
||||
```
|
||||
|
||||
**Impact**: Removes documentation, locales, desktop components, input framework
|
||||
**Risk**: Low - all removed components are unnecessary in headless container
|
||||
**Testing Required**: Verify Wireshark GUI renders correctly without icons/themes
|
||||
|
||||
### Phase 2: Experimental (Requires Testing, -200~500MB)
|
||||
|
||||
Higher-risk optimizations that need extensive validation.
|
||||
|
||||
#### 4. Alpine Linux Alternative (-500MB~1GB)
|
||||
|
||||
```dockerfile
|
||||
FROM alpine:3.19
|
||||
RUN apk add --no-cache wireshark xpra xvfb curl ...
|
||||
```
|
||||
|
||||
**Impact**: Could reduce image to ~500MB-1GB
|
||||
**Risk**: **High** - Wireshark and xpra have complex Qt/GTK dependencies
|
||||
**Challenges**:
|
||||
- Wireshark Qt dependencies may not be available in Alpine repos
|
||||
- xpra package availability and compatibility
|
||||
- X11 library differences
|
||||
- May require building dependencies from source
|
||||
|
||||
**Testing Required**:
|
||||
- [ ] Verify Wireshark package availability in Alpine
|
||||
- [ ] Test xpra compilation/installation on Alpine
|
||||
- [ ] Validate all GUI libraries work correctly
|
||||
- [ ] Full integration testing
|
||||
|
||||
**Recommendation**: Do not pursue unless Phase 1 insufficient
|
||||
|
||||
## Compression Feasibility Analysis
|
||||
|
||||
### Question: Can we reduce image size through compression?
|
||||
|
||||
**Short Answer**: **Not recommended** - limited benefit with performance trade-offs
|
||||
|
||||
### Current Compression Status
|
||||
|
||||
Docker images already use compression:
|
||||
|
||||
| Format | Size | Compression Rate | Use Case |
|
||||
|--------|------|------------------|----------|
|
||||
| Runtime size | 2.0 GB | - | Container running |
|
||||
| docker save (raw) | 1.9 GB | 5% | Docker internal compression |
|
||||
| docker save + gzip | 718 MB | **64%** | Standard transfer |
|
||||
| docker save + xz | 563 MB | **72%** | Maximum compression |
|
||||
| docker save + zstd | ~600 MB | **70%** | Fast compression |
|
||||
|
||||
### Binary Compression Analysis
|
||||
|
||||
#### File Already Stripped
|
||||
All binaries already have debug symbols removed:
|
||||
```bash
|
||||
wireshark: ELF 64-bit... stripped
|
||||
python3.13: ELF 64-bit... stripped
|
||||
libc.so.6: ELF 64-bit... stripped
|
||||
```
|
||||
|
||||
**No further stripping possible**
|
||||
|
||||
#### UPX Executable Compression (Limited Benefit)
|
||||
|
||||
Test results compressing major executables:
|
||||
|
||||
| Binary | Original | UPX Compressed | Savings | Startup Impact |
|
||||
|--------|----------|----------------|---------|----------------|
|
||||
| wireshark (11MB) | 11.0 MB | 4.2 MB | 62% | +0.3s |
|
||||
| Xvfb (2.1MB) | 2.1 MB | 0.9 MB | 57% | +0.1s |
|
||||
| python3.13 (6.6MB) | 6.6 MB | 2.8 MB | 58% | +0.2s |
|
||||
| **Total** | **19.7 MB** | **7.9 MB** | **60%** | **+0.6s** |
|
||||
|
||||
**Overall Impact**: Only ~20MB savings (1%) with 0.6s startup penalty
|
||||
|
||||
### Why Compression Has Limited Benefit
|
||||
|
||||
1. **Small Executable Footprint**: Binaries are only 74MB (3.7% of image)
|
||||
2. **Libraries Are Data Files**: `/usr/lib` contains mostly data, not code
|
||||
3. **Already Compressed**: Docker storage drivers compress layers automatically
|
||||
4. **Resource Files Dominate**: Fonts, icons, themes don't compress well
|
||||
|
||||
### Compression Trade-offs
|
||||
|
||||
| Method | Potential Savings | Performance Impact | Complexity | Risk |
|
||||
|--------|-------------------|-------------------|------------|------|
|
||||
| **File cleanup** | 400-500MB (20-25%) | None | Low | Low |
|
||||
| UPX compression | 50-100MB (2.5-5%) | +0.6s startup | Medium | Medium |
|
||||
| Layer squashing | 10-50MB (0.5-2.5%) | None | Low | Low |
|
||||
| Transfer compression | 1.4GB (70%) | None (transfer only) | None | None |
|
||||
|
||||
### Recommendation
|
||||
|
||||
**Do not pursue binary compression** because:
|
||||
- ✅ File cleanup is **4-10x more effective**
|
||||
- ✅ No performance penalty
|
||||
- ✅ Simpler build process
|
||||
- ✅ Better compatibility
|
||||
|
||||
**For transfer/storage optimization**, use standard tools:
|
||||
```bash
|
||||
# For archiving (use zstd for best speed/ratio)
|
||||
docker save gns3/web-wireshark:latest | \
|
||||
zstd -19 -o web-wireshark.tar.zst
|
||||
|
||||
# For maximum compression (slow)
|
||||
docker save gns3/web-wireshark:latest | \
|
||||
xz -9 -T 0 > web-wireshark.tar.xz
|
||||
```
|
||||
|
||||
## Implementation Plan
|
||||
|
||||
### Step 1: Create Optimized Dockerfile
|
||||
|
||||
Create `gns3server/agent/web_wireshark/docker/Dockerfile.optimized` with Phase 1 changes.
|
||||
|
||||
### Step 2: Local Testing
|
||||
|
||||
```bash
|
||||
# Build optimized image
|
||||
cd gns3server/agent/web_wireshark/docker
|
||||
docker build -f Dockerfile.optimized -t gns3/web-wireshark:optimized .
|
||||
|
||||
# Verify size reduction
|
||||
docker images | grep web-wireshark
|
||||
|
||||
# Test Wireshark functionality
|
||||
docker run --rm gns3/web-wireshark:optimized wireshark --version
|
||||
|
||||
# Test xpra functionality
|
||||
docker run --rm gns3/web-wireshark:optimized xpra --version
|
||||
```
|
||||
|
||||
### Step 3: Integration Testing
|
||||
|
||||
```bash
|
||||
# Test with actual GNS3 server
|
||||
pip install . && gns3server-web-wireshark-setup
|
||||
|
||||
# Start a test session
|
||||
python3 gns3server/agent/web_wireshark/manage_wireshark.py \
|
||||
--verbose start \
|
||||
--project-id "test-optimized" \
|
||||
--link-id "test-link-1" \
|
||||
--jwt-token "test-token" \
|
||||
--image "gns3/web-wireshark:optimized"
|
||||
|
||||
# Verify WebSocket connectivity and packet capture
|
||||
```
|
||||
|
||||
### Step 4: Production Rollout
|
||||
|
||||
1. Tag optimized image: `gns3/web-wireshark:v1.6-optimized`
|
||||
2. Deploy to staging environment
|
||||
3. Monitor for 1 week with real workloads
|
||||
4. If stable, promote to `latest` tag
|
||||
5. Keep old image available for rollback
|
||||
|
||||
## Testing Checklist
|
||||
|
||||
Before marking as complete:
|
||||
|
||||
- [ ] Wireshark launches without errors
|
||||
- [ ] xpra HTML5 client connects successfully
|
||||
- [ ] Packet capture works end-to-end
|
||||
- [ ] Packet analysis and filtering functional
|
||||
- [ ] WebSocket proxy integration works
|
||||
- [ ] Multi-session handling tested (3+ simultaneous captures)
|
||||
- [ ] Image size measured and documented
|
||||
- [ ] Tested on Docker 20.10, 24.x, 29.x
|
||||
- [ ] Startup time not negatively impacted
|
||||
- [ ] Memory/CPU usage unchanged
|
||||
- [ ] All existing tests pass
|
||||
|
||||
## Expected Results
|
||||
|
||||
### Phase 1: Safe Optimizations (Recommended)
|
||||
|
||||
| Metric | Current | Target | Improvement |
|
||||
|--------|---------|--------|-------------|
|
||||
| Image Size | 2GB | ~1.5GB | **-25% (500MB)** |
|
||||
| Pull Time | 3-5 min | 2-3 min | **-40%** |
|
||||
| Startup Time | 5-6s | 5-6s | No change |
|
||||
| Functionality | Full | Full | No regression |
|
||||
| Compatibility | All | All | No change |
|
||||
|
||||
### Phase 2: Experimental (If Needed)
|
||||
|
||||
| Metric | Current | Target | Improvement |
|
||||
|--------|---------|--------|-------------|
|
||||
| Image Size | 2GB | ~1GB | **-50% (1GB)** |
|
||||
| Pull Time | 3-5 min | 1-2 min | **-60%** |
|
||||
| Startup Time | 5-6s | 5-7s | Slight increase |
|
||||
| Functionality | Full | Full | Risk of regressions |
|
||||
| Compatibility | All | Alpine only | Significant testing required |
|
||||
|
||||
### Compression Comparison (Not Recommended)
|
||||
|
||||
| Method | Size | Savings | Trade-offs |
|
||||
|--------|------|---------|------------|
|
||||
| **File cleanup** | ~1.5GB | 500MB | None |
|
||||
| UPX compression | ~1.9GB | 100MB | +0.6s startup, compatibility risk |
|
||||
| Transfer compression | 600MB | 1.4GB | Transfer only, no runtime benefit |
|
||||
|
||||
## Related Files
|
||||
|
||||
| File | Current State | Changes Needed |
|
||||
|------|---------------|----------------|
|
||||
| `gns3server/agent/web_wireshark/docker/Dockerfile` | Current 2GB image | Create optimized variant |
|
||||
| `gns3server/agent/web_wireshark/setup_wireshark_image.py` | Pulls/builds current image | Support optimized image option |
|
||||
| `gns3server/schemas/config.py` | WebWiresharkSettings | Add image variant config |
|
||||
| `gns3server/agent/web_wireshark/WEB_WIRESHARK.md` | Documents current image | Update with optimization notes |
|
||||
|
||||
## Status
|
||||
|
||||
### Phase 1: Safe Optimizations
|
||||
|
||||
- [ ] Create `Dockerfile.optimized` with slim base + --no-install-recommends + cleanup
|
||||
- [ ] Local build and size verification
|
||||
- [ ] Functional testing (Wireshark, xpra, WebSocket)
|
||||
- [ ] Integration testing with GNS3 server
|
||||
- [ ] Document actual size reduction achieved
|
||||
- [ ] Deploy to staging for 1-week observation
|
||||
- [ ] Promote to production if stable
|
||||
|
||||
### Phase 2: Experimental (Only if Phase 1 insufficient)
|
||||
|
||||
- [ ] Research Alpine Wireshark/xpra package availability
|
||||
- [ ] Prototype Alpine build if feasible
|
||||
- [ ] Extensive compatibility testing
|
||||
- [ ] Performance benchmarking vs. Phase 1
|
||||
|
||||
## Notes
|
||||
|
||||
- All size estimates based on actual `docker history` and container filesystem analysis
|
||||
- Phase 1 optimizations are conservative and should be safe
|
||||
- Compression techniques (UPX, layer squashing) provide minimal benefit (1-5%)
|
||||
- File cleanup is **10x more effective** than compression (25% vs 2.5%)
|
||||
- Phase 2 requires significant research and testing effort
|
||||
- Backward compatibility must be maintained during transition
|
||||
- Consider maintaining both `latest` and `optimized` tags during migration period
|
||||
|
||||
### Key Findings from Analysis
|
||||
|
||||
1. **Major space waste**: 570MB of cleanable files (28.5% of image)
|
||||
- Locale files: 151MB → 6MB (keep only en_US)
|
||||
- ibus input framework: 130MB → 0MB (not needed in headless container)
|
||||
- Documentation: 76MB → 0MB (docs, man pages, help)
|
||||
- Desktop components: 37MB → 0MB (backgrounds, sounds)
|
||||
- Development packages: 50MB → 0MB (21 -dev packages)
|
||||
|
||||
2. **Compression not viable**:
|
||||
- Binaries already stripped (no debug symbols)
|
||||
- UPX only saves 20MB (1%) with 0.6s startup penalty
|
||||
- Docker already compresses layers internally
|
||||
- Resource files (fonts, themes) don't compress well
|
||||
|
||||
3. **Best approach**: Clean up unnecessary files rather than compress
|
||||
- 25x better compression ratio than UPX
|
||||
- No performance impact
|
||||
- Simpler build process
|
||||
- Better compatibility
|
||||
Loading…
x
Reference in New Issue
Block a user