gns3-server/docs/features/project-open-performance.md

Project Open Performance

Overview

Optimizations to accelerate project opening (POST /projects/{id}/open) and node creation for topologies with many nodes and links. The main bottlenecks were sequential link creation, redundant subprocess calls, and SQLite write contention.

Before vs After

Scenario	Before	After
20 IOU nodes + 20 links (project open)	~2s	~1s
40 QEMU nodes creation (MCP batch)	~40s	~1-2s

Optimizations

1. Parallel Link Creation

File: gns3server/controller/project.py

Links were created sequentially during project loading, each requiring up to 5 HTTP round-trips to the compute. Now uses Pool(concurrency=100) for parallel creation.

# Before: sequential loop
for link_data in topology.get("links", []):
    link = await self.add_link(...)
    await link.add_node(...)

# After: parallel Pool
pool = Pool(concurrency=100)
for link_data in topology.get("links", []):
    pool.append(self._create_link_from_topology_data, link_data)
await pool.join()

2. Batch UDP Port Allocation

Files: gns3server/api/routes/compute/compute.py, gns3server/controller/project.py, gns3server/controller/udp_link.py

During project loading, all required UDP ports are pre-allocated per compute in a single batch call before link creation begins. UDPLink.create() checks the pre-allocated pool first, falling back to individual allocation if unavailable.

# New batch endpoint
POST /projects/{id}/ports/udp/batch  →  {"count": N}  →  {"udp_ports": [...]}

3. IOU Image Subprocess Cache

File: gns3server/compute/iou/iou_vm.py

Each IOU VM creation spawned ld-linux --verify and iou-image -h subprocesses. With 20 nodes using the same image, this ran 40 redundant subprocesses. Now results are cached per image path at the class level.

# Class-level caches shared across all instances
IOUVM._loader_cache = {}           # image path → loader command
IOUVM._default_values_cache = {}   # image path → (ram, nvram)

Only the first node with a given image runs the subprocesses; subsequent nodes reuse cached values.

4. SQLite WAL Mode

File: gns3server/db/tasks.py

Write-Ahead Logging allows concurrent reads without blocking on writes. The PRAGMA is registered on engine.sync_engine instead of the Engine class to correctly fire for async engine connections.

@event.listens_for(engine.sync_engine, "connect")
def set_sqlite_pragma(dbapi_connection, connection_record):
    cursor = dbapi_connection.cursor()
    cursor.execute("PRAGMA journal_mode=WAL")
    cursor.execute("PRAGMA foreign_keys=ON")
    cursor.close()

Without WAL mode, concurrent API requests caused sqlite3.OperationalError: database is locked.

5. API Key Authentication O(1) Lookup

Files: gns3server/api/routes/controller/api_keys.py, gns3server/api/routes/controller/dependencies/authentication.py

Old format: gns3_<random> — required scanning ALL keys and running bcrypt on each (O(n)). New format: gns3_<api_key_id>_<random_secret> — extract UUID from token, single DB query (O(1)), single bcrypt.

# New auth flow
parts = token.split("_", 2)
key_id = UUID(parts[1])
secret = parts[2]
db_key = await api_keys_repo.get_api_key(key_id)  # O(1) lookup
if await asyncio.to_thread(bcrypt.checkpw, secret.encode(), db_key.key_hash.encode()):
    # Authenticated — 1 query + 1 bcrypt regardless of total key count

6. bcrypt in Thread Pool

File: gns3server/api/routes/controller/dependencies/authentication.py

bcrypt.checkpw() is CPU-bound (~1.3s per call) and was blocking the async event loop. With 5 API keys and 10 concurrent requests, this caused ~13s delay before any handler could start.

# Before: blocking the event loop
if bcrypt.checkpw(token.encode(), db_key.key_hash.encode()):
    ...

# After: offloaded to thread pool
if await asyncio.to_thread(bcrypt.checkpw, secret.encode(), db_key.key_hash.encode()):
    ...

7. Concurrency Settings

Setting	Before	After	File
Node creation Pool	5	100	controller/project.py
Link creation Pool	5	100	controller/project.py
MCP BATCH_MAX_WORKERS	10	100	api/routes/mcp/nodes.py
MCP HTTP timeout	10s	30s	agent/gns3_copilot/gns3_client/custom_gns3fy.py
HTTP connection pool	10 (default)	500/1000	agent/gns3_copilot/gns3_client/custom_gns3fy.py
Start nodes Pool	3	3 (unchanged)	controller/project.py

8. MCP Auth Returns JWT

File: gns3server/api/routes/mcp/__init__.py

When an MCP client connects with an API key, the _resolve_token function validates the key then returns a fresh short-lived JWT instead of the raw API key. The JWT is stored in a ContextVar and reused for all subsequent tool calls within the same SSE session — zero extra bcrypt.

if user:
    fresh_token = auth_service.create_access_token(user.username)
    return fresh_token  # JWT for subsequent REST API calls

Related Files

File	Changes
gns3server/controller/project.py	Parallel link creation, batch UDP, Pool(100)
gns3server/compute/iou/iou_vm.py	Image subprocess cache
gns3server/db/tasks.py	WAL mode + sync_engine event listener
gns3server/api/routes/compute/compute.py	Batch UDP endpoint
gns3server/controller/udp_link.py	Pre-allocated port consumption
gns3server/api/routes/controller/api_keys.py	O(1) key format
gns3server/api/routes/controller/dependencies/authentication.py	O(1) auth + thread pool bcrypt
gns3server/api/routes/mcp/__init__.py	Auth returns JWT, tool enhancements
gns3server/api/routes/mcp/nodes.py	fields filter, inherited template_id, name passthrough
gns3server/api/routes/mcp/links.py	fields filter, compact array format
gns3server/agent/gns3_copilot/gns3_client/custom_gns3fy.py	Timeout 30s, connection pool 500/1000
gns3server/utils/images.py	md5sum cache error → warning