Two exception paths could permanently kill the compute notification
chain (no more compute.updated events, no reconnection until a server
restart):
- connect() only caught ComputeError, but _run_http_query translates
HTTP status errors (401/403/404/...) into sibling ControllerError
subclasses (and a raw fastapi HTTPException for unexpected statuses).
Those escaped the fire-and-forget connect() task started at controller
startup and died silently. Now they notify clients, schedule an
exponential-backoff retry, and still re-raise for explicit callers.
The dead web.HTTP* except branches (never reached since
_run_http_query converts HTTP errors itself) are removed.
- _connect_notification() only caught aiohttp.ClientError. A malformed
frame (e.g. missing 'action') or any error raised while dispatching a
compute event (e.g. a pydantic ValidationError in
node.parse_node_response) escaped the task, skipped the reconnect
scheduling placed after the try block, and killed the stream forever.
Now any exception is logged with its traceback (the gather() future
holding it was never retrieved, so nothing was ever printed) and the
reconnect scheduling + final compute.updated emit live in the finally
block so every exit path recovers.
Also moves the usage-stats reset before the disconnect log line so the
emitted compute.updated snapshot is consistent.
The per-commit force_close=not _local optimisation reused TCP connections
for the loopback compute, but _session() is also used for the controller's
WebSocket heartbeat connection (_connect_notification -> ws_connect).
The different connector behaviour prevented the compute from receiving
pings, so no compute.updated events reached the WebUI and the compute
cache stayed empty.
host_ip resolved socket.gethostbyname on every access with no cache.
get_ip_on_same_subnet touches host_ip 2-4 times per link, so opening a
2500-link project issued thousands of blocking DNS calls inline on the
event loop — freezing all concurrent link coroutines each time. This is
the most likely cause of the 12-link/s throughput (1000x below what
Pool(concurrency=100) should deliver) and the burst+pause pattern.
- compute.host_ip: cache the resolution in _host_ip_cache, invalidate
on host setter change
- UDPLink.create: timing log splitting get_ip / ports / nio so the next
project-open confirms where time actually goes
When controller and compute share the same process, every HTTP request
to localhost was paying a full TCP handshake (force_close=True forced
connection teardown after each request). With 2500+ links each sending
two NIO POSTs, that's 5000 SYN->SYN-ACK->ACK cycles even for sub-ms
in-memory handlers. Switch to keep-alive for loopback compute
(127.0.0.1 / ::1 / localhost) while keeping force_close for remote
computes that may sit behind NAT/firewalls that drop idle connections.
Create async_iterable_to_stream() in gns3server.utils.asyncio that
converts an async iterable to an aiohttp StreamReader via a background
feeder task. This bypasses aiohttp's AsyncIterablePayload which can
cause 'Connection reset by peer' with certain HTTP servers.
Use it in _run_http_query for the __aiter__ data path.
- Stream file GET/POST through controller without buffering in memory
- Add recursive and subdirectory filtering to node file listing
- Replace file extension with magic-based file type detection
- Add DELETE endpoint for node and project files
- Include directories in listing response
- Add params and stream support to http_query
- Fix lambda closures, streamer exception scope, and delete error codes
When a remote compute is unreachable, the controller now uses exponential
backoff for reconnection attempts: 5s, 10s, 20s, 40s, 80s, then caps at
300s (5 minutes). Previously it retried every 5 seconds indefinitely.
User-initiated operations (open project, start node) still trigger an
immediate connection attempt, so recovery is not delayed in practice.
Related: #2704
This commit addresses issue #2703 where deleting a project with nodes
on remote compute nodes would result in long waits with no feedback
if those computes were unreachable.
Changes:
1. Compute connection status updates on connection failure
- When a compute fails to connect, update connected=False and last_error
- Send compute.updated notification to UI so users can see status
- This allows Web UI to display real-time connection status
2. Project deletion checks compute status before attempting deletion
- Check all computes used by the project are connected
- If any compute is disconnected, immediately reject deletion
- Provide clear error message indicating which computes are offline
- This prevents long timeouts and gives users immediate feedback
Benefits:
- Immediate feedback instead of 120-second timeouts
- Clear error messages about which computes are disconnected
- Prevents orphaned resources on offline computes
- Improves user experience by avoiding silent waits
Related: #2703
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove check for open project. Without this check a remote can be rebooted and will be usable once the main server polls it again. Without this the main server would need to open a project that already uses the remote server or restart the main server's gns3 process.
