When importing a project, the round-robin logic would attempt to distribute
nodes across all configured compute resources, including offline ones. This
caused import failures when any remote compute was unreachable.
This fix filters the compute list to only include connected computes before
round-robin distribution. If no remote computes are connected, all nodes
are assigned to the local compute.
This matches the approach used in project._get_disconnected_computes() and
prevents the issue where importing a project fails with:
"Cannot connect to compute 'X' with request POST /projects"
Fixes issue introduced in commit 90e3a8d6 (2017) which added round-robin
load balancing without considering offline compute nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Fix F821 undefined name 'status' error by importing the status
module from fastapi. This resolves build errors when using
status.HTTP_403_FORBIDDEN and status.HTTP_404_NOT_FOUND.
Add exception handling in stop_wrap_console to gracefully handle
ConnectionResetError, BrokenPipeError, and OSError when waiting
for console writer to close.
This prevents 500 errors when stopping QEMU nodes if the console
connection is reset before the writer finishes closing.
Fixes race condition where QEMU process exits and closes connections
before the console writer cleanup completes.
Remove MD5 checksum calculation from the node files API since
disk image files are dynamic and change frequently. MD5 calculation
was also causing significant performance overhead.
Changes:
- Remove md5sum field from NodeFile schema
- Remove MD5 calculation from list_node_files method
- Improve error handling for timestamp conversion
- Simplify code by removing lambda functions
Performance improvement:
- Response time reduced from ~0.5-1s to ~0.017s (30-60x faster)
- Especially beneficial for large files and multiple files
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add NodeFile to the schemas __init__.py exports to fix
import error when starting the server.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Enhance the node files API to include comprehensive file metadata:
- File size in bytes
- File creation time (ISO 8601 format)
- File modification time (ISO 8601 format)
- File extension
Create new NodeFile schema to support these additional fields
while keeping the existing ProjectFile schema for backward compatibility.
This provides users with better information to manage and identify
files in the Web UI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Implement issue #2719 - Add API endpoint to list project files
- Add GET /v3/projects/{project_id}/nodes/{node_id}/files endpoint
- Add list_node_files() method to Project class
- Add security checks to prevent path traversal
- Filter out .ghost temporary files
- Return file paths with MD5 checksums
- Require Node.Audit privilege
This allows users to discover dynamically created files
such as QEMU disk images created via the disk image API.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When closing a Docker node, if container deletion fails, the error
is silently ignored. This can lead to stale containers remaining on
the system and causing 409 conflicts when reopening projects.
Changes:
- Distinguish between 404 (container already removed, normal) and
other DockerError (deletion failed, needs attention)
- Log warning when deletion fails with error details
- Add comment explaining stale containers will be cleaned up on
project open (via automatic 409 conflict resolution)
This improves observability without blocking project close operations.
The root cause of stale containers can now be diagnosed from logs.
Fixes#2708
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a remote compute is unreachable, the controller now uses exponential
backoff for reconnection attempts: 5s, 10s, 20s, 40s, 80s, then caps at
300s (5 minutes). Previously it retried every 5 seconds indefinitely.
User-initiated operations (open project, start node) still trigger an
immediate connection attempt, so recovery is not delayed in practice.
Related: #2704
When opening a closed project with nodes on an offline remote compute,
open() would block for 120s trying to connect before eventually
failing. Now checks compute connectivity after loading the topology
file but before creating nodes, allowing immediate failure with a
clear error message.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously the disconnected compute check ran after open(), which would
block for 120s trying to connect to unreachable remote computes before
rejecting the deletion. Now reads the topology file directly to extract
compute IDs and checks connectivity before calling open(), enabling
immediate rejection of deletions involving offline computes.
Also removes the redundant post-open() check since the early check
covers both opened and closed project states.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously errors during close() and delete_on_computes() were silently
swallowed without any logging, making it difficult to diagnose failures
when remote computes are unreachable.
- close(): log warning instead of silent pass
- delete_on_computes(): wrap HTTP DELETE in try/except with warning log
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When deleting the AUTOIDLEPC project in autoidlepc(), if a remote compute
is disconnected, the deletion fails with ControllerForbiddenError due to
our compute connection status check. This causes:
- The project to remain in the controller's memory
- Subsequent autoidlepc calls to fail
- User must restart the server to recover
This fix adds exception handling to ensure the project is always removed
from the controller's memory, even if deletion fails.
Related to #2703
The initial fix used _project_created_on_compute to check for disconnected
computes before deletion, but this set gets reset during project.open(),
causing the check to fail.
Now uses self._computes which is loaded from the topology file and
persists through the open() call.
Related: #2703
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit addresses issue #2703 where deleting a project with nodes
on remote compute nodes would result in long waits with no feedback
if those computes were unreachable.
Changes:
1. Compute connection status updates on connection failure
- When a compute fails to connect, update connected=False and last_error
- Send compute.updated notification to UI so users can see status
- This allows Web UI to display real-time connection status
2. Project deletion checks compute status before attempting deletion
- Check all computes used by the project are connected
- If any compute is disconnected, immediately reject deletion
- Provide clear error message indicating which computes are offline
- This prevents long timeouts and gives users immediate feedback
Benefits:
- Immediate feedback instead of 120-second timeouts
- Clear error messages about which computes are disconnected
- Prevents orphaned resources on offline computes
- Improves user experience by avoiding silent waits
Related: #2703
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously only the "path" field was converted to a relative path for remote computes, causing issues with QEMU and VMware nodes that use different image path fields. Now all known image path fields are converted, ensuring consistent behavior across IOU, QEMU, Dynamips, and VMware node types.
When creating a node on a remote compute, the controller now sends
only the image filename instead of the absolute local path. The remote
compute will search for the image in its own configured images_directories.
If the image is not found, the compute returns ImageMissingError, which
triggers the controller's automatic image upload mechanism.
This fixes issue #2676 where remote computes reject paths from the
controller due to mismatched directory prefixes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When accessing /static/web-ui without trailing slash, the request would
fail with "RuntimeError: File at path ... is not a file" because:
1. The route /static/web-ui/{file_path:path} doesn't match paths without
trailing slash (Starlette's path regex requires the /)
2. The request falls through to StaticFiles mount, which tries to serve
the directory as a file
This fix:
- Sets html=True on StaticFiles mount to automatically redirect directory
URLs to trailing slash versions
- Adds os.path.isdir() check to handle empty file_path gracefully
Fixes#2680
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
