fix: make node working directory deletion robust

The rmtree error handler in BaseNode.delete() was chmod'ing the failed
path to S_IWRITE (0o200). On POSIX this strips the search permission
from directories, turning a transient deletion failure into a directory
that can no longer be traversed or deleted. The node deletion itself
silently "succeeds" (rmtree gives up once its error handler returns)
and a later project deletion then fails with EACCES. The handler also
never retried the failed operation, so it did not help on Windows
either (the platform it was written for).

The transient failure exists in practice: a concurrent MD5 checksum
computation caching its result in the node directory (e.g. a properties
request racing the deletion) can recreate a file after rmtree has
listed the directory, making the final rmdir fail with ENOTEMPTY.

- add the missing user permissions instead of replacing the whole mode,
  and retry the failed unlink/rmdir
- retry the whole deletion a few times to absorb files recreated while
  the directory is being deleted
- raise a ComputeError when the directory cannot be fully deleted
  instead of failing silently
This commit is contained in:
YueGuobin committed 2026-09-15 01:23:31 +08:00
1 parent 1c57bd7dca
commit 84e7ead465
3 files changed
+136 -4

No files matched your search

+31 -2
View File
@@ -340,14 +340,43 @@ class BaseNode:
"""
def set_rw(operation, name, exc):
os.chmod(name, stat.S_IWRITE)
# Add the missing user permissions instead of replacing the whole mode: chmod'ing
# a directory to S_IWRITE removes the search permission on POSIX systems, making
# the directory impossible to traverse or to delete. Also note that S_IWRITE
# clears the read-only attribute on Windows.
try:
if os.path.isdir(name) and not os.path.islink(name):
os.chmod(name, os.stat(name).st_mode | stat.S_IRWXU)
elif not os.path.islink(name):
os.chmod(name, os.stat(name).st_mode | stat.S_IRUSR | stat.S_IWUSR)
# retry the failed operation now that the permissions are fixed
# (retrying os.scandir is not possible, the directory is handled by the
# retry loop below)
if operation in (os.unlink, os.rmdir):
operation(name)
except OSError:
pass
directory = self.project.node_working_directory(self)
if os.path.exists(directory):
# Retry the deletion: a concurrent task (e.g. a MD5 checksum computation caching
# its result in the node directory) can recreate a file while the directory is
# being deleted, and shutil.rmtree silently gives up when its error handler returns
for attempt in range(3):
if not os.path.exists(directory):
return
try:
await wait_run_in_executor(shutil.rmtree, directory, onerror=set_rw)
except OSError as e:
raise ComputeError(f"Could not delete the node working directory: {e}")
if not os.path.exists(directory):
return
if attempt == 2:
raise ComputeError(
f"Could not delete the node working directory '{directory}': a file may have been "
"recreated in it or could not be removed"
)
log.warning(f"Could not completely delete the node working directory '{directory}', retrying")
await asyncio.sleep(0.1)
def start(self):
"""