fix(telnet): handle connection race condition during broadcast

Fixes two critical issues in telnet server when clients rapidly
   connect/disconnect during broadcast operations:

   1. **OSError [Errno 107]**: Transport endpoint not connected
      - Root cause: getpeername() called outside try block
      - Fix: Move getpeername() inside try block and catch OSError
      - Expand exception handling to include OSError and TimeoutError

   2. **KeyError**: Double deletion from connections dictionary
      - Root cause: Connection deleted in broadcast loop, then deleted
        again in top-level exception handler
      - Fix: Use dict.pop(key, None) instead of del dict[key]

   **Changes**:
   - Line 216: Add OSError to top-level exception handler
   - Line 227-228: Use pop() to avoid KeyError on double deletion
   - Line 305-316: Move getpeername() inside try block, expand
     exception types, use pop() for safe deletion

   **Impact**:
   - Prevents unhandled exceptions from propagating to asyncio event loop
   - Ensures proper resource cleanup even with race conditions
   - No resource leaks or zombie connections
   - Long-lived clients unaffected by rapid client disconnects

   **Test Script**:
   - Added stress test script to reproduce the issue
   - Supports IOU-L3, VPCS, and generic device types
   - Rapid clients send broadcast-triggering commands (show/run/OSPF)

   **Related**: Fixes bug documented in docs/bugs/telnet-server-connection-race-condition.md
This commit is contained in:
YueGuobin committed 2026-03-15 00:19:51 +08:00
1 parent 3abecc8492
commit e4faf673af
3 files changed
+677 -2

No files matched your search

+3 -2
View File
@@ -224,7 +224,8 @@ class AsyncioTelnetServer:
self._current_read.cancel()
await connection.disconnected()
del self._connections[network_writer]
# Use pop() to avoid KeyError if connection was already removed
self._connections.pop(network_writer, None)
async def close(self):
for writer, connection in self._connections.items():
@@ -311,7 +312,7 @@ class AsyncioTelnetServer:
except:
log.debug(f"Timeout while sending data to client: {client_info}, closing and removing from connection table.")
connection.close()
del self._connections[connection_key]
self._connections.pop(connection_key, None)
async def _read(self, cmd, buffer, location, reader):
""" Reads next op from the buffer or reader"""