sashatrask
2.1 KiB
Worker network OSError is logged as local I/O
Status: open, reproducible from the error-classification path.
Observed behavior
The accepted Linux worker logged the following safe summary during the live operator-experience validation:
2026-09-25T13:10:05.909Z worker slot 0: local I/O operation failed
The complete worker journal shows that reservation 1689 had already entered
assigned at 13:09:46.641Z. No runner phase had started. After the configured
error delay, the worker retried the same durable assignment, entered
preparing at 13:10:22.215Z, completed it, and received one
bundle_accepted receipt at 13:10:28Z.
The only operation between the successful assigned event and runner startup
is the authenticated assignment-status request in
WorkerSlot.step() (app/remote_worker_client.py, around lines 2141-2153).
The HTTPS client lets socket and transport OSError exceptions propagate.
safe_worker_error_summary() (app/remote_worker_client.py, around lines
128-135) maps every OSError to local I/O operation failed, including network
socket failures.
Impact
- No assignment, result, or progress data was lost in the observed incident.
- The durable retry behavior worked and did not rescan the target.
- The operator-facing message misclassifies a transient network failure as a local storage/filesystem problem, which can send diagnosis in the wrong direction.
- The original exception type is not retained in the safe worker log, so the transport subtype cannot be recovered after the fact.
Evidence
build/live-trace-20260925/linux-worker-state-interim.tar.gz- Worker event sequences 1292-1301 and the matching history row for reservation 1689
app/remote_worker_client.pystatus-read and safe-summary paths
Expected correction
Classify network/socket failures before the broad OSError branch and emit a
safe transport-specific summary. Keep filesystem/storage OSError failures as
local I/O. Add a regression test covering an OSError raised by
WorkerHTTPClient.status() and verify that retry behavior remains unchanged.