truf-server/WORKER_OPERATOR_EXPERIENCE_HANDOFF.md

# Worker Operator Experience Handoff

> Historical handoff. The current continuation entry point is
> `docs/session-handoff/README.md`. This file retains implementation and artifact
> provenance, but its stop point and immediate-next-actions section are obsolete.

Last updated: 2026-09-25

This is the historical implementation record for the OpenSpec change
`add-worker-operator-experience`. For current continuation instructions, read
`docs/session-handoff/README.md`. Do not repeat completed production validation
or rebuild accepted artifacts unless a current verification fails.

## User intent and constraints

- Continue autonomously from this handoff and finish the change end to end.
- The user explicitly requested a file handoff because invoking conversation
  compression appears to stop or destabilize all OpenCode sessions. Avoid
  proactively invoking the compression tool in the continuation session.
- Workspace: `D:\truf-workers`.
- Use the configured SSH server named `sec` only. Never call or connect through
  the configured server named `prod`.
- Never print or record tokens, credentials, the private admin prefix, raw
  targets/findings, runtime YAML, or worker command lines containing auth data.
- Do not add a new masking, redaction, credential-sandbox, or other security
  scope without explicit approval and an OpenSpec requirement.
- Do not remove or revert unrelated workspace files. The repository baseline is
  entirely untracked (`git status --short` shows the whole tree as `??`), so Git
  cannot provide a meaningful task-specific diff.
- Do not archive the OpenSpec change unless the user explicitly asks. Completing
  tasks and reporting "ready to archive" is expected.

## Current OpenSpec state

- Change: `add-worker-operator-experience`
- Schema: `spec-driven`
- Artifact status: proposal, design, specs, and tasks are complete.
- Apply progress before final closure: 23/27 tasks complete.
- File: `openspec/changes/add-worker-operator-experience/tasks.md`
- Tasks 1.1 through 5.5 are checked.
- Remaining unchecked tasks:
  - 6.1: validate the canonical from-zero operator guide.
  - 6.2: complete test matrix, reproducible packages, manifest registration,
    and documented identities.
  - 6.3: bounded Windows and WSL/Docker production validation and restoration.
  - 6.4: durable dated report with timings, watchdog evidence, snapshots,
    transcript, known limits, rollout, and rollback.

Do not check 6.1-6.4 until the remaining focused matrix, report, and strict
OpenSpec validation have passed.

## Implemented scope

The change now includes:

- Versioned worker phase/events and canonical transitions.
- Monotonic sequence handling and JSON/NDJSON contracts.
- Unified bounded diagnostics with deterministic identities.
- PostgreSQL progress/diagnostic persistence and admin queries.
- Server-owned global/per-source assignment deadline policy.
- Private local state, logs, history, diagnostic artifacts, and retention.
- Status, attach, logs/history, JSON/NDJSON, bounded follow/tail CLI behavior.
- Cross-platform worker supervisor, control protocol, drain/stop, shutdown
  receipts, stale instance handling, and recovery slots.
- Per-assignment contained runner, controller protocol, watchdog, timeout bundle
  publication, restart adoption, and abandoned-root cleanup.
- Authenticated progress endpoint and diagnostic ingestion.
- Admin assignment/progress/diagnostic experience.
- Windows portable and Linux image packaging for the supervisor runtime.
- Canonical operator runbook in `docs/remote-worker-operations.md`.

Important implementation files include:

- `app/worker_contracts.py`
- `app/worker_local_state.py`
- `app/worker_supervisor.py`
- `app/worker_cli.py`
- `app/worker_assignment_runner.py`
- `app/remote_worker_client.py`
- `app/worker_api.py`
- `app/scanner_db.py`
- `app/admin_api.py`
- `app/worker_package.py`
- `app/worker_package_builder.py`
- `docker/verify_packaged_workers.py`
- Worker-related tests under `tests/`

## Final correctness fixes

### Recovered ready-bundle transition

`WorkerSlot` could recover a published ready bundle while its persisted event
phase was still `assigned`. Upload code emitted `uploading` only from
`bundling/backoff`, then attempted the invalid transition
`assigned -> awaiting_receipt`.

Fix in `app/remote_worker_client.py`:

- Emit `UPLOADING` when the current event phase is `ASSIGNED`, as well as the
  existing bundling/backoff cases.
- Regression in `tests/test_worker_api.py` validates the event sequence
  `assigned -> uploading -> awaiting_receipt`.

The focused worker API/local-state/supervisor suite passed 100 tests after this
fix.

### Packaged E2E abandoned work invariant

Completed runner roots are intentionally retained under `work/abandoned` for at
least 60 seconds; retention maintenance normally runs every 300 seconds. The E2E
harness incorrectly required the total work file count to be zero, causing a
false `linux_direct_claims_timeout` after Linux had correctly claimed both direct
assignments.

Fix in `docker/verify_packaged_workers.py`:

- Linux and Windows work-tree identities now include `active_entries`.
- Files/directories beneath top-level `abandoned` are retained but not active.
- Direct-assignment and final-cleanup predicates require zero active entries,
  while preserving strict state and bundle identity checks.
- Outage marker waits also check worker liveness, so an exited worker fails
  immediately rather than timing out after four minutes.

### Windows `prepare-worker.ps1` ACL defect

Testing a freshly extracted ZIP exposed a real release bug. The old generated
script ran `icacls ... /grant:r ... /T`; on descendants this produced
inheritance-only ACEs, returned success, and made packaged `python.exe`
inaccessible.

Final fix in `app/worker_package_builder.py`:

1. Set private inheritable full-control ACEs for the current user, SYSTEM, and
   Administrators on the package root only.
2. Run `icacls (Join-Path $root '*') /inheritance:d /T /C` so each descendant
   converts inherited ACLs to explicit protected ACLs with the correct file or
   directory flags.

Regression in `tests/test_worker_package.py` checks the generated script and, on
Windows, executes it and verifies `private_directory_ready(root)` plus
`private_file_ready(child)`. `tests/test_worker_package.py` passes 20 tests.

Do not use the earlier `/reset /T` idea: inherited ACLs are not accepted because
runtime trust requires protected explicit ACLs.

### Watchdog test timing stabilization

The broad focused suite exposed two false failures because three tests created a
100 ms absolute watchdog deadline before runner protocol-root/state setup. Under
the complete Windows suite that setup could consume the deadline, exercising the
startup-deadline branch instead of the intended blocked-operation watchdog.

Test-only changes in `tests/test_worker_assignment_runner.py`:

- Affected tests:
  - `test_watchdog_kills_while_state_persistence_is_blocked`
  - `test_blocked_startup_gate_write_enters_preparing_timeout_result_path`
  - `test_watchdog_kills_while_event_drain_is_blocked`
- Scan deadline: 1 second -> 2 seconds.
- Watchdog deadline: 0.1 second -> 1 second.
- Injected block: 0.4 second -> 1.4 seconds.
- Kill bound: 0.3 second -> 1.3 seconds.

This preserves the independent watchdog assertion and does not weaken product
code. The exact three-test rerun passed: `3 passed in 5.17s`.

## Accepted reproducible artifacts

### Windows final pair: I and J

Paths:

- `build/operator-experience-validation/windows-i.zip`
- `build/operator-experience-validation/windows-i.zip.json`
- `build/operator-experience-validation/windows-j.zip`
- `build/operator-experience-validation/windows-j.zip.json`

Both independently built archives are identical:

- Bytes: `134850988`
- Archive SHA-256:
  `6ea9290736a059f1e17d8e89d9cf83506fa4abe2ba2f3731a7422a7b0f386e97`
- Package manifest identity:
  `78a962b2bd3fa411413c79e9a8ffb021608a08ff020b1ad851f4505ea634b2b6`
- Build-input identity:
  `6991ebbce6ae758c2bdd19a6ae934335aa585a50f86b18ccde8d88bca40ce436`
- Raw `worker-package.json` SHA-256:
  `e0b17d70fcb868fe39fac45ab6e05a17c6d40852e6034010fb63b6cab31f8a3c`

Acceptance used a fresh extraction, not the builder output:

- `build/pwe-final-i-extracted`
- The package's own corrected `prepare-worker.ps1` was run once.
- Direct package verification then passed.

The older Windows G/H archives are obsolete for acceptance because they contain
the broken preparation script. Their package manifest identity happens to be the
same because the support script is outside that manifest, but their archive
identity is not accepted. Do not publish or register G/H as final Windows ZIPs.

### Linux final pair: G and H

Tags:

- `truf-worker-test:operator-experience-final-3g`
- `truf-worker-test:operator-experience-final-3h`

Both were built with provenance disabled and are reproducible:

- Worker package identity:
  `45588f2cf406b41b239cfa3b8a9dc83fe84b587229bc997b2729016e1f0dde42`
- Image manifest / accepted image ID:
  `sha256:3a088f5743121d823aae132234a29730a84339cecbfda5fc601e8e942f9948c3`
- Config:
  `sha256:687a1c4c51c1b962c7fa7ea0cc4b04d159e7ba4f94ef347940c9fb225f7cb87d`
- Raw `worker-package.json` SHA-256:
  `ee926cce3c19e9e6094753f51fa902415bd7364c24fa649cd0c1b659c0aa4d60`

Extracted final manifest:

- `build/operator-experience-validation/linux-worker-package-g.json`

Test image:

- Tag: `truf-worker-test:operator-experience-final-3`
- ID:
  `sha256:1a22c396dbf329e20caf77f88b7c7a310bda86befbcf3b917f10ded3720ee712`

## Final packaged E2E

Passed run:

- Run ID: `35f3f52e232067c1`
- Safe summary: `build/pwe-35f3f52e232067c1/summary.json`
- Windows input: freshly extracted and prepared Windows I.
- Linux input: Linux G.
- Status: passed.
- Cleanup: complete.
- Foreign Docker state: unchanged.
- Windows and Linux normalized evidence matched.
- Restart, outage, durable bundle, direct assignment, direct bundle, receipt,
  shutdown, local cleanup, and cross-platform evidence gates all passed.

Do not copy raw target values from the summary into reports or chat. Only the
safe aggregate facts above are needed.

All Docker resources from final and diagnosed failed runs were cleaned by exact
owned IDs/names. Some local `build/pwe-*` failure evidence directories remain and
are safe to leave. `build/pwe-final-g` may still have unusable ACLs after running
the old broken preparation script; do not use it. `build/pwe-final-i-extracted`
is the accepted extracted Windows directory.

## Production validation evidence

Bounded production validation was completed before final package acceptance and
production was restored afterward.

Safe aggregate results:

- Assignments issued: 34.
- Accepted: 33.
- One intentional expected expiry.
- Accepted assignments ingested, settled, and projected: 33.
- Unresolved, precommit, and quarantine counts: zero.
- Natural timeout evidence reservation: 1453.
- Full-stage progress/watchdog evidence reservation: 1455.

Evidence files:

- `build/operator-experience-validation/final-evidence.json`
- `build/operator-experience-validation/progress-v3-evidence.json`
- `build/operator-experience-validation/timeout-evidence.json`
- `build/operator-experience-validation/server-baseline.json`

These files are the source for duration percentiles, phase/watchdog evidence,
diagnostic/admin snapshots, and reconciled counts in the final report. Derive
only aggregate/sanitized facts. Do not reproduce raw targets, findings, secrets,
or private route names.

Final production state after restoration:

- Operations controls: normal/open, revision 126.
- Standard WSL production worker user: enabled, assignment cap 1.
- Standard production device: enabled and not revoked.
- Temporary validation identities: disabled/revoked.
- Runtime canonical health: healthy.
- Edge remained up.

Do not repeat production assignments merely to write the report. Existing
evidence is sufficient.

## Registered trusted manifests

Registration was completed only after the final packaged E2E passed, using SSH
server `sec` only.

Remote paths:

- `/etc/truf/worker-packages/linux-worker-package-v2.json`
- `/etc/truf/worker-packages/windows-worker-package-v3.json`

Final remote SHA-256 values match the accepted manifests:

- Linux: `ee926cce3c19e9e6094753f51fa902415bd7364c24fa649cd0c1b659c0aa4d60`
- Windows: `e0b17d70fcb868fe39fac45ab6e05a17c6d40852e6034010fb63b6cab31f8a3c`

Both are `root:root` mode `0644`. Existing
`.pre-operator-experience` backups were preserved unchanged. Upload temp files
were removed. After registration, canonical runtime health succeeded and Docker
reported `truf-docker-runtime-1` healthy. No restart or config mutation was
needed.

## Test state

Completed checks:

- Worker API/local-state/supervisor focused suite: 100 passed.
- Worker package tests after ACL fix: 20 passed.
- Exact three watchdog timing tests after stabilization: 3 passed.
- Full packaged Windows/Linux E2E: passed, run `35f3f52e232067c1`.
- Production health after final manifest registration: passed.

The broad focused matrix was run before the watchdog test timing patch:

```powershell
python -B -m pytest tests/test_worker_api.py tests/test_worker_api_runtime.py tests/test_worker_assignment.py tests/test_worker_assignment_runner.py tests/test_worker_cli.py tests/test_worker_contracts.py tests/test_worker_local_state.py tests/test_worker_observability_db.py tests/test_worker_package.py tests/test_worker_runner_handoff_linux.py tests/test_worker_supervisor.py tests/test_remote_worker_db.py tests/test_scan_execution.py tests/test_admin_api.py -q
```

Result before the timing-only patch:

- 353 passed.
- 3 skipped.
- 2 false timing failures described above.

The two failures and the nearby equivalent test pass after the patch, but the
complete 14-file command has not yet been rerun. This is the exact current stop
point.

An unrestricted repository-wide pytest run is not a useful release gate in this
checkout because unrelated private/generated assets and platform assumptions are
absent. Its known baseline was `3031 passed, 134 skipped, 68 failed`. Do not try
to fix unrelated failures as part of this change. The focused change matrix,
packaged E2E, production proof, and strict OpenSpec validation are the gates.

## Immediate next actions

1. Rerun the exact 14-file focused matrix shown above. Expected result after the
   timing patch is 355 passed and 3 skipped. If it fails, diagnose only genuine
   worker-operator regressions; do not broaden scope.
2. Create the durable report:
   `docs/worker-operator-experience-validation-2026-09-24.md`.
3. In the report, include only sanitized aggregate evidence:
   - Scope and acceptance criteria.
   - Final Windows I/J and Linux G/H identities from this handoff.
   - Packaged E2E run `35f3f52e232067c1` and cleanup/foreign-state result.
   - Production issued/accepted/reconciled counts.
   - Duration percentiles derived from `final-evidence.json`.
   - Watchdog/full-stage evidence from `progress-v3-evidence.json`.
   - Natural timeout evidence from `timeout-evidence.json`.
   - Diagnostic/admin snapshot facts without private content.
   - Sanitized operator command transcript.
   - Known limits, especially no public registry/auto-updater and intentional
     abandoned-root retention.
   - Rollout and rollback/restoration facts, controls revision 126, and final
     healthy state.
4. Re-read `docs/remote-worker-operations.md` against task 6.1. It already covers
   package acquisition/build, Windows preparation, install/first run, lifecycle,
   status/attach/logs/history, phases/deadlines, diagnostics, drain/stop,
   recovery, update, and removal. Make only a minimal correction if the final
   artifact/report facts expose an actual gap.
5. Run strict validation:

   ```powershell
   openspec validate add-worker-operator-experience --strict
   ```

6. If the focused matrix, report, runbook review, and strict validation pass,
   change only task checkboxes 6.1-6.4 in
   `openspec/changes/add-worker-operator-experience/tasks.md` from `[ ]` to `[x]`.
7. Re-run `openspec instructions apply --change "add-worker-operator-experience" --json`
   and confirm progress 27/27 with state `all_done`.
8. Give the user a concise completion result and say the change is ready to
   archive. Do not archive it without an explicit request.

## Report safety checklist

Before saving or quoting the final report, verify it contains none of:

- Tokens or credentials.
- Raw worker targets or findings.
- Runtime YAML or secret environment values.
- The private admin route prefix.
- Worker argv/auth command lines.
- Unbounded log or diagnostic bodies.

Allowed report content includes hashes, aggregate counts, reservation numeric
IDs used as evidence references, phase names, durations/percentiles, safe test
counts, generic command names, and public artifact paths within this workspace.