truf-server/docs/worker-operator-experience-validation-2026-09-24.md

Worker Operator Experience Validation - 2026-09-24

Scope and acceptance

This report closes the release and production-proof work for OpenSpec change add-worker-operator-experience. Validation covered the shared worker event contract, local supervisor and contained runner, progress and diagnostics APIs, admin projections, reproducible Windows and Linux packages, packaged cross-platform operation, bounded production behavior, and final restoration.

Acceptance required:

• the focused unit, integration, protocol, and package matrix to pass;
• independently reproducible Windows and Linux artifacts with documented and registered package manifests;
• packaged Windows/Linux evidence for multi-slot operation, outage and restart recovery, durable bundles and receipts, shutdown, and local cleanup;
• bounded production evidence for progress, a full scan-stage timeout, diagnostics, reconciliation, and restoration; and
• a from-zero operator runbook covering acquisition through removal.

No production assignment was repeated to prepare this report. All production facts below are derived from the retained validation snapshots. Raw targets, findings, credentials, private routes, runtime configuration, and authenticated worker command lines are intentionally excluded.

Focused test matrix

The final 14-file worker-operator matrix completed on 2026-09-25:

355 passed, 3 skipped in 49.54s

The matrix includes worker API/runtime, assignment and contained-runner, CLI/contracts/local state, observability persistence, package, Linux handoff, supervisor, remote database, scan execution, and admin API coverage. The three independent-watchdog timing regressions also passed after their test setup bounds were stabilized. The timing change did not alter product deadlines or watchdog behavior.

An unrestricted repository-wide test run is not a release gate for this change: the checkout has unrelated missing private/generated assets and platform assumptions. The focused matrix, packaged E2E, production evidence, and strict OpenSpec validation are the scoped gates.

Reproducible artifacts

Windows portable package

Final independently built archives:

• build/operator-experience-validation/windows-i.zip
• build/operator-experience-validation/windows-j.zip

Both archives have the following identical identities:

Identity	Value
Archive bytes	134850988
Archive SHA-256	6ea9290736a059f1e17d8e89d9cf83506fa4abe2ba2f3731a7422a7b0f386e97
Package manifest identity	78a962b2bd3fa411413c79e9a8ffb021608a08ff020b1ad851f4505ea634b2b6
Build-input identity	6991ebbce6ae758c2bdd19a6ae934335aa585a50f86b18ccde8d88bca40ce436
Raw worker-package.json SHA-256	e0b17d70fcb868fe39fac45ab6e05a17c6d40852e6034010fb63b6cab31f8a3c

Acceptance used the fresh extraction at build/pwe-final-i-extracted. Its own prepare-worker.ps1 established protected explicit ACLs before direct package verification. Older G/H Windows archives are excluded because their preparation script could leave packaged executables inaccessible.

Linux worker image

Final independently built local tags:

• truf-worker-test:operator-experience-final-3g
• truf-worker-test:operator-experience-final-3h

Both provenance-disabled builds have the following identical identities:

Identity	Value
Worker package identity	45588f2cf406b41b239cfa3b8a9dc83fe84b587229bc997b2729016e1f0dde42
Image manifest / accepted image ID	sha256:3a088f5743121d823aae132234a29730a84339cecbfda5fc601e8e942f9948c3
Config SHA-256	sha256:687a1c4c51c1b962c7fa7ea0cc4b04d159e7ba4f94ef347940c9fb225f7cb87d
Raw worker-package.json SHA-256	ee926cce3c19e9e6094753f51fa902415bd7364c24fa649cd0c1b659c0aa4d60

The retained manifest snapshot is build/operator-experience-validation/linux-worker-package-g.json.

Trusted manifest registration

The accepted Linux and Windows manifests were registered after packaged E2E acceptance. Their remote SHA-256 values match the raw manifest hashes above. Both files are owned by root:root with mode 0644; pre-change backups remain intact and upload temporary files were removed. Registration required no runtime restart or configuration mutation, and canonical health remained successful.

Packaged Windows/Linux E2E

Run 35f3f52e232067c1 passed with the freshly extracted/prepared Windows I package and Linux G image. The safe summary is build/pwe-35f3f52e232067c1/summary.json.

The gate confirmed:

• real packaged Windows and Linux operation at two slots;
• server outage handling and restart recovery;
• durable and direct assignment bundle paths;
• authoritative receipt handling;
• graceful shutdown receipts;
• no active local work after completion while intentionally retained abandoned roots remained inactive;
• matching normalized cross-platform evidence; and
• complete cleanup of owned resources with foreign Docker state unchanged.

Production evidence

Reconciliation

The retained snapshot records 34 issued assignments: 33 accepted and one intentional expected expiry. All 33 accepted bundles were ingested, settled, and projected. Final unresolved, precommit, quarantine, and drain-blocker counts were zero.

Evidence sources:

• build/operator-experience-validation/final-evidence.json
• build/operator-experience-validation/progress-v3-evidence.json
• build/operator-experience-validation/timeout-evidence.json
• build/operator-experience-validation/server-baseline.json

Duration percentiles

The table reports every retained end-to-end metric group. Values are seconds. Sufficient means the server-side minimum sample count of five was met. Rows below that minimum are retained observations, not statistically sufficient percentile estimates.

Platform	Source	Outcome	Samples	p50	p95	p99	Sufficient
Linux	DockerHub	degraded	1	305	305	305	no
Linux	DockerHub	error	1	19	19	19	no
Linux	GitLab	error	1	5371	5371	5371	no
Linux	HuggingFace	expired	1	7219	7219	7219	no
Windows	DockerHub	clean	2	31	31.9	31.98	no
Windows	DockerHub	degraded	4	275.5	443.45	462.29	no
Windows	DockerHub	error	7	619	1679.5	1731.1	yes
Windows	GitLab	clean	3	27	873	948.2	no
Windows	GitLab	error	6	342.5	1723.75	1916.75	yes
Windows	HuggingFace	clean	1	1019	1019	1019	no
Windows	HuggingFace	error	7	971	3092.6	3452.12	yes

The snapshot contains 11 Linux and 21 Windows phase/outcome metric groups in total. Three Windows end-to-end error groups met the minimum; the other 29 phase/outcome groups did not. Rollout decisions must therefore preserve the sample-count qualification rather than treating all reported percentiles as stable capacity estimates.

Progress and watchdog evidence

Reservation 1455 is the retained complete-stage progress reference. It was acknowledged with an accepted bundle and persisted ten monotonic events spanning assigned, preparing, waiting_permit, scanning, filtering, cleaning, bundling, uploading, and awaiting_receipt. This demonstrates one coherent server-visible sequence across the complete local execution and upload boundary.

Independent watchdog fault-injection coverage passed for blocked state persistence, startup-gate persistence, and event draining. The contained runner tests verify bounded process-tree termination rather than relying on scanner cooperation. Packaged E2E additionally passed its watchdog, restart, durable bundle, and cleanup gates.

Natural full-stage timeout

Reservation 1453 is the retained natural timeout reference. The scan ended with one timeout / scan.stage_timeout / scanning diagnostic after 603.367 seconds. The diagnostic was current and available, with no body or process-log payload fabricated for the exception. The end-to-end assignment-resolution duration was 619 seconds.

The scan outcome was error, while the transport outcome was independently accepted: the bundle was acknowledged, the projection completed, and the diagnostic was attached to the authoritative result. This confirms that a hard scan-stage timeout remains a normal, uploadable terminal result and does not collapse scan, transport, and projection outcomes into one status.

Diagnostic and admin snapshots

The final diagnostic aggregate contains five grouped rows and 23 occurrences:

Category	Code	Phase	Occurrences
scanner	scan.result_error	scanning	20
timeout	scan.stage_timeout	scanning	1
assignment expiry	assignment.deadline_expired	assigned	1
network	scan.result_error	scanning	1

The retained snapshots also confirm separate assignment and scan outcomes, ordered progress, current diagnostic availability, accepted receipt state, ingestion/settlement/projection completion, duration metrics, and zero unresolved or precommit work. This is the durable machine-readable substitute for copying private admin pages or unbounded diagnostic bodies into the report.

Sanitized operator transcript

The release and validation sequence was:

1. Build the Windows package twice from the same reviewed inputs and compare the archive, package-manifest, build-input, and raw-manifest identities.
2. Extract Windows I into a fresh directory, run its packaged prepare-worker.ps1, and run direct package verification.
3. Build Linux G and H independently with provenance disabled and compare image, config, package, and raw-manifest identities.
4. Run docker/verify_packaged_workers.py with the accepted Windows extraction, Linux image, and isolated test image; retain only its safe summary and owned evidence directory.
5. Register the two accepted trusted manifests through the reviewed deployment path and recheck canonical runtime health.
6. Use typed operations to bound production dispatch, start at assignment cap 1, observe status/attach/history and server progress, exercise normal, timeout, outage, and restart paths, and reconcile accepted, ingested, settled, and projected counts.
7. Restore standard identities and normal/open controls, disable/revoke temporary validation identities, and recheck runtime and edge health.
8. Run the exact focused pytest matrix recorded above.

Authentication values, worker argv, raw targets/findings, private route names, and runtime configuration are omitted by design.

Known limits

• There is no public ZIP download, image registry, installer, or automatic updater. Release artifacts must move through a trusted channel and match a server-registered manifest.
• Completed runner roots move under top-level work/abandoned and are retained for at least 60 seconds; normal retention maintenance runs every 300 seconds. They are inactive evidence, not live work.
• Most retained percentile groups have fewer than five samples. Their values are useful validation observations but not stable performance baselines.
• Ownership fencing guarantees one authoritative acceptance, not exactly-once physical execution across a long partition and server-side expiry/reissue.
• Local state remains recovery authority until the server resolves the slot. Operators must not remove pending bundles or work trees to clear an alert.

Rollout, rollback, and restoration

Rollout uses the exact accepted package identities, begins with one user/device at server cap 1 and local parallelism 1, and requires one accepted, ingested, settled, and projected assignment before expansion. Caps and client count should increase in stages while unresolved/precommit counts, diagnostic availability, duration sample counts, and authenticated contact remain observable.

Rollback first sets the affected cap to 0, allows pending uploads to resolve, and obtains a graceful shutdown receipt. The operator then returns to the previous exact artifact while preserving the same private state tree or volume. Additive server progress and diagnostic records do not require schema rollback.

Final restored production state:

• operations controls normal/open at revision 126;
• standard WSL production worker user enabled at assignment cap 1;
• standard production device enabled and not revoked;
• temporary validation identities disabled/revoked;
• unresolved, precommit, quarantine, and drain blockers at zero;
• canonical runtime healthy; and
• edge service remained available.

Strict OpenSpec validation passed. The operationally validated change is ready for archival.