truf-server/openspec/changes/add-web-operations-control-plane/HANDOFF.md

# Implementation Handoff

Updated: 2026-09-21

This file preserves implementation context for the active OpenSpec change
`add-web-operations-control-plane`. It is a working handoff, not a normative
specification. The OpenSpec artifacts and `tasks.md` remain authoritative.

## User Direction

- Continue implementation autonomously and pragmatically.
- Avoid overengineering. Ask through the question tool only when a real product
  or architecture choice is unclear.
- Keep changes minimal and scoped to the current OpenSpec task.
- Do not modify or remove unrelated worktree changes.
- Do not touch the production server unless explicitly requested. The only
  previously approved remote host was `sec`; `prod` was explicitly excluded.

## Non-Negotiable Provider Architecture

These decisions are also recorded in the repository `AGENTS.md`.

- The server validates assignment shape and canonical immutable identity.
- Git planning may bind an exact commit.
- Docker planning may resolve a mutable tag to an immutable digest.
- These planning operations are not provider-access proofs.
- The worker is the final authority for real provider access.
- Discovery credentials do not enter direct assignment fields.
- Do not add server-side provider probes, durable public-access proof state,
  proof TTL/freshness migrations, broad worker environment scrubbing,
  credential sandboxes, or post-hoc redaction pipelines without explicit user
  approval and a new OpenSpec requirement/task.
- Worker ambient HOME/XDG/Git/Docker/provider environment belongs to the worker
  operator.
- Permanent provider outcomes include target-scoped auth/access/not-found.
- Retryable provider outcomes include network/rate-limit/provider 5xx failures.

## OpenSpec Progress

- Change: `add-web-operations-control-plane`
- Schema: `spec-driven`
- Completed through task 9.4.
- Current count: 50/58 complete.
- Current task: 9.5 (fixed host installation and production wiring).

## Major Completed Work

### Operations authority and distributed sources

- Added PostgreSQL/SQLite operation-control, operation, and append-only audit
  authority in migration 29.
- Discovery and dispatch pause gates and drain reconciliation are transactional.
- Server producers are discovery-only for GitLab, DockerHub, and HuggingFace.
- Protocol 2 and package schema 3 support exact GitLab, Docker direct, and
  HuggingFace Space assignment capabilities.
- New assignment claims are protocol 2; protocol 1 remains completion-only.

### Removed rejected infrastructure

- Docker anonymous-access proof/preflight machinery was removed.
- Proof columns, claim gates, resolver proof handling, and migration 30 were
  removed; migration count remains 29.
- Broad Docker/HuggingFace child-environment sanitization was removed.
- Docker tag-to-digest immutable planning remains.

### Worker/package and canary verification

- Windows/Linux packaged-worker E2E covers real GitLab scan/recovery plus
  synthetic DockerHub/HuggingFace protocol-2 claim-to-ingestion flows.
- Bounded test-only DockerHub canary covers discovery, enqueue, claim,
  worker-classified permanent/retryable failures, upload, ingestion,
  projection, replay, expiry, and drain.
- Synthetic test transport is test-only. The live public DockerHub canary is
  task 10.5.

### Runtime documents

- Strict bounded UTF-8 YAML loader rejects duplicate keys, including effective
  merge collisions, and exposes content-free errors.
- Combined config/secrets validator enforces exact types, bounds, core profile,
  auth pools/references, package capability evidence, and deployment paths.
- Startup, Supervisor launch, preview, and secrets import use the shared
  validator without replacing existing lifecycle/path security checks.
- Candidate files use fixed paths:
  - `/data/runtime-document-candidates/config.yaml`
  - `/data/runtime-document-candidates/secrets.yaml`
  - `/data/runtime-document-candidates/candidate.lock`
- Candidate storage uses private descriptor-based reads, raw SHA-256 CAS,
  durable atomic replacement, bounded config diffs, and aggregate-only secrets
  diffs.
- Candidate concurrency/error-redaction tests cover stale active/candidate
  revisions, hardlinks/symlinks, interrupted writes, rollback, lock contention,
  cancellation locals, and orphan temporary cleanup.

### Supervisor and operation services

- Control protocol stays schema 1 for compatibility.
- Legacy textual snapshot/command remains for CLI and health compatibility.
- Web-facing paths use only exact typed actions; they never call the generic
  parser or shell.
- Structured runtime/source snapshot covers every configured managed child.
- Typed lifecycle/settings support all practical Supervisor-managed children,
  including dashboard through its separate exact action.
- Bounded log tail accepts only an exact managed-source ID and line count.
- Durable operation methods support apply/restart, managed-source actions,
  worker-admin mutations, and content-free hash-chained audits.

### Web console through task 7.5

- Caddy strips inbound private admin headers and injects authenticated Basic
  username plus private edge marker.
- Backend trusts operator identity only with the private marker and stores it on
  request-local state.
- Shared navigation includes Workers / Dispatch, Overview, Search, Supervisor,
  Logs, Config, and Secrets as pages are implemented.
- Overview uses independently degradable bounded health components.
- Search provides persistent discovery pause/resume and exact producer controls.
- Workers / Dispatch provides dispatch pause/resume, drain controls/progress,
  package compatibility, users, devices, assignments, and deferred requeue.
- Supervisor page exposes structured state and exact typed controls for all
  managed children. Logs page exposes bounded allowlisted tails only.
- Every mutation implemented so far receives trusted actor attribution and a
  durable accepted/terminal audit operation.

## Important Web Decisions

- Admin pages are server-rendered and contain no application JavaScript.
- Every response remains `no-store` with restrictive security headers.
- Paths are relative so the random public Caddy admin prefix is preserved.
- Actor is the raw validated Basic-auth username; no actor form field exists.
- Exact form shapes reject missing, extra, and duplicate fields.
- POST mutations use canonical operation UUIDs embedded by the server.
- Plaintext device tokens appear only once in their direct POST response.
- Config/secrets content must never enter generic file-service scope.
- Worker status/upload/report/receipt endpoints remain independent of admin
  page health.

## Completed Task 7.6

Task text:

> Add config and plaintext secrets preview/save/apply pages with no-store
> rendering, revision/hash conflicts, and no value leakage outside the editor.

### Implemented UI and service behavior

- GET `/admin-internal/config`
- GET `/admin-internal/secrets`
- POST `/{config|secrets}/preview`
- POST `/{config|secrets}/save`
- POST `/{config|secrets}/apply`
- POST `/runtime/apply-both`
- Editors load the candidate when present, otherwise the active document.
- Config editor never receives plaintext secrets.
- Secrets plaintext appears only inside the escaped no-store textarea.
- Preview uses the shared validator and candidate pairing rules.
- Config diff now redacts every changed string value.
- Secrets diff is aggregate-only and contains no pool, entry, username, token,
  hash fragment, or secret length.
- Save stages only a candidate and never activates an active file.
- Production currently has no host-agent apply provider. Apply returns 503
  before operation creation. Host apply belongs to tasks 9.x.
- Save operations are durable and content-free:
  - `runtime.config.save`
  - `runtime.secrets.save`
- Accepted/terminal audit records contain only document name, hashes, byte
  counts, written flag, trusted actor, and fixed outcome/category.

### Active files

- `app/admin_api.py`
- `app/runtime_document_io.py`
- `app/runtime_document.py`
- `app/scanner_db.py`
- `app/worker_api.py`
- `tests/test_admin_api.py`
- `tests/test_runtime_document_io.py`
- `tests/test_operations_service.py`

### Completed replay and cancellation-safety fixes

#### 1. Save replay after the candidate changed length

Resolved problem:

- `_save_runtime_document_candidate_bytes()` previews the current candidate and
  recomputes `candidate_before_bytes` before asking the database to replay the
  existing operation.
- After a successful candidate write, current byte length may differ from the
  originally accepted before length.
- Reusing the same operation UUID then conflicts even though it is the exact
  request replay.

Implemented fix:

- Look up `runtime_operation(operation_id)` before creating a new document
  operation.
- If it exists, validate actor, action, target kind/ref, submitted active
  hashes, selected candidate-before hash, proposed candidate-after raw hash and
  byte count against persisted expected identity.
- Persist and validate the counterpart candidate hash as part of the document
  operation expected identity. The accepted operation must bind all four CAS
  hashes, not only the selected candidate.
- For a running replay:
  - active config/secrets and counterpart candidate must still match accepted
    hashes;
  - if selected candidate still has the accepted before hash, retry the save;
  - if selected candidate already has the accepted after hash, complete the
    operation without writing again;
  - any other state is a 409 revision conflict.
- Do not recompute accepted before-byte count from post-save state.
- Extend the real ScannerDB operation test and make the admin fake compare the
  complete immutable identity, so changed-length replay is covered.

#### 2. Preserve recovery identity on save errors

Resolved problem:

- A 503 `runtime document completion is pending` page renders pre-save state and
  generates a fresh operation UUID.
- Pressing Save from that page starts another operation instead of replaying the
  pending one.

Implemented fix:

- `_render_runtime_document_page()` should accept an optional operation UUID.
- Preview/error responses must keep the submitted operation UUID.
- After save failure/pending completion, reload editor state before rendering so
  hashes represent the physical candidate now on disk.
- The textarea may retain the submitted document only inside the editor.

#### 3. Apply replay after host-side file changes

Resolved problem:

- `request_runtime_apply()` verifies current candidate/active filesystem state
  before checking for an already persisted operation.
- If the host agent changed active files and its response was lost, an exact
  POST retry can fail revision verification instead of recognizing the durable
  operation.

Implemented fix:

- Check `runtime_operation(operation_id)` first.
- Validate actor, action, target kind/ref and persisted expected hash identity
  against the submitted request.
- Terminal success returns without filesystem verification.
- Terminal failure returns 409.
- Requested/running exact replay redispatches the same operation ID/action
  without re-verifying post-apply filesystem state.
- A fresh operation still performs full candidate verification before durable
  operation creation and dispatch.
- Provider dispatch is expected to be idempotent by operation ID; task 9.x host
  agent will enforce persisted-operation/hash checks.

#### 4. Cancellation traceback plaintext cleanup

Resolved problem:

- `asyncio.CancelledError` is a `BaseException` path and can leave raw
  URL-encoded body, decoded fields, or `document_text` in traceback locals.

Implemented fix:

- Add `try/finally` cleanup in `_form_fields()` for body, decoded text, parsed
  pairs, field mapping, and current chunk locals.
- Wrap the document branch of `_dispatch()` in `try/finally` and clear fields,
  hashes, editor, preview, operation ID, and document text locals.
- Add a cancellation test that injects cancellation after decoding and asserts
  the sentinel is absent from all `admin_api` traceback frame locals.

## Task 7.6 Verification

- PyCompile passed for all changed production and test modules.
- Focused runtime-document/admin/operation suites: `96 passed, 1 skipped`.
- Broader admin/runtime/operation/worker/edge/query suites: `179 passed, 1 skipped`.
- Container unit selection: 37 modules, 810 test definitions, no runner skips.
- Desktop/mobile Chrome checks passed for Config and Secrets: no body overflow,
  bounded scrollable textarea, prefix-safe relative navigation, no script,
  local storage or service worker use, and no console errors.
- The secrets sentinel occurred exactly once in serialized HTML and nowhere
  outside the textarea.
- Strict OpenSpec validation passed.
- `git diff --check` passed. The checkout still has no useful tracked baseline.
- Live PostgreSQL was unavailable locally; SQLite service tests and SQL-shape
  checks remain the current database coverage.

## Completed Task 7.7

Task 7.7 added durable operation-status and bounded paginated audit pages that
remain backed by database state across runtime/admin restarts.

### Implementation

- `ScannerDB.runtime_audit_events(before_event_id=None, limit=50)` returns a
  deterministic newest-first keyset page using event ID as the cursor.
- Cursor and page limits are strictly bounded. The query fetches at most one
  extra row to determine whether an older page exists.
- The audit read uses a SQLite transaction or PostgreSQL repeatable-read,
  read-only transaction and loads referenced operations in one bounded batch.
- Only operation-bound audit rows are exposed. Nullable unlinked storage events
  are excluded because they have no typed operation identity safe for the web
  console.
- Every returned event is revalidated against its normalized durable operation:
  actor, action, target, result/category, canonical expected/resulting identity,
  byte counts, parent hash metadata, timestamp, and event hash.
- Added explicit no-store server-rendered routes:
  - `/admin-internal/operations`
  - `/admin-internal/operations/{canonical-operation-uuid}`
  - `/admin-internal/audit`
  - `/admin-internal/audit?before={positive-event-id}`
- Unknown or noncanonical operation IDs return 404 without filesystem or agent
  probing. Duplicate, extra, or malformed query fields fail closed with 400.
- Operation status renders only normalized persisted safe fields and canonical
  expected/resulting identities.
- Audit renders only safe operation-bound event fields and hash identities.
- Apply POST redirects to the durable operation URL after successful dispatch.
- Relative URL roots preserve the random external admin prefix: list/audit pages
  use `.`, operation detail uses `..`, and apply redirects use
  `../operations/{operation_id}`.
- Operations and Audit were added to shared navigation.

### Task 7.7 tests and review

- Admin tests cover operation list/detail, audit pagination, random-prefix URL
  resolution, strict query shapes, unsupported methods, security headers,
  escaping, and absence of editor/Supervisor fixture secrets.
- SQLite service tests cover pagination without overlap, cursor exhaustion,
  operation completion identities, and close/reopen persistence.
- PostgreSQL integration coverage was added for pagination across ScannerDB
  reopen and rejection of audit UPDATE, DELETE, and TRUNCATE. All 59 bundled
  PostgreSQL tests were discovered but skipped locally because PostgreSQL is
  unavailable.
- An independent review found and then verified fixes for relative URL escape,
  nullable unlinked audit rows, and missing PostgreSQL coverage. Its final
  result reported no concrete findings.
- Focused admin/operation suites passed: 54 tests.
- Broader selected admin, runtime-document, operation/control/schema, worker API,
  edge deployment, and production-query checks passed; the unittest modules
  reported 183 tests with one expected Windows skip.
- Container unit selection passed: 37 modules, 814 test definitions, no runner
  skips.
- Desktop and 390px mobile browser checks passed for operation list, operation
  detail, and audit pages. The document had no horizontal overflow; wide tables
  scroll inside their bounded container. All links remained under the admin
  prefix. No scripts, local storage, service-worker controller, fixture-secret
  leakage, console warnings, or console errors were present.
- Strict OpenSpec validation and `git diff --check` passed.

## Completed Task 7.8

Task 7.8 adds admin API and browser coverage for routes, methods, exact form
shapes, actor spoofing, Origin/CSRF, stale forms, random-prefix navigation, CSP,
and secret non-retention.

### Production fixes already implemented

- `_dispatch()` rejects query fields before any state read or mutation. Only
  `GET /admin-internal/audit` accepts the exact optional `before` cursor; every
  other route and method requires an empty query.
- `_AdminRoute(Route)` upgrades Starlette method-only partial matches and calls
  the admin endpoint directly, so arbitrary methods such as `PROPFIND` receive
  the application's secured 405 response instead of an unprotected framework
  response.
- Supervisor controls were compacted without changing the typed backend:
  Dashboard and every managed source use collapsed `<details>` panels. Exact
  action routes, CSRF, operation UUIDs, allowlists, and no-JavaScript behavior
  remain unchanged. No shell, generic command/action field, or parser was added.

### API and edge test work implemented

- `tests/edge_e2e_client.py` now submits canonical operation UUIDs for user
  creation, expects the current 303 redirect, and verifies PostgreSQL operation
  plus accepted/succeeded audit rows attribute the authenticated Basic user,
  not spoofed inbound operator headers.
- A table-driven contract test covers all 45 exact POST routes. Each route
  rejects the wrong method, incomplete fields, and attacker Origin with secure
  headers and no side effects.
- All six stale control forms return secured 409 without a transition.
- Config/secrets preview, save, individual apply, and apply-both now have direct
  valid-route coverage.
- Eighteen stale runtime-document hash cases cover every active, selected
  candidate, and counterpart candidate hash before mutation/operation creation.
- All rendered Search and Supervisor exact actions/settings have successful
  dispatch coverage; semantically invalid keychecks loop mode remains excluded.
- CSRF/Origin tests now cover missing/duplicate CSRF and duplicate/wrong Origin,
  proving only the valid request mutates.
- Static random-prefix crawl covers root, overview, Search, Supervisor, Logs,
  Config, Secrets, Operations, operation detail, and Audit. Every link/form
  remains under the simulated private prefix; every response has strict headers
  and no script/browser-storage API references.
- The fixture secret occurs exactly once inside the Secrets textarea and nowhere
  outside it or on other pages. Form and textarea autocomplete are off.
- `tests/test_admin_browser.py` starts a real local ASGI fixture behind a
  simulated random external prefix and drives Chromium through pinned Playwright
  (`tests/requirements-browser.txt`). Its dedicated lane fails rather than
  silently skipping when Playwright is absent.
- The browser test visits every navigation page, validates exact response
  security headers and inline-script CSP enforcement, rejects unexpected console
  errors, checks desktop/mobile body overflow, and proves every link, stylesheet,
  and form remains under the random external prefix.
- The browser test also proves the plaintext fixture secret exists only in the
  Secrets textarea, no local/session/Cache/IndexedDB/service-worker persistence
  exists, and an unsaved sentinel is absent after a network reload. Native
  browser back-forward memory is not treated as application persistence.

### Final verification state

- PyCompile passed for `app/admin_api.py`, `tests/test_admin_api.py`, and
  `tests/test_admin_browser.py` plus `tests/edge_e2e_client.py`.
- `tests/test_admin_api.py` passed: 45 tests.
- `tests/test_admin_browser.py` passed in real local Chrome: 1 test.
- Broad selected admin/runtime-document/operation/control/schema/worker/edge/
  query suites passed: 190 unittest tests, one expected Windows symlink skip.
- Container selection passed: 37 modules, 821 test definitions, no runner skips.
- Full local edge E2E passed with current rebuilt images. It proved durable
  operation/audit attribution to the authenticated Basic user, spoofed operator
  header stripping, Origin/CSRF behavior, worker-route independence, fail2ban
  behavior, cleanup, and unchanged foreign Docker state.
- Desktop and 390px mobile real-browser checks passed with no body overflow,
  scripts, unexpected console errors, or secret leakage. Wide Audit content
  scrolls only inside its bounded table container.
- An independent read-only review found no remaining concrete task 7.8 defect.
- Strict OpenSpec validation and `git diff --check` passed.

## Completed Task 8.1

Task 8.1 defines the trusted logical-root policy and exact per-root permissions
and limits without opening files or prematurely implementing traversal, file I/O,
HTTP Files pages, or mutation audit behavior from tasks 8.2–8.4.

### Policy and configuration

- Added `app/managed_files.py` with immutable typed root, permission, limit, and
  registry records plus the exact operation enum: list, read, create-replace,
  and delete.
- Root IDs are bounded lowercase logical names; absolute host/container paths
  exist only in trusted server configuration and are hidden from root repr.
- Every root must explicitly provide all four permissions and all six limits:
  relative-path bytes, component bytes, path depth, listing entries, listing
  bytes, and file bytes. Per-root values cannot exceed fixed hard caps.
- The predefined existing roots are `runtime-logs`, `runtime-keychecks`, and
  `runtime-results` at their fixed runtime directories. They are forced to
  list/read only and cannot be renamed or granted mutation permissions.
- `runtime-results` permits files up to 256 MiB so active and rotated
  `scan_results.jsonl` and `found_secrets.jsonl` generations remain
  downloadable. Only active names and exact six-digit generation names are
  visible; locks, databases, ledgers, scan errors, temporary/quarantine
  directories, malformed generations, and recovery artifacts are excluded.
  Keycheck and log roots retain the narrower 64 MiB per-file bound, and custom
  roots cannot opt into the larger result bound.
- Future writable roots are permitted only as immediate children of the isolated
  `/data/managed-files` directory. No writable root is enabled by default.
- This narrow allowlist excludes config/secrets, candidates, PostgreSQL,
  application code, sockets/control state, host-agent metadata, raw result
  bundles/spool, runtime state/queues/caches/work, and imported archives in both
  normal and parent-container directions. Projected results and keycheck output
  are available only through their fixed read-only roots.
- Root mappings are deterministically ordered, duplicate paths are rejected,
  nested fields are exact, booleans cannot pass integer limits, and errors do
  not echo submitted IDs or paths.
- Missing roots mean an empty registry. Existing explicit `admin: null`
  compatibility remains an empty disabled admin configuration, while a present
  `managed_file_roots: null` or any other malformed section fails closed.

### Runtime wiring

- `app/config.linux.yaml` defines the read-only `runtime-logs`,
  `runtime-keychecks`, and `runtime-results` roots with bounded deployment
  defaults.
- `app/runtime_document.py` treats root IDs as a strict dynamic mapping, validates
  the policy even while admin is disabled, preserves omission compatibility, and
  includes the new dynamic shape in the pinned template schema hash.
- `app/worker_api.py` validates roots before DSN/secrets/assignment-builder work
  and passes the immutable registry into `AdminService`.
- `AdminService` accepts only a `ManagedFileRootRegistry` and otherwise defaults
  to an empty registry. No `/files` navigation or route exists yet; that belongs
  to task 8.4.
- Task 8.1 performs no `open`, stat, path resolution, directory creation, or
  network work. Descriptor opening and target traversal remain task 8.2.

### Verification

- Added `tests/test_managed_files.py`; 8 policy/configuration tests passed.
- Runtime-document validation passed: 22 tests.
- Worker runtime wiring passed: 12 tests.
- Admin API passed: 46 tests.
- Runtime-document I/O regressions passed: 25 tests with one expected Windows
  symlink skip.
- Container selection passed: 38 modules, 832 test definitions, no runner skips.
- Strict OpenSpec validation and `git diff --check` passed.
- Independent read-only review found no remaining concrete correctness or
  security findings.

## Completed Task 8.2

Task 8.2 adds the Linux descriptor-containment layer. It deliberately does not
yet enumerate directory entries, return file bytes, mutate files, add HTTP
routes, or write audit events; those belong to tasks 8.3 and 8.4.

### Canonical path and descriptor model

- `parse_managed_relative_path()` requires an exact UTF-8 string and returns
  unchanged components only after enforcing the configured byte, component, and
  depth limits.
- Empty, absolute, drive-qualified (including nested drive components),
  backslash, NUL, repeated-separator, trailing-separator, dot, and dot-dot paths
  fail before target traversal.
- `ManagedFileTraversal` is Linux-only for nonempty registries and fails closed
  unless `O_PATH`, `O_DIRECTORY`, `O_NOFOLLOW`, `O_CLOEXEC`, descriptor-relative
  `os.open`, and the required nonblocking flags are available.
- Each configured absolute root is opened from `/` one component at a time with
  `dir_fd`, `O_PATH | O_DIRECTORY | O_NOFOLLOW`, and retained for the traversal
  lifetime. Symlinked root or parent components cannot be followed.
- Each operation duplicates the retained root under a lock before walking. This
  prevents concurrent `close()` and descriptor-number reuse from redirecting a
  request; already-started operations remain anchored after close.
- Intermediate client components use the same `O_PATH | O_DIRECTORY |
  O_NOFOLLOW` traversal. A renamed/replaced configured pathname does not change
  the retained root object.
- Listing opens only a verified directory descriptor. `None` is the internal
  root-list sentinel; an empty client path remains invalid.
- Read targets are first opened with `O_PATH | O_NOFOLLOW`, checked as a
  single-link regular inode, then reopened through their own `/proc/self/fd`
  descriptor and revalidated by type, link count, device, and inode before a
  readable descriptor is returned. Devices/FIFOs/sockets therefore are not
  opened for reading before type rejection.
- Opened targets are context-managed; partial root walks, operation walks,
  ordinary exceptions, and `BaseException` paths close every descriptor they
  unambiguously own. `close()` is lock-safe and idempotent; use-after-close
  fails closed.
- Errors expose only bounded categories such as `invalid_path`, `unknown_root`,
  `operation_not_allowed`, `not_found`, `unsafe_target`, `root_unavailable`,
  `filesystem_unavailable`, and `closed`. They never echo client paths, root
  paths, errno text, or filenames.

### Adversarial coverage and packaging

- Traversal tests cover canonical ASCII/Unicode parsing, UTF-8 byte accounting,
  exact no-follow/dir-fd flags, partial-constructor and operation cancellation,
  close/open races, permissions, unknown roots, non-Linux behavior, and procfs
  failure classification.
- Real Linux tests cover nested list/read descriptor opens, root rename/path
  replacement anchoring, root/intermediate/final symlinks, hardlinks,
  directory-as-file, FIFO, Unix socket, component swap after intermediate open,
  idempotent close, and use-after-close.
- `.dockerignore` now admits only the new managed-file module/test explicitly,
  and `container_unit.py` selects the policy, mock traversal, and Linux kernel
  traversal classes.

### Verification

- Native managed-file suite: 17 tests, 13 passed and 4 expected Linux-only
  skips.
- WSL Linux managed-file suite: 17/17 passed.
- Rebuilt read-only Linux test image: 17/17 managed-file tests passed under UID
  10001 and the container audit fences.
- Related runtime-document, worker-runtime, admin, and runtime-document-I/O
  suites passed: 105 tests with one expected Windows symlink skip.
- Container selection passed: 38 modules, 841 test definitions, no runner skips.
- Strict OpenSpec validation and `git diff --check` passed.
- Final independent security review found no concrete task 8.2 findings.

## Completed Task 8.3

Task 8.3 adds the bounded listing/download and durable file-mutation service on
top of task 8.2 descriptor containment. It deliberately adds no HTTP route,
operator form, durable operation row, or audit event; those belong to task 8.4.

### Listing and download

- `list_directory()` lazily scans an already verified directory descriptor,
  counts every encountered entry against the configured work limit, bounds
  returned UTF-8 name bytes, and sorts accepted logical names deterministically.
- Listings expose only directories and bounded single-link regular files.
  Symlinks, hardlinks, special files, over-limit files, invalid names, and
  internal temporary names are omitted without revealing host paths.
- `download_file()` reuses the `O_PATH`/`/proc/self/fd` inode-safe open, rejects
  oversized files before reading, reads at most the configured bound, computes
  SHA-256 while reading, and verifies stable inode/type/link/size/mtime/ctime
  metadata before returning content.
- Public result records expose logical names, file/directory kind, bounded byte
  counts, hashes, and content only on the direct download result. Download
  content and host descriptors remain hidden from repr.

### Durable create/replace/delete

- Client relative components beginning with the reserved internal temporary
  prefix are rejected; the prefix is also hidden from listings.
- Create/replace accepts exact `bytes` up to the root file limit. `None` expected
  hash means create-only; a canonical lowercase SHA-256 means replace-only.
- Temporary files are created with random same-directory names using
  `O_RDWR | O_CREAT | O_EXCL | O_NOFOLLOW | O_CLOEXEC`, start private, are
  hardened to exact mode `0600`, and must be regular, single-link, and owned by
  the effective runtime UID.
- Writes handle short writes/interruption, verify exact size and SHA-256, fsync
  the temporary inode, and retain its descriptor through publication.
- Create publishes atomically with Linux `renameat2(RENAME_NOREPLACE)`, so an
  existing name is never overwritten. Replace revalidates the expected inode
  and hash before descriptor-relative `os.replace`; readers observe complete old
  or complete new files.
- Every mutation takes both an in-process lock and an advisory exclusive flock
  on the opened parent-directory inode. Separate traversal instances/processes
  using this service therefore implement one cooperative hash-CAS authority.
  Writable roots are dedicated immediate children of `/data/managed-files`;
  arbitrary SSH/root writers that ignore the lock are outside this contract.
- Namespace publication and unlink run inside a `finally`-protected parent
  directory fsync. A failure after visible mutation is reported only as bounded
  `durability_uncertain`/`concurrent_change`; the service never attempts an
  unsafe rollback over a later writer.
- Final published files are re-read and verified for expected hash/bytes,
  original staged inode, effective UID, exact `0600`, regular type, and one
  link. The parent descriptor is then revalidated.
- Replace with identical expected/proposed identity is an idempotent no-write.
- Delete requires the exact current SHA-256, revalidates the named inode,
  unlinks descriptor-relatively, and fsyncs the parent directory.
- Prepublication failure and `BaseException` paths unlink and directory-fsync
  the known temporary name before potentially interrupted descriptor close.
  Explicit `LOCK_UN` prevents an interrupted parent close from retaining the
  mutation lock. Cancellation is never hidden by an ordinary conflict/error.
- Errors remain content/path/errno-free categories such as `invalid_hash`,
  `invalid_content`, `limit_exceeded`, `hash_conflict`, `concurrent_change`,
  `durability_uncertain`, and the task-8.2 access categories.

### Verification

- Native Windows managed-file suite: 28 tests, 13 passed and 15 expected
  Linux-only skips.
- WSL Linux managed-file suite: 28/28 passed, including real `renameat2`, flock,
  inode/mode/owner checks, fsync, cleanup, and cross-instance concurrency.
- Rebuilt read-only Linux test image: 28/28 managed-file tests passed under UID
  10001 with network disabled; 1048 unrelated tests were deselected.
- Cross-instance and independent-process replace/replace and replace/delete
  probes produced exactly one winner.
- Related runtime-document, worker-runtime, admin, runtime-document-I/O, and
  operation-service suites passed: 121 tests with one expected Windows symlink
  skip.
- Container selection passed: 38 modules, 852 test definitions, no runner skips.
- Final independent security review found no concrete task 8.3 findings.

## Completed Task 8.4

Task 8.4 exposes the descriptor-safe managed-file service through exact
server-rendered admin routes and binds every accepted mutation to a durable,
content-free operation and audit lifecycle.

### Files pages and forms

- Shared admin navigation now includes `Files`.
- `GET /admin-internal/files` renders either the logical-root index or a bounded
  root/nested listing selected by exact `root_id` and optional canonical
  `relative_path` query fields.
- `GET /admin-internal/files/download` requires exact logical root/path fields
  and returns `application/octet-stream` with no-store/security headers, exact
  length, SHA-256 ETag, and a logical-leaf attachment filename. Ordinary files
  retain the bounded in-memory response. An allowed result projection is first
  copied and hashed in 64 KiB chunks into an anonymous stable snapshot on the
  same data volume, then streamed from that snapshot with bounded memory.
  Source revision/size changes retry once and then fail closed; only one result
  snapshot may exist at a time, and response completion/failure releases it.
  No host path is passed to a response object.
- Async request cancellation retains cleanup ownership until a background
  snapshot build finishes, while the streaming response closes the snapshot in
  a response-level `finally`; disconnects and ASGI send failures therefore
  cannot strand the sole result-download permit.
- Exact POST routes are `/files/create`, `/files/replace`, and `/files/delete`.
  They accept only CSRF, operation UUID, logical root ID, canonical relative
  path, canonical expected hash where required, and canonical URL-safe Base64
  content for create/replace.
- The existing bounded URL-encoded admin body is the explicit web-upload cap;
  the page displays that bound. No multipart parser, JavaScript, command field,
  actor field, absolute root, or generic action field was added.
- Root pages expose only logical IDs, permissions, configured limits, safe
  relative names/kinds/byte counts, and permission-appropriate forms. Private
  configured absolute roots and arbitrary file bytes never enter HTML.
- Query and form shapes reject duplicate, missing, extra, partial, empty, or
  noncanonical values before filesystem or database mutation.
- Managed-file errors map only to bounded HTTP outcomes (400/403/404/409/413/
  503) and never expose paths, errno text, content, or raw exceptions.

### Durable mutation authority

- ScannerDB recognizes exact actions `files.create`, `files.replace`, and
  `files.delete` with target kind `managed-file` and logical root ID as the
  bounded target reference.
- Canonical expected identity contains only root ID, relative path, expected
  SHA-256, proposed SHA-256, and proposed byte count. Successful resulting
  identity contains only before/after hashes and counts, outcome, and written
  flag. No migration was required; migration count remains 29.
- `create_runtime_managed_file_operation()` atomically commits a running
  operation and accepted audit before physical mutation.
- `complete_runtime_managed_file_operation()` commits one terminal audit and
  safe resulting identity, or the fixed content-free failure category
  `managed_file_mutation_failed`.
- Uploaded bytes, Base64, CSRF/Origin/auth data, credentials, absolute roots,
  exception text, and rendered content cannot be accepted by the operation or
  audit APIs.
- List and download remain read-only and create no operation/audit rows.

### Replay and execution serialization

- Each mutation holds a per-operation execution claim on a dedicated DB
  connection across terminal lookup, preflight, acceptance, physical CAS, and
  terminal completion.
- PostgreSQL uses a session advisory lock keyed by the canonical operation UUID.
  SQLite uses deterministic process-global lock stripes for local/test
  connections. Different operation IDs targeting one path remain serialized by
  task 8.3 parent-directory flock and hash CAS.
- Lock order is operation advisory claim, AdminService local lock, then traversal
  parent-directory lock. Release and connection close run through unconditional
  nested cleanup even under `BaseException`; cancellation remains primary and
  uploaded payload references are cleared.
- Exact terminal success validates immutable operation identity and returns
  without requiring current traversal/root availability. Terminal failure
  requires a new operation UUID.
- A fresh request validates current root permission/path and expected namespace
  state before accepted audit creation. Create requires absence; replace/delete
  require the exact expected hash. A fresh CAS loss always fails even if another
  writer produced matching bytes.
- A running exact replay may retry only from its accepted before state. It may
  complete without a second mutation only after observing the accepted desired
  after-state (or stable deletion), fsyncing the parent directory, and
  revalidating the same named inode/revision or absence after fsync.
- Matching replayed create/replace after-state must also be a private regular
  single-link file owned by the runtime UID with exact mode `0600`.
- Content-bearing parser, dispatcher, service, and traversal frames clear or
  release URL-encoded, Base64, decoded payload, and memory-view locals on normal,
  error, and cancellation paths.

### Traversal lifecycle

- Worker API lifespan constructs exactly one retained `ManagedFileTraversal`
  when admin is enabled and reuses it for all Files requests.
- Root-open failure safely degrades Files to 503 without taking down worker
  claim/status/upload/report routes and logs only a bounded category/type.
- Nested shutdown cleanup clears app state and closes the traversal exactly once
  even when assignment-reaper shutdown fails. Admin-disabled apps never open
  managed roots.

### Verification

- Focused host suites passed: operation service 19/19, admin API 54/54, Worker
  API 35/35, and real-browser admin coverage 1/1.
- Managed-file tests passed 30/30 on WSL Linux; native Windows ran the same 30
  with 17 expected Linux-only skips.
- Related runtime-document, runtime-document-I/O, worker-runtime, operation
  control/schema, edge-deployment, and production-query suites passed; the only
  skip was the expected Windows symlink case.
- Real Chromium covered Files navigation under the random prefix, security/CSP,
  no scripts/storage, mobile layout, and console cleanliness.
- Rebuilt read-only/no-network Linux test image passed all 46 selected
  managed-file/admin/operation/lifecycle tests under UID 10001; 1046 unrelated
  tests were deselected.
- Container selection passed: 38 modules, 868 test definitions, no runner skips.
- SQLite two-connection execution serialization and cancellation cleanup have
  dedicated regressions. A PostgreSQL two-session advisory-block/unlock/
  session-close integration test is committed and discovered, but skipped
  locally because bundled PostgreSQL is unavailable.
- Strict OpenSpec validation and `git diff --check` passed.
- Final independent security review found no concrete task 8.4 findings.

## Completed Task 8.5

Task 8.5 closes the managed-file service with adversarial traversal, link,
special-file, limit, concurrency, durability, transport-encoding, and
forbidden-root coverage.

### Production hardening

- Nested listings validate the full logical child path, not only the leaf name,
  before exposing an entry. Children beyond the configured path-byte or depth
  bound remain hidden.
- Temporary-file cleanup fsyncs the parent directory even when the allocated
  temporary name is already absent, preserving durable cleanup evidence.
- Admin dispatch requires byte-valued ASGI `raw_path` to be the exact canonical
  ASCII encoding of the decoded fixed route. Missing or percent-encoded route
  aliases fail with a secured 404; query and form values still decode exactly
  once through their normal parsers.
- Download stability again requires an unchanged regular single-link inode,
  including device, inode, size, link count, mtime and ctime. On a concurrent
  atomic replacement, download performs at most one canonical reopen and strict
  reread, so callers receive one complete version rather than mixed bytes.

### Adversarial coverage

- Policy tests explicitly exclude active secrets, candidate documents,
  PostgreSQL, host-agent metadata, Docker/runtime sockets, application code,
  raw bundles, spool/results/state/queues/cache/work, and imported archives.
- Portable tests prove FIFO, socket, character-device, block-device and
  directory modes are rejected before a readable `/proc/self/fd` reopen.
- Real Linux tests cover root and final symlink swaps, retained-descriptor
  anchoring, hardlinks, directories, FIFOs, hidden unsafe listing entries, and
  rejection by download/replace/delete.
- Exact and one-over tests cover total path bytes, component bytes, depth,
  listing entry/name-byte limits, nested logical paths, and file download/upload
  limits.
- Two traversal instances prove one winner for create/create and delete/delete;
  prior replace/replace and replace/delete tests remain. Reader replacement
  tests prove complete-version behavior, including a hostile in-place write,
  restored mtime, and atomic pathname replacement.
- Durability fault tests cover temporary inode fsync, already-absent temporary
  cleanup, cleanup unlink/fsync failures, and replace/delete parent-fsync
  uncertainty while asserting the resulting visible namespace.
- HTTP tests cover encoded and mixed-case traversal, backslash, NUL, drive
  prefixes, exact one-pass double decoding, encoded fixed-route aliases,
  forbidden logical roots, generic action/command/path/service field injection,
  and exact/over/streaming request-body limits before decode, traversal, or
  operation/audit creation.

### Writer authority boundary

- Writable roots are dedicated managed-service roots. Cooperating app instances
  and processes serialize mutations with the parent-directory advisory flock
  and hash CAS.
- Arbitrary SSH/root writers that deliberately ignore that flock are outside
  this authority contract; POSIX has no atomic primitive for
  `replace/unlink iff current content hash equals X` against such writers.
- The service still detects bounded concurrent drift wherever possible and
  never follows a swapped symlink target.

### Verification

- Managed-file tests passed 37/37 on WSL Linux; native Windows ran the same 37
  with 23 expected Linux-only skips.
- Native scoped review reported 71 passed plus the expected Linux skips, and the
  admin API adversarial suite passed.
- Broad runtime-document, operation/control/schema, Worker API/runtime,
  production-query, edge-deployment, and real-browser suites passed; the only
  broad skip was the expected Windows symlink case.
- Container selection passed: 38 modules, 878 test definitions, no runner
  skips.
- Rebuilt `truf-worker-test:task-8-5`; a read-only, no-network Linux run under
  UID 10001 passed all 55 selected managed-file/admin/operation/lifecycle tests,
  including all 37 Linux managed-file tests; 1047 unrelated tests were
  deselected.
- Independent final security review found no concrete in-scope correctness or
  security findings.
- Strict OpenSpec validation and `git diff --check` passed before artifact
  closure.

## Completed Task 9.4

Task 9.4 adds one fixed automatic rollback attempt, durable terminal evidence,
and a global failed-hold fence without adding request-selected lifecycle or file
surfaces.

### Byte-identical rollback

- Stopped-runtime proofs are operation-bound, purpose-bound (`forward` or
  `rollback`), uniquely issued, and single-use. Forward proof cannot authorize
  restoration and rollback proof cannot authorize candidate publication.
- `HostApplySession` retains authoritative original snapshots separately from
  observed active/candidate state and classifies publication as `original`,
  `partial`, or `candidate`.
- Exact mixed old/candidate replay is recovery-only. It never reapplies the
  candidate deployment.
- `restore_backups()` accepts no paths or payloads. It rereads fixed root-owned
  operation backups, accepts active bytes only when exactly original or exact
  candidate, stages byte-identical originals, rechecks CAS, atomically restores,
  adopts fixed runtime ownership/mode, fsyncs, and verifies the complete original
  pair. Backups and candidates remain as evidence.
- Exact root-owned originals left by interruption between rollback rename and
  ownership adoption are recognized and safely adopted on replay. Any unknown
  active bytes fail closed without overwrite.
- Publication state becomes `partial` before the first restoration mutation, so
  a failed second restore cannot be misreported as a complete candidate state.

### One-attempt recovery and failed hold

- Lifecycle execution retains a private mutable attempt before the first stop,
  including the original attested image/config identities and every observed
  stop/remove/recreate milestone.
- Any failure after lifecycle mutation quiesces only the exact fixed edge and
  runtime containers, restores selected backups, and recreates/verifies the
  original attested runtime and edge exactly once with the same strict aggregate
  health gates.
- Successful recovery emits `rolled_back` with both original active hashes and
  the bounded forward failure category. It never reports apply success.
- Recovery failure writes the global failed-hold marker before one containment
  stop, preserves operation/backups/candidates, emits `failed_hold`, and performs
  no third deployment attempt, restoration loop, or forced authority release.
- A global marker fences every different operation. The same operation may only
  finish a missing `failed_hold` phase/result; it skips document preparation and
  cannot resume forward or rollback lifecycle work.
- Pre-mutation validation/identity failure records `failed` without downtime or
  rollback.

### Durable phase and result evidence

- New `app/host_agent_state.py` uses only fixed operation/result/hold paths under
  `/var/lib/truf/host-agent`, canonical bounded JSON, no-follow stable reads,
  root metadata checks, same-directory atomic replacement, file fsync, directory
  fsync, and exact-byte idempotent replay.
- Closed phases are `prepared`, `forward_started`, `rollback_started`,
  `succeeded`, `failed`, `rolled_back`, and `failed_hold`; transitions bind the
  exact operation UUID, action, four request hashes, publication state, bounded
  category/detail, and containment evidence.
- The terminal phase is persisted before the immutable seven-field ScannerDB-
  compatible result. Missing results are deterministically reconstructed from a
  terminal phase without lifecycle mutation.
- A write that renamed phase evidence but could not confirm parent durability is
  classified `uncertain`. Terminal uncertainty never rolls back or contains an
  already healthy deployment; nonterminal rollback uncertainty durably fences
  and contains instead of retrying.
- `KeyboardInterrupt` and `SystemExit` propagate directly before mutation. After
  mutation, the one rollback/failed-hold safety path runs first and the original
  cancellation is still propagated, including result-write and phase-rename
  uncertainty windows.

### Verification

- Lifecycle tests passed 32 selected cases in the final Linux container and
  native Windows passed with four expected POSIX-only skips.
- Host apply passed 23/23 under WSL root; native passed 18 with five expected
  root/POSIX skips. Durable state passed 6/6, protocol passed 15/15.
- Broad operations, container-runtime, runtime-document/I/O, admin, Worker API,
  worker-runtime, and edge-deployment regressions passed.
- Container selection passed: 43 modules, 968 test definitions, no runner skips.
- Rebuilt `truf-worker-test:task-9-4`; a read-only, no-network container run as
  UID 10001 passed 76 selected host-agent tests with two expected distinct-root-
  ownership skips; 1119 unrelated tests were deselected.
- Independent security/correctness review completed repeated crash/cancellation
  review rounds and returned no findings after terminal ordering, early fencing,
  replay, uncertainty, and cancellation fixes.
- Strict OpenSpec validation and `git diff --check` passed before artifact
  closure.

## Completed Task 9.1

Task 9.1 establishes the authenticated, closed privileged-agent request
boundary without implementing file replacement, restart, health, or rollback.

### Protocol and client

- `app/host_agent_protocol.py` defines a strict canonical UTF-8 JSON protocol
  framed by a four-byte big-endian payload length.
- Requests are bounded to 1024 payload bytes and contain exactly operation UUID,
  one of `apply-config`, `apply-secrets`, `apply-both`, or `restart`, and the
  four active/candidate hash fields used by ScannerDB expected identity.
- Canonical nonzero UUID, lowercase SHA-256 values, and the action-specific
  candidate-null/hash matrix are enforced. Duplicate, missing, extra, reordered,
  whitespace-altered, non-finite, or incorrectly typed fields fail closed.
- No path, service, unit, command, argv, environment, Docker/Compose argument,
  timeout, actor, content, or generic options field exists.
- Each connection carries exactly one request and one response. The client
  half-closes its write side; the server requires EOF, rejecting trailing or
  pipelined bytes. Absolute monotonic deadlines prevent trickle extension.
- `app/host_agent_client.py` has no configurable socket path and uses only
  `/run/truf/host-agent.sock`. It requires a root-owned socket pathname and
  kernel `SO_PEERCRED` proving the connected server UID is root; there is no
  retry loop.

### Root agent boundary

- `app/host_agent_server.py` checks kernel peer credentials before reading any
  body and permits only the fixed runtime UID 10001. GID is intentionally not
  part of the identity contract.
- The server validates an inherited systemd socket as exact `AF_UNIX`, exact
  `SOCK_STREAM`, listening, fixed-path, socket-typed, and root-owned.
- `deploy/host-agent/truf_host_agent.py` requires root, no command-line
  arguments, and exactly one systemd-activated listener.
- The production task-9.1 handler always returns `unavailable`. It never claims
  `accepted` before task 9.2 has verified the persisted operation and durably
  assumed ownership.
- Systemd units, host path installation and socket permissions remain task 9.5;
  the agent executable is ready for that fixed activation contract but no unit
  was added early.

### Runtime integration boundary

- Admin apply provider calls now pass the persisted expected identity fields:
  active config/secrets hashes and candidate config/secrets hashes, in addition
  to operation ID and action.
- Production Worker API still installs no apply provider. An absent agent
  therefore preserves the existing 503 before operation creation. Wiring is
  deferred until persisted verification/execution exists; no request can
  currently trigger privileged lifecycle work.

### Verification

- Portable protocol/client/server tests passed 15/15, including exact schemas,
  candidate matrix, framing bounds, duplicate/extra fields, absolute trickle
  deadlines, response pipelining, cancellation cleanup, unauthorized-before-
  read ordering, listener type/listening checks, and inherited-FD cleanup.
- Real Linux tests passed 2/2 under fixed UID 10001 using kernel `SO_PEERCRED`.
- Rebuilt `truf-worker-test:task-9-1`; a read-only, no-network container run as
  UID 10001 passed all 17 selected host-agent tests, including the real Linux
  peer path; 1102 unrelated tests were deselected.
- Broad admin 57, operations 19, Worker API 35, worker-runtime 12, edge-static,
  and provider-hash-binding regressions passed.
- Container selection passed: 40 modules, 895 test definitions, no runner
  skips.
- Independent security review found no concrete task-9.1 correctness or
  security findings after exact `SO_TYPE`/`SO_ACCEPTCONN` hardening.
- Strict OpenSpec validation and `git diff --check` passed before artifact
  closure.

## Completed Task 9.2

Task 9.2 adds a fail-closed execution core for one persisted runtime apply under
fixed host paths. It remains deliberately unwired until task 9.3 can stop the
runtime before publication and task 9.5 installs protected DB/path authority.

### Persisted operation claim

- `ScannerDB.claim_runtime_operation_execution(...)` atomically binds the
  canonical operation UUID, action, runtime-deployment target, and exact four
  active/candidate hashes to the requested-to-running transition.
- The method uses the existing SQLite write transaction or PostgreSQL control
  and operation row locks. The host session separately requires PostgreSQL;
  SQLite support exists only for deterministic database tests.
- Requested/pending operations can be claimed once. An exact running/running
  operation replays; unknown, mismatched, terminal, or incoherent rows fail.
- Claim requires exactly one canonical accepted audit for the operation and
  verifies its actor/action/target/identity/time, global predecessor link, the
  predecessor's own payload hash, and the accepted event hash before transition.
- Schema-valid unlinked audit predecessors remain supported. No schema migration
  or new authority table was added.

### Fixed host apply session

- `app/host_agent_apply.py` defines only fixed constants: active config/secrets,
  candidate config/secrets, root apply lock, operation-ID backup directory, and
  the trusted packaged config template. Socket requests cannot provide paths,
  commands, services, environment, or options.
- `HostApplySession` takes the root-private nonblocking singleton filesystem lock
  before claim and keeps it through validation, backup, publication, and later
  task-9.3 lifecycle insertion points. The lock survives PostgreSQL shutdown.
- The session requires PostgreSQL authority, claims the exact persisted request,
  stable-reads regular single-link active/candidate/template files with nofollow,
  enforces fixed owner/group/mode and raw hash CAS, selects the action-specific
  effective pair, and calls the shared strict runtime-document validator.
- Package capability evidence is host-supplied trusted input, never a socket
  field. Production remains unwired until task 9.5 provides fixed protected
  package/DB/path mappings.

### Backup, publication, and replay

- Selected active bytes are backed up byte-for-byte under the canonical
  operation UUID using exclusive creation, root-only metadata, file fsync,
  reread/hash verification, and parent-directory fsync. Existing backups are
  accepted only when their bytes and identity exactly match.
- All selected replacements are staged and fsynced before publication. Stages
  remain root-owned until same-directory atomic rename, then receive fixed
  runtime ownership/mode and are fsynced again; the runtime UID cannot mutate a
  predictable stage before publication.
- Active and candidate snapshots are rechecked immediately before publication.
  On POSIX, the displaced active inode is retained and reread after rename, so a
  late in-place writer becomes a conservative partial result rather than silent
  data loss.
- Candidates and backups remain as evidence. A stale fixed stage from the same
  operation is durably removed before retry.
- Exact running replay distinguishes not-published, completely-published, and
  partially-published state. Complete replay requires exact backups, candidates,
  and active bytes, recovers a root-owned post-rename/pre-adoption file, and
  revalidates again at `replace()`. Partial replay requires valid backups and
  fails closed for task 9.4 recovery.
- Each config/secrets rename is atomic; the pair is not a filesystem transaction.
  A failure after any publication is categorized `partial`; task 9.2 does not
  invent rollback or failed-hold behavior ahead of task 9.4.
- `restart` still claims and validates the active pair but creates no backup and
  publishes no file.

### Production boundary

- `deploy/host-agent/truf_host_agent.py` was not wired to this core and still
  returns `unavailable`. Current deployment has no protected host PostgreSQL
  endpoint, fixed mounts/permissions, or safe stop/recreate sequence.
- Task 9.3 must guarantee stopped-runtime publication and perform fixed
  recreation/health checks. Task 9.4 owns rollback. Task 9.5 owns installation,
  host DB/path authority, systemd units, and permissions. Task 9.6 owns the full
  crash/fault matrix.

### Verification

- Host apply tests passed 17/17 under WSL root with distinct root/runtime
  ownership; native Windows passed with four expected POSIX-only skips.
- Operation tests passed 25/25, including exact claim/replay, mismatch,
  accepted/predecessor corruption, valid unlinked predecessor, missing, and
  terminal cases.
- Broad protocol 15, runtime-document 22, runtime-document I/O 25, admin 57,
  Worker API 35, worker-runtime 12, and edge deployment suites passed.
- Container selection passed: 41 modules, 918 test definitions, no runner skips.
- Rebuilt `truf-worker-test:task-9-2`; a read-only, no-network container run as
  UID 10001 passed 33 selected host-agent tests with one expected root-only
  ownership-recovery skip; 1108 unrelated tests were deselected.
- Independent security/correctness review found no findings after root-owned
  staging, displaced-inode detection, durable replay, and predecessor-hash fixes.
- Strict OpenSpec validation and `git diff --check` passed before artifact
  closure.

## Completed Task 9.3

Task 9.3 adds a fixed, fail-closed runtime/edge lifecycle and bounded aggregate
health verification. It remains deliberately unwired until rollback and
installation authority are complete.

### Fixed lifecycle authority

- `app/host_agent_lifecycle.py` accepts no production paths, services, commands,
  environment, or timeouts. It uses only `/usr/bin/docker`, project
  `truf-docker`, the two fixed Compose files, fixed edge environment, and the
  `runtime` and `edge` services.
- Preflight validates the fixed Compose model, captures immutable runtime/edge
  image IDs, resolves exactly one container per service, and inspects only a
  bounded selected projection. It never reads container environment, full state,
  health logs, or host mount source paths.
- Attestation binds IDs, images, users, entrypoint/command, runtime healthcheck,
  Compose labels/config hash, capabilities/security options, read-only and
  privileged state, namespaces, restart policy, resource limits, stop policy,
  log driver, destination-only mounts, ports/tmpfs, and absence of device access.
- Edge stops first with a fixed 30-second grace; runtime then receives the fixed
  600-second coordinated stop. Each captured container must prove exited, PID 0,
  exit 0, non-OOM, non-restarting/non-dead, and unchanged restart count.
- Immediately before publication, lifecycle re-resolves both service IDs,
  revalidates their stopped state and immutable image tags, and issues a private
  operation-bound stopped proof. `HostApplySession.replace()` rejects absent,
  forged, or wrong-operation proof.
- The executor removes only the stopped edge and runtime containers, recreates
  runtime with `--no-deps --no-build --pull never --force-recreate`, requires a
  new exact identity, waits under one bounded deadline, and recreates edge only
  after strict runtime health. Edge likewise requires a new exact identity,
  fixed network namespace, successful fixed Caddy validation, and bounded stable
  running observation.
- No `down`, kill, build/pull, volume/orphan, provision, systemd, fail2ban,
  request-selected action, automatic recreation retry, rollback, or terminal
  result reconciliation was added.

### Document and execution ordering

- `execute_fixed_forward()` keeps the task-9.2 singleton lock across backup,
  deployment preflight, pre-stop CAS, stop proof, publication, recreation, and
  health verification.
- `HostApplySession.revalidate_for_stop()` rechecks active/candidate identities
  after backup/preflight and before downtime; `replace()` repeats validation
  after clean stop and immediately before publication.
- Restart follows the same fixed stop/recreate/health sequence but does not back
  up or publish documents. Complete publication replay recreates and verifies
  once rather than treating file presence as runtime health evidence.

### Bounded aggregate health

- `container_runtime.py health --require-worker-api` preserves all existing
  authenticated Supervisor ACTIVE, PostgreSQL READY/schema/cutover/PG16/storage,
  and instance-bound ingester/projector lease checks, while requiring Worker API
  to be explicitly enabled and running.
- Strict Worker API health sends a fixed non-secret invalid Bearer credential to
  loopback `/api/v1/worker/claim` and requires the exact bounded 401 Bearer
  response. Authentication performs the normal ScannerDB lookup before rejection,
  proving the API-to-PostgreSQL path without a real credential or mutation.
- Ordinary Compose health remains unchanged; the strict flag is valid only for
  health/status. Production cannot pass strict mode until task 9.5 installs the
  intended enabled Worker API configuration.

### Command and deadline hardening

- Docker subprocesses use a fixed minimal environment, no shell, closed stdin,
  suppressed stderr, a new process session, bounded 16 KiB stdout, and monotonic
  phase deadlines.
- Timeout, output overflow, reader failure, cancellation, and a descendant that
  retains stdout terminate only the owned process group with bounded waits.
  Successful commands never signal a stale process-group ID, and the raw output
  descriptor has single-owner close semantics.
- Stop/recreate/publication commands run at most once. Only bounded state/health
  observation polls.

### Verification

- Lifecycle tests passed 18/18 on WSL, including real timeout, output-overflow,
  retained-descendant, and successful-process cleanup behavior; native Windows
  had four expected POSIX-only skips.
- Host apply tests passed 18/18 on WSL and protocol tests passed 15/15. The full
  container-runtime suite passed with strict Worker API positive and hostile
  response coverage.
- Broad operations 25, admin 57, Worker API 35, worker-runtime 12,
  runtime-document 22, runtime-document I/O 25, and edge deployment suites
  passed.
- Container selection passed: 42 modules, 943 test definitions, no runner
  skips.
- Rebuilt `truf-worker-test:task-9-3`; a read-only, no-network container run as
  UID 10001 passed 52 selected protocol/apply/lifecycle tests with one expected
  root-ownership skip; 1119 unrelated tests were deselected.
- The selected real Docker inspect format was exercised locally and parsed all
  49 expected keys; a legacy container with an extra bind/import command was
  correctly rejected.
- Independent security/correctness review found no findings after stopped-proof,
  pre-stop CAS, attestation, Worker API DB-path, deadline, process-group, and FD
  ownership fixes.
- Strict OpenSpec validation and `git diff --check` passed before artifact
  closure.

## Known Residual Boundaries

- Apply cannot execute in production until crash verification from task 9.6 is
  implemented. Returning 503 before operation creation remains correct when the
  fixed host-agent socket or result authority is unavailable.
- A process crash exactly after an external side effect and before terminal DB
  persistence is a distributed boundary. Stable operation IDs and idempotent
  replay are the intended mitigation; do not add a general coordinator here.
- Worker API self-stop/restart can terminate the admin request before terminal
  operation completion. Full external lifecycle reconciliation belongs to the
  host operation architecture, not task 7.5/7.6.
- Managed log pages display bounded child output as written. Do not add a
  post-hoc redaction pipeline without user approval.
- Candidate and active file handoff across privileged apply will be revalidated
  by the host agent; task 7.6 must not implement active replacement itself.

## Known Environment/Test Issues Unrelated to Current Work

- Several broad Supervisor safety tests fail in this checkout because external
  runtime authority fixture files such as `runtime/check-openrouter-keys.ps1`
  are absent or legacy fixture config is incomplete. Focused clean runs exclude
  only the documented affected classes; do not weaken production code for them.
- Bundled PostgreSQL integration tests skip when `initdb`/test DSN is not
  available.
- Windows symlink tests may skip when the process lacks symlink privilege.
- The worktree has no useful commit baseline and contains many pre-existing
  untracked/dirty files. Never perform broad resets or cleanup.
- Caddy 2.10.2 does not implement SIGUSR1 reload while production fail2ban
  documentation assumes it. Edge E2E uses an admin-enabled test reload path.
  This predates task 7.1 and needs a separate Caddy-upgrade/reload decision.
- Post-fail2ban-reload generic 404/401 responses in edge E2E do not reliably
  retain global security headers. Authenticated admin pages and mutations still
  enforce strict headers; the harness isolates the unrelated generic response
  behavior.

## Important Successful End-to-End Evidence

- Packaged Windows/Linux protocol-2 E2E passed with GitLab plus synthetic
  DockerHub/HuggingFace claim-to-ingestion.
- Full container E2E passed after strict runtime-document integration.
- Real edge E2E passed twice after rebuilding edge, fail2ban and runtime images;
  final safe result proved authenticated Basic username attribution, private
  marker acceptance, worker header stripping, worker availability, and fail2ban
  behavior.
- Tasks 7.2–7.8 received desktop/mobile browser checks with real Starlette
  routes and trusted headers; task 7.8 also has committed real-Chromium coverage.

## Task 9.5 Completion

- Added the fixed systemd socket/service, tmpfiles layout, root-only installer,
  installed `/usr/lib/truf-host-agent` entrypoints, graceful host-agent shutdown,
  PostgreSQL peer authority, async operation dispatch, and durable result
  reconciliation.
- Runtime Compose now uses exact long-form mounts with host-path creation
  disabled. Host config and secrets use the read-only `/etc/truf/runtime` to
  `/data/config` authority, while immutable package manifests use the separate
  root-owned `/etc/truf/worker-packages` to `/data/worker-packages` authority.
  Candidates, result handoff, host-agent socket, and PostgreSQL socket mounts
  retain their fixed access. Runtime-generated initialization state, lock, and
  PostgreSQL password remain in the private writable `/data` volume.
- Lifecycle inspection validates exact bind sources and the named data volume.
  The installer parses bounded rendered Compose JSON and independently requires
  the same source/target/read-only/create-host-path projection.
- Package manifests are stable-read as root-owned `0644` files beneath a fixed
  descriptor-walked root. Every descendant directory is root-owned and not
  group/other writable; host and container readers consume byte-identical files.
- The installer validates the full root-owned application tree, installed bytes,
  executables, Docker and agent sockets, units, tmpfiles, directories, and file
  modes. Repeat install stops active socket/service units before replacement,
  reloads systemd, enables the socket, and explicitly restarts it. Existing
  active config and secrets are never overwritten.

### Task 9.5 Verification

- Native focused runtime-document/security/host deployment suites passed 67
  tests with seven platform skips; the wider host deployment/lifecycle focused
  suites passed independently.
- WSL root stdlib suites passed 109/109. `systemd-analyze verify` accepted both
  units; warnings were limited to Windows-mounted source-file permissions.
- Canonical combined base+edge Compose rendered JSON passed the exact installer
  projection validator. Base Compose validation also passed under WSL Docker.
- The deny-by-default test image rebuilt successfully. Its UID 10001, read-only,
  no-network run passed all task-9.5 host-agent/deployment selections. Four of
  five initially failed broad tests passed alone after packaging `docker/verify.py`;
  the remaining deep-JSON hostile-input assertion belongs to task 9.6.
- Container selection is AST-clean with 47 modules and 994 test definitions.
- Three independent review passes found no remaining task-9.5 blocker after
  application-tree, mount-source, cleanup, manifest-root, Compose-projection,
  and repeat-install fixes.

## Task 9.6 Completion

- Host-result JSON rejects nesting deeper than 64 before parsing, using an
  iterative string-aware scanner so hostile depth is deterministic across Python
  versions without recursive traversal.
- Candidate validation after a durable database claim now terminalizes as an
  exact `failed/validation_failed` result before acceptance. A failed result
  publication is replayable, and non-original prepared state is never rewritten.
- Crash-boundary coverage now proves interrupted combined-backup replay, retained
  rollback evidence, restart-specific rollback, and durable rollback failure.
- Protocol coverage rejects deeply nested unknown request fields at the real
  server boundary without invoking the privileged handler.

### Task 9.6 Verification

- Native task-matrix suites passed 106 tests with nine platform skips; the WSL
  root stdlib matrix also exited successfully.
- Targeted UID 10001/Python 3.12 container tests passed 102 tests with two
  expected root-identity skips.
- The full hardened, read-only, no-network container suite passed 1219 tests
  with ten platform skips and one deprecation warning.
- Container selection is AST-clean with 47 modules and 1000 test definitions.
- Strict OpenSpec validation and diff checks passed. A final independent audit
  found no remaining task-9.6 blocker.

## Production Completion Status

- The authorized production host now uses the exact root-selected
  `shared-host-edge-v1` profile. Existing host Caddy remains the sole owner of
  ports 80/443, the managed Truf edge binds only `127.0.0.1:18766`, and the
  host agent has no lifecycle authority over host Caddy, X-UI, or another
  unrelated service.
- Task 10.4 production execution is complete: protocol-2 package, runtime,
  admin, edge, and agent checks passed; one reconciled restart succeeded; one
  health-triggered automatic rollback restored the original documents and
  reconciled as `rolled_back`; only then were production controls reopened.

## Task 10.1 Completion

- The production server image and lifecycle authority are remote-only and
  Git-only; TruffleHog remains confined to worker and test images.
- Explicit code authority covers the runtime-document, managed-file, and
  host-agent modules plus the core-profile launcher.
- Existing marked volumes gain only the fixed private `/data/managed-files`
  directory; all other incomplete or unsafe layouts still fail closed.
- Production runbooks now require the fixed host-agent installer, protected
  edge environment, active documents, package manifests, socket, and combined
  runtime/edge Compose deployment in the correct order.
- Container selection includes the operations, discovery-only, core-profile,
  multisource, and direct-source fixtures. The edge fixture advertises the
  exact GitLab, DockerHub, and HuggingFace profile and real-Caddy coverage
  traverses every new admin route plus operation detail.

## Task 10.1 Verification

- Focused native coverage passed 401 tests with eleven platform skips; the
  final edge deployment suite passed 32 tests with one skip.
- Container selection is AST-clean with 54 modules and 1058 definitions.
- A hardened container run passed 1282 tests before fixture-only corrections;
  all five corrected image/static/real-CLI cases then passed in the rebuilt
  image.
- A built production runtime image was verified to contain executable Git and
  no `/usr/local/bin/trufflehog`.
- Two independent reviews found no remaining task-10.1 blocker. Strict
  OpenSpec validation and diff checks passed.

## Task 10.2 Completion

- The offline rollback test removes the protocol-2 worker and operations
  authorities, applies the real stopped-runtime migration, and confirms all
  additive tables are restored.
- A live remote reservation and pre-commit bundle prevent drain completion;
  after both resolve, the authoritative state advances to `drained` with no
  blockers.
- A raw previous-image enqueue contract then opens the same database and writes
  using only legacy columns. The new control state, operation/audit evidence,
  and protocol-2 tables remain intact.
- The complete operations-control suite passed 13 tests on Windows and 13 under
  WSL. Container selection is AST-clean with 54 modules and 1059 definitions;
  strict OpenSpec validation and diff checks passed.

## Task 10.3 Completion

- Focused unit, browser, packaged-worker, edge, PostgreSQL, and formal
  verification gates completed without a remaining failure.
- The PostgreSQL fixture now supports an explicitly selected POSIX PostgreSQL
  binary directory while preserving its existing bundled-Windows default.
- Real PostgreSQL execution exposed and closed one exact local-admission
  recovery identity omission plus stale protocol-2 fixture limits and bundle
  path evidence.

## Task 10.3 Verification

- The real PostgreSQL 16 integration suite passed all 60 tests as UID 10001 in
  a read-only, no-network, capability-free container. Native Windows collected
  the same 60 tests and skipped them because the optional bundled PostgreSQL is
  absent.
- Packaged Windows and Linux worker verification passed after rebuilding the
  protocol-2 artifacts. The real edge E2E passed with 28 baseline responses,
  the exact core source profile, authenticated Caddy routes, and fail2ban
  restart, unban, and expiry evidence.
- Browser coverage passed. The focused task-10.1 matrix passed 401 tests with
  eleven platform skips, and the operations-control suite passed on Windows
  and WSL.
- Container selection is AST-clean with 54 modules and 1059 definitions.
  Strict OpenSpec validation and diff checks passed.

## Task 10.4 Completion

- The production cutover used `shared-host-edge-v1` while both explicit gates
  were paused and drain was authoritative with zero blockers. Root-owned profile,
  package, Compose projection, loopback edge, runtime, admin, agent, Caddy, and
  X-UI service checks passed without granting the Truf lifecycle authority any
  control over shared-host services.
- Reconciled restart operation `08dc7e8b-551f-5c64-b803-de905f040420`
  completed `succeeded/succeeded` with no safe failure category, healthy runtime
  and edge, and no failed hold.
- Reconciled rollback operation `6d09ea58-2c48-5366-b5ee-951c341fcd6f`
  deliberately applied a valid candidate with Worker API disabled. Strict
  forward health classified the failure as `health_check_failed`; the agent
  restored the byte-identical original config and secrets exactly once and
  completed `rolled_back/rolled_back` with no failed hold.
- Audited, revision-checked operations then canceled drain, resumed discovery,
  and resumed dispatch in that order. Controls advanced from revision 19 to 22
  and are now discovery open, dispatch open, and drain `normal`.
- After reopening, the active production worker contacted the server and
  received new DockerHub work. Lifecycle preflight, strict runtime health, and
  host-agent/Caddy/X-UI service checks remained healthy.

## Production Architecture Corrections

- Real candidate validation found that private writable runtime configuration
  could not also be the root-owned immutable package authority. Worker package
  manifests now use the separate read-only `/data/worker-packages` authority,
  while initialization, lock, and PostgreSQL password state use private files
  directly beneath `/data`.
- PostgreSQL maintenance and offline migration use the discovery-only server
  authority profile and therefore do not require the worker-only TruffleHog
  executable. The offline cutover reconciled two stale leases, migrated the
  production database, and established audited discovery and dispatch pauses.
- Production validation also corrected Worker API HTTP concurrency from one to
  eight while retaining the real assignment cap of one.
- DockerHub discovery now configures account metadata without creating scanner
  Docker directories or requiring scanner runtime initialization. Scanner
  execution keeps the original full-runtime guard and Docker config behavior.

## Task 10.4 Production Evidence

- The active thin-v6 runtime image is
  `sha256:dd55500ee2f947c0088a57061d1689025653fc0b08ed7fb22baea6816fb93270`
  under tag `truf-local:runtime`; the managed edge image is
  `sha256:6ec35cae4f4cf4bcd2bb1fe427b9cd2f16f8b5c2577442eae0941d806d362529`
  under tag `truf-local:edge`.
- Canonical health reports Supervisor `ACTIVE`, PostgreSQL `READY`, and healthy
  DockerHub, GitLab, HuggingFace, janitor, projector, ingester, and Worker API
  processes. Lifecycle attestation confirms host-network runtime with no
  published ports and a loopback-only edge.
- Root-owned protocol-2 Windows and Linux manifests advertise only GitLab,
  DockerHub, and HuggingFace. Candidate initialization passed against the real
  production volume before cutover.
- Public checks after rollback returned 401 for an invalid Worker API token,
  401 for the unauthenticated protected admin route, and 404 for an unrelated
  Truf path. Host Caddy, X-UI, and the Truf host agent remained active.
- Shared-host restart validation exposed a real Docker Compose identity nuance:
  `network_mode: service:runtime` changes the edge Compose config hash whenever
  runtime is recreated. Lifecycle now attests all fixed edge metadata before
  observing and pinning each forward or rollback edge hash; it never weakens the
  fixed topology checks.
- Strict runtime-health command failures are normalized to lifecycle health
  failures. This made the intentional rollback drill reconcile with the exact
  `health_check_failed` safe category rather than generic `apply_failed`.
- Root-only evidence for two diagnostic failed holds remains preserved under
  `/opt/truf-remote-server/staging/failed-hold-recovery-20260922` and
  `/opt/truf-remote-server/staging/failed-hold-recovery-v2-20260922`. Both holds
  were cleared only after strict health, identity, and evidence checks.
- The exact pre-v4 unit, previous runtime image and configuration, verified
  custom-format PostgreSQL backup, failed-hold evidence, and reconciled operation
  results remain available for rollback and audit. No X-UI or unrelated host
  service was modified.

### Task 10.4 Verification

- The final focused host-agent matrix passed 107 tests with fourteen platform
  skips. Edge deployment and container-runtime pytest coverage passed 221 tests
  with one platform skip.
- Runtime-document coverage passed 22 tests and worker-package coverage passed
  15 tests. Changed lifecycle and drill scripts compile successfully.
- Both standalone base+edge and shared-host base+edge Compose projections passed
  `config --quiet` with non-secret synthetic validation values; required edge
  variables still fail closed when omitted.
- Strict OpenSpec validation and diff checks passed after production evidence
  was reconciled and controls were reopened.

## Task 10.5 Production Evidence

- A single immutable DockerHub digest was processed independently by the
  packaged Windows worker and the hardened WSL worker with assignment caps and
  parallelism fixed at one. Both reservations produced accepted protocol-2
  bundles and acknowledged ingestion/projection records.
- A completed assignment replay returned the original accepted receipt without
  rescanning. A separate claim-only assignment expired at its real wall-clock
  deadline, refunded queue capacity and attempts, and replayed the same durable
  expiry receipt over the public Worker API.
- An audited drain advanced from `draining` to `drained` with zero live remote
  assignments and zero pre-commit bundles. Audited cancel restored `normal`
  while both explicit gates remained paused.
- Temporary canary devices were revoked, their token file was overwritten and
  removed, the normal 7200-second assignment TTL was restored, and no unresolved
  canary assignment remained.

## Task 10.6 Production Evidence

- Only after the DockerHub canary gates passed, audited operations enabled
  production discovery and dispatch and created one production WSL worker user
  and device with an active assignment cap of one.
- Real production cycles completed successfully for GitLab with `gl_1`, public
  HuggingFace with `hf_1`, and DockerHub with its real account pool. The final
  DockerHub cycle reported eleven available accounts and exited with code zero.
- The active server and both protocol-2 compatibility profiles contain exactly
  `gitlab`, `dockerhub`, and `huggingface`; GitHub remains excluded.
- Current controls are revision 22 with discovery and dispatch open and drain
  state `normal`. The production WSL worker is running detached with
  parallelism one; its audited user is enabled, its device is not revoked, and
  it contacted the server after the task-10.4 rollback gates reopened.
- Rollback evidence retains the previous image, exact units, previous active
  configuration, and verified database dump. The v4 unit and image identities
  were checked before and after activation.

## Managed Files Production Extension (2026-09-22)

- Production Files now exposes exactly three configured logical roots:
  `runtime-logs`, `runtime-keychecks`, and `runtime-results`. All are list/read
  only. Logs and keychecks retain the 64 MiB per-file limit; only the fixed
  result root permits 256 MiB files.
- The result root exposes only active `scan_results.jsonl` and
  `found_secrets.jsonl` projections and their exact six-digit generations.
  Internal locks, databases, ledgers, scan errors, malformed generations,
  recovery files, and temporary/quarantine directories remain unavailable.
- The deployment paused discovery and dispatch and allowed the existing remote
  DockerHub assignment to resolve naturally. Drain reached `drained` with zero
  blockers before any runtime or configuration activation; no assignment was
  cancelled.
- Runtime image `sha256:3b4d6e19e29e85a32b75d64265d75e71100d3f7f00221d21de544256dadd25cf`
  was activated with a fail-safe Compose transition. The preceding image is
  retained as `truf-local:runtime-pre-managed-files-v1`.
- Official restart operation `1978f770-c4a5-537d-b474-c3c9428cb79c`
  completed `succeeded/succeeded`, reconciled without a safe category or failed
  hold, and proved the full fixed lifecycle on the new immutable image before
  the root configuration changed.
- Official Preview/Save operation `87b78ff6-d8eb-5fb4-a897-83b52c3697e5`
  changed only the managed-root mapping. Apply operation
  `7a572971-08d2-5e51-be78-4fb7039a93d4` completed
  `succeeded/succeeded`, reconciled without rollback or failed hold. The active
  configuration identity is
  `e48797689ab0ee7328d21cf5c23d0ffedb979dba492bf49d1da3afe4b075cf5b`.
- Live runtime traversal verified all three exact policies, an empty keycheck
  listing, two allowlisted active result projections, stable snapshot
  length/hash, one-snapshot serialization, permit reuse, safe rejection of an
  internal result artifact, and denied result mutation.
- Live Worker API HTML rendered all three root IDs and the read-only keycheck
  and result pages. A permitted 146366-byte projection download matched its
  exact source SHA-256 and length through the streaming route; an internal
  result name returned 404. No file contents were emitted during validation.
- Final audited controls are revision 56 with discovery and dispatch open and
  drain `normal`. Canonical Worker API plus discovery-producer strict health and
  lifecycle preflight pass; failed hold is absent; host agent, host Caddy, and
  X-UI are active. Public checks remain 401 for invalid Worker authentication,
  401 for unauthenticated admin access, and 404 for an unrelated path.