Initial server source import

openspec/changes/add-web-operations-control-plane/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-19

openspec/changes/add-web-operations-control-plane/design.md

@@ -0,0 +1,155 @@
+## Context
+
+The current runtime combines provider discovery, queue admission, PostgreSQL claiming, and local scanner execution in the same `console_runner.py` source cycle. Supervisor can pause a child in memory, but that state is lost on restart and does not fence concurrent Worker API claims. The remote protocol supports GitHub and GitLab exact-Git assignments, while DockerHub and HuggingFace exist only as local scan paths. The typed admin UI can mutate worker users, devices, and queue rows, but routine runtime control and file changes still require SSH.
+
+The deployment is deliberately split across trust boundaries. Worker API/admin runs unprivileged in the read-only runtime container; either the standalone Truf edge or an existing root-owned host Caddy plus a loopback Truf edge owns public routing and injects a private edge marker; PostgreSQL is the durable queue authority; Supervisor exposes an authenticated loopback control protocol; systemd/Docker lifecycle control remains on the host. The exact ingress profile is root-installed policy, not runtime or request input. The design must retain those boundaries, keep uploads available during operational pauses, avoid a local server scanner, and never expose a shell or unrestricted host path.
+
+The default distributed core profile changes to exactly `gitlab`, `dockerhub`, and `huggingface`. Existing protocol-1 Git assignments may still be in flight when the new server is deployed, so result compatibility and migration ordering matter.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Run provider discovery as server-side producer processes that only search, normalize, and enqueue.
+- Persist and atomically enforce independent discovery pause, dispatch pause, and drain controls.
+- Complete remote claim, scan, upload, ingestion, and projection for GitLab, DockerHub, and HuggingFace.
+- Prove the first new-source path with a bounded public DockerHub immutable-digest canary.
+- Provide typed, server-rendered admin operations for runtime state, controls, Supervisor, logs, configuration, plaintext secrets, managed files, and audit history.
+- Apply configuration and secrets through validated candidates, backups, coordinated restart, health checks, and automatic rollback.
+- Keep operator identity, operation status, and audit records durable across admin/runtime restarts.
+- Support exact standalone and shared-host ingress profiles while preserving route confinement, loopback-only shared ingress, and the closed host-agent request schema.
+
+**Non-Goals:**
+
+- Local scanning or TruffleHog execution on the server.
+- Arbitrary shell commands, arbitrary Supervisor command strings, Docker socket access, or browsing host root.
+- PostgreSQL data-file access through the file page.
+- Private Docker registry credential delivery or Docker layer-plan transport in the first rollout.
+- Private HuggingFace credential delivery in the first canary; server-side discovery credentials remain supported.
+- A client-side single-page application or storage of configuration/secrets in browser persistence.
+- Replacing PostgreSQL queue, reservation, bundle, or projection authority.
+- Managing, restarting, or reconfiguring unrelated host Caddy sites, X-UI, or other shared-host services.
+- Runtime-selected topology, arbitrary ingress ports/upstreams, or wildcard/private-interface shared-edge binding.
+
+## Decisions
+
+### 1. Separate discovery into an explicit process role
+
+Supervisor will launch a `discovery-producer` role for each enabled core source instead of launching the existing combined source cycle. The role is part of the authenticated runtime bootstrap identity and is visible in structured Supervisor state.
+
+`console_runner.py` will expose a discovery-only cycle that performs provider requests, normalization, DockerHub retry/tag resolution where applicable, and idempotent queue admission. It will finish the source-cycle record with no scan requests and cannot call scan option preparation, scan-slot acquisition, target claiming, bundle staging, or scanner execution. Discovery runs according to its interval even when pending queue backlog exists.
+
+This is preferred over a mutable `enqueue_only` flag on the existing combined cycle because a distinct bootstrap role and call graph make accidental local scanner entry testable and fail-closed. The old combined path may remain for non-server/local workflows, but the server profile will not launch it.
+
+### 2. Make PostgreSQL the durable control authority
+
+An additive singleton control row will hold:
+
+- monotonically increasing `revision`;
+- explicit `discovery_paused` and `dispatch_paused` flags;
+- `drain_state` (`normal`, `draining`, or `drained`);
+- actor, operation ID, and update timestamps.
+
+Mutations use compare-and-swap on the expected revision and append an audit event in the same transaction. Explicit pause flags remain independent; entering drain overlays both effective gates, and cancelling drain does not clear pauses that the operator set explicitly.
+
+Discovery producers check the effective discovery gate before provider I/O and enforce it again in the same transaction as discovery queue admission. Source retry and Docker tag-resolution claims are also gated. Generic enqueue functions used by ingestion and maintenance are not globally disabled.
+
+`reserve_and_claim_target()` enforces the effective dispatch gate inside its existing claim transaction. This closes the race between an API pre-check and target reservation. Authentication, status, terminal reports, assignment expiry, result upload, receipt replay, `mark_result_bundle_ready()`, and ingestion remain available while dispatch is paused or draining.
+
+Drain is complete when there are no live remote assignments and no accepted result bundles that have not reached database commit. Pending target backlog, projection work, and keycheck work do not prevent `drained`; those workers can safely resume after restart. A reconciler advances `draining` to `drained` from database state.
+
+This is preferred over stopping Worker API or Supervisor-only pause state because in-flight workers must retain their upload path and all enforcement must survive process restart.
+
+### 3. Generalize assignments through source adapters and protocol 2
+
+The Git-specific assignment builder will become an adapter registry. Each adapter declares the canonical queue source, worker scan platform, planning kind, package capability, execution-snapshot validator, and claim/reconciliation behavior:
+
+| Queue source | Worker platform | Planning kind | Initial execution |
+| --- | --- | --- | --- |
+| `gitlab` | `gitlab` | `exact_git_v1` | Existing exact commit/snapshot path |
+| `dockerhub` | `docker` | `docker_direct_v1` | Public immutable digest reference |
+| `huggingface` | `huggingface` | `huggingface_space_v1` | Public Space identifier |
+
+The assignment API paths remain stable, but worker protocol/package compatibility advances to version 2 and manifests advertise explicit source/planning capabilities. Protocol-1 packages receive no new claims after cutover. Status, terminal report, upload, receipt replay, and immutable snapshot reconciliation remain available for already-issued protocol-1 assignments through their fixed expiry.
+
+Source selection will try other eligible configured sources when one queue has no claimable target instead of permanently choosing one source by request-ID modulo. Every successful claim still binds one fenced reservation, fixed expiry, device identity, immutable execution snapshot, and result-bundle identity.
+
+The first DockerHub canary uses an image resolved to an immutable digest and direct worker execution. Digest resolution is assignment planning, not proof that the worker can access the registry. The worker is the final access check and reports an inaccessible image using the bounded provider-failure result contract. The assignment does not serialize `DockerRegistryAuth`, process-local monotonic deadlines, server blob leases, or a Docker layer plan. The first HuggingFace canary follows the same worker-authoritative access model. Discovery drops Spaces that its existing provider response explicitly marks private, protected, gated, or disabled; the worker reports an inaccessible repository as non-retryable. Existing GitLab credential behavior remains, but provider discovery credentials are not assignment fields.
+
+Source adapters SHALL remain minimal. The server validates canonical target and assignment shape, performs only planning needed to identify the target, and leaves real provider access to the worker. A new per-target server access probe, durable public-access proof, proof freshness schema, broad child-environment credential scrubbing, credential sandbox, or post-hoc redaction pipeline is not implied by the credential non-transfer rule. Any such mechanism requires separate operator approval and an explicit OpenSpec requirement and task before implementation. Existing defensive code is not precedent for adding the same machinery to another source.
+
+### 4. Keep the admin interface typed and server-rendered
+
+`admin_api.py` will add explicit GET and POST routes for overview, search, dispatch/workers, Supervisor, logs, config, secrets, files, audit, and operation status. Forms retain exact field sets, bounded URL-encoded bodies, exact HTTPS Origin checks, CSRF, escaped output, CSP/HSTS/no-store headers, and POST/redirect/GET behavior. Unknown methods, route shapes, action names, source IDs, and file-root IDs fail closed.
+
+Caddy will strip any inbound operator header and inject the authenticated Basic-auth username alongside the existing trusted edge marker. The backend accepts the actor only with that marker and records it in control/audit rows. Plaintext secrets are rendered only in the dedicated no-store page; no JavaScript, local storage, or audit payload receives their values.
+
+The root-owned deployment profile is exactly `standalone-edge-v1` or `shared-host-edge-v1`. Standalone remains the default and owns host port 443. In shared-host mode, the existing host Caddy remains the sole owner of ports 80/443 and imports a fixed route-only snippet for only Worker API and the exact random admin prefix. It strips private/transit headers, injects an independent ingress marker, and proxies to the Truf edge at fixed loopback `127.0.0.1:18766`. The Truf edge rejects a missing marker before trusting the forwarded client address, binds only loopback, and retains Basic authentication, operator attribution, private backend marker, denylist, redacted logging, and security headers. It has no catch-all route for unrelated host applications.
+
+Admin will call new exact Supervisor actions for structured snapshot, one managed-source lifecycle action, and bounded log tail. Web input will never be forwarded to Supervisor's generic command parser. Long-running apply/restart operations return an operation ID and status page because the process serving the POST may be restarted.
+
+### 5. Share one strict configuration/secrets validator
+
+A side-effect-free validator will be used by preview, runtime startup, and the host operations agent. It will enforce bounded UTF-8 YAML, duplicate-key rejection, mapping roots, strict scalar types and bounds, known keys, the exact core profile, credential-pool entry schemas and unique names, reference integrity, package capabilities, and managed deployment paths. Validation errors identify fields but never echo secret values.
+
+Edits are candidate revisions, not direct active-file writes. Preview shows a structural/text diff with secret values redacted in audit and operation records. Candidate save and apply use expected SHA-256 hashes as compare-and-swap guards against stale forms or concurrent SSH changes.
+
+Configuration and secrets remain separate logical resources and are not exposed through the generic file browser.
+
+### 6. Use a narrow host operations agent for privileged lifecycle work
+
+A root-owned systemd socket/service will accept local requests from the runtime UID over a Unix socket. Its request schema contains only an operation UUID, one action enum (`apply-config`, `apply-secrets`, `apply-both`, or `restart`), and expected active/candidate hashes. It accepts no command, service name, path, environment, Compose argument, or shell text.
+
+Candidates live under a fixed host-managed bind directory shared read-only/read-write as required; active config/secrets will migrate from Docker-volume-only storage to fixed host-managed files before the agent is enabled. Immutable worker package manifests use a separate root-owned `/etc/truf/worker-packages` authority mapped read-only at `/data/worker-packages`; they never share the runtime-writable active-document trust root. Runtime-generated initialization state, lock, and PostgreSQL password remain in the private `/data` volume rather than the read-only active-document bind. The agent reads one root-owned exact profile and uses its fixed Compose files, network, ports, volume, capabilities, and Caddyfile. It never accepts that profile through its six-field request. The agent uses a singleton lock, revalidates candidate bytes, verifies all hashes, takes byte-identical backups, stops the fixed Truf runtime and edge, atomically replaces fixed files, recreates and attests that exact profile, and waits for Supervisor ACTIVE, PostgreSQL READY, Worker API, ingester, projector, and edge health. It never performs lifecycle actions on host Caddy, X-UI, or unrelated services. Failure restores backups and verifies the previous runtime. If both forward start and rollback fail, it enters a failed hold without deleting evidence or retrying indefinitely.
+
+The admin request and operation row are committed before the agent begins. The agent writes a bounded result envelope that the runtime reconciles into PostgreSQL after restart. The Docker socket is never mounted into the runtime container.
+
+### 7. Restrict managed files by logical root and descriptor-safe traversal
+
+The file page exposes configured logical roots for logs, backups, and selected result/export directories. The client submits a root ID plus canonical relative path, never an absolute root. Config, secrets, PostgreSQL storage, application code, sockets, host-agent metadata, and raw result bundles are excluded.
+
+Paths reject empty/absolute/drive-qualified/backslash/NUL/dot components and enforce byte, depth, listing, and file-size bounds. Linux traversal retains a root directory descriptor and uses component-wise `openat`/`dir_fd` operations with `O_NOFOLLOW`. Only single-link regular files are readable or replaceable; symlinks, hardlinks, reparse points, devices, FIFOs, and sockets are rejected. Writes use an exclusive same-directory temporary file, fsync, atomic replacement, directory fsync, and final owner/type/mode verification.
+
+Typed operations are limited to list, view/download, create/replace, and delete within roots that explicitly allow each action. Every mutation records actor, logical root/path, before/after hashes, byte counts, operation result, and timestamp, never file content.
+
+### 8. Persist operations and append-only audit records
+
+Additive PostgreSQL tables will store operation lifecycle and audit events. Operation rows contain typed action/target, requested/started/completed timestamps, safe status/category/detail, expected and resulting revisions/hashes, and host-agent reconciliation state. Audit events are append-only and include actor, operation ID, action, logical target, before/after identities, result, and a previous-event/hash-chain identity.
+
+Control mutation and its audit event commit atomically. File/config operation requests are audited when accepted and again when completed. Secret values, authorization headers, device tokens, provider tokens, CSRF values, and uploaded file bytes are forbidden from both schemas and logs.
+
+## Risks / Trade-offs
+
+- [A discovery code path accidentally reaches local scanning] -> Use a separate bootstrap role and call graph, omit TruffleHog from the server image/profile, and test that scan/claim functions are never invoked.
+- [Pause races enqueue or claim] -> Enforce gates transactionally at queue admission and reservation, not only in UI or process state.
+- [A runtime restart interrupts uploads] -> Drain to database-committed bundles before planned apply; preserve upload/status routes during pause; rely on fixed-expiry replay for network failures.
+- [Protocol-2 rollout strands old work] -> Stop protocol-1 issuance first, retain its reconciliation/upload readers until no unresolved assignments remain, and only then remove compatibility in a later change.
+- [Direct Docker scanning is less efficient than layer reuse] -> Accept the bandwidth cost for the first bounded canary; add layer-plan transport only after the simpler authority path is proven.
+- [A source-specific access check grows into preventive server or worker security infrastructure] -> Keep provider access worker-authoritative and require separate operator approval plus an explicit requirement/task before adding probes, durable proofs, broad environment scrubbing, sandboxes, or redaction pipelines.
+- [Plaintext secret editing exposes values to an operator browser] -> Require the existing protected admin boundary, no-store responses, no client persistence/scripts, bounded rendering, and value-free audit/log records.
+- [The host agent becomes a root command proxy] -> Use a closed action enum and fixed paths/units, peer-credential checks, hash CAS, no shell, and adversarial request-schema tests.
+- [Filesystem containment has TOCTOU or link attacks] -> Use retained directory descriptors and no-follow operations for every component; reject multi-link and non-regular files.
+- [Rollback binary cannot read an additive schema] -> Keep migrations additive, preserve old markers, avoid incompatible constraint rewrites, and test old-image rollback before production cutover.
+- [Shared-host ingress exposes a private listener] -> Require host networking only in the exact shared profile, no Docker-published ports, explicit loopback bind, an independent ingress marker, and metadata attestation before lifecycle work.
+- [A Truf route captures or disrupts another host application] -> Install only a fixed route-only host-Caddy snippet with no listener, catch-all, global policy, or unrelated lifecycle authority.
+- [Shared profile drift changes the trust boundary] -> Read one stable root-owned mode-0444 profile, reject unknown values and metadata, and attest exact Compose labels, mounts, network, ports, capabilities, and Caddyfile.
+- [Server-rendered pages are less dynamic] -> Prefer explicit refresh/status pages over JavaScript to preserve the current CSP and reduce secret-retention surface.
+
+## Migration Plan
+
+1. Add and test the control/audit/operation schema, discovery role, source adapters, protocol-2 package support, typed admin routes, validator, file service, and host agent while all new controls remain disabled.
+2. Build new server and Windows/Linux worker artifacts and verify code-authority/package manifests.
+3. Set dispatch paused, stop discovery, and allow or expire all protocol-1 assignments while continuing to accept their uploads.
+4. Stop runtime through the authenticated deployment path, create database and file backups, and run the additive migration under existing offline migration guards.
+5. Migrate active config/secrets to the fixed host-managed bind directory; install the exact root-owned ingress profile and socket-activated agent; start the new runtime and Truf edge with discovery and dispatch paused. In shared-host mode, install and validate the fixed route-only snippet in the existing host Caddy without granting the agent authority over that service.
+6. Verify health, admin actor attribution, control CAS/audit, Supervisor typed actions, managed-file containment, and rollback using a non-secret candidate.
+7. Publish protocol-2 worker packages. Enable a low-cap DockerHub public immutable-digest canary and verify search, enqueue, claim, scan, upload, ingestion, projection, expiry/replay, and drain.
+8. Enable GitLab and then public HuggingFace after the canary gates pass. Switch the default core profile exactly once and keep GitHub disabled.
+
+Rollback restores byte-identical config/secrets and the previous runtime/edge images while retaining additive database tables and audit evidence. Shared-host rollback does not modify or restart host Caddy or unrelated services. Rollback must not begin while protocol-2 DockerHub/HuggingFace assignments or pre-commit bundles are unresolved. If rollback health also fails, the agent leaves the Truf deployment stopped/held with backups intact for SSH recovery.
+
+## Open Questions
+
+- The retention duration and byte caps for browsable logs/backups/results need deployment defaults, but remain configurable within validator bounds.
+- Private DockerHub and HuggingFace worker credential delivery is deferred. It must not be designed or implemented without separate operator approval and a dedicated change defining only the agreed delivery and failure semantics.
+- Multi-operator authorization roles are deferred. This change records the Caddy Basic-auth username as actor but grants the existing admin policy uniformly.

openspec/changes/add-web-operations-control-plane/proposal.md

@@ -0,0 +1,32 @@
+## Why
+
+The remote deployment can accept GitHub and GitLab worker assignments, but it cannot continuously discover targets without also entering the local scan path, and routine operation still requires SSH and direct file edits. The server needs a web-operated control plane that keeps discovery, dispatch, remote scanning, configuration, and runtime supervision separate and makes the intended distributed pipeline usable end to end.
+
+## What Changes
+
+- Add discovery-only server producers for the core source set `gitlab`, `dockerhub`, and `huggingface`; they may search and enqueue targets but must never claim or scan them locally.
+- Add persistent controls for pausing discovery, pausing new assignment dispatch, and draining the system while continuing to accept uploads for existing assignments.
+- Extend remote assignments, worker packages, scan execution, and result acceptance to support full GitLab, DockerHub, and HuggingFace claim-to-ingestion cycles, with DockerHub as the first deployed end-to-end canary.
+- Add authenticated admin pages for overview/search controls, workers and dispatch, Supervisor status/commands/logs, runtime configuration, plaintext `secrets.yaml`, managed files, and an operation audit trail.
+- Apply configuration and secret changes as validated, backed-up operations with coordinated runtime restart, health verification, and automatic rollback rather than in-place live mutation.
+- Support two exact root-selected production ingress profiles: the standalone Truf edge and a shared-host edge behind an existing root-owned host Caddy, without adding topology, port, path, service, or command fields to the host-agent request.
+- Expose only explicitly managed project directories through the file page; host root, PostgreSQL data, Docker control sockets, and arbitrary shell execution remain outside the web interface.
+- **BREAKING** Replace GitHub in the default core source set with HuggingFace; the new default core set is exactly GitLab, DockerHub, and HuggingFace.
+- **BREAKING** Advance worker compatibility so packages that support only the current GitHub/GitLab assignment contract are not eligible for the new core-source profile and must be rebuilt.
+
+## Capabilities
+
+### New Capabilities
+
+- `distributed-core-source-processing`: Discovery-only production, persistent discovery/dispatch/drain controls, and remote-only processing for the configured core sources.
+- `multisource-worker-assignments`: Compatible worker packaging and fenced assignment/result lifecycles for GitLab, DockerHub, and HuggingFace.
+- `web-operations-console`: Authenticated web views and mutations for runtime overview, search, dispatch, workers, Supervisor, logs, and audited operations.
+- `managed-runtime-editing`: Validated editing and coordinated application of runtime configuration, plaintext secrets, and allowlisted project files with backup and rollback.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+The change affects source-cycle separation in `app/console_runner.py` and `app/supervisor.py`; queue and control state in `app/scanner_db.py`; worker assignment, package, client, scan execution, and API modules; the typed admin API and its HTML/CSS; standalone and shared-host Caddy/Compose profiles; profile-specific host lifecycle, installer, and denylist validation; runtime configuration and worker package manifests; and focused unit, integration, browser, and deployment tests. Existing queue and result authority remains PostgreSQL-backed, existing uploads remain accepted during drain, and no host filesystem or generic shell API is introduced.

openspec/changes/add-web-operations-control-plane/specs/distributed-core-source-processing/spec.md

@@ -0,0 +1,102 @@
+## ADDED Requirements
+
+### Requirement: Exact distributed core source set
+The server core profile SHALL contain exactly `gitlab`, `dockerhub`, and `huggingface`, and SHALL NOT start GitHub or any other discovery source as part of that profile.
+
+#### Scenario: Core profile starts
+- **WHEN** Supervisor starts the distributed core profile
+- **THEN** it starts one discovery producer for GitLab, DockerHub, and HuggingFace and no GitHub producer
+
+### Requirement: Discovery-only producer isolation
+Each core discovery producer SHALL search its provider, normalize targets, and admit them to PostgreSQL without claiming targets, acquiring scan slots, invoking a scanner, or staging result bundles locally.
+
+#### Scenario: GitLab discovery finds targets
+- **WHEN** the GitLab producer completes a provider search
+- **THEN** it enqueues the normalized targets and records zero local scan requests
+
+#### Scenario: DockerHub discovery resolves targets
+- **WHEN** the DockerHub producer processes pages, retries, tags, or digests
+- **THEN** it may persist discovery progress and immutable targets but never invokes TruffleHog or reserves those targets locally
+
+#### Scenario: HuggingFace discovery finds Spaces
+- **WHEN** the HuggingFace producer returns Space identifiers
+- **THEN** it drops records explicitly marked private, protected, gated, or disabled and enqueues the remaining identifiers without entering a local scan path or making a second per-Space verification request
+
+#### Scenario: Pending backlog exists
+- **WHEN** a scheduled discovery interval arrives while pending targets already exist
+- **THEN** the producer still performs the configured discovery cycle unless discovery is paused
+
+### Requirement: Durable operations control state
+The system SHALL persist discovery pause, dispatch pause, drain state, revision, actor, operation identity, and update timestamps in PostgreSQL so that control state survives process and host restarts.
+
+#### Scenario: Runtime restarts while paused
+- **WHEN** discovery and dispatch are paused and the runtime restarts
+- **THEN** both effective gates remain paused after startup
+
+#### Scenario: Stale control form is submitted
+- **WHEN** a mutation supplies a revision older than the current control revision
+- **THEN** the system rejects it without changing control state or writing a success audit event
+
+#### Scenario: Explicit pause coexists with drain
+- **WHEN** an operator explicitly pauses discovery, enters drain, and later cancels drain
+- **THEN** the explicit discovery pause remains set
+
+### Requirement: Transactional discovery admission gate
+The system SHALL enforce the effective discovery gate in the same transaction that admits discovered targets or claims discovery-specific retry work.
+
+#### Scenario: Pause races target admission
+- **WHEN** discovery pause commits before a producer admission transaction commits
+- **THEN** no newly discovered target is admitted by that transaction
+
+#### Scenario: Discovery is paused before provider request
+- **WHEN** a producer begins a cycle while discovery is effectively paused
+- **THEN** it performs no provider request and records a paused cycle outcome
+
+#### Scenario: Upload-derived work arrives during pause
+- **WHEN** result ingestion creates projection or keycheck work while discovery is paused
+- **THEN** that work remains admissible because it is not provider discovery
+
+### Requirement: Transactional dispatch gate
+The system SHALL enforce the effective dispatch gate within the reservation transaction so that no new remote assignment can be issued after dispatch pause or drain commits.
+
+#### Scenario: Dispatch pause races claim
+- **WHEN** dispatch pause commits before a worker claim transaction commits
+- **THEN** the claim returns a paused or no-work response and creates no reservation
+
+#### Scenario: Existing worker uploads while paused
+- **WHEN** dispatch is paused and a worker with an existing assignment reports status or uploads its result
+- **THEN** the server accepts the valid request under the existing assignment authority
+
+#### Scenario: Assignment expires while paused
+- **WHEN** an existing assignment expires during dispatch pause
+- **THEN** the reaper processes it normally without issuing replacement work
+
+### Requirement: Drain lifecycle
+Entering drain SHALL effectively pause discovery and dispatch while preserving status, terminal report, upload, receipt replay, ingestion, projection, and maintenance paths needed to finish accepted work.
+
+#### Scenario: Drain begins with active assignments
+- **WHEN** drain is requested while remote assignments are active
+- **THEN** the state becomes `draining`, no new targets or assignments are admitted, and existing workers retain their result path
+
+#### Scenario: Drain reaches completion
+- **WHEN** no live remote assignments remain and every accepted result bundle has reached database commit
+- **THEN** the reconciler advances the state to `drained`
+
+#### Scenario: Pending targets remain
+- **WHEN** pending queue targets remain but all issued assignments and pre-commit bundles are resolved
+- **THEN** drain may still become `drained`
+
+#### Scenario: Projection work remains
+- **WHEN** projection or keycheck work remains after its result bundle is database-committed
+- **THEN** that work does not prevent the control state from becoming `drained`
+
+### Requirement: Discovery process observability
+Supervisor and the operations console SHALL expose each discovery producer's source, role, lifecycle state, last cycle result, last successful discovery time, next scheduled run, and bounded safe error category.
+
+#### Scenario: Provider rejects credentials
+- **WHEN** a discovery producer receives a provider authorization error
+- **THEN** operations state reports the source and safe authorization category without exposing the credential or provider response body containing secrets
+
+#### Scenario: Producer is stopped
+- **WHEN** an operator stops a managed producer through a typed Supervisor action
+- **THEN** structured state identifies it as stopped without changing the persisted discovery pause flag

openspec/changes/add-web-operations-control-plane/specs/managed-runtime-editing/spec.md

@@ -0,0 +1,182 @@
+## ADDED Requirements
+
+### Requirement: Shared strict document validation
+Configuration and secrets preview, startup, and privileged apply SHALL use the same side-effect-free validator for bounded UTF-8 YAML, duplicate keys, mapping roots, strict scalar types and bounds, known keys, exact core profile, auth-pool schemas, unique entry names, reference integrity, package capabilities, and deployment paths.
+
+#### Scenario: Valid configuration is previewed
+- **WHEN** an operator submits a candidate satisfying the complete schema
+- **THEN** preview returns normalized validation success and a bounded diff without activating the candidate
+
+#### Scenario: Duplicate or unknown key is submitted
+- **WHEN** a candidate contains a duplicate mapping key or unsupported field
+- **THEN** validation fails before staging or restart and identifies the field without echoing secret values
+
+#### Scenario: Secret reference is invalid
+- **WHEN** configuration selects an auth-pool entry that does not exist in the candidate secrets
+- **THEN** combined validation fails and neither document becomes active
+
+### Requirement: Separate immutable package authority
+Worker package manifests SHALL resolve only beneath a separate root-owned, non-writable package authority, while runtime-generated initialization state, locks, and PostgreSQL credentials SHALL remain in the private writable data volume rather than the active config/secrets bind.
+
+#### Scenario: Package manifest uses the active-document directory
+- **WHEN** configuration references a package manifest beneath the config/secrets authority or outside the fixed package root
+- **THEN** validation fails before startup or privileged apply
+
+#### Scenario: Runtime initializes with active documents mounted read-only
+- **WHEN** the runtime initializes or restarts with the active config/secrets directory mounted read-only
+- **THEN** its initialization marker, singleton lock, and generated PostgreSQL password remain writable only through fixed private data paths
+
+### Requirement: Candidate revisions and compare-and-swap
+The system SHALL stage validated candidate revisions separately from active files and SHALL require expected active and candidate SHA-256 identities when saving or applying them.
+
+#### Scenario: Candidate is saved
+- **WHEN** an operator saves valid bytes against the current active hash
+- **THEN** the candidate is durably staged with a new hash and the active file is unchanged
+
+#### Scenario: Active file changed through SSH
+- **WHEN** the active hash differs from the expected hash submitted by a stale page
+- **THEN** save or apply fails without replacing either active file
+
+#### Scenario: Candidate changed concurrently
+- **WHEN** the supplied candidate hash is no longer current
+- **THEN** apply is rejected before host lifecycle changes begin
+
+### Requirement: Plaintext secrets editing without persistence leakage
+The protected secrets page SHALL allow authorized operators to view and edit the complete plaintext YAML document while responses remain no-store and secret values remain absent from browser persistence, application logs, diffs outside that page, operation status, and audit records.
+
+#### Scenario: Operator opens secrets page
+- **WHEN** an authenticated operator requests the dedicated secrets editor
+- **THEN** the current document is rendered in a server-side form over the protected no-store response
+
+#### Scenario: Secrets candidate is validated
+- **WHEN** an operator previews or saves changed secrets
+- **THEN** validation results name safe field paths and hashes but do not repeat credential values
+
+#### Scenario: Operator leaves the page
+- **WHEN** the browser navigates to another admin page
+- **THEN** the application has written no secret value to local storage, session storage, service workers, or client-side application state
+
+### Requirement: Closed host-agent protocol
+The privileged host operations agent SHALL accept only a canonical operation UUID, one fixed action enum, expected active hashes, and expected candidate hashes from the authorized runtime peer over a local Unix socket. Deployment profile is root-installed host policy and SHALL NOT be added to that request.
+
+#### Scenario: Valid apply request arrives
+- **WHEN** the authorized runtime UID submits an exact valid request
+- **THEN** the agent verifies the persisted operation and hashes before acquiring the singleton apply lock
+
+#### Scenario: Request includes path or command data
+- **WHEN** a request includes a service name, path, shell text, environment, Docker argument, or unknown field
+- **THEN** the agent rejects it before any privileged action
+
+#### Scenario: Request attempts topology selection
+- **WHEN** a request includes a profile, ingress port, upstream, Caddy path, unit, service, command, environment, Compose file, or Compose argument
+- **THEN** the agent rejects it before reading candidate documents or stopping the deployment
+
+#### Scenario: Unauthorized local peer connects
+- **WHEN** a process with an unapproved peer identity uses the socket
+- **THEN** the agent rejects the request regardless of its JSON body
+
+### Requirement: Coordinated apply with health verification
+For config, secrets, or combined apply, the host agent SHALL revalidate fixed candidate files, verify compare-and-swap hashes, create byte-identical backups, stop the fixed Truf deployment, atomically replace active files, recreate the exact root-selected runtime and edge profile, attest its fixed topology, and wait for required health before reporting success. In shared-host mode it SHALL NOT stop, restart, reload, reconfigure, or remove host Caddy, X-UI, or another unrelated service.
+
+#### Scenario: Combined apply succeeds
+- **WHEN** both candidates validate and the restarted deployment reaches Supervisor ACTIVE, PostgreSQL READY, Worker API, ingester, and projector health
+- **THEN** the operation completes successfully with resulting hashes and retained rollback evidence
+
+#### Scenario: Validation changes between preview and apply
+- **WHEN** host-side revalidation or hash verification differs from the accepted candidate operation
+- **THEN** the agent aborts before stopping the healthy runtime
+
+#### Scenario: Apply is requested while another is active
+- **WHEN** the singleton operation lock is held
+- **THEN** the second request is rejected or remains queued without overlapping lifecycle mutations
+
+#### Scenario: Shared-host profile is healthy
+- **WHEN** shared-host apply recreates a host-network runtime with no published ports and an edge bound only to fixed loopback, and all runtime and edge health checks pass
+- **THEN** the operation succeeds without a host-Caddy or X-UI lifecycle action
+
+#### Scenario: Shared-host topology drifts
+- **WHEN** runtime publishes a port, edge binds a non-loopback address, profile metadata changes, or Compose labels, mounts, network, capabilities, or Caddyfile differ from the fixed profile
+- **THEN** the agent rejects the deployment before candidate replacement or reports failed health without claiming success
+
+#### Scenario: Shared-host rollback succeeds
+- **WHEN** candidate health fails and the previous Truf runtime and edge are restored
+- **THEN** rollback completes exactly once without changing host Caddy, X-UI, or another unrelated service
+
+### Requirement: Automatic rollback and failed hold
+If the new deployment fails its bounded health check, the agent SHALL restore byte-identical backups and verify the previous deployment; if rollback also fails, it SHALL stop retrying and retain a failed-hold state and all evidence for SSH recovery.
+
+#### Scenario: New configuration fails startup
+- **WHEN** the restarted runtime cannot reach required health within the deadline
+- **THEN** the agent restores the prior active files and restarts the previous deployment
+
+#### Scenario: Rollback succeeds
+- **WHEN** the restored deployment reaches required health
+- **THEN** the operation records rolled-back status and safe failure category without claiming apply success
+
+#### Scenario: Rollback fails
+- **WHEN** neither the candidate nor restored deployment becomes healthy
+- **THEN** the agent enters failed hold, performs no replacement loop or forced authority release, and preserves backups and diagnostics
+
+### Requirement: Logical managed roots
+The generic file page SHALL address only configured logical roots with explicit read, create/replace, and delete permissions, and SHALL never accept an absolute root from a client.
+
+#### Scenario: Operator lists a managed runtime root
+- **WHEN** a valid logical root ID for logs, keycheck projections, or result projections and a canonical relative directory are requested
+- **THEN** the service returns a bounded read-only listing of permitted regular files and directories under that root
+
+#### Scenario: Operator downloads rotated result projections
+- **WHEN** the operator requests a permitted active or rotated result projection within its configured file-size bound
+- **THEN** the service returns that regular single-link file without granting mutation access or exposing the backing runtime path
+
+#### Scenario: Result projection names are allowlisted
+- **WHEN** the result-projection root is listed or read
+- **THEN** only active `scan_results.jsonl` and `found_secrets.jsonl` files and their exact six-digit generation names are visible, while locks, databases, ledgers, scan errors, temporary/quarantine directories, malformed generations, and recovery artifacts remain unavailable
+
+#### Scenario: Large result projection is downloaded
+- **WHEN** an allowed result projection is within the larger result-root byte bound
+- **THEN** the service copies and hashes an unchanged source revision into an anonymous same-volume snapshot using bounded chunks, permits only one such snapshot at a time, streams the snapshot with bounded memory, and releases the snapshot and concurrency slot after response completion or failure
+
+#### Scenario: Operator requests excluded storage
+- **WHEN** a request targets application code, config/secrets through the generic page, PostgreSQL storage, sockets, host-agent metadata, raw result bundles, or an unknown root
+- **THEN** the service rejects it without revealing host paths or existence details
+
+### Requirement: Descriptor-safe path containment
+Managed-file traversal and mutation SHALL use a retained root directory descriptor, component-wise no-follow operations, canonical relative components, and regular single-link file checks.
+
+#### Scenario: Relative traversal is attempted
+- **WHEN** a path contains an empty, dot, dot-dot, absolute, drive-qualified, backslash, NUL, over-depth, or over-length component
+- **THEN** the request is rejected before filesystem access outside the retained root
+
+#### Scenario: Symlink is swapped during access
+- **WHEN** a path component becomes a symlink between validation and open
+- **THEN** no-follow descriptor traversal fails without accessing the link target
+
+#### Scenario: Hardlink or special file is targeted
+- **WHEN** the final object is multi-linked or is not a regular file
+- **THEN** view, download, replace, and delete are rejected
+
+### Requirement: Durable bounded file mutation
+An allowed managed-file create or replace SHALL use an exclusive same-directory temporary regular file, bounded bytes, fsync, atomic descriptor-relative replacement, directory fsync, and final ownership/type/mode verification.
+
+#### Scenario: File replacement succeeds
+- **WHEN** an authorized bounded replacement is submitted against the current file hash
+- **THEN** readers observe either the complete old file or complete new file and audit records the safe before/after hashes
+
+#### Scenario: File is concurrently changed
+- **WHEN** the current file hash differs from the expected hash
+- **THEN** replacement fails without overwriting the concurrent change
+
+#### Scenario: Upload exceeds the root limit
+- **WHEN** submitted bytes exceed the configured bounded file size
+- **THEN** the service rejects and removes temporary data without changing the target
+
+### Requirement: Content-free operational audit
+Configuration, secrets, and managed-file operations SHALL record actor, typed action, logical target, timestamps, result, safe category, byte counts where applicable, and before/after hashes, but SHALL NOT record file contents or credentials.
+
+#### Scenario: Managed file is deleted
+- **WHEN** an allowed delete succeeds against the expected hash
+- **THEN** audit records the logical root/path and previous hash without retaining deleted content
+
+#### Scenario: Secrets apply fails
+- **WHEN** a secrets operation fails validation, startup, or rollback
+- **THEN** status and audit expose only the bounded failure category and document hashes

openspec/changes/add-web-operations-control-plane/specs/multisource-worker-assignments/spec.md

@@ -0,0 +1,120 @@
+## ADDED Requirements
+
+### Requirement: Protocol 2 source capabilities
+Protocol-2 worker packages SHALL advertise explicit source, worker-platform, and planning-kind capabilities for GitLab, DockerHub, and HuggingFace, and the server SHALL issue work only when the selected package supports the complete assignment capability.
+
+#### Scenario: Compatible package requests work
+- **WHEN** a protocol-2 package advertising the required capability requests a supported target
+- **THEN** the server may create an assignment using that capability
+
+#### Scenario: Package lacks planning capability
+- **WHEN** a package advertises the source but not the required planning kind
+- **THEN** the server rejects the claim without reserving a target
+
+#### Scenario: Package advertises unknown capability
+- **WHEN** a package manifest contains an unknown source, platform, or planning kind
+- **THEN** package validation fails closed
+
+### Requirement: Canonical multisource execution plans
+The server SHALL create and validate immutable execution snapshots using `exact_git_v1` for GitLab, `docker_direct_v1` for DockerHub, and `huggingface_space_v1` for HuggingFace.
+
+#### Scenario: GitLab assignment is issued
+- **WHEN** a GitLab target is claimed
+- **THEN** the assignment binds the existing exact commit and Git scan plan under `exact_git_v1`
+
+#### Scenario: DockerHub assignment is issued
+- **WHEN** a public DockerHub target is claimed
+- **THEN** the assignment binds an immutable digest reference under `docker_direct_v1` and does not depend on a mutable tag
+
+#### Scenario: HuggingFace assignment is issued
+- **WHEN** a public HuggingFace Space is claimed
+- **THEN** the assignment binds its canonical Space identifier under `huggingface_space_v1`
+
+#### Scenario: Snapshot shape does not match source
+- **WHEN** an execution snapshot's source, worker platform, or planning kind combination is invalid
+- **THEN** the server and worker reject it before scanner execution
+
+### Requirement: Fenced assignment authority for every source
+Every supported source assignment SHALL bind one user, device, target, fixed expiry, immutable execution snapshot, result reservation, and result bundle identity using the existing PostgreSQL authority model.
+
+#### Scenario: Lost claim response is retried
+- **WHEN** the server committed an assignment but the worker did not receive the response
+- **THEN** retrying the same admission request returns the same assignment and immutable execution snapshot
+
+#### Scenario: Stale worker uploads
+- **WHEN** a worker uploads with an expired, replaced, or mismatched reservation token
+- **THEN** the server rejects the upload without changing queue or bundle authority
+
+#### Scenario: Valid result commits
+- **WHEN** a valid assigned worker uploads and finalizes its bundle
+- **THEN** ingestion commits the queue result and downstream projection work exactly once
+
+### Requirement: Eligible source fallback
+The assignment service SHALL try other eligible configured sources when one supported source has no claimable target, while still issuing at most one assignment for an admission request.
+
+#### Scenario: Initially selected source is empty
+- **WHEN** the first eligible source has no claimable target and another eligible source does
+- **THEN** the same claim request may receive one assignment from the other source
+
+#### Scenario: All eligible sources are empty
+- **WHEN** no compatible source has a claimable target
+- **THEN** the claim returns no work and creates no reservation
+
+### Requirement: Legacy protocol-1 completion compatibility
+After protocol-2 cutover, the server SHALL stop issuing new claims to protocol-1 packages but SHALL continue status, terminal report, upload, receipt replay, and immutable snapshot reconciliation for already-issued protocol-1 assignments until they resolve or expire.
+
+#### Scenario: Protocol-1 package requests a new claim
+- **WHEN** a legacy GitHub/GitLab-only package requests new work after cutover
+- **THEN** the server returns an incompatibility response and creates no assignment
+
+#### Scenario: Existing protocol-1 assignment uploads
+- **WHEN** a legacy worker uploads a valid result for an assignment issued before cutover
+- **THEN** the server accepts and ingests it under its original immutable authority
+
+#### Scenario: Legacy snapshot is reconciled
+- **WHEN** the server reconstructs a lost response for an existing protocol-1 assignment
+- **THEN** it reads the original snapshot without rewriting it into protocol 2
+
+### Requirement: DockerHub end-to-end canary
+The rollout SHALL prove a bounded DockerHub `search -> enqueue -> claim -> scan -> upload -> ingestion -> projection` cycle using a public image resolved to an immutable digest before broader new-source enablement.
+
+#### Scenario: DockerHub canary succeeds
+- **WHEN** a canary producer discovers the configured public image and a compatible worker processes it
+- **THEN** the target reaches database-committed ingestion and projection under one fenced assignment
+
+#### Scenario: Mutable tag changes during canary
+- **WHEN** the discovered tag changes after queue admission
+- **THEN** the worker still scans the immutable digest bound in its assignment
+
+#### Scenario: Registry credentials would be required
+- **WHEN** the DockerHub canary target cannot be scanned without private registry credentials
+- **THEN** the worker returns a bounded inaccessible-provider result, the server applies its declared retryability, and no discovery or registry credential is transferred in the assignment
+
+### Requirement: HuggingFace remote processing
+The system SHALL support the same fenced claim-to-ingestion lifecycle for public HuggingFace Spaces without invoking the scanner on the server.
+
+#### Scenario: Public Space completes
+- **WHEN** a compatible worker claims and scans a public HuggingFace Space
+- **THEN** its result is uploaded, ingested, and projected under the bound assignment
+
+#### Scenario: Space is inaccessible without worker credentials
+- **WHEN** a tokenless worker cannot read a claimed HuggingFace Space because its repository is private, protected, removed, or otherwise unavailable
+- **THEN** it returns a non-retryable inaccessible result, the server does not retry that target, and the server discovery token is never exposed
+
+### Requirement: Worker-authoritative provider access
+The server SHALL validate canonical target and assignment authority but SHALL treat worker execution as the final provider-access check. A source SHALL NOT require a per-target server access probe, durable public-access proof, proof-freshness state, broad child-environment credential scrubbing, credential sandbox, or post-hoc redaction pipeline unless the operator separately approves an explicit OpenSpec requirement and implementation task.
+
+#### Scenario: Provider accessibility changes after discovery
+- **WHEN** a canonical target becomes inaccessible before worker execution
+- **THEN** the worker returns the source's bounded permanent or retryable provider-failure result and the server settles or retries it according to that result
+
+#### Scenario: Another source adapter is proposed
+- **WHEN** implementation would add preventive access proof or source-specific security infrastructure beyond the assignment's declared fields
+- **THEN** implementation pauses until the operator approves a dedicated requirement and task
+
+### Requirement: Credential and result secrecy
+Provider credentials, worker device tokens, authorization headers, and result contents SHALL NOT appear in operation status, audit records, Supervisor snapshots, or routine assignment logs.
+
+#### Scenario: Assignment logging occurs
+- **WHEN** any supported source assignment is created, retried, rejected, or completed
+- **THEN** logs identify bounded source and authority metadata without credential values or result payload bytes

openspec/changes/add-web-operations-control-plane/specs/web-operations-console/spec.md

@@ -0,0 +1,142 @@
+## ADDED Requirements
+
+### Requirement: Protected typed admin routes
+The operations console SHALL expose explicit server-rendered routes and exact mutation forms behind the existing random admin path, Caddy Basic authentication, trusted edge marker, exact same-origin check, CSRF validation, no-store responses, and restrictive security headers.
+
+#### Scenario: Authorized operator opens a page
+- **WHEN** Caddy authenticates the request and injects the trusted marker and operator identity
+- **THEN** the requested operations page renders escaped server-side HTML with no client-side secret persistence
+
+#### Scenario: Direct backend request lacks marker
+- **WHEN** a request reaches an admin route without the trusted edge marker
+- **THEN** the backend rejects it regardless of supplied operator headers
+
+#### Scenario: Mutation has stale or invalid CSRF
+- **WHEN** a POST has a missing, duplicate, or invalid CSRF value or wrong Origin
+- **THEN** the backend rejects the mutation without side effects
+
+#### Scenario: Unknown route or form action is submitted
+- **WHEN** a request contains an unsupported method, route shape, action, field, or duplicate field
+- **THEN** it fails closed without invoking Supervisor, database mutations, or host operations
+
+### Requirement: Trusted operator attribution
+Caddy SHALL strip any inbound operator identity header and inject the authenticated Basic-auth username, and the backend SHALL trust that identity only with the private edge marker.
+
+#### Scenario: Client spoofs operator header
+- **WHEN** a public request supplies its own operator identity header
+- **THEN** Caddy removes it and the audit actor is the authenticated Basic-auth user
+
+#### Scenario: Mutation is accepted
+- **WHEN** an authenticated operator performs a valid mutation
+- **THEN** the control or operation record and its audit event identify that operator
+
+### Requirement: Exact production ingress profiles
+The production deployment SHALL use exactly one root-installed profile: `standalone-edge-v1` or `shared-host-edge-v1`. The selected profile SHALL NOT be supplied by an admin request, host-agent request, runtime document, or other unprivileged input.
+
+#### Scenario: Standalone edge is selected
+- **WHEN** `standalone-edge-v1` is installed
+- **THEN** the managed Truf edge remains the sole Truf listener on host port 443 and retains the exact runtime-network-namespace contract
+
+#### Scenario: Shared-host edge is selected
+- **WHEN** `shared-host-edge-v1` is installed
+- **THEN** the existing root-owned host Caddy remains the sole owner of ports 80/443 and proxies only fixed Truf routes to a managed edge bound at `127.0.0.1:18766`
+
+#### Scenario: A request attempts to select topology
+- **WHEN** a request supplies a deployment mode, upstream, port, Caddy path, unit, service, command, or Compose argument
+- **THEN** it is rejected before lifecycle work
+
+### Requirement: Shared-host route confinement
+The shared-host profile SHALL install a fixed root-owned route-only host-Caddy snippet. It SHALL claim only `/api/v1/worker/*`, the exact random admin-prefix root, and that prefix's subtree. It SHALL strip inbound private and transit headers, inject an independent ingress marker and canonical client address, and preserve the managed edge's authentication, operator attribution, denylist, redacted logging, and security-header behavior without adding a listener, TLS policy, global error handler, trusted-proxy policy, catch-all, or unrelated route.
+
+#### Scenario: An unrelated host route is requested
+- **WHEN** a request does not match a Truf worker or admin path
+- **THEN** the Truf snippet does not handle or alter the request
+
+#### Scenario: The loopback Truf edge is unavailable
+- **WHEN** a matching route cannot reach `127.0.0.1:18766`
+- **THEN** host Caddy fails that Truf request without forwarding it to X-UI or another fallback upstream
+
+#### Scenario: A client supplies transit headers
+- **WHEN** a public request supplies an ingress marker, forwarded address, private edge marker, or operator identity
+- **THEN** host Caddy strips those values and injects only its reviewed ingress marker and observed client address
+
+### Requirement: Runtime overview
+The overview page SHALL report bounded structured health for Supervisor, PostgreSQL, required pipeline workers, discovery producers, queue status, active remote assignments, result bundles, operation controls, and recent operation outcomes.
+
+#### Scenario: Runtime is healthy
+- **WHEN** all required components hold valid authority and health
+- **THEN** the overview reports the runtime active and identifies each required component without exposing secrets
+
+#### Scenario: Component is unavailable
+- **WHEN** a health source times out or returns malformed state
+- **THEN** the overview reports that component unavailable without blocking the rest of the page
+
+### Requirement: Search controls
+The search page SHALL expose each core producer's structured state and typed start, stop, restart, pause, resume, and interval controls while clearly separating process lifecycle from the persistent discovery gate.
+
+#### Scenario: Operator pauses search
+- **WHEN** an operator submits pause with the current control revision
+- **THEN** the persistent discovery gate changes atomically and every producer stops admitting new discovered targets
+
+#### Scenario: Operator restarts one producer
+- **WHEN** an operator selects restart for an allowed producer ID
+- **THEN** only that managed discovery process restarts and the persistent pause state is unchanged
+
+### Requirement: Dispatch and drain controls
+The workers/dispatch page SHALL expose persistent dispatch pause/resume, drain start/cancel, drain progress, compatible package state, worker users/devices, assignment counts, and upload availability.
+
+#### Scenario: Operator pauses dispatch
+- **WHEN** the current revision is submitted to the pause action
+- **THEN** no new worker assignment can commit while valid existing uploads remain accepted
+
+#### Scenario: Operator starts drain
+- **WHEN** drain is started
+- **THEN** the page reports draining progress from authoritative assignment and bundle counts until the state becomes drained
+
+#### Scenario: Stale page attempts resume
+- **WHEN** another operator has changed the control revision before resume is submitted
+- **THEN** the console reports a revision conflict and does not overwrite the newer state
+
+### Requirement: Typed Supervisor operations
+The console SHALL use a closed Supervisor protocol for structured snapshot, allowlisted managed-source lifecycle actions, and bounded log tail, and SHALL NOT forward generic command strings.
+
+#### Scenario: Operator requests source status
+- **WHEN** the Supervisor page loads
+- **THEN** it displays structured source IDs, roles, phases, process state, restart state, and safe errors without parsing a text dashboard
+
+#### Scenario: Operator tails logs
+- **WHEN** an allowed managed source and bounded line count are submitted
+- **THEN** Supervisor returns only that source's bounded log tail
+
+#### Scenario: Input resembles a shell command
+- **WHEN** an operator submits command text, a path, or an unrecognized source ID
+- **THEN** the request is rejected and no generic Supervisor command or operating-system shell is called
+
+### Requirement: Durable asynchronous operation status
+Long-running restart and apply actions SHALL create a PostgreSQL operation record before execution and SHALL remain queryable by operation ID across runtime/admin restarts.
+
+#### Scenario: Apply restarts the admin process
+- **WHEN** the process that accepted an apply request terminates during the coordinated restart
+- **THEN** the operator can reopen the operation URL and observe reconciled success, rollback, or failure state
+
+#### Scenario: Unknown operation is requested
+- **WHEN** an operator requests an operation ID that does not exist or is not canonical
+- **THEN** the console returns not found without searching filesystem paths or host-agent state by user input
+
+### Requirement: Append-only audit view
+The audit page SHALL show bounded append-only events for accepted and completed controls, Supervisor actions, configuration/secrets operations, and managed-file mutations, including actor, action, logical target, time, result, and safe before/after identity.
+
+#### Scenario: Secret apply is audited
+- **WHEN** a secrets candidate is accepted and later applied or rolled back
+- **THEN** audit events record hashes and outcomes but no secret value, candidate bytes, authorization data, or CSRF value
+
+#### Scenario: Audit pagination is requested
+- **WHEN** an operator navigates audit history
+- **THEN** the backend returns a bounded deterministic page without unbounded database or browser output
+
+### Requirement: Existing worker API availability
+Adding the operations console SHALL NOT weaken or couple public worker endpoints to admin page availability.
+
+#### Scenario: Admin feature is disabled or unhealthy
+- **WHEN** the admin console is disabled or a Supervisor/host-agent status dependency is unavailable
+- **THEN** authenticated worker status, upload, terminal report, and receipt paths continue under their existing authority

openspec/changes/add-web-operations-control-plane/tasks.md

@@ -0,0 +1,92 @@
+## 1. PostgreSQL Operations Authority
+
+- [x] 1.1 Add an additive runtime-safety migration for the singleton operations control row, durable operation records, and append-only audit events
+- [x] 1.2 Add schema invariants, final-cutover checks, import/export handling, and migration-count fixtures for the new tables
+- [x] 1.3 Implement ScannerDB control-state reads and revision-checked discovery, dispatch, and drain mutations with atomic audit insertion
+- [x] 1.4 Enforce the discovery gate transactionally in provider admission, discovery retry, and Docker tag-resolution paths without blocking ingestion-derived work
+- [x] 1.5 Enforce the dispatch gate transactionally in remote reservation admission while preserving status, terminal report, upload, replay, expiry, and ingestion
+- [x] 1.6 Implement drain progress queries and reconciliation from live assignments and pre-commit result bundles
+- [x] 1.7 Add concurrent PostgreSQL tests for pause/admission races, stale revisions, restart persistence, drain completion, and uninterrupted uploads
+
+## 2. Discovery-Only Server Producers
+
+- [x] 2.1 Extract a discovery-only cycle for GitLab, DockerHub, and HuggingFace that performs provider work and enqueueing without scan preparation or claiming
+- [x] 2.2 Preserve DockerHub page cursors, retry-lane processing, tag resolution, and immutable target admission in the discovery role
+- [x] 2.3 Add an authenticated `discovery-producer` runtime bootstrap role and Supervisor managed-process type with structured state
+- [x] 2.4 Change the default distributed core profile to exactly GitLab, DockerHub, and HuggingFace and remove GitHub from that profile
+- [x] 2.5 Add tests proving each producer runs with backlog, respects persistent pause, reports safe state, and never enters local scanner/claim/bundle code
+
+## 3. Protocol-2 Multisource Assignments
+
+- [x] 3.1 Replace the Git-only assignment switch with source adapters that declare queue source, worker platform, planning kind, snapshot validation, and package capability
+- [x] 3.2 Implement GitLab `exact_git_v1`, DockerHub `docker_direct_v1`, and HuggingFace `huggingface_space_v1` execution-snapshot models and canonical validation
+- [x] 3.3 Generalize ScannerDB claim recovery and result-ready validation for the three planning kinds while retaining fixed reservation and device fences
+- [x] 3.4 Update source selection to try other compatible eligible queues while issuing at most one assignment per admission request
+- [x] 3.5 Advance worker/package manifests to protocol 2 with explicit source/platform/planning capabilities and fail-closed manifest validation
+- [x] 3.6 Update Windows and Linux worker clients to dispatch the bound Docker and HuggingFace scan platforms and validate protocol-2 snapshots before execution
+- [x] 3.7 Retain protocol-1 status, upload, terminal-report, receipt, and immutable reconciliation for existing assignments while refusing new protocol-1 claims
+- [x] 3.8 Add unit and PostgreSQL integration tests for capability matching, fallback, replay, expiry, stale upload rejection, source aliases, and legacy completion
+
+## 4. New-Source End-to-End Canaries
+
+Implementation guardrail: provider accessibility is finalized by the worker. New per-target server preflight/proof state, broad worker-environment credential scrubbing, or other source-specific defensive infrastructure requires separate operator approval and an explicit OpenSpec requirement/task before implementation.
+
+- [x] 4.1 Implement public DockerHub immutable-digest assignment execution without server registry-credential or layer-plan transport
+- [x] 4.2 Implement tokenless HuggingFace Space assignment execution with explicit discovery visibility filtering and non-retryable inaccessible results, without leaking server discovery credentials
+- [x] 4.3 Extend packaged-worker verification for Windows and Linux with synthetic DockerHub and HuggingFace claim-to-ingestion flows
+- [x] 4.4 Add bounded canary configuration and assertions for search, enqueue, claim, worker-classified provider failures, upload, ingestion, projection, replay, expiry, and drain without adding per-target server access proofs
+
+## 5. Shared Runtime Document Validation
+
+- [x] 5.1 Add a side-effect-free bounded YAML loader with duplicate-key rejection and secret-safe errors
+- [x] 5.2 Define strict configuration, core-profile, auth-pool, reference-integrity, package-capability, and deployment-path validation
+- [x] 5.3 Use the shared validator in preview, runtime startup, and secrets import without weakening existing runtime security checks
+- [x] 5.4 Implement fixed config/secrets candidate storage with private durable writes, SHA-256 compare-and-swap, and bounded redacted diffs
+- [x] 5.5 Add validation and concurrency tests for unknown keys, duplicate keys, invalid references, stale active hashes, stale candidates, and error redaction
+
+## 6. Typed Supervisor and Operations Services
+
+- [x] 6.1 Extend Supervisor control protocol with structured runtime/source snapshots and exact managed-source lifecycle actions
+- [x] 6.2 Add bounded log-tail actions keyed only by allowlisted managed source IDs and reject generic web command forwarding
+- [x] 6.3 Implement operation creation, lifecycle transition, bounded result reconciliation, and append-only audit service methods
+- [x] 6.4 Add Supervisor and operation-service tests for malformed actions, unknown sources, log bounds, restart races, and content-free audit records
+
+## 7. Web Operations Console
+
+- [x] 7.1 Add trusted Caddy operator-header stripping/injection and backend actor validation tied to the private edge marker
+- [x] 7.2 Add shared admin navigation and overview page with bounded runtime, queue, assignment, bundle, control, and operation health
+- [x] 7.3 Add Search pages and exact forms for persistent discovery controls plus typed producer lifecycle and interval actions
+- [x] 7.4 Add Workers/Dispatch pages and exact forms for pause/resume, drain start/cancel/progress, package compatibility, users, devices, and assignments
+- [x] 7.5 Add Supervisor status/action and bounded log pages without shell, path, or generic command inputs
+- [x] 7.6 Add config and plaintext secrets preview/save/apply pages with no-store rendering, revision/hash conflicts, and no value leakage outside the editor
+- [x] 7.7 Add durable operation-status and bounded paginated audit pages that survive runtime restart
+- [x] 7.8 Add admin API and browser tests for routes, methods, exact form shapes, actor spoofing, Origin/CSRF, stale forms, navigation, CSP, and secret non-retention
+
+## 8. Managed File Service
+
+- [x] 8.1 Define logical managed roots and per-root list/read/create-replace/delete permissions with bounded path, listing, and byte limits
+- [x] 8.2 Implement descriptor-relative Linux traversal with no-follow component opens and rejection of absolute, dot, drive, backslash, symlink, hardlink, and special-file targets
+- [x] 8.3 Implement bounded download, durable compare-and-swap create/replace, and expected-hash delete with private temporary files and directory fsync
+- [x] 8.4 Add typed Files pages/forms and content-free mutation audit events while keeping config, secrets, database, sockets, agent metadata, and raw bundles excluded
+- [x] 8.5 Add adversarial traversal, encoded traversal, symlink-swap, hardlink, special-file, limit, concurrent-replacement, and forbidden-root tests
+
+## 9. Privileged Host Operations Agent
+
+- [x] 9.1 Implement a root-owned Unix-socket agent with peer-credential checks and a closed request schema containing only operation ID, action enum, and expected hashes
+- [x] 9.2 Implement singleton locking, persisted-operation verification, host-side revalidation, fixed-path backups, and atomic config/secrets replacement
+- [x] 9.3 Implement fixed runtime/edge stop and recreation plus bounded health verification for Supervisor, PostgreSQL, Worker API, ingester, and projector
+- [x] 9.4 Implement byte-identical automatic rollback and failed-hold behavior without arbitrary services, paths, commands, or retry loops
+- [x] 9.5 Add systemd socket/service units, fixed host-managed config/candidate directories, permissions, and deployment installer validation
+- [x] 9.6 Add crash-boundary and hostile-request tests for validation, backup, replacement, restart, health failure, rollback success, rollback failure, and request-field injection
+
+## 10. Deployment Migration and Verification
+
+- [x] 10.1 Update container, code-authority, package, Caddy, systemd, and deployment fixtures for all new modules, profiles, routes, headers, sockets, and fixed managed paths
+- [x] 10.2 Add an offline migration/rollback test proving the previous image can coexist with additive tables after protocol-2 work is drained
+- [x] 10.3 Run focused unit and PostgreSQL integration suites, packaged worker verification, edge E2E, browser coverage, strict OpenSpec validation, and diff checks
+- [x] 10.4 On the approved production host, deploy one exact supported ingress profile with discovery and dispatch paused, verify protocol-2 packages plus runtime/admin/edge/agent health, complete one reconciled lifecycle operation and one successful automatic rollback drill, and only then issue or resume new work
+- [x] 10.5 Run the bounded DockerHub end-to-end canary and retain evidence for queue authority, ingestion, projection, expiry/replay, and drain
+- [x] 10.6 Enable GitLab and public HuggingFace only after canary gates pass, confirm GitHub remains excluded, and document rollback evidence
+- [x] 10.7 Add exact root-owned `standalone-edge-v1` and `shared-host-edge-v1` deployment profiles without changing the host-agent request schema
+- [x] 10.8 Add the shared-host Compose/Caddy route confinement and profile-specific lifecycle, installer, and denylist validation while preserving standalone behavior
+- [x] 10.9 Add dual-profile tests, deployment documentation, strict validation, and real hardened Caddy/Compose checks