truf-server/openspec/changes/run-bounded-docker-depth-experiment/specs/docker-depth-experiment/spec.md

ADDED Requirements

Requirement: Authoritative Docker image depth configuration

The system SHALL obtain separate ordinary and experiment Docker images-per-repository limits from validated configuration. While this experiment is configured, the ordinary FIFO resolver limit SHALL remain three and the dedicated experiment deep limit SHALL remain ten. The system SHALL reject invalid or incompatible collection configuration before secrets, state, database, network, or worker initialization.

Scenario: Disabled activation collection rollout

• WHEN experiment activation is disabled while provenance collection remains configured
• THEN Docker discovery SHALL persist fresh provenance while ordinary Docker resolver and scan admission remain paused, and the configured ordinary resolver depth SHALL remain three for later non-collection operation

Scenario: Reviewed false-to-true activation

• WHEN a disabled collection is reviewed and operational enabled changes from false to true without another configuration change
• THEN the frozen semantic configuration hash SHALL remain unchanged and enabled state SHALL be enforced separately at each activation or claim boundary

Scenario: Experiment depth ten

• WHEN the dedicated experiment resolver receives the validated deep limit of 10
• THEN it SHALL be allowed to select up to ten deterministic distinct image graphs without a hidden lower clamp

Scenario: Invalid depth

• WHEN Docker image depth is boolean, non-integer, below 1, above 10, or incompatible with the configured candidate-tag depth
• THEN startup SHALL fail before any runtime side effect

Scenario: Mixed discovery policies

• WHEN query overrides produce different effective pages or per-page policy values for experiment queries
• THEN startup SHALL fail because the current experiment pass authority represents one discovery policy

Scenario: Incompatible ordinary refresh

• WHEN experiment activation is enabled with periodic ordinary repository refresh greater than zero
• THEN startup SHALL fail before runtime side effects

Requirement: Durable many-to-many discovery provenance

The system SHALL record every fresh Docker query-to-repository observation in the same transaction as page admission while preserving one physical repository queue identity.

Scenario: Repository observed by two queries

• WHEN two Docker queries observe the same normalized repository anchor
• THEN the system SHALL retain two provenance relations and one repository queue row

Scenario: Page admission rolls back

• WHEN provenance persistence or repository admission fails
• THEN neither the page admission nor its provenance and retry progress SHALL commit partially

Scenario: Page admission races authority validation

• WHEN page ingestion and experiment validation execute concurrently at READ COMMITTED
• THEN both SHALL serialize on the experiment authority row and validation SHALL NOT observe a half-committed page, hold event, queue, binding, or reservation transition

Scenario: Pre-migration deep marker

• WHEN runner state contains a deep-dispatch marker but PostgreSQL has no complete deep pass for the pinned collection generation, policy, and ordered queries
• THEN each configured query SHALL be forced through one generation-current deep provenance pass before the normal 72-hour policy resumes

Scenario: Underreported result count

• WHEN a Docker discovery page reports a total count below its current absolute result bound or incoherent with an empty continuation
• THEN that page SHALL fail or be delegated and SHALL NOT provide terminal pass evidence

Requirement: Fresh complete cohort eligibility

The experiment SHALL use only fresh observations made under its pinned policy and SHALL remain in collecting state until one complete deep observation pass covers every configured query. It SHALL then preserve every query authority row and select up to ten eligible previously unscanned repositories per query, including an explicit actual count of zero when the completed pass produced none.

Scenario: Query has insufficient candidates

• WHEN the complete pinned deep pass leaves a configured query with fewer than ten fresh eligible repositories
• THEN the cohort SHALL retain exactly the available fresh repositories, SHALL NOT substitute historical or previously scanned targets, and SHALL report the unavailable count against the desired quota of ten

Scenario: Collection pass is incomplete

• WHEN no complete pinned deep pass covers all 61 configured queries
• THEN planning SHALL remain unavailable even if partial observations exist

Scenario: Legacy attribution exists

• WHEN a repository has only historical first-inserter queue attribution
• THEN that evidence SHALL NOT satisfy fresh experiment eligibility

Requirement: Bounded fair experiment cohort

The system SHALL select up to ten fresh repositories per configured query after the complete pass, resolve one newest distinct image from each selected repository when an eligible unseen image exists, and select image ranks 2 through 10 from one deterministic version-rich image-bearing repository for each applicable query, subject to a transactional ceiling of 1,200 unique immutable image targets. A selected repository that exhausts fresh replacements without an eligible image SHALL remain explicit terminal image-level scarcity rather than causing historical substitution.

Scenario: Complete 61-query authority

• WHEN all 61 query rows are planned from a complete pinned deep pass
• THEN the planned selections SHALL be no more than 1,159 before cross-query deduplication, honest scarcity SHALL reduce rather than inflate that count, and physical experiment targets SHALL never exceed 1,200

Scenario: Conclusive breadth zero

• WHEN a breadth repository resolves conclusively with no eligible immutable image
• THEN the resolver SHALL deterministically select the next fresh ranked repository and, when that pool is exhausted, SHALL terminally mark only that membership skipped with exact hashed no_eligible_physical_target evidence without consuming a physical target slot

Scenario: Complete breadth authority

• WHEN the experiment activates or completes
• THEN every selected breadth membership SHALL have either a valid rank-one selection bound to an eligible physical experiment target or exact terminal image-unavailability evidence, while zero-member queries SHALL remain explicit nonmissing repository scarcity evidence

Scenario: Query has no image-bearing membership

• WHEN every selected repository for a query terminates with valid image-unavailability evidence
• THEN that query SHALL have no deep probe and SHALL remain separately reportable without blocking activation

Scenario: Concurrent shared image selection

• WHEN concurrent query selections resolve to the same normalized immutable image
• THEN the image SHALL consume one physical target slot and SHALL retain every query selection relation

Requirement: Round-robin experiment scheduling

The system SHALL schedule breadth and depth work by pinned query ordinal rather than repository FIFO.

Scenario: Breadth precedes depth

• WHEN experiment targets become claimable
• THEN ranks 2-3 SHALL remain unclaimable until every physical rank-one target has a terminal latest experiment binding, and ranks 4-10 SHALL similarly wait for ranks 2-3

Scenario: Earlier wave is refunded

• WHEN a terminal attempt in an earlier wave is refunded and its physical target returns to pending
• THEN the earlier completion barrier SHALL reopen and later waves SHALL stop until its replacement attempt completes terminally

Scenario: Partial execution

• WHEN runtime stops before the experiment completes
• THEN completed work SHALL remain evenly attributable to the earliest unfinished round-robin wave

Requirement: Deterministic distinct graph selection

The resolver SHALL preserve the existing first-three selection semantics and SHALL select ranks 4 through 10 deterministically by marginal layer novelty and stable temporal/identity tie breaks.

Scenario: Duplicate tags share a graph

• WHEN multiple tags resolve to an identical ordered layer graph
• THEN the graph SHALL consume at most one image rank

Scenario: Fewer than ten valid graphs

• WHEN a deep repository has fewer than ten valid distinct image graphs
• THEN the resolver SHALL record the actual depth without inserting invalid or duplicate replacements

Requirement: Reversible fenced backlog hold

The system SHALL place non-cohort Docker repository anchors into a durable experiment-scoped cold state only when no active lease, resolver token, reservation, quarantine, or unrelated hold prevents the transition, and SHALL preserve the exact prior state for reviewed reactivation.

Scenario: Unfenced non-cohort anchor

• WHEN an eligible non-cohort unresolved repository is covered by the reviewed experiment hold policy
• THEN it SHALL become cold with a durable policy event and SHALL be ignored by ordinary resolver claims

Scenario: Independently fenced anchor

• WHEN a repository has an active or unrelated safety fence
• THEN the experiment SHALL leave it unchanged and record the hold conflict

Scenario: Reviewed release

• WHEN the experiment hold is released
• THEN the experiment SHALL already be completed and only unreversed cold events owned by that experiment SHALL restore their exact prior queue states

Scenario: Unsafe early release

• WHEN a reviewed release is requested from holding, resolving, active, draining, or held
• THEN release SHALL be rejected without changing owned cold events or experiment state

Scenario: Full reviewed search surface

• WHEN a reviewed hold or reactivation covers more than 100,000 rows up to the strict 61 * 30 * 100 search maximum
• THEN deterministic bounded parts SHALL cover every row, aggregate counts/hashes SHALL remain authoritative, and overflow beyond the reviewed 250,000-row ceiling SHALL fail rather than omit rows

Requirement: Fresh target safety

The experiment SHALL scan only previously unseen immutable image targets and SHALL NOT automatically reactivate completed, failed, quarantined, or independently cold targets.

Scenario: Selected image already completed

• WHEN a resolver selection conflicts with an immutable target already in done state
• THEN the system SHALL record hashed skip evidence and deterministically continue to the next eligible fresh graph/alias without requeueing the completed target or retrying the identical selected set

Scenario: Selected image is quarantined

• WHEN a resolver selection conflicts with quarantined work
• THEN the system SHALL skip it and continue to the next eligible fresh candidate or replacement repository and SHALL NOT bypass quarantine

Scenario: No safe replacement remains

• WHEN every fresh candidate is terminal, independently cold, quarantined, fenced, or otherwise ineligible
• THEN only that repository membership SHALL become terminal skipped with exact hashed image-unavailability evidence, no experiment target or capacity slot SHALL be consumed, and no candidate SHALL be reactivated

Scenario: Terminal scarcity evidence drifts

• WHEN terminal repository skip state lacks its exact reason, repository identity, ordinal, or canonical evidence hash
• THEN experiment authority validation SHALL move the experiment to held state before activation or further mutation

Requirement: Durable manifest and layer attribution

The system SHALL persist each selected immutable manifest and its ordered layer descriptors, including positions from base and top, and SHALL associate findings only when an exact layer digest is present.

Scenario: Exact layer digest finding

• WHEN a finding reports a Docker layer digest present at one or more manifest positions
• THEN the system SHALL persist every exact matching base/top position for that image

Scenario: Finding has no layer digest

• WHEN scanner evidence does not identify an exact layer digest
• THEN the report SHALL count the finding as layer-unattributed and SHALL NOT infer a position

Requirement: Reservation-bound experiment evidence

The system SHALL bind experiment targets to scan reservations atomically and SHALL use those bindings to exclude historical or unrelated scans from experiment results.

Scenario: Experiment target is reserved

• WHEN an experiment target receives scan capacity and a queue lease
• THEN its experiment binding, reservation, and queue transition SHALL commit atomically

Scenario: Reservation is retried

• WHEN the same experiment target requires a fenced retry
• THEN reporting SHALL preserve each bound attempt while deduplicating final physical target totals

Scenario: Admission capacity is saturated

• WHEN experiment admission cannot reserve pipeline or quarantine capacity
• THEN the aborted admission intent and experiment held transition SHALL commit atomically under the same experiment authority lock

Requirement: Safe experiment reporting

The system SHALL produce aggregate physical and per-query reports comparing image ranks 1-3 with ranks 4-10, including deduplicated findings, credential identities, keycheck outcomes, scan duration/errors, layer positions, overlap, coverage, and marginal minimum-rank yield without exposing secret material.

Scenario: Image belongs to multiple queries

• WHEN one physical image selection is attributed to multiple queries
• THEN global totals SHALL count it once while each relevant query report SHALL receive attribution and overlap SHALL be explicit

Scenario: Identity repeats at later rank

• WHEN a detector-secret or credential identity first appears at rank 2 and appears again at rank 7
• THEN marginal yield SHALL assign that identity to rank 2 and SHALL NOT recount it as new in ranks 4-10

Scenario: Report contains sensitive evidence

• WHEN aggregate reporting reads findings or keycheck records
• THEN output SHALL contain only approved IDs, hashes, enums, counts, durations, and positions and SHALL NOT contain raw credentials or secret-bearing excerpts

Requirement: Fail-closed experiment authority

Experiment collection and activation SHALL run only on managed PostgreSQL final-cutover with search-mode immutable-digest discovery. The canonical semantic configuration hash SHALL exclude operational enabled and include the pinned collection generation, exact ordered effective query policies, and Docker platform filter, OS, architecture, and candidate-count values. Enabled state SHALL be validated independently. The original cohort plan hash SHALL remain immutable across deterministic exclusions/replacements, and a second frozen runtime-selection hash SHALL bind effective repositories, terminal image-scarcity evidence, and the executed deep-probe choice before activation. Experiment authority validation and every experiment-owned mutation SHALL use an experiment-row-first locked protocol. An active experiment SHALL transition to held state when ordered queries, selector version, configuration hash, runtime-selection hash, terminal skip evidence, owned hold-event/queue history, capacity, or fencing invariants drift.

Scenario: Query list changes during execution

• WHEN the configured ordered query hash differs from the pinned experiment hash
• THEN new experiment claims SHALL stop and the experiment SHALL enter held state

Scenario: File-queue fallback is active

• WHEN PostgreSQL final-cutover authority is unavailable
• THEN experiment activation and mutation SHALL be rejected

Scenario: Executed deep probe drifts

• WHEN an executed is_deep_probe choice differs from the frozen runtime-selection hash
• THEN new claims SHALL stop and the experiment SHALL enter held state

Scenario: Disabled during holding

• WHEN operational enabled becomes false while the experiment is in holding
• THEN the experiment SHALL transition fail-closed to held

Scenario: Long remote graph resolution

• WHEN candidate manifest resolution spans the original 300-second lease
• THEN the worker SHALL renew before and between bounded remote stages under its exact owner/generation/token, verify the token after remote work, and perform no completion write after lease loss

Scenario: Runtime stops during resolver work

• WHEN authority validation finds an expired resolver fence left by an interrupted runtime
• THEN it SHALL atomically return the affected membership to pending and enter held(stale_resolver_fence), and a later claim MAY resume resolving only after full authority validation succeeds and no resolver fence remains; no other held reason SHALL automatically resume

Scenario: Repeated non-conclusive remote failures

• WHEN a resolver membership consumes three non-conclusive remote attempts
• THEN only that membership SHALL terminate instead of retrying forever or pinning the breadth barrier
• AND breadth work SHALL record exact hashed remote_unavailable_after_attempt_limit scarcity without a target slot, while deep work SHALL preserve already selected images and close at its achieved depth

Scenario: Systemic resolver conflict reaches its ceiling

• WHEN repeated evidence, capacity, configuration, or authority conflicts reach their safety ceiling
• THEN the experiment SHALL remain fail-closed in held state and SHALL NOT misclassify the conflict as target-local remote scarcity

Scenario: Legacy attempt-limit hold resumes under the simplified policy

• WHEN a fully validated claim encounters exactly one unfenced membership persisted as held(resolver_attempt_limit) by the earlier policy
• THEN it SHALL terminalize only that membership with exact hashed remote-unavailable evidence, clear the legacy experiment hold, and continue resolving the remaining cohort

Scenario: Reviewed disposition of a genuine attempt-limit hold

• WHEN an operator reviews an exact hash-only manifest for the single membership held by genuine remote failures and approves its SHA while sources are stopped
• THEN the system SHALL atomically use the first unchanged eligible fresh replacement repository and reset only that membership's attempts, or SHALL terminally record exact remote_unavailable_after_attempt_limit scarcity when the reviewed fresh replacement pool is exhausted
• AND it SHALL append a one-time immutable audit row, resume the experiment, preserve the global three-attempt policy, consume no target slot for a skip, and never expose or reactivate historical target identity

Scenario: Reviewed refund of attempts consumed by a retired local defect

• WHEN stopped-source offline review proves exactly two target-bound occurrences of the retired zero-graph limit defect for a membership and an operator approves the exact private manifest SHA
• THEN the system SHALL refund exactly two attempts, append a one-time immutable audit row, clear only that membership's obsolete local error, and resume the attempt-limit-held experiment without exposing the target identity
• AND memberships containing only partial or other remote failures SHALL remain unchanged, and the same recovery kind SHALL never refund a membership twice

Scenario: Owned hold history drifts

• WHEN an experiment-owned hold event, queue state, config/policy/manifest hash, audit hash, or reversal is changed outside the reviewed protocol
• THEN continuous authority validation SHALL stop mutation and hold the experiment before new rows or events commit