Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,252 @@
+## ADDED Requirements
+
+### Requirement: Authoritative Docker image depth configuration
+The system SHALL obtain separate ordinary and experiment Docker images-per-repository limits from validated configuration. While this experiment is configured, the ordinary FIFO resolver limit SHALL remain three and the dedicated experiment deep limit SHALL remain ten. The system SHALL reject invalid or incompatible collection configuration before secrets, state, database, network, or worker initialization.
+
+#### Scenario: Disabled activation collection rollout
+- **WHEN** experiment activation is disabled while provenance collection remains configured
+- **THEN** Docker discovery SHALL persist fresh provenance while ordinary Docker resolver and scan admission remain paused, and the configured ordinary resolver depth SHALL remain three for later non-collection operation
+
+#### Scenario: Reviewed false-to-true activation
+- **WHEN** a disabled collection is reviewed and operational `enabled` changes from false to true without another configuration change
+- **THEN** the frozen semantic configuration hash SHALL remain unchanged and enabled state SHALL be enforced separately at each activation or claim boundary
+
+#### Scenario: Experiment depth ten
+- **WHEN** the dedicated experiment resolver receives the validated deep limit of 10
+- **THEN** it SHALL be allowed to select up to ten deterministic distinct image graphs without a hidden lower clamp
+
+#### Scenario: Invalid depth
+- **WHEN** Docker image depth is boolean, non-integer, below 1, above 10, or incompatible with the configured candidate-tag depth
+- **THEN** startup SHALL fail before any runtime side effect
+
+#### Scenario: Mixed discovery policies
+- **WHEN** query overrides produce different effective pages or per-page policy values for experiment queries
+- **THEN** startup SHALL fail because the current experiment pass authority represents one discovery policy
+
+#### Scenario: Incompatible ordinary refresh
+- **WHEN** experiment activation is enabled with periodic ordinary repository refresh greater than zero
+- **THEN** startup SHALL fail before runtime side effects
+
+### Requirement: Durable many-to-many discovery provenance
+The system SHALL record every fresh Docker query-to-repository observation in the same transaction as page admission while preserving one physical repository queue identity.
+
+#### Scenario: Repository observed by two queries
+- **WHEN** two Docker queries observe the same normalized repository anchor
+- **THEN** the system SHALL retain two provenance relations and one repository queue row
+
+#### Scenario: Page admission rolls back
+- **WHEN** provenance persistence or repository admission fails
+- **THEN** neither the page admission nor its provenance and retry progress SHALL commit partially
+
+#### Scenario: Page admission races authority validation
+- **WHEN** page ingestion and experiment validation execute concurrently at READ COMMITTED
+- **THEN** both SHALL serialize on the experiment authority row and validation SHALL NOT observe a half-committed page, hold event, queue, binding, or reservation transition
+
+#### Scenario: Pre-migration deep marker
+- **WHEN** runner state contains a deep-dispatch marker but PostgreSQL has no complete deep pass for the pinned collection generation, policy, and ordered queries
+- **THEN** each configured query SHALL be forced through one generation-current deep provenance pass before the normal 72-hour policy resumes
+
+#### Scenario: Underreported result count
+- **WHEN** a Docker discovery page reports a total count below its current absolute result bound or incoherent with an empty continuation
+- **THEN** that page SHALL fail or be delegated and SHALL NOT provide terminal pass evidence
+
+### Requirement: Fresh complete cohort eligibility
+The experiment SHALL use only fresh observations made under its pinned policy and SHALL remain in collecting state until one complete deep observation pass covers every configured query. It SHALL then preserve every query authority row and select up to ten eligible previously unscanned repositories per query, including an explicit actual count of zero when the completed pass produced none.
+
+#### Scenario: Query has insufficient candidates
+- **WHEN** the complete pinned deep pass leaves a configured query with fewer than ten fresh eligible repositories
+- **THEN** the cohort SHALL retain exactly the available fresh repositories, SHALL NOT substitute historical or previously scanned targets, and SHALL report the unavailable count against the desired quota of ten
+
+#### Scenario: Collection pass is incomplete
+- **WHEN** no complete pinned deep pass covers all 61 configured queries
+- **THEN** planning SHALL remain unavailable even if partial observations exist
+
+#### Scenario: Legacy attribution exists
+- **WHEN** a repository has only historical first-inserter queue attribution
+- **THEN** that evidence SHALL NOT satisfy fresh experiment eligibility
+
+### Requirement: Bounded fair experiment cohort
+The system SHALL select up to ten fresh repositories per configured query after the complete pass, resolve one newest distinct image from each selected repository when an eligible unseen image exists, and select image ranks 2 through 10 from one deterministic version-rich image-bearing repository for each applicable query, subject to a transactional ceiling of 1,200 unique immutable image targets. A selected repository that exhausts fresh replacements without an eligible image SHALL remain explicit terminal image-level scarcity rather than causing historical substitution.
+
+#### Scenario: Complete 61-query authority
+- **WHEN** all 61 query rows are planned from a complete pinned deep pass
+- **THEN** the planned selections SHALL be no more than 1,159 before cross-query deduplication, honest scarcity SHALL reduce rather than inflate that count, and physical experiment targets SHALL never exceed 1,200
+
+#### Scenario: Conclusive breadth zero
+- **WHEN** a breadth repository resolves conclusively with no eligible immutable image
+- **THEN** the resolver SHALL deterministically select the next fresh ranked repository and, when that pool is exhausted, SHALL terminally mark only that membership `skipped` with exact hashed `no_eligible_physical_target` evidence without consuming a physical target slot
+
+#### Scenario: Complete breadth authority
+- **WHEN** the experiment activates or completes
+- **THEN** every selected breadth membership SHALL have either a valid rank-one selection bound to an eligible physical experiment target or exact terminal image-unavailability evidence, while zero-member queries SHALL remain explicit nonmissing repository scarcity evidence
+
+#### Scenario: Query has no image-bearing membership
+- **WHEN** every selected repository for a query terminates with valid image-unavailability evidence
+- **THEN** that query SHALL have no deep probe and SHALL remain separately reportable without blocking activation
+
+#### Scenario: Concurrent shared image selection
+- **WHEN** concurrent query selections resolve to the same normalized immutable image
+- **THEN** the image SHALL consume one physical target slot and SHALL retain every query selection relation
+
+### Requirement: Round-robin experiment scheduling
+The system SHALL schedule breadth and depth work by pinned query ordinal rather than repository FIFO.
+
+#### Scenario: Breadth precedes depth
+- **WHEN** experiment targets become claimable
+- **THEN** ranks 2-3 SHALL remain unclaimable until every physical rank-one target has a terminal latest experiment binding, and ranks 4-10 SHALL similarly wait for ranks 2-3
+
+#### Scenario: Earlier wave is refunded
+- **WHEN** a terminal attempt in an earlier wave is refunded and its physical target returns to pending
+- **THEN** the earlier completion barrier SHALL reopen and later waves SHALL stop until its replacement attempt completes terminally
+
+#### Scenario: Partial execution
+- **WHEN** runtime stops before the experiment completes
+- **THEN** completed work SHALL remain evenly attributable to the earliest unfinished round-robin wave
+
+### Requirement: Deterministic distinct graph selection
+The resolver SHALL preserve the existing first-three selection semantics and SHALL select ranks 4 through 10 deterministically by marginal layer novelty and stable temporal/identity tie breaks.
+
+#### Scenario: Duplicate tags share a graph
+- **WHEN** multiple tags resolve to an identical ordered layer graph
+- **THEN** the graph SHALL consume at most one image rank
+
+#### Scenario: Fewer than ten valid graphs
+- **WHEN** a deep repository has fewer than ten valid distinct image graphs
+- **THEN** the resolver SHALL record the actual depth without inserting invalid or duplicate replacements
+
+### Requirement: Reversible fenced backlog hold
+The system SHALL place non-cohort Docker repository anchors into a durable experiment-scoped cold state only when no active lease, resolver token, reservation, quarantine, or unrelated hold prevents the transition, and SHALL preserve the exact prior state for reviewed reactivation.
+
+#### Scenario: Unfenced non-cohort anchor
+- **WHEN** an eligible non-cohort unresolved repository is covered by the reviewed experiment hold policy
+- **THEN** it SHALL become cold with a durable policy event and SHALL be ignored by ordinary resolver claims
+
+#### Scenario: Independently fenced anchor
+- **WHEN** a repository has an active or unrelated safety fence
+- **THEN** the experiment SHALL leave it unchanged and record the hold conflict
+
+#### Scenario: Reviewed release
+- **WHEN** the experiment hold is released
+- **THEN** the experiment SHALL already be completed and only unreversed cold events owned by that experiment SHALL restore their exact prior queue states
+
+#### Scenario: Unsafe early release
+- **WHEN** a reviewed release is requested from holding, resolving, active, draining, or held
+- **THEN** release SHALL be rejected without changing owned cold events or experiment state
+
+#### Scenario: Full reviewed search surface
+- **WHEN** a reviewed hold or reactivation covers more than 100,000 rows up to the strict `61 * 30 * 100` search maximum
+- **THEN** deterministic bounded parts SHALL cover every row, aggregate counts/hashes SHALL remain authoritative, and overflow beyond the reviewed 250,000-row ceiling SHALL fail rather than omit rows
+
+### Requirement: Fresh target safety
+The experiment SHALL scan only previously unseen immutable image targets and SHALL NOT automatically reactivate completed, failed, quarantined, or independently cold targets.
+
+#### Scenario: Selected image already completed
+- **WHEN** a resolver selection conflicts with an immutable target already in done state
+- **THEN** the system SHALL record hashed skip evidence and deterministically continue to the next eligible fresh graph/alias without requeueing the completed target or retrying the identical selected set
+
+#### Scenario: Selected image is quarantined
+- **WHEN** a resolver selection conflicts with quarantined work
+- **THEN** the system SHALL skip it and continue to the next eligible fresh candidate or replacement repository and SHALL NOT bypass quarantine
+
+#### Scenario: No safe replacement remains
+- **WHEN** every fresh candidate is terminal, independently cold, quarantined, fenced, or otherwise ineligible
+- **THEN** only that repository membership SHALL become terminal `skipped` with exact hashed image-unavailability evidence, no experiment target or capacity slot SHALL be consumed, and no candidate SHALL be reactivated
+
+#### Scenario: Terminal scarcity evidence drifts
+- **WHEN** terminal repository skip state lacks its exact reason, repository identity, ordinal, or canonical evidence hash
+- **THEN** experiment authority validation SHALL move the experiment to held state before activation or further mutation
+
+### Requirement: Durable manifest and layer attribution
+The system SHALL persist each selected immutable manifest and its ordered layer descriptors, including positions from base and top, and SHALL associate findings only when an exact layer digest is present.
+
+#### Scenario: Exact layer digest finding
+- **WHEN** a finding reports a Docker layer digest present at one or more manifest positions
+- **THEN** the system SHALL persist every exact matching base/top position for that image
+
+#### Scenario: Finding has no layer digest
+- **WHEN** scanner evidence does not identify an exact layer digest
+- **THEN** the report SHALL count the finding as layer-unattributed and SHALL NOT infer a position
+
+### Requirement: Reservation-bound experiment evidence
+The system SHALL bind experiment targets to scan reservations atomically and SHALL use those bindings to exclude historical or unrelated scans from experiment results.
+
+#### Scenario: Experiment target is reserved
+- **WHEN** an experiment target receives scan capacity and a queue lease
+- **THEN** its experiment binding, reservation, and queue transition SHALL commit atomically
+
+#### Scenario: Reservation is retried
+- **WHEN** the same experiment target requires a fenced retry
+- **THEN** reporting SHALL preserve each bound attempt while deduplicating final physical target totals
+
+#### Scenario: Admission capacity is saturated
+- **WHEN** experiment admission cannot reserve pipeline or quarantine capacity
+- **THEN** the aborted admission intent and experiment `held` transition SHALL commit atomically under the same experiment authority lock
+
+### Requirement: Safe experiment reporting
+The system SHALL produce aggregate physical and per-query reports comparing image ranks 1-3 with ranks 4-10, including deduplicated findings, credential identities, keycheck outcomes, scan duration/errors, layer positions, overlap, coverage, and marginal minimum-rank yield without exposing secret material.
+
+#### Scenario: Image belongs to multiple queries
+- **WHEN** one physical image selection is attributed to multiple queries
+- **THEN** global totals SHALL count it once while each relevant query report SHALL receive attribution and overlap SHALL be explicit
+
+#### Scenario: Identity repeats at later rank
+- **WHEN** a detector-secret or credential identity first appears at rank 2 and appears again at rank 7
+- **THEN** marginal yield SHALL assign that identity to rank 2 and SHALL NOT recount it as new in ranks 4-10
+
+#### Scenario: Report contains sensitive evidence
+- **WHEN** aggregate reporting reads findings or keycheck records
+- **THEN** output SHALL contain only approved IDs, hashes, enums, counts, durations, and positions and SHALL NOT contain raw credentials or secret-bearing excerpts
+
+### Requirement: Fail-closed experiment authority
+Experiment collection and activation SHALL run only on managed PostgreSQL final-cutover with search-mode immutable-digest discovery. The canonical semantic configuration hash SHALL exclude operational `enabled` and include the pinned collection generation, exact ordered effective query policies, and Docker platform filter, OS, architecture, and candidate-count values. Enabled state SHALL be validated independently. The original cohort plan hash SHALL remain immutable across deterministic exclusions/replacements, and a second frozen runtime-selection hash SHALL bind effective repositories, terminal image-scarcity evidence, and the executed deep-probe choice before activation. Experiment authority validation and every experiment-owned mutation SHALL use an experiment-row-first locked protocol. An active experiment SHALL transition to held state when ordered queries, selector version, configuration hash, runtime-selection hash, terminal skip evidence, owned hold-event/queue history, capacity, or fencing invariants drift.
+
+#### Scenario: Query list changes during execution
+- **WHEN** the configured ordered query hash differs from the pinned experiment hash
+- **THEN** new experiment claims SHALL stop and the experiment SHALL enter held state
+
+#### Scenario: File-queue fallback is active
+- **WHEN** PostgreSQL final-cutover authority is unavailable
+- **THEN** experiment activation and mutation SHALL be rejected
+
+#### Scenario: Executed deep probe drifts
+- **WHEN** an executed `is_deep_probe` choice differs from the frozen runtime-selection hash
+- **THEN** new claims SHALL stop and the experiment SHALL enter held state
+
+#### Scenario: Disabled during holding
+- **WHEN** operational `enabled` becomes false while the experiment is in holding
+- **THEN** the experiment SHALL transition fail-closed to held
+
+#### Scenario: Long remote graph resolution
+- **WHEN** candidate manifest resolution spans the original 300-second lease
+- **THEN** the worker SHALL renew before and between bounded remote stages under its exact owner/generation/token, verify the token after remote work, and perform no completion write after lease loss
+
+#### Scenario: Runtime stops during resolver work
+- **WHEN** authority validation finds an expired resolver fence left by an interrupted runtime
+- **THEN** it SHALL atomically return the affected membership to pending and enter `held(stale_resolver_fence)`, and a later claim MAY resume `resolving` only after full authority validation succeeds and no resolver fence remains; no other held reason SHALL automatically resume
+
+#### Scenario: Repeated non-conclusive remote failures
+- **WHEN** a resolver membership consumes three non-conclusive remote attempts
+- **THEN** only that membership SHALL terminate instead of retrying forever or pinning the breadth barrier
+- **AND** breadth work SHALL record exact hashed `remote_unavailable_after_attempt_limit` scarcity without a target slot, while deep work SHALL preserve already selected images and close at its achieved depth
+
+#### Scenario: Systemic resolver conflict reaches its ceiling
+- **WHEN** repeated evidence, capacity, configuration, or authority conflicts reach their safety ceiling
+- **THEN** the experiment SHALL remain fail-closed in held state and SHALL NOT misclassify the conflict as target-local remote scarcity
+
+#### Scenario: Legacy attempt-limit hold resumes under the simplified policy
+- **WHEN** a fully validated claim encounters exactly one unfenced membership persisted as `held(resolver_attempt_limit)` by the earlier policy
+- **THEN** it SHALL terminalize only that membership with exact hashed remote-unavailable evidence, clear the legacy experiment hold, and continue resolving the remaining cohort
+
+#### Scenario: Reviewed disposition of a genuine attempt-limit hold
+- **WHEN** an operator reviews an exact hash-only manifest for the single membership held by genuine remote failures and approves its SHA while sources are stopped
+- **THEN** the system SHALL atomically use the first unchanged eligible fresh replacement repository and reset only that membership's attempts, or SHALL terminally record exact `remote_unavailable_after_attempt_limit` scarcity when the reviewed fresh replacement pool is exhausted
+- **AND** it SHALL append a one-time immutable audit row, resume the experiment, preserve the global three-attempt policy, consume no target slot for a skip, and never expose or reactivate historical target identity
+
+#### Scenario: Reviewed refund of attempts consumed by a retired local defect
+- **WHEN** stopped-source offline review proves exactly two target-bound occurrences of the retired zero-graph limit defect for a membership and an operator approves the exact private manifest SHA
+- **THEN** the system SHALL refund exactly two attempts, append a one-time immutable audit row, clear only that membership's obsolete local error, and resume the attempt-limit-held experiment without exposing the target identity
+- **AND** memberships containing only partial or other remote failures SHALL remain unchanged, and the same recovery kind SHALL never refund a membership twice
+
+#### Scenario: Owned hold history drifts
+- **WHEN** an experiment-owned hold event, queue state, config/policy/manifest hash, audit hash, or reversal is changed outside the reviewed protocol
+- **THEN** continuous authority validation SHALL stop mutation and hold the experiment before new rows or events commit