Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-09-12
@@ -0,0 +1,134 @@
+## Context
+
+The bounded Docker-depth experiment showed that all five currently alive
+credentials were already present in the newest selected image. Image ranks
+2-10 consumed about 7.5 scanner-hours, added twelve globally new historical
+credentials, and added no currently alive credential. The next experiment
+therefore spends a larger bounded budget on repository breadth while scanning
+only the newest eligible immutable image from each selected repository.
+
+The complete frozen 61-query discovery pass contains enough fresh provenance
+for this cohort. Most non-cohort repository anchors are currently held by the
+depth experiment and can become eligible only after that experiment completes
+and its exact reviewed cold events are reversed. PostgreSQL remains the sole
+authority for cohort, leases, reservations, policy events, capacity, and
+reporting evidence.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Select exactly 2,000 previously unscanned physical repositories from the
+  existing complete frozen discovery pass without using yield.
+- Balance selection across the 52 keywords that have an eligible remaining
+  pool: 38 repositories each, then one additional repository for the first 24
+  still-eligible keywords in pinned query order.
+- Physically deduplicate repositories across keywords while retaining their
+  frozen many-to-many keyword provenance.
+- Resolve and scan at most one newest eligible immutable image per selected
+  repository under the existing experiment authority and fencing protocol.
+- Hand authority over only after the depth experiment is completed and its
+  reviewed release has restored every owned non-cohort row.
+- Report globally deduplicated credential and currently-alive yield, scanner
+  cost, and secret-safe per-keyword attribution.
+
+**Non-Goals:**
+
+- Do not scan older image ranks or reinterpret `rank1` as a filesystem layer.
+- Do not infer an adaptive-depth trigger from the zero depth-only alive sample.
+- Do not reactivate scanned, failed, quarantined, independently cold, fenced,
+  or incompletely released repository rows.
+- Do not run the depth and breadth experiments concurrently.
+- Do not expose repositories, image targets, credentials, hashes, or raw
+  scanner evidence in operator output.
+
+## Decisions
+
+### Versioned experiment profiles
+
+The existing Docker experiment tables and worker protocol are reused. The
+reviewed selector version identifies one of two exact code-pinned profiles:
+the existing depth profile remains unchanged, while the breadth profile fixes
+61 queries, a per-query ceiling of 39, shallow/deep image depth of one, and a
+global physical target limit of 2,000. Configuration validation accepts only
+one complete reviewed profile; arbitrary mixtures remain invalid.
+
+The breadth profile's theoretical capacity is the global physical limit, not
+`query_count * repositories_per_query`, because cross-keyword physical
+deduplication and the global stop are part of planning. Schema bounds are
+widened only enough to store the reviewed profile. Runtime authority still
+compares every persisted value with the exact supplied reviewed profile.
+
+### Deterministic balanced physical selection
+
+Planning reads the existing complete frozen deep discovery pass and orders each
+query's eligible repository observations by search rank and stable queue ID.
+It walks pinned queries round-robin, taking the next candidate whose physical
+queue ID has not already been selected, until each query owns at most 39
+repositories or the global count reaches exactly 2,000. Duplicate observations
+are skipped within the selecting query rather than consuming quota.
+
+Given the reviewed frozen pool this produces 38 repositories for each of 52
+nonempty keywords and a 39th for the first 24 of those keywords in pinned order;
+the nine empty keywords remain explicit zero rows. Manifest generation fails
+closed unless it reaches exactly 2,000 unique physical repositories with this
+distribution. All fresh query observations remain in relational provenance and
+report attribution joins the selected physical repository back to that frozen
+evidence rather than duplicating scan work.
+
+### Exact prior-release eligibility
+
+A repository with no policy-event history remains eligible under the existing
+rules. A repository with history is eligible only when every event belongs to
+a completed/released Docker experiment and forms an exact cold/reactivate pair:
+the reverse event names the cold event, restores its recorded prior state, and
+matches experiment, config, policy, manifest, and audit evidence. Unreversed,
+unrelated, malformed, or mixed history is ineligible and causes no automatic
+repair. The old depth cohort is excluded independently through its persisted
+experiment membership.
+
+This exception permits the newly reviewed experiment to hold and later release
+rows that were safely released by the prior experiment without weakening the
+append-only policy audit chain.
+
+### Reviewed authority handoff
+
+The current depth experiment must first become `completed`. Runtime is stopped
+canonically, the terminal aggregate report and release manifest are generated,
+and exact-SHA reviewed release restores only that experiment's owned cold rows.
+The breadth cohort and hold manifests are then generated and applied while
+sources remain stopped. Activation rejects any other unreleased, nonterminal,
+or fenced Docker experiment authority.
+
+The breadth experiment uses the existing state machine, experiment-row-first
+locking, resolver tokens, target reservations, finite retries, capacity
+accounting, and reversible holds. Since both shallow and deep image limits are
+one, every image-bearing membership produces only selection rank one and a
+single dispatch wave.
+
+### Secret-safe decision report
+
+The terminal report treats target-scoped finding fingerprints as location
+evidence, not secret novelty. Primary outcomes are globally deduplicated
+credentials, currently alive credentials, and each per scanner-hour. It also
+reports physical repository/image coverage, scan states, keycheck completeness,
+and per-keyword attribution from frozen provenance. No raw identity or target
+material is emitted.
+
+## Risks / Trade-offs
+
+- [A selected repository has no eligible image] -> Use the existing deterministic
+  fresh replacement path; fail the reviewed cohort if exact 2,000 repository
+  ownership cannot be preserved before activation.
+- [Cross-keyword overlap biases ownership] -> Use pinned query-order round-robin,
+  preserve all frozen query provenance, and distinguish physical totals from
+  keyword attribution credits.
+- [Prior policy history is malformed] -> Leave the row ineligible and fail
+  closed rather than guessing or rewriting history.
+- [Two experiment authorities overlap] -> Reject planning/activation until the
+  prior experiment is completed, reviewed, released, and fence-free.
+- [The cohort is large] -> Keep the hard 2,000-target ceiling, existing shared
+  capacity limits, two Docker workers, and finite retry/hold behavior.
+- [Migration or rollout fails] -> Keep schema changes additive where possible,
+  stop before authority mutation, and retain manifests and audit rows for a
+  deterministic retry.
@@ -0,0 +1,50 @@
+## Why
+
+The completed portion of the Docker depth pilot found every currently alive
+credential in the newest selected image and no additional alive credential in
+older image ranks. A larger but still bounded rank-1 cohort is needed to test
+whether spending the same capacity on repository breadth produces better
+credential and currently-alive yield than blanket image depth.
+
+## What Changes
+
+- Add a separate rank-1-only Docker breadth experiment over exactly 2,000
+  previously unscanned physical repository anchors from the existing complete
+  frozen discovery pass, excluding the current depth-pilot cohort.
+- Balance the cohort without using prior yield: each of the 52 keywords with an
+  eligible remaining pool receives 38 repositories and 24 deterministically
+  selected keywords receive one additional repository; the 9 exhausted
+  keywords retain explicit zero coverage.
+- Deduplicate physical repositories across keywords while preserving all fresh
+  keyword provenance, and scan at most one newest eligible immutable image per
+  selected repository.
+- Keep the new experiment fail-closed until the current depth experiment is
+  terminal and its cold rows have passed reviewed release; never run two Docker
+  experiment authorities concurrently.
+- Persist a reviewable immutable cohort plan before activation and retain the
+  existing lease, reservation, fencing, finite-retry, capacity, and reversible
+  hold guarantees.
+- Report globally deduplicated credentials, currently alive credentials,
+  repository/image coverage, and both yield measures per scanner-hour, with
+  per-keyword attribution and no secret or target material.
+
+## Capabilities
+
+### New Capabilities
+
+- `docker-rank1-breadth-experiment`: A balanced, deterministic, physically
+  deduplicated 2,000-repository rank-1 experiment with reviewed authority
+  handoff, bounded execution, and secret-safe yield reporting.
+
+### Modified Capabilities
+
+## Impact
+
+- Docker experiment validation, cohort planning, authority handoff, resolver
+  admission, rank-1 target scheduling, and aggregate reporting.
+- Managed PostgreSQL experiment state and audit evidence, with migrations only
+  where the existing depth-experiment schema cannot represent the new plan.
+- Docker experiment configuration and focused unit/PostgreSQL integration
+  coverage.
+- Runtime operations require canonical stop, reviewed release of the completed
+  depth experiment, reviewed activation of this change, and canonical restart.
@@ -0,0 +1,97 @@
+## ADDED Requirements
+
+### Requirement: Reviewed rank-one breadth authority
+The system SHALL support a code-pinned rank-one breadth profile of 61 ordered
+queries, at most 39 owned repositories per query, one image per repository, and
+exactly 2,000 unique physical repository selections and target slots. It SHALL
+reject arbitrary limit combinations and SHALL preserve the existing depth
+profile unchanged.
+
+#### Scenario: Breadth profile is valid
+- **WHEN** configuration supplies the exact reviewed breadth selector and limits
+- **THEN** validation SHALL produce a distinct immutable authority hash with a physical target ceiling of 2,000
+
+#### Scenario: Profile limits are mixed
+- **WHEN** configuration combines a selector or limit from different reviewed profiles
+- **THEN** startup SHALL fail before secrets, database mutation, network work, or workers initialize
+
+#### Scenario: Prior authority is not released
+- **WHEN** another Docker experiment is nonterminal, unreleased, or fenced
+- **THEN** breadth cohort application and activation SHALL fail without changing either experiment
+
+### Requirement: Exact balanced physical cohort
+The system SHALL deterministically select exactly 2,000 previously unscanned
+physical repository anchors from the existing complete frozen discovery pass,
+excluding every repository in the depth cohort. Selection SHALL be independent
+of prior finding or credential yield.
+
+#### Scenario: Reviewed frozen pool is selected
+- **WHEN** 52 keywords have sufficient eligible repositories and nine have none
+- **THEN** each nonempty keyword SHALL own 38 unique repositories, the first 24 still-eligible keywords in pinned order SHALL own one additional repository, and empty keywords SHALL remain explicit zero rows
+
+#### Scenario: Repository appears under several keywords
+- **WHEN** the next candidate is already physically owned by an earlier round-robin selection
+- **THEN** it SHALL consume neither another physical slot nor the selecting keyword's quota, and selection SHALL continue to that keyword's next eligible candidate
+
+#### Scenario: Exact cohort cannot be formed
+- **WHEN** deterministic eligible selection cannot reach exactly 2,000 unique repositories with the reviewed distribution
+- **THEN** manifest generation or application SHALL fail closed and no partial experiment cohort SHALL activate
+
+### Requirement: Released policy history eligibility
+The system SHALL admit a previously held repository only when its complete
+policy-event history consists exclusively of authority-valid cold/reactivate
+pairs owned by completed and released Docker experiments.
+
+#### Scenario: Prior hold was exactly released
+- **WHEN** every prior cold event has one matching reverse event that restores its recorded state and matches experiment, config, policy, manifest, and audit evidence
+- **THEN** the unfenced unscanned repository MAY participate in the new reviewed cohort or hold manifest
+
+#### Scenario: Prior history is incomplete or unrelated
+- **WHEN** any policy event is unreversed, malformed, belongs to an unreleased experiment, or represents an unrelated policy
+- **THEN** the repository SHALL remain ineligible and its queue and event history SHALL remain unchanged
+
+### Requirement: Rank-one-only execution
+The system SHALL resolve at most the newest eligible immutable image for each
+selected repository and SHALL create no image selection above rank one.
+
+#### Scenario: Repository has an eligible newest image
+- **WHEN** its dedicated resolver completes under a valid owner, generation, token, and experiment authority
+- **THEN** exactly one rank-one selection MAY consume one physical experiment target slot
+
+#### Scenario: Candidate image is unsafe
+- **WHEN** the newest candidate is previously scanned, failed, quarantined, independently cold, fenced, or otherwise ineligible
+- **THEN** existing deterministic safe replacement rules SHALL apply without reactivating historical work or selecting an older image rank for depth
+
+#### Scenario: Breadth work is dispatched
+- **WHEN** rank-one targets become available
+- **THEN** they SHALL use one round-robin dispatch wave with existing reservation, capacity, retry, and terminal-binding guarantees
+
+### Requirement: Reviewed handoff and reversible holds
+The system SHALL activate the breadth experiment only through a stopped-runtime
+reviewed handoff after the depth experiment completes and releases its owned
+cold rows. Breadth-owned non-cohort holds SHALL remain exactly reversible.
+
+#### Scenario: Canonical handoff succeeds
+- **WHEN** runtime is stopped, the depth experiment is completed and fence-free, its exact release SHA is approved, and the breadth cohort and hold SHAs are approved
+- **THEN** release and breadth activation SHALL commit through their existing experiment-row-first fenced protocols before runtime restarts
+
+#### Scenario: Breadth release is reviewed
+- **WHEN** the breadth experiment later completes and its exact reactivation manifest is approved
+- **THEN** only unreversed breadth-owned cold events SHALL restore their recorded prior states
+
+### Requirement: Secret-safe breadth reporting
+The system SHALL report physical coverage, globally deduplicated credential and
+currently-alive yield, per-keyword attribution, scan cost, and both yield
+measures per scanner-hour without exposing secret or target material.
+
+#### Scenario: Credential repeats across keywords
+- **WHEN** one credential is found in a repository attributed to multiple frozen keyword observations
+- **THEN** global totals SHALL count it once while each eligible keyword attribution MAY receive an explicit non-additive credit
+
+#### Scenario: Report measures novelty
+- **WHEN** findings repeat across target-scoped locations
+- **THEN** the decision report SHALL distinguish finding locations from detector-secret identities and credential identities and SHALL use credentials and currently-alive credentials as primary outcomes
+
+#### Scenario: Report reads sensitive evidence
+- **WHEN** aggregate reporting accesses findings or keycheck rows
+- **THEN** output SHALL omit raw credentials, repositories, image targets, URLs, hashes, excerpts, DSNs, and configuration identities
@@ -0,0 +1,35 @@
+## 1. Reviewed Profile And Schema
+
+- [ ] 1.1 Add exact versioned depth and rank-one breadth profile validation while preserving the existing depth profile hashes and behavior.
+- [ ] 1.2 Widen PostgreSQL/SQLite experiment bounds only to the reviewed 39-repository and 2,000-target profile.
+- [ ] 1.3 Make persisted resolver authority dynamic from the exact reviewed profile instead of hardcoded depth constants.
+
+## 2. Deterministic Cohort
+
+- [ ] 2.1 Select fresh candidates from the existing complete frozen pass while excluding the prior depth cohort and unsafe queue state.
+- [ ] 2.2 Validate exact fully reversed prior Docker-experiment policy history without accepting unrelated or incomplete events.
+- [ ] 2.3 Implement physically deduplicated pinned-query round-robin planning to exactly 2,000 repositories with 38/39/0 keyword ownership.
+- [ ] 2.4 Preserve the old depth plan and manifest bytes and add strict breadth plan/manifest validation.
+- [ ] 2.5 Revalidate every reviewed cohort row and its provenance/history under locks before application.
+
+## 3. Runtime Authority
+
+- [ ] 3.1 Enforce that no other Docker experiment is nonterminal, unreleased, or fenced at breadth application and activation.
+- [ ] 3.2 Reuse reversible holds for safely released rows and retain exact append-only policy-event authority.
+- [ ] 3.3 Resolve at most one newest immutable image per repository and prohibit breadth selections above rank one.
+- [ ] 3.4 Keep experiment-row-first leases, reservations, capacity, finite retries, and terminal target reconciliation unchanged.
+
+## 4. Reporting And Verification
+
+- [ ] 4.1 Add secret-safe breadth reporting for globally deduplicated credentials, alive credentials, keyword attribution, coverage, scan-hours, and yield per scanner-hour.
+- [ ] 4.2 Add focused unit tests for profile validation, exact balanced physical selection, old-profile compatibility, and prior-release history.
+- [ ] 4.3 Add focused PostgreSQL integration coverage for authority handoff, locked cohort application, rank-one dispatch, and reversible holds.
+- [ ] 4.4 Run focused tests and strict OpenSpec validation.
+
+## 5. Reviewed Launch
+
+- [ ] 5.1 Canonically stop runtime and verify sources, workers, leases, reservations, and experiment fences are quiescent.
+- [ ] 5.2 Complete and report the two remaining depth targets or apply only an existing reviewed terminal protocol.
+- [ ] 5.3 Generate and approve the depth release manifest, then verify every owned cold event is exactly reversed.
+- [ ] 5.4 Generate, review, and apply the exact 2,000-repository breadth cohort and hold manifests.
+- [ ] 5.5 Update the reviewed runtime profile, canonically restart, and verify secret-safe live progress with no hold, exhausted retry, or fence anomaly.