Initial server source import
This commit is contained in:
sashatrask
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-09-06
|
||||
@@ -0,0 +1,80 @@
|
||||
## Context
|
||||
|
||||
The completed keyword-pruning change removed 38 globally zero-yield terms from discovery configuration, but `target_queue` admission still considers previously queued `pending` and due `deferred` rows without consulting current query policy. Since deployment, Docker work attributed to retired terms consumed about 20.6 scanner-hours and produced no strict-usable credential. GitHub Actions independently consumed one core worker while both fresh and retained work produced no strict-usable credential and its active backlog grew.
|
||||
|
||||
Queue rows are durable authority referenced by scan history, result reservations, Git and Docker coverage, deduplication, and retry state. Physical deletion or overloading quarantine would destroy or misrepresent that authority. Runtime configuration and PostgreSQL schema are lifecycle-protected and require coordinated offline changes.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Make policy-retired pending and deferred work explicitly unclaimable while preserving every durable record and field needed for audit or reversal.
|
||||
- Apply and reverse policy holds only through exact, reviewed, idempotent manifests under stopped-source lifecycle authority.
|
||||
- Prevent stale Docker resolver anchors from generating new digest children after the initial cold transition.
|
||||
- Pause GitHub Actions without deleting or rewriting its backlog.
|
||||
- Expose cold rows separately from active backlog in operational counts.
|
||||
|
||||
**Non-Goals:**
|
||||
- Deleting target queue rows or changing scan, finding, credential, result, deduplication, or coverage history.
|
||||
- Age-based retirement of mutable GitLab or HuggingFace targets.
|
||||
- Automatically colding every disabled source or all dormant historical backlogs in the first deployment.
|
||||
- Changing scanner concurrency, Docker scan policy, discovery keywords, or keycheck behavior.
|
||||
- Treating query retirement as evidence that a target can never become useful.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Add a first-class `cold` queue status
|
||||
|
||||
`target_queue.status='cold'` is the policy hold. Existing scan, legacy, and Docker resolver claim paths explicitly allow only `pending` and due `deferred`, so cold rows remain fail-closed even if a caller does not understand policy metadata. Cold is not a worker result disposition and ordinary scans cannot produce it.
|
||||
|
||||
Alternative: add a nullable metadata flag. Rejected because every current and older admission path would need to remember an additional predicate, making accidental claims likely. `deferred` is also unsuitable because it is a timed retry, and `quarantined` is reserved for pipeline-integrity failures with capacity accounting and review semantics.
|
||||
|
||||
### Use exact reviewed manifests and append-only audit events
|
||||
|
||||
Add an append-only `target_queue_policy_events` table recording each cold/reactivate transition, prior and next status, exact source/platform/query attribution, configuration and policy hashes, manifest hash, prior update fence, reason code, and reversal linkage. A private dry-run manifest lists only queue IDs and non-sensitive policy attribution; it never contains target values.
|
||||
|
||||
Apply locks rows in deterministic ID order and atomically validates the manifest fence, inserts one audit event, and changes only `status` and `updated_at`. Eligible cold transitions are limited to unfenced `pending` or `deferred` rows with no active queue/result/resolver lease or reservation. Reactivation restores the exact audited prior status and is explicit rather than an enqueue side effect.
|
||||
|
||||
Alternative: issue a one-off SQL update and infer reversal from `available_after`. Rejected because it is not reviewable, cannot prove the selected cohort, and loses the exact prior lifecycle state.
|
||||
|
||||
### Derive stale policy from canonical exact queries
|
||||
|
||||
Policy uses case-sensitive exact `(source, platform, query)` triples from canonical source configuration. Null/blank queries, unknown source/platform pairs, non-rotation provenance, and sole operational sentinel queries are not inferred as stale. Disabled sources retain their configured query policy; source disablement is not itself a retirement action.
|
||||
|
||||
The initial reviewed transition is scoped to DockerHub platform `docker`, covering every eligible pending/deferred resolver anchor and digest row whose own exact query is no longer configured. Dormant source families remain preserved and unclaimable by virtue of having no worker; they can be reviewed separately later rather than mutating a six-figure backlog in this deployment.
|
||||
|
||||
### Filter periodic Docker resolver claims by current query policy
|
||||
|
||||
Docker digest children inherit the resolver anchor query, and completed anchors can be periodically reclaimed. The runtime therefore passes the exact configured DockerHub query allowlist to resolver admission and excludes non-null anchors whose query is not allowed. This prevents completed stale anchors, which are outside the pending/deferred cold migration, from recreating policy-retired work.
|
||||
|
||||
The filter is exact and does not rewrite query attribution. Null legacy provenance remains excluded from automatic policy retirement and requires separate review.
|
||||
|
||||
### Preserve cold state until explicit reactivation
|
||||
|
||||
Ordinary queue synchronization, enqueue/upsert, rediscovery, retry, and completion logic must not turn a cold row back into pending/deferred. If a target becomes relevant through a retained query, an operator can review its audit lineage and explicitly reactivate it; implicit reactivation would make the policy hold non-deterministic and unaudited.
|
||||
|
||||
### Pause GitHub Actions at all configuration layers
|
||||
|
||||
Remove `github_actions` from `supervisor.enabled_sources` and set both supervisor-source and source-level enablement false. Keep its query and all queue/history rows unchanged. No cold transition is required for GHA because no worker remains able to claim its source/platform rows.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Earliest query attribution can cold a target rediscovered by a retained term] -> Preserve full audit lineage and require explicit reactivation; do not delete the row or overwrite its original query.
|
||||
- [A live or fenced row could be transitioned] -> Require coordinated source shutdown, exact row-update fences, reservation/lease checks, deterministic locks, and atomic all-or-nothing apply.
|
||||
- [Completed Docker anchors could bypass the migration] -> Filter retry and periodic resolver admission by the current exact query allowlist.
|
||||
- [Cold rows could inflate active backlog displays] -> Count `cold` separately and exclude it from pending/deferred operational backlog totals.
|
||||
- [A policy/config change between review and apply could invalidate the cohort] -> Fence the manifest with canonical configuration, query-policy, selection, and manifest hashes.
|
||||
- [Pausing GHA may miss future useful credentials] -> Preserve its entire queue and configuration for a coordinated future re-enable after explicit review.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add schema support, policy transition APIs, exact manifest planning/apply commands, Docker resolver query filtering, cold-preserving upsert behavior, and focused tests.
|
||||
2. Stop the authenticated runtime coordinately and verify sources are stopped and no active target/result/resolver fences block the selected Docker cohort.
|
||||
3. Start maintenance PostgreSQL, apply the additive schema migration, and generate a bounded private DockerHub stale-query cold manifest.
|
||||
4. Review aggregate counts and hashes, then apply the exact manifest atomically. Retain only the append-only database audit; remove the temporary private manifest after verification.
|
||||
5. Deploy the GHA-disabled configuration and restart through the canonical runtime lifecycle.
|
||||
6. Verify no cold Docker row is claimable, no stale completed anchor is resolver-claimable, cold and active counts reconcile, GHA has no child process, and the remaining pipeline is healthy.
|
||||
7. Roll back by stopping sources, generating an exact reactivation manifest from unreversed cold events, applying it atomically, restoring GHA configuration if desired, and restarting canonically.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None.
|
||||
@@ -0,0 +1,23 @@
|
||||
## Why
|
||||
|
||||
Removing zero-yield discovery terms stopped new discovery but left their pending and deferred targets claimable, so retired policy continued consuming scanner time. GitHub Actions also produced no strict-usable credential from either fresh work or its large retained backlog while that backlog kept growing, so it should no longer occupy a core worker.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Add an explicit, auditable, reversible `cold` lifecycle state for policy-retired target queue rows without deleting queue history, scans, deduplication, reservations, or coverage records.
|
||||
- Cold only unfenced `pending` and `deferred` rows whose exact source query is absent from the canonical configured query policy, using a reviewed offline manifest and coordinated lifecycle authority.
|
||||
- Prevent completed Docker resolver anchors attributed to retired queries from being periodically reclaimed and creating new stale digest children.
|
||||
- Preserve cold rows across ordinary enqueue, rediscovery, retry, and completion paths; require an explicit audited action to reactivate them.
|
||||
- Pause GitHub Actions by removing it from the active core and disabling both supervisor and source configuration, while preserving its complete queue and history.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `target-queue-policy-holds`: Auditable cold and reactivation transitions for policy-stale target queue work, including claim exclusion and Docker resolver filtering.
|
||||
|
||||
### Modified Capabilities
|
||||
- `discovery-keyword-pruning`: Retired queries stop both future discovery and claimable pending/deferred work while preserving every historical authority record.
|
||||
|
||||
## Impact
|
||||
|
||||
The change affects PostgreSQL target queue schema and migration authority in `app/scanner_db.py` and `app/migrate_runtime_safety.py`, Docker resolver admission and query plumbing in `app/console_runner.py`, core source selection in `app/config.yaml`, focused lifecycle/configuration tests, and dashboard/status aggregation where queue states are enumerated. Deployment requires a coordinated runtime stop, schema migration, reviewed cold manifest application, and canonical restart. No target, scan, finding, credential, result, deduplication, or coverage row is deleted.
|
||||
@@ -0,0 +1,20 @@
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Operational and historical authority is preserved
|
||||
Keyword retirement SHALL stop future discovery and SHALL permit existing unfenced pending/deferred targets attributed to retired exact queries to enter an audited, reversible cold state without deleting or rewriting historical authority.
|
||||
|
||||
#### Scenario: Dedicated source sentinels remain
|
||||
- **WHEN** archive and gist source rotations are loaded
|
||||
- **THEN** `gharchive`, `gharchive-files`, and `gists` SHALL remain as their sole configured query tokens
|
||||
|
||||
#### Scenario: Persisted rotation index remains valid
|
||||
- **WHEN** an existing query index exceeds a shortened query list
|
||||
- **THEN** normal modulo-based rotation SHALL select a valid configured query without a state-file edit
|
||||
|
||||
#### Scenario: Existing backlog is preserved but held
|
||||
- **WHEN** a previously admitted unfenced target is attributed to a retired exact query and selected by reviewed policy
|
||||
- **THEN** its queue row SHALL remain present with all attribution, retry, deduplication, scan, reservation, and coverage history preserved while its status becomes unclaimable `cold`
|
||||
|
||||
#### Scenario: Historical records remain unchanged
|
||||
- **WHEN** a stale-query cold transition is applied
|
||||
- **THEN** existing target scans, findings, credentials, keycheck results, completed queue rows, and coverage records SHALL NOT be deleted or rewritten
|
||||
@@ -0,0 +1,78 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Cold queue rows are not claimable
|
||||
The system SHALL represent policy-held target work with `target_queue.status='cold'`, and no scanner, legacy queue consumer, or Docker resolver SHALL claim a cold row.
|
||||
|
||||
#### Scenario: Scanner checks active backlog
|
||||
- **WHEN** a queue row has status `cold`
|
||||
- **THEN** the row SHALL be excluded from pending, due-deferred, retry, and periodic resolver admission
|
||||
|
||||
#### Scenario: Worker completes ordinary work
|
||||
- **WHEN** a scan result is ingested
|
||||
- **THEN** its queue disposition SHALL remain limited to ordinary lifecycle outcomes and SHALL NOT create a cold status
|
||||
|
||||
### Requirement: Policy holds are exact and audited
|
||||
The system SHALL transition queue rows into or out of cold state only through a reviewed, hash-fenced, append-only policy event under stopped-source lifecycle authority.
|
||||
|
||||
#### Scenario: Eligible stale row is held
|
||||
- **WHEN** an exact manifest entry still matches an unfenced `pending` or `deferred` row whose source query is absent from canonical policy
|
||||
- **THEN** the system SHALL atomically record the audit event and change only the row status and update timestamp to `cold`
|
||||
|
||||
#### Scenario: Selected row has an active fence
|
||||
- **WHEN** a selected row has a queue lease, resolver lease, current claim, active result reservation, or submitted content lease
|
||||
- **THEN** the policy apply SHALL fail closed without partially applying the manifest
|
||||
|
||||
#### Scenario: Exact apply is repeated
|
||||
- **WHEN** the same reviewed manifest is applied again after a successful transition
|
||||
- **THEN** the system SHALL report an idempotent duplicate without creating a second state transition
|
||||
|
||||
### Requirement: Cold state is explicitly reversible
|
||||
The system SHALL retain the exact prior queue status in policy audit history and SHALL require a reviewed reactivation manifest to restore a cold row.
|
||||
|
||||
#### Scenario: Cold row is rediscovered normally
|
||||
- **WHEN** enqueue, synchronization, rediscovery, retry, or completion logic encounters an existing cold row
|
||||
- **THEN** the row SHALL remain cold and its historical attribution SHALL remain unchanged
|
||||
|
||||
#### Scenario: Reviewed cold event is reactivated
|
||||
- **WHEN** an exact reactivation manifest references an unreversed cold event and all row fences still match
|
||||
- **THEN** the system SHALL restore the audited prior `pending` or `deferred` status and append a linked reactivation event
|
||||
|
||||
### Requirement: Stale-query selection follows canonical policy
|
||||
The system SHALL evaluate query staleness using case-sensitive exact source, platform, and query policy derived from canonical configuration.
|
||||
|
||||
#### Scenario: Configured query remains active
|
||||
- **WHEN** a queue row's exact source/platform/query triple remains configured
|
||||
- **THEN** automatic stale-policy planning SHALL NOT select the row
|
||||
|
||||
#### Scenario: Attribution cannot be classified safely
|
||||
- **WHEN** query attribution is null, blank, operational, non-rotation, or belongs to an unknown source/platform pair
|
||||
- **THEN** automatic planning SHALL skip and report the row rather than inferring retirement
|
||||
|
||||
#### Scenario: Initial Docker stale cohort is planned
|
||||
- **WHEN** DockerHub policy planning is scoped to platform `docker`
|
||||
- **THEN** it SHALL include eligible pending/deferred resolver anchors and digest rows attributed to removed exact queries and SHALL expose only aggregate counts plus non-sensitive manifest fields
|
||||
|
||||
### Requirement: Docker resolvers honor current query policy
|
||||
The system SHALL prevent completed or retryable Docker resolver anchors attributed to retired non-null queries from generating new digest queue rows.
|
||||
|
||||
#### Scenario: Periodic stale anchor becomes due
|
||||
- **WHEN** a completed Docker resolver anchor is due but its exact query is not in the configured DockerHub allowlist
|
||||
- **THEN** resolver admission SHALL leave the anchor unclaimed
|
||||
|
||||
#### Scenario: Configured anchor becomes due
|
||||
- **WHEN** a Docker resolver anchor's exact query remains configured and all ordinary claim fences pass
|
||||
- **THEN** resolver admission SHALL preserve the existing retry and periodic behavior
|
||||
|
||||
### Requirement: Source pause preserves backlog authority
|
||||
Pausing a source SHALL remove its worker from the active core without deleting or rewriting that source's queue or historical records.
|
||||
|
||||
#### Scenario: GitHub Actions is paused
|
||||
- **WHEN** canonical runtime configuration is loaded after this change
|
||||
- **THEN** GitHub Actions SHALL be absent from the supervisor core and disabled at both supervisor-source and source configuration layers while its configured query and persisted backlog remain intact
|
||||
|
||||
### Requirement: Cold work is separately observable
|
||||
Operational queue summaries SHALL report cold rows separately and SHALL NOT include them in active pending or deferred backlog counts.
|
||||
|
||||
#### Scenario: Queue state is summarized
|
||||
- **WHEN** an operator inspects canonical target queue status
|
||||
- **THEN** the summary SHALL expose a distinct cold count without representing those rows as retryable or claimable work
|
||||
@@ -0,0 +1,21 @@
|
||||
## 1. Durable Policy State
|
||||
|
||||
- [x] 1.1 Add the `cold` target queue lifecycle state, append-only policy-event schema, indexes, additive migration marker, and schema validation coverage.
|
||||
- [x] 1.2 Implement atomic, fenced, idempotent cold and reactivation database transitions that preserve all non-lifecycle queue authority.
|
||||
|
||||
## 2. Reviewed Policy Operations
|
||||
|
||||
- [x] 2.1 Implement exact canonical query-policy derivation and privacy-safe stale-row/reversal manifest planning.
|
||||
- [x] 2.2 Add stopped-source migration CLI dry-run/apply paths with manifest/config/policy/selection hash validation and bounded deterministic scope.
|
||||
|
||||
## 3. Runtime Enforcement
|
||||
|
||||
- [x] 3.1 Preserve cold rows across ordinary queue enqueue/synchronization and expose cold separately in canonical queue summaries.
|
||||
- [x] 3.2 Pass configured DockerHub query policy into retry/periodic resolver admission and exclude retired-query anchors.
|
||||
- [x] 3.3 Remove GitHub Actions from the active core and disable both supervisor and source layers without altering its query or persisted backlog.
|
||||
|
||||
## 4. Verification And Deployment
|
||||
|
||||
- [x] 4.1 Add focused unit and PostgreSQL integration tests for claim exclusion, exact selection, fenced transitions, idempotency, reversal, Docker resolver filtering, cold preservation, observability, and GHA pause.
|
||||
- [x] 4.2 Run targeted test suites and strict OpenSpec validation with no forbidden application bytecode artifacts.
|
||||
- [x] 4.3 Coordinately stop runtime, apply the additive schema migration and reviewed Docker stale-query cold manifest, remove temporary artifacts, restart canonically, and verify active backlog and pipeline health.
|
||||
Reference in New Issue
Block a user