Initial server source import
This commit is contained in:
sashatrask
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-08-25
|
||||
@@ -0,0 +1,87 @@
|
||||
## Context
|
||||
|
||||
GitHub, GitLab, and HuggingFace discovery return stable target identities together with remote update timestamps. The runner currently discards those timestamps and the PostgreSQL queue permanently deduplicates targets by `(source, normalized_target)`. A completed repository or Space is therefore never scanned again even when its content changes.
|
||||
|
||||
Production discovery is already saturated: less than one percent of fetched records are new target identities and the core queue is often empty. The change must restore changed-content coverage without turning every rediscovery into a rescan, growing one queue row per revision, or creating an unbounded backlog.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Preserve a bounded remote content-update signal for GitHub, GitLab, and HuggingFace targets.
|
||||
- Rescan a completed target only when discovery observes content newer than the revision covered by its latest claim.
|
||||
- Coalesce multiple remote updates into one mutable queue row and one pending follow-up.
|
||||
- Bound changed-target admission and expose it separately from new-target admission.
|
||||
- Roll out without treating every legacy row as changed.
|
||||
|
||||
**Non-Goals:**
|
||||
- Requeue terminal failures or actively leased/deferred targets.
|
||||
- Rescan every known target on a timer.
|
||||
- Change DockerHub's digest-based identity and refresh behavior.
|
||||
- Guarantee that provider activity timestamps always represent content changes.
|
||||
- Enable inactive historical sources or enlarge global scan concurrency.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Carry one normalized discovery record
|
||||
|
||||
The runner will represent eligible discovery results internally as a target plus an optional UTC `remote_modified_at`. GitHub uses `pushed_at`, GitLab uses `last_activity_at`, and HuggingFace uses `lastModified`. Missing, malformed, or non-monotonic timestamps remain valid discovery results but cannot trigger an updated-target rescan.
|
||||
|
||||
HuggingFace discovery will use a newest-modified feed so old Spaces changed recently are observable. Identity-only known-page stopping will be disabled when updated-target rescans are enabled; hard page and result limits remain the discovery bound.
|
||||
|
||||
Alternatives rejected:
|
||||
- Repository `updated_at` on GitHub, because metadata-only edits are not content pushes.
|
||||
- A HEAD-SHA request per repository, because it multiplies API traffic and rate-limit exposure.
|
||||
- Revision-aware early stopping, because a known first page does not prove later pages contain no changed targets.
|
||||
|
||||
### Keep one queue row and two remote timestamps
|
||||
|
||||
`target_queue` will gain nullable `remote_modified_at` and `scan_remote_modified_at` columns. Discovery monotonically advances `remote_modified_at`. Claiming atomically copies the currently observed value into `scan_remote_modified_at`, recording what that scan covers.
|
||||
|
||||
The separate claim snapshot is required because a remote update can arrive while a scan is running. On completion, a newer observed timestamp remains ahead of the claimed timestamp and is eligible for exactly one later scan. Encoding revisions into `normalized_target` was rejected because it would grow queue rows and weaken queue authority.
|
||||
|
||||
For a legacy completed row with no claim snapshot, `completed_at` is the rollout baseline. It is eligible only when the first valid remote timestamp observed is newer than that completion. This prevents a migration surge while still admitting updates that occurred after the historical scan.
|
||||
|
||||
### Observe and requeue atomically under a hard budget
|
||||
|
||||
A PostgreSQL transaction will upsert discovery observations and requeue at most `updated_rescan_max_per_cycle` eligible rows for one source. Eligibility requires:
|
||||
- `status='done'`;
|
||||
- a strictly newer observed timestamp than `scan_remote_modified_at`, or than `completed_at` for a legacy row;
|
||||
- elapsed `updated_rescan_cooldown_seconds` since completion;
|
||||
- no lease, reservation, claim, or resolver authority.
|
||||
|
||||
Eligible rows are locked with `FOR UPDATE SKIP LOCKED`. Requeue resets only retry/completion scheduling fields required for a normal pending claim. Failed, pending, deferred, in-progress, unresolved, unchanged, and invalid-timestamp rows are never promoted by this path.
|
||||
|
||||
The initial production setting is one updated target per source cycle. Existing backlog-first behavior remains enabled, so a source drains its admitted work before discovery can admit more; `refresh_registry` is not enabled by this change.
|
||||
|
||||
Alternatives rejected:
|
||||
- Global `requeue_done=True`, because it requeues unchanged rows on every cycle.
|
||||
- Comparing only remote time with local completion time forever, because provider and host clocks differ and a mid-scan update can be lost.
|
||||
- A separate maintenance queue, because it duplicates existing lease, reservation, and completion authority.
|
||||
|
||||
### Account for updated targets separately
|
||||
|
||||
`source_cycles` will gain `queued_updated_count`. `queued_new_count` keeps its current meaning. Source logs and dashboard aggregation will report changed-target admissions separately so rollout volume and yield can be audited.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [GitLab activity can change without a repository push] -> Use a one-target-per-cycle cap and cooldown; report updated admissions separately.
|
||||
- [Provider clock skew] -> Require strict monotonicity and use the claimed remote timestamp after the first revision-aware scan.
|
||||
- [More discovery API traffic after disabling identity-only early stop] -> Keep existing hard page/per-page limits and source intervals.
|
||||
- [Changed targets consume capacity without useful findings] -> Start at one per cycle and compare updated-target yield before increasing the cap.
|
||||
- [Schema rollout while runtime is active] -> Stop the authority-managed runtime, apply schema through the normal initialization path, run tests, then restart through `start_runtime.ps1`.
|
||||
- [Rollback leaves nullable columns] -> Disable the feature in source configuration; nullable columns and metrics are backward-compatible and can remain.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add nullable queue columns, the source-cycle counter, and a partial eligibility index through idempotent schema initialization.
|
||||
2. Deploy code and tests with updated-target rescans disabled by default.
|
||||
3. Enable GitHub, GitLab, and HuggingFace with a cap of one and a conservative cooldown; leave DockerHub unchanged.
|
||||
4. Restart the authority-managed runtime and verify discovery, queue authority, projection, and source health.
|
||||
5. Observe changed-target admission, completion, findings, and worker occupancy before changing any cap.
|
||||
|
||||
Rollback is configuration-first: disable updated-target rescans and restart the managed runtime. Existing pending work completes under normal queue semantics; no destructive data migration is required.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Whether production evidence supports different cooldowns per source after the initial canary.
|
||||
- Whether a later change should add provider-specific immutable revisions when APIs can supply them without extra requests.
|
||||
@@ -0,0 +1,28 @@
|
||||
## Why
|
||||
|
||||
GitHub, GitLab, and HuggingFace discovery continuously observe recently changed repositories and Spaces, but the queue permanently suppresses every URL or Space ID after its first scan. Production consequently fetched roughly 290,000 discovery results in the latest 24 hours while admitting less than one percent as targets, leaving scan capacity underused and ignoring secrets added to already-known projects.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Preserve the remote content-update timestamp supplied by GitHub, GitLab, and HuggingFace discovery.
|
||||
- Requeue a completed target only when the observed remote content timestamp is newer than its last completed scan.
|
||||
- Bound changed-target admission per source cycle and enforce a per-target cooldown.
|
||||
- Keep active, deferred, failed, and unchanged targets untouched.
|
||||
- Expose changed-target requeue counts separately from newly discovered target counts.
|
||||
- Leave DockerHub digest-based refresh behavior unchanged.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `updated-target-rescan`: Safely detect and rescan remotely changed core repository and Space targets under bounded rollout controls.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
None.
|
||||
|
||||
## Impact
|
||||
|
||||
- Discovery metadata and target preparation in `app/scanner.py` and `app/console_runner.py`.
|
||||
- PostgreSQL target queue state, migrations, cycle accounting, and dashboard observability in `app/scanner_db.py` and `app/dashboard.py`.
|
||||
- Source configuration for GitHub, GitLab, and HuggingFace.
|
||||
- Focused queue, discovery, and PostgreSQL integration tests.
|
||||
@@ -0,0 +1,87 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Discovery preserves remote content recency
|
||||
The system SHALL preserve a normalized remote content-update timestamp for GitHub, GitLab, and HuggingFace discovery records and SHALL keep DockerHub digest-based identity behavior unchanged.
|
||||
|
||||
#### Scenario: Source-specific remote timestamp is retained
|
||||
- **WHEN** GitHub supplies `pushed_at`, GitLab supplies `last_activity_at`, or HuggingFace supplies `lastModified`
|
||||
- **THEN** the queue observation stores the valid UTC timestamp with the target identity
|
||||
|
||||
#### Scenario: Missing or malformed remote timestamp
|
||||
- **WHEN** a discovery result has no valid remote content-update timestamp
|
||||
- **THEN** the target remains eligible for normal new-target admission but MUST NOT trigger an updated-target rescan
|
||||
|
||||
#### Scenario: DockerHub discovery
|
||||
- **WHEN** DockerHub resolves an image tag
|
||||
- **THEN** the existing digest identity and refresh behavior remain authoritative without updated-target promotion
|
||||
|
||||
### Requirement: Only changed completed targets are promoted
|
||||
The system SHALL promote a known target only when it is completed, its observed remote timestamp is strictly newer than the remote timestamp covered by its latest scan, and its configured cooldown has elapsed.
|
||||
|
||||
#### Scenario: Completed target changed after its covered revision
|
||||
- **WHEN** discovery observes a newer valid remote timestamp for a completed target after cooldown
|
||||
- **THEN** the target becomes pending for one normal fenced scan
|
||||
|
||||
#### Scenario: Unchanged target is rediscovered
|
||||
- **WHEN** discovery observes the same or an older remote timestamp
|
||||
- **THEN** the completed target remains unchanged and unclaimable
|
||||
|
||||
#### Scenario: Legacy completed target is first observed
|
||||
- **WHEN** a completed target has no claimed remote timestamp
|
||||
- **THEN** it is promoted only if the observed remote timestamp is strictly newer than its last completion time
|
||||
|
||||
#### Scenario: Non-completed target is rediscovered
|
||||
- **WHEN** the target is failed, pending, deferred, in progress, unresolved, leased, claimed, or reserved
|
||||
- **THEN** updated-target discovery MUST NOT alter its lifecycle or authority fields
|
||||
|
||||
#### Scenario: Update arrives during a scan
|
||||
- **WHEN** discovery records a newer remote timestamp after the active claim captured its scan timestamp
|
||||
- **THEN** completion preserves the newer observation and permits one bounded follow-up scan after cooldown
|
||||
|
||||
### Requirement: Updated-target admission is bounded
|
||||
The system SHALL enforce a source-configured hard maximum of updated-target promotions per discovery cycle and a per-target cooldown.
|
||||
|
||||
#### Scenario: Eligible changes exceed the cycle budget
|
||||
- **WHEN** more completed changed targets are eligible than the configured maximum
|
||||
- **THEN** at most the configured maximum are promoted and the remainder stay eligible for later cycles
|
||||
|
||||
#### Scenario: Cooldown has not elapsed
|
||||
- **WHEN** a changed completed target was scanned within the configured cooldown
|
||||
- **THEN** it remains completed until a later eligible cycle
|
||||
|
||||
#### Scenario: Concurrent discovery cycles
|
||||
- **WHEN** multiple workers observe the same changed target concurrently
|
||||
- **THEN** transactional row fencing permits at most one promotion for the covered revision
|
||||
|
||||
### Requirement: Updated discovery remains capable of seeing changed known targets
|
||||
The system SHALL NOT use identity-only known-page stopping for a source while updated-target rescans are enabled and SHALL keep discovery bounded by explicit page and result limits.
|
||||
|
||||
#### Scenario: Known identities appear on an early page
|
||||
- **WHEN** an early discovery page contains only known target identities
|
||||
- **THEN** discovery continues within its configured hard page limit so later changed targets can be observed
|
||||
|
||||
#### Scenario: HuggingFace Space was created long ago and recently updated
|
||||
- **WHEN** a known Space has a recent `lastModified` value
|
||||
- **THEN** update-sorted HuggingFace discovery can observe it independently of creation time
|
||||
|
||||
### Requirement: Claims record the covered remote revision
|
||||
The system SHALL atomically snapshot the newest observed remote timestamp when a target is claimed.
|
||||
|
||||
#### Scenario: Revision-aware target is claimed
|
||||
- **WHEN** a pending target receives a valid lease and reservation
|
||||
- **THEN** its scan-covered timestamp equals the newest remote timestamp observed before that claim
|
||||
|
||||
#### Scenario: Claim fails before authority is committed
|
||||
- **WHEN** capacity or fencing prevents the claim
|
||||
- **THEN** the scan-covered timestamp MUST NOT advance
|
||||
|
||||
### Requirement: Updated-target activity is separately observable
|
||||
The system SHALL report updated-target promotions separately from newly discovered target admissions in durable cycle metrics, source logs, and dashboard summaries.
|
||||
|
||||
#### Scenario: Cycle admits new and updated targets
|
||||
- **WHEN** a discovery cycle inserts new identities and promotes changed completed identities
|
||||
- **THEN** `queued_new_count` and `queued_updated_count` record the respective counts without overlap
|
||||
|
||||
#### Scenario: No changed targets are promoted
|
||||
- **WHEN** a cycle observes only unchanged or ineligible known targets
|
||||
- **THEN** `queued_updated_count` is zero
|
||||
@@ -0,0 +1,24 @@
|
||||
## 1. Queue State And Atomic Admission
|
||||
|
||||
- [x] 1.1 Add idempotent PostgreSQL schema support for observed and scan-covered remote timestamps, updated-target cycle counts, and eligibility indexing.
|
||||
- [x] 1.2 Implement atomic discovery observation and bounded promotion for changed completed targets while preserving all non-eligible lifecycle authority.
|
||||
- [x] 1.3 Snapshot the observed remote timestamp only when a legacy or slot-first target claim commits.
|
||||
|
||||
## 2. Discovery And Runner Integration
|
||||
|
||||
- [x] 2.1 Preserve GitHub `pushed_at`, GitLab `last_activity_at`, and HuggingFace `lastModified` through target preparation.
|
||||
- [x] 2.2 Make HuggingFace discovery newest-modified and disable identity-only page stopping when updated rescans are enabled.
|
||||
- [x] 2.3 Wire source-specific enablement, per-cycle caps, and cooldowns without changing DockerHub or enabling refresh-while-backlogged discovery.
|
||||
|
||||
## 3. Observability
|
||||
|
||||
- [x] 3.1 Persist and log `queued_updated_count` separately from new-target admission.
|
||||
- [x] 3.2 Add updated-target admission to dashboard source-cycle summaries.
|
||||
|
||||
## 4. Verification And Rollout
|
||||
|
||||
- [x] 4.1 Add focused discovery and runner tests for timestamp preservation, invalid timestamps, known-page behavior, and DockerHub isolation.
|
||||
- [x] 4.2 Add PostgreSQL integration tests for legacy baselines, changed and unchanged targets, cooldown and cap enforcement, concurrent fencing, claim snapshots, and mid-scan updates.
|
||||
- [x] 4.3 Run focused and full regression suites with bytecode writes disabled and run strict OpenSpec validation.
|
||||
- [x] 4.4 Restart the authority-managed runtime and verify PostgreSQL, pipeline, source, queue, and recorder health.
|
||||
- [x] 4.5 Observe a one-per-cycle production canary and compare changed-target admission, completion, findings, and worker occupancy before increasing any cap.
|
||||
Reference in New Issue
Block a user