sashatrask
1.5 KiB
1.5 KiB
Why
GitHub, GitLab, and HuggingFace discovery continuously observe recently changed repositories and Spaces, but the queue permanently suppresses every URL or Space ID after its first scan. Production consequently fetched roughly 290,000 discovery results in the latest 24 hours while admitting less than one percent as targets, leaving scan capacity underused and ignoring secrets added to already-known projects.
What Changes
- Preserve the remote content-update timestamp supplied by GitHub, GitLab, and HuggingFace discovery.
- Requeue a completed target only when the observed remote content timestamp is newer than its last completed scan.
- Bound changed-target admission per source cycle and enforce a per-target cooldown.
- Keep active, deferred, failed, and unchanged targets untouched.
- Expose changed-target requeue counts separately from newly discovered target counts.
- Leave DockerHub digest-based refresh behavior unchanged.
Capabilities
New Capabilities
updated-target-rescan: Safely detect and rescan remotely changed core repository and Space targets under bounded rollout controls.
Modified Capabilities
None.
Impact
- Discovery metadata and target preparation in
app/scanner.pyandapp/console_runner.py. - PostgreSQL target queue state, migrations, cycle accounting, and dashboard observability in
app/scanner_db.pyandapp/dashboard.py. - Source configuration for GitHub, GitLab, and HuggingFace.
- Focused queue, discovery, and PostgreSQL integration tests.