Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,31 @@
+## Why
+
+DockerHub discovery currently downloads its full configured page range before database deduplication, and one exhausted page discards every successful page while blocking keyword rotation. The authenticated 30-page window makes deeper discovery possible, but it needs incremental persistence, seen-page stopping, and durable retry delegation to remain efficient and avoid silent gaps or head-of-line blocking.
+
+## What Changes
+
+- **BREAKING**: replace complete-or-fail DockerHub pagination with page-level durable persistence; successful pages remain admitted when another page exhausts its two request attempts.
+- Stop an ordinary query pass after two consecutive nonempty pages whose repository identities were already known before the pass.
+- Run every DockerHub query with an effective ceiling of 30 pages and 100 results per page.
+- Bypass seen-page stopping for a full deep pass of each exact query at least once per 72-hour scheduling interval.
+- Delegate failed page/query acquisition to a fenced PostgreSQL discovery retry backlog before allowing the main keyword rotation to advance.
+- Add 12 DockerHub product/framework queries: `open-webui`, `ragflow`, `dify`, `flowise`, `crewai`, `n8n`, `langflow`, `autogen`, `browser-use`, `openhands`, `anythingllm`, and `agent-zero`.
+- Temporarily disable periodic re-resolution of completed DockerHub repository anchors while preserving initial resolution, partial/error resolver retries, immutable-digest scan retries, and normal keyword discovery.
+- Preserve authenticated fail-closed search, the two-GET page budget, target uniqueness, cold/failed target policy, and secret-safe diagnostics.
+
+## Capabilities
+
+### New Capabilities
+- `dockerhub-incremental-discovery`: Incremental DockerHub page admission, seen-page stopping, 72-hour deep passes, and durable failed-page retry work.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+- Affects DockerHub pagination and authentication integration in `app/scanner.py`, source-cycle orchestration/state in `app/console_runner.py`, PostgreSQL schema and retry claims in `app/scanner_db.py`, and DockerHub settings in `app/config.yaml`.
+- Adds a PostgreSQL discovery retry queue with bounded leases, fencing, backoff, and configured-query/policy validation.
+- Changes DockerHub query rotation from failure-blocking to durable retry delegation and adds an initial bounded backlog from 12 new deep searches.
+- Requires focused unit, SQL-shape, migration, PostgreSQL integration, source-state, configuration, and runtime health verification.
+- Does not add dependencies or credential formats and does not change Docker tag selection, Registry authentication, layer scanning, scan workers, keychecks, or immutable-digest retry policy.