Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,36 @@
+## 1. Input Processing
+
+- [x] 1.1 Add shared keycheck input state storage for per-service file path, file size, inode/signature if available, and last processed byte offset.
+- [x] 1.2 Extend shared keycheck input reader to support high-watermark processing with safe reset on file truncation or rotation.
+- [x] 1.3 Migrate hand-rolled readers in OpenAI, OpenRouter, and Gemini to the shared input reader.
+- [x] 1.4 Keep a bounded tail fallback for first run or missing state, with clear logging of the active input mode.
+
+## 2. Known-Key Occurrence Recording
+
+- [x] 2.1 Add a shared helper that resolves cached status for a key from checked/status files without provider API calls.
+- [x] 2.2 Add occurrence recording for skipped known keys, including service, cached status, source line, finding payload, and detector.
+- [x] 2.3 Mark cached occurrence rows distinctly from API-check rows in metadata.
+- [x] 2.4 Deduplicate cached occurrences per service/key/finding/source to avoid unbounded repeated rows.
+
+## 3. DB Write Reliability
+
+- [x] 3.1 Ensure per-result keycheck DB writes do not run schema initialization or migration DDL.
+- [x] 3.2 Add bounded lock retry/backoff for keycheck result and occurrence writes.
+- [x] 3.3 Add keycheck runner counters for DB write success, retry, and failure counts per service.
+- [x] 3.4 Log DB write failures without preventing status-file updates.
+
+## 4. Dashboard Visibility
+
+- [x] 4.1 Add dashboard panel comparing status-file current-state counts with latest DB observation timestamps.
+- [x] 4.2 Add keycheck pipeline health panel showing last completed service, current/stuck service, run duration, and DB write errors.
+- [x] 4.3 Add current usable-key view based on provider-specific access tiers and exact statuses.
+- [x] 4.4 Add historical source-yield view that includes cached known-key occurrences.
+- [x] 4.5 Label cached-status rows separately from fresh API-check rows in validation tables.
+
+## 5. Verification
+
+- [x] 5.1 Add a smoke test or script that runs a checker over a small fixture and verifies status-file writes plus DB occurrence rows.
+- [x] 5.2 Verify OpenAI, OpenRouter, Gemini, AWS, and GCP checkers use bounded/incremental input processing.
+- [x] 5.3 Verify a known alive key found in a new source/query creates a cached occurrence without an API recheck.
+- [x] 5.4 Verify dashboard can show current-state file counts even when DB observations are stale.
+- [x] 5.5 Run `python -m py_compile` on changed Python modules.