Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,52 @@
+## Context
+
+The current Streamlit dashboard has six visible pages. PostgreSQL is authoritative for scans, findings, queues, and keychecks, but several pages still read compatibility files or filter a pre-limited client-side frame. The dashboard is read-only, loopback-only, supervisor-managed, and disabled by default; those safety boundaries must remain.
+
+## Goals / Non-Goals
+
+**Goals:**
+- Present the operational summary and finding lookup on one understandable page.
+- Apply one selected UTC time window consistently to all historical statistics.
+- Use PostgreSQL for every result, queue, and validation value.
+- Accept convenient pasted lookup values without querying or rendering raw secret columns.
+- Keep queries bounded and preserve degraded behavior when PostgreSQL is unavailable.
+
+**Non-Goals:**
+- Add scanner lifecycle controls or expose the dashboard beyond loopback.
+- Add a REST service, database migration, or write path.
+- Preserve retired queue-file and compatibility-file diagnostics in the visible UI.
+- Turn the dashboard into a raw log or raw credential browser.
+
+## Decisions
+
+1. **Keep Streamlit and replace only the visible information architecture.** The existing supervisor launch, dependency isolation, read-only connection, redaction, and health checks are valuable. A new frontend/backend split would add operational surface without improving this local dashboard.
+
+2. **Render one page with four sections.** The order is lookup, time-window KPIs, source/alive breakdowns, then compact runtime health. The lookup is placed first because it is a direct operator task; all secondary diagnostics are omitted or collapsed.
+
+3. **Use bound UTC timestamps in SQL before aggregation and limits.** Presets cover 1 hour, 24 hours, 7 days, and 30 days; custom start/end values are converted to UTC. This replaces the current newest-5000-rows client filter.
+
+4. **Use only authoritative stores.** Historical metrics come from `source_cycles`, `target_scans`, `findings`, `keycheck_credentials`, `keycheck_current_state`, and `keycheck_results`. Current backlog and pipeline state come from PostgreSQL. Supervisor status and `scan_limiter.db` remain valid operational sidecars. Retired queue files, global runner state, TSV summaries, and JSONL metrics are not rendered.
+
+5. **Normalize lookup input before database access.** A 64-hex value is treated as a digest; numeric and UID-like values are searched by indexed identity; credential-like text is SHA-256 hashed in memory and only the digest reaches SQL. Short or structured non-secret metadata uses escaped `LIKE` predicates. No query selects raw payload columns, and the submitted raw value is never echoed.
+
+6. **Lookup overrides statistical filters.** Search results always show current validation state and all bounded matching origins regardless of the selected reporting period or access-tier preset. This avoids hiding a valid credential behind an unrelated dashboard filter.
+
+7. **Retain bounded rendering and query failure isolation.** Lookup origins and breakdowns have explicit limits. Query failure rolls back the PostgreSQL transaction and degrades only the affected section.
+
+## Risks / Trade-offs
+
+- **Raw pasted credentials traverse the local Streamlit session before hashing** -> Keep loopback-only authority, hash immediately, never log/echo/persist the input, and offer SHA-256 lookup as the safest path.
+- **Broad metadata lookup can be expensive** -> Escape wildcard characters, require a useful minimum length, search indexed exact identities first, and cap returned origins.
+- **Removing advanced pages hides forensic diagnostics** -> PostgreSQL and log files remain available to engineering tools; the operator dashboard intentionally prioritizes clarity.
+- **Existing helper functions may remain temporarily unused** -> Remove the visible routes first and prune only when tests establish that no safety helper depends on them.
+
+## Migration Plan
+
+1. Add and test the new query/lookup helpers.
+2. Switch `main()` to the single page and disable Streamlit telemetry.
+3. Run focused dashboard tests and a live supervisor-launched desktop/mobile smoke test.
+4. Keep dashboard startup disabled by default. Rollback is a source revert; no persisted data changes are involved.
+
+## Open Questions
+
+None required for the initial implementation.