Initial server source import
This commit is contained in:
sashatrask
@@ -0,0 +1,339 @@
|
||||
# End-to-End Scanner Validation
|
||||
|
||||
This runbook validates the real discovery, remote-worker, result-ingestion, and
|
||||
compatibility-projection path with a bounded cohort of 30-40 targets. It is an
|
||||
evidence procedure, not a claim that every repository feature and environment
|
||||
has been proven correct.
|
||||
|
||||
The completed 2026-09-22 production evidence is recorded in
|
||||
[`end-to-end-scanner-validation-2026-09-22.md`](end-to-end-scanner-validation-2026-09-22.md).
|
||||
|
||||
## Validation Questions
|
||||
|
||||
The run must answer all of the following:
|
||||
|
||||
1. Does bounded discovery create the expected immutable queue identities?
|
||||
2. Do remote workers receive each authoritative assignment with the correct
|
||||
source, target identity, execution snapshot, and lease fencing?
|
||||
3. Does each worker run the intended TruffleHog scan and upload a canonical
|
||||
protocol-2 result bundle?
|
||||
4. Does the server accept a result exactly once and make receipt replay
|
||||
idempotent?
|
||||
5. Does the ingester transactionally connect the reservation, queue row,
|
||||
target scan, findings, errors, and bundle record?
|
||||
6. Does the JSONL projector reproduce the Windows-compatible
|
||||
`scan_results.jsonl` and `found_secrets.jsonl` structures without becoming
|
||||
a second source of truth?
|
||||
7. Are naturally found or controlled-canary findings stored with safe identity,
|
||||
location, detector, verification, redaction, and provenance fields?
|
||||
8. After the test, are queues settled, projections caught up, no pipeline item
|
||||
quarantined, and the normal production configuration restored?
|
||||
|
||||
## Authorities and Expected Data Flow
|
||||
|
||||
The authoritative sequence is:
|
||||
|
||||
```text
|
||||
discovery cycle
|
||||
-> target_queue
|
||||
-> result_reservation / immutable remote assignment
|
||||
-> worker TruffleHog execution
|
||||
-> canonical .trb upload
|
||||
-> durable accepted receipt
|
||||
-> result ingester transaction
|
||||
-> target_scans + findings + errors + queue completion
|
||||
-> projection_jobs
|
||||
-> scan_results.jsonl + found_secrets.jsonl
|
||||
```
|
||||
|
||||
PostgreSQL is authoritative. Result bundles are durable pipeline artifacts.
|
||||
JSONL files are rebuildable compatibility projections and may lag briefly.
|
||||
`accepted` proves durable server receipt; `ingested` proves the database
|
||||
transaction completed. These states must not be treated as synonyms.
|
||||
|
||||
## Safety Rules
|
||||
|
||||
- Use only the approved test server and approved worker devices.
|
||||
- Never print device tokens, source credentials, raw secrets, active runtime
|
||||
YAML, worker argv, or complete unredacted findings into a terminal/log.
|
||||
- Query finding structure using IDs, hashes, redacted values, lengths, booleans,
|
||||
detector names, and location metadata. Review any raw secret only through the
|
||||
already protected admin workflow if explicitly required.
|
||||
- Apply and restore configuration through Preview -> Save candidate -> Apply.
|
||||
Do not edit the active runtime document in place.
|
||||
- Pause discovery and dispatch and drain before each runtime-document apply.
|
||||
- Record the original config SHA-256 and require byte-identical restoration at
|
||||
the end.
|
||||
- Use a unique test-run label and database high-water marks. Never infer the
|
||||
cohort from wall-clock time alone.
|
||||
- Do not delete queue, bundle, finding, projection, or receipt evidence to make
|
||||
a failed test look clean.
|
||||
|
||||
## Bounded Test Configuration
|
||||
|
||||
Use a temporary candidate derived from the active document. Preserve all
|
||||
secrets and unrelated settings. The exact candidate must be reviewed before it
|
||||
is applied.
|
||||
|
||||
### DockerHub
|
||||
|
||||
- `mode: search`
|
||||
- `pages: 1`
|
||||
- `per_page: 1`
|
||||
- `docker_images_per_repository: 1`
|
||||
- Use a reviewed finite query list for the test window.
|
||||
|
||||
One query is consumed per source cycle. An already-known or unsuitable search
|
||||
result can produce no new queue row, so the number of cycles is not the cohort
|
||||
size.
|
||||
|
||||
### GitLab
|
||||
|
||||
- `mode: search`
|
||||
- `pages: 1`
|
||||
- `per_page: 1`
|
||||
- Use a reviewed finite query list for the test window.
|
||||
- Keep current age, commit-boundary, exact-ref, visibility, and history-depth
|
||||
safety controls unless the test explicitly records a different expectation.
|
||||
|
||||
### HuggingFace
|
||||
|
||||
HuggingFace recent discovery does not support a real `per_page: 1` keyword
|
||||
test. Its API runner fetches newest-modified Spaces and the current API page
|
||||
size is fixed at 100; the configured query is only a rotation placeholder.
|
||||
|
||||
For a bounded cohort, use `mode: custom` with a reviewed private `target_file`
|
||||
containing a small list of Space IDs. Do not claim that changing `per_page` to
|
||||
1 bounded this source when it did not.
|
||||
|
||||
### Recommended Cohort
|
||||
|
||||
Target 36 authoritative terminal scans:
|
||||
|
||||
- 16 DockerHub immutable digest targets;
|
||||
- 16 GitLab exact-ref/commit-planned targets;
|
||||
- 4 HuggingFace custom Space targets.
|
||||
|
||||
The exact split may vary between 30 and 40 when discovery deduplicates known
|
||||
targets or a target becomes permanently inaccessible. Continue only until the
|
||||
recorded cohort reaches the agreed bound. Do not inflate discovery simply to
|
||||
hit an exact aesthetic number.
|
||||
|
||||
## Positive-Finding Requirement
|
||||
|
||||
A random public cohort may correctly produce zero findings. Zero findings
|
||||
cannot validate the finding-storage and `found_secrets.jsonl` path.
|
||||
|
||||
Include at least one separately identified, non-live controlled fixture that is
|
||||
expected to trigger an already approved detector. The fixture must contain no
|
||||
usable credential. Record its expected detector and identity before scanning.
|
||||
Do not weaken verification, introduce a new detector, or publish a real secret
|
||||
merely to force a positive result.
|
||||
|
||||
If no approved positive fixture is available, report the finding path as
|
||||
unverified by this run even if all zero-finding scans succeed.
|
||||
|
||||
## Phase 1: Baseline
|
||||
|
||||
With runtime healthy, record a secret-safe baseline:
|
||||
|
||||
- active config SHA-256 and semantic config SHA-256;
|
||||
- runtime-control revision and open/paused/drain state;
|
||||
- enabled source set and active worker package manifests;
|
||||
- remote worker/device count, recent contact, and package capability match;
|
||||
- high-water IDs for `target_queue`, `result_reservations`, `target_scans`,
|
||||
`findings`, `errors`, `result_bundles`, and `projection_jobs`;
|
||||
- queue counts by source and status;
|
||||
- active reservation count and oldest age;
|
||||
- pipeline worker readiness and capacity counters;
|
||||
- pending/leased/quarantined bundle, projection, and keycheck counts;
|
||||
- current projection stream/cursor identity;
|
||||
- byte size and final complete-line identity of active JSONL files.
|
||||
|
||||
The baseline collector must print aggregates and hashes only. It must not emit
|
||||
targets, assignment payloads, tokens, raw findings, or runtime documents.
|
||||
|
||||
## Phase 2: Apply the Test Candidate
|
||||
|
||||
1. Pause discovery.
|
||||
2. Pause dispatch.
|
||||
3. Start drain and wait for blocker count zero and `drained`.
|
||||
4. Preview the bounded candidate and review the semantic diff.
|
||||
5. Save and apply the candidate through the host-agent lifecycle.
|
||||
6. Require reconciled `succeeded`, no failed hold, strict runtime health, edge
|
||||
health, admin health, and Worker API health.
|
||||
7. Cancel drain, then resume discovery and dispatch in that order.
|
||||
|
||||
Do not continue if the lifecycle operation rolls back or enters failed hold.
|
||||
|
||||
## Phase 3: Build and Freeze the Cohort
|
||||
|
||||
Record the baseline `target_queue.id` high-water mark. Let the bounded sources
|
||||
cycle until 30-40 new eligible queue rows have been created after that mark.
|
||||
|
||||
Then:
|
||||
|
||||
1. Pause discovery so the cohort cannot grow.
|
||||
2. Leave dispatch open until the selected queue rows settle.
|
||||
3. Record cohort queue IDs and only their safe identities: source, normalized
|
||||
target hash, query hash, immutable planning kind, and creation order.
|
||||
4. Separate deduplicated, permanently inaccessible, deferred, retried, and
|
||||
actually assigned items. Do not count an API result as a scan.
|
||||
|
||||
The authoritative cohort is a fixed set of queue IDs, not "whatever completed
|
||||
during the same hour."
|
||||
|
||||
## Phase 4: Observe Remote Execution
|
||||
|
||||
For every cohort queue ID, verify:
|
||||
|
||||
- no more than one current authoritative reservation;
|
||||
- assignment package/platform capability matches the registered worker;
|
||||
- lease token and execution snapshot are bound but never printed;
|
||||
- Docker targets are immutable `repo@sha256` identities;
|
||||
- GitLab targets have the intended exact planning/ref identity;
|
||||
- HuggingFace targets use the direct Space execution kind;
|
||||
- terminal report classification is success, permanent target failure, or
|
||||
retryable provider failure as designed;
|
||||
- retries preserve queue identity and increment attempts without creating a
|
||||
second authoritative acceptance;
|
||||
- accepted receipt replay returns the same durable result.
|
||||
|
||||
Physical work can repeat after a lease expiry or network partition. Correctness
|
||||
means fencing permits one authoritative acceptance and one queue completion,
|
||||
not that duplicate physical execution is impossible.
|
||||
|
||||
## Phase 5: Validate Bundles and PostgreSQL Structure
|
||||
|
||||
For each accepted result, validate without dumping body content:
|
||||
|
||||
- bundle exists at the registered private relative path;
|
||||
- bundle byte count and SHA-256 match database metadata;
|
||||
- bundle schema/version, event ID/hash, reservation ID, queue ID, source,
|
||||
normalized target identity, execution snapshot identity, and scan policy are
|
||||
internally consistent;
|
||||
- result is ingested exactly once;
|
||||
- `target_queue.target_scan_id` references the corresponding `target_scans.id`;
|
||||
- queue completion is applied once with a terminal disposition;
|
||||
- `target_scans.queue_id` and claim lease identity refer back to the cohort row;
|
||||
- `target_scans.findings_count` and `error_count` equal actual child-row counts;
|
||||
- every finding/error references the same target scan, source, cycle, and run;
|
||||
- no legacy `raw_result_json`, publication outbox row, or orphan relation is
|
||||
introduced;
|
||||
- ingested bundle credit and pipeline-capacity counters are released exactly
|
||||
according to the durable state machine;
|
||||
- no cohort item enters `pipeline_quarantine`.
|
||||
|
||||
Aggregate checks must cover the entire cohort. Additionally inspect a small
|
||||
redacted structural sample from every source and every terminal disposition.
|
||||
|
||||
## Phase 6: Validate Findings
|
||||
|
||||
For every finding in the cohort, inspect structure only:
|
||||
|
||||
- stable `finding_uid` and finding fingerprint;
|
||||
- detector name/type and verification flag;
|
||||
- source, target hash, file path, line/commit/source timestamp where applicable;
|
||||
- redacted secret and secret/detector hashes;
|
||||
- provider and credential-kind enrichment;
|
||||
- required-context and raw-payload-omitted flags;
|
||||
- bounded source metadata and enrichment JSON decode successfully;
|
||||
- no unexpected raw-secret exposure in logs, queue rows, assignment metadata,
|
||||
admin list views, or compatibility scan summaries.
|
||||
|
||||
For the controlled positive fixture, require the expected finding to exist in
|
||||
PostgreSQL and to project once to `found_secrets.jsonl`.
|
||||
|
||||
## Phase 7: Validate Windows-Compatible Files
|
||||
|
||||
Use the active configured `global.results_dir`. The relevant compatibility
|
||||
outputs are:
|
||||
|
||||
- `scan_results.jsonl` for one sanitized scan event per projected scan;
|
||||
- `found_secrets.jsonl` for projected finding events;
|
||||
- their publication ledgers, active stream metadata, and rotated segments;
|
||||
- per-service keycheck result files only if keychecks run for the finding.
|
||||
|
||||
For the cohort, verify:
|
||||
|
||||
- every required `projection_job` reaches `completed`;
|
||||
- projector cursor and append ledger advance monotonically;
|
||||
- each cohort scan event appears exactly once by `scan_event_id`;
|
||||
- each cohort finding appears exactly once by `finding_uid`;
|
||||
- JSON lines parse and match the current compatibility schema;
|
||||
- scan summaries match PostgreSQL counts and terminal status;
|
||||
- finding projections are redacted as designed and preserve safe provenance;
|
||||
- active and rotated segments together contain the events; checking only the
|
||||
active file is insufficient when rotation occurs;
|
||||
- no torn-tail quarantine, duplicate append, skipped cursor, or unpublished
|
||||
completed job exists.
|
||||
|
||||
These files should have the same logical structure as the Windows deployment,
|
||||
but path separators and host/container root paths are platform-specific.
|
||||
|
||||
## Phase 8: Queue and Pipeline Closure
|
||||
|
||||
After all cohort rows settle, require:
|
||||
|
||||
- 30-40 cohort queue rows accounted for by terminal, deferred, or explicitly
|
||||
classified retry state;
|
||||
- no expired active reservation remains unreaped;
|
||||
- no queue row has multiple authoritative accepted results;
|
||||
- no accepted result remains un-ingested beyond the bounded pipeline window;
|
||||
- no completed scan remains unprojected beyond the bounded projector window;
|
||||
- no stale pipeline lease or capacity leak;
|
||||
- no unexpected quarantine;
|
||||
- runtime, Worker API, edge, host-agent, host Caddy, and unrelated host service
|
||||
health remain good.
|
||||
|
||||
The final report must show counts for discovered, deduplicated, assigned,
|
||||
retried, accepted, ingested, projected, succeeded, skipped/permanent,
|
||||
retryable/deferred, findings, errors, and quarantines.
|
||||
|
||||
## Phase 9: Restore Production Configuration
|
||||
|
||||
1. Pause discovery and dispatch.
|
||||
2. Drain to zero blockers.
|
||||
3. Apply the exact original runtime document through the normal lifecycle.
|
||||
4. Require byte-identical original config SHA-256, reconciled lifecycle success,
|
||||
strict health, and no failed hold.
|
||||
5. Cancel drain, resume discovery, then resume dispatch according to the
|
||||
original control state.
|
||||
6. Confirm worker contact and normal post-test assignment flow.
|
||||
|
||||
Do not restore by manually editing YAML or replacing files behind the
|
||||
host-agent.
|
||||
|
||||
## Pass Criteria
|
||||
|
||||
The run passes only when:
|
||||
|
||||
- at least 30 and at most 40 fixed-cohort rows are fully accounted for;
|
||||
- all accepted cohort results ingest exactly once;
|
||||
- all required cohort projections complete exactly once;
|
||||
- queue/reservation/scan/finding/error/bundle relationships are consistent;
|
||||
- the controlled positive finding reaches PostgreSQL and
|
||||
`found_secrets.jsonl`, or the report explicitly marks positive-finding
|
||||
validation incomplete because no approved fixture existed;
|
||||
- no unexplained retry, orphan, duplicate acceptance, capacity leak,
|
||||
quarantine, failed hold, or projection gap remains;
|
||||
- the original production config is restored exactly and services are healthy.
|
||||
|
||||
Any failure must retain its operation IDs, queue IDs, reservation IDs, hashes,
|
||||
safe categories, and aggregate evidence for diagnosis. A partial pass must not
|
||||
be reported as "100% scanner correctness."
|
||||
|
||||
## Evidence Report
|
||||
|
||||
Append or link a dated report containing:
|
||||
|
||||
- environment and worker package identities;
|
||||
- original/test/restored config hashes;
|
||||
- cohort definition and aggregate source split;
|
||||
- lifecycle operation IDs for test apply and restore;
|
||||
- queue and pipeline baseline/final aggregates;
|
||||
- per-stage reconciliation counts;
|
||||
- redacted structural examples for a scan, an error/skip, and a finding;
|
||||
- JSONL/ledger reconciliation counts;
|
||||
- deviations, retries, quarantines, and unresolved questions;
|
||||
- final verdict with explicit tested and untested boundaries.
|
||||
Reference in New Issue
Block a user