truf-server/docs/end-to-end-scanner-validation-2026-09-22.md

Scanner End-to-End Validation Evidence: 2026-09-22

Verdict

The bounded production validation passed on the approved sec deployment. It exercised the real PostgreSQL queue, protocol-2 remote assignment, existing Windows/WSL worker, TruffleHog execution, bundle upload, durable receipt, transactional ingestion, normalized findings/errors, and JSONL compatibility projection paths.

The evidence consists of:

• 36 ordinary public-target scans under realistic production backlog;
• one separately managed non-live synthetic GitLab fixture scan proving the positive finding path;
• exact append-region validation for scan_results.jsonl and found_secrets.jsonl against PostgreSQL reconstruction;
• byte-identical restoration of the original production config;
• cleanup of private validation target files; and
• audited reopening of discovery and dispatch.

This is strong bounded production evidence, not a claim that every source, failure mode, platform, detector, scale, or deployment environment is proven.

Safety Envelope

• Only sec was used. prod was never touched.
• Raw targets, raw findings, credentials, device tokens, runtime YAML, worker argv, the protected admin prefix, and edge markers were not printed.
• Configuration changes used managed Preview -> Save candidate -> Apply.
• Discovery and dispatch were paused and the runtime was drained before every apply.
• Long operations and monitors ran detached and were observed with bounded status polls.
• Host Caddy and X-UI remained outside the managed lifecycle.
• PostgreSQL remained the sole authority; JSONL was treated as a rebuildable compatibility projection.

Original Baseline

• Original active config SHA-256: f055a9f2506ab4fffa6953a95c2f6c07b1202e6558f4ce52bbf1b463ed6b1781.
• Drained baseline high-water IDs:
  • target queue: 1,534,069;
  • result reservations: 880;
  • target scans: 879;
  • findings: 7;
  • errors: 1,913.
• Runtime controls were revision 26, paused/paused, drained, blockers zero.
• One active worker device had contacted the server recently.
• All baseline orphan and referential invariants were zero.

Root-only baseline evidence:

• /opt/truf-remote-server/staging/scanner-validation-pre.json;
• /opt/truf-remote-server/staging/scanner-validation-drained.json;
• /opt/truf-remote-server/staging/scanner-validation-pre-discovery-v4.json;
• /opt/truf-remote-server/staging/scanner-validation-pre-dispatch-newest-v5.json.

Defects Found and Corrected

The validation exposed defects that synthetic tests had not modeled precisely. Each failure was contained by pause/drain, rollback, or failed-hold behavior before dispatch was opened.

Protected Config Parent

Discovery required the parent of a private file to be runtime-owned mode 0700, while the deployed contract intentionally uses root-owned mode 0755 /data/config with runtime-owned mode 0600 documents. Writable runtime directories still require private runtime ownership. Sensitive file parents now also accept a non-link root-owned directory with no group/other write bits and effective-user search access. The private file itself remains strictly checked.

Supervisor Startup Locking

PostgreSQL readiness previously launched core children and discovery producers while the supervisor held control_lock, then could perform another PostgreSQL query under that lock. Child bootstrap/entrypoint authentication needed the same lock and had bounded deadlines. Pipeline status refresh now happens before the lock, structured snapshots use cached-only status, core children start before source admission, and discovery producers use the source dependency gate.

Strict Discovery Health

Ordinary Docker health intentionally tolerates periodic producer waits. Managed lifecycle health now additionally uses explicit --require-discovery-producers and rejects enabled producers that are absent, blocked, never run, runtime-blocked, or waiting after a nonzero exit. Waiting after a successful exit remains valid.

Transient Strict-Health Probe

The first fixture apply encountered one bounded HuggingFace PostgreSQL connection timeout after every core worker had started successfully. The host lifecycle formerly performed only one strict probe after Docker health became healthy. It now retries only health-category strict failures inside the existing 240-second runtime-health deadline. Identity and metadata errors remain immediate failures, and persistent health failure still rolls back.

Managed Claim Order

The remote assignment path already supported oldest, newest, and balanced PostgreSQL admission, but the exact managed template omitted this field for the three core sources. The optional field is now represented and semantically validated. Temporary newest ordering allowed recent bounded discoveries to be tested against the real 1.5-million-row queue without direct SQL mutation or mass-hiding historical backlog. The restored original config omits the optional field and therefore uses the normal oldest default.

Candidate Base Authority

Candidate preparation originally used editor text that could represent an old candidate rather than active config. This inherited an earlier intentionally disabled Worker API setting. Candidate tools now read and hash-bind active config bytes explicitly before deriving changes.

Runtime Deployment Evidence

The corrected runtime was built as small derived immutable images rather than modifying a running container. The final validation image ID was:

sha256:5b9c86f68719d8c1f2358e0c4565dd2795ed608482968747feb961cf14908b5a

Prior images remain under rollback tags. Image Entrypoint, Cmd, User, source hashes, and in-image compilation were checked. An official lifecycle restart on the final image completed succeeded/succeeded, reconciled, without a safe category or failed hold.

Relevant local regression evidence accumulated during the run:

• runtime-document and worker-assignment tests: 45 passed;
• host lifecycle after transient-health retry: 34 passed, 4 platform skips;
• combined ACL/supervisor/health-focused suite: 246 passed, 9 platform skips;
• authenticated supervisor control class: 20 passed;
• focused compiles and git diff --check: passed.

Bounded Discovery

The temporary candidate enabled one-page/one-result search settings for GitLab and DockerHub and a four-item private custom file for HuggingFace. Dispatch remained paused. Five successful cycles for each source completed before the monitor's conservative time limit; no source cycle failed.

Because source cycles do not map directly to queue rows and uniqueness conflicts consume sequence values, queue high-water deltas were not treated as exact cohort membership. Eight new queue rows were observed: five GitLab pending and three DockerHub deferred. No direct queue updates were made.

Realistic 36-Scan Cohort

The worker processed exactly 36 new remote reservations, IDs 881 through 916, while discovery remained paused. A fail-closed monitor paused dispatch at the target and started drain. Final source mix:

Source	Scans
DockerHub	11
GitLab	14
HuggingFace	11
Total	36

All 36 reservations were remote, resolved, acknowledged, and bundle_accepted. They had 36 distinct queue IDs, bundle IDs, and scan event IDs, and every reservation had a receipt, payload hash, and execution-snapshot hash.

Results

Source	Result summary
DockerHub	8 clean, 3 degraded
GitLab	12 clean, 1 retryable API error, 1 permanent not-found
HuggingFace	11 clean

• Queue completion: 34 done, one deferred, one failed; no row remained fenced.
• Findings: zero, a valid outcome for random public targets.
• Errors: exactly two GitLab errors with queue dispositions matching their retryable/permanent categories.
• Quarantine: zero new rows.
• Bundle/projection capacity after completion: zero items and zero bytes.
• Existing unrelated keycheck capacity was unchanged.

Bundle and Projection Invariants

• 36 acknowledged bundles contained 110 frames.
• All bundle identities and counts matched their reservations and scans.
• Acknowledged physical .trb files were absent only after both pipeline artifact records reached durable deleted state, as designed.
• 36 scans used raw_result_storage=normalized_v2.
• 36 compatibility rows used expected bounded reconstruction.
• Exactly 36 projection jobs completed, one per scan, without duplicates or errors; all projection capacity was released.
• Physical append evidence covered 36 scan_results records and two scan_errors records.
• Every registered append generation/offset/length existed and matched its payload SHA-256, record count, and required JSON structure.
• scan_results.jsonl grew by exactly 83,752 bytes.
• found_secrets.jsonl did not change, matching zero random-target findings.
• All global queue/reservation and orphan invariants remained zero.

Root-only evidence:

• post snapshot: /opt/truf-remote-server/staging/scanner-validation-post-dispatch-newest-v5.json, SHA-256 45f9db81e88dbcbe4d6a04094dd1d892df06dd3f7cdd05eb9280c19da54aed94;
• aggregate report: /opt/truf-remote-server/staging/scanner-validation-cohort-report-v5.json, SHA-256 e0826587e4ba667de10a994d5842aae5fae8fa11dc071210c202b38b0e643bf4.

Controlled Positive Fixture

Random public targets produced no finding, so a separate one-target run used a public GitLab project whose README declares that its secret examples are generated and non-live. No detector or verification behavior was weakened.

• Fixture queue ID: 1,534,100.
• Reservation ID: 917.
• The immutable Git plan bound the approved exact head commit 2a09bd6767d39b95cf39ce4b5fd210721275d503.
• The reservation became acknowledged with bundle_accepted and a durable receipt.
• Queue completion was done with no remaining reservation fence.
• Target scan status was found with 116 findings and zero errors.
• All 116 findings used the existing OpenAI detector.
• Verified count was zero, consistent with the unchanged no-verification policy.
• All findings had distinct finding UIDs, nonempty identities, private raw material, redaction different from raw material, correct secret hashes, and complete non-omitted compatibility payloads.
• No raw finding value was emitted by validation tooling.
• Bundle retirement and both pipeline artifact tombstones were correct.
• The single projection job completed and released capacity.
• The registered scan_results region contained one record with exactly 116 findings and zero errors.
• The registered found_secrets region contained exactly 116 records.
• Both physical append regions matched the database payload SHA-256 and were byte-identical to fresh PostgreSQL compatibility reconstruction.
• No new quarantine row was created.

Root-only fixture report:

/opt/truf-remote-server/staging/scanner-validation-fixture-report-v2.json

SHA-256:

243a16b24bf9ae898bfdeb8f857c56ef1cf78e12e637730ff5b0674a250a4984

Restoration and Final State

The original 36,354 config bytes were passed through managed Preview, saved as a candidate, and applied through the host agent. Preview preserved the exact original SHA-256 and reported 169 semantic reversions.

• Restore Save operation: af2a3cc4-90ae-5831-b932-bbe78ceb2cab.
• Restore Apply operation: b12a2fc4-202a-578e-8dcf-b0a88cb028ef.
• Apply terminal state: succeeded/succeeded, reconciled, category None.
• Active and candidate config SHA-256 both equal the original f055a9f2506ab4fffa6953a95c2f6c07b1202e6558f4ce52bbf1b463ed6b1781.
• Lifecycle preflight and strict Worker API/discovery health passed.
• Runtime and edge were healthy; no failed hold existed.
• All private validation target/evidence files were removed.
• The root-only original backup was retained for audit.

The drained post-restore snapshot is root-only at /opt/truf-remote-server/staging/scanner-validation-post-restore-drained-v1.json, SHA-256 de4bd98d728dc551b10712a4afb6be2db0ac9111428d46fa5dbb16f8d2d611ca.

Final audited control transitions advanced revision 46 to 49 in this order:

1. cancel drain;
2. resume discovery;
3. resume dispatch.

Final state was discovery open, dispatch open, drain normal. The existing worker contacted the server within five minutes and immediately received normal production work. A live assignment after reopening is expected and is not a drain blocker because drain is no longer requested.

The final post-resume snapshot had zero orphan/referential invariants and preserved the original config SHA-256:

/opt/truf-remote-server/staging/scanner-validation-post-resume-final-v1.json

SHA-256:

42e19edae09550693d563b74631430cb1d2c1b807d636ccff20e545cebec3c2d

External route checks through existing host Caddy returned:

• invalid Worker API authentication: 401;
• unauthenticated protected admin route: 401;
• unrelated path: 404.

Host-agent, Caddy, and X-UI services remained active. Caddy and X-UI were not lifecycle targets.

Residual Limits

This validation does not prove:

• long-duration soak or high-concurrency behavior;
• every detector and verification provider;
• every source mode, browser, OS, architecture, or network failure;
• every secrets/config mutation and rotation case;
• HA or multi-server operation;
• resistance to an independent penetration test; or
• correctness of arbitrary unsupported Compose, ingress, or proxy layouts.

Within its declared scope, the real queue, worker, scanner, ingestion, findings, error, compatibility, restoration, and resumed-production paths all produced internally consistent durable evidence.