truf-server/docs/end-to-end-scanner-validation-2026-09-22.md

# Scanner End-to-End Validation Evidence: 2026-09-22

## Verdict

The bounded production validation passed on the approved `sec` deployment.
It exercised the real PostgreSQL queue, protocol-2 remote assignment, existing
Windows/WSL worker, TruffleHog execution, bundle upload, durable receipt,
transactional ingestion, normalized findings/errors, and JSONL compatibility
projection paths.

The evidence consists of:

- 36 ordinary public-target scans under realistic production backlog;
- one separately managed non-live synthetic GitLab fixture scan proving the
  positive finding path;
- exact append-region validation for `scan_results.jsonl` and
  `found_secrets.jsonl` against PostgreSQL reconstruction;
- byte-identical restoration of the original production config;
- cleanup of private validation target files; and
- audited reopening of discovery and dispatch.

This is strong bounded production evidence, not a claim that every source,
failure mode, platform, detector, scale, or deployment environment is proven.

## Safety Envelope

- Only `sec` was used. `prod` was never touched.
- Raw targets, raw findings, credentials, device tokens, runtime YAML, worker
  argv, the protected admin prefix, and edge markers were not printed.
- Configuration changes used managed Preview -> Save candidate -> Apply.
- Discovery and dispatch were paused and the runtime was drained before every
  apply.
- Long operations and monitors ran detached and were observed with bounded
  status polls.
- Host Caddy and X-UI remained outside the managed lifecycle.
- PostgreSQL remained the sole authority; JSONL was treated as a rebuildable
  compatibility projection.

## Original Baseline

- Original active config SHA-256:
  `f055a9f2506ab4fffa6953a95c2f6c07b1202e6558f4ce52bbf1b463ed6b1781`.
- Drained baseline high-water IDs:
  - target queue: 1,534,069;
  - result reservations: 880;
  - target scans: 879;
  - findings: 7;
  - errors: 1,913.
- Runtime controls were revision 26, paused/paused, `drained`, blockers zero.
- One active worker device had contacted the server recently.
- All baseline orphan and referential invariants were zero.

Root-only baseline evidence:

- `/opt/truf-remote-server/staging/scanner-validation-pre.json`;
- `/opt/truf-remote-server/staging/scanner-validation-drained.json`;
- `/opt/truf-remote-server/staging/scanner-validation-pre-discovery-v4.json`;
- `/opt/truf-remote-server/staging/scanner-validation-pre-dispatch-newest-v5.json`.

## Defects Found and Corrected

The validation exposed defects that synthetic tests had not modeled precisely.
Each failure was contained by pause/drain, rollback, or failed-hold behavior
before dispatch was opened.

### Protected Config Parent

Discovery required the parent of a private file to be runtime-owned mode 0700,
while the deployed contract intentionally uses root-owned mode 0755
`/data/config` with runtime-owned mode 0600 documents. Writable runtime
directories still require private runtime ownership. Sensitive file parents now
also accept a non-link root-owned directory with no group/other write bits and
effective-user search access. The private file itself remains strictly checked.

### Supervisor Startup Locking

PostgreSQL readiness previously launched core children and discovery producers
while the supervisor held `control_lock`, then could perform another PostgreSQL
query under that lock. Child bootstrap/entrypoint authentication needed the same
lock and had bounded deadlines. Pipeline status refresh now happens before the
lock, structured snapshots use cached-only status, core children start before
source admission, and discovery producers use the source dependency gate.

### Strict Discovery Health

Ordinary Docker health intentionally tolerates periodic producer waits. Managed
lifecycle health now additionally uses explicit
`--require-discovery-producers` and rejects enabled producers that are absent,
blocked, never run, runtime-blocked, or waiting after a nonzero exit. Waiting
after a successful exit remains valid.

### Transient Strict-Health Probe

The first fixture apply encountered one bounded HuggingFace PostgreSQL
connection timeout after every core worker had started successfully. The host
lifecycle formerly performed only one strict probe after Docker health became
healthy. It now retries only health-category strict failures inside the existing
240-second runtime-health deadline. Identity and metadata errors remain
immediate failures, and persistent health failure still rolls back.

### Managed Claim Order

The remote assignment path already supported `oldest`, `newest`, and `balanced`
PostgreSQL admission, but the exact managed template omitted this field for the
three core sources. The optional field is now represented and semantically
validated. Temporary `newest` ordering allowed recent bounded discoveries to be
tested against the real 1.5-million-row queue without direct SQL mutation or
mass-hiding historical backlog. The restored original config omits the optional
field and therefore uses the normal `oldest` default.

### Candidate Base Authority

Candidate preparation originally used editor text that could represent an old
candidate rather than active config. This inherited an earlier intentionally
disabled Worker API setting. Candidate tools now read and hash-bind active
config bytes explicitly before deriving changes.

## Runtime Deployment Evidence

The corrected runtime was built as small derived immutable images rather than
modifying a running container. The final validation image ID was:

`sha256:5b9c86f68719d8c1f2358e0c4565dd2795ed608482968747feb961cf14908b5a`

Prior images remain under rollback tags. Image Entrypoint, Cmd, User, source
hashes, and in-image compilation were checked. An official lifecycle restart on
the final image completed `succeeded/succeeded`, reconciled, without a safe
category or failed hold.

Relevant local regression evidence accumulated during the run:

- runtime-document and worker-assignment tests: 45 passed;
- host lifecycle after transient-health retry: 34 passed, 4 platform skips;
- combined ACL/supervisor/health-focused suite: 246 passed, 9 platform skips;
- authenticated supervisor control class: 20 passed;
- focused compiles and `git diff --check`: passed.

## Bounded Discovery

The temporary candidate enabled one-page/one-result search settings for GitLab
and DockerHub and a four-item private custom file for HuggingFace. Dispatch
remained paused. Five successful cycles for each source completed before the
monitor's conservative time limit; no source cycle failed.

Because source cycles do not map directly to queue rows and uniqueness conflicts
consume sequence values, queue high-water deltas were not treated as exact
cohort membership. Eight new queue rows were observed: five GitLab pending and
three DockerHub deferred. No direct queue updates were made.

## Realistic 36-Scan Cohort

The worker processed exactly 36 new remote reservations, IDs 881 through 916,
while discovery remained paused. A fail-closed monitor paused dispatch at the
target and started drain. Final source mix:

| Source | Scans |
|---|---:|
| DockerHub | 11 |
| GitLab | 14 |
| HuggingFace | 11 |
| Total | 36 |

All 36 reservations were remote, resolved, acknowledged, and
`bundle_accepted`. They had 36 distinct queue IDs, bundle IDs, and scan event
IDs, and every reservation had a receipt, payload hash, and execution-snapshot
hash.

### Results

| Source | Result summary |
|---|---|
| DockerHub | 8 clean, 3 degraded |
| GitLab | 12 clean, 1 retryable API error, 1 permanent not-found |
| HuggingFace | 11 clean |

- Queue completion: 34 done, one deferred, one failed; no row remained fenced.
- Findings: zero, a valid outcome for random public targets.
- Errors: exactly two GitLab errors with queue dispositions matching their
  retryable/permanent categories.
- Quarantine: zero new rows.
- Bundle/projection capacity after completion: zero items and zero bytes.
- Existing unrelated keycheck capacity was unchanged.

### Bundle and Projection Invariants

- 36 acknowledged bundles contained 110 frames.
- All bundle identities and counts matched their reservations and scans.
- Acknowledged physical `.trb` files were absent only after both pipeline
  artifact records reached durable `deleted` state, as designed.
- 36 scans used `raw_result_storage=normalized_v2`.
- 36 compatibility rows used expected bounded reconstruction.
- Exactly 36 projection jobs completed, one per scan, without duplicates or
  errors; all projection capacity was released.
- Physical append evidence covered 36 `scan_results` records and two
  `scan_errors` records.
- Every registered append generation/offset/length existed and matched its
  payload SHA-256, record count, and required JSON structure.
- `scan_results.jsonl` grew by exactly 83,752 bytes.
- `found_secrets.jsonl` did not change, matching zero random-target findings.
- All global queue/reservation and orphan invariants remained zero.

Root-only evidence:

- post snapshot:
  `/opt/truf-remote-server/staging/scanner-validation-post-dispatch-newest-v5.json`,
  SHA-256
  `45f9db81e88dbcbe4d6a04094dd1d892df06dd3f7cdd05eb9280c19da54aed94`;
- aggregate report:
  `/opt/truf-remote-server/staging/scanner-validation-cohort-report-v5.json`,
  SHA-256
  `e0826587e4ba667de10a994d5842aae5fae8fa11dc071210c202b38b0e643bf4`.

## Controlled Positive Fixture

Random public targets produced no finding, so a separate one-target run used a
public GitLab project whose README declares that its secret examples are
generated and non-live. No detector or verification behavior was weakened.

- Fixture queue ID: 1,534,100.
- Reservation ID: 917.
- The immutable Git plan bound the approved exact head commit
  `2a09bd6767d39b95cf39ce4b5fd210721275d503`.
- The reservation became acknowledged with `bundle_accepted` and a durable
  receipt.
- Queue completion was `done` with no remaining reservation fence.
- Target scan status was `found` with 116 findings and zero errors.
- All 116 findings used the existing OpenAI detector.
- Verified count was zero, consistent with the unchanged no-verification policy.
- All findings had distinct finding UIDs, nonempty identities, private raw
  material, redaction different from raw material, correct secret hashes, and
  complete non-omitted compatibility payloads.
- No raw finding value was emitted by validation tooling.
- Bundle retirement and both pipeline artifact tombstones were correct.
- The single projection job completed and released capacity.
- The registered `scan_results` region contained one record with exactly 116
  findings and zero errors.
- The registered `found_secrets` region contained exactly 116 records.
- Both physical append regions matched the database payload SHA-256 and were
  byte-identical to fresh PostgreSQL compatibility reconstruction.
- No new quarantine row was created.

Root-only fixture report:

`/opt/truf-remote-server/staging/scanner-validation-fixture-report-v2.json`

SHA-256:

`243a16b24bf9ae898bfdeb8f857c56ef1cf78e12e637730ff5b0674a250a4984`

## Restoration and Final State

The original 36,354 config bytes were passed through managed Preview, saved as
a candidate, and applied through the host agent. Preview preserved the exact
original SHA-256 and reported 169 semantic reversions.

- Restore Save operation:
  `af2a3cc4-90ae-5831-b932-bbe78ceb2cab`.
- Restore Apply operation:
  `b12a2fc4-202a-578e-8dcf-b0a88cb028ef`.
- Apply terminal state: `succeeded/succeeded`, reconciled, category `None`.
- Active and candidate config SHA-256 both equal the original
  `f055a9f2506ab4fffa6953a95c2f6c07b1202e6558f4ce52bbf1b463ed6b1781`.
- Lifecycle preflight and strict Worker API/discovery health passed.
- Runtime and edge were healthy; no failed hold existed.
- All private validation target/evidence files were removed.
- The root-only original backup was retained for audit.

The drained post-restore snapshot is root-only at
`/opt/truf-remote-server/staging/scanner-validation-post-restore-drained-v1.json`,
SHA-256
`de4bd98d728dc551b10712a4afb6be2db0ac9111428d46fa5dbb16f8d2d611ca`.

Final audited control transitions advanced revision 46 to 49 in this order:

1. cancel drain;
2. resume discovery;
3. resume dispatch.

Final state was discovery open, dispatch open, drain `normal`. The existing
worker contacted the server within five minutes and immediately received normal
production work. A live assignment after reopening is expected and is not a
drain blocker because drain is no longer requested.

The final post-resume snapshot had zero orphan/referential invariants and
preserved the original config SHA-256:

`/opt/truf-remote-server/staging/scanner-validation-post-resume-final-v1.json`

SHA-256:

`42e19edae09550693d563b74631430cb1d2c1b807d636ccff20e545cebec3c2d`

External route checks through existing host Caddy returned:

- invalid Worker API authentication: 401;
- unauthenticated protected admin route: 401;
- unrelated path: 404.

Host-agent, Caddy, and X-UI services remained active. Caddy and X-UI were not
lifecycle targets.

## Residual Limits

This validation does not prove:

- long-duration soak or high-concurrency behavior;
- every detector and verification provider;
- every source mode, browser, OS, architecture, or network failure;
- every secrets/config mutation and rotation case;
- HA or multi-server operation;
- resistance to an independent penetration test; or
- correctness of arbitrary unsupported Compose, ingress, or proxy layouts.

Within its declared scope, the real queue, worker, scanner, ingestion,
findings, error, compatibility, restoration, and resumed-production paths all
produced internally consistent durable evidence.