Initial server source import
This commit is contained in:
sashatrask
@@ -0,0 +1,473 @@
|
||||
# Worker Parallelism Validation, 2026-09-23
|
||||
|
||||
## Scope
|
||||
|
||||
This report records the production validation performed on `sec` using the local
|
||||
Windows computer:
|
||||
|
||||
- bounded discovery for GitLab, DockerHub, and HuggingFace;
|
||||
- a native Windows protocol-2 worker with client and server parallelism 3;
|
||||
- a dual-worker run with native Windows parallelism 1 and the existing trusted
|
||||
WSL production worker at cap 1;
|
||||
- authoritative reconciliation from discovery through queue, reservation,
|
||||
receipt, bundle ingestion, scan settlement, projection, and JSONL append;
|
||||
- restoration of production source/capacity settings and normal controls.
|
||||
|
||||
No provider credential, device token, raw target, raw finding, runtime YAML, or
|
||||
worker command line is included in this report.
|
||||
|
||||
## Result
|
||||
|
||||
The core validation passed.
|
||||
|
||||
- Native Windows reached exact unfinished concurrency 3 and never exceeded 3.
|
||||
- The dedicated Windows cohort accounted for 180 accepted assignments across all
|
||||
three sources.
|
||||
- Windows and WSL independently reached concurrency 1 at the same time, for exact
|
||||
combined concurrency 2, and never exceeded their individual caps.
|
||||
- All accepted results were ingested, queue-settled, projected, and appended as
|
||||
required.
|
||||
- Final cohort and global lineage checks reported zero violations.
|
||||
- No pipeline quarantine or failed hold was created.
|
||||
- Production controls and the WSL worker were restored.
|
||||
|
||||
The originally requested 10-hour observation was not completed continuously.
|
||||
After successful active parallelism-3 work, 14,105 seconds (about 3 hours 55
|
||||
minutes) of detached idle stability observation completed before the user changed
|
||||
the objective to the dual-worker test. The active concurrency evidence itself is
|
||||
complete; the residual limit is soak duration, not functional coverage.
|
||||
|
||||
## Baseline
|
||||
|
||||
The fresh pre-test snapshot was stored root-only on `sec`:
|
||||
|
||||
- Path: `/opt/truf-remote-server/staging/windows-p3-before2.json`
|
||||
- SHA-256: `469bcd69888cacce55523fef17508ee48191c4c879a6d39f2251abad4aa0efdb`
|
||||
- Active config SHA-256:
|
||||
`e48797689ab0ee7328d21cf5c23d0ffedb979dba492bf49d1da3afe4b075cf5b`
|
||||
- Config bytes: 37,233
|
||||
- Controls: revision 56, discovery open, dispatch open, drain normal
|
||||
- Existing blocker: one stale production assignment, allowed to settle naturally
|
||||
- Runtime image:
|
||||
`sha256:3b4d6e19e29e85a32b75d64265d75e71100d3f7f00221d21de544256dadd25cf`
|
||||
- Failed hold: absent
|
||||
- All recorded orphan/mismatch invariants: zero
|
||||
|
||||
Baseline high-water values included reservation 929, scan 926, projection job
|
||||
926, projection append 984, error 1919, finding 127, and zero quarantine rows.
|
||||
|
||||
The existing WSL assignment was never canceled or directly mutated. Dispatch was
|
||||
paused and the lease was allowed to resolve before changing worker caps.
|
||||
|
||||
## Windows Package
|
||||
|
||||
The trusted run artifact was built from pinned local caches:
|
||||
|
||||
- Package directory:
|
||||
`build/worker-parallelism3-20260922/package-fixed1`
|
||||
- Archive:
|
||||
`build/worker-parallelism3-20260922/truf-worker-windows-x86_64-fixed1.zip`
|
||||
- Archive SHA-256:
|
||||
`6ca460a47d31606c430e8c6f081443530892332cfc183fc0c6edfe7568a454cc`
|
||||
- Manifest file SHA-256:
|
||||
`cd55820e0a04a04ded9ba340b9c6e5c6259422aab80989180a6ba28ff3b77103`
|
||||
- Canonical code-manifest SHA-256:
|
||||
`e981da19283aad5971fc2865268bb8151b3540d71fbd057ae56d9d2379993374`
|
||||
- Platform: `windows-x86_64`
|
||||
- Protocol and bundle schema: v2
|
||||
- Capabilities: GitLab `exact_git_v1`, DockerHub `docker_direct_v1`,
|
||||
HuggingFace `huggingface_space_v1`
|
||||
|
||||
The registered Windows v2 manifest was stale relative to the current production
|
||||
authority files. It was not overwritten. The fixed package was registered as the
|
||||
new root-owned `windows-worker-package-v3.json` trust authority.
|
||||
|
||||
### Reproducibility defect
|
||||
|
||||
Two initial package builds differed only in four timestamp bytes in each
|
||||
pip-generated Windows launcher executable and the corresponding RECORD hashes.
|
||||
The package builder now normalizes the embedded launcher ZIP timestamps to the
|
||||
DOS epoch and recomputes the RECORD rows.
|
||||
|
||||
Changed local source:
|
||||
|
||||
- `app/worker_package_builder.py`
|
||||
- `tests/test_worker_package.py`
|
||||
|
||||
Verification:
|
||||
|
||||
- Two post-fix archives were byte-identical.
|
||||
- `python -m pytest tests/test_worker_package.py -q`: 16 passed.
|
||||
|
||||
## Discovery
|
||||
|
||||
The temporary discovery candidate was derived from the exact active config.
|
||||
|
||||
GitLab used 16 reviewed keyword queries, one API page, one result per page, and
|
||||
one target maximum per query. DockerHub used 16 reviewed keyword queries, one
|
||||
repository result per query, and one image per repository.
|
||||
|
||||
HuggingFace was deliberately reported differently: the configured implementation
|
||||
does not perform keyword search. It requests the newest-modified Spaces with a
|
||||
fixed API limit of 100. `pages: 1` therefore means one real recent page, not one
|
||||
keyword result. Repeated polls occurred while GitLab and DockerHub rotated their
|
||||
queries; deduplication bounded the admitted rows.
|
||||
|
||||
Final bounded discovery evidence:
|
||||
|
||||
- GitLab: 16/16 distinct successful query rotations, 7 new queue rows
|
||||
- DockerHub: 16/16 distinct successful query rotations, 10 new queue rows
|
||||
- HuggingFace: 17 successful recent-page cycles, 31 new queue rows
|
||||
- Discovery failures: 0
|
||||
- Hard limits: GitLab 20, DockerHub 20, HuggingFace 120; none exceeded
|
||||
|
||||
## Native Windows Parallelism 3
|
||||
|
||||
An isolated temporary server user and device were created with typed, audited
|
||||
`AdminService` operations. The one-time token was transferred privately, the
|
||||
server transfer file was deleted after successful launch, and the native worker
|
||||
used a dedicated protected `LOCALAPPDATA` state tree.
|
||||
|
||||
The first foreground launch exposed a local automation issue: the tool runner
|
||||
retained the descendant process and did not return. The worker itself survived.
|
||||
Subsequent starts used `Win32_Process.Create` so the worker was genuinely
|
||||
detached. No process command line was inspected.
|
||||
|
||||
### Capacity findings
|
||||
|
||||
Client `--parallelism 3` and server user cap 3 were not by themselves enough to
|
||||
permit three physical scans.
|
||||
|
||||
The first run reached only one active reservation because the production config
|
||||
had:
|
||||
|
||||
- `global.max_active_scans = 1`
|
||||
- 64 MiB maximum event size
|
||||
- 256 MiB projection backlog maximum
|
||||
- 128 MiB projection headroom
|
||||
|
||||
The capacity model reserves twice the event maximum per assignment, plus
|
||||
headroom. Three slots therefore require 512 MiB. The temporary test candidate
|
||||
changed only:
|
||||
|
||||
- `max_active_scans: 1 -> 3`
|
||||
- `projection_backlog_max_bytes: 256 MiB -> 512 MiB`
|
||||
|
||||
A second run still reached only one assignment. Admission evidence showed
|
||||
`pipeline_capacity_closed` on the keycheck axis. The stable keycheck backlog was
|
||||
127 items and each assignment reserved up to 2,000 items. The production limit
|
||||
of 4,096 allowed one reservation but not three. The temporary candidate changed:
|
||||
|
||||
- `keycheck_queue_max_items: 4096 -> 8192`
|
||||
|
||||
No unrelated limit was increased.
|
||||
|
||||
### Successful run
|
||||
|
||||
After both capacity corrections:
|
||||
|
||||
- Exact max unfinished concurrency: 3
|
||||
- Never exceeded: 3
|
||||
- Issued: 180
|
||||
- Accepted: 180
|
||||
- Prebundle failures: 0
|
||||
- Expired: 0
|
||||
- Unfinished after settlement: 0
|
||||
- Accepted-not-ingested: 0
|
||||
- Queue-not-settled: 0
|
||||
- Projection-not-completed: 0
|
||||
|
||||
Source totals:
|
||||
|
||||
| Source | Accepted |
|
||||
|---|---:|
|
||||
| DockerHub | 63 |
|
||||
| GitLab | 65 |
|
||||
| HuggingFace | 52 |
|
||||
|
||||
Scan outcomes were transport-successful but not necessarily target-clean:
|
||||
|
||||
| Source | Outcome | Count |
|
||||
|---|---|---:|
|
||||
| DockerHub | clean | 44 |
|
||||
| DockerHub | degraded | 18 |
|
||||
| DockerHub | error | 1 |
|
||||
| GitLab | clean | 57 |
|
||||
| GitLab | error | 8 |
|
||||
| HuggingFace | clean | 43 |
|
||||
| HuggingFace | error | 9 |
|
||||
|
||||
These scan statuses are provider/target results. They are not lost transport,
|
||||
ingestion, or projection records. All 180 authoritative results settled.
|
||||
|
||||
## Stability Observation
|
||||
|
||||
After the 180-assignment cap closed dispatch for the test identity, the worker
|
||||
remained online at cap 0 under detached monitoring.
|
||||
|
||||
- Successful checks: 43
|
||||
- Elapsed observation: 14,105 seconds
|
||||
- Runtime restarts: 0
|
||||
- Edge restarts: 0
|
||||
- Native worker OOM/restart: none
|
||||
- Pipeline debt: 0
|
||||
- Quarantine: 0
|
||||
- Global invariants: 0
|
||||
|
||||
A single strict-health probe can observe a discovery producer during its normal
|
||||
short startup window. Monitors were corrected to require failure across three
|
||||
attempts with 20-second delays rather than treating one transient sample as a
|
||||
production defect.
|
||||
|
||||
## Dual Worker Validation
|
||||
|
||||
The user then requested a second topology:
|
||||
|
||||
- native Windows worker: real client parallelism 1 and server cap 1;
|
||||
- existing trusted production WSL worker: cap 1;
|
||||
- desired combined concurrency: 2.
|
||||
|
||||
The native worker was cleanly stopped at cap 0, relaunched from the same trusted
|
||||
package and protected state with parallelism 1, and verified by PID/resource
|
||||
evidence. The production WSL token and command line were never inspected.
|
||||
|
||||
### Monitor corrections
|
||||
|
||||
Several fail-closed attempts improved the monitoring policy without losing or
|
||||
canceling work:
|
||||
|
||||
- A scanner subprocess can remain active without an HTTP contact until it
|
||||
reports. Contact age alone must not fail an identity with unfinished work.
|
||||
- A worker may honor a prior cap-0 `Retry-After` after caps change. Arming uses a
|
||||
separate 600-second threshold; the 180-second idle threshold applies only
|
||||
after dispatch has opened.
|
||||
- Operation IDs are one-use and identity-bound. Every retry used a fresh audited
|
||||
operation namespace.
|
||||
- A transient PostgreSQL connection timeout while reopening a read-only monitor
|
||||
connection terminated observability after caps and gates were already safely
|
||||
closed. The monitor now retries database opens three times with 20-second
|
||||
delays. A separate read-only settlement monitor completed the active run.
|
||||
|
||||
### Attempt 1 evidence
|
||||
|
||||
The first active dual run already proved max concurrency Windows 1, WSL 1,
|
||||
combined 2. It fail-closed because of the old contact policy while WSL was doing
|
||||
a long DockerHub scan.
|
||||
|
||||
- Windows: 19 accepted assignments
|
||||
- WSL: 2 accepted HuggingFace assignments
|
||||
- WSL: 1 DockerHub reservation naturally expired after its two-hour lease
|
||||
- No assignment was canceled
|
||||
|
||||
### Final attempt 4
|
||||
|
||||
The final bounded run used baseline reservation ID 1131.
|
||||
|
||||
- Issued: 30
|
||||
- Windows issued/accepted: 29/29
|
||||
- WSL issued: 1
|
||||
- WSL accepted: 0
|
||||
- WSL expired/refunded: 1 long DockerHub assignment after its two-hour lease
|
||||
- Max Windows concurrency: 1
|
||||
- Max WSL concurrency: 1
|
||||
- Max combined concurrency: 2
|
||||
- Accepted-not-ingested: 0
|
||||
- Queue-not-settled: 0
|
||||
- Projection-not-completed: 0
|
||||
- Cohort integrity violations: 0
|
||||
- Global invariant violations: 0
|
||||
- Failed holds in the cohort window: 0
|
||||
- Quarantine bytes/items: 0/0
|
||||
|
||||
Windows source/outcome totals in the final run:
|
||||
|
||||
| Source | Outcome | Count |
|
||||
|---|---|---:|
|
||||
| DockerHub | clean | 12 |
|
||||
| GitLab | clean | 7 |
|
||||
| HuggingFace | clean | 9 |
|
||||
| HuggingFace | error | 1 |
|
||||
|
||||
The WSL expiry does not invalidate the concurrency result: both independent
|
||||
clients simultaneously held one real reservation and never exceeded cap. Earlier
|
||||
in attempt 1, the same WSL path also completed two accepted HuggingFace results.
|
||||
|
||||
Authoritative dual evidence:
|
||||
|
||||
- `/opt/truf-remote-server/staging/windows-dual-worker-report.json`
|
||||
- SHA-256:
|
||||
`8fd03bf79960393406ef1a0f788d90db3ed3521d32ffa83f802c9d1c2c3509bf`
|
||||
|
||||
## Data Reconciliation
|
||||
|
||||
The validation distinguished assignment acceptance, bundle ingestion, queue
|
||||
settlement, and projection completion.
|
||||
|
||||
Checks covered:
|
||||
|
||||
- accepted reservation has a durable receipt;
|
||||
- accepted reservation has exactly one matching bundle;
|
||||
- reservation/bundle/scan event IDs and hashes agree;
|
||||
- target scan references the same queue row;
|
||||
- queue completion is `applied`;
|
||||
- projection job exists, matches the event identity, and is completed;
|
||||
- required `scan_results` append exists;
|
||||
- required `found_secrets` append exists when requested by the mask;
|
||||
- projection append event identity matches the job;
|
||||
- no orphan bundle, scan, finding, error, or projection job exists;
|
||||
- JSONL files are present, newline-terminated, and cursor offsets match.
|
||||
|
||||
All cohort and global mismatch counters were zero at the final report points.
|
||||
|
||||
## Code Defects Fixed
|
||||
|
||||
### Integer cap zero
|
||||
|
||||
`AdminService._validate_cap()` used `str(value or '')`, converting integer zero
|
||||
to an empty string even though cap 0 is valid. It now distinguishes `None` from
|
||||
zero.
|
||||
|
||||
Changed local source:
|
||||
|
||||
- `app/admin_api.py`
|
||||
- `tests/test_admin_api.py`
|
||||
|
||||
Verification:
|
||||
|
||||
- `python -m pytest tests/test_admin_api.py -q`: 62 passed.
|
||||
|
||||
The currently deployed runtime image predates this source fix. Run helpers used
|
||||
canonical string `"0"` through the same typed/audited API; no direct database
|
||||
mutation was used.
|
||||
|
||||
### Operational helpers
|
||||
|
||||
Run-specific helpers were added under `build/sec-deploy/` for config lifecycle,
|
||||
controls, typed worker administration, monitoring, observation, local launch,
|
||||
safe stop, settlement, and reporting. Python helpers passed `py_compile` and
|
||||
Ruff; PowerShell helpers passed parser validation before use.
|
||||
|
||||
## Configuration Restoration
|
||||
|
||||
Temporary active config identities were:
|
||||
|
||||
| Stage | SHA-256 |
|
||||
|---|---|
|
||||
| Bounded discovery + v3 profile | `ec2ae20f9a23ee9ec423c28f7da0b45db3eaff17abb3f484e1a1f0c9227fada4` |
|
||||
| Parallel scan/projection capacity | `4fee05f3a9ae001767f2de0388dcd5db75183dd954be8e024e98f47d4b64b6b1` |
|
||||
| Parallel keycheck capacity | `b3509aacc53ab0bfa1e6b13dd789f889d9fea34daf0f680579d2a54697393bf9` |
|
||||
| Restored production semantics + v3 trust | `c0966cac4f7f0610a813fa8732e91953f2e3e838ad880f91fd1a9437096925c7` |
|
||||
|
||||
The final config restores the original production discovery/source settings,
|
||||
global scan capacity, projection capacity, keycheck capacity, and supervisor
|
||||
interval. The only intentional permanent semantic difference from the initial
|
||||
config is the Windows compatibility profile path changing from stale v2 to the
|
||||
reproducible/current v3 manifest. Therefore the final config hash intentionally
|
||||
differs from the initial hash while retaining the same 37,233-byte size.
|
||||
|
||||
All four managed config apply operations succeeded and reconciled. The final
|
||||
apply operation was:
|
||||
|
||||
- `a9943967-372e-522a-8386-4f9465cc03f9`
|
||||
|
||||
The complete root-only operation export contains 69 operations for the run
|
||||
actor, all with status `succeeded`:
|
||||
|
||||
- `/opt/truf-remote-server/staging/windows-p3-operations.json`
|
||||
- SHA-256:
|
||||
`318aaa56cc2ba348e905b72fd63d7795e24cbd1aa9e84d0a75fa341744ef0acf`
|
||||
|
||||
## Final Production State
|
||||
|
||||
The temporary native process was stopped only after its local bundle/work trees
|
||||
and authoritative pipeline were empty.
|
||||
|
||||
Typed cleanup completed:
|
||||
|
||||
- temporary Windows cap: 0
|
||||
- temporary device: revoked
|
||||
- temporary user: disabled
|
||||
- server plaintext token transfer: absent
|
||||
- production WSL cap: 1
|
||||
|
||||
Controls after restoration:
|
||||
|
||||
- revision: 98
|
||||
- discovery: open
|
||||
- dispatch: open
|
||||
- drain: normal
|
||||
- effective gates: open
|
||||
|
||||
The production WSL container was idle but still honoring a previous cap-0 retry
|
||||
delay. With zero unfinished work it was safely restarted using the same container,
|
||||
token, and state. It immediately made a fresh server contact and received a new
|
||||
production assignment, proving restored admission.
|
||||
|
||||
At the final snapshot:
|
||||
|
||||
- production WSL contact was fresh and one real production assignment was active;
|
||||
- blocker count 1 was therefore expected, not a drain defect;
|
||||
- WSL container running, OOM false, restart count 0;
|
||||
- runtime healthy, restart count 0;
|
||||
- edge running, restart count 0;
|
||||
- host-agent, host Caddy, and X-UI active;
|
||||
- strict runtime health: ACTIVE/READY with DockerHub, GitLab, HuggingFace,
|
||||
janitor, JSONL projector, result ingester, and Worker API;
|
||||
- failed hold absent.
|
||||
|
||||
Runtime image remained:
|
||||
|
||||
`sha256:3b4d6e19e29e85a32b75d64265d75e71100d3f7f00221d21de544256dadd25cf`
|
||||
|
||||
## Final Snapshot
|
||||
|
||||
- Path: `/opt/truf-remote-server/staging/windows-p3-final.json`
|
||||
- SHA-256: `1e3ab6bdb50c5be1416974bf56235c220166a6b731e76a87d8abebb947aae074`
|
||||
- Bytes: 6,883
|
||||
- Config SHA-256:
|
||||
`c0966cac4f7f0610a813fa8732e91953f2e3e838ad880f91fd1a9437096925c7`
|
||||
- Reservation high-water: 1165
|
||||
- Scan high-water: 1159
|
||||
- Projection job high-water: 1159
|
||||
- Projection append high-water: 1239
|
||||
- Error high-water: 1945
|
||||
- Finding high-water: 127
|
||||
- Quarantine rows: 0
|
||||
- All six recorded global orphan/mismatch invariants: 0
|
||||
- `scan_results.jsonl`: 4,592,879 bytes, cursor exact, newline-terminated
|
||||
- `found_secrets.jsonl`: 146,366 bytes, cursor exact,
|
||||
newline-terminated
|
||||
|
||||
Final strict-health evidence:
|
||||
|
||||
- `/opt/truf-remote-server/staging/windows-p3-final-health.json`
|
||||
- SHA-256:
|
||||
`68c2b0dcd16591b2407fefcb5c3921aa7f58443011539ff912e2cb19b9fdef13`
|
||||
|
||||
## Retained Local Evidence
|
||||
|
||||
Per the user's explicit instruction on 2026-09-23, no additional local files were
|
||||
deleted during final reporting.
|
||||
|
||||
The following remain intentionally retained:
|
||||
|
||||
- protected local one-time token file;
|
||||
- dedicated native worker state directory;
|
||||
- worker stdout/stderr and PID evidence;
|
||||
- fixed package and both reproducibility builds;
|
||||
- local run-specific helpers and reports.
|
||||
|
||||
The token is no longer usable because the server device is revoked and user is
|
||||
disabled, but the local protected file remains until the user authorizes deletion.
|
||||
|
||||
## Residual Limits
|
||||
|
||||
- The continuous 10-hour soak was shortened by the later dual-worker request.
|
||||
- HuggingFace configured discovery is recent-page discovery, not keyword search.
|
||||
- Long DockerHub work twice reached the fixed two-hour WSL lease and expired;
|
||||
expiry/refund behavior was correct, but those targets did not produce bundles.
|
||||
- The local source fixes for package reproducibility and integer cap zero are
|
||||
tested in this checkout but were not deployed as a new production runtime
|
||||
image during this validation.
|
||||
- Local sensitive/state evidence is deliberately retained and requires a later
|
||||
explicit cleanup instruction.
|
||||
Reference in New Issue
Block a user