Initial server source import
This commit is contained in:
sashatrask
@@ -0,0 +1,2 @@
|
||||
schema: spec-driven
|
||||
created: 2026-09-30
|
||||
@@ -0,0 +1,70 @@
|
||||
## Context
|
||||
|
||||
Remote admission uses `result_bundle_max_event_bytes` for both the hard upload limit and the initial bundle reservation, and reserves twice that value for projection. With production's 64 MiB hard limit, one unresolved assignment consumes 64 MiB of bundle capacity and 128 MiB of projection capacity. Historical production data shows much smaller payloads, but the hard limit must remain available for outliers.
|
||||
|
||||
The production projector is singleton-owned. This allows one projection job at a time to expand from its baseline reservation into the protected projection headroom without allowing many simultaneous expansions to exhaust the backlog.
|
||||
|
||||
## Goals / Non-Goals
|
||||
|
||||
**Goals:**
|
||||
- Admit up to 50 unresolved remote assignments atomically across all users.
|
||||
- Charge 2 MiB baseline bundle and projection reservations per remote assignment.
|
||||
- Preserve the 64 MiB hard bundle limit and accept larger-than-baseline valid results with capacity backpressure.
|
||||
- Keep all capacity changes durable, replay-safe, and recoverable.
|
||||
|
||||
**Non-Goals:**
|
||||
- Change worker protocol, bundle format, assignment TTLs, or local scanner parallelism.
|
||||
- Preallocate RAM or disk for logical capacity counters.
|
||||
- Remove per-user assignment caps or disk-free safety checks.
|
||||
|
||||
## Decisions
|
||||
|
||||
### Persist a distinct bundle reservation
|
||||
|
||||
Add `reserved_bundle_bytes` to result reservations. `declared_bundle_bytes` remains the immutable hard upload bound; `reserved_bundle_bytes` is the amount charged to `pipeline_capacity.bundle_bytes`. Existing rows are backfilled with their declared value so migration does not change outstanding accounting.
|
||||
|
||||
Alternative considered: derive the reservation from current configuration during release. This is rejected because configuration may change while an assignment is unresolved.
|
||||
|
||||
### Use a 2 MiB baseline for both bundle and projection accounting
|
||||
|
||||
Add `remote_assignment_reserve_bytes`, set to 2 MiB in production. Remote admission charges this value for `reserved_bundle_bytes` and `reserved_projection_bytes`; local reservations retain their existing worst-case behavior.
|
||||
|
||||
Candidate reservations remain bounded by the execution snapshot. Production keycheck item and byte capacity will be sized for 50 such reservations.
|
||||
|
||||
### Expand bundle accounting when actual size is known
|
||||
|
||||
Bundle acceptance atomically grows `reserved_bundle_bytes` to `actual_bytes` before publishing the acceptance receipt. Growth must remain within configured bundle capacity. If capacity is unavailable, acceptance remains unresolved and the durable worker retries the same upload; replay cannot double-charge capacity.
|
||||
|
||||
### Expand projection accounting before append
|
||||
|
||||
After serialization determines the exact aggregate projection bytes, the singleton projector atomically grows the leased job's `capacity_bytes` when required. Growth may use configured projection headroom. If unavailable, the job returns to pending without quarantine or partial append and retries later.
|
||||
|
||||
Alternative considered: treat 2 MiB as a hard projection limit. This is rejected because a valid bundle below the 64 MiB hard limit must not become a deterministic projection failure merely for exceeding its baseline reservation.
|
||||
|
||||
### Enforce a global unresolved-assignment cap
|
||||
|
||||
Add `remote_assignment_max_active`, set to 50. Admission counts unresolved remote reservations while holding the same transactional locks used for per-user quota enforcement. The 51st claim receives normal no-work backpressure. Existing per-user caps still apply, so the effective limit is the minimum of global capacity, global assignment cap, and the user's cap.
|
||||
|
||||
### Production capacity values
|
||||
|
||||
Keep the 64 MiB hard result limit, 512 MiB bundle capacity, 256 MiB projection capacity, and 128 MiB projection headroom. Fifty 2 MiB baselines consume 100 MiB on each byte axis. Increase keycheck capacity to at least 100,000 items and 100 MiB; use 131,072 items and 128 MiB for bounded operational headroom. Set the intended production user's cap to 50.
|
||||
|
||||
## Risks / Trade-offs
|
||||
|
||||
- [Several near-maximum bundles arrive together] → Bundle acceptance atomically expands capacity and later uploads retry without losing their local durable bundle.
|
||||
- [Projection exceeds its baseline] → The singleton projector expands actual capacity before any append and defers without quarantine when headroom is busy.
|
||||
- [Configuration changes with unresolved work] → Every reservation persists charged bundle/projection values; reconciliation sums persisted values rather than current defaults.
|
||||
- [Fifty remote scans overload the small server on result return] → Scanning remains worker-side; API ingestion and projection retain byte, item, disk-free, singleton, and backpressure limits.
|
||||
- [Deployment rollback sees migrated rows] → The additive column is backfilled with prior declared values and remains compatible with restored conservative configuration; rollback code must be from a build that understands the migrated schema.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
1. Add and backfill `reserved_bundle_bytes`, then reconcile `pipeline_capacity` from durable rows.
|
||||
2. Deploy code with the new configuration fields at conservative values.
|
||||
3. Apply production capacity values and set the selected worker user cap to 50 through typed administration.
|
||||
4. Verify 50 baseline admissions in a bounded test, then reconcile all reservations, bundles, projection jobs, and capacity to zero.
|
||||
5. Roll back by setting the global and user caps to the prior value, draining unresolved work, and restoring conservative reservation settings; do not remove the additive column.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None.
|
||||
@@ -0,0 +1,22 @@
|
||||
## Why
|
||||
|
||||
Remote admission currently reserves each assignment at the absolute 64 MiB bundle limit and 128 MiB projection limit. Production results are much smaller (239 KiB maximum bundle and 273 KiB maximum projection across 6,733 observed remote results), so worst-case reservation blocks useful concurrency long before physical resources are under pressure.
|
||||
|
||||
## What Changes
|
||||
|
||||
- Keep the 64 MiB hard limit for an individual result bundle.
|
||||
- Reserve 2 MiB of bundle and projection capacity when issuing a remote assignment, independently of the hard result limit.
|
||||
- Permit a valid result or projection larger than 2 MiB to atomically acquire additional capacity based on its actual size, with retryable backpressure when capacity is temporarily unavailable.
|
||||
- Add an atomic global limit of 50 unresolved remote assignments in addition to existing per-user caps.
|
||||
- Size production pipeline and keycheck accounting so 50 baseline reservations fit without consuming emergency headroom.
|
||||
|
||||
## Capabilities
|
||||
|
||||
### New Capabilities
|
||||
- `remote-assignment-capacity`: Bounded global remote concurrency, separate hard result limits and baseline reservations, and actual-size overflow accounting.
|
||||
|
||||
### Modified Capabilities
|
||||
|
||||
## Impact
|
||||
|
||||
This affects remote assignment admission and result settlement in `worker_assignment.py` and `scanner_db.py`, projection capacity handling in `jsonl_projector.py`, runtime configuration validation, PostgreSQL schema migration, production capacity settings, and integration tests. Worker protocol and bundle format remain unchanged.
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: Hard result limit is separate from baseline reservation
|
||||
The system SHALL retain the configured hard result-bundle byte limit while charging each new remote assignment a separately configured 2 MiB baseline bundle reservation and 2 MiB baseline projection reservation.
|
||||
|
||||
#### Scenario: Remote assignment is issued
|
||||
- **WHEN** an eligible worker claims an assignment with available global and user quota
|
||||
- **THEN** the assignment advertises the unchanged hard bundle limit and pipeline accounting charges only the configured baseline bundle and projection reservations
|
||||
|
||||
#### Scenario: Local assignment is issued
|
||||
- **WHEN** the server admits a local scan
|
||||
- **THEN** its existing worst-case capacity reservation behavior remains unchanged
|
||||
|
||||
### Requirement: Actual bundle size expands accounting safely
|
||||
The system SHALL atomically grow a remote reservation's charged bundle bytes to the validated actual bundle size before issuing an acceptance receipt.
|
||||
|
||||
#### Scenario: Bundle exceeds baseline with capacity available
|
||||
- **WHEN** a valid bundle is larger than 2 MiB but does not exceed the hard result limit and aggregate bundle capacity is available
|
||||
- **THEN** the system increases the persisted reservation and pipeline capacity charge exactly once and accepts the bundle
|
||||
|
||||
#### Scenario: Bundle exceeds baseline without capacity available
|
||||
- **WHEN** a valid bundle requires additional bundle capacity that is temporarily unavailable
|
||||
- **THEN** the system does not issue an acceptance receipt and permits the worker to retry the identical durable upload later
|
||||
|
||||
### Requirement: Actual projection size expands accounting safely
|
||||
The system SHALL grow a leased projection job's capacity to its serialized aggregate byte size before appending output.
|
||||
|
||||
#### Scenario: Projection exceeds baseline with headroom available
|
||||
- **WHEN** serialization exceeds the baseline projection reservation and configured projection capacity is available
|
||||
- **THEN** the system atomically increases the job and pipeline charge before appending output
|
||||
|
||||
#### Scenario: Projection exceeds baseline without headroom available
|
||||
- **WHEN** the additional projection capacity is temporarily unavailable
|
||||
- **THEN** the system returns the untouched job to pending without quarantine and retries it later
|
||||
|
||||
### Requirement: Global remote assignment limit
|
||||
The system SHALL enforce a configured global maximum of 50 unresolved remote assignments atomically in addition to each user's assignment cap.
|
||||
|
||||
#### Scenario: Fiftieth assignment is admitted
|
||||
- **WHEN** 49 unresolved remote assignments exist and all other admission constraints are open
|
||||
- **THEN** one additional assignment is admitted
|
||||
|
||||
#### Scenario: Fifty-first assignment is refused
|
||||
- **WHEN** 50 unresolved remote assignments exist
|
||||
- **THEN** another claim receives normal no-work backpressure without creating a reservation
|
||||
|
||||
#### Scenario: Assignment resolves
|
||||
- **WHEN** an unresolved assignment receives a terminal resolution or expires
|
||||
- **THEN** its global slot becomes available for a subsequent claim
|
||||
|
||||
### Requirement: Capacity survives migration and reconciliation
|
||||
The system MUST preserve exact capacity accounting for reservations created before and after deployment.
|
||||
|
||||
#### Scenario: Existing reservation is migrated
|
||||
- **WHEN** the additive schema migration encounters a reservation without a distinct bundle reservation value
|
||||
- **THEN** it backfills the value from the existing declared bundle bytes
|
||||
|
||||
#### Scenario: Capacity is reconciled
|
||||
- **WHEN** pipeline capacity is rebuilt from durable state
|
||||
- **THEN** it sums each reservation's persisted bundle, projection, candidate, job, and quarantine charges exactly once
|
||||
@@ -0,0 +1,23 @@
|
||||
## 1. Capacity Model
|
||||
|
||||
- [x] 1.1 Add configuration validation for the 2 MiB remote baseline reservation and global active-assignment limit
|
||||
- [x] 1.2 Add and backfill persisted remote bundle reservation bytes in the runtime-safety schema
|
||||
- [x] 1.3 Charge persisted baseline bundle and projection bytes during remote admission while preserving local admission behavior
|
||||
- [x] 1.4 Enforce the global unresolved remote-assignment limit atomically with existing per-user quota
|
||||
|
||||
## 2. Actual-Size Expansion
|
||||
|
||||
- [x] 2.1 Expand bundle capacity atomically to validated actual bytes before remote acceptance
|
||||
- [x] 2.2 Expand projection-job capacity atomically before append and defer without quarantine when capacity is unavailable
|
||||
- [x] 2.3 Update refunds, cleanup, quarantine, and reconciliation to use persisted charged bytes
|
||||
|
||||
## 3. Production Configuration
|
||||
|
||||
- [x] 3.1 Configure a 2 MiB baseline, global limit 50, and keycheck capacity for 50 assignments
|
||||
- [x] 3.2 Update operator documentation with the distinct hard-limit, baseline-reservation, and backpressure semantics
|
||||
|
||||
## 4. Verification
|
||||
|
||||
- [x] 4.1 Add unit and PostgreSQL integration coverage for baseline admission, global quota, bundle expansion, projection expansion, retries, migration, and reconciliation
|
||||
- [x] 4.2 Run targeted worker, pipeline, migration, and packaged-worker tests
|
||||
- [x] 4.3 Validate a bounded 50-assignment production candidate and reconcile capacity and pipeline debt before rollout
|
||||
@@ -0,0 +1,27 @@
|
||||
## Production Validation
|
||||
|
||||
Validated and deployed on `sec` on 2026-09-30 through the fail-closed
|
||||
`deploy_capacity50.ps1` workflow.
|
||||
|
||||
- Baseline runtime image: `sha256:46f1cf1b92d1a7d93d06f690309d8c7eca45f64dc45ca310861c70fb419bb035`.
|
||||
- Deployed runtime image: `sha256:17b21c62f559220cd60e7972826fb439e5538c7c9443c4fdd1c3615751de5f8d`.
|
||||
- Active config SHA-256: `2cf2ac66b58f1b39c1e45dd0d562b9b6375be42fa2289cc9bbcbd1a310951ee7`.
|
||||
- Rollback tag: `truf-local:runtime-pre-capacity50-20260930`.
|
||||
- The additive `20260930_33_remote_assignment_capacity` migration and
|
||||
`reserved_bundle_bytes` backfill completed with zero null rows.
|
||||
- Runtime health was `ACTIVE`/`READY`; runtime and edge had zero restarts and no OOM.
|
||||
- Runtime control returned to `normal`; all bundle, projection, keycheck,
|
||||
quarantine, and unresolved-assignment counters were zero.
|
||||
- The 64 MiB hard limit, 2 MiB baseline, global cap 50, 131072 keycheck items,
|
||||
and 128 MiB keycheck bytes were active.
|
||||
- A production test user's cap was changed from 2 to 50 through typed admin,
|
||||
verified, and restored to 2. No test user retains the expanded cap.
|
||||
- The active Windows worker contacted the new Worker API after rollout.
|
||||
- The exact 50th-admitted/51st-refused behavior remains covered by the real
|
||||
PostgreSQL integration test; production was not flooded with synthetic work.
|
||||
|
||||
The live run also exercised rollback before cutover and after a rejected
|
||||
migration precondition. Both restored the original image/config, healthy runtime
|
||||
and edge, normal control state, and zero pipeline debt. The final workflow now
|
||||
quiesces singleton pipeline workers and releases their exact fenced leases before
|
||||
the schema migration.
|
||||
Reference in New Issue
Block a user