sashatrask
71 lines
5.5 KiB
Markdown
71 lines
5.5 KiB
Markdown
## Context
|
|
|
|
Remote admission uses `result_bundle_max_event_bytes` for both the hard upload limit and the initial bundle reservation, and reserves twice that value for projection. With production's 64 MiB hard limit, one unresolved assignment consumes 64 MiB of bundle capacity and 128 MiB of projection capacity. Historical production data shows much smaller payloads, but the hard limit must remain available for outliers.
|
|
|
|
The production projector is singleton-owned. This allows one projection job at a time to expand from its baseline reservation into the protected projection headroom without allowing many simultaneous expansions to exhaust the backlog.
|
|
|
|
## Goals / Non-Goals
|
|
|
|
**Goals:**
|
|
- Admit up to 50 unresolved remote assignments atomically across all users.
|
|
- Charge 2 MiB baseline bundle and projection reservations per remote assignment.
|
|
- Preserve the 64 MiB hard bundle limit and accept larger-than-baseline valid results with capacity backpressure.
|
|
- Keep all capacity changes durable, replay-safe, and recoverable.
|
|
|
|
**Non-Goals:**
|
|
- Change worker protocol, bundle format, assignment TTLs, or local scanner parallelism.
|
|
- Preallocate RAM or disk for logical capacity counters.
|
|
- Remove per-user assignment caps or disk-free safety checks.
|
|
|
|
## Decisions
|
|
|
|
### Persist a distinct bundle reservation
|
|
|
|
Add `reserved_bundle_bytes` to result reservations. `declared_bundle_bytes` remains the immutable hard upload bound; `reserved_bundle_bytes` is the amount charged to `pipeline_capacity.bundle_bytes`. Existing rows are backfilled with their declared value so migration does not change outstanding accounting.
|
|
|
|
Alternative considered: derive the reservation from current configuration during release. This is rejected because configuration may change while an assignment is unresolved.
|
|
|
|
### Use a 2 MiB baseline for both bundle and projection accounting
|
|
|
|
Add `remote_assignment_reserve_bytes`, set to 2 MiB in production. Remote admission charges this value for `reserved_bundle_bytes` and `reserved_projection_bytes`; local reservations retain their existing worst-case behavior.
|
|
|
|
Candidate reservations remain bounded by the execution snapshot. Production keycheck item and byte capacity will be sized for 50 such reservations.
|
|
|
|
### Expand bundle accounting when actual size is known
|
|
|
|
Bundle acceptance atomically grows `reserved_bundle_bytes` to `actual_bytes` before publishing the acceptance receipt. Growth must remain within configured bundle capacity. If capacity is unavailable, acceptance remains unresolved and the durable worker retries the same upload; replay cannot double-charge capacity.
|
|
|
|
### Expand projection accounting before append
|
|
|
|
After serialization determines the exact aggregate projection bytes, the singleton projector atomically grows the leased job's `capacity_bytes` when required. Growth may use configured projection headroom. If unavailable, the job returns to pending without quarantine or partial append and retries later.
|
|
|
|
Alternative considered: treat 2 MiB as a hard projection limit. This is rejected because a valid bundle below the 64 MiB hard limit must not become a deterministic projection failure merely for exceeding its baseline reservation.
|
|
|
|
### Enforce a global unresolved-assignment cap
|
|
|
|
Add `remote_assignment_max_active`, set to 50. Admission counts unresolved remote reservations while holding the same transactional locks used for per-user quota enforcement. The 51st claim receives normal no-work backpressure. Existing per-user caps still apply, so the effective limit is the minimum of global capacity, global assignment cap, and the user's cap.
|
|
|
|
### Production capacity values
|
|
|
|
Keep the 64 MiB hard result limit, 512 MiB bundle capacity, 256 MiB projection capacity, and 128 MiB projection headroom. Fifty 2 MiB baselines consume 100 MiB on each byte axis. Increase keycheck capacity to at least 100,000 items and 100 MiB; use 131,072 items and 128 MiB for bounded operational headroom. Set the intended production user's cap to 50.
|
|
|
|
## Risks / Trade-offs
|
|
|
|
- [Several near-maximum bundles arrive together] → Bundle acceptance atomically expands capacity and later uploads retry without losing their local durable bundle.
|
|
- [Projection exceeds its baseline] → The singleton projector expands actual capacity before any append and defers without quarantine when headroom is busy.
|
|
- [Configuration changes with unresolved work] → Every reservation persists charged bundle/projection values; reconciliation sums persisted values rather than current defaults.
|
|
- [Fifty remote scans overload the small server on result return] → Scanning remains worker-side; API ingestion and projection retain byte, item, disk-free, singleton, and backpressure limits.
|
|
- [Deployment rollback sees migrated rows] → The additive column is backfilled with prior declared values and remains compatible with restored conservative configuration; rollback code must be from a build that understands the migrated schema.
|
|
|
|
## Migration Plan
|
|
|
|
1. Add and backfill `reserved_bundle_bytes`, then reconcile `pipeline_capacity` from durable rows.
|
|
2. Deploy code with the new configuration fields at conservative values.
|
|
3. Apply production capacity values and set the selected worker user cap to 50 through typed administration.
|
|
4. Verify 50 baseline admissions in a bounded test, then reconcile all reservations, bundles, projection jobs, and capacity to zero.
|
|
5. Roll back by setting the global and user caps to the prior value, draining unresolved work, and restoring conservative reservation settings; do not remove the additive column.
|
|
|
|
## Open Questions
|
|
|
|
None.
|