Initial server source import

2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
@@ -0,0 +1,63 @@
+## Context
+
+GitLab repository scans use the shared TruffleHog Git command under the project's external supervisor, scan-slot leases, timeout enforcement, and Windows Job containment. Unlike Docker, Git commands still launch TruffleHog's embedded overseer. Twelve recent GitLab scans emitted no fatal diagnostic and no `finished scanning` marker, exited code 1, and were committed as non-retryable terminal failures after one attempt.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Give the existing external runtime sole lifecycle ownership for GitLab TruffleHog processes.
+- Require positive completion evidence for GitLab process success.
+- Retry incomplete GitLab scans through the existing three-attempt target policy.
+- Preserve partial findings and isolate behavior from other Git sources.
+- Validate with a GitLab-only canary and bounded exact-signature replay.
+
+**Non-Goals:**
+
+- Change GitLab discovery API behavior.
+- Change GitHub, package Git, or generic ad-hoc Git commands.
+- Upgrade TruffleHog or change detector selection.
+- Make clone-unavailable repositories retry forever.
+
+## Decisions
+
+### Enable local process mode only for configured GitLab scans
+
+`scan_git_repo` will accept an explicit lifecycle flag. GitLab source configuration enables it, causing the command to add `--local-dev`; callers that do not opt in retain the current command.
+
+Alternative considered: enable `--local-dev` for every Git source. Rejected because current production evidence is GitLab-specific and a narrow canary is easier to attribute and roll back.
+
+### Generalize explicit completion without changing existing defaults
+
+The diagnostic parser will accept an explicit completion-required option while preserving Docker as completion-required by default. GitLab local-process scans will enable that option. Missing completion or unexplained nonzero exit becomes retryable `command_incomplete`; a nonzero unexplained exit after completion becomes retryable `wrapper_exit`.
+
+Alternative considered: classify every Git RC=1 as retryable. Rejected because completion evidence distinguishes lifecycle interruption from explicit permanent diagnostics.
+
+### Reuse target retries and finding identity
+
+The existing maximum of three attempts, exponential delay, queue dispositions, finding UID assignment, and durable deduplication remain authoritative. Findings emitted before interruption are retained without marking the target complete.
+
+### Stage rollout and replay
+
+A controlled A/B run must first show that Git `--local-dev` completes a previously affected public target. Production then observes at least 100 GitLab attempts or two hours before a small exact-signature historical replay.
+
+## Risks / Trade-offs
+
+- [The flag behaves differently for Git than Docker] -> Require controlled A/B evidence and a GitLab-only canary.
+- [Retries increase clone traffic] -> Keep the existing attempt cap and replay only exact-signature targets in small batches.
+- [Completion logging changes upstream] -> Treat missing evidence as bounded incomplete coverage rather than success.
+- [Shared Git scanner plumbing leaks behavior] -> Use an explicit source-configured flag and assert non-GitLab command isolation.
+- [Partial findings repeat] -> Retain scan events while relying on existing stable finding identities downstream.
+
+## Migration Plan
+
+1. Run controlled baseline and local-process scans without exposing findings.
+2. Add command, completion, retry, and isolation regression tests.
+3. Enable the flag only for GitLab and restart the managed runtime.
+4. Observe at least 100 GitLab attempts or two hours with no new unexplained terminal RC=1.
+5. Replay one bounded exact-signature batch and verify queue disposition and deduplication.
+6. Roll back by disabling the GitLab lifecycle flag and restarting the source runtime.
+
+## Open Questions
+
+- Whether the policy should later become common to all externally supervised Git scans remains a separate change.