sashatrask/truf-server
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Initial server source import

Browse Source
This commit is contained in:
sashatrask
2026-09-30 20:30:56 +03:00
commit 170dd941b9
498 changed files with 261563 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
WINDOWS_IMPORT.md
+285
View File
@@ -0,0 +1,285 @@
# Windows Snapshot Import
## Current Status
As of 2026-09-16, source capture succeeded, but the target import failed during
maintenance cleanup and was manually interrupted. The destination remains
**failed, unmarked and stopped**, not verified-stopped or ready for normal run.
The user then explicitly authorized removal of copied SQLite databases,
backups, `found_secrets` outputs and archival files, preserving Windows originals.
**The staged snapshot is now intentionally incomplete: `files.tar` was deleted.**
Its retained manifest describes the original capture, not the reduced target.
Do not rewrite the manifest, retry import, restart the retained container, or
recapture/repopulate the removed copies automatically. The procedures below
describe the original full-snapshot workflow, not a resume procedure for this
pruned destination. Any future recovery needs a separately reviewed plan.
| Execution evidence | Result |
| --- | --- |
| Source snapshot publication | Manifest published 2026-09-15T19:36:28.010915+00:00; source supervisor/PG stopped flags true |
| Approved manifest SHA-256 | `08344147133c37d4b6f404cf4fac3e59d58f94917f1fa58a77cbb68c36db7e8a` |
| Original capture inventory | 50,501 files, 34,851,776,467 bytes; 49,897 active and 604 archival files; 61 tables and 38 sequences |
| Retained PostgreSQL dump | `database.dump`, 2,619,119,892 bytes; not deleted or modified by cleanup |
| Failed import container | `63286fd554f832fd3a1f073e7c923977e209149e940b684cdbf35e4479ebd5ba`; exited 129, PID 0, restarts 0, restart policy `no` |
| Pinned runtime/cleanup image | `sha256:ecf1ee044fd6e936359a5955e0a42b452b8098a3f9d822272ab373b697761de2` |
| Cleanup verification | 635 original Windows files checked for presence/size and unchanged metadata; original PG control hashes unchanged; Windows `postgres.exe` count 0 |
| Retained destination verification | Metadata of 49,866 remaining inventory files and 1,883 PG files unchanged; PG control/config hashes unchanged; initialized marker absent; application remains stopped |
| Final verified-stopped acceptance | NOT ACHIEVED; cleanup does not repair the failed import |
### Authorized Copy Cleanup
Only these copied locations were removed on 2026-09-16:
| Copied location/family | Files | Bytes removed |
| --- | ---: | ---: |
| `/data/windows-archive` including old SQLite backups and archived configurations | 604 | 10,827,425,254 |
| `/data/runtime-linux/results/scanner*.db` and associated WAL/SHM/journal files | 11 | 10,281,779,360 |
| `/data/runtime-linux/results/found_secrets.*`, including generations, manifest and publication ledger | 20 | 5,584,368,562 |
| Total from native `truf-docker_data` volume | 635 | 26,693,573,176 |
| Completed staging directory's `files.tar` on Windows D: | 1 | 34,917,959,680 |
Original `D:\truf`, `S:\postgres-data` and source bundles were not deleted or
modified. Target PostgreSQL, its dump, translated configuration, credentials,
proxies, queues, other result streams, bundles, caches and their required
publication ledgers were retained. The one-off cleanup used a network-disabled
utility container with only the verified native target volume writable; it did
not run PostgreSQL, the importer, scanners, providers or application services.
The volume gained approximately 26.69 GB of filesystem free space. Approximately
34.92 GB was freed on Windows D:. This did not compact the WSL VHDX on S: or
return all newly free ext4 blocks to the Windows host; S: reported
30,467,690,496 bytes free after cleanup. No WSL/storage reconfiguration was done.
Removing `found_secrets` files does not reset PostgreSQL projector cursors or
pending append/rotation proofs. A future authorized startup must first address
that projection state explicitly; removing files or their SQLite ledger alone
is not a safe live-cursor reset. Do not erase PostgreSQL findings, counters or
pipeline evidence to make the removed files appear consistent.
Reported regression evidence, not rerun by this documentation change: selected
Docker suite **663 passed, 7 Windows-only skipped**; synthetic snapshot tests
**58 passed**; pure config tests **12 passed**; host importer tests **66 passed,
1 POSIX-only skipped**. None is proof of this snapshot's capture or import E2E.
## Scope And Paths
The authorized operation copies the original logical database, proxies, secrets
and reviewed durable files. It does not move/delete the originals, migrate the
source schema, or execute providers, scanners, keycheckers or archived scripts.
Only exclusive PostgreSQL maintenance is allowed during capture/import. Leave
both the original supervisor and original PostgreSQL stopped after capture,
and the destination stopped after import.
Current private staging directory:
- Windows: `D:\truf-docker\docker\imports\windows-20260915-59a1c0aa23ec411b86f25c5eb9d2a4d3`
- WSL: `/mnt/d/truf-docker/docker/imports/windows-20260915-59a1c0aa23ec411b86f25c5eb9d2a4d3`
- Container: the same directory bound read-only at `/import`.
A completed snapshot contains exactly `manifest.json`, `files.tar` and
`database.dump`. Do not add reports or other files inside it. Windows staging
remains private to the capturing account and SYSTEM; preserve its ACLs rather
than making it world-readable for Docker. `docker/imports/` is excluded from
Git and the image build context. Never put dump/tar contents, credentials,
proxy values, application data or raw logs in Git, images or terminal output.
The directory listed above currently retains only `manifest.json` and
`database.dump` after the authorized cleanup. It is not a completed import input.
The destination is the base Compose native Linux volume `truf-docker_data`,
mounted at `/data`, not a Windows bind mount. Paths in braces below are reviewed
families; optional archival inputs are copied only when present.
| Original source | Destination within `/data` |
| --- | --- |
| `S:\postgres-data` via a full PG16 logical dump | `/data/postgres-linux`, independently initialized native Linux PG16 |
| `D:\truf\runtime\{results,queues,state,keychecks,postman_cache,result_spool}` | `/data/runtime-linux/{results,queues,state,keychecks,postman_cache,result_spool}` |
| `D:\truf\runtime\proxy.txt` | `/data/runtime-linux/proxy.txt` |
| `D:\truf\app\{secrets.yaml,trufflehog-custom-detectors.yaml}` | `/data/config/{secrets.yaml,trufflehog-custom-detectors.yaml}` |
| `D:\truf\app\config.yaml` | `/data/windows-archive/app/config.yaml`; translated profile at `/data/config/windows-import.yaml` |
| `S:\scanner-result-bundles\{tmp,ready,quarantine}` | `/data/scanner-result-bundles/{tmp,ready,quarantine}` |
| Reviewed archival inputs under `D:\truf` | `/data/windows-archive/` with their original relative paths |
Archival scope includes `D:\truf\state`, `runtime\backups`, `runtime\imports`,
non-authority JSON reports from `runtime\control`, `app\.streamlit\config.toml`,
`.env.postgres`, `docker-compose.postgres.yml`, `runner_state.json`,
`runtime\keychecks.7z`, `runtime\orkey.txt`, `runtime\check-openrouter-keys.ps1`,
`runtime\*.md`, root `checked_*.txt`/`todo_*.txt`, root/app `scanner.db*`,
app `config.yaml.*`/`secrets.yaml.*`, root result projection families and
`*.publication-ledger.sqlite3*`. The legacy copy tree is also archival:
`D:\truf\runtime\keychecks \u2014 \u043a\u043e\u043f\u0438\u044f`
(Unicode escapes describe the actual folder name, not a literal shell path).
Within `runtime\state`, `scan_limiter*.db*`, `*.tmp*` and
`janitor.cursor.json` are archival only, never active Linux authority/state.
Excluded: physical PGDATA/WAL, Windows PostgreSQL binaries/logs, live control
authority, locks/PIDs, `S:\scanner-work`, `runtime\downloads`, runtime git/traces/
freeze-diagnostics, `gharchive_cache`, `.git`, `.opencode`, tests and code caches.
Ordinary logs are excluded outside retained result/keycheck projection families;
`scan_errors.log*` is deliberately durable data, not a diagnostic to display.
The manifest records the actual selected inventory and exclusion counts.
The archived `.env.postgres` is never sourced or used for the target connection.
Provision generates `/data/postgres-password`; Linux uses this new
password and a different PG16 system identifier, not the source password or
physical cluster. Archived Windows configs are not executable runtime profiles.
## Capture And Capacity Gates
Main runs `D:\truf-docker\docker\windows_snapshot.py` using native PowerShell,
not context-mode or another Windows Job wrapper: original PostgreSQL correctly
refuses Job membership. The exporter temporarily starts only source maintenance
PostgreSQL and must confirm its stop before publication. Do not rerun capture
into the current attempt directory, reuse partial files, or kill a process that
is retaining authority while stop remains unconfirmed.
After capture exits 0 and publishes its final manifest, main records its digest
from the trusted Windows path, then checks that the WSL-visible manifest has
the same digest. Do not replace the approved pin with a newly computed digest
merely to bypass a mismatch. Native PowerShell digest command:
```powershell
(Get-FileHash -LiteralPath 'D:\truf-docker\docker\imports\windows-20260915-59a1c0aa23ec411b86f25c5eb9d2a4d3\manifest.json' -Algorithm SHA256).Hash.ToLowerInvariant()
```
Check Windows `D:` staging capacity and, independently, physical free space on
`S:`, which backs the Docker/WSL VHDX, and free space on native Linux `/data`.
Record the actual VHDX/daemon storage location; a large Linux `df` result does
not prove the Windows host can grow the VHDX. Budget its anticipated growth
while retaining at least **20 GiB physical free on S:**. The importer cannot
measure or enforce this host-side reserve.
The Linux preflight requires `file_bytes + database_bytes + 20 GiB` free, using
source physical database size from manifest metadata. Older v1 metadata without
that size uses `max(24 GiB, 4 * dump_bytes)` as the database estimate. At least
**20 GiB must still be free on Linux after import**. Check both host and guest
capacity during and after restoration; compressed dump size alone is not a
capacity estimate. Do not delete original data to make space.
## Offline Procedure
Run these steps separately in WSL Bash only after main approves the capture and
capacity evidence. Use the existing Linux Docker daemon and already-built,
importer-integrated `truf-local:runtime` image; no builds or pulls here. Keep the
fixed project/directory below. Do not use the generic initialize/start procedure
in `DOCKER_MIGRATION.md` for this full-schema snapshot.
```bash
TRUF_WINDOWS_SNAPSHOT='/mnt/d/truf-docker/docker/imports/windows-20260915-59a1c0aa23ec411b86f25c5eb9d2a4d3'
TRUF_WINDOWS_SNAPSHOT_SHA256='PENDING'
dci() {
if [[ ! "${TRUF_WINDOWS_SNAPSHOT_SHA256:-}" =~ ^[0-9a-f]{64}$ ]]; then
printf '%s\n' 'STOP: set the approved lowercase manifest SHA-256.' >&2
return 1
fi
sudo -n env \
TRUF_WINDOWS_SNAPSHOT="${TRUF_WINDOWS_SNAPSHOT:?Set the completed snapshot directory}" \
TRUF_WINDOWS_SNAPSHOT_SHA256="$TRUF_WINDOWS_SNAPSHOT_SHA256" \
docker compose \
--project-name truf-docker \
--project-directory /mnt/d/truf-docker \
--env-file /dev/null \
--file /mnt/d/truf-docker/compose.yaml \
--file /mnt/d/truf-docker/compose.snapshot-import.yaml \
"$@"
}
sha256sum "$TRUF_WINDOWS_SNAPSHOT/manifest.json"
dci config --quiet
sudo -n docker image inspect --format '{{.Id}}' truf-local:runtime
sudo -n docker volume inspect --format '{{.Name}} {{.Driver}} {{.Mountpoint}}' truf-docker_data
sudo -n docker container inspect --format '{{.State.Status}}' truf-docker-snapshot-import
```
Replace `PENDING` with the previously approved pin, not a credential. The
`sudo -n env NAME=value ... docker compose` form explicitly passes the two
non-secret interpolation inputs even when sudo strips shell exports. Do not
use `sudo -E` or pass source connection/provider variables. `--env-file /dev/null`
prevents implicit checkout `.env` loading, but does not sanitize shell exports;
use a clean operator shell and no unreviewed Docker/Compose overrides.
Main must separately confirm the image identity, **absence** of
`truf-docker_data`, and absence of the retained import container name before
provisioning. Only specific no-such-volume/no-such-container responses establish
absence; daemon/permission errors do not. If either already exists, stop for
review instead of adopting, overwriting or deleting it.
```bash
dci run --rm --no-deps --pull never -T provision
```
Proceed only after successful provision. This unchanged base service creates
the private layout, generated password and empty placeholders, not an application
schema. **Do not call `initialize`, `import-secrets`, or normal `run` first.**
The importer uses `initialize-empty` internally and restores the entire custom
dump into a virgin schema before raw table/sequence comparison and permitted
target-only recovery/migrations.
```bash
dci run --detach --no-deps --pull never -T \
--name truf-docker-snapshot-import runtime
```
This is the single retained maintenance container: no `--rm`, no automatic
restart, no dependency startup, no healthcheck and `network_mode: none`.
The override preserves the base image/entrypoint, non-root UID/GID, read-only
rootfs, capabilities/security policy, native `/data` volume, tmpfs and resource
limits. It adds only read-only `/import`; `create_host_path: false` rejects a
missing source directory rather than silently creating one.
Import starts with `/opt/truf/app/config.linux.yaml` in both the environment
and explicit `--config`. **Do not merge `compose.windows-import.yaml` here.**
The translated private profile does not exist at initial preflight; the importer
creates and selects it internally only after validating/extracting the snapshot.
## Stopped Verification
```bash
sudo -n docker container wait truf-docker-snapshot-import
sudo -n docker container inspect --format \
'status={{.State.Status}} exit={{.State.ExitCode}} oom={{.State.OOMKilled}} restarts={{.RestartCount}} network={{.HostConfig.NetworkMode}} restart={{.HostConfig.RestartPolicy.Name}} auto_remove={{.HostConfig.AutoRemove}}' \
truf-docker-snapshot-import
```
Require `status=exited exit=0 oom=false restarts=0 network=none restart=no
auto_remove=false`. `wait` prints the container exit code; the command's own
exit status alone is not import success. Waiting may take hours or hold while
maintenance stop is uncertain. Do not impose a timeout that kills the container.
Do not display raw `docker logs`, Compose logs, database logs, full environment
dumps or application data. Only structured importer numeric phase/count/byte
events and allowlisted aggregate/hash evidence are suitable for progress.
Phase 13 is emitted before final publication and is not success proof.
Main must privately inspect the following evidence from the stopped retained
container, for example with `docker cp` into a separate owner-only evidence
directory outside `/import`. Do not start another runtime to inspect it; `health`
and `status` are live readiness actions, not stopped-import verification.
- `/data/config/windows-import-manifest.json`: its byte SHA-256 equals the approved staging manifest pin; source stopped flags are true. The raw evidence, report and initialized marker all carry that same `manifest_sha256`. Report archive/dump hashes match this manifest and the verified staging files.
- `/data/config/windows-import-raw.json`: its SHA-256 matches report `raw_evidence_sha256`; `table_counts` equals manifest `database.table_counts`, `sequences_provided` is true and `sequences_verified` equals manifest `database.sequence_count`. The importer checks actual sequence values before transformations; this evidence records their verified count, not their values. Record only aggregate tables/rows/sequences.
- `/data/config/windows-import.yaml`: hash bytes without displaying values; SHA-256 matches report `config_sha256`.
- `/data/config/windows-import-report.json`: `status` is `verified-stopped`, `maintenance_stopped` is true, all `pipeline_after` counts are zero, final Linux reserve is at least 20 GiB, and cutover/migration/preserved-evidence checks succeeded. Record any reported fenced recovery or Postman rebasing; these may legitimately change final target counts or move incoming tmp/ready bundles after raw comparison.
- `/data/initialized.json`: exists as the last publication, has format `truf-container-data-v1`, `pg_major` 16 and the approved `manifest_sha256`; `import_report_sha256` matches the actual report bytes. Its system identifier equals report `linux_system_identifier` and differs from the source identifier.
- Confirm destination `/data/postgres-linux/postmaster.pid` is absent, original supervisor/PostgreSQL remain stopped, and host/guest space reserves still hold. Record all results in the PENDING table before declaring verified-stopped.
## Failure And Release
Any nonzero exit, OOM, missing/mismatched evidence or uncertain stop is not
verified-stopped. Retain the container, volume and private snapshot. A partial
import is failed/unmarked, not automatically resumable; early rejection can
leave no report. A report saying verified-stopped without a matching final
initialized marker and clean container exit is still insufficient.
Do not automatically retry/restart, overwrite, delete, remove locks/markers,
initialize, prune, run `down --volumes`, or force-kill an authority-holding
importer. Review the retained state first; any new attempt needs an explicit
decision and separately approved fresh destination, not cleanup by this runbook.
There is **no automatic normal run**. A future live run requires separate user
authorization after verified-stopped acceptance. Only then use base
`compose.yaml` plus `compose.windows-import.yaml`, without the snapshot override,
so runtime, health and status all select `/data/config/windows-import.yaml`.
Do not start either the original or destination supervisor as part of import.