Initial server source import
This commit is contained in:
sashatrask
@@ -0,0 +1,5 @@
|
||||
FROM truf-worker-test:test AS edge-e2e-runtime
|
||||
|
||||
USER 0:0
|
||||
COPY --chown=10001:10001 --chmod=0600 tests/edge_e2e_backend.py tests/edge_e2e_client.py /opt/truf/tests/
|
||||
USER 10001:10001
|
||||
@@ -0,0 +1,273 @@
|
||||
# Runtime Dependency Build
|
||||
|
||||
This directory records the public build inputs and pip-tools generator. It is not
|
||||
an application configuration directory and must never contain credentials.
|
||||
|
||||
## Pins
|
||||
|
||||
| Input | Pin |
|
||||
| --- | --- |
|
||||
| Python image | `python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254` |
|
||||
| Python version | `3.12.14` |
|
||||
| Linux amd64 manifest | `sha256:9c47360a2a0355e2da18516d0b1c2126ec22c195d2185e97347c9d98398c5bef` |
|
||||
| Linux arm64/v8 manifest | `sha256:d04f49f5882f49a3b91f874e75e19f0c265f7222da8659741a9d7eab148f22a9` |
|
||||
| Debian and Debian security snapshots | `20260914T000000Z` |
|
||||
| Git and git-man | `1:2.39.5-0+deb12u3` |
|
||||
| tini | `0.19.0-1+b3` |
|
||||
| CA certificates (already present in the pinned base) | `20250419~deb12u1` |
|
||||
| PGDG server, client, libpq | `16.15-1.pgdg12+2` |
|
||||
| PGDG common and client-common | `293.pgdg12+1` |
|
||||
| PGDG signing-key fingerprint | `B97B0AFCAA1A47F044F244A07FCC7D46ACCC4CF8` |
|
||||
| PGDG signing-key SHA-256 | `0144068502a1eddd2a0280ede10ef607d1ec592ce819940991203941564e8e76` |
|
||||
| TruffleHog | `3.97.4` |
|
||||
| TruffleHog Linux amd64 archive SHA-256 | `dc24007c2f233bd61c05beabeb44aa27ea9b43288166279209abe0458c5ce76b` |
|
||||
| TruffleHog Linux amd64 archive size | `34970205` bytes |
|
||||
| TruffleHog Linux arm64 archive SHA-256 | `7e65e771d2a247964056aa5edba0f8ae3945895e5dce867fe0ffbc7b0128239a` |
|
||||
| TruffleHog Windows amd64 archive SHA-256 | `6ce9a957ac62bfb19463048333d9e8481327dbbf5bdc0c43f5ab5327b9631fb9` |
|
||||
| Windows embeddable Python | `3.12.10`, SHA-256 `4acbed6dd1c744b0376e3b1cf57ce906f9dc9e95e68824584c8099a63025a3c3` |
|
||||
| Windows MinGit | `2.47.1.windows.1`, SHA-256 `50b04b55425b5c465d076cdb184f63a0cd0f86f6ec8bb4d5860114a713d2c29a` |
|
||||
| pip-tools | `7.6.1` |
|
||||
| Generator pip | `26.2.1` |
|
||||
| pytest | `8.4.2` |
|
||||
| httpx (test target only) | `0.28.1` |
|
||||
| zstandard | `0.23.0` |
|
||||
| pandas | `3.0.5` |
|
||||
| plotly | `7.0.0` |
|
||||
| streamlit | `1.63.0` |
|
||||
| psycopg and psycopg-binary | `3.3.5` |
|
||||
| boto3 and botocore | `1.43.94` |
|
||||
|
||||
TruffleHog's expected hashes were checked against the public release's
|
||||
[`trufflehog_3.97.4_checksums.txt`](https://github.com/trufflesecurity/trufflehog/releases/download/v3.97.4/trufflehog_3.97.4_checksums.txt).
|
||||
The PGDG key's primary OpenPGP fingerprint was independently calculated from the
|
||||
hash-pinned public key and matches the fingerprint above.
|
||||
|
||||
The Dockerfile pins the base index, download digests, PGDG package versions, and
|
||||
Debian snapshot. Apt verifies Debian signatures with its shipped archive keyring
|
||||
and PGDG signatures with the separately hash-pinned, repository-scoped armored
|
||||
key. Full GnuPG is not installed. Expired `Valid-Until` checks are disabled only
|
||||
for the immutable Debian snapshots, never signature verification. The official
|
||||
PGDG archive retains older package versions; apt preferences exclude every PGDG
|
||||
package except the five exact pins above.
|
||||
|
||||
The complete Debian Git package and HTTPS helper are retained. Native executable
|
||||
symlinks in the Python/Git tool directories, and PG client version-wrapper links,
|
||||
are replaced with root-owned regular hard links. Dependencies stay in the
|
||||
interpreter's `/usr/local/lib/python3.12/site-packages`, not a virtual environment
|
||||
or user site. Installation requires hashes and binary wheels and disables
|
||||
bytecode generation. The application lock includes both manifests, optional
|
||||
dashboard packages, and pytest in one environment. The test target layers its
|
||||
separate hash lock containing Starlette TestClient's `httpx` dependency; the
|
||||
runtime target does not contain `httpx`. The separate worker lock contains only
|
||||
Requests, PyYAML, zstandard, and their four transitives; it excludes PostgreSQL,
|
||||
server, dashboard, test, and detailed-keycheck dependencies.
|
||||
|
||||
`docker/worker-package-pins.json` is the canonical public build-input record for
|
||||
Linux and Windows worker artifacts. Worker manifests hash every application,
|
||||
dependency, scanner, detector-policy, Python-runtime, and complete Git-runtime
|
||||
file. The package grants no provider or detailed keycheck authority and contains
|
||||
no server or database credentials.
|
||||
|
||||
PostgreSQL service starts and automatic cluster creation are disabled during
|
||||
installation. No database is initialized by the build. Generated distribution
|
||||
snakeoil TLS keys are removed in the same layer. Package pins make dependency
|
||||
selection repeatable; this is not a claim of byte-for-byte identical OCI images
|
||||
across BuildKit versions or package-maintainer timestamp generation.
|
||||
|
||||
## Generate Locks
|
||||
|
||||
All four requirements lock files are generated by pip-tools on Linux with the pinned
|
||||
Python 3.12.14 interpreter. Do not edit any generated lock by hand. The initial
|
||||
compiler bootstrap installs only public `pip==26.2.1` and `pip-tools==7.6.1`, then
|
||||
resolves and hashes the compiler's entire dependency closure. Subsequent compiler
|
||||
installs use that generated hash lock.
|
||||
|
||||
The generator also copies the existing locks, so normal regeneration retains
|
||||
valid pins. Use pip-compile's `--upgrade` only for an intentional dependency
|
||||
refresh, then rebuild and revalidate the dependency image.
|
||||
|
||||
Run these Docker commands from the isolated source checkout (`D:\truf-workers` for
|
||||
this change), using WSL's
|
||||
`sudo -n docker -H unix:///var/run/docker.sock` in place of `docker` on this host.
|
||||
The generator receives only the three public application manifests and compiler
|
||||
inputs. There are no host bind mounts.
|
||||
|
||||
```sh
|
||||
docker build --target lock-generator -t truf-lock-generator:py3.12.14 .
|
||||
docker run --name truf-runtime-lock truf-lock-generator:py3.12.14
|
||||
docker cp truf-runtime-lock:/src/docker/requirements.lock docker/requirements.lock
|
||||
docker rm truf-runtime-lock
|
||||
docker run --name truf-test-lock truf-lock-generator:py3.12.14 \
|
||||
python3 -m piptools compile --generate-hashes --allow-unsafe \
|
||||
--resolver=backtracking --strip-extras --no-emit-index-url \
|
||||
--no-emit-trusted-host --index-url=https://pypi.org/simple \
|
||||
--pip-args=--only-binary=:all: \
|
||||
--output-file=docker/requirements-test.lock docker/requirements-test.in
|
||||
docker cp truf-test-lock:/src/docker/requirements-test.lock docker/requirements-test.lock
|
||||
docker rm truf-test-lock
|
||||
docker run --name truf-worker-lock truf-lock-generator:py3.12.14 \
|
||||
python3 -m piptools compile --generate-hashes --allow-unsafe \
|
||||
--resolver=backtracking --strip-extras --no-emit-index-url \
|
||||
--no-emit-trusted-host --index-url=https://pypi.org/simple \
|
||||
--pip-args=--only-binary=:all: \
|
||||
--output-file=docker/requirements-worker.lock docker/requirements-worker.in
|
||||
docker cp truf-worker-lock:/src/docker/requirements-worker.lock docker/requirements-worker.lock
|
||||
docker rm truf-worker-lock
|
||||
docker run --name truf-compiler-lock truf-lock-generator:py3.12.14 \
|
||||
python3 -m piptools compile --generate-hashes --allow-unsafe \
|
||||
--resolver=backtracking --strip-extras --no-emit-index-url \
|
||||
--no-emit-trusted-host --index-url=https://pypi.org/simple \
|
||||
--pip-args=--only-binary=:all: \
|
||||
--output-file=docker/build-dependencies/requirements.lock \
|
||||
docker/build-dependencies/requirements.in
|
||||
docker cp truf-compiler-lock:/src/docker/build-dependencies/requirements.lock docker/build-dependencies/requirements.lock
|
||||
docker rm truf-compiler-lock
|
||||
```
|
||||
|
||||
## Build Targets
|
||||
|
||||
```sh
|
||||
docker build --target dependencies -t truf-dependencies:py3.12.14-pg16.15-th3.97.4 .
|
||||
docker build --target runtime -t truf-runtime:local .
|
||||
docker build --target test -t truf-test:local .
|
||||
docker build --target worker -t truf-remote-worker:linux-x86_64 .
|
||||
```
|
||||
|
||||
## Remote Worker Artifacts
|
||||
|
||||
The `worker` target is independent of the server runtime. It uses the pinned image
|
||||
Python and includes the worker authority, required shared DB-free modules, worker
|
||||
dependency lock, detector policy, TruffleHog, CA roots, tini, and a package-local
|
||||
complete Git helper tree. The final build executes the scanner and Git version
|
||||
checks and verifies the full schema-3/protocol-2 capability package as unprivileged UID 10001. PostgreSQL
|
||||
tools, server runtime/control authority, test code, `httpx`, provider and detailed
|
||||
keycheck authority, server credentials, and database credentials are absent.
|
||||
|
||||
Build the Windows amd64 portable directory and deterministic ZIP from the same
|
||||
isolated checkout:
|
||||
|
||||
```powershell
|
||||
New-Item -ItemType Directory -Path dist -Force | Out-Null
|
||||
python -B app/worker_package_builder.py windows `
|
||||
--project-root . `
|
||||
--output dist/truf-worker-windows-x86_64 `
|
||||
--archive dist/truf-worker-windows-x86_64.zip `
|
||||
--cache build/worker-cache
|
||||
```
|
||||
|
||||
The builder downloads the hash-and-size-pinned Python 3.12.10 embeddable runtime,
|
||||
MinGit 2.47.1, and TruffleHog 3.97.4, installs the cross-platform worker lock with
|
||||
hash checking, verifies the complete package, and writes adjacent release JSON.
|
||||
After extracting the ZIP, run `prepare-worker.ps1` once to replace inherited ACLs,
|
||||
then use `run-worker.cmd`. Local files rely on private OS ACLs rather than
|
||||
application-layer encryption; the client has no TLS-verification bypass.
|
||||
|
||||
Build a complete distributable release (Windows ZIP, Linux image archive,
|
||||
Docker bundle, manifests, README, Compose file, and SHA-256 lists) with one
|
||||
PowerShell command:
|
||||
|
||||
```powershell
|
||||
.\build_worker_release.ps1 -ReleaseName release-YYYYMMDD-vN
|
||||
```
|
||||
|
||||
The script uses native Docker when available and otherwise uses the Docker Engine
|
||||
in `Ubuntu-24.04` through WSL. Use `-WslDistro NAME` for another distribution.
|
||||
The destination must not already exist; a failed build remains on disk for
|
||||
inspection and is never published as a partial replacement.
|
||||
|
||||
While the main context allowlist is pending, the dependency build can use this
|
||||
explicit two-file context from WSL. It does not send any application code, secret
|
||||
files, findings, state, or original checkout directories to the builder:
|
||||
|
||||
```sh
|
||||
tar -C /mnt/d/truf-docker -cf - Dockerfile docker/requirements.lock \
|
||||
| sudo -n docker -H unix:///var/run/docker.sock build --file Dockerfile \
|
||||
--target dependencies -t truf-dependencies:py3.12.14-pg16.15-th3.97.4 -
|
||||
```
|
||||
|
||||
`dependencies` stops before copying application code. `runtime-base` holds the
|
||||
production filesystem and launch configuration; `test` inherits those exact
|
||||
contents and adds private test sources. The last/default target, `runtime`, is a
|
||||
direct alias of `runtime-base` and contains no test tree. The test entrypoint uses
|
||||
the same interpreter isolation flags and the real `child_bootstrap.py` dependency
|
||||
path loader, without processing `.pth` files or starting the application.
|
||||
|
||||
Application and test copies are owned by `10001:10001`; their directories are
|
||||
`0700` and regular files `0600`. Build-time checks reject symlinks, special files,
|
||||
and cached bytecode in these controlled copies. No recursive permission changes
|
||||
are made to host paths or runtime-mounted data. The default user is `10001:10001`;
|
||||
the image precreates private `/data` and `/data/home` directories.
|
||||
|
||||
## Verified Results
|
||||
|
||||
Validated on 2026-09-15 using Ubuntu 24.04 under WSL2, stock Docker Engine 29.8.0,
|
||||
the local Unix socket, and `sudo -n`. Builds and generation were polled without
|
||||
printing full dependency logs. A temporary WSL keepalive prevented idle shutdown
|
||||
during detached lock generation; no host configuration changes were made.
|
||||
|
||||
| Result | Value |
|
||||
| --- | --- |
|
||||
| Dependency image tag | `truf-dependencies:py3.12.14-pg16.15-th3.97.4` |
|
||||
| Local dependency image ID | `sha256:3fbdc2ea6fd1199aa742f54fb653e433d79ad3f1a5e4bfeed8b413dc9f704eb5` |
|
||||
| Built and executed platform | `linux/amd64` |
|
||||
| Runtime lock | 49 exact package pins, 1059 SHA-256 wheel hashes |
|
||||
| Runtime lock SHA-256 | `82c69394b116fd8a762c3dea481682045af87c013974fcb78ac0529892188537` |
|
||||
| Compiler lock | 8 exact package pins, 8 SHA-256 wheel hashes |
|
||||
| Compiler lock SHA-256 | `0172b08004c6f2b0702ea9a472300cc63243492df2d3d782fd4dbaa612497fd2` |
|
||||
| Lock replay in the hash-locked generator image | Both files byte-for-byte identical |
|
||||
| `pip check` | No broken requirements |
|
||||
|
||||
- The `dependencies` and `lock-generator` targets built successfully with hash-required, binary-wheel-only installs.
|
||||
- All 49 package import checks passed normally, then with `-I -S -B` through the actual `child_bootstrap.py` loader for all 10 child kinds. Application entrypoints and providers were not executed.
|
||||
- Isolated `sys.path` contained only the interpreter ZIP path, standard library, `lib-dynload`, and `/usr/local/lib/python3.12/site-packages`. Neither the working directory nor the user site was added.
|
||||
- 14,202 dependency-tree permission checks found root ownership, no group/world write access, and no symlinks or bytecode files. The unprivileged UID could not write these dependencies.
|
||||
- Eleven selected native executables were regular root-owned files. Version and `ldd` checks passed for Python, Git, the Git HTTPS helper, tini, TruffleHog, and the six PG16 tools. TruffleHog is static; other checked ELF binaries had no missing shared libraries.
|
||||
- TruffleHog global, Git, filesystem, Docker, and Hugging Face help flags were checked, including the scanner's legacy `--local-dev` and `--log-level` options. No scans or provider requests were made.
|
||||
- No PostgreSQL `PG_VERSION` file or initialized cluster was present. No PostgreSQL server, application, or provider was started. Full GnuPG is absent.
|
||||
- Minimal bootstrap-only runtime and test fixtures exercised the actual copy stages: `10001:10001`, directories `0700`, files `0600`. Generated in-memory contexts containing a symlink or `.pyc` file were rejected by the build.
|
||||
- The test target's real tini/Python/bootstrap entrypoint reported `pytest 8.4.2` with networking disabled, a read-only root, and a UID-10001 private `/tmp` tmpfs. Pytest capture needs writable temporary storage even for `--version`.
|
||||
|
||||
The temporary runtime/test fixtures were deliberately incomplete and were used
|
||||
only for dependency and filesystem-policy verification. They are not deployable
|
||||
application images. No application test suite, PostgreSQL initialization test,
|
||||
provider integration, or arm64 build was run. The arm64 base and TruffleHog assets
|
||||
are pinned, but that platform still requires a native or emulated build/test.
|
||||
|
||||
## Integration Boundary
|
||||
|
||||
The main integration owns `.dockerignore`, `app/container_runtime.py`, Compose,
|
||||
and tests. The context allowlist must include the exact Dockerfile, dependency
|
||||
input/lock paths, this metadata file, the new runtime entrypoint, and the intended
|
||||
test files. Do not replace the deny-by-default context rules with directory-wide
|
||||
or wildcard exceptions. Keep `.env`, credentials, findings, original runtime
|
||||
directories, and generated caches excluded.
|
||||
|
||||
New exact metadata/input exceptions required in the main-owned `.dockerignore`:
|
||||
|
||||
```text
|
||||
!docker/requirements.in
|
||||
!docker/requirements.lock
|
||||
!docker/build-dependencies/requirements.in
|
||||
!docker/build-dependencies/requirements.lock
|
||||
!docker/build-dependencies/README.md
|
||||
```
|
||||
|
||||
The Dockerfile is already allowlisted. Main must separately allowlist its
|
||||
`app/container_runtime.py` and intended test source files once they exist. The
|
||||
test target copies only `app/` and `tests/`; repository tests that read root-level
|
||||
Compose, Docker, or PowerShell fixtures still need main-owned fixture integration.
|
||||
|
||||
Compose must select the runtime target, enforce a read-only root filesystem,
|
||||
provide newly initialized Linux writable volumes and a temporary filesystem as
|
||||
needed, and preserve the unprivileged UID/GID. Runtime startup, PostgreSQL
|
||||
initialization, provider execution, and existing-data integration are explicitly
|
||||
outside dependency-build validation.
|
||||
|
||||
For a read-only test image, provide private temporary storage, for example
|
||||
`--tmpfs /tmp:rw,nosuid,nodev,noexec,size=64m,mode=0700,uid=10001,gid=10001` for
|
||||
version/import checks. Main must choose temporary-storage size and execution
|
||||
policy appropriate for its full tests. After the entrypoint, allowlist, fixtures,
|
||||
and Compose changes are integrated, rebuild complete `runtime` and `test` images
|
||||
and perform the separately authorized integration checks. No files outside the
|
||||
Dockerfile and `docker/` build inputs were edited, and nothing was staged,
|
||||
committed, or pushed.
|
||||
@@ -0,0 +1,3 @@
|
||||
# Toolchain only; these packages are not copied from the generator into runtime.
|
||||
pip==26.2.1
|
||||
pip-tools==7.6.1
|
||||
@@ -0,0 +1,40 @@
|
||||
#
|
||||
# This file is autogenerated by pip-compile with Python 3.12
|
||||
# by the following command:
|
||||
#
|
||||
# See docker/build-dependencies/README.md for the pinned Python 3.12.14 pip-tools generation command.
|
||||
#
|
||||
--only-binary :all:
|
||||
|
||||
build==1.6.1 \
|
||||
--hash=sha256:ecd351a4be9d35a9eaaba244a7687143c9c7d4aea6ac964e7e7ddab20cbcf4e7
|
||||
# via pip-tools
|
||||
click==8.5.0 \
|
||||
--hash=sha256:255bc9599cf7748b4b1a446ccc735421bd08a2ae529a8b88597d3de5664ee360
|
||||
# via pip-tools
|
||||
packaging==26.3 \
|
||||
--hash=sha256:d7193f7c8e4e93f444fde0262bf90af30e16fa0ad0ad44cb553c87339b23cd1c
|
||||
# via
|
||||
# build
|
||||
# wheel
|
||||
pip-tools==7.6.1 \
|
||||
--hash=sha256:6111c8b4b07fd14b7223ca921485b0e96cf66e20bf94da95eeed9845f510cb8f
|
||||
# via -r docker/build-dependencies/requirements.in
|
||||
pyproject-hooks==1.2.0 \
|
||||
--hash=sha256:9e5c6bfa8dcc30091c74b0cf803c81fdd29d94f01992a7707bc97babb1141913
|
||||
# via
|
||||
# build
|
||||
# pip-tools
|
||||
wheel==0.48.0 \
|
||||
--hash=sha256:3217dcc807155e45db462d7ef2431f5ddda0d7273b700d05a67b271ceb1287ab
|
||||
# via pip-tools
|
||||
|
||||
# The following packages are considered to be unsafe in a requirements file:
|
||||
pip==26.2.1 \
|
||||
--hash=sha256:71138adf1f4ca900cdb7d289c21b7494329f2332b6d85f0e1c42108c0384ed3e
|
||||
# via
|
||||
# -r docker/build-dependencies/requirements.in
|
||||
# pip-tools
|
||||
setuptools==84.0.0 \
|
||||
--hash=sha256:51a52592b3b99e102b609654876bd65f19f999935166d1352678931132b0c670
|
||||
# via pip-tools
|
||||
@@ -0,0 +1 @@
|
||||
httpx==0.28.1
|
||||
@@ -0,0 +1,33 @@
|
||||
#
|
||||
# This file is autogenerated by pip-compile with Python 3.12
|
||||
# by the following command:
|
||||
#
|
||||
# See docker/build-dependencies/README.md for the pinned Python 3.12.14 pip-tools generation command.
|
||||
#
|
||||
--only-binary :all:
|
||||
|
||||
anyio==4.15.1 \
|
||||
--hash=sha256:6152fdbbf9a77fdec97731721bebf7c4c44f7c29b424b0065826173efc7ed101
|
||||
# via httpx
|
||||
certifi==2026.7.22 \
|
||||
--hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775
|
||||
# via
|
||||
# httpcore
|
||||
# httpx
|
||||
h11==0.16.0 \
|
||||
--hash=sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86
|
||||
# via httpcore
|
||||
httpcore==1.0.9 \
|
||||
--hash=sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55
|
||||
# via httpx
|
||||
httpx==0.28.1 \
|
||||
--hash=sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad
|
||||
# via -r docker/requirements-test.in
|
||||
idna==3.20 \
|
||||
--hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c
|
||||
# via
|
||||
# anyio
|
||||
# httpx
|
||||
typing-extensions==4.16.0 \
|
||||
--hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8
|
||||
# via anyio
|
||||
@@ -0,0 +1,3 @@
|
||||
PyYAML==6.0.3
|
||||
requests==2.34.2
|
||||
zstandard==0.23.0
|
||||
@@ -0,0 +1,365 @@
|
||||
#
|
||||
# This file is autogenerated by pip-compile with Python 3.12
|
||||
# by the following command:
|
||||
#
|
||||
# See docker/build-dependencies/README.md for the pinned Python 3.12.14 pip-tools generation command.
|
||||
#
|
||||
--only-binary :all:
|
||||
|
||||
certifi==2026.7.22 \
|
||||
--hash=sha256:62f22742b58a1a33014a2b6b706588a8d7e2a88ae7bd1a6ebe8c992928483775
|
||||
# via requests
|
||||
charset-normalizer==3.5.1 \
|
||||
--hash=sha256:00668ebb0609751758682eb0b5857e7c35b9f00e84dfdef062e103244ec94d45 \
|
||||
--hash=sha256:012a22b88a77ca2e59b98ac5889b0deb604147666032f45e6d6e217634d2550d \
|
||||
--hash=sha256:01e93745f7f219b703b60ba7afead36cfc4242782be5af484673fc500df12da5 \
|
||||
--hash=sha256:04368edf83514385ffc3e1cfd4546e595f4f1272dd23ba437a93a9cc3741d47b \
|
||||
--hash=sha256:0722590aabf9dc6a6c0343d523c05458fa2b5047dbe6302fd526bb570600753f \
|
||||
--hash=sha256:07ffd07412fc5d5e84cd8952acf9ff7e4ed7a708e69d1bada19d8ba91711353f \
|
||||
--hash=sha256:09a7bba9f739468c8e78c36a75c33768e53cb1959fc638f510454c14683f00d5 \
|
||||
--hash=sha256:0b2b1b3fa5670c127b246df1d0c059defd41f689a868a3b9d79df9b1cac42d22 \
|
||||
--hash=sha256:0c6dfb5ca6723eeed15aa8e564a014d69fcb8812f94eef11fe3631e0508199f5 \
|
||||
--hash=sha256:0d929fc574b4d6fd9e7c0f5c2ede8716a41911923aa7fa5fce38e0818aa4a1ac \
|
||||
--hash=sha256:13e3afe97712e8887cd516e960c63f0b93122971e5b5e4b2622fe7701771e838 \
|
||||
--hash=sha256:15f024313246a4ed976c60f440bb8d257815513a681d212ff74fd46f7d715a90 \
|
||||
--hash=sha256:195ce897c6153c0700078142cf8efe3e6454ca4cf4357499e4078dfd83396626 \
|
||||
--hash=sha256:19a3dd5aa73cef1c99687c4fc57db016a9c17104ae1185da88ba566a5d3bebe4 \
|
||||
--hash=sha256:1d1c7a53a6c2103925cdd6d7229f8c567379f211c869793df679f2e9f738c369 \
|
||||
--hash=sha256:1f5883d77fd409a261abb5dc8ccbe335720d798b1de4abb3b1d47ccbbc76b53b \
|
||||
--hash=sha256:21b82d8082f6f5e7f456ef0bd16323d08de1266efbfeb476e64b2a91d1471a4e \
|
||||
--hash=sha256:252d099029bcbea642f2a06c4ed5046bdf8b5a8150b64afa5e027e88b106e5ee \
|
||||
--hash=sha256:256dd4d85d9e4dc595e2bc983c980e73f62ddeb3165c58b4c3dfe78c5c8548c1 \
|
||||
--hash=sha256:26422d45fd13551cf564c58932f7d72b4f58b93b0fcf18c35ba6be12b46bb102 \
|
||||
--hash=sha256:2679de311c7946dde5d3b6f44941844133ff5c7cb86099c0061ab1e8901c20a8 \
|
||||
--hash=sha256:29880d17a8eb0b5cfdfd8944b468322928059aa35f1f5fa8ff22b149ec0b42f8 \
|
||||
--hash=sha256:2bced4061f000f7187254a02ad3433ae17eaf991747ceea2f478422590a5bba9 \
|
||||
--hash=sha256:2e9cf9253119d8e5d111f05d71626786fd3d6193817316eab1ca088cdb8593cf \
|
||||
--hash=sha256:2f06b7eae9dbe77fe1d644ca244dad508de8d302870a43f3c559b521270938a0 \
|
||||
--hash=sha256:2f293479cce755c75f1697e87c409b7ae4c555c7dfecb6e988ad13abba943031 \
|
||||
--hash=sha256:329fc3ccb63ad22d867d84c2adea759a64079a37ba4a343433b02c7a2816871e \
|
||||
--hash=sha256:343fb4f2821043bd87095f7b08a1a181febc8e36ac64212143bbfd0a0e1bc235 \
|
||||
--hash=sha256:3588e376b3ea2eea84976f67273d679f229e24c66dce7b82ae45aef04ff6e072 \
|
||||
--hash=sha256:35aea775dc2bd5f54cd84a1cd2696cc3207c479cb9cf0bd346f0d343e4300ddb \
|
||||
--hash=sha256:35fe081843b35aad20ffeccec3eeffbe637b15d14f3fb22cc1b59cd8ec17e93c \
|
||||
--hash=sha256:36047af20e17097c3bb9476c2b7655f2f7aa51322c0ba58c07695bedf755a950 \
|
||||
--hash=sha256:3617ac3cfd8b9888f145ad89dd6e692285834b0201c6074a5eeaad3fd4d668c2 \
|
||||
--hash=sha256:366ec70f5547c640d3ce1985722490f23faf4eb5216a7eeba78277490e78dacb \
|
||||
--hash=sha256:394fea06235c8543390050ed5f529187074b029fb027213f6c46ac11ab5d950e \
|
||||
--hash=sha256:3d27167433c0d5f18dc850f07d0b3816221984fecdc405d6c157a6f0b8f8e9e6 \
|
||||
--hash=sha256:3e5e1224c0a6a90e05843e07adfec669edebec17801c67072f51e59561d63c0b \
|
||||
--hash=sha256:41876ee62a3dddf48ff1121ad8f0798032aa03f2fd35f21f34a4cab14f18d8d2 \
|
||||
--hash=sha256:433c5a81eade63b47e522303bad236f59dba55ea6951746f5558355eeed8c75d \
|
||||
--hash=sha256:4582c27e8c889d64811987b5967fbd3ae0c823fe1fd933b543d55ac20bb475fa \
|
||||
--hash=sha256:485a0d363cafefcd2538a73c7c838daa2035f09b2c9f9b5e3133f80c6aeb84c2 \
|
||||
--hash=sha256:494b70049a4d69aec6e8137c13af4cf8db8c9f9820a1392ac293b0dd2987a818 \
|
||||
--hash=sha256:496846868fea80e479324862fa877f02411f2fd0f83b79ccee2607aa68b2a032 \
|
||||
--hash=sha256:4abdc5f9ad448c1ecbfae2974b820535d6bc6e7eef63babbab3d81cf46968c71 \
|
||||
--hash=sha256:4b599739b93b2cbeded49645ae3c8d1405c29ddfbceac1545c87a3f9580a9e96 \
|
||||
--hash=sha256:4bea7f8ebe90bbd7f0e4a2de42ca6924ba23e3e76418c408ff82f1d46fabd687 \
|
||||
--hash=sha256:4c4fb141a727957c93edfe5c32a26ceb6b5f6461d67146e2d39f51e16170bea8 \
|
||||
--hash=sha256:4c9548dc78002099910abaebc0a72ac58b7d30931869e0351c09b507dff4ece3 \
|
||||
--hash=sha256:4d26f14f041e83dd8edfd61f4cd4fa7285d31798b5bf1f28e70c367ba6c41d61 \
|
||||
--hash=sha256:4f298bdadb8f0b9e5672877f647d1be9373ef5320c9e2f049795e26cad28b6a9 \
|
||||
--hash=sha256:52ec005752a56ae79547a05c0139ca2501a0c866390b6115008456b9f0e7cde1 \
|
||||
--hash=sha256:55261ac0d2941c42f196dd576f543d87a8ee03cd6f5e30dfb4d807b2e3b9121a \
|
||||
--hash=sha256:56490c595a28b1bb27dfc583e816152a9767721ef58b2c03b13f954d2f707420 \
|
||||
--hash=sha256:58d3e12c88e0950bca850ae1f7c256055c097639c2edb9eb123af9807d8b15e4 \
|
||||
--hash=sha256:58d4aa13a59c969dbfdf9e6a9560e242cbfd9e8a8f50c2747714df1a423adf65 \
|
||||
--hash=sha256:59171c6e45bf07d0d5cab3b0bf81d945035530f6873398b3b531c31184d46663 \
|
||||
--hash=sha256:5b6d1386bf0096d26d3a863dc0a487a5b4eb9aa93cf5ba69683d29dde6b9d60f \
|
||||
--hash=sha256:5c0ea61a470e070686aa30892fed79e297d2c8d0ab46b8bcdf027d38c51da591 \
|
||||
--hash=sha256:5c84bec0ab5ae0c64bfe73a7d2adcb5ce73b467523fc27fd6a28ab2aa6cbe35a \
|
||||
--hash=sha256:5ca0555312ae2fe82715cada7fac375530c2f3349e1eaa1bcb33d0283ac79a18 \
|
||||
--hash=sha256:5d8531a6569d025f68e2321e7638fb7978f23db58e5f69f56913837aae03816e \
|
||||
--hash=sha256:5e2d0e146dcb57034f8b97dc58d2d512cb90aba253960ce449f695fec6a82c6f \
|
||||
--hash=sha256:5fc45d653ea8c9a20479167e11d4a0f8cb2fa3470737ab6f9c827532313187b7 \
|
||||
--hash=sha256:6199d5606e2bbf2b096cf64d03f8b6790c91081d5ac866b8e7bb6422738cc60c \
|
||||
--hash=sha256:62b55f6722735a6c472f88361cde6640608773d9443cebdbb51abf436a1fcdd3 \
|
||||
--hash=sha256:687c9ca3035544b113bea2055e180af96fb63c0c476e22a9180f51925186e7b7 \
|
||||
--hash=sha256:6b7430cf5728e68f6c462254009a6ef4086e1bea43cf2f57aa9c55fb4f50ff96 \
|
||||
--hash=sha256:6ba32c4d2abf1d2fe7cf27d280f4cca5664233b0f885549c7761719eb977f486 \
|
||||
--hash=sha256:6c9cdde8becb25a7fde49924511aa2644d6f8081cc8df8e9452724303348d8e3 \
|
||||
--hash=sha256:6df0ec430f9a831772c23ca5a224cba36517a58a84bb32c32bb59a9fa67c47f6 \
|
||||
--hash=sha256:6e2912d4babbc65196ac13c2f53468dc57fb8b9c25ef913e8c59ddf7c6dc0e1b \
|
||||
--hash=sha256:6e5e4d73d588ca5ed09df1b7dcd1b203d1df3c542e3f50d126c947d432b10731 \
|
||||
--hash=sha256:70055ff39b97c99e7ae40ea3e393fb62aa2e44dbd9b29f8d14f42fb0025c3959 \
|
||||
--hash=sha256:706bfd38730a5ac7a365793269a00f4e988178cec121391f4248d84ad8c972e9 \
|
||||
--hash=sha256:7235dc28fc6dd9d832ac7c7bce95367dedb85929f17368a0c2bee1e080b9acbf \
|
||||
--hash=sha256:774d157f112367ff4abd29019f38f023c24e00e56edc7829c20e358a5a913ad8 \
|
||||
--hash=sha256:77efcff2b23071c349402ac1066667a3d011f62398d81408c9b88ad991747c9e \
|
||||
--hash=sha256:789b8982559ae28dad2356519f841655756cdcd96616410590ae0b17454ee64f \
|
||||
--hash=sha256:7ac76cf9afd34929d76eb7fcb63be476a4853d8a96f0dcf2d0db68a0cbdf9885 \
|
||||
--hash=sha256:7c0c10730342b0c9b35dd1d619beb8214e520bd96a1f870f452680b238aab3e0 \
|
||||
--hash=sha256:823f82903d189af463d7df250ef1f7f696f3cee08cc8d91deb565e8d425f6506 \
|
||||
--hash=sha256:838648accb3a7fd9803fd45c87bce8509648eb0c11bc34e216141300977244f2 \
|
||||
--hash=sha256:854066be00447fa8de2ccbbe893e2ffc4b123ef16d897af794c1e18bd4a714b0 \
|
||||
--hash=sha256:85d5855daafc240cc045c026d7a15fd198a09b0fc8ff6f5ecbb5297b509cb11e \
|
||||
--hash=sha256:85de3134b5379856e323ba37c19c9256d39425f7b76a63af52b09fb4664c2e8f \
|
||||
--hash=sha256:87e4f41d375c0b9be2fb5251aee4b8a689169e134535aed81bf085c3b647451e \
|
||||
--hash=sha256:88ca277405c2d3b71c4e1c2ee0e7966e807bcba86a69d11e19ba199d18ae4491 \
|
||||
--hash=sha256:88e85ab89cb822c1e635f51d6d32e488f94e002e70e2f492bdb8b945543f345a \
|
||||
--hash=sha256:8ac8c94b6539074e0f40899301273ac8402b9b3e01c7b7ba269ff30340aaaf20 \
|
||||
--hash=sha256:8fe532b3c966d1fb794e0698e4589d0444017ae77fc0b31edea13c0e35bcc449 \
|
||||
--hash=sha256:9085f87b0e38a2b92b8923059b4e8789fe40d9279712d15dcc670048d77079af \
|
||||
--hash=sha256:90b7481fb62fbe172c558bc6fd1c4c98d82004a54a7551f20e11ac9bf0b8708c \
|
||||
--hash=sha256:92caef967d287a407085d61176fce4012b1dd62daed4eb6d5ceb26d3d2538712 \
|
||||
--hash=sha256:9362dd90aa7dab48c0054a21187791ccf05473f7dba5d92b8033ae62164675e7 \
|
||||
--hash=sha256:94d78ecec2605a8d0398b0f365d5f12a63248438516f5dac536a5eff7337df4a \
|
||||
--hash=sha256:94fbf1c0c6cc0d3d5e50f9a9313a8cdca90dd696d34b381cd1704f8c9e939f20 \
|
||||
--hash=sha256:950f23cb393f85543777b0433f082cddd25b51ab398eac7971146495679efe5f \
|
||||
--hash=sha256:96eefc178f8636b9c760c5829345307fd81cfae9ab1e80997dbddeb0f54ee9a3 \
|
||||
--hash=sha256:96fef3e886d6a9874b14f27fc193fbdc69d5d8035783d86aa4e1cea594e695f9 \
|
||||
--hash=sha256:977cdbd483a9cff38179bea4fd754289a6f2195c7abd414aba85410b3e66cc5e \
|
||||
--hash=sha256:978eab16f55b4ab2c2a745be9a0a840bf8f09a7f227d9c76eb30214d078865a5 \
|
||||
--hash=sha256:994e883d17c559cdfd38c84003c8b27d25424a1077272a17e7cd27bfe0bf57b2 \
|
||||
--hash=sha256:9ac4444d8d4fd4c4bd08bf451ed3167aa9e7ec6cdb41b648794f1d1103652e36 \
|
||||
--hash=sha256:9b5db6052055d34d41230fb78d7c439c23dc536a9896f6cb039e8dd92cfc1263 \
|
||||
--hash=sha256:9d9a0dc7cbe9bec24c3f767c9122c41fe5a1bc43f47cd099d00d393e09769de4 \
|
||||
--hash=sha256:9dbdd9205662134957cf0c324f639bdc5031c0ca056e2369e238db75187c0f11 \
|
||||
--hash=sha256:9eea3ab2597a5e65fe65296e2d6a84570845a6b55532d90333d740d48bbc850a \
|
||||
--hash=sha256:a2028475ba855475b8b4d3cfeb4994269c967aea8b9892dfba907f4263a863a3 \
|
||||
--hash=sha256:a3a370082ce34d0612f421e15fe011c53bb1feff21a26d06ad4fb244dab5a375 \
|
||||
--hash=sha256:a545775cfe815855ea32d7c27731d79da358ef2055b4a25830231b1622dd18aa \
|
||||
--hash=sha256:a5cbd90ecf0fc62e64726917ad083b73001f0563657a87ec3c0b504e277dc90d \
|
||||
--hash=sha256:a6d095662e73e74f0a49988e0593373e243e3a52e27bfeea0a859e88acf4a0f5 \
|
||||
--hash=sha256:a6dac12ff6b846103483683f60c5f8fee205121adc58ffd87e90a90a3af69e99 \
|
||||
--hash=sha256:a951ad59cad9145664a730d3036b40b844e74d2d3683da40111463cd3a83845d \
|
||||
--hash=sha256:aa1099b956fb795e686d073568f6dc002a0bb89765ea6d5b055dd7d9bf1b116c \
|
||||
--hash=sha256:aa2bb0b37202dca27175591f761108b5d34096ade1191ffe4808bdf6b1571488 \
|
||||
--hash=sha256:aae2ee51122d3ae968a3837d97dc24a0aeebb0dea23694422cd172bd30017cd6 \
|
||||
--hash=sha256:ab743e9bc90c1f73552ec33e10e3331315acd2c397b36065b591b0181de533cc \
|
||||
--hash=sha256:ac00177c4831ffa650f8609e4bdddd5fe09c03b1c0c47acece7e6ea20421598b \
|
||||
--hash=sha256:ac13b004224fb341e1e25a1ed5e19d32f57cdb2a403e01f003b46f051a550f6f \
|
||||
--hash=sha256:acaf604462bf330b0d07e7a07c1d6e4adac79e5fb13e9c5140590542cafacc00 \
|
||||
--hash=sha256:ae31a1a1db2ee6cc2942fccaf695c934bc7f3db9f2133a3fef1f367cf1a4ab10 \
|
||||
--hash=sha256:ae4a097991662cd4fff0ddc74e0fe7874f82e00042fa0ea00855645ed0c79598 \
|
||||
--hash=sha256:aea996a6aba25260827c9ea511d1addfde2da9eb686ac961838509086188b7e6 \
|
||||
--hash=sha256:b39b69b347e5e47a3b5b8cfc005c68c1ba347474e3960236c4944a8ecd174962 \
|
||||
--hash=sha256:b54e7e13267d49ffbfe68e25b3cbd774dab38fa37238f71265e91b36146eb21c \
|
||||
--hash=sha256:b9af956078716df40d985fb0dfeb2c2120c5ca92ba4ff4b388acfd01cdc14d08 \
|
||||
--hash=sha256:ba2f37ee79e6338845261a3c5b1784e5d1acdff2c0785b284f1b633033d136ab \
|
||||
--hash=sha256:ba501e667c17d8411f98e67a022d9604ef179aff0e459b7e292c796837c13573 \
|
||||
--hash=sha256:baf3775a2635e5a11fbd5e4e64ee69c7e86875d224a5c72aca4c141064589a90 \
|
||||
--hash=sha256:bb57753e36e4855b8ca375069482250a6246372331a3e4f3407eaebb007443f5 \
|
||||
--hash=sha256:bd6c173f04743d483881bffa1478d5a4624475b8cd1d2194956a75548e191c18 \
|
||||
--hash=sha256:be47f99644b208bff7766314013f9acf57b056b04191d570d68ad14022cf5b1d \
|
||||
--hash=sha256:c010f5581d9c612804cc59fcf7b524b707fbcb72828551237ab545bb5c7034af \
|
||||
--hash=sha256:c1dcc36dcb96abc02236e182d17e0f71430152a6c2c7447421da2d2dc144edea \
|
||||
--hash=sha256:c428c6c31eb5f4277d7f8eccaf767fbd548ddd5ce3c8b4f4cbbfab3d96b5904c \
|
||||
--hash=sha256:c658c50ac0c98cd755a2dd50b7977d3bca7df401dcc47fbdfa87db53ef7d4e8b \
|
||||
--hash=sha256:c71fb0d56c920c269cd3e2e3fe7c610e3f1fdb21a6ce60efa6430ff63676cea6 \
|
||||
--hash=sha256:c7b742bf31c88566b4bb6335a7f393bb322e580b6bb98df7bd0c25e6e3519ce8 \
|
||||
--hash=sha256:cc0329df4caaceb950d2f580b5ac716a377f7059624a0bafaeaf8a218c6ed774 \
|
||||
--hash=sha256:cc5d36d96478aa9c60654bd932525bf32964c62a7281eafdf16d85003a8d6004 \
|
||||
--hash=sha256:ce854f5f478050ade5a238731c4ca985a7d3b3cb53ff600a9b5c3b689b5f0a7a \
|
||||
--hash=sha256:ced3fdd71aaa83ce593746c2edb42b7a59cb4c19c8b5c407781c72e493aae55a \
|
||||
--hash=sha256:cee5dd7c6fb5dd52a0fe2a740f9bc6e3593f5f8b1788bde49de02086f30182b2 \
|
||||
--hash=sha256:cfa1c0cc3a8f9f53f1243a5a99ac36fd003880199383b37672e86ddda9cb07e2 \
|
||||
--hash=sha256:d1ee1e296209fdce05b81b663250eefa02213a2da7b41bf26f7829b8ba3545aa \
|
||||
--hash=sha256:d59b75732e9b6f27388e10c14b0259cc5f2e48c78627d185e6a177b58ad3cffe \
|
||||
--hash=sha256:d63600d620ad0064c3a748b950ac5ea38a80190e5498532efefa4b7b3f1da1f3 \
|
||||
--hash=sha256:dd732602a7009217f658d5863d12d79d373a4de0eebc111094bcdd3bb8e0a6cc \
|
||||
--hash=sha256:e06efa066f7dbadbc84ebc126a97c452a6451dfcf589d89d788484949e1cf795 \
|
||||
--hash=sha256:e199fb99720074809a7720f1c0b4d919eea8b87e88713e0f8f602f7bef543d9d \
|
||||
--hash=sha256:e4b018dc5a0eee4676e38fe84a47a427816c590b93b55d9025274ec4d6ffc2dc \
|
||||
--hash=sha256:e6621fb2a4988d6e53eedc455e5903e2679f3967b8acb3d639f1b63c14a2e893 \
|
||||
--hash=sha256:e71c909f353863b2b89c83de2ebed71ea6d0df8a6ef65a128193c5e650766bef \
|
||||
--hash=sha256:e90251c0c7bdd54a100a0dce3c07b7e637278c93af29dbf78ebb89a58c4bac7d \
|
||||
--hash=sha256:e9fbdce1e47394b09bc9f26ab117dfc8d6491977a11d86f592bb42c779db2fda \
|
||||
--hash=sha256:eb12fb2ba69ffa05f8695f61c69e591dc4b4a12ac3757ac8af8adb259bf56d17 \
|
||||
--hash=sha256:eda059b6bc8bc0812d626fd91a7ce01bf583df0a61296eff390fd94141a34e30 \
|
||||
--hash=sha256:f03ac127268b43ef4fe9e6ab6794a6794b49485a0cc0c1db79876d2f33f75bc7 \
|
||||
--hash=sha256:f298e218441525d3794428b4c8b8fb8662c6d3ea79925d4807ee6b9a96a3bca5 \
|
||||
--hash=sha256:f5542f9b941279d82d41eb0aa9f98eba36fe4df5c7086c651df7944935b37182 \
|
||||
--hash=sha256:f6f7deae3feb4edfa2efaf7c574fe88cbf055038a6abdb40188e4fff66d5699f \
|
||||
--hash=sha256:f9b1e28d0e8dbfa858abdba91d6b547beaf2df1a59bec6da6faae7b96a4991a9 \
|
||||
--hash=sha256:f9f8405c2c758532c74fed975dbee57be1f31a6e865c031870c79a6ed3212ada \
|
||||
--hash=sha256:fa48b1b63d639f9483e0633e092f5851e2348c352f1f9bb6c8182f87884ef876 \
|
||||
--hash=sha256:fb78f6e7fcd8ad785d28cd577168bc1aaee827b25bb8755638f694794ea98f0a \
|
||||
--hash=sha256:fbc597639158fd7c14d55e808718848319540f51b0e6746e3eefa59723a4a348 \
|
||||
--hash=sha256:fce8cbd4997efeb450bd298b54f755dcdff18d496f7a5ddbb4867c6d7c88fdc3 \
|
||||
--hash=sha256:fd0350afdc3aabd5576f60ea109228bd5538139713c7b094c5cd27c73a98bc6f \
|
||||
--hash=sha256:fd0a274c0e5f9a21565cd9d3dd749b61f96b7aa1e20a93aa1ba4029518f2e5c0 \
|
||||
--hash=sha256:fdb8a068947befafba9952162645dc2fecaeb400e64584829ed5e9b2fbe21a7f
|
||||
# via requests
|
||||
idna==3.20 \
|
||||
--hash=sha256:ab7ae7122974553370f0bdb919e1a960b2cd1bc1ef0276416d896db81c14582c
|
||||
# via requests
|
||||
pyyaml==6.0.3 \
|
||||
--hash=sha256:00c4bdeba853cc34e7dd471f16b4114f4162dc03e6b7afcc2128711f0eca823c \
|
||||
--hash=sha256:0150219816b6a1fa26fb4699fb7daa9caf09eb1999f3b70fb6e786805e80375a \
|
||||
--hash=sha256:02893d100e99e03eda1c8fd5c441d8c60103fd175728e23e431db1b589cf5ab3 \
|
||||
--hash=sha256:02ea2dfa234451bbb8772601d7b8e426c2bfa197136796224e50e35a78777956 \
|
||||
--hash=sha256:0f29edc409a6392443abf94b9cf89ce99889a1dd5376d94316ae5145dfedd5d6 \
|
||||
--hash=sha256:10892704fc220243f5305762e276552a0395f7beb4dbf9b14ec8fd43b57f126c \
|
||||
--hash=sha256:16249ee61e95f858e83976573de0f5b2893b3677ba71c9dd36b9cf8be9ac6d65 \
|
||||
--hash=sha256:1d37d57ad971609cf3c53ba6a7e365e40660e3be0e5175fa9f2365a379d6095a \
|
||||
--hash=sha256:1ebe39cb5fc479422b83de611d14e2c0d3bb2a18bbcb01f229ab3cfbd8fee7a0 \
|
||||
--hash=sha256:214ed4befebe12df36bcc8bc2b64b396ca31be9304b8f59e25c11cf94a4c033b \
|
||||
--hash=sha256:2283a07e2c21a2aa78d9c4442724ec1eb15f5e42a723b99cb3d822d48f5f7ad1 \
|
||||
--hash=sha256:22ba7cfcad58ef3ecddc7ed1db3409af68d023b7f940da23c6c2a1890976eda6 \
|
||||
--hash=sha256:27c0abcb4a5dac13684a37f76e701e054692a9b2d3064b70f5e4eb54810553d7 \
|
||||
--hash=sha256:28c8d926f98f432f88adc23edf2e6d4921ac26fb084b028c733d01868d19007e \
|
||||
--hash=sha256:2e71d11abed7344e42a8849600193d15b6def118602c4c176f748e4583246007 \
|
||||
--hash=sha256:34d5fcd24b8445fadc33f9cf348c1047101756fd760b4dacb5c3e99755703310 \
|
||||
--hash=sha256:37503bfbfc9d2c40b344d06b2199cf0e96e97957ab1c1b546fd4f87e53e5d3e4 \
|
||||
--hash=sha256:3c5677e12444c15717b902a5798264fa7909e41153cdf9ef7ad571b704a63dd9 \
|
||||
--hash=sha256:3ff07ec89bae51176c0549bc4c63aa6202991da2d9a6129d7aef7f1407d3f295 \
|
||||
--hash=sha256:41715c910c881bc081f1e8872880d3c650acf13dfa8214bad49ed4cede7c34ea \
|
||||
--hash=sha256:418cf3f2111bc80e0933b2cd8cd04f286338bb88bdc7bc8e6dd775ebde60b5e0 \
|
||||
--hash=sha256:44edc647873928551a01e7a563d7452ccdebee747728c1080d881d68af7b997e \
|
||||
--hash=sha256:4a2e8cebe2ff6ab7d1050ecd59c25d4c8bd7e6f400f5f82b96557ac0abafd0ac \
|
||||
--hash=sha256:4ad1906908f2f5ae4e5a8ddfce73c320c2a1429ec52eafd27138b7f1cbe341c9 \
|
||||
--hash=sha256:501a031947e3a9025ed4405a168e6ef5ae3126c59f90ce0cd6f2bfc477be31b7 \
|
||||
--hash=sha256:5190d403f121660ce8d1d2c1bb2ef1bd05b5f68533fc5c2ea899bd15f4399b35 \
|
||||
--hash=sha256:5498cd1645aa724a7c71c8f378eb29ebe23da2fc0d7a08071d89469bf1d2defb \
|
||||
--hash=sha256:5cf4e27da7e3fbed4d6c3d8e797387aaad68102272f8f9752883bc32d61cb87b \
|
||||
--hash=sha256:5e0b74767e5f8c593e8c9b5912019159ed0533c70051e9cce3e8b6aa699fcd69 \
|
||||
--hash=sha256:5ed875a24292240029e4483f9d4a4b8a1ae08843b9c54f43fcc11e404532a8a5 \
|
||||
--hash=sha256:5fcd34e47f6e0b794d17de1b4ff496c00986e1c83f7ab2fb8fcfe9616ff7477b \
|
||||
--hash=sha256:5fdec68f91a0c6739b380c83b951e2c72ac0197ace422360e6d5a959d8d97b2c \
|
||||
--hash=sha256:6344df0d5755a2c9a276d4473ae6b90647e216ab4757f8426893b5dd2ac3f369 \
|
||||
--hash=sha256:64386e5e707d03a7e172c0701abfb7e10f0fb753ee1d773128192742712a98fd \
|
||||
--hash=sha256:652cb6edd41e718550aad172851962662ff2681490a8a711af6a4d288dd96824 \
|
||||
--hash=sha256:66291b10affd76d76f54fad28e22e51719ef9ba22b29e1d7d03d6777a9174198 \
|
||||
--hash=sha256:66e1674c3ef6f541c35191caae2d429b967b99e02040f5ba928632d9a7f0f065 \
|
||||
--hash=sha256:6adc77889b628398debc7b65c073bcb99c4a0237b248cacaf3fe8a557563ef6c \
|
||||
--hash=sha256:79005a0d97d5ddabfeeea4cf676af11e647e41d81c9a7722a193022accdb6b7c \
|
||||
--hash=sha256:7c6610def4f163542a622a73fb39f534f8c101d690126992300bf3207eab9764 \
|
||||
--hash=sha256:7f047e29dcae44602496db43be01ad42fc6f1cc0d8cd6c83d342306c32270196 \
|
||||
--hash=sha256:8098f252adfa6c80ab48096053f512f2321f0b998f98150cea9bd23d83e1467b \
|
||||
--hash=sha256:850774a7879607d3a6f50d36d04f00ee69e7fc816450e5f7e58d7f17f1ae5c00 \
|
||||
--hash=sha256:8d1fab6bb153a416f9aeb4b8763bc0f22a5586065f86f7664fc23339fc1c1fac \
|
||||
--hash=sha256:8da9669d359f02c0b91ccc01cac4a67f16afec0dac22c2ad09f46bee0697eba8 \
|
||||
--hash=sha256:8dc52c23056b9ddd46818a57b78404882310fb473d63f17b07d5c40421e47f8e \
|
||||
--hash=sha256:9149cad251584d5fb4981be1ecde53a1ca46c891a79788c0df828d2f166bda28 \
|
||||
--hash=sha256:93dda82c9c22deb0a405ea4dc5f2d0cda384168e466364dec6255b293923b2f3 \
|
||||
--hash=sha256:96b533f0e99f6579b3d4d4995707cf36df9100d67e0c8303a0c55b27b5f99bc5 \
|
||||
--hash=sha256:9c57bb8c96f6d1808c030b1687b9b5fb476abaa47f0db9c0101f5e9f394e97f4 \
|
||||
--hash=sha256:9c7708761fccb9397fe64bbc0395abcae8c4bf7b0eac081e12b809bf47700d0b \
|
||||
--hash=sha256:9f3bfb4965eb874431221a3ff3fdcddc7e74e3b07799e0e84ca4a0f867d449bf \
|
||||
--hash=sha256:a33284e20b78bd4a18c8c2282d549d10bc8408a2a7ff57653c0cf0b9be0afce5 \
|
||||
--hash=sha256:a80cb027f6b349846a3bf6d73b5e95e782175e52f22108cfa17876aaeff93702 \
|
||||
--hash=sha256:b30236e45cf30d2b8e7b3e85881719e98507abed1011bf463a8fa23e9c3e98a8 \
|
||||
--hash=sha256:b3bc83488de33889877a0f2543ade9f70c67d66d9ebb4ac959502e12de895788 \
|
||||
--hash=sha256:b865addae83924361678b652338317d1bd7e79b1f4596f96b96c77a5a34b34da \
|
||||
--hash=sha256:b8bb0864c5a28024fac8a632c443c87c5aa6f215c0b126c449ae1a150412f31d \
|
||||
--hash=sha256:ba1cc08a7ccde2d2ec775841541641e4548226580ab850948cbfda66a1befcdc \
|
||||
--hash=sha256:bdb2c67c6c1390b63c6ff89f210c8fd09d9a1217a465701eac7316313c915e4c \
|
||||
--hash=sha256:c1ff362665ae507275af2853520967820d9124984e0f7466736aea23d8611fba \
|
||||
--hash=sha256:c2514fceb77bc5e7a2f7adfaa1feb2fb311607c9cb518dbc378688ec73d8292f \
|
||||
--hash=sha256:c3355370a2c156cffb25e876646f149d5d68f5e0a3ce86a5084dd0b64a994917 \
|
||||
--hash=sha256:c458b6d084f9b935061bc36216e8a69a7e293a2f1e68bf956dcd9e6cbcd143f5 \
|
||||
--hash=sha256:d0eae10f8159e8fdad514efdc92d74fd8d682c933a6dd088030f3834bc8e6b26 \
|
||||
--hash=sha256:ebc55a14a21cb14062aa4162f906cd962b28e2e9ea38f9b4391244cd8de4ae0b \
|
||||
--hash=sha256:eda16858a3cab07b80edaf74336ece1f986ba330fdb8ee0d6c0d68fe82bc96be \
|
||||
--hash=sha256:ee2922902c45ae8ccada2c5b501ab86c36525b883eff4255313a253a3160861c \
|
||||
--hash=sha256:efd7b85f94a6f21e4932043973a7ba2613b059c4a000551892ac9f1d11f5baf3 \
|
||||
--hash=sha256:f7057c9a337546edc7973c0d3ba84ddcdf0daa14533c2065749c9075001090e6 \
|
||||
--hash=sha256:fa160448684b4e94d80416c0fa4aac48967a969efe22931448d853ada8baf926 \
|
||||
--hash=sha256:fc09d0aa354569bc501d4e787133afc08552722d3ab34836a80547331bb5d4a0
|
||||
# via -r docker/requirements-worker.in
|
||||
requests==2.34.2 \
|
||||
--hash=sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0
|
||||
# via -r docker/requirements-worker.in
|
||||
urllib3==2.8.0 \
|
||||
--hash=sha256:0cf3cae568d36aa9576b28dfb35f11328f1cb974ca7647d9475ebb86c75ac6e3
|
||||
# via requests
|
||||
zstandard==0.23.0 \
|
||||
--hash=sha256:034b88913ecc1b097f528e42b539453fa82c3557e414b3de9d5632c80439a473 \
|
||||
--hash=sha256:0a7f0804bb3799414af278e9ad51be25edf67f78f916e08afdb983e74161b916 \
|
||||
--hash=sha256:11e3bf3c924853a2d5835b24f03eeba7fc9b07d8ca499e247e06ff5676461a15 \
|
||||
--hash=sha256:12a289832e520c6bd4dcaad68e944b86da3bad0d339ef7989fb7e88f92e96072 \
|
||||
--hash=sha256:1516c8c37d3a053b01c1c15b182f3b5f5eef19ced9b930b684a73bad121addf4 \
|
||||
--hash=sha256:157e89ceb4054029a289fb504c98c6a9fe8010f1680de0201b3eb5dc20aa6d9e \
|
||||
--hash=sha256:1bfe8de1da6d104f15a60d4a8a768288f66aa953bbe00d027398b93fb9680b26 \
|
||||
--hash=sha256:1e172f57cd78c20f13a3415cc8dfe24bf388614324d25539146594c16d78fcc8 \
|
||||
--hash=sha256:1fd7e0f1cfb70eb2f95a19b472ee7ad6d9a0a992ec0ae53286870c104ca939e5 \
|
||||
--hash=sha256:203d236f4c94cd8379d1ea61db2fce20730b4c38d7f1c34506a31b34edc87bdd \
|
||||
--hash=sha256:27d3ef2252d2e62476389ca8f9b0cf2bbafb082a3b6bfe9d90cbcbb5529ecf7c \
|
||||
--hash=sha256:29a2bc7c1b09b0af938b7a8343174b987ae021705acabcbae560166567f5a8db \
|
||||
--hash=sha256:2ef230a8fd217a2015bc91b74f6b3b7d6522ba48be29ad4ea0ca3a3775bf7dd5 \
|
||||
--hash=sha256:2ef3775758346d9ac6214123887d25c7061c92afe1f2b354f9388e9e4d48acfc \
|
||||
--hash=sha256:2f146f50723defec2975fb7e388ae3a024eb7151542d1599527ec2aa9cacb152 \
|
||||
--hash=sha256:2fb4535137de7e244c230e24f9d1ec194f61721c86ebea04e1581d9d06ea1269 \
|
||||
--hash=sha256:32ba3b5ccde2d581b1e6aa952c836a6291e8435d788f656fe5976445865ae045 \
|
||||
--hash=sha256:34895a41273ad33347b2fc70e1bff4240556de3c46c6ea430a7ed91f9042aa4e \
|
||||
--hash=sha256:379b378ae694ba78cef921581ebd420c938936a153ded602c4fea612b7eaa90d \
|
||||
--hash=sha256:38302b78a850ff82656beaddeb0bb989a0322a8bbb1bf1ab10c17506681d772a \
|
||||
--hash=sha256:3aa014d55c3af933c1315eb4bb06dd0459661cc0b15cd61077afa6489bec63bb \
|
||||
--hash=sha256:4051e406288b8cdbb993798b9a45c59a4896b6ecee2f875424ec10276a895740 \
|
||||
--hash=sha256:40b33d93c6eddf02d2c19f5773196068d875c41ca25730e8288e9b672897c105 \
|
||||
--hash=sha256:43da0f0092281bf501f9c5f6f3b4c975a8a0ea82de49ba3f7100e64d422a1274 \
|
||||
--hash=sha256:445e4cb5048b04e90ce96a79b4b63140e3f4ab5f662321975679b5f6360b90e2 \
|
||||
--hash=sha256:48ef6a43b1846f6025dde6ed9fee0c24e1149c1c25f7fb0a0585572b2f3adc58 \
|
||||
--hash=sha256:50a80baba0285386f97ea36239855f6020ce452456605f262b2d33ac35c7770b \
|
||||
--hash=sha256:519fbf169dfac1222a76ba8861ef4ac7f0530c35dd79ba5727014613f91613d4 \
|
||||
--hash=sha256:53dd9d5e3d29f95acd5de6802e909ada8d8d8cfa37a3ac64836f3bc4bc5512db \
|
||||
--hash=sha256:53ea7cdc96c6eb56e76bb06894bcfb5dfa93b7adcf59d61c6b92674e24e2dd5e \
|
||||
--hash=sha256:576856e8594e6649aee06ddbfc738fec6a834f7c85bf7cadd1c53d4a58186ef9 \
|
||||
--hash=sha256:59556bf80a7094d0cfb9f5e50bb2db27fefb75d5138bb16fb052b61b0e0eeeb0 \
|
||||
--hash=sha256:5d41d5e025f1e0bccae4928981e71b2334c60f580bdc8345f824e7c0a4c2a813 \
|
||||
--hash=sha256:61062387ad820c654b6a6b5f0b94484fa19515e0c5116faf29f41a6bc91ded6e \
|
||||
--hash=sha256:61f89436cbfede4bc4e91b4397eaa3e2108ebe96d05e93d6ccc95ab5714be512 \
|
||||
--hash=sha256:62136da96a973bd2557f06ddd4e8e807f9e13cbb0bfb9cc06cfe6d98ea90dfe0 \
|
||||
--hash=sha256:64585e1dba664dc67c7cdabd56c1e5685233fbb1fc1966cfba2a340ec0dfff7b \
|
||||
--hash=sha256:65308f4b4890aa12d9b6ad9f2844b7ee42c7f7a4fd3390425b242ffc57498f48 \
|
||||
--hash=sha256:66b689c107857eceabf2cf3d3fc699c3c0fe8ccd18df2219d978c0283e4c508a \
|
||||
--hash=sha256:6a41c120c3dbc0d81a8e8adc73312d668cd34acd7725f036992b1b72d22c1772 \
|
||||
--hash=sha256:6f77fa49079891a4aab203d0b1744acc85577ed16d767b52fc089d83faf8d8ed \
|
||||
--hash=sha256:72c68dda124a1a138340fb62fa21b9bf4848437d9ca60bd35db36f2d3345f373 \
|
||||
--hash=sha256:752bf8a74412b9892f4e5b58f2f890a039f57037f52c89a740757ebd807f33ea \
|
||||
--hash=sha256:76e79bc28a65f467e0409098fa2c4376931fd3207fbeb6b956c7c476d53746dd \
|
||||
--hash=sha256:774d45b1fac1461f48698a9d4b5fa19a69d47ece02fa469825b442263f04021f \
|
||||
--hash=sha256:77da4c6bfa20dd5ea25cbf12c76f181a8e8cd7ea231c673828d0386b1740b8dc \
|
||||
--hash=sha256:77ea385f7dd5b5676d7fd943292ffa18fbf5c72ba98f7d09fc1fb9e819b34c23 \
|
||||
--hash=sha256:80080816b4f52a9d886e67f1f96912891074903238fe54f2de8b786f86baded2 \
|
||||
--hash=sha256:80a539906390591dd39ebb8d773771dc4db82ace6372c4d41e2d293f8e32b8db \
|
||||
--hash=sha256:82d17e94d735c99621bf8ebf9995f870a6b3e6d14543b99e201ae046dfe7de70 \
|
||||
--hash=sha256:837bb6764be6919963ef41235fd56a6486b132ea64afe5fafb4cb279ac44f259 \
|
||||
--hash=sha256:84433dddea68571a6d6bd4fbf8ff398236031149116a7fff6f777ff95cad3df9 \
|
||||
--hash=sha256:8c24f21fa2af4bb9f2c492a86fe0c34e6d2c63812a839590edaf177b7398f700 \
|
||||
--hash=sha256:8ed7d27cb56b3e058d3cf684d7200703bcae623e1dcc06ed1e18ecda39fee003 \
|
||||
--hash=sha256:9206649ec587e6b02bd124fb7799b86cddec350f6f6c14bc82a2b70183e708ba \
|
||||
--hash=sha256:983b6efd649723474f29ed42e1467f90a35a74793437d0bc64a5bf482bedfa0a \
|
||||
--hash=sha256:98da17ce9cbf3bfe4617e836d561e433f871129e3a7ac16d6ef4c680f13a839c \
|
||||
--hash=sha256:9c236e635582742fee16603042553d276cca506e824fa2e6489db04039521e90 \
|
||||
--hash=sha256:9da6bc32faac9a293ddfdcb9108d4b20416219461e4ec64dfea8383cac186690 \
|
||||
--hash=sha256:a05e6d6218461eb1b4771d973728f0133b2a4613a6779995df557f70794fd60f \
|
||||
--hash=sha256:a0817825b900fcd43ac5d05b8b3079937073d2b1ff9cf89427590718b70dd840 \
|
||||
--hash=sha256:a4ae99c57668ca1e78597d8b06d5af837f377f340f4cce993b551b2d7731778d \
|
||||
--hash=sha256:a8c86881813a78a6f4508ef9daf9d4995b8ac2d147dcb1a450448941398091c9 \
|
||||
--hash=sha256:a8fffdbd9d1408006baaf02f1068d7dd1f016c6bcb7538682622c556e7b68e35 \
|
||||
--hash=sha256:a9b07268d0c3ca5c170a385a0ab9fb7fdd9f5fd866be004c4ea39e44edce47dd \
|
||||
--hash=sha256:ab19a2d91963ed9e42b4e8d77cd847ae8381576585bad79dbd0a8837a9f6620a \
|
||||
--hash=sha256:ac184f87ff521f4840e6ea0b10c0ec90c6b1dcd0bad2f1e4a9a1b4fa177982ea \
|
||||
--hash=sha256:b0e166f698c5a3e914947388c162be2583e0c638a4703fc6a543e23a88dea3c1 \
|
||||
--hash=sha256:b2170c7e0367dde86a2647ed5b6f57394ea7f53545746104c6b09fc1f4223573 \
|
||||
--hash=sha256:b4567955a6bc1b20e9c31612e615af6b53733491aeaa19a6b3b37f3b65477094 \
|
||||
--hash=sha256:b69bb4f51daf461b15e7b3db033160937d3ff88303a7bc808c67bbc1eaf98c78 \
|
||||
--hash=sha256:b8c0bd73aeac689beacd4e7667d48c299f61b959475cdbb91e7d3d88d27c56b9 \
|
||||
--hash=sha256:be9b5b8659dff1f913039c2feee1aca499cfbc19e98fa12bc85e037c17ec6ca5 \
|
||||
--hash=sha256:bf0a05b6059c0528477fba9054d09179beb63744355cab9f38059548fedd46a9 \
|
||||
--hash=sha256:c16842b846a8d2a145223f520b7e18b57c8f476924bda92aeee3a88d11cfc391 \
|
||||
--hash=sha256:c363b53e257246a954ebc7c488304b5592b9c53fbe74d03bc1c64dda153fb847 \
|
||||
--hash=sha256:c7c517d74bea1a6afd39aa612fa025e6b8011982a0897768a2f7c8ab4ebb78a2 \
|
||||
--hash=sha256:d20fd853fbb5807c8e84c136c278827b6167ded66c72ec6f9a14b863d809211c \
|
||||
--hash=sha256:d2240ddc86b74966c34554c49d00eaafa8200a18d3a5b6ffbf7da63b11d74ee2 \
|
||||
--hash=sha256:d477ed829077cd945b01fc3115edd132c47e6540ddcd96ca169facff28173057 \
|
||||
--hash=sha256:d50d31bfedd53a928fed6707b15a8dbeef011bb6366297cc435accc888b27c20 \
|
||||
--hash=sha256:dc1d33abb8a0d754ea4763bad944fd965d3d95b5baef6b121c0c9013eaf1907d \
|
||||
--hash=sha256:dc5d1a49d3f8262be192589a4b72f0d03b72dcf46c51ad5852a4fdc67be7b9e4 \
|
||||
--hash=sha256:e2d1a054f8f0a191004675755448d12be47fa9bebbcffa3cdf01db19f2d30a54 \
|
||||
--hash=sha256:e7792606d606c8df5277c32ccb58f29b9b8603bf83b48639b7aedf6df4fe8171 \
|
||||
--hash=sha256:ed1708dbf4d2e3a1c5c69110ba2b4eb6678262028afd6c6fbcc5a8dac9cda68e \
|
||||
--hash=sha256:f2d4380bf5f62daabd7b751ea2339c1a21d1c9463f1feb7fc2bdcea2c29c3160 \
|
||||
--hash=sha256:f3513916e8c645d0610815c257cbfd3242adfd5c4cfa78be514e5a3ebb42a41b \
|
||||
--hash=sha256:f8346bfa098532bc1fb6c7ef06783e969d87a99dd1d2a5a18a892c1d7a643c58 \
|
||||
--hash=sha256:f83fa6cae3fff8e98691248c9320356971b59678a17f20656a9e59cd32cee6d8 \
|
||||
--hash=sha256:fa6ce8b52c5987b3e34d5674b0ab529a4602b632ebab0a93b07bfb4dfc8f8a33 \
|
||||
--hash=sha256:fb2b1ecfef1e67897d336de3a0e3f52478182d6a47eda86cbd42504c5cbd009a \
|
||||
--hash=sha256:fc9ca1c9718cb3b06634c7c8dec57d24e9438b2aa9a0f02b8bb36bf478538880 \
|
||||
--hash=sha256:fd30d9c67d13d891f2360b2a120186729c111238ac63b43dbd37a5a40670b8ca \
|
||||
--hash=sha256:fd7699e8fd9969f455ef2926221e0233f81a2542921471382e77a9e2f2b57f4b \
|
||||
--hash=sha256:fe3b385d996ee0822fd46528d9f0443b880d4d05528fd26a9119a54ec3f91c69
|
||||
# via -r docker/requirements-worker.in
|
||||
@@ -0,0 +1,6 @@
|
||||
# Keep the runtime union tied to both application manifests, including the dashboard.
|
||||
-r ../app/requirements.txt
|
||||
-r ../app/requirements-keycheckers.txt
|
||||
|
||||
# One hash-locked environment is shared by the runtime and test targets.
|
||||
pytest==8.4.2
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,196 @@
|
||||
"""Pure Compose regressions: python -I -S -B docker/test_verify.py -v."""
|
||||
|
||||
from copy import deepcopy
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
import unittest
|
||||
|
||||
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
'truf_verify', Path(__file__).with_name('verify.py'))
|
||||
verify = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(verify)
|
||||
|
||||
|
||||
class ValidateComposeTests(unittest.TestCase):
|
||||
def setUp(self):
|
||||
# Only project/images are needed; skip all host setup and never run main.
|
||||
self.verifier = object.__new__(verify.Verifier)
|
||||
self.verifier.project = 'truf-worker-test-' + 'a' * 32
|
||||
self.verifier.images = {'runtime': 'sha256:' + '1' * 64,
|
||||
'test': 'sha256:' + '2' * 64}
|
||||
common = {
|
||||
'pull_policy': 'never', 'read_only': True, 'network_mode': 'none',
|
||||
'environment': dict(verify.PROXY_ENV), 'init': False,
|
||||
'user': '10001:10001', 'cap_drop': ['ALL'],
|
||||
'security_opt': ['no-new-privileges:true'], 'restart': 'no',
|
||||
'cpus': 2, 'mem_limit': 6442450944, 'pids_limit': 512,
|
||||
'shm_size': 268435456, 'stop_signal': 'SIGTERM',
|
||||
'stop_grace_period': '10m0s',
|
||||
'logging': {'driver': 'json-file',
|
||||
'options': {'max-size': '16m', 'max-file': '4'}},
|
||||
'tmpfs': [target + ':' + options for target, options in verify.TMPFS.items()],
|
||||
}
|
||||
data = {'type': 'volume', 'source': 'data', 'target': '/data', 'volume': {}}
|
||||
tools = {'type': 'volume', 'source': 'tools', 'target': '/opt/truf/tests',
|
||||
'read_only': True, 'volume': {'nocopy': True}}
|
||||
services = {name: deepcopy(common)
|
||||
for name in ('tools', 'provision', 'prepare', 'runtime', 'stopped')}
|
||||
for name, service in services.items():
|
||||
service['image'] = self.verifier.images['test' if name == 'tools' else 'runtime']
|
||||
service['volumes'] = [deepcopy(data)]
|
||||
# Compose 5 omits false nocopy/read_only; byte sizes are normalized integers.
|
||||
services['tools'].update(
|
||||
volumes=[{'type': 'volume', 'source': 'tools',
|
||||
'target': '/opt/truf/tests', 'volume': {}}],
|
||||
entrypoint=[*verify.PYTHON, '-c'], command=['pass'])
|
||||
services['provision'].update(
|
||||
user='0:0', cap_add=['CHOWN', 'DAC_OVERRIDE', 'FOWNER'],
|
||||
entrypoint=None, command=['provision'])
|
||||
services['prepare'].update(
|
||||
volumes=[deepcopy(data), deepcopy(tools)],
|
||||
entrypoint=[*verify.PYTHON, verify.DRIVER],
|
||||
command=['prepare', '--config', verify.CONFIG])
|
||||
services['runtime'].update(
|
||||
volumes=[deepcopy(data), deepcopy(tools)], entrypoint=None,
|
||||
command=['run', '--config', verify.CONFIG],
|
||||
healthcheck={'test': list(verify.HEALTH), 'interval': '5s',
|
||||
'timeout': '15s', 'start_period': '4m0s', 'retries': 3})
|
||||
services['stopped'].update(
|
||||
volumes=[dict(data, read_only=True, volume={'nocopy': True})],
|
||||
entrypoint=[*verify.PYTHON, '-c'], command=['pass'])
|
||||
self.value = {
|
||||
'name': self.verifier.project, 'services': services,
|
||||
'volumes': {name: {'name': self.verifier.project + '_' + name, 'driver': 'local'}
|
||||
for name in ('data', 'tools')},
|
||||
}
|
||||
|
||||
def test_normalized_fixture(self):
|
||||
self.verifier.validate_compose(self.value)
|
||||
|
||||
def test_rejects_production_project_and_volume_names(self):
|
||||
value = deepcopy(self.value)
|
||||
value['name'] = 'truf-docker'
|
||||
with self.assertRaisesRegex(verify.Failure, '^worker_test_project_guard$'):
|
||||
self.verifier.validate_compose(value)
|
||||
value = deepcopy(self.value)
|
||||
value['volumes']['data']['name'] = 'truf-docker_data'
|
||||
with self.assertRaisesRegex(verify.Failure, '^production_volume_forbidden$'):
|
||||
self.verifier.validate_compose(value)
|
||||
|
||||
def test_rejects_bind_mounts_and_published_ports(self):
|
||||
value = deepcopy(self.value)
|
||||
value['services']['runtime']['volumes'][0] = {
|
||||
'type': 'bind', 'source': r'D:\truf-docker', 'target': '/data',
|
||||
}
|
||||
with self.assertRaisesRegex(verify.Failure, '^bind_mount_forbidden$'):
|
||||
self.verifier.validate_compose(value)
|
||||
value = deepcopy(self.value)
|
||||
value['services']['runtime']['ports'] = [{'target': 5432, 'published': '5432'}]
|
||||
with self.assertRaisesRegex(verify.Failure, '^published_port_contract$'):
|
||||
self.verifier.validate_compose(value)
|
||||
|
||||
def test_rejects_shared_images_and_nonisolated_networks(self):
|
||||
value = deepcopy(self.value)
|
||||
value['services']['runtime']['image'] = 'truf-local:runtime'
|
||||
with self.assertRaisesRegex(verify.Failure, '^worker_test_image_reference_guard$'):
|
||||
self.verifier.validate_compose(value)
|
||||
value = deepcopy(self.value)
|
||||
value['services']['runtime']['network_mode'] = 'bridge'
|
||||
with self.assertRaisesRegex(verify.Failure, '^internal_network_contract$'):
|
||||
self.verifier.validate_compose(value)
|
||||
|
||||
def test_seed_accepts_false_or_omitted_nocopy(self):
|
||||
for volume in (None, {}, {'nocopy': False}):
|
||||
with self.subTest(volume=volume):
|
||||
value = deepcopy(self.value)
|
||||
mount = value['services']['tools']['volumes'][0]
|
||||
if volume is None:
|
||||
del mount['volume']
|
||||
else:
|
||||
mount['volume'] = volume
|
||||
self.verifier.validate_compose(value)
|
||||
|
||||
def test_seed_rejects_nocopy_other_than_false(self):
|
||||
for nocopy in (True, None, 0, 'false'):
|
||||
with self.subTest(nocopy=nocopy):
|
||||
value = deepcopy(self.value)
|
||||
value['services']['tools']['volumes'][0]['volume']['nocopy'] = nocopy
|
||||
with self.assertRaisesRegex(verify.Failure, '^tools_copy_up_contract$'):
|
||||
self.verifier.validate_compose(value)
|
||||
|
||||
def test_consumers_require_explicit_true_nocopy(self):
|
||||
for name in ('prepare', 'runtime'):
|
||||
for volume in (None, {}, {'nocopy': False}, {'nocopy': 1}, {'nocopy': 'true'}):
|
||||
with self.subTest(service=name, volume=volume):
|
||||
value = deepcopy(self.value)
|
||||
mount = value['services'][name]['volumes'][1]
|
||||
if volume is None:
|
||||
del mount['volume']
|
||||
else:
|
||||
mount['volume'] = volume
|
||||
with self.assertRaisesRegex(verify.Failure, '^tools_copy_up_contract$'):
|
||||
self.verifier.validate_compose(value)
|
||||
|
||||
def test_runtime_and_provision_inherit_entrypoint(self):
|
||||
for name in ('runtime', 'provision'):
|
||||
for omitted in (False, True):
|
||||
with self.subTest(service=name, omitted=omitted):
|
||||
value = deepcopy(self.value)
|
||||
if omitted:
|
||||
del value['services'][name]['entrypoint']
|
||||
self.verifier.validate_compose(value)
|
||||
|
||||
def test_runtime_and_provision_reject_entrypoint_overrides(self):
|
||||
for name, guard in (('runtime', 'production_entrypoint_contract'),
|
||||
('provision', 'prepare_command_contract')):
|
||||
for entrypoint in ([], ['/bin/sh', '-c'], ''):
|
||||
with self.subTest(service=name, entrypoint=entrypoint):
|
||||
value = deepcopy(self.value)
|
||||
value['services'][name]['entrypoint'] = entrypoint
|
||||
with self.assertRaisesRegex(verify.Failure, '^' + guard + '$'):
|
||||
self.verifier.validate_compose(value)
|
||||
|
||||
def test_foreign_snapshot_excludes_only_revalidated_owned_identities(self):
|
||||
owned_container = 'a' * 64
|
||||
foreign_container = 'b' * 64
|
||||
owned_volume = {'name': 'owned', 'identity': 'captured'}
|
||||
foreign_volume = {'name': 'foreign', 'identity': 'stable'}
|
||||
current = {'owned': owned_volume, 'foreign': foreign_volume}
|
||||
verifier = object.__new__(verify.Verifier)
|
||||
verifier.owned_containers = {owned_container: {'id': owned_container}}
|
||||
verifier.owned_volumes = {'owned': owned_volume}
|
||||
verifier.metadata_names = lambda kind: (
|
||||
{owned_container, foreign_container} if kind == 'container'
|
||||
else {'owned', 'foreign'}
|
||||
)
|
||||
verifier.container_metadata = lambda identifier: {'id': identifier}
|
||||
verifier.volume_metadata = lambda name: current[name]
|
||||
inspected = []
|
||||
verifier.inspect = lambda identifier, label: inspected.append((identifier, label))
|
||||
|
||||
snapshot = verifier.metadata_snapshot(exclude_owned=True)
|
||||
|
||||
self.assertEqual(snapshot, {
|
||||
'containers': {foreign_container: {'id': foreign_container}},
|
||||
'volumes': {'foreign': foreign_volume},
|
||||
})
|
||||
self.assertEqual(inspected, [(owned_container, 'foreign_owned_exclusion_guard')])
|
||||
|
||||
current['owned'] = {'name': 'owned', 'identity': 'replaced'}
|
||||
self.assertIn('owned', verifier.metadata_snapshot(exclude_owned=True)['volumes'])
|
||||
|
||||
def test_cleanup_uses_exact_resource_operations_and_foreign_guard(self):
|
||||
source = Path(verify.__file__).read_text(encoding='ascii')
|
||||
self.assertNotIn("compose_command('down'", source)
|
||||
self.assertNotIn("'--force-recreate'", source)
|
||||
self.assertIn("'cleanup_container_remove_guard'", source)
|
||||
self.assertIn("'cleanup_volume_identity_changed'", source)
|
||||
self.assertIn('self.assert_foreign_unchanged()', source)
|
||||
self.assertIn("'failure_stop_ownership_guard'", source)
|
||||
self.assertIn("'stage': label, 'class': failure_class", source)
|
||||
self.assertIn("'exit_code': exit_code", source)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,909 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the offline worker E2E against dedicated, already-built local images.
|
||||
|
||||
Run from Linux/WSL: python3 docker/verify.py [--keep]
|
||||
Requires Docker with Compose v2 (directly or via sudo -n docker), Linux named
|
||||
volumes, and Git. docker/test-results/latest.json must already be gitignored;
|
||||
this script never changes ignore files. No image builds, pulls, host data
|
||||
mounts, application edits, production Compose files, or broad cleanup.
|
||||
|
||||
Checks have a 3600-second aggregate budget; failure shutdown has a separate
|
||||
720-second budget. Each health wait is at most 240 seconds, and each stop uses
|
||||
600 seconds of grace. A forced/nonzero/OOM exit never counts as success.
|
||||
Failures retain owned Docker artifacts after a guarded stop attempt. --keep
|
||||
also retains them on success. Evidence contains only counts, hashes, image
|
||||
IDs, fixed statuses, and durations; command logs are never printed or saved.
|
||||
|
||||
Contract limits: prepare is not repaired or rerun after recreation. A stopped
|
||||
container loses its /run/truf tmpfs, so its shutdown receipt cannot be read
|
||||
afterward. Receipt-write success is inferred ONLY from the healthy foreground
|
||||
runtime's zero-exit contract; private PostgreSQL PID-file absence is separately
|
||||
checked using the same runtime image and a read-only data-volume mount.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import re
|
||||
import secrets
|
||||
import selectors
|
||||
import shutil
|
||||
import signal
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
|
||||
PYTHON = ['/usr/local/bin/python3', '-I', '-S', '-B']
|
||||
APP = '/opt/truf/app/container_runtime.py'
|
||||
DRIVER = '/opt/truf/tests/container_e2e.py'
|
||||
CONFIG = '/data/config/e2e.yaml'
|
||||
ENTRYPOINT = ['/usr/bin/tini', '--', '/usr/local/bin/python3', '-u', '-I', '-S', '-B', APP]
|
||||
HEALTH = ['CMD', *PYTHON, APP, 'health', '--config', CONFIG]
|
||||
SERVICES = {'tools', 'provision', 'prepare', 'runtime', 'stopped'}
|
||||
IMAGE_REFERENCES = {
|
||||
'runtime': 'truf-worker-test:runtime',
|
||||
'test': 'truf-worker-test:test',
|
||||
}
|
||||
TMPFS = {
|
||||
'/run/truf': 'rw,nosuid,nodev,noexec,size=64m,mode=0700,uid=10001,gid=10001',
|
||||
'/tmp': 'rw,nosuid,nodev,noexec,size=128m,mode=1777',
|
||||
}
|
||||
PROXY_ENV = {
|
||||
name: '*' if name.lower() == 'no_proxy' else ''
|
||||
for stem in ('http_proxy', 'https_proxy', 'ftp_proxy', 'all_proxy', 'no_proxy')
|
||||
for name in (stem, stem.upper())
|
||||
}
|
||||
MOUNTS = {
|
||||
'tools': {'/opt/truf/tests': ('tools', False)},
|
||||
'provision': {'/data': ('data', False)},
|
||||
'prepare': {'/data': ('data', False), '/opt/truf/tests': ('tools', True)},
|
||||
'runtime': {'/data': ('data', False), '/opt/truf/tests': ('tools', True)},
|
||||
'stopped': {'/data': ('data', True)},
|
||||
}
|
||||
LABEL = 'com.docker.compose.'
|
||||
MAX_OUTPUT = 128 * 1024
|
||||
MAX_DOCKER_RESOURCES = 1024
|
||||
MAX_SNAPSHOT_BYTES = 4 * 1024 * 1024
|
||||
HEX = re.compile(r'[a-f0-9]{64}')
|
||||
|
||||
CONTAINER_METADATA_FORMAT = (
|
||||
'{"id":{{json .Id}},"name":{{json .Name}},"image":{{json .Image}},'
|
||||
'"status":{{json .State.Status}},"running":{{json .State.Running}},'
|
||||
'"paused":{{json .State.Paused}},"restarting":{{json .State.Restarting}},'
|
||||
'"dead":{{json .State.Dead}},"mounts":{{json .Mounts}}}'
|
||||
)
|
||||
VOLUME_METADATA_FORMAT = (
|
||||
'{"name":{{json .Name}},"driver":{{json .Driver}},"scope":{{json .Scope}},'
|
||||
'"created":{{json .CreatedAt}},"mountpoint":{{json .Mountpoint}},'
|
||||
'"labels":{{json .Labels}},"options":{{json .Options}}}'
|
||||
)
|
||||
|
||||
INSPECT_FIELDS = {
|
||||
'id': '.Id', 'name': '.Name', 'image': '.Image', 'status': '.State.Status',
|
||||
'running': '.State.Running', 'pid': '.State.Pid',
|
||||
'exit_code': '.State.ExitCode', 'oom_killed': '.State.OOMKilled',
|
||||
'restarts': '.RestartCount', 'user': '.Config.User',
|
||||
'entrypoint': '.Config.Entrypoint', 'command': '.Config.Cmd',
|
||||
'stop_timeout': '.Config.StopTimeout',
|
||||
'stop_signal': '.Config.StopSignal', 'mounts': '.Mounts',
|
||||
'readonly': '.HostConfig.ReadonlyRootfs', 'network': '.HostConfig.NetworkMode',
|
||||
'cap_drop': '.HostConfig.CapDrop', 'cap_add': '.HostConfig.CapAdd',
|
||||
'security_opt': '.HostConfig.SecurityOpt', 'init': '.HostConfig.Init',
|
||||
'privileged': '.HostConfig.Privileged', 'pid_mode': '.HostConfig.PidMode',
|
||||
'ports': '.HostConfig.PortBindings', 'tmpfs': '.HostConfig.Tmpfs',
|
||||
'cpus': '.HostConfig.NanoCpus', 'memory': '.HostConfig.Memory',
|
||||
'pids_limit': '.HostConfig.PidsLimit', 'restart_policy': '.HostConfig.RestartPolicy',
|
||||
**{key: '(index .Config.Labels "' + LABEL + suffix + '")' for key, suffix in (
|
||||
('project', 'project'), ('service', 'service'), ('oneoff', 'oneoff'),
|
||||
('config_files', 'project.config_files'), ('working_dir', 'project.working_dir'),
|
||||
)},
|
||||
}
|
||||
# Do not inspect .State or .Config wholesale: health logs and environments can
|
||||
# contain credentials. Only these selected fields enter the host process.
|
||||
INSPECT_FORMAT = '{' + ','.join(
|
||||
json.dumps(key) + ':{{json ' + value + '}}' for key, value in INSPECT_FIELDS.items()
|
||||
) + (',"health_test":{{with index .Config "Healthcheck"}}{{json .Test}}{{else}}null{{end}}'
|
||||
',"health":{{with index .State "Health"}}{{json .Status}}{{else}}null{{end}}}')
|
||||
|
||||
|
||||
class Failure(Exception):
|
||||
"""Only fixed check labels, never subprocess output or exception messages."""
|
||||
|
||||
|
||||
def require(condition, label):
|
||||
if not condition:
|
||||
raise Failure(label)
|
||||
|
||||
|
||||
class Verifier:
|
||||
def __init__(self):
|
||||
self.script = Path(__file__).absolute()
|
||||
self.root = self.script.parent.parent.resolve(strict=True)
|
||||
self.file = self.root / 'compose.e2e.yaml'
|
||||
self.project = 'truf-worker-test-' + secrets.token_hex(16)
|
||||
self.started = time.monotonic()
|
||||
self.deadline = self.started + 3600
|
||||
self.docker = []
|
||||
self.compose = []
|
||||
self.images = {}
|
||||
self.owned_containers = {}
|
||||
self.owned_volumes = {}
|
||||
self.foreign_baseline = None
|
||||
self.mutated = False
|
||||
self.evidence_ready = False
|
||||
self.file_hashes = {}
|
||||
self.report = {
|
||||
'status': {'result': 'running', 'cleanup': 'not_started'},
|
||||
'counts': {'schema': 1},
|
||||
'hashes': {'project_sha256': hashlib.sha256(self.project.encode('ascii')).hexdigest()},
|
||||
'image_ids': self.images, 'durations': {},
|
||||
}
|
||||
allowed_env = (
|
||||
'PATH', 'HOME', 'XDG_CONFIG_HOME', 'XDG_RUNTIME_DIR', 'SSH_AUTH_SOCK',
|
||||
'DOCKER_HOST', 'DOCKER_CONTEXT', 'DOCKER_CONFIG', 'DOCKER_TLS_VERIFY',
|
||||
'DOCKER_CERT_PATH',
|
||||
)
|
||||
self.env = {name: os.environ[name] for name in allowed_env if name in os.environ}
|
||||
self.env['COMPOSE_DISABLE_ENV_FILE'] = '1'
|
||||
|
||||
def execute(self, label, args, *, timeout=30, capture=False, check=True):
|
||||
started = time.monotonic()
|
||||
end = min(self.deadline, started + timeout)
|
||||
require(end > started, 'aggregate_timeout')
|
||||
process = None
|
||||
output = bytearray()
|
||||
selector = selectors.DefaultSelector()
|
||||
counts = self.report['counts'].setdefault(label, {})
|
||||
counts['cli_calls'] = counts.get('cli_calls', 0) + 1
|
||||
self.report['status'][label] = 'running'
|
||||
try:
|
||||
process = subprocess.Popen(
|
||||
args, cwd=self.root, env=self.env, stdin=subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE if capture else subprocess.DEVNULL,
|
||||
stderr=subprocess.DEVNULL, start_new_session=True,
|
||||
)
|
||||
if capture:
|
||||
selector.register(process.stdout, selectors.EVENT_READ)
|
||||
while process.poll() is None or selector.get_map():
|
||||
remaining = end - time.monotonic()
|
||||
require(remaining > 0, label + '_timeout')
|
||||
if selector.get_map():
|
||||
for key, _ in selector.select(min(0.1, remaining)):
|
||||
chunk = os.read(key.fd, 65536)
|
||||
if not chunk:
|
||||
selector.unregister(key.fileobj)
|
||||
else:
|
||||
output.extend(chunk)
|
||||
require(len(output) <= MAX_OUTPUT, label + '_output_limit')
|
||||
else:
|
||||
time.sleep(min(0.05, remaining))
|
||||
code = process.returncode
|
||||
counts['exit_code' if code >= 0 else 'signal'] = abs(code)
|
||||
self.report['status'][label] = 'passed' if code == 0 else 'failed'
|
||||
if check and code != 0:
|
||||
error = Failure(label + '_command_failed')
|
||||
error.exit_code = code
|
||||
raise error
|
||||
return code, bytes(output)
|
||||
except OSError:
|
||||
self.report['status'][label] = 'failed'
|
||||
raise Failure(label + '_unavailable') from None
|
||||
except BaseException:
|
||||
self.report['status'][label] = 'failed'
|
||||
raise
|
||||
finally:
|
||||
selector.close()
|
||||
if process is not None:
|
||||
if process.poll() is None:
|
||||
# Terminate only the local CLI process group, not containers.
|
||||
# An interrupted Docker API operation can outlive its client;
|
||||
# failure handling discovers and stops owned containers.
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
except (ProcessLookupError, PermissionError):
|
||||
pass
|
||||
try:
|
||||
process.wait(timeout=2)
|
||||
except subprocess.TimeoutExpired:
|
||||
pass
|
||||
if process.stdout is not None:
|
||||
process.stdout.close()
|
||||
durations = self.report['durations']
|
||||
durations[label] = round(durations.get(label, 0) + time.monotonic() - started, 3)
|
||||
|
||||
def json_command(self, label, args, timeout=30):
|
||||
_, output = self.execute(label, args, timeout=timeout, capture=True)
|
||||
try:
|
||||
return json.loads(output)
|
||||
except (ValueError, UnicodeError):
|
||||
raise Failure(label + '_invalid_json') from None
|
||||
|
||||
def words(self, label, args):
|
||||
_, output = self.execute(label, [*self.docker, *args], capture=True)
|
||||
try:
|
||||
words = output.decode('ascii').split()
|
||||
except UnicodeError:
|
||||
raise Failure(label + '_invalid_inventory') from None
|
||||
require(len(words) <= 16, label + '_inventory_limit')
|
||||
return set(words)
|
||||
|
||||
def metadata_names(self, kind):
|
||||
options = (['--all', '--no-trunc', '--format', '{{.ID}}']
|
||||
if kind == 'container' else ['--format', '{{.Name}}'])
|
||||
_, output = self.execute(
|
||||
'foreign_' + kind + '_list', [*self.docker, kind, 'ls', *options], capture=True,
|
||||
)
|
||||
try:
|
||||
values = {line for line in output.decode('ascii').splitlines() if line}
|
||||
except UnicodeError:
|
||||
raise Failure('foreign_' + kind + '_inventory_encoding') from None
|
||||
require(len(values) <= MAX_DOCKER_RESOURCES, 'foreign_' + kind + '_inventory_bound')
|
||||
return values
|
||||
|
||||
def container_metadata(self, identifier):
|
||||
require(HEX.fullmatch(identifier), 'foreign_container_id_guard')
|
||||
value = self.json_command('foreign_container_inspect', [
|
||||
*self.docker, 'container', 'inspect', '--format', CONTAINER_METADATA_FORMAT,
|
||||
identifier,
|
||||
])
|
||||
require(
|
||||
value.get('id') == identifier and isinstance(value.get('name'), str)
|
||||
and HEX.fullmatch(str(value.get('image') or '').removeprefix('sha256:'))
|
||||
and value.get('status') in (
|
||||
'created', 'running', 'paused', 'restarting', 'removing', 'exited', 'dead',
|
||||
)
|
||||
and all(type(value.get(key)) is bool for key in (
|
||||
'running', 'paused', 'restarting', 'dead',
|
||||
))
|
||||
and isinstance(value.get('mounts'), list) and len(value['mounts']) <= 128,
|
||||
'foreign_container_metadata_guard',
|
||||
)
|
||||
mounts = [{
|
||||
key: mount.get(key) for key in (
|
||||
'Type', 'Name', 'Source', 'Destination', 'Driver', 'Mode', 'RW', 'Propagation',
|
||||
)
|
||||
} for mount in value['mounts']]
|
||||
return {
|
||||
'id': value['id'], 'name': value['name'], 'image': value['image'],
|
||||
'status': value['status'], 'running': value['running'], 'paused': value['paused'],
|
||||
'restarting': value['restarting'], 'dead': value['dead'],
|
||||
'mounts_sha256': hashlib.sha256(json.dumps(
|
||||
mounts, ensure_ascii=True, sort_keys=True, separators=(',', ':'),
|
||||
).encode('ascii')).hexdigest(),
|
||||
}
|
||||
|
||||
def volume_metadata(self, name):
|
||||
value = self.json_command('foreign_volume_inspect', [
|
||||
*self.docker, 'volume', 'inspect', '--format', VOLUME_METADATA_FORMAT, name,
|
||||
])
|
||||
require(
|
||||
value.get('name') == name and isinstance(value.get('driver'), str)
|
||||
and isinstance(value.get('scope'), str) and isinstance(value.get('mountpoint'), str)
|
||||
and (value.get('created') is None or isinstance(value['created'], str))
|
||||
and (value.get('labels') is None or isinstance(value['labels'], dict))
|
||||
and (value.get('options') is None or isinstance(value['options'], dict)),
|
||||
'foreign_volume_metadata_guard',
|
||||
)
|
||||
return {
|
||||
'name': name, 'driver': value['driver'], 'scope': value['scope'],
|
||||
'created': value['created'],
|
||||
'mountpoint_sha256': hashlib.sha256(value['mountpoint'].encode('utf-8')).hexdigest(),
|
||||
'labels_sha256': hashlib.sha256(json.dumps(
|
||||
value['labels'], ensure_ascii=True, sort_keys=True, separators=(',', ':'),
|
||||
).encode('ascii')).hexdigest(),
|
||||
'options_sha256': hashlib.sha256(json.dumps(
|
||||
value['options'], ensure_ascii=True, sort_keys=True, separators=(',', ':'),
|
||||
).encode('ascii')).hexdigest(),
|
||||
}
|
||||
|
||||
def metadata_snapshot(self, *, exclude_owned=False):
|
||||
containers = {}
|
||||
for identifier in sorted(self.metadata_names('container')):
|
||||
if exclude_owned and identifier in self.owned_containers:
|
||||
self.inspect(identifier, 'foreign_owned_exclusion_guard')
|
||||
continue
|
||||
containers[identifier] = self.container_metadata(identifier)
|
||||
volumes = {}
|
||||
for name in sorted(self.metadata_names('volume')):
|
||||
value = self.volume_metadata(name)
|
||||
if exclude_owned and self.owned_volumes.get(name) == value:
|
||||
continue
|
||||
volumes[name] = value
|
||||
snapshot = {'containers': containers, 'volumes': volumes}
|
||||
require(len(json.dumps(snapshot, ensure_ascii=True, sort_keys=True)) <= MAX_SNAPSHOT_BYTES,
|
||||
'foreign_metadata_snapshot_bound')
|
||||
return snapshot
|
||||
|
||||
def snapshot_foreign(self):
|
||||
require(self.foreign_baseline is None, 'foreign_snapshot_already_taken')
|
||||
self.foreign_baseline = self.metadata_snapshot()
|
||||
|
||||
def assert_foreign_unchanged(self):
|
||||
require(self.foreign_baseline is not None, 'foreign_snapshot_missing')
|
||||
require(self.metadata_snapshot(exclude_owned=True) == self.foreign_baseline,
|
||||
'foreign_docker_state_changed')
|
||||
self.report['status']['foreign_docker_state'] = 'unchanged'
|
||||
|
||||
def guard_files(self):
|
||||
require(re.fullmatch(r'truf-worker-test-[a-f0-9]{32}', self.project),
|
||||
'worker_test_project_guard')
|
||||
require(self.script == self.root / 'docker' / 'verify.py', 'verifier_path_guard')
|
||||
require(self.file == self.root / 'compose.e2e.yaml', 'compose_path_guard')
|
||||
for path in (self.script, self.file):
|
||||
require(path.resolve(strict=True) == path and path.is_file() and not path.is_symlink(),
|
||||
'canonical_file_required')
|
||||
require(path.stat().st_size <= 256 * 1024, 'source_file_size_bound')
|
||||
digest = hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
require(path not in self.file_hashes or self.file_hashes[path] == digest,
|
||||
'source_changed_during_verification')
|
||||
self.file_hashes[path] = digest
|
||||
|
||||
def compose_command(self, label, args, **kwargs):
|
||||
self.guard_files()
|
||||
return self.execute(label, [*self.compose, *args], **kwargs)
|
||||
|
||||
def inspect(self, container, label='inspect', timeout=30):
|
||||
require(HEX.fullmatch(container), 'container_id_guard')
|
||||
value = self.json_command(label, [
|
||||
*self.docker, 'container', 'inspect', '--format', INSPECT_FORMAT, container,
|
||||
], timeout=timeout)
|
||||
require(value['id'] == container and value['project'] == self.project
|
||||
and isinstance(value['name'], str)
|
||||
and value['name'].startswith('/' + self.project + '-')
|
||||
and value['service'] in SERVICES
|
||||
and value['config_files'] == str(self.file)
|
||||
and value['working_dir'] == str(self.root), 'container_ownership_guard')
|
||||
service = value['service']
|
||||
require(value['oneoff'] == ('False' if service == 'runtime' else 'True'),
|
||||
'container_role_guard')
|
||||
require(value['image'] == self.images['test' if service == 'tools' else 'runtime'],
|
||||
'container_image_guard')
|
||||
require(value['status'] in ('created', 'running', 'paused', 'restarting', 'removing', 'exited', 'dead'),
|
||||
'container_state_guard')
|
||||
require(all(type(value[key]) is int and value[key] >= 0 for key in ('exit_code', 'restarts', 'pid'))
|
||||
and all(type(value[key]) is bool for key in ('oom_killed', 'running')),
|
||||
'container_state_types_guard')
|
||||
identity = {key: value[key] for key in (
|
||||
'id', 'name', 'image', 'project', 'service', 'config_files', 'working_dir',
|
||||
)}
|
||||
require(self.owned_containers.setdefault(container, identity) == identity,
|
||||
'container_identity_changed')
|
||||
self.report['status']['container_' + service] = value['status']
|
||||
self.report['counts']['container_' + service] = {
|
||||
'exit_code': value['exit_code'], 'oom_killed': int(value['oom_killed']),
|
||||
'restarts': value['restarts'], 'running': int(value['running']),
|
||||
}
|
||||
return value
|
||||
|
||||
def inventory(self, *, complete=False):
|
||||
self.guard_files()
|
||||
found = {}
|
||||
for kind in ('container', 'volume', 'network'):
|
||||
options = ['--all', '--quiet', '--no-trunc'] if kind == 'container' else ['--format', '{{.Name}}']
|
||||
named = self.words('inventory_' + kind, [
|
||||
kind, 'ls', *options, '--filter', 'name=' + self.project,
|
||||
])
|
||||
labeled = self.words('ownership_' + kind, [
|
||||
kind, 'ls', *options, '--filter', 'label=' + LABEL + 'project=' + self.project,
|
||||
])
|
||||
require(named == labeled, 'resource_ownership_guard')
|
||||
found[kind] = named
|
||||
require(not found['network'], 'unexpected_project_network')
|
||||
expected = {self.project + '_data', self.project + '_tools'}
|
||||
require(found['volume'] <= expected, 'volume_name_guard')
|
||||
if complete:
|
||||
require(found['volume'] == expected, 'required_volumes_missing')
|
||||
for volume in sorted(found['volume']):
|
||||
value = self.json_command('inspect_volume', [
|
||||
*self.docker, 'volume', 'inspect', '--format',
|
||||
'{"name":{{json .Name}},"driver":{{json .Driver}},"options":{{json .Options}},'
|
||||
'"scope":{{json .Scope}},"project":{{json (index .Labels "com.docker.compose.project")}},'
|
||||
'"volume":{{json (index .Labels "com.docker.compose.volume")}}}', volume,
|
||||
])
|
||||
require(value['name'] == volume and value['project'] == self.project
|
||||
and value['volume'] in ('data', 'tools')
|
||||
and volume == self.project + '_' + value['volume']
|
||||
and value['driver'] == 'local' and not value['options']
|
||||
and value['scope'] == 'local', 'native_named_volume_guard')
|
||||
metadata = self.volume_metadata(volume)
|
||||
require(self.owned_volumes.setdefault(volume, metadata) == metadata,
|
||||
'volume_identity_changed')
|
||||
users = self.words('volume_users', [
|
||||
'container', 'ls', '--all', '--quiet', '--no-trunc', '--filter', 'volume=' + volume,
|
||||
])
|
||||
require(users <= found['container'], 'foreign_volume_user_guard')
|
||||
self.report['counts']['owned_resources'] = {
|
||||
kind + 's': len(names) for kind, names in found.items()
|
||||
}
|
||||
return [self.inspect(container) for container in sorted(found['container'])]
|
||||
|
||||
def validate_compose(self, value):
|
||||
require(value.get('name') == self.project, 'worker_test_project_guard')
|
||||
require(set(value['services']) == SERVICES and set(value['volumes']) == {'data', 'tools'}
|
||||
and not any(value.get(key) for key in ('networks', 'secrets', 'configs')),
|
||||
'compose_project_contract')
|
||||
for name, volume in value['volumes'].items():
|
||||
require(volume.get('name') == self.project + '_' + name,
|
||||
'production_volume_forbidden')
|
||||
require(
|
||||
volume.get('name') not in {'truf-docker_data', 'truf-docker_tools'}
|
||||
and volume.get('driver') == 'local'
|
||||
and set(volume) <= {'name', 'driver'}, 'compose_volume_contract')
|
||||
allowed = {
|
||||
'image', 'pull_policy', 'read_only', 'user', 'cap_drop', 'cap_add', 'security_opt',
|
||||
'network_mode', 'volumes', 'tmpfs', 'cpus', 'mem_limit', 'pids_limit', 'shm_size',
|
||||
'stop_signal', 'stop_grace_period', 'logging', 'restart', 'entrypoint', 'command',
|
||||
'healthcheck', 'environment', 'init', 'ports',
|
||||
}
|
||||
for name, service in value['services'].items():
|
||||
require(set(service) <= allowed, 'compose_service_options_guard')
|
||||
require(service['image'] == self.images['test' if name == 'tools' else 'runtime'],
|
||||
'worker_test_image_reference_guard')
|
||||
require(not service.get('ports'), 'published_port_contract')
|
||||
require(
|
||||
service.get('network_mode') == 'none', 'internal_network_contract')
|
||||
require(
|
||||
service['image'] == self.images['test' if name == 'tools' else 'runtime']
|
||||
and service.get('pull_policy') == 'never'
|
||||
and service.get('read_only') is True
|
||||
and service.get('environment') == PROXY_ENV
|
||||
and service.get('init') is False
|
||||
and service.get('user') == ('0:0' if name == 'provision' else '10001:10001')
|
||||
and set(service.get('cap_drop', ())) == {'ALL'}
|
||||
and set(service.get('cap_add', ())) == (
|
||||
{'CHOWN', 'DAC_OVERRIDE', 'FOWNER'} if name == 'provision' else set())
|
||||
and service.get('security_opt') == ['no-new-privileges:true']
|
||||
and service.get('restart') == 'no'
|
||||
and float(service['cpus']) == 2
|
||||
and int(service['mem_limit']) == 6 * 1024 ** 3
|
||||
and int(service['pids_limit']) == 512
|
||||
and int(service['shm_size']) == 256 * 1024 ** 2
|
||||
and service['stop_signal'] == 'SIGTERM'
|
||||
and service['logging'] == {
|
||||
'driver': 'json-file', 'options': {'max-size': '16m', 'max-file': '4'},
|
||||
}
|
||||
and set(service['tmpfs']) == {key + ':' + val for key, val in TMPFS.items()},
|
||||
'compose_isolation_contract')
|
||||
mounts = {}
|
||||
for mount in service['volumes']:
|
||||
require(mount.get('type') == 'volume', 'bind_mount_forbidden')
|
||||
require(set(mount) <= {'type', 'source', 'target', 'read_only', 'volume'}
|
||||
and set(mount.get('volume', {})) <= {'nocopy'}, 'compose_mount_guard')
|
||||
require(mount['target'] not in mounts, 'duplicate_mount_guard')
|
||||
mounts[mount['target']] = (mount['source'], mount.get('read_only', False))
|
||||
if mount['source'] == 'tools':
|
||||
require(mount.get('volume', {}).get('nocopy', False) is (name != 'tools'),
|
||||
'tools_copy_up_contract')
|
||||
require(mounts == MOUNTS[name], 'compose_mount_contract')
|
||||
runtime = value['services']['runtime']
|
||||
require(runtime.get('entrypoint') is None and runtime['command'] == ['run', '--config', CONFIG]
|
||||
and runtime['healthcheck']['test'] == HEALTH, 'production_entrypoint_contract')
|
||||
require(value['services']['provision'].get('entrypoint') is None
|
||||
and value['services']['provision']['command'] == ['provision']
|
||||
and value['services']['prepare']['entrypoint'] == [*PYTHON, DRIVER]
|
||||
and value['services']['prepare']['command'] == ['prepare', '--config', CONFIG],
|
||||
'prepare_command_contract')
|
||||
|
||||
def preflight(self, env_file):
|
||||
self.guard_files()
|
||||
for path, digest in self.file_hashes.items():
|
||||
self.report['hashes']['compose_sha256' if path == self.file else 'verifier_sha256'] = digest
|
||||
docker = shutil.which('docker')
|
||||
require(docker, 'docker_cli_required')
|
||||
candidates = [[docker]]
|
||||
sudo = shutil.which('sudo')
|
||||
if sudo:
|
||||
candidates.append([sudo, '-n', docker])
|
||||
for index, candidate in enumerate(candidates):
|
||||
try:
|
||||
code, output = self.execute('docker_probe_' + str(index), [
|
||||
*candidate, 'info', '--format', '{{json .OSType}}',
|
||||
], timeout=15, capture=True, check=False)
|
||||
except Failure:
|
||||
continue
|
||||
if code == 0:
|
||||
require(json.loads(output) == 'linux', 'linux_docker_daemon_required')
|
||||
self.docker = candidate
|
||||
break
|
||||
require(self.docker, 'docker_or_passwordless_sudo_required')
|
||||
self.execute('compose_available', [*self.docker, 'compose', 'version', '--short'])
|
||||
self.snapshot_foreign()
|
||||
for name in ('runtime', 'test'):
|
||||
value = self.json_command('image_' + name, [
|
||||
*self.docker, 'image', 'inspect', '--format',
|
||||
'{"id":{{json .Id}},"os":{{json .Os}},"user":{{json .Config.User}},'
|
||||
'"entrypoint":{{json .Config.Entrypoint}},"volumes":{{json (index .Config "Volumes")}}}',
|
||||
IMAGE_REFERENCES[name],
|
||||
])
|
||||
require(re.fullmatch(r'sha256:[a-f0-9]{64}', value['id'])
|
||||
and value['os'] == 'linux' and value['user'] == '10001:10001'
|
||||
and not value['volumes'], 'prebuilt_image_contract')
|
||||
if name == 'runtime':
|
||||
require(value['entrypoint'] == ENTRYPOINT, 'runtime_image_entrypoint_contract')
|
||||
self.images[name] = value['id']
|
||||
require(self.images['runtime'] != self.images['test'], 'distinct_image_targets_required')
|
||||
env_file.write('TRUF_WORKER_TEST_PROJECT=' + self.project + '\n'
|
||||
'TRUF_WORKER_TEST_RUNTIME_IMAGE=' + self.images['runtime'] + '\n'
|
||||
'TRUF_WORKER_TEST_TEST_IMAGE=' + self.images['test'] + '\n')
|
||||
env_file.flush()
|
||||
# An explicit env file works with sudo's environment reset, too. Never
|
||||
# load the checkout's .env or accept COMPOSE_FILE/PROJECT_NAME overrides.
|
||||
self.compose = [
|
||||
*self.docker, 'compose', '--ansi', 'never', '--project-name', self.project,
|
||||
'--project-directory', str(self.root), '--env-file', env_file.name,
|
||||
'--file', str(self.file),
|
||||
]
|
||||
self.validate_compose(self.json_command('compose_contract', [
|
||||
*self.compose, 'config', '--format', 'json',
|
||||
]))
|
||||
git = shutil.which('git')
|
||||
require(git, 'git_required_for_evidence_ignore_guard')
|
||||
code, _ = self.execute('evidence_ignore_guard', [
|
||||
git, 'check-ignore', '--quiet', '--no-index', '--', 'docker/test-results/latest.json',
|
||||
], check=False)
|
||||
require(code == 0, 'evidence_path_must_be_gitignored')
|
||||
require(not self.inventory(), 'fresh_project_required')
|
||||
require(not self.words('fresh_volumes', [
|
||||
'volume', 'ls', '--format', '{{.Name}}', '--filter', 'name=' + self.project,
|
||||
]), 'fresh_volumes_required')
|
||||
directory = self.root / 'docker' / 'test-results'
|
||||
require(not directory.is_symlink(), 'evidence_directory_guard')
|
||||
directory.mkdir(mode=0o700, exist_ok=True)
|
||||
require(directory.resolve(strict=True) == directory and directory.is_dir(),
|
||||
'evidence_directory_guard')
|
||||
self.evidence_ready = True
|
||||
|
||||
def summary(self, label, args, timeout):
|
||||
print(label + ': running', flush=True)
|
||||
code, output = self.execute(label, args, timeout=timeout, capture=True, check=False)
|
||||
try:
|
||||
value = json.loads(output)
|
||||
except (ValueError, UnicodeError):
|
||||
raise Failure(label + '_invalid_summary') from None
|
||||
require(isinstance(value, dict) and set(value) == {'counts', 'hashes'}
|
||||
and all(isinstance(value[key], dict) and len(value[key]) <= 128 for key in value),
|
||||
label + '_invalid_summary')
|
||||
require(all(re.fullmatch(r'[a-z][a-z0-9_:]{0,120}', key)
|
||||
and type(count) is int and 0 <= count <= 2 ** 63 - 1
|
||||
for key, count in value['counts'].items())
|
||||
and all(re.fullmatch(r'[a-z][a-z0-9_:]{0,120}_sha256', key)
|
||||
and isinstance(digest, str) and HEX.fullmatch(digest)
|
||||
for key, digest in value['hashes'].items()), label + '_invalid_summary')
|
||||
self.report['counts'][label].update(value['counts'])
|
||||
self.report['hashes'][label] = value['hashes']
|
||||
require(code == 0 and value['counts'].get('ok') == 1, label + '_check_failed')
|
||||
return value
|
||||
|
||||
def oneoff(self, service, label=None, timeout=180):
|
||||
self.guard_files()
|
||||
args = [*self.compose, 'run', '--rm', '--no-deps', '--pull', 'never', '-T', service]
|
||||
if service == 'provision':
|
||||
print('provision: running', flush=True)
|
||||
self.execute('provision', args, timeout=timeout)
|
||||
return None
|
||||
return self.summary(label or service, args, timeout)
|
||||
|
||||
def start(self, label, previous=None):
|
||||
if previous:
|
||||
value = self.inspect(previous, label + '_previous_ownership')
|
||||
require(value['status'] == 'exited' and not value['running'],
|
||||
label + '_previous_not_stopped')
|
||||
self.inspect(previous, label + '_previous_remove_guard')
|
||||
self.execute(label + '_remove_previous', [
|
||||
*self.docker, 'container', 'rm', previous,
|
||||
])
|
||||
self.compose_command(label, [
|
||||
'up', '--detach', '--no-deps', '--no-build', '--pull', 'never',
|
||||
'runtime',
|
||||
], timeout=120)
|
||||
containers = self.inventory(complete=True)
|
||||
runtimes = [value for value in containers if value['service'] == 'runtime']
|
||||
require(len(runtimes) == 1, 'single_runtime_required')
|
||||
value = runtimes[0]
|
||||
container = value['id']
|
||||
require(container != previous, 'new_runtime_container_required')
|
||||
require(value['entrypoint'] == ENTRYPOINT and value['command'] == ['run', '--config', CONFIG]
|
||||
and value['health_test'] == HEALTH and value['user'] == '10001:10001'
|
||||
and value['readonly'] is True and value['network'] == 'none'
|
||||
and set(value['cap_drop'] or ()) == {'ALL'} and not value['cap_add']
|
||||
and value['security_opt'] == ['no-new-privileges:true']
|
||||
and not value['init'] and not value['privileged'] and not value['pid_mode']
|
||||
and not value['ports'] and value['tmpfs'] == TMPFS
|
||||
and value['cpus'] == 2_000_000_000 and value['memory'] == 6 * 1024 ** 3
|
||||
and value['pids_limit'] == 512 and value['stop_timeout'] == 600
|
||||
and value['stop_signal'] == 'SIGTERM'
|
||||
and value['restart_policy'] == {'Name': 'no', 'MaximumRetryCount': 0},
|
||||
'actual_runtime_isolation_contract')
|
||||
mounts = {}
|
||||
for mount in value['mounts']:
|
||||
if mount['Type'] == 'tmpfs':
|
||||
require(mount['Destination'] in TMPFS, 'unexpected_tmpfs')
|
||||
continue
|
||||
require(mount['Type'] == 'volume' and mount['Destination'] not in mounts,
|
||||
'actual_named_mount_required')
|
||||
mounts[mount['Destination']] = (mount['Name'], not mount['RW'])
|
||||
require(mounts == {target: (self.project + '_' + name, ro)
|
||||
for target, (name, ro) in MOUNTS['runtime'].items()},
|
||||
'actual_runtime_mount_contract')
|
||||
self.report['hashes'][label + '_container_sha256'] = hashlib.sha256(container.encode('ascii')).hexdigest()
|
||||
health_started = time.monotonic()
|
||||
health_end = min(self.deadline, health_started + 240)
|
||||
while time.monotonic() < health_end:
|
||||
value = self.inspect(container, label + '_health', timeout=min(10, health_end - time.monotonic()))
|
||||
require(value['running'] and value['status'] == 'running'
|
||||
and not value['oom_killed'] and value['restarts'] == 0,
|
||||
label + '_runtime_exited_or_restarted')
|
||||
if value['health'] == 'healthy':
|
||||
self.report['durations'][label + '_ready'] = round(time.monotonic() - health_started, 3)
|
||||
return container
|
||||
time.sleep(min(2, max(0, health_end - time.monotonic())))
|
||||
raise Failure(label + '_health_timeout')
|
||||
|
||||
def driver(self, container, mode, label, timeout=240):
|
||||
self.inspect(container)
|
||||
return self.summary(label, [
|
||||
*self.docker, 'exec', '--user', '10001:10001', container, *PYTHON,
|
||||
DRIVER, mode, '--config', CONFIG, '--timeout', '180',
|
||||
], timeout)
|
||||
|
||||
def health(self, container, label):
|
||||
self.inspect(container)
|
||||
value = self.json_command(label, [
|
||||
*self.docker, 'exec', '--user', '10001:10001', container, *PYTHON,
|
||||
APP, 'health', '--config', CONFIG,
|
||||
], timeout=30)
|
||||
require(value == {
|
||||
'healthy': True, 'activation_state': 'ACTIVE', 'postgres': 'READY',
|
||||
'workers': ['gitlab', 'janitor', 'jsonl-projector', 'result-ingester'],
|
||||
}, label + '_authenticated_health_failed')
|
||||
self.report['counts'][label]['authenticated_health'] = 1
|
||||
|
||||
def stop(self, container, label):
|
||||
containers = self.inventory(complete=True)
|
||||
require(any(value['id'] == container and value['running'] for value in containers),
|
||||
label + '_runtime_not_running')
|
||||
print(label + ': stopping (600-second grace)', flush=True)
|
||||
self.inspect(container, label + '_ownership_guard')
|
||||
self.execute(label, [
|
||||
*self.docker, 'container', 'stop', '--time', '600', container,
|
||||
], timeout=660)
|
||||
value = self.inspect(container, label + '_inspect')
|
||||
self.report['counts'][label].update({
|
||||
'container_exit_code': value['exit_code'], 'oom_killed': int(value['oom_killed']),
|
||||
'restarts': value['restarts'],
|
||||
})
|
||||
require(value['status'] == 'exited' and not value['running'] and value['pid'] == 0
|
||||
and value['exit_code'] == 0 and value['oom_killed'] is False
|
||||
and value['restarts'] == 0, label + '_unclean_exit')
|
||||
self.oneoff('stopped', label + '_data')
|
||||
# Foreground supervisor.main returns zero only after receipt publication.
|
||||
# Do not claim to have read that receipt from a destroyed tmpfs.
|
||||
self.report['status'][label + '_receipt'] = 'exit_contract_only_tmpfs_removed'
|
||||
|
||||
def cleanup(self, keep):
|
||||
containers = self.inventory(complete=True)
|
||||
require(len(containers) == 1 and containers[0]['service'] == 'runtime'
|
||||
and containers[0]['status'] == 'exited' and containers[0]['exit_code'] == 0
|
||||
and not containers[0]['oom_killed'], 'cleanup_stopped_runtime_guard')
|
||||
if keep:
|
||||
self.assert_foreign_unchanged()
|
||||
self.report['status']['cleanup'] = 'kept'
|
||||
return
|
||||
self.report['status']['cleanup'] = 'running'
|
||||
container = containers[0]['id']
|
||||
self.inspect(container, 'cleanup_container_remove_guard')
|
||||
self.execute('remove_owned_container', [*self.docker, 'container', 'rm', container])
|
||||
for volume in sorted(self.owned_volumes):
|
||||
require(self.volume_metadata(volume) == self.owned_volumes[volume],
|
||||
'cleanup_volume_identity_changed')
|
||||
self.execute('remove_owned_volume', [*self.docker, 'volume', 'rm', volume])
|
||||
require(not self.inventory(), 'cleanup_containers_remaining')
|
||||
require(not self.words('cleanup_volumes', [
|
||||
'volume', 'ls', '--format', '{{.Name}}', '--filter', 'name=' + self.project,
|
||||
]), 'cleanup_volumes_remaining')
|
||||
self.assert_foreign_unchanged()
|
||||
self.report['status']['cleanup'] = 'removed'
|
||||
|
||||
def failure_stop(self):
|
||||
self.deadline = time.monotonic() + 720
|
||||
self.report['status']['cleanup'] = 'retained_after_failure'
|
||||
try:
|
||||
containers = self.inventory()
|
||||
active = [value for value in containers
|
||||
if value['running'] or value['status'] in ('restarting', 'paused')]
|
||||
if active:
|
||||
print('failure_stop: stopping owned containers (600-second grace)', flush=True)
|
||||
for value in active:
|
||||
container = value['id']
|
||||
self.inspect(container, 'failure_stop_ownership_guard')
|
||||
self.execute('failure_stop', [
|
||||
*self.docker, 'container', 'stop', '--time', '600', container,
|
||||
], timeout=660)
|
||||
remaining = self.inventory()
|
||||
self.report['counts']['failure_retained'] = {
|
||||
'containers': len(remaining), 'running': sum(int(value['running']) for value in remaining),
|
||||
}
|
||||
self.report['status']['failure_stop'] = (
|
||||
'incomplete' if any(value['running'] for value in remaining) else 'observed_stopped'
|
||||
)
|
||||
except (Exception, KeyboardInterrupt):
|
||||
# Loss of the daemon, changed files, or unproven ownership must never
|
||||
# lead to an unguarded down/prune/kill attempt or a false success.
|
||||
self.report['status']['failure_stop'] = 'failed_or_ownership_unproven'
|
||||
|
||||
def write_evidence(self):
|
||||
self.report['durations']['total'] = round(time.monotonic() - self.started, 3)
|
||||
directory = self.root / 'docker' / 'test-results'
|
||||
require(directory.resolve(strict=True) == directory, 'evidence_directory_guard')
|
||||
descriptor = os.open(directory, os.O_RDONLY | os.O_DIRECTORY | os.O_NOFOLLOW)
|
||||
temporary = '.' + self.project + '.json'
|
||||
try:
|
||||
try:
|
||||
details = os.stat('latest.json', dir_fd=descriptor, follow_symlinks=False)
|
||||
require(stat.S_ISREG(details.st_mode), 'evidence_file_guard')
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
output = os.open(temporary, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW,
|
||||
0o600, dir_fd=descriptor)
|
||||
with os.fdopen(output, 'w', encoding='ascii') as handle:
|
||||
json.dump(self.report, handle, sort_keys=True, indent=2, allow_nan=False)
|
||||
handle.write('\n')
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(temporary, 'latest.json', src_dir_fd=descriptor, dst_dir_fd=descriptor)
|
||||
os.fsync(descriptor)
|
||||
finally:
|
||||
try:
|
||||
os.unlink(temporary, dir_fd=descriptor)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
os.close(descriptor)
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
class Parser(argparse.ArgumentParser):
|
||||
def error(self, message):
|
||||
raise Failure('invalid_arguments')
|
||||
|
||||
parser = Parser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter, allow_abbrev=False)
|
||||
parser.add_argument('--keep', action='store_true', help='retain stopped, owned Docker artifacts on success too')
|
||||
args = parser.parse_args(argv)
|
||||
require(sys.platform == 'linux', 'linux_host_required_use_wsl_python3')
|
||||
verifier = Verifier()
|
||||
print('E2E project: ' + verifier.project, flush=True)
|
||||
|
||||
def interrupted(_signum, _frame):
|
||||
raise KeyboardInterrupt
|
||||
|
||||
previous = signal.signal(signal.SIGTERM, interrupted)
|
||||
# This private file contains only the project nonce and image IDs.
|
||||
with tempfile.NamedTemporaryFile(mode='w', encoding='ascii', prefix=verifier.project + '-', suffix='.env') as env_file:
|
||||
try:
|
||||
verifier.preflight(env_file)
|
||||
verifier.write_evidence()
|
||||
verifier.mutated = True
|
||||
verifier.oneoff('tools')
|
||||
verifier.oneoff('provision')
|
||||
verifier.oneoff('prepare')
|
||||
first = verifier.start('first_start')
|
||||
verifier.driver(first, 'run-local-pipeline', 'local_pipeline', timeout=240)
|
||||
verifier.driver(first, 'assert-pipeline', 'pipeline')
|
||||
baseline = verifier.driver(first, 'keycheck-fixture', 'first_keycheck', timeout=510)
|
||||
require(baseline['counts'].pop('http_requests', None) == 1, 'first_provider_request_count')
|
||||
verifier.health(first, 'first_authenticated_health')
|
||||
verifier.stop(first, 'first_stop')
|
||||
second = verifier.start('second_start', previous=first)
|
||||
persisted = verifier.driver(second, 'assert-persisted', 'persisted')
|
||||
require(persisted == baseline, 'persisted_public_summary_changed')
|
||||
checked = verifier.driver(second, 'keycheck-fixture', 'second_keycheck', timeout=510)
|
||||
require(checked['counts'].pop('http_requests', None) == 0, 'repeated_provider_made_http_requests')
|
||||
require(checked == baseline, 'repeated_provider_changed_public_summary')
|
||||
verifier.health(second, 'second_authenticated_health')
|
||||
verifier.driver(second, 'assert-remote-recovery', 'remote_recovery')
|
||||
verifier.driver(second, 'prepare-remote-transport', 'remote_transport')
|
||||
dockerhub_canary_prepare = verifier.driver(
|
||||
second, 'prepare-dockerhub-canary',
|
||||
'dockerhub_canary_prepare', timeout=510,
|
||||
)
|
||||
require(
|
||||
dockerhub_canary_prepare['counts'].get('search_pages') == 1
|
||||
and dockerhub_canary_prepare['counts'].get('cohort_targets') == 4
|
||||
and dockerhub_canary_prepare['counts'].get('digest_resolutions') == 4
|
||||
and dockerhub_canary_prepare['counts'].get('worker_provider_failures') == 2
|
||||
and dockerhub_canary_prepare['counts'].get('accepted_uploads') == 3
|
||||
and dockerhub_canary_prepare['counts'].get('expired_assignments') == 1
|
||||
and dockerhub_canary_prepare['counts'].get('drain_cycles') == 2
|
||||
and dockerhub_canary_prepare['counts'].get('pending_targets') == 1,
|
||||
'dockerhub_canary_prepare_evidence',
|
||||
)
|
||||
remote_full_prepare = verifier.driver(
|
||||
second, 'prepare-remote-full-race', 'remote_full_prepare', timeout=510,
|
||||
)
|
||||
require(remote_full_prepare['counts'].get('real_claims') == 2
|
||||
and remote_full_prepare['counts'].get('native_scans') == 3
|
||||
and remote_full_prepare['counts'].get('exact_expiries') == 1
|
||||
and remote_full_prepare['counts'].get('pending_restart') == 1,
|
||||
'remote_full_prepare_evidence')
|
||||
verifier.stop(second, 'second_stop')
|
||||
third = verifier.start('third_start', previous=second)
|
||||
remote_full_finish = verifier.driver(
|
||||
third, 'finish-remote-full-race', 'remote_full_finish', timeout=510,
|
||||
)
|
||||
require(remote_full_finish['counts'].get('concurrent_uploads') == 2
|
||||
and remote_full_finish['counts'].get('authoritative_receipts') == 1
|
||||
and remote_full_finish['counts'].get('authoritative_scans') == 1
|
||||
and remote_full_finish['counts'].get('expired_losers') == 1
|
||||
and remote_full_finish['counts'].get('completed_winners') == 1
|
||||
and remote_full_finish['counts'].get('cached_keychecks') == 1
|
||||
and remote_full_finish['counts'].get('provider_http_requests') == 0
|
||||
and remote_full_finish['counts'].get('projection_streams') == 3,
|
||||
'remote_full_finish_evidence')
|
||||
verifier.driver(
|
||||
third, 'assert-remote-transport-replay', 'remote_transport_replay',
|
||||
)
|
||||
dockerhub_canary_finish = verifier.driver(
|
||||
third, 'finish-dockerhub-canary',
|
||||
'dockerhub_canary_finish', timeout=510,
|
||||
)
|
||||
require(
|
||||
dockerhub_canary_finish['counts'].get('runtime_restarts') == 1
|
||||
and dockerhub_canary_finish['counts'].get('lost_claim_receipts') == 1
|
||||
and dockerhub_canary_finish['counts'].get('receipt_replays') == 1
|
||||
and dockerhub_canary_finish['counts'].get('conflicts_rejected') == 1
|
||||
and dockerhub_canary_finish['counts'].get('exactly_once') == 1
|
||||
and dockerhub_canary_finish['counts'].get('former_expiry_replays') == 1,
|
||||
'dockerhub_canary_finish_evidence',
|
||||
)
|
||||
verifier.health(third, 'third_authenticated_health')
|
||||
verifier.stop(third, 'third_stop')
|
||||
verifier.report['status']['checks'] = 'passed'
|
||||
# Persist the check results before deleting their Docker artifacts.
|
||||
verifier.write_evidence()
|
||||
verifier.cleanup(args.keep)
|
||||
verifier.report['status']['result'] = 'passed'
|
||||
except (Exception, KeyboardInterrupt) as exc:
|
||||
verifier.report['status']['result'] = 'failed'
|
||||
label = str(exc) if isinstance(exc, Failure) else (
|
||||
'interrupted' if isinstance(exc, KeyboardInterrupt) else 'verifier_exception'
|
||||
)
|
||||
failure_class = type(exc).__name__
|
||||
if not re.fullmatch(r'[a-z0-9_]{1,160}', label):
|
||||
label = 'verifier_failure'
|
||||
if not re.fullmatch(r'[A-Za-z][A-Za-z0-9_]{0,79}', failure_class):
|
||||
failure_class = 'Exception'
|
||||
exit_code = getattr(exc, 'exit_code', None)
|
||||
if type(exit_code) is not int or not -(2 ** 31) <= exit_code < 2 ** 31:
|
||||
exit_code = None
|
||||
verifier.report['failure'] = {
|
||||
'stage': label, 'class': failure_class,
|
||||
'exit_code': exit_code,
|
||||
}
|
||||
print('E2E failed: ' + label, flush=True)
|
||||
if verifier.mutated:
|
||||
verifier.failure_stop()
|
||||
finally:
|
||||
signal.signal(signal.SIGTERM, previous)
|
||||
if verifier.evidence_ready:
|
||||
try:
|
||||
verifier.write_evidence()
|
||||
except (Exception, KeyboardInterrupt):
|
||||
verifier.report['status']['result'] = 'failed'
|
||||
print('E2E failed: evidence_write_failed', flush=True)
|
||||
else:
|
||||
print('Evidence: docker/test-results/latest.json', flush=True)
|
||||
print('E2E ' + verifier.report['status']['result'] + '; project ' + verifier.project
|
||||
+ '; artifacts ' + verifier.report['status']['cleanup'], flush=True)
|
||||
return 0 if verifier.report['status']['result'] == 'passed' else 1
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except (Exception, KeyboardInterrupt) as exc:
|
||||
print('E2E failed: ' + (str(exc) if isinstance(exc, Failure) else 'verifier_exception'), flush=True)
|
||||
raise SystemExit(1) from None
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,880 @@
|
||||
"""Offline Windows export, never a supervisor launcher or a source migrator.
|
||||
|
||||
Run only after the canonical Windows supervisor stop:
|
||||
python -I -S -B docker/windows_snapshot.py capture --output D:\\truf-docker\\docker\\imports\\NAME
|
||||
|
||||
Output is numeric: phase, count, bytes; failures are phase, exit code. Phases:
|
||||
1 arguments, 2 source loading, 3 private output, 4 offline inventory, 5 maintenance
|
||||
start, 6 database, 7 confirmed stop (count = retries), 8 tar, 9 revalidation,
|
||||
10 publication. Exit 1 is a guarded failure; 124 is a client timeout.
|
||||
|
||||
An unconfirmed maintenance stop deliberately RETAINS authority and retries. Do
|
||||
not terminate this process to bypass that guard. There is no valid manifest
|
||||
until cleanup confirms STOPPED. Killing Windows/processes can defeat any lock.
|
||||
SIGINT/SIGBREAK only request cancellation while capture owns the source. They
|
||||
cannot unwind the original backend's spawned-but-not-yet-bookkept launch window.
|
||||
|
||||
Assumes intact original helper APIs/layout and existing credentials permitted to
|
||||
dump all data, read pg_control_system(), and observe all client sessions.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import contextlib
|
||||
import ctypes
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
import fnmatch
|
||||
import hashlib
|
||||
import importlib
|
||||
import importlib.util
|
||||
import json
|
||||
import ntpath
|
||||
import os
|
||||
from pathlib import Path, PureWindowsPath
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import tarfile
|
||||
import threading
|
||||
import time
|
||||
from types import SimpleNamespace
|
||||
from urllib.parse import unquote, urlsplit
|
||||
|
||||
|
||||
SOURCE_ROOT = Path(r'D:\truf')
|
||||
POSTGRES_DATA = Path(r'S:\postgres-data')
|
||||
BUNDLE_ROOT = Path(r'S:\scanner-result-bundles')
|
||||
IMPORTS_ROOT = Path(r'D:\truf-docker\docker\imports')
|
||||
KEYCHECK_COPY = 'keychecks \u2014 \u043a\u043e\u043f\u0438\u044f'
|
||||
BLOCK = 1024 * 1024
|
||||
QUERY_TIMEOUT = 30
|
||||
COUNT_TIMEOUT = 1800
|
||||
DUMP_TIMEOUT = 6 * 3600
|
||||
SESSION_TIMEOUT = 12 * 3600
|
||||
MAX_METADATA = 4 * BLOCK
|
||||
ACTIVE_DIRS = ('queues', 'state', 'keychecks', 'results', 'postman_cache', 'result_spool')
|
||||
EXCLUSION_POLICY = [
|
||||
'Only explicitly reviewed source roots and mappings are selected.',
|
||||
'No physical PGDATA/WAL, PostgreSQL binaries/logs, or active control authority.',
|
||||
'No runtime/downloads, git, traces, freeze-diagnostics, or S: scanner-work.',
|
||||
'No gharchive_cache, .git, .opencode, tests, code caches, *.lock*, or *.pid.',
|
||||
'Ordinary *.log* excluded outside result/keycheck projections; scan_errors.log* retained.',
|
||||
'Windows scratch databases, temporary state and janitor cursor are archival only.',
|
||||
]
|
||||
|
||||
|
||||
class Failure(Exception):
|
||||
def __init__(self, code=1):
|
||||
self.code = code
|
||||
super().__init__(code)
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _defer_signals():
|
||||
pending = False
|
||||
previous = {}
|
||||
|
||||
def request(_number, _frame):
|
||||
nonlocal pending
|
||||
pending = True
|
||||
|
||||
def checkpoint():
|
||||
if pending:
|
||||
raise Failure()
|
||||
|
||||
try:
|
||||
for number in (signal.SIGINT, getattr(signal, 'SIGBREAK', None)):
|
||||
if number is not None:
|
||||
previous[number] = signal.signal(number, request)
|
||||
yield checkpoint
|
||||
finally:
|
||||
for number, handler in previous.items():
|
||||
signal.signal(number, handler)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class File:
|
||||
source: Path
|
||||
size: int
|
||||
fingerprint: tuple
|
||||
|
||||
|
||||
@dataclass
|
||||
class Inventory:
|
||||
files: dict
|
||||
directories: dict
|
||||
exclusions: dict
|
||||
|
||||
|
||||
def _fingerprint(info):
|
||||
return (info.st_dev, info.st_ino, info.st_mode, info.st_nlink, info.st_size,
|
||||
info.st_mtime_ns, info.st_ctime_ns, getattr(info, 'st_file_attributes', 0))
|
||||
|
||||
|
||||
def _check_type(info, directory=False):
|
||||
if (getattr(info, 'st_file_attributes', 0) & 0x400
|
||||
or stat.S_ISLNK(info.st_mode)
|
||||
or not (stat.S_ISDIR(info.st_mode) if directory else stat.S_ISREG(info.st_mode))
|
||||
or (not directory and info.st_nlink != 1)):
|
||||
raise Failure()
|
||||
|
||||
|
||||
def _check_chain(path):
|
||||
# Inspect ancestors first: even lstat(child) otherwise traverses a junction.
|
||||
path = Path(path).absolute()
|
||||
for part in (*reversed(path.parents), path):
|
||||
info = part.lstat()
|
||||
if stat.S_ISLNK(info.st_mode) or getattr(info, 'st_file_attributes', 0) & 0x400:
|
||||
raise Failure()
|
||||
|
||||
|
||||
def _file_info(path):
|
||||
_check_chain(path)
|
||||
info = path.lstat()
|
||||
_check_type(info)
|
||||
return info
|
||||
|
||||
|
||||
def _same_windows_path(left, right):
|
||||
return ntpath.normcase(ntpath.normpath(str(left))) == ntpath.normcase(ntpath.normpath(str(right)))
|
||||
|
||||
|
||||
def _output_path(value):
|
||||
path = PureWindowsPath(value)
|
||||
name = path.name
|
||||
if (not path.is_absolute() or not _same_windows_path(path.parent, IMPORTS_ROOT)
|
||||
or any(part in ('.', '..') for part in re.split(r'[\\/]', value))
|
||||
or not re.fullmatch(r'[A-Za-z0-9][A-Za-z0-9_.-]{0,95}', name)
|
||||
or name.endswith('.')
|
||||
or re.fullmatch(r'CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9]', name.split('.')[0], re.I)):
|
||||
raise Failure()
|
||||
return Path(str(path))
|
||||
|
||||
|
||||
def _verify_private_acl(path, security, directory):
|
||||
security.reject_reparse_components(str(path))
|
||||
sddl = security._windows_private_sddl(str(path)).upper()
|
||||
alias = lambda sid: 'SY' if sid == 'S-1-5-18' else sid
|
||||
sid = alias(security._windows_current_user_sid().upper())
|
||||
owner = re.search(r'O:([^:()]+?)(?=[GDS]:|$)', sddl)
|
||||
aces = [ace.split(';') for ace in re.findall(r'\(([^()]*)\)', sddl)]
|
||||
expected = {sid, 'SY'}
|
||||
if (not owner or alias(owner.group(1)) != sid or 'D:P' not in sddl
|
||||
or len(aces) != len(expected)):
|
||||
raise Failure()
|
||||
trustees = set()
|
||||
for ace in aces:
|
||||
if (len(ace) != 6 or ace[:5] != ['A', 'OICI' if directory else '', 'FA', '', '']):
|
||||
raise Failure()
|
||||
trustees.add(alias(ace[5]))
|
||||
if trustees != expected:
|
||||
raise Failure()
|
||||
|
||||
|
||||
def _secure_path(path, security, directory=False):
|
||||
# The original public helper includes BA. Narrow its verified, empty path
|
||||
# using the same no-reparse Win32 primitives, before any sensitive write.
|
||||
harden = security.harden_private_directory if directory else security.harden_private_file
|
||||
harden(str(path))
|
||||
sid = security._windows_current_user_sid()
|
||||
trustees = ('SY',) if sid.upper() == 'S-1-5-18' else (sid, 'SY')
|
||||
flags = 'OICI' if directory else ''
|
||||
sddl = 'D:P' + ''.join(f'(A;{flags};FA;;;{trustee})' for trustee in trustees)
|
||||
descriptor = ctypes.c_void_p()
|
||||
if not security._CONVERT_SDDL(sddl, 1, ctypes.byref(descriptor), None):
|
||||
raise Failure()
|
||||
handle = None
|
||||
try:
|
||||
handle = security._CREATE_FILE(str(path), 0x60000, 7, None, 3, 0x2200000, None)
|
||||
if handle == ctypes.c_void_p(-1).value:
|
||||
raise Failure()
|
||||
info = security._BY_HANDLE_FILE_INFORMATION()
|
||||
if (not security._GET_FILE_INFORMATION(handle, ctypes.byref(info))
|
||||
or info.dwFileAttributes & 0x400
|
||||
or not security._SET_KERNEL_OBJECT_SECURITY(handle, 0x80000004, descriptor)):
|
||||
raise Failure()
|
||||
finally:
|
||||
if handle is not None and handle != ctypes.c_void_p(-1).value:
|
||||
security._CLOSE_HANDLE(handle)
|
||||
security._LOCAL_FREE(descriptor)
|
||||
_verify_private_acl(path, security, directory)
|
||||
|
||||
|
||||
def _prepare_output(output, security):
|
||||
if output.parent != IMPORTS_ROOT:
|
||||
raise Failure()
|
||||
_check_chain(IMPORTS_ROOT.parent)
|
||||
_check_type(IMPORTS_ROOT.parent.lstat(), directory=True)
|
||||
try:
|
||||
IMPORTS_ROOT.mkdir()
|
||||
except FileExistsError:
|
||||
_check_chain(IMPORTS_ROOT)
|
||||
_check_type(IMPORTS_ROOT.lstat(), directory=True)
|
||||
else:
|
||||
_secure_path(IMPORTS_ROOT, security, directory=True)
|
||||
output.mkdir() # Never reuse, overwrite or repair an existing snapshot.
|
||||
_secure_path(output, security, directory=True)
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _output_file(path, security):
|
||||
_check_chain(path.parent)
|
||||
with path.open('xb', buffering=0) as handle:
|
||||
_secure_path(path, security)
|
||||
info = _file_info(path)
|
||||
opened = os.fstat(handle.fileno())
|
||||
if (info.st_dev, info.st_ino) != (opened.st_dev, opened.st_ino):
|
||||
raise Failure()
|
||||
yield handle
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
|
||||
|
||||
def _destination(path, used, directories):
|
||||
parts = path.split('/')
|
||||
if (not parts or any(not p or p in ('.', '..') for p in parts)
|
||||
or any(ord(c) < 32 or ord(c) == 127 for c in path)
|
||||
or '\\' in path or ':' in path):
|
||||
raise Failure()
|
||||
folded = path.casefold()
|
||||
parents = {'/'.join(parts[:i]).casefold() for i in range(1, len(parts))}
|
||||
if folded in used or folded in directories or parents.intersection(used):
|
||||
raise Failure()
|
||||
used.add(folded)
|
||||
directories.update(parents)
|
||||
|
||||
|
||||
def _excluded(relative, control=False):
|
||||
parts = relative.casefold().split('/')
|
||||
name = parts[-1]
|
||||
if any(p in {'.git', '.opencode', 'tests', '__pycache__', '.pytest_cache',
|
||||
'.mypy_cache', '.ruff_cache', 'node_modules'} for p in parts):
|
||||
return 'code_cache_or_unreviewed_code'
|
||||
if 'gharchive_cache' in parts:
|
||||
return 'downloaded_archives'
|
||||
if any(fnmatch.fnmatchcase(p, '*.lock*') or p.endswith('.pid') for p in parts):
|
||||
return 'locks_or_pids'
|
||||
if name.endswith(('.pyc', '.pyo')):
|
||||
return 'code_cache_or_unreviewed_code'
|
||||
if control and any(re.search(
|
||||
r'supervisor|instance|token|authority|manifest|handshake|capability|(?:^|[._-])(?:pid|lock)(?:[._-]|$)',
|
||||
p) for p in parts[2:]):
|
||||
return 'control_authority'
|
||||
projection = any(p in ('results', 'keychecks', KEYCHECK_COPY.casefold())
|
||||
or p.startswith(('found_secrets.jsonl', 'scan_results.jsonl', 'scan_errors.log')) for p in parts)
|
||||
config_backup = len(parts) == 2 and parts[0] == 'app' and name.startswith(('config.yaml.', 'secrets.yaml.'))
|
||||
if (fnmatch.fnmatchcase(name, '*.log*') and not projection
|
||||
and not config_backup
|
||||
and not fnmatch.fnmatchcase(name, 'scan_errors.log*')
|
||||
and 'publication-ledger.sqlite3' not in name):
|
||||
return 'ordinary_logs'
|
||||
return None
|
||||
|
||||
|
||||
def _inventory():
|
||||
files, watched, excluded, used, parents = {}, {}, {}, set(), set()
|
||||
|
||||
def visit(path, relative, target, control=False, expect_directory=None):
|
||||
reason = _excluded(relative, control)
|
||||
if reason:
|
||||
excluded[reason] = excluded.get(reason, 0) + 1
|
||||
return
|
||||
try:
|
||||
_check_chain(path)
|
||||
info = path.lstat()
|
||||
except FileNotFoundError:
|
||||
watched[str(path)] = None
|
||||
return
|
||||
directory = stat.S_ISDIR(info.st_mode)
|
||||
_check_type(info, directory=directory)
|
||||
if expect_directory is not None and directory != expect_directory:
|
||||
raise Failure()
|
||||
if directory:
|
||||
watched[str(path)] = _fingerprint(info)
|
||||
with os.scandir(path) as entries:
|
||||
names = sorted(entry.name for entry in entries)
|
||||
for name in names:
|
||||
visit(path / name, relative + '/' + name, target + '/' + name, control)
|
||||
if _fingerprint(path.lstat()) != watched[str(path)]:
|
||||
raise Failure()
|
||||
return
|
||||
if control and not path.name.casefold().endswith('.json'):
|
||||
excluded['non_report_control'] = excluded.get('non_report_control', 0) + 1
|
||||
return
|
||||
if relative.casefold().startswith('runtime/state/'):
|
||||
scratch = relative.casefold().split('/')[2:]
|
||||
if any(fnmatch.fnmatchcase(p, 'scan_limiter*.db*')
|
||||
or fnmatch.fnmatchcase(p, '*.tmp*') or p == 'janitor.cursor.json' for p in scratch):
|
||||
target = 'windows-archive/' + relative
|
||||
_destination(target, used, parents)
|
||||
files[target] = File(path, info.st_size, _fingerprint(info))
|
||||
|
||||
for name in ACTIVE_DIRS:
|
||||
visit(SOURCE_ROOT / 'runtime' / name, 'runtime/' + name, 'runtime-linux/' + name,
|
||||
expect_directory=True)
|
||||
visit(SOURCE_ROOT / 'runtime/proxy.txt', 'runtime/proxy.txt', 'runtime-linux/proxy.txt',
|
||||
expect_directory=False)
|
||||
for name in ('secrets.yaml', 'trufflehog-custom-detectors.yaml'):
|
||||
visit(SOURCE_ROOT / 'app' / name, 'app/' + name, 'config/' + name, expect_directory=False)
|
||||
for relative in ('state', 'runtime/backups', 'runtime/imports', 'runtime/' + KEYCHECK_COPY):
|
||||
visit(SOURCE_ROOT / relative, relative, 'windows-archive/' + relative, expect_directory=True)
|
||||
for relative in (
|
||||
'app/config.yaml', 'app/.streamlit/config.toml', '.env.postgres', 'docker-compose.postgres.yml',
|
||||
'runner_state.json',
|
||||
'runtime/keychecks.7z', 'runtime/orkey.txt', 'runtime/check-openrouter-keys.ps1',
|
||||
):
|
||||
visit(SOURCE_ROOT / relative, relative, 'windows-archive/' + relative, expect_directory=False)
|
||||
for folder, patterns in (
|
||||
('', ('checked_*.txt', 'todo_*.txt', 'scanner.db*', 'found_secrets.jsonl*',
|
||||
'scan_results.jsonl*', 'scan_errors.log*', '*.publication-ledger.sqlite3*')),
|
||||
('app', ('config.yaml.*', 'secrets.yaml.*', 'scanner.db*')),
|
||||
('runtime', ('*.md',)),
|
||||
):
|
||||
path = SOURCE_ROOT / folder
|
||||
_check_chain(path)
|
||||
info = path.lstat()
|
||||
_check_type(info, directory=True)
|
||||
watched[str(path)] = _fingerprint(info)
|
||||
with os.scandir(path) as entries:
|
||||
names = sorted(entry.name for entry in entries
|
||||
if any(fnmatch.fnmatchcase(entry.name.casefold(), p) for p in patterns))
|
||||
for name in names:
|
||||
relative = folder + '/' + name if folder else name
|
||||
projection_family = not folder and name.casefold().startswith(
|
||||
('found_secrets.jsonl', 'scan_results.jsonl', 'scan_errors.log'))
|
||||
visit(path / name, relative, 'windows-archive/' + relative,
|
||||
expect_directory=None if projection_family else False)
|
||||
visit(SOURCE_ROOT / 'runtime/control', 'runtime/control', 'windows-archive/runtime/control',
|
||||
control=True, expect_directory=True)
|
||||
_check_chain(BUNDLE_ROOT)
|
||||
visit(BUNDLE_ROOT, 'scanner-result-bundles', 'scanner-result-bundles', expect_directory=True)
|
||||
return Inventory(files, watched, excluded)
|
||||
|
||||
|
||||
class HashWriter:
|
||||
def __init__(self, handle):
|
||||
self.handle = handle
|
||||
self.digest = hashlib.sha256()
|
||||
self.size = 0
|
||||
|
||||
def write(self, block):
|
||||
if self.handle.write(block) != len(block):
|
||||
raise Failure()
|
||||
self.digest.update(block)
|
||||
self.size += len(block)
|
||||
return len(block)
|
||||
|
||||
def metadata(self):
|
||||
return {'bytes': self.size, 'sha256': self.digest.hexdigest()}
|
||||
|
||||
|
||||
class HashReader:
|
||||
def __init__(self, handle):
|
||||
self.handle = handle
|
||||
self.digest = hashlib.sha256()
|
||||
self.size = 0
|
||||
|
||||
def read(self, size):
|
||||
block = self.handle.read(size)
|
||||
self.digest.update(block)
|
||||
self.size += len(block)
|
||||
return block
|
||||
|
||||
|
||||
def _write_tar(output, security, inventory, report):
|
||||
manifest_files = []
|
||||
with _output_file(output / 'files.tar', security) as handle:
|
||||
writer = HashWriter(handle)
|
||||
with tarfile.open(fileobj=writer, mode='w|', format=tarfile.PAX_FORMAT, copybufsize=BLOCK) as archive:
|
||||
for name, entry in sorted(inventory.files.items()):
|
||||
if _fingerprint(_file_info(entry.source)) != entry.fingerprint:
|
||||
raise Failure()
|
||||
with entry.source.open('rb', buffering=0) as source:
|
||||
before = _fingerprint(os.fstat(source.fileno()))
|
||||
# Windows Python 3.12 stat/fstat use different ctime bases.
|
||||
# Compare ctime within each API, not across the two APIs.
|
||||
if before[:6] + before[7:] != entry.fingerprint[:6] + entry.fingerprint[7:]:
|
||||
raise Failure()
|
||||
reader = HashReader(source)
|
||||
info = tarfile.TarInfo(name)
|
||||
info.size, info.mode, info.mtime = entry.size, 0o600, 0
|
||||
archive.addfile(info, reader)
|
||||
if (reader.size != entry.size
|
||||
or _fingerprint(os.fstat(source.fileno())) != before):
|
||||
raise Failure()
|
||||
if _fingerprint(_file_info(entry.source)) != entry.fingerprint:
|
||||
raise Failure()
|
||||
manifest_files.append({'path': name, 'size': entry.size, 'sha256': reader.digest.hexdigest()})
|
||||
report(8, len(manifest_files), writer.size)
|
||||
metadata = writer.metadata()
|
||||
return manifest_files, metadata
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _silence():
|
||||
with open(os.devnull, 'w', encoding='utf-8') as sink:
|
||||
with contextlib.redirect_stdout(sink), contextlib.redirect_stderr(sink):
|
||||
yield
|
||||
|
||||
|
||||
def _load_source():
|
||||
app = SOURCE_ROOT / 'app'
|
||||
for name in ('child_bootstrap.py', 'postgres_runtime.py', 'runtime_security.py'):
|
||||
_file_info(app / name)
|
||||
spec = importlib.util.spec_from_file_location('_snapshot_child_bootstrap', app / 'child_bootstrap.py')
|
||||
bootstrap = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(bootstrap)
|
||||
bootstrap._enable_dependency_paths('postgres-runtime')
|
||||
sys.path.insert(0, str(app))
|
||||
pg = importlib.import_module('postgres_runtime')
|
||||
security = importlib.import_module('runtime_security')
|
||||
for module in (pg, security):
|
||||
if not _same_windows_path(module.__file__, app / (module.__name__ + '.py')):
|
||||
raise Failure()
|
||||
# The original loader does not overwrite inherited credentials. Remove all
|
||||
# connection/path overrides first, so only the original .env can choose them.
|
||||
for key in list(os.environ):
|
||||
if key.upper().startswith(('PG', 'TRUF_', 'SCANNER_', 'TRUFFLEHOG_')) or key.upper() == 'DATABASE_URL':
|
||||
os.environ.pop(key, None)
|
||||
config_path, env_path = app / 'config.yaml', SOURCE_ROOT / '.env.postgres'
|
||||
inputs = {p: _fingerprint(_file_info(p)) for p in (config_path, env_path)}
|
||||
config = pg._load_config(str(config_path))
|
||||
expected = {'root_dir': SOURCE_ROOT, 'project_dir': app, 'runtime_dir': SOURCE_ROOT / 'runtime',
|
||||
'postgres_data_dir': POSTGRES_DATA, 'result_bundle_dir': BUNDLE_ROOT}
|
||||
for key, path in expected.items():
|
||||
if not _same_windows_path(config.get('global', {}).get(key, ''), path):
|
||||
raise Failure()
|
||||
for key, name in (('queue_dir', 'queues'), ('state_dir', 'state'), ('keycheck_dir', 'keychecks'),
|
||||
('results_dir', 'results'), ('postman_cache_dir', 'postman_cache'),
|
||||
('result_spool_dir', 'result_spool'), ('control_dir', 'control'), ('log_dir', 'logs')):
|
||||
value = config.get('global', {}).get(key)
|
||||
if value and not _same_windows_path(value, SOURCE_ROOT / 'runtime' / name):
|
||||
raise Failure()
|
||||
paths = pg.postgres_runtime_paths(config)
|
||||
if (not _same_windows_path(paths['data_dir'], POSTGRES_DATA)
|
||||
or not _same_windows_path(paths['postgres_dir'], SOURCE_ROOT / 'runtime/postgres')):
|
||||
raise Failure()
|
||||
security.preflight_lifecycle_paths(str(config_path), config)
|
||||
loaded = pg.load_postgres_environment(str(config_path), config)
|
||||
if not _same_windows_path(loaded or '', env_path):
|
||||
raise Failure()
|
||||
dsn = pg.canonical_database_url()
|
||||
if not dsn:
|
||||
raise Failure()
|
||||
identity_path = SOURCE_ROOT / 'runtime/postgres/cluster_identity.json'
|
||||
inputs[identity_path] = _fingerprint(_file_info(identity_path))
|
||||
return SimpleNamespace(pg=pg, security=security, config=config, dsn=dsn, inputs=inputs)
|
||||
|
||||
|
||||
def _supervisor_absent(source):
|
||||
supervisor = source.config.get('supervisor', {})
|
||||
paths = {SOURCE_ROOT / 'runtime/control/supervisor.instance.json',
|
||||
SOURCE_ROOT / 'runtime/logs/supervisor.instance.json',
|
||||
SOURCE_ROOT / 'runtime/logs/supervisor.pid'}
|
||||
for key in ('instance_file',):
|
||||
if supervisor.get(key):
|
||||
paths.add(Path(supervisor[key]))
|
||||
for key in ('control_dir', 'log_dir'):
|
||||
if supervisor.get(key):
|
||||
paths.add(Path(supervisor[key]) / 'supervisor.instance.json')
|
||||
if any(os.path.lexists(path) for path in paths):
|
||||
raise Failure()
|
||||
|
||||
|
||||
def _validate_identity(source, identity):
|
||||
values = source.pg.configured_cluster_values()
|
||||
parsed = urlsplit(source.dsn)
|
||||
if (identity['pg_major'] != 16 or not str(identity['system_identifier']).isdigit()
|
||||
or not _same_windows_path(identity['data_directory'], POSTGRES_DATA)
|
||||
or any(identity[k] != values[k] for k in ('database', 'user', 'port'))
|
||||
or parsed.scheme not in ('postgresql', 'postgres') or parsed.hostname != '127.0.0.1'
|
||||
or parsed.port != identity['port'] or unquote(parsed.username or '') != identity['user']
|
||||
or unquote(parsed.path[1:]) != identity['database'] or parsed.password is None
|
||||
or parsed.query or parsed.fragment):
|
||||
raise Failure()
|
||||
|
||||
|
||||
def _client_environment(dsn):
|
||||
parsed = urlsplit(dsn)
|
||||
env = {key: value for key, value in os.environ.items()
|
||||
if key.upper() in {'SYSTEMROOT', 'WINDIR', 'SYSTEMDRIVE', 'TEMP', 'TMP'}}
|
||||
env.update(PGHOST='127.0.0.1', PGHOSTADDR='127.0.0.1', PGPORT=str(parsed.port),
|
||||
PGDATABASE=unquote(parsed.path[1:]), PGUSER=unquote(parsed.username or ''),
|
||||
PGPASSWORD=unquote(parsed.password or ''), PGPASSFILE=os.devnull,
|
||||
PGSERVICEFILE=os.devnull, PGSSLMODE='disable', PGGSSENCMODE='disable',
|
||||
PGCONNECT_TIMEOUT='5', PGCLIENTENCODING='UTF8', PGAPPNAME='truf-windows-snapshot',
|
||||
PGOPTIONS=f'-c default_transaction_read_only=on -c statement_timeout={COUNT_TIMEOUT * 1000} '
|
||||
'-c lock_timeout=10000 -c idle_in_transaction_session_timeout=0 '
|
||||
'-c search_path=pg_catalog -c row_security=off',
|
||||
LC_ALL='C', LANG='C')
|
||||
return env
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _deadline(process, seconds):
|
||||
expired = threading.Event()
|
||||
|
||||
def expire():
|
||||
expired.set()
|
||||
try:
|
||||
process.kill()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
timer = threading.Timer(seconds, expire)
|
||||
timer.daemon = True
|
||||
timer.start()
|
||||
try:
|
||||
yield
|
||||
except BaseException:
|
||||
if expired.is_set():
|
||||
raise Failure(124) from None
|
||||
raise
|
||||
else:
|
||||
if expired.is_set():
|
||||
raise Failure(124)
|
||||
finally:
|
||||
timer.cancel()
|
||||
timer.join()
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def _client(command, env, timeout, interactive=False):
|
||||
process = subprocess.Popen(command, stdin=subprocess.PIPE if interactive else subprocess.DEVNULL,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env,
|
||||
creationflags=0x08000000, close_fds=True, bufsize=0)
|
||||
try:
|
||||
with _deadline(process, timeout):
|
||||
yield process
|
||||
if process.stdin is not None:
|
||||
process.stdin.close()
|
||||
if process.wait(timeout=10) != 0:
|
||||
raise Failure()
|
||||
finally:
|
||||
if process.poll() is None:
|
||||
process.kill()
|
||||
process.wait(timeout=10)
|
||||
if process.stdin is not None:
|
||||
process.stdin.close()
|
||||
process.stdout.close()
|
||||
|
||||
|
||||
def _query(process, sql, timeout=QUERY_TIMEOUT):
|
||||
with _deadline(process, timeout):
|
||||
payload = (sql + '\n').encode('utf-8')
|
||||
if process.stdin.write(payload) != len(payload):
|
||||
raise Failure()
|
||||
process.stdin.flush()
|
||||
line = process.stdout.readline(MAX_METADATA + 1)
|
||||
if not line.endswith(b'\n') or len(line) > MAX_METADATA:
|
||||
raise Failure()
|
||||
try:
|
||||
return json.loads(line)
|
||||
except (ValueError, UnicodeError):
|
||||
raise Failure() from None
|
||||
|
||||
|
||||
OTHER_CLIENTS = """(SELECT count(*) FROM pg_catalog.pg_stat_activity
|
||||
WHERE backend_type = 'client backend' AND pid <> pg_catalog.pg_backend_pid())"""
|
||||
DATABASE_METADATA = """BEGIN ISOLATION LEVEL REPEATABLE READ READ ONLY;
|
||||
SELECT pg_catalog.json_build_object(
|
||||
'version_num', current_setting('server_version_num')::integer,
|
||||
'system_identifier', (SELECT system_identifier::text FROM pg_catalog.pg_control_system()),
|
||||
'database_name', current_database(), 'user_name', current_user,
|
||||
'port', current_setting('port')::integer, 'data_directory', current_setting('data_directory'),
|
||||
'in_recovery', pg_is_in_recovery(), 'read_only', current_setting('transaction_read_only'),
|
||||
'all_sessions_visible', (SELECT rolsuper FROM pg_catalog.pg_roles WHERE rolname = current_user)
|
||||
OR pg_has_role(current_user, 'pg_read_all_stats', 'MEMBER'),
|
||||
'snapshot', pg_export_snapshot(), 'database_bytes', pg_database_size(current_database()),
|
||||
'tables', (SELECT COALESCE(json_agg(c.relname ORDER BY c.relname), '[]'::json)
|
||||
FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE n.nspname = 'public' AND c.relkind = 'r'),
|
||||
'sequences', (SELECT COALESCE(jsonb_agg(jsonb_build_array(n.nspname, c.relname)
|
||||
ORDER BY n.nspname, c.relname), '[]'::jsonb)
|
||||
FROM pg_catalog.pg_class c JOIN pg_catalog.pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE c.relkind = 'S' AND n.nspname <> 'information_schema' AND n.nspname !~ '^pg_'),
|
||||
'other_clients', """ + OTHER_CLIENTS + ');'
|
||||
|
||||
|
||||
def _validate_database(metadata, identity):
|
||||
if (type(metadata.get('version_num')) is not int or metadata['version_num'] // 10000 != 16
|
||||
or metadata.get('system_identifier') != identity['system_identifier']
|
||||
or metadata.get('database_name') != identity['database']
|
||||
or metadata.get('user_name') != identity['user'] or type(metadata.get('port')) is not int
|
||||
or metadata['port'] != identity['port']
|
||||
or not _same_windows_path(metadata.get('data_directory', ''), POSTGRES_DATA)
|
||||
or metadata.get('in_recovery') is not False or metadata.get('read_only') != 'on'
|
||||
or metadata.get('all_sessions_visible') is not True
|
||||
or type(metadata.get('other_clients')) is not int or metadata['other_clients'] != 0
|
||||
or not re.fullmatch(r'[0-9A-Fa-f]+-[0-9A-Fa-f]+-[0-9]+', metadata.get('snapshot', ''))
|
||||
or type(metadata.get('database_bytes')) is not int or metadata['database_bytes'] < 0):
|
||||
raise Failure()
|
||||
tables = metadata.get('tables')
|
||||
if (not isinstance(tables, list) or any(not isinstance(t, str) or not t or '\0' in t for t in tables)
|
||||
or len(tables) != len(set(tables))):
|
||||
raise Failure()
|
||||
sequences = metadata.get('sequences')
|
||||
if (not isinstance(sequences, list)
|
||||
or any(not isinstance(pair, list) or len(pair) != 2
|
||||
or any(not isinstance(name, str) or not name or '\0' in name for name in pair)
|
||||
for pair in sequences)
|
||||
or len(sequences) != len({tuple(pair) for pair in sequences})):
|
||||
raise Failure()
|
||||
|
||||
|
||||
def _sequence_states(process, sequences):
|
||||
states = {}
|
||||
for schema, name in sequences:
|
||||
quoted = '.'.join('"' + part.replace('"', '""') + '"' for part in (schema, name))
|
||||
value = _query(process, "SELECT pg_catalog.json_build_object('last_value', last_value, "
|
||||
"'is_called', is_called) FROM " + quoted + ';')
|
||||
if (not isinstance(value, dict) or set(value) != {'last_value', 'is_called'}
|
||||
or type(value['last_value']) is not int or type(value['is_called']) is not bool):
|
||||
raise Failure()
|
||||
states.setdefault(schema, {})[name] = value
|
||||
return states
|
||||
|
||||
|
||||
def _no_other_clients(process):
|
||||
# Activity views cache within a transaction; explicitly refresh before checking.
|
||||
_query(process, "SELECT json_build_object('cleared', pg_stat_clear_snapshot() IS NULL);")
|
||||
count = _query(process, 'SELECT ' + OTHER_CLIENTS + ';')
|
||||
if type(count) is not int or count != 0:
|
||||
raise Failure()
|
||||
|
||||
|
||||
def _capture_database(source, identity, output, inventory, report):
|
||||
report(6, 0, 0)
|
||||
env = _client_environment(source.dsn)
|
||||
binaries = SOURCE_ROOT / 'runtime/postgres/pgsql/bin'
|
||||
for name in ('psql', 'pg_dump'):
|
||||
path = binaries / (name + '.exe')
|
||||
before = _fingerprint(_file_info(path))
|
||||
result = subprocess.run([str(path), '--version'], stdin=subprocess.DEVNULL, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL, timeout=15, env=env, creationflags=0x08000000,
|
||||
close_fds=True)
|
||||
if (result.returncode != 0 or not re.fullmatch(
|
||||
rb'(?:psql|pg_dump) \(PostgreSQL\) 16(?:\.[0-9]+)*(?: \([^\r\n]*\))?\s*', result.stdout)
|
||||
or _fingerprint(_file_info(path)) != before):
|
||||
raise Failure()
|
||||
source.inputs[path] = before
|
||||
command = [str(binaries / 'psql.exe'), '-X', '-q', '-A', '-t', '-w', '-v', 'ON_ERROR_STOP=1', '-f', '-']
|
||||
with _client(command, env, SESSION_TIMEOUT, interactive=True) as process:
|
||||
metadata = _query(process, DATABASE_METADATA)
|
||||
_validate_database(metadata, identity)
|
||||
file_bytes = sum(entry.size for entry in inventory.files.values())
|
||||
if shutil.disk_usage(output).free < file_bytes + metadata['database_bytes'] + 512 * BLOCK:
|
||||
raise Failure()
|
||||
counts = {}
|
||||
for name in metadata['tables']:
|
||||
quoted = '"' + name.replace('"', '""') + '"'
|
||||
count = _query(process, 'SELECT count(*) FROM ONLY "public".' + quoted + ';', COUNT_TIMEOUT)
|
||||
if type(count) is not int or count < 0:
|
||||
raise Failure()
|
||||
counts[name] = count
|
||||
report(6, len(counts), 0)
|
||||
_no_other_clients(process)
|
||||
# Sequences are not MVCC-isolated, even in this exported-snapshot session.
|
||||
# With the source stopped, require their values to stay fixed across dump.
|
||||
sequences = _sequence_states(process, metadata['sequences'])
|
||||
dump = [str(binaries / 'pg_dump.exe'), '--format=custom', '--no-owner', '--no-acl',
|
||||
'--no-tablespaces', '--compress=1', '--no-password', '--lock-wait-timeout=10s',
|
||||
'--snapshot=' + metadata['snapshot']]
|
||||
dump_env = dict(env, PGOPTIONS=env['PGOPTIONS'].replace(
|
||||
f'statement_timeout={COUNT_TIMEOUT * 1000}', f'statement_timeout={DUMP_TIMEOUT * 1000}'))
|
||||
with _output_file(output / 'database.dump', source.security) as handle:
|
||||
writer = HashWriter(handle)
|
||||
prefix = b''
|
||||
with _client(dump, dump_env, DUMP_TIMEOUT) as dumping:
|
||||
while True:
|
||||
block = dumping.stdout.read(BLOCK)
|
||||
if not block:
|
||||
break
|
||||
if len(prefix) < 5:
|
||||
prefix = (prefix + block)[:5]
|
||||
writer.write(block)
|
||||
if prefix != b'PGDMP' or writer.size <= 5:
|
||||
raise Failure()
|
||||
dump_metadata = writer.metadata()
|
||||
_no_other_clients(process)
|
||||
if _sequence_states(process, metadata['sequences']) != sequences:
|
||||
raise Failure()
|
||||
report(6, len(counts), dump_metadata['bytes'])
|
||||
return {key: metadata[key] for key in ('version_num', 'system_identifier', 'database_name',
|
||||
'user_name', 'port', 'data_directory', 'database_bytes')} | {
|
||||
'table_counts': counts, 'table_count_mode': 'ONLY public ordinary tables; shared exported snapshot',
|
||||
'sequence_states': sequences, 'sequence_count': len(metadata['sequences']),
|
||||
'sequence_state_mode': 'Non-system schemas; non-MVCC values checked unchanged across dump in export session',
|
||||
'schema_migration_counts': {name: counts[name] for name in ('runtime_schema_migrations', 'schema_migrations')
|
||||
if name in counts}, **dump_metadata}
|
||||
|
||||
|
||||
def _stop_confirmed(source, backend, report):
|
||||
retries = 0
|
||||
while True:
|
||||
try:
|
||||
with _silence():
|
||||
result = source.pg.maintenance_stop(source.config, backend=backend)
|
||||
if not result.completed or not result.stopped or backend.probe().kind != source.pg.ProbeKind.STOPPED:
|
||||
raise Failure()
|
||||
return
|
||||
except BaseException:
|
||||
# Even Ctrl-C must not release authority over a possibly live source.
|
||||
retries += 1
|
||||
try:
|
||||
report(7, retries, 0)
|
||||
time.sleep(2)
|
||||
except BaseException:
|
||||
pass
|
||||
|
||||
|
||||
def _unchanged_inputs(source):
|
||||
for path, fingerprint in source.inputs.items():
|
||||
if _fingerprint(_file_info(path)) != fingerprint:
|
||||
raise Failure()
|
||||
|
||||
|
||||
def _publish_manifest(output, security, manifest):
|
||||
temporary = output / 'manifest.json.partial'
|
||||
with _output_file(temporary, security) as handle:
|
||||
writer = HashWriter(handle)
|
||||
for chunk in json.JSONEncoder(ensure_ascii=True, sort_keys=True, indent=2).iterencode(manifest):
|
||||
writer.write(chunk.encode('utf-8'))
|
||||
writer.write(b'\n')
|
||||
# Windows rename refuses an existing destination. A partial JSON is not valid
|
||||
# snapshot authority, even when all preceding large files were completed.
|
||||
os.rename(temporary, output / 'manifest.json')
|
||||
|
||||
|
||||
def capture(output, report):
|
||||
with _defer_signals() as checkpoint:
|
||||
def progress(number, count, size):
|
||||
if number != 7:
|
||||
checkpoint()
|
||||
report(number, count, size)
|
||||
|
||||
_capture(output, progress, checkpoint)
|
||||
|
||||
|
||||
def _capture(output, report, checkpoint):
|
||||
report(2, 0, 0)
|
||||
with _silence():
|
||||
source = _load_source()
|
||||
report(3, 0, 0)
|
||||
with _silence():
|
||||
_prepare_output(output, source.security)
|
||||
authority = source.security.ClusterAuthorityLock(source.config, endpoint_dsn=source.dsn)
|
||||
authority.acquire()
|
||||
backend, attempted, manifest, published = None, False, None, False
|
||||
try:
|
||||
report(4, 0, 0)
|
||||
with _silence():
|
||||
_supervisor_absent(source)
|
||||
identity = source.pg.verify_cluster_identity(source.config)
|
||||
_validate_identity(source, identity)
|
||||
backend = source.pg.PostgresBackend(source.config)
|
||||
if backend.probe().kind != source.pg.ProbeKind.STOPPED:
|
||||
raise Failure()
|
||||
inventory = _inventory()
|
||||
_unchanged_inputs(source)
|
||||
report(4, len(inventory.files), sum(entry.size for entry in inventory.files.values()))
|
||||
try:
|
||||
report(5, 0, 0)
|
||||
with _silence():
|
||||
_supervisor_absent(source)
|
||||
if backend.probe().kind != source.pg.ProbeKind.STOPPED:
|
||||
raise Failure()
|
||||
checkpoint()
|
||||
attempted = True
|
||||
# Never check cancellation inside the original start helper:
|
||||
# Popen precedes _accepted_start_at_monotonic. Its non-raising
|
||||
# signal fence lets that bookkeeping and close() finish first.
|
||||
if source.pg.maintenance_start(source.config, backend=backend).kind != source.pg.ProbeKind.READY:
|
||||
raise Failure()
|
||||
checkpoint()
|
||||
database = _capture_database(source, identity, output, inventory, report)
|
||||
finally:
|
||||
if attempted:
|
||||
report(7, 0, 0)
|
||||
_stop_confirmed(source, backend, report)
|
||||
checkpoint()
|
||||
files, archive = _write_tar(output, source.security, inventory, report)
|
||||
report(9, len(files), archive['bytes'])
|
||||
manifest = {
|
||||
'format': 'truf-windows-snapshot-v1', 'created_at': datetime.now(timezone.utc).isoformat(),
|
||||
'database': database, 'files': files, 'archive': archive,
|
||||
'source': {'root': str(SOURCE_ROOT), 'postgres_data_dir': str(POSTGRES_DATA),
|
||||
'supervisor_stopped': True, 'postgres_stopped': True},
|
||||
'exclusions': {'policy': EXCLUSION_POLICY, 'observed_entries': inventory.exclusions},
|
||||
'counts': {'files': len(files), 'file_bytes': sum(entry['size'] for entry in files),
|
||||
'active_files': sum(not entry['path'].startswith('windows-archive/') for entry in files),
|
||||
'archival_files': sum(entry['path'].startswith('windows-archive/') for entry in files),
|
||||
'public_tables': len(database['table_counts']), 'sequences': database['sequence_count']},
|
||||
}
|
||||
finally:
|
||||
# Do not use the lock's __exit__: an unconfirmed stop must retain it.
|
||||
if attempted:
|
||||
_stop_confirmed(source, backend, report)
|
||||
try:
|
||||
try:
|
||||
if manifest is not None:
|
||||
checkpoint()
|
||||
with _silence():
|
||||
_supervisor_absent(source)
|
||||
_unchanged_inputs(source)
|
||||
_verify_private_acl(output, source.security, directory=True)
|
||||
if _inventory() != inventory:
|
||||
raise Failure()
|
||||
report(10, len(manifest['files']), manifest['archive']['bytes'])
|
||||
_publish_manifest(output, source.security, manifest)
|
||||
published = True
|
||||
finally:
|
||||
with _silence():
|
||||
try:
|
||||
if backend is not None:
|
||||
backend.close()
|
||||
finally:
|
||||
authority.release()
|
||||
checkpoint()
|
||||
except BaseException:
|
||||
if published:
|
||||
(output / 'manifest.json').unlink()
|
||||
raise
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
phase = 1
|
||||
|
||||
def report(number, count=0, size=0):
|
||||
nonlocal phase
|
||||
phase = number
|
||||
print(number, count, size, flush=True)
|
||||
|
||||
environment, paths = os.environ.copy(), sys.path[:]
|
||||
try:
|
||||
if os.name != 'nt' or not (sys.flags.isolated and sys.flags.no_site and sys.dont_write_bytecode):
|
||||
raise Failure()
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('action', choices=('capture',))
|
||||
parser.add_argument('--output', required=True)
|
||||
with _silence():
|
||||
args = parser.parse_args(argv)
|
||||
output = _output_path(args.output)
|
||||
capture(output, report)
|
||||
return 0
|
||||
except BaseException as exc:
|
||||
timed_out = isinstance(exc, subprocess.TimeoutExpired) or isinstance(exc, Failure) and exc.code == 124
|
||||
code = 124 if timed_out else 1
|
||||
print(phase, code, file=sys.stderr, flush=True)
|
||||
return code
|
||||
finally:
|
||||
os.environ.clear()
|
||||
os.environ.update(environment)
|
||||
sys.path[:] = paths
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"schema": 1,
|
||||
"linux": {
|
||||
"aarch64": {
|
||||
"python_image_manifest": "sha256:d04f49f5882f49a3b91f874e75e19f0c265f7222da8659741a9d7eab148f22a9",
|
||||
"trufflehog_archive_sha256": "7e65e771d2a247964056aa5edba0f8ae3945895e5dce867fe0ffbc7b0128239a"
|
||||
},
|
||||
"ca_certificates": "20250419~deb12u1",
|
||||
"debian_snapshot": "20260914T000000Z",
|
||||
"git": "1:2.39.5-0+deb12u3",
|
||||
"tini": "0.19.0-1+b3",
|
||||
"trufflehog_version": "3.97.4",
|
||||
"x86_64": {
|
||||
"python_image_manifest": "sha256:9c47360a2a0355e2da18516d0b1c2126ec22c195d2185e97347c9d98398c5bef",
|
||||
"trufflehog_archive_bytes": 34970205,
|
||||
"trufflehog_archive_sha256": "dc24007c2f233bd61c05beabeb44aa27ea9b43288166279209abe0458c5ce76b"
|
||||
}
|
||||
},
|
||||
"python_image": "python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254",
|
||||
"python_version": "3.12.14",
|
||||
"windows": {
|
||||
"x86_64": {
|
||||
"git": {
|
||||
"archive_bytes": 47241394,
|
||||
"archive_sha256": "50b04b55425b5c465d076cdb184f63a0cd0f86f6ec8bb4d5860114a713d2c29a",
|
||||
"url": "https://github.com/git-for-windows/git/releases/download/v2.47.1.windows.1/MinGit-2.47.1-64-bit.zip",
|
||||
"version": "2.47.1.windows.1"
|
||||
},
|
||||
"python": {
|
||||
"archive_bytes": 11133606,
|
||||
"archive_sha256": "4acbed6dd1c744b0376e3b1cf57ce906f9dc9e95e68824584c8099a63025a3c3",
|
||||
"url": "https://www.python.org/ftp/python/3.12.10/python-3.12.10-embed-amd64.zip",
|
||||
"version": "3.12.10"
|
||||
},
|
||||
"trufflehog": {
|
||||
"archive_bytes": 73316636,
|
||||
"archive_sha256": "6ce9a957ac62bfb19463048333d9e8481327dbbf5bdc0c43f5ab5327b9631fb9",
|
||||
"url": "https://github.com/trufflesecurity/trufflehog/releases/download/v3.97.4/trufflehog_3.97.4_windows_amd64.tar.gz",
|
||||
"version": "3.97.4"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user