truf-server/deploy/edge/README.md

# Production edge deployment

This opt-in deployment keeps PostgreSQL, supervisor control, the standalone dashboard,
the Worker API, and its typed admin backend on the runtime container's loopback. The
root-owned deployment profile selects one of two exact Caddy topologies. Both keep the
private Caddy admin API on an unpublished Unix socket and preserve the same Worker API,
admin authentication, operator attribution, denylist, and header contract.

## Deployment profiles

If `/etc/truf/deployment-profile` is absent, `standalone-edge-v1` is selected. The
standalone profile publishes runtime TCP 443 and gives the managed edge only
`NET_BIND_SERVICE`.

For a host whose existing root-owned Caddy must remain the sole owner of ports 80/443,
install the shared profile before running the host-agent installer:

```sh
printf '%s\n' shared-host-edge-v1 | sudo install -m 0444 -o root -g root /dev/stdin /etc/truf/deployment-profile
```

`shared-host-edge-v1` runs the runtime in the host network namespace with no Docker
published ports. The managed Truf edge shares that namespace, has no capabilities, and
binds plain HTTP only at `127.0.0.1:18766`. The existing host Caddy imports the fixed
route-only `deploy/edge/host-caddy-shared.caddy` snippet inside the reviewed public site.
Install that import before any catch-all handler. It handles only `/api/v1/worker/*` and
the exact random admin prefix; it does not define a listener, TLS policy, global option,
or route for another application. The host agent never restarts or reconfigures host
Caddy or X-UI.

Profile changes are maintenance operations: stop the host agent first, require no active
apply or failed hold, install the exact root-owned mode-0444 value, validate the selected
Compose projection and host-Caddy configuration, then restart the agent. Never expose
profile selection through the admin or host-agent request.

### Choosing a topology

Use `standalone-edge-v1` on a dedicated host where the managed edge can own public TCP
443. The request path is:

```text
Internet -> managed Caddy :443 -> private runtime :8766
```

Use `shared-host-edge-v1` only when an existing root-owned Caddy must remain the sole
owner of public ports and TLS. The request path is:

```text
Internet -> host Caddy :443 -> 127.0.0.1:18766 -> managed Caddy -> private runtime :8766
```

Shared-host mode adds a one-time integration boundary, not a second public edge. The
operator installs the fixed route snippet, places its import before every catch-all,
supplies the independent ingress marker, and validates the complete host Caddy
configuration. After that bootstrap, runtime restart and document apply use the same
host-agent lifecycle as standalone mode. The host agent never owns the host Caddy
configuration or service lifecycle.

These are the only supported production topologies. Nginx, Traefik, an arbitrary Caddy
layout, or an ad-hoc Compose override is not equivalent to either profile. Add and test a
new exact deployment profile instead of translating private headers approximately. The
host agent validates the selected fixed Compose projection and rejects metadata drift.

The shared-host projection currently carries the constrained-host runtime limits declared
in `compose.shared-host.yaml`. A materially different CPU or memory envelope also requires
a reviewed profile change; do not hide it in an unvalidated local override.

### End-to-end host bootstrap

The repository provides fixed deployment components, not a universal VPS installer,
Ansible role, public image registry, or infrastructure module. Bootstrap a new host from
one reviewed release checkout as follows:

1. Install the reviewed Linux, Docker Engine and Compose plugin, systemd, Python 3, and
   fail2ban prerequisites; provision DNS and the selected TLS ownership boundary.
2. Install the release checkout root-owned at `/opt/truf` and choose exactly one deployment
   profile before installing the host agent.
3. In shared-host mode, install the fixed host-Caddy import, validate the complete host
   configuration, and prove an unavailable loopback edge cannot fall through to another
   application.
4. Create the protected edge directories, denylist state, and mode-0600 edge environment
   described below. Generate independent admin, edge, and shared-ingress values rather
   than copying values from another host.
5. Install and validate the fixed host agent. Its first install seeds an absent active
   config from `app/config.linux.yaml` and an absent secrets document as an empty mapping;
   repeat installation never replaces active documents.
6. Install every trusted worker-package manifest referenced by the runtime config beneath
   `/etc/truf/worker-packages` with the exact ownership and mode described below.
7. Review the private active config and secrets, then build `runtime` and `edge` from the
   same checkout with the exact base and selected profile Compose files.
8. Start the stack, explicitly enable Worker API and admin only after their private
   configuration is complete, and verify PostgreSQL, runtime, edge, HTTPS, admin, Worker
   API, host-agent, and unrelated host applications.

Initial host bootstrap is therefore intentionally more manual than later operation.
Normal config apply, restart, rollback, status, and audit are performed through the typed
control plane and fixed host agent after this trust boundary is established.

## Host agent and fixed runtime paths

Install the root-owned checkout at `/opt/truf`. Before invoking the host-agent installer,
prepare the edge state below and create the complete protected environment file that its
fixed combined-Compose validation consumes.

UID/GID 10001 is the numeric edge identity. The denylist directory is mounted, rather than
its file, so atomic replacement remains visible in the container.

```sh
sudo install -d -o 10001 -g 10001 -m 0700 /var/log/truf-edge
sudo install -d -o root -g 10001 -m 2750 /etc/truf-edge/denylist
sudo install -m 0640 -o root -g 10001 deploy/edge/admin-denylist.caddy /etc/truf-edge/denylist/admin-denylist.caddy
sudo install -d -o root -g root -m 0700 /var/lib/truf-edge
sudo install -m 0750 -o root -g root deploy/fail2ban/truf_caddy_admin_denylist.py /usr/local/sbin/truf-caddy-admin-denylist
```

Create `/etc/truf-edge/edge.env` as root with mode 0600. Generate a new admin segment
with `openssl rand -hex 32`. It must be exactly 64 lowercase hex characters (256 random
bits). Generate the bcrypt value interactively with the pinned edge image's
`caddy hash-password` command; never put the plaintext password in a command, file, or
Compose variable.

```dotenv
TRUF_EDGE_HOST=edge.example.net
TRUF_ADMIN_PREFIX=replace_with_64_lowercase_hex_characters
TRUF_ADMIN_USER=operator
TRUF_ADMIN_PASSWORD_HASH='$2a$14$replace_with_a_real_caddy_bcrypt_hash'
TRUF_ADMIN_EDGE_MARKER=replace_with_a_second_independent_64_character_hex_secret
TRUF_SHARED_INGRESS_MARKER=replace_with_a_third_independent_64_character_hex_secret
TRUF_EDGE_AUTH_LOG_DIR=/var/log/truf-edge
TRUF_EDGE_DENYLIST_DIR=/etc/truf-edge/denylist
```

`TRUF_SHARED_INGRESS_MARKER` is required only by `shared-host-edge-v1`. Host Caddy strips
any inbound transit/private headers, injects this marker and its observed client address,
and proxies to loopback. The managed edge rejects a missing marker before trusting that
address and removes the marker before proxying to the application.

Now install and validate the fixed host agent. The installer creates the fixed candidate,
result, PostgreSQL socket, and active-document paths and enables
`/run/truf/host-agent.sock`; Compose refuses to create missing bind sources.

```sh
sudo /usr/bin/python3 -I -S -B /opt/truf/deploy/host-agent/truf_host_agent_install.py install
sudo /usr/bin/python3 -I -S -B /opt/truf/deploy/host-agent/truf_host_agent_install.py validate
```

The active `/etc/truf/runtime/config.yaml` and `secrets.yaml` are UID/GID 10001 mode 0600
documents and are never overwritten by repeat installation. Any package manifest referenced
by the config must be installed beneath `/etc/truf/worker-packages` as a root-owned,
root:root mode 0644 regular file before validation. The runtime maps that immutable authority
read-only at `/data/worker-packages`; do not place manifests in the private active-document
directory.

Automatic TLS remains the default. A deployment that must use operator-provided
certificates can mount a root-owned, non-link `*.caddy` file under `/etc/caddy/tls` and
set `TRUF_EDGE_TLS_INCLUDE` to that absolute container path in a reviewed Compose
override. The include should contain only the site's `tls CERT KEY` directive. Never use
the repository's localhost test certificate or key in a deployment.

The normal `compose.yaml` remains private and unchanged. Confirm the host-agent socket is
active and rerun installer validation immediately before starting production edge. Always
supply the base file, the exact selected profile file, and the protected environment file.
For standalone:

```sh
docker compose --env-file /etc/truf-edge/edge.env -f compose.yaml -f compose.edge.yaml build runtime edge
docker compose --env-file /etc/truf-edge/edge.env -f compose.yaml -f compose.edge.yaml up -d
```

For shared host:

```sh
docker compose --env-file /etc/truf-edge/edge.env -f compose.yaml -f compose.shared-host.yaml build runtime edge
docker compose --env-file /etc/truf-edge/edge.env -f compose.yaml -f compose.shared-host.yaml up -d
```

Before starting shared host, validate the complete existing host Caddy configuration with
the snippet import in place. A matching request must fail at that Truf route if the
loopback edge is unavailable; it must never fall through to X-UI or another upstream.

Provisioning creates the private `/data/managed-files` namespace in the named data volume.
Each configured writable root must be a reviewed immediate child such as
`/data/managed-files/exports`, created with UID/GID 10001 and mode 0700 while the runtime is
stopped. Arbitrary host bind paths are not managed-file roots.

Worker admission remains disabled by `app/config.linux.yaml`. Configure the private
runtime config's worker sources, compatibility profiles, and hashed device credentials
before explicitly enabling `supervisor.worker_api.enabled`. The edge does not enable it.
The typed admin backend is disabled independently under `supervisor.worker_api.admin`.
Set its exact `origin` to `https://TRUF_EDGE_HOST`, set `edge_marker` to the same independent
256-bit value as `TRUF_ADMIN_EDGE_MARKER`, then explicitly enable it. Both authenticated
surfaces share the private runtime loopback port 8766. Caddy strips any inbound
`X-Truf-Admin-Edge` and `X-Truf-Admin-Operator`, sets the configured marker and the
authenticated Basic-auth username only after authentication, and rewrites the public
random prefix to the private `/admin-internal` backend path. The backend accepts the
operator identity only together with the private marker.
Worker API requests receive neither private admin header.

Build and client bootstrap instructions for Windows and Linux remote workers are in
`docs/remote-worker-operations.md`. Worker executables and images must be produced from a
reviewed release checkout; operators must not assemble Python, Git, TruffleHog, detector
policy, or dependencies manually on each worker.

## Fail2ban

Install the host files under their conventional names and enable fail2ban plus the expiry
timer. The jail counts only redacted `admin_auth_failure` JSON records. An initial Basic
challenge without credentials, Worker API authentication failures, and unrelated 404s do
not enter that log. The action changes only the matcher imported inside the secret admin
route; it does not create firewall rules and therefore does not block workers sharing an IP.

```sh
sudo install -m 0644 deploy/fail2ban/filter.d-truf-admin-auth.conf /etc/fail2ban/filter.d/truf-admin-auth.conf
sudo install -m 0644 deploy/fail2ban/jail.d-truf-admin-auth.local /etc/fail2ban/jail.d/truf-admin-auth.local
sudo install -m 0644 deploy/fail2ban/action.d-truf-caddy-admin-denylist.conf /etc/fail2ban/action.d/truf-caddy-admin-denylist.conf
sudo install -m 0644 deploy/fail2ban/fail2ban.d-truf-persistence.local /etc/fail2ban/fail2ban.d/truf-persistence.local
sudo install -m 0644 deploy/systemd/truf-caddy-admin-denylist-expire.service /etc/systemd/system/
sudo install -m 0644 deploy/systemd/truf-caddy-admin-denylist-expire.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now fail2ban truf-caddy-admin-denylist-expire.timer
sudo fail2ban-client status truf-admin-auth
```

Fail2ban persists jail state in `/var/lib/fail2ban/fail2ban.sqlite3`. The updater persists
canonical IPs and expiry timestamps in private
`/var/lib/truf-edge/admin-denylist.json`. It validates the complete Caddyfile in the running
edge container, reloads it through the private admin endpoint, and restores/reloads the
previous state if a command fails. Reload uses Caddy's private `/run/caddy-admin.sock` inside the edge
container. The socket is not mounted or published. Shared mode renders denylist matchers
against the marker-authenticated client address; standalone mode uses the direct peer.

## SSH recovery

Use fail2ban's normal unban first so its database and Caddy agree:

```sh
sudo fail2ban-client set truf-admin-auth unbanip 203.0.113.10
sudo /usr/local/sbin/truf-caddy-admin-denylist status
sudo /usr/local/sbin/truf-caddy-admin-denylist expire
```

If fail2ban is unavailable, run the updater's explicit unban over SSH:

```sh
sudo /usr/local/sbin/truf-caddy-admin-denylist unban 203.0.113.10
```

For recovery from a damaged generated snippet, stop the expiry timer and fail2ban, restore
`deploy/edge/admin-denylist.caddy` to `/etc/truf-edge/denylist/admin-denylist.caddy`, then
run Caddy validation and reload through the private Unix admin socket only after validation
succeeds. Reconcile each
remaining address with the updater before re-enabling the services. Do not use a global
firewall ban as a shortcut.