stack/scripts/testing/README.md

# Gate lanes

This tooling is for Sage's granted test gates on dragon-lin. It does not grant
itself a slot. A grant reference is an audit field, not an authentication token.
Until Sage adopts these lanes, the existing shared-slot procedure still applies.

Ask Sage with the lane, candidate SHA and exact command. Get a separate grant
for image preparation, suites, and the whole-slot paired proof. New state is
`/mnt/storage/scratch/mosaic-gate-lanes`. Old slot files are untouched.

Invoke the entry from the builder or canonical repository. The --checkout target
must be a separate clean scratch clone containing identical tooling source:

```bash
scripts/testing/gate-lane.sh run --lane a --candidate FULL_SHA \
  --checkout /mnt/storage/scratch/ROW-RUN/wt \
  --out /mnt/storage/scratch/ROW-RUN/out --grant SAGE_MESSAGE_ID \
  --image-receipt /mnt/storage/scratch/PREP/image-receipt.json
```

Each run creates a private root like `/mnt/storage/scratch/ga-ABCDEF` (30 bytes).
An explicit `--temp-root` must already be private, outside `/tmp`, and no longer
than 30 UTF-8 bytes after resolving symlinks. The sandbox's own `/tmp` and HOME
are unchanged. Host HOME stays real. Steps get `PI_AUTH_FILE` naming a private `{}`
fixture in the temp root, and the Docker guard refuses a Compose run whose
`PI_AUTH_FILE` isn't a regular file inside the temp root (row 244). A listener covering loopback port 9, an
unsafe privileged-port threshold or unreadable host inventory refuses startup.

Lane a leases ports 61000..61127, lane b 61128..61255. Refuse any band overlapping
the host's current ephemeral range or a live listener. Port-0 fixtures with a
held server keep using port 0. Never replace the literal dead Board :9 URLs.
Two Node test files run per lane, including nested shell-suite Node runners.

```bash
scripts/testing/gate-lane.sh status
scripts/testing/gate-lane.sh recover --run-id RUN_NONCE --grant SAGE_MESSAGE_ID
```

Status reports each lane, the image lock, unfinished receipts and unmanaged
test/landing/build processes as LEGACY BUSY. It prints no command lines or
environment values. A dead owner with an unfinished receipt is quarantined even
when flock is free. Recovery takes both lanes and the exclusive image lock.
Unknown process-group, scope or Docker ownership refuses recovery. Do not remove
lock files, old receipts, or another run's resources to clear a refusal.

An image receipt records exact COPY input hashes/modes, image ID, release tag,
base digest and platform. Both concurrent candidates must match those files and
that same image ID. Rebuilding identical files is not equivalent. Warm gates
refuse implicit Docker builds/pulls and recheck the ID at each step and the end.
External same-uid commands can ignore these locks. Sage must exclude other
builders; this is test coordination, not a security boundary against that uid.

Only under an explicit image-build grant:

```bash
scripts/testing/gate-lane.sh prepare-image --lane a --candidate FULL_SHA \
  --checkout /mnt/storage/scratch/ROW-PREP/wt \
  --out /mnt/storage/scratch/ROW-PREP/out --grant SAGE_MESSAGE_ID
```

Preparation holds both lanes and the exclusive image lock. Its build log,
receipt, count and wall time are separate from warm gate timings. The current
Containerfile uses a mutable FROM tag and apt indexes. A missing base digest
refuses provenance publication. No credential file is copied or printed.

Run records are immutable JSON files published by link after fsync. Every step
has a log, UTC/monotonic times, exit code, pass/fail/skip counts and log hash.
The first failed step stops the gate. Cleanup touches only exact project labels,
recorded process identities and fixture paths. A process leak fails the gate
even when cleanup succeeds. Failed logs and temp roots remain for diagnosis.
A cleanup failure leaves the lane quarantined. GATE_PASSED requires all steps
and cleanup. Required prerequisite skips fail the run; optional skips remain
visible and require acceptance review.

Home metadata is evidence only. Assert only on entries attributable to this
run's nonce or fixture paths. Other sessions normally change ~/.claude, ~/.pi
and ~/.cache, so there is no global unchanged-home assertion.

The row 197 acceptance packet is `agents/herb/work/queue-197/`: two candidate
solo baselines followed by three overlapping pairs, lane assignment swapped in
pair two. Keep later paired results after adoption. Six green gates establish
the initial acceptance bar, not a reliable estimate of rare flakes.

The preflight unit fixture uses the canonical checkout's existing node_modules
at `/mnt/storage/src/mosaic-stack/node_modules` through the required plain
symlink. It does not install dependencies. Unit run roots use the `gt-` prefix;
real lane roots use `ga-` or `gb-`. Resource helper tests inside a lane lease only
that lane's band through MOSAIC_GATE_STATE, including their child test files.
The exhaustion probe temporarily narrows its process-local band to a port it
already leased, so it never reserves the whole shared band.

`closed` call-site options are accepted for compatibility; every dynamic lease
reserves an unbound endpoint. Listener fixtures decide when to bind and close.
Fixture child caches put TMPDIR under `tmp/` next to `jiti/` and `cache/`.

Outside a lane, the resource helper file uses lane a with the shared default
state `/mnt/storage/scratch/mosaic-gate-lanes`. It leaves stable lock files in
place. A single-file run therefore respects a live lane-a gate's closed leases.
Its unrelated-session probe bounds both polling children at five seconds.