Compare commits

..
Author SHA1 Message Date
fred 5c35a250de test(fleet): name what the pane-boundary case's binary check rides on (#1241)
ci/woodpecker/pr/ci Pipeline was successful
Review finding from scooby. This case does not use run_start, so
install_pane_binaries' symlinks land under a home its launcher never consults
(HOME is the trusted parent here). It resolves mosaic and pi through
MOSAIC_RUNTIME_BIN=$FAKE_BIN instead. Valid path, valid green — and a trap for
anyone who later drops that env var believing the symlinks cover it, which
would break the #1241 binary check rather than exercise it.

Comment only; no behavior change. Harness rc=0.

Refs #1241.
2026-08-16 00:21:13 -05:00
fred 10a1f82031 test(fleet): cover the pane-pid-unresolved branch this PR shipped (#1241)
ci/woodpecker/pr/ci Pipeline was canceled
Review finding from scooby: this PR added a failure branch the harness
structurally could not reach. The fake tmux answered `has-session` only for
`=_holder:0.0`, so every non-holder agent landed in the session-is-gone branch
no matter what — the `elif` (tmux still reports the session, no pane PID after
the retries) had zero coverage and no way to get any.

That is the same shape as the bug this PR exists to fix, one layer down: a code
path shipped green where the gate that should measure it cannot. Less severe,
because the branch fails closed at exit 69 rather than reporting success — but
"the harness can't reach it" is the sentence that precedes the next silent
regression, so it gets closed here rather than filed.

`MOSAIC_TEST_HELD_SESSIONS` lets a case name targets the shim should also
answer for. It answers them only AFTER `new-session`, and that detail is the
whole trick: the launcher asks `has-session` about the same name twice — once
at line 255 where a yes means "already running, exit 0", and once at 417 where
a yes means "the session survived". A shim answering yes to both short-circuits
at the first and never reaches the branch under test. It would have looked like
coverage while measuring the idempotency path.

Both failure modes were measured, not reasoned about:
- toggle absent (the old shim): `code=pane-did-not-survive` — the case lands on
  the wrong branch, which is exactly the unreachability being reported.
- toggle answering unconditionally: launcher exits 0 via the idempotency
  short-circuit — "launcher reported success over a session with no resolvable
  pane PID".
- toggle gated on new-session: `code=pane-pid-unresolved`, exit 69.

The case also asserts the diagnostic is not `pane-did-not-survive` and does not
mention the heartbeat, so the two pane faults cannot collapse into one message.

Gates: bash -n · launcher harness rc=0 · test-fleet-units.sh (real tmux) rc=0 ·
fleet specs 342 passed.

Refs #1241.
2026-08-16 00:19:25 -05:00
fred 61a907a12f fix(fleet): fail the agent launcher when the pane cannot survive (#1241)
ci/woodpecker/pr/ci Pipeline was successful
`mosaic fleet start` returned 0 over three dead panes. The launcher knew,
and said the wrong thing at the wrong severity to the wrong layer.

The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared
environment. When that binary is absent the pane dies in under a second,
tmux destroys the session, and the diagnostic goes with it. The launcher
then found no PANE_PID, printed a WARNING about the *heartbeat sidecar*,
and exited 0 — so systemd logged "Finished ... successfully" and
`fleet start` reported success. `fleet ps` was the only component telling
the truth.

Two changes, both in start-agent-session.sh:

1. Before any effect, resolve `mosaic` and the roster's runtime against
   PANE_PATH — the pane's own view of the path, not the launcher's.
   `mosaic yolo <runtime>` calls checkRuntime(runtime) and looks for a
   binary named exactly like the runtime, so this asks the same question
   the pane will ask a moment later, while an operator can still see the
   answer. Absent binary -> exit 69, code=missing-binary, no session
   created.

2. Replace the dead-pane WARNING+exit-0. An absent session one second
   after new-session is a runtime that died on startup, not a heartbeat
   problem -> exit 69, code=pane-did-not-survive, with the command to run
   by hand to see why. A present session with no pane PID after five
   attempts -> code=pane-pid-unresolved. Neither branch kills the
   session; destroying a possibly-live pane on a guess is worse than
   leaving it for inspection.

Exit 69 (EX_UNAVAILABLE) is deliberate: the 64s already in this file mean
the projection was bad, and here the data is fine and the host is not
ready. Callers separate the cases by `code=`, the same way fail_env's
codes share 64.

This propagates for free. `fleet start` calls runChecked() for the holder
and each agent, and runChecked throws on non-zero, so layers 4 and 5 stop
lying without a TypeScript change. Two adjacent defects are left for a
follow-up issue rather than widened into this diff: the per-agent loop
aborts on the first failure instead of attempting all and reporting an
aggregate, and runChecked's bare throw surfaces the launcher's message
under a Node unhandled-rejection stack trace because program.parse() is
synchronous.

Tests:

- test-start-agent-session.sh gains three cases: `mosaic` absent from the
  pane path, the runtime absent from the pane path, and a pane that does
  not survive. Each was verified individually red against the unmodified
  origin/next launcher.
- The two cases asserting a valid launch now supply a pane PID. Until now
  the suite's one success path was itself a dead pane the launcher
  reported as fine.
- The harness fakes `npm` so PANE_PATH stops depending on whatever the
  host has installed, and fails loudly if the host provides `mosaic` or
  `pi` in the system path, where the missing-binary cases would not be
  measurable at all.
- test-fleet-units.sh gains a `pi` shim in its runtime bin. The real-tmux
  harness named `pi` in its roster and never installed it; the new
  preflight caught it.

Refs #1241
2026-08-15 23:56:53 -05:00
mos-dt-0 7a6fb024b4 docs: establish canonical documentation architecture (#1210)
ci/woodpecker/push/publish Pipeline failed
2026-08-13 17:56:13 +00:00
mos-dt-0 f82307c4dc fix(lease): raise capability-probe timeout to 10s on both halves (#869) (#1207)
ci/woodpecker/push/publish Pipeline failed
2026-08-13 17:28:00 +00:00
8 changed files with 511 additions and 19 deletions
+71
View File
@@ -0,0 +1,71 @@
# REPORT A1207
Date: 2026-08-13
Branch: `fix/869-lease-probe-timeout`
Starting head: `2373a5ad345fb316ad2460f6390baab1f45ba08f`
Base: `216cd72226cd9ee17eea461cfe7cd0e010a22f02`
## What changed
- Added Python behavior tests using isolated temporary directories and marker-writing fake `mosaic` executables. They prove that the supplied `PATH` wins over ambient `os.environ["PATH"]`, and that absent or empty supplied `PATH` values do not search ambient paths, platform defaults, or the current directory.
- Bound Python override behavior with executable fakes: a valid `MOSAIC_LEASE_VERSION_PROBE_COMMAND` wins over supplied and ambient `PATH`; an invalid override returns `None` without PATH fallback.
- Added a Python runner binding test that captures kwargs and requires `timeout=10.0`. Existing timeout, transport-error, and nonzero-exit checks remain fail-closed with `None`.
- Added the optional TypeScript dependency-injection seam `CapabilityProbeExecFile`, defaulting to the existing real `execFileSync` implementation. Production callers have no behavior change.
- Added TypeScript tests that capture child-process options and require exactly `timeout: 10_000`. Injected timeout, spawn-error, nonzero-exit, unparseable JSON, and malformed-object cases all return `null`.
- Removed the ambient no-dependency TypeScript smoke case that could execute a built checkout's real CLI. Default resolver and supervisor behavior retain their isolated tests, while capability transport tests now use an isolated artifact or the injected transport.
No Python production code changed relative to `2373a5ad`. The only production delta is the optional TypeScript child-process injection seam.
## Hermeticity incident and correction
An initial ambient-lookup mutation run exposed that the pre-existing Python "not resolvable" test left ambient process PATH uncontrolled. On this host, that mutation resolved and executed the host `mosaic` capability probe. A post-build intermediate TypeScript run also let the pre-existing no-dependency smoke case execute the checkout's built `dist/cli.js` capability probe. No `claude` process was run. I then isolated the Python test's ambient PATH, removed the TypeScript ambient smoke case, repeated the PATH mutation using only marker-writing temporary fakes, and repeated the final suites without either real probe path.
## Mutation evidence
Each mutation was applied independently, its focused suite was run, and the production source was restored before the final run.
| Mutation | Result | Reddened test name(s) |
| ------------------------------------------------------------------------------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `shutil.which("mosaic", path=environ.get("PATH", ""))` to ambient `shutil.which("mosaic")` | RED, three failures | `ProbeActivationCapabilityTest.test_supplied_path_wins_over_ambient_process_path`; `ProbeActivationCapabilityTest.test_absent_or_empty_supplied_path_never_falls_back_or_executes` for both absent and empty PATH subtests |
| Python `PROBE_TIMEOUT_SECONDS: 10.0` to `2.0` | RED, one failure | `ProbeActivationCapabilityTest.test_probe_passes_ten_second_timeout_to_runner` |
| TypeScript `LEASE_CAPABILITY_PROBE_TIMEOUT_MS: 10_000` to `2_000` | RED, one failure | `defaultCapabilityProbe > passes the exact ten-second timeout to the injected child-process transport` |
## Final test run
Dependencies were installed first with `pnpm install --frozen-lockfile`. Workspace dependencies were then built with `pnpm --filter '@mosaicstack/mosaic...' run build` so package type declarations were available.
```text
$ cd packages/mosaic && python3 src/mutator-gate/version_coupling_unittest.py
...................
----------------------------------------------------------------------
Ran 19 tests in 0.007s
OK
$ pnpm exec vitest run src/commands/lease-activation-probe.spec.ts
✓ src/commands/lease-activation-probe.spec.ts (20 tests) 80ms
Test Files 1 passed (1)
Tests 20 passed (20)
```
```text
$ pnpm exec prettier --check packages/mosaic/src/commands/lease-activation-probe.ts packages/mosaic/src/commands/lease-activation-probe.spec.ts
Checking formatting...
All matched files use Prettier code style!
$ pnpm --filter @mosaicstack/mosaic lint
> eslint src
$ pnpm --filter @mosaicstack/mosaic typecheck
> tsc --noEmit
$ python3 -m py_compile packages/mosaic/src/mutator-gate/version_coupling_unittest.py packages/mosaic/framework/tools/lease-broker/activation_version_gate.py
$ git diff --check
```
All commands above exited zero.
## Ambiguities skipped
None.
@@ -128,6 +128,14 @@ EOF
sleep 30
EOF
chmod 700 "$AGENT_BIN/mosaic"
# The launcher resolves the roster's runtime against PANE_PATH before it
# spawns anything (#1241), so the runtime this projection names has to be
# present here even though the fake `mosaic` above never execs it.
cat > "$AGENT_BIN/pi" <<'EOF'
#!/bin/sh
sleep 30
EOF
chmod 700 "$AGENT_BIN/pi"
server_environment_before=$(tmux -L "$TEST_SOCKET" show-environment -g | sort)
server_sessions_before=$(tmux -L "$TEST_SOCKET" list-sessions | sort)
if /usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin MOSAIC_HOME="$AGENT_HOME" \
@@ -286,6 +286,36 @@ _build_runtime_bin_prefix() {
MOSAIC_RUNTIME_BIN_PREFIX=$(_build_runtime_bin_prefix)
PANE_PATH=${MOSAIC_RUNTIME_BIN_PREFIX:+${MOSAIC_RUNTIME_BIN_PREFIX}:}/usr/local/bin:/usr/bin:/bin
# #1241. The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared
# environment. A binary missing from *that* path is a pane that dies in under a
# second, inside a session nobody is attached to, with its diagnostic scrolled
# into a pane tmux then destroys. Resolve both here, before any effect, where
# the failure is still attributable to the thing that caused it.
#
# `mosaic yolo <runtime>` runs checkRuntime(runtime) and the binary it looks for
# is named exactly like the runtime, so resolving the runtime name is the same
# question the pane will ask a moment later — asked while an operator can still
# see the answer.
_resolve_in_pane_path() {
PATH="$PANE_PATH" command -v -- "$1" 2>/dev/null
}
# Exit 69 (EX_UNAVAILABLE): the seat cannot be provided. Distinguished from the
# 64 (EX_USAGE) rejections above, which mean the projection itself was bad —
# here the data is fine and the host is not ready. Callers tell the individual
# cases apart by `code=`, the same way fail_env's many codes share exit 64.
fail_launch() {
local code="$1"
shift
echo "ERROR: agent launch aborted: code=${code} agent=${AGENT_NAME} $*" >&2
exit 69
}
for required_binary in mosaic "$MOSAIC_AGENT_RUNTIME"; do
_resolve_in_pane_path "$required_binary" >/dev/null ||
fail_launch missing-binary "'${required_binary}' is not on the pane PATH (${PANE_PATH})"
done
_ensure_claude_workdir_trusted() {
local workdir="$1"
local resolved
@@ -384,6 +414,19 @@ if [ -n "$PANE_PID" ]; then
_start_heartbeat_sidecar "$AGENT_NAME" "$PANE_PID" \
"$MOSAIC_HEARTBEAT_RUN_DIR" "$MOSAIC_HEARTBEAT_INTERVAL" || \
echo "WARNING: heartbeat sidecar could not be started for $AGENT_NAME" >&2
elif _tmux has-session -t "=${AGENT_NAME}:0.0" 2>/dev/null; then
# #1241. Session present, no pane PID after a second of retries. Whatever this
# is, it is not a seat an operator can use, so it is not a success either.
fail_launch pane-pid-unresolved \
"tmux reports the session but no pane PID after 5 attempts"
else
echo "WARNING: could not resolve pane PID for $AGENT_NAME — heartbeat sidecar not started" >&2
# #1241. This branch used to print a WARNING about the heartbeat sidecar and
# exit 0. It is not a heartbeat problem: tmux destroys a session when its pane
# command exits, so an absent session one second after new-session means the
# runtime died on startup. Reporting it as success is what let `fleet start`
# return 0 over three dead panes — the launcher knew, and said the wrong thing
# at the wrong severity to the wrong layer.
fail_launch pane-did-not-survive \
"the pane exited immediately and tmux destroyed the session;" \
"run 'mosaic yolo ${MOSAIC_AGENT_RUNTIME}' in ${MOSAIC_AGENT_WORKDIR} to see why"
fi
@@ -23,8 +23,26 @@ index=0
if [ "${args[0]:-}" = -L ]; then index=2; fi
case "${args[$index]:-}" in
has-session)
# The holder always answers. MOSAIC_TEST_HELD_SESSIONS lets a case add
# other targets that should answer too — without it there is no way to
# model "tmux still reports the session" for a non-holder agent, and the
# launcher's pane-pid-unresolved branch is unreachable from this harness.
#
# A listed target answers only AFTER new-session, because the launcher asks
# this question twice about the same name: once before launching, where a
# yes means "already running, nothing to do, exit 0", and once after, where
# a yes means "the session survived". A shim that answered yes to both
# would short-circuit at the first and never reach the branch under test —
# it would look like coverage and measure the idempotency path instead.
for argument in "${args[@]}"; do
[ "$argument" = '=_holder:0.0' ] && exit 0
case " ${MOSAIC_TEST_HELD_SESSIONS:-} " in
*" $argument "*)
if tr '\0' '\n' < "${MOSAIC_TEST_TMUX_CALLS:?}" | grep -qxF new-session; then
exit 0
fi
;;
esac
done
exit 1
;;
@@ -62,6 +80,30 @@ env -0 > "${MOSAIC_HOME:?}/fleet/pane-environment"
SHIM
chmod +x "$FAKE_BIN/mosaic"
# The runtime the rosters below name. The launcher resolves it against PANE_PATH
# before spawning (#1241), so it has to exist somewhere the pane would find it —
# not merely on the launcher's own PATH.
printf '#!/usr/bin/env bash\nexit 0\n' > "$FAKE_BIN/pi"
chmod +x "$FAKE_BIN/pi"
# PANE_PATH is derived partly from `npm config get prefix`. Left to the real npm
# it would splice whatever the host has installed into the path under test, and
# the missing-binary cases below would pass or fail by accident of the machine.
cat > "$FAKE_BIN/npm" <<'SHIM'
#!/usr/bin/env bash
printf '%s\n' "${MOSAIC_TEST_NPM_PREFIX:-/nonexistent}"
SHIM
chmod +x "$FAKE_BIN/npm"
# PANE_PATH always ends in the system path. A host that installs these there can
# not measure the missing-binary cases at all, and a green run would mean
# nothing — so say so instead of passing.
for host_binary in mosaic pi; do
if PATH=/usr/local/bin:/usr/bin:/bin command -v "$host_binary" >/dev/null 2>&1; then
fail "host provides '$host_binary' in the system path; missing-binary cases are not measurable here"
fi
done
write_generated() {
local home="$1"
local agent="$2"
@@ -81,6 +123,19 @@ MOSAIC_TMUX_SOCKET=mosaic-test
EOF
chmod 600 "$home/fleet/agents/$agent.env.generated"
mkdir -p "$home/work"
install_pane_binaries "$home"
}
# `$PANE_HOME/.npm-global/bin` is one of the prefixes the launcher folds into
# PANE_PATH, so this is the pane's own view of "installed", distinct from the
# launcher's PATH. Tests that need a binary *absent* remove it from here.
install_pane_binaries() {
local pane_home="$1"
mkdir -p "$pane_home/.npm-global/bin"
local binary
for binary in mosaic pi; do
ln -sf "$FAKE_BIN/$binary" "$pane_home/.npm-global/bin/$binary"
done
}
run_start() {
@@ -88,6 +143,7 @@ run_start() {
local agent="$2"
HOME="$home" PATH="$FAKE_BIN:$PATH" MOSAIC_TEST_TMUX_CALLS="$TMUX_CALLS" \
MOSAIC_TEST_PANE_PID="${MOSAIC_TEST_PANE_PID:-}" \
MOSAIC_TEST_HELD_SESSIONS="${MOSAIC_TEST_HELD_SESSIONS:-}" \
MOSAIC_TEST_HOME="$home" \
MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \
MOSAIC_HOME="$home" "$START" "$agent"
@@ -98,7 +154,10 @@ run_start() {
HOME_VALID="$ROOT/valid"
AGENT_VALID="coder0"
write_generated "$HOME_VALID" "$AGENT_VALID"
run_start "$HOME_VALID" "$AGENT_VALID"
# A live pane PID is part of what "valid launch" means. Until #1241 this case
# ran with none, so the suite's one success path was itself a dead pane the
# launcher reported as fine.
MOSAIC_TEST_PANE_PID=$$ run_start "$HOME_VALID" "$AGENT_VALID"
valid_args=$(tr '\0' '\n' < "$TMUX_CALLS")
echo "$valid_args" | grep -qF new-session || fail "valid generated projection did not reach tmux"
echo "$valid_args" | grep -qF 'mosaic' || fail "fixed mosaic launcher command missing"
@@ -245,6 +304,13 @@ PANE_BASH_ENV="$ROOT/pane-boundary.bash-env"
printf 'MOSAIC_RUNTIME_BIN=%s\n' "$FAKE_BIN" > \
"$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local"
chmod 600 "$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local"
# This case does not go through run_start, so its pane binaries come from
# MOSAIC_RUNTIME_BIN=$FAKE_BIN in the env.local written above — not from the
# symlinks install_pane_binaries planted under the generated home, which this
# launcher never consults because HOME here is the trusted parent. That is a
# legitimate resolution path, but it means dropping MOSAIC_RUNTIME_BIN from
# this case on the belief that the symlinks cover it would break the #1241
# binary check rather than exercise it.
LD_PRELOAD='/not/loaded/by-clean-bootstrap.so' \
BASH_ENV="$PANE_BASH_ENV" \
MOSAIC_UNTRUSTED_SENTINEL='must-not-reach-pane' \
@@ -258,6 +324,7 @@ PATH="$PANE_STALE_PATH" \
"MOSAIC_TEST_HOME=$PANE_TRUSTED_HOME" \
MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \
MOSAIC_TEST_EXECUTE_PANE=1 \
"MOSAIC_TEST_PANE_PID=$$" \
"$START" coder-pane-boundary
pane_args=$(tr '\0' '\n' < "$TMUX_CALLS")
echo "$pane_args" | grep -qxF "HOME=$PANE_TRUSTED_HOME" || \
@@ -392,6 +459,75 @@ echo "$interaction_policy_args" | grep -qF 'new-session' && \
echo "$output" | grep -qF 'operator interaction service requires runtime pi' || \
fail "interaction pinned-policy check did not follow strict parsing"
# #1241. The pane runs `mosaic yolo <runtime>` against PANE_PATH. A binary
# missing from that path is a launch failure, and it has to be named before the
# session is created — after it, the diagnostic dies with the pane.
assert_missing_pane_binary_rejected() {
local binary="$1"
local home="$ROOT/missing-$binary"
local agent="coder-missing-$binary"
write_generated "$home" "$agent"
rm -f "$home/.npm-global/bin/$binary"
: > "$TMUX_CALLS"
local output
if output=$(MOSAIC_TEST_PANE_PID=$$ run_start "$home" "$agent" 2>&1); then
fail "launch succeeded with '$binary' absent from the pane PATH"
fi
echo "$output" | grep -qF 'code=missing-binary' || fail "missing '$binary' diagnostic missing"
echo "$output" | grep -qF "'$binary'" || fail "missing-binary diagnostic did not name $binary"
if tr '\0' '\n' < "$TMUX_CALLS" | grep -qF new-session; then
fail "launcher created a session it knew would die ($binary absent)"
fi
}
assert_missing_pane_binary_rejected mosaic
assert_missing_pane_binary_rejected pi
# #1241. tmux destroys a session when its pane command exits, so no pane PID a
# second after new-session means the runtime died on startup. This used to be a
# WARNING about the heartbeat sidecar followed by exit 0 — three layers above it
# then reported a fleet that was not running.
: > "$TMUX_CALLS"
HOME_DEAD_PANE="$ROOT/dead-pane"
write_generated "$HOME_DEAD_PANE" "coder-dead-pane"
if output=$(MOSAIC_TEST_PANE_PID='' run_start "$HOME_DEAD_PANE" coder-dead-pane 2>&1); then
fail "launcher reported success over a pane that did not survive"
fi
echo "$output" | grep -qF 'code=pane-did-not-survive' || fail "dead-pane diagnostic missing"
if echo "$output" | grep -qiF 'heartbeat'; then
fail "dead pane is still being reported as a heartbeat-sidecar problem"
fi
tr '\0' '\n' < "$TMUX_CALLS" | grep -qF new-session || \
fail "dead-pane case did not reach the launch it is measuring"
# #1241, the other way a pane fails. Above, tmux destroyed the session and
# has-session said so. Here the session is still there and no PID comes back
# after the retries — a different fault (the pane is alive but unusable, or
# tmux is answering inconsistently) that an operator has to be told apart from
# a runtime that died on startup.
#
# This case exists because the branch that handles it shipped with nothing able
# to reach it: the shim answered has-session only for the holder, so every
# non-holder agent landed in the session-is-gone branch no matter what. A
# defensive branch nothing exercises is the same shape as the bug this whole
# change is about, one layer down.
: > "$TMUX_CALLS"
HOME_NO_PID="$ROOT/pane-no-pid"
write_generated "$HOME_NO_PID" "coder-no-pid"
if output=$(MOSAIC_TEST_PANE_PID='' MOSAIC_TEST_HELD_SESSIONS='=coder-no-pid:0.0' \
run_start "$HOME_NO_PID" coder-no-pid 2>&1); then
fail "launcher reported success over a session with no resolvable pane PID"
fi
echo "$output" | grep -qF 'code=pane-pid-unresolved' || \
fail "session-present/no-PID was not reported as pane-pid-unresolved: $output"
if echo "$output" | grep -qF 'code=pane-did-not-survive'; then
fail "a session tmux still reports was diagnosed as a destroyed session"
fi
if echo "$output" | grep -qiF 'heartbeat'; then
fail "an unresolvable pane PID is still being reported as a heartbeat-sidecar problem"
fi
# Exact stop derives the socket exclusively from the validated generated
# projection and ignores an ambient socket supplied by the caller.
: > "$TMUX_CALLS"
@@ -62,7 +62,14 @@ EXPECTED_ACTIVATION_CAPABILITY: Final[ActivationCapability] = {
# capability as compact JSON.
LEASE_CAPABILITY_PROBE_COMMAND: Final = "__lease-capability"
PROBE_TIMEOUT_SECONDS: Final = 2.0
# Budget for the out-of-process `mosaic __lease-capability` probe. The CLI
# is a Node program whose cold start alone measures 2.2-2.3s on a mid-range
# workstation (sb-it-1-dt, 2026-08-13), so a 2s budget made every launch on
# such hosts fail closed with the #869 skew message even though the
# capability matched. The timeout only bounds the pathological hang case —
# the happy path returns as soon as the probe exits — so a generous budget
# costs nothing on healthy hosts.
PROBE_TIMEOUT_SECONDS: Final = 10.0
# Override hook: a full shell-style command line (parsed with `shlex.split`)
# to run INSTEAD of resolving `mosaic` on PATH and appending the probe
@@ -88,7 +95,13 @@ def _resolve_probe_command(environ: Mapping[str, str]) -> list[str] | None:
if override:
parsed = shlex.split(override)
return parsed or None
resolved = shutil.which("mosaic")
# Resolve against the PROVIDED environment's PATH, not the ambient
# os.environ. Before this, a test passing a hermetic environ still
# resolved (and spawned) the host's real `mosaic` — masked only on hosts
# where the real probe happened to exceed the old 2s timeout. No PATH in
# the provided environment means nothing is resolvable (fail-closed),
# matching the probe's overall contract.
resolved = shutil.which("mosaic", path=environ.get("PATH", ""))
if resolved is None:
return None
return [resolved, LEASE_CAPABILITY_PROBE_COMMAND]
@@ -7,11 +7,13 @@ import { fileURLToPath } from 'node:url';
import {
LEASE_ACTIVATION_CAPABILITY,
LEASE_CAPABILITY_PROBE_COMMAND,
LEASE_CAPABILITY_PROBE_TIMEOUT_MS,
defaultCapabilityProbe,
defaultResolveCliEntry,
defaultSupervisorProbe,
leaseEnforcementActivatable,
registerLeaseCapabilityProbe,
type CapabilityProbeExecFile,
type LeaseActivationCapability,
type SupervisorProbeResult,
} from './lease-activation-probe.js';
@@ -35,6 +37,17 @@ const presentSupervisor: SupervisorProbeResult = {
socketPath: '/run/user/1000/mosaic-lease/broker.sock',
};
function withScratchCli<T>(run: (cliPath: string) => T): T {
const scratchDir = mkdtempSync(join(tmpdir(), 'mosaic-lease-capability-probe-'));
try {
const cliPath = join(scratchDir, 'cli.js');
writeFileSync(cliPath, '// isolated fake; injected execFile means this is never executed\n');
return run(cliPath);
} finally {
rmSync(scratchDir, { recursive: true, force: true });
}
}
describe('leaseEnforcementActivatable', () => {
it('is false when the activation capability is absent (null)', () => {
const result = leaseEnforcementActivatable({
@@ -100,15 +113,6 @@ describe('leaseEnforcementActivatable', () => {
});
expect(result).toBe(true);
});
it('uses the real default probes when no deps are injected (does not throw)', () => {
// No live broker / built CLI is guaranteed in a test environment, so this
// only asserts the predicate degrades to a safe boolean rather than
// throwing — the fail-closed behavior itself is covered by the injected
// cases above.
expect(() => leaseEnforcementActivatable()).not.toThrow();
expect(typeof leaseEnforcementActivatable()).toBe('boolean');
});
});
describe('defaultCapabilityProbe', () => {
@@ -127,6 +131,61 @@ describe('defaultCapabilityProbe', () => {
expect(result).toBeNull();
});
it('passes the exact ten-second timeout to the injected child-process transport', () => {
withScratchCli((cliPath) => {
let captured:
| {
file: string;
args: string[];
options: Parameters<CapabilityProbeExecFile>[2];
}
| undefined;
const execFile: CapabilityProbeExecFile = (file, args, options) => {
captured = { file, args, options };
return JSON.stringify(LEASE_ACTIVATION_CAPABILITY);
};
const result = defaultCapabilityProbe({ resolveCliEntry: () => cliPath, execFile });
expect(result).toEqual(LEASE_ACTIVATION_CAPABILITY);
expect(captured).toEqual({
file: process.execPath,
args: [cliPath, LEASE_CAPABILITY_PROBE_COMMAND],
options: {
encoding: 'utf-8',
timeout: 10_000,
stdio: ['ignore', 'pipe', 'ignore'],
},
});
expect(captured?.options.timeout).toBe(LEASE_CAPABILITY_PROBE_TIMEOUT_MS);
});
});
it.each([
['timeout', Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })],
['spawn error', Object.assign(new Error('spawn failed'), { code: 'ENOENT' })],
['nonzero exit', Object.assign(new Error('child exited 1'), { status: 1 })],
])('returns null (fail-closed) on child-process %s', (_failure, error) => {
withScratchCli((cliPath) => {
const execFile: CapabilityProbeExecFile = () => {
throw error;
};
expect(defaultCapabilityProbe({ resolveCliEntry: () => cliPath, execFile })).toBeNull();
});
});
it.each([
['unparseable JSON', 'not-json'],
['malformed object', JSON.stringify({ name: LEASE_ACTIVATION_CAPABILITY.name })],
])('returns null (fail-closed) on %s output', (_failure, output) => {
withScratchCli((cliPath) => {
const execFile: CapabilityProbeExecFile = () => output;
expect(defaultCapabilityProbe({ resolveCliEntry: () => cliPath, execFile })).toBeNull();
});
});
describe('positive path — injected resolver, isolated scratch dir (never the real dist/)', () => {
// A prior version of this test staged the stub cli.js at the package's
// REAL resolved dist/ path and relied on afterEach to clean up "only
@@ -55,6 +55,19 @@ export const LEASE_ACTIVATION_CAPABILITY: LeaseActivationCapability = {
/** Hidden CLI probe subcommand name — wired via {@link registerLeaseCapabilityProbe}. */
export const LEASE_CAPABILITY_PROBE_COMMAND = '__lease-capability';
/**
* Budget for the out-of-process capability probe. The probe launches a fresh
* Node process on the built CLI entrypoint, whose cold start alone measures
* 2.2-2.3s on a mid-range workstation (sb-it-1-dt, 2026-08-13) — so the
* previous 2s budget made the probe time out and report NO capability on
* such hosts, failing every launch with the #869 skew message even though
* the capability matched. The timeout only bounds the pathological hang
* case; the happy path returns as soon as the probe exits. Mirrors
* PROBE_TIMEOUT_SECONDS in the enforcement half
* (framework/tools/lease-broker/activation_version_gate.py).
*/
export const LEASE_CAPABILITY_PROBE_TIMEOUT_MS = 10_000;
function capabilityMatches(candidate: LeaseActivationCapability | null): boolean {
return (
candidate !== null &&
@@ -110,12 +123,28 @@ export function defaultResolveCliEntry(
return join(dirname(mainEntry), 'cli.js');
}
/** Narrow injectable seam for the synchronous child process used by the
* capability probe. */
export type CapabilityProbeExecFile = (
file: string,
args: string[],
options: {
encoding: BufferEncoding;
timeout: number;
stdio: ['ignore', 'pipe', 'ignore'];
},
) => string;
/** Injectable inputs for {@link defaultCapabilityProbe}. */
export interface CapabilityProbeDeps {
/** Resolve the CLI entrypoint (`cli.js`) to probe. Defaults to
* {@link defaultResolveCliEntry}. Inject to point at an isolated scratch
* location in tests — never at the real package's `dist/`. */
resolveCliEntry?: () => string;
/** Execute the resolved CLI entrypoint. Defaults to the real
* `execFileSync`. Inject so transport behavior and options can be tested
* without spawning a process. */
execFile?: CapabilityProbeExecFile;
}
/**
@@ -139,9 +168,10 @@ export function defaultCapabilityProbe(
const cliEntry = resolveCliEntry();
if (!existsSync(cliEntry)) return null;
const output = execFileSync(process.execPath, [cliEntry, LEASE_CAPABILITY_PROBE_COMMAND], {
const execFile: CapabilityProbeExecFile = deps.execFile ?? execFileSync;
const output = execFile(process.execPath, [cliEntry, LEASE_CAPABILITY_PROBE_COMMAND], {
encoding: 'utf-8',
timeout: 2000,
timeout: LEASE_CAPABILITY_PROBE_TIMEOUT_MS,
stdio: ['ignore', 'pipe', 'ignore'],
});
@@ -24,11 +24,15 @@ from __future__ import annotations
import importlib.util
import io
import os
import shlex
import subprocess
import sys
import tempfile
import unittest
from contextlib import redirect_stderr
from pathlib import Path
from unittest import mock
TOOLS_DIR = Path(__file__).parents[2] / "framework/tools/lease-broker"
@@ -57,6 +61,20 @@ def matching_capability() -> dict[str, object]:
return dict(VERSION_GATE.EXPECTED_ACTIVATION_CAPABILITY)
def write_fake_mosaic(directory: Path, marker: Path) -> Path:
directory.mkdir(parents=True, exist_ok=True)
executable = directory / "mosaic"
executable.write_text(
"#!/bin/sh\n"
f"printf '%s\\n' executed >> {shlex.quote(str(marker))}\n"
"printf '%s\\n' "
"'{\"name\":\"lease-runtime-activation\",\"version\":1}'\n",
encoding="utf-8",
)
executable.chmod(0o755)
return executable
class AssertActivationCapabilityMatchesTest(unittest.TestCase):
"""Unit-level coverage of `activation_version_gate.py`'s own assertion,
isolated from the launch-runtime.py seam it is wired into below."""
@@ -110,11 +128,102 @@ class ProbeActivationCapabilityTest(unittest.TestCase):
handling — never spawns a real `mosaic` process."""
def test_returns_none_when_mosaic_is_not_resolvable_on_path(self) -> None:
result = VERSION_GATE.default_probe_activation_capability(
{"PATH": "/nonexistent-bin-dir-for-869-c4-test"}
)
# Keep even a deliberate ambient-lookup mutation away from any host
# installation. The dedicated hermeticity tests below provide fake
# ambient executables and markers.
with mock.patch.dict(
os.environ, {"PATH": "/nonexistent-ambient-bin-dir-for-869-c4-test"}
):
result = VERSION_GATE.default_probe_activation_capability(
{"PATH": "/nonexistent-bin-dir-for-869-c4-test"}
)
self.assertIsNone(result)
def test_supplied_path_wins_over_ambient_process_path(self) -> None:
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
supplied_marker = root / "supplied.marker"
ambient_marker = root / "ambient.marker"
supplied_bin = root / "supplied-bin"
ambient_bin = root / "ambient-bin"
write_fake_mosaic(supplied_bin, supplied_marker)
write_fake_mosaic(ambient_bin, ambient_marker)
with mock.patch.dict(os.environ, {"PATH": str(ambient_bin)}):
result = VERSION_GATE.default_probe_activation_capability(
{"PATH": str(supplied_bin)}
)
self.assertEqual(result, matching_capability())
self.assertTrue(supplied_marker.exists())
self.assertFalse(ambient_marker.exists())
def test_absent_or_empty_supplied_path_never_falls_back_or_executes(self) -> None:
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
ambient_marker = root / "ambient.marker"
current_directory_marker = root / "current-directory.marker"
ambient_bin = root / "ambient-bin"
current_directory = root / "current-directory"
write_fake_mosaic(ambient_bin, ambient_marker)
write_fake_mosaic(current_directory, current_directory_marker)
original_directory = Path.cwd()
try:
os.chdir(current_directory)
with mock.patch.dict(os.environ, {"PATH": str(ambient_bin)}):
for supplied_environment in ({}, {"PATH": ""}):
with self.subTest(environ=supplied_environment):
result = VERSION_GATE.default_probe_activation_capability(
supplied_environment
)
self.assertIsNone(result)
self.assertFalse(ambient_marker.exists())
self.assertFalse(current_directory_marker.exists())
finally:
os.chdir(original_directory)
def test_valid_override_wins_and_invalid_override_does_not_fall_back_to_path(
self,
) -> None:
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
supplied_marker = root / "supplied.marker"
ambient_marker = root / "ambient.marker"
override_marker = root / "override.marker"
supplied_bin = root / "supplied-bin"
ambient_bin = root / "ambient-bin"
override_bin = root / "override-bin"
write_fake_mosaic(supplied_bin, supplied_marker)
write_fake_mosaic(ambient_bin, ambient_marker)
override_executable = write_fake_mosaic(override_bin, override_marker)
with mock.patch.dict(os.environ, {"PATH": str(ambient_bin)}):
result = VERSION_GATE.default_probe_activation_capability(
{
"PATH": str(supplied_bin),
VERSION_GATE.MOSAIC_COMMAND_OVERRIDE_VAR: str(override_executable),
}
)
self.assertEqual(result, matching_capability())
self.assertTrue(override_marker.exists())
self.assertFalse(supplied_marker.exists())
self.assertFalse(ambient_marker.exists())
override_marker.unlink()
result = VERSION_GATE.default_probe_activation_capability(
{
"PATH": str(supplied_bin),
VERSION_GATE.MOSAIC_COMMAND_OVERRIDE_VAR: str(
root / "invalid-override" / "mosaic"
),
}
)
self.assertIsNone(result)
self.assertFalse(override_marker.exists())
self.assertFalse(supplied_marker.exists())
self.assertFalse(ambient_marker.exists())
def test_override_command_is_parsed_and_the_probe_subcommand_is_not_double_appended(
self,
) -> None:
@@ -135,6 +244,29 @@ class ProbeActivationCapabilityTest(unittest.TestCase):
self.assertEqual(result, {"name": "lease-runtime-activation", "version": 1})
self.assertEqual(captured, [["/fake/mosaic", "__lease-capability"]])
def test_probe_passes_ten_second_timeout_to_runner(self) -> None:
captured_argv: list[str] = []
captured_kwargs: dict[str, object] = {}
class FakeCompleted:
returncode = 0
stdout = '{"name": "lease-runtime-activation", "version": 1}'
def fake_run(argv: list[str], **kwargs: object) -> FakeCompleted:
captured_argv.extend(argv)
captured_kwargs.update(kwargs)
return FakeCompleted()
result = VERSION_GATE.default_probe_activation_capability(
{VERSION_GATE.MOSAIC_COMMAND_OVERRIDE_VAR: "/fake/mosaic"},
run=fake_run,
)
self.assertEqual(result, matching_capability())
self.assertEqual(captured_argv, ["/fake/mosaic"])
self.assertEqual(captured_kwargs["timeout"], 10.0)
self.assertEqual(captured_kwargs["check"], False)
def test_fails_closed_on_nonzero_exit_malformed_json_and_missing_fields(self) -> None:
class NonZeroExit:
returncode = 1
@@ -174,7 +306,7 @@ class ProbeActivationCapabilityTest(unittest.TestCase):
def test_fails_closed_on_timeout_and_transport_error(self) -> None:
def timeout_run(*_args: object, **_kwargs: object) -> None:
raise subprocess.TimeoutExpired(cmd="mosaic", timeout=2.0)
raise subprocess.TimeoutExpired(cmd="mosaic", timeout=10.0)
def oserror_run(*_args: object, **_kwargs: object) -> None:
raise OSError("no such file or directory")