Compare commits

..
Author SHA1 Message Date
fred b61789fe26 fix(fleet): tell the operator when the fleet transport is missing (#1240)
ci/woodpecker/pr/ci Pipeline was successful
`mosaic fleet --help` reads "Manage the local Mosaic tmux fleet" and every
roster the CLI scaffolds sets `transport: tmux`, but neither `tools/install.sh`
nor `tools/_scripts/mosaic-doctor` contained the string "tmux" at all. A
greenfield host therefore came out of the installer able to install a fleet,
start a fleet, and run no seat, with `mosaic fleet ps` as the operator's first
and only signal.

Measured on mosaic-sbx-dev (Debian, no tmux, framework installed): `mosaic-doctor`
reported 11 warnings and not one of them named the reason no seat could launch.

The installer gets a warning, not a `require_cmd` hard failure: tmux is required
by the fleet, not by mosaic. Hosts that install this to run `mosaic claude` and
never scaffold a roster are common, and failing their install over a binary they
do not need would be wrong. The check runs in `--check` mode too — "what is the
state of this host" is the question `--check` is asked.

Both checks read the roster's own `transport:` rather than assuming tmux, so a
host declaring something else is pointed at the binary it actually needs instead
of at the wrong package.

The two implementations are deliberately parallel and each carries a comment
pointing at the other. They are separate because the installer must answer this
before the framework's own scripts are guaranteed to be on disk. One harness
drives BOTH from the shipped text — the functions are extracted from the scripts
by awk rather than copied — so the pair cannot drift silently, and the test
cannot keep passing after the shipped copy changes.

The harness is wired into `test:framework-shell`. Without that it would have
tripped the #1017 enumeration guard as UNENUMERATED, which is the guard doing
its job: a check nothing runs is not a check.

Evidence:
- red: the harness fails against origin/next ("could not extract
  fleet_declared_transport"); `grep -ci tmux` on both files at origin/next = 0.
- green on real hosts, all four branches:
  - dev (no tmux, no roster)  -> WARN naming tmux, points at `mosaic fleet init`
  - dev (no tmux, v2 roster)  -> WARN naming the roster, points at `mosaic fleet start`
  - dev installer --check     -> WARN saying start "reports success and no seat comes up"
  - canary (tmux present)     -> `[OK] Fleet transport available: tmux` under --verbose,
                                 silent by default (pass() is verbose-gated), installer silent
- harness green on node:24-alpine/busybox, the CI base image.
- `bash -n` x3, `pnpm typecheck` 45/45, fleet specs 342 passed,
  enumeration guard OK, its self-test OK, prettier clean.

Refs #1240. Upstream of #1237/#1243 and #1241/#1244: a correct fix for either of
those still leaves this host with no live seat.
2026-08-16 00:15:50 -05:00
mos-dt-0 7a6fb024b4 docs: establish canonical documentation architecture (#1210)
ci/woodpecker/push/publish Pipeline failed
2026-08-13 17:56:13 +00:00
mos-dt-0 f82307c4dc fix(lease): raise capability-probe timeout to 10s on both halves (#869) (#1207)
ci/woodpecker/push/publish Pipeline failed
2026-08-13 17:28:00 +00:00
9 changed files with 628 additions and 18 deletions
+71
View File
@@ -0,0 +1,71 @@
# REPORT A1207
Date: 2026-08-13
Branch: `fix/869-lease-probe-timeout`
Starting head: `2373a5ad345fb316ad2460f6390baab1f45ba08f`
Base: `216cd72226cd9ee17eea461cfe7cd0e010a22f02`
## What changed
- Added Python behavior tests using isolated temporary directories and marker-writing fake `mosaic` executables. They prove that the supplied `PATH` wins over ambient `os.environ["PATH"]`, and that absent or empty supplied `PATH` values do not search ambient paths, platform defaults, or the current directory.
- Bound Python override behavior with executable fakes: a valid `MOSAIC_LEASE_VERSION_PROBE_COMMAND` wins over supplied and ambient `PATH`; an invalid override returns `None` without PATH fallback.
- Added a Python runner binding test that captures kwargs and requires `timeout=10.0`. Existing timeout, transport-error, and nonzero-exit checks remain fail-closed with `None`.
- Added the optional TypeScript dependency-injection seam `CapabilityProbeExecFile`, defaulting to the existing real `execFileSync` implementation. Production callers have no behavior change.
- Added TypeScript tests that capture child-process options and require exactly `timeout: 10_000`. Injected timeout, spawn-error, nonzero-exit, unparseable JSON, and malformed-object cases all return `null`.
- Removed the ambient no-dependency TypeScript smoke case that could execute a built checkout's real CLI. Default resolver and supervisor behavior retain their isolated tests, while capability transport tests now use an isolated artifact or the injected transport.
No Python production code changed relative to `2373a5ad`. The only production delta is the optional TypeScript child-process injection seam.
## Hermeticity incident and correction
An initial ambient-lookup mutation run exposed that the pre-existing Python "not resolvable" test left ambient process PATH uncontrolled. On this host, that mutation resolved and executed the host `mosaic` capability probe. A post-build intermediate TypeScript run also let the pre-existing no-dependency smoke case execute the checkout's built `dist/cli.js` capability probe. No `claude` process was run. I then isolated the Python test's ambient PATH, removed the TypeScript ambient smoke case, repeated the PATH mutation using only marker-writing temporary fakes, and repeated the final suites without either real probe path.
## Mutation evidence
Each mutation was applied independently, its focused suite was run, and the production source was restored before the final run.
| Mutation | Result | Reddened test name(s) |
| ------------------------------------------------------------------------------------------ | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `shutil.which("mosaic", path=environ.get("PATH", ""))` to ambient `shutil.which("mosaic")` | RED, three failures | `ProbeActivationCapabilityTest.test_supplied_path_wins_over_ambient_process_path`; `ProbeActivationCapabilityTest.test_absent_or_empty_supplied_path_never_falls_back_or_executes` for both absent and empty PATH subtests |
| Python `PROBE_TIMEOUT_SECONDS: 10.0` to `2.0` | RED, one failure | `ProbeActivationCapabilityTest.test_probe_passes_ten_second_timeout_to_runner` |
| TypeScript `LEASE_CAPABILITY_PROBE_TIMEOUT_MS: 10_000` to `2_000` | RED, one failure | `defaultCapabilityProbe > passes the exact ten-second timeout to the injected child-process transport` |
## Final test run
Dependencies were installed first with `pnpm install --frozen-lockfile`. Workspace dependencies were then built with `pnpm --filter '@mosaicstack/mosaic...' run build` so package type declarations were available.
```text
$ cd packages/mosaic && python3 src/mutator-gate/version_coupling_unittest.py
...................
----------------------------------------------------------------------
Ran 19 tests in 0.007s
OK
$ pnpm exec vitest run src/commands/lease-activation-probe.spec.ts
✓ src/commands/lease-activation-probe.spec.ts (20 tests) 80ms
Test Files 1 passed (1)
Tests 20 passed (20)
```
```text
$ pnpm exec prettier --check packages/mosaic/src/commands/lease-activation-probe.ts packages/mosaic/src/commands/lease-activation-probe.spec.ts
Checking formatting...
All matched files use Prettier code style!
$ pnpm --filter @mosaicstack/mosaic lint
> eslint src
$ pnpm --filter @mosaicstack/mosaic typecheck
> tsc --noEmit
$ python3 -m py_compile packages/mosaic/src/mutator-gate/version_coupling_unittest.py packages/mosaic/framework/tools/lease-broker/activation_version_gate.py
$ git diff --check
```
All commands above exited zero.
## Ambiguities skipped
None.
@@ -225,6 +225,54 @@ else
warn "mosaic-ensure-sequential-thinking helper missing"
fi
# Fleet transport binary (#1240).
#
# `mosaic fleet --help` reads "Manage the local Mosaic tmux fleet" and every
# roster the CLI scaffolds sets `transport: tmux`, but nothing in the install
# path provides tmux and, until now, nothing here noticed it was absent. On a
# greenfield host that produced a fleet which installed clean, started clean,
# and had no live seat; `mosaic fleet ps` was the operator's first and only
# signal that anything was wrong.
#
# The roster's own `transport:` is read rather than assumed, so a host that
# declares something other than tmux is told about the binary it actually
# needs. Absent a roster the check still runs — `mosaic fleet init` will
# scaffold a tmux fleet on this host, and finding out beforehand is the point.
#
# `tools/install.sh` carries a deliberately parallel check at the end of its
# summary. The two are separate because the installer must be able to say this
# before the framework's own scripts are guaranteed to be on disk; keep their
# wording in step.
fleet_declared_transport() {
local roster="$MOSAIC_HOME/fleet/roster.yaml"
local declared=""
if [[ -f "$roster" ]]; then
declared="$(sed -n 's/^[[:space:]]*transport:[[:space:]]*//p' "$roster" | head -1 |
tr -d '"'\''' | tr -d '\r' | awk '{print $1}')"
fi
printf '%s\n' "${declared:-tmux}"
}
check_fleet_transport() {
local transport
transport="$(fleet_declared_transport)"
if command -v "$transport" >/dev/null 2>&1; then
pass "Fleet transport available: $transport"
return
fi
if [[ -f "$MOSAIC_HOME/fleet/roster.yaml" ]]; then
warn "Fleet transport '$transport' is not installed — this host has a roster and no seat can launch. Install it (e.g. sudo apt-get install -y $transport), then 'mosaic fleet start'."
else
warn "Fleet transport '$transport' is not installed — 'mosaic fleet' cannot run seats here. Install it (e.g. sudo apt-get install -y $transport) before 'mosaic fleet init'."
fi
}
check_fleet_transport
# Legacy migration surfaces should no longer contain symlink trees.
legacy_paths=(
"$HOME/.claude/agent-guides"
@@ -0,0 +1,215 @@
#!/usr/bin/env bash
# Covers the #1240 fleet-transport checks in `mosaic-doctor` and in
# `tools/install.sh`.
#
# Both checks answer the same question — "can a seat actually launch on this
# host?" — from two different places, because the installer has to be able to
# answer it before the framework's own scripts are guaranteed to be on disk.
# Two implementations of one rule is exactly the shape that drifts, so this
# harness drives BOTH, in one file, from the same table of cases.
#
# The functions are extracted from the shipped scripts rather than copied here.
# A test that carries its own copy of the logic is a test that keeps passing
# after the shipped copy changes — the failure mode this whole change is about.
# Extraction is by exact function header and a closing brace in column one; if
# either script is reshaped so that stops matching, the extraction yields
# nothing and this fails loudly instead of silently measuring an empty string.
set -euo pipefail
SCRIPT_DIR=$(cd -- "$(dirname -- "$0")" && pwd)
DOCTOR="$SCRIPT_DIR/mosaic-doctor"
# framework/tools/_scripts -> framework/tools -> framework -> mosaic -> packages -> repo
INSTALLER=$(cd -- "$SCRIPT_DIR/../../../../.." && pwd)/tools/install.sh
fail() {
echo "FAIL: $*" >&2
exit 1
}
[ -f "$DOCTOR" ] || fail "missing mosaic-doctor at $DOCTOR"
[ -f "$INSTALLER" ] || fail "missing install.sh at $INSTALLER"
ROOT=$(mktemp -d)
trap 'rm -rf "$ROOT"' EXIT
# The cases below run with PATH set to a directory that deliberately does not
# contain a shell, and a PATH assignment on a command also governs how that
# command is looked up — so bash has to be named absolutely or it becomes the
# thing that is missing.
BASH_BIN=$(command -v bash) || fail "host is missing 'bash'"
# A PATH containing exactly the utilities these functions use and nothing else.
# The absent-transport cases are only meaningful on a PATH where the transport
# is genuinely unresolvable, and this host (like most) has tmux in /usr/bin —
# so the system path cannot be part of the path under test.
FAKE_BIN="$ROOT/bin"
mkdir -p "$FAKE_BIN"
for utility in sed head tr awk; do
utility_path=$(command -v "$utility") || fail "host is missing '$utility'"
ln -s "$utility_path" "$FAKE_BIN/$utility"
done
if PATH="$FAKE_BIN" command -v tmux >/dev/null 2>&1; then
fail "'tmux' is resolvable on the minimal test path; absent-transport cases are not measurable"
fi
# Extract a function by its exact header, up to a closing brace in column one.
extract_function() {
local source_file="$1"
local function_name="$2"
local destination="$3"
awk -v name="$function_name" '
$0 == name "() {" { collecting = 1 }
collecting { print }
collecting && $0 == "}" { exit }
' "$source_file" > "$destination"
grep -qF "$function_name() {" "$destination" ||
fail "could not extract '$function_name' from $source_file — has it been renamed or reshaped?"
# An unterminated extraction would be a syntax error the moment it is sourced,
# but saying so here names the cause instead of leaving a bash parse error.
bash -n "$destination" ||
fail "extracted '$function_name' does not parse; the closing brace was probably not found"
}
extract_function "$DOCTOR" fleet_declared_transport "$ROOT/doctor-declared.sh"
extract_function "$DOCTOR" check_fleet_transport "$ROOT/doctor-check.sh"
extract_function "$INSTALLER" check_fleet_transport "$ROOT/installer-check.sh"
# Build a MOSAIC_HOME, optionally with a roster declaring a transport.
make_home() {
local home="$ROOT/$1"
local declared="${2-}"
rm -rf "$home"
mkdir -p "$home"
if [ -n "$declared" ]; then
mkdir -p "$home/fleet"
cat > "$home/fleet/roster.yaml" <<EOF
version: 2
generation: 1
transport: $declared
agents: []
EOF
fi
printf '%s\n' "$home"
}
# Run the doctor's check against a given home and path, capturing which
# reporter the check chose. The real `pass` prints only under `--verbose` and
# the real `warn` always prints; these stubs make both unconditional on
# purpose, because what is under test is the severity the check selects, not
# whether the default verbosity happens to show it. A check that warned where
# it should pass would otherwise be invisible here.
run_doctor_check() {
local home="$1"
local path="$2"
MOSAIC_HOME="$home" PATH="$path" "$BASH_BIN" --noprofile --norc -c '
set -euo pipefail
warn() { echo "[WARN] $*"; }
pass() { echo "[OK] $*"; }
MOSAIC_HOME="$1"
source "$2"
source "$3"
check_fleet_transport
' _ "$home" "$ROOT/doctor-declared.sh" "$ROOT/doctor-check.sh" 2>&1
}
run_installer_check() {
local home="$1"
local path="$2"
MOSAIC_HOME="$home" PATH="$path" "$BASH_BIN" --noprofile --norc -c '
set -euo pipefail
warn() { echo "[WARN] $*"; }
C="" RESET=""
MOSAIC_HOME="$1"
source "$2"
check_fleet_transport
' _ "$home" "$ROOT/installer-check.sh" 2>&1
}
# A transport that exists. Named tmux because that is what the default roster
# declares; the binary never runs, it only has to resolve.
PRESENT_BIN="$ROOT/present-bin"
mkdir -p "$PRESENT_BIN"
printf '#!/usr/bin/env bash\nexit 0\n' > "$PRESENT_BIN/tmux"
chmod +x "$PRESENT_BIN/tmux"
PATH_WITH_TMUX="$PRESENT_BIN:$FAKE_BIN"
# ── absent, no roster ────────────────────────────────────────────────────────
# Nothing has been configured yet, so the honest thing to point at is `init`.
home=$(make_home no-roster)
output=$(run_doctor_check "$home" "$FAKE_BIN")
echo "$output" | grep -qF '[WARN]' || fail "doctor did not warn when tmux was absent"
echo "$output" | grep -qF 'tmux' || fail "doctor warning did not name the transport"
echo "$output" | grep -qF 'mosaic fleet init' || fail "doctor did not point a rosterless host at init"
output=$(run_installer_check "$home" "$FAKE_BIN")
echo "$output" | grep -qF '[WARN]' || fail "installer did not warn when tmux was absent"
echo "$output" | grep -qF 'reports success and no seat comes up' ||
fail "installer warning did not say what the missing transport actually breaks"
# ── absent, roster present ───────────────────────────────────────────────────
# A configured fleet that cannot launch is a stronger statement than a
# hypothetical one, and the message says so.
home=$(make_home with-roster tmux)
output=$(run_doctor_check "$home" "$FAKE_BIN")
echo "$output" | grep -qF '[WARN]' || fail "doctor did not warn with a roster present and tmux absent"
echo "$output" | grep -qF 'roster' || fail "doctor did not mention the roster it found"
echo "$output" | grep -qF 'mosaic fleet start' || fail "doctor did not point a configured host at start"
# ── present ──────────────────────────────────────────────────────────────────
# Silence from the installer, and a pass (not a warning) from the audit.
for home_name in no-roster with-roster; do
home="$ROOT/$home_name"
output=$(run_doctor_check "$home" "$PATH_WITH_TMUX")
if echo "$output" | grep -qF '[WARN]'; then
fail "doctor warned about the transport while tmux was present ($home_name)"
fi
echo "$output" | grep -qF '[OK]' || fail "doctor did not record a pass with tmux present ($home_name)"
output=$(run_installer_check "$home" "$PATH_WITH_TMUX")
if [ -n "$output" ]; then
fail "installer was not silent with tmux present ($home_name): $output"
fi
done
# ── the roster declares something other than tmux ────────────────────────────
# The roster is read, not assumed. A host that declares a different transport
# is told about the binary it actually needs, and never about tmux — being sent
# to install the wrong package is worse than no advice at all.
home=$(make_home other-transport zellij)
output=$(run_doctor_check "$home" "$PATH_WITH_TMUX")
echo "$output" | grep -qF 'zellij' || fail "doctor ignored the roster's declared transport"
if echo "$output" | grep -qF 'tmux'; then
fail "doctor named tmux for a host whose roster declares zellij"
fi
output=$(run_installer_check "$home" "$PATH_WITH_TMUX")
echo "$output" | grep -qF 'zellij' || fail "installer ignored the roster's declared transport"
if echo "$output" | grep -qF 'tmux'; then
fail "installer named tmux for a host whose roster declares zellij"
fi
# ── a quoted or trailing-comment transport value ─────────────────────────────
# YAML permits both and neither is exotic; a check that installs `tmux"` or
# reads `tmux # default` as a binary name would send the operator nowhere.
home=$(make_home quoted-transport '"tmux" # the only transport today')
output=$(run_doctor_check "$home" "$PATH_WITH_TMUX")
echo "$output" | grep -qF '[OK] Fleet transport available: tmux' ||
fail "doctor did not parse a quoted/commented transport value: $output"
output=$(run_installer_check "$home" "$PATH_WITH_TMUX")
if [ -n "$output" ]; then
fail "installer did not parse a quoted/commented transport value: $output"
fi
echo "ok - fleet transport checks (mosaic-doctor + install.sh)"
@@ -62,7 +62,14 @@ EXPECTED_ACTIVATION_CAPABILITY: Final[ActivationCapability] = {
# capability as compact JSON.
LEASE_CAPABILITY_PROBE_COMMAND: Final = "__lease-capability"
PROBE_TIMEOUT_SECONDS: Final = 2.0
# Budget for the out-of-process `mosaic __lease-capability` probe. The CLI
# is a Node program whose cold start alone measures 2.2-2.3s on a mid-range
# workstation (sb-it-1-dt, 2026-08-13), so a 2s budget made every launch on
# such hosts fail closed with the #869 skew message even though the
# capability matched. The timeout only bounds the pathological hang case —
# the happy path returns as soon as the probe exits — so a generous budget
# costs nothing on healthy hosts.
PROBE_TIMEOUT_SECONDS: Final = 10.0
# Override hook: a full shell-style command line (parsed with `shlex.split`)
# to run INSTEAD of resolving `mosaic` on PATH and appending the probe
@@ -88,7 +95,13 @@ def _resolve_probe_command(environ: Mapping[str, str]) -> list[str] | None:
if override:
parsed = shlex.split(override)
return parsed or None
resolved = shutil.which("mosaic")
# Resolve against the PROVIDED environment's PATH, not the ambient
# os.environ. Before this, a test passing a hermetic environ still
# resolved (and spawned) the host's real `mosaic` — masked only on hosts
# where the real probe happened to exceed the old 2s timeout. No PATH in
# the provided environment means nothing is resolvable (fail-closed),
# matching the probe's overall contract.
resolved = shutil.which("mosaic", path=environ.get("PATH", ""))
if resolved is None:
return None
return [resolved, LEASE_CAPABILITY_PROBE_COMMAND]
+1 -1
View File
@@ -25,7 +25,7 @@
"lint": "eslint src",
"typecheck": "tsc --noEmit",
"test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh"
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh"
},
"dependencies": {
"@mosaicstack/brain": "workspace:*",
@@ -7,11 +7,13 @@ import { fileURLToPath } from 'node:url';
import {
LEASE_ACTIVATION_CAPABILITY,
LEASE_CAPABILITY_PROBE_COMMAND,
LEASE_CAPABILITY_PROBE_TIMEOUT_MS,
defaultCapabilityProbe,
defaultResolveCliEntry,
defaultSupervisorProbe,
leaseEnforcementActivatable,
registerLeaseCapabilityProbe,
type CapabilityProbeExecFile,
type LeaseActivationCapability,
type SupervisorProbeResult,
} from './lease-activation-probe.js';
@@ -35,6 +37,17 @@ const presentSupervisor: SupervisorProbeResult = {
socketPath: '/run/user/1000/mosaic-lease/broker.sock',
};
function withScratchCli<T>(run: (cliPath: string) => T): T {
const scratchDir = mkdtempSync(join(tmpdir(), 'mosaic-lease-capability-probe-'));
try {
const cliPath = join(scratchDir, 'cli.js');
writeFileSync(cliPath, '// isolated fake; injected execFile means this is never executed\n');
return run(cliPath);
} finally {
rmSync(scratchDir, { recursive: true, force: true });
}
}
describe('leaseEnforcementActivatable', () => {
it('is false when the activation capability is absent (null)', () => {
const result = leaseEnforcementActivatable({
@@ -100,15 +113,6 @@ describe('leaseEnforcementActivatable', () => {
});
expect(result).toBe(true);
});
it('uses the real default probes when no deps are injected (does not throw)', () => {
// No live broker / built CLI is guaranteed in a test environment, so this
// only asserts the predicate degrades to a safe boolean rather than
// throwing — the fail-closed behavior itself is covered by the injected
// cases above.
expect(() => leaseEnforcementActivatable()).not.toThrow();
expect(typeof leaseEnforcementActivatable()).toBe('boolean');
});
});
describe('defaultCapabilityProbe', () => {
@@ -127,6 +131,61 @@ describe('defaultCapabilityProbe', () => {
expect(result).toBeNull();
});
it('passes the exact ten-second timeout to the injected child-process transport', () => {
withScratchCli((cliPath) => {
let captured:
| {
file: string;
args: string[];
options: Parameters<CapabilityProbeExecFile>[2];
}
| undefined;
const execFile: CapabilityProbeExecFile = (file, args, options) => {
captured = { file, args, options };
return JSON.stringify(LEASE_ACTIVATION_CAPABILITY);
};
const result = defaultCapabilityProbe({ resolveCliEntry: () => cliPath, execFile });
expect(result).toEqual(LEASE_ACTIVATION_CAPABILITY);
expect(captured).toEqual({
file: process.execPath,
args: [cliPath, LEASE_CAPABILITY_PROBE_COMMAND],
options: {
encoding: 'utf-8',
timeout: 10_000,
stdio: ['ignore', 'pipe', 'ignore'],
},
});
expect(captured?.options.timeout).toBe(LEASE_CAPABILITY_PROBE_TIMEOUT_MS);
});
});
it.each([
['timeout', Object.assign(new Error('timed out'), { code: 'ETIMEDOUT' })],
['spawn error', Object.assign(new Error('spawn failed'), { code: 'ENOENT' })],
['nonzero exit', Object.assign(new Error('child exited 1'), { status: 1 })],
])('returns null (fail-closed) on child-process %s', (_failure, error) => {
withScratchCli((cliPath) => {
const execFile: CapabilityProbeExecFile = () => {
throw error;
};
expect(defaultCapabilityProbe({ resolveCliEntry: () => cliPath, execFile })).toBeNull();
});
});
it.each([
['unparseable JSON', 'not-json'],
['malformed object', JSON.stringify({ name: LEASE_ACTIVATION_CAPABILITY.name })],
])('returns null (fail-closed) on %s output', (_failure, output) => {
withScratchCli((cliPath) => {
const execFile: CapabilityProbeExecFile = () => output;
expect(defaultCapabilityProbe({ resolveCliEntry: () => cliPath, execFile })).toBeNull();
});
});
describe('positive path — injected resolver, isolated scratch dir (never the real dist/)', () => {
// A prior version of this test staged the stub cli.js at the package's
// REAL resolved dist/ path and relied on afterEach to clean up "only
@@ -55,6 +55,19 @@ export const LEASE_ACTIVATION_CAPABILITY: LeaseActivationCapability = {
/** Hidden CLI probe subcommand name — wired via {@link registerLeaseCapabilityProbe}. */
export const LEASE_CAPABILITY_PROBE_COMMAND = '__lease-capability';
/**
* Budget for the out-of-process capability probe. The probe launches a fresh
* Node process on the built CLI entrypoint, whose cold start alone measures
* 2.2-2.3s on a mid-range workstation (sb-it-1-dt, 2026-08-13) — so the
* previous 2s budget made the probe time out and report NO capability on
* such hosts, failing every launch with the #869 skew message even though
* the capability matched. The timeout only bounds the pathological hang
* case; the happy path returns as soon as the probe exits. Mirrors
* PROBE_TIMEOUT_SECONDS in the enforcement half
* (framework/tools/lease-broker/activation_version_gate.py).
*/
export const LEASE_CAPABILITY_PROBE_TIMEOUT_MS = 10_000;
function capabilityMatches(candidate: LeaseActivationCapability | null): boolean {
return (
candidate !== null &&
@@ -110,12 +123,28 @@ export function defaultResolveCliEntry(
return join(dirname(mainEntry), 'cli.js');
}
/** Narrow injectable seam for the synchronous child process used by the
* capability probe. */
export type CapabilityProbeExecFile = (
file: string,
args: string[],
options: {
encoding: BufferEncoding;
timeout: number;
stdio: ['ignore', 'pipe', 'ignore'];
},
) => string;
/** Injectable inputs for {@link defaultCapabilityProbe}. */
export interface CapabilityProbeDeps {
/** Resolve the CLI entrypoint (`cli.js`) to probe. Defaults to
* {@link defaultResolveCliEntry}. Inject to point at an isolated scratch
* location in tests — never at the real package's `dist/`. */
resolveCliEntry?: () => string;
/** Execute the resolved CLI entrypoint. Defaults to the real
* `execFileSync`. Inject so transport behavior and options can be tested
* without spawning a process. */
execFile?: CapabilityProbeExecFile;
}
/**
@@ -139,9 +168,10 @@ export function defaultCapabilityProbe(
const cliEntry = resolveCliEntry();
if (!existsSync(cliEntry)) return null;
const output = execFileSync(process.execPath, [cliEntry, LEASE_CAPABILITY_PROBE_COMMAND], {
const execFile: CapabilityProbeExecFile = deps.execFile ?? execFileSync;
const output = execFile(process.execPath, [cliEntry, LEASE_CAPABILITY_PROBE_COMMAND], {
encoding: 'utf-8',
timeout: 2000,
timeout: LEASE_CAPABILITY_PROBE_TIMEOUT_MS,
stdio: ['ignore', 'pipe', 'ignore'],
});
@@ -24,11 +24,15 @@ from __future__ import annotations
import importlib.util
import io
import os
import shlex
import subprocess
import sys
import tempfile
import unittest
from contextlib import redirect_stderr
from pathlib import Path
from unittest import mock
TOOLS_DIR = Path(__file__).parents[2] / "framework/tools/lease-broker"
@@ -57,6 +61,20 @@ def matching_capability() -> dict[str, object]:
return dict(VERSION_GATE.EXPECTED_ACTIVATION_CAPABILITY)
def write_fake_mosaic(directory: Path, marker: Path) -> Path:
directory.mkdir(parents=True, exist_ok=True)
executable = directory / "mosaic"
executable.write_text(
"#!/bin/sh\n"
f"printf '%s\\n' executed >> {shlex.quote(str(marker))}\n"
"printf '%s\\n' "
"'{\"name\":\"lease-runtime-activation\",\"version\":1}'\n",
encoding="utf-8",
)
executable.chmod(0o755)
return executable
class AssertActivationCapabilityMatchesTest(unittest.TestCase):
"""Unit-level coverage of `activation_version_gate.py`'s own assertion,
isolated from the launch-runtime.py seam it is wired into below."""
@@ -110,11 +128,102 @@ class ProbeActivationCapabilityTest(unittest.TestCase):
handling — never spawns a real `mosaic` process."""
def test_returns_none_when_mosaic_is_not_resolvable_on_path(self) -> None:
result = VERSION_GATE.default_probe_activation_capability(
{"PATH": "/nonexistent-bin-dir-for-869-c4-test"}
)
# Keep even a deliberate ambient-lookup mutation away from any host
# installation. The dedicated hermeticity tests below provide fake
# ambient executables and markers.
with mock.patch.dict(
os.environ, {"PATH": "/nonexistent-ambient-bin-dir-for-869-c4-test"}
):
result = VERSION_GATE.default_probe_activation_capability(
{"PATH": "/nonexistent-bin-dir-for-869-c4-test"}
)
self.assertIsNone(result)
def test_supplied_path_wins_over_ambient_process_path(self) -> None:
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
supplied_marker = root / "supplied.marker"
ambient_marker = root / "ambient.marker"
supplied_bin = root / "supplied-bin"
ambient_bin = root / "ambient-bin"
write_fake_mosaic(supplied_bin, supplied_marker)
write_fake_mosaic(ambient_bin, ambient_marker)
with mock.patch.dict(os.environ, {"PATH": str(ambient_bin)}):
result = VERSION_GATE.default_probe_activation_capability(
{"PATH": str(supplied_bin)}
)
self.assertEqual(result, matching_capability())
self.assertTrue(supplied_marker.exists())
self.assertFalse(ambient_marker.exists())
def test_absent_or_empty_supplied_path_never_falls_back_or_executes(self) -> None:
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
ambient_marker = root / "ambient.marker"
current_directory_marker = root / "current-directory.marker"
ambient_bin = root / "ambient-bin"
current_directory = root / "current-directory"
write_fake_mosaic(ambient_bin, ambient_marker)
write_fake_mosaic(current_directory, current_directory_marker)
original_directory = Path.cwd()
try:
os.chdir(current_directory)
with mock.patch.dict(os.environ, {"PATH": str(ambient_bin)}):
for supplied_environment in ({}, {"PATH": ""}):
with self.subTest(environ=supplied_environment):
result = VERSION_GATE.default_probe_activation_capability(
supplied_environment
)
self.assertIsNone(result)
self.assertFalse(ambient_marker.exists())
self.assertFalse(current_directory_marker.exists())
finally:
os.chdir(original_directory)
def test_valid_override_wins_and_invalid_override_does_not_fall_back_to_path(
self,
) -> None:
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
supplied_marker = root / "supplied.marker"
ambient_marker = root / "ambient.marker"
override_marker = root / "override.marker"
supplied_bin = root / "supplied-bin"
ambient_bin = root / "ambient-bin"
override_bin = root / "override-bin"
write_fake_mosaic(supplied_bin, supplied_marker)
write_fake_mosaic(ambient_bin, ambient_marker)
override_executable = write_fake_mosaic(override_bin, override_marker)
with mock.patch.dict(os.environ, {"PATH": str(ambient_bin)}):
result = VERSION_GATE.default_probe_activation_capability(
{
"PATH": str(supplied_bin),
VERSION_GATE.MOSAIC_COMMAND_OVERRIDE_VAR: str(override_executable),
}
)
self.assertEqual(result, matching_capability())
self.assertTrue(override_marker.exists())
self.assertFalse(supplied_marker.exists())
self.assertFalse(ambient_marker.exists())
override_marker.unlink()
result = VERSION_GATE.default_probe_activation_capability(
{
"PATH": str(supplied_bin),
VERSION_GATE.MOSAIC_COMMAND_OVERRIDE_VAR: str(
root / "invalid-override" / "mosaic"
),
}
)
self.assertIsNone(result)
self.assertFalse(override_marker.exists())
self.assertFalse(supplied_marker.exists())
self.assertFalse(ambient_marker.exists())
def test_override_command_is_parsed_and_the_probe_subcommand_is_not_double_appended(
self,
) -> None:
@@ -135,6 +244,29 @@ class ProbeActivationCapabilityTest(unittest.TestCase):
self.assertEqual(result, {"name": "lease-runtime-activation", "version": 1})
self.assertEqual(captured, [["/fake/mosaic", "__lease-capability"]])
def test_probe_passes_ten_second_timeout_to_runner(self) -> None:
captured_argv: list[str] = []
captured_kwargs: dict[str, object] = {}
class FakeCompleted:
returncode = 0
stdout = '{"name": "lease-runtime-activation", "version": 1}'
def fake_run(argv: list[str], **kwargs: object) -> FakeCompleted:
captured_argv.extend(argv)
captured_kwargs.update(kwargs)
return FakeCompleted()
result = VERSION_GATE.default_probe_activation_capability(
{VERSION_GATE.MOSAIC_COMMAND_OVERRIDE_VAR: "/fake/mosaic"},
run=fake_run,
)
self.assertEqual(result, matching_capability())
self.assertEqual(captured_argv, ["/fake/mosaic"])
self.assertEqual(captured_kwargs["timeout"], 10.0)
self.assertEqual(captured_kwargs["check"], False)
def test_fails_closed_on_nonzero_exit_malformed_json_and_missing_fields(self) -> None:
class NonZeroExit:
returncode = 1
@@ -174,7 +306,7 @@ class ProbeActivationCapabilityTest(unittest.TestCase):
def test_fails_closed_on_timeout_and_transport_error(self) -> None:
def timeout_run(*_args: object, **_kwargs: object) -> None:
raise subprocess.TimeoutExpired(cmd="mosaic", timeout=2.0)
raise subprocess.TimeoutExpired(cmd="mosaic", timeout=10.0)
def oserror_run(*_args: object, **_kwargs: object) -> None:
raise OSError("no such file or directory")
+42
View File
@@ -309,6 +309,43 @@ require_cmd() {
fi
}
# Fleet transport binary (#1240).
#
# `mosaic fleet --help` reads "Manage the local Mosaic tmux fleet" and every
# roster the CLI scaffolds sets `transport: tmux`, but nothing in this script
# provides tmux and, until now, nothing in it mentioned tmux at all. A
# greenfield host came out of this installer able to install a fleet, start a
# fleet, and run no seat — the operator's first signal was `mosaic fleet ps`.
#
# Not a `require_cmd`: tmux is required by the fleet, not by mosaic. Plenty of
# hosts install this to run `mosaic claude` and will never scaffold a roster,
# and failing their install over a binary they do not need would be wrong. It
# is a warning that names precisely what it blocks.
#
# `tools/_scripts/mosaic-doctor` carries a deliberately parallel check, so the
# same host state gets the same answer from an audit as from an install. They
# are separate implementations because this one has to work before the
# framework's scripts are guaranteed to be on disk; keep their wording in step.
check_fleet_transport() {
local transport=tmux
local roster="$MOSAIC_HOME/fleet/roster.yaml"
local declared=""
if [[ -f "$roster" ]]; then
declared="$(sed -n 's/^[[:space:]]*transport:[[:space:]]*//p' "$roster" | head -1 |
tr -d '"'\''' | tr -d '\r' | awk '{print $1}')"
[[ -n "$declared" ]] && transport="$declared"
fi
command -v "$transport" &>/dev/null && return 0
warn "Fleet transport '$transport' is not installed."
echo " The Mosaic fleet runs its agent seats inside $transport. Without it,"
echo " ${C}mosaic fleet start${RESET} reports success and no seat comes up."
echo " Install it before using the fleet, e.g. ${C}sudo apt-get install -y $transport${RESET}"
echo " (this does not affect ${C}mosaic claude${RESET} or the other single-runtime commands)."
}
installed_cli_version() {
local json
json="$(npm ls -g --depth=0 --json --prefix="$PREFIX" 2>/dev/null)" || true
@@ -870,6 +907,11 @@ if [[ "$FLAG_CHECK" == "false" ]]; then
ok "Done."
fi
# Fleet readiness (#1240). Runs in both normal and --check mode: "what is the
# state of this host" is exactly the question --check is asked, and a host that
# cannot run a seat should not have to discover it from `fleet ps`.
check_fleet_transport
} # end main
main "$@"