Compare commits

..
Author SHA1 Message Date
fargo 9d3e22b1c1 fix(fleet): activate the lease broker at install/start, place units through symlinks safely, refuse doomed launches (#1292)
ci/woodpecker/pr/ci Pipeline failed
Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.

Activation lands in the control plane, not the launcher:

- fleet install places ALL FOUR units through placeUnitFile — a placement
  helper that unlinks any by-path-enable symlink at the destination
  BEFORE copying (Node copyFile follows the link and overwrites the SEED
  template; measured on a throwaway systemd user instance 2026-08-17,
  with both cp and fs.copyFile), removes a stale wants-symlink pointing
  outside the active dir (readlink — readFile returns the target's
  content, not the link path), then copies and daemon-reloads. The same
  measurement showed systemctl enable <name> does NOT rewrite an existing
  by-path wants-symlink — reconciliation must be explicit. Idempotent:
  second install on by-path residue converges to the identical state.
  Until now the copy block named three units and omitted the broker, and
  the residue set / copy set were disjoint only by accident (fomo-lin
  survived copy-through because its one symlink was the one unit not
  copied); adding the broker made them intersect on first run. See the
  SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
  lifecycle effect, then RE-CHECK the socket (not unit state) and exit
  nonzero with a named code if it did not appear. Re-probed on every
  invocation — a RemainAfterExit=yes dead-looking-active unit can never
  make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
  first-class member; the socket is the signal (enabled-but-dead units
  report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
  effect (moved ahead of the ownership probe): absent -> exit 75
  (EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
  pane. The agent@ unit is Type=oneshot with no Restart=, so the message
  survives instead of looping. The preflight detects and refuses; it
  never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
  'mosaic fleet install (it reconciles either enable convention)' —
  written from the measurement; teaching a manual systemctl line could
  leave a host with competing wants-symlinks.

Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.

Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.

Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.

Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.
2026-08-17 18:38:22 -05:00
jason.woltje 8199261caa Merge pull request 'fix(ci): unwire test-start-agent-session.sh, restore its signed exclusion — unblocks every PR on next' (#1270) from fix/1269-ci-chain-unblock into next
ci/woodpecker/push/publish Pipeline failed
Reviewed-on: #1270
2026-08-17 20:44:59 +00:00
fred 57a2f2b40e docs(ci): point the exclusion at tracking issue #1271, not the closed first filing
ci/woodpecker/pr/ci Pipeline was successful
The first PR for this change was filed under the retired mos-dt-0 principal
(pr-create.sh has no --login flag and find_tea_login_for_host returns the first
host match) and was closed and refiled as #1270. That left in-tree references
pointing at a closed duplicate PR rather than at the burn-down issue, which is
the wrong target for them anyway: the open design question belongs on #1271.
2026-08-16 18:03:02 -05:00
fred 93c1de51e1 fix(ci): unwire test-start-agent-session.sh, restore its signed exclusion (#1269)
ci/woodpecker/pr/ci Pipeline was canceled
The `test` step has failed on every `next` pipeline since #1017 on exactly one
assertion, and it is the same one on unrelated PRs:

    FAIL: host provides 'pi' in the system path; missing-binary cases are not
    measurable here            (framework/tools/fleet/test-start-agent-session.sh:103)

Measured 2026-08-16 across pipelines 2444 (#1256), 2438 (#1240) and 2441
(#1017-quality): exactly one FAIL line in each full log, identical, this line.
Control `zzz-not-present-zzz` -> 0 on all three.

Cause. #1241 (5c35a250) added the guard: the suite shims fake mosaic/pi/npm into
$FAKE_BIN, but the constructed PANE_PATH always ends in the real system path, so
on a host that installs those binaries the missing-binary cases cannot be
measured and a green run would mean nothing. The guard says so instead of
passing. Its own pipeline 2430 was green only because the suite was CI-excluded
at the time, so the guard had never run in CI. #1017 (c56483eb) then enumerated
it and dropped the exclusion. The CI image installs
@earendil-works/[email protected].1 on purpose, so the precondition is
unsatisfiable there. Both commits are mine.

The guard is correct and is not being softened. A check that cannot measure its
property and reports success is the failure mode this repo has been cataloguing
all week; the error was wiring the suite into an image that violates its
precondition, so the wiring is what gets reverted.

Second effect, which is the reason this cost a day rather than an hour:
test:framework-shell is one && chain and this sat at position 44 of 48, so
glpi/test-list-http-status.sh, orchestrator/test-board-roll.sh,
woodpecker/test-ci-wait-exit-matrix.sh and _scripts/test-fleet-transport-check.sh
have not run at all since the merge. The pipeline reported one failure, never
"one failure plus four unrun". All four are green when run directly on
sb-it-1-dt, so the mask hid nothing broken -- but that is a local result on one
host, not a CI-image result.

Verification, with controls:
- enumeration guard OK (population 52, enumerated 36, signed-excluded 16).
- control A, exclusion line removed while unwired -> FAIL UNENUMERATED.
- control B, exclusion line kept while rewired -> FAIL CONTRADICTORY EXCLUSION.
  The gate discriminates in both directions, so its OK is load-bearing.
- the four formerly-masked suites: rc=0 each, run directly.
- the full chain cannot be run to completion on sb-it-1-dt: it stops earlier, at
  the lease-broker Invariant R test, because this host carries the quarantined
  operator-global pi 0.84.2 against a measured 0.84.1. That is host-specific and
  out of scope here -- CI pins 0.84.1, and the single FAIL line in those three
  pipelines proves positions 1-43 passed there.

Burn-down is to control the tail of PANE_PATH inside the test, not to remove pi
from the image. Recorded in the exclusion reason and in #1269.
2026-08-16 17:58:49 -05:00
24 changed files with 1019 additions and 1391 deletions
+12 -14
View File
@@ -115,19 +115,18 @@ gateway-backed agent catalog.
### Normative requirements
| ID | Requirement |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FCM-REQ-01` | The roster SHALL be the sole writable desired-state source for local fleet membership, launch policy, and persisted lifecycle target. Generated environment files, systemd enablement, tmux sessions, and heartbeat state SHALL be non-authoritative projections. |
| `FCM-REQ-02` | The implementation SHALL provide one executable structural contract for YAML/JSON input and one shared semantic validator. Roster load, profile validation, provision, migration, and apply SHALL reuse the existing baseline-plus-`roles.local` profile/persona resolver; a parallel role resolver is forbidden. |
| `FCM-REQ-03` | The local fleet CLI SHALL expose documented programmatic validate, show, plan, apply/reconcile, create, inspect, update, delete, start, stop, restart, status, verify, and doctor operations with stable JSON and exit-code behavior. Existing `fleet add/remove` compatibility aliases may remain during the stated deprecation window. |
| `FCM-REQ-04` | A fresh create SHALL persist `enabled:true` and `desired_state:stopped` unless an explicit persisted start is requested. The model SHALL distinguish enabled state, persisted desired state, and observed state. Migration, apply, reboot, and rollback SHALL not start an agent that was observed stopped before cutover. |
| `FCM-REQ-05` | The launch chain SHALL consume deterministic, digest-stamped generated input only. Optional local overrides SHALL be parsed as strict data, may not shadow authoritative generated keys, and may not contain arbitrary commands, credential values, channels, or unknown `MOSAIC_AGENT_*` keys. Forbidden legacy keys, including `MOSAIC_AGENT_COMMAND`, SHALL be privately quarantined before launch and reported only by key name and content hash. |
| `FCM-REQ-06` | Mutations and apply SHALL validate before mutation, use an expected generation/lock, write projections atomically, produce a deterministic plan, and emit recovery information on partial failure. Reconciliation SHALL act only on local, enabled, roster-owned projections and SHALL not kill unmanaged tmux sessions by fuzzy name. |
| `FCM-REQ-07` | Canonical required classes are `code`, `review`, `validator`, `orchestrator`, `team-leader`, `enhancer`, and `interaction`. `validator` issues an independent final certificate but has no merge authority; `merge-gate` remains sole approve-to-land/merge authority. Team-leader capacity is bounded by an orchestrator-issued lease, and interaction is request/status only. Tess and Ultron are configurable instance/display names, not required machine identities. |
| `FCM-REQ-08` | v1 migration SHALL be field-complete, reversible, and explicit about aliases, unresolved classes, lifecycle inference, generated-file regeneration, local override quarantine, schema-only remote/connector fields, and rollback. Every shipped example, profile, and service preset SHALL be migrated and executable, retained as an explicitly versioned v1 fixture, or retired with a replacement and deprecation note. |
| `FCM-REQ-09` | M1M5 SHALL remain local tmux/systemd control-plane work. Remote/SSH reconciliation, connector mutation, secret references, arbitrary command/channel overrides, gateway/API convergence, and UI configuration storage are excluded and require a separate PRD/threat model. |
| `FCM-REQ-10` | Documentation and examples are delivery gates. The M0 checklist at [docs/fleet/FLEET-CONFIG-DOCS-IA-CHECKLIST.md](./fleet/FLEET-CONFIG-DOCS-IA-CHECKLIST.md) and the baseline disposition inventory at [docs/fleet/LEGACY-EXAMPLE-PROFILE-DISPOSITION-INVENTORY.md](./fleet/LEGACY-EXAMPLE-PROFILE-DISPOSITION-INVENTORY.md) SHALL be maintained as acceptance evidence. |
| `FCM-REQ-11` | Fleet provisioning SHALL validate the fleet CLI and distinct runtimes requested by the roster against the exact PATH construction used by the runtime pane, through one shared implementation rather than the operator PATH or a parallel PATH model. Name resolution alone is insufficient: a resolved script's shebang interpreter SHALL also be reachable, and Node SHALL execute a side-effect-free version probe when it is that interpreter. `fleet install` and `install-systemd` SHALL fail before installation effects when a required executable is absent or unreachable. `fleet doctor` SHALL emit the same named checks as non-green evidence. Every runtime failure SHALL name the runtime, all requesting roster rows, the pane PATH searched, and an exact install command. |
| ID | Requirement |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `FCM-REQ-01` | The roster SHALL be the sole writable desired-state source for local fleet membership, launch policy, and persisted lifecycle target. Generated environment files, systemd enablement, tmux sessions, and heartbeat state SHALL be non-authoritative projections. |
| `FCM-REQ-02` | The implementation SHALL provide one executable structural contract for YAML/JSON input and one shared semantic validator. Roster load, profile validation, provision, migration, and apply SHALL reuse the existing baseline-plus-`roles.local` profile/persona resolver; a parallel role resolver is forbidden. |
| `FCM-REQ-03` | The local fleet CLI SHALL expose documented programmatic validate, show, plan, apply/reconcile, create, inspect, update, delete, start, stop, restart, status, verify, and doctor operations with stable JSON and exit-code behavior. Existing `fleet add/remove` compatibility aliases may remain during the stated deprecation window. |
| `FCM-REQ-04` | A fresh create SHALL persist `enabled:true` and `desired_state:stopped` unless an explicit persisted start is requested. The model SHALL distinguish enabled state, persisted desired state, and observed state. Migration, apply, reboot, and rollback SHALL not start an agent that was observed stopped before cutover. |
| `FCM-REQ-05` | The launch chain SHALL consume deterministic, digest-stamped generated input only. Optional local overrides SHALL be parsed as strict data, may not shadow authoritative generated keys, and may not contain arbitrary commands, credential values, channels, or unknown `MOSAIC_AGENT_*` keys. Forbidden legacy keys, including `MOSAIC_AGENT_COMMAND`, SHALL be privately quarantined before launch and reported only by key name and content hash. |
| `FCM-REQ-06` | Mutations and apply SHALL validate before mutation, use an expected generation/lock, write projections atomically, produce a deterministic plan, and emit recovery information on partial failure. Reconciliation SHALL act only on local, enabled, roster-owned projections and SHALL not kill unmanaged tmux sessions by fuzzy name. |
| `FCM-REQ-07` | Canonical required classes are `code`, `review`, `validator`, `orchestrator`, `team-leader`, `enhancer`, and `interaction`. `validator` issues an independent final certificate but has no merge authority; `merge-gate` remains sole approve-to-land/merge authority. Team-leader capacity is bounded by an orchestrator-issued lease, and interaction is request/status only. Tess and Ultron are configurable instance/display names, not required machine identities. |
| `FCM-REQ-08` | v1 migration SHALL be field-complete, reversible, and explicit about aliases, unresolved classes, lifecycle inference, generated-file regeneration, local override quarantine, schema-only remote/connector fields, and rollback. Every shipped example, profile, and service preset SHALL be migrated and executable, retained as an explicitly versioned v1 fixture, or retired with a replacement and deprecation note. |
| `FCM-REQ-09` | M1M5 SHALL remain local tmux/systemd control-plane work. Remote/SSH reconciliation, connector mutation, secret references, arbitrary command/channel overrides, gateway/API convergence, and UI configuration storage are excluded and require a separate PRD/threat model. |
| `FCM-REQ-10` | Documentation and examples are delivery gates. The M0 checklist at [docs/fleet/FLEET-CONFIG-DOCS-IA-CHECKLIST.md](./fleet/FLEET-CONFIG-DOCS-IA-CHECKLIST.md) and the baseline disposition inventory at [docs/fleet/LEGACY-EXAMPLE-PROFILE-DISPOSITION-INVENTORY.md](./fleet/LEGACY-EXAMPLE-PROFILE-DISPOSITION-INVENTORY.md) SHALL be maintained as acceptance evidence. |
### Acceptance criteria
@@ -139,7 +138,6 @@ gateway-backed agent catalog.
6. `AC-FCM-06`: A v1 roster migration previews field-by-field disposition, preserves observed stopped/running state, inventories rather than reconciles remote/schema-only entries, supports a canary and rollback, and classifies every shipped example, profile, and service preset according to the M0 inventory.
7. `AC-FCM-07`: Required role authority is validated: validator certificate is consumed but does not merge, merge-gate is the sole merge authority, team-leader leases do not change roster/credentials/authority, and interaction/Tess cannot claim orchestration or merge powers.
8. `AC-FCM-08`: Documentation, examples, migration, troubleshooting, operational recovery, package/update asset drift, schema/example/profile validation, independent code/security review, validator certificate, and terminal-green CI are complete before #758 closes.
9. `AC-FCM-09`: Red-first isolated tests create (a) a roster whose runtime exists on the operator PATH but is absent from the constructed pane PATH and (b) a greenfield pane where `mosaic` and a runtime resolve as Node-shebang scripts while Node is absent. They prove `fleet install` fails before effects, the launcher creates no doomed session, and `fleet doctor` reports named non-green checks. Diagnostics include the executable or runtime, all requesting rows, searched pane PATH, shebang dependency when present, and exact runtime install command; repeated rows are checked once per distinct runtime/effective pane path. Tests use temporary `--mosaic-home` state and fixture binaries, never host runtime mutation.
### M0 implementation gate
-22
View File
@@ -59,28 +59,6 @@ valid allowed local data can move to `.env.local`; invalid legacy input is priva
Diagnostics expose only rule code, key name, and a SHA-256 content hash. They do not reveal command
text, credentials, or other values.
## Pane executable preflight
The fleet install, install-systemd, and doctor commands plus the session launcher use
**pane-runtime-path.sh** as the single pane-PATH implementation. Install inspects every distinct
roster runtime and effective MOSAIC_RUNTIME_BIN pair before creating holder identity, tool,
projection, or unit files. Doctor reports the same checks as JSON.
A resolved command is not automatically executable. The helper reads a script shebang, unwraps the
common “/usr/bin/env node” and “/usr/bin/env -S node …” forms, then resolves the declared command
against the pane PATH. When Node is the declared interpreter, the helper runs the side-effect-free
“node --version” probe. It does not run “mosaic --version”, whose startup update check can write cache
state. Native binaries have no PATH-resolved shebang dependency and retain their normal executable
check. Failures name the executable or runtime, requesting roster rows, searched pane PATH,
dependency, and runtime install command.
Supported runtime install commands are:
- **Claude:** curl -fsSL https://claude.ai/install.sh | bash
- **Codex:** npm install -g @openai/codex
- **OpenCode:** npm install -g opencode-ai
- **Pi:** npm install -g @earendil-works/pi-coding-agent
## Launch and stop behavior
The launcher obtains the agent's socket only from the validated generated projection. It creates or
@@ -1,71 +0,0 @@
# #1256 — Fleet runtime preflight
**Agent:** tiny
**Branch:** `fix/1256-fleet-runtime-preflight` from `origin/next@476db12b92971634b67fd2057b7577ee5894e449`
**Issue:** `mosaicstack/stack#1256` blocker 1
**Adjacent PR:** `#1258` (`fix/1256-fleet-pane-path-node`) owns the bootstrapped-Node candidate and must remain a separate change
**Budget:** 30K-token soft cap; one bounded implementation lane
## Objective
Make fleet provisioning fail before installation effects when the roster names a runtime binary absent from the exact PATH the tmux pane will receive. Make `mosaic fleet doctor` report the same named runtime check. Diagnostics must name the runtime, every requesting roster row, the pane PATH searched, and an exact install command.
For Pi the exact command is:
```text
npm install -g @earendil-works/pi-coding-agent
```
## Constraints
- TDD: add the failing behavior test and capture RED before implementation.
- Runtime resolution uses the launcher's pane-PATH construction; a second PATH model is forbidden.
- Operator PATH is non-authoritative and must not cause a false pass.
- Tests use an isolated `--mosaic-home`/temporary HOME and never mutate host runtime binaries.
- No install, removal, or binary-resolution changes on sb-it-1-dt.
- PR targets `next` and requires a reviewer other than fred.
- Commit identity is `tiny <[email protected]>`.
- #1258's Node candidate is a dependency/adjacent change, never reimplemented here.
## Planned seam
1. Factor the shell pane-path builder/resolver into one sourceable and executable fleet helper.
2. Have `start-agent-session.sh` source that helper, preserving one definition of the pane PATH.
3. Have the TypeScript fleet command invoke the same helper under the unit-equivalent clean launcher environment.
4. Group roster rows by distinct runtime and effective pane PATH, then report requesting row names.
5. Run the preflight before `installFleet` performs any write.
6. Add the named result to roster-v2 `fleet doctor` JSON and set a failing exit when a runtime is absent.
7. Install/copy the helper alongside `start-agent-session.sh` and update framework manifest/docs as required.
This seam overlaps #1258 only at the location of the existing shell function. Development may use #1258 as a local dependency, but the final PR diff must exclude #1258's separately owned Node change after that PR lands or after an agreed rebase order.
## Acceptance evidence
| Requirement | Evidence |
|---|---|
| Missing Pi blocks install before effects | isolated CLI test: nonzero + no installed files/runner effects |
| Operator PATH cannot create false green | test puts Pi only on operator PATH and omits it from constructed pane PATH |
| Exact pane PATH reused | launcher and CLI call one shared shell helper; contract test exercises both |
| Actionable diagnosis | runtime + roster rows + searched PATH + exact install command assertions |
| Distinct runtimes | repeated rows produce one check with all row names |
| Doctor reports named check | JSON assertion + nonzero exit for missing runtime |
| Present runtime passes | isolated pane-path fixture with executable binary |
| No host mutation | tests use temporary HOME/Mosaic home and fixture binaries only |
| Baseline safety | focused tests, package typecheck/lint/format, full relevant suite, CI |
## Progress log
- 2026-08-16: Dispatch received from fred; issue #1256 and PR #1258 measured.
- 2026-08-16: Fresh clone created under `~/agent-work/tiny-fleet-runtime-preflight`; local Git identity pinned to tiny so retired global `mos-dt-0` identity cannot win.
- 2026-08-16: Design inspection found the pane PATH exists only inside `start-agent-session.sh`; the right seam is a shared shell helper rather than a parallel TypeScript reconstruction.
- 2026-08-16: RED measured on `origin/next@476db12b`: focused `fleet-roster-v2-dispatch.spec.ts` ran 11 tests; the new case failed because install returned success, wrote units for two agents, and emitted no `runtime=pi` diagnosis while Pi existed only on operator PATH.
- 2026-08-16: Factored pane home/PATH/resolution into sourceable and executable `pane-runtime-path.sh`; install invokes it before the first effect, doctor emits the same named checks, and the launcher sources it.
- 2026-08-16: Fred/rhodey review exposed the #1241 name-resolution blind spot: `mosaic` can resolve while its `#!/usr/bin/env node` interpreter cannot. Measurement confirmed every supported current Mosaic package shape is a Node-shebang script, but executing `mosaic --version` is not observational because CLI startup runs the cache-writing/network update checker before Commander handles the flag.
- 2026-08-16: Final executable check reads and unwraps direct and `/usr/bin/env` shebangs (including `env -S`), resolves the declared dependency against pane PATH, and runs only side-effect-free `node --version` when Node is declared. Native binaries do not inherit a permanent Node requirement. Install, doctor, and launcher share this implementation.
- 2026-08-16: Isolated greenfield fixture places resolved Mosaic and Pi Node-shebang scripts in pane-visible npm-global bin while using an empty system suffix; both checks become `unexecutable` with `dependency=node`, and install leaves holder/tools/units absent. No host binary or HOME is changed.
- 2026-08-16: GREEN evidence before #1258 rebase: focused install/doctor/preflight suites pass; `fleet.spec.ts` 209/209; full Vitest 87 files / 1,557 tests; launcher shell suite, typecheck, lint, build, and focused format check pass. Full framework-shell reaches an unrelated host-measurement drift in unchanged `invariant_r_unittest.py` (expected Pi 0.84.1, host resolves 0.84.2); no invariant was changed in this lane.
- 2026-08-16: Merge-order gate remains: `origin/next` is still `476db12b`; #1258 is unmerged at `6dc35e5`. Rebase after it lands, relocate its Node candidate into the helper with explicit provenance, rerun gates, then open the PR to `next` for an independent non-fred review.
@@ -51,12 +51,8 @@ See `docs/fleet/reference/generated-env-boundary.md` for the full contract.
## Manual canary sequence
Use the roster and the supported installer; do not pre-create the agent environment directory or
edit a generated projection. Before it writes any holder identity, tool, projection, or unit file,
`mosaic fleet install` validates the fleet CLI and every distinct roster runtime through the exact
pane PATH. The shared helper also unwraps `/usr/bin/env` shebangs, so a resolved Node script with no
pane-visible Node fails before effects. `mosaic fleet doctor` reports the same named executable
checks without mutation. After that preflight, install places the units and helpers and writes private
roster-derived projections before any service starts.
edit a generated projection. `mosaic fleet install` validates the roster, installs the units and
helpers, and writes private roster-derived projections before any service is started.
```bash
# Create a site-owned canary roster. Inspect an existing roster before using --force.
@@ -1,199 +0,0 @@
#!/usr/bin/env bash
# Canonical fleet-pane PATH construction and executable reachability checks.
#
# This file is both sourceable by start-agent-session.sh and executable by the
# fleet CLI preflight. Keep the pane PATH in one implementation: provisioning
# checks and the eventual pane must answer the same question.
mosaic_fleet_pane_home() {
local mosaic_home="$1"
local fallback_home="$2"
case "$mosaic_home" in
*/.config/mosaic) printf '%s' "${mosaic_home%/.config/mosaic}" ;;
*) printf '%s' "$fallback_home" ;;
esac
}
mosaic_fleet_build_runtime_bin_prefix() {
local pane_home="$1"
local runtime_bin="${2:-}"
local candidates=()
if [ -n "$runtime_bin" ]; then candidates+=("$runtime_bin"); fi
if command -v npm >/dev/null 2>&1; then
local npm_prefix
npm_prefix=$(npm config get prefix 2>/dev/null) || true
if [ -n "$npm_prefix" ]; then candidates+=("${npm_prefix}/bin"); fi
fi
candidates+=("$pane_home/.npm-global/bin" "$pane_home/.local/bin")
local prefix="" dir
for dir in "${candidates[@]}"; do
[ -d "$dir" ] || continue
case ":${prefix}:" in *":${dir}:"*) ;; *) prefix="${prefix:+$prefix:}$dir" ;; esac
done
printf '%s' "$prefix"
}
mosaic_fleet_build_pane_path() {
local pane_home="$1"
local runtime_bin="${2:-}"
local system_path="${3:-/usr/local/bin:/usr/bin:/bin}"
local prefix
prefix=$(mosaic_fleet_build_runtime_bin_prefix "$pane_home" "$runtime_bin")
printf '%s' "${prefix:+${prefix}:}${system_path}"
}
mosaic_fleet_resolve_in_pane_path() {
local pane_path="$1"
local binary="$2"
PATH="$pane_path" command -v -- "$binary" 2>/dev/null
}
# Sets executable evidence in MOSAIC_FLEET_EXECUTABLE_* and returns nonzero when
# a resolved script's shebang interpreter cannot run in the pane. Native/ELF
# binaries have no PATH-resolved interpreter dependency and pass the executable
# bit check. Node receives an additional side-effect-free `node --version`
# execution check; invoking `mosaic --version` itself is intentionally avoided
# because Mosaic performs a cache-writing/network update check at CLI startup.
mosaic_fleet_check_resolved_executable() {
local pane_path="$1"
local resolved="$2"
MOSAIC_FLEET_EXECUTABLE_DEPENDENCY=""
MOSAIC_FLEET_EXECUTABLE_PROBE=""
MOSAIC_FLEET_EXECUTABLE_EXIT=""
MOSAIC_FLEET_EXECUTABLE_OUTPUT=""
[ -x "$resolved" ] || {
MOSAIC_FLEET_EXECUTABLE_OUTPUT="resolved path is not executable"
return 70
}
local magic=""
IFS= read -r -n 2 magic < "$resolved" || true
[ "$magic" = '#!' ] || return 0
local shebang
IFS= read -r shebang < "$resolved" || true
shebang=${shebang%$'\r'}
shebang=${shebang#\#!}
local parts=()
read -r -a parts <<< "$shebang"
local interpreter="${parts[0]:-}"
[[ "$interpreter" = /* ]] && [ -x "$interpreter" ] || {
MOSAIC_FLEET_EXECUTABLE_DEPENDENCY="$interpreter"
MOSAIC_FLEET_EXECUTABLE_OUTPUT="shebang interpreter is absent or not executable"
return 70
}
local dependency="$interpreter"
local dependency_path="$interpreter"
if [ "${interpreter##*/}" = env ]; then
local index=1
if [ "${parts[$index]:-}" = -S ]; then index=$((index + 1)); fi
dependency="${parts[$index]:-}"
if [ -z "$dependency" ] || [[ "$dependency" = -* ]]; then
MOSAIC_FLEET_EXECUTABLE_DEPENDENCY="$dependency"
MOSAIC_FLEET_EXECUTABLE_OUTPUT="unsupported env shebang"
return 70
fi
fi
MOSAIC_FLEET_EXECUTABLE_DEPENDENCY="$dependency"
if [ "${dependency##*/}" = node ]; then
MOSAIC_FLEET_EXECUTABLE_PROBE="node --version"
fi
if [ "${interpreter##*/}" = env ]; then
if ! dependency_path=$(mosaic_fleet_resolve_in_pane_path "$pane_path" "$dependency"); then
MOSAIC_FLEET_EXECUTABLE_OUTPUT="shebang command is not on the pane PATH"
return 70
fi
fi
if [ "${dependency##*/}" = node ]; then
if MOSAIC_FLEET_EXECUTABLE_OUTPUT=$(PATH="$pane_path" "$dependency_path" --version 2>&1); then
MOSAIC_FLEET_EXECUTABLE_EXIT=0
else
MOSAIC_FLEET_EXECUTABLE_EXIT=$?
return 70
fi
fi
return 0
}
mosaic_fleet_runtime_path_main() {
local mosaic_home=""
local runtime_bin=""
local system_path="/usr/local/bin:/usr/bin:/bin"
local binary=""
local check_executable=0
while [ "$#" -gt 0 ]; do
case "$1" in
--mosaic-home)
[ "$#" -ge 2 ] || return 64
mosaic_home="$2"
shift 2
;;
--runtime-bin)
[ "$#" -ge 2 ] || return 64
runtime_bin="$2"
shift 2
;;
--binary)
[ "$#" -ge 2 ] || return 64
binary="$2"
shift 2
;;
--check-executable)
check_executable=1
shift
;;
# Test seam for measuring a greenfield host with no system Node. The
# launcher and production CLI omit it and retain the fixed system suffix.
--system-path)
[ "$#" -ge 2 ] || return 64
system_path="$2"
shift 2
;;
*) return 64 ;;
esac
done
[ -n "$mosaic_home" ] && [ -n "$binary" ] || return 64
local pane_home pane_path resolved
pane_home=$(mosaic_fleet_pane_home "$mosaic_home" "${HOME:-}")
# npm config is HOME-sensitive. Pin it to the derived pane home before asking
# for its prefix so an operator's unrelated npmrc cannot influence preflight.
HOME=$pane_home
export HOME
pane_path=$(mosaic_fleet_build_pane_path "$pane_home" "$runtime_bin" "$system_path")
if ! resolved=$(mosaic_fleet_resolve_in_pane_path "$pane_path" "$binary"); then
printf 'pane_path\0%s\0status\0missing\0binary_path\0\0dependency\0\0probe_command\0\0probe_exit\0\0probe_output\0\0' \
"$pane_path"
return 69
fi
if [ "$check_executable" -eq 1 ]; then
if mosaic_fleet_check_resolved_executable "$pane_path" "$resolved"; then
printf 'pane_path\0%s\0status\0present\0binary_path\0%s\0dependency\0%s\0probe_command\0%s\0probe_exit\0%s\0probe_output\0%s\0' \
"$pane_path" "$resolved" "$MOSAIC_FLEET_EXECUTABLE_DEPENDENCY" \
"$MOSAIC_FLEET_EXECUTABLE_PROBE" "$MOSAIC_FLEET_EXECUTABLE_EXIT" \
"$MOSAIC_FLEET_EXECUTABLE_OUTPUT"
return 0
fi
printf 'pane_path\0%s\0status\0unexecutable\0binary_path\0%s\0dependency\0%s\0probe_command\0%s\0probe_exit\0%s\0probe_output\0%s\0' \
"$pane_path" "$resolved" "$MOSAIC_FLEET_EXECUTABLE_DEPENDENCY" \
"$MOSAIC_FLEET_EXECUTABLE_PROBE" "$MOSAIC_FLEET_EXECUTABLE_EXIT" \
"$MOSAIC_FLEET_EXECUTABLE_OUTPUT"
return 70
fi
printf 'pane_path\0%s\0status\0present\0binary_path\0%s\0dependency\0\0probe_command\0\0probe_exit\0\0probe_output\0\0' \
"$pane_path" "$resolved"
return 0
}
if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then
set -euo pipefail
mosaic_fleet_runtime_path_main "$@"
fi
@@ -233,8 +233,36 @@ assert_owned_tmux_server() {
fail "tmux server ownership or environment validation failed"
}
# Validate exact server ownership before querying, cleaning, or creating any
# managed session. An unmanaged or contaminated named socket is never repaired.
# Lease-broker socket preflight (#1292). The gated runtime (`mosaic yolo …` →
# launch-runtime.py) registers with the broker or dies ~4 seconds in, with the
# diagnostic invisible because tmux destroys the dead pane. This check runs
# BEFORE any tmux effect — including the ownership probe below — so a host
# without a broker produces a named, surviving refusal instead of a doomed
# pane. Exit 75 (EX_TEMPFAIL), distinct from 64 (bad projection) and 69 (host
# not ready for other reasons); the agent@ unit is Type=oneshot with no
# Restart=, so the failed unit keeps its message instead of looping. Socket
# resolution matches launch.ts's defaultLeaseBrokerSocket precedence exactly.
# This preflight DETECTS and REFUSES — it never starts the broker (activation
# belongs to the fleet control plane; a component that both detects and fixes
# cannot be used to measure whether the fix worked).
broker_socket_path() {
if [ -n "${MOSAIC_LEASE_BROKER_SOCKET:-}" ]; then
printf '%s\n' "$MOSAIC_LEASE_BROKER_SOCKET"
return 0
fi
local runtime_dir="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
printf '%s\n' "${runtime_dir}/mosaic-lease/broker.sock"
}
if [ "$MODE" = "launch" ]; then
_broker_socket=$(broker_socket_path)
if [ ! -S "$_broker_socket" ]; then
echo "[fleet] FAIL_LAUNCH broker-absent: lease broker socket ${_broker_socket} missing; runtime launch denied (#1292)." >&2
echo "[fleet] remedy: systemctl --user enable --now mosaic-lease-broker.service (or reinstall via: mosaic fleet install)" >&2
exit 75
fi
fi
assert_owned_tmux_server
if [ "$MODE" = interaction ]; then
@@ -258,22 +286,46 @@ if _tmux has-session -t "=${AGENT_NAME}:0.0" 2>/dev/null; then
fi
# Systemd passes HOME as %h, and the installed service fixes MOSAIC_HOME under
# that home. The provisioning preflight executes this same helper under the
# unit's clean launcher environment, so operator PATH cannot produce a false
# green result for a binary the pane will never see.
SCRIPT_DIR=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
# shellcheck source=pane-runtime-path.sh
. "$SCRIPT_DIR/pane-runtime-path.sh"
PANE_HOME=$(mosaic_fleet_pane_home "$MOSAIC_HOME" "$HOME")
PANE_PATH=$(mosaic_fleet_build_pane_path "$PANE_HOME" "$MOSAIC_RUNTIME_BIN")
# that home. Derive the pane home from the canonical path when available so an
# inherited pane/session HOME cannot become runtime authority.
PANE_HOME=$HOME
case "$MOSAIC_HOME" in
*/.config/mosaic) PANE_HOME=${MOSAIC_HOME%/.config/mosaic} ;;
esac
# #1241/#1256. The pane runs `mosaic yolo <runtime>` under PANE_PATH with a
# cleared environment. Resolve both names and validate any shebang interpreter
# here, before an effect, where the failure remains attributable. Name
# resolution alone is insufficient: an `#!/usr/bin/env node` script resolves
# even when the pane cannot execute it because Node is absent.
_build_runtime_bin_prefix() {
local candidates=()
if [ -n "$MOSAIC_RUNTIME_BIN" ]; then candidates+=("$MOSAIC_RUNTIME_BIN"); fi
if command -v npm >/dev/null 2>&1; then
local npm_prefix
npm_prefix=$(npm config get prefix 2>/dev/null) || true
if [ -n "$npm_prefix" ]; then candidates+=("${npm_prefix}/bin"); fi
fi
candidates+=("$PANE_HOME/.npm-global/bin" "$PANE_HOME/.local/bin")
local prefix="" dir
for dir in "${candidates[@]}"; do
[ -d "$dir" ] || continue
case ":${prefix}:" in *":${dir}:"*) ;; *) prefix="${prefix:+$prefix:}$dir" ;; esac
done
printf '%s' "$prefix"
}
MOSAIC_RUNTIME_BIN_PREFIX=$(_build_runtime_bin_prefix)
PANE_PATH=${MOSAIC_RUNTIME_BIN_PREFIX:+${MOSAIC_RUNTIME_BIN_PREFIX}:}/usr/local/bin:/usr/bin:/bin
# #1241. The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared
# environment. A binary missing from *that* path is a pane that dies in under a
# second, inside a session nobody is attached to, with its diagnostic scrolled
# into a pane tmux then destroys. Resolve both here, before any effect, where
# the failure is still attributable to the thing that caused it.
#
# `mosaic yolo <runtime>` runs checkRuntime(runtime) and the binary it looks for
# is named exactly like the runtime, so resolving the runtime name is the same
# question the pane will ask a moment later — asked while an operator can still
# see the answer.
_resolve_in_pane_path() {
mosaic_fleet_resolve_in_pane_path "$PANE_PATH" "$1"
PATH="$PANE_PATH" command -v -- "$1" 2>/dev/null
}
# Exit 69 (EX_UNAVAILABLE): the seat cannot be provided. Distinguished from the
@@ -288,15 +340,8 @@ fail_launch() {
}
for required_binary in mosaic "$MOSAIC_AGENT_RUNTIME"; do
resolved_binary=$(_resolve_in_pane_path "$required_binary") ||
_resolve_in_pane_path "$required_binary" >/dev/null ||
fail_launch missing-binary "'${required_binary}' is not on the pane PATH (${PANE_PATH})"
if mosaic_fleet_check_resolved_executable "$PANE_PATH" "$resolved_binary"; then
continue
else
executable_exit=$?
fi
fail_launch unexecutable-binary \
"'${required_binary}' resolves to '${resolved_binary}' but dependency '${MOSAIC_FLEET_EXECUTABLE_DEPENDENCY:-unknown}' is not executable on the pane PATH (${PANE_PATH}); check_exit=${executable_exit} detail=${MOSAIC_FLEET_EXECUTABLE_OUTPUT:-unavailable}"
done
_ensure_claude_workdir_trusted() {
@@ -0,0 +1,216 @@
#!/usr/bin/env bash
# CI-fit regression suite for the #1292 lease-broker socket preflight in
# start-agent-session.sh.
#
# WHY THIS SUITE IS CI-FIT WHERE test-start-agent-session.sh IS NOT (#1017/#1270
# context): that older suite's precondition is "the host does not have the pi
# binary", which a CI image that ships pi violates — its guard correctly
# refuses to report a pass there, so it is excluded from the chain. THIS suite
# controls its own preconditions instead of inheriting them from the host: a
# fake tmux on PATH, a fake mosaic on PATH, a real unix socket created in a
# tmpdir, a hermetic env (env -i, fake HOME, GIT_CONFIG_GLOBAL severed). It
# never depends on what the host has installed, so a green here means the same
# thing on every host. Anyone adding cases: keep that property — no case may
# depend on host state.
#
# The failure this suite is written down to catch (#1292): a seat launched on a
# host with no lease broker dies ~4 seconds in at registration, with the
# diagnostic invisible because tmux destroys the dead pane. The preflight runs
# BEFORE any tmux effect and refuses with a NAMED code (exit 75, EX_TEMPFAIL)
# so the message survives. The agent@ unit is Type=oneshot with no Restart=,
# so a failed unit keeps its output instead of looping.
#
# Cases:
# 1. absent socket -> exit 75, message names broker-absent + socket path +
# remedy, and NO tmux session was ever created (the doomed-pane half).
# 2. present socket (real unix socket in tmpdir) -> proceeds PAST the
# preflight (the suite then stops at the next precondition, proving the
# preflight was not the refusal).
# 3. explicit MOSAIC_LEASE_BROKER_SOCKET wins over XDG_RUNTIME_DIR default.
# 4. --stop mode does NOT require the broker (teardown must not be fenced on
# a component whose absence is exactly what teardown may follow).
#
# Sabotage control, run by the developer (not in-suite): remove the preflight
# block from start-agent-session.sh, re-run — case 1 fails (a tmux session is
# created / exit is not 75), cases 2-4 still pass; restore byte-identically.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/agent-session-broker-preflight}"
FAKE_HOME="$WORK_DIR/home"
BIN_DIR="$WORK_DIR/bin"
ENV_DIR="$WORK_DIR/env"
SOCK_DIR="$WORK_DIR/sockets"
LOG_FILE="$WORK_DIR/tmux-calls.log"
rm -rf "$WORK_DIR"
# The script asserts a managed directory tree under MOSAIC_HOME: mosaic/,
# mosaic/fleet/, mosaic/fleet/agents/ — private (0700/0750-style) modes, no
# symlinks — plus a per-agent env projection. Build the full tree the launcher
# expects so the suite reaches the BROKER preflight rather than dying at
# environment validation.
mkdir -p "$FAKE_HOME/.config/mosaic/fleet/agents" "$BIN_DIR" "$SOCK_DIR"
chmod 700 "$FAKE_HOME/.config/mosaic" "$FAKE_HOME/.config/mosaic/fleet/agents"
chmod 750 "$FAKE_HOME/.config/mosaic/fleet"
cat > "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated" <<'ENVEOF'
MOSAIC_AGENT_NAME=preflight-test
MOSAIC_AGENT_CLASS=worker
MOSAIC_AGENT_RUNTIME=pi
MOSAIC_AGENT_MODEL=
MOSAIC_AGENT_REASONING=
MOSAIC_AGENT_TOOL_POLICY=code
MOSAIC_AGENT_WORKDIR=/tmp
MOSAIC_TMUX_SOCKET=mosaic-fleet
ENVEOF
chmod 600 "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated"
# ─── Fake tmux: records every invocation; new-session marks the marker. ────
: > "$LOG_FILE"
cat > "$BIN_DIR/tmux" <<SH
#!/usr/bin/env bash
printf 'tmux %s\n' "\$*" >> "$LOG_FILE"
if [[ "\$*" == *new-session* ]]; then
echo "TMUX-NEW-SESSION-INVOKED" >> "$LOG_FILE"
fi
exit 0
SH
chmod +x "$BIN_DIR/tmux"
# ─── Fake mosaic/pi binaries so the script proceeds past its own lookups. ───
for bin in mosaic pi claude; do
printf '#!/usr/bin/env bash\nexit 0\n' > "$BIN_DIR/$bin"
chmod +x "$BIN_DIR/$bin"
done
# ─── Minimal launch environment the script expects. ────────────────────────
# (Enough for the preflight to be reached; later stages will still fail in
# case 2 — that is expected and asserted.)
run_session_script() {
local mode="$1"; shift
(
cd "$WORK_DIR"
env -i HOME="$FAKE_HOME" PATH="$BIN_DIR:/usr/bin:/bin" \
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null \
MOSAIC_HOME="$FAKE_HOME/.config/mosaic" \
AGENT_NAME=preflight-test \
"$@" \
bash "$SCRIPT_DIR/start-agent-session.sh" $mode preflight-test
)
}
fail=0
assert() {
local desc="$1" expected="$2" actual="$3"
if [[ "$expected" != "$actual" ]]; then
echo "FAIL: $desc — expected '$expected', got '$actual'" >&2
fail=1
fi
}
assert_contains() {
local desc="$1" haystack="$2" needle="$3"
[[ "$haystack" == *"$needle"* ]] || { echo "FAIL: $desc — missing '$needle' in: $haystack" >&2; fail=1; }
}
assert_not_contains() {
local desc="$1" haystack="$2" needle="$3"
if [[ "$haystack" == *"$needle"* ]]; then
echo "FAIL: $desc — must not contain '$needle'" >&2
fail=1
fi
return 0
}
# ─── 1. Absent socket → named refusal, NO tmux session. ────────────────────
: > "$LOG_FILE"
stderr_file="$WORK_DIR/stderr-1.tmp"
set +e
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent.sock" 2>"$stderr_file")
rc=$?
set -e
assert "absent socket exit code" "75" "$rc"
err=$(cat "$stderr_file")
assert_contains "absent socket names the failure" "$err" "FAIL_LAUNCH broker-absent"
assert_contains "absent socket names the socket path" "$err" "$SOCK_DIR/absent.sock"
assert_contains "absent socket names a remedy" "$err" "mosaic fleet install"
log1=$(cat "$LOG_FILE")
assert_not_contains "absent socket must not create a tmux session" "$log1" "TMUX-NEW-SESSION-INVOKED"
# ─── 2. Present socket → passes the preflight. ─────────────────────────────
# Expected: ownership/env checks AFTER the preflight may refuse (fixture is
# minimal by design); the assertion is only that the refusal is NOT
# broker-absent and the exit is NOT 75.
# Create a REAL unix socket: a detached python holder binds it and stays alive
# for the duration (bash cannot create sockets; a foreground python would
# close the socket on exit and -S on a closed-but-unlinked path fails). Written
# as a script file + setsid nohup so no job-control/heredoc interaction with
# set -e can silently kill the suite.
# AF_UNIX binds cap at 108 path bytes; the suite's workdir exceeds that, so
# the live socket lives at a SHORT path under /tmp (unique per run, cleaned
# with the suite). The preflight takes its socket path explicitly, so this
# stays fully controlled.
LIVE_SOCK=$(mktemp -u /tmp/mosaic-preflight-XXXXXX.sock)
trap 'rm -f "$LIVE_SOCK"' EXIT
rm -f "$SOCK_DIR/live.sock" "$LIVE_SOCK"
cat > "$SOCK_DIR/holder.py" <<'PY'
import socket, sys, time
path = sys.argv[1]
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.bind(path)
s.listen(1)
time.sleep(120)
PY
python3 "$SOCK_DIR/holder.py" "$LIVE_SOCK" >/dev/null 2>"$SOCK_DIR/holder.err" &
HOLDER_PID=$!
# Wait for the socket object to exist (bind is near-instant, but do not race it).
for _ in $(seq 1 50); do
[ -S "$LIVE_SOCK" ] && break
sleep 0.1
done
if [ ! -S "$LIVE_SOCK" ]; then
echo "FAIL: could not create live socket fixture (holder pid $HOLDER_PID)" >&2
ps -p "$HOLDER_PID" -o pid,stat,cmd --no-headers >&2 || echo "(holder exited)" >&2
cat "$SOCK_DIR/holder.err" >&2 || true
exit 1
fi
: > "$LOG_FILE"
set +e
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$LIVE_SOCK" 2>"$WORK_DIR/stderr-2.tmp")
rc=$?
set -e
# The preflight PASSED if the failure (whatever later stage refused) is NOT
# the broker refusal, and tmux was reached or a later precondition named
# something else.
err2=$(cat "$WORK_DIR/stderr-2.tmp")
assert_not_contains "live socket must not refuse broker-absent" "$err2" "broker-absent"
if [[ "$rc" == "75" ]]; then
echo "FAIL: live socket — preflight still refused (exit 75) with a live socket" >&2
fail=1
fi
# ─── 3. Explicit socket env wins over XDG default. ─────────────────────────
set +e
out=$(run_session_script "" XDG_RUNTIME_DIR="$SOCK_DIR/no-runtime-here" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent2.sock" 2>"$WORK_DIR/stderr-3.tmp")
rc=$?
set -e
assert "explicit env wins (exit 75)" "75" "$rc"
assert_contains "explicit env path named" "$(cat "$WORK_DIR/stderr-3.tmp")" "$SOCK_DIR/absent2.sock"
# ─── 4. --stop is not fenced on the broker. ────────────────────────────────
: > "$LOG_FILE"
set +e
out=$(run_session_script "--stop" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent3.sock" 2>"$WORK_DIR/stderr-4.tmp")
rc=$?
set -e
err4=$(cat "$WORK_DIR/stderr-4.tmp")
assert_not_contains "--stop must not refuse broker-absent" "$err4" "broker-absent"
if [[ "$rc" == "75" ]]; then
echo "FAIL: --stop — exit 75 means teardown was fenced on the broker" >&2
fail=1
fi
kill "$HOLDER_PID" 2>/dev/null || true
if [[ "$fail" -eq 0 ]]; then
echo "start-agent-session lease-broker preflight regression passed"
fi
exit "$fail"
@@ -484,27 +484,6 @@ assert_missing_pane_binary_rejected() {
assert_missing_pane_binary_rejected mosaic
assert_missing_pane_binary_rejected pi
# #1256. Name resolution is not executable reachability. A script can resolve
# while its /usr/bin/env shebang command is absent from PANE_PATH; reject that
# before tmux creates the doomed session.
: > "$TMUX_CALLS"
HOME_UNEXECUTABLE="$ROOT/unexecutable-shebang"
write_generated "$HOME_UNEXECUTABLE" "coder-unexecutable"
rm -f "$HOME_UNEXECUTABLE/.npm-global/bin/mosaic"
printf '#!/usr/bin/env mosaic-test-absent-interpreter\n' > \
"$HOME_UNEXECUTABLE/.npm-global/bin/mosaic"
chmod +x "$HOME_UNEXECUTABLE/.npm-global/bin/mosaic"
if output=$(MOSAIC_TEST_PANE_PID=$$ run_start "$HOME_UNEXECUTABLE" coder-unexecutable 2>&1); then
fail "launcher accepted a resolved mosaic script with an absent shebang command"
fi
echo "$output" | grep -qF 'code=unexecutable-binary' || \
fail "unexecutable shebang diagnostic missing: $output"
echo "$output" | grep -qF 'mosaic-test-absent-interpreter' || \
fail "unexecutable shebang diagnostic did not name the missing dependency"
if tr '\0' '\n' < "$TMUX_CALLS" | grep -qF new-session; then
fail "launcher created a session after its shebang dependency check failed"
fi
# #1241. tmux destroys a session when its pane command exits, so no pane PID a
# second after new-session means the runtime died on startup. This used to be a
# WARNING about the heartbeat sidecar followed by exit 0 — three layers above it
@@ -39,3 +39,20 @@ packages/mosaic/framework/tools/tmux/test-send-message-verdict.sh | requires rea
# recorded judgement. These lines ARE that judgement, signed.)
packages/mosaic/framework/tools/orchestrator/smoke-test.sh | behavior smoke checks for coord continue/run workflows, run manually by orchestrator seats; unmeasured in CI; #1017 burndown
packages/mosaic/framework/tools/wake/validate-973/microtest-wake-assert.sh | #973 instrument self-test, run as a precondition of the validate-973 evidence procedure rather than as a standing CI suite; #1017 burndown candidate
# --- tools/fleet: precondition is unsatisfiable in the CI image (#1271) ---
# Signed by fred (sb-it-1-dt, 2026-08-16) at origin/next 476db12.
# This suite asserts the launcher's behaviour when `mosaic` and `pi` are MISSING.
# It shims fakes into $FAKE_BIN, but the constructed PANE_PATH always ends in the
# real system path, so on a host that installs those binaries the missing-binary
# cases cannot be measured at all. The suite's own guard (line 103) says so and
# fails rather than reporting a pass it cannot back. That guard is correct.
# The error was wiring the suite into CI: #1017 (c56483eb) enumerated it and
# dropped this exclusion, and the CI image provides `pi` in the system path, so
# it has failed on every pipeline since. Measured 2026-08-16 across pipelines
# 2444 (#1256), 2438 (#1240) and 2441 (#1017-quality): exactly one FAIL line in
# each full log, identical, this assertion; control `zzz-not-present-zzz` -> 0.
# Burn-down and the full measurement are tracked in #1271; unwired by PR #1270.
# Because test:framework-shell is one && chain and this sat at position 44 of 48,
# the four suites after it had not run at all since the merge.
packages/mosaic/framework/tools/fleet/test-start-agent-session.sh | precondition unsatisfiable in the CI image: asserts missing-binary behaviour, but PANE_PATH always ends in the system path and the image provides `pi` there; guard at line 103 fails by design rather than passing unmeasured. Burn down by controlling the tail of PANE_PATH inside the test. NOT by removing `pi` from the image: the CI image installs @earendil-works/[email protected] deliberately (measured in pipeline 2444's test-step log), and other suites depend on that pin. Burn-down tracked in #1271
+1 -1
View File
@@ -25,7 +25,7 @@
"lint": "eslint src",
"typecheck": "tsc --noEmit",
"test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/fleet/test-start-agent-session.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh"
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh"
},
"dependencies": {
"@mosaicstack/brain": "workspace:*",
@@ -0,0 +1,177 @@
import { lstat, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, describe, expect, it } from 'vitest';
import { placeUnitFile, resolveLeaseBrokerSocketForPreflight } from './fleet.js';
/**
* Unit-placement regression harness for #1292.
*
* The two measured defects this suite pins:
* 1. `systemctl enable <name>` does NOT rewrite an existing by-path
* wants-symlink so placement must remove stale residue explicitly, and
* acceptance asserts on the RESULTING SYMLINK TARGET, never on the enable
* call's argument (asserting the call cannot see where the link ended up).
* 2. Node's copyFile FOLLOWS a by-path symlink at the destination and
* overwrites the SEED template. Acceptance asserts on the SEED's bytes
* AND mtime unchanged which is the only check that can redden for
* finding 2. The symlink-target assertion catches finding 1; these are
* different defects with different failure modes.
*
* Fixtures are entirely inside tmpdirs (source template, active systemd dir,
* wants dir) no real host paths are touched by this suite.
*/
describe('placeUnitFile (#1292 unit placement)', () => {
const cleanup: string[] = [];
afterEach(async () => {
while (cleanup.length > 0) {
await rm(cleanup.pop()!, { recursive: true, force: true });
}
});
async function fixture() {
const root = await mkdtemp(join(tmpdir(), 'place-unit-'));
cleanup.push(root);
const seedDir = join(root, 'seed');
const activeDir = join(root, 'active');
await mkdir(seedDir, { recursive: true });
await mkdir(activeDir, { recursive: true });
const seedTemplate = join(seedDir, 'unit-under-test.service');
await writeFile(
seedTemplate,
'[Unit]\nDescription=seed template\n[Service]\nType=oneshot\nExecStart=/bin/true\n[Install]\nWantedBy=default.target\n',
);
const activeSource = join(root, 'active-source.service');
await writeFile(
activeSource,
'[Unit]\nDescription=active copy v2\n[Service]\nType=oneshot\nExecStart=/bin/true\n[Install]\nWantedBy=default.target\n',
);
return { root, seedDir, activeDir, seedTemplate, activeSource };
}
it('places a regular file on a clean host (negative control: no residue anywhere)', async () => {
const f = await fixture();
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
expect(result.unlinkedDestinationSymlink).toBe(false);
expect(result.removedStaleWantsSymlink).toBe(false);
const info = await lstat(join(f.activeDir, 'unit-under-test.service'));
expect(info.isSymbolicLink()).toBe(false);
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toContain(
'active copy v2',
);
// Seed untouched by construction — but assert it, so the clean-host case
// cannot silently regress into seed-mutation.
expect(await readFile(f.seedTemplate, 'utf8')).toContain('seed template');
});
it('by-path residue: unlinks destination symlink, places the file, seed bytes AND mtime unchanged (finding 2)', async () => {
const f = await fixture();
const seedBefore = await readFile(f.seedTemplate, 'utf8');
const mtimeBefore = (await lstat(f.seedTemplate)).mtimeMs;
// The fomo-lin convention: by-path enable left a symlink AT the unit name
// pointing at the seed template, plus a wants-symlink doing the same.
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
const wantsDir = join(f.activeDir, 'default.target.wants');
await mkdir(wantsDir, { recursive: true });
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
expect(result.unlinkedDestinationSymlink).toBe(true);
expect(result.removedStaleWantsSymlink).toBe(true);
// FINDING 2's check: the seed is byte-identical and its mtime did not move.
expect(await readFile(f.seedTemplate, 'utf8')).toBe(seedBefore);
expect((await lstat(f.seedTemplate)).mtimeMs).toBe(mtimeBefore);
// The destination is now a regular file carrying the ACTIVE content.
const destInfo = await lstat(join(f.activeDir, 'unit-under-test.service'));
expect(destInfo.isSymbolicLink()).toBe(false);
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toContain(
'active copy v2',
);
});
it('by-path residue: no wants-symlink remains pointing at the seed (finding 1 residue cleared)', async () => {
const f = await fixture();
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
const wantsDir = join(f.activeDir, 'default.target.wants');
await mkdir(wantsDir, { recursive: true });
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
// After placement the stale wants link is GONE (enable-by-name recreates
// it correctly). A link still present must not point at the seed.
try {
const link = await lstat(join(wantsDir, 'unit-under-test.service'));
if (link.isSymbolicLink()) {
const target = await readFile(join(wantsDir, 'unit-under-test.service'), 'utf8').catch(
async () => '',
);
expect(target).not.toContain('seed template');
}
} catch {
// absent wants link — the expected post-placement state
}
});
it('idempotence: second placement on a reconciled host is a no-op producing the identical final state', async () => {
const f = await fixture();
// Reconciled starting state: regular file at the name, wants link to the active copy.
await writeFile(
join(f.activeDir, 'unit-under-test.service'),
await readFile(f.activeSource, 'utf8'),
);
const wantsDir = join(f.activeDir, 'default.target.wants');
await mkdir(wantsDir, { recursive: true });
await symlink(
join(f.activeDir, 'unit-under-test.service'),
join(wantsDir, 'unit-under-test.service'),
);
const before = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
// No destructive step fired: no unlink, no wants removal.
expect(result.unlinkedDestinationSymlink).toBe(false);
expect(result.removedStaleWantsSymlink).toBe(false);
// Identical final state.
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toBe(before);
const link = await lstat(join(wantsDir, 'unit-under-test.service'));
expect(link.isSymbolicLink()).toBe(true);
});
it('double install on by-path residue converges to the identical reconciled state', async () => {
const f = await fixture();
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
const wantsDir = join(f.activeDir, 'default.target.wants');
await mkdir(wantsDir, { recursive: true });
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
const first = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
const secondRun = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
const second = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
expect(secondRun.unlinkedDestinationSymlink).toBe(false);
expect(second).toBe(first);
});
});
describe('resolveLeaseBrokerSocketForPreflight (#1292 preflight resolution)', () => {
it('explicit MOSAIC_LEASE_BROKER_SOCKET wins', () => {
expect(
resolveLeaseBrokerSocketForPreflight({ MOSAIC_LEASE_BROKER_SOCKET: '/custom/sock' }, 1000),
).toBe('/custom/sock');
});
it('XDG_RUNTIME_DIR next', () => {
expect(resolveLeaseBrokerSocketForPreflight({ XDG_RUNTIME_DIR: '/run/user/1001' }, 1000)).toBe(
'/run/user/1001/mosaic-lease/broker.sock',
);
});
it('falls back to /run/user/<uid>', () => {
expect(resolveLeaseBrokerSocketForPreflight({}, 1002)).toBe(
'/run/user/1002/mosaic-lease/broker.sock',
);
});
});
@@ -1,13 +1,9 @@
import { chmod, mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { join } from 'node:path';
import { Command } from 'commander';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { type FleetReconcileDeps } from '../fleet/fleet-reconciler.js';
import {
type FleetRuntimeProbeResult,
type FleetRuntimeProbeRunner,
} from '../fleet/fleet-runtime-preflight.js';
import { registerFleetCommand, type CommandResult, type FleetCommandDeps } from './fleet.js';
const roster = `
@@ -69,15 +65,12 @@ function program(
mosaicHome: string,
runner: FleetCommandDeps['runner'],
reconcileOverrides: Partial<FleetReconcileDeps> = {},
runtimeProbeRunner: FleetRuntimeProbeRunner = runtimeProbe('present'),
): Command {
const result = new Command();
result.exitOverride();
registerFleetCommand(result, {
mosaicHome,
runner,
frameworkRoot: resolve(process.cwd(), 'framework'),
runtimeProbeRunner,
reconcileDeps: {
homeDirectory: '/home/mosaic',
readHolderIdentity: async () => '11111111-1111-4111-8111-111111111111',
@@ -90,24 +83,6 @@ function program(
return result;
}
function runtimeProbe(status: 'present' | 'missing'): FleetRuntimeProbeRunner {
return async (_command, args): Promise<FleetRuntimeProbeResult> => {
const binaryFlag = args.indexOf('--binary');
const binary = binaryFlag >= 0 ? args[binaryFlag + 1] : undefined;
const effectiveStatus = binary === 'mosaic' ? 'present' : status;
return {
stdout:
`pane_path\u0000/fixture/runtime-bin:/usr/bin:/bin\u0000status\u0000${effectiveStatus}\u0000` +
`binary_path\u0000${effectiveStatus === 'present' ? `/fixture/runtime-bin/${binary ?? 'unknown'}` : ''}\u0000` +
`dependency\u0000${effectiveStatus === 'present' ? 'node' : ''}\u0000` +
`probe_command\u0000${effectiveStatus === 'present' ? 'node --version' : ''}\u0000` +
`probe_exit\u0000${effectiveStatus === 'present' ? '0' : ''}\u0000probe_output\u0000\u0000`,
stderr: '',
exitCode: effectiveStatus === 'present' ? 0 : 69,
};
};
}
function capture(): string[] {
const lines: string[] = [];
vi.spyOn(console, 'log').mockImplementation((value: string): void => {
@@ -177,64 +152,13 @@ describe('mosaic fleet reconciler commands', (): void => {
expect(lines.map((line: string): unknown => JSON.parse(line))).toMatchObject([
{ applied: false, lifecycle: 'not-applied' },
{
applied: false,
lifecycle: 'not-applied',
checks: {
fleetCliExecutable: [
{
check: 'fleet-cli-executable',
status: 'ok',
requestedBy: ['coder0'],
dependency: 'node',
probeCommand: 'node --version',
},
],
fleetRuntimeAvailability: [
{
check: 'fleet-runtime-available',
runtime: 'pi',
status: 'ok',
requestedBy: ['coder0'],
},
],
},
},
{ applied: false, lifecycle: 'not-applied' },
]);
expect(
calls.every((call: string[]): boolean => call[0] !== 'systemctl' || call[2] === 'show'),
).toBe(true);
});
it('reports a missing roster runtime as a named non-green doctor check', async (): Promise<void> => {
const home = await fleetHome();
const lines = capture();
await program(home, ownedRunner([]), {}, runtimeProbe('missing')).parseAsync([
'node',
'mosaic',
'fleet',
'doctor',
]);
expect(JSON.parse(lines.pop() ?? '')).toMatchObject({
applied: false,
checks: {
fleetRuntimeAvailability: [
{
check: 'fleet-runtime-available',
runtime: 'pi',
status: 'missing',
requestedBy: ['coder0'],
panePath: '/fixture/runtime-bin:/usr/bin:/bin',
installCommand: 'npm install -g @earendil-works/pi-coding-agent',
},
],
},
});
expect(process.exitCode).toBe(1);
});
it.each(['start', 'stop', 'restart'] as const)(
'uses exact roster-owned systemd targeting for %s',
async (operation: 'start' | 'stop' | 'restart'): Promise<void> => {
@@ -8,18 +8,10 @@ import {
type FleetReconcileCommand,
type FleetReconcileDeps,
} from '../fleet/fleet-reconciler.js';
import {
inspectFleetRuntimeAvailability,
type FleetRuntimeInspection,
type FleetRuntimePreflightCheck,
type FleetRuntimeProbeRunner,
} from '../fleet/fleet-runtime-preflight.js';
import { parseRosterV2 } from '../fleet/roster-v2.js';
export interface FleetReconcilerCommandDeps {
readonly runner: CommandRunner;
readonly runtimeProbeRunner?: FleetRuntimeProbeRunner;
readonly frameworkRoot?: string;
readonly mosaicHome?: string;
readonly reconcileDeps?: Omit<FleetReconcileDeps, 'runner' | 'mosaicHome'>;
}
@@ -79,10 +71,6 @@ export async function executeReconcilerCommand(
const mosaicHome = resolveMosaicHome(fleetCommand, deps);
const rosterPath = resolveRosterPath(fleetCommand, mosaicHome);
const roster = parseRosterV2(await readFile(rosterPath, 'utf8'), 'yaml');
const runtimeInspection =
operation === 'doctor'
? await inspectRuntimeAvailability(roster.agents, mosaicHome, deps)
: undefined;
const mutating = operation === 'apply' || operation === 'reconcile' || isLifecycle(operation);
const expectedGeneration = mutating
? parseExpectedGeneration(opts.expectedGeneration)
@@ -102,31 +90,8 @@ export async function executeReconcilerCommand(
...(deps.reconcileDeps ?? {}),
},
});
printJson(operation === 'doctor' ? { ...result, checks: runtimeInspection } : result);
const executableFailure =
runtimeInspection !== undefined &&
[...runtimeInspection.fleetCliExecutable, ...runtimeInspection.fleetRuntimeAvailability].some(
(check: FleetRuntimePreflightCheck): boolean => check.status !== 'ok',
);
process.exitCode =
result.recovery === undefined && result.cleanup === undefined && !executableFailure ? 0 : 1;
}
async function inspectRuntimeAvailability(
agents: readonly { readonly name: string; readonly runtime: string }[],
mosaicHome: string,
deps: FleetReconcilerCommandDeps,
): Promise<FleetRuntimeInspection> {
if (deps.frameworkRoot === undefined || deps.runtimeProbeRunner === undefined) {
throw new Error('Fleet doctor runtime preflight dependencies are unavailable.');
}
return inspectFleetRuntimeAvailability({
mosaicHome,
agentEnvDir: join(mosaicHome, 'fleet', 'agents'),
helperPath: join(deps.frameworkRoot, 'tools', 'fleet', 'pane-runtime-path.sh'),
agents,
runner: deps.runtimeProbeRunner,
});
printJson(result);
process.exitCode = result.recovery === undefined && result.cleanup === undefined ? 0 : 1;
}
function isLifecycle(operation: FleetReconcileCommand): boolean {
@@ -4,7 +4,6 @@ import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { Command } from 'commander';
import { afterEach, describe, expect, it, vi } from 'vitest';
import type { FleetRuntimeProbeRunner } from '../fleet/fleet-runtime-preflight.js';
import { registerFleetCommand, type CommandResult, type CommandRunner } from './fleet.js';
/**
@@ -70,7 +69,6 @@ agents:
let tempHome: string | undefined;
const savedHome = process.env.HOME;
const savedMosaicHome = process.env.MOSAIC_HOME;
const savedPath = process.env.PATH;
afterEach(async (): Promise<void> => {
vi.restoreAllMocks();
@@ -79,8 +77,6 @@ afterEach(async (): Promise<void> => {
else process.env.HOME = savedHome;
if (savedMosaicHome === undefined) delete process.env.MOSAIC_HOME;
else process.env.MOSAIC_HOME = savedMosaicHome;
if (savedPath === undefined) delete process.env.PATH;
else process.env.PATH = savedPath;
if (tempHome) await rm(tempHome, { recursive: true, force: true });
tempHome = undefined;
});
@@ -89,7 +85,7 @@ afterEach(async (): Promise<void> => {
* A HOME with a roster-v2 fleet and nothing else the greenfield shape, before
* anything has been installed, applied or started.
*/
async function v2Home(options: { withPaneRuntime?: boolean } = {}): Promise<string> {
async function v2Home(): Promise<string> {
tempHome = await mkdtemp(join(tmpdir(), 'mosaic-fleet-v2-dispatch-'));
process.env.HOME = tempHome;
delete process.env.MOSAIC_HOME;
@@ -101,12 +97,6 @@ async function v2Home(options: { withPaneRuntime?: boolean } = {}): Promise<stri
await writeFile(join(mosaicHome, 'fleet', 'roles', 'code.md'), '`class: code`\n\n# code\n', {
mode: 0o600,
});
const runtimeDir = join(tempHome, '.npm-global', 'bin');
await mkdir(runtimeDir, { recursive: true });
await writeFile(join(runtimeDir, 'mosaic'), '#!/bin/sh\nexit 0\n', { mode: 0o755 });
if (options.withPaneRuntime !== false) {
await writeFile(join(runtimeDir, 'pi'), '#!/bin/sh\nexit 0\n', { mode: 0o755 });
}
return mosaicHome;
}
@@ -123,17 +113,10 @@ const greenfieldRunner: CommandRunner = async (command): Promise<CommandResult>
return { stdout: '', stderr: '', exitCode: 1 };
};
function program(
runner: CommandRunner = greenfieldRunner,
runtimeProbeRunner?: FleetRuntimeProbeRunner,
): Command {
function program(runner: CommandRunner = greenfieldRunner): Command {
const result = new Command();
result.exitOverride();
registerFleetCommand(result, {
runner,
frameworkRoot: resolve(process.cwd(), 'framework'),
...(runtimeProbeRunner === undefined ? {} : { runtimeProbeRunner }),
});
registerFleetCommand(result, { runner, frameworkRoot: resolve(process.cwd(), 'framework') });
return result;
}
@@ -183,93 +166,6 @@ describe('mosaic fleet ps — roster v2', (): void => {
});
describe('mosaic fleet install — roster v2', (): void => {
it('rejects a roster runtime missing from the pane PATH before installing any files', async (): Promise<void> => {
const mosaicHome = await v2Home({ withPaneRuntime: false });
const operatorBin = join(tempHome!, 'operator-bin');
await mkdir(operatorBin, { recursive: true });
await writeFile(join(operatorBin, 'pi'), '#!/bin/sh\nexit 0\n', { mode: 0o755 });
process.env.PATH = `${operatorBin}:${savedPath ?? '/usr/bin:/bin'}`;
let message = '';
try {
await program().parseAsync([
'node',
'mosaic',
'fleet',
'--mosaic-home',
mosaicHome,
'install',
'--no-enable',
]);
} catch (error: unknown) {
message = error instanceof Error ? error.message : String(error);
}
expect(message).toContain('runtime=pi');
expect(message).toContain('requested_by=coder0,coder1');
expect(message).toContain('pane_path=');
expect(message).toContain('npm install -g @earendil-works/pi-coding-agent');
expect(message).not.toContain(operatorBin);
expect(
await exists(join(tempHome!, '.config', 'systemd', 'user', '[email protected]')),
).toBe(false);
expect(await exists(join(mosaicHome, 'fleet', 'holder.id'))).toBe(false);
expect(await exists(join(mosaicHome, 'tools'))).toBe(false);
});
it('rejects resolved Node-shebang commands when Node is absent from the pane PATH', async (): Promise<void> => {
const mosaicHome = await v2Home();
const runtimeDir = join(tempHome!, '.npm-global', 'bin');
const nodeScript = '#!/usr/bin/env node\nconsole.log("should-not-run");\n';
await writeFile(join(runtimeDir, 'mosaic'), nodeScript, { mode: 0o755 });
await writeFile(join(runtimeDir, 'pi'), nodeScript, { mode: 0o755 });
await writeFile(join(tempHome!, '.npmrc'), `prefix=${join(tempHome!, 'absent-prefix')}\n`);
const isolatedSystemPath = join(tempHome!, 'system-bin');
await mkdir(isolatedSystemPath, { recursive: true });
const isolatedProbeRunner: FleetRuntimeProbeRunner = async (
command,
args,
): Promise<CommandResult> =>
new Promise((settle) => {
const child = execFile(
command,
[...args, '--system-path', isolatedSystemPath],
{ encoding: 'utf8' },
(error, stdout, stderr) => {
settle({
stdout,
stderr,
exitCode: child.exitCode ?? (error === null ? 0 : 1),
});
},
);
});
let message = '';
try {
await program(greenfieldRunner, isolatedProbeRunner).parseAsync([
'node',
'mosaic',
'fleet',
'--mosaic-home',
mosaicHome,
'install',
'--no-enable',
]);
} catch (error: unknown) {
message = error instanceof Error ? error.message : String(error);
}
expect(message).toContain('check=fleet-cli-executable');
expect(message).toContain('binary=mosaic');
expect(message).toContain('dependency=node');
expect(message).toContain('check=fleet-runtime-available');
expect(message).toContain('runtime=pi');
expect(message).not.toContain('/usr/bin');
expect(await exists(join(mosaicHome, 'fleet', 'holder.id'))).toBe(false);
expect(await exists(join(mosaicHome, 'tools'))).toBe(false);
});
it('places the tool files and unit templates', async (): Promise<void> => {
const mosaicHome = await v2Home();
capture();
@@ -287,11 +183,9 @@ describe('mosaic fleet install — roster v2', (): void => {
]) {
expect(await exists(join(systemdUserDir, unit))).toBe(true);
}
for (const tool of ['start-agent-session.sh', 'pane-runtime-path.sh']) {
const toolPath = join(mosaicHome, 'tools', 'fleet', tool);
expect(await exists(toolPath)).toBe(true);
expect((await stat(toolPath)).mode & 0o777).toBe(0o755);
}
const launcher = join(mosaicHome, 'tools', 'fleet', 'start-agent-session.sh');
expect(await exists(launcher)).toBe(true);
expect((await stat(launcher)).mode & 0o777).toBe(0o755);
});
it('writes NO generated env — that file belongs to the reconciler (#791)', async (): Promise<void> => {
+110 -9
View File
@@ -835,13 +835,25 @@ describe('fleet command construction', () => {
};
const program = new Command();
program.exitOverride();
registerFleetCommand(program, { runner, mosaicHome: home });
// #1292: inject a present broker socket so the preflight passes and this
// spec keeps testing its ORIGINAL property (holder-before-agent ordering).
// The preflight's own refusal behavior has dedicated specs below.
registerFleetCommand(program, {
runner,
mosaicHome: home,
checkBrokerSocket: async () => true,
});
try {
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
await program.parseAsync(['node', 'mosaic', 'fleet', 'stop']);
expect(calls).toEqual([
// #1292: fleet start enables + starts the broker FIRST (enable is
// idempotent; the unit exists after install), re-checking the socket
// before any holder/agent lifecycle effect.
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
['systemctl', '--user', 'start', 'mosaic-tmux-holder.service'],
['systemctl', '--user', 'start', '[email protected]'],
['systemctl', '--user', 'stop', '[email protected]'],
@@ -852,6 +864,92 @@ describe('fleet command construction', () => {
}
});
it('fleet start refuses with a named error when the broker socket does not appear (#1292)', async () => {
const home = await tempDir();
const rosterPath = join(home, 'fleet', 'roster.yaml');
await mkdir(join(home, 'fleet'), { recursive: true });
await writeFile(
rosterPath,
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
'\n',
),
);
const calls: string[][] = [];
const runner: CommandRunner = async (command, args) => {
calls.push([command, ...args]);
return { stdout: '', stderr: '', exitCode: 0 };
};
const program = new Command();
program.exitOverride();
const errors: string[] = [];
const origError = console.error;
console.error = (...args: unknown[]) => {
errors.push(args.join(' '));
};
registerFleetCommand(program, {
runner,
mosaicHome: home,
checkBrokerSocket: async () => false,
});
try {
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
// Refused: no holder/agent starts were issued after the broker attempt.
expect(calls).toEqual([
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
]);
expect(errors.join('\n')).toContain('broker-absent');
expect(errors.join('\n')).toContain('mosaic fleet install');
} finally {
console.error = origError;
await rm(home, { recursive: true, force: true });
}
});
it('fleet start re-probes the broker on the SECOND invocation — no ActiveState trust (#1292 sticky half)', async () => {
const home = await tempDir();
const rosterPath = join(home, 'fleet', 'roster.yaml');
await mkdir(join(home, 'fleet'), { recursive: true });
await writeFile(
rosterPath,
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
'\n',
),
);
const calls: string[][] = [];
const runner: CommandRunner = async (command, args) => {
calls.push([command, ...args]);
return { stdout: '', stderr: '', exitCode: 0 };
};
const program = new Command();
program.exitOverride();
// Broker socket NEVER appears — the second start must refuse exactly like
// the first; RemainAfterExit-style stale unit state changes nothing
// because the check is the socket, not systemctl.
registerFleetCommand(program, {
runner,
mosaicHome: home,
checkBrokerSocket: async () => false,
});
const errors: string[] = [];
const origError = console.error;
console.error = (...args: unknown[]) => {
errors.push(args.join(' '));
};
try {
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
// Two invocations, each refusing after its own broker attempt:
expect(
calls.filter((c) => c.join(' ') === 'systemctl --user start [email protected]'),
).toHaveLength(0);
expect(errors.filter((e) => e.includes('broker-absent')).length).toBeGreaterThanOrEqual(2);
} finally {
console.error = origError;
await rm(home, { recursive: true, force: true });
}
});
it('waits for an in-flight restart to clear before relaunching (re-entry guard)', async () => {
const home = await tempDir();
const rosterPath = join(home, 'fleet', 'roster.yaml');
@@ -1277,10 +1375,6 @@ describe('fleet command construction', () => {
const home = await tempDir();
process.env.HOME = home;
delete process.env.MOSAIC_HOME;
const runtimeDir = join(home, '.npm-global', 'bin');
await mkdir(runtimeDir, { recursive: true });
await writeFile(join(runtimeDir, 'pi'), '#!/bin/sh\nexit 0\n', { mode: 0o755 });
await writeFile(join(runtimeDir, 'mosaic'), '#!/bin/sh\nexit 0\n', { mode: 0o755 });
const mosaicHome = join(home, '.config', 'mosaic');
const program = new Command();
program.exitOverride();
@@ -1319,10 +1413,6 @@ describe('fleet command construction', () => {
const originalHome = process.env.HOME;
const home = await tempDir();
process.env.HOME = home;
const runtimeDir = join(home, '.npm-global', 'bin');
await mkdir(runtimeDir, { recursive: true });
await writeFile(join(runtimeDir, 'pi'), '#!/bin/sh\nexit 0\n', { mode: 0o755 });
await writeFile(join(runtimeDir, 'mosaic'), '#!/bin/sh\nexit 0\n', { mode: 0o755 });
const mosaicHome = join(home, '.config', 'mosaic');
const agentEnvDir = join(mosaicHome, 'fleet', 'agents');
const fleetDir = join(mosaicHome, 'fleet');
@@ -2073,8 +2163,19 @@ describe('fleet install — auto-enable units for boot-survival', () => {
await enableFleetUnits(runner, minimalRoster, {});
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-lease-broker.service']);
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-tmux-holder.service']);
expect(calls).toContainEqual(['systemctl', '--user', 'enable', '[email protected]']);
// The broker must be enabled BEFORE the holder and agents: a start of any
// gated runtime without the broker is exactly the #1292 4-second death.
const brokerIndex = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user enable mosaic-lease-broker.service',
);
const holderIndex = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user enable mosaic-tmux-holder.service',
);
expect(brokerIndex).toBeGreaterThanOrEqual(0);
expect(brokerIndex).toBeLessThan(holderIndex);
});
it('install still succeeds when systemctl enable returns non-zero (non-fatal)', async () => {
+213 -80
View File
@@ -3,9 +3,11 @@ import {
access,
chmod,
copyFile,
lstat,
mkdir,
open,
readFile,
readlink,
stat,
unlink,
writeFile,
@@ -60,12 +62,6 @@ import {
writeAgentEnvironmentProjection,
writeManagedFleetRoster,
} from '../fleet/generated-env-boundary.js';
import {
assertFleetRuntimeAvailability,
FleetRuntimePreflightError,
inspectFleetRuntimeAvailability,
type FleetRuntimeProbeRunner,
} from '../fleet/fleet-runtime-preflight.js';
import { registerFleetBacklogCommand } from './fleet-backlog.js';
import { registerFleetPersonaCommand } from './fleet-personas.js';
import { registerFleetProfileCommand } from './fleet-profiles.js';
@@ -95,8 +91,8 @@ export type SleepFn = (ms: number) => Promise<void>;
export interface FleetCommandDeps {
runner?: CommandRunner;
/** Executes the pane-PATH helper under a clean launcher environment. */
runtimeProbeRunner?: FleetRuntimeProbeRunner;
/** Test seam for the #1292 fleet-start broker preflight (socket presence). */
checkBrokerSocket?: (path: string) => Promise<boolean> | boolean;
/** Injectable interactive runner for commands needing inherited TTY (e.g., `tmux attach`). */
interactiveRunner?: InteractiveRunner;
/**
@@ -805,6 +801,96 @@ export function buildSystemdEnableCommand(unit: string): string[] {
return ['systemctl', '--user', 'enable', unit];
}
/**
* Place a unit file into the ACTIVE systemd user directory, never through a
* symlink (#1292, measured 2026-08-17).
*
* SET-INDEPENDENCE (fomo-lin, 2026-08-17): the set of unit names carrying
* by-path residue and the set of unit names this install copies are
* INDEPENDENT. Until 0.0.50 they were disjoint only by accident of which
* units the install happened to name fomo-lin survived copy-through solely
* because its one by-path symlink (the broker) was the one unit the install
* did NOT copy. Adding the broker to the copy set made the intersection
* non-empty on the first run. Whoever adds a fifth unit to the placement
* list inherits this helper and its unlink step; do not place units with a
* bare copyFile.
*
* A host provisioned by the enable-by-path convention carries a symlink AT
* the unit-name path in ~/.config/systemd/user/ pointing at the shipped
* template under ~/.config/mosaic/systemd/user/. Node's copyFile FOLLOWS
* that link and overwrites the SEED template instead of placing the active
* unit (verified with fs.copyFile on a throwaway systemd user instance)
* silent, rc=0, and it mutates the directory every later reseed reads from.
* The same measurement showed `systemctl enable <name>` does NOT rewrite an
* existing by-path wants-symlink, so reconciliation must be explicit.
*
* Placement therefore: if the destination is a symlink, unlink it first
* (unlink copy copy-then-unlink would mutate the seed and then destroy
* the evidence that it did); then copy. Also removes a stale
* `default.target.wants/<name>` symlink that points outside the active
* directory (readlink NOT readFile, which follows the link and returns the
* target's CONTENT), so the subsequent enable-by-name recreates it against
* the active copy. Idempotent: on a clean or already-reconciled destination
* every step is a no-op (the copy rewrites identical bytes).
*
* Returns what was done, for assertions and install reporting.
*/
export interface PlaceUnitResult {
readonly unit: string;
readonly destination: string;
/** A symlink at the unit-name path was unlinked (by-path residue). */
readonly unlinkedDestinationSymlink: boolean;
/** A stale wants-symlink pointing outside the active dir was removed. */
readonly removedStaleWantsSymlink: boolean;
}
export async function placeUnitFile(
source: string,
systemdUserDir: string,
unit: string,
): Promise<PlaceUnitResult> {
const destination = join(systemdUserDir, unit);
let unlinkedDestinationSymlink = false;
try {
const destInfo = await lstat(destination);
if (destInfo.isSymbolicLink()) {
await unlink(destination);
unlinkedDestinationSymlink = true;
}
} catch {
// absent destination — nothing to unlink
}
await copyFile(source, destination);
let removedStaleWantsSymlink = false;
const wantsLink = join(systemdUserDir, 'default.target.wants', unit);
try {
const wantsInfo = await lstat(wantsLink);
if (wantsInfo.isSymbolicLink()) {
// readlink — NOT readFile: readFile FOLLOWS the link and returns the
// target file's CONTENT, which is not the question being asked.
let target: string | undefined;
try {
target = await readlink(wantsLink);
} catch {
target = undefined;
}
// Normalize (systemctl writes absolute targets; a relative one resolves
// against the wants dir). A wants-symlink pointing anywhere other than
// the active copy (the by-path convention points at the seed template)
// survives enable-by-name unchanged — remove it so enable recreates it.
if (target !== undefined && resolve(dirname(wantsLink), target) !== destination) {
await unlink(wantsLink);
removedStaleWantsSymlink = true;
}
}
} catch {
// absent wants link — nothing to reconcile
}
return { unit, destination, unlinkedDestinationSymlink, removedStaleWantsSymlink };
}
/**
* Returns the systemctl --user disable command for a given unit.
* Used by `fleet remove` so a removed agent's enabled unit cannot resurrect on
@@ -839,6 +925,22 @@ export async function enableFleetUnits(
let succeeded = 0;
let failed = 0;
// The lease broker ships with the fleet and every gated runtime needs it
// (#1292): seats die at lease registration without it, and no documented
// path ever enabled it. Enabled first — alongside the holder — and the
// unit must have been placed by installFleet's placeUnitFile step.
const brokerResult = await runner(
...splitCommand(buildSystemdEnableCommand('mosaic-lease-broker.service')),
);
if (brokerResult.exitCode === 0) {
succeeded++;
} else {
failed++;
process.stderr.write(
`Warning: could not enable mosaic-lease-broker.service: ${brokerResult.stderr || brokerResult.stdout || 'non-zero exit'}\n`,
);
}
const holderResult = await runner(
...splitCommand(buildSystemdEnableCommand('mosaic-tmux-holder.service')),
);
@@ -1437,10 +1539,6 @@ export function isSendAccepted(capturedOutput: string): SendVerifyResult {
export function registerFleetCommand(program: Command, deps: FleetCommandDeps = {}): Command {
const runner = deps.runner ?? runCommand;
const runtimeProbeRunner: FleetRuntimeProbeRunner =
deps.runtimeProbeRunner ??
(async (command: string, args: readonly string[]): Promise<CommandResult> =>
runCommand(command, [...args]));
const sleepFn = deps.sleepFn ?? defaultSleep;
const paths = resolveFleetPaths(deps.mosaicHome);
const frameworkRoot = deps.frameworkRoot ?? resolveFrameworkRoot();
@@ -1539,7 +1637,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.description('Install local fleet tools and user systemd units')
.option('--no-enable', 'Skip enabling units for boot-survival')
.action(async (opts: { enable?: boolean }) => {
await installFleet(cmd, frameworkRoot, runtimeProbeRunner);
await installFleet(cmd, frameworkRoot, runner);
// Unit enablement needs agent names only, so it reads either version.
const roster = await loadRosterReadModel(cmd);
await enableFleetUnits(runner, roster, opts);
@@ -1550,7 +1648,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.description('Install local fleet tools and user systemd units')
.option('--no-enable', 'Skip enabling units for boot-survival')
.action(async (opts: { enable?: boolean }) => {
await installFleet(cmd, frameworkRoot, runtimeProbeRunner);
await installFleet(cmd, frameworkRoot, runner);
// Unit enablement needs agent names only, so it reads either version.
const roster = await loadRosterReadModel(cmd);
await enableFleetUnits(runner, roster, opts);
@@ -1603,6 +1701,37 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
);
return;
}
if (action === 'start') {
// Broker preflight (#1292), re-probed on EVERY invocation: a
// gated runtime started without a live lease broker dies ~4s in
// while the unit reports active (RemainAfterExit) — enabling +
// starting here and then RE-CHECKING the socket refuses loudly
// instead of reporting rc0 over a doomed start. This is the
// second-start check as much as the first: it never trusts unit
// ActiveState.
await runChecked(runner, [
'systemctl',
'--user',
'enable',
'mosaic-lease-broker.service',
]);
await runChecked(runner, [
'systemctl',
'--user',
'start',
'mosaic-lease-broker.service',
]);
if (!(await brokerSocketPresent(deps))) {
console.error(
'[fleet] broker-absent: lease broker socket did not appear after enable+start (#1292).',
);
console.error(
'[fleet] remedy: mosaic fleet install (it reconciles either enable convention)',
);
process.exitCode = 1;
return;
}
}
if (action === 'restart') {
// Serialize the holder+agents teardown/relaunch behind the restart lock
// so a re-entrant restart waits for clean shutdown before relaunching,
@@ -2096,8 +2225,6 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
});
registerFleetReconcilerCommands(cmd, {
runner,
runtimeProbeRunner,
frameworkRoot,
mosaicHome: deps.mosaicHome,
reconcileDeps: deps.reconcileDeps,
});
@@ -2366,65 +2493,19 @@ export function registerFleetAgentCommands(
async function installFleet(
cmd: Command,
frameworkRoot: string,
runtimeProbeRunner: FleetRuntimeProbeRunner,
runner: CommandRunner,
): Promise<void> {
const activePaths = resolveFleetPaths(cmd.opts<{ mosaicHome: string }>().mosaicHome);
assertDefaultMosaicHomeForSystemd(activePaths.mosaicHome);
// Read and preflight before the first mkdir/copy/chmod/write. A successful
// install must mean every roster runtime is executable in the eventual pane,
// not merely visible to the operator who invoked this command.
// Read model first: every file this function places is roster-independent, and
// the v1 parser would reject a v2 roster before any of them were written.
const roster = await loadRosterReadModel(cmd);
const v1Roster = roster.version === 1 ? await loadRosterForCommand(cmd) : undefined;
const preflightV1Projections =
v1Roster === undefined
? []
: await Promise.all(
v1Roster.agents.map((agent: FleetAgent) =>
prepareAgentEnvironmentProjection({
mosaicHome: activePaths.mosaicHome,
agentEnvDir: activePaths.agentEnvDir,
agentName: agent.name,
generated: generateAgentEnvValues(v1Roster, agent),
}),
),
);
const preflightAgents =
v1Roster === undefined
? roster.agents
: v1Roster.agents.map((agent: FleetAgent, index: number) => {
const prepared = preflightV1Projections[index];
if (prepared === undefined) {
throw new Error(`Missing prepared environment projection for ${agent.name}.`);
}
const local = parseAgentEnvironment(prepared.local, 'local');
return {
name: agent.name,
runtime: agent.runtime,
runtimeBin: local['MOSAIC_RUNTIME_BIN'] ?? '',
};
});
const runtimeInspection = await inspectFleetRuntimeAvailability({
mosaicHome: activePaths.mosaicHome,
agentEnvDir: activePaths.agentEnvDir,
helperPath: join(frameworkRoot, 'tools', 'fleet', 'pane-runtime-path.sh'),
agents: preflightAgents,
runner: runtimeProbeRunner,
});
try {
assertFleetRuntimeAvailability(runtimeInspection);
} catch (error: unknown) {
if (error instanceof FleetRuntimePreflightError) {
cmd.error(error.message, { code: 'fleet.runtime-preflight', exitCode: 1 });
}
throw error;
}
await ensureFleetHolderIdentity(activePaths.mosaicHome);
await mkdir(activePaths.fleetToolsDir, { recursive: true });
await mkdir(activePaths.tmuxToolsDir, { recursive: true });
await mkdir(activePaths.systemdUserDir, { recursive: true });
const startAgentSessionPath = join(activePaths.fleetToolsDir, 'start-agent-session.sh');
const paneRuntimePath = join(activePaths.fleetToolsDir, 'pane-runtime-path.sh');
const startInteractionServicePath = join(
activePaths.fleetToolsDir,
'start-interaction-service.sh',
@@ -2438,7 +2519,6 @@ async function installFleet(
const agentSendPath = join(activePaths.tmuxToolsDir, 'agent-send.sh');
const executableToolPaths = [
startAgentSessionPath,
paneRuntimePath,
startInteractionServicePath,
startTmuxHolderPath,
printInteractionPolicyPath,
@@ -2449,7 +2529,6 @@ async function installFleet(
join(frameworkRoot, 'tools', 'fleet', 'start-agent-session.sh'),
startAgentSessionPath,
);
await copyFile(join(frameworkRoot, 'tools', 'fleet', 'pane-runtime-path.sh'), paneRuntimePath);
await copyFile(
join(frameworkRoot, 'tools', 'fleet', 'start-interaction-service.sh'),
startInteractionServicePath,
@@ -2467,18 +2546,40 @@ async function installFleet(
for (const toolPath of executableToolPaths) {
await chmod(toolPath, 0o755);
}
await copyFile(
join(frameworkRoot, 'systemd', 'user', 'mosaic-tmux-holder.service'),
join(activePaths.systemdUserDir, 'mosaic-tmux-holder.service'),
// Unit placement (#1292): every unit goes through placeUnitFile — never a
// bare copyFile — so a by-path-enable symlink at the destination is
// unlinked rather than written through (copy-through would silently
// overwrite the SEED template, measured 2026-08-17). The lease broker unit
// is placed here too: previously the install named three units and omitted
// the broker entirely, which is why no documented path ever enabled it.
const placedUnits = await Promise.all(
[
'mosaic-tmux-holder.service',
'[email protected]',
'[email protected]',
'mosaic-lease-broker.service',
].map((unit) =>
placeUnitFile(join(frameworkRoot, 'systemd', 'user', unit), activePaths.systemdUserDir, unit),
),
);
await copyFile(
join(frameworkRoot, 'systemd', 'user', '[email protected]'),
join(activePaths.systemdUserDir, '[email protected]'),
);
await copyFile(
join(frameworkRoot, 'systemd', 'user', '[email protected]'),
join(activePaths.systemdUserDir, '[email protected]'),
const reconciled = placedUnits.filter(
(result) => result.unlinkedDestinationSymlink || result.removedStaleWantsSymlink,
);
if (reconciled.length > 0) {
console.log(
`Reconciled ${reconciled.length} unit placement(s) from by-path enable residue: ${reconciled.map((r) => r.unit).join(', ')}`,
);
}
// systemd will not see a replaced unit file without a reload; do it once
// after all placements, before any enable call below. runCommand never
// rejects (it resolves exitCode 127 on spawn error), so a plain await with
// an exitCode check matches the rest of this file's systemctl handling.
const reloadResult = await runner(...splitCommand(['systemctl', '--user', 'daemon-reload']));
if (reloadResult.exitCode !== 0) {
process.stderr.write(
`Warning: systemctl --user daemon-reload after unit placement failed (non-systemd host?): ${reloadResult.stderr || reloadResult.stdout || 'non-zero exit'}\n`,
);
}
// On roster v2 the reconciler owns the generated env: `apply` writes it and
// `regen` rebuilds it, both from projectRosterV2AgentGeneratedEnv. Writing it
@@ -2493,9 +2594,7 @@ async function installFleet(
return;
}
if (v1Roster === undefined) {
throw new Error('Roster version changed while installing fleet files.');
}
const v1Roster = await loadRosterForCommand(cmd);
for (const agent of v1Roster.agents) {
await writeAgentEnvironmentProjection({
mosaicHome: activePaths.mosaicHome,
@@ -2677,6 +2776,40 @@ function splitCommand(command: string[]): [string, string[]] {
return [bin, args];
}
/**
* Lease-broker socket presence for the fleet-start preflight (#1292).
* Resolution precedence matches launch.ts's defaultLeaseBrokerSocket and
* start-agent-session.sh's broker_socket_path: explicit
* MOSAIC_LEASE_BROKER_SOCKET, else $XDG_RUNTIME_DIR/mosaic-lease/broker.sock,
* else /run/user/<uid>/mosaic-lease/broker.sock. Pure filesystem check this
* deliberately does NOT consult systemd state: a unit can be active
* (RemainAfterExit) with no live socket, and the socket is the thing the
* gated runtime connects to. Injectable via deps for tests.
*/
export function resolveLeaseBrokerSocketForPreflight(
env: NodeJS.ProcessEnv = process.env,
uid: number = typeof process.getuid === 'function' ? process.getuid() : 0,
): string {
if (env['MOSAIC_LEASE_BROKER_SOCKET']) return env['MOSAIC_LEASE_BROKER_SOCKET'];
const runtimeDir = env['XDG_RUNTIME_DIR'] ?? `/run/user/${uid}`;
return join(runtimeDir, 'mosaic-lease', 'broker.sock');
}
async function brokerSocketPresent(
deps: FleetCommandDeps,
env: NodeJS.ProcessEnv = process.env,
): Promise<boolean> {
const check = deps.checkBrokerSocket;
if (check) return check(resolveLeaseBrokerSocketForPreflight(env));
try {
const socketPath = resolveLeaseBrokerSocketForPreflight(env);
await access(socketPath, constants.S_IFSOCK);
return true;
} catch {
return false;
}
}
/** All supported fleet profile names. */
export type FleetProfile =
| 'general'
@@ -205,6 +205,12 @@ export async function runLeaseEnforcementDoctorCheck(
message:
`Lease-enforcement hooks (${matchedMarkers.join(', ')}) are wired in ~/.claude/settings.json, but ${reasons.join(' and ')}. ` +
'Every gated tool call will fail closed and BRICK this agent (see #869). ' +
'Remediate by activating the lease-broker supervisor (systemd unit + socket) or by removing the enforcement hooks from ~/.claude/settings.json.',
// #1292: one remedy, correct under BOTH enable conventions (by-path on
// the seed template, and copy-then-enable in the active dir). Written
// from the 2026-08-17 symlink measurement: `systemctl enable` by name
// does NOT rewrite an existing by-path wants-symlink, so teaching a
// manual systemctl line here could leave a host with two competing
// wants links. fleet install reconciles either shape.
'Remedy: run `mosaic fleet install` (it reconciles either enable convention), or remove the enforcement hooks from ~/.claude/settings.json.',
};
}
@@ -459,6 +459,7 @@ describe('FCM-M3-002 reconciler lifecycle acceptance', (): void => {
plan: {
generation: 7,
holder: 'owned',
broker: { unitInstalled: false, socketPresent: false },
agents: [
{
name: 'coder0',
@@ -92,6 +92,112 @@ async function run(command: FleetReconcileCommand, overrides: Partial<FleetRecon
}
describe('fleet roster-owned reconciler', (): void => {
// ── #1292: broker as first-class plan member + broker-first start ordering ──
it('reports broker unit and socket state in the plan (socket is the signal, not unit state)', async (): Promise<void> => {
const result = await run('status', {
statPath: async () => true,
checkBrokerSocket: async () => true,
});
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: true });
});
it('reports a dead broker as socketPresent=false even when the unit is installed (enabled-but-dead is the #1292 shape)', async (): Promise<void> => {
const result = await run('status', {
statPath: async () => true,
checkBrokerSocket: async () => false,
});
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: false });
});
it('reports broker-absent when neither seam is present (defaults false, never guesses healthy)', async (): Promise<void> => {
const result = await run('status');
expect(result.plan.broker).toEqual({ unitInstalled: false, socketPresent: false });
});
it('command start enables and starts the broker BEFORE the holder and any agent unit', async (): Promise<void> => {
const calls: string[][] = [];
const result = await run('start', {
runner: async (command, args) => {
calls.push([command, ...args]);
if (command === 'tmux' && args.includes('list-sessions')) {
return { stdout: '_holder\ncoder0\n', stderr: '', exitCode: 0 };
}
if (command === 'tmux' && args.includes('show-environment')) {
return {
stdout:
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
stderr: '',
exitCode: 0,
};
}
return { stdout: '', stderr: '', exitCode: 0 };
},
});
expect(result.lifecycle).toBe('complete');
const brokerEnable = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user enable mosaic-lease-broker.service',
);
const brokerStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start mosaic-lease-broker.service',
);
const holderStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start mosaic-tmux-holder.service',
);
const agentStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start [email protected]',
);
expect(brokerEnable).toBeGreaterThanOrEqual(0);
expect(brokerStart).toBeGreaterThan(brokerEnable);
// Holder start may be absent (holder 'owned' in this fixture); if present it must follow the broker.
if (holderStart >= 0) expect(holderStart).toBeGreaterThan(brokerStart);
expect(agentStart).toBeGreaterThan(brokerStart);
});
it('apply with a running desired agent also enables and starts the broker first', async (): Promise<void> => {
const calls: string[][] = [];
const runningRoster: FleetRosterV2 = {
...roster,
agents: roster.agents.map((agent) =>
agent.name === 'coder0'
? { ...agent, lifecycle: { enabled: true, desiredState: 'running' as const } }
: agent,
),
};
const result = await executeFleetReconcile({
roster: runningRoster,
command: 'apply',
expectedGeneration: 7,
deps: deps({
readRoster: async () => runningRoster,
runner: async (command, args) => {
calls.push([command, ...args]);
if (command === 'tmux' && args.includes('list-sessions')) {
return { stdout: '_holder\n', stderr: '', exitCode: 0 };
}
if (command === 'tmux' && args.includes('show-environment')) {
return {
stdout:
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
stderr: '',
exitCode: 0,
};
}
return { stdout: '', stderr: '', exitCode: 0 };
},
}),
});
expect(result.applied).toBe(true);
const brokerStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start mosaic-lease-broker.service',
);
const agentStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start [email protected]',
);
expect(brokerStart).toBeGreaterThanOrEqual(0);
expect(agentStart).toBeGreaterThan(brokerStart);
});
it('fails closed on a symlinked fleet ancestor without touching its target', async (): Promise<void> => {
const home = await lockHome();
const fleet = join(home, 'fleet');
@@ -43,6 +43,10 @@ export interface FleetReconcileDeps {
readonly overrideDir?: string;
readonly homeDirectory?: string;
readonly readHolderIdentity?: () => Promise<string>;
/** Test/observation seams for the lease-broker plan member (#1292). */
readonly statPath?: (path: string) => Promise<boolean> | boolean;
readonly checkBrokerSocket?: (path: string) => Promise<boolean> | boolean;
readonly brokerSocketEnv?: NodeJS.ProcessEnv;
readonly validateRoster?: (roster: FleetRosterV2) => Promise<void>;
readonly prepareProjections?: (roster: FleetRosterV2) => Promise<readonly unknown[]>;
readonly applyProjection?: (prepared: unknown) => Promise<unknown>;
@@ -74,6 +78,17 @@ export interface FleetReconcileObservedAgent {
export interface FleetReconcilePlan {
readonly generation: number;
readonly holder: 'owned' | 'missing' | 'ownership-mismatch';
/**
* Lease broker observation (#1292): every gated runtime registers with the
* broker or dies ~4s in a broker not in the plan cannot be reported as
* drifted, which made "broker died an hour ago" and "broker fine"
* produce identical output. `unitInstalled` = unit file present in the
* active dir; `socketPresent` = live broker at the resolved socket path.
*/
readonly broker: {
readonly unitInstalled: boolean;
readonly socketPresent: boolean;
};
readonly agents: readonly FleetReconcileObservedAgent[];
readonly unmanagedSessions: readonly string[];
}
@@ -314,6 +329,39 @@ function isObservational(command: FleetReconcileCommand): boolean {
return command === 'plan' || command === 'status' || command === 'verify' || command === 'doctor';
}
/**
* Observe the lease broker for the plan (#1292). Unit presence via systemctl
* is-system-running is NOT the signal a unit can be enabled-but-dead. The
* authoritative signal is the socket the gated runtimes connect to, matching
* broker-supervisor.ts's `checkBrokerSupervisorHealth` (healthy ===
* socketPresent). Injectable so tests drive every branch without a broker.
*/
async function observeBroker(deps: FleetReconcileDeps): Promise<FleetReconcilePlan['broker']> {
const homeDirectory = deps.homeDirectory ?? homedir();
const env = (deps.brokerSocketEnv ?? process.env) as NodeJS.ProcessEnv;
const uid = typeof process.getuid === 'function' ? process.getuid() : 0;
const runtimeDir = env['XDG_RUNTIME_DIR'] ?? `/run/user/${uid}`;
const socketPath =
env['MOSAIC_LEASE_BROKER_SOCKET'] ?? join(runtimeDir, 'mosaic-lease', 'broker.sock');
const configHome = env['XDG_CONFIG_HOME'] ?? join(homeDirectory, '.config');
const unitPath = join(configHome, 'systemd', 'user', 'mosaic-lease-broker.service');
const statPath = deps.statPath;
const checkBrokerSocket = deps.checkBrokerSocket;
let unitInstalled = false;
let socketPresent = false;
try {
unitInstalled = statPath ? await statPath(unitPath) : false;
} catch {
unitInstalled = false;
}
try {
socketPresent = checkBrokerSocket ? await checkBrokerSocket(socketPath) : false;
} catch {
socketPresent = false;
}
return { unitInstalled, socketPresent };
}
async function observeFleet(
roster: FleetRosterV2,
deps: FleetReconcileDeps,
@@ -324,10 +372,12 @@ async function observeFleet(
'-F',
'#{session_name}',
]);
const broker = await observeBroker(deps);
if (sessionsResult.exitCode !== 0) {
return {
generation: roster.generation,
holder: 'missing',
broker,
agents: await observeAgents(roster, deps, new Set<string>()),
unmanagedSessions: [],
};
@@ -350,6 +400,7 @@ async function observeFleet(
return {
generation: roster.generation,
holder,
broker,
agents: await observeAgents(roster, deps, sessions),
unmanagedSessions: Object.freeze(unmanagedSessions.sort()),
};
@@ -517,6 +568,24 @@ async function executeExplicitLifecycle(
}
}
try {
// Broker FIRST (#1292): a gated runtime started without a running lease
// broker dies ~4 seconds in at registration — enable the unit (install
// places it) and start it before any holder/agent lifecycle effect. The
// socket re-check after start is the same probe observeBroker uses, so a
// unit that starts but never produces a socket is caught here, not four
// seconds later inside a doomed seat.
if (request.command === 'start') {
await runChecked(request.deps, 'systemctl', [
'--user',
'enable',
'mosaic-lease-broker.service',
]);
await runChecked(request.deps, 'systemctl', [
'--user',
'start',
'mosaic-lease-broker.service',
]);
}
if (request.command === 'start' && plan.holder === 'missing') {
await runChecked(request.deps, 'systemctl', [
'--user',
@@ -562,6 +631,12 @@ async function applyDesiredLifecycle(
(agent: FleetRosterV2Agent): boolean =>
agent.lifecycle.enabled && agent.lifecycle.desiredState === 'running',
);
// Broker before any running agent, same ordering and reason as the
// command-driven path above (#1292).
if (needsRunningAgent) {
await runChecked(deps, 'systemctl', ['--user', 'enable', 'mosaic-lease-broker.service']);
await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-lease-broker.service']);
}
if (needsRunningAgent && plan.holder === 'missing') {
await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-tmux-holder.service']);
}
@@ -1,328 +0,0 @@
import { spawn } from 'node:child_process';
import { chmod, mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { afterEach, describe, expect, it } from 'vitest';
import {
inspectFleetRuntimeAvailability,
type FleetRuntimeProbeResult,
type FleetRuntimeProbeRunner,
} from './fleet-runtime-preflight.js';
const helperPath = resolve(process.cwd(), 'framework', 'tools', 'fleet', 'pane-runtime-path.sh');
let cleanup: string | undefined;
afterEach(async (): Promise<void> => {
if (cleanup !== undefined) await rm(cleanup, { recursive: true, force: true });
cleanup = undefined;
});
interface FleetFixture {
readonly root: string;
readonly mosaicHome: string;
readonly agentEnvDir: string;
readonly runtimeDir: string;
}
async function fleetHome(): Promise<FleetFixture> {
const root = await mkdtemp(join(tmpdir(), 'mosaic-fleet-runtime-preflight-'));
cleanup = root;
const mosaicHome = join(root, '.config', 'mosaic');
const agentEnvDir = join(mosaicHome, 'fleet', 'agents');
const runtimeDir = join(root, '.npm-global', 'bin');
await mkdir(agentEnvDir, { recursive: true, mode: 0o700 });
await mkdir(runtimeDir, { recursive: true });
for (const directory of [mosaicHome, join(mosaicHome, 'fleet'), agentEnvDir]) {
await chmod(directory, 0o700);
}
await writeExecutable(runtimeDir, 'mosaic', '#!/bin/sh\nexit 0\n');
return { root, mosaicHome, agentEnvDir, runtimeDir };
}
async function writeExecutable(directory: string, name: string, content: string): Promise<void> {
await mkdir(directory, { recursive: true });
await writeFile(join(directory, name), content, { mode: 0o755 });
}
const processRunner: FleetRuntimeProbeRunner = async (
command: string,
args: readonly string[],
): Promise<FleetRuntimeProbeResult> =>
new Promise((settle) => {
const child = spawn(command, [...args], { stdio: ['ignore', 'pipe', 'pipe'] });
let stdout = '';
let stderr = '';
child.stdout.setEncoding('utf8');
child.stderr.setEncoding('utf8');
child.stdout.on('data', (chunk: string): void => {
stdout += chunk;
});
child.stderr.on('data', (chunk: string): void => {
stderr += chunk;
});
child.on('error', (error: Error): void => {
settle({ stdout, stderr: `${stderr}${error.message}`, exitCode: 127 });
});
child.on('close', (code: number | null): void => {
settle({ stdout, stderr, exitCode: code ?? 1 });
});
});
describe('fleet runtime preflight', (): void => {
it('executes one distinct pane runtime and aggregates every requesting roster row', async (): Promise<void> => {
const fixture = await fleetHome();
await writeExecutable(fixture.runtimeDir, 'pi', '#!/bin/sh\nexit 0\n');
let probes = 0;
const inspection = await inspectFleetRuntimeAvailability({
mosaicHome: fixture.mosaicHome,
agentEnvDir: fixture.agentEnvDir,
helperPath,
agents: [
{ name: 'coder1', runtime: 'pi' },
{ name: 'coder0', runtime: 'pi' },
],
runner: async (command, args): Promise<FleetRuntimeProbeResult> => {
probes += 1;
return processRunner(command, args);
},
});
expect(probes).toBe(2);
expect(inspection.fleetCliExecutable).toEqual([
expect.objectContaining({
check: 'fleet-cli-executable',
status: 'ok',
requestedBy: ['coder0', 'coder1'],
binaryPath: join(fixture.runtimeDir, 'mosaic'),
dependency: '/bin/sh',
}),
]);
expect(inspection.fleetRuntimeAvailability).toEqual([
expect.objectContaining({
check: 'fleet-runtime-available',
runtime: 'pi',
status: 'ok',
requestedBy: ['coder0', 'coder1'],
binaryPath: join(fixture.runtimeDir, 'pi'),
dependency: '/bin/sh',
installCommand: 'npm install -g @earendil-works/pi-coding-agent',
}),
]);
expect(inspection.fleetRuntimeAvailability[0]?.panePath).toContain(fixture.runtimeDir);
});
it('returns an actionable non-green check when the pane PATH lacks the runtime', async (): Promise<void> => {
const fixture = await fleetHome();
const inspection = await inspectFleetRuntimeAvailability({
mosaicHome: fixture.mosaicHome,
agentEnvDir: fixture.agentEnvDir,
helperPath,
agents: [{ name: 'coder0', runtime: 'pi' }],
runner: processRunner,
});
expect(inspection.fleetCliExecutable[0]?.status).toBe('ok');
expect(inspection.fleetRuntimeAvailability).toEqual([
expect.objectContaining({
check: 'fleet-runtime-available',
runtime: 'pi',
status: 'missing',
requestedBy: ['coder0'],
installCommand: 'npm install -g @earendil-works/pi-coding-agent',
}),
]);
expect(inspection.fleetRuntimeAvailability[0]?.panePath).not.toContain(
process.env['PATH'] ?? 'operator-path-absent',
);
});
it('executes the side-effect-free Node version probe for Node-shebang commands', async (): Promise<void> => {
const fixture = await fleetHome();
const nodeScript = '#!/usr/bin/env node\nconsole.log("should-not-run");\n';
await writeExecutable(fixture.runtimeDir, 'mosaic', nodeScript);
await writeExecutable(fixture.runtimeDir, 'pi', nodeScript);
await writeExecutable(
fixture.runtimeDir,
'node',
'#!/bin/sh\n[ "$1" = --version ] || exit 9\nprintf "v-fixture-node\\n"\n',
);
const inspection = await inspectFleetRuntimeAvailability({
mosaicHome: fixture.mosaicHome,
agentEnvDir: fixture.agentEnvDir,
helperPath,
agents: [{ name: 'coder0', runtime: 'pi' }],
runner: processRunner,
});
for (const check of [
...inspection.fleetCliExecutable,
...inspection.fleetRuntimeAvailability,
]) {
expect(check).toMatchObject({
status: 'ok',
dependency: 'node',
probeCommand: 'node --version',
probeExit: 0,
probeOutput: 'v-fixture-node',
});
}
});
it('reddens when resolved Node-shebang commands cannot execute without pane Node', async (): Promise<void> => {
const fixture = await fleetHome();
const isolatedSystemPath = join(fixture.root, 'system-bin');
await mkdir(isolatedSystemPath, { recursive: true });
await writeFile(
join(fixture.root, '.npmrc'),
`prefix=${join(fixture.root, 'absent-prefix')}\n`,
);
const nodeScript = '#!/usr/bin/env node\nconsole.log("should-not-run");\n';
await writeExecutable(fixture.runtimeDir, 'mosaic', nodeScript);
await writeExecutable(fixture.runtimeDir, 'pi', nodeScript);
const inspection = await inspectFleetRuntimeAvailability({
mosaicHome: fixture.mosaicHome,
agentEnvDir: fixture.agentEnvDir,
helperPath,
agents: [{ name: 'coder0', runtime: 'pi' }],
runner: processRunner,
systemPath: isolatedSystemPath,
});
expect(inspection.fleetCliExecutable).toEqual([
expect.objectContaining({
check: 'fleet-cli-executable',
status: 'unexecutable',
binaryPath: join(fixture.runtimeDir, 'mosaic'),
dependency: 'node',
probeCommand: 'node --version',
}),
]);
expect(inspection.fleetRuntimeAvailability).toEqual([
expect.objectContaining({
check: 'fleet-runtime-available',
runtime: 'pi',
status: 'unexecutable',
binaryPath: join(fixture.runtimeDir, 'pi'),
dependency: 'node',
probeCommand: 'node --version',
}),
]);
expect(inspection.fleetCliExecutable[0]?.probeOutput).toBe(
'shebang command is not on the pane PATH',
);
expect(inspection.fleetCliExecutable[0]?.panePath).not.toContain('/usr/bin');
});
it('keeps distinct effective local runtime-bin paths as distinct checks', async (): Promise<void> => {
const fixture = await fleetHome();
const firstBin = join(fixture.root, 'first-bin');
const secondBin = join(fixture.root, 'second-bin');
for (const override of [
{ agent: 'coder0', runtimeBin: firstBin },
{ agent: 'coder1', runtimeBin: secondBin },
]) {
await writeExecutable(override.runtimeBin, 'pi', '#!/bin/sh\nexit 0\n');
await writeFile(
join(fixture.agentEnvDir, `${override.agent}.env.local`),
`MOSAIC_RUNTIME_BIN=${override.runtimeBin}\n`,
{ mode: 0o600 },
);
}
const inspection = await inspectFleetRuntimeAvailability({
mosaicHome: fixture.mosaicHome,
agentEnvDir: fixture.agentEnvDir,
helperPath,
agents: [
{ name: 'coder0', runtime: 'pi' },
{ name: 'coder1', runtime: 'pi' },
],
runner: processRunner,
});
expect(inspection.fleetCliExecutable).toHaveLength(2);
expect(inspection.fleetRuntimeAvailability).toHaveLength(2);
expect(inspection.fleetRuntimeAvailability.map((check) => check.requestedBy)).toEqual([
['coder0'],
['coder1'],
]);
expect(inspection.fleetRuntimeAvailability.map((check) => check.binaryPath)).toEqual([
join(firstBin, 'pi'),
join(secondBin, 'pi'),
]);
});
it('reports each distinct roster runtime with its exact install command', async (): Promise<void> => {
const fixture = await fleetHome();
let probes = 0;
const inspection = await inspectFleetRuntimeAvailability({
mosaicHome: fixture.mosaicHome,
agentEnvDir: fixture.agentEnvDir,
helperPath,
agents: [
{ name: 'pi-seat', runtime: 'pi' },
{ name: 'claude-seat', runtime: 'claude' },
{ name: 'codex-seat', runtime: 'codex' },
{ name: 'opencode-seat', runtime: 'opencode' },
],
runner: async (): Promise<FleetRuntimeProbeResult> => {
probes += 1;
return {
stdout:
'pane_path\u0000/fixture/bin:/usr/bin:/bin\u0000status\u0000missing\u0000' +
'binary_path\u0000\u0000probe_exit\u0000\u0000probe_output\u0000\u0000',
stderr: '',
exitCode: 69,
};
},
});
expect(probes).toBe(5);
expect(
inspection.fleetRuntimeAvailability.map((check) => ({
runtime: check.runtime,
installCommand: check.installCommand,
})),
).toEqual([
{
runtime: 'claude',
installCommand: 'curl -fsSL https://claude.ai/install.sh | bash',
},
{
runtime: 'codex',
installCommand: 'npm install -g @openai/codex',
},
{
runtime: 'opencode',
installCommand: 'npm install -g opencode-ai',
},
{
runtime: 'pi',
installCommand: 'npm install -g @earendil-works/pi-coding-agent',
},
]);
});
it('fails closed when the shared helper returns malformed evidence', async (): Promise<void> => {
const fixture = await fleetHome();
await expect(
inspectFleetRuntimeAvailability({
mosaicHome: fixture.mosaicHome,
agentEnvDir: fixture.agentEnvDir,
helperPath,
agents: [{ name: 'coder0', runtime: 'pi' }],
runner: async (): Promise<FleetRuntimeProbeResult> => ({
stdout: 'not-a-field-protocol',
stderr: '',
exitCode: 0,
}),
}),
).rejects.toThrow('malformed field output');
});
});
@@ -1,362 +0,0 @@
import { homedir } from 'node:os';
import { getInstallInstructions } from '../runtime/detector.js';
import type { RuntimeName } from '../types.js';
import { compareCodePoints } from './deterministic-order.js';
import {
GENERATED_AGENT_ENV_SUPPORTED_RUNTIMES,
readAgentLocalEnvironment,
} from './generated-env-boundary.js';
const RUNTIME_SET = new Set<string>(GENERATED_AGENT_ENV_SUPPORTED_RUNTIMES);
export interface FleetRuntimeRequestedAgent {
readonly name: string;
readonly runtime: string;
/** Planned effective local override, when provisioning has already prepared it. */
readonly runtimeBin?: string;
}
export interface FleetRuntimeProbeResult {
readonly stdout: string;
readonly stderr: string;
readonly exitCode: number;
}
export type FleetRuntimeProbeRunner = (
command: string,
args: readonly string[],
) => Promise<FleetRuntimeProbeResult>;
export interface FleetRuntimePreflightOptions {
readonly mosaicHome: string;
readonly agentEnvDir: string;
readonly helperPath: string;
readonly agents: readonly FleetRuntimeRequestedAgent[];
readonly runner: FleetRuntimeProbeRunner;
/** Test-only system suffix; production and the launcher use the helper default. */
readonly systemPath?: string;
}
export type FleetExecutableStatus = 'ok' | 'missing' | 'unexecutable';
interface FleetExecutableEvidence {
readonly status: FleetExecutableStatus;
readonly panePath: string;
readonly binaryPath?: string;
readonly dependency?: string;
readonly probeCommand?: string;
readonly probeExit?: number;
readonly probeOutput?: string;
}
export interface FleetCliExecutableCheck extends FleetExecutableEvidence {
readonly check: 'fleet-cli-executable';
readonly binary: 'mosaic';
readonly requestedBy: readonly string[];
}
export interface FleetRuntimeCheck extends FleetExecutableEvidence {
readonly check: 'fleet-runtime-available';
readonly runtime: RuntimeName;
readonly requestedBy: readonly string[];
readonly installCommand: string;
}
export type FleetRuntimePreflightCheck = FleetCliExecutableCheck | FleetRuntimeCheck;
export interface FleetRuntimeInspection {
readonly fleetCliExecutable: readonly FleetCliExecutableCheck[];
readonly fleetRuntimeAvailability: readonly FleetRuntimeCheck[];
}
interface EffectiveAgent {
readonly name: string;
readonly runtime: RuntimeName;
readonly runtimeBin: string;
}
interface PaneProbeGroup {
readonly runtimeBin: string;
readonly requestedBy: string[];
}
interface RuntimeProbeGroup extends PaneProbeGroup {
readonly runtime: RuntimeName;
}
interface BinaryProbeRequest {
readonly binary: string;
readonly runtimeBin: string;
}
export class FleetRuntimePreflightError extends Error {
readonly checks: readonly FleetRuntimePreflightCheck[];
constructor(checks: readonly FleetRuntimePreflightCheck[]) {
super(formatFleetRuntimePreflightError(checks));
this.name = FleetRuntimePreflightError.name;
this.checks = checks;
}
}
export class FleetRuntimeProbeError extends Error {
constructor(message: string) {
super(message);
this.name = FleetRuntimeProbeError.name;
}
}
/**
* Proves the fleet CLI and every distinct runtime/effective-bin pair resolve
* with an executable shebang interpreter through the eventual pane PATH. The
* helper runs under the unit's clean launcher environment, so operator PATH can
* neither create a false green nor provide a hidden interpreter.
*/
export async function inspectFleetRuntimeAvailability(
options: FleetRuntimePreflightOptions,
): Promise<FleetRuntimeInspection> {
const agents = await resolveEffectiveAgents(options);
const paneGroups = groupPaneRequests(agents);
const runtimeGroups = groupRuntimeRequests(agents);
const fleetCliExecutable: FleetCliExecutableCheck[] = [];
for (const group of paneGroups) {
const evidence = await probeBinary(options, {
binary: 'mosaic',
runtimeBin: group.runtimeBin,
});
fleetCliExecutable.push({
check: 'fleet-cli-executable',
binary: 'mosaic',
requestedBy: sortedRequestedBy(group.requestedBy),
...evidence,
});
}
const fleetRuntimeAvailability: FleetRuntimeCheck[] = [];
for (const group of runtimeGroups) {
const evidence = await probeBinary(options, {
binary: group.runtime,
runtimeBin: group.runtimeBin,
});
fleetRuntimeAvailability.push({
check: 'fleet-runtime-available',
runtime: group.runtime,
requestedBy: sortedRequestedBy(group.requestedBy),
installCommand: getInstallInstructions(group.runtime),
...evidence,
});
}
return Object.freeze({
fleetCliExecutable: Object.freeze(fleetCliExecutable),
fleetRuntimeAvailability: Object.freeze(fleetRuntimeAvailability),
});
}
export function assertFleetRuntimeAvailability(inspection: FleetRuntimeInspection): void {
const checks: FleetRuntimePreflightCheck[] = [
...inspection.fleetCliExecutable,
...inspection.fleetRuntimeAvailability,
];
const failures = checks.filter(
(check: FleetRuntimePreflightCheck): boolean => check.status !== 'ok',
);
if (failures.length > 0) throw new FleetRuntimePreflightError(failures);
}
export function formatFleetRuntimePreflightError(
checks: readonly FleetRuntimePreflightCheck[],
): string {
const lines = ['Fleet runtime preflight failed:'];
for (const check of checks) {
const dependency = check.dependency === undefined ? '' : ` dependency=${check.dependency}`;
const probe = check.probeCommand === undefined ? '' : ` dependency_probe=${check.probeCommand}`;
const execution =
check.status === 'unexecutable'
? ` probe_exit=${check.probeExit?.toString() ?? 'not-run'} ` +
`probe_output=${JSON.stringify(check.probeOutput ?? '')}`
: '';
if (check.check === 'fleet-cli-executable') {
lines.push(
`check=${check.check} binary=${check.binary} ` +
`requested_by=${check.requestedBy.join(',')} pane_path=${check.panePath}` +
`${dependency}${probe}${execution} ` +
'action=repair the Mosaic installation until its pane dependencies resolve',
);
continue;
}
lines.push(
`check=${check.check} runtime=${check.runtime} ` +
`requested_by=${check.requestedBy.join(',')} pane_path=${check.panePath}` +
`${dependency}${probe}${execution} install_command=${check.installCommand}`,
);
}
return lines.join('\n');
}
async function resolveEffectiveAgents(
options: FleetRuntimePreflightOptions,
): Promise<readonly EffectiveAgent[]> {
const agents: EffectiveAgent[] = [];
for (const agent of options.agents) {
if (!isRuntimeName(agent.runtime)) {
throw new FleetRuntimeProbeError(`Unsupported fleet runtime: ${agent.runtime}`);
}
const runtimeBin =
agent.runtimeBin ??
(
await readAgentLocalEnvironment({
mosaicHome: options.mosaicHome,
agentEnvDir: options.agentEnvDir,
agentName: agent.name,
})
)['MOSAIC_RUNTIME_BIN'] ??
'';
agents.push({ name: agent.name, runtime: agent.runtime, runtimeBin });
}
return agents;
}
function groupPaneRequests(agents: readonly EffectiveAgent[]): readonly PaneProbeGroup[] {
const groups = new Map<string, PaneProbeGroup>();
for (const agent of agents) {
const current = groups.get(agent.runtimeBin);
if (current === undefined) {
groups.set(agent.runtimeBin, { runtimeBin: agent.runtimeBin, requestedBy: [agent.name] });
} else {
current.requestedBy.push(agent.name);
}
}
return [...groups.values()].sort((left, right): number =>
compareCodePoints(left.runtimeBin, right.runtimeBin),
);
}
function groupRuntimeRequests(agents: readonly EffectiveAgent[]): readonly RuntimeProbeGroup[] {
const groups = new Map<string, RuntimeProbeGroup>();
for (const agent of agents) {
const key = JSON.stringify([agent.runtime, agent.runtimeBin]);
const current = groups.get(key);
if (current === undefined) {
groups.set(key, {
runtime: agent.runtime,
runtimeBin: agent.runtimeBin,
requestedBy: [agent.name],
});
} else {
current.requestedBy.push(agent.name);
}
}
return [...groups.values()].sort((left, right): number =>
compareCodePoints(
`${left.runtime}\u0000${left.runtimeBin}`,
`${right.runtime}\u0000${right.runtimeBin}`,
),
);
}
async function probeBinary(
options: FleetRuntimePreflightOptions,
probe: BinaryProbeRequest,
): Promise<FleetExecutableEvidence> {
const args = [
'-i',
`HOME=${process.env['HOME'] ?? homedir()}`,
'PATH=/usr/bin:/bin',
`MOSAIC_HOME=${options.mosaicHome}`,
'/bin/bash',
'--noprofile',
'--norc',
options.helperPath,
'--mosaic-home',
options.mosaicHome,
'--binary',
probe.binary,
'--check-executable',
];
if (probe.runtimeBin !== '') args.push('--runtime-bin', probe.runtimeBin);
if (options.systemPath !== undefined) args.push('--system-path', options.systemPath);
const result = await options.runner('/usr/bin/env', args);
const fields = parseNulFields(result.stdout);
const panePath = requiredField(fields, 'pane_path');
const status = requiredField(fields, 'status');
if (result.exitCode === 0 && status === 'present') {
return executableEvidence('ok', panePath, fields);
}
if (result.exitCode === 69 && status === 'missing') {
return { status: 'missing', panePath };
}
if (result.exitCode === 70 && status === 'unexecutable') {
return executableEvidence('unexecutable', panePath, fields);
}
throw new FleetRuntimeProbeError(
`Fleet executable probe failed: binary=${probe.binary} exit=${result.exitCode.toString()} ` +
`stderr=${JSON.stringify(result.stderr.trim())}`,
);
}
function executableEvidence(
status: 'ok' | 'unexecutable',
panePath: string,
fields: ReadonlyMap<string, string>,
): FleetExecutableEvidence {
const dependency = requiredField(fields, 'dependency');
const probeCommand = requiredField(fields, 'probe_command');
const probeExit = requiredField(fields, 'probe_exit');
const probeOutput = requiredField(fields, 'probe_output');
return {
status,
panePath,
binaryPath: requiredField(fields, 'binary_path'),
...(dependency === '' ? {} : { dependency }),
...(probeCommand === '' ? {} : { probeCommand }),
...(probeExit === '' ? {} : { probeExit: parseProbeExit(probeExit) }),
...(probeOutput === '' ? {} : { probeOutput }),
};
}
function sortedRequestedBy(requestedBy: readonly string[]): readonly string[] {
return Object.freeze(
[...requestedBy].sort((left: string, right: string): number => compareCodePoints(left, right)),
);
}
function parseNulFields(source: string): ReadonlyMap<string, string> {
const parts = source.split('\u0000');
if (parts.at(-1) === '') parts.pop();
if (parts.length % 2 !== 0) {
throw new FleetRuntimeProbeError('Fleet runtime probe returned malformed field output.');
}
const fields = new Map<string, string>();
for (let index = 0; index < parts.length; index += 2) {
const key = parts[index];
const value = parts[index + 1];
if (key === undefined || value === undefined || key === '' || fields.has(key)) {
throw new FleetRuntimeProbeError('Fleet runtime probe returned malformed field output.');
}
fields.set(key, value);
}
return fields;
}
function requiredField(fields: ReadonlyMap<string, string>, key: string): string {
const value = fields.get(key);
if (value === undefined) {
throw new FleetRuntimeProbeError(`Fleet runtime probe omitted ${key}.`);
}
return value;
}
function parseProbeExit(value: string): number {
const exitCode = Number(value);
if (!Number.isSafeInteger(exitCode) || exitCode < 0) {
throw new FleetRuntimeProbeError('Fleet runtime probe returned an invalid execution status.');
}
return exitCode;
}
function isRuntimeName(value: string): value is RuntimeName {
return RUNTIME_SET.has(value);
}
@@ -25,12 +25,6 @@ export interface AgentGeneratedProjectionDeletionOptions {
readonly agentName: string;
}
export interface AgentLocalEnvironmentReadOptions {
readonly mosaicHome: string;
readonly agentEnvDir: string;
readonly agentName: string;
}
export interface AgentEnvironmentProjectionResult {
readonly generatedPath: string;
readonly localPath: string;
@@ -151,23 +145,6 @@ export function parseAgentEnvironment(
return Object.freeze(values);
}
/**
* Reads one agent's optional local overrides through the same path, file-type,
* permission, key, and value boundary used by projection/launch handling.
*/
export async function readAgentLocalEnvironment(
options: AgentLocalEnvironmentReadOptions,
): Promise<Readonly<Record<string, string>>> {
if (!AGENT_NAME.test(options.agentName)) {
throw new AgentEnvBoundaryError('unsafe-agent-name', 'MOSAIC_AGENT_NAME', options.agentName);
}
await validatePrivateProjectionDirectory(options.mosaicHome, options.agentEnvDir);
const source = await readOptionalPrivateFile(
join(options.agentEnvDir, `${options.agentName}.env.local`),
);
return source === undefined ? Object.freeze({}) : parseAgentEnvironment(source, 'local');
}
/** Renders the roster-derived generated projection in a stable, complete key order. */
export function renderGeneratedAgentEnvironment(values: Readonly<Record<string, string>>): string {
const normalized = normalizeGeneratedValues(values);
+3 -3
View File
@@ -19,7 +19,7 @@ const RUNTIME_DEFS: Record<
label: 'Claude Code',
command: 'claude',
versionFlag: '--version',
installHint: 'curl -fsSL https://claude.ai/install.sh | bash',
installHint: 'npm install -g @anthropic-ai/claude-code',
},
codex: {
label: 'Codex',
@@ -31,13 +31,13 @@ const RUNTIME_DEFS: Record<
label: 'OpenCode',
command: 'opencode',
versionFlag: 'version',
installHint: 'npm install -g opencode-ai',
installHint: 'See https://opencode.ai for install instructions',
},
pi: {
label: 'Pi',
command: 'pi',
versionFlag: '--version',
installHint: 'npm install -g @earendil-works/pi-coding-agent',
installHint: 'curl -fsSL https://pi.dev/install.sh | sh',
},
};