Compare commits

..
Author SHA1 Message Date
fred 5c35a250de test(fleet): name what the pane-boundary case's binary check rides on (#1241)
ci/woodpecker/pr/ci Pipeline was successful
Review finding from scooby. This case does not use run_start, so
install_pane_binaries' symlinks land under a home its launcher never consults
(HOME is the trusted parent here). It resolves mosaic and pi through
MOSAIC_RUNTIME_BIN=$FAKE_BIN instead. Valid path, valid green — and a trap for
anyone who later drops that env var believing the symlinks cover it, which
would break the #1241 binary check rather than exercise it.

Comment only; no behavior change. Harness rc=0.

Refs #1241.
2026-08-16 00:21:13 -05:00
fred 10a1f82031 test(fleet): cover the pane-pid-unresolved branch this PR shipped (#1241)
ci/woodpecker/pr/ci Pipeline was canceled
Review finding from scooby: this PR added a failure branch the harness
structurally could not reach. The fake tmux answered `has-session` only for
`=_holder:0.0`, so every non-holder agent landed in the session-is-gone branch
no matter what — the `elif` (tmux still reports the session, no pane PID after
the retries) had zero coverage and no way to get any.

That is the same shape as the bug this PR exists to fix, one layer down: a code
path shipped green where the gate that should measure it cannot. Less severe,
because the branch fails closed at exit 69 rather than reporting success — but
"the harness can't reach it" is the sentence that precedes the next silent
regression, so it gets closed here rather than filed.

`MOSAIC_TEST_HELD_SESSIONS` lets a case name targets the shim should also
answer for. It answers them only AFTER `new-session`, and that detail is the
whole trick: the launcher asks `has-session` about the same name twice — once
at line 255 where a yes means "already running, exit 0", and once at 417 where
a yes means "the session survived". A shim answering yes to both short-circuits
at the first and never reaches the branch under test. It would have looked like
coverage while measuring the idempotency path.

Both failure modes were measured, not reasoned about:
- toggle absent (the old shim): `code=pane-did-not-survive` — the case lands on
  the wrong branch, which is exactly the unreachability being reported.
- toggle answering unconditionally: launcher exits 0 via the idempotency
  short-circuit — "launcher reported success over a session with no resolvable
  pane PID".
- toggle gated on new-session: `code=pane-pid-unresolved`, exit 69.

The case also asserts the diagnostic is not `pane-did-not-survive` and does not
mention the heartbeat, so the two pane faults cannot collapse into one message.

Gates: bash -n · launcher harness rc=0 · test-fleet-units.sh (real tmux) rc=0 ·
fleet specs 342 passed.

Refs #1241.
2026-08-16 00:19:25 -05:00
fred 61a907a12f fix(fleet): fail the agent launcher when the pane cannot survive (#1241)
ci/woodpecker/pr/ci Pipeline was successful
`mosaic fleet start` returned 0 over three dead panes. The launcher knew,
and said the wrong thing at the wrong severity to the wrong layer.

The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared
environment. When that binary is absent the pane dies in under a second,
tmux destroys the session, and the diagnostic goes with it. The launcher
then found no PANE_PID, printed a WARNING about the *heartbeat sidecar*,
and exited 0 — so systemd logged "Finished ... successfully" and
`fleet start` reported success. `fleet ps` was the only component telling
the truth.

Two changes, both in start-agent-session.sh:

1. Before any effect, resolve `mosaic` and the roster's runtime against
   PANE_PATH — the pane's own view of the path, not the launcher's.
   `mosaic yolo <runtime>` calls checkRuntime(runtime) and looks for a
   binary named exactly like the runtime, so this asks the same question
   the pane will ask a moment later, while an operator can still see the
   answer. Absent binary -> exit 69, code=missing-binary, no session
   created.

2. Replace the dead-pane WARNING+exit-0. An absent session one second
   after new-session is a runtime that died on startup, not a heartbeat
   problem -> exit 69, code=pane-did-not-survive, with the command to run
   by hand to see why. A present session with no pane PID after five
   attempts -> code=pane-pid-unresolved. Neither branch kills the
   session; destroying a possibly-live pane on a guess is worse than
   leaving it for inspection.

Exit 69 (EX_UNAVAILABLE) is deliberate: the 64s already in this file mean
the projection was bad, and here the data is fine and the host is not
ready. Callers separate the cases by `code=`, the same way fail_env's
codes share 64.

This propagates for free. `fleet start` calls runChecked() for the holder
and each agent, and runChecked throws on non-zero, so layers 4 and 5 stop
lying without a TypeScript change. Two adjacent defects are left for a
follow-up issue rather than widened into this diff: the per-agent loop
aborts on the first failure instead of attempting all and reporting an
aggregate, and runChecked's bare throw surfaces the launcher's message
under a Node unhandled-rejection stack trace because program.parse() is
synchronous.

Tests:

- test-start-agent-session.sh gains three cases: `mosaic` absent from the
  pane path, the runtime absent from the pane path, and a pane that does
  not survive. Each was verified individually red against the unmodified
  origin/next launcher.
- The two cases asserting a valid launch now supply a pane PID. Until now
  the suite's one success path was itself a dead pane the launcher
  reported as fine.
- The harness fakes `npm` so PANE_PATH stops depending on whatever the
  host has installed, and fails loudly if the host provides `mosaic` or
  `pi` in the system path, where the missing-binary cases would not be
  measurable at all.
- test-fleet-units.sh gains a `pi` shim in its runtime bin. The real-tmux
  harness named `pi` in its roster and never installed it; the new
  preflight caught it.

Refs #1241
2026-08-15 23:56:53 -05:00
6 changed files with 197 additions and 449 deletions
@@ -4,14 +4,6 @@ Documentation=https://git.mosaicstack.dev/mosaicstack/stack
Requires=mosaic-tmux-holder.service
After=mosaic-tmux-holder.service
PartOf=mosaic-tmux-holder.service
# Do not attempt a seat before its generated env exists. `install` enables this
# unit (WantedBy=default.target) but on a roster-v2 fleet the reconciler owns the
# generated env, so between `install` and the first `apply`/`regen --write` there
# is a boot window where ExecStart would run against an absent env file and the
# launcher would fail the unit. A skipped unit is the honest state for "enabled
# but not yet configured"; systemd re-evaluates the condition on every start, so
# the seat comes up on the next start once the reconciler has written env.
ConditionPathExists=%h/.config/mosaic/fleet/agents/%i.env.generated
[Service]
Type=oneshot
@@ -128,6 +128,14 @@ EOF
sleep 30
EOF
chmod 700 "$AGENT_BIN/mosaic"
# The launcher resolves the roster's runtime against PANE_PATH before it
# spawns anything (#1241), so the runtime this projection names has to be
# present here even though the fake `mosaic` above never execs it.
cat > "$AGENT_BIN/pi" <<'EOF'
#!/bin/sh
sleep 30
EOF
chmod 700 "$AGENT_BIN/pi"
server_environment_before=$(tmux -L "$TEST_SOCKET" show-environment -g | sort)
server_sessions_before=$(tmux -L "$TEST_SOCKET" list-sessions | sort)
if /usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin MOSAIC_HOME="$AGENT_HOME" \
@@ -286,6 +286,36 @@ _build_runtime_bin_prefix() {
MOSAIC_RUNTIME_BIN_PREFIX=$(_build_runtime_bin_prefix)
PANE_PATH=${MOSAIC_RUNTIME_BIN_PREFIX:+${MOSAIC_RUNTIME_BIN_PREFIX}:}/usr/local/bin:/usr/bin:/bin
# #1241. The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared
# environment. A binary missing from *that* path is a pane that dies in under a
# second, inside a session nobody is attached to, with its diagnostic scrolled
# into a pane tmux then destroys. Resolve both here, before any effect, where
# the failure is still attributable to the thing that caused it.
#
# `mosaic yolo <runtime>` runs checkRuntime(runtime) and the binary it looks for
# is named exactly like the runtime, so resolving the runtime name is the same
# question the pane will ask a moment later — asked while an operator can still
# see the answer.
_resolve_in_pane_path() {
PATH="$PANE_PATH" command -v -- "$1" 2>/dev/null
}
# Exit 69 (EX_UNAVAILABLE): the seat cannot be provided. Distinguished from the
# 64 (EX_USAGE) rejections above, which mean the projection itself was bad —
# here the data is fine and the host is not ready. Callers tell the individual
# cases apart by `code=`, the same way fail_env's many codes share exit 64.
fail_launch() {
local code="$1"
shift
echo "ERROR: agent launch aborted: code=${code} agent=${AGENT_NAME} $*" >&2
exit 69
}
for required_binary in mosaic "$MOSAIC_AGENT_RUNTIME"; do
_resolve_in_pane_path "$required_binary" >/dev/null ||
fail_launch missing-binary "'${required_binary}' is not on the pane PATH (${PANE_PATH})"
done
_ensure_claude_workdir_trusted() {
local workdir="$1"
local resolved
@@ -384,6 +414,19 @@ if [ -n "$PANE_PID" ]; then
_start_heartbeat_sidecar "$AGENT_NAME" "$PANE_PID" \
"$MOSAIC_HEARTBEAT_RUN_DIR" "$MOSAIC_HEARTBEAT_INTERVAL" || \
echo "WARNING: heartbeat sidecar could not be started for $AGENT_NAME" >&2
elif _tmux has-session -t "=${AGENT_NAME}:0.0" 2>/dev/null; then
# #1241. Session present, no pane PID after a second of retries. Whatever this
# is, it is not a seat an operator can use, so it is not a success either.
fail_launch pane-pid-unresolved \
"tmux reports the session but no pane PID after 5 attempts"
else
echo "WARNING: could not resolve pane PID for $AGENT_NAME — heartbeat sidecar not started" >&2
# #1241. This branch used to print a WARNING about the heartbeat sidecar and
# exit 0. It is not a heartbeat problem: tmux destroys a session when its pane
# command exits, so an absent session one second after new-session means the
# runtime died on startup. Reporting it as success is what let `fleet start`
# return 0 over three dead panes — the launcher knew, and said the wrong thing
# at the wrong severity to the wrong layer.
fail_launch pane-did-not-survive \
"the pane exited immediately and tmux destroyed the session;" \
"run 'mosaic yolo ${MOSAIC_AGENT_RUNTIME}' in ${MOSAIC_AGENT_WORKDIR} to see why"
fi
@@ -23,8 +23,26 @@ index=0
if [ "${args[0]:-}" = -L ]; then index=2; fi
case "${args[$index]:-}" in
has-session)
# The holder always answers. MOSAIC_TEST_HELD_SESSIONS lets a case add
# other targets that should answer too — without it there is no way to
# model "tmux still reports the session" for a non-holder agent, and the
# launcher's pane-pid-unresolved branch is unreachable from this harness.
#
# A listed target answers only AFTER new-session, because the launcher asks
# this question twice about the same name: once before launching, where a
# yes means "already running, nothing to do, exit 0", and once after, where
# a yes means "the session survived". A shim that answered yes to both
# would short-circuit at the first and never reach the branch under test —
# it would look like coverage and measure the idempotency path instead.
for argument in "${args[@]}"; do
[ "$argument" = '=_holder:0.0' ] && exit 0
case " ${MOSAIC_TEST_HELD_SESSIONS:-} " in
*" $argument "*)
if tr '\0' '\n' < "${MOSAIC_TEST_TMUX_CALLS:?}" | grep -qxF new-session; then
exit 0
fi
;;
esac
done
exit 1
;;
@@ -62,6 +80,30 @@ env -0 > "${MOSAIC_HOME:?}/fleet/pane-environment"
SHIM
chmod +x "$FAKE_BIN/mosaic"
# The runtime the rosters below name. The launcher resolves it against PANE_PATH
# before spawning (#1241), so it has to exist somewhere the pane would find it —
# not merely on the launcher's own PATH.
printf '#!/usr/bin/env bash\nexit 0\n' > "$FAKE_BIN/pi"
chmod +x "$FAKE_BIN/pi"
# PANE_PATH is derived partly from `npm config get prefix`. Left to the real npm
# it would splice whatever the host has installed into the path under test, and
# the missing-binary cases below would pass or fail by accident of the machine.
cat > "$FAKE_BIN/npm" <<'SHIM'
#!/usr/bin/env bash
printf '%s\n' "${MOSAIC_TEST_NPM_PREFIX:-/nonexistent}"
SHIM
chmod +x "$FAKE_BIN/npm"
# PANE_PATH always ends in the system path. A host that installs these there can
# not measure the missing-binary cases at all, and a green run would mean
# nothing — so say so instead of passing.
for host_binary in mosaic pi; do
if PATH=/usr/local/bin:/usr/bin:/bin command -v "$host_binary" >/dev/null 2>&1; then
fail "host provides '$host_binary' in the system path; missing-binary cases are not measurable here"
fi
done
write_generated() {
local home="$1"
local agent="$2"
@@ -81,6 +123,19 @@ MOSAIC_TMUX_SOCKET=mosaic-test
EOF
chmod 600 "$home/fleet/agents/$agent.env.generated"
mkdir -p "$home/work"
install_pane_binaries "$home"
}
# `$PANE_HOME/.npm-global/bin` is one of the prefixes the launcher folds into
# PANE_PATH, so this is the pane's own view of "installed", distinct from the
# launcher's PATH. Tests that need a binary *absent* remove it from here.
install_pane_binaries() {
local pane_home="$1"
mkdir -p "$pane_home/.npm-global/bin"
local binary
for binary in mosaic pi; do
ln -sf "$FAKE_BIN/$binary" "$pane_home/.npm-global/bin/$binary"
done
}
run_start() {
@@ -88,6 +143,7 @@ run_start() {
local agent="$2"
HOME="$home" PATH="$FAKE_BIN:$PATH" MOSAIC_TEST_TMUX_CALLS="$TMUX_CALLS" \
MOSAIC_TEST_PANE_PID="${MOSAIC_TEST_PANE_PID:-}" \
MOSAIC_TEST_HELD_SESSIONS="${MOSAIC_TEST_HELD_SESSIONS:-}" \
MOSAIC_TEST_HOME="$home" \
MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \
MOSAIC_HOME="$home" "$START" "$agent"
@@ -98,7 +154,10 @@ run_start() {
HOME_VALID="$ROOT/valid"
AGENT_VALID="coder0"
write_generated "$HOME_VALID" "$AGENT_VALID"
run_start "$HOME_VALID" "$AGENT_VALID"
# A live pane PID is part of what "valid launch" means. Until #1241 this case
# ran with none, so the suite's one success path was itself a dead pane the
# launcher reported as fine.
MOSAIC_TEST_PANE_PID=$$ run_start "$HOME_VALID" "$AGENT_VALID"
valid_args=$(tr '\0' '\n' < "$TMUX_CALLS")
echo "$valid_args" | grep -qF new-session || fail "valid generated projection did not reach tmux"
echo "$valid_args" | grep -qF 'mosaic' || fail "fixed mosaic launcher command missing"
@@ -245,6 +304,13 @@ PANE_BASH_ENV="$ROOT/pane-boundary.bash-env"
printf 'MOSAIC_RUNTIME_BIN=%s\n' "$FAKE_BIN" > \
"$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local"
chmod 600 "$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local"
# This case does not go through run_start, so its pane binaries come from
# MOSAIC_RUNTIME_BIN=$FAKE_BIN in the env.local written above — not from the
# symlinks install_pane_binaries planted under the generated home, which this
# launcher never consults because HOME here is the trusted parent. That is a
# legitimate resolution path, but it means dropping MOSAIC_RUNTIME_BIN from
# this case on the belief that the symlinks cover it would break the #1241
# binary check rather than exercise it.
LD_PRELOAD='/not/loaded/by-clean-bootstrap.so' \
BASH_ENV="$PANE_BASH_ENV" \
MOSAIC_UNTRUSTED_SENTINEL='must-not-reach-pane' \
@@ -258,6 +324,7 @@ PATH="$PANE_STALE_PATH" \
"MOSAIC_TEST_HOME=$PANE_TRUSTED_HOME" \
MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \
MOSAIC_TEST_EXECUTE_PANE=1 \
"MOSAIC_TEST_PANE_PID=$$" \
"$START" coder-pane-boundary
pane_args=$(tr '\0' '\n' < "$TMUX_CALLS")
echo "$pane_args" | grep -qxF "HOME=$PANE_TRUSTED_HOME" || \
@@ -392,6 +459,75 @@ echo "$interaction_policy_args" | grep -qF 'new-session' && \
echo "$output" | grep -qF 'operator interaction service requires runtime pi' || \
fail "interaction pinned-policy check did not follow strict parsing"
# #1241. The pane runs `mosaic yolo <runtime>` against PANE_PATH. A binary
# missing from that path is a launch failure, and it has to be named before the
# session is created — after it, the diagnostic dies with the pane.
assert_missing_pane_binary_rejected() {
local binary="$1"
local home="$ROOT/missing-$binary"
local agent="coder-missing-$binary"
write_generated "$home" "$agent"
rm -f "$home/.npm-global/bin/$binary"
: > "$TMUX_CALLS"
local output
if output=$(MOSAIC_TEST_PANE_PID=$$ run_start "$home" "$agent" 2>&1); then
fail "launch succeeded with '$binary' absent from the pane PATH"
fi
echo "$output" | grep -qF 'code=missing-binary' || fail "missing '$binary' diagnostic missing"
echo "$output" | grep -qF "'$binary'" || fail "missing-binary diagnostic did not name $binary"
if tr '\0' '\n' < "$TMUX_CALLS" | grep -qF new-session; then
fail "launcher created a session it knew would die ($binary absent)"
fi
}
assert_missing_pane_binary_rejected mosaic
assert_missing_pane_binary_rejected pi
# #1241. tmux destroys a session when its pane command exits, so no pane PID a
# second after new-session means the runtime died on startup. This used to be a
# WARNING about the heartbeat sidecar followed by exit 0 — three layers above it
# then reported a fleet that was not running.
: > "$TMUX_CALLS"
HOME_DEAD_PANE="$ROOT/dead-pane"
write_generated "$HOME_DEAD_PANE" "coder-dead-pane"
if output=$(MOSAIC_TEST_PANE_PID='' run_start "$HOME_DEAD_PANE" coder-dead-pane 2>&1); then
fail "launcher reported success over a pane that did not survive"
fi
echo "$output" | grep -qF 'code=pane-did-not-survive' || fail "dead-pane diagnostic missing"
if echo "$output" | grep -qiF 'heartbeat'; then
fail "dead pane is still being reported as a heartbeat-sidecar problem"
fi
tr '\0' '\n' < "$TMUX_CALLS" | grep -qF new-session || \
fail "dead-pane case did not reach the launch it is measuring"
# #1241, the other way a pane fails. Above, tmux destroyed the session and
# has-session said so. Here the session is still there and no PID comes back
# after the retries — a different fault (the pane is alive but unusable, or
# tmux is answering inconsistently) that an operator has to be told apart from
# a runtime that died on startup.
#
# This case exists because the branch that handles it shipped with nothing able
# to reach it: the shim answered has-session only for the holder, so every
# non-holder agent landed in the session-is-gone branch no matter what. A
# defensive branch nothing exercises is the same shape as the bug this whole
# change is about, one layer down.
: > "$TMUX_CALLS"
HOME_NO_PID="$ROOT/pane-no-pid"
write_generated "$HOME_NO_PID" "coder-no-pid"
if output=$(MOSAIC_TEST_PANE_PID='' MOSAIC_TEST_HELD_SESSIONS='=coder-no-pid:0.0' \
run_start "$HOME_NO_PID" coder-no-pid 2>&1); then
fail "launcher reported success over a session with no resolvable pane PID"
fi
echo "$output" | grep -qF 'code=pane-pid-unresolved' || \
fail "session-present/no-PID was not reported as pane-pid-unresolved: $output"
if echo "$output" | grep -qF 'code=pane-did-not-survive'; then
fail "a session tmux still reports was diagnosed as a destroyed session"
fi
if echo "$output" | grep -qiF 'heartbeat'; then
fail "an unresolvable pane PID is still being reported as a heartbeat-sidecar problem"
fi
# Exact stop derives the socket exclusively from the validated generated
# projection and ignores an ambient socket supplied by the caller.
: > "$TMUX_CALLS"
@@ -1,323 +0,0 @@
import { execFile } from 'node:child_process';
import { mkdir, mkdtemp, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { Command } from 'commander';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { registerFleetCommand, type CommandResult, type CommandRunner } from './fleet.js';
/**
* #1237: the v1-only commands (`ps`, `install`, `install-systemd`, `add`,
* `remove`) rejected a roster-v2 fleet outright, so a greenfield v2 box could
* never get its units placed. These tests pin the three behaviours that fix
* gives it, and the two it deliberately does NOT give it.
*
* The load-bearing negative is that `install` on v2 writes no generated env:
* the reconciler owns that file through projectRosterV2AgentGeneratedEnv, and a
* second writer here — necessarily through the v1 mapping — is exactly the
* drift the #791 single-SSOT invariant exists to prevent.
*/
const rosterV2 = `
version: 2
generation: 4
transport: tmux
tmux:
socket_name: mosaic-fleet
holder_session: _holder
defaults:
working_directory: /srv/mosaic
runtime: pi
runtimes:
pi:
reset_command: /new
agents:
- name: coder0
alias: Coder 0
class: code
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: high
tool_policy: code
working_directory: /srv/mosaic
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
- name: coder1
alias: Coder 1
class: code
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: medium
tool_policy: code
working_directory: /srv/other
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
`;
let tempHome: string | undefined;
const savedHome = process.env.HOME;
const savedMosaicHome = process.env.MOSAIC_HOME;
afterEach(async (): Promise<void> => {
vi.restoreAllMocks();
process.exitCode = undefined;
if (savedHome === undefined) delete process.env.HOME;
else process.env.HOME = savedHome;
if (savedMosaicHome === undefined) delete process.env.MOSAIC_HOME;
else process.env.MOSAIC_HOME = savedMosaicHome;
if (tempHome) await rm(tempHome, { recursive: true, force: true });
tempHome = undefined;
});
/**
* A HOME with a roster-v2 fleet and nothing else — the greenfield shape, before
* anything has been installed, applied or started.
*/
async function v2Home(): Promise<string> {
tempHome = await mkdtemp(join(tmpdir(), 'mosaic-fleet-v2-dispatch-'));
process.env.HOME = tempHome;
delete process.env.MOSAIC_HOME;
const mosaicHome = join(tempHome, '.config', 'mosaic');
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
await mkdir(join(mosaicHome, directory), { recursive: true, mode: 0o700 });
}
await writeFile(join(mosaicHome, 'fleet', 'roster.yaml'), rosterV2, { mode: 0o600 });
await writeFile(join(mosaicHome, 'fleet', 'roles', 'code.md'), '`class: code`\n\n# code\n', {
mode: 0o600,
});
return mosaicHome;
}
/**
* Stands in for a box where nothing is running: every systemctl and tmux probe
* fails the way it does before the holder has ever started. `ps` must survive
* this — it is the command an operator reaches for to find out *why* there is
* no seat, so it has to report the emptiness rather than fail on it.
*/
const greenfieldRunner: CommandRunner = async (command): Promise<CommandResult> => {
if (command === 'tmux') {
return { stdout: '', stderr: 'no server running on /tmp/tmux-1000/mosaic-fleet', exitCode: 1 };
}
return { stdout: '', stderr: '', exitCode: 1 };
};
function program(runner: CommandRunner = greenfieldRunner): Command {
const result = new Command();
result.exitOverride();
registerFleetCommand(result, { runner, frameworkRoot: resolve(process.cwd(), 'framework') });
return result;
}
function capture(): string[] {
const lines: string[] = [];
vi.spyOn(console, 'log').mockImplementation((value: string): void => {
lines.push(value);
});
return lines;
}
async function exists(path: string): Promise<boolean> {
try {
await stat(path);
return true;
} catch {
return false;
}
}
describe('mosaic fleet ps — roster v2', (): void => {
it('lists every v2 agent on a greenfield box with nothing running, and does not throw', async (): Promise<void> => {
await v2Home();
const lines = capture();
await expect(
program().parseAsync(['node', 'mosaic', 'fleet', 'ps', '--json']),
).resolves.toBeDefined();
const rows = JSON.parse(lines.join('\n')) as {
name: string;
runtime: string;
alias?: string;
paneAlive: boolean;
source: string;
}[];
expect(rows.map((row) => row.name).sort()).toEqual(['coder0', 'coder1']);
// The v2 roster's per-agent fields must survive the read model, not be
// flattened into defaults.
expect(rows.every((row) => row.runtime === 'pi')).toBe(true);
expect(rows.find((row) => row.name === 'coder0')?.alias).toBe('Coder 0');
// Nothing is running, and that is a report, not an error.
expect(rows.every((row) => row.paneAlive === false)).toBe(true);
expect(rows.every((row) => row.source === 'roster')).toBe(true);
expect(process.exitCode ?? 0).toBe(0);
});
});
describe('mosaic fleet install — roster v2', (): void => {
it('places the tool files and unit templates', async (): Promise<void> => {
const mosaicHome = await v2Home();
capture();
await expect(
program().parseAsync(['node', 'mosaic', 'fleet', 'install', '--no-enable']),
).resolves.toBeDefined();
// Units live in the systemd user dir, not under the Mosaic home.
const systemdUserDir = join(tempHome!, '.config', 'systemd', 'user');
for (const unit of [
'mosaic-tmux-holder.service',
'[email protected]',
'[email protected]',
]) {
expect(await exists(join(systemdUserDir, unit))).toBe(true);
}
const launcher = join(mosaicHome, 'tools', 'fleet', 'start-agent-session.sh');
expect(await exists(launcher)).toBe(true);
expect((await stat(launcher)).mode & 0o777).toBe(0o755);
});
it('writes NO generated env — that file belongs to the reconciler (#791)', async (): Promise<void> => {
const mosaicHome = await v2Home();
capture();
await program().parseAsync(['node', 'mosaic', 'fleet', 'install', '--no-enable']);
const agentDir = join(mosaicHome, 'fleet', 'agents');
expect(await readdir(agentDir)).toEqual([]);
});
it('tells the operator which command does own the env', async (): Promise<void> => {
await v2Home();
const lines = capture();
await program().parseAsync(['node', 'mosaic', 'fleet', 'install', '--no-enable']);
expect(lines.join('\n')).toContain('mosaic fleet apply');
});
});
describe('[email protected]', (): void => {
const unitPath = resolve(process.cwd(), 'framework', 'systemd', 'user', '[email protected]');
/** The single `ConditionPathExists=` value declared by the unit template. */
async function conditionPath(): Promise<string> {
const unit = await readFile(unitPath, 'utf8');
const matches = unit.match(/^ConditionPathExists=(.+)$/gm) ?? [];
expect(matches).toHaveLength(1);
return matches[0]!.slice('ConditionPathExists='.length).trim();
}
it('will not attempt a seat before the reconciler has written its env', async (): Promise<void> => {
// The pairing that makes "install writes no env" safe: install enables the
// unit (WantedBy=default.target) but does not start it, so without this
// condition a reboot between `install` and the first `apply` would run
// ExecStart against an absent env file and fail every seat unit.
expect(await conditionPath()).toBe('%h/.config/mosaic/fleet/agents/%i.env.generated');
});
/**
* The two halves of the guard's *effect*, which no assertion on the literal
* string can cover on its own.
*
* Measured end to end on a real box (canary, 2026-08-16) rather than inferred:
* with the condition, `systemctl --user start mosaic-agent@<name>` on an agent
* with no generated env returns rc=0, `Result=success`, `ConditionResult=no`,
* and journals "skipped, unmet condition check". With the condition removed by
* drop-in and nothing else changed, the same start returns rc=1,
* `Result=exit-code`, `ExecMainStatus=64`, and the unit enters `failed`.
*
* systemd is not available in this suite, so these two tests pin the parts
* that can drift in code: the condition naming a *different* file than the one
* the fleet actually writes, and the launcher quietly becoming tolerant of an
* absent env — either of which turns the condition into decoration while the
* literal-string assertion above still passes.
*/
it('guards exactly the file the fleet writes, so the two cannot drift apart', async (): Promise<void> => {
const mosaicHome = await v2Home();
const rendered = (await conditionPath()).replace('%h', tempHome!).replace('%i', 'coder0');
// The path an installed fleet actually places for this agent.
expect(rendered).toBe(join(mosaicHome, 'fleet', 'agents', 'coder0.env.generated'));
});
it('guards a real failure — the launcher rejects an absent generated env', async (): Promise<void> => {
await v2Home();
await program().parseAsync(['node', 'mosaic', 'fleet', 'install', '--no-enable']);
// Exactly what ExecStart runs, against the state the condition exists to
// catch: unit enabled, reconciler has not written env yet.
const launched = await new Promise<{ code: number | null; stderr: string }>((settle) => {
const child = execFile(
'/bin/bash',
[
'--noprofile',
'--norc',
join(tempHome!, '.config', 'mosaic', 'tools', 'fleet', 'start-agent-session.sh'),
'coder0',
],
{ env: { HOME: tempHome!, MOSAIC_AGENT_NAME: 'coder0', PATH: '/usr/bin:/bin' } },
(_error, _stdout, stderr) => {
settle({ code: child.exitCode, stderr });
},
);
});
expect(launched.code).not.toBe(0);
expect(launched.stderr).toContain('missing-file');
});
});
describe('mosaic fleet add / remove — roster v2', (): void => {
it('add refuses, and names the two-step v2 sequence instead of inventing defaults', async (): Promise<void> => {
await v2Home();
await expect(
program().parseAsync([
'node',
'mosaic',
'fleet',
'add',
'coder2',
'--runtime',
'pi',
'--class',
'code',
]),
).rejects.toThrow(/mosaic fleet create[\s\S]*mosaic fleet apply/);
});
it('remove refuses, and names delete plus apply', async (): Promise<void> => {
await v2Home();
await expect(
program().parseAsync(['node', 'mosaic', 'fleet', 'remove', 'coder1']),
).rejects.toThrow(/mosaic fleet delete coder1[\s\S]*mosaic fleet apply/);
});
// Note: this one passes on the unmodified tree too — there `remove` throws in
// the v1 parser, before it can touch anything. It is a regression guard on the
// ordering of the new guard clause, not evidence that the fix works.
it('refuses BEFORE mutating the roster', async (): Promise<void> => {
const mosaicHome = await v2Home();
const rosterPath = join(mosaicHome, 'fleet', 'roster.yaml');
const before = await readFile(rosterPath, 'utf8');
await expect(
program().parseAsync(['node', 'mosaic', 'fleet', 'remove', 'coder1']),
).rejects.toThrow();
expect(await readFile(rosterPath, 'utf8')).toBe(before);
});
});
+8 -116
View File
@@ -34,7 +34,6 @@ export {
resolveInstalledFleetRosterPath,
} from '../fleet/fleet-roster-v1.js';
export type { FleetAgent, FleetRoster } from '../fleet/fleet-roster-v1.js';
import { parseRosterV2 } from '../fleet/roster-v2.js';
import {
registerFleetAgentCrudCommands,
type FleetAgentCrudCommandDeps,
@@ -821,7 +820,7 @@ export function buildEnableLingerCommand(user: string): string[] {
*/
export async function enableFleetUnits(
runner: CommandRunner,
roster: { readonly agents: readonly { readonly name: string }[] },
roster: FleetRoster,
opts: { enable?: boolean },
): Promise<void> {
if (opts.enable === false) {
@@ -1528,8 +1527,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.option('--no-enable', 'Skip enabling units for boot-survival')
.action(async (opts: { enable?: boolean }) => {
await installFleet(cmd, frameworkRoot);
// Unit enablement needs agent names only, so it reads either version.
const roster = await loadRosterReadModel(cmd);
const roster = await loadRosterForCommand(cmd);
await enableFleetUnits(runner, roster, opts);
});
@@ -1539,8 +1537,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.option('--no-enable', 'Skip enabling units for boot-survival')
.action(async (opts: { enable?: boolean }) => {
await installFleet(cmd, frameworkRoot);
// Unit enablement needs agent names only, so it reads either version.
const roster = await loadRosterReadModel(cmd);
const roster = await loadRosterForCommand(cmd);
await enableFleetUnits(runner, roster, opts);
});
@@ -1691,9 +1688,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.action(async (opts: { json?: boolean }) => {
const commandOpts = cmd.opts<{ mosaicHome: string; roster?: string }>();
const activePaths = resolveFleetPaths(commandOpts.mosaicHome);
// ps only reads, so it takes the version-agnostic read model rather than
// the v1 parser, which rejects a v2 roster outright.
const roster = await loadRosterReadModel(cmd);
const roster = await loadRosterForCommand(cmd);
const { tenant_id, host } = getDefaultTenantAndHost();
const nowMs = Date.now();
@@ -1913,16 +1908,6 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
start: boolean;
},
) => {
if (await usesRosterV2ControlPlane(cmd)) {
// command.error, not a bare throw: this is operator guidance, and a
// bare throw reaches the top level uncaught and prints it under a Node
// stack trace. Measured on canary — the message is the whole point of
// the refusal, so it has to arrive readable.
cmd.error(rosterV2MutationGuidance('add', 'create', name), {
code: 'fleet.roster-v2',
exitCode: 1,
});
}
if (!VALID_FLEET_RUNTIMES.includes(opts.runtime)) {
throw new Error(
`Invalid runtime "${opts.runtime}". Valid runtimes: ${VALID_FLEET_RUNTIMES.join(', ')}.`,
@@ -1988,12 +1973,6 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.description('Remove an agent from the fleet roster')
.option('--keep-files', 'Skip deleting env and heartbeat files')
.action(async (name: string, opts: { keepFiles?: boolean }) => {
if (await usesRosterV2ControlPlane(cmd)) {
cmd.error(rosterV2MutationGuidance('remove', 'delete', name), {
code: 'fleet.roster-v2',
exitCode: 1,
});
}
const commandOpts = cmd.opts<{ mosaicHome: string; roster?: string }>();
const activePaths = resolveFleetPaths(commandOpts.mosaicHome);
const rosterPath = await resolveRosterPath(commandOpts.mosaicHome, commandOpts.roster);
@@ -2352,9 +2331,7 @@ export function registerFleetAgentCommands(
async function installFleet(cmd: Command, frameworkRoot: string): Promise<void> {
const activePaths = resolveFleetPaths(cmd.opts<{ mosaicHome: string }>().mosaicHome);
assertDefaultMosaicHomeForSystemd(activePaths.mosaicHome);
// Read model first: every file this function places is roster-independent, and
// the v1 parser would reject a v2 roster before any of them were written.
const roster = await loadRosterReadModel(cmd);
const roster = await loadRosterForCommand(cmd);
await ensureFleetHolderIdentity(activePaths.mosaicHome);
await mkdir(activePaths.fleetToolsDir, { recursive: true });
await mkdir(activePaths.tmuxToolsDir, { recursive: true });
@@ -2414,30 +2391,16 @@ async function installFleet(cmd: Command, frameworkRoot: string): Promise<void>
join(activePaths.systemdUserDir, '[email protected]'),
);
// On roster v2 the reconciler owns the generated env: `apply` writes it and
// `regen` rebuilds it, both from projectRosterV2AgentGeneratedEnv. Writing it
// here too — necessarily through the v1 mapping — would be the third writer of
// one file and would break the #791 single-SSOT invariant. So v2 gets the tool
// files and the units, and nothing else.
if (roster.version === 2) {
console.log(
`Installed fleet tools and systemd units for ${roster.agents.length} agent(s). ` +
`Generated env is owned by the reconciler on roster v2 — run: mosaic fleet apply --expected-generation <n>`,
);
return;
}
const v1Roster = await loadRosterForCommand(cmd);
for (const agent of v1Roster.agents) {
for (const agent of roster.agents) {
await writeAgentEnvironmentProjection({
mosaicHome: activePaths.mosaicHome,
agentEnvDir: activePaths.agentEnvDir,
agentName: agent.name,
generated: generateAgentEnvValues(v1Roster, agent),
generated: generateAgentEnvValues(roster, agent),
});
}
console.log(`Installed fleet files for ${v1Roster.agents.length} agent(s).`);
console.log(`Installed fleet files for ${roster.agents.length} agent(s).`);
}
async function loadRosterForCommand(cmd: Command): Promise<FleetRoster> {
@@ -2464,77 +2427,6 @@ async function usesRosterV2ControlPlane(cmd: Command): Promise<boolean> {
);
}
/**
* `add`/`remove` and `create`/`delete` are not two spellings of one operation.
* The v1 pair edits the roster *and* drives systemd; the v2 pair is documented
* as changing desired state "without runtime actions", leaving convergence to
* `apply`. `add` also collects four fields where a v2 agent requires eleven, so
* routing it to `create` would mean inventing provider, alias, reasoning and
* tool-policy defaults on the operator's behalf. Refusing with the real command
* is honest; silently guessing an agent's provider is not.
*/
function rosterV2MutationGuidance(
v1Command: 'add' | 'remove',
v2Command: 'create' | 'delete',
name: string,
): string {
const target = v2Command === 'delete' ? ` ${name}` : '';
return (
`mosaic fleet ${v1Command} does not operate on a roster-v2 fleet. ` +
`Roster v2 separates desired state from convergence:\n` +
` 1. mosaic fleet ${v2Command}${target} --expected-generation <current> ` +
`${v2Command === 'create' ? "--agent '<json>' " : ''}` +
`(edits the roster only)\n` +
` 2. mosaic fleet apply --expected-generation <new> (converges systemd and tmux)\n` +
`Read the current generation with: mosaic fleet status`
);
}
/**
* The read-only fields shared by roster v1 and v2, for the commands that only
* ever *read* the roster (`ps`, and unit enablement inside `install`).
*
* This is deliberately NOT a v2→v1 downshift. A downshifted `FleetRoster` would
* be accepted by `generateAgentEnvValues`, and that would make a third writer of
* `fleet/agents/<name>.env.generated` — through the v1 mapping — breaking the
* #791 single-SSOT invariant that {@link projectRosterV2AgentGeneratedEnv} is
* documented to hold. Keeping the read model this small makes that misuse
* impossible: there is nothing here to write a roster or an env file back from.
*/
interface FleetRosterReadModel {
readonly version: 1 | 2;
readonly tmux: { readonly socketName: string; readonly holderSession: string };
readonly agents: readonly {
readonly name: string;
readonly alias?: string;
readonly runtime: string;
}[];
}
/** Reads either roster version into the shared read-only view. */
async function loadRosterReadModel(cmd: Command): Promise<FleetRosterReadModel> {
const opts = cmd.opts<{ mosaicHome: string; roster?: string }>();
const path = await resolveRosterPath(opts.mosaicHome, opts.roster);
if (!(await usesRosterV2ControlPlane(cmd))) {
const v1 = await loadRosterAtPath(cmd, path);
return {
version: 1,
tmux: { socketName: v1.tmux.socketName, holderSession: v1.tmux.holderSession },
agents: v1.agents,
};
}
try {
const v2 = parseRosterV2(await readFleetRosterText(path), 'yaml');
return {
version: 2,
tmux: { socketName: v2.tmux.socketName, holderSession: v2.tmux.holderSession },
agents: v2.agents,
};
} catch (error) {
reportFleetRosterConfigurationError(cmd, error);
}
}
async function loadRosterFromAgentCommand(
command: Command,
mosaicHomeOverride?: string,