Compare commits

..
Author SHA1 Message Date
fred 6f5b4c3dc1 fix(fleet): restore ConditionPathExists dropped by my own red-check
ci/woodpecker/pr/ci Pipeline was successful
Self-inflicted and worth recording rather than quietly amending.

To prove the new tests were red without the fix I ran
`git checkout origin/next -- <fleet.ts> <[email protected]>`. That writes
the *index*, not just the working tree. Copying my versions back afterwards
restored the working tree only, so the unit file sat staged-as-origin/next and
modified-in-tree, and the next commit (67f5014c) committed the index — silently
removing the ConditionPathExists line that 463745e3 had added.

Nothing caught it. The spec reads the file from the working tree, so it stayed
10/10 green against a HEAD that no longer had the guard. Found by reading
`git status` after the push, not by any gate.

Verified by content, not by assumption:
  origin/next  0 occurrences
  463745e3     1
  67f5014c     0   <- the regression
  this commit  1

Refs #1237
2026-08-15 23:35:54 -05:00
fred 67f5014cc0 fix(fleet): refuse v2 add/remove cleanly, and pin the Condition's effect
Two follow-ups from the canary red->green run and scooby's review.

1. The v2 refusal in `add`/`remove` was a bare `throw`, which reaches the CLI
   top level uncaught and prints the guidance under a Node stack trace. The
   message *is* the point of the refusal, so it now goes through
   `command.error()` — the same clean path the roster-config error uses.
   Caught on canary, not in review: the unit tests asserted the message text
   and passed either way.

2. The unit-template test asserted only that ConditionPathExists is present.
   Presence is not effect. Added two tests for the parts that can drift in
   code while that assertion still passes: the condition resolving to exactly
   the file the fleet writes (%h/%i rendered against a real install), and the
   launcher genuinely failing on an absent generated env (exit 64,
   `missing-file`) — which is what makes the condition load-bearing rather
   than decorative.

systemd is not available in the suite, so the effect itself was measured on
canary (2026-08-16), roster v2 generation 3:

  with the condition:    start rc=0, Result=success, ConditionResult=no,
                         journal "skipped, unmet condition check"
  condition removed by
  drop-in, nothing else: start rc=1, Result=exit-code, ExecMainStatus=64,
                         unit failed, "agent environment rejected: missing-file"

Canary red->green for the three commands, same v2 roster, side by side:

  fleet ps               0.0.50-next.2413 rc=1  ->  branch rc=0 (3 agents listed)
  fleet install          0.0.50-next.2413 rc=1  ->  branch rc=0
  fleet remove <name>    0.0.50-next.2413 rc=1  ->  branch rc=1, refusal naming
                                                    delete + apply

All three previously failed with "Fleet roster has unknown field(s):
generation." The #791 negative was measured too: the six existing
*.env.generated files were untouched by `install` (mtimes 20+ minutes older
than the run).

Gates: typecheck 0, eslint 0, prettier clean, fleet specs 382 passed, new spec
10/10 with the fix and 9/10 red against origin/next (the 10th passes there for
an unrelated reason and is annotated as such). Full suite: only
mutator-gate.acceptance.spec.ts fails, pre-existing on origin/next.

Still true and still worth saying: a correct fix here shows install rc=0 and
start rc=0 and STILL no live seat. #1240 (tmux absent) is upstream, #1241
(start reports lifecycle-complete over dead panes) and the missing agent
runtime are downstream.

Refs #1237
Reviewed-by: scooby (by git comms; cannot file a Gitea review from fomo-lin)
2026-08-15 23:34:35 -05:00
fred 463745e314 fix(#1237): let ps/install work on a roster-v2 fleet, and refuse add/remove honestly
On a roster-v2 fleet, `ps`, `install`, `install-systemd`, `add` and `remove`
all failed in the v1 parser. The consequence was that a greenfield v2 box could
never get its unit templates placed, so nothing downstream could start.

The read-only commands get a narrow version-agnostic view of the roster
(version, socket name, holder session, and per agent name/alias/runtime).
This is deliberately not a v2 -> v1 downshift. A downshifted FleetRoster would
be accepted by generateAgentEnvValues, which would make a third writer of
fleet/agents/<name>.env.generated through the v1 mapping and break the #791
single-SSOT invariant that projectRosterV2AgentGeneratedEnv is documented to
hold. The view is too small to write a roster or an env file back from, so that
misuse is unavailable rather than merely discouraged.

So on a v2 roster `install` places the tool files and the unit templates,
enables the units, and writes no generated env at all. Env belongs to `apply`
and `regen`, both already v2-native.

That change alone would have traded an init-time failure for a boot-time one.
`install` enables mosaic-agent@<name>.service (WantedBy=default.target) without
starting it, so a reboot between `install` and the first `apply` would run
ExecStart against an absent env file and fail every seat unit, further from its
cause. The unit template now carries

  ConditionPathExists=%h/.config/mosaic/fleet/agents/%i.env.generated

which skips an enabled-but-unconfigured unit cleanly and starts it on the next
start once the reconciler has written env. On v1 it is a no-op, since v1
`install` writes env itself. Found in review by scooby.

`add` and `remove` are not routed to `create` and `delete`. They are different
operations: the v1 pair edits the roster and drives systemd, the v2 pair is
documented as changing desired state without runtime actions. `add` also
collects four fields where a v2 agent requires eleven, so routing it would mean
inventing an operator's provider, alias, reasoning and tool policy. On v2 both
now fail with the real two-step sequence instead.

Tests: 8 new, 7 of which are red before this change. Includes the greenfield
case scooby asked for — `ps` on a fresh v2 install with nothing running is rc=0
and lists every agent stopped, since that is the command an operator runs to
find out why there is no seat.

Note for anyone verifying this: a correct fix here shows `install` rc=0 and
`start` rc=0 and still no live seat. #1240 (tmux absent) is upstream, #1241
(start reports lifecycle-complete over dead panes) and the missing agent
runtime are downstream. A dead pane after this change is not a regression here.

Refs #1237, #791, #1240, #1241
2026-08-15 23:24:24 -05:00
6 changed files with 449 additions and 197 deletions
@@ -4,6 +4,14 @@ Documentation=https://git.mosaicstack.dev/mosaicstack/stack
Requires=mosaic-tmux-holder.service Requires=mosaic-tmux-holder.service
After=mosaic-tmux-holder.service After=mosaic-tmux-holder.service
PartOf=mosaic-tmux-holder.service PartOf=mosaic-tmux-holder.service
# Do not attempt a seat before its generated env exists. `install` enables this
# unit (WantedBy=default.target) but on a roster-v2 fleet the reconciler owns the
# generated env, so between `install` and the first `apply`/`regen --write` there
# is a boot window where ExecStart would run against an absent env file and the
# launcher would fail the unit. A skipped unit is the honest state for "enabled
# but not yet configured"; systemd re-evaluates the condition on every start, so
# the seat comes up on the next start once the reconciler has written env.
ConditionPathExists=%h/.config/mosaic/fleet/agents/%i.env.generated
[Service] [Service]
Type=oneshot Type=oneshot
@@ -128,14 +128,6 @@ EOF
sleep 30 sleep 30
EOF EOF
chmod 700 "$AGENT_BIN/mosaic" chmod 700 "$AGENT_BIN/mosaic"
# The launcher resolves the roster's runtime against PANE_PATH before it
# spawns anything (#1241), so the runtime this projection names has to be
# present here even though the fake `mosaic` above never execs it.
cat > "$AGENT_BIN/pi" <<'EOF'
#!/bin/sh
sleep 30
EOF
chmod 700 "$AGENT_BIN/pi"
server_environment_before=$(tmux -L "$TEST_SOCKET" show-environment -g | sort) server_environment_before=$(tmux -L "$TEST_SOCKET" show-environment -g | sort)
server_sessions_before=$(tmux -L "$TEST_SOCKET" list-sessions | sort) server_sessions_before=$(tmux -L "$TEST_SOCKET" list-sessions | sort)
if /usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin MOSAIC_HOME="$AGENT_HOME" \ if /usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin MOSAIC_HOME="$AGENT_HOME" \
@@ -286,36 +286,6 @@ _build_runtime_bin_prefix() {
MOSAIC_RUNTIME_BIN_PREFIX=$(_build_runtime_bin_prefix) MOSAIC_RUNTIME_BIN_PREFIX=$(_build_runtime_bin_prefix)
PANE_PATH=${MOSAIC_RUNTIME_BIN_PREFIX:+${MOSAIC_RUNTIME_BIN_PREFIX}:}/usr/local/bin:/usr/bin:/bin PANE_PATH=${MOSAIC_RUNTIME_BIN_PREFIX:+${MOSAIC_RUNTIME_BIN_PREFIX}:}/usr/local/bin:/usr/bin:/bin
# #1241. The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared
# environment. A binary missing from *that* path is a pane that dies in under a
# second, inside a session nobody is attached to, with its diagnostic scrolled
# into a pane tmux then destroys. Resolve both here, before any effect, where
# the failure is still attributable to the thing that caused it.
#
# `mosaic yolo <runtime>` runs checkRuntime(runtime) and the binary it looks for
# is named exactly like the runtime, so resolving the runtime name is the same
# question the pane will ask a moment later — asked while an operator can still
# see the answer.
_resolve_in_pane_path() {
PATH="$PANE_PATH" command -v -- "$1" 2>/dev/null
}
# Exit 69 (EX_UNAVAILABLE): the seat cannot be provided. Distinguished from the
# 64 (EX_USAGE) rejections above, which mean the projection itself was bad —
# here the data is fine and the host is not ready. Callers tell the individual
# cases apart by `code=`, the same way fail_env's many codes share exit 64.
fail_launch() {
local code="$1"
shift
echo "ERROR: agent launch aborted: code=${code} agent=${AGENT_NAME} $*" >&2
exit 69
}
for required_binary in mosaic "$MOSAIC_AGENT_RUNTIME"; do
_resolve_in_pane_path "$required_binary" >/dev/null ||
fail_launch missing-binary "'${required_binary}' is not on the pane PATH (${PANE_PATH})"
done
_ensure_claude_workdir_trusted() { _ensure_claude_workdir_trusted() {
local workdir="$1" local workdir="$1"
local resolved local resolved
@@ -414,19 +384,6 @@ if [ -n "$PANE_PID" ]; then
_start_heartbeat_sidecar "$AGENT_NAME" "$PANE_PID" \ _start_heartbeat_sidecar "$AGENT_NAME" "$PANE_PID" \
"$MOSAIC_HEARTBEAT_RUN_DIR" "$MOSAIC_HEARTBEAT_INTERVAL" || \ "$MOSAIC_HEARTBEAT_RUN_DIR" "$MOSAIC_HEARTBEAT_INTERVAL" || \
echo "WARNING: heartbeat sidecar could not be started for $AGENT_NAME" >&2 echo "WARNING: heartbeat sidecar could not be started for $AGENT_NAME" >&2
elif _tmux has-session -t "=${AGENT_NAME}:0.0" 2>/dev/null; then
# #1241. Session present, no pane PID after a second of retries. Whatever this
# is, it is not a seat an operator can use, so it is not a success either.
fail_launch pane-pid-unresolved \
"tmux reports the session but no pane PID after 5 attempts"
else else
# #1241. This branch used to print a WARNING about the heartbeat sidecar and echo "WARNING: could not resolve pane PID for $AGENT_NAME — heartbeat sidecar not started" >&2
# exit 0. It is not a heartbeat problem: tmux destroys a session when its pane
# command exits, so an absent session one second after new-session means the
# runtime died on startup. Reporting it as success is what let `fleet start`
# return 0 over three dead panes — the launcher knew, and said the wrong thing
# at the wrong severity to the wrong layer.
fail_launch pane-did-not-survive \
"the pane exited immediately and tmux destroyed the session;" \
"run 'mosaic yolo ${MOSAIC_AGENT_RUNTIME}' in ${MOSAIC_AGENT_WORKDIR} to see why"
fi fi
@@ -23,26 +23,8 @@ index=0
if [ "${args[0]:-}" = -L ]; then index=2; fi if [ "${args[0]:-}" = -L ]; then index=2; fi
case "${args[$index]:-}" in case "${args[$index]:-}" in
has-session) has-session)
# The holder always answers. MOSAIC_TEST_HELD_SESSIONS lets a case add
# other targets that should answer too — without it there is no way to
# model "tmux still reports the session" for a non-holder agent, and the
# launcher's pane-pid-unresolved branch is unreachable from this harness.
#
# A listed target answers only AFTER new-session, because the launcher asks
# this question twice about the same name: once before launching, where a
# yes means "already running, nothing to do, exit 0", and once after, where
# a yes means "the session survived". A shim that answered yes to both
# would short-circuit at the first and never reach the branch under test —
# it would look like coverage and measure the idempotency path instead.
for argument in "${args[@]}"; do for argument in "${args[@]}"; do
[ "$argument" = '=_holder:0.0' ] && exit 0 [ "$argument" = '=_holder:0.0' ] && exit 0
case " ${MOSAIC_TEST_HELD_SESSIONS:-} " in
*" $argument "*)
if tr '\0' '\n' < "${MOSAIC_TEST_TMUX_CALLS:?}" | grep -qxF new-session; then
exit 0
fi
;;
esac
done done
exit 1 exit 1
;; ;;
@@ -80,30 +62,6 @@ env -0 > "${MOSAIC_HOME:?}/fleet/pane-environment"
SHIM SHIM
chmod +x "$FAKE_BIN/mosaic" chmod +x "$FAKE_BIN/mosaic"
# The runtime the rosters below name. The launcher resolves it against PANE_PATH
# before spawning (#1241), so it has to exist somewhere the pane would find it —
# not merely on the launcher's own PATH.
printf '#!/usr/bin/env bash\nexit 0\n' > "$FAKE_BIN/pi"
chmod +x "$FAKE_BIN/pi"
# PANE_PATH is derived partly from `npm config get prefix`. Left to the real npm
# it would splice whatever the host has installed into the path under test, and
# the missing-binary cases below would pass or fail by accident of the machine.
cat > "$FAKE_BIN/npm" <<'SHIM'
#!/usr/bin/env bash
printf '%s\n' "${MOSAIC_TEST_NPM_PREFIX:-/nonexistent}"
SHIM
chmod +x "$FAKE_BIN/npm"
# PANE_PATH always ends in the system path. A host that installs these there can
# not measure the missing-binary cases at all, and a green run would mean
# nothing — so say so instead of passing.
for host_binary in mosaic pi; do
if PATH=/usr/local/bin:/usr/bin:/bin command -v "$host_binary" >/dev/null 2>&1; then
fail "host provides '$host_binary' in the system path; missing-binary cases are not measurable here"
fi
done
write_generated() { write_generated() {
local home="$1" local home="$1"
local agent="$2" local agent="$2"
@@ -123,19 +81,6 @@ MOSAIC_TMUX_SOCKET=mosaic-test
EOF EOF
chmod 600 "$home/fleet/agents/$agent.env.generated" chmod 600 "$home/fleet/agents/$agent.env.generated"
mkdir -p "$home/work" mkdir -p "$home/work"
install_pane_binaries "$home"
}
# `$PANE_HOME/.npm-global/bin` is one of the prefixes the launcher folds into
# PANE_PATH, so this is the pane's own view of "installed", distinct from the
# launcher's PATH. Tests that need a binary *absent* remove it from here.
install_pane_binaries() {
local pane_home="$1"
mkdir -p "$pane_home/.npm-global/bin"
local binary
for binary in mosaic pi; do
ln -sf "$FAKE_BIN/$binary" "$pane_home/.npm-global/bin/$binary"
done
} }
run_start() { run_start() {
@@ -143,7 +88,6 @@ run_start() {
local agent="$2" local agent="$2"
HOME="$home" PATH="$FAKE_BIN:$PATH" MOSAIC_TEST_TMUX_CALLS="$TMUX_CALLS" \ HOME="$home" PATH="$FAKE_BIN:$PATH" MOSAIC_TEST_TMUX_CALLS="$TMUX_CALLS" \
MOSAIC_TEST_PANE_PID="${MOSAIC_TEST_PANE_PID:-}" \ MOSAIC_TEST_PANE_PID="${MOSAIC_TEST_PANE_PID:-}" \
MOSAIC_TEST_HELD_SESSIONS="${MOSAIC_TEST_HELD_SESSIONS:-}" \
MOSAIC_TEST_HOME="$home" \ MOSAIC_TEST_HOME="$home" \
MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \ MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \
MOSAIC_HOME="$home" "$START" "$agent" MOSAIC_HOME="$home" "$START" "$agent"
@@ -154,10 +98,7 @@ run_start() {
HOME_VALID="$ROOT/valid" HOME_VALID="$ROOT/valid"
AGENT_VALID="coder0" AGENT_VALID="coder0"
write_generated "$HOME_VALID" "$AGENT_VALID" write_generated "$HOME_VALID" "$AGENT_VALID"
# A live pane PID is part of what "valid launch" means. Until #1241 this case run_start "$HOME_VALID" "$AGENT_VALID"
# ran with none, so the suite's one success path was itself a dead pane the
# launcher reported as fine.
MOSAIC_TEST_PANE_PID=$$ run_start "$HOME_VALID" "$AGENT_VALID"
valid_args=$(tr '\0' '\n' < "$TMUX_CALLS") valid_args=$(tr '\0' '\n' < "$TMUX_CALLS")
echo "$valid_args" | grep -qF new-session || fail "valid generated projection did not reach tmux" echo "$valid_args" | grep -qF new-session || fail "valid generated projection did not reach tmux"
echo "$valid_args" | grep -qF 'mosaic' || fail "fixed mosaic launcher command missing" echo "$valid_args" | grep -qF 'mosaic' || fail "fixed mosaic launcher command missing"
@@ -304,13 +245,6 @@ PANE_BASH_ENV="$ROOT/pane-boundary.bash-env"
printf 'MOSAIC_RUNTIME_BIN=%s\n' "$FAKE_BIN" > \ printf 'MOSAIC_RUNTIME_BIN=%s\n' "$FAKE_BIN" > \
"$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local" "$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local"
chmod 600 "$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local" chmod 600 "$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local"
# This case does not go through run_start, so its pane binaries come from
# MOSAIC_RUNTIME_BIN=$FAKE_BIN in the env.local written above — not from the
# symlinks install_pane_binaries planted under the generated home, which this
# launcher never consults because HOME here is the trusted parent. That is a
# legitimate resolution path, but it means dropping MOSAIC_RUNTIME_BIN from
# this case on the belief that the symlinks cover it would break the #1241
# binary check rather than exercise it.
LD_PRELOAD='/not/loaded/by-clean-bootstrap.so' \ LD_PRELOAD='/not/loaded/by-clean-bootstrap.so' \
BASH_ENV="$PANE_BASH_ENV" \ BASH_ENV="$PANE_BASH_ENV" \
MOSAIC_UNTRUSTED_SENTINEL='must-not-reach-pane' \ MOSAIC_UNTRUSTED_SENTINEL='must-not-reach-pane' \
@@ -324,7 +258,6 @@ PATH="$PANE_STALE_PATH" \
"MOSAIC_TEST_HOME=$PANE_TRUSTED_HOME" \ "MOSAIC_TEST_HOME=$PANE_TRUSTED_HOME" \
MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \ MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \
MOSAIC_TEST_EXECUTE_PANE=1 \ MOSAIC_TEST_EXECUTE_PANE=1 \
"MOSAIC_TEST_PANE_PID=$$" \
"$START" coder-pane-boundary "$START" coder-pane-boundary
pane_args=$(tr '\0' '\n' < "$TMUX_CALLS") pane_args=$(tr '\0' '\n' < "$TMUX_CALLS")
echo "$pane_args" | grep -qxF "HOME=$PANE_TRUSTED_HOME" || \ echo "$pane_args" | grep -qxF "HOME=$PANE_TRUSTED_HOME" || \
@@ -459,75 +392,6 @@ echo "$interaction_policy_args" | grep -qF 'new-session' && \
echo "$output" | grep -qF 'operator interaction service requires runtime pi' || \ echo "$output" | grep -qF 'operator interaction service requires runtime pi' || \
fail "interaction pinned-policy check did not follow strict parsing" fail "interaction pinned-policy check did not follow strict parsing"
# #1241. The pane runs `mosaic yolo <runtime>` against PANE_PATH. A binary
# missing from that path is a launch failure, and it has to be named before the
# session is created — after it, the diagnostic dies with the pane.
assert_missing_pane_binary_rejected() {
local binary="$1"
local home="$ROOT/missing-$binary"
local agent="coder-missing-$binary"
write_generated "$home" "$agent"
rm -f "$home/.npm-global/bin/$binary"
: > "$TMUX_CALLS"
local output
if output=$(MOSAIC_TEST_PANE_PID=$$ run_start "$home" "$agent" 2>&1); then
fail "launch succeeded with '$binary' absent from the pane PATH"
fi
echo "$output" | grep -qF 'code=missing-binary' || fail "missing '$binary' diagnostic missing"
echo "$output" | grep -qF "'$binary'" || fail "missing-binary diagnostic did not name $binary"
if tr '\0' '\n' < "$TMUX_CALLS" | grep -qF new-session; then
fail "launcher created a session it knew would die ($binary absent)"
fi
}
assert_missing_pane_binary_rejected mosaic
assert_missing_pane_binary_rejected pi
# #1241. tmux destroys a session when its pane command exits, so no pane PID a
# second after new-session means the runtime died on startup. This used to be a
# WARNING about the heartbeat sidecar followed by exit 0 — three layers above it
# then reported a fleet that was not running.
: > "$TMUX_CALLS"
HOME_DEAD_PANE="$ROOT/dead-pane"
write_generated "$HOME_DEAD_PANE" "coder-dead-pane"
if output=$(MOSAIC_TEST_PANE_PID='' run_start "$HOME_DEAD_PANE" coder-dead-pane 2>&1); then
fail "launcher reported success over a pane that did not survive"
fi
echo "$output" | grep -qF 'code=pane-did-not-survive' || fail "dead-pane diagnostic missing"
if echo "$output" | grep -qiF 'heartbeat'; then
fail "dead pane is still being reported as a heartbeat-sidecar problem"
fi
tr '\0' '\n' < "$TMUX_CALLS" | grep -qF new-session || \
fail "dead-pane case did not reach the launch it is measuring"
# #1241, the other way a pane fails. Above, tmux destroyed the session and
# has-session said so. Here the session is still there and no PID comes back
# after the retries — a different fault (the pane is alive but unusable, or
# tmux is answering inconsistently) that an operator has to be told apart from
# a runtime that died on startup.
#
# This case exists because the branch that handles it shipped with nothing able
# to reach it: the shim answered has-session only for the holder, so every
# non-holder agent landed in the session-is-gone branch no matter what. A
# defensive branch nothing exercises is the same shape as the bug this whole
# change is about, one layer down.
: > "$TMUX_CALLS"
HOME_NO_PID="$ROOT/pane-no-pid"
write_generated "$HOME_NO_PID" "coder-no-pid"
if output=$(MOSAIC_TEST_PANE_PID='' MOSAIC_TEST_HELD_SESSIONS='=coder-no-pid:0.0' \
run_start "$HOME_NO_PID" coder-no-pid 2>&1); then
fail "launcher reported success over a session with no resolvable pane PID"
fi
echo "$output" | grep -qF 'code=pane-pid-unresolved' || \
fail "session-present/no-PID was not reported as pane-pid-unresolved: $output"
if echo "$output" | grep -qF 'code=pane-did-not-survive'; then
fail "a session tmux still reports was diagnosed as a destroyed session"
fi
if echo "$output" | grep -qiF 'heartbeat'; then
fail "an unresolvable pane PID is still being reported as a heartbeat-sidecar problem"
fi
# Exact stop derives the socket exclusively from the validated generated # Exact stop derives the socket exclusively from the validated generated
# projection and ignores an ambient socket supplied by the caller. # projection and ignores an ambient socket supplied by the caller.
: > "$TMUX_CALLS" : > "$TMUX_CALLS"
@@ -0,0 +1,323 @@
import { execFile } from 'node:child_process';
import { mkdir, mkdtemp, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join, resolve } from 'node:path';
import { Command } from 'commander';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { registerFleetCommand, type CommandResult, type CommandRunner } from './fleet.js';
/**
* #1237: the v1-only commands (`ps`, `install`, `install-systemd`, `add`,
* `remove`) rejected a roster-v2 fleet outright, so a greenfield v2 box could
* never get its units placed. These tests pin the three behaviours that fix
* gives it, and the two it deliberately does NOT give it.
*
* The load-bearing negative is that `install` on v2 writes no generated env:
* the reconciler owns that file through projectRosterV2AgentGeneratedEnv, and a
* second writer here — necessarily through the v1 mapping — is exactly the
* drift the #791 single-SSOT invariant exists to prevent.
*/
const rosterV2 = `
version: 2
generation: 4
transport: tmux
tmux:
socket_name: mosaic-fleet
holder_session: _holder
defaults:
working_directory: /srv/mosaic
runtime: pi
runtimes:
pi:
reset_command: /new
agents:
- name: coder0
alias: Coder 0
class: code
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: high
tool_policy: code
working_directory: /srv/mosaic
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
- name: coder1
alias: Coder 1
class: code
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: medium
tool_policy: code
working_directory: /srv/other
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
`;
let tempHome: string | undefined;
const savedHome = process.env.HOME;
const savedMosaicHome = process.env.MOSAIC_HOME;
afterEach(async (): Promise<void> => {
vi.restoreAllMocks();
process.exitCode = undefined;
if (savedHome === undefined) delete process.env.HOME;
else process.env.HOME = savedHome;
if (savedMosaicHome === undefined) delete process.env.MOSAIC_HOME;
else process.env.MOSAIC_HOME = savedMosaicHome;
if (tempHome) await rm(tempHome, { recursive: true, force: true });
tempHome = undefined;
});
/**
* A HOME with a roster-v2 fleet and nothing else — the greenfield shape, before
* anything has been installed, applied or started.
*/
async function v2Home(): Promise<string> {
tempHome = await mkdtemp(join(tmpdir(), 'mosaic-fleet-v2-dispatch-'));
process.env.HOME = tempHome;
delete process.env.MOSAIC_HOME;
const mosaicHome = join(tempHome, '.config', 'mosaic');
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
await mkdir(join(mosaicHome, directory), { recursive: true, mode: 0o700 });
}
await writeFile(join(mosaicHome, 'fleet', 'roster.yaml'), rosterV2, { mode: 0o600 });
await writeFile(join(mosaicHome, 'fleet', 'roles', 'code.md'), '`class: code`\n\n# code\n', {
mode: 0o600,
});
return mosaicHome;
}
/**
* Stands in for a box where nothing is running: every systemctl and tmux probe
* fails the way it does before the holder has ever started. `ps` must survive
* this — it is the command an operator reaches for to find out *why* there is
* no seat, so it has to report the emptiness rather than fail on it.
*/
const greenfieldRunner: CommandRunner = async (command): Promise<CommandResult> => {
if (command === 'tmux') {
return { stdout: '', stderr: 'no server running on /tmp/tmux-1000/mosaic-fleet', exitCode: 1 };
}
return { stdout: '', stderr: '', exitCode: 1 };
};
function program(runner: CommandRunner = greenfieldRunner): Command {
const result = new Command();
result.exitOverride();
registerFleetCommand(result, { runner, frameworkRoot: resolve(process.cwd(), 'framework') });
return result;
}
function capture(): string[] {
const lines: string[] = [];
vi.spyOn(console, 'log').mockImplementation((value: string): void => {
lines.push(value);
});
return lines;
}
async function exists(path: string): Promise<boolean> {
try {
await stat(path);
return true;
} catch {
return false;
}
}
describe('mosaic fleet ps — roster v2', (): void => {
it('lists every v2 agent on a greenfield box with nothing running, and does not throw', async (): Promise<void> => {
await v2Home();
const lines = capture();
await expect(
program().parseAsync(['node', 'mosaic', 'fleet', 'ps', '--json']),
).resolves.toBeDefined();
const rows = JSON.parse(lines.join('\n')) as {
name: string;
runtime: string;
alias?: string;
paneAlive: boolean;
source: string;
}[];
expect(rows.map((row) => row.name).sort()).toEqual(['coder0', 'coder1']);
// The v2 roster's per-agent fields must survive the read model, not be
// flattened into defaults.
expect(rows.every((row) => row.runtime === 'pi')).toBe(true);
expect(rows.find((row) => row.name === 'coder0')?.alias).toBe('Coder 0');
// Nothing is running, and that is a report, not an error.
expect(rows.every((row) => row.paneAlive === false)).toBe(true);
expect(rows.every((row) => row.source === 'roster')).toBe(true);
expect(process.exitCode ?? 0).toBe(0);
});
});
describe('mosaic fleet install — roster v2', (): void => {
it('places the tool files and unit templates', async (): Promise<void> => {
const mosaicHome = await v2Home();
capture();
await expect(
program().parseAsync(['node', 'mosaic', 'fleet', 'install', '--no-enable']),
).resolves.toBeDefined();
// Units live in the systemd user dir, not under the Mosaic home.
const systemdUserDir = join(tempHome!, '.config', 'systemd', 'user');
for (const unit of [
'mosaic-tmux-holder.service',
'[email protected]',
'[email protected]',
]) {
expect(await exists(join(systemdUserDir, unit))).toBe(true);
}
const launcher = join(mosaicHome, 'tools', 'fleet', 'start-agent-session.sh');
expect(await exists(launcher)).toBe(true);
expect((await stat(launcher)).mode & 0o777).toBe(0o755);
});
it('writes NO generated env — that file belongs to the reconciler (#791)', async (): Promise<void> => {
const mosaicHome = await v2Home();
capture();
await program().parseAsync(['node', 'mosaic', 'fleet', 'install', '--no-enable']);
const agentDir = join(mosaicHome, 'fleet', 'agents');
expect(await readdir(agentDir)).toEqual([]);
});
it('tells the operator which command does own the env', async (): Promise<void> => {
await v2Home();
const lines = capture();
await program().parseAsync(['node', 'mosaic', 'fleet', 'install', '--no-enable']);
expect(lines.join('\n')).toContain('mosaic fleet apply');
});
});
describe('[email protected]', (): void => {
const unitPath = resolve(process.cwd(), 'framework', 'systemd', 'user', '[email protected]');
/** The single `ConditionPathExists=` value declared by the unit template. */
async function conditionPath(): Promise<string> {
const unit = await readFile(unitPath, 'utf8');
const matches = unit.match(/^ConditionPathExists=(.+)$/gm) ?? [];
expect(matches).toHaveLength(1);
return matches[0]!.slice('ConditionPathExists='.length).trim();
}
it('will not attempt a seat before the reconciler has written its env', async (): Promise<void> => {
// The pairing that makes "install writes no env" safe: install enables the
// unit (WantedBy=default.target) but does not start it, so without this
// condition a reboot between `install` and the first `apply` would run
// ExecStart against an absent env file and fail every seat unit.
expect(await conditionPath()).toBe('%h/.config/mosaic/fleet/agents/%i.env.generated');
});
/**
* The two halves of the guard's *effect*, which no assertion on the literal
* string can cover on its own.
*
* Measured end to end on a real box (canary, 2026-08-16) rather than inferred:
* with the condition, `systemctl --user start mosaic-agent@<name>` on an agent
* with no generated env returns rc=0, `Result=success`, `ConditionResult=no`,
* and journals "skipped, unmet condition check". With the condition removed by
* drop-in and nothing else changed, the same start returns rc=1,
* `Result=exit-code`, `ExecMainStatus=64`, and the unit enters `failed`.
*
* systemd is not available in this suite, so these two tests pin the parts
* that can drift in code: the condition naming a *different* file than the one
* the fleet actually writes, and the launcher quietly becoming tolerant of an
* absent env — either of which turns the condition into decoration while the
* literal-string assertion above still passes.
*/
it('guards exactly the file the fleet writes, so the two cannot drift apart', async (): Promise<void> => {
const mosaicHome = await v2Home();
const rendered = (await conditionPath()).replace('%h', tempHome!).replace('%i', 'coder0');
// The path an installed fleet actually places for this agent.
expect(rendered).toBe(join(mosaicHome, 'fleet', 'agents', 'coder0.env.generated'));
});
it('guards a real failure — the launcher rejects an absent generated env', async (): Promise<void> => {
await v2Home();
await program().parseAsync(['node', 'mosaic', 'fleet', 'install', '--no-enable']);
// Exactly what ExecStart runs, against the state the condition exists to
// catch: unit enabled, reconciler has not written env yet.
const launched = await new Promise<{ code: number | null; stderr: string }>((settle) => {
const child = execFile(
'/bin/bash',
[
'--noprofile',
'--norc',
join(tempHome!, '.config', 'mosaic', 'tools', 'fleet', 'start-agent-session.sh'),
'coder0',
],
{ env: { HOME: tempHome!, MOSAIC_AGENT_NAME: 'coder0', PATH: '/usr/bin:/bin' } },
(_error, _stdout, stderr) => {
settle({ code: child.exitCode, stderr });
},
);
});
expect(launched.code).not.toBe(0);
expect(launched.stderr).toContain('missing-file');
});
});
describe('mosaic fleet add / remove — roster v2', (): void => {
it('add refuses, and names the two-step v2 sequence instead of inventing defaults', async (): Promise<void> => {
await v2Home();
await expect(
program().parseAsync([
'node',
'mosaic',
'fleet',
'add',
'coder2',
'--runtime',
'pi',
'--class',
'code',
]),
).rejects.toThrow(/mosaic fleet create[\s\S]*mosaic fleet apply/);
});
it('remove refuses, and names delete plus apply', async (): Promise<void> => {
await v2Home();
await expect(
program().parseAsync(['node', 'mosaic', 'fleet', 'remove', 'coder1']),
).rejects.toThrow(/mosaic fleet delete coder1[\s\S]*mosaic fleet apply/);
});
// Note: this one passes on the unmodified tree too — there `remove` throws in
// the v1 parser, before it can touch anything. It is a regression guard on the
// ordering of the new guard clause, not evidence that the fix works.
it('refuses BEFORE mutating the roster', async (): Promise<void> => {
const mosaicHome = await v2Home();
const rosterPath = join(mosaicHome, 'fleet', 'roster.yaml');
const before = await readFile(rosterPath, 'utf8');
await expect(
program().parseAsync(['node', 'mosaic', 'fleet', 'remove', 'coder1']),
).rejects.toThrow();
expect(await readFile(rosterPath, 'utf8')).toBe(before);
});
});
+116 -8
View File
@@ -34,6 +34,7 @@ export {
resolveInstalledFleetRosterPath, resolveInstalledFleetRosterPath,
} from '../fleet/fleet-roster-v1.js'; } from '../fleet/fleet-roster-v1.js';
export type { FleetAgent, FleetRoster } from '../fleet/fleet-roster-v1.js'; export type { FleetAgent, FleetRoster } from '../fleet/fleet-roster-v1.js';
import { parseRosterV2 } from '../fleet/roster-v2.js';
import { import {
registerFleetAgentCrudCommands, registerFleetAgentCrudCommands,
type FleetAgentCrudCommandDeps, type FleetAgentCrudCommandDeps,
@@ -820,7 +821,7 @@ export function buildEnableLingerCommand(user: string): string[] {
*/ */
export async function enableFleetUnits( export async function enableFleetUnits(
runner: CommandRunner, runner: CommandRunner,
roster: FleetRoster, roster: { readonly agents: readonly { readonly name: string }[] },
opts: { enable?: boolean }, opts: { enable?: boolean },
): Promise<void> { ): Promise<void> {
if (opts.enable === false) { if (opts.enable === false) {
@@ -1527,7 +1528,8 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.option('--no-enable', 'Skip enabling units for boot-survival') .option('--no-enable', 'Skip enabling units for boot-survival')
.action(async (opts: { enable?: boolean }) => { .action(async (opts: { enable?: boolean }) => {
await installFleet(cmd, frameworkRoot); await installFleet(cmd, frameworkRoot);
const roster = await loadRosterForCommand(cmd); // Unit enablement needs agent names only, so it reads either version.
const roster = await loadRosterReadModel(cmd);
await enableFleetUnits(runner, roster, opts); await enableFleetUnits(runner, roster, opts);
}); });
@@ -1537,7 +1539,8 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.option('--no-enable', 'Skip enabling units for boot-survival') .option('--no-enable', 'Skip enabling units for boot-survival')
.action(async (opts: { enable?: boolean }) => { .action(async (opts: { enable?: boolean }) => {
await installFleet(cmd, frameworkRoot); await installFleet(cmd, frameworkRoot);
const roster = await loadRosterForCommand(cmd); // Unit enablement needs agent names only, so it reads either version.
const roster = await loadRosterReadModel(cmd);
await enableFleetUnits(runner, roster, opts); await enableFleetUnits(runner, roster, opts);
}); });
@@ -1688,7 +1691,9 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.action(async (opts: { json?: boolean }) => { .action(async (opts: { json?: boolean }) => {
const commandOpts = cmd.opts<{ mosaicHome: string; roster?: string }>(); const commandOpts = cmd.opts<{ mosaicHome: string; roster?: string }>();
const activePaths = resolveFleetPaths(commandOpts.mosaicHome); const activePaths = resolveFleetPaths(commandOpts.mosaicHome);
const roster = await loadRosterForCommand(cmd); // ps only reads, so it takes the version-agnostic read model rather than
// the v1 parser, which rejects a v2 roster outright.
const roster = await loadRosterReadModel(cmd);
const { tenant_id, host } = getDefaultTenantAndHost(); const { tenant_id, host } = getDefaultTenantAndHost();
const nowMs = Date.now(); const nowMs = Date.now();
@@ -1908,6 +1913,16 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
start: boolean; start: boolean;
}, },
) => { ) => {
if (await usesRosterV2ControlPlane(cmd)) {
// command.error, not a bare throw: this is operator guidance, and a
// bare throw reaches the top level uncaught and prints it under a Node
// stack trace. Measured on canary — the message is the whole point of
// the refusal, so it has to arrive readable.
cmd.error(rosterV2MutationGuidance('add', 'create', name), {
code: 'fleet.roster-v2',
exitCode: 1,
});
}
if (!VALID_FLEET_RUNTIMES.includes(opts.runtime)) { if (!VALID_FLEET_RUNTIMES.includes(opts.runtime)) {
throw new Error( throw new Error(
`Invalid runtime "${opts.runtime}". Valid runtimes: ${VALID_FLEET_RUNTIMES.join(', ')}.`, `Invalid runtime "${opts.runtime}". Valid runtimes: ${VALID_FLEET_RUNTIMES.join(', ')}.`,
@@ -1973,6 +1988,12 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.description('Remove an agent from the fleet roster') .description('Remove an agent from the fleet roster')
.option('--keep-files', 'Skip deleting env and heartbeat files') .option('--keep-files', 'Skip deleting env and heartbeat files')
.action(async (name: string, opts: { keepFiles?: boolean }) => { .action(async (name: string, opts: { keepFiles?: boolean }) => {
if (await usesRosterV2ControlPlane(cmd)) {
cmd.error(rosterV2MutationGuidance('remove', 'delete', name), {
code: 'fleet.roster-v2',
exitCode: 1,
});
}
const commandOpts = cmd.opts<{ mosaicHome: string; roster?: string }>(); const commandOpts = cmd.opts<{ mosaicHome: string; roster?: string }>();
const activePaths = resolveFleetPaths(commandOpts.mosaicHome); const activePaths = resolveFleetPaths(commandOpts.mosaicHome);
const rosterPath = await resolveRosterPath(commandOpts.mosaicHome, commandOpts.roster); const rosterPath = await resolveRosterPath(commandOpts.mosaicHome, commandOpts.roster);
@@ -2331,7 +2352,9 @@ export function registerFleetAgentCommands(
async function installFleet(cmd: Command, frameworkRoot: string): Promise<void> { async function installFleet(cmd: Command, frameworkRoot: string): Promise<void> {
const activePaths = resolveFleetPaths(cmd.opts<{ mosaicHome: string }>().mosaicHome); const activePaths = resolveFleetPaths(cmd.opts<{ mosaicHome: string }>().mosaicHome);
assertDefaultMosaicHomeForSystemd(activePaths.mosaicHome); assertDefaultMosaicHomeForSystemd(activePaths.mosaicHome);
const roster = await loadRosterForCommand(cmd); // Read model first: every file this function places is roster-independent, and
// the v1 parser would reject a v2 roster before any of them were written.
const roster = await loadRosterReadModel(cmd);
await ensureFleetHolderIdentity(activePaths.mosaicHome); await ensureFleetHolderIdentity(activePaths.mosaicHome);
await mkdir(activePaths.fleetToolsDir, { recursive: true }); await mkdir(activePaths.fleetToolsDir, { recursive: true });
await mkdir(activePaths.tmuxToolsDir, { recursive: true }); await mkdir(activePaths.tmuxToolsDir, { recursive: true });
@@ -2391,16 +2414,30 @@ async function installFleet(cmd: Command, frameworkRoot: string): Promise<void>
join(activePaths.systemdUserDir, '[email protected]'), join(activePaths.systemdUserDir, '[email protected]'),
); );
for (const agent of roster.agents) { // On roster v2 the reconciler owns the generated env: `apply` writes it and
// `regen` rebuilds it, both from projectRosterV2AgentGeneratedEnv. Writing it
// here too — necessarily through the v1 mapping — would be the third writer of
// one file and would break the #791 single-SSOT invariant. So v2 gets the tool
// files and the units, and nothing else.
if (roster.version === 2) {
console.log(
`Installed fleet tools and systemd units for ${roster.agents.length} agent(s). ` +
`Generated env is owned by the reconciler on roster v2 — run: mosaic fleet apply --expected-generation <n>`,
);
return;
}
const v1Roster = await loadRosterForCommand(cmd);
for (const agent of v1Roster.agents) {
await writeAgentEnvironmentProjection({ await writeAgentEnvironmentProjection({
mosaicHome: activePaths.mosaicHome, mosaicHome: activePaths.mosaicHome,
agentEnvDir: activePaths.agentEnvDir, agentEnvDir: activePaths.agentEnvDir,
agentName: agent.name, agentName: agent.name,
generated: generateAgentEnvValues(roster, agent), generated: generateAgentEnvValues(v1Roster, agent),
}); });
} }
console.log(`Installed fleet files for ${roster.agents.length} agent(s).`); console.log(`Installed fleet files for ${v1Roster.agents.length} agent(s).`);
} }
async function loadRosterForCommand(cmd: Command): Promise<FleetRoster> { async function loadRosterForCommand(cmd: Command): Promise<FleetRoster> {
@@ -2427,6 +2464,77 @@ async function usesRosterV2ControlPlane(cmd: Command): Promise<boolean> {
); );
} }
/**
* `add`/`remove` and `create`/`delete` are not two spellings of one operation.
* The v1 pair edits the roster *and* drives systemd; the v2 pair is documented
* as changing desired state "without runtime actions", leaving convergence to
* `apply`. `add` also collects four fields where a v2 agent requires eleven, so
* routing it to `create` would mean inventing provider, alias, reasoning and
* tool-policy defaults on the operator's behalf. Refusing with the real command
* is honest; silently guessing an agent's provider is not.
*/
function rosterV2MutationGuidance(
v1Command: 'add' | 'remove',
v2Command: 'create' | 'delete',
name: string,
): string {
const target = v2Command === 'delete' ? ` ${name}` : '';
return (
`mosaic fleet ${v1Command} does not operate on a roster-v2 fleet. ` +
`Roster v2 separates desired state from convergence:\n` +
` 1. mosaic fleet ${v2Command}${target} --expected-generation <current> ` +
`${v2Command === 'create' ? "--agent '<json>' " : ''}` +
`(edits the roster only)\n` +
` 2. mosaic fleet apply --expected-generation <new> (converges systemd and tmux)\n` +
`Read the current generation with: mosaic fleet status`
);
}
/**
* The read-only fields shared by roster v1 and v2, for the commands that only
* ever *read* the roster (`ps`, and unit enablement inside `install`).
*
* This is deliberately NOT a v2→v1 downshift. A downshifted `FleetRoster` would
* be accepted by `generateAgentEnvValues`, and that would make a third writer of
* `fleet/agents/<name>.env.generated` — through the v1 mapping — breaking the
* #791 single-SSOT invariant that {@link projectRosterV2AgentGeneratedEnv} is
* documented to hold. Keeping the read model this small makes that misuse
* impossible: there is nothing here to write a roster or an env file back from.
*/
interface FleetRosterReadModel {
readonly version: 1 | 2;
readonly tmux: { readonly socketName: string; readonly holderSession: string };
readonly agents: readonly {
readonly name: string;
readonly alias?: string;
readonly runtime: string;
}[];
}
/** Reads either roster version into the shared read-only view. */
async function loadRosterReadModel(cmd: Command): Promise<FleetRosterReadModel> {
const opts = cmd.opts<{ mosaicHome: string; roster?: string }>();
const path = await resolveRosterPath(opts.mosaicHome, opts.roster);
if (!(await usesRosterV2ControlPlane(cmd))) {
const v1 = await loadRosterAtPath(cmd, path);
return {
version: 1,
tmux: { socketName: v1.tmux.socketName, holderSession: v1.tmux.holderSession },
agents: v1.agents,
};
}
try {
const v2 = parseRosterV2(await readFleetRosterText(path), 'yaml');
return {
version: 2,
tmux: { socketName: v2.tmux.socketName, holderSession: v2.tmux.holderSession },
agents: v2.agents,
};
} catch (error) {
reportFleetRosterConfigurationError(cmd, error);
}
}
async function loadRosterFromAgentCommand( async function loadRosterFromAgentCommand(
command: Command, command: Command,
mosaicHomeOverride?: string, mosaicHomeOverride?: string,