fix(fleet): fail the agent launcher when the pane cannot survive (#1241)
ci/woodpecker/pr/ci Pipeline was successful
ci/woodpecker/pr/ci Pipeline was successful
`mosaic fleet start` returned 0 over three dead panes. The launcher knew, and said the wrong thing at the wrong severity to the wrong layer. The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared environment. When that binary is absent the pane dies in under a second, tmux destroys the session, and the diagnostic goes with it. The launcher then found no PANE_PID, printed a WARNING about the *heartbeat sidecar*, and exited 0 — so systemd logged "Finished ... successfully" and `fleet start` reported success. `fleet ps` was the only component telling the truth. Two changes, both in start-agent-session.sh: 1. Before any effect, resolve `mosaic` and the roster's runtime against PANE_PATH — the pane's own view of the path, not the launcher's. `mosaic yolo <runtime>` calls checkRuntime(runtime) and looks for a binary named exactly like the runtime, so this asks the same question the pane will ask a moment later, while an operator can still see the answer. Absent binary -> exit 69, code=missing-binary, no session created. 2. Replace the dead-pane WARNING+exit-0. An absent session one second after new-session is a runtime that died on startup, not a heartbeat problem -> exit 69, code=pane-did-not-survive, with the command to run by hand to see why. A present session with no pane PID after five attempts -> code=pane-pid-unresolved. Neither branch kills the session; destroying a possibly-live pane on a guess is worse than leaving it for inspection. Exit 69 (EX_UNAVAILABLE) is deliberate: the 64s already in this file mean the projection was bad, and here the data is fine and the host is not ready. Callers separate the cases by `code=`, the same way fail_env's codes share 64. This propagates for free. `fleet start` calls runChecked() for the holder and each agent, and runChecked throws on non-zero, so layers 4 and 5 stop lying without a TypeScript change. Two adjacent defects are left for a follow-up issue rather than widened into this diff: the per-agent loop aborts on the first failure instead of attempting all and reporting an aggregate, and runChecked's bare throw surfaces the launcher's message under a Node unhandled-rejection stack trace because program.parse() is synchronous. Tests: - test-start-agent-session.sh gains three cases: `mosaic` absent from the pane path, the runtime absent from the pane path, and a pane that does not survive. Each was verified individually red against the unmodified origin/next launcher. - The two cases asserting a valid launch now supply a pane PID. Until now the suite's one success path was itself a dead pane the launcher reported as fine. - The harness fakes `npm` so PANE_PATH stops depending on whatever the host has installed, and fails loudly if the host provides `mosaic` or `pi` in the system path, where the missing-binary cases would not be measurable at all. - test-fleet-units.sh gains a `pi` shim in its runtime bin. The real-tmux harness named `pi` in its roster and never installed it; the new preflight caught it. Refs #1241
This commit is contained in:
@@ -286,6 +286,36 @@ _build_runtime_bin_prefix() {
|
||||
MOSAIC_RUNTIME_BIN_PREFIX=$(_build_runtime_bin_prefix)
|
||||
PANE_PATH=${MOSAIC_RUNTIME_BIN_PREFIX:+${MOSAIC_RUNTIME_BIN_PREFIX}:}/usr/local/bin:/usr/bin:/bin
|
||||
|
||||
# #1241. The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared
|
||||
# environment. A binary missing from *that* path is a pane that dies in under a
|
||||
# second, inside a session nobody is attached to, with its diagnostic scrolled
|
||||
# into a pane tmux then destroys. Resolve both here, before any effect, where
|
||||
# the failure is still attributable to the thing that caused it.
|
||||
#
|
||||
# `mosaic yolo <runtime>` runs checkRuntime(runtime) and the binary it looks for
|
||||
# is named exactly like the runtime, so resolving the runtime name is the same
|
||||
# question the pane will ask a moment later — asked while an operator can still
|
||||
# see the answer.
|
||||
_resolve_in_pane_path() {
|
||||
PATH="$PANE_PATH" command -v -- "$1" 2>/dev/null
|
||||
}
|
||||
|
||||
# Exit 69 (EX_UNAVAILABLE): the seat cannot be provided. Distinguished from the
|
||||
# 64 (EX_USAGE) rejections above, which mean the projection itself was bad —
|
||||
# here the data is fine and the host is not ready. Callers tell the individual
|
||||
# cases apart by `code=`, the same way fail_env's many codes share exit 64.
|
||||
fail_launch() {
|
||||
local code="$1"
|
||||
shift
|
||||
echo "ERROR: agent launch aborted: code=${code} agent=${AGENT_NAME} $*" >&2
|
||||
exit 69
|
||||
}
|
||||
|
||||
for required_binary in mosaic "$MOSAIC_AGENT_RUNTIME"; do
|
||||
_resolve_in_pane_path "$required_binary" >/dev/null ||
|
||||
fail_launch missing-binary "'${required_binary}' is not on the pane PATH (${PANE_PATH})"
|
||||
done
|
||||
|
||||
_ensure_claude_workdir_trusted() {
|
||||
local workdir="$1"
|
||||
local resolved
|
||||
@@ -384,6 +414,19 @@ if [ -n "$PANE_PID" ]; then
|
||||
_start_heartbeat_sidecar "$AGENT_NAME" "$PANE_PID" \
|
||||
"$MOSAIC_HEARTBEAT_RUN_DIR" "$MOSAIC_HEARTBEAT_INTERVAL" || \
|
||||
echo "WARNING: heartbeat sidecar could not be started for $AGENT_NAME" >&2
|
||||
elif _tmux has-session -t "=${AGENT_NAME}:0.0" 2>/dev/null; then
|
||||
# #1241. Session present, no pane PID after a second of retries. Whatever this
|
||||
# is, it is not a seat an operator can use, so it is not a success either.
|
||||
fail_launch pane-pid-unresolved \
|
||||
"tmux reports the session but no pane PID after 5 attempts"
|
||||
else
|
||||
echo "WARNING: could not resolve pane PID for $AGENT_NAME — heartbeat sidecar not started" >&2
|
||||
# #1241. This branch used to print a WARNING about the heartbeat sidecar and
|
||||
# exit 0. It is not a heartbeat problem: tmux destroys a session when its pane
|
||||
# command exits, so an absent session one second after new-session means the
|
||||
# runtime died on startup. Reporting it as success is what let `fleet start`
|
||||
# return 0 over three dead panes — the launcher knew, and said the wrong thing
|
||||
# at the wrong severity to the wrong layer.
|
||||
fail_launch pane-did-not-survive \
|
||||
"the pane exited immediately and tmux destroyed the session;" \
|
||||
"run 'mosaic yolo ${MOSAIC_AGENT_RUNTIME}' in ${MOSAIC_AGENT_WORKDIR} to see why"
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user