fred
61a907a12f
ci/woodpecker/pr/ci Pipeline was successful
`mosaic fleet start` returned 0 over three dead panes. The launcher knew, and said the wrong thing at the wrong severity to the wrong layer. The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared environment. When that binary is absent the pane dies in under a second, tmux destroys the session, and the diagnostic goes with it. The launcher then found no PANE_PID, printed a WARNING about the *heartbeat sidecar*, and exited 0 — so systemd logged "Finished ... successfully" and `fleet start` reported success. `fleet ps` was the only component telling the truth. Two changes, both in start-agent-session.sh: 1. Before any effect, resolve `mosaic` and the roster's runtime against PANE_PATH — the pane's own view of the path, not the launcher's. `mosaic yolo <runtime>` calls checkRuntime(runtime) and looks for a binary named exactly like the runtime, so this asks the same question the pane will ask a moment later, while an operator can still see the answer. Absent binary -> exit 69, code=missing-binary, no session created. 2. Replace the dead-pane WARNING+exit-0. An absent session one second after new-session is a runtime that died on startup, not a heartbeat problem -> exit 69, code=pane-did-not-survive, with the command to run by hand to see why. A present session with no pane PID after five attempts -> code=pane-pid-unresolved. Neither branch kills the session; destroying a possibly-live pane on a guess is worse than leaving it for inspection. Exit 69 (EX_UNAVAILABLE) is deliberate: the 64s already in this file mean the projection was bad, and here the data is fine and the host is not ready. Callers separate the cases by `code=`, the same way fail_env's codes share 64. This propagates for free. `fleet start` calls runChecked() for the holder and each agent, and runChecked throws on non-zero, so layers 4 and 5 stop lying without a TypeScript change. Two adjacent defects are left for a follow-up issue rather than widened into this diff: the per-agent loop aborts on the first failure instead of attempting all and reporting an aggregate, and runChecked's bare throw surfaces the launcher's message under a Node unhandled-rejection stack trace because program.parse() is synchronous. Tests: - test-start-agent-session.sh gains three cases: `mosaic` absent from the pane path, the runtime absent from the pane path, and a pane that does not survive. Each was verified individually red against the unmodified origin/next launcher. - The two cases asserting a valid launch now supply a pane PID. Until now the suite's one success path was itself a dead pane the launcher reported as fine. - The harness fakes `npm` so PANE_PATH stops depending on whatever the host has installed, and fails loudly if the host provides `mosaic` or `pi` in the system path, where the missing-binary cases would not be measurable at all. - test-fleet-units.sh gains a `pi` shim in its runtime bin. The real-tmux harness named `pi` in its roster and never installed it; the new preflight caught it. Refs #1241
168 lines
8.4 KiB
Bash
Executable File
168 lines
8.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR=$(cd -- "$(dirname -- "$0")" && pwd)
|
|
HOLDER="$SCRIPT_DIR/mosaic-tmux-holder.service"
|
|
AGENT="$SCRIPT_DIR/[email protected]"
|
|
INTERACTION="$SCRIPT_DIR/[email protected]"
|
|
HOLDER_START="$SCRIPT_DIR/../../tools/fleet/start-tmux-holder.sh"
|
|
START_AGENT="$SCRIPT_DIR/../../tools/fleet/start-agent-session.sh"
|
|
|
|
fail() {
|
|
echo "FAIL: $*" >&2
|
|
exit 1
|
|
}
|
|
|
|
[ -f "$HOLDER" ] || fail "missing mosaic-tmux-holder.service"
|
|
[ -f "$AGENT" ] || fail "missing [email protected]"
|
|
[ -f "$INTERACTION" ] || fail "missing [email protected]"
|
|
[ -x "$HOLDER_START" ] || fail "missing executable start-tmux-holder.sh"
|
|
[ -x "$START_AGENT" ] || fail "missing executable start-agent-session.sh"
|
|
|
|
grep -qF 'ExecStart=' "$HOLDER" || fail "holder has no ExecStart"
|
|
grep -qF 'tmux -L' "$HOLDER" || fail "holder does not use named tmux socket"
|
|
grep -qF '_holder' "$HOLDER" || fail "holder session is not explicit"
|
|
grep -qF 'UnsetEnvironment=LD_PRELOAD BASH_ENV ENV' "$HOLDER" || \
|
|
fail "holder does not remove loader and shell-control variables"
|
|
grep -qF 'ExecStart=/usr/bin/env -i HOME=%h PATH=/usr/bin:/bin MOSAIC_TMUX_SOCKET=mosaic-fleet MOSAIC_TMUX_HOLDER=_holder /bin/bash --noprofile --norc %h/.config/mosaic/tools/fleet/start-tmux-holder.sh' "$HOLDER" || \
|
|
fail "holder does not clear manager environment before starting tmux"
|
|
grep -qF 'ExecStop=-/usr/bin/env -i HOME=%h PATH=/usr/bin:/bin MOSAIC_TMUX_SOCKET=mosaic-fleet /bin/bash --noprofile --norc -c' "$HOLDER" || \
|
|
fail "holder stop does not clear manager environment"
|
|
if grep -qF -- '/bin/bash -lc' "$HOLDER"; then
|
|
fail "holder must not start tmux through a login shell"
|
|
fi
|
|
grep -qF 'Requires=mosaic-tmux-holder.service' "$AGENT" || fail "agent does not require holder"
|
|
grep -qF 'start-agent-session.sh' "$AGENT" || fail "agent unit does not call start-agent-session.sh"
|
|
if grep -qE '^Environment(File)?=' "$AGENT" "$INTERACTION"; then
|
|
fail "agent units must not accept ambient or projection environment before strict parsing"
|
|
fi
|
|
grep -qF 'UnsetEnvironment=LD_PRELOAD BASH_ENV ENV' "$AGENT" || \
|
|
fail "agent unit does not remove loader and shell-control variables"
|
|
grep -qF 'UnsetEnvironment=LD_PRELOAD BASH_ENV ENV' "$INTERACTION" || \
|
|
fail "interaction unit does not remove loader and shell-control variables"
|
|
grep -qF 'ExecStart=/usr/bin/env -i HOME=%h MOSAIC_AGENT_NAME=%i PATH=/usr/bin:/bin /bin/bash --noprofile --norc' "$AGENT" || \
|
|
fail "agent unit does not clear bootstrap environment before strict parsing"
|
|
grep -qF 'start-agent-session.sh --stop %i' "$AGENT" || \
|
|
fail "agent stop does not use the validated exact-stop path"
|
|
grep -qF 'Requires=mosaic-tmux-holder.service' "$INTERACTION" || fail "interaction service does not require holder"
|
|
grep -qF 'ExecStart=/usr/bin/env -i HOME=%h MOSAIC_AGENT_NAME=%i PATH=/usr/bin:/bin /bin/bash --noprofile --norc' "$INTERACTION" || \
|
|
fail "interaction unit does not clear bootstrap environment before strict parsing"
|
|
grep -qF 'start-interaction-service.sh %i' "$INTERACTION" || fail "interaction service does not use shared strict parsing"
|
|
grep -qF 'start-agent-session.sh --stop %i' "$INTERACTION" || \
|
|
fail "interaction stop does not use the validated exact-stop path"
|
|
|
|
if command -v systemd-analyze >/dev/null 2>&1; then
|
|
systemd-analyze verify --user "$HOLDER" "$AGENT" "$INTERACTION" >/tmp/mosaic-fleet-systemd-verify.log 2>&1 || {
|
|
cat /tmp/mosaic-fleet-systemd-verify.log >&2
|
|
fail "systemd-analyze verify failed"
|
|
}
|
|
fi
|
|
|
|
# Real isolated socket regression: a preexisting server with an LD_PRELOAD
|
|
# constructor marker must fail closed, while a fresh named server is created.
|
|
if command -v tmux >/dev/null 2>&1 && command -v cc >/dev/null 2>&1; then
|
|
TEST_ROOT=$(mktemp -d)
|
|
TEST_SOCKET="mosaic-holder-test-$$"
|
|
trap 'tmux -L "$TEST_SOCKET" kill-server >/dev/null 2>&1 || true; rm -rf "$TEST_ROOT"' EXIT
|
|
MARKER="$TEST_ROOT/loader-marker"
|
|
LIBRARY="$TEST_ROOT/marker.so"
|
|
HOLDER_HOME="$TEST_ROOT/holder-home"
|
|
mkdir -p "$HOLDER_HOME/.config/mosaic/fleet/run"
|
|
chmod 700 "$HOLDER_HOME/.config" "$HOLDER_HOME/.config/mosaic" \
|
|
"$HOLDER_HOME/.config/mosaic/fleet" "$HOLDER_HOME/.config/mosaic/fleet/run"
|
|
printf '123e4567-e89b-12d3-a456-426614174000\n' > \
|
|
"$HOLDER_HOME/.config/mosaic/fleet/run/holder-owner"
|
|
chmod 600 "$HOLDER_HOME/.config/mosaic/fleet/run/holder-owner"
|
|
cat > "$TEST_ROOT/marker.c" <<'EOF'
|
|
#include <fcntl.h>
|
|
#include <stdlib.h>
|
|
#include <unistd.h>
|
|
__attribute__((constructor)) static void mark_loader(void) {
|
|
const char *path = getenv("MOSAIC_LOADER_MARKER");
|
|
if (path != NULL) {
|
|
int fd = open(path, O_WRONLY | O_CREAT | O_APPEND, 0600);
|
|
if (fd >= 0) { write(fd, "loaded\\n", 7); close(fd); }
|
|
}
|
|
}
|
|
EOF
|
|
cc -shared -fPIC -o "$LIBRARY" "$TEST_ROOT/marker.c"
|
|
MOSAIC_LOADER_MARKER="$MARKER" LD_PRELOAD="$LIBRARY" \
|
|
tmux -L "$TEST_SOCKET" new-session -d -s _holder 'sleep 60'
|
|
[ -s "$MARKER" ] || fail "contaminated fixture did not execute loader constructor"
|
|
server_pid=$(tmux -L "$TEST_SOCKET" display-message -p '#{pid}')
|
|
: > "$MARKER"
|
|
if /usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin \
|
|
MOSAIC_TMUX_SOCKET="$TEST_SOCKET" MOSAIC_TMUX_HOLDER=_holder "$HOLDER_START" \
|
|
>"$TEST_ROOT/holder.out" 2>&1; then
|
|
fail "holder adopted contaminated named server"
|
|
fi
|
|
grep -qF 'global environment does not match the owned-server contract' "$TEST_ROOT/holder.out" || \
|
|
fail "holder did not report contaminated server environment"
|
|
[ "$(tmux -L "$TEST_SOCKET" display-message -p '#{pid}')" = "$server_pid" ] || \
|
|
fail "holder replaced a contaminated server instead of failing closed"
|
|
[ ! -s "$MARKER" ] || fail "holder execution triggered a contaminated loader"
|
|
|
|
# Agent validation must reject the same unmanaged server without cleaning its
|
|
# global environment or adding a managed session.
|
|
AGENT_HOME="$HOLDER_HOME/.config/mosaic"
|
|
AGENT_NAME=loader-safe
|
|
AGENT_WORKDIR="$AGENT_HOME/work"
|
|
AGENT_BIN="$TEST_ROOT/agent-bin"
|
|
mkdir -p "$AGENT_HOME/fleet/agents" "$AGENT_WORKDIR" "$AGENT_BIN"
|
|
chmod 700 "$AGENT_HOME/fleet/agents"
|
|
cat > "$AGENT_HOME/fleet/agents/$AGENT_NAME.env.generated" <<EOF
|
|
MOSAIC_AGENT_NAME=$AGENT_NAME
|
|
MOSAIC_AGENT_CLASS=code
|
|
MOSAIC_AGENT_RUNTIME=pi
|
|
MOSAIC_AGENT_MODEL=
|
|
MOSAIC_AGENT_REASONING=
|
|
MOSAIC_AGENT_TOOL_POLICY=code
|
|
MOSAIC_AGENT_WORKDIR=$AGENT_WORKDIR
|
|
MOSAIC_TMUX_SOCKET=$TEST_SOCKET
|
|
EOF
|
|
printf 'MOSAIC_RUNTIME_BIN=%s\n' "$AGENT_BIN" > "$AGENT_HOME/fleet/agents/$AGENT_NAME.env.local"
|
|
chmod 600 "$AGENT_HOME/fleet/agents/$AGENT_NAME.env.generated" \
|
|
"$AGENT_HOME/fleet/agents/$AGENT_NAME.env.local"
|
|
cat > "$AGENT_BIN/mosaic" <<'EOF'
|
|
#!/bin/sh
|
|
sleep 30
|
|
EOF
|
|
chmod 700 "$AGENT_BIN/mosaic"
|
|
# The launcher resolves the roster's runtime against PANE_PATH before it
|
|
# spawns anything (#1241), so the runtime this projection names has to be
|
|
# present here even though the fake `mosaic` above never execs it.
|
|
cat > "$AGENT_BIN/pi" <<'EOF'
|
|
#!/bin/sh
|
|
sleep 30
|
|
EOF
|
|
chmod 700 "$AGENT_BIN/pi"
|
|
server_environment_before=$(tmux -L "$TEST_SOCKET" show-environment -g | sort)
|
|
server_sessions_before=$(tmux -L "$TEST_SOCKET" list-sessions | sort)
|
|
if /usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin MOSAIC_HOME="$AGENT_HOME" \
|
|
"$START_AGENT" "$AGENT_NAME" >"$TEST_ROOT/agent.out" 2>&1; then
|
|
fail "agent launcher adopted contaminated named server"
|
|
fi
|
|
[ "$(tmux -L "$TEST_SOCKET" display-message -p '#{pid}')" = "$server_pid" ] || \
|
|
fail "agent launcher changed unmanaged server PID"
|
|
[ "$(tmux -L "$TEST_SOCKET" show-environment -g | sort)" = "$server_environment_before" ] || \
|
|
fail "agent launcher changed unmanaged global environment"
|
|
[ "$(tmux -L "$TEST_SOCKET" list-sessions | sort)" = "$server_sessions_before" ] || \
|
|
fail "agent launcher changed unmanaged sessions"
|
|
tmux -L "$TEST_SOCKET" kill-server
|
|
/usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin \
|
|
MOSAIC_TMUX_SOCKET="$TEST_SOCKET" MOSAIC_TMUX_HOLDER=_holder "$HOLDER_START"
|
|
tmux -L "$TEST_SOCKET" has-session -t '=_holder:0.0' || fail "fresh holder was not created"
|
|
if tmux -L "$TEST_SOCKET" show-environment -g LD_PRELOAD 2>/dev/null | grep -q '^LD_PRELOAD='; then
|
|
fail "fresh holder retained LD_PRELOAD"
|
|
fi
|
|
/usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin MOSAIC_HOME="$AGENT_HOME" \
|
|
"$START_AGENT" "$AGENT_NAME"
|
|
tmux -L "$TEST_SOCKET" has-session -t "=$AGENT_NAME:0.0" || \
|
|
fail "agent did not launch on a valid owned server"
|
|
tmux -L "$TEST_SOCKET" kill-server
|
|
trap - EXIT
|
|
rm -rf "$TEST_ROOT"
|
|
fi
|
|
|
|
echo "ok - fleet systemd unit templates"
|