fargo
9d3e22b1c1
ci/woodpecker/pr/ci Pipeline failed
Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.
Activation lands in the control plane, not the launcher:
- fleet install places ALL FOUR units through placeUnitFile — a placement
helper that unlinks any by-path-enable symlink at the destination
BEFORE copying (Node copyFile follows the link and overwrites the SEED
template; measured on a throwaway systemd user instance 2026-08-17,
with both cp and fs.copyFile), removes a stale wants-symlink pointing
outside the active dir (readlink — readFile returns the target's
content, not the link path), then copies and daemon-reloads. The same
measurement showed systemctl enable <name> does NOT rewrite an existing
by-path wants-symlink — reconciliation must be explicit. Idempotent:
second install on by-path residue converges to the identical state.
Until now the copy block named three units and omitted the broker, and
the residue set / copy set were disjoint only by accident (fomo-lin
survived copy-through because its one symlink was the one unit not
copied); adding the broker made them intersect on first run. See the
SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
lifecycle effect, then RE-CHECK the socket (not unit state) and exit
nonzero with a named code if it did not appear. Re-probed on every
invocation — a RemainAfterExit=yes dead-looking-active unit can never
make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
first-class member; the socket is the signal (enabled-but-dead units
report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
effect (moved ahead of the ownership probe): absent -> exit 75
(EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
pane. The agent@ unit is Type=oneshot with no Restart=, so the message
survives instead of looping. The preflight detects and refuses; it
never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
'mosaic fleet install (it reconciles either enable convention)' —
written from the measurement; teaching a manual systemctl line could
leave a host with competing wants-symlinks.
Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.
Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.
Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.
Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.
217 lines
9.0 KiB
Bash
Executable File
217 lines
9.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# CI-fit regression suite for the #1292 lease-broker socket preflight in
|
|
# start-agent-session.sh.
|
|
#
|
|
# WHY THIS SUITE IS CI-FIT WHERE test-start-agent-session.sh IS NOT (#1017/#1270
|
|
# context): that older suite's precondition is "the host does not have the pi
|
|
# binary", which a CI image that ships pi violates — its guard correctly
|
|
# refuses to report a pass there, so it is excluded from the chain. THIS suite
|
|
# controls its own preconditions instead of inheriting them from the host: a
|
|
# fake tmux on PATH, a fake mosaic on PATH, a real unix socket created in a
|
|
# tmpdir, a hermetic env (env -i, fake HOME, GIT_CONFIG_GLOBAL severed). It
|
|
# never depends on what the host has installed, so a green here means the same
|
|
# thing on every host. Anyone adding cases: keep that property — no case may
|
|
# depend on host state.
|
|
#
|
|
# The failure this suite is written down to catch (#1292): a seat launched on a
|
|
# host with no lease broker dies ~4 seconds in at registration, with the
|
|
# diagnostic invisible because tmux destroys the dead pane. The preflight runs
|
|
# BEFORE any tmux effect and refuses with a NAMED code (exit 75, EX_TEMPFAIL)
|
|
# so the message survives. The agent@ unit is Type=oneshot with no Restart=,
|
|
# so a failed unit keeps its output instead of looping.
|
|
#
|
|
# Cases:
|
|
# 1. absent socket -> exit 75, message names broker-absent + socket path +
|
|
# remedy, and NO tmux session was ever created (the doomed-pane half).
|
|
# 2. present socket (real unix socket in tmpdir) -> proceeds PAST the
|
|
# preflight (the suite then stops at the next precondition, proving the
|
|
# preflight was not the refusal).
|
|
# 3. explicit MOSAIC_LEASE_BROKER_SOCKET wins over XDG_RUNTIME_DIR default.
|
|
# 4. --stop mode does NOT require the broker (teardown must not be fenced on
|
|
# a component whose absence is exactly what teardown may follow).
|
|
#
|
|
# Sabotage control, run by the developer (not in-suite): remove the preflight
|
|
# block from start-agent-session.sh, re-run — case 1 fails (a tmux session is
|
|
# created / exit is not 75), cases 2-4 still pass; restore byte-identically.
|
|
|
|
set -euo pipefail
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/agent-session-broker-preflight}"
|
|
FAKE_HOME="$WORK_DIR/home"
|
|
BIN_DIR="$WORK_DIR/bin"
|
|
ENV_DIR="$WORK_DIR/env"
|
|
SOCK_DIR="$WORK_DIR/sockets"
|
|
LOG_FILE="$WORK_DIR/tmux-calls.log"
|
|
|
|
rm -rf "$WORK_DIR"
|
|
# The script asserts a managed directory tree under MOSAIC_HOME: mosaic/,
|
|
# mosaic/fleet/, mosaic/fleet/agents/ — private (0700/0750-style) modes, no
|
|
# symlinks — plus a per-agent env projection. Build the full tree the launcher
|
|
# expects so the suite reaches the BROKER preflight rather than dying at
|
|
# environment validation.
|
|
mkdir -p "$FAKE_HOME/.config/mosaic/fleet/agents" "$BIN_DIR" "$SOCK_DIR"
|
|
chmod 700 "$FAKE_HOME/.config/mosaic" "$FAKE_HOME/.config/mosaic/fleet/agents"
|
|
chmod 750 "$FAKE_HOME/.config/mosaic/fleet"
|
|
cat > "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated" <<'ENVEOF'
|
|
MOSAIC_AGENT_NAME=preflight-test
|
|
MOSAIC_AGENT_CLASS=worker
|
|
MOSAIC_AGENT_RUNTIME=pi
|
|
MOSAIC_AGENT_MODEL=
|
|
MOSAIC_AGENT_REASONING=
|
|
MOSAIC_AGENT_TOOL_POLICY=code
|
|
MOSAIC_AGENT_WORKDIR=/tmp
|
|
MOSAIC_TMUX_SOCKET=mosaic-fleet
|
|
ENVEOF
|
|
chmod 600 "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated"
|
|
|
|
# ─── Fake tmux: records every invocation; new-session marks the marker. ────
|
|
: > "$LOG_FILE"
|
|
cat > "$BIN_DIR/tmux" <<SH
|
|
#!/usr/bin/env bash
|
|
printf 'tmux %s\n' "\$*" >> "$LOG_FILE"
|
|
if [[ "\$*" == *new-session* ]]; then
|
|
echo "TMUX-NEW-SESSION-INVOKED" >> "$LOG_FILE"
|
|
fi
|
|
exit 0
|
|
SH
|
|
chmod +x "$BIN_DIR/tmux"
|
|
|
|
# ─── Fake mosaic/pi binaries so the script proceeds past its own lookups. ───
|
|
for bin in mosaic pi claude; do
|
|
printf '#!/usr/bin/env bash\nexit 0\n' > "$BIN_DIR/$bin"
|
|
chmod +x "$BIN_DIR/$bin"
|
|
done
|
|
|
|
# ─── Minimal launch environment the script expects. ────────────────────────
|
|
# (Enough for the preflight to be reached; later stages will still fail in
|
|
# case 2 — that is expected and asserted.)
|
|
run_session_script() {
|
|
local mode="$1"; shift
|
|
(
|
|
cd "$WORK_DIR"
|
|
env -i HOME="$FAKE_HOME" PATH="$BIN_DIR:/usr/bin:/bin" \
|
|
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null \
|
|
MOSAIC_HOME="$FAKE_HOME/.config/mosaic" \
|
|
AGENT_NAME=preflight-test \
|
|
"$@" \
|
|
bash "$SCRIPT_DIR/start-agent-session.sh" $mode preflight-test
|
|
)
|
|
}
|
|
|
|
fail=0
|
|
assert() {
|
|
local desc="$1" expected="$2" actual="$3"
|
|
if [[ "$expected" != "$actual" ]]; then
|
|
echo "FAIL: $desc — expected '$expected', got '$actual'" >&2
|
|
fail=1
|
|
fi
|
|
}
|
|
assert_contains() {
|
|
local desc="$1" haystack="$2" needle="$3"
|
|
[[ "$haystack" == *"$needle"* ]] || { echo "FAIL: $desc — missing '$needle' in: $haystack" >&2; fail=1; }
|
|
}
|
|
assert_not_contains() {
|
|
local desc="$1" haystack="$2" needle="$3"
|
|
if [[ "$haystack" == *"$needle"* ]]; then
|
|
echo "FAIL: $desc — must not contain '$needle'" >&2
|
|
fail=1
|
|
fi
|
|
return 0
|
|
}
|
|
|
|
# ─── 1. Absent socket → named refusal, NO tmux session. ────────────────────
|
|
: > "$LOG_FILE"
|
|
stderr_file="$WORK_DIR/stderr-1.tmp"
|
|
set +e
|
|
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent.sock" 2>"$stderr_file")
|
|
rc=$?
|
|
set -e
|
|
assert "absent socket exit code" "75" "$rc"
|
|
err=$(cat "$stderr_file")
|
|
assert_contains "absent socket names the failure" "$err" "FAIL_LAUNCH broker-absent"
|
|
assert_contains "absent socket names the socket path" "$err" "$SOCK_DIR/absent.sock"
|
|
assert_contains "absent socket names a remedy" "$err" "mosaic fleet install"
|
|
log1=$(cat "$LOG_FILE")
|
|
assert_not_contains "absent socket must not create a tmux session" "$log1" "TMUX-NEW-SESSION-INVOKED"
|
|
|
|
# ─── 2. Present socket → passes the preflight. ─────────────────────────────
|
|
# Expected: ownership/env checks AFTER the preflight may refuse (fixture is
|
|
# minimal by design); the assertion is only that the refusal is NOT
|
|
# broker-absent and the exit is NOT 75.
|
|
# Create a REAL unix socket: a detached python holder binds it and stays alive
|
|
# for the duration (bash cannot create sockets; a foreground python would
|
|
# close the socket on exit and -S on a closed-but-unlinked path fails). Written
|
|
# as a script file + setsid nohup so no job-control/heredoc interaction with
|
|
# set -e can silently kill the suite.
|
|
# AF_UNIX binds cap at 108 path bytes; the suite's workdir exceeds that, so
|
|
# the live socket lives at a SHORT path under /tmp (unique per run, cleaned
|
|
# with the suite). The preflight takes its socket path explicitly, so this
|
|
# stays fully controlled.
|
|
LIVE_SOCK=$(mktemp -u /tmp/mosaic-preflight-XXXXXX.sock)
|
|
trap 'rm -f "$LIVE_SOCK"' EXIT
|
|
rm -f "$SOCK_DIR/live.sock" "$LIVE_SOCK"
|
|
cat > "$SOCK_DIR/holder.py" <<'PY'
|
|
import socket, sys, time
|
|
path = sys.argv[1]
|
|
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
|
s.bind(path)
|
|
s.listen(1)
|
|
time.sleep(120)
|
|
PY
|
|
python3 "$SOCK_DIR/holder.py" "$LIVE_SOCK" >/dev/null 2>"$SOCK_DIR/holder.err" &
|
|
HOLDER_PID=$!
|
|
# Wait for the socket object to exist (bind is near-instant, but do not race it).
|
|
for _ in $(seq 1 50); do
|
|
[ -S "$LIVE_SOCK" ] && break
|
|
sleep 0.1
|
|
done
|
|
if [ ! -S "$LIVE_SOCK" ]; then
|
|
echo "FAIL: could not create live socket fixture (holder pid $HOLDER_PID)" >&2
|
|
ps -p "$HOLDER_PID" -o pid,stat,cmd --no-headers >&2 || echo "(holder exited)" >&2
|
|
cat "$SOCK_DIR/holder.err" >&2 || true
|
|
exit 1
|
|
fi
|
|
: > "$LOG_FILE"
|
|
set +e
|
|
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$LIVE_SOCK" 2>"$WORK_DIR/stderr-2.tmp")
|
|
rc=$?
|
|
set -e
|
|
# The preflight PASSED if the failure (whatever later stage refused) is NOT
|
|
# the broker refusal, and tmux was reached or a later precondition named
|
|
# something else.
|
|
err2=$(cat "$WORK_DIR/stderr-2.tmp")
|
|
assert_not_contains "live socket must not refuse broker-absent" "$err2" "broker-absent"
|
|
if [[ "$rc" == "75" ]]; then
|
|
echo "FAIL: live socket — preflight still refused (exit 75) with a live socket" >&2
|
|
fail=1
|
|
fi
|
|
|
|
# ─── 3. Explicit socket env wins over XDG default. ─────────────────────────
|
|
set +e
|
|
out=$(run_session_script "" XDG_RUNTIME_DIR="$SOCK_DIR/no-runtime-here" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent2.sock" 2>"$WORK_DIR/stderr-3.tmp")
|
|
rc=$?
|
|
set -e
|
|
assert "explicit env wins (exit 75)" "75" "$rc"
|
|
assert_contains "explicit env path named" "$(cat "$WORK_DIR/stderr-3.tmp")" "$SOCK_DIR/absent2.sock"
|
|
|
|
# ─── 4. --stop is not fenced on the broker. ────────────────────────────────
|
|
: > "$LOG_FILE"
|
|
set +e
|
|
out=$(run_session_script "--stop" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent3.sock" 2>"$WORK_DIR/stderr-4.tmp")
|
|
rc=$?
|
|
set -e
|
|
err4=$(cat "$WORK_DIR/stderr-4.tmp")
|
|
assert_not_contains "--stop must not refuse broker-absent" "$err4" "broker-absent"
|
|
if [[ "$rc" == "75" ]]; then
|
|
echo "FAIL: --stop — exit 75 means teardown was fenced on the broker" >&2
|
|
fail=1
|
|
fi
|
|
|
|
kill "$HOLDER_PID" 2>/dev/null || true
|
|
|
|
if [[ "$fail" -eq 0 ]]; then
|
|
echo "start-agent-session lease-broker preflight regression passed"
|
|
fi
|
|
exit "$fail"
|