Compare commits

..
Author SHA1 Message Date
veronica 0ca192baaa fix(tmux): split the rule pair without piping into head
ci/woodpecker/pr/ci Pipeline was successful
ci/woodpecker/manual/ci Pipeline was successful
CI 2604 red on scripts/pipefail-early-exit.test.mjs. Two lines I added in the
previous commit tripped its scan of send-message.sh:

  [ "$(printf '%s\n' "$rule_lines" | grep -c .)" -eq 2 ] || return 1
  top=$(printf '%s\n' "$rule_lines" | head -1)

The second is a real violation of the rule the guard enforces: `head` exits
after its count, the producer takes SIGPIPE, and under `set -euo pipefail` the
caller aborts with rc=141 and no output. That is the same failure mode
test-mosaic-worktree-large-repo.sh exists to pin.

The first is the guard reading `-eq` as an early-exit grep flag
(`-[A-Za-z]*q` matches `-eq`), so `grep ... | ... -eq 2` looks like `grep -q`
on one line. A false positive, but the fix removes the shape either way.

Both go away by splitting the captured line numbers with parameter expansion
instead of a second pass through the pipe. With one rule captured, both halves
resolve to the same value and the new `[ "$top" != "$bottom" ]` test rejects it,
which is the answer that case wanted anyway.

Verified: node --test scripts/pipefail-early-exit.test.mjs green on the
load-bearing assertion; test-send-message-verdict.sh still PASS=5 FAIL=0;
test-send-message-socket.sh rc=0; agent-send.test.sh PASS=19 FAIL=0.
2026-08-21 17:10:40 -05:00
veronica dc6db5f14e fix(tmux): locate the REPL input box by shape, not by a Claude-only glyph
ci/woodpecker/pr/ci Pipeline failed
send-message.sh confirmed delivery by grepping the captured pane for a
prompt glyph (`❯`, a leading `>`, or `│ >`). Those are Claude Code shapes.
pi draws its input box as two horizontal `─` rules with the input between
them and no glyph anywhere, so on a pi pane the grep matched nothing, the
`delivered` and `draft` branches were unreachable, and every send exited 2
with "may be UNDELIVERED" after burning all its retries — on messages that
had in fact been delivered.

Measured on five live seats: pi seats vision, tess and medic returned 0
glyph matches on non-empty captures (3653 / 1086 / 1890 bytes); Claude
seats fred and tuesday returned 3 and 1.

This is a different arm from #1257. On the measured host
`tmux capture-pane -t <session> -p` and `-t <session>:0.0 -p` both return
the same 3653 bytes, so #1257's empty-capture mechanism does not reproduce
here; fixing only that would leave every pi send at rc=2 and would verify
green against a Claude pane.

The probe becomes locate_input_box(), which returns a status code instead
of a string. Found-but-empty is a real answer (an empty input box is what
a submitted message leaves behind) and is now distinguishable from
not-found, which the old `[ -z "$promptline" ]` test conflated. Box
detection anchors on the LAST pair of `─` rules: agent output can contain
its own rules, but nothing is drawn below the input box except the status
line.

Adding a runtime means adding its shape in that one function. A missing
shape does not degrade gracefully — it turns every send to that runtime
into a false "may be UNDELIVERED", which is exactly what this fixes.

Tests: two fixtures added to test-send-message-verdict.sh driving a
glyphless box-drawn REPL, one that submits (expect exit 0 delivered) and
one that holds the text in the box (expect exit 2 draft). Patched suite
PASS=5 FAIL=0. Control against the pristine origin/next send-message.sh:
PASS=3 FAIL=2, fixture 4 failing as `rc=2 ... REPL input prompt not
locatable after 3 attempts — message may be UNDELIVERED`, which is the
defect reproduced in a test. Sibling suites unchanged and green:
test-send-message-socket.sh rc=0, agent-send.test.sh PASS=19 FAIL=0.

Note the suite is in test-enumeration-exclusions.txt (the CI image ships
no tmux), so these fixtures are verified locally only. That exclusion and
its #1017 burndown reason are unchanged by this commit.

Not addressed here: auto-submit-drafts.sh lines 20 and 26 carry the same
Claude-only `❯` assumption. Separate surface, separate change.

Closes #1362
2026-08-21 16:54:39 -05:00
fred 24462f460e docs(git-wrappers): correct the stale --login fallback and document the test suite's identity split (#1354)
ci/woodpecker/push/publish Pipeline was successful
2026-08-21 14:42:36 +00:00
veronicaandfred a480ee83dc docs(W4): document contract — stamp kind and status on 104 live docs (#1350)
ci/woodpecker/push/publish Pipeline was canceled
Co-authored-by: veronica <[email protected]>
2026-08-21 14:07:36 +00:00
fred fd43ed5420 fix(git-wrappers): no --login must use the caller's own credential, not a guessed shared login (#1352)
ci/woodpecker/push/publish Pipeline was successful
2026-08-21 04:10:42 +00:00
fargoandfred 1d84bc3f3d fix(fleet): lease-broker activation, symlink-safe unit placement, named launch refusal — Wall 6 (#1292) (#1297)
ci/woodpecker/push/publish Pipeline was successful
Co-authored-by: fargo <[email protected]>
2026-08-21 00:40:29 +00:00
16 changed files with 1414 additions and 85 deletions
@@ -257,8 +257,36 @@ assert_owned_tmux_server() {
fail "tmux server ownership or environment validation failed" fail "tmux server ownership or environment validation failed"
} }
# Validate exact server ownership before querying, cleaning, or creating any # Lease-broker socket preflight (#1292). The gated runtime (`mosaic yolo …` →
# managed session. An unmanaged or contaminated named socket is never repaired. # launch-runtime.py) registers with the broker or dies ~4 seconds in, with the
# diagnostic invisible because tmux destroys the dead pane. This check runs
# BEFORE any tmux effect — including the ownership probe below — so a host
# without a broker produces a named, surviving refusal instead of a doomed
# pane. Exit 75 (EX_TEMPFAIL), distinct from 64 (bad projection) and 69 (host
# not ready for other reasons); the agent@ unit is Type=oneshot with no
# Restart=, so the failed unit keeps its message instead of looping. Socket
# resolution matches launch.ts's defaultLeaseBrokerSocket precedence exactly.
# This preflight DETECTS and REFUSES — it never starts the broker (activation
# belongs to the fleet control plane; a component that both detects and fixes
# cannot be used to measure whether the fix worked).
broker_socket_path() {
if [ -n "${MOSAIC_LEASE_BROKER_SOCKET:-}" ]; then
printf '%s\n' "$MOSAIC_LEASE_BROKER_SOCKET"
return 0
fi
local runtime_dir="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
printf '%s\n' "${runtime_dir}/mosaic-lease/broker.sock"
}
if [ "$MODE" = "launch" ]; then
_broker_socket=$(broker_socket_path)
if [ ! -S "$_broker_socket" ]; then
echo "[fleet] FAIL_LAUNCH broker-absent: lease broker socket ${_broker_socket} missing; runtime launch denied (#1292)." >&2
echo "[fleet] remedy: systemctl --user enable --now mosaic-lease-broker.service (or reinstall via: mosaic fleet install)" >&2
exit 75
fi
fi
assert_owned_tmux_server assert_owned_tmux_server
if [ "$MODE" = interaction ]; then if [ "$MODE" = interaction ]; then
@@ -0,0 +1,222 @@
#!/usr/bin/env bash
# CI-fit regression suite for the #1292 lease-broker socket preflight in
# start-agent-session.sh.
#
# WHY THIS SUITE IS CI-FIT WHERE test-start-agent-session.sh IS NOT (#1017/#1270
# context): that older suite's precondition is "the host does not have the pi
# binary", which a CI image that ships pi violates — its guard correctly
# refuses to report a pass there, so it is excluded from the chain. THIS suite
# controls its own preconditions instead of inheriting them from the host: a
# fake tmux on PATH, a fake mosaic on PATH, a real unix socket created in a
# tmpdir, a hermetic env (env -i, fake HOME, GIT_CONFIG_GLOBAL severed). It
# never depends on what the host has installed, so a green here means the same
# thing on every host. Anyone adding cases: keep that property — no case may
# depend on host state.
#
# The failure this suite is written down to catch (#1292): a seat launched on a
# host with no lease broker dies ~4 seconds in at registration, with the
# diagnostic invisible because tmux destroys the dead pane. The preflight runs
# BEFORE any tmux effect and refuses with a NAMED code (exit 75, EX_TEMPFAIL)
# so the message survives. The agent@ unit is Type=oneshot with no Restart=,
# so a failed unit keeps its output instead of looping.
#
# Cases:
# 1. absent socket -> exit 75, message names broker-absent + socket path +
# remedy, and NO tmux session was ever created (the doomed-pane half).
# 2. present socket (real unix socket in tmpdir) -> proceeds PAST the
# preflight (the suite then stops at the next precondition, proving the
# preflight was not the refusal).
# 3. explicit MOSAIC_LEASE_BROKER_SOCKET wins over XDG_RUNTIME_DIR default.
# 4. --stop mode does NOT require the broker (teardown must not be fenced on
# a component whose absence is exactly what teardown may follow).
#
# Sabotage control, run by the developer (not in-suite): remove the preflight
# block from start-agent-session.sh, re-run — case 1 fails (a tmux session is
# created / exit is not 75), cases 2-4 still pass; restore byte-identically.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/agent-session-broker-preflight}"
FAKE_HOME="$WORK_DIR/home"
BIN_DIR="$WORK_DIR/bin"
ENV_DIR="$WORK_DIR/env"
SOCK_DIR="$WORK_DIR/sockets"
LOG_FILE="$WORK_DIR/tmux-calls.log"
rm -rf "$WORK_DIR"
# The script asserts a managed directory tree under MOSAIC_HOME: mosaic/,
# mosaic/fleet/, mosaic/fleet/agents/ — private (0700/0750-style) modes, no
# symlinks — plus a per-agent env projection. Build the full tree the launcher
# expects so the suite reaches the BROKER preflight rather than dying at
# environment validation.
mkdir -p "$FAKE_HOME/.config/mosaic/fleet/agents" "$BIN_DIR" "$SOCK_DIR"
chmod 700 "$FAKE_HOME/.config/mosaic" "$FAKE_HOME/.config/mosaic/fleet/agents"
chmod 750 "$FAKE_HOME/.config/mosaic/fleet"
cat > "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated" <<'ENVEOF'
MOSAIC_AGENT_NAME=preflight-test
MOSAIC_GIT_IDENTITY=preflight-test
MOSAIC_AGENT_CLASS=worker
MOSAIC_AGENT_RUNTIME=pi
MOSAIC_AGENT_MODEL=
MOSAIC_AGENT_REASONING=
MOSAIC_AGENT_TOOL_POLICY=code
MOSAIC_AGENT_WORKDIR=/tmp
MOSAIC_TMUX_SOCKET=mosaic-fleet
ENVEOF
chmod 600 "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated"
# ─── Fake tmux: records every invocation; new-session marks the marker. ────
: > "$LOG_FILE"
cat > "$BIN_DIR/tmux" <<SH
#!/usr/bin/env bash
printf 'tmux %s\n' "\$*" >> "$LOG_FILE"
if [[ "\$*" == *new-session* ]]; then
echo "TMUX-NEW-SESSION-INVOKED" >> "$LOG_FILE"
fi
exit 0
SH
chmod +x "$BIN_DIR/tmux"
# ─── Fake mosaic/pi binaries so the script proceeds past its own lookups. ───
for bin in mosaic pi claude; do
printf '#!/usr/bin/env bash\nexit 0\n' > "$BIN_DIR/$bin"
chmod +x "$BIN_DIR/$bin"
done
# ─── Minimal launch environment the script expects. ────────────────────────
# (Enough for the preflight to be reached; later stages will still fail in
# case 2 — that is expected and asserted.)
run_session_script() {
local mode="$1"; shift
(
cd "$WORK_DIR"
env -i HOME="$FAKE_HOME" PATH="$BIN_DIR:/usr/bin:/bin" \
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null \
MOSAIC_HOME="$FAKE_HOME/.config/mosaic" \
AGENT_NAME=preflight-test \
"$@" \
bash "$SCRIPT_DIR/start-agent-session.sh" $mode preflight-test
)
}
fail=0
assert() {
local desc="$1" expected="$2" actual="$3"
if [[ "$expected" != "$actual" ]]; then
echo "FAIL: $desc — expected '$expected', got '$actual'" >&2
fail=1
fi
}
assert_contains() {
local desc="$1" haystack="$2" needle="$3"
[[ "$haystack" == *"$needle"* ]] || { echo "FAIL: $desc — missing '$needle' in: $haystack" >&2; fail=1; }
}
assert_not_contains() {
local desc="$1" haystack="$2" needle="$3"
if [[ "$haystack" == *"$needle"* ]]; then
echo "FAIL: $desc — must not contain '$needle'" >&2
fail=1
fi
return 0
}
# ─── 1. Absent socket → named refusal, NO tmux session. ────────────────────
: > "$LOG_FILE"
stderr_file="$WORK_DIR/stderr-1.tmp"
set +e
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent.sock" 2>"$stderr_file")
rc=$?
set -e
assert "absent socket exit code" "75" "$rc"
err=$(cat "$stderr_file")
assert_contains "absent socket names the failure" "$err" "FAIL_LAUNCH broker-absent"
assert_contains "absent socket names the socket path" "$err" "$SOCK_DIR/absent.sock"
assert_contains "absent socket names a remedy" "$err" "mosaic fleet install"
log1=$(cat "$LOG_FILE")
assert_not_contains "absent socket must not create a tmux session" "$log1" "TMUX-NEW-SESSION-INVOKED"
# ─── 2. Present socket → passes the preflight. ─────────────────────────────
# Expected: ownership/env checks AFTER the preflight may refuse (fixture is
# minimal by design); the assertion is only that the refusal is NOT
# broker-absent and the exit is NOT 75.
# Create a REAL unix socket: a detached python holder binds it and stays alive
# for the duration (bash cannot create sockets; a foreground python would
# close the socket on exit and -S on a closed-but-unlinked path fails). Written
# as a script file + setsid nohup so no job-control/heredoc interaction with
# set -e can silently kill the suite.
# AF_UNIX binds cap at 108 path bytes; the suite's workdir exceeds that, so
# the live socket lives at a SHORT path under /tmp (unique per run, cleaned
# with the suite). The preflight takes its socket path explicitly, so this
# stays fully controlled.
# A real unix socket at a SHORT absolute path (AF_UNIX limit is 108 bytes,
# so the repo-deep SOCK_DIR cannot host it). The name is composed, not
# `mktemp -u`: the CI image's mktemp dialect rejects that invocation
# (pipeline 2562: "mktemp: : Invalid argument"), and no pre-existing file is
# wanted anyway — the holder binds it fresh.
LIVE_SOCK="/tmp/mosaic-preflight-$RANDOM-$$.sock"
trap 'rm -f "$LIVE_SOCK"' EXIT
rm -f "$SOCK_DIR/live.sock" "$LIVE_SOCK"
cat > "$SOCK_DIR/holder.py" <<'PY'
import socket, sys, time
path = sys.argv[1]
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.bind(path)
s.listen(1)
time.sleep(120)
PY
python3 "$SOCK_DIR/holder.py" "$LIVE_SOCK" >/dev/null 2>"$SOCK_DIR/holder.err" &
HOLDER_PID=$!
# Wait for the socket object to exist (bind is near-instant, but do not race it).
for _ in $(seq 1 50); do
[ -S "$LIVE_SOCK" ] && break
sleep 0.1
done
if [ ! -S "$LIVE_SOCK" ]; then
echo "FAIL: could not create live socket fixture (holder pid $HOLDER_PID)" >&2
ps -p "$HOLDER_PID" -o pid,stat,cmd --no-headers >&2 || echo "(holder exited)" >&2
cat "$SOCK_DIR/holder.err" >&2 || true
exit 1
fi
: > "$LOG_FILE"
set +e
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$LIVE_SOCK" 2>"$WORK_DIR/stderr-2.tmp")
rc=$?
set -e
# The preflight PASSED if the failure (whatever later stage refused) is NOT
# the broker refusal, and tmux was reached or a later precondition named
# something else.
err2=$(cat "$WORK_DIR/stderr-2.tmp")
assert_not_contains "live socket must not refuse broker-absent" "$err2" "broker-absent"
if [[ "$rc" == "75" ]]; then
echo "FAIL: live socket — preflight still refused (exit 75) with a live socket" >&2
fail=1
fi
# ─── 3. Explicit socket env wins over XDG default. ─────────────────────────
set +e
out=$(run_session_script "" XDG_RUNTIME_DIR="$SOCK_DIR/no-runtime-here" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent2.sock" 2>"$WORK_DIR/stderr-3.tmp")
rc=$?
set -e
assert "explicit env wins (exit 75)" "75" "$rc"
assert_contains "explicit env path named" "$(cat "$WORK_DIR/stderr-3.tmp")" "$SOCK_DIR/absent2.sock"
# ─── 4. --stop is not fenced on the broker. ────────────────────────────────
: > "$LOG_FILE"
set +e
out=$(run_session_script "--stop" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent3.sock" 2>"$WORK_DIR/stderr-4.tmp")
rc=$?
set -e
err4=$(cat "$WORK_DIR/stderr-4.tmp")
assert_not_contains "--stop must not refuse broker-absent" "$err4" "broker-absent"
if [[ "$rc" == "75" ]]; then
echo "FAIL: --stop — exit 75 means teardown was fenced on the broker" >&2
fail=1
fi
kill "$HOLDER_PID" 2>/dev/null || true
if [[ "$fail" -eq 0 ]]; then
echo "start-agent-session lease-broker preflight regression passed"
fi
exit "$fail"
+29 -1
View File
@@ -30,7 +30,9 @@ The Gitea API token is **never passed on a curl command line.** An `Authorizatio
### `--login` override ### `--login` override
Both `pr-review.sh` and `issue-comment.sh` accept an optional `--login <name>` flag that overrides the automatically detected Gitea login for that single invocation. The override selects **which credential the REST write, the `/user` identity lookup, and the read-back all use** — its token is resolved from the tea config for that login name (`get_gitea_token_for_login`), falling back to the repo host's credential when no login is named. The resolved login is **host- and port-bound**: the login's configured URL host **and effective port** (the scheme's default port — 80 for `http`, 443 for `https` — applies when a port is omitted, symmetrically on both sides) must match the repo remote's, so a login name shared across hosts (or an override configured for a different Gitea, including one on a different port of the same host) can never send one host's credential to another — a host or port mismatch fails closed rather than leaking a cross-host token. Resolving the acting identity and the read-back from the _same_ login that performs the write is essential: a write performed under an overridden login must be verified against that login's identity, not the host default's. Callers who need a different login than the host default should pass `--login <reviewer-login>`. Both `pr-review.sh` and `issue-comment.sh` accept an optional `--login <name>` flag that overrides the automatically detected Gitea login for that single invocation. The override selects **which credential the REST write, the `/user` identity lookup, and the read-back all use** — its token is resolved from the tea config for that login name (`get_gitea_token_for_login`).
**With no `--login`, there is no tea lookup at all.** The acting credential is the calling identity's own, resolved by `get_gitea_token` (see "Per-agent Gitea identity" below), and there is deliberately no fallback from it. These wrappers previously _guessed_ a login from the repo host and looked that guess up in the tea config; on a shared-account host the guess resolved to the shared login, so an unqualified call authored its write as that account rather than as the caller. Since `get_gitea_token_for_login` matches by login **name** and performs no authentication check, a dead shared credential still resolved at rc=0 and the identity-aware resolver was never reached. A caller passing no `--login` is asking to act as itself, so `--login` is now the only route to the tea store (#1351). The resolved login is **host- and port-bound**: the login's configured URL host **and effective port** (the scheme's default port — 80 for `http`, 443 for `https` — applies when a port is omitted, symmetrically on both sides) must match the repo remote's, so a login name shared across hosts (or an override configured for a different Gitea, including one on a different port of the same host) can never send one host's credential to another — a host or port mismatch fails closed rather than leaking a cross-host token. Resolving the acting identity and the read-back from the _same_ login that performs the write is essential: a write performed under an overridden login must be verified against that login's identity, not the host default's. Callers who need a different login than the host default should pass `--login <reviewer-login>`.
As a durable successor to this mechanism, consider giving each reviewer/approver slot its own dedicated Gitea login credential, so that author≠reviewer holds at the credential level rather than relying on wrapper-level `--login` bookkeeping. This is a recommendation for future hardening, not something implemented by this flag. As a durable successor to this mechanism, consider giving each reviewer/approver slot its own dedicated Gitea login credential, so that author≠reviewer holds at the credential level rather than relying on wrapper-level `--login` bookkeeping. This is a recommendation for future hardening, not something implemented by this flag.
@@ -125,6 +127,32 @@ otherwise careful never to touch. Because identity is already resolved per-workt
(`mosaic.gitIdentity`), the correct granularity for registering the helper is per-clone (`mosaic.gitIdentity`), the correct granularity for registering the helper is per-clone
too, so a documented manual step is the right shape here, not a global auto-write. too, so a documented manual step is the right shape here, not a global auto-write.
### Running these tests
`MOSAIC_GIT_IDENTITY` is inherited into each test's sandbox `HOME`, and **the tests disagree
about which value they need**, so no single ambient value passes all 29. Measured on `next` at
`a480ee83`, two full passes differing only in that variable:
| tests | identity exported | identity unset |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | -------------- |
| `gitea-login-resolution`, `issue-comment-readback`, `issue-create-interactive-auth`, `pr-edit`, `pr-merge-gitea-empty-uid`, `pr-metadata-gitea` | **fail** | pass |
| `issue-close-fail-closed` | pass | **fail** |
| remaining 22 | pass | pass |
The six fail because inside a sandbox `HOME` the identity has no `fleet/agents/<id>/`
directory, so it is classified as a **service identity**, its store is unpopulated, and the
resolver correctly refuses with `Refusing to borrow another slot's token`. That is the
documented fail-closed behaviour above, reached from a state the test never intended.
`issue-close-fail-closed` is the mirror image: it asserts that no comment POST is attempted, so
it needs an identity resolving to an empty slot, and with the variable unset the shared account
answers and the POST goes through.
These read as wrapper regressions rather than as environment. Two seats independently
misdiagnosed them as a patch defect while reviewing #1352. Until each test controls its own
value (#1353), `env -u MOSAIC_GIT_IDENTITY` is the closest thing to a clean run at 28/29, with
`issue-close-fail-closed` the expected failure — and **"the suite passes" is not a statement
anyone can make here without naming the ambient value that produced it.**
### PowerShell parity ### PowerShell parity
`detect-platform.ps1`'s Gitea wrappers authenticate through `tea` CLI logins `detect-platform.ps1`'s Gitea wrappers authenticate through `tea` CLI logins
@@ -102,9 +102,17 @@ gitea_resolve_api_for_login() {
return 1 return 1
} }
else else
GITEA_API_TOKEN=$(get_gitea_token_for_login "$effective_login" "$host") \ # NO --login: the acting credential is this identity's own token and there
|| GITEA_API_TOKEN=$(get_gitea_token "$host") || { # is deliberately no tea-config fallback. get_gitea_token_for_login matches
echo "Error: Gitea token not found for login '$effective_login' (comment write/read-back)" >&2 # by login NAME and performs no authentication check, and with no --login
# that name was a HOST GUESS resolving to a shared account. A live shared
# token would therefore have authored every seat's comment as that
# account, making Gate-16 author-is-not-reviewer unenforceable fleet-wide;
# a dead one is only what made the defect visible. get_gitea_token fails
# loud on a fleet host when no identity resolves, and that refusal is the
# correct outcome, not a case to fall back from.
GITEA_API_TOKEN=$(get_gitea_token "$host") || {
echo "Error: no Gitea credential resolved for the acting identity on host '$host' (comment write/read-back). Set MOSAIC_GIT_IDENTITY=<agent-id>, or pass --login <name> to use a named tea credential." >&2
return 1 return 1
} }
fi fi
@@ -335,15 +343,12 @@ if [[ "$PLATFORM" == "github" ]]; then
gh issue comment "$ISSUE_NUMBER" --body "$COMMENT" gh issue comment "$ISSUE_NUMBER" --body "$COMMENT"
echo "Added comment to GitHub issue #$ISSUE_NUMBER" echo "Added comment to GitHub issue #$ISSUE_NUMBER"
elif [[ "$PLATFORM" == "gitea" ]]; then elif [[ "$PLATFORM" == "gitea" ]]; then
# Resolve the login this comment should be attributed to: the --login # A --login override selects a NAMED tea credential and is the only way to
# override when given, otherwise the detected default for this repo's host. # reach the tea store. With no --login there is deliberately no guess: the
# A --login override always wins. Otherwise name this repo host's login only # comment is attributed to this identity's own credential, resolved by
# as a best effort: the login name merely selects a per-login token, and # gitea_resolve_api_for_login. The guess this replaced named a SHARED
# gitea_resolve_api_for_login falls back to the host credential # account, selecting an identity the caller never asked to act as.
# (get_gitea_token) when no tea login is named, so the default credential
# still resolves even when the host tea has no matching login entry.
EFFECTIVE_LOGIN="$LOGIN_OVERRIDE" EFFECTIVE_LOGIN="$LOGIN_OVERRIDE"
[[ -n "$EFFECTIVE_LOGIN" ]] || EFFECTIVE_LOGIN=$(get_gitea_login 2>/dev/null || true)
# Bind the REST endpoint + token to the effective login, then derive the # Bind the REST endpoint + token to the effective login, then derive the
# acting identity from that SAME credential (GET /user). The write below and # acting identity from that SAME credential (GET /user). The write below and
@@ -12,7 +12,7 @@
# concurrent record cannot masquerade as this write and a no-op fails closed. # concurrent record cannot masquerade as this write and a no-op fails closed.
# #
# --login override: the default login is resolved from the local tea login list # --login override: the default login is resolved from the local tea login list
# for this repo's host (get_gitea_login_for_host). Pass --login <name> to # for this repo's host from the acting identity's own credential. Pass --login <name> to
# override it for this invocation only. The REST write, the /user identity read, # override it for this invocation only. The REST write, the /user identity read,
# and every read-back are ALL performed with the token of the EFFECTIVE login, # and every read-back are ALL performed with the token of the EFFECTIVE login,
# so the write and its verification bind to the same identity. # so the write and its verification bind to the same identity.
@@ -372,9 +372,17 @@ gitea_resolve_api_for_login() {
return 1 return 1
} }
else else
GITEA_API_TOKEN=$(get_gitea_token_for_login "$effective_login" "$host") \ # NO --login: the acting credential is this identity's own token and there
|| GITEA_API_TOKEN=$(get_gitea_token "$host") || { # is deliberately no tea-config fallback. get_gitea_token_for_login matches
echo "Error: Gitea token not found for login '$effective_login' (review write/read-back)" >&2 # by login NAME and performs no authentication check, and with no --login
# that name was a HOST GUESS resolving to a shared account. A live shared
# token would therefore have authored every seat's review as that
# account, making Gate-16 author-is-not-reviewer unenforceable fleet-wide;
# a dead one is only what made the defect visible. get_gitea_token fails
# loud on a fleet host when no identity resolves, and that refusal is the
# correct outcome, not a case to fall back from.
GITEA_API_TOKEN=$(get_gitea_token "$host") || {
echo "Error: no Gitea credential resolved for the acting identity on host '$host' (review write/read-back). Set MOSAIC_GIT_IDENTITY=<agent-id>, or pass --login <name> to use a named tea credential." >&2
return 1 return 1
} }
fi fi
@@ -698,7 +706,7 @@ if [[ "$PLATFORM" == "github" ]]; then
elif [[ "$PLATFORM" == "gitea" ]]; then elif [[ "$PLATFORM" == "gitea" ]]; then
case $ACTION in case $ACTION in
approve) approve)
# Best-effort host for the tea-login GUESS only (gitea_resolve_api_for_login # Best-effort host for credential resolution only (gitea_resolve_api_for_login
# below re-derives the real host from HOST_OVERRIDE/remote independently and # below re-derives the real host from HOST_OVERRIDE/remote independently and
# is authoritative). Prefer an explicit -H/--host; otherwise best-effort # is authoritative). Prefer an explicit -H/--host; otherwise best-effort
# git-remote inference, tolerating its ABSENCE (a bare `get_remote_host` here # git-remote inference, tolerating its ABSENCE (a bare `get_remote_host` here
@@ -706,15 +714,13 @@ elif [[ "$PLATFORM" == "gitea" ]]; then
# SILENTLY — exit 1, zero output — even though -r/-H are exactly the flags # SILENTLY — exit 1, zero output — even though -r/-H are exactly the flags
# that support running with no usable origin at all). # that support running with no usable origin at all).
host="${HOST_OVERRIDE:-$(get_remote_host 2>/dev/null || true)}" host="${HOST_OVERRIDE:-$(get_remote_host 2>/dev/null || true)}"
# A --login override always wins. Otherwise name this host's login # A --login override selects a NAMED tea credential and is the only
# only as a best effort: the login name merely selects a per-login # way to reach the tea store. With no --login there is deliberately no
# token, and gitea_resolve_api_for_login falls back to the host # guess: gitea_resolve_api_for_login resolves this identity's own token.
# credential (get_gitea_token) when no tea login is named — so a host # The guess this replaced named a SHARED account, selecting an identity
# tea's login list need not enumerate exotic (e.g. ported) hosts for # the caller never asked to act as. The single resolved token is then
# the default credential to resolve. The single resolved token is # used for the write, the /user identity, and the read-back.
# then used for the write, the /user identity, and the read-back.
EFFECTIVE_LOGIN="$LOGIN_OVERRIDE" EFFECTIVE_LOGIN="$LOGIN_OVERRIDE"
[[ -n "$EFFECTIVE_LOGIN" ]] || EFFECTIVE_LOGIN=$(get_gitea_login_for_host "$host" 2>/dev/null || true)
# Bind the REST endpoint + token to the effective login, then derive # Bind the REST endpoint + token to the effective login, then derive
# the acting identity from that SAME credential so the review submit # the acting identity from that SAME credential so the review submit
# and its read-back verify against the identity that performed them. # and its read-back verify against the identity that performed them.
@@ -735,7 +741,7 @@ elif [[ "$PLATFORM" == "gitea" ]]; then
echo "Error: Comment required for request-changes" echo "Error: Comment required for request-changes"
exit 1 exit 1
fi fi
# Best-effort host for the tea-login GUESS only (gitea_resolve_api_for_login # Best-effort host for credential resolution only (gitea_resolve_api_for_login
# below re-derives the real host from HOST_OVERRIDE/remote independently and # below re-derives the real host from HOST_OVERRIDE/remote independently and
# is authoritative). Prefer an explicit -H/--host; otherwise best-effort # is authoritative). Prefer an explicit -H/--host; otherwise best-effort
# git-remote inference, tolerating its ABSENCE (a bare `get_remote_host` here # git-remote inference, tolerating its ABSENCE (a bare `get_remote_host` here
@@ -743,15 +749,13 @@ elif [[ "$PLATFORM" == "gitea" ]]; then
# SILENTLY — exit 1, zero output — even though -r/-H are exactly the flags # SILENTLY — exit 1, zero output — even though -r/-H are exactly the flags
# that support running with no usable origin at all). # that support running with no usable origin at all).
host="${HOST_OVERRIDE:-$(get_remote_host 2>/dev/null || true)}" host="${HOST_OVERRIDE:-$(get_remote_host 2>/dev/null || true)}"
# A --login override always wins. Otherwise name this host's login # A --login override selects a NAMED tea credential and is the only
# only as a best effort: the login name merely selects a per-login # way to reach the tea store. With no --login there is deliberately no
# token, and gitea_resolve_api_for_login falls back to the host # guess: gitea_resolve_api_for_login resolves this identity's own token.
# credential (get_gitea_token) when no tea login is named — so a host # The guess this replaced named a SHARED account, selecting an identity
# tea's login list need not enumerate exotic (e.g. ported) hosts for # the caller never asked to act as. The single resolved token is then
# the default credential to resolve. The single resolved token is # used for the write, the /user identity, and the read-back.
# then used for the write, the /user identity, and the read-back.
EFFECTIVE_LOGIN="$LOGIN_OVERRIDE" EFFECTIVE_LOGIN="$LOGIN_OVERRIDE"
[[ -n "$EFFECTIVE_LOGIN" ]] || EFFECTIVE_LOGIN=$(get_gitea_login_for_host "$host" 2>/dev/null || true)
gitea_resolve_api_for_login "$EFFECTIVE_LOGIN" "${LOGIN_OVERRIDE:+explicit}" || exit 1 gitea_resolve_api_for_login "$EFFECTIVE_LOGIN" "${LOGIN_OVERRIDE:+explicit}" || exit 1
ACTING_LOGIN=$(gitea_authenticated_login) || exit 1 ACTING_LOGIN=$(gitea_authenticated_login) || exit 1
head_sha=$(gitea_pr_head_sha "$PR_NUMBER") || exit 1 head_sha=$(gitea_pr_head_sha "$PR_NUMBER") || exit 1
@@ -766,7 +770,7 @@ elif [[ "$PLATFORM" == "gitea" ]]; then
echo "Error: Comment required" echo "Error: Comment required"
exit 1 exit 1
fi fi
# Best-effort host for the tea-login GUESS only (gitea_resolve_api_for_login # Best-effort host for credential resolution only (gitea_resolve_api_for_login
# below re-derives the real host from HOST_OVERRIDE/remote independently and # below re-derives the real host from HOST_OVERRIDE/remote independently and
# is authoritative). Prefer an explicit -H/--host; otherwise best-effort # is authoritative). Prefer an explicit -H/--host; otherwise best-effort
# git-remote inference, tolerating its ABSENCE (a bare `get_remote_host` here # git-remote inference, tolerating its ABSENCE (a bare `get_remote_host` here
@@ -774,15 +778,13 @@ elif [[ "$PLATFORM" == "gitea" ]]; then
# SILENTLY — exit 1, zero output — even though -r/-H are exactly the flags # SILENTLY — exit 1, zero output — even though -r/-H are exactly the flags
# that support running with no usable origin at all). # that support running with no usable origin at all).
host="${HOST_OVERRIDE:-$(get_remote_host 2>/dev/null || true)}" host="${HOST_OVERRIDE:-$(get_remote_host 2>/dev/null || true)}"
# A --login override always wins. Otherwise name this host's login # A --login override selects a NAMED tea credential and is the only
# only as a best effort: the login name merely selects a per-login # way to reach the tea store. With no --login there is deliberately no
# token, and gitea_resolve_api_for_login falls back to the host # guess: gitea_resolve_api_for_login resolves this identity's own token.
# credential (get_gitea_token) when no tea login is named — so a host # The guess this replaced named a SHARED account, selecting an identity
# tea's login list need not enumerate exotic (e.g. ported) hosts for # the caller never asked to act as. The single resolved token is then
# the default credential to resolve. The single resolved token is # used for the write, the /user identity, and the read-back.
# then used for the write, the /user identity, and the read-back.
EFFECTIVE_LOGIN="$LOGIN_OVERRIDE" EFFECTIVE_LOGIN="$LOGIN_OVERRIDE"
[[ -n "$EFFECTIVE_LOGIN" ]] || EFFECTIVE_LOGIN=$(get_gitea_login_for_host "$host" 2>/dev/null || true)
gitea_resolve_api_for_login "$EFFECTIVE_LOGIN" "${LOGIN_OVERRIDE:+explicit}" || exit 1 gitea_resolve_api_for_login "$EFFECTIVE_LOGIN" "${LOGIN_OVERRIDE:+explicit}" || exit 1
ACTING_LOGIN=$(gitea_authenticated_login) || exit 1 ACTING_LOGIN=$(gitea_authenticated_login) || exit 1
comment_id=$(gitea_create_comment_verified "$PR_NUMBER" "$COMMENT" "$ACTING_LOGIN") || { comment_id=$(gitea_create_comment_verified "$PR_NUMBER" "$COMMENT" "$ACTING_LOGIN") || {
@@ -32,7 +32,9 @@
# 0 delivered (submitted) or queued (agent busy; will process when free) # 0 delivered (submitted) or queued (agent busy; will process when free)
# 1 tmux target not found # 1 tmux target not found
# 2 submission NOT confirmed — either still an unsubmitted draft, or the REPL # 2 submission NOT confirmed — either still an unsubmitted draft, or the REPL
# input prompt could not be located to confirm the message actually landed. # input box could not be located to confirm the message actually landed.
# Locating the box is runtime-specific; see locate_input_box() below, and
# add a shape there before pointing this tool at a new runtime.
# Delivery is NEVER inferred from absence of evidence: if we cannot positively # Delivery is NEVER inferred from absence of evidence: if we cannot positively
# see the input box clear of the message (or the queued banner), we fail loud # see the input box clear of the message (or the queued banner), we fail loud
# so the sender learns immediately instead of a silent worker->lead stall. # so the sender learns immediately instead of a silent worker->lead stall.
@@ -97,10 +99,50 @@ printf '%s' "$MSG" | "${tmux_cmd[@]}" load-buffer -b "$BUF" -
# would otherwise accumulate forever. # would otherwise accumulate forever.
sleep 0.5 sleep 0.5
# Locate the REPL input box in a captured pane. Prints the box's contents on
# stdout and returns 0 when the box was FOUND; returns 1 when it could not be
# located at all. Found-but-empty is a real, distinct answer (an empty input box
# is what a submitted message leaves behind), so the caller must branch on the
# return code, never on whether the output is empty.
#
# Two REPL shapes are recognised:
# * a prompt-glyph line — ``, a leading `>`, or `│ >`. Claude Code and most
# readline REPLs.
# * a box drawn as two horizontal `─` rules with the input between them and NO
# prompt glyph anywhere. pi renders this. Anchoring on the LAST rule pair is
# what makes it safe: agent output can contain its own rules, but nothing is
# drawn below the input box except the status line.
#
# Adding a runtime means adding its shape HERE. A shape that is missing does not
# degrade gracefully: it turns every send to that runtime into a false
# "may be UNDELIVERED", which is what #1362 measured on pi and #1257 on another
# arm of the same probe.
locate_input_box() {
local pane=$1 glyph_line rule_lines top bottom
glyph_line=$(printf '%s\n' "$pane" | grep -E '|^>|│ >' | tail -1)
if [ -n "$glyph_line" ]; then printf '%s\n' "$glyph_line"; return 0; fi
rule_lines=$(printf '%s\n' "$pane" | grep -nE '^[[:space:]]*─{4,}[[:space:]]*$' | cut -d: -f1 | tail -2)
[ -n "$rule_lines" ] || return 1
# Split the (at most two) captured line numbers with parameter expansion. Not
# `head -1`: piping into an early-exiting consumer SIGPIPEs the producer, which
# under `set -euo pipefail` aborts the caller with rc=141 and no output. The
# scripts/pipefail-early-exit.test.mjs guard reds on that shape, correctly.
# With one rule captured both halves resolve to the same value and the
# ordering test below rejects it, which is the answer we want anyway.
top=${rule_lines%%$'\n'*}
bottom=${rule_lines##*$'\n'}
[ "$top" != "$bottom" ] || return 1
[ "$bottom" -gt "$top" ] || return 1
# An empty range (adjacent rules) prints nothing and still returns 0: found,
# empty, which is the delivered shape.
printf '%s\n' "$pane" | sed -n "$((top + 1)),$((bottom - 1))p"
return 0
}
# 2) Submit, then POSITIVELY confirm submission; flush with another Enter if it is # 2) Submit, then POSITIVELY confirm submission; flush with another Enter if it is
# still a draft. Success requires positive evidence — the queued banner, OR the # still a draft. Success requires positive evidence — the queued banner, OR the
# REPL input box located AND clear of our message tail. The historical bug was # REPL input box located AND clear of our message tail. The historical bug was
# treating ABSENCE of a draft as delivery: if the prompt glyph was never matched # treating ABSENCE of a draft as delivery: if the input box was never located
# (wrong pane / prompt-glyph drift), an unsubmitted message read as "delivered" # (wrong pane / prompt-glyph drift), an unsubmitted message read as "delivered"
# and worker->lead relays stalled silently. We now default to UNCONFIRMED and only # and worker->lead relays stalled silently. We now default to UNCONFIRMED and only
# upgrade to delivered on positive evidence; anything we cannot confirm fails loud. # upgrade to delivered on positive evidence; anything we cannot confirm fails loud.
@@ -113,15 +155,14 @@ for attempt in $(seq 1 $((RETRIES + 1))); do
if grep -qF "$QUEUED_RE" <<<"$pane"; then if grep -qF "$QUEUED_RE" <<<"$pane"; then
status="queued"; break status="queued"; break
fi fi
# Locate the REPL input box (prompt glyph). If we cannot see it, we have NO # If we cannot see the input box, we have NO evidence of submission state —
# evidence of submission state — stay UNCONFIRMED and retry; never infer delivery. # stay UNCONFIRMED and retry; never infer delivery.
promptline=$(printf '%s' "$pane" | grep -E '|^>|│ >' | tail -1) if ! inputbox=$(locate_input_box "$pane"); then
if [ -z "$promptline" ]; then
status="unconfirmed"; continue status="unconfirmed"; continue
fi fi
# Input box located AND still carrying our tail => unsubmitted draft. Flush + retry. # Input box located AND still carrying our tail => unsubmitted draft. Flush + retry.
# (Submitted messages scroll up into history; a draft stays on the line.) # (Submitted messages scroll up into history; a draft stays in the box.)
if [ -n "$snippet" ] && grep -qF "$snippet" <<<"$promptline"; then if [ -n "$snippet" ] && grep -qF "$snippet" <<<"$inputbox"; then
status="draft"; continue status="draft"; continue
fi fi
# Input box located AND clear of our tail => positively submitted. This is the # Input box located AND clear of our tail => positively submitted. This is the
@@ -135,6 +176,6 @@ case "$status" in
delivered) echo "✓ delivered to $TARGET"; exit 0 ;; delivered) echo "✓ delivered to $TARGET"; exit 0 ;;
queued) echo "✓ queued to $TARGET (agent busy — will process when it returns to prompt)"; exit 0 ;; queued) echo "✓ queued to $TARGET (agent busy — will process when it returns to prompt)"; exit 0 ;;
draft) echo "✗ still an unsubmitted draft on $TARGET after $RETRIES flush attempts" >&2; exit 2 ;; draft) echo "✗ still an unsubmitted draft on $TARGET after $RETRIES flush attempts" >&2; exit 2 ;;
unconfirmed) echo "✗ could not confirm submission on $TARGET: REPL input prompt not locatable after $((RETRIES + 1)) attempts — message may be UNDELIVERED (check target/pane, retry, or escalate)" >&2; exit 2 ;; unconfirmed) echo "✗ could not confirm submission on $TARGET: REPL input box not locatable after $((RETRIES + 1)) attempts — message may be UNDELIVERED (check target/pane, retry, or escalate)" >&2; exit 2 ;;
*) echo "✗ could not confirm submission on $TARGET (unexpected state '$status')" >&2; exit 2 ;; *) echo "✗ could not confirm submission on $TARGET (unexpected state '$status')" >&2; exit 2 ;;
esac esac
@@ -10,6 +10,13 @@
# "could not confirm submission"). # "could not confirm submission").
# 3. DRAFT — a ` `-prompt pane that never submits (message stays on the # 3. DRAFT — a ` `-prompt pane that never submits (message stays on the
# input line) => exit 2, stderr "unsubmitted draft". # input line) => exit 2, stderr "unsubmitted draft".
# 4. DELIVERED — a pane whose input box is two `─` rules with NO prompt glyph
# (box shape) anywhere (pi's shape) and which submits => exit 0. Pre-#1362
# the glyph probe could not see this box at all, so EVERY send
# to such a pane reported "may be UNDELIVERED" while landing.
# 5. DRAFT — the same glyphless box, holding our tail across every flush
# (box shape) Enter => exit 2, stderr "unsubmitted draft". Pre-#1362 this
# also reported unconfirmed, so the true state was invisible.
set -uo pipefail set -uo pipefail
HERE=$(cd -- "$(dirname -- "$0")" && pwd) HERE=$(cd -- "$(dirname -- "$0")" && pwd)
@@ -69,6 +76,56 @@ else
fi fi
fi fi
# --- Fixtures 4 and 5: a pi-shaped pane. The input box is two `─` rules with the
# text between them and NO prompt glyph anywhere, so the glyph probe alone can
# never locate it and every send reports "may be UNDELIVERED" (#1362). The
# renderer below is the shape, not the runtime: MODE=clear submits (box empties),
# MODE=keep leaves the text sitting in the box.
cat > "$TMP/pibox.sh" <<'PIBOX'
#!/usr/bin/env bash
MODE=${1:-clear}
RULE=$(printf '─%.0s' $(seq 1 60))
buf=""
draw() {
printf '\033[H\033[2J'
printf 'fixture output line\n\n'
printf '%s\n' "$RULE"
printf '%s\n' "$buf"
printf '%s\n' "$RULE"
printf '~/fixture (main)\n'
printf 'tok 0 model fixture\n'
}
draw
while IFS= read -r line; do
# keep: hold the tail across every flush Enter, which is what a stuck draft does.
if [ "$MODE" = keep ]; then [ -n "$line" ] && buf=$line; else buf=""; fi
draw
done
PIBOX
chmod +x "$TMP/pibox.sh"
tmux -L "$SOCKET" new-session -d -s pibox -c "$TMP" "exec bash '$TMP/pibox.sh' clear"
sleep 0.3
out=$("$SEND" -L "$SOCKET" -t "=pibox" -m "pi fixture four delivered ok" 2>"$TMP/e4"); rc=$?
if [ "$rc" -eq 0 ] && grep -qF "✓ delivered" <<<"$out"; then
ok "delivered: glyphless box-drawn REPL that submits => exit 0 ✓ delivered"
else
no "delivered: glyphless box-drawn REPL that submits => exit 0 ✓ delivered" "rc=$rc out=[$out] err=[$(cat "$TMP/e4")]"
fi
tmux -L "$SOCKET" new-session -d -s piboxdraft -c "$TMP" "exec bash '$TMP/pibox.sh' keep"
sleep 0.3
if out=$("$SEND" -L "$SOCKET" -t "=piboxdraft" -r 1 -m "pi fixture five stuck in the box" 2>"$TMP/e5"); then
no "draft: glyphless box-drawn pane holding our tail must NOT report success" "expected exit 2, got 0 (out=[$out])"
else
rc=$?
if [ "$rc" -eq 2 ] && grep -qF "unsubmitted draft" "$TMP/e5"; then
ok "draft: message left in a glyphless box => exit 2 + 'unsubmitted draft'"
else
no "draft: message left in a glyphless box => exit 2 + stderr" "rc=$rc err=[$(cat "$TMP/e5")]"
fi
fi
echo "---" echo "---"
echo "PASS=$PASS FAIL=$FAIL" echo "PASS=$PASS FAIL=$FAIL"
[ "$FAIL" -eq 0 ] [ "$FAIL" -eq 0 ]
+1 -1
View File
@@ -25,7 +25,7 @@
"lint": "eslint src", "lint": "eslint src",
"typecheck": "tsc --noEmit", "typecheck": "tsc --noEmit",
"test": "vitest run --passWithNoTests && pnpm run test:framework-shell", "test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 framework/tools/quality/scripts/test-framework-drift-check.py && bash framework/tools/quality/scripts/test-framework-drift-doctor.sh && bash framework/systemd/user/test-fleet-units.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/lease-broker/revoke_noop_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-edit.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-no-status.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-fork-ci-status.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/git/test-explain-diagnostic-status-neutral.sh && bash framework/tools/git/test-detect-platform-outside-repo.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh && bash framework/tools/_scripts/test-brain-home-check.sh" "test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 framework/tools/quality/scripts/test-framework-drift-check.py && bash framework/tools/quality/scripts/test-framework-drift-doctor.sh && bash framework/systemd/user/test-fleet-units.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/lease-broker/revoke_noop_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-edit.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-no-status.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-fork-ci-status.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/git/test-explain-diagnostic-status-neutral.sh && bash framework/tools/git/test-detect-platform-outside-repo.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh && bash framework/tools/_scripts/test-brain-home-check.sh && bash framework/tools/fleet/test-agent-session-broker-preflight.sh"
}, },
"dependencies": { "dependencies": {
"@mosaicstack/brain": "workspace:*", "@mosaicstack/brain": "workspace:*",
@@ -0,0 +1,177 @@
import { lstat, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { afterEach, describe, expect, it } from 'vitest';
import { placeUnitFile, resolveLeaseBrokerSocketForPreflight } from './fleet.js';
/**
* Unit-placement regression harness for #1292.
*
* The two measured defects this suite pins:
* 1. `systemctl enable <name>` does NOT rewrite an existing by-path
* wants-symlink — so placement must remove stale residue explicitly, and
* acceptance asserts on the RESULTING SYMLINK TARGET, never on the enable
* call's argument (asserting the call cannot see where the link ended up).
* 2. Node's copyFile FOLLOWS a by-path symlink at the destination and
* overwrites the SEED template. Acceptance asserts on the SEED's bytes
* AND mtime — unchanged — which is the only check that can redden for
* finding 2. The symlink-target assertion catches finding 1; these are
* different defects with different failure modes.
*
* Fixtures are entirely inside tmpdirs (source template, active systemd dir,
* wants dir) — no real host paths are touched by this suite.
*/
describe('placeUnitFile (#1292 unit placement)', () => {
const cleanup: string[] = [];
afterEach(async () => {
while (cleanup.length > 0) {
await rm(cleanup.pop()!, { recursive: true, force: true });
}
});
async function fixture() {
const root = await mkdtemp(join(tmpdir(), 'place-unit-'));
cleanup.push(root);
const seedDir = join(root, 'seed');
const activeDir = join(root, 'active');
await mkdir(seedDir, { recursive: true });
await mkdir(activeDir, { recursive: true });
const seedTemplate = join(seedDir, 'unit-under-test.service');
await writeFile(
seedTemplate,
'[Unit]\nDescription=seed template\n[Service]\nType=oneshot\nExecStart=/bin/true\n[Install]\nWantedBy=default.target\n',
);
const activeSource = join(root, 'active-source.service');
await writeFile(
activeSource,
'[Unit]\nDescription=active copy v2\n[Service]\nType=oneshot\nExecStart=/bin/true\n[Install]\nWantedBy=default.target\n',
);
return { root, seedDir, activeDir, seedTemplate, activeSource };
}
it('places a regular file on a clean host (negative control: no residue anywhere)', async () => {
const f = await fixture();
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
expect(result.unlinkedDestinationSymlink).toBe(false);
expect(result.removedStaleWantsSymlink).toBe(false);
const info = await lstat(join(f.activeDir, 'unit-under-test.service'));
expect(info.isSymbolicLink()).toBe(false);
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toContain(
'active copy v2',
);
// Seed untouched by construction — but assert it, so the clean-host case
// cannot silently regress into seed-mutation.
expect(await readFile(f.seedTemplate, 'utf8')).toContain('seed template');
});
it('by-path residue: unlinks destination symlink, places the file, seed bytes AND mtime unchanged (finding 2)', async () => {
const f = await fixture();
const seedBefore = await readFile(f.seedTemplate, 'utf8');
const mtimeBefore = (await lstat(f.seedTemplate)).mtimeMs;
// The fomo-lin convention: by-path enable left a symlink AT the unit name
// pointing at the seed template, plus a wants-symlink doing the same.
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
const wantsDir = join(f.activeDir, 'default.target.wants');
await mkdir(wantsDir, { recursive: true });
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
expect(result.unlinkedDestinationSymlink).toBe(true);
expect(result.removedStaleWantsSymlink).toBe(true);
// FINDING 2's check: the seed is byte-identical and its mtime did not move.
expect(await readFile(f.seedTemplate, 'utf8')).toBe(seedBefore);
expect((await lstat(f.seedTemplate)).mtimeMs).toBe(mtimeBefore);
// The destination is now a regular file carrying the ACTIVE content.
const destInfo = await lstat(join(f.activeDir, 'unit-under-test.service'));
expect(destInfo.isSymbolicLink()).toBe(false);
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toContain(
'active copy v2',
);
});
it('by-path residue: no wants-symlink remains pointing at the seed (finding 1 residue cleared)', async () => {
const f = await fixture();
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
const wantsDir = join(f.activeDir, 'default.target.wants');
await mkdir(wantsDir, { recursive: true });
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
// After placement the stale wants link is GONE (enable-by-name recreates
// it correctly). A link still present must not point at the seed.
try {
const link = await lstat(join(wantsDir, 'unit-under-test.service'));
if (link.isSymbolicLink()) {
const target = await readFile(join(wantsDir, 'unit-under-test.service'), 'utf8').catch(
async () => '',
);
expect(target).not.toContain('seed template');
}
} catch {
// absent wants link — the expected post-placement state
}
});
it('idempotence: second placement on a reconciled host is a no-op producing the identical final state', async () => {
const f = await fixture();
// Reconciled starting state: regular file at the name, wants link to the active copy.
await writeFile(
join(f.activeDir, 'unit-under-test.service'),
await readFile(f.activeSource, 'utf8'),
);
const wantsDir = join(f.activeDir, 'default.target.wants');
await mkdir(wantsDir, { recursive: true });
await symlink(
join(f.activeDir, 'unit-under-test.service'),
join(wantsDir, 'unit-under-test.service'),
);
const before = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
// No destructive step fired: no unlink, no wants removal.
expect(result.unlinkedDestinationSymlink).toBe(false);
expect(result.removedStaleWantsSymlink).toBe(false);
// Identical final state.
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toBe(before);
const link = await lstat(join(wantsDir, 'unit-under-test.service'));
expect(link.isSymbolicLink()).toBe(true);
});
it('double install on by-path residue converges to the identical reconciled state', async () => {
const f = await fixture();
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
const wantsDir = join(f.activeDir, 'default.target.wants');
await mkdir(wantsDir, { recursive: true });
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
const first = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
const secondRun = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
const second = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
expect(secondRun.unlinkedDestinationSymlink).toBe(false);
expect(second).toBe(first);
});
});
describe('resolveLeaseBrokerSocketForPreflight (#1292 preflight resolution)', () => {
it('explicit MOSAIC_LEASE_BROKER_SOCKET wins', () => {
expect(
resolveLeaseBrokerSocketForPreflight({ MOSAIC_LEASE_BROKER_SOCKET: '/custom/sock' }, 1000),
).toBe('/custom/sock');
});
it('XDG_RUNTIME_DIR next', () => {
expect(resolveLeaseBrokerSocketForPreflight({ XDG_RUNTIME_DIR: '/run/user/1001' }, 1000)).toBe(
'/run/user/1001/mosaic-lease/broker.sock',
);
});
it('falls back to /run/user/<uid>', () => {
expect(resolveLeaseBrokerSocketForPreflight({}, 1002)).toBe(
'/run/user/1002/mosaic-lease/broker.sock',
);
});
});
@@ -73,6 +73,10 @@ function program(
runner, runner,
reconcileDeps: { reconcileDeps: {
homeDirectory: '/home/mosaic', homeDirectory: '/home/mosaic',
// Deterministic broker presence: without a seam the reconciler probes the
// REAL host socket (#1297 F3), making every CLI start test answer the
// host's broker state instead of its own property.
checkBrokerSocket: async () => true,
readHolderIdentity: async () => '11111111-1111-4111-8111-111111111111', readHolderIdentity: async () => '11111111-1111-4111-8111-111111111111',
validateRoster: async () => undefined, validateRoster: async () => undefined,
prepareProjections: async () => [{ agentName: 'coder0' }], prepareProjections: async () => [{ agentName: 'coder0' }],
+193 -2
View File
@@ -1,11 +1,24 @@
import { chmod, lstat, mkdir, mkdtemp, readFile, rm, stat, writeFile } from 'node:fs/promises'; import {
chmod,
lstat,
mkdir,
mkdtemp,
readFile,
readlink,
rm,
stat,
symlink,
writeFile,
} from 'node:fs/promises';
import { tmpdir } from 'node:os'; import { tmpdir } from 'node:os';
import { dirname, join, resolve } from 'node:path'; import { dirname, join, resolve } from 'node:path';
import { createServer } from 'node:net';
import { Command } from 'commander'; import { Command } from 'commander';
import { afterEach, describe, expect, it, vi } from 'vitest'; import { afterEach, describe, expect, it, vi } from 'vitest';
import { import {
acquireRestartLock, acquireRestartLock,
addAgentToRoster, addAgentToRoster,
brokerSocketPresent,
buildAgentSendCommand, buildAgentSendCommand,
buildAgentWatchAttachCommand, buildAgentWatchAttachCommand,
buildAgentWatchCommand, buildAgentWatchCommand,
@@ -42,6 +55,7 @@ import {
parseSystemdShow, parseSystemdShow,
parseTmuxListPanes, parseTmuxListPanes,
parseTmuxListSessions, parseTmuxListSessions,
placeUnitFile,
registerFleetCommand, registerFleetCommand,
removeAgentFromRoster, removeAgentFromRoster,
resolveFleetPaths, resolveFleetPaths,
@@ -50,6 +64,7 @@ import {
RESTART_LOCK_STALE_MS, RESTART_LOCK_STALE_MS,
RUNTIME_ACCEPTABLE_COMMANDS, RUNTIME_ACCEPTABLE_COMMANDS,
serializeRosterToYaml, serializeRosterToYaml,
UnitPlacementError,
VERIFY_DEFAULT_TIMEOUT_MS, VERIFY_DEFAULT_TIMEOUT_MS,
VERIFY_POLL_INTERVAL_MS, VERIFY_POLL_INTERVAL_MS,
type AgentPsRow, type AgentPsRow,
@@ -836,13 +851,25 @@ describe('fleet command construction', () => {
}; };
const program = new Command(); const program = new Command();
program.exitOverride(); program.exitOverride();
registerFleetCommand(program, { runner, mosaicHome: home }); // #1292: inject a present broker socket so the preflight passes and this
// spec keeps testing its ORIGINAL property (holder-before-agent ordering).
// The preflight's own refusal behavior has dedicated specs below.
registerFleetCommand(program, {
runner,
mosaicHome: home,
checkBrokerSocket: async () => true,
});
try { try {
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']); await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
await program.parseAsync(['node', 'mosaic', 'fleet', 'stop']); await program.parseAsync(['node', 'mosaic', 'fleet', 'stop']);
expect(calls).toEqual([ expect(calls).toEqual([
// #1292: fleet start enables + starts the broker FIRST (enable is
// idempotent; the unit exists after install), re-checking the socket
// before any holder/agent lifecycle effect.
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
['systemctl', '--user', 'start', 'mosaic-tmux-holder.service'], ['systemctl', '--user', 'start', 'mosaic-tmux-holder.service'],
['systemctl', '--user', 'start', '[email protected]'], ['systemctl', '--user', 'start', '[email protected]'],
['systemctl', '--user', 'stop', '[email protected]'], ['systemctl', '--user', 'stop', '[email protected]'],
@@ -853,6 +880,92 @@ describe('fleet command construction', () => {
} }
}); });
it('fleet start refuses with a named error when the broker socket does not appear (#1292)', async () => {
const home = await tempDir();
const rosterPath = join(home, 'fleet', 'roster.yaml');
await mkdir(join(home, 'fleet'), { recursive: true });
await writeFile(
rosterPath,
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
'\n',
),
);
const calls: string[][] = [];
const runner: CommandRunner = async (command, args) => {
calls.push([command, ...args]);
return { stdout: '', stderr: '', exitCode: 0 };
};
const program = new Command();
program.exitOverride();
const errors: string[] = [];
const origError = console.error;
console.error = (...args: unknown[]) => {
errors.push(args.join(' '));
};
registerFleetCommand(program, {
runner,
mosaicHome: home,
checkBrokerSocket: async () => false,
});
try {
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
// Refused: no holder/agent starts were issued after the broker attempt.
expect(calls).toEqual([
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
]);
expect(errors.join('\n')).toContain('broker-absent');
expect(errors.join('\n')).toContain('mosaic fleet install');
} finally {
console.error = origError;
await rm(home, { recursive: true, force: true });
}
});
it('fleet start re-probes the broker on the SECOND invocation — no ActiveState trust (#1292 sticky half)', async () => {
const home = await tempDir();
const rosterPath = join(home, 'fleet', 'roster.yaml');
await mkdir(join(home, 'fleet'), { recursive: true });
await writeFile(
rosterPath,
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
'\n',
),
);
const calls: string[][] = [];
const runner: CommandRunner = async (command, args) => {
calls.push([command, ...args]);
return { stdout: '', stderr: '', exitCode: 0 };
};
const program = new Command();
program.exitOverride();
// Broker socket NEVER appears — the second start must refuse exactly like
// the first; RemainAfterExit-style stale unit state changes nothing
// because the check is the socket, not systemctl.
registerFleetCommand(program, {
runner,
mosaicHome: home,
checkBrokerSocket: async () => false,
});
const errors: string[] = [];
const origError = console.error;
console.error = (...args: unknown[]) => {
errors.push(args.join(' '));
};
try {
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
// Two invocations, each refusing after its own broker attempt:
expect(
calls.filter((c) => c.join(' ') === 'systemctl --user start [email protected]'),
).toHaveLength(0);
expect(errors.filter((e) => e.includes('broker-absent')).length).toBeGreaterThanOrEqual(2);
} finally {
console.error = origError;
await rm(home, { recursive: true, force: true });
}
});
it('waits for an in-flight restart to clear before relaunching (re-entry guard)', async () => { it('waits for an in-flight restart to clear before relaunching (re-entry guard)', async () => {
const home = await tempDir(); const home = await tempDir();
const rosterPath = join(home, 'fleet', 'roster.yaml'); const rosterPath = join(home, 'fleet', 'roster.yaml');
@@ -2066,8 +2179,19 @@ describe('fleet install — auto-enable units for boot-survival', () => {
await enableFleetUnits(runner, minimalRoster, {}); await enableFleetUnits(runner, minimalRoster, {});
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-lease-broker.service']);
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-tmux-holder.service']); expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-tmux-holder.service']);
expect(calls).toContainEqual(['systemctl', '--user', 'enable', '[email protected]']); expect(calls).toContainEqual(['systemctl', '--user', 'enable', '[email protected]']);
// The broker must be enabled BEFORE the holder and agents: a start of any
// gated runtime without the broker is exactly the #1292 4-second death.
const brokerIndex = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user enable mosaic-lease-broker.service',
);
const holderIndex = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user enable mosaic-tmux-holder.service',
);
expect(brokerIndex).toBeGreaterThanOrEqual(0);
expect(brokerIndex).toBeLessThan(holderIndex);
}); });
it('install still succeeds when systemctl enable returns non-zero (non-fatal)', async () => { it('install still succeeds when systemctl enable returns non-zero (non-fatal)', async () => {
@@ -4362,3 +4486,70 @@ describe('fleet ps — heartbeat path resolution', () => {
); );
}); });
}); });
describe('#1297 review: the real broker probe, exercised without any seam', () => {
it('brokerSocketPresent answers a REAL unix socket via stat().isSocket() (access(S_IFSOCK) threw ERR_OUT_OF_RANGE)', async () => {
const dir = await tempDir();
const sockPath = join(dir, 'broker.sock');
const server = createServer();
await new Promise<void>((resolve) => {
server.listen(sockPath, resolve);
});
try {
// A live unix socket answers true through the REAL probe — no seam.
expect(await brokerSocketPresent({}, { MOSAIC_LEASE_BROKER_SOCKET: sockPath })).toBe(true);
// Discrimination is by file type: a regular file that EXISTS is not a
// socket. The old implementation could not reach either verdict — it
// threw ERR_OUT_OF_RANGE (node >= 24) and the catch answered false.
const notASocket = join(dir, 'not-a-sock');
await writeFile(notASocket, 'x');
expect(await brokerSocketPresent({}, { MOSAIC_LEASE_BROKER_SOCKET: notASocket })).toBe(false);
// Absent path: false, not a throw.
expect(
await brokerSocketPresent({}, { MOSAIC_LEASE_BROKER_SOCKET: join(dir, 'gone.sock') }),
).toBe(false);
} finally {
await new Promise<void>((resolve) => {
server.close(() => resolve());
});
}
await rm(dir, { recursive: true, force: true });
});
// EACCES-based unlink failure requires a non-root uid: root bypasses
// directory mode bits (CAP_DAC_OVERRIDE), so the abort path cannot be
// triggered this way under CI's root runner. Skipped there, exercised on
// every non-root dev host.
const itUnlessRoot =
typeof process.getuid === 'function' && process.getuid() === 0 ? it.skip : it;
itUnlessRoot(
'placeUnitFile aborts with UnitPlacementError when unlink fails — never copies through a live symlink',
async () => {
const dir = await tempDir();
const unitDir = join(dir, 'systemd', 'user');
await mkdir(unitDir, { recursive: true });
// By-path residue: destination is a symlink pointing somewhere else.
const residueTarget = join(dir, 'residue-target');
await writeFile(residueTarget, 'RESIDUE-BYTES');
const destination = join(unitDir, 'x.service');
await symlink(residueTarget, destination);
const source = join(dir, 'seed.service');
await writeFile(source, 'UNIT-BYTES');
// Read-only unit dir: unlink now fails EACCES (test runs as the owner,
// not root, so mode bits are enforced).
await chmod(unitDir, 0o500);
try {
await expect(placeUnitFile(source, unitDir, 'x.service')).rejects.toThrow(
UnitPlacementError,
);
} finally {
await chmod(unitDir, 0o700);
}
// The copy-through never happened: residue bytes intact, destination
// still the symlink (abort, not overwrite-through).
expect(await readFile(residueTarget, 'utf8')).toBe('RESIDUE-BYTES');
expect(await readlink(destination)).toBe(residueTarget);
await rm(dir, { recursive: true, force: true });
},
);
});
+259 -14
View File
@@ -1,11 +1,13 @@
import { constants } from 'node:fs'; import { constants, type Stats } from 'node:fs';
import { import {
access, access,
chmod, chmod,
copyFile, copyFile,
lstat,
mkdir, mkdir,
open, open,
readFile, readFile,
readlink,
stat, stat,
unlink, unlink,
writeFile, writeFile,
@@ -90,6 +92,8 @@ export type SleepFn = (ms: number) => Promise<void>;
export interface FleetCommandDeps { export interface FleetCommandDeps {
runner?: CommandRunner; runner?: CommandRunner;
/** Test seam for the #1292 fleet-start broker preflight (socket presence). */
checkBrokerSocket?: (path: string) => Promise<boolean> | boolean;
/** Injectable interactive runner for commands needing inherited TTY (e.g., `tmux attach`). */ /** Injectable interactive runner for commands needing inherited TTY (e.g., `tmux attach`). */
interactiveRunner?: InteractiveRunner; interactiveRunner?: InteractiveRunner;
/** /**
@@ -815,6 +819,135 @@ export function buildSystemdEnableCommand(unit: string): string[] {
return ['systemctl', '--user', 'enable', unit]; return ['systemctl', '--user', 'enable', unit];
} }
/**
* Place a unit file into the ACTIVE systemd user directory, never through a
* symlink (#1292, measured 2026-08-17).
*
* ⚠ SET-INDEPENDENCE (fomo-lin, 2026-08-17): the set of unit names carrying
* by-path residue and the set of unit names this install copies are
* INDEPENDENT. Until 0.0.50 they were disjoint only by accident of which
* units the install happened to name — fomo-lin survived copy-through solely
* because its one by-path symlink (the broker) was the one unit the install
* did NOT copy. Adding the broker to the copy set made the intersection
* non-empty on the first run. Whoever adds a fifth unit to the placement
* list inherits this helper and its unlink step; do not place units with a
* bare copyFile.
*
* A host provisioned by the enable-by-path convention carries a symlink AT
* the unit-name path in ~/.config/systemd/user/ pointing at the shipped
* template under ~/.config/mosaic/systemd/user/. Node's copyFile FOLLOWS
* that link and overwrites the SEED template instead of placing the active
* unit (verified with fs.copyFile on a throwaway systemd user instance) —
* silent, rc=0, and it mutates the directory every later reseed reads from.
* The same measurement showed `systemctl enable <name>` does NOT rewrite an
* existing by-path wants-symlink, so reconciliation must be explicit.
*
* Placement therefore: if the destination is a symlink, unlink it first
* (unlink → copy — copy-then-unlink would mutate the seed and then destroy
* the evidence that it did); then copy. Also removes a stale
* `default.target.wants/<name>` symlink that points outside the active
* directory (readlink — NOT readFile, which follows the link and returns the
* target's CONTENT), so the subsequent enable-by-name recreates it against
* the active copy. Idempotent: on a clean or already-reconciled destination
* every step is a no-op (the copy rewrites identical bytes).
*
* Returns what was done, for assertions and install reporting.
*/
export interface PlaceUnitResult {
readonly unit: string;
readonly destination: string;
/** A symlink at the unit-name path was unlinked (by-path residue). */
readonly unlinkedDestinationSymlink: boolean;
/** A stale wants-symlink pointing outside the active dir was removed. */
readonly removedStaleWantsSymlink: boolean;
}
/**
* placeUnitFile failed. Thrown BEFORE any copy: no destination bytes were
* written, so a residue target cannot have been clobbered by a copy-through
* (#1297 review F2).
*/
export class UnitPlacementError extends Error {
constructor(
readonly unit: string,
message: string,
) {
super(message);
this.name = UnitPlacementError.name;
}
}
function isErrnoException(error: unknown): error is NodeJS.ErrnoException {
return error instanceof Error && 'code' in error && typeof error.code === 'string';
}
export async function placeUnitFile(
source: string,
systemdUserDir: string,
unit: string,
): Promise<PlaceUnitResult> {
const destination = join(systemdUserDir, unit);
let unlinkedDestinationSymlink = false;
// Destination-absent and unlink-FAILED are different outcomes and must not
// share a catch (#1297 review F2): a swallowed unlink error used to fall
// through to copyFile through the still-live symlink, silently reintroducing
// the exact copy-through this helper exists to prevent.
let destinationInfo: Stats | undefined;
try {
destinationInfo = await lstat(destination);
} catch (error) {
if (!isErrnoException(error) || error.code !== 'ENOENT') {
throw new UnitPlacementError(
unit,
`cannot inspect destination ${destination}: ${error instanceof Error ? error.message : String(error)}`,
);
}
// ENOENT: absent destination — nothing to unlink, copy below is safe.
}
if (destinationInfo?.isSymbolicLink()) {
try {
await unlink(destination);
} catch (error) {
// Abort BEFORE the copy: proceeding would run copyFile through the
// still-live symlink and overwrite the residue target's bytes.
throw new UnitPlacementError(
unit,
`cannot unlink destination symlink ${destination}: ${error instanceof Error ? error.message : String(error)}`,
);
}
unlinkedDestinationSymlink = true;
}
await copyFile(source, destination);
let removedStaleWantsSymlink = false;
const wantsLink = join(systemdUserDir, 'default.target.wants', unit);
try {
const wantsInfo = await lstat(wantsLink);
if (wantsInfo.isSymbolicLink()) {
// readlink — NOT readFile: readFile FOLLOWS the link and returns the
// target file's CONTENT, which is not the question being asked.
let target: string | undefined;
try {
target = await readlink(wantsLink);
} catch {
target = undefined;
}
// Normalize (systemctl writes absolute targets; a relative one resolves
// against the wants dir). A wants-symlink pointing anywhere other than
// the active copy (the by-path convention points at the seed template)
// survives enable-by-name unchanged — remove it so enable recreates it.
if (target !== undefined && resolve(dirname(wantsLink), target) !== destination) {
await unlink(wantsLink);
removedStaleWantsSymlink = true;
}
}
} catch {
// absent wants link — nothing to reconcile
}
return { unit, destination, unlinkedDestinationSymlink, removedStaleWantsSymlink };
}
/** /**
* Returns the systemctl --user disable command for a given unit. * Returns the systemctl --user disable command for a given unit.
* Used by `fleet remove` so a removed agent's enabled unit cannot resurrect on * Used by `fleet remove` so a removed agent's enabled unit cannot resurrect on
@@ -849,6 +982,22 @@ export async function enableFleetUnits(
let succeeded = 0; let succeeded = 0;
let failed = 0; let failed = 0;
// The lease broker ships with the fleet and every gated runtime needs it
// (#1292): seats die at lease registration without it, and no documented
// path ever enabled it. Enabled first — alongside the holder — and the
// unit must have been placed by installFleet's placeUnitFile step.
const brokerResult = await runner(
...splitCommand(buildSystemdEnableCommand('mosaic-lease-broker.service')),
);
if (brokerResult.exitCode === 0) {
succeeded++;
} else {
failed++;
process.stderr.write(
`Warning: could not enable mosaic-lease-broker.service: ${brokerResult.stderr || brokerResult.stdout || 'non-zero exit'}\n`,
);
}
const holderResult = await runner( const holderResult = await runner(
...splitCommand(buildSystemdEnableCommand('mosaic-tmux-holder.service')), ...splitCommand(buildSystemdEnableCommand('mosaic-tmux-holder.service')),
); );
@@ -1545,7 +1694,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.description('Install local fleet tools and user systemd units') .description('Install local fleet tools and user systemd units')
.option('--no-enable', 'Skip enabling units for boot-survival') .option('--no-enable', 'Skip enabling units for boot-survival')
.action(async (opts: { enable?: boolean }) => { .action(async (opts: { enable?: boolean }) => {
await installFleet(cmd, frameworkRoot); await installFleet(cmd, frameworkRoot, runner);
// Unit enablement needs agent names only, so it reads either version. // Unit enablement needs agent names only, so it reads either version.
const roster = await loadRosterReadModel(cmd); const roster = await loadRosterReadModel(cmd);
await enableFleetUnits(runner, roster, opts); await enableFleetUnits(runner, roster, opts);
@@ -1556,7 +1705,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
.description('Install local fleet tools and user systemd units') .description('Install local fleet tools and user systemd units')
.option('--no-enable', 'Skip enabling units for boot-survival') .option('--no-enable', 'Skip enabling units for boot-survival')
.action(async (opts: { enable?: boolean }) => { .action(async (opts: { enable?: boolean }) => {
await installFleet(cmd, frameworkRoot); await installFleet(cmd, frameworkRoot, runner);
// Unit enablement needs agent names only, so it reads either version. // Unit enablement needs agent names only, so it reads either version.
const roster = await loadRosterReadModel(cmd); const roster = await loadRosterReadModel(cmd);
await enableFleetUnits(runner, roster, opts); await enableFleetUnits(runner, roster, opts);
@@ -1609,6 +1758,37 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
); );
return; return;
} }
if (action === 'start') {
// Broker preflight (#1292), re-probed on EVERY invocation: a
// gated runtime started without a live lease broker dies ~4s in
// while the unit reports active (RemainAfterExit) — enabling +
// starting here and then RE-CHECKING the socket refuses loudly
// instead of reporting rc0 over a doomed start. This is the
// second-start check as much as the first: it never trusts unit
// ActiveState.
await runChecked(runner, [
'systemctl',
'--user',
'enable',
'mosaic-lease-broker.service',
]);
await runChecked(runner, [
'systemctl',
'--user',
'start',
'mosaic-lease-broker.service',
]);
if (!(await brokerSocketPresent(deps))) {
console.error(
'[fleet] broker-absent: lease broker socket did not appear after enable+start (#1292).',
);
console.error(
'[fleet] remedy: mosaic fleet install (it reconciles either enable convention)',
);
process.exitCode = 1;
return;
}
}
if (action === 'restart') { if (action === 'restart') {
// Serialize the holder+agents teardown/relaunch behind the restart lock // Serialize the holder+agents teardown/relaunch behind the restart lock
// so a re-entrant restart waits for clean shutdown before relaunching, // so a re-entrant restart waits for clean shutdown before relaunching,
@@ -2367,7 +2547,11 @@ export function registerFleetAgentCommands(
}); });
} }
async function installFleet(cmd: Command, frameworkRoot: string): Promise<void> { async function installFleet(
cmd: Command,
frameworkRoot: string,
runner: CommandRunner,
): Promise<void> {
const activePaths = resolveFleetPaths(cmd.opts<{ mosaicHome: string }>().mosaicHome); const activePaths = resolveFleetPaths(cmd.opts<{ mosaicHome: string }>().mosaicHome);
assertDefaultMosaicHomeForSystemd(activePaths.mosaicHome); assertDefaultMosaicHomeForSystemd(activePaths.mosaicHome);
// Read model first: every file this function places is roster-independent, and // Read model first: every file this function places is roster-independent, and
@@ -2419,18 +2603,40 @@ async function installFleet(cmd: Command, frameworkRoot: string): Promise<void>
for (const toolPath of executableToolPaths) { for (const toolPath of executableToolPaths) {
await chmod(toolPath, 0o755); await chmod(toolPath, 0o755);
} }
await copyFile( // Unit placement (#1292): every unit goes through placeUnitFile — never a
join(frameworkRoot, 'systemd', 'user', 'mosaic-tmux-holder.service'), // bare copyFile — so a by-path-enable symlink at the destination is
join(activePaths.systemdUserDir, 'mosaic-tmux-holder.service'), // unlinked rather than written through (copy-through would silently
// overwrite the SEED template, measured 2026-08-17). The lease broker unit
// is placed here too: previously the install named three units and omitted
// the broker entirely, which is why no documented path ever enabled it.
const placedUnits = await Promise.all(
[
'mosaic-tmux-holder.service',
'[email protected]',
'[email protected]',
'mosaic-lease-broker.service',
].map((unit) =>
placeUnitFile(join(frameworkRoot, 'systemd', 'user', unit), activePaths.systemdUserDir, unit),
),
); );
await copyFile( const reconciled = placedUnits.filter(
join(frameworkRoot, 'systemd', 'user', '[email protected]'), (result) => result.unlinkedDestinationSymlink || result.removedStaleWantsSymlink,
join(activePaths.systemdUserDir, '[email protected]'),
);
await copyFile(
join(frameworkRoot, 'systemd', 'user', '[email protected]'),
join(activePaths.systemdUserDir, '[email protected]'),
); );
if (reconciled.length > 0) {
console.log(
`Reconciled ${reconciled.length} unit placement(s) from by-path enable residue: ${reconciled.map((r) => r.unit).join(', ')}`,
);
}
// systemd will not see a replaced unit file without a reload; do it once
// after all placements, before any enable call below. runCommand never
// rejects (it resolves exitCode 127 on spawn error), so a plain await with
// an exitCode check matches the rest of this file's systemctl handling.
const reloadResult = await runner(...splitCommand(['systemctl', '--user', 'daemon-reload']));
if (reloadResult.exitCode !== 0) {
process.stderr.write(
`Warning: systemctl --user daemon-reload after unit placement failed (non-systemd host?): ${reloadResult.stderr || reloadResult.stdout || 'non-zero exit'}\n`,
);
}
// On roster v2 the reconciler owns the generated env: `apply` writes it and // On roster v2 the reconciler owns the generated env: `apply` writes it and
// `regen` rebuilds it, both from projectRosterV2AgentGeneratedEnv. Writing it // `regen` rebuilds it, both from projectRosterV2AgentGeneratedEnv. Writing it
@@ -2627,6 +2833,45 @@ function splitCommand(command: string[]): [string, string[]] {
return [bin, args]; return [bin, args];
} }
/**
* Lease-broker socket presence for the fleet-start preflight (#1292).
* Resolution precedence matches launch.ts's defaultLeaseBrokerSocket and
* start-agent-session.sh's broker_socket_path: explicit
* MOSAIC_LEASE_BROKER_SOCKET, else $XDG_RUNTIME_DIR/mosaic-lease/broker.sock,
* else /run/user/<uid>/mosaic-lease/broker.sock. Pure filesystem check — this
* deliberately does NOT consult systemd state: a unit can be active
* (RemainAfterExit) with no live socket, and the socket is the thing the
* gated runtime connects to. Injectable via deps for tests.
*/
export function resolveLeaseBrokerSocketForPreflight(
env: NodeJS.ProcessEnv = process.env,
uid: number = typeof process.getuid === 'function' ? process.getuid() : 0,
): string {
if (env['MOSAIC_LEASE_BROKER_SOCKET']) return env['MOSAIC_LEASE_BROKER_SOCKET'];
const runtimeDir = env['XDG_RUNTIME_DIR'] ?? `/run/user/${uid}`;
return join(runtimeDir, 'mosaic-lease', 'broker.sock');
}
export async function brokerSocketPresent(
deps: FleetCommandDeps,
env: NodeJS.ProcessEnv = process.env,
): Promise<boolean> {
const check = deps.checkBrokerSocket;
const socketPath = resolveLeaseBrokerSocketForPreflight(env);
if (check) return check(socketPath);
// S_IFSOCK (0xC000) is a file-TYPE constant, not an access() mode (0-7):
// access(path, S_IFSOCK) throws ERR_OUT_OF_RANGE on node >= 24 (measured on
// v24.18.0, #1297 review F1) and cannot succeed on any node — the old catch
// swallowed the throw, so this probe could NEVER return true and every
// un-seamed call reported the broker absent. stat() + isSocket() is the real
// check and matches the bash side's [ -S ].
try {
return (await stat(socketPath)).isSocket();
} catch {
return false;
}
}
/** All supported fleet profile names. */ /** All supported fleet profile names. */
export type FleetProfile = export type FleetProfile =
| 'general' | 'general'
@@ -205,6 +205,12 @@ export async function runLeaseEnforcementDoctorCheck(
message: message:
`Lease-enforcement hooks (${matchedMarkers.join(', ')}) are wired in ~/.claude/settings.json, but ${reasons.join(' and ')}. ` + `Lease-enforcement hooks (${matchedMarkers.join(', ')}) are wired in ~/.claude/settings.json, but ${reasons.join(' and ')}. ` +
'Every gated tool call will fail closed and BRICK this agent (see #869). ' + 'Every gated tool call will fail closed and BRICK this agent (see #869). ' +
'Remediate by activating the lease-broker supervisor (systemd unit + socket) or by removing the enforcement hooks from ~/.claude/settings.json.', // #1292: one remedy, correct under BOTH enable conventions (by-path on
// the seed template, and copy-then-enable in the active dir). Written
// from the 2026-08-17 symlink measurement: `systemctl enable` by name
// does NOT rewrite an existing by-path wants-symlink, so teaching a
// manual systemctl line here could leave a host with two competing
// wants links. fleet install reconciles either shape.
'Remedy: run `mosaic fleet install` (it reconciles either enable convention), or remove the enforcement hooks from ~/.claude/settings.json.',
}; };
} }
@@ -169,6 +169,16 @@ function reconcileDeps(host: FakeLifecycleHost): FleetReconcileDeps {
applyProjection: async () => undefined, applyProjection: async () => undefined,
readRoster: async () => host.roster, readRoster: async () => host.roster,
acquireMutationLock: async () => async () => undefined, acquireMutationLock: async () => async () => undefined,
// Hermetic broker observation (#1297 F3): without this, the plan probes
// the REAL host filesystem, so the "stable JSON" fixtures answered true
// on any machine with a live lease broker and false elsewhere. Pointing
// both paths at fixtures that do not exist pins socketPresent:false and
// unitInstalled:false on every host, which is what these fixtures assert.
brokerSocketEnv: {
MOSAIC_LEASE_BROKER_SOCKET: '/nonexistent/mosaic-lease/broker.sock',
XDG_CONFIG_HOME: '/nonexistent/mosaic-config',
XDG_RUNTIME_DIR: '/nonexistent/run',
},
}; };
} }
@@ -459,6 +469,7 @@ describe('FCM-M3-002 reconciler lifecycle acceptance', (): void => {
plan: { plan: {
generation: 7, generation: 7,
holder: 'owned', holder: 'owned',
broker: { unitInstalled: false, socketPresent: false },
agents: [ agents: [
{ {
name: 'coder0', name: 'coder0',
@@ -1,6 +1,7 @@
import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises'; import { chmod, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os'; import { tmpdir } from 'node:os';
import { join } from 'node:path'; import { join } from 'node:path';
import { createServer } from 'node:net';
import { afterEach, describe, expect, it } from 'vitest'; import { afterEach, describe, expect, it } from 'vitest';
import { import {
acquirePrivateReconcileLock, acquirePrivateReconcileLock,
@@ -92,6 +93,179 @@ async function run(command: FleetReconcileCommand, overrides: Partial<FleetRecon
} }
describe('fleet roster-owned reconciler', (): void => { describe('fleet roster-owned reconciler', (): void => {
// ── #1292: broker as first-class plan member + broker-first start ordering ──
it('reports broker unit and socket state in the plan (socket is the signal, not unit state)', async (): Promise<void> => {
const result = await run('status', {
statPath: async () => true,
checkBrokerSocket: async () => true,
});
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: true });
});
it('reports a dead broker as socketPresent=false even when the unit is installed (enabled-but-dead is the #1292 shape)', async (): Promise<void> => {
const result = await run('status', {
statPath: async () => true,
checkBrokerSocket: async () => false,
});
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: false });
});
it('probes the REAL filesystem when no seam is injected — live socket and unit report healthy, absent paths report absent (#1297 F3)', async (): Promise<void> => {
const dir = await mkdtemp(join(tmpdir(), 'mosaic-broker-probe-'));
cleanup = dir;
const configHome = join(dir, 'config');
const unitDir = join(configHome, 'systemd', 'user');
await mkdir(unitDir, { recursive: true });
await writeFile(join(unitDir, 'mosaic-lease-broker.service'), '[Unit]\n');
const sockPath = join(dir, 'broker.sock');
const server = createServer();
await new Promise<void>((resolve) => {
server.listen(sockPath, resolve);
});
try {
const result = await run('status', {
brokerSocketEnv: {
MOSAIC_LEASE_BROKER_SOCKET: sockPath,
XDG_CONFIG_HOME: configHome,
XDG_RUNTIME_DIR: dir,
},
});
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: true });
// Absent paths through the SAME seam-less path answer false — this is
// the half the old default got right; healthy is the half it got wrong.
const absent = await run('status', {
brokerSocketEnv: {
MOSAIC_LEASE_BROKER_SOCKET: join(dir, 'gone.sock'),
XDG_CONFIG_HOME: join(dir, 'gone-config'),
},
});
expect(absent.plan.broker).toEqual({ unitInstalled: false, socketPresent: false });
} finally {
await new Promise<void>((resolve) => {
server.close(() => resolve());
});
}
});
it('command start refuses with a named error when the broker socket does not appear after enable+start (#1297 F3)', async (): Promise<void> => {
const calls: string[][] = [];
await expect(
run('start', {
checkBrokerSocket: async () => false,
runner: async (command, args) => {
calls.push([command, ...args]);
if (command === 'tmux' && args.includes('list-sessions')) {
return { stdout: '_holder\ncoder0\n', stderr: '', exitCode: 0 };
}
if (command === 'tmux' && args.includes('show-environment')) {
return {
stdout:
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
stderr: '',
exitCode: 0,
};
}
return { stdout: '', stderr: '', exitCode: 0 };
},
}),
).rejects.toThrow(/broker-absent/);
// Refused: broker enable+start attempted, no holder/agent unit touched.
const agentStarts = calls.filter(
(c) => c.join(' ') === 'systemctl --user start [email protected]',
);
expect(agentStarts).toHaveLength(0);
});
it('command start enables and starts the broker BEFORE the holder and any agent unit', async (): Promise<void> => {
const calls: string[][] = [];
const result = await run('start', {
// Deterministic broker presence: without the seam this test answers the
// HOST's broker state (passes on a machine with a live broker, refuses
// on CI), not the ordering property it exists for (#1297 follow-up).
checkBrokerSocket: async () => true,
runner: async (command, args) => {
calls.push([command, ...args]);
if (command === 'tmux' && args.includes('list-sessions')) {
return { stdout: '_holder\ncoder0\n', stderr: '', exitCode: 0 };
}
if (command === 'tmux' && args.includes('show-environment')) {
return {
stdout:
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
stderr: '',
exitCode: 0,
};
}
return { stdout: '', stderr: '', exitCode: 0 };
},
});
expect(result.lifecycle).toBe('complete');
const brokerEnable = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user enable mosaic-lease-broker.service',
);
const brokerStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start mosaic-lease-broker.service',
);
const holderStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start mosaic-tmux-holder.service',
);
const agentStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start [email protected]',
);
expect(brokerEnable).toBeGreaterThanOrEqual(0);
expect(brokerStart).toBeGreaterThan(brokerEnable);
// Holder start may be absent (holder 'owned' in this fixture); if present it must follow the broker.
if (holderStart >= 0) expect(holderStart).toBeGreaterThan(brokerStart);
expect(agentStart).toBeGreaterThan(brokerStart);
});
it('apply with a running desired agent also enables and starts the broker first', async (): Promise<void> => {
const calls: string[][] = [];
const runningRoster: FleetRosterV2 = {
...roster,
agents: roster.agents.map((agent) =>
agent.name === 'coder0'
? { ...agent, lifecycle: { enabled: true, desiredState: 'running' as const } }
: agent,
),
};
const result = await executeFleetReconcile({
roster: runningRoster,
command: 'apply',
expectedGeneration: 7,
deps: deps({
readRoster: async () => runningRoster,
// Deterministic broker presence (see start-ordering test note).
checkBrokerSocket: async () => true,
runner: async (command, args) => {
calls.push([command, ...args]);
if (command === 'tmux' && args.includes('list-sessions')) {
return { stdout: '_holder\n', stderr: '', exitCode: 0 };
}
if (command === 'tmux' && args.includes('show-environment')) {
return {
stdout:
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
stderr: '',
exitCode: 0,
};
}
return { stdout: '', stderr: '', exitCode: 0 };
},
}),
});
expect(result.applied).toBe(true);
const brokerStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start mosaic-lease-broker.service',
);
const agentStart = calls.findIndex(
(c) => c.join(' ') === 'systemctl --user start [email protected]',
);
expect(brokerStart).toBeGreaterThanOrEqual(0);
expect(agentStart).toBeGreaterThan(brokerStart);
});
it('fails closed on a symlinked fleet ancestor without touching its target', async (): Promise<void> => { it('fails closed on a symlinked fleet ancestor without touching its target', async (): Promise<void> => {
const home = await lockHome(); const home = await lockHome();
const fleet = join(home, 'fleet'); const fleet = join(home, 'fleet');
@@ -375,6 +549,8 @@ describe('fleet roster-owned reconciler', (): void => {
expectedGeneration: 7, expectedGeneration: 7,
deps: deps({ deps: deps({
readRoster: async () => runningRoster, readRoster: async () => runningRoster,
// Deterministic broker presence (see start-ordering test note).
checkBrokerSocket: async () => true,
runner: async (command, args) => { runner: async (command, args) => {
calls.push([command, ...args]); calls.push([command, ...args]);
if (command === 'tmux' && args.includes('list-sessions')) { if (command === 'tmux' && args.includes('list-sessions')) {
+149 -13
View File
@@ -1,5 +1,5 @@
import { constants } from 'node:fs'; import { constants } from 'node:fs';
import { lstat, open, readFile, unlink, type FileHandle } from 'node:fs/promises'; import { lstat, open, readFile, stat, unlink, type FileHandle } from 'node:fs/promises';
import { randomUUID } from 'node:crypto'; import { randomUUID } from 'node:crypto';
import { homedir } from 'node:os'; import { homedir } from 'node:os';
import { join } from 'node:path'; import { join } from 'node:path';
@@ -44,6 +44,10 @@ export interface FleetReconcileDeps {
readonly overrideDir?: string; readonly overrideDir?: string;
readonly homeDirectory?: string; readonly homeDirectory?: string;
readonly readHolderIdentity?: () => Promise<string>; readonly readHolderIdentity?: () => Promise<string>;
/** Test/observation seams for the lease-broker plan member (#1292). */
readonly statPath?: (path: string) => Promise<boolean> | boolean;
readonly checkBrokerSocket?: (path: string) => Promise<boolean> | boolean;
readonly brokerSocketEnv?: NodeJS.ProcessEnv;
readonly validateRoster?: (roster: FleetRosterV2) => Promise<void>; readonly validateRoster?: (roster: FleetRosterV2) => Promise<void>;
readonly prepareProjections?: (roster: FleetRosterV2) => Promise<readonly unknown[]>; readonly prepareProjections?: (roster: FleetRosterV2) => Promise<readonly unknown[]>;
readonly applyProjection?: (prepared: unknown) => Promise<unknown>; readonly applyProjection?: (prepared: unknown) => Promise<unknown>;
@@ -75,6 +79,17 @@ export interface FleetReconcileObservedAgent {
export interface FleetReconcilePlan { export interface FleetReconcilePlan {
readonly generation: number; readonly generation: number;
readonly holder: 'owned' | 'missing' | 'ownership-mismatch'; readonly holder: 'owned' | 'missing' | 'ownership-mismatch';
/**
* Lease broker observation (#1292): every gated runtime registers with the
* broker or dies ~4s in a broker not in the plan cannot be reported as
* drifted, which made "broker died an hour ago" and "broker fine"
* produce identical output. `unitInstalled` = unit file present in the
* active dir; `socketPresent` = live broker at the resolved socket path.
*/
readonly broker: {
readonly unitInstalled: boolean;
readonly socketPresent: boolean;
};
readonly agents: readonly FleetReconcileObservedAgent[]; readonly agents: readonly FleetReconcileObservedAgent[];
readonly unmanagedSessions: readonly string[]; readonly unmanagedSessions: readonly string[];
} }
@@ -246,7 +261,17 @@ export async function executeFleetReconcile(
lifecycle: 'complete', lifecycle: 'complete',
plan, plan,
}; };
} catch { } catch (error: unknown) {
// A named lifecycle precondition (broker-absent after enable+start,
// #1297 F3) must surface as itself — converting it to the generic
// recoverable result would hide the diagnosis and report a clean
// refusal where a loud one is the point.
if (
error instanceof FleetReconcileError &&
error.code === 'lifecycle-precondition-failed'
) {
throw error;
}
result = { result = {
applied: false, applied: false,
authoritativeRoster: 'unchanged', authoritativeRoster: 'unchanged',
@@ -315,6 +340,63 @@ function isObservational(command: FleetReconcileCommand): boolean {
return command === 'plan' || command === 'status' || command === 'verify' || command === 'doctor'; return command === 'plan' || command === 'status' || command === 'verify' || command === 'doctor';
} }
/**
* Observe the lease broker for the plan (#1292). Unit presence via systemctl
* is-system-running is NOT the signal a unit can be enabled-but-dead. The
* authoritative signal is the socket the gated runtimes connect to, matching
* broker-supervisor.ts's `checkBrokerSupervisorHealth` (healthy ===
* socketPresent). Injectable so tests drive every branch without a broker.
*/
function resolveBrokerSocketPath(env: NodeJS.ProcessEnv): string {
const uid = typeof process.getuid === 'function' ? process.getuid() : 0;
const runtimeDir = env['XDG_RUNTIME_DIR'] ?? `/run/user/${uid}`;
return env['MOSAIC_LEASE_BROKER_SOCKET'] ?? join(runtimeDir, 'mosaic-lease', 'broker.sock');
}
/**
* Probe the broker socket. Seams take precedence, but with no seam injected
* the REAL stat().isSocket() runs (#1297 review F3): production passes no
* seams, and defaulting to false made plan/status/doctor report a healthy
* broker as absent a dead broker was indistinguishable from noise.
*/
async function brokerSocketPresent(
deps: FleetReconcileDeps,
env: NodeJS.ProcessEnv,
): Promise<boolean> {
const socketPath = resolveBrokerSocketPath(env);
const check = deps.checkBrokerSocket;
if (check) return check(socketPath);
try {
return (await stat(socketPath)).isSocket();
} catch {
return false;
}
}
async function observeBroker(deps: FleetReconcileDeps): Promise<FleetReconcilePlan['broker']> {
const homeDirectory = deps.homeDirectory ?? homedir();
const env = (deps.brokerSocketEnv ?? process.env) as NodeJS.ProcessEnv;
const configHome = env['XDG_CONFIG_HOME'] ?? join(homeDirectory, '.config');
const unitPath = join(configHome, 'systemd', 'user', 'mosaic-lease-broker.service');
const statPath = deps.statPath;
let unitInstalled = false;
let socketPresent = false;
try {
// Same principle as the socket probe: no seam → look at the real
// filesystem. A unit file placed by installFleet (or a by-path residue
// symlink resolving to it) satisfies stat().isFile().
unitInstalled = statPath ? await statPath(unitPath) : (await stat(unitPath)).isFile();
} catch {
unitInstalled = false;
}
try {
socketPresent = await brokerSocketPresent(deps, env);
} catch {
socketPresent = false;
}
return { unitInstalled, socketPresent };
}
async function observeFleet( async function observeFleet(
roster: FleetRosterV2, roster: FleetRosterV2,
deps: FleetReconcileDeps, deps: FleetReconcileDeps,
@@ -325,10 +407,12 @@ async function observeFleet(
'-F', '-F',
'#{session_name}', '#{session_name}',
]); ]);
const broker = await observeBroker(deps);
if (sessionsResult.exitCode !== 0) { if (sessionsResult.exitCode !== 0) {
return { return {
generation: roster.generation, generation: roster.generation,
holder: 'missing', holder: 'missing',
broker,
agents: await observeAgents(roster, deps, new Set<string>()), agents: await observeAgents(roster, deps, new Set<string>()),
unmanagedSessions: [], unmanagedSessions: [],
}; };
@@ -351,6 +435,7 @@ async function observeFleet(
return { return {
generation: roster.generation, generation: roster.generation,
holder, holder,
broker,
agents: await observeAgents(roster, deps, sessions), agents: await observeAgents(roster, deps, sessions),
unmanagedSessions: Object.freeze(unmanagedSessions.sort()), unmanagedSessions: Object.freeze(unmanagedSessions.sort()),
}; };
@@ -507,6 +592,17 @@ async function executeExplicitLifecycle(
plan: FleetReconcilePlan, plan: FleetReconcilePlan,
agents: readonly FleetRosterV2Agent[], agents: readonly FleetRosterV2Agent[],
): Promise<FleetReconcileResult> { ): Promise<FleetReconcileResult> {
const lifecycleApplyFailed = (): FleetReconcileResult => ({
applied: false,
authoritativeRoster: 'unchanged',
projections: 'not-applied',
lifecycle: 'incomplete',
plan,
recovery: {
code: 'lifecycle-apply-failed',
action: 'rerun-after-inspecting-owned-resources',
},
});
if (request.command === 'start') { if (request.command === 'start') {
for (const agent of agents) { for (const agent of agents) {
if (!agent.lifecycle.enabled) { if (!agent.lifecycle.enabled) {
@@ -517,6 +613,40 @@ async function executeExplicitLifecycle(
} }
} }
} }
// Broker FIRST (#1292): a gated runtime started without a running lease
// broker dies ~4 seconds in at registration — enable the unit (install
// places it) and start it before any holder/agent lifecycle effect.
try {
if (request.command === 'start') {
await runChecked(request.deps, 'systemctl', [
'--user',
'enable',
'mosaic-lease-broker.service',
]);
await runChecked(request.deps, 'systemctl', [
'--user',
'start',
'mosaic-lease-broker.service',
]);
}
} catch {
return lifecycleApplyFailed();
}
if (request.command === 'start') {
// Socket re-check after start, as a NAMED precondition (#1297 review
// F3) — the same protection the v1 path in commands/fleet.ts has had all
// along: the unit reporting active is not the signal; the socket is.
// Deliberately outside the try/catch above: a swallowed FleetReconcileError
// here read as a generic recoverable failure, hiding the named refusal.
// Runs BEFORE any holder/agent unit is touched so nothing doomed starts.
const env = (request.deps.brokerSocketEnv ?? process.env) as NodeJS.ProcessEnv;
if (!(await brokerSocketPresent(request.deps, env))) {
throw new FleetReconcileError(
'lifecycle-precondition-failed',
'broker-absent: lease broker socket did not appear after enable+start (#1292; #1297 F3). Remedy: mosaic fleet install.',
);
}
}
try { try {
if (request.command === 'start' && plan.holder === 'missing') { if (request.command === 'start' && plan.holder === 'missing') {
await runChecked(request.deps, 'systemctl', [ await runChecked(request.deps, 'systemctl', [
@@ -533,17 +663,7 @@ async function executeExplicitLifecycle(
]); ]);
} }
} catch { } catch {
return { return lifecycleApplyFailed();
applied: false,
authoritativeRoster: 'unchanged',
projections: 'not-applied',
lifecycle: 'incomplete',
plan,
recovery: {
code: 'lifecycle-apply-failed',
action: 'rerun-after-inspecting-owned-resources',
},
};
} }
return { return {
applied: true, applied: true,
@@ -563,6 +683,22 @@ async function applyDesiredLifecycle(
(agent: FleetRosterV2Agent): boolean => (agent: FleetRosterV2Agent): boolean =>
agent.lifecycle.enabled && agent.lifecycle.desiredState === 'running', agent.lifecycle.enabled && agent.lifecycle.desiredState === 'running',
); );
// Broker before any running agent, same ordering and reason as the
// command-driven path above (#1292).
if (needsRunningAgent) {
await runChecked(deps, 'systemctl', ['--user', 'enable', 'mosaic-lease-broker.service']);
await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-lease-broker.service']);
// Same socket re-check as the explicit start path (#1297 F3): apply with
// running desired agents starts gated runtimes too, and a broker that
// starts but never binds dooms them the same way.
const env = (deps.brokerSocketEnv ?? process.env) as NodeJS.ProcessEnv;
if (!(await brokerSocketPresent(deps, env))) {
throw new FleetReconcileError(
'lifecycle-precondition-failed',
'broker-absent: lease broker socket did not appear after enable+start (#1292; #1297 F3). Remedy: mosaic fleet install.',
);
}
}
if (needsRunningAgent && plan.holder === 'missing') { if (needsRunningAgent && plan.holder === 'missing') {
await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-tmux-holder.service']); await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-tmux-holder.service']);
} }