Compare commits

..
Author SHA1 Message Date
fred 57a2f2b40e docs(ci): point the exclusion at tracking issue #1271, not the closed first filing
ci/woodpecker/pr/ci Pipeline was successful
The first PR for this change was filed under the retired mos-dt-0 principal
(pr-create.sh has no --login flag and find_tea_login_for_host returns the first
host match) and was closed and refiled as #1270. That left in-tree references
pointing at a closed duplicate PR rather than at the burn-down issue, which is
the wrong target for them anyway: the open design question belongs on #1271.
2026-08-16 18:03:02 -05:00
fred 93c1de51e1 fix(ci): unwire test-start-agent-session.sh, restore its signed exclusion (#1269)
ci/woodpecker/pr/ci Pipeline was canceled
The `test` step has failed on every `next` pipeline since #1017 on exactly one
assertion, and it is the same one on unrelated PRs:

    FAIL: host provides 'pi' in the system path; missing-binary cases are not
    measurable here            (framework/tools/fleet/test-start-agent-session.sh:103)

Measured 2026-08-16 across pipelines 2444 (#1256), 2438 (#1240) and 2441
(#1017-quality): exactly one FAIL line in each full log, identical, this line.
Control `zzz-not-present-zzz` -> 0 on all three.

Cause. #1241 (5c35a250) added the guard: the suite shims fake mosaic/pi/npm into
$FAKE_BIN, but the constructed PANE_PATH always ends in the real system path, so
on a host that installs those binaries the missing-binary cases cannot be
measured and a green run would mean nothing. The guard says so instead of
passing. Its own pipeline 2430 was green only because the suite was CI-excluded
at the time, so the guard had never run in CI. #1017 (c56483eb) then enumerated
it and dropped the exclusion. The CI image installs
@earendil-works/[email protected].1 on purpose, so the precondition is
unsatisfiable there. Both commits are mine.

The guard is correct and is not being softened. A check that cannot measure its
property and reports success is the failure mode this repo has been cataloguing
all week; the error was wiring the suite into an image that violates its
precondition, so the wiring is what gets reverted.

Second effect, which is the reason this cost a day rather than an hour:
test:framework-shell is one && chain and this sat at position 44 of 48, so
glpi/test-list-http-status.sh, orchestrator/test-board-roll.sh,
woodpecker/test-ci-wait-exit-matrix.sh and _scripts/test-fleet-transport-check.sh
have not run at all since the merge. The pipeline reported one failure, never
"one failure plus four unrun". All four are green when run directly on
sb-it-1-dt, so the mask hid nothing broken -- but that is a local result on one
host, not a CI-image result.

Verification, with controls:
- enumeration guard OK (population 52, enumerated 36, signed-excluded 16).
- control A, exclusion line removed while unwired -> FAIL UNENUMERATED.
- control B, exclusion line kept while rewired -> FAIL CONTRADICTORY EXCLUSION.
  The gate discriminates in both directions, so its OK is load-bearing.
- the four formerly-masked suites: rc=0 each, run directly.
- the full chain cannot be run to completion on sb-it-1-dt: it stops earlier, at
  the lease-broker Invariant R test, because this host carries the quarantined
  operator-global pi 0.84.2 against a measured 0.84.1. That is host-specific and
  out of scope here -- CI pins 0.84.1, and the single FAIL line in those three
  pipelines proves positions 1-43 passed there.

Burn-down is to control the tail of PANE_PATH inside the test, not to remove pi
from the image. Recorded in the exclusion reason and in #1269.
2026-08-16 17:58:49 -05:00
4 changed files with 18 additions and 98 deletions
@@ -268,16 +268,6 @@ esac
_build_runtime_bin_prefix() {
local candidates=()
if [ -n "$MOSAIC_RUNTIME_BIN" ]; then candidates+=("$MOSAIC_RUNTIME_BIN"); fi
# A host with no system Node gets one bootstrapped here by tools/install.sh, which
# records it in ~/.profile. The fleet unit runs `env -i ... bash --noprofile --norc`
# by design, so ~/.profile is never read and the directory has to be named here.
# The npm probe below cannot cover this: it reports a package prefix
# (~/.npm-global), never a Node runtime directory. It sits ahead of the npm probe so
# the bootstrapped runtime wins on a host that has both — that is the one the installer
# verified — while an explicit MOSAIC_RUNTIME_BIN still outranks it.
# Runtime binaries are `#!/usr/bin/env node`, so without this the pane resolves the
# binary and then dies on `env: 'node': No such file or directory`.
candidates+=("$PANE_HOME/.mosaic/node/current/bin")
if command -v npm >/dev/null 2>&1; then
local npm_prefix
npm_prefix=$(npm config get prefix 2>/dev/null) || true
@@ -348,93 +348,6 @@ for blocked in LD_PRELOAD= BASH_ENV= MOSAIC_UNTRUSTED_SENTINEL=; do
echo "$pane_environment" | grep -qF "$blocked" && fail "runtime pane received $blocked"
done
# #1256. On a host with no system Node, tools/install.sh bootstraps one into
# ~/.mosaic/node/ and writes that directory to ~/.profile. The fleet unit runs
# `env -i ... bash --noprofile --norc`, so ~/.profile is never read — correctly, by
# design — and _build_runtime_bin_prefix does not list the bootstrap directory. Its
# `npm config get prefix` branch cannot cover the gap either: the installer points
# npm's prefix at ~/.npm-global, so that branch contributes the npm-global directory
# and never the Node one, however it resolves.
#
# The property under test is not "the string is in PATH". It is that the pane can
# EXECUTE a Node-shebang runtime binary — which is what `mosaic` is
# (`#!/usr/bin/env node`) and what actually failed: measured on a greenfield VM as
# `env: 'node': No such file or directory` after a clean install that reported success.
#
# So this case runs the pane for real and requires it to have run. A PATH-substring
# assertion would pass on a fix that put the directory in the wrong position, and it
# would keep passing if the pane later stopped running for some unrelated reason.
: > "$TMUX_CALLS"
HOME_NODE="$ROOT/bootstrap-node/.config/mosaic"
write_generated "$HOME_NODE" "coder-node"
NODE_PANE_HOME="${HOME_NODE%/.config/mosaic}"
NODE_BOOTSTRAP_BIN="$NODE_PANE_HOME/.mosaic/node/current/bin"
mkdir -p "$NODE_BOOTSTRAP_BIN"
# The bootstrapped runtime. It records that it ran, which is the evidence this case
# turns on: no node reachable from the pane means no marker.
cat > "$NODE_BOOTSTRAP_BIN/node" <<'SHIM'
#!/usr/bin/env bash
set -euo pipefail
env -0 > "${MOSAIC_HOME:?}/fleet/pane-environment"
SHIM
chmod +x "$NODE_BOOTSTRAP_BIN/node"
# write_generated plants its symlinks under the MOSAIC_HOME it is given; here the
# pane's HOME is the trusted parent, so the pane's view of "installed" is this
# directory instead. `pi` is what #1241 resolves against PANE_PATH; `mosaic` is what
# the pane then executes, and it is a Node script — not a bash script that would run
# anywhere and quietly hide the defect.
mkdir -p "$NODE_PANE_HOME/.npm-global/bin"
ln -sf "$FAKE_BIN/pi" "$NODE_PANE_HOME/.npm-global/bin/pi"
printf '#!/usr/bin/env node\n' > "$NODE_PANE_HOME/.npm-global/bin/mosaic"
chmod +x "$NODE_PANE_HOME/.npm-global/bin/mosaic"
# The npm branch is modelled ALIVE and still cannot close the gap, which is the
# stronger statement. An earlier draft of this case tried to model npm as absent —
# true on a real bootstrap host, where npm lives only in the Node directory — and it
# refused to run anywhere npm is in the system path, i.e. most machines. It was also
# the weaker claim: it would have proven only that a dead branch supplies nothing.
#
# On a bootstrap host the installer sets npm's prefix to ~/.npm-global. So even with
# `command -v npm` true and the branch executing, `npm config get prefix` yields the
# npm-global directory and never the Node one. The gap does not depend on whether
# that branch runs.
NODE_LAUNCHER_BIN="$ROOT/bootstrap-node-launcher-bin"
mkdir -p "$NODE_LAUNCHER_BIN"
ln -sf "$FAKE_BIN/tmux" "$NODE_LAUNCHER_BIN/tmux"
ln -sf "$FAKE_BIN/npm" "$NODE_LAUNCHER_BIN/npm"
/usr/bin/env -i \
"HOME=$NODE_PANE_HOME" \
"PATH=$NODE_LAUNCHER_BIN:/usr/bin:/bin" \
"MOSAIC_HOME=$HOME_NODE" \
"MOSAIC_TEST_TMUX_CALLS=$TMUX_CALLS" \
"MOSAIC_TEST_HOME=$NODE_PANE_HOME" \
"MOSAIC_TEST_NPM_PREFIX=$NODE_PANE_HOME/.npm-global" \
MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \
MOSAIC_TEST_EXECUTE_PANE=1 \
"MOSAIC_TEST_PANE_PID=$$" \
"$START" coder-node
[ -f "$HOME_NODE/fleet/pane-environment" ] || \
fail "pane could not execute a Node-shebang runtime: $NODE_BOOTSTRAP_BIN is absent from PANE_PATH (#1256)"
node_pane_environment=$(tr '\0' '\n' < "$HOME_NODE/fleet/pane-environment")
# Colon-pad and match a whole element. A regex with `(^|:)` after `.*` looks like it
# does this and does not: an anchor cannot match mid-pattern, so it silently requires
# a leading colon and rejects the directory in FIRST position — which is where THIS
# FIXTURE puts it: it runs under `env -i` with no MOSAIC_RUNTIME_BIN, so the bootstrap
# directory leads. That is a property of the fixture, not of the fix — in general the
# directory sits second, after MOSAIC_RUNTIME_BIN. The colon padding makes the
# assertion position-independent either way, which is why it is written this way and
# not with an anchor. That produced a failure reading "pane ran but PANE_PATH does not
# carry <dir>" against a PATH whose first element was that dir.
node_pane_path=":$(printf '%s\n' "$node_pane_environment" | sed -n 's/^PATH=//p' | head -1):"
case "$node_pane_path" in
*":$NODE_BOOTSTRAP_BIN:"*) ;;
*) fail "pane ran but PANE_PATH does not carry $NODE_BOOTSTRAP_BIN (PATH=$node_pane_path)" ;;
esac
write_interaction_generated() {
local home="$1"
local agent="$2"
@@ -39,3 +39,20 @@ packages/mosaic/framework/tools/tmux/test-send-message-verdict.sh | requires rea
# recorded judgement. These lines ARE that judgement, signed.)
packages/mosaic/framework/tools/orchestrator/smoke-test.sh | behavior smoke checks for coord continue/run workflows, run manually by orchestrator seats; unmeasured in CI; #1017 burndown
packages/mosaic/framework/tools/wake/validate-973/microtest-wake-assert.sh | #973 instrument self-test, run as a precondition of the validate-973 evidence procedure rather than as a standing CI suite; #1017 burndown candidate
# --- tools/fleet: precondition is unsatisfiable in the CI image (#1271) ---
# Signed by fred (sb-it-1-dt, 2026-08-16) at origin/next 476db12.
# This suite asserts the launcher's behaviour when `mosaic` and `pi` are MISSING.
# It shims fakes into $FAKE_BIN, but the constructed PANE_PATH always ends in the
# real system path, so on a host that installs those binaries the missing-binary
# cases cannot be measured at all. The suite's own guard (line 103) says so and
# fails rather than reporting a pass it cannot back. That guard is correct.
# The error was wiring the suite into CI: #1017 (c56483eb) enumerated it and
# dropped this exclusion, and the CI image provides `pi` in the system path, so
# it has failed on every pipeline since. Measured 2026-08-16 across pipelines
# 2444 (#1256), 2438 (#1240) and 2441 (#1017-quality): exactly one FAIL line in
# each full log, identical, this assertion; control `zzz-not-present-zzz` -> 0.
# Burn-down and the full measurement are tracked in #1271; unwired by PR #1270.
# Because test:framework-shell is one && chain and this sat at position 44 of 48,
# the four suites after it had not run at all since the merge.
packages/mosaic/framework/tools/fleet/test-start-agent-session.sh | precondition unsatisfiable in the CI image: asserts missing-binary behaviour, but PANE_PATH always ends in the system path and the image provides `pi` there; guard at line 103 fails by design rather than passing unmeasured. Burn down by controlling the tail of PANE_PATH inside the test. NOT by removing `pi` from the image: the CI image installs @earendil-works/[email protected] deliberately (measured in pipeline 2444's test-step log), and other suites depend on that pin. Burn-down tracked in #1271
+1 -1
View File
@@ -25,7 +25,7 @@
"lint": "eslint src",
"typecheck": "tsc --noEmit",
"test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/fleet/test-start-agent-session.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh"
"test:framework-shell": "bash framework/tools/quality/scripts/check-test-enumeration.sh && bash framework/tools/quality/scripts/test-check-test-enumeration.sh && python3 src/lease-broker/daemon_deadline_unittest.py && python3 src/lease-broker/normative_fragments_unittest.py && python3 src/lease-broker/promotion_binding_unittest.py && python3 src/lease-broker/promotion_trigger_unittest.py && python3 src/lease-broker/receipt_challenge_unittest.py && python3 src/lease-broker/context_recovery_unittest.py && python3 src/lease-broker/recovery_runtime_unittest.py && python3 src/lease-broker/recovery_b1_adversarial_unittest.py && python3 src/lease-broker/receipt_observer_client_unittest.py && python3 src/lease-broker/invariant_r_unittest.py && python3 src/lease-broker/framework_skill_portability_unittest.py && python3 src/mutator-gate/runtime_tools_unittest.py && python3 src/mutator-gate/runtime_launch_guard_unittest.py && python3 src/mutator-gate/version_coupling_unittest.py && python3 framework/tools/lease-broker/check-runtime-launches.py --root ../.. && bash framework/tools/codex/test-pr-diff-context.sh && bash framework/tools/qa/test-deps-preflight.sh && bash framework/tools/git/test-pr-review-gitea-comment.sh && bash framework/tools/git/test-pr-review-repo-host-override.sh && bash framework/tools/git/test-ci-queue-wait-branch-absent.sh && bash framework/tools/git/test-ci-queue-wait-tristate.sh && bash framework/tools/git/test-ci-queue-wait-github-checks.sh && bash framework/tools/git/test-pr-merge-queue-branch.sh && bash framework/tools/git/test-pr-merge-head-pin.sh && bash framework/tools/git/test-pr-merge-message-field.sh && bash framework/tools/git/test-git-credential-mosaic.sh && bash framework/tools/git/test-gitea-token-identity.sh && bash framework/tools/woodpecker/test-terminal-green-contract.sh && bash framework/tools/_scripts/test-install-ordering-guard.sh && bash framework/tools/_scripts/test-mosaic-init-rce.sh && bash framework/tools/tmux/agent-send.test.sh && bash framework/tools/wake/test-wake-store-ack.sh && bash framework/tools/wake/test-wake-store-enqueue-race.sh && bash framework/tools/wake/test-wake-digest-hmac.sh && bash framework/tools/wake/test-wake-digest-quarantine.sh && bash framework/tools/wake/test-wake-detector.sh && bash framework/tools/wake/test-wake-fn-oracle.sh && bash framework/tools/wake/test-wake-reconcile.sh && bash framework/tools/wake/test-wake-beacon.sh && bash framework/tools/wake/test-wake-preimage.sh && bash framework/tools/wake/test-wake-install.sh && bash framework/tools/glpi/test-list-http-status.sh && bash framework/tools/orchestrator/test-board-roll.sh && bash framework/tools/woodpecker/test-ci-wait-exit-matrix.sh && bash framework/tools/_scripts/test-fleet-transport-check.sh"
},
"dependencies": {
"@mosaicstack/brain": "workspace:*",