worker worktrees run QA gates before pnpm install → eslint/prettier/tsc/vitest 'not found' (131/147 fleet errors) #856

New Issue

jason.woltje · 2026-07-20T08:54:49Z

jason.woltje commented

2026-07-20 08:54:49 +00:00

Problem

Worker git-worktrees under ~/agent-work/* run the mosaic-stack QA gate suite (pnpm test / lint / typecheck / format:check) before node_modules is installed, so every JS toolchain binary fails not found:

> @mosaicstack/mosaic@0.0.48 typecheck ... > tsc --noEmit
sh: 1: tsc: not found
> @mosaicstack/mosaic@0.0.48 lint    ... > eslint src
sh: 1: eslint: not found
> mosaic-stack@ format:check ... > prettier --check "**/*.{ts,tsx,js,jsx,json,md}"
sh: 1: prettier: not found
> @mosaicstack/mosaic@0.0.48 test    ... > vitest run --passWithNoTests
sh: 1: vitest: not found

Evidence (24h fleet digest, 2026-07-20 — digest-20260720T084758Z)

pi runtime contributed 131 of 147 fleet tool-errors this window; the dominant cluster is the four "not found" gate failures above. Confirmed across multiple independent worktrees, so this is systemic, not a one-off:

~/agent-work/wi6-833-constrained-recovery-command/packages/mosaic — node_modules/.bin MISSING (verified at both worktree root and packages/mosaic)
~/agent-work/review-wi5-832-{ba2716c1,e5526270,e196abfd}/packages/mosaic
~/agent-work/stack-wi5-832-build/packages/mosaic

Root cause

pnpm workspaces do not share node_modules across separate git worktrees — each fresh worktree needs its own pnpm install. The worker runs its delivery-cycle QA gates in a newly created worktree that was never installed, so node_modules/.bin is empty and every gate binary resolves to not found. (The correct flow — "create worktree → install dependencies → run gates" — is already documented in prior codex fix scratchpads.)

Neither tools/qa/qa-hook-stdin.sh nor tools/tmux/dispatch-fresh.sh performs (or preflights) the install; the gate suite is invoked by the worker itself, so there is no guardrail that node_modules exists first.

Blast radius

False-red gates: a not found exit is indistinguishable from a genuine test/lint failure to a worker — risks a worker treating installed-but-untested code as broken, or churning re-runs.
Wasted tool calls + error-signal noise: inflates the fleet error count (131/147 this window) and buries real signal.
Low severity per incident, but high frequency and fully preventable.

Proposed fix (candidates — MS-LEAD's lane to choose)

Bootstrap install (preferred): whatever creates the worker worktree (worker-launch / orchestrator worktree-add path) runs pnpm install --frozen-lockfile (or --offline against the shared store) immediately after git worktree add, before the worker begins its delivery cycle.
Gate-runner preflight: a thin wrapper the delivery cycle calls that checks for node_modules/.bin/<tool> and, if absent, either installs on demand or fails with an explicit deps not installed — run pnpm install message instead of raw sh: not found. This kills the false-red ambiguity even if (1) is skipped.

Either lands in the Mosaic framework (packages/mosaic/framework/tools/*, SSOT mosaicstack/stack).

Effort

Small — one install step at the worktree-bootstrap seam, or a ~20-line preflight wrapper. No behavior change to gates themselves.

Filed by enhance nightly pass 2026-07-20. Evidence: fleet digest digest-20260720T084758Z; pi sessions 2026-07-19T23-50-18…, …T21-31-31…, …T21-21-30…, 2026-07-20T04-03-54…. Shared-runtime → route/merge via Mos + MS-LEAD; enhance does not modify the framework directly (AGENTS.md rule 38).

## Problem Worker git-worktrees under `~/agent-work/*` run the mosaic-stack QA gate suite (`pnpm test` / `lint` / `typecheck` / `format:check`) **before `node_modules` is installed**, so every JS toolchain binary fails `not found`: ``` > @mosaicstack/mosaic@0.0.48 typecheck ... > tsc --noEmit sh: 1: tsc: not found > @mosaicstack/mosaic@0.0.48 lint ... > eslint src sh: 1: eslint: not found > mosaic-stack@ format:check ... > prettier --check "**/*.{ts,tsx,js,jsx,json,md}" sh: 1: prettier: not found > @mosaicstack/mosaic@0.0.48 test ... > vitest run --passWithNoTests sh: 1: vitest: not found ``` ## Evidence (24h fleet digest, 2026-07-20 — digest-20260720T084758Z) pi runtime contributed **131 of 147** fleet tool-errors this window; the dominant cluster is the four "not found" gate failures above. Confirmed across **multiple independent worktrees**, so this is systemic, not a one-off: - `~/agent-work/wi6-833-constrained-recovery-command/packages/mosaic` — `node_modules/.bin` **MISSING** (verified at both worktree root and `packages/mosaic`) - `~/agent-work/review-wi5-832-{ba2716c1,e5526270,e196abfd}/packages/mosaic` - `~/agent-work/stack-wi5-832-build/packages/mosaic` ## Root cause pnpm workspaces do **not** share `node_modules` across separate git worktrees — each fresh worktree needs its own `pnpm install`. The worker runs its delivery-cycle QA gates in a newly created worktree that was never installed, so `node_modules/.bin` is empty and every gate binary resolves to `not found`. (The correct flow — "create worktree → **install dependencies** → run gates" — is already documented in prior codex fix scratchpads.) Neither `tools/qa/qa-hook-stdin.sh` nor `tools/tmux/dispatch-fresh.sh` performs (or preflights) the install; the gate suite is invoked by the worker itself, so there is no guardrail that node_modules exists first. ## Blast radius - **False-red gates:** a `not found` exit is indistinguishable from a genuine test/lint failure to a worker — risks a worker treating installed-but-untested code as broken, or churning re-runs. - **Wasted tool calls + error-signal noise:** inflates the fleet error count (131/147 this window) and buries real signal. - Low severity per incident, but high frequency and fully preventable. ## Proposed fix (candidates — MS-LEAD's lane to choose) 1. **Bootstrap install (preferred):** whatever creates the worker worktree (worker-launch / orchestrator worktree-add path) runs `pnpm install --frozen-lockfile` (or `--offline` against the shared store) immediately after `git worktree add`, before the worker begins its delivery cycle. 2. **Gate-runner preflight:** a thin wrapper the delivery cycle calls that checks for `node_modules/.bin/<tool>` and, if absent, either installs on demand or fails with an explicit `deps not installed — run pnpm install` message instead of raw `sh: not found`. This kills the false-red ambiguity even if (1) is skipped. Either lands in the Mosaic framework (`packages/mosaic/framework/tools/*`, SSOT mosaicstack/stack). ## Effort Small — one install step at the worktree-bootstrap seam, or a ~20-line preflight wrapper. No behavior change to gates themselves. --- _Filed by `enhance` nightly pass 2026-07-20. Evidence: fleet digest digest-20260720T084758Z; pi sessions `2026-07-19T23-50-18…`, `…T21-31-31…`, `…T21-21-30…`, `2026-07-20T04-03-54…`. Shared-runtime → route/merge via Mos + MS-LEAD; enhance does not modify the framework directly (AGENTS.md rule 38)._

jason.woltje added the fleet-enhancement label 2026-07-20 08:54:49 +00:00

jason.woltje referenced this issue from a commit

2026-07-20 08:59:10 +00:00

mos: route #856 (worktree-deps, 89% errors, false-red gates) TOP park-window priority to MS-LEAD (approach-1 bootstrap install + approach-2 preflight defense, deployment-care seam); ack #841 2nd sighting + footgun hygiene + disk/ #72 carries

jason.woltje referenced this issue from a commit

2026-07-20 09:10:17 +00:00

mos: lock #857 named-executor clearance contract (CI-1942 terminal-green + durable exact-head RoR @2e8e8aa8 + Closes #835 + 6-check); concur #856 approach shape, deployment-timing stays mos

jason.woltje referenced this issue from a commit

2026-07-20 09:14:05 +00:00

mos: (repost) lock #857 named-executor clearance contract (CI-1942 terminal-green + durable exact-head RoR @2e8e8aa8 + Closes #835 + 6-check); concur #856 shape, deployment-timing stays mos

jason.woltje referenced this issue from a commit

2026-07-20 09:17:22 +00:00