stack/docs/scratchpads/rm-01-reproducible-checkout.md

RM-01 — Reproducible checkout

• Task/ref: RM-01 (docs/remediation/TASKS.md, internal mission tracking)
• Objective: make checkout/install/typecheck hooks fail on code rather than environmental residue, for root CI and non-root seats.
• Scope: pnpm store configuration, transactional Husky installation, dependency/generated-state preflight, checkout regression tests, developer documentation.
• Constraints: isolated worktree; no skip-switch fixes; no writes under /root or /tmp; workers do not edit docs/remediation/TASKS.md; author does not review or merge.
• Acceptance: AC1–AC8 from the orchestrator dispatch/addendum.
• Plan:
  1. Add RED-first tests for missing dependencies, stale/foreign .next, and interrupted hook installation.
  2. Implement environment-overridable HOME-based pnpm store defaults, deterministic preflight, and transactional hook installation.
  3. Run focused tests, install/build/baseline gates, and explicit AC negative controls.
  4. Obtain independent review, push after queue guard, open PR, and send evidence to mos-remediation.
• Budget: orchestrator estimate 6K/60K; no explicit hard token cap. Keep scope to RM-01 and avoid unrelated cleanup.
• Risks: 97%-full shared /tmp; native dependency install size; root-owned fixtures may require Docker for realistic verification.

Progress / evidence

• Worktree created at /home/hermes/agent-work/rm-01 from origin/main 06e0d403.
• /tmp baseline: 28G used, 889M available (97%); worktree and planned store are on /home.
• Root causes confirmed from source: committed .npmrc pins /root; prepare invokes Husky directly; web typecheck includes generated .next types without validating ownership/freshness.

Checkpoint evidence (c45e5e19)

• AC1 IN PROGRESS: non-root pnpm install --frozen-lockfile --store-dir "$HOME/.local/share/pnpm/store" exited 0; pnpm exec turbo run typecheck --force exited 0 (45/45 uncached). Clean CI-container run not performed.
• AC2 DONE: with node_modules absent, pnpm preflight exited 42 with MOSAIC_PREFLIGHT_MISSING_DEPS and run pnpm install; after install it exited 0.
• AC3 DONE: appending export const x: number = "s" to packages/types/src/index.ts made pnpm -w typecheck exit 2 with TS2322; reverting made it exit 0.
• AC4 IN PROGRESS: local pnpm -w build exited 0 and git status --porcelain showed no generated residue beyond the intended RM-01 source changes. Fresh-clone proof not performed.
• AC5 DONE: non-root install exited 0; pnpm store path resolved /home/hermes/.local/share/pnpm/store/v10; no /root write was attempted.
• AC6 IN PROGRESS: focused failure/rollback tests passed, but final review found a concurrent-install race. Two installers can both observe .husky/_ absent; after one installs successfully, the losing install's catch path can quarantine the winner's active hooks and restore stale Git config (scripts/install-hooks.mjs, activation/catch transaction). A RED regression is committed after the checkpoint.
• AC7 DONE: install/store/worktree were on /home; full pnpm -w build exited 0; /tmp usage changed by 4096 bytes during the build (23,805,173,760 → 23,805,177,856 bytes), not materially.
• AC8 DONE for the implemented path: store resolves under $HOME; test/quarantine/build state resolves under the worktree; no implemented component requires a writable path outside $HOME or the worktree.

Continuation evidence

• AC6 DONE: the committed race reproducer was observed RED (node --test --test-name-pattern='a competing successful installer is not removed by the losing process' scripts/install-hooks.test.mjs, exit 1/ENOENT), then passed after cleanup became ownership-safe. The losing installer never removes an active hook set or restores Git configuration it did not activate. pnpm test:checkout passes 21/21, exit 0, including the original race and a post-rename peer-replacement regression.
• Generated-state remediation: replaced mtime inference with a source/build-input fingerprint, written only after a serialized successful Next build with unchanged inputs. Failed/interrupted/overlapping builds leave no trusted marker. The fingerprint uses Next's own environment loader, covers resolved NEXT_PUBLIC_* values, inherited TypeScript configuration, lock/workspace inputs, and rejects symlink inputs.
• Baseline: pnpm typecheck, pnpm lint, and pnpm format:check each exit 0. Local pnpm test still exits 97 only at the pre-existing Bash BASH_LINENO convention guard (#973/#1003), after checkout tests and package tests pass; this is not reported as a green full-suite result.
• Automated review remediation: resolved findings for peer-hook ownership, stale/failed build markers, build-input changes, expanded environment inputs, inherited TypeScript config, symlink inputs, and overlapping build serialization. Independent PR review remains assigned to rev-974.
• AC1 DONE at 0f706119: a clean clone created inside git.mosaicstack.dev/mosaicstack/stack/ci-base:latest ran the exact acceptance sequence pnpm install --frozen-lockfile && pnpm -w typecheck; exit 0 with 45/45 uncached typecheck tasks successful. An earlier bind-mounted clone attempt exited 1 because root in the container rejected the host-owned Git directory; that failed attempt is not counted as evidence.
• AC4 DONE at 0f706119: in that same fresh clone and CI image, pnpm -w build completed 25/25 tasks and the immediately following git status --porcelain was empty; combined assertion exit 0.
• Push BLOCKED after the required queue guard: git push origin fix/rm-01-reproducible-checkout was rejected by Gitea with User permission denied for writing / pre-receive hook declined, despite MOSAIC_GIT_IDENTITY=f10-coder resolving username f10-coder from the provisioned gitea-mosaicstack-f10-coder.token.

Review remediation — restated AC2

• Independent review correctly found that an added symlink under a successfully built .next tree passed preflight. The exact reviewer control, ln -s /etc/hosts apps/web/.next/reviewer-symlink && pnpm preflight, was observed passing before remediation.
• The original blanket symlink wording conflicts with AC4 because canonical Next output: 'standalone' emits legitimate pnpm dependency symlinks. The coordinator independently verified 42 such links and approved the operative restatement: .next itself must not be a symlink; descendant symlinks must exactly match the successful build's certified manifest.
• RED-first controls were observed failing together against the prior implementation (exit 1): .next root, added, removed, retargeted, tampered-manifest, and canonical-style certified-link cases. The build now publishes the manifest atomically before the existing source certification commit marker; that marker binds the manifest SHA-256. Missing/partial/modified manifests remain untrusted.
• GREEN evidence: the six-case symlink control passes; the exact reviewer-added link exits 43; removing it restores preflight exit 0. The added RED-first build-publication control also proves a symlinked .next cannot redirect certification writes outside the checkout. pnpm test:checkout passes 23 top-level tests / 29 including subtests. Canonical pnpm --filter @mosaicstack/web build and the following pnpm preflight both exit 0.
• Threat-model ruling: the manifest detects accidental, independent, stale, and foreign-residue mutation—the class exposed by the five-month-stale .next that produced 19 phantom TS2307 errors. It does not defend against a same-UID actor able to rewrite both manifest and marker consistently (CWE-345); no local worktree construction can without an external trust anchor. RM-59 tracks the residual: executor/spine-side attestation outside worktree authority, dependent on RM-12, RM-21, and RM-25.
• AC8 concrete proof at df7530ae: a clean clone ran in ci-base:latest with Docker --read-only; its only writable mounts were /workspace (the worktree) and /home/ci (HOME, with NPM_CONFIG_STORE_DIR=/home/ci/store). pnpm install --frozen-lockfile && pnpm -w typecheck exited 0 with 45/45 uncached tasks. This proves the implemented checkout path requires no writable location outside $HOME and the worktree. An initial fixture attempt failed only because Git required /workspace safe-directory setup; it is not counted as evidence.

Handoff

1. Keep the newly committed RED tests red until implementing: (a) source-fingerprint marker support for valid incremental .next output, and (b) ownership-safe concurrent hook activation.
2. The latest automated review rejected oldest-generated-file mtime as a false positive for valid incremental Next output. Use a source-content fingerprint marker written only after successful next build; do not continue tuning mtimes.
3. For Husky, generation in an isolated temporary Git repo avoids mutating real core.hooksPath during staging. Preserve that design. Fix the losing concurrent process so it never removes a peer's completed hook set or restores stale config.
4. Codex review runs in a read-only sandbox, so its attempts to run the fixture-writing Node tests report opaque test-file failures. The same tests run normally in the worktree.
5. Full pnpm test is not green on this host: it exits 97 at the pre-existing Bash BASH_LINENO convention guard (#1003), after the changed checkout tests and package tests pass. Do not weaken that gate.