RM-01: reproducible checkout — the pre-push gate fails on code, not environment (#1027)

Co-authored-by: mos-dt-0 <[email protected]>

docs/scratchpads/rm-01-reproducible-checkout.md

@@ -0,0 +1,58 @@
+# RM-01 — Reproducible checkout
+
+- Task/ref: RM-01 (`docs/remediation/TASKS.md`, internal mission tracking)
+- Objective: make checkout/install/typecheck hooks fail on code rather than environmental residue, for root CI and non-root seats.
+- Scope: pnpm store configuration, transactional Husky installation, dependency/generated-state preflight, checkout regression tests, developer documentation.
+- Constraints: isolated worktree; no skip-switch fixes; no writes under `/root` or `/tmp`; workers do not edit `docs/remediation/TASKS.md`; author does not review or merge.
+- Acceptance: AC1–AC8 from the orchestrator dispatch/addendum.
+- Plan:
+  1. Add RED-first tests for missing dependencies, stale/foreign `.next`, and interrupted hook installation.
+  2. Implement environment-overridable HOME-based pnpm store defaults, deterministic preflight, and transactional hook installation.
+  3. Run focused tests, install/build/baseline gates, and explicit AC negative controls.
+  4. Obtain independent review, push after queue guard, open PR, and send evidence to `mos-remediation`.
+- Budget: orchestrator estimate 6K/60K; no explicit hard token cap. Keep scope to RM-01 and avoid unrelated cleanup.
+- Risks: 97%-full shared `/tmp`; native dependency install size; root-owned fixtures may require Docker for realistic verification.
+
+## Progress / evidence
+
+- Worktree created at `/home/hermes/agent-work/rm-01` from `origin/main` `06e0d403`.
+- `/tmp` baseline: 28G used, 889M available (97%); worktree and planned store are on `/home`.
+- Root causes confirmed from source: committed `.npmrc` pins `/root`; `prepare` invokes Husky directly; web typecheck includes generated `.next` types without validating ownership/freshness.
+
+## Checkpoint evidence (c45e5e19)
+
+- AC1 IN PROGRESS: non-root `pnpm install --frozen-lockfile --store-dir "$HOME/.local/share/pnpm/store"` exited 0; `pnpm exec turbo run typecheck --force` exited 0 (45/45 uncached). Clean CI-container run not performed.
+- AC2 DONE: with `node_modules` absent, `pnpm preflight` exited 42 with `MOSAIC_PREFLIGHT_MISSING_DEPS` and `run pnpm install`; after install it exited 0.
+- AC3 DONE: appending `export const x: number = "s"` to `packages/types/src/index.ts` made `pnpm -w typecheck` exit 2 with TS2322; reverting made it exit 0.
+- AC4 IN PROGRESS: local `pnpm -w build` exited 0 and `git status --porcelain` showed no generated residue beyond the intended RM-01 source changes. Fresh-clone proof not performed.
+- AC5 DONE: non-root install exited 0; `pnpm store path` resolved `/home/hermes/.local/share/pnpm/store/v10`; no `/root` write was attempted.
+- AC6 IN PROGRESS: focused failure/rollback tests passed, but final review found a concurrent-install race. Two installers can both observe `.husky/_` absent; after one installs successfully, the losing install's catch path can quarantine the winner's active hooks and restore stale Git config (`scripts/install-hooks.mjs`, activation/catch transaction). A RED regression is committed after the checkpoint.
+- AC7 DONE: install/store/worktree were on `/home`; full `pnpm -w build` exited 0; `/tmp` usage changed by 4096 bytes during the build (23,805,173,760 → 23,805,177,856 bytes), not materially.
+- AC8 DONE for the implemented path: store resolves under `$HOME`; test/quarantine/build state resolves under the worktree; no implemented component requires a writable path outside `$HOME` or the worktree.
+
+## Continuation evidence
+
+- AC6 DONE: the committed race reproducer was observed RED (`node --test --test-name-pattern='a competing successful installer is not removed by the losing process' scripts/install-hooks.test.mjs`, exit 1/ENOENT), then passed after cleanup became ownership-safe. The losing installer never removes an active hook set or restores Git configuration it did not activate. `pnpm test:checkout` passes 21/21, exit 0, including the original race and a post-rename peer-replacement regression.
+- Generated-state remediation: replaced mtime inference with a source/build-input fingerprint, written only after a serialized successful Next build with unchanged inputs. Failed/interrupted/overlapping builds leave no trusted marker. The fingerprint uses Next's own environment loader, covers resolved `NEXT_PUBLIC_*` values, inherited TypeScript configuration, lock/workspace inputs, and rejects symlink inputs.
+- Baseline: `pnpm typecheck`, `pnpm lint`, and `pnpm format:check` each exit 0. Local `pnpm test` still exits 97 only at the pre-existing Bash `BASH_LINENO` convention guard (#973/#1003), after checkout tests and package tests pass; this is not reported as a green full-suite result.
+- Automated review remediation: resolved findings for peer-hook ownership, stale/failed build markers, build-input changes, expanded environment inputs, inherited TypeScript config, symlink inputs, and overlapping build serialization. Independent PR review remains assigned to rev-974.
+- AC1 DONE at `0f706119`: a clean clone created inside `git.mosaicstack.dev/mosaicstack/stack/ci-base:latest` ran the exact acceptance sequence `pnpm install --frozen-lockfile && pnpm -w typecheck`; exit 0 with 45/45 uncached typecheck tasks successful. An earlier bind-mounted clone attempt exited 1 because root in the container rejected the host-owned Git directory; that failed attempt is not counted as evidence.
+- AC4 DONE at `0f706119`: in that same fresh clone and CI image, `pnpm -w build` completed 25/25 tasks and the immediately following `git status --porcelain` was empty; combined assertion exit 0.
+- Push BLOCKED after the required queue guard: `git push origin fix/rm-01-reproducible-checkout` was rejected by Gitea with `User permission denied for writing` / `pre-receive hook declined`, despite `MOSAIC_GIT_IDENTITY=f10-coder` resolving username `f10-coder` from the provisioned `gitea-mosaicstack-f10-coder.token`.
+
+## Review remediation — restated AC2
+
+- Independent review correctly found that an added symlink under a successfully built `.next` tree passed preflight. The exact reviewer control, `ln -s /etc/hosts apps/web/.next/reviewer-symlink && pnpm preflight`, was observed passing before remediation.
+- The original blanket symlink wording conflicts with AC4 because canonical Next `output: 'standalone'` emits legitimate pnpm dependency symlinks. The coordinator independently verified 42 such links and approved the operative restatement: `.next` itself must not be a symlink; descendant symlinks must exactly match the successful build's certified manifest.
+- RED-first controls were observed failing together against the prior implementation (exit 1): `.next` root, added, removed, retargeted, tampered-manifest, and canonical-style certified-link cases. The build now publishes the manifest atomically before the existing source certification commit marker; that marker binds the manifest SHA-256. Missing/partial/modified manifests remain untrusted.
+- GREEN evidence: the six-case symlink control passes; the exact reviewer-added link exits 43; removing it restores preflight exit 0. The added RED-first build-publication control also proves a symlinked `.next` cannot redirect certification writes outside the checkout. `pnpm test:checkout` passes 23 top-level tests / 29 including subtests. Canonical `pnpm --filter @mosaicstack/web build` and the following `pnpm preflight` both exit 0.
+- Threat-model ruling: the manifest detects accidental, independent, stale, and foreign-residue mutation—the class exposed by the five-month-stale `.next` that produced 19 phantom TS2307 errors. It does not defend against a same-UID actor able to rewrite both manifest and marker consistently (CWE-345); no local worktree construction can without an external trust anchor. RM-59 tracks the residual: executor/spine-side attestation outside worktree authority, dependent on RM-12, RM-21, and RM-25.
+- AC8 concrete proof at `df7530ae`: a clean clone ran in `ci-base:latest` with Docker `--read-only`; its only writable mounts were `/workspace` (the worktree) and `/home/ci` (`HOME`, with `NPM_CONFIG_STORE_DIR=/home/ci/store`). `pnpm install --frozen-lockfile && pnpm -w typecheck` exited 0 with 45/45 uncached tasks. This proves the implemented checkout path requires no writable location outside `$HOME` and the worktree. An initial fixture attempt failed only because Git required `/workspace` safe-directory setup; it is not counted as evidence.
+
+## Handoff
+
+1. Keep the newly committed RED tests red until implementing: (a) source-fingerprint marker support for valid incremental `.next` output, and (b) ownership-safe concurrent hook activation.
+2. The latest automated review rejected oldest-generated-file mtime as a false positive for valid incremental Next output. Use a source-content fingerprint marker written only after successful `next build`; do not continue tuning mtimes.
+3. For Husky, generation in an isolated temporary Git repo avoids mutating real `core.hooksPath` during staging. Preserve that design. Fix the losing concurrent process so it never removes a peer's completed hook set or restores stale config.
+4. Codex review runs in a read-only sandbox, so its attempts to run the fixture-writing Node tests report opaque test-file failures. The same tests run normally in the worktree.
+5. Full `pnpm test` is not green on this host: it exits 97 at the pre-existing Bash `BASH_LINENO` convention guard (#1003), after the changed checkout tests and package tests pass. Do not weaken that gate.