stack/docs/scratchpads/1050-install-state-machine-red-fixture.md

# #1050 — Installer P0–P9 state machine and red-first fixture

## Objective

Implement C1 from the canonical greenfield-install PRD v2: a transactional P0–P9 installer spine, a side-effect-free P0–P8 `--check`, and a lane-parametric Debian/glibc non-root from-zero fixture. The acceptance milestone is an attributable RED on the pre-C1 installer while preserving P3 PASS.

## Authority and scope

- Canonical requirements: `jason.woltje/jarvis-brain` `docs/plans/2026-08-04-greenfield-install-blockers-PRD-v2.md`. Currency was re-derived after compaction: authenticated fetch resolved `origin/main` to `cb23e5fbc8a282fa967b93d7a134fa48d11b4bb1`; the PRD and charters are byte-identical to the previously read remote copies.
- Tracking: `mosaicstack/stack#1050` on `git.mosaicstack.dev` (author read back as `be-coder-05`).
- Historical implementation base: `origin/next` `4df478cdd150fdf8d52ea109f02ade5d85017acd`. Delivery PR #1054 targets `main` under L0's trunk-only rule; `next` remains a non-merging integration lane.
- Out of scope: PATH, skills, headless wizard/identity, activation remediation, #869 wiring, RM-02, main promotion.
- `docs/TASKS.md` is orchestrator-single-writer and is not modified by this worker.

## Plan

1. Pre-register the canonical phase/output/side-effect-free/fault-injection checks and observe RED against the base installer.
2. Commit the immutable red-first acceptance fixture before implementation.
3. Add the state-machine/journal/postcondition spine without repairing P4/P5/P8 symptoms.
4. Wire the expected-RED from-zero fixture into Woodpecker using Debian/glibc and a non-root target user.
5. Run shell/static baselines, situational container validation, code review, security review, then deliver through a PR to `next` under the coordinator-owned merge path.

## Budget

- Working estimate: 32K reasoning/output tokens.
- Hard external cap: none stated.
- Adaptation: keep implementation in shell surfaces already in scope; no package dependency install unless repository gates require it.

## Pre-registered acceptance checks

| ID | Exact case | Expected pre-fix result |
|---|---|---|
| C1-R1 | `tools/e2e-install-test.sh --lane next` in a clean Debian 12 container as uid 1001 | non-zero; P3 PASS; P4 `NOT-MEASURED / UNDECLARED`; P5/P6/P8 FAIL with own reasons |
| C1-R2 | `tools/install-state-machine.test.sh` phase table case | RED because base installer does not enumerate canonical P0–P9 contracts |
| C1-R3 | side-effect-free `--check` case over a fingerprinted HOME | RED because base `--check` is version-only rather than P0–P8 predicates |
| C1-R4 | fault injection after each P2…P8 | RED because base installer has no injectable durable journal/rollback state |
| C1-R5 | Docker unavailable | base harness incorrectly exits 0; replacement must fail non-zero |
| C1-R6 | lane resolution | bare checkout is forbidden; fixture must pass `--next` and assert the resolved prerelease version |
| C1-R7 | same Debian fixture with `git` absent vs present | absent: P1 FAIL while legacy installer exits 0 and sync degrades; present: P1 PASS and observed store/runtime containment 101/101 |

## Progress

- [x] Charter, doctrine, delivery/CI/QA/docs guides read and re-anchored after compaction.
- [x] Canonical PRD v2/v3 addenda and charters read from fetched `origin/main`; numbering reconciles with the TL spec. No numbering conflict found. INV-B/C/D are binding and implemented without renumbering.
- [x] Target base reachability verified with `merge-base --is-ancestor`.
- [x] Issue #1050 created and provider author read back.
- [x] Initial RED captured; TL rejected P4's repo-root count as a false RED. Four populations disagree (framework payload 1, repo root 13, sync store 101 in the fixture, W-jarvis observation 7), so C1 now requires a checkout-free declared shipped-set artifact and reports P4 `NOT-MEASURED / UNDECLARED` until C5 supplies it.
- [x] P6 strengthens #869: the two dead enforcement hooks reproduce from zero on a clean broker-less container. C1 asserts the breach but neither wires nor unwires it.
- [x] P1 false pass identified from the P4 evidence row: `git` is absent from the Debian base and was undeclared even though skill sync shells out to it. C1 adds `git` to P1; the fixture matrix preserves absent/present controls. The prior claim that web1's missing runtime skills reproduce this greenfield mechanism is withdrawn by the TL and is not carried here.
- [x] Corrected RED transcript captured and reported, including the git-present/absent controls and strict P3 PASS.
- [x] State-machine implementation complete: private pre-mutation journal/snapshot, P0–P8 `--check`, P2–P8 fault seam, rollback, durable manifest/journal seal, action-status persistence, safe rollback roots, and stale-projection recovery.
- [x] Debian/glibc checkout fixture now packages the complete current checkout, verifies its digest in-container, and reaches the expected attributable RED without host inheritance. CI compares its exact final phase map/reasons to `tools/fixtures/greenfield-expected-red.tsv`; the fixture remains red while the detector job is green only on an exact match.
- [ ] Reviews complete. Reviews 80 (`rev-security-02`) and 81 (`rev-974`) requested changes at `3934e03f`; their eight non-overlapping detector findings are being remediated red-first. Current remediation adds canonical-image portability, absolute P3 CLI propagation, exact expected-RED schema/cardinality, passwd-HOME binding, created-path owner/mode policy, real-action P2–P8 fault injection, verified non-empty remote installer execution, and seeded secret-canary/redacted diagnostics. Both old verdicts become void when the remediation head moves and require fresh independent review.
- [x] Successor remediation for review 90 is RED-first and recorded in `docs/reports/verification/1050-successor-remediation/`: the manifest now binds the complete supported final P0 reason; P4 rejects an incomplete created-path walk instead of discarding `find` failure; and the TERM no-exit control is independent of filesystem enumeration order while retaining a proven RED mutation. Pipeline 2224's 32/2 result was a path-order-sensitive control, not evidence that the resume bug's premise became stale. The enumeration-class sweep additionally replaced order-dependent `find | head -1` source-root selection with a checked complete inventory requiring exactly one extracted root.
- [x] C1 fix round for reviews 92/93: Blocker B was completed first. RED was reproduced before implementation: a planted `{"status":"in-progress"}` record makes the complete real walk fail the suite at P2, while the same planted defect beneath a target-owned mode-0100 directory makes real `find` fail and the frozen suite falsely exit 0 with `installer next lane tests passed`. One shared helper now captures and checks the complete NUL-delimited population before testing absence across the primary and copied harness sites; grep no-match is distinct from read failure. The same attack child drives the full suite RED with a named enumeration error, while ordinary native and root/container runs pass. Blocker A then regenerated the exact installer sidecar; provider-fetch validation remains the required post-push bar. Working estimate: 16K tokens; no external hard cap; 60% context is the stop/report gate.

## Risks / blockers

- The deployed create wrappers do not expose `--dry-run`; identity preflight was performed through `pr-merge.sh --dry-run` on the same HOMELAB repo, which resolved `git.mosaicstack.dev` + `be-coder-05`. The issue create then fell back from tea to the API but provider read-back confirmed author `be-coder-05`.
- `next` is a non-merging integration lane; PR #1054 targets `main`. The old “pending promotion to main” caution dissolved when the base moved. #1050 remains open after merge and closes only after Jarvis validates the greenfield behavior.
- #869 must remain staged and inactive.
- Late sequencing input MB-BRAIN-01 is accommodated without implementation or renumbering: P2 covers installer distribution only; P5 owns requested credential capability; P7 leaves an ordered seam for credential-dependent resource provisioning after P5.

## Remediation review controls

- B1 RED: the next-lane harness failed immediately under `ci-base:latest` as root/musl; it now models uid 1001/glibc explicitly and uses Python tree fingerprints instead of GNU `find -printf`.
- B2 RED: framework/runtime linking consumed bare `mosaic` from PATH after P3 had committed an absolute path. The unified installer now exports/passes `MOSAIC_CLI_PATH`; the linker invokes that absolute artifact, and wizard auto-launch has no stale-PATH fallback.
- B3 RED: a one-row manifest (`exit=1`) certified any exit-1 log. Full-manifest validation now requires the exact three cases, one exit and P0–P9 row each, pinned require/forbid populations, and rejects malformed/duplicate/unknown rows; shrink is a negative control.
- B4 RED: uid 1001 with a passwd HOME different from ambient HOME produced P0 PASS. P0 now binds uid, username, passwd HOME and shell and explicitly rejects root and sudo-with-inherited-HOME controls.
- B5 RED: mode-0777 CLI, mode-0644 identity, and mode-0755 credential storage passed. P3/P4/P5 now apply target owner/group plus executable/shared/private policies; framework credential storage is created 0700.
- B6 RED: fault injection only wrote `.selftest-*` files. The synthetic path was removed; the P2–P8 matrix enters the normal action flow, proves an action observation occurred, injects after each real phase, and fingerprints rollback.
- B7 RED: an HTTP-200 empty body exits zero when piped to Bash. The fetched installer must now be non-empty, digest-equal to `tools/install.sh.sha256`, and that exact file is executed; failed/empty/mismatch controls are blocking and CI has a remote immutable-commit arm.
- B8 RED: raw combined command output was duplicated to terminal and `commands.log`. Both capture layers now redact before output/persistence; a seeded canary is positively emitted by the fake credential-capable registry and must remain absent from terminal, command log, npmrc, generated files and observed argv. The real greenfield fixture also scans those populations.
- Advisory code review findings are fixed: URL userinfo redaction now handles raw `@`, token-only and percent-encoded forms, repeated `:`, multiple URLs, Authorization/Basic, npm `_auth`, and Cookie headers in both capture layers; the real greenfield path positively emits its canary through `state_run_captured`; verified-fetch removes its temporary body after successful execution; and plaintext diagnostics exist only in process-substitution pipes rather than interruptible temporary files.
- Advisory security review's independent trust-root finding is **DEFERRED by canonical PRD v2 §3**, which explicitly excludes signed provenance. README now states precisely that the same-origin sidecar detects empty/corrupt/inconsistent publication but cannot authenticate against repository/server compromise; no stronger claim remains.
- The web1 no-manifest representativeness observation is recorded but intentionally not acted on: it is explicitly outside these eight blockers. This remediation does not weaken or otherwise change P9's manifest-presence assertion.

## Verification log

- `bash -n` and ShellCheck pass for all changed shell surfaces; `git diff --check` passes.
- `bash tools/install-state-machine.test.sh` passes, including exact P0–P8 rows, passwd-HOME/privilege discrimination, owner/group/mode attacks, persisted P4/P6 action failures, no synthetic fault implementation, unsafe/overlapping/symlink roots, and fatal journal initialization.
- `bash tools/install-next-lane.test.sh` passes inside `ci-base:latest`, including exact `@next` versions, immutable source fallback, source-build/archive-failure rollback, offline `--dev`, explicit refs, prerelease suffix mismatch, absolute P3 CLI propagation, secret redaction, real-action P2–P8 rollback, and stale projection recovery.
- Comparator controls pass for verdict drift, unexpected exit, manifest shrink, missing phases, duplicate rows, unknown cases, and unknown kinds. Verified-fetch controls pass for successful execution and failed/empty/digest-mismatch rejection.
- `bash tools/e2e-install-test.sh --lane next --source checkout --git present` returns the required expected RED in clean Debian/glibc as uid 1001: installer P0/P1/P2/P3/P7 PASS; P4/P5/P6/P8 and P9 blocking; no `Done.` claim; checkout archive digest pinned and current framework installer exercised. `tools/verify-greenfield-expected-red.sh` converts that expected detector result into a green CI assertion and fails on any unreviewed verdict drift.
- Earlier repository gates passed: `pnpm typecheck`, `pnpm lint`, `pnpm format:check`, upgrade manifest/rollback/durable-snapshot/migration suites, and focused `@mosaicstack/mosaic` tests with an isolated npm prefix. Full exact-remediation rerun is required before push.
- Review-93 RED evidence at frozen `378bc1a`: isolated positive-control full-suite exit `1` with `P2 left an in-progress transaction`; isolated permission-failure attack full-suite exit `0` with final `installer next lane tests passed`. No binary shadowing or PATH interception was used; the failure came from a real target-owned mode-0100 directory.
- Fix-round GREEN: `pnpm test:installer`, native next-lane, root/`ci-base:latest` next-lane, state-machine, verified-fetch, Bash syntax, ShellCheck, `pnpm typecheck`, `pnpm lint`, `pnpm format:check`, and `git diff --check` pass. The direct attack child exits 1 and names `P2 fault-state enumeration failed`. All 81 Mosaic Vitest files / 1508 tests pass under an isolated npm prefix; the wider framework-shell chain reaches the pre-existing #973 Bash-line-number gate and exits 97, matching the known host-specific condition rather than this delta. Codex code review approved at 0.93 confidence with zero findings; Codex security review reported no risk at 0.96 confidence with zero findings.
- Sidecar RED-first documentation control from the repository root: `sha256sum -c tools/install.sh.sha256` exits 1 with `install.sh: FAILED open or read` because the sidecar path is relative to `tools/`. Sidecar GREEN with the command actually executed: `(cd tools && sha256sum -c install.sh.sha256)` exits 0 with `install.sh: OK`; the workflow's separate expected/actual comparison resolves `e59cb441a2f37ae9150f8eae470238e9d858a1816df93343d9784a6796676096`. This is not substituted for the required immutable provider-fetch arm at the pushed head.

## Round 3 — remote-arm downstream source binding

### Objective and constraints

- Tracking remains `mosaicstack/stack#1050`, delivery PR #1054, branch `feat/1050-install-state-machine-red-fixture`; `docs/TASKS.md` remains orchestrator-single-writer.
- Bind the remote arm's downstream stack framework/source archive to the same immutable `${CI_COMMIT_SHA}` as the digest-verified `install.sh`, while retaining `--next` and its exact resolved CLI/gateway lane-version assertion.
- RED first: the realised source commit/digest assertion must reject the existing stale `origin/next` substitution; GREEN must restore the exact expected `P6 FAIL` and pass the comparator. R7 then deletes the binding and requires RED again.
- Fenced out: the expected-RED manifest (including `P6=FAIL`), #869 activation, and #1068 digest-before-comparator sequencing.
- Push budget: one force-with-lease push pinned to provider head `ff3f0d29f1763bed44a60610d073036112e66b77`; run the queue guard first; do not poll CI after push.
- Working estimate: 14K reasoning/output tokens; no external hard token cap. Scope reduction order: reuse the existing local-source archive seam, add one realised-state assertion, avoid installer behavior changes.

### Corrected reference citation

The transferable reference is only **the container-image acquisition path in `fa-install.sh`**: resolve mutable input once to `Docker-Content-Digest`, fail closed if unresolved, render `@sha256`, then verify running images against the resolved digest. It is not a claim about USC's installer as a whole; the sibling root bundle extraction in `Install-FieldAgentOnPanel.ps1:82` is unpinned. The applicable structure is `RESOLVE-ONCE -> PIN -> FAIL-CLOSED ON RESOLUTION -> VERIFY-AFTER`, applied per acquisition path.

### Remote-arm acquisition-path census before the fix

Code-path enumeration only; execution of the complete list is **NOT MEASURED** in this round yet. Logical payload paths are counted once even when one command is invoked repeatedly or installs a dependency graph.

| # | Acquisition path taken by the arm | Pre-fix binding state | Scope disposition |
|---|---|---|---|
| 1 | Woodpecker checkout containing the workflow, fixture, sidecar, and comparator | CI commit checkout (commit-bound by runner contract) | existing arm substrate |
| 2 | `node:22-bookworm-slim` step image | mutable image tag | unpinned sibling; outside this fix fence |
| 3 | Debian package set acquired by `apt-get update/install` | repository-current, no package versions/snapshot | unpinned sibling; outside this fix fence |
| 4 | remote `tools/install.sh` | exact `${CI_COMMIT_SHA}` URL plus checked SHA-256 | in scope; already bound first hop |
| 5 | `@mosaicstack/mosaic@next` CLI package graph | mutable lane resolved to an exact top-level version, exact version installed, realised top-level version verified | lane assertion; must remain |
| 6 | `@mosaicstack/gateway@next` package graph | later re-resolution wins and is verified against itself; drift from the first value can be silently accepted | partially constrained TOCTOU sibling; outside this fix |
| 7 | `mosaicstack/stack` framework/source archive consumed by `ensure_monorepo` | `next` resolves once to `4df478cdd150fdf8d52ea109f02ade5d85017acd`, not the arm's CI commit | **in-scope defect; bind to CI commit** |
| 8 | sequential-thinking MCP package warmed via unversioned `npx -y @modelcontextprotocol/server-sequential-thinking` | mutable package resolution | unpinned sibling; outside this fix fence |
| 9 | Excalidraw npm dependency graph | shipped `package-lock.json` exact versions + registry integrity hashes; `npm install` consumes the lock | already lock/integrity-bound, separate from CI-source identity |
| 10 | canonical skills repo cloned by `mosaic-sync-skills` | mutable default branch of `mosaic/agent-skills.git` | unpinned sibling; outside this fix fence |

Pre-fix mechanism breakdown (do not blend unlike strengths): **2 / 10 intended-CI-commit-bound** (#1 checkout, #4 installer); **1 / 10 resolve-then-pin with comparison to the authoritative value** (#5 CLI); **1 / 10 partially constrained because verify-after compares #6 gateway to its later re-resolution rather than an authoritative pin**; **1 / 10 lockfile+integrity-bound** (#9 Excalidraw); **1 / 10 internally pinned to an immutable commit+digest but bound to the wrong arm identity** (#7 framework at `4df478cd`); **4 / 10 unpinned siblings** (#2, #3, #8, #10). The in-scope same-CI stack payload ratio is **1 / 2** (`install.sh` yes; framework/source no).

Resolution-failure behavior is separately stated rather than hidden in the counts. For #5/#6, a metadata-resolution failure during P1 blocks before mutation. Later package metadata/install failure enters the documented source fallback, but that fallback consumes the immutable stack archive already resolved in P2; failure to resolve or fetch that source is fatal. For #5, the P2 CLI value is authoritative and a later realised mismatch blocks P3 and rolls back. For #6, no authoritative P2 gateway value persists; a later same-suffix gateway drift can be silently accepted because the later value is compared with itself. Changing that TOCTOU path is outside this fix. The new #7 path accepts the 40-hex CI commit directly, fetches only its exact commit URL, rejects failed/empty acquisition, computes and passes the exact body's digest through the existing local-source seam, and rejects any realised manifest commit/digest mismatch without falling back to `next`.

Denominator corrections were sent to and accepted by the lane lead before implementation. Four expressly unpinned siblings make `7/10` impossible. Post-fix, the primary mechanism breakdown is **fully constrained 5 / 10** (#1, #4, #5, #7, #9), **partially constrained 1 / 10** (#6), and **unpinned 4 / 10** (#2, #3, #8, #10). The meaningful repaired ratio is **2 / 2 same-CI stack payload hops**. The fix does not increase coverage; it moves #7 from wrong identity to intended identity. Every census revision moved assurance downward under additional questioning, never upward, so the census is a lower bound on defects and an upper bound on assurance.

### Implementation plan

1. Add the realised source commit+archive-digest assertion and pass the expected immutable identity through the remote fixture; run the current stale-`next` mechanism and capture RED before changing acquisition.
2. Fetch the stack archive from the exact commit URL once, fail closed on failed/empty acquisition, compute its digest, pass that exact body through the installer's existing local-source archive seam, and verify the manifest's realised `sourceCommit` and `sourceSha256` against those values.
3. Run exact-source GREEN: require `P6 FAIL` values and a passing `next-git-present` comparator; run focused installer/baseline gates.
4. R7: delete the binding in a temporary mutant, run the same assertion to RED, then restore and re-run GREEN.
5. Independent code/security review, commit with command-scoped identity, queue guard, one force-with-lease push pinned to `ff3f0d29`; stop without CI polling and report `believed-fixed, pending jarvis validation`.

### Executed outcome

- RED-first stale substitution: fixture `rc=1`; realised source `4df478cdd150fdf8d52ea109f02ade5d85017acd` / `3e6d831efe13c3b2c0501507099d4a566af5abf877dacf85e5d7e4284d35e5c7` rejected against expected `ff3f0d29f1763bed44a60610d073036112e66b77` / `944c6db1b01b23c83169e6dc83e0d31262b1d24a2825270441745eb65c269c25`; comparator `rc=1`, 8/19 mismatches.
- Exact-source reproducibility, N=5 identical final-tree executions: every run had fixture `rc=1`, `@mosaicstack/mosaic@next=0.0.50-next.2207`, P3 PASS, realised source `ff3f0d29…` / `944c6db1…`, SOURCE-CONTROL PASS, `installer_exit=1`, `done_claims=0`, P6 FAIL, and P9 FAIL. Comparator `rc=0` occurred 4/5 (19/19 checks; action-failure reason); comparator `rc=1` occurred 1/5 (18/19; dead hooks active count `2`). The rate is the finding; no comparator verdict is claimed.
- Timings/load: runs 1–2 NOT MEASURED; run 3 rc1 elapsed 912s, load 7.79/7.51/8.93 -> 12.28/14.82/11.79; run 4 rc0 elapsed 938s, load 12.28/14.82/11.79 -> 9.84/15.18/13.83; run 5 rc0 elapsed 954s, load 9.84/15.18/13.83 -> 3.94/4.76/8.18. Load does not explain this sample monotonically.
- The `P6=FAIL` row remains untouched; P6 failed 5/5 while its reason signal varied. No comparator widening was performed. Checkout-source control independently exhibited the rc1/dead-hooks outcome, but its reproducibility was not separately measured.
- #869 out-of-scope finding: current `defaultSupervisorProbe` checks bundled supervisor artifact presence and a resolvable socket path, not socket existence. The 2000ms capability-probe timeout is a code-read hypothesis for variance, NOT MEASURED as causal.
- Final-tree R7 deleted the three local-source binding exports temporarily. The realised source reverted to `4df478cd…` / `3e6d831…`, SOURCE-CONTROL failed against `ff3f0d29…` / `944c6db…`, and comparator `rc=1` with 8/19 mismatches. The subject file SHA-256 was `a93113565aa69f2c6f3d792b78251021bb3bbe3f7813d5fed547ab0099fa3b98` before mutation and after restoration.
- Trust boundary: the exact-commit URL trusts the configured repository provider's authenticated commit-to-archive mapping. The computed digest pins transfer/consumption but does not authenticate against repository/TLS compromise; signed provenance remains the canonical PRD v2 §3 deferral. Initial Codex security review retained this as HIGH/CWE-494; no stronger claim or out-of-scope signing change was made.
- Final baselines: Bash syntax, ShellCheck, `pnpm test:installer`, `pnpm typecheck` (45/45), `pnpm lint` (25/25), `pnpm format:check`, and `git diff --check` pass. Codex code review APPROVE confidence 0.92 with zero findings; after explicit trust-boundary documentation, security re-review risk NONE confidence 0.96 with zero findings. The initial HIGH trust-root finding remains recorded as the signed-provenance deferral.
- Full evidence and named paths: `docs/reports/verification/1050-c1-fix-round/09-round3-source-binding.txt`.

## Round 4 — pipeline case-coverage denominator

- Pipeline 2242 at `0e2eef1c` superseded the lane lead's earlier pipeline-2229 ruling: requirements 1–3 were already satisfied. The temporary local P6-consumer edit started while that ruling was in flight was restored; `tools/e2e-install-test.sh`, the expected-RED manifest, the per-case verifier, and #869 remain unchanged in the final tree.
- Requirement 5 adds a pipeline-level instrument above the three per-case invocations. Expected names are derived from the manifest, markers are scoped by pipeline+workflow run, and each marker is written only after that exact per-case verifier succeeds. Every producer depends on initialization; the final step depends on the complete case/contract matrix, runs after success or failure, and requires exact expected/actual set equality rather than count equality.
- Measured firing controls: skipped arm `3/2` rc1; count inflation `3/3` rc1 with one missing and one unexpected name; stale re-initialized run `3/0` rc1; exact set `3/3` rc0; future manifest case `4/3` rc1. Codex review found and blocked two independent defects: first the missing `depends_on` graph, then comment lines being parsed as case names. Both were accepted; dependency edges are regression-asserted, blank/comments are excluded, and the focused test now consumes the production manifest directly. Final Codex code re-review APPROVE confidence 0.94 and security re-review risk NONE confidence 0.96, both with zero findings. Woodpecker strict lint, Bash syntax, ShellCheck, focused/full installer tests, typecheck, lint, Prettier, and diff check pass.
- Full evidence: `docs/reports/verification/1050-c1-fix-round/10-round4-case-coverage.txt`.