stack/docs/scratchpads/1050-install-state-machine-red-fixture.md

#1050 — Installer P0–P9 state machine and red-first fixture

Objective

Implement C1 from the canonical greenfield-install PRD v2: a transactional P0–P9 installer spine, a side-effect-free P0–P8 --check, and a lane-parametric Debian/glibc non-root from-zero fixture. The acceptance milestone is an attributable RED on the pre-C1 installer while preserving P3 PASS.

Authority and scope

• Canonical requirements: jason.woltje/jarvis-brain docs/plans/2026-08-04-greenfield-install-blockers-PRD-v2.md. Currency was re-derived after compaction: authenticated fetch resolved origin/main to cb23e5fbc8a282fa967b93d7a134fa48d11b4bb1; the PRD and charters are byte-identical to the previously read remote copies.
• Tracking: mosaicstack/stack#1050 on git.mosaicstack.dev (author read back as be-coder-05).
• Historical implementation base: origin/next 4df478cdd150fdf8d52ea109f02ade5d85017acd. Delivery PR #1054 targets main under L0's trunk-only rule; next remains a non-merging integration lane.
• Out of scope: PATH, skills, headless wizard/identity, activation remediation, #869 wiring, RM-02, main promotion.
• docs/TASKS.md is orchestrator-single-writer and is not modified by this worker.

Plan

1. Pre-register the canonical phase/output/side-effect-free/fault-injection checks and observe RED against the base installer.
2. Commit the immutable red-first acceptance fixture before implementation.
3. Add the state-machine/journal/postcondition spine without repairing P4/P5/P8 symptoms.
4. Wire the expected-RED from-zero fixture into Woodpecker using Debian/glibc and a non-root target user.
5. Run shell/static baselines, situational container validation, code review, security review, then deliver through a PR to next under the coordinator-owned merge path.

Budget

• Working estimate: 32K reasoning/output tokens.
• Hard external cap: none stated.
• Adaptation: keep implementation in shell surfaces already in scope; no package dependency install unless repository gates require it.

Pre-registered acceptance checks

ID	Exact case	Expected pre-fix result
C1-R1	tools/e2e-install-test.sh --lane next in a clean Debian 12 container as uid 1001	non-zero; P3 PASS; P4 NOT-MEASURED / UNDECLARED; P5/P6/P8 FAIL with own reasons
C1-R2	tools/install-state-machine.test.sh phase table case	RED because base installer does not enumerate canonical P0–P9 contracts
C1-R3	side-effect-free --check case over a fingerprinted HOME	RED because base --check is version-only rather than P0–P8 predicates
C1-R4	fault injection after each P2…P8	RED because base installer has no injectable durable journal/rollback state
C1-R5	Docker unavailable	base harness incorrectly exits 0; replacement must fail non-zero
C1-R6	lane resolution	bare checkout is forbidden; fixture must pass --next and assert the resolved prerelease version
C1-R7	same Debian fixture with git absent vs present	absent: P1 FAIL while legacy installer exits 0 and sync degrades; present: P1 PASS and observed store/runtime containment 101/101

Progress

• Charter, doctrine, delivery/CI/QA/docs guides read and re-anchored after compaction.
• Canonical PRD v2/v3 addenda and charters read from fetched origin/main; numbering reconciles with the TL spec. No numbering conflict found. INV-B/C/D are binding and implemented without renumbering.
• Target base reachability verified with merge-base --is-ancestor.
• Issue #1050 created and provider author read back.
• Initial RED captured; TL rejected P4's repo-root count as a false RED. Four populations disagree (framework payload 1, repo root 13, sync store 101 in the fixture, W-jarvis observation 7), so C1 now requires a checkout-free declared shipped-set artifact and reports P4 NOT-MEASURED / UNDECLARED until C5 supplies it.
• P6 strengthens #869: the two dead enforcement hooks reproduce from zero on a clean broker-less container. C1 asserts the breach but neither wires nor unwires it.
• P1 false pass identified from the P4 evidence row: git is absent from the Debian base and was undeclared even though skill sync shells out to it. C1 adds git to P1; the fixture matrix preserves absent/present controls. The prior claim that web1's missing runtime skills reproduce this greenfield mechanism is withdrawn by the TL and is not carried here.
• Corrected RED transcript captured and reported, including the git-present/absent controls and strict P3 PASS.
• State-machine implementation complete: private pre-mutation journal/snapshot, P0–P8 --check, P2–P8 fault seam, rollback, durable manifest/journal seal, action-status persistence, safe rollback roots, and stale-projection recovery.
• Debian/glibc checkout fixture now packages the complete current checkout, verifies its digest in-container, and reaches the expected attributable RED without host inheritance. CI compares its exact final phase map/reasons to tools/fixtures/greenfield-expected-red.tsv; the fixture remains red while the detector job is green only on an exact match.
• Reviews complete. Reviews 80 (rev-security-02) and 81 (rev-974) requested changes at 3934e03f; their eight non-overlapping detector findings are being remediated red-first. Current remediation adds canonical-image portability, absolute P3 CLI propagation, exact expected-RED schema/cardinality, passwd-HOME binding, created-path owner/mode policy, real-action P2–P8 fault injection, verified non-empty remote installer execution, and seeded secret-canary/redacted diagnostics. Both old verdicts become void when the remediation head moves and require fresh independent review.
• Successor remediation for review 90 is RED-first and recorded in docs/reports/verification/1050-successor-remediation/: the manifest now binds the complete supported final P0 reason; P4 rejects an incomplete created-path walk instead of discarding find failure; and the TERM no-exit control is independent of filesystem enumeration order while retaining a proven RED mutation. Pipeline 2224's 32/2 result was a path-order-sensitive control, not evidence that the resume bug's premise became stale. The enumeration-class sweep additionally replaced order-dependent find | head -1 source-root selection with a checked complete inventory requiring exactly one extracted root.
• C1 fix round for reviews 92/93: Blocker B was completed first. RED was reproduced before implementation: a planted {"status":"in-progress"} record makes the complete real walk fail the suite at P2, while the same planted defect beneath a target-owned mode-0100 directory makes real find fail and the frozen suite falsely exit 0 with installer next lane tests passed. One shared helper now captures and checks the complete NUL-delimited population before testing absence across the primary and copied harness sites; grep no-match is distinct from read failure. The same attack child drives the full suite RED with a named enumeration error, while ordinary native and root/container runs pass. Blocker A then regenerated the exact installer sidecar; provider-fetch validation remains the required post-push bar. Working estimate: 16K tokens; no external hard cap; 60% context is the stop/report gate.

Risks / blockers

• The deployed create wrappers do not expose --dry-run; identity preflight was performed through pr-merge.sh --dry-run on the same HOMELAB repo, which resolved git.mosaicstack.dev + be-coder-05. The issue create then fell back from tea to the API but provider read-back confirmed author be-coder-05.
• next is a non-merging integration lane; PR #1054 targets main. The old “pending promotion to main” caution dissolved when the base moved. #1050 remains open after merge and closes only after Jarvis validates the greenfield behavior.
• #869 must remain staged and inactive.
• Late sequencing input MB-BRAIN-01 is accommodated without implementation or renumbering: P2 covers installer distribution only; P5 owns requested credential capability; P7 leaves an ordered seam for credential-dependent resource provisioning after P5.

Remediation review controls

• B1 RED: the next-lane harness failed immediately under ci-base:latest as root/musl; it now models uid 1001/glibc explicitly and uses Python tree fingerprints instead of GNU find -printf.
• B2 RED: framework/runtime linking consumed bare mosaic from PATH after P3 had committed an absolute path. The unified installer now exports/passes MOSAIC_CLI_PATH; the linker invokes that absolute artifact, and wizard auto-launch has no stale-PATH fallback.
• B3 RED: a one-row manifest (exit=1) certified any exit-1 log. Full-manifest validation now requires the exact three cases, one exit and P0–P9 row each, pinned require/forbid populations, and rejects malformed/duplicate/unknown rows; shrink is a negative control.
• B4 RED: uid 1001 with a passwd HOME different from ambient HOME produced P0 PASS. P0 now binds uid, username, passwd HOME and shell and explicitly rejects root and sudo-with-inherited-HOME controls.
• B5 RED: mode-0777 CLI, mode-0644 identity, and mode-0755 credential storage passed. P3/P4/P5 now apply target owner/group plus executable/shared/private policies; framework credential storage is created 0700.
• B6 RED: fault injection only wrote .selftest-* files. The synthetic path was removed; the P2–P8 matrix enters the normal action flow, proves an action observation occurred, injects after each real phase, and fingerprints rollback.
• B7 RED: an HTTP-200 empty body exits zero when piped to Bash. The fetched installer must now be non-empty, digest-equal to tools/install.sh.sha256, and that exact file is executed; failed/empty/mismatch controls are blocking and CI has a remote immutable-commit arm.
• B8 RED: raw combined command output was duplicated to terminal and commands.log. Both capture layers now redact before output/persistence; a seeded canary is positively emitted by the fake credential-capable registry and must remain absent from terminal, command log, npmrc, generated files and observed argv. The real greenfield fixture also scans those populations.
• Advisory code review findings are fixed: URL userinfo redaction now handles raw @, token-only and percent-encoded forms, repeated :, multiple URLs, Authorization/Basic, npm _auth, and Cookie headers in both capture layers; the real greenfield path positively emits its canary through state_run_captured; verified-fetch removes its temporary body after successful execution; and plaintext diagnostics exist only in process-substitution pipes rather than interruptible temporary files.
• Advisory security review's independent trust-root finding is DEFERRED by canonical PRD v2 §3, which explicitly excludes signed provenance. README now states precisely that the same-origin sidecar detects empty/corrupt/inconsistent publication but cannot authenticate against repository/server compromise; no stronger claim remains.
• The web1 no-manifest representativeness observation is recorded but intentionally not acted on: it is explicitly outside these eight blockers. This remediation does not weaken or otherwise change P9's manifest-presence assertion.

Verification log

• bash -n and ShellCheck pass for all changed shell surfaces; git diff --check passes.
• bash tools/install-state-machine.test.sh passes, including exact P0–P8 rows, passwd-HOME/privilege discrimination, owner/group/mode attacks, persisted P4/P6 action failures, no synthetic fault implementation, unsafe/overlapping/symlink roots, and fatal journal initialization.
• bash tools/install-next-lane.test.sh passes inside ci-base:latest, including exact @next versions, immutable source fallback, source-build/archive-failure rollback, offline --dev, explicit refs, prerelease suffix mismatch, absolute P3 CLI propagation, secret redaction, real-action P2–P8 rollback, and stale projection recovery.
• Comparator controls pass for verdict drift, unexpected exit, manifest shrink, missing phases, duplicate rows, unknown cases, and unknown kinds. Verified-fetch controls pass for successful execution and failed/empty/digest-mismatch rejection.
• bash tools/e2e-install-test.sh --lane next --source checkout --git present returns the required expected RED in clean Debian/glibc as uid 1001: installer P0/P1/P2/P3/P7 PASS; P4/P5/P6/P8 and P9 blocking; no Done. claim; checkout archive digest pinned and current framework installer exercised. tools/verify-greenfield-expected-red.sh converts that expected detector result into a green CI assertion and fails on any unreviewed verdict drift.
• Earlier repository gates passed: pnpm typecheck, pnpm lint, pnpm format:check, upgrade manifest/rollback/durable-snapshot/migration suites, and focused @mosaicstack/mosaic tests with an isolated npm prefix. Full exact-remediation rerun is required before push.
• Review-93 RED evidence at frozen 378bc1a: isolated positive-control full-suite exit 1 with P2 left an in-progress transaction; isolated permission-failure attack full-suite exit 0 with final installer next lane tests passed. No binary shadowing or PATH interception was used; the failure came from a real target-owned mode-0100 directory.
• Fix-round GREEN: pnpm test:installer, native next-lane, root/ci-base:latest next-lane, state-machine, verified-fetch, Bash syntax, ShellCheck, pnpm typecheck, pnpm lint, pnpm format:check, and git diff --check pass. The direct attack child exits 1 and names P2 fault-state enumeration failed. All 81 Mosaic Vitest files / 1508 tests pass under an isolated npm prefix; the wider framework-shell chain reaches the pre-existing #973 Bash-line-number gate and exits 97, matching the known host-specific condition rather than this delta. Codex code review approved at 0.93 confidence with zero findings; Codex security review reported no risk at 0.96 confidence with zero findings.
• Sidecar RED-first documentation control from the repository root: sha256sum -c tools/install.sh.sha256 exits 1 with install.sh: FAILED open or read because the sidecar path is relative to tools/. Sidecar GREEN with the command actually executed: (cd tools && sha256sum -c install.sh.sha256) exits 0 with install.sh: OK; the workflow's separate expected/actual comparison resolves e59cb441a2f37ae9150f8eae470238e9d858a1816df93343d9784a6796676096. This is not substituted for the required immutable provider-fetch arm at the pushed head.

Round 3 — remote-arm downstream source binding

Objective and constraints

• Tracking remains mosaicstack/stack#1050, delivery PR #1054, branch feat/1050-install-state-machine-red-fixture; docs/TASKS.md remains orchestrator-single-writer.
• Bind the remote arm's downstream stack framework/source archive to the same immutable ${CI_COMMIT_SHA} as the digest-verified install.sh, while retaining --next and its exact resolved CLI/gateway lane-version assertion.
• RED first: the realised source commit/digest assertion must reject the existing stale origin/next substitution; GREEN must restore the exact expected P6 FAIL and pass the comparator. R7 then deletes the binding and requires RED again.
• Fenced out: the expected-RED manifest (including P6=FAIL), #869 activation, and #1068 digest-before-comparator sequencing.
• Push budget: one force-with-lease push pinned to provider head ff3f0d29f1763bed44a60610d073036112e66b77; run the queue guard first; do not poll CI after push.
• Working estimate: 14K reasoning/output tokens; no external hard token cap. Scope reduction order: reuse the existing local-source archive seam, add one realised-state assertion, avoid installer behavior changes.

Corrected reference citation

The transferable reference is only the container-image acquisition path in fa-install.sh: resolve mutable input once to Docker-Content-Digest, fail closed if unresolved, render @sha256, then verify running images against the resolved digest. It is not a claim about USC's installer as a whole; the sibling root bundle extraction in Install-FieldAgentOnPanel.ps1:82 is unpinned. The applicable structure is RESOLVE-ONCE -> PIN -> FAIL-CLOSED ON RESOLUTION -> VERIFY-AFTER, applied per acquisition path.

Remote-arm acquisition-path census before the fix

Code-path enumeration only; execution of the complete list is NOT MEASURED in this round yet. Logical payload paths are counted once even when one command is invoked repeatedly or installs a dependency graph.

#	Acquisition path taken by the arm	Pre-fix binding state	Scope disposition
1	Woodpecker checkout containing the workflow, fixture, sidecar, and comparator	CI commit checkout (commit-bound by runner contract)	existing arm substrate
2	node:22-bookworm-slim step image	mutable image tag	unpinned sibling; outside this fix fence
3	Debian package set acquired by apt-get update/install	repository-current, no package versions/snapshot	unpinned sibling; outside this fix fence
4	remote tools/install.sh	exact ${CI_COMMIT_SHA} URL plus checked SHA-256	in scope; already bound first hop
5	@mosaicstack/mosaic@next CLI package graph	mutable lane resolved to an exact top-level version, exact version installed, realised top-level version verified	lane assertion; must remain
6	@mosaicstack/gateway@next package graph	later re-resolution wins and is verified against itself; drift from the first value can be silently accepted	partially constrained TOCTOU sibling; outside this fix
7	mosaicstack/stack framework/source archive consumed by ensure_monorepo	next resolves once to 4df478cdd150fdf8d52ea109f02ade5d85017acd, not the arm's CI commit	in-scope defect; bind to CI commit
8	sequential-thinking MCP package warmed via unversioned npx -y @modelcontextprotocol/server-sequential-thinking	mutable package resolution	unpinned sibling; outside this fix fence
9	Excalidraw npm dependency graph	shipped package-lock.json exact versions + registry integrity hashes; npm install consumes the lock	already lock/integrity-bound, separate from CI-source identity
10	canonical skills repo cloned by mosaic-sync-skills	mutable default branch of mosaic/agent-skills.git	unpinned sibling; outside this fix fence

Pre-fix mechanism breakdown (do not blend unlike strengths): 2 / 10 intended-CI-commit-bound (#1 checkout, #4 installer); 1 / 10 resolve-then-pin with comparison to the authoritative value (#5 CLI); 1 / 10 partially constrained because verify-after compares #6 gateway to its later re-resolution rather than an authoritative pin; 1 / 10 lockfile+integrity-bound (#9 Excalidraw); 1 / 10 internally pinned to an immutable commit+digest but bound to the wrong arm identity (#7 framework at 4df478cd); 4 / 10 unpinned siblings (#2, #3, #8, #10). The in-scope same-CI stack payload ratio is 1 / 2 (install.sh yes; framework/source no).

Resolution-failure behavior is separately stated rather than hidden in the counts. For #5/#6, a metadata-resolution failure during P1 blocks before mutation. Later package metadata/install failure enters the documented source fallback, but that fallback consumes the immutable stack archive already resolved in P2; failure to resolve or fetch that source is fatal. For #5, the P2 CLI value is authoritative and a later realised mismatch blocks P3 and rolls back. For #6, no authoritative P2 gateway value persists; a later same-suffix gateway drift can be silently accepted because the later value is compared with itself. Changing that TOCTOU path is outside this fix. The new #7 path accepts the 40-hex CI commit directly, fetches only its exact commit URL, rejects failed/empty acquisition, computes and passes the exact body's digest through the existing local-source seam, and rejects any realised manifest commit/digest mismatch without falling back to next.

Denominator corrections were sent to and accepted by the lane lead before implementation. Four expressly unpinned siblings make 7/10 impossible. Post-fix, the primary mechanism breakdown is fully constrained 5 / 10 (#1, #4, #5, #7, #9), partially constrained 1 / 10 (#6), and unpinned 4 / 10 (#2, #3, #8, #10). The meaningful repaired ratio is 2 / 2 same-CI stack payload hops. The fix does not increase coverage; it moves #7 from wrong identity to intended identity. Every census revision moved assurance downward under additional questioning, never upward, so the census is a lower bound on defects and an upper bound on assurance.

Implementation plan

1. Add the realised source commit+archive-digest assertion and pass the expected immutable identity through the remote fixture; run the current stale-next mechanism and capture RED before changing acquisition.
2. Fetch the stack archive from the exact commit URL once, fail closed on failed/empty acquisition, compute its digest, pass that exact body through the installer's existing local-source archive seam, and verify the manifest's realised sourceCommit and sourceSha256 against those values.
3. Run exact-source GREEN: require P6 FAIL values and a passing next-git-present comparator; run focused installer/baseline gates.
4. R7: delete the binding in a temporary mutant, run the same assertion to RED, then restore and re-run GREEN.
5. Independent code/security review, commit with command-scoped identity, queue guard, one force-with-lease push pinned to ff3f0d29; stop without CI polling and report believed-fixed, pending jarvis validation.

Executed outcome

• RED-first stale substitution: fixture rc=1; realised source 4df478cdd150fdf8d52ea109f02ade5d85017acd / 3e6d831efe13c3b2c0501507099d4a566af5abf877dacf85e5d7e4284d35e5c7 rejected against expected ff3f0d29f1763bed44a60610d073036112e66b77 / 944c6db1b01b23c83169e6dc83e0d31262b1d24a2825270441745eb65c269c25; comparator rc=1, 8/19 mismatches.
• Exact-source reproducibility, N=5 identical final-tree executions: every run had fixture rc=1, @mosaicstack/mosaic@next=0.0.50-next.2207, P3 PASS, realised source ff3f0d29… / 944c6db1…, SOURCE-CONTROL PASS, installer_exit=1, done_claims=0, P6 FAIL, and P9 FAIL. Comparator rc=0 occurred 4/5 (19/19 checks; action-failure reason); comparator rc=1 occurred 1/5 (18/19; dead hooks active count 2). The rate is the finding; no comparator verdict is claimed.
• Timings/load: runs 1–2 NOT MEASURED; run 3 rc1 elapsed 912s, load 7.79/7.51/8.93 -> 12.28/14.82/11.79; run 4 rc0 elapsed 938s, load 12.28/14.82/11.79 -> 9.84/15.18/13.83; run 5 rc0 elapsed 954s, load 9.84/15.18/13.83 -> 3.94/4.76/8.18. Load does not explain this sample monotonically.
• The P6=FAIL row remains untouched; P6 failed 5/5 while its reason signal varied. No comparator widening was performed. Checkout-source control independently exhibited the rc1/dead-hooks outcome, but its reproducibility was not separately measured.
• #869 out-of-scope finding: current defaultSupervisorProbe checks bundled supervisor artifact presence and a resolvable socket path, not socket existence. The 2000ms capability-probe timeout is a code-read hypothesis for variance, NOT MEASURED as causal.
• Final-tree R7 deleted the three local-source binding exports temporarily. The realised source reverted to 4df478cd… / 3e6d831…, SOURCE-CONTROL failed against ff3f0d29… / 944c6db…, and comparator rc=1 with 8/19 mismatches. The subject file SHA-256 was a93113565aa69f2c6f3d792b78251021bb3bbe3f7813d5fed547ab0099fa3b98 before mutation and after restoration.
• Trust boundary: the exact-commit URL trusts the configured repository provider's authenticated commit-to-archive mapping. The computed digest pins transfer/consumption but does not authenticate against repository/TLS compromise; signed provenance remains the canonical PRD v2 §3 deferral. Initial Codex security review retained this as HIGH/CWE-494; no stronger claim or out-of-scope signing change was made.
• Final baselines: Bash syntax, ShellCheck, pnpm test:installer, pnpm typecheck (45/45), pnpm lint (25/25), pnpm format:check, and git diff --check pass. Codex code review APPROVE confidence 0.92 with zero findings; after explicit trust-boundary documentation, security re-review risk NONE confidence 0.96 with zero findings. The initial HIGH trust-root finding remains recorded as the signed-provenance deferral.
• Full evidence and named paths: docs/reports/verification/1050-c1-fix-round/09-round3-source-binding.txt.

Round 4 — pipeline case-coverage denominator

• Pipeline 2242 at 0e2eef1c superseded the lane lead's earlier pipeline-2229 ruling: requirements 1–3 were already satisfied. The temporary local P6-consumer edit started while that ruling was in flight was restored; tools/e2e-install-test.sh, the expected-RED manifest, the per-case verifier, and #869 remain unchanged in the final tree.
• Requirement 5 adds a pipeline-level instrument above the three per-case invocations. Expected names are derived from the manifest, markers are scoped by pipeline+workflow run, and each marker is written only after that exact per-case verifier succeeds. Every producer depends on initialization; the final step depends on the complete case/contract matrix, runs after success or failure, and requires exact expected/actual set equality rather than count equality.
• Measured firing controls: skipped arm 3/2 rc1; count inflation 3/3 rc1 with one missing and one unexpected name; stale re-initialized run 3/0 rc1; exact set 3/3 rc0; future manifest case 4/3 rc1. Codex review found and blocked two independent defects: first the missing depends_on graph, then comment lines being parsed as case names. Both were accepted; dependency edges are regression-asserted, blank/comments are excluded, and the focused test now consumes the production manifest directly. Final Codex code re-review APPROVE confidence 0.94 and security re-review risk NONE confidence 0.96, both with zero findings. Woodpecker strict lint, Bash syntax, ShellCheck, focused/full installer tests, typecheck, lint, Prettier, and diff check pass.
• Full evidence: docs/reports/verification/1050-c1-fix-round/10-round4-case-coverage.txt.