Files
stack/docs/remediation/TASKS.md
T
ba98f88891 docs(remediation): reconciled execution backlog from two independent decompositions
TASK-1 complete. planner-opus (robustness, 38 tasks/8 dissents) and planner-sol
(pragmatic, 25 tasks/10 defers/7 dissents) each decomposed the 4-build plan without
seeing the other's work. Both decomps are committed alongside the reconciliation so
the disagreements stay auditable rather than being flattened into a consensus.

Reconciled into 58 tasks across P0-P5 (docs/remediation/TASKS.md):
- 7 independent convergences, treated as settled because neither planner could see
  the other. The headline: BOTH reject the charter's wire-in point
  (mosaic_orchestrator.py::run_single_task) because that controller is disabled and
  references a dispatcher absent from this checkout — wiring it would produce a
  stranded executor, the same built-but-unwired disease one layer up.
- 7 genuine disagreements ADJUDICATED, not averaged. The cost estimates are ~18x
  apart; rather than split the difference, the plan adopts sol's scope with opus's
  rigor and treats the first-dogfood gate as a hard budget checkpoint.
- 3 decisions escalated (wire-in point, rollback artifact + availability trade,
  queue-guard ownership vs parked PR #1023). No task blocked on them is dispatched.

Keystone dogfood case recorded (TASKS.md 1a): merged PR #868 shipped a file failing
pnpm format:check, then an unrelated PR reformatted it as a side effect, so main went
green and the gate's failure to fire left no trace. Detection must therefore be
per-merge-commit against that commit's own tree.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
2026-07-31 17:25:04 -05:00

26 KiB
Raw Blame History

Remediation Backlog — Reconciled Execution Plan

Owner: mos-remediation (sole writer). Workers read; they never modify this file. Sources: DECOMP-OPUS.md (robustness, 38 tasks / 8 dissents) and DECOMP-SOL.md (pragmatic, 25 tasks / 10 defers / 7 dissents), produced independently — neither planner read the other. Charter: MISSION.md. Status: PLANNING — this backlog is proposed, not yet dispatched. Three items need a Mos/Jason ruling before the affected tasks dispatch (§5).

Provenance of the inputs (both clean). planner-opus ran in a fresh session throughout. planner-sol initially began work at 64.3% dirty context despite a brief instructing it to reset; that run was interrupted and discarded before it produced any output, the seat was reset out-of-band to 0.0%, and the brief was re-dispatched. DECOMP-SOL.md is the product of the clean run only (it peaked at ~26% context). Both decompositions are therefore clean-context artifacts and are weighted equally here. The discarded dirty run is banked as dogfood seed D-4 and as task RM-58 — the failure it demonstrates is that asking an agent to reset is not enforcement.


1. What the two planners agreed on without collusion

Independent convergence is the strongest signal available here, because neither planner could see the other's file. Where both arrived at the same conclusion from opposite biases, I treat it as settled.

# Convergent finding OPUS SOL
C1 The charter's wire-in point is wrong. Do NOT wire the choke point into mosaic_orchestrator.py::run_single_task. D2 (headline) Dissent 1
C2 P0 hygiene/gate work must precede the spine, not follow it. D1, phase P0 G0, SOL-01/02
C3 No new deployable microservice; the executor is a library + the coord daemon. implicit throughout Dissent 3
C4 Comms adapters beyond tmux are out of scope for this mission. D7 DEFER 4/5/6
C5 The "100 rotations lossless" bar is a late conformance gate, not an early tax. D4 Dissent 7
C6 Redis is a derived hot path, never an authority; PG commits first. R-013, R-054 SOL-11, SOL-21
C7 Reuse packages/coord; do NOT revive the untracked apps/coordinator residue. R-042 SOL-15 AC5

C1 is the single most consequential output of this exercise. The charter (MISSION.md) and my kickoff instruction both name mosaic_orchestrator.py::run_single_task:126-276 as the integration point. Both planners independently rejected it on the same evidence: that controller is "enabled": false (.mosaic/orchestrator/config.json:2) and references a dispatcher path (tools/macp/dispatcher/pi_runner.ts) that does not exist in this checkout. Wiring the new choke point into a disabled rail produces a stranded executor — the identical built-but-unwired disease, one layer up, that would look "done" in a PR. The live paths are packages/mosaic/src/commands/launch.ts and packages/coord/src/runner.ts. This contradicts the charter and is escalated as DECISION-1 (§5).


1a. ★ KEYSTONE DOGFOOD CASE — an inert gate that erased its own evidence

A merged commit shipped past pnpm format:check — and then the evidence quietly erased itself.

Verified chain (blob-level, under the repo's own prettier config, at the file's real path):

commit state of packages/mosaic/framework/tools/orchestrator/README.md
b79336a8merged PR #868 blob 3ee7f104FAILS pnpm format:check
48fd1df2 — merged PR #872 (unrelated: ci-queue-wait 404 handling) blob 3d3bb132 — passes; incidentally reformatted by that PR's lint-staged
current origin/main (06e0d403) passes — the gate now looks green

So: PR #868 merged a file that fails a required gate ⇒ the CI format gate did not block it. The gate was inert for that merge. Then an unrelated later PR's pre-commit hook reformatted the file as a side effect, so main went green again without anyone ever learning the gate had failed to fire.

Correction on record: my first report to Mos said "format:check is RED on main now." That was true of the main my checkout was pinned to (b79336a8) and is no longer true of current main, which advanced mid-session. The inert-gate finding itself is unchanged and verified; only its present-tense framing was wrong. The hygiene PR therefore carries the .prettierignore fix only — the README needs no fix today.

The self-erasure is the important part. An inert gate that is masked by unrelated downstream commits produces no lasting artifact, which is precisely why this class survives for months. Detection cannot rely on "is main currently red" — it must be per-merge-commit.

This matters more than the one-line fix:

  • It is the P-QUEUE-001 / P-CONFORMANCE-001 class ("gate-6 was inert fleet-wide"), reproduced in the repository this mission is remediating, discovered incidentally.
  • It independently validates OPUS premise A1 ("every gate is inert until proven otherwise") with live evidence rather than argument — which is why RM-02 is adopted as the keystone (§2, X2).
  • The file fix rides in its own hygiene PR. The inert gate itself is NOT quiet-patched. Per Mos: it stays a first-class backlog item, because patching the symptom would destroy the signal.

Binding requirement on RM-02 and RM-55: the gate registry and the conformance harness must assert "every merged commit passed every required gate" — evaluated per merge commit, against that commit's own tree, not against current main. As the table above proves, a "is main green today" check would have reported all-clear. A merged-commit-that-fails-a-required-gate is the exact detection signal, and it must be a registered must-fail case. A gate that cannot prove it blocked something has not been shown to work.


2. Where they genuinely disagree (not averaged — adjudicated)

# Axis OPUS SOL My ruling
X1 Total cost 38 tasks, ~5.3M tok 25 tasks, ~294K tok ~18× apart. Not reconcilable by splitting. They measure different things: SOL explicitly excludes orchestration/review/iteration overhead and assumes one remediation pass; OPUS prices the full loop. Adopt SOL's scope with OPUS's rigor, and treat SOL's G1 as a hard budget checkpoint (§4). Re-estimate empirically after the first three merged PRs rather than trusting either number.
X2 Gate registry (OPUS R-002) Keystone; blocks all P2 Absent; only a queue-guard fix ADOPT OPUS. Empirically validated in this very session: I found pnpm format:check red on main via merged PR #868 — a required gate that did not block. OPUS's premise A1 ("every gate is inert until proven otherwise") is not theoretical; it reproduced today, unprompted. Scope it tighter than 120K.
X3 Drizzle PG first-install defect (R-010) Hidden blocker; everything downstream depends on it Not mentioned ADOPT OPUS. packages/db/src/migrate.ts:30-38 carries a TODO admitting postgres-tier first-install fails today. The spine has only ever been proven on PGlite. Every later migration silently depends on this. SOL missed it.
X4 Rollback artifact for the hard cutover D3: hard cutover needs a rehearsed rollback snapshot SOL-07: import-only, explicitly no dual-write Both obey "no flat-file interim." OPUS wants a one-directional snapshot nothing reads as authority. I read that as compatible with the directive, but it is Jason's call → DECISION-2 (§5).
X5 Where the queue guard sits P0, independent of spine SOL-02, also early Agree it is P0. But ownership collides with parked PR #1023DECISION-3 (§5).
X6 Report-only rollout D5: only with a hard expiry, else withdraw not raised ADOPT OPUS. A report-only gate is by definition inert; expiry is the mechanism that stops it becoming the new fail-open.
X7 Availability trade (FC-7/FC-11) D8: "no DB ⇒ fleet stops" must be pre-committed in writing not raised Genuine availability regression, correctly identified. Needs Jason → folded into DECISION-2.

3. Reconciled DAG

Phases run in order; marks a hard barrier. src shows lineage (O=opus, S=sol, O+S=both). Estimates are given as a range (SOL low / OPUS high) rather than a fabricated midpoint — the spread is itself information, and X1 says we calibrate on real merged PRs.

P0 — Make gates provable, and stop the fleet re-bricking

No gate-introducing task in any later phase may merge before RM-02.

id task src depends_on est (S/O) tier
RM-01 Reproducible non-root checkout; pre-push gate fails on code, not env (banks D-1/D-2/D-5) O+S 6K / 60K codex
RM-02 Gate registry + negative-control CI check (anti-inert-gate harness) ★keystone O RM-01 — / 120K opus
RM-03 Queue-guard fail-closed rework (unknown/no-status/malformed ⇒ ≠0) O+S RM-02 8K / 100K sonnet
RM-04 Activation/version coherence; block launch on skew, fail SAFE; honest doctor labels O+S RM-01 (in S-01) / 140K sonnet
RM-05 Break-glass replaces the three silent MOSAIC BYPASS fail-opens O RM-04, RM-02 — / 120K opus

RM-05 must not merge before RM-04. The bypasses exist because the lease-broker daemon was never deployed on this host — removing the fail-open before deployment coherence is real re-creates the 2026-07-22 bricking incident. Hard edge, from OPUS.

P1 — Durable spine (PG)

No migration may merge before RM-10.

id task src depends_on est (S/O) tier
RM-10 Fix the Drizzle postgres-tier first-install defect ★hidden blocker O RM-01 — / 90K sonnet
RM-11 Orchestration spine schema (tasks, attempts, gate_results, hash-chained ledger, typed claims) O+S RM-10 12K / 160K opus
RM-12 Spine client, fail-closed connection (no silent PGlite in prod) O RM-11 — / 80K sonnet
RM-13 Atomic claims/transitions + transactional outbox + reconciliation sweeper O+S RM-12 12K / 140K opus

P2 — The single choke point

RM-25 (no-second-path) lands in the same milestone as RM-20, or the choke point is optional.

id task src depends_on est (S/O) tier
RM-20 Canonical MACP contract completion (Task/Result/Event/Claim/tri-state outcome) S 8K / (in R-020) codex
RM-21 Production TaskExecutor backed by @mosaicstack/macp ★keystone O+S RM-12, RM-02, RM-20 16K / 220K opus
RM-22 Gate-runner hardening: fail_on, timeouts, empty gate set = failure O RM-21 — / 120K sonnet
RM-23 Hash-chained MACPEvent ledger in PG + lifecycle EventType extension O+S RM-21, RM-11 — / 160K opus
RM-24 Seat identity from MOSAIC_AGENT_NAME + mandatory tri-state write outcomes O+S RM-21 (in S-03) / 150K opus
RM-25 No-second-path gate: terminal status writable only by the executor O RM-21, RM-23 — / 140K opus
RM-26 packages/coord submits through the executor (retire direct spawn) O+S RM-21 16K / 140K sonnet
RM-27 mosaic yolo/claude/codex/pi launch path records typed Task + events O RM-21, RM-23 — / 160K sonnet
RM-28 Delete the Forge stub executor (empty-gate-list "success"); Forge submits through the real one O+S RM-21 10K / 90K codex
RM-29 One-shot flat-file import + cutover readiness audit (dry-run, idempotent, no dual-write) S RM-13 8K / (in R-062) codex

★ G1 — FIRST DOGFOOD. Stop here and prove it. One live fleet task travels PG claim → TaskExecutor → worker → gates → terminal PG result/event, with no flat-file state. Adopted from SOL wholesale. If G1 cannot carry a real task, do not build Redis, rotation, comms, or conformance — remediate instead. This is the budget escape hatch (§4).

P3 — Rotation lifecycle (finish the Mission Control Plane)

id task src depends_on est (S/O) tier
RM-30 Typed state claims (source/confidence/TTL) with HMAC integrity, fail-closed O+S RM-11, RM-21 (in S-03) / 170K opus
RM-31 Contract-hash binding; stale generation loses mutation authority mechanically O+S RM-21, RM-30 12K / 180K opus
RM-32 Durable compaction/token sensor (per-runtime thresholds, PreCompact event) O RM-23, RM-31 — / 130K sonnet
RM-33 Typed checkpoint writer (structured claims, never transcript) + digest O+S RM-30, RM-32 12K / 150K opus
RM-34 Rotation daemon: watch → checkpoint → revoke → kill → relaunch → rehydrate O+S RM-33, RM-26 16K / 240K opus
RM-35 Rehydration attestation gate: refuse to act on an incomplete claim set O RM-33 — / 130K opus
RM-36 Broker-independent recovery; remove silent bypass; honest capability labels S RM-34 12K / (in R-004) sonnet
RM-37 Delete /compact and continue from the persistent-seat path (substitution, not removal) O+S RM-34, RM-44 (in S-16) / 60K codex

P4 — Comms service

RM-50 (one roster-owned socket per host) precedes identity-addressed delivery.

id task src depends_on est (S/O) tier
RM-40 comms/v1 envelope + protocol-version negotiation, LOUD reject O+S RM-11, RM-31 8K / 140K opus
RM-41 Comms service: PG state machine PENDING→RECEIVED→CONSUMED→DEAD-LETTER O+S RM-40, RM-13 16K / 200K opus
RM-42 tmux transport as a dumb adapter; durable retry before cursor advance O+S RM-41, RM-50 (in S-19) / 160K sonnet
RM-43 Per-class coalescing + supersede (the stale-consumed-as-live fix) O+S RM-41 12K / 130K sonnet
RM-44 Redis Streams hot delivery + provenance guard (Redis is never authority) O+S RM-41, RM-13 12K / 170K opus
RM-45 Retire direct tmux sends; only the service may write a pane O+S RM-42, RM-43 (in S-20) / 100K codex

P5 — Retirements, hygiene, conformance

id task src depends_on est (S/O) tier
RM-50 One roster-owned socket/host; quarantine unmanaged; stale-session GC O+S RM-04 14K / 150K sonnet
RM-51 Auto-sync allowlist (never auto-stage unknown paths) + worktree/lease isolation O+S RM-02 8K / 110K sonnet
RM-52 Retire the Python controller + duplicate MACP islands (3 → 1) O+S RM-26, RM-27, RM-25, RM-28 14K / 110K codex
RM-53 Flat-file orchestration → DB hard cutover, with rehearsed rollback artifact O+S RM-27, RM-30, RM-34, RM-29 (in S-10) / 200K opus
RM-54 Fleet-wide inert-gate audit against the RM-02 registry O RM-02 — / 120K sonnet
RM-55 Conformance harness: fault-inject the live failure classes on real artifacts O+S RM-35, RM-41, RM-53 18K / 260K opus
RM-56 Retirement proof: CI asserts all three retirements are complete and stay complete O RM-52, RM-45, RM-53 — / 90K codex
RM-57 Operator cutover docs + activation proof; map all 15 decisions to evidence S RM-04, RM-36, RM-45, RM-55 6K / — codex
RM-58 Mechanical pre-dispatch context reset — the orchestrator resets a seat out-of-band and verifies it, rather than asking the agent to reset itself mos-remediation (D-4) RM-31, RM-50 8K sonnet

Critical path: RM-01 → RM-02 → RM-10 → RM-11 → RM-12 → RM-21 → RM-23 → RM-31 → RM-33 → RM-34 → RM-53 → RM-55.


4. Execution discipline

  • Every row is one PR. Author ≠ reviewer; rev-974 is the mosaicstack reviewer identity.
  • Pre-registered, diff-blind acceptance checks are committed BEFORE the reviewer reads the diff. Both decomps wrote their ACs in runnable ⇒0 / ⇒≠0 form specifically to make this possible.
  • Every gate-introducing task carries at least one registered must-fail negative control. This is RM-02's whole purpose; a gate with no proven failure path manufactures evidence.
  • Cost tiers: codex for mechanical/unambiguous, sonnet for normal feature work, opus reserved for security/integrity/cross-cutting-invariant tasks. SOL priced 0 opus tokens; OPUS priced 14 opus tasks. I am keeping opus only where the failure is integrity, not merely complexity.
  • G1 is the budget checkpoint. If the first-dogfood slice overruns SOL's estimate by >3×, stop and re-plan rather than spending the remainder. X1 says neither estimate is trustworthy until calibrated.
  • Defer list adopted from SOL (10 items): mission dashboard/TUI, PRD-to-board auto-decomposition, heuristic churn scoring, Discord/Slack/Telegram adapters, public MCP comms surface, protocol-v2 negotiation, multi-region PG/Redis, event analytics UI.

5. Blocking decisions (need Mos, or Jason via Mos)

These are escalated because they change the charter, the accepted directive, or another lane's ownership — none is a question I should answer unilaterally.

DECISION-1 — the wire-in point (changes MISSION.md). Both planners independently reject wiring the choke point into mosaic_orchestrator.py::run_single_task, because that controller is disabled and references a non-existent dispatcher. They propose wiring packages/coord/src/runner.ts + packages/mosaic/src/commands/launch.ts and deleting the Python rail instead. This makes RM-52 a deletion task rather than an integration task, and moves Build 1's acceptance onto a live mosaic yolo invocation. My recommendation: accept — wiring a disabled rail reproduces the exact disease this mission exists to cure.

DECISION-2 — rollback artifact + the availability trade (needs Jason). (a) Does "hard cutover, no flat-file interim" permit a rehearsed, one-directional, read-as-authority- by-nobody rollback snapshot (OPUS D3)? (b) Do we pre-commit in writing that "no DB ⇒ the fleet stops" and "unpersistable event ⇒ the operation fails" (OPUS D8)? That is a real availability regression versus today's limping flat-file fleet — correct for a system whose defining failure is silent continuation, but it should be a decision, not a 2am discovery. My recommendation: yes to both; a rollback snapshot nothing reads is not an interim tracking system.

DECISION-3 — RM-03 ownership vs. parked PR #1023 (needs Mos). P-QUEUE-001 is absorbed into this mission, but PR #1023 is explicitly parked under Mos. Two lanes can legitimately claim it. A third recursion on the gate-6 defect — performed by the remediation itself — would be the postmortem's own anti-pattern. Not dispatching RM-03 until Mos rules.


6. Status

phase state
Decomposition DONE — both planners delivered independently
Reconciliation DONE — this document
Blocking decisions OPEN — 3 escalated to Mos (§5)
Dispatch NOT STARTED — RM-01 is dispatchable now; it depends on nothing and blocks everything