ba98f88891
TASK-1 complete. planner-opus (robustness, 38 tasks/8 dissents) and planner-sol (pragmatic, 25 tasks/10 defers/7 dissents) each decomposed the 4-build plan without seeing the other's work. Both decomps are committed alongside the reconciliation so the disagreements stay auditable rather than being flattened into a consensus. Reconciled into 58 tasks across P0-P5 (docs/remediation/TASKS.md): - 7 independent convergences, treated as settled because neither planner could see the other. The headline: BOTH reject the charter's wire-in point (mosaic_orchestrator.py::run_single_task) because that controller is disabled and references a dispatcher absent from this checkout — wiring it would produce a stranded executor, the same built-but-unwired disease one layer up. - 7 genuine disagreements ADJUDICATED, not averaged. The cost estimates are ~18x apart; rather than split the difference, the plan adopts sol's scope with opus's rigor and treats the first-dogfood gate as a hard budget checkpoint. - 3 decisions escalated (wire-in point, rollback artifact + availability trade, queue-guard ownership vs parked PR #1023). No task blocked on them is dispatched. Keystone dogfood case recorded (TASKS.md 1a): merged PR #868 shipped a file failing pnpm format:check, then an unrelated PR reformatted it as a side effect, so main went green and the gate's failure to fire left no trace. Detection must therefore be per-merge-commit against that commit's own tree. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
259 lines
26 KiB
Markdown
259 lines
26 KiB
Markdown
# Remediation Backlog — Reconciled Execution Plan
|
||
|
||
**Owner:** `mos-remediation` (sole writer). Workers read; they never modify this file.
|
||
**Sources:** [`DECOMP-OPUS.md`](./DECOMP-OPUS.md) (robustness, 38 tasks / 8 dissents) and
|
||
[`DECOMP-SOL.md`](./DECOMP-SOL.md) (pragmatic, 25 tasks / 10 defers / 7 dissents), produced
|
||
**independently** — neither planner read the other. Charter: [`MISSION.md`](./MISSION.md).
|
||
**Status:** PLANNING — this backlog is proposed, not yet dispatched. Three items need a Mos/Jason
|
||
ruling before the affected tasks dispatch (§5).
|
||
|
||
> **Provenance of the inputs (both clean).** `planner-opus` ran in a fresh session throughout.
|
||
> `planner-sol` initially began work at 64.3% dirty context despite a brief instructing it to reset;
|
||
> that run was **interrupted and discarded before it produced any output**, the seat was reset
|
||
> out-of-band to 0.0%, and the brief was re-dispatched. `DECOMP-SOL.md` is the product of the clean
|
||
> run only (it peaked at ~26% context). Both decompositions are therefore clean-context artifacts and
|
||
> are weighted equally here. The discarded dirty run is banked as dogfood seed D-4 and as task RM-58 —
|
||
> the failure it demonstrates is that _asking_ an agent to reset is not enforcement.
|
||
|
||
---
|
||
|
||
## 1. What the two planners agreed on without collusion
|
||
|
||
Independent convergence is the strongest signal available here, because neither planner could see the
|
||
other's file. Where both arrived at the same conclusion from opposite biases, I treat it as settled.
|
||
|
||
| # | Convergent finding | OPUS | SOL |
|
||
| --- | --------------------------------------------------------------------------------------------------------------------- | ------------------- | -------------- |
|
||
| C1 | **The charter's wire-in point is wrong.** Do NOT wire the choke point into `mosaic_orchestrator.py::run_single_task`. | D2 (headline) | Dissent 1 |
|
||
| C2 | P0 hygiene/gate work must precede the spine, not follow it. | D1, phase P0 | G0, SOL-01/02 |
|
||
| C3 | No new deployable microservice; the executor is a library + the coord daemon. | implicit throughout | Dissent 3 |
|
||
| C4 | Comms adapters beyond tmux are out of scope for this mission. | D7 | DEFER 4/5/6 |
|
||
| C5 | The "100 rotations lossless" bar is a late conformance gate, not an early tax. | D4 | Dissent 7 |
|
||
| C6 | Redis is a derived hot path, never an authority; PG commits first. | R-013, R-054 | SOL-11, SOL-21 |
|
||
| C7 | Reuse `packages/coord`; do NOT revive the untracked `apps/coordinator` residue. | R-042 | SOL-15 AC5 |
|
||
|
||
**C1 is the single most consequential output of this exercise.** The charter (`MISSION.md`) and my
|
||
kickoff instruction both name `mosaic_orchestrator.py::run_single_task:126-276` as the integration
|
||
point. Both planners independently rejected it on the same evidence: that controller is
|
||
`"enabled": false` (`.mosaic/orchestrator/config.json:2`) and references a dispatcher path
|
||
(`tools/macp/dispatcher/pi_runner.ts`) that does not exist in this checkout. Wiring the new choke
|
||
point into a disabled rail produces **a stranded executor — the identical built-but-unwired disease,
|
||
one layer up, that would look "done" in a PR.** The live paths are
|
||
`packages/mosaic/src/commands/launch.ts` and `packages/coord/src/runner.ts`.
|
||
This contradicts the charter and is escalated as **DECISION-1** (§5).
|
||
|
||
---
|
||
|
||
## 1a. ★ KEYSTONE DOGFOOD CASE — an inert gate that erased its own evidence
|
||
|
||
**A merged commit shipped past `pnpm format:check` — and then the evidence quietly erased itself.**
|
||
|
||
Verified chain (blob-level, under the repo's own prettier config, at the file's real path):
|
||
|
||
| commit | state of `packages/mosaic/framework/tools/orchestrator/README.md` |
|
||
| ------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
|
||
| `b79336a8` — **merged PR #868** | blob `3ee7f104` — **FAILS** `pnpm format:check` |
|
||
| `48fd1df2` — merged PR #872 (unrelated: ci-queue-wait 404 handling) | blob `3d3bb132` — passes; incidentally reformatted by that PR's `lint-staged` |
|
||
| current `origin/main` (`06e0d403`) | passes — **the gate now looks green** |
|
||
|
||
So: PR #868 merged a file that fails a required gate ⇒ **the CI format gate did not block it.** The
|
||
gate was inert for that merge. Then an unrelated later PR's pre-commit hook reformatted the file as a
|
||
side effect, so `main` went green again **without anyone ever learning the gate had failed to fire.**
|
||
|
||
> **Correction on record:** my first report to Mos said "format:check is RED on main _now_." That was
|
||
> true of the `main` my checkout was pinned to (`b79336a8`) and is **no longer true of current `main`**,
|
||
> which advanced mid-session. The inert-gate finding itself is unchanged and verified; only its
|
||
> present-tense framing was wrong. The hygiene PR therefore carries the `.prettierignore` fix only —
|
||
> the README needs no fix today.
|
||
|
||
**The self-erasure is the important part.** An inert gate that is masked by unrelated downstream
|
||
commits produces no lasting artifact, which is precisely why this class survives for months. Detection
|
||
cannot rely on "is `main` currently red" — it must be per-merge-commit.
|
||
|
||
This matters more than the one-line fix:
|
||
|
||
- It is the **P-QUEUE-001 / P-CONFORMANCE-001 class** ("gate-6 was inert fleet-wide"), reproduced in
|
||
the repository this mission is remediating, discovered incidentally.
|
||
- It independently **validates OPUS premise A1** ("every gate is inert until proven otherwise") with
|
||
live evidence rather than argument — which is why RM-02 is adopted as the keystone (§2, X2).
|
||
- The file fix rides in its own hygiene PR. **The inert gate itself is NOT quiet-patched.** Per Mos:
|
||
it stays a first-class backlog item, because patching the symptom would destroy the signal.
|
||
|
||
**Binding requirement on RM-02 and RM-55:** the gate registry and the conformance harness must assert
|
||
**"every merged commit passed every required gate"** — evaluated **per merge commit, against that
|
||
commit's own tree**, not against current `main`. As the table above proves, a "is main green today"
|
||
check would have reported all-clear. A merged-commit-that-fails-a-required-gate is the exact detection
|
||
signal, and it must be a registered must-fail case. A gate that cannot prove it blocked something has
|
||
not been shown to work.
|
||
|
||
---
|
||
|
||
## 2. Where they genuinely disagree (not averaged — adjudicated)
|
||
|
||
| # | Axis | OPUS | SOL | My ruling |
|
||
| --- | ------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| X1 | **Total cost** | 38 tasks, ~5.3M tok | 25 tasks, ~294K tok | **~18× apart.** Not reconcilable by splitting. They measure different things: SOL explicitly excludes orchestration/review/iteration overhead and assumes one remediation pass; OPUS prices the full loop. Adopt **SOL's scope** with **OPUS's rigor**, and treat SOL's G1 as a hard budget checkpoint (§4). Re-estimate empirically after the first three merged PRs rather than trusting either number. |
|
||
| X2 | **Gate registry (OPUS R-002)** | Keystone; blocks all P2 | Absent; only a queue-guard fix | **ADOPT OPUS.** Empirically validated in this very session: I found `pnpm format:check` red on `main` via merged PR #868 — a required gate that did not block. OPUS's premise A1 ("every gate is inert until proven otherwise") is not theoretical; it reproduced today, unprompted. Scope it tighter than 120K. |
|
||
| X3 | **Drizzle PG first-install defect (R-010)** | Hidden blocker; everything downstream depends on it | Not mentioned | **ADOPT OPUS.** `packages/db/src/migrate.ts:30-38` carries a TODO admitting postgres-tier first-install fails today. The spine has only ever been proven on PGlite. Every later migration silently depends on this. SOL missed it. |
|
||
| X4 | **Rollback artifact for the hard cutover** | D3: hard cutover needs a rehearsed rollback snapshot | SOL-07: import-only, explicitly no dual-write | Both obey "no flat-file interim." OPUS wants a one-directional snapshot nothing reads as authority. I read that as compatible with the directive, but it is Jason's call → **DECISION-2** (§5). |
|
||
| X5 | **Where the queue guard sits** | P0, independent of spine | SOL-02, also early | Agree it is P0. But ownership collides with **parked PR #1023** → **DECISION-3** (§5). |
|
||
| X6 | **Report-only rollout** | D5: only with a hard expiry, else withdraw | not raised | **ADOPT OPUS.** A report-only gate is by definition inert; expiry is the mechanism that stops it becoming the new fail-open. |
|
||
| X7 | **Availability trade (FC-7/FC-11)** | D8: "no DB ⇒ fleet stops" must be pre-committed in writing | not raised | Genuine availability regression, correctly identified. Needs Jason → folded into **DECISION-2**. |
|
||
|
||
---
|
||
|
||
## 3. Reconciled DAG
|
||
|
||
Phases run in order; `⛔` marks a hard barrier. `src` shows lineage (`O`=opus, `S`=sol, `O+S`=both).
|
||
Estimates are given as a **range** (SOL low / OPUS high) rather than a fabricated midpoint — the
|
||
spread is itself information, and X1 says we calibrate on real merged PRs.
|
||
|
||
### P0 — Make gates provable, and stop the fleet re-bricking
|
||
|
||
⛔ _No gate-introducing task in any later phase may merge before RM-02._
|
||
|
||
| id | task | src | depends_on | est (S/O) | tier |
|
||
| ----- | -------------------------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ |
|
||
| RM-01 | Reproducible non-root checkout; pre-push gate fails on **code, not env** (banks D-1/D-2/D-5) | O+S | — | 6K / 60K | codex |
|
||
| RM-02 | **Gate registry + negative-control CI check** (anti-inert-gate harness) ★keystone | O | RM-01 | — / 120K | opus |
|
||
| RM-03 | Queue-guard fail-closed rework (`unknown`/`no-status`/malformed ⇒ ≠0) | O+S | RM-02 | 8K / 100K | sonnet |
|
||
| RM-04 | Activation/version coherence; block launch on skew, fail SAFE; honest `doctor` labels | O+S | RM-01 | (in S-01) / 140K | sonnet |
|
||
| RM-05 | Break-glass replaces the three silent `MOSAIC BYPASS` fail-opens | O | RM-04, RM-02 | — / 120K | opus |
|
||
|
||
> ⚠ **RM-05 must not merge before RM-04.** The bypasses exist because the lease-broker daemon was
|
||
> never _deployed_ on this host — removing the fail-open before deployment coherence is real
|
||
> re-creates the 2026-07-22 bricking incident. Hard edge, from OPUS.
|
||
|
||
### P1 — Durable spine (PG)
|
||
|
||
⛔ _No migration may merge before RM-10._
|
||
|
||
| id | task | src | depends_on | est (S/O) | tier |
|
||
| ----- | --------------------------------------------------------------------------------------------- | --- | ---------- | ---------- | ------ |
|
||
| RM-10 | **Fix the Drizzle postgres-tier first-install defect** ★hidden blocker | O | RM-01 | — / 90K | sonnet |
|
||
| RM-11 | Orchestration spine schema (tasks, attempts, gate_results, hash-chained ledger, typed claims) | O+S | RM-10 | 12K / 160K | opus |
|
||
| RM-12 | Spine client, fail-closed connection (no silent PGlite in prod) | O | RM-11 | — / 80K | sonnet |
|
||
| RM-13 | Atomic claims/transitions + transactional outbox + reconciliation sweeper | O+S | RM-12 | 12K / 140K | opus |
|
||
|
||
### P2 — The single choke point
|
||
|
||
⛔ _RM-25 (no-second-path) lands in the same milestone as RM-20, or the choke point is optional._
|
||
|
||
| id | task | src | depends_on | est (S/O) | tier |
|
||
| ----- | ---------------------------------------------------------------------------------------------- | --- | ------------------- | ---------------- | ------ |
|
||
| RM-20 | Canonical MACP contract completion (Task/Result/Event/Claim/tri-state outcome) | S | — | 8K / (in R-020) | codex |
|
||
| RM-21 | **Production `TaskExecutor`** backed by `@mosaicstack/macp` ★keystone | O+S | RM-12, RM-02, RM-20 | 16K / 220K | opus |
|
||
| RM-22 | Gate-runner hardening: `fail_on`, timeouts, **empty gate set = failure** | O | RM-21 | — / 120K | sonnet |
|
||
| RM-23 | Hash-chained MACPEvent ledger in PG + lifecycle EventType extension | O+S | RM-21, RM-11 | — / 160K | opus |
|
||
| RM-24 | Seat identity from `MOSAIC_AGENT_NAME` + **mandatory** tri-state write outcomes | O+S | RM-21 | (in S-03) / 150K | opus |
|
||
| RM-25 | **No-second-path gate:** terminal status writable only by the executor | O | RM-21, RM-23 | — / 140K | opus |
|
||
| RM-26 | `packages/coord` submits through the executor (retire direct spawn) | O+S | RM-21 | 16K / 140K | sonnet |
|
||
| RM-27 | `mosaic yolo/claude/codex/pi` launch path records typed Task + events | O | RM-21, RM-23 | — / 160K | sonnet |
|
||
| RM-28 | Delete the Forge stub executor (empty-gate-list "success"); Forge submits through the real one | O+S | RM-21 | 10K / 90K | codex |
|
||
| RM-29 | One-shot flat-file import + cutover readiness audit (dry-run, idempotent, no dual-write) | S | RM-13 | 8K / (in R-062) | codex |
|
||
|
||
> **★ G1 — FIRST DOGFOOD. Stop here and prove it.** One live fleet task travels
|
||
> PG claim → TaskExecutor → worker → gates → terminal PG result/event, with **no** flat-file state.
|
||
> Adopted from SOL wholesale. If G1 cannot carry a real task, **do not build Redis, rotation, comms,
|
||
> or conformance** — remediate instead. This is the budget escape hatch (§4).
|
||
|
||
### P3 — Rotation lifecycle (finish the Mission Control Plane)
|
||
|
||
| id | task | src | depends_on | est (S/O) | tier |
|
||
| ----- | -------------------------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ |
|
||
| RM-30 | Typed state claims (source/confidence/TTL) with HMAC integrity, fail-closed | O+S | RM-11, RM-21 | (in S-03) / 170K | opus |
|
||
| RM-31 | Contract-hash binding; stale generation loses mutation authority **mechanically** | O+S | RM-21, RM-30 | 12K / 180K | opus |
|
||
| RM-32 | Durable compaction/token sensor (per-runtime thresholds, PreCompact event) | O | RM-23, RM-31 | — / 130K | sonnet |
|
||
| RM-33 | Typed checkpoint writer (structured claims, never transcript) + digest | O+S | RM-30, RM-32 | 12K / 150K | opus |
|
||
| RM-34 | **Rotation daemon:** watch → checkpoint → revoke → kill → relaunch → rehydrate | O+S | RM-33, RM-26 | 16K / 240K | opus |
|
||
| RM-35 | Rehydration attestation gate: refuse to act on an incomplete claim set | O | RM-33 | — / 130K | opus |
|
||
| RM-36 | Broker-independent recovery; remove silent bypass; honest capability labels | S | RM-34 | 12K / (in R-004) | sonnet |
|
||
| RM-37 | Delete `/compact and continue` from the persistent-seat path (**substitution**, not removal) | O+S | RM-34, RM-44 | (in S-16) / 60K | codex |
|
||
|
||
### P4 — Comms service
|
||
|
||
⛔ _RM-50 (one roster-owned socket per host) precedes identity-addressed delivery._
|
||
|
||
| id | task | src | depends_on | est (S/O) | tier |
|
||
| ----- | ---------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ |
|
||
| RM-40 | `comms/v1` envelope + protocol-version negotiation, LOUD reject | O+S | RM-11, RM-31 | 8K / 140K | opus |
|
||
| RM-41 | Comms service: PG state machine PENDING→RECEIVED→CONSUMED→DEAD-LETTER | O+S | RM-40, RM-13 | 16K / 200K | opus |
|
||
| RM-42 | tmux transport as a **dumb adapter**; durable retry before cursor advance | O+S | RM-41, RM-50 | (in S-19) / 160K | sonnet |
|
||
| RM-43 | Per-class coalescing + supersede (the stale-consumed-as-live fix) | O+S | RM-41 | 12K / 130K | sonnet |
|
||
| RM-44 | Redis Streams hot delivery + provenance guard (**Redis is never authority**) | O+S | RM-41, RM-13 | 12K / 170K | opus |
|
||
| RM-45 | Retire direct tmux sends; only the service may write a pane | O+S | RM-42, RM-43 | (in S-20) / 100K | codex |
|
||
|
||
### P5 — Retirements, hygiene, conformance
|
||
|
||
| id | task | src | depends_on | est (S/O) | tier |
|
||
| ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | -------------------------- | ---------------- | ------ |
|
||
| RM-50 | One roster-owned socket/host; quarantine unmanaged; stale-session GC | O+S | RM-04 | 14K / 150K | sonnet |
|
||
| RM-51 | Auto-sync **allowlist** (never auto-stage unknown paths) + worktree/lease isolation | O+S | RM-02 | 8K / 110K | sonnet |
|
||
| RM-52 | Retire the Python controller + duplicate MACP islands (3 → 1) | O+S | RM-26, RM-27, RM-25, RM-28 | 14K / 110K | codex |
|
||
| RM-53 | Flat-file orchestration → DB hard cutover, with rehearsed rollback artifact | O+S | RM-27, RM-30, RM-34, RM-29 | (in S-10) / 200K | opus |
|
||
| RM-54 | Fleet-wide inert-gate audit against the RM-02 registry | O | RM-02 | — / 120K | sonnet |
|
||
| RM-55 | **Conformance harness:** fault-inject the live failure classes on real artifacts | O+S | RM-35, RM-41, RM-53 | 18K / 260K | opus |
|
||
| RM-56 | Retirement proof: CI asserts all three retirements are complete **and stay complete** | O | RM-52, RM-45, RM-53 | — / 90K | codex |
|
||
| RM-57 | Operator cutover docs + activation proof; map all 15 decisions to evidence | S | RM-04, RM-36, RM-45, RM-55 | 6K / — | codex |
|
||
| RM-58 | **Mechanical pre-dispatch context reset** — the orchestrator resets a seat out-of-band and verifies it, rather than asking the agent to reset itself | mos-remediation (D-4) | RM-31, RM-50 | 8K | sonnet |
|
||
|
||
**Critical path:** `RM-01 → RM-02 → RM-10 → RM-11 → RM-12 → RM-21 → RM-23 → RM-31 → RM-33 → RM-34 → RM-53 → RM-55`.
|
||
|
||
---
|
||
|
||
## 4. Execution discipline
|
||
|
||
- **Every row is one PR.** Author ≠ reviewer; `rev-974` is the mosaicstack reviewer identity.
|
||
- **Pre-registered, diff-blind acceptance checks are committed BEFORE the reviewer reads the diff.**
|
||
Both decomps wrote their ACs in runnable `⇒0` / `⇒≠0` form specifically to make this possible.
|
||
- **Every gate-introducing task carries at least one registered must-fail negative control.** This is
|
||
RM-02's whole purpose; a gate with no proven failure path manufactures evidence.
|
||
- **Cost tiers:** codex for mechanical/unambiguous, sonnet for normal feature work, opus reserved for
|
||
security/integrity/cross-cutting-invariant tasks. SOL priced 0 opus tokens; OPUS priced 14 opus
|
||
tasks. I am keeping opus only where the failure is _integrity_, not merely complexity.
|
||
- **G1 is the budget checkpoint.** If the first-dogfood slice overruns SOL's estimate by >3×, stop and
|
||
re-plan rather than spending the remainder. X1 says neither estimate is trustworthy until calibrated.
|
||
- **Defer list adopted from SOL** (10 items): mission dashboard/TUI, PRD-to-board auto-decomposition,
|
||
heuristic churn scoring, Discord/Slack/Telegram adapters, public MCP comms surface, protocol-v2
|
||
negotiation, multi-region PG/Redis, event analytics UI.
|
||
|
||
---
|
||
|
||
## 5. Blocking decisions (need Mos, or Jason via Mos)
|
||
|
||
These are escalated because they change the charter, the accepted directive, or another lane's
|
||
ownership — none is a question I should answer unilaterally.
|
||
|
||
**DECISION-1 — the wire-in point (changes `MISSION.md`).**
|
||
Both planners independently reject wiring the choke point into
|
||
`mosaic_orchestrator.py::run_single_task`, because that controller is disabled and references a
|
||
non-existent dispatcher. They propose wiring `packages/coord/src/runner.ts` +
|
||
`packages/mosaic/src/commands/launch.ts` and **deleting** the Python rail instead. This makes RM-52 a
|
||
deletion task rather than an integration task, and moves Build 1's acceptance onto a live
|
||
`mosaic yolo` invocation. _My recommendation: accept — wiring a disabled rail reproduces the exact
|
||
disease this mission exists to cure._
|
||
|
||
**DECISION-2 — rollback artifact + the availability trade (needs Jason).**
|
||
(a) Does "hard cutover, no flat-file interim" permit a **rehearsed, one-directional, read-as-authority-
|
||
by-nobody** rollback snapshot (OPUS D3)? (b) Do we pre-commit in writing that "no DB ⇒ the fleet
|
||
stops" and "unpersistable event ⇒ the operation fails" (OPUS D8)? That is a real availability
|
||
regression versus today's limping flat-file fleet — correct for a system whose defining failure is
|
||
_silent continuation_, but it should be a decision, not a 2am discovery.
|
||
_My recommendation: yes to both; a rollback snapshot nothing reads is not an interim tracking system._
|
||
|
||
**DECISION-3 — RM-03 ownership vs. parked PR #1023 (needs Mos).**
|
||
P-QUEUE-001 is absorbed into this mission, but PR #1023 is explicitly parked under Mos. Two lanes can
|
||
legitimately claim it. A third recursion on the gate-6 defect — performed by the remediation itself —
|
||
would be the postmortem's own anti-pattern. _Not dispatching RM-03 until Mos rules._
|
||
|
||
---
|
||
|
||
## 6. Status
|
||
|
||
| phase | state |
|
||
| ------------------ | ------------------------------------------------------------------------------------ |
|
||
| Decomposition | DONE — both planners delivered independently |
|
||
| Reconciliation | DONE — this document |
|
||
| Blocking decisions | **OPEN — 3 escalated to Mos (§5)** |
|
||
| Dispatch | NOT STARTED — RM-01 is dispatchable now; it depends on nothing and blocks everything |
|