# Remediation Backlog — Reconciled Execution Plan **Owner:** `mos-remediation` (sole writer). Workers read; they never modify this file. **Sources:** [`DECOMP-OPUS.md`](./DECOMP-OPUS.md) (robustness, 38 tasks / 8 dissents) and [`DECOMP-SOL.md`](./DECOMP-SOL.md) (pragmatic, 25 tasks / 10 defers / 7 dissents), produced **independently** — neither planner read the other. Charter: [`MISSION.md`](./MISSION.md). **Status:** PLANNING — this backlog is proposed, not yet dispatched. Three items need a Mos/Jason ruling before the affected tasks dispatch (§5). > **Provenance of the inputs (both clean).** `planner-opus` ran in a fresh session throughout. > `planner-sol` initially began work at 64.3% dirty context despite a brief instructing it to reset; > that run was **interrupted and discarded before it produced any output**, the seat was reset > out-of-band to 0.0%, and the brief was re-dispatched. `DECOMP-SOL.md` is the product of the clean > run only (it peaked at ~26% context). Both decompositions are therefore clean-context artifacts and > are weighted equally here. The discarded dirty run is banked as dogfood seed D-4 and as task RM-58 — > the failure it demonstrates is that _asking_ an agent to reset is not enforcement. --- ## 1. What the two planners agreed on without collusion Independent convergence is the strongest signal available here, because neither planner could see the other's file. Where both arrived at the same conclusion from opposite biases, I treat it as settled. | # | Convergent finding | OPUS | SOL | | --- | --------------------------------------------------------------------------------------------------------------------- | ------------------- | -------------- | | C1 | **The charter's wire-in point is wrong.** Do NOT wire the choke point into `mosaic_orchestrator.py::run_single_task`. | D2 (headline) | Dissent 1 | | C2 | P0 hygiene/gate work must precede the spine, not follow it. | D1, phase P0 | G0, SOL-01/02 | | C3 | No new deployable microservice; the executor is a library + the coord daemon. | implicit throughout | Dissent 3 | | C4 | Comms adapters beyond tmux are out of scope for this mission. | D7 | DEFER 4/5/6 | | C5 | The "100 rotations lossless" bar is a late conformance gate, not an early tax. | D4 | Dissent 7 | | C6 | Redis is a derived hot path, never an authority; PG commits first. | R-013, R-054 | SOL-11, SOL-21 | | C7 | Reuse `packages/coord`; do NOT revive the untracked `apps/coordinator` residue. | R-042 | SOL-15 AC5 | **C1 is the single most consequential output of this exercise.** The charter (`MISSION.md`) and my kickoff instruction both name `mosaic_orchestrator.py::run_single_task:126-276` as the integration point. Both planners independently rejected it on the same evidence: that controller is `"enabled": false` (`.mosaic/orchestrator/config.json:2`) and references a dispatcher path (`tools/macp/dispatcher/pi_runner.ts`) that does not exist in this checkout. Wiring the new choke point into a disabled rail produces **a stranded executor — the identical built-but-unwired disease, one layer up, that would look "done" in a PR.** The live paths are `packages/mosaic/src/commands/launch.ts` and `packages/coord/src/runner.ts`. This contradicted the charter and was escalated as DECISION-1 — **now RULED in the planners' favour by Mos (§5)**. The corrected target is a new production Node `TaskExecutor` on the live dispatch path (`packages/mosaic` launch + `packages/coord`) that Coord/Forge/live dispatch submit through; the Python rail is deleted, not ported. --- ## 1a. ★ KEYSTONE DOGFOOD CASE — an inert gate that erased its own evidence **A merged commit shipped past `pnpm format:check` — and then the evidence quietly erased itself.** Verified chain (blob-level, under the repo's own prettier config, at the file's real path): | commit | state of `packages/mosaic/framework/tools/orchestrator/README.md` | | ------------------------------------------------------------------- | ----------------------------------------------------------------------------- | | `b79336a8` — **merged PR #868** | blob `3ee7f104` — **FAILS** `pnpm format:check` | | `48fd1df2` — merged PR #872 (unrelated: ci-queue-wait 404 handling) | blob `3d3bb132` — passes; incidentally reformatted by that PR's `lint-staged` | | current `origin/main` (`06e0d403`) | passes — **the gate now looks green** | So: PR #868 merged a file that fails a required gate ⇒ **the CI format gate did not block it.** The gate was inert for that merge. Then an unrelated later PR's pre-commit hook reformatted the file as a side effect, so `main` went green again **without anyone ever learning the gate had failed to fire.** > **Correction on record:** my first report to Mos said "format:check is RED on main _now_." That was > true of the `main` my checkout was pinned to (`b79336a8`) and is **no longer true of current `main`**, > which advanced mid-session. The inert-gate finding itself is unchanged and verified; only its > present-tense framing was wrong. The hygiene PR therefore carries the `.prettierignore` fix only — > the README needs no fix today. ### Third live instance, same class — the queue guard, hit by this orchestrator Running the **mandated** pre-push guard during TASK-0: ``` $ ~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose push [ci-queue-wait] platform=gitea purpose=push branch=main sha=06e0d403… [ci-queue-wait] state=unknown purpose=push branch=main $ echo $? → 0 ``` **Two distinct defects in one tool**, both feeding RM-03: 1. **Wrong exit** — `state=unknown` ⇒ `exit 0`. The defect at `ci-queue-wait.sh:282-288` that OPUS documented and that PR #1023 is parked on. A required gate returned PASS on an indeterminate result. 2. **Wrong branch** — it evaluated `branch=main`, not the branch actually being pushed. Even a correctly-exiting guard would have been answering the wrong question. **Standing doctrine (Mos):** until RM-03 lands, a green from this guard carries **zero information** and must not be cited as merge evidence. Rely on reviewer clearance + real CI. Three independent live instances in a single session — format gate, agent context reset, queue guard — is the class confirmed, not anecdote. ### D-7 — shared-tmpfs contention → cascading ENOSPC (live incident, 2026-07-31) The shared 30 G `/tmp` hit **100% ENOSPC** mid-session. It broke tool calls in **two different seats** (mine and Mos's) — a single full disk degrades every agent on the host at once. Recurring: prior incidents 2026-06-18 and 2026-07-17. Attribution matters, because the wrong owner cleans the wrong thing. Measured: | path | size | last modified | owner | | ---------------------------------------------- | --------- | ---------------------------- | ------------------------------------------------- | | `…/-src-mosaic-stack/6d2faee6…` (this session) | **88 K** | live | mos-remediation | | `…/-src-mosaic-stack/c743185d…` | **3.6 G** | **2026-07-22** (9 days dead) | abandoned session, same project path | | `…/claude-1001/pnpm-store` | **1.6 G** | **2026-07-23** (8 days dead) | abandoned; the live store is correctly on `$HOME` | So ~5.2 G — the bulk of the pressure — is **dead session scratch that nothing will ever read again**. This is not a quota problem; it is **P-FLEET-001's stale-session GC, applied to disk instead of tmux sessions.** The same missing capability (nothing owns reaping dead ephemeral state) produces both the orphaned-session failure and this one. Reaping dead-session scratch belongs in RM-50 alongside stale tmux-session GC. **Added to RM-01 as acceptance criteria:** heavy build artifacts (node_modules, package stores, build output) must land on the main disk in the worktree, never on the shared 30 G `/tmp`. **The self-erasure is the important part.** An inert gate that is masked by unrelated downstream commits produces no lasting artifact, which is precisely why this class survives for months. Detection cannot rely on "is `main` currently red" — it must be per-merge-commit. This matters more than the one-line fix: - It is the **P-QUEUE-001 / P-CONFORMANCE-001 class** ("gate-6 was inert fleet-wide"), reproduced in the repository this mission is remediating, discovered incidentally. - It independently **validates OPUS premise A1** ("every gate is inert until proven otherwise") with live evidence rather than argument — which is why RM-02 is adopted as the keystone (§2, X2). - The file fix rides in its own hygiene PR. **The inert gate itself is NOT quiet-patched.** Per Mos: it stays a first-class backlog item, because patching the symptom would destroy the signal. **Binding requirement on RM-02 and RM-55:** the gate registry and the conformance harness must assert **"every merged commit passed every required gate"** — evaluated **per merge commit, against that commit's own tree**, not against current `main`. As the table above proves, a "is main green today" check would have reported all-clear. A merged-commit-that-fails-a-required-gate is the exact detection signal, and it must be a registered must-fail case. A gate that cannot prove it blocked something has not been shown to work. --- ## 2. Where they genuinely disagree (not averaged — adjudicated) | # | Axis | OPUS | SOL | My ruling | | --- | ------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | X1 | **Total cost** | 38 tasks, ~5.3M tok | 25 tasks, ~294K tok | **~18× apart.** Not reconcilable by splitting. They measure different things: SOL explicitly excludes orchestration/review/iteration overhead and assumes one remediation pass; OPUS prices the full loop. Adopt **SOL's scope** with **OPUS's rigor**, and treat SOL's G1 as a hard budget checkpoint (§4). Re-estimate empirically after the first three merged PRs rather than trusting either number. | | X2 | **Gate registry (OPUS R-002)** | Keystone; blocks all P2 | Absent; only a queue-guard fix | **ADOPT OPUS.** Empirically validated in this very session: I found `pnpm format:check` red on `main` via merged PR #868 — a required gate that did not block. OPUS's premise A1 ("every gate is inert until proven otherwise") is not theoretical; it reproduced today, unprompted. Scope it tighter than 120K. | | X3 | **Drizzle PG first-install defect (R-010)** | Hidden blocker; everything downstream depends on it | Not mentioned | **ADOPT OPUS.** `packages/db/src/migrate.ts:30-38` carries a TODO admitting postgres-tier first-install fails today. The spine has only ever been proven on PGlite. Every later migration silently depends on this. SOL missed it. | | X4 | **Rollback artifact for the hard cutover** | D3: hard cutover needs a rehearsed rollback snapshot | SOL-07: import-only, explicitly no dual-write | Both obey "no flat-file interim." OPUS wants a one-directional snapshot nothing reads as authority. I read that as compatible with the directive, but it is Jason's call → **DECISION-2** (§5). | | X5 | **Where the queue guard sits** | P0, independent of spine | SOL-02, also early | Agree it is P0. But ownership collides with **parked PR #1023** → **DECISION-3** (§5). | | X6 | **Report-only rollout** | D5: only with a hard expiry, else withdraw | not raised | **ADOPT OPUS.** A report-only gate is by definition inert; expiry is the mechanism that stops it becoming the new fail-open. | | X7 | **Availability trade (FC-7/FC-11)** | D8: "no DB ⇒ fleet stops" must be pre-committed in writing | not raised | Genuine availability regression, correctly identified. Needs Jason → folded into **DECISION-2**. | --- ## 3. Reconciled DAG Phases run in order; `⛔` marks a hard barrier. `src` shows lineage (`O`=opus, `S`=sol, `O+S`=both). Estimates are given as a **range** (SOL low / OPUS high) rather than a fabricated midpoint — the spread is itself information, and X1 says we calibrate on real merged PRs. ### P0 — Make gates provable, and stop the fleet re-bricking ⛔ _No gate-introducing task in any later phase may merge before RM-02._ | id | task | src | depends_on | est (S/O) | tier | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ | ---------------- | ------ | | RM-01 | Reproducible non-root checkout; gate fails on **code, not env**; heavy artifacts OFF shared `/tmp` (banks D-1/D-2/D-5/D-7) | O+S+live | — | 6K / 60K | codex | | RM-02 | **Gate registry + negative-control CI check** (anti-inert-gate harness) ★keystone | O | RM-01 | — / 120K | opus | | RM-03 ⏸**HOLD** | Queue-guard: **two** defects — (a) `unknown`/`no-status`/malformed ⇒ ≠0, (b) guard evaluates `branch=main` instead of the branch being pushed | O+S+live | RM-02 | 8K / 100K | sonnet | | RM-04 | Activation/version coherence; block launch on skew, fail SAFE; honest `doctor` labels | O+S | RM-01 | (in S-01) / 140K | sonnet | | RM-05 | Break-glass replaces the three silent `MOSAIC BYPASS` fail-opens | O | RM-04, RM-02 | — / 120K | opus | > ⚠ **RM-05 must not merge before RM-04.** The bypasses exist because the lease-broker daemon was > never _deployed_ on this host — removing the fail-open before deployment coherence is real > re-creates the 2026-07-22 bricking incident. Hard edge, from OPUS. ### P1 — Durable spine (PG) ⛔ _No migration may merge before RM-10._ | id | task | src | depends_on | est (S/O) | tier | | ----- | --------------------------------------------------------------------------------------------- | --- | ---------- | ---------- | ------ | | RM-10 | **Fix the Drizzle postgres-tier first-install defect** ★hidden blocker | O | RM-01 | — / 90K | sonnet | | RM-11 | Orchestration spine schema (tasks, attempts, gate_results, hash-chained ledger, typed claims) | O+S | RM-10 | 12K / 160K | opus | | RM-12 | Spine client, fail-closed connection (no silent PGlite in prod) | O | RM-11 | — / 80K | sonnet | | RM-13 | Atomic claims/transitions + transactional outbox + reconciliation sweeper | O+S | RM-12 | 12K / 140K | opus | ### P2 — The single choke point ⛔ _RM-25 (no-second-path) lands in the same milestone as RM-20, or the choke point is optional._ | id | task | src | depends_on | est (S/O) | tier | | ----- | ---------------------------------------------------------------------------------------------- | --- | ------------------- | ---------------- | ------ | | RM-20 | Canonical MACP contract completion (Task/Result/Event/Claim/tri-state outcome) | S | — | 8K / (in R-020) | codex | | RM-21 | **Production `TaskExecutor`** backed by `@mosaicstack/macp` ★keystone | O+S | RM-12, RM-02, RM-20 | 16K / 220K | opus | | RM-22 | Gate-runner hardening: `fail_on`, timeouts, **empty gate set = failure** | O | RM-21 | — / 120K | sonnet | | RM-23 | Hash-chained MACPEvent ledger in PG + lifecycle EventType extension | O+S | RM-21, RM-11 | — / 160K | opus | | RM-24 | Seat identity from `MOSAIC_AGENT_NAME` + **mandatory** tri-state write outcomes | O+S | RM-21 | (in S-03) / 150K | opus | | RM-25 | **No-second-path gate:** terminal status writable only by the executor | O | RM-21, RM-23 | — / 140K | opus | | RM-26 | `packages/coord` submits through the executor (retire direct spawn) | O+S | RM-21 | 16K / 140K | sonnet | | RM-27 | `mosaic yolo/claude/codex/pi` launch path records typed Task + events | O | RM-21, RM-23 | — / 160K | sonnet | | RM-28 | Delete the Forge stub executor (empty-gate-list "success"); Forge submits through the real one | O+S | RM-21 | 10K / 90K | codex | | RM-29 | One-shot flat-file import + cutover readiness audit (dry-run, idempotent, no dual-write) | S | RM-13 | 8K / (in R-062) | codex | > **★ G1 — FIRST DOGFOOD. Stop here and prove it.** One live fleet task travels > PG claim → TaskExecutor → worker → gates → terminal PG result/event, with **no** flat-file state. > Adopted from SOL wholesale. If G1 cannot carry a real task, **do not build Redis, rotation, comms, > or conformance** — remediate instead. This is the budget escape hatch (§4). ### P3 — Rotation lifecycle (finish the Mission Control Plane) | id | task | src | depends_on | est (S/O) | tier | | ----- | -------------------------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ | | RM-30 | Typed state claims (source/confidence/TTL) with HMAC integrity, fail-closed | O+S | RM-11, RM-21 | (in S-03) / 170K | opus | | RM-31 | Contract-hash binding; stale generation loses mutation authority **mechanically** | O+S | RM-21, RM-30 | 12K / 180K | opus | | RM-32 | Durable compaction/token sensor (per-runtime thresholds, PreCompact event) | O | RM-23, RM-31 | — / 130K | sonnet | | RM-33 | Typed checkpoint writer (structured claims, never transcript) + digest | O+S | RM-30, RM-32 | 12K / 150K | opus | | RM-34 | **Rotation daemon:** watch → checkpoint → revoke → kill → relaunch → rehydrate | O+S | RM-33, RM-26 | 16K / 240K | opus | | RM-35 | Rehydration attestation gate: refuse to act on an incomplete claim set | O | RM-33 | — / 130K | opus | | RM-36 | Broker-independent recovery; remove silent bypass; honest capability labels | S | RM-34 | 12K / (in R-004) | sonnet | | RM-37 | Delete `/compact and continue` from the persistent-seat path (**substitution**, not removal) | O+S | RM-34, RM-44 | (in S-16) / 60K | codex | ### P4 — Comms service ⛔ _RM-50 (one roster-owned socket per host) precedes identity-addressed delivery._ | id | task | src | depends_on | est (S/O) | tier | | ----- | ---------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ | | RM-40 | `comms/v1` envelope + protocol-version negotiation, LOUD reject | O+S | RM-11, RM-31 | 8K / 140K | opus | | RM-41 | Comms service: PG state machine PENDING→RECEIVED→CONSUMED→DEAD-LETTER | O+S | RM-40, RM-13 | 16K / 200K | opus | | RM-42 | tmux transport as a **dumb adapter**; durable retry before cursor advance | O+S | RM-41, RM-50 | (in S-19) / 160K | sonnet | | RM-43 | Per-class coalescing + supersede (the stale-consumed-as-live fix) | O+S | RM-41 | 12K / 130K | sonnet | | RM-44 | Redis Streams hot delivery + provenance guard (**Redis is never authority**) | O+S | RM-41, RM-13 | 12K / 170K | opus | | RM-45 | Retire direct tmux sends; only the service may write a pane | O+S | RM-42, RM-43 | (in S-20) / 100K | codex | ### P5 — Retirements, hygiene, conformance | id | task | src | depends_on | est (S/O) | tier | | ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | -------------------------- | ---------------- | ------ | | RM-50 | One roster-owned socket/host; quarantine unmanaged; stale-session GC | O+S | RM-04 | 14K / 150K | sonnet | | RM-51 | Auto-sync **allowlist** (never auto-stage unknown paths) + worktree/lease isolation | O+S | RM-02 | 8K / 110K | sonnet | | RM-52 | Retire the Python controller + duplicate MACP islands (3 → 1) | O+S | RM-26, RM-27, RM-25, RM-28 | 14K / 110K | codex | | RM-53 | Flat-file orchestration → DB hard cutover, with rehearsed rollback artifact | O+S | RM-27, RM-30, RM-34, RM-29 | (in S-10) / 200K | opus | | RM-54 | Fleet-wide inert-gate audit against the RM-02 registry | O | RM-02 | — / 120K | sonnet | | RM-55 | **Conformance harness:** fault-inject the live failure classes on real artifacts | O+S | RM-35, RM-41, RM-53 | 18K / 260K | opus | | RM-56 | Retirement proof: CI asserts all three retirements are complete **and stay complete** | O | RM-52, RM-45, RM-53 | — / 90K | codex | | RM-57 | Operator cutover docs + activation proof; map all 15 decisions to evidence | S | RM-04, RM-36, RM-45, RM-55 | 6K / — | codex | | RM-58 | **Mechanical pre-dispatch context reset** — the orchestrator resets a seat out-of-band and verifies it, rather than asking the agent to reset itself | mos-remediation (D-4) | RM-31, RM-50 | 8K | sonnet | **Critical path:** `RM-01 → RM-02 → RM-10 → RM-11 → RM-12 → RM-21 → RM-23 → RM-31 → RM-33 → RM-34 → RM-53 → RM-55`. --- ## 4. Execution discipline - **Every row is one PR.** Author ≠ reviewer; `rev-974` is the mosaicstack reviewer identity. - **Pre-registered, diff-blind acceptance checks are committed BEFORE the reviewer reads the diff.** Both decomps wrote their ACs in runnable `⇒0` / `⇒≠0` form specifically to make this possible. - **Every gate-introducing task carries at least one registered must-fail negative control.** This is RM-02's whole purpose; a gate with no proven failure path manufactures evidence. - **Cost tiers:** codex for mechanical/unambiguous, sonnet for normal feature work, opus reserved for security/integrity/cross-cutting-invariant tasks. SOL priced 0 opus tokens; OPUS priced 14 opus tasks. I am keeping opus only where the failure is _integrity_, not merely complexity. - **G1 is the budget checkpoint.** If the first-dogfood slice overruns SOL's estimate by >3×, stop and re-plan rather than spending the remainder. X1 says neither estimate is trustworthy until calibrated. - **Defer list adopted from SOL** (10 items): mission dashboard/TUI, PRD-to-board auto-decomposition, heuristic churn scoring, Discord/Slack/Telegram adapters, public MCP comms surface, protocol-v2 negotiation, multi-region PG/Redis, event analytics UI. --- ## 5. Decisions — all three ruled by Mos, 2026-07-31 **DECISION-1 — the wire-in point. ✅ RULED: accept the planners (Mos, 2026-07-31).** The charter's `mosaic_orchestrator.py::run_single_task` target is the **disabled Python controller this mission retires**; wiring the new choke point into the rail we are deleting is wrong. > **Corrected target (authoritative):** a **new production Node `TaskExecutor`** sitting on the > **live dispatch path** — `packages/mosaic` launch + `packages/coord` — which Coord, Forge, and live > dispatch all **submit through**. This is the MACP scout's _full_ recommendation ("replace the block > **with** a Node executor **and** make Coord/Forge submit through it"), not a resurrection of the > Python controller. RM-52 is therefore a **deletion** task, and Build 1's acceptance is measured on a > live `mosaic yolo` invocation. Mos ruled this resolvable from the already-accepted retire-the-Python-rail decision — his authority, not a Jason escalation. RM-21/RM-26/RM-27/RM-52 all take the corrected target. **DECISION-2 — rollback artifact + availability trade. ⏸ JASON-PENDING — NOT BLOCKING.** The DB build is phases away, so this is queued for Jason's next session rather than escalated now. **Binding requirement in the meantime (Mos, from P-RECOVERY-001):** the DB spine **must NOT be a single-point hard-stop.** Design for a broker-independent / degraded mode **plus** a rollback artifact. Jason finalises only the specific availability target. This reverses my earlier reading of OPUS D8 ("the fallback is: the fleet stops") — that answer is **not** pre-committed; a degraded mode is now a design requirement on RM-12, RM-13, RM-23, RM-36 and RM-53. **DECISION-3 — RM-03 vs. parked PR #1023. ✅ RULED: HOLD RM-03 (Mos, 2026-07-31).** Do **not** open a third gate-6 lane — that is the postmortem's own anti-pattern performed by the remediation. PR #1023 sits in Jason's **parked delivery stack**; its disposition (close, or supersede by RM-03) is Jason's at his next session. - **PR #1023 → `SUPERSEDED-PENDING-JASON`.** RM-03 stays `HOLD`; when Jason rules, RM-03 proceeds as the single correct lane. - **RM-02 and RM-55 proceed independently and are NOT held.** The per-merge-commit gate-assertion requirement is the _conformance_ capability, not the gate-6 fix itself — different scope, no ownership collision. --- ## 6. Status | phase | state | | ------------------ | ------------------------------------------------------------------------------------ | | Decomposition | DONE — both planners delivered independently | | Reconciliation | DONE — this document | | Blocking decisions | **OPEN — 3 escalated to Mos (§5)** | | Dispatch | NOT STARTED — RM-01 is dispatchable now; it depends on nothing and blocks everything |