diff --git a/docs/remediation/BOARD.md b/docs/remediation/BOARD.md index 2b2d59db..9d97981b 100644 --- a/docs/remediation/BOARD.md +++ b/docs/remediation/BOARD.md @@ -1,6 +1,6 @@ # mos-remediation — LIVE BOARD (keep < 8 KB) -**Phase:** PLANNING — adversarial task decomposition IN FLIGHT. +**Phase:** PLANNING COMPLETE — reconciled backlog landed; 3 decisions block dispatch. **Updated:** 2026-07-31 (mos-remediation orchestrator; seat active on `mosaic-fleet`). ## Head @@ -8,23 +8,28 @@ - Mission charter + 15 decisions + 4-build plan: PERSISTED (`docs/remediation/MISSION.md`). - HOLD lifted for this workstream (Jason 2026-07-31). Nothing implemented yet — planning first. - Orchestrator seat `mos-remediation` is LIVE and owns the mission. Residency attestation: PASS. -- TASK-0 (env + push mission package) IN PROGRESS — checkout deps repaired, gates being verified honestly. -- TASK-1 (adversarial decomp) DISPATCHED for real, both planners on GUARANTEED-CLEAN context. +- **TASK-0 DONE** — checkout repaired, all three gates green HONESTLY (no `--no-verify`), branch pushed. +- **TASK-1 DONE** — both planners delivered independently on clean context; reconciled into `TASKS.md` + (58 tasks across P0–P5, 7 convergences, 7 adjudicated disagreements, 3 escalated decisions). +- **NEXT ACTION IS NOT MINE:** DECISION-1/2/3 (`TASKS.md` §5) must be ruled before P0 dispatch. + RM-01 is dispatchable immediately regardless — it depends on nothing and blocks everything. ## In-flight -| Task | Owner | State | -| ---------------------------------------------------- | --------------- | --------------------------------------------- | -| TASK-0 env repair + push `remediation/mission-setup` | mos-remediation | IN PROGRESS — deps installed; gates verifying | -| Decomp (robustness side) → `DECOMP-OPUS.md` | planner-opus | WORKING (clean session) | -| Decomp (pragmatic side) → `DECOMP-SOL.md` | planner-sol | WORKING (reset to 0.0% ctx before dispatch) | -| Reconcile both decomps → `docs/remediation/TASKS.md` | mos-remediation | BLOCKED on the two decomps | +| Task | Owner | State | +| ------------------------------------------------- | --------------- | ----------------------------------------------------- | +| PR: `remediation/mission-setup` (docs-only) | mos-remediation | branch pushed; PR to open | +| PR: `fix/hygiene-inert-format-gate` (full gate) | mos-remediation | branch pushed; needs rev-974 review + CI | +| DECISION-1 wire-in point (charter change) | Mos / Jason | ESCALATED — both planners reject the charter's target | +| DECISION-2 rollback artifact + availability trade | Jason | ESCALATED | +| DECISION-3 RM-03 vs parked PR #1023 ownership | Mos | ESCALATED | +| RM-01 reproducible checkout (unblocks everything) | unassigned | READY TO DISPATCH | ## Fleet seats - mos-remediation — project orchestrator (Claude, /src/mosaic-stack, socket `mosaic-fleet`) — ACTIVE -- planner-opus — adversarial planner (robustness), Opus 5, socket `default` — WORKING -- planner-sol — adversarial planner (pragmatic), gpt-5.6-sol, socket `default` — WORKING +- planner-opus — adversarial planner (robustness), Opus 5, socket `default` — DELIVERED, idle +- planner-sol — adversarial planner (pragmatic), gpt-5.6-sol, socket `default` — DELIVERED, idle - rev-974 — mosaicstack reviewer identity (id 16, write:repository) — idle, on call - Mos (mos-claude) — lead coordinator, socket `default` — relay path to Jason @@ -40,6 +45,9 @@ ## Sequencing (from MISSION.md) 1. Spine + choke-point service (MACP wiring @ mosaic_orchestrator.py::run_single_task) + PG/Redis + ⚠ **CONTESTED — see DECISION-1.** Both planners independently reject this wire-in point: that + controller is `"enabled": false` and references a dispatcher that does not exist here. Charter text + left UNCHANGED pending Mos/Jason ruling; do not treat it as settled. 2. Rotation daemon (finish Mission Control Plane, reuse packages/coord) 3. Comms service (envelope→service→PG/Redis→adapters) 4. Hygiene + conformance harness @@ -64,6 +72,14 @@ Only an out-of-band `/new` driven by the orchestrator guaranteed clean state. Confirms the postmortem thesis: **instructions are not enforcement.** Reset must be a mechanical pre-dispatch step, not a request. +- **D-5 / P-QUEUE-001 + P-CONFORMANCE-001 — KEYSTONE: an inert gate that erased its own evidence.** + Merged PR #868 (`b79336a8`) shipped a file that FAILS `pnpm format:check` ⇒ the CI format gate did + not block. An unrelated later PR (#872) then reformatted that file via its own `lint-staged`, so + `main` went green again and nobody learned the gate had failed to fire. Verified blob-level under the + repo's own config. **Detection must be per-merge-commit against that commit's own tree** — a "is main + green today" check reports all-clear on this exact defect. Binding on RM-02/RM-55. Full chain in + `TASKS.md` §1a. NOT quiet-patched, by Mos's ruling: patching the symptom destroys the signal. + ## Decisions log - 2026-07-31 — Mission set up by Mos post-postmortem (15/15 decided). Dogfood posture active. diff --git a/docs/remediation/DECOMP-OPUS.md b/docs/remediation/DECOMP-OPUS.md new file mode 100644 index 00000000..d35acb79 --- /dev/null +++ b/docs/remediation/DECOMP-OPUS.md @@ -0,0 +1,1110 @@ +# DECOMP-OPUS — Adversarial Task Decomposition (ROBUSTNESS SIDE) + +**Author:** `planner-opus` (web1), robustness/pessimist lane. +**Inputs read:** `MISSION.md`, `MACP-WIRING-SCOUT.md`, `BOARD.md`, plus first-hand inspection of +`/src/mosaic-stack` (`packages/{macp,coord,db,queue,forge,mosaic}`, `apps/`, +`packages/mosaic/framework/tools/orchestrator-matrix/controller/mosaic_orchestrator.py`), +`~/.config/mosaic/tools/{git,tmux,lease-broker}/`, and +`jarvis-brain/docs/scratchpads/postmortem/REMEDIATION-DISCUSSION-STATE.md`. +**Not read (by charter):** `DECOMP-SOL.md`. +**Output:** 38 tasks, 8 dissents. + +--- + +## 0. Operating premise of this decomposition + +Three assumptions drive every ordering choice below. All three are load-bearing; if you reject one, +the DAG changes. + +**A1 — Every gate is inert until proven otherwise.** Gate-6 (`ci-queue-wait.sh`) passed green +fleet-wide while classifying `unknown` → `exit 0`, and its fix recursed the same defect. I verified +the defect is still live at `~/.config/mosaic/tools/git/ci-queue-wait.sh:282-288`: + +``` +terminal-success|terminal-failure|unknown) exit 0 # "unknown" == pass +*) echo "unrecognized state ... proceeding conservatively"; exit 0 +``` + +`no-status` also exits 0 unless `--require-status` is passed, and nothing passes it +(`pr-merge.sh:98`). A gate whose failure path is unreachable is worse than no gate: it manufactures +evidence. **Therefore: no task in this plan may introduce a gate without a registered +must-exit-nonzero negative control, and the registry check itself is a gate (R-002).** + +**A2 — Fail-open is the resting state of this system.** Confirmed instances, all live today: + +| Site | Failure mode | +| ----------------------------------------- | --------------------------------------------------------------- | +| `ci-queue-wait.sh:282,286` | unknown/unrecognized CI state → proceed | +| `send-message.sh:111` | indeterminate submission → `✓ sent`, exit 0 | +| `packages/forge/src/cli.ts:13-31` | stub `TaskExecutor` reports completion with **empty gate list** | +| `lease-broker/mutator-gate.py:140` | broker socket unset → `return 0` (all tool calls allowed) | +| `lease-broker/revoke-lease.py:57` | broker socket unset → `return 0` (lease never revoked) | +| `lease-broker/receipt-observer-client.py` | entire client replaced by a `sys.exit(0)` stub | + +Four of six are _silent_. Every task below names its fail-closed point explicitly (§4). + +**A3 — The substrate is real, but "wire it in" understates the work.** `packages/db` already has +`missions`, `tasks`, `mission_tasks`, `agents`, `interaction_{sessions,inbox,outbox,checkpoints,handoffs}` +with idempotency-key unique indexes and a compaction-epoch column. `packages/queue` has a +`QueueAdapter` with bullmq + local adapters. That is genuinely more than the charter credits. +But see §6 — roughly a third of this plan is greenfield regardless of the slogan. + +--- + +## 1. Phase map (ordered; hard barriers between phases marked ⛔) + +``` +P0 Make gates provable + stop the fleet re-bricking R-001..R-005 + ⛔ barrier: no gate-introducing task may merge before R-002 +P1 Durable spine (PG) R-010..R-013 + ⛔ barrier: no migration may merge before R-010 +P2 Single choke-point executor (MACP wire-in) R-020..R-030 + ⛔ barrier: R-025 (no-second-path) must land in the same milestone as R-020 +P3 Rotation lifecycle (Mission Control Plane, finished) R-040..R-045 + ⛔ barrier: R-045 (delete /compact guidance) may not merge before R-042 is live +P4 Comms service R-050..R-055 + ⛔ barrier: R-060 (one socket per host) must precede R-052 identity addressing +P5 Retirements, hygiene, conformance R-060..R-065 +``` + +**Deliberate divergence from `MISSION.md` sequencing:** the charter starts at "spine + choke-point +(builds 1+2)". I insert P0 ahead of it. Rationale in Dissent D1. + +--- + +## 2. Master task table + +`tier` = suggested runtime. `codex` = mechanical, spec is unambiguous. `sonnet` = normal feature work. +`opus` = security/integrity/design-ambiguity or cross-cutting invariant. + +| id | title | build | depends_on | est tok | tier | +| ----- | ---------------------------------------------------------------------------------------- | ----- | -------------------------- | ------- | ------ | +| R-001 | Reproducible checkout + pre-push gate that fails on code, not env | 5 | — | 60k | codex | +| R-002 | Gate registry + negative-control CI check (anti-inert-gate harness) | 5 | R-001 | 120k | opus | +| R-003 | Queue-guard fail-closed rework (`unknown`/`no-status`/malformed) | 1 | R-002 | 100k | sonnet | +| R-004 | Break-glass replaces the three silent `MOSAIC BYPASS` fail-opens | 1 | R-005, R-002 | 120k | opus | +| R-005 | Activation/version coherence: block launch on host↔repo skew, fail SAFE | 5 | R-001 | 140k | sonnet | +| R-010 | Fix the Drizzle postgres-tier first-install migration defect | 2 | R-001 | 90k | sonnet | +| R-011 | Orchestration spine schema (macp_tasks, attempts, gate_results, ledger, claims) | 2 | R-010 | 160k | opus | +| R-012 | Spine client with fail-closed connection semantics (no silent PGlite in prod) | 2 | R-011 | 80k | sonnet | +| R-013 | Transactional outbox tables + reconciliation sweeper (PG-first) | 2 | R-012 | 140k | opus | +| R-020 | Production `TaskExecutor` backed by `@mosaicstack/macp` | 1 | R-012, R-002 | 220k | opus | +| R-021 | Gate-runner hardening: structured `GateEntry`, `fail_on`, timeout, empty-gate-set = fail | 1 | R-020 | 120k | sonnet | +| R-022 | Hash-chained append-only MACPEvent ledger in PG; lifecycle EventType extension | 1 | R-020, R-011 | 160k | opus | +| R-023 | `packages/coord` submits work through the executor (retire direct spawn) | 1 | R-020 | 140k | sonnet | +| R-024 | `mosaic yolo\|claude\|codex\|pi` launch path records a typed Task + events | 1 | R-020, R-022 | 160k | sonnet | +| R-025 | No-second-path gate: terminal status writable only by the executor | 1 | R-020, R-022 | 140k | opus | +| R-026 | Delete the Forge stub executor; Forge submits through the real one | 1 | R-020 | 90k | codex | +| R-027 | Seat identity from `MOSAIC_AGENT_NAME` + mandatory tri-state write outcomes | 1 | R-020 | 150k | opus | +| R-028 | Typed state claims (source/confidence/TTL) with HMAC integrity, fail-closed | 1 | R-011, R-020 | 170k | opus | +| R-029 | Contract-hash binding; stale generation loses mutation authority mechanically | 1 | R-020, R-028 | 180k | opus | +| R-030 | Retire the Python controller + `plugins/macp` island (island count 3 → 1) | 1 | R-023, R-024, R-025, R-026 | 110k | codex | +| R-040 | Durable compaction/token sensor (per-runtime thresholds, PreCompact event) | 3 | R-022, R-029 | 130k | sonnet | +| R-041 | Typed checkpoint writer (structured claims, not transcript) + digest | 3 | R-028, R-040 | 150k | opus | +| R-042 | Rotation daemon: watch → checkpoint → kill → relaunch → rehydrate | 3 | R-041, R-023 | 240k | opus | +| R-043 | Rehydration attestation gate: refuse to act on an incomplete claim set | 3 | R-041 | 130k | opus | +| R-044 | Fault-injected rotation soak (100× incl. kill-at-worst-moment) | 3 | R-042, R-043 | 180k | sonnet | +| R-045 | Delete `/compact and continue` from the persistent-seat path (substitution) | 3 | R-042, R-044 | 60k | codex | +| R-050 | `comms/v1` envelope + protocol-version negotiation, LOUD reject | 4 | R-011 | 140k | opus | +| R-051 | Comms service: PG state machine PENDING→RECEIVED→CONSUMED→DEAD-LETTER | 4 | R-050, R-013 | 200k | opus | +| R-052 | tmux transport adapter behind the service; durable retry before cursor advance | 4 | R-051, R-060 | 160k | sonnet | +| R-053 | Per-class coalescing + supersede (stale-consumed-as-live fix) | 4 | R-051 | 130k | sonnet | +| R-054 | Redis hot path + provenance guard (Redis may never be read as authority) | 4 | R-051, R-013 | 170k | opus | +| R-055 | Retire direct tmux sends; only the service may write a pane | 4 | R-052, R-053 | 100k | codex | +| R-060 | One roster-owned socket/host, quarantine unmanaged, stale-session GC | 5 | R-005 | 150k | sonnet | +| R-061 | Auto-sync allowlist (never auto-stage unknown paths) + worktree isolation | 5 | R-002 | 110k | sonnet | +| R-062 | Flat-file orchestration tracking → DB cutover, with rollback artifact | 2/5 | R-024, R-028, R-042 | 200k | opus | +| R-063 | Conformance harness: fault injection of the six live failure classes | 5 | R-044, R-051, R-062 | 260k | opus | +| R-064 | Fleet-wide inert-gate audit against the R-002 registry | 5 | R-002 | 120k | sonnet | +| R-065 | Retirement proof: CI asserts all three retirements are complete and stay complete | 5 | R-030, R-055, R-062 | 90k | codex | + +**Totals:** 38 tasks, ~5.3M estimated worker tokens. Critical path (longest chain): +`R-001 → R-010 → R-011 → R-012 → R-020 → R-022 → R-029 → R-040 → R-041 → R-042 → R-044 → R-062 → R-063` += 13 tasks, ~2.2M tokens. Everything else parallelizes against it. + +--- + +## 3. Task detail + +Acceptance criteria are written **diff-blind**: each is a command a reviewer can pre-register and run +against the merged branch without having read the implementation. `⇒0` / `⇒≠0` denote required exit +codes. Every gate-introducing task carries at least one **must-fail** case — that is the whole point. + +### P0 — Make gates provable + +--- + +**R-001 — Reproducible checkout + pre-push gate that fails on code, not env** +_build 5 · depends: — · 60k · codex_ + +The mosaic-stack checkout on web1 cannot push: `pre-push` typecheck fails on `Cannot find module +commander/@mosaicstack/*` because deps were never installed. A gate that reds on environment is +indistinguishable from a gate that reds on defect, so the fleet learns to bypass it. This is the +mission's own TASK-0 and it blocks literally every other PR. + +- AC1: from a clean clone in the CI container, `pnpm install --frozen-lockfile && pnpm -w typecheck` ⇒0. +- AC2: `scripts/preflight.sh` (or equivalent named in the PR) run with node_modules removed ⇒≠0 with a + message containing `run pnpm install`, and ⇒0 after install. **Negative control: the missing-deps + case must be distinguishable by exit code or a machine-greppable marker from a real type error.** +- AC3: introduce `export const x: number = "s"` in any package → gate ⇒≠0. Revert → ⇒0. +- AC4: `git status --porcelain` in a fresh clone after a full build is empty (no generated residue). +- Dogfood: live finding 2026-07-31 — "pre-push gate red on env not code"; `.bak`/`.mosaic-bak` and + untracked `apps/coordinator` residue in the checkout. +- Fail-closed point: none introduced; this restores the _ability_ to fail closed. + +--- + +**R-002 — Gate registry + negative-control CI check (anti-inert-gate harness)** ★ keystone +_build 5 · depends: R-001 · 120k · opus_ + +The generalization of P-QUEUE-001. A machine-readable registry (`gates.manifest.json` or equivalent) +where every deterministic gate in the repo declares: invocation, the input classes it must +distinguish, and for **each** class the required exit code. A CI job runs every declared case and +fails if (a) any must-fail case exits 0, (b) any gate in the tools tree is absent from the registry, +or (c) a gate has zero must-fail cases registered. + +- AC1: `pnpm gate:verify` ⇒0 on main. +- AC2: mutate any registered gate's failure branch to `exit 0` → `pnpm gate:verify` ⇒≠0 naming that + gate. **This is the meta-negative-control; a reviewer must run it.** +- AC3: add a new executable under the gates path without a registry entry → ⇒≠0 with `unregistered gate`. +- AC4: registry entry with zero must-fail cases → ⇒≠0 with `no negative control`. +- AC5: the check runs in CI on every PR, not only on gate-file changes (an inert gate is usually made + inert by a change _elsewhere_, e.g. an env var going unset). +- Dogfood: gate-6 inert fleet-wide; #1019 fix (PR #1023 @f6334080) recursed the same defect — + rev-974 review id 58 finding (3): "tests don't assert exit outcomes." +- Fail-closed point: **unregistered gate ⇒ build fails.** Not "warn". +- Secretly greenfield: yes, small. Accept it — it is the cheapest insurance in the plan. + +--- + +**R-003 — Queue-guard fail-closed rework** +_build 1 · depends: R-002 · 100k · sonnet_ + +`ci-queue-wait.sh` must stop treating `unknown`, unrecognized, and `no-status` as pass. Spec is +rev-974 review id 58: (1) `unknown` must not exit 0; (2) payload via stdin/temp file, never argv +(ARG_MAX rc126 at ~150 KiB); (3) tests assert exit outcomes. + +- AC1: synthetic status payloads for each class ⇒ `pending` (loops then times out ⇒124), + `terminal-success` ⇒0, `terminal-failure` ⇒≠0, `no-status` ⇒≠0, `malformed JSON` ⇒≠0, + `unknown` ⇒≠0. Each is a registered R-002 case. +- AC2: a 150 KiB status payload ⇒ not 126, and classification unchanged. +- AC3: `--force-proceed` (if provided) requires a reason, emits a WARN to stderr, and is a registered + break-glass per R-004 — never a silent default. +- AC4: `pr-merge.sh` invokes the guard with `--require-status`; removing that flag ⇒ R-002 fails. +- Dogfood: gate-6 inert + #1019 recursion. +- Fail-closed point: **every non-green classification blocks the merge.** +- ⚠ **Authority conflict:** #1019/PR #1023 is explicitly PARKED under Mos ("delivery stack still + parked"), while Build 1 absorbs QUEUE and HOLD is lifted for this workstream. The orchestrator must + resolve ownership with Mos _before dispatch_, or two lanes will rework the same branch. Flagged, not + assumed. + +--- + +**R-004 — Break-glass replaces the three silent `MOSAIC BYPASS` fail-opens** +_build 1 · depends: R-005, R-002 · 120k · opus_ + +Convert `mutator-gate.py:140`, `revoke-lease.py:57`, and the `receipt-observer-client.py` stub from +permanent silent fail-open into LOUD + AUDITED + TEMPORARY break-glass: explicit opt-in token with an +expiry timestamp, a WARN on stderr every invocation, an audit line written on every use, and refusal +once expired. + +- AC1: with break-glass unset and the broker absent → gate ⇒≠0 (denies) with `GATE_UNAVAILABLE`. +- AC2: with a valid unexpired break-glass token → ⇒0, **and** an audit record exists containing seat, + reason, expiry, and invocation count. +- AC3: with an expired token → ⇒≠0. (Registered must-fail case.) +- AC4: `grep -rn "MOSAIC BYPASS" ~/.config/mosaic/tools/` returns nothing after deploy; a CI check + fails if the marker reappears without a matching registry entry. +- AC5: the Stop-hook client never blocks a turn — observational hooks exit 0 on transport failure + **but record a delivery-failure event**; silence is the bug, not the exit code. +- Dogfood: Pi brick (fail-closed gate whose recovery needed the gate it was blocked by) + the + 2026-07-22 fleet incident that motivated all three bypasses. +- Fail-closed point: **broker absent + no break-glass = deny.** +- ⚠ **Ordering hazard (severe):** merging R-004 before R-005 re-bricks the fleet exactly as on + 2026-07-22. The bypasses exist because the broker daemon was never _deployed_ on this host, not + because the code was wrong. Removing the fail-open before deployment coherence is real recreates the + incident. **Hard edge: R-005 → R-004.** + +--- + +**R-005 — Activation/version coherence: block launch on host↔repo skew, fail SAFE** +_build 5 · depends: R-001 · 140k · sonnet_ + +P-ACTIVATION-001. Transactional install of CLI + hooks + broker + version; launch refuses to start a +seat whose deployed framework version disagrees with the repo/roster expectation. "Fail SAFE" per +Jason's caveat = clear message + exact fix command, not a bare hard stop. + +- AC1: with matching versions, `mosaic claude` launches ⇒0. +- AC2: with a deliberately skewed deployed version, launch ⇒≠0 and stdout contains both versions and a + copy-pasteable remediation command. (Registered must-fail case.) +- AC3: interrupted install (kill mid-way) leaves the previous version fully functional — no + half-installed hook set. Verified by: install, `kill -9` at a scripted point, then run the previous + version's smoke test ⇒0. +- AC4: `mosaic doctor` reports broker deployed/undeployed **honestly** — the label must change when + the daemon is stopped. (Direct fix for P-RECOVERY-001's "honest capability labeling".) +- Dogfood: the 2026-07-22 incident root cause ("#828 lease-broker daemon + launch-time env injection + were never deployed on this host"); the messy-checkout finding. +- Fail-closed point: **version skew blocks launch.** + +### P1 — Durable spine + +--- + +**R-010 — Fix the Drizzle postgres-tier first-install migration defect** ★ hidden blocker +_build 2 · depends: R-001 · 90k · sonnet_ + +`packages/db/src/migrate.ts:30-38` carries a TODO admitting that **postgres-tier first-install fails +today**: Drizzle wraps all migrations in one transaction (breaking 0009's `ALTER TYPE ADD VALUE` → +`SET DEFAULT` sequence) and `drizzle/meta/_journal.json` orders 0009 before 0008, which the +postgres-js migrator silently skips by `created_at < folderMillis`. The PGlite path sidesteps both — +which means the spine has only ever been proven on the embedded tier. + +- AC1: against a **fresh empty** Postgres 16 container, `runMigrations()` ⇒0 and every table in + `schema.ts` exists (assert by count and by name list). +- AC2: `drizzle.__drizzle_migrations` contains one row per migration file, in file order. +- AC3: re-running `runMigrations()` on the migrated DB ⇒0 and is a no-op (row count unchanged). +- AC4: a deliberately corrupted journal ⇒≠0 (must not silently skip). Registered must-fail case. +- Dogfood: synthetic — no live incident yet, because nobody has first-installed the PG tier. That is + precisely the danger: the first real deploy is the discovery event. +- Fail-closed point: **skipped migration ⇒ error, never silent skip.** +- ⚠ **Ordering hazard:** every later task that adds a migration (R-011, R-013, R-022, R-028, R-050, + R-051) silently depends on this. Without it they pass on PGlite in CI and fail on the real deploy. + +--- + +**R-011 — Orchestration spine schema** +_build 2 · depends: R-010 · 160k · opus_ + +New tables, **not** overloads of existing ones: `macp_tasks` (typed MACP `Task`), `task_attempts`, +`gate_results`, `macp_events` (append-only, monotonic sequence, `prev_hash`/`hash` chain), +`state_claims` (typed claim with source/confidence/TTL + HMAC), `seat_identities`, +`contract_bindings`. Reuse `missions`/`mission_tasks` for mission linkage. + +- AC1: migration applies to fresh PG ⇒0 (per R-010 AC1). +- AC2: `UPDATE macp_events SET ...` or `DELETE FROM macp_events` ⇒ error (append-only enforced by + trigger/rule or role grants, not convention). Registered must-fail case. +- AC3: inserting an event whose `prev_hash` does not match the current head ⇒ error. +- AC4: a verification function reports gap/tamper on an artificially altered chain ⇒≠0. +- AC5: `state_claims` rows carry a non-null TTL; a query helper returns expired claims as _expired_, + never as valid. +- Dogfood: P-AUDIT-001's duplicate ledger (`.mosaic/orchestrator/events.ndjson` written by + `mosaic_orchestrator.py:129-133` and only _read_ by `plugins/mosaic-framework`). +- Fail-closed point: **ledger mutation attempt = hard DB error.** +- Design note (deliberate divergence): the existing `events` table (`schema.ts:264`) is a generic + `type/title/description/date` record — a calendar-shaped table. Overloading it as the audit ledger + would defeat append-only enforcement. Separate table. + +--- + +**R-012 — Spine client with fail-closed connection semantics** +_build 2 · depends: R-011 · 80k · sonnet_ + +`packages/db` exposes both `createDb` (postgres) and `createPgliteDb` (embedded). If production can +ever silently land on PGlite, two seats will happily write to two different "systems of record." + +- AC1: with `NODE_ENV=production` and no `DATABASE_URL`, client construction ⇒ throws (not a fallback). + Registered must-fail case. +- AC2: with `DATABASE_URL` set to an unreachable host, the first operation ⇒ throws within the + configured timeout; no in-memory substitute is created. +- AC3: PGlite construction in production requires an explicit `MOSAIC_ALLOW_EMBEDDED_DB=1`; absent it, + ⇒ throws. +- AC4: a health endpoint/CLI reports which tier is in use, and the string differs between tiers. +- Dogfood: synthetic (the split-brain this prevents has not happened yet _because_ the spine is not + wired — it would happen on week one of Build 2). +- Fail-closed point: **no DB ⇒ no work.** The choke point must not degrade to local files. + +--- + +**R-013 — Transactional outbox + reconciliation sweeper** +_build 2 · depends: R-012 · 140k · opus_ + +PG commit first, then enqueue. A sweeper deterministically re-enqueues rows that committed in PG but +never reached the hot queue. `interaction_outbox` already has the right shape +(`idempotency_key` unique per session, status enum, `content_digest`) — reuse it rather than adding a +fourth queue concept. + +- AC1: kill the process between PG commit and enqueue → sweeper re-enqueues within N seconds; the + consumer sees the message exactly once (idempotency key enforced). +- AC2: enqueue the same idempotency key twice → one delivery, one PG row. Registered must-fail case is + the inverse: a duplicate delivery ⇒ test fails. +- AC3: with Redis unavailable, writes still commit to PG and are marked `pending`; nothing is lost and + nothing reports delivered. +- AC4: sweeper is idempotent — running it twice concurrently produces no duplicates (advisory lock). +- Dogfood: MACP scout notification BOUNCE ("tmux target not found") — never RECEIVED, no retry, found + only by manual liveness check. +- Fail-closed point: **an un-acked message is never marked delivered.** + +### P2 — The single choke point + +--- + +**R-020 — Production `TaskExecutor` backed by `@mosaicstack/macp`** ★ keystone +_build 1 · depends: R-012, R-002 · 220k · opus_ + +One Node executor: validate a typed `Task` (schemas exist at `packages/macp/src/schemas`), call +`resolveCredentials`, run the process, call `runGates`, emit `MACPEvent`s, persist a `TaskResult`. +Exposed as a programmatic API **and** a CLI. Note the scout's observation that there is no exported +programmatic `submit` — that is this task. + +- AC1: submitting a Task that fails schema validation ⇒≠0 before any process spawn; assert no child + process was created and no event row exists. Registered must-fail case. +- AC2: a Task whose provider credentials cannot be resolved ⇒≠0 with `CredentialError`; **assert the + credential value never appears in stdout, stderr, the event ledger, or the task row.** +- AC3: a successful Task produces, in order: `task.assigned`, `task.started`, `rail.check.*` per gate, + `task.completed` — all in `macp_events`, chain-valid. +- AC4: a Task whose gate fails is recorded `failed`/`gated`, never `completed`. +- AC5: killing the executor mid-run leaves the task in a non-terminal state that the sweeper can + reclaim — **never** `completed`. +- AC6: two executors racing on the same task id → exactly one runs (row-level claim). +- Dogfood: the entire "stranded MACP" finding — zero production calls to + `runGates`/`emitEvent`/`resolveCredentials`. +- Fail-closed points: schema-invalid ⇒ reject; credentials unresolved ⇒ reject; crash ⇒ non-terminal. +- ⚠ **I dissent from the scout's chosen wire-in point.** See Dissent D2 — wiring into + `mosaic_orchestrator.py::run_single_task` puts the choke point inside a controller that is + `"enabled": false`. Wire the live path instead (R-023/R-024) and delete the Python rail (R-030). + +--- + +**R-021 — Gate-runner hardening** +_build 1 · depends: R-020 · 120k · sonnet_ + +The Python rail runs raw string gates (`mosaic_orchestrator.py:213-235`) with no `type`/`fail_on` +support. The package runner supports structured `GateEntry`. Hardening: honor `fail_on`, enforce +per-gate timeouts distinct from the task timeout, and **treat an empty gate list as failure for +gate-required task types** rather than as vacuous success. + +- AC1: `quality_gates: []` on a `coding` task ⇒ task not `completed`; error names `no gates +configured`. Registered must-fail case — this is the Forge-stub bug generalized. +- AC2: a gate exceeding its timeout ⇒ `timed_out: true` and `passed: false` (never `passed: true`). +- AC3: `fail_on: blocker` with 3 non-blocking findings ⇒ pass; with 1 blocker ⇒ fail. +- AC4: a gate command that does not exist (rc 127) ⇒ fail, not skip. +- Dogfood: `packages/forge/src/cli.ts:13-31` stub executor "immediately reports completion with empty + gates" — an inert gate by construction. +- Fail-closed point: **absent/unrunnable gates fail the task.** + +--- + +**R-022 — Hash-chained MACPEvent ledger + lifecycle EventType extension** +_build 1 · depends: R-020, R-011 · 160k · opus_ + +Emitter targets PG (via R-011), not a caller-supplied NDJSON path. Extend `EventType` from +task-centric to lifecycle-first-class: `session.launched`, `mission.generated`, +`compaction.detected`, `session.rotated`, `recovery.started`, `breakglass.used`, +`inbox.received`, `inbox.consumed`, `terminal.disposition`. Executor-emitted, so lines are identical +across Claude/Codex/Pi. + +- AC1: run the same Task under two runtimes → event sequences are byte-identical modulo + `event_id`/`timestamp`/`source`. **This is the runtime-neutrality claim; assert it, don't assume it.** +- AC2: chain verification ⇒0 on a healthy ledger, ⇒≠0 after any row alteration. Registered must-fail case. +- AC3: an emit that fails to persist ⇒ the operation it describes fails (no fire-and-forget logging on + the authority path). +- AC4: `grep -rn "events.ndjson"` in production paths returns nothing (the duplicate island is gone — + enforced permanently by R-065). +- Dogfood: P-AUDIT-001 duplicate Python ledger; `#1018` reconstruction required manual archaeology. +- Fail-closed point: **unpersistable event ⇒ operation fails.** (Yes, this trades availability for + auditability. That is the decision on record.) + +--- + +**R-023 — `packages/coord` submits through the executor** +_build 1 · depends: R-020 · 140k · sonnet_ + +`packages/coord/src/runner.ts:397-427` spawns a child process directly and tracks tasks in +`docs/TASKS.md` + `.mosaic/orchestrator/mission.json`. Replace the spawn with an executor submit; +keep coord's session-lock/crash-recovery, which are good and already durable. + +- AC1: `mosaic coord run` on a fixture mission produces `macp_tasks` + `macp_events` rows; no direct + `spawn` of the runtime remains in `runner.ts` (assert by AST/grep in the R-025 check). +- AC2: crash recovery still works — kill mid-task, resume, and the task is reclaimed exactly once. +- AC3: coord no longer _writes_ `docs/TASKS.md` as authority (read-only mirror at most, per R-062). +- Dogfood: coord/macp "disconnected islands" finding. +- Fail-closed point: **coord cannot complete a task the executor did not run.** + +--- + +**R-024 — Launch path records a typed Task** +_build 1 · depends: R-020, R-022 · 160k · sonnet_ + +`packages/mosaic/src/commands/launch.ts:730-843,1102-1167` dispatches `mosaic yolo|claude|codex|pi` +straight to harness launch with no Task, no gates, no events. This is the path humans and the fleet +actually use — leaving it outside the choke point leaves the choke point decorative. + +- AC1: `mosaic yolo …` creates a `macp_tasks` row and `session.launched` event before the + harness starts. +- AC2: the launch is refused if the spine is unreachable (per R-012). Registered must-fail case. +- AC3: seat identity is bound at launch and present on every subsequent event from that seat. +- AC4: an interactive/ephemeral launch class is supported explicitly and is _labeled_ as ungated — + honest capability labeling, not a silent hole. +- Dogfood: "direct `mosaic yolo|claude|codex|opencode|pi` also bypasses it" (scout, §2). +- Fail-closed point: **no spine ⇒ no launch** (for managed seats). + +--- + +**R-025 — No-second-path gate** ★ the invariant that makes R-020 mean anything +_build 1 · depends: R-020, R-022 · 140k · opus_ + +A choke point is only a choke point if nothing else can write a terminal outcome. Two layers: +(1) DB — only the executor's role may insert/transition to terminal status; (2) CI — a static check +that fails the build if any file outside the executor writes terminal task status, appends to the +ledger, or spawns a runtime process directly. + +- AC1: a test that connects as a non-executor role and attempts to set `status='completed'` ⇒ error. + Registered must-fail case. +- AC2: add a file that writes terminal status → CI ⇒≠0 naming the file. **A reviewer must actually add + such a file and watch it fail.** +- AC3: the check enumerates its own coverage (which paths it scanned) so an empty scan is visible — + an inert _checker_ is the same disease. +- AC4: known-legitimate exceptions live in an explicit allowlist with a reason string; an empty-reason + entry ⇒≠0. +- Dogfood: three parallel MACP islands, each able to declare a task complete. +- Fail-closed point: **unknown writer ⇒ build fails / DB rejects.** +- ⚠ **Ordering hazard:** if R-025 lands a milestone _after_ R-020, the intervening period is when a + fourth island gets built, and R-025 then arrives as a large adversarial refactor nobody wants to + merge. Ship them together. + +--- + +**R-026 — Delete the Forge stub executor** +_build 1 · depends: R-020 · 90k · codex_ + +- AC1: `mosaic forge run` on a fixture pipeline produces real gate results; a fixture with a failing + gate ⇒≠0. Registered must-fail case (today it would exit 0 with empty gates). +- AC2: `grep -n "stub" packages/forge/src/cli.ts` returns nothing; type-only MACP imports become value + imports. +- Dogfood: `packages/forge/src/cli.ts:13-31,167,185`. +- Fail-closed point: **no executor injected ⇒ forge refuses to run** (not "runs with a stub"). + +--- + +**R-027 — Seat identity + mandatory tri-state write outcomes** +_build 1 · depends: R-020 · 150k · opus_ + +Two halves of P-WRAPPER-001. (a) Identity derives from `MOSAIC_AGENT_NAME` at launch, survives +respawn, resolved through the executor's credential binding, fail-closed. (b) Every write/send +wrapper returns exactly one of `written+verified` / `written+unverified` / `not-written`, with distinct +exit codes. Today `send-message.sh:111` prints `✓ sent … (submission state indeterminate)` and +**exits 0** — a written+unverified reported as success, the discarded-measurement anti-pattern. + +- AC1: respawn a seat pane; the seat's identity is unchanged and `MOSAIC_GIT_IDENTITY` resolves + without being inlined by hand. (rev-974's respawn dropped it → token-not-found → could not post.) +- AC2: with identity unresolvable, the wrapper ⇒≠0 and performs no write. Registered must-fail case. +- AC3: force an indeterminate submission (target pane exists, no verification possible) ⇒ exit code + distinct from both success and hard failure, and the string `unverified` on stderr. +- AC4: a caller that treats any nonzero as success is caught by an R-002 registry case. +- AC5: safe target metadata (repo / id / head SHA) is named in the wrapper contract and verified by + construction — a review posted to the wrong SHA ⇒≠0. +- Dogfood: rev-974 identity drop (OpenBrain 33bef845); pepper written+verified; the + written+unverified silent-success class. +- Fail-closed point: **cannot verify ⇒ never report success.** + +--- + +**R-028 — Typed state claims with HMAC integrity** +_build 1 · depends: R-011, R-020 · 170k · opus_ + +Replace the prose-blob checkpoint with typed claims: `{key, value, source, confidence, ttl, +refreshed_at}`, signed. A corrupt or unsigned claim set refuses to rehydrate and forces refresh. + +- AC1: write a claim set, flip one byte, attempt rehydrate ⇒≠0 with `integrity`. Registered must-fail case. +- AC2: an expired claim is returned as expired and cannot satisfy a required-claim check. +- AC3: an unsigned claim set (legacy/prose) ⇒ refuse, with a named migration path. +- AC4: the HMAC key is resolved through the credential path and never appears in any output. +- Dogfood: `MOS-ORCHESTRATION-BOARD-LIVE.md` is the model-maintained manual prototype of this; + Jason's "assuming data not corrupted" is the requirement being mechanized. +- Fail-closed point: **corrupt checkpoint ⇒ refuse-rehydrate + force refresh.** + +--- + +**R-029 — Contract-hash binding; stale generation loses authority** +_build 1 · depends: R-020, R-028 · 180k · opus_ + +Session binds a hash of (Constitution + AGENTS + runtime contract + skill set) at launch **and** at +rotation. On policy change or compaction-detected, the seat must re-attest before acting. The executor +refuses mutations carrying a stale generation. + +- AC1: submit a task with a stale contract hash ⇒ rejected with `stale_generation`. Registered must-fail case. +- AC2: change any directive file → the next submit from an unrefreshed seat ⇒ rejected. +- AC3: after re-anchor, the same submit ⇒0. +- AC4: the rejection is _mechanical_ — assert it happens with the seat's LLM removed from the loop + (a scripted client reproduces it). +- Dogfood: the postmortem's opening incident — a compacted orchestrator that kept acting after its + directives rotted; this very mission's `CLAUDE.md` context-rot guard is the prose version. +- Fail-closed point: **stale hash ⇒ mutations rejected.** +- Note: this is the single most likely task to produce fleet-wide breakage on rollout. Ship behind a + report-only mode first (log rejections without enforcing) for one measured window, then flip. + Report-only mode must itself be time-boxed and expire — see D5. + +--- + +**R-030 — Retire the Python controller + `plugins/macp` island** +_build 1 · depends: R-023, R-024, R-025, R-026 · 110k · codex_ + +Delete `mosaic_orchestrator.py`'s exec/gate/event block and the redefined types in +`plugins/macp/src/macp-runtime.ts:43-77`; repoint the OpenClaw ACP backend at the Node executor. Also +removes the dangling reference to `tools/macp/dispatcher/pi_runner.ts`, which does not exist in this +checkout. + +- AC1: island count is 1 — `grep -rn "def emit_event\|def append_event" packages/` returns nothing. +- AC2: `plugins/macp` integration test drives a task end-to-end through the Node executor. +- AC3: `.mosaic/orchestrator/{tasks,state}.json` and `events.ndjson` are no longer written by any code + path (assert by running a full mission and checking mtimes). +- AC4: a re-added duplicate emitter ⇒ R-025/R-065 check fails. +- Dogfood: "three parallel islands, none wired." +- Fail-closed point: n/a — this is a deletion; its guard is R-065. + +### P3 — Rotation lifecycle + +--- + +**R-040 — Durable compaction/token sensor** +_build 3 · depends: R-022, R-029 · 130k · sonnet_ + +Nothing durable tracks compactions today (the Claude compaction hook "revokes leases only"). Add a +per-seat token/compaction counter persisted to the spine, with per-runtime thresholds +(Claude ~200k, Codex/Pi ~372k) as configuration, not constants in code. + +- AC1: driving a seat past the configured threshold emits `compaction.pending` with the observed count. +- AC2: an actual harness compaction emits `compaction.detected` — assert by triggering a real compaction, + not by unit-mocking the hook. (A mocked sensor is an inert sensor.) +- AC3: sensor failure (hook not installed) ⇒ the seat is reported _unmonitored_, and managed seats + refuse to start unmonitored. Registered must-fail case. +- AC4: counters survive a seat respawn. +- Dogfood: this orchestrator lineage itself — a compacted session that kept operating. +- Fail-closed point: **no sensor ⇒ seat not managed ⇒ launch refused.** + +--- + +**R-041 — Typed checkpoint writer** +_build 3 · depends: R-028, R-040 · 150k · opus_ + +Structured essentials only — goal, current state, completed, blocked, next steps, constraints +(mission-control FR-5) — as signed claims. **Never the transcript.** Atomic write + digest, into +`interaction_checkpoints` (which already has `checkpoint_id`, `content_digest`, `compaction_epoch`). + +- AC1: checkpoint written; digest verifies; `compaction_epoch` increments monotonically. +- AC2: kill the writer mid-write → no partial checkpoint is readable (atomic rename or tx). Registered + must-fail case: a torn checkpoint must be rejected, not repaired-by-guess. +- AC3: the checkpoint contains no transcript text — assert by size bound and by a content check that + rejects raw conversational turns. +- AC4: required claim classes are declared in a manifest; a checkpoint missing one is _invalid at write + time_, not at read time. +- Dogfood: the abandoned coordinator PoC "compacted THE CHAT (the anti-pattern) not canonical mission + state." +- Fail-closed point: **incomplete claim set ⇒ checkpoint rejected.** + +--- + +**R-042 — Rotation daemon** ★ the build-3 deliverable +_build 3 · depends: R-041, R-023 · 240k · opus_ + +The missing piece: a deterministic program (no LLM in the loop) that watches the sensor, checkpoints +atomically, kills the old session, launches a fresh one, and rehydrates. Reuse coord's +`buildContinuationPrompt`, session lock, `writeAtomicJson`, and crash recovery. The abandoned +`apps/coordinator` PoC stalled exactly here: `_spawn_agent()` was a stub and `_check_context()` only +_logged_ "rotation needed" and never called `trigger_rotation()`. + +- AC1: a seat crossing threshold rotates without operator action; the new seat reports the same mission + and next action. +- AC2: **`_check_context`-equivalent regression test:** a test asserts that threshold-crossing actually + _invokes_ rotation (not merely logs it). This is the specific historical failure — assert the call, + not the log line. Registered must-fail case: stub out the rotation call → test ⇒≠0. +- AC3: no LLM call occurs in the rotation path — assert by running the daemon with network/model access + denied; rotation still completes. +- AC4: rotation is idempotent under double-trigger (only one new session). +- AC5: if checkpoint or rehydration fails, the daemon does **not** kill the old session. +- Dogfood: `apps/coordinator` residue (Jan–Feb 2026, deliberately abandoned). +- Fail-closed points: **checkpoint failure ⇒ no kill**; **rehydration failure ⇒ no promote** (R-043). +- ⚠ Secretly greenfield: the daemon does not exist in any form. `packages/coord` has primitives, not a + daemon. Budget accordingly. + +--- + +**R-043 — Rehydration attestation gate** +_build 3 · depends: R-041 · 130k · opus_ + +The pessimist's core objection to rotation: an incomplete rehydration produces a clean-looking session +with a silent gap — worse than a compacted one, because nothing signals the loss. A fresh seat must +attest that every required claim class is present and digest-valid _before_ it is permitted to act. + +- AC1: with a complete claim set, the new seat's first mutation ⇒0. +- AC2: with one required claim class removed, the new seat's first mutation ⇒ rejected with + `rehydration_incomplete`, and the seat is quarantined, not silently continued. Registered must-fail case. +- AC3: attestation is enforced executor-side (R-029 machinery), so a seat cannot self-certify. +- AC4: the operator sees a distinct, greppable state for `rehydrated-degraded`. +- Dogfood: this session lineage's hand-run `mosaic-context-refresh` "fail-closed residency attestation" + — the prose version of exactly this gate. +- Fail-closed point: **incomplete rehydration ⇒ seat may not mutate.** + +--- + +**R-044 — Fault-injected rotation soak** +_build 3 · depends: R-042, R-043 · 180k · sonnet_ + +The "100 rotations lossless" bar, made adversarial. 100 clean rotations prove almost nothing; the +failure mode is a kill at the worst moment. + +- AC1: 100 consecutive rotations, claim-set diff empty at every boundary (machine-compared, not + eyeballed). +- AC2: fault matrix, each ≥10 iterations: kill during checkpoint write; kill after checkpoint before + old-session kill; kill after kill before relaunch; DB unavailable during checkpoint; corrupt + checkpoint; two daemons racing; contract hash changed mid-rotation. **Each must end in a defined + state — either fully rotated or fully not; never half.** +- AC3: any iteration ending `rehydrated-degraded` fails the suite. +- AC4: the suite runs against real runtime artifacts (real seats, real spine), not mocks — + P-CONFORMANCE-001 criterion (1). +- Dogfood: all of P-LIFECYCLE-001; the harness is the P-CONFORMANCE down payment. +- Fail-closed point: **any half-state ⇒ suite red ⇒ rotation does not ship.** + +--- + +**R-045 — Delete `/compact and continue` from the persistent-seat path** +_build 3 · depends: R-042, R-044 · 60k · codex_ + +Removal = substitution. Keep it for ephemeral seats (one doctrine per session class). + +- AC1: `grep -rn "compact and continue"` in orchestrator/persistent guidance returns nothing. +- AC2: ephemeral-seat guidance still contains it, and a test asserts the two session classes are + distinguished by a machine-readable attribute, not by prose. +- AC3: a persistent seat attempting the old flow is redirected to rotation by the daemon. +- Dogfood: P-GUIDE-001; the "live irony" that the deciding session was itself a hand-re-anchored + compacted orchestrator. +- Fail-closed point: n/a (guidance change) — but see the hazard. +- ⚠ **Ordering hazard (severe):** merging this before R-042/R-044 are _live on the fleet_ removes the + coping mechanism while leaving the failure in place. Hard edge, and the orchestrator should verify + deployment, not just merge. + +### P4 — Comms service + +--- + +**R-050 — `comms/v1` envelope + protocol-version negotiation** +_build 4 · depends: R-011 · 140k · opus_ + +Version the protocol, not the participants. Envelope carries `comms/v1`; receiver rejects unsupported +versions LOUDLY; N-version compatibility window; framework version is diagnostic-only (avoids the N² +matrix). This is the first breaking comms change, so the negotiation must exist before anything rides +on it. + +- AC1: a `comms/v1` envelope round-trips; digest and correlation id preserved. +- AC2: a `comms/v99` envelope ⇒ rejected with a machine-readable `unsupported_protocol_version` and a + visible operator message. Registered must-fail case. +- AC3: a malformed/unsigned envelope ⇒ rejected (never best-effort parsed). +- AC4: the supported-version window is data, not code branches; shrinking it in config immediately + rejects the dropped version. +- Dogfood: P-AUTHORITY-001; "bare tmux = comms P0/PoC" with no version field at all. +- Fail-closed point: **unknown protocol version ⇒ reject, loudly.** + +--- + +**R-051 — Comms service: PG state machine** +_build 4 · depends: R-050, R-013 · 200k · opus_ + +Sole-path service. `PENDING → RECEIVED → CONSUMED → DEAD-LETTER`, ack cursor, per-recipient filter. +`interaction_inbox` already provides the unique `(session_id, idempotency_key)` index and the +status enum — extend rather than fork. + +- AC1: a message with no consumer remains `PENDING` and is retried; it is never reported delivered. +- AC2: a bounced transport attempt does not advance the cursor. Registered must-fail case: force a + bounce, assert the cursor is unchanged and a retry occurs. +- AC3: after N failed attempts the message goes `DEAD-LETTER` and raises a visible operator signal — + dead-lettering silently is the same bug in a new hat. +- AC4: `RECEIVED` and `CONSUMED` are distinct and both observable; a manual liveness check is never + required to learn a message's fate. +- Dogfood: the MACP scout bounce ("tmux target not found") — never RECEIVED, no retry, discovered only + by hand. +- Fail-closed point: **un-acked ⇒ not consumed ⇒ retried.** + +--- + +**R-052 — tmux transport adapter behind the service** +_build 4 · depends: R-051, R-060 · 160k · sonnet_ + +tmux becomes the first dumb adapter, not the authority. Durable retry before cursor advance. + +- AC1: sending to a nonexistent pane ⇒ message stays `PENDING`, retried, and the sender receives + `not-written` (R-027 tri-state), not `✓`. +- AC2: sending to a busy pane ⇒ `written+unverified` until the receipt is observed, then `RECEIVED`. +- AC3: adapter failure never loses the message — kill the adapter mid-send, assert redelivery. +- AC4: the adapter cannot mark anything `CONSUMED`; only the recipient's ack can. Registered must-fail case. +- Dogfood: scout-bounce; `send-message.sh:71` (`tmux target not found` → exit 1, no retry) and `:111`. +- Fail-closed point: **transport cannot self-certify delivery.** +- ⚠ Ordering: requires R-060 — identity addressing is meaningless while seats are split across the + `default` and `mosaic-fleet` sockets (that split _was_ the bounce). + +--- + +**R-053 — Per-class coalescing + supersede** +_build 4 · depends: R-051 · 130k · sonnet_ + +The `#1018` case: pepper's "awaiting your word" arrived _after_ the PR had merged — stale-consumed-as-live. + +- AC1: two messages of the same class for the same subject where the later supersedes → the recipient + sees one, the newer, with the older marked superseded (not deleted — auditability). +- AC2: a message whose subject has reached a terminal state is delivered _marked stale_, never as live + actionable. Registered must-fail case: assert an actionable-class stale message does not present as + actionable. +- AC3: ordering within a class is monotonic per sender. +- AC4: coalescing never drops a message with no successor. +- Dogfood: `#1018` stale-consumed. +- Fail-closed point: **ambiguous freshness ⇒ mark stale, never present as live.** + +--- + +**R-054 — Redis hot path + provenance guard** +_build 4 · depends: R-051, R-013 · 170k · opus_ + +Redis Streams + PEL as a derived, rebuildable accelerator. PG remains the protected tier. + +- AC1: flush Redis entirely → the sweeper rebuilds the hot queue from PG with zero message loss and + zero duplicates. This is the acceptance test that matters. +- AC2: every read from Redis carries a provenance flag; a code path that returns Redis-derived data as + authoritative ⇒ CI check fails (registered with R-025's machinery). Registered must-fail case: add + such a path and watch the build go red. +- AC3: Redis unavailable ⇒ service degrades to PG-only and _says so_; it does not fail open to + "assume delivered". +- AC4: split-brain injection (Redis has a message PG does not) → reconciliation resolves toward PG and + emits an event. +- Dogfood: synthetic. Justification: the moment a fast tier exists, someone reads it as truth — this + guard is cheap now and impossible later. +- Fail-closed point: **Redis is never authority.** + +--- + +**R-055 — Retire direct tmux sends** +_build 4 · depends: R-052, R-053 · 100k · codex_ + +- AC1: a CI check fails on any direct `tmux send-keys` to an agent pane outside the adapter. +- AC2: `agent-send.sh` becomes a thin client of the service; its exit codes remain tri-state per R-027. +- AC3: existing fleet scripts are migrated; the check enumerates what it scanned (no silent empty scan). +- Dogfood: the whole bare-tmux P0 comms substrate. +- Fail-closed point: **out-of-band send ⇒ build fails.** + +### P5 — Retirements, hygiene, conformance + +--- + +**R-060 — One roster-owned socket/host, quarantine, stale GC** +_build 5 · depends: R-005 · 150k · sonnet_ + +- AC1: every roster seat resolves to the single roster-owned socket; a seat launched on another socket + is quarantined and reported. Registered must-fail case. +- AC2: `fleet status` lists unmanaged sessions distinctly; count is nonzero in a test that plants one. +- AC3: stale-session GC reaps or rotates by max-age/max-context, with the decision recorded as an event. +- AC4: GC never reaps a session holding an active mission claim. +- Dogfood: scout-bounce root cause — "seats split default vs mosaic-fleet socket." +- Fail-closed point: **unmanaged seat ⇒ quarantined, not addressed.** + +--- + +**R-061 — Auto-sync allowlist + worktree isolation** +_build 5 · depends: R-002 · 110k · sonnet_ + +`brain-sync.sh:99` does `git add -- "$@"`; `session-end.sh:48` stages `data/ views/ domains/ docs/ '*.md'`. +The measured incident (517bd5c26) staged agent-authored `docs/postmortem-spec/site/*` mid-write. + +- AC1: staging is allowlist-driven; a path outside the allowlist is never staged. Registered must-fail + case: place an unknown path, run sync, assert it is untouched **and** reported. +- AC2: a half-written agent-authored file under an agent-owned path is never swept (lease/worktree check). +- AC3: the sync reports what it declined to stage (silence on decline is how this gets re-broken). +- Dogfood: P-WORKFLOW-001, sweep 517bd5c26. +- Fail-closed point: **unknown path ⇒ not staged, and said out loud.** +- Note: criterion (1) of P-WORKFLOW-001 is partially obviated by R-062 (DB tracking), but (2)+(3) + remain for code/docs in project repos. Do not cancel this task on the strength of R-062. + +--- + +**R-062 — Flat-file orchestration tracking → DB cutover, with rollback artifact** ★ highest data risk +_build 2/5 · depends: R-024, R-028, R-042 · 200k · opus_ + +Jason's directive: hard cutover, no flat-file interim. I comply on the _write_ path and dissent on the +unqualified form (D3). Scope: mosaic fleet orchestration state only — missions, agent-task +assignments, `MOS-ORCHESTRATION-BOARD-{LIVE,LEDGER}`, `docs/TASKS.md`, `.mosaic/orchestrator/*`. +**jarvis-brain PDA flat files (`data/projects`, `data/tasks`) are explicitly out of scope and must be +proven untouched.** + +- AC1: a one-shot importer migrates the current board + ledger + `docs/TASKS.md` into the spine; + re-running it is idempotent (no duplicate missions/tasks). +- AC2: **in-flight safety** — a mission with a running task is migrated without the task being lost or + double-claimed. Test by migrating while a task is mid-execution; assert exactly-once completion. +- AC3: after cutover, flat files are no longer written (mtime unchanged over a full mission run). +- AC4: a **read-only export** regenerates a human-readable board from the DB on demand. This is the + rollback artifact, not a fallback tracking system — it is never read back as authority (enforced by + R-025's writer check). +- AC5: documented, tested rollback: restore the pre-cutover snapshot and resume flat-file tracking + within one operator step. Exercised once in a rehearsal before the real cutover. +- AC6: `git diff --stat` over `jarvis-brain/data/` across the cutover is empty. +- Dogfood: the flat-file clobber class (P-WORKFLOW-001); the co-location of fleet state inside the + brain repo, which the postmortem names as part of the bug. +- Fail-closed points: **importer aborts on any ambiguity rather than guessing**; **cutover refuses to + run while a rotation is in flight.** + +--- + +**R-063 — Conformance harness** +_build 5 · depends: R-044, R-051, R-062 · 260k · opus_ + +Fault-inject the six live failure classes against real runtime artifacts on the DB substrate: +compaction/rotation, broker-unavailable, delivery-bounce, identity-drop, stale-hash, queue-inert. +Flat-file cases become regression guards. + +- AC1: each of the six classes has an injector, an expected-outcome assertion, and a _demonstrated + red_ — the harness must be shown failing when the corresponding fix is reverted. **A harness that has + never been red is an inert gate.** +- AC2: the six seed incidents are reproduced by id: Pi brick, scout-bounce, gate-6 inert, #1019 + recursion, identity drift, auto-sync sweep, #1018 stale-consumed. +- AC3: harness runs in CI on a schedule and on release; results land in the event ledger. +- AC4: harness self-reports coverage (which classes ran); a skipped class is a failure, not a silent pass. +- Dogfood: it _is_ the dogfood — the automated form of the standing directive. +- Fail-closed point: **skipped class ⇒ suite red.** + +--- + +**R-064 — Fleet-wide inert-gate audit** +_build 5 · depends: R-002 · 120k · sonnet_ + +Apply the R-002 registry to every existing gate in `~/.config/mosaic/tools/` and the repo. Expect +casualties beyond gate-6. + +- AC1: every gate is registered; the audit report lists each gate with its must-fail case and the + observed exit code. +- AC2: every gate found inert is either fixed in this PR or filed with an issue id in the registry; + an unexplained inert gate ⇒ CI red. +- AC3: the report is committed as evidence (this is the "prove it" artifact for P-QUEUE-001's + interim doctrine: _queue-guard green = zero information until proven otherwise_). +- Dogfood: gate-6 was inert **fleet-wide** and nobody knew; the base rate of inert gates is unknown and + that is itself the finding. +- Fail-closed point: **inert gate without a filed issue ⇒ red.** + +--- + +**R-065 — Retirement proof** +_build 5 · depends: R-030, R-055, R-062 · 90k · codex_ + +A CI check asserting the three cross-cutting retirements are complete **and stay complete**: no +flat-file orchestration tracking writes, exactly one MACP implementation, no silent bypass markers. + +- AC1: reintroduce each retired pattern in a scratch branch → check ⇒≠0, once per pattern. Registered + must-fail cases (three of them). +- AC2: the check enumerates what it scanned. +- AC3: allowlisted exceptions carry an expiry date; an expired exception ⇒≠0. +- Dogfood: the entire "built-but-unwired / retired-but-resurrected" disease. +- Fail-closed point: **resurrection ⇒ build fails.** + +--- + +## 4. Fail-closed register + +Every point where this plan chooses safety over availability. If any of these is later softened, the +corresponding failure class returns. + +| # | Point | Task | What it costs when it fires | +| ----- | ------------------------------------------------------------- | ------------ | ------------------------------------------------------- | +| FC-1 | Unregistered gate ⇒ build fails | R-002 | New gates need a negative control before merge | +| FC-2 | Unknown/no-status/malformed CI state ⇒ merge blocked | R-003 | Merges wait on flaky status APIs | +| FC-3 | Broker absent + no break-glass ⇒ deny | R-004 | Requires R-005 first or the fleet bricks | +| FC-4 | Version skew ⇒ launch blocked | R-005 | Seats refuse to start after a partial upgrade | +| FC-5 | Skipped migration ⇒ error | R-010 | First-install must be fixed, not worked around | +| FC-6 | Ledger mutation ⇒ DB error | R-011 | No "cleanup" of bad events; only compensating entries | +| FC-7 | No DB ⇒ no managed work | R-012, R-024 | Spine outage stops the fleet. Accepted deliberately | +| FC-8 | Schema-invalid / credentials unresolved ⇒ reject before spawn | R-020 | Bad task definitions surface loudly | +| FC-9 | Executor crash ⇒ non-terminal state | R-020 | Requires a reclaim sweeper | +| FC-10 | Empty/unrunnable gate set ⇒ task fails | R-021 | Gate-less task types must be declared explicitly | +| FC-11 | Unpersistable event ⇒ operation fails | R-022 | Auditability outranks availability | +| FC-12 | Unknown terminal-status writer ⇒ build/DB rejects | R-025 | Every new writer needs an allowlist entry with a reason | +| FC-13 | Identity unresolvable ⇒ no write | R-027 | Respawned seats must re-derive identity or stop | +| FC-14 | Cannot verify ⇒ never report success | R-027 | Callers must handle a third exit code | +| FC-15 | Corrupt checkpoint ⇒ refuse rehydrate | R-028, R-041 | Forces a refresh cycle | +| FC-16 | Stale contract hash ⇒ mutations rejected | R-029 | Highest blast radius; ship report-only first | +| FC-17 | Unmonitored seat ⇒ launch refused | R-040 | Hook install becomes mandatory | +| FC-18 | Checkpoint failure ⇒ old session not killed | R-042 | Rotation stalls rather than losing state | +| FC-19 | Incomplete rehydration ⇒ seat may not mutate | R-043 | Quarantined seats need operator attention | +| FC-20 | Unknown protocol version ⇒ reject loudly | R-050 | Rollout needs the N-version window honored | +| FC-21 | Un-acked ⇒ not consumed ⇒ retried | R-051 | Duplicate-tolerant consumers required | +| FC-22 | Transport cannot self-certify delivery | R-052 | tmux "✓" becomes tri-state | +| FC-23 | Ambiguous freshness ⇒ marked stale | R-053 | Some genuinely-live messages arrive marked stale | +| FC-24 | Redis never authority | R-054 | Extra PG round-trip on the read path | +| FC-25 | Unmanaged seat ⇒ quarantined | R-060 | Ad-hoc tmux seats stop being addressable | +| FC-26 | Unknown path ⇒ not staged, and reported | R-061 | Sync gets noisier, deliberately | +| FC-27 | Importer aborts on ambiguity | R-062 | Cutover may need manual disambiguation | +| FC-28 | Skipped conformance class ⇒ suite red | R-063 | No partial conformance claims | +| FC-29 | Retirement resurrection ⇒ build fails | R-065 | Exceptions must carry expiry dates | + +--- + +## 5. Ordering hazards (later task silently depends on an earlier invariant) + +**H-1 — R-004 before R-005 re-bricks the fleet.** The three bypasses are load-bearing _because the +broker was never deployed_. Remove the fail-open before deployment coherence exists and you reproduce +2026-07-22 exactly, including the Pi brick. Hardest edge in the DAG. + +**H-2 — Every migration-adding task silently depends on R-010.** R-011/R-013/R-022/R-028/R-050/R-051 +will all pass in CI on PGlite and fail on the first real Postgres install. The defect is documented in +a TODO and has never been hit because nobody first-installed the PG tier. + +**H-3 — R-025 must ship with R-020, not after.** The choke point's only real property is exclusivity. +The gap between "executor exists" and "executor is the only path" is exactly when island #4 gets built. + +**H-4 — R-045 before R-042/R-044 are _deployed_ removes the coping mechanism.** Merge ≠ live. The +orchestrator must confirm rotation is running on the fleet, not just merged. + +**H-5 — R-052 depends on R-060.** Identity addressing across two tmux sockets is the bounce that +motivated the whole comms build. Building the adapter first just relocates the bug. + +**H-6 — R-062 depends on R-042.** Cutting over tracking while rotation is half-built means a rotation +lands mid-migration with state split across both substrates. R-062 must also refuse to run during an +in-flight rotation (FC-27). + +**H-7 — R-029 depends on R-028's claim signing.** A contract hash that is not itself integrity-protected +is a suggestion. Also: R-029 has the widest blast radius of any task here; report-only first. + +**H-8 — R-021's "empty gate set = failure" invariant is assumed by R-026.** If R-026 lands first, +Forge's stub is replaced by a real executor that will happily run zero gates. + +**H-9 — R-063 assumes every earlier fix is _revertible in a test harness_.** If tasks land without +feature flags or clean revert points, "demonstrate the harness red" (AC1) becomes impossible and the +conformance suite silently becomes decorative. **Every P0–P4 task should land with a documented way to +disable its fix in a test context.** This is a cross-cutting requirement the orchestrator should add to +the charter, not a per-task note. + +**H-10 — R-002's registry is only as good as its enumeration.** If the registry is populated by hand +from a partial list, unregistered gates keep passing. R-002 AC3 (unregistered-gate detection) is the +load-bearing half; R-064 is its audit. + +--- + +## 6. Secretly-greenfield register + +"Finish, don't re-spec" is accurate at the _specification_ level and misleading at the code level. Of +38 tasks, these have no existing implementation to finish: + +| Task | Reality | +| ----------- | --------------------------------------------------------------------------------------------------- | +| R-002 | Gate registry + negative-control runner — new, no precedent in repo | +| R-011 | Hash-chained append-only ledger — the existing `events` table is calendar-shaped | +| R-013 | Outbox sweeper — tables exist, sweeper does not | +| R-020 | Executor — the package has **no exported programmatic submit** (scout §1); only a CLI placeholder | +| R-025 | No-second-path enforcement — entirely new | +| R-028 | HMAC claim integrity — new | +| R-029 | Contract-hash binding — new | +| R-040 | Durable compaction sensor — "nothing durable today" | +| R-042 | Rotation daemon — the PoC's `_spawn_agent()` was a stub; `packages/coord` has primitives, no daemon | +| R-043 | Rehydration attestation — new | +| R-050/R-051 | comms/v1 envelope + service — tables exist, protocol and service do not | +| R-054 | Provenance guard — new | +| R-063 | Conformance harness — new | + +**Estimate: ~13 of 38 tasks (≈40% of tokens) are new construction.** What genuinely exists and should +be _reused, not rebuilt_: `packages/macp` types/credential-resolver/gate-runner/risk-floor; +`packages/db` schema + drizzle migrations (post-R-010); `packages/queue` `QueueAdapter`; +`packages/coord` mission/session/capsule/crash-recovery primitives; `interaction_*` tables; +PG + Redis already running in-stack. That is a real head start — it is just not "wiring." + +--- + +## 7. Migration & rollback (the hard cutover) + +Per Jason: hard cutover, no flat-file interim. Concretely, R-062 executes in five steps, each of which +must be reversible until the last: + +1. **Rehearsal** (throwaway DB): run the importer against a copy of the current board/ledger/TASKS.md. + Diff the reconstructed board against the flat original. Any unexplained delta ⇒ stop. +2. **Freeze window**: no rotation in flight (FC-27), no task mid-execution beyond a known set. +3. **Snapshot**: tag the flat-file state (git tag + tarball) — this is the rollback artifact. +4. **Import + flip**: importer runs; writers switch to the spine in one commit; flat files become + read-only outputs. +5. **Verification window**: for one measured period, a scheduled job regenerates the human board from + the DB and diffs it against the frozen snapshot's _shape_ (not content). Divergence in shape ⇒ + investigate. **This is not a dual-write and not an interim tracking system** — it is a + one-directional export that proves the DB is complete. + +**Rollback trigger:** any of — importer AC1 idempotency failure, a lost in-flight task (AC2), or spine +unavailability exceeding the agreed window. **Rollback action:** restore the snapshot, revert the +writer flip, resume flat-file tracking. Must be exercised once in rehearsal or it is not a rollback +plan, it is a wish. + +--- + +## 8. Dissent + +**D1 — The charter's sequencing starts one phase too late.** +`MISSION.md` sequences "spine + choke-point first." I insert P0 (R-001…R-005) ahead of it. Reason: +every task in Builds 1–4 introduces gates, and this fleet has demonstrated — twice, on the same +defect — that it cannot detect an inert gate. Building the choke point first means its gates are +unverifiable _by construction_, and we would be shipping the exact class of artifact the postmortem +indicts. R-001 is also a hard blocker (nothing can be pushed today). Cost of the insertion: ~420k +tokens, roughly 8% of the plan. I consider it the highest-leverage 8% here. + +**D2 — I disagree with the scout's chosen wire-in point.** ★ headline dissent +`MACP-WIRING-SCOUT.md` names the "single integration point" as replacing the exec/gate/event block in +`mosaic_orchestrator.py::run_single_task` (:126-276). But that controller is **disabled** +(`.mosaic/orchestrator/config.json:2` — `"enabled": false`) and references a dispatcher path +(`tools/macp/dispatcher/pi_runner.ts`) that does not exist in this checkout. Wiring the new choke +point into a disabled rail produces a stranded executor — the identical disease, one layer up, and it +would look "done" in a PR. + +The paths that actually carry work today are `packages/mosaic/src/commands/launch.ts` (`mosaic +yolo|claude|codex|pi`, :730-843, :1102-1167) and `packages/coord/src/runner.ts` (:397-427). **Wire +those (R-023, R-024) and delete the Python rail (R-030) rather than porting it.** The scout's analysis +is excellent and its file:line evidence is what let me reach this conclusion — I am disagreeing with +its recommendation, not its findings. Concrete consequence: R-030 becomes a deletion task, not an +integration task, and Build 1's acceptance must be measured on a live `mosaic yolo` invocation. + +**D3 — "Hard cutover, no interim" is right about tracking and wrong about evidence.** +I comply with the directive: one write path, no flat-file interim, no dual-write. But a hard cutover +with **no rollback artifact** will, on the balance of this codebase's history, lose an in-flight +mission. R-062 AC4/AC5 add a one-directional read-only export and a rehearsed rollback. These are not +an interim tracking system — nothing reads them as authority (enforced by R-025) — they are the +snapshot you need at 3am. If the orchestrator judges even this to violate the directive, escalate to +Jason rather than silently dropping it; the difference between "no interim" and "no rollback" is worth +one question. + +**D4 — The "100 rotations lossless" bar as written is not a test, it is a demo.** +100 clean rotations exercise the happy path 100 times. The failure mode is a kill at the worst moment. +R-044 replaces the bar with a fault matrix and defines "lossless" as a machine-compared claim-set diff. +I would not accept Build 3 on 100 clean rotations alone, and I would treat any suite that has never +been observed red as unproven (R-063 AC1). + +**D5 — Report-only modes must expire, or they become the new fail-open.** +I recommend report-only rollout for R-029 (contract hash) and R-025 (no-second-path) because their +blast radius is fleet-wide. But a report-only gate is definitionally inert, and this organization has +proven it will not notice. **Every report-only mode must carry a hard expiry timestamp after which it +enforces or refuses to start** — same mechanism as R-004's break-glass. Without that clause I withdraw +the report-only recommendation and prefer a hard flip with a scheduled window. + +**D6 — P-QUEUE-001 should not be scheduled inside Build 1, and its ownership is currently ambiguous.** +The queue guard is a shell script in the shared tools path; it has no dependency on the spine or the +executor, and it is P0. Scheduling it "inside Build 1" delays a P0 fix behind a 1M-token foundation. +R-003 sits in P0 instead. Separately: PR #1023 is explicitly **parked under Mos** while HOLD is lifted +for this workstream — two lanes can legitimately claim it. **The orchestrator must resolve this with +Mos before dispatching R-003.** A third recursion on this defect would be the postmortem's own +anti-pattern, performed by the remediation. + +**D7 — Build 4's adapter fan-out (Matrix/Discord/Slack/Telegram) does not belong in this mission.** +The comms _service_, the envelope, the state machine, and one adapter (tmux) fix the observed failures. +Additional adapters are reach, not correctness, and each one adds a delivery-semantics surface that +R-063 must then cover. I recommend explicitly deferring adapters beyond tmux to a follow-on mission, +and saying so in the charter so it is a decision rather than a slip. + +**D8 — FC-7 and FC-11 are real costs, and I want them acknowledged in writing, not discovered.** +"No DB ⇒ no managed work" and "unpersistable event ⇒ operation fails" mean a Postgres outage stops the +fleet, where today a flat-file fleet limps on. That is the correct trade for a system whose defining +failure is _silent_ continuation — but it is a genuine availability regression, and the first time it +fires at 2am someone will be tempted to add a fallback. **Pre-commit to the answer now:** the fallback +is "the fleet stops," and the mitigation is spine availability (backups, restart policy, monitoring), +not a degraded write path. If that is not acceptable to Jason, the right time to say so is before R-012, +not during the incident. + +--- + +## 9. Handoff notes for the orchestrator + +- **Dispatch order for the first wave (parallelizable):** R-001 → then R-002 and R-005 concurrently → + then R-010 concurrently with R-003; R-004 waits on R-005. +- **Do not dispatch any P2 task before R-002 is merged.** That is the whole argument of D1; if it gets + compressed away under schedule pressure, the plan's central claim goes with it. +- **Pre-registration:** every AC above is written to be committed _before_ the diff is read. They are + drafts — the reviewer should tighten them per task, but the must-fail cases are non-negotiable. +- **Charter addendum recommended (H-9):** every P0–P4 task lands with a documented way to disable its + fix in a test context, so R-063 can demonstrate red. +- **Three decisions needing a human or Mos:** (1) R-003 ownership vs. parked PR #1023 [D6]; + (2) rollback artifact vs. "no interim" reading [D3]; (3) accepting the FC-7/FC-11 availability + trade [D8]. diff --git a/docs/remediation/DECOMP-SOL.md b/docs/remediation/DECOMP-SOL.md new file mode 100644 index 00000000..8fbbef65 --- /dev/null +++ b/docs/remediation/DECOMP-SOL.md @@ -0,0 +1,424 @@ +# Adversarial Decomposition — Pragmatic / Shortest-Path Side + +**Planner:** `planner-sol` +**Bias:** make one real fleet task pass through one enforced path as early as possible; reuse before building. +**Scope source:** `MISSION.md`, `MACP-WIRING-SCOUT.md`, `BOARD.md`, existing `@mosaicstack/macp`, `packages/coord`, PG/Valkey, Tess durable inbox/outbox, and the Mission Control PRD. + +## Executive position + +The first useful milestone is **not** “complete Builds 1 and 2.” It is this narrow vertical slice: + +> A DB-backed mission task is atomically claimed by `packages/coord`, executed by one Node `@mosaicstack/macp` TaskExecutor, gated, and terminally recorded with identity-bound events and a tri-state mutation result. No `docs/TASKS.md`, `mission.json`, `tasks.json`, Python gate loop, or NDJSON ledger participates. + +That slice is **SOL-03 → SOL-04 → SOL-05 → SOL-06 → SOL-07 → SOL-08**, estimated at **72K tokens**, mostly Codex. PG polling is acceptable for this first proof. Redis acceleration follows only after correctness is observable. This is the shortest path that is both dogfoodable and not throwaway work. + +### Cost posture + +- **25 PR tasks, ~294K tokens total:** ~164K Codex, ~130K Sonnet, **0K Opus**. +- First live choke-point dogfood: ~72K on the hard path; SOL-01 and SOL-02 can run beside it. +- Opus is not justified for planned implementation. Escalate only if an independent security review finds an unresolved architecture-level authority flaw. +- Every row is one PR. Estimates include implementation, focused tests, docs affected by that PR, and one remediation pass—not orchestration/reviewer overhead. + +## Gates and critical path + +| gate | opens when | proof required before downstream work | +| ------------------------------------------- | -------------- | --------------------------------------------------------------------------------------------------------- | +| **G0 — trustworthy launch gates** | SOL-01, SOL-02 | non-root checkout works; queue status cannot become false-green | +| **G1 — first dogfood / minimum viable cut** | SOL-08 | one live fleet task completes DB → MACP executor → gates → DB with no flat-file state | +| **G2 — Builds 1+2 closed** | SOL-09..SOL-12 | all producers use the executor; duplicate islands retired; Redis loss is recoverable from PG | +| **G3 — rotation real** | SOL-13..SOL-16 | stale generation cannot mutate; fresh session resumes typed state; Pi-brick recovery works without broker | +| **G4 — sole-path comms real** | SOL-17..SOL-22 | roster identity is stable; bounced/stale messages converge through PG/Redis and adapters | +| **G5 — mission proof** | SOL-23..SOL-25 | workflow sweep cannot capture unknown files; fault bank passes, including 100 rotations | + +**Critical path:** `03 → 04 → 05 → 06 → 08 → 10 → 12 → 13 → 14 → 15 → 16 → 17 → 18 → 19 → 20 → 21 → 22 → 24 → 25`. + +## Ordered task list + +### SOL-01 — Repair activation coherence and non-root checkout hygiene + +- **build:** 5 (hygiene; pulled forward) +- **depends_on:** — +- **acceptance criteria (diff-blind testable):** + 1. A clean non-root checkout can run dependency/bootstrap preparation without accessing `/root`. + 2. A root CI checkout still uses an isolated writable pnpm store. + 3. Activation installs CLI, hooks, broker/runtime assets, and version manifest transactionally: induced failure leaves the prior complete generation active. + 4. Launch with a deliberately skewed component version is rejected with the exact repair command; diagnostics remain usable. + 5. No test uses `--no-verify` or suppresses hooks. +- **dogfood seed:** BOARD D-1 root-pinned `.npmrc` and D-2 root-owned Husky path. +- **est. tokens:** 6K +- **suggested runtime tier:** codex + +### SOL-02 — Make queue guard fail-safe with exit-asserting tests + +- **build:** 1 +- **depends_on:** — +- **acceptance criteria (diff-blind testable):** + 1. Fixture responses `pending`, `success`, `failure`, no-status, malformed JSON, provider error, and unknown status produce explicitly asserted process exits. + 2. Only terminal success/no-active-queue returns 0; unknown, malformed, and transport failure return non-zero with actionable output. + 3. A payload larger than 150 KiB is consumed without argv expansion or truncation. + 4. At least one mutant changes unknown→success and is killed by the test suite. +- **dogfood seed:** inert gate-6 and recursive #1019 failure (unknown→exit 0; ARG_MAX). +- **est. tokens:** 8K +- **suggested runtime tier:** codex + +### SOL-03 — Complete the canonical MACP contract, not another protocol + +- **build:** 1 +- **depends_on:** — +- **acceptance criteria (diff-blind testable):** + 1. `@mosaicstack/macp` validates typed Task, TaskResult, lifecycle Event, state Claim, and `verified | written-unverified | failed` mutation outcome records. + 2. Claims require source, confidence, issued-at, TTL/expiry, refresh instruction, and HMAC integrity; tamper/expiry returns a typed refusal, never partial data. + 3. Lifecycle events include launch, mission generation, checkpoint, rotation, recovery, inbox receipt, and terminal disposition while preserving existing task events. + 4. `MOSAIC_AGENT_NAME` is required for mutating execution and appears in credential/actor binding; missing identity fails closed. + 5. Target metadata requires repository identity, task/record ID, and head or generation where applicable. +- **dogfood seed:** rev-974 identity drift plus the three observed write outcomes. +- **est. tokens:** 8K +- **suggested runtime tier:** codex + +### SOL-04 — Add the narrow PG orchestration spine schema + +- **build:** 2 +- **depends_on:** SOL-03 +- **acceptance criteria (diff-blind testable):** + 1. Migration up creates mission, task, dependency, task-claim, MACP event, typed state-claim, session-generation, and dispatch-outbox records with tenant/mission keys and uniqueness constraints. + 2. The database rejects a task without a mission, a dependency outside its mission, duplicate idempotency keys, and terminal→running regression. + 3. Event and claim records reference canonical mission/task/generation identities; claims store integrity metadata. + 4. Migration rollback on an empty test DB succeeds; rerunning migration is safe. + 5. No comms-specific “universal message” schema is invented here; Build 4 reuses/extends existing interaction inbox/outbox tables. +- **dogfood seed:** mission convention existed but a lane could act with no mechanically valid mission/task. +- **est. tokens:** 12K +- **suggested runtime tier:** codex + +### SOL-05 — Implement atomic PG task claims, transitions, ledger, and outbox + +- **build:** 2 +- **depends_on:** SOL-04 +- **acceptance criteria (diff-blind testable):** + 1. Two concurrent claimers for one runnable task yield exactly one lease owner. + 2. Dependencies are evaluated transactionally; an unmet dependency can never be claimed. + 3. Claim, state transition, MACP event, and dispatch-outbox append commit atomically or all roll back. + 4. Expired leases are reclaimable with a higher fencing generation; stale owners cannot complete or mutate. + 5. Querying mission status is derived solely from PG and returns the next runnable task deterministically. +- **dogfood seed:** model-maintained live board and stale claims surviving session changes. +- **est. tokens:** 12K +- **suggested runtime tier:** codex + +### SOL-06 — Build the one production Node MACP TaskExecutor + +- **build:** 1 +- **depends_on:** SOL-03, SOL-05 +- **acceptance criteria (diff-blind testable):** + 1. A public Node executor accepts only a claimed canonical MACP Task, resolves credentials/identity, runs the worker, runs structured gates, and persists terminal result/events through SOL-05. + 2. Worker exit 0 plus a failed gate cannot produce `completed`; worker failure cannot skip terminal ledger emission. + 3. Claude, Codex, and Pi fixture backends emit the same runtime-neutral lifecycle sequence. + 4. Every mutation returns one mandatory tri-state outcome; callers cannot compile while discarding it. + 5. Crash after worker success but before terminal commit leaves a recoverable fenced claim and no false completion. +- **dogfood seed:** stranded MACP, gate-6 inert completion, and `written-unverified` being treated as success. +- **est. tokens:** 16K +- **suggested runtime tier:** sonnet + +### SOL-07 — Provide one-shot flat-file import and cutover readiness audit + +- **build:** 2 +- **depends_on:** SOL-05 +- **acceptance criteria (diff-blind testable):** + 1. A dry-run parses existing mission/TASKS artifacts, reports unsupported/ambiguous rows, and performs zero writes. + 2. Apply is idempotent and records source digests; repeated apply creates no duplicates. + 3. Unknown status, dangling dependency, duplicate task ID, and malformed table block import with row-level diagnostics. + 4. Readiness reports “cutover-ready” only when imported PG projections exactly match source counts/dependencies/statuses. + 5. This command is migration-only; it exposes no dual-write or ongoing sync mode. +- **dogfood seed:** current remediation board/TASKS state needs a clean DB landing without silently losing tasks. +- **est. tokens:** 8K +- **suggested runtime tier:** codex + +### SOL-08 — Hard-cut `packages/coord` to PG and dogfood one live task + +- **build:** 1 +- **depends_on:** SOL-06, SOL-07 +- **acceptance criteria (diff-blind testable):** + 1. `mosaic coord run/status/continue` reads and mutates PG only; absent/unmigrated DB state fails with the SOL-07 repair path. + 2. No fallback reads/writes `docs/TASKS.md`, mission JSON, task JSON, state JSON, results JSON, or events NDJSON. + 3. A live canary task assigned to a fleet seat travels PG claim → TaskExecutor → worker → gate → terminal PG result/event and closes only after gate success. + 4. Killing the coordinator after claim and restarting it neither duplicates execution nor allows the stale lease to close the task. + 5. Evidence query shows actor seat, mission/task, target metadata, gate results, and tri-state outcome. +- **dogfood seed:** this remediation mission itself; reproduce a gate-6-style non-null task and identity-bound write. +- **est. tokens:** 16K +- **suggested runtime tier:** sonnet + +> **G1 FIRST-DOGFOOD:** stop and validate here before broadening. If SOL-08 cannot carry a real task, do not build Redis, rotation, comms, or UI. + +### SOL-09 — Route Forge and OpenClaw/MACP producers through TaskExecutor + +- **build:** 1 +- **depends_on:** SOL-08 +- **acceptance criteria (diff-blind testable):** + 1. Forge and the OpenClaw MACP runtime submit the canonical Task type to SOL-06; neither executes a worker or gate itself. + 2. Their success callbacks are derived from canonical terminal results, not local/stub completion. + 3. A failed canonical gate is observed identically from Coord, Forge, and OpenClaw fixtures. + 4. Repository search plus an executable import boundary test finds no production-local redefinition of Task/TaskResult/GateResult on these paths. +- **dogfood seed:** Forge’s immediate empty-gate completion and the plugin’s redefined MACP-shaped result. +- **est. tokens:** 10K +- **suggested runtime tier:** codex + +### SOL-10 — Retire flat-file orchestration and the disabled duplicate rail + +- **build:** 1 +- **depends_on:** SOL-09 +- **acceptance criteria (diff-blind testable):** + 1. The Python controller execution/gate/event path, `tasks_md_sync`, plugin-local protocol types, orphaned context loader, and production flat-file orchestration writers/readers are absent from shipped assets. + 2. Framework guides/templates/startup context point to DB mission commands, not `docs/TASKS.md` as orchestration SoR. + 3. A regression scan fails CI if production code reintroduces `events.ndjson`, `tasks.json`, `mission.json`, or `docs/TASKS.md` orchestration mutation. + 4. jarvis-brain PDA flat files and unrelated project docs remain untouched. + 5. Upgrade removes/quarantines obsolete generated rail files without deleting user source/docs. +- **dogfood seed:** three parallel islands and stale `.mosaic/orchestrator/mission.json` 0/0 residue. +- **est. tokens:** 14K +- **suggested runtime tier:** sonnet + +### SOL-11 — Add Redis hot dispatch as a derived outbox consumer + +- **build:** 2 +- **depends_on:** SOL-08 +- **acceptance criteria (diff-blind testable):** + 1. Task creation commits mission/task/outbox in PG before any Redis enqueue. + 2. Induced Redis failure leaves the task durable and pending; a sweeper later enqueues it exactly once logically. + 3. Deleting the Redis queue and rebuilding from PG restores all non-terminal dispatches without reviving terminal tasks. + 4. Duplicate delivery is neutralized by PG claim fencing/idempotency. + 5. Existing `packages/queue` adapter/config is reused; no second broker API is introduced. +- **dogfood seed:** inert/unknown queue transport and broker outage during task dispatch. +- **est. tokens:** 12K +- **suggested runtime tier:** codex + +### SOL-12 — Lock Builds 1+2 with black-box failure cases + +- **build:** 2 +- **depends_on:** SOL-02, SOL-10, SOL-11 +- **acceptance criteria (diff-blind testable):** + 1. A black-box suite proves: unknown queue status blocks; malformed task blocks; gate failure blocks completion; dropped identity blocks mutation; HMAC corruption forces refresh; Redis loss recovers from PG; stale lease cannot close. + 2. The suite invokes shipped CLI/service boundaries, not internal mocks. + 3. Every asserted failure checks process/result status and durable terminal/non-terminal state. + 4. The same canary task succeeds under Claude, Codex, and Pi adapters or a documented unavailable-runtime fixture fails explicitly. +- **dogfood seed:** gate-6/#1019, identity drift, and built-but-unwired MACP. +- **est. tokens:** 8K +- **suggested runtime tier:** sonnet + +### SOL-13 — Bind session authority to contract hash and generation + +- **build:** 3 +- **depends_on:** SOL-12 +- **acceptance criteria (diff-blind testable):** + 1. Launch computes a stable hash over the effective Constitution/AGENTS/runtime/skills set and stores it with session generation. + 2. Policy change or compaction detection marks the generation stale before any subsequent mutation. + 3. A stale/mismatched generation can read diagnostics but cannot claim, write task state, acknowledge comms, merge, or close. + 4. Re-attestation creates a new generation; old credentials/leases remain fenced. + 5. Hash input order/path normalization is deterministic across two clean launches. +- **dogfood seed:** compacted orchestrator losing directives and D-4 ignoring an in-message reset. +- **est. tokens:** 12K +- **suggested runtime tier:** sonnet + +### SOL-14 — Persist compact typed rotation checkpoints using Coord primitives + +- **build:** 3 +- **depends_on:** SOL-13 +- **acceptance criteria (diff-blind testable):** + 1. Checkpoint contains mission/task, completed/blocked state, next three actions, constraints, claims, contract hash/generation, and cursors—never transcript text. + 2. Checkpoint is HMAC-verified before rehydration; corrupt/expired/missing required claims refuse resume and request deterministic refresh. + 3. Writing checkpoint and rotation-intent event is atomic in PG. + 4. Existing Coord continuation capsule semantics are reused; no competing handoff schema/file is created. +- **dogfood seed:** manual MOS-ORCHESTRATION-BOARD checkpoint and incomplete-rehydration risk. +- **est. tokens:** 12K +- **suggested runtime tier:** codex + +### SOL-15 — Finish the deterministic coordinator rotation daemon + +- **build:** 3 +- **depends_on:** SOL-14 +- **acceptance criteria (diff-blind testable):** + 1. Configured token threshold triggers checkpoint → revoke old authority → terminate → launch fresh → verify rehydration in that order. + 2. Compaction-detected is a backstop that forces the same rotation path; it never requests recursive compaction. + 3. A launch failure leaves the mission recoverable and visibly paused, not assigned to two active generations. + 4. Ephemeral seats die/respawn without mission checkpoint; persistent/orchestrator seats rotate. + 5. The implementation extends `packages/coord`; untracked `apps/coordinator` residue is not revived. +- **dogfood seed:** planner-sol dirty-context dispatch and the old coordinator’s log-only `_check_context()` behavior. +- **est. tokens:** 16K +- **suggested runtime tier:** sonnet + +### SOL-16 — Add broker-independent recovery and remove silent MOSAIC BYPASS + +- **build:** 3 +- **depends_on:** SOL-15 +- **acceptance criteria (diff-blind testable):** + 1. With Redis/broker unavailable, a diagnostic/bootstrap command can inspect PG mission state, repair broker configuration, and resume without traversing the broker gate. + 2. Normal recovery remains broker-gated and is labeled as such. + 3. Break-glass requires explicit scope and expiry, emits a durable event, displays a loud banner, and auto-expires; permanent/silent bypass text or behavior is absent. + 4. The Pi-brick fixture recovers the broker, then returns to normal gated operation without editing source/config by hand. + 5. Orchestrator guidance removes “/compact and continue” only after the rotation command is available; ephemeral guidance remains explicit. +- **dogfood seed:** Pi brick and silent `MOSAIC BYPASS 2026-07-22`. +- **est. tokens:** 12K +- **suggested runtime tier:** sonnet + +### SOL-17 — Converge each host on one roster-owned lifecycle domain + +- **build:** 5 (hygiene; hard prerequisite for addressed comms) +- **depends_on:** SOL-16 +- **acceptance criteria (diff-blind testable):** + 1. Reconcile establishes exactly one roster-declared tmux socket/lifecycle domain per host. + 2. Unknown sessions are reported and quarantined; they are never killed without positive unmanaged classification. + 3. Max-age/max-context stale sessions invoke SOL-15 rotation for persistent seats or reap for ephemerals. + 4. Seat identity survives respawn and equals the roster/MOSAIC_AGENT_NAME binding. + 5. A fixture matching the current four unmanaged remediation seats converges them or produces explicit quarantine actions. +- **dogfood seed:** scout-bounce and BOARD D-3 seats split between default and `mosaic-fleet` sockets. +- **est. tokens:** 14K +- **suggested runtime tier:** codex + +### SOL-18 — Publish authenticated `comms/v1` envelope and compatibility rules + +- **build:** 4 +- **depends_on:** SOL-13, SOL-17 +- **acceptance criteria (diff-blind testable):** + 1. Envelope validates protocol version, message/idempotency ID, sender/recipient seat identity, class, ordering/coalesce key, creation/expiry, correlation, payload digest, and authentication. + 2. Current and immediately previous supported protocol versions are accepted; unsupported versions are rejected loudly with supported range. + 3. Framework/runtime version is diagnostic metadata and never the compatibility key. + 4. Forged sender, changed recipient/payload, expired envelope, and replay with conflicting content fail closed. +- **dogfood seed:** wrong-socket bare tmux message with no authoritative sender/recipient receipt. +- **est. tokens:** 8K +- **suggested runtime tier:** codex + +### SOL-19 — Build the logical PG-first comms service with tmux adapter + +- **build:** 4 +- **depends_on:** SOL-18 +- **acceptance criteria (diff-blind testable):** + 1. Sending commits envelope/payload and PENDING state in PG before adapter delivery. + 2. State machine enforces PENDING → RECEIVED → CONSUMED or DEAD-LETTER; illegal regressions are rejected. + 3. Recipient-filtered claims and append/coalesce policy are deterministic by message class. + 4. tmux is a dumb adapter: delivery failure changes no PG authority state and is retryable. + 5. Existing Tess durable repository/state-machine patterns are extended or generalized; no new deployable microservice or second inbox framework appears. +- **dogfood seed:** MACP scout bounce that was discovered only by manual liveness check. +- **est. tokens:** 16K +- **suggested runtime tier:** sonnet + +### SOL-20 — Make `agent-send` use the sole path and prove stale-message handling + +- **build:** 4 +- **depends_on:** SOL-19 +- **acceptance criteria (diff-blind testable):** + 1. Normal `agent-send` creates a comms/v1 record and observes RECEIVED/CONSUMED; it cannot directly invoke tmux. + 2. A wrong/missing socket leaves PENDING with retry diagnostics, then reaches RECEIVED after roster repair without resending. + 3. A stale coalescible message arriving after a newer terminal message is marked superseded/consumed and is not surfaced as live work. + 4. Duplicate identical send is idempotent; same ID with changed content is rejected. + 5. Inbox receipt/terminal disposition emits canonical MACP lifecycle events. +- **dogfood seed:** scout-bounce and #1018 stale-consumed message arriving after merge. +- **est. tokens:** 12K +- **suggested runtime tier:** codex + +### SOL-21 — Add Redis Streams hot delivery and PG reconciliation + +- **build:** 4 +- **depends_on:** SOL-11, SOL-20 +- **acceptance criteria (diff-blind testable):** + 1. PG commit precedes XADD; induced XADD failure is repaired by sweeper. + 2. Consumer uses a PEL; ack sequence is PG CONSUMED commit before XACK. + 3. Redis flush/restart rebuilds pending delivery from PG without duplicating consumed messages. + 4. Pending, abandoned, and dead-letter transitions are observable with bounded retry/backoff. + 5. Existing Redis/queue connection/configuration is reused. +- **dogfood seed:** delivery bounce plus broker loss between durable write and hot enqueue. +- **est. tokens:** 12K +- **suggested runtime tier:** codex + +### SOL-22 — Prove adapter pluggability with the existing Matrix connector + +- **build:** 4 +- **depends_on:** SOL-21 +- **acceptance criteria (diff-blind testable):** + 1. Existing Matrix connector consumes/produces comms/v1 through SOL-19 without owning authority state. + 2. The same envelope can fail tmux and later deliver through Matrix while producing one logical message lifecycle. + 3. Matrix retry/reconnect cannot regress PG state or duplicate CONSUMED work. + 4. Removing Matrix availability leaves PG/Redis/tmux behavior intact. +- **dogfood seed:** cross-socket scout notification bounce; alternate reach must not become alternate authority. +- **est. tokens:** 8K +- **suggested runtime tier:** codex + +### SOL-23 — Constrain auto-sync and agent writes by allowlist and lease + +- **build:** 5 +- **depends_on:** SOL-10 +- **acceptance criteria (diff-blind testable):** + 1. Auto-sync stages only an explicit allowlist; an unknown modified/untracked docs/source file remains unstaged and is reported. + 2. Agent source/docs writes require the correct worktree/lease; two seats cannot acquire the same mutable target concurrently. + 3. Generated files are positively identified, not inferred by denylist. + 4. The measured annotation/index mid-write fixture cannot be swept into an unrelated commit. + 5. DB orchestration state is absent from repository staging concerns. +- **dogfood seed:** auto-sync sweep commit `517bd5c26` capturing agent-authored docs mid-write. +- **est. tokens:** 8K +- **suggested runtime tier:** codex + +### SOL-24 — Build the real-artifact lifecycle conformance harness + +- **build:** 5 +- **depends_on:** SOL-16, SOL-17, SOL-22, SOL-23 +- **acceptance criteria (diff-blind testable):** + 1. Harness launches shipped CLI/runtime artifacts and fault-injects compaction, broker outage, delivery bounce, identity drop, stale contract hash, queue unknown/malformed, Redis loss, and auto-sync collision. + 2. One deterministic test executes 100 sequential rotations with no lost/duplicated task, claim, receipt, or terminal disposition. + 3. Tests assert DB state/event order and process exits, not log substrings alone. + 4. Harness runs against isolated PG/Redis namespaces and cleans only resources it created. + 5. Every banked dogfood seed has a named case and evidence output suitable for CI/release attachment. +- **dogfood seed:** the complete failure bank: Pi brick, scout-bounce, gate-6/#1019, identity drift, auto-sync, #1018 stale-consumed, D-4 dirty context. +- **est. tokens:** 18K +- **suggested runtime tier:** sonnet + +### SOL-25 — Complete operator cutover docs and activation proof + +- **build:** 5 +- **depends_on:** SOL-01, SOL-02, SOL-10, SOL-16, SOL-22, SOL-24 +- **acceptance criteria (diff-blind testable):** + 1. Operator docs give exact DB import/cutover, rollback-before-cutover, recovery, break-glass expiry, rotation, comms, quarantine, and conformance commands. + 2. Link/command checks find no orchestrator instruction to mutate flat-file mission/tasks, use silent bypass, direct-tmux normal comms, or “compact and continue” a persistent seat. + 3. A clean non-root install activates one coherent version and runs the conformance smoke subset. + 4. Release evidence maps all 15 decisions and every live seed to a passing check or an explicit deferred item below. +- **dogfood seed:** activation skew plus the tendency to leave built fixes unwired or undocumented. +- **est. tokens:** 6K +- **suggested runtime tier:** codex + +## Explicit DEFER list (10) + +These are not rejected; they are **past first dogfood** and should not delay G1/G2. Each is gold-plating unless a live failure makes it necessary. + +1. **DEFER — Mission dashboard/TUI views.** CLI/DB queries are enough to operate and prove the spine. +2. **DEFER — PRD-to-board automatic decomposition.** This is LLM/judgment-heavy and unrelated to enforcing already-decided tasks. +3. **DEFER — General heuristic churn scoring.** Implement token threshold + compaction sensor first; repeated-tool-loop inference can follow measured need. +4. **DEFER — Discord comms adapter.** Existing plugin reach remains; migrate only after tmux+Matrix prove the service contract. +5. **DEFER — Slack comms adapter.** No current dogfood dependency. +6. **DEFER — Telegram comms adapter.** No current dogfood dependency. +7. **DEFER — Public MCP comms surface.** `agent-send` and service API are sufficient for the mission proof. +8. **DEFER — Protocol-v2 features/general negotiation framework.** Ship v1 with a bounded current/previous acceptance window; do not predict v2. +9. **DEFER — Multi-region/HA PG or Redis.** Existing in-stack PG+Redis and rebuildability satisfy current failure classes. +10. **DEFER — Event analytics/search UI and long-term warehouse.** Indexed PG evidence plus CLI queries is enough for audit/conformance. + +## Suspect abstractions register + +| proposed thing | verdict | +| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Canonical Node `TaskExecutor` | **JUSTIFIED:** explicitly required single choke point; wraps existing MACP functions rather than replacing them. | +| PG repository methods | **JUSTIFIED but narrow:** ordinary adapters around existing Drizzle/DB patterns, not a new “state platform.” | +| Rotation daemon | **JUSTIFIED:** finishes `packages/coord`; do not revive `apps/coordinator` or create another service. | +| Comms service | **JUSTIFIED only as a logical in-process boundary:** reuse Tess durable inbox/outbox and existing connectors; no new deployable microservice this cycle. | +| Universal queue/broker abstraction | **SUSPECT / DO NOT BUILD:** reuse `packages/queue`, PG outbox, and Redis Streams/BullMQ configuration already present. | +| Universal envelope/state framework | **SUSPECT / DO NOT BUILD:** MACP Task/Claim/Event and comms/v1 have different bounded purposes. | +| Generic compatibility-negotiation engine | **SUSPECT / DEFER:** a small supported-version check meets v1 needs. | + +## Dissent (7) + +1. **Do not wire new production behavior into `mosaic_orchestrator.py::run_single_task`.** The scout correctly identified the duplicated block, but BOARD’s later ruling says the disabled Python rail is residue and must be retired. Building a Node bridge only to delete it is throwaway. Put the Node TaskExecutor in `@mosaicstack/macp`, route the live Coord path to it at SOL-08, migrate remaining producers at SOL-09, then delete the Python block at SOL-10. +2. **Redis is not on the first-dogfood critical path.** PG claim/polling is sufficient for one real task and exposes correctness earlier. Add Redis only after G1; otherwise queue debugging obscures whether the choke point works. +3. **“One choke-point service” does not justify a new deployable service.** An exported executor plus Coord daemon is enough. A new Nest app, RPC protocol, deployment, auth layer, and health plane would be greenfield. +4. **The Mission Control PRD’s file-first and board-regeneration assumptions are superseded.** Keep its mission/rotation semantics, but obey the accepted hard DB cutover; do not implement its file-first milestones or PRD-to-board generator now. +5. **Do not gate every interactive runtime launch as if it were a mission task.** Enforce every tracked task/data mutation at the executor/DB authority boundary. Ephemeral interactive shells may launch, but receive no task mutation authority unless attached to a valid claim. +6. **Do not implement broad “churn intelligence.”** Token threshold and compaction-detected are deterministic sensors. Repeated-loop semantic detection is expensive, noisy, and premature until telemetry demonstrates a gap. +7. **The 100-rotation test is a final conformance bar, not an early unit-test tax.** First prove one rotation, then fault cases, then 100 repetitions in SOL-24. Requiring 100 before G3 would delay feedback without changing the design. + +## Orchestrator reconciliation notes + +- Pre-register each task’s acceptance checks from this document before showing implementation diffs to its reviewer. Author and reviewer remain different seats. +- SOL-07 permits a one-time import, **not** an interim store: no shadow writes, dual reads, or sync daemon. +- G1 is the budget escape hatch. If the 72K hard-path slice does not work, stop and remediate instead of spending the remaining ~222K. +- Docs belong in each behavior-changing PR where required; SOL-25 is cross-link/cutover validation, not permission to postpone essential docs. diff --git a/docs/remediation/TASKS.md b/docs/remediation/TASKS.md new file mode 100644 index 00000000..d2c07a50 --- /dev/null +++ b/docs/remediation/TASKS.md @@ -0,0 +1,258 @@ +# Remediation Backlog — Reconciled Execution Plan + +**Owner:** `mos-remediation` (sole writer). Workers read; they never modify this file. +**Sources:** [`DECOMP-OPUS.md`](./DECOMP-OPUS.md) (robustness, 38 tasks / 8 dissents) and +[`DECOMP-SOL.md`](./DECOMP-SOL.md) (pragmatic, 25 tasks / 10 defers / 7 dissents), produced +**independently** — neither planner read the other. Charter: [`MISSION.md`](./MISSION.md). +**Status:** PLANNING — this backlog is proposed, not yet dispatched. Three items need a Mos/Jason +ruling before the affected tasks dispatch (§5). + +> **Provenance of the inputs (both clean).** `planner-opus` ran in a fresh session throughout. +> `planner-sol` initially began work at 64.3% dirty context despite a brief instructing it to reset; +> that run was **interrupted and discarded before it produced any output**, the seat was reset +> out-of-band to 0.0%, and the brief was re-dispatched. `DECOMP-SOL.md` is the product of the clean +> run only (it peaked at ~26% context). Both decompositions are therefore clean-context artifacts and +> are weighted equally here. The discarded dirty run is banked as dogfood seed D-4 and as task RM-58 — +> the failure it demonstrates is that _asking_ an agent to reset is not enforcement. + +--- + +## 1. What the two planners agreed on without collusion + +Independent convergence is the strongest signal available here, because neither planner could see the +other's file. Where both arrived at the same conclusion from opposite biases, I treat it as settled. + +| # | Convergent finding | OPUS | SOL | +| --- | --------------------------------------------------------------------------------------------------------------------- | ------------------- | -------------- | +| C1 | **The charter's wire-in point is wrong.** Do NOT wire the choke point into `mosaic_orchestrator.py::run_single_task`. | D2 (headline) | Dissent 1 | +| C2 | P0 hygiene/gate work must precede the spine, not follow it. | D1, phase P0 | G0, SOL-01/02 | +| C3 | No new deployable microservice; the executor is a library + the coord daemon. | implicit throughout | Dissent 3 | +| C4 | Comms adapters beyond tmux are out of scope for this mission. | D7 | DEFER 4/5/6 | +| C5 | The "100 rotations lossless" bar is a late conformance gate, not an early tax. | D4 | Dissent 7 | +| C6 | Redis is a derived hot path, never an authority; PG commits first. | R-013, R-054 | SOL-11, SOL-21 | +| C7 | Reuse `packages/coord`; do NOT revive the untracked `apps/coordinator` residue. | R-042 | SOL-15 AC5 | + +**C1 is the single most consequential output of this exercise.** The charter (`MISSION.md`) and my +kickoff instruction both name `mosaic_orchestrator.py::run_single_task:126-276` as the integration +point. Both planners independently rejected it on the same evidence: that controller is +`"enabled": false` (`.mosaic/orchestrator/config.json:2`) and references a dispatcher path +(`tools/macp/dispatcher/pi_runner.ts`) that does not exist in this checkout. Wiring the new choke +point into a disabled rail produces **a stranded executor — the identical built-but-unwired disease, +one layer up, that would look "done" in a PR.** The live paths are +`packages/mosaic/src/commands/launch.ts` and `packages/coord/src/runner.ts`. +This contradicts the charter and is escalated as **DECISION-1** (§5). + +--- + +## 1a. ★ KEYSTONE DOGFOOD CASE — an inert gate that erased its own evidence + +**A merged commit shipped past `pnpm format:check` — and then the evidence quietly erased itself.** + +Verified chain (blob-level, under the repo's own prettier config, at the file's real path): + +| commit | state of `packages/mosaic/framework/tools/orchestrator/README.md` | +| ------------------------------------------------------------------- | ----------------------------------------------------------------------------- | +| `b79336a8` — **merged PR #868** | blob `3ee7f104` — **FAILS** `pnpm format:check` | +| `48fd1df2` — merged PR #872 (unrelated: ci-queue-wait 404 handling) | blob `3d3bb132` — passes; incidentally reformatted by that PR's `lint-staged` | +| current `origin/main` (`06e0d403`) | passes — **the gate now looks green** | + +So: PR #868 merged a file that fails a required gate ⇒ **the CI format gate did not block it.** The +gate was inert for that merge. Then an unrelated later PR's pre-commit hook reformatted the file as a +side effect, so `main` went green again **without anyone ever learning the gate had failed to fire.** + +> **Correction on record:** my first report to Mos said "format:check is RED on main _now_." That was +> true of the `main` my checkout was pinned to (`b79336a8`) and is **no longer true of current `main`**, +> which advanced mid-session. The inert-gate finding itself is unchanged and verified; only its +> present-tense framing was wrong. The hygiene PR therefore carries the `.prettierignore` fix only — +> the README needs no fix today. + +**The self-erasure is the important part.** An inert gate that is masked by unrelated downstream +commits produces no lasting artifact, which is precisely why this class survives for months. Detection +cannot rely on "is `main` currently red" — it must be per-merge-commit. + +This matters more than the one-line fix: + +- It is the **P-QUEUE-001 / P-CONFORMANCE-001 class** ("gate-6 was inert fleet-wide"), reproduced in + the repository this mission is remediating, discovered incidentally. +- It independently **validates OPUS premise A1** ("every gate is inert until proven otherwise") with + live evidence rather than argument — which is why RM-02 is adopted as the keystone (§2, X2). +- The file fix rides in its own hygiene PR. **The inert gate itself is NOT quiet-patched.** Per Mos: + it stays a first-class backlog item, because patching the symptom would destroy the signal. + +**Binding requirement on RM-02 and RM-55:** the gate registry and the conformance harness must assert +**"every merged commit passed every required gate"** — evaluated **per merge commit, against that +commit's own tree**, not against current `main`. As the table above proves, a "is main green today" +check would have reported all-clear. A merged-commit-that-fails-a-required-gate is the exact detection +signal, and it must be a registered must-fail case. A gate that cannot prove it blocked something has +not been shown to work. + +--- + +## 2. Where they genuinely disagree (not averaged — adjudicated) + +| # | Axis | OPUS | SOL | My ruling | +| --- | ------------------------------------------- | ---------------------------------------------------------- | --------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| X1 | **Total cost** | 38 tasks, ~5.3M tok | 25 tasks, ~294K tok | **~18× apart.** Not reconcilable by splitting. They measure different things: SOL explicitly excludes orchestration/review/iteration overhead and assumes one remediation pass; OPUS prices the full loop. Adopt **SOL's scope** with **OPUS's rigor**, and treat SOL's G1 as a hard budget checkpoint (§4). Re-estimate empirically after the first three merged PRs rather than trusting either number. | +| X2 | **Gate registry (OPUS R-002)** | Keystone; blocks all P2 | Absent; only a queue-guard fix | **ADOPT OPUS.** Empirically validated in this very session: I found `pnpm format:check` red on `main` via merged PR #868 — a required gate that did not block. OPUS's premise A1 ("every gate is inert until proven otherwise") is not theoretical; it reproduced today, unprompted. Scope it tighter than 120K. | +| X3 | **Drizzle PG first-install defect (R-010)** | Hidden blocker; everything downstream depends on it | Not mentioned | **ADOPT OPUS.** `packages/db/src/migrate.ts:30-38` carries a TODO admitting postgres-tier first-install fails today. The spine has only ever been proven on PGlite. Every later migration silently depends on this. SOL missed it. | +| X4 | **Rollback artifact for the hard cutover** | D3: hard cutover needs a rehearsed rollback snapshot | SOL-07: import-only, explicitly no dual-write | Both obey "no flat-file interim." OPUS wants a one-directional snapshot nothing reads as authority. I read that as compatible with the directive, but it is Jason's call → **DECISION-2** (§5). | +| X5 | **Where the queue guard sits** | P0, independent of spine | SOL-02, also early | Agree it is P0. But ownership collides with **parked PR #1023** → **DECISION-3** (§5). | +| X6 | **Report-only rollout** | D5: only with a hard expiry, else withdraw | not raised | **ADOPT OPUS.** A report-only gate is by definition inert; expiry is the mechanism that stops it becoming the new fail-open. | +| X7 | **Availability trade (FC-7/FC-11)** | D8: "no DB ⇒ fleet stops" must be pre-committed in writing | not raised | Genuine availability regression, correctly identified. Needs Jason → folded into **DECISION-2**. | + +--- + +## 3. Reconciled DAG + +Phases run in order; `⛔` marks a hard barrier. `src` shows lineage (`O`=opus, `S`=sol, `O+S`=both). +Estimates are given as a **range** (SOL low / OPUS high) rather than a fabricated midpoint — the +spread is itself information, and X1 says we calibrate on real merged PRs. + +### P0 — Make gates provable, and stop the fleet re-bricking + +⛔ _No gate-introducing task in any later phase may merge before RM-02._ + +| id | task | src | depends_on | est (S/O) | tier | +| ----- | -------------------------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ | +| RM-01 | Reproducible non-root checkout; pre-push gate fails on **code, not env** (banks D-1/D-2/D-5) | O+S | — | 6K / 60K | codex | +| RM-02 | **Gate registry + negative-control CI check** (anti-inert-gate harness) ★keystone | O | RM-01 | — / 120K | opus | +| RM-03 | Queue-guard fail-closed rework (`unknown`/`no-status`/malformed ⇒ ≠0) | O+S | RM-02 | 8K / 100K | sonnet | +| RM-04 | Activation/version coherence; block launch on skew, fail SAFE; honest `doctor` labels | O+S | RM-01 | (in S-01) / 140K | sonnet | +| RM-05 | Break-glass replaces the three silent `MOSAIC BYPASS` fail-opens | O | RM-04, RM-02 | — / 120K | opus | + +> ⚠ **RM-05 must not merge before RM-04.** The bypasses exist because the lease-broker daemon was +> never _deployed_ on this host — removing the fail-open before deployment coherence is real +> re-creates the 2026-07-22 bricking incident. Hard edge, from OPUS. + +### P1 — Durable spine (PG) + +⛔ _No migration may merge before RM-10._ + +| id | task | src | depends_on | est (S/O) | tier | +| ----- | --------------------------------------------------------------------------------------------- | --- | ---------- | ---------- | ------ | +| RM-10 | **Fix the Drizzle postgres-tier first-install defect** ★hidden blocker | O | RM-01 | — / 90K | sonnet | +| RM-11 | Orchestration spine schema (tasks, attempts, gate_results, hash-chained ledger, typed claims) | O+S | RM-10 | 12K / 160K | opus | +| RM-12 | Spine client, fail-closed connection (no silent PGlite in prod) | O | RM-11 | — / 80K | sonnet | +| RM-13 | Atomic claims/transitions + transactional outbox + reconciliation sweeper | O+S | RM-12 | 12K / 140K | opus | + +### P2 — The single choke point + +⛔ _RM-25 (no-second-path) lands in the same milestone as RM-20, or the choke point is optional._ + +| id | task | src | depends_on | est (S/O) | tier | +| ----- | ---------------------------------------------------------------------------------------------- | --- | ------------------- | ---------------- | ------ | +| RM-20 | Canonical MACP contract completion (Task/Result/Event/Claim/tri-state outcome) | S | — | 8K / (in R-020) | codex | +| RM-21 | **Production `TaskExecutor`** backed by `@mosaicstack/macp` ★keystone | O+S | RM-12, RM-02, RM-20 | 16K / 220K | opus | +| RM-22 | Gate-runner hardening: `fail_on`, timeouts, **empty gate set = failure** | O | RM-21 | — / 120K | sonnet | +| RM-23 | Hash-chained MACPEvent ledger in PG + lifecycle EventType extension | O+S | RM-21, RM-11 | — / 160K | opus | +| RM-24 | Seat identity from `MOSAIC_AGENT_NAME` + **mandatory** tri-state write outcomes | O+S | RM-21 | (in S-03) / 150K | opus | +| RM-25 | **No-second-path gate:** terminal status writable only by the executor | O | RM-21, RM-23 | — / 140K | opus | +| RM-26 | `packages/coord` submits through the executor (retire direct spawn) | O+S | RM-21 | 16K / 140K | sonnet | +| RM-27 | `mosaic yolo/claude/codex/pi` launch path records typed Task + events | O | RM-21, RM-23 | — / 160K | sonnet | +| RM-28 | Delete the Forge stub executor (empty-gate-list "success"); Forge submits through the real one | O+S | RM-21 | 10K / 90K | codex | +| RM-29 | One-shot flat-file import + cutover readiness audit (dry-run, idempotent, no dual-write) | S | RM-13 | 8K / (in R-062) | codex | + +> **★ G1 — FIRST DOGFOOD. Stop here and prove it.** One live fleet task travels +> PG claim → TaskExecutor → worker → gates → terminal PG result/event, with **no** flat-file state. +> Adopted from SOL wholesale. If G1 cannot carry a real task, **do not build Redis, rotation, comms, +> or conformance** — remediate instead. This is the budget escape hatch (§4). + +### P3 — Rotation lifecycle (finish the Mission Control Plane) + +| id | task | src | depends_on | est (S/O) | tier | +| ----- | -------------------------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ | +| RM-30 | Typed state claims (source/confidence/TTL) with HMAC integrity, fail-closed | O+S | RM-11, RM-21 | (in S-03) / 170K | opus | +| RM-31 | Contract-hash binding; stale generation loses mutation authority **mechanically** | O+S | RM-21, RM-30 | 12K / 180K | opus | +| RM-32 | Durable compaction/token sensor (per-runtime thresholds, PreCompact event) | O | RM-23, RM-31 | — / 130K | sonnet | +| RM-33 | Typed checkpoint writer (structured claims, never transcript) + digest | O+S | RM-30, RM-32 | 12K / 150K | opus | +| RM-34 | **Rotation daemon:** watch → checkpoint → revoke → kill → relaunch → rehydrate | O+S | RM-33, RM-26 | 16K / 240K | opus | +| RM-35 | Rehydration attestation gate: refuse to act on an incomplete claim set | O | RM-33 | — / 130K | opus | +| RM-36 | Broker-independent recovery; remove silent bypass; honest capability labels | S | RM-34 | 12K / (in R-004) | sonnet | +| RM-37 | Delete `/compact and continue` from the persistent-seat path (**substitution**, not removal) | O+S | RM-34, RM-44 | (in S-16) / 60K | codex | + +### P4 — Comms service + +⛔ _RM-50 (one roster-owned socket per host) precedes identity-addressed delivery._ + +| id | task | src | depends_on | est (S/O) | tier | +| ----- | ---------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ | +| RM-40 | `comms/v1` envelope + protocol-version negotiation, LOUD reject | O+S | RM-11, RM-31 | 8K / 140K | opus | +| RM-41 | Comms service: PG state machine PENDING→RECEIVED→CONSUMED→DEAD-LETTER | O+S | RM-40, RM-13 | 16K / 200K | opus | +| RM-42 | tmux transport as a **dumb adapter**; durable retry before cursor advance | O+S | RM-41, RM-50 | (in S-19) / 160K | sonnet | +| RM-43 | Per-class coalescing + supersede (the stale-consumed-as-live fix) | O+S | RM-41 | 12K / 130K | sonnet | +| RM-44 | Redis Streams hot delivery + provenance guard (**Redis is never authority**) | O+S | RM-41, RM-13 | 12K / 170K | opus | +| RM-45 | Retire direct tmux sends; only the service may write a pane | O+S | RM-42, RM-43 | (in S-20) / 100K | codex | + +### P5 — Retirements, hygiene, conformance + +| id | task | src | depends_on | est (S/O) | tier | +| ----- | ---------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | -------------------------- | ---------------- | ------ | +| RM-50 | One roster-owned socket/host; quarantine unmanaged; stale-session GC | O+S | RM-04 | 14K / 150K | sonnet | +| RM-51 | Auto-sync **allowlist** (never auto-stage unknown paths) + worktree/lease isolation | O+S | RM-02 | 8K / 110K | sonnet | +| RM-52 | Retire the Python controller + duplicate MACP islands (3 → 1) | O+S | RM-26, RM-27, RM-25, RM-28 | 14K / 110K | codex | +| RM-53 | Flat-file orchestration → DB hard cutover, with rehearsed rollback artifact | O+S | RM-27, RM-30, RM-34, RM-29 | (in S-10) / 200K | opus | +| RM-54 | Fleet-wide inert-gate audit against the RM-02 registry | O | RM-02 | — / 120K | sonnet | +| RM-55 | **Conformance harness:** fault-inject the live failure classes on real artifacts | O+S | RM-35, RM-41, RM-53 | 18K / 260K | opus | +| RM-56 | Retirement proof: CI asserts all three retirements are complete **and stay complete** | O | RM-52, RM-45, RM-53 | — / 90K | codex | +| RM-57 | Operator cutover docs + activation proof; map all 15 decisions to evidence | S | RM-04, RM-36, RM-45, RM-55 | 6K / — | codex | +| RM-58 | **Mechanical pre-dispatch context reset** — the orchestrator resets a seat out-of-band and verifies it, rather than asking the agent to reset itself | mos-remediation (D-4) | RM-31, RM-50 | 8K | sonnet | + +**Critical path:** `RM-01 → RM-02 → RM-10 → RM-11 → RM-12 → RM-21 → RM-23 → RM-31 → RM-33 → RM-34 → RM-53 → RM-55`. + +--- + +## 4. Execution discipline + +- **Every row is one PR.** Author ≠ reviewer; `rev-974` is the mosaicstack reviewer identity. +- **Pre-registered, diff-blind acceptance checks are committed BEFORE the reviewer reads the diff.** + Both decomps wrote their ACs in runnable `⇒0` / `⇒≠0` form specifically to make this possible. +- **Every gate-introducing task carries at least one registered must-fail negative control.** This is + RM-02's whole purpose; a gate with no proven failure path manufactures evidence. +- **Cost tiers:** codex for mechanical/unambiguous, sonnet for normal feature work, opus reserved for + security/integrity/cross-cutting-invariant tasks. SOL priced 0 opus tokens; OPUS priced 14 opus + tasks. I am keeping opus only where the failure is _integrity_, not merely complexity. +- **G1 is the budget checkpoint.** If the first-dogfood slice overruns SOL's estimate by >3×, stop and + re-plan rather than spending the remainder. X1 says neither estimate is trustworthy until calibrated. +- **Defer list adopted from SOL** (10 items): mission dashboard/TUI, PRD-to-board auto-decomposition, + heuristic churn scoring, Discord/Slack/Telegram adapters, public MCP comms surface, protocol-v2 + negotiation, multi-region PG/Redis, event analytics UI. + +--- + +## 5. Blocking decisions (need Mos, or Jason via Mos) + +These are escalated because they change the charter, the accepted directive, or another lane's +ownership — none is a question I should answer unilaterally. + +**DECISION-1 — the wire-in point (changes `MISSION.md`).** +Both planners independently reject wiring the choke point into +`mosaic_orchestrator.py::run_single_task`, because that controller is disabled and references a +non-existent dispatcher. They propose wiring `packages/coord/src/runner.ts` + +`packages/mosaic/src/commands/launch.ts` and **deleting** the Python rail instead. This makes RM-52 a +deletion task rather than an integration task, and moves Build 1's acceptance onto a live +`mosaic yolo` invocation. _My recommendation: accept — wiring a disabled rail reproduces the exact +disease this mission exists to cure._ + +**DECISION-2 — rollback artifact + the availability trade (needs Jason).** +(a) Does "hard cutover, no flat-file interim" permit a **rehearsed, one-directional, read-as-authority- +by-nobody** rollback snapshot (OPUS D3)? (b) Do we pre-commit in writing that "no DB ⇒ the fleet +stops" and "unpersistable event ⇒ the operation fails" (OPUS D8)? That is a real availability +regression versus today's limping flat-file fleet — correct for a system whose defining failure is +_silent continuation_, but it should be a decision, not a 2am discovery. +_My recommendation: yes to both; a rollback snapshot nothing reads is not an interim tracking system._ + +**DECISION-3 — RM-03 ownership vs. parked PR #1023 (needs Mos).** +P-QUEUE-001 is absorbed into this mission, but PR #1023 is explicitly parked under Mos. Two lanes can +legitimately claim it. A third recursion on the gate-6 defect — performed by the remediation itself — +would be the postmortem's own anti-pattern. _Not dispatching RM-03 until Mos rules._ + +--- + +## 6. Status + +| phase | state | +| ------------------ | ------------------------------------------------------------------------------------ | +| Decomposition | DONE — both planners delivered independently | +| Reconciliation | DONE — this document | +| Blocking decisions | **OPEN — 3 escalated to Mos (§5)** | +| Dispatch | NOT STARTED — RM-01 is dispatchable now; it depends on nothing and blocks everything |