mosaicstack/stack
1
0
Fork 1
Code Issues 139 Pull Requests 5 Actions Packages Projects Releases 15 Wiki Activity

docs(remediation): apply Mos rulings on D-1/D-2/D-3; bank D-7 tmpfs ENOSPC

Browse Source 
D-1 RULED — corrected wire-in target is a new production Node TaskExecutor on the
LIVE dispatch path (packages/mosaic launch + packages/coord) that Coord/Forge/live
dispatch submit through. The Python rail is deleted, not ported. RM-21/26/27/52 updated.

D-2 — Jason-pending and explicitly NON-blocking. Binding meanwhile: the DB spine must
not be a single-point hard-stop; degraded mode + rollback artifact become design
requirements on RM-12/13/23/36/53. This reverses the earlier recommendation to
pre-commit 'no DB means the fleet stops'.

D-3 — RM-03 HOLD; PR #1023 marked SUPERSEDED-PENDING-JASON; RM-02/RM-55 proceed
independently since conformance assertion is not the gate-6 fix itself.

RM-03 now carries BOTH queue-guard defects: wrong exit on unknown, and evaluating
branch=main rather than the branch being pushed.

D-7 — shared 30G /tmp hit 100% ENOSPC, degrading two seats. ~5.2G of it is session
scratch dead for 8-9 days; this session's own footprint is 88K. Same missing
capability as P-FLEET-001 stale-session GC, applied to disk. Folded into RM-50, and
RM-01 gains artifacts-off-shared-tmp acceptance criteria.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
mos-dt-0
2026-07-31 17:25:04 -05:00
committed by mosaic-coder
co-authored by Claude Opus 5
parent d4f68040ab
commit 1324385bba
1 changed files with 88 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/remediation/TASKS.md
+88 -30
View File
@@ -40,7 +40,10 @@ point. Both planners independently rejected it on the same evidence: that contro
point into a disabled rail produces **a stranded executor — the identical built-but-unwired disease,
one layer up, that would look "done" in a PR.** The live paths are
`packages/mosaic/src/commands/launch.ts` and `packages/coord/src/runner.ts`.
This contradicts the charter and is escalated as **DECISION-1** (§5).
This contradicted the charter and was escalated as DECISION-1 — **now RULED in the planners' favour by
Mos (§5)**. The corrected target is a new production Node `TaskExecutor` on the live dispatch path
(`packages/mosaic` launch + `packages/coord`) that Coord/Forge/live dispatch submit through; the
Python rail is deleted, not ported.
---
@@ -66,6 +69,53 @@ side effect, so `main` went green again **without anyone ever learning the gate
> present-tense framing was wrong. The hygiene PR therefore carries the `.prettierignore` fix only —
> the README needs no fix today.
### Third live instance, same class — the queue guard, hit by this orchestrator
Running the **mandated** pre-push guard during TASK-0:
```
$ ~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose push
[ci-queue-wait] platform=gitea purpose=push branch=main sha=06e0d403…
[ci-queue-wait] state=unknown purpose=push branch=main
$ echo $? → 0
```
**Two distinct defects in one tool**, both feeding RM-03:
1. **Wrong exit** — `state=unknown` ⇒ `exit 0`. The defect at `ci-queue-wait.sh:282-288` that OPUS
documented and that PR #1023 is parked on. A required gate returned PASS on an indeterminate result.
2. **Wrong branch** — it evaluated `branch=main`, not the branch actually being pushed. Even a
correctly-exiting guard would have been answering the wrong question.
**Standing doctrine (Mos):** until RM-03 lands, a green from this guard carries **zero information**
and must not be cited as merge evidence. Rely on reviewer clearance + real CI.
Three independent live instances in a single session — format gate, agent context reset, queue guard —
is the class confirmed, not anecdote.
### D-7 — shared-tmpfs contention → cascading ENOSPC (live incident, 2026-07-31)
The shared 30 G `/tmp` hit **100% ENOSPC** mid-session. It broke tool calls in **two different seats**
(mine and Mos's) — a single full disk degrades every agent on the host at once. Recurring: prior
incidents 2026-06-18 and 2026-07-17.
Attribution matters, because the wrong owner cleans the wrong thing. Measured:
| path | size | last modified | owner |
| ---------------------------------------------- | --------- | ---------------------------- | ------------------------------------------------- |
| `…/-src-mosaic-stack/6d2faee6…` (this session) | **88 K** | live | mos-remediation |
| `…/-src-mosaic-stack/c743185d…` | **3.6 G** | **2026-07-22** (9 days dead) | abandoned session, same project path |
| `…/claude-1001/pnpm-store` | **1.6 G** | **2026-07-23** (8 days dead) | abandoned; the live store is correctly on `$HOME` |
So ~5.2 G — the bulk of the pressure — is **dead session scratch that nothing will ever read again**.
This is not a quota problem; it is **P-FLEET-001's stale-session GC, applied to disk instead of tmux
sessions.** The same missing capability (nothing owns reaping dead ephemeral state) produces both the
orphaned-session failure and this one. Reaping dead-session scratch belongs in RM-50 alongside stale
tmux-session GC.
**Added to RM-01 as acceptance criteria:** heavy build artifacts (node_modules, package stores, build
output) must land on the main disk in the worktree, never on the shared 30 G `/tmp`.
**The self-erasure is the important part.** An inert gate that is masked by unrelated downstream
commits produces no lasting artifact, which is precisely why this class survives for months. Detection
cannot rely on "is `main` currently red" — it must be per-merge-commit.
@@ -112,13 +162,13 @@ spread is itself information, and X1 says we calibrate on real merged PRs.
⛔ _No gate-introducing task in any later phase may merge before RM-02._
| id | task | src | depends_on | est (S/O) | tier |
| ----- | -------------------------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ |
| RM-01 | Reproducible non-root checkout; pre-push gate fails on **code, not env** (banks D-1/D-2/D-5) | O+S | — | 6K / 60K | codex |
| RM-02 | **Gate registry + negative-control CI check** (anti-inert-gate harness) ★keystone | O | RM-01 | — / 120K | opus |
| RM-03 | Queue-guard fail-closed rework (`unknown`/`no-status`/malformed ⇒ ≠0) | O+S | RM-02 | 8K / 100K | sonnet |
| RM-04 | Activation/version coherence; block launch on skew, fail SAFE; honest `doctor` labels | O+S | RM-01 | (in S-01) / 140K | sonnet |
| RM-05 | Break-glass replaces the three silent `MOSAIC BYPASS` fail-opens | O | RM-04, RM-02 | — / 120K | opus |
| id | task | src | depends_on | est (S/O) | tier |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ | ---------------- | ------ |
| RM-01 | Reproducible non-root checkout; gate fails on **code, not env**; heavy artifacts OFF shared `/tmp` (banks D-1/D-2/D-5/D-7) | O+S+live | — | 6K / 60K | codex |
| RM-02 | **Gate registry + negative-control CI check** (anti-inert-gate harness) ★keystone | O | RM-01 | — / 120K | opus |
| RM-03 ⏸**HOLD** | Queue-guard: **two** defects — (a) `unknown`/`no-status`/malformed ⇒ ≠0, (b) guard evaluates `branch=main` instead of the branch being pushed | O+S+live | RM-02 | 8K / 100K | sonnet |
| RM-04 | Activation/version coherence; block launch on skew, fail SAFE; honest `doctor` labels | O+S | RM-01 | (in S-01) / 140K | sonnet |
| RM-05 | Break-glass replaces the three silent `MOSAIC BYPASS` fail-opens | O | RM-04, RM-02 | — / 120K | opus |
> ⚠ **RM-05 must not merge before RM-04.** The bypasses exist because the lease-broker daemon was
> never _deployed_ on this host — removing the fail-open before deployment coherence is real
@@ -219,32 +269,40 @@ spread is itself information, and X1 says we calibrate on real merged PRs.
---
## 5. Blocking decisions (need Mos, or Jason via Mos)
## 5. Decisions — all three ruled by Mos, 2026-07-31
These are escalated because they change the charter, the accepted directive, or another lane's
ownership — none is a question I should answer unilaterally.
**DECISION-1 — the wire-in point. ✅ RULED: accept the planners (Mos, 2026-07-31).**
The charter's `mosaic_orchestrator.py::run_single_task` target is the **disabled Python controller
this mission retires**; wiring the new choke point into the rail we are deleting is wrong.
**DECISION-1 — the wire-in point (changes `MISSION.md`).**
Both planners independently reject wiring the choke point into
`mosaic_orchestrator.py::run_single_task`, because that controller is disabled and references a
non-existent dispatcher. They propose wiring `packages/coord/src/runner.ts` +
`packages/mosaic/src/commands/launch.ts` and **deleting** the Python rail instead. This makes RM-52 a
deletion task rather than an integration task, and moves Build 1's acceptance onto a live
`mosaic yolo` invocation. _My recommendation: accept — wiring a disabled rail reproduces the exact
disease this mission exists to cure._
> **Corrected target (authoritative):** a **new production Node `TaskExecutor`** sitting on the
> **live dispatch path** — `packages/mosaic` launch + `packages/coord` — which Coord, Forge, and live
> dispatch all **submit through**. This is the MACP scout's _full_ recommendation ("replace the block
> **with** a Node executor **and** make Coord/Forge submit through it"), not a resurrection of the
> Python controller. RM-52 is therefore a **deletion** task, and Build 1's acceptance is measured on a
> live `mosaic yolo` invocation.
**DECISION-2 — rollback artifact + the availability trade (needs Jason).**
(a) Does "hard cutover, no flat-file interim" permit a **rehearsed, one-directional, read-as-authority-
by-nobody** rollback snapshot (OPUS D3)? (b) Do we pre-commit in writing that "no DB ⇒ the fleet
stops" and "unpersistable event ⇒ the operation fails" (OPUS D8)? That is a real availability
regression versus today's limping flat-file fleet — correct for a system whose defining failure is
_silent continuation_, but it should be a decision, not a 2am discovery.
_My recommendation: yes to both; a rollback snapshot nothing reads is not an interim tracking system._
Mos ruled this resolvable from the already-accepted retire-the-Python-rail decision — his authority,
not a Jason escalation. RM-21/RM-26/RM-27/RM-52 all take the corrected target.
**DECISION-3 — RM-03 ownership vs. parked PR #1023 (needs Mos).**
P-QUEUE-001 is absorbed into this mission, but PR #1023 is explicitly parked under Mos. Two lanes can
legitimately claim it. A third recursion on the gate-6 defect — performed by the remediation itself —
would be the postmortem's own anti-pattern. _Not dispatching RM-03 until Mos rules._
**DECISION-2 — rollback artifact + availability trade. ⏸ JASON-PENDING — NOT BLOCKING.**
The DB build is phases away, so this is queued for Jason's next session rather than escalated now.
**Binding requirement in the meantime (Mos, from P-RECOVERY-001):** the DB spine **must NOT be a
single-point hard-stop.** Design for a broker-independent / degraded mode **plus** a rollback
artifact. Jason finalises only the specific availability target. This reverses my earlier reading of
OPUS D8 ("the fallback is: the fleet stops") — that answer is **not** pre-committed; a degraded mode
is now a design requirement on RM-12, RM-13, RM-23, RM-36 and RM-53.
**DECISION-3 — RM-03 vs. parked PR #1023. ✅ RULED: HOLD RM-03 (Mos, 2026-07-31).**
Do **not** open a third gate-6 lane — that is the postmortem's own anti-pattern performed by the
remediation. PR #1023 sits in Jason's **parked delivery stack**; its disposition (close, or supersede
by RM-03) is Jason's at his next session.
- **PR #1023 → `SUPERSEDED-PENDING-JASON`.** RM-03 stays `HOLD`; when Jason rules, RM-03 proceeds as
the single correct lane.
- **RM-02 and RM-55 proceed independently and are NOT held.** The per-merge-commit gate-assertion
requirement is the _conformance_ capability, not the gate-6 fix itself — different scope, no
ownership collision.
---