docs(remediation): apply Mos rulings on D-1/D-2/D-3; bank D-7 tmpfs ENOSPC
D-1 RULED — corrected wire-in target is a new production Node TaskExecutor on the LIVE dispatch path (packages/mosaic launch + packages/coord) that Coord/Forge/live dispatch submit through. The Python rail is deleted, not ported. RM-21/26/27/52 updated. D-2 — Jason-pending and explicitly NON-blocking. Binding meanwhile: the DB spine must not be a single-point hard-stop; degraded mode + rollback artifact become design requirements on RM-12/13/23/36/53. This reverses the earlier recommendation to pre-commit 'no DB means the fleet stops'. D-3 — RM-03 HOLD; PR #1023 marked SUPERSEDED-PENDING-JASON; RM-02/RM-55 proceed independently since conformance assertion is not the gate-6 fix itself. RM-03 now carries BOTH queue-guard defects: wrong exit on unknown, and evaluating branch=main rather than the branch being pushed. D-7 — shared 30G /tmp hit 100% ENOSPC, degrading two seats. ~5.2G of it is session scratch dead for 8-9 days; this session's own footprint is 88K. Same missing capability as P-FLEET-001 stale-session GC, applied to disk. Folded into RM-50, and RM-01 gains artifacts-off-shared-tmp acceptance criteria. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
This commit is contained in:
mos-dt-0
committed by
mosaic-coder
mosaic-coder
co-authored by
Claude Opus 5
Claude Opus 5
parent
d4f68040ab
commit
1324385bba
+88
-30
@@ -40,7 +40,10 @@ point. Both planners independently rejected it on the same evidence: that contro
|
|||||||
point into a disabled rail produces **a stranded executor — the identical built-but-unwired disease,
|
point into a disabled rail produces **a stranded executor — the identical built-but-unwired disease,
|
||||||
one layer up, that would look "done" in a PR.** The live paths are
|
one layer up, that would look "done" in a PR.** The live paths are
|
||||||
`packages/mosaic/src/commands/launch.ts` and `packages/coord/src/runner.ts`.
|
`packages/mosaic/src/commands/launch.ts` and `packages/coord/src/runner.ts`.
|
||||||
This contradicts the charter and is escalated as **DECISION-1** (§5).
|
This contradicted the charter and was escalated as DECISION-1 — **now RULED in the planners' favour by
|
||||||
|
Mos (§5)**. The corrected target is a new production Node `TaskExecutor` on the live dispatch path
|
||||||
|
(`packages/mosaic` launch + `packages/coord`) that Coord/Forge/live dispatch submit through; the
|
||||||
|
Python rail is deleted, not ported.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -66,6 +69,53 @@ side effect, so `main` went green again **without anyone ever learning the gate
|
|||||||
> present-tense framing was wrong. The hygiene PR therefore carries the `.prettierignore` fix only —
|
> present-tense framing was wrong. The hygiene PR therefore carries the `.prettierignore` fix only —
|
||||||
> the README needs no fix today.
|
> the README needs no fix today.
|
||||||
|
|
||||||
|
### Third live instance, same class — the queue guard, hit by this orchestrator
|
||||||
|
|
||||||
|
Running the **mandated** pre-push guard during TASK-0:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ ~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose push
|
||||||
|
[ci-queue-wait] platform=gitea purpose=push branch=main sha=06e0d403…
|
||||||
|
[ci-queue-wait] state=unknown purpose=push branch=main
|
||||||
|
$ echo $? → 0
|
||||||
|
```
|
||||||
|
|
||||||
|
**Two distinct defects in one tool**, both feeding RM-03:
|
||||||
|
|
||||||
|
1. **Wrong exit** — `state=unknown` ⇒ `exit 0`. The defect at `ci-queue-wait.sh:282-288` that OPUS
|
||||||
|
documented and that PR #1023 is parked on. A required gate returned PASS on an indeterminate result.
|
||||||
|
2. **Wrong branch** — it evaluated `branch=main`, not the branch actually being pushed. Even a
|
||||||
|
correctly-exiting guard would have been answering the wrong question.
|
||||||
|
|
||||||
|
**Standing doctrine (Mos):** until RM-03 lands, a green from this guard carries **zero information**
|
||||||
|
and must not be cited as merge evidence. Rely on reviewer clearance + real CI.
|
||||||
|
|
||||||
|
Three independent live instances in a single session — format gate, agent context reset, queue guard —
|
||||||
|
is the class confirmed, not anecdote.
|
||||||
|
|
||||||
|
### D-7 — shared-tmpfs contention → cascading ENOSPC (live incident, 2026-07-31)
|
||||||
|
|
||||||
|
The shared 30 G `/tmp` hit **100% ENOSPC** mid-session. It broke tool calls in **two different seats**
|
||||||
|
(mine and Mos's) — a single full disk degrades every agent on the host at once. Recurring: prior
|
||||||
|
incidents 2026-06-18 and 2026-07-17.
|
||||||
|
|
||||||
|
Attribution matters, because the wrong owner cleans the wrong thing. Measured:
|
||||||
|
|
||||||
|
| path | size | last modified | owner |
|
||||||
|
| ---------------------------------------------- | --------- | ---------------------------- | ------------------------------------------------- |
|
||||||
|
| `…/-src-mosaic-stack/6d2faee6…` (this session) | **88 K** | live | mos-remediation |
|
||||||
|
| `…/-src-mosaic-stack/c743185d…` | **3.6 G** | **2026-07-22** (9 days dead) | abandoned session, same project path |
|
||||||
|
| `…/claude-1001/pnpm-store` | **1.6 G** | **2026-07-23** (8 days dead) | abandoned; the live store is correctly on `$HOME` |
|
||||||
|
|
||||||
|
So ~5.2 G — the bulk of the pressure — is **dead session scratch that nothing will ever read again**.
|
||||||
|
This is not a quota problem; it is **P-FLEET-001's stale-session GC, applied to disk instead of tmux
|
||||||
|
sessions.** The same missing capability (nothing owns reaping dead ephemeral state) produces both the
|
||||||
|
orphaned-session failure and this one. Reaping dead-session scratch belongs in RM-50 alongside stale
|
||||||
|
tmux-session GC.
|
||||||
|
|
||||||
|
**Added to RM-01 as acceptance criteria:** heavy build artifacts (node_modules, package stores, build
|
||||||
|
output) must land on the main disk in the worktree, never on the shared 30 G `/tmp`.
|
||||||
|
|
||||||
**The self-erasure is the important part.** An inert gate that is masked by unrelated downstream
|
**The self-erasure is the important part.** An inert gate that is masked by unrelated downstream
|
||||||
commits produces no lasting artifact, which is precisely why this class survives for months. Detection
|
commits produces no lasting artifact, which is precisely why this class survives for months. Detection
|
||||||
cannot rely on "is `main` currently red" — it must be per-merge-commit.
|
cannot rely on "is `main` currently red" — it must be per-merge-commit.
|
||||||
@@ -112,13 +162,13 @@ spread is itself information, and X1 says we calibrate on real merged PRs.
|
|||||||
|
|
||||||
⛔ _No gate-introducing task in any later phase may merge before RM-02._
|
⛔ _No gate-introducing task in any later phase may merge before RM-02._
|
||||||
|
|
||||||
| id | task | src | depends_on | est (S/O) | tier |
|
| id | task | src | depends_on | est (S/O) | tier |
|
||||||
| ----- | -------------------------------------------------------------------------------------------- | --- | ------------ | ---------------- | ------ |
|
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | -------- | ------------ | ---------------- | ------ |
|
||||||
| RM-01 | Reproducible non-root checkout; pre-push gate fails on **code, not env** (banks D-1/D-2/D-5) | O+S | — | 6K / 60K | codex |
|
| RM-01 | Reproducible non-root checkout; gate fails on **code, not env**; heavy artifacts OFF shared `/tmp` (banks D-1/D-2/D-5/D-7) | O+S+live | — | 6K / 60K | codex |
|
||||||
| RM-02 | **Gate registry + negative-control CI check** (anti-inert-gate harness) ★keystone | O | RM-01 | — / 120K | opus |
|
| RM-02 | **Gate registry + negative-control CI check** (anti-inert-gate harness) ★keystone | O | RM-01 | — / 120K | opus |
|
||||||
| RM-03 | Queue-guard fail-closed rework (`unknown`/`no-status`/malformed ⇒ ≠0) | O+S | RM-02 | 8K / 100K | sonnet |
|
| RM-03 ⏸**HOLD** | Queue-guard: **two** defects — (a) `unknown`/`no-status`/malformed ⇒ ≠0, (b) guard evaluates `branch=main` instead of the branch being pushed | O+S+live | RM-02 | 8K / 100K | sonnet |
|
||||||
| RM-04 | Activation/version coherence; block launch on skew, fail SAFE; honest `doctor` labels | O+S | RM-01 | (in S-01) / 140K | sonnet |
|
| RM-04 | Activation/version coherence; block launch on skew, fail SAFE; honest `doctor` labels | O+S | RM-01 | (in S-01) / 140K | sonnet |
|
||||||
| RM-05 | Break-glass replaces the three silent `MOSAIC BYPASS` fail-opens | O | RM-04, RM-02 | — / 120K | opus |
|
| RM-05 | Break-glass replaces the three silent `MOSAIC BYPASS` fail-opens | O | RM-04, RM-02 | — / 120K | opus |
|
||||||
|
|
||||||
> ⚠ **RM-05 must not merge before RM-04.** The bypasses exist because the lease-broker daemon was
|
> ⚠ **RM-05 must not merge before RM-04.** The bypasses exist because the lease-broker daemon was
|
||||||
> never _deployed_ on this host — removing the fail-open before deployment coherence is real
|
> never _deployed_ on this host — removing the fail-open before deployment coherence is real
|
||||||
@@ -219,32 +269,40 @@ spread is itself information, and X1 says we calibrate on real merged PRs.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 5. Blocking decisions (need Mos, or Jason via Mos)
|
## 5. Decisions — all three ruled by Mos, 2026-07-31
|
||||||
|
|
||||||
These are escalated because they change the charter, the accepted directive, or another lane's
|
**DECISION-1 — the wire-in point. ✅ RULED: accept the planners (Mos, 2026-07-31).**
|
||||||
ownership — none is a question I should answer unilaterally.
|
The charter's `mosaic_orchestrator.py::run_single_task` target is the **disabled Python controller
|
||||||
|
this mission retires**; wiring the new choke point into the rail we are deleting is wrong.
|
||||||
|
|
||||||
**DECISION-1 — the wire-in point (changes `MISSION.md`).**
|
> **Corrected target (authoritative):** a **new production Node `TaskExecutor`** sitting on the
|
||||||
Both planners independently reject wiring the choke point into
|
> **live dispatch path** — `packages/mosaic` launch + `packages/coord` — which Coord, Forge, and live
|
||||||
`mosaic_orchestrator.py::run_single_task`, because that controller is disabled and references a
|
> dispatch all **submit through**. This is the MACP scout's _full_ recommendation ("replace the block
|
||||||
non-existent dispatcher. They propose wiring `packages/coord/src/runner.ts` +
|
> **with** a Node executor **and** make Coord/Forge submit through it"), not a resurrection of the
|
||||||
`packages/mosaic/src/commands/launch.ts` and **deleting** the Python rail instead. This makes RM-52 a
|
> Python controller. RM-52 is therefore a **deletion** task, and Build 1's acceptance is measured on a
|
||||||
deletion task rather than an integration task, and moves Build 1's acceptance onto a live
|
> live `mosaic yolo` invocation.
|
||||||
`mosaic yolo` invocation. _My recommendation: accept — wiring a disabled rail reproduces the exact
|
|
||||||
disease this mission exists to cure._
|
|
||||||
|
|
||||||
**DECISION-2 — rollback artifact + the availability trade (needs Jason).**
|
Mos ruled this resolvable from the already-accepted retire-the-Python-rail decision — his authority,
|
||||||
(a) Does "hard cutover, no flat-file interim" permit a **rehearsed, one-directional, read-as-authority-
|
not a Jason escalation. RM-21/RM-26/RM-27/RM-52 all take the corrected target.
|
||||||
by-nobody** rollback snapshot (OPUS D3)? (b) Do we pre-commit in writing that "no DB ⇒ the fleet
|
|
||||||
stops" and "unpersistable event ⇒ the operation fails" (OPUS D8)? That is a real availability
|
|
||||||
regression versus today's limping flat-file fleet — correct for a system whose defining failure is
|
|
||||||
_silent continuation_, but it should be a decision, not a 2am discovery.
|
|
||||||
_My recommendation: yes to both; a rollback snapshot nothing reads is not an interim tracking system._
|
|
||||||
|
|
||||||
**DECISION-3 — RM-03 ownership vs. parked PR #1023 (needs Mos).**
|
**DECISION-2 — rollback artifact + availability trade. ⏸ JASON-PENDING — NOT BLOCKING.**
|
||||||
P-QUEUE-001 is absorbed into this mission, but PR #1023 is explicitly parked under Mos. Two lanes can
|
The DB build is phases away, so this is queued for Jason's next session rather than escalated now.
|
||||||
legitimately claim it. A third recursion on the gate-6 defect — performed by the remediation itself —
|
**Binding requirement in the meantime (Mos, from P-RECOVERY-001):** the DB spine **must NOT be a
|
||||||
would be the postmortem's own anti-pattern. _Not dispatching RM-03 until Mos rules._
|
single-point hard-stop.** Design for a broker-independent / degraded mode **plus** a rollback
|
||||||
|
artifact. Jason finalises only the specific availability target. This reverses my earlier reading of
|
||||||
|
OPUS D8 ("the fallback is: the fleet stops") — that answer is **not** pre-committed; a degraded mode
|
||||||
|
is now a design requirement on RM-12, RM-13, RM-23, RM-36 and RM-53.
|
||||||
|
|
||||||
|
**DECISION-3 — RM-03 vs. parked PR #1023. ✅ RULED: HOLD RM-03 (Mos, 2026-07-31).**
|
||||||
|
Do **not** open a third gate-6 lane — that is the postmortem's own anti-pattern performed by the
|
||||||
|
remediation. PR #1023 sits in Jason's **parked delivery stack**; its disposition (close, or supersede
|
||||||
|
by RM-03) is Jason's at his next session.
|
||||||
|
|
||||||
|
- **PR #1023 → `SUPERSEDED-PENDING-JASON`.** RM-03 stays `HOLD`; when Jason rules, RM-03 proceeds as
|
||||||
|
the single correct lane.
|
||||||
|
- **RM-02 and RM-55 proceed independently and are NOT held.** The per-merge-commit gate-assertion
|
||||||
|
requirement is the _conformance_ capability, not the gate-6 fix itself — different scope, no
|
||||||
|
ownership collision.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user