Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2a8281a6f0 | ||
|
|
5b63ea3236 | ||
|
|
64f6558ee4 | ||
|
|
f799e8b155 | ||
|
|
4992e49097 | ||
|
|
e86c712f5c | ||
|
|
87e67a765f |
No files matched your search
@@ -0,0 +1,111 @@
|
||||
# Row 302 (#1809): assertion preservation
|
||||
|
||||
Base 0e30a85fa. `git diff 0e30a85fa -- 'packages/*/tests/*'` on an existing file changes four lines
|
||||
and nothing else. Each one is ruled by Mos (mos-1414, amended by mos-1415 and mos-1416, relayed in
|
||||
sage-3206, sage-3209 and sage-3210). No assertion is removed or loosened. Each one now pins the new
|
||||
code or exit for the same call.
|
||||
|
||||
## 1. packages/bus/tests/threads.test.mjs:394 (mos-1414 a)
|
||||
|
||||
Old:
|
||||
|
||||
```
|
||||
assert.throws(() => call('a', 'message.receive'), { code: 'not-holder' });
|
||||
```
|
||||
|
||||
New:
|
||||
|
||||
```
|
||||
assert.throws(() => call('a', 'message.receive'), { code: 'launcher-released' });
|
||||
```
|
||||
|
||||
:392 (threadIdle's own second refusal, not-holder) and :395 (the pending check) are unchanged.
|
||||
|
||||
## 2. packages/harness/tests/runner.test.mjs:771 (mos-1415)
|
||||
|
||||
Old:
|
||||
|
||||
```
|
||||
assert.equal(out.code, EXIT.session, out.stderr);
|
||||
```
|
||||
|
||||
New:
|
||||
|
||||
```
|
||||
assert.equal(out.code, 0, out.stderr);
|
||||
```
|
||||
|
||||
out.stderr stays as the message.
|
||||
|
||||
## 3. packages/harness/tests/runner.test.mjs:772 (mos-1415)
|
||||
|
||||
Old:
|
||||
|
||||
```
|
||||
assert.match(out.stderr, new RegExp(`runner: landing turn 1 for decision ${d1.id} \\(vikunja:1/2\\): ok\n[\\s\\S]*runner: session stopped working: not-holder\n`));
|
||||
```
|
||||
|
||||
New:
|
||||
|
||||
```
|
||||
assert.match(out.stderr, new RegExp(`runner: landing turn 1 for decision ${d1.id} \\(vikunja:1/2\\): ok\n[\\s\\S]*runner: session released by the launcher\n`));
|
||||
```
|
||||
|
||||
Only the last line of the pattern changes. The landing-turn part is the same, word for word.
|
||||
|
||||
## 4. packages/bus/tests/land.test.mjs:464 (mos-1416)
|
||||
|
||||
Old:
|
||||
|
||||
```
|
||||
assert.throws(() => w.call('merge-gate', 'message.receive'), /not-holder/);
|
||||
```
|
||||
|
||||
New:
|
||||
|
||||
```
|
||||
assert.throws(() => w.call('merge-gate', 'message.receive'), /launcher-released/);
|
||||
```
|
||||
|
||||
:457, :463 and :466 are unchanged.
|
||||
|
||||
## The not-holder grep, rerun on the build (mos-1414)
|
||||
|
||||
`git grep -n "not-holder\|session stopped working\|session-invalid"` over packages/cli, seat and harness
|
||||
tests and scripts/, outside the new files:
|
||||
|
||||
- runner.test.mjs:558, `session stopped working: unauthenticated`. Not a release. Stays.
|
||||
- runner.test.mjs:628 (a comment) and :647, the limit stop. The limit revoke isn't threadIdle or
|
||||
landRelease, so it stays not-holder and exit 22 (Dewey Q3).
|
||||
- session.test.mjs:116, a list of end reasons. No call.
|
||||
|
||||
In packages/bus/tests, only end-launch, land and threads call threadIdle or landRelease. Their other
|
||||
not-holder pins: end-launch:198, threads:383-385 and :392, land:457 and :463 are the ops' own refusals.
|
||||
threads:493 follows assistant.stop. threads:416 releases a2-run as the test's last line. The fix reaches
|
||||
none of them, so no hit goes to Mos.
|
||||
|
||||
## 5. packages/bus/tests/launcher-released.test.mjs:89, a new test's own line (d6, sage-3294, Mos ruling sage-3293)
|
||||
|
||||
This file is new in row 302, so nothing at the base changes. The line changed between the pushed tests
|
||||
commit f799e8b15 and d6, after the first gate run went red on it (gizmoduck-0388). It's recorded here
|
||||
because the change narrows what the line counts.
|
||||
|
||||
Old:
|
||||
|
||||
```
|
||||
assert.equal(store.get('SELECT count(*) AS n FROM deliveries WHERE message=?', later.id).n, 0, 'nothing delivered to the released run');
|
||||
```
|
||||
|
||||
New:
|
||||
|
||||
```
|
||||
assert.equal(store.get('SELECT count(*) AS n FROM deliveries WHERE message=? AND holder_run=?', later.id, 'a-run').n, 0, 'nothing delivered to the released run');
|
||||
assert.deepEqual(store.all('SELECT op, holder_run FROM deliveries WHERE message=?', later.id).map((r) => ({ ...r })), [{ op: 'routed', holder_run: null }]);
|
||||
```
|
||||
|
||||
The old line counted every delivery row for the message. `#route` (broker.mjs:6482) writes one routed
|
||||
row for each message, and `#recipient` (:832-834) gives it holder_run null when no run holds the role.
|
||||
After threadIdle no run holds thread A, so the count was 1, with no row for a-run. The first new line
|
||||
keeps the claim it was meant to make: no row names the released run. The second pins the whole set,
|
||||
one routed row held by nobody. That is tighter than the old line on everything except the null-holder
|
||||
row, which it requires.
|
||||
@@ -0,0 +1,444 @@
|
||||
# Row 302 (#1809, Vikunja 588): the idle-stop race
|
||||
|
||||
Owner Gizmoduck. Reviewer Dewey. Gate owner Sage. Brief:
|
||||
docs/plans/2026-10-23_idle-stop-race-row.md, "Brief: the idle-stop race (Vikunja 588)". Base 0e30a85fa.
|
||||
This is the design, written static only. Nothing gets built until Dewey approves it (sage-3200).
|
||||
|
||||
## The race, as the code has it at 0e30a85fa
|
||||
|
||||
- `packages/cli/src/launcher.mjs:703-717`. The idle tick calls the `threadIdle` op. On `released` it adds
|
||||
the run to `stopping`, records `thread-idle`, and only then sends SIGTERM (:716).
|
||||
- `packages/bus/src/broker.mjs:1044-1057`. `threadIdle` releases the claim in the same synchronous call
|
||||
(`#claimEnd(..., 'release', null)`, :1055). The comment at :1040-1043 says the order is deliberate:
|
||||
row 121 r2, Dewey's F3 in 28591.
|
||||
- `broker.mjs:1002-1008`. `#agent` is the only place a thread run's own call gets `not-holder`, because
|
||||
its session's latest claim row isn't a `claim` of its generation. `message.receive` (:5586-5591) and
|
||||
`state` (:5805-5807) both go through it, and so do `message.send` and `message.read`.
|
||||
- `packages/harness/src/runner.mjs:55`. `SESSION_GONE` holds `unauthenticated`, `not-holder` and
|
||||
`run-ended`. Four sites set `code = EXIT.session` (22) on any of them, without checking `stopping`:
|
||||
the landing `state` (:500-506), `message.receive` (:526-531), the turn's `state` (:545-552) and the
|
||||
reply (:575-582).
|
||||
- `packages/seat/src/session.mjs:340` maps 22 to `session-invalid`.
|
||||
|
||||
So if a runner call reaches the broker after the release and before SIGTERM sets `stopping`, the run
|
||||
exits 22 by itself and ends `session-invalid`. A call already in flight when the release lands counts
|
||||
too. Row 292's diagnostic saw this once in about 100 idle stops. The clear test's run r846b8d083502 logged
|
||||
`not-holder` and then `exiting 22`.
|
||||
|
||||
## Design: a distinct answer for a run the idle stop released (Mos's suggestion, adopted)
|
||||
|
||||
### Broker (`packages/bus/src/broker.mjs`)
|
||||
|
||||
1. Add a new in-memory set, `#idled`, keyed `business:run`, next to `#ended`, `#received` and `#stops`.
|
||||
`threadIdle` adds the run's key in the same synchronous call that releases it, and only when it
|
||||
returns `{released:true}`. Release and mark happen together, so no call can be answered between them.
|
||||
2. In `#agent`, the refusal becomes
|
||||
`fail(this.#idled.has(s.business + ':' + s.run) ? 'idle-released' : 'not-holder')`. Nothing else in
|
||||
`#agent` changes. A run the idle stop released now gets `idle-released` at every call after the
|
||||
release, whether it's a receive, `state`, a send or a read, and whether or not another run has
|
||||
claimed the session since. Every other path that takes a claim away still answers `not-holder`:
|
||||
- `assistant.stop` (:5975), `instance.end` (:6009), the chief lifecycle revoke (:4682) and
|
||||
`role.revoke` (:5230, :5235)
|
||||
- the session clear (:1146)
|
||||
- the limit stop (`#limitStop`, :4842-4855). It releases a thread claim and then refuses
|
||||
`not-holder` on purpose, so that the run exits 22.
|
||||
- another run's newer generation, where the run wasn't idle-released
|
||||
3. `endLaunch` (:1300) drops the key along with `#ended.add`. After the end, the run's caps are gone and
|
||||
its calls answer `unauthenticated`, as they do today. The set lives only in memory, like `#runs`. A
|
||||
broker that restarts loses `#runs` too, so an old run's calls answer `unauthenticated` there either
|
||||
way.
|
||||
4. `threadIdle`'s own refusals don't change. A second `threadIdle` on a released run is still
|
||||
`not-holder` (threads.test:392). The answer comes from the `#agent` path only, for the run's own
|
||||
calls.
|
||||
|
||||
Why a new code and not a flag on `not-holder`: the wire carries only `error: <code>`
|
||||
(server.mjs:96-160), and the client accepts only `^[a-z-]{1,64}$` (client.mjs:42-43). A new code changes
|
||||
neither of them. Tools and docs that switch on `not-holder` keep their meaning, because `not-holder`
|
||||
still means "your claim was taken away".
|
||||
|
||||
### Runner (`packages/harness/src/runner.mjs`)
|
||||
|
||||
5. A new constant, `IDLE_RELEASED = "idle-released"`. It is not added to `SESSION_GONE`. At each of the
|
||||
four sites, `idle-released` is checked before `SESSION_GONE`. The runner logs
|
||||
`runner: session released by the launcher's idle stop`, leaves `code` at `EXIT.stopped` (0), sets a
|
||||
local `released`, and leaves the loop the same way the site does for `SESSION_GONE` today.
|
||||
6. The `role.release` at exit (:589-595) is skipped when `released`. The claim is already released, so
|
||||
the call would only be refused and logged.
|
||||
7. If the runner didn't handle the code, the receive loop would count `idle-released` as an
|
||||
unreachable broker and retry (:532-537). The run would then end at SIGTERM (0), or after 30 polls
|
||||
(23). Every site handles it so that neither of those decides how the run ends.
|
||||
|
||||
How this meets mos-1411 condition 1: `unauthenticated` (the caps went at the end), `run-ended`, and every
|
||||
`not-holder` that isn't an idle release still exit 22 and end `session-invalid`. Only the broker's own
|
||||
record that `threadIdle` released this run maps to `stopped/0`. The launcher's `stopping` set and SIGTERM
|
||||
play no part in the mapping. That also covers a poll that lands before the signal, and a call that was
|
||||
in flight when the release landed.
|
||||
|
||||
### Launcher and session.mjs
|
||||
|
||||
8. No change. The runner exits 0 by itself, and SIGTERM's start-time check (:716) then finds no process,
|
||||
which the code already handles. `endReason(0)` is `stopped`.
|
||||
|
||||
## Tests
|
||||
|
||||
### The forcing test (mos-1411 condition 2): new file `packages/cli/tests/idle-stop-race.test.mjs`
|
||||
|
||||
This is a thread launch on the same fixture shape as thread-launch.test.mjs (`start(t, prepared, { ops })`
|
||||
replaces launch ops, :78-101), with `sessionDefaults.pollInterval` set to 100. The test replaces the
|
||||
`threadIdle` op:
|
||||
|
||||
```js
|
||||
threadIdle: async (m) => {
|
||||
const r = runtime.threadIdle(m.record);
|
||||
if (r.released) window.set(m.record.run, await held(m.record.run)); // returns only after the runner's own exit, or 20 s
|
||||
return r;
|
||||
}
|
||||
```
|
||||
|
||||
`held(run)` waits for two things, in order. First, an `action.refused` event whose actor is `run` and
|
||||
whose seq is after the release's `thread_claims` row. That proves a runner call reached the broker inside
|
||||
the window. Second, the runner's own exit (the run's `runner: exiting N` line in runner.log, or its pid
|
||||
gone), with no SIGTERM sent. The launcher sends SIGTERM only after the op returns, so the window stays
|
||||
open until the runner has decided how it ends. The test then asserts all of the following:
|
||||
|
||||
- `held` saw a refusal from the run after the release, and the runner exited before the op returned.
|
||||
- The refusal's code is `idle-released`.
|
||||
- The run's launch `end` event is `['stopped', 0, null]`.
|
||||
- runner.log has the idle-released line and no `session stopped working:` line.
|
||||
|
||||
Against the code today, the refusal is `not-holder`, the runner exits 22 inside the window, and the end
|
||||
is `['session-invalid', 22, null]`. The test fails, and fails every time, not at some rate: the
|
||||
window is held open until a call has landed in it. If the runner fix is dropped, the runner retries and
|
||||
doesn't exit inside the window, so `held` times out and the "exited before the op returned" assertion
|
||||
fails. If the broker fix is dropped, the end is 22.
|
||||
|
||||
A control test in the same file: after an idle release, a `role.revoke` the human approved on another
|
||||
thread's run (or `assistant.stop`) still ends that run `['session-invalid', 22, null]`. That shows a
|
||||
real not-holder still exits 22.
|
||||
|
||||
### Bus: new tests in `packages/bus/tests/idle-release.test.mjs` (a new file, so no existing assertion moves)
|
||||
|
||||
- After `threadIdle` releases a-run, a-run's `message.receive`, `state`, `message.send` and
|
||||
`message.read` each refuse `idle-released`. Each writes an `action.refused` event with code
|
||||
`idle-released`, and none of them delivers a message or writes a receipt.
|
||||
- `not-holder` still comes back for: a run released by `assistant.stop`, a run released by a session
|
||||
clear, a limit stop (stop then receive), a run whose session another run claimed without any idle
|
||||
release, and a second `threadIdle` on a released run. After `endLaunch`, the idle-released run's
|
||||
calls get `unauthenticated`, as today (its caps go at :1310-1311).
|
||||
- A role run's `role.release` followed by its own receive still gets `not-holder` (no idle).
|
||||
|
||||
### Harness: no new runner.test case
|
||||
|
||||
runner.test.mjs's `setup` (:105) binds role runs only, and its claims sit in `role_claims` (:876), so it
|
||||
can't reach `threadIdle`. The runner's own exit at receive is covered by the cli forcing test above,
|
||||
which runs the real runner. The forcing test reaches only the site that lands in the window, which in
|
||||
practice is the receive poll. The landing `state`, the turn's `state` and the reply sites get the same
|
||||
four lines, but no test forces each one. Q4 asks whether Dewey wants a per-site case. The existing 22
|
||||
tests (:539-560, :630-647, :772) stay unchanged and pass unchanged.
|
||||
|
||||
## The one existing assertion that changes (goes to Mos through Sage, per the brief)
|
||||
|
||||
`packages/bus/tests/threads.test.mjs:394`,
|
||||
`assert.throws(() => call('a', 'message.receive'), { code: 'not-holder' })`. This is a receive by the run
|
||||
that `threadIdle` released at :388, so under this design it answers `idle-released`. The assertion is
|
||||
changed, not loosened: the new pin is `{ code: 'idle-released' }`, and the property right below it
|
||||
(:395, the message stays `pending` for the next holder) stays. I've found no other existing assertion
|
||||
that pins `not-holder` for an idle-released run's own call. The build reruns
|
||||
`git grep -n "not-holder" 'packages/*/tests/*'` against every idle path before in-review, and names any
|
||||
other hit for Mos first.
|
||||
|
||||
## Docs
|
||||
|
||||
- packages/bus/README.md:320-326 (threadIdle). Add: after a release, the run's own calls answer
|
||||
`idle-released`, not `not-holder`. Also say why.
|
||||
- packages/harness/README.md:295-302, the exit table. Row 0 becomes "stopped, by a signal, or because the
|
||||
launcher's idle stop released the session (`idle-released`)". Row 22 is unchanged.
|
||||
- packages/bus/README.md:207 (`assistant.stop`): no change. It still gives `not-holder`.
|
||||
|
||||
## Open questions for Dewey
|
||||
|
||||
- Q1. `landRelease` (broker.mjs:4144-4165, launcher.mjs:876-889) has the same shape for a role run's
|
||||
land-relaunch: it releases, then the launcher sends SIGTERM, so a poll in between ends the run 22. I
|
||||
recommend folding it in. That means one line in `landRelease` (add to `#idled` on release), one bus
|
||||
assertion, and one cli forcing case like the one above. It falls outside this brief's words, so it
|
||||
needs Sage's yes. If the answer is no, I'll write it up as a follow-up row with the same evidence.
|
||||
- Q2. The code name `idle-released`. It names the cause, and if Q1 is yes it covers land-relaunch too,
|
||||
since that is also a release made by the launcher. A more general name like `launcher-released` would
|
||||
fit both. I prefer `released-by-launcher` only if Q1 is yes. Dewey picks.
|
||||
- Q3. A limit stop on a run that was already idle-released. `#limitStop` (:4842-4855) finds no claim to
|
||||
end and refuses `not-holder`, so the run ends 22. I'd leave that alone: a limit hit is a real stop and
|
||||
ends session-invalid, as it does today. It's possible only if the limit checker answers inside the
|
||||
window.
|
||||
|
||||
- Q4. Per-site coverage in the runner. Only the receive site is forced. I can add a cli case per site
|
||||
(hold the window until the run's landing `state`, or a turn's `state`, lands in it), at about one
|
||||
fixture each. I'd add them only if Dewey asks, since all four sites share one helper in the build.
|
||||
|
||||
## Gate (brief)
|
||||
|
||||
Dewey approves this design, then the manifest. Runs: the new forcing test, then
|
||||
compactions-launch.test.mjs whole, then the bus, cli and harness suites whole, then every
|
||||
`scripts/test-*.sh`. The mos-1411 point 4 rerun rule doesn't apply to this row's own gate.
|
||||
|
||||
Mutants. Each one must die on the forcing test, with the bus test also named where it applies:
|
||||
|
||||
- M1. `threadIdle` doesn't add to `#idled`. Dies on the forcing test (end 22) and the bus test.
|
||||
- M2. `#agent` ignores `#idled`. Same deaths as M1.
|
||||
- M3. The runner drops the `idle-released` branch at receive. Dies on the forcing test (no exit inside
|
||||
the window).
|
||||
- M4. The runner maps `idle-released` to `EXIT.session`. Dies on the forcing test (end 22).
|
||||
- M5. `#claimEnd` marks every release, not just threadIdle's. Dies on the bus not-holder controls and
|
||||
the cli control.
|
||||
- M6. `endLaunch` doesn't drop the key. This is not a correctness mutant: caps are gone, so the answer
|
||||
is `unauthenticated` either way. I'll drop it unless Dewey wants an assertion on it, and record it as
|
||||
an equivalent mutant.
|
||||
|
||||
Limits will be set in the run plan after the design round (mos-1275: `timeout -k 10` at three times or
|
||||
more the longest run).
|
||||
|
||||
## After Dewey's d1 review (dewey-0875, 05:32Z): APPROVE on D1, folded here before any build
|
||||
|
||||
D1 (required). role.claim's thread branch (broker.mjs:5154-5173) doesn't go through `#agent` and
|
||||
refuses only `run-revoked`, the session fence and `thread-already-held`. So an idle-released run could
|
||||
claim its free session again. Its key would still be in `#idled`, and a later real not-holder on that
|
||||
claim would answer `idle-released` and exit 0. Fix: the thread branch of role.claim refuses
|
||||
`idle-released` for a run in `#idled`, before the insert. The key stays, because the run's claim is over.
|
||||
Bus assertion: after threadIdle releases a-run, a-run's role.claim refuses `idle-released`, and no
|
||||
thread_claims row is added. Mutant M7: drop that check. It dies on that assertion.
|
||||
|
||||
Correction to "Harness: no new runner.test case". That's true for threadIdle only. runner.test reaches
|
||||
landRelease, which is a role-run path: :742-772 releases coder-run with `ctx.broker.landRelease` and no
|
||||
launcher, and pins `out.code === EXIT.session` (:771) and `session stopped working: not-holder` (:772).
|
||||
If Q1 is yes, those two lines change (to stopped/0 and the released line). They go to Mos through Sage
|
||||
with threads.test:394, and runner.test:742-772 then serves as the land-relaunch forcing case. No launcher means no
|
||||
SIGTERM, so the receive always lands after the release. The cli case for land-relaunch is dropped. If Q1
|
||||
is no, the follow-up row cites runner.test:771-772 as its evidence.
|
||||
|
||||
Q2. The code is `launcher-released` if Q1 is yes, and `idle-released` if Q1 is no. Every name in this
|
||||
file reads that way.
|
||||
|
||||
Q3. Agreed. A limit stop on an idled run stays not-holder and exits 22.
|
||||
|
||||
Q4. No per-site cases. Both ops return `released:false` while a delivered message is unread
|
||||
(threadIdle via `t.unread`, landRelease via its count). The runner reads only after the reply
|
||||
(runner.mjs:575-580), so a run can't be released mid-turn. landRelease keeps a run that hasn't received
|
||||
yet (`landing:true`), and thread runs have no landing. So only the receive site can see the code. All
|
||||
four sites keep the shared helper, and the three unreachable ones get a one-line comment saying why.
|
||||
|
||||
N1. The cli control must reach the runner before any launcher signal. It uses a role.revoke on the run
|
||||
(no launcher stop), or holds the signal the same way the forcing test does. assistant.stop gets checked
|
||||
first in case it goes through a launcher stop. It also asserts the run's runner.log has
|
||||
`session stopped working: not-holder` and `exiting 22`.
|
||||
N2. The control keeps "after an idle release", so `#idled` isn't empty when M5 runs.
|
||||
N3. M6 is equivalent. The key drop in endLaunch stays, so the set doesn't grow.
|
||||
N4. The other-pin grep also covers runner.test `session stopped working`. :558 (unauthenticated) and
|
||||
:647 (revoke) stay, and :772 changes only under Q1.
|
||||
Docs: if Q1 is yes, landRelease joins the bus README line and the harness exit-table row 0.
|
||||
|
||||
## Round d2 (sage-3204, 05:32Z)
|
||||
|
||||
Line numbers. Sage's check against 0e30a85fa is right, and the references above are now corrected. The
|
||||
second threadIdle not-holder pin is threads.test:392 and stays. The receive pin that changes is :394,
|
||||
and the pending check is :395 and stays. The idle release is at :388.
|
||||
|
||||
Sage asked for a grep of the land-relaunch tests for a not-holder pin on the relaunched run's own call.
|
||||
`git grep -l "landRelease\|land-relaunch" 'packages/*/tests/*'` lists 8 files. In those files,
|
||||
`not-holder|session stopped working|EXIT.session|session-invalid` hits only land.test.mjs and
|
||||
runner.test.mjs:
|
||||
|
||||
- land.test.mjs:464, `assert.throws(() => w.call('merge-gate', 'message.receive'), /not-holder/)`. This is
|
||||
the released run's own receive after `landRelease` released it (:458). Under Q1 yes, it answers
|
||||
`launcher-released`, and the regex fails. That's a third old-test change for Mos through Sage. The new
|
||||
pin is `/launcher-released/`.
|
||||
- land.test.mjs:457 (a non-holder run's landRelease) and :463 (a second landRelease) are landRelease's
|
||||
own refusals. They stay not-holder, the same as threads.test:392.
|
||||
- runner.test.mjs:771-772 change under Q1, as in the section above. runner.test.mjs:548, :557-558 and
|
||||
:646-647 are unauthenticated, unauthenticated and a limit stop. They stay.
|
||||
- The six cli files have no hit.
|
||||
|
||||
So under Q1 yes, the Mos package is threads.test:394, land.test:464 and runner.test:771-772. Under Q1 no,
|
||||
it's threads.test:394 alone. Nothing gets built until Mos rules and Sage gives the go.
|
||||
|
||||
## Round d3: as built (after mos-1414/1415/1416 and sage-3220)
|
||||
|
||||
Q1 is yes, so the code is `launcher-released` everywhere. The broker set is `#launcherReleased`, not
|
||||
`#idled`. Every `idle-released` above reads `launcher-released`.
|
||||
|
||||
### Correction to d1: message.send doesn't go through `#agent`
|
||||
|
||||
d1 said `message.send` and `message.read` both go through `#agent`. `message.read` does. `message.send`
|
||||
doesn't. So a released run's send isn't refused `launcher-released`. Only `message.receive`, `state`,
|
||||
`message.read` and `role.claim` answer it, and the bus test asserts exactly those four. That narrows
|
||||
what Dewey approved in d1, so I'm sending it to Dewey directly (sage-3220).
|
||||
|
||||
### Broker (packages/bus/src/broker.mjs)
|
||||
|
||||
- `#launcherReleased = new Set()`, keyed `business:run`, next to `#stops`, with a comment.
|
||||
- threadIdle and landRelease add the key after their release transaction.
|
||||
- `#agent` refuses `launcher-released` for a run in the set, and `not-holder` otherwise.
|
||||
- role.claim (D1) refuses `launcher-released` right after the `agent-required` check, before the thread
|
||||
and role branches. So it covers landRelease's role runs as well as thread runs.
|
||||
- endLaunch deletes the key after `#ended.add(key)`.
|
||||
- Q3 needs no code. A limit stop runs `#limitStop` (broker.mjs:4482) before any `#agent` check, and it
|
||||
fails `not-holder` itself (:4862), so a limit stop after a launcher release still exits 22.
|
||||
|
||||
### Runner (packages/harness/src/runner.mjs)
|
||||
|
||||
- `const LAUNCHER_RELEASED = "launcher-released"`.
|
||||
- The shared helper `ended(err, say = true)`. For `launcher-released`, it logs
|
||||
`runner: session released by the launcher`, sets `released` and leaves the exit code at
|
||||
`EXIT.stopped`. For the SESSION_GONE codes it keeps the old behaviour (`EXIT.session`, and the log
|
||||
line unless `say` is false).
|
||||
- It's used at the landing state check, receive, the turn state check and the reply. The reply site
|
||||
passes `say = false`, so its logging is the same as before. The three sites that can't see the code
|
||||
(Q4) carry a comment saying why.
|
||||
- At exit, the final `role.release` is skipped when `released`, because the launcher already released.
|
||||
|
||||
### Tests as built
|
||||
|
||||
- `packages/cli/tests/idle-stop-race.test.mjs`. This is the threadIdle forcing case. Inside the first
|
||||
released run's held threadIdle op, the human's `assistant.stop` revokes the other thread's holder
|
||||
(N1: no launcher signal can come while the op is held, launcher.mjs:662-663). The op waits for both
|
||||
runners' verdicts and end events. The idle run logs `session released by the launcher`, ends
|
||||
`["stopped", 0, null]` before its thread-idle line, and has `exiting 0` and no `session stopped working`.
|
||||
The control logs `session stopped working: not-holder`, ends `["session-invalid", 22, null]` and
|
||||
has `exiting 22`. The Store shows `launcher-released` and no `not-holder` refusals for the idle run,
|
||||
and `not-holder` for the control.
|
||||
- `packages/bus/tests/launcher-released.test.mjs` (renamed from idle-release.test.mjs). It has three
|
||||
tests. (1) threadIdle: receive, state, read and role.claim refuse `launcher-released`, role.claim adds
|
||||
no thread_claims row, the four refusals are logged, nothing is delivered, threadIdle's own second
|
||||
refusal stays not-holder, and after endLaunch the answer is unauthenticated. (2) The control (N2:
|
||||
after an idle release): assistant.stop's run answers not-holder and run-revoked, pm's own role.release
|
||||
answers not-holder, and a new holder for the released thread doesn't change the released run's answer.
|
||||
(3) landRelease: receive and state refuse `launcher-released`, role.claim refuses and adds no
|
||||
role_claims row, landRelease's own second refusal stays not-holder, and coder's own role.release
|
||||
answers not-holder.
|
||||
- runner.test.mjs:742-772 is the land-relaunch forcing case (mos-1415). There's no cli land-relaunch case.
|
||||
- The four old lines Mos ruled are in ASSERTION-PRESERVATION.md. The diff on old test files is those four
|
||||
lines and nothing else.
|
||||
|
||||
### Docs
|
||||
|
||||
The bus README's threadIdle paragraph names `launcher-released`, the four calls that answer it,
|
||||
landRelease, and that every other lost claim stays not-holder. The cli README's land-relaunch paragraph
|
||||
says calls between the release and the signal refuse `launcher-released` and the runner exits 0. The
|
||||
harness README's exit table row 0 names the launcher release.
|
||||
|
||||
### Red first (mos-1415)
|
||||
|
||||
There are two commits. The first holds only the tests and the four ruled pin edits. The second holds the
|
||||
fix (broker.mjs, runner.mjs) and the READMEs. The gate's first stage runs on the tests-only commit, and
|
||||
these two must fail there:
|
||||
|
||||
- idle-stop-race.test.mjs: the idle run ends `["session-invalid", 22, null]`, or its verdict is
|
||||
`session stopped working: not-holder`, whichever lands first.
|
||||
- runner.test.mjs, the case at :742: out.code is 22, not 0.
|
||||
|
||||
The bus test files and threads.test:394 and land.test:464 fail there too, and the packet records them.
|
||||
The red run doesn't count as the gate.
|
||||
|
||||
### Mutants (each run against the fix commit, restored and checked by sha256 after)
|
||||
|
||||
- M1. threadIdle doesn't add the key. Dies: idle-stop-race (end 22), launcher-released.test test 1,
|
||||
threads.test:394.
|
||||
- M2. `#agent` ignores the set. Dies: the same as M1, plus launcher-released.test test 3,
|
||||
land.test:464 and runner.test:771.
|
||||
- M3. The runner's helper drops the `launcher-released` branch. Dies: idle-stop-race (the run doesn't
|
||||
exit in the window, so `window.idle` is null) and runner.test:771-772.
|
||||
- M4. The runner maps `launcher-released` to `EXIT.session`. Dies: idle-stop-race (end 22) and
|
||||
runner.test:771.
|
||||
- M5. `#agent` answers `launcher-released` for every lost claim. Dies: the idle-stop-race control
|
||||
(verdict and end 22), launcher-released.test test 2, and test 3's coder line.
|
||||
- M6. endLaunch doesn't drop the key. Equivalent (N3), not run.
|
||||
- M7. role.claim drops the D1 check. Dies: launcher-released.test tests 1 and 3 (the claim succeeds).
|
||||
- M8. landRelease doesn't add the key. Dies: runner.test:771-772, land.test:464 and
|
||||
launcher-released.test test 3.
|
||||
|
||||
### Run plan (mos-1275 limits, `timeout -k 10`)
|
||||
|
||||
1. idle-stop-race.test.mjs alone, 600 s. Then runner.test.mjs whole, 900 s.
|
||||
2. compactions-launch.test.mjs whole, 600 s.
|
||||
3. The bus, harness and cli suites whole: bus 1200 s, harness 1200 s, cli 2100 s. 292's cli suite took
|
||||
630 s.
|
||||
4. Every scripts/test-*.sh, 900 s each.
|
||||
5. M1-M5, M7 and M8, each on its named files at the limits above.
|
||||
|
||||
TMPDIR comes from `mktemp -d /mnt/storage/scratch/tmp/gXXX`. Every run is slot work and starts only on a
|
||||
CLEAR from Sage.
|
||||
|
||||
## Round d4: Dewey's early static read (dewey-0878, 05:47Z)
|
||||
|
||||
C1. The d3 "correction" about message.send was wrong, and d1 was right. `message.send` calls
|
||||
`#consumeAuthority` (broker.mjs:5506), which calls `#checkAuthority`, which calls `#agent` (:3749).
|
||||
So a released run's send refuses `launcher-released`, and so does every verb that reaches `#agent`. There
|
||||
are 29 call sites, including role.release and every authority check. role.claim refuses it before the
|
||||
claim (D1). I misread the send path in d3, and no code depended on that reading. The runner's reply site
|
||||
already handles the code. The bus README now says "every call that needs the claim", not a list of four.
|
||||
Bus test 1 adds a message.send refusal, so the refused list is five `launcher-released`.
|
||||
|
||||
N1. Nothing killed a mutant that drops `&& !released` at runner exit (runner.mjs:596). That run calls
|
||||
role.release, logs `runner: role.release: launcher-released` and still exits 0. idle-stop-race now
|
||||
asserts the idle run's runner.log has no `runner: role.release:` line.
|
||||
- M9. Drop `&& !released` at runner.mjs:596. It dies on idle-stop-race (the new doesNotMatch).
|
||||
|
||||
N2. idle-stop-race waits for the whole held window (`"revoked" in window`), then asserts
|
||||
`window.control` is non-null with a message. If the other thread's run isn't holding when A's first
|
||||
release lands, the test now fails with that message instead of a bare "timed out waiting".
|
||||
|
||||
The mutants to run are M1-M5, M7, M8 and M9. These test edits are in a third commit, after the fix.
|
||||
ad9fa9f16 stays the red-first commit for the two forcing cases mos-1415 names.
|
||||
|
||||
## Round d5: red-first reordered (sage-3225, 05:49Z)
|
||||
|
||||
Sage pointed out that 8cf2c6d54 changed idle-stop-race.test.mjs and launcher-released.test.mjs after
|
||||
ad9fa9f16. So a red run on ad9fa9f16 would test older copies than the ones that land. I took the first of
|
||||
Sage's two options and reordered the commits locally, before any push. The order is now:
|
||||
|
||||
1. Tests: the final test files and this packet, over unfixed product files.
|
||||
2. The fix: broker.mjs, runner.mjs and the three READMEs.
|
||||
|
||||
ad9fa9f16, 6e5f782ae and 8cf2c6d54 are superseded and were never pushed. The old tip is kept locally as
|
||||
gizmoduck/row302-pre-reorder. The fix commit's tree matches 8cf2c6d54's except for this BUILD.md note.
|
||||
The "third commit" in d4 now lives in these two commits.
|
||||
|
||||
The red run uses the tests commit as it stands, with these test blobs (identical in the fix commit):
|
||||
|
||||
| Blob | File |
|
||||
| --- | --- |
|
||||
| 52343932b231691ba2b4fc117547ea7f6f504704 | packages/bus/tests/land.test.mjs |
|
||||
| 6c83c9ede41247fd50d40eeac3e8a16a75efacbf | packages/bus/tests/launcher-released.test.mjs |
|
||||
| 27193a745c4ed8d4936de96c1768ab010a572856 | packages/bus/tests/threads.test.mjs |
|
||||
| 9dfacfea15a487c4e7056ebae3e311cb514b0467 | packages/cli/tests/idle-stop-race.test.mjs |
|
||||
| 0792431b60314a17ed74b3df0c5b3748dc3c1286 | packages/harness/tests/runner.test.mjs |
|
||||
|
||||
The red run checks these blobs with `git rev-parse HEAD:<path>` before it starts.
|
||||
|
||||
## Round d6: the first gate run's red in launcher-released.test.mjs:89 (gizmoduck-0388, sage-3294)
|
||||
|
||||
The slot opened at 08:15Z (sage-3289). The gate tree was 649ec67ab, a local merge of 64f6558ee onto
|
||||
origin/refactor 1ac655005. The red stage on f799e8b15 passed: seven expected reds, with the pins at
|
||||
threads.test.mjs:394, land.test.mjs:464 and runner.test.mjs:771 (:772 sits behind :771 in one test).
|
||||
On 649ec67ab, idle-stop-race 1/1, runner.test 40/40 and compactions-launch 5/5 went green. The bus
|
||||
suite then had 875 of 876 pass. The one red was this row's own test, at launcher-released.test.mjs:89,
|
||||
1 !== 0. The gate stopped there, and the red went to Sage before any fix or rerun.
|
||||
|
||||
The test was wrong. `#route` (broker.mjs:6482) writes one routed delivery row for every message.
|
||||
After threadIdle no run holds thread A, so `#recipient` (:832-834) returns null and `#delivery`
|
||||
(:6612) writes holder_run null. The line counted every row for the message. Nothing was delivered to
|
||||
a-run. ASSERTION-PRESERVATION.md entry 5 has the old and new lines.
|
||||
|
||||
The fix narrows the count to rows for a-run and adds the line Sage made mandatory: the message has
|
||||
exactly one row, routed, holder_run null. bus rows come from node:sqlite with a null prototype, and the
|
||||
tests use assert/strict, so the second line spreads each row before deepEqual. That is the idiom in
|
||||
sessions.test.mjs:385 and session-clear.test.mjs:564. Sage's draft compared store.all directly, which
|
||||
would fail on the prototype alone.
|
||||
|
||||
Packaging (sage-3294): this is a third commit on gizmoduck/row302 after the pushed f799e8b15 and
|
||||
64f6558ee. It touches only that test line, this note and ASSERTION-PRESERVATION.md. It goes out as a
|
||||
fast-forward push in a gap Sage names, after Mos rules (sage-3293). Then:
|
||||
|
||||
1. The red run uses the final test files over f799e8b15's product: f799e8b15 checked out detached, with
|
||||
the five test files from the new tip checked out on top. The script checks the new blobs first.
|
||||
2. The whole gate runs again on a fresh merge of the new tip onto origin/refactor.
|
||||
|
||||
Dewey reviews the delta. The first run's logs stay in gizmo-302-out/gate-run1/.
|
||||
@@ -0,0 +1,112 @@
|
||||
# Row 302 (#1809): gate results
|
||||
|
||||
The slot ran from 08:15Z on 2026-10-23 (sage-3289), with Sage as gate owner and Dewey as reviewer.
|
||||
Logs: run1/gate-run.log and run2/gate-run.log. Script: run2/run-gate.sh and run2/mutants.py. The
|
||||
full per-file logs stay in /mnt/storage/scratch/gizmo-302-out/gate-run1/ and gate/.
|
||||
|
||||
## Commits
|
||||
|
||||
| Commit | What |
|
||||
| --- | --- |
|
||||
| f799e8b15 | red first: the tests over unfixed product (d5) |
|
||||
| 64f6558ee | the fix |
|
||||
| 5b63ea323 | d6: launcher-released.test.mjs:89, Mos ruling mos-1430 (relayed in sage-3297) |
|
||||
|
||||
The 5b63ea323 subject still says "Mos ruling pending sage-3293; local, not pushed". I wrote it before
|
||||
mos-1430 arrived, and Sage said not to amend. The ruling is mos-1430, and the commit was pushed as a
|
||||
fast-forward at about 08:23Z.
|
||||
|
||||
## Run 1 (08:18Z): stopped on a red outside the four pins
|
||||
|
||||
- Gate tree: 649ec67ab, a local merge of 64f6558ee onto origin/refactor 1ac655005.
|
||||
- Red stage: as expected (the same reds as run 2).
|
||||
- Green on 649ec67ab: idle-stop-race 1/1, runner.test 40/40 and compactions-launch 5/5. Then bus had
|
||||
875 of 876 pass. The red was this row's own test, at launcher-released.test.mjs:89, 1 !== 0.
|
||||
- It stopped at 08:19:28Z and went to Sage (gizmoduck-0388), then Mos. BUILD.md d6 has the cause.
|
||||
|
||||
## Run 2 (08:23:41Z to 08:38:14Z, 873 s)
|
||||
|
||||
The gate tree was 783602e06ed92f99c7c71d1e9c016a0f9e2c9b28, a local merge (not pushed). Parent 1 is
|
||||
origin/refactor 1ac655005, and parent 2 is 5b63ea323. The 12 row paths equal 5b63ea323's, and none
|
||||
of them changes in 0e30a85fa..1ac655005.
|
||||
|
||||
### Red: f799e8b15's product with the final test files from 5b63ea323
|
||||
|
||||
The five final test blobs:
|
||||
|
||||
| Blob | File |
|
||||
| --- | --- |
|
||||
| 52343932b231691ba2b4fc117547ea7f6f504704 | packages/bus/tests/land.test.mjs |
|
||||
| 6fb6e15c2fd34f742f9aab718ff654ab2704e198 | packages/bus/tests/launcher-released.test.mjs (new in d6) |
|
||||
| 27193a745c4ed8d4936de96c1768ab010a572856 | packages/bus/tests/threads.test.mjs |
|
||||
| 9dfacfea15a487c4e7056ebae3e311cb514b0467 | packages/cli/tests/idle-stop-race.test.mjs |
|
||||
| 0792431b60314a17ed74b3df0c5b3748dc3c1286 | packages/harness/tests/runner.test.mjs |
|
||||
|
||||
The script checked each blob with hash-object on the red tree. Only launcher-released differs from
|
||||
f799e8b15, and no product file differs from it.
|
||||
|
||||
There were seven reds, and nothing else failed:
|
||||
|
||||
- idle-stop-race: the released run ended with 'session stopped working: not-holder', exit 22 not 0
|
||||
- runner.test landing case: pin :771, with 22 vs 0
|
||||
- threads.test threadIdle case: pin :394
|
||||
- land.test landRelease case: pin :464
|
||||
- launcher-released: all three tests
|
||||
|
||||
On the bus files, 33 of 38 passed and 5 failed.
|
||||
|
||||
Pin :772 sits after :771 in the same test, so :771 stops the test first. :771's message is out.stderr,
|
||||
and that shows the old "session stopped working: not-holder" line and no "session released by the
|
||||
launcher" line. :89's two new lines never run on red, because the earlier refusals fail first. Run 2's
|
||||
green bus stage is their first green.
|
||||
|
||||
### Green on 783602e06
|
||||
|
||||
| Run | Result | Wall |
|
||||
| --- | --- | --- |
|
||||
| idle-stop-race | 1/1 | 2 s |
|
||||
| runner.test whole | 40/40 | 12 s |
|
||||
| compactions-launch | 5/5 | 14 s |
|
||||
| bus | 876/876 | 24 s |
|
||||
| harness | 151/151 | 13 s |
|
||||
| cli | 918/918 | 615 s |
|
||||
| test-auth | 15 | |
|
||||
| test-conductor | 17 | |
|
||||
| test-config | 55 | |
|
||||
| test-discord | 81 | |
|
||||
| test-extension-package | 18 | |
|
||||
| test-foundation | 44 | |
|
||||
| test-land-work | 86 | |
|
||||
| test-queue | node 156/156, suite 28 | |
|
||||
| test-release | 14 | |
|
||||
| test-task | 91 | |
|
||||
|
||||
Every scripts/test-*.sh ended with rc 0 and 0 failed. The node suites skipped nothing and left no
|
||||
todo. The tree was clean after the tests.
|
||||
|
||||
The shell scripts skipped two things:
|
||||
|
||||
- test-queue: "queue verify and render --check: this checkout is not the queue's canonical root".
|
||||
The gate runs in a worktree, not /mnt/storage/src/mosaic-stack.
|
||||
- test-task: "live provider cases (row 244: no provider call in the gate)", printed twice. It printed
|
||||
no other skip line, so no Docker group was skipped.
|
||||
|
||||
### Mutants on 783602e06
|
||||
|
||||
Each kill file ran whole, and every listed test name had to fail. After each mutant the file was
|
||||
checked out again and matched its sha256. At the end, broker.mjs was 0c6dbbc5… and runner.mjs was
|
||||
72ec39eb…, both equal to 5b63ea323's, and the tree was clean.
|
||||
|
||||
| Mutant | Change | Killed by |
|
||||
| --- | --- | --- |
|
||||
| M1 | no add after threadIdle's release | launcher-released (threadIdle), idle-stop-race |
|
||||
| M2 | #agent always not-holder | land, launcher-released (threadIdle, landRelease), idle-stop-race, runner landing case |
|
||||
| M3 | runner ignores launcher-released | idle-stop-race, runner landing case |
|
||||
| M4 | a released run exits session | idle-stop-race, runner landing case |
|
||||
| M5 | every lost claim reads launcher-released | launcher-released (other way), idle-stop-race |
|
||||
| M7 | role.claim's D1 check removed | launcher-released (threadIdle, landRelease) |
|
||||
| M8 | no add after landRelease's release | land, launcher-released (landRelease), runner landing case |
|
||||
| M9 | exit releases a launcher-released run | idle-stop-race |
|
||||
|
||||
All eight were killed by rule, and none survived. When the run ended, no node process had its cwd in
|
||||
the worktree.
|
||||
@@ -0,0 +1,57 @@
|
||||
blobs: the five d5 test blobs match in f799e8b15561ea2c0f0735e5328ba149c7a1a7b5 and 649ec67abcd315ac877bbad65742017b998137e4
|
||||
merge: 649ec67abcd315ac877bbad65742017b998137e4 = 64f6558ee4cc536447c720eb20b1f675f32ffbf0 onto 1ac655005e611157605a0dcc5adf7a37f9bca2b0; 12 row paths equal to 64f6558ee4cc536447c720eb20b1f675f32ffbf0, none touched in 0e30a85fa..1ac655005e611157605a0dcc5adf7a37f9bca2b0
|
||||
TIP 649ec67abcd315ac877bbad65742017b998137e4 (merge) FIX 64f6558ee4cc536447c720eb20b1f675f32ffbf0 RED f799e8b15561ea2c0f0735e5328ba149c7a1a7b5 base 0e30a85fa onto 1ac655005e611157605a0dcc5adf7a37f9bca2b0 origin/refactor 1ac655005e611157605a0dcc5adf7a37f9bca2b0 start 2026-10-23T08:18:31Z
|
||||
loadavg: 1.63 1.48 1.89 5/6323 1532949
|
||||
== 1. red first on f799e8b15561ea2c0f0735e5328ba149c7a1a7b5 (unfixed product, final tests)
|
||||
TMPDIR /mnt/storage/scratch/tmp/ggiJ
|
||||
idle-stop-race rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
TMPDIR /mnt/storage/scratch/tmp/g2U9
|
||||
runner-land rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
TMPDIR /mnt/storage/scratch/tmp/g3Lr
|
||||
bus rc=1 ℹ tests 38 ℹ pass 33 ℹ fail 5 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
✖ failing tests:
|
||||
✖ landRelease releases the holding run by land-relaunch once it has asked message.receive and nothing delivered to it is unread, and waiting messages go to the next holder (34.180349ms)
|
||||
✖ REQ-THREAD-2 REQ-LAUNCH-1: threadIdle releases only a thread with nothing waiting, and only for its holder (33.321208ms)
|
||||
✖ row 302 a run landRelease released answers launcher-released, and claims nothing again (36.389735ms)
|
||||
✖ row 302 a run threadIdle released answers launcher-released on receive, state, read and send, logs each, and claims nothing again (39.497664ms)
|
||||
✖ row 302 with a run launcher-released, a run that lost its claim another way still answers not-holder (39.379523ms)
|
||||
17: AssertionError [ERR_ASSERTION]: runner: acme/assistant claimed by run ra10cf293beb6
|
||||
21: + actual - expected
|
||||
32: actual: 'session stopped working: not-holder',
|
||||
33: expected: 'session released by the launcher',
|
||||
15: AssertionError [ERR_ASSERTION]: runner: demo/coder claimed by run coder-run
|
||||
35: actual: 22,
|
||||
36: expected: 0,
|
||||
pin red at threads.test.mjs:394:
|
||||
pin red at land.test.mjs:464:
|
||||
pin red at runner.test.mjs:771:
|
||||
:772 pattern absent from stderr (old not-holder line present)
|
||||
every red on f799e8b15561ea2c0f0735e5328ba149c7a1a7b5:
|
||||
✖ a run in a landing turn isn't released for a second held decision until its landing turns end and it has read its messages (602.229598ms)
|
||||
✖ failing tests:
|
||||
✖ landRelease releases the holding run by land-relaunch once it has asked message.receive and nothing delivered to it is unread, and waiting messages go to the next holder (34.180349ms)
|
||||
✖ REQ-THREAD-2 REQ-LAUNCH-1: threadIdle releases only a thread with nothing waiting, and only for its holder (33.321208ms)
|
||||
✖ row 302 a run landRelease released answers launcher-released, and claims nothing again (36.389735ms)
|
||||
✖ row 302 a run threadIdle released answers launcher-released on receive, state, read and send, logs each, and claims nothing again (39.497664ms)
|
||||
✖ row 302 REQ-THREAD-2 an idle-released run whose call lands before the SIGTERM ends stopped/0; a run whose claim the human revoked in the same window still ends session-invalid/22 (1672.44448ms)
|
||||
✖ row 302 with a run launcher-released, a run that lost its claim another way still answers not-holder (39.379523ms)
|
||||
red stage: both forcing cases and the four pins red
|
||||
wall 5 s
|
||||
== 2. green on 649ec67abcd315ac877bbad65742017b998137e4
|
||||
TMPDIR /mnt/storage/scratch/tmp/gaYE
|
||||
idle-stop-race rc=0 ℹ tests 1 ℹ pass 1 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
wall 1 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/g9t4
|
||||
runner rc=0 ℹ tests 40 ℹ pass 40 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
wall 12 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gwAp
|
||||
compactions-launch rc=0 ℹ tests 5 ℹ pass 5 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
wall 15 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gPAB
|
||||
bus rc=1 ℹ tests 876 ℹ pass 875 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
RED: bus, stop at 2026-10-23T08:19:28Z, wall 57 s
|
||||
430:✖ row 302 a run threadIdle released answers launcher-released on receive, state, read and send, logs each, and claims nothing again (81.984434ms)
|
||||
931:✖ failing tests:
|
||||
934:✖ row 302 a run threadIdle released answers launcher-released on receive, state, read and send, logs each, and claims nothing again (81.984434ms)
|
||||
935: AssertionError [ERR_ASSERTION]: nothing delivered to the released run
|
||||
exit 1
|
||||
@@ -0,0 +1,187 @@
|
||||
blobs: the five final test blobs (d5, launcher-released from d6) match in 5b63ea3236d82e14956594b41a00cbd7f7dc5f71 and 783602e06ed92f99c7c71d1e9c016a0f9e2c9b28
|
||||
merge: 783602e06ed92f99c7c71d1e9c016a0f9e2c9b28 = 5b63ea3236d82e14956594b41a00cbd7f7dc5f71 onto 1ac655005e611157605a0dcc5adf7a37f9bca2b0; 12 row paths equal to 5b63ea3236d82e14956594b41a00cbd7f7dc5f71, none touched in 0e30a85fa..1ac655005e611157605a0dcc5adf7a37f9bca2b0
|
||||
TIP 783602e06ed92f99c7c71d1e9c016a0f9e2c9b28 (merge) FIX 5b63ea3236d82e14956594b41a00cbd7f7dc5f71 RED f799e8b15561ea2c0f0735e5328ba149c7a1a7b5 base 0e30a85fa onto 1ac655005e611157605a0dcc5adf7a37f9bca2b0 origin/refactor 1ac655005e611157605a0dcc5adf7a37f9bca2b0 start 2026-10-23T08:23:41Z
|
||||
loadavg: 1.49 1.75 1.94 4/6320 1592551
|
||||
== 1. red first: f799e8b15561ea2c0f0735e5328ba149c7a1a7b5 product with the final test files from 5b63ea3236d82e14956594b41a00cbd7f7dc5f71 checked out on top
|
||||
red tree: product of f799e8b15561ea2c0f0735e5328ba149c7a1a7b5, tests of 5b63ea3236d82e14956594b41a00cbd7f7dc5f71: packages/bus/tests/launcher-released.test.mjs
|
||||
TMPDIR /mnt/storage/scratch/tmp/gHpj
|
||||
idle-stop-race rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
TMPDIR /mnt/storage/scratch/tmp/gIww
|
||||
runner-land rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
TMPDIR /mnt/storage/scratch/tmp/g8I5
|
||||
bus rc=1 ℹ tests 38 ℹ pass 33 ℹ fail 5 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
✖ failing tests:
|
||||
✖ landRelease releases the holding run by land-relaunch once it has asked message.receive and nothing delivered to it is unread, and waiting messages go to the next holder (37.255433ms)
|
||||
✖ REQ-THREAD-2 REQ-LAUNCH-1: threadIdle releases only a thread with nothing waiting, and only for its holder (30.900781ms)
|
||||
✖ row 302 a run landRelease released answers launcher-released, and claims nothing again (36.454968ms)
|
||||
✖ row 302 a run threadIdle released answers launcher-released on receive, state, read and send, logs each, and claims nothing again (40.167008ms)
|
||||
✖ row 302 with a run launcher-released, a run that lost its claim another way still answers not-holder (38.485853ms)
|
||||
17: AssertionError [ERR_ASSERTION]: runner: acme/assistant claimed by run r3f96e5142838
|
||||
21: + actual - expected
|
||||
32: actual: 'session stopped working: not-holder',
|
||||
33: expected: 'session released by the launcher',
|
||||
15: AssertionError [ERR_ASSERTION]: runner: demo/coder claimed by run coder-run
|
||||
35: actual: 22,
|
||||
36: expected: 0,
|
||||
pin red at threads.test.mjs:394:
|
||||
pin red at land.test.mjs:464:
|
||||
pin red at runner.test.mjs:771:
|
||||
:772 pattern absent from stderr (old not-holder line present)
|
||||
every red on f799e8b15561ea2c0f0735e5328ba149c7a1a7b5:
|
||||
✖ a run in a landing turn isn't released for a second held decision until its landing turns end and it has read its messages (604.606436ms)
|
||||
✖ failing tests:
|
||||
✖ landRelease releases the holding run by land-relaunch once it has asked message.receive and nothing delivered to it is unread, and waiting messages go to the next holder (37.255433ms)
|
||||
✖ REQ-THREAD-2 REQ-LAUNCH-1: threadIdle releases only a thread with nothing waiting, and only for its holder (30.900781ms)
|
||||
✖ row 302 a run landRelease released answers launcher-released, and claims nothing again (36.454968ms)
|
||||
✖ row 302 a run threadIdle released answers launcher-released on receive, state, read and send, logs each, and claims nothing again (40.167008ms)
|
||||
✖ row 302 REQ-THREAD-2 an idle-released run whose call lands before the SIGTERM ends stopped/0; a run whose claim the human revoked in the same window still ends session-invalid/22 (1604.296753ms)
|
||||
✖ row 302 with a run launcher-released, a run that lost its claim another way still answers not-holder (38.485853ms)
|
||||
red stage: both forcing cases and the four pins red
|
||||
wall 4 s
|
||||
== 2. green on 783602e06ed92f99c7c71d1e9c016a0f9e2c9b28
|
||||
TMPDIR /mnt/storage/scratch/tmp/gMtc
|
||||
idle-stop-race rc=0 ℹ tests 1 ℹ pass 1 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
wall 2 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gLSw
|
||||
runner rc=0 ℹ tests 40 ℹ pass 40 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
wall 12 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gTQ3
|
||||
compactions-launch rc=0 ℹ tests 5 ℹ pass 5 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
wall 14 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gAbg
|
||||
bus rc=0 ℹ tests 876 ℹ pass 876 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
wall 24 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gzOd
|
||||
harness rc=0 ℹ tests 151 ℹ pass 151 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
wall 13 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gxSF
|
||||
cli rc=0 ℹ tests 918 ℹ pass 918 ℹ fail 0 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
wall 615 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gL7z
|
||||
test-auth rc=0
|
||||
|
||||
selftest: 15 passed, 0 failed
|
||||
wall 1 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/ghcv
|
||||
test-conductor rc=0
|
||||
|
||||
selftest: 17 passed, 0 failed
|
||||
wall 6 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gCsn
|
||||
test-config rc=0
|
||||
|
||||
selftest: 55 passed, 0 failed
|
||||
wall 3 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/guh1
|
||||
test-discord rc=0
|
||||
|
||||
discord suite: 81 passed, 0 failed
|
||||
wall 7 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gVDP
|
||||
test-extension-package rc=0
|
||||
|
||||
extension package selftest: 18 passed, 0 failed
|
||||
wall 1 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/g1V8
|
||||
test-foundation rc=0
|
||||
|
||||
selftest: 44 passed, 0 failed
|
||||
wall 38 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gYsp
|
||||
test-land-work rc=0
|
||||
---
|
||||
land-work selftests: 86 passed, 0 failed
|
||||
wall 11 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gucV
|
||||
test-queue rc=0 ℹ tests 156 ℹ pass 156 ℹ fail 0
|
||||
|
||||
queue suite: 28 passed, 0 failed
|
||||
wall 21 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gGva
|
||||
test-release rc=0
|
||||
|
||||
selftest: 14 passed, 0 failed
|
||||
wall 3 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/g4vO
|
||||
test-task rc=0
|
||||
|
||||
selftest: 91 passed, 0 failed
|
||||
wall 16 s
|
||||
after tests: clean
|
||||
== skips
|
||||
suite/test-queue.log:skip queue verify and render --check: this checkout (/mnt/storage/scratch/gizmo-302) is not the queue's canonical root (/mnt/storage/src/mosaic-stack)
|
||||
suite/test-task.log:skip live provider cases (row 244: no provider call in the gate)
|
||||
suite/test-task.log:skip live provider cases (row 244: no provider call in the gate)
|
||||
== 3. mutants
|
||||
TMPDIR /mnt/storage/scratch/tmp/gDiY
|
||||
M1.launcher-released rc=1 ℹ tests 3 ℹ pass 1 ℹ fail 2 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 a run threadIdle released answers launcher-released
|
||||
TMPDIR /mnt/storage/scratch/tmp/glVj
|
||||
M1.idle-stop-race rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 REQ-THREAD-2 an idle-released run
|
||||
M1 KILLED by rule
|
||||
wall 2 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/grHO
|
||||
M2.land rc=1 ℹ tests 18 ℹ pass 17 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: landRelease releases the holding run by land-relaunch
|
||||
TMPDIR /mnt/storage/scratch/tmp/giID
|
||||
M2.launcher-released rc=1 ℹ tests 3 ℹ pass 0 ℹ fail 3 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 a run threadIdle released answers launcher-released
|
||||
killed: row 302 a run landRelease released answers launcher-released
|
||||
TMPDIR /mnt/storage/scratch/tmp/gXTo
|
||||
M2.idle-stop-race rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 REQ-THREAD-2 an idle-released run
|
||||
TMPDIR /mnt/storage/scratch/tmp/gTkJ
|
||||
M2.runner rc=1 ℹ tests 40 ℹ pass 39 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: a run in a landing turn isn't released for a second held decision
|
||||
M2 KILLED by rule
|
||||
wall 15 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/ge9B
|
||||
M3.idle-stop-race rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 REQ-THREAD-2 an idle-released run
|
||||
TMPDIR /mnt/storage/scratch/tmp/gYUi
|
||||
M3.runner rc=1 ℹ tests 40 ℹ pass 39 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: a run in a landing turn isn't released for a second held decision
|
||||
M3 KILLED by rule
|
||||
wall 33 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gw5X
|
||||
M4.idle-stop-race rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 REQ-THREAD-2 an idle-released run
|
||||
TMPDIR /mnt/storage/scratch/tmp/g90Y
|
||||
M4.runner rc=1 ℹ tests 40 ℹ pass 39 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: a run in a landing turn isn't released for a second held decision
|
||||
M4 KILLED by rule
|
||||
wall 14 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gNlw
|
||||
M5.launcher-released rc=1 ℹ tests 3 ℹ pass 1 ℹ fail 2 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 with a run launcher-released, a run that lost its claim
|
||||
TMPDIR /mnt/storage/scratch/tmp/gCk7
|
||||
M5.idle-stop-race rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 REQ-THREAD-2 an idle-released run
|
||||
M5 KILLED by rule
|
||||
wall 2 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gaHU
|
||||
M7.launcher-released rc=1 ℹ tests 3 ℹ pass 1 ℹ fail 2 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 a run threadIdle released answers launcher-released
|
||||
killed: row 302 a run landRelease released answers launcher-released
|
||||
M7 KILLED by rule
|
||||
wall 1 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/g3K9
|
||||
M8.land rc=1 ℹ tests 18 ℹ pass 17 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: landRelease releases the holding run by land-relaunch
|
||||
TMPDIR /mnt/storage/scratch/tmp/ge6z
|
||||
M8.launcher-released rc=1 ℹ tests 3 ℹ pass 2 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 a run landRelease released answers launcher-released
|
||||
TMPDIR /mnt/storage/scratch/tmp/gWPU
|
||||
M8.runner rc=1 ℹ tests 40 ℹ pass 39 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: a run in a landing turn isn't released for a second held decision
|
||||
M8 KILLED by rule
|
||||
wall 13 s
|
||||
TMPDIR /mnt/storage/scratch/tmp/gqwU
|
||||
M9.idle-stop-race rc=1 ℹ tests 1 ℹ pass 0 ℹ fail 1 ℹ cancelled 0 ℹ skipped 0 ℹ todo 0
|
||||
killed: row 302 REQ-THREAD-2 an idle-released run
|
||||
M9 KILLED by rule
|
||||
wall 2 s
|
||||
end 2026-10-23T08:38:14Z wall 873 s
|
||||
node with cwd in the worktree: 0
|
||||
exit 0
|
||||
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env python3
|
||||
# Row 302 (#1809) mutants, from BUILD.md d3 and d4. Each replaces one exact string, once, in one file.
|
||||
# Kill rule: every (test file, failing test name) listed fails in that file's run.
|
||||
# mutants.py list | apply M | file M | kills M
|
||||
import sys
|
||||
|
||||
BROKER = "packages/bus/src/broker.mjs"
|
||||
RUNNER = "packages/harness/src/runner.mjs"
|
||||
ADD = " this.#launcherReleased.add(key);\n"
|
||||
|
||||
IDLE = ("packages/cli/tests/idle-stop-race.test.mjs", "row 302 REQ-THREAD-2 an idle-released run")
|
||||
LR1 = ("packages/bus/tests/launcher-released.test.mjs", "row 302 a run threadIdle released answers launcher-released")
|
||||
LR2 = ("packages/bus/tests/launcher-released.test.mjs", "row 302 with a run launcher-released, a run that lost its claim")
|
||||
LR3 = ("packages/bus/tests/launcher-released.test.mjs", "row 302 a run landRelease released answers launcher-released")
|
||||
LAND = ("packages/bus/tests/land.test.mjs", "landRelease releases the holding run by land-relaunch")
|
||||
RUN = ("packages/harness/tests/runner.test.mjs", "a run in a landing turn isn't released for a second held decision")
|
||||
|
||||
M = {
|
||||
"M1": (BROKER, "this.#claimEnd(session, this.#own(session), 'release', null));\n" + ADD,
|
||||
"this.#claimEnd(session, this.#own(session), 'release', null));\n", [IDLE, LR1]),
|
||||
"M2": (BROKER, "fail(this.#launcherReleased.has(s.business + ':' + s.run) ? 'launcher-released' : 'not-holder');",
|
||||
"fail('not-holder');", [IDLE, LR1, LR3, LAND, RUN]),
|
||||
"M3": (RUNNER, " if (err.code === LAUNCHER_RELEASED) {", " if (false) {", [IDLE, RUN]),
|
||||
"M4": (RUNNER, " released = true;\n return true;", " released = true;\n code = EXIT.session;\n return true;", [IDLE, RUN]),
|
||||
"M5": (BROKER, "? 'launcher-released' : 'not-holder')", "? 'launcher-released' : 'launcher-released')", [IDLE, LR2]),
|
||||
"M7": (BROKER, " if (this.#launcherReleased.has(s.business + ':' + s.run)) fail('launcher-released');\n", "", [LR1, LR3]),
|
||||
"M8": (BROKER, "'land-relaunch'));\n" + ADD, "'land-relaunch'));\n", [LR3, LAND, RUN]),
|
||||
"M9": (RUNNER, "if (code === EXIT.stopped && !released) {", "if (code === EXIT.stopped) {", [IDLE]),
|
||||
}
|
||||
|
||||
cmd = sys.argv[1]
|
||||
if cmd == "list":
|
||||
print(" ".join(M))
|
||||
elif cmd == "file":
|
||||
print(M[sys.argv[2]][0])
|
||||
elif cmd == "kills":
|
||||
for f, name in M[sys.argv[2]][3]:
|
||||
print(f + "\t" + name)
|
||||
elif cmd == "apply":
|
||||
path, old, new, _ = M[sys.argv[2]]
|
||||
s = open(path).read()
|
||||
if s.count(old) != 1:
|
||||
sys.exit(f"{sys.argv[2]}: anchor found {s.count(old)} times in {path}")
|
||||
open(path, "w").write(s.replace(old, new))
|
||||
print(path)
|
||||
else:
|
||||
sys.exit("usage: mutants.py list|apply M|file M|kills M")
|
||||
@@ -0,0 +1,147 @@
|
||||
#!/bin/bash
|
||||
# Row 302 (#1809, Vikunja 588) gate. Plan: agents/gizmoduck/work/queue-302/BUILD.md d3 "Run plan", d4, d5.
|
||||
# 0. Preflight: worktree at TIP and clean, the five test blobs from d5. TIP is the local merge of 64f6558ee
|
||||
# onto origin/refactor 1ac655005 (sage-3289); the 12 row paths are unchanged in BASE..1ac655005.
|
||||
# 1. Red (mos-1415): f799e8b15 (final tests, unfixed product). idle-stop-race and the runner.test land case
|
||||
# must fail. The bus files are run and recorded. A green forcing case here is a STOP.
|
||||
# 2. On TIP: idle-stop-race alone 600 s, runner.test whole 900 s, compactions-launch whole 600 s,
|
||||
# bus 1200 s, harness 1200 s, cli 2100 s, then the scripts/test-*.sh at 900 s each.
|
||||
# 3. M1-M5, M7, M8, M9 (mutants.py). Each kill-listed file runs whole; every listed test name must fail.
|
||||
# After each: git checkout, sha256 equal to the candidate's, tree clean. A survivor is a STOP to Sage.
|
||||
# mos-1272: a first-run red in a new test is placed and reported after; any other red stops the run.
|
||||
set -o pipefail
|
||||
WT=/mnt/storage/scratch/gizmo-302
|
||||
# Run 2 (BUILD d6): FIX is the d6 tip; TIP is a fresh merge of it onto ONTO, set when the run is cleared.
|
||||
TIP=${TIP:?set TIP to the run-2 merge sha}
|
||||
FIX=5b63ea3236d82e14956594b41a00cbd7f7dc5f71
|
||||
ONTO=${ONTO:?set ONTO to origin/refactor at the merge}
|
||||
BR=gizmoduck/row302-gate
|
||||
RED=f799e8b15561ea2c0f0735e5328ba149c7a1a7b5
|
||||
BASE=0e30a85fa
|
||||
OUT=/mnt/storage/scratch/gizmo-302-out/gate
|
||||
PY=/mnt/storage/scratch/gizmo-302-out/mutants.py
|
||||
cd "$WT" || exit 9
|
||||
[ "$(git rev-parse HEAD)" = "$TIP" ] || { echo "not on the gate commit"; exit 9; }
|
||||
[ -z "$(git status --porcelain --untracked-files=no)" ] || { echo "dirty"; exit 9; }
|
||||
TESTS=""
|
||||
while read -r blob f; do
|
||||
[ "$(git rev-parse "$TIP:$f")" = "$blob" ] && [ "$(git rev-parse "$FIX:$f")" = "$blob" ] || { echo "blob mismatch $f"; exit 9; }
|
||||
TESTS="$TESTS $f"
|
||||
done <<'EOF'
|
||||
52343932b231691ba2b4fc117547ea7f6f504704 packages/bus/tests/land.test.mjs
|
||||
6fb6e15c2fd34f742f9aab718ff654ab2704e198 packages/bus/tests/launcher-released.test.mjs
|
||||
27193a745c4ed8d4936de96c1768ab010a572856 packages/bus/tests/threads.test.mjs
|
||||
9dfacfea15a487c4e7056ebae3e311cb514b0467 packages/cli/tests/idle-stop-race.test.mjs
|
||||
0792431b60314a17ed74b3df0c5b3748dc3c1286 packages/harness/tests/runner.test.mjs
|
||||
EOF
|
||||
echo "blobs: the five final test blobs (d5, launcher-released from d6) match in $FIX and $TIP"
|
||||
[ -L node_modules ] || ln -s /mnt/storage/src/mosaic-stack/node_modules node_modules
|
||||
[ -L node_modules ] || { echo "no node_modules link"; exit 8; }
|
||||
[ "$(git rev-parse $TIP^1) $(git rev-parse $TIP^2)" = "$ONTO $FIX" ] || { echo "merge parents wrong"; exit 9; }
|
||||
paths=$(git diff --name-only $BASE $FIX)
|
||||
[ "$(echo "$paths" | wc -l)" -eq 12 ] || { echo "row paths not 12"; exit 9; }
|
||||
over=$(git diff --name-only $BASE $ONTO -- $paths)
|
||||
[ -z "$over" ] || { echo "BASE..ONTO touches row paths:"; echo "$over"; exit 9; }
|
||||
git diff --quiet $FIX $TIP -- $paths || { echo "merge changed a row path"; exit 9; }
|
||||
echo "merge: $TIP = $FIX onto $ONTO; 12 row paths equal to $FIX, none touched in $BASE..$ONTO"
|
||||
[ -e "$OUT" ] && { echo "$OUT exists; move it first"; exit 9; }
|
||||
mkdir -p "$OUT/red" "$OUT/suite" "$OUT/mut"
|
||||
newtmp() { export TMPDIR=$(mktemp -d /mnt/storage/scratch/tmp/gXXX); echo " TMPDIR $TMPDIR"; }
|
||||
echo "TIP $TIP (merge) FIX $FIX RED $RED base $BASE onto $ONTO origin/refactor $(git rev-parse origin/refactor) start $(date -u +%FT%TZ)"; echo "loadavg: $(cat /proc/loadavg)"
|
||||
T0=$SECONDS; t0=$SECONDS; lap() { echo " wall $((SECONDS - t0)) s"; t0=$SECONDS; }
|
||||
counts() { grep -E '^ℹ (tests|pass|fail|cancelled|skipped|todo) ' "$1" | tr '\n' ' '; echo; }
|
||||
timed() { # log dir limit args... ; sets rc
|
||||
local log=$1 dir=$2 lim=$3; shift 3
|
||||
newtmp
|
||||
(cd "$dir" && timeout -k 10 "$lim" "$@") > "$log" 2>&1
|
||||
rc=$?
|
||||
if [ $rc -eq 124 ] || [ $rc -eq 137 ]; then
|
||||
echo "TIMEOUT after $lim s: $(basename "$log" .log). To Sage."
|
||||
case $(basename "$log") in test-*.log) echo "SHELL-PHASE TIMEOUT (mos-1275): to Mos unplaced";; esac
|
||||
fi
|
||||
echo "$(basename "$log" .log) rc=$rc $(counts "$log")"
|
||||
}
|
||||
green() { # log dir limit args... ; any red stops
|
||||
timed "$@"
|
||||
if [ $rc -ne 0 ]; then
|
||||
echo "RED: $(basename "$1" .log), stop at $(date -u +%FT%TZ), wall $((SECONDS - T0)) s"
|
||||
grep -nE '^not ok|✖|AssertionError|Error:' "$1" | head -40
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
echo "== 1. red first: $RED product with the final test files from $FIX checked out on top"
|
||||
git checkout -q --detach "$RED" || exit 9
|
||||
git checkout -q "$FIX" -- $TESTS || exit 9
|
||||
while read -r f; do [ "$(git hash-object "$f")" = "$(git rev-parse "$FIX:$f")" ] || { echo "red tree blob wrong $f"; exit 9; }; done < <(echo $TESTS | tr ' ' '\n')
|
||||
[ -z "$(git diff --name-only HEAD -- 'packages/*/src/*')" ] || exit 9
|
||||
echo " red tree: product of $RED, tests of $FIX: $(git diff --cached --name-only HEAD | tr '\n' ' ')"
|
||||
[ -z "$(git diff --name-only $BASE HEAD -- 'packages/*/src/*')" ] || { echo "red commit changes product files"; git checkout -q $BR; exit 9; }
|
||||
timed "$OUT/red/idle-stop-race.log" "$WT/packages/cli" 600 node --test tests/idle-stop-race.test.mjs; r1=$rc
|
||||
timed "$OUT/red/runner-land.log" "$WT/packages/harness" 600 node --test --test-name-pattern="^a run in a landing turn isn't released for a second held decision" tests/runner.test.mjs; r2=$rc
|
||||
timed "$OUT/red/bus.log" "$WT/packages/bus" 600 node --test tests/launcher-released.test.mjs tests/threads.test.mjs tests/land.test.mjs
|
||||
grep -E '✖ ' "$OUT/red/bus.log" | sort -u | sed 's/^/ /' | head -20
|
||||
git reset -q --hard "$RED" && git checkout -q $BR || exit 9
|
||||
[ "$(git rev-parse HEAD)" = "$TIP" ] && [ -z "$(git status --porcelain --untracked-files=no)" ] || { echo "not back on TIP clean"; exit 9; }
|
||||
red_ok=1
|
||||
[ $r1 -ne 0 ] && grep -qF "✖ row 302 REQ-THREAD-2 an idle-released run" "$OUT/red/idle-stop-race.log" || { echo " idle-stop-race NOT red on $RED"; red_ok=0; }
|
||||
[ $r2 -ne 0 ] && grep -qF "✖ a run in a landing turn isn't released for a second held decision" "$OUT/red/runner-land.log" || { echo " runner.test land case NOT red on $RED"; red_ok=0; }
|
||||
grep -nE 'AssertionError|actual|expected' "$OUT/red/idle-stop-race.log" | head -12 | sed 's/^/ /'
|
||||
grep -nE 'AssertionError|actual|expected' "$OUT/red/runner-land.log" | head -8 | sed 's/^/ /'
|
||||
# The four ruled pins (mos-1414/1415/1416). :772 sits after :771 in one test, so :771 stops it first;
|
||||
# :771's message is out.stderr, so the log shows the old "stopped working: not-holder" line and no
|
||||
# "released by the launcher" line, which is the line :772 now pins.
|
||||
grep -qF "✖ REQ-THREAD-2 REQ-LAUNCH-1: threadIdle releases only a thread with nothing waiting" "$OUT/red/bus.log" || { echo " threads.test threadIdle case NOT red"; red_ok=0; }
|
||||
grep -qF "✖ landRelease releases the holding run by land-relaunch" "$OUT/red/bus.log" || { echo " land.test landRelease case NOT red"; red_ok=0; }
|
||||
for pin in "threads.test.mjs:394:" "land.test.mjs:464:"; do
|
||||
grep -qF "$pin" "$OUT/red/bus.log" && echo " pin red at $pin" || { echo " pin $pin not in the failure stacks"; red_ok=0; }
|
||||
done
|
||||
grep -qF "runner.test.mjs:771:" "$OUT/red/runner-land.log" && echo " pin red at runner.test.mjs:771:" || { echo " pin runner.test.mjs:771 not in the failure stack"; red_ok=0; }
|
||||
grep -qF "session stopped working: not-holder" "$OUT/red/runner-land.log" && ! grep -qF "session released by the launcher" "$OUT/red/runner-land.log" \
|
||||
&& echo " :772 pattern absent from stderr (old not-holder line present)" || { echo " :772 evidence missing"; red_ok=0; }
|
||||
echo " every red on $RED:"; cat "$OUT"/red/*.log | grep -E '^✖ |^ ✖ |✖ ' | sed 's/^ *//' | sort -u | sed 's/^/ /'
|
||||
[ $red_ok -eq 1 ] && echo " red stage: both forcing cases and the four pins red" || { echo "STOP: red stage failed, to Sage"; exit 3; }
|
||||
lap
|
||||
|
||||
echo "== 2. green on $TIP"
|
||||
green "$OUT/suite/idle-stop-race.log" "$WT/packages/cli" 600 node --test tests/idle-stop-race.test.mjs; lap
|
||||
green "$OUT/suite/runner.log" "$WT/packages/harness" 900 node --test tests/runner.test.mjs; lap
|
||||
green "$OUT/suite/compactions-launch.log" "$WT/packages/cli" 600 node --test tests/compactions-launch.test.mjs; lap
|
||||
green "$OUT/suite/bus.log" "$WT/packages/bus" 1200 node --test tests/*.test.mjs; lap
|
||||
green "$OUT/suite/harness.log" "$WT/packages/harness" 1200 node --test tests/*.test.mjs; lap
|
||||
green "$OUT/suite/cli.log" "$WT/packages/cli" 2100 node --test tests/*.test.mjs; lap
|
||||
for s in "$WT"/scripts/test-*.sh; do
|
||||
n=$(basename "$s" .sh)
|
||||
green "$OUT/suite/$n.log" "$WT" 900 bash "$s"
|
||||
tail -2 "$OUT/suite/$n.log" | sed 's/^/ /'; lap
|
||||
done
|
||||
[ -z "$(git status --porcelain --untracked-files=no)" ] && echo "after tests: clean" || { echo "after tests: DIRTY"; exit 1; }
|
||||
echo "== skips"
|
||||
grep -HE '^ℹ skipped [1-9]|^ℹ todo [1-9]' "$OUT"/suite/*.log | sed "s#$OUT/##"
|
||||
grep -HiE '^(skip|# skip|SKIP)|skipped' "$OUT"/suite/test-*.log | grep -v '^ℹ' | sed "s#$OUT/##" | head -40
|
||||
|
||||
echo "== 3. mutants"
|
||||
limit() { case $1 in *cli/*) echo 600;; *runner.test*) echo 900;; *) echo 600;; esac; }
|
||||
for m in $(python3 -B "$PY" list); do
|
||||
src=$(python3 -B "$PY" file "$m")
|
||||
sum=$(git show "HEAD:$src" | sha256sum | cut -c1-64)
|
||||
python3 -B "$PY" apply "$m" >/dev/null || { echo "$m apply-failed"; git checkout -- "$src"; exit 8; }
|
||||
git diff > "$OUT/mut/$m.diff"
|
||||
ok=1
|
||||
for f in $(python3 -B "$PY" kills "$m" | cut -f1 | sort -u); do
|
||||
pkg=${f%%/tests/*}; rel=tests/${f#*/tests/}
|
||||
log="$OUT/mut/$m.$(basename "$f" .test.mjs).log"
|
||||
timed "$log" "$WT/$pkg" "$(limit "$f")" node --test "$rel"
|
||||
[ $rc -ne 0 ] || { echo " $m survived in $f"; ok=0; }
|
||||
while IFS=$'\t' read -r kf name; do
|
||||
[ "$kf" = "$f" ] || continue
|
||||
grep -qF "✖ $name" "$log" && echo " killed: $name" || { echo " NOT failing: $name"; ok=0; }
|
||||
done < <(python3 -B "$PY" kills "$m")
|
||||
done
|
||||
git checkout -- "$src"
|
||||
[ "$(sha256sum "$WT/$src" | cut -c1-64)" = "$sum" ] && [ -z "$(git status --porcelain --untracked-files=no)" ] || { echo "$m restore-failed"; exit 7; }
|
||||
if [ $ok -eq 1 ]; then echo "$m KILLED by rule"; else echo "STOP: $m kill rule not met, to Sage"; exit 4; fi
|
||||
lap
|
||||
done
|
||||
echo "end $(date -u +%FT%TZ) wall $((SECONDS - T0)) s"
|
||||
cd /; echo "node with cwd in the worktree: $(for p in $(pgrep -x node); do readlink /proc/$p/cwd; done 2>/dev/null | grep -c "$WT")"
|
||||
@@ -324,6 +324,14 @@ verb. It rechecks the thread and releases the run's claim in one call, and retur
|
||||
launcher's snapshot. The launcher signals the run only after a release, so a message sent
|
||||
after the snapshot gets a turn from the same run or waits as `pending` for the next one. It
|
||||
refuses `not-holder` for a run that doesn't hold that thread and `run-ended` for an ended run.
|
||||
After a release, and until the run ends, every call of the run's that needs the claim
|
||||
(`message.receive`, `state`, `message.read`, `message.send` and every other verb that checks
|
||||
the holder) refuses `launcher-released`, not `not-holder` (row 302). `role.claim` refuses it
|
||||
too, before the claim, so the run can't claim again. A call
|
||||
the broker answers after the release and before the signal is that stop, not a lost claim, so
|
||||
the runner exits 0. The launch-landing pass's `landRelease` (`packages/cli/README.md`) answers
|
||||
the same way for the role run it releases. Any other lost claim (a stop, a revoke, a clear, a
|
||||
limit, a newer holder) still refuses `not-holder`.
|
||||
|
||||
A message delivered to a thread run that then ended without reading it is stranded, as
|
||||
`stranded` (Agent state, below) describes for a role. It counts as neither `pending` nor
|
||||
|
||||
@@ -419,6 +419,10 @@ export class Broker {
|
||||
#hostRuns = new Map();
|
||||
// Row 125 (#1619, REQ-LIMIT-4): a run a limit stops, by business:run, until its next receive or state.
|
||||
#stops = new Map();
|
||||
// Row 302 (#1809): runs the launcher released, by business:run, until endLaunch. threadIdle and
|
||||
// landRelease release and then signal, so the run's own calls in between answer launcher-released,
|
||||
// not not-holder, and the runner exits stopped instead of as a lost session.
|
||||
#launcherReleased = new Set();
|
||||
#clock = 0;
|
||||
#processing;
|
||||
// credentialStatus is the runtime's metadata-only Credentials.status(); the broker never gets use().
|
||||
@@ -1003,7 +1007,8 @@ export class Broker {
|
||||
if (s.reader) fail('read-only');
|
||||
if (s.human) fail('agent-required');
|
||||
const h = this.#own(s);
|
||||
if (h?.op !== 'claim' || (s.thread ? h.generation !== this.#runGeneration(s) : h.holder_run !== s.run)) fail('not-holder');
|
||||
if (h?.op !== 'claim' || (s.thread ? h.generation !== this.#runGeneration(s) : h.holder_run !== s.run))
|
||||
fail(this.#launcherReleased.has(s.business + ':' + s.run) ? 'launcher-released' : 'not-holder');
|
||||
return h;
|
||||
}
|
||||
// Row 264a (#1769, T2): the generation a thread run holds, its latest claim's in its session, or
|
||||
@@ -1053,6 +1058,7 @@ export class Broker {
|
||||
if (t?.holder !== record.run) fail('not-holder');
|
||||
if (t.pending || t.unread) return { released: false, pending: t.pending, unread: t.unread };
|
||||
this.#store.transaction(() => this.#claimEnd(session, this.#own(session), 'release', null));
|
||||
this.#launcherReleased.add(key);
|
||||
return { released: true };
|
||||
}
|
||||
// Row 271 (264d, #1776, D7 steps 3-5): the business's clears still in the launcher's hands, as of
|
||||
@@ -1307,6 +1313,7 @@ export class Broker {
|
||||
const exitCode = record.exitCode ?? null;
|
||||
if (exitCode !== null && !Number.isSafeInteger(exitCode)) fail('invalid-request');
|
||||
this.#ended.add(key);
|
||||
this.#launcherReleased.delete(key);
|
||||
for (const [cap, s] of this.#sessions)
|
||||
if (!s.human && !s.reader && s.business === record.business && s.run === record.run) this.#sessions.delete(cap);
|
||||
return this.#store.transaction(() => {
|
||||
@@ -4161,6 +4168,7 @@ export class Broker {
|
||||
);
|
||||
if (unread) return { released: false, unread };
|
||||
this.#store.transaction(() => this.#claimEnd(session, h, 'release', null, null, 'land-relaunch'));
|
||||
this.#launcherReleased.add(key);
|
||||
return { released: true };
|
||||
}
|
||||
// Row 194 (#1692, D1.1): the launcher's lifecycle pass. Trusted launcher API, not socket verbs.
|
||||
@@ -5149,6 +5157,9 @@ export class Broker {
|
||||
case 'role.claim': {
|
||||
keys(a, []);
|
||||
if (s.human) fail('agent-required');
|
||||
// Row 302 (#1809, Dewey's D1): a run the launcher released is being stopped and doesn't claim
|
||||
// again, or a later not-holder on that claim would answer launcher-released.
|
||||
if (this.#launcherReleased.has(s.business + ':' + s.run)) fail('launcher-released');
|
||||
// Row 121: a thread run claims its thread, so idle threads hold nothing (REQ-THREAD-2).
|
||||
// Row 264a (#1769): the claim is the run's session's, with the session's next generation.
|
||||
if (s.thread) {
|
||||
|
||||
@@ -461,7 +461,7 @@ test('landRelease releases the holding run by land-relaunch once it has asked me
|
||||
{ op: 'release', holder_run: 'merge-gate-run', by: 'land-relaunch', decision: null, reason: null },
|
||||
);
|
||||
assert.throws(() => release(), /not-holder/, 'released once');
|
||||
assert.throws(() => w.call('merge-gate', 'message.receive'), /not-holder/);
|
||||
assert.throws(() => w.call('merge-gate', 'message.receive'), /launcher-released/);
|
||||
w.b.endLaunch({ business: 'demo', run: 'merge-gate-run', reason: 'exited' });
|
||||
assert.throws(() => release(), /run-ended/);
|
||||
assert.equal(w.store.get("SELECT count(*) AS n FROM deliveries WHERE message=? AND op='delivered'", waiting.id).n, 0, 'still waiting');
|
||||
|
||||
@@ -0,0 +1,151 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtempSync, readFileSync, rmSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { Store } from '../src/store.mjs';
|
||||
import { Broker } from '../src/broker.mjs';
|
||||
|
||||
// Row 302 (#1809, Vikunja 588): threadIdle and landRelease release a run's claim and the launcher
|
||||
// signals it after. The run's own calls in between answer launcher-released, so its runner exits
|
||||
// stopped. Every other way a run loses its claim still answers not-holder, and a released run
|
||||
// doesn't claim again (Dewey's D1 in dewey-0875).
|
||||
|
||||
const shipped = (name) =>
|
||||
JSON.parse(readFileSync(fileURLToPath(new URL(`../../../roles/${name}.json`, import.meta.url)), 'utf8'));
|
||||
|
||||
function setup(t, roles, definitions) {
|
||||
const root = mkdtempSync(join(tmpdir(), 'bus-launcher-released-'));
|
||||
const store = new Store(root);
|
||||
t.after(() => {
|
||||
store.close();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
const businesses = {
|
||||
demo: {
|
||||
id: 'demo',
|
||||
human: 'jason',
|
||||
arbiters: { technical: 'pm', delivery: 'pm' },
|
||||
roles,
|
||||
...(definitions ? { definitions } : {}),
|
||||
projects: { stack: { root: '/nonexistent' } },
|
||||
},
|
||||
};
|
||||
const b = new Broker({ store, businesses });
|
||||
const caps = { human: b.bindHuman({ business: 'demo', human: 'jason', via: 'cli', outsideAgent: true }) };
|
||||
const call = (who, verb, args = {}) => b.request(caps[who], { verb, args });
|
||||
const launch = (name, role, extra = {}) => {
|
||||
caps[name] = b.bindLaunch({ business: 'demo', role, run: `${name}-run`, harness: 'pi', ...extra });
|
||||
};
|
||||
const refused = (run) =>
|
||||
store.all("SELECT body FROM events WHERE kind='action.refused' AND actor_run=? ORDER BY seq", run).map((e) => JSON.parse(e.body).code);
|
||||
return { b, store, caps, call, launch, refused };
|
||||
}
|
||||
|
||||
// Two threads of the assistant, each with its own run holding it.
|
||||
function threads(t) {
|
||||
const w = setup(
|
||||
t,
|
||||
{
|
||||
chief: { definition: 'chief-of-staff', authority: { withinRole: ['message.send'], crossRole: [] } },
|
||||
assistant: { definition: 'assistant', authority: { withinRole: ['message.send'], crossRole: [] } },
|
||||
pm: { definition: 'pm', authority: { withinRole: ['message.send'], crossRole: [] } },
|
||||
},
|
||||
{ 'chief-of-staff': {}, assistant: { messaging: shipped('assistant').messaging }, pm: {} },
|
||||
);
|
||||
const project = w.call('human', 'project.create', { name: 'stack work', repo: 'stack' }).id;
|
||||
const A = w.call('human', 'thread.create', { project, name: 'alpha' }).id;
|
||||
const B = w.call('human', 'thread.create', { project, name: 'beta' }).id;
|
||||
for (const r of ['chief', 'pm']) {
|
||||
w.launch(r, r);
|
||||
w.call(r, 'role.claim');
|
||||
}
|
||||
w.launch('a', 'assistant', { thread: A });
|
||||
w.launch('b', 'assistant', { thread: B });
|
||||
w.call('a', 'role.claim');
|
||||
w.call('b', 'role.claim');
|
||||
const idle = (run, thread = A) => w.b.threadIdle({ business: 'demo', thread, run });
|
||||
return { ...w, A, B, idle };
|
||||
}
|
||||
|
||||
test('row 302 a run threadIdle released answers launcher-released on receive, state, read and send, logs each, and claims nothing again', (t) => {
|
||||
const { b, store, call, refused, idle, A } = threads(t);
|
||||
const sent = call('human', 'thread.send', { thread: A, body: 'for A' });
|
||||
assert.deepEqual(call('a', 'message.receive').map((m) => m.id), [sent.id]);
|
||||
call('a', 'message.read', { id: sent.id });
|
||||
assert.deepEqual(idle('a-run'), { released: true });
|
||||
const later = call('human', 'thread.send', { thread: A, body: 'later for A' });
|
||||
assert.throws(() => call('a', 'message.receive'), { code: 'launcher-released' });
|
||||
assert.throws(() => call('a', 'state'), { code: 'launcher-released' });
|
||||
assert.throws(() => call('a', 'message.read', { id: later.id }), { code: 'launcher-released' });
|
||||
// message.send needs the claim too, through #consumeAuthority (dewey-0878 C1).
|
||||
assert.throws(() => call('a', 'message.send', { to: 'chief', body: 'from a released run' }), { code: 'launcher-released' });
|
||||
// D1: no new claim for the released run, so a later not-holder can't be misreported.
|
||||
const claims = store.get('SELECT count(*) AS n FROM thread_claims').n;
|
||||
assert.throws(() => call('a', 'role.claim'), { code: 'launcher-released' });
|
||||
assert.equal(store.get('SELECT count(*) AS n FROM thread_claims').n, claims);
|
||||
assert.deepEqual(refused('a-run'), Array(5).fill('launcher-released'));
|
||||
// #route writes one routed row per message; with no holder its holder_run is null (broker #recipient).
|
||||
assert.equal(store.get('SELECT count(*) AS n FROM deliveries WHERE message=? AND holder_run=?', later.id, 'a-run').n, 0, 'nothing delivered to the released run');
|
||||
assert.deepEqual(store.all('SELECT op, holder_run FROM deliveries WHERE message=?', later.id).map((r) => ({ ...r })), [{ op: 'routed', holder_run: null }]);
|
||||
// threadIdle's own refusal is unchanged.
|
||||
assert.throws(() => idle('a-run'), { code: 'not-holder' });
|
||||
// After the end the run's caps are gone.
|
||||
b.endLaunch({ business: 'demo', run: 'a-run', reason: 'idle' });
|
||||
assert.throws(() => call('a', 'message.receive'), { code: 'unauthenticated' });
|
||||
});
|
||||
|
||||
test('row 302 with a run launcher-released, a run that lost its claim another way still answers not-holder', (t) => {
|
||||
const { call, launch, idle, A, B } = threads(t);
|
||||
assert.deepEqual(idle('a-run'), { released: true });
|
||||
// The human's stop revokes the other thread's holder.
|
||||
assert.deepEqual(call('human', 'assistant.stop', { instance: 'assistant' }).stopped, [{ thread: B, run: 'b-run' }]);
|
||||
assert.throws(() => call('b', 'message.receive'), { code: 'not-holder' });
|
||||
assert.throws(() => call('b', 'state'), { code: 'not-holder' });
|
||||
assert.throws(() => call('b', 'role.claim'), { code: 'run-revoked' });
|
||||
// A role run that released its own claim.
|
||||
call('pm', 'role.release');
|
||||
assert.throws(() => call('pm', 'message.receive'), { code: 'not-holder' });
|
||||
// A new run holds the released thread; the released run's answer stays its own.
|
||||
launch('a2', 'assistant', { thread: A });
|
||||
call('a2', 'role.claim');
|
||||
assert.throws(() => call('a', 'message.receive'), { code: 'launcher-released' });
|
||||
assert.deepEqual(call('a2', 'message.receive'), []);
|
||||
});
|
||||
|
||||
const MERGE = 'git.merge.protected';
|
||||
const TASK = 'vikunja:1/2';
|
||||
|
||||
test('row 302 a run landRelease released answers launcher-released, and claims nothing again', (t) => {
|
||||
const w = setup(t, {
|
||||
pm: { authority: { withinRole: ['message.send'], crossRole: [] } },
|
||||
coder: { authority: { withinRole: ['git.push.working', 'review.request', 'message.send'], crossRole: [] } },
|
||||
reviewer: { authority: { withinRole: ['review.verdict', 'message.send'], crossRole: [] } },
|
||||
'merge-gate': { authority: { withinRole: [], crossRole: [MERGE] } },
|
||||
});
|
||||
for (const role of ['pm', 'coder', 'reviewer', 'merge-gate']) {
|
||||
w.launch(role, role);
|
||||
w.call(role, 'role.claim');
|
||||
}
|
||||
const commit = 'a'.repeat(40);
|
||||
w.b.checkPush(w.caps.coder, TASK);
|
||||
w.b.recordPush(w.caps.coder, { task_ref: TASK, project: 'stack', branch: 'feat', ref: 'work/vikunja-1-2', commit, previous: null });
|
||||
const r = w.call('coder', 'review.request', { task_ref: TASK, commit, to: 'reviewer' });
|
||||
w.call('reviewer', 'review.verdict', { task_ref: TASK, request: r.request, verdict: 'approve', body: 'good' });
|
||||
const options = [{ key: 'yes', text: 'Land it' }, { key: 'no', text: 'Not yet' }];
|
||||
const d = w.call('merge-gate', 'decision.raise', { action: MERGE, target: TASK, question: 'Land?', options, recommendation: 'yes', blocking: false });
|
||||
w.call('pm', 'decision.resolve', { id: d.id, choice: 'yes' });
|
||||
const release = () => w.b.landRelease({ business: 'demo', run: 'merge-gate-run', decision: d.id });
|
||||
for (const m of w.call('merge-gate', 'message.receive')) w.call('merge-gate', 'message.read', { id: m.id });
|
||||
assert.deepEqual(release(), { released: true });
|
||||
assert.throws(() => w.call('merge-gate', 'message.receive'), { code: 'launcher-released' });
|
||||
assert.throws(() => w.call('merge-gate', 'state'), { code: 'launcher-released' });
|
||||
const claims = w.store.get('SELECT count(*) AS n FROM role_claims').n;
|
||||
assert.throws(() => w.call('merge-gate', 'role.claim'), { code: 'launcher-released' });
|
||||
assert.equal(w.store.get('SELECT count(*) AS n FROM role_claims').n, claims);
|
||||
// landRelease's own refusal is unchanged, and a role run that released itself still answers not-holder.
|
||||
assert.throws(() => release(), { code: 'not-holder' });
|
||||
w.call('coder', 'role.release');
|
||||
assert.throws(() => w.call('coder', 'message.receive'), { code: 'not-holder' });
|
||||
});
|
||||
@@ -391,7 +391,7 @@ test('REQ-THREAD-2 REQ-LAUNCH-1: threadIdle releases only a thread with nothing
|
||||
// Once released, the run can't take the thread's next message; it waits for the next holder.
|
||||
assert.throws(() => idle('a-run'), { code: 'not-holder' });
|
||||
const later = call('human', 'thread.send', { thread: A, body: 'later for A' });
|
||||
assert.throws(() => call('a', 'message.receive'), { code: 'not-holder' });
|
||||
assert.throws(() => call('a', 'message.receive'), { code: 'launcher-released' });
|
||||
assert.equal(state()[A].pending, 1);
|
||||
assert.equal(b.endLaunch({ business: 'demo', run: 'a-run', reason: 'idle' }).released, false);
|
||||
assert.throws(() => idle('a-run'), { code: 'run-ended' });
|
||||
|
||||
@@ -563,6 +563,8 @@ each holder that `launch.instances` lists:
|
||||
per run as `run <run>: landRelease refused: <code>`. After a release the
|
||||
launch log gets a `land-relaunch` record (`instance`, `run`,
|
||||
`decision`) and the runner gets SIGTERM, as `mosaic stop` sends it. The
|
||||
run's calls between the release and the signal refuse `launcher-released`,
|
||||
so the runner exits 0 either way (row 302). The
|
||||
next tick starts the instance with the decision in its `landing`.
|
||||
|
||||
The launcher keeps none of this on disk. A host that restarts ends the
|
||||
|
||||
@@ -0,0 +1,204 @@
|
||||
// Row 302 (#1809, Vikunja 588): the idle stop releases a thread run's claim in the threadIdle op
|
||||
// and sends SIGTERM only after the op returns. A runner call the broker answers in between used to
|
||||
// get not-holder, and the run ended session-invalid/22 instead of stopped/0. This test holds the op
|
||||
// open until a call has landed in that window and the runner has exited, so it doesn't depend on
|
||||
// timing. The launcher skips a tick while one is running (launcher.mjs, the thread pass), so nothing
|
||||
// else signals a run while the op is held.
|
||||
//
|
||||
// The host is in process, as in thread-launch.test.mjs, with the same fixture.
|
||||
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { existsSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { launchDoc } from "../../business/tests/helpers.mjs";
|
||||
import { BusError } from "../../bus/src/broker.mjs";
|
||||
import { startBroker } from "../../bus/src/runtime.mjs";
|
||||
import { Store } from "../../bus/src/store.mjs";
|
||||
import { gitEnv } from "../../seat/src/checkout.mjs";
|
||||
import { sandboxProblem } from "../../seat/src/sandbox.mjs";
|
||||
import { launchLogFile, SESSION_CREDENTIALS } from "../../seat/src/session.mjs";
|
||||
import { bootConfig, loadSystem } from "../src/config.mjs";
|
||||
import { createLauncher } from "../src/launcher.mjs";
|
||||
import { fakeHarnesses, fixture, tmp } from "./helpers.mjs";
|
||||
|
||||
const ADAPTER = fileURLToPath(new URL("./fixtures/thread-adapter.mjs", import.meta.url));
|
||||
const skip = sandboxProblem() ?? false;
|
||||
|
||||
async function until(fn, ms = 20000) {
|
||||
const end = Date.now() + ms;
|
||||
while (Date.now() < end) {
|
||||
const v = await fn();
|
||||
if (v) return v;
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
}
|
||||
throw new Error("timed out waiting");
|
||||
}
|
||||
|
||||
function gitIn(cwd, ...args) {
|
||||
const r = spawnSync("git", ["-c", "user.name=test", "-c", "[email protected]", "-C", cwd, ...args], { env: gitEnv(), encoding: "utf8" });
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
}
|
||||
|
||||
function prepare(t) {
|
||||
const root = tmp(t, "mosaic-idle-race-");
|
||||
const f = fixture(root, "acme", (doc) => {
|
||||
doc.roles.chief.vars = { harness: "pi", model: "claude-opus-5-5", thinking: "high" };
|
||||
doc.roles.assistant = { definition: "assistant", vars: { harness: "pi", model: "claude-sonnet-5-5", thinking: "high" } };
|
||||
doc.launch = launchDoc("chief", ["coder", "assistant"], { opus: 1, sonnet: 2 });
|
||||
doc.launch.threadIdleSeconds = 1;
|
||||
return doc;
|
||||
});
|
||||
for (const k of SESSION_CREDENTIALS.flat()) delete f.env[k];
|
||||
const project = f.doc.projects.stack.root;
|
||||
gitIn(project, "init", "-q", "-b", "main");
|
||||
writeFileSync(join(project, "file.txt"), "one\n");
|
||||
gitIn(project, "add", "file.txt");
|
||||
gitIn(project, "commit", "-q", "-m", "one");
|
||||
const adapter = join(root, "adapter.sh");
|
||||
writeFileSync(adapter, `exec '${process.execPath}' '${ADAPTER}'\n`, { mode: 0o644 });
|
||||
return { root, f, adapter };
|
||||
}
|
||||
|
||||
// thread-launch.test.mjs's start(), with threadIdle replaced by the caller.
|
||||
async function start(t, { f, adapter }, threadIdle) {
|
||||
const system = loadSystem({ env: f.env });
|
||||
const { trackers, ...boot } = bootConfig({ system, businessId: "acme", env: f.env });
|
||||
assert.equal(trackers, undefined);
|
||||
const runtime = await startBroker(boot);
|
||||
const ops = {
|
||||
identity: (m) => runtime.identity(m.cap),
|
||||
authorizeLaunch: (m) => runtime.authorizeLaunch(m.cap, m.instance),
|
||||
refuse: (m) => runtime.refuse(m.cap, m.code, m.subject),
|
||||
endLaunch: (m) => runtime.endLaunch(m.record),
|
||||
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
|
||||
threadsState: (m) => runtime.threadsState(m.business),
|
||||
threadIdle: (m) => threadIdle(runtime, m),
|
||||
mayLaunch: (m) => runtime.mayLaunch(m),
|
||||
landHeld: (m) => runtime.landHeld(m.record),
|
||||
landHolders: (m) => runtime.landHolders(m.business),
|
||||
landRelease: (m) => runtime.landRelease(m.record),
|
||||
};
|
||||
const refusing = (what, fn) => async (m) => {
|
||||
try {
|
||||
return await fn(m);
|
||||
} catch (e) {
|
||||
const code = e instanceof BusError ? e.code : `${what}-refused`;
|
||||
throw Object.assign(new Error(`${what} refused: ${code}`), { code });
|
||||
}
|
||||
};
|
||||
const hooks = [];
|
||||
let closing = null;
|
||||
const host = {
|
||||
business: "acme",
|
||||
path: runtime.path,
|
||||
dataRoot: f.dataRoot,
|
||||
op: refusing("launch-op", (m) => {
|
||||
if (!Object.hasOwn(ops, m.op)) throw new BusError("invalid-host-request");
|
||||
return ops[m.op](m);
|
||||
}),
|
||||
bindLaunch: refusing("launch bind", (r) => runtime.bindLaunch(r)),
|
||||
beforeClose: (fn) => void hooks.push(fn),
|
||||
};
|
||||
const launcher = createLauncher({
|
||||
host,
|
||||
system,
|
||||
env: f.env,
|
||||
makeLoader: await fakeHarnesses(t, { pi: adapter, "claude-code": adapter }),
|
||||
sessionDefaults: { pollInterval: 100 },
|
||||
harnessVersion: "0.85.1",
|
||||
readOnly: [ADAPTER],
|
||||
threadTickMs: 100,
|
||||
log: () => {},
|
||||
});
|
||||
const close = () =>
|
||||
(closing ??= (async () => {
|
||||
for (const fn of hooks) await fn(0);
|
||||
await runtime.close();
|
||||
})());
|
||||
t.after(close);
|
||||
await launcher.listen();
|
||||
const human = runtime.broker.bindHuman({ business: "acme", human: f.doc.human, via: "cli", outsideAgent: true });
|
||||
const call = (verb, args = {}) => runtime.broker.request(human, { verb, args });
|
||||
return { runtime, launcher, call, close };
|
||||
}
|
||||
|
||||
const launchLog = (f) =>
|
||||
existsSync(launchLogFile(f.dataRoot, "acme"))
|
||||
? readFileSync(launchLogFile(f.dataRoot, "acme"), "utf8").trim().split("\n").map((l) => JSON.parse(l))
|
||||
: [];
|
||||
const runnerLog = (f, run) => {
|
||||
const p = join(f.dataRoot, "launches", "acme", run, "runner.log");
|
||||
return existsSync(p) ? readFileSync(p, "utf8") : "";
|
||||
};
|
||||
const ended = (f, run) => launchLog(f).find((e) => e.event === "end" && e.run === run);
|
||||
// The runner's own verdict on a refusal: the launcher-release line, or a lost session's.
|
||||
const verdict = (f, run) => /runner: (session released by the launcher|session stopped working: [a-z-]+)\n/.exec(runnerLog(f, run))?.[1] ?? null;
|
||||
const settled = async (fn) => {
|
||||
try {
|
||||
return await until(fn);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
async function ask(call, thread, body) {
|
||||
const sent = call("thread.send", { thread, body });
|
||||
await until(() => call("thread.messages", { thread }).find((m) => m.in_reply_to === sent.id && m.class === "RESULT"));
|
||||
}
|
||||
|
||||
test("row 302 REQ-THREAD-2 an idle-released run whose call lands before the SIGTERM ends stopped/0; a run whose claim the human revoked in the same window still ends session-invalid/22", { skip, timeout: 120000 }, async (t) => {
|
||||
const prepared = prepare(t);
|
||||
const { f } = prepared;
|
||||
// The first run the idle stop releases. Inside its held window, the human stops the other
|
||||
// thread's run, which no launcher pass can signal while the op is held.
|
||||
let window = null;
|
||||
let host = null;
|
||||
const threadIdle = async (runtime, m) => {
|
||||
const r = runtime.threadIdle(m.record);
|
||||
if (!r.released || window) return r;
|
||||
window = { run: m.record.run, thread: m.record.thread };
|
||||
const other = runtime.threadsState("acme").find((x) => x.thread !== m.record.thread && x.holder);
|
||||
window.control = other ? { run: other.holder, stopped: host.call("assistant.stop", { instance: "assistant" }).stopped } : null;
|
||||
// Both runners decide on their own before the op returns and the launcher can signal.
|
||||
window.idle = await settled(() => verdict(f, window.run) && ended(f, window.run) && verdict(f, window.run));
|
||||
window.revoked = other ? await settled(() => verdict(f, other.holder) && ended(f, other.holder) && verdict(f, other.holder)) : null;
|
||||
return r;
|
||||
};
|
||||
host = await start(t, prepared, threadIdle);
|
||||
const project = host.call("project.create", { name: "stack work", repo: "stack" }).id;
|
||||
const A = host.call("thread.create", { project, name: "alpha" }).id;
|
||||
const B = host.call("thread.create", { project, name: "beta" }).id;
|
||||
await Promise.all([ask(host.call, A, "for A"), ask(host.call, B, "for B")]);
|
||||
// The op sets window.revoked last, so this waits for the whole window (dewey-0878 N2).
|
||||
await until(() => window && "revoked" in window, 60000);
|
||||
assert.ok(window.control, "the other thread's run held B when A's first release landed");
|
||||
await until(() => ended(f, window.run) && ended(f, window.control.run), 60000);
|
||||
|
||||
const { run, control } = window;
|
||||
assert.deepEqual(control.stopped.map((s) => s.run), [control.run], "the human's stop took only the other thread's run");
|
||||
// The released run refused in the window and exited by itself before the op returned.
|
||||
assert.equal(window.idle, "session released by the launcher", runnerLog(f, run));
|
||||
const log = launchLog(f);
|
||||
const end = (r) => log.find((e) => e.event === "end" && e.run === r);
|
||||
assert.deepEqual([end(run).reason, end(run).exitCode, end(run).signal], ["stopped", 0, null]);
|
||||
assert.ok(log.indexOf(end(run)) < log.findIndex((e) => e.event === "thread-idle" && e.run === run), "the end came before the idle stop's signal");
|
||||
assert.doesNotMatch(runnerLog(f, run), /session stopped working/);
|
||||
assert.match(runnerLog(f, run), /runner: exiting 0\n/);
|
||||
// The launcher already released the claim, so the runner doesn't release it at exit (dewey-0878 N1).
|
||||
assert.doesNotMatch(runnerLog(f, run), /runner: role\.release:/);
|
||||
// The control: a real not-holder in the same window still ends 22, with no signal.
|
||||
assert.equal(window.revoked, "session stopped working: not-holder", runnerLog(f, control.run));
|
||||
assert.deepEqual([end(control.run).reason, end(control.run).exitCode, end(control.run).signal], ["session-invalid", 22, null]);
|
||||
assert.match(runnerLog(f, control.run), /runner: exiting 22\n/);
|
||||
|
||||
await host.close();
|
||||
const store = new Store(f.dataRoot);
|
||||
t.after(() => store.close());
|
||||
const codes = (r) => store.all("SELECT body FROM events WHERE kind='action.refused' AND actor_run=? ORDER BY seq", r).map((e) => JSON.parse(e.body).code);
|
||||
assert.ok(codes(run).includes("launcher-released"), JSON.stringify(codes(run)));
|
||||
assert.ok(!codes(run).includes("not-holder"), JSON.stringify(codes(run)));
|
||||
assert.ok(codes(control.run).includes("not-holder"), JSON.stringify(codes(control.run)));
|
||||
});
|
||||
@@ -293,7 +293,7 @@ it never touches the disk and isn't in the environment or argv.
|
||||
|
||||
| Exit | Meaning | Launch end reason |
|
||||
|---|---|---|
|
||||
| 0 | stopped | `stopped` |
|
||||
| 0 | stopped: by a signal, or the launcher's idle stop or land-relaunch released the claim (`launcher-released`) | `stopped` |
|
||||
| 2 | bad run directory or stdin | `failed` |
|
||||
| 20 | founder credentials | `founder-credentials` |
|
||||
| 21 | `role.claim` refused | `claim-refused` |
|
||||
|
||||
@@ -53,6 +53,9 @@ export const FOUNDER_ENV = Object.freeze([
|
||||
const USABLE = new Set(["valid", "expiring", "rotation-due"]);
|
||||
// Capabilities that stopped working: the run ended, the claim was revoked.
|
||||
const SESSION_GONE = new Set(["unauthenticated", "not-holder", "run-ended"]);
|
||||
// Row 302 (#1809): the launcher's idle stop or land-relaunch released the claim and will stop the run.
|
||||
// That is a stop, not a lost session, so the run exits stopped even when its call lands before the signal.
|
||||
const LAUNCHER_RELEASED = "launcher-released";
|
||||
const BODY_MAX = 32000;
|
||||
const DEFAULTS = Object.freeze({ turnTimeout: 900, pollInterval: 2000, brokerRetries: 30 });
|
||||
|
||||
@@ -482,6 +485,20 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
|
||||
|
||||
let code = EXIT.stopped;
|
||||
let failures = 0;
|
||||
// Whether a broker refusal ends the run, and how: a launcher release stops it, a gone session is 22.
|
||||
// The reply site has logged its failure already, so it passes say=false.
|
||||
let released = false;
|
||||
const ended = (err, say = true) => {
|
||||
if (err.code === LAUNCHER_RELEASED) {
|
||||
log("runner: session released by the launcher");
|
||||
released = true;
|
||||
return true;
|
||||
}
|
||||
if (!SESSION_GONE.has(err.code)) return false;
|
||||
if (say) log(`runner: session stopped working: ${err.code}`);
|
||||
code = EXIT.session;
|
||||
return true;
|
||||
};
|
||||
let n = 0;
|
||||
const sleep = (ms) => new Promise((r) => {
|
||||
const t = setTimeout(r, ms);
|
||||
@@ -500,9 +517,8 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
|
||||
try {
|
||||
block = stateBlock(await client.call("state"), new Date().toISOString());
|
||||
} catch (err) {
|
||||
if (SESSION_GONE.has(err.code)) {
|
||||
log(`runner: session stopped working: ${err.code}`);
|
||||
code = EXIT.session;
|
||||
// A run in its landing turns hasn't received, so landRelease keeps it; thread runs have no landing.
|
||||
if (ended(err)) {
|
||||
gone = true;
|
||||
break;
|
||||
}
|
||||
@@ -524,11 +540,7 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
|
||||
messages = await client.call("message.receive");
|
||||
failures = 0;
|
||||
} catch (e) {
|
||||
if (SESSION_GONE.has(e.code)) {
|
||||
log(`runner: session stopped working: ${e.code}`);
|
||||
code = EXIT.session;
|
||||
break;
|
||||
}
|
||||
if (ended(e)) break;
|
||||
if (++failures >= session.brokerRetries) {
|
||||
log(`runner: broker unreachable after ${failures} tries: ${e.code ?? e.message}`);
|
||||
code = EXIT.broker;
|
||||
@@ -545,11 +557,8 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
|
||||
try {
|
||||
block = stateBlock(await client.call("state"), new Date().toISOString());
|
||||
} catch (e) {
|
||||
if (SESSION_GONE.has(e.code)) {
|
||||
log(`runner: session stopped working: ${e.code}`);
|
||||
code = EXIT.session;
|
||||
break loop;
|
||||
}
|
||||
// Neither release happens while a delivered message is unread, so this sees one only by a bug.
|
||||
if (ended(e)) break loop;
|
||||
unread = `could not read the current state: ${e.code ?? e.message}`;
|
||||
}
|
||||
++n;
|
||||
@@ -576,17 +585,15 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
|
||||
await client.call("message.read", { id: m.id });
|
||||
} catch (e) {
|
||||
log(`runner: reply to ${m.id} failed: ${e.code ?? e.message}`);
|
||||
if (SESSION_GONE.has(e.code)) {
|
||||
code = EXIT.session;
|
||||
break loop;
|
||||
}
|
||||
// As for the state above: the message isn't read yet, so no launcher release has happened.
|
||||
if (ended(e, false)) break loop;
|
||||
}
|
||||
}
|
||||
if (!messages.length) await sleep(session.pollInterval);
|
||||
}
|
||||
server.close();
|
||||
rmSync(session.toolSocket, { force: true });
|
||||
if (code === EXIT.stopped) {
|
||||
if (code === EXIT.stopped && !released) {
|
||||
try {
|
||||
await client.call("role.release");
|
||||
} catch (e) {
|
||||
|
||||
@@ -768,8 +768,8 @@ test("a run in a landing turn isn't released for a second held decision until it
|
||||
);
|
||||
assert.equal(ctx.store.get("SELECT count(*) AS n FROM deliveries WHERE holder_run='coder-run' AND op='read'").n, 2);
|
||||
const out = await r.exited;
|
||||
assert.equal(out.code, EXIT.session, out.stderr);
|
||||
assert.match(out.stderr, new RegExp(`runner: landing turn 1 for decision ${d1.id} \\(vikunja:1/2\\): ok\n[\\s\\S]*runner: session stopped working: not-holder\n`));
|
||||
assert.equal(out.code, 0, out.stderr);
|
||||
assert.match(out.stderr, new RegExp(`runner: landing turn 1 for decision ${d1.id} \\(vikunja:1/2\\): ok\n[\\s\\S]*runner: session released by the launcher\n`));
|
||||
});
|
||||
|
||||
test("a malformed landing exits 2 before the claim, with no turn", async (t) => {
|
||||
|
||||
Reference in new issue
Block a user