fix(bus): retry chief after processing resumes
This commit is contained in:
1 parent
13f81354e8
commit
26f1218b9c
4 files changed
+102
-2
No files matched your search
@@ -6299,3 +6299,5 @@ R5 quote fallback room (#797 D3, 2026-11-06, MVP3 lead): r4 review P2 found that
|
||||
|
||||
2026-11-06 UTC | #797 stall-guard integration merge | Before: stall-guard tip 6b2c925b9 and integration 5255e4e74 each supplied a Claude activity route; managed Pi also had two potential marker producers. After: BASE's Claude stream helper is authoritative, the Pi monitor is retained and its extension duplicate is disabled for managed turns, runner metadata suppression and newline filtering coexist, instance-running-only cooldown remains, and known/unknown child-signal receipts survive on both paths. Claude empty-delta handling follows BASE's selected helper; helper SHA pin updated. Node 26 final harness 309 pass, CLI main/lane/release 885/61/251 pass with one FED R7 skip, separately verified guard E2Es 11 pass, bus excluding project-files 1615 pass. Bus project-files had 5 pass then timeout/cancellation on both candidate and exact BASE; gate is not green. One earlier harness run failed the known Claude row-66 timing assertion (exact BASE full harness 277/277 and two file runs 14/14; older-base reproduction recorded by prior review); final rerun passed. An initial signal test invoked the removed Claude monitor and was adapted. Queue refused linked worktree; canonical queue read returned nothing. Local merge candidate only, no push or deployment. Full counts, caveats and evidence: /mnt/storage/scratch/mosaic-lead/mvp3/stall-guard-int-report.md and stall-guard-int-evidence/.
|
||||
2026-11-06 UTC | #797 supervisor schema guard implementation | Before: the 3s-only accounting guard refused the 3t archive store. After: the lead-authorized closed 3s/3t fallback preserves 3s compatibility and accepts 3t; store.mjs performs load-time file reads and in-memory schema/migration execution, so it is not imported into the supervisor. The real-Store current-version regression failed on BASE with test-store-schema-refused; synthetic 3s accepts, 3r/3u/missing refuse. DDL diff adds only project_archives objects. README documents owner diagnosis and restoration or campaign end without reset. Sequential Node 26 whole-file gates: budget 102/0, isolated identity 8/0, browser 13/0, lifecycle 2/0; no skips. Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/supervisor-3t-evidence/; local candidate for independent review, no push or deployment.
|
||||
|
||||
2026-11-06 UTC | #797 resume chief implementation | Before: halted and launches-disabled receipts counted as the chief's launch attempt, so a resting request could not relaunch after resume at an unchanged prerequisite. After: continuation lookup excludes only those host-gate receipts while preserving visible refusals and earlier real attempt failures. Three real-bus/launcher regressions fail behaviorally on 13f81354e and pass with the fix, including bounded repeated ticks and a launch-failed retry guard. Node 26 touched file 12 pass; bus excluding project-files 1642 pass; serial CLI 1207 pass, 1 base-reproduced FED R7 skip. Initial concurrent CLI 1199 pass/8 fail/1 skip: four timing cases passed serially on base; four Claude wrapper failures reproduced on base and passed with the direct installed binary. Evidence: /mnt/storage/scratch/mosaic-lead/mvp3/resume-chief-evidence/; report: /mnt/storage/scratch/mosaic-lead/mvp3/resume-chief-report.md. One local commit for independent review; no push, merge, deployment or live host action.
|
||||
@@ -2096,3 +2096,4 @@ are never rewritten or removed; corrections are new entries.
|
||||
2026-11-06 UTC | ui797 stall-guard review-fix implementation sub-agent | #797 three corrections on feat/ui797-stall-guard-r2 from 3e754c96a | Adapter stdout parity, signal-specific child-death receipts, instance-running-only cooldown; Node 26 gates and negative controls recorded in /mnt/storage/scratch/mosaic-lead/mvp3/stall-guard-r2-report.md. Local commits only; queue refused linked worktree; independent review pending. No push, merge, deployment, live-host contact, model call, secret, role, policy, contract or schema changes.
|
||||
2026-11-06 UTC | ui797 stall-guard integration implementation sub-agent | #797 integration of 5255e4e74 into 6b2c925b9 | Reconciled Claude/Pi activity sources, retained cooldown and signal receipts; final Node 26 harness 309 pass, CLI 1197 pass/1 base-reproduced skip, six guard E2Es within 11 pass, bus 1615 pass plus project-files 5 pass/1 timeout also on exact BASE; earlier row-66 timing failure and all comparisons retained. Local merge candidate for independent review; no push, deployment, real model call or protected-host contact. Report: /mnt/storage/scratch/mosaic-lead/mvp3/stall-guard-int-report.md.
|
||||
2026-11-06 UTC | supervisor-3t implementation sub-agent | #797 closed 3s/3t supervisor guard and recovery guidance above 37f38d8e1 | Real-store regression red on BASE; sequential whole-file gates 125 pass, 0 fail/skip, evidence in /mnt/storage/scratch/mosaic-lead/mvp3/supervisor-3t-evidence/. Queue refused linked worktree; one local commit for independent review, no push, deployment or live access.
|
||||
2026-11-06 UTC | resume-chief implementation sub-agent | #797 F-resume-chief on fix/mvp3-resume-chief from 13f81354e | Host-gate receipts no longer consume chief attempts; three behavioral base controls, touched file 12 pass, bus 1642 pass, serial CLI 1207 pass/1 base-reproduced skip. Initial concurrent timing and Claude-wrapper failures, baseline comparisons, and report retained under /mnt/storage/scratch/mosaic-lead/mvp3/. Queue refused linked worktree; local commit only, independent review pending; no push, merge or deployment.
|
||||
@@ -222,7 +222,9 @@ export function createContinuations({ journal, services }) {
|
||||
const intent = row.result.intent;
|
||||
if (intent.revision !== state.revision) continue;
|
||||
const context = contextFor(state, intent.target);
|
||||
const lastChief = effects(state).findLast(e => e.stage === 'committed' && e.result?.type === 'launch' && e.result.key === intent.key && e.result.instance === context.chief);
|
||||
// A host-wide gate records a visible refusal, but never attempted the chief.
|
||||
// Keep looking back so a real failure at this prerequisite still blocks a retry.
|
||||
const lastChief = effects(state).findLast(e => e.stage === 'committed' && e.result?.type === 'launch' && e.result.key === intent.key && e.result.instance === context.chief && !['processing-halted', 'launches-disabled'].includes(e.result.code));
|
||||
out.push({ ...intent, ...context, chief_attempt: lastChief?.result?.prerequisite ?? null });
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { readFileSync, writeFileSync } from 'node:fs';
|
||||
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { fixture, tmp, withoutHostCredentials, fakeHarnesses } from './helpers.mjs';
|
||||
@@ -199,3 +199,98 @@ test('archived product mappings stay out of new manifests while the business sti
|
||||
assert.deepEqual(JSON.parse(readFileSync(join(session.runDir, 'bundle/manifest.json'))).projects, ['stack']);
|
||||
await launcher.close();
|
||||
});
|
||||
|
||||
// A resting request has an outstanding PM continuation and no running chief. These
|
||||
// tests use the same broker throughout; only the launcher is replaced for --no-launch.
|
||||
async function restingRequest(t, { launches = true, brokenHolders = false } = {}) {
|
||||
let launcher;
|
||||
// Close the launcher before requestFixture closes its bus and removes its root.
|
||||
t.after(async () => { if (launcher) await launcher.close(); });
|
||||
const { requestFixture } = await import('../../bus/tests/request-fixture.mjs');
|
||||
const f = await requestFixture(t);
|
||||
withoutHostCredentials(f.f.env, { piKey: true });
|
||||
f.bus.broker.endLaunch({ business: 'demo', run: 'chief-run', reason: 'exit', exitCode: 0 });
|
||||
await f.bus.requestLifecycle.recover();
|
||||
const fake = fileURLToPath(new URL('../../harness/tests/fixtures/fake-adapter.mjs', import.meta.url));
|
||||
const adapter = join(f.f.dataRoot, 'resume-chief.sh');
|
||||
writeFileSync(adapter, `exec '${process.execPath}' '${fake}'\n`);
|
||||
// The real bus host creates this before constructing its read-only launcher.
|
||||
mkdirSync(join(f.f.dataRoot, 'bus-host'), { recursive: true, mode: 0o700 });
|
||||
const makeLoader = await fakeHarnesses(t, { pi: adapter, 'claude-code': adapter });
|
||||
const host = requestHost({ bus: f.bus, f: f.f, caps: f.caps });
|
||||
const op = host.op;
|
||||
let chiefChecks = 0, inChiefLaunch = false;
|
||||
host.op = m => {
|
||||
if (m.op === 'request.chief') { chiefChecks++; inChiefLaunch = true; }
|
||||
if (brokenHolders && inChiefLaunch && m.op === 'landHolders') return Promise.resolve({ malformed: true });
|
||||
return op(m);
|
||||
};
|
||||
async function start(enabled) {
|
||||
launcher = createLauncher({ host, system: f.system, env: f.f.env, launches: enabled, makeLoader,
|
||||
harnessVersion: 'fixture', readOnly: [fake], threadTickMs: 3600000, sessionDefaults: { pollInterval: 25 } });
|
||||
await launcher.listen();
|
||||
return launcher;
|
||||
}
|
||||
await start(launches);
|
||||
return { f, host, start, get launcher() { return launcher; }, chiefChecks: () => chiefChecks,
|
||||
async replace(enabled = true) { await launcher.close(); launcher = null; return start(enabled); } };
|
||||
}
|
||||
|
||||
async function expectQuietTicks(x, code) {
|
||||
await x.launcher.tick();
|
||||
const { f } = x;
|
||||
assert.equal((await f.call('human', 'request.get', { request: f.request })).launch_refusal.code, code);
|
||||
const effects = f.journal.effects('demo', f.request).length;
|
||||
const messages = f.store.get('SELECT count(*) n FROM messages').n;
|
||||
for (let i = 0; i < 5; i++) await x.launcher.tick();
|
||||
assert.equal(x.launcher.registry.sessions.length, 0, 'ticks do not launch');
|
||||
assert.equal(x.chiefChecks(), 0, 'ticks never enter chief launch');
|
||||
assert.equal(f.journal.effects('demo', f.request).length, effects, 'identical refusal receipts are deduplicated');
|
||||
assert.equal(f.store.get('SELECT count(*) n FROM messages').n, messages, 'ticks do not flood messages');
|
||||
}
|
||||
|
||||
test('a resting chief launches on a later tick after processing resumes', { timeout: 30000 }, async t => {
|
||||
const x = await restingRequest(t), { f } = x;
|
||||
const halted = await f.call('human', 'processing.halt');
|
||||
await expectQuietTicks(x, 'processing-halted');
|
||||
await f.call('human', 'processing.resume', { generation: halted.generation });
|
||||
await x.launcher.tick();
|
||||
assert.equal(x.launcher.registry.sessions.length, 1, 'resume must launch without restarting the bus');
|
||||
assert.equal(x.launcher.registry.sessions[0].instance, 'chief');
|
||||
assert.equal((await f.call('human', 'request.get', { request: f.request })).launch_refusal, null);
|
||||
});
|
||||
|
||||
test('a no-launch host remains quiet and an enabled launcher can subsequently start the chief', { timeout: 30000 }, async t => {
|
||||
const x = await restingRequest(t, { launches: false });
|
||||
await expectQuietTicks(x, 'launches-disabled');
|
||||
await x.replace();
|
||||
await x.launcher.tick();
|
||||
assert.equal(x.launcher.registry.sessions.length, 1, 'enabling launches must not require a bus restart');
|
||||
assert.equal(x.launcher.registry.sessions[0].instance, 'chief');
|
||||
assert.equal((await x.f.call('human', 'request.get', { request: x.f.request })).launch_refusal, null);
|
||||
});
|
||||
|
||||
test('a real launch failure after resume consumes the attempt at the unchanged prerequisite', { timeout: 30000 }, async t => {
|
||||
const x = await restingRequest(t, { brokenHolders: true }), { f } = x;
|
||||
const halted = await f.call('human', 'processing.halt');
|
||||
await expectQuietTicks(x, 'processing-halted');
|
||||
await f.call('human', 'processing.resume', { generation: halted.generation });
|
||||
await x.launcher.tick();
|
||||
assert.equal((await f.call('human', 'request.get', { request: f.request })).launch_refusal.code, 'launch-failed');
|
||||
assert.equal(x.chiefChecks(), 1, 'the chief launch was attempted once');
|
||||
const prerequisite = f.bus.requestLaunches('demo')[0].prerequisite;
|
||||
const haltedAgain = await f.call('human', 'processing.halt');
|
||||
await x.launcher.tick();
|
||||
assert.equal((await f.call('human', 'request.get', { request: f.request })).launch_refusal.code, 'launch-failed', 'the identical earlier halt receipt is deduplicated');
|
||||
await f.call('human', 'processing.resume', { generation: haltedAgain.generation });
|
||||
await x.replace(false);
|
||||
await x.launcher.tick();
|
||||
assert.equal((await f.call('human', 'request.get', { request: f.request })).launch_refusal.code, 'launches-disabled');
|
||||
await x.replace();
|
||||
assert.equal(f.bus.requestLaunches('demo')[0].prerequisite, prerequisite);
|
||||
const effects = f.journal.effects('demo', f.request).length;
|
||||
for (let i = 0; i < 5; i++) await x.launcher.tick();
|
||||
assert.equal(x.chiefChecks(), 1, 'a real failure does not retry at the same prerequisite');
|
||||
assert.equal(f.journal.effects('demo', f.request).length, effects);
|
||||
assert.equal(x.launcher.registry.sessions.length, 0);
|
||||
});
|
||||
Reference in new issue
Block a user