Compare commits

..
Author SHA1 Message Date
jason.woltje a7362a8fc5 test(webui): tasks-board: the fixture task is due in 2099, so the card still reads due, not overdue, after 20 Nov 2026 passes; the expected label carries the year (#1873, lane chat-fail) 2026-11-19 18:20:43 -06:00
jason.woltje 5a07f9e1f9 test(bus): chat-fail pickup-launch: the fixture runs its Claude model on claude-code, since a legacy Claude entry counts as OAuth and runs there only; the host launcher's env carries a stand-in session login, because claude-code's preflight needs one and this launcher boots with no proxy, and a stand-in adapter path skips only the binary check, so each launch still stops at the model gate with model-provider-unavailable and no assertion changed (#1873, lane chat-fail) 2026-11-19 17:08:06 -06:00
jason.woltje 07eca4a670 fix(chat-fail): review round 3: the link, email and mixed-run redaction patterns start only where a run without spaces starts, so redacting the whole kept stdout stays linear (a 128000-character run took 10 to 15 s, now a few ms) and matches the same text; a long text that redacts below the read limit is not cut and keeps its first word, and a cause that is only [redacted] is dropped; the proxy page reads relogin, plural and past forms of an expired sign-in as a sign-in (#1873, lane chat-fail) 2026-11-19 16:36:16 -06:00
jason.woltje fc9e4dfca8 fix(chat-fail): review round 2: the F1 cause is redacted over the whole text before the tail cut, so a cut that lands between a key word and its value no longer shows the value; the account sign-in words in the proxy page match whole words only ("assigning" and "logging" no longer read as a sign-in) and a prefixed or "is expired" token counts (#1873, lane chat-fail) 2026-11-19 16:36:16 -06:00
jason.woltje 10a35ce65a fix(chat-fail): review round 1: the F1 cause decodes %XX, HTML references and NFKC forms before the secret and redaction checks, redacts the value after a quoted or space-separated key word and Windows and UNC paths, and is read from the tail of the output, with the cause on the retry log line too; a launch-failed start posts only when the same message is refused twice, project-pending and processing-halted are waits, and a refused post is tried again and logged once; the proxy reason refuses an impossible date, checks quota_blocked before the message words and no longer reads a bare "expired" as a sign-in (#1873, lane chat-fail) 2026-11-19 16:36:16 -06:00
jason.woltje 3b9cac5452 fix(bus): chat-fail F3: an account the proxy reports as unavailable or in error carries one plain reason with the next step, built only from the fields of the existing account read (requires_sign_in, refresh_error, quota_blocked, next_retry_after, and status_message matched against fixed patterns and never shown); with nothing more it says the model service gave no reason. No new proxy call; working, turned-off and unknown rows keep their exact fields (#1873, lane chat-fail) 2026-11-19 16:36:16 -06:00
jason.woltje bc6d3565b0 fix(bus): chat-fail F2: a refused thread start and a proxy-stall stop each put one plain line in the chat, with the cause and next step, once per unanswered message and not once per retry; a start that only waits (a free place, a run still ending, a held lock, a lost race) puts none. The line comes from the launcher through a trusted launch op, not a socket verb, from the thread's instance with from_run bus, so no schema change and no new refusal code (#1873, lane chat-fail) 2026-11-19 16:36:16 -06:00
jason.woltje 72ea7272ec fix(harness): chat-fail F1: the turn-failure code moves into the mounted harness failure-notice.mjs, so a sandboxed runner loads it; the separate file was outside RUNNER_FILES and every sandboxed run died with ERR_MODULE_NOT_FOUND (#1873, lane chat-fail) 2026-11-19 16:36:16 -06:00
jason.woltje 27a359590e fix(harness): chat-fail F1: a failed turn's chat line carries the harness's error as a plain cause and next step, cleaned, redacted and capped, with the technical reason under it (#1873, lane chat-fail) 2026-11-19 16:36:16 -06:00
jason.woltje 6c5ec23b47 test(model-proxy): settings-p3 ride-fence: the launch fixture runs its Claude model on claude-code, since a legacy Claude entry counts as OAuth and runs there only; a stand-in adapter path skips only the binary check, so the before launch still reaches the limit checks and the fenced launch after Ride all is still refused with proxy-uncertain (#1873, lane settings-p3) 2026-11-19 15:44:31 -06:00
jason.woltje dd0a8612c8 fix(webui, model-proxy): settings-p3 review P2-1, P3-1 to P3-3: the done line counts an organization that can still ride, so an owner who can't send is not told every organization can see the models; an organization that took cliproxy during a Ride all run reads as already using it, not refused; the stopped page's refusal repaint is tested before the page renders again; a stand-in organization with no bus status (the single-business Console) reads that moving is not available there, with no Move button, from the status summary as well as after a run (#1873, lane settings-p3) 2026-11-19 15:16:59 -06:00
jason.woltje 9276277970 test(webui): settings-p3 shots: the Model setup shot shows the Ride all box below the restart list, with one organization that can ride, one skipped and one on stand-in models; a full-page shot grows the window to the page so the fixed footer covers nothing (#1873, lane settings-p3) 2026-11-19 15:04:56 -06:00
jason.woltje 517a3fe467 fix(webui): settings-p3 P3-1 to P3-11, rebase re-check 1 to 4, Routing help: Model setup says what to do for a held init lock or a partial init, with the README path under Technical details, and both lines are tested; the triage-drawer fixture case says its pair is synthetic; a model-key collision names a step; a fixture refusal leaves Restart on the stopped page but not primary; a ready catalogue the bus picked up late keeps its organization on Model setup's restart list with a Restart, holds the next step, and gets a Restart beside its notice on Routing; the README names the Model setup page; the NO_GUESS pin counts both uses; a reason that is not a stop reads "Last check:"; the why line sits under the headline; orgDownReason tells every waiting caller, each with its own name; the done line does not say every organization while the Ride all box lists one; Ride all is the next step only when the owner can send; the step button reads "Go to Ride all"; Ride all text says model service; the single-business fixture move says nothing about a bus it can't see; the Routing help names no harness (#1873, lane settings-p3) 2026-11-19 15:04:56 -06:00
jason.woltje 149712816b fix(model-proxy): settings-p3 R1, R2: Ride all reads each organization's file and change log again before its add, not only the receipts, so an add and remove saved on its Settings page during a run is not put back; a list that can't be read again, a change log that can't be read or an organization no longer listed skips the add as unreadable; with no bus status (the single-business Console) a fixture move answers not-available instead of bus-running, and whether its bus runs is unknown (#1873, lane settings-p3) 2026-11-19 15:04:56 -06:00
46 changed files with 1157 additions and 93 deletions

No files matched your search

+111
View File
@@ -554,6 +554,62 @@ const INSTALLER_COMMENT = 'mosaic-installer-';
// business ID_PATTERN, which starts with [a-z0-9], so no instance can be named this.
const NOTICE_ROLE = '_bus';
const NOTICE_RUN = 'bus';
// #1873 chat-fail F2: the one plain line a refused thread start, or a run the model proxy stopped
// for silence (proxy-stall), leaves in the conversation for each message it left unanswered
// (failureNotice below). It is here, not in a file of its own, because the runner imports this
// file in the session sandbox, which mounts only RUNNER_FILES (packages/seat/src/sandbox.mjs).
//
// The line has the form a failed turn's reply has (packages/harness/src/failure-notice.mjs), so the
// chat renders it the same way: "The turn failed: <cause and next step>\nDetails: <code>". The
// text is fixed per code. A refusal's own reason is never put in it: it can name paths, models
// and accounts. Codes are the launcher's existing refusal codes and the existing proxy-stall stop
// reason; none is added here.
const MODEL = "the model this agent uses isn't available from the model service now. Pick another model, or check Settings > Model setup.";
const PROXY = "the model proxy isn't ready. Check Settings > Model setup.";
const ACCOUNT = "the agent's model account isn't signed in or set up. Sign in again, or check the account in Settings > Model setup.";
const LIMIT = "the agent reached a usage limit. Wait for it to reset, or check the limits in Settings.";
const SETUP = "the agent's setup can't start now. Check the agent in Agents.";
const OTHER = "the assistant couldn't start. Try again later, or check the agent in Agents.";
const LINES = new Map([
['proxy-stall', "the model service gave no answer in time, so the turn was stopped. Try again, or check the account in Settings > Model setup."],
['model-provider-unavailable', MODEL],
['model-catalogue-unavailable', MODEL],
['model-harness-incompatible', MODEL],
['proxy-uncertain', PROXY],
['no-session-credential', PROXY],
['account-credential-missing', ACCOUNT],
['account-credential-ambiguous', ACCOUNT],
['account-is-credential', ACCOUNT],
['credential-unavailable', ACCOUNT],
['limit-reached', LIMIT],
['limit-check-failed', LIMIT],
['harness-off', SETUP],
['harness-refused', SETUP],
['harness-failed', SETUP],
['unsupported-harness', SETUP],
['thinking-unsupported', SETUP],
['plugin-disabled', SETUP],
['skill-not-enabled', SETUP],
['skill-refused', SETUP],
['no-role-contract', SETUP],
['definition-holds-credential', SETUP],
['project-archived', "this project is archived, so the assistant can't reply here."],
]);
// A start that waits for a free place, a run that is still ending, a held lock, a lost race, a
// project still being set up or an operator's pause starts by itself once that clears: no line.
const WAITS = new Set(['capacity-full', 'instance-running', 'legacy-run-live', 'session-workspace-busy',
'processing-generation-changed', 'proxy-lock-busy', 'project-pending', 'processing-halted']);
const NOTICE_CODE = /^[a-z][a-z-]{0,63}$/;
// The body for a code, or null when the code leaves no line.
function failureNoticeBody(code) {
if (typeof code !== 'string' || !NOTICE_CODE.test(code) || WAITS.has(code)) return null;
const line = LINES.get(code) ?? (code.startsWith('proxy-') ? PROXY : OTHER);
return `The turn failed: ${line}\nDetails: ${code === 'proxy-stall' ? 'stopped' : 'start refused'}: ${code}`;
}
// Row 143 (#1637, REQ-PROJ-4): a human save of this file through project.file.save sends one notice to each
// instance of these definitions. Their answer goes to the instance of SPEC_ANSWER_TO, the chief of staff,
// because only the user and the chief of staff may send to the assistant (REQ-ASSIST-2).
@@ -2006,6 +2062,61 @@ export class Broker {
this.#launcherReleased.add(key);
return { released: true };
}
// #1873 chat-fail F2: a thread start the launcher refused, or a run the model proxy stopped for
// silence (proxy-stall), leaves the person who wrote no answer. This puts one plain line in the
// conversation for each message from the human, a user or the tester it left unanswered, as a
// RESULT in reply to it, the way #landMessage reports a refused landing. A message in a thread is
// to or from the thread's instance (trigger message_threads_match), so the line is from the
// instance, with from_run NOTICE_RUN: no run sent it, the bus did, and a failed turn's own line is
// the instance's too. With { thread, session } the messages are the session's pending ones, for a
// refused start; with { run } they are the ones delivered to that run and never read, for a
// stopped run. A message the instance has already answered gets no line, so a refusal the
// launcher meets on every tick leaves one line per message. The text is fixed per code
// (failureNoticeBody above). Trusted launcher API, deliberately NOT a socket verb.
failureNotice(record) {
keys(record, ['business', 'code', 'thread', 'session', 'run'], ['business', 'code']);
this.#business(identifier(record.business));
if (typeof record.code !== 'string' || !NOTICE_CODE.test(record.code)) fail('invalid-request');
const byRun = record.run !== undefined;
if (byRun === (record.thread !== undefined || record.session !== undefined)) fail('invalid-request');
let session = null, instance, where, param;
if (byRun) {
const run = this.#runs.get(record.business + ':' + identifier(record.run)) ?? fail('unknown-run');
session = run.session ?? null;
instance = run.role;
where = `EXISTS(SELECT 1 FROM deliveries d WHERE d.message=m.id AND d.op='delivered' AND d.holder_run=?)
AND NOT EXISTS(SELECT 1 FROM deliveries d WHERE d.message=m.id AND d.op='read')`;
param = run.run;
} else {
session = identifier(record.session);
const se = this.#store.get('SELECT instance FROM sessions WHERE id=? AND thread=? AND business=?', session, identifier(record.thread), record.business) ?? fail('session-not-found');
instance = se.instance;
where = `NOT EXISTS(SELECT 1 FROM deliveries d WHERE d.message=m.id AND d.op IN ('delivered','read'))`;
param = null;
}
const body = failureNoticeBody(record.code);
if (body === null) return { posted: 0 };
const scope = session ? 'm.id IN (SELECT message FROM message_threads WHERE session=?)' : 'NOT EXISTS(SELECT 1 FROM message_threads mt WHERE mt.message=m.id)';
return this.#store.transaction(() => {
const open = this.#store.all(
`SELECT m.id, m.from_role FROM messages m WHERE m.business=? AND m.to_role=? AND ${scope}
AND m.from_role IN ('human','user','tester') AND ${where}
AND NOT EXISTS(SELECT 1 FROM message_cancels x WHERE x.message=m.id)
AND NOT EXISTS(SELECT 1 FROM messages n WHERE n.in_reply_to=m.id AND n.from_role=m.to_role)
ORDER BY m.seq`,
...[record.business, instance, ...(session ? [session] : []), ...(param === null ? [] : [param])],
);
for (const m of open) {
const mid = randomUUID();
this.#store.run(
'INSERT INTO messages(id,at,business,from_role,from_run,to_role,class,in_reply_to,decision,corrects,body) VALUES(?,?,?,?,?,?,?,?,?,?,?)',
mid, this.#now(), record.business, instance, NOTICE_RUN, m.from_role, 'RESULT', m.id, null, null, body,
);
this.#route(record.business, mid, m.from_role, session);
}
return { posted: open.length };
});
}
// Row 271 (264d, #1776, D7 steps 3-5): the business's clears still in the launcher's hands, as of
// each op's latest row. An open op waits for its drained run's stop and exit proof, then its
// retirement. A retired op whose successor hasn't started, and wasn't cancelled, waits for its
+3
View File
@@ -28,6 +28,7 @@ const LAUNCH_OPS = new Set([
'landHeld',
'threadsState',
'threadIdle',
'failureNotice',
'mayLaunch',
'landHolders',
'landRelease',
@@ -67,6 +68,8 @@ function launchOp(m) {
// Row 121 (#1615): the launcher's view of the business's threads, and its idle release.
if (m.op === 'threadsState') return runtime.threadsState(m.business);
if (m.op === 'threadIdle') return runtime.threadIdle(m.record);
// #1873 chat-fail F2: one plain line per message a refused thread start or a proxy-stall stop left unanswered.
if (m.op === 'failureNotice') return runtime.failureNotice(m.record);
// Row 125 (#1619, REQ-LIMIT-4): the limit checks' answer before a launch (async).
if (m.op === 'mayLaunch') return runtime.mayLaunch(m);
// Row 156 (#1651): the launcher's landing pass: who holds a merge decision, and the stop of a
+40 -2
View File
@@ -20,6 +20,42 @@ function view(id, provider, status, priority) {
return { accountId: keyId(id), provider, displayName: `${providerName(provider)} account ${keyId(id)}`,
status, priority: priorityOk(priority) ? priority : 0 };
}
// #1873 chat-fail F3: one plain line under an account the proxy reports as not working, with
// the next step. It is built only from the structured fields of the account read this page
// already makes. status_message is matched against fixed patterns and never shown; a label,
// email, recovery hint or refresh error message is never read into the line.
const SIGN_IN = 'The account needs to sign in again. Sign in again, or add another account.';
const NO_REASON = 'The model service gave no reason. Wait and try again later, sign in again, or add another account.';
const USAGE_TEXT = /usage limit|quota|rate.?limit|limit reached|too many requests|\b429\b|insufficient.?balance|credit balance/i;
// A bare "expired" is not enough: a usage window expires too.
const SIGN_IN_TEXT = /\b401\b|unauthori[sz]ed|invalid.?grant|\b(?:\w*[_-])?(?:tokens?|sessions?|credentials?|logins?|auth\w*|grants?)\s+(?:(?:has|have|is|are|was|were)\s+)?(?:been\s+)?expired|\brevoked\b|\bsign.?in\b|\b(?:re-?)?log.?in\b|\bre-?auth/i;
const RFC3339 = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})(?:\.(\d{1,9}))?(Z|[+-]\d{2}:\d{2})$/;
// A reset time the read gives, as "YYYY-MM-DD HH:MM UTC" made from the parsed date, only when
// it is a valid RFC 3339 time still ahead of now. Anything else gives no time.
function resetTime(value, now) {
const m = typeof value === 'string' ? RFC3339.exec(value) : null;
if (!m || +m[2] < 1 || +m[2] > 12 || +m[3] < 1 || +m[3] > 31 || +m[4] > 23 || +m[5] > 59 || +m[6] > 59) return null;
// A day the month doesn't have (02-31) would roll into the next month, so it is refused.
const day = new Date(Date.UTC(2000, +m[2] - 1, +m[3]));
day.setUTCFullYear(+m[1]);
if (day.getUTCMonth() !== +m[2] - 1 || day.getUTCDate() !== +m[3]) return null;
const at = Date.parse(`${m[1]}-${m[2]}-${m[3]}T${m[4]}:${m[5]}:${m[6]}${m[8]}`);
if (!Number.isFinite(at) || at <= now) return null;
return `${new Date(at).toISOString().slice(0, 16).replace('T', ' ')} UTC`;
}
function accountReason(row, now) {
const message = typeof row.status_message === 'string' ? row.status_message.slice(0, 4096) : '';
const time = resetTime(row.next_retry_after, now);
const usage = time ? `The account is out of usage. It resets at ${time}. Wait until then, or add another account.`
: 'The account is out of usage. Wait until it resets, or add another account.';
// The structured flags come first, then the message patterns.
if (row.requires_sign_in === true || (object(row.refresh_error) && Object.keys(row.refresh_error).length > 0)) return SIGN_IN;
if (row.quota_blocked === true) return usage;
if (SIGN_IN_TEXT.test(message)) return SIGN_IN;
if (USAGE_TEXT.test(message)) return usage;
if (time) return `The model service paused this account until ${time}. Wait until then, sign in again, or add another account.`;
return NO_REASON;
}
export class ProxyPage {
#client; #tag; #records; #lock; #org; #dirty = false; #now; #timeout; #onChange; #pending = new Map(); #writes = Promise.resolve();
constructor({ client, instanceTag, keyRecords = () => [], now = Date.now, loginTimeoutSeconds = 300, onChange = () => {}, lock = null, org = null }) {
@@ -51,7 +87,8 @@ export class ProxyPage {
const provider = ['claude', 'codex'].includes(row.provider) ? row.provider : 'other';
const status = row.disabled === true ? 'disabled' : row.unavailable === true ? 'unavailable'
: ['ready', 'active', 'error', 'disabled', 'unavailable'].includes(row.status) ? row.status : 'unknown';
rows.push({ ...view(row.id, provider, status, row.priority), source: 'oauth', name: row.name });
rows.push({ ...view(row.id, provider, status, row.priority), source: 'oauth', name: row.name,
...(['unavailable', 'error'].includes(status) ? { reason: accountReason(row, this.#now()) } : {}) });
}
for (const [groupIndex, group] of groups.entries()) {
if (!Array.isArray(group?.keys)) continue;
@@ -69,7 +106,8 @@ export class ProxyPage {
}
#public(row) {
const { accountId, provider, displayName, status, priority } = row;
return { accountId, provider, displayName, status, priority, ...(row.source === 'apikey' && provider === 'z.ai' ? { models: row.models } : {}) };
return { accountId, provider, displayName, status, priority, ...(row.source === 'apikey' && provider === 'z.ai' ? { models: row.models } : {}),
...(typeof row.reason === 'string' ? { reason: row.reason } : {}) };
}
#session(state, scope, { reportTimeout = false } = {}) {
const entry = stateOk(state) ? this.#pending.get(state) : null;
+2
View File
@@ -1864,6 +1864,8 @@ export async function startBroker({
},
threadsState: (business) => broker.threadsState(business),
threadIdle: (r) => broker.threadIdle(r),
// #1873 chat-fail F2: the plain line a refused thread start or a proxy-stall stop leaves.
failureNotice: (r) => broker.failureNotice(r),
// Row 125 (#1619, REQ-LIMIT-4): the launcher asks the limit checks before every launch.
mayLaunch: async ({ business, instance, provider, model, account = null }) => {
processingWork();
@@ -76,6 +76,7 @@ export async function ownershipFixture(t,{launches=true,listed=true,keep=false,r
case 'landRelease':return bus.landRelease(m.record);
case 'threadsState':return bus.threadsState(m.business);
case 'threadIdle':return bus.threadIdle(m.record);
case 'failureNotice':return bus.failureNotice(m.record);
case 'lifecycleRequests':return bus.lifecycleRequests(m.business);
case 'lifecycleTake':return bus.lifecycleTake(m.record);
case 'lifecycleDelete':return bus.lifecycleDelete(m.record);
+11 -4
View File
@@ -80,7 +80,9 @@ function instance(t) {
// A real organization bus on the instance, started with no proxy unless `binding.proxy` gives one.
async function orgBus(t, f, id, { providers = ['cliproxy'], binding = {} } = {}) {
const doc = businessDoc(f.root, id);
doc.plugins = { harness: ['pi'], 'model-provider': providers };
// The coder's model is a Claude model on a legacy entry, which counts as OAuth and runs on
// claude-code only, so the organization enables claude-code.
doc.plugins = { harness: ['claude-code'], 'model-provider': providers };
writeJson(join(f.configDir, 'businesses', `${id}.json`), doc);
const business = loadBusiness(id, { dir: f.configDir, rolesDir: f.rolesDir }), resolved = {};
for (const name of Object.keys(business.roles)) resolved[name] = resolveInstance({ system: f.system, business, instance: name });
@@ -113,7 +115,7 @@ async function pickModel(o) {
const view = await o.ask('routes');
assert.equal(view.catalogue.status, 'ready');
assert.ok(view.catalogue.models.some(m => m.key === MODEL));
await o.ask('routes.set', { revision: view.revision, routes: { coder: { harness: 'pi', model: MODEL, thinking: 'low', fallback: null } } });
await o.ask('routes.set', { revision: view.revision, routes: { coder: { harness: 'claude-code', model: MODEL, thinking: 'low', fallback: null } } });
}
// The host launcher as cli.mjs builds it for a bus that booted with no proxy.json: its system has
@@ -123,7 +125,10 @@ async function hostLauncher(t, f, o, { system = {} } = {}) {
initializeControl(o.dataRoot);
// A launch whose role names a skill that is not enabled refuses before the model gate.
if (!fs.existsSync(join(f.dataRoot, 'skills-enabled'))) fs.cpSync(REPO_SKILLS, join(f.dataRoot, 'skills-enabled'), { recursive: true });
const env = withoutHostCredentials(humanEnv({ MOSAIC_CONFIG: join(f.configDir, 'config.json'), MOSAIC_ROLES_DIR: f.rolesDir }), { piKey: true });
// claude-code's preflight needs a session login or a proxy key, and this launcher boots with no
// proxy, so the env carries a stand-in login (as launcher.test.mjs does). No adapter ever runs.
const env = withoutHostCredentials(humanEnv({ MOSAIC_CONFIG: join(f.configDir, 'config.json'), MOSAIC_ROLES_DIR: f.rolesDir }));
env.CLAUDE_CODE_OAUTH_TOKEN = 'stand-in';
const answers = [], ops = [];
const host = { business: o.id, beforeClose: () => {}, op: async m => {
ops.push(m.op);
@@ -134,7 +139,9 @@ async function hostLauncher(t, f, o, { system = {} } = {}) {
return {};
} };
const cliproxy = JSON.stringify(pathToFileURL(new URL('../../plugins/model-provider/cliproxy/index.mjs', import.meta.url).pathname).href);
const makeLoader = await fakeHarnesses(t, { pi: null }, write => write('model-provider', 'cliproxy', `export * from ${cliproxy};\n`));
// The stand-in adapter path skips only the binary check; the launch stops at the model gate or
// authorizeLaunch, before any adapter runs.
const makeLoader = await fakeHarnesses(t, { 'claude-code': '/fixture-unused' }, write => write('model-provider', 'cliproxy', `export * from ${cliproxy};\n`));
// The validated system config shape (mosaic-config.mjs), as cli.mjs passes it with the org root.
const config = { configVersion: 1, environment: 'development', dataRoot: o.dataRoot, instanceRoot: f.dataRoot,
execution: { backend: 'docker', provider: 'zai', model: 'glm-5.3-flash', adapter: 'mock' }, ...system };
@@ -0,0 +1,82 @@
// #1873 chat-fail F3: an account the proxy reports as not working carries one plain line,
// built only from the structured fields of the proxy's own account read. No upstream text,
// label or email is ever reflected, and no new proxy call is made.
import test from 'node:test';
import assert from 'node:assert/strict';
import { fakeManagement } from './proxy-page-fixture.mjs';
const PRIVATE = ['p4-private-label', 'p4-private-email', 'p4-private-token', 'p4-private-message'];
const oauth = extra => ({ id: 'oauth-one.json', name: 'oauth-one.json', provider: 'claude', label: PRIVATE[0],
email: PRIVATE[1], token: PRIVATE[2], status: 'ready', priority: 3, ...extra });
const SIGN_IN = 'The account needs to sign in again. Sign in again, or add another account.';
const USAGE = 'The account is out of usage. Wait until it resets, or add another account.';
const usageAt = time => `The account is out of usage. It resets at ${time}. Wait until then, or add another account.`;
const PAUSED = time => `The model service paused this account until ${time}. Wait until then, sign in again, or add another account.`;
const NO_REASON = 'The model service gave no reason. Wait and try again later, sign in again, or add another account.';
async function reasonOf(t, extra) {
const s = await fakeManagement(t, { files: [oauth(extra)] });
const result = await s.call('proxy.accounts');
const text = JSON.stringify(result);
for (const value of PRIVATE) assert.equal(text.includes(value), false, `${value} reached the page`);
assert.deepEqual(s.state.calls, ['GET /v8/management/credentials', 'GET /v8/management/config/api-keys/claude'], 'no new proxy call');
return result.accounts[0];
}
test('an account that must sign in again says so, from requires_sign_in or a refresh error', async t => {
for (const extra of [{ status: 'error', requires_sign_in: true, next_action: 'sign_in', recovery_hint: PRIVATE[3] },
{ status: 'error', refresh_error: { code: 'invalid_grant', message: PRIVATE[3], http_status: 400 } },
{ unavailable: true, status_message: `401 Unauthorized ${PRIVATE[3]}` },
{ status: 'error', status_message: `token expired ${PRIVATE[3]}` }])
assert.equal((await reasonOf(t, extra)).reason, SIGN_IN, JSON.stringify(extra));
});
test('an account out of usage names the reset time when the read carries one', async t => {
assert.equal((await reasonOf(t, { unavailable: true, quota_blocked: true, next_retry_after: '2099-03-04T05:06:07Z' })).reason,
usageAt('2099-03-04 05:06 UTC'));
assert.equal((await reasonOf(t, { status: 'error', status_message: `usage limit reached ${PRIVATE[3]}`, next_retry_after: '2099-03-04T05:06:07+02:00' })).reason,
usageAt('2099-03-04 03:06 UTC'));
// No time, a past time or a value that is not a time gives no time, never the value itself.
for (const next_retry_after of [undefined, '2000-01-01T00:00:00Z', PRIVATE[3], '2099-13-40T99:99:99Z', 12345, { at: 1 }])
assert.equal((await reasonOf(t, { unavailable: true, quota_blocked: true, next_retry_after })).reason, USAGE, String(next_retry_after));
});
test('a paused account with only a retry time says until when', async t => {
assert.equal((await reasonOf(t, { unavailable: true, status_message: PRIVATE[3], next_retry_after: '2099-03-04T05:06:07.123456789Z' })).reason,
PAUSED('2099-03-04 05:06 UTC'));
});
test('a failing account with nothing more in the read says the model service gave no reason', async t => {
for (const extra of [{ status: 'error' }, { unavailable: true }, { status: 'unavailable', status_message: PRIVATE[3] },
{ status: 'error', status_message: 42, refresh_error: null, requires_sign_in: 'yes', quota_blocked: 1 }])
assert.equal((await reasonOf(t, extra)).reason, NO_REASON, JSON.stringify(extra));
});
test('a working, turned-off or unknown account keeps exactly its fields and gets no line', async t => {
for (const extra of [{}, { status: 'active', status_message: 'usage limit reached' }, { disabled: true, requires_sign_in: true }, { status: 'mystery' }])
assert.deepEqual(Object.keys(await reasonOf(t, extra)).sort(), ['accountId', 'displayName', 'priority', 'provider', 'status'], JSON.stringify(extra));
});
test('every line is plain: no em dash, no control character, at most one sentence pair', () => {
for (const line of [SIGN_IN, USAGE, usageAt('2099-03-04 05:06 UTC'), PAUSED('2099-03-04 05:06 UTC'), NO_REASON]) {
assert.doesNotMatch(line, /[—\u0000-\u001f\u007f]/);
assert.ok(line.length <= 160, line);
}
});
// Review round 1: a day the month doesn't have is refused, not rolled into the next month.
test('an impossible date gives no reset time; a real leap day does', async t => {
for (const next_retry_after of ['2099-02-31T00:00:00Z', '2099-04-31T10:00:00Z', '2099-02-29T00:00:00Z'])
assert.equal((await reasonOf(t, { unavailable: true, quota_blocked: true, next_retry_after })).reason, USAGE, next_retry_after);
assert.equal((await reasonOf(t, { unavailable: true, quota_blocked: true, next_retry_after: '2096-02-29T01:02:03Z' })).reason, usageAt('2096-02-29 01:02 UTC'));
});
// Review round 1: quota_blocked wins over a message word, and a bare "expired" is not a sign-in.
test('an account blocked on usage whose message says expired gets the usage line', async t => {
assert.equal((await reasonOf(t, { unavailable: true, quota_blocked: true, status_message: `usage window expired ${PRIVATE[3]}` })).reason, USAGE);
assert.equal((await reasonOf(t, { status: 'error', status_message: `trial window expired ${PRIVATE[3]}` })).reason, NO_REASON);
for (const status_message of ['session expired', 'credentials expired', 'OAuth token has expired'])
assert.equal((await reasonOf(t, { status: 'error', status_message })).reason, SIGN_IN, status_message);
});
// Review round 2: the sign-in words match whole words only, and a prefixed or "is expired" token counts.
test('a message word inside another word is not a sign-in; a prefixed or "is expired" token is', async t => {
for (const status_message of ['assigning a slot', 'logging paused', 'design input'])
assert.equal((await reasonOf(t, { status: 'error', status_message })).reason, NO_REASON, status_message);
for (const status_message of ['access_token expired', 'token is expired', 'refresh-token has been expired', 'please sign in'])
assert.equal((await reasonOf(t, { status: 'error', status_message })).reason, SIGN_IN, status_message);
});
// Review round 3: "relogin", plurals and past forms of "expired" read as a sign-in too.
test('relogin, plural and past forms of an expired sign-in say sign in again', async t => {
for (const status_message of ['relogin required', 'tokens have expired', 'session was expired', 'credentials were expired'])
assert.equal((await reasonOf(t, { status: 'error', status_message })).reason, SIGN_IN, status_message);
});
+77
View File
@@ -656,3 +656,80 @@ test('C2-09a: creative bindLaunch without a thread refuses thread-required', t =
assert.equal(ctx.count('SELECT count(*) n FROM threads'), 0);
assert.equal(ctx.count('SELECT count(*) n FROM role_claims'), 0);
});
// #1873 chat-fail F2: the launcher's trusted failureNotice puts one plain line in the conversation
// for each message a refused start or a stalled run left unanswered.
const LIMIT_LINE = "The turn failed: the agent reached a usage limit. Wait for it to reset, or check the limits in Settings.\nDetails: start refused: limit-reached";
const STALL_LINE = "The turn failed: the model service gave no answer in time, so the turn was stopped. Try again, or check the account in Settings > Model setup.\nDetails: stopped: proxy-stall";
const lines = (store, id) => store.all("SELECT * FROM messages WHERE in_reply_to=? AND from_run='bus'", id);
test('#1873 F2: a refused start puts one line per pending message in its session, once, and none for a wait', (t) => {
const { b, store, call, A, B } = twoThreads(t);
b.endLaunch({ business: 'demo', run: 'a-run', reason: 'idle' });
const session = store.get('SELECT session FROM session_defaults WHERE thread=?', A).session;
const notice = (code, extra = {}) => b.failureNotice({ business: 'demo', thread: A, session, code, ...extra });
const one = call('human', 'thread.send', { thread: A, body: 'one' });
const two = call('human', 'thread.send', { thread: A, body: 'two' });
const other = call('human', 'thread.send', { thread: B, body: 'for B' });
// A start that waits for a free place leaves no line.
for (const code of ['capacity-full', 'instance-running', 'legacy-run-live', 'session-workspace-busy', 'processing-generation-changed', 'proxy-lock-busy', 'project-pending', 'processing-halted']) assert.deepEqual(notice(code), { posted: 0 });
assert.deepEqual(notice('limit-reached'), { posted: 2 });
for (const sent of [one, two]) {
const [line] = lines(store, sent.id);
assert.equal(line.body, LIMIT_LINE);
assert.deepEqual([line.class, line.from_role, line.to_role, line.from_run], ['RESULT', 'assistant', 'human', 'bus']);
assert.equal(store.get('SELECT session FROM message_threads WHERE message=?', line.id).session, session, 'the line is in the session');
}
assert.deepEqual(lines(store, other.id), [], "another thread's message gets none");
// Every retry meets the same refusal: still one line each, under any code.
assert.deepEqual(notice('limit-reached'), { posted: 0 });
assert.deepEqual(notice('model-provider-unavailable'), { posted: 0 });
assert.equal(lines(store, one.id).length, 1);
// The message stays pending, so the next start still takes it.
assert.equal(b.threadsState('demo').find((s) => s.thread === A).pending, 2);
const three = call('human', 'thread.send', { thread: A, body: 'three' });
assert.deepEqual(notice('some-new-code'), { posted: 1 });
assert.match(lines(store, three.id)[0].body, /^The turn failed: the assistant couldn't start\. .*\nDetails: start refused: some-new-code$/);
});
test('#1873 F2: a stalled run puts one line on each message delivered to it and unread, thread or role', (t) => {
const { b, store, call, A } = twoThreads(t);
const taken = call('human', 'thread.send', { thread: A, body: 'taken' });
assert.deepEqual(call('a', 'message.receive').map((m) => m.id), [taken.id]);
const waiting = call('human', 'thread.send', { thread: A, body: 'still waiting' });
assert.deepEqual(b.failureNotice({ business: 'demo', run: 'a-run', code: 'proxy-stall' }), { posted: 1 });
assert.equal(lines(store, taken.id)[0].body, STALL_LINE);
assert.deepEqual(lines(store, waiting.id), [], 'a message the run never took waits for the next run');
assert.deepEqual(b.failureNotice({ business: 'demo', run: 'a-run', code: 'proxy-stall' }), { posted: 0 });
// A read message was answered by the turn: no line.
const read = call('human', 'thread.send', { thread: A, body: 'read' });
call('a', 'message.receive');
call('a', 'message.read', { id: read.id });
assert.deepEqual(b.failureNotice({ business: 'demo', run: 'a-run', code: 'proxy-stall' }), { posted: 1 }, 'only waiting, delivered now');
assert.deepEqual(lines(store, read.id), []);
// A role run: the human's message to the chief, outside any thread.
const asked = call('human', 'message.send', { to: 'chief', class: 'REQUEST', body: 'chief, please' });
assert.ok(call('chief', 'message.receive').some((m) => m.id === asked.id));
assert.deepEqual(b.failureNotice({ business: 'demo', run: 'chief-run', code: 'proxy-stall' }), { posted: 1 });
const [line] = lines(store, asked.id);
assert.deepEqual([line.body, line.from_role, line.to_role], [STALL_LINE, 'chief', 'human']);
assert.equal(store.get('SELECT count(*) n FROM message_threads WHERE message=?', line.id).n, 0);
});
test('#1873 F2: failureNotice refuses a malformed record before any write', (t) => {
const { b, store, A } = twoThreads(t);
const session = store.get('SELECT session FROM session_defaults WHERE thread=?', A).session;
const before = store.get('SELECT count(*) n FROM messages').n;
const bad = [
[{ business: 'demo', code: 'limit-reached' }, 'invalid-request'],
[{ business: 'demo', code: 'limit-reached', run: 'a-run', thread: A, session }, 'invalid-request'],
[{ business: 'demo', code: 'Limit Reached', run: 'a-run' }, 'invalid-request'],
[{ business: 'demo', code: 'x'.repeat(65), run: 'a-run' }, 'invalid-request'],
[{ business: 'demo', code: 7, run: 'a-run' }, 'invalid-request'],
[{ business: 'demo', code: 'limit-reached', run: 'a-run', why: 'secret text' }, 'invalid-request'],
[{ business: 'demo', code: 'limit-reached', run: 'nobody-run' }, 'unknown-run'],
[{ business: 'demo', code: 'limit-reached', thread: A, session: 'no-such-session' }, 'session-not-found'],
];
for (const [record, code] of bad) assert.throws(() => b.failureNotice(record), { code }, JSON.stringify(record));
assert.equal(store.get('SELECT count(*) n FROM messages').n, before);
});
+34 -1
View File
@@ -302,6 +302,11 @@ export function createLauncher({
let ticking = false;
const stopping = new Set();
const refusedThreads = new Map();
// #1873 chat-fail F2: the refusal and message each thread session's plain line was asked for.
const noticedThreads = new Map();
// #1873 F2: the last refusal seen per thread start, and the last failed post of its line.
const refusalSeen = new Map();
const noticeFailed = new Map();
const startedFor = new Map();
// Row 271 (264d): each clear's stop time (by op), and each clear refusal logged once (by op and code).
const clearStops = new Map();
@@ -896,7 +901,13 @@ export function createLauncher({
}
children.set(run, exited);
const outputWatch = admittedProxy ? watchModelOutput({ runDir, armAtFirstTurn: true, keyId: modelKeyId, seconds: system.proxy.firstOutputSeconds ?? 120,
stop: () => host.op({ op: 'modelStall', business: b.id, role: instance, run, keyId: modelKeyId }), receipt: record }) : null;
stop: async () => {
const out = await host.op({ op: 'modelStall', business: b.id, role: instance, run, keyId: modelKeyId });
// #1873 chat-fail F2: the message the stopped turn was on gets one plain line, as a failed turn's.
try { await host.op({ op: 'failureNotice', record: { business: b.id, run, code: 'proxy-stall' } }); }
catch (err) { log(`run ${run}: failure line refused: ${err.code ?? 'failed'}`); }
return out;
}, receipt: record }) : null;
exited.then(async ({ code, signal }) => {
outputWatch?.cancel();
let receipt;
@@ -1066,8 +1077,30 @@ export function createLauncher({
await start({ instance: t.instance, launchedBy: b.human, ...where, legacy: t.legacy, legacyThread: legacyThreads.has(t.thread), ...(await check(t.instance, where)) });
startedFor.set(key, signature);
refusedThreads.delete(key);
noticedThreads.delete(key);
refusalSeen.delete(key);
noticeFailed.delete(key);
} catch (e) {
const code = e instanceof Refusal ? e.code : "launch-failed";
// #1873 chat-fail F2: the person who wrote gets one plain line per message the refusal
// leaves unanswered. Asked once per code and message; the bus keeps it to one per message.
// launch-failed can be a lost race that the next try wins, so it gets a line only when
// the same message is refused with it twice in a row. A post that fails is tried again
// on the next tick, and its failure is logged once.
const noticed = `${code}|${signature}`;
const repeated = refusalSeen.get(key) === noticed;
refusalSeen.set(key, noticed);
if (noticedThreads.get(key) !== noticed && (code !== "launch-failed" || repeated)) {
try {
await host.op({ op: "failureNotice", record: { business: b.id, thread: t.thread, session: t.session, code } });
noticedThreads.set(key, noticed);
noticeFailed.delete(key);
} catch (err) {
const failed = `${noticed}|${err.code ?? "failed"}`;
if (noticeFailed.get(key) !== failed) log(`thread ${t.thread} session ${t.session}: failure line refused: ${err.code ?? "failed"}`);
noticeFailed.set(key, failed);
}
}
// Logged once per code, not once per tick.
if (refusedThreads.get(key) === code) return;
refusedThreads.set(key, code);
+1
View File
@@ -105,6 +105,7 @@ async function start(t, { f, adapter }, { businessId = "acme", orgRoot, copyTran
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
threadsState: (m) => runtime.threadsState(m.business),
threadIdle: (m) => runtime.threadIdle(m.record),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
@@ -99,6 +99,7 @@ async function start(t, { f, adapter }) {
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
threadsState: (m) => runtime.threadsState(m.business),
threadIdle: (m) => runtime.threadIdle(m.record),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
+1
View File
@@ -30,6 +30,7 @@ const ops = {
credentialStatus: m => runtime.credentialStatus(m.business, m.role),
threadsState: m => runtime.threadsState(m.business),
threadIdle: m => runtime.threadIdle(m.record),
failureNotice: m => runtime.failureNotice(m.record),
mayLaunch: m => runtime.mayLaunch(m),
landHeld: m => runtime.landHeld(m.record),
landHolders: m => runtime.landHolders(m.business),
@@ -77,6 +77,7 @@ async function start(t, { f, adapter }, threadIdle) {
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
threadsState: (m) => runtime.threadsState(m.business),
threadIdle: (m) => threadIdle(runtime, m),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
@@ -95,6 +95,7 @@ async function start(t, { f, adapter }, { tickMs = 100, ops: replace } = {}) {
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
threadsState: (m) => runtime.threadsState(m.business),
threadIdle: (m) => runtime.threadIdle(m.record),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
@@ -48,6 +48,7 @@ function requestHost(x) {
landHeld: m => x.bus.landHeld(m.record), landHolders: m => x.bus.landHolders(m.business),
sessionClears: m => x.bus.sessionClears(m.business),
threadsState: m => x.bus.threadsState(m.business), lifecycleRequests: m => x.bus.lifecycleRequests(m.business),
failureNotice: m => x.bus.failureNotice(m.record),
'request.admission': m => x.bus.admission(m), 'request.launches': m => x.bus.requestLaunches(m.business),
'request.launchResult': m => x.bus.requestLaunchResult(m.record), 'request.manifest': m => x.bus.requestManifest(m.record),
'request.release': m => x.bus.requestRelease(m.cap, m.record), 'request.chief': m => x.bus.requestChief(m.business),
+1
View File
@@ -95,6 +95,7 @@ async function start(t, { f, adapter }, options = {}) {
definitionScan: (m) => runtime.definitionScan({ business: m.business, definition: m.definition, entry: m.entry }),
threadsState: (m) => runtime.threadsState(m.business),
threadIdle: (m) => runtime.threadIdle(m.record),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
@@ -98,6 +98,7 @@ export async function managedLaunchFixture(t,{proxy=true,launches=true,refusePro
case 'landRelease':return bus.landRelease(m.record);
case 'threadsState':return bus.threadsState(m.business);
case 'threadIdle':return bus.threadIdle(m.record);
case 'failureNotice':return bus.failureNotice(m.record);
case 'lifecycleRequests':return bus.lifecycleRequests(m.business);
case 'sessionClears':return bus.sessionClears(m.business);
default:throw Error('fixture unrequested trusted operation');
@@ -11,8 +11,11 @@ async function launched(t,options){const f=await managedLaunchFixture(t,options)
test('653 actual managed proxy launch stays idle unarmed then stops first silent turn through trusted runtime',{skip,timeout:30000},async t=>{
const {f,s}=await launched(t);assert.equal(f.trace.successfulPreflights,1);assert.deepEqual(f.trace.enableBeforeSpawn,[true]);assert.equal(f.trace.bindings[0].modelKeyId,f.expectedKeyId());assert.equal(s.modelKeyId,f.expectedKeyId());assert.notEqual(s.pid,s.runnerPid);
await pause(1250);assert.equal(f.trace.stalls.length,0,'idle beyond configured initial budget does not stop');assert.equal(fs.existsSync(join(s.runDir,'proxy-first-turn')),false);assert.equal(startTimeOf(s.runnerPid),s.runnerStartTime);
const before=process.hrtime.bigint();f.send('SILENT');await waitFor(()=>fs.existsSync(join(s.runDir,'proxy-first-turn')),'actual managed adapter turn publishes marker');const stamp=BigInt(fs.readFileSync(join(s.runDir,'proxy-first-turn'),'utf8').trim().split(' ')[1]);assert.ok(stamp>=before&&stamp<=process.hrtime.bigint());assert.equal(fs.existsSync(join(s.runDir,'proxy-first-output')),false);
await waitFor(()=>f.trace.stalls.length===1,'actual caller watcher reaches trusted modelStall');assert.deepEqual(f.trace.stalls,[{run:s.run,keyId:s.modelKeyId,stopped:true}]);await waitFor(()=>f.sessions().length===0,'normal managed exit unregisters session');assert.equal(f.launchLog().some(r=>r.action==='proxy-stall'&&r.keyIds?.includes(s.modelKeyId)&&!r.code),true);f.assertHistoryGuard();f.valueFree();
const before=process.hrtime.bigint();const sent=f.send('SILENT');await waitFor(()=>fs.existsSync(join(s.runDir,'proxy-first-turn')),'actual managed adapter turn publishes marker');const stamp=BigInt(fs.readFileSync(join(s.runDir,'proxy-first-turn'),'utf8').trim().split(' ')[1]);assert.ok(stamp>=before&&stamp<=process.hrtime.bigint());assert.equal(fs.existsSync(join(s.runDir,'proxy-first-output')),false);
await waitFor(()=>f.trace.stalls.length===1,'actual caller watcher reaches trusted modelStall');assert.deepEqual(f.trace.stalls,[{run:s.run,keyId:s.modelKeyId,stopped:true}]);await waitFor(()=>f.sessions().length===0,'normal managed exit unregisters session');assert.equal(f.launchLog().some(r=>r.action==='proxy-stall'&&r.keyIds?.includes(s.modelKeyId)&&!r.code),true);f.assertHistoryGuard();
// #1873 chat-fail F2: the stopped turn's message gets one plain line, as a failed turn's.
const line=await f.reply(sent);assert.equal(line.body,"The turn failed: the model service gave no answer in time, so the turn was stopped. Try again, or check the account in Settings > Model setup.\nDetails: stopped: proxy-stall");assert.deepEqual([line.from_role,line.from_run,line.class],['chief','bus','RESULT']);
f.valueFree();
});
test('653 actual managed proxy output cancels caller watcher and first timestamp never resets',{skip,timeout:30000},async t=>{
const {f,s}=await launched(t);const first=f.send('OUTPUT');const answer=await f.reply(first);assert.equal(answer.body,'fx653-private-output');const file=join(s.runDir,'proxy-first-turn'),stamp=fs.readFileSync(file,'utf8');assert.equal(fs.statSync(file).mode&0o777,0o600);assert.equal(fs.readFileSync(join(s.runDir,'proxy-first-output'),'utf8'),'output\n');
@@ -95,6 +95,7 @@ function requestHost(x) {
landHeld: m => x.bus.landHeld(m.record), landHolders: m => x.bus.landHolders(m.business),
sessionClears: m => x.bus.sessionClears(m.business),
threadsState: m => x.bus.threadsState(m.business), lifecycleRequests: m => x.bus.lifecycleRequests(m.business),
failureNotice: m => x.bus.failureNotice(m.record),
'request.admission': m => x.bus.admission(m), 'request.launches': m => x.bus.requestLaunches(m.business),
'request.launchResult': m => x.bus.requestLaunchResult(m.record), 'request.manifest': m => x.bus.requestManifest(m.record),
'request.release': m => x.bus.requestRelease(m.cap, m.record), 'request.chief': m => x.bus.requestChief(m.business),
@@ -96,6 +96,7 @@ async function start(t, { f, adapter, access }, threadsState) {
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
threadsState: (m) => threadsState(runtime, m),
threadIdle: (m) => runtime.threadIdle(m.record),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
@@ -92,6 +92,7 @@ async function start(t, { f, adapter }, { tickMs = 100, stopTimeoutMs } = {}) {
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
threadsState: (m) => runtime.threadsState(m.business),
threadIdle: (m) => runtime.threadIdle(m.record),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
@@ -112,6 +112,7 @@ async function start(t, { f, adapter }, { tickMs = 100, ops: replace } = {}) {
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
threadsState: (m) => runtime.threadsState(m.business),
threadIdle: (m) => runtime.threadIdle(m.record),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
+73
View File
@@ -93,6 +93,7 @@ async function start(t, { f, adapter }, { tickMs = 100, ops: replace } = {}) {
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
threadsState: (m) => runtime.threadsState(m.business),
threadIdle: (m) => runtime.threadIdle(m.record),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
@@ -496,6 +497,78 @@ test("REQ-LIMIT-3 REQ-LIMIT-4 a thread start whose instance is at its token limi
assert.equal(launchLog(f).filter((e) => e.event === "refused" && e.thread === A).length, 1);
});
// #1873 chat-fail F2: a refused start leaves one plain line in the chat for each message it left
// unanswered, with the cause and next step, never one per retry and never the refusal's reason.
const LIMIT_LINE = "The turn failed: the agent reached a usage limit. Wait for it to reset, or check the limits in Settings.\nDetails: start refused: limit-reached";
test("#1873 F2 a refused thread start puts one plain line per unanswered message in the chat, not one per retry", { skip, timeout: 60000 }, async (t) => {
const prepared = prepare(t, { edit: atLimit });
const { f } = prepared;
const host = await start(t, prepared);
const project = host.call("project.create", { name: "stack work", repo: "stack" }).id;
const A = host.call("thread.create", { project, name: "alpha" }).id;
const first = host.call("thread.send", { thread: A, body: "for A" });
const lines = (sent) => host.call("thread.messages", { thread: A }).filter((m) => m.in_reply_to === sent.id);
const [line] = await until(() => lines(first).length && lines(first));
assert.equal(line.body, LIMIT_LINE);
assert.deepEqual([line.from_role, line.from_run], ["assistant", "bus"]);
assert.equal(line.to_role, "human");
assert.equal(line.class, "RESULT");
assert.doesNotMatch(line.body, /token-limit|tokens\.instances|"limit"/, "the refusal's reason stays out of the chat");
for (let i = 0; i < 3; i++) await host.launcher.tick();
assert.equal(lines(first).length, 1, "a retry adds no line");
// A second message gets its own line; the first keeps its one.
const second = host.call("thread.send", { thread: A, body: "again" });
await until(() => lines(second).length);
for (let i = 0; i < 3; i++) await host.launcher.tick();
assert.deepEqual(lines(second).map((m) => m.body), [LIMIT_LINE]);
assert.equal(lines(first).length, 1);
assert.deepEqual(threadRuns(f, A), []);
// The messages stay pending: the start is retried and answers them once the limit allows.
assert.equal(launchLog(f).filter((e) => e.event === "refused" && e.thread === A).length, 1);
});
// #1873 review round 1 (P2-2): launch-failed can be a lost race that the next try wins, so one
// such refusal posts nothing and the answer arrives alone; the same refusal twice posts one line.
const failedLines = (host, thread) => host.call("thread.messages", { thread }).filter((m) => m.body.startsWith("The turn failed"));
test("#1873 F2 one launch-failed refusal that the next start clears leaves no failure line", { skip, timeout: 60000 }, async (t) => {
const prepared = prepare(t);
let throws = 1;
const host = await start(t, prepared, { ops: (runtime) => ({ landHeld: (m) => { if (throws-- > 0) throw new Error("fixture: lost race"); return runtime.landHeld(m.record); } }) });
const project = host.call("project.create", { name: "stack work", repo: "stack" }).id;
const A = host.call("thread.create", { project, name: "alpha" }).id;
const sent = host.call("thread.send", { thread: A, body: "for A" });
const reply = await until(() => host.call("thread.messages", { thread: A }).find((m) => m.in_reply_to === sent.id && m.class === "RESULT"));
assert.match(reply.body, /THREAD \{/, "the first reply is the answer");
assert.equal(throws, -1, "the first start was refused");
assert.deepEqual(failedLines(host, A), []);
});
test("#1873 F2 the same launch-failed refusal twice for one message posts one line", { skip, timeout: 60000 }, async (t) => {
const prepared = prepare(t);
const host = await start(t, prepared, { ops: () => ({ landHeld: () => { throw new Error("fixture: always refused"); } }) });
const project = host.call("project.create", { name: "stack work", repo: "stack" }).id;
const A = host.call("thread.create", { project, name: "alpha" }).id;
host.call("thread.send", { thread: A, body: "for A" });
const [line] = await until(() => failedLines(host, A).length && failedLines(host, A));
assert.match(line.body, /\nDetails: start refused: launch-failed$/);
for (let i = 0; i < 3; i++) await host.launcher.tick();
assert.equal(failedLines(host, A).length, 1);
});
// #1873 review round 1 (P3-3): a line whose post fails is posted on a later tick, once.
test("#1873 F2 a failure line whose post is refused once arrives on a later tick, exactly once", { skip, timeout: 60000 }, async (t) => {
const prepared = prepare(t, { edit: atLimit });
let throws = 1;
const host = await start(t, prepared, { ops: (runtime) => ({ failureNotice: (m) => { if (throws-- > 0) throw new Error("fixture: bus busy"); return runtime.failureNotice(m.record); } }) });
const project = host.call("project.create", { name: "stack work", repo: "stack" }).id;
const A = host.call("thread.create", { project, name: "alpha" }).id;
host.call("thread.send", { thread: A, body: "for A" });
const [line] = await until(() => failedLines(host, A).length && failedLines(host, A));
assert.equal(line.body, LIMIT_LINE);
assert.equal(throws, -1, "the first post was refused");
for (let i = 0; i < 3; i++) await host.launcher.tick();
assert.equal(failedLines(host, A).length, 1);
assert.equal(host.logs.filter((l) => /failure line refused/.test(l)).length, 1);
});
test("REQ-LIMIT-4 a thread run at its token limit stops at the end of its turn: the reply lands, and the end record names the limit", { skip, timeout: 60000 }, async (t) => {
const prepared = prepare(t, { edit: atLimit });
const { f } = prepared;
@@ -97,6 +97,7 @@ async function start(t, { f, adapter, access }) {
credentialStatus: (m) => runtime.credentialStatus(m.business, m.role),
threadsState: (m) => runtime.threadsState(m.business),
threadIdle: (m) => runtime.threadIdle(m.record),
failureNotice: (m) => runtime.failureNotice(m.record),
mayLaunch: (m) => runtime.mayLaunch(m),
landHeld: (m) => runtime.landHeld(m.record),
landHolders: (m) => runtime.landHolders(m.business),
+153
View File
@@ -1,2 +1,155 @@
// Shared by the sender and recipient's reply-cycle check.
export const LENGTH_FAILURE_NOTICE_PREFIX = 'Mosaic turn failure: ';
// #1873 chat-fail F1: the plain cause of a failed turn. It lives here, not in a file of its own,
// because the session sandbox mounts only RUNNER_FILES (packages/seat/src/sandbox.mjs) and this
// file is already one of them.
//
// The adapter contract (adapters/README.md) puts the answer on stdout and makes a
// nonzero exit a failure. A harness that ends on an error result may still print that
// result on stdout before it exits nonzero. That text is the only cause the runner
// has, so the chat line carries it, made safe first:
//
// 0. %XX escapes and HTML character references are decoded and the text is put in NFKC
// form, so an encoded or full-width email, path or secret is seen as itself.
// 1. ANSI escapes go, every control, format, separator, surrogate and private-use
// character goes, and whitespace collapses to single spaces.
// 2. If a secret the turn's environment holds (a variable named like a key, token,
// secret, password or credential) appears in the text, even split by whitespace or
// controls, no text is shown at all.
// 3. Links, email addresses, absolute and Windows paths, "Bearer x" values, the value after
// a key word ("key: x", "\"api_key\":\"x\"", "token x"), known key prefixes and long
// letter-and-digit runs become [redacted].
// 4. The rest is cut to CAUSE_MAX characters.
//
// Only the last READ_MAX characters are classified and shown. A harness prints its error
// result last, and an answer streamed before the failure may itself talk about limits or
// signing in. Redaction runs over the whole text first, so the cut never parts a key word
// from its value.
//
// Three causes are recognised from the text and shown as a fixed plain line with the
// next step: the account is out of usage, its sign-in expired or was refused, or it
// doesn't offer the model. Anything else shows the cleaned text. The fixed lines never
// carry any of the error's own words.
export const CAUSE_MAX = 300;
const READ_MAX = 2000;
export const PLAIN = Object.freeze({
usage: "the model account is out of usage. Check Settings > Model setup.",
signin: "the model account's sign-in expired or was refused. Sign in again in Settings > Model setup.",
model: "the model this agent uses isn't offered by its account. Pick another model in Settings > Model setup.",
});
export const WITHHELD = "the error text held a secret, so it isn't shown. Check Settings > Model setup.";
const ANSI = /\u001b(?:\[[0-?]*[ -/]*[@-~]|\][^\u0007\u001b]*(?:\u0007|\u001b\\)?|[@-Z\\-_])/g;
// Cf (format: zero-width, bidi overrides) is removed so a split token rejoins and is seen.
const INVISIBLE = /[\p{Cf}\p{Cs}\p{Co}]/gu;
// Cc (C0, DEL, C1), line and paragraph separators become spaces.
const BREAKS = /[\p{Cc}\p{Zl}\p{Zp}]/gu;
const USAGE = /usage limit|hit your [\w -]{0,24}limit|(?<!context |output |token |length )limit reached|credit balance is too low|insufficient[_ ]quota|quota (?:exceeded|exhausted)|exceeded your current quota|out of (?:usage|credits)/i;
const SIGNIN = /(?:oauth |access |auth )?token (?:has )?expired|invalid api key|invalid x-api-key|invalid bearer token|authentication_error|authentication failed|run \/login|not logged in|unauthori[sz]ed|(?:error|status)[: ]+401\b|sign[- ]?in (?:again|required|expired)|log ?in (?:again|required|expired)|invalid_grant/i;
const MODEL = /issue with the selected model|model_not_found|\bmodel\b.{0,80}\b(?:not found|does not exist|not available|isn't available|not supported|may not exist|not offered|no access)|not_found_error.{0,80}\bmodel\b|unknown model|no such model/i;
const ENTITY = { amp: "&", lt: "<", gt: ">", quot: '"', apos: "'", commat: "@", lowbar: "_", period: ".", colon: ":", sol: "/", bsol: "\\", num: "#", equals: "=", hyphen: "-", dash: "-" };
function codePoint(n) {
return Number.isInteger(n) && n > 0 && n <= 0x10ffff && (n < 0xd800 || n > 0xdfff) ? String.fromCodePoint(n) : " ";
}
// %XX escapes (twice encoded included) and HTML character references are decoded, then NFKC
// folds full-width and compatibility forms to their plain letters.
export function decodeText(text) {
let out = String(text);
for (let i = 0; i < 3; i++) {
const next = out
.replace(/(?:%[0-9a-f]{2})+/gi, run => { try { return decodeURIComponent(run); } catch { return run.replace(/%([0-9a-f]{2})/gi, (_, h) => String.fromCharCode(parseInt(h, 16))); } })
.replace(/&#(\d{1,7});?|&#x([0-9a-f]{1,6});?|&([a-z]{2,7});/gi, (all, dec, hex, name) =>
dec ? codePoint(Number(dec)) : hex ? codePoint(parseInt(hex, 16)) : ENTITY[name.toLowerCase()] ?? all);
if (next === out) break;
out = next;
}
return out.normalize("NFKC");
}
// The text with every space, control, format, mark, surrogate and private-use character gone,
// for the env-secret check.
function compactText(text) {
return decodeText(text).replace(ANSI, "").replace(/[\p{Cc}\p{Cf}\p{Cs}\p{Co}\p{M}\p{Z}\s]/gu, "");
}
const SECRET_NAME = /KEY|TOKEN|SECRET|PASS|CRED|AUTH|COOKIE/i;
// Values from the turn's environment that must never reach a chat line.
export function secretValues(env) {
const out = [];
for (const [name, value] of Object.entries(env ?? {})) {
if (typeof value !== "string" || !SECRET_NAME.test(name)) continue;
const compact = compactText(value);
if (compact.length >= 8) out.push(compact);
}
return out;
}
// A key word, with any prefix joined by _ or - (access_token, client_secret, x-api-key).
const KEY_WORD = "(?:\\w*[_-])?(?:api[_-]?key|key|token|secret|password|passwd|authorization|cookie|session)";
// Redaction runs over the whole text, up to the runner's 128000 characters, so a pattern that
// could start anywhere inside a long run without spaces starts only where the run starts: it
// matches the same text in linear time, not quadratic.
const REDACT = [
/(?<![a-z0-9+.-])[a-z0-9+.-]+:\/\/\S*/gi, // any link, its userinfo and query included, from the run's start
/\bwww\.\S+/gi,
/(?<!\S)(?=\S*\S@\S)\S+/g, // email addresses and user@host: the whole run holding the @
/(?:^|(?<=[\s("'=:]))(?:~|\.{1,2})?\/\S+/g, // absolute and home paths
/\b[A-Za-z]:\\\S*/g, // Windows paths
/\\\\\S+/g, // UNC paths
/\b(?:bearer|basic)\s+\S+/gi,
// The value after a key word: "key: x", "key=x" and the JSON form "\"api_key\":\"x\"" always;
// "token x" with only a space when x is quoted, holds a digit or is 8 characters or more.
new RegExp(`\\b${KEY_WORD}["']?\\s*[:=]\\s*(?:"[^"]*"?|'[^']*'?|[^\\s,;}]+)`, "gi"),
new RegExp(`\\b${KEY_WORD}["']?\\s+(?:"[^"]*"?|'[^']*'?|(?=[^\\s"',;}]*\\d)[^\\s"',;}]+|[^\\s"',;}]{8,})`, "gi"),
/\b(?:sk|msk|rk|pk|ghp|gho|ghs|glpat|xox[a-z])[-_]\S+/gi,
/\beyJ\S+/g, // JWT
// A long run that mixes letters and digits, or any very long run, reads as a key.
/[A-Za-z0-9_\-+/=.]{32,}/g,
/(?<![A-Za-z0-9_\-+/=.])(?=[A-Za-z0-9_\-+/=.]*\d)(?=[A-Za-z0-9_\-+/=.]*[A-Za-z])[A-Za-z0-9_\-+/=.]{16,}/g,
];
export function cleanText(text) {
return String(text).replace(ANSI, "").replace(INVISIBLE, "").replace(BREAKS, " ").replace(/\s+/g, " ").trim();
}
// A text read from its tail keeps its end, where the error is; any other keeps its start.
function cap(text, fromEnd = false, n = CAUSE_MAX) {
const chars = Array.from(text);
if (chars.length <= n) return text;
return fromEnd ? "..." + chars.slice(-(n - 3)).join("") : chars.slice(0, n - 3).join("") + "...";
}
// The plain cause for a failed turn's stdout, or null when there is no text.
// `secrets` is secretValues(env) for the turn.
export function turnCause(stdout, secrets = []) {
if (typeof stdout !== "string") return null;
const decoded = decodeText(stdout);
const clean = cleanText(decoded);
if (!clean) return null;
// The secret check covers the whole text, not only the part that is read.
const compact = compactText(stdout);
if (secrets.some(value => compact.includes(value))) return WITHHELD;
// Only the tail is classified and shown. Redaction runs over the whole text before any cut,
// so a cut that lands between a key word and its value can't leave the value bare.
const tail = clean.length > READ_MAX;
const read = tail ? clean.slice(-READ_MAX) : clean;
if (USAGE.test(read)) return PLAIN.usage;
if (SIGNIN.test(read)) return PLAIN.signin;
if (MODEL.test(read)) return PLAIN.model;
let shown = clean;
for (const re of REDACT) shown = shown.replace(re, "[redacted]");
shown = shown.replace(/(?:\[redacted\]\s*){2,}/g, "[redacted] ").trim();
// The cut drops the partial word it lands in; a redacted text short enough needs no cut.
if (shown.length > READ_MAX) shown = shown.slice(-READ_MAX).replace(/^\S+\s*/, "");
// Text that redacts to nothing but [redacted] says nothing; the technical reason stands alone.
return shown && shown !== "[redacted]" ? cap(shown, tail) : null;
}
// The chat body for a failed turn: the plain cause first, the technical reason after.
export const DETAILS = "\nDetails: ";
export function failureBody(turn) {
return turn.cause ? `The turn failed: ${turn.cause}${DETAILS}${turn.reason}` : `The turn failed: ${turn.reason}`;
}
+10 -5
View File
@@ -41,7 +41,7 @@ import { createServer, connect } from "node:net";
import { basename, join } from "node:path";
import { fileURLToPath, pathToFileURL } from "node:url";
import { Client } from "../../bus/src/client.mjs";
import { LENGTH_FAILURE_NOTICE_PREFIX } from "./failure-notice.mjs";
import { LENGTH_FAILURE_NOTICE_PREFIX, failureBody, secretValues, turnCause } from "./failure-notice.mjs";
import { ID_PATTERN } from "../../organization/src/vocabulary.mjs";
import { firstOutputMarker, firstTurnMarker } from './first-output.mjs';
import { GUARD_CODES, FIXED_TEXTS } from "../../library/src/reply-protocol.mjs";
@@ -508,7 +508,12 @@ function runTurn({ session, request, n, env, onChild }) {
const missing = missingCommand(stderrTail(errPath));
return resolve({ ok: false, reason: `adapter exited 127${missing ? ` (command not found: ${missing})` : ""}` });
}
if (code !== 0) return resolve({ ok: false, reason: signal ? `adapter killed by ${signal}` : `adapter exited ${code}` });
if (signal) return resolve({ ok: false, reason: `adapter killed by ${signal}` });
if (code !== 0) {
// #1873 chat-fail F1: an adapter that ends on an error result prints it on stdout.
const cause = turnCause(out, secretValues(env));
return resolve({ ok: false, reason: `adapter exited ${code}`, ...(cause ? { cause } : {}) });
}
// A "marker" harness (row 65: the module's turnEnd()) writes the marker when its turn ends.
if (session.turnEnd === "marker" && !existsSync(marker)) return resolve({ ok: false, reason: `${session.harness} exited before the turn ended (no turn marker)` });
resolve({ ok: true, text: out.trim() });
@@ -801,11 +806,11 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
catch (e) { log(`runner: failure record for ${m.id} failed: ${e.code ?? e.message}`); if (ended(e, false)) break loop; }
continue;
}
if (!turn.ok) log(`runner: turn ${n} for message ${m.id} failed: ${turn.reason}`);
if (!turn.ok) log(`runner: turn ${n} for message ${m.id} failed: ${turn.reason}${turn.cause ? ` (${turn.cause})` : ""}`);
try {
if (!isLengthFailureNotice(m) && m.class !== "RESULT" && (m.class === "REQUEST" || ["human", "tester", "user"].includes(m.from_role) || m.in_reply_to_class !== "RESULT")) {
if (!library) {
const body = turn.ok ? turn.text || "(the session gave no answer)" : `The turn failed: ${turn.reason}`;
const body = turn.ok ? turn.text || "(the session gave no answer)" : failureBody(turn);
await client.call("message.send", { to: m.from_role, body: clip(body) + unslop.note(), class: "RESULT", in_reply_to: m.id });
unslop.sent();
} else {
@@ -858,7 +863,7 @@ export async function main(runDir, { stdin = process.stdin, env = process.env, l
await client.call("message.read", { id: m.id });
continue;
}
if (!retry.ok) log(`runner: retry turn ${n} for message ${m.id} failed: ${retry.reason}`);
if (!retry.ok) log(`runner: retry turn ${n} for message ${m.id} failed: ${retry.reason}${retry.cause ? ` (${retry.cause})` : ""}`);
try { await send(retry); }
catch (again) {
if (!GUARD_CODES.includes(again.code) && again.code !== "request-too-large") throw again;
@@ -205,3 +205,14 @@ test('a library continuation that succeeds sends the cited RESULT once', async t
assert.equal(f.count(),2);
assert.equal(f.store.get("SELECT count(*) n FROM messages WHERE from_role='coder'").n,1);
});
// #1873 chat-fail review round 1: a failed citation retry logs its plain cause, as the first turn does.
test('a failed citation retry logs the plain cause of the failure', async t => {
let sends = 0;
const f = await setup(t, commit => [{ text: citedBody([fixtureCite(commit)]) }, { text: 'Claude AI usage limit reached', exitCode: 1 }], {
guard: async () => { if (sends++ === 0) throw new GuardError('citation-unavailable', 'The library is busy.'); return null; },
});
await f.call(f.human, 'message.send', { to: 'coder', body: 'Fixture question.', class: 'REQUEST' });
await until(() => /runner: retry turn \d+ for message \S+ failed: /.test(f.log()));
assert.match(f.log(), /runner: retry turn \d+ for message \S+ failed: adapter exited 1 \(the model account is out of usage/);
});
+59 -1
View File
@@ -322,6 +322,63 @@ test("r127 an adapter exit 127 names the program that wasn't found, through the
r.child.kill("SIGTERM"); await r.exited;
});
// #1873 chat-fail F1: a turn that ended on the harness's error result showed only "adapter exited 1".
// The real claude adapter prints the CLI's final error result on stdout and exits with the CLI's code.
test("chat-fail F1 an error result from the harness reaches the chat line as a plain cause and next step", async t => {
const ctx = await setup(t, {session:{harness:"claude-code", turnEnd:"exit", adapter:new URL("../../../adapters/claude/adapter.sh", import.meta.url).pathname}});
const sessionFile = join(ctx.runDir, "session.json");
const session = JSON.parse(readFileSync(sessionFile, "utf8"));
const bin = join(ctx.dir, "model-fixture");
const result = {type:"result", subtype:"success", is_error:true, result:"You've hit your limit \u00b7 resets 3pm (UTC)"};
writeFileSync(bin, `#!${process.execPath}\nconsole.log(${JSON.stringify(JSON.stringify(result))});\nprocess.exit(1);\n`, {mode:0o755});
session.harnessEnv = {MOSAIC_CLAUDE_BIN:bin, MOSAIC_THINKING:"low", MOSAIC_CLAUDE_SETTINGS:session.bundle.policy, MOSAIC_CLAUDE_MCP_CONFIG:session.bundle.tools};
writeFileSync(sessionFile, JSON.stringify(session));
ctx.inbox = [];
const r = startRunner(ctx.runDir, ctx.cap);
t.after(() => r.child.kill("SIGKILL"));
const sent = ctx.call(ctx.pm, "message.send", {to:"coder", body:"hello", class:"REQUEST"});
const [reply] = await until(replies(ctx));
await until(() => ctx.store.get("SELECT 1 FROM deliveries WHERE message=? AND op='read'", sent.id));
assert.equal(reply.body, "The turn failed: the model account is out of usage. Check Settings > Model setup.\nDetails: adapter exited 1");
assert.match(r.stderr(), /failed: adapter exited 1 \(the model account is out of usage/);
r.child.kill("SIGTERM"); await r.exited;
});
test("chat-fail F1 error text is cleaned, redacted and capped; a held secret withholds it; no text keeps the bare reason", async t => {
const secret = "fixture-secret-0123456789abcdef";
const ctx = await setup(t, {session:{harnessEnv:{MOSAIC_FIXTURE_TOKEN:secret}}});
const adapter = join(ctx.dir, "adapter.sh");
// HOSTILE: ANSI, C0, C1, DEL, bidi and zero-width characters, a link with a query and userinfo, an
// email, a path, a Bearer value, key prefixes, a JWT and a long text. LEAK: the turn's own secret,
// split by a newline and a zero-width space. QUIET: nothing on stdout.
writeFileSync(adapter, [
'case "$MOSAIC_REQUEST" in',
// The long text comes first: the harness prints its error result last, and only the tail is read.
`*HOSTILE*) head -c 2000 /dev/zero | tr '\\0' 'w'; printf '\\n\\033[31mAPI Error: 500\\033[0m \\001\\177\\302\\205 boom \\342\\200\\256evil\\342\\200\\213 https://u:[email protected]/x?key=abc [email protected] /home/someone/.config/x Bearer abc.def sk-ant-api03-zzz msk-abcdef eyJhbGciOi.eyJzdWIi.sig AbCdEf0123456789XyZ\\n'; exit 1 ;;`,
`*LEAK*) printf 'failed: fixture-secret-01234\\n5678\\342\\200\\2139abcdef\\n'; exit 1 ;;`,
`*QUIET*) exit 1 ;;`,
'esac',
].join("\n") + "\n");
ctx.inbox = [];
const r = startRunner(ctx.runDir, ctx.cap);
t.after(() => r.child.kill("SIGKILL"));
for (const body of ["HOSTILE", "LEAK", "QUIET"]) ctx.call(ctx.pm, "message.send", {to:"coder", body});
await until(() => replies(ctx)() && ctx.inbox.length >= 3);
const [hostile, leak, quiet] = ctx.inbox.map(m => m.body);
const [cause, details] = hostile.slice("The turn failed: ".length).split("\nDetails: ");
assert.equal(details, "adapter exited 1");
// The long token before the error is redacted as a whole: it shows, if at all, as one [redacted].
assert.match(cause, /^(?:\[redacted\] )?API Error: 500 boom evil /);
assert.ok(Array.from(cause).length <= 300, cause.length);
assert.equal(cause.includes("www"), false, "the long text before the error is not shown");
assert.doesNotMatch(hostile, /[\u0000-\u0009\u000b-\u001f\u007f-\u009f\u200b-\u200f\u202a-\u202e\u2066-\u2069\ufeff]/u, "no control or format character but the one separating newline");
for (const bad of ["pw@", "example.com", "key=abc", "mail@", "/home/someone", "abc.def", "sk-ant", "msk-abcdef", "eyJhbGciOi", "AbCdEf0123456789XyZ"]) assert.equal(hostile.includes(bad), false, bad);
assert.equal(leak, "The turn failed: the error text held a secret, so it isn't shown. Check Settings > Model setup.\nDetails: adapter exited 1");
assert.equal(quiet, "The turn failed: adapter exited 1");
for (const line of [...ctx.inbox.map(m => m.body), r.stderr()]) assert.equal(line.replace(/\s|\u200b/g, "").includes(secret), false);
r.child.kill("SIGTERM"); await r.exited;
});
test("r127 an adapter whose own command is missing names it; a 127 with no such line keeps the bare reason", async t => {
const ctx = await setup(t);
const adapter = join(ctx.dir, "adapter.sh");
@@ -394,7 +451,8 @@ for (const [harness, exit, activity] of [['claude-code', 1, false], ['claude-cod
const r = startRunner(ctx.runDir, ctx.cap, { MOSAIC_MODEL_ACTIVITY_SOURCE: 'adapter' });
t.after(async () => { r.child.kill('SIGTERM'); await r.exited; });
const reply = await ask(ctx);
if (exit) assert.match(reply.body, /The turn failed: adapter exited 1/);
// #1873 chat-fail F1: the error text on stdout is the turn's plain cause.
if (exit) assert.equal(reply.body, 'The turn failed: API Error: 400 synthetic-result\nDetails: adapter exited 1');
else assert.equal(reply.body, 'API Error: 400 synthetic-result');
const marker = join(ctx.runDir, 'proxy-first-output');
assert.equal(existsSync(marker), activity);
@@ -0,0 +1,114 @@
// #1873 chat-fail F1: the plain cause of a failed turn, from the harness's error text.
import { test } from "node:test";
import assert from "node:assert/strict";
import { CAUSE_MAX, PLAIN, WITHHELD, cleanText, failureBody, secretValues, turnCause } from "../src/failure-notice.mjs";
test("the three recognised causes map to a fixed plain line that carries none of the error's words", () => {
for (const text of ["Claude AI usage limit reached|1700000000", "You've hit your limit \u00b7 resets 3pm (UTC)",
"5-hour limit reached \u2219 resets 9pm", "Credit balance is too low", '{"error":{"type":"insufficient_quota"}}', "Quota exceeded for this account"])
assert.equal(turnCause(text), PLAIN.usage, text);
for (const text of ["OAuth token has expired. Please obtain a new token or refresh your existing token.", "Invalid API key \u00b7 Please run /login",
'API Error: 401 {"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}', "Not logged in"])
assert.equal(turnCause(text), PLAIN.signin, text);
for (const text of ["There's an issue with the selected model (some-model). It may not exist or you may not have access to it.",
'{"error":{"type":"not_found_error","message":"model: some-model"}}', "model_not_found", "The model some-model does not exist"])
assert.equal(turnCause(text), PLAIN.model, text);
// Other limits are not the account's usage.
for (const text of ["Context limit reached \u00b7 /compact or /clear to continue", "output limit reached"])
assert.notEqual(turnCause(text), PLAIN.usage, text);
for (const line of Object.values(PLAIN)) assert.doesNotMatch(line, /\u2014/, "no em dash in UI text");
});
test("no text, or text that is only controls or one long token, gives no cause", () => {
for (const text of [undefined, null, "", " \n\t", "\u0000\u001b[0m\u200b", "x".repeat(5000)]) assert.equal(turnCause(text), null, JSON.stringify(text));
});
test("control, format and separator characters go and the cause is capped", () => {
assert.equal(cleanText("a\u0000b\u0007c\u001b[1;31md\u001b]0;title\u0007e\u007ff\u0085g\u2028h\u2029i\u202ej\u2066k\u200bl\ufeffm\tn\r\no"), "a b cde f g h ijklm n o");
const long = turnCause("word ".repeat(400));
assert.ok(Array.from(long).length <= CAUSE_MAX && long.endsWith("..."), long.length);
// Astral characters are cut whole, never into a lone surrogate.
const astral = turnCause("\u{1f600}".repeat(400));
assert.equal(/[\ud800-\udfff]/u.test(astral.replace(/\u{1f600}/gu, "")), false);
});
test("secrets, links, emails, paths and key-like runs never reach the cause", () => {
const secret = "s3cr3t-value-for-the-fixture";
const secrets = secretValues({ MOSAIC_MODEL_KEY: secret, MOSAIC_PROXY_TOKEN: "tok-0123456789", PATH: "/usr/bin:/bin", SHORT_KEY: "abc" });
assert.deepEqual(secrets, [secret, "tok-0123456789"]);
for (const text of [`boom ${secret}`, `boom ${secret.slice(0, 9)}\n${secret.slice(9)}`, `boom ${secret.slice(0, 5)}\u200b${secret.slice(5)}`, `boom ${secret.slice(0, 5)}\u0000${secret.slice(5)}`])
assert.equal(turnCause(text, secrets), WITHHELD, JSON.stringify(text));
const shown = turnCause("API Error: 500 at https://user:[email protected]/a?token=zz and www.host.example/q, [email protected], /home/u/.config/mosaic-dev/x ~/x ./y "
+ "Bearer abc.def Authorization: Basic Zm9v api_key=k1 token: k2 sk-ant-api03-k3 ghp_k4 xoxb-k5 eyJa.b.c " + "A1".repeat(10) + " " + "z".repeat(40), secrets);
for (const bad of ["user:pw", "host.example", "token=zz", "ops@", "/home/u", "~/x", "./y", "abc.def", "Zm9v", "k1", "k2", "sk-ant", "ghp_", "xoxb", "eyJa", "A1A1", "zzzz"])
assert.equal(shown.includes(bad), false, `${bad} in ${shown}`);
assert.match(shown, /^API Error: 500 at \[redacted\]/);
});
test("the chat body puts the plain cause first and the technical reason after", () => {
assert.equal(failureBody({ reason: "adapter exited 1", cause: PLAIN.usage }), `The turn failed: ${PLAIN.usage}\nDetails: adapter exited 1`);
assert.equal(failureBody({ reason: "adapter exited 1" }), "The turn failed: adapter exited 1");
});
test("review round 1: quoted keys, short or letters-only values and a space separator are redacted", () => {
for (const [text, value] of [['{"error":"bad","password":"hunter2hunte"}', "hunter2hunte"], ['{"api_key":"abcdefghijklmno"}', "abcdefghijklmno"],
['auth failed for key "Zq8xPw2LmN4"', "Zq8xPw2LmN4"], ["token abcd1234efgh567 rejected", "abcd1234efgh567"],
["secret qwertyuiopasdfghjklzxcvbnmqwer", "qwertyuiop"], ['{"access_token":"shortvalue1"}', "shortvalue1"], ["client_secret=abc", "abc"]]) {
const shown = turnCause(text);
assert.ok(shown === null || !shown.includes(value), `${value} in ${shown}`);
}
// Ordinary words after "key" or "session" are left alone.
assert.equal(turnCause("API Error: 500 the key point is the session ended"), "API Error: 500 the key point is the session ended");
});
test("review round 1: encoded emails, Windows and UNC paths and encoded env secrets never reach the cause", () => {
for (const [text, bad] of [["mail jason%40example.com", "example"], ["mail jason%2540example.com", "example"], ["mail jason\uff20example.com", "example"],
["mail jason&#64;example.com", "example"], ["read C:\\Users\\jason\\.claude\\creds.json", "jason"], ["read \\\\server\\share\\creds", "server"]]) {
const shown = turnCause(text);
assert.ok(shown === null || !shown.includes(bad), `${bad} in ${shown}`);
}
const secrets = secretValues({ MOSAIC_PROXY_KEY: "pxy_0123456789abcd" });
for (const text of ["boom pxy%5F0123456789abcd", "boom pxy&#95;0123456789abcd", "boom pxy&#x5f;0123456789abcd", "boom pxy_01234\u030156789abcd", "boom \uff50xy_0123456789abcd"])
assert.equal(turnCause(text, secrets), WITHHELD, JSON.stringify(text));
});
test("review round 1: a long output is read from its end, so an answer that talks about limits is not the cause", () => {
const cause = turnCause("Here is how limits work: usage limit reached means wait. " + "x ".repeat(1500) + "Error: adapter crashed: segfault");
assert.notEqual(cause, PLAIN.usage);
assert.match(cause, /segfault$/);
assert.equal(turnCause("y ".repeat(1500) + "Claude AI usage limit reached"), PLAIN.usage);
});
// Review round 2: the tail cut can land between a key word and its value. Redaction runs
// before the cut, so the value is still redacted with its key word gone.
test("review round 2: a tail cut between a key word and its value never shows the value", () => {
for (const [key, value] of [["password:", "Zq8xPw2LmN"], ["Bearer", "Zq8xPw2LmN"], ['"api_key":', '"Zq8xPw2LmN"'], ["token", "Zq8xPw2LmN4"]]) {
// The last 2000 characters start just before the value, so the cut parts it from its key
// word; a near miss on either side is checked too.
for (let shift = -2; shift <= 2; shift++) {
const end = " then the adapter crashed";
const head = ` ${value} https://e.example/`;
const tail = head + "a".repeat(2000 - head.length - end.length + shift) + end;
const cause = turnCause("w ".repeat(300) + key + tail);
assert.ok(cause !== null && !cause.includes("Zq8xPw2LmN"), `${key} shift ${shift}: ${cause}`);
assert.match(cause, /then the adapter crashed$/);
}
}
});
// Review round 3: redaction runs over the whole kept stdout (up to 128000 characters), so a long
// run with no spaces (base64, a data URI, minified JSON) must not make it slow.
test("review round 3: a 128000-character run with no spaces is redacted in well under a second", () => {
const N = 128000;
for (const run of ["a".repeat(N), "a.".repeat(N / 2), "a-".repeat(N / 2), "a/".repeat(N / 2), "a:".repeat(N / 2), 'a"'.repeat(N / 2)]) {
const started = performance.now();
const cause = turnCause(`${run} then the adapter crashed`);
const took = performance.now() - started;
assert.ok(took < 1000, `${run.slice(0, 4)}... took ${Math.round(took)} ms`);
assert.match(cause, /then the adapter crashed$/);
}
});
// Review round 3: a long text that redacts down below the read limit isn't cut, so its first word stays.
test("review round 3: a long text that redaction makes short keeps its first word", () => {
assert.equal(turnCause(`Crash: adapter died at https://e.example/${"a".repeat(2500)}`), "Crash: adapter died at [redacted]");
});
+1 -1
View File
@@ -64,7 +64,7 @@ code is in that same tree, a restart picks it up. If it is in another tree,
the unit keeps the old PIN until it is installed again from the new tree:
stop it, remove its unit file, then `mosaic proxy unit install` from the new
tree, because install never replaces a unit. Then run `mosaic proxy fetch` (or Fetch on the Console's
Model proxy settings), which installs the new release beside the old one
Model setup page), which installs the new release beside the old one
under `<dataRoot>/opt/cliproxyapi/<version>/`. Then restart the proxy with
`mosaic proxy unit restart`. Before the fetch, the proxy fails to start with
`proxy-install-missing`, and keeps failing until the fetch is done. Restart the bus and
+21 -6
View File
@@ -60,17 +60,24 @@ export function createProxyAdmin({ dataRoot, configDir, instance = null, unitDir
// intent. Any other failure leaves it: the add may have happened.
async function rideOne(rider, wasRidden) {
const id = rider.id, local = decide(rider, wasRidden);
if (local === 'fixture') return { id, outcome: 'fixture', running: !(await stopped(rider)) };
// settings-p3 (R2): with no bus status (the single-business Console) whether its bus runs is unknown (null).
if (local === 'fixture') return { id, outcome: 'fixture', running: typeof rider.status === 'function' ? !(await stopped(rider)) : null };
if (local !== 'ready') return { id, outcome: local === 'already' ? 'already' : 'skipped', reason: local };
let current;
try { current = await rider.call('plugins'); } catch { return { id, outcome: 'skipped', reason: 'bus-down' }; }
try { await rider.call('proxy.accounts'); } catch (error) { return { id, outcome: 'skipped', reason: error?.code === 'not-configured' ? 'no-proxy-client' : 'proxy-check-failed' }; }
const providers = Array.isArray(current?.plugins?.['model-provider']) ? current.plugins['model-provider'] : [];
if (providers.includes('cliproxy')) return { id, outcome: 'already' };
// The receipts are read again here: a removal saved while this run worked on earlier
// organizations is on file now, and the decision above came from the run's first read.
const now = ridden()(id);
if (now !== false) return { id, outcome: 'skipped', reason: now === null ? 'unreadable' : 'removed' };
// The organization (its file and change log) and the receipts are read again here: an add and
// remove saved on its Settings page, or a removal saved while this run worked on earlier
// organizations, is on file now, and the decision above came from the run's first read
// (settings-p3 R1). A read that fails, or an organization no longer listed, is skipped as unreadable.
let fresh;
try { fresh = (await riders()).find(r => r?.id === id); } catch { fresh = null; }
if (!fresh) return { id, outcome: 'skipped', reason: 'unreadable' };
const again = decide(fresh, ridden());
if (again === 'already') return { id, outcome: 'already' };
if (again !== 'ready') return { id, outcome: 'skipped', reason: again };
try { receipt({ op: 'ride-intent', org: id }); } catch { return { id, outcome: 'skipped', reason: 'not-recorded' }; }
try { await rider.call('plugins.set', { revision: current.revision, plugins: { 'model-provider': [...providers, 'cliproxy'] } }); }
catch (error) {
@@ -83,7 +90,12 @@ export function createProxyAdmin({ dataRoot, configDir, instance = null, unitDir
return { id, outcome: 'added' };
}
async function rideSummary() {
try { const wasRidden = ridden(); return (await riders()).map(r => ({ id: r.id, state: decide(r, wasRidden) })); } catch { return null; }
// settings-p3 (review P3-3): a stand-in organization with no bus status (the single-business
// Console) can't be moved; running null says so before any Ride all run.
try {
const wasRidden = ridden();
return (await riders()).map(r => { const state = decide(r, wasRidden); return state === 'fixture' && typeof r.status !== 'function' ? { id: r.id, state, running: null } : { id: r.id, state }; });
} catch { return null; }
}
function nextSteps({ release, init, unitState, fence, lock }) {
if (!release.fetched || release.problem) return ['fetch'];
@@ -221,6 +233,9 @@ export function createProxyAdmin({ dataRoot, configDir, instance = null, unitDir
if (!rider) return { org, moved: false, reason: 'unknown' };
const local = decide(rider, ridden());
if (local !== 'fixture') return { org, moved: false, reason: local };
// settings-p3 (R2): with no bus status (the single-business Console) its bus can't be shown
// stopped, so stopping it would not help: moving is not available there.
if (typeof rider.status !== 'function') return { org, moved: false, reason: 'not-available' };
if (!(await stopped(rider))) return { org, moved: false, reason: 'bus-running' };
if (typeof move !== 'function') return { org, moved: false, reason: 'not-available' };
const owner = await ownerModels();
@@ -102,10 +102,12 @@ test('after Ride all the launcher asks admission for a direct anthropic route an
};
const launchOf = async riding => {
const root = tmp(t);
// A Claude model on a legacy entry counts as OAuth and runs on claude-code only. The stand-in adapter
// path skips only the binary check; the launch stops at authorizeLaunch, before any adapter runs.
const f = fixture(root, 'acme', doc => {
doc.plugins = { ...doc.plugins, harness: ['pi'] };
doc.plugins = { ...doc.plugins, harness: ['claude-code'] };
doc.launch.models['claude-opus-5-5'] = { provider: 'anthropic', model: 'claude-opus-5-5' };
doc.roles.coder.vars = { harness: 'pi', model: 'claude-opus-5-5', thinking: 'low' };
doc.roles.coder.vars = { harness: 'claude-code', model: 'claude-opus-5-5', thinking: 'low' };
return doc;
});
if (riding) await ride(join(root, 'config', 'businesses', 'acme.json'));
@@ -119,7 +121,7 @@ test('after Ride all the launcher asks admission for a direct anthropic route an
if (x.op === 'authorizeLaunch') throw Object.assign(Error('fixture authorization stop'), { code: 'fixture-policy' });
return {};
} };
const makeLoader = await fakeHarnesses(t, { pi: null }, write => write('model-provider', 'cliproxy', `export * from ${JSON.stringify(new URL('../../plugins/model-provider/cliproxy/index.mjs', import.meta.url).href)};`));
const makeLoader = await fakeHarnesses(t, { 'claude-code': '/fixture-unused' }, write => write('model-provider', 'cliproxy', `export * from ${JSON.stringify(new URL('../../plugins/model-provider/cliproxy/index.mjs', import.meta.url).href)};`));
const system = { ...JSON.parse(readFileSync(f.config, 'utf8')), proxy: { port: 1, binary: '/fixture-unused' } };
const launcher = createLauncher({ host, system, env: f.env, makeLoader, bwrap: '/usr/bin/true', log: () => {} });
return { answer: await launcher.fromSession('fixture-cap', 'coder'), ops };
+45 -3
View File
@@ -201,6 +201,43 @@ test('a removal saved while Ride all works on earlier organizations is read befo
assert.equal(g2.calls.some(([v]) => v === 'plugins.set'), false);
});
test('settings-p3 R1: the change log is read again before each add, and a read that fails skips the add', async t => {
// While Ride all works on 'first', the owner adds cliproxy to 'late' on its Settings page and takes it
// off again. Only its change log shows that; the receipts do not. The run's first read showed nothing.
let lateLog = '';
const first = bus({ providers: [] }), late = bus({ providers: [] });
const firstCall = async (verb, args) => { if (verb === 'plugins') lateLog = committed([{ path: 'plugins.model-provider', old: [], new: ['cliproxy'] }]) + committed([{ path: 'plugins.model-provider', old: ['cliproxy'], new: [] }]); return first.call(verb, args); };
const f = admin(t, { riders: async () => [{ id: 'first', fixture: false, cliproxy: false, changes: '', call: firstCall }, { id: 'late', fixture: false, cliproxy: false, changes: lateLog, call: late.call }] });
const r = await f.admin.rideAll();
assert.deepEqual(r.added, ['first']); assert.deepEqual(r.skipped, [{ id: 'late', reason: 'removed' }]);
assert.equal(late.calls.some(([v]) => v === 'plugins.set'), false);
assert.deepEqual(receipts(f.dataRoot).filter(x => x.org === 'late').map(x => x.op), []);
// Fail closed: a change log that can no longer be read, an organization that is no longer listed,
// or a list that can't be read again skips the add as unreadable, with no intent and no plugins.set.
for (const [name, again] of [
['log unreadable', b => [{ id: 'o', fixture: false, cliproxy: false, changes: null, call: b.call }]],
['no longer listed', () => []],
['list throws', () => { throw new Error('read failed'); }],
]) {
const b = bus({ providers: [] });
let reads = 0;
const g = admin(t, { riders: async () => (reads++ === 0 ? [{ id: 'o', fixture: false, cliproxy: false, changes: '', call: b.call }] : again(b)) });
assert.deepEqual((await g.admin.rideAll()).skipped, [{ id: 'o', reason: 'unreadable' }], name);
assert.equal(b.calls.some(([v]) => v === 'plugins.set'), false, name);
assert.deepEqual(receipts(g.dataRoot).filter(x => x.op === 'ride-intent'), [], name);
}
});
test('settings-p3 review P3-1: an organization that took cliproxy during the run reads as already using it, with no add', async t => {
const b = bus({ providers: [] });
let reads = 0;
const f = admin(t, { riders: async () => [{ id: 'o', fixture: false, cliproxy: reads++ > 0, changes: '', call: b.call }] });
const r = await f.admin.rideAll();
assert.deepEqual([r.added, r.already, r.skipped], [[], ['o'], []]);
assert.equal(b.calls.some(([v]) => v === 'plugins.set'), false);
assert.deepEqual(receipts(f.dataRoot).filter(x => x.op === 'ride-intent'), []);
});
test('an intent receipt that cannot be written stops that add before plugins.set', async t => {
const b = bus({ providers: [] });
const f = admin(t, { riders: async () => [{ id: 'w', fixture: false, cliproxy: false, changes: '', call: b.call }] });
@@ -388,10 +425,15 @@ test('a fixture organization moves only when stopped, with a proxy, a riding bus
assert.deepEqual(await f.admin.rideMove({ org: 'nope', principal }), { org: 'nope', moved: false, reason: 'unknown' });
assert.deepEqual(await f.admin.rideMove({ org: 'real', principal }), { org: 'real', moved: false, reason: 'ready' });
assert.deepEqual(await f.admin.rideMove({ org: 'live', principal }), { org: 'live', moved: false, reason: 'bus-running' });
// A slow or busy bus that does not answer, a status that cannot be read, or no status at all is running.
for (const org of ['busy', 'unproven', 'no-status']) assert.deepEqual(await f.admin.rideMove({ org, principal }), { org, moved: false, reason: 'bus-running' }, org);
// A slow or busy bus that does not answer, or a status that cannot be read, is running.
for (const org of ['busy', 'unproven']) assert.deepEqual(await f.admin.rideMove({ org, principal }), { org, moved: false, reason: 'bus-running' }, org);
// settings-p3 R2: no status at all (the single-business Console) can never show the bus stopped, so
// "stop its bus" would not help: moving is not available there, and whether it runs is unknown.
assert.deepEqual(await f.admin.rideMove({ org: 'no-status', principal }), { org: 'no-status', moved: false, reason: 'not-available' });
const listed = await f.admin.rideAll();
assert.deepEqual(listed.fixtures, [{ id: 'stand-in', running: false }, { id: 'live', running: true }, { id: 'busy', running: true }, { id: 'unproven', running: true }, { id: 'no-status', running: true }]);
assert.deepEqual(listed.fixtures, [{ id: 'stand-in', running: false }, { id: 'live', running: true }, { id: 'busy', running: true }, { id: 'unproven', running: true }, { id: 'no-status', running: null }]);
// settings-p3 review P3-3: the status summary says so before any run, without asking any bus.
assert.deepEqual((await f.admin.status()).ride.filter(r => r.state === 'fixture'), [{ id: 'stand-in', state: 'fixture' }, { id: 'live', state: 'fixture' }, { id: 'busy', state: 'fixture' }, { id: 'unproven', state: 'fixture' }, { id: 'no-status', state: 'fixture', running: null }]);
ownerNow = { ...owner, proxy: false };
assert.deepEqual(await f.admin.rideMove({ org: 'stand-in', principal }), { org: 'stand-in', moved: false, reason: 'no-proxy' });
ownerNow = { ...owner, riding: false };
+6 -1
View File
@@ -10,10 +10,15 @@ export { createProxyAdmin };
// #1873: one organization's catalogue row for the setup page. A count is shown only when the bus
// answered with a model list; otherwise models is null and the page shows a dash, never a made-up 0.
const RESTART_TO_LAUNCH = 'Restart this organization to start agents on these models.';
export async function orgCatalogue(call) {
try {
const value = await call('routes');
return { status: value?.catalogue?.status ?? 'unknown', models: Array.isArray(value?.catalogue?.models) ? value.catalogue.models.length : null };
const status = value?.catalogue?.status ?? 'unknown';
// settings-p3 (re-check 3 P3-5): a bus that picked the service up after it started reads ready but
// starts agents on these models only after a restart (bus runtime.mjs RESTART_TO_LAUNCH).
const launchRestart = status === 'ready' && Array.isArray(value.catalogue.notices) && value.catalogue.notices.includes(RESTART_TO_LAUNCH);
return { status, models: Array.isArray(value?.catalogue?.models) ? value.catalogue.models.length : null, ...(launchRestart ? { launchRestart: true } : {}) };
} catch { return { status: 'bus-unavailable', models: null }; }
}
+9 -5
View File
@@ -1378,7 +1378,9 @@
// fixture-only catalogue is ready with no model proxy). Before the setting is read it falls
// back to the catalogue status, as before.
const proxyOn=harnessSettings?(Array.isArray(harnessSettings.plugins?.['model-provider'])?harnessSettings.plugins['model-provider']:[]).includes('cliproxy'):v.catalogue?.status!=='disabled';
const catalogueInfo=v.catalogue?.status==='ready'?`<p class="small muted">Model catalogue, observed ${txt(window.MosaicLocalTime(v.catalogue.observedAt))}.</p>${(v.catalogue.notices??[]).map(n=>`<p class="small warn">${txt(n)}</p>`).join('')}`:v.catalogue?.status==='unavailable'?catalogueLine(proxyOn,v.catalogue):'<p class="small muted">Model catalogue not enabled.</p>';
// settings-p3 (re-check 3 P3-5): a ready catalogue the bus picked up late says agents start on these
// models after a restart; that notice gets the organization's Restart (not primary: Save routes is).
const catalogueInfo=v.catalogue?.status==='ready'?`<p class="small muted">Model catalogue, observed ${txt(window.MosaicLocalTime(v.catalogue.observedAt))}.</p>${(v.catalogue.notices??[]).map(n=>`<p class="small warn">${txt(n)}${n===window.MosaicConsole?.RESTART_TO_LAUNCH&&window.MosaicOrg?.id?` ${window.MosaicConsole.orgRestartButton(window.MosaicOrg.id,{name:window.MosaicOrg.label||window.MosaicOrg.id})}`:''}</p>`).join('')}`:v.catalogue?.status==='unavailable'?catalogueLine(proxyOn,v.catalogue):'<p class="small muted">Model catalogue not enabled.</p>';
const providerControl=canSend&&!isTester?`<button type="button" id="provider-switch" data-enabled="${proxyOn}">${proxyOn?'Stop using the model service':'Use your models'}</button><p role="status" id="provider-switch-status"></p>`:'';
const harnessControl=canSend&&!isTester?`<fieldset class="harness-control"><legend>Harnesses</legend>${harnessSettings ? [...new Set([...harnessSettings.available.harness,...(harnessSettings.plugins.harness??[])])].map(name=>`<label><input type="checkbox" data-harness="${esc(name)}"${(harnessDraft??harnessSettings.plugins.harness??[]).includes(name)?' checked':''}${harnessBusy?' disabled':''}> ${txt(name)}</label>`).join('')+` <button type="button" id="harness-save"${harnessBusy?' disabled':''}>Save harnesses</button>` : '<p>Harnesses unavailable.</p><button type="button" id="harness-read">Read harnesses again</button>'}<p role="status" id="harness-status">${txt(harnessReceipt)}</p></fieldset>`:'';
const off=v.harnesses.length?'':'<p class="small warn">No harness is switched on for this organization, so every launch is refused, and so is a save that changes a harness. Switch one on under Harnesses.</p>';
@@ -1507,7 +1509,7 @@
if (page==='tokens' && s.launch && !s.launch.refused) takeSnapshot(s.launch);
const notify = typeof n.binding==='string' ? `<code>${txt(n.binding)}</code>` : n.refused ? `Could not be read: <code>${txt(n.refused)}</code>` : '<span class="muted">None set</span>';
const contents={limits:()=>launchForm(s.launch),routing:()=>routesSection(routes,instances),agents:()=>agentsSection(instances)+agentLevels,assistant:()=>assistantSection(s.launch,names),tokens:()=>tokenLimits(limitRows,limitsRefused)+tokenUsage(usage),names:()=>testerNames(names,instances),about:()=>`<section class="panel"><dl class="kv"><dt>Address</dt><dd><code>${txt(s.address)||'unknown'}</code></dd><dt>Release</dt><dd>${txt(s.release)||'unknown'}</dd><dt>Business</dt><dd>${txt(s.business)||none}</dd><dt>Board</dt><dd><code>${txt(s.board)||'unknown'}</code></dd><dt>Notifier binding</dt><dd>${notify}</dd></dl></section>`, 'model-proxy':()=>'<section class="panel"><a href="#/settings/model-proxy">Open Model setup</a></section>'};
const helpers={limits:'Leave a limit blank for no limit. Zero stops new launches. Running sessions keep running when limits or models change.',routing:"Each agent’s harness, model and thinking, and the fallback a launch takes when the primary harness fails its preflight. A fallback sets all three or none. Running sessions keep their route. The next launch reads the change. A harness is the program that runs the model, such as Claude Code. Preflight is the check a launch makes before it starts.",agents:'Each agent with its definition, launch membership and runs. A launch refuses until the agent has a route in Routing.',assistant:'The definition stays an assistant one. Holder and credentials stay hand edits.',tokens:'Token limits the bus checks, and token use read from session transcripts.'};
const helpers={limits:'Leave a limit blank for no limit. Zero stops new launches. Running sessions keep running when limits or models change.',routing:"Each agent’s harness, model and thinking, and the fallback a launch takes when the primary harness fails its preflight. A fallback sets all three or none. Running sessions keep their route. The next launch reads the change. A harness is the program that runs the model. Preflight is the check a launch makes before it starts.",agents:'Each agent with its definition, launch membership and runs. A launch refuses until the agent has a route in Routing.',assistant:'The definition stays an assistant one. Holder and credentials stay hand edits.',tokens:'Token limits the bus checks, and token use read from session transcripts.'};
return settingsShell(page,contents[page](),`${get.oldest()?`Read ${when(get.oldest())}.`: 'Reading…'} ${helpers[page]||''}`);
}
let settingsFocusDisclosure=false;
@@ -1730,11 +1732,13 @@
const page = path.split('/')[2] || 'appearance', id = window.MosaicOrg?.id, row = id ? window.MosaicOrg.row?.(id) : null;
const org = txt(window.MosaicOrg?.label || id || ''), name = settingsPages.find(([p]) => p === page)?.[1] || 'this page';
if (id && row?.busHealthy === false && !refusal(err)) {
const restart = window.MosaicConsole?.orgRestartButton?.(id, { name: window.MosaicOrg.label || id, primary: true }) || '';
const state = row.busStopped ? `${org} is stopped` : `${org} isn't running`;
// Why it is down, from the read the dashboard row uses (MosaicConsole.orgDownReason).
const label = window.MosaicOrg.label || id, whyText = v => !v ? '' : v.plain && v.plain !== `${label} did not start.` ? txt(v.plain) : `Why it stopped: ${txt(v.detail)}`;
// Why it is down, from the read the dashboard row uses (MosaicConsole.orgDownReason). Only a
// refused start is a reason it stopped; anything else is what the last check saw (P3-9).
const label = window.MosaicOrg.label || id, whyText = v => !v ? '' : v.plain && v.plain !== `${label} did not start.` ? txt(v.plain) : `${v.refused === false ? 'Last check' : 'Why it stopped'}: ${txt(v.detail)}`;
const why = window.MosaicConsole?.orgDownReason?.(id, label, v => { const p = document.querySelector(`[data-org-down="${CSS.escape(id)}"] [data-org-down-why]`); if (p) { p.innerHTML = whyText(v); p.hidden = !p.innerHTML; } }) ?? null;
// settings-p3 (re-check 3 P3-4): after a refusal a restart can't clear, Restart is not the primary.
const restart = window.MosaicConsole?.orgRestartButton?.(id, { name: label, primary: !(window.MosaicConsole?.orgRefusal?.(id)?.restart === false || why?.restart === false) }) || '';
return settingsShell(page, `<section class="panel org-down" data-org-down="${txt(id)}"><div class="banner err" role="alert"><p><b>${state}, so ${txt(name)} can't be read.</b> Start it again to see and change this page. Nothing was written.</p><p data-org-down-why${why ? '' : ' hidden'}>${whyText(why)}</p>${restart ? `<p>${restart}</p>` : '<p class="small">Ask the instance owner to restart it.</p>'}</div><p class="small">This instance's own pages still work: <a href="#/settings/appearance">Appearance</a>, <a href="#/settings/model-proxy">Model setup</a>${triageOwner ? ', <a href="#/settings/triage">Triage</a>' : ''} and <a href="#/settings/console">Console</a>.</p></section>`);
}
return settingsShell(page, `<div class="banner ${refusal(err) ? 'ref' : 'err'}" role="alert"><p><b>${txt(failTitle(err))}.</b> ${refusal(err) ? `<code>${txt(err.code)}</code>. ` : ''}${why(err, err.code === 'queue-refused')} ${NO_GUESS}</p>${retry}</div>`);
+6 -1
View File
@@ -186,7 +186,12 @@
return `<ul class="chat-cites">${list.map(c => `<li>${c.badge ? `<span class="chat-scope chat-scope-${c.badge}">${txt(c.badge)}</span> ` : ''}${c.link ? `<a href="${esc(c.link)}" rel="noopener noreferrer" target="_blank">${txt(c.ref)}</a>` : `<code>${txt(c.ref)}</code>`} <q>${txt(c.quote)}</q>${c.agent ? ` <span class="chat-label chat-agent-written">agent-written: ${txt(c.agent)}</span>` : ''}${c.changed ? ' <span class="chat-label chat-changed">changed since</span>' : ''}</li>`).join('')}</ul>`;
}
function messageBody(m) {
if (m.from_role !== 'human' && typeof m.body === 'string' && m.body.startsWith('The turn failed: ')) return `<div class="chat-body">The assistant couldn't reply. Try again, or check Agents.<div class="small">Details: ${txt(m.body.slice('The turn failed: '.length))}</div></div>`;
if (m.from_role !== 'human' && typeof m.body === 'string' && m.body.startsWith('The turn failed: ')) {
// #1873 chat-fail: "The turn failed: <plain cause and next step>\nDetails: <technical reason>".
const rest = m.body.slice('The turn failed: '.length), at = rest.indexOf('\nDetails: ');
if (at > 0) return `<div class="chat-body">The assistant couldn't reply: ${txt(rest.slice(0, at))}<div class="small">Details: ${txt(rest.slice(at + '\nDetails: '.length))}</div></div>`;
return `<div class="chat-body">The assistant couldn't reply. Try again, or check Agents.<div class="small">Details: ${txt(rest)}</div></div>`;
}
const r = renderReply?.(m.body, { check: m.check ?? null, name: m.from_role, changed: changedBy(m.check) });
if (r?.kind === 'reply') return `<div class="chat-body">${prose(r.text)}</div>${r.citations.length ? cites(r.citations) : ''}`;
if (r?.kind === 'notice') {
+26 -10
View File
@@ -207,7 +207,9 @@
// The refusal the bus host wrote, with one plain line in front when its code is known.
function refusalLine(name, message) {
const text = String(message || 'The restart was refused. Nothing was retried.');
if (/\bfixture-proxy-configured\b/.test(text)) return { plain: `${name} runs on test models, which can't share this machine with your model service. It did not start.`, detail: text };
// settings-p3 (re-check 3 P3-4): a restart is refused the same way again, so `restart: false`
// tells the page not to keep Restart as its primary action.
if (/\bfixture-proxy-configured\b/.test(text)) return { plain: `${name} runs on test models, which can't share this machine with your model service. It did not start.`, detail: text, restart: false };
return { plain: `${name} did not start.`, detail: text };
}
// #1873 review P1-1, P2-2: the cause a running bus gives for a catalogue it cannot read
@@ -218,7 +220,7 @@
// plain line, so the page shows the bus's words.
function catalogueCause(name, notices, reason) {
// Two different models that slug to one key: the catalogue names it in `reason`, with no notice.
if (reason === 'model-key-collision') return { code: reason, plain: `Two of your models get the same short name, so ${name} can't tell them apart. A restart won't change that.`, restart: false, fix: 'service', detail: reason };
if (reason === 'model-key-collision') return { code: reason, plain: `Two of your models get the same short name, so ${name} can't tell them apart. A restart won't change that. Take one of them out of an API key's Models list on Model setup. If both come from signed-in accounts, this can't be changed from Console yet.`, restart: false, fix: 'service', detail: reason };
const list = (Array.isArray(notices) ? notices : []).filter(n => typeof n === 'string' && n);
if (!list.length) return null;
const detail = list.join(' '), code = /\(([a-z][a-z-]*)\)/.exec(detail)?.[1] ?? (/has not been set up/.test(detail) ? 'proxy-not-set-up' : null);
@@ -240,8 +242,8 @@
return `<span class="org-restart" data-org-restart-box="${esc(id)}"><button type="button" class="btn${primary ? ' primary' : ''}" data-org-restart="${esc(id)}" data-org-name="${esc(name)}" data-label="${esc(label)}"${primary ? ' data-primary' : ''}${busy ? ' disabled' : ''}>${esc(busy ? 'Restarting…' : label)}</button> <span role="status" data-org-restart-receipt class="${r && !r.ok ? 'org-restart-refused' : ''}">${receipt}</span></span>`;
}
const paintOrg = id => document.querySelectorAll(`[data-org-restart-box="${CSS.escape(id)}"]`).forEach(el => {
const b = el.querySelector('[data-org-restart]');
el.outerHTML = orgRestartButton(id, { name: b?.dataset.orgName || id, label: b?.dataset.label, primary: b?.hasAttribute('data-primary') });
const b = el.querySelector('[data-org-restart]'), r = orgReceipts.get(id);
el.outerHTML = orgRestartButton(id, { name: b?.dataset.orgName || id, label: b?.dataset.label, primary: b?.hasAttribute('data-primary') && r?.restart !== false });
});
async function restartOrg(id, name) {
if (orgBusy.has(id) || !orgAllowed()) return null;
@@ -275,12 +277,21 @@
// (workspace-read.mjs). It is the reason the dashboard row shows. One read per organization at a
// time, kept for 15 seconds; until it lands this returns what was read before, or null. A reason
// that only restates "not running" is not a reason.
// settings-p3 (re-check 3 P3-9): only a refused start is a stop reason (`refused: true`); any other
// reason (a slow or failed read, a host that does not answer yet) is what the last check saw.
// P3-11: the raw reason is kept and each caller's line is built with its own name when used, and
// every caller that asked while the read was pending is told when it lands.
const downReasons = new Map();
const downLine = (raw, name) => !raw ? null : raw.refused ? { ...refusalLine(name, raw.text), refused: true } : { plain: null, detail: raw.text, refused: false };
function orgDownReason(id, name = id, onRead = null) {
if (typeof id !== 'string' || !id) return null;
const got = downReasons.get(id);
if (got && (got.pending || Date.now() - got.at < 15000)) return got.value ?? null;
downReasons.set(id, { at: got?.at ?? 0, value: got?.value ?? null, pending: true });
if (got && (got.pending || Date.now() - got.at < 15000)) {
if (got.pending && typeof onRead === 'function') got.waiters.push([name, onRead]);
return downLine(got.raw, name);
}
const entry = { at: got?.at ?? 0, raw: got?.raw ?? null, pending: true, waiters: typeof onRead === 'function' ? [[name, onRead]] : [] };
downReasons.set(id, entry);
nativeFetch(`/api/workspace?${new URLSearchParams({ org: id, sections: 'tasks' })}`, { cache: 'no-store', redirect: 'manual' })
.then(r => (r.ok ? r.json() : null))
.then(body => {
@@ -288,11 +299,16 @@
const reason = row && row.state !== 'ok' && typeof row.reason === 'string' ? row.reason.trim() : '';
if (!reason || /^the bus is not (running|answering)\.?$/i.test(reason)) return null;
const refused = /^the bus refused to start: ([\s\S]+)$/.exec(reason);
return refused ? refusalLine(name, refused[1]) : { plain: null, detail: reason };
return refused ? { refused: true, text: refused[1] } : { refused: false, text: reason };
})
.catch(() => null)
.then(value => { downReasons.set(id, { at: Date.now(), value, pending: false }); onRead?.(value); });
return got?.value ?? null;
.then(raw => {
downReasons.set(id, { at: Date.now(), raw, pending: false, waiters: [] });
for (const [who, callback] of entry.waiters) { try { callback(downLine(raw, who)); } catch { /* one caller's repaint does not stop the others */ } }
});
return downLine(got?.raw ?? null, name);
}
window.MosaicConsole = Object.freeze({ ready, settingsMarkup, busButton, attachBus, orgRestartButton, restartOrg, catalogueCause, orgDownReason, orgRefusal: id => orgReceipts.get(id) ?? null, get busRestart() { return busAllowed(); }, get config() { return config; } });
// The notice the bus adds to a ready catalogue it picked up after it started (bus runtime.mjs RESTART_TO_LAUNCH).
const RESTART_TO_LAUNCH = 'Restart this organization to start agents on these models.';
window.MosaicConsole = Object.freeze({ RESTART_TO_LAUNCH, ready, settingsMarkup, busButton, attachBus, orgRestartButton, restartOrg, catalogueCause, orgDownReason, orgRefusal: id => orgReceipts.get(id) ?? null, get busRestart() { return busAllowed(); }, get config() { return config; } });
})();
+42 -29
View File
@@ -170,7 +170,7 @@ window.MosaicLocalTime = value => Number.isFinite(Date.parse(value)) ? new Date(
const bus = notRead(o.bus) ? 'bus not read' : refused ? `not running: its last start failed at ${when(o.bus.lastStart.at)} (bus ${clean(o.bus.state) || 'not read'})${why ? '' : '. Restart it on the dashboard to see why'}` : `bus ${clean(o.bus.state) || 'not read'}`;
const proc = notRead(o.processing) ? 'processing not read' : o.processing.invalid === true ? 'processing state not valid' : o.processing.halted === true ? 'processing stopped' : 'processing on';
// A known refusal reads as its plain line; any other shows the bus's words.
const reason = !why ? '' : why.plain && why.plain !== `${orgName(o.id)} did not start.` ? ` ${why.plain}` : ` Why it stopped: ${clean(why.detail).replace(/\.$/, '')}.`;
const reason = !why ? '' : why.plain && why.plain !== `${orgName(o.id)} did not start.` ? ` ${why.plain}` : ` ${why.refused === false ? 'Last check' : 'Why it stopped'}: ${clean(why.detail).replace(/\.$/, '')}.`;
lines.push(`${orgName(o.id)}: ${bus}, ${proc}, config ${configWords(o.config)}${reason ? `.${reason}` : ''}`);
}
return lines;
@@ -568,8 +568,8 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
+ '<p>The model service lets your organizations use your own model accounts. Each step is checked on this machine. Keys are never shown.</p>'
+ '<div id="proxy-setup-next" class="proxy-setup-next" role="status" aria-live="polite"></div>'
+ '<p id="proxy-setup-message" role="status" aria-live="polite"></p>'
+ '<div id="proxy-setup-restart" class="proxy-setup-restart" hidden><h3 id="proxy-setup-restart-title">Organizations that can\'t see your models</h3>'
+ '<p class="small muted">An organization connects to the model service the next time it reads its models. These still can\'t see your models, or are not running. Each one says why. Where a restart helps, it has a Restart button: the organization stops for a moment and starts again.</p>'
+ '<div id="proxy-setup-restart" class="proxy-setup-restart" hidden><h3 id="proxy-setup-restart-title">Organizations that can\'t use your models yet</h3>'
+ '<p class="small muted">An organization connects to the model service the next time it reads its models. These still can\'t see your models, need a restart before their agents start on them, or are not running. Each one says why. Where a restart helps, it has a Restart button: the organization stops for a moment and starts again.</p>'
+ '<ul id="proxy-setup-restart-list" class="proxy-setup-restart-list" aria-labelledby="proxy-setup-restart-title"></ul></div>'
+ '<div id="proxy-ride" class="proxy-ride" hidden><h3>Use your models in every organization</h3><p id="proxy-ride-text"></p>'
+ '<button type="button" id="proxy-ride-go" class="btn">Ride all</button><p id="proxy-ride-message" role="status" aria-live="polite"></p>'
@@ -791,13 +791,15 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
row(dl, 'Init lock', !initLock.present ? 'None' : initLock.clearable ? 'Stale: the bus is stopped and no model service files exist' : `Held: ${initLock.blockers.join('; ')}`);
if (state.restart?.confirmed) row(dl, 'Last restart', `Confirmed at ${window.MosaicLocalTime(state.restart.observedAt)}, new pid ${state.restart.pid}`);
}
// settings-p3 (re-check 3 P3-1): where the hand steps for a held init lock or a partial init are written.
const README_STEPS = ['The hand steps for a stale init lock and a partial init are in ', 'packages/model-proxy/README.md'];
function paintNext() {
const next = state.next?.[0], box = $('proxy-setup-next');
const steps = {
fetch: ['Fetch the pinned release', `Next: fetch CLIProxyAPI ${state.release.version}. Every file is checked against its pinned SHA-256 before it is installed.`, fetchRelease],
'init-lock-clear': ['Clear the stale init lock', 'Next: an init lock is left from an init that did not finish. The bus is stopped and no model service files exist, so it can be cleared.', clearLock],
'init-lock-wait': ['Check again', `An init lock is held and cannot be cleared: ${state.initLock.blockers.join('; ')}. Stop the bus and remove partial state as the model-proxy README says, then check again.`, load],
'init-repair': ['Check again', 'Setup is partly done. Init never overwrites existing model service files; resolve them as the model-proxy README says, then check again.', load],
'init-lock-wait': ['Check again', `An init lock is held and cannot be cleared: ${state.initLock.blockers.join('; ')}. Stop the bus, then check again. Model service files that are left must be resolved by hand. Do not delete them to make init run again.`, load, README_STEPS],
'init-repair': ['Check again', 'Setup is partly done. Init never overwrites model service files that already exist, so they must be resolved by hand. Do not delete them to make init run again. Then check again.', load, README_STEPS],
init: ['Initialize the model service', 'Next: initialize. The bus generates the management key unless you paste your own.', () => { $('proxy-setup-init-box').open = true; form.elements.port.focus(); }],
'unit-install': ['Install the model service', `Next: install ${state.unit.unit}.`, () => unitAction('install')],
'unit-start': ['Start the model service', `Next: start ${state.unit.unit}.`, () => unitAction('start')],
@@ -806,23 +808,24 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
'fence-clear': ['Clear the fence', 'The model service was restarted from this page and the new process answers on its port. Clear the fence with that evidence.', clearFence],
accounts: ['Add an account', state.next?.includes('ride-all') ? 'The model service is running. Sign in to a model account or add an API key below. Then use Ride all below.' : 'The model service is running. Sign in to a model account or add an API key below.', () => $('proxy-signin-title')?.scrollIntoView({ block: 'start' })],
'triage-route': ['Open Triage settings', 'Next: choose the model Triage uses. Triage is the assistant that answers questions about this instance.', () => { location.hash = '#/settings/triage'; }],
'ride-all': ['Ride all', 'Next: let your organizations use your models. See Ride all below.', () => $('proxy-ride')?.scrollIntoView({ block: 'start' })],
'ride-all': ['Go to Ride all', 'Next: let your organizations use your models. Ride all is below.', () => $('proxy-ride')?.scrollIntoView({ block: 'start' })],
};
const step = next === 'accounts' ? afterAccounts() : next;
if (['restart-orgs', 'checking-orgs', 'orgs-blocked', 'done'].includes(step)) {
const names = needRestart().filter(restartHelps).map(o => nameOf(o.id));
const p = document.createElement('p'); p.dataset.next = step;
p.textContent = step === 'done' ? 'The model service is set up and every organization can see your models.'
// settings-p3 (rebase re-check P3-1): the Ride all box below still lists an organization, so not "every".
p.textContent = step === 'done' ? (rideListed() ? 'The model service is set up. Some organizations still need a step under Use your models in every organization.' : 'The model service is set up and every organization can see your models.')
: step === 'checking-orgs' ? 'Checking why each organization can\'t see your models.'
: step === 'orgs-blocked' ? 'The model service is set up. Some organizations still can\'t see your models. Each one says why under Organizations that can\'t see your models.'
: `Next: restart ${names.length === 1 ? names[0] : `${names.slice(0, -1).join(', ')} and ${names.at(-1)}`}. ${names.length === 1 ? 'It' : 'Each'} has a Restart button under Organizations that can't see your models.`;
: step === 'orgs-blocked' ? 'The model service is set up. Some organizations still can\'t see your models. Each one says why under Organizations that can\'t use your models yet.'
: `Next: restart ${names.length === 1 ? names[0] : `${names.slice(0, -1).join(', ')} and ${names.at(-1)}`}. ${names.length === 1 ? 'It' : 'Each'} has a Restart button under Organizations that can't use your models yet.`;
box.replaceChildren(p); return;
}
const [label, text, action, command] = steps[step] ?? ['Check again', 'Status read.', load];
const p = document.createElement('p'); p.textContent = text;
// A step only a terminal can finish keeps its exact command under Technical details.
const cli = command ? Object.assign(document.createElement('details'), { className: 'proxy-setup-cli' }) : null;
if (cli) { const s = document.createElement('summary'), q = document.createElement('p'), c = document.createElement('code'); s.textContent = 'Technical details'; q.className = 'small muted'; q.textContent = 'Run this in a terminal on this machine: '; c.textContent = command; q.append(c); cli.append(s, q); }
if (cli) { const s = document.createElement('summary'), q = document.createElement('p'), c = document.createElement('code'), [lead, code] = Array.isArray(command) ? command : ['Run this in a terminal on this machine: ', command]; s.textContent = 'Technical details'; q.className = 'small muted'; q.textContent = lead; c.textContent = code; q.append(c); cli.append(s, q); }
const button = document.createElement('button'); button.type = 'button'; button.className = 'btn primary'; button.id = 'proxy-setup-go'; button.dataset.next = step ?? '';
button.textContent = state.busy ? 'Working…' : label;
const write = !['init-lock-wait', 'init-repair', 'accounts', 'init', 'ride-all', 'triage-route'].includes(step);
@@ -835,7 +838,10 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
const running = () => state?.next?.[0] === 'accounts';
// An organization that cannot see the models while the service runs. "disabled" is left out: that
// organization does not use the model service, and a restart would not change it.
const needRestart = () => (running() && Array.isArray(state.organizations) ? state.organizations : []).filter(o => ['unavailable', 'bus-unavailable'].includes(o.catalogue?.status));
// settings-p3 (re-check 3 P3-5): a ready catalogue whose bus picked up the service late says a
// restart is needed before agents start on these models (status: launchRestart). It is listed too.
const launchRestart = o => o.catalogue?.status === 'ready' && o.catalogue.launchRestart === true;
const needRestart = () => (running() && Array.isArray(state.organizations) ? state.organizations : []).filter(o => ['unavailable', 'bus-unavailable'].includes(o.catalogue?.status) || launchRestart(o));
// #1873 review P1-1: why each listed organization can't see the models, from its own routes
// read (GET /api/orgs/<id>/bus/routes, the read Routing uses). The status read carries only a
// status and a count. A Restart shows only when the bus names no cause or one a fresh start
@@ -853,13 +859,14 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
.then(got => { if (causes.get(id)?.round !== round) return; causes.set(id, { round, ...got }); if (enabled && state) { paintNext(); paintRestart(); } });
}
const unread = o => o.catalogue?.status === 'unavailable' && !('known' in (causes.get(o.id) ?? {}));
const restartHelps = o => { const c = causes.get(o.id); return o.catalogue?.status === 'unavailable' && c?.known === true && !c.ready && (!c.cause || c.cause.restart); };
const seenReady = o => o.catalogue?.status === 'ready' || causes.get(o.id)?.ready === true;
const restartHelps = o => { const c = causes.get(o.id); return launchRestart(o) || (o.catalogue?.status === 'unavailable' && c?.known === true && !c.ready && (!c.cause || c.cause.restart)); };
const seenReady = o => !launchRestart(o) && (o.catalogue?.status === 'ready' || causes.get(o.id)?.ready === true);
function afterAccounts() {
if (!accountCount) return 'accounts';
// ride-default: Ride all is next while an organization can ride now. Stand-in organizations alone
// do not hold the step: their Move buttons are optional and Ride all would be disabled.
if (state.next?.includes('ride-all') && Array.isArray(state.ride) && state.ride.some(r => r.state === 'ready')) return 'ride-all';
// settings-p3 (rebase re-check P3-2): only when the owner can send, so the step never leads to a disabled Ride all.
if (sendOk && state.next?.includes('ride-all') && Array.isArray(state.ride) && state.ride.some(r => r.state === 'ready')) return 'ride-all';
const waiting = needRestart().filter(o => !seenReady(o));
if (waiting.some(unread)) return 'checking-orgs';
if (waiting.some(restartHelps)) return 'restart-orgs';
@@ -883,7 +890,8 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
const status = o.catalogue?.status, name = nameOf(o.id), c = causes.get(o.id);
const routing = `<a href="#/orgs/${encodeURIComponent(o.id)}/settings/routing">Open ${esc(name)}'s Routing</a>`;
let badge, word, why, restart = true, fix = '', detail = '';
if (seenReady(o)) [badge, word, why, restart] = ['badge-ok', 'Ready', `${name} can see your models now.`, false];
if (launchRestart(o)) [badge, word, why] = ['badge-attn', 'Restart needed', `${name} can see your models. Its agents start on them after a restart.`];
else if (seenReady(o)) [badge, word, why, restart] = ['badge-ok', 'Ready', `${name} can see your models now.`, false];
else if (status === 'bus-unavailable') [badge, word, why] = ['badge-danger', 'Not running', `${name} is not running.`];
else if (status !== 'unavailable') [badge, word, why] = ['badge-muted', 'Unknown', `${name}'s models could not be read.`];
else {
@@ -918,17 +926,17 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
// #1873 ride-default: Ride all and the fixture move. The server decides each organization again;
// this code only words the answer. Every write asks first, in plain words, what changes.
const RIDE_REASONS = {
removed: 'The model proxy was on it, or Ride all may have added it, and it is off now. Ride all leaves it off. Add it on its Settings page to use it.',
'not-recorded': 'The proxy could not record the change first, so it was not made.',
removed: 'It used your model service, or Ride all may have added it, and it is off now. Ride all leaves it off. To use it there, choose Use your models on its Routing page.',
'not-recorded': 'Console could not record the change first, so it was not made.',
'bus-down': 'Its bus is not running. Start it, then ride again.',
'no-proxy-client': 'Its bus started before the proxy was set up. Restart its bus, then ride again.',
'proxy-check-failed': 'Its bus could not check the proxy. Check the proxy, then ride again.',
unreadable: 'Its organization file, its change log or the proxy\'s receipts could not be read. A damaged line in the proxy\'s receipts must be repaired by hand before Ride all adds again.',
'no-proxy-client': 'Its bus started before the model service was set up. Restart its bus, then ride again.',
'proxy-check-failed': 'Its bus could not check the model service. Check that the model service runs, then ride again.',
unreadable: 'Its organization file, its change log or the model service\'s receipts could not be read. A damaged line in the receipts must be repaired by hand before Ride all adds again.',
failed: 'Its bus refused the change.',
'bus-running': 'Its bus is running, or its state could not be read. Stop or recover its bus, then move it.',
'no-proxy': 'Set up the model proxy first.',
'no-riding-bus': 'No organization uses the proxy yet, so no model list was read. Ride all first.',
'no-model': 'The proxy lists no model for the harness its agents use.',
'no-proxy': 'Set up the model service first.',
'no-riding-bus': 'No organization uses the model service yet, so no model list was read. Ride all first.',
'no-model': 'The model service lists no model for the harness its agents use.',
invalid: 'The moved organization file did not pass its check. Nothing was written.',
'revision-conflict': 'The organization changed while it was being moved. Try again.',
'org-forbidden': 'Moving organizations is not available on this Console yet. It works on a local development Console.',
@@ -938,21 +946,26 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
};
const rideReason = code => RIDE_REASONS[code] ?? `The change was refused (${code}).`;
let rideResult = null;
// What the Ride all box lists (paintRide): organizations that can ride, skipped ones and stand-in ones.
const rideListed = () => { const ride = Array.isArray(state.ride) ? state.ride : [];
return ride.filter(r => r.state === 'ready').length + (rideResult?.skipped ?? ride.filter(r => !['ready', 'already', 'fixture'].includes(r.state))).length + (rideResult?.fixtures ?? ride.filter(r => r.state === 'fixture')).length; };
const li = text => { const el = document.createElement('li'); el.textContent = text; return el; };
function paintRide() {
const box = $('proxy-ride'), ride = Array.isArray(state.ride) ? state.ride : null;
box.hidden = !ride || !state.init?.initialized;
if (box.hidden) return;
const ready = ride.filter(r => r.state === 'ready').length, fixtures = ride.filter(r => r.state === 'fixture');
$('proxy-ride-text').textContent = 'Ride all adds this proxy to each organization that does not use it yet. '
+ 'Admission gets stricter there: every launch route is checked, direct Anthropic routes too, and while the proxy is down or fenced every launch in that organization is refused. '
$('proxy-ride-text').textContent = 'Ride all adds your model service to each organization that does not use it yet. '
+ 'Admission gets stricter there: every launch route is checked, direct Anthropic routes too, and while the model service is down or fenced every launch in that organization is refused. '
+ (ready ? `${ready} ${ready === 1 ? 'organization can' : 'organizations can'} ride now.` : 'No organization is waiting to ride.');
const go = $('proxy-ride-go'); go.disabled = !sendOk || Boolean(state.busy) || !ready;
const skips = (rideResult?.skipped ?? ride.filter(r => !['ready', 'already', 'fixture'].includes(r.state)).map(r => ({ id: r.id, reason: r.state })));
$('proxy-ride-skips').replaceChildren(...skips.map(s => li(`Skipped ${s.id}: ${rideReason(s.reason)}`)));
const listed = rideResult?.fixtures ?? fixtures.map(f => ({ id: f.id }));
const listed = rideResult?.fixtures ?? fixtures.map(f => ({ id: f.id, running: f.running }));
$('proxy-ride-fixtures').replaceChildren(...listed.map(f => {
const item = li(`${f.id} uses stand-in models, can be moved to the owner's models.${f.running ? ' Its bus is running: stop it first.' : ''} `);
// running null: no bus status here (the single-business Console), so moving is not available.
if (f.running === null) return li(`${f.id} uses stand-in models. Moving it is not available on this Console.`);
const item = li(`${f.id} uses stand-in models, can be moved to the owner's models.${f.running === true ? ' Its bus is running: stop it first.' : ''} `);
const move = document.createElement('button'); move.type = 'button'; move.className = 'btn'; move.textContent = `Move ${f.id}`;
move.dataset.rideMove = f.id; move.disabled = !sendOk || Boolean(state.busy);
move.addEventListener('click', () => { if (!move.disabled) rideMove(f.id); });
@@ -961,7 +974,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
}
const rideSay = (text, kind) => receipt('proxy-ride-message', text, kind);
const rideAll = () => {
if (!window.confirm('Ride all adds this model proxy to each organization that can take it. Admission gets stricter there: every launch route is checked, direct Anthropic routes too, and while the proxy is down or fenced every launch in that organization is refused. Continue?')) return;
if (!window.confirm('Ride all adds your model service to each organization that can take it. Admission gets stricter there: every launch route is checked, direct Anthropic routes too, and while the model service is down or fenced every launch in that organization is refused. Continue?')) return;
return act('ride-all', async () => {
const result = await admin('ride-all', {}); addReceipt(result.receipt); rideResult = result;
const parts = [`Added to ${result.added.length}`, `already using it ${result.already.length}`, `skipped ${result.skipped.length}`];
@@ -970,7 +983,7 @@ if (document.documentElement.hasAttribute('data-settings-proxy-document') && !lo
});
};
const rideMove = id => {
if (!window.confirm(`Move ${id} to your models? Its stand-in models are replaced by a model your proxy lists. After this, its agents spend real tokens on your account. Its bus stays stopped: start it when you are ready.`)) return;
if (!window.confirm(`Move ${id} to your models? Its stand-in models are replaced by a model your model service lists. After this, its agents spend real tokens on your account. Its bus stays stopped: start it when you are ready.`)) return;
return act('ride-move', async () => {
const result = await admin('ride-move', { org: id });
if (!result.moved) { rideSay(`${id} was not moved. ${rideReason(result.reason)}`, 'error'); return; }
+1 -1
View File
@@ -107,7 +107,7 @@ table.s1-table td code{font-size:12.5px;white-space:nowrap}
.org-restart [data-org-restart-detail]{display:block}
.routes-catalogue{flex-direction:column;align-items:stretch}
.routes-catalogue>p{flex:none}
.settings-content .org-down .banner p{margin:0 0 8px}
.settings-content .org-down .banner p{margin:0 0 8px;flex-basis:100%}
.settings-content .org-down .banner p:last-child{margin-bottom:0}
.routes-apply{margin:12px 0 0;padding-left:18px;display:grid;gap:8px}
.routes-apply li{overflow-wrap:anywhere}
+13 -1
View File
@@ -3,7 +3,7 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { request } from 'node:http';
import { mkdtempSync, rmSync } from 'node:fs';
import { mkdtempSync, rmSync, readFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { startServer } from '../src/serve.mjs';
@@ -207,6 +207,18 @@ test('an organization whose bus does not answer shows no model count, not a made
assert.deepEqual(await orgCatalogue(async () => ({ catalogue: { status: 'enabled', models: [] } })), { status: 'enabled', models: 0 });
});
// settings-p3 (re-check 3 P3-5): a ready catalogue the bus picked up after it started carries the bus's
// restart notice; Model setup gets launchRestart so it keeps that organization listed. The notice text is
// copied from packages/bus/src/runtime.mjs (not exported there); the pins below fail if the copies drift.
test('a ready catalogue with the bus restart notice is marked launchRestart, and every copy of the notice matches the bus', async () => {
const notice = 'Restart this organization to start agents on these models.';
assert.deepEqual(await orgCatalogue(async () => ({ catalogue: { status: 'ready', models: [{ id: 'a' }], notices: [notice] } })), { status: 'ready', models: 1, launchRestart: true });
assert.deepEqual(await orgCatalogue(async () => ({ catalogue: { status: 'ready', models: [{ id: 'a' }], notices: ['something else'] } })), { status: 'ready', models: 1 });
assert.deepEqual(await orgCatalogue(async () => ({ catalogue: { status: 'unavailable', models: [], notices: [notice] } })), { status: 'unavailable', models: 0 });
for (const file of ['../../bus/src/runtime.mjs', '../src/proxy-admin.mjs', '../src/public/console-restart.js'])
assert.ok(readFileSync(new URL(file, import.meta.url), 'utf8').includes(`RESTART_TO_LAUNCH = '${notice}'`), file);
});
test('ride move takes one organization id and passes the signed-in owner as the principal', async t => {
const f = await devServer(t);
for (const body of [{}, { org: '' }, { org: '../x' }, { org: 'A' }, { org: 7 }, { org: 'x', extra: 1 }])
@@ -174,7 +174,7 @@ test('Ride all confirms the stricter admission, paints skips and stand-in organi
// The first Ride all's own reload paints, then waits on the accounts read for good.
await b.evaluate('document.querySelector("#proxy-ride-go").click()');
await wait('window.__writes.some(w=>w.path==="/api/proxy/admin/ride-all")');
assert.match(await b.evaluate('window.__confirms[0]'), /Admission gets stricter there: every launch route is checked, direct Anthropic routes too, and while the proxy is down or fenced every launch in that organization is refused/);
assert.match(await b.evaluate('window.__confirms[0]'), /Admission gets stricter there: every launch route is checked, direct Anthropic routes too, and while the model service is down or fenced every launch in that organization is refused/);
await wait('document.querySelector("#proxy-ride-message").textContent.includes("Added to 1")');
await wait('document.querySelector("#proxy-ride-go").disabled===false');
// The second answer lands while that load is still in flight.
@@ -183,7 +183,7 @@ test('Ride all confirms the stricter admission, paints skips and stand-in organi
await wait('window.__writes.filter(w=>w.path==="/api/proxy/admin/ride-all").length===2');
await wait('document.querySelector("#proxy-ride-message").textContent.includes("skipped 1")');
await wait('document.querySelectorAll("#proxy-ride-skips li").length===1');
assert.equal(await b.evaluate('document.querySelector("#proxy-ride-skips li").textContent'), 'Skipped north: Its bus started before the proxy was set up. Restart its bus, then ride again.');
assert.equal(await b.evaluate('document.querySelector("#proxy-ride-skips li").textContent'), 'Skipped north: Its bus started before the model service was set up. Restart its bus, then ride again.');
assert.match(await b.evaluate('document.querySelector("#proxy-ride-fixtures li").textContent'), /Its bus is running: stop it first\./);
assert.equal(await b.evaluate('document.querySelector("#proxy-ride-go").disabled'), false, 'the box is not left busy');
await b.evaluate('document.querySelector("[data-ride-move=west]").click()');
@@ -208,10 +208,12 @@ test('Ride all confirms the stricter admission, paints skips and stand-in organi
await wait('window.__writes.filter(w=>w.path==="/api/proxy/admin/ride-all").length===3');
await wait('document.querySelectorAll("#proxy-ride-skips li").length===3');
assert.deepEqual(JSON.parse(await b.evaluate('JSON.stringify([...document.querySelectorAll("#proxy-ride-skips li")].map(li=>li.textContent))')), [
'Skipped north: The model proxy was on it, or Ride all may have added it, and it is off now. Ride all leaves it off. Add it on its Settings page to use it.',
'Skipped south: The proxy could not record the change first, so it was not made.',
'Skipped north: It used your model service, or Ride all may have added it, and it is off now. Ride all leaves it off. To use it there, choose Use your models on its Routing page.',
'Skipped south: Console could not record the change first, so it was not made.',
// Re-check 3 (P3-3): a damaged receipts line stops every add until someone repairs it.
'Skipped east: Its organization file, its change log or the proxy\'s receipts could not be read. A damaged line in the proxy\'s receipts must be repaired by hand before Ride all adds again.']);
'Skipped east: Its organization file, its change log or the model service\'s receipts could not be read. A damaged line in the receipts must be repaired by hand before Ride all adds again.']);
// settings-p3 (rebase re-check P3-4): the Ride all box, its confirms and its lines say model service, not proxy.
assert.doesNotMatch(await b.evaluate('document.querySelector("#proxy-ride").innerText + window.__confirms.join(" ")'), /proxy/i);
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
});
@@ -240,8 +242,8 @@ test('settings-s1 B2: once the service runs, the organizations that cannot see t
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-go")'), null, 'the next step is the list, so no second button');
// acme's bus names no cause, so a restart connects it. beta is not running: it keeps its Restart
// and its refusal, and does not hold the next step.
assert.match(await b.evaluate('document.querySelector("#proxy-setup-next").textContent'), /^Next: restart acme\. It has a Restart button under Organizations that can't see your models\./);
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-restart-title").textContent'), "Organizations that can't see your models");
assert.match(await b.evaluate('document.querySelector("#proxy-setup-next").textContent'), /^Next: restart acme\. It has a Restart button under Organizations that can't use your models yet\./);
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-restart-title").textContent'), "Organizations that can't use your models yet");
// A restart reads the status again; the organization stays on the list, now ready, so the owner sees it worked.
await b.evaluate(`window.__scenario['/api/proxy/admin/status'] = ${JSON.stringify({ ...running, organizations: [{ ...orgs[0], catalogue: { status: 'ready', models: 3 } }, ...orgs.slice(1)] })}`);
await b.evaluate('document.querySelector("[data-org-restart=acme]").focus()');
@@ -271,7 +273,7 @@ test('settings-s1 B2: Restart shows only where the organization\'s own cause all
assert.deepEqual(rows.lab, { badge: 'badge badge-attn', why: "lab runs on test models, which can't share this machine with your model service. This can't be changed from Console yet.", restart: null, link: null, detail: 'Technical details' });
assert.deepEqual(rows.lock, { badge: 'badge badge-attn', why: "lock can't open the model service's shared lock. A restart opens it again.", restart: 'Restart lock', link: null, detail: 'Technical details' });
// Two models that slug to one key: a restart reads the same catalogue again (review 2, P2-A).
assert.deepEqual(rows.clash, { badge: 'badge badge-attn', why: "Two of your models get the same short name, so clash can't tell them apart. A restart won't change that.", restart: null, link: null, detail: 'Technical details' });
assert.deepEqual(rows.clash, { badge: 'badge badge-attn', why: "Two of your models get the same short name, so clash can't tell them apart. A restart won't change that. Take one of them out of an API key's Models list on Model setup. If both come from signed-in accounts, this can't be changed from Console yet.", restart: null, link: null, detail: 'Technical details' });
assert.deepEqual(rows.odd, { badge: 'badge badge-attn', why: "odd can't see your models, and its reason could not be read. Open odd's Routing", restart: null, link: '#/orgs/odd/settings/routing', detail: null });
assert.equal(await b.evaluate('document.querySelector("[data-restart-org=lab] details").textContent.includes("fixture-proxy-configured")'), true, 'the bus line is kept under Technical details');
assert.match(await b.evaluate('document.querySelector("#proxy-setup-next").textContent'), /^Next: restart acme and lock\./);
@@ -330,7 +332,9 @@ test('settings-s1 B3 with ride-default: Ride all follows accounts while an organ
const { f, b, wait } = await w15Fixture(t, { scenario: { ...scenario(riding), ...answering(1), '/api/triage/route': { route: null } } });
await b.navigate(f.base + '/#/settings/model-proxy');
await wait('document.querySelector("#proxy-setup-go")?.dataset.next==="ride-all"');
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-go").textContent'), 'Ride all');
// settings-p3 (rebase re-check P3-3): the step button is not a second "Ride all"; it goes to the box's own button.
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-go").textContent'), 'Go to Ride all');
assert.equal(await b.evaluate('[...document.querySelectorAll("#proxy-setup button")].filter(e=>e.checkVisibility()&&e.textContent==="Ride all").length'), 1, 'one Ride all button');
assert.equal(await b.evaluate('document.querySelector("#proxy-ride").checkVisibility()'), true, 'the Ride all box is not folded under Technical details');
assert.equal(await b.evaluate('document.querySelector("#proxy-ride-go").disabled'), false);
// Ride all adds north. Only a stand-in organization is left: the step moves on to the Triage route, and Move stays offered.
@@ -342,6 +346,92 @@ test('settings-s1 B3 with ride-default: Ride all follows accounts while an organ
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
});
// settings-p3 (rebase re-check P3-2): an owner who can't send gets no Ride all step: Ride all is disabled for them.
test('settings-p3: Ride all is not the next step when the owner cannot send', { timeout: 90000 }, async t => {
const riding = { ...running, next: ['accounts', 'ride-all'], organizations: [{ id: 'acme', proxy: 'shared', catalogue: { status: 'ready', models: 2 } }], ride: [{ id: 'north', state: 'ready' }] };
const { f, b, wait } = await w15Fixture(t, { scenario: { ...scenario(riding, { proxyAdminSend: false }), ...answering(1), '/api/triage/route': { route: null } } });
await b.navigate(f.base + '/#/settings/model-proxy');
await wait('document.querySelector("#proxy-setup-go")?.dataset.next==="triage-route"');
assert.equal(await b.evaluate('document.querySelector("#proxy-ride-go").disabled'), true);
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
});
// settings-p3 (rebase re-check P3-1): with a stand-in or skipped organization still in the Ride all box,
// the done line does not say every organization can see the models.
test('settings-p3: the done line does not say every organization while the Ride all box still lists one', { timeout: 90000 }, async t => {
const done = { ...running, organizations: [{ id: 'acme', proxy: 'shared', catalogue: { status: 'ready', models: 2 } }], ride: [{ id: 'acme', state: 'already' }, { id: 'west', state: 'fixture' }] };
const { f, b, wait } = await w15Fixture(t, { scenario: { ...scenario(done), ...answering(1), '/api/triage/route': savedRoute } });
await b.navigate(f.base + '/#/settings/model-proxy');
await wait('document.querySelector("#proxy-setup-next [data-next]")?.dataset.next==="done"');
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-next").textContent'), 'The model service is set up. Some organizations still need a step under Use your models in every organization.');
assert.equal(await b.evaluate('document.querySelector("#proxy-ride").checkVisibility() && document.querySelector("#proxy-ride h3").textContent'), 'Use your models in every organization');
// Once nothing is listed there, the line says every organization.
await b.evaluate(`window.__scenario['/api/proxy/admin/status'] = ${JSON.stringify({ ...done, ride: [{ id: 'acme', state: 'already' }] })}`);
await b.evaluate('document.dispatchEvent(new CustomEvent("mosaic-org-restarted", { detail: { id: "acme" } })); true');
await wait('document.querySelector("#proxy-setup-next").textContent==="The model service is set up and every organization can see your models."');
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
});
// settings-p3 review P2-1: an owner who can't send gets no Ride all step, so the done line shows while an
// organization can still ride; it must not say every organization can see the models.
test('settings-p3: the done line does not say every organization while one can still ride and the owner cannot send', { timeout: 90000 }, async t => {
const done = { ...running, organizations: [{ id: 'acme', proxy: 'shared', catalogue: { status: 'ready', models: 2 } }], ride: [{ id: 'acme', state: 'already' }, { id: 'north', state: 'ready' }] };
const { f, b, wait } = await w15Fixture(t, { scenario: { ...scenario(done, { proxyAdminSend: false }), ...answering(1), '/api/triage/route': savedRoute } });
await b.navigate(f.base + '/#/settings/model-proxy');
await wait('document.querySelector("#proxy-setup-next [data-next]")?.dataset.next==="done"');
assert.equal(await b.evaluate('document.querySelector("#proxy-setup-next").textContent'), 'The model service is set up. Some organizations still need a step under Use your models in every organization.');
assert.match(await b.evaluate('document.querySelector("#proxy-ride-text").textContent'), /1 organization can ride now\./);
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
});
// settings-p3 review P3-3: on a single-business Console a stand-in organization has no bus status
// (running null), so the page says moving is not available there and offers no Move button.
test('settings-p3: a stand-in organization with no bus status is not offered a Move', { timeout: 90000 }, async t => {
const state = { ...running, organizations: [{ id: 'acme', proxy: 'shared', catalogue: { status: 'ready', models: 2 } }], ride: [{ id: 'acme', state: 'already' }, { id: 'west', state: 'fixture', running: null }, { id: 'east', state: 'fixture' }] };
const { f, b, wait } = await w15Fixture(t, { scenario: { ...scenario(state), ...answering(1), '/api/triage/route': savedRoute } });
await b.navigate(f.base + '/#/settings/model-proxy');
await wait('document.querySelector("#proxy-ride-fixtures li")');
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("#proxy-ride-fixtures li")].map(li => li.firstChild.textContent.trim())'),
['west uses stand-in models. Moving it is not available on this Console.', "east uses stand-in models, can be moved to the owner's models."]);
assert.equal(await b.evaluate('document.querySelector("[data-ride-move=west]")'), null);
assert.equal(await b.evaluate('document.querySelector("[data-ride-move=east]").disabled'), false);
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
});
// settings-p3 (re-check 3 P3-5): a ready catalogue the bus picked up late (launchRestart) keeps the
// organization listed with its Restart, holds the next step, and Model setup never says done.
test('settings-p3: an organization whose agents start on the models only after a restart is listed and holds the next step', { timeout: 90000 }, async t => {
const orgs = [{ id: 'late', proxy: 'shared', catalogue: { status: 'ready', models: 2, launchRestart: true } }];
const { f, b, wait } = await w15Fixture(t, { scenario: { ...scenario({ ...running, organizations: orgs }, { orgSetup: true }), ...answering(), '/api/triage/route': savedRoute } });
await b.navigate(f.base + '/#/settings/model-proxy');
await wait('document.querySelectorAll("#proxy-setup-restart-list li").length===1');
assert.deepEqual(await b.evaluate('(li=>[li.querySelector(".badge").className, li.querySelector(".badge").textContent, li.querySelector(":scope > .small").textContent, li.querySelector("[data-org-restart]")?.textContent ?? null])(document.querySelector("[data-restart-org=late]"))'),
['badge badge-attn', 'Restart needed', 'late can see your models. Its agents start on them after a restart.', 'Restart late']);
assert.match(await b.evaluate('document.querySelector("#proxy-setup-next").textContent'), /^Next: restart late\./);
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
});
// settings-p3 (re-check 3 P3-1, P3-7): the init lock and partial init lines say what to do in plain
// words, never point at "the README" in the text, and keep the README path under Technical details.
test('settings-p3: the init lock and partial init steps say what to do, with the README path under Technical details', { timeout: 90000 }, async t => {
const held = { ...fetched, initLock: { present: true, clearable: false, blockers: ['the bus for acme is running'] }, next: ['init-lock-wait'] };
const { f, b, wait } = await w15Fixture(t, { scenario: scenario(held) });
await b.navigate(f.base + '/#/settings/model-proxy');
await wait('document.querySelector("#proxy-setup-go")?.dataset.next==="init-lock-wait"');
const read = 'JSON.stringify({text: document.querySelector("#proxy-setup-next > p").textContent, summary: document.querySelector("#proxy-setup-next details summary")?.textContent, lead: document.querySelector("#proxy-setup-next details p")?.textContent, code: document.querySelector("#proxy-setup-next details code")?.textContent, open: document.querySelector("#proxy-setup-next details")?.open, button: document.querySelector("#proxy-setup-go").textContent})';
assert.deepEqual(JSON.parse(await b.evaluate(read)), {
text: 'An init lock is held and cannot be cleared: the bus for acme is running. Stop the bus, then check again. Model service files that are left must be resolved by hand. Do not delete them to make init run again.',
summary: 'Technical details', lead: 'The hand steps for a stale init lock and a partial init are in packages/model-proxy/README.md', code: 'packages/model-proxy/README.md', open: false, button: 'Check again' });
await b.evaluate(`window.__scenario['/api/proxy/admin/status'] = ${JSON.stringify({ ...fetched, init: { initialized: false, partial: true, files: [{ label: 'proxy settings', present: true }, { label: 'management key', present: false }] }, next: ['init-repair'] })}`);
await b.evaluate('document.querySelector("#proxy-setup-go").click()');
await wait('document.querySelector("#proxy-setup-go")?.dataset.next==="init-repair"');
const repair = JSON.parse(await b.evaluate(read));
assert.equal(repair.text, 'Setup is partly done. Init never overwrites model service files that already exist, so they must be resolved by hand. Do not delete them to make init run again. Then check again.');
assert.equal(repair.code, 'packages/model-proxy/README.md');
assert.doesNotMatch(await b.evaluate('document.querySelector("#proxy-setup-next").innerText'), /README|proxy/i, 'nothing technical shows until Technical details is opened');
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
});
test('settings-s1 B4: only the next step has a button; done steps, Fence, Init lock and the service controls sit under Technical details', { timeout: 90000 }, async t => {
const { f, b, wait } = await w15Fixture(t, { scenario: { ...scenario(running), ...answering(0), '/api/triage/route': savedRoute } });
await b.navigate(f.base + '/#/settings/model-proxy');
@@ -77,6 +77,14 @@ test('failed assistant turns explain recovery in conversation and messages',asyn
let text=await b.evaluate(`document.querySelector('.chat-message-agent').textContent`);assert.match(text,/The assistant couldn.t reply\. Try again, or check Agents\./);assert.match(text,/adapter exited 127/);
await b.evaluate(`location.hash='#/messages'`);await wait(`document.querySelector('.s1-message')`);text=await b.evaluate(`document.querySelector('#message-list').textContent`);assert.match(text,/The assistant couldn.t reply\. Try again, or check Agents\./);assert.match(text,/adapter exited 127/);
});
// #1873 chat-fail: a failed turn with a plain cause shows the cause and next step, with the technical reason under it.
test('a failed turn with a plain cause shows it in the conversation with its details under it',async t=>{
const extra=`fixtureMessages[1].body='The turn failed: the model account is out of usage. Check Settings > Model setup.\\nDetails: adapter exited 1 <b>x</b>';`;
const {b,wait}=await setup(t,'/projects/threads/'+T,390,extra);await wait(`document.querySelector('.chat-message-agent .chat-body')`);
const r=await b.evaluate(`(()=>{const e=document.querySelector('.chat-message-agent .chat-body');return {text:e.textContent,small:e.querySelector('.small')?.textContent,bold:!!e.querySelector('b')}})()`);
assert.match(r.text,/^The assistant couldn.t reply: the model account is out of usage\. Check Settings > Model setup\./);
assert.equal(r.small,'Details: adapter exited 1 <b>x</b>');assert.equal(r.bold,false);
});
test('queue refusal redacts the placeholder and shows its code only once',async t=>{
const extra=`const prior=window.fetch;window.fetch=async(u,o)=>String(u)==='/api/queue'?new Response(JSON.stringify({error:'queue-refused',message:'<path> is a linked worktree; the queue runs only in the canonical checkout'}),{status:503,headers:{'content-type':'application/json'}}):prior(u,o);`;
const {b,wait}=await setup(t,'/queue',390,extra);await wait(`document.querySelector('#bus-view .banner')`);const text=await b.evaluate(`document.querySelector('#bus-view .banner').textContent`);assert.doesNotMatch(text,/<path>/);assert.equal((text.match(/queue-refused/g)||[]).length,1);
@@ -23,7 +23,7 @@ const LIST = [
const ROLES = { acme: { chief: { definition: 'chief-of-staff' }, coder: { definition: 'coder' } }, gamma: { chief: { definition: 'chief-of-staff' } } };
// The routes acme's bus answers; a test swaps it to show a restart that brought the catalogue up.
const state = { view: null };
const state = { view: null, gamma: null };
async function fixture() {
const { checkRootAgents } = await import('../../organization/src/index.mjs');
@@ -32,7 +32,7 @@ async function fixture() {
const empty = () => Promise.resolve([]);
const member = org => ({
id: org.id, doc: { appearance: { theme: org.theme }, roles: ROLES[org.id], projects: {} },
status: async () => (org.busHealthy ? { healthy: true, pid: 4242, boot: null } : { healthy: false, boot: { ok: false, at: at(90), lastGood: null, reason: 'stopped by the owner' } }),
status: async () => (!org.busHealthy && state.gamma ? state.gamma : org.busHealthy ? { healthy: true, pid: 4242, boot: null } : { healthy: false, boot: { ok: false, at: at(90), lastGood: null, reason: 'stopped by the owner' } }),
options: () => ({ bus: { inbox: empty, agents: empty, 'thread.claims': empty, tasks: empty, usage: () => Promise.resolve({}), limits: () => Promise.resolve({}), decisions: empty }, chat: { threads: empty, projects: empty } }),
});
const organizations = {
@@ -90,8 +90,8 @@ const stub = ({ service = SERVICE.running, restart = DONE, extra = {} } = {}) =>
};
})();`;
async function setup(t, { viewport = [1440, 900], path, init, triage = null, view = VIEW }) {
state.view = view;
async function setup(t, { viewport = [1440, 900], path, init, triage = null, view = VIEW, gamma = null }) {
state.view = view; state.gamma = gamma;
const { organizations, agentLevels } = await fixture();
const server = await startServer({ port: 0, sso: createDevOwner(), organizations, agentLevels, triage, send: async () => ({}), log: () => {} });
t.after(() => close(server));
@@ -213,13 +213,27 @@ test('settings-s1 B6: a stopped organization keeps its refused restart on screen
assert.equal(await text('[data-org-down=gamma] [data-org-down-why]'), 'Why it stopped: stopped by the owner');
// One Restart on the page: the header's bus banner steps aside.
assert.equal(await b.evaluate('[...document.querySelectorAll("[data-org-restart], #bus-restart")].filter(e => e.offsetParent).length'), 1);
// settings-p3 (re-check 3 P3-10): the why line sits under the headline, not beside it, at 1440.
assert.equal(await b.evaluate('(([a, w]) => w.getBoundingClientRect().top >= a.getBoundingClientRect().bottom)([...document.querySelectorAll("[data-org-down=gamma] .banner > p")])'), true, 'why line under the headline');
const primary = 'document.querySelector("[data-org-down=gamma] [data-org-restart=gamma]").classList.contains("primary")';
assert.equal(await b.evaluate(primary), true, 'Restart is primary before any refusal');
// The refusal's own repaint (console-restart.js paintOrg) is read when it lands, before the page renders again.
await b.evaluate(`window.__firstPaint = null; new MutationObserver((_, o) => { const d = document.querySelector("[data-org-down=gamma] [data-org-restart-detail]"); if (d) { window.__firstPaint = d.closest("[data-org-restart-box]").querySelector("[data-org-restart]").classList.contains("primary"); o.disconnect(); } }).observe(document.body, { subtree: true, childList: true }); true`);
await b.evaluate('document.querySelector("[data-org-down=gamma] [data-org-restart=gamma]").click(); true');
await wait('window.__writes.length === 1 && document.querySelector("[data-org-down=gamma] [data-org-restart-detail]")');
assert.equal(await b.evaluate('window.__firstPaint'), false, 'the refusal repaint does not keep Restart primary');
// The page renders again after the restart; the refusal stays, with its plain line first.
await new Promise(r => setTimeout(r, 500));
const receipt = await text('[data-org-down=gamma] [data-org-restart-receipt]');
assert.match(receipt, /^Gamma Home runs on test models, which can't share this machine with your model service\. It did not start\./);
assert.match(receipt, /bus start refused: fixture-proxy-configured/);
// settings-p3 (re-check 3 P3-4): a restart is refused the same way again, so Restart stays but is not
// the primary action, both in place and after the page renders again.
assert.equal(await b.evaluate(primary), false, 'Restart is not primary after the fixture refusal');
await b.evaluate('document.querySelector("[data-org-down=gamma]").dataset.old = "1"; location.hash = "#/orgs/gamma/settings/appearance"');
await wait('location.hash.endsWith("appearance") && !document.querySelector("[data-org-down][data-old]")');
await b.evaluate('location.hash = "#/orgs/gamma/settings/routing"'); await wait('document.querySelector("[data-org-down=gamma]:not([data-old]) [data-org-restart=gamma]")');
assert.equal(await b.evaluate(primary), false, 'still not primary after rendering again');
const links = await b.evaluate('[...document.querySelectorAll("[data-org-down=gamma] a")].map(a => [a.textContent, a.getAttribute("href")])');
assert.deepEqual(links, [['Appearance', '#/settings/appearance'], ['Model setup', '#/settings/model-proxy'], ['Triage', '#/settings/triage'], ['Console', '#/settings/console']]);
// A link from the stopped page opens an instance page.
@@ -228,6 +242,46 @@ test('settings-s1 B6: a stopped organization keeps its refused restart on screen
assert.notEqual(await text('h1'), 'Not found');
});
// settings-p3 (re-check 3 P3-5): a ready catalogue the bus picked up after it started carries the bus's
// restart notice (runtime.mjs RESTART_TO_LAUNCH); Routing puts the organization's Restart beside it,
// not primary, since Save routes is. Also (bus.js help): the Routing help names no harness.
test('settings-p3: a ready catalogue with the restart notice offers the organization Restart beside it, and the help names no harness', { timeout: 90000 }, async t => {
const notice = 'Restart this organization to start agents on these models.';
const view = { ...VIEW, catalogue: { status: 'ready', observedAt: new Date().toISOString(), models: [], notices: [notice] } };
const { b, wait } = await setup(t, { path: '/#/orgs/acme/settings/routing', init: stub(), view });
await wait('document.querySelector(".routing-panel p.warn [data-org-restart=acme]")');
assert.deepEqual(await b.evaluate('(p => [p.firstChild.textContent.trim(), p.querySelector("[data-org-restart]").textContent, p.querySelector("[data-org-restart]").classList.contains("primary")])(document.querySelector(".routing-panel p.warn [data-org-restart=acme]").closest("p"))'),
[notice, 'Restart Acme Studio', false]);
assert.equal(await b.evaluate('window.MosaicConsole.RESTART_TO_LAUNCH'), notice);
const help = await b.evaluate('document.querySelector(".settings-helper").textContent');
assert.match(help, /A harness is the program that runs the model\. Preflight/);
assert.doesNotMatch(help, /Claude|such as/);
await b.evaluate('document.querySelector(".routing-panel p.warn [data-org-restart=acme]").focus(); true');
assert.notEqual(await b.evaluate('getComputedStyle(document.activeElement).outlineStyle'), 'none');
});
// settings-p3 (re-check 3 P3-9): a reason that is not a stop (a host that runs but does not answer yet)
// is labeled as the last check, not as why the bus stopped.
test('settings-p3: a stopped page labels a reason that is not a stop as the last check', { timeout: 90000 }, async t => {
const { wait, text } = await setup(t, { path: '/#/orgs/gamma/settings/routing', init: stub(), gamma: { healthy: false, pid: 4343, boot: { ok: true, at: at(5), lastGood: at(5) } } });
await wait('document.querySelector("[data-org-down=gamma] [data-org-down-why]:not([hidden])")');
assert.equal(await text('[data-org-down=gamma] [data-org-down-why]'), 'Last check: the bus host is running but does not answer yet');
});
// settings-p3 (re-check 3 P3-11): two callers that ask while one read is pending are each told when it
// lands, each with a line that names its own organization label.
test('settings-p3: every caller of orgDownReason that asks while the read is pending gets its own line', { timeout: 90000 }, async t => {
const { b, wait } = await setup(t, { path: '/#/settings/appearance', init: stub(), gamma: { healthy: false, boot: { ok: false, at: at(5), lastGood: null, reason: 'bus start refused: fixture-proxy-configured' } } });
await wait('window.MosaicConsole?.orgDownReason');
await b.evaluate('window.__lines = []; MosaicConsole.orgDownReason("gamma", "First", v => __lines.push(["a", v?.plain])); MosaicConsole.orgDownReason("gamma", "Second", v => __lines.push(["b", v?.plain])); true');
await wait('window.__lines.length === 2');
assert.deepEqual(await b.evaluate('window.__lines.sort()'), [
['a', "First runs on test models, which can't share this machine with your model service. It did not start."],
['b', "Second runs on test models, which can't share this machine with your model service. It did not start."]]);
// The cached answer is built for the caller who asks, not kept from the first one.
assert.equal(await b.evaluate('MosaicConsole.orgDownReason("gamma", "Third").plain'), "Third runs on test models, which can't share this machine with your model service. It did not start.");
});
// Shots for the build report, current beside proposed: run with MOSAIC_UX_SHOTS=<dir>, on this tree and on
// the base tree with this file copied in. Each waits only for what both trees draw, then settles. At 390
// every visible select must show its chosen text in full, and no page scrolls sideways.
@@ -243,6 +297,9 @@ test('shots', { timeout: 600000, skip: !process.env.MOSAIC_UX_SHOTS }, async t =
unit: { unit: 'mosaic-proxy.service', installed: true, active: 'active', healthy: true, pid: 4242, port: 18432, listening: true },
fence: { set: false }, initLock: { present: false, clearable: false, blockers: [] }, fetch: { state: 'idle' }, restart: null, busy: null,
organizations: [{ id: 'acme', proxy: 'shared', catalogue: { status: 'unavailable', models: 0 } }, { id: 'gamma', proxy: 'shared', catalogue: { status: 'bus-unavailable' } }],
// settings-p3: the Ride all box below the restart list, with one organization that can ride, one
// skipped because its bus is down and one on stand-in models.
ride: [{ id: 'acme', state: 'already' }, { id: 'north', state: 'ready' }, { id: 'gamma', state: 'bus-down' }, { id: 'west', state: 'fixture' }],
ownProxy: { available: false, state: 'not built', reason: 'The configuration has one proxy per instance.' }, next: ['accounts'] };
const triageRead = { route: { model: 'sonnet', harness: 'claude-code', cap: 200000, reset: '00:00', timeZone: 'Europe/Berlin', revision: 1, updatedAt: T }, revision: 1, send: true,
choices: { status: 'ready', models: [{ key: 'sonnet', label: 'Sonnet', available: true, harnesses: ['claude-code', 'pi'] }, { key: 'glm-flash', label: 'GLM Flash', available: true, harnesses: ['pi'] }] },
@@ -299,7 +356,10 @@ test('shots', { timeout: 600000, skip: !process.env.MOSAIC_UX_SHOTS }, async t =
// The drawer is fixed to the window, so its shot is the window; every other page is shot whole.
if (page.viewportOnly) await f.b.screenshot(join(dir, shot));
else {
const metrics = await f.b.call('Page.getLayoutMetrics'), size = metrics.cssContentSize || metrics.contentSize;
let metrics = await f.b.call('Page.getLayoutMetrics'), size = metrics.cssContentSize || metrics.contentSize;
// The footer is fixed to the window: grow the window to the page so it does not cover the page.
await f.b.viewport(width, Math.ceil(size.height)); await settle();
metrics = await f.b.call('Page.getLayoutMetrics'); size = metrics.cssContentSize || metrics.contentSize;
const { data } = await f.b.call('Page.captureScreenshot', { format: 'png', captureBeyondViewport: true, clip: { x: 0, y: 0, width: Math.ceil(size.width), height: Math.ceil(size.height), scale: 1 } });
await writeFile(join(dir, shot), Buffer.from(data, 'base64'));
}
+3 -1
View File
@@ -50,7 +50,9 @@ test('shipped strings are unchanged, and the web keeps its write boundary', () =
['bus.js', 'No page at this address.'], ['app.js', 'Nothing is waiting on you.'], ['bus.js', 'Nothing is waiting on you.'],
['bus.js', 'No tasks in this business yet.'], ['bus.js', 'Tasks appear after the PM creates them or the poller reads them from Vikunja.'],
['bus.js', 'No open tasks.'], ['bus.js', 'No role is held right now.'], ['bus.js', 'Nothing of this kind in the trail.'],
['bus.js', 'Console shows nothing rather than a guess. Nothing was written.'],
// settings-p3 (re-check 3 P3-8): the line is declared once as NO_GUESS and used by the failed view and
// the failed Settings read; counting the uses catches dropping it from either page.
['bus.js', 'Console shows nothing rather than a guess. Nothing was written.'], ['bus.js', '${NO_GUESS}', 2],
]) assert.equal(plain[file].split(s).length - 1, n, `${file}: ${s}`);
// Row 252 (#1753, mos-1321): the one decision control is the Console's grant Approve and Deny, one
// fetch(url, ...) in decidePost, called only for a decision's confirmation and its resolve. Its labels
@@ -27,7 +27,7 @@ function boardOrgs({ refuse = null, tasksFail = [] } = {}) {
forge: [{ id: 'api-gateway', repo: 'gateway', name: 'API gateway', at: '2026-11-09T10:00:00.000Z' }, { id: 'billing', repo: 'billing', name: 'Billing', at: '2026-11-08T10:00:00.000Z' }],
};
const store = [
{ n: 41, title: 'Rate limit the public API', state: 'todo', group: 'api-gateway', assignees: ['coder'], priority: 4, due: '2026-11-20T00:00:00.000Z' },
{ n: 41, title: 'Rate limit the public API', state: 'todo', group: 'api-gateway', assignees: ['coder'], priority: 4, due: '2099-11-20T00:00:00.000Z' },
{ n: 42, title: 'Invoice export', state: 'in-progress', group: 'billing', assignees: [], priority: 0, due: null },
{ n: 43, title: 'Retire the v1 gateway', state: 'done', group: 'api-gateway', assignees: ['reviewer'], priority: 2, due: null },
].map(t => ({ ...t, rev: 0 }));
@@ -90,7 +90,7 @@ test('the board merges every organization\'s tasks, tags each with its org, and
[['ddigital', 'V-112', true, 'Vikunja'], ['ddigital', 'V-118', true, 'Vikunja'], ['forge', '#41', true, 'built-in'], ['forge', '#42', true, 'built-in'], ['forge', '#43', true, 'built-in']]);
// Columns: the store's states, plus one for the Vikunja task whose bucket the bus does not name.
assert.deepEqual(await b.evaluate('[...document.querySelectorAll(".tb-col h3")].map(h => h.textContent)'), ['To do', 'In progress', 'Review', 'Blocked', 'Done', 'State not read']);
assert.deepEqual(await b.evaluate('[...document.querySelector("[data-tb-col=todo] [data-tb-task=\'forge|task:41\'] .bot").children].map(e => e.textContent.trim())'), ['API gateway', 'coder', 'P4', 'due 20 Nov']);
assert.deepEqual(await b.evaluate('[...document.querySelector("[data-tb-col=todo] [data-tb-task=\'forge|task:41\'] .bot").children].map(e => e.textContent.trim())'), ['API gateway', 'coder', 'P4', 'due 20 Nov 2099']);
// Only built-in store cards can be dragged; a Vikunja card is read only.
assert.equal(await b.evaluate('document.querySelector("[data-tb-task=\'forge|task:41\']").draggable'), true);
assert.equal(await b.evaluate('document.querySelector("[data-tb-task=\'ddigital|vikunja:3/112\']").draggable'), false);
@@ -558,8 +558,14 @@ const WORKSPACE_REASON = reason => `(() => { const f = window.fetch; window.fetc
test('settings-s1 B6: a refused last start shows the reason the workspace read carries', async t => {
const failedAt = at(30);
for (const [reason, expect] of [
// The fixture case is a synthetic pair: lastStart.ok false with a fixture refusal reason. The host
// does not write it. The fixture refusal is thrown after recordBootOk (cli.mjs), so boot status stays
// ok and the workspace read says "the bus is not running". The case pins the line the page would show
// once a later bus lane records that refusal.
['the bus refused to start: bus start refused: fixture-proxy-configured', ' Acme Studio runs on test models, which can\'t share this machine with your model service. It did not start.'],
['the bus refused to start: stopped by the owner', ' Why it stopped: stopped by the owner.'],
// settings-p3 (re-check 3 P3-9): a reason that is not a stop is labeled as the last check.
['the bus state was not read within 2000 ms', ' Last check: the bus state was not read within 2000 ms.'],
]) {
const snapshot = SNAPSHOT();
snapshot.organizations.list[0] = { id: 'acme', bus: { state: 'stopped', lastStart: { ok: false, at: failedAt } }, processing: { halted: false, invalid: false }, config: { valid: true, exit: 0, code: null } };