Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0ba50111f6 | ||
|
|
dab07f0ff5 | ||
|
|
561ff77e48 | ||
|
|
e1642be760 | ||
|
|
b23ccdfc12 | ||
|
|
57af33630d | ||
|
|
17b972b92d | ||
|
|
d531d7181d | ||
|
|
67e1ff193d |
No files matched your search
@@ -1452,3 +1452,73 @@ the model key against a catalogue yet.
|
||||
|
||||
The Console does not read this file: the Triage host keeps `route: null` until
|
||||
the wiring slice.
|
||||
|
||||
## Triage key (#1873)
|
||||
|
||||
`mosaic triage key issue|rotate|retire|status` manage Triage's own model
|
||||
proxy client key. Every verb refuses inside an agent run before anything is
|
||||
read (exit 3), and takes no options.
|
||||
|
||||
The key is `mst-<tag>-triage-<43 base64url>`, where `<tag>` is the first 12
|
||||
hex of sha256 of the config directory, the tag on the instance's own `msk-`
|
||||
sentinel. An org bus reconciles only its `mso-<tag>-<org>-key-` names and a
|
||||
no-org bus only its `msk-<tag>-` names, so neither removes it. The file is
|
||||
`<configDir>/secrets/triage/model`: `{version, key, keyId, issuedAt,
|
||||
rotateBy, retiring}`. The directory is 0700 and the file 0600, owned by the
|
||||
running user. A link, a non-file, a second hard link, a wrong mode or owner, a
|
||||
file over 64 KiB, or a record that does not match the shape refuses every verb
|
||||
with `triage-key-unreadable` (exit 3).
|
||||
|
||||
- `issue` adds one key and writes the file. It refuses `triage-key-issued`
|
||||
when a key is already issued; that is `rotate`.
|
||||
- `rotate` adds a new key and moves the old one to `retiring`, with a retire
|
||||
time of now plus `rotateGraceMinutes` from proxy.json, where the bus reads
|
||||
it (default 120). `rotateBy` is 30 days on, as for an org key. An old key
|
||||
the proxy no longer holds is dropped at once.
|
||||
- `retire` removes retiring keys whose grace has passed or that the proxy no
|
||||
longer holds. It changes nothing before then.
|
||||
|
||||
Nothing runs on a timer. The retire time is the earliest a retiring key may
|
||||
go; the key stays live in the proxy until someone runs `retire` (or a later
|
||||
`rotate` or `issue`) after it. `rotateBy` works the same way: `status` marks
|
||||
it due, and only `rotate` replaces the key.
|
||||
- `status` prints whether a key is issued, its key id, its age, its rotate-by
|
||||
day, each retiring key, what the proxy list holds, a set fence, any key in
|
||||
the Triage namespace the file does not hold, and the next command. It takes
|
||||
no lock and writes nothing.
|
||||
|
||||
`issue`, `rotate` and `retire` hold the instance proxy lock that org buses
|
||||
take (`<dataRoot>/proxy`). Each reads the list, refuses without this
|
||||
instance's sentinel (`triage-key-proxy-open`), reads it again and refuses if
|
||||
it moved (`triage-key-raced`), sets the fence, writes, reads back and clears
|
||||
the fence. The proxy changes first and the file second. Each run that
|
||||
changes something appends one line of key ids, never keys, to
|
||||
`<dataRoot>/proxy/mosaic-keys.receipts.jsonl`. That file is checked under the
|
||||
lock before anything changes; an unsafe one refuses
|
||||
`triage-key-receipts-unsafe`. A receipt line that still fails after the change
|
||||
refuses `triage-key-unrecorded`. No output, error or receipt holds a key.
|
||||
|
||||
The file is written to a 0600 temp file in the same directory, synced, renamed
|
||||
into place, and the directory is synced. A failed directory sync after the
|
||||
rename refuses `triage-key-unsynced`: the file is in place but may not survive
|
||||
a power loss. `proxy-lock-busy` means another proxy writer holds the lock; if
|
||||
none is running, `mosaic proxy fence status` shows the lock state.
|
||||
|
||||
After a crash:
|
||||
|
||||
- Killed after the proxy write and before the read-back: the fence stays set
|
||||
and every proxy writer on the instance refuses `proxy-uncertain`. `status`
|
||||
shows the fence. Restart the proxy, run `mosaic proxy fence clear
|
||||
--proxy-restarted`, then run the verb again.
|
||||
- Killed after the read-back and before the file write: the proxy holds a key
|
||||
the file does not. `status` names it; the next `issue`, `rotate` or `retire`
|
||||
removes it. A file that could not be written refuses
|
||||
`triage-key-unwritten` and says to run the verb again.
|
||||
- Killed after the temp file is written and before its rename: a
|
||||
`.model.<pid>.<hex>.tmp` file that may hold a key stays in the triage
|
||||
directory. `status` counts it; the next `issue`, `rotate` or `retire`
|
||||
removes it under the lock.
|
||||
- A `retire` killed after the proxy write leaves a retiring key the proxy no
|
||||
longer holds. `status` marks it; the next `retire` drops it from the file.
|
||||
|
||||
Nothing gives the key to the Triage host yet; that is the wiring slice.
|
||||
@@ -0,0 +1,392 @@
|
||||
// #1873 triage-key: Triage's own model proxy client key, issued from the owner's terminal
|
||||
// (design report reports/triage-route-design.md slice 3, Jason's ruling 2026-11-18).
|
||||
//
|
||||
// <configDir>/secrets/triage/model (directory 0700, file 0600, the running user's)
|
||||
// { version, key, keyId, issuedAt, rotateBy, retiring: [{ key, keyId, issuedAt, retireAt }] }
|
||||
//
|
||||
// The key is `mst-<instance tag>-triage-<43 base64url>`, where the tag is the first 12 hex of
|
||||
// sha256(configDir), the tag the instance's own `msk-` sentinel carries. An org bus reconciles only
|
||||
// its own `mso-<tag>-<org>-key-` names and a no-org bus only `msk-<tag>-`, so neither touches it.
|
||||
//
|
||||
// Every proxy write runs under the instance proxy lock (packages/bus/src/proxy-lock.mjs), the lock
|
||||
// org buses take: read the list, check this instance's sentinel, fence, PUT, read back, confirm.
|
||||
// The proxy changes first and the file second, as the bus does for an agent key. A crash between
|
||||
// leaves a key in the Triage namespace that the file does not hold; the next issue, rotate or retire
|
||||
// removes it, and status names it. A crash after the PUT and before the read-back leaves the lock's
|
||||
// uncertain fence, which every proxy writer on the instance refuses until a human clears it
|
||||
// (`mosaic proxy fence clear --proxy-restarted`).
|
||||
//
|
||||
// Nothing runs on a timer. rotate records when the old key's grace ends; the key stays in the proxy
|
||||
// until a later retire, rotate or issue removes it, and status says when that is due. The same goes
|
||||
// for rotateBy: status marks it due; only rotate replaces the key.
|
||||
//
|
||||
// Nothing here prints, logs or records a key. Receipts and output carry key ids only (the first 12
|
||||
// hex of sha256(key), the id the bus uses). Messages name files and options, never values.
|
||||
// Nothing wires the key into the Triage host or the Console; that is a later slice.
|
||||
import { closeSync, constants, fchmodSync, fstatSync, fsyncSync, lstatSync, mkdirSync, openSync, readdirSync, readFileSync, renameSync, unlinkSync, writeSync } from "node:fs";
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
import { join } from "node:path";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import { createProxyLock } from "../../bus/src/proxy-lock.mjs";
|
||||
import { loadManagementClient } from "../../model-proxy/src/client.mjs";
|
||||
|
||||
export const KEY_VERSION = 1;
|
||||
// The org key lifetime and the rotate grace default. The bus holds both as literals
|
||||
// (packages/bus/src/model-keys.mjs #issue: 30 days to rotateBy; packages/bus/src/runtime.mjs:
|
||||
// `proxy.rotateGraceMinutes ?? 120`) and exports neither, and this lane may not change the bus.
|
||||
// packages/cli/tests/triage-key.test.mjs drives the bus's own ModelKeys with the same clock and
|
||||
// proves these give the same rotateBy and retireAt, so a change on either side goes red there.
|
||||
// The grace itself is read from proxy.json first, where the bus reads it.
|
||||
export const KEY_LIFETIME_DAYS = 30;
|
||||
export const DEFAULT_ROTATE_GRACE_MINUTES = 120;
|
||||
const FIELDS = Object.freeze(["version", "key", "keyId", "issuedAt", "rotateBy", "retiring"]);
|
||||
const RETIRING_FIELDS = Object.freeze(["key", "keyId", "issuedAt", "retireAt"]);
|
||||
const MAX_BYTES = 64 * 1024;
|
||||
const MAX_RETIRING = 16;
|
||||
const SECRET = /^[A-Za-z0-9_-]{43}$/;
|
||||
const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
|
||||
const DAY = /^\d{4}-\d{2}-\d{2}$/;
|
||||
|
||||
export class TriageKeyError extends Error {
|
||||
constructor(code, message, exitCode = 3) {
|
||||
super(message);
|
||||
this.name = "TriageKeyError";
|
||||
this.code = code;
|
||||
this.exitCode = exitCode;
|
||||
}
|
||||
}
|
||||
const unreadable = (why) => new TriageKeyError("triage-key-unreadable", `the Triage key store is unreadable: ${why}`);
|
||||
|
||||
export const keyIdOf = (key) => createHash("sha256").update(key).digest("hex").slice(0, 12);
|
||||
export const instanceTag = (configDir) => createHash("sha256").update(configDir).digest("hex").slice(0, 12);
|
||||
export const keyPrefix = (configDir) => `mst-${instanceTag(configDir)}-triage-`;
|
||||
const sentinelPrefix = (configDir) => `msk-${instanceTag(configDir)}-sentinel-`;
|
||||
const ownKey = (configDir) => (k) => typeof k === "string" && k.startsWith(keyPrefix(configDir)) && SECRET.test(k.slice(keyPrefix(configDir).length));
|
||||
// Any name under the prefix is Triage's to remove, well formed or not, as the bus treats its own
|
||||
// `msk-<tag>-` names. Only a well-formed key may be stored in the file.
|
||||
const inNamespace = (configDir) => (k) => typeof k === "string" && k.startsWith(keyPrefix(configDir));
|
||||
|
||||
export function keyPaths(configDir) {
|
||||
const secrets = join(configDir, "secrets");
|
||||
const dir = join(secrets, "triage");
|
||||
return { secrets, dir, file: join(dir, "model") };
|
||||
}
|
||||
|
||||
const plain = (v) => v !== null && typeof v === "object" && !Array.isArray(v) && Object.getPrototypeOf(v) === Object.prototype;
|
||||
const exactKeys = (v, keys) => plain(v) && Object.keys(v).length === keys.length && keys.every((k) => Object.hasOwn(v, k));
|
||||
const canonicalTime = (s) => typeof s === "string" && ISO.test(s) && !Number.isNaN(Date.parse(s)) && new Date(s).toISOString() === s;
|
||||
const canonicalDay = (s) => typeof s === "string" && DAY.test(s) && !Number.isNaN(Date.parse(`${s}T00:00:00Z`)) && new Date(`${s}T00:00:00Z`).toISOString().slice(0, 10) === s;
|
||||
|
||||
export function validRecord(doc, configDir) {
|
||||
const own = ownKey(configDir);
|
||||
return exactKeys(doc, FIELDS)
|
||||
&& doc.version === KEY_VERSION
|
||||
&& own(doc.key) && doc.keyId === keyIdOf(doc.key)
|
||||
&& canonicalTime(doc.issuedAt) && canonicalDay(doc.rotateBy)
|
||||
&& Array.isArray(doc.retiring) && doc.retiring.length <= MAX_RETIRING
|
||||
&& doc.retiring.every((r) => exactKeys(r, RETIRING_FIELDS) && own(r.key) && r.key !== doc.key && r.keyId === keyIdOf(r.key)
|
||||
&& canonicalTime(r.issuedAt) && canonicalTime(r.retireAt))
|
||||
&& new Set(doc.retiring.map((r) => r.key)).size === doc.retiring.length;
|
||||
}
|
||||
|
||||
const uidOf = (opts) => opts.uid ?? process.getuid();
|
||||
|
||||
// A real directory, the running user's, mode 0700. Missing: false, or made when `create` is on.
|
||||
function checkDir(dir, uid, what, { create }) {
|
||||
let st;
|
||||
try { st = lstatSync(dir); }
|
||||
catch (e) {
|
||||
if (e.code !== "ENOENT") throw unreadable(`${what} cannot be read`);
|
||||
if (!create) return false;
|
||||
try { mkdirSync(dir, { mode: 0o700 }); } catch { throw unreadable(`${what} cannot be made`); }
|
||||
try {
|
||||
const fd = openSync(dir, constants.O_RDONLY | constants.O_DIRECTORY | constants.O_NOFOLLOW);
|
||||
try { fchmodSync(fd, 0o700); } finally { closeSync(fd); }
|
||||
} catch { throw unreadable(`${what} cannot be made`); }
|
||||
st = lstatSync(dir);
|
||||
}
|
||||
if (st.isSymbolicLink()) throw unreadable(`${what} is a link`);
|
||||
if (!st.isDirectory()) throw unreadable(`${what} is not a directory`);
|
||||
if (st.uid !== uid) throw unreadable(`${what} is not owned by this user`);
|
||||
if ((st.mode & 0o777) !== 0o700) throw unreadable(`${what} mode is not 0700`);
|
||||
return true;
|
||||
}
|
||||
|
||||
function checkFile(fd, uid, what, fail = unreadable) {
|
||||
const st = fstatSync(fd);
|
||||
if (!st.isFile()) throw fail(`${what} is not a regular file`);
|
||||
if (st.nlink !== 1) throw fail(`${what} has more than one link`);
|
||||
if (st.uid !== uid) throw fail(`${what} is not owned by this user`);
|
||||
if ((st.mode & 0o777) !== 0o600) throw fail(`${what} mode is not 0600`);
|
||||
return st;
|
||||
}
|
||||
|
||||
// The stored record, or null when no key is issued. Throws triage-key-unreadable on anything unsafe.
|
||||
// opts.uid (both directories and the file) and opts.fileUid (the file only) are test seams for the
|
||||
// owner checks.
|
||||
export function readKey(configDir, opts = {}) {
|
||||
const { secrets, dir, file } = keyPaths(configDir);
|
||||
const uid = uidOf(opts);
|
||||
if (!checkDir(secrets, uid, "the secrets directory", { create: false })) return null;
|
||||
if (!checkDir(dir, uid, "the triage secrets directory", { create: false })) return null;
|
||||
let fd;
|
||||
try { fd = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); }
|
||||
catch (e) {
|
||||
if (e.code === "ENOENT") return null;
|
||||
if (e.code === "ELOOP") throw unreadable("the key file is a link");
|
||||
throw unreadable("the key file cannot be opened");
|
||||
}
|
||||
try {
|
||||
const st = checkFile(fd, opts.fileUid ?? uid, "the key file");
|
||||
if (st.size > MAX_BYTES) throw unreadable("the key file is too large");
|
||||
let doc;
|
||||
try { doc = JSON.parse(readFileSync(fd, "utf8")); } catch { throw unreadable("the key file is not JSON"); }
|
||||
if (!validRecord(doc, configDir)) throw unreadable("the key file does not match the key shape");
|
||||
return doc;
|
||||
} finally { closeSync(fd); }
|
||||
}
|
||||
|
||||
// Both directories, checked (and the triage one made) before any proxy write, so a store that cannot
|
||||
// be written refuses while the proxy is still untouched.
|
||||
function prepareDirs(configDir, uid) {
|
||||
const { secrets, dir } = keyPaths(configDir);
|
||||
if (!checkDir(secrets, uid, "the secrets directory", { create: false })) throw unreadable("the secrets directory is missing");
|
||||
checkDir(dir, uid, "the triage secrets directory", { create: true });
|
||||
}
|
||||
|
||||
// A temp file writeKey left when it was killed before its rename. It may hold a key.
|
||||
const TEMP = /^\.model\.\d+\.[0-9a-f]{12}\.tmp$/;
|
||||
const tempsIn = (dir) => { try { return readdirSync(dir).filter((n) => TEMP.test(n)); } catch { return []; } };
|
||||
// Under the lock no other Triage writer runs, so every temp file left in the directory is dead.
|
||||
function removeTemps(configDir) {
|
||||
const { dir } = keyPaths(configDir);
|
||||
for (const name of tempsIn(dir)) {
|
||||
try { if (!lstatSync(join(dir, name)).isDirectory()) unlinkSync(join(dir, name)); }
|
||||
catch { throw unreadable("a leftover temp file in the triage secrets directory cannot be removed"); }
|
||||
}
|
||||
}
|
||||
|
||||
// A directory sync that failed after the rename: the file is in place but may not be durable.
|
||||
class NotSynced extends Error {}
|
||||
|
||||
// Atomic: a 0600 temp file in the same directory, fsync, rename, fsync of the directory.
|
||||
function writeKey(configDir, record, uid, hook = () => {}) {
|
||||
const { dir, file } = keyPaths(configDir);
|
||||
prepareDirs(configDir, uid);
|
||||
const temp = join(dir, `.model.${process.pid}.${randomBytes(6).toString("hex")}.tmp`);
|
||||
let fd;
|
||||
try { fd = openSync(temp, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW, 0o600); }
|
||||
catch { throw unreadable("the key file cannot be written"); }
|
||||
try {
|
||||
try { fchmodSync(fd, 0o600); writeSync(fd, `${JSON.stringify(record, null, 2)}\n`); fsyncSync(fd); }
|
||||
finally { closeSync(fd); }
|
||||
hook("rename");
|
||||
renameSync(temp, file);
|
||||
} catch { try { unlinkSync(temp); } catch {} throw unreadable("the key file cannot be written"); }
|
||||
try {
|
||||
hook("sync");
|
||||
const dfd = openSync(dir, constants.O_RDONLY | constants.O_DIRECTORY);
|
||||
try { fsyncSync(dfd); } finally { closeSync(dfd); }
|
||||
} catch { throw new NotSynced(); }
|
||||
}
|
||||
|
||||
// One line per write in the proxy lock's receipts file, ids only. `receiptsReady` runs under the
|
||||
// lock before any change, so an unsafe receipts file refuses while nothing has changed yet.
|
||||
const RECEIPTS_FLAGS = constants.O_WRONLY | constants.O_APPEND | constants.O_CREAT | constants.O_NOFOLLOW | constants.O_NONBLOCK;
|
||||
const receiptsUnsafe = (why) => new TriageKeyError("triage-key-receipts-unsafe", `${why}; nothing was changed`);
|
||||
function receiptsReady(stateDir, uid) {
|
||||
let fd;
|
||||
try { fd = openSync(join(stateDir, "mosaic-keys.receipts.jsonl"), RECEIPTS_FLAGS, 0o600); }
|
||||
catch { throw receiptsUnsafe("the proxy receipts file cannot be opened"); }
|
||||
try { checkFile(fd, uid, "the proxy receipts file", receiptsUnsafe); } finally { closeSync(fd); }
|
||||
}
|
||||
function receipt(stateDir, uid, entry) {
|
||||
const unrecorded = new TriageKeyError("triage-key-unrecorded", "the change was made but its receipt line was not written; run mosaic triage key status");
|
||||
let fd;
|
||||
try { fd = openSync(join(stateDir, "mosaic-keys.receipts.jsonl"), RECEIPTS_FLAGS, 0o600); } catch { throw unrecorded; }
|
||||
try {
|
||||
checkFile(fd, uid, "the proxy receipts file", () => unrecorded);
|
||||
writeSync(fd, `${JSON.stringify(entry)}\n`);
|
||||
fsyncSync(fd);
|
||||
} catch { throw unrecorded; }
|
||||
finally { closeSync(fd); }
|
||||
}
|
||||
|
||||
// The grace the bus uses: proxy.json's rotateGraceMinutes, or the bus default.
|
||||
export const graceMinutes = (proxy) => proxy?.rotateGraceMinutes ?? DEFAULT_ROTATE_GRACE_MINUTES;
|
||||
export const rotateByFor = (nowMs) => new Date(nowMs + KEY_LIFETIME_DAYS * 86400_000).toISOString().slice(0, 10);
|
||||
export const retireAtFor = (nowMs, proxy) => new Date(nowMs + graceMinutes(proxy) * 60_000).toISOString();
|
||||
|
||||
const proxyFailure = (code) => {
|
||||
const words = {
|
||||
"proxy-lock-busy": "another proxy writer holds the instance proxy lock; try again when it ends, and if no writer is running, see mosaic proxy fence status",
|
||||
"proxy-lock-uncertain": "the instance proxy lock is in a state this verb will not guess about; check <dataRoot>/proxy by hand",
|
||||
"proxy-lock-invalid": "the instance proxy lock directory is missing or unsafe, or names another proxy",
|
||||
"proxy-uncertain": "an earlier proxy write has an uncertain outcome; see mosaic proxy fence status",
|
||||
"proxy-management-key-invalid": "the proxy management key file is missing or unsafe",
|
||||
"proxy-client-invalid": "the proxy settings are not usable",
|
||||
"proxy-client-down": "the proxy did not answer",
|
||||
"proxy-client-refused": "the proxy refused the management request",
|
||||
"proxy-client-redirect": "the proxy answered with a redirect",
|
||||
"proxy-client-response": "the proxy gave an answer this verb cannot use",
|
||||
};
|
||||
return new TriageKeyError(code, words[code] ?? "the proxy write failed");
|
||||
};
|
||||
|
||||
// opts.uid, opts.now and opts.hook(stage) are test seams; a stage hook may throw or kill the process.
|
||||
function context({ configDir, system, uid, fileUid, now, hook }) {
|
||||
if (!system?.proxy || !Number.isInteger(system.proxy.port)) throw new TriageKeyError("triage-key-no-proxy", "the instance has no model proxy set up (proxy.json)");
|
||||
return {
|
||||
configDir, dataRoot: system.dataRoot, proxy: system.proxy, stateDir: join(system.dataRoot, "proxy"),
|
||||
uid: uid ?? process.getuid(), fileUid, now: now ?? Date.now, hook: hook ?? (() => {}),
|
||||
};
|
||||
}
|
||||
|
||||
function lockFor(c) {
|
||||
try { return createProxyLock({ stateDir: c.stateDir, identity: `${c.configDir}:${c.proxy.port}` }); }
|
||||
catch (e) { throw proxyFailure(e.code ?? "proxy-lock-invalid"); }
|
||||
}
|
||||
|
||||
async function clientFor(c) {
|
||||
try { return await loadManagementClient({ configDir: c.configDir, proxy: c.proxy }); }
|
||||
catch (e) { throw proxyFailure(e.code ?? "proxy-management-key-invalid"); }
|
||||
}
|
||||
|
||||
function checkSentinel(c, keys) {
|
||||
const prefix = sentinelPrefix(c.configDir);
|
||||
if (!keys.some((k) => k.startsWith(prefix) && SECRET.test(k.slice(prefix.length)))) {
|
||||
throw new TriageKeyError("triage-key-proxy-open", "the proxy's key list has no sentinel for this instance, so it may be another proxy");
|
||||
}
|
||||
}
|
||||
|
||||
// Under the instance proxy lock: read the list, check the sentinel, run `plan`, write the result.
|
||||
// `plan(keys, record)` returns { after, record, removed, added, done }; `record` null leaves the
|
||||
// file alone. The proxy changes first, then the file, then one receipt line.
|
||||
async function change(c, op, plan) {
|
||||
const lock = lockFor(c);
|
||||
const client = await clientFor(c);
|
||||
try {
|
||||
return await lock.run(async () => {
|
||||
const before = await client.getKeys();
|
||||
checkSentinel(c, before);
|
||||
const current = readKey(c.configDir, { uid: c.uid, fileUid: c.fileUid });
|
||||
const result = plan(before, current);
|
||||
if (result.record) prepareDirs(c.configDir, c.uid);
|
||||
removeTemps(c.configDir);
|
||||
receiptsReady(c.stateDir, c.uid);
|
||||
const after = result.after;
|
||||
const removed = before.filter((k) => !after.includes(k));
|
||||
const added = after.filter((k) => !before.includes(k));
|
||||
// A key in another namespace never leaves the list.
|
||||
if (removed.some((k) => !inNamespace(c.configDir)(k))) throw new TriageKeyError("triage-key-failed", "the change would remove a key outside the Triage namespace");
|
||||
if (added.length || removed.length) {
|
||||
// As the bus does: a list that moved since the first read is not overwritten.
|
||||
if (!isDeepStrictEqual(await client.getKeys(), before)) throw new TriageKeyError("triage-key-raced", "the proxy's key list changed while this verb ran; run it again");
|
||||
lock.fence({ org: null, op: `triage.${op}` });
|
||||
await client.putKeys(after);
|
||||
c.hook("put");
|
||||
const confirmed = await client.getKeys();
|
||||
if (!confirmed.some((k) => k.startsWith(sentinelPrefix(c.configDir))) || added.some((k) => !confirmed.includes(k)) || removed.some((k) => confirmed.includes(k))) {
|
||||
throw new TriageKeyError("triage-key-unconfirmed", "the proxy did not confirm the change; the instance proxy fence stays set, see mosaic proxy fence status");
|
||||
}
|
||||
lock.confirm();
|
||||
}
|
||||
c.hook("write");
|
||||
let unsynced = false;
|
||||
if (result.record) {
|
||||
try { writeKey(c.configDir, result.record, c.uid, c.hook); }
|
||||
catch (e) {
|
||||
if (e instanceof NotSynced) unsynced = true;
|
||||
else throw new TriageKeyError("triage-key-unwritten", `the proxy list changed but ${keyPaths(c.configDir).file} was not written (${e.message}); fix the file's directory, then run mosaic triage key ${op} again, which clears what this run left`);
|
||||
}
|
||||
}
|
||||
c.hook("receipt");
|
||||
// A run that changed neither the proxy nor the file leaves no receipt.
|
||||
if (added.length || removed.length || result.record) receipt(c.stateDir, c.uid, { op: `triage.key.${op}`, time: new Date(c.now()).toISOString(), addedKeyIds: added.map(keyIdOf), removedKeyIds: removed.map(keyIdOf) });
|
||||
// The file is in place and the receipt written; only its durability is in doubt.
|
||||
if (unsynced) throw new TriageKeyError("triage-key-unsynced", `${keyPaths(c.configDir).file} was written but its directory could not be synced, so it may not survive a power loss; run mosaic triage key status`);
|
||||
return { ...result.done, addedKeyIds: added.map(keyIdOf), removedKeyIds: removed.map(keyIdOf) };
|
||||
});
|
||||
} catch (e) {
|
||||
if (e instanceof TriageKeyError) throw e;
|
||||
throw proxyFailure(typeof e?.code === "string" && /^proxy-[a-z-]+$/.test(e.code) ? e.code : "triage-key-failed");
|
||||
}
|
||||
}
|
||||
|
||||
const newKey = (configDir) => keyPrefix(configDir) + randomBytes(32).toString("base64url");
|
||||
|
||||
// Keys in the Triage namespace that the file does not hold: left by a crash between the PUT and the
|
||||
// file write, or by a file that was removed by hand.
|
||||
const orphansOf = (configDir, keys, record) => {
|
||||
const held = record ? [record.key, ...record.retiring.map((r) => r.key)] : [];
|
||||
return keys.filter((k) => inNamespace(configDir)(k) && !held.includes(k));
|
||||
};
|
||||
|
||||
export async function issueKey(opts) {
|
||||
const c = context(opts);
|
||||
return change(c, "issue", (before, current) => {
|
||||
if (current) throw new TriageKeyError("triage-key-issued", "a Triage key is already issued; use mosaic triage key rotate");
|
||||
const orphans = orphansOf(c.configDir, before, null);
|
||||
const key = newKey(c.configDir), nowMs = c.now();
|
||||
const record = { version: KEY_VERSION, key, keyId: keyIdOf(key), issuedAt: new Date(nowMs).toISOString(), rotateBy: rotateByFor(nowMs), retiring: [] };
|
||||
return { after: [...before.filter((k) => !orphans.includes(k)), key], record, done: { keyId: record.keyId, rotateBy: record.rotateBy } };
|
||||
});
|
||||
}
|
||||
|
||||
// Retiring keys whose grace has passed, or that the proxy no longer holds.
|
||||
const dueOf = (record, keys, nowMs) => record.retiring.filter((r) => Date.parse(r.retireAt) <= nowMs || !keys.includes(r.key));
|
||||
|
||||
export async function rotateKey(opts) {
|
||||
const c = context(opts);
|
||||
return change(c, "rotate", (before, current) => {
|
||||
if (!current) throw new TriageKeyError("triage-key-not-issued", "no Triage key is issued; use mosaic triage key issue");
|
||||
const nowMs = c.now();
|
||||
const orphans = orphansOf(c.configDir, before, current);
|
||||
const due = dueOf(current, before, nowMs);
|
||||
const keep = before.includes(current.key);
|
||||
const retiring = [...current.retiring.filter((r) => !due.includes(r)),
|
||||
...(keep ? [{ key: current.key, keyId: current.keyId, issuedAt: current.issuedAt, retireAt: retireAtFor(nowMs, c.proxy) }] : [])];
|
||||
if (retiring.length > MAX_RETIRING) throw new TriageKeyError("triage-key-failed", "too many keys are retiring; run mosaic triage key retire once their grace has passed");
|
||||
const key = newKey(c.configDir);
|
||||
const record = { version: KEY_VERSION, key, keyId: keyIdOf(key), issuedAt: new Date(nowMs).toISOString(), rotateBy: rotateByFor(nowMs), retiring };
|
||||
const drop = [...orphans, ...due.map((r) => r.key)];
|
||||
return {
|
||||
after: [...before.filter((k) => !drop.includes(k)), key], record,
|
||||
done: { keyId: record.keyId, rotateBy: record.rotateBy, old: keep ? { keyId: current.keyId, retireAt: retiring.at(-1).retireAt } : { keyId: current.keyId, lost: true } },
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
export async function retireKeys(opts) {
|
||||
const c = context(opts);
|
||||
return change(c, "retire", (before, current) => {
|
||||
const nowMs = c.now();
|
||||
const orphans = orphansOf(c.configDir, before, current);
|
||||
const due = current ? dueOf(current, before, nowMs) : [];
|
||||
const record = current && due.length ? { ...current, retiring: current.retiring.filter((r) => !due.includes(r)) } : null;
|
||||
const drop = [...orphans, ...due.map((r) => r.key)];
|
||||
return { after: before.filter((k) => !drop.includes(k)), record, done: { droppedKeyIds: due.map((r) => r.keyId) } };
|
||||
});
|
||||
}
|
||||
|
||||
// Read only: the file, and when the proxy is set up, what its list holds. No lock, no write.
|
||||
// The fence is checked by name only, so status writes nothing, not even the lock's identity file.
|
||||
export async function keyStatus(opts) {
|
||||
const { configDir, system, uid, fileUid } = opts;
|
||||
const record = readKey(configDir, { uid, fileUid });
|
||||
const proxy = { state: "not-set-up", fence: false, leftoverTemps: tempsIn(keyPaths(configDir).dir).length };
|
||||
if (system?.proxy && Number.isInteger(system.proxy.port)) {
|
||||
try { lstatSync(join(system.dataRoot, "proxy", "mosaic-keys.uncertain")); proxy.fence = true; }
|
||||
catch (e) { if (e.code !== "ENOENT") proxy.fence = true; }
|
||||
try {
|
||||
const client = await loadManagementClient({ configDir, proxy: system.proxy });
|
||||
const keys = await client.getKeys();
|
||||
const prefix = sentinelPrefix(configDir);
|
||||
proxy.state = keys.some((k) => k.startsWith(prefix) && SECRET.test(k.slice(prefix.length))) ? "read" : "no-sentinel";
|
||||
proxy.holdsKey = record ? keys.includes(record.key) : null;
|
||||
proxy.retiringHeld = record ? record.retiring.map((r) => keys.includes(r.key)) : [];
|
||||
proxy.orphanKeyIds = orphansOf(configDir, keys, record).map(keyIdOf);
|
||||
} catch (e) { proxy.state = "unreadable"; proxy.code = typeof e?.code === "string" && /^proxy-[a-z-]+$/.test(e.code) ? e.code : "proxy-client-response"; }
|
||||
}
|
||||
return { record, proxy };
|
||||
}
|
||||
@@ -13,15 +13,23 @@
|
||||
// Nothing here wires the route into the Triage host: the Console keeps `route: null` until the
|
||||
// wiring slice. `set` checks the harness against the repository's harness modules; it does not
|
||||
// check the model key against a catalogue, since none exists outside an organization's bus yet.
|
||||
//
|
||||
// mosaic triage key issue | rotate | retire | status
|
||||
//
|
||||
// Triage's own model proxy client key (packages/cli/src/triage-key.mjs). Human verbs as well,
|
||||
// refused in an agent run before anything is read. No output names a key, only its id.
|
||||
import { lstatSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { readRoute, RouteStoreError, writeRoute } from "../../webui/src/triage-route-store.mjs";
|
||||
import { loadSystem } from "./config.mjs";
|
||||
import { configDir } from "../../organization/src/organization-config.mjs";
|
||||
import { issueKey, keyPaths, keyStatus, retireKeys, rotateKey, TriageKeyError } from "./triage-key.mjs";
|
||||
import { CliError } from "./errors.mjs";
|
||||
import { refuseInsideAgent } from "./transport.mjs";
|
||||
|
||||
export const TRIAGE_USAGE = "mosaic triage route show | set --model KEY --harness NAME --cap TOKENS --time-zone ZONE [--revision N]";
|
||||
export const TRIAGE_USAGE = "mosaic triage route show | set --model KEY --harness NAME --cap TOKENS --time-zone ZONE [--revision N] | key issue | rotate | retire | status";
|
||||
const KEY_VERBS = ["issue", "rotate", "retire", "status"];
|
||||
const HARNESS_ROOT = fileURLToPath(new URL("../../plugins/harness/", import.meta.url));
|
||||
const VALUES = ["--model", "--harness", "--cap", "--time-zone", "--revision"];
|
||||
const WHOLE = /^(0|[1-9][0-9]*)$/;
|
||||
@@ -73,10 +81,89 @@ function show(route) {
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
// A key error prints its code and fixed words. Anything else becomes one fixed line, so no value
|
||||
// that an unexpected error might carry reaches the terminal.
|
||||
function keyError(error) {
|
||||
if (error instanceof CliError) return error;
|
||||
if (error instanceof TriageKeyError) return new CliError(`${error.code}: ${error.message}`, error.exitCode);
|
||||
return new CliError("triage-key-failed: the key verb stopped on an unexpected error; run mosaic triage key status", 3);
|
||||
}
|
||||
|
||||
const age = (ms) => {
|
||||
const m = Math.max(0, Math.floor(ms / 60_000));
|
||||
return `${Math.floor(m / 1440)}d ${Math.floor((m % 1440) / 60)}h ${m % 60}m`;
|
||||
};
|
||||
const ids = (list) => (list.length ? list.join(", ") : "none");
|
||||
|
||||
function statusText({ record, proxy }, nowMs, file) {
|
||||
const lines = [`triage key: ${record ? "issued" : "not issued"}`, `file: ${file}`];
|
||||
const rotateDue = record && Date.parse(`${record.rotateBy}T00:00:00Z`) <= nowMs;
|
||||
const retireDue = record ? record.retiring.filter((r) => Date.parse(r.retireAt) <= nowMs) : [];
|
||||
if (record) {
|
||||
lines.push(`key id: ${record.keyId}`, `issued: ${record.issuedAt} (age ${age(nowMs - Date.parse(record.issuedAt))})`, `rotate by: ${record.rotateBy}${rotateDue ? " (due)" : ""}`);
|
||||
record.retiring.forEach((r, i) => {
|
||||
const held = proxy.retiringHeld?.[i];
|
||||
lines.push(`retiring: ${r.keyId}, live until a retire after ${r.retireAt}${Date.parse(r.retireAt) <= nowMs ? " (due)" : ""}${held === false ? " (not in the proxy list)" : ""}`);
|
||||
});
|
||||
}
|
||||
const state = {
|
||||
"not-set-up": "not set up for this instance (no proxy.json)",
|
||||
"no-sentinel": "its key list has no sentinel for this instance",
|
||||
unreadable: `not read (${proxy.code})`,
|
||||
read: record ? (proxy.holdsKey ? "holds the key" : "does not hold the key") : "read",
|
||||
}[proxy.state];
|
||||
lines.push(`proxy: ${state}`);
|
||||
if (proxy.fence) lines.push("proxy fence: set, an earlier proxy write has an uncertain outcome");
|
||||
if (proxy.leftoverTemps) lines.push(`leftover temp files in the triage secrets directory: ${proxy.leftoverTemps} (the next issue, rotate or retire removes them)`);
|
||||
if (proxy.orphanKeyIds?.length) lines.push(`keys in the Triage namespace the file does not hold: ${ids(proxy.orphanKeyIds)}`);
|
||||
let next;
|
||||
if (proxy.fence) next = "restart the proxy, then mosaic proxy fence clear --proxy-restarted";
|
||||
else if (proxy.state !== "read") next = proxy.state === "not-set-up" ? "set up the model proxy first (mosaic proxy init)" : "check the proxy with mosaic proxy status";
|
||||
else if (!record) next = "mosaic triage key issue";
|
||||
else if (!proxy.holdsKey || rotateDue) next = "mosaic triage key rotate";
|
||||
else if (retireDue.length || proxy.leftoverTemps || proxy.orphanKeyIds?.length || proxy.retiringHeld?.includes(false)) next = "mosaic triage key retire";
|
||||
else next = "nothing";
|
||||
lines.push(`next: ${next}`, "");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function keyCommand(sub, args, io, deps) {
|
||||
if (!KEY_VERBS.includes(sub)) throw usage("unknown triage key command");
|
||||
if (args.length) throw usage(args[0].startsWith("--") ? "unknown option" : "unexpected argument");
|
||||
try {
|
||||
const system = deps.system ?? loadSystem({ env: io.env });
|
||||
const dir = configDir(io.env);
|
||||
const now = deps.now ? () => deps.now().getTime() : Date.now;
|
||||
const opts = { configDir: dir, system, now, uid: deps.triageKeyUid, fileUid: deps.triageKeyFileUid, hook: deps.triageKeyHook };
|
||||
if (sub === "status") {
|
||||
io.stdout.write(statusText(await keyStatus(opts), now(), keyPaths(dir).file));
|
||||
return;
|
||||
}
|
||||
if (sub === "issue") {
|
||||
const r = await issueKey(opts);
|
||||
io.stdout.write(`triage key issued: ${r.keyId}, rotate by ${r.rotateBy}\nfile: ${keyPaths(dir).file}\n`);
|
||||
if (r.removedKeyIds.length) io.stdout.write(`removed keys this file did not hold: ${ids(r.removedKeyIds)}\n`);
|
||||
return;
|
||||
}
|
||||
if (sub === "rotate") {
|
||||
const r = await rotateKey(opts);
|
||||
io.stdout.write(`triage key rotated: ${r.keyId}, rotate by ${r.rotateBy}\n`);
|
||||
io.stdout.write(r.old.lost
|
||||
? `old key ${r.old.keyId} was not in the proxy list; nothing to retire\n`
|
||||
: `old key ${r.old.keyId} stays live until mosaic triage key retire runs at or after ${r.old.retireAt}\n`);
|
||||
if (r.removedKeyIds.length) io.stdout.write(`removed: ${ids(r.removedKeyIds)}\n`);
|
||||
return;
|
||||
}
|
||||
const r = await retireKeys(opts);
|
||||
io.stdout.write(r.removedKeyIds.length ? `triage key retire: removed ${ids(r.removedKeyIds)}\n` : "triage key retire: nothing to remove\n");
|
||||
} catch (e) { throw keyError(e); }
|
||||
}
|
||||
|
||||
export function triageCommand(rest, io, deps = {}) {
|
||||
refuseInsideAgent(process.env);
|
||||
refuseInsideAgent(io.env);
|
||||
const [noun, sub, ...args] = rest;
|
||||
if (noun === "key" && sub !== undefined) return keyCommand(sub, args, io, deps);
|
||||
if (noun !== "route" || !["show", "set"].includes(sub)) throw usage("unknown triage command");
|
||||
const { values, positional } = parse(args);
|
||||
if (positional.length) throw usage("unexpected argument");
|
||||
|
||||
@@ -0,0 +1,695 @@
|
||||
// #1873 triage-key: `mosaic triage key issue|rotate|retire|status`, Triage's own model proxy key.
|
||||
// A stand-in management server on loopback and a temporary config directory only; nothing here
|
||||
// reads a real config, proxy or secret. Every stdout, stderr, receipt and file the verbs leave is
|
||||
// scanned for every key the stand-in ever saw, at the end of each test and once more at the end.
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import http from "node:http";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
import { run } from "../src/cli.mjs";
|
||||
import { AGENT_MARKERS } from "../src/transport.mjs";
|
||||
import { DEFAULT_ROTATE_GRACE_MINUTES, keyPaths, retireAtFor, rotateByFor } from "../src/triage-key.mjs";
|
||||
import { ModelKeys } from "../../bus/src/model-keys.mjs";
|
||||
import { ManagementClient } from "../../model-proxy/src/client.mjs";
|
||||
import { startBroker } from "../../bus/src/runtime.mjs";
|
||||
import { busOrganization } from "../../bus/src/organization-context.mjs";
|
||||
import { Registry } from "../../seat/src/session.mjs";
|
||||
import { loadBusiness } from "../../organization/src/organization-config.mjs";
|
||||
import { resolveInstance } from "../../organization/src/resolve.mjs";
|
||||
import { businessDoc, rolesCopy, systemFor, writeJson } from "../../organization/tests/helpers.mjs";
|
||||
import { humanEnv, io } from "./helpers.mjs";
|
||||
|
||||
const REPO = new URL("../../../", import.meta.url).pathname;
|
||||
const NOW = Date.parse("2026-11-18T12:00:00.000Z");
|
||||
const MIN = 60_000, HOUR = 60 * MIN, DAY = 24 * HOUR;
|
||||
const MANAGE = "fx1873-manage";
|
||||
const KEY_PATH = "/v8/management/config/access/api-keys";
|
||||
const sha = (s) => createHash("sha256").update(s).digest("hex");
|
||||
const tag = (dir) => sha(dir).slice(0, 12);
|
||||
const id = (key) => sha(key).slice(0, 12);
|
||||
const b64 = () => randomBytes(32).toString("base64url");
|
||||
const iso = (ms) => new Date(ms).toISOString();
|
||||
|
||||
// Every output any verb produced in this file, and every Triage key the stand-in ever held.
|
||||
const OUTPUTS = [];
|
||||
const SEEN = new Set();
|
||||
const VERBS = ["issue", "rotate", "retire", "status"];
|
||||
|
||||
function leaks(text, keys = SEEN) {
|
||||
for (const key of keys) {
|
||||
const secret = key.slice(-43);
|
||||
if (text.includes(key) || text.includes(secret)) return true;
|
||||
for (let i = 0; i + 8 <= secret.length; i++) if (text.includes(secret.slice(i, i + 8))) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function walk(dir, skip, out = []) {
|
||||
for (const name of fs.readdirSync(dir)) {
|
||||
const p = join(dir, name), st = fs.lstatSync(p);
|
||||
if (st.isDirectory()) walk(p, skip, out);
|
||||
else if (st.isFile() && !skip.includes(p)) out.push(p);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function fixture(t, { sentinel = true } = {}) {
|
||||
const root = fs.realpathSync(fs.mkdtempSync(join(tmpdir(), "tk-")));
|
||||
const configDir = join(root, "config");
|
||||
fs.mkdirSync(join(configDir, "secrets"), { recursive: true, mode: 0o700 });
|
||||
fs.chmodSync(configDir, 0o700); fs.chmodSync(join(configDir, "secrets"), 0o700);
|
||||
fs.writeFileSync(join(configDir, "secrets", "proxy-management.key"), MANAGE, { mode: 0o600 });
|
||||
fs.mkdirSync(join(root, "proxy"), { mode: 0o700 });
|
||||
const prefix = `mst-${tag(configDir)}-triage-`;
|
||||
const state = {
|
||||
keys: [...(sentinel ? [`msk-${tag(configDir)}-sentinel-${b64()}`] : []), "foreign-client-key", `mso-${tag(configDir)}-demo-key-${b64()}`],
|
||||
gets: 0, puts: [], ignorePut: false, dropSentinelOnPut: false, onGet: null,
|
||||
};
|
||||
const server = http.createServer(async (req, res) => {
|
||||
if (req.url !== KEY_PATH) {
|
||||
const key = req.headers.authorization?.slice(7);
|
||||
res.writeHead(state.keys.includes(key) ? 200 : 401); res.end("{}"); return;
|
||||
}
|
||||
if (req.headers.authorization !== `Bearer ${MANAGE}`) { res.writeHead(401); res.end("{}"); return; }
|
||||
if (req.method === "GET") { state.gets++; const body = JSON.stringify(state.keys); state.onGet?.(state.gets); res.end(body); return; }
|
||||
const chunks = []; for await (const c of req) chunks.push(c);
|
||||
const next = JSON.parse(Buffer.concat(chunks));
|
||||
for (const k of next) if (/^mst-.*-[A-Za-z0-9_-]{43}$/.test(k)) SEEN.add(k);
|
||||
state.puts.push(next);
|
||||
if (state.dropSentinelOnPut) state.keys = next.filter((k) => !k.startsWith("msk-") || !k.includes("-sentinel-"));
|
||||
else if (!state.ignorePut) state.keys = next;
|
||||
res.end('{"status":"ok"}');
|
||||
});
|
||||
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
|
||||
const port = server.address().port;
|
||||
const closers = [];
|
||||
t.after(async () => {
|
||||
for (const close of closers) await close();
|
||||
server.closeAllConnections(); await new Promise((resolve) => server.close(resolve));
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
const f = {
|
||||
root, configDir, prefix, state, server, port, closers, now: NOW,
|
||||
system: { dataRoot: root, proxy: { port, binary: "/fixture-unused" } },
|
||||
file: keyPaths(configDir).file, dir: keyPaths(configDir).dir,
|
||||
env: (extra = {}) => humanEnv({ MOSAIC_CONFIG: join(configDir, "config.json"), ...extra }),
|
||||
async cli(argv, { env, deps = {}, system = f.system } = {}) {
|
||||
const s = io(env ?? f.env());
|
||||
const code = await run(argv, s, { system, now: () => new Date(f.now), ...deps });
|
||||
const out = { code, stdout: s.out.text, stderr: s.err.text };
|
||||
OUTPUTS.push(out.stdout, out.stderr);
|
||||
return out;
|
||||
},
|
||||
// The same verb in a child process that SIGKILLs itself at `stage`.
|
||||
crash(argv, stage) {
|
||||
const script = `
|
||||
import { run } from ${JSON.stringify(new URL("../src/cli.mjs", import.meta.url).href)};
|
||||
import { io } from ${JSON.stringify(new URL("./helpers.mjs", import.meta.url).href)};
|
||||
const p = JSON.parse(process.env.TK_CRASH);
|
||||
const s = io(process.env);
|
||||
const code = await run(p.argv, s, { system: p.system, now: () => new Date(p.now),
|
||||
triageKeyHook: (stage) => { if (stage === p.stage) process.kill(process.pid, "SIGKILL"); } });
|
||||
process.stdout.write(s.out.text); process.stderr.write(s.err.text); process.exitCode = code;`;
|
||||
const env = f.env({ TK_CRASH: JSON.stringify({ argv, stage, system: f.system, now: f.now }) });
|
||||
return new Promise((resolve) => {
|
||||
const child = spawn(process.execPath, ["--input-type=module", "-e", script], { env, stdio: ["ignore", "pipe", "pipe"] });
|
||||
let stdout = "", stderr = "";
|
||||
child.stdout.on("data", (d) => { stdout += d; });
|
||||
child.stderr.on("data", (d) => { stderr += d; });
|
||||
child.on("exit", (code, signal) => { OUTPUTS.push(stdout, stderr); resolve({ code, signal, stdout, stderr }); });
|
||||
});
|
||||
},
|
||||
record: () => JSON.parse(fs.readFileSync(f.file, "utf8")),
|
||||
triageKeys: () => state.keys.filter((k) => k.startsWith(prefix)),
|
||||
receipts: () => {
|
||||
const file = join(root, "proxy", "mosaic-keys.receipts.jsonl");
|
||||
return fs.existsSync(file) ? fs.readFileSync(file, "utf8").split("\n").filter(Boolean).map((l) => JSON.parse(l)) : [];
|
||||
},
|
||||
fenced: () => fs.existsSync(join(root, "proxy", "mosaic-keys.uncertain")),
|
||||
// No file under the fixture but the key file itself holds a key; neither does any output.
|
||||
scan(alsoKeyFiles = []) {
|
||||
for (const k of state.keys) if (/^mst-.*-[A-Za-z0-9_-]{43}$/.test(k)) SEEN.add(k);
|
||||
for (const p of walk(root, [f.file, join(configDir, "secrets", "proxy-management.key"), ...alsoKeyFiles])) {
|
||||
assert.equal(leaks(fs.readFileSync(p, "latin1")), false, `a key is in ${p}`);
|
||||
}
|
||||
for (const text of OUTPUTS) assert.equal(leaks(text), false, "a key is in an output");
|
||||
},
|
||||
};
|
||||
return f;
|
||||
}
|
||||
|
||||
const ok = (r) => assert.equal(r.code, 0, r.stderr);
|
||||
const refusal = (r, code, words) => {
|
||||
assert.equal(r.code, 3, r.stderr);
|
||||
assert.equal(r.stdout, "");
|
||||
assert.match(r.stderr, new RegExp(`^mosaic: ${code}: ${words ?? ""}`));
|
||||
};
|
||||
|
||||
test("issue adds one key under the lock, writes the 0600 file, and records ids only", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const before = [...f.state.keys];
|
||||
const r = await f.cli(["triage", "key", "issue"]);
|
||||
ok(r);
|
||||
const rec = f.record();
|
||||
assert.deepEqual(Object.keys(rec), ["version", "key", "keyId", "issuedAt", "rotateBy", "retiring"]);
|
||||
assert.match(rec.key, new RegExp(`^${f.prefix}[A-Za-z0-9_-]{43}$`));
|
||||
assert.equal(rec.keyId, id(rec.key));
|
||||
assert.equal(rec.issuedAt, iso(NOW));
|
||||
assert.equal(rec.rotateBy, "2026-12-18");
|
||||
assert.deepEqual(rec.retiring, []);
|
||||
assert.equal(r.stdout, `triage key issued: ${rec.keyId}, rotate by 2026-12-18\nfile: ${f.file}\n`);
|
||||
assert.deepEqual(f.state.keys, [...before, rec.key]);
|
||||
assert.equal(f.state.puts.length, 1);
|
||||
const file = fs.lstatSync(f.file), dir = fs.lstatSync(f.dir);
|
||||
assert.equal(file.mode & 0o777, 0o600); assert.equal(dir.mode & 0o777, 0o700);
|
||||
assert.equal(file.uid, process.getuid()); assert.equal(dir.uid, process.getuid());
|
||||
assert.deepEqual(f.receipts(), [{ op: "triage.key.issue", time: iso(NOW), addedKeyIds: [rec.keyId], removedKeyIds: [] }]);
|
||||
assert.equal(f.fenced(), false);
|
||||
assert.equal(fs.readFileSync(join(f.root, "proxy", "keys.generation"), "utf8"), "1\n");
|
||||
// A second issue refuses and changes nothing; rotate is the verb for that.
|
||||
const again = await f.cli(["triage", "key", "issue"]);
|
||||
refusal(again, "triage-key-issued", "a Triage key is already issued; use mosaic triage key rotate\n$");
|
||||
assert.equal(f.state.puts.length, 1);
|
||||
assert.deepEqual(f.record(), rec);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("status reads the file and the list and writes nothing", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const none = await f.cli(["triage", "key", "status"]);
|
||||
ok(none);
|
||||
assert.equal(none.stdout, `triage key: not issued\nfile: ${f.file}\nproxy: read\nnext: mosaic triage key issue\n`);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const rec = f.record(), puts = f.state.puts.length, receipts = f.receipts().length;
|
||||
f.now = NOW + 3 * DAY + 4 * HOUR + 5 * MIN;
|
||||
const shown = await f.cli(["triage", "key", "status"]);
|
||||
ok(shown);
|
||||
assert.equal(shown.stdout, [
|
||||
"triage key: issued", `file: ${f.file}`, `key id: ${rec.keyId}`, `issued: ${iso(NOW)} (age 3d 4h 5m)`,
|
||||
"rotate by: 2026-12-18", "proxy: holds the key", "next: nothing", "",
|
||||
].join("\n"));
|
||||
f.now = Date.parse("2026-12-18T00:00:00.000Z");
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /rotate by: 2026-12-18 \(due\)\n[^]*next: mosaic triage key rotate\n$/);
|
||||
f.now = NOW + HOUR;
|
||||
f.state.keys = f.state.keys.filter((k) => k !== rec.key);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /proxy: does not hold the key\nnext: mosaic triage key rotate\n$/);
|
||||
assert.equal(f.state.puts.length, puts);
|
||||
assert.equal(f.receipts().length, receipts);
|
||||
assert.equal(fs.readdirSync(join(f.root, "proxy")).includes("mosaic-keys.lock"), false);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("rotate adds a new key and keeps the old one for the bus grace; retire removes it once due", async (t) => {
|
||||
const f = await fixture(t);
|
||||
refusal(await f.cli(["triage", "key", "rotate"]), "triage-key-not-issued", "no Triage key is issued; use mosaic triage key issue\n$");
|
||||
assert.equal(f.state.puts.length, 0);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const first = f.record();
|
||||
f.now = NOW + HOUR;
|
||||
const r = await f.cli(["triage", "key", "rotate"]);
|
||||
ok(r);
|
||||
const second = f.record();
|
||||
const retireAt = iso(NOW + HOUR + 120 * MIN);
|
||||
assert.notEqual(second.key, first.key);
|
||||
assert.equal(second.issuedAt, iso(NOW + HOUR));
|
||||
assert.deepEqual(second.retiring, [{ key: first.key, keyId: first.keyId, issuedAt: first.issuedAt, retireAt }]);
|
||||
assert.equal(r.stdout, `triage key rotated: ${second.keyId}, rotate by 2026-12-18\nold key ${first.keyId} stays live until mosaic triage key retire runs at or after ${retireAt}\n`);
|
||||
assert.deepEqual(f.triageKeys(), [first.key, second.key]);
|
||||
f.now = NOW + HOUR + 119 * MIN;
|
||||
const early = await f.cli(["triage", "key", "retire"]);
|
||||
ok(early);
|
||||
assert.equal(early.stdout, "triage key retire: nothing to remove\n");
|
||||
assert.equal(f.state.puts.length, 2);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, new RegExp(`retiring: ${first.keyId}, live until a retire after ${retireAt}\\n[^]*next: nothing\\n$`));
|
||||
f.now = NOW + HOUR + 120 * MIN;
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, new RegExp(`retiring: ${first.keyId}, live until a retire after ${retireAt} \\(due\\)\\n[^]*next: mosaic triage key retire\\n$`));
|
||||
const done = await f.cli(["triage", "key", "retire"]);
|
||||
ok(done);
|
||||
assert.equal(done.stdout, `triage key retire: removed ${first.keyId}\n`);
|
||||
assert.deepEqual(f.triageKeys(), [second.key]);
|
||||
assert.deepEqual(f.record(), { ...second, retiring: [] });
|
||||
assert.deepEqual(f.receipts().map((e) => e.op), ["triage.key.issue", "triage.key.rotate", "triage.key.retire"]);
|
||||
assert.deepEqual(f.receipts()[2].removedKeyIds, [first.keyId]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("rotate takes the grace from proxy.json, where the bus reads it, and drops an old key the proxy lost", async (t) => {
|
||||
const f = await fixture(t);
|
||||
f.system = { ...f.system, proxy: { ...f.system.proxy, rotateGraceMinutes: 30 } };
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const first = f.record();
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
const second = f.record();
|
||||
assert.equal(second.retiring[0].retireAt, iso(NOW + 30 * MIN));
|
||||
// The proxy lost the current key: rotate drops it at once, as the bus does for an agent key.
|
||||
f.state.keys = f.state.keys.filter((k) => k !== second.key);
|
||||
const r = await f.cli(["triage", "key", "rotate"]);
|
||||
ok(r);
|
||||
assert.match(r.stdout, new RegExp(`old key ${second.keyId} was not in the proxy list; nothing to retire\\n$`));
|
||||
assert.deepEqual(f.record().retiring.map((e) => e.keyId), [first.keyId]);
|
||||
// A retiring key past its time, or no longer in the list, leaves on the next rotate.
|
||||
f.now = NOW + 31 * MIN;
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
const last = f.record();
|
||||
assert.deepEqual(last.retiring.map((e) => e.keyId), [id(f.triageKeys()[0])]);
|
||||
assert.equal(f.triageKeys().includes(first.key), false);
|
||||
assert.equal(f.triageKeys().length, 2);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("lifetime and grace match the bus's own ModelKeys for the same clock and proxy settings", async (t) => {
|
||||
const runtime = fs.readFileSync(join(REPO, "packages/bus/src/runtime.mjs"), "utf8");
|
||||
const literal = runtime.match(/rotateGraceMinutes: proxy\.rotateGraceMinutes \?\? (\d+)/);
|
||||
assert.ok(literal, "the bus no longer reads rotateGraceMinutes as this test expects");
|
||||
assert.equal(Number(literal[1]), DEFAULT_ROTATE_GRACE_MINUTES);
|
||||
for (const grace of [undefined, 45]) {
|
||||
const f = await fixture(t);
|
||||
if (grace !== undefined) f.system = { ...f.system, proxy: { ...f.system.proxy, rotateGraceMinutes: grace } };
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
f.now = NOW + 3 * HOUR;
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
const ours = f.record();
|
||||
// The bus, driven the way runtime.mjs drives it, on its own in-memory list.
|
||||
let keys = [`msk-${tag(f.configDir)}-sentinel-${b64()}`], record = null, now = NOW;
|
||||
const credentials = { registerModel() {}, modelRecord: () => record, writeModel: (_, r) => { record = r; }, status: () => [] };
|
||||
const client = { getKeys: async () => [...keys], putKeys: async (k) => { keys = [...k]; return { ok: true }; } };
|
||||
const bus = new ModelKeys({ credentials, client, configDir: f.configDir, baseUrl: "http://127.0.0.1:1", now: () => now,
|
||||
...(grace === undefined ? {} : { rotateGraceMinutes: f.system.proxy.rotateGraceMinutes ?? Number(literal[1]) }) });
|
||||
await bus.reconcile();
|
||||
await bus.ensure("demo/x");
|
||||
assert.equal(record.rotateBy, rotateByFor(NOW));
|
||||
now = NOW + 3 * HOUR;
|
||||
await bus.rotate("demo/x");
|
||||
assert.equal(ours.rotateBy, record.rotateBy);
|
||||
assert.equal(ours.retiring[0].retireAt, record.retiring[0].retireAt);
|
||||
assert.equal(ours.retiring[0].retireAt, retireAtFor(now, f.system.proxy));
|
||||
f.scan();
|
||||
}
|
||||
});
|
||||
|
||||
test("an agent caller is refused before anything is read, for every key verb", async (t) => {
|
||||
const f = await fixture(t);
|
||||
for (const marker of AGENT_MARKERS) {
|
||||
for (const verb of VERBS) {
|
||||
const r = await f.cli(["triage", "key", verb], { env: { ...f.env(), [marker]: "1" } });
|
||||
assert.equal(r.code, 3, marker);
|
||||
assert.equal(r.stderr, `mosaic: refused: this runs only from a human shell, outside any agent run (found ${marker} in the environment)\n`);
|
||||
}
|
||||
const saved = process.env[marker];
|
||||
process.env[marker] = "1";
|
||||
try {
|
||||
for (const verb of VERBS) {
|
||||
const r = await f.cli(["triage", "key", verb]);
|
||||
assert.equal(r.code, 3, marker);
|
||||
assert.match(r.stderr, new RegExp(`found ${marker} in the environment`));
|
||||
}
|
||||
} finally { if (saved === undefined) delete process.env[marker]; else process.env[marker] = saved; }
|
||||
}
|
||||
assert.equal(f.state.gets, 0);
|
||||
assert.equal(f.state.puts.length, 0);
|
||||
assert.equal(fs.existsSync(f.dir), false);
|
||||
assert.deepEqual(fs.readdirSync(join(f.root, "proxy")), []);
|
||||
});
|
||||
|
||||
test("usage: unknown key verbs, options and arguments exit 4 and echo no value", async (t) => {
|
||||
const f = await fixture(t);
|
||||
for (const argv of [["triage", "key"], ["triage", "key", "nope"], ["triage", "key", "issue", "extra"], ["triage", "key", "status", "--json"]]) {
|
||||
const r = await f.cli(argv);
|
||||
assert.equal(r.code, 4, argv.join(" "));
|
||||
assert.equal(r.stdout, "");
|
||||
assert.match(r.stderr, /\nusage: mosaic triage route show [^\n]*\| key issue \| rotate \| retire \| status\n$/);
|
||||
}
|
||||
const r = await f.cli(["triage", "key", "rotate", "--key", "secret-value"]);
|
||||
assert.equal(r.code, 4);
|
||||
assert.match(r.stderr, /^mosaic: unknown option\n/);
|
||||
assert.doesNotMatch(r.stderr, /secret-value/);
|
||||
assert.equal(f.state.gets, 0);
|
||||
});
|
||||
|
||||
test("an unsafe key store refuses every verb before any proxy write", async (t) => {
|
||||
const uid = process.getuid();
|
||||
const cases = [
|
||||
["file mode", (f) => fs.chmodSync(f.file, 0o644), {}, "the key file mode is not 0600"],
|
||||
["file owner", () => {}, { triageKeyFileUid: uid + 1 }, "the key file is not owned by this user"],
|
||||
["file link", (f) => { fs.renameSync(f.file, join(f.dir, "real")); fs.symlinkSync(join(f.dir, "real"), f.file); }, {}, "the key file is a link"],
|
||||
["file is a directory", (f) => { fs.rmSync(f.file); fs.mkdirSync(f.file, { mode: 0o700 }); }, {}, "the key file is not a regular file"],
|
||||
["file is a FIFO", (f) => { fs.rmSync(f.file); assert.equal(spawnSync("mkfifo", ["-m", "600", f.file]).status, 0); }, {}, "the key file is not a regular file"],
|
||||
["hard link", (f) => fs.linkSync(f.file, join(f.dir, "second")), {}, "the key file has more than one link"],
|
||||
["not JSON", (f) => fs.writeFileSync(f.file, "{"), {}, "the key file is not JSON"],
|
||||
["extra field", (f) => fs.writeFileSync(f.file, JSON.stringify({ ...f.record(), extra: 1 })), {}, "the key file does not match the key shape"],
|
||||
["another instance's key", (f) => { const r = f.record(); const key = `mst-${tag("/elsewhere")}-triage-${b64()}`; fs.writeFileSync(f.file, JSON.stringify({ ...r, key, keyId: id(key) })); }, {}, "the key file does not match the key shape"],
|
||||
["wrong key id", (f) => fs.writeFileSync(f.file, JSON.stringify({ ...f.record(), keyId: "000000000000" })), {}, "the key file does not match the key shape"],
|
||||
["too large", (f) => fs.writeFileSync(f.file, " ".repeat(64 * 1024 + 1)), {}, "the key file is too large"],
|
||||
["triage directory mode", (f) => fs.chmodSync(f.dir, 0o755), {}, "the triage secrets directory mode is not 0700"],
|
||||
["triage directory link", (f) => { fs.renameSync(f.dir, join(f.configDir, "secrets", "moved")); fs.symlinkSync(join(f.configDir, "secrets", "moved"), f.dir); }, {}, "the triage secrets directory is a link"],
|
||||
["triage directory is a file", (f) => { fs.rmSync(f.dir, { recursive: true }); fs.writeFileSync(f.dir, "", { mode: 0o600 }); }, {}, "the triage secrets directory is not a directory"],
|
||||
["secrets mode", (f) => fs.chmodSync(join(f.configDir, "secrets"), 0o750), {}, "the secrets directory mode is not 0700"],
|
||||
["owner", () => {}, { triageKeyUid: uid + 1 }, "the secrets directory is not owned by this user"],
|
||||
];
|
||||
for (const [name, spoil, deps, words] of cases) {
|
||||
const f = await fixture(t);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
spoil(f);
|
||||
const keys = [...f.state.keys], puts = f.state.puts.length;
|
||||
for (const verb of VERBS) {
|
||||
const r = await f.cli(["triage", "key", verb], { deps });
|
||||
refusal(r, "triage-key-unreadable", `the Triage key store is unreadable: ${words.replace(/[.()]/g, "\\$&")}\n$`);
|
||||
assert.equal(f.state.puts.length, puts, `${name}: ${verb}`);
|
||||
}
|
||||
assert.deepEqual(f.state.keys, keys);
|
||||
// The spoils above move or link the key file itself; those copies are the key file.
|
||||
f.scan([join(f.dir, "real"), join(f.dir, "second"), join(f.configDir, "secrets", "moved", "model")]);
|
||||
}
|
||||
// No secrets directory at all: status says not issued; issue refuses before the proxy changes.
|
||||
const f = await fixture(t);
|
||||
fs.rmSync(join(f.configDir, "secrets"), { recursive: true });
|
||||
fs.mkdirSync(join(f.configDir, "secrets"), { mode: 0o700 });
|
||||
fs.writeFileSync(join(f.configDir, "secrets", "proxy-management.key"), MANAGE, { mode: 0o600 });
|
||||
fs.chmodSync(join(f.configDir, "secrets"), 0o755);
|
||||
refusal(await f.cli(["triage", "key", "issue"]), "triage-key-unreadable", "the Triage key store is unreadable: the secrets directory mode is not 0700\n$");
|
||||
assert.equal(f.state.puts.length, 0);
|
||||
});
|
||||
|
||||
test("the instance proxy lock, the fence, the sentinel and the proxy itself each refuse plainly", async (t) => {
|
||||
// Lock held by a live process: busy after the lock's wait, and nothing changes.
|
||||
const f = await fixture(t);
|
||||
const lockFile = join(f.root, "proxy", "mosaic-keys.lock");
|
||||
ok(await f.cli(["triage", "key", "status"]));
|
||||
fs.writeFileSync(lockFile, `${JSON.stringify({ pid: process.pid, token: "a".repeat(32) })}\n`, { mode: 0o600 });
|
||||
const busy = await f.cli(["triage", "key", "issue"]);
|
||||
refusal(busy, "proxy-lock-busy", "another proxy writer holds the instance proxy lock; try again when it ends, and if no writer is running, see mosaic proxy fence status\n$");
|
||||
assert.equal(f.state.gets, 1);
|
||||
// A lock left by a process that is gone is reaped, and the verb goes ahead.
|
||||
const dead = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||
fs.writeFileSync(lockFile, `${JSON.stringify({ pid: dead, token: "b".repeat(32) })}\n`, { mode: 0o600 });
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
assert.equal(fs.existsSync(lockFile), false);
|
||||
assert.equal(f.triageKeys().length, 1);
|
||||
fs.rmSync(f.file);
|
||||
f.state.keys = f.state.keys.filter((k) => !k.startsWith(f.prefix));
|
||||
// A set fence: every write refuses, status names it and the way out.
|
||||
const putsBefore = f.state.puts.length;
|
||||
fs.writeFileSync(join(f.root, "proxy", "mosaic-keys.uncertain"), "{}\n", { mode: 0o600 });
|
||||
for (const verb of ["issue", "rotate", "retire"]) refusal(await f.cli(["triage", "key", verb]), "proxy-uncertain");
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /proxy fence: set, an earlier proxy write has an uncertain outcome\nnext: restart the proxy, then mosaic proxy fence clear --proxy-restarted\n$/);
|
||||
assert.equal(f.state.puts.length, putsBefore);
|
||||
// No sentinel for this instance (another instance's does not count).
|
||||
const g = await fixture(t, { sentinel: false });
|
||||
g.state.keys.push(`msk-${tag("/elsewhere")}-sentinel-${b64()}`);
|
||||
refusal(await g.cli(["triage", "key", "issue"]), "triage-key-proxy-open");
|
||||
assert.match((await g.cli(["triage", "key", "status"])).stdout, /proxy: its key list has no sentinel for this instance\nnext: check the proxy with mosaic proxy status\n$/);
|
||||
assert.equal(g.state.puts.length, 0);
|
||||
// No proxy set up, no management key, proxy down.
|
||||
const h = await fixture(t);
|
||||
const bare = { dataRoot: h.root };
|
||||
refusal(await h.cli(["triage", "key", "issue"], { system: bare }), "triage-key-no-proxy");
|
||||
assert.equal((await h.cli(["triage", "key", "status"], { system: bare })).stdout,
|
||||
`triage key: not issued\nfile: ${h.file}\nproxy: not set up for this instance (no proxy.json)\nnext: set up the model proxy first (mosaic proxy init)\n`);
|
||||
fs.chmodSync(join(h.configDir, "secrets", "proxy-management.key"), 0o644);
|
||||
refusal(await h.cli(["triage", "key", "issue"]), "proxy-management-key-invalid");
|
||||
fs.chmodSync(join(h.configDir, "secrets", "proxy-management.key"), 0o600);
|
||||
h.server.closeAllConnections(); await new Promise((resolve) => h.server.close(resolve));
|
||||
h.server.close = (cb) => cb();
|
||||
refusal(await h.cli(["triage", "key", "issue"]), "proxy-client-down");
|
||||
assert.match((await h.cli(["triage", "key", "status"])).stdout, /proxy: not read \(proxy-client-down\)\n/);
|
||||
assert.equal(fs.existsSync(h.file), false);
|
||||
f.scan(); g.scan(); h.scan();
|
||||
});
|
||||
|
||||
test("a write the proxy does not confirm keeps the fence; a list that moves mid-verb is not overwritten", async (t) => {
|
||||
const f = await fixture(t);
|
||||
f.state.ignorePut = true;
|
||||
refusal(await f.cli(["triage", "key", "issue"]), "triage-key-unconfirmed");
|
||||
assert.equal(f.fenced(), true);
|
||||
assert.equal(fs.existsSync(f.file), false);
|
||||
// The store's directory was made and checked before the proxy changed, not after.
|
||||
assert.equal(fs.lstatSync(f.dir).mode & 0o777, 0o700);
|
||||
assert.deepEqual(f.receipts(), []);
|
||||
// A PUT the proxy takes but whose read-back has lost this instance's sentinel is not confirmed.
|
||||
const s = await fixture(t);
|
||||
s.state.dropSentinelOnPut = true;
|
||||
refusal(await s.cli(["triage", "key", "issue"]), "triage-key-unconfirmed");
|
||||
assert.equal(s.fenced(), true);
|
||||
assert.equal(fs.existsSync(s.file), false);
|
||||
s.scan();
|
||||
const g = await fixture(t);
|
||||
g.state.onGet = (n) => { if (n === 1) g.state.keys = [...g.state.keys, "late-foreign-key"]; };
|
||||
refusal(await g.cli(["triage", "key", "issue"]), "triage-key-raced");
|
||||
assert.equal(g.state.puts.length, 0);
|
||||
assert.equal(g.fenced(), false);
|
||||
f.scan(); g.scan();
|
||||
});
|
||||
|
||||
test("keys outside the Triage namespace are never touched; Triage keys the file does not hold are removed", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const agent = `msk-${tag(f.configDir)}-${b64()}`, other = `mst-${tag("/elsewhere")}-triage-${b64()}`;
|
||||
const orphan = `${f.prefix}${b64()}`, odd = `${f.prefix}short`;
|
||||
f.state.keys.push(agent, other, orphan, odd);
|
||||
const foreign = f.state.keys.filter((k) => k !== orphan && k !== odd);
|
||||
const r = await f.cli(["triage", "key", "issue"]);
|
||||
ok(r);
|
||||
// A malformed name under the Triage prefix is still Triage's, as the bus treats its own prefix.
|
||||
assert.match(r.stdout, new RegExp(`removed keys this file did not hold: ${id(orphan)}, ${id(odd)}\\n$`));
|
||||
for (const verb of ["rotate", "retire"]) { f.now += 3 * HOUR; ok(await f.cli(["triage", "key", verb])); }
|
||||
for (const k of foreign) assert.ok(f.state.keys.includes(k), "a key outside the namespace left the list");
|
||||
assert.equal(f.state.keys.includes(orphan), false);
|
||||
assert.equal(f.state.keys.includes(odd), false);
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("crash after the PUT, before the read-back: the fence holds; status says so; clear, then issue recovers", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const crashed = await f.crash(["triage", "key", "issue"], "put");
|
||||
assert.equal(crashed.signal, "SIGKILL");
|
||||
assert.equal(f.fenced(), true);
|
||||
assert.equal(f.triageKeys().length, 1);
|
||||
assert.equal(fs.existsSync(f.file), false);
|
||||
const status = await f.cli(["triage", "key", "status"]);
|
||||
ok(status);
|
||||
assert.match(status.stdout, new RegExp(`proxy fence: set[^\\n]*\\nkeys in the Triage namespace the file does not hold: ${id(f.triageKeys()[0])}\\nnext: restart the proxy, then mosaic proxy fence clear --proxy-restarted\\n$`));
|
||||
refusal(await f.cli(["triage", "key", "issue"]), "proxy-uncertain");
|
||||
ok(await f.cli(["proxy", "fence", "clear", "--proxy-restarted"]));
|
||||
assert.equal(f.fenced(), false);
|
||||
const stray = f.triageKeys()[0];
|
||||
const r = await f.cli(["triage", "key", "issue"]);
|
||||
ok(r);
|
||||
assert.match(r.stdout, new RegExp(`removed keys this file did not hold: ${id(stray)}\\n$`));
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("crash after the read-back, before the file write: no fence; status names the stray key; issue removes it", async (t) => {
|
||||
const f = await fixture(t);
|
||||
assert.equal((await f.crash(["triage", "key", "issue"], "write")).signal, "SIGKILL");
|
||||
assert.equal(f.fenced(), false);
|
||||
assert.equal(fs.existsSync(f.file), false);
|
||||
const [stray] = f.triageKeys();
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout,
|
||||
new RegExp(`^triage key: not issued\\n[^]*keys in the Triage namespace the file does not hold: ${id(stray)}\\nnext: mosaic triage key issue\\n$`));
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
// The same crash in rotate: the file keeps the old key, the stray new one is named, retire removes it.
|
||||
const held = f.record();
|
||||
assert.equal((await f.crash(["triage", "key", "rotate"], "write")).signal, "SIGKILL");
|
||||
assert.deepEqual(f.record(), held);
|
||||
const strayNew = f.triageKeys().find((k) => k !== held.key);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout,
|
||||
new RegExp(`proxy: holds the key\\nkeys in the Triage namespace the file does not hold: ${id(strayNew)}\\nnext: mosaic triage key retire\\n$`));
|
||||
const retired = await f.cli(["triage", "key", "retire"]);
|
||||
ok(retired);
|
||||
assert.equal(retired.stdout, `triage key retire: removed ${id(strayNew)}\n`);
|
||||
assert.deepEqual(f.triageKeys(), [held.key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("file written but the old key not yet retired: status shows it due; a crash inside retire recovers on the next retire", async (t) => {
|
||||
const f = await fixture(t);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const first = f.record();
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
f.now = NOW + 121 * MIN;
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, new RegExp(`retiring: ${first.keyId}, live until a retire after [^\\n]* \\(due\\)\\n[^]*next: mosaic triage key retire\\n$`));
|
||||
// retire dies after the proxy dropped the key but before the file was rewritten.
|
||||
assert.equal((await f.crash(["triage", "key", "retire"], "write")).signal, "SIGKILL");
|
||||
assert.equal(f.state.keys.includes(first.key), false);
|
||||
assert.equal(f.record().retiring.length, 1);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /\(due\) \(not in the proxy list\)\n[^]*next: mosaic triage key retire\n$/);
|
||||
const puts = f.state.puts.length;
|
||||
const r = await f.cli(["triage", "key", "retire"]);
|
||||
ok(r);
|
||||
assert.equal(f.state.puts.length, puts);
|
||||
assert.deepEqual(f.record().retiring, []);
|
||||
// And a retire killed after its PUT leaves the fence, like any other write.
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
f.now += 121 * MIN;
|
||||
assert.equal((await f.crash(["triage", "key", "retire"], "put")).signal, "SIGKILL");
|
||||
assert.equal(f.fenced(), true);
|
||||
refusal(await f.cli(["triage", "key", "retire"]), "proxy-uncertain");
|
||||
ok(await f.cli(["proxy", "fence", "clear", "--proxy-restarted"]));
|
||||
ok(await f.cli(["triage", "key", "retire"]));
|
||||
assert.deepEqual(f.record().retiring, []);
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("a file that cannot be written after the proxy changed is reported with its recovery", async (t) => {
|
||||
const f = await fixture(t);
|
||||
ok(await f.cli(["triage", "key", "status"]));
|
||||
fs.mkdirSync(f.dir, { mode: 0o700 });
|
||||
const r = await f.cli(["triage", "key", "issue"], { deps: { triageKeyHook: (stage) => { if (stage === "write") fs.chmodSync(f.dir, 0o755); } } });
|
||||
refusal(r, "triage-key-unwritten", "the proxy list changed but ");
|
||||
assert.match(r.stderr, /run mosaic triage key issue again, which clears what this run left\n$/);
|
||||
assert.equal(f.triageKeys().length, 1);
|
||||
fs.chmodSync(f.dir, 0o700);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("crash after the temp file, before its rename: status counts it; the next verb removes it", async (t) => {
|
||||
const f = await fixture(t);
|
||||
assert.equal((await f.crash(["triage", "key", "issue"], "rename")).signal, "SIGKILL");
|
||||
assert.equal(fs.existsSync(f.file), false);
|
||||
const temps = fs.readdirSync(f.dir).filter((n) => n.endsWith(".tmp"));
|
||||
assert.equal(temps.length, 1);
|
||||
assert.match(temps[0], /^\.model\.\d+\.[0-9a-f]{12}\.tmp$/);
|
||||
// The temp file holds the key the proxy now has; that is why it must go.
|
||||
const [stray] = f.triageKeys();
|
||||
assert.ok(fs.readFileSync(join(f.dir, temps[0]), "utf8").includes(stray));
|
||||
const status = await f.cli(["triage", "key", "status"]);
|
||||
ok(status);
|
||||
assert.match(status.stdout, new RegExp(`leftover temp files in the triage secrets directory: 1 \\(the next issue, rotate or retire removes them\\)\\nkeys in the Triage namespace the file does not hold: ${id(stray)}\\nnext: mosaic triage key issue\\n$`));
|
||||
// Something else in the directory is not a leftover and stays.
|
||||
fs.writeFileSync(join(f.dir, "notes.tmp"), "", { mode: 0o600 });
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
assert.deepEqual(fs.readdirSync(f.dir).sort(), ["model", "notes.tmp"]);
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
// Retire, with nothing else to do, still removes a leftover.
|
||||
assert.equal((await f.crash(["triage", "key", "rotate"], "rename")).signal, "SIGKILL");
|
||||
ok(await f.cli(["triage", "key", "retire"]));
|
||||
assert.deepEqual(fs.readdirSync(f.dir).sort(), ["model", "notes.tmp"]);
|
||||
assert.doesNotMatch((await f.cli(["triage", "key", "status"])).stdout, /leftover/);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("the receipts file is checked before the proxy changes; a receipt that fails after is named", async (t) => {
|
||||
const receipts = (f) => join(f.root, "proxy", "mosaic-keys.receipts.jsonl");
|
||||
const spoils = [
|
||||
["mode", (f) => { fs.writeFileSync(receipts(f), "", { mode: 0o644 }); fs.chmodSync(receipts(f), 0o644); }, "the proxy receipts file mode is not 0600"],
|
||||
["link", (f) => { fs.writeFileSync(join(f.root, "elsewhere"), "", { mode: 0o600 }); fs.symlinkSync(join(f.root, "elsewhere"), receipts(f)); }, "the proxy receipts file cannot be opened"],
|
||||
["FIFO", (f) => assert.equal(spawnSync("mkfifo", ["-m", "600", receipts(f)]).status, 0), "the proxy receipts file cannot be opened"],
|
||||
];
|
||||
for (const [name, spoil, words] of spoils) {
|
||||
const f = await fixture(t);
|
||||
spoil(f);
|
||||
refusal(await f.cli(["triage", "key", "issue"]), "triage-key-receipts-unsafe", `${words}; nothing was changed\n$`);
|
||||
assert.equal(f.state.puts.length, 0, name);
|
||||
assert.equal(f.fenced(), false, name);
|
||||
assert.equal(fs.existsSync(f.file), false, name);
|
||||
f.scan();
|
||||
}
|
||||
// Safe before the change, unsafe at the receipt: the change stands and the refusal says so.
|
||||
const f = await fixture(t);
|
||||
const r = await f.cli(["triage", "key", "issue"], { deps: { triageKeyHook: (stage) => { if (stage === "receipt") fs.chmodSync(receipts(f), 0o644); } } });
|
||||
refusal(r, "triage-key-unrecorded", "the change was made but its receipt line was not written; run mosaic triage key status\n$");
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
assert.equal(f.fenced(), false);
|
||||
assert.equal(fs.readFileSync(receipts(f), "utf8"), "");
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("a directory sync that fails after the rename is named, and the receipt is still written", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const r = await f.cli(["triage", "key", "issue"], { deps: { triageKeyHook: (stage) => { if (stage === "sync") throw new Error("EIO"); } } });
|
||||
refusal(r, "triage-key-unsynced", `${f.file.replace(/[.]/g, "\\.")} was written but its directory could not be synced, so it may not survive a power loss; run mosaic triage key status\n$`);
|
||||
const rec = f.record();
|
||||
assert.deepEqual(f.triageKeys(), [rec.key]);
|
||||
assert.deepEqual(f.receipts().map((e) => e.addedKeyIds), [[rec.keyId]]);
|
||||
assert.equal(f.fenced(), false);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /proxy: holds the key\nnext: nothing\n$/);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("an unexpected error prints fixed words, never the error's own text", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const secret = `mst-${tag(f.configDir)}-triage-${b64()}`;
|
||||
SEEN.add(secret);
|
||||
// Outside the proxy write: the verb's own fallback line.
|
||||
const broken = { dataRoot: f.root, get proxy() { throw new Error(`boom ${secret}`); } };
|
||||
for (const verb of VERBS) {
|
||||
const r = await f.cli(["triage", "key", verb], { system: broken });
|
||||
assert.equal(r.code, 3, verb);
|
||||
assert.equal(r.stderr, "mosaic: triage-key-failed: the key verb stopped on an unexpected error; run mosaic triage key status\n");
|
||||
}
|
||||
// Inside the proxy write: the write's own fixed line, and the fence stays for a human.
|
||||
const r = await f.cli(["triage", "key", "issue"], { deps: { triageKeyHook: (stage) => { if (stage === "put") throw new Error(`boom ${secret}`); } } });
|
||||
refusal(r, "triage-key-failed", "the proxy write failed\n$");
|
||||
assert.equal(f.fenced(), true);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
// The org bus reconciles its own `mso-` namespace at start and a no-org bus its `msk-` one; a
|
||||
// Triage key sits in neither, so both leave it. Stray keys in each namespace show the reconcile ran.
|
||||
test("an org bus that starts and reconciles, and a no-org reconcile, both leave the Triage key alone", async (t) => {
|
||||
const f = await fixture(t);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const triage = f.record().key;
|
||||
const strayOrg = `mso-${tag(f.configDir)}-demo-key-${b64()}`;
|
||||
f.state.keys.push(strayOrg);
|
||||
const rolesDir = rolesCopy(f.root), system = systemFor(f.root), doc = businessDoc(f.root, "demo");
|
||||
doc.plugins = { ...doc.plugins, harness: ["pi"] };
|
||||
const file = writeJson(join(f.configDir, "businesses", "demo.json"), doc);
|
||||
assert.ok(file);
|
||||
const business = loadBusiness("demo", { dir: f.configDir, rolesDir }), resolved = {};
|
||||
for (const instance of Object.keys(business.roles)) resolved[instance] = resolveInstance({ system, business, instance });
|
||||
fs.mkdirSync(join(f.root, "data"), { mode: 0o700 });
|
||||
new Registry(join(f.root, "data"));
|
||||
const bus = await startBroker({ dataRoot: join(f.root, "data"), businesses: { demo: busOrganization(business, resolved) },
|
||||
organization: { dir: f.configDir, rolesDir, system }, proxy: { port: f.port, binary: "/fixture-unused" },
|
||||
org: { id: "demo" }, instanceDataRoot: f.root, instanceConfigDir: f.configDir });
|
||||
f.closers.push(() => bus.close());
|
||||
const orgKey = (await bus.modelPreflight({ business: "demo", role: "chief" })).key;
|
||||
assert.ok(orgKey.startsWith(`mso-${tag(f.configDir)}-demo-key-`));
|
||||
assert.equal(f.state.keys.includes(strayOrg), false, "the org reconcile did not run");
|
||||
assert.ok(f.state.keys.includes(triage), "the org bus removed the Triage key");
|
||||
// A no-org reconcile on the same instance.
|
||||
const strayAgent = `msk-${tag(f.configDir)}-${b64()}`;
|
||||
f.state.keys.push(strayAgent);
|
||||
const client = new ManagementClient({ baseUrl: `http://127.0.0.1:${f.port}`, key: MANAGE });
|
||||
const keys = new ModelKeys({ credentials: { status: () => [], modelRecord: () => null }, client, configDir: f.configDir, baseUrl: `http://127.0.0.1:${f.port}` });
|
||||
await keys.reconcile();
|
||||
assert.equal(f.state.keys.includes(strayAgent), false, "the no-org reconcile did not run");
|
||||
assert.ok(f.state.keys.includes(triage), "the no-org reconcile removed the Triage key");
|
||||
// Triage still works under the shared lock while the org bus runs, and leaves the org key.
|
||||
f.now += HOUR;
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
assert.ok(f.state.keys.includes(orgKey));
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /proxy: holds the key\n/);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("scripts/mosaic routes triage key status with the instance's own config directory", (t) => {
|
||||
const home = fs.mkdtempSync(join(tmpdir(), "tkm-"));
|
||||
t.after(() => fs.rmSync(home, { recursive: true, force: true }));
|
||||
const env = humanEnv({ HOME: home, MOSAIC_CONFIG: join(home, ".config/mosaic-tktest/config.json") });
|
||||
const setup = spawnSync(join(REPO, "scripts/bootstrap.sh"), ["--instance", "tktest"], { env, encoding: "utf8" });
|
||||
assert.equal(setup.status, 0, setup.stderr);
|
||||
const r = spawnSync(join(REPO, "scripts/mosaic"), ["triage", "key", "status"], { env, encoding: "utf8", timeout: 60000 });
|
||||
OUTPUTS.push(r.stdout, r.stderr);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.equal(r.stdout, `triage key: not issued\nfile: ${join(home, ".config/mosaic-tktest/secrets/triage/model")}\nproxy: not set up for this instance (no proxy.json)\nnext: set up the model proxy first (mosaic proxy init)\n`);
|
||||
const refused = spawnSync(join(REPO, "scripts/mosaic"), ["triage", "key", "issue"], { env: { ...env, CLAUDECODE: "1" }, encoding: "utf8", timeout: 60000 });
|
||||
assert.equal(refused.status, 3);
|
||||
assert.match(refused.stderr, /found CLAUDECODE in the environment/);
|
||||
});
|
||||
|
||||
test("no output of any verb in this file holds any key the stand-in saw", () => {
|
||||
assert.ok(SEEN.size >= 20, `only ${SEEN.size} keys were seen`);
|
||||
assert.ok(OUTPUTS.length >= 100);
|
||||
for (const text of OUTPUTS) assert.equal(leaks(text), false);
|
||||
});
|
||||
@@ -38,6 +38,7 @@ import { createTriageHost } from './triage-host.mjs';
|
||||
import { composeSnapshot, readSystemConfig, readOrgConfig } from './triage-snapshot.mjs';
|
||||
import { busesStopped } from '../../cli/src/proxy-admin-host.mjs';
|
||||
import { agentLevelsWriteRoutes } from './agent-levels-write.mjs';
|
||||
import { triageRouteRoutes } from './triage-route-http.mjs';
|
||||
import { RESTART_EXIT } from '../../cli/src/org-console.mjs';
|
||||
import { ID_PATTERN, systemOnly } from '../../organization/src/vocabulary.mjs';
|
||||
import { selectorScope } from '../../organization/src/selector.mjs';
|
||||
@@ -262,8 +263,24 @@ try {
|
||||
const trail = system ? actionTrail(join(system.dataRoot, 'console')) : null;
|
||||
// #1873 A2: Settings > Agents writes for the root and project levels, beside the levels read.
|
||||
const agentLevelRoutes = organizations ? [agentLevelsWriteRoutes({ configDir: configDir(), rolesDir: rolesDir(), trail })] : [];
|
||||
// #1873 triage-settings: Settings > Triage reads and saves the route store for the instance owner.
|
||||
// The model is checked against the catalogue each active organization's bus answers with, as an
|
||||
// organization's route change is. The Triage host above still gets route: null; saving starts nothing.
|
||||
const triageRoutes = system && (devOwner || ssoOn) ? [triageRouteRoutes({
|
||||
dataRoot: system.dataRoot, log: line => console.log(line),
|
||||
catalogues: async () => {
|
||||
const read = async call => { try { return (await call('routes'))?.catalogue ?? { status: 'unknown' }; } catch { return { status: 'bus-unavailable' }; } };
|
||||
if (!organizations) return ownerCall ? [await read(ownerCall)] : [];
|
||||
// Every active organization is asked at once, so a hung bus costs one timeout, not one each.
|
||||
const active = (await listOrgs({ configDir: configDir(), dataRoot: system.dataRoot, instance })).filter(org => org.state === 'active');
|
||||
return Promise.all(active.map(org => {
|
||||
let paths; try { paths = orgPaths(org.root); } catch { return { status: 'bus-unavailable' }; }
|
||||
return read(busCaller({ dataRoot: org.root, socket: paths.socket, business: org.id, timeoutMs: 5000 }));
|
||||
}));
|
||||
},
|
||||
})] : [];
|
||||
try {
|
||||
server = await startServer({ restart, trail, ...(lan ? { host: lan.bind, devAllowHosts: lan.allowHosts } : {}), board: options.board, port: options.port, bus, reads, preview: options.preview, sso, proof, send, settingsWrite, projectFiles, projectFile, libraryFile, libraryStatus, librarySync, libraryRevert, librarySave, projectSave, chat, nameWrite, assistantStop, instancesWrite, routesWrite, agentWrite, proxyCall, proxyAdmin, triage, processing, decisionWrite, requestRead, requestWrite, userOptions, federation: system?.federation ?? null, organizations, agentLevels, routes: agentLevelRoutes, preselect: options.business ?? null });
|
||||
server = await startServer({ restart, trail, ...(lan ? { host: lan.bind, devAllowHosts: lan.allowHosts } : {}), board: options.board, port: options.port, bus, reads, preview: options.preview, sso, proof, send, settingsWrite, projectFiles, projectFile, libraryFile, libraryStatus, librarySync, libraryRevert, librarySave, projectSave, chat, nameWrite, assistantStop, instancesWrite, routesWrite, agentWrite, proxyCall, proxyAdmin, triage, processing, decisionWrite, requestRead, requestWrite, userOptions, federation: system?.federation ?? null, organizations, agentLevels, routes: [...agentLevelRoutes, ...triageRoutes], preselect: options.business ?? null });
|
||||
} catch (err) { sso?.close(); throw err; }
|
||||
if (testerSocket) {
|
||||
const call = busCaller({ dataRoot: system.dataRoot, socket: socketPath(system.dataRoot), business: options.business ?? null, cli: TESTER_CLI });
|
||||
|
||||
@@ -321,7 +321,7 @@
|
||||
// render. That read marks nothing. Row 150 (#1644): since row 142, `mosaic talk` prints only the
|
||||
// reply to its own message, so a reply to a Console send stays unread in `mosaic inbox` until
|
||||
// the human marks it read with `mosaic inbox --read`.
|
||||
let isTester = false, orgSetup = false;
|
||||
let isTester = false, orgSetup = false, triageOwner = false;
|
||||
// Both Docs and project direction use the same owner-only records setup action.
|
||||
window.MosaicRecords = {
|
||||
markup(project) {
|
||||
@@ -1431,13 +1431,13 @@
|
||||
});
|
||||
});
|
||||
// Organizations is writable when the server allows org setup, which it does in organization mode where send is off.
|
||||
const readonlyPage = page => !['appearance','business','console'].includes(page) && !(page === 'organizations' && orgSetup) && !canSend && !isTester;
|
||||
const settingsPages = [['appearance','Appearance'],['business',window.MosaicOrg ? 'Organization' : 'Business'],['limits','Launch limits'],['organizations','Organizations'],['routing','Routing'],['agents','Agents'],['assistant','Assistant'],['tokens','Token use'],['model-proxy','Model proxy'],['console','Console'],['about','About']];
|
||||
const readonlyPage = page => !['appearance','business','console','triage'].includes(page) && !(page === 'organizations' && orgSetup) && !canSend && !isTester;
|
||||
const settingsPages = [['appearance','Appearance'],['business',window.MosaicOrg ? 'Organization' : 'Business'],['limits','Launch limits'],['organizations','Organizations'],['routing','Routing'],['agents','Agents'],['assistant','Assistant'],['tokens','Token use'],['model-proxy','Model proxy'],['triage','Triage'],['console','Console'],['about','About']];
|
||||
function markSettingsDraft(page,dirty){const link=document.querySelector(`#settings-nav a[href="#/settings/${page}"]`), dot='<span class="settings-dot"><span class="sr-only">unsaved changes</span></span>';if(link && dirty && !link.querySelector('.settings-dot'))link.insertAdjacentHTML('beforeend',dot);const button=$('settings-disclosure');if(dirty && link?.getAttribute('aria-current')==='page' && button && !button.querySelector('.settings-dot'))button.querySelector('strong').insertAdjacentHTML('beforeend',dot);}
|
||||
function settingsShell(page, body, helper='') {
|
||||
const pages=isTester ? [['appearance','Appearance'],['business','Business'],['routing','Routing'],['agents','Agents'],['names','Seat names']] : settingsPages;
|
||||
const pages=isTester ? [['appearance','Appearance'],['business','Business'],['routing','Routing'],['agents','Agents'],['names','Seat names']] : settingsPages.filter(([id])=>id!=='triage'||triageOwner);
|
||||
const label=pages.find(([id])=>id===page)?.[1] || 'Appearance';
|
||||
const isDirty=id=>(id==='routing'&&routesDraft)||(id==='limits'&&settingsInputs)||(id==='assistant'&&(Object.keys(assistantDrafts).length||Object.keys(nameDrafts).length))||(id==='agents'&&(instanceDrafts.add||instanceDrafts.change||window.MosaicAgentLevels?.hasDraft()));
|
||||
const isDirty=id=>(id==='routing'&&routesDraft)||(id==='limits'&&settingsInputs)||(id==='assistant'&&(Object.keys(assistantDrafts).length||Object.keys(nameDrafts).length))||(id==='agents'&&(instanceDrafts.add||instanceDrafts.change||window.MosaicAgentLevels?.hasDraft()))||(id==='triage'&&window.MosaicTriageRoute?.hasDraft());
|
||||
const dot='<span class="settings-dot"><span class="sr-only">unsaved changes</span></span>';
|
||||
const links=pages.map(([id,name],i)=>`${i===0?'<div class="settings-group">Your browser</div>':i===1?'<div class="settings-group">This instance</div>':''}<a href="#/settings/${id}"${id===page?' aria-current="page"':''}>${name}${isDirty(id)?dot:''}</a>`).join('') + (!isTester ? '<div class="settings-group">Members and invites <span class="muted">not built</span></div>' : '');
|
||||
return `${head('Settings','Your browser and this instance.')}<div class="settings-layout"><div><button type="button" id="settings-disclosure" aria-expanded="false" aria-controls="settings-nav"><small>Section</small><strong>${label}${isDirty(page)?dot:''}</strong><span aria-hidden="true">⌄</span></button><nav id="settings-nav" aria-label="Settings sections">${links}</nav></div><div class="settings-content settings-${page}${readonlyPage(page)?' readonly':''}"><h2 id="settings-page-title" tabindex="-1">${label}</h2><p class="settings-helper">${helper}</p>${readonlyPage(page)?'<p class="settings-readonly">Read only. Saving requires signed-in send mode.</p>':''}${body}</div></div>`;
|
||||
@@ -1447,6 +1447,8 @@
|
||||
if (page==='appearance') return settingsShell(page,appearance(),'These choices apply in this browser only.');
|
||||
if (page==='organizations') { await configRead; return settingsShell(page,window.MosaicOrgCreate.markup,'Create an organization from the template, edit it, and finish its setup. Each step names the command it equals.'); }
|
||||
if (page==='model-proxy') { await configRead; return settingsShell(page,window.MosaicProxy.markup,'Accounts used by this instance, their priority, and the client keys assigned to agents.'); }
|
||||
// #1873 triage-settings: Settings > Triage (triage-settings.js), the instance owner's only; anyone else gets no page.
|
||||
if (page==='triage') { await configRead; if (!triageOwner || !window.MosaicTriageRoute) return notFound(); return settingsShell(page,await window.MosaicTriageRoute.section(),'The model, harness and daily token cap Triage uses. Only the instance owner sees this page.'); }
|
||||
// F3: Settings > Console (console-restart.js). The server checks the owner, Origin and trail.
|
||||
if (page==='console') { await configRead; await window.MosaicConsole?.ready; return settingsShell(page,window.MosaicConsole?.settingsMarkup()??'<section class="panel"><p class="small warn">The Console controls did not load. Reload the page to try again.</p></section>','Restart this Console on the same address and configuration.'); }
|
||||
const s = await get.api('settings'), n = s.notifier || {};
|
||||
@@ -1910,13 +1912,14 @@
|
||||
return;
|
||||
}
|
||||
if (path === '/settings/model-proxy' && shown === route && !navigated) return;
|
||||
if (path === '/settings/triage' && shown === route && !navigated) return;
|
||||
if (path === '/settings/organizations' && shown?.split('?')[0] === path && !navigated) return;
|
||||
if (($('agent-dialog')?.open || document.querySelector('.agent-more[aria-expanded=true]')) && !navigated) { schedule(); return; }
|
||||
if (navigated || (shown !== route && !samePage)) { unmountEditor(); $('bus-view').innerHTML = loadingView(SOURCES[path.split('/')[1]]); }
|
||||
let html, state = 'fresh';
|
||||
const live = reader(true);
|
||||
try {
|
||||
if (path !== '/organization' && !wsPage(path) && !(path === '/agents' && window.MosaicOrg) && path !== '/messages' && !['/settings','/settings/appearance','/settings/model-proxy','/settings/organizations'].includes(path)) {
|
||||
if (path !== '/organization' && !wsPage(path) && !(path === '/agents' && window.MosaicOrg) && path !== '/messages' && !['/settings','/settings/appearance','/settings/model-proxy','/settings/triage','/settings/organizations'].includes(path)) {
|
||||
try { await live('messages'); }
|
||||
catch (err) { messageCount = null; if (err.status === 403 || err.code === 'signed-out') throw err; }
|
||||
}
|
||||
@@ -2113,6 +2116,8 @@
|
||||
canMessage = !isTester && c?.send === true;
|
||||
chatSend = !isTester && c?.chatSend === true;
|
||||
orgSetup = !isTester && c?.orgSetup === true;
|
||||
// #1873 triage-settings: /api/config says proxyAdmin to the instance owner only, the same owner /api/triage/route serves.
|
||||
triageOwner = !isTester && c?.proxyAdmin === true;
|
||||
window.MosaicOrgBusStopped = !!window.MosaicOrg && c?.devOwner === true && orgSetup && c?.chatSend === false;
|
||||
// The header menu offers "New organization" to the owner of a Console with organizations; nobody else can create one.
|
||||
if (orgSetup && !$('org-new-link')) { const a = document.createElement('a'); a.id = 'org-new-link'; a.href = NEW_ORG; a.setAttribute('role', 'menuitem'); a.textContent = 'New organization'; document.querySelector('#avatar-menu > a[href="#/settings"]')?.after(a); }
|
||||
|
||||
@@ -43,6 +43,7 @@ window.MosaicLocalTime = value => Number.isFinite(Date.parse(value)) ? new Date(
|
||||
x.addEventListener('click', close);
|
||||
hd.append(title, x);
|
||||
const state = el('p', 'triage-state'); state.id = 'triage-state'; state.setAttribute('role', 'status');
|
||||
const usage = el('p', 'note triage-usage'); usage.id = 'triage-usage'; usage.hidden = true;
|
||||
const healthSec = el('details', 'triage-health-sec'); healthSec.open = true;
|
||||
const healthSum = el('summary', '', 'Health');
|
||||
const health = el('ul', 'triage-health'); health.id = 'triage-health';
|
||||
@@ -51,10 +52,10 @@ window.MosaicLocalTime = value => Number.isFinite(Date.parse(value)) ? new Date(
|
||||
const transcript = el('ol', 'triage-transcript'); transcript.id = 'triage-transcript'; transcript.tabIndex = 0;
|
||||
transcript.setAttribute('aria-label', 'Triage conversation');
|
||||
const compose = el('div', 'triage-compose');
|
||||
drawer.append(hd, state, healthSec, convo, transcript, compose);
|
||||
drawer.append(hd, state, usage, healthSec, convo, transcript, compose);
|
||||
drawer.addEventListener('keydown', e => { if (e.key === 'Escape') { e.preventDefault(); close(); } });
|
||||
document.body.append(drawer);
|
||||
parts = { title, state, health, transcript, compose, box: null, send: null, status: null, off: null };
|
||||
parts = { title, state, usage, health, transcript, compose, box: null, send: null, status: null, off: null };
|
||||
}
|
||||
|
||||
// The message box is built once and never rewritten or removed by a read, so a draft survives every
|
||||
@@ -109,6 +110,19 @@ window.MosaicLocalTime = value => Number.isFinite(Date.parse(value)) ? new Date(
|
||||
};
|
||||
|
||||
const notRead = v => v === 'not read' || v === null || v === undefined || typeof v !== 'object';
|
||||
// #1873 triage-cap: one line, today's count against the stored cap and the next reset. Only what the
|
||||
// read holds: a field the read says is not set or not read is said so, never filled in.
|
||||
const usageLine = view => {
|
||||
if (!Object.hasOwn(view, 'usageToday')) return '';
|
||||
const u = view.usageToday, c = view.cap;
|
||||
if (u === 'not set' || c === 'not set') return 'Daily token cap: not set.';
|
||||
if (notRead(u) || notRead(c)) return "Today's token usage: not read.";
|
||||
const n = v => (Number.isSafeInteger(v) ? v.toLocaleString() : 'not read');
|
||||
const reset = typeof view.resetsAt === 'string' && Number.isFinite(Date.parse(view.resetsAt)) ? ` Resets at ${when(view.resetsAt)}.` : '';
|
||||
const k = Number.isSafeInteger(u.unknownTurns) && u.unknownTurns > 0 ? u.unknownTurns : 0;
|
||||
const unknown = k ? ` ${k} ${k === 1 ? 'turn has' : 'turns have'} no usage reading.` : '';
|
||||
return `Today: ${n(u.count)} of ${n(c.limit)} tokens.${reset}${unknown}`;
|
||||
};
|
||||
const configWords = c => (notRead(c) ? 'not read' : c.valid === true ? 'valid' : `not valid${c.code ? ` (${clean(c.code)})` : ''}`);
|
||||
function healthLines(s) {
|
||||
if (notRead(s)) return ['Health: not read'];
|
||||
@@ -173,6 +187,9 @@ window.MosaicLocalTime = value => Number.isFinite(Date.parse(value)) ? new Date(
|
||||
const t = parts.transcript, atBottom = t.scrollHeight - t.scrollTop - t.clientHeight < 24, top = t.scrollTop;
|
||||
parts.state.textContent = stateLine(view);
|
||||
parts.state.dataset.kind = view.route !== 'set' ? 'off' : view.overCapUntil ? 'cap' : view.running === true ? 'running' : 'stopped';
|
||||
const usage = usageLine(view);
|
||||
parts.usage.textContent = usage;
|
||||
parts.usage.hidden = !usage;
|
||||
const lines = healthLines(view.snapshot);
|
||||
const healthKey = JSON.stringify(lines);
|
||||
if (healthKey !== lastHealth) { lastHealth = healthKey; parts.health.replaceChildren(...lines.map(l => el('li', l.endsWith('not read') ? 'not-read' : '', l))); }
|
||||
@@ -328,7 +345,7 @@ window.MosaicLocalTime = value => Number.isFinite(Date.parse(value)) ? new Date(
|
||||
return orgOf(hash) || !seg || system(hash) ? hash : `#/orgs/${seg}${path}`;
|
||||
};
|
||||
// /api/workspace reads and writes across organizations (U1, U4b), so it keeps its own path.
|
||||
const unscoped = /^\/api\/(?:workspace(?:\/|\?|$)|orgs(?:\/|$)|themes(?:\?|$)|org-template(?:\?|$)|org-presets(?:\?|$)|org-admin(?:\?|$)|org-validate(?:\?|$)|agents\/(?:levels|root)(?:\?|$))/;
|
||||
const unscoped = /^\/api\/(?:workspace(?:\/|\?|$)|orgs(?:\/|$)|themes(?:\?|$)|org-template(?:\?|$)|org-presets(?:\?|$)|org-admin(?:\?|$)|org-validate(?:\?|$)|agents\/(?:levels|root)(?:\?|$)|triage\/route(?:\?|$))/;
|
||||
window.fetch = (input, init) => {
|
||||
if (typeof input !== 'string' || !input.startsWith('/api/') || unscoped.test(input)) return nativeFetch(input, init);
|
||||
const send = () => nativeFetch(current ? `/api/orgs/${current}/${input.slice('/api/'.length)}` : input, init);
|
||||
|
||||
@@ -7,7 +7,7 @@
|
||||
<link rel="stylesheet" href="/processing.css"><script src="/processing.js" defer></script>
|
||||
<link rel="stylesheet" href="/org-create.css"><script src="/org-create.js" defer></script>
|
||||
<link rel="stylesheet" href="/onboarding.css"><script src="/onboarding.js" defer></script><link rel="stylesheet" href="/assistant.css">
|
||||
<script src="/workspace-data.js" defer></script><script src="/workspace.js" defer></script><script src="/tasks-board.js" defer></script><script src="/console-restart.js" defer></script><script src="/display.js" defer></script><script src="/inbox.js" defer></script><script src="/files.js" defer></script><script src="/project-remote.js" defer></script><link rel="stylesheet" href="/autonomy.css"><script src="/autonomy.js" defer></script><link rel="stylesheet" href="/packs.css"><script src="/packs.js" defer></script><link rel="stylesheet" href="/workflows.css"><script src="/workflows.js" defer></script><script src="/federation.js" defer></script><script src="/brand.js" defer></script><script src="/app.js" defer></script><script src="/markdown-preview.js" defer></script><script src="/projects.js" defer></script><script src="/chat.js" defer></script><script src="/project-archive.js" defer></script><script src="/agents.js" defer></script><script src="/assistant.js" defer></script><script src="/bus.js" defer></script><script src="/requests.js" defer></script></head>
|
||||
<script src="/workspace-data.js" defer></script><script src="/workspace.js" defer></script><script src="/tasks-board.js" defer></script><script src="/console-restart.js" defer></script><script src="/triage-settings.js" defer></script><script src="/display.js" defer></script><script src="/inbox.js" defer></script><script src="/files.js" defer></script><script src="/project-remote.js" defer></script><link rel="stylesheet" href="/autonomy.css"><script src="/autonomy.js" defer></script><link rel="stylesheet" href="/packs.css"><script src="/packs.js" defer></script><link rel="stylesheet" href="/workflows.css"><script src="/workflows.js" defer></script><script src="/federation.js" defer></script><script src="/brand.js" defer></script><script src="/app.js" defer></script><script src="/markdown-preview.js" defer></script><script src="/projects.js" defer></script><script src="/chat.js" defer></script><script src="/project-archive.js" defer></script><script src="/agents.js" defer></script><script src="/assistant.js" defer></script><script src="/bus.js" defer></script><script src="/requests.js" defer></script></head>
|
||||
<body>
|
||||
<div hidden aria-hidden="true"><select id="palette"></select><select id="mode"><option value="system">system</option><option value="light">light</option><option value="dim">dim</option><option value="dark">dark</option></select></div>
|
||||
<a class="skip" href="#main">Skip to content</a>
|
||||
|
||||
@@ -832,6 +832,7 @@ html[data-organizations] .ws-drawer-acts .note{flex-basis:100%}
|
||||
#triage-drawer h2:focus{outline:none}
|
||||
#triage-drawer .x{margin-left:auto;flex:none;width:32px;height:32px;border:1px solid var(--line);border-radius:6px;background:var(--canvas);color:var(--strong);font-size:20px;line-height:1;cursor:pointer}
|
||||
#triage-drawer .triage-state{margin:0;padding:10px 14px;flex:none;font-size:14px;font-weight:600;color:var(--strong);border-bottom:1px solid var(--line);overflow-wrap:anywhere}
|
||||
#triage-drawer .triage-usage{margin:0;padding:6px 14px;flex:none;font-size:13px;color:var(--muted);border-bottom:1px solid var(--line);overflow-wrap:anywhere}
|
||||
#triage-drawer .triage-state[data-kind=off],#triage-drawer .triage-state[data-kind=cap]{color:var(--warning);background:color-mix(in srgb,var(--warning) 9%,var(--surface))}
|
||||
#triage-drawer .triage-state[data-kind=running]::before{content:"";display:inline-block;width:8px;height:8px;margin-right:7px;border-radius:50%;background:var(--success);vertical-align:1px}
|
||||
#triage-drawer .triage-health-sec{flex:none;padding:8px 14px;border-bottom:1px solid var(--line)}
|
||||
|
||||
@@ -0,0 +1,133 @@
|
||||
/* #1873 triage-settings: Settings > Triage, the instance owner's Triage model route. It reads and
|
||||
writes /api/triage/route (triage-route-http.mjs); the server checks the owner, the Origin and Fetch
|
||||
Metadata, the catalogue and the time zone, and the store keeps the revision and the receipt. This
|
||||
file only asks and reports. bus.js offers the page only when /api/config says proxyAdmin, which is
|
||||
the same owner the route serves. The fields live here, so a repaint keeps what the owner typed. */
|
||||
(() => {
|
||||
'use strict';
|
||||
const $ = id => document.getElementById(id);
|
||||
const esc = s => String(s).replace(/[&<>"']/g, c => ({ '&': '&', '<': '<', '>': '>', '"': '"', "'": ''' })[c]);
|
||||
const PATH = '/api/triage/route';
|
||||
// The time zones this browser knows, offered as suggestions only; the server decides.
|
||||
const zones = (() => { try { return Intl.supportedValuesOf('timeZone'); } catch { return []; } })();
|
||||
// draft holds the typed fields; base is the revision they were read at, which a save sends, so a
|
||||
// change saved elsewhere since then answers as stale instead of being written over.
|
||||
let data = null, readLine = '', draft = null, base = 0, busy = false, line = '', conflict = false;
|
||||
|
||||
const call = init => window.fetch(PATH, { cache: 'no-store', redirect: 'manual', ...init });
|
||||
// The stored route's fields, or empty ones: there is no default model, harness, cap or zone.
|
||||
const fromRoute = r => ({ model: r?.model ?? '', harness: r?.harness ?? '', cap: r ? String(r.cap) : '', timeZone: r?.timeZone ?? '' });
|
||||
|
||||
async function read() {
|
||||
readLine = '';
|
||||
let r, a = null;
|
||||
try { r = await call({ method: 'GET' }); } catch { data = null; readLine = 'The Triage route could not be read. Reload the page to try again.'; return; }
|
||||
try { a = await r.json(); } catch { /* refused below */ }
|
||||
if (r.type === 'opaqueredirect') { data = null; readLine = 'Sign in again, then reload.'; return; }
|
||||
if (!r.ok || !a || typeof a !== 'object') { data = null; readLine = typeof a?.message === 'string' ? a.message : 'The Triage route could not be read.'; return; }
|
||||
// An untouched page follows the store; typed fields stay, with the revision they were read at.
|
||||
const untouched = draft === null || JSON.stringify(draft) === JSON.stringify(fromRoute(data?.route ?? null));
|
||||
data = a;
|
||||
if (untouched) { draft = fromRoute(a.route); base = Number.isSafeInteger(a.revision) ? a.revision : 0; }
|
||||
}
|
||||
|
||||
const models = () => (Array.isArray(data?.choices?.models) ? data.choices.models : []);
|
||||
const harnessesOf = key => models().find(m => m.key === key)?.harnesses ?? [];
|
||||
|
||||
function stored() {
|
||||
const r = data?.route;
|
||||
if (!r) return `<p id="triage-route-stored"><strong>Not set.</strong> ${esc(data?.routeMessage ?? 'No Triage route is set.')}</p>`;
|
||||
return `<p id="triage-route-stored">Stored route: <strong>${esc(r.model)}</strong> on <strong>${esc(r.harness)}</strong>, cap ${esc(Number(r.cap).toLocaleString('en-US'))} tokens a day, reset ${esc(r.reset)} ${esc(r.timeZone)}. Revision ${esc(r.revision)}, saved ${esc(r.updatedAt)}.</p>`;
|
||||
}
|
||||
|
||||
function form() {
|
||||
const d = draft ?? fromRoute(null);
|
||||
const send = data?.send === true;
|
||||
const off = busy || !send ? ' disabled' : '';
|
||||
const list = models();
|
||||
const offered = list.some(m => m.key === d.model);
|
||||
const modelOptions = `<option value=""${d.model === '' ? ' selected' : ''}>Choose a model</option>`
|
||||
+ (d.model && !offered ? `<option value="${esc(d.model)}" selected>${esc(d.model)} (not offered now)</option>` : '')
|
||||
+ list.map(m => `<option value="${esc(m.key)}"${m.key === d.model ? ' selected' : ''}${m.available ? '' : ' disabled'}>${esc(m.label && m.label !== m.key ? `${m.label} (${m.key})` : m.key)}${m.available ? '' : ' (unavailable)'}</option>`).join('');
|
||||
const hs = harnessesOf(d.model);
|
||||
const harnessOptions = `<option value=""${d.harness === '' ? ' selected' : ''}>${d.model ? 'Choose a harness' : 'Choose a model first'}</option>`
|
||||
+ (d.harness && !hs.includes(d.harness) ? `<option value="${esc(d.harness)}" selected>${esc(d.harness)} (not offered for this model)</option>` : '')
|
||||
+ hs.map(h => `<option value="${esc(h)}"${h === d.harness ? ' selected' : ''}>${esc(h)}</option>`).join('');
|
||||
const status = data?.choices?.status;
|
||||
const catalogue = status === 'ready' ? '' : `<p class="small warn" id="triage-route-catalogue">No organization bus offered a model catalogue (${esc(status ?? 'unknown')}), so no route can be saved now.</p>`;
|
||||
const conflictButton = conflict ? ' <button type="button" class="btn" id="triage-route-reload">Reload route</button>' : '';
|
||||
return `${catalogue}<form id="triage-route-form" novalidate><div class="filters">`
|
||||
+ `<label>Model <select id="triage-route-model"${off}>${modelOptions}</select></label>`
|
||||
+ `<label>Harness <select id="triage-route-harness"${off}>${harnessOptions}</select></label>`
|
||||
+ `<label>Daily cap, tokens <input id="triage-route-cap" type="number" min="1" step="1" inputmode="numeric" value="${esc(d.cap)}"${off}></label>`
|
||||
+ `<label>Reset <input id="triage-route-reset" value="00:00" readonly aria-readonly="true"></label>`
|
||||
+ `<label>Time zone <input id="triage-route-zone" list="triage-route-zones" autocomplete="off" spellcheck="false" placeholder="Region/City" value="${esc(d.timeZone)}"${off}></label>`
|
||||
+ `</div><datalist id="triage-route-zones">${zones.map(z => `<option value="${esc(z)}"></option>`).join('')}</datalist>`
|
||||
+ `<div class="settings-actions"><button type="submit" id="triage-route-save"${off}>${busy ? 'Saving…' : 'Save'}</button>${conflictButton} <span role="status" id="triage-route-line" tabindex="-1">${esc(line)}</span></div>`
|
||||
+ `</form>${send ? '' : '<p class="small muted" id="triage-route-readonly">This Console is read-only, so the route cannot be saved here.</p>'}`;
|
||||
}
|
||||
|
||||
function markup() {
|
||||
const body = data === null
|
||||
? `<p class="small warn">${esc(readLine || 'The Triage route could not be read.')}</p><p role="status" id="triage-route-line" tabindex="-1">${esc(line)}</p>`
|
||||
: `${stored()}${form()}<p class="small muted" id="triage-route-note">${esc(data.note ?? 'Saving a route does not start Triage. The key and the wiring are separate.')}</p>`;
|
||||
return `<section class="panel" id="triage-route" aria-labelledby="triage-route-title"><h3 id="triage-route-title">Triage model route</h3>${body}</section>`;
|
||||
}
|
||||
const repaint = () => { const panel = $('triage-route'); if (panel) panel.outerHTML = markup(); };
|
||||
|
||||
// Called by bus.js when the page draws: a fresh read each time, the typed fields kept.
|
||||
async function section() { await read(); return markup(); }
|
||||
|
||||
async function save() {
|
||||
if (busy || data?.send !== true) return;
|
||||
const d = draft ?? fromRoute(null);
|
||||
const cap = /^\d+$/.test(d.cap.trim()) ? Number(d.cap.trim()) : d.cap.trim() === '' ? null : Number.NaN;
|
||||
const body = { model: d.model, harness: d.harness, cap: Number.isFinite(cap) ? cap : d.cap, timeZone: d.timeZone.trim(), revision: base };
|
||||
busy = true; conflict = false; line = 'Saving…'; repaint();
|
||||
let r, a = null;
|
||||
try { r = await call({ method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify(body) }); }
|
||||
catch { busy = false; line = 'Outcome unknown. Reload the route to see whether it was saved.'; conflict = true; repaint(); return; }
|
||||
try { a = await r.json(); } catch { /* refused below */ }
|
||||
busy = false;
|
||||
if (r.ok && a?.route) {
|
||||
data = { ...data, route: a.route, revision: a.revision, routeMessage: undefined };
|
||||
draft = fromRoute(a.route); base = a.revision;
|
||||
line = `Saved. Revision ${a.revision}.`;
|
||||
repaint();
|
||||
$('triage-route-line')?.focus?.();
|
||||
return;
|
||||
}
|
||||
if (r.type === 'opaqueredirect') line = 'Sign in again, then retry.';
|
||||
else line = typeof a?.message === 'string' ? a.message : 'The route was refused. Nothing was saved.';
|
||||
if (r.status === 409 && a?.error === 'triage-route-stale') conflict = true;
|
||||
repaint();
|
||||
$(conflict ? 'triage-route-reload' : 'triage-route-save')?.focus();
|
||||
}
|
||||
|
||||
async function reload() {
|
||||
if (busy) return;
|
||||
draft = null; conflict = false; line = '';
|
||||
await read();
|
||||
line = data ? 'Route read again. Your fields now show the stored route.' : '';
|
||||
repaint();
|
||||
}
|
||||
|
||||
const field = { 'triage-route-model': 'model', 'triage-route-harness': 'harness', 'triage-route-cap': 'cap', 'triage-route-zone': 'timeZone' };
|
||||
document.addEventListener('input', e => {
|
||||
const k = field[e.target?.id]; if (!k || !$('triage-route')) return;
|
||||
draft = { ...(draft ?? fromRoute(null)), [k]: e.target.value };
|
||||
});
|
||||
document.addEventListener('change', e => {
|
||||
if (e.target?.id !== 'triage-route-model') return;
|
||||
// A new model narrows the harness list; a harness the model does not take is cleared.
|
||||
const d = draft = { ...(draft ?? fromRoute(null)), model: e.target.value };
|
||||
const hs = harnessesOf(d.model);
|
||||
if (!hs.includes(d.harness)) d.harness = hs.length === 1 ? hs[0] : '';
|
||||
repaint(); $('triage-route-model')?.focus();
|
||||
});
|
||||
document.addEventListener('submit', e => { if (e.target?.id !== 'triage-route-form') return; e.preventDefault(); save(); });
|
||||
document.addEventListener('click', e => { if (e.target?.id === 'triage-route-reload') reload(); });
|
||||
|
||||
const hasDraft = () => !!(draft && data && JSON.stringify(draft) !== JSON.stringify(fromRoute(data.route)));
|
||||
window.MosaicTriageRoute = { section, hasDraft };
|
||||
})();
|
||||
@@ -1117,16 +1117,21 @@
|
||||
};
|
||||
const DRAWER_LIMIT = 5, STOP_REASON = 'Stopped from the agent drawer.';
|
||||
const STOP_UNKNOWN = 'Outcome unknown. Read the agents again before trying; nothing was retried.';
|
||||
// The Stop state of the drawer's agent survives repaints: `confirm` while it asks, `busy` while the
|
||||
// SP20: an organization that refuses a Stop or a Restart, in the same line for both.
|
||||
const orgRefusal = (e, orgName) => e === 'org-archived' ? `${orgName} is archived, so its agents are not changed. Nothing changed.`
|
||||
: e === 'org-unavailable' ? `${orgName} is not available to this Console. Nothing changed.` : '';
|
||||
// Each drawer agent's question and Stop survive repaints: `confirm` while it asks, `busy` while the
|
||||
// request is open, then the result in words beside the button. SP19: `kind` says whether Stop or
|
||||
// Restart asks (no kind is Stop). While either asks or runs, both buttons wait.
|
||||
let stopState = null; // { id, kind, confirm, busy, text }
|
||||
// Restart asks (no kind is Stop). While either asks or runs, both buttons wait. SP20: kept per agent,
|
||||
// as restarts are, so two Stops in two drawers each keep their line. An entry goes when the drawer
|
||||
// closes after it finished, when its question is cancelled, or when its Restart starts.
|
||||
const stops = new Map(); // id -> { kind, confirm, busy, text }
|
||||
// SP19: each drawer agent's restart, kept per agent so several can run at once and none loses its
|
||||
// line when another drawer asks or stops: `busy` while it runs, then its outcome in words. An entry
|
||||
// goes when a new question starts on that agent, or when the drawer closes after it finished.
|
||||
const restarts = new Map(); // id -> { busy, text }
|
||||
// The state a drawer draws: its own question or Stop first, else its restart.
|
||||
const actState = id => stopState?.id === id ? stopState : restarts.has(id) ? { id, kind: 'restart', ...restarts.get(id) } : null;
|
||||
const actState = id => stops.has(id) ? { id, ...stops.get(id) } : restarts.has(id) ? { id, kind: 'restart', ...restarts.get(id) } : null;
|
||||
const RESTART_REASON = 'restart from the agent drawer';
|
||||
let drawerFocus = false, drawerShown = null;
|
||||
// The drawer holds at most one history entry. A row opened from the page without a drawer pushes one
|
||||
@@ -1396,7 +1401,7 @@
|
||||
// Closing goes Back when a row link on this page pushed the current entry; otherwise it drops
|
||||
// `agent` from the address in place. Either way the page repaints without it and focus returns to the row.
|
||||
function closeDrawer() {
|
||||
if (!stopState?.busy) stopState = null;
|
||||
for (const [k, st] of stops) if (!st.busy) stops.delete(k);
|
||||
for (const [k, r] of restarts) if (!r.busy) restarts.delete(k);
|
||||
if (stateOf()?.[DRAWER_MARK] && queryOf().has('agent')) { history.back(); return; }
|
||||
history.replaceState(unmarked(), '', sansAgent());
|
||||
@@ -1433,10 +1438,11 @@
|
||||
}
|
||||
async function stopAgent(d) {
|
||||
const id = d.dataset.wsDrawer, org = d.dataset.drawerOrgId, name = id.slice(org.length + 1), orgName = d.dataset.drawerOrgName;
|
||||
stopState = { id, busy: true, text: `Stopping ${name}…` };
|
||||
stops.set(id, { busy: true, text: `Stopping ${name}…` });
|
||||
const repaint = () => document.dispatchEvent(new CustomEvent('mosaic-workspace-render'));
|
||||
const paint = () => { const now = document.querySelector(`[data-ws-drawer="${CSS.escape(id)}"]`); if (!now) return; const s = now.querySelector('[data-drawer-status]'); if (s) s.textContent = stopState.text; const b = now.querySelector('[data-drawer-stop]'); if (b) { b.disabled = !!stopState.busy; b.firstChild.textContent = stopState.busy ? 'Stopping…' : 'Stop'; } const rb = now.querySelector('[data-drawer-restart]'); if (rb) rb.disabled = !!stopState.busy; now.querySelector('[data-drawer-stop-confirm]')?.remove(); };
|
||||
const finish = text => { stopState = { id, text }; paint(); const b = document.querySelector(`[data-ws-drawer="${CSS.escape(id)}"] [data-drawer-stop]`); b?.focus({ preventScroll: true }); };
|
||||
// Paint this agent's drawer in place, when it is on screen.
|
||||
const paint = () => { const now = document.querySelector(`[data-ws-drawer="${CSS.escape(id)}"]`), st = stops.get(id); if (!now || !st) return; const s = now.querySelector('[data-drawer-status]'); if (s) s.textContent = st.text; const b = now.querySelector('[data-drawer-stop]'); if (b) { b.disabled = !!st.busy; b.firstChild.textContent = st.busy ? 'Stopping…' : 'Stop'; } const rb = now.querySelector('[data-drawer-restart]'); if (rb) rb.disabled = !!st.busy; now.querySelector('[data-drawer-stop-confirm]')?.remove(); };
|
||||
const finish = text => { stops.set(id, { text }); paint(); const b = document.querySelector(`[data-ws-drawer="${CSS.escape(id)}"] [data-drawer-stop]`); b?.focus({ preventScroll: true }); };
|
||||
paint();
|
||||
let res, body = null;
|
||||
try { res = await fetch(`/api/orgs/${enc(org)}/agent/terminate`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ instance: name, reason: STOP_REASON }), cache: 'no-store', redirect: 'manual' }); }
|
||||
@@ -1444,6 +1450,8 @@
|
||||
if (res.type === 'opaqueredirect' || res.status === 401) return finish('Not stopped: sign in again, then read the agents.');
|
||||
try { body = await res.json(); } catch { body = null; }
|
||||
if (!res.ok || body?.sent !== true && !body?.request) {
|
||||
const orgWhy = orgRefusal(body?.error, orgName);
|
||||
if (orgWhy) return finish(`Not stopped: ${orgWhy}`);
|
||||
if (body?.sent === false && typeof body.message === 'string' && body.message) return finish(`Not stopped: ${body.message}`);
|
||||
if (res.status === 403 && body?.error === 'read-only') return finish(`Not stopped: this Console only reads agents in ${orgName} (read-only).`);
|
||||
if (body?.sent === false && typeof body.error === 'string') return finish(`Not stopped: the bus refused (${body.error}).`);
|
||||
@@ -1451,14 +1459,14 @@
|
||||
}
|
||||
const request = typeof body?.request === 'string' ? body.request : '';
|
||||
if (!/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/.test(request)) return finish(STOP_UNKNOWN);
|
||||
stopState = { id, busy: true, text: `Stop asked: request ${request} is open.` }; paint();
|
||||
stops.set(id, { busy: true, text: `Stop asked: request ${request} is open.` }); paint();
|
||||
for (let tries = 0; tries < 90; tries++) {
|
||||
await new Promise(r => setTimeout(r, 2000));
|
||||
let r = null;
|
||||
try { const got = await fetch(`/api/orgs/${enc(org)}/agent/request?id=${enc(request)}`, { cache: 'no-store', redirect: 'manual' }); if (got.type === 'opaqueredirect') return finish('Sign in again, then read the agents.'); r = got.ok ? await got.json() : null; } catch { continue; }
|
||||
if (r?.request === request && r.state === 'closed') {
|
||||
if (r.reason) return finish(`Not stopped: ${r.reason}.`);
|
||||
stopState = { id, text: `Stopped ${name}.` }; repaint(); return;
|
||||
stops.set(id, { text: `Stopped ${name}.` }); repaint(); return;
|
||||
}
|
||||
}
|
||||
finish(`Request ${request} is still open. Read the agents again later.`);
|
||||
@@ -1472,8 +1480,8 @@
|
||||
if (d?.status === 403 && e === 'read-only') return `this Console only reads agents in ${orgName} (read-only).`;
|
||||
if (d?.status === 403 && (e === 'unavailable' || e === 'owner-only')) return `this Console is not the owner of ${orgName}, so it does not restart its agents.`;
|
||||
if (e === 'no-bus-host') return `the bus of ${orgName} is not running. Nothing changed.`;
|
||||
if (e === 'org-archived') return `${orgName} is archived, so its agents are not changed. Nothing changed.`;
|
||||
if (e === 'org-unavailable') return `${orgName} is not available to this Console. Nothing changed.`;
|
||||
const orgWhy = orgRefusal(e, orgName);
|
||||
if (orgWhy) return orgWhy;
|
||||
if (d?.result?.sent === false && typeof m === 'string' && m) return m;
|
||||
if (d?.result?.sent === false && typeof e === 'string') return `the bus refused (${e}).`;
|
||||
return STOP_UNKNOWN;
|
||||
@@ -1501,11 +1509,11 @@
|
||||
const thread = d.querySelector('[data-drawer-restart]')?.dataset.drawerThread === 'yes';
|
||||
if (restarts.get(id)?.busy) return;
|
||||
// The question is answered: from now on this agent's drawer draws its restart.
|
||||
if (stopState?.id === id) stopState = null;
|
||||
stops.delete(id);
|
||||
restarts.set(id, { busy: true, text: '' });
|
||||
// Paint this agent's drawer in place, when it is on screen and no question of its own holds it.
|
||||
const paint = () => {
|
||||
const now = document.querySelector(`[data-ws-drawer="${CSS.escape(id)}"]`), r = restarts.get(id); if (!now || !r || stopState?.id === id) return;
|
||||
const now = document.querySelector(`[data-ws-drawer="${CSS.escape(id)}"]`), r = restarts.get(id); if (!now || !r || stops.has(id)) return;
|
||||
const s = now.querySelector('[data-drawer-status]'); if (s) s.textContent = r.text;
|
||||
const rb = now.querySelector('[data-drawer-restart]'), sb = now.querySelector('[data-drawer-stop]');
|
||||
if (rb) { rb.disabled = r.busy; rb.firstChild.textContent = r.busy ? 'Restarting…' : 'Restart'; }
|
||||
@@ -1548,11 +1556,11 @@
|
||||
if (e.target.closest('[data-drawer-close]')) { closeDrawer(); return; }
|
||||
const id = d.dataset.wsDrawer;
|
||||
// SP19: nothing starts while this agent's Stop or Restart runs.
|
||||
const held = (stopState?.id === id && stopState.busy) || !!restarts.get(id)?.busy;
|
||||
const held = !!stops.get(id)?.busy || !!restarts.get(id)?.busy;
|
||||
const ask = e.target.closest('[data-drawer-stop]') ? 'stop' : e.target.closest('[data-drawer-restart]') ? 'restart' : null;
|
||||
if (ask && !held) {
|
||||
const b = d.querySelector(`[data-drawer-${ask}]`), name = d.dataset.drawerName, who = `${name} in ${d.dataset.drawerOrgName}`;
|
||||
stopState = { id, kind: ask, confirm: true, text: '' }; restarts.delete(id);
|
||||
stops.set(id, { kind: ask, confirm: true, text: '' }); restarts.delete(id);
|
||||
d.querySelector('.ws-drawer-confirm')?.remove();
|
||||
b.insertAdjacentHTML('afterend', ask === 'stop' ? stopConfirm(name, who) : restartConfirm(name, who, b.dataset.drawerThread === 'yes'));
|
||||
for (const x of d.querySelectorAll('[data-drawer-stop], [data-drawer-restart]')) x.disabled = true;
|
||||
@@ -1569,7 +1577,7 @@
|
||||
// Cancel the Stop or Restart question: nothing sent, both buttons enabled again, focus on the one that asked.
|
||||
function cancelStop(d) {
|
||||
const kind = d.querySelector('[data-drawer-restart-confirm]') ? 'restart' : 'stop';
|
||||
stopState = null;
|
||||
stops.delete(d.dataset.wsDrawer);
|
||||
d.querySelector('.ws-drawer-confirm')?.remove();
|
||||
for (const x of d.querySelectorAll('[data-drawer-stop], [data-drawer-restart]')) x.disabled = false;
|
||||
d.querySelector(`[data-drawer-${kind}]`)?.focus({ preventScroll: true });
|
||||
|
||||
@@ -66,7 +66,7 @@ const files = new Map([
|
||||
['/', ['index.html', 'text/html; charset=utf-8']],
|
||||
['/proxy.js', ['proxy.js', 'text/javascript; charset=utf-8']],
|
||||
['/project-remote.js', ['project-remote.js', 'text/javascript; charset=utf-8']],
|
||||
...['app.js', 'brand.js', 'bus.js', 'markdown-preview.js', 'chat.js', 'project-archive.js', 'processing.js', 'agents.js', 'federation.js', 'requests.js', 'org-create.js', 'console-restart.js', 'onboarding.js', 'workspace-data.js', 'workspace.js', 'tasks-board.js', 'assistant.js'].map(f => ['/' + f, [f, 'text/javascript; charset=utf-8']]),
|
||||
...['app.js', 'brand.js', 'bus.js', 'markdown-preview.js', 'chat.js', 'project-archive.js', 'processing.js', 'agents.js', 'federation.js', 'requests.js', 'org-create.js', 'console-restart.js', 'onboarding.js', 'workspace-data.js', 'workspace.js', 'tasks-board.js', 'assistant.js', 'triage-settings.js'].map(f => ['/' + f, [f, 'text/javascript; charset=utf-8']]),
|
||||
['/files.js', ['files.js', 'text/javascript; charset=utf-8']],
|
||||
['/projects.js', ['projects.js', 'text/javascript; charset=utf-8']],
|
||||
['/projects.css', ['projects.css', 'text/css; charset=utf-8']],
|
||||
|
||||
@@ -32,6 +32,17 @@
|
||||
// directory and HOME (HOME not counted against the stack's own code), here and again in
|
||||
// hostedBinds. A refused answer reaches no spawn, answers triage-route-invalid, and writes
|
||||
// ROUTE_REFUSED to the transcript; a refusal at the binds also removes what the start wrote.
|
||||
//
|
||||
// #1873 triage-cap: the relay counts each turn. usage.begin takes a baseline of the harness's
|
||||
// transcript with the transcript usage-source the bus uses; usage.end takes the delta (the bus's
|
||||
// own rules, triage-usage.mjs) and appends one line to <dataRoot>/triage/usage.jsonl. When a cap is
|
||||
// stored (triage-route-store.mjs), the daily-quota plugin decides over the lines in that file: before
|
||||
// a start (admit), before the runner takes queued messages (message.receive) and before each
|
||||
// message's turn (the runner's state read outside a turn). Over the cap, or with the store or the
|
||||
// file unreadable, the turn in flight finishes, every message still queued gets a host line and
|
||||
// leaves the queue, and the runner is told its session is gone. Nothing is held in memory: each
|
||||
// check reads the file again, so a restart counts what the file holds. With no cap stored, nothing
|
||||
// is refused for usage and the lines are still written.
|
||||
|
||||
import { spawn } from "node:child_process";
|
||||
import { randomBytes, timingSafeEqual } from "node:crypto";
|
||||
@@ -48,6 +59,9 @@ import { builtinTools } from "../../harness/src/bundle.mjs";
|
||||
import { FOUNDER_ENV } from "../../harness/src/runner.mjs";
|
||||
import { extractCards, plainText } from "./triage-cards.mjs";
|
||||
import { SNAPSHOT_FILE, writeSnapshot } from "./triage-snapshot.mjs";
|
||||
import { readRoute } from "./triage-route-store.mjs";
|
||||
import { ID as USAGE_ID, appendUsage, capState, delta, nextReset, readUsage, snapshot, usagePath } from "./triage-usage.mjs";
|
||||
import { readTurn } from "../../plugins/usage-source/transcript/index.mjs";
|
||||
|
||||
export const RUNNER = RUNNER_FILES[0];
|
||||
export const RELAY_REFUSED = "triage-refused";
|
||||
@@ -58,6 +72,8 @@ export const UNANSWERED = "Triage stopped before it answered this message. Send
|
||||
export const NOT_STARTED = "Triage did not start, so this message was not delivered.";
|
||||
export const NOT_TAKEN = "Triage stopped before it read this message. Send it again.";
|
||||
export const ROUTE_REFUSED = "Triage did not start: the model route's answer did not pass the host's checks, so this message was not delivered.";
|
||||
export const overCapNotTaken = (until) => `Triage reached its daily token cap before it took this message. Send it again after ${until}.`;
|
||||
export const USAGE_NOT_READ = "Triage could not read its token usage or its cap, so it did not take this message.";
|
||||
// An over-cap answer is held for at most this long; a later time is not taken.
|
||||
export const CAP_HOLD_MAX_MS = 48 * 3600 * 1000;
|
||||
const ISO_TIME = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d{1,3})?Z$/;
|
||||
@@ -179,7 +195,10 @@ function sameCap(a, b) {
|
||||
// The relay server. `onSend({ body, in_reply_to, result })` keeps a human-bound send, `result` true
|
||||
// only for class RESULT, the class the runner gives its answer to a turn; `take()` returns the
|
||||
// queued owner messages; `read(id)` drops one that was answered.
|
||||
export function createRelay({ socket, cap, take, read, onSend, log = () => {} }) {
|
||||
// #1873 triage-cap: `usage(phase, turn)` (a promise) counts the turn; the relay answers usage.begin
|
||||
// and usage.end only once it settles. `stopped(verb)` is asked before state and message.receive;
|
||||
// true answers "not-holder", which ends the runner's loop after the turn in flight.
|
||||
export function createRelay({ socket, cap, take, read, onSend, usage = null, stopped = () => false, log = () => {} }) {
|
||||
const answer = (conn, value) => { try { conn.end(JSON.stringify(value) + "\n"); } catch {} };
|
||||
const refuse = (conn) => answer(conn, { ok: false, error: RELAY_REFUSED, reason: "the Triage host answers only a message to the person" });
|
||||
const server = createServer((conn) => {
|
||||
@@ -200,6 +219,13 @@ export function createRelay({ socket, cap, take, read, onSend, log = () => {} })
|
||||
const args = req.args && typeof req.args === "object" && !Array.isArray(req.args) ? req.args : {};
|
||||
const verb = req.verb;
|
||||
try {
|
||||
if (usage && (verb === "usage.begin" || verb === "usage.end")) {
|
||||
return void Promise.resolve()
|
||||
.then(() => usage(verb === "usage.begin" ? "begin" : "end", args.turn))
|
||||
.catch((e) => log(`triage relay: ${verb} not counted: ${e?.code ?? "error"}`))
|
||||
.then(() => answer(conn, { ok: true, result: {} }));
|
||||
}
|
||||
if ((verb === "state" || verb === "message.receive") && stopped(verb)) return answer(conn, { ok: false, error: "not-holder" });
|
||||
if (Object.hasOwn(LIFECYCLE, verb)) return answer(conn, { ok: true, result: LIFECYCLE[verb]() });
|
||||
if (verb === "message.receive") return answer(conn, { ok: true, result: take() });
|
||||
if (verb === "message.read") { read(args.id); return answer(conn, { ok: true, result: {} }); }
|
||||
@@ -235,11 +261,13 @@ export function createRelay({ socket, cap, take, read, onSend, log = () => {} })
|
||||
// harnessModule async (name) -> the harness module (envNames, installRoots) or null; the
|
||||
// repository's packages/plugins/harness/<name> by default
|
||||
// home HOME, which no mount may reach outside the stack's own code; the Console user's by default
|
||||
export function createTriageHost({ dataRoot, route = null, readSnapshot = async () => null, env = process.env, node = process.execPath, bwrap = BWRAP, now = () => new Date(), devOwner = false, log = () => {}, roleDir = join(REPO, "roles"), readOnly = [], pollInterval = null, proxyPort = null, tokenPattern = null, harnessModule = repoHarnessModule, home = homedir() }) {
|
||||
// usageSource { readTurn({ harness, root }) } the transcript usage-source; the repository's own by default
|
||||
export function createTriageHost({ dataRoot, route = null, readSnapshot = async () => null, env = process.env, node = process.execPath, bwrap = BWRAP, now = () => new Date(), devOwner = false, log = () => {}, roleDir = join(REPO, "roles"), readOnly = [], pollInterval = null, proxyPort = null, tokenPattern = null, harnessModule = repoHarnessModule, home = homedir(), usageSource = { readTurn } }) {
|
||||
if (typeof dataRoot !== "string" || !dataRoot.startsWith("/")) throw new TriageError("triage-no-data-root", "no data root");
|
||||
if (proxyPort !== null && !(Number.isInteger(proxyPort) && proxyPort > 0 && proxyPort <= 65535)) throw new TypeError("proxyPort must be a port number or null");
|
||||
if (tokenPattern !== null && !(tokenPattern instanceof RegExp)) throw new TypeError("tokenPattern must be a RegExp or null");
|
||||
if (typeof harnessModule !== "function") throw new TypeError("harnessModule must be a function");
|
||||
if (typeof usageSource?.readTurn !== "function") throw new TypeError("usageSource must have readTurn");
|
||||
const tokenShape = tokenPattern && new RegExp(`^(?:${tokenPattern.source})$`, tokenPattern.flags.replace(/[gmy]/g, ""));
|
||||
// What no mount source may lie in or around, besides the run directory and the fixed list. HOME
|
||||
// does not refuse the stack's own code when the stack runs from under it (a copy or a worktree).
|
||||
@@ -257,6 +285,10 @@ export function createTriageHost({ dataRoot, route = null, readSnapshot = async
|
||||
let cap = null;
|
||||
let overCapUntil = null; // the route's over-cap answer, held until its time passes
|
||||
const queue = [];
|
||||
let usage = null; // this run's counting: { run, root, harness, baselines, ended, inTurn }
|
||||
let usageChain = Promise.resolve(); // usage calls and exit records, one at a time
|
||||
let usageLost = false; // a turn's line could not be written: the count is short, so a stored cap refuses
|
||||
const badSeen = new Set();
|
||||
|
||||
const ensureDirs = () => {
|
||||
mkdirSync(dataRoot, { recursive: true, mode: 0o700 });
|
||||
@@ -320,14 +352,16 @@ export function createTriageHost({ dataRoot, route = null, readSnapshot = async
|
||||
writeFileSync(bundle.prompt, hostPrompt(contract, { snapshotPath: join(workspace, "health", SNAPSHOT_FILE), steps }), { mode: 0o600, flag: "wx" });
|
||||
writeFileSync(bundle.policy, JSON.stringify(policy, null, 2) + "\n", { mode: 0o600, flag: "wx" });
|
||||
writeFileSync(bundle.tools, JSON.stringify(tools, null, 2) + "\n", { mode: 0o600, flag: "wx" });
|
||||
const run = `triage-${now().toISOString().replace(/[^0-9]/g, "").slice(0, 14)}`;
|
||||
const session = {
|
||||
bundle, adapter: admitted.adapter, harness: admitted.harness, harnessEnv: admitted.harnessEnv, turnEnd: admitted.turnEnd,
|
||||
workspace, sessionDir: join(a, "session"), toolSocket: join(a, "tools.sock"), turnMarker: join(a, "turn.done"),
|
||||
brokerSocket: join(a, "relay.sock"), business: "system", instance: "triage", run: `triage-${now().toISOString().replace(/[^0-9]/g, "").slice(0, 14)}`,
|
||||
brokerSocket: join(a, "relay.sock"), business: "system", instance: "triage", run,
|
||||
model: admitted.model, provider: admitted.provider ?? "",
|
||||
...(Number.isInteger(pollInterval) && pollInterval >= 100 ? { pollInterval } : {}),
|
||||
};
|
||||
writeFileSync(join(a, "session.json"), JSON.stringify(session, null, 2) + "\n", { mode: 0o600, flag: "wx" });
|
||||
return run;
|
||||
};
|
||||
|
||||
// The route's answer, checked. It returns a copy taken once, so a getter or a later change to
|
||||
@@ -363,13 +397,114 @@ export function createTriageHost({ dataRoot, route = null, readSnapshot = async
|
||||
if (e?.code === "hosted-binds-refused") refuse(e.fence);
|
||||
throw e;
|
||||
}
|
||||
return { admitted: a, installRoots };
|
||||
// Where the harness keeps its session files, as the bus binds it (the module's transcript()).
|
||||
// Only a path inside the run directory is read; anything else reads as no transcript.
|
||||
let transcriptRoot = null;
|
||||
try {
|
||||
const r = typeof mod.transcript === "function" ? mod.transcript({ runDir: paths.agent }) : null;
|
||||
if (typeof r === "string" && r.startsWith(paths.agent + "/") && !r.split("/").includes("..")) transcriptRoot = r;
|
||||
} catch { transcriptRoot = null; }
|
||||
return { admitted: a, installRoots, transcriptRoot };
|
||||
};
|
||||
|
||||
// #1873 triage-cap. A baseline or a reading of the transcript, with the bus's codes.
|
||||
const sample = async (u) => {
|
||||
if (u.root === null) return { status: "unknown", code: "no-transcript" };
|
||||
try { return snapshot(await usageSource.readTurn({ harness: u.harness, root: u.root })); }
|
||||
catch { return { status: "unknown", code: "source-failed" }; }
|
||||
};
|
||||
// A turn id the runner names, or "unnamed". The first begin for a turn wins, as on the bus.
|
||||
const turnKey = (turn) => (typeof turn === "string" && USAGE_ID.test(turn) ? turn : null);
|
||||
const usageBegin = (u, turn) => {
|
||||
const named = turnKey(turn);
|
||||
if (!named) log("triage host: usage.begin without a usable turn id");
|
||||
const k = named ?? "unnamed";
|
||||
u.inTurn = true;
|
||||
if (!u.baselines.has(k)) u.baselines.set(k, sample(u));
|
||||
};
|
||||
// One line per turn: the delta since its baseline, known or unknown.
|
||||
const finishTurn = async (u, k) => {
|
||||
const base = u.baselines.has(k) ? await u.baselines.get(k) : null;
|
||||
u.baselines.delete(k);
|
||||
const result = delta(base, await sample(u));
|
||||
const known = result.status === "known";
|
||||
if (!known) log(`triage host: turn usage unknown: ${result.code}`);
|
||||
const line = { at: now().toISOString(), run: u.run, turn: k, status: result.status,
|
||||
input: known ? result.input : null, output: known ? result.output : null, cacheRead: known ? result.cacheRead : null, cacheWrite: known ? result.cacheWrite : null };
|
||||
try {
|
||||
appendUsage(usagePath(dataRoot), line);
|
||||
} catch (e) {
|
||||
usageLost = true;
|
||||
log(`triage host: usage line not written: ${e?.code ?? "error"}`);
|
||||
}
|
||||
};
|
||||
const usageEnd = async (u, turn) => {
|
||||
u.inTurn = false;
|
||||
const named = turnKey(turn);
|
||||
if (!named) log("triage host: usage.end without a usable turn id");
|
||||
if (named && u.ended.has(named)) { log("triage host: usage.end repeated for a turn; not counted twice"); return; }
|
||||
if (named) u.ended.add(named);
|
||||
await finishTurn(u, named ?? "unnamed");
|
||||
};
|
||||
const chainUsage = (fn) => (usageChain = usageChain.then(fn).catch((e) => log(`triage host: usage failed: ${e?.code ?? "error"}`)));
|
||||
|
||||
// The stored cap's state, read from the store and the usage file each time.
|
||||
// { state: "not set" } no cap stored
|
||||
// { state: "route-unreadable" } | { state: "usage-unreadable", cap, resetsAt? } refused
|
||||
// { state: "under" | "over", cap, usageToday, resetsAt }
|
||||
const capNow = () => {
|
||||
let stored;
|
||||
try { stored = readRoute(dataRoot); } catch { return { state: "route-unreadable" }; }
|
||||
if (!stored) return { state: "not set" };
|
||||
const at = now().getTime();
|
||||
let resetsAt;
|
||||
try { resetsAt = new Date(nextReset(stored.cap, at)).toISOString(); } catch { return { state: "usage-unreadable", cap: stored.cap }; }
|
||||
if (usageLost) return { state: "usage-unreadable", cap: stored.cap, resetsAt };
|
||||
let read;
|
||||
try { read = readUsage(usagePath(dataRoot)); } catch { return { state: "usage-unreadable", cap: stored.cap, resetsAt }; }
|
||||
for (const b of read.bad) {
|
||||
const key = `${read.ino}:${b.line}`;
|
||||
if (badSeen.has(key)) continue;
|
||||
badSeen.add(key);
|
||||
log(`triage host: usage.jsonl line ${b.line} is not a usage line; it counts as an unknown turn`);
|
||||
}
|
||||
try {
|
||||
const s = capState({ cap: stored.cap, records: read.records, now: at });
|
||||
return { state: s.allow ? "under" : "over", cap: stored.cap, usageToday: s.usageToday, resetsAt: s.resetsAt };
|
||||
} catch { return { state: "usage-unreadable", cap: stored.cap, resetsAt }; }
|
||||
};
|
||||
const unsafeRoot = () => {
|
||||
let st;
|
||||
try { st = lstatSync(paths.root); } catch { return false; }
|
||||
return st.isSymbolicLink() || !st.isDirectory() || (typeof process.getuid === "function" && st.uid !== process.getuid());
|
||||
};
|
||||
const refusedByCap = (c) => c.state === "over" || c.state === "usage-unreadable" || c.state === "route-unreadable";
|
||||
// Before the runner takes messages or starts a message's turn: over the cap, every message still
|
||||
// queued gets a host line and leaves the queue, and the runner's loop ends. That holds with nothing
|
||||
// queued too: an idle session over the cap ends at its next receive, and the next message is
|
||||
// refused by admit. A state read inside a turn (a continuation) is not stopped: the turn finishes.
|
||||
const capStop = (verb) => {
|
||||
const u = usage;
|
||||
if (verb === "state" && u?.inTurn) return false;
|
||||
const c = capNow();
|
||||
if (!refusedByCap(c)) return false;
|
||||
log(`triage host: stopping the session at ${verb}: ${c.state === "over" ? "over the daily token cap" : "usage or cap not read"}`);
|
||||
// A message whose turn began and was never answered says so; one never begun says why not.
|
||||
const begun = (id) => Boolean(u && (u.baselines.has(id) || u.ended.has(id)));
|
||||
for (const m of queue.splice(0)) {
|
||||
const text = begun(m.id) ? UNANSWERED : c.state === "over" ? overCapNotTaken(c.resetsAt) : USAGE_NOT_READ;
|
||||
try { append({ at: now().toISOString(), from: "host", in_reply_to: m.id, text }); }
|
||||
catch { log("triage host: transcript write failed"); }
|
||||
}
|
||||
return true;
|
||||
};
|
||||
|
||||
const start = async (answer) => {
|
||||
const { admitted, installRoots } = await checkAdmitted(answer);
|
||||
const { admitted, installRoots, transcriptRoot } = await checkAdmitted(answer);
|
||||
ensureDirs();
|
||||
writeBundle(admitted);
|
||||
const run = writeBundle(admitted);
|
||||
const u = { run, root: transcriptRoot, harness: admitted.harness, baselines: new Map(), ended: new Set(), inTurn: false };
|
||||
usage = u;
|
||||
cap = randomBytes(32).toString("hex");
|
||||
relay = await createRelay({
|
||||
socket: join(paths.agent, "relay.sock"), cap,
|
||||
@@ -393,6 +528,8 @@ export function createTriageHost({ dataRoot, route = null, readSnapshot = async
|
||||
const i = in_reply_to && result === true ? queue.findIndex((m) => m.id === in_reply_to) : -1;
|
||||
if (i >= 0) queue.splice(i, 1);
|
||||
},
|
||||
usage: (phase, turn) => chainUsage(() => (phase === "begin" ? usageBegin(u, turn) : usageEnd(u, turn))),
|
||||
stopped: capStop,
|
||||
log,
|
||||
});
|
||||
let list;
|
||||
@@ -427,6 +564,9 @@ export function createTriageHost({ dataRoot, route = null, readSnapshot = async
|
||||
if (child !== mine) return;
|
||||
lastExit = { code: Number.isInteger(code) ? code : null, signal: signal ?? null, at: now().toISOString() };
|
||||
child = null;
|
||||
// A turn the session began and never ended (killed, crashed) still spent tokens: it gets its line.
|
||||
// The open turns are read when this runs, after any usage.end already in the chain.
|
||||
chainUsage(async () => { for (const k of [...u.baselines.keys()]) await finishTurn(u, k); });
|
||||
// Every message still queued leaves it, so none holds a slot with no session to answer it, and
|
||||
// the transcript says which: one the session took and never answered, or one it never read.
|
||||
for (const m of queue.splice(0)) {
|
||||
@@ -456,6 +596,15 @@ export function createTriageHost({ dataRoot, route = null, readSnapshot = async
|
||||
if (!route || typeof route.admit !== "function") return { ok: false, code: "triage-no-route" };
|
||||
const held = capUntil();
|
||||
if (held !== null) return { ok: false, code: "triage-over-cap", until: held };
|
||||
// A turn still being counted (a session that just ended mid-turn, or a usage.end in flight) is in
|
||||
// the file before the count is read.
|
||||
await usageChain;
|
||||
const c = capNow();
|
||||
// The store sits in the Triage directory: one that is a link or not the Console user's is the
|
||||
// unsafe directory start would refuse, not an unreadable route.
|
||||
if (c.state === "route-unreadable") return { ok: false, code: unsafeRoot() ? "triage-dir-unsafe" : "triage-route-unreadable" };
|
||||
if (c.state === "usage-unreadable") return { ok: false, code: "triage-usage-unreadable" };
|
||||
if (c.state === "over") return { ok: false, code: "triage-over-cap", until: c.resetsAt };
|
||||
let a;
|
||||
try { a = await route.admit(); } catch { return { ok: false, code: "triage-route-unreadable" }; }
|
||||
if (a?.ok === true) return a;
|
||||
@@ -474,9 +623,15 @@ export function createTriageHost({ dataRoot, route = null, readSnapshot = async
|
||||
get pid() { return child?.pid ?? null; },
|
||||
routeSet: () => Boolean(route && typeof route.admit === "function"),
|
||||
async state() {
|
||||
const c = capNow();
|
||||
const unset = c.state === "not set";
|
||||
return {
|
||||
route: this.routeSet() ? "set" : "not set",
|
||||
overCapUntil: capUntil(),
|
||||
overCapUntil: c.state === "over" ? c.resetsAt : capUntil(),
|
||||
// #1873 triage-cap: today's count against the stored cap. Names and numbers only.
|
||||
usageToday: unset ? "not set" : c.usageToday ?? "not read",
|
||||
cap: unset ? "not set" : c.cap ? { limit: c.cap.limit, reset: c.cap.reset, timeZone: c.cap.timeZone } : "not read",
|
||||
resetsAt: unset ? "not set" : c.resetsAt ?? "not read",
|
||||
running: child !== null,
|
||||
lastExit,
|
||||
snapshot: last,
|
||||
@@ -493,7 +648,7 @@ export function createTriageHost({ dataRoot, route = null, readSnapshot = async
|
||||
if (text.length > MESSAGE_MAX) throw new TriageError("triage-message-too-long", `a message holds at most ${MESSAGE_MAX} characters`);
|
||||
if (queue.length >= QUEUE_MAX) throw new TriageError("triage-busy", "Triage has too many messages waiting");
|
||||
const admitted = await admit();
|
||||
if (!admitted.ok) throw Object.assign(new TriageError(admitted.code, admitted.code === "triage-over-cap" ? "Triage is over its daily cap" : admitted.code === "triage-no-route" ? "no model route is set for Triage" : "the model route refused"), admitted.until ? { until: admitted.until } : {});
|
||||
if (!admitted.ok) throw Object.assign(new TriageError(admitted.code, admitted.code === "triage-over-cap" ? "Triage is over its daily cap" : admitted.code === "triage-no-route" ? "no model route is set for Triage" : admitted.code === "triage-usage-unreadable" ? "Triage's token usage could not be read" : admitted.code === "triage-route-unreadable" ? "the model route could not be read" : admitted.code === "triage-dir-unsafe" ? "a Triage directory is not a real directory" : "the model route refused"), admitted.until ? { until: admitted.until } : {});
|
||||
await refresh();
|
||||
const at = now().toISOString();
|
||||
const id = `owner-${randomBytes(6).toString("hex")}`;
|
||||
@@ -525,6 +680,7 @@ export function createTriageHost({ dataRoot, route = null, readSnapshot = async
|
||||
}
|
||||
relay?.close();
|
||||
relay = null;
|
||||
await usageChain;
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -31,6 +31,7 @@ const MESSAGES = Object.freeze({
|
||||
"triage-route-refused": "the model route refused to start Triage",
|
||||
"triage-route-invalid": "the model route's answer is not usable",
|
||||
"triage-route-unreadable": "the model route could not be read",
|
||||
"triage-usage-unreadable": "Triage's token usage could not be read, so it takes no message",
|
||||
"triage-over-cap": "Triage is over its daily token cap and starts again after the time shown",
|
||||
"triage-busy": "Triage has too many messages waiting",
|
||||
"triage-dir-unsafe": "a Triage directory is not safe to use",
|
||||
@@ -69,6 +70,8 @@ export async function triageView(triage, { send, now = Date.now } = {}) {
|
||||
...(s.route === "set" ? {} : { routeMessage: NO_ROUTE_MESSAGE }),
|
||||
overCapUntil: s.overCapUntil ?? null,
|
||||
...(s.overCapUntil ? { capMessage: capMessage(s.overCapUntil) } : {}),
|
||||
// #1873 triage-cap: today's count, the stored cap and the next reset, or "not set" / "not read".
|
||||
...(Object.hasOwn(s, "usageToday") ? { usageToday: s.usageToday, cap: s.cap, resetsAt: s.resetsAt } : {}),
|
||||
running: s.running,
|
||||
lastExit: s.lastExit,
|
||||
snapshot: s.snapshot,
|
||||
|
||||
@@ -0,0 +1,263 @@
|
||||
// #1873 triage-settings: the owner's read and write of the Triage model route, for Settings > Triage.
|
||||
// GET and POST /api/triage/route register through serve.mjs's route table, so the server's session
|
||||
// and Origin checks run first; this module then takes the proxy-admin gate (the instance owner's own
|
||||
// session on the public name, never a tester or a granted user) and the same-origin checks again.
|
||||
// A write goes through the route store (triage-route-store.mjs), so the lock, the revision compare
|
||||
// and the receipt are the store's. Before it, the model and harness are checked the way an
|
||||
// organization's route change is (checkRoutes in packages/organization/src/launch-settings.mjs):
|
||||
// offered by the catalogue, available, and compatible with the harness.
|
||||
//
|
||||
// The catalogue lives in each organization's bus (the trusted bus boot builds it; nothing outside
|
||||
// a bus holds the proxy client), so the Console reads it through each active organization's
|
||||
// `routes` verb, as the proxy setup page does, and merges the answers (mergeCatalogues).
|
||||
//
|
||||
// Saving a route starts nothing: the Triage host is not wired to the store in this slice.
|
||||
// No key, path, env value or exception text reaches a response or a log line.
|
||||
import { readdirSync, lstatSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { checkRoutes } from '../../organization/src/launch-settings.mjs';
|
||||
import { isDailyModelQuota } from '../../plugins/src/model-quota.mjs';
|
||||
import { readRoute, writeRoute } from './triage-route-store.mjs';
|
||||
|
||||
export const TRIAGE_ROUTE_PATH = '/api/triage/route';
|
||||
export const BODY_MAX = 16384;
|
||||
export const POST_KEYS = Object.freeze(['model', 'harness', 'cap', 'timeZone', 'revision']);
|
||||
export const NOT_SET_MESSAGE = 'No Triage route is set.';
|
||||
export const NOT_STARTED_NOTE = 'Saving a route does not start Triage. The key and the wiring are separate.';
|
||||
|
||||
const HARNESS_ROOT = fileURLToPath(new URL('../../plugins/harness/', import.meta.url));
|
||||
// The same shapes the store holds (triage-route-store.mjs).
|
||||
const MODEL_KEY = /^[a-z0-9][a-z0-9._-]{0,63}$/;
|
||||
const HARNESS_NAME = /^[a-z][a-z0-9-]{0,31}$/;
|
||||
const ACTOR = /^[a-z][a-z0-9-]{0,31}$/;
|
||||
// The store's zone rule (its knownZone): isDailyModelQuota's zone shape, then Intl.
|
||||
const knownZone = zone => typeof zone === 'string' && zone !== ''
|
||||
&& isDailyModelQuota('000000000000', { limit: 0, since: '2000-01-01T00:00:00.000Z', reset: '00:00', timeZone: zone });
|
||||
// The zone as Intl names it, so "america/chicago" is kept as "America/Chicago". A known zone that
|
||||
// Intl renames to something the store would not take is kept as typed.
|
||||
export function canonicalZone(zone) {
|
||||
try {
|
||||
const named = new Intl.DateTimeFormat('en-US', { timeZone: zone }).resolvedOptions().timeZone;
|
||||
return knownZone(named) ? named : zone;
|
||||
} catch { return zone; }
|
||||
}
|
||||
|
||||
const STATUS = Object.freeze({
|
||||
'human-required': 403,
|
||||
'read-only': 403,
|
||||
'cross-origin': 403,
|
||||
'cross-site': 403,
|
||||
'method-not-allowed': 405,
|
||||
'invalid-request': 400,
|
||||
'triage-cap-invalid': 400,
|
||||
'triage-zone-required': 400,
|
||||
'triage-zone-unknown': 400,
|
||||
'unknown-model': 400,
|
||||
'model-harness-incompatible': 400,
|
||||
'harness-off': 400,
|
||||
'triage-route-invalid': 400,
|
||||
'triage-route-stale': 409,
|
||||
'triage-route-busy': 409,
|
||||
'model-provider-unavailable': 503,
|
||||
'triage-route-unreadable': 503,
|
||||
'triage-route-unrecorded': 500,
|
||||
'outcome-unknown': 503,
|
||||
});
|
||||
// Fixed words for each code. The store's own messages name lock files, so none is forwarded.
|
||||
const MESSAGES = Object.freeze({
|
||||
'human-required': 'Only the instance owner can read or set the Triage route.',
|
||||
'read-only': 'This Console was started without writes, so the route cannot be saved.',
|
||||
'cross-origin': 'The request did not come from this Console page.',
|
||||
'cross-site': 'The request did not come from this Console page.',
|
||||
'method-not-allowed': 'This address takes GET and POST only.',
|
||||
'invalid-request': 'The request is not a Triage route. It takes model, harness, cap, timeZone and revision.',
|
||||
'triage-cap-invalid': 'The cap must be a whole number of tokens, 1 or more.',
|
||||
'triage-zone-required': 'A time zone is required. There is no default.',
|
||||
'triage-zone-unknown': 'That time zone is not known. Use a name like Europe/Berlin or America/Chicago.',
|
||||
'unknown-model': 'The model catalogue does not offer that model.',
|
||||
'model-harness-incompatible': 'That harness cannot run that model.',
|
||||
'harness-off': 'No harness module of that name is installed.',
|
||||
'triage-route-invalid': 'The route was refused as invalid.',
|
||||
'triage-route-stale': 'The route changed since this page read it. Reload the route before saving again.',
|
||||
'triage-route-busy': 'Another save of the route is in progress. Try again.',
|
||||
'model-provider-unavailable': 'The model catalogue cannot confirm that model now, so the route was not saved.',
|
||||
'triage-route-unreadable': 'The stored route could not be read.',
|
||||
'triage-route-unrecorded': 'The route was saved, but its receipt line was not written.',
|
||||
'outcome-unknown': 'The route service did not answer.',
|
||||
});
|
||||
|
||||
const json = (res, status, value) => {
|
||||
res.writeHead(status, { 'content-type': 'application/json', 'cache-control': 'no-store' });
|
||||
res.end(JSON.stringify(value));
|
||||
};
|
||||
const one = (req, name) => (req.headersDistinct[name]?.length === 1 ? req.headersDistinct[name][0] : null);
|
||||
const plain = v => v !== null && typeof v === 'object' && !Array.isArray(v);
|
||||
const identity = m => JSON.stringify([m.provider, m.model, m.module, m.upstream]);
|
||||
|
||||
// The installed harness modules: a directory under packages/plugins/harness with an index.mjs.
|
||||
export function installedHarnesses(root = HARNESS_ROOT) {
|
||||
try {
|
||||
return readdirSync(root).filter(name => HARNESS_NAME.test(name) && (() => {
|
||||
try { return lstatSync(join(root, name, 'index.mjs')).isFile(); } catch { return false; }
|
||||
})()).sort();
|
||||
} catch { return []; }
|
||||
}
|
||||
|
||||
// Each active organization's catalogue, merged into the one the Triage route is checked against.
|
||||
// Only ready answers count. A key is offered when a ready catalogue lists it, and dropped when two
|
||||
// catalogues disagree on what it is (provider, model, module, upstream). It is available only when
|
||||
// every catalogue listing it says so, and its harnesses are those every listing names that are
|
||||
// installed here. Status is ready when one catalogue is; otherwise the first status seen.
|
||||
export function mergeCatalogues(list, harnesses) {
|
||||
const ready = (Array.isArray(list) ? list : []).filter(c => c?.status === 'ready' && Array.isArray(c.models));
|
||||
if (!ready.length) {
|
||||
const first = (Array.isArray(list) ? list : []).find(c => typeof c?.status === 'string')?.status;
|
||||
return { status: first && first !== 'ready' ? first : 'unavailable', models: [] };
|
||||
}
|
||||
const merged = new Map(), dropped = new Set();
|
||||
for (const catalogue of ready) {
|
||||
for (const m of catalogue.models) {
|
||||
if (!plain(m) || typeof m.key !== 'string' || !MODEL_KEY.test(m.key) || dropped.has(m.key)) continue;
|
||||
const names = Array.isArray(m.harnesses) ? m.harnesses.filter(h => typeof h === 'string') : [];
|
||||
const held = merged.get(m.key);
|
||||
if (!held) {
|
||||
merged.set(m.key, { key: m.key, label: typeof m.label === 'string' && m.label ? m.label : m.key, provider: m.provider, model: m.model, module: m.module, upstream: m.upstream, available: m.available === true, harnesses: names });
|
||||
continue;
|
||||
}
|
||||
if (identity(held) !== identity(m)) { merged.delete(m.key); dropped.add(m.key); continue; }
|
||||
held.available = held.available && m.available === true;
|
||||
held.harnesses = held.harnesses.filter(h => names.includes(h));
|
||||
}
|
||||
}
|
||||
const installed = new Set(harnesses);
|
||||
const models = [...merged.values()].map(m => ({ ...m, harnesses: m.harnesses.filter(h => installed.has(h)) })).sort((a, b) => a.key.localeCompare(b.key));
|
||||
return { status: 'ready', models };
|
||||
}
|
||||
|
||||
// The organization route check, called as it is for an organization's change: one role whose
|
||||
// vars name the harness and model, launch.models holding the catalogue's own entry for that key
|
||||
// (none when it is not offered), plugins.harness the installed harness modules, and
|
||||
// plugins.model-provider the entry's module. It throws unknown-model, model-provider-unavailable,
|
||||
// model-harness-incompatible or harness-off.
|
||||
export function checkTriageRoute({ model, harness, catalogue, harnesses }) {
|
||||
if (catalogue?.status !== 'ready') {
|
||||
const error = new Error('model-provider-unavailable'); error.code = 'model-provider-unavailable'; throw error;
|
||||
}
|
||||
const offered = catalogue.models.find(m => m.key === model) ?? null;
|
||||
const entry = offered ? { provider: offered.provider, model: offered.model, module: offered.module, upstream: offered.upstream } : null;
|
||||
checkRoutes({
|
||||
proposed: {
|
||||
roles: { triage: { vars: { harness, model } } },
|
||||
launch: { models: entry ? { [model]: entry } : {} },
|
||||
plugins: { harness: [...harnesses], 'model-provider': entry ? [entry.module] : [] },
|
||||
},
|
||||
old: null,
|
||||
levels: () => null,
|
||||
catalogue,
|
||||
});
|
||||
}
|
||||
|
||||
// What the signed-in owner is recorded as: their account id, or the dev local owner's subject.
|
||||
export function ownerActor(session) {
|
||||
if (typeof session?.account === 'string' && ACTOR.test(session.account)) return session.account;
|
||||
if (session?.provider === 'local-dev' && typeof session.sub === 'string' && ACTOR.test(session.sub)) return session.sub;
|
||||
return null;
|
||||
}
|
||||
|
||||
const view = doc => doc ? {
|
||||
model: doc.model, harness: doc.harness, cap: doc.cap.limit, reset: doc.cap.reset, timeZone: doc.cap.timeZone,
|
||||
revision: doc.revision, updatedAt: doc.updatedAt,
|
||||
} : null;
|
||||
// The panel's choices: no provider, upstream or module, only what a select shows.
|
||||
const choices = catalogue => ({
|
||||
status: catalogue.status,
|
||||
models: catalogue.models.map(m => ({ key: m.key, label: m.label, available: m.available, harnesses: [...m.harnesses] })),
|
||||
});
|
||||
|
||||
async function readBody(req) {
|
||||
if ((req.headers['content-type'] ?? '').split(';')[0].trim().toLowerCase() !== 'application/json') throw Error();
|
||||
const chunks = [];
|
||||
let size = 0;
|
||||
for await (const chunk of req) {
|
||||
size += chunk.length;
|
||||
if (size > BODY_MAX) throw Error();
|
||||
chunks.push(chunk);
|
||||
}
|
||||
const value = JSON.parse(Buffer.concat(chunks));
|
||||
if (!plain(value) || Object.keys(value).some(k => !POST_KEYS.includes(k))) throw Error();
|
||||
return value;
|
||||
}
|
||||
|
||||
// Registers both routes. `catalogues` returns each active organization's `routes` catalogue (or a
|
||||
// status when its bus did not answer); `harnesses` lists the installed harness modules.
|
||||
export function triageRouteRoutes({ dataRoot, catalogues, harnesses = installedHarnesses, log = () => {} }) {
|
||||
async function catalogue() {
|
||||
let list;
|
||||
try { list = await catalogues(); } catch { list = []; }
|
||||
return mergeCatalogues(list, harnesses());
|
||||
}
|
||||
async function handle(req, res, context) {
|
||||
const done = (status, value) => {
|
||||
// Method, path and status only.
|
||||
log(`${new Date().toISOString()} ${req.method} ${TRIAGE_ROUTE_PATH} ${status}`);
|
||||
return json(res, status, value);
|
||||
};
|
||||
const refuse = (code, extra = {}) => done(STATUS[code] ?? 503, { error: code, message: MESSAGES[code] ?? MESSAGES['outcome-unknown'], ...extra });
|
||||
if (req.method !== 'GET' && req.method !== 'POST') return refuse('method-not-allowed');
|
||||
// The proxy-admin gate: the instance owner's own session, never a tester or a granted user.
|
||||
if (context.owner !== true || !context.session) return refuse('human-required');
|
||||
// A write needs this Console's Origin and same-origin Fetch Metadata, exactly once each. A read
|
||||
// refuses another site's Origin or Fetch Metadata and lets their absence through, as the packs and
|
||||
// request reads do: a browser sends no Origin on a same-origin GET, and Chromium sends no
|
||||
// Fetch Metadata to a LAN http origin (serve.mjs marks only a LAN write).
|
||||
if (req.method === 'POST') {
|
||||
if (one(req, 'origin') !== context.origin) return refuse('cross-origin');
|
||||
if (one(req, 'sec-fetch-site') !== 'same-origin') return refuse('cross-site');
|
||||
} else {
|
||||
if (req.headersDistinct.origin && one(req, 'origin') !== context.origin) return refuse('cross-origin');
|
||||
if (req.headersDistinct['sec-fetch-site'] && !['same-origin', 'none'].includes(one(req, 'sec-fetch-site'))) return refuse('cross-site');
|
||||
}
|
||||
if (req.method === 'GET') {
|
||||
let doc;
|
||||
try { doc = readRoute(dataRoot); } catch (error) { return refuse(error?.code === 'triage-route-unreadable' ? 'triage-route-unreadable' : 'outcome-unknown'); }
|
||||
const route = view(doc);
|
||||
return done(200, {
|
||||
route, ...(route ? {} : { routeMessage: NOT_SET_MESSAGE }), revision: route ? route.revision : 0,
|
||||
choices: choices(await catalogue()), send: context.send === true, note: NOT_STARTED_NOTE,
|
||||
});
|
||||
}
|
||||
if (context.send !== true) return refuse('read-only');
|
||||
let body;
|
||||
try { body = await readBody(req); } catch { return refuse('invalid-request'); }
|
||||
const { model, harness, cap, timeZone, revision } = body;
|
||||
if (typeof model !== 'string' || !MODEL_KEY.test(model)) return refuse('unknown-model');
|
||||
if (typeof harness !== 'string' || !HARNESS_NAME.test(harness)) return refuse('harness-off');
|
||||
if (!Number.isSafeInteger(cap) || cap < 1) return refuse('triage-cap-invalid');
|
||||
if (timeZone === undefined || timeZone === null || timeZone === '') return refuse('triage-zone-required');
|
||||
if (!knownZone(timeZone)) return refuse('triage-zone-unknown');
|
||||
const zone = canonicalZone(timeZone);
|
||||
if (!Number.isSafeInteger(revision) || revision < 0) return refuse('invalid-request');
|
||||
try { checkTriageRoute({ model, harness, catalogue: await catalogue(), harnesses: harnesses() }); }
|
||||
catch (error) {
|
||||
const code = ['unknown-model', 'model-provider-unavailable', 'model-harness-incompatible', 'harness-off'].includes(error?.code) ? error.code : 'outcome-unknown';
|
||||
return refuse(code);
|
||||
}
|
||||
const by = ownerActor(context.session);
|
||||
if (by === null) return refuse('human-required');
|
||||
try {
|
||||
const doc = writeRoute(dataRoot, { model, harness, limit: cap, timeZone: zone, by, revision });
|
||||
return done(200, { route: view(doc), revision: doc.revision, note: NOT_STARTED_NOTE });
|
||||
} catch (error) {
|
||||
const code = typeof error?.code === 'string' && Object.hasOwn(MESSAGES, error.code) && error.code.startsWith('triage-route-') ? error.code : 'outcome-unknown';
|
||||
if (code !== 'triage-route-stale') return refuse(code);
|
||||
let stored = null;
|
||||
try { stored = view(readRoute(dataRoot)); } catch { /* the conflict still stands */ }
|
||||
return refuse(code, { revision: stored ? stored.revision : 0 });
|
||||
}
|
||||
}
|
||||
return registered => {
|
||||
registered.add('GET', TRIAGE_ROUTE_PATH, handle);
|
||||
registered.add('POST', TRIAGE_ROUTE_PATH, handle);
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,220 @@
|
||||
// #1873 triage-cap: Triage's token usage and its daily cap.
|
||||
//
|
||||
// The Triage host appends one line per turn to <dataRoot>/triage/usage.jsonl:
|
||||
// { at, run, turn, status, input, output, cacheRead, cacheWrite }
|
||||
// A known turn carries four non-negative safe integers; an unknown turn carries four nulls. The file
|
||||
// sits outside the run directory, so a new start never clears it. It is held to the route store's
|
||||
// file checks: a regular file, one link, the running user's, mode 0600. A link, a non-file, a wrong
|
||||
// mode or owner refuses the read and the append alike, and the append leaves the file as it was.
|
||||
// Lines are only ever appended. Nothing here prunes the file.
|
||||
//
|
||||
// The cap is the daily-quota plugin's own rule (packages/plugins/limit-check/daily-quota), called
|
||||
// with the organization limits shape and the stored cap as its one leaf, for one account named
|
||||
// "triage". No counting rule lives here: the count, the window, the DST handling and the hit all
|
||||
// come from mayLaunch and report. nextReset only asks report when the window turns over.
|
||||
import { closeSync, constants, fchmodSync, fstatSync, fsyncSync, openSync, readSync, writeSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { mayLaunch, report } from "../../plugins/limit-check/daily-quota/index.mjs";
|
||||
|
||||
export const USAGE_ACCOUNT = "triage";
|
||||
export const USAGE_FIELDS = Object.freeze(["at", "run", "turn", "status", "input", "output", "cacheRead", "cacheWrite"]);
|
||||
const KINDS = ["input", "output", "cacheWrite", "cacheRead"];
|
||||
const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
|
||||
// A run id is the host's own triage-<14 digits>; a turn id is the runner's message id, maybe with
|
||||
// ":retry". Both are held to one short, plain shape so a line never carries free text.
|
||||
export const ID = /^[A-Za-z0-9:._-]{1,80}$/;
|
||||
// 32 MiB holds well over a hundred thousand lines; a larger file refuses rather than stalls a read.
|
||||
export const USAGE_MAX_BYTES = 32 * 1024 * 1024;
|
||||
const HOUR = 3600000;
|
||||
|
||||
export class UsageError extends Error {
|
||||
constructor(why) {
|
||||
super(`the Triage usage file is unreadable: ${why}`);
|
||||
this.name = "UsageError";
|
||||
this.code = "triage-usage-unreadable";
|
||||
}
|
||||
}
|
||||
const unreadable = (why) => new UsageError(why);
|
||||
|
||||
export const usagePath = (dataRoot) => join(dataRoot, "triage", "usage.jsonl");
|
||||
|
||||
const plain = (v) => v !== null && typeof v === "object" && !Array.isArray(v) && Object.getPrototypeOf(v) === Object.prototype;
|
||||
const canonicalTime = (s) => typeof s === "string" && ISO.test(s) && !Number.isNaN(Date.parse(s)) && new Date(s).toISOString() === s;
|
||||
const count = (n) => Number.isSafeInteger(n) && n >= 0;
|
||||
|
||||
// One line's shape, exactly.
|
||||
export function validLine(v) {
|
||||
if (!plain(v) || Object.keys(v).length !== USAGE_FIELDS.length || !USAGE_FIELDS.every((k) => Object.hasOwn(v, k))) return false;
|
||||
if (!canonicalTime(v.at) || typeof v.run !== "string" || !ID.test(v.run) || typeof v.turn !== "string" || !ID.test(v.turn)) return false;
|
||||
if (v.status === "known") return KINDS.every((k) => count(v[k]));
|
||||
if (v.status === "unknown") return KINDS.every((k) => v[k] === null);
|
||||
return false;
|
||||
}
|
||||
|
||||
// An opened file's checks, as the route store holds route.json to.
|
||||
function checkFile(fd, uid) {
|
||||
const st = fstatSync(fd);
|
||||
if (!st.isFile()) throw unreadable("usage.jsonl is not a regular file");
|
||||
if (st.nlink !== 1) throw unreadable("usage.jsonl has more than one link");
|
||||
if (st.uid !== uid) throw unreadable("usage.jsonl is not owned by this user");
|
||||
if ((st.mode & 0o777) !== 0o600) throw unreadable("usage.jsonl mode is not 0600");
|
||||
return st;
|
||||
}
|
||||
|
||||
const uidOf = (opts) => opts.uid ?? process.getuid();
|
||||
|
||||
// Append one line. A torn last line (a write that died) is closed off with a newline first, so the
|
||||
// new line always stands alone and the torn one stays one bad line.
|
||||
export function appendUsage(file, line, opts = {}) {
|
||||
if (!validLine(line)) throw unreadable("the line does not match the usage shape");
|
||||
const text = JSON.stringify(Object.fromEntries(USAGE_FIELDS.map((k) => [k, line[k]]))) + "\n";
|
||||
const flags = constants.O_RDWR | constants.O_APPEND | constants.O_NOFOLLOW | constants.O_NONBLOCK;
|
||||
let fd;
|
||||
try { fd = openSync(file, flags); }
|
||||
catch (e) {
|
||||
if (e.code === "ELOOP") throw unreadable("usage.jsonl is a link");
|
||||
if (e.code !== "ENOENT") throw unreadable("usage.jsonl cannot be opened");
|
||||
try { fd = openSync(file, flags | constants.O_CREAT | constants.O_EXCL, 0o600); }
|
||||
catch { throw unreadable("usage.jsonl cannot be made"); }
|
||||
// The umask may have narrowed the mode; set it on the descriptor just made.
|
||||
try { fchmodSync(fd, 0o600); } catch { closeSync(fd); throw unreadable("usage.jsonl cannot be made"); }
|
||||
}
|
||||
try {
|
||||
const st = checkFile(fd, uidOf(opts));
|
||||
let lead = "";
|
||||
if (st.size > 0) {
|
||||
const last = Buffer.alloc(1);
|
||||
if (readSync(fd, last, 0, 1, st.size - 1) !== 1) throw unreadable("usage.jsonl cannot be read");
|
||||
if (last[0] !== 0x0a) lead = "\n";
|
||||
}
|
||||
try {
|
||||
writeSync(fd, lead + text);
|
||||
fsyncSync(fd);
|
||||
} catch { throw unreadable("usage.jsonl cannot be written"); }
|
||||
} finally { closeSync(fd); }
|
||||
}
|
||||
|
||||
const nulls = { input: null, output: null, cacheRead: null, cacheWrite: null };
|
||||
|
||||
// Every line as a daily-quota record for the triage account. A line that does not parse or does
|
||||
// not match the shape (a torn last line, an edit by hand) is refused: it becomes one unknown turn,
|
||||
// keyed by its line number, dated by the next good line (or the file's change time when none
|
||||
// follows), and is listed in `bad` so the host can log it. No file reads as no turns.
|
||||
export function readUsage(file, opts = {}) {
|
||||
const maxBytes = opts.maxBytes ?? USAGE_MAX_BYTES;
|
||||
let fd;
|
||||
try { fd = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); }
|
||||
catch (e) {
|
||||
if (e.code === "ENOENT") return { records: [], bad: [] };
|
||||
if (e.code === "ELOOP") throw unreadable("usage.jsonl is a link");
|
||||
throw unreadable("usage.jsonl cannot be opened");
|
||||
}
|
||||
let st;
|
||||
let buf;
|
||||
try {
|
||||
st = checkFile(fd, uidOf(opts));
|
||||
if (st.size > maxBytes) throw unreadable("usage.jsonl is too large");
|
||||
buf = Buffer.alloc(st.size);
|
||||
let got = 0;
|
||||
while (got < st.size) {
|
||||
const n = readSync(fd, buf, got, st.size - got, got);
|
||||
if (n === 0) break;
|
||||
got += n;
|
||||
}
|
||||
buf = buf.subarray(0, got);
|
||||
} finally { closeSync(fd); }
|
||||
const lines = buf.toString("utf8").split("\n");
|
||||
if (lines.at(-1) === "") lines.pop();
|
||||
const parsed = lines.map((text) => {
|
||||
try {
|
||||
const v = JSON.parse(text);
|
||||
return validLine(v) ? v : null;
|
||||
} catch { return null; }
|
||||
});
|
||||
const records = new Array(parsed.length);
|
||||
const bad = [];
|
||||
let nextAt = st.mtime.toISOString();
|
||||
for (let i = parsed.length - 1; i >= 0; i--) {
|
||||
const v = parsed[i];
|
||||
if (v) {
|
||||
records[i] = { account: USAGE_ACCOUNT, ...v };
|
||||
nextAt = v.at;
|
||||
} else {
|
||||
records[i] = { account: USAGE_ACCOUNT, at: nextAt, run: null, turn: `line-${i + 1}`, status: "unknown", ...nulls };
|
||||
bad.unshift({ line: i + 1 });
|
||||
}
|
||||
}
|
||||
return { records, bad, ino: st.ino };
|
||||
}
|
||||
|
||||
// The organization limits shape with the stored cap as its one leaf.
|
||||
export const limitsFor = (cap) => ({ tokens: { daily: { [USAGE_ACCOUNT]: cap } } });
|
||||
const SUBJECT = Object.freeze({ account: USAGE_ACCOUNT });
|
||||
const iso = (ms) => new Date(ms).toISOString();
|
||||
|
||||
// The next reset after nowMs, or after the cap's `since` when that is later. A probe record at that
|
||||
// instant counts while the window holding t still starts at or before it; the reset is the first
|
||||
// millisecond at which report stops counting it. A day is at most 25 hours, so 49 hours always
|
||||
// holds the next one.
|
||||
export function nextReset(cap, nowMs) {
|
||||
const from = Math.max(nowMs, Date.parse(cap.since));
|
||||
const limits = limitsFor(cap);
|
||||
const probe = [{ account: USAGE_ACCOUNT, at: iso(from), run: "probe", turn: "probe", status: "known", input: 1, output: 0, cacheRead: 0, cacheWrite: 0 }];
|
||||
const counts = (t) => report({ limits, records: probe, now: iso(t) })[0].count > 0;
|
||||
let lo = from;
|
||||
let hi = from + 49 * HOUR;
|
||||
if (counts(hi)) throw unreadable("the cap has no reset in the next two days");
|
||||
while (hi - lo > 1) {
|
||||
const mid = Math.floor((lo + hi) / 2);
|
||||
if (counts(mid)) lo = mid; else hi = mid;
|
||||
}
|
||||
return hi;
|
||||
}
|
||||
|
||||
// Today's count, the decision and the next reset for the stored cap.
|
||||
export function capState({ cap, records, now }) {
|
||||
const at = iso(now);
|
||||
const limits = limitsFor(cap);
|
||||
const decision = mayLaunch({ limits, subject: SUBJECT, records, now: at });
|
||||
const [r] = report({ limits, records, now: at });
|
||||
return {
|
||||
allow: decision.allow,
|
||||
hit: decision.allow ? null : decision.hit,
|
||||
usageToday: { count: r.count, ...r.kinds, unknownTurns: r.unknownTurns },
|
||||
resetsAt: iso(nextReset(cap, now)),
|
||||
};
|
||||
}
|
||||
|
||||
// The turn delta, copied from packages/bus/src/usage.mjs (snapshot, delta; not exported there and
|
||||
// that package is not this lane's to change). Same codes, same rules: the turn's usage is the sum
|
||||
// of the transcript entries that are new since the turn began; a changed or missing earlier entry
|
||||
// is "transcript-rewritten", and no new entry is "no-new-usage".
|
||||
const unknown = (code) => ({ status: "unknown", code });
|
||||
export function snapshot(value) {
|
||||
if (value?.status === "unknown") return unknown(/^transcript-(path|size|line|entries|records|changed|parse|usage|harness)$/.test(value.code) ? value.code : "source-unknown");
|
||||
if (value?.status !== "known" || !value.entries || typeof value.entries !== "object" || Array.isArray(value.entries)) return unknown("source-invalid");
|
||||
const entries = Object.entries(value.entries);
|
||||
if (entries.length > 20000) return unknown("source-invalid");
|
||||
for (const [id, v] of entries) {
|
||||
if (id.length > 512 || !v || Object.keys(v).length !== 4 || KINDS.some((k) => !Number.isSafeInteger(v[k]) || v[k] < 0)) return unknown("source-invalid");
|
||||
}
|
||||
return { status: "known", entries: value.entries };
|
||||
}
|
||||
export function delta(before, after) {
|
||||
if (!before || before.status !== "known") return unknown("baseline-unknown");
|
||||
if (after.status !== "known") return after;
|
||||
for (const [id, v] of Object.entries(before.entries)) {
|
||||
if (!Object.hasOwn(after.entries, id) || KINDS.some((k) => after.entries[id][k] !== v[k])) return unknown("transcript-rewritten");
|
||||
}
|
||||
const sum = { input: 0, output: 0, cacheWrite: 0, cacheRead: 0 };
|
||||
let added = false;
|
||||
for (const [id, v] of Object.entries(after.entries)) {
|
||||
if (Object.hasOwn(before.entries, id)) continue;
|
||||
added = true;
|
||||
for (const k of KINDS) {
|
||||
sum[k] += v[k];
|
||||
if (!Number.isSafeInteger(sum[k])) return unknown("usage-overflow");
|
||||
}
|
||||
}
|
||||
return added ? { status: "known", ...sum } : unknown("no-new-usage");
|
||||
}
|
||||
@@ -910,3 +910,124 @@ for (const [label, answers, line] of [
|
||||
assert.equal(posts.filter(p => p.url.endsWith('/agent/launch')).length, launched ? 1 : 0);
|
||||
assert.equal(await b.evaluate('window.__answers.length'), 0, 'every queued answer was asked for');
|
||||
});
|
||||
|
||||
// SP20 (#1873): an organization refusal of Stop says so in one plain line, the line Restart gives.
|
||||
for (const [label, answer, line] of [
|
||||
['the organization is archived', { status: 409, body: { error: 'org-archived', message: 'organization is archived' } },
|
||||
/^Not stopped: Acme Studio is archived, so its agents are not changed\. Nothing changed\.$/],
|
||||
['the organization is not available', { status: 404, body: { error: 'org-unavailable' } },
|
||||
/^Not stopped: Acme Studio is not available to this Console\. Nothing changed\.$/],
|
||||
]) test(`SP20: a Stop refusal says so in the drawer: ${label}`, async t => {
|
||||
const { b, wait } = await setup(t, { hash: '#/agents?agent=acme%3Acoder' });
|
||||
const d = '[data-ws-drawer="acme:coder"]';
|
||||
await wait(`${open('acme:coder')} && document.querySelector('${d} [data-drawer-stop]')`);
|
||||
await b.evaluate(`window.__answers.push(${JSON.stringify(answer)})`);
|
||||
await b.evaluate(`document.querySelector('${d} [data-drawer-stop]').click()`);
|
||||
await wait(`document.querySelector('${d} [data-drawer-stop-yes]')`);
|
||||
await b.evaluate(`document.querySelector('${d} [data-drawer-stop-yes]').click()`);
|
||||
await wait(`/^Not stopped: /.test(${status(d)}) && ${usable(d)}`);
|
||||
assert.match(await b.evaluate(status(d)), line);
|
||||
assert.equal(await b.evaluate(`document.activeElement === document.querySelector('${d} [data-drawer-stop]')`), true, 'focus is back on Stop');
|
||||
assert.deepEqual((await b.evaluate('window.__posts')).map(p => [p.url, p.method]), [['/api/orgs/acme/agent/terminate', 'POST']]);
|
||||
});
|
||||
|
||||
// SP20: each Stop keeps its own state, as each restart does. Another drawer's question or Stop does not
|
||||
// lose a running Stop's line or its running state, and a Stop that ends does not drop another drawer's question.
|
||||
test('SP20: two Stops in two drawers each keep their own line and running state', async t => {
|
||||
const data = D => { D.acme.instances.ideas = { definition: 'ideas', running: ['run-4'] }; };
|
||||
const { b, wait } = await setup(t, { hash: '#/agents?agent=acme%3Acoder', data });
|
||||
const d = '[data-ws-drawer="acme:coder"]', i = '[data-ws-drawer="acme:ideas"]';
|
||||
const refused = who => ({ status: 409, body: { error: 'instance-protected', message: `instance-protected: ${who} is protected from ending. Nothing changed.`, sent: false } });
|
||||
const state = x => `[${status(x)}, document.querySelector('${x} [data-drawer-stop]')?.firstChild.textContent, document.querySelector('${x} [data-drawer-stop]')?.disabled, document.querySelector('${x} [data-drawer-restart]')?.disabled]`;
|
||||
await wait(`${open('acme:coder')} && document.querySelector('${d} [data-drawer-stop]')`);
|
||||
// Coder's Stop runs: its terminate is held.
|
||||
await b.evaluate(`window.__answers.push({ hold: new Promise(r => { window.__release = r; }), ...${JSON.stringify(refused('coder'))} })`);
|
||||
await b.evaluate(`document.querySelector('${d} [data-drawer-stop]').click()`);
|
||||
await wait(`document.querySelector('${d} [data-drawer-stop-yes]')`);
|
||||
await b.evaluate(`document.querySelector('${d} [data-drawer-stop-yes]').click()`);
|
||||
await wait(`window.__posts.length === 1`);
|
||||
// Ideas asks to stop and is cancelled; back on coder, its Stop still shows as running.
|
||||
await b.evaluate('location.hash = "#/agents?agent=acme%3Aideas"');
|
||||
await wait(`${open('acme:ideas')} && document.querySelector('${i} [data-drawer-stop]')`);
|
||||
await b.evaluate(`document.querySelector('${i} [data-drawer-stop]').click()`);
|
||||
await wait(`document.querySelector('${i} [data-drawer-stop-no]')`);
|
||||
await b.evaluate(`document.querySelector('${i} [data-drawer-stop-no]').click()`);
|
||||
await b.evaluate('location.hash = "#/agents?agent=acme%3Acoder"');
|
||||
await wait(`${open('acme:coder')} && document.querySelector('${d} [data-drawer-stop]')`);
|
||||
assert.deepEqual(await b.evaluate(state(d)), ['Stopping coder…', 'Stopping…', true, true]);
|
||||
// Ideas' Stop question is open when coder's Stop is refused; each drawer keeps its own.
|
||||
await b.evaluate('location.hash = "#/agents?agent=acme%3Aideas"');
|
||||
await wait(`${open('acme:ideas')} && document.querySelector('${i} [data-drawer-stop]')`);
|
||||
await b.evaluate(`document.querySelector('${i} [data-drawer-stop]').click()`);
|
||||
await wait(`document.querySelector('${i} [data-drawer-stop-no]')`);
|
||||
await b.evaluate('window.__release()');
|
||||
await b.evaluate('new Promise(r => setTimeout(r, 300))');
|
||||
await b.evaluate('location.hash = "#/agents?agent=acme%3Acoder"');
|
||||
await wait(`${open('acme:coder')} && /^Not stopped: /.test(${status(d)}) && ${usable(d)}`);
|
||||
assert.equal(await b.evaluate(status(d)), 'Not stopped: instance-protected: coder is protected from ending. Nothing changed.');
|
||||
await b.evaluate('location.hash = "#/agents?agent=acme%3Aideas"');
|
||||
await wait(`${open('acme:ideas')} && document.querySelector('${i} [data-drawer-stop]')`);
|
||||
assert.equal(await b.evaluate(`!!document.querySelector('${i} [data-drawer-stop-yes]')`), true, 'ideas keeps its question');
|
||||
// Ideas' Stop runs and is refused; coder's line is still there after, and ideas' is too.
|
||||
await b.evaluate(`window.__answers.push(${JSON.stringify(refused('ideas'))})`);
|
||||
await b.evaluate(`document.querySelector('${i} [data-drawer-stop-yes]').click()`);
|
||||
await wait(`/^Not stopped: /.test(${status(i)}) && ${usable(i)}`);
|
||||
await b.evaluate('location.hash = "#/agents?agent=acme%3Acoder"');
|
||||
await wait(`${open('acme:coder')} && document.querySelector('${d} [data-drawer-stop]')`);
|
||||
assert.deepEqual(await b.evaluate(state(d)), ['Not stopped: instance-protected: coder is protected from ending. Nothing changed.', 'Stop', false, false]);
|
||||
await b.evaluate('location.hash = "#/agents?agent=acme%3Aideas"');
|
||||
await wait(`${open('acme:ideas')} && document.querySelector('${i} [data-drawer-stop]')`);
|
||||
assert.deepEqual(await b.evaluate(state(i)), ['Not stopped: instance-protected: ideas is protected from ending. Nothing changed.', 'Stop', false, false]);
|
||||
assert.deepEqual((await b.evaluate('window.__posts')).map(p => JSON.parse(p.body).instance), ['coder', 'ideas']);
|
||||
});
|
||||
|
||||
// SP20 review: two Stops run at once. One that ends, here the one the bus takes, which repaints the page,
|
||||
// does not end the other's running state or line, and each drawer keeps its own line after.
|
||||
test('SP20: two Stops running at once in two drawers each keep their own state', async t => {
|
||||
const data = D => { D.acme.instances.ideas = { definition: 'ideas', running: ['run-4'] }; };
|
||||
const { b, wait } = await setup(t, { hash: '#/agents?agent=acme%3Acoder', scripts: [FAST], data });
|
||||
const d = '[data-ws-drawer="acme:coder"]', i = '[data-ws-drawer="acme:ideas"]';
|
||||
const REQ = '0f0e0d0c-0b0a-4908-8706-050403020101';
|
||||
const state = x => `[${status(x)}, document.querySelector('${x} [data-drawer-stop]')?.firstChild.textContent, document.querySelector('${x} [data-drawer-stop]')?.disabled, document.querySelector('${x} [data-drawer-restart]')?.disabled]`;
|
||||
const ask = async x => {
|
||||
await b.evaluate(`document.querySelector('${x} [data-drawer-stop]').click()`);
|
||||
await wait(`document.querySelector('${x} [data-drawer-stop-yes]')`);
|
||||
await b.evaluate(`document.querySelector('${x} [data-drawer-stop-yes]').click()`);
|
||||
};
|
||||
const show = async (agent, x) => {
|
||||
await b.evaluate(`location.hash = "#/agents?agent=acme%3A${agent}"`);
|
||||
await wait(`${open(`acme:${agent}`)} && document.querySelector('${x} [data-drawer-stop]')`);
|
||||
};
|
||||
// Both terminates are held: coder's the bus takes, ideas' it refuses.
|
||||
await b.evaluate(`window.__answers.push(
|
||||
{ hold: new Promise(r => { window.__releaseCoder = r; }), status: 200, body: { sent: true, request: '${REQ}' } },
|
||||
{ hold: new Promise(r => { window.__releaseIdeas = r; }), status: 409, body: { error: 'instance-protected', message: 'instance-protected: ideas is protected from ending. Nothing changed.', sent: false } },
|
||||
{ status: 200, body: { request: '${REQ}', state: 'closed' } })`);
|
||||
await wait(`${open('acme:coder')} && document.querySelector('${d} [data-drawer-stop]')`);
|
||||
await ask(d);
|
||||
await wait('window.__posts.length === 1');
|
||||
await show('ideas', i);
|
||||
await ask(i);
|
||||
await wait('window.__posts.length === 2');
|
||||
assert.deepEqual(await b.evaluate(state(i)), ['Stopping ideas…', 'Stopping…', true, true]);
|
||||
// Coder's Stop ends while ideas' drawer is on screen; ideas' Stop still runs.
|
||||
await b.evaluate('window.__releaseCoder()');
|
||||
await wait('window.__posts.length === 3');
|
||||
await b.evaluate('new Promise(r => setTimeout(r, 300))');
|
||||
await wait(`${open('acme:ideas')} && document.querySelector('${i} [data-drawer-stop]')`);
|
||||
assert.deepEqual(await b.evaluate(state(i)), ['Stopping ideas…', 'Stopping…', true, true]);
|
||||
await show('coder', d);
|
||||
assert.deepEqual(await b.evaluate(state(d)), ['Stopped coder.', 'Stop', false, false]);
|
||||
// Ideas' Stop is refused; each drawer keeps its own line.
|
||||
await show('ideas', i);
|
||||
assert.deepEqual(await b.evaluate(state(i)), ['Stopping ideas…', 'Stopping…', true, true]);
|
||||
await b.evaluate('window.__releaseIdeas()');
|
||||
await wait(`/^Not stopped: /.test(${status(i)}) && ${usable(i)}`);
|
||||
assert.deepEqual(await b.evaluate(state(i)), ['Not stopped: instance-protected: ideas is protected from ending. Nothing changed.', 'Stop', false, false]);
|
||||
await show('coder', d);
|
||||
assert.deepEqual(await b.evaluate(state(d)), ['Stopped coder.', 'Stop', false, false]);
|
||||
assert.deepEqual((await b.evaluate('window.__posts')).map(p => [p.url, p.method]), [
|
||||
['/api/orgs/acme/agent/terminate', 'POST'], ['/api/orgs/acme/agent/terminate', 'POST'], [`/api/orgs/acme/agent/request?id=${REQ}`, 'GET']]);
|
||||
assert.deepEqual((await b.evaluate('window.__posts')).slice(0, 2).map(p => JSON.parse(p.body).instance), ['coder', 'ideas']);
|
||||
assert.equal(await b.evaluate('window.__answers.length'), 0, 'every queued answer was asked for');
|
||||
});
|
||||
@@ -18,8 +18,8 @@ test('served Console browser: real board fixtures, keyboard, drafts, receipts, t
|
||||
assert.equal(await b.evaluate('document.querySelector("#session-count").textContent'), '3 of 4');
|
||||
await b.evaluate('document.querySelector("#hide-offline").click()');
|
||||
assert.equal(await b.evaluate('document.querySelectorAll("table.sessions tbody tr").length'), 4);
|
||||
// Rows 129, 154, 191, 194 and 294: chat.js, editor.js, processing.js, agents.js and federation.js are the scripts added, and #1839 adds requests.js (the intake panel under an owner's chat message), all same origin; the list is the pin, so another script is a visible change. #797 adds project-archive.js immediately after chat.js. F2 adds onboarding.js after org-create.js, and F3 adds console-restart.js after it. U2 (#1873) adds workspace-data.js and workspace.js before console-restart.js, and U4b adds tasks-board.js after workspace.js. U5 (#1873) added panes.js (Side by side) just before bus.js; SP6 replaces it with assistant.js (the Assistant page), which bus.js mounts.
|
||||
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("script")].map(s => s.getAttribute("src"))'), ['/editor.js', '/processing.js', '/org-create.js', '/onboarding.js', '/workspace-data.js', '/workspace.js', '/tasks-board.js', '/console-restart.js', '/display.js', '/inbox.js', '/files.js', '/project-remote.js', '/autonomy.js', '/packs.js', '/workflows.js', '/federation.js', '/brand.js', '/app.js', '/markdown-preview.js', '/projects.js', '/chat.js', '/project-archive.js', '/agents.js', '/assistant.js', '/bus.js', '/requests.js']);
|
||||
// Rows 129, 154, 191, 194 and 294: chat.js, editor.js, processing.js, agents.js and federation.js are the scripts added, and #1839 adds requests.js (the intake panel under an owner's chat message), all same origin; the list is the pin, so another script is a visible change. #797 adds project-archive.js immediately after chat.js. F2 adds onboarding.js after org-create.js, and F3 adds console-restart.js after it. U2 (#1873) adds workspace-data.js and workspace.js before console-restart.js, and U4b adds tasks-board.js after workspace.js. triage-settings (#1873) adds triage-settings.js after console-restart.js. U5 (#1873) added panes.js (Side by side) just before bus.js; SP6 replaces it with assistant.js (the Assistant page), which bus.js mounts.
|
||||
assert.deepEqual(await b.evaluate('[...document.querySelectorAll("script")].map(s => s.getAttribute("src"))'), ['/editor.js', '/processing.js', '/org-create.js', '/onboarding.js', '/workspace-data.js', '/workspace.js', '/tasks-board.js', '/console-restart.js', '/triage-settings.js', '/display.js', '/inbox.js', '/files.js', '/project-remote.js', '/autonomy.js', '/packs.js', '/workflows.js', '/federation.js', '/brand.js', '/app.js', '/markdown-preview.js', '/projects.js', '/chat.js', '/project-archive.js', '/agents.js', '/assistant.js', '/bus.js', '/requests.js']);
|
||||
assert.equal(await b.evaluate(`document.querySelectorAll('script[src="/processing.js"]').length`), 1);
|
||||
assert.equal(await b.evaluate(`document.querySelectorAll('script[src="/chat.js"]').length`), 1);
|
||||
assert.equal(await b.evaluate('!!window.injected'), false);
|
||||
|
||||
+14
-1
@@ -8,8 +8,12 @@
|
||||
// SLEEP hang until killed
|
||||
// REPLY-SLEEP send the person an answer (class RESULT) to this message through the tool socket, then hang until killed
|
||||
// NOTE-SLEEP send the person a progress note (class INFO) on this message through the tool socket, then hang until killed
|
||||
// SPEND <n> add one usage entry of <n> input tokens to the session's spend file (session/spend.jsonl,
|
||||
// which the host tests' usage source reads as the transcript), then answer "spent <n>"
|
||||
// GATED-SPEND <n> write work/started, wait for work/gate, then spend <n> as SPEND does
|
||||
// anything else answer "echo: <body>"
|
||||
import { existsSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { appendFileSync, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { randomBytes } from "node:crypto";
|
||||
import { connect } from "node:net";
|
||||
import { join } from "node:path";
|
||||
|
||||
@@ -64,6 +68,15 @@ if (body === "PROBE") {
|
||||
const id = request.match(/^Message (\S+) from /m)?.[1];
|
||||
await toolCall("send_message", { to: "human", body: "still looking", class: "INFO", in_reply_to: id });
|
||||
setInterval(() => {}, 1000);
|
||||
} else if (/^(GATED-)?SPEND \d{1,9}$/.test(body)) {
|
||||
const n = Number(body.split(" ")[1]);
|
||||
if (body.startsWith("GATED-")) {
|
||||
writeFileSync(join(workspace, "started"), "x");
|
||||
while (!existsSync(join(workspace, "gate"))) await new Promise((r) => setTimeout(r, 50));
|
||||
}
|
||||
mkdirSync(join(workspace, "..", "session"), { recursive: true });
|
||||
appendFileSync(join(workspace, "..", "session", "spend.jsonl"), JSON.stringify({ id: randomBytes(6).toString("hex"), n }) + "\n");
|
||||
process.stdout.write(`spent ${n}`);
|
||||
} else if (body === "SLEEP") {
|
||||
setInterval(() => {}, 1000);
|
||||
} else {
|
||||
|
||||
@@ -63,7 +63,7 @@ function fakeHost(over = {}) {
|
||||
const triage = {
|
||||
snapshotAt: () => Date.now(),
|
||||
refresh: async () => null,
|
||||
state: async () => { h.reads += 1; return { route: h.route, running: h.running, lastExit: h.lastExit, overCapUntil: h.overCapUntil, snapshot: h.snapshot, transcript: h.transcript }; },
|
||||
state: async () => { h.reads += 1; return { route: h.route, running: h.running, lastExit: h.lastExit, overCapUntil: h.overCapUntil, snapshot: h.snapshot, transcript: h.transcript, ...(Object.hasOwn(h, 'usageToday') ? { usageToday: h.usageToday, cap: h.cap, resetsAt: h.resetsAt } : {}) }; },
|
||||
message: async text => {
|
||||
h.sent.push(text);
|
||||
const e = h.next; h.next = null;
|
||||
@@ -98,6 +98,9 @@ async function setup(t, { host = fakeHost(), triage = host.triage, send = true,
|
||||
|
||||
// The page focuses its heading when its first render ends (bus.js). Open after that, as a person would, or
|
||||
// that late focus can land after the drawer's own.
|
||||
// The Settings > Triage panel script (triage-settings, #1873) is a static file every page loads, like
|
||||
// every other page module; it carries no data. These checks are about Triage traffic, so they skip it.
|
||||
const SETTINGS_SCRIPT = 'GET /triage-settings.js';
|
||||
const open = async f => { await f.wait('document.querySelector("#triage-open")'); await f.wait('document.activeElement?.tagName === "H1"', 100).catch(() => {}); await f.click('#triage-open'); await f.wait('["off", "cap", "running", "stopped"].includes(document.querySelector("#triage-drawer #triage-state")?.dataset.kind)'); };
|
||||
const type = async (f, words) => { await f.b.evaluate('document.querySelector("#triage-text").focus()'); await f.b.call('Input.insertText', { text: words }); };
|
||||
const sendAndWait = async (f, before) => { await f.click('#triage-send'); await f.wait(`document.querySelector("#triage-send-status").textContent.trim() && document.querySelector("#triage-send-status").textContent !== ${JSON.stringify(before)} && !document.querySelector("#triage-send").disabled`); return f.text('#triage-send-status'); };
|
||||
@@ -109,7 +112,7 @@ test('item 1: a session that is not the instance owner gets no entry, no drawer
|
||||
await f.wait('document.querySelector("#dashboard-kpis")');
|
||||
await sleep(1500);
|
||||
assert.equal(await f.b.evaluate('document.querySelector("#triage-open, #triage-drawer, [id^=triage]")'), null);
|
||||
assert.deepEqual(f.requests.filter(r => /triage/i.test(r)), []);
|
||||
assert.deepEqual(f.requests.filter(r => /triage/i.test(r) && r !== SETTINGS_SCRIPT), []);
|
||||
}
|
||||
// A Triage host, but a signed-in session that is not the root owner.
|
||||
{
|
||||
@@ -119,7 +122,7 @@ test('item 1: a session that is not the instance owner gets no entry, no drawer
|
||||
await sleep(1500);
|
||||
assert.equal(await f.b.evaluate('document.querySelector("[id^=triage]")'), null);
|
||||
assert.equal(await f.b.evaluate('document.body.innerHTML.includes("Triage")'), false, 'no Triage words in the page');
|
||||
assert.deepEqual(f.requests.filter(r => /triage/i.test(r)), []);
|
||||
assert.deepEqual(f.requests.filter(r => /triage/i.test(r) && r !== SETTINGS_SCRIPT), []);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -135,7 +138,7 @@ test('item 1: the owner gets one Triage entry in the top bar on the dashboard an
|
||||
assert.equal(await f.b.evaluate('document.querySelector("#triage-open").getAttribute("aria-expanded")'), 'true');
|
||||
// From inside an organization the read is still the system route, never the organization's.
|
||||
assert.ok(f.triageReads() >= 1);
|
||||
assert.deepEqual(f.requests.filter(r => /triage/.test(r) && r !== 'GET /api/triage'), []);
|
||||
assert.deepEqual(f.requests.filter(r => /triage/.test(r) && r !== 'GET /api/triage' && r !== SETTINGS_SCRIPT), []);
|
||||
assert.ok(f.requests.includes('GET /api/config'), 'the unscoped config read');
|
||||
// The same entry on the system dashboard.
|
||||
await f.b.evaluate('location.hash = "#/dashboard"');
|
||||
@@ -437,3 +440,33 @@ test('triage-followups: with no drawer open the first render still focuses the h
|
||||
const f = await setup(t);
|
||||
await f.wait('document.activeElement?.tagName === "H1"');
|
||||
});
|
||||
|
||||
test('triage-cap: one usage line from the read: not set, the count against the cap with the reset, not read, and markup as text', async t => {
|
||||
const f = await setup(t, { host: fakeHost({ usageToday: 'not set', cap: 'not set', resetsAt: 'not set' }) });
|
||||
await open(f);
|
||||
assert.equal(await f.text('#triage-usage'), 'Daily token cap: not set.');
|
||||
assert.equal(await f.b.evaluate('document.querySelector("#triage-usage").hidden'), false);
|
||||
const resetsAt = new Date(Date.now() + 5 * 3600000).toISOString();
|
||||
const local = await f.b.evaluate(`window.MosaicLocalTime(${JSON.stringify(resetsAt)})`);
|
||||
Object.assign(f.h, { usageToday: { count: 1234, input: 1200, output: 34, cacheWrite: 0, cacheRead: 0, unknownTurns: 2 }, cap: { limit: 200000, reset: '00:00', timeZone: 'UTC' }, resetsAt });
|
||||
await f.wait('/^Today:/.test(document.querySelector("#triage-usage").textContent)', 400);
|
||||
const line = await f.text('#triage-usage');
|
||||
const [count, limit] = await f.b.evaluate('[(1234).toLocaleString(), (200000).toLocaleString()]');
|
||||
assert.ok(line.startsWith(`Today: ${count} of ${limit} tokens. Resets at `), line);
|
||||
assert.ok(line.includes(local), line);
|
||||
assert.ok(line.endsWith(' 2 turns have no usage reading.'), line);
|
||||
Object.assign(f.h, { usageToday: 'not read', cap: { limit: 200000, reset: '00:00', timeZone: 'UTC' }, resetsAt: 'not read' });
|
||||
await f.wait('/not read/.test(document.querySelector("#triage-usage").textContent)', 400);
|
||||
assert.equal(await f.text('#triage-usage'), "Today's token usage: not read.");
|
||||
// A count that is not a whole number is not shown as one; markup stays text.
|
||||
Object.assign(f.h, { usageToday: { count: '<img src=x onerror=alert(1)>', unknownTurns: 0 }, cap: { limit: 1.5 }, resetsAt: '<b>x</b>' });
|
||||
await f.wait('/^Today: not read/.test(document.querySelector("#triage-usage").textContent)', 400);
|
||||
assert.equal(await f.text('#triage-usage'), 'Today: not read of not read tokens.');
|
||||
assert.equal(await f.b.evaluate('document.querySelector("#triage-usage").children.length'), 0);
|
||||
});
|
||||
|
||||
test('triage-cap: a read without the usage fields shows no usage line', async t => {
|
||||
const f = await setup(t);
|
||||
await open(f);
|
||||
assert.equal(await f.b.evaluate('document.querySelector("#triage-usage").hidden'), true);
|
||||
});
|
||||
@@ -5,15 +5,18 @@
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { connect } from "node:net";
|
||||
import { existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { appendFileSync, chmodSync, existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { dirname, join } from "node:path";
|
||||
import { Client } from "../../bus/src/client.mjs";
|
||||
import { REPO, RUNNER_FILES, checkHostedRoute, sandboxProblem } from "../../seat/src/sandbox.mjs";
|
||||
import { isProviderVariable } from "../../seat/src/session.mjs";
|
||||
import {
|
||||
NOT_STARTED, NOT_TAKEN, RELAY_REFUSED, ROUTE_ENV_ALLOW, ROUTE_REFUSED, SPAWN_ENV_ALLOW, UNANSWERED, createRelay, createTriageHost, hostPrompt,
|
||||
refusedHarnessName, triageEnv,
|
||||
USAGE_NOT_READ, overCapNotTaken, refusedHarnessName, triageEnv,
|
||||
} from "../src/triage-host.mjs";
|
||||
import { writeRoute } from "../src/triage-route-store.mjs";
|
||||
import { appendUsage, readUsage, usagePath } from "../src/triage-usage.mjs";
|
||||
import { triageView } from "../src/triage-http.mjs";
|
||||
|
||||
const FIXTURE = new URL("./fixtures/triage-adapter.mjs", import.meta.url).pathname;
|
||||
const skip = sandboxProblem() ?? false;
|
||||
@@ -114,7 +117,8 @@ const REFUSED_TRIED = [
|
||||
// The fake harness module: it declares the probe names and every refused one, so only the host's
|
||||
// own refusal stops those. Its one install root is <root>/install.
|
||||
const DECLARED = Object.freeze(["TRIAGE_PROBE", "FAKE_OK", "FAKE_TWO", ...REFUSED_TRIED]);
|
||||
const fakeModule = (roots, names = DECLARED) => async (name) => name === "fake" ? { envNames: names, installRoots: () => roots } : null;
|
||||
// Its transcript is <run>/session, where the stub adapter's SPEND writes (spendSource reads it).
|
||||
const fakeModule = (roots, names = DECLARED) => async (name) => name === "fake" ? { envNames: names, installRoots: () => roots, transcript: ({ runDir }) => join(runDir, "session") } : null;
|
||||
|
||||
function host(t, root, opts = {}) {
|
||||
const dataRoot = join(root, "data");
|
||||
@@ -674,3 +678,497 @@ test("with devOwner set, a card's bus start carries the dev-owner flag", { skip,
|
||||
const [reply] = await repliesOf(h, 1);
|
||||
assert.deepEqual(reply.cards, [{ step: "bus-start", org: "alpha", command: "mosaic bus start alpha --dev-local-owner --detach" }]);
|
||||
});
|
||||
|
||||
// #1873 triage-cap. The host counts each turn into <dataRoot>/triage/usage.jsonl and holds the stored
|
||||
// cap through daily-quota. The usage source here reads <run>/session/spend.jsonl, which the stub
|
||||
// adapter's SPEND writes (or a test writes by hand), one entry per spend.
|
||||
const spendSource = {
|
||||
readTurn: async ({ root }) => {
|
||||
let text;
|
||||
try { text = readFileSync(join(root, "spend.jsonl"), "utf8"); }
|
||||
catch (e) { if (e.code === "ENOENT") return { status: "known", entries: {} }; throw e; }
|
||||
const entries = {};
|
||||
for (const l of text.split("\n").filter(Boolean)) {
|
||||
const { id, n } = JSON.parse(l);
|
||||
entries[id] = { input: n, output: 0, cacheWrite: 0, cacheRead: 0 };
|
||||
}
|
||||
return { status: "known", entries };
|
||||
},
|
||||
};
|
||||
const DAY = "2026-11-18T00:00:00.000Z";
|
||||
// Store a cap the way the owner's verb does: the route store, its since at `since`.
|
||||
// The data root's triage directory, as the host's ensureDirs makes it (a test writes before any start).
|
||||
const triageDir = (dataRoot) => {
|
||||
mkdirSync(join(dataRoot, "triage"), { recursive: true, mode: 0o700 });
|
||||
chmodSync(dataRoot, 0o700);
|
||||
chmodSync(join(dataRoot, "triage"), 0o700);
|
||||
};
|
||||
const storeCap = (dataRoot, limit, timeZone = "UTC", since = DAY) => triageDir(dataRoot) ??
|
||||
writeRoute(dataRoot, { model: "glm-5", harness: "claude-code", limit, timeZone, by: "owner" }, { now: () => new Date(since) });
|
||||
const usageLine = (over = {}) => ({ at: "2026-11-18T10:00:00.000Z", run: "triage-20261118100000", turn: "owner-0123456789ab", status: "known", input: 10, output: 0, cacheRead: 0, cacheWrite: 0, ...over });
|
||||
const usageLines = (dataRoot) => readUsage(usagePath(dataRoot)).records;
|
||||
let spendId = 0;
|
||||
const spend = (dataRoot, n) => {
|
||||
mkdirSync(join(dataRoot, "triage", "agent", "session"), { recursive: true });
|
||||
appendFileSync(join(dataRoot, "triage", "agent", "session", "spend.jsonl"), JSON.stringify({ id: `s${++spendId}`, n }) + "\n");
|
||||
};
|
||||
const counting = (root) => { const r = { asked: 0, admit: async () => { r.asked++; return answer(root); } }; return r; };
|
||||
// A launcher that keeps the relay open and leaves the capability it was handed, so a test can make
|
||||
// the runner's calls itself without the sandbox.
|
||||
function standIn(root) {
|
||||
const f = join(root, "stand-in");
|
||||
writeFileSync(f, `#!/bin/sh\n: > '${join(root, "spawned")}'\ncat > '${join(root, "stdin.json")}'\nexec sleep 600\n`, { mode: 0o755 });
|
||||
return f;
|
||||
}
|
||||
async function runnerClient(root, dataRoot) {
|
||||
await waitFor(() => { try { return JSON.parse(readFileSync(join(root, "stdin.json"), "utf8")).cap; } catch { return null; } });
|
||||
const cap = JSON.parse(readFileSync(join(root, "stdin.json"), "utf8")).cap;
|
||||
return new Client({ path: join(dataRoot, "triage", "agent", "relay.sock"), cap });
|
||||
}
|
||||
|
||||
test("cap: a hit before a start refuses with the next reset, asks no route and spawns nothing; at 00:00 in the stored zone it starts", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
let clock = Date.parse("2026-11-18T12:00:00.000Z");
|
||||
const route = counting(root);
|
||||
const { h, dataRoot } = host(t, root, { route, bwrap: spy(root), usageSource: spendSource, now: () => new Date(clock) });
|
||||
storeCap(dataRoot, 100);
|
||||
appendUsage(usagePath(dataRoot), usageLine({ input: 60, output: 30, cacheRead: 5, cacheWrite: 5 }));
|
||||
await assert.rejects(h.message("hello"), (e) => e.code === "triage-over-cap" && e.until === "2026-11-19T00:00:00.000Z");
|
||||
assert.equal(route.asked, 0, "the route is not asked over the cap");
|
||||
assert.equal(existsSync(join(root, "spawned")), false);
|
||||
assert.deepEqual((await h.state()).transcript, [], "a refused message is never written");
|
||||
const s = await h.state();
|
||||
assert.deepEqual([s.overCapUntil, s.resetsAt], ["2026-11-19T00:00:00.000Z", "2026-11-19T00:00:00.000Z"]);
|
||||
assert.deepEqual(s.usageToday, { count: 100, input: 60, output: 30, cacheWrite: 5, cacheRead: 5, unknownTurns: 0 });
|
||||
assert.deepEqual(s.cap, { limit: 100, reset: "00:00", timeZone: "UTC" });
|
||||
clock = Date.parse("2026-11-18T23:59:59.999Z");
|
||||
await assert.rejects(h.message("hello"), (e) => e.code === "triage-over-cap");
|
||||
clock = Date.parse("2026-11-19T00:00:00.000Z");
|
||||
const s2 = await h.state();
|
||||
assert.deepEqual([s2.overCapUntil, s2.usageToday.count, s2.resetsAt], [null, 0, "2026-11-20T00:00:00.000Z"]);
|
||||
await h.message("hello");
|
||||
await waitFor(() => existsSync(join(root, "spawned")));
|
||||
assert.equal(route.asked, 1);
|
||||
});
|
||||
|
||||
test("cap: the reset is 00:00 in the stored zone on a DST day (America/Chicago, a 25-hour day)", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
let clock = Date.parse("2026-11-01T12:00:00.000Z");
|
||||
const route = counting(root);
|
||||
const { h, dataRoot } = host(t, root, { route, bwrap: spy(root), usageSource: spendSource, now: () => new Date(clock) });
|
||||
storeCap(dataRoot, 100, "America/Chicago", "2026-10-31T00:00:00.000Z");
|
||||
// 2026-11-01 in Chicago runs from 05:00Z (CDT) to 06:00Z the next day (CST).
|
||||
appendUsage(usagePath(dataRoot), usageLine({ at: "2026-11-01T05:00:00.000Z", input: 100 }));
|
||||
await assert.rejects(h.message("a"), (e) => e.code === "triage-over-cap" && e.until === "2026-11-02T06:00:00.000Z");
|
||||
clock = Date.parse("2026-11-02T05:59:59.999Z");
|
||||
await assert.rejects(h.message("b"), (e) => e.code === "triage-over-cap" && e.until === "2026-11-02T06:00:00.000Z");
|
||||
clock = Date.parse("2026-11-02T06:00:00.000Z");
|
||||
await h.message("c");
|
||||
await waitFor(() => existsSync(join(root, "spawned")));
|
||||
assert.equal(route.asked, 1);
|
||||
// A line from before the day began does not count: 04:59:59.999Z was still 2026-10-31 in Chicago.
|
||||
const other = scratch();
|
||||
const o = host(t, other, { route: counting(other), bwrap: spy(other), usageSource: spendSource, now: () => new Date("2026-11-01T12:00:00.000Z") });
|
||||
storeCap(o.dataRoot, 100, "America/Chicago", "2026-10-30T00:00:00.000Z");
|
||||
appendUsage(usagePath(o.dataRoot), usageLine({ at: "2026-11-01T04:59:59.999Z", input: 100 }));
|
||||
assert.equal((await o.h.state()).usageToday.count, 0);
|
||||
});
|
||||
|
||||
test("cap: the relay writes one line per turn, known or unknown, and a session that ends mid-turn still gets its line", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const logs = [];
|
||||
const { h, dataRoot } = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z"), log: (l) => logs.push(l) });
|
||||
const first = await h.message("one");
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
const run = JSON.parse(readFileSync(join(dataRoot, "triage", "agent", "session.json"), "utf8")).run;
|
||||
assert.match(run, /^triage-\d{14}$/);
|
||||
spend(dataRoot, 7); // before the turn: the baseline holds it
|
||||
assert.deepEqual(await c.call("usage.begin", { turn: first.id }), {});
|
||||
spend(dataRoot, 40);
|
||||
// A repeated begin for the same turn keeps the first baseline, so the tokens before it still count.
|
||||
assert.deepEqual(await c.call("usage.begin", { turn: first.id }), {});
|
||||
spend(dataRoot, 2);
|
||||
await c.call("usage.end", { turn: first.id });
|
||||
// A repeated end does not count the turn twice.
|
||||
await c.call("usage.end", { turn: first.id });
|
||||
await c.call("usage.begin", { turn: `${first.id}:retry` });
|
||||
await c.call("usage.end", { turn: `${first.id}:retry` }); // nothing new: unknown
|
||||
const lines = usageLines(dataRoot);
|
||||
assert.deepEqual(lines.map((l) => [l.run, l.turn, l.status, l.input, l.output, l.cacheRead, l.cacheWrite]), [
|
||||
[run, first.id, "known", 42, 0, 0, 0],
|
||||
[run, `${first.id}:retry`, "unknown", null, null, null, null],
|
||||
]);
|
||||
assert.ok(lines.every((l) => l.at === "2026-11-18T12:00:00.000Z"), "the line's time is the host's clock");
|
||||
assert.ok(logs.includes("triage host: usage.end repeated for a turn; not counted twice"));
|
||||
assert.ok(logs.includes("triage host: turn usage unknown: no-new-usage"));
|
||||
assert.equal(lstatSync(usagePath(dataRoot)).mode & 0o777, 0o600);
|
||||
// A turn begun and never ended: the session is killed; the host writes its line.
|
||||
await c.call("usage.begin", { turn: "owner-0123456789ff" });
|
||||
spend(dataRoot, 5);
|
||||
process.kill(-h.pid, "SIGKILL");
|
||||
await waitFor(() => !h.running);
|
||||
await h.stop();
|
||||
assert.deepEqual(usageLines(dataRoot).slice(2).map((l) => [l.turn, l.status, l.input]), [["owner-0123456789ff", "known", 5]]);
|
||||
assert.equal(usageLines(dataRoot).length, 3, "the killed turn has one line, not two");
|
||||
});
|
||||
|
||||
test("cap: a turn that crosses the cap finishes; the runner is then refused and every queued message gets a host line", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const logs = [];
|
||||
const route = counting(root);
|
||||
const { h, dataRoot } = host(t, root, { route, bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z"), log: (l) => logs.push(l) });
|
||||
storeCap(dataRoot, 100);
|
||||
const first = await h.message("one");
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
assert.deepEqual((await c.call("message.receive")).map((m) => m.id), [first.id]);
|
||||
await c.call("state");
|
||||
await c.call("usage.begin", { turn: first.id });
|
||||
// Queued while the turn runs, under the cap: admitted.
|
||||
const second = await h.message("two");
|
||||
const third = await h.message("three");
|
||||
spend(dataRoot, 150);
|
||||
// Inside the turn (a continuation's state read) the runner is not stopped.
|
||||
assert.equal((await c.call("state")).tasks.total, 0);
|
||||
await c.call("usage.end", { turn: first.id });
|
||||
await c.call("message.send", { to: "human", body: "answer", class: "RESULT", in_reply_to: first.id });
|
||||
await c.call("message.read", { id: first.id });
|
||||
// The next receive is refused: the session is gone, the queue drains with a host line each.
|
||||
await assert.rejects(c.call("message.receive"), (e) => e.code === "not-holder");
|
||||
const tr = (await h.state()).transcript;
|
||||
assert.deepEqual(tr.filter((e) => e.from === "triage").map((e) => [e.in_reply_to, e.text]), [[first.id, "answer"]]);
|
||||
const until = "2026-11-19T00:00:00.000Z";
|
||||
assert.deepEqual(tr.filter((e) => e.from === "host").map((e) => [e.in_reply_to, e.text]), [[second.id, overCapNotTaken(until)], [third.id, overCapNotTaken(until)]]);
|
||||
assert.ok(logs.includes("triage host: stopping the session at message.receive: over the daily token cap"), logs.join(" | "));
|
||||
await assert.rejects(h.message("four"), (e) => e.code === "triage-over-cap" && e.until === until);
|
||||
assert.equal(route.asked, 3);
|
||||
// The session ends: nothing is left to answer, so no second line.
|
||||
process.kill(-h.pid, "SIGKILL");
|
||||
await waitFor(() => !h.running);
|
||||
assert.equal((await h.state()).transcript.filter((e) => e.from === "host").length, 2);
|
||||
});
|
||||
|
||||
test("cap: messages taken together: once a turn crosses the cap, the next message's turn does not start", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const logs = [];
|
||||
const { h, dataRoot } = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z"), log: (l) => logs.push(l) });
|
||||
storeCap(dataRoot, 100);
|
||||
const [a, b] = await Promise.all([h.message("one"), h.message("two")]);
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
assert.deepEqual((await c.call("message.receive")).map((m) => m.id), [a.id, b.id]);
|
||||
await c.call("state");
|
||||
await c.call("usage.begin", { turn: a.id });
|
||||
spend(dataRoot, 100);
|
||||
await c.call("usage.end", { turn: a.id });
|
||||
await c.call("message.send", { to: "human", body: "answer", class: "RESULT", in_reply_to: a.id });
|
||||
await c.call("message.read", { id: a.id });
|
||||
// The runner reads the state before each message's turn: refused, so b's turn never begins.
|
||||
await assert.rejects(c.call("state"), (e) => e.code === "not-holder");
|
||||
const hostLines = (await h.state()).transcript.filter((e) => e.from === "host");
|
||||
assert.deepEqual(hostLines.map((e) => [e.in_reply_to, e.text]), [[b.id, overCapNotTaken("2026-11-19T00:00:00.000Z")]]);
|
||||
assert.ok(logs.includes("triage host: stopping the session at state: over the daily token cap"), logs.join(" | "));
|
||||
assert.equal(usageLines(dataRoot).length, 1);
|
||||
});
|
||||
|
||||
test("cap: a restart counts what the file holds: a new host on the same data root refuses at the same count", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const first = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
storeCap(first.dataRoot, 100);
|
||||
const m = await first.h.message("one");
|
||||
const c = await runnerClient(root, first.dataRoot);
|
||||
await c.call("usage.begin", { turn: m.id });
|
||||
spend(first.dataRoot, 60);
|
||||
await c.call("usage.end", { turn: m.id });
|
||||
// The second turn is open when the host stops: its line is written on the way down.
|
||||
await c.call("usage.begin", { turn: "owner-0123456789ee" });
|
||||
spend(first.dataRoot, 45);
|
||||
await first.h.stop();
|
||||
assert.deepEqual(usageLines(first.dataRoot).map((l) => [l.turn, l.input]), [[m.id, 60], ["owner-0123456789ee", 45]]);
|
||||
const route = counting(root);
|
||||
const second = host(t, root, { route, bwrap: spy(root), usageSource: spendSource, now: () => new Date("2026-11-18T13:00:00.000Z") });
|
||||
assert.equal((await second.h.state()).usageToday.count, 105);
|
||||
await assert.rejects(second.h.message("two"), (e) => e.code === "triage-over-cap" && e.until === "2026-11-19T00:00:00.000Z");
|
||||
assert.equal(route.asked, 0);
|
||||
});
|
||||
|
||||
test("cap: a message whose turn began and was not answered gets the unanswered line when the cap stops the session", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const { h, dataRoot } = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
storeCap(dataRoot, 100);
|
||||
const [a, b] = await Promise.all([h.message("one"), h.message("two")]);
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
await c.call("message.receive");
|
||||
await c.call("usage.begin", { turn: a.id });
|
||||
spend(dataRoot, 100);
|
||||
await c.call("usage.end", { turn: a.id });
|
||||
// The turn's answer is lost (a halted processing state leaves a message unread): a stays queued.
|
||||
await assert.rejects(c.call("state"), (e) => e.code === "not-holder");
|
||||
const lines = (await h.state()).transcript.filter((e) => e.from === "host").map((e) => [e.in_reply_to, e.text]);
|
||||
assert.deepEqual(lines, [[a.id, UNANSWERED], [b.id, overCapNotTaken("2026-11-19T00:00:00.000Z")]]);
|
||||
});
|
||||
|
||||
test("cap: under the sandbox, the runner's turns are counted, a turn that crosses the cap answers, the queued message drains, and a restart counts the file", { skip, timeout: 120000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const logs = [];
|
||||
const route = stubRoute(root);
|
||||
const clock = () => new Date("2026-11-18T12:00:00.000Z");
|
||||
const { h, dataRoot } = host(t, root, { route, usageSource: spendSource, now: clock, log: (l) => logs.push(l) });
|
||||
storeCap(dataRoot, 1000);
|
||||
const one = await h.message("SPEND 300");
|
||||
const echo = await h.message("plain");
|
||||
const replies = await repliesOf(h, 2);
|
||||
assert.deepEqual(replies.map((e) => [e.text.replace(/\n\n.*$/s, ""), e.in_reply_to]), [["spent 300", one.id], ["echo: plain", echo.id]]);
|
||||
const lines = usageLines(dataRoot);
|
||||
assert.deepEqual(lines.map((l) => [l.turn, l.status, l.input]), [[one.id, "known", 300], [echo.id, "unknown", null]]);
|
||||
assert.match(lines[0].run, /^triage-\d{14}$/);
|
||||
await h.stop();
|
||||
// A restart: a new host on the same data root counts the file.
|
||||
const again = host(t, root, { route, usageSource: spendSource, now: clock, log: (l) => logs.push(l) });
|
||||
const s = await again.h.state();
|
||||
assert.deepEqual([s.usageToday.count, s.usageToday.unknownTurns], [300, 1]);
|
||||
const big = await again.h.message("GATED-SPEND 800");
|
||||
await waitFor(() => existsSync(join(dataRoot, "triage", "agent", "work", "started")));
|
||||
const queued = await again.h.message("queued");
|
||||
writeFileSync(join(dataRoot, "triage", "agent", "work", "gate"), "x");
|
||||
const [answer] = await repliesOf(again.h, 3).then((l) => l.slice(2));
|
||||
assert.deepEqual([answer.text.replace(/\n\n.*$/s, ""), answer.in_reply_to], ["spent 800", big.id]);
|
||||
const [note] = await repliesOf(again.h, 1, "host");
|
||||
assert.deepEqual([note.in_reply_to, note.text], [queued.id, overCapNotTaken("2026-11-19T00:00:00.000Z")]);
|
||||
await waitFor(() => !again.h.running);
|
||||
assert.equal((await again.h.state()).lastExit.code, 22, "the runner ends as a session that is gone");
|
||||
assert.equal((await again.h.state()).transcript.filter((e) => e.from === "host").length, 1);
|
||||
await assert.rejects(again.h.message("after"), (e) => e.code === "triage-over-cap");
|
||||
assert.equal(usageLines(dataRoot).at(-1).input, 800);
|
||||
});
|
||||
|
||||
test("cap: a hit from another count before a queued message: the file is read again at each check", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const { h, dataRoot } = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
storeCap(dataRoot, 1000);
|
||||
const first = await h.message("one");
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
await c.call("message.receive");
|
||||
const second = await h.message("two");
|
||||
// Under the cap the runner takes the queued message.
|
||||
assert.deepEqual((await c.call("message.receive")).map((m) => m.id), [second.id]);
|
||||
await c.call("usage.begin", { turn: second.id });
|
||||
const third = await h.message("three");
|
||||
appendUsage(usagePath(dataRoot), usageLine({ input: 1000 }));
|
||||
// Over the cap inside second's turn: a continuation's state read is not stopped; the turn finishes.
|
||||
assert.equal((await c.call("state")).tasks.total, 0);
|
||||
spend(dataRoot, 4);
|
||||
await c.call("usage.end", { turn: second.id });
|
||||
await c.call("message.send", { to: "human", body: "answer", class: "RESULT", in_reply_to: second.id });
|
||||
await c.call("message.read", { id: second.id });
|
||||
await assert.rejects(c.call("message.receive"), (e) => e.code === "not-holder");
|
||||
const until = overCapNotTaken("2026-11-19T00:00:00.000Z");
|
||||
const hostLines = (await h.state()).transcript.filter((e) => e.from === "host").map((e) => [e.in_reply_to, e.text]);
|
||||
assert.deepEqual(hostLines.sort(), [[first.id, until], [third.id, until]].sort(), "every message still queued leaves it");
|
||||
assert.deepEqual(usageLines(dataRoot).at(-1).input, 4, "the finished turn is counted");
|
||||
});
|
||||
|
||||
test("cap: an idle session over the cap ends at its next receive, with nothing queued", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const { h, dataRoot } = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
storeCap(dataRoot, 1000);
|
||||
const m = await h.message("one");
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
await c.call("message.receive");
|
||||
await c.call("message.send", { to: "human", body: "answer", class: "RESULT", in_reply_to: m.id });
|
||||
await c.call("message.read", { id: m.id });
|
||||
appendUsage(usagePath(dataRoot), usageLine({ input: 1000 }));
|
||||
await assert.rejects(c.call("message.receive"), (e) => e.code === "not-holder");
|
||||
assert.equal((await h.state()).transcript.filter((e) => e.from === "host").length, 0, "nothing was queued, so no host line");
|
||||
});
|
||||
|
||||
test("cap: the count waits for a turn still being counted: usage.end answers after its line, and a message right after a mid-turn kill is refused", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
let gate = null; // while set, the usage source waits on it
|
||||
const close = () => { let release; const promise = new Promise((r) => { release = r; }); gate = { promise, release: () => { gate = null; release(); } }; return gate; };
|
||||
const gated = { readTurn: async (a) => { if (gate) await gate.promise; return spendSource.readTurn(a); } };
|
||||
const route = counting(root);
|
||||
// A failed assertion must not leave the gate shut: the host's stop waits for the usage chain. After hooks
|
||||
// run in the order they were added, so this one goes before the host's stop.
|
||||
t.after(() => gate?.release());
|
||||
const { h, dataRoot } = host(t, root, { route, bwrap: standIn(root), usageSource: gated, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
storeCap(dataRoot, 100);
|
||||
const m = await h.message("one");
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
await c.call("message.receive");
|
||||
await c.call("usage.begin", { turn: m.id });
|
||||
spend(dataRoot, 60);
|
||||
let g = close();
|
||||
let answered = false;
|
||||
const end = c.call("usage.end", { turn: m.id }).then(() => { answered = true; });
|
||||
await new Promise((r) => setTimeout(r, 200));
|
||||
assert.equal(answered, false, "usage.end is not answered before its line is written");
|
||||
assert.equal(usageLines(dataRoot).length, 0);
|
||||
g.release();
|
||||
await end;
|
||||
assert.deepEqual(usageLines(dataRoot).map((l) => l.input), [60]);
|
||||
await c.call("message.send", { to: "human", body: "answer", class: "RESULT", in_reply_to: m.id });
|
||||
await c.call("message.read", { id: m.id });
|
||||
// A second turn crosses the cap and the session is killed before its usage.end.
|
||||
const two = await h.message("two");
|
||||
assert.deepEqual((await c.call("message.receive")).map((x) => x.id), [two.id]);
|
||||
await c.call("usage.begin", { turn: two.id });
|
||||
spend(dataRoot, 50);
|
||||
g = close();
|
||||
process.kill(-h.pid, "SIGKILL");
|
||||
await waitFor(() => !h.running);
|
||||
const asked = route.asked;
|
||||
setTimeout(() => g.release(), 200);
|
||||
await assert.rejects(h.message("three"), (e) => e.code === "triage-over-cap" && e.until === "2026-11-19T00:00:00.000Z");
|
||||
assert.equal(route.asked, asked, "the route is not asked: the open turn was counted first");
|
||||
assert.deepEqual(usageLines(dataRoot).map((l) => [l.turn, l.input]), [[m.id, 60], [two.id, 50]]);
|
||||
});
|
||||
|
||||
test("cap: a harness transcript path outside the run directory reads as no transcript, so the turn's line is unknown", { timeout: 30000 }, async (t) => {
|
||||
for (const label of ["a sibling of the run directory", "a path through .. from inside it"]) {
|
||||
const root = scratch();
|
||||
const logs = [];
|
||||
let target = null;
|
||||
const mod = async (name) => name === "fake" ? { envNames: DECLARED, installRoots: () => [join(root, "install")], transcript: ({ runDir }) => (target = label.startsWith("a sibling") ? join(runDir, "..", "outside") : `${runDir}/session/../../outside`) } : null;
|
||||
const { h, dataRoot } = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, harnessModule: mod, now: () => new Date("2026-11-18T12:00:00.000Z"), log: (l) => logs.push(l) });
|
||||
const m = await h.message("one");
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
mkdirSync(target, { recursive: true });
|
||||
await c.call("usage.begin", { turn: m.id });
|
||||
appendFileSync(join(target, "spend.jsonl"), JSON.stringify({ id: "x1", n: 5 }) + "\n");
|
||||
await c.call("usage.end", { turn: m.id });
|
||||
assert.deepEqual(usageLines(dataRoot).map((l) => [l.status, l.input]), [["unknown", null]], label);
|
||||
assert.ok(logs.includes("triage host: turn usage unknown: baseline-unknown"), label);
|
||||
}
|
||||
});
|
||||
|
||||
test("cap: a torn last line is refused, logged once, and counts as an unknown turn; the next line stands alone", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const logs = [];
|
||||
const { h, dataRoot } = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z"), log: (l) => logs.push(l) });
|
||||
storeCap(dataRoot, 1000);
|
||||
appendUsage(usagePath(dataRoot), usageLine({ input: 30 }));
|
||||
appendFileSync(usagePath(dataRoot), '{"at":"2026-11-18T11:00:00.000Z","run":"triage-2026111');
|
||||
const s = await h.state();
|
||||
assert.deepEqual([s.usageToday.count, s.usageToday.unknownTurns], [30, 1]);
|
||||
await h.state();
|
||||
assert.deepEqual(logs.filter((l) => /usage\.jsonl line/.test(l)), ["triage host: usage.jsonl line 2 is not a usage line; it counts as an unknown turn"]);
|
||||
// Under the cap, a message starts, and its turn's line is written after the torn one.
|
||||
const m = await h.message("one");
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
await c.call("usage.begin", { turn: m.id });
|
||||
spend(dataRoot, 20);
|
||||
await c.call("usage.end", { turn: m.id });
|
||||
const lines = usageLines(dataRoot);
|
||||
assert.deepEqual(lines.map((l) => [l.turn, l.status, l.input]), [["owner-0123456789ab", "known", 30], ["line-2", "unknown", null], [m.id, "known", 20]]);
|
||||
const s2 = await h.state();
|
||||
assert.deepEqual([s2.usageToday.count, s2.usageToday.unknownTurns], [50, 1]);
|
||||
});
|
||||
|
||||
test("cap: a usage file with a wrong mode, a link or a non-file refuses every message and reads as not read", { timeout: 30000 }, async (t) => {
|
||||
const cases = [
|
||||
["mode 0644", (f) => { appendUsage(f, usageLine()); chmodSync(f, 0o644); }],
|
||||
["mode 0400", (f) => { appendUsage(f, usageLine()); chmodSync(f, 0o400); }],
|
||||
["a link", (f) => { writeFileSync(f + ".real", JSON.stringify(usageLine()) + "\n", { mode: 0o600 }); symlinkSync(f + ".real", f); }],
|
||||
["a directory", (f) => mkdirSync(f)],
|
||||
];
|
||||
for (const [label, make] of cases) {
|
||||
const root = scratch();
|
||||
const route = counting(root);
|
||||
const { h, dataRoot } = host(t, root, { route, bwrap: spy(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
storeCap(dataRoot, 1000);
|
||||
make(usagePath(dataRoot));
|
||||
await assert.rejects(h.message("hello"), (e) => e.code === "triage-usage-unreadable" && e.message === "Triage's token usage could not be read", label);
|
||||
assert.equal(route.asked, 0, label);
|
||||
assert.equal(existsSync(join(root, "spawned")), false, label);
|
||||
const s = await h.state();
|
||||
assert.deepEqual([s.usageToday, s.cap, s.resetsAt], ["not read", { limit: 1000, reset: "00:00", timeZone: "UTC" }, "2026-11-19T00:00:00.000Z"], label);
|
||||
}
|
||||
});
|
||||
|
||||
test("cap: a line the host cannot write leaves the count short, so the stored cap refuses from then on", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const logs = [];
|
||||
const { h, dataRoot } = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z"), log: (l) => logs.push(l) });
|
||||
storeCap(dataRoot, 1000);
|
||||
const m = await h.message("one");
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
await c.call("message.receive");
|
||||
await c.call("usage.begin", { turn: m.id });
|
||||
spend(dataRoot, 5);
|
||||
appendUsage(usagePath(dataRoot), usageLine({ input: 1 }));
|
||||
chmodSync(usagePath(dataRoot), 0o644);
|
||||
await c.call("usage.end", { turn: m.id });
|
||||
assert.ok(logs.includes("triage host: usage line not written: triage-usage-unreadable"), logs.join(" | "));
|
||||
chmodSync(usagePath(dataRoot), 0o600);
|
||||
assert.equal(usageLines(dataRoot).length, 1, "the file is as it was");
|
||||
// The file reads again, but a turn is missing from it: refused, and the state says not read.
|
||||
await assert.rejects(h.message("two"), (e) => e.code === "triage-usage-unreadable");
|
||||
assert.equal((await h.state()).usageToday, "not read");
|
||||
await c.call("message.send", { to: "human", body: "answer", class: "RESULT", in_reply_to: m.id });
|
||||
const later = await h.state();
|
||||
assert.equal(later.transcript.filter((e) => e.from === "host").length, 0);
|
||||
});
|
||||
|
||||
test("cap: with no route stored the fields say not set, nothing is refused for usage, and each turn's line is still written", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const { h, dataRoot } = host(t, root, { route: counting(root), bwrap: standIn(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
const s = await h.state();
|
||||
assert.deepEqual([s.usageToday, s.cap, s.resetsAt, s.overCapUntil], ["not set", "not set", "not set", null]);
|
||||
triageDir(dataRoot);
|
||||
// A usage file over any count, even a broken one, refuses nothing while no cap is stored.
|
||||
appendUsage(usagePath(dataRoot), usageLine({ input: 10 ** 9 }));
|
||||
appendFileSync(usagePath(dataRoot), "not json");
|
||||
const m = await h.message("one");
|
||||
const c = await runnerClient(root, dataRoot);
|
||||
assert.deepEqual((await c.call("message.receive")).map((x) => x.id), [m.id]);
|
||||
await c.call("state");
|
||||
await c.call("usage.begin", { turn: m.id });
|
||||
spend(dataRoot, 3);
|
||||
await c.call("usage.end", { turn: m.id });
|
||||
assert.deepEqual(usageLines(dataRoot).at(-1).input, 3);
|
||||
const two = await h.message("two");
|
||||
assert.deepEqual((await c.call("message.receive")).map((x) => x.id), [two.id]);
|
||||
});
|
||||
|
||||
test("cap: an unreadable route store refuses every message and reads as not read", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const route = counting(root);
|
||||
const { h, dataRoot } = host(t, root, { route, bwrap: spy(root), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
storeCap(dataRoot, 1000);
|
||||
chmodSync(join(dataRoot, "triage", "route.json"), 0o644);
|
||||
await assert.rejects(h.message("hello"), (e) => e.code === "triage-route-unreadable" && e.message === "the model route could not be read");
|
||||
assert.equal(route.asked, 0);
|
||||
const s = await h.state();
|
||||
assert.deepEqual([s.usageToday, s.cap, s.resetsAt], ["not read", "not read", "not read"]);
|
||||
// A running session: the store turns unreadable, the next receive is refused and the queue drains.
|
||||
const other = scratch();
|
||||
const r = host(t, other, { route: counting(other), bwrap: standIn(other), usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
storeCap(r.dataRoot, 1000);
|
||||
const m = await r.h.message("one");
|
||||
const c = await runnerClient(other, r.dataRoot);
|
||||
await c.call("message.receive");
|
||||
const two = await r.h.message("two");
|
||||
chmodSync(join(r.dataRoot, "triage", "route.json"), 0o644);
|
||||
await assert.rejects(c.call("message.receive"), (e) => e.code === "not-holder");
|
||||
const hostLines = (await r.h.state()).transcript.filter((e) => e.from === "host").map((e) => [e.in_reply_to, e.text]);
|
||||
assert.deepEqual(hostLines.sort(), [[m.id, USAGE_NOT_READ], [two.id, USAGE_NOT_READ]].sort());
|
||||
});
|
||||
|
||||
test("cap: the read carries usageToday, cap and resetsAt, with and without a stored route, and no path, key or env value", { timeout: 30000 }, async (t) => {
|
||||
const root = scratch();
|
||||
const env = { ...CONSOLE_ENV, ANTHROPIC_AUTH_TOKEN: "tok-secret-value" };
|
||||
const route = { admit: async () => answer(root, { env: { ANTHROPIC_BASE_URL: "http://127.0.0.1:9", ANTHROPIC_AUTH_TOKEN: "tok-secret-value" } }) };
|
||||
const { h, dataRoot } = host(t, root, { route, bwrap: spy(root), env, usageSource: spendSource, now: () => new Date("2026-11-18T12:00:00.000Z") });
|
||||
const unset = await triageView(h, { send: true });
|
||||
assert.deepEqual([unset.usageToday, unset.cap, unset.resetsAt], ["not set", "not set", "not set"]);
|
||||
storeCap(dataRoot, 500, "Asia/Kolkata", "2026-11-17T00:00:00.000Z");
|
||||
appendUsage(usagePath(dataRoot), usageLine({ at: "2026-11-18T11:00:00.000Z", input: 120, output: 3 }));
|
||||
await h.message("hello");
|
||||
const view = await triageView(h, { send: true });
|
||||
assert.deepEqual(view.usageToday, { count: 123, input: 120, output: 3, cacheWrite: 0, cacheRead: 0, unknownTurns: 0 });
|
||||
assert.deepEqual(view.cap, { limit: 500, reset: "00:00", timeZone: "Asia/Kolkata" });
|
||||
assert.equal(view.resetsAt, "2026-11-18T18:30:00.000Z");
|
||||
const text = JSON.stringify([unset, view]);
|
||||
for (const bad of [dataRoot, root, "usage.jsonl", "route.json", "glm-5", "claude-code", "tok-secret-value", "from-the-console", "since", "2026-11-17T00:00:00.000Z", "triage-2026"]) {
|
||||
assert.equal(text.includes(bad), false, bad);
|
||||
}
|
||||
});
|
||||
@@ -4,7 +4,7 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { request } from 'node:http';
|
||||
import { mkdtempSync, existsSync, readFileSync } from 'node:fs';
|
||||
import { mkdtempSync, mkdirSync, existsSync, readFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { startServer } from '../src/serve.mjs';
|
||||
import { createDevOwner } from '../src/dev-owner.mjs';
|
||||
@@ -12,6 +12,8 @@ import { consoleAccess } from '../src/access.mjs';
|
||||
import { createSsoSession } from './helpers/sso-session.mjs';
|
||||
import { NO_ROUTE_MESSAGE, capMessage, triageView, triageHTTP, SNAPSHOT_MAX_AGE_MS } from '../src/triage-http.mjs';
|
||||
import { createTriageHost, TriageError } from '../src/triage-host.mjs';
|
||||
import { writeRoute } from '../src/triage-route-store.mjs';
|
||||
import { appendUsage, usagePath } from '../src/triage-usage.mjs';
|
||||
|
||||
const LEAK = '/home/owner/.config/mosaic-dev/secret-token-0123456789';
|
||||
|
||||
@@ -239,6 +241,21 @@ test('with the real host and no route: the read says so in plain words, the writ
|
||||
assert.equal(JSON.parse(readFileSync(join(host.paths.health, 'snapshot.json'), 'utf8')).snapshotVersion, 1, 'the read wrote the snapshot');
|
||||
});
|
||||
|
||||
test('triage-cap: the read carries usageToday, cap and resetsAt from the real host, and no path, model or harness', async t => {
|
||||
const dataRoot = join(mkdtempSync(join(process.env.TMPDIR, 'th-')), 'data');
|
||||
mkdirSync(join(dataRoot, 'triage'), { recursive: true, mode: 0o700 });
|
||||
writeRoute(dataRoot, { model: 'glm-5', harness: 'claude-code', limit: 1000, timeZone: 'UTC', by: 'owner' }, { now: () => new Date('2026-11-18T00:00:00.000Z') });
|
||||
appendUsage(usagePath(dataRoot), { at: '2026-11-18T09:00:00.000Z', run: 'triage-20261118090000', turn: 'owner-0123456789ab', status: 'known', input: 30, output: 5, cacheRead: 2, cacheWrite: 1 });
|
||||
const host = createTriageHost({ dataRoot, readSnapshot: async () => null, now: () => new Date('2026-11-18T10:00:00.000Z'), bwrap: '/nonexistent/bwrap' });
|
||||
const f = await devServer(t, { triage: host });
|
||||
const read = await send(f.port, '/api/triage', { headers: { host: f.headers.host } });
|
||||
assert.equal(read.status, 200);
|
||||
assert.deepEqual(read.json.usageToday, { count: 38, input: 30, output: 5, cacheWrite: 1, cacheRead: 2, unknownTurns: 0 });
|
||||
assert.deepEqual(read.json.cap, { limit: 1000, reset: '00:00', timeZone: 'UTC' });
|
||||
assert.equal(read.json.resetsAt, '2026-11-19T00:00:00.000Z');
|
||||
for (const leak of [dataRoot, 'glm-5', 'claude-code', 'usage.jsonl', '"since"']) assert.equal(read.text.includes(leak), false, leak);
|
||||
});
|
||||
|
||||
test('the route module keeps its own write checks when called alone: --send, one exact Origin, one same-origin Fetch Metadata', async () => {
|
||||
const triage = fakeTriage();
|
||||
const call = async (headers, { send = true } = {}) => {
|
||||
|
||||
@@ -0,0 +1,393 @@
|
||||
// #1873 triage-settings: GET and POST /api/triage/route over the real HTTP boundary. Only the instance
|
||||
// owner reaches them, a write is same-origin, the model and harness are checked as an organization's
|
||||
// route change is, the store does the write (one receipt), and no key or path reaches a response.
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { request } from 'node:http';
|
||||
import { mkdtempSync, readFileSync, rmSync, writeFileSync, existsSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { startServer } from '../src/serve.mjs';
|
||||
import { createDevOwner } from '../src/dev-owner.mjs';
|
||||
import { consoleAccess } from '../src/access.mjs';
|
||||
import { createSsoSession } from './helpers/sso-session.mjs';
|
||||
import { routePaths, readRoute, writeRoute } from '../src/triage-route-store.mjs';
|
||||
import { triageRouteRoutes, mergeCatalogues, installedHarnesses, ownerActor, TRIAGE_ROUTE_PATH, NOT_STARTED_NOTE } from '../src/triage-route-http.mjs';
|
||||
|
||||
const SECRET = 'sk-triage-route-sentinel-0123456789abcdef';
|
||||
const T = '2026-11-18T12:00:00.000Z';
|
||||
const model = (key, extra = {}) => ({ key, module: 'anthropic', upstream: 'cliproxy', provider: 'anthropic', model: `claude-${key}`, label: key.toUpperCase(), available: true, reason: null, harnesses: ['claude-code', 'pi'], ...extra });
|
||||
// One catalogue as an organization's `routes` verb answers. A leaked field would carry the sentinel.
|
||||
const READY = () => ({ status: 'ready', observedAt: T, notices: [`key ${SECRET} at /srv/proxy/keys.json`], models: [
|
||||
model('sonnet', { reason: `${SECRET} /home/owner/.config/mosaic-dev/proxy.json` }),
|
||||
model('glm-flash', { module: 'opencode', provider: 'zai', model: 'glm-5.3-flash', harnesses: ['pi'] }),
|
||||
model('down', { available: false, reason: 'proxy says no' }),
|
||||
] });
|
||||
const BODY = { model: 'sonnet', harness: 'claude-code', cap: 200000, timeZone: 'Europe/Berlin', revision: 0 };
|
||||
|
||||
const send = (port, path, { method = 'GET', headers = {}, body } = {}) => new Promise((resolve, reject) => {
|
||||
const req = request({ host: '127.0.0.1', port, path, method, headers }, res => {
|
||||
let text = ''; res.on('data', c => { text += c; }); res.on('end', () => resolve({ status: res.statusCode, text, json: (() => { try { return JSON.parse(text); } catch { return null; } })() }));
|
||||
});
|
||||
req.on('error', reject); req.end(body === undefined ? undefined : typeof body === 'string' ? body : JSON.stringify(body));
|
||||
});
|
||||
const receipts = root => { const f = routePaths(root).receipts; return existsSync(f) ? readFileSync(f, 'utf8').split('\n').filter(Boolean).map(l => JSON.parse(l)) : []; };
|
||||
|
||||
async function devServer(t, { catalogues = async () => [READY()], harnesses = () => ['claude-code', 'pi'], ...options } = {}) {
|
||||
const root = mkdtempSync(join(tmpdir(), 'tr-http-'));
|
||||
t.after(() => rmSync(root, { recursive: true, force: true }));
|
||||
const logs = [], routeLogs = [];
|
||||
const s = await startServer({ port: 0, board: 'http://127.0.0.1:9', sso: createDevOwner(), send: async () => ({}), log: line => logs.push(line),
|
||||
routes: [triageRouteRoutes({ dataRoot: root, catalogues, harnesses, log: line => routeLogs.push(line) })], ...options });
|
||||
t.after(() => new Promise(r => s.close(r)));
|
||||
const port = s.address().port;
|
||||
const origin = `http://127.0.0.1:${port}`;
|
||||
const read = { host: `127.0.0.1:${port}`, 'sec-fetch-site': 'same-origin' };
|
||||
const write = { ...read, origin, 'content-type': 'application/json' };
|
||||
const get = (headers = read) => send(port, TRIAGE_ROUTE_PATH, { headers });
|
||||
const post = (body = BODY, headers = write) => send(port, TRIAGE_ROUTE_PATH, { method: 'POST', headers, body });
|
||||
return { s, port, root, logs, routeLogs, read, write, get, post };
|
||||
}
|
||||
|
||||
test('GET without a stored route says it is not set and offers the catalogue models with their harnesses', async t => {
|
||||
const f = await devServer(t);
|
||||
const r = await f.get();
|
||||
assert.equal(r.status, 200);
|
||||
assert.equal(r.json.route, null);
|
||||
assert.equal(r.json.routeMessage, 'No Triage route is set.');
|
||||
assert.equal(r.json.revision, 0);
|
||||
assert.equal(r.json.send, true);
|
||||
assert.equal(r.json.note, NOT_STARTED_NOTE);
|
||||
assert.deepEqual(r.json.choices, { status: 'ready', models: [
|
||||
{ key: 'down', label: 'DOWN', available: false, harnesses: ['claude-code', 'pi'] },
|
||||
{ key: 'glm-flash', label: 'GLM-FLASH', available: true, harnesses: ['pi'] },
|
||||
{ key: 'sonnet', label: 'SONNET', available: true, harnesses: ['claude-code', 'pi'] },
|
||||
] });
|
||||
assert.equal(r.text.includes(SECRET), false);
|
||||
assert.equal(r.text.includes(f.root), false);
|
||||
assert.equal(r.text.includes('/srv/') || r.text.includes('/home/'), false);
|
||||
});
|
||||
|
||||
test('GET with a stored route returns its model, harness, cap, reset, zone, revision and time, and no path or actor detail', async t => {
|
||||
const f = await devServer(t);
|
||||
writeRoute(f.root, { model: 'sonnet', harness: 'pi', limit: 5000, timeZone: 'America/Chicago', by: 'local-dev-owner' }, { now: () => new Date(T) });
|
||||
const r = await f.get();
|
||||
assert.equal(r.status, 200);
|
||||
assert.deepEqual(r.json.route, { model: 'sonnet', harness: 'pi', cap: 5000, reset: '00:00', timeZone: 'America/Chicago', revision: 1, updatedAt: T });
|
||||
assert.equal(r.json.revision, 1);
|
||||
assert.equal(Object.hasOwn(r.json, 'routeMessage'), false);
|
||||
assert.equal(r.text.includes(f.root), false);
|
||||
});
|
||||
|
||||
test('POST writes through the store: one receipt, the signed-in owner as updatedBy, the revision moves', async t => {
|
||||
const f = await devServer(t);
|
||||
const r = await f.post();
|
||||
assert.equal(r.status, 200, r.text);
|
||||
assert.equal(r.json.route.model, 'sonnet');
|
||||
assert.equal(r.json.route.harness, 'claude-code');
|
||||
assert.equal(r.json.route.cap, 200000);
|
||||
assert.equal(r.json.route.timeZone, 'Europe/Berlin');
|
||||
assert.equal(r.json.route.reset, '00:00');
|
||||
assert.equal(r.json.revision, 1);
|
||||
assert.equal(r.json.note, NOT_STARTED_NOTE);
|
||||
const doc = readRoute(f.root);
|
||||
assert.equal(doc.updatedBy, 'local-dev-owner');
|
||||
assert.equal(doc.cap.limit, 200000);
|
||||
const lines = receipts(f.root);
|
||||
assert.equal(lines.length, 1);
|
||||
assert.equal(lines[0].by, 'local-dev-owner');
|
||||
assert.equal(lines[0].revision, 1);
|
||||
const again = await f.post({ ...BODY, harness: 'pi', revision: 1 });
|
||||
assert.equal(again.status, 200);
|
||||
assert.equal(receipts(f.root).length, 2);
|
||||
for (const line of f.routeLogs) assert.match(line, /^\S+ (GET|POST) \/api\/triage\/route \d{3}$/);
|
||||
});
|
||||
|
||||
test('a stale revision answers 409 with the stored revision and writes nothing', async t => {
|
||||
const f = await devServer(t);
|
||||
assert.equal((await f.post()).status, 200);
|
||||
const stale = await f.post({ ...BODY, cap: 9 });
|
||||
assert.equal(stale.status, 409);
|
||||
assert.equal(stale.json.error, 'triage-route-stale');
|
||||
assert.equal(stale.json.revision, 1);
|
||||
assert.match(stale.json.message, /changed since this page read it/);
|
||||
assert.equal(readRoute(f.root).cap.limit, 200000);
|
||||
assert.equal(receipts(f.root).length, 1);
|
||||
});
|
||||
|
||||
test('a model the catalogue does not offer, or one it marks unavailable, is refused and nothing is written', async t => {
|
||||
const f = await devServer(t);
|
||||
const unknown = await f.post({ ...BODY, model: 'opus' });
|
||||
assert.equal(unknown.status, 400);
|
||||
assert.equal(unknown.json.error, 'unknown-model');
|
||||
const down = await f.post({ ...BODY, model: 'down' });
|
||||
assert.equal(down.status, 503);
|
||||
assert.equal(down.json.error, 'model-provider-unavailable');
|
||||
for (const bad of [7, ['sonnet'], { key: 'sonnet' }, 'BAD KEY', '']) {
|
||||
const r = await f.post({ ...BODY, model: bad });
|
||||
assert.equal(r.status, 400, JSON.stringify(bad));
|
||||
assert.equal(r.json.error, 'unknown-model', JSON.stringify(bad));
|
||||
}
|
||||
assert.equal(readRoute(f.root), null);
|
||||
assert.deepEqual(receipts(f.root), []);
|
||||
});
|
||||
|
||||
test('a harness the model is not compatible with, or one not installed, is refused', async t => {
|
||||
const f = await devServer(t);
|
||||
const wrong = await f.post({ ...BODY, model: 'glm-flash', harness: 'claude-code' });
|
||||
assert.equal(wrong.status, 400);
|
||||
assert.equal(wrong.json.error, 'model-harness-incompatible');
|
||||
// Installed here is only pi: the catalogue's claude-code is not offered and not accepted.
|
||||
const g = await devServer(t, { harnesses: () => ['pi'] });
|
||||
assert.deepEqual((await g.get()).json.choices.models.find(m => m.key === 'sonnet').harnesses, ['pi']);
|
||||
const off = await g.post({ ...BODY, harness: 'claude-code' });
|
||||
assert.equal(off.status, 400);
|
||||
assert.ok(['model-harness-incompatible', 'harness-off'].includes(off.json.error), off.text);
|
||||
const bad = await f.post({ ...BODY, harness: 'Not A Name' });
|
||||
assert.equal(bad.json.error, 'harness-off');
|
||||
assert.equal(readRoute(f.root), null);
|
||||
assert.equal(readRoute(g.root), null);
|
||||
});
|
||||
|
||||
test('the installed harness check is the organization one: a compatible harness no module provides is harness-off', async t => {
|
||||
const catalogue = READY(); catalogue.models[0].harnesses = ['claude-code', 'pi', 'ghost'];
|
||||
// harnesses() lists ghost so the merged catalogue keeps it; the check gets the installed list without it.
|
||||
let calls = 0;
|
||||
const f = await devServer(t, { catalogues: async () => [catalogue], harnesses: () => (++calls % 2 === 1 ? ['claude-code', 'pi', 'ghost'] : ['claude-code', 'pi']) });
|
||||
const r = await f.post({ ...BODY, harness: 'ghost' });
|
||||
assert.equal(r.json.error, 'harness-off', r.text);
|
||||
});
|
||||
|
||||
test('a catalogue that is not ready, or no organization bus answering, refuses the write and shows no choices', async t => {
|
||||
for (const catalogues of [async () => [{ status: 'starting', models: [] }], async () => [{ status: 'bus-unavailable' }], async () => [], async () => { throw new Error(SECRET); }]) {
|
||||
const f = await devServer(t, { catalogues });
|
||||
const r = await f.get();
|
||||
assert.equal(r.status, 200);
|
||||
assert.deepEqual(r.json.choices.models, []);
|
||||
assert.notEqual(r.json.choices.status, 'ready');
|
||||
const w = await f.post();
|
||||
assert.equal(w.status, 503);
|
||||
assert.equal(w.json.error, 'model-provider-unavailable');
|
||||
assert.equal(w.text.includes(SECRET) || r.text.includes(SECRET), false);
|
||||
assert.equal(readRoute(f.root), null);
|
||||
}
|
||||
});
|
||||
|
||||
test('a missing or unknown time zone is refused; there is no default', async t => {
|
||||
const f = await devServer(t);
|
||||
const { timeZone, ...noZone } = BODY;
|
||||
for (const body of [noZone, { ...BODY, timeZone: '' }, { ...BODY, timeZone: null }]) {
|
||||
const r = await f.post(body);
|
||||
assert.equal(r.status, 400, JSON.stringify(body));
|
||||
assert.equal(r.json.error, 'triage-zone-required', JSON.stringify(body));
|
||||
}
|
||||
for (const zone of ['Mars/Olympus', 'not a zone', 42]) {
|
||||
const r = await f.post({ ...BODY, timeZone: zone });
|
||||
assert.equal(r.status, 400, String(zone));
|
||||
assert.equal(r.json.error, 'triage-zone-unknown', String(zone));
|
||||
}
|
||||
assert.equal(readRoute(f.root), null);
|
||||
const lower = await f.post({ ...BODY, timeZone: 'america/chicago' });
|
||||
assert.equal(lower.status, 200, JSON.stringify(lower.json));
|
||||
assert.equal(readRoute(f.root).cap.timeZone, 'America/Chicago', 'a zone in another case is stored in its canonical spelling');
|
||||
assert.equal(lower.json.route.timeZone, 'America/Chicago');
|
||||
});
|
||||
|
||||
test('a cap that is not a positive whole number is refused', async t => {
|
||||
const f = await devServer(t);
|
||||
for (const cap of [0, -1, 1.5, '1000', null, undefined, Number.MAX_SAFE_INTEGER + 1, true]) {
|
||||
const body = { ...BODY, cap }; if (cap === undefined) delete body.cap;
|
||||
const r = await f.post(body);
|
||||
assert.equal(r.status, 400, String(cap));
|
||||
assert.equal(r.json.error, 'triage-cap-invalid', String(cap));
|
||||
}
|
||||
assert.equal((await f.post({ ...BODY, cap: 1 })).status, 200, 'one token is a cap');
|
||||
});
|
||||
|
||||
test('a body with another key, a bad revision, a wrong type or too many bytes is invalid-request', async t => {
|
||||
const f = await devServer(t);
|
||||
for (const body of [{ ...BODY, by: 'mallory' }, { ...BODY, revision: -1 }, { ...BODY, revision: '0' }, [BODY], 'null', '{', JSON.stringify({ ...BODY, model: 'a'.repeat(17000) })]) {
|
||||
const r = await f.post(body);
|
||||
assert.equal(r.status, 400, typeof body === 'string' ? body.slice(0, 20) : JSON.stringify(body).slice(0, 60));
|
||||
assert.equal(r.json.error, 'invalid-request');
|
||||
}
|
||||
const { revision, ...noRevision } = BODY;
|
||||
assert.equal((await f.post(noRevision)).json.error, 'invalid-request', 'the revision is required');
|
||||
const text = await f.post(BODY, { ...f.write, 'content-type': 'text/plain' });
|
||||
assert.equal(text.json.error, 'invalid-request');
|
||||
assert.equal(readRoute(f.root), null);
|
||||
});
|
||||
|
||||
test('both routes refuse a cross-origin request; a read may omit the metadata; a write needs both checks exactly once', async t => {
|
||||
const f = await devServer(t);
|
||||
// Read: a foreign Origin or cross-site Fetch Metadata is refused. Missing metadata is the repo GET convention.
|
||||
const evil = await f.get({ ...f.read, origin: 'http://evil.example' });
|
||||
assert.equal(evil.status, 403);
|
||||
assert.match(evil.json.error, /^cross-origin/, 'the serve gate or the handler refuses it');
|
||||
const bare = await f.get({ host: f.read.host });
|
||||
assert.equal(bare.status, 200, 'a read with no Origin and no Fetch Metadata is allowed');
|
||||
assert.equal((await f.get({ ...f.read, 'sec-fetch-site': 'none' })).status, 200, 'a typed address is allowed');
|
||||
const cross = await f.get({ ...f.read, 'sec-fetch-site': 'cross-site' });
|
||||
assert.equal(cross.status, 403);
|
||||
assert.equal(cross.json.error, 'cross-site');
|
||||
assert.equal((await f.get({ host: f.read.host, 'sec-fetch-site': 'same-site' })).status, 403);
|
||||
assert.equal((await f.get({ ...f.read, origin: `http://127.0.0.1:${f.port}` })).status, 200, 'a matching Origin on a read is fine');
|
||||
// Write: missing Origin, foreign Origin, missing or cross-site Fetch Metadata.
|
||||
const { origin, ...noOrigin } = f.write;
|
||||
const missing = await f.post(BODY, noOrigin);
|
||||
assert.equal(missing.status, 403);
|
||||
assert.match(missing.json.error, /^cross-origin/, 'the serve gate or the handler refuses it');
|
||||
assert.equal((await f.post(BODY, { ...f.write, origin: 'http://evil.example' })).status, 403);
|
||||
const { 'sec-fetch-site': _, ...noSite } = f.write;
|
||||
assert.equal((await f.post(BODY, noSite)).status, 403);
|
||||
assert.equal((await f.post(BODY, { ...f.write, 'sec-fetch-site': 'same-site' })).status, 403);
|
||||
assert.equal(readRoute(f.root), null);
|
||||
assert.deepEqual(receipts(f.root), []);
|
||||
});
|
||||
|
||||
test('a LAN dev host reads the route with no Fetch Metadata and writes with its own Origin', async t => {
|
||||
const f = await devServer(t, { devAllowHosts: ['lanbox'] });
|
||||
const host = `lanbox:${f.port}`;
|
||||
const r = await f.get({ host });
|
||||
assert.equal(r.status, 200);
|
||||
assert.equal(r.json.route, null);
|
||||
const w = await f.post(BODY, { host, origin: `http://${host}`, 'content-type': 'application/json' });
|
||||
assert.equal(w.status, 200, JSON.stringify(w.json));
|
||||
assert.equal(readRoute(f.root).model, BODY.model);
|
||||
const foreign = await f.post(BODY, { host, origin: 'http://evil.example', 'content-type': 'application/json' });
|
||||
assert.equal(foreign.status, 403);
|
||||
assert.equal(readRoute(f.root).revision, 1);
|
||||
});
|
||||
|
||||
test('the handler holds the same-origin line itself: a doubled Origin or Sec-Fetch-Site is refused', async t => {
|
||||
const root = mkdtempSync(join(tmpdir(), 'tr-unit-'));
|
||||
t.after(() => rmSync(root, { recursive: true, force: true }));
|
||||
let handler;
|
||||
triageRouteRoutes({ dataRoot: root, catalogues: async () => [READY()], harnesses: () => ['claude-code', 'pi'] })({ add: (method, path, fn) => { handler = fn; } });
|
||||
const call = (method, headersDistinct, extra = {}) => new Promise(resolve => {
|
||||
const res = { writeHead(status) { this.status = status; }, end(text) { resolve({ status: this.status, json: JSON.parse(text) }); } };
|
||||
const req = { method, headersDistinct, headers: { 'content-type': 'application/json' }, async *[Symbol.asyncIterator]() { yield Buffer.from(JSON.stringify(BODY)); } };
|
||||
handler(req, res, { owner: true, session: { provider: 'local-dev', sub: 'local-dev-owner' }, origin: 'https://console.example', send: true, ...extra });
|
||||
});
|
||||
const o = 'https://console.example';
|
||||
for (const method of ['PUT', 'DELETE', 'PATCH']) assert.equal((await call(method, { origin: [o], 'sec-fetch-site': ['same-origin'] })).json.error, 'method-not-allowed', method);
|
||||
assert.equal(readRoute(root), null);
|
||||
assert.equal((await call('GET', { origin: [o, o], 'sec-fetch-site': ['same-origin'] })).json.error, 'cross-origin');
|
||||
assert.equal((await call('GET', { 'sec-fetch-site': ['same-origin', 'same-origin'] })).json.error, 'cross-site');
|
||||
assert.equal((await call('GET', { origin: ['https://evil.example'], 'sec-fetch-site': ['same-origin'] })).json.error, 'cross-origin');
|
||||
assert.equal((await call('POST', { origin: [o, o], 'sec-fetch-site': ['same-origin'] })).json.error, 'cross-origin');
|
||||
assert.equal((await call('POST', { origin: [o], 'sec-fetch-site': ['same-origin', 'same-origin'] })).json.error, 'cross-site');
|
||||
assert.equal((await call('POST', { origin: [o], 'sec-fetch-site': ['same-origin'] }, { send: false })).json.error, 'read-only');
|
||||
assert.equal((await call('GET', { 'sec-fetch-site': ['same-origin'] }, { owner: false })).json.error, 'human-required');
|
||||
assert.equal((await call('POST', { origin: [o], 'sec-fetch-site': ['same-origin'] }, { owner: false })).json.error, 'human-required');
|
||||
assert.equal((await call('POST', { origin: [o], 'sec-fetch-site': ['same-origin'] }, { session: { provider: 'oidc', sub: 'Owner Sub' } })).json.error, 'human-required', 'an owner with no recordable name is refused');
|
||||
assert.equal(readRoute(root), null);
|
||||
assert.equal((await call('POST', { origin: [o], 'sec-fetch-site': ['same-origin'] })).status, 200);
|
||||
});
|
||||
|
||||
test('a read-only Console reads the route but refuses a write', async t => {
|
||||
const f = await devServer(t, { send: null });
|
||||
const r = await f.get();
|
||||
assert.equal(r.status, 200);
|
||||
assert.equal(r.json.send, false);
|
||||
const w = await f.post();
|
||||
assert.equal(w.status, 403);
|
||||
assert.equal(w.json.error, 'read-only');
|
||||
assert.equal(readRoute(f.root), null);
|
||||
});
|
||||
|
||||
test('an SSO root owner reads and writes as their account; a granted user and a signed-out caller reach neither route', async t => {
|
||||
const access = { accessVersion: 1, revision: 1,
|
||||
users: [{ id: 'alice', home: 'local', subjects: [{ provider: 'writer', sub: 'alice-sub' }] }],
|
||||
grants: [{ id: 'g-1', user: 'alice', business: 'demo', projects: ['*'], instances: ['*'], level: 'collaborator' }] };
|
||||
const auth = await createSsoSession(t, { host: 'console.example', users: consoleAccess({ load: () => access, business: () => 'demo' }) });
|
||||
const root = mkdtempSync(join(tmpdir(), 'tr-sso-'));
|
||||
t.after(() => rmSync(root, { recursive: true, force: true }));
|
||||
const s = await startServer({ port: 0, sso: auth.sso, send: async () => ({}), log: () => {},
|
||||
routes: [triageRouteRoutes({ dataRoot: root, catalogues: async () => [READY()], harnesses: () => ['claude-code', 'pi'] })] });
|
||||
t.after(() => new Promise(r => s.close(r)));
|
||||
const port = s.address().port;
|
||||
const owner = auth.writeHeaders(await auth.signIn(port));
|
||||
assert.equal((await send(port, TRIAGE_ROUTE_PATH, { headers: owner })).status, 200);
|
||||
const saved = await send(port, TRIAGE_ROUTE_PATH, { method: 'POST', headers: owner, body: BODY });
|
||||
assert.equal(saved.status, 200, saved.text);
|
||||
assert.match(readRoute(root).updatedBy, /^u-[0-9a-f]{12}$/);
|
||||
const alice = auth.writeHeaders(await auth.signIn(port, { sub: 'alice-sub' }));
|
||||
const aliceRead = await send(port, TRIAGE_ROUTE_PATH, { headers: alice });
|
||||
assert.equal(aliceRead.status, 403);
|
||||
assert.ok(['owner-only', 'human-required'].includes(aliceRead.json.error), aliceRead.text);
|
||||
assert.equal(aliceRead.text.includes('sonnet'), false, 'a granted user sees no route');
|
||||
const aliceWrite = await send(port, TRIAGE_ROUTE_PATH, { method: 'POST', headers: alice, body: { ...BODY, revision: 1, cap: 7 } });
|
||||
assert.equal(aliceWrite.status, 403);
|
||||
const nobody = await send(port, TRIAGE_ROUTE_PATH, { headers: { host: 'console.example', 'sec-fetch-site': 'same-origin' } });
|
||||
assert.equal(nobody.status, 401);
|
||||
const nobodyWrite = await send(port, TRIAGE_ROUTE_PATH, { method: 'POST', headers: { host: 'console.example', origin: 'https://console.example', 'sec-fetch-site': 'same-origin', 'content-type': 'application/json' }, body: { ...BODY, revision: 1 } });
|
||||
assert.equal(nobodyWrite.status, 401);
|
||||
// Cross-origin as the owner, on the public name.
|
||||
assert.equal((await send(port, TRIAGE_ROUTE_PATH, { method: 'POST', headers: { ...owner, origin: 'https://evil.example' }, body: { ...BODY, revision: 1 } })).status, 403);
|
||||
assert.equal((await send(port, TRIAGE_ROUTE_PATH, { headers: { ...owner, origin: 'https://evil.example' } })).status, 403);
|
||||
assert.equal(readRoute(root).revision, 1);
|
||||
assert.equal(readRoute(root).cap.limit, 200000);
|
||||
assert.equal(receipts(root).length, 1);
|
||||
});
|
||||
|
||||
test('the tester listener reaches neither route', async t => {
|
||||
const { startTesterSocket } = await import('../src/tester-socket.mjs');
|
||||
const { testerRequest } = await import('../../cli/src/tester-http.mjs');
|
||||
const root = mkdtempSync(join(tmpdir(), 'tr-tester-'));
|
||||
t.after(() => rmSync(root, { recursive: true, force: true }));
|
||||
const s = await startTesterSocket({ dataRoot: root, log: () => {}, routes: [triageRouteRoutes({ dataRoot: root, catalogues: async () => [READY()], harnesses: () => ['claude-code', 'pi'] })] });
|
||||
t.after(() => new Promise(r => s.close(r)));
|
||||
for (const method of ['GET', 'POST']) {
|
||||
const r = await testerRequest({ dataRoot: root, path: TRIAGE_ROUTE_PATH, method, body: method === 'POST' ? BODY : undefined });
|
||||
assert.equal(r.status, 403, method);
|
||||
}
|
||||
assert.equal(readRoute(root), null);
|
||||
});
|
||||
|
||||
test('a busy store and an unreadable store answer with fixed words and no path', async t => {
|
||||
const f = await devServer(t);
|
||||
assert.equal((await f.post()).status, 200);
|
||||
const { lock, file } = routePaths(f.root);
|
||||
writeFileSync(lock, JSON.stringify({ pid: process.pid, at: T }), { mode: 0o600 });
|
||||
const busy = await f.post({ ...BODY, revision: 1 });
|
||||
assert.equal(busy.status, 409);
|
||||
assert.equal(busy.json.error, 'triage-route-busy');
|
||||
assert.equal(busy.text.includes(f.root) || busy.text.includes('route.lock'), false);
|
||||
rmSync(lock);
|
||||
writeFileSync(file, '{not json', { mode: 0o600 });
|
||||
const broken = await f.get();
|
||||
assert.equal(broken.status, 503);
|
||||
assert.equal(broken.json.error, 'triage-route-unreadable');
|
||||
assert.equal(broken.text.includes(f.root) || broken.text.includes('route.json'), false);
|
||||
const brokenWrite = await f.post({ ...BODY, revision: 1 });
|
||||
assert.equal(brokenWrite.json.error, 'triage-route-unreadable');
|
||||
assert.equal(brokenWrite.text.includes(f.root), false);
|
||||
});
|
||||
|
||||
test('no response or log line carries the sentinel, the data root or a store path', async t => {
|
||||
const f = await devServer(t);
|
||||
const all = [await f.get(), await f.post(), await f.get(), await f.post({ ...BODY, cap: 0 }), await f.post({ ...BODY, model: 'opus' }), await f.post({ ...BODY, revision: 0 }), await f.post(BODY, { ...f.write, origin: 'http://evil.example' })];
|
||||
const text = all.map(r => r.text).join('\n') + f.routeLogs.join('\n') + f.logs.join('\n');
|
||||
for (const needle of [SECRET, f.root, 'route.json', 'route.lock', 'route-receipts', '/srv/', '/home/', 'cliproxy', 'anthropic']) assert.equal(text.includes(needle), false, needle);
|
||||
});
|
||||
|
||||
test('mergeCatalogues: a key two catalogues disagree on is dropped; availability and harnesses must hold everywhere', () => {
|
||||
const a = { status: 'ready', models: [model('sonnet'), model('glm', { harnesses: ['pi'] }), model('x', { available: false })] };
|
||||
const b = { status: 'ready', models: [model('sonnet', { harnesses: ['pi'] }), model('glm', { upstream: 'other' }), model('x'), model('y', { available: false })] };
|
||||
a.models.push(model('y'));
|
||||
const merged = mergeCatalogues([a, b, { status: 'bus-unavailable' }], ['claude-code', 'pi']);
|
||||
assert.equal(merged.status, 'ready');
|
||||
assert.deepEqual(merged.models.map(m => [m.key, m.available, m.harnesses]), [['sonnet', true, ['pi']], ['x', false, ['claude-code', 'pi']], ['y', false, ['claude-code', 'pi']]]);
|
||||
assert.deepEqual(mergeCatalogues([{ status: 'starting' }], ['pi']), { status: 'starting', models: [] });
|
||||
assert.deepEqual(mergeCatalogues([], ['pi']), { status: 'unavailable', models: [] });
|
||||
assert.deepEqual(mergeCatalogues([{ status: 'ready', models: [{ key: 'BAD KEY', harnesses: ['pi'] }, null] }], ['pi']).models, []);
|
||||
});
|
||||
|
||||
test('installedHarnesses lists the harness modules on disk; ownerActor names the account or the dev owner', () => {
|
||||
assert.deepEqual(installedHarnesses(), ['claude-code', 'pi']);
|
||||
assert.equal(ownerActor({ account: 'u-0123456789ab', provider: 'writer', sub: 'x' }), 'u-0123456789ab');
|
||||
assert.equal(ownerActor({ provider: 'local-dev', sub: 'local-dev-owner' }), 'local-dev-owner');
|
||||
assert.equal(ownerActor({ provider: 'writer', sub: 'owner-sub' }), null);
|
||||
assert.equal(ownerActor(null), null);
|
||||
});
|
||||
@@ -0,0 +1,262 @@
|
||||
// #1873 triage-settings: Settings > Triage in a real browser. The owner sees "not set", picks a model,
|
||||
// a harness from that model's list, a cap and a time zone, and saves in three clicks; a refusal is one
|
||||
// line and keeps the fields; a stale revision offers a re-read; a second Save while one runs does
|
||||
// nothing; anyone /api/config does not call the owner gets no Triage page and no route read.
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { w15Fixture } from './w15-fixture.mjs';
|
||||
|
||||
const T = '2026-11-18T12:00:00.000Z';
|
||||
const NOTE = 'Saving a route does not start Triage. The key and the wiring are separate.';
|
||||
const choices = { status: 'ready', models: [
|
||||
{ key: 'down', label: 'Down', available: false, harnesses: ['claude-code'] },
|
||||
{ key: 'glm-flash', label: 'GLM Flash', available: true, harnesses: ['pi'] },
|
||||
{ key: 'sonnet', label: 'Sonnet', available: true, harnesses: ['claude-code', 'pi'] },
|
||||
] };
|
||||
const unset = { route: null, routeMessage: 'No Triage route is set.', revision: 0, choices, send: true, note: NOTE };
|
||||
const route = (over = {}) => ({ model: 'sonnet', harness: 'pi', cap: 200000, reset: '00:00', timeZone: 'Europe/Berlin', revision: 1, updatedAt: T, ...over });
|
||||
const config = (over = {}) => ({ send: true, devOwner: true, sso: true, proxyAdmin: true, proxyAdminSend: true, ...over });
|
||||
const scenario = (read = unset, extra = {}, cfg = {}) => ({ '/api/config': config(cfg), '/api/triage/route': read, ...extra });
|
||||
const writes = b => b.evaluate('window.__writes.filter(w=>w.path==="/api/triage/route").map(w=>({method:w.method,body:JSON.parse(w.body)}))');
|
||||
// The fixture answers in the page, so no resource timing exists: count the route reads in a wrapper.
|
||||
const countReads = b => b.call('Page.addScriptToEvaluateOnNewDocument', { source: '(()=>{window.__routeReads=0;const prev=window.fetch;window.fetch=(i,init)=>{try{if(new URL(typeof i==="string"?i:i.url,location.href).pathname==="/api/triage/route"&&(init?.method??"GET")==="GET")window.__routeReads++;}catch{}return prev(i,init);};})()' });
|
||||
const reads = b => b.evaluate('window.__routeReads');
|
||||
const pick = (b, id, value) => b.evaluate(`(()=>{const el=document.getElementById(${JSON.stringify(id)});el.value=${JSON.stringify(value)};el.dispatchEvent(new Event('input',{bubbles:true}));el.dispatchEvent(new Event('change',{bubbles:true}));})()`);
|
||||
const type = (b, id, value) => b.evaluate(`(()=>{const el=document.getElementById(${JSON.stringify(id)});el.value=${JSON.stringify(value)};el.dispatchEvent(new Event('input',{bubbles:true}));})()`);
|
||||
const text = (b, id) => b.evaluate(`document.getElementById(${JSON.stringify(id)})?.textContent ?? null`);
|
||||
const value = (b, id) => b.evaluate(`document.getElementById(${JSON.stringify(id)})?.value ?? null`);
|
||||
const options = (b, id) => b.evaluate(`[...document.getElementById(${JSON.stringify(id)}).options].map(o=>[o.value,o.textContent,o.disabled])`);
|
||||
async function fill(b, { model = 'sonnet', harness = 'claude-code', cap = '150000', zone = 'America/Chicago' } = {}) {
|
||||
await pick(b, 'triage-route-model', model);
|
||||
if (harness !== null) await pick(b, 'triage-route-harness', harness);
|
||||
await type(b, 'triage-route-cap', cap);
|
||||
await type(b, 'triage-route-zone', zone);
|
||||
}
|
||||
|
||||
test('the owner sees Triage in Settings, "not set" before a first save, the catalogue choices and the note', { timeout: 90000 }, async t => {
|
||||
const { f, b, wait } = await w15Fixture(t, { scenario: scenario() });
|
||||
await countReads(b);
|
||||
await b.navigate(f.base + '/#/settings');
|
||||
await wait('document.querySelector(\'#settings-nav a[href="#/settings/triage"]\')');
|
||||
await b.evaluate('document.querySelector(\'#settings-nav a[href="#/settings/triage"]\').click()');
|
||||
await wait('document.getElementById("triage-route-stored")');
|
||||
assert.match(await text(b, 'triage-route-stored'), /^Not set\. No Triage route is set\.$/);
|
||||
assert.ok(await reads(b) >= 1, 'the owner page reads the route (the counter works)');
|
||||
assert.equal(await text(b, 'triage-route-note'), NOTE);
|
||||
assert.equal(await text(b, 'settings-page-title'), 'Triage');
|
||||
assert.deepEqual(await options(b, 'triage-route-model'), [['', 'Choose a model', false], ['down', 'Down (down) (unavailable)', true], ['glm-flash', 'GLM Flash (glm-flash)', false], ['sonnet', 'Sonnet (sonnet)', false]]);
|
||||
assert.deepEqual(await options(b, 'triage-route-harness'), [['', 'Choose a model first', false]]);
|
||||
assert.equal(await value(b, 'triage-route-cap'), '', 'no default cap');
|
||||
assert.equal(await value(b, 'triage-route-zone'), '', 'no default time zone');
|
||||
assert.equal(await value(b, 'triage-route-reset'), '00:00');
|
||||
assert.equal(await b.evaluate('document.getElementById("triage-route-reset").readOnly'), true);
|
||||
assert.equal(await b.evaluate('document.querySelector(".settings-readonly")'), null);
|
||||
assert.equal(await b.evaluate('document.getElementById("triage-route-catalogue")'), null);
|
||||
// The harness list follows the model; a model with one harness picks it.
|
||||
await pick(b, 'triage-route-model', 'glm-flash');
|
||||
assert.deepEqual(await options(b, 'triage-route-harness'), [['', 'Choose a harness', false], ['pi', 'pi', false]]);
|
||||
assert.equal(await value(b, 'triage-route-harness'), 'pi');
|
||||
await pick(b, 'triage-route-model', 'sonnet');
|
||||
assert.deepEqual((await options(b, 'triage-route-harness')).map(o => o[0]), ['', 'claude-code', 'pi']);
|
||||
assert.equal(await value(b, 'triage-route-harness'), 'pi', 'a harness the new model takes is kept');
|
||||
assert.deepEqual(await writes(b), []);
|
||||
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
|
||||
assert.deepEqual(await b.evaluate('window.__errors'), []);
|
||||
});
|
||||
|
||||
test('Save posts the fields and the read revision once, then shows the stored route; three clicks from Settings', { timeout: 90000 }, async t => {
|
||||
const saved = route({ harness: 'claude-code', cap: 150000, timeZone: 'America/Chicago' });
|
||||
const { f, b, wait } = await w15Fixture(t, { scenario: scenario(unset, { 'POST /api/triage/route': { body: { route: saved, revision: 1, note: NOTE } } }) });
|
||||
await b.navigate(f.base + '/#/settings');
|
||||
await wait('document.querySelector(\'#settings-nav a[href="#/settings/triage"]\')');
|
||||
await b.evaluate('document.querySelector(\'#settings-nav a[href="#/settings/triage"]\').click()');
|
||||
await wait('document.getElementById("triage-route-model")');
|
||||
await fill(b, { zone: ' America/Chicago ' });
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await wait('document.getElementById("triage-route-line")?.textContent==="Saved. Revision 1."');
|
||||
assert.equal(await b.evaluate('document.activeElement?.id'), 'triage-route-line', 'focus lands on the saved line, not the page body');
|
||||
assert.deepEqual(await writes(b), [{ method: 'POST', body: { model: 'sonnet', harness: 'claude-code', cap: 150000, timeZone: 'America/Chicago', revision: 0 } }]);
|
||||
assert.match(await text(b, 'triage-route-stored'), /Stored route: sonnet on claude-code, cap 150,000 tokens a day, reset 00:00 America\/Chicago\. Revision 1/);
|
||||
assert.equal(await value(b, 'triage-route-cap'), '150000');
|
||||
// The next save carries the new revision.
|
||||
await type(b, 'triage-route-cap', '9000');
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await wait('window.__writes.filter(w=>w.path==="/api/triage/route").length===2');
|
||||
assert.equal((await writes(b))[1].body.revision, 1);
|
||||
assert.deepEqual(await b.evaluate('window.__unexpected'), []);
|
||||
});
|
||||
|
||||
test('a stored route fills the fields; each refusal is one plain line and keeps the fields', { timeout: 90000 }, async t => {
|
||||
const { f, b, wait } = await w15Fixture(t, { scenario: scenario({ ...unset, route: route(), routeMessage: undefined, revision: 1 }) });
|
||||
await b.navigate(f.base + '/#/settings/triage');
|
||||
await wait('document.getElementById("triage-route-model")?.value==="sonnet"');
|
||||
assert.equal(await value(b, 'triage-route-harness'), 'pi');
|
||||
assert.equal(await value(b, 'triage-route-cap'), '200000');
|
||||
assert.equal(await value(b, 'triage-route-zone'), 'Europe/Berlin');
|
||||
assert.match(await text(b, 'triage-route-stored'), /^Stored route: sonnet on pi, cap 200,000 tokens a day, reset 00:00 Europe\/Berlin\. Revision 1, saved 2026-11-18T12:00:00\.000Z\.$/);
|
||||
for (const [error, message, status] of [['triage-zone-unknown', 'The time zone is not one this Console knows.', 400], ['model-harness-incompatible', 'That harness does not run this model.', 400], ['model-provider-unavailable', 'The model provider is not available now.', 503]]) {
|
||||
await b.evaluate(`Object.assign(window.__scenario, {'POST /api/triage/route': {status: ${status}, body: {error: ${JSON.stringify(error)}, message: ${JSON.stringify(message)}}}})`);
|
||||
await fill(b, { zone: 'Mars/Olympus', cap: '777' });
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await wait(`document.getElementById("triage-route-line")?.textContent===${JSON.stringify(message)}`);
|
||||
assert.equal(await value(b, 'triage-route-model'), 'sonnet');
|
||||
assert.equal(await value(b, 'triage-route-harness'), 'claude-code');
|
||||
assert.equal(await value(b, 'triage-route-cap'), '777');
|
||||
assert.equal(await value(b, 'triage-route-zone'), 'Mars/Olympus');
|
||||
assert.equal(await b.evaluate('document.getElementById("triage-route-reload")'), null, 'only a conflict offers a re-read');
|
||||
assert.equal(await b.evaluate('document.getElementById("triage-route-save").disabled'), false);
|
||||
}
|
||||
assert.match(await text(b, 'triage-route-stored'), /Revision 1/, 'a refusal leaves the stored line alone');
|
||||
// A refusal with no readable answer still says one line.
|
||||
await b.evaluate(`Object.assign(window.__scenario, {'POST /api/triage/route': {status: 500, body: {}}})`);
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await wait('document.getElementById("triage-route-line")?.textContent==="The route was refused. Nothing was saved."');
|
||||
assert.equal(await value(b, 'triage-route-cap'), '777');
|
||||
// Leaving the page and coming back reads the route again and keeps what was typed.
|
||||
await b.evaluate('location.hash="#/settings/appearance"');
|
||||
await wait('!document.getElementById("triage-route")');
|
||||
await b.evaluate('location.hash="#/settings/triage"');
|
||||
await wait('document.getElementById("triage-route-cap")');
|
||||
assert.equal(await value(b, 'triage-route-cap'), '777');
|
||||
assert.equal(await value(b, 'triage-route-zone'), 'Mars/Olympus');
|
||||
assert.match(await text(b, 'triage-route-stored'), /Revision 1/);
|
||||
});
|
||||
|
||||
test('a stale revision says so, keeps the fields, and Reload route reads the stored route again', { timeout: 90000 }, async t => {
|
||||
const message = 'The Triage route changed since this page read it. Reload the route, then save again.';
|
||||
const { f, b, wait } = await w15Fixture(t, { scenario: scenario(unset, { 'POST /api/triage/route': { status: 409, body: { error: 'triage-route-stale', message, revision: 2 } } }) });
|
||||
await b.navigate(f.base + '/#/settings/triage');
|
||||
await wait('document.getElementById("triage-route-model")');
|
||||
await fill(b, { cap: '5000' });
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await wait(`document.getElementById("triage-route-line")?.textContent===${JSON.stringify(message)}`);
|
||||
assert.equal(await value(b, 'triage-route-cap'), '5000');
|
||||
assert.equal(await b.evaluate('document.activeElement?.id'), 'triage-route-reload');
|
||||
await b.evaluate(`window.__scenario['/api/triage/route'] = ${JSON.stringify({ ...unset, route: route({ revision: 2, model: 'glm-flash', cap: 42 }), routeMessage: undefined, revision: 2 })}`);
|
||||
await b.evaluate('document.getElementById("triage-route-reload").click()');
|
||||
await wait('document.getElementById("triage-route-model")?.value==="glm-flash"');
|
||||
assert.equal(await value(b, 'triage-route-cap'), '42');
|
||||
assert.match(await text(b, 'triage-route-stored'), /Revision 2/);
|
||||
assert.equal(await b.evaluate('document.getElementById("triage-route-reload")'), null);
|
||||
await b.evaluate(`window.__scenario['POST /api/triage/route'] = {body: {route: ${JSON.stringify(route({ revision: 3, model: 'glm-flash', cap: 42 }))}, revision: 3, note: ${JSON.stringify(NOTE)}}}`);
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await wait('document.getElementById("triage-route-line")?.textContent==="Saved. Revision 3."');
|
||||
assert.equal((await writes(b)).at(-1).body.revision, 2, 'the save after a re-read carries the stored revision');
|
||||
});
|
||||
|
||||
test('an untouched page follows the store; typed fields keep the revision they were read at, so a later save is stale', { timeout: 90000 }, async t => {
|
||||
const stored = (over, revision) => ({ ...unset, route: route({ revision, ...over }), routeMessage: undefined, revision });
|
||||
const message = 'The Triage route changed since this page read it. Reload the route, then save again.';
|
||||
const { f, b, wait } = await w15Fixture(t, { scenario: scenario(stored({}, 1), { 'POST /api/triage/route': { status: 409, body: { error: 'triage-route-stale', message, revision: 3 } } }) });
|
||||
const away = async () => {
|
||||
await b.evaluate('location.hash="#/settings/appearance"');
|
||||
await wait('!document.getElementById("triage-route")');
|
||||
await b.evaluate('location.hash="#/settings/triage"');
|
||||
await wait('document.getElementById("triage-route-cap")');
|
||||
};
|
||||
const dot = 'document.querySelector(\'#settings-nav a[href="#/settings/triage"] .settings-dot\') !== null';
|
||||
await b.navigate(f.base + '/#/settings/triage');
|
||||
await wait('document.getElementById("triage-route-model")?.value==="sonnet"');
|
||||
// Saved elsewhere while this page sat untouched: the page shows the new route, with no unsaved mark.
|
||||
await b.evaluate(`window.__scenario['/api/triage/route'] = ${JSON.stringify(stored({ model: 'glm-flash', cap: 42 }, 2))}`);
|
||||
await away();
|
||||
assert.equal(await value(b, 'triage-route-model'), 'glm-flash');
|
||||
assert.equal(await value(b, 'triage-route-cap'), '42');
|
||||
assert.match(await text(b, 'triage-route-stored'), /Revision 2/);
|
||||
assert.equal(await b.evaluate(dot), false);
|
||||
// Typed at revision 2, then saved elsewhere again: the typed fields stay and the save sends revision 2.
|
||||
await type(b, 'triage-route-cap', '9000');
|
||||
await b.evaluate(`window.__scenario['/api/triage/route'] = ${JSON.stringify(stored({ model: 'sonnet', cap: 1 }, 3))}`);
|
||||
await away();
|
||||
assert.equal(await value(b, 'triage-route-model'), 'glm-flash');
|
||||
assert.equal(await value(b, 'triage-route-cap'), '9000');
|
||||
assert.match(await text(b, 'triage-route-stored'), /Revision 3/);
|
||||
assert.equal(await b.evaluate(dot), true, 'the typed fields are marked unsaved');
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await wait(`document.getElementById("triage-route-line")?.textContent===${JSON.stringify(message)}`);
|
||||
const last = (await writes(b)).at(-1).body;
|
||||
assert.equal(last.revision, 2, 'the save carries the revision the typed fields were read at, not the newest read');
|
||||
assert.equal(last.model, 'glm-flash');
|
||||
assert.equal(last.cap, 9000);
|
||||
});
|
||||
|
||||
test('a second Save while one runs does nothing', { timeout: 90000 }, async t => {
|
||||
const { f, b, wait } = await w15Fixture(t, { scenario: scenario(unset, { 'POST /api/triage/route': { body: { route: route(), revision: 1, note: NOTE } } }) });
|
||||
await b.navigate(f.base + '/#/settings/triage');
|
||||
await wait('document.getElementById("triage-route-model")');
|
||||
// Hold the POST until the test lets it go.
|
||||
await b.evaluate('(()=>{const prev=window.fetch;window.__posts=0;window.fetch=(input,init)=>init?.method==="POST"?(window.__posts++,new Promise(r=>{window.__release=()=>r(prev(input,init));})):prev(input,init);})()');
|
||||
await fill(b);
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await wait('typeof window.__release==="function"');
|
||||
assert.equal(await text(b, 'triage-route-save'), 'Saving…');
|
||||
assert.equal(await b.evaluate('document.getElementById("triage-route-save").disabled'), true);
|
||||
// A click on the disabled button, a form submit (Enter) and a direct requestSubmit all do nothing.
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await b.evaluate('document.getElementById("triage-route-form").requestSubmit()');
|
||||
await b.evaluate('document.getElementById("triage-route-form").dispatchEvent(new Event("submit",{bubbles:true,cancelable:true}))');
|
||||
assert.equal(await b.evaluate('window.__writes.filter(w=>w.path==="/api/triage/route").length'), 0, 'the held POST is not recorded yet');
|
||||
assert.equal(await b.evaluate('window.__posts'), 1, 'no second POST left the page');
|
||||
await b.evaluate('window.__release()');
|
||||
await wait('document.getElementById("triage-route-line")?.textContent==="Saved. Revision 1."');
|
||||
assert.equal((await writes(b)).length, 1);
|
||||
});
|
||||
|
||||
test('a read-only Console shows the route with Save off and posts nothing', { timeout: 90000 }, async t => {
|
||||
const { f, b, wait } = await w15Fixture(t, { scenario: scenario({ ...unset, send: false }, {}, { send: false, proxyAdminSend: false }) });
|
||||
await b.navigate(f.base + '/#/settings/triage');
|
||||
await wait('document.getElementById("triage-route-save")');
|
||||
assert.equal(await b.evaluate('document.getElementById("triage-route-save").disabled'), true);
|
||||
assert.match(await text(b, 'triage-route-readonly'), /read-only/);
|
||||
assert.equal(await b.evaluate('document.querySelector(".settings-readonly")'), null, 'the panel says read-only once, from the route\'s own answer');
|
||||
await b.evaluate('document.getElementById("triage-route-form").dispatchEvent(new Event("submit",{bubbles:true,cancelable:true}))');
|
||||
assert.deepEqual(await writes(b), []);
|
||||
});
|
||||
|
||||
test('a catalogue that is not ready says so and offers no model', { timeout: 90000 }, async t => {
|
||||
const { f, b, wait } = await w15Fixture(t, { scenario: scenario({ ...unset, choices: { status: 'bus-unavailable', models: [] } }) });
|
||||
await b.navigate(f.base + '/#/settings/triage');
|
||||
await wait('document.getElementById("triage-route-catalogue")');
|
||||
assert.match(await text(b, 'triage-route-catalogue'), /No organization bus offered a model catalogue \(bus-unavailable\)/);
|
||||
assert.deepEqual(await options(b, 'triage-route-model'), [['', 'Choose a model', false]]);
|
||||
});
|
||||
|
||||
test('a non-owner sees no Triage section and the page reads no route', { timeout: 90000 }, async t => {
|
||||
const { f, b, wait } = await w15Fixture(t, { scenario: scenario(unset, {}, { proxyAdmin: undefined, proxyAdminSend: undefined, user: true }) });
|
||||
await countReads(b);
|
||||
await b.navigate(f.base + '/#/settings');
|
||||
await wait('document.querySelector(\'#settings-nav a[href="#/settings/appearance"]\')');
|
||||
assert.equal(await b.evaluate('document.querySelector(\'#settings-nav a[href="#/settings/triage"]\')'), null);
|
||||
await b.navigate(f.base + '/#/settings/triage');
|
||||
await wait('document.querySelector(".nf")');
|
||||
assert.equal(await b.evaluate('document.getElementById("triage-route")'), null);
|
||||
assert.equal(await reads(b), 0, 'no route read for a non-owner');
|
||||
assert.deepEqual(await writes(b), []);
|
||||
});
|
||||
|
||||
// Shots for the build report: run with MOSAIC_UX_SHOTS=<dir>. Not set, filled and one refusal, at 390
|
||||
// and 1440, light and dark.
|
||||
test('shots', { timeout: 300000, skip: !process.env.MOSAIC_UX_SHOTS }, async t => {
|
||||
const { f, b, wait, capture } = await w15Fixture(t, { slice: 'triage', scenario: scenario(unset, { 'POST /api/triage/route': { status: 400, body: { error: 'triage-zone-unknown', message: 'The time zone is not one this Console knows.' } } }) });
|
||||
const results = [];
|
||||
for (const scheme of ['light', 'dark']) {
|
||||
await b.call('Emulation.setEmulatedMedia', { features: [{ name: 'prefers-color-scheme', value: scheme }] });
|
||||
for (const width of [390, 1440]) {
|
||||
await b.viewport(width, width === 390 ? 844 : 900);
|
||||
// A fresh document each round, so the kept fields of the last round do not carry over.
|
||||
await b.navigate(`${f.base}/?shot=${scheme}-${width}#/settings/triage`);
|
||||
await wait('document.getElementById("triage-route-model") && !!window.MosaicAppearance');
|
||||
await b.evaluate(`window.MosaicAppearance.ready.then(() => window.MosaicAppearance.set({ mode: ${JSON.stringify(scheme)} }))`);
|
||||
// No sideways scroll at either width.
|
||||
assert.equal(await b.evaluate('document.documentElement.scrollWidth <= document.documentElement.clientWidth'), true, `${scheme} ${width} scrolls sideways`);
|
||||
results.push(await capture(`notset-${scheme}`, width, '#/settings/triage'));
|
||||
await fill(b);
|
||||
results.push(await capture(`filled-${scheme}`, width, '#/settings/triage'));
|
||||
await type(b, 'triage-route-zone', 'Mars/Olympus');
|
||||
await b.evaluate('document.getElementById("triage-route-save").click()');
|
||||
await wait('document.getElementById("triage-route-line")?.textContent.includes("time zone")');
|
||||
results.push(await capture(`refusal-${scheme}`, width, '#/settings/triage'));
|
||||
}
|
||||
}
|
||||
for (const r of results) { assert.deepEqual(r.errors, [], r.full); assert.deepEqual(r.unexpected, [], r.full); }
|
||||
});
|
||||
@@ -0,0 +1,205 @@
|
||||
// #1873 triage-cap: Triage's usage file and its daily cap. One line per turn in
|
||||
// <dataRoot>/triage/usage.jsonl, held to the same file checks as the route store; the cap is the
|
||||
// daily-quota plugin's own rule over those lines, with the stored cap as its one leaf.
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { chmodSync, linkSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { join } from "node:path";
|
||||
import { mayLaunch, report } from "../../plugins/limit-check/daily-quota/index.mjs";
|
||||
import {
|
||||
USAGE_ACCOUNT, USAGE_FIELDS, appendUsage, capState, delta, limitsFor, nextReset, readUsage, snapshot, usagePath,
|
||||
} from "../src/triage-usage.mjs";
|
||||
|
||||
const scratch = () => mkdtempSync(join(process.env.TMPDIR, "tu-"));
|
||||
const fileIn = () => {
|
||||
const dir = join(scratch(), "triage");
|
||||
mkdirSync(dir, { mode: 0o700 });
|
||||
return join(dir, "usage.jsonl");
|
||||
};
|
||||
const line = (over = {}) => ({ at: "2026-11-18T12:00:00.000Z", run: "triage-20261118120000", turn: "owner-0123456789ab", status: "known", input: 10, output: 5, cacheRead: 2, cacheWrite: 1, ...over });
|
||||
const unknownLine = (over = {}) => line({ status: "unknown", input: null, output: null, cacheRead: null, cacheWrite: null, ...over });
|
||||
const cap = (over = {}) => ({ limit: 100, since: "2026-10-01T00:00:00.000Z", reset: "00:00", timeZone: "UTC", ...over });
|
||||
const ms = (s) => Date.parse(s);
|
||||
|
||||
test("the usage file is <dataRoot>/triage/usage.jsonl and a line holds exactly its eight fields", () => {
|
||||
assert.equal(usagePath("/d"), "/d/triage/usage.jsonl");
|
||||
assert.deepEqual(USAGE_FIELDS, ["at", "run", "turn", "status", "input", "output", "cacheRead", "cacheWrite"]);
|
||||
assert.equal(USAGE_ACCOUNT, "triage");
|
||||
});
|
||||
|
||||
test("each append writes one line, 0600, in field order; a torn last line is closed off before the next line", () => {
|
||||
const file = fileIn();
|
||||
appendUsage(file, line());
|
||||
appendUsage(file, unknownLine({ turn: "owner-0123456789ac" }));
|
||||
assert.equal(lstatSync(file).mode & 0o777, 0o600);
|
||||
const text = readFileSync(file, "utf8");
|
||||
assert.equal(text.split("\n").length, 3);
|
||||
assert.deepEqual(Object.keys(JSON.parse(text.split("\n")[0])), USAGE_FIELDS);
|
||||
// A crash in a write leaves a torn line. The next append starts on a line of its own.
|
||||
writeFileSync(file, '{"at":"2026-11-18T12:0', { flag: "a" });
|
||||
appendUsage(file, line({ turn: "owner-0123456789ad" }));
|
||||
const lines = readFileSync(file, "utf8").split("\n");
|
||||
assert.equal(lines.length, 5);
|
||||
assert.equal(JSON.parse(lines[3]).turn, "owner-0123456789ad");
|
||||
// A line that does not match its shape is never written.
|
||||
for (const bad of [{ ...line(), extra: 1 }, line({ at: "yesterday" }), line({ input: -1 }), line({ status: "unknown" }), unknownLine({ input: 3 }), line({ turn: "a b" }), line({ run: "" })]) {
|
||||
assert.throws(() => appendUsage(file, bad), (e) => e.code === "triage-usage-unreadable", JSON.stringify(bad));
|
||||
}
|
||||
assert.equal(readFileSync(file, "utf8").split("\n").length, 5);
|
||||
});
|
||||
|
||||
test("a narrow umask still leaves a new usage file at 0600", () => {
|
||||
const file = fileIn();
|
||||
const old = process.umask(0o277);
|
||||
try { appendUsage(file, line()); } finally { process.umask(old); }
|
||||
assert.equal(lstatSync(file).mode & 0o777, 0o600);
|
||||
});
|
||||
|
||||
// Each unsafe state refuses both the read and the append, and the append leaves the target alone.
|
||||
function unsafe(make) {
|
||||
const file = fileIn();
|
||||
const opts = make(file) ?? {};
|
||||
return { file, opts };
|
||||
}
|
||||
test("a wrong mode, a wrong owner, a symlink, a hard link or a non-file refuses the read and the append", () => {
|
||||
const cases = {
|
||||
"mode 0644": (f) => { appendUsage(f, line()); chmodSync(f, 0o644); },
|
||||
"mode 0400": (f) => { appendUsage(f, line()); chmodSync(f, 0o400); },
|
||||
"owner": (f) => { appendUsage(f, line()); return { uid: process.getuid() + 1 }; },
|
||||
"symlink": (f) => { const real = join(f, "..", "real.jsonl"); writeFileSync(real, "", { mode: 0o600 }); symlinkSync(real, f); },
|
||||
"dangling symlink": (f) => { symlinkSync(join(f, "..", "absent.jsonl"), f); },
|
||||
"hard link": (f) => { appendUsage(f, line()); linkSync(f, join(f, "..", "other.jsonl")); },
|
||||
"directory": (f) => { mkdirSync(f, { mode: 0o700 }); },
|
||||
"fifo": (f) => { execFileSync("mkfifo", ["-m", "600", f]); },
|
||||
};
|
||||
for (const [label, make] of Object.entries(cases)) {
|
||||
const { file, opts } = unsafe(make);
|
||||
const before = lstatSync(file);
|
||||
assert.throws(() => readUsage(file, opts), (e) => e.code === "triage-usage-unreadable", `read: ${label}`);
|
||||
assert.throws(() => appendUsage(file, line(), opts), (e) => e.code === "triage-usage-unreadable", `append: ${label}`);
|
||||
const after = lstatSync(file);
|
||||
assert.deepEqual([after.size, after.mode, after.ino], [before.size, before.mode, before.ino], `untouched: ${label}`);
|
||||
}
|
||||
// The dangling link's target is not made.
|
||||
const { file } = unsafe((f) => { symlinkSync(join(f, "..", "absent.jsonl"), f); });
|
||||
try { appendUsage(file, line()); } catch {}
|
||||
assert.throws(() => lstatSync(join(file, "..", "absent.jsonl")), (e) => e.code === "ENOENT");
|
||||
});
|
||||
|
||||
test("a usage file over the size bound refuses before it is parsed", () => {
|
||||
const file = fileIn();
|
||||
appendUsage(file, line());
|
||||
appendUsage(file, line({ turn: "owner-0123456789ac" }));
|
||||
const size = lstatSync(file).size;
|
||||
assert.equal(readUsage(file, { maxBytes: size }).records.length, 2);
|
||||
assert.throws(() => readUsage(file, { maxBytes: size - 1 }), (e) => e.code === "triage-usage-unreadable");
|
||||
});
|
||||
|
||||
test("no file reads as no turns; good lines read back as records for the triage account", () => {
|
||||
const file = fileIn();
|
||||
assert.deepEqual(readUsage(file), { records: [], bad: [] });
|
||||
appendUsage(file, line());
|
||||
appendUsage(file, unknownLine({ turn: "owner-0123456789ac" }));
|
||||
const { records, bad } = readUsage(file);
|
||||
assert.deepEqual(bad, []);
|
||||
assert.deepEqual(records, [{ account: "triage", ...line() }, { account: "triage", ...unknownLine({ turn: "owner-0123456789ac" }) }]);
|
||||
});
|
||||
|
||||
test("a torn last line is refused and counts as one unknown turn; a wrong-shaped line the same", () => {
|
||||
const file = fileIn();
|
||||
appendUsage(file, line());
|
||||
writeFileSync(file, '{"at":"2026-11-18T13:00:00.000Z","run":"triage-2026', { flag: "a" });
|
||||
const torn = readUsage(file);
|
||||
assert.equal(torn.records.length, 2);
|
||||
assert.deepEqual(torn.bad.map((b) => b.line), [2]);
|
||||
const [, b] = torn.records;
|
||||
assert.deepEqual([b.status, b.run, b.turn, b.input, b.output, b.cacheRead, b.cacheWrite], ["unknown", null, "line-2", null, null, null, null]);
|
||||
assert.ok(!Number.isNaN(Date.parse(b.at)), "a torn last line takes the file's change time");
|
||||
const [r] = report({ limits: limitsFor(cap()), records: torn.records, now: "2026-11-18T23:00:00.000Z" });
|
||||
assert.deepEqual([r.count, r.unknownTurns], [18, 1]);
|
||||
// Closed off by the next append, it stays one unknown turn, dated by the line after it.
|
||||
appendUsage(file, line({ at: "2026-11-18T14:00:00.000Z", turn: "owner-0123456789ac", input: 1, output: 0, cacheRead: 0, cacheWrite: 0 }));
|
||||
writeFileSync(file, JSON.stringify({ ...line(), input: "9" }) + "\n", { flag: "a" });
|
||||
const later = readUsage(file);
|
||||
assert.deepEqual(later.bad.map((x) => x.line), [2, 4]);
|
||||
assert.equal(later.records[1].at, "2026-11-18T14:00:00.000Z");
|
||||
const [again] = report({ limits: limitsFor(cap()), records: later.records, now: "2026-11-18T23:00:00.000Z" });
|
||||
assert.deepEqual([again.count, again.unknownTurns], [19, 2]);
|
||||
});
|
||||
|
||||
test("the limits copy the organization shape: one daily leaf for the triage account", () => {
|
||||
assert.deepEqual(limitsFor(cap()), { tokens: { daily: { triage: cap() } } });
|
||||
const records = [{ account: "triage", ...line({ input: 100, output: 0, cacheRead: 0, cacheWrite: 0 }) }];
|
||||
assert.deepEqual(mayLaunch({ limits: limitsFor(cap()), subject: { account: "triage" }, records, now: "2026-11-18T13:00:00.000Z" }).allow, false);
|
||||
});
|
||||
|
||||
test("the cap state: under, at and over the cap, unknown turns adding nothing, and the cap's since", () => {
|
||||
const now = ms("2026-11-18T18:00:00.000Z");
|
||||
const recs = (...counts) => counts.map((n, i) => ({ account: "triage", ...line({ turn: `owner-00000000000${i}`, input: n, output: 0, cacheRead: 0, cacheWrite: 0 }) }));
|
||||
const under = capState({ cap: cap(), records: recs(40, 59), now });
|
||||
assert.equal(under.allow, true);
|
||||
assert.deepEqual(under.usageToday, { count: 99, input: 99, output: 0, cacheRead: 0, cacheWrite: 0, unknownTurns: 0 });
|
||||
assert.equal(under.resetsAt, "2026-11-19T00:00:00.000Z");
|
||||
const at = capState({ cap: cap(), records: recs(40, 60), now });
|
||||
assert.equal(at.allow, false);
|
||||
assert.equal(at.usageToday.count, 100);
|
||||
const withUnknown = capState({ cap: cap(), records: [...recs(40), { account: "triage", ...unknownLine({ turn: "owner-00000000000f" }) }], now });
|
||||
assert.deepEqual([withUnknown.allow, withUnknown.usageToday.count, withUnknown.usageToday.unknownTurns], [true, 40, 1]);
|
||||
// Lines before the cap's since or before today's reset do not count.
|
||||
const old = [{ account: "triage", ...line({ at: "2026-11-17T23:59:59.999Z", input: 500, output: 0, cacheRead: 0, cacheWrite: 0 }) }];
|
||||
assert.equal(capState({ cap: cap(), records: old, now }).usageToday.count, 0);
|
||||
const since = [{ account: "triage", ...line({ at: "2026-11-18T10:00:00.000Z", input: 500, output: 0, cacheRead: 0, cacheWrite: 0 }) }];
|
||||
assert.equal(capState({ cap: cap({ since: "2026-11-18T11:00:00.000Z" }), records: since, now }).allow, true);
|
||||
// A cap of 0 takes no turn, as the plugin's own rule says.
|
||||
assert.equal(capState({ cap: cap({ limit: 0 }), records: [], now }).allow, false);
|
||||
});
|
||||
|
||||
test("the reset is 00:00 in the stored zone: the next one is the time shown and the day turns over there", () => {
|
||||
const z = (timeZone) => cap({ timeZone });
|
||||
assert.equal(new Date(nextReset(z("UTC"), ms("2026-11-18T18:00:00.000Z"))).toISOString(), "2026-11-19T00:00:00.000Z");
|
||||
assert.equal(new Date(nextReset(z("UTC"), ms("2026-11-18T00:00:00.000Z"))).toISOString(), "2026-11-19T00:00:00.000Z", "at the reset, the next is a day on");
|
||||
assert.equal(new Date(nextReset(z("UTC"), ms("2026-11-18T23:59:59.999Z"))).toISOString(), "2026-11-19T00:00:00.000Z");
|
||||
assert.equal(new Date(nextReset(z("Asia/Kolkata"), ms("2026-11-18T18:00:00.000Z"))).toISOString(), "2026-11-18T18:30:00.000Z");
|
||||
// A since after now: the reset after it.
|
||||
assert.equal(new Date(nextReset(cap({ since: "2026-11-19T06:00:00.000Z" }), ms("2026-11-18T18:00:00.000Z"))).toISOString(), "2026-11-20T00:00:00.000Z");
|
||||
});
|
||||
|
||||
test("a DST day: America/Chicago falls back on 2026-11-01 (25 hours) and springs forward on 2026-03-08 (23 hours)", () => {
|
||||
const chicago = cap({ timeZone: "America/Chicago", since: "2026-01-01T00:00:00.000Z", limit: 50 });
|
||||
// Fall back: the day runs from 05:00Z (00:00 CDT) to 06:00Z next day (00:00 CST).
|
||||
assert.equal(new Date(nextReset(chicago, ms("2026-11-01T12:00:00.000Z"))).toISOString(), "2026-11-02T06:00:00.000Z");
|
||||
assert.equal(new Date(nextReset(chicago, ms("2026-11-01T05:30:00.000Z"))).toISOString(), "2026-11-02T06:00:00.000Z", "inside the repeated hour's day");
|
||||
const fall = [
|
||||
{ account: "triage", ...line({ at: "2026-11-01T04:59:59.999Z", turn: "owner-000000000001", input: 30, output: 0, cacheRead: 0, cacheWrite: 0 }) },
|
||||
{ account: "triage", ...line({ at: "2026-11-01T05:00:00.000Z", turn: "owner-000000000002", input: 30, output: 0, cacheRead: 0, cacheWrite: 0 }) },
|
||||
{ account: "triage", ...line({ at: "2026-11-02T05:30:00.000Z", turn: "owner-000000000003", input: 30, output: 0, cacheRead: 0, cacheWrite: 0 }) },
|
||||
];
|
||||
const lateFall = capState({ cap: chicago, records: fall, now: ms("2026-11-02T05:59:59.999Z") });
|
||||
assert.deepEqual([lateFall.allow, lateFall.usageToday.count, lateFall.resetsAt], [false, 60, "2026-11-02T06:00:00.000Z"], "the 25th hour still counts the day");
|
||||
const nextDay = capState({ cap: chicago, records: fall, now: ms("2026-11-02T06:00:00.000Z") });
|
||||
assert.deepEqual([nextDay.allow, nextDay.usageToday.count, nextDay.resetsAt], [true, 0, "2026-11-03T06:00:00.000Z"]);
|
||||
// Spring forward: the day runs from 06:00Z (00:00 CST) to 05:00Z next day (00:00 CDT).
|
||||
assert.equal(new Date(nextReset(chicago, ms("2026-03-08T12:00:00.000Z"))).toISOString(), "2026-03-09T05:00:00.000Z");
|
||||
const spring = [{ account: "triage", ...line({ at: "2026-03-08T06:00:00.000Z", input: 60, output: 0, cacheRead: 0, cacheWrite: 0 }) }];
|
||||
assert.equal(capState({ cap: chicago, records: spring, now: ms("2026-03-09T04:59:59.999Z") }).allow, false);
|
||||
assert.equal(capState({ cap: chicago, records: spring, now: ms("2026-03-09T05:00:00.000Z") }).allow, true);
|
||||
});
|
||||
|
||||
test("the turn delta keeps the bus's rules: new entries summed, a rewritten or missing baseline unknown, no new usage unknown", () => {
|
||||
const e = (input) => ({ input, output: 1, cacheWrite: 0, cacheRead: 0 });
|
||||
const begin = snapshot({ status: "known", entries: { a: e(5) } });
|
||||
assert.deepEqual(delta(begin, snapshot({ status: "known", entries: { a: e(5), b: e(7), c: e(1) } })), { status: "known", input: 8, output: 2, cacheWrite: 0, cacheRead: 0 });
|
||||
assert.deepEqual(delta(begin, snapshot({ status: "known", entries: { a: e(5) } })), { status: "unknown", code: "no-new-usage" });
|
||||
assert.deepEqual(delta(begin, snapshot({ status: "known", entries: { a: e(6), b: e(1) } })), { status: "unknown", code: "transcript-rewritten" });
|
||||
assert.deepEqual(delta(begin, snapshot({ status: "known", entries: { b: e(1) } })), { status: "unknown", code: "transcript-rewritten" });
|
||||
assert.deepEqual(delta(null, snapshot({ status: "known", entries: { b: e(1) } })), { status: "unknown", code: "baseline-unknown" });
|
||||
assert.deepEqual(delta(snapshot({ status: "unknown", code: "transcript-size" }), snapshot({ status: "known", entries: {} })), { status: "unknown", code: "baseline-unknown" });
|
||||
assert.deepEqual(delta(begin, snapshot({ status: "unknown", code: "transcript-parse" })), { status: "unknown", code: "transcript-parse" });
|
||||
assert.deepEqual(snapshot({ status: "unknown", code: "anything" }), { status: "unknown", code: "source-unknown" });
|
||||
assert.deepEqual(snapshot({ status: "known", entries: { a: { input: -1, output: 0, cacheWrite: 0, cacheRead: 0 } } }), { status: "unknown", code: "source-invalid" });
|
||||
assert.deepEqual(snapshot({ status: "known", entries: { a: { ...e(1), extra: 0 } } }), { status: "unknown", code: "source-invalid" });
|
||||
assert.deepEqual(snapshot(null), { status: "unknown", code: "source-invalid" });
|
||||
const big = { status: "known", entries: { a: e(Number.MAX_SAFE_INTEGER), b: e(1) } };
|
||||
assert.deepEqual(delta(snapshot({ status: "known", entries: {} }), snapshot(big)), { status: "unknown", code: "usage-overflow" });
|
||||
});
|
||||
Reference in new issue
Block a user