feat(cli): mosaic triage key issue|rotate|retire|status, Triage's own model proxy key (#1873) #1991
No files matched your search
@@ -1452,3 +1452,73 @@ the model key against a catalogue yet.
|
||||
|
||||
The Console does not read this file: the Triage host keeps `route: null` until
|
||||
the wiring slice.
|
||||
|
||||
## Triage key (#1873)
|
||||
|
||||
`mosaic triage key issue|rotate|retire|status` manage Triage's own model
|
||||
proxy client key. Every verb refuses inside an agent run before anything is
|
||||
read (exit 3), and takes no options.
|
||||
|
||||
The key is `mst-<tag>-triage-<43 base64url>`, where `<tag>` is the first 12
|
||||
hex of sha256 of the config directory, the tag on the instance's own `msk-`
|
||||
sentinel. An org bus reconciles only its `mso-<tag>-<org>-key-` names and a
|
||||
no-org bus only its `msk-<tag>-` names, so neither removes it. The file is
|
||||
`<configDir>/secrets/triage/model`: `{version, key, keyId, issuedAt,
|
||||
rotateBy, retiring}`. The directory is 0700 and the file 0600, owned by the
|
||||
running user. A link, a non-file, a second hard link, a wrong mode or owner, a
|
||||
file over 64 KiB, or a record that does not match the shape refuses every verb
|
||||
with `triage-key-unreadable` (exit 3).
|
||||
|
||||
- `issue` adds one key and writes the file. It refuses `triage-key-issued`
|
||||
when a key is already issued; that is `rotate`.
|
||||
- `rotate` adds a new key and moves the old one to `retiring`, with a retire
|
||||
time of now plus `rotateGraceMinutes` from proxy.json, where the bus reads
|
||||
it (default 120). `rotateBy` is 30 days on, as for an org key. An old key
|
||||
the proxy no longer holds is dropped at once.
|
||||
- `retire` removes retiring keys whose grace has passed or that the proxy no
|
||||
longer holds. It changes nothing before then.
|
||||
|
||||
Nothing runs on a timer. The retire time is the earliest a retiring key may
|
||||
go; the key stays live in the proxy until someone runs `retire` (or a later
|
||||
`rotate` or `issue`) after it. `rotateBy` works the same way: `status` marks
|
||||
it due, and only `rotate` replaces the key.
|
||||
- `status` prints whether a key is issued, its key id, its age, its rotate-by
|
||||
day, each retiring key, what the proxy list holds, a set fence, any key in
|
||||
the Triage namespace the file does not hold, and the next command. It takes
|
||||
no lock and writes nothing.
|
||||
|
||||
`issue`, `rotate` and `retire` hold the instance proxy lock that org buses
|
||||
take (`<dataRoot>/proxy`). Each reads the list, refuses without this
|
||||
instance's sentinel (`triage-key-proxy-open`), reads it again and refuses if
|
||||
it moved (`triage-key-raced`), sets the fence, writes, reads back and clears
|
||||
the fence. The proxy changes first and the file second. Each run that
|
||||
changes something appends one line of key ids, never keys, to
|
||||
`<dataRoot>/proxy/mosaic-keys.receipts.jsonl`. That file is checked under the
|
||||
lock before anything changes; an unsafe one refuses
|
||||
`triage-key-receipts-unsafe`. A receipt line that still fails after the change
|
||||
refuses `triage-key-unrecorded`. No output, error or receipt holds a key.
|
||||
|
||||
The file is written to a 0600 temp file in the same directory, synced, renamed
|
||||
into place, and the directory is synced. A failed directory sync after the
|
||||
rename refuses `triage-key-unsynced`: the file is in place but may not survive
|
||||
a power loss. `proxy-lock-busy` means another proxy writer holds the lock; if
|
||||
none is running, `mosaic proxy fence status` shows the lock state.
|
||||
|
||||
After a crash:
|
||||
|
||||
- Killed after the proxy write and before the read-back: the fence stays set
|
||||
and every proxy writer on the instance refuses `proxy-uncertain`. `status`
|
||||
shows the fence. Restart the proxy, run `mosaic proxy fence clear
|
||||
--proxy-restarted`, then run the verb again.
|
||||
- Killed after the read-back and before the file write: the proxy holds a key
|
||||
the file does not. `status` names it; the next `issue`, `rotate` or `retire`
|
||||
removes it. A file that could not be written refuses
|
||||
`triage-key-unwritten` and says to run the verb again.
|
||||
- Killed after the temp file is written and before its rename: a
|
||||
`.model.<pid>.<hex>.tmp` file that may hold a key stays in the triage
|
||||
directory. `status` counts it; the next `issue`, `rotate` or `retire`
|
||||
removes it under the lock.
|
||||
- A `retire` killed after the proxy write leaves a retiring key the proxy no
|
||||
longer holds. `status` marks it; the next `retire` drops it from the file.
|
||||
|
||||
Nothing gives the key to the Triage host yet; that is the wiring slice.
|
||||
@@ -0,0 +1,392 @@
|
||||
// #1873 triage-key: Triage's own model proxy client key, issued from the owner's terminal
|
||||
// (design report reports/triage-route-design.md slice 3, Jason's ruling 2026-11-18).
|
||||
//
|
||||
// <configDir>/secrets/triage/model (directory 0700, file 0600, the running user's)
|
||||
// { version, key, keyId, issuedAt, rotateBy, retiring: [{ key, keyId, issuedAt, retireAt }] }
|
||||
//
|
||||
// The key is `mst-<instance tag>-triage-<43 base64url>`, where the tag is the first 12 hex of
|
||||
// sha256(configDir), the tag the instance's own `msk-` sentinel carries. An org bus reconciles only
|
||||
// its own `mso-<tag>-<org>-key-` names and a no-org bus only `msk-<tag>-`, so neither touches it.
|
||||
//
|
||||
// Every proxy write runs under the instance proxy lock (packages/bus/src/proxy-lock.mjs), the lock
|
||||
// org buses take: read the list, check this instance's sentinel, fence, PUT, read back, confirm.
|
||||
// The proxy changes first and the file second, as the bus does for an agent key. A crash between
|
||||
// leaves a key in the Triage namespace that the file does not hold; the next issue, rotate or retire
|
||||
// removes it, and status names it. A crash after the PUT and before the read-back leaves the lock's
|
||||
// uncertain fence, which every proxy writer on the instance refuses until a human clears it
|
||||
// (`mosaic proxy fence clear --proxy-restarted`).
|
||||
//
|
||||
// Nothing runs on a timer. rotate records when the old key's grace ends; the key stays in the proxy
|
||||
// until a later retire, rotate or issue removes it, and status says when that is due. The same goes
|
||||
// for rotateBy: status marks it due; only rotate replaces the key.
|
||||
//
|
||||
// Nothing here prints, logs or records a key. Receipts and output carry key ids only (the first 12
|
||||
// hex of sha256(key), the id the bus uses). Messages name files and options, never values.
|
||||
// Nothing wires the key into the Triage host or the Console; that is a later slice.
|
||||
import { closeSync, constants, fchmodSync, fstatSync, fsyncSync, lstatSync, mkdirSync, openSync, readdirSync, readFileSync, renameSync, unlinkSync, writeSync } from "node:fs";
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
import { join } from "node:path";
|
||||
import { isDeepStrictEqual } from "node:util";
|
||||
import { createProxyLock } from "../../bus/src/proxy-lock.mjs";
|
||||
import { loadManagementClient } from "../../model-proxy/src/client.mjs";
|
||||
|
||||
export const KEY_VERSION = 1;
|
||||
// The org key lifetime and the rotate grace default. The bus holds both as literals
|
||||
// (packages/bus/src/model-keys.mjs #issue: 30 days to rotateBy; packages/bus/src/runtime.mjs:
|
||||
// `proxy.rotateGraceMinutes ?? 120`) and exports neither, and this lane may not change the bus.
|
||||
// packages/cli/tests/triage-key.test.mjs drives the bus's own ModelKeys with the same clock and
|
||||
// proves these give the same rotateBy and retireAt, so a change on either side goes red there.
|
||||
// The grace itself is read from proxy.json first, where the bus reads it.
|
||||
export const KEY_LIFETIME_DAYS = 30;
|
||||
export const DEFAULT_ROTATE_GRACE_MINUTES = 120;
|
||||
const FIELDS = Object.freeze(["version", "key", "keyId", "issuedAt", "rotateBy", "retiring"]);
|
||||
const RETIRING_FIELDS = Object.freeze(["key", "keyId", "issuedAt", "retireAt"]);
|
||||
const MAX_BYTES = 64 * 1024;
|
||||
const MAX_RETIRING = 16;
|
||||
const SECRET = /^[A-Za-z0-9_-]{43}$/;
|
||||
const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$/;
|
||||
const DAY = /^\d{4}-\d{2}-\d{2}$/;
|
||||
|
||||
export class TriageKeyError extends Error {
|
||||
constructor(code, message, exitCode = 3) {
|
||||
super(message);
|
||||
this.name = "TriageKeyError";
|
||||
this.code = code;
|
||||
this.exitCode = exitCode;
|
||||
}
|
||||
}
|
||||
const unreadable = (why) => new TriageKeyError("triage-key-unreadable", `the Triage key store is unreadable: ${why}`);
|
||||
|
||||
export const keyIdOf = (key) => createHash("sha256").update(key).digest("hex").slice(0, 12);
|
||||
export const instanceTag = (configDir) => createHash("sha256").update(configDir).digest("hex").slice(0, 12);
|
||||
export const keyPrefix = (configDir) => `mst-${instanceTag(configDir)}-triage-`;
|
||||
const sentinelPrefix = (configDir) => `msk-${instanceTag(configDir)}-sentinel-`;
|
||||
const ownKey = (configDir) => (k) => typeof k === "string" && k.startsWith(keyPrefix(configDir)) && SECRET.test(k.slice(keyPrefix(configDir).length));
|
||||
// Any name under the prefix is Triage's to remove, well formed or not, as the bus treats its own
|
||||
// `msk-<tag>-` names. Only a well-formed key may be stored in the file.
|
||||
const inNamespace = (configDir) => (k) => typeof k === "string" && k.startsWith(keyPrefix(configDir));
|
||||
|
||||
export function keyPaths(configDir) {
|
||||
const secrets = join(configDir, "secrets");
|
||||
const dir = join(secrets, "triage");
|
||||
return { secrets, dir, file: join(dir, "model") };
|
||||
}
|
||||
|
||||
const plain = (v) => v !== null && typeof v === "object" && !Array.isArray(v) && Object.getPrototypeOf(v) === Object.prototype;
|
||||
const exactKeys = (v, keys) => plain(v) && Object.keys(v).length === keys.length && keys.every((k) => Object.hasOwn(v, k));
|
||||
const canonicalTime = (s) => typeof s === "string" && ISO.test(s) && !Number.isNaN(Date.parse(s)) && new Date(s).toISOString() === s;
|
||||
const canonicalDay = (s) => typeof s === "string" && DAY.test(s) && !Number.isNaN(Date.parse(`${s}T00:00:00Z`)) && new Date(`${s}T00:00:00Z`).toISOString().slice(0, 10) === s;
|
||||
|
||||
export function validRecord(doc, configDir) {
|
||||
const own = ownKey(configDir);
|
||||
return exactKeys(doc, FIELDS)
|
||||
&& doc.version === KEY_VERSION
|
||||
&& own(doc.key) && doc.keyId === keyIdOf(doc.key)
|
||||
&& canonicalTime(doc.issuedAt) && canonicalDay(doc.rotateBy)
|
||||
&& Array.isArray(doc.retiring) && doc.retiring.length <= MAX_RETIRING
|
||||
&& doc.retiring.every((r) => exactKeys(r, RETIRING_FIELDS) && own(r.key) && r.key !== doc.key && r.keyId === keyIdOf(r.key)
|
||||
&& canonicalTime(r.issuedAt) && canonicalTime(r.retireAt))
|
||||
&& new Set(doc.retiring.map((r) => r.key)).size === doc.retiring.length;
|
||||
}
|
||||
|
||||
const uidOf = (opts) => opts.uid ?? process.getuid();
|
||||
|
||||
// A real directory, the running user's, mode 0700. Missing: false, or made when `create` is on.
|
||||
function checkDir(dir, uid, what, { create }) {
|
||||
let st;
|
||||
try { st = lstatSync(dir); }
|
||||
catch (e) {
|
||||
if (e.code !== "ENOENT") throw unreadable(`${what} cannot be read`);
|
||||
if (!create) return false;
|
||||
try { mkdirSync(dir, { mode: 0o700 }); } catch { throw unreadable(`${what} cannot be made`); }
|
||||
try {
|
||||
const fd = openSync(dir, constants.O_RDONLY | constants.O_DIRECTORY | constants.O_NOFOLLOW);
|
||||
try { fchmodSync(fd, 0o700); } finally { closeSync(fd); }
|
||||
} catch { throw unreadable(`${what} cannot be made`); }
|
||||
st = lstatSync(dir);
|
||||
}
|
||||
if (st.isSymbolicLink()) throw unreadable(`${what} is a link`);
|
||||
if (!st.isDirectory()) throw unreadable(`${what} is not a directory`);
|
||||
if (st.uid !== uid) throw unreadable(`${what} is not owned by this user`);
|
||||
if ((st.mode & 0o777) !== 0o700) throw unreadable(`${what} mode is not 0700`);
|
||||
return true;
|
||||
}
|
||||
|
||||
function checkFile(fd, uid, what, fail = unreadable) {
|
||||
const st = fstatSync(fd);
|
||||
if (!st.isFile()) throw fail(`${what} is not a regular file`);
|
||||
if (st.nlink !== 1) throw fail(`${what} has more than one link`);
|
||||
if (st.uid !== uid) throw fail(`${what} is not owned by this user`);
|
||||
if ((st.mode & 0o777) !== 0o600) throw fail(`${what} mode is not 0600`);
|
||||
return st;
|
||||
}
|
||||
|
||||
// The stored record, or null when no key is issued. Throws triage-key-unreadable on anything unsafe.
|
||||
// opts.uid (both directories and the file) and opts.fileUid (the file only) are test seams for the
|
||||
// owner checks.
|
||||
export function readKey(configDir, opts = {}) {
|
||||
const { secrets, dir, file } = keyPaths(configDir);
|
||||
const uid = uidOf(opts);
|
||||
if (!checkDir(secrets, uid, "the secrets directory", { create: false })) return null;
|
||||
if (!checkDir(dir, uid, "the triage secrets directory", { create: false })) return null;
|
||||
let fd;
|
||||
try { fd = openSync(file, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); }
|
||||
catch (e) {
|
||||
if (e.code === "ENOENT") return null;
|
||||
if (e.code === "ELOOP") throw unreadable("the key file is a link");
|
||||
throw unreadable("the key file cannot be opened");
|
||||
}
|
||||
try {
|
||||
const st = checkFile(fd, opts.fileUid ?? uid, "the key file");
|
||||
if (st.size > MAX_BYTES) throw unreadable("the key file is too large");
|
||||
let doc;
|
||||
try { doc = JSON.parse(readFileSync(fd, "utf8")); } catch { throw unreadable("the key file is not JSON"); }
|
||||
if (!validRecord(doc, configDir)) throw unreadable("the key file does not match the key shape");
|
||||
return doc;
|
||||
} finally { closeSync(fd); }
|
||||
}
|
||||
|
||||
// Both directories, checked (and the triage one made) before any proxy write, so a store that cannot
|
||||
// be written refuses while the proxy is still untouched.
|
||||
function prepareDirs(configDir, uid) {
|
||||
const { secrets, dir } = keyPaths(configDir);
|
||||
if (!checkDir(secrets, uid, "the secrets directory", { create: false })) throw unreadable("the secrets directory is missing");
|
||||
checkDir(dir, uid, "the triage secrets directory", { create: true });
|
||||
}
|
||||
|
||||
// A temp file writeKey left when it was killed before its rename. It may hold a key.
|
||||
const TEMP = /^\.model\.\d+\.[0-9a-f]{12}\.tmp$/;
|
||||
const tempsIn = (dir) => { try { return readdirSync(dir).filter((n) => TEMP.test(n)); } catch { return []; } };
|
||||
// Under the lock no other Triage writer runs, so every temp file left in the directory is dead.
|
||||
function removeTemps(configDir) {
|
||||
const { dir } = keyPaths(configDir);
|
||||
for (const name of tempsIn(dir)) {
|
||||
try { if (!lstatSync(join(dir, name)).isDirectory()) unlinkSync(join(dir, name)); }
|
||||
catch { throw unreadable("a leftover temp file in the triage secrets directory cannot be removed"); }
|
||||
}
|
||||
}
|
||||
|
||||
// A directory sync that failed after the rename: the file is in place but may not be durable.
|
||||
class NotSynced extends Error {}
|
||||
|
||||
// Atomic: a 0600 temp file in the same directory, fsync, rename, fsync of the directory.
|
||||
function writeKey(configDir, record, uid, hook = () => {}) {
|
||||
const { dir, file } = keyPaths(configDir);
|
||||
prepareDirs(configDir, uid);
|
||||
const temp = join(dir, `.model.${process.pid}.${randomBytes(6).toString("hex")}.tmp`);
|
||||
let fd;
|
||||
try { fd = openSync(temp, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW, 0o600); }
|
||||
catch { throw unreadable("the key file cannot be written"); }
|
||||
try {
|
||||
try { fchmodSync(fd, 0o600); writeSync(fd, `${JSON.stringify(record, null, 2)}\n`); fsyncSync(fd); }
|
||||
finally { closeSync(fd); }
|
||||
hook("rename");
|
||||
renameSync(temp, file);
|
||||
} catch { try { unlinkSync(temp); } catch {} throw unreadable("the key file cannot be written"); }
|
||||
try {
|
||||
hook("sync");
|
||||
const dfd = openSync(dir, constants.O_RDONLY | constants.O_DIRECTORY);
|
||||
try { fsyncSync(dfd); } finally { closeSync(dfd); }
|
||||
} catch { throw new NotSynced(); }
|
||||
}
|
||||
|
||||
// One line per write in the proxy lock's receipts file, ids only. `receiptsReady` runs under the
|
||||
// lock before any change, so an unsafe receipts file refuses while nothing has changed yet.
|
||||
const RECEIPTS_FLAGS = constants.O_WRONLY | constants.O_APPEND | constants.O_CREAT | constants.O_NOFOLLOW | constants.O_NONBLOCK;
|
||||
const receiptsUnsafe = (why) => new TriageKeyError("triage-key-receipts-unsafe", `${why}; nothing was changed`);
|
||||
function receiptsReady(stateDir, uid) {
|
||||
let fd;
|
||||
try { fd = openSync(join(stateDir, "mosaic-keys.receipts.jsonl"), RECEIPTS_FLAGS, 0o600); }
|
||||
catch { throw receiptsUnsafe("the proxy receipts file cannot be opened"); }
|
||||
try { checkFile(fd, uid, "the proxy receipts file", receiptsUnsafe); } finally { closeSync(fd); }
|
||||
}
|
||||
function receipt(stateDir, uid, entry) {
|
||||
const unrecorded = new TriageKeyError("triage-key-unrecorded", "the change was made but its receipt line was not written; run mosaic triage key status");
|
||||
let fd;
|
||||
try { fd = openSync(join(stateDir, "mosaic-keys.receipts.jsonl"), RECEIPTS_FLAGS, 0o600); } catch { throw unrecorded; }
|
||||
try {
|
||||
checkFile(fd, uid, "the proxy receipts file", () => unrecorded);
|
||||
writeSync(fd, `${JSON.stringify(entry)}\n`);
|
||||
fsyncSync(fd);
|
||||
} catch { throw unrecorded; }
|
||||
finally { closeSync(fd); }
|
||||
}
|
||||
|
||||
// The grace the bus uses: proxy.json's rotateGraceMinutes, or the bus default.
|
||||
export const graceMinutes = (proxy) => proxy?.rotateGraceMinutes ?? DEFAULT_ROTATE_GRACE_MINUTES;
|
||||
export const rotateByFor = (nowMs) => new Date(nowMs + KEY_LIFETIME_DAYS * 86400_000).toISOString().slice(0, 10);
|
||||
export const retireAtFor = (nowMs, proxy) => new Date(nowMs + graceMinutes(proxy) * 60_000).toISOString();
|
||||
|
||||
const proxyFailure = (code) => {
|
||||
const words = {
|
||||
"proxy-lock-busy": "another proxy writer holds the instance proxy lock; try again when it ends, and if no writer is running, see mosaic proxy fence status",
|
||||
"proxy-lock-uncertain": "the instance proxy lock is in a state this verb will not guess about; check <dataRoot>/proxy by hand",
|
||||
"proxy-lock-invalid": "the instance proxy lock directory is missing or unsafe, or names another proxy",
|
||||
"proxy-uncertain": "an earlier proxy write has an uncertain outcome; see mosaic proxy fence status",
|
||||
"proxy-management-key-invalid": "the proxy management key file is missing or unsafe",
|
||||
"proxy-client-invalid": "the proxy settings are not usable",
|
||||
"proxy-client-down": "the proxy did not answer",
|
||||
"proxy-client-refused": "the proxy refused the management request",
|
||||
"proxy-client-redirect": "the proxy answered with a redirect",
|
||||
"proxy-client-response": "the proxy gave an answer this verb cannot use",
|
||||
};
|
||||
return new TriageKeyError(code, words[code] ?? "the proxy write failed");
|
||||
};
|
||||
|
||||
// opts.uid, opts.now and opts.hook(stage) are test seams; a stage hook may throw or kill the process.
|
||||
function context({ configDir, system, uid, fileUid, now, hook }) {
|
||||
if (!system?.proxy || !Number.isInteger(system.proxy.port)) throw new TriageKeyError("triage-key-no-proxy", "the instance has no model proxy set up (proxy.json)");
|
||||
return {
|
||||
configDir, dataRoot: system.dataRoot, proxy: system.proxy, stateDir: join(system.dataRoot, "proxy"),
|
||||
uid: uid ?? process.getuid(), fileUid, now: now ?? Date.now, hook: hook ?? (() => {}),
|
||||
};
|
||||
}
|
||||
|
||||
function lockFor(c) {
|
||||
try { return createProxyLock({ stateDir: c.stateDir, identity: `${c.configDir}:${c.proxy.port}` }); }
|
||||
catch (e) { throw proxyFailure(e.code ?? "proxy-lock-invalid"); }
|
||||
}
|
||||
|
||||
async function clientFor(c) {
|
||||
try { return await loadManagementClient({ configDir: c.configDir, proxy: c.proxy }); }
|
||||
catch (e) { throw proxyFailure(e.code ?? "proxy-management-key-invalid"); }
|
||||
}
|
||||
|
||||
function checkSentinel(c, keys) {
|
||||
const prefix = sentinelPrefix(c.configDir);
|
||||
if (!keys.some((k) => k.startsWith(prefix) && SECRET.test(k.slice(prefix.length)))) {
|
||||
throw new TriageKeyError("triage-key-proxy-open", "the proxy's key list has no sentinel for this instance, so it may be another proxy");
|
||||
}
|
||||
}
|
||||
|
||||
// Under the instance proxy lock: read the list, check the sentinel, run `plan`, write the result.
|
||||
// `plan(keys, record)` returns { after, record, removed, added, done }; `record` null leaves the
|
||||
// file alone. The proxy changes first, then the file, then one receipt line.
|
||||
async function change(c, op, plan) {
|
||||
const lock = lockFor(c);
|
||||
const client = await clientFor(c);
|
||||
try {
|
||||
return await lock.run(async () => {
|
||||
const before = await client.getKeys();
|
||||
checkSentinel(c, before);
|
||||
const current = readKey(c.configDir, { uid: c.uid, fileUid: c.fileUid });
|
||||
const result = plan(before, current);
|
||||
if (result.record) prepareDirs(c.configDir, c.uid);
|
||||
removeTemps(c.configDir);
|
||||
receiptsReady(c.stateDir, c.uid);
|
||||
const after = result.after;
|
||||
const removed = before.filter((k) => !after.includes(k));
|
||||
const added = after.filter((k) => !before.includes(k));
|
||||
// A key in another namespace never leaves the list.
|
||||
if (removed.some((k) => !inNamespace(c.configDir)(k))) throw new TriageKeyError("triage-key-failed", "the change would remove a key outside the Triage namespace");
|
||||
if (added.length || removed.length) {
|
||||
// As the bus does: a list that moved since the first read is not overwritten.
|
||||
if (!isDeepStrictEqual(await client.getKeys(), before)) throw new TriageKeyError("triage-key-raced", "the proxy's key list changed while this verb ran; run it again");
|
||||
lock.fence({ org: null, op: `triage.${op}` });
|
||||
await client.putKeys(after);
|
||||
c.hook("put");
|
||||
const confirmed = await client.getKeys();
|
||||
if (!confirmed.some((k) => k.startsWith(sentinelPrefix(c.configDir))) || added.some((k) => !confirmed.includes(k)) || removed.some((k) => confirmed.includes(k))) {
|
||||
throw new TriageKeyError("triage-key-unconfirmed", "the proxy did not confirm the change; the instance proxy fence stays set, see mosaic proxy fence status");
|
||||
}
|
||||
lock.confirm();
|
||||
}
|
||||
c.hook("write");
|
||||
let unsynced = false;
|
||||
if (result.record) {
|
||||
try { writeKey(c.configDir, result.record, c.uid, c.hook); }
|
||||
catch (e) {
|
||||
if (e instanceof NotSynced) unsynced = true;
|
||||
else throw new TriageKeyError("triage-key-unwritten", `the proxy list changed but ${keyPaths(c.configDir).file} was not written (${e.message}); fix the file's directory, then run mosaic triage key ${op} again, which clears what this run left`);
|
||||
}
|
||||
}
|
||||
c.hook("receipt");
|
||||
// A run that changed neither the proxy nor the file leaves no receipt.
|
||||
if (added.length || removed.length || result.record) receipt(c.stateDir, c.uid, { op: `triage.key.${op}`, time: new Date(c.now()).toISOString(), addedKeyIds: added.map(keyIdOf), removedKeyIds: removed.map(keyIdOf) });
|
||||
// The file is in place and the receipt written; only its durability is in doubt.
|
||||
if (unsynced) throw new TriageKeyError("triage-key-unsynced", `${keyPaths(c.configDir).file} was written but its directory could not be synced, so it may not survive a power loss; run mosaic triage key status`);
|
||||
return { ...result.done, addedKeyIds: added.map(keyIdOf), removedKeyIds: removed.map(keyIdOf) };
|
||||
});
|
||||
} catch (e) {
|
||||
if (e instanceof TriageKeyError) throw e;
|
||||
throw proxyFailure(typeof e?.code === "string" && /^proxy-[a-z-]+$/.test(e.code) ? e.code : "triage-key-failed");
|
||||
}
|
||||
}
|
||||
|
||||
const newKey = (configDir) => keyPrefix(configDir) + randomBytes(32).toString("base64url");
|
||||
|
||||
// Keys in the Triage namespace that the file does not hold: left by a crash between the PUT and the
|
||||
// file write, or by a file that was removed by hand.
|
||||
const orphansOf = (configDir, keys, record) => {
|
||||
const held = record ? [record.key, ...record.retiring.map((r) => r.key)] : [];
|
||||
return keys.filter((k) => inNamespace(configDir)(k) && !held.includes(k));
|
||||
};
|
||||
|
||||
export async function issueKey(opts) {
|
||||
const c = context(opts);
|
||||
return change(c, "issue", (before, current) => {
|
||||
if (current) throw new TriageKeyError("triage-key-issued", "a Triage key is already issued; use mosaic triage key rotate");
|
||||
const orphans = orphansOf(c.configDir, before, null);
|
||||
const key = newKey(c.configDir), nowMs = c.now();
|
||||
const record = { version: KEY_VERSION, key, keyId: keyIdOf(key), issuedAt: new Date(nowMs).toISOString(), rotateBy: rotateByFor(nowMs), retiring: [] };
|
||||
return { after: [...before.filter((k) => !orphans.includes(k)), key], record, done: { keyId: record.keyId, rotateBy: record.rotateBy } };
|
||||
});
|
||||
}
|
||||
|
||||
// Retiring keys whose grace has passed, or that the proxy no longer holds.
|
||||
const dueOf = (record, keys, nowMs) => record.retiring.filter((r) => Date.parse(r.retireAt) <= nowMs || !keys.includes(r.key));
|
||||
|
||||
export async function rotateKey(opts) {
|
||||
const c = context(opts);
|
||||
return change(c, "rotate", (before, current) => {
|
||||
if (!current) throw new TriageKeyError("triage-key-not-issued", "no Triage key is issued; use mosaic triage key issue");
|
||||
const nowMs = c.now();
|
||||
const orphans = orphansOf(c.configDir, before, current);
|
||||
const due = dueOf(current, before, nowMs);
|
||||
const keep = before.includes(current.key);
|
||||
const retiring = [...current.retiring.filter((r) => !due.includes(r)),
|
||||
...(keep ? [{ key: current.key, keyId: current.keyId, issuedAt: current.issuedAt, retireAt: retireAtFor(nowMs, c.proxy) }] : [])];
|
||||
if (retiring.length > MAX_RETIRING) throw new TriageKeyError("triage-key-failed", "too many keys are retiring; run mosaic triage key retire once their grace has passed");
|
||||
const key = newKey(c.configDir);
|
||||
const record = { version: KEY_VERSION, key, keyId: keyIdOf(key), issuedAt: new Date(nowMs).toISOString(), rotateBy: rotateByFor(nowMs), retiring };
|
||||
const drop = [...orphans, ...due.map((r) => r.key)];
|
||||
return {
|
||||
after: [...before.filter((k) => !drop.includes(k)), key], record,
|
||||
done: { keyId: record.keyId, rotateBy: record.rotateBy, old: keep ? { keyId: current.keyId, retireAt: retiring.at(-1).retireAt } : { keyId: current.keyId, lost: true } },
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
export async function retireKeys(opts) {
|
||||
const c = context(opts);
|
||||
return change(c, "retire", (before, current) => {
|
||||
const nowMs = c.now();
|
||||
const orphans = orphansOf(c.configDir, before, current);
|
||||
const due = current ? dueOf(current, before, nowMs) : [];
|
||||
const record = current && due.length ? { ...current, retiring: current.retiring.filter((r) => !due.includes(r)) } : null;
|
||||
const drop = [...orphans, ...due.map((r) => r.key)];
|
||||
return { after: before.filter((k) => !drop.includes(k)), record, done: { droppedKeyIds: due.map((r) => r.keyId) } };
|
||||
});
|
||||
}
|
||||
|
||||
// Read only: the file, and when the proxy is set up, what its list holds. No lock, no write.
|
||||
// The fence is checked by name only, so status writes nothing, not even the lock's identity file.
|
||||
export async function keyStatus(opts) {
|
||||
const { configDir, system, uid, fileUid } = opts;
|
||||
const record = readKey(configDir, { uid, fileUid });
|
||||
const proxy = { state: "not-set-up", fence: false, leftoverTemps: tempsIn(keyPaths(configDir).dir).length };
|
||||
if (system?.proxy && Number.isInteger(system.proxy.port)) {
|
||||
try { lstatSync(join(system.dataRoot, "proxy", "mosaic-keys.uncertain")); proxy.fence = true; }
|
||||
catch (e) { if (e.code !== "ENOENT") proxy.fence = true; }
|
||||
try {
|
||||
const client = await loadManagementClient({ configDir, proxy: system.proxy });
|
||||
const keys = await client.getKeys();
|
||||
const prefix = sentinelPrefix(configDir);
|
||||
proxy.state = keys.some((k) => k.startsWith(prefix) && SECRET.test(k.slice(prefix.length))) ? "read" : "no-sentinel";
|
||||
proxy.holdsKey = record ? keys.includes(record.key) : null;
|
||||
proxy.retiringHeld = record ? record.retiring.map((r) => keys.includes(r.key)) : [];
|
||||
proxy.orphanKeyIds = orphansOf(configDir, keys, record).map(keyIdOf);
|
||||
} catch (e) { proxy.state = "unreadable"; proxy.code = typeof e?.code === "string" && /^proxy-[a-z-]+$/.test(e.code) ? e.code : "proxy-client-response"; }
|
||||
}
|
||||
return { record, proxy };
|
||||
}
|
||||
@@ -13,15 +13,23 @@
|
||||
// Nothing here wires the route into the Triage host: the Console keeps `route: null` until the
|
||||
// wiring slice. `set` checks the harness against the repository's harness modules; it does not
|
||||
// check the model key against a catalogue, since none exists outside an organization's bus yet.
|
||||
//
|
||||
// mosaic triage key issue | rotate | retire | status
|
||||
//
|
||||
// Triage's own model proxy client key (packages/cli/src/triage-key.mjs). Human verbs as well,
|
||||
// refused in an agent run before anything is read. No output names a key, only its id.
|
||||
import { lstatSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { readRoute, RouteStoreError, writeRoute } from "../../webui/src/triage-route-store.mjs";
|
||||
import { loadSystem } from "./config.mjs";
|
||||
import { configDir } from "../../organization/src/organization-config.mjs";
|
||||
import { issueKey, keyPaths, keyStatus, retireKeys, rotateKey, TriageKeyError } from "./triage-key.mjs";
|
||||
import { CliError } from "./errors.mjs";
|
||||
import { refuseInsideAgent } from "./transport.mjs";
|
||||
|
||||
export const TRIAGE_USAGE = "mosaic triage route show | set --model KEY --harness NAME --cap TOKENS --time-zone ZONE [--revision N]";
|
||||
export const TRIAGE_USAGE = "mosaic triage route show | set --model KEY --harness NAME --cap TOKENS --time-zone ZONE [--revision N] | key issue | rotate | retire | status";
|
||||
const KEY_VERBS = ["issue", "rotate", "retire", "status"];
|
||||
const HARNESS_ROOT = fileURLToPath(new URL("../../plugins/harness/", import.meta.url));
|
||||
const VALUES = ["--model", "--harness", "--cap", "--time-zone", "--revision"];
|
||||
const WHOLE = /^(0|[1-9][0-9]*)$/;
|
||||
@@ -73,10 +81,89 @@ function show(route) {
|
||||
].join("\n");
|
||||
}
|
||||
|
||||
// A key error prints its code and fixed words. Anything else becomes one fixed line, so no value
|
||||
// that an unexpected error might carry reaches the terminal.
|
||||
function keyError(error) {
|
||||
if (error instanceof CliError) return error;
|
||||
if (error instanceof TriageKeyError) return new CliError(`${error.code}: ${error.message}`, error.exitCode);
|
||||
return new CliError("triage-key-failed: the key verb stopped on an unexpected error; run mosaic triage key status", 3);
|
||||
}
|
||||
|
||||
const age = (ms) => {
|
||||
const m = Math.max(0, Math.floor(ms / 60_000));
|
||||
return `${Math.floor(m / 1440)}d ${Math.floor((m % 1440) / 60)}h ${m % 60}m`;
|
||||
};
|
||||
const ids = (list) => (list.length ? list.join(", ") : "none");
|
||||
|
||||
function statusText({ record, proxy }, nowMs, file) {
|
||||
const lines = [`triage key: ${record ? "issued" : "not issued"}`, `file: ${file}`];
|
||||
const rotateDue = record && Date.parse(`${record.rotateBy}T00:00:00Z`) <= nowMs;
|
||||
const retireDue = record ? record.retiring.filter((r) => Date.parse(r.retireAt) <= nowMs) : [];
|
||||
if (record) {
|
||||
lines.push(`key id: ${record.keyId}`, `issued: ${record.issuedAt} (age ${age(nowMs - Date.parse(record.issuedAt))})`, `rotate by: ${record.rotateBy}${rotateDue ? " (due)" : ""}`);
|
||||
record.retiring.forEach((r, i) => {
|
||||
const held = proxy.retiringHeld?.[i];
|
||||
lines.push(`retiring: ${r.keyId}, live until a retire after ${r.retireAt}${Date.parse(r.retireAt) <= nowMs ? " (due)" : ""}${held === false ? " (not in the proxy list)" : ""}`);
|
||||
});
|
||||
}
|
||||
const state = {
|
||||
"not-set-up": "not set up for this instance (no proxy.json)",
|
||||
"no-sentinel": "its key list has no sentinel for this instance",
|
||||
unreadable: `not read (${proxy.code})`,
|
||||
read: record ? (proxy.holdsKey ? "holds the key" : "does not hold the key") : "read",
|
||||
}[proxy.state];
|
||||
lines.push(`proxy: ${state}`);
|
||||
if (proxy.fence) lines.push("proxy fence: set, an earlier proxy write has an uncertain outcome");
|
||||
if (proxy.leftoverTemps) lines.push(`leftover temp files in the triage secrets directory: ${proxy.leftoverTemps} (the next issue, rotate or retire removes them)`);
|
||||
if (proxy.orphanKeyIds?.length) lines.push(`keys in the Triage namespace the file does not hold: ${ids(proxy.orphanKeyIds)}`);
|
||||
let next;
|
||||
if (proxy.fence) next = "restart the proxy, then mosaic proxy fence clear --proxy-restarted";
|
||||
else if (proxy.state !== "read") next = proxy.state === "not-set-up" ? "set up the model proxy first (mosaic proxy init)" : "check the proxy with mosaic proxy status";
|
||||
else if (!record) next = "mosaic triage key issue";
|
||||
else if (!proxy.holdsKey || rotateDue) next = "mosaic triage key rotate";
|
||||
else if (retireDue.length || proxy.leftoverTemps || proxy.orphanKeyIds?.length || proxy.retiringHeld?.includes(false)) next = "mosaic triage key retire";
|
||||
else next = "nothing";
|
||||
lines.push(`next: ${next}`, "");
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
async function keyCommand(sub, args, io, deps) {
|
||||
if (!KEY_VERBS.includes(sub)) throw usage("unknown triage key command");
|
||||
if (args.length) throw usage(args[0].startsWith("--") ? "unknown option" : "unexpected argument");
|
||||
try {
|
||||
const system = deps.system ?? loadSystem({ env: io.env });
|
||||
const dir = configDir(io.env);
|
||||
const now = deps.now ? () => deps.now().getTime() : Date.now;
|
||||
const opts = { configDir: dir, system, now, uid: deps.triageKeyUid, fileUid: deps.triageKeyFileUid, hook: deps.triageKeyHook };
|
||||
if (sub === "status") {
|
||||
io.stdout.write(statusText(await keyStatus(opts), now(), keyPaths(dir).file));
|
||||
return;
|
||||
}
|
||||
if (sub === "issue") {
|
||||
const r = await issueKey(opts);
|
||||
io.stdout.write(`triage key issued: ${r.keyId}, rotate by ${r.rotateBy}\nfile: ${keyPaths(dir).file}\n`);
|
||||
if (r.removedKeyIds.length) io.stdout.write(`removed keys this file did not hold: ${ids(r.removedKeyIds)}\n`);
|
||||
return;
|
||||
}
|
||||
if (sub === "rotate") {
|
||||
const r = await rotateKey(opts);
|
||||
io.stdout.write(`triage key rotated: ${r.keyId}, rotate by ${r.rotateBy}\n`);
|
||||
io.stdout.write(r.old.lost
|
||||
? `old key ${r.old.keyId} was not in the proxy list; nothing to retire\n`
|
||||
: `old key ${r.old.keyId} stays live until mosaic triage key retire runs at or after ${r.old.retireAt}\n`);
|
||||
if (r.removedKeyIds.length) io.stdout.write(`removed: ${ids(r.removedKeyIds)}\n`);
|
||||
return;
|
||||
}
|
||||
const r = await retireKeys(opts);
|
||||
io.stdout.write(r.removedKeyIds.length ? `triage key retire: removed ${ids(r.removedKeyIds)}\n` : "triage key retire: nothing to remove\n");
|
||||
} catch (e) { throw keyError(e); }
|
||||
}
|
||||
|
||||
export function triageCommand(rest, io, deps = {}) {
|
||||
refuseInsideAgent(process.env);
|
||||
refuseInsideAgent(io.env);
|
||||
const [noun, sub, ...args] = rest;
|
||||
if (noun === "key" && sub !== undefined) return keyCommand(sub, args, io, deps);
|
||||
if (noun !== "route" || !["show", "set"].includes(sub)) throw usage("unknown triage command");
|
||||
const { values, positional } = parse(args);
|
||||
if (positional.length) throw usage("unexpected argument");
|
||||
|
||||
@@ -0,0 +1,695 @@
|
||||
// #1873 triage-key: `mosaic triage key issue|rotate|retire|status`, Triage's own model proxy key.
|
||||
// A stand-in management server on loopback and a temporary config directory only; nothing here
|
||||
// reads a real config, proxy or secret. Every stdout, stderr, receipt and file the verbs leave is
|
||||
// scanned for every key the stand-in ever saw, at the end of each test and once more at the end.
|
||||
import { test } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import fs from "node:fs";
|
||||
import http from "node:http";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
import { run } from "../src/cli.mjs";
|
||||
import { AGENT_MARKERS } from "../src/transport.mjs";
|
||||
import { DEFAULT_ROTATE_GRACE_MINUTES, keyPaths, retireAtFor, rotateByFor } from "../src/triage-key.mjs";
|
||||
import { ModelKeys } from "../../bus/src/model-keys.mjs";
|
||||
import { ManagementClient } from "../../model-proxy/src/client.mjs";
|
||||
import { startBroker } from "../../bus/src/runtime.mjs";
|
||||
import { busOrganization } from "../../bus/src/organization-context.mjs";
|
||||
import { Registry } from "../../seat/src/session.mjs";
|
||||
import { loadBusiness } from "../../organization/src/organization-config.mjs";
|
||||
import { resolveInstance } from "../../organization/src/resolve.mjs";
|
||||
import { businessDoc, rolesCopy, systemFor, writeJson } from "../../organization/tests/helpers.mjs";
|
||||
import { humanEnv, io } from "./helpers.mjs";
|
||||
|
||||
const REPO = new URL("../../../", import.meta.url).pathname;
|
||||
const NOW = Date.parse("2026-11-18T12:00:00.000Z");
|
||||
const MIN = 60_000, HOUR = 60 * MIN, DAY = 24 * HOUR;
|
||||
const MANAGE = "fx1873-manage";
|
||||
const KEY_PATH = "/v8/management/config/access/api-keys";
|
||||
const sha = (s) => createHash("sha256").update(s).digest("hex");
|
||||
const tag = (dir) => sha(dir).slice(0, 12);
|
||||
const id = (key) => sha(key).slice(0, 12);
|
||||
const b64 = () => randomBytes(32).toString("base64url");
|
||||
const iso = (ms) => new Date(ms).toISOString();
|
||||
|
||||
// Every output any verb produced in this file, and every Triage key the stand-in ever held.
|
||||
const OUTPUTS = [];
|
||||
const SEEN = new Set();
|
||||
const VERBS = ["issue", "rotate", "retire", "status"];
|
||||
|
||||
function leaks(text, keys = SEEN) {
|
||||
for (const key of keys) {
|
||||
const secret = key.slice(-43);
|
||||
if (text.includes(key) || text.includes(secret)) return true;
|
||||
for (let i = 0; i + 8 <= secret.length; i++) if (text.includes(secret.slice(i, i + 8))) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
function walk(dir, skip, out = []) {
|
||||
for (const name of fs.readdirSync(dir)) {
|
||||
const p = join(dir, name), st = fs.lstatSync(p);
|
||||
if (st.isDirectory()) walk(p, skip, out);
|
||||
else if (st.isFile() && !skip.includes(p)) out.push(p);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function fixture(t, { sentinel = true } = {}) {
|
||||
const root = fs.realpathSync(fs.mkdtempSync(join(tmpdir(), "tk-")));
|
||||
const configDir = join(root, "config");
|
||||
fs.mkdirSync(join(configDir, "secrets"), { recursive: true, mode: 0o700 });
|
||||
fs.chmodSync(configDir, 0o700); fs.chmodSync(join(configDir, "secrets"), 0o700);
|
||||
fs.writeFileSync(join(configDir, "secrets", "proxy-management.key"), MANAGE, { mode: 0o600 });
|
||||
fs.mkdirSync(join(root, "proxy"), { mode: 0o700 });
|
||||
const prefix = `mst-${tag(configDir)}-triage-`;
|
||||
const state = {
|
||||
keys: [...(sentinel ? [`msk-${tag(configDir)}-sentinel-${b64()}`] : []), "foreign-client-key", `mso-${tag(configDir)}-demo-key-${b64()}`],
|
||||
gets: 0, puts: [], ignorePut: false, dropSentinelOnPut: false, onGet: null,
|
||||
};
|
||||
const server = http.createServer(async (req, res) => {
|
||||
if (req.url !== KEY_PATH) {
|
||||
const key = req.headers.authorization?.slice(7);
|
||||
res.writeHead(state.keys.includes(key) ? 200 : 401); res.end("{}"); return;
|
||||
}
|
||||
if (req.headers.authorization !== `Bearer ${MANAGE}`) { res.writeHead(401); res.end("{}"); return; }
|
||||
if (req.method === "GET") { state.gets++; const body = JSON.stringify(state.keys); state.onGet?.(state.gets); res.end(body); return; }
|
||||
const chunks = []; for await (const c of req) chunks.push(c);
|
||||
const next = JSON.parse(Buffer.concat(chunks));
|
||||
for (const k of next) if (/^mst-.*-[A-Za-z0-9_-]{43}$/.test(k)) SEEN.add(k);
|
||||
state.puts.push(next);
|
||||
if (state.dropSentinelOnPut) state.keys = next.filter((k) => !k.startsWith("msk-") || !k.includes("-sentinel-"));
|
||||
else if (!state.ignorePut) state.keys = next;
|
||||
res.end('{"status":"ok"}');
|
||||
});
|
||||
await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve));
|
||||
const port = server.address().port;
|
||||
const closers = [];
|
||||
t.after(async () => {
|
||||
for (const close of closers) await close();
|
||||
server.closeAllConnections(); await new Promise((resolve) => server.close(resolve));
|
||||
fs.rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
const f = {
|
||||
root, configDir, prefix, state, server, port, closers, now: NOW,
|
||||
system: { dataRoot: root, proxy: { port, binary: "/fixture-unused" } },
|
||||
file: keyPaths(configDir).file, dir: keyPaths(configDir).dir,
|
||||
env: (extra = {}) => humanEnv({ MOSAIC_CONFIG: join(configDir, "config.json"), ...extra }),
|
||||
async cli(argv, { env, deps = {}, system = f.system } = {}) {
|
||||
const s = io(env ?? f.env());
|
||||
const code = await run(argv, s, { system, now: () => new Date(f.now), ...deps });
|
||||
const out = { code, stdout: s.out.text, stderr: s.err.text };
|
||||
OUTPUTS.push(out.stdout, out.stderr);
|
||||
return out;
|
||||
},
|
||||
// The same verb in a child process that SIGKILLs itself at `stage`.
|
||||
crash(argv, stage) {
|
||||
const script = `
|
||||
import { run } from ${JSON.stringify(new URL("../src/cli.mjs", import.meta.url).href)};
|
||||
import { io } from ${JSON.stringify(new URL("./helpers.mjs", import.meta.url).href)};
|
||||
const p = JSON.parse(process.env.TK_CRASH);
|
||||
const s = io(process.env);
|
||||
const code = await run(p.argv, s, { system: p.system, now: () => new Date(p.now),
|
||||
triageKeyHook: (stage) => { if (stage === p.stage) process.kill(process.pid, "SIGKILL"); } });
|
||||
process.stdout.write(s.out.text); process.stderr.write(s.err.text); process.exitCode = code;`;
|
||||
const env = f.env({ TK_CRASH: JSON.stringify({ argv, stage, system: f.system, now: f.now }) });
|
||||
return new Promise((resolve) => {
|
||||
const child = spawn(process.execPath, ["--input-type=module", "-e", script], { env, stdio: ["ignore", "pipe", "pipe"] });
|
||||
let stdout = "", stderr = "";
|
||||
child.stdout.on("data", (d) => { stdout += d; });
|
||||
child.stderr.on("data", (d) => { stderr += d; });
|
||||
child.on("exit", (code, signal) => { OUTPUTS.push(stdout, stderr); resolve({ code, signal, stdout, stderr }); });
|
||||
});
|
||||
},
|
||||
record: () => JSON.parse(fs.readFileSync(f.file, "utf8")),
|
||||
triageKeys: () => state.keys.filter((k) => k.startsWith(prefix)),
|
||||
receipts: () => {
|
||||
const file = join(root, "proxy", "mosaic-keys.receipts.jsonl");
|
||||
return fs.existsSync(file) ? fs.readFileSync(file, "utf8").split("\n").filter(Boolean).map((l) => JSON.parse(l)) : [];
|
||||
},
|
||||
fenced: () => fs.existsSync(join(root, "proxy", "mosaic-keys.uncertain")),
|
||||
// No file under the fixture but the key file itself holds a key; neither does any output.
|
||||
scan(alsoKeyFiles = []) {
|
||||
for (const k of state.keys) if (/^mst-.*-[A-Za-z0-9_-]{43}$/.test(k)) SEEN.add(k);
|
||||
for (const p of walk(root, [f.file, join(configDir, "secrets", "proxy-management.key"), ...alsoKeyFiles])) {
|
||||
assert.equal(leaks(fs.readFileSync(p, "latin1")), false, `a key is in ${p}`);
|
||||
}
|
||||
for (const text of OUTPUTS) assert.equal(leaks(text), false, "a key is in an output");
|
||||
},
|
||||
};
|
||||
return f;
|
||||
}
|
||||
|
||||
const ok = (r) => assert.equal(r.code, 0, r.stderr);
|
||||
const refusal = (r, code, words) => {
|
||||
assert.equal(r.code, 3, r.stderr);
|
||||
assert.equal(r.stdout, "");
|
||||
assert.match(r.stderr, new RegExp(`^mosaic: ${code}: ${words ?? ""}`));
|
||||
};
|
||||
|
||||
test("issue adds one key under the lock, writes the 0600 file, and records ids only", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const before = [...f.state.keys];
|
||||
const r = await f.cli(["triage", "key", "issue"]);
|
||||
ok(r);
|
||||
const rec = f.record();
|
||||
assert.deepEqual(Object.keys(rec), ["version", "key", "keyId", "issuedAt", "rotateBy", "retiring"]);
|
||||
assert.match(rec.key, new RegExp(`^${f.prefix}[A-Za-z0-9_-]{43}$`));
|
||||
assert.equal(rec.keyId, id(rec.key));
|
||||
assert.equal(rec.issuedAt, iso(NOW));
|
||||
assert.equal(rec.rotateBy, "2026-12-18");
|
||||
assert.deepEqual(rec.retiring, []);
|
||||
assert.equal(r.stdout, `triage key issued: ${rec.keyId}, rotate by 2026-12-18\nfile: ${f.file}\n`);
|
||||
assert.deepEqual(f.state.keys, [...before, rec.key]);
|
||||
assert.equal(f.state.puts.length, 1);
|
||||
const file = fs.lstatSync(f.file), dir = fs.lstatSync(f.dir);
|
||||
assert.equal(file.mode & 0o777, 0o600); assert.equal(dir.mode & 0o777, 0o700);
|
||||
assert.equal(file.uid, process.getuid()); assert.equal(dir.uid, process.getuid());
|
||||
assert.deepEqual(f.receipts(), [{ op: "triage.key.issue", time: iso(NOW), addedKeyIds: [rec.keyId], removedKeyIds: [] }]);
|
||||
assert.equal(f.fenced(), false);
|
||||
assert.equal(fs.readFileSync(join(f.root, "proxy", "keys.generation"), "utf8"), "1\n");
|
||||
// A second issue refuses and changes nothing; rotate is the verb for that.
|
||||
const again = await f.cli(["triage", "key", "issue"]);
|
||||
refusal(again, "triage-key-issued", "a Triage key is already issued; use mosaic triage key rotate\n$");
|
||||
assert.equal(f.state.puts.length, 1);
|
||||
assert.deepEqual(f.record(), rec);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("status reads the file and the list and writes nothing", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const none = await f.cli(["triage", "key", "status"]);
|
||||
ok(none);
|
||||
assert.equal(none.stdout, `triage key: not issued\nfile: ${f.file}\nproxy: read\nnext: mosaic triage key issue\n`);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const rec = f.record(), puts = f.state.puts.length, receipts = f.receipts().length;
|
||||
f.now = NOW + 3 * DAY + 4 * HOUR + 5 * MIN;
|
||||
const shown = await f.cli(["triage", "key", "status"]);
|
||||
ok(shown);
|
||||
assert.equal(shown.stdout, [
|
||||
"triage key: issued", `file: ${f.file}`, `key id: ${rec.keyId}`, `issued: ${iso(NOW)} (age 3d 4h 5m)`,
|
||||
"rotate by: 2026-12-18", "proxy: holds the key", "next: nothing", "",
|
||||
].join("\n"));
|
||||
f.now = Date.parse("2026-12-18T00:00:00.000Z");
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /rotate by: 2026-12-18 \(due\)\n[^]*next: mosaic triage key rotate\n$/);
|
||||
f.now = NOW + HOUR;
|
||||
f.state.keys = f.state.keys.filter((k) => k !== rec.key);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /proxy: does not hold the key\nnext: mosaic triage key rotate\n$/);
|
||||
assert.equal(f.state.puts.length, puts);
|
||||
assert.equal(f.receipts().length, receipts);
|
||||
assert.equal(fs.readdirSync(join(f.root, "proxy")).includes("mosaic-keys.lock"), false);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("rotate adds a new key and keeps the old one for the bus grace; retire removes it once due", async (t) => {
|
||||
const f = await fixture(t);
|
||||
refusal(await f.cli(["triage", "key", "rotate"]), "triage-key-not-issued", "no Triage key is issued; use mosaic triage key issue\n$");
|
||||
assert.equal(f.state.puts.length, 0);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const first = f.record();
|
||||
f.now = NOW + HOUR;
|
||||
const r = await f.cli(["triage", "key", "rotate"]);
|
||||
ok(r);
|
||||
const second = f.record();
|
||||
const retireAt = iso(NOW + HOUR + 120 * MIN);
|
||||
assert.notEqual(second.key, first.key);
|
||||
assert.equal(second.issuedAt, iso(NOW + HOUR));
|
||||
assert.deepEqual(second.retiring, [{ key: first.key, keyId: first.keyId, issuedAt: first.issuedAt, retireAt }]);
|
||||
assert.equal(r.stdout, `triage key rotated: ${second.keyId}, rotate by 2026-12-18\nold key ${first.keyId} stays live until mosaic triage key retire runs at or after ${retireAt}\n`);
|
||||
assert.deepEqual(f.triageKeys(), [first.key, second.key]);
|
||||
f.now = NOW + HOUR + 119 * MIN;
|
||||
const early = await f.cli(["triage", "key", "retire"]);
|
||||
ok(early);
|
||||
assert.equal(early.stdout, "triage key retire: nothing to remove\n");
|
||||
assert.equal(f.state.puts.length, 2);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, new RegExp(`retiring: ${first.keyId}, live until a retire after ${retireAt}\\n[^]*next: nothing\\n$`));
|
||||
f.now = NOW + HOUR + 120 * MIN;
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, new RegExp(`retiring: ${first.keyId}, live until a retire after ${retireAt} \\(due\\)\\n[^]*next: mosaic triage key retire\\n$`));
|
||||
const done = await f.cli(["triage", "key", "retire"]);
|
||||
ok(done);
|
||||
assert.equal(done.stdout, `triage key retire: removed ${first.keyId}\n`);
|
||||
assert.deepEqual(f.triageKeys(), [second.key]);
|
||||
assert.deepEqual(f.record(), { ...second, retiring: [] });
|
||||
assert.deepEqual(f.receipts().map((e) => e.op), ["triage.key.issue", "triage.key.rotate", "triage.key.retire"]);
|
||||
assert.deepEqual(f.receipts()[2].removedKeyIds, [first.keyId]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("rotate takes the grace from proxy.json, where the bus reads it, and drops an old key the proxy lost", async (t) => {
|
||||
const f = await fixture(t);
|
||||
f.system = { ...f.system, proxy: { ...f.system.proxy, rotateGraceMinutes: 30 } };
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const first = f.record();
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
const second = f.record();
|
||||
assert.equal(second.retiring[0].retireAt, iso(NOW + 30 * MIN));
|
||||
// The proxy lost the current key: rotate drops it at once, as the bus does for an agent key.
|
||||
f.state.keys = f.state.keys.filter((k) => k !== second.key);
|
||||
const r = await f.cli(["triage", "key", "rotate"]);
|
||||
ok(r);
|
||||
assert.match(r.stdout, new RegExp(`old key ${second.keyId} was not in the proxy list; nothing to retire\\n$`));
|
||||
assert.deepEqual(f.record().retiring.map((e) => e.keyId), [first.keyId]);
|
||||
// A retiring key past its time, or no longer in the list, leaves on the next rotate.
|
||||
f.now = NOW + 31 * MIN;
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
const last = f.record();
|
||||
assert.deepEqual(last.retiring.map((e) => e.keyId), [id(f.triageKeys()[0])]);
|
||||
assert.equal(f.triageKeys().includes(first.key), false);
|
||||
assert.equal(f.triageKeys().length, 2);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("lifetime and grace match the bus's own ModelKeys for the same clock and proxy settings", async (t) => {
|
||||
const runtime = fs.readFileSync(join(REPO, "packages/bus/src/runtime.mjs"), "utf8");
|
||||
const literal = runtime.match(/rotateGraceMinutes: proxy\.rotateGraceMinutes \?\? (\d+)/);
|
||||
assert.ok(literal, "the bus no longer reads rotateGraceMinutes as this test expects");
|
||||
assert.equal(Number(literal[1]), DEFAULT_ROTATE_GRACE_MINUTES);
|
||||
for (const grace of [undefined, 45]) {
|
||||
const f = await fixture(t);
|
||||
if (grace !== undefined) f.system = { ...f.system, proxy: { ...f.system.proxy, rotateGraceMinutes: grace } };
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
f.now = NOW + 3 * HOUR;
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
const ours = f.record();
|
||||
// The bus, driven the way runtime.mjs drives it, on its own in-memory list.
|
||||
let keys = [`msk-${tag(f.configDir)}-sentinel-${b64()}`], record = null, now = NOW;
|
||||
const credentials = { registerModel() {}, modelRecord: () => record, writeModel: (_, r) => { record = r; }, status: () => [] };
|
||||
const client = { getKeys: async () => [...keys], putKeys: async (k) => { keys = [...k]; return { ok: true }; } };
|
||||
const bus = new ModelKeys({ credentials, client, configDir: f.configDir, baseUrl: "http://127.0.0.1:1", now: () => now,
|
||||
...(grace === undefined ? {} : { rotateGraceMinutes: f.system.proxy.rotateGraceMinutes ?? Number(literal[1]) }) });
|
||||
await bus.reconcile();
|
||||
await bus.ensure("demo/x");
|
||||
assert.equal(record.rotateBy, rotateByFor(NOW));
|
||||
now = NOW + 3 * HOUR;
|
||||
await bus.rotate("demo/x");
|
||||
assert.equal(ours.rotateBy, record.rotateBy);
|
||||
assert.equal(ours.retiring[0].retireAt, record.retiring[0].retireAt);
|
||||
assert.equal(ours.retiring[0].retireAt, retireAtFor(now, f.system.proxy));
|
||||
f.scan();
|
||||
}
|
||||
});
|
||||
|
||||
test("an agent caller is refused before anything is read, for every key verb", async (t) => {
|
||||
const f = await fixture(t);
|
||||
for (const marker of AGENT_MARKERS) {
|
||||
for (const verb of VERBS) {
|
||||
const r = await f.cli(["triage", "key", verb], { env: { ...f.env(), [marker]: "1" } });
|
||||
assert.equal(r.code, 3, marker);
|
||||
assert.equal(r.stderr, `mosaic: refused: this runs only from a human shell, outside any agent run (found ${marker} in the environment)\n`);
|
||||
}
|
||||
const saved = process.env[marker];
|
||||
process.env[marker] = "1";
|
||||
try {
|
||||
for (const verb of VERBS) {
|
||||
const r = await f.cli(["triage", "key", verb]);
|
||||
assert.equal(r.code, 3, marker);
|
||||
assert.match(r.stderr, new RegExp(`found ${marker} in the environment`));
|
||||
}
|
||||
} finally { if (saved === undefined) delete process.env[marker]; else process.env[marker] = saved; }
|
||||
}
|
||||
assert.equal(f.state.gets, 0);
|
||||
assert.equal(f.state.puts.length, 0);
|
||||
assert.equal(fs.existsSync(f.dir), false);
|
||||
assert.deepEqual(fs.readdirSync(join(f.root, "proxy")), []);
|
||||
});
|
||||
|
||||
test("usage: unknown key verbs, options and arguments exit 4 and echo no value", async (t) => {
|
||||
const f = await fixture(t);
|
||||
for (const argv of [["triage", "key"], ["triage", "key", "nope"], ["triage", "key", "issue", "extra"], ["triage", "key", "status", "--json"]]) {
|
||||
const r = await f.cli(argv);
|
||||
assert.equal(r.code, 4, argv.join(" "));
|
||||
assert.equal(r.stdout, "");
|
||||
assert.match(r.stderr, /\nusage: mosaic triage route show [^\n]*\| key issue \| rotate \| retire \| status\n$/);
|
||||
}
|
||||
const r = await f.cli(["triage", "key", "rotate", "--key", "secret-value"]);
|
||||
assert.equal(r.code, 4);
|
||||
assert.match(r.stderr, /^mosaic: unknown option\n/);
|
||||
assert.doesNotMatch(r.stderr, /secret-value/);
|
||||
assert.equal(f.state.gets, 0);
|
||||
});
|
||||
|
||||
test("an unsafe key store refuses every verb before any proxy write", async (t) => {
|
||||
const uid = process.getuid();
|
||||
const cases = [
|
||||
["file mode", (f) => fs.chmodSync(f.file, 0o644), {}, "the key file mode is not 0600"],
|
||||
["file owner", () => {}, { triageKeyFileUid: uid + 1 }, "the key file is not owned by this user"],
|
||||
["file link", (f) => { fs.renameSync(f.file, join(f.dir, "real")); fs.symlinkSync(join(f.dir, "real"), f.file); }, {}, "the key file is a link"],
|
||||
["file is a directory", (f) => { fs.rmSync(f.file); fs.mkdirSync(f.file, { mode: 0o700 }); }, {}, "the key file is not a regular file"],
|
||||
["file is a FIFO", (f) => { fs.rmSync(f.file); assert.equal(spawnSync("mkfifo", ["-m", "600", f.file]).status, 0); }, {}, "the key file is not a regular file"],
|
||||
["hard link", (f) => fs.linkSync(f.file, join(f.dir, "second")), {}, "the key file has more than one link"],
|
||||
["not JSON", (f) => fs.writeFileSync(f.file, "{"), {}, "the key file is not JSON"],
|
||||
["extra field", (f) => fs.writeFileSync(f.file, JSON.stringify({ ...f.record(), extra: 1 })), {}, "the key file does not match the key shape"],
|
||||
["another instance's key", (f) => { const r = f.record(); const key = `mst-${tag("/elsewhere")}-triage-${b64()}`; fs.writeFileSync(f.file, JSON.stringify({ ...r, key, keyId: id(key) })); }, {}, "the key file does not match the key shape"],
|
||||
["wrong key id", (f) => fs.writeFileSync(f.file, JSON.stringify({ ...f.record(), keyId: "000000000000" })), {}, "the key file does not match the key shape"],
|
||||
["too large", (f) => fs.writeFileSync(f.file, " ".repeat(64 * 1024 + 1)), {}, "the key file is too large"],
|
||||
["triage directory mode", (f) => fs.chmodSync(f.dir, 0o755), {}, "the triage secrets directory mode is not 0700"],
|
||||
["triage directory link", (f) => { fs.renameSync(f.dir, join(f.configDir, "secrets", "moved")); fs.symlinkSync(join(f.configDir, "secrets", "moved"), f.dir); }, {}, "the triage secrets directory is a link"],
|
||||
["triage directory is a file", (f) => { fs.rmSync(f.dir, { recursive: true }); fs.writeFileSync(f.dir, "", { mode: 0o600 }); }, {}, "the triage secrets directory is not a directory"],
|
||||
["secrets mode", (f) => fs.chmodSync(join(f.configDir, "secrets"), 0o750), {}, "the secrets directory mode is not 0700"],
|
||||
["owner", () => {}, { triageKeyUid: uid + 1 }, "the secrets directory is not owned by this user"],
|
||||
];
|
||||
for (const [name, spoil, deps, words] of cases) {
|
||||
const f = await fixture(t);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
spoil(f);
|
||||
const keys = [...f.state.keys], puts = f.state.puts.length;
|
||||
for (const verb of VERBS) {
|
||||
const r = await f.cli(["triage", "key", verb], { deps });
|
||||
refusal(r, "triage-key-unreadable", `the Triage key store is unreadable: ${words.replace(/[.()]/g, "\\$&")}\n$`);
|
||||
assert.equal(f.state.puts.length, puts, `${name}: ${verb}`);
|
||||
}
|
||||
assert.deepEqual(f.state.keys, keys);
|
||||
// The spoils above move or link the key file itself; those copies are the key file.
|
||||
f.scan([join(f.dir, "real"), join(f.dir, "second"), join(f.configDir, "secrets", "moved", "model")]);
|
||||
}
|
||||
// No secrets directory at all: status says not issued; issue refuses before the proxy changes.
|
||||
const f = await fixture(t);
|
||||
fs.rmSync(join(f.configDir, "secrets"), { recursive: true });
|
||||
fs.mkdirSync(join(f.configDir, "secrets"), { mode: 0o700 });
|
||||
fs.writeFileSync(join(f.configDir, "secrets", "proxy-management.key"), MANAGE, { mode: 0o600 });
|
||||
fs.chmodSync(join(f.configDir, "secrets"), 0o755);
|
||||
refusal(await f.cli(["triage", "key", "issue"]), "triage-key-unreadable", "the Triage key store is unreadable: the secrets directory mode is not 0700\n$");
|
||||
assert.equal(f.state.puts.length, 0);
|
||||
});
|
||||
|
||||
test("the instance proxy lock, the fence, the sentinel and the proxy itself each refuse plainly", async (t) => {
|
||||
// Lock held by a live process: busy after the lock's wait, and nothing changes.
|
||||
const f = await fixture(t);
|
||||
const lockFile = join(f.root, "proxy", "mosaic-keys.lock");
|
||||
ok(await f.cli(["triage", "key", "status"]));
|
||||
fs.writeFileSync(lockFile, `${JSON.stringify({ pid: process.pid, token: "a".repeat(32) })}\n`, { mode: 0o600 });
|
||||
const busy = await f.cli(["triage", "key", "issue"]);
|
||||
refusal(busy, "proxy-lock-busy", "another proxy writer holds the instance proxy lock; try again when it ends, and if no writer is running, see mosaic proxy fence status\n$");
|
||||
assert.equal(f.state.gets, 1);
|
||||
// A lock left by a process that is gone is reaped, and the verb goes ahead.
|
||||
const dead = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||
fs.writeFileSync(lockFile, `${JSON.stringify({ pid: dead, token: "b".repeat(32) })}\n`, { mode: 0o600 });
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
assert.equal(fs.existsSync(lockFile), false);
|
||||
assert.equal(f.triageKeys().length, 1);
|
||||
fs.rmSync(f.file);
|
||||
f.state.keys = f.state.keys.filter((k) => !k.startsWith(f.prefix));
|
||||
// A set fence: every write refuses, status names it and the way out.
|
||||
const putsBefore = f.state.puts.length;
|
||||
fs.writeFileSync(join(f.root, "proxy", "mosaic-keys.uncertain"), "{}\n", { mode: 0o600 });
|
||||
for (const verb of ["issue", "rotate", "retire"]) refusal(await f.cli(["triage", "key", verb]), "proxy-uncertain");
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /proxy fence: set, an earlier proxy write has an uncertain outcome\nnext: restart the proxy, then mosaic proxy fence clear --proxy-restarted\n$/);
|
||||
assert.equal(f.state.puts.length, putsBefore);
|
||||
// No sentinel for this instance (another instance's does not count).
|
||||
const g = await fixture(t, { sentinel: false });
|
||||
g.state.keys.push(`msk-${tag("/elsewhere")}-sentinel-${b64()}`);
|
||||
refusal(await g.cli(["triage", "key", "issue"]), "triage-key-proxy-open");
|
||||
assert.match((await g.cli(["triage", "key", "status"])).stdout, /proxy: its key list has no sentinel for this instance\nnext: check the proxy with mosaic proxy status\n$/);
|
||||
assert.equal(g.state.puts.length, 0);
|
||||
// No proxy set up, no management key, proxy down.
|
||||
const h = await fixture(t);
|
||||
const bare = { dataRoot: h.root };
|
||||
refusal(await h.cli(["triage", "key", "issue"], { system: bare }), "triage-key-no-proxy");
|
||||
assert.equal((await h.cli(["triage", "key", "status"], { system: bare })).stdout,
|
||||
`triage key: not issued\nfile: ${h.file}\nproxy: not set up for this instance (no proxy.json)\nnext: set up the model proxy first (mosaic proxy init)\n`);
|
||||
fs.chmodSync(join(h.configDir, "secrets", "proxy-management.key"), 0o644);
|
||||
refusal(await h.cli(["triage", "key", "issue"]), "proxy-management-key-invalid");
|
||||
fs.chmodSync(join(h.configDir, "secrets", "proxy-management.key"), 0o600);
|
||||
h.server.closeAllConnections(); await new Promise((resolve) => h.server.close(resolve));
|
||||
h.server.close = (cb) => cb();
|
||||
refusal(await h.cli(["triage", "key", "issue"]), "proxy-client-down");
|
||||
assert.match((await h.cli(["triage", "key", "status"])).stdout, /proxy: not read \(proxy-client-down\)\n/);
|
||||
assert.equal(fs.existsSync(h.file), false);
|
||||
f.scan(); g.scan(); h.scan();
|
||||
});
|
||||
|
||||
test("a write the proxy does not confirm keeps the fence; a list that moves mid-verb is not overwritten", async (t) => {
|
||||
const f = await fixture(t);
|
||||
f.state.ignorePut = true;
|
||||
refusal(await f.cli(["triage", "key", "issue"]), "triage-key-unconfirmed");
|
||||
assert.equal(f.fenced(), true);
|
||||
assert.equal(fs.existsSync(f.file), false);
|
||||
// The store's directory was made and checked before the proxy changed, not after.
|
||||
assert.equal(fs.lstatSync(f.dir).mode & 0o777, 0o700);
|
||||
assert.deepEqual(f.receipts(), []);
|
||||
// A PUT the proxy takes but whose read-back has lost this instance's sentinel is not confirmed.
|
||||
const s = await fixture(t);
|
||||
s.state.dropSentinelOnPut = true;
|
||||
refusal(await s.cli(["triage", "key", "issue"]), "triage-key-unconfirmed");
|
||||
assert.equal(s.fenced(), true);
|
||||
assert.equal(fs.existsSync(s.file), false);
|
||||
s.scan();
|
||||
const g = await fixture(t);
|
||||
g.state.onGet = (n) => { if (n === 1) g.state.keys = [...g.state.keys, "late-foreign-key"]; };
|
||||
refusal(await g.cli(["triage", "key", "issue"]), "triage-key-raced");
|
||||
assert.equal(g.state.puts.length, 0);
|
||||
assert.equal(g.fenced(), false);
|
||||
f.scan(); g.scan();
|
||||
});
|
||||
|
||||
test("keys outside the Triage namespace are never touched; Triage keys the file does not hold are removed", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const agent = `msk-${tag(f.configDir)}-${b64()}`, other = `mst-${tag("/elsewhere")}-triage-${b64()}`;
|
||||
const orphan = `${f.prefix}${b64()}`, odd = `${f.prefix}short`;
|
||||
f.state.keys.push(agent, other, orphan, odd);
|
||||
const foreign = f.state.keys.filter((k) => k !== orphan && k !== odd);
|
||||
const r = await f.cli(["triage", "key", "issue"]);
|
||||
ok(r);
|
||||
// A malformed name under the Triage prefix is still Triage's, as the bus treats its own prefix.
|
||||
assert.match(r.stdout, new RegExp(`removed keys this file did not hold: ${id(orphan)}, ${id(odd)}\\n$`));
|
||||
for (const verb of ["rotate", "retire"]) { f.now += 3 * HOUR; ok(await f.cli(["triage", "key", verb])); }
|
||||
for (const k of foreign) assert.ok(f.state.keys.includes(k), "a key outside the namespace left the list");
|
||||
assert.equal(f.state.keys.includes(orphan), false);
|
||||
assert.equal(f.state.keys.includes(odd), false);
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("crash after the PUT, before the read-back: the fence holds; status says so; clear, then issue recovers", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const crashed = await f.crash(["triage", "key", "issue"], "put");
|
||||
assert.equal(crashed.signal, "SIGKILL");
|
||||
assert.equal(f.fenced(), true);
|
||||
assert.equal(f.triageKeys().length, 1);
|
||||
assert.equal(fs.existsSync(f.file), false);
|
||||
const status = await f.cli(["triage", "key", "status"]);
|
||||
ok(status);
|
||||
assert.match(status.stdout, new RegExp(`proxy fence: set[^\\n]*\\nkeys in the Triage namespace the file does not hold: ${id(f.triageKeys()[0])}\\nnext: restart the proxy, then mosaic proxy fence clear --proxy-restarted\\n$`));
|
||||
refusal(await f.cli(["triage", "key", "issue"]), "proxy-uncertain");
|
||||
ok(await f.cli(["proxy", "fence", "clear", "--proxy-restarted"]));
|
||||
assert.equal(f.fenced(), false);
|
||||
const stray = f.triageKeys()[0];
|
||||
const r = await f.cli(["triage", "key", "issue"]);
|
||||
ok(r);
|
||||
assert.match(r.stdout, new RegExp(`removed keys this file did not hold: ${id(stray)}\\n$`));
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("crash after the read-back, before the file write: no fence; status names the stray key; issue removes it", async (t) => {
|
||||
const f = await fixture(t);
|
||||
assert.equal((await f.crash(["triage", "key", "issue"], "write")).signal, "SIGKILL");
|
||||
assert.equal(f.fenced(), false);
|
||||
assert.equal(fs.existsSync(f.file), false);
|
||||
const [stray] = f.triageKeys();
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout,
|
||||
new RegExp(`^triage key: not issued\\n[^]*keys in the Triage namespace the file does not hold: ${id(stray)}\\nnext: mosaic triage key issue\\n$`));
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
// The same crash in rotate: the file keeps the old key, the stray new one is named, retire removes it.
|
||||
const held = f.record();
|
||||
assert.equal((await f.crash(["triage", "key", "rotate"], "write")).signal, "SIGKILL");
|
||||
assert.deepEqual(f.record(), held);
|
||||
const strayNew = f.triageKeys().find((k) => k !== held.key);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout,
|
||||
new RegExp(`proxy: holds the key\\nkeys in the Triage namespace the file does not hold: ${id(strayNew)}\\nnext: mosaic triage key retire\\n$`));
|
||||
const retired = await f.cli(["triage", "key", "retire"]);
|
||||
ok(retired);
|
||||
assert.equal(retired.stdout, `triage key retire: removed ${id(strayNew)}\n`);
|
||||
assert.deepEqual(f.triageKeys(), [held.key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("file written but the old key not yet retired: status shows it due; a crash inside retire recovers on the next retire", async (t) => {
|
||||
const f = await fixture(t);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const first = f.record();
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
f.now = NOW + 121 * MIN;
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, new RegExp(`retiring: ${first.keyId}, live until a retire after [^\\n]* \\(due\\)\\n[^]*next: mosaic triage key retire\\n$`));
|
||||
// retire dies after the proxy dropped the key but before the file was rewritten.
|
||||
assert.equal((await f.crash(["triage", "key", "retire"], "write")).signal, "SIGKILL");
|
||||
assert.equal(f.state.keys.includes(first.key), false);
|
||||
assert.equal(f.record().retiring.length, 1);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /\(due\) \(not in the proxy list\)\n[^]*next: mosaic triage key retire\n$/);
|
||||
const puts = f.state.puts.length;
|
||||
const r = await f.cli(["triage", "key", "retire"]);
|
||||
ok(r);
|
||||
assert.equal(f.state.puts.length, puts);
|
||||
assert.deepEqual(f.record().retiring, []);
|
||||
// And a retire killed after its PUT leaves the fence, like any other write.
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
f.now += 121 * MIN;
|
||||
assert.equal((await f.crash(["triage", "key", "retire"], "put")).signal, "SIGKILL");
|
||||
assert.equal(f.fenced(), true);
|
||||
refusal(await f.cli(["triage", "key", "retire"]), "proxy-uncertain");
|
||||
ok(await f.cli(["proxy", "fence", "clear", "--proxy-restarted"]));
|
||||
ok(await f.cli(["triage", "key", "retire"]));
|
||||
assert.deepEqual(f.record().retiring, []);
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("a file that cannot be written after the proxy changed is reported with its recovery", async (t) => {
|
||||
const f = await fixture(t);
|
||||
ok(await f.cli(["triage", "key", "status"]));
|
||||
fs.mkdirSync(f.dir, { mode: 0o700 });
|
||||
const r = await f.cli(["triage", "key", "issue"], { deps: { triageKeyHook: (stage) => { if (stage === "write") fs.chmodSync(f.dir, 0o755); } } });
|
||||
refusal(r, "triage-key-unwritten", "the proxy list changed but ");
|
||||
assert.match(r.stderr, /run mosaic triage key issue again, which clears what this run left\n$/);
|
||||
assert.equal(f.triageKeys().length, 1);
|
||||
fs.chmodSync(f.dir, 0o700);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("crash after the temp file, before its rename: status counts it; the next verb removes it", async (t) => {
|
||||
const f = await fixture(t);
|
||||
assert.equal((await f.crash(["triage", "key", "issue"], "rename")).signal, "SIGKILL");
|
||||
assert.equal(fs.existsSync(f.file), false);
|
||||
const temps = fs.readdirSync(f.dir).filter((n) => n.endsWith(".tmp"));
|
||||
assert.equal(temps.length, 1);
|
||||
assert.match(temps[0], /^\.model\.\d+\.[0-9a-f]{12}\.tmp$/);
|
||||
// The temp file holds the key the proxy now has; that is why it must go.
|
||||
const [stray] = f.triageKeys();
|
||||
assert.ok(fs.readFileSync(join(f.dir, temps[0]), "utf8").includes(stray));
|
||||
const status = await f.cli(["triage", "key", "status"]);
|
||||
ok(status);
|
||||
assert.match(status.stdout, new RegExp(`leftover temp files in the triage secrets directory: 1 \\(the next issue, rotate or retire removes them\\)\\nkeys in the Triage namespace the file does not hold: ${id(stray)}\\nnext: mosaic triage key issue\\n$`));
|
||||
// Something else in the directory is not a leftover and stays.
|
||||
fs.writeFileSync(join(f.dir, "notes.tmp"), "", { mode: 0o600 });
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
assert.deepEqual(fs.readdirSync(f.dir).sort(), ["model", "notes.tmp"]);
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
// Retire, with nothing else to do, still removes a leftover.
|
||||
assert.equal((await f.crash(["triage", "key", "rotate"], "rename")).signal, "SIGKILL");
|
||||
ok(await f.cli(["triage", "key", "retire"]));
|
||||
assert.deepEqual(fs.readdirSync(f.dir).sort(), ["model", "notes.tmp"]);
|
||||
assert.doesNotMatch((await f.cli(["triage", "key", "status"])).stdout, /leftover/);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("the receipts file is checked before the proxy changes; a receipt that fails after is named", async (t) => {
|
||||
const receipts = (f) => join(f.root, "proxy", "mosaic-keys.receipts.jsonl");
|
||||
const spoils = [
|
||||
["mode", (f) => { fs.writeFileSync(receipts(f), "", { mode: 0o644 }); fs.chmodSync(receipts(f), 0o644); }, "the proxy receipts file mode is not 0600"],
|
||||
["link", (f) => { fs.writeFileSync(join(f.root, "elsewhere"), "", { mode: 0o600 }); fs.symlinkSync(join(f.root, "elsewhere"), receipts(f)); }, "the proxy receipts file cannot be opened"],
|
||||
["FIFO", (f) => assert.equal(spawnSync("mkfifo", ["-m", "600", receipts(f)]).status, 0), "the proxy receipts file cannot be opened"],
|
||||
];
|
||||
for (const [name, spoil, words] of spoils) {
|
||||
const f = await fixture(t);
|
||||
spoil(f);
|
||||
refusal(await f.cli(["triage", "key", "issue"]), "triage-key-receipts-unsafe", `${words}; nothing was changed\n$`);
|
||||
assert.equal(f.state.puts.length, 0, name);
|
||||
assert.equal(f.fenced(), false, name);
|
||||
assert.equal(fs.existsSync(f.file), false, name);
|
||||
f.scan();
|
||||
}
|
||||
// Safe before the change, unsafe at the receipt: the change stands and the refusal says so.
|
||||
const f = await fixture(t);
|
||||
const r = await f.cli(["triage", "key", "issue"], { deps: { triageKeyHook: (stage) => { if (stage === "receipt") fs.chmodSync(receipts(f), 0o644); } } });
|
||||
refusal(r, "triage-key-unrecorded", "the change was made but its receipt line was not written; run mosaic triage key status\n$");
|
||||
assert.deepEqual(f.triageKeys(), [f.record().key]);
|
||||
assert.equal(f.fenced(), false);
|
||||
assert.equal(fs.readFileSync(receipts(f), "utf8"), "");
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("a directory sync that fails after the rename is named, and the receipt is still written", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const r = await f.cli(["triage", "key", "issue"], { deps: { triageKeyHook: (stage) => { if (stage === "sync") throw new Error("EIO"); } } });
|
||||
refusal(r, "triage-key-unsynced", `${f.file.replace(/[.]/g, "\\.")} was written but its directory could not be synced, so it may not survive a power loss; run mosaic triage key status\n$`);
|
||||
const rec = f.record();
|
||||
assert.deepEqual(f.triageKeys(), [rec.key]);
|
||||
assert.deepEqual(f.receipts().map((e) => e.addedKeyIds), [[rec.keyId]]);
|
||||
assert.equal(f.fenced(), false);
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /proxy: holds the key\nnext: nothing\n$/);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("an unexpected error prints fixed words, never the error's own text", async (t) => {
|
||||
const f = await fixture(t);
|
||||
const secret = `mst-${tag(f.configDir)}-triage-${b64()}`;
|
||||
SEEN.add(secret);
|
||||
// Outside the proxy write: the verb's own fallback line.
|
||||
const broken = { dataRoot: f.root, get proxy() { throw new Error(`boom ${secret}`); } };
|
||||
for (const verb of VERBS) {
|
||||
const r = await f.cli(["triage", "key", verb], { system: broken });
|
||||
assert.equal(r.code, 3, verb);
|
||||
assert.equal(r.stderr, "mosaic: triage-key-failed: the key verb stopped on an unexpected error; run mosaic triage key status\n");
|
||||
}
|
||||
// Inside the proxy write: the write's own fixed line, and the fence stays for a human.
|
||||
const r = await f.cli(["triage", "key", "issue"], { deps: { triageKeyHook: (stage) => { if (stage === "put") throw new Error(`boom ${secret}`); } } });
|
||||
refusal(r, "triage-key-failed", "the proxy write failed\n$");
|
||||
assert.equal(f.fenced(), true);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
// The org bus reconciles its own `mso-` namespace at start and a no-org bus its `msk-` one; a
|
||||
// Triage key sits in neither, so both leave it. Stray keys in each namespace show the reconcile ran.
|
||||
test("an org bus that starts and reconciles, and a no-org reconcile, both leave the Triage key alone", async (t) => {
|
||||
const f = await fixture(t);
|
||||
ok(await f.cli(["triage", "key", "issue"]));
|
||||
const triage = f.record().key;
|
||||
const strayOrg = `mso-${tag(f.configDir)}-demo-key-${b64()}`;
|
||||
f.state.keys.push(strayOrg);
|
||||
const rolesDir = rolesCopy(f.root), system = systemFor(f.root), doc = businessDoc(f.root, "demo");
|
||||
doc.plugins = { ...doc.plugins, harness: ["pi"] };
|
||||
const file = writeJson(join(f.configDir, "businesses", "demo.json"), doc);
|
||||
assert.ok(file);
|
||||
const business = loadBusiness("demo", { dir: f.configDir, rolesDir }), resolved = {};
|
||||
for (const instance of Object.keys(business.roles)) resolved[instance] = resolveInstance({ system, business, instance });
|
||||
fs.mkdirSync(join(f.root, "data"), { mode: 0o700 });
|
||||
new Registry(join(f.root, "data"));
|
||||
const bus = await startBroker({ dataRoot: join(f.root, "data"), businesses: { demo: busOrganization(business, resolved) },
|
||||
organization: { dir: f.configDir, rolesDir, system }, proxy: { port: f.port, binary: "/fixture-unused" },
|
||||
org: { id: "demo" }, instanceDataRoot: f.root, instanceConfigDir: f.configDir });
|
||||
f.closers.push(() => bus.close());
|
||||
const orgKey = (await bus.modelPreflight({ business: "demo", role: "chief" })).key;
|
||||
assert.ok(orgKey.startsWith(`mso-${tag(f.configDir)}-demo-key-`));
|
||||
assert.equal(f.state.keys.includes(strayOrg), false, "the org reconcile did not run");
|
||||
assert.ok(f.state.keys.includes(triage), "the org bus removed the Triage key");
|
||||
// A no-org reconcile on the same instance.
|
||||
const strayAgent = `msk-${tag(f.configDir)}-${b64()}`;
|
||||
f.state.keys.push(strayAgent);
|
||||
const client = new ManagementClient({ baseUrl: `http://127.0.0.1:${f.port}`, key: MANAGE });
|
||||
const keys = new ModelKeys({ credentials: { status: () => [], modelRecord: () => null }, client, configDir: f.configDir, baseUrl: `http://127.0.0.1:${f.port}` });
|
||||
await keys.reconcile();
|
||||
assert.equal(f.state.keys.includes(strayAgent), false, "the no-org reconcile did not run");
|
||||
assert.ok(f.state.keys.includes(triage), "the no-org reconcile removed the Triage key");
|
||||
// Triage still works under the shared lock while the org bus runs, and leaves the org key.
|
||||
f.now += HOUR;
|
||||
ok(await f.cli(["triage", "key", "rotate"]));
|
||||
assert.ok(f.state.keys.includes(orgKey));
|
||||
assert.match((await f.cli(["triage", "key", "status"])).stdout, /proxy: holds the key\n/);
|
||||
f.scan();
|
||||
});
|
||||
|
||||
test("scripts/mosaic routes triage key status with the instance's own config directory", (t) => {
|
||||
const home = fs.mkdtempSync(join(tmpdir(), "tkm-"));
|
||||
t.after(() => fs.rmSync(home, { recursive: true, force: true }));
|
||||
const env = humanEnv({ HOME: home, MOSAIC_CONFIG: join(home, ".config/mosaic-tktest/config.json") });
|
||||
const setup = spawnSync(join(REPO, "scripts/bootstrap.sh"), ["--instance", "tktest"], { env, encoding: "utf8" });
|
||||
assert.equal(setup.status, 0, setup.stderr);
|
||||
const r = spawnSync(join(REPO, "scripts/mosaic"), ["triage", "key", "status"], { env, encoding: "utf8", timeout: 60000 });
|
||||
OUTPUTS.push(r.stdout, r.stderr);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.equal(r.stdout, `triage key: not issued\nfile: ${join(home, ".config/mosaic-tktest/secrets/triage/model")}\nproxy: not set up for this instance (no proxy.json)\nnext: set up the model proxy first (mosaic proxy init)\n`);
|
||||
const refused = spawnSync(join(REPO, "scripts/mosaic"), ["triage", "key", "issue"], { env: { ...env, CLAUDECODE: "1" }, encoding: "utf8", timeout: 60000 });
|
||||
assert.equal(refused.status, 3);
|
||||
assert.match(refused.stderr, /found CLAUDECODE in the environment/);
|
||||
});
|
||||
|
||||
test("no output of any verb in this file holds any key the stand-in saw", () => {
|
||||
assert.ok(SEEN.size >= 20, `only ${SEEN.size} keys were seen`);
|
||||
assert.ok(OUTPUTS.length >= 100);
|
||||
for (const text of OUTPUTS) assert.equal(leaks(text), false);
|
||||
});
|
||||
Reference in new issue
Block a user