Compare commits
6
Commits
harness-scan
...
refactor
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ebe2af86de | ||
|
|
1947a056fb | ||
|
|
7ffe932250 | ||
|
|
900daf5213 | ||
|
|
f9dae1ebad | ||
|
|
f063158875 |
No files matched your search
+1
-1
@@ -333,7 +333,7 @@ For local development on a named instance only, `mosaic bus start BUSINESS --dev
|
||||
|
||||
`mosaic console start --dev-local-owner --dev-insecure-bind ADDR --dev-allow-host NAME[,NAME...]` opens that dev owner Console to other machines, for UI testing from another device on a trusted LAN. It has no sign-in: anyone who can reach ADDR:PORT acts as the owner. Start, and `mosaic console status` while that Console runs, print `WARNING: dev owner Console listens on ADDR:PORT with no sign-in; anyone who can reach it acts as the owner.` on stderr. ADDR is `0.0.0.0` or an IPv4 address, and a specific address must also be one of the names. NAME is a host name or IPv4 address without a port; give several as one comma-separated value. The Console answers only `NAME:<bound port>` and the loopback names, and takes a write only when Origin is `http://` plus that same Host. A browser on such an http origin sends no Fetch Metadata, so for an allowed NAME a write with that exact Origin and no `Sec-Fetch-Site` counts as same-origin; a `Sec-Fetch-Site` the browser did send is still checked. Both flags are refused without `--dev-local-owner`, and `--dev-insecure-bind` is refused without a `--dev-allow-host`. SSO, preview and tester Consoles stay on loopback. The Console state file records `bind` and `allowHosts`. `mosaic console status` prints the state as JSON (`bind` is `127.0.0.1` for a loopback Console) and never moves a stale state file.
|
||||
|
||||
`mosaic console restart` and `mosaic org reload ORG` are the Console's Settings controls (F3) from a terminal (U1, #1873). Each sends one request to the running Console named in the instance's console state, as the page sends it (the Console's own Host and Origin, `Sec-Fetch-Site: same-origin`, a JSON `{}` body), so the Console applies the same owner, `--send` and origin checks and writes the same action trail, `<dataRoot>/console/actions.jsonl`. Both refuse inside an agent run (exit 3) before reading anything, and refuse with exit 3 when no Console runs. `console restart` works only for a Console started in the foreground by `mosaic console start` (otherwise the Console answers `not-supervised`); it waits up to 30 seconds for the relaunched Console to answer and prints `{"outcome":"restarted","from":OLD,"boot":NEW}`. `org reload ORG` restarts a running bus or starts a stopped one and prints the Console's done receipt. A request that left but got no answer is reported as outcome unknown (exit 1) and is never retried; check `mosaic console status`, `mosaic bus status ORG` and the trail. `mosaic bus restart ORG` remains the trail-less host verb.
|
||||
`mosaic console restart` and `mosaic org reload ORG` are the Console's Settings controls (F3) from a terminal (U1, #1873). Each sends one request to the running Console named in the instance's console state, as the page sends it (the Console's own Host and Origin, `Sec-Fetch-Site: same-origin`, a JSON `{}` body), so the Console applies the same owner, `--send` and origin checks and writes the same action trail, `<dataRoot>/console/actions.jsonl`. Both refuse inside an agent run (exit 3) before reading anything, and refuse with exit 3 when no Console runs. `console restart` works only for a Console started in the foreground by `mosaic console start`, or by `mosaic console start --dev-local-owner --detach`, which leaves a detached supervisor (`packages/cli/src/console-supervisor.mjs`, its own session) running the Console as the foreground start does; its receipt adds the supervisor's `supervisor` pid, and its later warnings go to `console/supervisor-*.log` under the data root. `mosaic console stop` stops that Console, and its supervisor then exits. `mosaic console start --dev-local-owner` refuses inside an agent run (exit 3), with or without `--detach`, checking the starting process's own environment as well as the one given for the Console; a start without `--dev-local-owner` is unchanged. A detached start whose supervisor does not reply within 60 seconds signals the supervisor's process group, then kills it, and clears `console/start.lock` only when the lock names that supervisor (pid and start time) and it is gone. Any other Console answers `not-supervised`. `console restart` waits up to 30 seconds for the relaunched Console to answer and prints `{"outcome":"restarted","from":OLD,"boot":NEW}`. `org reload ORG` restarts a running bus or starts a stopped one and prints the Console's done receipt. A request that left but got no answer is reported as outcome unknown (exit 1) and is never retried; check `mosaic console status`, `mosaic bus status ORG` and the trail. `mosaic bus restart ORG` remains the trail-less host verb.
|
||||
|
||||
The instance processing switch is separate from per-business launches off/on.
|
||||
`mosaic halt` durably stops managed processing across every business in the
|
||||
|
||||
@@ -595,6 +595,8 @@ async function consoleCommand(rest, io, deps) {
|
||||
// org-lan-bind: refused here, before any config is read, unless both flags come with
|
||||
// --dev-local-owner. ADDR is 0.0.0.0 or an IPv4 address; NAME[,NAME...] has no ports.
|
||||
const devOwner = parsed.flags.has("--dev-local-owner"), bind = parsed.values["--dev-insecure-bind"] ?? null;
|
||||
// D1: an agent run never starts a dev-local-owner Console (org-console.mjs consoleStart checks again).
|
||||
if (devOwner) { refuseInsideAgent(process.env); refuseInsideAgent(io.env); }
|
||||
const names = Object.hasOwn(parsed.values, "--dev-allow-host") ? [parsed.values["--dev-allow-host"]] : [];
|
||||
let lan;
|
||||
try { lan = devLanOptions({ devOwner, bind, names }); } catch (err) { throw usage(err.message); }
|
||||
@@ -703,7 +705,8 @@ async function bus(rest, io, deps) {
|
||||
if (["stop", "restart", "status", "recover"].includes(sub)) {
|
||||
const parsed = parse(args, { flags: sub === "status" ? ["--json"] : [] });
|
||||
if (parsed.positional.length !== 1) throw usage(`bus ${sub} takes one organization`);
|
||||
if (sub === "recover") devOwnerInstance(io.env);
|
||||
// D2 security review: recovery clears host state, so an agent run is refused before any read.
|
||||
if (sub === "recover") { refuseInsideAgent(io.env); devOwnerInstance(io.env); }
|
||||
const ctx = organizationContext(io, deps), id = parsed.positional[0];
|
||||
const result = sub === "stop" ? await busStop(ctx, id) : sub === "restart" ? await busRestart(ctx, id) : sub === "recover" ? await busRecover(ctx, id) : await busStatus(ctx, id);
|
||||
io.stdout.write(`${JSON.stringify(result, null, parsed.flags.has("--json") ? 2 : 0)}\n`);
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
// D1 (#1873): the supervisor of a detached dev-local-owner Console. `mosaic console start
|
||||
// --dev-local-owner --detach` starts this process in its own session (org-console.mjs
|
||||
// detachedSupervisor). It starts the Console exactly as a foreground start does, with --supervised,
|
||||
// and relaunches it when the owner restarts it (F3). Its one stdout line is the start's reply:
|
||||
// { receipt, warnings } or { error, exitCode, warnings }. Later warnings go to stderr, the log.
|
||||
import { consoleStart } from './org-console.mjs';
|
||||
import { CliError } from './errors.mjs';
|
||||
|
||||
const { configDir, dataRoot, instance, port, bind, allowHosts } = JSON.parse(process.argv[2]);
|
||||
const warnings = [];
|
||||
let started = false;
|
||||
const warn = s => { if (started) process.stderr.write(s); else warnings.push(s); };
|
||||
// The starting CLI can be gone before the reply (Ctrl-C, a wrapper's timeout). The Console is
|
||||
// started and its state names it, so a reply nobody reads is dropped and supervision goes on.
|
||||
process.stdout.on('error', () => {});
|
||||
const reply = value => new Promise(resolve => process.stdout.write(`${JSON.stringify(value)}\n`, () => resolve()));
|
||||
// A stop signal during the start (the CLI's start timeout signals this process group) is kept and
|
||||
// acted on once the start settles, so a failed start still releases the start lock.
|
||||
let stopAsked = false;
|
||||
const early = () => { stopAsked = true; };
|
||||
for (const signal of ['SIGINT', 'SIGTERM']) process.on(signal, early);
|
||||
let receipt;
|
||||
try {
|
||||
receipt = await consoleStart({ configDir, dataRoot, instance, env: process.env }, { devLocalOwner: true, port, bind, allowHosts, warn });
|
||||
} catch (e) {
|
||||
const exitCode = e instanceof CliError ? e.exitCode : 1;
|
||||
if (!(e instanceof CliError)) process.stderr.write(`${e?.stack ?? e}\n`);
|
||||
await reply({ ...(e instanceof CliError ? { error: e.message } : {}), exitCode, warnings });
|
||||
process.exit(exitCode);
|
||||
}
|
||||
started = true;
|
||||
// A stop signal ends supervision cleanly, as in the foreground (cli.mjs consoleCommand).
|
||||
const stop = () => receipt.stop();
|
||||
for (const signal of ['SIGINT', 'SIGTERM']) { process.off(signal, early); process.on(signal, stop); }
|
||||
if (stopAsked) stop();
|
||||
await reply({ receipt, warnings });
|
||||
const code = await receipt.done;
|
||||
if (code !== 0) process.stderr.write(`mosaic: console stopped unexpectedly (${code})\n`);
|
||||
process.exit(code);
|
||||
@@ -2,7 +2,7 @@
|
||||
import { randomBytes } from 'node:crypto';
|
||||
import { processState } from '../../seat/src/process-state.mjs';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { constants, closeSync, lstatSync, mkdirSync, openSync, readFileSync, fstatSync, unlinkSync, writeFileSync } from 'node:fs';
|
||||
import { constants, closeSync, linkSync, lstatSync, mkdirSync, openSync, readFileSync, fstatSync, renameSync, unlinkSync, writeFileSync } from 'node:fs';
|
||||
import { connect } from 'node:net';
|
||||
import { join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
@@ -151,6 +151,27 @@ export async function busRestart(ctx, id) {
|
||||
return busStart(ctx, id, { detach: true, ...options });
|
||||
}
|
||||
|
||||
// D2 security review: clear only the entries busRecover inspected. Each is renamed aside (atomic)
|
||||
// and the renamed entry is checked by device and inode; an entry swapped after the inspection is
|
||||
// put back with everything moved so far, and the recovery refuses before any receipt or deletion.
|
||||
// The deletion then names only the aside entries, never the live paths.
|
||||
export function setAside(entries) {
|
||||
const tag = randomBytes(4).toString('hex'), moved = [];
|
||||
const restore = () => {
|
||||
// link() refuses an existing path, so a put-back never replaces whatever now holds the name; an
|
||||
// entry that cannot go back stays under its .recovering name for a person to inspect.
|
||||
for (const m of moved.slice().reverse()) { try { linkSync(m.aside, m.path); unlinkSync(m.aside); } catch { /* left aside */ } }
|
||||
};
|
||||
for (const entry of entries) {
|
||||
const aside = `${entry.path}.recovering-${tag}`;
|
||||
let current;
|
||||
try { renameSync(entry.path, aside); moved.push({ path: entry.path, aside }); current = lstatSync(aside); }
|
||||
catch { restore(); fail('bus recovery state changed'); }
|
||||
if (current.dev !== entry.st.dev || current.ino !== entry.st.ino) { restore(); fail('bus recovery state changed'); }
|
||||
}
|
||||
return { restore, remove: () => { for (const m of moved.slice().reverse()) unlinkSync(m.aside); } };
|
||||
}
|
||||
|
||||
// Recover only while the org lifecycle fence excludes new starts and deletion.
|
||||
export async function busRecover(ctx, id) {
|
||||
return withOrgLifecycle(ctx, id, async () => {
|
||||
@@ -177,10 +198,12 @@ export async function busRecover(ctx, id) {
|
||||
const h = state.value;
|
||||
if (!Number.isSafeInteger(h?.pid) || h.pid <= 0 || typeof h.startTime !== 'string' || !/^\d+$/.test(h.startTime) || typeof h.startedAt !== 'string' || !Number.isFinite(Date.parse(h.startedAt)) || h.business !== id) fail('invalid or mismatched bus host state');
|
||||
let p; try { p = processState(h.pid); } catch { fail('bus host identity is uncertain'); }
|
||||
// A live pid with the recorded start time is the recorded process: never proof of death. Its argv
|
||||
// only picks the message (D2 security review: a mismatching argv used to read as a dead host).
|
||||
if (p.alive && p.startTime === h.startTime) {
|
||||
const argv = cmdlineOf(h.pid); if (!argv || !argv.length) fail('bus host identity is uncertain');
|
||||
const i = argv.indexOf(CLI);
|
||||
const argv = cmdlineOf(h.pid), i = argv?.indexOf(CLI) ?? -1;
|
||||
if (i >= 0 && argv[i + 1] === 'bus' && argv[i + 2] === 'start' && argv[i + 3] === id) fail(`bus host pid ${h.pid} is alive`);
|
||||
fail('bus host identity is uncertain');
|
||||
}
|
||||
if (lock && (lock.value?.pid !== h.pid || lock.value?.startTime !== h.startTime || lock.value?.business !== id || lock.value?.startedAt !== h.startedAt)) fail('bus host lock does not match host state');
|
||||
if (writer) {
|
||||
@@ -198,14 +221,43 @@ export async function busRecover(ctx, id) {
|
||||
if (occupied) fail('bus socket answers or is uncertain');
|
||||
}
|
||||
const entries = [state, lock, writer, socket].filter(Boolean);
|
||||
for (const entry of entries) {
|
||||
const current = lstatSync(entry.path);
|
||||
if (current.dev !== entry.st.dev || current.ino !== entry.st.ino) fail('bus recovery state changed');
|
||||
}
|
||||
const aside = setAside(entries);
|
||||
const at = new Date().toISOString();
|
||||
const receipt = join(org.paths.hostDir, `recovery-${at.replace(/[:.]/g, '-')}-${randomBytes(4).toString('hex')}.json`);
|
||||
writeFileSync(receipt, JSON.stringify({ action: 'bus.recover', at, org: id, stopped: true, host: h, lock: lock?.value ?? null, writer: writer?.value ?? null, socket: Boolean(socket) }, null, 2) + '\n', { mode: 0o600, flag: 'wx' });
|
||||
for (const entry of entries.slice().reverse()) unlinkSync(entry.path);
|
||||
return { org: id, recovered: true, pid: h.pid, receipt };
|
||||
try { writeFileSync(receipt, JSON.stringify({ action: 'bus.recover', at, org: id, stopped: true, host: h, lock: lock?.value ?? null, writer: writer?.value ?? null, socket: Boolean(socket) }, null, 2) + '\n', { mode: 0o600, flag: 'wx' }); }
|
||||
catch (e) { aside.restore(); throw e; }
|
||||
aside.remove();
|
||||
const result = { org: id, recovered: true, pid: h.pid, receipt };
|
||||
// The dead host's start options, for busRevive. Not part of the printed receipt.
|
||||
const o = h.options && typeof h.options === 'object' ? h.options : {};
|
||||
Object.defineProperty(result, 'options', { value: Object.freeze({ devLocalOwner: o.devLocalOwner === true, pm: o.pm === true, noLaunch: o.noLaunch === true }) });
|
||||
return result;
|
||||
});
|
||||
}
|
||||
|
||||
// D2 (P0-3): bring back a bus whose host was killed, as a Console unit stop does to the hosts the
|
||||
// Console started. busRecover is the proof: it refuses a live or unprovable host, a lock that does
|
||||
// not match the dead host, a live writer and an answering socket. The host restarts on the options
|
||||
// the dead host recorded, never wider; `pm: false` also keeps the chief from launching.
|
||||
export function reviveStartOptions(recorded, { pm = true } = {}) {
|
||||
const o = recorded && typeof recorded === 'object' ? recorded : {};
|
||||
return { detach: true, devLocalOwner: o.devLocalOwner === true, pm: pm === true && o.pm === true, noLaunch: o.noLaunch === true };
|
||||
}
|
||||
export async function busRevive(ctx, id, { pm = true } = {}) {
|
||||
await target(ctx, id);
|
||||
const recovered = await busRecover(ctx, id);
|
||||
const options = reviveStartOptions(recovered.options, { pm });
|
||||
const cleared = { pid: recovered.pid, receipt: recovered.receipt };
|
||||
let started;
|
||||
try { started = await busStart(ctx, id, options); }
|
||||
catch (e) {
|
||||
// The dead host's state is gone by now: the caller must not report "nothing changed".
|
||||
if (e && typeof e === 'object') { try { Object.defineProperty(e, 'recovered', { value: cleared }); } catch { /* frozen: the caller sees a plain failure */ } }
|
||||
throw e;
|
||||
}
|
||||
const receipt = { ...started, recovered: cleared };
|
||||
Object.defineProperty(receipt, 'done', { value: started.done });
|
||||
// The chief was not relaunched although the dead host was started with --pm.
|
||||
Object.defineProperty(receipt, 'pmDropped', { value: recovered.options.pm === true && !options.pm });
|
||||
return receipt;
|
||||
}
|
||||
@@ -3,13 +3,14 @@ import { randomBytes } from 'node:crypto';
|
||||
import { processState } from '../../seat/src/process-state.mjs';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { request as httpRequest } from 'node:http';
|
||||
import { constants, closeSync, fstatSync, lstatSync, mkdirSync, openSync, readFileSync, rmSync, renameSync, writeFileSync } from 'node:fs';
|
||||
import { constants, closeSync, fstatSync, lstatSync, mkdirSync, openSync, readFileSync, rmSync, renameSync, writeFileSync, writeSync } from 'node:fs';
|
||||
import { basename, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { listOrgs, resolveOrg } from '../../organization/src/org-registry.mjs';
|
||||
import { cmdlineOf, startTimeOf } from './host.mjs';
|
||||
import { CliError } from './errors.mjs';
|
||||
import { devLanOptions } from './dev-local-owner.mjs';
|
||||
import { refuseInsideAgent } from './transport.mjs';
|
||||
|
||||
const WEBUI = fileURLToPath(new URL('../../webui/src/cli.mjs', import.meta.url));
|
||||
const forbidden = new Set([7332, 7340, 7341, 33223, 17450, 17443, 44307]);
|
||||
@@ -63,8 +64,25 @@ function takeLock(ctx) {
|
||||
checkDir(ctx);
|
||||
const path = join(directory(ctx), 'start.lock');
|
||||
let fd; try { fd = openSync(path, 'wx', 0o600); } catch { fail('console start lock exists'); }
|
||||
// D1: the lock names its holder, so a detached start that timed out can clear the lock of the
|
||||
// supervisor it killed (clearDeadLock) and no other.
|
||||
try { writeSync(fd, JSON.stringify({ pid: process.pid, startTime: startTimeOf(process.pid) })); } catch { /* an unnamed lock is only never cleared */ }
|
||||
return () => { closeSync(fd); rmSync(path, { force: true }); };
|
||||
}
|
||||
// Removes start.lock only when it names this exact process (pid and start time) and that process
|
||||
// is gone. A lock that is unreadable, unnamed or another holder's stays.
|
||||
function clearDeadLock(ctx, pid, startTime) {
|
||||
if (!startTime) return;
|
||||
const path = join(directory(ctx), 'start.lock');
|
||||
let owner;
|
||||
try {
|
||||
const st = lstatSync(path);
|
||||
if (!st.isFile() || st.uid !== process.getuid()) return;
|
||||
owner = JSON.parse(readFileSync(path, 'utf8'));
|
||||
} catch { return; }
|
||||
if (owner?.pid !== pid || owner.startTime !== startTime || startTimeOf(pid) === startTime) return;
|
||||
rmSync(path, { force: true });
|
||||
}
|
||||
// The exit code a supervised Console uses to ask for a relaunch (EX_TEMPFAIL). Only the Console's
|
||||
// own restart route (POST /api/console/restart, a human action) exits with it.
|
||||
export const RESTART_EXIT = 75;
|
||||
@@ -153,15 +171,76 @@ function supervise(ctx, first, args, lan, warn) {
|
||||
})();
|
||||
return { done, stop };
|
||||
}
|
||||
// D1 (#1873): `--dev-local-owner --detach` runs the foreground supervision above in a process of
|
||||
// its own session (console-supervisor.mjs), so Restart Console works on a detached dev Console. The
|
||||
// supervisor takes the start lock and writes the console state as a foreground start does. This
|
||||
// process waits for the supervisor's one-line reply, passes its warnings on and leaves it running.
|
||||
const SUPERVISOR = fileURLToPath(new URL('./console-supervisor.mjs', import.meta.url));
|
||||
async function detachedSupervisor(ctx, { port, bind, allowHosts, warn }) {
|
||||
checkDir(ctx);
|
||||
lastStamp = Math.max(Date.now(), lastStamp + 1);
|
||||
const name = `supervisor-${process.pid}-${lastStamp}.log`;
|
||||
const fd = openSync(join(directory(ctx), name), constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW, 0o600);
|
||||
let child;
|
||||
try {
|
||||
child = spawn(process.execPath, [SUPERVISOR, JSON.stringify({ configDir: ctx.configDir, dataRoot: ctx.dataRoot, instance: ctx.instance, port, bind, allowHosts })],
|
||||
{ env: ctx.env ?? process.env, detached: true, stdio: ['ignore', 'pipe', fd] });
|
||||
} finally { closeSync(fd); }
|
||||
const supervisorStart = child.pid ? startTimeOf(child.pid) : null;
|
||||
const done = new Promise(resolve => { child.once('error', () => resolve(1)); child.once('exit', code => resolve(code ?? 1)); });
|
||||
// The log is named under the data root, never by host path (PR #1916).
|
||||
const unclear = `console failed to start; see console/${name} under the data root`;
|
||||
let text = '', timer;
|
||||
child.stdout.setEncoding('utf8');
|
||||
const line = await Promise.race([
|
||||
new Promise(resolve => {
|
||||
child.stdout.on('data', c => { text += c; const n = text.indexOf('\n'); if (n >= 0) resolve(text.slice(0, n)); else if (text.length > 65536) resolve(null); });
|
||||
child.stdout.once('end', () => resolve(null));
|
||||
}),
|
||||
new Promise(resolve => { timer = setTimeout(() => resolve(undefined), 60000); }),
|
||||
]);
|
||||
clearTimeout(timer);
|
||||
child.stdout.destroy();
|
||||
// The supervisor leads its own process group, with the Console it may have started: a timed-out
|
||||
// start signals that group, then kills it if it is still there after 10 seconds. A supervisor
|
||||
// killed while it held the start lock never released it, so the lock it named is cleared.
|
||||
if (line === undefined) {
|
||||
const group = signal => { try { process.kill(-child.pid, signal); } catch { /* already gone */ } };
|
||||
group('SIGTERM');
|
||||
let grace; const exited = await Promise.race([done.then(() => true), new Promise(r => { grace = setTimeout(() => r(false), 10000); })]);
|
||||
clearTimeout(grace);
|
||||
if (!exited) { group('SIGKILL'); await done; }
|
||||
clearDeadLock(ctx, child.pid, supervisorStart);
|
||||
fail(unclear);
|
||||
}
|
||||
let reply = null;
|
||||
try { reply = JSON.parse(line); } catch { /* no reply */ }
|
||||
for (const w of Array.isArray(reply?.warnings) ? reply.warnings : []) if (typeof w === 'string') warn(w);
|
||||
child.unref();
|
||||
if (typeof reply?.receipt?.url !== 'string' || !Number.isSafeInteger(reply.receipt.pid)) {
|
||||
const code = [1, 2, 3, 4].includes(reply?.exitCode) ? reply.exitCode : 1;
|
||||
throw new CliError(typeof reply?.error === 'string' ? reply.error : unclear, code);
|
||||
}
|
||||
const result = { ...reply.receipt, supervisor: child.pid };
|
||||
Object.defineProperty(result, 'done', { value: done });
|
||||
Object.defineProperty(result, 'stop', { value: () => child.kill('SIGTERM') });
|
||||
return result;
|
||||
}
|
||||
// `bind` and `allowHosts` are --dev-insecure-bind and --dev-allow-host (org-lan-bind): the dev
|
||||
// owner Console on the LAN with no sign-in. Neither is set for a loopback Console.
|
||||
// In the foreground the Console runs with --supervised: its restart route exits RESTART_EXIT and
|
||||
// this process relaunches it in place (F3). A detached Console has no supervisor.
|
||||
// this process relaunches it in place (F3). A detached dev-local-owner Console runs under a
|
||||
// detached supervisor (D1); any other detached Console has none.
|
||||
export async function consoleStart(ctx, { devLocalOwner = false, detach = false, port = 'auto', bind = null, allowHosts = [], warn = s => process.stderr.write(s) } = {}) {
|
||||
const value = port === 'auto' ? 0 : Number(port);
|
||||
if (!(port === 'auto' || (typeof port === 'string' && /^\d+$/.test(port) && value >= 1 && value <= 65535 && !forbidden.has(value)))) fail('invalid or forbidden console port');
|
||||
let lan;
|
||||
try { lan = devLanOptions({ devOwner: devLocalOwner, bind, names: allowHosts }); } catch (e) { throw new CliError(e.message, 4); }
|
||||
// D1 fix round (security review P1): a dev-local-owner Console acts as the owner with no sign-in,
|
||||
// so an agent run never starts one, detached or not. Other starts are unchanged. This process's own
|
||||
// environment is checked as well as the one given for the Console (security re-check P1).
|
||||
if (devLocalOwner) { refuseInsideAgent(process.env); if (ctx.env) refuseInsideAgent(ctx.env); }
|
||||
if (detach && devLocalOwner) return detachedSupervisor(ctx, { port, bind, allowHosts, warn });
|
||||
const release = takeLock(ctx);
|
||||
let run, args;
|
||||
try {
|
||||
|
||||
@@ -115,21 +115,47 @@ test('mosaic console restart relaunches a supervised Console and the trail recor
|
||||
assert.deepEqual(warnings, ['mosaic: console asked for a restart; relaunching on the same arguments\n']);
|
||||
});
|
||||
|
||||
test('a detached Console refuses mosaic console restart, and org reload of an unknown org, with no trail', { timeout: 90000 }, async t => {
|
||||
// D1 (#1873): a detached dev-owner Console runs under a detached supervisor, so the terminal restart
|
||||
// works on it as on a foreground one. Before D1 this Console refused with not-supervised.
|
||||
test('a detached dev-owner Console restarts through mosaic console restart, and org reload of an unknown org is refused with no trail', { timeout: 120000 }, async t => {
|
||||
const { ctx, env, trailFile } = await instance(t);
|
||||
const result = await consoleStart(ctx, { devLocalOwner: true, detach: true, port: 'auto', warn: () => {} });
|
||||
const port = Number(new URL(result.url).port);
|
||||
assert.ok(!FORBIDDEN.includes(port));
|
||||
const r = await mosaic(env, ['console', 'restart']);
|
||||
assert.equal(r.status, 3, r.stderr);
|
||||
assert.match(r.stderr, /console restart refused \(not-supervised\)/);
|
||||
// The real organization service refuses an org that does not exist before any trail line.
|
||||
const unknown = await mosaic(env, ['org', 'reload', 'nosuch']);
|
||||
assert.equal(unknown.status, 3, unknown.stderr);
|
||||
assert.match(unknown.stderr, /org reload refused \(org-[a-z-]+\)/);
|
||||
assert.equal(existsSync(trailFile), false);
|
||||
const before = (await get(port, '/api/config')).body;
|
||||
assert.equal(before.consoleRestart, true);
|
||||
const r = await mosaic(env, ['console', 'restart']);
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
const answer = JSON.parse(r.stdout);
|
||||
assert.equal(answer.outcome, 'restarted'); assert.equal(answer.from, before.boot);
|
||||
assert.equal((await get(port, '/api/config')).body.boot, answer.boot);
|
||||
assert.deepEqual(trailLines(trailFile), [{ by: DEV, action: 'console.restart', outcome: 'restarting', boot: before.boot }]);
|
||||
assert.equal((await consoleStop(ctx)).stopped, true);
|
||||
// The supervisor is unref'd, so a timer keeps this process awake while it exits.
|
||||
let code; result.done.then(c => { code = c; });
|
||||
for (let i = 0; i < 200 && code === undefined; i++) await new Promise(r => setTimeout(r, 50));
|
||||
assert.equal(code, 0);
|
||||
});
|
||||
|
||||
// A dev-owner Console nothing supervises (here one in this process, started with no restart hook)
|
||||
// still refuses the terminal restart and stays up, with no trail line.
|
||||
test('an unsupervised Console refuses mosaic console restart with not-supervised and stays up', { timeout: 90000 }, async t => {
|
||||
const { ctx, env, trailFile } = await instance(t);
|
||||
const server = await startServer({ port: 0, board: 'http://127.0.0.1:9', sso: createDevOwner(), send: async () => ({}), log: () => {}, trail: actionTrail(join(ctx.dataRoot, 'console')) });
|
||||
t.after(() => close(server));
|
||||
const port = server.address().port;
|
||||
assert.ok(!FORBIDDEN.includes(port));
|
||||
recordConsole(ctx, `http://127.0.0.1:${port}/`);
|
||||
const r = await mosaic(env, ['console', 'restart']);
|
||||
assert.equal(r.status, 3, r.stderr);
|
||||
assert.match(r.stderr, /console restart refused \(not-supervised\)/);
|
||||
assert.equal((await get(port, '/api/config')).status, 200);
|
||||
assert.equal(existsSync(trailFile), false);
|
||||
assert.equal((await consoleStop(ctx)).stopped, true);
|
||||
});
|
||||
|
||||
// The F3 bus control through the Console's own route: a stopped bus starts and a running one restarts.
|
||||
|
||||
@@ -1,15 +1,22 @@
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, readdirSync, readFileSync, rmSync, existsSync, statSync } from 'node:fs';
|
||||
import { get, request } from 'node:http';
|
||||
import { join } from 'node:path';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { reserveOrg, activateOrg } from '../../business/src/org-registry.mjs';
|
||||
import { humanEnv } from './helpers.mjs';
|
||||
import { consoleStart, consoleStatus, consoleStop } from '../src/org-console.mjs';
|
||||
import { consoleStart, consoleStatus, consoleStop, consoleRestart } from '../src/org-console.mjs';
|
||||
|
||||
const REPO = new URL('../../../', import.meta.url).pathname;
|
||||
// A detached supervisor's handle is unref'd, so its exit is awaited with a timer that keeps the loop alive.
|
||||
async function exitOf(done, ms = 20000) {
|
||||
let code; done.then(c => { code = c; });
|
||||
for (const deadline = Date.now() + ms; code === undefined; await new Promise(r => setTimeout(r, 100))) if (Date.now() > deadline) assert.fail('the supervisor did not exit');
|
||||
return code;
|
||||
}
|
||||
test('detached console reports its live loopback URL and stops', { timeout: 60000 }, async t => {
|
||||
const home = mkdtempSync(join(tmpdir(), 's1-'));
|
||||
const env = humanEnv({ HOME: home, MOSAIC_CONFIG: join(home, '.config/mosaic-s1test/config.json') });
|
||||
@@ -31,11 +38,13 @@ test('detached console reports its live loopback URL and stops', { timeout: 6000
|
||||
assert.ok(Number.isSafeInteger(result.pid));
|
||||
// A selector producer (rename r2): the Console child gets the active organization as --business.
|
||||
const argv = readFileSync(`/proc/${result.pid}/cmdline`, 'utf8').split('\0').slice(0, -1);
|
||||
assert.deepEqual(argv, [process.execPath, join(REPO, 'packages/webui/src/cli.mjs'), 'serve', '--port', '0', '--business', 'alpha', '--dev-local-owner']);
|
||||
// D1 (#1873): a detached dev-owner Console runs under a detached supervisor, so it is --supervised.
|
||||
assert.deepEqual(argv, [process.execPath, join(REPO, 'packages/webui/src/cli.mjs'), 'serve', '--port', '0', '--business', 'alpha', '--dev-local-owner', '--supervised']);
|
||||
assert.ok(![7332, 7340, 7341, 33223, 17450, 17443, 44307, 7350].includes(Number(url.port)));
|
||||
const status = await new Promise((resolve, reject) => get(url, res => { res.resume(); resolve(res.statusCode); }).on('error', reject));
|
||||
assert.equal(status, 200);
|
||||
assert.equal((await consoleStop(ctx)).stopped, true);
|
||||
assert.equal(await exitOf(result.done), 0);
|
||||
});
|
||||
|
||||
test('mosaic console start and stop dispatch through the CLI', { timeout: 60000 }, async t => {
|
||||
@@ -198,15 +207,208 @@ test('a foreground console relaunches on the same address when the owner restart
|
||||
assert.equal(consoleStatus(ctx).running, false);
|
||||
});
|
||||
|
||||
test('a detached console has no supervisor, so it refuses a restart and stays up', { timeout: 60000 }, async t => {
|
||||
// D1 (#1873, readiness audit P1-1): Restart Console on a detached dev-owner Console. The start runs
|
||||
// the foreground supervision in a process of its own session; the Console is that process's child.
|
||||
const SUPERVISOR = join(REPO, 'packages/cli/src/console-supervisor.mjs');
|
||||
const statOf = pid => readFileSync(`/proc/${pid}/stat`, 'utf8').replace(/^.*\) /, '').split(' ');
|
||||
test('a detached dev-owner console runs under a detached supervisor, so the owner restarts it in place', { timeout: 120000 }, async t => {
|
||||
const { ctx, mosaic } = await instance(t);
|
||||
const warnings = [];
|
||||
const result = await consoleStart(ctx, { devLocalOwner: true, detach: true, port: 'auto', warn: s => warnings.push(s) });
|
||||
t.after(() => { try { process.kill(result.supervisor, 'SIGTERM'); } catch {} });
|
||||
assert.deepEqual(warnings, []);
|
||||
assert.deepEqual(Object.keys(result), ['url', 'pid', 'supervisor']);
|
||||
const port = Number(new URL(result.url).port);
|
||||
assert.ok(![7332, 7340, 7341, 33223, 17450, 17443, 44307, 7350].includes(port));
|
||||
const sup = argvOf(result.supervisor);
|
||||
assert.deepEqual(sup.slice(0, 2), [process.execPath, SUPERVISOR]);
|
||||
assert.deepEqual(JSON.parse(sup[2]), { configDir: ctx.configDir, dataRoot: ctx.dataRoot, instance: 's1test', port: 'auto', bind: null, allowHosts: [] });
|
||||
// The supervisor leads its own session; the Console is its child, started as in the foreground.
|
||||
assert.equal(Number(statOf(result.supervisor)[3]), result.supervisor);
|
||||
assert.equal(Number(statOf(result.pid)[1]), result.supervisor);
|
||||
assert.deepEqual(argvOf(result.pid), [process.execPath, WEBUI, 'serve', '--port', '0', '--business', 'alpha', '--dev-local-owner', '--supervised']);
|
||||
const before = (await call(port, '/api/config')).body;
|
||||
assert.equal(before.consoleRestart, true);
|
||||
const stateFile = join(ctx.dataRoot, 'console/console.json'), trailFile = join(ctx.dataRoot, 'console/actions.jsonl');
|
||||
// Another origin or site restarts nothing.
|
||||
for (const headers of [{ origin: 'http://evil.example' }, { origin: null }, { 'sec-fetch-site': 'cross-site' }, { host: `evil.example:${port}` }]) {
|
||||
assert.equal((await call(port, '/api/console/restart', { method: 'POST', body: '{}', headers })).status, 403, JSON.stringify(headers));
|
||||
}
|
||||
assert.equal(existsSync(trailFile), false);
|
||||
// The page's request, as `mosaic console restart` sends it: 202, then a new boot on the same address.
|
||||
const restarted = await consoleRestart(ctx);
|
||||
assert.equal(restarted.outcome, 'restarted'); assert.equal(restarted.from, before.boot); assert.notEqual(restarted.boot, before.boot);
|
||||
// The relaunched Console answers before the supervisor writes its state (relaunch), so wait for it.
|
||||
const saved = await relaunchedState(ctx, result.pid);
|
||||
assert.notEqual(saved.pid, result.pid);
|
||||
assert.equal(saved.url, result.url);
|
||||
assert.equal(Number(statOf(saved.pid)[1]), result.supervisor);
|
||||
assert.deepEqual(argvOf(saved.pid), [process.execPath, WEBUI, 'serve', '--port', String(port), '--business', 'alpha', '--dev-local-owner', '--supervised']);
|
||||
const after = (await call(port, '/api/config')).body;
|
||||
assert.equal(after.boot, restarted.boot); assert.equal(after.consoleRestart, true);
|
||||
const [line, ...more] = readFileSync(trailFile, 'utf8').split('\n').filter(Boolean).map(l => JSON.parse(l));
|
||||
assert.deepEqual(more, []);
|
||||
assert.deepEqual({ ...line, at: undefined }, { at: undefined, by: DEV, action: 'console.restart', outcome: 'restarting', boot: before.boot });
|
||||
// The supervisor's later warnings go to its log under the data root.
|
||||
const logs = readdirSync(join(ctx.dataRoot, 'console')).filter(x => /^supervisor-\d+-\d+\.log$/.test(x));
|
||||
assert.equal(logs.length, 1);
|
||||
assert.equal(statSync(join(ctx.dataRoot, 'console', logs[0])).mode & 0o777, 0o600);
|
||||
assert.equal(readFileSync(join(ctx.dataRoot, 'console', logs[0]), 'utf8'), 'mosaic: console asked for a restart; relaunching on the same arguments\n');
|
||||
// A second start is refused by the supervisor's start and says so here; nothing else starts.
|
||||
await assert.rejects(consoleStart(ctx, { devLocalOwner: true, detach: true }), e => e.exitCode === 3 && /console state exists/.test(e.message));
|
||||
const cli = mosaic(['console', 'start', '--dev-local-owner', '--detach', '--port', 'auto']);
|
||||
assert.equal(cli.status, 3); assert.match(cli.stderr, /console state exists/); assert.equal(cli.stdout, '');
|
||||
assert.deepEqual(consoleStatus(ctx), { running: true, pid: saved.pid, url: result.url, bind: '127.0.0.1' });
|
||||
// A stop ends the Console and then the supervisor, cleanly.
|
||||
assert.deepEqual(await consoleStop(ctx), { stopped: true, pid: saved.pid });
|
||||
assert.equal(await exitOf(result.done), 0);
|
||||
assert.deepEqual(consoleStatus(ctx), { running: false });
|
||||
});
|
||||
|
||||
test('a supervisor stopped by a signal stops its detached dev-owner console', { timeout: 60000 }, async t => {
|
||||
const { ctx } = await instance(t);
|
||||
const result = await consoleStart(ctx, { devLocalOwner: true, detach: true, port: 'auto', warn: () => {} });
|
||||
t.after(() => { try { process.kill(result.supervisor, 'SIGTERM'); } catch {} });
|
||||
result.stop();
|
||||
assert.equal(await exitOf(result.done), 0);
|
||||
assert.equal(consoleStatus(ctx).running, false);
|
||||
});
|
||||
|
||||
// relaunch() writes console.json after the new Console answers; read it once it names a new pid.
|
||||
async function relaunchedState(ctx, oldPid) {
|
||||
const file = join(ctx.dataRoot, 'console/console.json');
|
||||
for (const deadline = Date.now() + 15000; ; await new Promise(r => setTimeout(r, 50))) {
|
||||
const saved = JSON.parse(readFileSync(file, 'utf8'));
|
||||
if (saved.pid !== oldPid || Date.now() > deadline) return saved;
|
||||
}
|
||||
}
|
||||
|
||||
// D1 fix round (build review P2): the starting CLI can die before it reads the supervisor's reply
|
||||
// (Ctrl-C, a wrapper's timeout). The reply is dropped, and the supervisor keeps supervising.
|
||||
test('a supervisor whose starting CLI is gone keeps supervising, so Restart Console still relaunches', { timeout: 60000 }, async t => {
|
||||
const { ctx } = await instance(t);
|
||||
const child = spawn(process.execPath, [SUPERVISOR, JSON.stringify({ configDir: ctx.configDir, dataRoot: ctx.dataRoot, instance: 's1test', port: 'auto', bind: null, allowHosts: [] })],
|
||||
{ env: ctx.env, detached: true, stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
t.after(() => { try { process.kill(child.pid, 'SIGTERM'); } catch {} });
|
||||
const exit = new Promise(resolve => child.once('exit', code => resolve(code)));
|
||||
let stderr = ''; child.stderr.setEncoding('utf8').on('data', c => { stderr += c; });
|
||||
child.stdout.destroy();
|
||||
let status = consoleStatus(ctx);
|
||||
for (const deadline = Date.now() + 30000; !status.running; status = consoleStatus(ctx)) {
|
||||
if (Date.now() > deadline) assert.fail(`the console did not start: ${stderr}`);
|
||||
await new Promise(r => setTimeout(r, 100));
|
||||
}
|
||||
// The reply follows the state write; give it time to meet the closed pipe.
|
||||
await new Promise(r => setTimeout(r, 1000));
|
||||
assert.equal(child.exitCode, null, stderr);
|
||||
assert.equal(Number(statOf(status.pid)[1]), child.pid);
|
||||
const restarted = await consoleRestart(ctx);
|
||||
assert.equal(restarted.outcome, 'restarted');
|
||||
const after = await relaunchedState(ctx, status.pid);
|
||||
assert.notEqual(after.pid, status.pid); assert.equal(after.url, status.url);
|
||||
assert.equal(Number(statOf(after.pid)[1]), child.pid);
|
||||
assert.deepEqual(await consoleStop(ctx), { stopped: true, pid: after.pid });
|
||||
assert.equal(await exit, 0);
|
||||
assert.doesNotMatch(stderr, /EPIPE|Unhandled/);
|
||||
});
|
||||
|
||||
// D1 fix round (security review P2): a stop signal while the supervisor holds the start lock is kept
|
||||
// until the start settles; then the Console stops, the lock is released and the supervisor exits.
|
||||
test('a supervisor signalled during its start stops the console it started and releases the start lock', { timeout: 60000 }, async t => {
|
||||
const { ctx } = await instance(t);
|
||||
const child = spawn(process.execPath, [SUPERVISOR, JSON.stringify({ configDir: ctx.configDir, dataRoot: ctx.dataRoot, instance: 's1test', port: 'auto', bind: null, allowHosts: [] })],
|
||||
{ env: ctx.env, detached: true, stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
t.after(() => { try { process.kill(-child.pid, 'SIGKILL'); } catch {} });
|
||||
const exit = new Promise(resolve => child.once('exit', (code, signal) => resolve({ code, signal })));
|
||||
let stdout = '', stderr = '';
|
||||
child.stdout.setEncoding('utf8').on('data', c => { stdout += c; });
|
||||
child.stderr.setEncoding('utf8').on('data', c => { stderr += c; });
|
||||
const lock = join(ctx.dataRoot, 'console/start.lock');
|
||||
for (const deadline = Date.now() + 20000; !existsSync(lock); await new Promise(r => setTimeout(r, 2))) {
|
||||
if (Date.now() > deadline) assert.fail(`the supervisor never took the start lock: ${stderr}`);
|
||||
}
|
||||
process.kill(child.pid, 'SIGTERM');
|
||||
assert.deepEqual(await exit, { code: 0, signal: null }, stderr);
|
||||
assert.equal(existsSync(lock), false);
|
||||
// As after a foreground stop by signal, the state file stays and reads stale, not running.
|
||||
assert.equal(consoleStatus(ctx).running, false);
|
||||
// The started Console was this supervisor's child, and it is gone.
|
||||
const reply = JSON.parse(stdout.trim().split('\n')[0]);
|
||||
assert.equal(existsSync(`/proc/${reply.receipt.pid}`), false);
|
||||
});
|
||||
|
||||
// D1 fix round (security review P1): an agent run never starts a dev-local-owner Console.
|
||||
test('an agent run cannot start a dev-local-owner console; a start without it is unchanged', { timeout: 60000 }, async t => {
|
||||
const { ctx, mosaic } = await instance(t);
|
||||
const agent = { ...ctx, env: { ...ctx.env, MOSAIC_RUN_ID: 'run-1' } };
|
||||
for (const detach of [true, false]) {
|
||||
await assert.rejects(consoleStart(agent, { devLocalOwner: true, detach, port: 'auto', warn: () => {} }), e => e.exitCode === 3 && /only from a human shell/.test(e.message));
|
||||
}
|
||||
// A clean environment given for the Console does not hide this process's own marker (re-check P1).
|
||||
process.env.MOSAIC_RUN_ID = 'run-2';
|
||||
try {
|
||||
for (const detach of [true, false]) {
|
||||
await assert.rejects(consoleStart(ctx, { devLocalOwner: true, detach, port: 'auto', warn: () => {} }), e => e.exitCode === 3 && /found MOSAIC_RUN_ID/.test(e.message));
|
||||
}
|
||||
} finally { delete process.env.MOSAIC_RUN_ID; }
|
||||
const agentCli = args => spawnSync(join(REPO, 'scripts/mosaic'), args, { env: { ...ctx.env, MOSAIC_AGENT_RUN: '1' }, encoding: 'utf8', timeout: 30000 });
|
||||
const refused = agentCli(['console', 'start', '--dev-local-owner', '--detach', '--port', 'auto']);
|
||||
assert.equal(refused.status, 3); assert.equal(refused.stdout, ''); assert.match(refused.stderr, /only from a human shell/);
|
||||
const consoleDir = join(ctx.dataRoot, 'console');
|
||||
assert.deepEqual(existsSync(consoleDir) ? readdirSync(consoleDir) : [], []);
|
||||
assert.deepEqual(consoleStatus(ctx), { running: false });
|
||||
const plain = agentCli(['console', 'start', '--detach', '--port', 'auto']);
|
||||
assert.equal(plain.status, 0, plain.stderr);
|
||||
assert.deepEqual(Object.keys(JSON.parse(plain.stdout)), ['url', 'pid']);
|
||||
assert.equal((await consoleStop(ctx)).stopped, true);
|
||||
});
|
||||
|
||||
// D1 security re-check P2: a supervisor that hangs holding the start lock is killed with its group
|
||||
// after the 60 s start timeout and 10 s grace. Its lock names it, so the start clears it and the
|
||||
// next start proceeds. The hook blocks the supervisor's event loop once it holds the lock (at its
|
||||
// console state read), so its stop handler never runs and only the group SIGKILL ends it.
|
||||
test('a detached start that times out clears the start lock of the supervisor it killed', { timeout: 150000 }, async t => {
|
||||
const { ctx } = await instance(t);
|
||||
const hook = join(dirname(ctx.dataRoot), 'hang-after-lock.mjs');
|
||||
writeFileSync(hook, `import fs from 'node:fs'; import { syncBuiltinESMExports } from 'node:module';
|
||||
if (process.argv[1]?.endsWith('console-supervisor.mjs')) { const real = fs.openSync;
|
||||
fs.openSync = (...args) => { if (String(args[0]).endsWith('/console/console.json')) Atomics.wait(new Int32Array(new SharedArrayBuffer(4)), 0, 0); return real(...args); };
|
||||
syncBuiltinESMExports(); }\n`);
|
||||
const hung = { ...ctx, env: { ...ctx.env, NODE_OPTIONS: `--import=${pathToFileURL(hook).href}` } };
|
||||
const lock = join(ctx.dataRoot, 'console/start.lock');
|
||||
const began = Date.now();
|
||||
await assert.rejects(consoleStart(hung, { devLocalOwner: true, detach: true, port: 'auto', warn: () => {} }),
|
||||
e => e.exitCode === 3 && /^console failed to start; see console\/supervisor-\d+-\d+\.log under the data root$/.test(e.message));
|
||||
assert.ok(Date.now() - began >= 69000, 'the start waited out its timeout and grace');
|
||||
assert.equal(existsSync(lock), false);
|
||||
assert.deepEqual(consoleStatus(ctx), { running: false });
|
||||
// The next start is not blocked by the killed supervisor's lock.
|
||||
const result = await consoleStart(ctx, { devLocalOwner: true, detach: true, port: 'auto', warn: () => {} });
|
||||
assert.equal((await consoleStop(ctx)).stopped, true);
|
||||
assert.equal(await exitOf(result.done), 0);
|
||||
});
|
||||
|
||||
// Without --dev-local-owner a detached start is what it was before D1: no supervisor, restart refused.
|
||||
test('a detached console without the dev owner has no supervisor, so it refuses a restart and stays up', { timeout: 60000 }, async t => {
|
||||
const { ctx, mosaic } = await instance(t);
|
||||
const warnings = [];
|
||||
const result = await consoleStart(ctx, { detach: true, port: 'auto', warn: s => warnings.push(s) });
|
||||
assert.deepEqual(warnings, []);
|
||||
assert.deepEqual(Object.keys(result), ['url', 'pid']);
|
||||
const port = Number(new URL(result.url).port);
|
||||
assert.ok(!argvOf(result.pid).includes('--supervised'));
|
||||
assert.deepEqual(argvOf(result.pid), [process.execPath, WEBUI, 'serve', '--port', '0', '--business', 'alpha']);
|
||||
assert.equal(readdirSync(join(ctx.dataRoot, 'console')).filter(x => x.startsWith('supervisor-')).length, 0);
|
||||
assert.equal((await call(port, '/api/config')).body.consoleRestart, false);
|
||||
const r = await call(port, '/api/console/restart', { method: 'POST', body: '{}' });
|
||||
assert.equal(r.status, 409); assert.equal(r.body.error, 'not-supervised');
|
||||
// Without a dev owner there is no owner session to restart as: refused before the supervisor check.
|
||||
assert.equal(r.status, 403, JSON.stringify(r.body)); assert.deepEqual(r.body, { error: 'owner-only' });
|
||||
assert.equal((await call(port, '/api/config')).status, 200);
|
||||
assert.equal(existsSync(join(ctx.dataRoot, 'console/actions.jsonl')), false);
|
||||
assert.equal((await consoleStop(ctx)).stopped, true);
|
||||
// The CLI's receipt keeps its two fields.
|
||||
const start = mosaic(['console', 'start', '--detach', '--port', 'auto']);
|
||||
assert.equal(start.status, 0, start.stderr);
|
||||
assert.deepEqual(Object.keys(JSON.parse(start.stdout)), ['url', 'pid']);
|
||||
assert.equal(start.stderr, '');
|
||||
assert.equal((await consoleStop(ctx)).stopped, true);
|
||||
});
|
||||
@@ -21,8 +21,9 @@ async function fixture() {
|
||||
const env = humanEnv({ HOME: root, MOSAIC_CONFIG: join(configDir, 'config.json') });
|
||||
const moduleUrl = new URL('../src/cli.mjs', import.meta.url).href;
|
||||
const code = `import { run } from ${JSON.stringify(moduleUrl)}; process.exitCode = await run(JSON.parse(process.argv[1]), undefined, {system: {dataRoot: process.argv[2]}});`;
|
||||
const call = async (...args) => { const x = await exec(process.execPath, ['--input-type=module', '-e', code, JSON.stringify(args), dataRoot], { env }); return { out: { text: x.stdout }, err: { text: x.stderr } }; };
|
||||
return { root, ctx, org, call };
|
||||
const callWith = async (extra, ...args) => { const x = await exec(process.execPath, ['--input-type=module', '-e', code, JSON.stringify(args), dataRoot], { env: { ...env, ...extra } }); return { out: { text: x.stdout }, err: { text: x.stderr } }; };
|
||||
const call = (...args) => callWith({}, ...args);
|
||||
return { root, ctx, org, call, callWith };
|
||||
}
|
||||
const dead = 2147483647;
|
||||
const json = (path, value) => writeFileSync(path, JSON.stringify(value), { mode: 0o600 });
|
||||
@@ -121,8 +122,9 @@ test('org lock show and clear require a provably dead holder', async () => {
|
||||
|
||||
test('bus recover preserves evidence for dead and wrong-process hosts without signals', async () => {
|
||||
const f = await fixture(), dir = join(f.org.root, 'bus-host'); mkdirSync(dir, { mode: 0o700 });
|
||||
// A wrong process is a live pid whose start time is not the recorded one (the pid was reused).
|
||||
for (const pid of [dead, process.pid]) {
|
||||
const state = { pid, startTime: startTimeOf(pid) ?? '1', business: 'alpha', startedAt: '2026-01-01T00:00:00Z' };
|
||||
const state = { pid, startTime: '1', business: 'alpha', startedAt: '2026-01-01T00:00:00Z' };
|
||||
json(join(dir, 'host.json'), state); json(join(dir, 'host.lock'), state);
|
||||
const x = await f.call('bus', 'recover', 'alpha');
|
||||
const result = JSON.parse(x.out.text);
|
||||
@@ -138,6 +140,26 @@ test('bus recover preserves evidence for dead and wrong-process hosts without si
|
||||
assert.equal(process.pid > 0, true);
|
||||
});
|
||||
|
||||
test('D2: bus recover refuses a live pid with the recorded start time, whatever its argv', async () => {
|
||||
const f = await fixture(), dir = join(f.org.root, 'bus-host'); mkdirSync(dir, { mode: 0o700 });
|
||||
const state = { pid: process.pid, startTime: startTimeOf(process.pid), business: 'alpha', startedAt: '2026-01-01T00:00:00Z' };
|
||||
assert.match(state.startTime, /^\d+$/);
|
||||
json(join(dir, 'host.json'), state); json(join(dir, 'host.lock'), state);
|
||||
await assert.rejects(f.call('bus', 'recover', 'alpha'), /bus host identity is uncertain/);
|
||||
assert.deepEqual(JSON.parse(readFileSync(join(dir, 'host.json'), 'utf8')), state);
|
||||
assert.equal(existsSync(join(dir, 'host.lock')), true);
|
||||
assert.ok(!readdirSync(dir).some(x => x.startsWith('recovery-')));
|
||||
});
|
||||
|
||||
test('D2: bus recover refuses inside an agent run before it reads anything', async () => {
|
||||
const f = await fixture(), dir = join(f.org.root, 'bus-host'); mkdirSync(dir, { mode: 0o700 });
|
||||
const state = { pid: dead, startTime: '1', business: 'alpha', startedAt: '2026-01-01T00:00:00Z' };
|
||||
json(join(dir, 'host.json'), state); json(join(dir, 'host.lock'), state);
|
||||
await assert.rejects(f.callWith({ MOSAIC_AGENT_RUN: '1' }, 'bus', 'recover', 'alpha'), e => e.code === 3 && /outside any agent run/.test(e.stderr));
|
||||
assert.equal(existsSync(join(dir, 'host.json')), true);
|
||||
assert.ok(!readdirSync(dir).some(x => x.startsWith('recovery-')));
|
||||
});
|
||||
|
||||
test('lock recovery refuses malformed or unreadable holders', async () => {
|
||||
const f = await fixture(), path = join(f.ctx.configDir, 'org-lifecycle-alpha.lock');
|
||||
for (const value of [{ pid: 0, at: '2026-01-01T00:00:00Z' }, { pid: dead }, null]) {
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
// D2 (P0-3): a Console unit stop (KillMode=mixed) SIGKILLs the bus hosts the Console started. These
|
||||
// tests start real bus hosts, kill them the same way, and check that busRevive and the Console-start
|
||||
// revive bring them back, and that a live host or a live writer lock is never taken over.
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawn, spawnSync } from 'node:child_process';
|
||||
import { existsSync, lstatSync, mkdtempSync, readFileSync, readdirSync, renameSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { reserveOrg, activateOrg, listOrgs } from '../../organization/src/org-registry.mjs';
|
||||
import { businessDoc, rolesCopy, writeJson } from '../../organization/tests/helpers.mjs';
|
||||
import { humanEnv, notifyConfig } from './helpers.mjs';
|
||||
import { busRevive, busStart, busStatus, busStop, setAside } from '../src/org-bus.mjs';
|
||||
import { reviveBuses } from '../../webui/src/bus-revive.mjs';
|
||||
|
||||
const REPO = new URL('../../../', import.meta.url).pathname;
|
||||
const IDS = ['alpha', 'beta'];
|
||||
|
||||
async function instance(t) {
|
||||
const home = mkdtempSync(join(tmpdir(), 'rv-'));
|
||||
const env = humanEnv({ HOME: home, MOSAIC_CONFIG: join(home, '.config/mosaic-rvtest/config.json') });
|
||||
const setup = spawnSync(join(REPO, 'scripts/bootstrap.sh'), ['--instance', 'rvtest'], { env, encoding: 'utf8' });
|
||||
assert.equal(setup.status, 0, setup.stderr);
|
||||
const ctx = { configDir: join(home, '.config/mosaic-rvtest'), dataRoot: join(home, '.mosaic-rvtest'), instance: 'rvtest', env };
|
||||
env.MOSAIC_ROLES_DIR = rolesCopy(home);
|
||||
const roots = {};
|
||||
for (const id of IDS) {
|
||||
const org = await reserveOrg(ctx, id);
|
||||
await activateOrg(ctx, id);
|
||||
writeJson(join(ctx.configDir, 'businesses', `${id}.json`), businessDoc(home, id));
|
||||
notifyConfig(org.root, id, null);
|
||||
roots[id] = org.root;
|
||||
}
|
||||
const extra = [];
|
||||
t.after(async () => {
|
||||
for (const child of extra) { try { child.kill('SIGKILL'); } catch {} }
|
||||
for (const id of IDS) { try { await busStop(ctx, id); } catch {} }
|
||||
rmSync(home, { recursive: true, force: true });
|
||||
});
|
||||
return { ctx, roots, extra };
|
||||
}
|
||||
// What a Console unit stop does to a host it started: SIGKILL to the host and every process under it.
|
||||
// The host is a session leader (detached), so its process group is the test's own; `done` resolves
|
||||
// once the host is reaped, and the loop waits for the rest of the group to go.
|
||||
async function kill(receipt) {
|
||||
process.kill(-receipt.pid, 'SIGKILL');
|
||||
await receipt.done;
|
||||
for (let i = 0; i < 100; i++) {
|
||||
try { process.kill(-receipt.pid, 0); } catch (e) { if (e.code === 'ESRCH') return; throw e; }
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
}
|
||||
throw new Error(`process group ${receipt.pid} did not go`);
|
||||
}
|
||||
const host = root => JSON.parse(readFileSync(join(root, 'bus-host/host.json'), 'utf8'));
|
||||
const recoveries = root => readdirSync(join(root, 'bus-host')).filter(name => /^recovery-.*\.json$/.test(name));
|
||||
|
||||
test('D2: a SIGKILLed bus host comes back from one revive, on the options it was started with', { timeout: 120000 }, async t => {
|
||||
const { ctx, roots } = await instance(t);
|
||||
const first = await busStart(ctx, 'alpha', { detach: true, devLocalOwner: true, noLaunch: true });
|
||||
await kill(first);
|
||||
// What a Console unit stop leaves behind: the dead host's state, lock and socket.
|
||||
assert.equal(existsSync(join(roots.alpha, 'bus-host/host.json')), true);
|
||||
await assert.rejects(busStatus(ctx, 'alpha'), e => e.name === 'CliError' && /stale/.test(e.message));
|
||||
const revived = await busRevive(ctx, 'alpha');
|
||||
assert.notEqual(revived.pid, first.pid);
|
||||
assert.equal(revived.recovered.pid, first.pid);
|
||||
assert.equal((await busStatus(ctx, 'alpha')).healthy, true);
|
||||
assert.deepEqual(host(roots.alpha).options, { devLocalOwner: true, pm: false, noLaunch: true });
|
||||
assert.equal(recoveries(roots.alpha).length, 1);
|
||||
const receipt = JSON.parse(readFileSync(revived.recovered.receipt, 'utf8'));
|
||||
assert.equal(receipt.host.pid, first.pid);
|
||||
});
|
||||
|
||||
test('D2: a live bus host is never taken over, and its state is left as it was', { timeout: 120000 }, async t => {
|
||||
const { ctx, roots } = await instance(t);
|
||||
const live = await busStart(ctx, 'alpha', { detach: true, noLaunch: true });
|
||||
const before = readFileSync(join(roots.alpha, 'bus-host/host.json'), 'utf8');
|
||||
await assert.rejects(busRevive(ctx, 'alpha'), e => e.name === 'CliError' && e.message === `bus host pid ${live.pid} is alive`);
|
||||
assert.equal(readFileSync(join(roots.alpha, 'bus-host/host.json'), 'utf8'), before);
|
||||
assert.equal(recoveries(roots.alpha).length, 0);
|
||||
const status = await busStatus(ctx, 'alpha');
|
||||
assert.equal(status.healthy, true);
|
||||
assert.equal(status.pid, live.pid);
|
||||
});
|
||||
|
||||
test('D2: a writer lock held by a live process refuses the revive, even with the host dead', { timeout: 120000 }, async t => {
|
||||
const { ctx, roots, extra } = await instance(t);
|
||||
const first = await busStart(ctx, 'alpha', { detach: true, noLaunch: true });
|
||||
await kill(first);
|
||||
// A live process the test started holds the writer lock.
|
||||
const holder = spawn('sleep', ['60'], { stdio: 'ignore' }); extra.push(holder);
|
||||
writeFileSync(join(roots.alpha, 'bus/writer.lock'), JSON.stringify({ pid: holder.pid, at: new Date().toISOString() }), { mode: 0o600 });
|
||||
const lockBefore = readFileSync(join(roots.alpha, 'bus/writer.lock'), 'utf8');
|
||||
await assert.rejects(busRevive(ctx, 'alpha'), e => e.name === 'CliError' && e.message === `bus writer pid ${holder.pid} is alive`);
|
||||
assert.equal(readFileSync(join(roots.alpha, 'bus/writer.lock'), 'utf8'), lockBefore);
|
||||
assert.equal(existsSync(join(roots.alpha, 'bus-host/host.json')), true);
|
||||
assert.equal(recoveries(roots.alpha).length, 0);
|
||||
// The Console-start revive leaves it too, and logs the refusal.
|
||||
const lines = [], trail = [];
|
||||
const results = await reviveBuses({ ctx, listOrgs, busStatus, busRevive, env: {}, trail: e => trail.push(e), log: line => lines.push(line) });
|
||||
assert.deepEqual(results.find(r => r.id === 'alpha'), { id: 'alpha', outcome: 'refused' });
|
||||
assert.ok(lines.some(line => line.includes(`alpha: refused (bus writer pid ${holder.pid} is alive)`)), lines.join('\n'));
|
||||
assert.deepEqual(trail.filter(e => e.org === 'alpha').map(e => e.outcome), ['requested', 'refused']);
|
||||
assert.equal(existsSync(join(roots.alpha, 'bus-host/host.json')), true);
|
||||
});
|
||||
|
||||
test('D2: Console start brings both killed buses back and reattaches to one that still runs', { timeout: 180000 }, async t => {
|
||||
const { ctx, roots } = await instance(t);
|
||||
const a = await busStart(ctx, 'alpha', { detach: true, devLocalOwner: true, noLaunch: true });
|
||||
const b = await busStart(ctx, 'beta', { detach: true, devLocalOwner: true, noLaunch: true });
|
||||
await kill(a); await kill(b);
|
||||
const trail = [], lines = [];
|
||||
const results = await reviveBuses({ ctx, listOrgs, busStatus, busRevive, env: {}, trail: e => trail.push(e), log: line => lines.push(line) });
|
||||
assert.deepEqual(results.map(r => [r.id, r.outcome]), [['alpha', 'revived'], ['beta', 'revived']]);
|
||||
for (const id of IDS) {
|
||||
const status = await busStatus(ctx, id);
|
||||
assert.equal(status.healthy, true, id);
|
||||
assert.equal(host(roots[id]).options.devLocalOwner, true);
|
||||
assert.deepEqual(trail.filter(e => e.org === id).map(e => [e.action, e.outcome, e.by.sub]), [['bus.revive', 'requested', 'local-dev-owner'], ['bus.revive', 'done', 'local-dev-owner']]);
|
||||
}
|
||||
// A second Console start finds them running and leaves them.
|
||||
const pids = Object.fromEntries(await Promise.all(IDS.map(async id => [id, (await busStatus(ctx, id)).pid])));
|
||||
const again = await reviveBuses({ ctx, listOrgs, busStatus, busRevive, env: {}, trail: e => trail.push(e), log: () => {} });
|
||||
assert.deepEqual(again.map(r => [r.id, r.outcome]), [['alpha', 'running'], ['beta', 'running']]);
|
||||
for (const id of IDS) assert.equal((await busStatus(ctx, id)).pid, pids[id]);
|
||||
});
|
||||
|
||||
test('D2: a start that fails after the dead state was cleared says so (error.recovered)', { timeout: 120000 }, async t => {
|
||||
const { ctx, roots } = await instance(t);
|
||||
const first = await busStart(ctx, 'alpha', { detach: true, noLaunch: true });
|
||||
await kill(first);
|
||||
// A record busStart refuses (--pm with --no-launch), rewritten in place: the clear goes through and
|
||||
// the start that follows fails, the way a host that cannot come up would.
|
||||
const file = join(roots.alpha, 'bus-host/host.json');
|
||||
writeFileSync(file, JSON.stringify({ ...host(roots.alpha), options: { devLocalOwner: false, pm: true, noLaunch: true } }));
|
||||
await assert.rejects(busRevive(ctx, 'alpha'), e => e.name === 'CliError' && /--pm or --no-launch/.test(e.message) && e.recovered?.pid === first.pid);
|
||||
assert.equal(existsSync(file), false);
|
||||
assert.equal(recoveries(roots.alpha).length, 1);
|
||||
});
|
||||
|
||||
test('D2: an entry swapped after inspection is put back and nothing is removed', t => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'aside-'));
|
||||
t.after(() => rmSync(dir, { recursive: true, force: true }));
|
||||
const a = join(dir, 'host.json'), b = join(dir, 'writer.lock');
|
||||
writeFileSync(a, 'dead host', { mode: 0o600 }); writeFileSync(b, 'dead writer', { mode: 0o600 });
|
||||
const entries = [{ path: a, st: lstatSync(a) }, { path: b, st: lstatSync(b) }];
|
||||
// The swap: the inspected writer.lock is moved off (its inode stays in use) and a new one takes the name.
|
||||
renameSync(b, join(dir, 'kept')); writeFileSync(b, 'live writer', { mode: 0o600 });
|
||||
assert.throws(() => setAside(entries), e => e.name === 'CliError' && e.message === 'bus recovery state changed');
|
||||
assert.equal(readFileSync(a, 'utf8'), 'dead host');
|
||||
assert.equal(readFileSync(b, 'utf8'), 'live writer');
|
||||
assert.deepEqual(readdirSync(dir).sort(), ['host.json', 'kept', 'writer.lock']);
|
||||
// Unswapped, the entries go aside and remove() deletes only those.
|
||||
renameSync(join(dir, 'kept'), b);
|
||||
const fresh = [{ path: a, st: lstatSync(a) }, { path: b, st: lstatSync(b) }];
|
||||
const aside = setAside(fresh);
|
||||
assert.equal(existsSync(a) || existsSync(b), false);
|
||||
writeFileSync(a, 'a new host', { mode: 0o600 });
|
||||
aside.remove();
|
||||
assert.deepEqual(readdirSync(dir), ['host.json']);
|
||||
assert.equal(readFileSync(a, 'utf8'), 'a new host');
|
||||
});
|
||||
@@ -21,7 +21,8 @@ baseline CLI.
|
||||
| Inspect or clear a dead lifecycle lock | `mosaic org lock show <id>` / `mosaic org lock clear <id>` | CLI only |
|
||||
| Start, stop or restart a bus | `mosaic bus start <id>` / `mosaic bus stop <id>` / `mosaic bus restart <id>` | S6 integration hook |
|
||||
| Restart (or first start) an organization's bus | `mosaic bus restart <id>`, or `mosaic bus start <id>` for a bus never started | Enabled (F3): the owner's **Restart bus** control, offered where a page says the bus is down or a change needs a restart; trailed |
|
||||
| Restart the Console | Stop and run `mosaic console start` again | Enabled (F3) when `mosaic console start` runs it in the foreground: Settings > Console; the owner only; trailed. Other Consoles answer `not-supervised` |
|
||||
| Restart the Console | Stop and run `mosaic console start` again | Enabled (F3) when `mosaic console start` runs it in the foreground, or with `--dev-local-owner --detach` under a detached supervisor (D1): Settings > Console; the owner only; trailed. Other Consoles answer `not-supervised` |
|
||||
| Save, review, confirm or reverse an organization's autonomy | Save only: edit the autonomy block and run `mosaic org update ORG --from FILE --if-revision N`; no autonomy verb yet (readiness P1-3) | Enabled (D1) on a `--dev-local-owner` Console: the owner only, on the selected organization's bus, behind a fence that passes only the four console verbs `autonomy.set.console`, `autonomy.review.console`, `decision.confirm.console` and `decision.reverse.console` |
|
||||
| Create chat project linked to a configured repository | `mosaic project create <name> --repo <project> --org <id>` | Enabled |
|
||||
| Create a thread | `mosaic thread create <project> <name> --org <id>` | Planned enabled |
|
||||
| Open a thread | `mosaic thread open <thread> --org <id>` | Planned enabled |
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
// AU: owner Console transport only. Identity and organization never come from a JSON body.
|
||||
export const AUTONOMY_ROUTE=/^\/api\/autonomy(?:\/(confirm|reverse|review))?$/;
|
||||
// D1 (#1873): the owner writes this route makes, and a fence that passes only those to a console transport.
|
||||
export const AUTONOMY_WRITE_VERBS=Object.freeze(['autonomy.set.console','autonomy.review.console','decision.confirm.console','decision.reverse.console']);
|
||||
export const autonomyCaller=call=>async(verb,args,options)=>{if(!AUTONOMY_WRITE_VERBS.includes(verb))throw new Error(`the autonomy writer does not call ${verb}`);return call(verb,args,options);};
|
||||
const one=(req,k)=>req.headersDistinct[k]?.length===1?req.headersDistinct[k][0]:null;
|
||||
const json=(res,status,value)=>{res.writeHead(status,{'content-type':'application/json','cache-control':'no-store'});res.end(JSON.stringify(value));};
|
||||
export async function autonomyHTTP(req,res,{path,read,write,session,live,origin}) {
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
// D2 (P0-3): when a dev-local-owner Console starts, it reattaches to the organization buses that
|
||||
// still run and brings back the ones a Console unit stop killed. The unit's KillMode=mixed sends
|
||||
// SIGKILL to the bus hosts the Console started; a host killed that way leaves its host.json behind,
|
||||
// which a host stopped on purpose (mosaic bus stop, Stop in the page) removes as it closes.
|
||||
//
|
||||
// For each active organization:
|
||||
// - a bus whose state checks out (running, or stopped cleanly) is left as it is;
|
||||
// - a bus whose state does not check out goes to busRevive, whose busRecover clears the state only
|
||||
// when the host is proven dead (no live pid with that start time, the lock is the dead host's, no
|
||||
// live writer, no socket that answers). Anything else refuses and the state stays for a person.
|
||||
// The chief is not relaunched (pm: false): no agent starts without a person's press.
|
||||
// Every attempt is a trail line before anything changes; a trail that cannot be written skips it.
|
||||
// A Console whose own environment carries an agent run's markers revives nothing (the human-only
|
||||
// rule `mosaic bus recover` and Restart bus's `mosaic org reload` follow).
|
||||
// While an organization's revive runs, revivePending(id) holds it, so a Restart bus press waits for
|
||||
// it instead of racing it for the lifecycle lock.
|
||||
import { DEV_OWNER } from '../../cli/src/dev-local-owner.mjs';
|
||||
import { AGENT_MARKERS } from '../../cli/src/transport.mjs';
|
||||
|
||||
const isCliError = e => e?.name === 'CliError' && Number.isInteger(e.exitCode);
|
||||
const BY = Object.freeze({ ...DEV_OWNER, name: 'Console start' });
|
||||
const pending = new Map();
|
||||
|
||||
export const revivePending = id => pending.get(id) ?? null;
|
||||
|
||||
export async function reviveBuses({ ctx, listOrgs, busStatus, busRevive, trail, log = () => {}, env = process.env }) {
|
||||
const results = [];
|
||||
if (AGENT_MARKERS.some(k => env?.[k])) { log('bus revive: this Console runs inside an agent run; no bus was checked'); return results; }
|
||||
let orgs;
|
||||
try { orgs = await listOrgs(ctx); }
|
||||
catch { log('bus revive: the organization list could not be read; no bus was checked'); return results; }
|
||||
const active = (orgs ?? []).filter(org => org?.state === 'active').map(org => org.id);
|
||||
const settle = new Map();
|
||||
for (const id of active) {
|
||||
let done;
|
||||
const wait = new Promise(resolve => { done = resolve; });
|
||||
pending.set(id, wait);
|
||||
settle.set(id, () => { if (pending.get(id) === wait) pending.delete(id); done(); });
|
||||
}
|
||||
try {
|
||||
for (const id of active) {
|
||||
try { results.push(await reviveOne({ ctx, id, busStatus, busRevive, trail, log })); }
|
||||
finally { settle.get(id)(); }
|
||||
}
|
||||
} finally { for (const release of settle.values()) release(); }
|
||||
return results;
|
||||
}
|
||||
|
||||
async function reviveOne({ ctx, id, busStatus, busRevive, trail, log }) {
|
||||
let status = null;
|
||||
try { status = await busStatus(ctx, id); }
|
||||
catch (e) {
|
||||
if (!isCliError(e)) { log(`bus revive: ${id}: bus state could not be read; left as it is`); return { id, outcome: 'uncertain' }; }
|
||||
}
|
||||
if (status) {
|
||||
const outcome = status.healthy ? 'running' : Number.isSafeInteger(status.pid) ? 'not-answering' : 'stopped';
|
||||
if (outcome === 'running') log(`bus revive: ${id}: bus answers; reattached`);
|
||||
return { id, outcome };
|
||||
}
|
||||
try { trail({ by: BY, action: 'bus.revive', org: id, outcome: 'requested' }); }
|
||||
catch { log(`bus revive: ${id}: the action trail could not be written; nothing was recovered`); return { id, outcome: 'trail-unavailable' }; }
|
||||
const note = outcome => { try { trail({ by: BY, action: 'bus.revive', org: id, outcome, ...(outcome === 'failed' ? { recovered: true } : {}) }); } catch { log(`bus revive: ${id}: trail write failed after ${outcome}`); } };
|
||||
try {
|
||||
const receipt = await busRevive(ctx, id, { pm: false });
|
||||
note('done');
|
||||
log(`bus revive: ${id}: dead bus host cleared; bus started (pid ${receipt.pid})`);
|
||||
if (receipt.pmDropped) log(`bus revive: ${id}: the dead host was started with --pm; the chief was not relaunched. mosaic bus stop ${id}; mosaic bus start ${id} --pm brings it back`);
|
||||
return { id, outcome: 'revived', pid: receipt.pid };
|
||||
} catch (e) {
|
||||
// The reason names the organization id and pids only, never a host path (busRecover's and
|
||||
// busStart's messages); an error that is not a CliError shows its code, not its text.
|
||||
const reason = isCliError(e) ? e.message : `unexpected error${typeof e?.code === 'string' ? ` ${e.code}` : ''}`;
|
||||
if (e?.recovered) {
|
||||
note('failed');
|
||||
log(`bus revive: ${id}: dead bus host cleared, but the bus did not start (${reason}); see mosaic bus status ${id}`);
|
||||
return { id, outcome: 'failed' };
|
||||
}
|
||||
note('refused');
|
||||
log(`bus revive: ${id}: refused (${reason}); left for mosaic bus status ${id}`);
|
||||
return { id, outcome: 'refused' };
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import { busCaller, busBusiness, userCaller, consoleCall } from './bus.mjs';
|
||||
import { organizationConsole } from './organizations.mjs';
|
||||
import { consoleBus, userOptionsFor, agentCaller, TESTER_AGENT_VERBS } from './console-bus.mjs';
|
||||
import { SYSTEM_TYPE_VERBS } from './system-types-http.mjs';
|
||||
import { autonomyCaller } from './autonomy-http.mjs';
|
||||
import { consoleAccess } from './access.mjs';
|
||||
import { accessContextFor } from '../../cli/src/access.mjs';
|
||||
import { accessFilePath, loadAccess } from '../../organization/src/access.mjs';
|
||||
@@ -14,7 +15,7 @@ import { startTesterSocket } from './tester-socket.mjs';
|
||||
import { TESTER_CLI } from '../../cli/src/transport.mjs';
|
||||
import { views } from '../../bus/src/views.mjs';
|
||||
import { loadSystem, socketPath, orgPaths, REPO, rolesDir } from '../../cli/src/config.mjs';
|
||||
import { busStatus as orgBusStatus } from '../../cli/src/org-bus.mjs';
|
||||
import { busStatus as orgBusStatus, busRevive as orgBusRevive } from '../../cli/src/org-bus.mjs';
|
||||
import { listOrgs, resolveOrg } from '../../organization/src/org-registry.mjs';
|
||||
import { showOrg } from '../../organization/src/org-service.mjs';
|
||||
import { configDir } from '../../organization/src/organization-config.mjs';
|
||||
@@ -28,6 +29,7 @@ import { readFileSync, statSync } from 'node:fs';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { isAbsolute, join } from 'node:path';
|
||||
import { actionTrail } from './action-trail.mjs';
|
||||
import { reviveBuses } from './bus-revive.mjs';
|
||||
import { agentLevelsWriteRoutes } from './agent-levels-write.mjs';
|
||||
import { RESTART_EXIT } from '../../cli/src/org-console.mjs';
|
||||
import { ID_PATTERN } from '../../organization/src/vocabulary.mjs';
|
||||
@@ -182,6 +184,10 @@ try {
|
||||
// Phase P2 (#1873): the owner's System types writes, the organization's enable switch (plugins.set
|
||||
// with its revision) and a project's selection, over the owner's human transport under --send only.
|
||||
systemTypesWrite: session.user === undefined && sendOn ? agentCaller(owner, SYSTEM_TYPE_VERBS) : null,
|
||||
// D1 (#1873): the owner's autonomy writes (save, review, confirm, reverse) on this organization's bus.
|
||||
// These options exist only on a --dev-local-owner Console (`organizations` above). The fence passes
|
||||
// those four verbs and nothing else the console transport carries.
|
||||
autonomyWrite: session.user === undefined && sendOn ? autonomyCaller(consoleCall({ dataRoot: root, socket: paths.socket, business: id })) : null,
|
||||
reads: consoleReads({ root: REPO, queueRoot: options.checkout ?? REPO, system, configDir: configDir(), rolesDir: rolesDir(), business: id, orgRoot: root }),
|
||||
};
|
||||
},
|
||||
@@ -235,6 +241,9 @@ try {
|
||||
console.log(`Listening: ${lan.bind}:${port}, allowed Host names ${lan.allowHosts.map(name => `${name}:${port}`).join(', ')}`);
|
||||
}
|
||||
console.log(`Mosaic Console: http://${urlHost}:${port}/\nBoard: ${options.board}\n${busLine}${queueLine}${previewLine}${ssoLines}${proofLine}${testerServer ? `\nTester Console: unix:${testerServer.address()} (binds the tester)` : ''}\nCtrl-C stops this server.`);
|
||||
// D2 (P0-3): the dev Console reattaches to the buses that still run and brings back the ones its
|
||||
// unit stop killed. It runs after the Console answers, so a slow bus never delays the page.
|
||||
if (organizations && trail) reviveBuses({ ctx: { configDir: configDir(), dataRoot: system.dataRoot, instance }, listOrgs, busStatus: orgBusStatus, busRevive: orgBusRevive, trail, log: line => console.log(line) }).catch(() => console.log('bus revive: stopped on an unexpected error'));
|
||||
for (const signal of ['SIGINT', 'SIGTERM']) process.once(signal, () => { sso?.close(); server.close(); testerServer?.close(); });
|
||||
} catch (err) {
|
||||
console.error(`refused: ${err.message}`);
|
||||
|
||||
@@ -16,7 +16,8 @@
|
||||
// POST /api/orgs/ID/setup/queue C2 queueInit mosaic org queue init
|
||||
// POST /api/orgs/ID/setup/project/P C2 projectRepoInit mosaic org project repo init
|
||||
// POST /api/orgs/ID/setup/bus C3 busStart mosaic bus start --detach
|
||||
// POST /api/orgs/ID/restart/bus C3 busRestart mosaic bus restart (F3; busStart when stopped)
|
||||
// POST /api/orgs/ID/restart/bus C3 busRestart mosaic bus restart (F3; busStart when stopped;
|
||||
// busRevive for a host that died, dev-local owner only, D2)
|
||||
//
|
||||
// GET /api/orgs (the list) and other /api/orgs/ID/... routes are S5's; this
|
||||
// handler returns false for them.
|
||||
@@ -27,6 +28,8 @@ import { checkProject, initProject, projectRepo, recordContext } from '../../pro
|
||||
import { refuseInsideAgent } from '../../cli/src/transport.mjs';
|
||||
import { join } from 'node:path';
|
||||
import { actor } from './action-trail.mjs';
|
||||
import { DEV_OWNER } from '../../cli/src/dev-local-owner.mjs';
|
||||
import { revivePending } from './bus-revive.mjs';
|
||||
|
||||
const ID = /^[a-z0-9][a-z0-9._-]{0,63}$/;
|
||||
const KEY = /^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/;
|
||||
@@ -51,6 +54,7 @@ export function verbFor(step, id, opts = {}) {
|
||||
if (step === 'records') return `mosaic project init ${id} ${opts.project}`;
|
||||
if (step === 'bus') return `mosaic bus start ${id} --dev-local-owner --detach`;
|
||||
if (step === 'bus-restart') return `mosaic bus restart ${id}`;
|
||||
if (step === 'bus-revive') return `mosaic bus recover ${id}; mosaic bus start ${id} --dev-local-owner --detach`;
|
||||
throw new Error(`unknown step ${step}`);
|
||||
}
|
||||
|
||||
@@ -563,7 +567,8 @@ async function route(req, res, { path, search, org, admin, principal, write, log
|
||||
// F3 (REQ-WEB-3): one control restarts an organization's bus and answers with a done receipt. A
|
||||
// running host restarts on the options it was started with (C3 busRestart keeps --dev-local-owner);
|
||||
// a stopped one starts as setup starts it. The trail line is written before anything changes, and
|
||||
// a trail that cannot be written refuses the restart. Stale host state is never cleared here.
|
||||
// a trail that cannot be written refuses the restart. Stale host state is cleared only for the
|
||||
// dev-local owner, and only through busRevive (D2), whose busRecover refuses a host it cannot prove dead.
|
||||
async function restartBus(req, res, { sub, write, id, ctx, service, admin, log, trail, by }) {
|
||||
const step = 'bus-restart';
|
||||
if (sub !== 'bus' || !write) return json(res, 404, { error: 'not found' });
|
||||
@@ -571,11 +576,14 @@ async function restartBus(req, res, { sub, write, id, ctx, service, admin, log,
|
||||
try { await service.showOrg(ctx, id); } catch (e) { return refusal(res, e, { step, id }); }
|
||||
const bus = await need(admin, 'bus', res, step); if (!bus) return;
|
||||
if (!trail) return json(res, 503, { error: 'trail-unavailable', step, message: 'this Console has no action trail, so it does not restart buses' });
|
||||
// D2: the dev Console's start-up revive may be bringing this bus back; the press acts on the result.
|
||||
await revivePending(id);
|
||||
let status;
|
||||
try { status = await bus.busStatus(ctx, id); }
|
||||
catch (e) {
|
||||
if (['org-unknown', 'org-mismatch', 'org-unavailable', 'org-archived'].includes(e?.code)) return refusal(res, e, { step, id });
|
||||
if (isCliError(e)) {
|
||||
if (isDevOwner(by) && typeof bus.busRevive === 'function') return reviveBus(res, { id, ctx, bus, log, trail, by, step });
|
||||
// Refused before anything changed, but the owner asked: the trail records the attempt.
|
||||
try { trail({ by, action: 'bus.restart', org: id, outcome: 'refused', running: null }); } catch { log(`org-admin: ${id} bus restart: trail write failed after refused`); }
|
||||
return json(res, 409, { error: 'bus-refused', step, message: `${plainMessage(ctx, e.message)}. Inspect it with mosaic bus status ${id}; mosaic bus recover ${id} clears a host that died.` });
|
||||
@@ -600,3 +608,33 @@ async function restartBus(req, res, { sub, write, id, ctx, service, admin, log,
|
||||
try { healthy = (await bus.busStatus(ctx, id)).healthy; } catch { /* the receipt still stands */ }
|
||||
return json(res, 200, { outcome: 'done', step, restarted: running, healthy, receipt: forReceipt(ctx, receipt), verb: running ? verbFor('bus-restart', id) : verbFor('bus', id) });
|
||||
}
|
||||
|
||||
// The dev-local owner exactly (packages/cli/src/dev-local-owner.mjs). A signed-in session carries
|
||||
// its provider's issuer, so no SSO user matches.
|
||||
const isDevOwner = by => by?.provider === DEV_OWNER.provider && by.iss === DEV_OWNER.iss && by.sub === DEV_OWNER.sub;
|
||||
|
||||
// D2 (P0-3): the dev Console's Restart bus on a bus whose state did not check out. A Console unit
|
||||
// stop kills the hosts the Console started and leaves their state behind; busRevive clears that
|
||||
// state only when busRecover proves the host dead, then starts the bus on its recorded options.
|
||||
// A live or unprovable host, or a lock that is not the dead host's, refuses and nothing changes.
|
||||
async function reviveBus(res, { id, ctx, bus, log, trail, by, step }) {
|
||||
try { trail({ by, action: 'bus.restart', org: id, outcome: 'requested', running: false, recover: true }); }
|
||||
catch { return json(res, 500, { error: 'trail-unavailable', step, message: 'the action trail could not be written, so nothing was recovered' }); }
|
||||
const note = outcome => { try { trail({ by, action: 'bus.restart', org: id, outcome, recover: true }); } catch { log(`org-admin: ${id} bus recover: trail write failed after ${outcome}`); } };
|
||||
let receipt;
|
||||
try { receipt = await bus.busRevive(ctx, id); }
|
||||
catch (e) {
|
||||
// busRevive marks a failure after busRecover cleared the dead host's state (e.recovered).
|
||||
const cleared = Boolean(e?.recovered);
|
||||
note(cleared ? 'failed' : 'refused');
|
||||
if (!cleared && e?.code === 'org-locked') return json(res, 409, { error: 'bus-busy', step, message: `another bus or lifecycle action on ${id} is running, and nothing was changed. Press Restart bus again when it ends.` });
|
||||
if (cleared) return json(res, 409, { error: 'bus-failed', step, message: isCliError(e) ? plainMessage(ctx, `The dead host's state was cleared, but ${e.message}`) : `The dead host's state was cleared, but the bus did not start. Check it with mosaic bus status ${id}.` });
|
||||
if (!isCliError(e)) return refusal(res, e, { step, id, unknownOk: true });
|
||||
return json(res, 409, { error: 'bus-refused', step, message: `${plainMessage(ctx, e.message)}. Inspect it with mosaic bus status ${id}.` });
|
||||
}
|
||||
note('done');
|
||||
log(`org-admin: ${id} bus recovered and started`);
|
||||
let healthy = null;
|
||||
try { healthy = (await bus.busStatus(ctx, id)).healthy; } catch { /* the receipt still stands */ }
|
||||
return json(res, 200, { outcome: 'done', step, restarted: false, recovered: true, healthy, receipt: forReceipt(ctx, receipt), verb: verbFor('bus-revive', id) });
|
||||
}
|
||||
@@ -146,7 +146,9 @@
|
||||
if (response.type === 'opaqueredirect') throw new Error('Sign in again, then retry.');
|
||||
const body = await response.json();
|
||||
if (!response.ok || body.outcome !== 'done' || body.step !== 'bus-restart') throw new Error(body.message || 'The restart was refused. Nothing was retried.');
|
||||
busReceipt = `${body.restarted ? 'Bus restarted' : 'Bus started'} at ${clock()}. ${body.healthy ? 'It answers.' : `It has not answered yet. Check it with ${body.verb.replace('restart', 'status')}.`}`;
|
||||
// D2: a recovered bus's verb is two commands; its check is the status verb.
|
||||
const check = body.recovered ? `mosaic bus status ${id}` : body.verb.replace('restart', 'status');
|
||||
busReceipt = `${body.recovered ? 'Dead bus host cleared; bus started' : body.restarted ? 'Bus restarted' : 'Bus started'} at ${clock()}. ${body.healthy ? 'It answers.' : `It has not answered yet. Check it with ${check}.`}`;
|
||||
// The page's send flags were read while the bus was in its old state; read them again. This
|
||||
// read goes through the page's fetch, which display.js scopes to the selected organization.
|
||||
// The organization list goes first, so the view the config event redraws shows the bus as it is now.
|
||||
|
||||
@@ -859,7 +859,7 @@ export async function startServer({ host = '127.0.0.1', port = 7330, board = DEF
|
||||
let boundAddress = null, boundPort = null;
|
||||
const server = createServer(async (req, res) => {
|
||||
// A granted user's request rebinds these below; every other request keeps the owner's.
|
||||
let { bus, reads, send, settingsWrite, projectFiles, projectFile, projectSave, libraryStatus, libraryRevert, chat, nameWrite, assistantStop, instancesWrite, routesWrite, agentWrite, proxyCall, processing, decisionWrite, requestRead, requestWrite, systemTypesWrite, upstream, files } = owner, granted = false, chatGrants = [];
|
||||
let { bus, reads, send, settingsWrite, projectFiles, projectFile, projectSave, libraryStatus, libraryRevert, chat, nameWrite, assistantStop, instancesWrite, routesWrite, agentWrite, proxyCall, processing, decisionWrite, requestRead, requestWrite, systemTypesWrite, upstream, files } = owner, granted = false, chatGrants = [], autonomyWrite = null;
|
||||
res.setHeader('x-content-type-options', 'nosniff');
|
||||
res.setHeader('referrer-policy', 'no-referrer');
|
||||
// Under --sso the Sign out form's POST is redirected to the provider's end-session endpoint, and
|
||||
@@ -954,7 +954,7 @@ export async function startServer({ host = '127.0.0.1', port = 7330, board = DEF
|
||||
const instanceAction = originalPath === '/api/console/restart';
|
||||
if (session && !session.rootOwner && session.user === undefined && orgId === null && !orgList && !hook?.publicRead && !instanceAction && path !== '/api/config' && path !== '/healthz' && !files.has(path)) return json(res, 404, { error: 'not-found' });
|
||||
if (session && !session.rootOwner && session.user === undefined) {
|
||||
bus = reads = send = settingsWrite = projectFiles = projectFile = projectSave = libraryStatus = libraryRevert = chat = nameWrite = assistantStop = instancesWrite = routesWrite = agentWrite = proxyCall = processing = decisionWrite = requestRead = requestWrite = systemTypesWrite = upstream = null;
|
||||
bus = reads = send = settingsWrite = projectFiles = projectFile = projectSave = libraryStatus = libraryRevert = chat = nameWrite = assistantStop = instancesWrite = routesWrite = agentWrite = proxyCall = processing = decisionWrite = requestRead = requestWrite = systemTypesWrite = autonomyWrite = upstream = null;
|
||||
}
|
||||
if (session?.user !== undefined && !session.rootOwner && !instanceAction) {
|
||||
if (!organizations && !userRoute(path, req.method)) return json(res, 403, { error: 'owner-only', message: 'this part of the Console is the business owner\'s' });
|
||||
@@ -991,7 +991,7 @@ export async function startServer({ host = '127.0.0.1', port = 7330, board = DEF
|
||||
({ bus = null, reads = null, send = null, settingsWrite = null, projectFiles = null, projectFile = null,
|
||||
projectSave = null, libraryStatus = null, libraryRevert = null, chat = null, nameWrite = null,
|
||||
assistantStop = null, instancesWrite = null, routesWrite = null, agentWrite = null, proxyCall = null,
|
||||
processing = null, decisionWrite = null, requestRead = null, requestWrite = null, systemTypesWrite = null, upstream = null } = selected);
|
||||
processing = null, decisionWrite = null, requestRead = null, requestWrite = null, systemTypesWrite = null, autonomyWrite = null, upstream = null } = selected);
|
||||
}
|
||||
// F3: the owner's settings writes reach the selected organization's bus too (limits, routing,
|
||||
// agents, assistant, instances, model proxy). The routes below keep their own checks.
|
||||
@@ -1052,7 +1052,9 @@ export async function startServer({ host = '127.0.0.1', port = 7330, board = DEF
|
||||
// Phase P2 (#1873): the owner's System types read and writes on the selected organization.
|
||||
if(autonomyPath) {
|
||||
if(tester || granted || !named || sso===null)return json(res,403,{error:'owner-only'});
|
||||
return autonomyHTTP(req,res,{path,read:requestRead,write:send?requestWrite:null,session:ssoSession,
|
||||
// D1 (#1873): an organization's own autonomy writer (`autonomyWrite`, the four autonomy verbs only);
|
||||
// without one the route keeps the request writer, as before.
|
||||
return autonomyHTTP(req,res,{path,read:requestRead,write:send?(autonomyWrite??requestWrite):null,session:ssoSession,
|
||||
live:()=>{const live=sso.session(req);return live?.rootOwner===true?live:null;},origin:localOrigin??sso.publicOrigin});
|
||||
}
|
||||
if (systemTypesPath) {
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
// D2 (P0-3): the dev Console's start-up revive (bus-revive.mjs), on fakes. The real-host paths
|
||||
// (a SIGKILLed host, a live host, a live writer lock) are in packages/cli/tests/org-revive.test.mjs.
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { reviveBuses, revivePending } from '../src/bus-revive.mjs';
|
||||
import { reviveStartOptions } from '../../cli/src/org-bus.mjs';
|
||||
|
||||
const cliError = (message, extra = {}) => Object.assign(new Error(message), { name: 'CliError', exitCode: 3 }, extra);
|
||||
const orgs = ids => async () => ids.map(id => ({ id, state: 'active' }));
|
||||
|
||||
function run({ ids = ['alpha'], status, revive, trail, env = {} }) {
|
||||
const calls = [], lines = [], trailed = [];
|
||||
const done = reviveBuses({
|
||||
ctx: {}, listOrgs: orgs(ids),
|
||||
busStatus: async (ctx, id) => status(id),
|
||||
busRevive: async (ctx, id, opts) => { calls.push([id, opts]); return revive(id); },
|
||||
trail: trail ?? (e => trailed.push(e)),
|
||||
log: line => lines.push(line), env,
|
||||
});
|
||||
return { done, calls, lines, trailed };
|
||||
}
|
||||
|
||||
test('the start-up revive asks busRevive for pm:false, so no chief launches without a press', async () => {
|
||||
const r = run({ status: () => { throw cliError('stale bus host lock'); }, revive: () => ({ pid: 77 }) });
|
||||
assert.deepEqual(await r.done, [{ id: 'alpha', outcome: 'revived', pid: 77 }]);
|
||||
assert.deepEqual(r.calls, [['alpha', { pm: false }]]);
|
||||
});
|
||||
|
||||
test('reviveStartOptions keeps the recorded options, never wider, and pm only when both allow it', () => {
|
||||
assert.deepEqual(reviveStartOptions({ devLocalOwner: true, pm: true, noLaunch: false }, { pm: false }), { detach: true, devLocalOwner: true, pm: false, noLaunch: false });
|
||||
assert.deepEqual(reviveStartOptions({ devLocalOwner: true, pm: true, noLaunch: false }), { detach: true, devLocalOwner: true, pm: true, noLaunch: false });
|
||||
assert.deepEqual(reviveStartOptions({ pm: false, noLaunch: true }, { pm: true }), { detach: true, devLocalOwner: false, pm: false, noLaunch: true });
|
||||
// Anything but a literal true is false: a malformed record widens nothing.
|
||||
assert.deepEqual(reviveStartOptions({ devLocalOwner: 'yes', pm: 1 }), { detach: true, devLocalOwner: false, pm: false, noLaunch: false });
|
||||
assert.deepEqual(reviveStartOptions(null), { detach: true, devLocalOwner: false, pm: false, noLaunch: false });
|
||||
});
|
||||
|
||||
test('a dropped --pm is logged with the way back', async () => {
|
||||
const r = run({ status: () => { throw cliError('stale bus host lock'); }, revive: () => Object.defineProperty({ pid: 9 }, 'pmDropped', { value: true }) });
|
||||
await r.done;
|
||||
assert.ok(r.lines.some(l => l === 'bus revive: alpha: the dead host was started with --pm; the chief was not relaunched. mosaic bus stop alpha; mosaic bus start alpha --pm brings it back'), r.lines.join('\n'));
|
||||
});
|
||||
|
||||
test('running, not-answering, stopped and unreadable buses are left alone and never trailed', async () => {
|
||||
const states = { a: { healthy: true, pid: 1 }, b: { healthy: false, pid: 2 }, c: { healthy: false, pid: null } };
|
||||
const r = run({ ids: ['a', 'b', 'c', 'd'], status: id => { if (id === 'd') throw new Error('EACCES /secret/path'); return states[id]; }, revive: () => assert.fail('revived') });
|
||||
assert.deepEqual(await r.done, [{ id: 'a', outcome: 'running' }, { id: 'b', outcome: 'not-answering' }, { id: 'c', outcome: 'stopped' }, { id: 'd', outcome: 'uncertain' }]);
|
||||
assert.deepEqual(r.trailed, []);
|
||||
assert.ok(!r.lines.join('\n').includes('/secret/path'));
|
||||
});
|
||||
|
||||
test('a trail that cannot be written skips the revive', async () => {
|
||||
const r = run({ status: () => { throw cliError('stale bus host lock'); }, revive: () => assert.fail('revived without a trail line'), trail: () => { throw new Error('disk full'); } });
|
||||
assert.deepEqual(await r.done, [{ id: 'alpha', outcome: 'trail-unavailable' }]);
|
||||
assert.deepEqual(r.calls, []);
|
||||
});
|
||||
|
||||
test('a start that fails after the dead state was cleared is "failed", not "refused"', async () => {
|
||||
const r = run({ status: () => { throw cliError('stale bus host lock'); }, revive: () => { throw cliError('bus failed to start; see bus-host/host.log in the runtime directory of organization alpha', { recovered: { pid: 4 } }); } });
|
||||
assert.deepEqual(await r.done, [{ id: 'alpha', outcome: 'failed' }]);
|
||||
assert.deepEqual(r.trailed.map(e => [e.outcome, e.recovered]), [['requested', undefined], ['failed', true]]);
|
||||
assert.match(r.lines.at(-1), /^bus revive: alpha: dead bus host cleared, but the bus did not start \(bus failed to start;/);
|
||||
// A non-CliError shows its code, never its text.
|
||||
const q = run({ status: () => { throw cliError('stale bus host lock'); }, revive: () => { throw Object.assign(new Error('EIO /home/someone/x'), { code: 'org-locked' }); } });
|
||||
assert.deepEqual(await q.done, [{ id: 'alpha', outcome: 'refused' }]);
|
||||
assert.equal(q.lines.at(-1), 'bus revive: alpha: refused (unexpected error org-locked); left for mosaic bus status alpha');
|
||||
});
|
||||
|
||||
test('revivePending holds each organization until its own revive ends', async () => {
|
||||
let release;
|
||||
const gate = new Promise(resolve => { release = resolve; });
|
||||
const r = run({ ids: ['alpha', 'beta'], status: id => { if (id === 'alpha') throw cliError('stale bus host lock'); return { healthy: true, pid: 3 }; }, revive: async () => { await gate; return { pid: 5 }; } });
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
const alpha = revivePending('alpha'), beta = revivePending('beta');
|
||||
assert.ok(alpha instanceof Promise); assert.ok(beta instanceof Promise);
|
||||
assert.equal(revivePending('gamma'), null);
|
||||
let settled = false; alpha.then(() => { settled = true; });
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
assert.equal(settled, false);
|
||||
release();
|
||||
await r.done;
|
||||
assert.equal(settled, true);
|
||||
assert.equal(revivePending('alpha'), null); assert.equal(revivePending('beta'), null);
|
||||
});
|
||||
|
||||
test('a Console that runs inside an agent run revives nothing and reads no organization', async () => {
|
||||
for (const marker of ['MOSAIC_AGENT_RUN', 'CLAUDECODE', 'CODEX_THREAD_ID']) {
|
||||
let listed = false;
|
||||
const lines = [];
|
||||
const results = await reviveBuses({ ctx: {}, listOrgs: async () => { listed = true; return []; }, busStatus: () => assert.fail('read'), busRevive: () => assert.fail('revived'), trail: () => assert.fail('trailed'), log: line => lines.push(line), env: { [marker]: '1' } });
|
||||
assert.deepEqual(results, []);
|
||||
assert.equal(listed, false);
|
||||
assert.deepEqual(lines, ['bus revive: this Console runs inside an agent run; no bus was checked']);
|
||||
}
|
||||
});
|
||||
@@ -565,3 +565,112 @@ test('a bus that refuses to restart is a visible failure and the trail says refu
|
||||
assert.deepEqual(tr.lines().slice(2), [{ by: DEV, action: 'bus.restart', org: 'acme', outcome: 'refused', running: null }]);
|
||||
assert.equal(c.calls.slice(before).filter(y => ['busStart', 'busRestart'].includes(y[0])).length, 0);
|
||||
});
|
||||
|
||||
// D2 (P0-3): on the dev Console, Restart bus on a host that died (its state left behind by a
|
||||
// Console unit stop) clears that state through busRevive and starts the bus. busRevive's
|
||||
// busRecover is the proof; a live or unprovable host refuses and nothing starts.
|
||||
const cliError = message => Object.assign(new Error(message), { name: 'CliError', exitCode: 3 });
|
||||
const withRevive = (c, revive) => c.admin.bus().then(bus => {
|
||||
bus.busStatus = async () => { throw cliError('stale bus host lock'); };
|
||||
bus.busRevive = async (ctx, id, opts) => { c.calls.push(['busRevive', id, opts]); return revive(bus, id); };
|
||||
return bus;
|
||||
});
|
||||
|
||||
test('D2: the dev owner\'s Restart bus brings back a host that died, and the trail records the recovery', async t => {
|
||||
const tr = trailIn(t);
|
||||
const c = await console_(t, { trail: tr.trail });
|
||||
await c.call('POST', '/api/orgs', { id: 'acme', doc: browserDoc('acme') });
|
||||
await withRevive(c, (bus, id) => {
|
||||
bus.busStatus = async () => ({ org: id, healthy: true, pid: 5151 });
|
||||
return { org: id, pid: 5151, socket: '/data/org-runtime/acme/bus/broker.sock', root: '/data/org-runtime/acme', recovered: { pid: 4242, receipt: '/data/org-runtime/acme/bus-host/recovery-1.json' } };
|
||||
});
|
||||
const r = await c.call('POST', '/api/orgs/acme/restart/bus', {});
|
||||
assert.equal(r.status, 200, r.raw);
|
||||
assert.deepEqual(r.body, { outcome: 'done', step: 'bus-restart', restarted: false, recovered: true, healthy: true,
|
||||
receipt: { org: 'acme', pid: 5151, socket: '<data root>/org-runtime/acme/bus/broker.sock', root: '<data root>/org-runtime/acme', recovered: { pid: 4242, receipt: '<data root>/org-runtime/acme/bus-host/recovery-1.json' } },
|
||||
verb: 'mosaic bus recover acme; mosaic bus start acme --dev-local-owner --detach' });
|
||||
assert.deepEqual(c.calls.filter(x => ['busRevive', 'busStart', 'busRestart'].includes(x[0])), [['busRevive', 'acme', undefined]]);
|
||||
assert.deepEqual(tr.lines(), [
|
||||
{ by: DEV, action: 'bus.restart', org: 'acme', outcome: 'requested', running: false, recover: true },
|
||||
{ by: DEV, action: 'bus.restart', org: 'acme', outcome: 'done', recover: true },
|
||||
]);
|
||||
});
|
||||
|
||||
test('D2: a host that is alive or unprovable still refuses on the dev Console, and nothing starts', async t => {
|
||||
const tr = trailIn(t);
|
||||
const c = await console_(t, { trail: tr.trail });
|
||||
await c.call('POST', '/api/orgs', { id: 'acme', doc: browserDoc('acme') });
|
||||
for (const reason of ['bus host pid 4242 is alive', 'bus host lock does not match host state', 'bus socket answers or is uncertain', 'bus host identity is uncertain']) {
|
||||
await withRevive(c, () => { throw cliError(reason); });
|
||||
const r = await c.call('POST', '/api/orgs/acme/restart/bus', {});
|
||||
assert.equal(r.status, 409, reason); assert.equal(r.body.error, 'bus-refused');
|
||||
assert.equal(r.body.message, `${reason}. Inspect it with mosaic bus status acme.`);
|
||||
}
|
||||
assert.equal(c.calls.filter(x => ['busStart', 'busRestart'].includes(x[0])).length, 0);
|
||||
assert.deepEqual(tr.lines().map(e => e.outcome), ['requested', 'refused', 'requested', 'refused', 'requested', 'refused', 'requested', 'refused']);
|
||||
// A host that was cleared but did not come back says so, without host paths.
|
||||
await withRevive(c, () => { throw Object.assign(cliError('bus failed to start; see bus-host/host.log in the runtime directory of organization acme'), { recovered: { pid: 4242 } }); });
|
||||
const f = await c.call('POST', '/api/orgs/acme/restart/bus', {});
|
||||
assert.equal(f.status, 409); assert.equal(f.body.error, 'bus-failed');
|
||||
assert.equal(f.body.message, "The dead host's state was cleared, but bus failed to start; see bus-host/host.log in the runtime directory of organization acme");
|
||||
assert.deepEqual(tr.lines().slice(-1), [{ by: DEV, action: 'bus.restart', org: 'acme', outcome: 'failed', recover: true }]);
|
||||
// Another lifecycle action holds the organization's lock: nothing changed, and the page says to press again.
|
||||
await withRevive(c, () => { throw Object.assign(new Error('organization lifecycle lock is held; inspect the lock and resume'), { code: 'org-locked' }); });
|
||||
const l = await c.call('POST', '/api/orgs/acme/restart/bus', {});
|
||||
assert.equal(l.status, 409); assert.equal(l.body.error, 'bus-busy');
|
||||
assert.equal(l.body.message, 'another bus or lifecycle action on acme is running, and nothing was changed. Press Restart bus again when it ends.');
|
||||
assert.deepEqual(tr.lines().slice(-1).map(e => e.outcome), ['refused']);
|
||||
// No trail, no recovery.
|
||||
const broken = await console_(t, { trail: () => { throw new Error('disk full'); } });
|
||||
await broken.call('POST', '/api/orgs', { id: 'acme', doc: browserDoc('acme') });
|
||||
await withRevive(broken, () => assert.fail('busRevive ran without a trail line'));
|
||||
const b = await broken.call('POST', '/api/orgs/acme/restart/bus', {});
|
||||
assert.equal(b.status, 500); assert.equal(b.body.error, 'trail-unavailable');
|
||||
});
|
||||
|
||||
test('D2: a signed-in (non-dev) owner keeps the stale-state refusal and never reaches busRevive', async t => {
|
||||
const { Readable } = await import('node:stream');
|
||||
const tr = trailIn(t);
|
||||
const f = fakes();
|
||||
await f.service.createOrg({}, 'acme', template('acme'));
|
||||
const bus = await f.admin.bus();
|
||||
bus.busStatus = async () => { throw cliError('stale bus host lock'); };
|
||||
bus.busRevive = async () => assert.fail('busRevive reached from a signed-in session');
|
||||
const call = async session => {
|
||||
const req = Object.assign(Readable.from([Buffer.from('{}')]), { method: 'POST', headersDistinct: { origin: ['http://console.example'], 'sec-fetch-site': ['same-origin'], 'content-type': ['application/json'] } });
|
||||
const res = { writeHead(status) { this.status = status; }, end(body) { this.body = JSON.parse(body); } };
|
||||
await orgAdminHTTP(req, res, { path: '/api/orgs/acme/restart/bus', admin: f.admin, session, owner: true, send: true, origin: 'http://console.example', trail: tr.trail });
|
||||
return res;
|
||||
};
|
||||
// An SSO owner, and a session that names the local-dev provider under another issuer.
|
||||
for (const session of [{ provider: 'fixture', iss: 'https://issuer.example', sub: 'owner', name: 'jason', rootOwner: true }, { provider: 'local-dev', iss: 'https://issuer.example', sub: 'local-dev-owner', rootOwner: true }]) {
|
||||
const r = await call(session);
|
||||
assert.equal(r.status, 409); assert.equal(r.body.error, 'bus-refused');
|
||||
assert.equal(r.body.message, 'stale bus host lock. Inspect it with mosaic bus status acme; mosaic bus recover acme clears a host that died.');
|
||||
}
|
||||
assert.deepEqual(tr.lines().map(e => [e.outcome, e.running, e.recover]), [['refused', null, undefined], ['refused', null, undefined]]);
|
||||
});
|
||||
|
||||
test('D2: a press during the Console start-up revive waits for it and acts on the bus it brought back', async t => {
|
||||
const { reviveBuses } = await import('../src/bus-revive.mjs');
|
||||
const tr = trailIn(t);
|
||||
const c = await console_(t, { trail: tr.trail });
|
||||
await c.call('POST', '/api/orgs', { id: 'acme', doc: browserDoc('acme') });
|
||||
const bus = await c.admin.bus();
|
||||
let up = false, release;
|
||||
const gate = new Promise(resolve => { release = resolve; });
|
||||
bus.busStatus = async () => { if (!up) throw cliError('stale bus host lock'); return { org: 'acme', healthy: true, pid: 6161 }; };
|
||||
bus.busRevive = async () => assert.fail('the press revived a bus the Console was already bringing back');
|
||||
bus.busRestart = async (ctx, id) => { c.calls.push(['busRestart', id]); return { org: id, pid: 6262 }; };
|
||||
const startup = reviveBuses({ ctx: {}, env: {}, listOrgs: async () => [{ id: 'acme', state: 'active' }], busStatus: bus.busStatus,
|
||||
busRevive: async () => { await gate; up = true; return { pid: 6161 }; }, trail: () => {}, log: () => {} });
|
||||
await new Promise(resolve => setImmediate(resolve));
|
||||
const press = c.call('POST', '/api/orgs/acme/restart/bus', {});
|
||||
await new Promise(resolve => setTimeout(resolve, 50));
|
||||
assert.equal(c.calls.filter(x => x[0] === 'busRestart').length, 0);
|
||||
release();
|
||||
await startup;
|
||||
const r = await press;
|
||||
assert.equal(r.status, 200, r.raw); assert.equal(r.body.restarted, true);
|
||||
assert.deepEqual(c.calls.filter(x => x[0] === 'busRestart'), [['busRestart', 'acme']]);
|
||||
});
|
||||
@@ -0,0 +1,173 @@
|
||||
// D1 (#1873, readiness audit P0-2): on a Console started with --dev-local-owner, an organization's
|
||||
// autonomy page read `writable:false` and refused every owner save. The organization options built in
|
||||
// cli.mjs carried no writer the autonomy route could use. This runs the real cli.mjs serve
|
||||
// --dev-local-owner against a real organization record; only the bus transports cli.mjs builds are
|
||||
// replaced (loader hook), so no bus or human proof is involved.
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { mkdirSync, readFileSync, renameSync, chmodSync, writeFileSync } from 'node:fs';
|
||||
import { request } from 'node:http';
|
||||
import { join } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { tmp, fixture } from '../../cli/tests/helpers.mjs';
|
||||
import { fakeOidc } from './helpers/fake-oidc.mjs';
|
||||
import { ownerAccounts } from './helpers/owner-accounts.mjs';
|
||||
import { SESSION_COOKIE } from '../src/sso.mjs';
|
||||
import { createOrg } from '../../organization/src/org-service.mjs';
|
||||
import { orgPaths } from '../../cli/src/config.mjs';
|
||||
import { DEV_OWNER } from '../../cli/src/dev-local-owner.mjs';
|
||||
|
||||
const AUTONOMY = { revision: 3, autonomy: { default: 'near' } };
|
||||
const HOST = 'console.example.test', ORIGIN = `https://${HOST}`;
|
||||
|
||||
// `sso`: a deployed Console instead (--sso --send, a fake OP on loopback, the owner signed in).
|
||||
async function consoleFor(t, { sso = false } = {}) {
|
||||
const root = tmp(t, 'o-'), f = fixture(root), home = join(root, 'home'), configDir = join(home, '.config', 'mosaic-test');
|
||||
mkdirSync(join(home, '.config'), { recursive: true }); renameSync(join(root, 'config'), configDir);
|
||||
chmodSync(configDir, 0o700); chmodSync(join(configDir, 'businesses'), 0o700);
|
||||
writeFileSync(join(f.dataRoot, 'instance.json'), JSON.stringify({ instanceVersion: 1, name: 'test' }));
|
||||
const doc = structuredClone(f.doc); doc.id = 'alpha'; doc.human = DEV_OWNER.sub; doc.projects.stack.root = '/__mosaic_org__/alpha/projects/stack';
|
||||
const org = await createOrg({ configDir, dataRoot: f.dataRoot, instance: 'test', principal: DEV_OWNER }, 'alpha', doc);
|
||||
const log = join(root, 'calls.jsonl'); writeFileSync(log, '');
|
||||
const busURL = new URL('../src/bus.mjs', import.meta.url).href, orgBusURL = new URL('../../cli/src/org-bus.mjs', import.meta.url).href;
|
||||
// The console transport answers the autonomy verbs from fixed rows and logs (kind, business, socket, verb, args, subject).
|
||||
const busShim = join(root, 'bus-shim.mjs'), orgBusShim = join(root, 'org-bus-shim.mjs'), hook = join(root, 'hook.mjs');
|
||||
writeFileSync(busShim, `import { appendFileSync } from 'node:fs';
|
||||
import { BusReadError } from ${JSON.stringify(busURL)};
|
||||
export * from ${JSON.stringify(busURL)};
|
||||
const record = (kind, o, verb, args, extra = {}) => appendFileSync(${JSON.stringify(log)}, JSON.stringify({ kind, business: o.business ?? null, socket: o.socket ?? null, verb, args, ...extra }) + '\\n');
|
||||
export const busCaller = o => async (verb, args) => { record('human', o, verb, args); return verb === 'plugins' ? { plugins: { channel: ['cli', 'webui'] } } : []; };
|
||||
export const consoleCall = o => async (verb, args, { subject }) => {
|
||||
record('console', o, verb, args, { subject });
|
||||
if (verb === 'autonomy.get.console') return { revision: 3, autonomy: { default: 'in' } };
|
||||
if (verb === 'autonomy.set.console') return { revision: 4, autonomy: args.autonomy, receipt: { action: 'autonomy.set', revision: 4 } };
|
||||
if (['autonomy.review.console', 'decision.confirm.console', 'decision.reverse.console'].includes(verb)) return { answered: verb };
|
||||
throw new BusReadError('unavailable', 'fixture: ' + verb);
|
||||
};
|
||||
`);
|
||||
writeFileSync(orgBusShim, `export * from ${JSON.stringify(orgBusURL)};\nexport const busStatus = async () => ({ healthy: true });\n`);
|
||||
writeFileSync(hook, `import { registerHooks } from 'node:module';
|
||||
registerHooks({ resolve(specifier, context, next) {
|
||||
if (context.parentURL?.endsWith('/webui/src/cli.mjs')) {
|
||||
if (specifier === './bus.mjs') return { url: ${JSON.stringify(pathToFileURL(busShim).href)}, shortCircuit: true };
|
||||
if (specifier === '../../cli/src/org-bus.mjs') return { url: ${JSON.stringify(pathToFileURL(orgBusShim).href)}, shortCircuit: true };
|
||||
}
|
||||
return next(specifier, context);
|
||||
} });
|
||||
`);
|
||||
const env = { ...f.env, HOME: home, MOSAIC_CONFIG: join(configDir, 'config.json'), NODE_OPTIONS: `--import=${pathToFileURL(hook).href}` };
|
||||
let op = null, a = null;
|
||||
if (sso) {
|
||||
op = await fakeOidc(); t.after(() => op.close());
|
||||
a = op.issuer('a');
|
||||
const provider = { id: a.name, issuer: a.issuer, clientId: a.clientId, clientSecret: { slot: a.name }, algs: [...new Set(a.keys.map(k => k.alg))], subjects: [{ sub: `user-${a.name}`, name: `person-${a.name}` }] };
|
||||
const config = JSON.parse(readFileSync(env.MOSAIC_CONFIG, 'utf8'));
|
||||
writeFileSync(env.MOSAIC_CONFIG, JSON.stringify({ ...config, console: { sso: { publicOrigin: ORIGIN, providers: [provider] } } }));
|
||||
mkdirSync(join(configDir, 'secrets', 'mosaic-stack'), { recursive: true, mode: 0o700 });
|
||||
writeFileSync(join(configDir, 'secrets', 'mosaic-stack', a.name), `${a.clientSecret}\n`, { mode: 0o600 });
|
||||
writeFileSync(join(configDir, 'accounts.json'), JSON.stringify({ accountsVersion: 1, revision: 1, users: ownerAccounts({ providers: [provider] }).load().users.map(u => ({ ...u, created: '2026-11-10T00:00:00Z' })), memberships: [], invites: [] }), { mode: 0o600 });
|
||||
writeFileSync(join(root, 'ca.pem'), op.cert);
|
||||
env.NODE_EXTRA_CA_CERTS = join(root, 'ca.pem');
|
||||
}
|
||||
const mode = sso ? ['--sso', '--send'] : ['--dev-local-owner'];
|
||||
const child = spawn(process.execPath, [new URL('../src/cli.mjs', import.meta.url).pathname, 'serve', '--port', '0', '--board', 'http://127.0.0.1:9', ...mode], { env, stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
let out = ''; child.stdout.on('data', c => out += c); child.stderr.on('data', c => out += c);
|
||||
t.after(() => { if (child.exitCode === null) child.kill('SIGTERM'); });
|
||||
for (let i = 0; i < 300 && !/Mosaic Console: (http:\/\/127\.0\.0\.1:\d+\/)/.test(out); i++) {
|
||||
if (child.exitCode !== null) break;
|
||||
await new Promise(r => setTimeout(r, 50));
|
||||
}
|
||||
const base = /Mosaic Console: (http:\/\/127\.0\.0\.1:(\d+)\/)/.exec(out);
|
||||
assert.ok(base, `the Console did not start:\n${out}`);
|
||||
const calls = () => readFileSync(log, 'utf8').trim().split('\n').filter(Boolean).map(line => JSON.parse(line));
|
||||
return { port: Number(base[2]), calls, org, op, a, rootSocket: join(f.dataRoot, 'bus', 'broker.sock') };
|
||||
}
|
||||
|
||||
function call(port, path, { method = 'GET', body = null, headers = {}, host = `127.0.0.1:${port}`, origin = `http://${host}` } = {}) {
|
||||
const h = { host, ...(method === 'GET' ? {} : { origin, 'sec-fetch-site': 'same-origin', 'content-type': 'application/json' }), ...headers };
|
||||
for (const k of Object.keys(h)) if (h[k] === null) delete h[k];
|
||||
return new Promise((resolve, reject) => {
|
||||
const r = request({ host: '127.0.0.1', port, path, method, headers: h }, res => {
|
||||
let text = ''; res.setEncoding('utf8'); res.on('data', c => { text += c; });
|
||||
res.on('end', () => { let json = null; try { json = JSON.parse(text); } catch {} resolve({ status: res.statusCode, headers: res.headers, body: json }); });
|
||||
});
|
||||
r.on('error', reject); r.end(body ?? undefined);
|
||||
});
|
||||
}
|
||||
|
||||
test('the dev-local owner reads an organization autonomy page as writable and a save reaches that organization bus', { timeout: 60000 }, async t => {
|
||||
const { port, calls, org } = await consoleFor(t);
|
||||
const subject = { provider: DEV_OWNER.provider, iss: DEV_OWNER.iss, sub: DEV_OWNER.sub };
|
||||
const read = await call(port, '/api/orgs/alpha/autonomy');
|
||||
assert.equal(read.status, 200, JSON.stringify(read.body));
|
||||
assert.deepEqual(read.body, { revision: 3, autonomy: { default: 'in' }, writable: true });
|
||||
const saved = await call(port, '/api/orgs/alpha/autonomy', { method: 'POST', body: JSON.stringify(AUTONOMY) });
|
||||
assert.equal(saved.status, 200, JSON.stringify(saved.body));
|
||||
assert.deepEqual(saved.body, { revision: 4, autonomy: { default: 'near' }, receipt: { action: 'autonomy.set', revision: 4 }, writable: true });
|
||||
const socket = orgPaths(org.root).socket;
|
||||
assert.deepEqual(calls().filter(c => c.kind === 'console').map(({ business, socket: s, verb, args, subject: who }) => ({ business, socket: s, verb, args, subject: who })), [
|
||||
{ business: 'alpha', socket, verb: 'autonomy.get.console', args: {}, subject },
|
||||
{ business: 'alpha', socket, verb: 'autonomy.set.console', args: AUTONOMY, subject },
|
||||
]);
|
||||
// Review, confirm and reverse go to the same organization bus as the owner (D1 fix round).
|
||||
for (const [step, body, verb] of [['review', { decisionClass: 'merge' }, 'autonomy.review.console'], ['confirm', { id: 'd-1', confirm: true, reason: 'checked' }, 'decision.confirm.console'], ['reverse', { id: 'd-1' }, 'decision.reverse.console']]) {
|
||||
const r = await call(port, `/api/orgs/alpha/autonomy/${step}`, { method: 'POST', body: JSON.stringify(body) });
|
||||
assert.equal(r.status, 200, `${step}: ${JSON.stringify(r.body)}`);
|
||||
assert.deepEqual(r.body, { answered: verb, writable: true });
|
||||
assert.deepEqual(calls().filter(c => c.kind === 'console').at(-1), { kind: 'console', business: 'alpha', socket, verb, args: body, subject });
|
||||
}
|
||||
// The write keeps the route's own checks: another site saves nothing.
|
||||
const before = calls().length;
|
||||
for (const headers of [{ origin: 'http://evil.example' }, { 'sec-fetch-site': 'cross-site' }]) {
|
||||
const r = await call(port, '/api/orgs/alpha/autonomy', { method: 'POST', body: JSON.stringify(AUTONOMY), headers });
|
||||
assert.equal(r.status, 403, JSON.stringify(headers));
|
||||
}
|
||||
assert.equal(calls().length, before);
|
||||
// Request intake and project remote writes stay unavailable for the organizations release (README inventory).
|
||||
const config = (await call(port, '/api/orgs/alpha/config')).body;
|
||||
assert.equal(config.requestWrites, false); assert.equal(config.projectRemoteWrites, false);
|
||||
});
|
||||
|
||||
test('the organization autonomy writer passes the four autonomy verbs and nothing else', async () => {
|
||||
// Imported here so the other tests in this file also run against BASE, which has no such export.
|
||||
const { AUTONOMY_WRITE_VERBS, autonomyCaller } = await import('../src/autonomy-http.mjs');
|
||||
const seen = [];
|
||||
const write = autonomyCaller(async (verb, args, options) => { seen.push(verb); return { verb, args, options }; });
|
||||
for (const verb of AUTONOMY_WRITE_VERBS) assert.equal((await write(verb, {}, { subject: {} })).verb, verb);
|
||||
assert.deepEqual(AUTONOMY_WRITE_VERBS, ['autonomy.set.console', 'autonomy.review.console', 'decision.confirm.console', 'decision.reverse.console']);
|
||||
for (const verb of ['autonomy.get.console', 'request.draft.console', 'project.remote.set.console', 'decision.resolve.console', 'hire.start.console', 'prd.revoke.console']) {
|
||||
await assert.rejects(write(verb, {}, { subject: {} }), /does not call/);
|
||||
}
|
||||
assert.deepEqual(seen, AUTONOMY_WRITE_VERBS);
|
||||
});
|
||||
|
||||
// Without --dev-local-owner nothing changed (D1 acceptance). A deployed Console (--sso --send) has no
|
||||
// organization routes, and its own autonomy route still reads and writes through the instance's request
|
||||
// transport (the root bus, no business), for its signed-in owner, exactly as before D1.
|
||||
test('a deployed --sso --send Console is unchanged: no organization autonomy route, and the root route keeps the request transport', { timeout: 60000 }, async t => {
|
||||
const { port, calls, op, a, rootSocket } = await consoleFor(t, { sso: true });
|
||||
const login = await call(port, '/auth/login', { host: HOST });
|
||||
assert.equal(login.status, 303);
|
||||
const loginCookie = [login.headers['set-cookie'] ?? []].flat()[0].split(';')[0];
|
||||
const granted = op.grant(a, login.headers.location);
|
||||
const state = new URL(login.headers.location).searchParams.get('state');
|
||||
const signed = await call(port, `/auth/callback?${new URLSearchParams({ code: granted.code, state })}`, { host: HOST, headers: { cookie: loginCookie } });
|
||||
const cookie = [signed.headers['set-cookie'] ?? []].flat().find(c => c.startsWith(`${SESSION_COOKIE}=`) && !c.includes('Max-Age=0'))?.split(';')[0];
|
||||
assert.ok(cookie, `no session: ${signed.status}`);
|
||||
const orgRead = await call(port, '/api/orgs/alpha/autonomy', { host: HOST, headers: { cookie } });
|
||||
assert.equal(orgRead.status, 404); assert.deepEqual(orgRead.body, { error: 'not found' });
|
||||
const orgSave = await call(port, '/api/orgs/alpha/autonomy', { method: 'POST', body: JSON.stringify(AUTONOMY), host: HOST, origin: ORIGIN, headers: { cookie } });
|
||||
assert.equal(orgSave.status, 403); assert.deepEqual(orgSave.body, { error: 'read-only', message: 'the Console preview is read-only' });
|
||||
const read = await call(port, '/api/autonomy', { host: HOST, headers: { cookie } });
|
||||
assert.equal(read.status, 200, JSON.stringify(read.body));
|
||||
assert.deepEqual(read.body, { revision: 3, autonomy: { default: 'in' }, writable: true });
|
||||
const saved = await call(port, '/api/autonomy', { method: 'POST', body: JSON.stringify(AUTONOMY), host: HOST, origin: ORIGIN, headers: { cookie } });
|
||||
assert.equal(saved.status, 200, JSON.stringify(saved.body));
|
||||
assert.deepEqual(saved.body, { revision: 4, autonomy: { default: 'near' }, receipt: { action: 'autonomy.set', revision: 4 }, writable: true });
|
||||
const subject = { provider: a.name, iss: a.issuer, sub: `user-${a.name}` };
|
||||
assert.deepEqual(calls().filter(c => c.kind === 'console').map(({ business, socket, verb, subject: who }) => ({ business, socket, verb, subject: who })), [
|
||||
{ business: null, socket: rootSocket, verb: 'autonomy.get.console', subject },
|
||||
{ business: null, socket: rootSocket, verb: 'autonomy.set.console', subject },
|
||||
]);
|
||||
});
|
||||
Reference in new issue
Block a user