stack-release: rehearse, the D2 rehearsal on a scratch root, and bus start --no-launch (row 163, #1660)
rehearse ID BUSINESS --backup ACTIVATION|PATH --port N builds <dataRoot>/rehearsals/<stamp>/ from a backup, guards the scratch config (mos-0961, mos-0962, mos-0963), runs the migration checks when the schema differs, starts the target's bus host with --no-launch and its Console on the scratch root, runs D1's gate checks but the PM's, and checks the backup and live config files are unchanged. bus start --no-launch refuses every launch, starts no thread tick, leaves an earlier host's sessions.json and runs alone, and keeps a read-only Registry.
This commit is contained in:
jason.woltje committed
1 parent
4f02486d56
commit
5805006993
9 files changed
+828
-14
No files matched your search
@@ -528,6 +528,68 @@ on PATH; the checkout's `scripts/mosaic` exits 3 with `human-required` or
|
||||
`tester-required`, and queue writes stay on it ([packages/cli/README.md, One
|
||||
pinned copy](../packages/cli/README.md#one-pinned-copy-row-85-1579)).
|
||||
|
||||
#### Rehearsal on a scratch root (D2, row 163)
|
||||
|
||||
```
|
||||
scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]
|
||||
```
|
||||
|
||||
`rehearse` starts the target copy's bus host and Console on a copy of a
|
||||
backup and runs `activate`'s gate checks on them, all but the PM's, while the
|
||||
live stack keeps running. It reads the live config and a backup of the live
|
||||
store, so Jason or Mos runs it, as a window's pre-step. No development seat
|
||||
runs it against live data. It stops and starts no unit and writes no pointer
|
||||
or log line.
|
||||
|
||||
`--backup` takes an activation id under `<dataRoot>/backups/stack/` or an
|
||||
absolute path. `--port` is the scratch Console's loopback port, 1024 to
|
||||
65535. The run refuses `port-reserved` for 7332, 7333 and the pointer's
|
||||
`--port` and `--tester-port`, and `port-in-use` for a port something holds;
|
||||
it never picks another port. `--timeout` (default 60 seconds) bounds the
|
||||
gate and each stop.
|
||||
|
||||
Before it makes anything, it checks the pointer, the copy (present,
|
||||
reachable from refactor, a readable schema, a bus host with `--no-launch`),
|
||||
the port and the backup's `SHA256SUMS`. Then it makes
|
||||
`<dataRoot>/rehearsals/<UTC second>-<copy's first 12>/`, mode 0700:
|
||||
|
||||
- `data/bus/` holds a copy of the backup's store files, and
|
||||
`data/notify/BUSINESS/notify.json` is `{"notifyVersion":1,"binding":null}`.
|
||||
- `config/config.json` is the live config with `dataRoot` set to `data/`.
|
||||
- `config/businesses/BUSINESS.json` is the live business file with every
|
||||
`tracker.*` variable removed and every credential reference, the roles'
|
||||
and `tracker.sync`'s, pointed at its own 0600 file under `placeholders/`
|
||||
holding `rehearsal-placeholder-token`. `business.diff` shows the changes.
|
||||
Roles come from the target copy's `roles/`, as the live bus reads them.
|
||||
- `steps.log` holds the same PASS and FAIL lines the run prints, with times;
|
||||
`host.log`, `console.log` and, after a migration, `migrate.log` sit beside it.
|
||||
|
||||
A guard then loads the scratch config with the target's code and refuses
|
||||
with a named check: `data-root-live` when the scratch data root resolves to
|
||||
the live one, `sessions-live` when a `sessions.json` there names a running
|
||||
pid (the run never deletes the file), `data-root-scratch`, `notify-off`,
|
||||
`tracker-configured` when `bootConfig` still sets a tracker, and
|
||||
`credential-ref-live` for an env reference or a file outside `placeholders/`.
|
||||
When the backup's schema differs from the target's, the run checks that the
|
||||
target refuses the store, migrates it, opens it, migrates it again as a
|
||||
no-op, and that the pointer's copy refuses the migrated store.
|
||||
|
||||
It starts `scripts/mosaic bus start BUSINESS --no-launch` and the Console's
|
||||
`serve --port N --business BUSINESS` from the target copy, without `--send`.
|
||||
The gate needs the host's `launches disabled` line, a live `bus status` host
|
||||
with this run's pid and a boot from this start, the Console's address line,
|
||||
`/healthz` 200, and both command lines on the target copy. Then it checks
|
||||
that the host wrote no `sessions.json` and no `launches`, stops the Console
|
||||
and the host (SIGTERM, then SIGKILL and a FAIL after the timeout), checks that
|
||||
the host released `writer.lock`, and compares the backup and the live config
|
||||
and business files with sums it took before. The last line is
|
||||
`REHEARSAL_PASS` (exit 0) or `REHEARSAL_FAIL` (exit 1), then the directory.
|
||||
Usage exits 4. Nothing prunes rehearsal directories.
|
||||
|
||||
`bus start --no-launch` refuses every launch with `launches-disabled`,
|
||||
starts no thread ticker, and leaves an earlier host's `sessions.json` and
|
||||
its runs alone. No unit passes it, and `--pm` with it is a usage error.
|
||||
|
||||
## Console service (`scripts/webui-service.sh`)
|
||||
|
||||
```bash
|
||||
|
||||
@@ -57,7 +57,7 @@ export const USAGE = `usage:
|
||||
mosaic stop <run>
|
||||
mosaic launches off|on [--business <id>]
|
||||
mosaic launches list [--json]
|
||||
mosaic bus start <business> [--pm]
|
||||
mosaic bus start <business> [--pm | --no-launch]
|
||||
mosaic bus stop
|
||||
mosaic bus status [--json]`;
|
||||
|
||||
@@ -482,8 +482,11 @@ async function launches(rest, io, deps) {
|
||||
async function bus(rest, io, deps) {
|
||||
const [sub, ...args] = rest;
|
||||
if (sub === "start") {
|
||||
const parsed = parse(args, { flags: ["--pm"] });
|
||||
const parsed = parse(args, { flags: ["--pm", "--no-launch"] });
|
||||
if (parsed.positional.length !== 1) throw usage("bus start takes one business");
|
||||
// Row 163: --no-launch is the D2 rehearsal's host. No unit passes it.
|
||||
const launches = !parsed.flags.has("--no-launch");
|
||||
if (!launches && parsed.flags.has("--pm")) throw usage("bus start takes --pm or --no-launch, not both");
|
||||
const businessId = parsed.positional[0];
|
||||
const system = deps.system ?? loadSystem({ env: io.env });
|
||||
// Fail closed, and say so (vikunja:19/302): a refused load is recorded where the Console and
|
||||
@@ -500,9 +503,9 @@ async function bus(rest, io, deps) {
|
||||
const binding = readNotifyConfig(system.dataRoot, businessId);
|
||||
const host = await startHost({ boot, business: businessId, notifier: binding ? { binding } : null });
|
||||
try {
|
||||
const launcher = createLauncher({ host, system, env: io.env, tracker: Boolean(boot.trackers?.[businessId]), ...(deps.launcher ?? {}) });
|
||||
const launcher = createLauncher({ host, system, env: io.env, tracker: Boolean(boot.trackers?.[businessId]), launches, ...(deps.launcher ?? {}) });
|
||||
await launcher.listen();
|
||||
io.stdout.write(`bus host up: business ${businessId}, socket ${host.path}, notifier ${binding ?? "off"}\n`);
|
||||
io.stdout.write(`bus host up: business ${businessId}, socket ${host.path}, notifier ${binding ?? "off"}${launches ? "" : ", launches disabled"}\n`);
|
||||
if (parsed.flags.has("--pm")) {
|
||||
const pm = await launcher.launchPm();
|
||||
io.stdout.write(`launched ${pm.instance}: run ${pm.run}, ${pm.harness} ${pm.model}\n`);
|
||||
|
||||
@@ -153,7 +153,11 @@ function call(name, what, fn) {
|
||||
}
|
||||
|
||||
// host: startHost()'s handle. tracker: true when the broker has task verbs for
|
||||
// the business. makeLoader (the plug-in loader's factory, which tests point at a
|
||||
// the business. launches: false for `bus start --no-launch` (row 163, the D2
|
||||
// rehearsal): start() refuses launches-disabled, listen() starts no ticker, a
|
||||
// session's launch request is refused, recover() kills and ends nothing, and
|
||||
// sessions.json stays as it was.
|
||||
// makeLoader (the plug-in loader's factory, which tests point at a
|
||||
// scratch tree), bwrap, readOnly, sessionDefaults, harnessVersion,
|
||||
// stopTimeoutMs, threadTickMs and git (createGitSocket options) exist for
|
||||
// the tests; the command line never sets them.
|
||||
@@ -162,6 +166,7 @@ export function createLauncher({
|
||||
system,
|
||||
env = process.env,
|
||||
tracker = false,
|
||||
launches = true,
|
||||
makeLoader = createLoader,
|
||||
bwrap = BWRAP,
|
||||
readOnly = [],
|
||||
@@ -187,7 +192,7 @@ export function createLauncher({
|
||||
} catch (e) {
|
||||
throw new CliError(`${e.message}; check which sessions still run, then move the file aside`, 3);
|
||||
}
|
||||
const registry = new Registry(dataRoot);
|
||||
const registry = new Registry(dataRoot, { readOnly: !launches });
|
||||
const children = new Map();
|
||||
let chain = Promise.resolve();
|
||||
let closed = false;
|
||||
@@ -323,6 +328,8 @@ export function createLauncher({
|
||||
|
||||
// Bundle, spawn, bind, register, hand over the capability.
|
||||
async function start({ instance, launchedBy, resolved, model, account = null, thread = null }) {
|
||||
// Every launch path ends here, so a --no-launch host spawns nothing whichever path asked.
|
||||
if (!launches) throw new Refusal("launches-disabled", "this host was started with --no-launch");
|
||||
// A thread's store is checked first, so a bad id or an unsafe directory
|
||||
// launches nothing.
|
||||
let store = null;
|
||||
@@ -627,6 +634,11 @@ export function createLauncher({
|
||||
await host.op({ op: "refuse", cap, code: "not-a-role" }).catch(() => {});
|
||||
return { ok: false, error: "not-a-role" };
|
||||
}
|
||||
if (!launches) {
|
||||
await host.op({ op: "refuse", cap, code: "launches-disabled" }).catch(() => {});
|
||||
record({ event: "refused", by: who.role, instance, code: "launches-disabled" });
|
||||
return { ok: false, error: "launches-disabled" };
|
||||
}
|
||||
let checked;
|
||||
try {
|
||||
if (typeof instance !== "string") throw new Refusal("invalid-request");
|
||||
@@ -675,6 +687,13 @@ export function createLauncher({
|
||||
// This broker never bound those runs, so each is bound again first with
|
||||
// its identical launch record, which the broker accepts.
|
||||
async function recover() {
|
||||
// Row 163 (mos-0961): a --no-launch host never signals or ends an earlier host's session. Its
|
||||
// data root may hold a sessions.json that names live processes, so it only says what it left.
|
||||
if (!launches) {
|
||||
for (const s of leftover) log(`run ${s.run} (${s.instance}) from an earlier host: left as it is, this host was started with --no-launch`);
|
||||
leftover = [];
|
||||
return;
|
||||
}
|
||||
for (const s of leftover) {
|
||||
if (startTimeOf(s.pid) === s.startTime && (cmdlineOf(s.pid) ?? [])[0] === bwrap) {
|
||||
try {
|
||||
@@ -744,7 +763,7 @@ export function createLauncher({
|
||||
});
|
||||
chmodSync(path, 0o600);
|
||||
await gitSocket.listen();
|
||||
if (b.launch) ticker = setInterval(() => tick().catch((e) => log(`thread tick: ${e.message}`)), threadTickMs);
|
||||
if (b.launch && launches) ticker = setInterval(() => tick().catch((e) => log(`thread tick: ${e.message}`)), threadTickMs);
|
||||
}
|
||||
|
||||
// SIGTERM every runner, wait, then end what's left. Runs before the host
|
||||
|
||||
@@ -811,6 +811,29 @@ test("bus start runs until bus stop; status reports it while it runs", async (t)
|
||||
assert.match(again.stdout, /no host runs/);
|
||||
});
|
||||
|
||||
// Row 163 (mos-0961): the D2 rehearsal's host.
|
||||
test("bus start --no-launch says launches are disabled in its host-up line and stops cleanly; with --pm it is a usage error", async (t) => {
|
||||
const f = fixture(tmp(t));
|
||||
notifyConfig(f.dataRoot, "acme", null);
|
||||
const both = spawnSync(process.execPath, [CLI, "bus", "start", "acme", "--pm", "--no-launch"], { env: f.env, encoding: "utf8", timeout: 40000 });
|
||||
assert.equal(both.status, 4, both.stderr);
|
||||
assert.match(both.stderr, /bus start takes --pm or --no-launch, not both/);
|
||||
assert.equal(existsSync(join(f.dataRoot, "bus", "writer.lock")), false);
|
||||
const child = spawn(process.execPath, [CLI, "bus", "start", "acme", "--no-launch"], { env: f.env, stdio: ["ignore", "pipe", "pipe"] });
|
||||
t.after(() => child.exitCode === null && child.kill("SIGKILL"));
|
||||
let out = "";
|
||||
child.stdout.on("data", (b) => (out += b));
|
||||
child.stderr.on("data", (b) => (out += b));
|
||||
await until(() => /bus host up: business acme.*\n/.test(out), 30000);
|
||||
assert.match(out, /bus host up: business acme, socket \S+, notifier off, launches disabled\n/);
|
||||
// The launcher got launches false: its Registry is read-only, so no sessions.json appears.
|
||||
assert.equal(existsSync(join(f.dataRoot, "bus-host", "sessions.json")), false);
|
||||
const exit = once(child, "exit");
|
||||
const stop = spawnSync(process.execPath, [CLI, "bus", "stop"], { env: f.env, encoding: "utf8", timeout: 70000 });
|
||||
assert.equal(stop.status, 0, stop.stderr);
|
||||
assert.equal((await exit)[0], 0, out);
|
||||
});
|
||||
|
||||
test("bus-service.sh renders the unit and installs it into a given directory", (t) => {
|
||||
const dir = tmp(t);
|
||||
const script = join(REPO, "scripts", "bus-service.sh");
|
||||
@@ -818,6 +841,8 @@ test("bus-service.sh renders the unit and installs it into a given directory", (
|
||||
assert.equal(render.status, 0, render.stderr);
|
||||
assert.match(render.stdout, new RegExp(`ExecStart=${REPO.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}/scripts/mosaic bus start %i`));
|
||||
assert.match(render.stdout, /RestartPreventExitStatus=2 3 4/);
|
||||
// Row 163: --no-launch is for the D2 rehearsal's own host only; no unit starts a bus with it.
|
||||
assert.doesNotMatch(render.stdout, /--no-launch/);
|
||||
// One session credential file, optional so Pi sessions start without it.
|
||||
// The README's paragraph names the same file; nothing else does.
|
||||
const files = render.stdout.match(/^EnvironmentFile=.*$/gm);
|
||||
|
||||
@@ -15,11 +15,12 @@ import { chmodSync, existsSync, mkdirSync, readFileSync, readdirSync, renameSync
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { Client } from "../../bus/src/client.mjs";
|
||||
import { gitSocketPath, launchLogFile, launchSocketPath, readSessions, Registry, SESSION_CREDENTIALS, sessionsFile, workspaceDir } from "../../seat/src/session.mjs";
|
||||
import { sandboxProblem } from "../../seat/src/sandbox.mjs";
|
||||
import { gitSocketPath, launchLogFile, launchSocketPath, readSessions, Registry, runsDir, SESSION_CREDENTIALS, sessionsFile, workspaceDir } from "../../seat/src/session.mjs";
|
||||
import { BWRAP, sandboxProblem } from "../../seat/src/sandbox.mjs";
|
||||
import { childrenOf } from "../../seat/src/proc.mjs";
|
||||
import { gitEnv } from "../../seat/src/checkout.mjs";
|
||||
import { bootConfig, loadSystem } from "../src/config.mjs";
|
||||
import { startHost, startTimeOf } from "../src/host.mjs";
|
||||
import { cmdlineOf, startHost, startTimeOf } from "../src/host.mjs";
|
||||
import { createLauncher } from "../src/launcher.mjs";
|
||||
import { mockAnthropic, REPO } from "../../harness/tests/helpers.mjs";
|
||||
import { fakeHarnesses, fixture, tmp } from "./helpers.mjs";
|
||||
@@ -901,6 +902,56 @@ for (const exited of [false, true]) {
|
||||
});
|
||||
}
|
||||
|
||||
// Row 163 (mos-0961): the D2 rehearsal host. A live bwrap process the test owns stands in for an
|
||||
// earlier host's session; its outer bwrap is what recover() would kill, and its child stands in
|
||||
// for the runner that close() would signal.
|
||||
test("bus start --no-launch leaves an earlier host's live session alone: no signal, no end, sessions.json as it was", { skip, timeout: 30000 }, async (t) => {
|
||||
const given = prepare(t);
|
||||
const { f } = given;
|
||||
const box = spawn(BWRAP, ["--ro-bind", "/", "/", "--dev", "/dev", "--proc", "/proc", "--unshare-pid", "--die-with-parent", "sleep", "60"], { stdio: "ignore" });
|
||||
t.after(() => box.kill("SIGKILL"));
|
||||
const pid = box.pid;
|
||||
const startTime = await until(() => startTimeOf(pid));
|
||||
const runnerPid = await until(() => childrenOf(pid)[0]);
|
||||
const runnerStartTime = await until(() => startTimeOf(runnerPid));
|
||||
assert.equal(cmdlineOf(pid)[0], BWRAP, "recover() would take it for a session");
|
||||
new Registry(f.dataRoot).add({ business: "acme", instance: "chief", run: "run-left", harness: "pi", pid, startTime, runnerPid, runnerStartTime });
|
||||
const before = readFileSync(sessionsFile(f.dataRoot), "utf8");
|
||||
|
||||
const s = await setup(t, undefined, given, { launches: false });
|
||||
assert.ok(s.logs.includes("run run-left (chief) from an earlier host: left as it is, this host was started with --no-launch"), s.logs.join("\n"));
|
||||
assert.equal(startTimeOf(pid), startTime, "listen() sent no signal");
|
||||
assert.equal(readFileSync(sessionsFile(f.dataRoot), "utf8"), before);
|
||||
assert.deepEqual((await s.client.call("trail", { subject: "run-left" })).filter((r) => r.kind === "session.ended"), []);
|
||||
assert.equal(await s.close(), 0);
|
||||
assert.equal(startTimeOf(pid), startTime, "close() sent no signal either");
|
||||
assert.equal(startTimeOf(runnerPid), runnerStartTime);
|
||||
assert.equal(readFileSync(sessionsFile(f.dataRoot), "utf8"), before);
|
||||
});
|
||||
|
||||
test("bus start --no-launch refuses every launch: the PM launch and a session's request, each logged, nothing spawned", { skip, timeout: 30000 }, async (t) => {
|
||||
const s = await setup(t, undefined, undefined, { launches: false });
|
||||
await assert.rejects(s.launcher.launchPm(), (e) => e.exitCode === 3 && /^PM launch refused: launches-disabled/.test(e.message));
|
||||
const sock = launchSocketPath(s.f.dataRoot);
|
||||
assert.deepEqual(await ask(sock, `${JSON.stringify({ cap: s.cto.cap, instance: "coder" })}\n`), { ok: false, error: "launches-disabled" });
|
||||
assert.deepEqual(log(s.f).map((e) => [e.event, e.instance, e.code]), [["refused", "chief", "launches-disabled"], ["refused", "coder", "launches-disabled"]]);
|
||||
assert.deepEqual(readSessions(s.f.dataRoot), []);
|
||||
const runs = runsDir(s.f.dataRoot, "acme");
|
||||
assert.deepEqual(existsSync(runs) ? readdirSync(runs) : [], [], "no run directory");
|
||||
assert.equal(await s.close(), 0);
|
||||
});
|
||||
|
||||
test("bus start --no-launch starts no thread tick; the same host with launches runs it", { skip, timeout: 30000 }, async (t) => {
|
||||
for (const launches of [false, true]) {
|
||||
const asked = [];
|
||||
const wrap = (host) => Object.create(host, { op: { value: async (m) => (["threadsState", "landHolders"].includes(m.op) && asked.push(m.op), host.op(m)) } });
|
||||
const s = await setup(t, undefined, undefined, { launches, threadTickMs: 50, wrap });
|
||||
await new Promise((r) => setTimeout(r, 400));
|
||||
assert.equal(asked.length > 0, launches, `launches ${launches}: ${asked.join(", ")}`);
|
||||
assert.equal(await s.close(), 0);
|
||||
}
|
||||
});
|
||||
|
||||
test("a malformed sessions.json refuses the host start with exit 3 and stays as it was", { timeout: 30000 }, async (t) => {
|
||||
const given = prepare(t);
|
||||
const file = sessionsFile(given.f.dataRoot);
|
||||
|
||||
@@ -1705,3 +1705,335 @@ test('T54 row 188: a carried pair gets the proof file checks, and a proof file i
|
||||
await noLeak(e);
|
||||
});
|
||||
});
|
||||
|
||||
// --- rehearse (D2, row 163; the brief's "Stack rehearsal on a scratch root") ---
|
||||
// Copies of their own in RCOPIES: the real scripts/mosaic, config loader, business loader, roles and
|
||||
// mosaic-config.mjs, with stand-ins for the bus host (cli.mjs), the Console, the Store and
|
||||
// migrate-schema.mjs. A rehearsal starts its processes with only HOME, PATH and MOSAIC_CONFIG, so
|
||||
// the stand-ins take their faults from $HOME/faults and record what they did in $HOME.
|
||||
|
||||
// The bus host. bus start writes the store's writer.lock, a meta row in the store, and its state
|
||||
// beside the scratch store, and prints the host-up line; without --no-launch it starts the PM, which here means a line in
|
||||
// $HOME/starts.log and a sessions.json. bus status --json answers from that state.
|
||||
function rHost(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
|
||||
const id=path.basename(path.resolve(import.meta.dirname,'..','..','..')),home=process.env.HOME,a=process.argv.slice(2);
|
||||
const fault=(n)=>fs.existsSync(path.join(home,'faults',n));
|
||||
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
|
||||
const data=JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8')).dataRoot,state=path.join(data,'fixture-host.json');
|
||||
const flags=["--pm","--no-launch"];
|
||||
if(a.join(' ')==='bus status --json'){
|
||||
if(fault(`status-${id}`))process.exit(1);
|
||||
let s={host:null,boot:null};try{s=JSON.parse(fs.readFileSync(state,'utf8'));s.host.live=alive(s.host.pid)}catch{}
|
||||
if(fault('status-other-host')&&s.host)s.host.pid=process.ppid;
|
||||
console.log(JSON.stringify(s));process.exit(0);
|
||||
}
|
||||
if(a[0]!=='bus'||a[1]!=='start'||!a.slice(3).every((f)=>flags.includes(f))){console.error(`fixture host: ${a.join(' ')}`);process.exit(4)}
|
||||
fs.appendFileSync(path.join(home,'host-args.log'),`${a.join(' ')}\n`);
|
||||
if(fault(`host-fails-${id}`)){console.error('fixture host: refused');process.exit(1)}
|
||||
const b=a[2],off=a.includes('--no-launch'),lock=path.join(data,'bus','writer.lock'),at=new Date().toISOString();
|
||||
fs.writeFileSync(lock,String(process.pid),{flag:'wx'});
|
||||
const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(data,'bus','bus.sqlite'));
|
||||
db.prepare("INSERT OR REPLACE INTO meta(key,value) VALUES('fixture_host_start',?)").run(at);db.close();
|
||||
fs.writeFileSync(state,JSON.stringify({host:{business:b,pid:process.pid,live:true,startedAt:at,notifier:'off'},boot:{ok:true,business:b,at}}));
|
||||
if(!off||fault('host-launches')){
|
||||
fs.appendFileSync(path.join(home,'starts.log'),`pm-${b}\n`);
|
||||
fs.mkdirSync(path.join(data,'bus-host'),{recursive:true});
|
||||
fs.writeFileSync(path.join(data,'bus-host','sessions.json'),JSON.stringify({sessionsVersion:1,sessions:[{pid:process.pid}]}));
|
||||
}
|
||||
console.log(`bus host up: business ${b}, socket ${path.join(data,'bus','broker.sock')}, notifier off${off?', launches disabled':''}`);
|
||||
process.on('SIGTERM',()=>{if(fault('host-ignores-term'))return;if(!fault('host-keeps-lock'))fs.rmSync(lock,{force:true});process.exit(0)});
|
||||
setInterval(()=>{},1<<30);
|
||||
}
|
||||
|
||||
// The Console: serve on the given loopback port; /healthz answers 200 unless healthz-<copy> is set.
|
||||
function rConsole(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),http=process.getBuiltinModule('node:http');
|
||||
const id=path.basename(path.resolve(import.meta.dirname,'..','..','..')),home=process.env.HOME,a=process.argv.slice(2);
|
||||
const fault=(n)=>fs.existsSync(path.join(home,'faults',n));
|
||||
fs.appendFileSync(path.join(home,'console-args.log'),`${a.join(' ')}\n`);
|
||||
if(a[0]!=='serve'){console.error('fixture Console: refused');process.exit(1)}
|
||||
const s=http.createServer((q,r)=>{r.writeHead(q.url==='/healthz'&&!fault(`healthz-${id}`)?200:500);r.end()});
|
||||
s.listen(Number(a[a.indexOf('--port')+1]),'127.0.0.1',()=>console.log(`Mosaic Console: http://127.0.0.1:${s.address().port}/\nBoard: fixture`));
|
||||
process.on('SIGTERM',()=>process.exit(0));
|
||||
}
|
||||
|
||||
// migrate-schema.mjs: sets its copy's SCHEMA_VERSION, or exit 1 under migrate-fails.
|
||||
function rMigrate(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
|
||||
const home=process.env.HOME,data=process.argv[2];
|
||||
const v=/^export const SCHEMA_VERSION = '([A-Za-z0-9]+)';$/m.exec(fs.readFileSync(path.join(import.meta.dirname,'store.mjs'),'utf8'))[1];
|
||||
fs.appendFileSync(path.join(home,'migrate.log'),`${data}\n`);
|
||||
if(fs.existsSync(path.join(home,'faults','migrate-fails'))){console.error('fixture migrate: failed');process.exit(1)}
|
||||
const db=new DatabaseSync(path.join(data,'bus','bus.sqlite'));db.prepare("UPDATE meta SET value=? WHERE key='schema_version'").run(v);db.close();
|
||||
console.log(`bus schema ${v} ready`);
|
||||
}
|
||||
|
||||
// The copy's Store: refuses a store at another version with schema-mismatch, as the real one does.
|
||||
const rStore=(v,froms)=>`${storeText(v,froms)}import {DatabaseSync} from 'node:sqlite';
|
||||
import {join} from 'node:path';
|
||||
export class Store {
|
||||
constructor(root) {
|
||||
this.db = new DatabaseSync(join(root, 'bus', 'bus.sqlite'));
|
||||
if (this.db.prepare("SELECT value FROM meta WHERE key='schema_version'").get()?.value !== SCHEMA_VERSION) { this.db.close(); throw Error('schema-mismatch'); }
|
||||
}
|
||||
get(sql, ...args) { return this.db.prepare(sql).get(...args); }
|
||||
close() { this.db.close(); }
|
||||
}
|
||||
`;
|
||||
|
||||
// B (3g) and C (3h) have --no-launch. A is a copy from before row 163, whose host has no such flag.
|
||||
// C1's scripts/mosaic runs B's host, as a copy-check failure needs.
|
||||
const RCOPIES=join(base,'rcopies');mkdirSync(RCOPIES);
|
||||
for(const [id,v,froms] of [[B,'3g',['3f']],[C,'3h',['3f','3g']],[A,'3g',['3f']],[C1,'3g',['3f']]]){
|
||||
const d=join(RCOPIES,id);
|
||||
for(const p of ['scripts/mosaic','scripts/mosaic-config.mjs','packages/cli/src/config.mjs','packages/cli/src/errors.mjs','packages/bus/src/business.mjs','packages/plugins/src/kinds.mjs','packages/business/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true});
|
||||
if(id===C1)writeFileSync(join(d,'scripts','mosaic'),`#!/usr/bin/env bash\nexec node "${join(RCOPIES,B,'packages','cli','src','cli.mjs')}" "$@"\n`,{mode:0o755});
|
||||
for(const [f,text] of [['packages/cli/src/cli.mjs',id===A?'// fixture host from before row 163\nprocess.exit(1);\n':script(rHost)],['packages/webui/src/cli.mjs',script(rConsole)],['packages/bus/src/store.mjs',rStore(v,froms)],['packages/bus/src/migrate-schema.mjs',script(rMigrate)],['packages/bus/src/broker.mjs','export class BusError extends Error { constructor(code) { super(code); this.code = code; } }\n']]){mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text)}
|
||||
execFileSync('chmod',['-R','a-w',d]);
|
||||
}
|
||||
|
||||
const TESTER='7499';
|
||||
const freePort=async()=>{const s=createServer();s.listen(0,'127.0.0.1');await once(s,'listening');const p=String(s.address().port);await new Promise((r)=>s.close(r));return p};
|
||||
const rh=(e,...p)=>join(e.env.HOME,...p);
|
||||
const rfault=(e,name)=>{mkdirSync(rh(e,'faults'),{recursive:true});writeFileSync(rh(e,'faults',name),'')};
|
||||
|
||||
// A root for one rehearsal: a stopped live store at `store`, a pointer by hand on `copy` with the
|
||||
// Console on e.port and the tester listener on TESTER, and a backup as backup_store writes one.
|
||||
// A rehearsal's leftovers, found by their scratch MOSAIC_CONFIG, are killed after the test.
|
||||
async function rmk(t,{copy=B,store='3g'}={}){
|
||||
const e=await mk(t);
|
||||
t.after(()=>{
|
||||
for(const pid of readdirSync('/proc').filter((n)=>/^\d+$/.test(n)))
|
||||
try{if(readFileSync(`/proc/${pid}/environ`,'utf8').includes(`MOSAIC_CONFIG=${e.data}/rehearsals/`))process.kill(Number(pid),'SIGKILL')}catch{}
|
||||
});
|
||||
const db=new DatabaseSync(e.store);
|
||||
db.exec('PRAGMA journal_mode=WAL');
|
||||
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
|
||||
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
|
||||
db.prepare("INSERT INTO meta VALUES ('schema_version', ?)").run(store);
|
||||
for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`);
|
||||
db.close();
|
||||
mkdirSync(e.state,{recursive:true});
|
||||
writeFileSync(e.pointer,JSON.stringify({copy,business:'acme',pm:true,claude:e.claude,console:['--port',e.port,'--business','acme','--tester-port',TESTER],schema:store,activation:'manual',at:'2026-10-20T00:00:00Z'})+'\n');
|
||||
e.act=`20261020T000000Z-${copy.slice(0,12)}`;e.backup=join(e.data,'backups','stack',e.act);
|
||||
mkdirSync(join(e.backup,'bus'),{recursive:true,mode:0o700});
|
||||
cpSync(e.store,join(e.backup,'bus','bus.sqlite'));
|
||||
writeFileSync(join(e.backup,'SHA256SUMS'),`${sha(e.store)} bus.sqlite\n`);
|
||||
for(const [f,text] of [['DATA_ROOT',e.data],['SCHEMA',store],['STAMP','2026-10-20T00:00:00.000Z']])writeFileSync(join(e.backup,f),`${text}\n`);
|
||||
e.biz=join(dirname(e.cfg),'businesses','acme.json');
|
||||
return e;
|
||||
}
|
||||
|
||||
// What a rehearsal may never change: the live store, config, business file, pointer and log, and
|
||||
// the backup, by sha256 of every file.
|
||||
const untouched=(e)=>JSON.stringify([...[e.store,e.cfg,e.biz,e.pointer].map((f)=>[f,sha(f)]),existsSync(e.log),...readdirSync(e.backup,{recursive:true}).sort().map((f)=>join(e.backup,f)).filter((f)=>statSync(f).isFile()).map((f)=>[f,sha(f)])]);
|
||||
|
||||
async function rehearse(e,id,...args){
|
||||
const ch=spawn('bash',[e.sr??SR,'rehearse',id,'acme',...args,'--copies',RCOPIES,'--bin',e.bin],{env:e.env,stdio:['ignore','pipe','pipe']});
|
||||
let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d});
|
||||
const timer=setTimeout(()=>ch.kill('SIGKILL'),120000);
|
||||
const [code]=await once(ch,'close');clearTimeout(timer);
|
||||
const m=/\nrehearsal directory: (.*)\n$/.exec(out);
|
||||
return {code,out,err,dir:m?m[1]:null,names:out.split('\n').filter((l)=>/^(PASS|FAIL) /.test(l)).map((l)=>l.replace(/^(PASS|FAIL) ([^:]*).*/,'$1 $2'))};
|
||||
}
|
||||
// A script of its own whose text has `from` replaced, for the guard tests. The replacement is a
|
||||
// function so a shell "$$" in `to` stays as written.
|
||||
function ownScript(e,from,to){
|
||||
const own=join(e.root,'repo');cpSync(repo,own,{recursive:true});
|
||||
e.sr=join(own,'scripts','stack-release.sh');
|
||||
const s=readFileSync(e.sr,'utf8');assert.ok(s.includes(from),`${e.sr} has no ${from}`);writeFileSync(e.sr,s.replace(from,()=>to));
|
||||
}
|
||||
const rDirs=(e)=>existsSync(join(e.data,'rehearsals'))?readdirSync(join(e.data,'rehearsals')):[];
|
||||
const PRE=['PASS copy','PASS port','PASS backup','PASS scratch-root','PASS data-root-live','PASS sessions-live','PASS data-root-scratch','PASS notify-off','PASS tracker-configured','PASS credential-ref-live'];
|
||||
const POST=['PASS console-stopped','PASS host-stopped','PASS host-released-store','PASS backup-unchanged','PASS live-config-unchanged'];
|
||||
|
||||
test('R1 rehearse passes on the same schema; the scratch root holds its files and the live files stay byte-identical',async(t)=>{
|
||||
const e=await rmk(t),before=untouched(e),port=await freePort();
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',port,'--timeout','10');
|
||||
assert.equal(r.code,0,r.out+r.err);
|
||||
assert.deepEqual(r.names,[...PRE,'PASS migration','PASS gate','PASS no-session',...POST]);
|
||||
assert.ok(r.out.endsWith(`REHEARSAL_PASS\nrehearsal directory: ${r.dir}\n`),r.out);
|
||||
assert.equal(r.err,'');
|
||||
assert.equal(untouched(e),before);
|
||||
const R=r.dir;assert.equal(dirname(R),join(e.data,'rehearsals'));assert.match(basename(R),new RegExp(`^\\d{8}T\\d{6}Z-${B.slice(0,12)}$`));
|
||||
for(const d of [dirname(R),R,join(R,'placeholders'),join(R,'config')])assert.equal(statSync(d).mode&0o777,0o700,d);
|
||||
for(const f of ['steps.log','backup.before','live.before','business.diff','host.log','console.log','config/config.json','data/bus/bus.sqlite'])assert.ok(existsSync(join(R,f)),f);
|
||||
// The host wrote the scratch store, a copy: the backup has no such row.
|
||||
const metaOf=(f)=>{const db=new DatabaseSync(f,{readOnly:true});try{return db.prepare("SELECT value FROM meta WHERE key='fixture_host_start'").get()?.value}finally{db.close()}};
|
||||
assert.match(metaOf(join(R,'data','bus','bus.sqlite')),/^\d{4}-/);assert.equal(metaOf(join(e.backup,'bus','bus.sqlite')),undefined);
|
||||
// The scratch config differs from the live one only in dataRoot.
|
||||
assert.deepEqual(JSON.parse(readFileSync(join(R,'config','config.json'),'utf8')),{...JSON.parse(readFileSync(e.cfg,'utf8')),dataRoot:join(R,'data')});
|
||||
// The notifier config: binding null at 0600.
|
||||
const nf=join(R,'data','notify','acme','notify.json');
|
||||
assert.deepEqual(JSON.parse(readFileSync(nf,'utf8')),{notifyVersion:1,binding:null});assert.equal(statSync(nf).mode&0o777,0o600);
|
||||
// The scratch business: no tracker.* variable, and every reference a 0600 placeholder with its date kept.
|
||||
const live=JSON.parse(readFileSync(e.biz,'utf8')),sb=JSON.parse(readFileSync(join(R,'config','businesses','acme.json'),'utf8'));
|
||||
assert.ok(Object.keys(live.vars).some((k)=>k.startsWith('tracker.')));
|
||||
assert.deepEqual(Object.keys(sb.vars).filter((k)=>k.startsWith('tracker.')),[]);
|
||||
const refs=[...Object.entries(sb.roles).flatMap(([n,r])=>Object.entries(r.credentials??{}).map(([s,ref])=>[`${n}-${s}`,ref,live.roles[n].credentials[s]])),...Object.entries(sb.tracker.sync.credentials).map(([s,ref])=>[`@sync-${s}`,ref,live.tracker.sync.credentials[s]])];
|
||||
assert.ok(refs.length>=2);
|
||||
for(const [name,ref,old] of refs){
|
||||
const f=join(R,'placeholders',name);
|
||||
assert.deepEqual(ref,{...old,file:f});assert.equal(readFileSync(f,'utf8'),'rehearsal-placeholder-token');assert.equal(statSync(f).mode&0o777,0o600);
|
||||
}
|
||||
assert.deepEqual(readdirSync(join(R,'placeholders')).sort(),refs.map(([n])=>n).sort());
|
||||
// Everything else in the business is as it was.
|
||||
const strip=(b)=>{const c=structuredClone(b);for(const k of Object.keys(c.vars))if(k.startsWith('tracker.'))delete c.vars[k];for(const r of Object.values(c.roles))delete r.credentials;delete c.tracker.sync.credentials;return c};
|
||||
assert.deepEqual(strip(sb),strip(live));
|
||||
const diff=readFileSync(join(R,'business.diff'),'utf8');assert.match(diff,/^-.*"tracker\.baseUrl"/m);assert.match(diff,/^\+.*placeholders\/pm-/m);
|
||||
// The host ran with launches disabled and started no PM; the Console got no --send.
|
||||
assert.deepEqual(lines(rh(e,'host-args.log')),['bus start acme --no-launch']);
|
||||
assert.deepEqual(lines(rh(e,'console-args.log')),[`serve --port ${port} --business acme`]);
|
||||
assert.equal(existsSync(rh(e,'starts.log')),false);
|
||||
assert.equal(existsSync(join(R,'data','bus','writer.lock')),false);
|
||||
assert.deepEqual(lines(join(R,'steps.log')).map((l)=>l.replace(/^\S+ /,'')),r.out.split('\n').filter((l)=>/^(PASS|FAIL) |^REHEARSAL_/.test(l)));
|
||||
});
|
||||
|
||||
test('R2 rehearse runs the migration checks when the schema differs; the live store stays at 3g',async(t)=>{
|
||||
const e=await rmk(t),before=untouched(e),port=await freePort();
|
||||
const r=await rehearse(e,C,'--backup',e.backup,'--port',port,'--timeout','10');
|
||||
assert.equal(r.code,0,r.out+r.err);
|
||||
assert.deepEqual(r.names,[...PRE,'PASS new-refuses-old-store','PASS migrate','PASS new-opens-migrated','PASS migrate-again','PASS old-refuses-migrated','PASS gate','PASS no-session',...POST]);
|
||||
assert.ok(r.out.includes('PASS migrate: 3g to 3h\n'),r.out);
|
||||
assert.equal(untouched(e),before);assert.equal(storeVersion(e.store,false),'3g');
|
||||
assert.equal(storeVersion(join(r.dir,'data','bus','bus.sqlite'),false),'3h');
|
||||
assert.equal(storeVersion(join(r.dir,'refuse','bus','bus.sqlite'),false),'3g');
|
||||
assert.deepEqual(lines(rh(e,'migrate.log')),[join(r.dir,'data'),join(r.dir,'data')]);
|
||||
});
|
||||
|
||||
test('R3 a failing migration gives REHEARSAL_FAIL migrate and starts nothing',async(t)=>{
|
||||
const e=await rmk(t),before=untouched(e);rfault(e,'migrate-fails');
|
||||
const r=await rehearse(e,C,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.deepEqual(r.names,[...PRE,'PASS new-refuses-old-store','FAIL migrate','PASS backup-unchanged','PASS live-config-unchanged']);
|
||||
assert.ok(r.out.includes('FAIL migrate: migrate-schema.mjs exit 1: fixture migrate: failed\n'),r.out);
|
||||
assert.ok(r.out.endsWith(`REHEARSAL_FAIL\nrehearsal directory: ${r.dir}\n`),r.out);
|
||||
assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false);
|
||||
assert.equal(untouched(e),before);
|
||||
});
|
||||
|
||||
test('R4 a host that does not start, does not answer status, reports another pid, launches anyway or keeps its lock each fails by name',async(t)=>{
|
||||
for(const [f,name,why] of [[`host-fails-${B}`,'host-up','exited'],[`status-${B}`,'bus-status','bus status failed'],['status-other-host','bus-status','the host this run started'],['host-launches','no-session','bus-host/sessions.json'],['host-keeps-lock','host-released-store','writer.lock is still there']]){
|
||||
const e=await rmk(t),before=untouched(e);rfault(e,f);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,f+r.out+r.err);
|
||||
assert.ok(r.names.includes(`FAIL ${name}`),f+r.out);assert.equal(r.names.filter((n)=>n.startsWith('FAIL')).length,1,f+r.out);
|
||||
assert.ok(r.out.split('\n').some((l)=>l.startsWith(`FAIL ${name}: `)&&l.includes(why)),f+r.out);
|
||||
assert.ok(r.names.includes('PASS console-stopped')&&r.names.includes('PASS backup-unchanged'),f+r.out);
|
||||
assert.ok(r.out.endsWith(`REHEARSAL_FAIL\nrehearsal directory: ${r.dir}\n`),r.out);
|
||||
assert.equal(untouched(e),before);
|
||||
}
|
||||
});
|
||||
|
||||
test('R5 a host that ignores SIGTERM gets SIGKILL and FAIL host-stopped',async(t)=>{
|
||||
const e=await rmk(t);rfault(e,'host-ignores-term');
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','2');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.ok(/^FAIL host-stopped: pid \d+ was still running 2s after SIGTERM; sent SIGKILL$/m.test(r.out),r.out);
|
||||
});
|
||||
|
||||
test('R6 a Console that does not answer /healthz gives REHEARSAL_FAIL healthz',async(t)=>{
|
||||
const e=await rmk(t);rfault(e,`healthz-${B}`);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.ok(r.out.includes('FAIL healthz: no answer within 3s\n'),r.out);
|
||||
assert.deepEqual(r.names.filter((n)=>n.startsWith('FAIL')),['FAIL healthz']);
|
||||
});
|
||||
|
||||
test('R7 a host on another copy fails copy-match',async(t)=>{
|
||||
const e=await rmk(t);
|
||||
const r=await rehearse(e,C1,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.ok(r.out.includes(`FAIL copy-match: the host's command line is not ${join(RCOPIES,C1)}'s cli.mjs bus start acme --no-launch\n`),r.out);
|
||||
});
|
||||
|
||||
test('R8 a scratch port in use or reserved refuses, and the rehearsal takes no other port',async(t)=>{
|
||||
const e=await rmk(t),before=untouched(e);
|
||||
const held=createServer();held.listen(0,'127.0.0.1');await once(held,'listening');t.after(()=>held.close());
|
||||
const p=String(held.address().port);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',p);
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.equal(r.out,`PASS copy: ${B}, schema 3g\nFAIL port-in-use: 127.0.0.1:${p} is in use; the rehearsal takes no other port\nREHEARSAL_FAIL\n`);
|
||||
for(const q of ['7332','7333',e.port,TESTER]){
|
||||
const s=await rehearse(e,B,'--backup',e.act,'--port',q);
|
||||
assert.equal(s.code,1,s.out+s.err);
|
||||
assert.equal(s.out,`PASS copy: ${B}, schema 3g\nFAIL port-reserved: --port ${q} is a live Console port (7332, 7333 and the pointer's: ${e.port} ${TESTER})\nREHEARSAL_FAIL\n`);
|
||||
}
|
||||
assert.deepEqual(rDirs(e),[]);assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false);
|
||||
assert.equal(untouched(e),before);
|
||||
});
|
||||
|
||||
test('R9 mos-0963: a role reference left outside placeholders/ fails credential-ref-live before the host starts',async(t)=>{
|
||||
const e=await rmk(t);
|
||||
// The live pm gitea reference names a file that is there, as a live token would be.
|
||||
const tok=join(e.root,'live-gitea.token');writeFileSync(tok,'live-token-value',{mode:0o600});
|
||||
const b=JSON.parse(readFileSync(e.biz,'utf8'));b.roles.pm.credentials.gitea.file=tok;writeFileSync(e.biz,JSON.stringify(b,null,2)+'\n',{mode:0o600});
|
||||
const before=untouched(e);
|
||||
ownScript(e,'for(const [name,r] of Object.entries(b.roles??{})){strip(r?.vars);swap(name,r?.credentials)}','for(const [name,r] of Object.entries(b.roles??{})){strip(r?.vars);if(name!=="pm")swap(name,r?.credentials)}');
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.ok(r.out.includes(`FAIL credential-ref-live: pm gitea is ${tok}, not a file in ${join(r.dir,'placeholders')}\n`),r.out);
|
||||
assert.deepEqual(r.names.slice(-3),['FAIL credential-ref-live','PASS backup-unchanged','PASS live-config-unchanged']);
|
||||
assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false);
|
||||
assert.equal(untouched(e),before);
|
||||
});
|
||||
|
||||
test('R10 mos-0962: a scratch business that still sets a tracker fails tracker-configured',async(t)=>{
|
||||
const e=await rmk(t);
|
||||
const proj=join(e.root,'project');mkdirSync(join(proj,'.mosaic'),{recursive:true});
|
||||
writeFileSync(join(proj,'.mosaic','project.json'),JSON.stringify({projectVersion:1,id:'stack',vars:{'tracker.project':7}}),{mode:0o600});
|
||||
const b=JSON.parse(readFileSync(e.biz,'utf8'));b.projects.stack.root=proj;writeFileSync(e.biz,JSON.stringify(b,null,2)+'\n',{mode:0o600});
|
||||
ownScript(e,'if(k.startsWith("tracker."))delete vars[k]','if(k.startsWith("tracker-kept."))delete vars[k]');
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.ok(r.out.includes('FAIL tracker-configured: bootConfig sets trackers for acme\n'),r.out);
|
||||
assert.equal(existsSync(rh(e,'host-args.log')),false);
|
||||
});
|
||||
|
||||
test('R11 mos-0961: a scratch config on the live data root, or a sessions.json naming a running pid, refuses by name',async(t)=>{
|
||||
// mosaic-config already refuses the live path itself, which contains the scratch config directory;
|
||||
// the guard catches a path that reaches the live data root through a link.
|
||||
const e=await rmk(t),link=join(e.root,'live-link');symlinkSync(e.data,link);
|
||||
ownScript(e,'c.dataRoot=data;',`c.dataRoot=${JSON.stringify(link)};`);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.ok(r.out.includes(`FAIL data-root-live: the scratch config names dataRoot ${link}, which resolves to the live data root ${e.data}\n`),r.out);
|
||||
assert.equal(existsSync(rh(e,'host-args.log')),false);
|
||||
const f=await rmk(t);
|
||||
ownScript(f,`printf '{"notifyVersion":1,"binding":null}\\n' > "$R/data/notify/$BUSINESS/notify.json"`,`printf '{"notifyVersion":1,"binding":null}\\n' > "$R/data/notify/$BUSINESS/notify.json" && mkdir "$R/data/bus-host" && printf '{"sessionsVersion":1,"sessions":[{"pid":%s}]}\\n' "$$" > "$R/data/bus-host/sessions.json"`);
|
||||
const s=await rehearse(f,B,'--backup',f.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(s.code,1,s.out+s.err);
|
||||
assert.ok(/^FAIL sessions-live: \S+\/bus-host\/sessions\.json names running pid\(s\) \d+$/m.test(s.out),s.out);
|
||||
// The file is left as it was: the rehearsal never deletes it and retries.
|
||||
assert.ok(existsSync(join(s.dir,'data','bus-host','sessions.json')));
|
||||
assert.equal(existsSync(rh(f,'host-args.log')),false);
|
||||
});
|
||||
|
||||
test('R12 mos-0962: a notifier config with a binding fails notify-off',async(t)=>{
|
||||
const e=await rmk(t);
|
||||
ownScript(e,`printf '{"notifyVersion":1,"binding":null}\\n'`,`printf '{"notifyVersion":1,"binding":"discord"}\\n'`);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.ok(r.out.includes('FAIL notify-off: '),r.out);
|
||||
assert.equal(existsSync(rh(e,'host-args.log')),false);
|
||||
});
|
||||
|
||||
test('R13 the pre-checks: an old copy, a backup that fails its sums, a missing backup or pointer, and usage',async(t)=>{
|
||||
const e=await rmk(t),before=untouched(e),port=await freePort();
|
||||
const old=await rehearse(e,A,'--backup',e.act,'--port',port);
|
||||
assert.equal(old.code,1);assert.equal(old.out,`FAIL copy-without-no-launch: ${A}'s bus host has no --no-launch flag\nREHEARSAL_FAIL\n`);
|
||||
const none=await rehearse(e,B,'--backup','20261020T000000Z-000000000000','--port',port);
|
||||
assert.equal(none.code,1);assert.ok(none.out.includes(`FAIL backup-missing: ${join(e.data,'backups','stack','20261020T000000Z-000000000000')} has no bus/bus.sqlite and SHA256SUMS\n`),none.out);
|
||||
const f=await rmk(t);appendFileSync(join(f.backup,'bus','bus.sqlite'),'x');
|
||||
const bad=await rehearse(f,B,'--backup',f.act,'--port',port);
|
||||
assert.equal(bad.code,1);assert.ok(bad.out.includes(`FAIL backup-sums: ${f.backup}/bus does not match its SHA256SUMS\n`),bad.out);
|
||||
const g=await rmk(t);rmSync(g.pointer);
|
||||
const np=await rehearse(g,B,'--backup',g.act,'--port',port);
|
||||
assert.equal(np.code,1);assert.match(np.out,/^FAIL no-pointer: /);
|
||||
for(const args of [['--port',port],['--backup',e.act],['--backup',e.act,'--port','80'],['--backup',e.act,'--port','70000'],['--backup','relative/dir','--port',port],['--backup',e.act,'--port',port,'--pm'],['--backup',e.act,'--port',port,'--plan']]){
|
||||
const u=await rehearse(e,B,...args);assert.equal(u.code,4,args.join(' ')+u.out+u.err);assert.match(u.err,/^usage: /);
|
||||
}
|
||||
assert.deepEqual([...rDirs(e),...rDirs(f),...rDirs(g)],[]);
|
||||
assert.equal(untouched(e),before);
|
||||
});
|
||||
@@ -186,17 +186,22 @@ export function endReason(code, signal) {
|
||||
|
||||
// The host's view of its running sessions, kept in memory and mirrored to
|
||||
// sessions.json for `mosaic stop` and `mosaic launches list`.
|
||||
// readOnly (row 163, a --no-launch host): sessions.json stays as an earlier host left it, so a
|
||||
// later host still finds and ends those runs, and add() refuses.
|
||||
export class Registry {
|
||||
#dataRoot;
|
||||
#sessions = [];
|
||||
constructor(dataRoot) {
|
||||
#readOnly;
|
||||
constructor(dataRoot, { readOnly = false } = {}) {
|
||||
this.#dataRoot = dataRoot;
|
||||
this.#write();
|
||||
this.#readOnly = readOnly;
|
||||
if (!readOnly) this.#write();
|
||||
}
|
||||
get sessions() {
|
||||
return this.#sessions.map((s) => ({ ...s }));
|
||||
}
|
||||
add(entry) {
|
||||
if (this.#readOnly) throw new SeatError("this host keeps no sessions (--no-launch)", 2);
|
||||
this.#sessions.push({ ...entry });
|
||||
this.#write();
|
||||
}
|
||||
|
||||
@@ -133,6 +133,21 @@ test("Registry mirrors to sessions.json; readSessions marks live entries; bad fi
|
||||
assert.throws(() => readSessions(root), (e) => e instanceof SeatError && /malformed/.test(e.message));
|
||||
});
|
||||
|
||||
// Row 163: a --no-launch host's registry leaves an earlier host's sessions.json as it was.
|
||||
test("a read-only Registry never writes sessions.json and refuses add()", (t) => {
|
||||
const root = scratch(t);
|
||||
new Registry(root).add({ business: "acme", instance: "chief", run: "r1", runnerPid: process.pid, runnerStartTime: startTimeOf(process.pid) });
|
||||
const before = readFileSync(sessionsFile(root), "utf8");
|
||||
const r = new Registry(root, { readOnly: true });
|
||||
assert.equal(readFileSync(sessionsFile(root), "utf8"), before);
|
||||
assert.deepEqual(r.sessions, []);
|
||||
assert.throws(() => r.add({ business: "acme", instance: "coder", run: "r2" }), (e) => e instanceof SeatError && /keeps no sessions/.test(e.message));
|
||||
assert.equal(readFileSync(sessionsFile(root), "utf8"), before);
|
||||
const empty = scratch(t);
|
||||
new Registry(empty, { readOnly: true });
|
||||
assert.equal(existsSync(sessionsFile(empty)), false);
|
||||
});
|
||||
|
||||
// Row 121 (#1615): a thread instance runs once per thread, so the registry
|
||||
// keys a thread run by its thread too.
|
||||
test("REQ-THREAD-1 Registry.running keys a thread run by business, instance and thread", (t) => {
|
||||
|
||||
+304
-2
@@ -15,6 +15,8 @@
|
||||
# 4. this script never prunes backups, rollback sets, post-attempt directories or copies.
|
||||
# Decisions 94 and 102 still decide who starts the Console; webui-service.sh keeps having no start
|
||||
# or stop verb. status and --plan only read, and every seat may run them.
|
||||
# rehearse (D2, row 163) stops and starts no unit and writes only its own rehearsal directory, but
|
||||
# it reads the live config and a backup of the live store, so Jason or Mos runs it.
|
||||
#
|
||||
# Most functions below run as `f || handler`, and bash ignores set -e inside such a call, so
|
||||
# every step that can fail carries its own `|| return 1`.
|
||||
@@ -28,7 +30,7 @@ BIN="$HOME/.npm-global/bin"
|
||||
UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
|
||||
CONSOLE_UNIT=mosaic-webui.service
|
||||
TIMEOUT=60 WANT_PM=0 AFTER_HARNESS=0 MIGRATE=0 PLAN=0 RESTORE="" EVENTS="" TESTER_PORT=""
|
||||
PROOF_HEADER="" PROOF_FILE="" NO_PROOF=0
|
||||
PROOF_HEADER="" PROOF_FILE="" NO_PROOF=0 R_BACKUP="" R_PORT=""
|
||||
|
||||
usage() {
|
||||
cat >&2 <<'USAGE'
|
||||
@@ -38,6 +40,7 @@ usage: stack-release.sh adopt BUSINESS
|
||||
stack-release.sh pm-on BUSINESS [--timeout S]
|
||||
stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan]
|
||||
stack-release.sh status BUSINESS
|
||||
stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]
|
||||
common options: --copies DIR, --bin DIR
|
||||
--events N is the total of the per-table counts the post-backup-events refusal prints: events and
|
||||
every table in COUNTED.
|
||||
@@ -84,11 +87,14 @@ parse_opts() {
|
||||
--no-proof) [ "$OP" = activate ] || usage; NO_PROOF=1; shift ;;
|
||||
--migrate) [ "$OP" = activate ] || usage; MIGRATE=1; shift ;;
|
||||
--restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || usage; RESTORE="$2"; shift 2 ;;
|
||||
--backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
|
||||
--port) [ "$OP" = rehearse ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{1,5}$ ]] && [ "$((10#$2))" -ge 1024 ] && [ "$((10#$2))" -le 65535 ] || usage; R_PORT="$((10#$2))"; shift 2 ;;
|
||||
--events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
[ -z "$EVENTS" ] || [ -n "$RESTORE" ] || usage
|
||||
[ "$OP" != rehearse ] || { [ -n "$R_BACKUP" ] && [ -n "$R_PORT" ]; } || usage
|
||||
# A bad tester port is a usage error; --plan lists it as tester-port-invalid instead (row 180).
|
||||
[ -z "$TESTER_PORT" ] || [ "$PLAN" = 1 ] || valid_tester_port "$TESTER_PORT" 7332 || usage
|
||||
# --pm-after-harness replaces --pm (row 169 item 1), so the two never go together.
|
||||
@@ -1155,13 +1161,307 @@ do_status() {
|
||||
fi
|
||||
}
|
||||
|
||||
# --- The rehearsal (D2, row 163) ---
|
||||
|
||||
# rehearse runs the new copy's bus host (--no-launch) and Console on a scratch root built from a
|
||||
# backup, under <dataRoot>/rehearsals/<stamp>/, and gates them as activate does. It never writes the
|
||||
# live store, the live config files, the backup, the pointer, the log or a unit. Nothing prunes
|
||||
# the rehearsal directory. Design, rulings (mos-0961, mos-0962, mos-0963) and the deviations from
|
||||
# the brief: agents/sage/work/queue-163/BUILD.md.
|
||||
|
||||
# One PASS or FAIL line per check, on stdout and in steps.log. Lines from before the directory
|
||||
# exists wait in R_EARLY and go in first.
|
||||
r_line() {
|
||||
echo "$*"
|
||||
if [ -z "$STEPS" ]; then R_EARLY+=("$(now) $*"); else echo "$(now) $*" >> "$STEPS" || true; fi
|
||||
}
|
||||
r_pass() { r_line "PASS $1${2:+: $2}"; }
|
||||
r_fail() { r_line "FAIL $1${2:+: $2}"; R_RC=1; }
|
||||
r_abort() { r_fail "$@"; r_end; }
|
||||
|
||||
r_alive() { local s; s="$(cat "/proc/$1/stat" 2>/dev/null)" || return 1; s="${s##*) }"; [ "${s:0:1}" != Z ]; }
|
||||
r_gone() {
|
||||
local deadline=$((SECONDS + TIMEOUT))
|
||||
while r_alive "$1"; do [ "$SECONDS" -lt "$deadline" ] || return 1; sleep 1; done
|
||||
}
|
||||
|
||||
# Stops the Console, then the host, with SIGTERM within --timeout; one still there gets SIGKILL
|
||||
# and a FAIL line. The host has to leave the store's writer.lock behind it.
|
||||
r_stop() {
|
||||
local name pid
|
||||
for name in console host; do
|
||||
if [ "$name" = console ]; then pid="$R_CONSOLE_PID"; R_CONSOLE_PID=""; else pid="$R_HOST_PID"; R_HOST_PID=""; fi
|
||||
[ -n "$pid" ] || continue
|
||||
kill -TERM "$pid" 2>/dev/null || true
|
||||
if r_gone "$pid"; then r_pass "$name-stopped" "pid $pid"
|
||||
else kill -KILL "$pid" 2>/dev/null || true; r_fail "$name-stopped" "pid $pid was still running ${TIMEOUT}s after SIGTERM; sent SIGKILL"
|
||||
fi
|
||||
wait "$pid" 2>/dev/null || true
|
||||
done
|
||||
[ -n "$R_STARTED" ] || return 0
|
||||
R_STARTED=""
|
||||
if [ -e "$R/data/bus/writer.lock" ]; then r_fail host-released-store "$R/data/bus/writer.lock is still there"; else r_pass host-released-store; fi
|
||||
}
|
||||
|
||||
r_tree_sums() { (cd "$1" && find . -type f -print0 | sort -z | xargs -0 -r sha256sum); }
|
||||
|
||||
# The last checks, then the last line. The backup and the live config files are compared with the
|
||||
# sums taken before the rehearsal directory was made.
|
||||
r_end() {
|
||||
trap - INT TERM HUP
|
||||
r_stop
|
||||
if [ -n "$R" ]; then
|
||||
if [ "$(r_tree_sums "$R_BK")" = "$(cat "$R/backup.before")" ]; then r_pass backup-unchanged; else r_fail backup-unchanged "$R_BK differs from $R/backup.before"; fi
|
||||
if [ "$(sha256sum "$LIVE_CFG" "$LIVE_BIZ" 2>&1)" = "$(cat "$R/live.before")" ]; then r_pass live-config-unchanged; else r_fail live-config-unchanged "$LIVE_CFG or $LIVE_BIZ differs from $R/live.before"; fi
|
||||
fi
|
||||
if [ "$R_RC" = 0 ]; then r_line REHEARSAL_PASS; else r_line REHEARSAL_FAIL; fi
|
||||
[ -z "$R" ] || echo "rehearsal directory: $R"
|
||||
exit "$R_RC"
|
||||
}
|
||||
|
||||
r_port_free() {
|
||||
node -e 'const s=require("node:net").createServer();s.once("error",()=>process.exit(1));s.listen(Number(process.argv[1]),"127.0.0.1",()=>s.close(()=>process.exit(0)));' -- "$1"
|
||||
}
|
||||
|
||||
# The scratch config: the live one with dataRoot set to the scratch data root.
|
||||
R_CONFIG_JS='
|
||||
const fs=require("node:fs");const [file,data]=process.argv.slice(1);
|
||||
const c=JSON.parse(fs.readFileSync(file,"utf8"));c.dataRoot=data;process.stdout.write(JSON.stringify(c,null,2)+"\n");'
|
||||
|
||||
# The scratch business file (mos-0962, mos-0963): the live file without its tracker.* variables,
|
||||
# business and role, and with every credential reference, the roles and tracker.sync, moved to its
|
||||
# own 0600 placeholder file. The service and its rotateBy or expires date stay. The tracker key
|
||||
# stays because the loader requires it. Mode `normalize` prints the live file as this writes it,
|
||||
# for the diff.
|
||||
R_BUSINESS_JS='
|
||||
const fs=require("node:fs"),path=require("node:path");const [mode,live,out,ph]=process.argv.slice(1);
|
||||
const b=JSON.parse(fs.readFileSync(live,"utf8"));
|
||||
if(mode==="normalize"){process.stdout.write(JSON.stringify(b,null,2)+"\n");process.exit(0)}
|
||||
const DATE={gitea:"rotateBy",vikunja:"expires"};
|
||||
const strip=(vars)=>{if(vars&&typeof vars==="object")for(const k of Object.keys(vars))if(k.startsWith("tracker."))delete vars[k]};
|
||||
const swap=(name,refs)=>{
|
||||
if(!refs||typeof refs!=="object")return;
|
||||
for(const s of Object.keys(refs)){
|
||||
const file=path.join(ph,`${name}-${s}`),old=refs[s],ref={file};
|
||||
fs.writeFileSync(file,"rehearsal-placeholder-token",{mode:0o600,flag:"wx"});
|
||||
if(old&&typeof old==="object"&&Object.hasOwn(DATE,s)&&old[DATE[s]]!==undefined)ref[DATE[s]]=old[DATE[s]];
|
||||
refs[s]=ref;
|
||||
}
|
||||
};
|
||||
strip(b.vars);
|
||||
for(const [name,r] of Object.entries(b.roles??{})){strip(r?.vars);swap(name,r?.credentials)}
|
||||
swap("@sync",b.tracker?.sync?.credentials);
|
||||
fs.writeFileSync(out,JSON.stringify(b,null,2)+"\n",{mode:0o600,flag:"wx"});'
|
||||
|
||||
# The guard, before the host starts. It reads the scratch config through the new copy's own
|
||||
# loadSystem and bootConfig, as the host will, and prints one line per check:
|
||||
# data-root-live the data root is not the live one (mos-0961)
|
||||
# sessions-live no sessions.json there names a running process (mos-0961)
|
||||
# data-root-scratch the data root is the scratch one
|
||||
# notify-off the notifier config is {"notifyVersion":1,"binding":null}, 0600 (mos-0962)
|
||||
# tracker-configured bootConfig sets no trackers (mos-0962)
|
||||
# credential-ref-live every credential reference is a file under placeholders/ (mos-0963)
|
||||
# Exit 0 when all pass, 1 on the first FAIL.
|
||||
R_GUARD_JS='
|
||||
const fs=require("node:fs"),path=require("node:path");const {pathToFileURL}=require("node:url");
|
||||
const [copy,id,live,scratch,ph]=process.argv.slice(1);
|
||||
const real=(p)=>{try{return fs.realpathSync(p)}catch{return null}};
|
||||
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,"utf8");return s[s.lastIndexOf(")")+2]!=="Z"}catch{return false}};
|
||||
const pass=(n,w)=>console.log(`PASS ${n}${w?": "+w:""}`);
|
||||
const fail=(n,w)=>{console.log(`FAIL ${n}: ${w}`);process.exit(1)};
|
||||
(async()=>{
|
||||
let c,system,config;
|
||||
try{c=await import(pathToFileURL(`${copy}/packages/cli/src/config.mjs`).href);system=c.loadSystem({env:process.env})}
|
||||
catch(e){fail("scratch-config",String(e?.message??e).split("\n")[0])}
|
||||
const root=real(system.dataRoot);
|
||||
if(root===null)fail("data-root-scratch",`${system.dataRoot} does not exist`);
|
||||
if(root===real(live))fail("data-root-live",`the scratch config names dataRoot ${system.dataRoot}, which resolves to the live data root ${live}`);
|
||||
pass("data-root-live",`${system.dataRoot} is not the live data root`);
|
||||
const sf=path.join(root,"bus-host","sessions.json");
|
||||
if(fs.existsSync(sf)){
|
||||
let doc;try{doc=JSON.parse(fs.readFileSync(sf,"utf8"))}catch{fail("sessions-live",`${sf} is unreadable`)}
|
||||
if(!Array.isArray(doc?.sessions))fail("sessions-live",`${sf} is malformed`);
|
||||
const running=doc.sessions.flatMap((s)=>[s?.pid,s?.runnerPid]).filter((p)=>Number.isSafeInteger(p)&&p>0&&alive(p));
|
||||
if(running.length)fail("sessions-live",`${sf} names running pid(s) ${running.join(", ")}`);
|
||||
}
|
||||
pass("sessions-live","no running session named in the data root");
|
||||
if(root!==real(scratch))fail("data-root-scratch",`the scratch config resolves to ${system.dataRoot}, not ${scratch}`);
|
||||
pass("data-root-scratch");
|
||||
const nf=path.join(root,"notify",id,"notify.json");
|
||||
let st,doc;try{st=fs.lstatSync(nf);doc=JSON.parse(fs.readFileSync(nf,"utf8"))}catch{fail("notify-off",`${nf} is missing or not JSON`)}
|
||||
if(!st.isFile()||(st.mode&0o777)!==0o600||JSON.stringify(doc)!==JSON.stringify({notifyVersion:1,binding:null}))fail("notify-off",`${nf} is not {"notifyVersion":1,"binding":null} at mode 0600`);
|
||||
pass("notify-off","binding null, so no Discord bridge");
|
||||
try{config=c.bootConfig({system,businessId:id,env:process.env})}catch(e){fail("boot-config",String(e?.message??e).split("\n")[0])}
|
||||
if(config.trackers!==undefined)fail("tracker-configured",`bootConfig sets trackers for ${Object.keys(config.trackers).join(", ")}`);
|
||||
pass("tracker-configured","bootConfig sets no trackers");
|
||||
const b=config.businesses?.[id],phr=real(ph),refs=[];
|
||||
for(const [role,r] of Object.entries(b?.roles??{}))for(const [s,ref] of Object.entries(r?.credentials??{}))refs.push([`${role} ${s}`,ref]);
|
||||
for(const [s,ref] of Object.entries(b?.tracker?.sync?.credentials??{}))refs.push([`@sync ${s}`,ref]);
|
||||
for(const [what,ref] of refs){
|
||||
if(ref?.env!==undefined)fail("credential-ref-live",`${what} is env ${ref.env}`);
|
||||
const f=typeof ref?.file==="string"?real(ref.file):null;
|
||||
if(f===null||phr===null||path.dirname(f)!==phr)fail("credential-ref-live",`${what} is ${ref?.file}, not a file in ${ph}`);
|
||||
}
|
||||
pass("credential-ref-live",`${refs.length} reference(s), each a placeholder`);
|
||||
})().catch((e)=>fail("guard",String(e?.message??e).split("\n")[0]));'
|
||||
|
||||
# Opens a data root's store with a copy's Store, as the bus does at start, with no migration
|
||||
# (de352dfaf's verify-open.mjs). Prints OPEN_OK <version> or REFUSED <message>.
|
||||
R_OPEN_JS='
|
||||
const {pathToFileURL}=require("node:url");const [copy,root]=process.argv.slice(1);
|
||||
(async()=>{
|
||||
const {Store}=await import(pathToFileURL(`${copy}/packages/bus/src/store.mjs`).href);
|
||||
let s;
|
||||
try{s=new Store(root);process.stdout.write(`OPEN_OK ${s.get("SELECT value FROM meta WHERE key=?","schema_version").value}\n`)}
|
||||
catch(e){process.stdout.write(`REFUSED ${e.message}\n`)}
|
||||
finally{s?.close()}
|
||||
})();'
|
||||
r_open() { node -e "$R_OPEN_JS" -- "$1" "$2" 2>&1 || true; }
|
||||
|
||||
# Every process this rehearsal starts gets only HOME, PATH and the scratch MOSAIC_CONFIG.
|
||||
r_env() { env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" "$@"; }
|
||||
|
||||
# The de352dfaf checks, when the backup's schema is not the new copy's.
|
||||
r_migration() {
|
||||
local copy="$COPIES/$TO" old="$COPIES/$P_COPY" v out rc
|
||||
v="$(node -e "$STORE_JS" -- version "$R/data/bus/bus.sqlite" 2>&1)" || r_abort store-unreadable "$v"
|
||||
if [ "$v" = "$T_SCHEMA" ]; then r_pass migration "not needed: the backup is already $v"; return 0; fi
|
||||
in_chain "$v" || r_abort store-newer "the backup is $v, which $TO's migration chain does not start from"
|
||||
(umask 077; mkdir "$R/refuse" "$R/refuse/bus" && cp "$R_BK"/bus/* "$R/refuse/bus/") || r_abort scratch-build "cannot copy the backup to $R/refuse"
|
||||
out="$(r_open "$copy" "$R/refuse")"
|
||||
[ "$out" = "REFUSED schema-mismatch" ] || r_abort new-refuses-old-store "$TO opened the $v store: $out"
|
||||
r_pass new-refuses-old-store "$TO refuses the $v store at a normal open"
|
||||
rc=0; out="$(r_env node "$copy/packages/bus/src/migrate-schema.mjs" "$R/data" 2>&1)" || rc=$?
|
||||
printf '%s\n' "$out" >> "$R/migrate.log"
|
||||
[ "$rc" = 0 ] && [ "$(tail -n 1 <<< "$out")" = "bus schema $T_SCHEMA ready" ] || r_abort migrate "migrate-schema.mjs exit $rc: $(tail -n 1 <<< "$out")"
|
||||
r_pass migrate "$v to $T_SCHEMA"
|
||||
out="$(r_open "$copy" "$R/data")"
|
||||
[ "$out" = "OPEN_OK $T_SCHEMA" ] || r_abort new-opens-migrated "$out"
|
||||
r_pass new-opens-migrated
|
||||
rc=0; out="$(r_env node "$copy/packages/bus/src/migrate-schema.mjs" "$R/data" 2>&1)" || rc=$?
|
||||
printf '%s\n' "$out" >> "$R/migrate.log"
|
||||
[ "$rc" = 0 ] && [ "$(tail -n 1 <<< "$out")" = "bus schema $T_SCHEMA ready" ] || r_abort migrate-again "second migrate-schema.mjs exit $rc: $(tail -n 1 <<< "$out")"
|
||||
r_pass migrate-again "a second run is a no-op"
|
||||
out="$(r_open "$old" "$R/data")"
|
||||
[ "$out" = "REFUSED schema-mismatch" ] || r_abort old-refuses-migrated "the pointer's copy $P_COPY opened the $T_SCHEMA store: $out"
|
||||
r_pass old-refuses-migrated "$P_COPY refuses the $T_SCHEMA store"
|
||||
}
|
||||
|
||||
# One pass of D1's gate checks but the PM's, against the processes this run started. Sets R_GATE
|
||||
# to the first failing check and R_WHY to what it saw.
|
||||
r_gate_once() {
|
||||
local copy="$COPIES/$TO" out
|
||||
R_GATE=host-up R_WHY=""
|
||||
r_alive "$R_HOST_PID" || { R_WHY="the host (pid $R_HOST_PID) exited; see $R/host.log"; return 1; }
|
||||
grep -q "^bus host up: business $BUSINESS, .*, launches disabled\$" "$R/host.log" || { R_WHY="no \"bus host up ... launches disabled\" line in $R/host.log"; return 1; }
|
||||
R_GATE=bus-status
|
||||
out="$(r_env "$copy/scripts/mosaic" bus status --json 2>/dev/null)" || { R_WHY="bus status failed"; return 1; }
|
||||
R_WHY="$(node -e '
|
||||
const [b,main,since]=process.argv.slice(2),s=JSON.parse(process.argv[1]);
|
||||
const why=!s.host?"no host":s.host.business!==b?`host for ${s.host.business}`:s.host.live!==true?"host not live":String(s.host.pid)!==main?`host pid ${s.host.pid} is not ${main}, the host this run started`:!s.boot||s.boot.ok!==true?"boot status not ok":s.boot.business!==b?`boot status for ${s.boot.business}`:!(Date.parse(s.boot.at)>=Number(since))?"boot status is from before this start":"";
|
||||
process.stdout.write(why);' -- "$out" "$BUSINESS" "$R_HOST_PID" "$START_MS" 2>&1)" || { R_WHY="bus status unreadable"; return 1; }
|
||||
[ -z "$R_WHY" ] || return 1
|
||||
R_GATE=console-up
|
||||
r_alive "$R_CONSOLE_PID" || { R_WHY="the Console (pid $R_CONSOLE_PID) exited; see $R/console.log"; return 1; }
|
||||
grep -qxF "Mosaic Console: http://127.0.0.1:$R_PORT/" "$R/console.log" || { R_WHY="no \"Mosaic Console: http://127.0.0.1:$R_PORT/\" line in $R/console.log"; return 1; }
|
||||
R_GATE=healthz
|
||||
node -e '
|
||||
fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>process.exit(r.status===200?0:1),()=>process.exit(1));' -- "$R_PORT" || return 1
|
||||
R_GATE=copy-match
|
||||
tr '\0' ' ' < "/proc/$R_HOST_PID/cmdline" | grep -qF " $copy/packages/cli/src/cli.mjs bus start $BUSINESS --no-launch" || { R_WHY="the host's command line is not $copy's cli.mjs bus start $BUSINESS --no-launch"; return 1; }
|
||||
tr '\0' ' ' < "/proc/$R_CONSOLE_PID/cmdline" | grep -qF " $copy/packages/webui/src/cli.mjs serve " || { R_WHY="the Console's command line is not $copy's webui cli.mjs serve"; return 1; }
|
||||
R_GATE=""
|
||||
}
|
||||
|
||||
r_gate() {
|
||||
local deadline=$((SECONDS + TIMEOUT))
|
||||
until r_gate_once; do
|
||||
[ "$SECONDS" -lt "$deadline" ] || { r_fail "$R_GATE" "${R_WHY:-no answer within ${TIMEOUT}s}"; return 1; }
|
||||
sleep 1
|
||||
done
|
||||
r_pass gate "host-up, bus-status, console-up, healthz and copy-match on $TO"
|
||||
}
|
||||
|
||||
# No role session: the host wrote no sessions.json and no launches directory or log.
|
||||
r_no_session() {
|
||||
local found=""
|
||||
[ ! -e "$R/data/bus-host/sessions.json" ] || found+=" bus-host/sessions.json"
|
||||
[ ! -e "$R/data/launches" ] || found+=" launches"
|
||||
if [ -z "$found" ]; then r_pass no-session "no sessions.json and no launches under $R/data"; else r_fail no-session "found$found under $R/data"; fi
|
||||
}
|
||||
|
||||
do_rehearse() {
|
||||
TO="$ID"
|
||||
trap 'r_fail signal "the rehearsal was interrupted"; r_end' INT TERM HUP
|
||||
read_pointer || r_abort no-pointer "no $POINTER; the pointer's copy is the old copy the checks use"
|
||||
[ "$P_BUSINESS" = "$BUSINESS" ] || r_abort business-mismatch "the pointer is for $P_BUSINESS"
|
||||
copy_exists "$TO" 2>/dev/null || r_abort copy-missing "no copy $TO in $COPIES"
|
||||
copy_ok "$TO" 2>/dev/null || r_abort copy-unreachable "$TO is not reachable from refactor and origin/refactor"
|
||||
target_schema "$TO" || r_abort copy-schema-unknown "cannot read SCHEMA_VERSION and its chain from $COPIES/$TO/packages/bus/src/store.mjs"
|
||||
# mos-0961: an older copy's host has no --no-launch and would start the PM on its own.
|
||||
grep -qF -- '"--no-launch"' "$COPIES/$TO/packages/cli/src/cli.mjs" 2>/dev/null || r_abort copy-without-no-launch "$TO's bus host has no --no-launch flag"
|
||||
r_pass copy "$TO, schema $T_SCHEMA"
|
||||
local i
|
||||
local -a reserved=(7332 7333 "$(console_port)")
|
||||
for ((i = 0; i < ${#P_CONSOLE[@]}; i++)); do [ "${P_CONSOLE[i]}" != --tester-port ] || reserved+=("${P_CONSOLE[i + 1]:-}"); done
|
||||
for i in "${reserved[@]}"; do [ "$((10#$R_PORT))" != "$i" ] || r_abort port-reserved "--port $R_PORT is a live Console port (7332, 7333 and the pointer's: ${reserved[*]:2})"; done
|
||||
r_port_free "$R_PORT" || r_abort port-in-use "127.0.0.1:$R_PORT is in use; the rehearsal takes no other port"
|
||||
r_pass port "127.0.0.1:$R_PORT is free"
|
||||
case "$R_BACKUP" in /*) R_BK="$R_BACKUP" ;; *) R_BK="$DATA/backups/stack/$R_BACKUP" ;; esac
|
||||
[ -f "$R_BK/bus/bus.sqlite" ] && [ -f "$R_BK/SHA256SUMS" ] || r_abort backup-missing "$R_BK has no bus/bus.sqlite and SHA256SUMS"
|
||||
(cd "$R_BK/bus" && sha256sum -c --quiet ../SHA256SUMS) >/dev/null 2>&1 || r_abort backup-sums "$R_BK/bus does not match its SHA256SUMS"
|
||||
r_pass backup "$R_BK matches its SHA256SUMS"
|
||||
LIVE_CFG="$(realpath -m "${MOSAIC_CONFIG:-$HOME/.config/mosaic-dev/config.json}")"
|
||||
LIVE_BIZ="$(dirname "$LIVE_CFG")/businesses/$BUSINESS.json"
|
||||
[ -f "$LIVE_BIZ" ] || r_abort business-missing "no $LIVE_BIZ"
|
||||
|
||||
local stamp before live
|
||||
before="$(r_tree_sums "$R_BK")" && live="$(sha256sum "$LIVE_CFG" "$LIVE_BIZ" 2>&1)" || r_abort sums "cannot read $R_BK or the live config files"
|
||||
stamp="$(date -u +%Y%m%dT%H%M%SZ)-${TO:0:12}"
|
||||
(umask 077; mkdir -p "$DATA/rehearsals" && mkdir "$DATA/rehearsals/$stamp") || r_abort rehearsal-dir "cannot make $DATA/rehearsals/$stamp (one rehearsal per second and copy)"
|
||||
R="$DATA/rehearsals/$stamp" STEPS="$DATA/rehearsals/$stamp/steps.log"
|
||||
printf '%s\n' "${R_EARLY[@]}" > "$STEPS" && printf '%s\n' "$before" > "$R/backup.before" && printf '%s\n' "$live" > "$R/live.before" ||
|
||||
r_abort scratch-build "cannot write the sums to $R"
|
||||
(
|
||||
umask 077
|
||||
mkdir "$R/config" "$R/config/businesses" "$R/placeholders" "$R/data" "$R/data/bus" "$R/data/notify" "$R/data/notify/$BUSINESS" &&
|
||||
cp "$R_BK"/bus/* "$R/data/bus/" &&
|
||||
node -e "$R_CONFIG_JS" -- "$LIVE_CFG" "$R/data" > "$R/config/config.json" &&
|
||||
node -e "$R_BUSINESS_JS" -- write "$LIVE_BIZ" "$R/config/businesses/$BUSINESS.json" "$R/placeholders" &&
|
||||
printf '{"notifyVersion":1,"binding":null}\n' > "$R/data/notify/$BUSINESS/notify.json"
|
||||
) 2>> "$STEPS" || r_abort scratch-build "cannot build the scratch root under $R; see $STEPS"
|
||||
local rc=0
|
||||
diff -u --label "$LIVE_BIZ (as JSON.stringify writes it)" --label "$R/config/businesses/$BUSINESS.json" \
|
||||
<(node -e "$R_BUSINESS_JS" -- normalize "$LIVE_BIZ") "$R/config/businesses/$BUSINESS.json" > "$R/business.diff" || rc=$?
|
||||
[ "$rc" -le 1 ] || r_abort scratch-build "cannot write $R/business.diff"
|
||||
r_pass scratch-root "$R/data from the backup, config and business file under $R/config, business.diff beside them"
|
||||
|
||||
local out
|
||||
rc=0; out="$(r_env node -e "$R_GUARD_JS" -- "$COPIES/$TO" "$BUSINESS" "$DATA" "$R/data" "$R/placeholders" 2>&1)" || rc=$?
|
||||
while IFS= read -r i; do r_line "$i"; done <<< "$out"
|
||||
[ "$rc" = 0 ] || { R_RC=1; r_end; }
|
||||
|
||||
r_migration
|
||||
|
||||
START_MS="$(date +%s%3N)"
|
||||
R_STARTED=1
|
||||
# env, not r_env: a function in the background is a subshell, and $! has to be the process
|
||||
# itself (env execs scripts/mosaic, which execs node), for the bus-status pid check and the stop.
|
||||
env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" "$COPIES/$TO/scripts/mosaic" bus start "$BUSINESS" --no-launch > "$R/host.log" 2>&1 &
|
||||
R_HOST_PID=$!
|
||||
env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" node "$COPIES/$TO/packages/webui/src/cli.mjs" serve --port "$R_PORT" --business "$BUSINESS" > "$R/console.log" 2>&1 &
|
||||
R_CONSOLE_PID=$!
|
||||
if r_gate; then r_no_session; fi
|
||||
r_end
|
||||
}
|
||||
|
||||
# --- Dispatch ---
|
||||
|
||||
[ $# -gt 0 ] || usage
|
||||
OP="$1"; shift
|
||||
ID=""
|
||||
case "$OP" in
|
||||
activate) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;;
|
||||
activate|rehearse) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;;
|
||||
adopt|pm-on|rollback|status) [ $# -ge 1 ] || usage; BUSINESS="$1"; shift ;;
|
||||
-h|--help) usage 0 ;;
|
||||
*) usage ;;
|
||||
@@ -1179,10 +1479,12 @@ BUS_UNIT="mosaic-bus@$BUSINESS.service"
|
||||
FROM="" TO="" ACT="" RS="" POST="" BACKUP="" TAKEN="" RESTORED="" STEPS="" COUNT="" UNCLEAN="" FAIL=""
|
||||
STORE_SCHEMA="" START_EPOCH="" START_MS=0 GATE="" GATE_WHY="" S_CHANGED=""
|
||||
P_COPY="" P_PM=0 P_PENDING=0 P_SCHEMA="" P_CLAUDE="" P_CONSOLE=() RUN_PM=0 RUN_PENDING=0 RUN_CONSOLE=() DROP_TESTER="" PROOF_PLAN="" PROOF_STEP="" PLAN_REFUSE=()
|
||||
R="" R_RC=0 R_EARLY=() R_BK="" R_HOST_PID="" R_CONSOLE_PID="" R_STARTED="" LIVE_CFG="" LIVE_BIZ=""
|
||||
case "$OP" in
|
||||
adopt) do_adopt ;;
|
||||
activate) do_activate ;;
|
||||
pm-on) do_pm_on ;;
|
||||
rollback) do_rollback ;;
|
||||
status) do_status ;;
|
||||
rehearse) do_rehearse ;;
|
||||
esac
|
||||
Reference in new issue
Block a user