diff --git a/agents/sage/work/queue-163/BUILD.md b/agents/sage/work/queue-163/BUILD.md new file mode 100644 index 000000000..84270fa3d --- /dev/null +++ b/agents/sage/work/queue-163/BUILD.md @@ -0,0 +1,119 @@ +# Row 163 (D2): `stack-release.sh rehearse` and `bus start --no-launch` + +Gitea #1660. Brief: docs/plans/2026-10-18_stack-release.md, "Stack rehearsal on a scratch root +(D2)", blob b1500d96ad27. Reviewer: Filbert. Rulings: mos-0961, mos-0962, mos-0963. +Base: 4f02486d5 (origin/refactor). D2 lands after the Part B window (P19 385), because Part B's +window file names the stack-release.sh blob (mos-0958, mos-0959). + +## What it does + +`rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]` builds +`/rehearsals/-/` (0700) from a backup, runs a guard on the scratch +config with the target copy's code, runs the migration checks when the schema differs, starts the +target's `bus start BUSINESS --no-launch` and Console on the scratch root, gates them with D1's +checks minus the PM's, checks that no session started, stops both, and compares the backup and the +live config and business files with sums taken first. One PASS or FAIL line per check, then +REHEARSAL_PASS (0) or REHEARSAL_FAIL (1). Usage exits 4. docs/TOOLS.md has the operator text. + +`bus start --no-launch` (brief item 4): `start()` refuses `launches-disabled`, so every launch path +spawns nothing; a session's launch request is refused and logged; `listen()` starts no thread +ticker; `recover()` signals and ends nothing and logs what it left; the Registry is read-only, so +sessions.json stays as an earlier host left it and `add()` refuses. The host-up line ends +`, launches disabled`. `--pm` with it is a usage error. No unit passes it. + +## Rulings carried + +- mos-0961: `data-root-live` refuses a scratch dataRoot that resolves to the live data root; + `sessions-live` refuses when a sessions.json in the scratch data root names a running pid. The run + never deletes the file or retries. recover() under --no-launch is covered by the launcher test + "bus start --no-launch leaves an earlier host's live session alone: no signal, no end, + sessions.json as it was". +- mos-0962: every tracker.* variable is stripped, business and role level; `business.diff` keeps the + change; `tracker-configured` fails if bootConfig still sets trackers; notify.json is asserted to be + exactly {"notifyVersion":1,"binding":null} at 0600. +- mos-0963: every role credential reference and @sync's is moved to its own 0600 file under + `/placeholders/` holding `rehearsal-placeholder-token`; `credential-ref-live` fails for + an env reference or a file whose real directory is not the placeholders directory. Test R9 leaves + one role ref on a file outside placeholders and gets the FAIL; mutant K1 drops the check and R9 + fails. + +## Outbound calls + +The rehearsal reaches nothing off the host: + +- reads: the live config.json, the live business file, the pointer, the backup, the copies + directory, the target copy's roles/, git in the canonical checkout (`copy_ok`'s merge-base); +- writes: only `/rehearsals//`; +- processes: the target's bus host (`--no-launch`) and Console (`serve --port N --business B`, no + `--send`), both with `env -i HOME PATH MOSAIC_CONFIG=`; +- network: 127.0.0.1:N for /healthz, and the host's sockets under the scratch data root. + +No model call: launches are disabled and no session starts (`no-session`). No Vikunja: tracker.* +is stripped and `tracker-configured` checks bootConfig. No Discord: notify binding null. No Gitea: +every credential ref is a placeholder file, and the git socket serves only bound sessions, of +which there are none. + +`projects.stack.root` stays at the live checkout, /mnt/storage/src/mosaic-stack. The business +file's projects are copied unchanged. Under --no-launch nothing makes a checkout from it, because +checkouts are made only in `start()`, which refuses. + +## Deviations from the brief + +1. Item 1 says to copy "the business file, roles and organization file from the live config + directory". Roles come from the target copy's `roles/`, as the live bus reads them (rolesDir is + `/roles` unless MOSAIC_ROLES_DIR is set, and `env -i` leaves it unset). The live config + directory has no organization file; bootConfig's organization entry is the config dir itself, + which here is the scratch config dir. +2. The business file is rewritten, not copied (mos-0962, mos-0963), and `business.diff` records + the change. +3. The `tracker` key stays and `tracker.sync.credentials` becomes a placeholder: the business + schema requires both, so stripping the key (mos-0962's wording) makes the loader refuse. + bootConfig sets trackers only with tracker.baseUrl and a project's tracker.project, and both are + stripped; `tracker-configured` checks the result. Authority: mos-0964 (2026-10-20T14:35:30Z), + which accepts this and replaces the "@sync still present" clause of mos-0963 item 2: + business.mjs:197 calls checkTracker on every load, and checkTracker requires tracker and + tracker.sync (:57, :59) and checks sync.credentials (:63). @sync stays with its ref on a + placeholder, and `credential-ref-live` covers it with the role refs. Evidence: R10 (a tracker + still set fails `tracker-configured`), K2 (the check dropped, R10 fails), K10 (the tracker.* + strip dropped, R1 fails on `tracker-configured`), and K9 (the @sync swap dropped, R1 fails on `credential-ref-live`). +4. mosaic-config already refuses a scratch config that names the live data root lexically, because + the rehearsal directory sits inside the live data root and a dataRoot may not contain the config + dir. The guard's `data-root-live` catches the form that reaches the live root through a link + (R11). +5. Files owned: the brief lists stack-release.sh, its test, TOOLS.md and the packet. Item 4's flag + adds packages/cli/src/cli.mjs, packages/cli/src/launcher.mjs, packages/seat/src/session.mjs and + their tests (host.test.mjs, launcher.test.mjs, session.test.mjs). +6. The brief's "free loopback port" is an operator-given `--port`, refused when reserved (7332, + 7333, the pointer's --port and --tester-port) or in use, never replaced by another port. +7. Brief item 3's "in the foreground": the host runs as a child of the script for the length of + the gate, then gets SIGTERM; it is not a unit and not detached. + +## Tests + +packages/cli/tests/stack-release.test.mjs, rehearse section (stand-in host, Console, migrate and +Store in four fixture copies; real scripts/mosaic, mosaic-config, cli config and the business +loader): + +- R1 passes on the same schema: files, modes, placeholders, business.diff, the host's meta row in + the scratch store and not in the backup, args logs, steps.log equal to the output, live files + byte-identical. +- R2 migration path 3g to 3h; R3 migrate fails. +- R4 host fails, status fails, status names another pid, host launches, host keeps writer.lock. +- R5 SIGTERM ignored: SIGKILL and FAIL host-stopped. R6 healthz. R7 copy-match. +- R8 port in use, and 7332, 7333, the pointer's --port and --tester-port reserved. +- R9 credential-ref-live (mos-0963). R10 tracker-configured (mos-0962). R11 data-root-live through + a link and sessions-live on a running pid (mos-0961). R12 notify-off. +- R13 old copy, backup missing, backup sums, no pointer, usage. + +packages/cli/tests/launcher.test.mjs: recover() under --no-launch leaves an earlier host's live +session alone; every launch is refused and logged; no thread tick (and the same host with +launches runs it). packages/cli/tests/host.test.mjs: the host-up line, no sessions.json, --pm +with --no-launch is usage, and no unit renders --no-launch. packages/seat/tests/session.test.mjs: +a read-only Registry never writes and refuses add(). + +## Mutants + +33, one per behaviour, each killed (mutants.py, mutants-run.log and mutants-rerun.log beside this file). The first run left K11 +(bus-status pid), K21 (scratch store linked to the backup), K27 (ticker under --no-launch) and +K31 (cli drops launches); each got a test and the rerun killed all four. K21 now dies on +FAIL backup-unchanged, the check it exists for. diff --git a/docs/TOOLS.md b/docs/TOOLS.md index ba59079d9..f2892ca04 100644 --- a/docs/TOOLS.md +++ b/docs/TOOLS.md @@ -528,6 +528,68 @@ on PATH; the checkout's `scripts/mosaic` exits 3 with `human-required` or `tester-required`, and queue writes stay on it ([packages/cli/README.md, One pinned copy](../packages/cli/README.md#one-pinned-copy-row-85-1579)). +#### Rehearsal on a scratch root (D2, row 163) + +``` +scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S] +``` + +`rehearse` starts the target copy's bus host and Console on a copy of a +backup and runs `activate`'s gate checks on them, all but the PM's, while the +live stack keeps running. It reads the live config and a backup of the live +store, so Jason or Mos runs it, as a window's pre-step. No development seat +runs it against live data. It stops and starts no unit and writes no pointer +or log line. + +`--backup` takes an activation id under `/backups/stack/` or an +absolute path. `--port` is the scratch Console's loopback port, 1024 to +65535. The run refuses `port-reserved` for 7332, 7333 and the pointer's +`--port` and `--tester-port`, and `port-in-use` for a port something holds; +it never picks another port. `--timeout` (default 60 seconds) bounds the +gate and each stop. + +Before it makes anything, it checks the pointer, the copy (present, +reachable from refactor, a readable schema, a bus host with `--no-launch`), +the port and the backup's `SHA256SUMS`. Then it makes +`/rehearsals/-/`, mode 0700: + +- `data/bus/` holds a copy of the backup's store files, and + `data/notify/BUSINESS/notify.json` is `{"notifyVersion":1,"binding":null}`. +- `config/config.json` is the live config with `dataRoot` set to `data/`. +- `config/businesses/BUSINESS.json` is the live business file with every + `tracker.*` variable removed and every credential reference, the roles' + and `tracker.sync`'s, pointed at its own 0600 file under `placeholders/` + holding `rehearsal-placeholder-token`. `business.diff` shows the changes. + Roles come from the target copy's `roles/`, as the live bus reads them. +- `steps.log` holds the same PASS and FAIL lines the run prints, with times; + `host.log`, `console.log` and, after a migration, `migrate.log` sit beside it. + +A guard then loads the scratch config with the target's code and refuses +with a named check: `data-root-live` when the scratch data root resolves to +the live one, `sessions-live` when a `sessions.json` there names a running +pid (the run never deletes the file), `data-root-scratch`, `notify-off`, +`tracker-configured` when `bootConfig` still sets a tracker, and +`credential-ref-live` for an env reference or a file outside `placeholders/`. +When the backup's schema differs from the target's, the run checks that the +target refuses the store, migrates it, opens it, migrates it again as a +no-op, and that the pointer's copy refuses the migrated store. + +It starts `scripts/mosaic bus start BUSINESS --no-launch` and the Console's +`serve --port N --business BUSINESS` from the target copy, without `--send`. +The gate needs the host's `launches disabled` line, a live `bus status` host +with this run's pid and a boot from this start, the Console's address line, +`/healthz` 200, and both command lines on the target copy. Then it checks +that the host wrote no `sessions.json` and no `launches`, stops the Console +and the host (SIGTERM, then SIGKILL and a FAIL after the timeout), checks that +the host released `writer.lock`, and compares the backup and the live config +and business files with sums it took before. The last line is +`REHEARSAL_PASS` (exit 0) or `REHEARSAL_FAIL` (exit 1), then the directory. +Usage exits 4. Nothing prunes rehearsal directories. + +`bus start --no-launch` refuses every launch with `launches-disabled`, +starts no thread ticker, and leaves an earlier host's `sessions.json` and +its runs alone. No unit passes it, and `--pm` with it is a usage error. + ## Console service (`scripts/webui-service.sh`) ```bash diff --git a/packages/cli/src/cli.mjs b/packages/cli/src/cli.mjs index 37b11ea0d..10ba53e95 100644 --- a/packages/cli/src/cli.mjs +++ b/packages/cli/src/cli.mjs @@ -57,7 +57,7 @@ export const USAGE = `usage: mosaic stop mosaic launches off|on [--business ] mosaic launches list [--json] - mosaic bus start [--pm] + mosaic bus start [--pm | --no-launch] mosaic bus stop mosaic bus status [--json]`; @@ -482,8 +482,11 @@ async function launches(rest, io, deps) { async function bus(rest, io, deps) { const [sub, ...args] = rest; if (sub === "start") { - const parsed = parse(args, { flags: ["--pm"] }); + const parsed = parse(args, { flags: ["--pm", "--no-launch"] }); if (parsed.positional.length !== 1) throw usage("bus start takes one business"); + // Row 163: --no-launch is the D2 rehearsal's host. No unit passes it. + const launches = !parsed.flags.has("--no-launch"); + if (!launches && parsed.flags.has("--pm")) throw usage("bus start takes --pm or --no-launch, not both"); const businessId = parsed.positional[0]; const system = deps.system ?? loadSystem({ env: io.env }); // Fail closed, and say so (vikunja:19/302): a refused load is recorded where the Console and @@ -500,9 +503,9 @@ async function bus(rest, io, deps) { const binding = readNotifyConfig(system.dataRoot, businessId); const host = await startHost({ boot, business: businessId, notifier: binding ? { binding } : null }); try { - const launcher = createLauncher({ host, system, env: io.env, tracker: Boolean(boot.trackers?.[businessId]), ...(deps.launcher ?? {}) }); + const launcher = createLauncher({ host, system, env: io.env, tracker: Boolean(boot.trackers?.[businessId]), launches, ...(deps.launcher ?? {}) }); await launcher.listen(); - io.stdout.write(`bus host up: business ${businessId}, socket ${host.path}, notifier ${binding ?? "off"}\n`); + io.stdout.write(`bus host up: business ${businessId}, socket ${host.path}, notifier ${binding ?? "off"}${launches ? "" : ", launches disabled"}\n`); if (parsed.flags.has("--pm")) { const pm = await launcher.launchPm(); io.stdout.write(`launched ${pm.instance}: run ${pm.run}, ${pm.harness} ${pm.model}\n`); diff --git a/packages/cli/src/launcher.mjs b/packages/cli/src/launcher.mjs index fb0f9f647..5bd36a86d 100644 --- a/packages/cli/src/launcher.mjs +++ b/packages/cli/src/launcher.mjs @@ -153,7 +153,11 @@ function call(name, what, fn) { } // host: startHost()'s handle. tracker: true when the broker has task verbs for -// the business. makeLoader (the plug-in loader's factory, which tests point at a +// the business. launches: false for `bus start --no-launch` (row 163, the D2 +// rehearsal): start() refuses launches-disabled, listen() starts no ticker, a +// session's launch request is refused, recover() kills and ends nothing, and +// sessions.json stays as it was. +// makeLoader (the plug-in loader's factory, which tests point at a // scratch tree), bwrap, readOnly, sessionDefaults, harnessVersion, // stopTimeoutMs, threadTickMs and git (createGitSocket options) exist for // the tests; the command line never sets them. @@ -162,6 +166,7 @@ export function createLauncher({ system, env = process.env, tracker = false, + launches = true, makeLoader = createLoader, bwrap = BWRAP, readOnly = [], @@ -187,7 +192,7 @@ export function createLauncher({ } catch (e) { throw new CliError(`${e.message}; check which sessions still run, then move the file aside`, 3); } - const registry = new Registry(dataRoot); + const registry = new Registry(dataRoot, { readOnly: !launches }); const children = new Map(); let chain = Promise.resolve(); let closed = false; @@ -323,6 +328,8 @@ export function createLauncher({ // Bundle, spawn, bind, register, hand over the capability. async function start({ instance, launchedBy, resolved, model, account = null, thread = null }) { + // Every launch path ends here, so a --no-launch host spawns nothing whichever path asked. + if (!launches) throw new Refusal("launches-disabled", "this host was started with --no-launch"); // A thread's store is checked first, so a bad id or an unsafe directory // launches nothing. let store = null; @@ -627,6 +634,11 @@ export function createLauncher({ await host.op({ op: "refuse", cap, code: "not-a-role" }).catch(() => {}); return { ok: false, error: "not-a-role" }; } + if (!launches) { + await host.op({ op: "refuse", cap, code: "launches-disabled" }).catch(() => {}); + record({ event: "refused", by: who.role, instance, code: "launches-disabled" }); + return { ok: false, error: "launches-disabled" }; + } let checked; try { if (typeof instance !== "string") throw new Refusal("invalid-request"); @@ -675,6 +687,13 @@ export function createLauncher({ // This broker never bound those runs, so each is bound again first with // its identical launch record, which the broker accepts. async function recover() { + // Row 163 (mos-0961): a --no-launch host never signals or ends an earlier host's session. Its + // data root may hold a sessions.json that names live processes, so it only says what it left. + if (!launches) { + for (const s of leftover) log(`run ${s.run} (${s.instance}) from an earlier host: left as it is, this host was started with --no-launch`); + leftover = []; + return; + } for (const s of leftover) { if (startTimeOf(s.pid) === s.startTime && (cmdlineOf(s.pid) ?? [])[0] === bwrap) { try { @@ -744,7 +763,7 @@ export function createLauncher({ }); chmodSync(path, 0o600); await gitSocket.listen(); - if (b.launch) ticker = setInterval(() => tick().catch((e) => log(`thread tick: ${e.message}`)), threadTickMs); + if (b.launch && launches) ticker = setInterval(() => tick().catch((e) => log(`thread tick: ${e.message}`)), threadTickMs); } // SIGTERM every runner, wait, then end what's left. Runs before the host diff --git a/packages/cli/tests/host.test.mjs b/packages/cli/tests/host.test.mjs index 63dab06c3..3be5f1fa8 100644 --- a/packages/cli/tests/host.test.mjs +++ b/packages/cli/tests/host.test.mjs @@ -811,6 +811,29 @@ test("bus start runs until bus stop; status reports it while it runs", async (t) assert.match(again.stdout, /no host runs/); }); +// Row 163 (mos-0961): the D2 rehearsal's host. +test("bus start --no-launch says launches are disabled in its host-up line and stops cleanly; with --pm it is a usage error", async (t) => { + const f = fixture(tmp(t)); + notifyConfig(f.dataRoot, "acme", null); + const both = spawnSync(process.execPath, [CLI, "bus", "start", "acme", "--pm", "--no-launch"], { env: f.env, encoding: "utf8", timeout: 40000 }); + assert.equal(both.status, 4, both.stderr); + assert.match(both.stderr, /bus start takes --pm or --no-launch, not both/); + assert.equal(existsSync(join(f.dataRoot, "bus", "writer.lock")), false); + const child = spawn(process.execPath, [CLI, "bus", "start", "acme", "--no-launch"], { env: f.env, stdio: ["ignore", "pipe", "pipe"] }); + t.after(() => child.exitCode === null && child.kill("SIGKILL")); + let out = ""; + child.stdout.on("data", (b) => (out += b)); + child.stderr.on("data", (b) => (out += b)); + await until(() => /bus host up: business acme.*\n/.test(out), 30000); + assert.match(out, /bus host up: business acme, socket \S+, notifier off, launches disabled\n/); + // The launcher got launches false: its Registry is read-only, so no sessions.json appears. + assert.equal(existsSync(join(f.dataRoot, "bus-host", "sessions.json")), false); + const exit = once(child, "exit"); + const stop = spawnSync(process.execPath, [CLI, "bus", "stop"], { env: f.env, encoding: "utf8", timeout: 70000 }); + assert.equal(stop.status, 0, stop.stderr); + assert.equal((await exit)[0], 0, out); +}); + test("bus-service.sh renders the unit and installs it into a given directory", (t) => { const dir = tmp(t); const script = join(REPO, "scripts", "bus-service.sh"); @@ -818,6 +841,8 @@ test("bus-service.sh renders the unit and installs it into a given directory", ( assert.equal(render.status, 0, render.stderr); assert.match(render.stdout, new RegExp(`ExecStart=${REPO.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")}/scripts/mosaic bus start %i`)); assert.match(render.stdout, /RestartPreventExitStatus=2 3 4/); + // Row 163: --no-launch is for the D2 rehearsal's own host only; no unit starts a bus with it. + assert.doesNotMatch(render.stdout, /--no-launch/); // One session credential file, optional so Pi sessions start without it. // The README's paragraph names the same file; nothing else does. const files = render.stdout.match(/^EnvironmentFile=.*$/gm); diff --git a/packages/cli/tests/launcher.test.mjs b/packages/cli/tests/launcher.test.mjs index 06f4abe49..bd8b6ea18 100644 --- a/packages/cli/tests/launcher.test.mjs +++ b/packages/cli/tests/launcher.test.mjs @@ -15,11 +15,12 @@ import { chmodSync, existsSync, mkdirSync, readFileSync, readdirSync, renameSync import { join } from "node:path"; import { fileURLToPath } from "node:url"; import { Client } from "../../bus/src/client.mjs"; -import { gitSocketPath, launchLogFile, launchSocketPath, readSessions, Registry, SESSION_CREDENTIALS, sessionsFile, workspaceDir } from "../../seat/src/session.mjs"; -import { sandboxProblem } from "../../seat/src/sandbox.mjs"; +import { gitSocketPath, launchLogFile, launchSocketPath, readSessions, Registry, runsDir, SESSION_CREDENTIALS, sessionsFile, workspaceDir } from "../../seat/src/session.mjs"; +import { BWRAP, sandboxProblem } from "../../seat/src/sandbox.mjs"; +import { childrenOf } from "../../seat/src/proc.mjs"; import { gitEnv } from "../../seat/src/checkout.mjs"; import { bootConfig, loadSystem } from "../src/config.mjs"; -import { startHost, startTimeOf } from "../src/host.mjs"; +import { cmdlineOf, startHost, startTimeOf } from "../src/host.mjs"; import { createLauncher } from "../src/launcher.mjs"; import { mockAnthropic, REPO } from "../../harness/tests/helpers.mjs"; import { fakeHarnesses, fixture, tmp } from "./helpers.mjs"; @@ -901,6 +902,56 @@ for (const exited of [false, true]) { }); } +// Row 163 (mos-0961): the D2 rehearsal host. A live bwrap process the test owns stands in for an +// earlier host's session; its outer bwrap is what recover() would kill, and its child stands in +// for the runner that close() would signal. +test("bus start --no-launch leaves an earlier host's live session alone: no signal, no end, sessions.json as it was", { skip, timeout: 30000 }, async (t) => { + const given = prepare(t); + const { f } = given; + const box = spawn(BWRAP, ["--ro-bind", "/", "/", "--dev", "/dev", "--proc", "/proc", "--unshare-pid", "--die-with-parent", "sleep", "60"], { stdio: "ignore" }); + t.after(() => box.kill("SIGKILL")); + const pid = box.pid; + const startTime = await until(() => startTimeOf(pid)); + const runnerPid = await until(() => childrenOf(pid)[0]); + const runnerStartTime = await until(() => startTimeOf(runnerPid)); + assert.equal(cmdlineOf(pid)[0], BWRAP, "recover() would take it for a session"); + new Registry(f.dataRoot).add({ business: "acme", instance: "chief", run: "run-left", harness: "pi", pid, startTime, runnerPid, runnerStartTime }); + const before = readFileSync(sessionsFile(f.dataRoot), "utf8"); + + const s = await setup(t, undefined, given, { launches: false }); + assert.ok(s.logs.includes("run run-left (chief) from an earlier host: left as it is, this host was started with --no-launch"), s.logs.join("\n")); + assert.equal(startTimeOf(pid), startTime, "listen() sent no signal"); + assert.equal(readFileSync(sessionsFile(f.dataRoot), "utf8"), before); + assert.deepEqual((await s.client.call("trail", { subject: "run-left" })).filter((r) => r.kind === "session.ended"), []); + assert.equal(await s.close(), 0); + assert.equal(startTimeOf(pid), startTime, "close() sent no signal either"); + assert.equal(startTimeOf(runnerPid), runnerStartTime); + assert.equal(readFileSync(sessionsFile(f.dataRoot), "utf8"), before); +}); + +test("bus start --no-launch refuses every launch: the PM launch and a session's request, each logged, nothing spawned", { skip, timeout: 30000 }, async (t) => { + const s = await setup(t, undefined, undefined, { launches: false }); + await assert.rejects(s.launcher.launchPm(), (e) => e.exitCode === 3 && /^PM launch refused: launches-disabled/.test(e.message)); + const sock = launchSocketPath(s.f.dataRoot); + assert.deepEqual(await ask(sock, `${JSON.stringify({ cap: s.cto.cap, instance: "coder" })}\n`), { ok: false, error: "launches-disabled" }); + assert.deepEqual(log(s.f).map((e) => [e.event, e.instance, e.code]), [["refused", "chief", "launches-disabled"], ["refused", "coder", "launches-disabled"]]); + assert.deepEqual(readSessions(s.f.dataRoot), []); + const runs = runsDir(s.f.dataRoot, "acme"); + assert.deepEqual(existsSync(runs) ? readdirSync(runs) : [], [], "no run directory"); + assert.equal(await s.close(), 0); +}); + +test("bus start --no-launch starts no thread tick; the same host with launches runs it", { skip, timeout: 30000 }, async (t) => { + for (const launches of [false, true]) { + const asked = []; + const wrap = (host) => Object.create(host, { op: { value: async (m) => (["threadsState", "landHolders"].includes(m.op) && asked.push(m.op), host.op(m)) } }); + const s = await setup(t, undefined, undefined, { launches, threadTickMs: 50, wrap }); + await new Promise((r) => setTimeout(r, 400)); + assert.equal(asked.length > 0, launches, `launches ${launches}: ${asked.join(", ")}`); + assert.equal(await s.close(), 0); + } +}); + test("a malformed sessions.json refuses the host start with exit 3 and stays as it was", { timeout: 30000 }, async (t) => { const given = prepare(t); const file = sessionsFile(given.f.dataRoot); diff --git a/packages/cli/tests/stack-release.test.mjs b/packages/cli/tests/stack-release.test.mjs index b5f9c2119..67a96cefa 100644 --- a/packages/cli/tests/stack-release.test.mjs +++ b/packages/cli/tests/stack-release.test.mjs @@ -1705,3 +1705,335 @@ test('T54 row 188: a carried pair gets the proof file checks, and a proof file i await noLeak(e); }); }); + +// --- rehearse (D2, row 163; the brief's "Stack rehearsal on a scratch root") --- +// Copies of their own in RCOPIES: the real scripts/mosaic, config loader, business loader, roles and +// mosaic-config.mjs, with stand-ins for the bus host (cli.mjs), the Console, the Store and +// migrate-schema.mjs. A rehearsal starts its processes with only HOME, PATH and MOSAIC_CONFIG, so +// the stand-ins take their faults from $HOME/faults and record what they did in $HOME. + +// The bus host. bus start writes the store's writer.lock, a meta row in the store, and its state +// beside the scratch store, and prints the host-up line; without --no-launch it starts the PM, which here means a line in +// $HOME/starts.log and a sessions.json. bus status --json answers from that state. +function rHost(){ + const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'); + const id=path.basename(path.resolve(import.meta.dirname,'..','..','..')),home=process.env.HOME,a=process.argv.slice(2); + const fault=(n)=>fs.existsSync(path.join(home,'faults',n)); + const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}}; + const data=JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8')).dataRoot,state=path.join(data,'fixture-host.json'); + const flags=["--pm","--no-launch"]; + if(a.join(' ')==='bus status --json'){ + if(fault(`status-${id}`))process.exit(1); + let s={host:null,boot:null};try{s=JSON.parse(fs.readFileSync(state,'utf8'));s.host.live=alive(s.host.pid)}catch{} + if(fault('status-other-host')&&s.host)s.host.pid=process.ppid; + console.log(JSON.stringify(s));process.exit(0); + } + if(a[0]!=='bus'||a[1]!=='start'||!a.slice(3).every((f)=>flags.includes(f))){console.error(`fixture host: ${a.join(' ')}`);process.exit(4)} + fs.appendFileSync(path.join(home,'host-args.log'),`${a.join(' ')}\n`); + if(fault(`host-fails-${id}`)){console.error('fixture host: refused');process.exit(1)} + const b=a[2],off=a.includes('--no-launch'),lock=path.join(data,'bus','writer.lock'),at=new Date().toISOString(); + fs.writeFileSync(lock,String(process.pid),{flag:'wx'}); + const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(data,'bus','bus.sqlite')); + db.prepare("INSERT OR REPLACE INTO meta(key,value) VALUES('fixture_host_start',?)").run(at);db.close(); + fs.writeFileSync(state,JSON.stringify({host:{business:b,pid:process.pid,live:true,startedAt:at,notifier:'off'},boot:{ok:true,business:b,at}})); + if(!off||fault('host-launches')){ + fs.appendFileSync(path.join(home,'starts.log'),`pm-${b}\n`); + fs.mkdirSync(path.join(data,'bus-host'),{recursive:true}); + fs.writeFileSync(path.join(data,'bus-host','sessions.json'),JSON.stringify({sessionsVersion:1,sessions:[{pid:process.pid}]})); + } + console.log(`bus host up: business ${b}, socket ${path.join(data,'bus','broker.sock')}, notifier off${off?', launches disabled':''}`); + process.on('SIGTERM',()=>{if(fault('host-ignores-term'))return;if(!fault('host-keeps-lock'))fs.rmSync(lock,{force:true});process.exit(0)}); + setInterval(()=>{},1<<30); +} + +// The Console: serve on the given loopback port; /healthz answers 200 unless healthz- is set. +function rConsole(){ + const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),http=process.getBuiltinModule('node:http'); + const id=path.basename(path.resolve(import.meta.dirname,'..','..','..')),home=process.env.HOME,a=process.argv.slice(2); + const fault=(n)=>fs.existsSync(path.join(home,'faults',n)); + fs.appendFileSync(path.join(home,'console-args.log'),`${a.join(' ')}\n`); + if(a[0]!=='serve'){console.error('fixture Console: refused');process.exit(1)} + const s=http.createServer((q,r)=>{r.writeHead(q.url==='/healthz'&&!fault(`healthz-${id}`)?200:500);r.end()}); + s.listen(Number(a[a.indexOf('--port')+1]),'127.0.0.1',()=>console.log(`Mosaic Console: http://127.0.0.1:${s.address().port}/\nBoard: fixture`)); + process.on('SIGTERM',()=>process.exit(0)); +} + +// migrate-schema.mjs: sets its copy's SCHEMA_VERSION, or exit 1 under migrate-fails. +function rMigrate(){ + const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite'); + const home=process.env.HOME,data=process.argv[2]; + const v=/^export const SCHEMA_VERSION = '([A-Za-z0-9]+)';$/m.exec(fs.readFileSync(path.join(import.meta.dirname,'store.mjs'),'utf8'))[1]; + fs.appendFileSync(path.join(home,'migrate.log'),`${data}\n`); + if(fs.existsSync(path.join(home,'faults','migrate-fails'))){console.error('fixture migrate: failed');process.exit(1)} + const db=new DatabaseSync(path.join(data,'bus','bus.sqlite'));db.prepare("UPDATE meta SET value=? WHERE key='schema_version'").run(v);db.close(); + console.log(`bus schema ${v} ready`); +} + +// The copy's Store: refuses a store at another version with schema-mismatch, as the real one does. +const rStore=(v,froms)=>`${storeText(v,froms)}import {DatabaseSync} from 'node:sqlite'; +import {join} from 'node:path'; +export class Store { + constructor(root) { + this.db = new DatabaseSync(join(root, 'bus', 'bus.sqlite')); + if (this.db.prepare("SELECT value FROM meta WHERE key='schema_version'").get()?.value !== SCHEMA_VERSION) { this.db.close(); throw Error('schema-mismatch'); } + } + get(sql, ...args) { return this.db.prepare(sql).get(...args); } + close() { this.db.close(); } +} +`; + +// B (3g) and C (3h) have --no-launch. A is a copy from before row 163, whose host has no such flag. +// C1's scripts/mosaic runs B's host, as a copy-check failure needs. +const RCOPIES=join(base,'rcopies');mkdirSync(RCOPIES); +for(const [id,v,froms] of [[B,'3g',['3f']],[C,'3h',['3f','3g']],[A,'3g',['3f']],[C1,'3g',['3f']]]){ + const d=join(RCOPIES,id); + for(const p of ['scripts/mosaic','scripts/mosaic-config.mjs','packages/cli/src/config.mjs','packages/cli/src/errors.mjs','packages/bus/src/business.mjs','packages/plugins/src/kinds.mjs','packages/business/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true}); + if(id===C1)writeFileSync(join(d,'scripts','mosaic'),`#!/usr/bin/env bash\nexec node "${join(RCOPIES,B,'packages','cli','src','cli.mjs')}" "$@"\n`,{mode:0o755}); + for(const [f,text] of [['packages/cli/src/cli.mjs',id===A?'// fixture host from before row 163\nprocess.exit(1);\n':script(rHost)],['packages/webui/src/cli.mjs',script(rConsole)],['packages/bus/src/store.mjs',rStore(v,froms)],['packages/bus/src/migrate-schema.mjs',script(rMigrate)],['packages/bus/src/broker.mjs','export class BusError extends Error { constructor(code) { super(code); this.code = code; } }\n']]){mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text)} + execFileSync('chmod',['-R','a-w',d]); +} + +const TESTER='7499'; +const freePort=async()=>{const s=createServer();s.listen(0,'127.0.0.1');await once(s,'listening');const p=String(s.address().port);await new Promise((r)=>s.close(r));return p}; +const rh=(e,...p)=>join(e.env.HOME,...p); +const rfault=(e,name)=>{mkdirSync(rh(e,'faults'),{recursive:true});writeFileSync(rh(e,'faults',name),'')}; + +// A root for one rehearsal: a stopped live store at `store`, a pointer by hand on `copy` with the +// Console on e.port and the tester listener on TESTER, and a backup as backup_store writes one. +// A rehearsal's leftovers, found by their scratch MOSAIC_CONFIG, are killed after the test. +async function rmk(t,{copy=B,store='3g'}={}){ + const e=await mk(t); + t.after(()=>{ + for(const pid of readdirSync('/proc').filter((n)=>/^\d+$/.test(n))) + try{if(readFileSync(`/proc/${pid}/environ`,'utf8').includes(`MOSAIC_CONFIG=${e.data}/rehearsals/`))process.kill(Number(pid),'SIGKILL')}catch{} + }); + const db=new DatabaseSync(e.store); + db.exec('PRAGMA journal_mode=WAL'); + db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)'); + db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)'); + db.prepare("INSERT INTO meta VALUES ('schema_version', ?)").run(store); + for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`); + db.close(); + mkdirSync(e.state,{recursive:true}); + writeFileSync(e.pointer,JSON.stringify({copy,business:'acme',pm:true,claude:e.claude,console:['--port',e.port,'--business','acme','--tester-port',TESTER],schema:store,activation:'manual',at:'2026-10-20T00:00:00Z'})+'\n'); + e.act=`20261020T000000Z-${copy.slice(0,12)}`;e.backup=join(e.data,'backups','stack',e.act); + mkdirSync(join(e.backup,'bus'),{recursive:true,mode:0o700}); + cpSync(e.store,join(e.backup,'bus','bus.sqlite')); + writeFileSync(join(e.backup,'SHA256SUMS'),`${sha(e.store)} bus.sqlite\n`); + for(const [f,text] of [['DATA_ROOT',e.data],['SCHEMA',store],['STAMP','2026-10-20T00:00:00.000Z']])writeFileSync(join(e.backup,f),`${text}\n`); + e.biz=join(dirname(e.cfg),'businesses','acme.json'); + return e; +} + +// What a rehearsal may never change: the live store, config, business file, pointer and log, and +// the backup, by sha256 of every file. +const untouched=(e)=>JSON.stringify([...[e.store,e.cfg,e.biz,e.pointer].map((f)=>[f,sha(f)]),existsSync(e.log),...readdirSync(e.backup,{recursive:true}).sort().map((f)=>join(e.backup,f)).filter((f)=>statSync(f).isFile()).map((f)=>[f,sha(f)])]); + +async function rehearse(e,id,...args){ + const ch=spawn('bash',[e.sr??SR,'rehearse',id,'acme',...args,'--copies',RCOPIES,'--bin',e.bin],{env:e.env,stdio:['ignore','pipe','pipe']}); + let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d}); + const timer=setTimeout(()=>ch.kill('SIGKILL'),120000); + const [code]=await once(ch,'close');clearTimeout(timer); + const m=/\nrehearsal directory: (.*)\n$/.exec(out); + return {code,out,err,dir:m?m[1]:null,names:out.split('\n').filter((l)=>/^(PASS|FAIL) /.test(l)).map((l)=>l.replace(/^(PASS|FAIL) ([^:]*).*/,'$1 $2'))}; +} +// A script of its own whose text has `from` replaced, for the guard tests. The replacement is a +// function so a shell "$$" in `to` stays as written. +function ownScript(e,from,to){ + const own=join(e.root,'repo');cpSync(repo,own,{recursive:true}); + e.sr=join(own,'scripts','stack-release.sh'); + const s=readFileSync(e.sr,'utf8');assert.ok(s.includes(from),`${e.sr} has no ${from}`);writeFileSync(e.sr,s.replace(from,()=>to)); +} +const rDirs=(e)=>existsSync(join(e.data,'rehearsals'))?readdirSync(join(e.data,'rehearsals')):[]; +const PRE=['PASS copy','PASS port','PASS backup','PASS scratch-root','PASS data-root-live','PASS sessions-live','PASS data-root-scratch','PASS notify-off','PASS tracker-configured','PASS credential-ref-live']; +const POST=['PASS console-stopped','PASS host-stopped','PASS host-released-store','PASS backup-unchanged','PASS live-config-unchanged']; + +test('R1 rehearse passes on the same schema; the scratch root holds its files and the live files stay byte-identical',async(t)=>{ + const e=await rmk(t),before=untouched(e),port=await freePort(); + const r=await rehearse(e,B,'--backup',e.act,'--port',port,'--timeout','10'); + assert.equal(r.code,0,r.out+r.err); + assert.deepEqual(r.names,[...PRE,'PASS migration','PASS gate','PASS no-session',...POST]); + assert.ok(r.out.endsWith(`REHEARSAL_PASS\nrehearsal directory: ${r.dir}\n`),r.out); + assert.equal(r.err,''); + assert.equal(untouched(e),before); + const R=r.dir;assert.equal(dirname(R),join(e.data,'rehearsals'));assert.match(basename(R),new RegExp(`^\\d{8}T\\d{6}Z-${B.slice(0,12)}$`)); + for(const d of [dirname(R),R,join(R,'placeholders'),join(R,'config')])assert.equal(statSync(d).mode&0o777,0o700,d); + for(const f of ['steps.log','backup.before','live.before','business.diff','host.log','console.log','config/config.json','data/bus/bus.sqlite'])assert.ok(existsSync(join(R,f)),f); + // The host wrote the scratch store, a copy: the backup has no such row. + const metaOf=(f)=>{const db=new DatabaseSync(f,{readOnly:true});try{return db.prepare("SELECT value FROM meta WHERE key='fixture_host_start'").get()?.value}finally{db.close()}}; + assert.match(metaOf(join(R,'data','bus','bus.sqlite')),/^\d{4}-/);assert.equal(metaOf(join(e.backup,'bus','bus.sqlite')),undefined); + // The scratch config differs from the live one only in dataRoot. + assert.deepEqual(JSON.parse(readFileSync(join(R,'config','config.json'),'utf8')),{...JSON.parse(readFileSync(e.cfg,'utf8')),dataRoot:join(R,'data')}); + // The notifier config: binding null at 0600. + const nf=join(R,'data','notify','acme','notify.json'); + assert.deepEqual(JSON.parse(readFileSync(nf,'utf8')),{notifyVersion:1,binding:null});assert.equal(statSync(nf).mode&0o777,0o600); + // The scratch business: no tracker.* variable, and every reference a 0600 placeholder with its date kept. + const live=JSON.parse(readFileSync(e.biz,'utf8')),sb=JSON.parse(readFileSync(join(R,'config','businesses','acme.json'),'utf8')); + assert.ok(Object.keys(live.vars).some((k)=>k.startsWith('tracker.'))); + assert.deepEqual(Object.keys(sb.vars).filter((k)=>k.startsWith('tracker.')),[]); + const refs=[...Object.entries(sb.roles).flatMap(([n,r])=>Object.entries(r.credentials??{}).map(([s,ref])=>[`${n}-${s}`,ref,live.roles[n].credentials[s]])),...Object.entries(sb.tracker.sync.credentials).map(([s,ref])=>[`@sync-${s}`,ref,live.tracker.sync.credentials[s]])]; + assert.ok(refs.length>=2); + for(const [name,ref,old] of refs){ + const f=join(R,'placeholders',name); + assert.deepEqual(ref,{...old,file:f});assert.equal(readFileSync(f,'utf8'),'rehearsal-placeholder-token');assert.equal(statSync(f).mode&0o777,0o600); + } + assert.deepEqual(readdirSync(join(R,'placeholders')).sort(),refs.map(([n])=>n).sort()); + // Everything else in the business is as it was. + const strip=(b)=>{const c=structuredClone(b);for(const k of Object.keys(c.vars))if(k.startsWith('tracker.'))delete c.vars[k];for(const r of Object.values(c.roles))delete r.credentials;delete c.tracker.sync.credentials;return c}; + assert.deepEqual(strip(sb),strip(live)); + const diff=readFileSync(join(R,'business.diff'),'utf8');assert.match(diff,/^-.*"tracker\.baseUrl"/m);assert.match(diff,/^\+.*placeholders\/pm-/m); + // The host ran with launches disabled and started no PM; the Console got no --send. + assert.deepEqual(lines(rh(e,'host-args.log')),['bus start acme --no-launch']); + assert.deepEqual(lines(rh(e,'console-args.log')),[`serve --port ${port} --business acme`]); + assert.equal(existsSync(rh(e,'starts.log')),false); + assert.equal(existsSync(join(R,'data','bus','writer.lock')),false); + assert.deepEqual(lines(join(R,'steps.log')).map((l)=>l.replace(/^\S+ /,'')),r.out.split('\n').filter((l)=>/^(PASS|FAIL) |^REHEARSAL_/.test(l))); +}); + +test('R2 rehearse runs the migration checks when the schema differs; the live store stays at 3g',async(t)=>{ + const e=await rmk(t),before=untouched(e),port=await freePort(); + const r=await rehearse(e,C,'--backup',e.backup,'--port',port,'--timeout','10'); + assert.equal(r.code,0,r.out+r.err); + assert.deepEqual(r.names,[...PRE,'PASS new-refuses-old-store','PASS migrate','PASS new-opens-migrated','PASS migrate-again','PASS old-refuses-migrated','PASS gate','PASS no-session',...POST]); + assert.ok(r.out.includes('PASS migrate: 3g to 3h\n'),r.out); + assert.equal(untouched(e),before);assert.equal(storeVersion(e.store,false),'3g'); + assert.equal(storeVersion(join(r.dir,'data','bus','bus.sqlite'),false),'3h'); + assert.equal(storeVersion(join(r.dir,'refuse','bus','bus.sqlite'),false),'3g'); + assert.deepEqual(lines(rh(e,'migrate.log')),[join(r.dir,'data'),join(r.dir,'data')]); +}); + +test('R3 a failing migration gives REHEARSAL_FAIL migrate and starts nothing',async(t)=>{ + const e=await rmk(t),before=untouched(e);rfault(e,'migrate-fails'); + const r=await rehearse(e,C,'--backup',e.act,'--port',await freePort(),'--timeout','3'); + assert.equal(r.code,1,r.out+r.err); + assert.deepEqual(r.names,[...PRE,'PASS new-refuses-old-store','FAIL migrate','PASS backup-unchanged','PASS live-config-unchanged']); + assert.ok(r.out.includes('FAIL migrate: migrate-schema.mjs exit 1: fixture migrate: failed\n'),r.out); + assert.ok(r.out.endsWith(`REHEARSAL_FAIL\nrehearsal directory: ${r.dir}\n`),r.out); + assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false); + assert.equal(untouched(e),before); +}); + +test('R4 a host that does not start, does not answer status, reports another pid, launches anyway or keeps its lock each fails by name',async(t)=>{ + for(const [f,name,why] of [[`host-fails-${B}`,'host-up','exited'],[`status-${B}`,'bus-status','bus status failed'],['status-other-host','bus-status','the host this run started'],['host-launches','no-session','bus-host/sessions.json'],['host-keeps-lock','host-released-store','writer.lock is still there']]){ + const e=await rmk(t),before=untouched(e);rfault(e,f); + const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3'); + assert.equal(r.code,1,f+r.out+r.err); + assert.ok(r.names.includes(`FAIL ${name}`),f+r.out);assert.equal(r.names.filter((n)=>n.startsWith('FAIL')).length,1,f+r.out); + assert.ok(r.out.split('\n').some((l)=>l.startsWith(`FAIL ${name}: `)&&l.includes(why)),f+r.out); + assert.ok(r.names.includes('PASS console-stopped')&&r.names.includes('PASS backup-unchanged'),f+r.out); + assert.ok(r.out.endsWith(`REHEARSAL_FAIL\nrehearsal directory: ${r.dir}\n`),r.out); + assert.equal(untouched(e),before); + } +}); + +test('R5 a host that ignores SIGTERM gets SIGKILL and FAIL host-stopped',async(t)=>{ + const e=await rmk(t);rfault(e,'host-ignores-term'); + const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','2'); + assert.equal(r.code,1,r.out+r.err); + assert.ok(/^FAIL host-stopped: pid \d+ was still running 2s after SIGTERM; sent SIGKILL$/m.test(r.out),r.out); +}); + +test('R6 a Console that does not answer /healthz gives REHEARSAL_FAIL healthz',async(t)=>{ + const e=await rmk(t);rfault(e,`healthz-${B}`); + const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3'); + assert.equal(r.code,1,r.out+r.err); + assert.ok(r.out.includes('FAIL healthz: no answer within 3s\n'),r.out); + assert.deepEqual(r.names.filter((n)=>n.startsWith('FAIL')),['FAIL healthz']); +}); + +test('R7 a host on another copy fails copy-match',async(t)=>{ + const e=await rmk(t); + const r=await rehearse(e,C1,'--backup',e.act,'--port',await freePort(),'--timeout','3'); + assert.equal(r.code,1,r.out+r.err); + assert.ok(r.out.includes(`FAIL copy-match: the host's command line is not ${join(RCOPIES,C1)}'s cli.mjs bus start acme --no-launch\n`),r.out); +}); + +test('R8 a scratch port in use or reserved refuses, and the rehearsal takes no other port',async(t)=>{ + const e=await rmk(t),before=untouched(e); + const held=createServer();held.listen(0,'127.0.0.1');await once(held,'listening');t.after(()=>held.close()); + const p=String(held.address().port); + const r=await rehearse(e,B,'--backup',e.act,'--port',p); + assert.equal(r.code,1,r.out+r.err); + assert.equal(r.out,`PASS copy: ${B}, schema 3g\nFAIL port-in-use: 127.0.0.1:${p} is in use; the rehearsal takes no other port\nREHEARSAL_FAIL\n`); + for(const q of ['7332','7333',e.port,TESTER]){ + const s=await rehearse(e,B,'--backup',e.act,'--port',q); + assert.equal(s.code,1,s.out+s.err); + assert.equal(s.out,`PASS copy: ${B}, schema 3g\nFAIL port-reserved: --port ${q} is a live Console port (7332, 7333 and the pointer's: ${e.port} ${TESTER})\nREHEARSAL_FAIL\n`); + } + assert.deepEqual(rDirs(e),[]);assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false); + assert.equal(untouched(e),before); +}); + +test('R9 mos-0963: a role reference left outside placeholders/ fails credential-ref-live before the host starts',async(t)=>{ + const e=await rmk(t); + // The live pm gitea reference names a file that is there, as a live token would be. + const tok=join(e.root,'live-gitea.token');writeFileSync(tok,'live-token-value',{mode:0o600}); + const b=JSON.parse(readFileSync(e.biz,'utf8'));b.roles.pm.credentials.gitea.file=tok;writeFileSync(e.biz,JSON.stringify(b,null,2)+'\n',{mode:0o600}); + const before=untouched(e); + ownScript(e,'for(const [name,r] of Object.entries(b.roles??{})){strip(r?.vars);swap(name,r?.credentials)}','for(const [name,r] of Object.entries(b.roles??{})){strip(r?.vars);if(name!=="pm")swap(name,r?.credentials)}'); + const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3'); + assert.equal(r.code,1,r.out+r.err); + assert.ok(r.out.includes(`FAIL credential-ref-live: pm gitea is ${tok}, not a file in ${join(r.dir,'placeholders')}\n`),r.out); + assert.deepEqual(r.names.slice(-3),['FAIL credential-ref-live','PASS backup-unchanged','PASS live-config-unchanged']); + assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false); + assert.equal(untouched(e),before); +}); + +test('R10 mos-0962: a scratch business that still sets a tracker fails tracker-configured',async(t)=>{ + const e=await rmk(t); + const proj=join(e.root,'project');mkdirSync(join(proj,'.mosaic'),{recursive:true}); + writeFileSync(join(proj,'.mosaic','project.json'),JSON.stringify({projectVersion:1,id:'stack',vars:{'tracker.project':7}}),{mode:0o600}); + const b=JSON.parse(readFileSync(e.biz,'utf8'));b.projects.stack.root=proj;writeFileSync(e.biz,JSON.stringify(b,null,2)+'\n',{mode:0o600}); + ownScript(e,'if(k.startsWith("tracker."))delete vars[k]','if(k.startsWith("tracker-kept."))delete vars[k]'); + const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3'); + assert.equal(r.code,1,r.out+r.err); + assert.ok(r.out.includes('FAIL tracker-configured: bootConfig sets trackers for acme\n'),r.out); + assert.equal(existsSync(rh(e,'host-args.log')),false); +}); + +test('R11 mos-0961: a scratch config on the live data root, or a sessions.json naming a running pid, refuses by name',async(t)=>{ + // mosaic-config already refuses the live path itself, which contains the scratch config directory; + // the guard catches a path that reaches the live data root through a link. + const e=await rmk(t),link=join(e.root,'live-link');symlinkSync(e.data,link); + ownScript(e,'c.dataRoot=data;',`c.dataRoot=${JSON.stringify(link)};`); + const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3'); + assert.equal(r.code,1,r.out+r.err); + assert.ok(r.out.includes(`FAIL data-root-live: the scratch config names dataRoot ${link}, which resolves to the live data root ${e.data}\n`),r.out); + assert.equal(existsSync(rh(e,'host-args.log')),false); + const f=await rmk(t); + ownScript(f,`printf '{"notifyVersion":1,"binding":null}\\n' > "$R/data/notify/$BUSINESS/notify.json"`,`printf '{"notifyVersion":1,"binding":null}\\n' > "$R/data/notify/$BUSINESS/notify.json" && mkdir "$R/data/bus-host" && printf '{"sessionsVersion":1,"sessions":[{"pid":%s}]}\\n' "$$" > "$R/data/bus-host/sessions.json"`); + const s=await rehearse(f,B,'--backup',f.act,'--port',await freePort(),'--timeout','3'); + assert.equal(s.code,1,s.out+s.err); + assert.ok(/^FAIL sessions-live: \S+\/bus-host\/sessions\.json names running pid\(s\) \d+$/m.test(s.out),s.out); + // The file is left as it was: the rehearsal never deletes it and retries. + assert.ok(existsSync(join(s.dir,'data','bus-host','sessions.json'))); + assert.equal(existsSync(rh(f,'host-args.log')),false); +}); + +test('R12 mos-0962: a notifier config with a binding fails notify-off',async(t)=>{ + const e=await rmk(t); + ownScript(e,`printf '{"notifyVersion":1,"binding":null}\\n'`,`printf '{"notifyVersion":1,"binding":"discord"}\\n'`); + const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3'); + assert.equal(r.code,1,r.out+r.err); + assert.ok(r.out.includes('FAIL notify-off: '),r.out); + assert.equal(existsSync(rh(e,'host-args.log')),false); +}); + +test('R13 the pre-checks: an old copy, a backup that fails its sums, a missing backup or pointer, and usage',async(t)=>{ + const e=await rmk(t),before=untouched(e),port=await freePort(); + const old=await rehearse(e,A,'--backup',e.act,'--port',port); + assert.equal(old.code,1);assert.equal(old.out,`FAIL copy-without-no-launch: ${A}'s bus host has no --no-launch flag\nREHEARSAL_FAIL\n`); + const none=await rehearse(e,B,'--backup','20261020T000000Z-000000000000','--port',port); + assert.equal(none.code,1);assert.ok(none.out.includes(`FAIL backup-missing: ${join(e.data,'backups','stack','20261020T000000Z-000000000000')} has no bus/bus.sqlite and SHA256SUMS\n`),none.out); + const f=await rmk(t);appendFileSync(join(f.backup,'bus','bus.sqlite'),'x'); + const bad=await rehearse(f,B,'--backup',f.act,'--port',port); + assert.equal(bad.code,1);assert.ok(bad.out.includes(`FAIL backup-sums: ${f.backup}/bus does not match its SHA256SUMS\n`),bad.out); + const g=await rmk(t);rmSync(g.pointer); + const np=await rehearse(g,B,'--backup',g.act,'--port',port); + assert.equal(np.code,1);assert.match(np.out,/^FAIL no-pointer: /); + for(const args of [['--port',port],['--backup',e.act],['--backup',e.act,'--port','80'],['--backup',e.act,'--port','70000'],['--backup','relative/dir','--port',port],['--backup',e.act,'--port',port,'--pm'],['--backup',e.act,'--port',port,'--plan']]){ + const u=await rehearse(e,B,...args);assert.equal(u.code,4,args.join(' ')+u.out+u.err);assert.match(u.err,/^usage: /); + } + assert.deepEqual([...rDirs(e),...rDirs(f),...rDirs(g)],[]); + assert.equal(untouched(e),before); +}); diff --git a/packages/seat/src/session.mjs b/packages/seat/src/session.mjs index fa3e8ae77..d844c79b4 100644 --- a/packages/seat/src/session.mjs +++ b/packages/seat/src/session.mjs @@ -186,17 +186,22 @@ export function endReason(code, signal) { // The host's view of its running sessions, kept in memory and mirrored to // sessions.json for `mosaic stop` and `mosaic launches list`. +// readOnly (row 163, a --no-launch host): sessions.json stays as an earlier host left it, so a +// later host still finds and ends those runs, and add() refuses. export class Registry { #dataRoot; #sessions = []; - constructor(dataRoot) { + #readOnly; + constructor(dataRoot, { readOnly = false } = {}) { this.#dataRoot = dataRoot; - this.#write(); + this.#readOnly = readOnly; + if (!readOnly) this.#write(); } get sessions() { return this.#sessions.map((s) => ({ ...s })); } add(entry) { + if (this.#readOnly) throw new SeatError("this host keeps no sessions (--no-launch)", 2); this.#sessions.push({ ...entry }); this.#write(); } diff --git a/packages/seat/tests/session.test.mjs b/packages/seat/tests/session.test.mjs index 23d288c2b..36c99463c 100644 --- a/packages/seat/tests/session.test.mjs +++ b/packages/seat/tests/session.test.mjs @@ -133,6 +133,21 @@ test("Registry mirrors to sessions.json; readSessions marks live entries; bad fi assert.throws(() => readSessions(root), (e) => e instanceof SeatError && /malformed/.test(e.message)); }); +// Row 163: a --no-launch host's registry leaves an earlier host's sessions.json as it was. +test("a read-only Registry never writes sessions.json and refuses add()", (t) => { + const root = scratch(t); + new Registry(root).add({ business: "acme", instance: "chief", run: "r1", runnerPid: process.pid, runnerStartTime: startTimeOf(process.pid) }); + const before = readFileSync(sessionsFile(root), "utf8"); + const r = new Registry(root, { readOnly: true }); + assert.equal(readFileSync(sessionsFile(root), "utf8"), before); + assert.deepEqual(r.sessions, []); + assert.throws(() => r.add({ business: "acme", instance: "coder", run: "r2" }), (e) => e instanceof SeatError && /keeps no sessions/.test(e.message)); + assert.equal(readFileSync(sessionsFile(root), "utf8"), before); + const empty = scratch(t); + new Registry(empty, { readOnly: true }); + assert.equal(existsSync(sessionsFile(empty)), false); +}); + // Row 121 (#1615): a thread instance runs once per thread, so the registry // keys a thread run by its thread too. test("REQ-THREAD-1 Registry.running keys a thread run by business, instance and thread", (t) => { diff --git a/scripts/stack-release.sh b/scripts/stack-release.sh index 2586a2a37..0053f35d1 100755 --- a/scripts/stack-release.sh +++ b/scripts/stack-release.sh @@ -15,6 +15,8 @@ # 4. this script never prunes backups, rollback sets, post-attempt directories or copies. # Decisions 94 and 102 still decide who starts the Console; webui-service.sh keeps having no start # or stop verb. status and --plan only read, and every seat may run them. +# rehearse (D2, row 163) stops and starts no unit and writes only its own rehearsal directory, but +# it reads the live config and a backup of the live store, so Jason or Mos runs it. # # Most functions below run as `f || handler`, and bash ignores set -e inside such a call, so # every step that can fail carries its own `|| return 1`. @@ -28,7 +30,7 @@ BIN="$HOME/.npm-global/bin" UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user" CONSOLE_UNIT=mosaic-webui.service TIMEOUT=60 WANT_PM=0 AFTER_HARNESS=0 MIGRATE=0 PLAN=0 RESTORE="" EVENTS="" TESTER_PORT="" -PROOF_HEADER="" PROOF_FILE="" NO_PROOF=0 +PROOF_HEADER="" PROOF_FILE="" NO_PROOF=0 R_BACKUP="" R_PORT="" usage() { cat >&2 <<'USAGE' @@ -38,6 +40,7 @@ usage: stack-release.sh adopt BUSINESS stack-release.sh pm-on BUSINESS [--timeout S] stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan] stack-release.sh status BUSINESS + stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S] common options: --copies DIR, --bin DIR --events N is the total of the per-table counts the post-backup-events refusal prints: events and every table in COUNTED. @@ -84,11 +87,14 @@ parse_opts() { --no-proof) [ "$OP" = activate ] || usage; NO_PROOF=1; shift ;; --migrate) [ "$OP" = activate ] || usage; MIGRATE=1; shift ;; --restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || usage; RESTORE="$2"; shift 2 ;; + --backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;; + --port) [ "$OP" = rehearse ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{1,5}$ ]] && [ "$((10#$2))" -ge 1024 ] && [ "$((10#$2))" -le 65535 ] || usage; R_PORT="$((10#$2))"; shift 2 ;; --events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;; *) usage ;; esac done [ -z "$EVENTS" ] || [ -n "$RESTORE" ] || usage + [ "$OP" != rehearse ] || { [ -n "$R_BACKUP" ] && [ -n "$R_PORT" ]; } || usage # A bad tester port is a usage error; --plan lists it as tester-port-invalid instead (row 180). [ -z "$TESTER_PORT" ] || [ "$PLAN" = 1 ] || valid_tester_port "$TESTER_PORT" 7332 || usage # --pm-after-harness replaces --pm (row 169 item 1), so the two never go together. @@ -1155,13 +1161,307 @@ do_status() { fi } +# --- The rehearsal (D2, row 163) --- + +# rehearse runs the new copy's bus host (--no-launch) and Console on a scratch root built from a +# backup, under /rehearsals//, and gates them as activate does. It never writes the +# live store, the live config files, the backup, the pointer, the log or a unit. Nothing prunes +# the rehearsal directory. Design, rulings (mos-0961, mos-0962, mos-0963) and the deviations from +# the brief: agents/sage/work/queue-163/BUILD.md. + +# One PASS or FAIL line per check, on stdout and in steps.log. Lines from before the directory +# exists wait in R_EARLY and go in first. +r_line() { + echo "$*" + if [ -z "$STEPS" ]; then R_EARLY+=("$(now) $*"); else echo "$(now) $*" >> "$STEPS" || true; fi +} +r_pass() { r_line "PASS $1${2:+: $2}"; } +r_fail() { r_line "FAIL $1${2:+: $2}"; R_RC=1; } +r_abort() { r_fail "$@"; r_end; } + +r_alive() { local s; s="$(cat "/proc/$1/stat" 2>/dev/null)" || return 1; s="${s##*) }"; [ "${s:0:1}" != Z ]; } +r_gone() { + local deadline=$((SECONDS + TIMEOUT)) + while r_alive "$1"; do [ "$SECONDS" -lt "$deadline" ] || return 1; sleep 1; done +} + +# Stops the Console, then the host, with SIGTERM within --timeout; one still there gets SIGKILL +# and a FAIL line. The host has to leave the store's writer.lock behind it. +r_stop() { + local name pid + for name in console host; do + if [ "$name" = console ]; then pid="$R_CONSOLE_PID"; R_CONSOLE_PID=""; else pid="$R_HOST_PID"; R_HOST_PID=""; fi + [ -n "$pid" ] || continue + kill -TERM "$pid" 2>/dev/null || true + if r_gone "$pid"; then r_pass "$name-stopped" "pid $pid" + else kill -KILL "$pid" 2>/dev/null || true; r_fail "$name-stopped" "pid $pid was still running ${TIMEOUT}s after SIGTERM; sent SIGKILL" + fi + wait "$pid" 2>/dev/null || true + done + [ -n "$R_STARTED" ] || return 0 + R_STARTED="" + if [ -e "$R/data/bus/writer.lock" ]; then r_fail host-released-store "$R/data/bus/writer.lock is still there"; else r_pass host-released-store; fi +} + +r_tree_sums() { (cd "$1" && find . -type f -print0 | sort -z | xargs -0 -r sha256sum); } + +# The last checks, then the last line. The backup and the live config files are compared with the +# sums taken before the rehearsal directory was made. +r_end() { + trap - INT TERM HUP + r_stop + if [ -n "$R" ]; then + if [ "$(r_tree_sums "$R_BK")" = "$(cat "$R/backup.before")" ]; then r_pass backup-unchanged; else r_fail backup-unchanged "$R_BK differs from $R/backup.before"; fi + if [ "$(sha256sum "$LIVE_CFG" "$LIVE_BIZ" 2>&1)" = "$(cat "$R/live.before")" ]; then r_pass live-config-unchanged; else r_fail live-config-unchanged "$LIVE_CFG or $LIVE_BIZ differs from $R/live.before"; fi + fi + if [ "$R_RC" = 0 ]; then r_line REHEARSAL_PASS; else r_line REHEARSAL_FAIL; fi + [ -z "$R" ] || echo "rehearsal directory: $R" + exit "$R_RC" +} + +r_port_free() { + node -e 'const s=require("node:net").createServer();s.once("error",()=>process.exit(1));s.listen(Number(process.argv[1]),"127.0.0.1",()=>s.close(()=>process.exit(0)));' -- "$1" +} + +# The scratch config: the live one with dataRoot set to the scratch data root. +R_CONFIG_JS=' +const fs=require("node:fs");const [file,data]=process.argv.slice(1); +const c=JSON.parse(fs.readFileSync(file,"utf8"));c.dataRoot=data;process.stdout.write(JSON.stringify(c,null,2)+"\n");' + +# The scratch business file (mos-0962, mos-0963): the live file without its tracker.* variables, +# business and role, and with every credential reference, the roles and tracker.sync, moved to its +# own 0600 placeholder file. The service and its rotateBy or expires date stay. The tracker key +# stays because the loader requires it. Mode `normalize` prints the live file as this writes it, +# for the diff. +R_BUSINESS_JS=' +const fs=require("node:fs"),path=require("node:path");const [mode,live,out,ph]=process.argv.slice(1); +const b=JSON.parse(fs.readFileSync(live,"utf8")); +if(mode==="normalize"){process.stdout.write(JSON.stringify(b,null,2)+"\n");process.exit(0)} +const DATE={gitea:"rotateBy",vikunja:"expires"}; +const strip=(vars)=>{if(vars&&typeof vars==="object")for(const k of Object.keys(vars))if(k.startsWith("tracker."))delete vars[k]}; +const swap=(name,refs)=>{ + if(!refs||typeof refs!=="object")return; + for(const s of Object.keys(refs)){ + const file=path.join(ph,`${name}-${s}`),old=refs[s],ref={file}; + fs.writeFileSync(file,"rehearsal-placeholder-token",{mode:0o600,flag:"wx"}); + if(old&&typeof old==="object"&&Object.hasOwn(DATE,s)&&old[DATE[s]]!==undefined)ref[DATE[s]]=old[DATE[s]]; + refs[s]=ref; + } +}; +strip(b.vars); +for(const [name,r] of Object.entries(b.roles??{})){strip(r?.vars);swap(name,r?.credentials)} +swap("@sync",b.tracker?.sync?.credentials); +fs.writeFileSync(out,JSON.stringify(b,null,2)+"\n",{mode:0o600,flag:"wx"});' + +# The guard, before the host starts. It reads the scratch config through the new copy's own +# loadSystem and bootConfig, as the host will, and prints one line per check: +# data-root-live the data root is not the live one (mos-0961) +# sessions-live no sessions.json there names a running process (mos-0961) +# data-root-scratch the data root is the scratch one +# notify-off the notifier config is {"notifyVersion":1,"binding":null}, 0600 (mos-0962) +# tracker-configured bootConfig sets no trackers (mos-0962) +# credential-ref-live every credential reference is a file under placeholders/ (mos-0963) +# Exit 0 when all pass, 1 on the first FAIL. +R_GUARD_JS=' +const fs=require("node:fs"),path=require("node:path");const {pathToFileURL}=require("node:url"); +const [copy,id,live,scratch,ph]=process.argv.slice(1); +const real=(p)=>{try{return fs.realpathSync(p)}catch{return null}}; +const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,"utf8");return s[s.lastIndexOf(")")+2]!=="Z"}catch{return false}}; +const pass=(n,w)=>console.log(`PASS ${n}${w?": "+w:""}`); +const fail=(n,w)=>{console.log(`FAIL ${n}: ${w}`);process.exit(1)}; +(async()=>{ + let c,system,config; + try{c=await import(pathToFileURL(`${copy}/packages/cli/src/config.mjs`).href);system=c.loadSystem({env:process.env})} + catch(e){fail("scratch-config",String(e?.message??e).split("\n")[0])} + const root=real(system.dataRoot); + if(root===null)fail("data-root-scratch",`${system.dataRoot} does not exist`); + if(root===real(live))fail("data-root-live",`the scratch config names dataRoot ${system.dataRoot}, which resolves to the live data root ${live}`); + pass("data-root-live",`${system.dataRoot} is not the live data root`); + const sf=path.join(root,"bus-host","sessions.json"); + if(fs.existsSync(sf)){ + let doc;try{doc=JSON.parse(fs.readFileSync(sf,"utf8"))}catch{fail("sessions-live",`${sf} is unreadable`)} + if(!Array.isArray(doc?.sessions))fail("sessions-live",`${sf} is malformed`); + const running=doc.sessions.flatMap((s)=>[s?.pid,s?.runnerPid]).filter((p)=>Number.isSafeInteger(p)&&p>0&&alive(p)); + if(running.length)fail("sessions-live",`${sf} names running pid(s) ${running.join(", ")}`); + } + pass("sessions-live","no running session named in the data root"); + if(root!==real(scratch))fail("data-root-scratch",`the scratch config resolves to ${system.dataRoot}, not ${scratch}`); + pass("data-root-scratch"); + const nf=path.join(root,"notify",id,"notify.json"); + let st,doc;try{st=fs.lstatSync(nf);doc=JSON.parse(fs.readFileSync(nf,"utf8"))}catch{fail("notify-off",`${nf} is missing or not JSON`)} + if(!st.isFile()||(st.mode&0o777)!==0o600||JSON.stringify(doc)!==JSON.stringify({notifyVersion:1,binding:null}))fail("notify-off",`${nf} is not {"notifyVersion":1,"binding":null} at mode 0600`); + pass("notify-off","binding null, so no Discord bridge"); + try{config=c.bootConfig({system,businessId:id,env:process.env})}catch(e){fail("boot-config",String(e?.message??e).split("\n")[0])} + if(config.trackers!==undefined)fail("tracker-configured",`bootConfig sets trackers for ${Object.keys(config.trackers).join(", ")}`); + pass("tracker-configured","bootConfig sets no trackers"); + const b=config.businesses?.[id],phr=real(ph),refs=[]; + for(const [role,r] of Object.entries(b?.roles??{}))for(const [s,ref] of Object.entries(r?.credentials??{}))refs.push([`${role} ${s}`,ref]); + for(const [s,ref] of Object.entries(b?.tracker?.sync?.credentials??{}))refs.push([`@sync ${s}`,ref]); + for(const [what,ref] of refs){ + if(ref?.env!==undefined)fail("credential-ref-live",`${what} is env ${ref.env}`); + const f=typeof ref?.file==="string"?real(ref.file):null; + if(f===null||phr===null||path.dirname(f)!==phr)fail("credential-ref-live",`${what} is ${ref?.file}, not a file in ${ph}`); + } + pass("credential-ref-live",`${refs.length} reference(s), each a placeholder`); +})().catch((e)=>fail("guard",String(e?.message??e).split("\n")[0]));' + +# Opens a data root's store with a copy's Store, as the bus does at start, with no migration +# (de352dfaf's verify-open.mjs). Prints OPEN_OK or REFUSED . +R_OPEN_JS=' +const {pathToFileURL}=require("node:url");const [copy,root]=process.argv.slice(1); +(async()=>{ + const {Store}=await import(pathToFileURL(`${copy}/packages/bus/src/store.mjs`).href); + let s; + try{s=new Store(root);process.stdout.write(`OPEN_OK ${s.get("SELECT value FROM meta WHERE key=?","schema_version").value}\n`)} + catch(e){process.stdout.write(`REFUSED ${e.message}\n`)} + finally{s?.close()} +})();' +r_open() { node -e "$R_OPEN_JS" -- "$1" "$2" 2>&1 || true; } + +# Every process this rehearsal starts gets only HOME, PATH and the scratch MOSAIC_CONFIG. +r_env() { env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" "$@"; } + +# The de352dfaf checks, when the backup's schema is not the new copy's. +r_migration() { + local copy="$COPIES/$TO" old="$COPIES/$P_COPY" v out rc + v="$(node -e "$STORE_JS" -- version "$R/data/bus/bus.sqlite" 2>&1)" || r_abort store-unreadable "$v" + if [ "$v" = "$T_SCHEMA" ]; then r_pass migration "not needed: the backup is already $v"; return 0; fi + in_chain "$v" || r_abort store-newer "the backup is $v, which $TO's migration chain does not start from" + (umask 077; mkdir "$R/refuse" "$R/refuse/bus" && cp "$R_BK"/bus/* "$R/refuse/bus/") || r_abort scratch-build "cannot copy the backup to $R/refuse" + out="$(r_open "$copy" "$R/refuse")" + [ "$out" = "REFUSED schema-mismatch" ] || r_abort new-refuses-old-store "$TO opened the $v store: $out" + r_pass new-refuses-old-store "$TO refuses the $v store at a normal open" + rc=0; out="$(r_env node "$copy/packages/bus/src/migrate-schema.mjs" "$R/data" 2>&1)" || rc=$? + printf '%s\n' "$out" >> "$R/migrate.log" + [ "$rc" = 0 ] && [ "$(tail -n 1 <<< "$out")" = "bus schema $T_SCHEMA ready" ] || r_abort migrate "migrate-schema.mjs exit $rc: $(tail -n 1 <<< "$out")" + r_pass migrate "$v to $T_SCHEMA" + out="$(r_open "$copy" "$R/data")" + [ "$out" = "OPEN_OK $T_SCHEMA" ] || r_abort new-opens-migrated "$out" + r_pass new-opens-migrated + rc=0; out="$(r_env node "$copy/packages/bus/src/migrate-schema.mjs" "$R/data" 2>&1)" || rc=$? + printf '%s\n' "$out" >> "$R/migrate.log" + [ "$rc" = 0 ] && [ "$(tail -n 1 <<< "$out")" = "bus schema $T_SCHEMA ready" ] || r_abort migrate-again "second migrate-schema.mjs exit $rc: $(tail -n 1 <<< "$out")" + r_pass migrate-again "a second run is a no-op" + out="$(r_open "$old" "$R/data")" + [ "$out" = "REFUSED schema-mismatch" ] || r_abort old-refuses-migrated "the pointer's copy $P_COPY opened the $T_SCHEMA store: $out" + r_pass old-refuses-migrated "$P_COPY refuses the $T_SCHEMA store" +} + +# One pass of D1's gate checks but the PM's, against the processes this run started. Sets R_GATE +# to the first failing check and R_WHY to what it saw. +r_gate_once() { + local copy="$COPIES/$TO" out + R_GATE=host-up R_WHY="" + r_alive "$R_HOST_PID" || { R_WHY="the host (pid $R_HOST_PID) exited; see $R/host.log"; return 1; } + grep -q "^bus host up: business $BUSINESS, .*, launches disabled\$" "$R/host.log" || { R_WHY="no \"bus host up ... launches disabled\" line in $R/host.log"; return 1; } + R_GATE=bus-status + out="$(r_env "$copy/scripts/mosaic" bus status --json 2>/dev/null)" || { R_WHY="bus status failed"; return 1; } + R_WHY="$(node -e ' +const [b,main,since]=process.argv.slice(2),s=JSON.parse(process.argv[1]); +const why=!s.host?"no host":s.host.business!==b?`host for ${s.host.business}`:s.host.live!==true?"host not live":String(s.host.pid)!==main?`host pid ${s.host.pid} is not ${main}, the host this run started`:!s.boot||s.boot.ok!==true?"boot status not ok":s.boot.business!==b?`boot status for ${s.boot.business}`:!(Date.parse(s.boot.at)>=Number(since))?"boot status is from before this start":""; +process.stdout.write(why);' -- "$out" "$BUSINESS" "$R_HOST_PID" "$START_MS" 2>&1)" || { R_WHY="bus status unreadable"; return 1; } + [ -z "$R_WHY" ] || return 1 + R_GATE=console-up + r_alive "$R_CONSOLE_PID" || { R_WHY="the Console (pid $R_CONSOLE_PID) exited; see $R/console.log"; return 1; } + grep -qxF "Mosaic Console: http://127.0.0.1:$R_PORT/" "$R/console.log" || { R_WHY="no \"Mosaic Console: http://127.0.0.1:$R_PORT/\" line in $R/console.log"; return 1; } + R_GATE=healthz + node -e ' +fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>process.exit(r.status===200?0:1),()=>process.exit(1));' -- "$R_PORT" || return 1 + R_GATE=copy-match + tr '\0' ' ' < "/proc/$R_HOST_PID/cmdline" | grep -qF " $copy/packages/cli/src/cli.mjs bus start $BUSINESS --no-launch" || { R_WHY="the host's command line is not $copy's cli.mjs bus start $BUSINESS --no-launch"; return 1; } + tr '\0' ' ' < "/proc/$R_CONSOLE_PID/cmdline" | grep -qF " $copy/packages/webui/src/cli.mjs serve " || { R_WHY="the Console's command line is not $copy's webui cli.mjs serve"; return 1; } + R_GATE="" +} + +r_gate() { + local deadline=$((SECONDS + TIMEOUT)) + until r_gate_once; do + [ "$SECONDS" -lt "$deadline" ] || { r_fail "$R_GATE" "${R_WHY:-no answer within ${TIMEOUT}s}"; return 1; } + sleep 1 + done + r_pass gate "host-up, bus-status, console-up, healthz and copy-match on $TO" +} + +# No role session: the host wrote no sessions.json and no launches directory or log. +r_no_session() { + local found="" + [ ! -e "$R/data/bus-host/sessions.json" ] || found+=" bus-host/sessions.json" + [ ! -e "$R/data/launches" ] || found+=" launches" + if [ -z "$found" ]; then r_pass no-session "no sessions.json and no launches under $R/data"; else r_fail no-session "found$found under $R/data"; fi +} + +do_rehearse() { + TO="$ID" + trap 'r_fail signal "the rehearsal was interrupted"; r_end' INT TERM HUP + read_pointer || r_abort no-pointer "no $POINTER; the pointer's copy is the old copy the checks use" + [ "$P_BUSINESS" = "$BUSINESS" ] || r_abort business-mismatch "the pointer is for $P_BUSINESS" + copy_exists "$TO" 2>/dev/null || r_abort copy-missing "no copy $TO in $COPIES" + copy_ok "$TO" 2>/dev/null || r_abort copy-unreachable "$TO is not reachable from refactor and origin/refactor" + target_schema "$TO" || r_abort copy-schema-unknown "cannot read SCHEMA_VERSION and its chain from $COPIES/$TO/packages/bus/src/store.mjs" + # mos-0961: an older copy's host has no --no-launch and would start the PM on its own. + grep -qF -- '"--no-launch"' "$COPIES/$TO/packages/cli/src/cli.mjs" 2>/dev/null || r_abort copy-without-no-launch "$TO's bus host has no --no-launch flag" + r_pass copy "$TO, schema $T_SCHEMA" + local i + local -a reserved=(7332 7333 "$(console_port)") + for ((i = 0; i < ${#P_CONSOLE[@]}; i++)); do [ "${P_CONSOLE[i]}" != --tester-port ] || reserved+=("${P_CONSOLE[i + 1]:-}"); done + for i in "${reserved[@]}"; do [ "$((10#$R_PORT))" != "$i" ] || r_abort port-reserved "--port $R_PORT is a live Console port (7332, 7333 and the pointer's: ${reserved[*]:2})"; done + r_port_free "$R_PORT" || r_abort port-in-use "127.0.0.1:$R_PORT is in use; the rehearsal takes no other port" + r_pass port "127.0.0.1:$R_PORT is free" + case "$R_BACKUP" in /*) R_BK="$R_BACKUP" ;; *) R_BK="$DATA/backups/stack/$R_BACKUP" ;; esac + [ -f "$R_BK/bus/bus.sqlite" ] && [ -f "$R_BK/SHA256SUMS" ] || r_abort backup-missing "$R_BK has no bus/bus.sqlite and SHA256SUMS" + (cd "$R_BK/bus" && sha256sum -c --quiet ../SHA256SUMS) >/dev/null 2>&1 || r_abort backup-sums "$R_BK/bus does not match its SHA256SUMS" + r_pass backup "$R_BK matches its SHA256SUMS" + LIVE_CFG="$(realpath -m "${MOSAIC_CONFIG:-$HOME/.config/mosaic-dev/config.json}")" + LIVE_BIZ="$(dirname "$LIVE_CFG")/businesses/$BUSINESS.json" + [ -f "$LIVE_BIZ" ] || r_abort business-missing "no $LIVE_BIZ" + + local stamp before live + before="$(r_tree_sums "$R_BK")" && live="$(sha256sum "$LIVE_CFG" "$LIVE_BIZ" 2>&1)" || r_abort sums "cannot read $R_BK or the live config files" + stamp="$(date -u +%Y%m%dT%H%M%SZ)-${TO:0:12}" + (umask 077; mkdir -p "$DATA/rehearsals" && mkdir "$DATA/rehearsals/$stamp") || r_abort rehearsal-dir "cannot make $DATA/rehearsals/$stamp (one rehearsal per second and copy)" + R="$DATA/rehearsals/$stamp" STEPS="$DATA/rehearsals/$stamp/steps.log" + printf '%s\n' "${R_EARLY[@]}" > "$STEPS" && printf '%s\n' "$before" > "$R/backup.before" && printf '%s\n' "$live" > "$R/live.before" || + r_abort scratch-build "cannot write the sums to $R" + ( + umask 077 + mkdir "$R/config" "$R/config/businesses" "$R/placeholders" "$R/data" "$R/data/bus" "$R/data/notify" "$R/data/notify/$BUSINESS" && + cp "$R_BK"/bus/* "$R/data/bus/" && + node -e "$R_CONFIG_JS" -- "$LIVE_CFG" "$R/data" > "$R/config/config.json" && + node -e "$R_BUSINESS_JS" -- write "$LIVE_BIZ" "$R/config/businesses/$BUSINESS.json" "$R/placeholders" && + printf '{"notifyVersion":1,"binding":null}\n' > "$R/data/notify/$BUSINESS/notify.json" + ) 2>> "$STEPS" || r_abort scratch-build "cannot build the scratch root under $R; see $STEPS" + local rc=0 + diff -u --label "$LIVE_BIZ (as JSON.stringify writes it)" --label "$R/config/businesses/$BUSINESS.json" \ + <(node -e "$R_BUSINESS_JS" -- normalize "$LIVE_BIZ") "$R/config/businesses/$BUSINESS.json" > "$R/business.diff" || rc=$? + [ "$rc" -le 1 ] || r_abort scratch-build "cannot write $R/business.diff" + r_pass scratch-root "$R/data from the backup, config and business file under $R/config, business.diff beside them" + + local out + rc=0; out="$(r_env node -e "$R_GUARD_JS" -- "$COPIES/$TO" "$BUSINESS" "$DATA" "$R/data" "$R/placeholders" 2>&1)" || rc=$? + while IFS= read -r i; do r_line "$i"; done <<< "$out" + [ "$rc" = 0 ] || { R_RC=1; r_end; } + + r_migration + + START_MS="$(date +%s%3N)" + R_STARTED=1 + # env, not r_env: a function in the background is a subshell, and $! has to be the process + # itself (env execs scripts/mosaic, which execs node), for the bus-status pid check and the stop. + env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" "$COPIES/$TO/scripts/mosaic" bus start "$BUSINESS" --no-launch > "$R/host.log" 2>&1 & + R_HOST_PID=$! + env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" node "$COPIES/$TO/packages/webui/src/cli.mjs" serve --port "$R_PORT" --business "$BUSINESS" > "$R/console.log" 2>&1 & + R_CONSOLE_PID=$! + if r_gate; then r_no_session; fi + r_end +} + # --- Dispatch --- [ $# -gt 0 ] || usage OP="$1"; shift ID="" case "$OP" in - activate) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;; + activate|rehearse) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;; adopt|pm-on|rollback|status) [ $# -ge 1 ] || usage; BUSINESS="$1"; shift ;; -h|--help) usage 0 ;; *) usage ;; @@ -1179,10 +1479,12 @@ BUS_UNIT="mosaic-bus@$BUSINESS.service" FROM="" TO="" ACT="" RS="" POST="" BACKUP="" TAKEN="" RESTORED="" STEPS="" COUNT="" UNCLEAN="" FAIL="" STORE_SCHEMA="" START_EPOCH="" START_MS=0 GATE="" GATE_WHY="" S_CHANGED="" P_COPY="" P_PM=0 P_PENDING=0 P_SCHEMA="" P_CLAUDE="" P_CONSOLE=() RUN_PM=0 RUN_PENDING=0 RUN_CONSOLE=() DROP_TESTER="" PROOF_PLAN="" PROOF_STEP="" PLAN_REFUSE=() +R="" R_RC=0 R_EARLY=() R_BK="" R_HOST_PID="" R_CONSOLE_PID="" R_STARTED="" LIVE_CFG="" LIVE_BIZ="" case "$OP" in adopt) do_adopt ;; activate) do_activate ;; pm-on) do_pm_on ;; rollback) do_rollback ;; status) do_status ;; + rehearse) do_rehearse ;; esac