fix(cli): M3 review round: re-pin migration, rehearsal provider (#1873)

An installed unit must equal the render but for the copy id, so a
re-pin that adds --sso refused render-refused. case2-unit gains --check
(every render check, nothing written) and --check-new (a first install's
checks). stack-upgrade runs both checks before anything stops, and
--next-args reads root's <instance>.args.next: equal before -- but for
--copy, different after it. After the stage it moves the installed unit
files into a new <instance>.replaced.XXXXXX directory and renders a
first install. A failure moves them back unchanged and restarts the old
copy, and confirm swaps the args file. The run sheet gives the commands,
the rollback past the migration included.

A stamped rehearsal now starts an isolated provider on 127.0.0.1 port 0
with a certificate and keys made for the run. It serves discovery and a
JWKS for the live provider ids and answers 403 to authorize and token.
Only the scratch Console trusts its certificate. The gate requires each
provider ready (sso-ready); provider-discovery requires discovery and
JWKS fetches and no code or token request. The provider stops with the
run, and by itself once the run's shell is gone.

The system-path test helper asserts the log's from copy again. The
pinned-copy sign-in requests are bounded, and cleanup waits for each
Console to exit.
This commit is contained in:
jason.woltje committed 2026-11-10 21:38:17 -06:00
1 parent db822c5b9d
commit 9acc378555
10 files changed
+766 -116

No files matched your search

+57 -13
View File
@@ -808,9 +808,10 @@ the port and the backup's `SHA256SUMS`. Then it makes
- `steps.log` holds the same PASS and FAIL lines the run prints, with times;
`host.log`, `console.log` and, after a migration, `migrate.log` sit beside it.
- When the live config has `console.sso` (M3, #1873), the scratch copy keeps
its provider ids, client ids, algorithms and subjects, and moves the public
origin and every issuer to `rehearsal.invalid`, a name that never resolves.
Each client secret slot is a 0600 file under `config/secrets/mosaic-stack/`
its provider ids, client ids, algorithms and subjects and moves the public
origin to `rehearsal.invalid`, a name that never resolves. For a stamped
target each issuer becomes `ORIGIN/ID/` at the rehearsal provider below;
otherwise it moves to `rehearsal.invalid` too. Each client secret slot is a 0600 file under `config/secrets/mosaic-stack/`
holding `rehearsal-placeholder-secret`; the live secret is never read.
A stamped target copy (one with `.stack-build.json`) serves only with `--sso`,
@@ -819,10 +820,27 @@ so its rehearsal needs the live instance's human-made SSO steps. Without a live
beside the config it refuses `accounts-not-initialized`. Both name the run-sheet
step. Otherwise it copies the live `accounts.json` into `config/`, writes
`data/.deployed` with the target copy's own `writeDeployment`, prints
`PASS deployment`, and starts the Console with `--sso`. The scratch Console
runs every startup check, finds no provider at `rehearsal.invalid`, and starts
degraded. The live `accounts.json` joins the end check `live-config-unchanged`.
The rehearsal adds no subject and no account.
`PASS deployment`, and starts the Console with `--sso`.
Before that, a stamped run starts a rehearsal provider and prints
`PASS provider-start`. It is a node process on `127.0.0.1` port 0 that serves
HTTPS with a certificate made for this run, under `provider/` (mode 0700). For
each live provider id it serves discovery and a JWKS whose public key it made
in memory for the first algorithm it can make from that provider's list. Its
authorize and token routes answer 403: it issues no code and no token, and
it reads no live secret. It logs each request to `provider/requests.log`,
deletes its TLS key file once it has read it, and exits on SIGTERM or when
the rehearsal's process is gone. `provider-start` fails when it exits early,
writes no origin within the timeout, writes an origin that is not
`https://127.0.0.1:PORT`, or leaves its key file. Only the scratch Console gets
`NODE_EXTRA_CA_CERTS` naming `provider/cert.pem`; the host does not.
The scratch Console runs every startup check against that provider. The gate
adds `sso-ready`: `/healthz` must report each provider `ready`. After the gate,
`provider-discovery` requires a 200 discovery and JWKS request per provider and
no authorize or token request. The end stops the provider with the Console and
host and prints `provider-stopped`. The live `accounts.json` joins the end check
`live-config-unchanged`. The rehearsal adds no subject and no account.
A guard then loads the scratch config with the target's code and refuses
with a named check: `data-root-live` when the scratch data root resolves to
@@ -830,8 +848,10 @@ the live one, `sessions-live` when a `sessions.json` there names a running
pid (the run never deletes the file), `data-root-scratch`, `notify-off`,
`tracker-configured` when `bootConfig` still sets a tracker,
`credential-ref-live` for an env reference or a file outside `placeholders/`,
and `sso-isolated` when a `console.sso` origin or issuer is not on
`rehearsal.invalid` or a slot is not its scratch placeholder.
and `sso-isolated` when a `console.sso` origin is not on `rehearsal.invalid`,
an issuer is not `ORIGIN/ID/` at the rehearsal provider (not on
`rehearsal.invalid` for an unstamped target), or a slot is not its scratch
placeholder.
When the backup's schema differs from the target's, the run checks that the
target refuses the store, migrates it, opens it, migrates it again as a
no-op, and that the pointer's copy refuses the migrated store.
@@ -840,7 +860,8 @@ It starts `scripts/mosaic bus start BUSINESS --no-launch` and the Console's
`serve --port N --business BUSINESS` from the target copy, without `--send`.
The gate needs the host's `launches disabled` line, a live `bus status` host
with this run's pid and a boot from this start, the Console's address line,
`/healthz` 200, and both command lines on the target copy. Then it checks
`/healthz` 200 (and `sso-ready` for a stamped target), and both command lines
on the target copy. Then it checks
that the host wrote no `sessions.json` and no `launches`, stops the Console
and the host (SIGTERM, then SIGKILL and a FAIL after the timeout), checks that
the host released `writer.lock`, and compares the backup and the live config
@@ -927,8 +948,9 @@ the target migrates the store and the run lacks `--migrate` (row 233). Under
```
scripts/stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
[--rehearse-port N] [--timeout S]
scripts/stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
[--rehearse-port N] [--next-args] [--timeout S]
scripts/stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--next-args]
[--timeout S]
```
Root runs it from a root-owned checkout of the reviewed commit, never from a
@@ -968,7 +990,13 @@ user's clone `<home>/src/mosaic-stack`:
(`args-differ`), and
`activate SHA BUSINESS --units external --plan --expect-from OLD` exits 0
(`plan-refused`, which covers `open-activation`, `migrate-required` and
`pointer-differs`).
`pointer-differs`). Then the render checks (#1873 M3):
`case2-unit.sh --check` renders OLD from the args file and compares the
installed units with it, so a rollback could render them, then checks
SHA's render the same way. Either refusal is `render-refused`. A re-pin
that changes the serve flags fails the second check, because an
installed unit must equal the render but for the copy id (row 195); it
takes `--next-args`, below.
4. The rehearsal, `rehearse SHA BUSINESS --backup last --port 7339`
(`--rehearse-port`). Exit 0 goes on. Exit 3 means no backup in the log
and goes on only with `--no-rehearsal`, logging `rehearsal-skipped`;
@@ -983,6 +1011,22 @@ user's clone `<home>/src/mosaic-stack`:
still names the old one, and root moves the new file (the result line
names it) into place by hand.
`--next-args` changes the serve flags, as the SSO migration does
(`docs/guides/host-run-sheet.md` section 7). The run reads
`/etc/mosaic-stack/dev.args.next` as it reads the args file
(`args-unreadable`). It must equal the args file before `--` but for
`--copy`'s value (`next-args-differ`) and differ from it after `--`
(`next-args-same`). The target's check is `case2-unit.sh --check-new`, a
first install's checks from the next args, without the installed units'.
After A3, root moves the installed unit files into a new 0700 directory
`/etc/mosaic-stack/dev.replaced.XXXXXX` (step `units-aside`), so A4 renders
as a first install. A failure from there moves them back unchanged
(`units-back`), runs `daemon-reload` and starts the old units with no render.
After A5 the old args file is copied into the replaced directory, the next
args file is removed (step `next-args`), and the new args file, made from
the next args, moves into place. The result line names the replaced
directory. Nothing removes it.
A failure in A2 or a stage that exits 3 starts the old units again and gates
them (A2f). A later failure stops both units, runs `revert`, renders the old
copy and runs `confirm --rollback OLD --cause A3|A4|A5`. `rollback` is the
+34 -6
View File
@@ -446,8 +446,36 @@ stack picks or invents a subject: the subjects come from `console.sso`, which a
4. Provision the dedicated uid. `mosaic` must keep section 1's shape: a non-login shell, the only member
of its primary group, and owner of its config directory and data root. An accounts instance never runs
under a human or development seat uid, and never under the `webui-service.sh` user units.
5. Re-pin. Put `--sso` after `--` in the args file (`factory`: `--sso --send`). For `library`, also remove
the proxy-preview trio and the proof pair. Then run `case2-unit.sh` or `stack-upgrade.sh upgrade`.
5. Re-pin with `--next-args`. Leave `/etc/mosaic-stack/dev.args` as it is. Write the new flags to
`/etc/mosaic-stack/dev.args.next`, root's, mode 0640: the same lines before `--`, and after `--` the
serve flags with `--sso` added (`factory`: `--sso --send`). For `library`, also remove the
proxy-preview trio (`--public-host`, `--auth-header`, `--allow-user`) and the proof pair
(`--proof-header`, `--proof-file`). Then run
`stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA --next-args`.
A plain re-pin refuses `render-refused` here, because an installed unit must equal the render but for
the copy id. With `--next-args`, before anything stops, the driver checks the installed copy against
`dev.args` and the target as a first install from `dev.args.next`. It refuses `next-args-differ` when
the files differ before `--` (other than `--copy`), and `next-args-same` when they do not differ after
it. After the stage it moves the installed unit files into a new 0700 directory
`/etc/mosaic-stack/dev.replaced.XXXXXX` and renders the target as a first install. A failure from
there moves them back unchanged and starts the old copy again. After `confirm` passes,
`dev.args.next` becomes `dev.args`, and the replaced directory keeps the old unit files and the old
`dev.args`. The result line names the directory. Nothing removes it; root removes it by hand once no
rollback needs it.
To roll back past the migration, put the replaced args back as the next args and roll back with
`--next-args`:
```sh
install -m 0640 -o root -g root /etc/mosaic-stack/dev.replaced.XXXXXX/dev.args /etc/mosaic-stack/dev.args.next
stack-upgrade.sh rollback --instance dev --to <old copy id> --next-args
```
For `library` this is the only rollback to a copy older than M3: `case2-unit.sh` refuses `--sso` for
an unstamped library copy (#416), so the old copy comes back with its old flags, proxy-preview
admission included, as before the migration. A `factory` rollback may instead keep `--sso --send`
with a plain `rollback`.
If a step is missing, `case2-unit.sh` refuses before it writes a unit and prints the missing steps,
numbered in this order, for example:
@@ -461,11 +489,11 @@ case2-unit: refusing the units for <sha>: a stamped copy's Console starts only w
```
Without `console.sso`, the render does not check `accounts.json`, because its subjects name providers SSO
has not configured yet. The next render after step 2 checks it. Rollback keeps `--sso`, accounts.json and
`.deployed` in place (`docs/plans/2026-11-10_accounts-design.md`, section 7).
has not configured yet. The next render after step 2 checks it. A rollback never removes accounts.json,
its receipts or `.deployed` (`docs/plans/2026-11-10_accounts-design.md`, section 7).
Precheck line: none. `stack-upgrade.sh` refuses `args-unreadable` or `console-unit-legacy` on any of these
rules, before it stops a unit.
Precheck line: none. `stack-upgrade.sh` refuses `args-unreadable`, `console-unit-legacy`, `render-refused`,
`next-args-differ` or `next-args-same` on any of these rules, before it stops a unit.
## 8. Enrollment, once per host
+9 -1
View File
@@ -1298,7 +1298,8 @@ business file.
### Rollback
Rollback keeps OIDC. Nothing in it removes `--sso` or turns sign-in off.
Rollback keeps OIDC. Nothing in it removes `--sso` or turns sign-in off,
except the library rollback past the migration below.
- The supported rollback is a pinned code and config rollback: install the
earlier stack copy with the same unit flags, including `--sso`, through
@@ -1306,6 +1307,13 @@ Rollback keeps OIDC. Nothing in it removes `--sso` or turns sign-in off.
dedicated system units. The accounts instance must not use the
`webui-service.sh` user-unit path. Leave accounts.json, its receipts and `.deployed` in
place. BASE code ignores them, and the next roll-forward needs them.
- A library host rolled back to a copy older than M3 is the exception.
`case2-unit.sh` refuses `--sso` for an unstamped library copy (#416), so
that rollback renders the old copy from the args file the migration
replaced, through `stack-upgrade.sh rollback ... --next-args`
(`docs/guides/host-run-sheet.md` section 7). The old copy comes back with
its old flags, proxy-preview admission included, as before the migration.
accounts.json and `.deployed` stay in place.
- Before the first accounts org exists, that rollback restores today's
behavior completely. `console.sso` owners sign in as before, and FED R6
users keep their root grants. Account-only users and pending invites stop
+60 -1
View File
@@ -364,7 +364,7 @@ test("the unit takes --org or --business on either side of --, renders --busines
test("the unit refuses a bad selector in either scope, and an outer and serve pair that differ, before writing a unit", (t) => {
const h = hub(t);
const serve = ["--port", "443", "--business", "acme"];
const usage = "usage: case2-unit.sh --user NAME --home DIR --business ID --copy ID --node PATH [--profile factory|library] [--proxy] [--pm] [--claude PATH] -- SERVE FLAGS\n";
const usage = "usage: case2-unit.sh [--check|--check-new] --user NAME --home DIR --business ID --copy ID --node PATH [--profile factory|library] [--proxy] [--pm] [--claude PATH] -- SERVE FLAGS\n";
const head = ["--user", "mosaic", "--home", h.home, "--copy", COPY_A, "--node", USR_NODE, "--profile", "factory"];
for (const [label, argv, code, err] of [
["outer unequal, old first", [...head, "--business", "acme", "--org", "beta", "--", ...serve], 4, "case2-unit: --business acme and --org beta conflict\n"],
@@ -886,6 +886,65 @@ test("M3 a stamped copy's query refuses on noise or a failed query, before any w
}
});
// #1873 M3: stack-upgrade.sh checks each render before it stops anything. --check runs a render's
// checks, the installed units' included, and writes nothing; --check-new skips only the installed
// units' checks, for --next-args, which moves them aside before its render.
const checkArgv = (h, { mode = "--check", profile = "library", copy = COPY_A, flags } = {}) =>
[mode, "--user", "mosaic", "--home", h.home, "--business", "acme", "--copy", copy, "--node", USR_NODE, "--profile", profile, "--",
...(flags ?? (profile === "factory" ? factoryFlags() : libraryFlags(h)))];
const unitsNow = (h) => [busFile(h), webuiFile(h)].map((f) => (existsSync(f) ? read(f) : null));
const logNow = (h) => (existsSync(h.log) ? read(h.log) : "");
test("#1873 --check passes on a matching install and on a first install, refuses an edited unit, and never writes or calls systemctl", (t) => {
const h = hub(t);
const empty = unit(h, { argv: checkArgv(h) });
assert.equal(empty.code, 0, empty.out + empty.err);
assert.equal(empty.out, `case2-unit: check passed: the units for copy ${COPY_A} render, and the installed ones match them; nothing was written\n`);
noUnits(h);
assert.equal(unit(h).code, 0);
const units = unitsNow(h), log = logNow(h);
for (const copy of [COPY_A, COPY_B]) {
const r = unit(h, { argv: checkArgv(h, { copy }) });
assert.equal(r.code, 0, r.out + r.err);
assert.deepEqual(unitsNow(h), units, copy);
assert.equal(logNow(h), log, `${copy}: no systemctl call`);
}
// The installed-unit match stays: an edit, or other serve flags, refuses as the render would.
writeFileSync(busFile(h), units[0].replace(/^Description=.*$/m, "Description=edited"));
refuses(unit(h, { argv: checkArgv(h, { copy: COPY_B }) }), /differs from the render in more than the copy id/);
writeFileSync(busFile(h), units[0]);
const flags = libraryFlags(h).map((f) => (f === "melanie" ? "someone" : f));
refuses(unit(h, { argv: checkArgv(h, { copy: COPY_B, flags }) }), /installed .*: ExecStart= must be exactly once/);
assert.deepEqual(unitsNow(h), units);
assert.equal(logNow(h), log);
});
for (const profile of ["library", "factory"]) {
test(`#1873 an installed pre-M3 ${profile} unit: --check and the render refuse a stamped copy's SSO flags, and --check-new passes with the prerequisites, writing nothing`, (t) => {
const h = hub(t);
assert.equal(unit(h, { profile }).code, 0);
stamp(h, COPY_B);
stampSso(h);
stampAccounts(h);
stampDeployed(h);
const flags = profile === "factory" ? STAMP_FACTORY : STAMP_LIBRARY;
const units = unitsNow(h), log = logNow(h);
refuses(unit(h, { argv: checkArgv(h, { profile, copy: COPY_B, flags }) }), /installed .*: ExecStart= must be exactly once/);
refuses(unit(h, { profile, copy: COPY_B, flags }), /installed .*: ExecStart= must be exactly once/);
const r = unit(h, { argv: checkArgv(h, { mode: "--check-new", profile, copy: COPY_B, flags }) });
assert.equal(r.code, 0, r.out + r.err);
assert.equal(r.out, `case2-unit: check passed: the units for copy ${COPY_B} render as a first install; the installed ones were not compared; nothing was written\n`);
// --check-new still runs the stamped copy's checks: the old flags, or a missing prerequisite, refuse.
refuses(unit(h, { argv: checkArgv(h, { mode: "--check-new", profile, copy: COPY_B }) }), /a stamped copy's Console starts only with SSO accounts|is a stamped copy, and proxy-preview admission/);
rmSync(accountsFile(h));
refuses(unit(h, { argv: checkArgv(h, { mode: "--check-new", profile, copy: COPY_B, flags }) }), /run `mosaic accounts init` as mosaic, a human verb/);
assert.deepEqual(unitsNow(h), units);
assert.equal(logNow(h), log, "no systemctl call");
assert.equal(existsSync(accountsFile(h)), false, "the check never writes accounts.json");
assert.equal(statSync(deployedFile(h)).mode & 0o777, 0o444);
});
}
test("row253 factory refuses retired TCP and a value after the socket marker", (t) => {
const h = hub(t);
refuses(unit(h, { profile: "factory", flags: ['--port','443','--business','acme','--tester-port','7333'] }), /case2-unit.sh retires tester TCP; use --tester-socket/);
+188 -39
View File
@@ -1578,15 +1578,37 @@ function rootUp(e,copy,flags){
.replace(/^ExecStart=.*$/m,`ExecStart=${node} ${dir}/packages/webui/src/cli.mjs serve --checkout ${repo} ${flags.join(' ')}`));
execFileSync(join(SBIN,'systemctl'),['--user','start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:'0'}});sysctl(e,'start',e.con);
}
// One system-path run: OP (activate TO, or rollback --to TO) staged, root's units for TO with FLAGS, confirm.
// One system-path run: OP (activate TO, or rollback --to TO) planned and staged from the pointer's
// copy, root's units for TO with FLAGS, confirm. The plan, the staged set and both log lines are this
// script's output; the unit flags are root's render (case2-unit.test.mjs pins that render), so the
// pointer pinning them is confirm's read of root's unit, not a conversion of this script's.
async function viaSystem(e,op,to,flags,...more){
const x=X(e);rootDown(e);
let r=await prun(x,...(op==='activate'?['activate',to,'acme']:['rollback','acme','--to',to]),...more);assert.equal(r.code,0,r.out+r.err);
const act=logLines(e).at(-1).activation;
const x=X(e),from=pointer(e).copy,p0=readFileSync(e.pointer,'utf8'),w0=sha(join(e.bin,e.wrapper));
const args=op==='activate'?['activate',to,'acme']:['rollback','acme','--to',to],inst=e.flags.length?`${e.flags.join(' ')} `:'';
rootDown(e);
const con0=sha(join(e.units,e.con));
const out=await planOnly(x,0,...args,...more),plan=out.join('\n');
assert.deepEqual([out.find((l)=>l.startsWith('plan: ')),out.find((l)=>l.startsWith('from: ')),out.find((l)=>l.startsWith('to: '))],[`plan: ${op} for acme`,`from: ${from}`,`to: ${to}`],plan);
assert.ok(out.includes(`units: external; root installs the units for ${to} and starts them, then confirm gates them`),plan);
assert.ok(out.includes(`CLI: scripts/stack-copy.sh wrapper ${to} ${inst}--copies ${COPIES} --bin ${e.bin}`),plan);
// Root renders the units, so the plan names no unit, Console flag or refusal of this script's.
assert.deepEqual(out.filter((l)=>/^(bus|pm|proof|send|sso|Console):|^(drop|refuses) /.test(l)),[],plan);
let r=await prun(x,...args,...more);assert.equal(r.code,0,r.out+r.err);
const st=logLines(e).at(-1),act=st.activation,rs=setOf(e,act);
assert.deepEqual([st.op,st.from,st.to,st.result,st.units],[op,from,to,'started','external']);
assert.equal(r.out.split('\n').at(-2),`stack-release: ${op} staged ${to} for acme as ${act}; root installs and starts the units, then runs confirm ${to} acme --units external`);
// Staged, not live: the pointer stays; the set holds the pointer, the wrapper and the Console unit root stopped.
assert.equal(readFileSync(e.pointer,'utf8'),p0);
assert.equal(readFileSync(join(rs,'AWAITING-UNITS'),'utf8'),`${to}\n`);
assert.equal(readFileSync(join(rs,'pointer'),'utf8'),p0);assert.equal(sha(join(rs,'wrapper')),w0);
assert.ok(readFileSync(join(rs,'UNITS.sha256'),'utf8').includes(`${con0} ${join(e.units,e.con)}\n`));
assert.ok(readFileSync(join(e.bin,e.wrapper),'utf8').includes(join(COPIES,to)),'the wrapper names the target copy');
rootUp(e,to,flags);
r=await prun(x,'confirm',to,'acme');assert.equal(r.code,0,r.out+r.err);
assert.equal(r.out.split('\n').at(-2),`stack-release: ${op} ok: ${to} is live for acme`);
const l=logLines(e).at(-1);assert.deepEqual([l.op,l.to,l.result,l.units,l.activation],[op,to,'ok','external',act]);
const l=logLines(e).at(-1);assert.deepEqual([l.op,l.from,l.to,l.result,l.units,l.activation],[op,from,to,'ok','external',act]);
assert.ok(!existsSync(join(rs,'AWAITING-UNITS')));
assert.deepEqual([pointer(e).copy,pointer(e).console],[to,flags]);
return act;
}
@@ -2102,6 +2124,7 @@ function rHost(){
}
if(a[0]!=='bus'||a[1]!=='start'||!a.slice(3).every((f)=>flags.includes(f))){console.error(`fixture host: ${a.join(' ')}`);process.exit(4)}
fs.appendFileSync(path.join(home,'host-args.log'),`${a.join(' ')}\n`);
fs.appendFileSync(path.join(home,'host-ca.log'),`${process.env.NODE_EXTRA_CA_CERTS??'none'}\n`);
if(fault(`host-fails-${id}`)){console.error('fixture host: refused');process.exit(1)}
const b=a[2],off=a.includes('--no-launch'),lock=path.join(data,'bus','writer.lock'),at=new Date().toISOString();
fs.writeFileSync(lock,String(process.pid),{flag:'wx'});
@@ -2122,14 +2145,35 @@ function rHost(){
}
// The Console: serve on the given loopback port; /healthz answers 200 unless healthz-<copy> is set.
function rConsole(){
// M3 (#1873): under --sso it runs its copy's real createSso on the scratch console.sso, with the
// scratch slot placeholders, as the real Console does, and /healthz carries sso.health(). Faults:
// console-hold waits for $HOME/release before it starts; console-no-sso reports every provider
// ready without asking any; console-asks-token asks provider a's token endpoint once.
async function rConsole(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),http=process.getBuiltinModule('node:http');
const id=path.basename(path.resolve(import.meta.dirname,'..','..','..')),home=process.env.HOME,a=process.argv.slice(2);
const fault=(n)=>fs.existsSync(path.join(home,'faults',n));
const c=JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8'));
fs.appendFileSync(path.join(home,'console-args.log'),`${a.join(' ')}\n`);
fs.appendFileSync(path.join(home,'console-context.jsonl'),JSON.stringify({argv:a,dataRoot:JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8')).dataRoot})+'\n');
fs.appendFileSync(path.join(home,'console-context.jsonl'),JSON.stringify({argv:a,dataRoot:c.dataRoot,ca:process.env.NODE_EXTRA_CA_CERTS??null})+'\n');
if(a[0]!=='serve'){console.error('fixture Console: refused');process.exit(1)}
const s=http.createServer((q,r)=>{r.writeHead(q.url==='/healthz'&&!fault(`healthz-${id}`)?200:500);r.end()});
while(fault('console-hold')&&!fs.existsSync(path.join(home,'release')))await new Promise((r)=>setTimeout(r,50));
let health=null;
if(a.includes('--sso')){
const ps=c.console.sso.providers;
if(fault('console-no-sso'))health=()=>Object.fromEntries(ps.map((p)=>[p.id,'ready']));
else{
const {createSso}=await import(path.join(import.meta.dirname,'sso.mjs'));
const secrets=new Map(ps.map((p)=>[p.id,fs.readFileSync(path.join(path.dirname(process.env.MOSAIC_CONFIG),'secrets','mosaic-stack',p.clientSecret.slot),'utf8')]));
let sso;try{sso=await createSso(c.console.sso,{secrets,log:()=>{}})}catch(err){console.error(`fixture Console: ${err.message}`);process.exit(2)}
health=()=>sso.health();
}
if(fault('console-asks-token'))await fetch(`${ps[0].issuer}token`).catch(()=>{});
}
const s=http.createServer((q,r)=>{
const ok=q.url==='/healthz'&&!fault(`healthz-${id}`);r.writeHead(ok?200:500,{'content-type':'application/json'});
r.end(ok&&health?JSON.stringify({ok:true,sso:health()}):'');
});
s.listen(Number(a[a.indexOf('--port')+1]),'127.0.0.1',()=>console.log(`Mosaic Console: http://127.0.0.1:${s.address().port}/\nBoard: fixture`));
process.on('SIGTERM',()=>process.exit(0));
}
@@ -2173,15 +2217,23 @@ for(const [id,v,froms] of [[B,'3g',['3f']],[C,'3h',['3f','3g']],[A,'3g',['3f']],
execFileSync('chmod',['-R','a-w',d]);
}
// M3 (#1873): SCOPIES holds B as a stamped copy: RCOPIES's B with a read-only .stack-build.json
// naming its commit, the Console's sso-config.mjs the config loader needs for a console key, and
// the deployment.mjs whose writeDeployment the rehearsal runs.
// naming its commit, the Console's sso-config.mjs the config loader needs for a console key, its
// sso.mjs the stand-in Console runs, and the deployment.mjs whose writeDeployment the rehearsal runs.
const SCOPIES=join(base,'scopies');mkdirSync(SCOPIES);
{
const d=join(SCOPIES,B);cpSync(join(RCOPIES,B),d,{recursive:true});execFileSync('chmod',['-R','u+w',d]);
for(const p of ['packages/webui/src/sso-config.mjs','packages/cli/src/deployment.mjs'])cpSync(join(repo,p),join(d,p));
for(const p of ['packages/webui/src/sso-config.mjs','packages/webui/src/sso.mjs','packages/cli/src/deployment.mjs'])cpSync(join(repo,p),join(d,p));
// sso.mjs imports oauth4webapi.
symlinkSync(join(ROOT,'node_modules'),join(d,'node_modules'));
writeFileSync(join(d,'.stack-build.json'),JSON.stringify({commit:B}),{mode:0o444});
execFileSync('chmod',['-R','a-w',d]);
}
// UCOPIES holds B unstamped but able to load a console key: SCOPIES's B without .stack-build.json.
const UCOPIES=join(base,'ucopies');mkdirSync(UCOPIES);
{
const d=join(UCOPIES,B);cpSync(join(SCOPIES,B),d,{recursive:true,verbatimSymlinks:true});execFileSync('chmod',['-R','u+w',d]);
rmSync(join(d,'.stack-build.json'));execFileSync('chmod',['-R','a-w',d]);
}
const TESTER='7499';
// Vikunja #782: this leases a port, closes it and hands the number to `stack-release.sh rehearse`, which
@@ -2194,12 +2246,13 @@ const rfault=(e,name)=>{mkdirSync(rh(e,'faults'),{recursive:true});writeFileSync
// A root for one rehearsal: a stopped live store at `store`, a pointer by hand on `copy` with the
// Console on e.port and the tester listener on TESTER, and a backup as backup_store writes one.
// A rehearsal's leftovers, found by their scratch MOSAIC_CONFIG, are killed after the test.
// A rehearsal's leftovers, found by their scratch MOSAIC_CONFIG or a rehearsal path in their command
// line (the rehearsal provider), are killed after the test.
async function rmk(t,{copy=B,store='3g'}={}){
const e=await mk(t);
t.after(()=>{
for(const pid of readdirSync('/proc').filter((n)=>/^\d+$/.test(n)))
try{if(readFileSync(`/proc/${pid}/environ`,'utf8').includes(`MOSAIC_CONFIG=${e.data}/rehearsals/`))process.kill(Number(pid),'SIGKILL')}catch{}
try{if(readFileSync(`/proc/${pid}/environ`,'utf8').includes(`MOSAIC_CONFIG=${e.data}/rehearsals/`)||readFileSync(`/proc/${pid}/cmdline`,'utf8').includes(`${e.data}/rehearsals/`))process.kill(Number(pid),'SIGKILL')}catch{}
});
const db=new DatabaseSync(e.store);
db.exec('PRAGMA journal_mode=WAL');
@@ -2479,9 +2532,10 @@ test('R15 a rehearsal stopped by TERM, INT or HUP fails signal and still stops t
// --- M3 (#1873): rehearsing a stamped copy (the brief's item 1) ---
// A stamped copy serves only with --sso, console.sso, accounts.json and a .deployed naming its
// instance. The rehearsal gives its scratch Console all four: the live console.sso with origin and
// issuers on rehearsal.invalid, a placeholder per slot, a copy of the live accounts.json, and a
// .deployed from the copy's writeDeployment. The live slot has a read watch: it is never opened.
// instance. The rehearsal gives its scratch Console all four: the live console.sso with its origin
// on rehearsal.invalid and each issuer at the rehearsal's own provider, a placeholder per slot, a
// copy of the live accounts.json, and a .deployed from the copy's writeDeployment. The live slot
// has a read watch: it is never opened.
const ACCOUNTS={accountsVersion:1,revision:1,users:[{id:'u-000000000001',subjects:[{provider:'a',sub:'subject-1'}],name:'Jason',instanceRole:'operator',state:'active',authGen:1,created:'2026-11-10T00:00:00Z'}],memberships:[],invites:[]};
async function smk(t,{sso=true,accounts=true}={}){
const e=await rmk(t);e.copies=SCOPIES;
@@ -2490,7 +2544,10 @@ async function smk(t,{sso=true,accounts=true}={}){
if(accounts)writeFileSync(e.accounts,JSON.stringify(ACCOUNTS),{mode:0o600});
return e;
}
const SPRE=[...PRE.slice(0,4),'PASS deployment',...PRE.slice(4)];
const SPRE=[...PRE.slice(0,3),'PASS provider-start',PRE[3],'PASS deployment',...PRE.slice(4)];
const SPOST=['PASS console-stopped','PASS host-stopped','PASS provider-stopped',...POST.slice(2)];
// The rehearsal provider's origin and pid, from its provider-start line.
const provOf=(out)=>{const m=/^PASS provider-start: pid (\d+) at (https:\/\/127\.0\.0\.1:\d+): /m.exec(out);return m?{pid:Number(m[1]),origin:m[2]}:null};
// A real stamped copy of this checkout, as deployment-boundary.test.mjs builds one.
function stampedCopy(t){
const sha40='a'.repeat(40),root=mkdtempSync(join(tmpdir(),'rs-'));t.after(()=>rmSync(root,{recursive:true,force:true}));
@@ -2503,8 +2560,8 @@ function stampedCopy(t){
}
// Starts the copy's real Console on port 0 against the rehearsal's scratch config. Resolves with
// {code,out,err} on exit, or with the bound port once it prints its address line.
async function realServe(t,copy,e,dir){
const ch=spawn(process.execPath,[join(copy,'packages/webui/src/cli.mjs'),'serve','--port','0','--sso','--board','http://127.0.0.1:9'],{env:{HOME:e.env.HOME,PATH:e.env.PATH,TMPDIR:e.env.TMPDIR,MOSAIC_CONFIG:join(dir,'config','config.json')},stdio:['ignore','pipe','pipe']});
async function realServe(t,copy,e,dir,more={}){
const ch=spawn(process.execPath,[join(copy,'packages/webui/src/cli.mjs'),'serve','--port','0','--sso','--board','http://127.0.0.1:9'],{env:{HOME:e.env.HOME,PATH:e.env.PATH,TMPDIR:e.env.TMPDIR,MOSAIC_CONFIG:join(dir,'config','config.json'),...more},stdio:['ignore','pipe','pipe']});
t.after(async()=>{if(ch.exitCode===null&&ch.signalCode===null){const done=once(ch,'close');ch.kill('SIGKILL');await done}});
let out='',err='';ch.stderr.on('data',(d)=>{err+=d});
return new Promise((res)=>{
@@ -2514,21 +2571,32 @@ async function realServe(t,copy,e,dir){
});
}
test('M3 R16 a stamped copy rehearses with --sso against rehearsal.invalid, a scratch accounts copy and its own .deployed; the live files stay as they were',async(t)=>{
test('M3 R16 a stamped copy rehearses with --sso against its own provider, a scratch accounts copy and its own .deployed; the live files stay as they were',async(t)=>{
const st=(f)=>{const x=statSync(f);return [x.ino,x.size,x.mtimeMs,x.mode]};
const e=await smk(t),before=untouched(e),accounts=sha(e.accounts),slot=st(slotPath(e)),port=await freePort();
const r=await rehearse(e,B,'--backup',e.act,'--port',port,'--timeout','10');
assert.equal(r.code,0,r.out+r.err);
assert.deepEqual(r.names,[...SPRE,'PASS migration','PASS gate','PASS no-session',...POST]);
const R=r.dir;
assert.deepEqual(r.names,[...SPRE,'PASS migration','PASS gate','PASS no-session','PASS provider-discovery',...SPOST]);
const R=r.dir,prov=provOf(r.out);assert.ok(prov,r.out);
assert.ok(r.out.includes(`PASS provider-start: pid ${prov.pid} at ${prov.origin}: discovery and a JWKS from keys made for this run, no code and no token; its certificate ${join(R,'provider','cert.pem')} is the trust the scratch Console alone gets\n`),r.out);
assert.ok(r.out.includes(`PASS deployment: ${B} is stamped: ${join(R,'data','.deployed')} from its writeDeployment, the live accounts.json copied to ${join(R,'config')}, the Console starts with --sso\n`),r.out);
assert.ok(r.out.includes('PASS sso-isolated: 1 provider(s), each at rehearsal.invalid with a placeholder secret\n'),r.out);
// The Console started with --sso, and only on the scratch config.
assert.ok(r.out.includes(`PASS sso-isolated: 1 provider(s), each at the rehearsal provider ${prov.origin} with a placeholder secret\n`),r.out);
assert.ok(r.out.includes(`PASS gate: host-up, bus-status, console-up, healthz, sso-ready and copy-match on ${B}\n`),r.out);
assert.ok(r.out.includes(`PASS provider-discovery: the Console fetched each provider's discovery and JWKS from ${prov.origin} and asked for no code or token\n`),r.out);
// The Console started with --sso, only on the scratch config, and alone trusts the provider's
// certificate; the host gets no added trust.
assert.deepEqual(lines(rh(e,'console-args.log')),[`serve --port ${port} --business acme --sso`]);
assert.deepEqual(lines(rh(e,'console-context.jsonl')).map((l)=>JSON.parse(l).ca),[join(R,'provider','cert.pem')]);
assert.deepEqual(lines(rh(e,'host-ca.log')),['none']);
// The provider served discovery and the JWKS, got no other request, kept no TLS key and is gone.
assert.deepEqual(lines(join(R,'provider','requests.log')).sort(),['GET /a/.well-known/openid-configuration 200','GET /a/jwks 200']);
assert.deepEqual(readdirSync(join(R,'provider')).sort(),['cert.pem','origin','provider.log','requests.log']);
assert.equal(statSync(join(R,'provider')).mode&0o777,0o700);
assert.equal(existsSync(`/proc/${prov.pid}`),false,`provider pid ${prov.pid} is still there`);
// The scratch config: the live one with dataRoot, origin and issuer moved; provider id, client id,
// algorithms, slot name and subjects as they were. No subject is added.
const live=JSON.parse(readFileSync(e.cfg,'utf8')),sc=JSON.parse(readFileSync(join(R,'config','config.json'),'utf8'));
const want=structuredClone(live);want.dataRoot=join(R,'data');want.console.sso.publicOrigin='https://rehearsal.invalid';want.console.sso.providers[0].issuer='https://rehearsal.invalid/a';
const want=structuredClone(live);want.dataRoot=join(R,'data');want.console.sso.publicOrigin='https://rehearsal.invalid';want.console.sso.providers[0].issuer=`${prov.origin}/a/`;
assert.deepEqual(sc,want);assert.deepEqual(sc.console.sso.providers[0].subjects,SSO_KEY.sso.providers[0].subjects);
// The slot is a scratch placeholder, 0600 in a 0700 directory.
const ph=join(R,'config','secrets','mosaic-stack',SLOT);
@@ -2545,16 +2613,30 @@ test('M3 R16 a stamped copy rehearses with --sso against rehearsal.invalid, a sc
await noLeak(e);
});
test('M3 R17 the real stamped Console accepts the rehearsal\'s scratch prerequisites and starts degraded; without .deployed or accounts.json it refuses',async(t)=>{
const e=await smk(t);
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','10');
assert.equal(r.code,0,r.out+r.err);
test('M3 R17 the real stamped Console finds the rehearsal provider ready with the scratch prerequisites, starts degraded once it is gone, and refuses without .deployed or accounts.json',async(t)=>{
const e=await smk(t);rfault(e,'console-hold');
const copy=stampedCopy(t);
// Every startup check passes: stamp, --sso, .deployed, accounts, console.sso and its slot. The
// provider at rehearsal.invalid is unreachable, so the Console serves degraded and /healthz answers.
// While the stand-in Console holds, the rehearsal provider is up: the real Console of a stamped
// copy passes every startup check (stamp, --sso, .deployed, accounts, console.sso and its slot)
// and, trusting only the provider's certificate, finds provider a ready.
let during=null;
e.during=async()=>{
try{
for(let i=0;i<400&&!existsSync(rh(e,'console-args.log'));i++)await new Promise((r)=>setTimeout(r,50));
const dir=join(e.data,'rehearsals',rDirs(e)[0]);
const up=await realServe(t,copy,e,dir,{NODE_EXTRA_CA_CERTS:join(dir,'provider','cert.pem')});
during={up:up.port?await (await fetch(`http://127.0.0.1:${up.port}/healthz`)).json():null,why:`${up.code} ${up.out} ${up.err}`};
if(up.ch){const done=once(up.ch,'close');up.ch.kill('SIGTERM');await done}
}finally{writeFileSync(rh(e,'release'),'')}
};
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','30');
assert.equal(r.code,0,r.out+r.err);
assert.deepEqual(during?.up,{ok:true,sso:{a:'ready'}},during?.why);
// After the rehearsal its provider is gone: the same Console serves degraded and /healthz says so.
const up=await realServe(t,copy,e,r.dir);
assert.ok(up.port,`${up.code} ${up.out} ${up.err}`);
assert.equal((await fetch(`http://127.0.0.1:${up.port}/healthz`)).status,200);
const h=await fetch(`http://127.0.0.1:${up.port}/healthz`);
assert.equal(h.status,200);assert.deepEqual(await h.json(),{ok:true,sso:{a:'unreachable'}});
const done=once(up.ch,'close');up.ch.kill('SIGTERM');await done;
// The same start without each prerequisite refuses before it listens.
const dep=join(r.dir,'data','.deployed'),acc=join(r.dir,'config','accounts.json'),keep=readFileSync(dep);
@@ -2589,19 +2671,86 @@ test('M3 R18 a stamped copy refuses sso-not-prepared without a live console.sso
});
test('M3 R19 sso-isolated fails before the host starts when the scratch origin, an issuer or a slot is not the rehearsal\'s',async(t)=>{
// stamped: the issuer must be the rehearsal provider's; unstamped (no provider): on rehearsal.invalid.
for(const [stamped,from,to,why] of [
[true,'sso.publicOrigin="https://rehearsal.invalid";','','console.sso.publicOrigin is https://console.example.test, not https://rehearsal.invalid'],
[true,'p.issuer=`${op||','p.issuer=p.issuer||`${op||','console.sso provider a issuer is https://op.example.test/, not ORIGIN/a/ at the rehearsal provider'],
[false,'p.issuer=`${op||"https://rehearsal.invalid"}','p.issuer=`${op||"https://op.example.test"}','console.sso provider a issuer is https://op.example.test/a, not on rehearsal.invalid'],
[true,'"rehearsal-placeholder-secret\\n",{mode:0o600,flag:"wx"}','"other\\n",{mode:0o600,flag:"wx"}','console.sso provider a has no scratch placeholder secret'],
]){
let e;if(stamped)e=await smk(t);else{e=await rmk(t);e.copies=UCOPIES;await ssoConfig(e)}
ownScript(e,from,to);
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
assert.equal(r.code,1,why+r.out+r.err);
const prov=provOf(r.out);assert.equal(!!prov,stamped,r.out);
assert.ok(r.out.includes(`FAIL sso-isolated: ${why.replace('ORIGIN',prov?.origin)}\n`),r.out);
assert.deepEqual(r.names.slice(stamped?-4:-3),['FAIL sso-isolated',...(stamped?['PASS provider-stopped']:[]),'PASS backup-unchanged','PASS live-config-unchanged']);
assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false);
if(prov)assert.equal(existsSync(`/proc/${prov.pid}`),false);
await noLeak(e);
}
});
test('M3 R20 the rehearsal provider\'s failure controls: no start, no trust, no discovery, a token request, a signal and a SIGKILL each fail and stop it',async(t)=>{
const ends=['PASS backup-unchanged','PASS live-config-unchanged'];
// The provider cannot start, leaves its TLS key, offers a non-loopback origin or never writes one:
// provider-start fails before the scratch build, and no host or Console starts.
for(const [from,to,why] of [
['sso.publicOrigin="https://rehearsal.invalid";','','console.sso.publicOrigin is https://console.example.test, not https://rehearsal.invalid'],
['p.issuer=`https://rehearsal.invalid/${encodeURIComponent(String(p.id))}`','0','console.sso provider a issuer is https://op.example.test/, not on rehearsal.invalid'],
['"rehearsal-placeholder-secret\\n",{mode:0o600,flag:"wx"}','"other\\n",{mode:0o600,flag:"wx"}','console.sso provider a has no scratch placeholder secret'],
['spawnSync("openssl"','spawnSync("openssl-missing-for-r20"','the rehearsal provider exited: rehearsal provider: cannot make the certificate: openssl exit ENOENT'],
['finally{fs.rmSync(tls,{force:true})}','finally{}','the rehearsal provider left its TLS key at RDIR/provider/tls-key.pem'],
['origin=`https://127.0.0.1:${','origin=`https://localhost:${','the rehearsal provider\'s origin https://localhost:PORT is not https on 127.0.0.1'],
['fs.renameSync(path.join(dir,"origin.tmp"),path.join(dir,"origin"));','','the rehearsal provider wrote no origin within 3s'],
]){
const e=await smk(t);ownScript(e,from,to);
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
assert.equal(r.code,1,why+r.out+r.err);
assert.ok(r.out.includes(`FAIL sso-isolated: ${why}\n`),r.out);
assert.deepEqual(r.names.slice(-3),['FAIL sso-isolated','PASS backup-unchanged','PASS live-config-unchanged']);
assert.equal(r.code,1,why+r.out+r.err);assert.ok(r.dir,r.out+r.err);
const line=r.out.split('\n').find((l)=>l.startsWith('FAIL provider-start: '))??'';
const re=new RegExp(`^FAIL provider-start: ${why.replace(/[.*+?^${}()|[\]\\]/g,'\\$&').replace('RDIR',r.dir.replace(/[.*+?^${}()|[\]\\]/g,'\\$&')).replace('PORT','\\d+')}$`);
assert.match(line,re,r.out);
assert.deepEqual(r.names,[...PRE.slice(0,3),'FAIL provider-start',...(why.includes('exited')?[]:['PASS provider-stopped']),...ends],r.out);
assert.equal(existsSync(join(r.dir,'config')),false);
assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false);
await noLeak(e);
}
// The Console cannot reach the provider (its discovery drops the connection, or the Console has
// no trust for its certificate): it starts degraded and the gate fails sso-ready.
for(const [from,to] of [
[' send(200,{issuer,authorization_endpoint',' return q.socket.destroy();send(200,{issuer,authorization_endpoint'],
['r_ca=(NODE_EXTRA_CA_CERTS="$R/provider/cert.pem")','r_ca=()'],
]){
const e=await smk(t);ownScript(e,from,to);
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','5');
assert.equal(r.code,1,from+r.out+r.err);
assert.ok(r.out.includes('FAIL sso-ready: provider a is unreachable, not ready\n'),r.out);
assert.deepEqual(r.names,[...SPRE,'PASS migration','FAIL sso-ready',...SPOST],r.out);
assert.equal(existsSync(`/proc/${provOf(r.out).pid}`),false);
}
// A Console that reports ready without asking the provider, or asks it for a token: the gate
// passes and provider-discovery fails.
for(const [f,why] of [
['console-no-sso','no GET /a/.well-known/openid-configuration 200; no GET /a/jwks 200'],
['console-asks-token','a request for GET /a/token 403'],
]){
const e=await smk(t);rfault(e,f);
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','10');
assert.equal(r.code,1,f+r.out+r.err);
assert.ok(r.out.includes(`FAIL provider-discovery: ${why}; see ${join(r.dir,'provider','requests.log')}\n`),r.out);
assert.deepEqual(r.names,[...SPRE,'PASS migration','PASS gate','PASS no-session','FAIL provider-discovery',...SPOST],r.out);
}
// A signal while the gate waits stops the Console, host and provider.
const e=await smk(t);rfault(e,`healthz-${B}`);
e.during=async(ch)=>{for(let i=0;i<200&&!existsSync(rh(e,'console-args.log'));i++)await new Promise((r)=>setTimeout(r,50));ch.kill('SIGTERM')};
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','30');
assert.equal(r.code,1,r.out+r.err);
assert.deepEqual(r.names,[...SPRE,'PASS migration','FAIL signal',...SPOST],r.out);
assert.equal(existsSync(`/proc/${provOf(r.out).pid}`),false);
// SIGKILL leaves no trap to run: the provider sees its parent gone and exits by itself.
const k=await smk(t);rfault(k,`healthz-${B}`);
k.during=async(ch)=>{for(let i=0;i<200&&!existsSync(rh(k,'console-args.log'));i++)await new Promise((r)=>setTimeout(r,50));ch.kill('SIGKILL')};
const rk=await rehearse(k,B,'--backup',k.act,'--port',await freePort(),'--timeout','30');
assert.equal(rk.code,null,rk.out+rk.err);
const pk=provOf(rk.out);assert.ok(pk,rk.out);
for(let i=0;i<60&&existsSync(`/proc/${pk.pid}`);i++)await new Promise((r)=>setTimeout(r,100));
assert.equal(existsSync(`/proc/${pk.pid}`),false,`provider pid ${pk.pid} outlived the killed rehearsal`);
});
// --- Row 233 (#1735): --backup last, the newest backup the log records (sage-1650, sage-1653) ---
+112 -5
View File
@@ -47,12 +47,19 @@ function fakeSudo(K){
// case2-unit.sh: logs its argv, then installs the copy's units as the fixture's user units (its
// WorkingDirectory under the stack's copies) and starts the optional proxy, bus, then Console,
// as case2-unit.sh does. The business is --business's value, or --org's when that is the spelling. faults/render-fails-<copy> exits 1 first; faults/render-probe-<copy> (one-shot) first runs
// as case2-unit.sh does. The business is --business's value, or --org's when that is the spelling. A first
// --check or --check-new (#1873) logs its argv and exits 1 under faults/check-fails-<copy>, else 0,
// writing nothing. faults/render-fails-<copy> exits 1 first; faults/render-probe-<copy> (one-shot) first runs
// probe.sh, a second driver, and records its exit and stderr.
function fakeRender(K){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
const a=process.argv.slice(2),sd=process.env.SD_STATE,f=(n)=>path.join(sd,'faults',n);
fs.appendFileSync(path.join(sd,'render.log'),JSON.stringify(a)+'\n');
if(a[0]==='--check'||a[0]==='--check-new'){
const c=a[a.indexOf('--copy')+1];
if(fs.existsSync(f(`check-fails-${c}`))){console.error(`fixture render: ${a[0]} ${c} fails`);process.exit(1)}
process.exit(0);
}
const cut=a.indexOf('--'),pre=a.slice(0,cut),serve=a.slice(cut+1),get=(k)=>pre[pre.indexOf(k)+1];
const copy=get('--copy'),copies=path.join(get('--home'),'.local','share','mosaic-stack','copies');
if(fs.existsSync(f(`render-probe-${copy}`))){
@@ -147,7 +154,7 @@ const arm=(e,key,text)=>writeFileSync(join(e.sd,'arm',key),text);
// The step log: every line `<time> <step> rc=N ...` or the result line, root:root 0640 (here the
// test user's), and nothing the stack's user printed. Returns the lines without the time.
const STEP=/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ (start|fetch|ff|copy|hostcheck|units|plan|rehearse|rehearse-store|rehearsal-skipped|stop-console|stop-bus|stop-proxy|wait-down|stage|render|confirm|args|revert|render-old|confirm-rollback|gate|start-proxy|start-bus|start-console|stop-units) rc=\d+ |^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ result exit=[0-35] [a-z-]+ /;
const STEP=/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ (start|fetch|ff|copy|hostcheck|units|plan|rehearse|rehearse-store|rehearsal-skipped|render-old-check|render-check|stop-console|stop-bus|stop-proxy|wait-down|units-aside|units-back|daemon-reload|next-args|stage|render|confirm|args|revert|render-old|confirm-rollback|gate|start-proxy|start-bus|start-console|stop-units) rc=\d+ |^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ result exit=[0-35] [a-z-]+ /;
function slog(e){
const l=lines(e.slog);
for(const x of l){assert.match(x,STEP);assert.ok(!/stack-release:|fixture rehearse|PASS |FAIL |REHEARSAL/.test(x),x)}
@@ -165,7 +172,7 @@ const live=(e)=>[units(e)[e.bus].state,units(e)[e.con].state,units(e)[e.bus].wd.
// Nothing the run did reached the args file: the same bytes and mode, and no temporary file left.
const argsKept=(e)=>{assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args'])};
const argsCopy=(e)=>{const l=readFileSync(e.args,'utf8').split('\n');return l[l.indexOf('--copy')+1]};
const PRE=['start','fetch','ff','copy','hostcheck','units','plan','rehearse','rehearse-store'];
const PRE=['start','fetch','ff','copy','hostcheck','units','plan','render-old-check','render-check','rehearse','rehearse-store'];
const A2=['stop-console','stop-bus','wait-down','wait-down'];
const R=[...A2,'revert','render-old','confirm-rollback'];
@@ -394,7 +401,7 @@ test('row 233: sequence 7: two upgrades and a rollback --to the previous copy wi
const rb=await drive(e,C,'rollback','--instance','dev','--to',C1,'--restore',a2,...TO);assert.equal(rb.code,0,rb.all);
assert.equal(rb.out.split('\n').at(-2),`stack-upgrade: rollback ok: ${C1} is live for acme`);
// A rollback fetches, copies and rehearses nothing.
assert.deepEqual(names(e).slice(n),['start','hostcheck','units','plan',...A2,'stage','render','confirm','args','result']);
assert.deepEqual(names(e).slice(n),['start','hostcheck','units','plan','render-old-check','render-check',...A2,'stage','render','confirm','args','result']);
const ra=lastAct(e),l3=final(e,ra);assert.deepEqual([l3.op,l3.result,l3.from,l3.to,l3.backup],['rollback','ok',C,C1,a2]);
assert.deepEqual([pointer(e).copy,pointer(e).schema],[C1,'3g']);assert.equal(storeVersion(e.store,true),'3g');
assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
@@ -632,7 +639,9 @@ test('rename r2: an upgrade from an args file that spells both selectors --org k
const after=e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`);assert.notEqual(after,e.argsText);
assert.equal(readFileSync(e.args,'utf8'),after);
assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args']);
assert.deepEqual(lines(join(e.sd,'render.log')).map((x)=>JSON.parse(x)),[after.trimEnd().split('\n')]);
// #1873: before anything stops, --check renders the installed copy, then the target, from the same lines.
const al=after.trimEnd().split('\n');
assert.deepEqual(lines(join(e.sd,'render.log')).map((x)=>JSON.parse(x)),[['--check',...e.argsText.trimEnd().split('\n')],['--check',...al],al]);
// The stack's user's steps get the business as a word, never a spelling.
for(const k of ['activate-plan','rehearse','activate','confirm'])assert.ok(JSON.parse(lines(join(e.sd,'sudo.log')).find((x)=>JSON.parse(x).key===k)).cmd.includes('acme'),k);
assert.ok(!lines(join(e.sd,'sudo.log')).some((x)=>JSON.parse(x).cmd.includes('--org')));
@@ -842,3 +851,101 @@ test('row 233: usage exits 4 before any log or lock',async(t)=>{
}
assert.ok(!existsSync(dirname(e.slog)));assert.ok(!existsSync(e.run));
});
// --- #1873 M3: render checks before anything stops, and --next-args ---
// The installed units' bytes, for the byte-for-byte checks across a moved-aside render.
const unitBytes=(e)=>Object.fromEntries([e.tmpl,e.con].map((u)=>[u,readFileSync(join(e.units,u),'utf8')]));
const renders=(e)=>lines(join(e.sd,'render.log')).map((l)=>JSON.parse(l));
const copyOf=(a)=>a[a.indexOf('--copy')+1];
// The next args: the args file's lines with the serve flags in another order, root's 0640.
const nextText=(e)=>e.argsText.replace(`--\n--port\n${e.port}\n--business\nacme\n`,`--\n--business\nacme\n--port\n${e.port}\n`);
const writeNext=(e,text=nextText(e),mode=0o640)=>{const f=join(e.etc,'dev.args.next');writeFileSync(f,text);chmodSync(f,mode);return f};
const replacedDirs=(e)=>readdirSync(e.etc).filter((n)=>/^dev\.replaced\.[A-Za-z0-9]{6}$/.test(n)).sort();
test('#1873: a target whose render check fails refuses render-refused, naming --next-args, before anything stops',async(t)=>{
const e=await useed(t);
fault(e,`check-fails-${C1}`);
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
assert.deepEqual(names(e),['start','fetch','ff','copy','hostcheck','units','plan','render-old-check','render-check','result']);
assert.match(result(e),new RegExp(`^result exit=3 render-refused case2-unit.sh --check refuses ${C1}'s units from .*dev\\.args; .*takes --next-args .*\\(nothing was stopped\\)$`));
assert.deepEqual(renders(e).map((a)=>[a[0],copyOf(a)]),[['--check',B],['--check',C1]]);
assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
});
test('#1873: an installed copy whose render check fails refuses at render-old-check, before the target\'s check',async(t)=>{
const e=await useed(t);
fault(e,`check-fails-${B}`);
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
assert.deepEqual(names(e),['start','fetch','ff','copy','hostcheck','units','plan','render-old-check','result']);
assert.match(result(e),new RegExp(`^result exit=3 render-refused case2-unit.sh --check refuses ${B}'s units from .*dev\\.args, so a rollback could not render them`));
assert.deepEqual(renders(e).map((a)=>[a[0],copyOf(a)]),[['--check',B]]);
assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
});
test('#1873: --next-args moves the old units aside, renders the new flags as a first install, keeps the old units and args, and rolls back the same way',async(t)=>{
const e=await useed(t);
const before=unitBytes(e),next=nextText(e);writeNext(e);
const r=await upgrade(e,C1,'--next-args');assert.equal(r.code,0,r.all);
assert.deepEqual(names(e),[...PRE,...A2,'stage','units-aside','render','confirm','next-args','args','result']);
const [d]=replacedDirs(e);assert.ok(d,readdirSync(e.etc).join(' '));
assert.equal(result(e),`result exit=0 ok upgrade ok: ${C1} is live for acme; the replaced units and args file are in ${join(e.etc,d)}`);
assert.deepEqual(readdirSync(e.etc).sort(),['dev.args',d]);
assert.equal(readFileSync(e.args,'utf8'),next.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`));assert.equal(statSync(e.args).mode&0o777,0o640);
const rd=join(e.etc,d);assert.equal(statSync(rd).mode&0o777,0o700);
assert.deepEqual(readdirSync(rd).sort(),['dev.args',e.tmpl,e.con].sort());
for(const [u,text] of Object.entries(before))assert.equal(readFileSync(join(rd,u),'utf8'),text,u);
assert.equal(readFileSync(join(rd,'dev.args'),'utf8'),e.argsText);assert.equal(statSync(join(rd,'dev.args')).mode&0o777,0o640);
const serve=(a)=>a.slice(a.indexOf('--')+1);
const rl=renders(e);assert.deepEqual(rl.map((a)=>[a[0],copyOf(a)]),[['--check',B],['--check-new',C1],['--user',C1]]);
assert.deepEqual(serve(rl[0]),['--port',String(e.port),'--business','acme']);
assert.deepEqual(serve(rl[1]),['--business','acme','--port',String(e.port)]);assert.deepEqual(serve(rl[2]),serve(rl[1]));
assert.deepEqual(live(e),['active','active',C1,C1]);
// The run sheet's rollback: the replaced args file as the next args, then rollback --next-args.
copyFileSync(join(rd,'dev.args'),join(e.etc,'dev.args.next'));chmodSync(join(e.etc,'dev.args.next'),0o640);
const after=unitBytes(e),m=slog(e).length,n=renders(e).length;
const rb=await drive(e,C1,'rollback','--instance','dev','--to',B,'--next-args',...TO);assert.equal(rb.code,0,rb.all);
assert.deepEqual(names(e).slice(m),['start','hostcheck','units','plan','render-old-check','render-check',...A2,'stage','units-aside','render','confirm','next-args','args','result']);
assert.deepEqual(renders(e).slice(n).map((a)=>[a[0],copyOf(a)]),[['--check',C1],['--check-new',B],['--user',B]]);
assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.deepEqual(live(e),['active','active',B,B]);
const d2=replacedDirs(e).find((x)=>x!==d);assert.ok(d2);assert.deepEqual(readdirSync(e.etc).sort(),['dev.args',d,d2].sort());
for(const [u,text] of Object.entries(after))assert.equal(readFileSync(join(e.etc,d2,u),'utf8'),text,u);
assert.equal(readFileSync(join(e.etc,d2,'dev.args'),'utf8'),next.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`));
});
test('#1873: a failed render under --next-args moves the old units back unchanged and restarts them, with no render of the old copy: exit 1',async(t)=>{
const e=await useed(t);
const before=unitBytes(e),next=nextText(e);writeNext(e);
fault(e,`render-fails-${C1}`);
const r=await upgrade(e,C1,'--next-args');assert.equal(r.code,1,r.all);
assert.deepEqual(names(e),[...PRE,...A2,'stage','units-aside','render',...A2,'revert','units-back','daemon-reload','start-bus','start-console','confirm-rollback','result']);
assert.match(result(e),new RegExp(`^result exit=1 failed A4 failed; rolled back to ${B}`));
assert.deepEqual(renders(e).map((a)=>[a[0],copyOf(a)]),[['--check',B],['--check-new',C1],['--user',C1]]);
for(const [u,text] of Object.entries(before))assert.equal(readFileSync(join(e.units,u),'utf8'),text,u);
assert.deepEqual(readdirSync(e.etc).sort(),['dev.args','dev.args.next']);
assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.equal(readFileSync(join(e.etc,'dev.args.next'),'utf8'),next);
assert.equal(final(e,lastAct(e)).rollback,'ok');assert.deepEqual(live(e),['active','active',B,B]);
});
test('#1873: the next args must be root\'s, equal before -- but for --copy, differ after it, and render as a first install',async(t)=>{
const e=await useed(t);
const tryNext=async(text,mode,code,why,at='plan')=>{
if(text===null)rmSync(join(e.etc,'dev.args.next'),{force:true});else writeNext(e,text,mode);
const n=existsSync(e.slog)?slog(e).length:0;
const r=await upgrade(e,C1,'--next-args');assert.equal(r.code,3,`${why}: ${r.all}`);
assert.match(slog(e).at(-1),new RegExp(`^result exit=3 ${code} `),why);
const ran=names(e).slice(n);assert.ok(!ran.some((x)=>/^(stop|stage|units-aside|render$|confirm)/.test(x)),`${why}: ${ran}`);
if(at)assert.equal(ran.at(-2),at,`${why}: ${ran}`);
};
await tryNext(null,0,'args-unreadable','no next file','start');
await tryNext(nextText(e),0o660,'args-unreadable','group write','start');
await tryNext(nextText(e).replace('--home\n','--pm\n--home\n'),0o640,'next-args-differ','a flag before --','start');
await tryNext(nextText(e).replace('--user\nfixture\n','--user\nother\n'),0o640,'next-args-differ','another user','start');
await tryNext(e.argsText,0o640,'next-args-same','the same lines','start');
await tryNext(e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C}\n`),0o640,'next-args-same','only --copy differs','start');
fault(e,`check-fails-${C1}`);
await tryNext(nextText(e),0o640,'render-refused','the first-install check','render-check');
assert.match(result(e),new RegExp(`^result exit=3 render-refused case2-unit.sh --check-new refuses ${C1}'s units from .*dev\\.args\\.next`));
assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);
assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.deepEqual(replacedDirs(e),[]);
});
+21 -7
View File
@@ -234,13 +234,16 @@ test('a copy that meets an empty target directory replaces it by rename, not by
// prerequisites (M3): console.sso and its slot, .deployed and accounts.json.
async function serve(t, cli, args, env) {
const child = spawn(process.execPath, [cli, 'serve', '--port', '0', '--board', 'http://127.0.0.1:9', ...args], { env, stdio: ['ignore', 'pipe', 'pipe'] });
t.after(() => child.kill());
const exited = once(child, 'exit');
// Cleanup waits for the Console to exit, so no child outlives the test.
t.after(async () => { if (child.exitCode === null && child.signalCode === null) child.kill('SIGKILL'); await exited; });
let out = '', err = '';
child.stderr.on('data', (c) => { err += c; });
const timer = setTimeout(() => child.kill('SIGKILL'), 60_000);
const port = await new Promise((resolve, reject) => {
child.stdout.on('data', (c) => { out += c; const m = out.match(/^Mosaic Console: http:\/\/127\.0\.0\.1:(\d+)\/$/m); if (m && out.includes('Ctrl-C')) resolve(Number(m[1])); });
child.on('exit', (code) => reject(new Error(`exited ${code}: ${out}${err}`)));
});
child.on('exit', (code, signal) => reject(new Error(`exited ${code ?? signal} before listening: ${out}${err}`)));
}).finally(() => clearTimeout(timer));
return { out, port };
}
@@ -271,13 +274,24 @@ test('a stamped copy refuses the former unauthenticated Console start', async t
});
const PUBLIC = 'console.test';
// Each request is bounded: a stall, an aborted or incomplete response, or a
// body that is not JSON rejects with the path instead of hanging the test.
function request(port, path, headers = {}) {
return new Promise((resolve, reject) => {
const req = httpRequest({ host: '127.0.0.1', port, path, headers: { host: PUBLIC, ...headers } }, (res) => {
let text = '';
res.setEncoding('utf8');
res.on('data', (c) => { text += c; });
res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: text ? JSON.parse(text) : null }));
res.on('aborted', () => reject(new Error(`${path}: response aborted`)));
res.on('error', reject);
res.on('end', () => {
if (!res.complete) return reject(new Error(`${path}: incomplete response`));
let body = null;
try { body = text ? JSON.parse(text) : null; } catch (e) { return reject(new Error(`${path}: ${res.statusCode} body is not JSON: ${text.slice(0, 200)}`)); }
resolve({ status: res.statusCode, headers: res.headers, body });
});
});
req.setTimeout(10_000, () => req.destroy(new Error(`${path}: no response within 10s`)));
req.on('error', reject); req.end();
});
}
@@ -295,7 +309,7 @@ async function signIn(port, op, iss) {
return async (path) => { const r = await request(port, path, { cookie: session }); return { status: r.status, body: r.body }; };
}
test('serve runs from a copy: /healthz 200, the config check, roles and RELEASE from the copy, the queue from --checkout', async (t) => {
test('serve runs from a copy: /healthz 200, the config check, roles and RELEASE from the copy, the queue from --checkout', { timeout: 240_000 }, async (t) => {
const { dir, script, head } = fixtureRepo(t);
const copies = join(dir, 'copies');
assert.equal((await run(dir, script, ['copy', 'refactor', '--copies', copies])).status, 0);
@@ -335,7 +349,7 @@ test('serve runs from a copy: /healthz 200, the config check, roles and RELEASE
const { out, port } = await serve(t, cli, args, env);
assert.match(out, /^Bus: acme$/m, 'the config check ran from the copy');
assert.match(out, /^Queue: read from the --checkout directory$/m);
const health = await fetch(`http://127.0.0.1:${port}/healthz`);
const health = await fetch(`http://127.0.0.1:${port}/healthz`, { signal: AbortSignal.timeout(10_000) });
assert.equal(health.status, 200);
// Under --sso the body adds each provider's state beside ok (serve.mjs /healthz).
const hb = await health.json();
@@ -358,7 +372,7 @@ test('serve runs from a copy: /healthz 200, the config check, roles and RELEASE
// no packages/queue, so it fails closed.
const plain = await serve(t, cli, ['--business', 'acme', '--sso'], env);
assert.doesNotMatch(plain.out, /^Queue:/m);
assert.equal((await fetch(`http://127.0.0.1:${plain.port}/healthz`)).status, 200);
assert.equal((await fetch(`http://127.0.0.1:${plain.port}/healthz`, { signal: AbortSignal.timeout(10_000) })).status, 200);
const none = await (await signIn(plain.port, op, iss))('/api/queue');
assert.equal(none.status, 503);
assert.equal(none.body.error, 'queue-refused');
+20 -5
View File
@@ -5,9 +5,15 @@
# mosaic-proxy.service under --proxy, from the existing templates, which it
# doesn't change.
#
# case2-unit.sh --user NAME --home DIR --business ID --copy ID --node PATH
# [--profile factory|library] [--proxy] [--pm] [--claude PATH]
# -- SERVE FLAGS
# case2-unit.sh [--check|--check-new] --user NAME --home DIR --business ID
# --copy ID --node PATH [--profile factory|library] [--proxy]
# [--pm] [--claude PATH] -- SERVE FLAGS
#
# --check runs every check a render runs, the installed units' included, then
# exits 0 without writing, enabling or starting anything. --check-new does the
# same as a first install would, without the installed units' checks; it is for
# stack-upgrade.sh --next-args, which moves the installed units aside before its
# render (docs/guides/host-run-sheet.md section 7). Neither writes a file.
#
# --org ID is the same option as --business ID (rename design r2), in each of
# two scopes: this script's own flags before --, and the serve flags after it.
@@ -85,7 +91,7 @@ PROXY_UNIT=mosaic-proxy.service
BUS_UNIT=[email protected]
WEBUI_UNIT=mosaic-webui.service
usage() { echo 'usage: case2-unit.sh --user NAME --home DIR --business ID --copy ID --node PATH [--profile factory|library] [--proxy] [--pm] [--claude PATH] -- SERVE FLAGS' >&2; exit 4; }
usage() { echo 'usage: case2-unit.sh [--check|--check-new] --user NAME --home DIR --business ID --copy ID --node PATH [--profile factory|library] [--proxy] [--pm] [--claude PATH] -- SERVE FLAGS' >&2; exit 4; }
die() { echo "case2-unit: $*" >&2; exit 1; }
# A value the unit carries as itself: no space, %, $, quote, backslash, & or |.
plain() { case "$1" in *[!A-Za-z0-9/._+@-]*) return 1 ;; esac; }
@@ -116,7 +122,8 @@ serve_refused() {
*) die "serve $2" ;;
esac
}
PROFILE=library USER_NAME='' HOME_DIR='' BUSINESS='' COPY_ID='' NODE='' PROXY=0 PM=0 CLAUDE='' CLAUDE_SET=0
PROFILE=library USER_NAME='' HOME_DIR='' BUSINESS='' COPY_ID='' NODE='' PROXY=0 PM=0 CLAUDE='' CLAUDE_SET=0 CHECK=''
case "${1-}" in --check|--check-new) CHECK="$1"; shift ;; esac
while [ $# -gt 0 ]; do
case "$1" in
--business|--org) BUSINESS="${2-}"; selector_take outer id outer_refused business_id_ok "$1" "${@:2:1}"; shift 2 ;;
@@ -720,6 +727,7 @@ else
|| die "installed $PROXY_UNIT requires --proxy; nothing was changed"
fi
for pair in "${PAIRS[@]}"; do
[ "$CHECK" != --check-new ] || break
IFS=: read -r kind name var <<<"$pair"
path="$UNIT_DIR/$name"
[ ! -L "$path" ] || die "refusing: $path is a symlink"
@@ -737,6 +745,13 @@ read -r -a ids <<<"$installed_ids"
for old in "${ids[@]}"; do
[ "$old" = "${ids[0]}" ] || die "the installed units name different copies, ${ids[0]} and $old"
done
if [ "$CHECK" = --check ]; then
echo "case2-unit: check passed: the units for copy $COPY_ID render, and the installed ones match them; nothing was written"
exit 0
elif [ "$CHECK" = --check-new ]; then
echo "case2-unit: check passed: the units for copy $COPY_ID render as a first install; the installed ones were not compared; nothing was written"
exit 0
fi
# `systemctl start` leaves a running unit on the copy it started from, so
# note which units run before the write (row 195 owns the restart).
+137 -21
View File
@@ -2041,12 +2041,16 @@ r_gone() {
while r_alive "$1"; do [ "$SECONDS" -lt "$deadline" ] || return 1; sleep 1; done
}
# Stops the Console, then the host, with SIGTERM within --timeout; one still there gets SIGKILL
# and a FAIL line. The host has to leave the store's writer.lock behind it.
# Stops the Console, the host, then the rehearsal provider, with SIGTERM within --timeout; one
# still there gets SIGKILL and a FAIL line. The host has to leave the store's writer.lock behind it.
r_stop() {
local name pid
for name in console host; do
if [ "$name" = console ]; then pid="$R_CONSOLE_PID"; R_CONSOLE_PID=""; else pid="$R_HOST_PID"; R_HOST_PID=""; fi
for name in console host provider; do
case "$name" in
console) pid="$R_CONSOLE_PID"; R_CONSOLE_PID="" ;;
host) pid="$R_HOST_PID"; R_HOST_PID="" ;;
provider) pid="$R_PROVIDER_PID"; R_PROVIDER_PID="" ;;
esac
[ -n "$pid" ] || continue
kill -TERM "$pid" 2>/dev/null || true
if r_gone "$pid"; then r_pass "$name-stopped" "pid $pid"
@@ -2080,20 +2084,77 @@ r_port_free() {
}
# The scratch config: the live one with dataRoot set to the scratch data root. M3 (#1873): a
# console.sso keeps its provider ids, client ids, algorithms and subjects, but the public origin and
# every issuer move to rehearsal.invalid, a name that never resolves (RFC 6761). The scratch Console
# reaches no real provider: discovery fails as unreachable and the Console starts degraded, after
# every other startup check. No subject is added; the live config's own are the only ones.
# console.sso keeps its provider ids, client ids, algorithms and subjects; the public origin moves to
# rehearsal.invalid, a name that never resolves (RFC 6761). For a stamped copy, whose Console signs
# in, every issuer moves to the rehearsal's own provider (R_PROVIDER_JS, the origin argument). An
# unstamped copy's Console runs without --sso, so its issuers move to rehearsal.invalid too. No
# subject is added; the live config's own are the only ones.
R_CONFIG_JS='
const fs=require("node:fs");const [file,data]=process.argv.slice(1);
const fs=require("node:fs");const [file,data,op]=process.argv.slice(1);
const c=JSON.parse(fs.readFileSync(file,"utf8"));c.dataRoot=data;
const sso=c.console?.sso;
if(sso&&typeof sso==="object"){
sso.publicOrigin="https://rehearsal.invalid";
for(const p of Array.isArray(sso.providers)?sso.providers:[])if(p&&typeof p==="object")p.issuer=`https://rehearsal.invalid/${encodeURIComponent(String(p.id))}`;
for(const p of Array.isArray(sso.providers)?sso.providers:[])if(p&&typeof p==="object")p.issuer=`${op||"https://rehearsal.invalid"}/${encodeURIComponent(String(p.id))}${op?"/":""}`;
}
process.stdout.write(JSON.stringify(c,null,2)+"\n");'
# M3 (#1873): the rehearsal's own OpenID provider, for a stamped copy's --sso Console. An https
# server on 127.0.0.1, port 0, with a certificate openssl makes for this run in DIR. Its TLS key is
# read into memory and its file removed before the server listens; cert.pem stays for the scratch
# Console's NODE_EXTRA_CA_CERTS. Per configured provider, at /<id>/, it serves discovery and a JWKS
# holding one public key made in memory for the first of the provider's algorithms it can make.
# Authorization, token and every other route answer 403 or 404: it issues no code, no token and no
# sign-in. Each request is a line in requests.log. It writes origin when listening and exits when
# PARENT (the rehearsal) is gone, on SIGTERM, or on an error.
R_PROVIDER_JS='
const fs=require("node:fs"),path=require("node:path"),https=require("node:https"),crypto=require("node:crypto"),{spawnSync}=require("node:child_process");
const [dir,cfg,parent]=process.argv.slice(1);
const die=(m)=>{console.error(`rehearsal provider: ${m}`);process.exit(1)};
const sso=JSON.parse(fs.readFileSync(cfg,"utf8")).console?.sso;
if(!sso||!Array.isArray(sso.providers)||!sso.providers.length)die("the live config has no console.sso providers");
const GEN={RS:["rsa",{modulusLength:2048}],PS:["rsa",{modulusLength:2048}],ES256:["ec",{namedCurve:"P-256"}],ES384:["ec",{namedCurve:"P-384"}],ES512:["ec",{namedCurve:"P-521"}],Ed25519:["ed25519",{}],EdDSA:["ed25519",{}]};
const keyFor=(alg)=>{
const g=GEN[alg]??GEN[String(alg).slice(0,2)]??(/^ML-DSA-(44|65|87)$/.test(alg)?[alg.toLowerCase(),{}]:null);
if(!g)return null;
try{const {publicKey}=crypto.generateKeyPairSync(...g);return {...publicKey.export({format:"jwk"}),kid:`rehearsal-${crypto.randomBytes(4).toString("hex")}`,use:"sig",alg}}catch{return null}
};
const providers=new Map();
for(const p of sso.providers){
const id=String(p?.id),algs=Array.isArray(p?.algs)?p.algs:[];
let jwk=null;for(const a of algs){jwk=keyFor(a);if(jwk)break}
if(!jwk)die(`provider ${id}: none of ${algs.join(", ")||"no algorithms"} is one this provider can make a key for`);
providers.set(encodeURIComponent(id),jwk);
}
const tls=path.join(dir,"tls-key.pem"),certFile=path.join(dir,"cert.pem");
const r=spawnSync("openssl",["req","-x509","-newkey","ec","-pkeyopt","ec_paramgen_curve:P-256","-nodes","-days","1","-subj","/CN=127.0.0.1","-addext","subjectAltName=IP:127.0.0.1","-keyout",tls,"-out",certFile],{encoding:"utf8"});
let key,cert;
try{if(r.status!==0){const why=String(r.stderr??"").trim().split("\n").pop();throw new Error(`openssl exit ${r.status??r.error?.code}${why?`: ${why}`:""}`)};key=fs.readFileSync(tls);cert=fs.readFileSync(certFile);fs.chmodSync(certFile,0o600)}
catch(e){die(`cannot make the certificate: ${e.message}`)}
finally{fs.rmSync(tls,{force:true})}
const log=(q,status)=>fs.appendFileSync(path.join(dir,"requests.log"),`${q.method} ${q.url} ${status}\n`);
let origin;
const server=https.createServer({key,cert,requestTimeout:5000,headersTimeout:5000},(q,res)=>{
const send=(status,body)=>{log(q,status);res.writeHead(status,{"content-type":"application/json","cache-control":"no-store"});res.end(JSON.stringify(body))};
if(q.method!=="GET")return send(405,{error:"GET only"});
const m=/^\/([^/?#]+)\/(\.well-known\/openid-configuration|jwks|authorize|token)$/.exec(q.url);
const jwk=m?providers.get(m[1]):undefined;
if(!jwk)return send(404,{error:"not found"});
const issuer=`${origin}/${m[1]}/`;
if(m[2]==="jwks")return send(200,{keys:[jwk]});
if(m[2]!==".well-known/openid-configuration")return send(403,{error:"the rehearsal provider issues no code and no token"});
send(200,{issuer,authorization_endpoint:`${issuer}authorize`,token_endpoint:`${issuer}token`,jwks_uri:`${issuer}jwks`,response_types_supported:["code"],subject_types_supported:["public"],id_token_signing_alg_values_supported:[jwk.alg],code_challenge_methods_supported:["S256"]});
});
server.on("error",(e)=>die(e.message));
server.listen(0,"127.0.0.1",()=>{
origin=`https://127.0.0.1:${server.address().port}`;
fs.writeFileSync(path.join(dir,"origin.tmp"),`${origin}\n`,{mode:0o600});fs.renameSync(path.join(dir,"origin.tmp"),path.join(dir,"origin"));
console.log(`rehearsal provider: ${origin}, ${providers.size} provider(s)`);
});
const stop=()=>{server.close();server.closeAllConnections();process.exit(0)};
process.on("SIGTERM",stop);process.on("SIGINT",stop);process.on("SIGHUP",stop);
setInterval(()=>{if(!fs.existsSync(`/proc/${parent}`))stop()},1000);'
# M3 (#1873): one 0600 placeholder per console.sso client secret slot, in the scratch config
# directory where readSlot looks. Never the live slot: the rehearsal reads no secret.
R_SLOTS_JS='
@@ -2145,12 +2206,13 @@ fs.writeFileSync(out,JSON.stringify(b,null,2)+"\n",{mode:0o600,flag:"wx"});'
# notify-off the notifier config is {"notifyVersion":1,"binding":null}, 0600 (mos-0962)
# tracker-configured bootConfig sets no trackers (mos-0962)
# credential-ref-live every credential reference is a file under placeholders/ (mos-0963)
# sso-isolated every console.sso origin and issuer is rehearsal.invalid and every client
# secret slot is a scratch placeholder (M3, #1873)
# sso-isolated the console.sso origin is rehearsal.invalid, every issuer is the
# rehearsal provider's (OP, a stamped copy) or on rehearsal.invalid, and every
# client secret slot is a scratch placeholder (M3, #1873)
# Exit 0 when all pass, 1 on the first FAIL.
R_GUARD_JS='
const fs=require("node:fs"),path=require("node:path");const {pathToFileURL}=require("node:url");
const [copy,id,live,scratch,ph]=process.argv.slice(1);
const [copy,id,live,scratch,ph,op]=process.argv.slice(1);
const real=(p)=>{try{return fs.realpathSync(p)}catch{return null}};
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,"utf8");return s[s.lastIndexOf(")")+2]!=="Z"}catch{return false}};
const pass=(n,w)=>console.log(`PASS ${n}${w?": "+w:""}`);
@@ -2195,11 +2257,13 @@ const fail=(n,w)=>{console.log(`FAIL ${n}: ${w}`);process.exit(1)};
if(host(sso.publicOrigin)!=="rehearsal.invalid")fail("sso-isolated",`console.sso.publicOrigin is ${sso.publicOrigin}, not https://rehearsal.invalid`);
const sd=real(path.join(path.dirname(path.resolve(process.env.MOSAIC_CONFIG)),"secrets","mosaic-stack"));
for(const p of sso.providers){
if(host(p.issuer)!=="rehearsal.invalid")fail("sso-isolated",`console.sso provider ${p.id} issuer is ${p.issuer}, not on rehearsal.invalid`);
const want=op?`${op}/${encodeURIComponent(String(p.id))}/`:null;
if(op&&p.issuer!==want)fail("sso-isolated",`console.sso provider ${p.id} issuer is ${p.issuer}, not ${want} at the rehearsal provider`);
if(!op&&host(p.issuer)!=="rehearsal.invalid")fail("sso-isolated",`console.sso provider ${p.id} issuer is ${p.issuer}, not on rehearsal.invalid`);
const f=real(path.join(path.dirname(path.resolve(process.env.MOSAIC_CONFIG)),"secrets","mosaic-stack",p.clientSecret.slot));
if(f===null||sd===null||path.dirname(f)!==sd||fs.readFileSync(f,"utf8")!=="rehearsal-placeholder-secret\n")fail("sso-isolated",`console.sso provider ${p.id} has no scratch placeholder secret`);
}
pass("sso-isolated",`${sso.providers.length} provider(s), each at rehearsal.invalid with a placeholder secret`);
pass("sso-isolated",`${sso.providers.length} provider(s), each ${op?`at the rehearsal provider ${op}`:"at rehearsal.invalid"} with a placeholder secret`);
})().catch((e)=>fail("guard",String(e?.message??e).split("\n")[0]));'
# Opens a data root's store with a copy's Store, as the bus does at start, with no migration
@@ -2267,6 +2331,18 @@ process.stdout.write(why);' -- "$out" "$BUSINESS" "$R_HOST_PID" "$START_MS" 2>&1
R_GATE=healthz
node -e '
fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>process.exit(r.status===200?0:1),()=>process.exit(1));' -- "$R_PORT" || return 1
# M3 (#1873): a stamped copy's Console must have found every configured provider at the
# rehearsal provider, not started degraded.
if [ -n "$R_PROVIDER" ]; then
R_GATE=sso-ready
R_WHY="$(node -e '
const ids=JSON.parse(require("node:fs").readFileSync(process.argv[2],"utf8")).console.sso.providers.map((p)=>String(p.id));
fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>r.json()).then((h)=>{
const s=h?.sso&&typeof h.sso==="object"?h.sso:{};
process.stdout.write(ids.filter((id)=>s[id]!=="ready").map((id)=>`provider ${id} is ${s[id]??"missing"}, not ready`).join("; "));
},(e)=>process.stdout.write(`healthz unreadable: ${e.message}`));' -- "$R_PORT" "$R/config/config.json" 2>&1)" || R_WHY="${R_WHY:-healthz unreadable}"
[ -z "$R_WHY" ] || return 1
fi
R_GATE=copy-match
tr '\0' ' ' < "/proc/$R_HOST_PID/cmdline" | grep -qF " $copy/packages/cli/src/cli.mjs bus start $BUSINESS --no-launch" || { R_WHY="the host's command line is not $copy's cli.mjs bus start $BUSINESS --no-launch"; return 1; }
tr '\0' ' ' < "/proc/$R_CONSOLE_PID/cmdline" | grep -qF " $copy/packages/webui/src/cli.mjs serve " || { R_WHY="the Console's command line is not $copy's webui cli.mjs serve"; return 1; }
@@ -2279,7 +2355,43 @@ r_gate() {
[ "$SECONDS" -lt "$deadline" ] || { r_fail "$R_GATE" "${R_WHY:-no answer within ${TIMEOUT}s}"; return 1; }
sleep 1
done
r_pass gate "host-up, bus-status, console-up, healthz and copy-match on $TO"
r_pass gate "host-up, bus-status, console-up, healthz${R_PROVIDER:+, sso-ready} and copy-match on $TO"
}
# M3 (#1873): starts R_PROVIDER_JS for a stamped copy, in $R/provider, and waits for its origin.
r_provider() {
local deadline=$((SECONDS + TIMEOUT))
mkdir -m 700 "$R/provider" || r_abort provider-start "cannot make $R/provider"
env -i HOME="$HOME" PATH="$PATH" node -e "$R_PROVIDER_JS" -- "$R/provider" "$LIVE_CFG" "$$" > "$R/provider/provider.log" 2>&1 &
R_PROVIDER_PID=$!
until [ -s "$R/provider/origin" ]; do
r_alive "$R_PROVIDER_PID" || { wait "$R_PROVIDER_PID" 2>/dev/null || true; R_PROVIDER_PID=""; r_abort provider-start "the rehearsal provider exited: $(tail -n 1 "$R/provider/provider.log" 2>/dev/null)"; }
[ "$SECONDS" -lt "$deadline" ] || r_abort provider-start "the rehearsal provider wrote no origin within ${TIMEOUT}s"
sleep 0.2
done
R_PROVIDER="$(cat "$R/provider/origin")"
[[ "$R_PROVIDER" =~ ^https://127\.0\.0\.1:[0-9]{1,5}$ ]] || r_abort provider-start "the rehearsal provider's origin $R_PROVIDER is not https on 127.0.0.1"
[ ! -e "$R/provider/tls-key.pem" ] || r_abort provider-start "the rehearsal provider left its TLS key at $R/provider/tls-key.pem"
r_pass provider-start "pid $R_PROVIDER_PID at $R_PROVIDER: discovery and a JWKS from keys made for this run, no code and no token; its certificate $R/provider/cert.pem is the trust the scratch Console alone gets"
}
# M3 (#1873): the Console fetched each provider's discovery and JWKS from the rehearsal provider and
# asked it for no code and no token.
r_provider_seen() {
local why
why="$(node -e '
const fs=require("node:fs");const [cfg,log]=process.argv.slice(1);
const lines=fs.existsSync(log)?fs.readFileSync(log,"utf8").split("\n").filter(Boolean):[];
const why=[];
for(const p of JSON.parse(fs.readFileSync(cfg,"utf8")).console.sso.providers){
const seg=encodeURIComponent(String(p.id));
for(const what of [".well-known/openid-configuration","jwks"])if(!lines.includes(`GET /${seg}/${what} 200`))why.push(`no GET /${seg}/${what} 200`);
}
for(const l of lines)if(/\/(authorize|token)\b/.test(l))why.push(`a request for ${l}`);
process.stdout.write(why.join("; "));' -- "$R/config/config.json" "$R/provider/requests.log" 2>&1)" || why="${why:-cannot read $R/provider/requests.log}"
if [ -z "$why" ]; then r_pass provider-discovery "the Console fetched each provider's discovery and JWKS from $R_PROVIDER and asked for no code or token"
else r_fail provider-discovery "$why; see $R/provider/requests.log"
fi
}
# No role session: the host wrote no sessions.json and no launches directory or log.
@@ -2343,11 +2455,12 @@ do_rehearse() {
R="$DATA/rehearsals/$stamp" STEPS="$DATA/rehearsals/$stamp/steps.log"
printf '%s\n' "${R_EARLY[@]}" > "$STEPS" && printf '%s\n' "$before" > "$R/backup.before" && printf '%s\n' "$live" > "$R/live.before" ||
r_abort scratch-build "cannot write the sums to $R"
[ -z "$R_STAMPED" ] || r_provider
(
umask 077
mkdir "$R/config" "$R/config/businesses" "$R/placeholders" "$R/data" "$R/data/bus" "$R/data/notify" "$R/data/notify/$BUSINESS" &&
cp "$R_BK"/bus/* "$R/data/bus/" &&
node -e "$R_CONFIG_JS" -- "$LIVE_CFG" "$R/data" > "$R/config/config.json" &&
node -e "$R_CONFIG_JS" -- "$LIVE_CFG" "$R/data" "$R_PROVIDER" > "$R/config/config.json" &&
node -e "$R_BUSINESS_JS" -- write "$LIVE_BIZ" "$R/config/businesses/$BUSINESS.json" "$R/placeholders" &&
printf '{"notifyVersion":1,"binding":null}\n' > "$R/data/notify/$BUSINESS/notify.json" &&
{ ! node -e 'process.exit(JSON.parse(require("node:fs").readFileSync(process.argv[1],"utf8")).console?.sso?0:1)' -- "$R/config/config.json" ||
@@ -2364,7 +2477,7 @@ do_rehearse() {
[ -z "$R_STAMPED" ] || r_pass deployment "$TO is stamped: $R/data/.deployed from its writeDeployment, the live accounts.json copied to $R/config, the Console starts with --sso"
local out
rc=0; out="$(r_env node -e "$R_GUARD_JS" -- "$COPIES/$TO" "$BUSINESS" "$DATA" "$R/data" "$R/placeholders" 2>&1)" || rc=$?
rc=0; out="$(r_env node -e "$R_GUARD_JS" -- "$COPIES/$TO" "$BUSINESS" "$DATA" "$R/data" "$R/placeholders" "$R_PROVIDER" 2>&1)" || rc=$?
while IFS= read -r i; do r_line "$i"; done <<< "$out"
[ "$rc" = 0 ] || { R_RC=1; r_end; }
@@ -2379,9 +2492,12 @@ do_rehearse() {
local -a tester_rehearsal=()
if grep -q 'MOSAIC_TESTER_SOCKET_V1' "$COPIES/$TO/packages/webui/src/cli.mjs"; then tester_rehearsal=(--tester-socket); fi
[ -z "$R_STAMPED" ] || tester_rehearsal+=(--sso)
env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" node "$COPIES/$TO/packages/webui/src/cli.mjs" serve --port "$R_PORT" --business "$BUSINESS" "${tester_rehearsal[@]}" > "$R/console.log" 2>&1 &
# M3 (#1873): the rehearsal provider's certificate is trusted by the scratch Console only.
local -a r_ca=()
[ -z "$R_PROVIDER" ] || r_ca=(NODE_EXTRA_CA_CERTS="$R/provider/cert.pem")
env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" "${r_ca[@]}" node "$COPIES/$TO/packages/webui/src/cli.mjs" serve --port "$R_PORT" --business "$BUSINESS" "${tester_rehearsal[@]}" > "$R/console.log" 2>&1 &
R_CONSOLE_PID=$!
if r_gate; then r_no_session; fi
if r_gate; then r_no_session; [ -z "$R_PROVIDER" ] || r_provider_seen; fi
r_end
}
@@ -2419,7 +2535,7 @@ FROM="" TO="" ACT="" RS="" POST="" BACKUP="" TAKEN="" RESTORED="" STEPS="" COUNT
STORE_SCHEMA="" START_EPOCH="" START_MS=0 GATE="" GATE_WHY="" S_CHANGED=""
P_COPY="" P_PM=0 P_PENDING=0 P_SCHEMA="" P_CLAUDE="" P_CONSOLE=() RUN_PM=0 RUN_PENDING=0 RUN_CONSOLE=() DROP_TESTER="" PROOF_PLAN="" PROOF_STEP="" SEND_PLAN="" SEND_STEP="" PLAN_REFUSE=()
BASE_CONSOLE=() SERVE_FLAGS=() SSO_PLAN="" SSO_STEP="" SSO_DROP="" SSO_NO_TARGET=""
R="" R_RC=0 R_EARLY=() R_BK="" R_HOST_PID="" R_CONSOLE_PID="" R_STARTED="" LIVE_CFG="" LIVE_BIZ="" LIVE_FILES=() R_STAMPED=""
R="" R_RC=0 R_EARLY=() R_BK="" R_HOST_PID="" R_CONSOLE_PID="" R_PROVIDER_PID="" R_PROVIDER="" R_STARTED="" LIVE_CFG="" LIVE_BIZ="" LIVE_FILES=() R_STAMPED=""
O_ACT="" FLOOR=0
case "$OP" in
adopt) do_adopt ;;
+128 -18
View File
@@ -6,8 +6,9 @@
# agents/darkwing/work/queue-233/BUILD.md.
#
# stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
# [--rehearse-port N] [--timeout S]
# stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
# [--rehearse-port N] [--next-args] [--timeout S]
# stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--next-args]
# [--timeout S]
#
# Root runs it from a root-owned checkout of the reviewed commit, never from a copy, because the
# stack's user can write the copies. The unit values come from /etc/mosaic-stack/<instance>.args,
@@ -20,6 +21,20 @@
# output but exit codes and the rehearsal's BACKUP_STORE line (sage-1653), which it copies into its
# step log after a match against a fixed pattern and decides nothing on.
#
# Before anything stops, case2-unit.sh --check renders the installed copy from the args file and
# compares the installed units with it, then checks the target's render the same way; either
# refusal is render-refused, with the units left running (#1873 M3).
#
# --next-args changes the serve flags, which a re-render otherwise refuses (row 195's installed-unit
# match): /etc/mosaic-stack/<instance>.args.next, root's with no group or other write, must equal
# the args file before `--` but for --copy's value, and differ from it after `--`. The target
# renders from it through case2-unit.sh --check-new before anything stops. After A3, root moves the
# installed unit files into a new <instance>.replaced.XXXXXX directory beside the args file, and A4
# renders from the next args as a first install. A failure from there moves the replaced files back
# unchanged, restarts them on the old copy and gates it, with no render. After confirm, the next
# args become the args file, and the replaced directory keeps the old unit files and args file for
# a later rollback with --next-args (docs/guides/host-run-sheet.md section 7).
#
# --no-rehearsal is for a stack whose log records no backup. The rehearsal runs either way: exit 3
# from it (no-backup) goes on only with the flag, and exit 0 with the flag refuses
# rehearsal-required (sage-1650, sage-1652).
@@ -47,8 +62,8 @@ REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
usage() {
cat >&2 <<'USAGE'
usage: stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal] [--rehearse-port N] [--timeout S]
stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
usage: stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal] [--rehearse-port N] [--next-args] [--timeout S]
stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--next-args] [--timeout S]
USAGE
exit 4
}
@@ -70,7 +85,7 @@ ARGV=("$@")
[ $# -gt 0 ] || usage
OP="$1"; shift
case "$OP" in upgrade|rollback) ;; *) usage ;; esac
INSTANCE="" BUNDLE="" SHA="" MIGRATE=0 NO_REHEARSAL=0 R_PORT=7339 TIMEOUT=60 TO="" RESTORE="" EVENTS=""
INSTANCE="" BUNDLE="" SHA="" MIGRATE=0 NO_REHEARSAL=0 R_PORT=7339 TIMEOUT=60 TO="" RESTORE="" EVENTS="" NEXT=0 ASIDE="" MOVED=() REPLACED_NOTE=""
while [ $# -gt 0 ]; do
case "$1" in
--instance) [ $# -ge 2 ] || usage; INSTANCE="$2"; shift 2 ;;
@@ -79,6 +94,7 @@ while [ $# -gt 0 ]; do
--migrate) [ "$OP" = upgrade ] || usage; MIGRATE=1; shift ;;
--no-rehearsal) [ "$OP" = upgrade ] || usage; NO_REHEARSAL=1; shift ;;
--rehearse-port) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{3,4}$ ]] && [ "$2" -le 65535 ] || usage; R_PORT="$2"; shift 2 ;;
--next-args) NEXT=1; shift ;;
--timeout) [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
--to) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; TO="$2"; shift 2 ;;
--restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ $ID_RE ]] || usage; RESTORE="$2"; shift 2 ;;
@@ -125,8 +141,8 @@ logl "start rc=0 argv:$(printf ' %q' "$0" "${ARGV[@]}")"
# The selector scopes: selector.sh defines functions only, from this root-owned checkout, as RENDER is.
. "$REPO/scripts/selector.sh" || refuse checkout-unreadable "cannot read $REPO/scripts/selector.sh"
args_business_ok() { [[ "$1" =~ ^[a-z0-9][a-z0-9-]*$ ]]; }
args_outer_refused() { refuse args-unreadable "$ETC/$INSTANCE.args before --: $2"; }
args_serve_refused() { refuse args-unreadable "$ETC/$INSTANCE.args after --: $2"; }
args_outer_refused() { refuse args-unreadable "$ARGS_FILE before --: $2"; }
args_serve_refused() { refuse args-unreadable "$ARGS_FILE after --: $2"; }
# I2's file: one case2-unit.sh argument per line, owned by root with no group or other write. Sets
# ARGS (the lines), S_USER, S_HOME, BUSINESS, NODE, ARGS_COPY, ARGS_PROFILE and ARGS_PROXY, and the paths derived
@@ -140,7 +156,8 @@ args_serve_refused() { refuse args-unreadable "$ETC/$INSTANCE.args after --: $2"
# says which of them take a value, so a value is never taken for a selector; a token that grammar
# lacks refuses, as the render would.
read_args() {
local f="$ETC/$INSTANCE.args" st i sep copies=0 v profile=library served
ARGS_FILE="${1:-$ETC/$INSTANCE.args}"
local f="$ARGS_FILE" st i sep copies=0 v profile=library served
[ -f "$f" ] && [ ! -L "$f" ] || refuse args-unreadable "no regular file $f"
st="$(stat -c '%u %a' "$f")" || refuse args-unreadable "cannot stat $f"
[ "${st% *}" = "$(id -u)" ] && (( (8#${st#* } & 8#022) == 0 )) || refuse args-unreadable "$f must be $(id -un)'s with no group or other write (it is uid ${st% *}, mode ${st#* })"
@@ -224,17 +241,82 @@ legacy_console() {
done
}
# RENDER_ARGS: the args file's lines with copy $1 in place of --copy's value.
# RENDER_ARGS: the lines of array $2 (default ARGS) with copy $1 in place of --copy's value.
args_with() {
local i past=""
local -n lines_="${2:-ARGS}"
RENDER_ARGS=()
for ((i = 0; i < ${#ARGS[@]}; i++)); do
RENDER_ARGS+=("${ARGS[i]}")
[ "${ARGS[i]}" != -- ] || past=1
if [ -z "$past" ] && [ "${ARGS[i]}" = --copy ]; then RENDER_ARGS+=("$1"); i=$((i + 1)); fi
for ((i = 0; i < ${#lines_[@]}; i++)); do
RENDER_ARGS+=("${lines_[i]}")
[ "${lines_[i]}" != -- ] || past=1
if [ -z "$past" ] && [ "${lines_[i]}" = --copy ]; then RENDER_ARGS+=("$1"); i=$((i + 1)); fi
done
}
# OLD_ARGS: the args file's lines, which the installed units were rendered from. Under --next-args,
# ARGS becomes the next file's lines, which must equal OLD_ARGS before -- but for --copy's value and
# differ after it.
read_next() {
OLD_ARGS=("${ARGS[@]}")
[ "$NEXT" = 1 ] || return 0
local f="$ETC/$INSTANCE.args.next" a b i copy="$ARGS_COPY"
unset SELECTOR_SEEN_outer SELECTOR_VALUE_outer SELECTOR_SEEN_serve SELECTOR_VALUE_serve
read_args "$f"
# read_old checks the installed units against the args file's copy, not the next file's.
ARGS_COPY="$copy"
args_with COPY OLD_ARGS; a=("${RENDER_ARGS[@]}")
args_with COPY; b=("${RENDER_ARGS[@]}")
for ((i = 0; i < ${#a[@]} && i < ${#b[@]}; i++)); do
[ "${a[i]}" = "${b[i]}" ] || refuse next-args-differ "$f must equal $ETC/$INSTANCE.args before -- but for --copy's value; line $((i + 1)) differs"
[ "${a[i]}" != -- ] || break
done
[ "${a[*]}" != "${b[*]}" ] || [ "${#a[@]}" != "${#b[@]}" ] ||
refuse next-args-same "$f has the same serve flags as $ETC/$INSTANCE.args; run without --next-args"
}
# Before anything stops: the installed units against OLD's render from the args file, so a rollback
# render-old works, then copy $1's render, against the installed units or, under --next-args, as a
# first install from the next args.
render_checks() {
args_with "$OLD" OLD_ARGS
step render-old-check "$RENDER" --check "${RENDER_ARGS[@]}" ||
refuse render-refused "case2-unit.sh --check refuses $OLD's units from $ETC/$INSTANCE.args, so a rollback could not render them; its message is above"
args_with "$1"
if [ "$NEXT" = 1 ]; then
step render-check "$RENDER" --check-new "${RENDER_ARGS[@]}" ||
refuse render-refused "case2-unit.sh --check-new refuses $1's units from $ETC/$INSTANCE.args.next; its message is above"
else
step render-check "$RENDER" --check "${RENDER_ARGS[@]}" ||
refuse render-refused "case2-unit.sh --check refuses $1's units from $ETC/$INSTANCE.args; its message is above. A re-pin that changes the serve flags takes --next-args (docs/guides/host-run-sheet.md section 7)"
fi
}
# --next-args, after A3: the installed unit files into a new directory beside the args file, so A4
# renders as a first install. MOVED lists the files moved, for units_back.
units_aside() {
local u rc=0
local -a files=([email protected] "$CON_UNIT")
[ "$ARGS_PROXY" = 0 ] || files+=("$PROXY_UNIT")
ASIDE="$(mktemp -d "$ETC/$INSTANCE.replaced.XXXXXX")" && chmod 0700 "$ASIDE" || rc=1
if [ "$rc" = 0 ]; then
for u in "${files[@]}"; do
mv -T -- "$UNIT_DIR/$u" "$ASIDE/$u" || { rc=1; break; }
MOVED+=("$u")
done
fi
logl "units-aside rc=$rc dir: ${ASIDE:-none} moved: ${MOVED[*]}"
return "$rc"
}
# The moved files back, unchanged, over whatever A4 wrote, then the replaced directory removed.
units_back() {
local u rc=0
for u in "${MOVED[@]}"; do mv -fT -- "$ASIDE/$u" "$UNIT_DIR/$u" || rc=1; done
[ "$rc" != 0 ] || rmdir -- "$ASIDE" || rc=1
logl "units-back rc=$rc dir: $ASIDE moved: ${MOVED[*]}"
return "$rc"
}
# The new args file, written beside the old one before anything stops, moved into place only after
# confirm passes. $1 is the copy.
prepare_args() {
@@ -334,9 +416,25 @@ rollback_to_old() {
stop_wait || { stop_units; finish 2 rollback-failed "$cause failed and the units did not stop for the rollback; $STOPPED_UNITS are stopped"; }
step revert "${AS_USER[@]}" "$SR" revert "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" ||
{ stop_units; finish 2 rollback-failed "$cause failed and revert failed; $STOPPED_UNITS are stopped"; }
args_with "$OLD"
step render-old "$RENDER" "${RENDER_ARGS[@]}" ||
{ stop_units; finish 2 rollback-failed "$cause failed and the render of $OLD failed; $STOPPED_UNITS are stopped"; }
if [ -n "$ASIDE" ]; then
# --next-args: the replaced files back as they were, then the start render-old would do.
units_back ||
{ stop_units; finish 2 rollback-failed "$cause failed and the replaced units in $ASIDE did not all move back; $STOPPED_UNITS are stopped"; }
step daemon-reload "$SYSTEMCTL" daemon-reload ||
{ stop_units; finish 2 rollback-failed "$cause failed and daemon-reload failed after the replaced units moved back; $STOPPED_UNITS are stopped"; }
if [ "$ARGS_PROXY" = 1 ]; then
step start-proxy "$SYSTEMCTL" start "$PROXY_UNIT" ||
{ stop_units; finish 2 rollback-failed "$cause failed and the proxy did not start for $OLD; $STOPPED_UNITS are stopped"; }
fi
step start-bus "$SYSTEMCTL" start "$BUS_UNIT" ||
{ stop_units; finish 2 rollback-failed "$cause failed and the bus did not start for $OLD; $STOPPED_UNITS are stopped"; }
step start-console "$SYSTEMCTL" start "$CON_UNIT" ||
{ stop_units; finish 2 rollback-failed "$cause failed and the Console did not start for $OLD; $STOPPED_UNITS are stopped"; }
else
args_with "$OLD" OLD_ARGS
step render-old "$RENDER" "${RENDER_ARGS[@]}" ||
{ stop_units; finish 2 rollback-failed "$cause failed and the render of $OLD failed; $STOPPED_UNITS are stopped"; }
fi
# confirm --rollback exits 1 when OLD passed its gate and 2 when it failed; anything else is a
# refusal, with OLD's gate never run.
local rc=0
@@ -360,18 +458,27 @@ run_steps() {
*) rollback_to_old "$id" A3 ;;
esac
# A4
if [ "$NEXT" = 1 ]; then units_aside || rollback_to_old "$id" A4; fi
args_with "$id"
step render "$RENDER" "${RENDER_ARGS[@]}" || rollback_to_old "$id" A4
# A5
step confirm "${AS_USER[@]}" "$SR" confirm "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" --timeout "$TIMEOUT" ||
rollback_to_old "$id" A5
if [ "$NEXT" = 1 ]; then
# The old args file joins the replaced units before the next one takes its place.
cp -p -- "$ETC/$INSTANCE.args" "$ASIDE/$INSTANCE.args" && rm -f -- "$ETC/$INSTANCE.args.next" &&
logl "next-args rc=0 replaced: $ASIDE" ||
{ logl "next-args rc=1 replaced: $ASIDE"; finish 5 args-failed "$id is live and confirmed, but the old args file did not reach $ASIDE; $ETC/$INSTANCE.args still names $OLD; copy it there, then put $NEW_ARGS in its place and remove $ETC/$INSTANCE.args.next"; }
fi
if mv -f "$NEW_ARGS" "$ETC/$INSTANCE.args"; then logl "args rc=0 copy: $id"
else logl "args rc=1 copy: $id"; finish 5 args-failed "$id is live and confirmed, but $ETC/$INSTANCE.args still names $OLD; put $NEW_ARGS in its place"
fi
REPLACED_NOTE="${ASIDE:+; the replaced units and args file are in $ASIDE}"
}
do_upgrade() {
read_args
read_next
legacy_console
# Before A1: the bundle into the stack's user's clone, then the copy from it (I1).
step fetch "${AS_USER[@]}" git -C "$CLONE" fetch -q "$BUNDLE" refs/heads/refactor:refs/remotes/origin/refactor ||
@@ -390,9 +497,10 @@ do_upgrade() {
local -a migrate=()
[ "$MIGRATE" = 0 ] || migrate=(--migrate)
plan activate "$SHA" "$BUSINESS" "${migrate[@]}"
render_checks "$SHA"
rehearse
run_steps "$SHA" activate "$SHA" "$BUSINESS" "${migrate[@]}"
finish 0 ok "upgrade ok: $SHA is live for $BUSINESS"
finish 0 ok "upgrade ok: $SHA is live for $BUSINESS$REPLACED_NOTE"
}
# The rehearsal against the log's newest backup, before anything stops. Its output goes to the
@@ -419,6 +527,7 @@ rehearse() {
# check: the target is an older copy than this checkout, and no rehearsal.
do_rollback() {
read_args
read_next
legacy_console
[ -x "$COPIES_DIR/$TO/scripts/mosaic" ] || refuse copy-missing "no $COPIES_DIR/$TO/scripts/mosaic"
hostcheck
@@ -427,8 +536,9 @@ do_rollback() {
[ -z "$RESTORE" ] || restore=(--restore "$RESTORE")
[ -z "$EVENTS" ] || restore+=(--events "$EVENTS")
plan rollback "$BUSINESS" --to "$TO" "${restore[@]}"
render_checks "$TO"
run_steps "$TO" rollback "$BUSINESS" --to "$TO" "${restore[@]}"
finish 0 ok "rollback ok: $TO is live for $BUSINESS"
finish 0 ok "rollback ok: $TO is live for $BUSINESS$REPLACED_NOTE"
}
case "$OP" in