fix(cli): M3 review round: re-pin migration, rehearsal provider (#1873)
An installed unit must equal the render but for the copy id, so a re-pin that adds --sso refused render-refused. case2-unit gains --check (every render check, nothing written) and --check-new (a first install's checks). stack-upgrade runs both checks before anything stops, and --next-args reads root's <instance>.args.next: equal before -- but for --copy, different after it. After the stage it moves the installed unit files into a new <instance>.replaced.XXXXXX directory and renders a first install. A failure moves them back unchanged and restarts the old copy, and confirm swaps the args file. The run sheet gives the commands, the rollback past the migration included. A stamped rehearsal now starts an isolated provider on 127.0.0.1 port 0 with a certificate and keys made for the run. It serves discovery and a JWKS for the live provider ids and answers 403 to authorize and token. Only the scratch Console trusts its certificate. The gate requires each provider ready (sso-ready); provider-discovery requires discovery and JWKS fetches and no code or token request. The provider stops with the run, and by itself once the run's shell is gone. The system-path test helper asserts the log's from copy again. The pinned-copy sign-in requests are bounded, and cleanup waits for each Console to exit.
This commit is contained in:
1 parent
db822c5b9d
commit
9acc378555
10 files changed
+766
-116
No files matched your search
+57
-13
@@ -808,9 +808,10 @@ the port and the backup's `SHA256SUMS`. Then it makes
|
||||
- `steps.log` holds the same PASS and FAIL lines the run prints, with times;
|
||||
`host.log`, `console.log` and, after a migration, `migrate.log` sit beside it.
|
||||
- When the live config has `console.sso` (M3, #1873), the scratch copy keeps
|
||||
its provider ids, client ids, algorithms and subjects, and moves the public
|
||||
origin and every issuer to `rehearsal.invalid`, a name that never resolves.
|
||||
Each client secret slot is a 0600 file under `config/secrets/mosaic-stack/`
|
||||
its provider ids, client ids, algorithms and subjects and moves the public
|
||||
origin to `rehearsal.invalid`, a name that never resolves. For a stamped
|
||||
target each issuer becomes `ORIGIN/ID/` at the rehearsal provider below;
|
||||
otherwise it moves to `rehearsal.invalid` too. Each client secret slot is a 0600 file under `config/secrets/mosaic-stack/`
|
||||
holding `rehearsal-placeholder-secret`; the live secret is never read.
|
||||
|
||||
A stamped target copy (one with `.stack-build.json`) serves only with `--sso`,
|
||||
@@ -819,10 +820,27 @@ so its rehearsal needs the live instance's human-made SSO steps. Without a live
|
||||
beside the config it refuses `accounts-not-initialized`. Both name the run-sheet
|
||||
step. Otherwise it copies the live `accounts.json` into `config/`, writes
|
||||
`data/.deployed` with the target copy's own `writeDeployment`, prints
|
||||
`PASS deployment`, and starts the Console with `--sso`. The scratch Console
|
||||
runs every startup check, finds no provider at `rehearsal.invalid`, and starts
|
||||
degraded. The live `accounts.json` joins the end check `live-config-unchanged`.
|
||||
The rehearsal adds no subject and no account.
|
||||
`PASS deployment`, and starts the Console with `--sso`.
|
||||
|
||||
Before that, a stamped run starts a rehearsal provider and prints
|
||||
`PASS provider-start`. It is a node process on `127.0.0.1` port 0 that serves
|
||||
HTTPS with a certificate made for this run, under `provider/` (mode 0700). For
|
||||
each live provider id it serves discovery and a JWKS whose public key it made
|
||||
in memory for the first algorithm it can make from that provider's list. Its
|
||||
authorize and token routes answer 403: it issues no code and no token, and
|
||||
it reads no live secret. It logs each request to `provider/requests.log`,
|
||||
deletes its TLS key file once it has read it, and exits on SIGTERM or when
|
||||
the rehearsal's process is gone. `provider-start` fails when it exits early,
|
||||
writes no origin within the timeout, writes an origin that is not
|
||||
`https://127.0.0.1:PORT`, or leaves its key file. Only the scratch Console gets
|
||||
`NODE_EXTRA_CA_CERTS` naming `provider/cert.pem`; the host does not.
|
||||
|
||||
The scratch Console runs every startup check against that provider. The gate
|
||||
adds `sso-ready`: `/healthz` must report each provider `ready`. After the gate,
|
||||
`provider-discovery` requires a 200 discovery and JWKS request per provider and
|
||||
no authorize or token request. The end stops the provider with the Console and
|
||||
host and prints `provider-stopped`. The live `accounts.json` joins the end check
|
||||
`live-config-unchanged`. The rehearsal adds no subject and no account.
|
||||
|
||||
A guard then loads the scratch config with the target's code and refuses
|
||||
with a named check: `data-root-live` when the scratch data root resolves to
|
||||
@@ -830,8 +848,10 @@ the live one, `sessions-live` when a `sessions.json` there names a running
|
||||
pid (the run never deletes the file), `data-root-scratch`, `notify-off`,
|
||||
`tracker-configured` when `bootConfig` still sets a tracker,
|
||||
`credential-ref-live` for an env reference or a file outside `placeholders/`,
|
||||
and `sso-isolated` when a `console.sso` origin or issuer is not on
|
||||
`rehearsal.invalid` or a slot is not its scratch placeholder.
|
||||
and `sso-isolated` when a `console.sso` origin is not on `rehearsal.invalid`,
|
||||
an issuer is not `ORIGIN/ID/` at the rehearsal provider (not on
|
||||
`rehearsal.invalid` for an unstamped target), or a slot is not its scratch
|
||||
placeholder.
|
||||
When the backup's schema differs from the target's, the run checks that the
|
||||
target refuses the store, migrates it, opens it, migrates it again as a
|
||||
no-op, and that the pointer's copy refuses the migrated store.
|
||||
@@ -840,7 +860,8 @@ It starts `scripts/mosaic bus start BUSINESS --no-launch` and the Console's
|
||||
`serve --port N --business BUSINESS` from the target copy, without `--send`.
|
||||
The gate needs the host's `launches disabled` line, a live `bus status` host
|
||||
with this run's pid and a boot from this start, the Console's address line,
|
||||
`/healthz` 200, and both command lines on the target copy. Then it checks
|
||||
`/healthz` 200 (and `sso-ready` for a stamped target), and both command lines
|
||||
on the target copy. Then it checks
|
||||
that the host wrote no `sessions.json` and no `launches`, stops the Console
|
||||
and the host (SIGTERM, then SIGKILL and a FAIL after the timeout), checks that
|
||||
the host released `writer.lock`, and compares the backup and the live config
|
||||
@@ -927,8 +948,9 @@ the target migrates the store and the run lacks `--migrate` (row 233). Under
|
||||
|
||||
```
|
||||
scripts/stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
|
||||
[--rehearse-port N] [--timeout S]
|
||||
scripts/stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
|
||||
[--rehearse-port N] [--next-args] [--timeout S]
|
||||
scripts/stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--next-args]
|
||||
[--timeout S]
|
||||
```
|
||||
|
||||
Root runs it from a root-owned checkout of the reviewed commit, never from a
|
||||
@@ -968,7 +990,13 @@ user's clone `<home>/src/mosaic-stack`:
|
||||
(`args-differ`), and
|
||||
`activate SHA BUSINESS --units external --plan --expect-from OLD` exits 0
|
||||
(`plan-refused`, which covers `open-activation`, `migrate-required` and
|
||||
`pointer-differs`).
|
||||
`pointer-differs`). Then the render checks (#1873 M3):
|
||||
`case2-unit.sh --check` renders OLD from the args file and compares the
|
||||
installed units with it, so a rollback could render them, then checks
|
||||
SHA's render the same way. Either refusal is `render-refused`. A re-pin
|
||||
that changes the serve flags fails the second check, because an
|
||||
installed unit must equal the render but for the copy id (row 195); it
|
||||
takes `--next-args`, below.
|
||||
4. The rehearsal, `rehearse SHA BUSINESS --backup last --port 7339`
|
||||
(`--rehearse-port`). Exit 0 goes on. Exit 3 means no backup in the log
|
||||
and goes on only with `--no-rehearsal`, logging `rehearsal-skipped`;
|
||||
@@ -983,6 +1011,22 @@ user's clone `<home>/src/mosaic-stack`:
|
||||
still names the old one, and root moves the new file (the result line
|
||||
names it) into place by hand.
|
||||
|
||||
`--next-args` changes the serve flags, as the SSO migration does
|
||||
(`docs/guides/host-run-sheet.md` section 7). The run reads
|
||||
`/etc/mosaic-stack/dev.args.next` as it reads the args file
|
||||
(`args-unreadable`). It must equal the args file before `--` but for
|
||||
`--copy`'s value (`next-args-differ`) and differ from it after `--`
|
||||
(`next-args-same`). The target's check is `case2-unit.sh --check-new`, a
|
||||
first install's checks from the next args, without the installed units'.
|
||||
After A3, root moves the installed unit files into a new 0700 directory
|
||||
`/etc/mosaic-stack/dev.replaced.XXXXXX` (step `units-aside`), so A4 renders
|
||||
as a first install. A failure from there moves them back unchanged
|
||||
(`units-back`), runs `daemon-reload` and starts the old units with no render.
|
||||
After A5 the old args file is copied into the replaced directory, the next
|
||||
args file is removed (step `next-args`), and the new args file, made from
|
||||
the next args, moves into place. The result line names the replaced
|
||||
directory. Nothing removes it.
|
||||
|
||||
A failure in A2 or a stage that exits 3 starts the old units again and gates
|
||||
them (A2f). A later failure stops both units, runs `revert`, renders the old
|
||||
copy and runs `confirm --rollback OLD --cause A3|A4|A5`. `rollback` is the
|
||||
|
||||
@@ -446,8 +446,36 @@ stack picks or invents a subject: the subjects come from `console.sso`, which a
|
||||
4. Provision the dedicated uid. `mosaic` must keep section 1's shape: a non-login shell, the only member
|
||||
of its primary group, and owner of its config directory and data root. An accounts instance never runs
|
||||
under a human or development seat uid, and never under the `webui-service.sh` user units.
|
||||
5. Re-pin. Put `--sso` after `--` in the args file (`factory`: `--sso --send`). For `library`, also remove
|
||||
the proxy-preview trio and the proof pair. Then run `case2-unit.sh` or `stack-upgrade.sh upgrade`.
|
||||
5. Re-pin with `--next-args`. Leave `/etc/mosaic-stack/dev.args` as it is. Write the new flags to
|
||||
`/etc/mosaic-stack/dev.args.next`, root's, mode 0640: the same lines before `--`, and after `--` the
|
||||
serve flags with `--sso` added (`factory`: `--sso --send`). For `library`, also remove the
|
||||
proxy-preview trio (`--public-host`, `--auth-header`, `--allow-user`) and the proof pair
|
||||
(`--proof-header`, `--proof-file`). Then run
|
||||
`stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA --next-args`.
|
||||
|
||||
A plain re-pin refuses `render-refused` here, because an installed unit must equal the render but for
|
||||
the copy id. With `--next-args`, before anything stops, the driver checks the installed copy against
|
||||
`dev.args` and the target as a first install from `dev.args.next`. It refuses `next-args-differ` when
|
||||
the files differ before `--` (other than `--copy`), and `next-args-same` when they do not differ after
|
||||
it. After the stage it moves the installed unit files into a new 0700 directory
|
||||
`/etc/mosaic-stack/dev.replaced.XXXXXX` and renders the target as a first install. A failure from
|
||||
there moves them back unchanged and starts the old copy again. After `confirm` passes,
|
||||
`dev.args.next` becomes `dev.args`, and the replaced directory keeps the old unit files and the old
|
||||
`dev.args`. The result line names the directory. Nothing removes it; root removes it by hand once no
|
||||
rollback needs it.
|
||||
|
||||
To roll back past the migration, put the replaced args back as the next args and roll back with
|
||||
`--next-args`:
|
||||
|
||||
```sh
|
||||
install -m 0640 -o root -g root /etc/mosaic-stack/dev.replaced.XXXXXX/dev.args /etc/mosaic-stack/dev.args.next
|
||||
stack-upgrade.sh rollback --instance dev --to <old copy id> --next-args
|
||||
```
|
||||
|
||||
For `library` this is the only rollback to a copy older than M3: `case2-unit.sh` refuses `--sso` for
|
||||
an unstamped library copy (#416), so the old copy comes back with its old flags, proxy-preview
|
||||
admission included, as before the migration. A `factory` rollback may instead keep `--sso --send`
|
||||
with a plain `rollback`.
|
||||
|
||||
If a step is missing, `case2-unit.sh` refuses before it writes a unit and prints the missing steps,
|
||||
numbered in this order, for example:
|
||||
@@ -461,11 +489,11 @@ case2-unit: refusing the units for <sha>: a stamped copy's Console starts only w
|
||||
```
|
||||
|
||||
Without `console.sso`, the render does not check `accounts.json`, because its subjects name providers SSO
|
||||
has not configured yet. The next render after step 2 checks it. Rollback keeps `--sso`, accounts.json and
|
||||
`.deployed` in place (`docs/plans/2026-11-10_accounts-design.md`, section 7).
|
||||
has not configured yet. The next render after step 2 checks it. A rollback never removes accounts.json,
|
||||
its receipts or `.deployed` (`docs/plans/2026-11-10_accounts-design.md`, section 7).
|
||||
|
||||
Precheck line: none. `stack-upgrade.sh` refuses `args-unreadable` or `console-unit-legacy` on any of these
|
||||
rules, before it stops a unit.
|
||||
Precheck line: none. `stack-upgrade.sh` refuses `args-unreadable`, `console-unit-legacy`, `render-refused`,
|
||||
`next-args-differ` or `next-args-same` on any of these rules, before it stops a unit.
|
||||
|
||||
## 8. Enrollment, once per host
|
||||
|
||||
|
||||
@@ -1298,7 +1298,8 @@ business file.
|
||||
|
||||
### Rollback
|
||||
|
||||
Rollback keeps OIDC. Nothing in it removes `--sso` or turns sign-in off.
|
||||
Rollback keeps OIDC. Nothing in it removes `--sso` or turns sign-in off,
|
||||
except the library rollback past the migration below.
|
||||
|
||||
- The supported rollback is a pinned code and config rollback: install the
|
||||
earlier stack copy with the same unit flags, including `--sso`, through
|
||||
@@ -1306,6 +1307,13 @@ Rollback keeps OIDC. Nothing in it removes `--sso` or turns sign-in off.
|
||||
dedicated system units. The accounts instance must not use the
|
||||
`webui-service.sh` user-unit path. Leave accounts.json, its receipts and `.deployed` in
|
||||
place. BASE code ignores them, and the next roll-forward needs them.
|
||||
- A library host rolled back to a copy older than M3 is the exception.
|
||||
`case2-unit.sh` refuses `--sso` for an unstamped library copy (#416), so
|
||||
that rollback renders the old copy from the args file the migration
|
||||
replaced, through `stack-upgrade.sh rollback ... --next-args`
|
||||
(`docs/guides/host-run-sheet.md` section 7). The old copy comes back with
|
||||
its old flags, proxy-preview admission included, as before the migration.
|
||||
accounts.json and `.deployed` stay in place.
|
||||
- Before the first accounts org exists, that rollback restores today's
|
||||
behavior completely. `console.sso` owners sign in as before, and FED R6
|
||||
users keep their root grants. Account-only users and pending invites stop
|
||||
|
||||
@@ -364,7 +364,7 @@ test("the unit takes --org or --business on either side of --, renders --busines
|
||||
test("the unit refuses a bad selector in either scope, and an outer and serve pair that differ, before writing a unit", (t) => {
|
||||
const h = hub(t);
|
||||
const serve = ["--port", "443", "--business", "acme"];
|
||||
const usage = "usage: case2-unit.sh --user NAME --home DIR --business ID --copy ID --node PATH [--profile factory|library] [--proxy] [--pm] [--claude PATH] -- SERVE FLAGS\n";
|
||||
const usage = "usage: case2-unit.sh [--check|--check-new] --user NAME --home DIR --business ID --copy ID --node PATH [--profile factory|library] [--proxy] [--pm] [--claude PATH] -- SERVE FLAGS\n";
|
||||
const head = ["--user", "mosaic", "--home", h.home, "--copy", COPY_A, "--node", USR_NODE, "--profile", "factory"];
|
||||
for (const [label, argv, code, err] of [
|
||||
["outer unequal, old first", [...head, "--business", "acme", "--org", "beta", "--", ...serve], 4, "case2-unit: --business acme and --org beta conflict\n"],
|
||||
@@ -886,6 +886,65 @@ test("M3 a stamped copy's query refuses on noise or a failed query, before any w
|
||||
}
|
||||
});
|
||||
|
||||
// #1873 M3: stack-upgrade.sh checks each render before it stops anything. --check runs a render's
|
||||
// checks, the installed units' included, and writes nothing; --check-new skips only the installed
|
||||
// units' checks, for --next-args, which moves them aside before its render.
|
||||
const checkArgv = (h, { mode = "--check", profile = "library", copy = COPY_A, flags } = {}) =>
|
||||
[mode, "--user", "mosaic", "--home", h.home, "--business", "acme", "--copy", copy, "--node", USR_NODE, "--profile", profile, "--",
|
||||
...(flags ?? (profile === "factory" ? factoryFlags() : libraryFlags(h)))];
|
||||
const unitsNow = (h) => [busFile(h), webuiFile(h)].map((f) => (existsSync(f) ? read(f) : null));
|
||||
const logNow = (h) => (existsSync(h.log) ? read(h.log) : "");
|
||||
|
||||
test("#1873 --check passes on a matching install and on a first install, refuses an edited unit, and never writes or calls systemctl", (t) => {
|
||||
const h = hub(t);
|
||||
const empty = unit(h, { argv: checkArgv(h) });
|
||||
assert.equal(empty.code, 0, empty.out + empty.err);
|
||||
assert.equal(empty.out, `case2-unit: check passed: the units for copy ${COPY_A} render, and the installed ones match them; nothing was written\n`);
|
||||
noUnits(h);
|
||||
assert.equal(unit(h).code, 0);
|
||||
const units = unitsNow(h), log = logNow(h);
|
||||
for (const copy of [COPY_A, COPY_B]) {
|
||||
const r = unit(h, { argv: checkArgv(h, { copy }) });
|
||||
assert.equal(r.code, 0, r.out + r.err);
|
||||
assert.deepEqual(unitsNow(h), units, copy);
|
||||
assert.equal(logNow(h), log, `${copy}: no systemctl call`);
|
||||
}
|
||||
// The installed-unit match stays: an edit, or other serve flags, refuses as the render would.
|
||||
writeFileSync(busFile(h), units[0].replace(/^Description=.*$/m, "Description=edited"));
|
||||
refuses(unit(h, { argv: checkArgv(h, { copy: COPY_B }) }), /differs from the render in more than the copy id/);
|
||||
writeFileSync(busFile(h), units[0]);
|
||||
const flags = libraryFlags(h).map((f) => (f === "melanie" ? "someone" : f));
|
||||
refuses(unit(h, { argv: checkArgv(h, { copy: COPY_B, flags }) }), /installed .*: ExecStart= must be exactly once/);
|
||||
assert.deepEqual(unitsNow(h), units);
|
||||
assert.equal(logNow(h), log);
|
||||
});
|
||||
|
||||
for (const profile of ["library", "factory"]) {
|
||||
test(`#1873 an installed pre-M3 ${profile} unit: --check and the render refuse a stamped copy's SSO flags, and --check-new passes with the prerequisites, writing nothing`, (t) => {
|
||||
const h = hub(t);
|
||||
assert.equal(unit(h, { profile }).code, 0);
|
||||
stamp(h, COPY_B);
|
||||
stampSso(h);
|
||||
stampAccounts(h);
|
||||
stampDeployed(h);
|
||||
const flags = profile === "factory" ? STAMP_FACTORY : STAMP_LIBRARY;
|
||||
const units = unitsNow(h), log = logNow(h);
|
||||
refuses(unit(h, { argv: checkArgv(h, { profile, copy: COPY_B, flags }) }), /installed .*: ExecStart= must be exactly once/);
|
||||
refuses(unit(h, { profile, copy: COPY_B, flags }), /installed .*: ExecStart= must be exactly once/);
|
||||
const r = unit(h, { argv: checkArgv(h, { mode: "--check-new", profile, copy: COPY_B, flags }) });
|
||||
assert.equal(r.code, 0, r.out + r.err);
|
||||
assert.equal(r.out, `case2-unit: check passed: the units for copy ${COPY_B} render as a first install; the installed ones were not compared; nothing was written\n`);
|
||||
// --check-new still runs the stamped copy's checks: the old flags, or a missing prerequisite, refuse.
|
||||
refuses(unit(h, { argv: checkArgv(h, { mode: "--check-new", profile, copy: COPY_B }) }), /a stamped copy's Console starts only with SSO accounts|is a stamped copy, and proxy-preview admission/);
|
||||
rmSync(accountsFile(h));
|
||||
refuses(unit(h, { argv: checkArgv(h, { mode: "--check-new", profile, copy: COPY_B, flags }) }), /run `mosaic accounts init` as mosaic, a human verb/);
|
||||
assert.deepEqual(unitsNow(h), units);
|
||||
assert.equal(logNow(h), log, "no systemctl call");
|
||||
assert.equal(existsSync(accountsFile(h)), false, "the check never writes accounts.json");
|
||||
assert.equal(statSync(deployedFile(h)).mode & 0o777, 0o444);
|
||||
});
|
||||
}
|
||||
|
||||
test("row253 factory refuses retired TCP and a value after the socket marker", (t) => {
|
||||
const h = hub(t);
|
||||
refuses(unit(h, { profile: "factory", flags: ['--port','443','--business','acme','--tester-port','7333'] }), /case2-unit.sh retires tester TCP; use --tester-socket/);
|
||||
|
||||
@@ -1578,15 +1578,37 @@ function rootUp(e,copy,flags){
|
||||
.replace(/^ExecStart=.*$/m,`ExecStart=${node} ${dir}/packages/webui/src/cli.mjs serve --checkout ${repo} ${flags.join(' ')}`));
|
||||
execFileSync(join(SBIN,'systemctl'),['--user','start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:'0'}});sysctl(e,'start',e.con);
|
||||
}
|
||||
// One system-path run: OP (activate TO, or rollback --to TO) staged, root's units for TO with FLAGS, confirm.
|
||||
// One system-path run: OP (activate TO, or rollback --to TO) planned and staged from the pointer's
|
||||
// copy, root's units for TO with FLAGS, confirm. The plan, the staged set and both log lines are this
|
||||
// script's output; the unit flags are root's render (case2-unit.test.mjs pins that render), so the
|
||||
// pointer pinning them is confirm's read of root's unit, not a conversion of this script's.
|
||||
async function viaSystem(e,op,to,flags,...more){
|
||||
const x=X(e);rootDown(e);
|
||||
let r=await prun(x,...(op==='activate'?['activate',to,'acme']:['rollback','acme','--to',to]),...more);assert.equal(r.code,0,r.out+r.err);
|
||||
const act=logLines(e).at(-1).activation;
|
||||
const x=X(e),from=pointer(e).copy,p0=readFileSync(e.pointer,'utf8'),w0=sha(join(e.bin,e.wrapper));
|
||||
const args=op==='activate'?['activate',to,'acme']:['rollback','acme','--to',to],inst=e.flags.length?`${e.flags.join(' ')} `:'';
|
||||
rootDown(e);
|
||||
const con0=sha(join(e.units,e.con));
|
||||
const out=await planOnly(x,0,...args,...more),plan=out.join('\n');
|
||||
assert.deepEqual([out.find((l)=>l.startsWith('plan: ')),out.find((l)=>l.startsWith('from: ')),out.find((l)=>l.startsWith('to: '))],[`plan: ${op} for acme`,`from: ${from}`,`to: ${to}`],plan);
|
||||
assert.ok(out.includes(`units: external; root installs the units for ${to} and starts them, then confirm gates them`),plan);
|
||||
assert.ok(out.includes(`CLI: scripts/stack-copy.sh wrapper ${to} ${inst}--copies ${COPIES} --bin ${e.bin}`),plan);
|
||||
// Root renders the units, so the plan names no unit, Console flag or refusal of this script's.
|
||||
assert.deepEqual(out.filter((l)=>/^(bus|pm|proof|send|sso|Console):|^(drop|refuses) /.test(l)),[],plan);
|
||||
let r=await prun(x,...args,...more);assert.equal(r.code,0,r.out+r.err);
|
||||
const st=logLines(e).at(-1),act=st.activation,rs=setOf(e,act);
|
||||
assert.deepEqual([st.op,st.from,st.to,st.result,st.units],[op,from,to,'started','external']);
|
||||
assert.equal(r.out.split('\n').at(-2),`stack-release: ${op} staged ${to} for acme as ${act}; root installs and starts the units, then runs confirm ${to} acme --units external`);
|
||||
// Staged, not live: the pointer stays; the set holds the pointer, the wrapper and the Console unit root stopped.
|
||||
assert.equal(readFileSync(e.pointer,'utf8'),p0);
|
||||
assert.equal(readFileSync(join(rs,'AWAITING-UNITS'),'utf8'),`${to}\n`);
|
||||
assert.equal(readFileSync(join(rs,'pointer'),'utf8'),p0);assert.equal(sha(join(rs,'wrapper')),w0);
|
||||
assert.ok(readFileSync(join(rs,'UNITS.sha256'),'utf8').includes(`${con0} ${join(e.units,e.con)}\n`));
|
||||
assert.ok(readFileSync(join(e.bin,e.wrapper),'utf8').includes(join(COPIES,to)),'the wrapper names the target copy');
|
||||
rootUp(e,to,flags);
|
||||
r=await prun(x,'confirm',to,'acme');assert.equal(r.code,0,r.out+r.err);
|
||||
assert.equal(r.out.split('\n').at(-2),`stack-release: ${op} ok: ${to} is live for acme`);
|
||||
const l=logLines(e).at(-1);assert.deepEqual([l.op,l.to,l.result,l.units,l.activation],[op,to,'ok','external',act]);
|
||||
const l=logLines(e).at(-1);assert.deepEqual([l.op,l.from,l.to,l.result,l.units,l.activation],[op,from,to,'ok','external',act]);
|
||||
assert.ok(!existsSync(join(rs,'AWAITING-UNITS')));
|
||||
assert.deepEqual([pointer(e).copy,pointer(e).console],[to,flags]);
|
||||
return act;
|
||||
}
|
||||
|
||||
@@ -2102,6 +2124,7 @@ function rHost(){
|
||||
}
|
||||
if(a[0]!=='bus'||a[1]!=='start'||!a.slice(3).every((f)=>flags.includes(f))){console.error(`fixture host: ${a.join(' ')}`);process.exit(4)}
|
||||
fs.appendFileSync(path.join(home,'host-args.log'),`${a.join(' ')}\n`);
|
||||
fs.appendFileSync(path.join(home,'host-ca.log'),`${process.env.NODE_EXTRA_CA_CERTS??'none'}\n`);
|
||||
if(fault(`host-fails-${id}`)){console.error('fixture host: refused');process.exit(1)}
|
||||
const b=a[2],off=a.includes('--no-launch'),lock=path.join(data,'bus','writer.lock'),at=new Date().toISOString();
|
||||
fs.writeFileSync(lock,String(process.pid),{flag:'wx'});
|
||||
@@ -2122,14 +2145,35 @@ function rHost(){
|
||||
}
|
||||
|
||||
// The Console: serve on the given loopback port; /healthz answers 200 unless healthz-<copy> is set.
|
||||
function rConsole(){
|
||||
// M3 (#1873): under --sso it runs its copy's real createSso on the scratch console.sso, with the
|
||||
// scratch slot placeholders, as the real Console does, and /healthz carries sso.health(). Faults:
|
||||
// console-hold waits for $HOME/release before it starts; console-no-sso reports every provider
|
||||
// ready without asking any; console-asks-token asks provider a's token endpoint once.
|
||||
async function rConsole(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),http=process.getBuiltinModule('node:http');
|
||||
const id=path.basename(path.resolve(import.meta.dirname,'..','..','..')),home=process.env.HOME,a=process.argv.slice(2);
|
||||
const fault=(n)=>fs.existsSync(path.join(home,'faults',n));
|
||||
const c=JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8'));
|
||||
fs.appendFileSync(path.join(home,'console-args.log'),`${a.join(' ')}\n`);
|
||||
fs.appendFileSync(path.join(home,'console-context.jsonl'),JSON.stringify({argv:a,dataRoot:JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8')).dataRoot})+'\n');
|
||||
fs.appendFileSync(path.join(home,'console-context.jsonl'),JSON.stringify({argv:a,dataRoot:c.dataRoot,ca:process.env.NODE_EXTRA_CA_CERTS??null})+'\n');
|
||||
if(a[0]!=='serve'){console.error('fixture Console: refused');process.exit(1)}
|
||||
const s=http.createServer((q,r)=>{r.writeHead(q.url==='/healthz'&&!fault(`healthz-${id}`)?200:500);r.end()});
|
||||
while(fault('console-hold')&&!fs.existsSync(path.join(home,'release')))await new Promise((r)=>setTimeout(r,50));
|
||||
let health=null;
|
||||
if(a.includes('--sso')){
|
||||
const ps=c.console.sso.providers;
|
||||
if(fault('console-no-sso'))health=()=>Object.fromEntries(ps.map((p)=>[p.id,'ready']));
|
||||
else{
|
||||
const {createSso}=await import(path.join(import.meta.dirname,'sso.mjs'));
|
||||
const secrets=new Map(ps.map((p)=>[p.id,fs.readFileSync(path.join(path.dirname(process.env.MOSAIC_CONFIG),'secrets','mosaic-stack',p.clientSecret.slot),'utf8')]));
|
||||
let sso;try{sso=await createSso(c.console.sso,{secrets,log:()=>{}})}catch(err){console.error(`fixture Console: ${err.message}`);process.exit(2)}
|
||||
health=()=>sso.health();
|
||||
}
|
||||
if(fault('console-asks-token'))await fetch(`${ps[0].issuer}token`).catch(()=>{});
|
||||
}
|
||||
const s=http.createServer((q,r)=>{
|
||||
const ok=q.url==='/healthz'&&!fault(`healthz-${id}`);r.writeHead(ok?200:500,{'content-type':'application/json'});
|
||||
r.end(ok&&health?JSON.stringify({ok:true,sso:health()}):'');
|
||||
});
|
||||
s.listen(Number(a[a.indexOf('--port')+1]),'127.0.0.1',()=>console.log(`Mosaic Console: http://127.0.0.1:${s.address().port}/\nBoard: fixture`));
|
||||
process.on('SIGTERM',()=>process.exit(0));
|
||||
}
|
||||
@@ -2173,15 +2217,23 @@ for(const [id,v,froms] of [[B,'3g',['3f']],[C,'3h',['3f','3g']],[A,'3g',['3f']],
|
||||
execFileSync('chmod',['-R','a-w',d]);
|
||||
}
|
||||
// M3 (#1873): SCOPIES holds B as a stamped copy: RCOPIES's B with a read-only .stack-build.json
|
||||
// naming its commit, the Console's sso-config.mjs the config loader needs for a console key, and
|
||||
// the deployment.mjs whose writeDeployment the rehearsal runs.
|
||||
// naming its commit, the Console's sso-config.mjs the config loader needs for a console key, its
|
||||
// sso.mjs the stand-in Console runs, and the deployment.mjs whose writeDeployment the rehearsal runs.
|
||||
const SCOPIES=join(base,'scopies');mkdirSync(SCOPIES);
|
||||
{
|
||||
const d=join(SCOPIES,B);cpSync(join(RCOPIES,B),d,{recursive:true});execFileSync('chmod',['-R','u+w',d]);
|
||||
for(const p of ['packages/webui/src/sso-config.mjs','packages/cli/src/deployment.mjs'])cpSync(join(repo,p),join(d,p));
|
||||
for(const p of ['packages/webui/src/sso-config.mjs','packages/webui/src/sso.mjs','packages/cli/src/deployment.mjs'])cpSync(join(repo,p),join(d,p));
|
||||
// sso.mjs imports oauth4webapi.
|
||||
symlinkSync(join(ROOT,'node_modules'),join(d,'node_modules'));
|
||||
writeFileSync(join(d,'.stack-build.json'),JSON.stringify({commit:B}),{mode:0o444});
|
||||
execFileSync('chmod',['-R','a-w',d]);
|
||||
}
|
||||
// UCOPIES holds B unstamped but able to load a console key: SCOPIES's B without .stack-build.json.
|
||||
const UCOPIES=join(base,'ucopies');mkdirSync(UCOPIES);
|
||||
{
|
||||
const d=join(UCOPIES,B);cpSync(join(SCOPIES,B),d,{recursive:true,verbatimSymlinks:true});execFileSync('chmod',['-R','u+w',d]);
|
||||
rmSync(join(d,'.stack-build.json'));execFileSync('chmod',['-R','a-w',d]);
|
||||
}
|
||||
|
||||
const TESTER='7499';
|
||||
// Vikunja #782: this leases a port, closes it and hands the number to `stack-release.sh rehearse`, which
|
||||
@@ -2194,12 +2246,13 @@ const rfault=(e,name)=>{mkdirSync(rh(e,'faults'),{recursive:true});writeFileSync
|
||||
|
||||
// A root for one rehearsal: a stopped live store at `store`, a pointer by hand on `copy` with the
|
||||
// Console on e.port and the tester listener on TESTER, and a backup as backup_store writes one.
|
||||
// A rehearsal's leftovers, found by their scratch MOSAIC_CONFIG, are killed after the test.
|
||||
// A rehearsal's leftovers, found by their scratch MOSAIC_CONFIG or a rehearsal path in their command
|
||||
// line (the rehearsal provider), are killed after the test.
|
||||
async function rmk(t,{copy=B,store='3g'}={}){
|
||||
const e=await mk(t);
|
||||
t.after(()=>{
|
||||
for(const pid of readdirSync('/proc').filter((n)=>/^\d+$/.test(n)))
|
||||
try{if(readFileSync(`/proc/${pid}/environ`,'utf8').includes(`MOSAIC_CONFIG=${e.data}/rehearsals/`))process.kill(Number(pid),'SIGKILL')}catch{}
|
||||
try{if(readFileSync(`/proc/${pid}/environ`,'utf8').includes(`MOSAIC_CONFIG=${e.data}/rehearsals/`)||readFileSync(`/proc/${pid}/cmdline`,'utf8').includes(`${e.data}/rehearsals/`))process.kill(Number(pid),'SIGKILL')}catch{}
|
||||
});
|
||||
const db=new DatabaseSync(e.store);
|
||||
db.exec('PRAGMA journal_mode=WAL');
|
||||
@@ -2479,9 +2532,10 @@ test('R15 a rehearsal stopped by TERM, INT or HUP fails signal and still stops t
|
||||
|
||||
// --- M3 (#1873): rehearsing a stamped copy (the brief's item 1) ---
|
||||
// A stamped copy serves only with --sso, console.sso, accounts.json and a .deployed naming its
|
||||
// instance. The rehearsal gives its scratch Console all four: the live console.sso with origin and
|
||||
// issuers on rehearsal.invalid, a placeholder per slot, a copy of the live accounts.json, and a
|
||||
// .deployed from the copy's writeDeployment. The live slot has a read watch: it is never opened.
|
||||
// instance. The rehearsal gives its scratch Console all four: the live console.sso with its origin
|
||||
// on rehearsal.invalid and each issuer at the rehearsal's own provider, a placeholder per slot, a
|
||||
// copy of the live accounts.json, and a .deployed from the copy's writeDeployment. The live slot
|
||||
// has a read watch: it is never opened.
|
||||
const ACCOUNTS={accountsVersion:1,revision:1,users:[{id:'u-000000000001',subjects:[{provider:'a',sub:'subject-1'}],name:'Jason',instanceRole:'operator',state:'active',authGen:1,created:'2026-11-10T00:00:00Z'}],memberships:[],invites:[]};
|
||||
async function smk(t,{sso=true,accounts=true}={}){
|
||||
const e=await rmk(t);e.copies=SCOPIES;
|
||||
@@ -2490,7 +2544,10 @@ async function smk(t,{sso=true,accounts=true}={}){
|
||||
if(accounts)writeFileSync(e.accounts,JSON.stringify(ACCOUNTS),{mode:0o600});
|
||||
return e;
|
||||
}
|
||||
const SPRE=[...PRE.slice(0,4),'PASS deployment',...PRE.slice(4)];
|
||||
const SPRE=[...PRE.slice(0,3),'PASS provider-start',PRE[3],'PASS deployment',...PRE.slice(4)];
|
||||
const SPOST=['PASS console-stopped','PASS host-stopped','PASS provider-stopped',...POST.slice(2)];
|
||||
// The rehearsal provider's origin and pid, from its provider-start line.
|
||||
const provOf=(out)=>{const m=/^PASS provider-start: pid (\d+) at (https:\/\/127\.0\.0\.1:\d+): /m.exec(out);return m?{pid:Number(m[1]),origin:m[2]}:null};
|
||||
// A real stamped copy of this checkout, as deployment-boundary.test.mjs builds one.
|
||||
function stampedCopy(t){
|
||||
const sha40='a'.repeat(40),root=mkdtempSync(join(tmpdir(),'rs-'));t.after(()=>rmSync(root,{recursive:true,force:true}));
|
||||
@@ -2503,8 +2560,8 @@ function stampedCopy(t){
|
||||
}
|
||||
// Starts the copy's real Console on port 0 against the rehearsal's scratch config. Resolves with
|
||||
// {code,out,err} on exit, or with the bound port once it prints its address line.
|
||||
async function realServe(t,copy,e,dir){
|
||||
const ch=spawn(process.execPath,[join(copy,'packages/webui/src/cli.mjs'),'serve','--port','0','--sso','--board','http://127.0.0.1:9'],{env:{HOME:e.env.HOME,PATH:e.env.PATH,TMPDIR:e.env.TMPDIR,MOSAIC_CONFIG:join(dir,'config','config.json')},stdio:['ignore','pipe','pipe']});
|
||||
async function realServe(t,copy,e,dir,more={}){
|
||||
const ch=spawn(process.execPath,[join(copy,'packages/webui/src/cli.mjs'),'serve','--port','0','--sso','--board','http://127.0.0.1:9'],{env:{HOME:e.env.HOME,PATH:e.env.PATH,TMPDIR:e.env.TMPDIR,MOSAIC_CONFIG:join(dir,'config','config.json'),...more},stdio:['ignore','pipe','pipe']});
|
||||
t.after(async()=>{if(ch.exitCode===null&&ch.signalCode===null){const done=once(ch,'close');ch.kill('SIGKILL');await done}});
|
||||
let out='',err='';ch.stderr.on('data',(d)=>{err+=d});
|
||||
return new Promise((res)=>{
|
||||
@@ -2514,21 +2571,32 @@ async function realServe(t,copy,e,dir){
|
||||
});
|
||||
}
|
||||
|
||||
test('M3 R16 a stamped copy rehearses with --sso against rehearsal.invalid, a scratch accounts copy and its own .deployed; the live files stay as they were',async(t)=>{
|
||||
test('M3 R16 a stamped copy rehearses with --sso against its own provider, a scratch accounts copy and its own .deployed; the live files stay as they were',async(t)=>{
|
||||
const st=(f)=>{const x=statSync(f);return [x.ino,x.size,x.mtimeMs,x.mode]};
|
||||
const e=await smk(t),before=untouched(e),accounts=sha(e.accounts),slot=st(slotPath(e)),port=await freePort();
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',port,'--timeout','10');
|
||||
assert.equal(r.code,0,r.out+r.err);
|
||||
assert.deepEqual(r.names,[...SPRE,'PASS migration','PASS gate','PASS no-session',...POST]);
|
||||
const R=r.dir;
|
||||
assert.deepEqual(r.names,[...SPRE,'PASS migration','PASS gate','PASS no-session','PASS provider-discovery',...SPOST]);
|
||||
const R=r.dir,prov=provOf(r.out);assert.ok(prov,r.out);
|
||||
assert.ok(r.out.includes(`PASS provider-start: pid ${prov.pid} at ${prov.origin}: discovery and a JWKS from keys made for this run, no code and no token; its certificate ${join(R,'provider','cert.pem')} is the trust the scratch Console alone gets\n`),r.out);
|
||||
assert.ok(r.out.includes(`PASS deployment: ${B} is stamped: ${join(R,'data','.deployed')} from its writeDeployment, the live accounts.json copied to ${join(R,'config')}, the Console starts with --sso\n`),r.out);
|
||||
assert.ok(r.out.includes('PASS sso-isolated: 1 provider(s), each at rehearsal.invalid with a placeholder secret\n'),r.out);
|
||||
// The Console started with --sso, and only on the scratch config.
|
||||
assert.ok(r.out.includes(`PASS sso-isolated: 1 provider(s), each at the rehearsal provider ${prov.origin} with a placeholder secret\n`),r.out);
|
||||
assert.ok(r.out.includes(`PASS gate: host-up, bus-status, console-up, healthz, sso-ready and copy-match on ${B}\n`),r.out);
|
||||
assert.ok(r.out.includes(`PASS provider-discovery: the Console fetched each provider's discovery and JWKS from ${prov.origin} and asked for no code or token\n`),r.out);
|
||||
// The Console started with --sso, only on the scratch config, and alone trusts the provider's
|
||||
// certificate; the host gets no added trust.
|
||||
assert.deepEqual(lines(rh(e,'console-args.log')),[`serve --port ${port} --business acme --sso`]);
|
||||
assert.deepEqual(lines(rh(e,'console-context.jsonl')).map((l)=>JSON.parse(l).ca),[join(R,'provider','cert.pem')]);
|
||||
assert.deepEqual(lines(rh(e,'host-ca.log')),['none']);
|
||||
// The provider served discovery and the JWKS, got no other request, kept no TLS key and is gone.
|
||||
assert.deepEqual(lines(join(R,'provider','requests.log')).sort(),['GET /a/.well-known/openid-configuration 200','GET /a/jwks 200']);
|
||||
assert.deepEqual(readdirSync(join(R,'provider')).sort(),['cert.pem','origin','provider.log','requests.log']);
|
||||
assert.equal(statSync(join(R,'provider')).mode&0o777,0o700);
|
||||
assert.equal(existsSync(`/proc/${prov.pid}`),false,`provider pid ${prov.pid} is still there`);
|
||||
// The scratch config: the live one with dataRoot, origin and issuer moved; provider id, client id,
|
||||
// algorithms, slot name and subjects as they were. No subject is added.
|
||||
const live=JSON.parse(readFileSync(e.cfg,'utf8')),sc=JSON.parse(readFileSync(join(R,'config','config.json'),'utf8'));
|
||||
const want=structuredClone(live);want.dataRoot=join(R,'data');want.console.sso.publicOrigin='https://rehearsal.invalid';want.console.sso.providers[0].issuer='https://rehearsal.invalid/a';
|
||||
const want=structuredClone(live);want.dataRoot=join(R,'data');want.console.sso.publicOrigin='https://rehearsal.invalid';want.console.sso.providers[0].issuer=`${prov.origin}/a/`;
|
||||
assert.deepEqual(sc,want);assert.deepEqual(sc.console.sso.providers[0].subjects,SSO_KEY.sso.providers[0].subjects);
|
||||
// The slot is a scratch placeholder, 0600 in a 0700 directory.
|
||||
const ph=join(R,'config','secrets','mosaic-stack',SLOT);
|
||||
@@ -2545,16 +2613,30 @@ test('M3 R16 a stamped copy rehearses with --sso against rehearsal.invalid, a sc
|
||||
await noLeak(e);
|
||||
});
|
||||
|
||||
test('M3 R17 the real stamped Console accepts the rehearsal\'s scratch prerequisites and starts degraded; without .deployed or accounts.json it refuses',async(t)=>{
|
||||
const e=await smk(t);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','10');
|
||||
assert.equal(r.code,0,r.out+r.err);
|
||||
test('M3 R17 the real stamped Console finds the rehearsal provider ready with the scratch prerequisites, starts degraded once it is gone, and refuses without .deployed or accounts.json',async(t)=>{
|
||||
const e=await smk(t);rfault(e,'console-hold');
|
||||
const copy=stampedCopy(t);
|
||||
// Every startup check passes: stamp, --sso, .deployed, accounts, console.sso and its slot. The
|
||||
// provider at rehearsal.invalid is unreachable, so the Console serves degraded and /healthz answers.
|
||||
// While the stand-in Console holds, the rehearsal provider is up: the real Console of a stamped
|
||||
// copy passes every startup check (stamp, --sso, .deployed, accounts, console.sso and its slot)
|
||||
// and, trusting only the provider's certificate, finds provider a ready.
|
||||
let during=null;
|
||||
e.during=async()=>{
|
||||
try{
|
||||
for(let i=0;i<400&&!existsSync(rh(e,'console-args.log'));i++)await new Promise((r)=>setTimeout(r,50));
|
||||
const dir=join(e.data,'rehearsals',rDirs(e)[0]);
|
||||
const up=await realServe(t,copy,e,dir,{NODE_EXTRA_CA_CERTS:join(dir,'provider','cert.pem')});
|
||||
during={up:up.port?await (await fetch(`http://127.0.0.1:${up.port}/healthz`)).json():null,why:`${up.code} ${up.out} ${up.err}`};
|
||||
if(up.ch){const done=once(up.ch,'close');up.ch.kill('SIGTERM');await done}
|
||||
}finally{writeFileSync(rh(e,'release'),'')}
|
||||
};
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','30');
|
||||
assert.equal(r.code,0,r.out+r.err);
|
||||
assert.deepEqual(during?.up,{ok:true,sso:{a:'ready'}},during?.why);
|
||||
// After the rehearsal its provider is gone: the same Console serves degraded and /healthz says so.
|
||||
const up=await realServe(t,copy,e,r.dir);
|
||||
assert.ok(up.port,`${up.code} ${up.out} ${up.err}`);
|
||||
assert.equal((await fetch(`http://127.0.0.1:${up.port}/healthz`)).status,200);
|
||||
const h=await fetch(`http://127.0.0.1:${up.port}/healthz`);
|
||||
assert.equal(h.status,200);assert.deepEqual(await h.json(),{ok:true,sso:{a:'unreachable'}});
|
||||
const done=once(up.ch,'close');up.ch.kill('SIGTERM');await done;
|
||||
// The same start without each prerequisite refuses before it listens.
|
||||
const dep=join(r.dir,'data','.deployed'),acc=join(r.dir,'config','accounts.json'),keep=readFileSync(dep);
|
||||
@@ -2589,19 +2671,86 @@ test('M3 R18 a stamped copy refuses sso-not-prepared without a live console.sso
|
||||
});
|
||||
|
||||
test('M3 R19 sso-isolated fails before the host starts when the scratch origin, an issuer or a slot is not the rehearsal\'s',async(t)=>{
|
||||
// stamped: the issuer must be the rehearsal provider's; unstamped (no provider): on rehearsal.invalid.
|
||||
for(const [stamped,from,to,why] of [
|
||||
[true,'sso.publicOrigin="https://rehearsal.invalid";','','console.sso.publicOrigin is https://console.example.test, not https://rehearsal.invalid'],
|
||||
[true,'p.issuer=`${op||','p.issuer=p.issuer||`${op||','console.sso provider a issuer is https://op.example.test/, not ORIGIN/a/ at the rehearsal provider'],
|
||||
[false,'p.issuer=`${op||"https://rehearsal.invalid"}','p.issuer=`${op||"https://op.example.test"}','console.sso provider a issuer is https://op.example.test/a, not on rehearsal.invalid'],
|
||||
[true,'"rehearsal-placeholder-secret\\n",{mode:0o600,flag:"wx"}','"other\\n",{mode:0o600,flag:"wx"}','console.sso provider a has no scratch placeholder secret'],
|
||||
]){
|
||||
let e;if(stamped)e=await smk(t);else{e=await rmk(t);e.copies=UCOPIES;await ssoConfig(e)}
|
||||
ownScript(e,from,to);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,why+r.out+r.err);
|
||||
const prov=provOf(r.out);assert.equal(!!prov,stamped,r.out);
|
||||
assert.ok(r.out.includes(`FAIL sso-isolated: ${why.replace('ORIGIN',prov?.origin)}\n`),r.out);
|
||||
assert.deepEqual(r.names.slice(stamped?-4:-3),['FAIL sso-isolated',...(stamped?['PASS provider-stopped']:[]),'PASS backup-unchanged','PASS live-config-unchanged']);
|
||||
assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false);
|
||||
if(prov)assert.equal(existsSync(`/proc/${prov.pid}`),false);
|
||||
await noLeak(e);
|
||||
}
|
||||
});
|
||||
|
||||
test('M3 R20 the rehearsal provider\'s failure controls: no start, no trust, no discovery, a token request, a signal and a SIGKILL each fail and stop it',async(t)=>{
|
||||
const ends=['PASS backup-unchanged','PASS live-config-unchanged'];
|
||||
// The provider cannot start, leaves its TLS key, offers a non-loopback origin or never writes one:
|
||||
// provider-start fails before the scratch build, and no host or Console starts.
|
||||
for(const [from,to,why] of [
|
||||
['sso.publicOrigin="https://rehearsal.invalid";','','console.sso.publicOrigin is https://console.example.test, not https://rehearsal.invalid'],
|
||||
['p.issuer=`https://rehearsal.invalid/${encodeURIComponent(String(p.id))}`','0','console.sso provider a issuer is https://op.example.test/, not on rehearsal.invalid'],
|
||||
['"rehearsal-placeholder-secret\\n",{mode:0o600,flag:"wx"}','"other\\n",{mode:0o600,flag:"wx"}','console.sso provider a has no scratch placeholder secret'],
|
||||
['spawnSync("openssl"','spawnSync("openssl-missing-for-r20"','the rehearsal provider exited: rehearsal provider: cannot make the certificate: openssl exit ENOENT'],
|
||||
['finally{fs.rmSync(tls,{force:true})}','finally{}','the rehearsal provider left its TLS key at RDIR/provider/tls-key.pem'],
|
||||
['origin=`https://127.0.0.1:${','origin=`https://localhost:${','the rehearsal provider\'s origin https://localhost:PORT is not https on 127.0.0.1'],
|
||||
['fs.renameSync(path.join(dir,"origin.tmp"),path.join(dir,"origin"));','','the rehearsal provider wrote no origin within 3s'],
|
||||
]){
|
||||
const e=await smk(t);ownScript(e,from,to);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,why+r.out+r.err);
|
||||
assert.ok(r.out.includes(`FAIL sso-isolated: ${why}\n`),r.out);
|
||||
assert.deepEqual(r.names.slice(-3),['FAIL sso-isolated','PASS backup-unchanged','PASS live-config-unchanged']);
|
||||
assert.equal(r.code,1,why+r.out+r.err);assert.ok(r.dir,r.out+r.err);
|
||||
const line=r.out.split('\n').find((l)=>l.startsWith('FAIL provider-start: '))??'';
|
||||
const re=new RegExp(`^FAIL provider-start: ${why.replace(/[.*+?^${}()|[\]\\]/g,'\\$&').replace('RDIR',r.dir.replace(/[.*+?^${}()|[\]\\]/g,'\\$&')).replace('PORT','\\d+')}$`);
|
||||
assert.match(line,re,r.out);
|
||||
assert.deepEqual(r.names,[...PRE.slice(0,3),'FAIL provider-start',...(why.includes('exited')?[]:['PASS provider-stopped']),...ends],r.out);
|
||||
assert.equal(existsSync(join(r.dir,'config')),false);
|
||||
assert.equal(existsSync(rh(e,'host-args.log')),false);assert.equal(existsSync(rh(e,'console-args.log')),false);
|
||||
await noLeak(e);
|
||||
}
|
||||
// The Console cannot reach the provider (its discovery drops the connection, or the Console has
|
||||
// no trust for its certificate): it starts degraded and the gate fails sso-ready.
|
||||
for(const [from,to] of [
|
||||
[' send(200,{issuer,authorization_endpoint',' return q.socket.destroy();send(200,{issuer,authorization_endpoint'],
|
||||
['r_ca=(NODE_EXTRA_CA_CERTS="$R/provider/cert.pem")','r_ca=()'],
|
||||
]){
|
||||
const e=await smk(t);ownScript(e,from,to);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','5');
|
||||
assert.equal(r.code,1,from+r.out+r.err);
|
||||
assert.ok(r.out.includes('FAIL sso-ready: provider a is unreachable, not ready\n'),r.out);
|
||||
assert.deepEqual(r.names,[...SPRE,'PASS migration','FAIL sso-ready',...SPOST],r.out);
|
||||
assert.equal(existsSync(`/proc/${provOf(r.out).pid}`),false);
|
||||
}
|
||||
// A Console that reports ready without asking the provider, or asks it for a token: the gate
|
||||
// passes and provider-discovery fails.
|
||||
for(const [f,why] of [
|
||||
['console-no-sso','no GET /a/.well-known/openid-configuration 200; no GET /a/jwks 200'],
|
||||
['console-asks-token','a request for GET /a/token 403'],
|
||||
]){
|
||||
const e=await smk(t);rfault(e,f);
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','10');
|
||||
assert.equal(r.code,1,f+r.out+r.err);
|
||||
assert.ok(r.out.includes(`FAIL provider-discovery: ${why}; see ${join(r.dir,'provider','requests.log')}\n`),r.out);
|
||||
assert.deepEqual(r.names,[...SPRE,'PASS migration','PASS gate','PASS no-session','FAIL provider-discovery',...SPOST],r.out);
|
||||
}
|
||||
// A signal while the gate waits stops the Console, host and provider.
|
||||
const e=await smk(t);rfault(e,`healthz-${B}`);
|
||||
e.during=async(ch)=>{for(let i=0;i<200&&!existsSync(rh(e,'console-args.log'));i++)await new Promise((r)=>setTimeout(r,50));ch.kill('SIGTERM')};
|
||||
const r=await rehearse(e,B,'--backup',e.act,'--port',await freePort(),'--timeout','30');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.deepEqual(r.names,[...SPRE,'PASS migration','FAIL signal',...SPOST],r.out);
|
||||
assert.equal(existsSync(`/proc/${provOf(r.out).pid}`),false);
|
||||
// SIGKILL leaves no trap to run: the provider sees its parent gone and exits by itself.
|
||||
const k=await smk(t);rfault(k,`healthz-${B}`);
|
||||
k.during=async(ch)=>{for(let i=0;i<200&&!existsSync(rh(k,'console-args.log'));i++)await new Promise((r)=>setTimeout(r,50));ch.kill('SIGKILL')};
|
||||
const rk=await rehearse(k,B,'--backup',k.act,'--port',await freePort(),'--timeout','30');
|
||||
assert.equal(rk.code,null,rk.out+rk.err);
|
||||
const pk=provOf(rk.out);assert.ok(pk,rk.out);
|
||||
for(let i=0;i<60&&existsSync(`/proc/${pk.pid}`);i++)await new Promise((r)=>setTimeout(r,100));
|
||||
assert.equal(existsSync(`/proc/${pk.pid}`),false,`provider pid ${pk.pid} outlived the killed rehearsal`);
|
||||
});
|
||||
|
||||
// --- Row 233 (#1735): --backup last, the newest backup the log records (sage-1650, sage-1653) ---
|
||||
|
||||
@@ -47,12 +47,19 @@ function fakeSudo(K){
|
||||
|
||||
// case2-unit.sh: logs its argv, then installs the copy's units as the fixture's user units (its
|
||||
// WorkingDirectory under the stack's copies) and starts the optional proxy, bus, then Console,
|
||||
// as case2-unit.sh does. The business is --business's value, or --org's when that is the spelling. faults/render-fails-<copy> exits 1 first; faults/render-probe-<copy> (one-shot) first runs
|
||||
// as case2-unit.sh does. The business is --business's value, or --org's when that is the spelling. A first
|
||||
// --check or --check-new (#1873) logs its argv and exits 1 under faults/check-fails-<copy>, else 0,
|
||||
// writing nothing. faults/render-fails-<copy> exits 1 first; faults/render-probe-<copy> (one-shot) first runs
|
||||
// probe.sh, a second driver, and records its exit and stderr.
|
||||
function fakeRender(K){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
|
||||
const a=process.argv.slice(2),sd=process.env.SD_STATE,f=(n)=>path.join(sd,'faults',n);
|
||||
fs.appendFileSync(path.join(sd,'render.log'),JSON.stringify(a)+'\n');
|
||||
if(a[0]==='--check'||a[0]==='--check-new'){
|
||||
const c=a[a.indexOf('--copy')+1];
|
||||
if(fs.existsSync(f(`check-fails-${c}`))){console.error(`fixture render: ${a[0]} ${c} fails`);process.exit(1)}
|
||||
process.exit(0);
|
||||
}
|
||||
const cut=a.indexOf('--'),pre=a.slice(0,cut),serve=a.slice(cut+1),get=(k)=>pre[pre.indexOf(k)+1];
|
||||
const copy=get('--copy'),copies=path.join(get('--home'),'.local','share','mosaic-stack','copies');
|
||||
if(fs.existsSync(f(`render-probe-${copy}`))){
|
||||
@@ -147,7 +154,7 @@ const arm=(e,key,text)=>writeFileSync(join(e.sd,'arm',key),text);
|
||||
|
||||
// The step log: every line `<time> <step> rc=N ...` or the result line, root:root 0640 (here the
|
||||
// test user's), and nothing the stack's user printed. Returns the lines without the time.
|
||||
const STEP=/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ (start|fetch|ff|copy|hostcheck|units|plan|rehearse|rehearse-store|rehearsal-skipped|stop-console|stop-bus|stop-proxy|wait-down|stage|render|confirm|args|revert|render-old|confirm-rollback|gate|start-proxy|start-bus|start-console|stop-units) rc=\d+ |^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ result exit=[0-35] [a-z-]+ /;
|
||||
const STEP=/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ (start|fetch|ff|copy|hostcheck|units|plan|rehearse|rehearse-store|rehearsal-skipped|render-old-check|render-check|stop-console|stop-bus|stop-proxy|wait-down|units-aside|units-back|daemon-reload|next-args|stage|render|confirm|args|revert|render-old|confirm-rollback|gate|start-proxy|start-bus|start-console|stop-units) rc=\d+ |^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ result exit=[0-35] [a-z-]+ /;
|
||||
function slog(e){
|
||||
const l=lines(e.slog);
|
||||
for(const x of l){assert.match(x,STEP);assert.ok(!/stack-release:|fixture rehearse|PASS |FAIL |REHEARSAL/.test(x),x)}
|
||||
@@ -165,7 +172,7 @@ const live=(e)=>[units(e)[e.bus].state,units(e)[e.con].state,units(e)[e.bus].wd.
|
||||
// Nothing the run did reached the args file: the same bytes and mode, and no temporary file left.
|
||||
const argsKept=(e)=>{assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args'])};
|
||||
const argsCopy=(e)=>{const l=readFileSync(e.args,'utf8').split('\n');return l[l.indexOf('--copy')+1]};
|
||||
const PRE=['start','fetch','ff','copy','hostcheck','units','plan','rehearse','rehearse-store'];
|
||||
const PRE=['start','fetch','ff','copy','hostcheck','units','plan','render-old-check','render-check','rehearse','rehearse-store'];
|
||||
const A2=['stop-console','stop-bus','wait-down','wait-down'];
|
||||
const R=[...A2,'revert','render-old','confirm-rollback'];
|
||||
|
||||
@@ -394,7 +401,7 @@ test('row 233: sequence 7: two upgrades and a rollback --to the previous copy wi
|
||||
const rb=await drive(e,C,'rollback','--instance','dev','--to',C1,'--restore',a2,...TO);assert.equal(rb.code,0,rb.all);
|
||||
assert.equal(rb.out.split('\n').at(-2),`stack-upgrade: rollback ok: ${C1} is live for acme`);
|
||||
// A rollback fetches, copies and rehearses nothing.
|
||||
assert.deepEqual(names(e).slice(n),['start','hostcheck','units','plan',...A2,'stage','render','confirm','args','result']);
|
||||
assert.deepEqual(names(e).slice(n),['start','hostcheck','units','plan','render-old-check','render-check',...A2,'stage','render','confirm','args','result']);
|
||||
const ra=lastAct(e),l3=final(e,ra);assert.deepEqual([l3.op,l3.result,l3.from,l3.to,l3.backup],['rollback','ok',C,C1,a2]);
|
||||
assert.deepEqual([pointer(e).copy,pointer(e).schema],[C1,'3g']);assert.equal(storeVersion(e.store,true),'3g');
|
||||
assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
|
||||
@@ -632,7 +639,9 @@ test('rename r2: an upgrade from an args file that spells both selectors --org k
|
||||
const after=e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`);assert.notEqual(after,e.argsText);
|
||||
assert.equal(readFileSync(e.args,'utf8'),after);
|
||||
assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args']);
|
||||
assert.deepEqual(lines(join(e.sd,'render.log')).map((x)=>JSON.parse(x)),[after.trimEnd().split('\n')]);
|
||||
// #1873: before anything stops, --check renders the installed copy, then the target, from the same lines.
|
||||
const al=after.trimEnd().split('\n');
|
||||
assert.deepEqual(lines(join(e.sd,'render.log')).map((x)=>JSON.parse(x)),[['--check',...e.argsText.trimEnd().split('\n')],['--check',...al],al]);
|
||||
// The stack's user's steps get the business as a word, never a spelling.
|
||||
for(const k of ['activate-plan','rehearse','activate','confirm'])assert.ok(JSON.parse(lines(join(e.sd,'sudo.log')).find((x)=>JSON.parse(x).key===k)).cmd.includes('acme'),k);
|
||||
assert.ok(!lines(join(e.sd,'sudo.log')).some((x)=>JSON.parse(x).cmd.includes('--org')));
|
||||
@@ -842,3 +851,101 @@ test('row 233: usage exits 4 before any log or lock',async(t)=>{
|
||||
}
|
||||
assert.ok(!existsSync(dirname(e.slog)));assert.ok(!existsSync(e.run));
|
||||
});
|
||||
|
||||
// --- #1873 M3: render checks before anything stops, and --next-args ---
|
||||
|
||||
// The installed units' bytes, for the byte-for-byte checks across a moved-aside render.
|
||||
const unitBytes=(e)=>Object.fromEntries([e.tmpl,e.con].map((u)=>[u,readFileSync(join(e.units,u),'utf8')]));
|
||||
const renders=(e)=>lines(join(e.sd,'render.log')).map((l)=>JSON.parse(l));
|
||||
const copyOf=(a)=>a[a.indexOf('--copy')+1];
|
||||
// The next args: the args file's lines with the serve flags in another order, root's 0640.
|
||||
const nextText=(e)=>e.argsText.replace(`--\n--port\n${e.port}\n--business\nacme\n`,`--\n--business\nacme\n--port\n${e.port}\n`);
|
||||
const writeNext=(e,text=nextText(e),mode=0o640)=>{const f=join(e.etc,'dev.args.next');writeFileSync(f,text);chmodSync(f,mode);return f};
|
||||
const replacedDirs=(e)=>readdirSync(e.etc).filter((n)=>/^dev\.replaced\.[A-Za-z0-9]{6}$/.test(n)).sort();
|
||||
|
||||
test('#1873: a target whose render check fails refuses render-refused, naming --next-args, before anything stops',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,`check-fails-${C1}`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
|
||||
assert.deepEqual(names(e),['start','fetch','ff','copy','hostcheck','units','plan','render-old-check','render-check','result']);
|
||||
assert.match(result(e),new RegExp(`^result exit=3 render-refused case2-unit.sh --check refuses ${C1}'s units from .*dev\\.args; .*takes --next-args .*\\(nothing was stopped\\)$`));
|
||||
assert.deepEqual(renders(e).map((a)=>[a[0],copyOf(a)]),[['--check',B],['--check',C1]]);
|
||||
assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
|
||||
});
|
||||
|
||||
test('#1873: an installed copy whose render check fails refuses at render-old-check, before the target\'s check',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,`check-fails-${B}`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
|
||||
assert.deepEqual(names(e),['start','fetch','ff','copy','hostcheck','units','plan','render-old-check','result']);
|
||||
assert.match(result(e),new RegExp(`^result exit=3 render-refused case2-unit.sh --check refuses ${B}'s units from .*dev\\.args, so a rollback could not render them`));
|
||||
assert.deepEqual(renders(e).map((a)=>[a[0],copyOf(a)]),[['--check',B]]);
|
||||
assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
|
||||
});
|
||||
|
||||
test('#1873: --next-args moves the old units aside, renders the new flags as a first install, keeps the old units and args, and rolls back the same way',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const before=unitBytes(e),next=nextText(e);writeNext(e);
|
||||
const r=await upgrade(e,C1,'--next-args');assert.equal(r.code,0,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage','units-aside','render','confirm','next-args','args','result']);
|
||||
const [d]=replacedDirs(e);assert.ok(d,readdirSync(e.etc).join(' '));
|
||||
assert.equal(result(e),`result exit=0 ok upgrade ok: ${C1} is live for acme; the replaced units and args file are in ${join(e.etc,d)}`);
|
||||
assert.deepEqual(readdirSync(e.etc).sort(),['dev.args',d]);
|
||||
assert.equal(readFileSync(e.args,'utf8'),next.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`));assert.equal(statSync(e.args).mode&0o777,0o640);
|
||||
const rd=join(e.etc,d);assert.equal(statSync(rd).mode&0o777,0o700);
|
||||
assert.deepEqual(readdirSync(rd).sort(),['dev.args',e.tmpl,e.con].sort());
|
||||
for(const [u,text] of Object.entries(before))assert.equal(readFileSync(join(rd,u),'utf8'),text,u);
|
||||
assert.equal(readFileSync(join(rd,'dev.args'),'utf8'),e.argsText);assert.equal(statSync(join(rd,'dev.args')).mode&0o777,0o640);
|
||||
const serve=(a)=>a.slice(a.indexOf('--')+1);
|
||||
const rl=renders(e);assert.deepEqual(rl.map((a)=>[a[0],copyOf(a)]),[['--check',B],['--check-new',C1],['--user',C1]]);
|
||||
assert.deepEqual(serve(rl[0]),['--port',String(e.port),'--business','acme']);
|
||||
assert.deepEqual(serve(rl[1]),['--business','acme','--port',String(e.port)]);assert.deepEqual(serve(rl[2]),serve(rl[1]));
|
||||
assert.deepEqual(live(e),['active','active',C1,C1]);
|
||||
// The run sheet's rollback: the replaced args file as the next args, then rollback --next-args.
|
||||
copyFileSync(join(rd,'dev.args'),join(e.etc,'dev.args.next'));chmodSync(join(e.etc,'dev.args.next'),0o640);
|
||||
const after=unitBytes(e),m=slog(e).length,n=renders(e).length;
|
||||
const rb=await drive(e,C1,'rollback','--instance','dev','--to',B,'--next-args',...TO);assert.equal(rb.code,0,rb.all);
|
||||
assert.deepEqual(names(e).slice(m),['start','hostcheck','units','plan','render-old-check','render-check',...A2,'stage','units-aside','render','confirm','next-args','args','result']);
|
||||
assert.deepEqual(renders(e).slice(n).map((a)=>[a[0],copyOf(a)]),[['--check',C1],['--check-new',B],['--user',B]]);
|
||||
assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.deepEqual(live(e),['active','active',B,B]);
|
||||
const d2=replacedDirs(e).find((x)=>x!==d);assert.ok(d2);assert.deepEqual(readdirSync(e.etc).sort(),['dev.args',d,d2].sort());
|
||||
for(const [u,text] of Object.entries(after))assert.equal(readFileSync(join(e.etc,d2,u),'utf8'),text,u);
|
||||
assert.equal(readFileSync(join(e.etc,d2,'dev.args'),'utf8'),next.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`));
|
||||
});
|
||||
|
||||
test('#1873: a failed render under --next-args moves the old units back unchanged and restarts them, with no render of the old copy: exit 1',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const before=unitBytes(e),next=nextText(e);writeNext(e);
|
||||
fault(e,`render-fails-${C1}`);
|
||||
const r=await upgrade(e,C1,'--next-args');assert.equal(r.code,1,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage','units-aside','render',...A2,'revert','units-back','daemon-reload','start-bus','start-console','confirm-rollback','result']);
|
||||
assert.match(result(e),new RegExp(`^result exit=1 failed A4 failed; rolled back to ${B}`));
|
||||
assert.deepEqual(renders(e).map((a)=>[a[0],copyOf(a)]),[['--check',B],['--check-new',C1],['--user',C1]]);
|
||||
for(const [u,text] of Object.entries(before))assert.equal(readFileSync(join(e.units,u),'utf8'),text,u);
|
||||
assert.deepEqual(readdirSync(e.etc).sort(),['dev.args','dev.args.next']);
|
||||
assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.equal(readFileSync(join(e.etc,'dev.args.next'),'utf8'),next);
|
||||
assert.equal(final(e,lastAct(e)).rollback,'ok');assert.deepEqual(live(e),['active','active',B,B]);
|
||||
});
|
||||
|
||||
test('#1873: the next args must be root\'s, equal before -- but for --copy, differ after it, and render as a first install',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const tryNext=async(text,mode,code,why,at='plan')=>{
|
||||
if(text===null)rmSync(join(e.etc,'dev.args.next'),{force:true});else writeNext(e,text,mode);
|
||||
const n=existsSync(e.slog)?slog(e).length:0;
|
||||
const r=await upgrade(e,C1,'--next-args');assert.equal(r.code,3,`${why}: ${r.all}`);
|
||||
assert.match(slog(e).at(-1),new RegExp(`^result exit=3 ${code} `),why);
|
||||
const ran=names(e).slice(n);assert.ok(!ran.some((x)=>/^(stop|stage|units-aside|render$|confirm)/.test(x)),`${why}: ${ran}`);
|
||||
if(at)assert.equal(ran.at(-2),at,`${why}: ${ran}`);
|
||||
};
|
||||
await tryNext(null,0,'args-unreadable','no next file','start');
|
||||
await tryNext(nextText(e),0o660,'args-unreadable','group write','start');
|
||||
await tryNext(nextText(e).replace('--home\n','--pm\n--home\n'),0o640,'next-args-differ','a flag before --','start');
|
||||
await tryNext(nextText(e).replace('--user\nfixture\n','--user\nother\n'),0o640,'next-args-differ','another user','start');
|
||||
await tryNext(e.argsText,0o640,'next-args-same','the same lines','start');
|
||||
await tryNext(e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C}\n`),0o640,'next-args-same','only --copy differs','start');
|
||||
fault(e,`check-fails-${C1}`);
|
||||
await tryNext(nextText(e),0o640,'render-refused','the first-install check','render-check');
|
||||
assert.match(result(e),new RegExp(`^result exit=3 render-refused case2-unit.sh --check-new refuses ${C1}'s units from .*dev\\.args\\.next`));
|
||||
assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);
|
||||
assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.deepEqual(replacedDirs(e),[]);
|
||||
});
|
||||
@@ -234,13 +234,16 @@ test('a copy that meets an empty target directory replaces it by rename, not by
|
||||
// prerequisites (M3): console.sso and its slot, .deployed and accounts.json.
|
||||
async function serve(t, cli, args, env) {
|
||||
const child = spawn(process.execPath, [cli, 'serve', '--port', '0', '--board', 'http://127.0.0.1:9', ...args], { env, stdio: ['ignore', 'pipe', 'pipe'] });
|
||||
t.after(() => child.kill());
|
||||
const exited = once(child, 'exit');
|
||||
// Cleanup waits for the Console to exit, so no child outlives the test.
|
||||
t.after(async () => { if (child.exitCode === null && child.signalCode === null) child.kill('SIGKILL'); await exited; });
|
||||
let out = '', err = '';
|
||||
child.stderr.on('data', (c) => { err += c; });
|
||||
const timer = setTimeout(() => child.kill('SIGKILL'), 60_000);
|
||||
const port = await new Promise((resolve, reject) => {
|
||||
child.stdout.on('data', (c) => { out += c; const m = out.match(/^Mosaic Console: http:\/\/127\.0\.0\.1:(\d+)\/$/m); if (m && out.includes('Ctrl-C')) resolve(Number(m[1])); });
|
||||
child.on('exit', (code) => reject(new Error(`exited ${code}: ${out}${err}`)));
|
||||
});
|
||||
child.on('exit', (code, signal) => reject(new Error(`exited ${code ?? signal} before listening: ${out}${err}`)));
|
||||
}).finally(() => clearTimeout(timer));
|
||||
return { out, port };
|
||||
}
|
||||
|
||||
@@ -271,13 +274,24 @@ test('a stamped copy refuses the former unauthenticated Console start', async t
|
||||
});
|
||||
|
||||
const PUBLIC = 'console.test';
|
||||
// Each request is bounded: a stall, an aborted or incomplete response, or a
|
||||
// body that is not JSON rejects with the path instead of hanging the test.
|
||||
function request(port, path, headers = {}) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const req = httpRequest({ host: '127.0.0.1', port, path, headers: { host: PUBLIC, ...headers } }, (res) => {
|
||||
let text = '';
|
||||
res.setEncoding('utf8');
|
||||
res.on('data', (c) => { text += c; });
|
||||
res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body: text ? JSON.parse(text) : null }));
|
||||
res.on('aborted', () => reject(new Error(`${path}: response aborted`)));
|
||||
res.on('error', reject);
|
||||
res.on('end', () => {
|
||||
if (!res.complete) return reject(new Error(`${path}: incomplete response`));
|
||||
let body = null;
|
||||
try { body = text ? JSON.parse(text) : null; } catch (e) { return reject(new Error(`${path}: ${res.statusCode} body is not JSON: ${text.slice(0, 200)}`)); }
|
||||
resolve({ status: res.statusCode, headers: res.headers, body });
|
||||
});
|
||||
});
|
||||
req.setTimeout(10_000, () => req.destroy(new Error(`${path}: no response within 10s`)));
|
||||
req.on('error', reject); req.end();
|
||||
});
|
||||
}
|
||||
@@ -295,7 +309,7 @@ async function signIn(port, op, iss) {
|
||||
return async (path) => { const r = await request(port, path, { cookie: session }); return { status: r.status, body: r.body }; };
|
||||
}
|
||||
|
||||
test('serve runs from a copy: /healthz 200, the config check, roles and RELEASE from the copy, the queue from --checkout', async (t) => {
|
||||
test('serve runs from a copy: /healthz 200, the config check, roles and RELEASE from the copy, the queue from --checkout', { timeout: 240_000 }, async (t) => {
|
||||
const { dir, script, head } = fixtureRepo(t);
|
||||
const copies = join(dir, 'copies');
|
||||
assert.equal((await run(dir, script, ['copy', 'refactor', '--copies', copies])).status, 0);
|
||||
@@ -335,7 +349,7 @@ test('serve runs from a copy: /healthz 200, the config check, roles and RELEASE
|
||||
const { out, port } = await serve(t, cli, args, env);
|
||||
assert.match(out, /^Bus: acme$/m, 'the config check ran from the copy');
|
||||
assert.match(out, /^Queue: read from the --checkout directory$/m);
|
||||
const health = await fetch(`http://127.0.0.1:${port}/healthz`);
|
||||
const health = await fetch(`http://127.0.0.1:${port}/healthz`, { signal: AbortSignal.timeout(10_000) });
|
||||
assert.equal(health.status, 200);
|
||||
// Under --sso the body adds each provider's state beside ok (serve.mjs /healthz).
|
||||
const hb = await health.json();
|
||||
@@ -358,7 +372,7 @@ test('serve runs from a copy: /healthz 200, the config check, roles and RELEASE
|
||||
// no packages/queue, so it fails closed.
|
||||
const plain = await serve(t, cli, ['--business', 'acme', '--sso'], env);
|
||||
assert.doesNotMatch(plain.out, /^Queue:/m);
|
||||
assert.equal((await fetch(`http://127.0.0.1:${plain.port}/healthz`)).status, 200);
|
||||
assert.equal((await fetch(`http://127.0.0.1:${plain.port}/healthz`, { signal: AbortSignal.timeout(10_000) })).status, 200);
|
||||
const none = await (await signIn(plain.port, op, iss))('/api/queue');
|
||||
assert.equal(none.status, 503);
|
||||
assert.equal(none.body.error, 'queue-refused');
|
||||
|
||||
+20
-5
@@ -5,9 +5,15 @@
|
||||
# mosaic-proxy.service under --proxy, from the existing templates, which it
|
||||
# doesn't change.
|
||||
#
|
||||
# case2-unit.sh --user NAME --home DIR --business ID --copy ID --node PATH
|
||||
# [--profile factory|library] [--proxy] [--pm] [--claude PATH]
|
||||
# -- SERVE FLAGS
|
||||
# case2-unit.sh [--check|--check-new] --user NAME --home DIR --business ID
|
||||
# --copy ID --node PATH [--profile factory|library] [--proxy]
|
||||
# [--pm] [--claude PATH] -- SERVE FLAGS
|
||||
#
|
||||
# --check runs every check a render runs, the installed units' included, then
|
||||
# exits 0 without writing, enabling or starting anything. --check-new does the
|
||||
# same as a first install would, without the installed units' checks; it is for
|
||||
# stack-upgrade.sh --next-args, which moves the installed units aside before its
|
||||
# render (docs/guides/host-run-sheet.md section 7). Neither writes a file.
|
||||
#
|
||||
# --org ID is the same option as --business ID (rename design r2), in each of
|
||||
# two scopes: this script's own flags before --, and the serve flags after it.
|
||||
@@ -85,7 +91,7 @@ PROXY_UNIT=mosaic-proxy.service
|
||||
BUS_UNIT=[email protected]
|
||||
WEBUI_UNIT=mosaic-webui.service
|
||||
|
||||
usage() { echo 'usage: case2-unit.sh --user NAME --home DIR --business ID --copy ID --node PATH [--profile factory|library] [--proxy] [--pm] [--claude PATH] -- SERVE FLAGS' >&2; exit 4; }
|
||||
usage() { echo 'usage: case2-unit.sh [--check|--check-new] --user NAME --home DIR --business ID --copy ID --node PATH [--profile factory|library] [--proxy] [--pm] [--claude PATH] -- SERVE FLAGS' >&2; exit 4; }
|
||||
die() { echo "case2-unit: $*" >&2; exit 1; }
|
||||
# A value the unit carries as itself: no space, %, $, quote, backslash, & or |.
|
||||
plain() { case "$1" in *[!A-Za-z0-9/._+@-]*) return 1 ;; esac; }
|
||||
@@ -116,7 +122,8 @@ serve_refused() {
|
||||
*) die "serve $2" ;;
|
||||
esac
|
||||
}
|
||||
PROFILE=library USER_NAME='' HOME_DIR='' BUSINESS='' COPY_ID='' NODE='' PROXY=0 PM=0 CLAUDE='' CLAUDE_SET=0
|
||||
PROFILE=library USER_NAME='' HOME_DIR='' BUSINESS='' COPY_ID='' NODE='' PROXY=0 PM=0 CLAUDE='' CLAUDE_SET=0 CHECK=''
|
||||
case "${1-}" in --check|--check-new) CHECK="$1"; shift ;; esac
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--business|--org) BUSINESS="${2-}"; selector_take outer id outer_refused business_id_ok "$1" "${@:2:1}"; shift 2 ;;
|
||||
@@ -720,6 +727,7 @@ else
|
||||
|| die "installed $PROXY_UNIT requires --proxy; nothing was changed"
|
||||
fi
|
||||
for pair in "${PAIRS[@]}"; do
|
||||
[ "$CHECK" != --check-new ] || break
|
||||
IFS=: read -r kind name var <<<"$pair"
|
||||
path="$UNIT_DIR/$name"
|
||||
[ ! -L "$path" ] || die "refusing: $path is a symlink"
|
||||
@@ -737,6 +745,13 @@ read -r -a ids <<<"$installed_ids"
|
||||
for old in "${ids[@]}"; do
|
||||
[ "$old" = "${ids[0]}" ] || die "the installed units name different copies, ${ids[0]} and $old"
|
||||
done
|
||||
if [ "$CHECK" = --check ]; then
|
||||
echo "case2-unit: check passed: the units for copy $COPY_ID render, and the installed ones match them; nothing was written"
|
||||
exit 0
|
||||
elif [ "$CHECK" = --check-new ]; then
|
||||
echo "case2-unit: check passed: the units for copy $COPY_ID render as a first install; the installed ones were not compared; nothing was written"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# `systemctl start` leaves a running unit on the copy it started from, so
|
||||
# note which units run before the write (row 195 owns the restart).
|
||||
|
||||
+137
-21
@@ -2041,12 +2041,16 @@ r_gone() {
|
||||
while r_alive "$1"; do [ "$SECONDS" -lt "$deadline" ] || return 1; sleep 1; done
|
||||
}
|
||||
|
||||
# Stops the Console, then the host, with SIGTERM within --timeout; one still there gets SIGKILL
|
||||
# and a FAIL line. The host has to leave the store's writer.lock behind it.
|
||||
# Stops the Console, the host, then the rehearsal provider, with SIGTERM within --timeout; one
|
||||
# still there gets SIGKILL and a FAIL line. The host has to leave the store's writer.lock behind it.
|
||||
r_stop() {
|
||||
local name pid
|
||||
for name in console host; do
|
||||
if [ "$name" = console ]; then pid="$R_CONSOLE_PID"; R_CONSOLE_PID=""; else pid="$R_HOST_PID"; R_HOST_PID=""; fi
|
||||
for name in console host provider; do
|
||||
case "$name" in
|
||||
console) pid="$R_CONSOLE_PID"; R_CONSOLE_PID="" ;;
|
||||
host) pid="$R_HOST_PID"; R_HOST_PID="" ;;
|
||||
provider) pid="$R_PROVIDER_PID"; R_PROVIDER_PID="" ;;
|
||||
esac
|
||||
[ -n "$pid" ] || continue
|
||||
kill -TERM "$pid" 2>/dev/null || true
|
||||
if r_gone "$pid"; then r_pass "$name-stopped" "pid $pid"
|
||||
@@ -2080,20 +2084,77 @@ r_port_free() {
|
||||
}
|
||||
|
||||
# The scratch config: the live one with dataRoot set to the scratch data root. M3 (#1873): a
|
||||
# console.sso keeps its provider ids, client ids, algorithms and subjects, but the public origin and
|
||||
# every issuer move to rehearsal.invalid, a name that never resolves (RFC 6761). The scratch Console
|
||||
# reaches no real provider: discovery fails as unreachable and the Console starts degraded, after
|
||||
# every other startup check. No subject is added; the live config's own are the only ones.
|
||||
# console.sso keeps its provider ids, client ids, algorithms and subjects; the public origin moves to
|
||||
# rehearsal.invalid, a name that never resolves (RFC 6761). For a stamped copy, whose Console signs
|
||||
# in, every issuer moves to the rehearsal's own provider (R_PROVIDER_JS, the origin argument). An
|
||||
# unstamped copy's Console runs without --sso, so its issuers move to rehearsal.invalid too. No
|
||||
# subject is added; the live config's own are the only ones.
|
||||
R_CONFIG_JS='
|
||||
const fs=require("node:fs");const [file,data]=process.argv.slice(1);
|
||||
const fs=require("node:fs");const [file,data,op]=process.argv.slice(1);
|
||||
const c=JSON.parse(fs.readFileSync(file,"utf8"));c.dataRoot=data;
|
||||
const sso=c.console?.sso;
|
||||
if(sso&&typeof sso==="object"){
|
||||
sso.publicOrigin="https://rehearsal.invalid";
|
||||
for(const p of Array.isArray(sso.providers)?sso.providers:[])if(p&&typeof p==="object")p.issuer=`https://rehearsal.invalid/${encodeURIComponent(String(p.id))}`;
|
||||
for(const p of Array.isArray(sso.providers)?sso.providers:[])if(p&&typeof p==="object")p.issuer=`${op||"https://rehearsal.invalid"}/${encodeURIComponent(String(p.id))}${op?"/":""}`;
|
||||
}
|
||||
process.stdout.write(JSON.stringify(c,null,2)+"\n");'
|
||||
|
||||
# M3 (#1873): the rehearsal's own OpenID provider, for a stamped copy's --sso Console. An https
|
||||
# server on 127.0.0.1, port 0, with a certificate openssl makes for this run in DIR. Its TLS key is
|
||||
# read into memory and its file removed before the server listens; cert.pem stays for the scratch
|
||||
# Console's NODE_EXTRA_CA_CERTS. Per configured provider, at /<id>/, it serves discovery and a JWKS
|
||||
# holding one public key made in memory for the first of the provider's algorithms it can make.
|
||||
# Authorization, token and every other route answer 403 or 404: it issues no code, no token and no
|
||||
# sign-in. Each request is a line in requests.log. It writes origin when listening and exits when
|
||||
# PARENT (the rehearsal) is gone, on SIGTERM, or on an error.
|
||||
R_PROVIDER_JS='
|
||||
const fs=require("node:fs"),path=require("node:path"),https=require("node:https"),crypto=require("node:crypto"),{spawnSync}=require("node:child_process");
|
||||
const [dir,cfg,parent]=process.argv.slice(1);
|
||||
const die=(m)=>{console.error(`rehearsal provider: ${m}`);process.exit(1)};
|
||||
const sso=JSON.parse(fs.readFileSync(cfg,"utf8")).console?.sso;
|
||||
if(!sso||!Array.isArray(sso.providers)||!sso.providers.length)die("the live config has no console.sso providers");
|
||||
const GEN={RS:["rsa",{modulusLength:2048}],PS:["rsa",{modulusLength:2048}],ES256:["ec",{namedCurve:"P-256"}],ES384:["ec",{namedCurve:"P-384"}],ES512:["ec",{namedCurve:"P-521"}],Ed25519:["ed25519",{}],EdDSA:["ed25519",{}]};
|
||||
const keyFor=(alg)=>{
|
||||
const g=GEN[alg]??GEN[String(alg).slice(0,2)]??(/^ML-DSA-(44|65|87)$/.test(alg)?[alg.toLowerCase(),{}]:null);
|
||||
if(!g)return null;
|
||||
try{const {publicKey}=crypto.generateKeyPairSync(...g);return {...publicKey.export({format:"jwk"}),kid:`rehearsal-${crypto.randomBytes(4).toString("hex")}`,use:"sig",alg}}catch{return null}
|
||||
};
|
||||
const providers=new Map();
|
||||
for(const p of sso.providers){
|
||||
const id=String(p?.id),algs=Array.isArray(p?.algs)?p.algs:[];
|
||||
let jwk=null;for(const a of algs){jwk=keyFor(a);if(jwk)break}
|
||||
if(!jwk)die(`provider ${id}: none of ${algs.join(", ")||"no algorithms"} is one this provider can make a key for`);
|
||||
providers.set(encodeURIComponent(id),jwk);
|
||||
}
|
||||
const tls=path.join(dir,"tls-key.pem"),certFile=path.join(dir,"cert.pem");
|
||||
const r=spawnSync("openssl",["req","-x509","-newkey","ec","-pkeyopt","ec_paramgen_curve:P-256","-nodes","-days","1","-subj","/CN=127.0.0.1","-addext","subjectAltName=IP:127.0.0.1","-keyout",tls,"-out",certFile],{encoding:"utf8"});
|
||||
let key,cert;
|
||||
try{if(r.status!==0){const why=String(r.stderr??"").trim().split("\n").pop();throw new Error(`openssl exit ${r.status??r.error?.code}${why?`: ${why}`:""}`)};key=fs.readFileSync(tls);cert=fs.readFileSync(certFile);fs.chmodSync(certFile,0o600)}
|
||||
catch(e){die(`cannot make the certificate: ${e.message}`)}
|
||||
finally{fs.rmSync(tls,{force:true})}
|
||||
const log=(q,status)=>fs.appendFileSync(path.join(dir,"requests.log"),`${q.method} ${q.url} ${status}\n`);
|
||||
let origin;
|
||||
const server=https.createServer({key,cert,requestTimeout:5000,headersTimeout:5000},(q,res)=>{
|
||||
const send=(status,body)=>{log(q,status);res.writeHead(status,{"content-type":"application/json","cache-control":"no-store"});res.end(JSON.stringify(body))};
|
||||
if(q.method!=="GET")return send(405,{error:"GET only"});
|
||||
const m=/^\/([^/?#]+)\/(\.well-known\/openid-configuration|jwks|authorize|token)$/.exec(q.url);
|
||||
const jwk=m?providers.get(m[1]):undefined;
|
||||
if(!jwk)return send(404,{error:"not found"});
|
||||
const issuer=`${origin}/${m[1]}/`;
|
||||
if(m[2]==="jwks")return send(200,{keys:[jwk]});
|
||||
if(m[2]!==".well-known/openid-configuration")return send(403,{error:"the rehearsal provider issues no code and no token"});
|
||||
send(200,{issuer,authorization_endpoint:`${issuer}authorize`,token_endpoint:`${issuer}token`,jwks_uri:`${issuer}jwks`,response_types_supported:["code"],subject_types_supported:["public"],id_token_signing_alg_values_supported:[jwk.alg],code_challenge_methods_supported:["S256"]});
|
||||
});
|
||||
server.on("error",(e)=>die(e.message));
|
||||
server.listen(0,"127.0.0.1",()=>{
|
||||
origin=`https://127.0.0.1:${server.address().port}`;
|
||||
fs.writeFileSync(path.join(dir,"origin.tmp"),`${origin}\n`,{mode:0o600});fs.renameSync(path.join(dir,"origin.tmp"),path.join(dir,"origin"));
|
||||
console.log(`rehearsal provider: ${origin}, ${providers.size} provider(s)`);
|
||||
});
|
||||
const stop=()=>{server.close();server.closeAllConnections();process.exit(0)};
|
||||
process.on("SIGTERM",stop);process.on("SIGINT",stop);process.on("SIGHUP",stop);
|
||||
setInterval(()=>{if(!fs.existsSync(`/proc/${parent}`))stop()},1000);'
|
||||
|
||||
# M3 (#1873): one 0600 placeholder per console.sso client secret slot, in the scratch config
|
||||
# directory where readSlot looks. Never the live slot: the rehearsal reads no secret.
|
||||
R_SLOTS_JS='
|
||||
@@ -2145,12 +2206,13 @@ fs.writeFileSync(out,JSON.stringify(b,null,2)+"\n",{mode:0o600,flag:"wx"});'
|
||||
# notify-off the notifier config is {"notifyVersion":1,"binding":null}, 0600 (mos-0962)
|
||||
# tracker-configured bootConfig sets no trackers (mos-0962)
|
||||
# credential-ref-live every credential reference is a file under placeholders/ (mos-0963)
|
||||
# sso-isolated every console.sso origin and issuer is rehearsal.invalid and every client
|
||||
# secret slot is a scratch placeholder (M3, #1873)
|
||||
# sso-isolated the console.sso origin is rehearsal.invalid, every issuer is the
|
||||
# rehearsal provider's (OP, a stamped copy) or on rehearsal.invalid, and every
|
||||
# client secret slot is a scratch placeholder (M3, #1873)
|
||||
# Exit 0 when all pass, 1 on the first FAIL.
|
||||
R_GUARD_JS='
|
||||
const fs=require("node:fs"),path=require("node:path");const {pathToFileURL}=require("node:url");
|
||||
const [copy,id,live,scratch,ph]=process.argv.slice(1);
|
||||
const [copy,id,live,scratch,ph,op]=process.argv.slice(1);
|
||||
const real=(p)=>{try{return fs.realpathSync(p)}catch{return null}};
|
||||
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,"utf8");return s[s.lastIndexOf(")")+2]!=="Z"}catch{return false}};
|
||||
const pass=(n,w)=>console.log(`PASS ${n}${w?": "+w:""}`);
|
||||
@@ -2195,11 +2257,13 @@ const fail=(n,w)=>{console.log(`FAIL ${n}: ${w}`);process.exit(1)};
|
||||
if(host(sso.publicOrigin)!=="rehearsal.invalid")fail("sso-isolated",`console.sso.publicOrigin is ${sso.publicOrigin}, not https://rehearsal.invalid`);
|
||||
const sd=real(path.join(path.dirname(path.resolve(process.env.MOSAIC_CONFIG)),"secrets","mosaic-stack"));
|
||||
for(const p of sso.providers){
|
||||
if(host(p.issuer)!=="rehearsal.invalid")fail("sso-isolated",`console.sso provider ${p.id} issuer is ${p.issuer}, not on rehearsal.invalid`);
|
||||
const want=op?`${op}/${encodeURIComponent(String(p.id))}/`:null;
|
||||
if(op&&p.issuer!==want)fail("sso-isolated",`console.sso provider ${p.id} issuer is ${p.issuer}, not ${want} at the rehearsal provider`);
|
||||
if(!op&&host(p.issuer)!=="rehearsal.invalid")fail("sso-isolated",`console.sso provider ${p.id} issuer is ${p.issuer}, not on rehearsal.invalid`);
|
||||
const f=real(path.join(path.dirname(path.resolve(process.env.MOSAIC_CONFIG)),"secrets","mosaic-stack",p.clientSecret.slot));
|
||||
if(f===null||sd===null||path.dirname(f)!==sd||fs.readFileSync(f,"utf8")!=="rehearsal-placeholder-secret\n")fail("sso-isolated",`console.sso provider ${p.id} has no scratch placeholder secret`);
|
||||
}
|
||||
pass("sso-isolated",`${sso.providers.length} provider(s), each at rehearsal.invalid with a placeholder secret`);
|
||||
pass("sso-isolated",`${sso.providers.length} provider(s), each ${op?`at the rehearsal provider ${op}`:"at rehearsal.invalid"} with a placeholder secret`);
|
||||
})().catch((e)=>fail("guard",String(e?.message??e).split("\n")[0]));'
|
||||
|
||||
# Opens a data root's store with a copy's Store, as the bus does at start, with no migration
|
||||
@@ -2267,6 +2331,18 @@ process.stdout.write(why);' -- "$out" "$BUSINESS" "$R_HOST_PID" "$START_MS" 2>&1
|
||||
R_GATE=healthz
|
||||
node -e '
|
||||
fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>process.exit(r.status===200?0:1),()=>process.exit(1));' -- "$R_PORT" || return 1
|
||||
# M3 (#1873): a stamped copy's Console must have found every configured provider at the
|
||||
# rehearsal provider, not started degraded.
|
||||
if [ -n "$R_PROVIDER" ]; then
|
||||
R_GATE=sso-ready
|
||||
R_WHY="$(node -e '
|
||||
const ids=JSON.parse(require("node:fs").readFileSync(process.argv[2],"utf8")).console.sso.providers.map((p)=>String(p.id));
|
||||
fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>r.json()).then((h)=>{
|
||||
const s=h?.sso&&typeof h.sso==="object"?h.sso:{};
|
||||
process.stdout.write(ids.filter((id)=>s[id]!=="ready").map((id)=>`provider ${id} is ${s[id]??"missing"}, not ready`).join("; "));
|
||||
},(e)=>process.stdout.write(`healthz unreadable: ${e.message}`));' -- "$R_PORT" "$R/config/config.json" 2>&1)" || R_WHY="${R_WHY:-healthz unreadable}"
|
||||
[ -z "$R_WHY" ] || return 1
|
||||
fi
|
||||
R_GATE=copy-match
|
||||
tr '\0' ' ' < "/proc/$R_HOST_PID/cmdline" | grep -qF " $copy/packages/cli/src/cli.mjs bus start $BUSINESS --no-launch" || { R_WHY="the host's command line is not $copy's cli.mjs bus start $BUSINESS --no-launch"; return 1; }
|
||||
tr '\0' ' ' < "/proc/$R_CONSOLE_PID/cmdline" | grep -qF " $copy/packages/webui/src/cli.mjs serve " || { R_WHY="the Console's command line is not $copy's webui cli.mjs serve"; return 1; }
|
||||
@@ -2279,7 +2355,43 @@ r_gate() {
|
||||
[ "$SECONDS" -lt "$deadline" ] || { r_fail "$R_GATE" "${R_WHY:-no answer within ${TIMEOUT}s}"; return 1; }
|
||||
sleep 1
|
||||
done
|
||||
r_pass gate "host-up, bus-status, console-up, healthz and copy-match on $TO"
|
||||
r_pass gate "host-up, bus-status, console-up, healthz${R_PROVIDER:+, sso-ready} and copy-match on $TO"
|
||||
}
|
||||
|
||||
# M3 (#1873): starts R_PROVIDER_JS for a stamped copy, in $R/provider, and waits for its origin.
|
||||
r_provider() {
|
||||
local deadline=$((SECONDS + TIMEOUT))
|
||||
mkdir -m 700 "$R/provider" || r_abort provider-start "cannot make $R/provider"
|
||||
env -i HOME="$HOME" PATH="$PATH" node -e "$R_PROVIDER_JS" -- "$R/provider" "$LIVE_CFG" "$$" > "$R/provider/provider.log" 2>&1 &
|
||||
R_PROVIDER_PID=$!
|
||||
until [ -s "$R/provider/origin" ]; do
|
||||
r_alive "$R_PROVIDER_PID" || { wait "$R_PROVIDER_PID" 2>/dev/null || true; R_PROVIDER_PID=""; r_abort provider-start "the rehearsal provider exited: $(tail -n 1 "$R/provider/provider.log" 2>/dev/null)"; }
|
||||
[ "$SECONDS" -lt "$deadline" ] || r_abort provider-start "the rehearsal provider wrote no origin within ${TIMEOUT}s"
|
||||
sleep 0.2
|
||||
done
|
||||
R_PROVIDER="$(cat "$R/provider/origin")"
|
||||
[[ "$R_PROVIDER" =~ ^https://127\.0\.0\.1:[0-9]{1,5}$ ]] || r_abort provider-start "the rehearsal provider's origin $R_PROVIDER is not https on 127.0.0.1"
|
||||
[ ! -e "$R/provider/tls-key.pem" ] || r_abort provider-start "the rehearsal provider left its TLS key at $R/provider/tls-key.pem"
|
||||
r_pass provider-start "pid $R_PROVIDER_PID at $R_PROVIDER: discovery and a JWKS from keys made for this run, no code and no token; its certificate $R/provider/cert.pem is the trust the scratch Console alone gets"
|
||||
}
|
||||
|
||||
# M3 (#1873): the Console fetched each provider's discovery and JWKS from the rehearsal provider and
|
||||
# asked it for no code and no token.
|
||||
r_provider_seen() {
|
||||
local why
|
||||
why="$(node -e '
|
||||
const fs=require("node:fs");const [cfg,log]=process.argv.slice(1);
|
||||
const lines=fs.existsSync(log)?fs.readFileSync(log,"utf8").split("\n").filter(Boolean):[];
|
||||
const why=[];
|
||||
for(const p of JSON.parse(fs.readFileSync(cfg,"utf8")).console.sso.providers){
|
||||
const seg=encodeURIComponent(String(p.id));
|
||||
for(const what of [".well-known/openid-configuration","jwks"])if(!lines.includes(`GET /${seg}/${what} 200`))why.push(`no GET /${seg}/${what} 200`);
|
||||
}
|
||||
for(const l of lines)if(/\/(authorize|token)\b/.test(l))why.push(`a request for ${l}`);
|
||||
process.stdout.write(why.join("; "));' -- "$R/config/config.json" "$R/provider/requests.log" 2>&1)" || why="${why:-cannot read $R/provider/requests.log}"
|
||||
if [ -z "$why" ]; then r_pass provider-discovery "the Console fetched each provider's discovery and JWKS from $R_PROVIDER and asked for no code or token"
|
||||
else r_fail provider-discovery "$why; see $R/provider/requests.log"
|
||||
fi
|
||||
}
|
||||
|
||||
# No role session: the host wrote no sessions.json and no launches directory or log.
|
||||
@@ -2343,11 +2455,12 @@ do_rehearse() {
|
||||
R="$DATA/rehearsals/$stamp" STEPS="$DATA/rehearsals/$stamp/steps.log"
|
||||
printf '%s\n' "${R_EARLY[@]}" > "$STEPS" && printf '%s\n' "$before" > "$R/backup.before" && printf '%s\n' "$live" > "$R/live.before" ||
|
||||
r_abort scratch-build "cannot write the sums to $R"
|
||||
[ -z "$R_STAMPED" ] || r_provider
|
||||
(
|
||||
umask 077
|
||||
mkdir "$R/config" "$R/config/businesses" "$R/placeholders" "$R/data" "$R/data/bus" "$R/data/notify" "$R/data/notify/$BUSINESS" &&
|
||||
cp "$R_BK"/bus/* "$R/data/bus/" &&
|
||||
node -e "$R_CONFIG_JS" -- "$LIVE_CFG" "$R/data" > "$R/config/config.json" &&
|
||||
node -e "$R_CONFIG_JS" -- "$LIVE_CFG" "$R/data" "$R_PROVIDER" > "$R/config/config.json" &&
|
||||
node -e "$R_BUSINESS_JS" -- write "$LIVE_BIZ" "$R/config/businesses/$BUSINESS.json" "$R/placeholders" &&
|
||||
printf '{"notifyVersion":1,"binding":null}\n' > "$R/data/notify/$BUSINESS/notify.json" &&
|
||||
{ ! node -e 'process.exit(JSON.parse(require("node:fs").readFileSync(process.argv[1],"utf8")).console?.sso?0:1)' -- "$R/config/config.json" ||
|
||||
@@ -2364,7 +2477,7 @@ do_rehearse() {
|
||||
[ -z "$R_STAMPED" ] || r_pass deployment "$TO is stamped: $R/data/.deployed from its writeDeployment, the live accounts.json copied to $R/config, the Console starts with --sso"
|
||||
|
||||
local out
|
||||
rc=0; out="$(r_env node -e "$R_GUARD_JS" -- "$COPIES/$TO" "$BUSINESS" "$DATA" "$R/data" "$R/placeholders" 2>&1)" || rc=$?
|
||||
rc=0; out="$(r_env node -e "$R_GUARD_JS" -- "$COPIES/$TO" "$BUSINESS" "$DATA" "$R/data" "$R/placeholders" "$R_PROVIDER" 2>&1)" || rc=$?
|
||||
while IFS= read -r i; do r_line "$i"; done <<< "$out"
|
||||
[ "$rc" = 0 ] || { R_RC=1; r_end; }
|
||||
|
||||
@@ -2379,9 +2492,12 @@ do_rehearse() {
|
||||
local -a tester_rehearsal=()
|
||||
if grep -q 'MOSAIC_TESTER_SOCKET_V1' "$COPIES/$TO/packages/webui/src/cli.mjs"; then tester_rehearsal=(--tester-socket); fi
|
||||
[ -z "$R_STAMPED" ] || tester_rehearsal+=(--sso)
|
||||
env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" node "$COPIES/$TO/packages/webui/src/cli.mjs" serve --port "$R_PORT" --business "$BUSINESS" "${tester_rehearsal[@]}" > "$R/console.log" 2>&1 &
|
||||
# M3 (#1873): the rehearsal provider's certificate is trusted by the scratch Console only.
|
||||
local -a r_ca=()
|
||||
[ -z "$R_PROVIDER" ] || r_ca=(NODE_EXTRA_CA_CERTS="$R/provider/cert.pem")
|
||||
env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json" "${r_ca[@]}" node "$COPIES/$TO/packages/webui/src/cli.mjs" serve --port "$R_PORT" --business "$BUSINESS" "${tester_rehearsal[@]}" > "$R/console.log" 2>&1 &
|
||||
R_CONSOLE_PID=$!
|
||||
if r_gate; then r_no_session; fi
|
||||
if r_gate; then r_no_session; [ -z "$R_PROVIDER" ] || r_provider_seen; fi
|
||||
r_end
|
||||
}
|
||||
|
||||
@@ -2419,7 +2535,7 @@ FROM="" TO="" ACT="" RS="" POST="" BACKUP="" TAKEN="" RESTORED="" STEPS="" COUNT
|
||||
STORE_SCHEMA="" START_EPOCH="" START_MS=0 GATE="" GATE_WHY="" S_CHANGED=""
|
||||
P_COPY="" P_PM=0 P_PENDING=0 P_SCHEMA="" P_CLAUDE="" P_CONSOLE=() RUN_PM=0 RUN_PENDING=0 RUN_CONSOLE=() DROP_TESTER="" PROOF_PLAN="" PROOF_STEP="" SEND_PLAN="" SEND_STEP="" PLAN_REFUSE=()
|
||||
BASE_CONSOLE=() SERVE_FLAGS=() SSO_PLAN="" SSO_STEP="" SSO_DROP="" SSO_NO_TARGET=""
|
||||
R="" R_RC=0 R_EARLY=() R_BK="" R_HOST_PID="" R_CONSOLE_PID="" R_STARTED="" LIVE_CFG="" LIVE_BIZ="" LIVE_FILES=() R_STAMPED=""
|
||||
R="" R_RC=0 R_EARLY=() R_BK="" R_HOST_PID="" R_CONSOLE_PID="" R_PROVIDER_PID="" R_PROVIDER="" R_STARTED="" LIVE_CFG="" LIVE_BIZ="" LIVE_FILES=() R_STAMPED=""
|
||||
O_ACT="" FLOOR=0
|
||||
case "$OP" in
|
||||
adopt) do_adopt ;;
|
||||
|
||||
+128
-18
@@ -6,8 +6,9 @@
|
||||
# agents/darkwing/work/queue-233/BUILD.md.
|
||||
#
|
||||
# stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
|
||||
# [--rehearse-port N] [--timeout S]
|
||||
# stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
|
||||
# [--rehearse-port N] [--next-args] [--timeout S]
|
||||
# stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--next-args]
|
||||
# [--timeout S]
|
||||
#
|
||||
# Root runs it from a root-owned checkout of the reviewed commit, never from a copy, because the
|
||||
# stack's user can write the copies. The unit values come from /etc/mosaic-stack/<instance>.args,
|
||||
@@ -20,6 +21,20 @@
|
||||
# output but exit codes and the rehearsal's BACKUP_STORE line (sage-1653), which it copies into its
|
||||
# step log after a match against a fixed pattern and decides nothing on.
|
||||
#
|
||||
# Before anything stops, case2-unit.sh --check renders the installed copy from the args file and
|
||||
# compares the installed units with it, then checks the target's render the same way; either
|
||||
# refusal is render-refused, with the units left running (#1873 M3).
|
||||
#
|
||||
# --next-args changes the serve flags, which a re-render otherwise refuses (row 195's installed-unit
|
||||
# match): /etc/mosaic-stack/<instance>.args.next, root's with no group or other write, must equal
|
||||
# the args file before `--` but for --copy's value, and differ from it after `--`. The target
|
||||
# renders from it through case2-unit.sh --check-new before anything stops. After A3, root moves the
|
||||
# installed unit files into a new <instance>.replaced.XXXXXX directory beside the args file, and A4
|
||||
# renders from the next args as a first install. A failure from there moves the replaced files back
|
||||
# unchanged, restarts them on the old copy and gates it, with no render. After confirm, the next
|
||||
# args become the args file, and the replaced directory keeps the old unit files and args file for
|
||||
# a later rollback with --next-args (docs/guides/host-run-sheet.md section 7).
|
||||
#
|
||||
# --no-rehearsal is for a stack whose log records no backup. The rehearsal runs either way: exit 3
|
||||
# from it (no-backup) goes on only with the flag, and exit 0 with the flag refuses
|
||||
# rehearsal-required (sage-1650, sage-1652).
|
||||
@@ -47,8 +62,8 @@ REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
usage() {
|
||||
cat >&2 <<'USAGE'
|
||||
usage: stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal] [--rehearse-port N] [--timeout S]
|
||||
stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
|
||||
usage: stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal] [--rehearse-port N] [--next-args] [--timeout S]
|
||||
stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--next-args] [--timeout S]
|
||||
USAGE
|
||||
exit 4
|
||||
}
|
||||
@@ -70,7 +85,7 @@ ARGV=("$@")
|
||||
[ $# -gt 0 ] || usage
|
||||
OP="$1"; shift
|
||||
case "$OP" in upgrade|rollback) ;; *) usage ;; esac
|
||||
INSTANCE="" BUNDLE="" SHA="" MIGRATE=0 NO_REHEARSAL=0 R_PORT=7339 TIMEOUT=60 TO="" RESTORE="" EVENTS=""
|
||||
INSTANCE="" BUNDLE="" SHA="" MIGRATE=0 NO_REHEARSAL=0 R_PORT=7339 TIMEOUT=60 TO="" RESTORE="" EVENTS="" NEXT=0 ASIDE="" MOVED=() REPLACED_NOTE=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--instance) [ $# -ge 2 ] || usage; INSTANCE="$2"; shift 2 ;;
|
||||
@@ -79,6 +94,7 @@ while [ $# -gt 0 ]; do
|
||||
--migrate) [ "$OP" = upgrade ] || usage; MIGRATE=1; shift ;;
|
||||
--no-rehearsal) [ "$OP" = upgrade ] || usage; NO_REHEARSAL=1; shift ;;
|
||||
--rehearse-port) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{3,4}$ ]] && [ "$2" -le 65535 ] || usage; R_PORT="$2"; shift 2 ;;
|
||||
--next-args) NEXT=1; shift ;;
|
||||
--timeout) [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
|
||||
--to) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; TO="$2"; shift 2 ;;
|
||||
--restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ $ID_RE ]] || usage; RESTORE="$2"; shift 2 ;;
|
||||
@@ -125,8 +141,8 @@ logl "start rc=0 argv:$(printf ' %q' "$0" "${ARGV[@]}")"
|
||||
# The selector scopes: selector.sh defines functions only, from this root-owned checkout, as RENDER is.
|
||||
. "$REPO/scripts/selector.sh" || refuse checkout-unreadable "cannot read $REPO/scripts/selector.sh"
|
||||
args_business_ok() { [[ "$1" =~ ^[a-z0-9][a-z0-9-]*$ ]]; }
|
||||
args_outer_refused() { refuse args-unreadable "$ETC/$INSTANCE.args before --: $2"; }
|
||||
args_serve_refused() { refuse args-unreadable "$ETC/$INSTANCE.args after --: $2"; }
|
||||
args_outer_refused() { refuse args-unreadable "$ARGS_FILE before --: $2"; }
|
||||
args_serve_refused() { refuse args-unreadable "$ARGS_FILE after --: $2"; }
|
||||
|
||||
# I2's file: one case2-unit.sh argument per line, owned by root with no group or other write. Sets
|
||||
# ARGS (the lines), S_USER, S_HOME, BUSINESS, NODE, ARGS_COPY, ARGS_PROFILE and ARGS_PROXY, and the paths derived
|
||||
@@ -140,7 +156,8 @@ args_serve_refused() { refuse args-unreadable "$ETC/$INSTANCE.args after --: $2"
|
||||
# says which of them take a value, so a value is never taken for a selector; a token that grammar
|
||||
# lacks refuses, as the render would.
|
||||
read_args() {
|
||||
local f="$ETC/$INSTANCE.args" st i sep copies=0 v profile=library served
|
||||
ARGS_FILE="${1:-$ETC/$INSTANCE.args}"
|
||||
local f="$ARGS_FILE" st i sep copies=0 v profile=library served
|
||||
[ -f "$f" ] && [ ! -L "$f" ] || refuse args-unreadable "no regular file $f"
|
||||
st="$(stat -c '%u %a' "$f")" || refuse args-unreadable "cannot stat $f"
|
||||
[ "${st% *}" = "$(id -u)" ] && (( (8#${st#* } & 8#022) == 0 )) || refuse args-unreadable "$f must be $(id -un)'s with no group or other write (it is uid ${st% *}, mode ${st#* })"
|
||||
@@ -224,17 +241,82 @@ legacy_console() {
|
||||
done
|
||||
}
|
||||
|
||||
# RENDER_ARGS: the args file's lines with copy $1 in place of --copy's value.
|
||||
# RENDER_ARGS: the lines of array $2 (default ARGS) with copy $1 in place of --copy's value.
|
||||
args_with() {
|
||||
local i past=""
|
||||
local -n lines_="${2:-ARGS}"
|
||||
RENDER_ARGS=()
|
||||
for ((i = 0; i < ${#ARGS[@]}; i++)); do
|
||||
RENDER_ARGS+=("${ARGS[i]}")
|
||||
[ "${ARGS[i]}" != -- ] || past=1
|
||||
if [ -z "$past" ] && [ "${ARGS[i]}" = --copy ]; then RENDER_ARGS+=("$1"); i=$((i + 1)); fi
|
||||
for ((i = 0; i < ${#lines_[@]}; i++)); do
|
||||
RENDER_ARGS+=("${lines_[i]}")
|
||||
[ "${lines_[i]}" != -- ] || past=1
|
||||
if [ -z "$past" ] && [ "${lines_[i]}" = --copy ]; then RENDER_ARGS+=("$1"); i=$((i + 1)); fi
|
||||
done
|
||||
}
|
||||
|
||||
# OLD_ARGS: the args file's lines, which the installed units were rendered from. Under --next-args,
|
||||
# ARGS becomes the next file's lines, which must equal OLD_ARGS before -- but for --copy's value and
|
||||
# differ after it.
|
||||
read_next() {
|
||||
OLD_ARGS=("${ARGS[@]}")
|
||||
[ "$NEXT" = 1 ] || return 0
|
||||
local f="$ETC/$INSTANCE.args.next" a b i copy="$ARGS_COPY"
|
||||
unset SELECTOR_SEEN_outer SELECTOR_VALUE_outer SELECTOR_SEEN_serve SELECTOR_VALUE_serve
|
||||
read_args "$f"
|
||||
# read_old checks the installed units against the args file's copy, not the next file's.
|
||||
ARGS_COPY="$copy"
|
||||
args_with COPY OLD_ARGS; a=("${RENDER_ARGS[@]}")
|
||||
args_with COPY; b=("${RENDER_ARGS[@]}")
|
||||
for ((i = 0; i < ${#a[@]} && i < ${#b[@]}; i++)); do
|
||||
[ "${a[i]}" = "${b[i]}" ] || refuse next-args-differ "$f must equal $ETC/$INSTANCE.args before -- but for --copy's value; line $((i + 1)) differs"
|
||||
[ "${a[i]}" != -- ] || break
|
||||
done
|
||||
[ "${a[*]}" != "${b[*]}" ] || [ "${#a[@]}" != "${#b[@]}" ] ||
|
||||
refuse next-args-same "$f has the same serve flags as $ETC/$INSTANCE.args; run without --next-args"
|
||||
}
|
||||
|
||||
# Before anything stops: the installed units against OLD's render from the args file, so a rollback
|
||||
# render-old works, then copy $1's render, against the installed units or, under --next-args, as a
|
||||
# first install from the next args.
|
||||
render_checks() {
|
||||
args_with "$OLD" OLD_ARGS
|
||||
step render-old-check "$RENDER" --check "${RENDER_ARGS[@]}" ||
|
||||
refuse render-refused "case2-unit.sh --check refuses $OLD's units from $ETC/$INSTANCE.args, so a rollback could not render them; its message is above"
|
||||
args_with "$1"
|
||||
if [ "$NEXT" = 1 ]; then
|
||||
step render-check "$RENDER" --check-new "${RENDER_ARGS[@]}" ||
|
||||
refuse render-refused "case2-unit.sh --check-new refuses $1's units from $ETC/$INSTANCE.args.next; its message is above"
|
||||
else
|
||||
step render-check "$RENDER" --check "${RENDER_ARGS[@]}" ||
|
||||
refuse render-refused "case2-unit.sh --check refuses $1's units from $ETC/$INSTANCE.args; its message is above. A re-pin that changes the serve flags takes --next-args (docs/guides/host-run-sheet.md section 7)"
|
||||
fi
|
||||
}
|
||||
|
||||
# --next-args, after A3: the installed unit files into a new directory beside the args file, so A4
|
||||
# renders as a first install. MOVED lists the files moved, for units_back.
|
||||
units_aside() {
|
||||
local u rc=0
|
||||
local -a files=([email protected] "$CON_UNIT")
|
||||
[ "$ARGS_PROXY" = 0 ] || files+=("$PROXY_UNIT")
|
||||
ASIDE="$(mktemp -d "$ETC/$INSTANCE.replaced.XXXXXX")" && chmod 0700 "$ASIDE" || rc=1
|
||||
if [ "$rc" = 0 ]; then
|
||||
for u in "${files[@]}"; do
|
||||
mv -T -- "$UNIT_DIR/$u" "$ASIDE/$u" || { rc=1; break; }
|
||||
MOVED+=("$u")
|
||||
done
|
||||
fi
|
||||
logl "units-aside rc=$rc dir: ${ASIDE:-none} moved: ${MOVED[*]}"
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
# The moved files back, unchanged, over whatever A4 wrote, then the replaced directory removed.
|
||||
units_back() {
|
||||
local u rc=0
|
||||
for u in "${MOVED[@]}"; do mv -fT -- "$ASIDE/$u" "$UNIT_DIR/$u" || rc=1; done
|
||||
[ "$rc" != 0 ] || rmdir -- "$ASIDE" || rc=1
|
||||
logl "units-back rc=$rc dir: $ASIDE moved: ${MOVED[*]}"
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
# The new args file, written beside the old one before anything stops, moved into place only after
|
||||
# confirm passes. $1 is the copy.
|
||||
prepare_args() {
|
||||
@@ -334,9 +416,25 @@ rollback_to_old() {
|
||||
stop_wait || { stop_units; finish 2 rollback-failed "$cause failed and the units did not stop for the rollback; $STOPPED_UNITS are stopped"; }
|
||||
step revert "${AS_USER[@]}" "$SR" revert "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and revert failed; $STOPPED_UNITS are stopped"; }
|
||||
args_with "$OLD"
|
||||
step render-old "$RENDER" "${RENDER_ARGS[@]}" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the render of $OLD failed; $STOPPED_UNITS are stopped"; }
|
||||
if [ -n "$ASIDE" ]; then
|
||||
# --next-args: the replaced files back as they were, then the start render-old would do.
|
||||
units_back ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the replaced units in $ASIDE did not all move back; $STOPPED_UNITS are stopped"; }
|
||||
step daemon-reload "$SYSTEMCTL" daemon-reload ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and daemon-reload failed after the replaced units moved back; $STOPPED_UNITS are stopped"; }
|
||||
if [ "$ARGS_PROXY" = 1 ]; then
|
||||
step start-proxy "$SYSTEMCTL" start "$PROXY_UNIT" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the proxy did not start for $OLD; $STOPPED_UNITS are stopped"; }
|
||||
fi
|
||||
step start-bus "$SYSTEMCTL" start "$BUS_UNIT" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the bus did not start for $OLD; $STOPPED_UNITS are stopped"; }
|
||||
step start-console "$SYSTEMCTL" start "$CON_UNIT" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the Console did not start for $OLD; $STOPPED_UNITS are stopped"; }
|
||||
else
|
||||
args_with "$OLD" OLD_ARGS
|
||||
step render-old "$RENDER" "${RENDER_ARGS[@]}" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the render of $OLD failed; $STOPPED_UNITS are stopped"; }
|
||||
fi
|
||||
# confirm --rollback exits 1 when OLD passed its gate and 2 when it failed; anything else is a
|
||||
# refusal, with OLD's gate never run.
|
||||
local rc=0
|
||||
@@ -360,18 +458,27 @@ run_steps() {
|
||||
*) rollback_to_old "$id" A3 ;;
|
||||
esac
|
||||
# A4
|
||||
if [ "$NEXT" = 1 ]; then units_aside || rollback_to_old "$id" A4; fi
|
||||
args_with "$id"
|
||||
step render "$RENDER" "${RENDER_ARGS[@]}" || rollback_to_old "$id" A4
|
||||
# A5
|
||||
step confirm "${AS_USER[@]}" "$SR" confirm "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" --timeout "$TIMEOUT" ||
|
||||
rollback_to_old "$id" A5
|
||||
if [ "$NEXT" = 1 ]; then
|
||||
# The old args file joins the replaced units before the next one takes its place.
|
||||
cp -p -- "$ETC/$INSTANCE.args" "$ASIDE/$INSTANCE.args" && rm -f -- "$ETC/$INSTANCE.args.next" &&
|
||||
logl "next-args rc=0 replaced: $ASIDE" ||
|
||||
{ logl "next-args rc=1 replaced: $ASIDE"; finish 5 args-failed "$id is live and confirmed, but the old args file did not reach $ASIDE; $ETC/$INSTANCE.args still names $OLD; copy it there, then put $NEW_ARGS in its place and remove $ETC/$INSTANCE.args.next"; }
|
||||
fi
|
||||
if mv -f "$NEW_ARGS" "$ETC/$INSTANCE.args"; then logl "args rc=0 copy: $id"
|
||||
else logl "args rc=1 copy: $id"; finish 5 args-failed "$id is live and confirmed, but $ETC/$INSTANCE.args still names $OLD; put $NEW_ARGS in its place"
|
||||
fi
|
||||
REPLACED_NOTE="${ASIDE:+; the replaced units and args file are in $ASIDE}"
|
||||
}
|
||||
|
||||
do_upgrade() {
|
||||
read_args
|
||||
read_next
|
||||
legacy_console
|
||||
# Before A1: the bundle into the stack's user's clone, then the copy from it (I1).
|
||||
step fetch "${AS_USER[@]}" git -C "$CLONE" fetch -q "$BUNDLE" refs/heads/refactor:refs/remotes/origin/refactor ||
|
||||
@@ -390,9 +497,10 @@ do_upgrade() {
|
||||
local -a migrate=()
|
||||
[ "$MIGRATE" = 0 ] || migrate=(--migrate)
|
||||
plan activate "$SHA" "$BUSINESS" "${migrate[@]}"
|
||||
render_checks "$SHA"
|
||||
rehearse
|
||||
run_steps "$SHA" activate "$SHA" "$BUSINESS" "${migrate[@]}"
|
||||
finish 0 ok "upgrade ok: $SHA is live for $BUSINESS"
|
||||
finish 0 ok "upgrade ok: $SHA is live for $BUSINESS$REPLACED_NOTE"
|
||||
}
|
||||
|
||||
# The rehearsal against the log's newest backup, before anything stops. Its output goes to the
|
||||
@@ -419,6 +527,7 @@ rehearse() {
|
||||
# check: the target is an older copy than this checkout, and no rehearsal.
|
||||
do_rollback() {
|
||||
read_args
|
||||
read_next
|
||||
legacy_console
|
||||
[ -x "$COPIES_DIR/$TO/scripts/mosaic" ] || refuse copy-missing "no $COPIES_DIR/$TO/scripts/mosaic"
|
||||
hostcheck
|
||||
@@ -427,8 +536,9 @@ do_rollback() {
|
||||
[ -z "$RESTORE" ] || restore=(--restore "$RESTORE")
|
||||
[ -z "$EVENTS" ] || restore+=(--events "$EVENTS")
|
||||
plan rollback "$BUSINESS" --to "$TO" "${restore[@]}"
|
||||
render_checks "$TO"
|
||||
run_steps "$TO" rollback "$BUSINESS" --to "$TO" "${restore[@]}"
|
||||
finish 0 ok "rollback ok: $TO is live for $BUSINESS"
|
||||
finish 0 ok "rollback ok: $TO is live for $BUSINESS$REPLACED_NOTE"
|
||||
}
|
||||
|
||||
case "$OP" in
|
||||
|
||||
Reference in new issue
Block a user