row 233: BUILD.md, the gate and m1-m8 results on d99166ade (1201/0, 8 of 8 killed) and the N1 fix that leaves the gate standing (sage-1707)

row 233: the lock test kills its flock holder's process group (N1, sage-1707)
stack-upgrade.test.mjs:312 spawned `flock LOCK sleep 60` and SIGKILLed only the flock, so its `sleep 60` child outlived the test. Spawn it detached and kill the group with process.kill(-pid), as row 232's T20 does (stack-release.test.mjs:370). Test only.
2026-10-20 23:43:28 -05:00 · 2026-10-20 23:43:21 -05:00 · 2026-10-20 23:23:00 -05:00 · 2026-10-20 23:00:36 -05:00
7 changed files with 1433 additions and 4 deletions
@@ -0,0 +1,249 @@
+# Row 233 (I3): root's upgrade driver and the host run sheet
+
+Brief: `docs/plans/2026-10-20_installer-rows.md` section I3, blob `e61291bcf1c6` (commit a0e3d7cfd), re-pinned
+by Sage (sage-1685, rev 2631) from `f35e7e57daee` (ad6f0be7a). The later blob only appends I4; the I3 text
+is the same. Gitea #1735, Vikunja P19
+#432. Owner Darkwing, reviewer Filbert, gate owner Sage. Design input: row 195 DESIGN r6,
+`agents/darkwing/work/queue-195/DESIGN.md` at b7239bb66, sections 2, 4 and 7. Where the brief and r6
+differ, the brief wins. Sage's rulings during the build: sage-1650 and sage-1652 (`--backup last`, the
+`--no-rehearsal` mapping, `--rehearse-port`, hostcheck), and sage-1653 (`from_log newest`, the two external
+plan refusals, the `BACKUP_STORE` line).
+
+## Files
+
+Base: e65a3ab4d, row 232's landing, on `darkwing/row233-r1`. The build started on ea1e85c7b
+(`darkwing/row233`, row 232's candidate e6ca230d2 and its BUILD.md) before row 232 landed. The three row 232
+files this row changes are identical on the two bases, so the uncommitted diff moved across unchanged.
+
+- `scripts/stack-upgrade.sh` (new): `upgrade` and `rollback`, root's side of r6 A1-A5, A2f and R1-R4.
+- `packages/cli/tests/stack-upgrade.test.mjs` (new, 20 tests).
+- `docs/guides/host-run-sheet.md` (new).
+- `docs/TOOLS.md`: a new subsection "Root's upgrade driver (`scripts/stack-upgrade.sh`, row 233)", plus the
+  row 232 lines below.
+- Row 232 files, changed on Sage's rulings (the brief's files list doesn't name them):
+  - `scripts/stack-release.sh`:
+    - `rehearse --backup last` takes the newest backup the log records (`from_log newest`) and prints
+      `PASS backup-last`. With none it fails `no-backup`, exits 3 and makes no rehearsal directory.
+      A newest backup that isn't an activation id aborts `backup-unreadable`, exit 1.
+    - The rehearsal prints `BACKUP_STORE <activation id> schema <version>` before its migration step.
+    - `--plan` under `--units external` refuses `open-activation` when any rollback set holds
+      `AWAITING-UNITS`, and `migrate-required` for a migrating target without `--migrate`. Root reads
+      only the plan's exit, so these two have to be refusals there. User mode prints the same bytes.
+  - `packages/cli/tests/stack-release.test.mjs`: three new `row 233:` tests for `--backup last`.
+  - `docs/TOOLS.md`: `--backup ACTIVATION|PATH|last` in the rehearse usage, a paragraph on `last`, and a
+    paragraph on the two external plan refusals.
+- This file.
+
+## The brief, item by item
+
+| Brief item | Where |
+|---|---|
+| Root runs from a root-owned checkout, never a copy | `head-mismatch` (A1); TOOLS.md; run sheet section 6 |
+| I2's render with `/etc/mosaic-stack/<instance>.args`, `--copy` the one changed value | `read_args`, `args_with`; the args test |
+| The args file changes only after a confirmed run | `prepare_args` before A2, `mv` after A5; sequences 1 and 7, m6 |
+| `copy --bundle` as the stack's user before A1 | the `fetch`, `ff`, `copy` steps; sequence 1's sudo keys |
+| A rehearsal before A2, a failed one refuses with nothing stopped | `rehearse`; three rehearsal tests, m4 |
+| The root lock, held to the end, rollback included | `flock -n 9`; the lock test and sequence 3's probe, m1, m2 |
+| A1 refuses `open-activation` | stack-release.sh's external plan; the AWAITING-UNITS test, m3 |
+| `migrate-required` without `--migrate` | stack-release.sh's external plan; its test, m5 |
+| The step log, 0640, one line per step | `logl`, `step`; every test checks lines against `STEP` |
+| Exit codes 0-3 | `finish`, `refuse`; exit 4 is usage |
+| duckworth-0055's notes | the next section |
+| Root parses no stack user output beyond exits | the step log test; the one exception is sage-1653's `BACKUP_STORE` |
+| The host run sheet | `docs/guides/host-run-sheet.md` |
+
+## duckworth-0055's notes on r6 (brief line 169)
+
+- N1, A2's stop timeout under `gate --since`. Row 232's BUILD.md left two pieces to I3. The run sheet's
+  A2f line is in section 6 ("When a run doesn't exit 0", exit 2). The test is `N1: a bus that outlives
+  A2's timeout keeps its old start`: a bus that ignores `stop` keeps its old `ActiveEnterTimestamp`,
+  `wait-down` logs `rc=1 ... state: active`, A2f restarts and gates, the gate fails `stale-start`, and the
+  driver stops both units and exits 2. Sequence 4 is the other half: an A3 exit 3 goes to A2f and exits
+  1 with the old copy's gate passing.
+- N2, the claude path under `/opt/mosaic-stack`. Run sheet section 3 installs it there as root with
+  managed settings. The args file's `--claude` carries it, and `case2-unit.sh` already refuses a binary
+  that isn't root's or sits under the home. The driver passes the args file through and adds no claude
+  check of its own. F6 is I2's.
+
+## Sage's rulings during the build
+
+- sage-1650 and sage-1652. Root always runs the rehearsal and maps its exit:
+  - 0 goes on, or refuses `rehearsal-required` with `--no-rehearsal`.
+  - 3 refuses `no-backup`, or with `--no-rehearsal` logs `rehearsal-skipped` and goes on.
+  - Anything else refuses `rehearsal-failed`.
+
+  `--rehearse-port N`, default 7339. Hostcheck: a missing file or a nonzero `--check` refuses
+  `profile-drift`.
+- sage-1653. `from_log newest`, and the external plan refusals `open-activation` and `migrate-required`.
+  Residual: the `BACKUP_STORE` id and schema land in root's step log, but nothing ties that backup to the
+  one the activation's own backup step takes later. A stopped backup before the upgrade would close that
+  gap; it's a later row, not this one.
+
+## Tests
+
+`stack-upgrade.test.mjs` runs the driver as the test's user against a scratch root, with the test seams
+`STACK_UPGRADE_*`. The stand-ins are a fake `sudo` (logs its argv, checks the driver's form, then runs the
+real command as the test's user, r6's B1 pattern), a fake `case2-unit.sh` render that installs and starts
+the fixture units, a fake `systemctl` for root's system units, and a shell `hostcheck.sh`. The stack's
+user's steps run the real `stack-release.sh` and `stack-copy.sh` from a clone made from a real bundle. The
+units, copies and store are `stack-release-fixture.mjs`'s.
+
+| Test | Covers |
+|---|---|
+| sequence 1 | confirm fails healthz, rollback with `--cause A5`, exit 1; every sudo call in the driver's form |
+| sequence 2 | `activate --migrate` fails at the wrapper, store restored, `--cause A3`, exit 1 |
+| sequence 3 | render fails, `--cause A4`; a probe run during `render-old` refuses `upgrade-running` |
+| sequence 4 | stage exits 3, A2f restarts and gates the old units, exit 1 |
+| N1 | a unit that won't stop: A2f's gate fails `stale-start`, exit 2, units stopped |
+| sequence 5 | confirm of the wrong copy refused, `--cause A5`, exit 1 |
+| sequence 6 | revert fails, exit 2; the next run refuses `interrupted-run` at its plan |
+| sequence 7 | two upgrades and a `rollback --to` with `--restore`; the args file after each confirm; `to-not-previous` |
+| sequence 8 | a migration that writes and fails, store back byte for byte, exit 1 |
+| lock | a second run while the lock is held: one log line, no call |
+| open-activation | any `AWAITING-UNITS` refuses at A1, nothing stopped |
+| migrate-required | refused without `--migrate`, no rehearsal, nothing stopped |
+| rehearsal-failed | the rehearsal's argv, exit 3, nothing stopped |
+| `BACKUP_STORE` | the line after a match; four malformed lines log `BACKUP_STORE none` |
+| no-backup | the real rehearsal with no backup refuses; `--no-rehearsal` logs the skip and goes on |
+| backup-last | the real rehearsal names the newest backup; `--no-rehearsal` with one refuses `rehearsal-required` |
+| args file | group write, other write, two `--copy`, an unknown flag, a symlink, `args-differ` |
+| hostcheck | nonzero `--check` and a missing file refuse `profile-drift` before the plan |
+| head-mismatch | checkout not at `--sha`; a commit the clone can't fast-forward to refuses `fetch-failed` |
+| usage | twelve bad argvs exit 4 with no log and no lock |
+
+The step log check (`STEP`) runs in every test that reads the log: each line must be a step line or the
+result line, and none may hold the stack user's output (`stack-release:`, `PASS `, `FAIL `, `REHEARSAL`,
+the fake rehearsal's text).
+
+## Mutants
+
+`~/darkwing-scratch/r233/mutants.sh`, not committed because it edits the worktree. It has the brief's six
+mutants and two for sage-1650's mapping. The method is row 232's: a checked backup, exact replacements, a
+syntax check, the named killers, then the file put back with its sha256 checked. One control runs first:
+the whole `stack-upgrade.test.mjs` with no change. `DRY=1` matched each replacement once on the candidate
+(2026-10-21).
+
+| Mutant | File | Killers |
+|---|---|---|
+| m1 drop the flock | stack-upgrade.sh | `a second run while the lock is held`, `sequence 3:` |
+| m2 release the lock when the rollback starts | stack-upgrade.sh | `sequence 3:` |
+| m3 skip A1's open-activation check | stack-release.sh | `A1 refuses open-activation` |
+| m4 skip the rehearsal | stack-upgrade.sh | `the rehearsal runs after the plan`, `sequence 1:` |
+| m5 accept a migrating target without `--migrate` | stack-release.sh | `migrate-required` |
+| m6 write the args file before confirm | stack-upgrade.sh | `sequence 7:`, `sequence 1:` |
+| m7 `--no-rehearsal` accepted with a backup | stack-upgrade.sh | `refuses rehearsal-required` |
+| m8 drop the `rehearsal-skipped` line | stack-upgrade.sh | `a real rehearsal with no backup` |
+
+## Deviations
+
+D1. The brief has A1 refuse `open-activation` for an open `AWAITING-UNITS` or a non-empty `interrupted`
+set. The interrupted case refuses with row 232's existing code, `interrupted-run`; only `AWAITING-UNITS`
+gives `open-activation`. Both reach root as `plan-refused`, exit 3 (sequence 6, the open-activation test).
+
+D2. Root copies one line of the stack user's output into its log, `BACKUP_STORE`, on sage-1653. It
+matches a fixed pattern first, takes the last match, logs `BACKUP_STORE none` otherwise, and decides
+nothing on it.
+
+D3. The stack user's steps carry no `--instance`. The driver takes `--instance dev` only, row 190's default
+instance, which the stack user's commands select with no flag. A named instance's units are a later row.
+
+D4. Most tests fake the rehearsal in the fake `sudo`: the fixture's stub copies can't pass a real one. Two
+tests run the real rehearsal (`faults/real-rehearse`): no-backup and backup-last. `stack-release.test.mjs`
+covers `--backup last` itself.
+
+D5. Any nonzero `--plan` exit refuses `plan-refused`, exit 3, whatever the code. The stack user's output
+above it names the refusal; root doesn't parse it.
+
+D6. `rollback --restore ACT` is accepted without `--events`, as `stack-release.sh rollback` accepts it.
+The brief's usage pairs them. `--events` without `--restore` is a usage error.
+
+D7. The refusal of any `STACK_UPGRADE_*` seam when run as root has no test, because the tests don't run
+as root. It is three lines at the top of the script.
+
+D8. The fake `sudo` keeps the test's environment, where real `sudo` resets it. The driver passes
+everything the stack user needs through `env PATH=...` and `-H`, and the fake checks that form.
+
+D9. Exit 3 means nothing stopped, not nothing changed. A refusal after `fetch` leaves the stack user's
+clone fast-forwarded and the new copy made. Both are the user's own state, and the next run reuses them.
+TOOLS.md and the run sheet say so.
+
+D10. The brief's usage has neither `--no-rehearsal` nor `--rehearse-port`. Both come from sage-1650 and
+sage-1652.
+
+## Decisions
+
+- `rollback` has no HEAD check. Its target is an older copy than root's checkout, which already exists.
+  It fetches, copies and rehearses nothing (sequence 7).
+- The new args file is written next to the old one (`.dev.args.XXXXXX`, same mode) before A2, so a full
+  `/etc` refuses before anything stops. It moves into place after A5. A2f and the rollback remove it; a failed move keeps it and names it.
+- Every stack user step gets `--copies` explicitly, so the copies directory comes from the args file's
+  `--home`, never from the user's environment.
+- The stack's user does the fetch, the fast-forward and the copy, so root never runs git on a tree that
+  user can write.
+- A4 is the render, and `case2-unit.sh` starts the bus, then the Console. The forward path has no
+  `systemctl start` of its own; only A2f does.
+
+## Run environment
+
+Worktree `/mnt/storage/scratch/darkwing-r233`, `TMPDIR=/mnt/storage/scratch/tmp/r233`, node from PATH. One
+whole `stack-upgrade.test.mjs` run takes about 56 s.
+
+## Results
+
+### Singles, run 1, 2026-10-21, Sage's slot (sage-1689)
+
+`~/darkwing-scratch/r233/singles.sh` on `darkwing/row233-r1` (base e65a3ab4d, uncommitted tree), 03:48:20Z
+to 03:59:00Z. These are the test files that read a changed script. `stack-instances.test.mjs` reads
+`stack-release.sh`, so it joined the three named in the plan.
+
+| File | Result |
+|---|---|
+| `stack-upgrade.test.mjs` | 20 pass, 0 fail |
+| `stack-release.test.mjs` | 194 pass, 1 fail |
+| `stack-release-external.test.mjs` | 25 pass, 0 fail |
+| `stack-instances.test.mjs` | 23 pass, 0 fail |
+
+The failure was my test, not the script. "row 233: --backup last takes the newest backup..." compares
+`untouched(e)` before and after the run, and `untouched` includes `existsSync(e.log)`. The test took its
+before snapshot ahead of `rlog()`, which writes the log, so that flag went from false to true. Every hash in
+the snapshot matched. The fix takes the snapshot after `rlog()`. No other row 233 test calls `untouched`.
+
+### Singles, run 2
+
+`~/darkwing-scratch/r233/rerun.sh` on aacfa2d7a, Sage's slot (sage-1695), 04:13:05Z to 04:21:28Z:
+`stack-release.test.mjs` whole, 195 pass, 0 fail, 0 skipped. The fixed test passed. The other three files
+did not change after run 1. The gate and the mutants run next, in Sage's slot.
+
+### Gate and mutants, 2026-10-21, Sage's slot (sage-1704)
+
+`~/darkwing-scratch/r233/gate.sh` then `mutants.sh`, chained behind `busy.sh`, on d99166ade (code
+aacfa2d7a; d99166ade changes only this file). Counts are an awk sum over `gate-r1.out`.
+
+| Part | Time (UTC) | Result |
+|---|---|---|
+| cli suite (`node --test tests/*.test.mjs`) | 04:29:01-04:37:27 | 616 pass, 0 fail, 0 skipped |
+| the 10 `scripts/test-*.sh` | 04:37:27-04:39:38 | selftests 429 passed, 0 failed; `test-queue`'s node part 156 pass, 0 fail |
+| gate total | | 1201 pass, 0 fail, 11 suites, `GATE_PASSED` |
+| c0 control | 04:39:38-04:40:35 | exit 0 |
+| m1-m8 | 04:40:35-04:41:25 | 8 of 8 killed, every one exit 1 |
+
+Every restore passed its sha256 check, and `git diff` was clean at d99166ade afterwards. The EXIT trap
+removed the `darkwing-r233_default` network.
+
+N1, found in the process sweep after the run: a `sleep 60` was left with its cwd in `packages/cli`.
+`stack-upgrade.test.mjs:312` spawns `flock LOCK sleep 60`, and its `t.after` SIGKILLs only the flock, so
+flock's child outlives the test by up to 60 s. It exited on its own. Row 232's T20
+(`stack-release.test.mjs:370`) spawns its holder `detached` and kills the group with `process.kill(-pid)`.
+The fix is the same two lines. That test is one of m1's killers.
+
+Sage ruled it a fix now (sage-1707), and 08db6ac3f makes it, test only. After the gate's d99166ade
+the only change outside this file is that test:
+
+```
+$ git diff --stat d99166ade 08db6ac3f
+ packages/cli/tests/stack-upgrade.test.mjs | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+```
+
+The gate stands on Sage's ruling. The rerun is the whole `stack-upgrade.test.mjs` singly plus m1, in
+Sage's slot, with a sweep for a leftover `sleep` afterwards.
@@ -669,7 +669,7 @@ pinned copy](../packages/cli/README.md#one-pinned-copy-row-85-1579)).
 #### Rehearsal on a scratch root (D2, row 163)

 ```
-scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]
+scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH|last --port N [--timeout S]
 ```

 `rehearse` starts the target copy's bus host and Console on a copy of a
@@ -680,7 +680,11 @@ runs it against live data. It stops and starts no unit and writes no pointer
 or log line.

 `--backup` takes an activation id under `<dataRoot>/backups/stack/` or an
-absolute path. `--port` is the scratch Console's loopback port, 1024 to
+absolute path. `--backup last` (row 233) takes the newest backup the
+activation log records and prints `PASS backup-last` naming it. With none in
+the log it fails `no-backup`, exits 3 and makes no directory. After the
+backup's checks it prints `BACKUP_STORE <activation id> schema <version>`.
+`--port` is the scratch Console's loopback port, 1024 to
 65535. The run refuses `port-reserved` for 7332, 7333 and the pointer's
 `--port` and `--tester-port`, and `port-in-use` for a port something holds;
 it never picks another port. `--timeout` (default 60 seconds) bounds the
@@ -779,6 +783,82 @@ instance (the business's `launch.by`), for this business, whose run started at
 or after the floor. `status` adds an
 `awaiting units:` line for an open activation.

+Under `--units external`, `activate --plan` also refuses `open-activation`
+while any rollback set holds `AWAITING-UNITS`, and `migrate-required` when
+the target migrates the store and the run lacks `--migrate` (row 233). Under
+`--units user` the plan's bytes are unchanged.
+
+#### Root's upgrade driver (`scripts/stack-upgrade.sh`, row 233)
+
+```
+scripts/stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
+    [--rehearse-port N] [--timeout S]
+scripts/stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
+```
+
+Root runs it from a root-owned checkout of the reviewed commit, never from a
+copy, because the stack's user can write the copies. Each run needs Mos's go,
+and `--migrate` needs Jason present (decision 96). The host must be prepared
+first (`docs/guides/host-run-sheet.md`). Only the default instance, `dev`, is
+accepted; a named instance's units are a later row.
+
+The unit values come from `/etc/mosaic-stack/dev.args`, which I2's install
+writes: one `case2-unit.sh` argument per line, the serve flags after `--`,
+owned by root with no group or other write, exactly one `--copy`, and that
+copy the one both installed units run. Otherwise the run refuses
+`args-unreadable` or `args-differ`. The driver writes the new copy id into
+that file only after `confirm` passes.
+
+`upgrade`, in order. Every stack user step runs as
+`sudo -u USER -H env PATH=<node dir>:/usr/local/bin:/usr/bin:/bin` from the
+user's clone `<home>/src/mosaic-stack`:
+
+1. Takes `flock -n` on `/run/mosaic-stack/upgrade-dev.lock` and holds it to
+   the end, rollback included. A second run refuses `upgrade-running`, and
+   a lock it can't open refuses `lock-failed`.
+2. Fetches the bundle's `refactor` into the clone and fast-forwards it to
+   SHA (`fetch-failed`), then `stack-copy.sh copy --bundle FILE SHA`
+   (`copy-failed`).
+3. A1: this checkout's HEAD is SHA (`head-mismatch`), the copy exists
+   (`copy-missing`), `/usr/local/lib/mosaic-stack/hostcheck.sh --check`
+   exits 0 (`profile-drift`, row 240), both units name one copy
+   (`units-unreadable`, `units-disagree`) and it is the args file's
+   (`args-differ`), and
+   `activate SHA BUSINESS --units external --plan --expect-from OLD` exits 0
+   (`plan-refused`, which covers `open-activation`, `migrate-required` and
+   `pointer-differs`).
+4. The rehearsal, `rehearse SHA BUSINESS --backup last --port 7339`
+   (`--rehearse-port`). Exit 0 goes on. Exit 3 means no backup in the log
+   and goes on only with `--no-rehearsal`, logging `rehearsal-skipped`;
+   without the flag it refuses `no-backup`. Exit 0 with `--no-rehearsal`
+   refuses `rehearsal-required`; any other exit refuses `rehearsal-failed`.
+5. A2 stops the Console, then the bus, and waits for both to read inactive
+   or failed. A3 is `activate --units external`, A4 the render with the new
+   `--copy` (which also starts the bus, then the Console), A5 `confirm`.
+   The driver writes the new args file beside the old one before A2
+   (`args-failed`, exit 3) and moves it into place after A5. A failed move
+   exits 1 `args-failed`: the new copy is live and confirmed, and root puts
+   the file in place by hand.
+
+A failure in A2 or a stage that exits 3 starts the old units again and gates
+them (A2f). A later failure stops both units, runs `revert`, renders the old
+copy and runs `confirm --rollback OLD --cause A3|A4|A5`. `rollback` is the
+same from A1 on, with `rollback BUSINESS --to ID`, no fetch, no copy and no
+rehearsal; `--restore` and `--events` pass through.
+
+Root reads no output the stack user wrote except exit codes and the
+rehearsal's `BACKUP_STORE` line, which it copies into its log after a match
+against a fixed pattern (`BACKUP_STORE none` otherwise). The step log
+`/var/log/mosaic-stack/upgrade-dev.log`, 0640, gets one
+`<time> <step> rc=N argv: ...` line per step and a last
+`<time> result exit=N <code> <text>` line.
+
+Exit 0 ok, 1 failed with the old stack running and gated, 2 rollback-failed
+with both units stopped, 3 refused with nothing stopped, 4 usage. A refusal
+after step 2 leaves the user's clone fast-forwarded and the new copy made. After exit
+2 the activation stays open, so the next run's plan refuses until root
+finishes it.
+
 ## Console service (`scripts/webui-service.sh`)

 ```bash
@@ -0,0 +1,240 @@
+# Host run sheet: preparing a host for the stack
+
+ops-01 runs this on a fresh host before install (I2), and once per host for enrollment. Jason places the
+credentials in section 9. Brief: `docs/plans/2026-10-20_installer-rows.md`, I3 (row 233, #1735). The
+host design is Darkwing's r6 (`agents/darkwing/work/queue-195/DESIGN.md`, section 2) as the brief
+changes it.
+
+The hosts are .53 (profile `factory`) and .54 (profile `library`). Both get every section unless a
+section says otherwise. Every command runs as root unless it starts with `sudo -u mosaic`.
+
+Each item names the `scripts/case2-precheck.sh` line that verifies it. Install's precheck runs as the
+stack's user with `--business ID --profile factory|library`. Where no precheck line checks an item, the
+item says "precheck line missing", and Gosalyn adds the line to I2. Today's precheck (C2-13a) has these
+gaps; C2-13b adds the library lines (`library.sync`, persona modes, no Discord binding).
+
+## 1. The stack's user and its home
+
+```sh
+useradd --system --user-group --home-dir /srv/mosaic --create-home --shell /usr/sbin/nologin mosaic
+chmod 0700 /srv/mosaic
+install -d -o root -g root -m 0755 /etc/mosaic-stack
+install -d -o root -g root -m 0750 /var/log/mosaic-stack
+```
+
+`mosaic` gets no sudo rule and no supplementary group. Check with `sudo -l -U mosaic` (expect "not allowed
+to run sudo") and `id -G mosaic` (one number).
+
+Precheck lines:
+- `ok      HOME is /srv/mosaic`
+- `ok      sudo -n true fails as mosaic`
+- `ok      id -G prints one group (<gid>)`
+- `/srv/mosaic` at 0700: precheck line missing.
+
+## 2. node, pinned under /opt
+
+The pin is the version and sha256 the window names. `<v>` below is that version, for example `v24.x.y`.
+
+```sh
+install -d -o root -g root -m 0755 /etc/mosaic-stack/node-keys /opt/mosaic-stack
+cd "$(mktemp -d)"
+curl -fsSLO https://nodejs.org/dist/<v>/node-<v>-linux-x64.tar.xz
+curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt
+curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt.asc
+gpg --no-default-keyring --keyring /etc/mosaic-stack/node-keys/pubring.kbx --verify SHASUMS256.txt.asc SHASUMS256.txt
+grep " node-<v>-linux-x64.tar.xz\$" SHASUMS256.txt | sha256sum -c -
+echo "<pinned sha256>  node-<v>-linux-x64.tar.xz" | sha256sum -c -
+tar -xJf node-<v>-linux-x64.tar.xz --no-same-owner
+mv node-<v>-linux-x64 /opt/mosaic-stack/node-<v>
+chown -R root:root /opt/mosaic-stack/node-<v>
+chmod -R go-w /opt/mosaic-stack/node-<v>
+```
+
+The keyring holds the Node release keys from the nodejs/release-keys repository, nothing else. Stop on any
+failed check; don't unpack. Rollback: `rm -r /opt/mosaic-stack/node-<v>`.
+
+Precheck lines:
+- `ok      node <v>` (version 24 or later only).
+- The `--node` path in the args file must be a root-owned regular file outside the home. `case2-unit.sh`
+  refuses otherwise at render, which is not a precheck line.
+- The installed version equals the pin: precheck line missing.
+
+## 3. claude, factory only (.53)
+
+Root installs it under `/opt`, never under the stack's home (this replaces r6's `/srv/mosaic/.local/bin`).
+The pin is 2.1.304 unless the window names a newer one.
+
+```sh
+install -d -o root -g root -m 0755 /opt/mosaic-stack/claude-<pin> /etc/claude-code
+npm install --prefix /opt/mosaic-stack/claude-<pin> @anthropic-ai/claude-code@<pin>
+chown -R root:root /opt/mosaic-stack/claude-<pin>
+printf '%s\n' '{"env":{"DISABLE_AUTOUPDATER":"1"}}' > /etc/claude-code/managed-settings.json
+chown root:root /etc/claude-code/managed-settings.json
+chmod 0644 /etc/claude-code/managed-settings.json
+```
+
+Rollback: `rm -r /opt/mosaic-stack/claude-<pin> /etc/claude-code/managed-settings.json`.
+
+The args file's `--claude` names the binary under `/opt/mosaic-stack/claude-<pin>`. `case2-unit.sh`
+refuses one that isn't a root-owned regular file outside the home.
+
+Precheck lines:
+- `ok      claude replaces its prompt with --system-prompt-file (<n> input tokens)`. It runs the claude the bus
+  unit's `MOSAIC_CLAUDE_BIN` names, so it needs the model credential from section 9.
+- Managed settings present with the autoupdater off: precheck line missing.
+
+## 4. bubblewrap, AppArmor and the drift stop
+
+Both hosts. These are r6's items, unchanged.
+
+```sh
+apt-get install bubblewrap=0.9.0-1ubuntu0.3
+stat -c '%U:%G %a' /usr/bin/bwrap > /etc/mosaic-stack/bwrap.orig    # expect root:root 755
+
+cd "$(mktemp -d)"
+apt-get download apparmor-profiles=4.0.1really4.0.1-0ubuntu0.24.04.9
+dpkg-deb --fsys-tarfile apparmor-profiles_*.deb | tar -x ./usr/share/apparmor/extra-profiles/bwrap-userns-restrict
+echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9  usr/share/apparmor/extra-profiles/bwrap-userns-restrict" | sha256sum -c -
+install -o root -g root -m 0644 usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/bwrap-userns-restrict
+apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict
+
+dpkg-statoverride --update --add root mosaic 0750 /usr/bin/bwrap
+```
+
+`aa-status` then lists `bwrap` and `unpriv_bwrap` in enforce mode. Install the one profile file, never the
+whole `apparmor-profiles` package, whose 20 other profiles would load at the next boot. Rollback, in
+reverse order: `dpkg-statoverride --remove /usr/bin/bwrap`, then `chown` and `chmod` to the values in
+`bwrap.orig`; `apparmor_parser -R /etc/apparmor.d/bwrap-userns-restrict`, then `rm` it;
+`apt-get remove bubblewrap`.
+
+The no-new-privs test must print `bwrap//&unpriv_bwrap (enforce)`:
+
+```sh
+systemd-run --wait --pipe --uid=mosaic --gid=mosaic -p NoNewPrivileges=yes -p ProtectSystem=strict \
+  -p ProtectHome=yes -p PrivateTmp=yes \
+  /usr/bin/bwrap --unshare-user --unshare-pid --ro-bind / / cat /proc/self/attr/current
+```
+
+The P9 checks: another user's `/usr/bin/bwrap --version` fails with Permission denied, `getent group mosaic`
+lists no member, and `sysctl fs.protected_hardlinks` prints 1. `sysctl
+kernel.apparmor_restrict_unprivileged_userns` prints 1 too (Ubuntu 24.04's default); leave it, so nothing
+else on the host gains user namespaces.
+
+The drift stop is row 240's (I4, #1746): `hostcheck.sh` and its pins in `/usr/local/lib/mosaic-stack/`, the
+apt hook `/etc/apt/apt.conf.d/80mosaic-hostcheck`
+(`DPkg::Post-Invoke { "/usr/local/lib/mosaic-stack/hostcheck.sh --enforce || true"; };`) and the boot unit
+`mosaic-hostcheck.service`. Install them from row 240's run sheet lines once it lands. Until then
+`stack-upgrade.sh` refuses `profile-drift` on every run, because it runs `hostcheck.sh --check` and a
+missing file refuses.
+
+Precheck lines:
+- `ok      bwrap --unshare-all runs`
+- The AppArmor profile, the statoverride, the no-new-privs result and the P9 checks: precheck line missing.
+  A line that runs `/usr/local/lib/mosaic-stack/hostcheck.sh --check` would cover them once row 240 lands.
+
+## 5. The port floor
+
+Leave `net.ipv4.ip_unprivileged_port_start` at its default (1024) or higher. The Console's port is below
+it and the unit gets `CAP_NET_BIND_SERVICE`, so no other `mosaic` process can bind that port.
+
+Precheck line: `ok      net.ipv4.ip_unprivileged_port_start <n> is above the Console's port <port>`.
+
+## 6. The release transport
+
+A release is a git bundle of `refs/heads/refactor` plus the 40-character commit id (I1, mos-1118 choice 2).
+Mos's go names both, with the bundle's sha256. `<sha>` below is that id.
+
+First install:
+
+```sh
+# Copy the bundle in, then check it before anything reads it.
+echo "<bundle sha256>  /root/<sha>.bundle" | sha256sum -c -
+git bundle verify /root/<sha>.bundle
+
+# Root's checkout. stack-upgrade.sh and case2-unit.sh run from here, never from a copy.
+git clone -q -b refactor /root/<sha>.bundle /root/mosaic-stack
+git -C /root/mosaic-stack checkout -q --detach <sha>
+git -C /root/mosaic-stack rev-parse HEAD        # must print <sha>
+
+# The stack's user's copy of the bundle, and its own clone.
+install -d -o mosaic -g mosaic -m 0700 /srv/mosaic/incoming /srv/mosaic/src
+install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
+sudo -u mosaic -H git clone -q -b refactor /srv/mosaic/incoming/<sha>.bundle /srv/mosaic/src/mosaic-stack
+```
+
+Each upgrade, before `stack-upgrade.sh upgrade`:
+
+```sh
+echo "<bundle sha256>  /root/<sha>.bundle" | sha256sum -c -
+git bundle verify /root/<sha>.bundle
+git -C /root/mosaic-stack fetch -q /root/<sha>.bundle refs/heads/refactor:refs/remotes/origin/refactor
+git -C /root/mosaic-stack checkout -q --detach <sha>
+install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
+/root/mosaic-stack/scripts/stack-upgrade.sh upgrade --instance dev --bundle /srv/mosaic/incoming/<sha>.bundle --sha <sha>
+```
+
+The driver fetches the bundle into the stack's user's clone and makes the copy itself. It refuses
+`head-mismatch` when root's checkout isn't at `<sha>`. Usage, exits and the step log are in
+`docs/TOOLS.md`, under "Root's upgrade driver". Each run needs Mos's go, and `--migrate` needs Jason present
+(decision 96).
+
+When a run doesn't exit 0, send Mos the exit and the last lines of `/var/log/mosaic-stack/upgrade-dev.log`:
+- Exit 3: refused before anything stopped. The last line names the code. The stack's user's clone and the
+  new copy may already hold the new commit; the units still run the old one.
+- Exit 1: the old copy runs and passed its gate. Either the driver restarted it after A2 or a refused A3
+  (A2f), or it rolled back after A3, A4 or A5.
+- Exit 2: both units are stopped and the activation stays open, so the next run refuses at its plan until
+  it is finished with `confirm` or `revert`. One way here: a unit that doesn't stop within `--timeout`
+  keeps its old start time, A2f's gate fails it `stale-start`, and the driver stops both (duckworth-0055
+  N1). That is on purpose; a unit that won't stop is a reason to stop.
+
+Precheck lines: none. Root's checkout at `<sha>` and the stack's user's clone: precheck line missing.
+After install, `stack-copy.sh check --units external` reads the copy the units run.
+
+## 7. The args file
+
+`/etc/mosaic-stack/dev.args` holds the `case2-unit.sh` arguments, one per line, with the serve flags after a
+line holding `--`. I2's install writes it; ops-01 doesn't write it by hand. It must be root's, with no group
+or other write (`chmod 0644` or tighter). Before `--` it takes only `--user`, `--home`, `--business`,
+`--copy` (exactly once), `--node`, `--profile`, `--claude` and `--pm`. `stack-upgrade.sh` changes only the
+`--copy` line, and only after confirm passes.
+
+Precheck line: none. `stack-upgrade.sh` refuses `args-unreadable` on any of these rules.
+
+## 8. Enrollment, once per host
+
+The machine-grant enrollment is Gizmoduck's design r6, commit f9039682e, section 3, steps E1-E4. ops-01's
+Q-O1 is answered (mos-1125). The host side is the `tools/host/` files: `mosaic-keys`, `mosaic-keys-write`,
+their sudoers line and the sshd drop-in `/etc/ssh/sshd_config.d/<nn>-mosaic-keys.conf`. The verbs that run
+it (`hosts.enroll`, the `host-enroll` decision) are rows 234 and 235, not yet on refactor. This item has no
+commands until they land.
+
+## 9. Credential slots
+
+Jason places these, all in one sitting per host. No value appears here or in any message. Check a placed
+file with `stat -c '%U %a %n' PATH`, never `cat`. `<business>` is the host's business id.
+
+### .53, factory
+
+| Path | File | Mode | Owner | Profile | Precheck line |
+|---|---|---|---|---|---|
+| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `CLAUDE_CODE_OAUTH_TOKEN=...` (Q185) | 0600 | mosaic | factory | `claude replaces its prompt with --system-prompt-file` runs claude with it. The file's mode and owner: precheck line missing. |
+| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
+
+### .54, library
+
+| Path | File | Mode | Owner | Profile | Precheck line |
+|---|---|---|---|---|---|
+| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `ZAI_API_KEY=...` | 0600 | mosaic | library | `<path> sets ZAI_API_KEY` |
+| `/srv/mosaic/.config/mosaic-dev/secrets/` | the Console's proof file, the path the args file's `--proof-file` names | 0600 or 0400 | mosaic | library | none. `case2-unit.sh` refuses at render. Precheck line missing. |
+| `/srv/mosaic/.config/mosaic-dev/secrets/<business>/` | `library-gitea` | 0600 | mosaic | library | C2-13b's `library.sync` (not landed) |
+| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
+
+The installer key, `<configDir>/secrets/hosts/<business>/<hostId>/installer`, is not a slot anyone places.
+Enrollment (section 8) writes it under the enrolling stack's config directory and deletes it when the host
+leaves.
+
+## 10. Console start
+
+None by hand. I2's `case2-unit.sh` enables both units, starts the bus, then the Console (Q189, mos-1122).
+`stack-upgrade.sh` stops and starts both units itself.
@@ -2062,6 +2062,49 @@ test('R15 a rehearsal stopped by TERM, INT or HUP fails signal and still stops t
 }
 });

+// --- Row 233 (#1735): --backup last, the newest backup the log records (sage-1650, sage-1653) ---
+
+// The log's lines, as the runs write them, with only the fields from_log reads set.
+function rlog(e,...ls){
+ mkdirSync(e.state,{recursive:true});
+ writeFileSync(e.log,ls.map(([op,to,backup],i)=>JSON.stringify({at:`2026-10-20T00:00:0${i}Z`,op,from:null,to,backup,activation:`20261020T00000${i}Z-${to.slice(0,12)}`,result:'ok'})+'\n').join(''));
+}
+
+test('row 233: --backup last takes the newest backup the log records; a later line with backup null does not hide it, and the store line names it',async(t)=>{
+ const e=await rmk(t);
+ // An older backup that isn't on disk: a run that took it would fail backup-missing.
+ rlog(e,['activate',C1,`20261019T000000Z-${C1.slice(0,12)}`],['activate',B,e.act],['rollback',B,null]);
+ const l0=readFileSync(e.log,'utf8'),before=untouched(e);
+ const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','10');
+ assert.equal(r.code,0,r.out+r.err);
+ assert.deepEqual(r.names,['PASS backup-last',...PRE,'PASS migration','PASS gate','PASS no-session',...POST]);
+ assert.ok(r.out.startsWith(`PASS backup-last: the log's newest backup is ${e.act}\n`),r.out);
+ assert.ok(r.out.includes(`\nBACKUP_STORE ${e.act} schema 3g\n`),r.out);
+ assert.equal(readFileSync(e.log,'utf8'),l0);assert.equal(untouched(e),before);
+ // The rehearsal's own steps.log has the PASS and FAIL lines only.
+ assert.deepEqual(lines(join(r.dir,'steps.log')).map((l)=>l.replace(/^\S+ /,'')),r.out.split('\n').filter((l)=>/^(PASS|FAIL) |^REHEARSAL_/.test(l)));
+});
+
+test('row 233: --backup last with no backup in the log, or no log, exits 3 with no-backup and makes no rehearsal directory',async(t)=>{
+ const e=await rmk(t);
+ for(const how of ['no log','adopt only']){
+  if(how==='adopt only')rlog(e,['adopt',B,null]);
+  const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','3');
+  assert.equal(r.code,3,how+r.out+r.err);
+  assert.equal(r.out,'FAIL no-backup: the log records no backup\nREHEARSAL_FAIL\n',how);
+  assert.equal(r.dir,null);assert.deepEqual(rDirs(e),[],how);assert.ok(!existsSync(join(e.data,'rehearsals')),how);
+ }
+});
+
+test('row 233: --backup last refuses a newest backup that is not an activation id, with exit 1',async(t)=>{
+ const e=await rmk(t);
+ rlog(e,['activate',B,'../../etc']);
+ const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','3');
+ assert.equal(r.code,1,r.out+r.err);
+ assert.equal(r.out,'FAIL backup-unreadable: the log\'s newest backup is not an activation id\nREHEARSAL_FAIL\n');
+ assert.deepEqual(rDirs(e),[]);
+});
+
 // --- Row 190 (#1688): stack instances on one host. Design and the T-I table:
 // agents/darkwing/work/queue-190/DESIGN.md. T-I1, T-I2, T-I5, T-I10 and the loader half of T-I7 are in
 // stack-instances.test.mjs; these need the units, the stand-ins and stack-release.sh. Every existing
@@ -0,0 +1,449 @@
+// scripts/stack-upgrade.sh (row 233, I3, #1735): root's upgrade and rollback of a stack whose units are
+// root's, driving stack-release.sh --units external (row 232). Design: r6 sections 4 and 7,
+// agents/darkwing/work/queue-195/DESIGN.md, as the I3 brief changes them; deviations:
+// agents/darkwing/work/queue-233/BUILD.md. The stand-ins and the module fixture are
+// stack-release-fixture.mjs's. Root's sudo, case2-unit.sh render and row 240 hostcheck are stand-ins
+// here; the stack's user's steps are the real stack-release.sh from the user's clone, except the
+// rehearsal, which the fixture's stub copies can't pass (its own tests are in stack-release.test.mjs).
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import {execFileSync,spawn} from 'node:child_process';
+import {mkdirSync,writeFileSync,readFileSync,existsSync,readdirSync,rmSync,chmodSync,copyFileSync,symlinkSync,statSync} from 'node:fs';
+import {DatabaseSync} from 'node:sqlite';
+import {join,dirname} from 'node:path';
+import {once} from 'node:events';
+import {B,C,C1,U,COPIES,NODE,SBIN,base,fault,git,lines,logLines,mk,pointer,repo,sha,storeVersion,sysctl,units} from './stack-release-fixture.mjs';
+
+// --- Root's stand-ins. Each is serialized into its own file with K, its baked paths. ---
+
+// sudo -u USER -H env PATH=P CMD...: the form the driver must use, logged to sudo.log, then CMD with
+// SBIN before P. A one-shot hook arm/<key>.sh runs first and arm/<key>.id replaces CMD's ID. The
+// rehearsal prints a fixed BACKUP_STORE line and exits faults/rehearse-rc unless faults/real-rehearse.
+function fakeSudo(K){
+ const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
+ const a=process.argv.slice(2),sd=process.env.SD_STATE,f=(n)=>path.join(sd,'faults',n);
+ if(a[0]!=='-u'||!a[1]||a[2]!=='-H'||a[3]!=='env'||!/^PATH=/.test(a[4]||'')||a.length<6){console.error(`fake sudo: not the driver's form: ${a.join(' ')}`);process.exit(99)}
+ const P=a[4].slice(5),cmd=a.slice(5);
+ let key=path.basename(cmd[0]);
+ if(cmd[0].endsWith('/scripts/stack-release.sh')){
+  key=cmd[1];
+  if(['activate','rollback'].includes(key)&&cmd.includes('--plan'))key+='-plan';
+  if(key==='confirm'&&cmd.includes('--rollback'))key='confirm-rollback';
+ }else if(cmd[0]==='git')key=`git-${cmd[3]}`;
+ fs.appendFileSync(path.join(sd,'sudo.log'),JSON.stringify({user:a[1],path:P,key,cmd})+'\n');
+ const env={...process.env,PATH:`${K.sbin}:${P}`},arm=(n)=>path.join(sd,'arm',`${key}.${n}`);
+ if(fs.existsSync(arm('sh'))){const run=arm('sh')+'.ran';fs.renameSync(arm('sh'),run);cp.spawnSync('bash',[run],{env,stdio:'inherit'})}
+ if(fs.existsSync(arm('id'))){cmd[2]=fs.readFileSync(arm('id'),'utf8').trim();fs.rmSync(arm('id'))}
+ if(key==='rehearse'&&!fs.existsSync(f('real-rehearse'))){
+  console.log(`fixture rehearse: ${cmd.slice(1).join(' ')}`);
+  let line='BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g';try{line=fs.readFileSync(f('rehearse-store-line'),'utf8').replace(/\n$/,'')}catch{}
+  if(line)console.log(line);
+  let rc=0;try{rc=Number(fs.readFileSync(f('rehearse-rc'),'utf8'))}catch{}
+  process.exit(rc);
+ }
+ const r=cp.spawnSync(cmd[0],cmd.slice(1),{env,stdio:'inherit'});process.exit(r.status??1);
+}
+
+// case2-unit.sh: logs its argv, then installs the copy's units as the fixture's user units (its
+// WorkingDirectory under the stack's copies) and starts the bus, then the Console, as case2-unit.sh
+// does. faults/render-fails-<copy> exits 1 first; faults/render-probe-<copy> (one-shot) first runs
+// probe.sh, a second driver, and records its exit and stderr.
+function fakeRender(K){
+ const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
+ const a=process.argv.slice(2),sd=process.env.SD_STATE,f=(n)=>path.join(sd,'faults',n);
+ fs.appendFileSync(path.join(sd,'render.log'),JSON.stringify(a)+'\n');
+ const cut=a.indexOf('--'),pre=a.slice(0,cut),serve=a.slice(cut+1),get=(k)=>pre[pre.indexOf(k)+1];
+ const copy=get('--copy'),copies=path.join(get('--home'),'.local','share','mosaic-stack','copies');
+ if(fs.existsSync(f(`render-probe-${copy}`))){
+  fs.rmSync(f(`render-probe-${copy}`));
+  const p=cp.spawnSync('bash',[path.join(sd,'probe.sh')],{encoding:'utf8'});
+  fs.writeFileSync(path.join(sd,'probe.json'),JSON.stringify({code:p.status,err:p.stderr}));
+ }
+ if(fs.existsSync(f(`render-fails-${copy}`))){console.error(`fixture render: ${copy} fails`);process.exit(1)}
+ const sh=(n,...x)=>cp.execFileSync('bash',[path.join(K.repo,'scripts',n),...x],{stdio:['ignore','ignore','inherit']});
+ sh('bus-service.sh','install','--copy',copy,'--copies',copies,...(pre.includes('--pm')?['--pm']:[]),'--claude',get('--claude'));
+ sh('webui-service.sh','install','--copy',copy,'--copies',copies,...serve);
+ for(const u of [`mosaic-bus@${get('--business')}.service`,'mosaic-webui.service'])cp.execFileSync(path.join(K.sbin,'systemctl'),['start',u],{stdio:'inherit'});
+}
+
+// Root's systemctl: the fixture's, with systemd's no-op start of an active unit, and a one-shot
+// faults/stop-noop-<unit> that leaves the unit running, as a stop past its timeout would.
+function fakeSystemctl(K){
+ const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
+ const a=process.argv.slice(2),sd=process.env.SD_STATE;
+ fs.appendFileSync(path.join(sd,'root-systemctl.log'),a.join(' ')+'\n');
+ const noop=path.join(sd,'faults',`stop-noop-${a[1]}`);
+ if(a[0]==='stop'&&fs.existsSync(noop)){fs.rmSync(noop);process.exit(0)}
+ if(a[0]==='start'){let s={};try{s=JSON.parse(fs.readFileSync(path.join(sd,'units.json'),'utf8'))}catch{}if((s[a[1]]||{}).state==='active')process.exit(0)}
+ const r=cp.spawnSync(path.join(K.sbin,'systemctl'),a,{stdio:'inherit'});process.exit(r.status??1);
+}
+
+const FAKES=join(base,'upgrade-fakes');mkdirSync(FAKES);
+const K={sbin:SBIN,repo};
+for(const [n,fn] of [['sudo',fakeSudo],['render',fakeRender],['systemctl',fakeSystemctl]])writeFileSync(join(FAKES,n),`#!${NODE}\n(${fn})(${JSON.stringify(K)});\n`,{mode:0o755});
+writeFileSync(join(FAKES,'hostcheck'),'#!/bin/sh\necho "$*" >> "$SD_STATE/hostcheck.log"\nexit "$(cat "$SD_STATE/faults/hostcheck-rc" 2>/dev/null || echo 0)"\n',{mode:0o755});
+const BUNDLE=join(base,'fixture.bundle');git(repo,'bundle','create','-q',BUNDLE,'refactor');
+// Root's checkout of a reviewed commit: a clone of the fixture repo, detached at it.
+const checkouts=new Map();
+const checkout=(id)=>{
+ if(!checkouts.has(id)){const d=join(base,`root-${id.slice(0,12)}`);git(base,'clone','-q','--no-checkout',repo,d);git(d,'checkout','-q','--detach',id);checkouts.set(id,d)}
+ return checkouts.get(id);
+};
+const TO=['--timeout','3'];
+
+// --- Per-test stack ---
+
+// The stack's user's home: its copies (the fixture's, through a link), its clone, its bin. Both units
+// installed on `copy` and active, adopted under --units external, and the I2 args file naming it.
+async function useed(t,{copy=B}={}){
+ const e=await mk(t),home=e.env.HOME;
+ e.hc=join(home,'.local','share','mosaic-stack','copies');mkdirSync(dirname(e.hc),{recursive:true});symlinkSync(COPIES,e.hc);
+ const bin=join(home,'.npm-global','bin');mkdirSync(bin,{recursive:true});
+ writeFileSync(join(bin,'mosaic'),readFileSync(join(e.bin,'mosaic'),'utf8').replaceAll(`${e.bin}/`,`${bin}/`),{mode:0o755});
+ copyFileSync(join(e.bin,'mosaic-old'),join(bin,'mosaic-old'));chmodSync(join(bin,'mosaic-old'),0o755);
+ e.bin=bin;e.clone=join(home,'src','mosaic-stack');git(base,'clone','-q',repo,e.clone);
+ const db=new DatabaseSync(e.store);
+ db.exec('PRAGMA journal_mode=WAL');
+ db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
+ db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
+ db.prepare("INSERT INTO meta VALUES ('schema_version', '3g')").run();
+ for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`);
+ db.close();
+ const sh=(n,...a)=>execFileSync('bash',[join(e.clone,'scripts',n),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
+ sh('bus-service.sh','install','--copy',copy,'--copies',e.hc,'--claude',e.claude);
+ sh('webui-service.sh','install','--copy',copy,'--copies',e.hc,'--port',e.port,'--business','acme');
+ sh('stack-copy.sh','wrapper',copy,'--copies',e.hc,'--bin',e.bin);
+ execFileSync(join(SBIN,'systemctl'),['start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:'120'}});
+ sysctl(e,'start',e.con);
+ sh('stack-release.sh','adopt','acme','--units','external','--copies',e.hc,'--bin',e.bin);
+ e.etc=join(e.root,'etc');mkdirSync(e.etc);e.args=join(e.etc,'dev.args');
+ e.argsText=`--user\nfixture\n--home\n${home}\n--business\nacme\n--copy\n${copy}\n--node\n${NODE}\n--claude\n${e.claude}\n--\n--port\n${e.port}\n--business\nacme\n`;
+ writeFileSync(e.args,e.argsText);chmodSync(e.args,0o640);
+ e.run=join(e.root,'run');e.slog=join(e.root,'log','upgrade-dev.log');e.lockf=join(e.run,'upgrade-dev.lock');
+ mkdirSync(join(e.sd,'arm'));
+ e.denv={...e.env,STACK_UPGRADE_ETC:e.etc,STACK_UPGRADE_RUN:e.run,STACK_UPGRADE_LOG_DIR:dirname(e.slog),STACK_UPGRADE_UNIT_DIR:e.units,STACK_UPGRADE_HOSTCHECK:join(FAKES,'hostcheck'),STACK_UPGRADE_RENDER:join(FAKES,'render'),STACK_UPGRADE_SUDO:join(FAKES,'sudo'),STACK_UPGRADE_SYSTEMCTL:join(FAKES,'systemctl')};
+ writeFileSync(join(e.sd,'probe.sh'),`exec bash ${join(checkout(C1),'scripts','stack-upgrade.sh')} upgrade --instance dev --bundle ${BUNDLE} --sha ${C1} --timeout 3\n`);
+ writeFileSync(join(e.sd,'snap.jsonl'),'');
+ return e;
+}
+
+// The driver from root's checkout of `at`.
+async function drive(e,at,...args){
+ const ch=spawn('bash',[join(checkout(at),'scripts','stack-upgrade.sh'),...args],{env:e.denv,stdio:['ignore','pipe','pipe']});
+ let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d});
+ const timer=setTimeout(()=>ch.kill('SIGKILL'),180000);
+ const [code]=await once(ch,'close');clearTimeout(timer);
+ return {code,out,err,all:out+err};
+}
+const upgrade=(e,id,...a)=>drive(e,id,'upgrade','--instance','dev','--bundle',BUNDLE,'--sha',id,...TO,...a);
+const arm=(e,key,text)=>writeFileSync(join(e.sd,'arm',key),text);
+
+// The step log: every line `<time> <step> rc=N ...` or the result line, root:root 0640 (here the
+// test user's), and nothing the stack's user printed. Returns the lines without the time.
+const STEP=/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ (start|fetch|ff|copy|hostcheck|units|plan|rehearse|rehearse-store|rehearsal-skipped|stop-console|stop-bus|wait-down|stage|render|confirm|args|revert|render-old|confirm-rollback|gate|start-bus|start-console|stop-units) rc=\d+ |^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ result exit=[0-3] [a-z-]+ /;
+function slog(e){
+ const l=lines(e.slog);
+ for(const x of l){assert.match(x,STEP);assert.ok(!/stack-release:|fixture rehearse|PASS |FAIL |REHEARSAL/.test(x),x)}
+ assert.equal(statSync(e.slog).mode&0o777,0o640);
+ return l.map((x)=>x.slice(21));
+}
+const names=(e)=>slog(e).map((x)=>x.split(' ')[0]);
+const result=(e)=>slog(e).at(-1);
+const sudoKeys=(e)=>lines(join(e.sd,'sudo.log')).map((l)=>JSON.parse(l).key);
+const lastAct=(e)=>logLines(e).filter((o)=>o.result==='started').at(-1).activation;
+const mine=(e,a)=>logLines(e).filter((o)=>o.activation===a);
+const open=(e)=>{const m=new Set();for(const o of logLines(e)){if(!o.activation)continue;if(o.result==='started')m.add(o.activation);else m.delete(o.activation)}return [...m]};
+const final=(e,a)=>{const [s,f,...more]=mine(e,a);assert.equal(s.result,'started');assert.equal(more.length,0);return f};
+const live=(e)=>[units(e)[e.bus].state,units(e)[e.con].state,units(e)[e.bus].wd.split('/').at(-1),units(e)[e.con].wd.split('/').at(-1)];
+// Nothing the run did reached the args file: the same bytes and mode, and no temporary file left.
+const argsKept=(e)=>{assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args'])};
+const argsCopy=(e)=>{const l=readFileSync(e.args,'utf8').split('\n');return l[l.indexOf('--copy')+1]};
+const PRE=['start','fetch','ff','copy','hostcheck','units','plan','rehearse','rehearse-store'];
+const A2=['stop-console','stop-bus','wait-down','wait-down'];
+const R=[...A2,'revert','render-old','confirm-rollback'];
+
+// --- r6 sequences 1-8, through the driver ---
+
+test('row 233: sequence 1: confirm fails healthz, the driver reverts, renders the old copy and confirm --rollback --cause A5 passes: exit 1',async(t)=>{
+ const e=await useed(t);
+ fault(e,`healthz-${C1}`);
+ const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
+ assert.equal(r.err.split('\n').at(-2),`stack-upgrade: failed: A5 failed; rolled back to ${B}, which passed the gate`);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm',...R,'result']);
+ assert.equal(result(e),`result exit=1 failed A5 failed; rolled back to ${B}, which passed the gate`);
+ const a=lastAct(e),l=final(e,a);assert.deepEqual([l.op,l.result,l.code,l.rollback,l.from,l.to],['activate','gate-failed','healthz','ok',B,C1]);
+ assert.deepEqual(open(e),[]);assert.deepEqual(live(e),['active','active',B,B]);
+ argsKept(e);
+ // The stack's user's steps went through sudo in the driver's form, as fixture, the pinned node first.
+ for(const l of lines(join(e.sd,'sudo.log')).map((x)=>JSON.parse(x)))assert.deepEqual([l.user,l.path],['fixture',`${dirname(NODE)}:/usr/local/bin:/usr/bin:/bin`]);
+ assert.deepEqual(sudoKeys(e),['git-fetch','git-merge','stack-copy.sh','activate-plan','rehearse','activate','confirm','revert','confirm-rollback']);
+});
+
+test('row 233: sequence 2: activate --migrate fails at the wrapper, revert restores the store, confirm --rollback --cause A3: exit 1',async(t)=>{
+ const e=await useed(t);
+ const s0=sha(e.store);
+ arm(e,'activate.sh',`chmod 0555 ${e.bin}\n`);arm(e,'revert.sh',`chmod 0755 ${e.bin}\n`);
+ const r=await upgrade(e,C,'--migrate');assert.equal(r.code,1,r.all);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage',...R,'result']);
+ assert.equal(result(e),`result exit=1 failed A3 failed; rolled back to ${B}, which passed the gate`);
+ const l=final(e,lastAct(e));assert.deepEqual([l.result,l.code,l.rollback],['failed','wrapper','ok']);
+ assert.equal(storeVersion(e.store,true),'3g');assert.deepEqual(live(e),['active','active',B,B]);
+ // The bus holds the store open again; its bytes are the backup's once it stops.
+ sysctl(e,'stop',e.bus);assert.equal(sha(e.store),s0);
+ argsKept(e);
+});
+
+test('row 233: sequence 3: root\'s render of the new copy fails, the driver rolls back with --cause A4, and holds the lock to the end: exit 1',async(t)=>{
+ const e=await useed(t);
+ fault(e,`render-fails-${C1}`);fault(e,`render-probe-${B}`);
+ const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
+ // The probe ran inside render-old, during the rollback, and its one line is its refusal.
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render',...A2,'revert','result','render-old','confirm-rollback','result']);
+ const p=JSON.parse(readFileSync(join(e.sd,'probe.json'),'utf8'));assert.equal(p.code,3,p.err);
+ assert.equal(p.err,`stack-upgrade: upgrade-running: another stack-upgrade.sh run holds ${e.lockf} (nothing was stopped)\n`);
+ assert.match(slog(e).at(-4),/^result exit=3 upgrade-running /);
+ assert.match(slog(e).find((x)=>x.startsWith('render ')),/^render rc=1 /);
+ const l=final(e,lastAct(e));assert.deepEqual([l.result,l.code,l.rollback],['failed','units-start','ok']);
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: sequence 4: a capture failure is exit 3 from the stage, and A2f restarts the old units and gates them: exit 1',async(t)=>{
+ const e=await useed(t);
+ arm(e,'activate.sh',`: > ${join(e.sd,'faults','no-fragment')}\n`);arm(e,'gate.sh',`rm ${join(e.sd,'faults','no-fragment')}\n`);
+ const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','start-bus','start-console','gate','result']);
+ assert.match(slog(e).find((x)=>x.startsWith('stage ')),/^stage rc=3 /);
+ assert.equal(result(e),`result exit=1 failed A3 failed; ${B} is running again and passed the gate`);
+ const a=lastAct(e);assert.deepEqual(mine(e,a).map((o)=>[o.result,o.code]),[['started',undefined],['refused','capture-failed']]);
+ assert.deepEqual(open(e),[]);assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: N1: a bus that outlives A2\'s timeout keeps its old start, so A2f\'s gate fails stale-start: exit 2, both units stopped',async(t)=>{
+ const e=await useed(t);
+ fault(e,`stop-noop-${e.bus}`);
+ const r=await upgrade(e,C1);assert.equal(r.code,2,r.all);
+ assert.deepEqual(names(e),[...PRE,...A2,'start-bus','start-console','gate','stop-units','result']);
+ assert.ok(slog(e).includes(`wait-down rc=1 unit: ${e.bus} state: active`),slog(e).join('\n'));
+ assert.ok(r.out.includes(`gate failed on ${B}: stale-start`),r.out);
+ assert.equal(result(e),`result exit=2 rollback-failed A2 failed and ${B} failed its gate after the restart; both units are stopped`);
+ assert.deepEqual(live(e).slice(0,2),['inactive','inactive']);
+ assert.ok(!sudoKeys(e).includes('activate'));argsKept(e);
+});
+
+test('row 233: sequence 5: confirm of the wrong copy is refused, and the driver rolls back with --cause A5: exit 1',async(t)=>{
+ const e=await useed(t);
+ arm(e,'confirm.id',B);
+ const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
+ assert.ok(r.out.includes('refusing open-activation')||r.err.includes('refusing open-activation'),r.all);
+ assert.match(slog(e).find((x)=>x.startsWith('confirm ')),/^confirm rc=3 /);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm',...R,'result']);
+ const l=final(e,lastAct(e));assert.deepEqual([l.result,l.code,l.rollback],['gate-failed','unrecorded','ok']);
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: sequence 6: a revert that fails stops both units at exit 2, and the next run refuses at its --plan with nothing stopped',async(t)=>{
+ const e=await useed(t);
+ fault(e,`render-fails-${C1}`);
+ arm(e,'revert.sh',`for d in ${join(e.state,'stack-rollback')}/*/; do [ -f "$d/AWAITING-UNITS" ] && rm "$d/wrapper"; done\n`);
+ const r=await upgrade(e,C1);assert.equal(r.code,2,r.all);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render',...A2,'revert','stop-units','result']);
+ assert.equal(result(e),`result exit=2 rollback-failed A4 failed and revert failed; both units are stopped`);
+ const a=lastAct(e);assert.deepEqual(open(e),[a]);assert.deepEqual(live(e).slice(0,2),['inactive','inactive']);
+ argsKept(e);
+ const n=slog(e).length,sc=lines(join(e.sd,'root-systemctl.log')).length;
+ rmSync(join(e.sd,'faults',`render-fails-${C1}`));
+ const again=await upgrade(e,C1);assert.equal(again.code,3,again.all);
+ assert.ok(again.out.includes(`refuses interrupted-run: activation ${a} started and has no final line`),again.out);
+ assert.deepEqual(names(e).slice(n),PRE.slice(0,7).concat('result'));
+ assert.match(result(e),/^result exit=3 plan-refused /);
+ assert.equal(lines(join(e.sd,'root-systemctl.log')).length,sc);argsKept(e);
+});
+
+test('row 233: sequence 7: two upgrades and a rollback --to the previous copy with --restore, the args file following each confirm',async(t)=>{
+ const e=await useed(t);
+ // The args file at confirm still names the old copy; the new one moves in after it.
+ arm(e,'confirm.sh',`cp ${e.args} ${join(e.sd,'args-at-confirm')}\n`);
+ const one=await upgrade(e,C1);assert.equal(one.code,0,one.all);
+ assert.equal(one.out.split('\n').at(-2),`stack-upgrade: upgrade ok: ${C1} is live for acme`);
+ assert.equal(readFileSync(join(e.sd,'args-at-confirm'),'utf8'),e.argsText);
+ assert.equal(readFileSync(e.args,'utf8'),e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`));
+ assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args']);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm','args','result']);
+ assert.ok(slog(e).includes(`args rc=0 copy: ${C1}`));
+ const a1=lastAct(e);
+ const two=await upgrade(e,C,'--migrate');assert.equal(two.code,0,two.all);
+ const a2=lastAct(e);assert.equal(argsCopy(e),C);
+ const l1=final(e,a1),l2=final(e,a2);
+ assert.deepEqual([l1.op,l1.result,l1.from,l1.to],['activate','ok',B,C1]);
+ assert.deepEqual([l2.op,l2.result,l2.from,l2.to,l2.backup],['activate','ok',C1,C,a2]);
+ assert.equal(storeVersion(e.store,true),'3h');
+ const n=slog(e).length;
+ const rb=await drive(e,C,'rollback','--instance','dev','--to',C1,'--restore',a2,...TO);assert.equal(rb.code,0,rb.all);
+ assert.equal(rb.out.split('\n').at(-2),`stack-upgrade: rollback ok: ${C1} is live for acme`);
+ // A rollback fetches, copies and rehearses nothing.
+ assert.deepEqual(names(e).slice(n),['start','hostcheck','units','plan',...A2,'stage','render','confirm','args','result']);
+ const ra=lastAct(e),l3=final(e,ra);assert.deepEqual([l3.op,l3.result,l3.from,l3.to,l3.backup],['rollback','ok',C,C1,a2]);
+ assert.deepEqual([pointer(e).copy,pointer(e).schema],[C1,'3g']);assert.equal(storeVersion(e.store,true),'3g');
+ assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
+ const m=slog(e).length,sc=lines(join(e.sd,'root-systemctl.log')).length;
+ const no=await drive(e,C,'rollback','--instance','dev','--to',B,...TO);assert.equal(no.code,3,no.all);
+ assert.ok(no.all.includes(`refusing to-not-previous: --to ${B} is not the previous copy; the log names ${C}`),no.all);
+ assert.deepEqual(names(e).slice(m),['start','hostcheck','units','plan','result']);
+ assert.equal(lines(join(e.sd,'root-systemctl.log')).length,sc);assert.equal(argsCopy(e),C1);
+});
+
+test('row 233: sequence 8: a migration that writes and fails: the driver reverts to the backup byte for byte, --cause A3: exit 1',async(t)=>{
+ const e=await useed(t);
+ const s0=sha(e.store);
+ fault(e,'migrate-fails');
+ const r=await upgrade(e,C,'--migrate');assert.equal(r.code,1,r.all);
+ assert.match(slog(e).find((x)=>x.startsWith('stage ')),/^stage rc=1 /);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage',...R,'result']);
+ const a=lastAct(e),l=final(e,a);assert.deepEqual([l.result,l.code,l.rollback],['failed','migrate-failed','ok']);
+ sysctl(e,'stop',e.bus);assert.equal(sha(e.store),s0);assert.equal(storeVersion(e.store,false),'3g');
+ argsKept(e);
+});
+
+// --- The lock, A1's refusals and the rehearsal ---
+
+test('row 233: a second run while the lock is held refuses upgrade-running with one log line and no call',async(t)=>{
+ const e=await useed(t);
+ mkdirSync(e.run,{recursive:true});
+ const holder=spawn('flock',[e.lockf,'sleep','60'],{detached:true,stdio:'ignore'});
+ t.after(()=>{try{process.kill(-holder.pid,'SIGKILL')}catch{}});
+ for(let i=0;i<100&&execFileSync('bash',['-c','flock -n "$1" true && echo free || echo held','_',e.lockf],{encoding:'utf8'}).trim()==='free';i++)await new Promise((r)=>setTimeout(r,50));
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.equal(r.err,`stack-upgrade: upgrade-running: another stack-upgrade.sh run holds ${e.lockf} (nothing was stopped)\n`);
+ assert.deepEqual(names(e),['result']);
+ assert.ok(!existsSync(join(e.sd,'sudo.log')));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: A1 refuses open-activation on any AWAITING-UNITS file, through the stack\'s user\'s --plan, with nothing stopped',async(t)=>{
+ const e=await useed(t);
+ const fake=join(e.state,'stack-rollback',`20000101T000000Z-${C1.slice(0,12)}`);mkdirSync(fake,{recursive:true});writeFileSync(join(fake,'AWAITING-UNITS'),`${C1}\n`);
+ const logs=logLines(e).length;
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes('refuses open-activation'),r.out);
+ assert.deepEqual(names(e),[...PRE.slice(0,7),'result']);assert.match(result(e),/^result exit=3 plan-refused /);
+ assert.equal(logLines(e).length,logs);assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: a target that migrates refuses migrate-required without --migrate, with nothing stopped',async(t)=>{
+ const e=await useed(t);
+ const r=await upgrade(e,C);assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes('refuses migrate-required'),r.out);
+ assert.deepEqual(names(e),[...PRE.slice(0,7),'result']);assert.match(result(e),/^result exit=3 plan-refused /);
+ assert.ok(!sudoKeys(e).includes('rehearse'));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: the rehearsal runs after the plan and before A2; a failed one refuses rehearsal-failed with nothing stopped',async(t)=>{
+ const e=await useed(t);
+ fault(e,'rehearse-rc','1');
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes('fixture rehearse: '),r.out);
+ assert.deepEqual(names(e),[...PRE,'result']);
+ assert.equal(result(e),'result exit=3 rehearsal-failed the rehearsal exited 1 (nothing was stopped)');
+ const rh=lines(join(e.sd,'sudo.log')).map((l)=>JSON.parse(l)).find((l)=>l.key==='rehearse');
+ assert.deepEqual(rh.cmd.slice(1),['rehearse',C1,'acme','--copies',e.hc,'--backup','last','--port','7339','--timeout','3']);
+ assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: the rehearsal\'s BACKUP_STORE line goes to the step log after a match, a malformed one as none, the rest of its output never',async(t)=>{
+ const e=await useed(t);
+ fault(e,'rehearse-rc','1');
+ const r=await upgrade(e,C1,'--rehearse-port','7449');assert.equal(r.code,3,r.all);
+ assert.ok(slog(e).includes('rehearse-store rc=0 BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g'),slog(e).join('\n'));
+ assert.ok(slog(e).find((x)=>x.startsWith('rehearse ')).includes(' --port 7449 '));
+ for(const bad of ['BACKUP_STORE ../../etc schema 3g','BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g; rm -rf /','stack-release: BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g','']){
+  fault(e,'rehearse-store-line',bad);
+  const n=slog(e).length;
+  const b=await upgrade(e,C1);assert.equal(b.code,3,b.all);
+  assert.deepEqual(slog(e).slice(n).filter((x)=>x.startsWith('rehearse-store ')),['rehearse-store rc=0 BACKUP_STORE none'],bad);
+ }
+});
+
+test('row 233: a real rehearsal with no backup in the log refuses no-backup, and goes on only with --no-rehearsal, which logs the skip',async(t)=>{
+ const e=await useed(t);
+ fault(e,'real-rehearse');
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes('FAIL no-backup: the log records no backup\nREHEARSAL_FAIL\n'),r.out);
+ assert.equal(result(e),'result exit=3 no-backup the log records no backup to rehearse against; only a go that names --no-rehearsal goes on (nothing was stopped)');
+ assert.ok(slog(e).includes('rehearse-store rc=0 BACKUP_STORE none'));
+ assert.ok(!existsSync(join(e.data,'rehearsals')));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
+ const n=slog(e).length;
+ const s=await upgrade(e,C1,'--no-rehearsal');assert.equal(s.code,0,s.all);
+ assert.deepEqual(names(e).slice(n),[...PRE,'rehearsal-skipped',...A2,'stage','render','confirm','args','result']);
+ assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
+});
+
+test('row 233: a real rehearsal after an upgrade names the log\'s newest backup; --no-rehearsal with a backup refuses rehearsal-required',async(t)=>{
+ const e=await useed(t);
+ const one=await upgrade(e,C1);assert.equal(one.code,0,one.all);
+ const a1=lastAct(e);
+ fault(e,'real-rehearse');
+ const n=slog(e).length;
+ // The stub copies fail the rehearsal right after it names the backup (copy-without-no-launch).
+ const r=await upgrade(e,C,'--migrate');assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes(`PASS backup-last: the log's newest backup is ${a1}\n`),r.out);
+ assert.deepEqual(names(e).slice(n),[...PRE,'result']);assert.match(result(e),/^result exit=3 rehearsal-failed the rehearsal exited 1 /);
+ rmSync(join(e.sd,'faults','real-rehearse'));
+ const m=slog(e).length;
+ const q=await upgrade(e,C,'--migrate','--no-rehearsal');assert.equal(q.code,3,q.all);
+ assert.deepEqual(names(e).slice(m),[...PRE,'result']);
+ assert.equal(result(e),'result exit=3 rehearsal-required the log records a backup and the rehearsal ran; --no-rehearsal is only for a stack with none (nothing was stopped)');
+ assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
+});
+
+// --- Before A1 and the args file ---
+
+test('row 233: the args file must be the running user\'s, with no group or other write, one --copy, and the copy the units run',async(t)=>{
+ const e=await useed(t);
+ const tryArgs=async(text,mode,code,why)=>{
+  writeFileSync(e.args,text);chmodSync(e.args,mode);
+  const n=existsSync(e.slog)?slog(e).length:0;
+  const r=await upgrade(e,C1);assert.equal(r.code,3,`${why}: ${r.all}`);
+  assert.match(slog(e).at(-1),new RegExp(`^result exit=3 ${code} `),why);
+  assert.ok(!slog(e).slice(n).some((x)=>/^(stop|stage|render|confirm)/.test(x)),why);
+ };
+ await tryArgs(e.argsText,0o660,'args-unreadable','group write');
+ await tryArgs(e.argsText,0o642,'args-unreadable','other write');
+ await tryArgs(e.argsText.replace('--\n',`--copy\n${C1}\n--\n`),0o640,'args-unreadable','two --copy');
+ await tryArgs(e.argsText.replace('--\n','--tester\n--\n'),0o640,'args-unreadable','an unknown flag');
+ await tryArgs(e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`),0o640,'args-differ','the units run B');
+ rmSync(e.args);symlinkSync(join(e.sd,'probe.sh'),e.args);
+ const l=await upgrade(e,C1);assert.equal(l.code,3);assert.match(result(e),/^result exit=3 args-unreadable /);
+ assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);
+});
+
+test('row 233: hostcheck (row 240): a nonzero --check or a missing file refuses profile-drift before the plan',async(t)=>{
+ const e=await useed(t);
+ fault(e,'hostcheck-rc','1');
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.match(result(e),/^result exit=3 profile-drift /);
+ assert.deepEqual(lines(join(e.sd,'hostcheck.log')),['--check']);
+ assert.deepEqual(sudoKeys(e),['git-fetch','git-merge','stack-copy.sh']);
+ e.denv.STACK_UPGRADE_HOSTCHECK=join(e.root,'no-hostcheck');
+ const m=await upgrade(e,C1);assert.equal(m.code,3,m.all);
+ assert.equal(result(e),`result exit=3 profile-drift no executable ${join(e.root,'no-hostcheck')} (row 240) (nothing was stopped)`);
+ assert.ok(!sudoKeys(e).includes('activate-plan'));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));argsKept(e);
+});
+
+test('row 233: the checkout must be at --sha (head-mismatch), and a commit the clone can\'t fast-forward to refuses fetch-failed',async(t)=>{
+ const e=await useed(t);
+ const r=await drive(e,C1,'upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C,'--migrate',...TO);assert.equal(r.code,3,r.all);
+ assert.equal(result(e),`result exit=3 head-mismatch this checkout, ${checkout(C1)}, is at ${C1}, not ${C} (nothing was stopped)`);
+ const u=await drive(e,U,'upgrade','--instance','dev','--bundle',BUNDLE,'--sha',U,...TO);assert.equal(u.code,3,u.all);
+ assert.match(result(e),/^result exit=3 fetch-failed .* fast-forward /);
+ assert.ok(!sudoKeys(e).includes('activate-plan'));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));argsKept(e);
+});
+
+test('row 233: usage exits 4 before any log or lock',async(t)=>{
+ const e=await useed(t);
+ for(const a of [[],['status'],['upgrade','--instance','dev','--bundle',BUNDLE],['upgrade','--instance','other','--bundle',BUNDLE,'--sha',C1],['upgrade','--instance','dev','--bundle','rel/x','--sha',C1],['upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C1.slice(0,12)],['upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C1,'--to',B],['upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C1,'--rehearse-port','80'],['rollback','--instance','dev','--to',B,'--migrate'],['rollback','--instance','dev','--to',B,'--events','3'],['rollback','--instance','dev'],['rollback','--instance','dev','--to',B,'--restore','x']]){
+  const r=await drive(e,C1,...a);assert.equal(r.code,4,a.join(' '));assert.match(r.err,/^usage: stack-upgrade.sh upgrade /,a.join(' '));
+ }
+ assert.ok(!existsSync(dirname(e.slog)));assert.ok(!existsSync(e.run));
+});
@@ -130,7 +130,7 @@ parse_opts() {
            --sso) [ "$OP" = activate ] || usage; SSO=1; shift ;;
            --migrate) [ "$OP" = activate ] || usage; MIGRATE=1; shift ;;
            --restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || usage; RESTORE="$2"; shift 2 ;;
-            --backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
+            --backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [ "$2" = last ] || [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
            --port) [ "$OP" = rehearse ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{1,5}$ ]] && [ "$((10#$2))" -ge 1024 ] && [ "$((10#$2))" -le 65535 ] || usage; R_PORT="$((10#$2))"; shift 2 ;;
            --events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
            --units) [ "$OP" != rehearse ] && [ $# -ge 2 ] && { [ "$2" = user ] || [ "$2" = external ]; } || usage; UNITS="$2"; shift 2 ;;
@@ -220,7 +220,8 @@ for(const k of open.keys())process.stdout.write(k+"\n");' -- "$LOG"

 # Reads the log's ok lines, newest first. `previous` prints rollback's target: the first `to`
 # that is not the pointer's copy (release.sh's rule). `backup` prints the backup field of the
-# line that made the pointer's copy live, the backup a rollback would restore. `arrival` prints the
+# line that made the pointer's copy live, the backup a rollback would restore. `newest` prints the
+# backup field of the newest line that has one, rehearse's --backup last (row 233). `arrival` prints the
 # activation and `from` of the line that moved the stack onto the pointer's copy from another one,
 # past any same-copy re-pin after it (row 200, sage-1389).
 from_log() {
@@ -233,6 +234,7 @@ for(const l of lines.reverse()){
  if(o.result!=="ok"||!["adopt","activate","rollback"].includes(o.op)||!o.to)continue;
  if(mode==="previous"&&o.to!==current){process.stdout.write(o.to+"\n");process.exit(0)}
  if(mode==="backup"&&o.to===current){if(o.backup)process.stdout.write(o.backup+"\n");process.exit(o.backup?0:1)}
+  if(mode==="newest"&&o.backup){process.stdout.write(o.backup+"\n");process.exit(0)}
  if(mode==="arrival"&&o.to===current&&o.from!==current){process.stdout.write(`${o.activation??""} ${o.from??""}\n`);process.exit(0)}
 }
 process.exit(1);' -- "$LOG" "$1" "$P_COPY"
@@ -1262,6 +1264,8 @@ plan() {
        echo "needs: --migrate$([ "$MIGRATE" = 1 ] && echo ' (given)' || echo ' (not given; the run would roll back with migration-needs-flag)')"
        echo "needs: Jason present for this run (decision 96 condition 2)"
        [ "$MIGRATE" = 1 ] || early=1
+        # Row 233: root reads only the plan's exit, so under external a missing --migrate refuses here.
+        [ "$UNITS" = user ] || [ "$MIGRATE" = 1 ] || PLAN_REFUSE+=("migrate-required: the store is $S_SCHEMA and $TO migrates it to $T_SCHEMA; the go names --migrate")
    elif [ "$OP" = activate ]; then
        echo "migration: none possible; $S_SCHEMA is not in $T_SCHEMA's migration chain, so the run would roll back with store-newer"
        early=1
@@ -1452,6 +1456,13 @@ ext_pointer_checks() {
    local open
    open="$(interrupted | paste -sd' ' -)"
    [ -z "$open" ] || check_refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)"
+    # Row 233: root can't read the stack's state, so the plan lists any rollback set still waiting
+    # for units, whatever the log says.
+    local m sets=""
+    local -a markers
+    mapfile -t markers < <(compgen -G "$STATE/stack-rollback/*/AWAITING-UNITS" || true)
+    for m in "${markers[@]}"; do sets+=" $(basename "$(dirname "$m")")"; done
+    [ -z "$sets" ] || check_refuse open-activation "rollback set(s)$sets hold AWAITING-UNITS; root finishes them with confirm or revert first"
 }

 # A3's checks before `started`. Root has stopped the units and installs them after this run, so
@@ -1973,6 +1984,9 @@ r_env() { env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json"
 r_migration() {
    local copy="$COPIES/$TO" old="$COPIES/$P_COPY" v out rc
    v="$(node -e "$STORE_JS" -- version "$R/data/bus/bus.sqlite" 2>&1)" || r_abort store-unreadable "$v"
+    # Row 233 (sage-1653): which store the rehearsal ran against, for root's step log. Output only:
+    # the rehearsal's own steps.log has the backup's path on its backup line.
+    echo "BACKUP_STORE $R_BACKUP schema $v"
    if [ "$v" = "$T_SCHEMA" ]; then r_pass migration "not needed: the backup is already $v"; return 0; fi
    in_chain "$v" || r_abort store-newer "the backup is $v, which $TO's migration chain does not start from"
    (umask 077; mkdir "$R/refuse" "$R/refuse/bus" && cp "$R_BK"/bus/* "$R/refuse/bus/") || r_abort scratch-build "cannot copy the backup to $R/refuse"
@@ -2043,6 +2057,13 @@ do_rehearse() {
    trap 'r_fail signal "the rehearsal was interrupted"; r_end' INT TERM HUP
    read_pointer || r_abort no-pointer "no $POINTER; the pointer's copy is the old copy the checks use"
    [ "$P_BUSINESS" = "$BUSINESS" ] || r_abort business-mismatch "the pointer is for $P_BUSINESS"
+    # Row 233 (sage-1650, sage-1653): last is the newest backup the log records. With none, exit 3
+    # and no directory, the one rehearse refusal stack-upgrade.sh reads as no-backup.
+    if [ "$R_BACKUP" = last ]; then
+        R_BACKUP="$(from_log newest)" || { r_fail no-backup "the log records no backup"; R_RC=3; r_end; }
+        [[ "$R_BACKUP" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || r_abort backup-unreadable "the log's newest backup is not an activation id"
+        r_pass backup-last "the log's newest backup is $R_BACKUP"
+    fi
    copy_exists "$TO" 2>/dev/null || r_abort copy-missing "no copy $TO in $COPIES"
    copy_ok "$TO" 2>/dev/null || r_abort copy-unreachable "$TO is not reachable from refactor and origin/refactor"
    target_schema "$TO" || r_abort copy-schema-unknown "cannot read SCHEMA_VERSION and its chain from $COPIES/$TO/packages/bus/src/store.mjs"
@@ -0,0 +1,347 @@
+#!/usr/bin/env bash
+# Row 233 (I3, #1735): root's upgrade and rollback for a stack whose units are root's system units
+# (stack-release.sh --units external, row 232). r6's activate A1-A5, A2f, R1-R4 and rollback
+# (agents/darkwing/work/queue-195/DESIGN.md), as the I3 brief changes them
+# (docs/plans/2026-10-20_installer-rows.md). Deviations and decisions:
+# agents/darkwing/work/queue-233/BUILD.md.
+#
+#   stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
+#                    [--rehearse-port N] [--timeout S]
+#   stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
+#
+# Root runs it from a root-owned checkout of the reviewed commit, never from a copy, because the
+# stack's user can write the copies. The unit values come from /etc/mosaic-stack/<instance>.args,
+# which I2's install writes: one case2-unit.sh argument per line, the serve flags after `--`. Only
+# --copy changes, and the file changes only after confirm passes. The stack's user runs its steps
+# from its own clone, <home>/src/mosaic-stack, through sudo. Root reads none of that user's output
+# but exit codes and the rehearsal's BACKUP_STORE line (sage-1653), which it copies into its step
+# log after a match against a fixed pattern and decides nothing on.
+#
+# --no-rehearsal is for a stack whose log records no backup. The rehearsal runs either way: exit 3
+# from it (no-backup) goes on only with the flag, and exit 0 with the flag refuses
+# rehearsal-required (sage-1650, sage-1652).
+#
+# The lock /run/mosaic-stack/upgrade-<instance>.lock is held from before A1 to the end of the run,
+# rollback included; every child runs with the lock's descriptor closed. The step log
+# /var/log/mosaic-stack/upgrade-<instance>.log gets one line per step, `<time> <step> rc=N argv:
+# <argv>`, and a last `<time> result exit=N <code> <text>` line.
+#
+# Exit 0 ok, 1 failed with the old stack running, 2 rollback-failed with the units stopped,
+# 3 refused with nothing stopped, 4 usage.
+#
+# Test seams, for packages/cli/tests/stack-upgrade.test.mjs only. Run as root, it refuses any:
+#   STACK_UPGRADE_ETC        default /etc/mosaic-stack
+#   STACK_UPGRADE_RUN        default /run/mosaic-stack
+#   STACK_UPGRADE_LOG_DIR    default /var/log/mosaic-stack
+#   STACK_UPGRADE_UNIT_DIR   default /etc/systemd/system
+#   STACK_UPGRADE_HOSTCHECK  default /usr/local/lib/mosaic-stack/hostcheck.sh (row 240)
+#   STACK_UPGRADE_RENDER     default scripts/case2-unit.sh in this checkout
+#   STACK_UPGRADE_SUDO       default sudo
+#   STACK_UPGRADE_SYSTEMCTL  default systemctl
+set -uo pipefail
+REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+usage() {
+    cat >&2 <<'USAGE'
+usage: stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal] [--rehearse-port N] [--timeout S]
+       stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
+USAGE
+    exit 4
+}
+
+if [ "$(id -u)" = 0 ]; then
+    for v in ${!STACK_UPGRADE_@}; do echo "stack-upgrade: refusing: $v is a test seam, and root takes none" >&2; exit 3; done
+fi
+ETC="${STACK_UPGRADE_ETC:-/etc/mosaic-stack}"
+RUN_DIR="${STACK_UPGRADE_RUN:-/run/mosaic-stack}"
+LOG_DIR="${STACK_UPGRADE_LOG_DIR:-/var/log/mosaic-stack}"
+UNIT_DIR="${STACK_UPGRADE_UNIT_DIR:-/etc/systemd/system}"
+HOSTCHECK="${STACK_UPGRADE_HOSTCHECK:-/usr/local/lib/mosaic-stack/hostcheck.sh}"
+RENDER="${STACK_UPGRADE_RENDER:-$REPO/scripts/case2-unit.sh}"
+SUDO="${STACK_UPGRADE_SUDO:-sudo}"
+SYSTEMCTL="${STACK_UPGRADE_SYSTEMCTL:-systemctl}"
+ID_RE='^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$'
+
+ARGV=("$@")
+[ $# -gt 0 ] || usage
+OP="$1"; shift
+case "$OP" in upgrade|rollback) ;; *) usage ;; esac
+INSTANCE="" BUNDLE="" SHA="" MIGRATE=0 NO_REHEARSAL=0 R_PORT=7339 TIMEOUT=60 TO="" RESTORE="" EVENTS=""
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --instance) [ $# -ge 2 ] || usage; INSTANCE="$2"; shift 2 ;;
+        --bundle) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^/[A-Za-z0-9/._+-]+$ ]] || usage; BUNDLE="$2"; shift 2 ;;
+        --sha) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; SHA="$2"; shift 2 ;;
+        --migrate) [ "$OP" = upgrade ] || usage; MIGRATE=1; shift ;;
+        --no-rehearsal) [ "$OP" = upgrade ] || usage; NO_REHEARSAL=1; shift ;;
+        --rehearse-port) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{3,4}$ ]] && [ "$2" -le 65535 ] || usage; R_PORT="$2"; shift 2 ;;
+        --timeout) [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
+        --to) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; TO="$2"; shift 2 ;;
+        --restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ $ID_RE ]] || usage; RESTORE="$2"; shift 2 ;;
+        --events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
+        *) usage ;;
+    esac
+done
+# The default instance only, whose name is dev (row 190); a named instance's units are a later row.
+[ "$INSTANCE" = dev ] || usage
+if [ "$OP" = upgrade ]; then [ -n "$BUNDLE" ] && [ -n "$SHA" ] || usage; else [ -n "$TO" ] || usage; fi
+[ -z "$EVENTS" ] || [ -n "$RESTORE" ] || usage
+
+# --- The step log and the lock ---
+
+umask 027
+LOG="$LOG_DIR/upgrade-$INSTANCE.log"
+LOCK="$RUN_DIR/upgrade-$INSTANCE.lock"
+{ mkdir -p "$LOG_DIR" "$RUN_DIR" && touch "$LOG" && chmod 0640 "$LOG"; } 2>/dev/null ||
+    { echo "stack-upgrade: refusing log-failed: cannot write $LOG or make $RUN_DIR; nothing was changed" >&2; exit 3; }
+logl() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >> "$LOG" || true; }
+# The run's last line, in the log and on the terminal, then the exit.
+finish() {
+    local rc="$1" code="$2"; shift 2
+    logl "result exit=$rc $code $*"
+    if [ "$rc" = 0 ]; then echo "stack-upgrade: $*"; else echo "stack-upgrade: $code: $*" >&2; fi
+    exit "$rc"
+}
+refuse() { local code="$1"; shift; finish 3 "$code" "$* (nothing was stopped)"; }
+# One step: $1 its name, the rest the command, run with the lock's descriptor closed. One log line
+# with its exit and argv; the command's output goes to the terminal only.
+step() {
+    local name="$1" rc=0; shift
+    "$@" 9>&- || rc=$?
+    logl "$name rc=$rc argv:$(printf ' %q' "$@")"
+    return "$rc"
+}
+
+exec 9>> "$LOCK" || refuse lock-failed "cannot open $LOCK"
+flock -n 9 || refuse upgrade-running "another stack-upgrade.sh run holds $LOCK"
+logl "start rc=0 argv:$(printf ' %q' "$0" "${ARGV[@]}")"
+
+# --- The args file and the stack's user ---
+
+# I2's file: one case2-unit.sh argument per line, owned by root with no group or other write. Sets
+# ARGS (the lines), S_USER, S_HOME, BUSINESS, NODE and ARGS_COPY, and the paths derived from them.
+read_args() {
+    local f="$ETC/$INSTANCE.args" st i copies=0 v
+    [ -f "$f" ] && [ ! -L "$f" ] || refuse args-unreadable "no regular file $f"
+    st="$(stat -c '%u %a' "$f")" || refuse args-unreadable "cannot stat $f"
+    [ "${st% *}" = "$(id -u)" ] && (( (8#${st#* } & 8#022) == 0 )) || refuse args-unreadable "$f must be $(id -un)'s with no group or other write (it is uid ${st% *}, mode ${st#* })"
+    mapfile -t ARGS < "$f" || refuse args-unreadable "cannot read $f"
+    S_USER="" S_HOME="" BUSINESS="" NODE="" ARGS_COPY=""
+    for ((i = 0; i < ${#ARGS[@]}; i++)); do
+        case "${ARGS[i]}" in
+            --) break ;;
+            --pm) ;;
+            --user|--home|--business|--copy|--node|--profile|--claude)
+                v="${ARGS[i + 1]:-}"
+                case "${ARGS[i]}" in
+                    --user) S_USER="$v" ;; --home) S_HOME="$v" ;; --business) BUSINESS="$v" ;; --node) NODE="$v" ;;
+                    --copy) ARGS_COPY="$v"; copies=$((copies + 1)) ;;
+                esac
+                i=$((i + 1)) ;;
+            *) refuse args-unreadable "$f has ${ARGS[i]} before --, which case2-unit.sh does not take" ;;
+        esac
+    done
+    [ "$copies" = 1 ] || refuse args-unreadable "$f needs exactly one --copy before --, not $copies"
+    [[ "$S_USER" =~ ^[a-z_][a-z0-9_-]*$ ]] && [[ "$S_HOME" =~ ^/[A-Za-z0-9/._+-]+$ ]] && [[ "$BUSINESS" =~ ^[a-z0-9][a-z0-9-]*$ ]] &&
+        [[ "$NODE" =~ ^/[A-Za-z0-9/._+-]+$ ]] && [[ "$ARGS_COPY" =~ ^[0-9a-f]{40}$ ]] ||
+        refuse args-unreadable "$f lacks a plain --user, --home, --business, --node or --copy"
+    COPIES_DIR="$S_HOME/.local/share/mosaic-stack/copies"
+    CLONE="$S_HOME/src/mosaic-stack"
+    SR="$CLONE/scripts/stack-release.sh"
+    BUS_UNIT="mosaic-bus@$BUSINESS.service" CON_UNIT=mosaic-webui.service
+    # The stack's user's steps: its own HOME and default config, the pinned node first on PATH.
+    AS_USER=("$SUDO" -u "$S_USER" -H env "PATH=$(dirname "$NODE"):/usr/local/bin:/usr/bin:/bin")
+}
+
+# RENDER_ARGS: the args file's lines with copy $1 in place of --copy's value.
+args_with() {
+    local i past=""
+    RENDER_ARGS=()
+    for ((i = 0; i < ${#ARGS[@]}; i++)); do
+        RENDER_ARGS+=("${ARGS[i]}")
+        [ "${ARGS[i]}" != -- ] || past=1
+        if [ -z "$past" ] && [ "${ARGS[i]}" = --copy ]; then RENDER_ARGS+=("$1"); i=$((i + 1)); fi
+    done
+}
+
+# The new args file, written beside the old one before anything stops, moved into place only after
+# confirm passes. $1 is the copy.
+prepare_args() {
+    args_with "$1"
+    NEW_ARGS="$(mktemp "$ETC/.$INSTANCE.args.XXXXXX")" &&
+        printf '%s\n' "${RENDER_ARGS[@]}" > "$NEW_ARGS" && chmod --reference="$ETC/$INSTANCE.args" "$NEW_ARGS" ||
+        { rm -f "${NEW_ARGS:-}"; refuse args-failed "cannot write the new args file in $ETC"; }
+}
+drop_args() { rm -f "${NEW_ARGS:-}"; }
+
+# --- A1 ---
+
+# OLD: the copy both installed units run, from their WorkingDirectory under the stack's copies.
+read_old() {
+    local f ids=() id n
+    for f in "$UNIT_DIR/[email protected]" "$UNIT_DIR/$CON_UNIT"; do
+        [ -f "$f" ] && [ ! -L "$f" ] || refuse units-unreadable "no regular file $f"
+        n="$(grep -c '^WorkingDirectory=' "$f")"
+        id="$(awk -v c="WorkingDirectory=$COPIES_DIR/" 'index($0, c) == 1 { print substr($0, length(c) + 1) }' "$f")"
+        [ "$n" = 1 ] && [[ "$id" =~ ^[0-9a-f]{40}$ ]] || refuse units-unreadable "$f names no copy under $COPIES_DIR"
+        ids+=("$id")
+    done
+    [ "${ids[0]}" = "${ids[1]}" ] || refuse units-disagree "$UNIT_DIR's units name ${ids[0]} and ${ids[1]}"
+    OLD="${ids[0]}"
+    logl "units rc=0 old: $OLD"
+    [ "$ARGS_COPY" = "$OLD" ] || refuse args-differ "$ETC/$INSTANCE.args names $ARGS_COPY and the units name $OLD"
+}
+
+# r6 A1 and row 240: the host's profile still matches, on every host.
+hostcheck() {
+    [ -f "$HOSTCHECK" ] && [ -x "$HOSTCHECK" ] || refuse profile-drift "no executable $HOSTCHECK (row 240)"
+    step hostcheck "$HOSTCHECK" --check || refuse profile-drift "$HOSTCHECK --check failed"
+}
+
+# The stack's user's --plan for the run, $@ its op and arguments; any refusal stops here.
+plan() {
+    local rc=0
+    step plan "${AS_USER[@]}" "$SR" "$@" --units external --copies "$COPIES_DIR" --plan --expect-from "$OLD" || rc=$?
+    [ "$rc" = 0 ] || refuse plan-refused "the stack's user's $1 --plan exited $rc; its output above names the refusal"
+}
+
+# --- The units ---
+
+stop_wait() {
+    local u s deadline=$((SECONDS + TIMEOUT)) rc=0
+    step stop-console "$SYSTEMCTL" stop "$CON_UNIT" || rc=1
+    step stop-bus "$SYSTEMCTL" stop "$BUS_UNIT" || rc=1
+    for u in "$CON_UNIT" "$BUS_UNIT"; do
+        while :; do
+            s="$("$SYSTEMCTL" is-active "$u" 9>&- 2>/dev/null)"
+            case "$s" in inactive|failed) break ;; esac
+            [ "$SECONDS" -lt "$deadline" ] || { rc=1; break; }
+            sleep 1
+        done
+        case "$s" in inactive|failed) logl "wait-down rc=0 unit: $u state: $s" ;; *) logl "wait-down rc=1 unit: $u state: ${s:-unknown}" ;; esac
+    done
+    return "$rc"
+}
+stop_units() { "$SYSTEMCTL" stop "$CON_UNIT" 9>&- || true; "$SYSTEMCTL" stop "$BUS_UNIT" 9>&- || true; logl "stop-units rc=0 units: $CON_UNIT $BUS_UNIT"; }
+
+# A2f: the old units back on the copy they never left, gated on a start at or after this second.
+# An A2 that timed out leaves a unit on its old start, which the gate refuses as stale-start.
+a2f() {
+    local cause="$1" since
+    drop_args
+    since="$(date -u +%s)"
+    step start-bus "$SYSTEMCTL" start "$BUS_UNIT"
+    step start-console "$SYSTEMCTL" start "$CON_UNIT"
+    if step gate "${AS_USER[@]}" "$SR" gate "$OLD" "$BUSINESS" --units external --copies "$COPIES_DIR" --since "$since" --timeout "$TIMEOUT"; then
+        finish 1 failed "$cause failed; $OLD is running again and passed the gate"
+    fi
+    stop_units
+    finish 2 rollback-failed "$cause failed and $OLD failed its gate after the restart; both units are stopped"
+}
+
+# R1-R4: back to OLD after the stack's user staged $1. $2 is the cause: A3, A4 or A5.
+rollback_to_old() {
+    local id="$1" cause="$2"
+    drop_args
+    stop_wait || { stop_units; finish 2 rollback-failed "$cause failed and the units did not stop for the rollback; both units are stopped"; }
+    step revert "${AS_USER[@]}" "$SR" revert "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" ||
+        { stop_units; finish 2 rollback-failed "$cause failed and revert failed; both units are stopped"; }
+    args_with "$OLD"
+    step render-old "$RENDER" "${RENDER_ARGS[@]}" ||
+        { stop_units; finish 2 rollback-failed "$cause failed and the render of $OLD failed; both units are stopped"; }
+    # confirm --rollback exits 1 when OLD passed its gate and 2 when it failed; anything else is a
+    # refusal, with OLD's gate never run.
+    local rc=0
+    step confirm-rollback "${AS_USER[@]}" "$SR" confirm "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" --rollback "$OLD" --cause "$cause" --timeout "$TIMEOUT" || rc=$?
+    [ "$rc" != 1 ] || finish 1 failed "$cause failed; rolled back to $OLD, which passed the gate"
+    stop_units
+    finish 2 rollback-failed "$cause failed and the rollback to $OLD did not pass (confirm --rollback exited $rc); both units are stopped"
+}
+
+# A2 to A5 for copy $1, once A1 and any rehearsal passed. $@ after it is the stack's user's op.
+run_steps() {
+    local id="$1" rc=0; shift
+    prepare_args "$id"
+    # A2
+    stop_wait || a2f A2
+    # A3
+    step stage "${AS_USER[@]}" "$SR" "$@" --units external --copies "$COPIES_DIR" || rc=$?
+    case "$rc" in
+        0) ;;
+        3) a2f A3 ;;
+        *) rollback_to_old "$id" A3 ;;
+    esac
+    # A4
+    args_with "$id"
+    step render "$RENDER" "${RENDER_ARGS[@]}" || rollback_to_old "$id" A4
+    # A5
+    step confirm "${AS_USER[@]}" "$SR" confirm "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" --timeout "$TIMEOUT" ||
+        rollback_to_old "$id" A5
+    if mv -f "$NEW_ARGS" "$ETC/$INSTANCE.args"; then logl "args rc=0 copy: $id"
+    else logl "args rc=1 copy: $id"; finish 1 args-failed "$id is live and confirmed, but $ETC/$INSTANCE.args still names $OLD; put $NEW_ARGS in its place"
+    fi
+}
+
+do_upgrade() {
+    read_args
+    # Before A1: the bundle into the stack's user's clone, then the copy from it (I1).
+    step fetch "${AS_USER[@]}" git -C "$CLONE" fetch -q "$BUNDLE" refs/heads/refactor:refs/remotes/origin/refactor ||
+        refuse fetch-failed "the stack's user could not fetch $BUNDLE into $CLONE"
+    step ff "${AS_USER[@]}" git -C "$CLONE" merge -q --ff-only "$SHA" ||
+        refuse fetch-failed "the stack's user could not fast-forward $CLONE's refactor to $SHA"
+    step copy "${AS_USER[@]}" "$CLONE/scripts/stack-copy.sh" copy --bundle "$BUNDLE" "$SHA" --copies "$COPIES_DIR" ||
+        refuse copy-failed "the stack's user could not make copy $SHA from $BUNDLE"
+    # A1
+    local head
+    head="$(git -C "$REPO" rev-parse HEAD 2>/dev/null)"
+    [ "$head" = "$SHA" ] || refuse head-mismatch "this checkout, $REPO, is at ${head:-unknown}, not $SHA"
+    [ -x "$COPIES_DIR/$SHA/scripts/mosaic" ] || refuse copy-missing "no $COPIES_DIR/$SHA/scripts/mosaic"
+    hostcheck
+    read_old
+    local -a migrate=()
+    [ "$MIGRATE" = 0 ] || migrate=(--migrate)
+    plan activate "$SHA" "$BUSINESS" "${migrate[@]}"
+    rehearse
+    run_steps "$SHA" activate "$SHA" "$BUSINESS" "${migrate[@]}"
+    finish 0 ok "upgrade ok: $SHA is live for $BUSINESS"
+}
+
+# The rehearsal against the log's newest backup, before anything stops. Its output goes to the
+# terminal; root keeps only the BACKUP_STORE line, so the go and the log show which store it used.
+rehearse() {
+    local cap rc bs
+    cap="$(mktemp "$RUN_DIR/rehearse-$INSTANCE.XXXXXX")" || refuse rehearsal-failed "cannot make a capture file in $RUN_DIR"
+    local -a cmd=("${AS_USER[@]}" "$SR" rehearse "$SHA" "$BUSINESS" --copies "$COPIES_DIR" --backup last --port "$R_PORT" --timeout "$TIMEOUT")
+    "${cmd[@]}" 9>&- 2>&1 | tee "$cap" 9>&-
+    rc="${PIPESTATUS[0]}"
+    logl "rehearse rc=$rc argv:$(printf ' %q' "${cmd[@]}")"
+    bs="$(grep -E '^BACKUP_STORE [0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])? schema [0-9a-z]{1,16}$' "$cap" | tail -n 1)"
+    rm -f "$cap"
+    logl "rehearse-store rc=0 ${bs:-BACKUP_STORE none}"
+    case "$rc" in
+        0) [ "$NO_REHEARSAL" = 0 ] || refuse rehearsal-required "the log records a backup and the rehearsal ran; --no-rehearsal is only for a stack with none" ;;
+        3) [ "$NO_REHEARSAL" = 1 ] || refuse no-backup "the log records no backup to rehearse against; only a go that names --no-rehearsal goes on"
+           logl "rehearsal-skipped rc=0 the log records no backup and the go names --no-rehearsal" ;;
+        *) refuse rehearsal-failed "the rehearsal exited $rc" ;;
+    esac
+}
+
+# r6 rollback: to the log's previous copy, which the stack's user checks against --to. No HEAD
+# check: the target is an older copy than this checkout, and no rehearsal.
+do_rollback() {
+    read_args
+    [ -x "$COPIES_DIR/$TO/scripts/mosaic" ] || refuse copy-missing "no $COPIES_DIR/$TO/scripts/mosaic"
+    hostcheck
+    read_old
+    local -a restore=()
+    [ -z "$RESTORE" ] || restore=(--restore "$RESTORE")
+    [ -z "$EVENTS" ] || restore+=(--events "$EVENTS")
+    plan rollback "$BUSINESS" --to "$TO" "${restore[@]}"
+    run_steps "$TO" rollback "$BUSINESS" --to "$TO" "${restore[@]}"
+    finish 0 ok "rollback ok: $TO is live for $BUSINESS"
+}
+
+case "$OP" in
+    upgrade) do_upgrade ;;
+    rollback) do_rollback ;;
+esac
Author	SHA1	Message	Date
jason.woltje	1a1dc80caa	row 233: BUILD.md, the gate and m1-m8 results on `d99166ade` (1201/0, 8 of 8 killed) and the N1 fix that leaves the gate standing (sage-1707)	2026-10-20 23:43:28 -05:00
jason.woltje	08db6ac3f5	row 233: the lock test kills its flock holder's process group (N1, sage-1707) stack-upgrade.test.mjs:312 spawned `flock LOCK sleep 60` and SIGKILLed only the flock, so its `sleep 60` child outlived the test. Spawn it detached and kill the group with process.kill(-pid), as row 232's T20 does (stack-release.test.mjs:370). Test only.	2026-10-20 23:43:21 -05:00
jason.woltje	d99166ade2	row 233: BUILD.md, singles run 2 on `aacfa2d7a` (stack-release 195/0)	2026-10-20 23:23:00 -05:00
jason.woltje	aacfa2d7a4	stack-upgrade: root's upgrade and rollback driver for root-owned units, with the host run sheet; stack-release --backup last and the external plan refusals (row 233, #1735 , installer I3)	2026-10-20 23:00:36 -05:00