row 233: BUILD.md, the gate and m1-m8 results on d99166ade (1201/0, 8 of 8 killed) and the N1 fix that leaves the gate standing (sage-1707)

row 233: the lock test kills its flock holder's process group (N1, sage-1707)
stack-upgrade.test.mjs:312 spawned `flock LOCK sleep 60` and SIGKILLed only the flock, so its `sleep 60` child outlived the test. Spawn it detached and kill the group with process.kill(-pid), as row 232's T20 does (stack-release.test.mjs:370). Test only.
2026-10-20 23:43:28 -05:00 · 2026-10-20 23:43:21 -05:00 · 2026-10-20 23:23:00 -05:00 · 2026-10-20 23:00:36 -05:00
15 changed files with 1470 additions and 633 deletions
@@ -0,0 +1,249 @@
+# Row 233 (I3): root's upgrade driver and the host run sheet
+
+Brief: `docs/plans/2026-10-20_installer-rows.md` section I3, blob `e61291bcf1c6` (commit a0e3d7cfd), re-pinned
+by Sage (sage-1685, rev 2631) from `f35e7e57daee` (ad6f0be7a). The later blob only appends I4; the I3 text
+is the same. Gitea #1735, Vikunja P19
+#432. Owner Darkwing, reviewer Filbert, gate owner Sage. Design input: row 195 DESIGN r6,
+`agents/darkwing/work/queue-195/DESIGN.md` at b7239bb66, sections 2, 4 and 7. Where the brief and r6
+differ, the brief wins. Sage's rulings during the build: sage-1650 and sage-1652 (`--backup last`, the
+`--no-rehearsal` mapping, `--rehearse-port`, hostcheck), and sage-1653 (`from_log newest`, the two external
+plan refusals, the `BACKUP_STORE` line).
+
+## Files
+
+Base: e65a3ab4d, row 232's landing, on `darkwing/row233-r1`. The build started on ea1e85c7b
+(`darkwing/row233`, row 232's candidate e6ca230d2 and its BUILD.md) before row 232 landed. The three row 232
+files this row changes are identical on the two bases, so the uncommitted diff moved across unchanged.
+
+- `scripts/stack-upgrade.sh` (new): `upgrade` and `rollback`, root's side of r6 A1-A5, A2f and R1-R4.
+- `packages/cli/tests/stack-upgrade.test.mjs` (new, 20 tests).
+- `docs/guides/host-run-sheet.md` (new).
+- `docs/TOOLS.md`: a new subsection "Root's upgrade driver (`scripts/stack-upgrade.sh`, row 233)", plus the
+  row 232 lines below.
+- Row 232 files, changed on Sage's rulings (the brief's files list doesn't name them):
+  - `scripts/stack-release.sh`:
+    - `rehearse --backup last` takes the newest backup the log records (`from_log newest`) and prints
+      `PASS backup-last`. With none it fails `no-backup`, exits 3 and makes no rehearsal directory.
+      A newest backup that isn't an activation id aborts `backup-unreadable`, exit 1.
+    - The rehearsal prints `BACKUP_STORE <activation id> schema <version>` before its migration step.
+    - `--plan` under `--units external` refuses `open-activation` when any rollback set holds
+      `AWAITING-UNITS`, and `migrate-required` for a migrating target without `--migrate`. Root reads
+      only the plan's exit, so these two have to be refusals there. User mode prints the same bytes.
+  - `packages/cli/tests/stack-release.test.mjs`: three new `row 233:` tests for `--backup last`.
+  - `docs/TOOLS.md`: `--backup ACTIVATION|PATH|last` in the rehearse usage, a paragraph on `last`, and a
+    paragraph on the two external plan refusals.
+- This file.
+
+## The brief, item by item
+
+| Brief item | Where |
+|---|---|
+| Root runs from a root-owned checkout, never a copy | `head-mismatch` (A1); TOOLS.md; run sheet section 6 |
+| I2's render with `/etc/mosaic-stack/<instance>.args`, `--copy` the one changed value | `read_args`, `args_with`; the args test |
+| The args file changes only after a confirmed run | `prepare_args` before A2, `mv` after A5; sequences 1 and 7, m6 |
+| `copy --bundle` as the stack's user before A1 | the `fetch`, `ff`, `copy` steps; sequence 1's sudo keys |
+| A rehearsal before A2, a failed one refuses with nothing stopped | `rehearse`; three rehearsal tests, m4 |
+| The root lock, held to the end, rollback included | `flock -n 9`; the lock test and sequence 3's probe, m1, m2 |
+| A1 refuses `open-activation` | stack-release.sh's external plan; the AWAITING-UNITS test, m3 |
+| `migrate-required` without `--migrate` | stack-release.sh's external plan; its test, m5 |
+| The step log, 0640, one line per step | `logl`, `step`; every test checks lines against `STEP` |
+| Exit codes 0-3 | `finish`, `refuse`; exit 4 is usage |
+| duckworth-0055's notes | the next section |
+| Root parses no stack user output beyond exits | the step log test; the one exception is sage-1653's `BACKUP_STORE` |
+| The host run sheet | `docs/guides/host-run-sheet.md` |
+
+## duckworth-0055's notes on r6 (brief line 169)
+
+- N1, A2's stop timeout under `gate --since`. Row 232's BUILD.md left two pieces to I3. The run sheet's
+  A2f line is in section 6 ("When a run doesn't exit 0", exit 2). The test is `N1: a bus that outlives
+  A2's timeout keeps its old start`: a bus that ignores `stop` keeps its old `ActiveEnterTimestamp`,
+  `wait-down` logs `rc=1 ... state: active`, A2f restarts and gates, the gate fails `stale-start`, and the
+  driver stops both units and exits 2. Sequence 4 is the other half: an A3 exit 3 goes to A2f and exits
+  1 with the old copy's gate passing.
+- N2, the claude path under `/opt/mosaic-stack`. Run sheet section 3 installs it there as root with
+  managed settings. The args file's `--claude` carries it, and `case2-unit.sh` already refuses a binary
+  that isn't root's or sits under the home. The driver passes the args file through and adds no claude
+  check of its own. F6 is I2's.
+
+## Sage's rulings during the build
+
+- sage-1650 and sage-1652. Root always runs the rehearsal and maps its exit:
+  - 0 goes on, or refuses `rehearsal-required` with `--no-rehearsal`.
+  - 3 refuses `no-backup`, or with `--no-rehearsal` logs `rehearsal-skipped` and goes on.
+  - Anything else refuses `rehearsal-failed`.
+
+  `--rehearse-port N`, default 7339. Hostcheck: a missing file or a nonzero `--check` refuses
+  `profile-drift`.
+- sage-1653. `from_log newest`, and the external plan refusals `open-activation` and `migrate-required`.
+  Residual: the `BACKUP_STORE` id and schema land in root's step log, but nothing ties that backup to the
+  one the activation's own backup step takes later. A stopped backup before the upgrade would close that
+  gap; it's a later row, not this one.
+
+## Tests
+
+`stack-upgrade.test.mjs` runs the driver as the test's user against a scratch root, with the test seams
+`STACK_UPGRADE_*`. The stand-ins are a fake `sudo` (logs its argv, checks the driver's form, then runs the
+real command as the test's user, r6's B1 pattern), a fake `case2-unit.sh` render that installs and starts
+the fixture units, a fake `systemctl` for root's system units, and a shell `hostcheck.sh`. The stack's
+user's steps run the real `stack-release.sh` and `stack-copy.sh` from a clone made from a real bundle. The
+units, copies and store are `stack-release-fixture.mjs`'s.
+
+| Test | Covers |
+|---|---|
+| sequence 1 | confirm fails healthz, rollback with `--cause A5`, exit 1; every sudo call in the driver's form |
+| sequence 2 | `activate --migrate` fails at the wrapper, store restored, `--cause A3`, exit 1 |
+| sequence 3 | render fails, `--cause A4`; a probe run during `render-old` refuses `upgrade-running` |
+| sequence 4 | stage exits 3, A2f restarts and gates the old units, exit 1 |
+| N1 | a unit that won't stop: A2f's gate fails `stale-start`, exit 2, units stopped |
+| sequence 5 | confirm of the wrong copy refused, `--cause A5`, exit 1 |
+| sequence 6 | revert fails, exit 2; the next run refuses `interrupted-run` at its plan |
+| sequence 7 | two upgrades and a `rollback --to` with `--restore`; the args file after each confirm; `to-not-previous` |
+| sequence 8 | a migration that writes and fails, store back byte for byte, exit 1 |
+| lock | a second run while the lock is held: one log line, no call |
+| open-activation | any `AWAITING-UNITS` refuses at A1, nothing stopped |
+| migrate-required | refused without `--migrate`, no rehearsal, nothing stopped |
+| rehearsal-failed | the rehearsal's argv, exit 3, nothing stopped |
+| `BACKUP_STORE` | the line after a match; four malformed lines log `BACKUP_STORE none` |
+| no-backup | the real rehearsal with no backup refuses; `--no-rehearsal` logs the skip and goes on |
+| backup-last | the real rehearsal names the newest backup; `--no-rehearsal` with one refuses `rehearsal-required` |
+| args file | group write, other write, two `--copy`, an unknown flag, a symlink, `args-differ` |
+| hostcheck | nonzero `--check` and a missing file refuse `profile-drift` before the plan |
+| head-mismatch | checkout not at `--sha`; a commit the clone can't fast-forward to refuses `fetch-failed` |
+| usage | twelve bad argvs exit 4 with no log and no lock |
+
+The step log check (`STEP`) runs in every test that reads the log: each line must be a step line or the
+result line, and none may hold the stack user's output (`stack-release:`, `PASS `, `FAIL `, `REHEARSAL`,
+the fake rehearsal's text).
+
+## Mutants
+
+`~/darkwing-scratch/r233/mutants.sh`, not committed because it edits the worktree. It has the brief's six
+mutants and two for sage-1650's mapping. The method is row 232's: a checked backup, exact replacements, a
+syntax check, the named killers, then the file put back with its sha256 checked. One control runs first:
+the whole `stack-upgrade.test.mjs` with no change. `DRY=1` matched each replacement once on the candidate
+(2026-10-21).
+
+| Mutant | File | Killers |
+|---|---|---|
+| m1 drop the flock | stack-upgrade.sh | `a second run while the lock is held`, `sequence 3:` |
+| m2 release the lock when the rollback starts | stack-upgrade.sh | `sequence 3:` |
+| m3 skip A1's open-activation check | stack-release.sh | `A1 refuses open-activation` |
+| m4 skip the rehearsal | stack-upgrade.sh | `the rehearsal runs after the plan`, `sequence 1:` |
+| m5 accept a migrating target without `--migrate` | stack-release.sh | `migrate-required` |
+| m6 write the args file before confirm | stack-upgrade.sh | `sequence 7:`, `sequence 1:` |
+| m7 `--no-rehearsal` accepted with a backup | stack-upgrade.sh | `refuses rehearsal-required` |
+| m8 drop the `rehearsal-skipped` line | stack-upgrade.sh | `a real rehearsal with no backup` |
+
+## Deviations
+
+D1. The brief has A1 refuse `open-activation` for an open `AWAITING-UNITS` or a non-empty `interrupted`
+set. The interrupted case refuses with row 232's existing code, `interrupted-run`; only `AWAITING-UNITS`
+gives `open-activation`. Both reach root as `plan-refused`, exit 3 (sequence 6, the open-activation test).
+
+D2. Root copies one line of the stack user's output into its log, `BACKUP_STORE`, on sage-1653. It
+matches a fixed pattern first, takes the last match, logs `BACKUP_STORE none` otherwise, and decides
+nothing on it.
+
+D3. The stack user's steps carry no `--instance`. The driver takes `--instance dev` only, row 190's default
+instance, which the stack user's commands select with no flag. A named instance's units are a later row.
+
+D4. Most tests fake the rehearsal in the fake `sudo`: the fixture's stub copies can't pass a real one. Two
+tests run the real rehearsal (`faults/real-rehearse`): no-backup and backup-last. `stack-release.test.mjs`
+covers `--backup last` itself.
+
+D5. Any nonzero `--plan` exit refuses `plan-refused`, exit 3, whatever the code. The stack user's output
+above it names the refusal; root doesn't parse it.
+
+D6. `rollback --restore ACT` is accepted without `--events`, as `stack-release.sh rollback` accepts it.
+The brief's usage pairs them. `--events` without `--restore` is a usage error.
+
+D7. The refusal of any `STACK_UPGRADE_*` seam when run as root has no test, because the tests don't run
+as root. It is three lines at the top of the script.
+
+D8. The fake `sudo` keeps the test's environment, where real `sudo` resets it. The driver passes
+everything the stack user needs through `env PATH=...` and `-H`, and the fake checks that form.
+
+D9. Exit 3 means nothing stopped, not nothing changed. A refusal after `fetch` leaves the stack user's
+clone fast-forwarded and the new copy made. Both are the user's own state, and the next run reuses them.
+TOOLS.md and the run sheet say so.
+
+D10. The brief's usage has neither `--no-rehearsal` nor `--rehearse-port`. Both come from sage-1650 and
+sage-1652.
+
+## Decisions
+
+- `rollback` has no HEAD check. Its target is an older copy than root's checkout, which already exists.
+  It fetches, copies and rehearses nothing (sequence 7).
+- The new args file is written next to the old one (`.dev.args.XXXXXX`, same mode) before A2, so a full
+  `/etc` refuses before anything stops. It moves into place after A5. A2f and the rollback remove it; a failed move keeps it and names it.
+- Every stack user step gets `--copies` explicitly, so the copies directory comes from the args file's
+  `--home`, never from the user's environment.
+- The stack's user does the fetch, the fast-forward and the copy, so root never runs git on a tree that
+  user can write.
+- A4 is the render, and `case2-unit.sh` starts the bus, then the Console. The forward path has no
+  `systemctl start` of its own; only A2f does.
+
+## Run environment
+
+Worktree `/mnt/storage/scratch/darkwing-r233`, `TMPDIR=/mnt/storage/scratch/tmp/r233`, node from PATH. One
+whole `stack-upgrade.test.mjs` run takes about 56 s.
+
+## Results
+
+### Singles, run 1, 2026-10-21, Sage's slot (sage-1689)
+
+`~/darkwing-scratch/r233/singles.sh` on `darkwing/row233-r1` (base e65a3ab4d, uncommitted tree), 03:48:20Z
+to 03:59:00Z. These are the test files that read a changed script. `stack-instances.test.mjs` reads
+`stack-release.sh`, so it joined the three named in the plan.
+
+| File | Result |
+|---|---|
+| `stack-upgrade.test.mjs` | 20 pass, 0 fail |
+| `stack-release.test.mjs` | 194 pass, 1 fail |
+| `stack-release-external.test.mjs` | 25 pass, 0 fail |
+| `stack-instances.test.mjs` | 23 pass, 0 fail |
+
+The failure was my test, not the script. "row 233: --backup last takes the newest backup..." compares
+`untouched(e)` before and after the run, and `untouched` includes `existsSync(e.log)`. The test took its
+before snapshot ahead of `rlog()`, which writes the log, so that flag went from false to true. Every hash in
+the snapshot matched. The fix takes the snapshot after `rlog()`. No other row 233 test calls `untouched`.
+
+### Singles, run 2
+
+`~/darkwing-scratch/r233/rerun.sh` on aacfa2d7a, Sage's slot (sage-1695), 04:13:05Z to 04:21:28Z:
+`stack-release.test.mjs` whole, 195 pass, 0 fail, 0 skipped. The fixed test passed. The other three files
+did not change after run 1. The gate and the mutants run next, in Sage's slot.
+
+### Gate and mutants, 2026-10-21, Sage's slot (sage-1704)
+
+`~/darkwing-scratch/r233/gate.sh` then `mutants.sh`, chained behind `busy.sh`, on d99166ade (code
+aacfa2d7a; d99166ade changes only this file). Counts are an awk sum over `gate-r1.out`.
+
+| Part | Time (UTC) | Result |
+|---|---|---|
+| cli suite (`node --test tests/*.test.mjs`) | 04:29:01-04:37:27 | 616 pass, 0 fail, 0 skipped |
+| the 10 `scripts/test-*.sh` | 04:37:27-04:39:38 | selftests 429 passed, 0 failed; `test-queue`'s node part 156 pass, 0 fail |
+| gate total | | 1201 pass, 0 fail, 11 suites, `GATE_PASSED` |
+| c0 control | 04:39:38-04:40:35 | exit 0 |
+| m1-m8 | 04:40:35-04:41:25 | 8 of 8 killed, every one exit 1 |
+
+Every restore passed its sha256 check, and `git diff` was clean at d99166ade afterwards. The EXIT trap
+removed the `darkwing-r233_default` network.
+
+N1, found in the process sweep after the run: a `sleep 60` was left with its cwd in `packages/cli`.
+`stack-upgrade.test.mjs:312` spawns `flock LOCK sleep 60`, and its `t.after` SIGKILLs only the flock, so
+flock's child outlives the test by up to 60 s. It exited on its own. Row 232's T20
+(`stack-release.test.mjs:370`) spawns its holder `detached` and kills the group with `process.kill(-pid)`.
+The fix is the same two lines. That test is one of m1's killers.
+
+Sage ruled it a fix now (sage-1707), and 08db6ac3f makes it, test only. After the gate's d99166ade
+the only change outside this file is that test:
+
+```
+$ git diff --stat d99166ade 08db6ac3f
+ packages/cli/tests/stack-upgrade.test.mjs | 4 ++--
+ 1 file changed, 2 insertions(+), 2 deletions(-)
+```
+
+The gate stands on Sage's ruling. The rerun is the whole `stack-upgrade.test.mjs` singly plus m1, in
+Sage's slot, with a sweep for a leftover `sleep` afterwards.
@@ -1,71 +0,0 @@
-# Row 241: one decision lands once (#1747, P19 441)
-
-Megavolt, 2026-10-21 (from `date -u`). Brief: `docs/plans/2026-10-20_agent-credentials.md`
-§ "Brief: row 237 follow-up, one decision lands once, with tests at the broker", c25785d3d. Source:
-row 237 residual 3, megavolt-0068, megavolt-0069 and sage-1651. Base: origin/refactor f5753c0e8, row 210's landing.
-Reviewer Duckworth, gate owner Sage. Tests only; no source changes.
-
-## Why landing has no row 237 race
-
-Row 237 fixed publish: two tickets on one decision both pushed, because the decision is spent only
-after the push, and a push to a remote fork can't be taken back. Landing has the same ticket shape
-(`checkLand` opens a ticket, `recordLand` spends the decision), but three guards keep one decision
-from landing twice:
-
-1. **land-busy, the outer guard.** `doLand` in `packages/cli/src/git-socket.mjs:351` refuses
-   `land-busy` while another landing runs. The slot is taken before the first await, and freed only
-   when the landing ends. There is one git socket per launcher, and one host per dataRoot (host.lock).
-   So through the socket two landings never overlap, and a later one stops at `checkLand` with
-   `decision-consumed`. Even past that, applying the same reviewed patch to the new main fails
-   `base-conflict` in `prepareLand`.
-2. **The CAS on main.** `finishLand` (`packages/seat/src/working.mjs:391`) moves main with
-   `update-ref main <commit> <old>`, so a landing prepared on a tip that has since moved refuses
-   `base-moved` and writes nothing.
-3. **The recordLand recheck.** `finishLand` calls `record()`, which is `recordLand`, under the same
-   repository lock. `recordLand` runs `#landFacts` again (`packages/bus/src/broker.mjs:963`), which
-   refuses `decision-consumed` once the decision is spent. When `record()` throws, `finishLand` moves
-   main back.
-
-Guards 2 and 3 can't be reached end to end, because guard 1 stops a second landing first. So they are
-pinned below the socket, and no test bypasses land-busy.
-
-### The unrecorded-commit window
-
-Between the `update-ref` and the end of `record()`, main names the squash, which isn't recorded yet.
-That window is under the repository's lock, so no other push, review or landing in this host sees
-it. If `record()` refuses, main moves back. If moving back fails, `finishLand` refuses
-`rollback-failed` and main keeps naming an unrecorded commit (tested at `working.test.mjs:555`, the
-main.lock case). That is existing behaviour, unchanged here.
-
-## The test
-
-`packages/bus/tests/land.test.mjs:312`. Two tickets pass `checkLand` on one decision before either
-records. `recordLand` on the first lands. `recordLand` on the second refuses `decision-consumed`
-(:319), and leaves one merge `action.allowed`, one RESULT and no `action.refused`. `landRefused`, as
-the socket would call it, then files the refusal and ends the ticket.
-
-**The assert that pins the recordLand recheck is :319.** The `#spent` line is shared with
-`checkLand`, so dropping it also fails :144 (a `checkLand` after the decision is spent). Only :319
-shows the recheck at record time.
-
-Single file on this branch: land.test.mjs 18/18.
-Whole suites on this branch before in-review: bus 500/500, seat 164/164, cli 565/565 (02:52 to 03:01Z).
-
-## Mutants
-
-| Mutant | Change | Killed by |
-|---|---|---|
-| M1 | drop the `#spent` line in `#landFacts` (broker.mjs:963) | land.test.mjs: :312 at :319, and :94 at :144 (16/2) |
-| M2 | drop `land.old` from the `update-ref` in `finishLand` (working.mjs:391) | working.test.mjs:555 at :567, the `base-moved` assert (25/1) |
-| M3 | drop the `land-busy` throw (git-socket.mjs:351) | git-socket.test.mjs:812 at :825, the `land-busy` assert (23/1) |
-
-Each run was one test file on base f5753c0e8 (row 210's landing, which moved the `#spent` line from
-:945 to :963), 2026-10-21 02:51Z. Each assert failed with "Missing expected exception" or "rejection".
-`cmp` confirmed each source matched its saved copy after the restore. M2 is killed by the existing
-working.test:555, so per the brief working.test.mjs isn't changed. Logs: `/mnt/storage/scratch/megavolt-r241/`
-(m1.log to m3.log, with each diff).
-
-## Gate
-
-Per the brief: Duckworth approves the manifest; the bus, seat and cli suites and every
-`scripts/test-*.sh` are green on Sage's gate; the three mutants are killed.
@@ -1,108 +0,0 @@
-# Row 242 build: a retry after publish-timeout pushes once
-
-Row 242, #1748, P19 440. Brief: docs/plans/2026-10-20_agent-credentials.md, "Brief: row 237
-follow-up N1, a retry after publish-timeout pushes once" (9d165be8c). Base: origin/refactor
-8fede5fc6 (row 210 landed f5753c0e8).
-
-## The defect
-
-The git socket polls publishResult until publishDeadlineMs, then refuses publish-timeout
-(packages/cli/src/git-socket.mjs:433) while the bus process's push (runtime publishPush) can
-still be running. The socket then calls publishRefused, which went straight to
-broker.publishRefused: it filed the refusal and deleted the ticket. checkPublish no longer saw an
-open ticket on the decision, so a retry got a new ticket and pushed a second ref while the first
-push was still on its way to the fork.
-
-## Fix chosen: hold the decision until the push ends, then record a push that reached the fork
-
-packages/bus/src/runtime.mjs, `publishRefused` and `settle`:
-
- If the bus has no push for the ticket, publishRefused goes to the broker as before.
- If the push is still running, the refusal code is held on the entry and the ticket stays open.
-  checkPublish refuses another publish on the decision with publish-busy (row 237's check),
-  the same as when the socket is still polling.
- When the push ends, `settle` runs:
-  - A push that reached the fork is recorded with broker.recordPublish, which spends the
-    decision. The bus logs `pushed after the git socket refused <code>; recorded`.
-  - A push that failed is filed as publishRefused with the socket's code.
-  - If recordPublish refuses with a BusError, that code is filed instead.
- If the push has already ended when the refusal arrives, it's settled at once. The sage-1440 test
-  relies on this: its push fails fast, so publish-timeout is filed once, immediately, as before.
-
-Why hold rather than kill. Killing the git client doesn't undo a push the fork has already
-received: receive-pack can update the ref after the client dies. After a kill the bus would have
-to ls-remote the fork to learn what landed, and the result could still race the server. Holding
-waits for the push's own outcome. Each git run in publishRemote already has its own limit
-(publishRemote defaults to checkout.mjs GIT_LIMITS), so a hold is bounded. Holding also reuses the existing publish-busy
-state, and the refusal text is unchanged.
-
-Why record and not only hold. If the hold just filed publish-timeout once the push ended, the
-ticket would close and the decision would be free again with a ref on the fork. A retry would then
-push a second ref. Recording the late push spends the decision, so the retry gets
-decision-consumed.
-
-The publish-timeout refusal text is unchanged (brief condition).
-
-## Test
-
-packages/cli/tests/publish-socket.test.mjs:302, "row 242: a retry after publish-timeout refuses
-publish-busy while the bus process's push runs; the late push is recorded and the decision pushes
-once".
-
- publishDeadlineMs 150 and a store token. The fork's pre-receive hook writes `held`, then loops
-  until the test writes `release`.
- The first publish gives publish-timeout. The test waits for `held`.
- A retry to branch `other` gives publish-busy (:322). Refused events: ["publish-busy"]. No
-  git.push.remote allowed event.
- The test writes `release` and waits until there is exactly one allowed event, with ref
-  refs/heads/coder/feat and the work commit.
- The next retry gives decision-consumed (:330).
- Exactly one git-receive-pack request reached the fake Gitea, and the fork has only
-  refs/heads/coder/feat. Refused events: ["publish-busy", "decision-consumed"]. No TOKEN appears in
-  events or logs.
-
-Fail fast (sage-1681): the body is wrapped in try/finally, and the finally writes `release`. Under
-the first mutant run, the assert at the retry failed and the hook was never released. setup's
-teardown waits for the fake Gitea, which waited for the held push, so the file hung until I stopped
-it. Now a failed assert ends the hook and the run. Each wait is bounded at 10 s with an assert.fail.
-
-## Mutants
-
-Script: /mnt/storage/scratch/megavolt-r242/mutants.sh. Each mutant: save, one-line sed, run
-publish-socket.test.mjs whole, restore, cmp.
-
-| Mutant | Change | Expected kill |
-|---|---|---|
-| M1 | export `publishRefused: (ticket, args) => broker.publishRefused(ticket, args)` again (removes the fix) | :302 at :322: the retry gets a new ticket and its own push, so publish-timeout, not publish-busy (seen in the stopped 03:06Z run) |
-| M2 | `settle` files publishRefused without trying recordPublish (hold only) | :302 at the wait for one allowed event, then :330 |
-
-Results (Sage's slot, sage-1686, 2026-10-21T03:35:00Z to 03:35:15Z): both killed, both restored
-(cmp ok).
-
- M1: publish-socket 8/1, the row 242 test fails at :322 in 0.5 s.
- M2: publish-socket 8/1, the row 242 test fails at :327, the wait for one allowed event, through
-  the assert.fail inside `until` (:311) after its 10 s bound, in 10.3 s.
-
-## Suites
-
-On Sage's slot (sage-1686), on base 8fede5fc6 plus this row:
-
- bus `node --test tests/*.test.mjs`: 499/499, 03:35:20Z to 03:35:34Z. This is one fewer than
-  Sage's 241 gate (500) because the base predates row 241's land.test case.
- cli `node --test tests/*.test.mjs`: 566/566, 03:35:34Z to 03:43:47Z. That's 565 plus the
-  row 242 test.
-
-Singles before the slot: publish-socket.test.mjs 9/9, bus publish.test.mjs 13/13. The seat suite
-and scripts/test-*.sh are left for Sage's landing gate.
-
-## Residuals
-
- The agent was told publish-timeout, but the push is recorded later and the run isn't
-  notified. The allowed event and the bus log line are the only trace. A later publish on the
-  decision gets decision-consumed, which tells the agent the decision was used. Whether the run
-  should get a RESULT for a late record is a follow-up question for Sage.
- A late push whose recordPublish refuses (Duckworth N2) is filed with that code: the ref stays on
-  the fork and the decision is not spent, the same residual as an in-band push whose record
-  refuses.
- The hold lives in the bus process's memory, like the broker's open tickets. A bus restart during
-  a hold drops both, the same as today for any publish in flight.
@@ -669,7 +669,7 @@ pinned copy](../packages/cli/README.md#one-pinned-copy-row-85-1579)).
 #### Rehearsal on a scratch root (D2, row 163)

 ```
-scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]
+scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH|last --port N [--timeout S]
 ```

 `rehearse` starts the target copy's bus host and Console on a copy of a
@@ -680,7 +680,11 @@ runs it against live data. It stops and starts no unit and writes no pointer
 or log line.

 `--backup` takes an activation id under `<dataRoot>/backups/stack/` or an
-absolute path. `--port` is the scratch Console's loopback port, 1024 to
+absolute path. `--backup last` (row 233) takes the newest backup the
+activation log records and prints `PASS backup-last` naming it. With none in
+the log it fails `no-backup`, exits 3 and makes no directory. After the
+backup's checks it prints `BACKUP_STORE <activation id> schema <version>`.
+`--port` is the scratch Console's loopback port, 1024 to
 65535. The run refuses `port-reserved` for 7332, 7333 and the pointer's
 `--port` and `--tester-port`, and `port-in-use` for a port something holds;
 it never picks another port. `--timeout` (default 60 seconds) bounds the
@@ -779,6 +783,82 @@ instance (the business's `launch.by`), for this business, whose run started at
 or after the floor. `status` adds an
 `awaiting units:` line for an open activation.

+Under `--units external`, `activate --plan` also refuses `open-activation`
+while any rollback set holds `AWAITING-UNITS`, and `migrate-required` when
+the target migrates the store and the run lacks `--migrate` (row 233). Under
+`--units user` the plan's bytes are unchanged.
+
+#### Root's upgrade driver (`scripts/stack-upgrade.sh`, row 233)
+
+```
+scripts/stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
+    [--rehearse-port N] [--timeout S]
+scripts/stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
+```
+
+Root runs it from a root-owned checkout of the reviewed commit, never from a
+copy, because the stack's user can write the copies. Each run needs Mos's go,
+and `--migrate` needs Jason present (decision 96). The host must be prepared
+first (`docs/guides/host-run-sheet.md`). Only the default instance, `dev`, is
+accepted; a named instance's units are a later row.
+
+The unit values come from `/etc/mosaic-stack/dev.args`, which I2's install
+writes: one `case2-unit.sh` argument per line, the serve flags after `--`,
+owned by root with no group or other write, exactly one `--copy`, and that
+copy the one both installed units run. Otherwise the run refuses
+`args-unreadable` or `args-differ`. The driver writes the new copy id into
+that file only after `confirm` passes.
+
+`upgrade`, in order. Every stack user step runs as
+`sudo -u USER -H env PATH=<node dir>:/usr/local/bin:/usr/bin:/bin` from the
+user's clone `<home>/src/mosaic-stack`:
+
+1. Takes `flock -n` on `/run/mosaic-stack/upgrade-dev.lock` and holds it to
+   the end, rollback included. A second run refuses `upgrade-running`, and
+   a lock it can't open refuses `lock-failed`.
+2. Fetches the bundle's `refactor` into the clone and fast-forwards it to
+   SHA (`fetch-failed`), then `stack-copy.sh copy --bundle FILE SHA`
+   (`copy-failed`).
+3. A1: this checkout's HEAD is SHA (`head-mismatch`), the copy exists
+   (`copy-missing`), `/usr/local/lib/mosaic-stack/hostcheck.sh --check`
+   exits 0 (`profile-drift`, row 240), both units name one copy
+   (`units-unreadable`, `units-disagree`) and it is the args file's
+   (`args-differ`), and
+   `activate SHA BUSINESS --units external --plan --expect-from OLD` exits 0
+   (`plan-refused`, which covers `open-activation`, `migrate-required` and
+   `pointer-differs`).
+4. The rehearsal, `rehearse SHA BUSINESS --backup last --port 7339`
+   (`--rehearse-port`). Exit 0 goes on. Exit 3 means no backup in the log
+   and goes on only with `--no-rehearsal`, logging `rehearsal-skipped`;
+   without the flag it refuses `no-backup`. Exit 0 with `--no-rehearsal`
+   refuses `rehearsal-required`; any other exit refuses `rehearsal-failed`.
+5. A2 stops the Console, then the bus, and waits for both to read inactive
+   or failed. A3 is `activate --units external`, A4 the render with the new
+   `--copy` (which also starts the bus, then the Console), A5 `confirm`.
+   The driver writes the new args file beside the old one before A2
+   (`args-failed`, exit 3) and moves it into place after A5. A failed move
+   exits 1 `args-failed`: the new copy is live and confirmed, and root puts
+   the file in place by hand.
+
+A failure in A2 or a stage that exits 3 starts the old units again and gates
+them (A2f). A later failure stops both units, runs `revert`, renders the old
+copy and runs `confirm --rollback OLD --cause A3|A4|A5`. `rollback` is the
+same from A1 on, with `rollback BUSINESS --to ID`, no fetch, no copy and no
+rehearsal; `--restore` and `--events` pass through.
+
+Root reads no output the stack user wrote except exit codes and the
+rehearsal's `BACKUP_STORE` line, which it copies into its log after a match
+against a fixed pattern (`BACKUP_STORE none` otherwise). The step log
+`/var/log/mosaic-stack/upgrade-dev.log`, 0640, gets one
+`<time> <step> rc=N argv: ...` line per step and a last
+`<time> result exit=N <code> <text>` line.
+
+Exit 0 ok, 1 failed with the old stack running and gated, 2 rollback-failed
+with both units stopped, 3 refused with nothing stopped, 4 usage. A refusal
+after step 2 leaves the user's clone fast-forwarded and the new copy made. After exit
+2 the activation stays open, so the next run's plan refuses until root
+finishes it.
+
 ## Console service (`scripts/webui-service.sh`)

 ```bash
@@ -0,0 +1,240 @@
+# Host run sheet: preparing a host for the stack
+
+ops-01 runs this on a fresh host before install (I2), and once per host for enrollment. Jason places the
+credentials in section 9. Brief: `docs/plans/2026-10-20_installer-rows.md`, I3 (row 233, #1735). The
+host design is Darkwing's r6 (`agents/darkwing/work/queue-195/DESIGN.md`, section 2) as the brief
+changes it.
+
+The hosts are .53 (profile `factory`) and .54 (profile `library`). Both get every section unless a
+section says otherwise. Every command runs as root unless it starts with `sudo -u mosaic`.
+
+Each item names the `scripts/case2-precheck.sh` line that verifies it. Install's precheck runs as the
+stack's user with `--business ID --profile factory|library`. Where no precheck line checks an item, the
+item says "precheck line missing", and Gosalyn adds the line to I2. Today's precheck (C2-13a) has these
+gaps; C2-13b adds the library lines (`library.sync`, persona modes, no Discord binding).
+
+## 1. The stack's user and its home
+
+```sh
+useradd --system --user-group --home-dir /srv/mosaic --create-home --shell /usr/sbin/nologin mosaic
+chmod 0700 /srv/mosaic
+install -d -o root -g root -m 0755 /etc/mosaic-stack
+install -d -o root -g root -m 0750 /var/log/mosaic-stack
+```
+
+`mosaic` gets no sudo rule and no supplementary group. Check with `sudo -l -U mosaic` (expect "not allowed
+to run sudo") and `id -G mosaic` (one number).
+
+Precheck lines:
+- `ok      HOME is /srv/mosaic`
+- `ok      sudo -n true fails as mosaic`
+- `ok      id -G prints one group (<gid>)`
+- `/srv/mosaic` at 0700: precheck line missing.
+
+## 2. node, pinned under /opt
+
+The pin is the version and sha256 the window names. `<v>` below is that version, for example `v24.x.y`.
+
+```sh
+install -d -o root -g root -m 0755 /etc/mosaic-stack/node-keys /opt/mosaic-stack
+cd "$(mktemp -d)"
+curl -fsSLO https://nodejs.org/dist/<v>/node-<v>-linux-x64.tar.xz
+curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt
+curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt.asc
+gpg --no-default-keyring --keyring /etc/mosaic-stack/node-keys/pubring.kbx --verify SHASUMS256.txt.asc SHASUMS256.txt
+grep " node-<v>-linux-x64.tar.xz\$" SHASUMS256.txt | sha256sum -c -
+echo "<pinned sha256>  node-<v>-linux-x64.tar.xz" | sha256sum -c -
+tar -xJf node-<v>-linux-x64.tar.xz --no-same-owner
+mv node-<v>-linux-x64 /opt/mosaic-stack/node-<v>
+chown -R root:root /opt/mosaic-stack/node-<v>
+chmod -R go-w /opt/mosaic-stack/node-<v>
+```
+
+The keyring holds the Node release keys from the nodejs/release-keys repository, nothing else. Stop on any
+failed check; don't unpack. Rollback: `rm -r /opt/mosaic-stack/node-<v>`.
+
+Precheck lines:
+- `ok      node <v>` (version 24 or later only).
+- The `--node` path in the args file must be a root-owned regular file outside the home. `case2-unit.sh`
+  refuses otherwise at render, which is not a precheck line.
+- The installed version equals the pin: precheck line missing.
+
+## 3. claude, factory only (.53)
+
+Root installs it under `/opt`, never under the stack's home (this replaces r6's `/srv/mosaic/.local/bin`).
+The pin is 2.1.304 unless the window names a newer one.
+
+```sh
+install -d -o root -g root -m 0755 /opt/mosaic-stack/claude-<pin> /etc/claude-code
+npm install --prefix /opt/mosaic-stack/claude-<pin> @anthropic-ai/claude-code@<pin>
+chown -R root:root /opt/mosaic-stack/claude-<pin>
+printf '%s\n' '{"env":{"DISABLE_AUTOUPDATER":"1"}}' > /etc/claude-code/managed-settings.json
+chown root:root /etc/claude-code/managed-settings.json
+chmod 0644 /etc/claude-code/managed-settings.json
+```
+
+Rollback: `rm -r /opt/mosaic-stack/claude-<pin> /etc/claude-code/managed-settings.json`.
+
+The args file's `--claude` names the binary under `/opt/mosaic-stack/claude-<pin>`. `case2-unit.sh`
+refuses one that isn't a root-owned regular file outside the home.
+
+Precheck lines:
+- `ok      claude replaces its prompt with --system-prompt-file (<n> input tokens)`. It runs the claude the bus
+  unit's `MOSAIC_CLAUDE_BIN` names, so it needs the model credential from section 9.
+- Managed settings present with the autoupdater off: precheck line missing.
+
+## 4. bubblewrap, AppArmor and the drift stop
+
+Both hosts. These are r6's items, unchanged.
+
+```sh
+apt-get install bubblewrap=0.9.0-1ubuntu0.3
+stat -c '%U:%G %a' /usr/bin/bwrap > /etc/mosaic-stack/bwrap.orig    # expect root:root 755
+
+cd "$(mktemp -d)"
+apt-get download apparmor-profiles=4.0.1really4.0.1-0ubuntu0.24.04.9
+dpkg-deb --fsys-tarfile apparmor-profiles_*.deb | tar -x ./usr/share/apparmor/extra-profiles/bwrap-userns-restrict
+echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9  usr/share/apparmor/extra-profiles/bwrap-userns-restrict" | sha256sum -c -
+install -o root -g root -m 0644 usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/bwrap-userns-restrict
+apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict
+
+dpkg-statoverride --update --add root mosaic 0750 /usr/bin/bwrap
+```
+
+`aa-status` then lists `bwrap` and `unpriv_bwrap` in enforce mode. Install the one profile file, never the
+whole `apparmor-profiles` package, whose 20 other profiles would load at the next boot. Rollback, in
+reverse order: `dpkg-statoverride --remove /usr/bin/bwrap`, then `chown` and `chmod` to the values in
+`bwrap.orig`; `apparmor_parser -R /etc/apparmor.d/bwrap-userns-restrict`, then `rm` it;
+`apt-get remove bubblewrap`.
+
+The no-new-privs test must print `bwrap//&unpriv_bwrap (enforce)`:
+
+```sh
+systemd-run --wait --pipe --uid=mosaic --gid=mosaic -p NoNewPrivileges=yes -p ProtectSystem=strict \
+  -p ProtectHome=yes -p PrivateTmp=yes \
+  /usr/bin/bwrap --unshare-user --unshare-pid --ro-bind / / cat /proc/self/attr/current
+```
+
+The P9 checks: another user's `/usr/bin/bwrap --version` fails with Permission denied, `getent group mosaic`
+lists no member, and `sysctl fs.protected_hardlinks` prints 1. `sysctl
+kernel.apparmor_restrict_unprivileged_userns` prints 1 too (Ubuntu 24.04's default); leave it, so nothing
+else on the host gains user namespaces.
+
+The drift stop is row 240's (I4, #1746): `hostcheck.sh` and its pins in `/usr/local/lib/mosaic-stack/`, the
+apt hook `/etc/apt/apt.conf.d/80mosaic-hostcheck`
+(`DPkg::Post-Invoke { "/usr/local/lib/mosaic-stack/hostcheck.sh --enforce || true"; };`) and the boot unit
+`mosaic-hostcheck.service`. Install them from row 240's run sheet lines once it lands. Until then
+`stack-upgrade.sh` refuses `profile-drift` on every run, because it runs `hostcheck.sh --check` and a
+missing file refuses.
+
+Precheck lines:
+- `ok      bwrap --unshare-all runs`
+- The AppArmor profile, the statoverride, the no-new-privs result and the P9 checks: precheck line missing.
+  A line that runs `/usr/local/lib/mosaic-stack/hostcheck.sh --check` would cover them once row 240 lands.
+
+## 5. The port floor
+
+Leave `net.ipv4.ip_unprivileged_port_start` at its default (1024) or higher. The Console's port is below
+it and the unit gets `CAP_NET_BIND_SERVICE`, so no other `mosaic` process can bind that port.
+
+Precheck line: `ok      net.ipv4.ip_unprivileged_port_start <n> is above the Console's port <port>`.
+
+## 6. The release transport
+
+A release is a git bundle of `refs/heads/refactor` plus the 40-character commit id (I1, mos-1118 choice 2).
+Mos's go names both, with the bundle's sha256. `<sha>` below is that id.
+
+First install:
+
+```sh
+# Copy the bundle in, then check it before anything reads it.
+echo "<bundle sha256>  /root/<sha>.bundle" | sha256sum -c -
+git bundle verify /root/<sha>.bundle
+
+# Root's checkout. stack-upgrade.sh and case2-unit.sh run from here, never from a copy.
+git clone -q -b refactor /root/<sha>.bundle /root/mosaic-stack
+git -C /root/mosaic-stack checkout -q --detach <sha>
+git -C /root/mosaic-stack rev-parse HEAD        # must print <sha>
+
+# The stack's user's copy of the bundle, and its own clone.
+install -d -o mosaic -g mosaic -m 0700 /srv/mosaic/incoming /srv/mosaic/src
+install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
+sudo -u mosaic -H git clone -q -b refactor /srv/mosaic/incoming/<sha>.bundle /srv/mosaic/src/mosaic-stack
+```
+
+Each upgrade, before `stack-upgrade.sh upgrade`:
+
+```sh
+echo "<bundle sha256>  /root/<sha>.bundle" | sha256sum -c -
+git bundle verify /root/<sha>.bundle
+git -C /root/mosaic-stack fetch -q /root/<sha>.bundle refs/heads/refactor:refs/remotes/origin/refactor
+git -C /root/mosaic-stack checkout -q --detach <sha>
+install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
+/root/mosaic-stack/scripts/stack-upgrade.sh upgrade --instance dev --bundle /srv/mosaic/incoming/<sha>.bundle --sha <sha>
+```
+
+The driver fetches the bundle into the stack's user's clone and makes the copy itself. It refuses
+`head-mismatch` when root's checkout isn't at `<sha>`. Usage, exits and the step log are in
+`docs/TOOLS.md`, under "Root's upgrade driver". Each run needs Mos's go, and `--migrate` needs Jason present
+(decision 96).
+
+When a run doesn't exit 0, send Mos the exit and the last lines of `/var/log/mosaic-stack/upgrade-dev.log`:
+- Exit 3: refused before anything stopped. The last line names the code. The stack's user's clone and the
+  new copy may already hold the new commit; the units still run the old one.
+- Exit 1: the old copy runs and passed its gate. Either the driver restarted it after A2 or a refused A3
+  (A2f), or it rolled back after A3, A4 or A5.
+- Exit 2: both units are stopped and the activation stays open, so the next run refuses at its plan until
+  it is finished with `confirm` or `revert`. One way here: a unit that doesn't stop within `--timeout`
+  keeps its old start time, A2f's gate fails it `stale-start`, and the driver stops both (duckworth-0055
+  N1). That is on purpose; a unit that won't stop is a reason to stop.
+
+Precheck lines: none. Root's checkout at `<sha>` and the stack's user's clone: precheck line missing.
+After install, `stack-copy.sh check --units external` reads the copy the units run.
+
+## 7. The args file
+
+`/etc/mosaic-stack/dev.args` holds the `case2-unit.sh` arguments, one per line, with the serve flags after a
+line holding `--`. I2's install writes it; ops-01 doesn't write it by hand. It must be root's, with no group
+or other write (`chmod 0644` or tighter). Before `--` it takes only `--user`, `--home`, `--business`,
+`--copy` (exactly once), `--node`, `--profile`, `--claude` and `--pm`. `stack-upgrade.sh` changes only the
+`--copy` line, and only after confirm passes.
+
+Precheck line: none. `stack-upgrade.sh` refuses `args-unreadable` on any of these rules.
+
+## 8. Enrollment, once per host
+
+The machine-grant enrollment is Gizmoduck's design r6, commit f9039682e, section 3, steps E1-E4. ops-01's
+Q-O1 is answered (mos-1125). The host side is the `tools/host/` files: `mosaic-keys`, `mosaic-keys-write`,
+their sudoers line and the sshd drop-in `/etc/ssh/sshd_config.d/<nn>-mosaic-keys.conf`. The verbs that run
+it (`hosts.enroll`, the `host-enroll` decision) are rows 234 and 235, not yet on refactor. This item has no
+commands until they land.
+
+## 9. Credential slots
+
+Jason places these, all in one sitting per host. No value appears here or in any message. Check a placed
+file with `stat -c '%U %a %n' PATH`, never `cat`. `<business>` is the host's business id.
+
+### .53, factory
+
+| Path | File | Mode | Owner | Profile | Precheck line |
+|---|---|---|---|---|---|
+| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `CLAUDE_CODE_OAUTH_TOKEN=...` (Q185) | 0600 | mosaic | factory | `claude replaces its prompt with --system-prompt-file` runs claude with it. The file's mode and owner: precheck line missing. |
+| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
+
+### .54, library
+
+| Path | File | Mode | Owner | Profile | Precheck line |
+|---|---|---|---|---|---|
+| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `ZAI_API_KEY=...` | 0600 | mosaic | library | `<path> sets ZAI_API_KEY` |
+| `/srv/mosaic/.config/mosaic-dev/secrets/` | the Console's proof file, the path the args file's `--proof-file` names | 0600 or 0400 | mosaic | library | none. `case2-unit.sh` refuses at render. Precheck line missing. |
+| `/srv/mosaic/.config/mosaic-dev/secrets/<business>/` | `library-gitea` | 0600 | mosaic | library | C2-13b's `library.sync` (not landed) |
+| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
+
+The installer key, `<configDir>/secrets/hosts/<business>/<hostId>/installer`, is not a slot anyone places.
+Enrollment (section 8) writes it under the enrolling stack's config directory and deletes it when the host
+leaves.
+
+## 10. Console start
+
+None by hand. I2's `case2-unit.sh` enables both units, starts the bus, then the Console (Q189, mos-1122).
+`stack-upgrade.sh` stops and starts both units itself.
@@ -33,7 +33,7 @@ Gaps found while working go to `docs/plans/DEFERRED.md`, not here.

 <!-- mosaic-queue:begin -->

-Generated from `docs/plans/queue.json` revision 2638 by `queue render`. Do not edit between the markers; change the queue with `scripts/mosaic queue`.
+Generated from `docs/plans/queue.json` revision 2627 by `queue render`. Do not edit between the markers; change the queue with `scripts/mosaic queue`.

 | # | Piece | Owner | Issues | State | After | Gate | Brief | Note |
 |---|---|---|---|---|---|---|---|---|
@@ -268,8 +268,8 @@ Generated from `docs/plans/queue.json` revision 2638 by `queue render`. Do not e
 | 229 | Row B: per-instance pi agent dir for Chet's Codex login, design only, starts after 10-27 (#428) | gosalyn; reviewers webby | #1730 | queued | — | A live check, with Jason's or Mos's go, that two clients on one ChatGPT login don't revoke each other's refresh tokens. No build row starts before it passes. Held until after 10-27. (sage) | `docs/plans/2026-10-20_case2-rows.md` § Brief: row B, a per-instance pi agent dir for Chet's Codex login (#428) | — |
 | 230 | Case 2 C2-13a unit render and host precheck (case2-unit.sh, case2-precheck.sh) | gosalyn; reviewers webby | #1732 | done | — | Webby approves the manifest. The test file and the cli suite are green on Sage's gate, along with every scripts/test-*.sh. A run on a host is a separate window with Jason's go. (sage) | `docs/plans/2026-10-20_case2-rows.md` § C2-13a unit render and host precheck | Landed c8bc5cf4c on refactor (base 390ea5261), pushed. Patch-id f4b49eb68997 matches git diff 722136772..96768766c on the 5 manifest paths (manifest fbb0d7e42c67, Webby approve r2 rev 2563). packages/cli/README.md 3-way with row 199, hunks apart. Gosalyn gate on 96768766c: cli 541/0, scripts 429/0, mutants 10/10. Merged tree check: case2-unit 30/0, host 42/0. |
 | 231 | Case 2 C2-01b survey escapes, list item lines and attachment embeds (row 212 N1-N3) | gosalyn; reviewers webby | #1733 | briefed | 212 done | Webby approves the manifest. The library tests are green on Sage's gate, along with every scripts/test-*.sh. (sage) | `docs/plans/2026-10-20_case2-rows.md` § C2-01b survey escapes, list item lines and attachment embeds | — |
-| 232 | Installer I1: stack-release external units and stack-copy bundle copies (MVP, Q187, Vikunja #432) | darkwing; reviewers filbert | #1734 | done | — | Filbert approves the manifest. The cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) | `docs/plans/2026-10-20_installer-rows.md` § I1 stack-release external units and bundle copies | Landed e65a3ab4d on refactor (base 0444acd08), pushed. Patch-id 5adfbab9 matches git diff 8fede5fc6..da2da2f6c on the 9 manifest paths; tree entries (modes too) equal da2da2f6c, files pass manifest 7af3c943. Filbert approve r1 29829, r2 29865. Darkwing's landing gate on da2da2f6c: cli 593/0, scripts 429/0 (+156 queue node tests). Mutants 17/17. |
-| 233 | Installer I3: stack-upgrade.sh root driver and the ops-01 host run sheet (MVP, Q187, Vikunja #432) | darkwing; reviewers filbert | #1735 | in-progress | 232 done | Filbert approves the manifest. The test file, the cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) | `docs/plans/2026-10-20_installer-rows.md` § I3 the upgrade command and the host run sheet | — |
+| 232 | Installer I1: stack-release external units and stack-copy bundle copies (MVP, Q187, Vikunja #432) | darkwing; reviewers filbert | #1734 | in-review, round 2, request posted | — | Filbert approves the manifest. The cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) | `docs/plans/2026-10-20_installer-rows.md` § I1 stack-release external units and bundle copies | — |
+| 233 | Installer I3: stack-upgrade.sh root driver and the ops-01 host run sheet (MVP, Q187, Vikunja #432) | darkwing; reviewers filbert | #1735 | briefed | 232 done | Filbert approves the manifest. The test file, the cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) | `docs/plans/2026-10-20_installer-rows.md` § I3 the upgrade command and the host run sheet | — |
 | 234 | Machine grant row a: host enrollment, store records and migration 3i-3j (#436a, design r6 f9039682e, lands by 10-25 before the .53/.54 installs) | gizmoduck; reviewers duckworth | #1740 | in-progress | 210 done | Duckworth approves the manifest. The bus, business and cli suites and every scripts/test-*.sh are green on Sage's gate. The design's section 10 row a tests and mutants pass, the migration test included. (sage) (sage) | `docs/plans/2026-10-20_host-credentials.md` § Machine grant row a: enrollment, store records and migration 3i-3j | — |
 | 235 | Machine grant row b: raise binding, executor, reconcile and revoke (#436, design r6 f9039682e, C1, N1, G27, G28) | gizmoduck; reviewers duckworth | #1741 | queued | 234 done | Duckworth approves the manifest. The bus, business and cli suites and every scripts/test-*.sh are green on Sage's gate. Section 10's row b tests and mutants pass, G27, G28 and the upsert mutant included. (sage) (sage) | `docs/plans/2026-10-20_host-credentials.md` § Machine grant row b: raise binding, executor, reconcile and revoke | — |
 | 236 | #436b Gitea grant path build: seat accounts and tokens through an approved grant, schema 3j-3k (design r5 1be535aca, due 10-27 at risk; live waits for Jason's section 8 approval and bootstrap step) | megavolt; reviewers dewey | #1742 | queued | 210 done, 234 done | Dewey approves the manifest. The bus, business and cli suites and every scripts/test-*.sh are green on Sage's gate. Section 9's mutants are killed, N9's restart test included. (sage) (sage) | `docs/plans/2026-10-20_agent-credentials.md` § Brief: #436b Gitea grant path, build (schema 3j-3k) | — |
@@ -277,8 +277,8 @@ Generated from `docs/plans/queue.json` revision 2638 by `queue render`. Do not e
 | 238 | row 200 follow-up N1: the sso-config hint carries --instance NAME (#1744) | launchpad; reviewers webby | #1744 | in-review, round 1, request posted | 200 done | Webby approves the manifest; stack-release.test.mjs whole, cli suite and every scripts/test-*.sh green on Sage's gate; the mutant killed (sage) | `docs/plans/2026-10-20_stack-release-sso.md` § Brief: row 200 follow-up N1, the sso-config hint names the instance | — |
 | 239 | Row 194 follow-up: the credential revoke runs under the writer's lock, refusals after a revoke say so, and a stale-revision fire test (duckworth-0074, mos-1158) | honker; reviewers duckworth | #1745 | queued | 194 done | Duckworth approves the manifest. The bus, business, cli and webui suites and every scripts/test-*.sh are green on Sage's gate, and the mutants are killed. (sage) (sage) | `docs/plans/2026-10-20_agent-lifecycle.md` § Brief: row 194 follow-up, the revoke under the writer's lock and a fire test | — |
 | 240 | Installer I4: hostcheck, the bwrap profile check, pins and boot unit, from r6 B2 (MVP, Q187, Vikunja #432; split from I3) | darkwing; reviewers filbert | #1746 | queued | 233 done | Filbert approves the manifest. The test file, the cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) (sage) | `docs/plans/2026-10-20_installer-rows.md` § I4 hostcheck: the bwrap profile check, pins and boot unit | — |
-| 241 | Row 237 follow-up: one decision lands once, a two-ticket broker land test and three mutants on the landing guards (megavolt-0069, sage-1651, P19 441) | megavolt; reviewers duckworth | #1747 | done | 237 done | Duckworth approves the manifest. The bus, seat and cli suites and every scripts/test-*.sh are green on Sage's gate, and the three mutants are killed. (sage) (sage) | `docs/plans/2026-10-20_agent-credentials.md` § Brief: row 237 follow-up, one decision lands once, with tests at the broker | Landed 5d4a61776 on refactor, pushed (base e65a3ab4d). Blobs from 3e3ed303e on 2 paths; patch-id a1649f80 equals git diff f5753c0e8..3e3ed303e on those paths; ls-tree modes equal; manifest e8266864 verifies on the landed files. Sage slot on bbf3b1ee8: M1-M3 killed; gate 15 packages 2797/0, 10 script suites 429/0. Duckworth approve 29870 rev 2628. |
-| 242 | Row 237 follow-up N1: a retry after publish-timeout pushes once; hold the decision or kill and wait for the push, a slow-push retry test and a mutant (duckworth-0069, P19 440) | megavolt; reviewers duckworth | #1748 | in-review, round 1, request posted | 237 done | Duckworth approves the manifest. The bus and cli suites and every scripts/test-*.sh are green on Sage's gate, and the mutant is killed. (sage) (sage) | `docs/plans/2026-10-20_agent-credentials.md` § Brief: row 237 follow-up N1, a retry after publish-timeout pushes once | — |
+| 241 | Row 237 follow-up: one decision lands once, a two-ticket broker land test and three mutants on the landing guards (megavolt-0069, sage-1651, P19 441) | megavolt; reviewers duckworth | #1747 | in-review, round 1, request posted | 237 done | Duckworth approves the manifest. The bus, seat and cli suites and every scripts/test-*.sh are green on Sage's gate, and the three mutants are killed. (sage) (sage) | `docs/plans/2026-10-20_agent-credentials.md` § Brief: row 237 follow-up, one decision lands once, with tests at the broker | — |
+| 242 | Row 237 follow-up N1: a retry after publish-timeout pushes once; hold the decision or kill and wait for the push, a slow-push retry test and a mutant (duckworth-0069, P19 440) | megavolt; reviewers duckworth | #1748 | in-progress | 237 done | Duckworth approves the manifest. The bus and cli suites and every scripts/test-*.sh are green on Sage's gate, and the mutant is killed. (sage) (sage) | `docs/plans/2026-10-20_agent-credentials.md` § Brief: row 237 follow-up N1, a retry after publish-timeout pushes once | — |

 <!-- mosaic-queue:end -->

@@ -1,7 +1,7 @@
 {
  "version": 1,
  "canonicalRoot": "/mnt/storage/src/mosaic-stack",
-  "revision": 2638,
+  "revision": 2627,
  "rows": [
    {
      "id": 1,
@@ -22631,7 +22631,7 @@
      "closes": [
        1734
      ],
-      "state": "done",
+      "state": "in-review",
      "previousState": null,
      "required": false,
      "requiredSince": null,
@@ -22730,12 +22730,15 @@
          }
        ]
      },
-      "claim": null,
-      "note": "Landed e65a3ab4d on refactor (base 0444acd08), pushed. Patch-id 5adfbab9 matches git diff 8fede5fc6..da2da2f6c on the 9 manifest paths; tree entries (modes too) equal da2da2f6c, files pass manifest 7af3c943. Filbert approve r1 29829, r2 29865. Darkwing's landing gate on da2da2f6c: cli 593/0, scripts 429/0 (+156 queue node tests). Mutants 17/17.",
+      "claim": {
+        "seat": "darkwing",
+        "op": "darkwing-232-inprogress-r1"
+      },
+      "note": null,
      "blockedReason": null,
      "createdAt": "2026-10-20T22:56:29.929Z",
-      "updatedAt": "2026-10-21T03:20:36.830Z",
-      "updatedBy": "sage"
+      "updatedAt": "2026-10-21T03:01:22.609Z",
+      "updatedBy": "filbert"
    },
    {
      "id": 233,
@@ -22747,7 +22750,7 @@
      "closes": [
        1735
      ],
-      "state": "in-progress",
+      "state": "briefed",
      "previousState": null,
      "required": false,
      "requiredSince": null,
@@ -22756,7 +22759,7 @@
      "brief": {
        "path": "docs/plans/2026-10-20_installer-rows.md",
        "anchor": "I3 the upgrade command and the host run sheet",
-        "blob": "e61291bcf1c6e9f43f599ca94fe9c6eb0dce1383"
+        "blob": "f35e7e57daee39a760159e069994f213d733b167"
      },
      "after": [
        {
@@ -22768,15 +22771,12 @@
        "filbert"
      ],
      "review": null,
-      "claim": {
-        "seat": "darkwing",
-        "op": "darkwing-233-inprogress-r1"
-      },
+      "claim": null,
      "note": null,
      "blockedReason": null,
      "createdAt": "2026-10-20T22:56:59.719Z",
-      "updatedAt": "2026-10-21T03:25:49.757Z",
-      "updatedBy": "darkwing"
+      "updatedAt": "2026-10-21T01:30:41.511Z",
+      "updatedBy": "sage"
    },
    {
      "id": 234,
@@ -23178,7 +23178,7 @@
      "closes": [
        1747
      ],
-      "state": "done",
+      "state": "in-review",
      "previousState": null,
      "required": false,
      "requiredSince": null,
@@ -23247,24 +23247,18 @@
              }
            ],
            "duplicateRisk": false,
-            "receipts": [
-              {
-                "reviewer": "duckworth",
-                "op": "duckworth-r241-record-1",
-                "at": "2026-10-21T03:06:16.441Z",
-                "verdict": "approve",
-                "comment": 29870,
-                "candidate": "e82668642589eefca8b22433f1b0c7a24432f1993c0050f0a4b74c1146c93c2f"
-              }
-            ]
+            "receipts": []
          }
        ]
      },
-      "claim": null,
-      "note": "Landed 5d4a61776 on refactor, pushed (base e65a3ab4d). Blobs from 3e3ed303e on 2 paths; patch-id a1649f80 equals git diff f5753c0e8..3e3ed303e on those paths; ls-tree modes equal; manifest e8266864 verifies on the landed files. Sage slot on bbf3b1ee8: M1-M3 killed; gate 15 packages 2797/0, 10 script suites 429/0. Duckworth approve 29870 rev 2628.",
+      "claim": {
+        "seat": "megavolt",
+        "op": "megavolt-241-start-1"
+      },
+      "note": null,
      "blockedReason": null,
      "createdAt": "2026-10-21T01:52:55.772Z",
-      "updatedAt": "2026-10-21T03:35:52.657Z",
+      "updatedAt": "2026-10-21T03:04:33.826Z",
      "updatedBy": "sage"
    },
    {
@@ -23277,7 +23271,7 @@
      "closes": [
        1748
      ],
-      "state": "in-review",
+      "state": "in-progress",
      "previousState": null,
      "required": false,
      "requiredSince": null,
@@ -23297,55 +23291,7 @@
      "reviewers": [
        "duckworth"
      ],
-      "review": {
-        "rounds": [
-          {
-            "n": 1,
-            "op": "megavolt-242-inreview-r1",
-            "by": "megavolt",
-            "at": "2026-10-21T03:44:17.331Z",
-            "issue": 1748,
-            "candidate": {
-              "kind": "manifest",
-              "digest": "1fbfab6c68e8d76d2d69b13c415257280d4fa8c5584e24d5dfd13dca1420a3da",
-              "text": "d245f805bfc9ad8a0064a8898effa05148e9c3fb7deb22b85f7af499e0ded369  agents/megavolt/work/queue-242/BUILD.md\ncfecc67698ad18a94288ba4e852b372d92c8d1cb984494256f929b03a467baad  packages/bus/src/runtime.mjs\na0783575b7da4e3b8e1ae161d7031fdf62a8a1a402a48ac898ef1b5c2d378d9b  packages/cli/tests/publish-socket.test.mjs\n"
-            },
-            "request": "comment",
-            "attempts": [
-              {
-                "op": "megavolt-242-inreview-r1",
-                "by": "megavolt",
-                "at": "2026-10-21T03:44:17.331Z",
-                "state": "failed",
-                "comment": null,
-                "transport": {
-                  "outcome": "failed",
-                  "status": null,
-                  "comment": null,
-                  "detail": "pre-send: the credential or account check failed"
-                },
-                "resolutions": []
-              },
-              {
-                "op": "sage-242-request-1",
-                "by": "sage",
-                "at": "2026-10-21T03:45:41.063Z",
-                "state": "posted",
-                "comment": 29888,
-                "transport": {
-                  "outcome": "posted",
-                  "status": 201,
-                  "comment": 29888,
-                  "detail": "created"
-                },
-                "resolutions": []
-              }
-            ],
-            "duplicateRisk": false,
-            "receipts": []
-          }
-        ]
-      },
+      "review": null,
      "claim": {
        "seat": "megavolt",
        "op": "megavolt-242-start-1"
@@ -23353,8 +23299,8 @@
      "note": null,
      "blockedReason": null,
      "createdAt": "2026-10-21T01:54:53.103Z",
-      "updatedAt": "2026-10-21T03:46:05.721Z",
-      "updatedBy": "sage"
+      "updatedAt": "2026-10-21T03:03:20.738Z",
+      "updatedBy": "megavolt"
    }
  ],
  "log": [
@@ -93011,268 +92957,6 @@
        "receipt": "ok sage-241-request-1-resolve rev 2627 row 241 round 1 request sage-241-request-1 requesting→posted comment 29868"
      },
      "viewSha": "ec48b0f767195fb8ddaa8e67e9526a533ef4ddc789d6b11c0998befbf32433de"
-    },
-    {
-      "rev": 2628,
-      "op": "duckworth-r241-record-1",
-      "verb": "review-record",
-      "args": {
-        "id": 241,
-        "verdict": "approve",
-        "comment": 29870,
-        "candidate": "e82668642589eefca8b22433f1b0c7a24432f1993c0050f0a4b74c1146c93c2f"
-      },
-      "by": "duckworth",
-      "at": "2026-10-21T03:06:16.441Z",
-      "semantics": 2,
-      "result": {
-        "row": 241,
-        "round": 1,
-        "verdict": "approve",
-        "receipt": "ok duckworth-r241-record-1 rev 2628 row 241 round 1 approve by duckworth (comment 29870)"
-      },
-      "viewSha": "094f036ccb430caef42be43e00643081c1ed2f7ec8253b865faba8dd28ad7298"
-    },
-    {
-      "rev": 2629,
-      "op": "sage-232-landing-note",
-      "verb": "note",
-      "args": {
-        "id": 232,
-        "text": "Landed e65a3ab4d on refactor (base 0444acd08), pushed. Patch-id 5adfbab9 matches git diff 8fede5fc6..da2da2f6c on the 9 manifest paths; tree entries (modes too) equal da2da2f6c, files pass manifest 7af3c943. Filbert approve r1 29829, r2 29865. Darkwing's landing gate on da2da2f6c: cli 593/0, scripts 429/0 (+156 queue node tests). Mutants 17/17."
-      },
-      "by": "sage",
-      "at": "2026-10-21T03:20:12.732Z",
-      "semantics": 2,
-      "result": {
-        "row": 232,
-        "field": "note",
-        "receipt": "ok sage-232-landing-note rev 2629 row 232 note"
-      },
-      "viewSha": "e0f5ff1d8b8c1592a39c64a2df6b32d4d8f2a65e4ee095566a01989d491250ea"
-    },
-    {
-      "rev": 2630,
-      "op": "sage-232-done",
-      "verb": "move",
-      "args": {
-        "id": 232,
-        "to": "done",
-        "reason": null,
-        "candidate": null,
-        "evidence": null,
-        "issue": null
-      },
-      "by": "sage",
-      "at": "2026-10-21T03:20:36.830Z",
-      "semantics": 2,
-      "result": {
-        "row": 232,
-        "from": "in-review",
-        "to": "done",
-        "round": 2,
-        "issue": null,
-        "candidate": null,
-        "receipt": "ok sage-232-done rev 2630 row 232 in-review→done round 2"
-      },
-      "viewSha": "8d3708d10c659ec2d626c28be1cb87089cd606574278e9b51b99fae515928050"
-    },
-    {
-      "rev": 2631,
-      "op": "sage-233-repin-brief-1",
-      "verb": "set",
-      "args": {
-        "id": 233,
-        "field": "brief",
-        "value": "docs/plans/2026-10-20_installer-rows.md#I3 the upgrade command and the host run sheet",
-        "reason": null
-      },
-      "by": "sage",
-      "at": "2026-10-21T03:25:06.360Z",
-      "semantics": 2,
-      "result": {
-        "row": 233,
-        "field": "brief",
-        "from": {
-          "path": "docs/plans/2026-10-20_installer-rows.md",
-          "anchor": "I3 the upgrade command and the host run sheet",
-          "blob": "f35e7e57daee39a760159e069994f213d733b167"
-        },
-        "to": {
-          "path": "docs/plans/2026-10-20_installer-rows.md",
-          "anchor": "I3 the upgrade command and the host run sheet",
-          "blob": "e61291bcf1c6e9f43f599ca94fe9c6eb0dce1383"
-        },
-        "receipt": "ok sage-233-repin-brief-1 rev 2631 row 233 brief: docs/plans/2026-10-20_installer-rows.md § I3 the upgrade command and the host run sheet @f35e7e57daee→docs/plans/2026-10-20_installer-rows.md § I3 the upgrade command and the host run sheet @e61291bcf1c6"
-      },
-      "viewSha": "da749491f801e3ec47f908872aa4453fc3caa30b23e4194a205ddfd9d076241e"
-    },
-    {
-      "rev": 2632,
-      "op": "darkwing-233-inprogress-r1",
-      "verb": "move",
-      "args": {
-        "id": 233,
-        "to": "in-progress",
-        "reason": null,
-        "candidate": null,
-        "evidence": null,
-        "issue": null
-      },
-      "by": "darkwing",
-      "at": "2026-10-21T03:25:49.757Z",
-      "semantics": 2,
-      "result": {
-        "row": 233,
-        "from": "briefed",
-        "to": "in-progress",
-        "round": null,
-        "issue": null,
-        "candidate": null,
-        "receipt": "ok darkwing-233-inprogress-r1 rev 2632 row 233 briefed→in-progress"
-      },
-      "viewSha": "f77ca73f963f35d77733bfe6fa5e8ad3a4144d3ae6b7aa2fcba1d62eb3da4e12"
-    },
-    {
-      "rev": 2633,
-      "op": "sage-241-landed-note",
-      "verb": "note",
-      "args": {
-        "id": 241,
-        "text": "Landed 5d4a61776 on refactor, pushed (base e65a3ab4d). Blobs from 3e3ed303e on 2 paths; patch-id a1649f80 equals git diff f5753c0e8..3e3ed303e on those paths; ls-tree modes equal; manifest e8266864 verifies on the landed files. Sage slot on bbf3b1ee8: M1-M3 killed; gate 15 packages 2797/0, 10 script suites 429/0. Duckworth approve 29870 rev 2628."
-      },
-      "by": "sage",
-      "at": "2026-10-21T03:35:10.676Z",
-      "semantics": 2,
-      "result": {
-        "row": 241,
-        "field": "note",
-        "receipt": "ok sage-241-landed-note rev 2633 row 241 note"
-      },
-      "viewSha": "60a6381f3fa3ff4ca5774baed90a9b383fd69b09bf88afc6c5aa123fa0379208"
-    },
-    {
-      "rev": 2634,
-      "op": "sage-241-done",
-      "verb": "move",
-      "args": {
-        "id": 241,
-        "to": "done",
-        "reason": null,
-        "candidate": null,
-        "evidence": null,
-        "issue": null
-      },
-      "by": "sage",
-      "at": "2026-10-21T03:35:52.657Z",
-      "semantics": 2,
-      "result": {
-        "row": 241,
-        "from": "in-review",
-        "to": "done",
-        "round": 1,
-        "issue": null,
-        "candidate": null,
-        "receipt": "ok sage-241-done rev 2634 row 241 in-review→done round 1"
-      },
-      "viewSha": "369663313667edd50358fd170a5e652a06332ec394918089a02cc79b19eed63f"
-    },
-    {
-      "rev": 2635,
-      "op": "megavolt-242-inreview-r1",
-      "verb": "move",
-      "args": {
-        "id": 242,
-        "to": "in-review",
-        "reason": null,
-        "candidate": "/mnt/storage/scratch/megavolt-r242/manifest-r1-635c10924.txt",
-        "evidence": null,
-        "issue": null
-      },
-      "by": "megavolt",
-      "at": "2026-10-21T03:44:17.331Z",
-      "semantics": 2,
-      "result": {
-        "row": 242,
-        "from": "in-progress",
-        "to": "in-review",
-        "round": 1,
-        "issue": 1748,
-        "candidate": {
-          "kind": "manifest",
-          "digest": "1fbfab6c68e8d76d2d69b13c415257280d4fa8c5584e24d5dfd13dca1420a3da",
-          "text": "d245f805bfc9ad8a0064a8898effa05148e9c3fb7deb22b85f7af499e0ded369  agents/megavolt/work/queue-242/BUILD.md\ncfecc67698ad18a94288ba4e852b372d92c8d1cb984494256f929b03a467baad  packages/bus/src/runtime.mjs\na0783575b7da4e3b8e1ae161d7031fdf62a8a1a402a48ac898ef1b5c2d378d9b  packages/cli/tests/publish-socket.test.mjs\n"
-        },
-        "receipt": "ok megavolt-242-inreview-r1 rev 2635 row 242 in-progress→in-review round 1 on #1748; request megavolt-242-inreview-r1 requesting"
-      },
-      "viewSha": "432a911c27ffb0374b1ca49cb60215cb4211ce4c8a7f02bdfff91054f6a54e69"
-    },
-    {
-      "rev": 2636,
-      "op": "megavolt-242-inreview-r1.outcome",
-      "verb": "review-outcome",
-      "args": {
-        "id": 242,
-        "attempt": "megavolt-242-inreview-r1",
-        "outcome": "failed",
-        "status": null,
-        "comment": null,
-        "detail": "pre-send: the credential or account check failed"
-      },
-      "by": "megavolt",
-      "at": "2026-10-21T03:44:42.695Z",
-      "semantics": 2,
-      "result": {
-        "row": 242,
-        "round": 1,
-        "attempt": "megavolt-242-inreview-r1",
-        "state": "failed",
-        "receipt": "ok megavolt-242-inreview-r1.outcome rev 2636 row 242 round 1 request megavolt-242-inreview-r1 failed"
-      },
-      "viewSha": "42cd5e62d4b725838d45d409f48444e769c173db4825dcdc22e33c86a297ecda"
-    },
-    {
-      "rev": 2637,
-      "op": "sage-242-request-1",
-      "verb": "review-request",
-      "args": {
-        "id": 242
-      },
-      "by": "sage",
-      "at": "2026-10-21T03:45:41.063Z",
-      "semantics": 2,
-      "result": {
-        "row": 242,
-        "round": 1,
-        "attempt": "sage-242-request-1",
-        "state": "requesting",
-        "receipt": "ok sage-242-request-1 rev 2637 row 242 round 1 request sage-242-request-1 requesting"
-      },
-      "viewSha": "bc6018e64090bd29a7770b4f03511edc1314428a99d9114b3b274f2d248ce228"
-    },
-    {
-      "rev": 2638,
-      "op": "sage-242-request-1.outcome",
-      "verb": "review-outcome",
-      "args": {
-        "id": 242,
-        "attempt": "sage-242-request-1",
-        "outcome": "posted",
-        "status": 201,
-        "comment": 29888,
-        "detail": "created"
-      },
-      "by": "sage",
-      "at": "2026-10-21T03:46:05.721Z",
-      "semantics": 2,
-      "result": {
-        "row": 242,
-        "round": 1,
-        "attempt": "sage-242-request-1",
-        "state": "posted",
-        "receipt": "ok sage-242-request-1.outcome rev 2638 row 242 round 1 request sage-242-request-1 posted comment 29888"
-      },
-      "viewSha": "d7bf764deb95140e4b2748863b838f5598f57162a0388adcfb24b4ca6cb90449"
    }
  ]
 }
@@ -257,42 +257,9 @@ export async function startBroker({
          if (!(e instanceof BusError)) process.stderr.write(`bus: publish git-failed: ${JSON.stringify({ name: e?.name, code: e?.code })}\n`);
          Object.assign(entry, { state: 'refused', code: e instanceof BusError ? e.code : 'git-failed' });
        },
-      ).then(() => {
-        if (entry.held) settle(ticket, entry, entry.held);
-      });
+      );
      return { state: 'running' };
    }
-    // Row 242 (#1748): the git socket gave up on a publish (publish-timeout) whose push it never
-    // collected. Its publishRefused waits for the push: until that ends the ticket stays open, so
-    // checkPublish refuses another publish on the decision with publish-busy (row 237). A push that
-    // reached the fork is then recorded, which spends the decision; one that didn't is filed with
-    // the socket's code. Either way the decision can't push a second ref.
-    function settle(ticket, entry, code) {
-      remotes.delete(ticket);
-      try {
-        if (entry.state === 'pushed') {
-          try {
-            broker.recordPublish(ticket, { ref: entry.ref, commit: entry.commit, previous: entry.previous });
-            process.stderr.write(`bus: publish ${ticket}: pushed after the git socket refused ${code}; recorded\n`);
-            return;
-          } catch (e) {
-            if (!(e instanceof BusError)) throw e;
-            code = e.code;
-          }
-        }
-        broker.publishRefused(ticket, { code });
-      } catch (e) {
-        process.stderr.write(`bus: publish ${ticket}: not settled: ${JSON.stringify({ name: e?.name, code: e?.code })}\n`);
-      }
-    }
-    function publishRefused(ticket, args) {
-      const entry = remotes.get(ticket);
-      if (!entry) return broker.publishRefused(ticket, args);
-      if (!/^[a-z-]{1,64}$/.test(args?.code ?? '')) throw new BusError('invalid-request');
-      if (entry.state === 'running') entry.held = args.code;
-      else settle(ticket, entry, args.code);
-      return args.code;
-    }
    function publishResult(ticket) {
      const entry = remotes.get(ticket);
      if (!entry) throw new BusError('publish-unknown');
@@ -329,7 +296,7 @@ export async function startBroker({
      publishPush,
      publishResult,
      recordPublish: (ticket, pushed) => broker.recordPublish(ticket, pushed),
-      publishRefused,
+      publishRefused: (ticket, args) => broker.publishRefused(ticket, args),
      landHeld: (record) => broker.landHeld(record),
      // Row 156 (#1651): the launcher's landing pass.
      landHolders: (business) => broker.landHolders(business),
@@ -306,28 +306,6 @@ test('a ticket outlives the run that asked, and recordLand checks the task again
  assert.throws(() => w.b.landRefused(ticket, { code: 'gate-failed' }), /land-unknown/);
 });

-// Row 241 (#1747): one decision lands once. Two tickets on one decision can't both reach
-// recordLand through the git socket (land-busy, git-socket.mjs), so this pins the broker's own
-// guard: recordLand runs #landFacts again, and a decision spent by the first refuses the second.
-test('two tickets on one decision: the first recordLand lands, the second refuses decision-consumed and records nothing', (t) => {
-  const w = setup(t);
-  const { d } = w.ready();
-  const first = w.check({ task_ref: TASK, decision: d.id }).ticket;
-  const second = w.check({ task_ref: TASK, decision: d.id }).ticket;
-  assert.notEqual(first, second);
-  assert.deepEqual(w.b.recordLand(first, w.land), { task_ref: TASK, commit: B });
-  assert.throws(() => w.b.recordLand(second, { ...w.land, commit: C, previous: B }), /decision-consumed/);
-  assert.equal(w.merges().length, 1);
-  assert.equal(w.merges()[0].body.commit, B);
-  assert.equal(w.results().length, 1);
-  assert.deepEqual(w.events('action.refused'), []);
-  // The git socket then ends the second ticket with the broker's code, as it does any record refusal.
-  assert.equal(w.b.landRefused(second, { code: 'decision-consumed', notify: true }), 'decision-consumed');
-  assert.deepEqual(w.events('action.refused').map((e) => e.body), [{ code: 'decision-consumed' }]);
-  assert.equal(w.merges().length, 1);
-  assert.throws(() => w.b.recordLand(second, w.land), /land-unknown/);
-});
-
 test('landRefused files the code on the task, tells the role only with notify, and ends the ticket', (t) => {
  const w = setup(t);
  const { d } = w.ready();
@@ -38,10 +38,7 @@
 //          one op at a time within 10 s. The poll ends at publishDeadlineMs
 //          (publish-timeout), past git's own limit on each of the four git
 //          runs. recordPublish files the push; any refusal after checkPublish
-//          is filed once, by publishRefused. After publish-timeout the bus
-//          holds the decision until its push ends (a retry gets publish-busy),
-//          then records a push that reached the fork or files the refusal
-//          (row 242, #1748).
+//          is filed once, by publishRefused.
 //
 // A refusal is {"ok": false, "error": "<code>"}, and the connection ends.
 // The host checks, in order: identity (a role of this business), the request
@@ -430,9 +427,8 @@ export function createGitSocket({ host, business: b, dataRoot, env = process.env
          await op({ op: "publishPush", ticket });
          const until = Date.now() + publishDeadlineMs;
          for (;;) {
-            // The bus's git may still finish after this. The bus holds the decision until it does,
-            // then records a push that reached the fork (row 242); the fork can still hold the
-            // commit with no record if recordPublish refuses, as after an in-band push.
+            // The bus's git may still finish after this; the fork could then hold the commit with
+            // no record, as when recordPublish refuses after a push.
            if (Date.now() >= until) {
              log(`git socket: publish ${task_ref}: publish-timeout; the bus process's push may still reach the fork`);
              throw new Refusal("publish-timeout");
@@ -5,7 +5,7 @@
 import { test } from "node:test";
 import assert from "node:assert/strict";
 import { spawnSync } from "node:child_process";
-import { chmodSync, existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
+import { chmodSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
 import { connect } from "node:net";
 import { tmpdir } from "node:os";
 import { join } from "node:path";
@@ -298,41 +298,3 @@ test("sage-1440: the publishResult poll is bounded: publish-timeout, filed once,
  assert.equal(w.events("action.allowed").filter((e) => e.body.action === "git.push.remote" && e.body.operation !== "decision.raise").length, 0);
  assert.ok(w.logs.some((l) => l.includes("publish-timeout")));
 });
-
-test("row 242: a retry after publish-timeout refuses publish-busy while the bus process's push runs; the late push is recorded and the decision pushes once", async (t) => {
-  const w = await setup(t, { publishDeadlineMs: 150 });
-  w.bus.credentials.issue("demo/coder", "gitea", TOKEN, "2099-01-01");
-  // The fork holds the push in pre-receive until the test writes `release`; `held` says it's there.
-  const held = join(w.root, "held");
-  const release = join(w.root, "release");
-  writeFileSync(join(w.forkRepo, "hooks", "pre-receive"), `#!/bin/sh\n: > '${held}'\nwhile [ ! -e '${release}' ]; do sleep 0.05; done\ncat > /dev/null\n`, { mode: 0o755 });
-  const until = async (what, ok) => {
-    for (const end = Date.now() + 10_000; !ok(); ) {
-      if (Date.now() > end) assert.fail(`timed out waiting for ${what}`);
-      await new Promise((r) => setTimeout(r, 20));
-    }
-  };
-  const allowed = () => w.events("action.allowed").filter((e) => e.body.action === "git.push.remote" && e.body.operation !== "decision.raise");
-  // A failed assert must still let the hook end: teardown waits for the fake Gitea, which waits for the push.
-  try {
-    const d = w.approve();
-    assert.deepEqual(await w.publish(d), { ok: false, error: "publish-timeout" });
-    await until("the push to reach pre-receive", () => existsSync(held));
-    // The socket gave up, but the push still runs: the decision is held, and nothing is filed yet.
-    assert.deepEqual(await w.publish(d, { branch: "other" }), { ok: false, error: "publish-busy" });
-    assert.deepEqual(w.events("action.refused").map((e) => e.body.code), ["publish-busy"]);
-    assert.equal(allowed().length, 0);
-    writeFileSync(release, "");
-    // The push reached the fork, so it's recorded, which spends the decision.
-    await until("the late push to be recorded", () => allowed().length === 1);
-    assert.equal(allowed()[0].body.ref, "refs/heads/coder/feat");
-    assert.equal(allowed()[0].body.commit, w.work);
-    assert.deepEqual(await w.publish(d, { branch: "other" }), { ok: false, error: "decision-consumed" });
-    assert.equal(w.gitea.requests.filter((r) => r.path.endsWith("/git-receive-pack")).length, 1);
-    assert.equal(git(w.root, `--git-dir=${w.forkRepo}`, "for-each-ref", "--format=%(refname)", "refs/heads/coder/"), "refs/heads/coder/feat");
-    assert.deepEqual(w.events("action.refused").map((e) => e.body.code), ["publish-busy", "decision-consumed"]);
-  } finally {
-    writeFileSync(release, "");
-  }
-  assert.equal(JSON.stringify([w.events("action.allowed"), w.events("action.refused"), w.logs]).includes(TOKEN), false);
-});
@@ -2062,6 +2062,49 @@ test('R15 a rehearsal stopped by TERM, INT or HUP fails signal and still stops t
 }
 });

+// --- Row 233 (#1735): --backup last, the newest backup the log records (sage-1650, sage-1653) ---
+
+// The log's lines, as the runs write them, with only the fields from_log reads set.
+function rlog(e,...ls){
+ mkdirSync(e.state,{recursive:true});
+ writeFileSync(e.log,ls.map(([op,to,backup],i)=>JSON.stringify({at:`2026-10-20T00:00:0${i}Z`,op,from:null,to,backup,activation:`20261020T00000${i}Z-${to.slice(0,12)}`,result:'ok'})+'\n').join(''));
+}
+
+test('row 233: --backup last takes the newest backup the log records; a later line with backup null does not hide it, and the store line names it',async(t)=>{
+ const e=await rmk(t);
+ // An older backup that isn't on disk: a run that took it would fail backup-missing.
+ rlog(e,['activate',C1,`20261019T000000Z-${C1.slice(0,12)}`],['activate',B,e.act],['rollback',B,null]);
+ const l0=readFileSync(e.log,'utf8'),before=untouched(e);
+ const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','10');
+ assert.equal(r.code,0,r.out+r.err);
+ assert.deepEqual(r.names,['PASS backup-last',...PRE,'PASS migration','PASS gate','PASS no-session',...POST]);
+ assert.ok(r.out.startsWith(`PASS backup-last: the log's newest backup is ${e.act}\n`),r.out);
+ assert.ok(r.out.includes(`\nBACKUP_STORE ${e.act} schema 3g\n`),r.out);
+ assert.equal(readFileSync(e.log,'utf8'),l0);assert.equal(untouched(e),before);
+ // The rehearsal's own steps.log has the PASS and FAIL lines only.
+ assert.deepEqual(lines(join(r.dir,'steps.log')).map((l)=>l.replace(/^\S+ /,'')),r.out.split('\n').filter((l)=>/^(PASS|FAIL) |^REHEARSAL_/.test(l)));
+});
+
+test('row 233: --backup last with no backup in the log, or no log, exits 3 with no-backup and makes no rehearsal directory',async(t)=>{
+ const e=await rmk(t);
+ for(const how of ['no log','adopt only']){
+  if(how==='adopt only')rlog(e,['adopt',B,null]);
+  const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','3');
+  assert.equal(r.code,3,how+r.out+r.err);
+  assert.equal(r.out,'FAIL no-backup: the log records no backup\nREHEARSAL_FAIL\n',how);
+  assert.equal(r.dir,null);assert.deepEqual(rDirs(e),[],how);assert.ok(!existsSync(join(e.data,'rehearsals')),how);
+ }
+});
+
+test('row 233: --backup last refuses a newest backup that is not an activation id, with exit 1',async(t)=>{
+ const e=await rmk(t);
+ rlog(e,['activate',B,'../../etc']);
+ const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','3');
+ assert.equal(r.code,1,r.out+r.err);
+ assert.equal(r.out,'FAIL backup-unreadable: the log\'s newest backup is not an activation id\nREHEARSAL_FAIL\n');
+ assert.deepEqual(rDirs(e),[]);
+});
+
 // --- Row 190 (#1688): stack instances on one host. Design and the T-I table:
 // agents/darkwing/work/queue-190/DESIGN.md. T-I1, T-I2, T-I5, T-I10 and the loader half of T-I7 are in
 // stack-instances.test.mjs; these need the units, the stand-ins and stack-release.sh. Every existing
@@ -0,0 +1,449 @@
+// scripts/stack-upgrade.sh (row 233, I3, #1735): root's upgrade and rollback of a stack whose units are
+// root's, driving stack-release.sh --units external (row 232). Design: r6 sections 4 and 7,
+// agents/darkwing/work/queue-195/DESIGN.md, as the I3 brief changes them; deviations:
+// agents/darkwing/work/queue-233/BUILD.md. The stand-ins and the module fixture are
+// stack-release-fixture.mjs's. Root's sudo, case2-unit.sh render and row 240 hostcheck are stand-ins
+// here; the stack's user's steps are the real stack-release.sh from the user's clone, except the
+// rehearsal, which the fixture's stub copies can't pass (its own tests are in stack-release.test.mjs).
+import test from 'node:test';
+import assert from 'node:assert/strict';
+import {execFileSync,spawn} from 'node:child_process';
+import {mkdirSync,writeFileSync,readFileSync,existsSync,readdirSync,rmSync,chmodSync,copyFileSync,symlinkSync,statSync} from 'node:fs';
+import {DatabaseSync} from 'node:sqlite';
+import {join,dirname} from 'node:path';
+import {once} from 'node:events';
+import {B,C,C1,U,COPIES,NODE,SBIN,base,fault,git,lines,logLines,mk,pointer,repo,sha,storeVersion,sysctl,units} from './stack-release-fixture.mjs';
+
+// --- Root's stand-ins. Each is serialized into its own file with K, its baked paths. ---
+
+// sudo -u USER -H env PATH=P CMD...: the form the driver must use, logged to sudo.log, then CMD with
+// SBIN before P. A one-shot hook arm/<key>.sh runs first and arm/<key>.id replaces CMD's ID. The
+// rehearsal prints a fixed BACKUP_STORE line and exits faults/rehearse-rc unless faults/real-rehearse.
+function fakeSudo(K){
+ const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
+ const a=process.argv.slice(2),sd=process.env.SD_STATE,f=(n)=>path.join(sd,'faults',n);
+ if(a[0]!=='-u'||!a[1]||a[2]!=='-H'||a[3]!=='env'||!/^PATH=/.test(a[4]||'')||a.length<6){console.error(`fake sudo: not the driver's form: ${a.join(' ')}`);process.exit(99)}
+ const P=a[4].slice(5),cmd=a.slice(5);
+ let key=path.basename(cmd[0]);
+ if(cmd[0].endsWith('/scripts/stack-release.sh')){
+  key=cmd[1];
+  if(['activate','rollback'].includes(key)&&cmd.includes('--plan'))key+='-plan';
+  if(key==='confirm'&&cmd.includes('--rollback'))key='confirm-rollback';
+ }else if(cmd[0]==='git')key=`git-${cmd[3]}`;
+ fs.appendFileSync(path.join(sd,'sudo.log'),JSON.stringify({user:a[1],path:P,key,cmd})+'\n');
+ const env={...process.env,PATH:`${K.sbin}:${P}`},arm=(n)=>path.join(sd,'arm',`${key}.${n}`);
+ if(fs.existsSync(arm('sh'))){const run=arm('sh')+'.ran';fs.renameSync(arm('sh'),run);cp.spawnSync('bash',[run],{env,stdio:'inherit'})}
+ if(fs.existsSync(arm('id'))){cmd[2]=fs.readFileSync(arm('id'),'utf8').trim();fs.rmSync(arm('id'))}
+ if(key==='rehearse'&&!fs.existsSync(f('real-rehearse'))){
+  console.log(`fixture rehearse: ${cmd.slice(1).join(' ')}`);
+  let line='BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g';try{line=fs.readFileSync(f('rehearse-store-line'),'utf8').replace(/\n$/,'')}catch{}
+  if(line)console.log(line);
+  let rc=0;try{rc=Number(fs.readFileSync(f('rehearse-rc'),'utf8'))}catch{}
+  process.exit(rc);
+ }
+ const r=cp.spawnSync(cmd[0],cmd.slice(1),{env,stdio:'inherit'});process.exit(r.status??1);
+}
+
+// case2-unit.sh: logs its argv, then installs the copy's units as the fixture's user units (its
+// WorkingDirectory under the stack's copies) and starts the bus, then the Console, as case2-unit.sh
+// does. faults/render-fails-<copy> exits 1 first; faults/render-probe-<copy> (one-shot) first runs
+// probe.sh, a second driver, and records its exit and stderr.
+function fakeRender(K){
+ const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
+ const a=process.argv.slice(2),sd=process.env.SD_STATE,f=(n)=>path.join(sd,'faults',n);
+ fs.appendFileSync(path.join(sd,'render.log'),JSON.stringify(a)+'\n');
+ const cut=a.indexOf('--'),pre=a.slice(0,cut),serve=a.slice(cut+1),get=(k)=>pre[pre.indexOf(k)+1];
+ const copy=get('--copy'),copies=path.join(get('--home'),'.local','share','mosaic-stack','copies');
+ if(fs.existsSync(f(`render-probe-${copy}`))){
+  fs.rmSync(f(`render-probe-${copy}`));
+  const p=cp.spawnSync('bash',[path.join(sd,'probe.sh')],{encoding:'utf8'});
+  fs.writeFileSync(path.join(sd,'probe.json'),JSON.stringify({code:p.status,err:p.stderr}));
+ }
+ if(fs.existsSync(f(`render-fails-${copy}`))){console.error(`fixture render: ${copy} fails`);process.exit(1)}
+ const sh=(n,...x)=>cp.execFileSync('bash',[path.join(K.repo,'scripts',n),...x],{stdio:['ignore','ignore','inherit']});
+ sh('bus-service.sh','install','--copy',copy,'--copies',copies,...(pre.includes('--pm')?['--pm']:[]),'--claude',get('--claude'));
+ sh('webui-service.sh','install','--copy',copy,'--copies',copies,...serve);
+ for(const u of [`mosaic-bus@${get('--business')}.service`,'mosaic-webui.service'])cp.execFileSync(path.join(K.sbin,'systemctl'),['start',u],{stdio:'inherit'});
+}
+
+// Root's systemctl: the fixture's, with systemd's no-op start of an active unit, and a one-shot
+// faults/stop-noop-<unit> that leaves the unit running, as a stop past its timeout would.
+function fakeSystemctl(K){
+ const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
+ const a=process.argv.slice(2),sd=process.env.SD_STATE;
+ fs.appendFileSync(path.join(sd,'root-systemctl.log'),a.join(' ')+'\n');
+ const noop=path.join(sd,'faults',`stop-noop-${a[1]}`);
+ if(a[0]==='stop'&&fs.existsSync(noop)){fs.rmSync(noop);process.exit(0)}
+ if(a[0]==='start'){let s={};try{s=JSON.parse(fs.readFileSync(path.join(sd,'units.json'),'utf8'))}catch{}if((s[a[1]]||{}).state==='active')process.exit(0)}
+ const r=cp.spawnSync(path.join(K.sbin,'systemctl'),a,{stdio:'inherit'});process.exit(r.status??1);
+}
+
+const FAKES=join(base,'upgrade-fakes');mkdirSync(FAKES);
+const K={sbin:SBIN,repo};
+for(const [n,fn] of [['sudo',fakeSudo],['render',fakeRender],['systemctl',fakeSystemctl]])writeFileSync(join(FAKES,n),`#!${NODE}\n(${fn})(${JSON.stringify(K)});\n`,{mode:0o755});
+writeFileSync(join(FAKES,'hostcheck'),'#!/bin/sh\necho "$*" >> "$SD_STATE/hostcheck.log"\nexit "$(cat "$SD_STATE/faults/hostcheck-rc" 2>/dev/null || echo 0)"\n',{mode:0o755});
+const BUNDLE=join(base,'fixture.bundle');git(repo,'bundle','create','-q',BUNDLE,'refactor');
+// Root's checkout of a reviewed commit: a clone of the fixture repo, detached at it.
+const checkouts=new Map();
+const checkout=(id)=>{
+ if(!checkouts.has(id)){const d=join(base,`root-${id.slice(0,12)}`);git(base,'clone','-q','--no-checkout',repo,d);git(d,'checkout','-q','--detach',id);checkouts.set(id,d)}
+ return checkouts.get(id);
+};
+const TO=['--timeout','3'];
+
+// --- Per-test stack ---
+
+// The stack's user's home: its copies (the fixture's, through a link), its clone, its bin. Both units
+// installed on `copy` and active, adopted under --units external, and the I2 args file naming it.
+async function useed(t,{copy=B}={}){
+ const e=await mk(t),home=e.env.HOME;
+ e.hc=join(home,'.local','share','mosaic-stack','copies');mkdirSync(dirname(e.hc),{recursive:true});symlinkSync(COPIES,e.hc);
+ const bin=join(home,'.npm-global','bin');mkdirSync(bin,{recursive:true});
+ writeFileSync(join(bin,'mosaic'),readFileSync(join(e.bin,'mosaic'),'utf8').replaceAll(`${e.bin}/`,`${bin}/`),{mode:0o755});
+ copyFileSync(join(e.bin,'mosaic-old'),join(bin,'mosaic-old'));chmodSync(join(bin,'mosaic-old'),0o755);
+ e.bin=bin;e.clone=join(home,'src','mosaic-stack');git(base,'clone','-q',repo,e.clone);
+ const db=new DatabaseSync(e.store);
+ db.exec('PRAGMA journal_mode=WAL');
+ db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
+ db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
+ db.prepare("INSERT INTO meta VALUES ('schema_version', '3g')").run();
+ for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`);
+ db.close();
+ const sh=(n,...a)=>execFileSync('bash',[join(e.clone,'scripts',n),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
+ sh('bus-service.sh','install','--copy',copy,'--copies',e.hc,'--claude',e.claude);
+ sh('webui-service.sh','install','--copy',copy,'--copies',e.hc,'--port',e.port,'--business','acme');
+ sh('stack-copy.sh','wrapper',copy,'--copies',e.hc,'--bin',e.bin);
+ execFileSync(join(SBIN,'systemctl'),['start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:'120'}});
+ sysctl(e,'start',e.con);
+ sh('stack-release.sh','adopt','acme','--units','external','--copies',e.hc,'--bin',e.bin);
+ e.etc=join(e.root,'etc');mkdirSync(e.etc);e.args=join(e.etc,'dev.args');
+ e.argsText=`--user\nfixture\n--home\n${home}\n--business\nacme\n--copy\n${copy}\n--node\n${NODE}\n--claude\n${e.claude}\n--\n--port\n${e.port}\n--business\nacme\n`;
+ writeFileSync(e.args,e.argsText);chmodSync(e.args,0o640);
+ e.run=join(e.root,'run');e.slog=join(e.root,'log','upgrade-dev.log');e.lockf=join(e.run,'upgrade-dev.lock');
+ mkdirSync(join(e.sd,'arm'));
+ e.denv={...e.env,STACK_UPGRADE_ETC:e.etc,STACK_UPGRADE_RUN:e.run,STACK_UPGRADE_LOG_DIR:dirname(e.slog),STACK_UPGRADE_UNIT_DIR:e.units,STACK_UPGRADE_HOSTCHECK:join(FAKES,'hostcheck'),STACK_UPGRADE_RENDER:join(FAKES,'render'),STACK_UPGRADE_SUDO:join(FAKES,'sudo'),STACK_UPGRADE_SYSTEMCTL:join(FAKES,'systemctl')};
+ writeFileSync(join(e.sd,'probe.sh'),`exec bash ${join(checkout(C1),'scripts','stack-upgrade.sh')} upgrade --instance dev --bundle ${BUNDLE} --sha ${C1} --timeout 3\n`);
+ writeFileSync(join(e.sd,'snap.jsonl'),'');
+ return e;
+}
+
+// The driver from root's checkout of `at`.
+async function drive(e,at,...args){
+ const ch=spawn('bash',[join(checkout(at),'scripts','stack-upgrade.sh'),...args],{env:e.denv,stdio:['ignore','pipe','pipe']});
+ let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d});
+ const timer=setTimeout(()=>ch.kill('SIGKILL'),180000);
+ const [code]=await once(ch,'close');clearTimeout(timer);
+ return {code,out,err,all:out+err};
+}
+const upgrade=(e,id,...a)=>drive(e,id,'upgrade','--instance','dev','--bundle',BUNDLE,'--sha',id,...TO,...a);
+const arm=(e,key,text)=>writeFileSync(join(e.sd,'arm',key),text);
+
+// The step log: every line `<time> <step> rc=N ...` or the result line, root:root 0640 (here the
+// test user's), and nothing the stack's user printed. Returns the lines without the time.
+const STEP=/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ (start|fetch|ff|copy|hostcheck|units|plan|rehearse|rehearse-store|rehearsal-skipped|stop-console|stop-bus|wait-down|stage|render|confirm|args|revert|render-old|confirm-rollback|gate|start-bus|start-console|stop-units) rc=\d+ |^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ result exit=[0-3] [a-z-]+ /;
+function slog(e){
+ const l=lines(e.slog);
+ for(const x of l){assert.match(x,STEP);assert.ok(!/stack-release:|fixture rehearse|PASS |FAIL |REHEARSAL/.test(x),x)}
+ assert.equal(statSync(e.slog).mode&0o777,0o640);
+ return l.map((x)=>x.slice(21));
+}
+const names=(e)=>slog(e).map((x)=>x.split(' ')[0]);
+const result=(e)=>slog(e).at(-1);
+const sudoKeys=(e)=>lines(join(e.sd,'sudo.log')).map((l)=>JSON.parse(l).key);
+const lastAct=(e)=>logLines(e).filter((o)=>o.result==='started').at(-1).activation;
+const mine=(e,a)=>logLines(e).filter((o)=>o.activation===a);
+const open=(e)=>{const m=new Set();for(const o of logLines(e)){if(!o.activation)continue;if(o.result==='started')m.add(o.activation);else m.delete(o.activation)}return [...m]};
+const final=(e,a)=>{const [s,f,...more]=mine(e,a);assert.equal(s.result,'started');assert.equal(more.length,0);return f};
+const live=(e)=>[units(e)[e.bus].state,units(e)[e.con].state,units(e)[e.bus].wd.split('/').at(-1),units(e)[e.con].wd.split('/').at(-1)];
+// Nothing the run did reached the args file: the same bytes and mode, and no temporary file left.
+const argsKept=(e)=>{assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args'])};
+const argsCopy=(e)=>{const l=readFileSync(e.args,'utf8').split('\n');return l[l.indexOf('--copy')+1]};
+const PRE=['start','fetch','ff','copy','hostcheck','units','plan','rehearse','rehearse-store'];
+const A2=['stop-console','stop-bus','wait-down','wait-down'];
+const R=[...A2,'revert','render-old','confirm-rollback'];
+
+// --- r6 sequences 1-8, through the driver ---
+
+test('row 233: sequence 1: confirm fails healthz, the driver reverts, renders the old copy and confirm --rollback --cause A5 passes: exit 1',async(t)=>{
+ const e=await useed(t);
+ fault(e,`healthz-${C1}`);
+ const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
+ assert.equal(r.err.split('\n').at(-2),`stack-upgrade: failed: A5 failed; rolled back to ${B}, which passed the gate`);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm',...R,'result']);
+ assert.equal(result(e),`result exit=1 failed A5 failed; rolled back to ${B}, which passed the gate`);
+ const a=lastAct(e),l=final(e,a);assert.deepEqual([l.op,l.result,l.code,l.rollback,l.from,l.to],['activate','gate-failed','healthz','ok',B,C1]);
+ assert.deepEqual(open(e),[]);assert.deepEqual(live(e),['active','active',B,B]);
+ argsKept(e);
+ // The stack's user's steps went through sudo in the driver's form, as fixture, the pinned node first.
+ for(const l of lines(join(e.sd,'sudo.log')).map((x)=>JSON.parse(x)))assert.deepEqual([l.user,l.path],['fixture',`${dirname(NODE)}:/usr/local/bin:/usr/bin:/bin`]);
+ assert.deepEqual(sudoKeys(e),['git-fetch','git-merge','stack-copy.sh','activate-plan','rehearse','activate','confirm','revert','confirm-rollback']);
+});
+
+test('row 233: sequence 2: activate --migrate fails at the wrapper, revert restores the store, confirm --rollback --cause A3: exit 1',async(t)=>{
+ const e=await useed(t);
+ const s0=sha(e.store);
+ arm(e,'activate.sh',`chmod 0555 ${e.bin}\n`);arm(e,'revert.sh',`chmod 0755 ${e.bin}\n`);
+ const r=await upgrade(e,C,'--migrate');assert.equal(r.code,1,r.all);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage',...R,'result']);
+ assert.equal(result(e),`result exit=1 failed A3 failed; rolled back to ${B}, which passed the gate`);
+ const l=final(e,lastAct(e));assert.deepEqual([l.result,l.code,l.rollback],['failed','wrapper','ok']);
+ assert.equal(storeVersion(e.store,true),'3g');assert.deepEqual(live(e),['active','active',B,B]);
+ // The bus holds the store open again; its bytes are the backup's once it stops.
+ sysctl(e,'stop',e.bus);assert.equal(sha(e.store),s0);
+ argsKept(e);
+});
+
+test('row 233: sequence 3: root\'s render of the new copy fails, the driver rolls back with --cause A4, and holds the lock to the end: exit 1',async(t)=>{
+ const e=await useed(t);
+ fault(e,`render-fails-${C1}`);fault(e,`render-probe-${B}`);
+ const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
+ // The probe ran inside render-old, during the rollback, and its one line is its refusal.
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render',...A2,'revert','result','render-old','confirm-rollback','result']);
+ const p=JSON.parse(readFileSync(join(e.sd,'probe.json'),'utf8'));assert.equal(p.code,3,p.err);
+ assert.equal(p.err,`stack-upgrade: upgrade-running: another stack-upgrade.sh run holds ${e.lockf} (nothing was stopped)\n`);
+ assert.match(slog(e).at(-4),/^result exit=3 upgrade-running /);
+ assert.match(slog(e).find((x)=>x.startsWith('render ')),/^render rc=1 /);
+ const l=final(e,lastAct(e));assert.deepEqual([l.result,l.code,l.rollback],['failed','units-start','ok']);
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: sequence 4: a capture failure is exit 3 from the stage, and A2f restarts the old units and gates them: exit 1',async(t)=>{
+ const e=await useed(t);
+ arm(e,'activate.sh',`: > ${join(e.sd,'faults','no-fragment')}\n`);arm(e,'gate.sh',`rm ${join(e.sd,'faults','no-fragment')}\n`);
+ const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','start-bus','start-console','gate','result']);
+ assert.match(slog(e).find((x)=>x.startsWith('stage ')),/^stage rc=3 /);
+ assert.equal(result(e),`result exit=1 failed A3 failed; ${B} is running again and passed the gate`);
+ const a=lastAct(e);assert.deepEqual(mine(e,a).map((o)=>[o.result,o.code]),[['started',undefined],['refused','capture-failed']]);
+ assert.deepEqual(open(e),[]);assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: N1: a bus that outlives A2\'s timeout keeps its old start, so A2f\'s gate fails stale-start: exit 2, both units stopped',async(t)=>{
+ const e=await useed(t);
+ fault(e,`stop-noop-${e.bus}`);
+ const r=await upgrade(e,C1);assert.equal(r.code,2,r.all);
+ assert.deepEqual(names(e),[...PRE,...A2,'start-bus','start-console','gate','stop-units','result']);
+ assert.ok(slog(e).includes(`wait-down rc=1 unit: ${e.bus} state: active`),slog(e).join('\n'));
+ assert.ok(r.out.includes(`gate failed on ${B}: stale-start`),r.out);
+ assert.equal(result(e),`result exit=2 rollback-failed A2 failed and ${B} failed its gate after the restart; both units are stopped`);
+ assert.deepEqual(live(e).slice(0,2),['inactive','inactive']);
+ assert.ok(!sudoKeys(e).includes('activate'));argsKept(e);
+});
+
+test('row 233: sequence 5: confirm of the wrong copy is refused, and the driver rolls back with --cause A5: exit 1',async(t)=>{
+ const e=await useed(t);
+ arm(e,'confirm.id',B);
+ const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
+ assert.ok(r.out.includes('refusing open-activation')||r.err.includes('refusing open-activation'),r.all);
+ assert.match(slog(e).find((x)=>x.startsWith('confirm ')),/^confirm rc=3 /);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm',...R,'result']);
+ const l=final(e,lastAct(e));assert.deepEqual([l.result,l.code,l.rollback],['gate-failed','unrecorded','ok']);
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: sequence 6: a revert that fails stops both units at exit 2, and the next run refuses at its --plan with nothing stopped',async(t)=>{
+ const e=await useed(t);
+ fault(e,`render-fails-${C1}`);
+ arm(e,'revert.sh',`for d in ${join(e.state,'stack-rollback')}/*/; do [ -f "$d/AWAITING-UNITS" ] && rm "$d/wrapper"; done\n`);
+ const r=await upgrade(e,C1);assert.equal(r.code,2,r.all);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render',...A2,'revert','stop-units','result']);
+ assert.equal(result(e),`result exit=2 rollback-failed A4 failed and revert failed; both units are stopped`);
+ const a=lastAct(e);assert.deepEqual(open(e),[a]);assert.deepEqual(live(e).slice(0,2),['inactive','inactive']);
+ argsKept(e);
+ const n=slog(e).length,sc=lines(join(e.sd,'root-systemctl.log')).length;
+ rmSync(join(e.sd,'faults',`render-fails-${C1}`));
+ const again=await upgrade(e,C1);assert.equal(again.code,3,again.all);
+ assert.ok(again.out.includes(`refuses interrupted-run: activation ${a} started and has no final line`),again.out);
+ assert.deepEqual(names(e).slice(n),PRE.slice(0,7).concat('result'));
+ assert.match(result(e),/^result exit=3 plan-refused /);
+ assert.equal(lines(join(e.sd,'root-systemctl.log')).length,sc);argsKept(e);
+});
+
+test('row 233: sequence 7: two upgrades and a rollback --to the previous copy with --restore, the args file following each confirm',async(t)=>{
+ const e=await useed(t);
+ // The args file at confirm still names the old copy; the new one moves in after it.
+ arm(e,'confirm.sh',`cp ${e.args} ${join(e.sd,'args-at-confirm')}\n`);
+ const one=await upgrade(e,C1);assert.equal(one.code,0,one.all);
+ assert.equal(one.out.split('\n').at(-2),`stack-upgrade: upgrade ok: ${C1} is live for acme`);
+ assert.equal(readFileSync(join(e.sd,'args-at-confirm'),'utf8'),e.argsText);
+ assert.equal(readFileSync(e.args,'utf8'),e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`));
+ assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args']);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm','args','result']);
+ assert.ok(slog(e).includes(`args rc=0 copy: ${C1}`));
+ const a1=lastAct(e);
+ const two=await upgrade(e,C,'--migrate');assert.equal(two.code,0,two.all);
+ const a2=lastAct(e);assert.equal(argsCopy(e),C);
+ const l1=final(e,a1),l2=final(e,a2);
+ assert.deepEqual([l1.op,l1.result,l1.from,l1.to],['activate','ok',B,C1]);
+ assert.deepEqual([l2.op,l2.result,l2.from,l2.to,l2.backup],['activate','ok',C1,C,a2]);
+ assert.equal(storeVersion(e.store,true),'3h');
+ const n=slog(e).length;
+ const rb=await drive(e,C,'rollback','--instance','dev','--to',C1,'--restore',a2,...TO);assert.equal(rb.code,0,rb.all);
+ assert.equal(rb.out.split('\n').at(-2),`stack-upgrade: rollback ok: ${C1} is live for acme`);
+ // A rollback fetches, copies and rehearses nothing.
+ assert.deepEqual(names(e).slice(n),['start','hostcheck','units','plan',...A2,'stage','render','confirm','args','result']);
+ const ra=lastAct(e),l3=final(e,ra);assert.deepEqual([l3.op,l3.result,l3.from,l3.to,l3.backup],['rollback','ok',C,C1,a2]);
+ assert.deepEqual([pointer(e).copy,pointer(e).schema],[C1,'3g']);assert.equal(storeVersion(e.store,true),'3g');
+ assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
+ const m=slog(e).length,sc=lines(join(e.sd,'root-systemctl.log')).length;
+ const no=await drive(e,C,'rollback','--instance','dev','--to',B,...TO);assert.equal(no.code,3,no.all);
+ assert.ok(no.all.includes(`refusing to-not-previous: --to ${B} is not the previous copy; the log names ${C}`),no.all);
+ assert.deepEqual(names(e).slice(m),['start','hostcheck','units','plan','result']);
+ assert.equal(lines(join(e.sd,'root-systemctl.log')).length,sc);assert.equal(argsCopy(e),C1);
+});
+
+test('row 233: sequence 8: a migration that writes and fails: the driver reverts to the backup byte for byte, --cause A3: exit 1',async(t)=>{
+ const e=await useed(t);
+ const s0=sha(e.store);
+ fault(e,'migrate-fails');
+ const r=await upgrade(e,C,'--migrate');assert.equal(r.code,1,r.all);
+ assert.match(slog(e).find((x)=>x.startsWith('stage ')),/^stage rc=1 /);
+ assert.deepEqual(names(e),[...PRE,...A2,'stage',...R,'result']);
+ const a=lastAct(e),l=final(e,a);assert.deepEqual([l.result,l.code,l.rollback],['failed','migrate-failed','ok']);
+ sysctl(e,'stop',e.bus);assert.equal(sha(e.store),s0);assert.equal(storeVersion(e.store,false),'3g');
+ argsKept(e);
+});
+
+// --- The lock, A1's refusals and the rehearsal ---
+
+test('row 233: a second run while the lock is held refuses upgrade-running with one log line and no call',async(t)=>{
+ const e=await useed(t);
+ mkdirSync(e.run,{recursive:true});
+ const holder=spawn('flock',[e.lockf,'sleep','60'],{detached:true,stdio:'ignore'});
+ t.after(()=>{try{process.kill(-holder.pid,'SIGKILL')}catch{}});
+ for(let i=0;i<100&&execFileSync('bash',['-c','flock -n "$1" true && echo free || echo held','_',e.lockf],{encoding:'utf8'}).trim()==='free';i++)await new Promise((r)=>setTimeout(r,50));
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.equal(r.err,`stack-upgrade: upgrade-running: another stack-upgrade.sh run holds ${e.lockf} (nothing was stopped)\n`);
+ assert.deepEqual(names(e),['result']);
+ assert.ok(!existsSync(join(e.sd,'sudo.log')));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: A1 refuses open-activation on any AWAITING-UNITS file, through the stack\'s user\'s --plan, with nothing stopped',async(t)=>{
+ const e=await useed(t);
+ const fake=join(e.state,'stack-rollback',`20000101T000000Z-${C1.slice(0,12)}`);mkdirSync(fake,{recursive:true});writeFileSync(join(fake,'AWAITING-UNITS'),`${C1}\n`);
+ const logs=logLines(e).length;
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes('refuses open-activation'),r.out);
+ assert.deepEqual(names(e),[...PRE.slice(0,7),'result']);assert.match(result(e),/^result exit=3 plan-refused /);
+ assert.equal(logLines(e).length,logs);assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: a target that migrates refuses migrate-required without --migrate, with nothing stopped',async(t)=>{
+ const e=await useed(t);
+ const r=await upgrade(e,C);assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes('refuses migrate-required'),r.out);
+ assert.deepEqual(names(e),[...PRE.slice(0,7),'result']);assert.match(result(e),/^result exit=3 plan-refused /);
+ assert.ok(!sudoKeys(e).includes('rehearse'));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
+ assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: the rehearsal runs after the plan and before A2; a failed one refuses rehearsal-failed with nothing stopped',async(t)=>{
+ const e=await useed(t);
+ fault(e,'rehearse-rc','1');
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes('fixture rehearse: '),r.out);
+ assert.deepEqual(names(e),[...PRE,'result']);
+ assert.equal(result(e),'result exit=3 rehearsal-failed the rehearsal exited 1 (nothing was stopped)');
+ const rh=lines(join(e.sd,'sudo.log')).map((l)=>JSON.parse(l)).find((l)=>l.key==='rehearse');
+ assert.deepEqual(rh.cmd.slice(1),['rehearse',C1,'acme','--copies',e.hc,'--backup','last','--port','7339','--timeout','3']);
+ assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
+});
+
+test('row 233: the rehearsal\'s BACKUP_STORE line goes to the step log after a match, a malformed one as none, the rest of its output never',async(t)=>{
+ const e=await useed(t);
+ fault(e,'rehearse-rc','1');
+ const r=await upgrade(e,C1,'--rehearse-port','7449');assert.equal(r.code,3,r.all);
+ assert.ok(slog(e).includes('rehearse-store rc=0 BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g'),slog(e).join('\n'));
+ assert.ok(slog(e).find((x)=>x.startsWith('rehearse ')).includes(' --port 7449 '));
+ for(const bad of ['BACKUP_STORE ../../etc schema 3g','BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g; rm -rf /','stack-release: BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g','']){
+  fault(e,'rehearse-store-line',bad);
+  const n=slog(e).length;
+  const b=await upgrade(e,C1);assert.equal(b.code,3,b.all);
+  assert.deepEqual(slog(e).slice(n).filter((x)=>x.startsWith('rehearse-store ')),['rehearse-store rc=0 BACKUP_STORE none'],bad);
+ }
+});
+
+test('row 233: a real rehearsal with no backup in the log refuses no-backup, and goes on only with --no-rehearsal, which logs the skip',async(t)=>{
+ const e=await useed(t);
+ fault(e,'real-rehearse');
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes('FAIL no-backup: the log records no backup\nREHEARSAL_FAIL\n'),r.out);
+ assert.equal(result(e),'result exit=3 no-backup the log records no backup to rehearse against; only a go that names --no-rehearsal goes on (nothing was stopped)');
+ assert.ok(slog(e).includes('rehearse-store rc=0 BACKUP_STORE none'));
+ assert.ok(!existsSync(join(e.data,'rehearsals')));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
+ const n=slog(e).length;
+ const s=await upgrade(e,C1,'--no-rehearsal');assert.equal(s.code,0,s.all);
+ assert.deepEqual(names(e).slice(n),[...PRE,'rehearsal-skipped',...A2,'stage','render','confirm','args','result']);
+ assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
+});
+
+test('row 233: a real rehearsal after an upgrade names the log\'s newest backup; --no-rehearsal with a backup refuses rehearsal-required',async(t)=>{
+ const e=await useed(t);
+ const one=await upgrade(e,C1);assert.equal(one.code,0,one.all);
+ const a1=lastAct(e);
+ fault(e,'real-rehearse');
+ const n=slog(e).length;
+ // The stub copies fail the rehearsal right after it names the backup (copy-without-no-launch).
+ const r=await upgrade(e,C,'--migrate');assert.equal(r.code,3,r.all);
+ assert.ok(r.out.includes(`PASS backup-last: the log's newest backup is ${a1}\n`),r.out);
+ assert.deepEqual(names(e).slice(n),[...PRE,'result']);assert.match(result(e),/^result exit=3 rehearsal-failed the rehearsal exited 1 /);
+ rmSync(join(e.sd,'faults','real-rehearse'));
+ const m=slog(e).length;
+ const q=await upgrade(e,C,'--migrate','--no-rehearsal');assert.equal(q.code,3,q.all);
+ assert.deepEqual(names(e).slice(m),[...PRE,'result']);
+ assert.equal(result(e),'result exit=3 rehearsal-required the log records a backup and the rehearsal ran; --no-rehearsal is only for a stack with none (nothing was stopped)');
+ assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
+});
+
+// --- Before A1 and the args file ---
+
+test('row 233: the args file must be the running user\'s, with no group or other write, one --copy, and the copy the units run',async(t)=>{
+ const e=await useed(t);
+ const tryArgs=async(text,mode,code,why)=>{
+  writeFileSync(e.args,text);chmodSync(e.args,mode);
+  const n=existsSync(e.slog)?slog(e).length:0;
+  const r=await upgrade(e,C1);assert.equal(r.code,3,`${why}: ${r.all}`);
+  assert.match(slog(e).at(-1),new RegExp(`^result exit=3 ${code} `),why);
+  assert.ok(!slog(e).slice(n).some((x)=>/^(stop|stage|render|confirm)/.test(x)),why);
+ };
+ await tryArgs(e.argsText,0o660,'args-unreadable','group write');
+ await tryArgs(e.argsText,0o642,'args-unreadable','other write');
+ await tryArgs(e.argsText.replace('--\n',`--copy\n${C1}\n--\n`),0o640,'args-unreadable','two --copy');
+ await tryArgs(e.argsText.replace('--\n','--tester\n--\n'),0o640,'args-unreadable','an unknown flag');
+ await tryArgs(e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`),0o640,'args-differ','the units run B');
+ rmSync(e.args);symlinkSync(join(e.sd,'probe.sh'),e.args);
+ const l=await upgrade(e,C1);assert.equal(l.code,3);assert.match(result(e),/^result exit=3 args-unreadable /);
+ assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);
+});
+
+test('row 233: hostcheck (row 240): a nonzero --check or a missing file refuses profile-drift before the plan',async(t)=>{
+ const e=await useed(t);
+ fault(e,'hostcheck-rc','1');
+ const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
+ assert.match(result(e),/^result exit=3 profile-drift /);
+ assert.deepEqual(lines(join(e.sd,'hostcheck.log')),['--check']);
+ assert.deepEqual(sudoKeys(e),['git-fetch','git-merge','stack-copy.sh']);
+ e.denv.STACK_UPGRADE_HOSTCHECK=join(e.root,'no-hostcheck');
+ const m=await upgrade(e,C1);assert.equal(m.code,3,m.all);
+ assert.equal(result(e),`result exit=3 profile-drift no executable ${join(e.root,'no-hostcheck')} (row 240) (nothing was stopped)`);
+ assert.ok(!sudoKeys(e).includes('activate-plan'));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));argsKept(e);
+});
+
+test('row 233: the checkout must be at --sha (head-mismatch), and a commit the clone can\'t fast-forward to refuses fetch-failed',async(t)=>{
+ const e=await useed(t);
+ const r=await drive(e,C1,'upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C,'--migrate',...TO);assert.equal(r.code,3,r.all);
+ assert.equal(result(e),`result exit=3 head-mismatch this checkout, ${checkout(C1)}, is at ${C1}, not ${C} (nothing was stopped)`);
+ const u=await drive(e,U,'upgrade','--instance','dev','--bundle',BUNDLE,'--sha',U,...TO);assert.equal(u.code,3,u.all);
+ assert.match(result(e),/^result exit=3 fetch-failed .* fast-forward /);
+ assert.ok(!sudoKeys(e).includes('activate-plan'));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));argsKept(e);
+});
+
+test('row 233: usage exits 4 before any log or lock',async(t)=>{
+ const e=await useed(t);
+ for(const a of [[],['status'],['upgrade','--instance','dev','--bundle',BUNDLE],['upgrade','--instance','other','--bundle',BUNDLE,'--sha',C1],['upgrade','--instance','dev','--bundle','rel/x','--sha',C1],['upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C1.slice(0,12)],['upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C1,'--to',B],['upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C1,'--rehearse-port','80'],['rollback','--instance','dev','--to',B,'--migrate'],['rollback','--instance','dev','--to',B,'--events','3'],['rollback','--instance','dev'],['rollback','--instance','dev','--to',B,'--restore','x']]){
+  const r=await drive(e,C1,...a);assert.equal(r.code,4,a.join(' '));assert.match(r.err,/^usage: stack-upgrade.sh upgrade /,a.join(' '));
+ }
+ assert.ok(!existsSync(dirname(e.slog)));assert.ok(!existsSync(e.run));
+});
@@ -130,7 +130,7 @@ parse_opts() {
            --sso) [ "$OP" = activate ] || usage; SSO=1; shift ;;
            --migrate) [ "$OP" = activate ] || usage; MIGRATE=1; shift ;;
            --restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || usage; RESTORE="$2"; shift 2 ;;
-            --backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
+            --backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [ "$2" = last ] || [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
            --port) [ "$OP" = rehearse ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{1,5}$ ]] && [ "$((10#$2))" -ge 1024 ] && [ "$((10#$2))" -le 65535 ] || usage; R_PORT="$((10#$2))"; shift 2 ;;
            --events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
            --units) [ "$OP" != rehearse ] && [ $# -ge 2 ] && { [ "$2" = user ] || [ "$2" = external ]; } || usage; UNITS="$2"; shift 2 ;;
@@ -220,7 +220,8 @@ for(const k of open.keys())process.stdout.write(k+"\n");' -- "$LOG"

 # Reads the log's ok lines, newest first. `previous` prints rollback's target: the first `to`
 # that is not the pointer's copy (release.sh's rule). `backup` prints the backup field of the
-# line that made the pointer's copy live, the backup a rollback would restore. `arrival` prints the
+# line that made the pointer's copy live, the backup a rollback would restore. `newest` prints the
+# backup field of the newest line that has one, rehearse's --backup last (row 233). `arrival` prints the
 # activation and `from` of the line that moved the stack onto the pointer's copy from another one,
 # past any same-copy re-pin after it (row 200, sage-1389).
 from_log() {
@@ -233,6 +234,7 @@ for(const l of lines.reverse()){
  if(o.result!=="ok"||!["adopt","activate","rollback"].includes(o.op)||!o.to)continue;
  if(mode==="previous"&&o.to!==current){process.stdout.write(o.to+"\n");process.exit(0)}
  if(mode==="backup"&&o.to===current){if(o.backup)process.stdout.write(o.backup+"\n");process.exit(o.backup?0:1)}
+  if(mode==="newest"&&o.backup){process.stdout.write(o.backup+"\n");process.exit(0)}
  if(mode==="arrival"&&o.to===current&&o.from!==current){process.stdout.write(`${o.activation??""} ${o.from??""}\n`);process.exit(0)}
 }
 process.exit(1);' -- "$LOG" "$1" "$P_COPY"
@@ -1262,6 +1264,8 @@ plan() {
        echo "needs: --migrate$([ "$MIGRATE" = 1 ] && echo ' (given)' || echo ' (not given; the run would roll back with migration-needs-flag)')"
        echo "needs: Jason present for this run (decision 96 condition 2)"
        [ "$MIGRATE" = 1 ] || early=1
+        # Row 233: root reads only the plan's exit, so under external a missing --migrate refuses here.
+        [ "$UNITS" = user ] || [ "$MIGRATE" = 1 ] || PLAN_REFUSE+=("migrate-required: the store is $S_SCHEMA and $TO migrates it to $T_SCHEMA; the go names --migrate")
    elif [ "$OP" = activate ]; then
        echo "migration: none possible; $S_SCHEMA is not in $T_SCHEMA's migration chain, so the run would roll back with store-newer"
        early=1
@@ -1452,6 +1456,13 @@ ext_pointer_checks() {
    local open
    open="$(interrupted | paste -sd' ' -)"
    [ -z "$open" ] || check_refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)"
+    # Row 233: root can't read the stack's state, so the plan lists any rollback set still waiting
+    # for units, whatever the log says.
+    local m sets=""
+    local -a markers
+    mapfile -t markers < <(compgen -G "$STATE/stack-rollback/*/AWAITING-UNITS" || true)
+    for m in "${markers[@]}"; do sets+=" $(basename "$(dirname "$m")")"; done
+    [ -z "$sets" ] || check_refuse open-activation "rollback set(s)$sets hold AWAITING-UNITS; root finishes them with confirm or revert first"
 }

 # A3's checks before `started`. Root has stopped the units and installs them after this run, so
@@ -1973,6 +1984,9 @@ r_env() { env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json"
 r_migration() {
    local copy="$COPIES/$TO" old="$COPIES/$P_COPY" v out rc
    v="$(node -e "$STORE_JS" -- version "$R/data/bus/bus.sqlite" 2>&1)" || r_abort store-unreadable "$v"
+    # Row 233 (sage-1653): which store the rehearsal ran against, for root's step log. Output only:
+    # the rehearsal's own steps.log has the backup's path on its backup line.
+    echo "BACKUP_STORE $R_BACKUP schema $v"
    if [ "$v" = "$T_SCHEMA" ]; then r_pass migration "not needed: the backup is already $v"; return 0; fi
    in_chain "$v" || r_abort store-newer "the backup is $v, which $TO's migration chain does not start from"
    (umask 077; mkdir "$R/refuse" "$R/refuse/bus" && cp "$R_BK"/bus/* "$R/refuse/bus/") || r_abort scratch-build "cannot copy the backup to $R/refuse"
@@ -2043,6 +2057,13 @@ do_rehearse() {
    trap 'r_fail signal "the rehearsal was interrupted"; r_end' INT TERM HUP
    read_pointer || r_abort no-pointer "no $POINTER; the pointer's copy is the old copy the checks use"
    [ "$P_BUSINESS" = "$BUSINESS" ] || r_abort business-mismatch "the pointer is for $P_BUSINESS"
+    # Row 233 (sage-1650, sage-1653): last is the newest backup the log records. With none, exit 3
+    # and no directory, the one rehearse refusal stack-upgrade.sh reads as no-backup.
+    if [ "$R_BACKUP" = last ]; then
+        R_BACKUP="$(from_log newest)" || { r_fail no-backup "the log records no backup"; R_RC=3; r_end; }
+        [[ "$R_BACKUP" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || r_abort backup-unreadable "the log's newest backup is not an activation id"
+        r_pass backup-last "the log's newest backup is $R_BACKUP"
+    fi
    copy_exists "$TO" 2>/dev/null || r_abort copy-missing "no copy $TO in $COPIES"
    copy_ok "$TO" 2>/dev/null || r_abort copy-unreachable "$TO is not reachable from refactor and origin/refactor"
    target_schema "$TO" || r_abort copy-schema-unknown "cannot read SCHEMA_VERSION and its chain from $COPIES/$TO/packages/bus/src/store.mjs"
@@ -0,0 +1,347 @@
+#!/usr/bin/env bash
+# Row 233 (I3, #1735): root's upgrade and rollback for a stack whose units are root's system units
+# (stack-release.sh --units external, row 232). r6's activate A1-A5, A2f, R1-R4 and rollback
+# (agents/darkwing/work/queue-195/DESIGN.md), as the I3 brief changes them
+# (docs/plans/2026-10-20_installer-rows.md). Deviations and decisions:
+# agents/darkwing/work/queue-233/BUILD.md.
+#
+#   stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
+#                    [--rehearse-port N] [--timeout S]
+#   stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
+#
+# Root runs it from a root-owned checkout of the reviewed commit, never from a copy, because the
+# stack's user can write the copies. The unit values come from /etc/mosaic-stack/<instance>.args,
+# which I2's install writes: one case2-unit.sh argument per line, the serve flags after `--`. Only
+# --copy changes, and the file changes only after confirm passes. The stack's user runs its steps
+# from its own clone, <home>/src/mosaic-stack, through sudo. Root reads none of that user's output
+# but exit codes and the rehearsal's BACKUP_STORE line (sage-1653), which it copies into its step
+# log after a match against a fixed pattern and decides nothing on.
+#
+# --no-rehearsal is for a stack whose log records no backup. The rehearsal runs either way: exit 3
+# from it (no-backup) goes on only with the flag, and exit 0 with the flag refuses
+# rehearsal-required (sage-1650, sage-1652).
+#
+# The lock /run/mosaic-stack/upgrade-<instance>.lock is held from before A1 to the end of the run,
+# rollback included; every child runs with the lock's descriptor closed. The step log
+# /var/log/mosaic-stack/upgrade-<instance>.log gets one line per step, `<time> <step> rc=N argv:
+# <argv>`, and a last `<time> result exit=N <code> <text>` line.
+#
+# Exit 0 ok, 1 failed with the old stack running, 2 rollback-failed with the units stopped,
+# 3 refused with nothing stopped, 4 usage.
+#
+# Test seams, for packages/cli/tests/stack-upgrade.test.mjs only. Run as root, it refuses any:
+#   STACK_UPGRADE_ETC        default /etc/mosaic-stack
+#   STACK_UPGRADE_RUN        default /run/mosaic-stack
+#   STACK_UPGRADE_LOG_DIR    default /var/log/mosaic-stack
+#   STACK_UPGRADE_UNIT_DIR   default /etc/systemd/system
+#   STACK_UPGRADE_HOSTCHECK  default /usr/local/lib/mosaic-stack/hostcheck.sh (row 240)
+#   STACK_UPGRADE_RENDER     default scripts/case2-unit.sh in this checkout
+#   STACK_UPGRADE_SUDO       default sudo
+#   STACK_UPGRADE_SYSTEMCTL  default systemctl
+set -uo pipefail
+REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+
+usage() {
+    cat >&2 <<'USAGE'
+usage: stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal] [--rehearse-port N] [--timeout S]
+       stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
+USAGE
+    exit 4
+}
+
+if [ "$(id -u)" = 0 ]; then
+    for v in ${!STACK_UPGRADE_@}; do echo "stack-upgrade: refusing: $v is a test seam, and root takes none" >&2; exit 3; done
+fi
+ETC="${STACK_UPGRADE_ETC:-/etc/mosaic-stack}"
+RUN_DIR="${STACK_UPGRADE_RUN:-/run/mosaic-stack}"
+LOG_DIR="${STACK_UPGRADE_LOG_DIR:-/var/log/mosaic-stack}"
+UNIT_DIR="${STACK_UPGRADE_UNIT_DIR:-/etc/systemd/system}"
+HOSTCHECK="${STACK_UPGRADE_HOSTCHECK:-/usr/local/lib/mosaic-stack/hostcheck.sh}"
+RENDER="${STACK_UPGRADE_RENDER:-$REPO/scripts/case2-unit.sh}"
+SUDO="${STACK_UPGRADE_SUDO:-sudo}"
+SYSTEMCTL="${STACK_UPGRADE_SYSTEMCTL:-systemctl}"
+ID_RE='^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$'
+
+ARGV=("$@")
+[ $# -gt 0 ] || usage
+OP="$1"; shift
+case "$OP" in upgrade|rollback) ;; *) usage ;; esac
+INSTANCE="" BUNDLE="" SHA="" MIGRATE=0 NO_REHEARSAL=0 R_PORT=7339 TIMEOUT=60 TO="" RESTORE="" EVENTS=""
+while [ $# -gt 0 ]; do
+    case "$1" in
+        --instance) [ $# -ge 2 ] || usage; INSTANCE="$2"; shift 2 ;;
+        --bundle) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^/[A-Za-z0-9/._+-]+$ ]] || usage; BUNDLE="$2"; shift 2 ;;
+        --sha) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; SHA="$2"; shift 2 ;;
+        --migrate) [ "$OP" = upgrade ] || usage; MIGRATE=1; shift ;;
+        --no-rehearsal) [ "$OP" = upgrade ] || usage; NO_REHEARSAL=1; shift ;;
+        --rehearse-port) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{3,4}$ ]] && [ "$2" -le 65535 ] || usage; R_PORT="$2"; shift 2 ;;
+        --timeout) [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
+        --to) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; TO="$2"; shift 2 ;;
+        --restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ $ID_RE ]] || usage; RESTORE="$2"; shift 2 ;;
+        --events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
+        *) usage ;;
+    esac
+done
+# The default instance only, whose name is dev (row 190); a named instance's units are a later row.
+[ "$INSTANCE" = dev ] || usage
+if [ "$OP" = upgrade ]; then [ -n "$BUNDLE" ] && [ -n "$SHA" ] || usage; else [ -n "$TO" ] || usage; fi
+[ -z "$EVENTS" ] || [ -n "$RESTORE" ] || usage
+
+# --- The step log and the lock ---
+
+umask 027
+LOG="$LOG_DIR/upgrade-$INSTANCE.log"
+LOCK="$RUN_DIR/upgrade-$INSTANCE.lock"
+{ mkdir -p "$LOG_DIR" "$RUN_DIR" && touch "$LOG" && chmod 0640 "$LOG"; } 2>/dev/null ||
+    { echo "stack-upgrade: refusing log-failed: cannot write $LOG or make $RUN_DIR; nothing was changed" >&2; exit 3; }
+logl() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >> "$LOG" || true; }
+# The run's last line, in the log and on the terminal, then the exit.
+finish() {
+    local rc="$1" code="$2"; shift 2
+    logl "result exit=$rc $code $*"
+    if [ "$rc" = 0 ]; then echo "stack-upgrade: $*"; else echo "stack-upgrade: $code: $*" >&2; fi
+    exit "$rc"
+}
+refuse() { local code="$1"; shift; finish 3 "$code" "$* (nothing was stopped)"; }
+# One step: $1 its name, the rest the command, run with the lock's descriptor closed. One log line
+# with its exit and argv; the command's output goes to the terminal only.
+step() {
+    local name="$1" rc=0; shift
+    "$@" 9>&- || rc=$?
+    logl "$name rc=$rc argv:$(printf ' %q' "$@")"
+    return "$rc"
+}
+
+exec 9>> "$LOCK" || refuse lock-failed "cannot open $LOCK"
+flock -n 9 || refuse upgrade-running "another stack-upgrade.sh run holds $LOCK"
+logl "start rc=0 argv:$(printf ' %q' "$0" "${ARGV[@]}")"
+
+# --- The args file and the stack's user ---
+
+# I2's file: one case2-unit.sh argument per line, owned by root with no group or other write. Sets
+# ARGS (the lines), S_USER, S_HOME, BUSINESS, NODE and ARGS_COPY, and the paths derived from them.
+read_args() {
+    local f="$ETC/$INSTANCE.args" st i copies=0 v
+    [ -f "$f" ] && [ ! -L "$f" ] || refuse args-unreadable "no regular file $f"
+    st="$(stat -c '%u %a' "$f")" || refuse args-unreadable "cannot stat $f"
+    [ "${st% *}" = "$(id -u)" ] && (( (8#${st#* } & 8#022) == 0 )) || refuse args-unreadable "$f must be $(id -un)'s with no group or other write (it is uid ${st% *}, mode ${st#* })"
+    mapfile -t ARGS < "$f" || refuse args-unreadable "cannot read $f"
+    S_USER="" S_HOME="" BUSINESS="" NODE="" ARGS_COPY=""
+    for ((i = 0; i < ${#ARGS[@]}; i++)); do
+        case "${ARGS[i]}" in
+            --) break ;;
+            --pm) ;;
+            --user|--home|--business|--copy|--node|--profile|--claude)
+                v="${ARGS[i + 1]:-}"
+                case "${ARGS[i]}" in
+                    --user) S_USER="$v" ;; --home) S_HOME="$v" ;; --business) BUSINESS="$v" ;; --node) NODE="$v" ;;
+                    --copy) ARGS_COPY="$v"; copies=$((copies + 1)) ;;
+                esac
+                i=$((i + 1)) ;;
+            *) refuse args-unreadable "$f has ${ARGS[i]} before --, which case2-unit.sh does not take" ;;
+        esac
+    done
+    [ "$copies" = 1 ] || refuse args-unreadable "$f needs exactly one --copy before --, not $copies"
+    [[ "$S_USER" =~ ^[a-z_][a-z0-9_-]*$ ]] && [[ "$S_HOME" =~ ^/[A-Za-z0-9/._+-]+$ ]] && [[ "$BUSINESS" =~ ^[a-z0-9][a-z0-9-]*$ ]] &&
+        [[ "$NODE" =~ ^/[A-Za-z0-9/._+-]+$ ]] && [[ "$ARGS_COPY" =~ ^[0-9a-f]{40}$ ]] ||
+        refuse args-unreadable "$f lacks a plain --user, --home, --business, --node or --copy"
+    COPIES_DIR="$S_HOME/.local/share/mosaic-stack/copies"
+    CLONE="$S_HOME/src/mosaic-stack"
+    SR="$CLONE/scripts/stack-release.sh"
+    BUS_UNIT="mosaic-bus@$BUSINESS.service" CON_UNIT=mosaic-webui.service
+    # The stack's user's steps: its own HOME and default config, the pinned node first on PATH.
+    AS_USER=("$SUDO" -u "$S_USER" -H env "PATH=$(dirname "$NODE"):/usr/local/bin:/usr/bin:/bin")
+}
+
+# RENDER_ARGS: the args file's lines with copy $1 in place of --copy's value.
+args_with() {
+    local i past=""
+    RENDER_ARGS=()
+    for ((i = 0; i < ${#ARGS[@]}; i++)); do
+        RENDER_ARGS+=("${ARGS[i]}")
+        [ "${ARGS[i]}" != -- ] || past=1
+        if [ -z "$past" ] && [ "${ARGS[i]}" = --copy ]; then RENDER_ARGS+=("$1"); i=$((i + 1)); fi
+    done
+}
+
+# The new args file, written beside the old one before anything stops, moved into place only after
+# confirm passes. $1 is the copy.
+prepare_args() {
+    args_with "$1"
+    NEW_ARGS="$(mktemp "$ETC/.$INSTANCE.args.XXXXXX")" &&
+        printf '%s\n' "${RENDER_ARGS[@]}" > "$NEW_ARGS" && chmod --reference="$ETC/$INSTANCE.args" "$NEW_ARGS" ||
+        { rm -f "${NEW_ARGS:-}"; refuse args-failed "cannot write the new args file in $ETC"; }
+}
+drop_args() { rm -f "${NEW_ARGS:-}"; }
+
+# --- A1 ---
+
+# OLD: the copy both installed units run, from their WorkingDirectory under the stack's copies.
+read_old() {
+    local f ids=() id n
+    for f in "$UNIT_DIR/[email protected]" "$UNIT_DIR/$CON_UNIT"; do
+        [ -f "$f" ] && [ ! -L "$f" ] || refuse units-unreadable "no regular file $f"
+        n="$(grep -c '^WorkingDirectory=' "$f")"
+        id="$(awk -v c="WorkingDirectory=$COPIES_DIR/" 'index($0, c) == 1 { print substr($0, length(c) + 1) }' "$f")"
+        [ "$n" = 1 ] && [[ "$id" =~ ^[0-9a-f]{40}$ ]] || refuse units-unreadable "$f names no copy under $COPIES_DIR"
+        ids+=("$id")
+    done
+    [ "${ids[0]}" = "${ids[1]}" ] || refuse units-disagree "$UNIT_DIR's units name ${ids[0]} and ${ids[1]}"
+    OLD="${ids[0]}"
+    logl "units rc=0 old: $OLD"
+    [ "$ARGS_COPY" = "$OLD" ] || refuse args-differ "$ETC/$INSTANCE.args names $ARGS_COPY and the units name $OLD"
+}
+
+# r6 A1 and row 240: the host's profile still matches, on every host.
+hostcheck() {
+    [ -f "$HOSTCHECK" ] && [ -x "$HOSTCHECK" ] || refuse profile-drift "no executable $HOSTCHECK (row 240)"
+    step hostcheck "$HOSTCHECK" --check || refuse profile-drift "$HOSTCHECK --check failed"
+}
+
+# The stack's user's --plan for the run, $@ its op and arguments; any refusal stops here.
+plan() {
+    local rc=0
+    step plan "${AS_USER[@]}" "$SR" "$@" --units external --copies "$COPIES_DIR" --plan --expect-from "$OLD" || rc=$?
+    [ "$rc" = 0 ] || refuse plan-refused "the stack's user's $1 --plan exited $rc; its output above names the refusal"
+}
+
+# --- The units ---
+
+stop_wait() {
+    local u s deadline=$((SECONDS + TIMEOUT)) rc=0
+    step stop-console "$SYSTEMCTL" stop "$CON_UNIT" || rc=1
+    step stop-bus "$SYSTEMCTL" stop "$BUS_UNIT" || rc=1
+    for u in "$CON_UNIT" "$BUS_UNIT"; do
+        while :; do
+            s="$("$SYSTEMCTL" is-active "$u" 9>&- 2>/dev/null)"
+            case "$s" in inactive|failed) break ;; esac
+            [ "$SECONDS" -lt "$deadline" ] || { rc=1; break; }
+            sleep 1
+        done
+        case "$s" in inactive|failed) logl "wait-down rc=0 unit: $u state: $s" ;; *) logl "wait-down rc=1 unit: $u state: ${s:-unknown}" ;; esac
+    done
+    return "$rc"
+}
+stop_units() { "$SYSTEMCTL" stop "$CON_UNIT" 9>&- || true; "$SYSTEMCTL" stop "$BUS_UNIT" 9>&- || true; logl "stop-units rc=0 units: $CON_UNIT $BUS_UNIT"; }
+
+# A2f: the old units back on the copy they never left, gated on a start at or after this second.
+# An A2 that timed out leaves a unit on its old start, which the gate refuses as stale-start.
+a2f() {
+    local cause="$1" since
+    drop_args
+    since="$(date -u +%s)"
+    step start-bus "$SYSTEMCTL" start "$BUS_UNIT"
+    step start-console "$SYSTEMCTL" start "$CON_UNIT"
+    if step gate "${AS_USER[@]}" "$SR" gate "$OLD" "$BUSINESS" --units external --copies "$COPIES_DIR" --since "$since" --timeout "$TIMEOUT"; then
+        finish 1 failed "$cause failed; $OLD is running again and passed the gate"
+    fi
+    stop_units
+    finish 2 rollback-failed "$cause failed and $OLD failed its gate after the restart; both units are stopped"
+}
+
+# R1-R4: back to OLD after the stack's user staged $1. $2 is the cause: A3, A4 or A5.
+rollback_to_old() {
+    local id="$1" cause="$2"
+    drop_args
+    stop_wait || { stop_units; finish 2 rollback-failed "$cause failed and the units did not stop for the rollback; both units are stopped"; }
+    step revert "${AS_USER[@]}" "$SR" revert "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" ||
+        { stop_units; finish 2 rollback-failed "$cause failed and revert failed; both units are stopped"; }
+    args_with "$OLD"
+    step render-old "$RENDER" "${RENDER_ARGS[@]}" ||
+        { stop_units; finish 2 rollback-failed "$cause failed and the render of $OLD failed; both units are stopped"; }
+    # confirm --rollback exits 1 when OLD passed its gate and 2 when it failed; anything else is a
+    # refusal, with OLD's gate never run.
+    local rc=0
+    step confirm-rollback "${AS_USER[@]}" "$SR" confirm "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" --rollback "$OLD" --cause "$cause" --timeout "$TIMEOUT" || rc=$?
+    [ "$rc" != 1 ] || finish 1 failed "$cause failed; rolled back to $OLD, which passed the gate"
+    stop_units
+    finish 2 rollback-failed "$cause failed and the rollback to $OLD did not pass (confirm --rollback exited $rc); both units are stopped"
+}
+
+# A2 to A5 for copy $1, once A1 and any rehearsal passed. $@ after it is the stack's user's op.
+run_steps() {
+    local id="$1" rc=0; shift
+    prepare_args "$id"
+    # A2
+    stop_wait || a2f A2
+    # A3
+    step stage "${AS_USER[@]}" "$SR" "$@" --units external --copies "$COPIES_DIR" || rc=$?
+    case "$rc" in
+        0) ;;
+        3) a2f A3 ;;
+        *) rollback_to_old "$id" A3 ;;
+    esac
+    # A4
+    args_with "$id"
+    step render "$RENDER" "${RENDER_ARGS[@]}" || rollback_to_old "$id" A4
+    # A5
+    step confirm "${AS_USER[@]}" "$SR" confirm "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" --timeout "$TIMEOUT" ||
+        rollback_to_old "$id" A5
+    if mv -f "$NEW_ARGS" "$ETC/$INSTANCE.args"; then logl "args rc=0 copy: $id"
+    else logl "args rc=1 copy: $id"; finish 1 args-failed "$id is live and confirmed, but $ETC/$INSTANCE.args still names $OLD; put $NEW_ARGS in its place"
+    fi
+}
+
+do_upgrade() {
+    read_args
+    # Before A1: the bundle into the stack's user's clone, then the copy from it (I1).
+    step fetch "${AS_USER[@]}" git -C "$CLONE" fetch -q "$BUNDLE" refs/heads/refactor:refs/remotes/origin/refactor ||
+        refuse fetch-failed "the stack's user could not fetch $BUNDLE into $CLONE"
+    step ff "${AS_USER[@]}" git -C "$CLONE" merge -q --ff-only "$SHA" ||
+        refuse fetch-failed "the stack's user could not fast-forward $CLONE's refactor to $SHA"
+    step copy "${AS_USER[@]}" "$CLONE/scripts/stack-copy.sh" copy --bundle "$BUNDLE" "$SHA" --copies "$COPIES_DIR" ||
+        refuse copy-failed "the stack's user could not make copy $SHA from $BUNDLE"
+    # A1
+    local head
+    head="$(git -C "$REPO" rev-parse HEAD 2>/dev/null)"
+    [ "$head" = "$SHA" ] || refuse head-mismatch "this checkout, $REPO, is at ${head:-unknown}, not $SHA"
+    [ -x "$COPIES_DIR/$SHA/scripts/mosaic" ] || refuse copy-missing "no $COPIES_DIR/$SHA/scripts/mosaic"
+    hostcheck
+    read_old
+    local -a migrate=()
+    [ "$MIGRATE" = 0 ] || migrate=(--migrate)
+    plan activate "$SHA" "$BUSINESS" "${migrate[@]}"
+    rehearse
+    run_steps "$SHA" activate "$SHA" "$BUSINESS" "${migrate[@]}"
+    finish 0 ok "upgrade ok: $SHA is live for $BUSINESS"
+}
+
+# The rehearsal against the log's newest backup, before anything stops. Its output goes to the
+# terminal; root keeps only the BACKUP_STORE line, so the go and the log show which store it used.
+rehearse() {
+    local cap rc bs
+    cap="$(mktemp "$RUN_DIR/rehearse-$INSTANCE.XXXXXX")" || refuse rehearsal-failed "cannot make a capture file in $RUN_DIR"
+    local -a cmd=("${AS_USER[@]}" "$SR" rehearse "$SHA" "$BUSINESS" --copies "$COPIES_DIR" --backup last --port "$R_PORT" --timeout "$TIMEOUT")
+    "${cmd[@]}" 9>&- 2>&1 | tee "$cap" 9>&-
+    rc="${PIPESTATUS[0]}"
+    logl "rehearse rc=$rc argv:$(printf ' %q' "${cmd[@]}")"
+    bs="$(grep -E '^BACKUP_STORE [0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])? schema [0-9a-z]{1,16}$' "$cap" | tail -n 1)"
+    rm -f "$cap"
+    logl "rehearse-store rc=0 ${bs:-BACKUP_STORE none}"
+    case "$rc" in
+        0) [ "$NO_REHEARSAL" = 0 ] || refuse rehearsal-required "the log records a backup and the rehearsal ran; --no-rehearsal is only for a stack with none" ;;
+        3) [ "$NO_REHEARSAL" = 1 ] || refuse no-backup "the log records no backup to rehearse against; only a go that names --no-rehearsal goes on"
+           logl "rehearsal-skipped rc=0 the log records no backup and the go names --no-rehearsal" ;;
+        *) refuse rehearsal-failed "the rehearsal exited $rc" ;;
+    esac
+}
+
+# r6 rollback: to the log's previous copy, which the stack's user checks against --to. No HEAD
+# check: the target is an older copy than this checkout, and no rehearsal.
+do_rollback() {
+    read_args
+    [ -x "$COPIES_DIR/$TO/scripts/mosaic" ] || refuse copy-missing "no $COPIES_DIR/$TO/scripts/mosaic"
+    hostcheck
+    read_old
+    local -a restore=()
+    [ -z "$RESTORE" ] || restore=(--restore "$RESTORE")
+    [ -z "$EVENTS" ] || restore+=(--events "$EVENTS")
+    plan rollback "$BUSINESS" --to "$TO" "${restore[@]}"
+    run_steps "$TO" rollback "$BUSINESS" --to "$TO" "${restore[@]}"
+    finish 0 ok "rollback ok: $TO is live for $BUSINESS"
+}
+
+case "$OP" in
+    upgrade) do_upgrade ;;
+    rollback) do_rollback ;;
+esac
Author	SHA1	Message	Date
jason.woltje	1a1dc80caa	row 233: BUILD.md, the gate and m1-m8 results on `d99166ade` (1201/0, 8 of 8 killed) and the N1 fix that leaves the gate standing (sage-1707)	2026-10-20 23:43:28 -05:00
jason.woltje	08db6ac3f5	row 233: the lock test kills its flock holder's process group (N1, sage-1707) stack-upgrade.test.mjs:312 spawned `flock LOCK sleep 60` and SIGKILLed only the flock, so its `sleep 60` child outlived the test. Spawn it detached and kill the group with process.kill(-pid), as row 232's T20 does (stack-release.test.mjs:370). Test only.	2026-10-20 23:43:21 -05:00
jason.woltje	d99166ade2	row 233: BUILD.md, singles run 2 on `aacfa2d7a` (stack-release 195/0)	2026-10-20 23:23:00 -05:00
jason.woltje	aacfa2d7a4	stack-upgrade: root's upgrade and rollback driver for root-owned units, with the host run sheet; stack-release --backup last and the external plan refusals (row 233, #1735 , installer I3)	2026-10-20 23:00:36 -05:00