Files

865 lines
50 KiB
Markdown

# Row 162 (D1, #1659): stack activate and rollback, health-gated
Owner: Darkwing. Reviewer: Filbert. Gate owner: Sage.
Brief: `docs/plans/2026-10-18_stack-release.md`, section "## Stack activate
and rollback, health-gated (D1)", commit cd9c071ae, blob cfc468c9df88.
Base: df4d5702f for round 2 (origin/refactor after row 125 landed; see
"Candidate round 2"). Round 1 was on cfde62f9f (origin/refactor after row 164 landed). The draft
started on cd9c071ae and moved up with `git checkout --detach`, through
70504d23a (row 124) to cfde62f9f. None of the 162 paths is among row
164's 21; the only base change on them is row 129's appended "Console
Chat" section in `docs/TOOLS.md`, and the TOOLS.md patch applied cleanly.
The store schema on refactor is 3h. The live copy 52f35e7a0 is at 3f,
not 3g as this packet said before 2026-10-18T19:40Z: 3g came with row
141 (de352dfaf), which isn't live yet.
Worktree: `~/darkwing-scratch/r162/wt`.
Status: built and tested on Sage's slot (19:25:10Z to 19:46:14Z on
2026-10-18). `stack-release.test.mjs` 58/58 and `stack-copy.test.mjs`
12/12, each run singly. 47 mutants killed before the steps change, then
M41, M45b, M46 and M47 to M49 killed after it. The scratch-root plan for
Mos is in `scratch-plan.txt` next to this file.
## Files
| File | Change |
|---|---|
| `scripts/stack-release.sh` | new: `adopt`, `activate`, `rollback`, `status` |
| `scripts/stack-copy.sh` | a `reachable` function split out of `copy_commit`, and a guard so the file can be sourced for its helpers |
| `packages/cli/tests/stack-release.test.mjs` | new |
| `docs/TOOLS.md` | the "Pinned stack update" section only |
| `packages/cli/README.md` | the pinned-stack paragraph only |
| this packet | |
Nothing else changes. `bus-service.sh` and `webui-service.sh` are called
as they are.
## Shape of the script
```
scripts/stack-release.sh adopt BUSINESS
scripts/stack-release.sh activate ID BUSINESS [--pm] [--migrate] [--timeout S] [--plan]
scripts/stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan]
scripts/stack-release.sh status BUSINESS
common options: --copies DIR, --bin DIR (passed to stack-copy.sh and the install scripts)
```
Exit codes: 0 ok; 1 the gate failed and the rollback inside the run passed;
2 `rollback-failed`, the stack is left stopped; 3 a refusal, nothing
changed; 4 usage. Usage errors write no log line. Every other outcome
writes exactly one final line.
`--pm` on `activate` must match the pointer's `pm`. The brief has
`activate` install "exactly the pointer's `--pm`", so the flag is a check
the operator states, not a way to change it. A mismatch refuses
`unit-flags-drift`.
The header comment states decision 96's four conditions (item 10) and
points at decisions 94 and 102 for who starts the Console.
### Where things come from
- Data root: `load_config` from `scripts/common.sh`, run from the canonical
checkout (`MOSAIC_CONFIG` is honoured, as everywhere). The bus resolves
its root the same way, so both see one root.
- The canonical checkout: the `REPO` of this script. `bus-service.sh`,
`webui-service.sh` and `stack-copy.sh` are this checkout's, as in the
window plans. `webui-service.sh install` already refuses outside the
canonical checkout.
- Unit names: `[email protected]` and `mosaic-webui.service`.
Unit directory: `${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user`, the
default both install scripts write to.
- `claude`: `MOSAIC_CLAUDE_BIN=` from `systemctl --user show
mosaic-bus@B.service -p Environment`.
- `pm`: a `--pm` token after `bus start B` in the bus unit's `ExecStart`
`argv[]`.
- `console`: the `argv[]` tokens after `serve` in the Console unit's
`ExecStart`, minus `--checkout X`. The install script adds `--checkout`
itself.
- The copy's `SCHEMA_VERSION` and chain: text read from the copy's
`packages/bus/src/store.mjs`. `export const SCHEMA_VERSION = 'X';` must
match exactly once, and the chain is the `from: 'X'` lines in file
order. The copy's code is never imported or run for this. A copy where
either read fails refuses `copy-schema-unknown`. The migration file
names would also work, but `3b.sql` is a base, not a step, so the
store.mjs lines are the cleaner source.
### Sharing with stack-copy.sh
`copy_commit`'s reachability loop moves into `reachable FULL`, and
`copy_commit` calls it. The dispatch at the bottom gets
`[ "${BASH_SOURCE[0]}" = "$0" ] || return 0`, so sourcing defines the
functions and runs nothing. `stack-release.sh` sources it inside a
subshell function, so stack-copy's `die`, `parse` and globals never mix
with its own:
```
copy_ok() ( source "$REPO/scripts/stack-copy.sh"; ID="$1"; COPIES="$COPIES_DIR"; copy_root >/dev/null && reachable "$ID" )
```
`check` and `wrapper` run as commands, as an operator would run them.
## Pointer and log (item 1)
`<dataRoot>/state/stack-active.json`, written to a temp file in `state/`
and renamed:
```json
{"copy":"<40 hex>","business":"mosaic-stack","pm":true,
"claude":"/home/u/.local/bin/claude","console":["--port","7332","--business","mosaic-stack"],
"schema":"3g","activation":"20261018T170000Z-52f35e7a0abc","at":"2026-10-18T17:00:00Z"}
```
`schema` is the copy's `SCHEMA_VERSION` at the time it went live. On
`adopt` that is the adopted copy's, since a running host proves the store
is at its copy's schema.
`<dataRoot>/state/stack-activation-log.jsonl`, appended with `>>`, one
JSON object per line:
```json
{"at":"...","op":"activate","from":"<copy>","to":"<copy>","backup":null,"result":"ok"}
{"at":"...","op":"activate","from":"...","to":"...","backup":null,"result":"refused","code":"unit-flags-drift"}
{"at":"...","op":"activate","from":"...","to":"...","backup":"<activation>","result":"gate-failed","code":"healthz","rollback":"ok"}
{"at":"...","op":"activate","from":"...","to":"...","backup":"<activation>","result":"rollback-failed","code":"bus-status","rollback":"failed","rollbackCode":"healthz"}
```
Brief item 4 says "each step is logged". Ruling 3: a run that gets past
the refusals writes `result: "started"` before step 1 and one final line
at the end. The per-step detail goes to stdout and to `steps.log` inside
the run's rollback set. `rollback` reads final lines only.
A `started` line with no final line after it for the same activation is
an interrupted run. `status` shows it, and every later `adopt`,
`activate` or `rollback` refuses `interrupted-run`, naming the
activation. The script has no verb to clear it. Clearing it is Jason's or
Mos's call (ruling 3).
Every `adopt`, `activate` and `rollback` takes `flock -n` on
`state/stack-release.lock` first and refuses `run-locked` when another
run holds it. `status` and `--plan` take no lock (ruling 6).
Activation id: `<UTC %Y%m%dT%H%M%SZ>-<first 12 of the target copy>`. The
rollback set directory is made with `mkdir -m 0700` and no `-p`, so a
clash fails before anything changes.
## adopt (item 2)
1. The run lock, else `run-locked`. No `interrupted-run` in the log.
2. No pointer, else `already-adopted`.
3. `stack-copy.sh check --business B` passes, else `check-failed` with
its output.
4. Read `claude`, `pm` and `console` from `systemctl show`. No
`MOSAIC_CLAUDE_BIN` refuses `no-claude-pin` (ruling 5).
5. Write the pointer, then log `ok`.
It stops, starts and installs nothing.
## activate (items 3 to 6)
### Refusals before anything changes (item 3)
The checks run in this order. Each one gives exit 3, one log line with its
code, and no other write:
1. `run-locked`, then `interrupted-run`. `run-locked` writes no log
line, because the lock holder may be writing the log.
2. `no-pointer`, then `business-mismatch`: the pointer names another
business. The brief names no code for it; the pointer holds one
business, so a run for another one would install units the pointer
doesn't describe.
3. `copy-missing` (`copy_root` fails) or `copy-unreachable` (`reachable`
fails). The brief names no codes for these two, so these are mine.
4. `copy-schema-unknown`, from the store.mjs read above.
5. `already-active`: ID is the pointer's copy.
6. `instance-busy` (ruling 1): another business's `mosaic-bus@` instance
is in any state but inactive or failed, since the copy and wrapper
switch reach it too; or this business's instance is activating,
deactivating or reloading. The message names the instance and its
state. The read is `install_guard`'s `list-units` call. A literal
reading (refuse any running instance) would refuse every activate,
because the bus is running when activate starts.
7. `drop-in-override`: the `install_guard` scan of
`systemd-analyze --user unit-paths`, naming the file.
8. `no-claude-pin` (ruling 5): the effective bus unit has no
`MOSAIC_CLAUDE_BIN`.
9. `unit-flags-drift`: effective `pm`, `claude` or `console` differ from
the pointer's, or `--pm` differs from the pointer's `pm`. Prints
`pointer:` and `unit:` lines for each field that differs.
### The run (item 4)
1. Capture to `state/stack-rollback/<activation>/` (0700): the two unit
files from the unit directory, the drop-in directories
`[email protected]`, `mosaic-bus@B.service.d` and
`mosaic-webui.service.d` under that directory when present, the
wrapper `$BIN/mosaic` with `stat -c %a` in `WRAPPER_MODE`, the
`systemctl show` output for both units (it includes `DropInPaths`), and
`SHA256SUMS` over the captured files. `cp -p` throughout. This is
`capture-rollback.sh`. The `started` line is already written, so a
failure here writes the final line `refused` with code
`capture-failed` and exits 3. Nothing has stopped, and only the
partial rollback set exists.
2. `systemctl --user stop mosaic-webui.service`, then the bus unit. Poll
`is-active` for both to read inactive or failed, then the stopped
check: no `<dataRoot>/bus/writer.lock`, and no `/proc/*/fd` link
resolving under `<dataRoot>/bus/`. Bounded by `--timeout`. A timeout
goes to item 6 with code `stop-timeout`.
3. Schema read: see "The store read" below. A `-wal` or `-shm` beside
the store goes to item 6 with `store-unclean`.
- The same as the copy's `SCHEMA_VERSION`: no backup.
- In the chain and older, no `--migrate`: `migration-needs-flag`,
item 6.
- Older with `--migrate`: back up, then `node
<copy>/packages/bus/src/migrate-schema.mjs <dataRoot>`. A non-zero
exit goes to item 6 with code `migrate-failed` and the store
restored.
- Newer, or not in the chain: `store-newer`, item 6.
- No `bus.sqlite` at all (a fresh root): no backup, the new host
creates the store.
4. `bus-service.sh install --copy ID --copies DIR [--pm] --claude PATH`,
then `webui-service.sh install --copy ID --copies DIR <console
flags>`, then `stack-copy.sh wrapper ID --copies DIR --bin DIR`. Any
non-zero exit goes to item 6 with code `install-bus`, `install-console`
or `wrapper`.
5. Start the bus, then the Console.
From step 2 on, `trap` on INT, TERM and HUP sends the run to item 6, once.
A Ctrl-C mid-window leaves the old stack started, not a stopped one. The
one exception is after a `rollback --restore` has swapped the store; see
ruling 4 under rollback.
Most of the script's functions run as `f || handler`. Bash turns `set -e`
off inside such a call, so every step in them that can fail carries its
own `|| return 1` or sits in an `&&` chain.
### The backup
`<dataRoot>/backups/stack/<activation>/`, as `backup.sh` does it: umask
077, `mkdir -m 700` for the directory and `bus/`, `cp -p` of `bus.sqlite`
(only that file: a `-wal` or `-shm` was already refused as
`store-unclean`), `SOURCE.sha256` taken before the copy and
checked against `SHA256SUMS` after it, `MODES.txt`, `LISTING.txt`,
`DATA_ROOT`, `STAMP` (UTC ISO with milliseconds, the same form as
`events.at`), `SCHEMA` (the version read in step 3) and `ROLLBACK_SET`.
`backup.sh`, `restore.sh` and `stopped-check.sh` here mean Sage's
committed window scripts in `agents/sage/work/window-plan/repin-de352dfaf/`.
`stack-release.sh` carries its own copies of their steps and doesn't call
them. The stopped check runs again after the copy. Any failure goes to item 6
before the migration runs.
### The store read
`node -e` with `node:sqlite`:
`` new DatabaseSync(`${pathToFileURL(file).href}?immutable=1`, { readOnly: true }) ``,
then `SELECT value FROM meta WHERE key='schema_version'`, then close.
The probe settled the open mode. It ran on Sage's slot (sage-0738),
16:02:53Z to 16:03:08Z, node v26.8.1, on a WAL-mode fixture store in
`~/darkwing-scratch/r162/probe/store/`, never the live store or a copy of
it. The fixture was rebuilt before each trial and closed cleanly, so only
`bus.sqlite` was present at the start of each one.
| Open | Read | Files after | Main file mtime, sha256 |
|---|---|---|---|
| `{ readOnly: true }`, store stopped | version and count right | `bus.sqlite-shm` and `bus.sqlite-wal` created and left behind | unchanged |
| `file:...?immutable=1` with `readOnly`, store stopped | version and count right | none created | unchanged |
| `{ readOnly: true }` while a WAL connection holds the store | sees the holder's row still in the WAL | the holder's `-wal` and `-shm`, nothing new | not compared |
A plain read-only open of a stopped store writes two files next to it, so
it fails Sage's condition, and immutable is the mode. Immutable tells
SQLite the file can't change and makes it skip the WAL entirely. That is
sound only when there is no WAL to skip. So the rule (Sage approved it in
sage-0739):
- Before every stopped-store read, `bus.sqlite-wal` and `bus.sqlite-shm`
must both be absent. A clean close checkpoints and removes them. One
left behind means the bus did not close the store cleanly, and an
immutable read could return a stale version or count. The run refuses
`store-unclean`, naming the file, and goes to item 6. `stack-release.sh`
never opens, copies or moves that store. Whatever happens to it next is
Jason's or Mos's call. There is no copy fallback.
- With both absent, the read uses immutable.
- An unreadable version goes to item 6 with code `schema-unreadable`.
The `--restore` count in item 7 step 5 is taken before the stop, from a
store the running bus holds. There the `-wal` and `-shm` belong to the
bus, the bus unit must be active, and the read is `{ readOnly: true }`,
which the probe shows adds no file. If the bus unit isn't active, the
count is a stopped-store read under the rule above.
The no-write test below catches a read that creates a file: it compares
the file set and the sums under `bus/` before and after.
### The health gate (item 5)
Polled every second until all pass or `--timeout` (default 60) runs out.
The failing check named at the end is the first one still failing:
| Check | Code |
|---|---|
| `systemctl --user is-active` for both units | `units-inactive` |
| `<copy>/scripts/mosaic bus status --json` exits 0, `host.business` is B, `host.live` is true, `host.pid` is the bus unit's `MainPID`, `boot.ok` is true, `boot.business` is B, `boot.at` is at or after the moment step 5 began | `bus-status` |
| `GET http://127.0.0.1:<port>/healthz` returns 200, port from the `--port` flag in `console`, else 7332 (the `webui-service.sh` default) | `healthz` |
| `stack-copy.sh check --business B` exits 0 | `copy-check` |
| with `pm`: exactly one `launched` line in the bus unit's journal since step 5, and its run is live in `<copy>/scripts/mosaic launches list --json`, for B, with `startedAt` at or after the moment step 5 began | `pm-launch-unconfirmed` |
Ruling 2 (sage-0736, refined in sage-0738): the check must match the PM
role without reading the business file (Jason's org file) and without
naming `launch.by`. The session entries carry `instance`, but nothing in
them or in `launches/<business>.jsonl` marks a launch as the PM's. The
`bus start --pm` path is the only code that knows. It prints
`launched <instance>: run <run>, <harness> <model>` to stdout
(`packages/cli/src/cli.mjs:503`, the only place that prints such a line),
and under the unit that stdout is the journal.
So the gate reads `journalctl --user -u mosaic-bus@B.service --since
@<epoch seconds of step 5> -o cat`. The `@` form is unambiguous. This
host's journalctl reads a bare time as local CDT, and in Q14 an empty
window passed for a clean start that way. Of the lines it returns, it
keeps those starting with `launched `. It passes only when:
- there is exactly one such line;
- it matches `^launched (\S+): run (\S+), `;
- that run is live in `launches list --json` for B, with `startedAt` at
or after step 5.
Zero lines, two or more, a line that doesn't match, or a run that isn't
live all give the one code `pm-launch-unconfirmed`, and the message says
which. A human `mosaic launch` in the same window writes no line to the
unit's journal, so it can't pass for the PM.
The review should know that this check depends on the wording of that
stdout line. A change to it in `cli.mjs` makes the gate fail closed, not
pass. A test pins the dependency: T10b's unparseable-line subtest feeds a
line that still starts with `launched ` but has a later word changed
(`job` for `run`) and expects the refusal.
Follow-up, not in D1 (Sage files it if wanted): a bus-side PM marker, for
example a `pm: true` field on the launch event the `--pm` path writes,
would let the gate drop the journal read and the stdout-format
dependency.
`MainPID` equals `host.pid` because `scripts/mosaic` ends in `exec node
...cli.mjs`. Every `bus start` that loads the business calls
`recordBootOk` (`packages/cli/src/cli.mjs:494`), which writes `boot.at` as
the current time, so a `boot.at` from before step 5 means the status
describes an earlier start.
The healthz fetch is `node -e fetch(...)` with a 5 s abort, as
`webui-service.sh status` does it.
On a pass: rename the pointer into place, log `ok`, exit 0.
### Rollback inside the run (item 6)
1. Stop the Console, then the bus, and wait as in step 2.
2. Put back the two unit files and the drop-in directories from the
rollback set (temp file and rename each), and the wrapper bytes and
mode. Check them against `SHA256SUMS`. `daemon-reload`.
3. If step 3 of this run took a backup, restore it as `restore.sh` does
(a `rollback --restore` backup is never restored here): check
`DATA_ROOT`, the stopped check and `sha256sum -c`; rename `bus` to
`bus.post-attempt-<UTC stamp>`; `mkdir -m 700 bus`; copy in; check the
sums and that the file set matches exactly. Nothing is deleted.
4. Start the bus, then the Console. Run the gate against the pointer's
copy.
5. Log `gate-failed` (or the step's code) with `rollback: ok`, exit 1. If
the rollback's gate fails: print both results, stop both units, log
`rollback-failed` with both codes, exit 2. No retry.
## rollback (item 7)
1. `no-pointer`.
2. Target: scan the log newest first for `result: "ok"` lines with
`op` adopt, activate or rollback, and take the first `to` that isn't
the pointer's copy. None found refuses `no-previous`. This is
`release.sh`'s rule, and it shares that rule's property: two rollbacks
in a row go back and forth between two copies.
3. The item 3 refusals apply to the target (copy, schema, busy,
drop-ins, flag drift).
4. `schema-differs`: the pointer's `schema` is not the target's
`SCHEMA_VERSION` and there's no `--restore`. Decided from the pointer
before anything stops.
5. With `--restore ACTIVATION`: `backups/stack/ACTIVATION/` must exist
with `SHA256SUMS`, a `DATA_ROOT` equal to this root, and a `SCHEMA`
equal to the target's `SCHEMA_VERSION`, else `restore-mismatch`. Print
the post-attempt path the run will keep, and the count of events with
`at` after the backup's `STAMP`, from a read-only open of the store
before anything stops. Count above 0 and `--events N` not equal to it
refuses `post-backup-events`. No change.
6. Steps 4.1 to 4.5 with the target. In step 3, the stopped store's
version must equal the pointer's `schema`, else `store-differs`, item
6. With `--restore`, the count is read again after the stop. A
different count goes to item 6 with `post-backup-events`, since an
event landed between the check and the stop. Then the restore runs in
place of the migration.
7. The gate, as for activate. A pass renames the pointer (with the
target's schema) and logs `ok` with `backup: ACTIVATION`.
Ruling 4 (sage-0736, confirmed in sage-0738) has two cases:
- A `rollback --restore` that fails after the restore has moved the old
`bus/` aside: the run does not go through item 6. That covers the gate
and also an install, start or signal failure after the swap; the script
treats every one the same and logs `rollbackCode: "not-attempted"`. A
restore that fails before the move goes through item 6 like any other
step, since the store hasn't changed. Putting the newer store back would
be a second swap, which is a retry under another name. The run prints
both results, stops both units, logs `rollback-failed` and exits 2. The
post-attempt directory and the restored `bus` stay where they are for
Jason. The unit files and wrapper also stay as the run installed them,
since they match the restored store. Nothing is swapped back. Tested in
T14b.
- A `rollback` without `--restore` whose gate fails: the store was never
swapped, so the run goes through item 6 as `activate` does. It puts back
the unit files and wrapper from its rollback set, restarts the pointer's
copy and runs the gate on it. Exit 1 with `rollback: ok`, or exit 2 with
`rollback-failed` if that gate fails too. Tested in T14c.
## --plan (item 8)
It reads the pointer, the log, the target copy's store.mjs and the
store's schema version. It writes nothing: no lock, no log line, no file
under the data root.
Mos takes the `--plan` output to Jason before the combined re-pin
(mos-0741, sage-0745), so a migration must be stated plainly. For a
store at 3f and refactor at 3h, `activate --plan` prints:
```
store schema: 3f (read from the stopped store)
target schema: 3h
migration: 3f to 3h, after a backup of the stopped store
migration steps: 3f to 3g, 3g to 3h
needs: --migrate (not given; the run would roll back with migration-needs-flag)
needs: Jason present for this run (decision 96 condition 2)
```
I added the `migration steps:` line in the slot on 2026-10-18, before the
scratch-root run (see "The scratch-root plan for Mos"). It walks the target store.mjs's `from:` and
`to:` pairs from the store's version, as text, the same way
`copy_schema` reads the version, so nothing in the copy runs. When the
lines don't pair up, or the walk doesn't reach the target's version, it
prints `migration steps: cannot be read from <file>` and the rest of the
plan is unchanged. The run itself still calls `migrate-schema.mjs`, which
applies the copy's own chain; the steps line only names it.
The store version comes from the store itself. While the bus unit is
active the read is `{ readOnly: true }`, which adds no file next to a
store the bus holds (the probe). With the bus stopped and no `-wal` or
`-shm`, it is the immutable read. With no store file, a `-wal` or `-shm`
present, or a failed read, it falls back to the pointer's `schema` and
says which case applied. When the store and the pointer disagree, a
`note:` line says so; the run itself goes by the stopped store.
Other cases print `migration: none` (same version), a `store-newer`
warning (not in the target's chain) or, for rollback, `restore: ... needs
--restore ACTIVATION` with the activation the log's backup field names
for the pointer's copy. Then the exact `bus-service.sh`,
`webui-service.sh` and `stack-copy.sh wrapper` command lines. Exit 0.
The refusals it can see before a run (no-pointer, business-mismatch,
copy, already-active, no-previous) still print, with exit 3 and no log
line.
After the migration or restore lines comes the `commits:` block
(sage-0778): `commits: the run adds N` when the pointer's copy is an
ancestor of the target, `commits: the run takes away N` when the target
is an ancestor of the pointer's copy (a rollback), then the
`git log --oneline --no-merges` lines, newest first, at most 200. When
neither copy is an ancestor of the other, `--plan` refuses
`copies-unrelated`. My 16:53Z message to Sage said "refuse if FROM isn't
an ancestor of TO". The code is wider than that on purpose, so a
rollback still lists what it takes away. T25 and T26 test it.
### The scratch-root plan for Mos (sage-0837)
Mos asked (mos-0765) for the combined re-pin's `--plan` output with the
3g to 3h migration and rows 148, 141, 143, 128, 124 and 149. Before any
live adopt there is no pointer, so I run it against a scratch root under
`~/darkwing-scratch/r162/`: a data root, a config naming it, a
hand-written pointer and a copies directory, all scratch. Nothing under
`~/.mosaic-dev`, the live copies directory or the units is written.
- The live copy id: `systemctl --user show mosaic-bus@mosaic-stack.service
-p WorkingDirectory` at 2026-10-18T19:32:16Z gave
`.../copies/52f35e7a0933e9fa845f59860a5de1709f9e5756`. It reads the
unit's property and writes nothing. sage-0859 also allowed a read-only
`systemctl --user is-active` on the bus unit name; the run never made
that call, because with no store file `plan_store` doesn't reach it.
- The pointer's `schema` is that copy's `SCHEMA_VERSION`, read with
`git show 52f35e7a0:packages/bus/src/store.mjs`: `3f`. I had expected
3g. That is why the steps line exists: `migration: 3f to 3h` alone
doesn't show the 3g to 3h step Mos asked for, and hand-writing 3g would
have been a false pointer.
- The target copy, from `scratch-plan-steps.log`:
`(cd ~/darkwing-scratch/r162/wt && scripts/stack-copy.sh copy
606fafd43154ded993a503933681198e737987cd --copies
~/darkwing-scratch/r162/plan/copies --allow-worktree)` at 19:45:54Z,
copied by 19:45:56Z. Removed at 19:45:56Z with `chmod -R u+w` and
`rm -rf` of that one copy directory; the copies dir was left with 0
entries. 606fafd43 was origin/refactor at the run; it moved past
cfde62f9f after sage-0859.
- A header I put above the script's output names the scratch root and
says the pointer is hand-written. The script prints no such header
itself.
- The scratch root has no `bus/bus.sqlite`, so the store schema line
reads `from the pointer; there is no store file`. `--plan` runs only
`check_target` and `plan`, not the unit checks, so the only other reads
are git in the checkout and the copies directory. The data root held
only `state/stack-active.json` and an empty `bus/` after the run.
The result, `scratch-plan.txt` (134 lines, exit 0): `migration: 3f to
3h`, `migration steps: 3f to 3g, 3g to 3h`, both `needs:` lines, then
`commits: the run adds 112`. All six rows are in the list (grep hits per
row: 148 7, 141 5, 143 8, 128 9, 124 9, 149 18). The range also carries
row 164, the row 165 brief and everything else since 52f35e7a0.
Mos then narrowed the window target to `8e7ea89a1`, the six rows only
(mos-0810 D1-D2, sage-0871). I reran the same plan in a new slot with
`r1/plan-8e7.sh`, a copy of plan.sh with the target, the scratch root
(`plan-8e7`) and the output file changed. It ran from this worktree
while it still matched the r1 manifest. The result is
`scratch-plan-8e7ea89a1.txt` (114 lines, exit 0). The `migration` and
`migration steps` lines are unchanged. The plan lists 92 commits, the
same count as `git rev-list --count 52f35e7a0..8e7ea89a1`. Each of the
six landing commits appears once. Row 164's `008746b6e` and `850484a7d`
are not in it. Of the 92, only the six touch code paths. The step log
is `scratch-plan-8e7ea89a1-steps.log`. Its "plan: written to" line names
the wrong file, and `live-reads.md` explains why. Both 606fafd43 files
stay as they were, since this review uses them.
Sage forwards the output to Mos as a scratch-root plan. The real
`--plan` against the live pointer runs again in Mos's window before any
adopt.
## status (item 9)
The pointer, the last five log lines, any interrupted run,
`stack-copy.sh check` output and result, `<pointer copy>/scripts/mosaic
bus status`, whether the unit flags match the pointer, and the drift:
whether the copy is an ancestor of `refactor` and
`git rev-list --count COPY..refactor`. It only reads.
## Tests (`packages/cli/tests/stack-release.test.mjs`)
Fixture, built once per file:
- A temp canonical repo, made the way `stack-copy.test.mjs` `setup` does
it: `git ls-files -co` of scripts and packages copied in, `git init -b
refactor`, the light package.json. I copy the setup code rather than
import it, because importing a test file runs its tests too. The base
commit is F.
- On `refactor`, empty commits A, B, C1 and C (subjects `row 901: A` to
`row 904: C`). A side branch from A holds S and merges into `refactor`
as M, with `--no-ff`. A branch named `unreached` from A holds U.
`refactor` and `origin/refactor` both point at M.
- Read-only stub copies under a temp `copies/<id>/` for F, A, B, C1, C, S
and U. M has no copy, so it is the `copy-missing` case. Each copy has
`scripts/mosaic`, `packages/webui/src/cli.mjs` (present only, never
run), `packages/bus/src/store.mjs` and
`packages/bus/src/migrate-schema.mjs`. The store file in A, B, C1, S
and U says `3g` with a `from: '3f'` step. C says `3h` with `3f` and
`3g` steps. F has no `SCHEMA_VERSION`, which gives
`copy-schema-unknown`.
- The stub migrate records the `backups/stack/` entries it saw, and
whether each has `SHA256SUMS`, in `migrate.json`, then sets the store
to `3h` with `node:sqlite`. The `migrate-fails` fault sets
`3h-partial` and exits 1.
- The stand-ins `systemctl`, `journalctl` and `systemd-analyze` are node
scripts with a `#!<node>` shebang in a `sbin/` directory that comes
first on `PATH`. All three exist, so the real ones never run. A
`package.json` with `"type":"commonjs"` in the fixture's base
directory covers them, the holder and the copies' `scripts/mosaic`.
Per test: a temp `HOME`, `XDG_CONFIG_HOME`, `XDG_DATA_HOME`, data root,
`MOSAIC_CONFIG` (a valid config naming the temp data root), `bin/` with
the Q81 wrapper, `mosaic-old` and a fake `claude`, and a stand-in state
directory with a `faults/` directory. A fault is a file named for the
copy it hits, such as `healthz-<id>`, so one fault can fail the new copy
and leave the rollback target alone.
`seed` builds the stack the way an operator would: a real WAL-mode
`bus.sqlite` with a `meta` row and three events, then the real
`bus-service.sh install`, `webui-service.sh install` and
`stack-copy.sh wrapper` against the fixture copy, then a start of both
units. The pointer comes from a real `adopt`, or is written by hand with
no log (T23, T25), or is left out.
The stand-in `systemctl`:
- `show UNIT -p ...` reads the installed unit file and its drop-ins
(`[email protected]` and `<unit>.d`), fills in `%i` and `%h`, and
answers `LoadState`, `WorkingDirectory`, `ExecStart={ path=... ;
argv[]=... ; }`, `Environment`, `MainPID` and `DropInPaths`. The
`check-<id>` fault moves the Console's `WorkingDirectory`, so
`stack-copy.sh check` fails.
- `list-units`, `is-active`, `is-enabled`, `start`, `stop` and
`daemon-reload` keep unit state in `units.json` (written through a temp
file and a rename) and append every call to `calls.log`, so the order
is testable.
- `start` of the bus runs a holder process. The holder opens
`bus.sqlite` in WAL mode and keeps it open, writes `bus/writer.lock`
and a `host.json` with the business, its pid and a boot status, and
closes the store and removes the lock on SIGTERM. That gives the real
`-wal` and `-shm` files of a held store, and a clean close on stop. The
holder starts through `bash -c` with every fd above 2 closed, so it
never inherits the script's fd 9 lock. `SD_CLOCK_OFFSET` backdates the
seeded host and its PM run by two minutes, so they predate any run's
start.
- `start` of the bus with `--pm` in the unit's ExecStart appends the
`launched` line, as `cli.mjs:503` writes it, to the stand-in journal,
and adds the run to `launches.json`. A `pm-<id>` fault picks another
case: no line, two lines, a changed word, a run missing from the list,
an early `startedAt`, a human launch with no line, a run for another
business, a dead run.
- `stop` sends SIGTERM and waits for the holder to exit. The one-shot
`stop-leaves-wal` and `stop-leaves-shm` faults leave an empty `-wal` or
`-shm` beside the store, as an unclean close would. Each bus start and
stop records the name, size, mtime and sha256 of every file under
`bus/` in `snap.jsonl`, so T22 can compare the store before and after
the script's read.
- `start-dies-<id>` sets the bus to `failed`.
The stub `scripts/mosaic` prints `bus status --json` from `host.json`,
with `live` recomputed from `/proc`, and `launches list --json` from
`launches.json`. The `status-<id>` and `launches-<id>` faults make it
fail. The Console is an HTTP server inside the test process on a free
port. It answers 200 when the stand-in says the Console is active, and
500 when a `healthz-<id>` fault names the Console unit's current copy.
That is why the test runs the script with async `spawn`: `spawnSync`
would block the server.
`systemd-analyze --user unit-paths` prints the temp unit directory.
The stand-in `journalctl` logs the `--since` argument it got and prints
the stand-in journal lines for the unit at or after it. It accepts
`--since` only as `@<epoch seconds>` or `YYYY-MM-DD HH:MM:SS UTC`, and
exits 1 on any other form, so a bare local time gives no lines.
Every `run` call also checks T17: the listing of `backups/stack/`,
`state/stack-rollback/`, `bus.post-attempt-*` and `copies/` taken before
the run is a subset of the one after.
Clean-up kills only the holder pids the stand-in logged, after checking
each one's `/proc/<pid>/cmdline` names this file's holder.
Gate failure cases run with `--timeout 3`.
| # | Case | Brief |
|---|---|---|
| T1 | `adopt` writes the pointer (claude and console from `show`) and one `ok` line; a second adopt refuses `already-adopted`; a failing check refuses `check-failed` with no pointer | item 2 |
| T2 | `activate` installs exactly the pointer's `--pm`, `--claude` and Console flags (unit file text) | items 3, 4 |
| T3 | a hand-edited unit (each of pm, claude, a console flag) refuses `unit-flags-drift`, prints the difference, changes nothing | item 3 |
| T4 | each item 3 refusal: units, wrapper, pointer and store unchanged by sums, exactly one new log line with the code, no `start` or `stop` in `calls.log`; `instance-busy` has two subtests (another business's instance active, this one's activating) and both name the instance and state | item 3, ruling 1 |
| T5 | no-schema activate: `calls.log` shows stop Console, stop bus, daemon-reloads, start bus, start Console in order; pointer names C1; no `backups/stack/` | item 4 |
| T6 | B to C without `--migrate`: `migration-needs-flag`, old stack started, pointer unchanged | item 4.3 |
| T7 | B to C with `--migrate`: backup exists before the migrate stub ran (the stub records whether it saw the backup's sums), pointer schema `3h` | item 4.3 |
| T8 | B to C with `--migrate` and a failing migration: post-attempt dir exists, restored files match the backup's sums, old stack started | item 6 |
| T9 | `store-newer` (store at a version outside C's chain) | item 4.3 |
| T10 | each gate check failing alone (five subtests): rollback, pointer unchanged, unit files and wrapper back byte for byte, log names the check | items 5, 6 |
| T10b | the PM check, one subtest per case, each refusing `pm-launch-unconfirmed` with the case named: no `launched` line; two lines; a line that starts with `launched ` but has a later word changed (`launched mosaic-pm: job R, ...`); one line whose run isn't in `launches list`; one line whose run has `startedAt` before step 5; a live human `mosaic launch` session after step 5 with no journal line; a run for another business; a run that isn't live. A last subtest passes with one well-formed line and a live run, and checks the `--since` argument the stand-in received starts with `@` | item 5, ruling 2 |
| T11 | rollback gate fails too: `rollback-failed`, exit 2, both units stopped, no `start` after the failing one | item 6 |
| T12 | `rollback` finds the previous copy from the log | item 7 |
| T13 | `rollback` across a migration refuses `schema-differs`; with `--restore` it restores and goes live on the old copy | item 7 |
| T14 | `--restore` with post-backup events and no or a wrong `--events N` refuses `post-backup-events`; the right N proceeds | item 7 |
| T14b | a `rollback --restore` whose gate fails: `rollback-failed`, exit 2, both units stopped, the post-attempt dir and the restored `bus` in place, no rename after the restore | item 7, ruling 4 |
| T14c | a `rollback` without `--restore` whose gate fails: item 6 runs, the unit files and wrapper are back byte for byte from the run's rollback set, the pointer's copy is started and gated, exit 1 with `rollback: ok`; the store's file set and sums are unchanged | item 7, ruling 4 |
| T15 | `--plan` on activate and rollback prints the migration or restore and the flags; units, pointer, log and store unchanged. After an activate to C with `--migrate`, `rollback --plan` prints, line for line, the restore with the activation and `commits: the run takes away 2` | item 8 |
| T25 | sage-0778 ask 1: pointer on B, `activate S --plan` (S is on a side branch, so neither copy is an ancestor of the other) exits 3 with exactly `stack-release: refusing copies-unrelated: neither <B> nor <S> is an ancestor of the other, so the plan cannot list the commits`, empty stdout, no log, no lock | item 8, sage-0778 |
| T26 | sage-0778 ask 2: `activate C --plan`, with and without `--migrate`, prints the whole output line for line: the migration line, the `--migrate` need, the Jason line, then `commits: the run adds 2` and the two `row 904` and `row 903` lines, then the three install lines. `activate C1 --plan` prints `migration: none` | item 8, sage-0778 |
| T27 | the steps line: store at `3f`, `activate C --plan` prints `migration: 3f to 3h` then `migration steps: 3f to 3g, 3g to 3h`; `activate C1 --plan` (C1's fixture store.mjs has `from:` lines and no `to:` lines) prints `migration steps: cannot be read from <C1>/packages/bus/src/store.mjs` and still exits 0; `activate A --plan` (A's `to:` names `3x`, so the walk stops short of `3g`) prints the same for A | item 8, sage-0859 |
| T23 | sage-0745: the store at `3g` and stopped, the pointer on B, target C at `3h`. `activate C B --plan` exits 0 and prints `3g`, `3h`, `--migrate` and `Jason present ... decision 96 condition 2`. Afterwards the store's file set, mtime and sha256, the listing under the data root, the unit files and the wrapper are unchanged; there is no log file and no lock file. A second subtest with the bus active reads the version through the held read and prints the same lines | item 8, mos-0741 |
| T24 | `business-mismatch` (pointer for B, run for another business) and `capture-failed` (the rollback set's parent made read-only): each exits 3 with one `refused` line, no `stop` in `calls.log`; `capture-failed` follows a `started` line, so `status` shows no interrupted run afterwards. An activate ID that isn't 40 hex is a usage error, exit 4, no log | item 3, item 4.1 |
| T16 | `status` prints the pointer, five log lines, check, bus status and drift, and writes nothing | item 9 |
| T17 | in every `run` call: nothing under `backups/stack/`, `state/stack-rollback/`, `bus.post-attempt-*` or `copies/` is ever removed (the listing before the run is a subset of the one after). The store sums are checked in each test through its own fingerprint | item 10 |
| T18 | `stack-copy.test.mjs` still passes with `reachable` split out and the source guard | stack-copy.sh change |
| T19 | a `started` line with no final line: `status` shows the interrupted activation; `adopt`, `activate` and `rollback` each refuse `interrupted-run` naming it, with nothing changed | ruling 3 |
| T20 | a held `state/stack-release.lock`: `adopt`, `activate` and `rollback` refuse `run-locked` and change nothing; `status` and `--plan` still run | ruling 6 |
| T21 | a bus unit with no `MOSAIC_CLAUDE_BIN`: `adopt` and `activate` refuse `no-claude-pin` | ruling 5 |
| T22 | the store read (sage-0739), three subtests. A `bus.sqlite-wal` left after the stop refuses `store-unclean` naming the file, goes to item 6, and leaves every file under `bus/` with the same name, size, mtime and sha256. The same with a `bus.sqlite-shm`. The clean case: an activate reads the version, and afterwards `bus/` holds only `bus.sqlite`, with its mtime and sha256 the same as before the read | item 4.3, the store read |
## Mutants
One per behavior. Each is a single edit to a scratch copy of
`stack-release.sh` or `stack-copy.sh`, run against the file and expected
to fail the named test. The run script puts each mutant in its own
process group and kills the group on timeout.
| M | Mutation | Killed by |
|---|---|---|
| M1 | adopt skips the already-adopted check | T1 |
| M2 | adopt reads console flags without dropping `--checkout` | T1, T2 |
| M3 | activate installs without `--claude` | T2 |
| M4 | drift check ignores console flags | T3 |
| M5 | `instance-busy` check removed | T4 |
| M6 | `drop-in-override` check removed | T4 |
| M7 | `copy-unreachable` check removed | T4 |
| M8 | a refusal logs nothing | T4 |
| M9 | the bus stops before the Console | T5 |
| M10 | the Console starts before the bus | T5 |
| M11 | the migration runs without `--migrate` | T6 |
| M12 | the backup is taken after the migration | T7 |
| M13 | item 6 skips the store restore | T8 |
| M14 | `store-newer` treated as same | T9 |
| M15 to M19 | each gate check made to always pass | T10, one subtest each |
| M20 | the pointer is written before the gate | T10 |
| M21 | item 6 skips the wrapper restore | T10 |
| M22 | a failed rollback gate retries | T11 |
| M23 | rollback picks the newest `to` without skipping the current copy | T12 |
| M24 | `schema-differs` check removed | T13 |
| M25 | `--events` not compared | T14 |
| M26 | `--plan` writes a log line | T15 |
| M27 | restore deletes the post-attempt directory | T13, T8 (the run killed it there, not in T17) |
| M28 | `reachable` returns 0 always | T4, T18 |
| M29 | `instance-busy` ignores this business's transitional state | T4 (activating subtest) |
| M30 | the PM check accepts any live session started after step 5 | T10b (human launch subtest) |
| M31 | a `rollback --restore` gate failure goes through item 6 | T14b |
| M32 | the interrupted-run check removed | T19 |
| M33 | the run lock removed | T20 |
| M34 | the `no-claude-pin` check removed | T21 |
| M35 | `--since` given a bare local time instead of `@<epoch>` | T10b (pass subtest) |
| M36 | the PM check takes the first `launched` line instead of requiring exactly one | T10b (two-lines subtest) |
| M37 | the PM check skips the `startedAt` comparison | T10b (early-run subtest) |
| M38 | the `store-unclean` check removed | T22 (`-wal` and `-shm` subtests) |
| M39 | the store read uses `{ readOnly: true }` without immutable | T22 (clean subtest) |
| M40 | a `rollback` without `--restore` whose gate fails exits 2 without item 6 | T14c |
| M41 | `--plan` omits the `Jason present` line | T23 |
| M42 | `--plan` takes the run lock or logs `started` | T23, T20 |
| M43 | `--plan` reads the store with `{ readOnly: true }` while stopped, without immutable | T23 (file set) |
| M44 | a capture failure leaves the `started` line without a final line | T24 |
| M45 | the `copies-unrelated` refusal removed, or the `commits:` block left out | T25, T26 |
| M46 | `--plan` omits the migration line | T26, T23 |
| M47 | `--plan` omits the `migration steps:` line | T26, T23, T27 |
| M48 | `plan_steps` drops the odd-line-count check | T27 |
| M49 | `plan_steps` prints a partial walk that doesn't reach the target | T27 (the A case) |
## Candidate round 1
`r1/build.patch` against BASE `cfde62f9f` (sha256
`f5d28a751cb77c4054c598b36dd4dd996d7372bb47a0a8030c39bc8517b8a397`).
`r1/candidate-manifest.sha256` (sha256
`091693c51d137e4cd31e75ee22aef4f22c307e15abce6f91f6c518e3ff78fdd6`) lists
the 5 files in "Files". I applied `r1/build.patch` with `git apply
--index` to a fresh worktree at cfde62f9f: `sha256sum -c` passed on
every line, `git diff --cached --check` was clean, the staged names match
the manifest (5 files, +1604 -45), and the worktree was removed after.
`scripts/stack-release.sh` is mode 100755, the test file 100644. The
TOOLS.md diff is one hunk, inside "Pinned stack update".
### Checks round 1
On Sage's slot, 2026-10-18T19:25:10Z to 19:46:14Z, worktree at
cfde62f9f, TMPDIR `~/darkwing-scratch/r162/tmp`, each file run singly
with `timeout 600 node --test --test-reporter=spec`:
| Test file | Pass | Fail | Run |
|---|---|---|---|
| `packages/cli/tests/stack-release.test.mjs` | 58 | 0 | 19:43:19Z, after the steps change |
| `packages/cli/tests/stack-copy.test.mjs` | 12 | 0 | 19:45:27Z |
The first run in the slot, before the capture `mkdir` fix, was 56/57
with T24 failing (see "Notes"); the second was 57/57; the steps change
added T27. Mutants: all 47 killed against the 57-test file, then M41,
M45b, M46 and the new M47 to M49 killed against the 58-test file. The
mutant runner restores each file from a pristine copy and checks its
sha256 after every mutant. `sweep.sh` found no process of mine after.
Logs: `r1/stack-release-test.log`, `r1/stack-copy-test.log`,
`r1/mutants.summary`, `r1/mutants-steps.summary` and the runner
`r1/mutants.py`.
Not run here: the cli, bus and webui suites and the `scripts/test-*.sh`.
They are Sage's gate at landing.
## Candidate round 2 (sage-0882)
Row 125 landed as aa625565f, with its packet at df4d5702f (origin/refactor). It changed two of
this row's five paths, `docs/TOOLS.md` and `packages/cli/README.md`, so the r1 manifest no
longer described the tree this row would land on. Round 2 makes no source change.
I applied `r1/build.patch` with `git apply --index` to a fresh worktree at df4d5702f. It
applied without conflict, `git diff --cached --check` was clean, and the staged names are the
same 5 files (+1604 -45). The modes are unchanged: both scripts 100755, the test file 100644.
The worktree was removed after.
`r2/build.patch` against BASE `df4d5702f` (sha256
`336503b59a894c8ee0449dba5c36feb242a80404c9f262e96740def231fa541b`).
`r2/candidate-manifest.sha256` (sha256
`c213a801681b5dd9029520d606b3303cb30ed2fbfff8e320b0f7d79d483bd15c`).
What moved from r1:
- In `r2/build.patch`, only the two `index` lines for `docs/TOOLS.md` and
`packages/cli/README.md`, the base and result blob ids. Every hunk header, context line and
added or removed line is the same as in r1, because row 125's additions to both files sit
below this row's hunks.
- In the manifest, only the sha256 of those two files, since the files now carry row 125's
text too. `stack-release.test.mjs`, `stack-copy.sh` and `stack-release.sh` keep their r1
sha256s.
Nothing was rerun. `stack-release.test.mjs` and `stack-copy.test.mjs` read neither doc file,
and the scripts are byte-identical to r1, so the round 1 checks stand.
## Rulings (Sage, sage-0736, with sage-0738, sage-0739 and sage-0745)
1. `instance-busy`: approved as read. Refuse when another business's
`mosaic-bus@` instance is active, because the copy and wrapper switch
reach it too, or this business's is activating, deactivating or
reloading. The message names the instance and state. Test both cases.
2. PM check: approved without reading the business file, and it must
match the PM role: a live launch for B whose role is the PM, started at
or after this run's bus start. Don't name `launch.by`. sage-0738
accepted the journal reading in "The health gate" above, with an
unambiguous `--since`, exactly one line, and the one code
`pm-launch-unconfirmed` for every failure. Each refusal is tested
(T10b), and the stdout-line dependency is stated for the review. The
bus-side PM marker is a follow-up outside D1.
3. Logging: `started` and a final line in the jsonl, step detail in
`steps.log`. An unfinished `started` is an interrupted run: `status`
shows it and the next `adopt`, `activate` or `rollback` refuses
`interrupted-run`, naming the activation. Clearing it is Jason's or
Mos's. Tested.
4. A rollback whose own gate fails stops, prints both results, leaves the
stack stopped and writes `rollback-failed`. No further renames: a
second swap is a retry under another name. The post-attempt dir and
the restored bus stay for Jason. sage-0738 confirmed my reading: after
a restore, the unit files and wrapper stay as installed, and a
rollback that didn't restore goes through item 6 (T14b, T14c).
5. `no-claude-pin`: approved for both `adopt` and `activate`.
6. Run lock: approved, `flock -n` on `state/stack-release.lock`, code
`run-locked`. `status` and `--plan` take no lock.
SQLite probe: on Sage's slot (Rocko first, then me), on a fixture store
in my scratch, never the live store or a copy of it. Try `readOnly` and
`file:...?immutable=1`, and check no `-shm` or `-wal` appears and the
main file's mtime and hash hold. If neither works, the copy fallback.
Result in "The store read": readOnly left `-shm` and `-wal`, immutable
left nothing. sage-0739 approved immutable only with no `-wal` or `-shm`
present, `store-unclean` otherwise, no copy fallback, tested in T22.
sage-0745 (mos-0741): Mos takes the combined re-pin's `--plan` preview to
Jason, so `activate --plan` names the store's version and the target's
(3g to 3h there), that `--migrate` is needed, and that Jason must be
present under decision 96 condition 2. Tested in T23 with a fixture
store one version behind, which also checks that `--plan` writes
nothing. Base moved to 70504d23a, since row 124 touched
`packages/bus/src/store.mjs` and `schema.sql`.
## Notes
- T24 failed on the first run in the slot: `mkdir`'s "Permission denied"
reached stderr ahead of the `capture-failed` refusal. Both `mkdir`
calls in `capture` now send stderr to /dev/null; the refusal names the
path. The rerun passed.
- The install scripts render from the canonical checkout's templates. A
later rollback to an older copy renders that copy's units with today's
templates, and the in-run rollback restores the captured bytes. The
window plans had the same property.
- Nothing here prunes. Disk use grows by one rollback set per run and one
store copy per migrating run.
- The new TOOLS.md "Pinned stack update" section numbers the drop-in
removal as step 4, so the `drop-in-override` message says
`TOOLS.md step 4`. The brief's item 3 text still says step 3, from the
old seven-step list.