stack/agents/darkwing/work/queue-162/BUILD.md

Row 162 (D1, #1659): stack activate and rollback, health-gated

Owner: Darkwing. Reviewer: Filbert. Gate owner: Sage. Brief: docs/plans/2026-10-18_stack-release.md, section "## Stack activate and rollback, health-gated (D1)", commit cd9c071ae, blob cfc468c9df. Base: df4d5702f for round 2 (origin/refactor after row 125 landed; see "Candidate round 2"). Round 1 was on cfde62f9f (origin/refactor after row 164 landed). The draft started on cd9c071ae and moved up with git checkout --detach, through 70504d23a (row 124) to cfde62f9f. None of the 162 paths is among row 164's 21; the only base change on them is row 129's appended "Console Chat" section in docs/TOOLS.md, and the TOOLS.md patch applied cleanly. The store schema on refactor is 3h. The live copy 52f35e7a0 is at 3f, not 3g as this packet said before 2026-10-18T19:40Z: 3g came with row 141 (de352dfaf), which isn't live yet. Worktree: ~/darkwing-scratch/r162/wt.

Status: built and tested on Sage's slot (19:25:10Z to 19:46:14Z on 2026-10-18). stack-release.test.mjs 58/58 and stack-copy.test.mjs 12/12, each run singly. 47 mutants killed before the steps change, then M41, M45b, M46 and M47 to M49 killed after it. The scratch-root plan for Mos is in scratch-plan.txt next to this file.

Files

File	Change
scripts/stack-release.sh	new: adopt, activate, rollback, status
scripts/stack-copy.sh	a reachable function split out of copy_commit, and a guard so the file can be sourced for its helpers
packages/cli/tests/stack-release.test.mjs	new
docs/TOOLS.md	the "Pinned stack update" section only
packages/cli/README.md	the pinned-stack paragraph only
this packet

Nothing else changes. bus-service.sh and webui-service.sh are called as they are.

Shape of the script

scripts/stack-release.sh adopt BUSINESS
scripts/stack-release.sh activate ID BUSINESS [--pm] [--migrate] [--timeout S] [--plan]
scripts/stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan]
scripts/stack-release.sh status BUSINESS
common options: --copies DIR, --bin DIR (passed to stack-copy.sh and the install scripts)

Exit codes: 0 ok; 1 the gate failed and the rollback inside the run passed; 2 rollback-failed, the stack is left stopped; 3 a refusal, nothing changed; 4 usage. Usage errors write no log line. Every other outcome writes exactly one final line.

--pm on activate must match the pointer's pm. The brief has activate install "exactly the pointer's --pm", so the flag is a check the operator states, not a way to change it. A mismatch refuses unit-flags-drift.

The header comment states decision 96's four conditions (item 10) and points at decisions 94 and 102 for who starts the Console.

Where things come from

• Data root: load_config from scripts/common.sh, run from the canonical checkout (MOSAIC_CONFIG is honoured, as everywhere). The bus resolves its root the same way, so both see one root.
• The canonical checkout: the REPO of this script. bus-service.sh, webui-service.sh and stack-copy.sh are this checkout's, as in the window plans. webui-service.sh install already refuses outside the canonical checkout.
• Unit names: [email protected] and mosaic-webui.service. Unit directory: ${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user, the default both install scripts write to.
• claude: MOSAIC_CLAUDE_BIN= from systemctl --user show [email protected] -p Environment.
• pm: a --pm token after bus start B in the bus unit's ExecStart argv[].
• console: the argv[] tokens after serve in the Console unit's ExecStart, minus --checkout X. The install script adds --checkout itself.
• The copy's SCHEMA_VERSION and chain: text read from the copy's packages/bus/src/store.mjs. export const SCHEMA_VERSION = 'X'; must match exactly once, and the chain is the from: 'X' lines in file order. The copy's code is never imported or run for this. A copy where either read fails refuses copy-schema-unknown. The migration file names would also work, but 3b.sql is a base, not a step, so the store.mjs lines are the cleaner source.

Sharing with stack-copy.sh

copy_commit's reachability loop moves into reachable FULL, and copy_commit calls it. The dispatch at the bottom gets [ "${BASH_SOURCE[0]}" = "$0" ] || return 0, so sourcing defines the functions and runs nothing. stack-release.sh sources it inside a subshell function, so stack-copy's die, parse and globals never mix with its own:

copy_ok() ( source "$REPO/scripts/stack-copy.sh"; ID="$1"; COPIES="$COPIES_DIR"; copy_root >/dev/null && reachable "$ID" )

check and wrapper run as commands, as an operator would run them.

Pointer and log (item 1)

<dataRoot>/state/stack-active.json, written to a temp file in state/ and renamed:

{"copy":"<40 hex>","business":"mosaic-stack","pm":true,
 "claude":"/home/u/.local/bin/claude","console":["--port","7332","--business","mosaic-stack"],
 "schema":"3g","activation":"20261018T170000Z-52f35e7a0abc","at":"2026-10-18T17:00:00Z"}

schema is the copy's SCHEMA_VERSION at the time it went live. On adopt that is the adopted copy's, since a running host proves the store is at its copy's schema.

<dataRoot>/state/stack-activation-log.jsonl, appended with >>, one JSON object per line:

{"at":"...","op":"activate","from":"<copy>","to":"<copy>","backup":null,"result":"ok"}
{"at":"...","op":"activate","from":"...","to":"...","backup":null,"result":"refused","code":"unit-flags-drift"}
{"at":"...","op":"activate","from":"...","to":"...","backup":"<activation>","result":"gate-failed","code":"healthz","rollback":"ok"}
{"at":"...","op":"activate","from":"...","to":"...","backup":"<activation>","result":"rollback-failed","code":"bus-status","rollback":"failed","rollbackCode":"healthz"}

Brief item 4 says "each step is logged". Ruling 3: a run that gets past the refusals writes result: "started" before step 1 and one final line at the end. The per-step detail goes to stdout and to steps.log inside the run's rollback set. rollback reads final lines only.

A started line with no final line after it for the same activation is an interrupted run. status shows it, and every later adopt, activate or rollback refuses interrupted-run, naming the activation. The script has no verb to clear it. Clearing it is Jason's or Mos's call (ruling 3).

Every adopt, activate and rollback takes flock -n on state/stack-release.lock first and refuses run-locked when another run holds it. status and --plan take no lock (ruling 6).

Activation id: <UTC %Y%m%dT%H%M%SZ>-<first 12 of the target copy>. The rollback set directory is made with mkdir -m 0700 and no -p, so a clash fails before anything changes.

adopt (item 2)

1. The run lock, else run-locked. No interrupted-run in the log.
2. No pointer, else already-adopted.
3. stack-copy.sh check --business B passes, else check-failed with its output.
4. Read claude, pm and console from systemctl show. No MOSAIC_CLAUDE_BIN refuses no-claude-pin (ruling 5).
5. Write the pointer, then log ok.

It stops, starts and installs nothing.

activate (items 3 to 6)

Refusals before anything changes (item 3)

The checks run in this order. Each one gives exit 3, one log line with its code, and no other write:

1. run-locked, then interrupted-run. run-locked writes no log line, because the lock holder may be writing the log.
2. no-pointer, then business-mismatch: the pointer names another business. The brief names no code for it; the pointer holds one business, so a run for another one would install units the pointer doesn't describe.
3. copy-missing (copy_root fails) or copy-unreachable (reachable fails). The brief names no codes for these two, so these are mine.
4. copy-schema-unknown, from the store.mjs read above.
5. already-active: ID is the pointer's copy.
6. instance-busy (ruling 1): another business's mosaic-bus@ instance is in any state but inactive or failed, since the copy and wrapper switch reach it too; or this business's instance is activating, deactivating or reloading. The message names the instance and its state. The read is install_guard's list-units call. A literal reading (refuse any running instance) would refuse every activate, because the bus is running when activate starts.
7. drop-in-override: the install_guard scan of systemd-analyze --user unit-paths, naming the file.
8. no-claude-pin (ruling 5): the effective bus unit has no MOSAIC_CLAUDE_BIN.
9. unit-flags-drift: effective pm, claude or console differ from the pointer's, or --pm differs from the pointer's pm. Prints pointer: and unit: lines for each field that differs.

The run (item 4)

1. Capture to state/stack-rollback/<activation>/ (0700): the two unit files from the unit directory, the drop-in directories [email protected], [email protected] and mosaic-webui.service.d under that directory when present, the wrapper $BIN/mosaic with stat -c %a in WRAPPER_MODE, the systemctl show output for both units (it includes DropInPaths), and SHA256SUMS over the captured files. cp -p throughout. This is capture-rollback.sh. The started line is already written, so a failure here writes the final line refused with code capture-failed and exits 3. Nothing has stopped, and only the partial rollback set exists.
2. systemctl --user stop mosaic-webui.service, then the bus unit. Poll is-active for both to read inactive or failed, then the stopped check: no <dataRoot>/bus/writer.lock, and no /proc/*/fd link resolving under <dataRoot>/bus/. Bounded by --timeout. A timeout goes to item 6 with code stop-timeout.
3. Schema read: see "The store read" below. A -wal or -shm beside the store goes to item 6 with store-unclean.
   • The same as the copy's SCHEMA_VERSION: no backup.
   • In the chain and older, no --migrate: migration-needs-flag, item 6.
   • Older with --migrate: back up, then node <copy>/packages/bus/src/migrate-schema.mjs <dataRoot>. A non-zero exit goes to item 6 with code migrate-failed and the store restored.
   • Newer, or not in the chain: store-newer, item 6.
   • No bus.sqlite at all (a fresh root): no backup, the new host creates the store.
4. bus-service.sh install --copy ID --copies DIR [--pm] --claude PATH, then webui-service.sh install --copy ID --copies DIR <console flags>, then stack-copy.sh wrapper ID --copies DIR --bin DIR. Any non-zero exit goes to item 6 with code install-bus, install-console or wrapper.
5. Start the bus, then the Console.

From step 2 on, trap on INT, TERM and HUP sends the run to item 6, once. A Ctrl-C mid-window leaves the old stack started, not a stopped one. The one exception is after a rollback --restore has swapped the store; see ruling 4 under rollback.

Most of the script's functions run as f || handler. Bash turns set -e off inside such a call, so every step in them that can fail carries its own || return 1 or sits in an && chain.

The backup

<dataRoot>/backups/stack/<activation>/, as backup.sh does it: umask 077, mkdir -m 700 for the directory and bus/, cp -p of bus.sqlite (only that file: a -wal or -shm was already refused as store-unclean), SOURCE.sha256 taken before the copy and checked against SHA256SUMS after it, MODES.txt, LISTING.txt, DATA_ROOT, STAMP (UTC ISO with milliseconds, the same form as events.at), SCHEMA (the version read in step 3) and ROLLBACK_SET. backup.sh, restore.sh and stopped-check.sh here mean Sage's committed window scripts in agents/sage/work/window-plan/repin-de352dfaf/. stack-release.sh carries its own copies of their steps and doesn't call them. The stopped check runs again after the copy. Any failure goes to item 6 before the migration runs.

The store read

node -e with node:sqlite: new DatabaseSync(`${pathToFileURL(file).href}?immutable=1`, { readOnly: true }), then SELECT value FROM meta WHERE key='schema_version', then close.

The probe settled the open mode. It ran on Sage's slot (sage-0738), 16:02:53Z to 16:03:08Z, node v26.8.1, on a WAL-mode fixture store in ~/darkwing-scratch/r162/probe/store/, never the live store or a copy of it. The fixture was rebuilt before each trial and closed cleanly, so only bus.sqlite was present at the start of each one.

Open	Read	Files after	Main file mtime, sha256
{ readOnly: true }, store stopped	version and count right	bus.sqlite-shm and bus.sqlite-wal created and left behind	unchanged
file:...?immutable=1 with readOnly, store stopped	version and count right	none created	unchanged
{ readOnly: true } while a WAL connection holds the store	sees the holder's row still in the WAL	the holder's -wal and -shm, nothing new	not compared

A plain read-only open of a stopped store writes two files next to it, so it fails Sage's condition, and immutable is the mode. Immutable tells SQLite the file can't change and makes it skip the WAL entirely. That is sound only when there is no WAL to skip. So the rule (Sage approved it in sage-0739):

• Before every stopped-store read, bus.sqlite-wal and bus.sqlite-shm must both be absent. A clean close checkpoints and removes them. One left behind means the bus did not close the store cleanly, and an immutable read could return a stale version or count. The run refuses store-unclean, naming the file, and goes to item 6. stack-release.sh never opens, copies or moves that store. Whatever happens to it next is Jason's or Mos's call. There is no copy fallback.
• With both absent, the read uses immutable.
• An unreadable version goes to item 6 with code schema-unreadable.

The --restore count in item 7 step 5 is taken before the stop, from a store the running bus holds. There the -wal and -shm belong to the bus, the bus unit must be active, and the read is { readOnly: true }, which the probe shows adds no file. If the bus unit isn't active, the count is a stopped-store read under the rule above.

The no-write test below catches a read that creates a file: it compares the file set and the sums under bus/ before and after.

The health gate (item 5)

Polled every second until all pass or --timeout (default 60) runs out. The failing check named at the end is the first one still failing:

Check	Code
systemctl --user is-active for both units	units-inactive
<copy>/scripts/mosaic bus status --json exits 0, host.business is B, host.live is true, host.pid is the bus unit's MainPID, boot.ok is true, boot.business is B, boot.at is at or after the moment step 5 began	bus-status
GET http://127.0.0.1:<port>/healthz returns 200, port from the --port flag in console, else 7332 (the webui-service.sh default)	healthz
stack-copy.sh check --business B exits 0	copy-check
with pm: exactly one launched line in the bus unit's journal since step 5, and its run is live in <copy>/scripts/mosaic launches list --json, for B, with startedAt at or after the moment step 5 began	pm-launch-unconfirmed

Ruling 2 (sage-0736, refined in sage-0738): the check must match the PM role without reading the business file (Jason's org file) and without naming launch.by. The session entries carry instance, but nothing in them or in launches/<business>.jsonl marks a launch as the PM's. The bus start --pm path is the only code that knows. It prints launched <instance>: run <run>, <harness> <model> to stdout (packages/cli/src/cli.mjs:503, the only place that prints such a line), and under the unit that stdout is the journal.

So the gate reads journalctl --user -u [email protected] --since @<epoch seconds of step 5> -o cat. The @ form is unambiguous. This host's journalctl reads a bare time as local CDT, and in Q14 an empty window passed for a clean start that way. Of the lines it returns, it keeps those starting with launched . It passes only when:

• there is exactly one such line;
• it matches ^launched (\S+): run (\S+), ;
• that run is live in launches list --json for B, with startedAt at or after step 5.

Zero lines, two or more, a line that doesn't match, or a run that isn't live all give the one code pm-launch-unconfirmed, and the message says which. A human mosaic launch in the same window writes no line to the unit's journal, so it can't pass for the PM.

The review should know that this check depends on the wording of that stdout line. A change to it in cli.mjs makes the gate fail closed, not pass. A test pins the dependency: T10b's unparseable-line subtest feeds a line that still starts with launched but has a later word changed (job for run) and expects the refusal.

Follow-up, not in D1 (Sage files it if wanted): a bus-side PM marker, for example a pm: true field on the launch event the --pm path writes, would let the gate drop the journal read and the stdout-format dependency.

MainPID equals host.pid because scripts/mosaic ends in exec node ...cli.mjs. Every bus start that loads the business calls recordBootOk (packages/cli/src/cli.mjs:494), which writes boot.at as the current time, so a boot.at from before step 5 means the status describes an earlier start.

The healthz fetch is node -e fetch(...) with a 5 s abort, as webui-service.sh status does it.

On a pass: rename the pointer into place, log ok, exit 0.

Rollback inside the run (item 6)

1. Stop the Console, then the bus, and wait as in step 2.
2. Put back the two unit files and the drop-in directories from the rollback set (temp file and rename each), and the wrapper bytes and mode. Check them against SHA256SUMS. daemon-reload.
3. If step 3 of this run took a backup, restore it as restore.sh does (a rollback --restore backup is never restored here): check DATA_ROOT, the stopped check and sha256sum -c; rename bus to bus.post-attempt-<UTC stamp>; mkdir -m 700 bus; copy in; check the sums and that the file set matches exactly. Nothing is deleted.
4. Start the bus, then the Console. Run the gate against the pointer's copy.
5. Log gate-failed (or the step's code) with rollback: ok, exit 1. If the rollback's gate fails: print both results, stop both units, log rollback-failed with both codes, exit 2. No retry.

rollback (item 7)

1. no-pointer.
2. Target: scan the log newest first for result: "ok" lines with op adopt, activate or rollback, and take the first to that isn't the pointer's copy. None found refuses no-previous. This is release.sh's rule, and it shares that rule's property: two rollbacks in a row go back and forth between two copies.
3. The item 3 refusals apply to the target (copy, schema, busy, drop-ins, flag drift).
4. schema-differs: the pointer's schema is not the target's SCHEMA_VERSION and there's no --restore. Decided from the pointer before anything stops.
5. With --restore ACTIVATION: backups/stack/ACTIVATION/ must exist with SHA256SUMS, a DATA_ROOT equal to this root, and a SCHEMA equal to the target's SCHEMA_VERSION, else restore-mismatch. Print the post-attempt path the run will keep, and the count of events with at after the backup's STAMP, from a read-only open of the store before anything stops. Count above 0 and --events N not equal to it refuses post-backup-events. No change.
6. Steps 4.1 to 4.5 with the target. In step 3, the stopped store's version must equal the pointer's schema, else store-differs, item 6. With --restore, the count is read again after the stop. A different count goes to item 6 with post-backup-events, since an event landed between the check and the stop. Then the restore runs in place of the migration.
7. The gate, as for activate. A pass renames the pointer (with the target's schema) and logs ok with backup: ACTIVATION.

Ruling 4 (sage-0736, confirmed in sage-0738) has two cases:

• A rollback --restore that fails after the restore has moved the old bus/ aside: the run does not go through item 6. That covers the gate and also an install, start or signal failure after the swap; the script treats every one the same and logs rollbackCode: "not-attempted". A restore that fails before the move goes through item 6 like any other step, since the store hasn't changed. Putting the newer store back would be a second swap, which is a retry under another name. The run prints both results, stops both units, logs rollback-failed and exits 2. The post-attempt directory and the restored bus stay where they are for Jason. The unit files and wrapper also stay as the run installed them, since they match the restored store. Nothing is swapped back. Tested in T14b.
• A rollback without --restore whose gate fails: the store was never swapped, so the run goes through item 6 as activate does. It puts back the unit files and wrapper from its rollback set, restarts the pointer's copy and runs the gate on it. Exit 1 with rollback: ok, or exit 2 with rollback-failed if that gate fails too. Tested in T14c.

--plan (item 8)

It reads the pointer, the log, the target copy's store.mjs and the store's schema version. It writes nothing: no lock, no log line, no file under the data root.

Mos takes the --plan output to Jason before the combined re-pin (mos-0741, sage-0745), so a migration must be stated plainly. For a store at 3f and refactor at 3h, activate --plan prints:

store schema: 3f (read from the stopped store)
target schema: 3h
migration: 3f to 3h, after a backup of the stopped store
migration steps: 3f to 3g, 3g to 3h
needs: --migrate (not given; the run would roll back with migration-needs-flag)
needs: Jason present for this run (decision 96 condition 2)

I added the migration steps: line in the slot on 2026-10-18, before the scratch-root run (see "The scratch-root plan for Mos"). It walks the target store.mjs's from: and to: pairs from the store's version, as text, the same way copy_schema reads the version, so nothing in the copy runs. When the lines don't pair up, or the walk doesn't reach the target's version, it prints migration steps: cannot be read from <file> and the rest of the plan is unchanged. The run itself still calls migrate-schema.mjs, which applies the copy's own chain; the steps line only names it.

The store version comes from the store itself. While the bus unit is active the read is { readOnly: true }, which adds no file next to a store the bus holds (the probe). With the bus stopped and no -wal or -shm, it is the immutable read. With no store file, a -wal or -shm present, or a failed read, it falls back to the pointer's schema and says which case applied. When the store and the pointer disagree, a note: line says so; the run itself goes by the stopped store.

Other cases print migration: none (same version), a store-newer warning (not in the target's chain) or, for rollback, restore: ... needs --restore ACTIVATION with the activation the log's backup field names for the pointer's copy. Then the exact bus-service.sh, webui-service.sh and stack-copy.sh wrapper command lines. Exit 0. The refusals it can see before a run (no-pointer, business-mismatch, copy, already-active, no-previous) still print, with exit 3 and no log line.

After the migration or restore lines comes the commits: block (sage-0778): commits: the run adds N when the pointer's copy is an ancestor of the target, commits: the run takes away N when the target is an ancestor of the pointer's copy (a rollback), then the git log --oneline --no-merges lines, newest first, at most 200. When neither copy is an ancestor of the other, --plan refuses copies-unrelated. My 16:53Z message to Sage said "refuse if FROM isn't an ancestor of TO". The code is wider than that on purpose, so a rollback still lists what it takes away. T25 and T26 test it.

The scratch-root plan for Mos (sage-0837)

Mos asked (mos-0765) for the combined re-pin's --plan output with the 3g to 3h migration and rows 148, 141, 143, 128, 124 and 149. Before any live adopt there is no pointer, so I run it against a scratch root under ~/darkwing-scratch/r162/: a data root, a config naming it, a hand-written pointer and a copies directory, all scratch. Nothing under ~/.mosaic-dev, the live copies directory or the units is written.

• The live copy id: systemctl --user show [email protected] -p WorkingDirectory at 2026-10-18T19:32:16Z gave .../copies/52f35e7a0933e9fa845f59860a5de1709f9e5756. It reads the unit's property and writes nothing. sage-0859 also allowed a read-only systemctl --user is-active on the bus unit name; the run never made that call, because with no store file plan_store doesn't reach it.
• The pointer's schema is that copy's SCHEMA_VERSION, read with git show 52f35e7a0:packages/bus/src/store.mjs: 3f. I had expected 3g. That is why the steps line exists: migration: 3f to 3h alone doesn't show the 3g to 3h step Mos asked for, and hand-writing 3g would have been a false pointer.
• The target copy, from scratch-plan-steps.log: (cd ~/darkwing-scratch/r162/wt && scripts/stack-copy.sh copy 606fafd43154ded993a503933681198e737987cd --copies ~/darkwing-scratch/r162/plan/copies --allow-worktree) at 19:45:54Z, copied by 19:45:56Z. Removed at 19:45:56Z with chmod -R u+w and rm -rf of that one copy directory; the copies dir was left with 0 entries. 606fafd43 was origin/refactor at the run; it moved past cfde62f9f after sage-0859.
• A header I put above the script's output names the scratch root and says the pointer is hand-written. The script prints no such header itself.
• The scratch root has no bus/bus.sqlite, so the store schema line reads from the pointer; there is no store file. --plan runs only check_target and plan, not the unit checks, so the only other reads are git in the checkout and the copies directory. The data root held only state/stack-active.json and an empty bus/ after the run.

The result, scratch-plan.txt (134 lines, exit 0): migration: 3f to 3h, migration steps: 3f to 3g, 3g to 3h, both needs: lines, then commits: the run adds 112. All six rows are in the list (grep hits per row: 148 7, 141 5, 143 8, 128 9, 124 9, 149 18). The range also carries row 164, the row 165 brief and everything else since 52f35e7a0.

Mos then narrowed the window target to 8e7ea89a1, the six rows only (mos-0810 D1-D2, sage-0871). I reran the same plan in a new slot with r1/plan-8e7.sh, a copy of plan.sh with the target, the scratch root (plan-8e7) and the output file changed. It ran from this worktree while it still matched the r1 manifest. The result is scratch-plan-8e7ea89a1.txt (114 lines, exit 0). The migration and migration steps lines are unchanged. The plan lists 92 commits, the same count as git rev-list --count 52f35e7a0..8e7ea89a1. Each of the six landing commits appears once. Row 164's 008746b6e and 850484a7d are not in it. Of the 92, only the six touch code paths. The step log is scratch-plan-8e7ea89a1-steps.log. Its "plan: written to" line names the wrong file, and live-reads.md explains why. Both 606fafd43 files stay as they were, since this review uses them.

Sage forwards the output to Mos as a scratch-root plan. The real --plan against the live pointer runs again in Mos's window before any adopt.

status (item 9)

The pointer, the last five log lines, any interrupted run, stack-copy.sh check output and result, <pointer copy>/scripts/mosaic bus status, whether the unit flags match the pointer, and the drift: whether the copy is an ancestor of refactor and git rev-list --count COPY..refactor. It only reads.

Tests (packages/cli/tests/stack-release.test.mjs)

Fixture, built once per file:

• A temp canonical repo, made the way stack-copy.test.mjs setup does it: git ls-files -co of scripts and packages copied in, git init -b refactor, the light package.json. I copy the setup code rather than import it, because importing a test file runs its tests too. The base commit is F.
• On refactor, empty commits A, B, C1 and C (subjects row 901: A to row 904: C). A side branch from A holds S and merges into refactor as M, with --no-ff. A branch named unreached from A holds U. refactor and origin/refactor both point at M.
• Read-only stub copies under a temp copies/<id>/ for F, A, B, C1, C, S and U. M has no copy, so it is the copy-missing case. Each copy has scripts/mosaic, packages/webui/src/cli.mjs (present only, never run), packages/bus/src/store.mjs and packages/bus/src/migrate-schema.mjs. The store file in A, B, C1, S and U says 3g with a from: '3f' step. C says 3h with 3f and 3g steps. F has no SCHEMA_VERSION, which gives copy-schema-unknown.
• The stub migrate records the backups/stack/ entries it saw, and whether each has SHA256SUMS, in migrate.json, then sets the store to 3h with node:sqlite. The migrate-fails fault sets 3h-partial and exits 1.
• The stand-ins systemctl, journalctl and systemd-analyze are node scripts with a #!<node> shebang in a sbin/ directory that comes first on PATH. All three exist, so the real ones never run. A package.json with "type":"commonjs" in the fixture's base directory covers them, the holder and the copies' scripts/mosaic.

Per test: a temp HOME, XDG_CONFIG_HOME, XDG_DATA_HOME, data root, MOSAIC_CONFIG (a valid config naming the temp data root), bin/ with the Q81 wrapper, mosaic-old and a fake claude, and a stand-in state directory with a faults/ directory. A fault is a file named for the copy it hits, such as healthz-<id>, so one fault can fail the new copy and leave the rollback target alone.

seed builds the stack the way an operator would: a real WAL-mode bus.sqlite with a meta row and three events, then the real bus-service.sh install, webui-service.sh install and stack-copy.sh wrapper against the fixture copy, then a start of both units. The pointer comes from a real adopt, or is written by hand with no log (T23, T25), or is left out.

The stand-in systemctl:

• show UNIT -p ... reads the installed unit file and its drop-ins ([email protected] and <unit>.d), fills in %i and %h, and answers LoadState, WorkingDirectory, ExecStart={ path=... ; argv[]=... ; }, Environment, MainPID and DropInPaths. The check-<id> fault moves the Console's WorkingDirectory, so stack-copy.sh check fails.
• list-units, is-active, is-enabled, start, stop and daemon-reload keep unit state in units.json (written through a temp file and a rename) and append every call to calls.log, so the order is testable.
• start of the bus runs a holder process. The holder opens bus.sqlite in WAL mode and keeps it open, writes bus/writer.lock and a host.json with the business, its pid and a boot status, and closes the store and removes the lock on SIGTERM. That gives the real -wal and -shm files of a held store, and a clean close on stop. The holder starts through bash -c with every fd above 2 closed, so it never inherits the script's fd 9 lock. SD_CLOCK_OFFSET backdates the seeded host and its PM run by two minutes, so they predate any run's start.
• start of the bus with --pm in the unit's ExecStart appends the launched line, as cli.mjs:503 writes it, to the stand-in journal, and adds the run to launches.json. A pm-<id> fault picks another case: no line, two lines, a changed word, a run missing from the list, an early startedAt, a human launch with no line, a run for another business, a dead run.
• stop sends SIGTERM and waits for the holder to exit. The one-shot stop-leaves-wal and stop-leaves-shm faults leave an empty -wal or -shm beside the store, as an unclean close would. Each bus start and stop records the name, size, mtime and sha256 of every file under bus/ in snap.jsonl, so T22 can compare the store before and after the script's read.
• start-dies-<id> sets the bus to failed.

The stub scripts/mosaic prints bus status --json from host.json, with live recomputed from /proc, and launches list --json from launches.json. The status-<id> and launches-<id> faults make it fail. The Console is an HTTP server inside the test process on a free port. It answers 200 when the stand-in says the Console is active, and 500 when a healthz-<id> fault names the Console unit's current copy. That is why the test runs the script with async spawn: spawnSync would block the server.

systemd-analyze --user unit-paths prints the temp unit directory.

The stand-in journalctl logs the --since argument it got and prints the stand-in journal lines for the unit at or after it. It accepts --since only as @<epoch seconds> or YYYY-MM-DD HH:MM:SS UTC, and exits 1 on any other form, so a bare local time gives no lines.

Every run call also checks T17: the listing of backups/stack/, state/stack-rollback/, bus.post-attempt-* and copies/ taken before the run is a subset of the one after.

Clean-up kills only the holder pids the stand-in logged, after checking each one's /proc/<pid>/cmdline names this file's holder.

Gate failure cases run with --timeout 3.

#	Case	Brief
T1	adopt writes the pointer (claude and console from show) and one ok line; a second adopt refuses already-adopted; a failing check refuses check-failed with no pointer	item 2
T2	activate installs exactly the pointer's --pm, --claude and Console flags (unit file text)	items 3, 4
T3	a hand-edited unit (each of pm, claude, a console flag) refuses unit-flags-drift, prints the difference, changes nothing	item 3
T4	each item 3 refusal: units, wrapper, pointer and store unchanged by sums, exactly one new log line with the code, no start or stop in calls.log; instance-busy has two subtests (another business's instance active, this one's activating) and both name the instance and state	item 3, ruling 1
T5	no-schema activate: calls.log shows stop Console, stop bus, daemon-reloads, start bus, start Console in order; pointer names C1; no backups/stack/	item 4
T6	B to C without --migrate: migration-needs-flag, old stack started, pointer unchanged	item 4.3
T7	B to C with --migrate: backup exists before the migrate stub ran (the stub records whether it saw the backup's sums), pointer schema 3h	item 4.3
T8	B to C with --migrate and a failing migration: post-attempt dir exists, restored files match the backup's sums, old stack started	item 6
T9	store-newer (store at a version outside C's chain)	item 4.3
T10	each gate check failing alone (five subtests): rollback, pointer unchanged, unit files and wrapper back byte for byte, log names the check	items 5, 6
T10b	the PM check, one subtest per case, each refusing pm-launch-unconfirmed with the case named: no launched line; two lines; a line that starts with launched but has a later word changed (launched mosaic-pm: job R, ...); one line whose run isn't in launches list; one line whose run has startedAt before step 5; a live human mosaic launch session after step 5 with no journal line; a run for another business; a run that isn't live. A last subtest passes with one well-formed line and a live run, and checks the --since argument the stand-in received starts with @	item 5, ruling 2
T11	rollback gate fails too: rollback-failed, exit 2, both units stopped, no start after the failing one	item 6
T12	rollback finds the previous copy from the log	item 7
T13	rollback across a migration refuses schema-differs; with --restore it restores and goes live on the old copy	item 7
T14	--restore with post-backup events and no or a wrong --events N refuses post-backup-events; the right N proceeds	item 7
T14b	a rollback --restore whose gate fails: rollback-failed, exit 2, both units stopped, the post-attempt dir and the restored bus in place, no rename after the restore	item 7, ruling 4
T14c	a rollback without --restore whose gate fails: item 6 runs, the unit files and wrapper are back byte for byte from the run's rollback set, the pointer's copy is started and gated, exit 1 with rollback: ok; the store's file set and sums are unchanged	item 7, ruling 4
T15	--plan on activate and rollback prints the migration or restore and the flags; units, pointer, log and store unchanged. After an activate to C with --migrate, rollback --plan prints, line for line, the restore with the activation and commits: the run takes away 2	item 8
T25	sage-0778 ask 1: pointer on B, activate S --plan (S is on a side branch, so neither copy is an ancestor of the other) exits 3 with exactly stack-release: refusing copies-unrelated: neither <B> nor <S> is an ancestor of the other, so the plan cannot list the commits, empty stdout, no log, no lock	item 8, sage-0778
T26	sage-0778 ask 2: activate C --plan, with and without --migrate, prints the whole output line for line: the migration line, the --migrate need, the Jason line, then commits: the run adds 2 and the two row 904 and row 903 lines, then the three install lines. activate C1 --plan prints migration: none	item 8, sage-0778
T27	the steps line: store at 3f, activate C --plan prints migration: 3f to 3h then migration steps: 3f to 3g, 3g to 3h; activate C1 --plan (C1's fixture store.mjs has from: lines and no to: lines) prints migration steps: cannot be read from <C1>/packages/bus/src/store.mjs and still exits 0; activate A --plan (A's to: names 3x, so the walk stops short of 3g) prints the same for A	item 8, sage-0859
T23	sage-0745: the store at 3g and stopped, the pointer on B, target C at 3h. activate C B --plan exits 0 and prints 3g, 3h, --migrate and Jason present ... decision 96 condition 2. Afterwards the store's file set, mtime and sha256, the listing under the data root, the unit files and the wrapper are unchanged; there is no log file and no lock file. A second subtest with the bus active reads the version through the held read and prints the same lines	item 8, mos-0741
T24	business-mismatch (pointer for B, run for another business) and capture-failed (the rollback set's parent made read-only): each exits 3 with one refused line, no stop in calls.log; capture-failed follows a started line, so status shows no interrupted run afterwards. An activate ID that isn't 40 hex is a usage error, exit 4, no log	item 3, item 4.1
T16	status prints the pointer, five log lines, check, bus status and drift, and writes nothing	item 9
T17	in every run call: nothing under backups/stack/, state/stack-rollback/, bus.post-attempt-* or copies/ is ever removed (the listing before the run is a subset of the one after). The store sums are checked in each test through its own fingerprint	item 10
T18	stack-copy.test.mjs still passes with reachable split out and the source guard	stack-copy.sh change
T19	a started line with no final line: status shows the interrupted activation; adopt, activate and rollback each refuse interrupted-run naming it, with nothing changed	ruling 3
T20	a held state/stack-release.lock: adopt, activate and rollback refuse run-locked and change nothing; status and --plan still run	ruling 6
T21	a bus unit with no MOSAIC_CLAUDE_BIN: adopt and activate refuse no-claude-pin	ruling 5
T22	the store read (sage-0739), three subtests. A bus.sqlite-wal left after the stop refuses store-unclean naming the file, goes to item 6, and leaves every file under bus/ with the same name, size, mtime and sha256. The same with a bus.sqlite-shm. The clean case: an activate reads the version, and afterwards bus/ holds only bus.sqlite, with its mtime and sha256 the same as before the read	item 4.3, the store read

Mutants

One per behavior. Each is a single edit to a scratch copy of stack-release.sh or stack-copy.sh, run against the file and expected to fail the named test. The run script puts each mutant in its own process group and kills the group on timeout.

M	Mutation	Killed by
M1	adopt skips the already-adopted check	T1
M2	adopt reads console flags without dropping --checkout	T1, T2
M3	activate installs without --claude	T2
M4	drift check ignores console flags	T3
M5	instance-busy check removed	T4
M6	drop-in-override check removed	T4
M7	copy-unreachable check removed	T4
M8	a refusal logs nothing	T4
M9	the bus stops before the Console	T5
M10	the Console starts before the bus	T5
M11	the migration runs without --migrate	T6
M12	the backup is taken after the migration	T7
M13	item 6 skips the store restore	T8
M14	store-newer treated as same	T9
M15 to M19	each gate check made to always pass	T10, one subtest each
M20	the pointer is written before the gate	T10
M21	item 6 skips the wrapper restore	T10
M22	a failed rollback gate retries	T11
M23	rollback picks the newest to without skipping the current copy	T12
M24	schema-differs check removed	T13
M25	--events not compared	T14
M26	--plan writes a log line	T15
M27	restore deletes the post-attempt directory	T13, T8 (the run killed it there, not in T17)
M28	reachable returns 0 always	T4, T18
M29	instance-busy ignores this business's transitional state	T4 (activating subtest)
M30	the PM check accepts any live session started after step 5	T10b (human launch subtest)
M31	a rollback --restore gate failure goes through item 6	T14b
M32	the interrupted-run check removed	T19
M33	the run lock removed	T20
M34	the no-claude-pin check removed	T21
M35	--since given a bare local time instead of @<epoch>	T10b (pass subtest)
M36	the PM check takes the first launched line instead of requiring exactly one	T10b (two-lines subtest)
M37	the PM check skips the startedAt comparison	T10b (early-run subtest)
M38	the store-unclean check removed	T22 (-wal and -shm subtests)
M39	the store read uses { readOnly: true } without immutable	T22 (clean subtest)
M40	a rollback without --restore whose gate fails exits 2 without item 6	T14c
M41	--plan omits the Jason present line	T23
M42	--plan takes the run lock or logs started	T23, T20
M43	--plan reads the store with { readOnly: true } while stopped, without immutable	T23 (file set)
M44	a capture failure leaves the started line without a final line	T24
M45	the copies-unrelated refusal removed, or the commits: block left out	T25, T26
M46	--plan omits the migration line	T26, T23
M47	--plan omits the migration steps: line	T26, T23, T27
M48	plan_steps drops the odd-line-count check	T27
M49	plan_steps prints a partial walk that doesn't reach the target	T27 (the A case)

Candidate round 1

r1/build.patch against BASE cfde62f9f (sha256 f5d28a751cb77c4054c598b36dd4dd996d7372bb47a0a8030c39bc8517b8a397). r1/candidate-manifest.sha256 (sha256 091693c51d137e4cd31e75ee22aef4f22c307e15abce6f91f6c518e3ff78fdd6) lists the 5 files in "Files". I applied r1/build.patch with git apply --index to a fresh worktree at cfde62f9f: sha256sum -c passed on every line, git diff --cached --check was clean, the staged names match the manifest (5 files, +1604 -45), and the worktree was removed after. scripts/stack-release.sh is mode 100755, the test file 100644. The TOOLS.md diff is one hunk, inside "Pinned stack update".

Checks round 1

On Sage's slot, 2026-10-18T19:25:10Z to 19:46:14Z, worktree at cfde62f9f, TMPDIR ~/darkwing-scratch/r162/tmp, each file run singly with timeout 600 node --test --test-reporter=spec:

Test file	Pass	Fail	Run
packages/cli/tests/stack-release.test.mjs	58	0	19:43:19Z, after the steps change
packages/cli/tests/stack-copy.test.mjs	12	0	19:45:27Z

The first run in the slot, before the capture mkdir fix, was 56/57 with T24 failing (see "Notes"); the second was 57/57; the steps change added T27. Mutants: all 47 killed against the 57-test file, then M41, M45b, M46 and the new M47 to M49 killed against the 58-test file. The mutant runner restores each file from a pristine copy and checks its sha256 after every mutant. sweep.sh found no process of mine after.

Logs: r1/stack-release-test.log, r1/stack-copy-test.log, r1/mutants.summary, r1/mutants-steps.summary and the runner r1/mutants.py.

Not run here: the cli, bus and webui suites and the scripts/test-*.sh. They are Sage's gate at landing.

Candidate round 2 (sage-0882)

Row 125 landed as aa625565f, with its packet at df4d5702f (origin/refactor). It changed two of this row's five paths, docs/TOOLS.md and packages/cli/README.md, so the r1 manifest no longer described the tree this row would land on. Round 2 makes no source change.

I applied r1/build.patch with git apply --index to a fresh worktree at df4d5702f. It applied without conflict, git diff --cached --check was clean, and the staged names are the same 5 files (+1604 -45). The modes are unchanged: both scripts 100755, the test file 100644. The worktree was removed after.

r2/build.patch against BASE df4d5702f (sha256 336503b59a894c8ee0449dba5c36feb242a80404c9f262e96740def231fa541b). r2/candidate-manifest.sha256 (sha256 c213a801681b5dd9029520d606b3303cb30ed2fbfff8e320b0f7d79d483bd15c).

What moved from r1:

• In r2/build.patch, only the two index lines for docs/TOOLS.md and packages/cli/README.md, the base and result blob ids. Every hunk header, context line and added or removed line is the same as in r1, because row 125's additions to both files sit below this row's hunks.
• In the manifest, only the sha256 of those two files, since the files now carry row 125's text too. stack-release.test.mjs, stack-copy.sh and stack-release.sh keep their r1 sha256s.

Nothing was rerun. stack-release.test.mjs and stack-copy.test.mjs read neither doc file, and the scripts are byte-identical to r1, so the round 1 checks stand.

Rulings (Sage, sage-0736, with sage-0738, sage-0739 and sage-0745)

1. instance-busy: approved as read. Refuse when another business's mosaic-bus@ instance is active, because the copy and wrapper switch reach it too, or this business's is activating, deactivating or reloading. The message names the instance and state. Test both cases.
2. PM check: approved without reading the business file, and it must match the PM role: a live launch for B whose role is the PM, started at or after this run's bus start. Don't name launch.by. sage-0738 accepted the journal reading in "The health gate" above, with an unambiguous --since, exactly one line, and the one code pm-launch-unconfirmed for every failure. Each refusal is tested (T10b), and the stdout-line dependency is stated for the review. The bus-side PM marker is a follow-up outside D1.
3. Logging: started and a final line in the jsonl, step detail in steps.log. An unfinished started is an interrupted run: status shows it and the next adopt, activate or rollback refuses interrupted-run, naming the activation. Clearing it is Jason's or Mos's. Tested.
4. A rollback whose own gate fails stops, prints both results, leaves the stack stopped and writes rollback-failed. No further renames: a second swap is a retry under another name. The post-attempt dir and the restored bus stay for Jason. sage-0738 confirmed my reading: after a restore, the unit files and wrapper stay as installed, and a rollback that didn't restore goes through item 6 (T14b, T14c).
5. no-claude-pin: approved for both adopt and activate.
6. Run lock: approved, flock -n on state/stack-release.lock, code run-locked. status and --plan take no lock.

SQLite probe: on Sage's slot (Rocko first, then me), on a fixture store in my scratch, never the live store or a copy of it. Try readOnly and file:...?immutable=1, and check no -shm or -wal appears and the main file's mtime and hash hold. If neither works, the copy fallback. Result in "The store read": readOnly left -shm and -wal, immutable left nothing. sage-0739 approved immutable only with no -wal or -shm present, store-unclean otherwise, no copy fallback, tested in T22.

sage-0745 (mos-0741): Mos takes the combined re-pin's --plan preview to Jason, so activate --plan names the store's version and the target's (3g to 3h there), that --migrate is needed, and that Jason must be present under decision 96 condition 2. Tested in T23 with a fixture store one version behind, which also checks that --plan writes nothing. Base moved to 70504d23a, since row 124 touched packages/bus/src/store.mjs and schema.sql.

Notes

• T24 failed on the first run in the slot: mkdir's "Permission denied" reached stderr ahead of the capture-failed refusal. Both mkdir calls in capture now send stderr to /dev/null; the refusal names the path. The rerun passed.
• The install scripts render from the canonical checkout's templates. A later rollback to an older copy renders that copy's units with today's templates, and the in-run rollback restores the captured bytes. The window plans had the same property.
• Nothing here prunes. Disk use grows by one rollback set per run and one store copy per migrating run.
• The new TOOLS.md "Pinned stack update" section numbers the drop-in removal as step 4, so the drop-in-override message says TOOLS.md step 4. The brief's item 3 text still says step 3, from the old seven-step list.