stack/docs/plans/2026-10-18_stack-release.md

Stack release, rollback and rehearsal (D1, D2), 2026-10-18

Status: written by Sage, lead, for decision 96's D track (Q97). Decision 96 orders S7, then M1-M2, then D1-D2 beside I1, then I2-I6. I1 is written (docs/plans/2026-10-18_containerized-install-design.md), so D1 and D2 come before row 157 (I2). It follows docs/plans/BRIEF-TEMPLATE.md. Two rows. Line numbers are at 7dfda5182.

Background

Every live change since row 85 goes out as a new copy and a re-pin (docs/plans/2026-10-12_one-copy-stack.md). The bus, the Console and the human CLI all run from one read-only copy:

• scripts/stack-copy.sh copy (:34-60) makes the copy;
• bus-service.sh install --copy (:132-170) and webui-service.sh install --copy (:242-281) write the units;
• stack-copy.sh wrapper (:61-74) switches the CLI;
• stack-copy.sh check (:87-98) confirms all three point at the same copy.

What happens between those steps is manual. docs/TOOLS.md:343-381 ("Pinned stack update") lists seven operator steps. Rollback means repeating steps 2-6 with the earlier id, and nothing records which copy is live or which copy came before it. The bus has no health endpoint. The Console's /healthz returns a fixed 200 (packages/webui/src/serve.mjs:355). A schema change has no backup in product code: packages/bus/README.md:678 says only "after a consistent backup". The store migrates forward only (packages/bus/src/store.mjs:21-50), and there is no down-migration.

The real windows filled these gaps with plan-local scripts. The de352dfaf window (agents/sage/work/window-plan/repin-de352dfaf/) had these:

• capture-rollback.sh: the old wrapper and unit files, with sums;
• stopped-check.sh: no writer.lock holder and no bus process;
• backup.sh: an owner-only copy of the stopped store, with sums;
• rehearse.sh: the migration run on disposable copies of the backup;
• restore.sh: the backup put back, with the post-attempt store moved aside and nothing deleted;
• verify-live.sh and verify-new.sh: the effective paths.

scripts/release.sh already has the shape decision 96 names for the worker image:

• a health gate before the switch (release.sh:41-57);
• an atomic pointer at state/active.json, written to a temp file and renamed (:74-80);
• an append-only state/activation-log.jsonl;
• rollback to the last different tag through the same gate (:133-143).

D1 gives the stack that shape. D2 makes the rehearsal a product step.

Stack activate and rollback, health-gated (D1)

Problem

A re-pin is a hand-run sequence of stops, installs, a wrapper switch and starts, checked by eye. The window plans carry the backup, restore and path checks as one-off scripts. The live stack has no pointer and no log, so "what is live and what was before it" lives in a plan file. A bad start has no gate that puts the old copy back.

Owner and reviewer

• Owner: darkwing (Sage, 2026-10-18).
• Reviewer: filbert.

Files owned

• scripts/stack-release.sh (new): verbs adopt, activate, rollback and status
• scripts/stack-copy.sh: only to share the copy-id and copies-directory helpers, if the owner factors them out; copy, wrapper and check keep their behavior
• packages/cli/tests/stack-release.test.mjs (new)
• docs/TOOLS.md, the "Pinned stack update" section, and packages/cli/README.md, the paragraph on the pinned stack
• the packet agents/<owner>/work/queue-<row>/

What ships

1. A pointer and a log, the same shape as release.sh:

   • <dataRoot>/state/stack-active.json holds { copy, business, pm, claude, console, schema, activation, at }. It is written to a temp file and renamed. claude is the bus unit's --claude pin, and console is the Console unit's flag list (--port, --business, --board, --send and the public-host, auth-header and allow-user set). webui-service.sh saves its flags nowhere else, so the pointer is their record (Darkwing's survey, 2026-10-18).
   • <dataRoot>/state/stack-activation-log.jsonl is append-only. Each line has at, op, from, to, backup, result and, on a refusal or a failed gate, code. op is one of adopt, activate or rollback.
   • Both are files under state/, which AGENTS.md's data map already gives to the release pointer and its logs.

2. adopt BUSINESS records the stack that is running now as the first pointer, and changes nothing. It refuses unless stack-copy.sh check passes and the bus unit's copy is the Console's and the wrapper's. The live stack is already on one copy (52f35e7a0), so no mixed-setup path is needed. A pointer that already exists refuses already-adopted. adopt reads claude and console from the effective units with systemctl --user show -p ExecStart -p Environment, the way bus-service.sh status reads --pm (:200-209).

3. activate ID BUSINESS [--pm] moves the stack from the pointer's copy to copy ID. It refuses, with a code, a log line and no change, when any of these holds:

   • there's no pointer, no-pointer;
   • the copy is missing or isn't reachable from refactor and origin/refactor, using the rule stack-copy.sh already enforces;
   • any instance is busy, instance-busy, using the check bus-service.sh install_guard uses (:100-130);
   • a bus drop-in overrides ExecStart or WorkingDirectory, drop-in-override. It is named, and the operator removes it as in TOOLS.md step 3;
   • the target is already the pointer's copy, already-active;
   • the effective units' --pm, --claude or Console flags differ from the pointer's, unit-flags-drift. The difference is printed. A hand edit between runs is never carried forward or reverted without anyone seeing it; the operator settles it first.

4. The order inside activate is as follows. Each step is logged, and a failure at any step goes to item 6.

   1. Capture the rollback set under <dataRoot>/state/stack-rollback/<activation>/, mode 0700: the installed bus and Console unit files, their drop-in directories, and the wrapper's bytes and mode, with SHA256SUMS. This is capture-rollback.sh as product code.
   2. Stop the Console, then the bus. Wait for both to be inactive and for the stopped check to pass: no writer.lock holder and no bus process for the data root (stopped-check.sh).
   3. Read the stopped store's schema version without writing. The owner picks the read, for example a read-only SQLite open of the version the store records, and the packet names it. If the version is the new copy's SCHEMA_VERSION, there's no backup. If it's older and the run has no --migrate, refuse migration-needs-flag through item 6. With --migrate, back up the stopped store to <dataRoot>/backups/stack/<activation>/ (backup.sh: 0700, sums, a source and copy comparison, the stopped check again after the copy), then run the new copy's migrate-schema.mjs. If it's newer than the new copy's, or not in the copy's chain, refuse store-newer. The stack is stopped by then, so the refusal goes through item 6 and starts the old stack again.
   4. Install the bus unit with bus-service.sh install --copy ID, with exactly the --pm and --claude pin the pointer recorded. Then install the Console with webui-service.sh install --copy ID with exactly the pointer's console flags. Then switch the wrapper with stack-copy.sh wrapper ID.
   5. Start the bus, then the Console.

5. The health gate runs after the start and passes only when all of these hold within a timeout (default 60 s, --timeout):

   • systemctl --user is-active reports both units active;
   • the copy's scripts/mosaic bus status --json exits 0 and shows a live host for BUSINESS whose pid is the bus unit's MainPID, with the boot status ok (packages/cli/src/cli.mjs:499-514). The store opens only at its own schema, so a running host means the schema matches;
   • the Console's /healthz answers 200 on its configured port;
   • stack-copy.sh check passes;
   • with --pm, scripts/mosaic launches list shows the PM.

   On a pass, the pointer is renamed into place and the log says ok. The pointer changes only on a pass.

6. A failed gate, or a failure after step 2, rolls back inside the same run. It stops the Console and the bus, then puts back the rollback set's unit files, drop-ins and wrapper bytes and runs daemon-reload. If step 3 migrated, it restores the backup the way restore.sh does: the current bus/ is renamed to bus.post-attempt-<stamp>, nothing is deleted, and the restore is checked against the backup's sums. Then it starts the old stack and runs the same gate against the pointer's copy. The log records gate-failed with the failing check and the outcome of the rollback. If the rollback's own gate fails, the script stops, prints both results and leaves the stack stopped, with a rollback-failed line. It never retries.

7. rollback BUSINESS moves to the copy the log names as the one before the pointer's, the same way release.sh rollback finds it. It runs steps 4.1-4.5 and the gate with that copy as the target. If the store's schema is newer than that copy's, it refuses schema-differs unless --restore ACTIVATION names the backup taken when the store was migrated. With --restore, it prints the post-attempt path it keeps and how many bus events were written after the backup's stamp, then restores. If that count is above zero, it refuses post-backup-events unless --events N repeats the printed count.

8. activate ID BUSINESS --plan and rollback BUSINESS --plan print the target, whether a migration or a restore is expected and the flags that would be installed. The store schema comes from a read-only read of the store the running bus holds, or of the stopped store when the bus is down. With no store file it falls back to the pointer's schema and says so (sage-0745, mos-0741; this replaced the first draft's "pointer only, no store read"). The target side is the copy's SCHEMA_VERSION. They change nothing and write no log line, so the window knows beforehand whether Jason must be present.

9. status BUSINESS prints the pointer, the last five log lines, the live units' copies, the result of stack-copy.sh check, bus status and the drift from refactor. It only reads.

10. Who runs it. Decision 96 covers the script stopping and starting the live units under four conditions (Mos, mos-0716), and the header comment states all four:

• every run of adopt, activate or rollback happens on Mos's go, and Jason or Mos runs it;
• a run that migrates the store, an activate --migrate, happens with Jason present;
• a rollback --restore with events written after the backup waits for Jason, which is why it needs --events N;
• the script never prunes backups, rollback sets, post-attempt directories or copies.

Decisions 94 and 102 still decide who starts the Console. webui-service.sh keeps having no start or stop verb (:50). The release script is the one path that stops and starts. status and --plan are open to every seat.

Tests run with a stand-in systemctl and systemd-analyze on PATH (install_guard also calls systemd-analyze --user unit-paths), temporary XDG_CONFIG_HOME and XDG_DATA_HOME, a temporary data root, and fake copies whose scripts/mosaic and Console are small stubs, as packages/cli/tests/stack-copy.test.mjs and packages/webui/tests/service.test.mjs do. The cases:

• adopt writes the pointer and one log line only when the check passes, with claude and console read from the stand-in's show output. A second adopt refuses.
• activate installs exactly the pointer's --pm, --claude and Console flags, and a stand-in unit whose flags differ refuses unit-flags-drift with no change.
• Each refusal in item 3 leaves the units, the wrapper, the pointer and the store unchanged, and it writes one log line with its code.
• An activate with no schema change: the stand-in sees the stops, the installs and the starts in that order. The pointer names the new copy, and no backup directory exists.
• An activate with a schema change and no --migrate refuses migration-needs-flag and starts the old stack. With --migrate it makes the backup before the migration. A fixture migration that fails restores the backup and starts the old stack. The post-attempt directory exists, and the restored files match the backup's sums.
• Each check in item 5, failing on its own, rolls back. The pointer is unchanged, the old unit files and wrapper bytes come back byte-for-byte, and the log names that check.
• A failing rollback gate stops with rollback-failed and starts nothing more.
• rollback finds the previous copy from the log, refuses schema-differs without --restore, and restores with it. With post-backup events, --restore without a matching --events N refuses post-backup-events.
• --plan reports the expected migration or restore and the flags, and leaves the units, pointer, log and store unchanged.
• The store is never opened for writing outside migrate-schema.mjs and the restore, and nothing under backups/, state/stack-rollback/, bus.post-attempt-* or the copies directory is removed.

Mutants: one per behavior above, with the killing test in the packet.

Out of scope

• Any live run of adopt, activate or rollback. The first adopt and the first activate happen in a Mos window, with Mos's go.
• The Discord unit (docs/plans/2026-10-14_discord-pinned-copy.md). It joins in a follow-up once D1 lands.
• Pruning copies, rollback sets or backups.
• A down-migration, any change to the store's schema chain, and any change to migrate-schema.mjs.
• A health endpoint in the bus. The gate uses bus status.
• Any change to the human proof (packages/bus/src/human.mjs) or to packages/queue.
• CI, which is D4.

Gate

Filbert approves on the row's issue, naming the candidate manifest. The cli, bus and webui node suites and every scripts/test-*.sh pass on Sage's gate rerun. The landing changes nothing live. Sage sends Mos the commit and stack-release.sh status output from a scratch root. The first live adopt waits for Mos's go.

Stack rehearsal on a scratch root (D2)

Problem

Decision 96's D2 is staging: trying a candidate before live takes it. Today the trial is rehearse.sh, a plan-local script that checks only the migration on copies of the backup. Nothing starts the new copy's bus host and Console against real data before the live window, so a failure that shows only at start is found with the live stack stopped.

Owner and reviewer

• Owner: sage (row 163, briefed 2026-10-20).
• Reviewer: filbert.

Files owned

• scripts/stack-release.sh: the rehearse verb
• packages/cli/tests/stack-release.test.mjs: the rehearsal cases
• docs/TOOLS.md, the same section as D1
• the packet agents/<owner>/work/queue-<row>/

What ships

1. rehearse ID BUSINESS --backup ACTIVATION|PATH builds a scratch stack under a new 0700 directory, <dataRoot>/rehearsals/<stamp>/:

   • a data root holding copies of the backup's store files;
   • the business file, roles and organization file copied from the live config directory, read only;
   • a scratch config that names the scratch data root.

   The live store, live config and the backup are never written. The backup's sums are checked before and after.

2. On the scratch root it runs the de352dfaf rehearsal checks with the new copy: the new code refuses the unmigrated store at a normal open, migrate-schema.mjs succeeds, a second run is a no-op, and the old copy refuses the migrated store. Those checks apply only when the schema differs.

3. It starts the new copy's bus host in the foreground on the scratch root with launches disabled, so no role session starts and no model is called. It starts the new copy's Console on a free loopback port with no send. Then it runs D1's gate checks against both, except the PM check. It stops both and prints one PASS or FAIL line per check, then REHEARSAL_PASS or REHEARSAL_FAIL.

4. "Launches disabled" is a host flag the owner adds if the host lacks one. Its default is off, and the live units never pass it. If the host can't start without launching, the row stops and Sage takes the design back to Mos. No rehearsal calls a model.

5. The rehearsal directory stays for reading. Nothing prunes it, and it is 0700 because it holds bus data.

6. Who runs it. rehearse reads a backup of the live store, so only Jason or Mos run it, as the window's pre-step (the old P-steps). No development seat runs it against live data. Tests use fixture stores.

Tests:

• The rehearsal passes on a fixture store and copy, and the scratch root holds the expected files.
• The live store, the live config files and the backup are byte-identical before and after.
• A migration that fails, a new copy whose host doesn't start, and a Console that doesn't answer each give REHEARSAL_FAIL naming the check.
• No role session starts: the fixture's launcher stub records no start.
• A scratch port that is in use refuses and picks no other port that could be the live Console's.

Mutants: one per behavior above.

Out of scope

• A standing staging stack with its own units, Console, business and model turns. That needs a second system config beside ~/.config/mosaic-dev/config.json (AGENTS.md invariant 2) and model spend, so it's Jason's call. It is not briefed here.
• Everything D1's out-of-scope list names.

Gate

As D1. D2 lands after D1, because both edit stack-release.sh and its test file.

D2 also lands after the Part B window (P19 385) has run. The Part B window file names the stack-release.sh blob, and any change to that blob means Mos checks the window again (mos-0958, mos-0959). The build and review can go ahead before then.