stack/docs/plans/2026-10-18_stack-release.md

# Stack release, rollback and rehearsal (D1, D2), 2026-10-18

Status: written by Sage, lead, for decision 96's D track (Q97). Decision 96
orders S7, then M1-M2, then D1-D2 beside I1, then I2-I6. I1 is written
(`docs/plans/2026-10-18_containerized-install-design.md`), so D1 and D2
come before row 157 (I2). It follows `docs/plans/BRIEF-TEMPLATE.md`. Two
rows. Line numbers are at 7dfda5182.

## Background

Every live change since row 85 goes out as a new copy and a re-pin
(`docs/plans/2026-10-12_one-copy-stack.md`). The bus, the Console and the
human CLI all run from one read-only copy:
- `scripts/stack-copy.sh copy` (:34-60) makes the copy;
- `bus-service.sh install --copy` (:132-170) and
  `webui-service.sh install --copy` (:242-281) write the units;
- `stack-copy.sh wrapper` (:61-74) switches the CLI;
- `stack-copy.sh check` (:87-98) confirms all three point at the same copy.

What happens between those steps is manual. `docs/TOOLS.md:343-381`
("Pinned stack update") lists seven operator steps. Rollback means
repeating steps 2-6 with the earlier id, and nothing records which copy
is live or which copy came before it. The bus has no health endpoint. The
Console's `/healthz` returns a fixed 200 (`packages/webui/src/serve.mjs:355`).
A schema change has no backup in product code: `packages/bus/README.md:678`
says only "after a consistent backup". The store migrates forward only
(`packages/bus/src/store.mjs:21-50`), and there is no down-migration.

The real windows filled these gaps with plan-local scripts. The de352dfaf
window (`agents/sage/work/window-plan/repin-de352dfaf/`) had these:
- `capture-rollback.sh`: the old wrapper and unit files, with sums;
- `stopped-check.sh`: no writer.lock holder and no bus process;
- `backup.sh`: an owner-only copy of the stopped store, with sums;
- `rehearse.sh`: the migration run on disposable copies of the backup;
- `restore.sh`: the backup put back, with the post-attempt store moved
  aside and nothing deleted;
- `verify-live.sh` and `verify-new.sh`: the effective paths.

`scripts/release.sh` already has the shape decision 96 names for the
worker image:
- a health gate before the switch (`release.sh:41-57`);
- an atomic pointer at `state/active.json`, written to a temp file and
  renamed (:74-80);
- an append-only `state/activation-log.jsonl`;
- `rollback` to the last different tag through the same gate (:133-143).

D1 gives the stack that shape. D2 makes the rehearsal a product step.

## Stack activate and rollback, health-gated (D1)

### Problem

A re-pin is a hand-run sequence of stops, installs, a wrapper switch and
starts, checked by eye. The window plans carry the backup, restore and
path checks as one-off scripts. The live stack has no pointer and no log,
so "what is live and what was before it" lives in a plan file. A bad
start has no gate that puts the old copy back.

### Owner and reviewer

- Owner: darkwing (Sage, 2026-10-18).
- Reviewer: filbert.

### Files owned

- `scripts/stack-release.sh` (new): verbs `adopt`, `activate`,
  `rollback` and `status`
- `scripts/stack-copy.sh`: only to share the copy-id and copies-directory
  helpers, if the owner factors them out; `copy`, `wrapper` and `check`
  keep their behavior
- `packages/cli/tests/stack-release.test.mjs` (new)
- `docs/TOOLS.md`, the "Pinned stack update" section, and
  `packages/cli/README.md`, the paragraph on the pinned stack
- the packet `agents/<owner>/work/queue-<row>/`

### What ships

1. A pointer and a log, the same shape as `release.sh`:
   - `<dataRoot>/state/stack-active.json` holds `{ copy, business, pm,
     claude, console, schema, activation, at }`. It is written to a temp
     file and renamed. `claude` is the bus unit's `--claude` pin, and
     `console` is the Console unit's flag list (`--port`, `--business`,
     `--board`, `--send` and the public-host, auth-header and allow-user
     set). `webui-service.sh` saves its flags nowhere else, so the
     pointer is their record (Darkwing's survey, 2026-10-18).
   - `<dataRoot>/state/stack-activation-log.jsonl` is append-only. Each
     line has `at`, `op`, `from`, `to`, `backup`, `result` and, on a
     refusal or a failed gate, `code`. `op` is one of `adopt`,
     `activate` or `rollback`.
   - Both are files under `state/`, which AGENTS.md's data map already
     gives to the release pointer and its logs.
2. `adopt BUSINESS` records the stack that is running now as the first
   pointer, and changes nothing. It refuses unless `stack-copy.sh check`
   passes and the bus unit's copy is the Console's and the wrapper's.
   The live stack is already on one copy (52f35e7a0), so no mixed-setup
   path is needed. A pointer that already exists refuses
   `already-adopted`. `adopt` reads `claude` and `console` from the
   effective units with `systemctl --user show -p ExecStart -p
   Environment`, the way `bus-service.sh status` reads `--pm`
   (:200-209).
3. `activate ID BUSINESS [--pm]` moves the stack from the pointer's copy
   to copy `ID`. It refuses, with a code, a log line and no change, when
   any of these holds:
   - there's no pointer, `no-pointer`;
   - the copy is missing or isn't reachable from `refactor` and
     `origin/refactor`, using the rule `stack-copy.sh` already enforces;
   - any instance is busy, `instance-busy`, using the check
     `bus-service.sh install_guard` uses (:100-130);
   - a bus drop-in overrides `ExecStart` or `WorkingDirectory`,
     `drop-in-override`. It is named, and the operator removes it as in
     TOOLS.md step 3;
   - the target is already the pointer's copy, `already-active`;
   - the effective units' `--pm`, `--claude` or Console flags differ from
     the pointer's, `unit-flags-drift`. The difference is printed. A hand
     edit between runs is never carried forward or reverted without
     anyone seeing it; the operator settles it first.
4. The order inside `activate` is as follows. Each step is logged, and a
   failure at any step goes to item 6.
   1. Capture the rollback set under
      `<dataRoot>/state/stack-rollback/<activation>/`, mode 0700: the
      installed bus and Console unit files, their drop-in directories,
      and the wrapper's bytes and mode, with SHA256SUMS. This is
      `capture-rollback.sh` as product code.
   2. Stop the Console, then the bus. Wait for both to be inactive and
      for the stopped check to pass: no `writer.lock` holder and no bus
      process for the data root (`stopped-check.sh`).
   3. Read the stopped store's schema version without writing. The
      owner picks the read, for example a read-only SQLite open of the
      version the store records, and the packet names it. If the version
      is the new copy's `SCHEMA_VERSION`, there's no backup. If it's
      older and the run has no `--migrate`, refuse `migration-needs-flag`
      through item 6. With `--migrate`, back up the stopped store to
      `<dataRoot>/backups/stack/<activation>/` (`backup.sh`: 0700, sums,
      a source and copy comparison, the stopped check again after the
      copy), then run the new copy's `migrate-schema.mjs`. If it's newer
      than the new copy's, or not in the copy's chain, refuse
      `store-newer`. The stack is stopped by then, so the refusal goes
      through item 6 and starts the old stack again.
   4. Install the bus unit with `bus-service.sh install --copy ID`,
      with exactly the `--pm` and `--claude` pin the pointer recorded.
      Then install the Console with `webui-service.sh install --copy ID`
      with exactly the pointer's `console` flags. Then switch the wrapper
      with
      `stack-copy.sh wrapper ID`.
   5. Start the bus, then the Console.
5. The health gate runs after the start and passes only when all of
   these hold within a timeout (default 60 s, `--timeout`):
   - `systemctl --user is-active` reports both units active;
   - the copy's `scripts/mosaic bus status --json` exits 0 and shows a
     live host for BUSINESS whose pid is the bus unit's MainPID, with the
     boot status ok (`packages/cli/src/cli.mjs:499-514`). The store opens
     only at its own schema, so a running host means the schema matches;
   - the Console's `/healthz` answers 200 on its configured port;
   - `stack-copy.sh check` passes;
   - with `--pm`, `scripts/mosaic launches list` shows the PM.

   On a pass, the pointer is renamed into place and the log says `ok`.
   The pointer changes only on a pass.
6. A failed gate, or a failure after step 2, rolls back inside the same
   run. It stops the Console and the bus, then puts back the rollback
   set's unit files, drop-ins and wrapper bytes and runs `daemon-reload`.
   If step 3 migrated, it restores the backup the way `restore.sh` does:
   the current `bus/` is renamed to `bus.post-attempt-<stamp>`, nothing
   is deleted, and the restore is checked against the backup's sums.
   Then it starts the old stack and runs the same gate against the
   pointer's copy. The log records `gate-failed` with the failing check
   and the outcome of the rollback. If the rollback's own gate fails,
   the script stops, prints both results and leaves the stack stopped,
   with a `rollback-failed` line. It never retries.
7. `rollback BUSINESS` moves to the copy the log names as the one
   before the pointer's, the same way `release.sh rollback` finds it. It
   runs steps 4.1-4.5 and the gate with that copy as the target. If the
   store's schema is newer than that copy's, it refuses
   `schema-differs` unless `--restore ACTIVATION` names the backup taken
   when the store was migrated. With `--restore`, it prints the
   post-attempt path it keeps and how many bus events were written after
   the backup's stamp, then restores. If that count is above zero, it
   refuses `post-backup-events` unless `--events N` repeats the printed
   count.
8. `activate ID BUSINESS --plan` and `rollback BUSINESS --plan` print
   the target, whether a migration or a restore is expected and the
   flags that would be installed. The store schema comes from a
   read-only read of the store the running bus holds, or of the stopped
   store when the bus is down. With no store file it falls back to the
   pointer's `schema` and says so (sage-0745, mos-0741; this replaced
   the first draft's "pointer only, no store read"). The target side is the
   copy's `SCHEMA_VERSION`. They change nothing and write no log line,
   so the window knows beforehand whether Jason must be present.
9. `status BUSINESS` prints the pointer, the last five log lines, the
   live units' copies, the result of `stack-copy.sh check`, `bus status`
   and the drift from `refactor`. It only reads.
10. Who runs it. Decision 96 covers the script stopping and starting
   the live units under four conditions (Mos, mos-0716), and the
   header comment states all four:
   - every run of `adopt`, `activate` or `rollback` happens on Mos's
     go, and Jason or Mos runs it;
   - a run that migrates the store, an `activate --migrate`, happens
     with Jason present;
   - a `rollback --restore` with events written after the backup waits
     for Jason, which is why it needs `--events N`;
   - the script never prunes backups, rollback sets, post-attempt
     directories or copies.

   Decisions 94 and 102 still decide who starts the Console.
   `webui-service.sh` keeps having no start or stop verb (:50). The
   release script is the one path that stops and starts. `status` and
   `--plan` are open to every seat.

Tests run with a stand-in `systemctl` and `systemd-analyze` on `PATH`
(`install_guard` also calls `systemd-analyze --user unit-paths`), temporary
`XDG_CONFIG_HOME` and `XDG_DATA_HOME`, a temporary data root, and fake
copies whose `scripts/mosaic` and Console are small stubs, as
`packages/cli/tests/stack-copy.test.mjs` and
`packages/webui/tests/service.test.mjs` do. The cases:

- `adopt` writes the pointer and one log line only when the check passes,
  with `claude` and `console` read from the stand-in's `show` output.
  A second `adopt` refuses.
- `activate` installs exactly the pointer's `--pm`, `--claude` and
  Console flags, and a stand-in unit whose flags differ refuses
  `unit-flags-drift` with no change.
- Each refusal in item 3 leaves the units, the wrapper, the pointer and
  the store unchanged, and it writes one log line with its code.
- An `activate` with no schema change: the stand-in sees the stops, the
  installs and the starts in that order. The pointer names the new
  copy, and no backup directory exists.
- An `activate` with a schema change and no `--migrate` refuses
  `migration-needs-flag` and starts the old stack. With `--migrate` it
  makes the backup before the migration. A fixture migration that fails restores the backup and
  starts the old stack. The post-attempt directory exists, and the
  restored files match the backup's sums.
- Each check in item 5, failing on its own, rolls back. The pointer is
  unchanged, the old unit files and wrapper bytes come back
  byte-for-byte, and the log names that check.
- A failing rollback gate stops with `rollback-failed` and starts
  nothing more.
- `rollback` finds the previous copy from the log, refuses
  `schema-differs` without `--restore`, and restores with it. With
  post-backup events, `--restore` without a matching `--events N`
  refuses `post-backup-events`.
- `--plan` reports the expected migration or restore and the flags, and
  leaves the units, pointer, log and store unchanged.
- The store is never opened for writing outside `migrate-schema.mjs` and
  the restore, and nothing under `backups/`, `state/stack-rollback/`,
  `bus.post-attempt-*` or the copies directory is removed.

Mutants: one per behavior above, with the killing test in the packet.

### Out of scope

- Any live run of `adopt`, `activate` or `rollback`. The first `adopt` and
  the first `activate` happen in a Mos window, with Mos's go.
- The Discord unit (`docs/plans/2026-10-14_discord-pinned-copy.md`). It
  joins in a follow-up once D1 lands.
- Pruning copies, rollback sets or backups.
- A down-migration, any change to the store's schema chain, and any change
  to `migrate-schema.mjs`.
- A health endpoint in the bus. The gate uses `bus status`.
- Any change to the human proof (`packages/bus/src/human.mjs`) or to
  `packages/queue`.
- CI, which is D4.

### Gate

Filbert approves on the row's issue, naming the candidate manifest. The
cli, bus and webui node suites and every `scripts/test-*.sh` pass on
Sage's gate rerun. The landing changes nothing live. Sage sends Mos the
commit and `stack-release.sh status` output from a scratch root. The
first live `adopt` waits for Mos's go.

## Stack rehearsal on a scratch root (D2)

### Problem

Decision 96's D2 is staging: trying a candidate before live takes it.
Today the trial is `rehearse.sh`, a plan-local script that checks only the
migration on copies of the backup. Nothing starts the new copy's bus host
and Console against real data before the live window, so a failure that
shows only at start is found with the live stack stopped.

### Owner and reviewer

- Owner: sage (row 163, briefed 2026-10-20).
- Reviewer: filbert.

### Files owned

- `scripts/stack-release.sh`: the `rehearse` verb
- `packages/cli/tests/stack-release.test.mjs`: the rehearsal cases
- `docs/TOOLS.md`, the same section as D1
- the packet `agents/<owner>/work/queue-<row>/`

### What ships

1. `rehearse ID BUSINESS --backup ACTIVATION|PATH` builds a scratch stack
   under a new 0700 directory, `<dataRoot>/rehearsals/<stamp>/`:
   - a data root holding copies of the backup's store files;
   - the business file, roles and organization file copied from the live
     config directory, read only;
   - a scratch config that names the scratch data root.

   The live store, live config and the backup are never written. The
   backup's sums are checked before and after.
2. On the scratch root it runs the de352dfaf rehearsal checks with the
   new copy: the new code refuses the unmigrated store at a normal open,
   `migrate-schema.mjs` succeeds, a second run is a no-op, and the old
   copy refuses the migrated store. Those checks apply only when the
   schema differs.
3. It starts the new copy's bus host in the foreground on the scratch
   root with launches disabled, so no role session starts and no model
   is called. It starts the new copy's Console on a free loopback port
   with no send. Then it runs D1's gate checks against both, except the
   PM check. It stops both and prints one PASS or FAIL line per check,
   then `REHEARSAL_PASS` or `REHEARSAL_FAIL`.
4. "Launches disabled" is a host flag the owner adds if the host lacks
   one. Its default is off, and the live units never pass it. If the
   host can't start without launching, the row stops and Sage takes the
   design back to Mos. No rehearsal calls a model.
5. The rehearsal directory stays for reading. Nothing prunes it, and it
   is 0700 because it holds bus data.
6. Who runs it. `rehearse` reads a backup of the live store, so only
   Jason or Mos run it, as the window's pre-step (the old P-steps). No
   development seat runs it against live data. Tests use fixture stores.

Tests:
- The rehearsal passes on a fixture store and copy, and the scratch root
  holds the expected files.
- The live store, the live config files and the backup are byte-identical
  before and after.
- A migration that fails, a new copy whose host doesn't start, and a
  Console that doesn't answer each give `REHEARSAL_FAIL` naming the check.
- No role session starts: the fixture's launcher stub records no start.
- A scratch port that is in use refuses and picks no other port that
  could be the live Console's.

Mutants: one per behavior above.

### Out of scope

- A standing staging stack with its own units, Console, business and
  model turns. That needs a second system config beside
  `~/.config/mosaic-dev/config.json` (AGENTS.md invariant 2) and model
  spend, so it's Jason's call. It is not briefed here.
- Everything D1's out-of-scope list names.

### Gate

As D1. D2 lands after D1, because both edit `stack-release.sh` and its
test file.

D2 also lands after the Part B window (P19 385) has run. The Part B window
file names the `stack-release.sh` blob, and any change to that blob means
Mos checks the window again (mos-0958, mos-0959). The build and review can
go ahead before then.