plans: D1 and D2 brief, health-gated stack activate and rollback and the scratch-root rehearsal (#1659, #1660)
Decision 96 D track. D1 turns the de352dfaf window scripts and the
release.sh pointer-and-log shape into stack-release.sh adopt, activate,
rollback and status. D2 adds rehearse on a scratch root with launches
disabled and no model calls. A standing staging stack is left to Jason.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
jason.woltjeandClaude Opus 5.5 committed
1 parent
e270538675
commit
0df05f2c5c
1 file changed
+305
@@ -0,0 +1,305 @@
|
||||
# Stack release, rollback and rehearsal (D1, D2), 2026-10-18
|
||||
|
||||
Status: written by Sage, lead, for decision 96's D track (Q97). Decision 96
|
||||
orders S7, then M1-M2, then D1-D2 beside I1, then I2-I6. I1 is written
|
||||
(`docs/plans/2026-10-18_containerized-install-design.md`), so D1 and D2
|
||||
come before row 157 (I2). It follows `docs/plans/BRIEF-TEMPLATE.md`. Two
|
||||
rows. Line numbers are at 7dfda5182.
|
||||
|
||||
## Background
|
||||
|
||||
Every live change since row 85 goes out as a new copy and a re-pin
|
||||
(`docs/plans/2026-10-12_one-copy-stack.md`). The bus, the Console and the
|
||||
human CLI all run from one read-only copy:
|
||||
- `scripts/stack-copy.sh copy` (:34-60) makes the copy;
|
||||
- `bus-service.sh install --copy` (:132-170) and
|
||||
`webui-service.sh install --copy` (:242-281) write the units;
|
||||
- `stack-copy.sh wrapper` (:61-74) switches the CLI;
|
||||
- `stack-copy.sh check` (:87-98) confirms all three point at the same copy.
|
||||
|
||||
What happens between those steps is manual. `docs/TOOLS.md:343-381`
|
||||
("Pinned stack update") lists seven operator steps. Rollback means
|
||||
repeating steps 2-6 with the earlier id, and nothing records which copy
|
||||
is live or which copy came before it. The bus has no health endpoint. The
|
||||
Console's `/healthz` returns a fixed 200 (`packages/webui/src/serve.mjs:355`).
|
||||
A schema change has no backup in product code: `packages/bus/README.md:678`
|
||||
says only "after a consistent backup". The store migrates forward only
|
||||
(`packages/bus/src/store.mjs:21-50`), and there is no down-migration.
|
||||
|
||||
The real windows filled these gaps with plan-local scripts. The de352dfaf
|
||||
window (`agents/sage/work/window-plan/repin-de352dfaf/`) had these:
|
||||
- `capture-rollback.sh`: the old wrapper and unit files, with sums;
|
||||
- `stopped-check.sh`: no writer.lock holder and no bus process;
|
||||
- `backup.sh`: an owner-only copy of the stopped store, with sums;
|
||||
- `rehearse.sh`: the migration run on disposable copies of the backup;
|
||||
- `restore.sh`: the backup put back, with the post-attempt store moved
|
||||
aside and nothing deleted;
|
||||
- `verify-live.sh` and `verify-new.sh`: the effective paths.
|
||||
|
||||
`scripts/release.sh` already has the shape decision 96 names for the
|
||||
worker image:
|
||||
- a health gate before the switch (`release.sh:41-57`);
|
||||
- an atomic pointer at `state/active.json`, written to a temp file and
|
||||
renamed (:74-80);
|
||||
- an append-only `state/activation-log.jsonl`;
|
||||
- `rollback` to the last different tag through the same gate (:133-143).
|
||||
|
||||
D1 gives the stack that shape. D2 makes the rehearsal a product step.
|
||||
|
||||
## Stack activate and rollback, health-gated (D1)
|
||||
|
||||
### Problem
|
||||
|
||||
A re-pin is a hand-run sequence of stops, installs, a wrapper switch and
|
||||
starts, checked by eye. The window plans carry the backup, restore and
|
||||
path checks as one-off scripts. The live stack has no pointer and no log,
|
||||
so "what is live and what was before it" lives in a plan file. A bad
|
||||
start has no gate that puts the old copy back.
|
||||
|
||||
### Owner and reviewer
|
||||
|
||||
- Owner: set when the row is briefed.
|
||||
- Reviewer: filbert.
|
||||
|
||||
### Files owned
|
||||
|
||||
- `scripts/stack-release.sh` (new): verbs `adopt`, `activate`,
|
||||
`rollback` and `status`
|
||||
- `scripts/stack-copy.sh`: only to share the copy-id and copies-directory
|
||||
helpers, if the owner factors them out; `copy`, `wrapper` and `check`
|
||||
keep their behavior
|
||||
- `packages/cli/tests/stack-release.test.mjs` (new)
|
||||
- `docs/TOOLS.md`, the "Pinned stack update" section, and
|
||||
`packages/cli/README.md`, the paragraph on the pinned stack
|
||||
- the packet `agents/<owner>/work/queue-<row>/`
|
||||
|
||||
### What ships
|
||||
|
||||
1. A pointer and a log, the same shape as `release.sh`:
|
||||
- `<dataRoot>/state/stack-active.json` holds `{ copy, business, pm,
|
||||
schema, activation, at }`. It is written to a temp file and renamed.
|
||||
- `<dataRoot>/state/stack-activation-log.jsonl` is append-only. Each
|
||||
line has `at`, `op`, `from`, `to`, `backup`, `result` and, on a
|
||||
refusal or a failed gate, `code`. `op` is one of `adopt`,
|
||||
`activate` or `rollback`.
|
||||
- Both are files under `state/`, which AGENTS.md's data map already
|
||||
gives to the release pointer and its logs.
|
||||
2. `adopt BUSINESS` records the stack that is running now as the first
|
||||
pointer, and changes nothing. It refuses unless `stack-copy.sh check`
|
||||
passes and the bus unit's copy is the Console's and the wrapper's.
|
||||
The live stack is already on one copy (52f35e7a0), so no mixed-setup
|
||||
path is needed. A pointer that already exists refuses
|
||||
`already-adopted`.
|
||||
3. `activate ID BUSINESS [--pm]` moves the stack from the pointer's copy
|
||||
to copy `ID`. It refuses, with a code, a log line and no change, when
|
||||
any of these holds:
|
||||
- there's no pointer, `no-pointer`;
|
||||
- the copy is missing or isn't reachable from `refactor` and
|
||||
`origin/refactor`, using the rule `stack-copy.sh` already enforces;
|
||||
- any instance is busy, `instance-busy`, using the check
|
||||
`bus-service.sh install_guard` uses (:100-130);
|
||||
- a bus drop-in overrides `ExecStart` or `WorkingDirectory`,
|
||||
`drop-in-override`. It is named, and the operator removes it as in
|
||||
TOOLS.md step 3;
|
||||
- the target is already the pointer's copy, `already-active`.
|
||||
4. The order inside `activate` is as follows. Each step is logged, and a
|
||||
failure at any step goes to item 6.
|
||||
1. Capture the rollback set under
|
||||
`<dataRoot>/state/stack-rollback/<activation>/`, mode 0700: the
|
||||
installed bus and Console unit files, their drop-in directories,
|
||||
and the wrapper's bytes and mode, with SHA256SUMS. This is
|
||||
`capture-rollback.sh` as product code.
|
||||
2. Stop the Console, then the bus. Wait for both to be inactive and
|
||||
for the stopped check to pass: no `writer.lock` holder and no bus
|
||||
process for the data root (`stopped-check.sh`).
|
||||
3. Read the stopped store's schema version without writing. The
|
||||
owner picks the read, for example a read-only SQLite open of the
|
||||
version the store records, and the packet names it. If the version
|
||||
is the new copy's `SCHEMA_VERSION`, there's no backup. If it's
|
||||
older, back up the stopped store to
|
||||
`<dataRoot>/backups/stack/<activation>/` (`backup.sh`: 0700, sums,
|
||||
a source and copy comparison, the stopped check again after the
|
||||
copy), then run the new copy's `migrate-schema.mjs`. If it's newer
|
||||
than the new copy's, or not in the copy's chain, refuse
|
||||
`store-newer`. The stack is stopped by then, so the refusal goes
|
||||
through item 6 and starts the old stack again.
|
||||
4. Install the bus unit with `bus-service.sh install --copy ID`,
|
||||
keeping `--pm` and the `--claude` pin the pointer recorded. Then
|
||||
install the Console with `webui-service.sh install --copy ID`,
|
||||
keeping its saved flags. Then switch the wrapper with
|
||||
`stack-copy.sh wrapper ID`.
|
||||
5. Start the bus, then the Console.
|
||||
5. The health gate runs after the start and passes only when all of
|
||||
these hold within a timeout (default 60 s, `--timeout`):
|
||||
- `systemctl --user is-active` reports both units active;
|
||||
- the copy's `scripts/mosaic bus status --json` exits 0 and shows a
|
||||
live host for BUSINESS whose pid is the bus unit's MainPID, with the
|
||||
boot status ok (`packages/cli/src/cli.mjs:499-514`). The store opens
|
||||
only at its own schema, so a running host means the schema matches;
|
||||
- the Console's `/healthz` answers 200 on its configured port;
|
||||
- `stack-copy.sh check` passes;
|
||||
- with `--pm`, `scripts/mosaic launches list` shows the PM.
|
||||
|
||||
On a pass, the pointer is renamed into place and the log says `ok`.
|
||||
The pointer changes only on a pass.
|
||||
6. A failed gate, or a failure after step 2, rolls back inside the same
|
||||
run. It stops the Console and the bus, then puts back the rollback
|
||||
set's unit files, drop-ins and wrapper bytes and runs `daemon-reload`.
|
||||
If step 3 migrated, it restores the backup the way `restore.sh` does:
|
||||
the current `bus/` is renamed to `bus.post-attempt-<stamp>`, nothing
|
||||
is deleted, and the restore is checked against the backup's sums.
|
||||
Then it starts the old stack and runs the same gate against the
|
||||
pointer's copy. The log records `gate-failed` with the failing check
|
||||
and the outcome of the rollback. If the rollback's own gate fails,
|
||||
the script stops, prints both results and leaves the stack stopped,
|
||||
with a `rollback-failed` line. It never retries.
|
||||
7. `rollback BUSINESS` moves to the copy the log names as the one
|
||||
before the pointer's, the same way `release.sh rollback` finds it. It
|
||||
runs steps 4.1-4.5 and the gate with that copy as the target. If the
|
||||
store's schema is newer than that copy's, it refuses
|
||||
`schema-differs` unless `--restore ACTIVATION` names the backup taken
|
||||
when the store was migrated. With `--restore`, it prints the
|
||||
post-attempt path it keeps and how many bus events were written after
|
||||
the backup's stamp, then restores.
|
||||
8. `status BUSINESS` prints the pointer, the last five log lines, the
|
||||
live units' copies, the result of `stack-copy.sh check`, `bus status`
|
||||
and the drift from `refactor`. It only reads.
|
||||
9. Who runs it. `adopt`, `activate` and `rollback` stop and start the
|
||||
live units, so they are run by Jason or Mos only, and each run is a
|
||||
live step that needs Mos's go, as TOOLS.md says today. Decisions 94
|
||||
and 102 still decide who starts the Console. `webui-service.sh`
|
||||
keeps having no start or stop verb (:50). The release script is the
|
||||
one path that stops and starts, and its header comment says who may
|
||||
run it. `status` is open to every seat.
|
||||
|
||||
Tests run with a stand-in `systemctl` on `PATH`, temporary
|
||||
`XDG_CONFIG_HOME` and `XDG_DATA_HOME`, a temporary data root, and fake
|
||||
copies whose `scripts/mosaic` and Console are small stubs, as
|
||||
`packages/cli/tests/stack-copy.test.mjs` and
|
||||
`packages/webui/tests/service.test.mjs` do. The cases:
|
||||
|
||||
- `adopt` writes the pointer and one log line only when the check passes.
|
||||
A second `adopt` refuses.
|
||||
- Each refusal in item 3 leaves the units, the wrapper, the pointer and
|
||||
the store unchanged, and it writes one log line with its code.
|
||||
- An `activate` with no schema change: the stand-in sees the stops, the
|
||||
installs and the starts in that order. The pointer names the new
|
||||
copy, and no backup directory exists.
|
||||
- An `activate` with a schema change makes the backup before the
|
||||
migration. A fixture migration that fails restores the backup and
|
||||
starts the old stack. The post-attempt directory exists, and the
|
||||
restored files match the backup's sums.
|
||||
- Each check in item 5, failing on its own, rolls back. The pointer is
|
||||
unchanged, the old unit files and wrapper bytes come back
|
||||
byte-for-byte, and the log names that check.
|
||||
- A failing rollback gate stops with `rollback-failed` and starts
|
||||
nothing more.
|
||||
- `rollback` finds the previous copy from the log, refuses
|
||||
`schema-differs` without `--restore`, and restores with it.
|
||||
- The store is never opened for writing outside `migrate-schema.mjs` and
|
||||
the restore, and nothing under `backups/` or `bus.post-attempt-*` is
|
||||
removed.
|
||||
|
||||
Mutants: one per behavior above, with the killing test in the packet.
|
||||
|
||||
### Out of scope
|
||||
|
||||
- Any live run of `adopt`, `activate` or `rollback`. The first `adopt` and
|
||||
the first `activate` happen in a Mos window, with Mos's go.
|
||||
- The Discord unit (`docs/plans/2026-10-14_discord-pinned-copy.md`). It
|
||||
joins in a follow-up once D1 lands.
|
||||
- Pruning copies, rollback sets or backups.
|
||||
- A down-migration, any change to the store's schema chain, and any change
|
||||
to `migrate-schema.mjs`.
|
||||
- A health endpoint in the bus. The gate uses `bus status`.
|
||||
- Any change to the human proof (`packages/bus/src/human.mjs`) or to
|
||||
`packages/queue`.
|
||||
- CI, which is D4.
|
||||
|
||||
### Gate
|
||||
|
||||
Filbert approves on the row's issue, naming the candidate manifest. The
|
||||
cli, bus and webui node suites and every `scripts/test-*.sh` pass on
|
||||
Sage's gate rerun. The landing changes nothing live. Sage sends Mos the
|
||||
commit and `stack-release.sh status` output from a scratch root. The
|
||||
first live `adopt` waits for Mos's go.
|
||||
|
||||
## Stack rehearsal on a scratch root (D2)
|
||||
|
||||
### Problem
|
||||
|
||||
Decision 96's D2 is staging: trying a candidate before live takes it.
|
||||
Today the trial is `rehearse.sh`, a plan-local script that checks only the
|
||||
migration on copies of the backup. Nothing starts the new copy's bus host
|
||||
and Console against real data before the live window, so a failure that
|
||||
shows only at start is found with the live stack stopped.
|
||||
|
||||
### Owner and reviewer
|
||||
|
||||
- Owner: set when the row is briefed.
|
||||
- Reviewer: filbert.
|
||||
|
||||
### Files owned
|
||||
|
||||
- `scripts/stack-release.sh`: the `rehearse` verb
|
||||
- `packages/cli/tests/stack-release.test.mjs`: the rehearsal cases
|
||||
- `docs/TOOLS.md`, the same section as D1
|
||||
- the packet `agents/<owner>/work/queue-<row>/`
|
||||
|
||||
### What ships
|
||||
|
||||
1. `rehearse ID BUSINESS --backup ACTIVATION|PATH` builds a scratch stack
|
||||
under a new 0700 directory, `<dataRoot>/rehearsals/<stamp>/`:
|
||||
- a data root holding copies of the backup's store files;
|
||||
- the business file, roles and organization file copied from the live
|
||||
config directory, read only;
|
||||
- a scratch config that names the scratch data root.
|
||||
|
||||
The live store, live config and the backup are never written. The
|
||||
backup's sums are checked before and after.
|
||||
2. On the scratch root it runs the de352dfaf rehearsal checks with the
|
||||
new copy: the new code refuses the unmigrated store at a normal open,
|
||||
`migrate-schema.mjs` succeeds, a second run is a no-op, and the old
|
||||
copy refuses the migrated store. Those checks apply only when the
|
||||
schema differs.
|
||||
3. It starts the new copy's bus host in the foreground on the scratch
|
||||
root with launches disabled, so no role session starts and no model
|
||||
is called. It starts the new copy's Console on a free loopback port
|
||||
with no send. Then it runs D1's gate checks against both, except the
|
||||
PM check. It stops both and prints one PASS or FAIL line per check,
|
||||
then `REHEARSAL_PASS` or `REHEARSAL_FAIL`.
|
||||
4. "Launches disabled" is a host flag the owner adds if the host lacks
|
||||
one. Its default is off, and the live units never pass it. If the
|
||||
host can't start without launching, the row stops and Sage takes the
|
||||
design back to Mos. No rehearsal calls a model.
|
||||
5. The rehearsal directory stays for reading. Nothing prunes it, and it
|
||||
is 0700 because it holds bus data.
|
||||
6. Who runs it. `rehearse` reads a backup of the live store, so only
|
||||
Jason or Mos run it, as the window's pre-step (the old P-steps). No
|
||||
development seat runs it against live data. Tests use fixture stores.
|
||||
|
||||
Tests:
|
||||
- The rehearsal passes on a fixture store and copy, and the scratch root
|
||||
holds the expected files.
|
||||
- The live store, the live config files and the backup are byte-identical
|
||||
before and after.
|
||||
- A migration that fails, a new copy whose host doesn't start, and a
|
||||
Console that doesn't answer each give `REHEARSAL_FAIL` naming the check.
|
||||
- No role session starts: the fixture's launcher stub records no start.
|
||||
- A scratch port that is in use refuses and picks no other port that
|
||||
could be the live Console's.
|
||||
|
||||
Mutants: one per behavior above.
|
||||
|
||||
### Out of scope
|
||||
|
||||
- A standing staging stack with its own units, Console, business and
|
||||
model turns. That needs a second system config beside
|
||||
`~/.config/mosaic-dev/config.json` (AGENTS.md invariant 2) and model
|
||||
spend, so it's Jason's call. It is not briefed here.
|
||||
- Everything D1's out-of-scope list names.
|
||||
|
||||
### Gate
|
||||
|
||||
As D1. D2 lands after D1, because both edit `stack-release.sh` and its
|
||||
test file.
|
||||
Reference in new issue
Block a user