stack/agents/darkwing/work/queue-233/BUILD.md

# Row 233 (I3): root's upgrade driver and the host run sheet

Brief: `docs/plans/2026-10-20_installer-rows.md` section I3, blob `e61291bcf1c6` (commit a0e3d7cfd), re-pinned
by Sage (sage-1685, rev 2631) from `f35e7e57daee` (ad6f0be7a). The later blob only appends I4; the I3 text
is the same. Gitea #1735, Vikunja P19
#432. Owner Darkwing, reviewer Filbert, gate owner Sage. Design input: row 195 DESIGN r6,
`agents/darkwing/work/queue-195/DESIGN.md` at b7239bb66, sections 2, 4 and 7. Where the brief and r6
differ, the brief wins. Sage's rulings during the build: sage-1650 and sage-1652 (`--backup last`, the
`--no-rehearsal` mapping, `--rehearse-port`, hostcheck), and sage-1653 (`from_log newest`, the two external
plan refusals, the `BACKUP_STORE` line).

## Files

Base: e65a3ab4d, row 232's landing, on `darkwing/row233-r1`. The build started on ea1e85c7b
(`darkwing/row233`, row 232's candidate e6ca230d2 and its BUILD.md) before row 232 landed. The three row 232
files this row changes are identical on the two bases, so the uncommitted diff moved across unchanged.

- `scripts/stack-upgrade.sh` (new): `upgrade` and `rollback`, root's side of r6 A1-A5, A2f and R1-R4.
- `packages/cli/tests/stack-upgrade.test.mjs` (new, 20 tests).
- `docs/guides/host-run-sheet.md` (new).
- `docs/TOOLS.md`: a new subsection "Root's upgrade driver (`scripts/stack-upgrade.sh`, row 233)", plus the
  row 232 lines below.
- Row 232 files, changed on Sage's rulings (the brief's files list doesn't name them):
  - `scripts/stack-release.sh`:
    - `rehearse --backup last` takes the newest backup the log records (`from_log newest`) and prints
      `PASS backup-last`. With none it fails `no-backup`, exits 3 and makes no rehearsal directory.
      A newest backup that isn't an activation id aborts `backup-unreadable`, exit 1.
    - The rehearsal prints `BACKUP_STORE <activation id> schema <version>` before its migration step.
    - `--plan` under `--units external` refuses `open-activation` when any rollback set holds
      `AWAITING-UNITS`, and `migrate-required` for a migrating target without `--migrate`. Root reads
      only the plan's exit, so these two have to be refusals there. User mode prints the same bytes.
  - `packages/cli/tests/stack-release.test.mjs`: three new `row 233:` tests for `--backup last`.
  - `docs/TOOLS.md`: `--backup ACTIVATION|PATH|last` in the rehearse usage, a paragraph on `last`, and a
    paragraph on the two external plan refusals.
- This file.

## The brief, item by item

| Brief item | Where |
|---|---|
| Root runs from a root-owned checkout, never a copy | `head-mismatch` (A1); TOOLS.md; run sheet section 6 |
| I2's render with `/etc/mosaic-stack/<instance>.args`, `--copy` the one changed value | `read_args`, `args_with`; the args test |
| The args file changes only after a confirmed run | `prepare_args` before A2, `mv` after A5; sequences 1 and 7, m6 |
| `copy --bundle` as the stack's user before A1 | the `fetch`, `ff`, `copy` steps; sequence 1's sudo keys |
| A rehearsal before A2, a failed one refuses with nothing stopped | `rehearse`; three rehearsal tests, m4 |
| The root lock, held to the end, rollback included | `flock -n 9`; the lock test and sequence 3's probe, m1, m2 |
| A1 refuses `open-activation` | stack-release.sh's external plan; the AWAITING-UNITS test, m3 |
| `migrate-required` without `--migrate` | stack-release.sh's external plan; its test, m5 |
| The step log, 0640, one line per step | `logl`, `step`; every test checks lines against `STEP` |
| Exit codes 0-3 | `finish`, `refuse`; exit 4 is usage |
| duckworth-0055's notes | the next section |
| Root parses no stack user output beyond exits | the step log test; the one exception is sage-1653's `BACKUP_STORE` |
| The host run sheet | `docs/guides/host-run-sheet.md` |

## duckworth-0055's notes on r6 (brief line 169)

- N1, A2's stop timeout under `gate --since`. Row 232's BUILD.md left two pieces to I3. The run sheet's
  A2f line is in section 6 ("When a run doesn't exit 0", exit 2). The test is `N1: a bus that outlives
  A2's timeout keeps its old start`: a bus that ignores `stop` keeps its old `ActiveEnterTimestamp`,
  `wait-down` logs `rc=1 ... state: active`, A2f restarts and gates, the gate fails `stale-start`, and the
  driver stops both units and exits 2. Sequence 4 is the other half: an A3 exit 3 goes to A2f and exits
  1 with the old copy's gate passing.
- N2, the claude path under `/opt/mosaic-stack`. Run sheet section 3 installs it there as root with
  managed settings. The args file's `--claude` carries it, and `case2-unit.sh` already refuses a binary
  that isn't root's or sits under the home. The driver passes the args file through and adds no claude
  check of its own. F6 is I2's.

## Sage's rulings during the build

- sage-1650 and sage-1652. Root always runs the rehearsal and maps its exit:
  - 0 goes on, or refuses `rehearsal-required` with `--no-rehearsal`.
  - 3 refuses `no-backup`, or with `--no-rehearsal` logs `rehearsal-skipped` and goes on.
  - Anything else refuses `rehearsal-failed`.

  `--rehearse-port N`, default 7339. Hostcheck: a missing file or a nonzero `--check` refuses
  `profile-drift`.
- sage-1653. `from_log newest`, and the external plan refusals `open-activation` and `migrate-required`.
  Residual: the `BACKUP_STORE` id and schema land in root's step log, but nothing ties that backup to the
  one the activation's own backup step takes later. A stopped backup before the upgrade would close that
  gap; it's a later row, not this one.

## Tests

`stack-upgrade.test.mjs` runs the driver as the test's user against a scratch root, with the test seams
`STACK_UPGRADE_*`. The stand-ins are a fake `sudo` (logs its argv, checks the driver's form, then runs the
real command as the test's user, r6's B1 pattern), a fake `case2-unit.sh` render that installs and starts
the fixture units, a fake `systemctl` for root's system units, and a shell `hostcheck.sh`. The stack's
user's steps run the real `stack-release.sh` and `stack-copy.sh` from a clone made from a real bundle. The
units, copies and store are `stack-release-fixture.mjs`'s.

| Test | Covers |
|---|---|
| sequence 1 | confirm fails healthz, rollback with `--cause A5`, exit 1; every sudo call in the driver's form |
| sequence 2 | `activate --migrate` fails at the wrapper, store restored, `--cause A3`, exit 1 |
| sequence 3 | render fails, `--cause A4`; a probe run during `render-old` refuses `upgrade-running` |
| sequence 4 | stage exits 3, A2f restarts and gates the old units, exit 1 |
| N1 | a unit that won't stop: A2f's gate fails `stale-start`, exit 2, units stopped |
| sequence 5 | confirm of the wrong copy refused, `--cause A5`, exit 1 |
| sequence 6 | revert fails, exit 2; the next run refuses `interrupted-run` at its plan |
| sequence 7 | two upgrades and a `rollback --to` with `--restore`; the args file after each confirm; `to-not-previous` |
| sequence 8 | a migration that writes and fails, store back byte for byte, exit 1 |
| lock | a second run while the lock is held: one log line, no call |
| open-activation | any `AWAITING-UNITS` refuses at A1, nothing stopped |
| migrate-required | refused without `--migrate`, no rehearsal, nothing stopped |
| rehearsal-failed | the rehearsal's argv, exit 3, nothing stopped |
| `BACKUP_STORE` | the line after a match; four malformed lines log `BACKUP_STORE none` |
| no-backup | the real rehearsal with no backup refuses; `--no-rehearsal` logs the skip and goes on |
| backup-last | the real rehearsal names the newest backup; `--no-rehearsal` with one refuses `rehearsal-required` |
| args file | group write, other write, two `--copy`, an unknown flag, a symlink, `args-differ` |
| hostcheck | nonzero `--check` and a missing file refuse `profile-drift` before the plan |
| head-mismatch | checkout not at `--sha`; a commit the clone can't fast-forward to refuses `fetch-failed` |
| usage | twelve bad argvs exit 4 with no log and no lock |

The step log check (`STEP`) runs in every test that reads the log: each line must be a step line or the
result line, and none may hold the stack user's output (`stack-release:`, `PASS `, `FAIL `, `REHEARSAL`,
the fake rehearsal's text).

## Mutants

`~/darkwing-scratch/r233/mutants.sh`, not committed because it edits the worktree. It has the brief's six
mutants and two for sage-1650's mapping. The method is row 232's: a checked backup, exact replacements, a
syntax check, the named killers, then the file put back with its sha256 checked. One control runs first:
the whole `stack-upgrade.test.mjs` with no change. `DRY=1` matched each replacement once on the candidate
(2026-10-21).

| Mutant | File | Killers |
|---|---|---|
| m1 drop the flock | stack-upgrade.sh | `a second run while the lock is held`, `sequence 3:` |
| m2 release the lock when the rollback starts | stack-upgrade.sh | `sequence 3:` |
| m3 skip A1's open-activation check | stack-release.sh | `A1 refuses open-activation` |
| m4 skip the rehearsal | stack-upgrade.sh | `the rehearsal runs after the plan`, `sequence 1:` |
| m5 accept a migrating target without `--migrate` | stack-release.sh | `migrate-required` |
| m6 write the args file before confirm | stack-upgrade.sh | `sequence 7:`, `sequence 1:` |
| m7 `--no-rehearsal` accepted with a backup | stack-upgrade.sh | `refuses rehearsal-required` |
| m8 drop the `rehearsal-skipped` line | stack-upgrade.sh | `a real rehearsal with no backup` |

## Deviations

D1. The brief has A1 refuse `open-activation` for an open `AWAITING-UNITS` or a non-empty `interrupted`
set. The interrupted case refuses with row 232's existing code, `interrupted-run`; only `AWAITING-UNITS`
gives `open-activation`. Both reach root as `plan-refused`, exit 3 (sequence 6, the open-activation test).

D2. Root copies one line of the stack user's output into its log, `BACKUP_STORE`, on sage-1653. It
matches a fixed pattern first, takes the last match, logs `BACKUP_STORE none` otherwise, and decides
nothing on it.

D3. The stack user's steps carry no `--instance`. The driver takes `--instance dev` only, row 190's default
instance, which the stack user's commands select with no flag. A named instance's units are a later row.

D4. Most tests fake the rehearsal in the fake `sudo`: the fixture's stub copies can't pass a real one. Two
tests run the real rehearsal (`faults/real-rehearse`): no-backup and backup-last. `stack-release.test.mjs`
covers `--backup last` itself.

D5. Any nonzero `--plan` exit refuses `plan-refused`, exit 3, whatever the code. The stack user's output
above it names the refusal; root doesn't parse it.

D6. `rollback --restore ACT` is accepted without `--events`, as `stack-release.sh rollback` accepts it.
The brief's usage pairs them. `--events` without `--restore` is a usage error.

D7. The refusal of any `STACK_UPGRADE_*` seam when run as root has no test, because the tests don't run
as root. It is three lines at the top of the script.

D8. The fake `sudo` keeps the test's environment, where real `sudo` resets it. The driver passes
everything the stack user needs through `env PATH=...` and `-H`, and the fake checks that form.

D9. Exit 3 means nothing stopped, not nothing changed. A refusal after `fetch` leaves the stack user's
clone fast-forwarded and the new copy made. Both are the user's own state, and the next run reuses them.
TOOLS.md and the run sheet say so.

D10. The brief's usage has neither `--no-rehearsal` nor `--rehearse-port`. Both come from sage-1650 and
sage-1652.

## Decisions

- `rollback` has no HEAD check. Its target is an older copy than root's checkout, which already exists.
  It fetches, copies and rehearses nothing (sequence 7).
- The new args file is written next to the old one (`.dev.args.XXXXXX`, same mode) before A2, so a full
  `/etc` refuses before anything stops. It moves into place after A5. A2f and the rollback remove it; a failed move keeps it and names it.
- Every stack user step gets `--copies` explicitly, so the copies directory comes from the args file's
  `--home`, never from the user's environment.
- The stack's user does the fetch, the fast-forward and the copy, so root never runs git on a tree that
  user can write.
- A4 is the render, and `case2-unit.sh` starts the bus, then the Console. The forward path has no
  `systemctl start` of its own; only A2f does.

## Run environment

Worktree `/mnt/storage/scratch/darkwing-r233`, `TMPDIR=/mnt/storage/scratch/tmp/r233`, node from PATH. One
whole `stack-upgrade.test.mjs` run takes about 56 s.

## Results

### Singles, run 1, 2026-10-21, Sage's slot (sage-1689)

`~/darkwing-scratch/r233/singles.sh` on `darkwing/row233-r1` (base e65a3ab4d, uncommitted tree), 03:48:20Z
to 03:59:00Z. These are the test files that read a changed script. `stack-instances.test.mjs` reads
`stack-release.sh`, so it joined the three named in the plan.

| File | Result |
|---|---|
| `stack-upgrade.test.mjs` | 20 pass, 0 fail |
| `stack-release.test.mjs` | 194 pass, 1 fail |
| `stack-release-external.test.mjs` | 25 pass, 0 fail |
| `stack-instances.test.mjs` | 23 pass, 0 fail |

The failure was my test, not the script. "row 233: --backup last takes the newest backup..." compares
`untouched(e)` before and after the run, and `untouched` includes `existsSync(e.log)`. The test took its
before snapshot ahead of `rlog()`, which writes the log, so that flag went from false to true. Every hash in
the snapshot matched. The fix takes the snapshot after `rlog()`. No other row 233 test calls `untouched`.

### Singles, run 2

`~/darkwing-scratch/r233/rerun.sh` on aacfa2d7a, Sage's slot (sage-1695), 04:13:05Z to 04:21:28Z:
`stack-release.test.mjs` whole, 195 pass, 0 fail, 0 skipped. The fixed test passed. The other three files
did not change after run 1. The gate and the mutants run next, in Sage's slot.

### Gate and mutants, 2026-10-21, Sage's slot (sage-1704)

`~/darkwing-scratch/r233/gate.sh` then `mutants.sh`, chained behind `busy.sh`, on d99166ade (code
aacfa2d7a; d99166ade changes only this file). Counts are an awk sum over `gate-r1.out`.

| Part | Time (UTC) | Result |
|---|---|---|
| cli suite (`node --test tests/*.test.mjs`) | 04:29:01-04:37:27 | 616 pass, 0 fail, 0 skipped |
| the 10 `scripts/test-*.sh` | 04:37:27-04:39:38 | selftests 429 passed, 0 failed; `test-queue`'s node part 156 pass, 0 fail |
| gate total | | 1201 pass, 0 fail, 11 suites, `GATE_PASSED` |
| c0 control | 04:39:38-04:40:35 | exit 0 |
| m1-m8 | 04:40:35-04:41:25 | 8 of 8 killed, every one exit 1 |

Every restore passed its sha256 check, and `git diff` was clean at d99166ade afterwards. The EXIT trap
removed the `darkwing-r233_default` network.

N1, found in the process sweep after the run: a `sleep 60` was left with its cwd in `packages/cli`.
`stack-upgrade.test.mjs:312` spawns `flock LOCK sleep 60`, and its `t.after` SIGKILLs only the flock, so
flock's child outlives the test by up to 60 s. It exited on its own. Row 232's T20
(`stack-release.test.mjs:370`) spawns its holder `detached` and kills the group with `process.kill(-pid)`.
The fix is the same two lines. That test is one of m1's killers.

Sage ruled it a fix now (sage-1707), and 08db6ac3f makes it, test only. After the gate's d99166ade
the only change outside this file is that test:

```
$ git diff --stat d99166ade 08db6ac3f
 packages/cli/tests/stack-upgrade.test.mjs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
```

The gate stands on Sage's ruling. The rerun is the whole `stack-upgrade.test.mjs` singly plus m1, in
Sage's slot, with a sweep for a leftover `sleep` afterwards.