stack/agents/darkwing/work/queue-233/BUILD.md

Row 233 (I3): root's upgrade driver and the host run sheet

Brief: docs/plans/2026-10-20_installer-rows.md section I3, blob e61291bcf1c6 (commit a0e3d7cfd), re-pinned by Sage (sage-1685, rev 2631) from f35e7e57daee (ad6f0be7a). The later blob only appends I4; the I3 text is the same. Gitea #1735, Vikunja P19 #432. Owner Darkwing, reviewer Filbert, gate owner Sage. Design input: row 195 DESIGN r6, agents/darkwing/work/queue-195/DESIGN.md at b7239bb66, sections 2, 4 and 7. Where the brief and r6 differ, the brief wins. Sage's rulings during the build: sage-1650 and sage-1652 (--backup last, the --no-rehearsal mapping, --rehearse-port, hostcheck), and sage-1653 (from_log newest, the two external plan refusals, the BACKUP_STORE line).

Files

Base: e65a3ab4d, row 232's landing, on darkwing/row233-r1. The build started on ea1e85c7b (darkwing/row233, row 232's candidate e6ca230d2 and its BUILD.md) before row 232 landed. The three row 232 files this row changes are identical on the two bases, so the uncommitted diff moved across unchanged.

• scripts/stack-upgrade.sh (new): upgrade and rollback, root's side of r6 A1-A5, A2f and R1-R4.
• packages/cli/tests/stack-upgrade.test.mjs (new, 20 tests).
• docs/guides/host-run-sheet.md (new).
• docs/TOOLS.md: a new subsection "Root's upgrade driver (scripts/stack-upgrade.sh, row 233)", plus the row 232 lines below.
• Row 232 files, changed on Sage's rulings (the brief's files list doesn't name them):
  • scripts/stack-release.sh:
    • rehearse --backup last takes the newest backup the log records (from_log newest) and prints PASS backup-last. With none it fails no-backup, exits 3 and makes no rehearsal directory. A newest backup that isn't an activation id aborts backup-unreadable, exit 1.
    • The rehearsal prints BACKUP_STORE <activation id> schema <version> before its migration step.
    • --plan under --units external refuses open-activation when any rollback set holds AWAITING-UNITS, and migrate-required for a migrating target without --migrate. Root reads only the plan's exit, so these two have to be refusals there. User mode prints the same bytes.
  • packages/cli/tests/stack-release.test.mjs: three new row 233: tests for --backup last.
  • docs/TOOLS.md: --backup ACTIVATION|PATH|last in the rehearse usage, a paragraph on last, and a paragraph on the two external plan refusals.
• This file.

The brief, item by item

Brief item	Where
Root runs from a root-owned checkout, never a copy	head-mismatch (A1); TOOLS.md; run sheet section 6
I2's render with /etc/mosaic-stack/<instance>.args, --copy the one changed value	read_args, args_with; the args test
The args file changes only after a confirmed run	prepare_args before A2, mv after A5; sequences 1 and 7, m6
copy --bundle as the stack's user before A1	the fetch, ff, copy steps; sequence 1's sudo keys
A rehearsal before A2, a failed one refuses with nothing stopped	rehearse; three rehearsal tests, m4
The root lock, held to the end, rollback included	flock -n 9; the lock test and sequence 3's probe, m1, m2
A1 refuses open-activation	stack-release.sh's external plan; the AWAITING-UNITS test, m3
migrate-required without --migrate	stack-release.sh's external plan; its test, m5
The step log, 0640, one line per step	logl, step; every test checks lines against STEP
Exit codes 0-3	finish, refuse; exit 4 is usage
duckworth-0055's notes	the next section
Root parses no stack user output beyond exits	the step log test; the one exception is sage-1653's BACKUP_STORE
The host run sheet	docs/guides/host-run-sheet.md

duckworth-0055's notes on r6 (brief line 169)

• N1, A2's stop timeout under gate --since. Row 232's BUILD.md left two pieces to I3. The run sheet's A2f line is in section 6 ("When a run doesn't exit 0", exit 2). The test is N1: a bus that outlives A2's timeout keeps its old start: a bus that ignores stop keeps its old ActiveEnterTimestamp, wait-down logs rc=1 ... state: active, A2f restarts and gates, the gate fails stale-start, and the driver stops both units and exits 2. Sequence 4 is the other half: an A3 exit 3 goes to A2f and exits 1 with the old copy's gate passing.
• N2, the claude path under /opt/mosaic-stack. Run sheet section 3 installs it there as root with managed settings. The args file's --claude carries it, and case2-unit.sh already refuses a binary that isn't root's or sits under the home. The driver passes the args file through and adds no claude check of its own. F6 is I2's.

Sage's rulings during the build

• sage-1650 and sage-1652. Root always runs the rehearsal and maps its exit:

  • 0 goes on, or refuses rehearsal-required with --no-rehearsal.
  • 3 refuses no-backup, or with --no-rehearsal logs rehearsal-skipped and goes on.
  • Anything else refuses rehearsal-failed.

  --rehearse-port N, default 7339. Hostcheck: a missing file or a nonzero --check refuses profile-drift.

• sage-1653. from_log newest, and the external plan refusals open-activation and migrate-required. Residual: the BACKUP_STORE id and schema land in root's step log, but nothing ties that backup to the one the activation's own backup step takes later. A stopped backup before the upgrade would close that gap; it's a later row, not this one.

Tests

stack-upgrade.test.mjs runs the driver as the test's user against a scratch root, with the test seams STACK_UPGRADE_*. The stand-ins are a fake sudo (logs its argv, checks the driver's form, then runs the real command as the test's user, r6's B1 pattern), a fake case2-unit.sh render that installs and starts the fixture units, a fake systemctl for root's system units, and a shell hostcheck.sh. The stack's user's steps run the real stack-release.sh and stack-copy.sh from a clone made from a real bundle. The units, copies and store are stack-release-fixture.mjs's.

Test	Covers
sequence 1	confirm fails healthz, rollback with --cause A5, exit 1; every sudo call in the driver's form
sequence 2	activate --migrate fails at the wrapper, store restored, --cause A3, exit 1
sequence 3	render fails, --cause A4; a probe run during render-old refuses upgrade-running
sequence 4	stage exits 3, A2f restarts and gates the old units, exit 1
N1	a unit that won't stop: A2f's gate fails stale-start, exit 2, units stopped
sequence 5	confirm of the wrong copy refused, --cause A5, exit 1
sequence 6	revert fails, exit 2; the next run refuses interrupted-run at its plan
sequence 7	two upgrades and a rollback --to with --restore; the args file after each confirm; to-not-previous
sequence 8	a migration that writes and fails, store back byte for byte, exit 1
lock	a second run while the lock is held: one log line, no call
open-activation	any AWAITING-UNITS refuses at A1, nothing stopped
migrate-required	refused without --migrate, no rehearsal, nothing stopped
rehearsal-failed	the rehearsal's argv, exit 3, nothing stopped
BACKUP_STORE	the line after a match; four malformed lines log BACKUP_STORE none
no-backup	the real rehearsal with no backup refuses; --no-rehearsal logs the skip and goes on
backup-last	the real rehearsal names the newest backup; --no-rehearsal with one refuses rehearsal-required
args file	group write, other write, two --copy, an unknown flag, a symlink, args-differ
hostcheck	nonzero --check and a missing file refuse profile-drift before the plan
head-mismatch	checkout not at --sha; a commit the clone can't fast-forward to refuses fetch-failed
usage	twelve bad argvs exit 4 with no log and no lock

The step log check (STEP) runs in every test that reads the log: each line must be a step line or the result line, and none may hold the stack user's output (stack-release:, PASS , FAIL , REHEARSAL, the fake rehearsal's text).

Mutants

~/darkwing-scratch/r233/mutants.sh, not committed because it edits the worktree. It has the brief's six mutants and two for sage-1650's mapping. The method is row 232's: a checked backup, exact replacements, a syntax check, the named killers, then the file put back with its sha256 checked. One control runs first: the whole stack-upgrade.test.mjs with no change. DRY=1 matched each replacement once on the candidate (2026-10-21).

Mutant	File	Killers
m1 drop the flock	stack-upgrade.sh	a second run while the lock is held, sequence 3:
m2 release the lock when the rollback starts	stack-upgrade.sh	sequence 3:
m3 skip A1's open-activation check	stack-release.sh	A1 refuses open-activation
m4 skip the rehearsal	stack-upgrade.sh	the rehearsal runs after the plan, sequence 1:
m5 accept a migrating target without --migrate	stack-release.sh	migrate-required
m6 write the args file before confirm	stack-upgrade.sh	sequence 7:, sequence 1:
m7 --no-rehearsal accepted with a backup	stack-upgrade.sh	refuses rehearsal-required
m8 drop the rehearsal-skipped line	stack-upgrade.sh	a real rehearsal with no backup

Deviations

D1. The brief has A1 refuse open-activation for an open AWAITING-UNITS or a non-empty interrupted set. The interrupted case refuses with row 232's existing code, interrupted-run; only AWAITING-UNITS gives open-activation. Both reach root as plan-refused, exit 3 (sequence 6, the open-activation test).

D2. Root copies one line of the stack user's output into its log, BACKUP_STORE, on sage-1653. It matches a fixed pattern first, takes the last match, logs BACKUP_STORE none otherwise, and decides nothing on it.

D3. The stack user's steps carry no --instance. The driver takes --instance dev only, row 190's default instance, which the stack user's commands select with no flag. A named instance's units are a later row.

D4. Most tests fake the rehearsal in the fake sudo: the fixture's stub copies can't pass a real one. Two tests run the real rehearsal (faults/real-rehearse): no-backup and backup-last. stack-release.test.mjs covers --backup last itself.

D5. Any nonzero --plan exit refuses plan-refused, exit 3, whatever the code. The stack user's output above it names the refusal; root doesn't parse it.

D6. rollback --restore ACT is accepted without --events, as stack-release.sh rollback accepts it. The brief's usage pairs them. --events without --restore is a usage error.

D7. The refusal of any STACK_UPGRADE_* seam when run as root has no test, because the tests don't run as root. It is three lines at the top of the script.

D8. The fake sudo keeps the test's environment, where real sudo resets it. The driver passes everything the stack user needs through env PATH=... and -H, and the fake checks that form.

D9. Exit 3 means nothing stopped, not nothing changed. A refusal after fetch leaves the stack user's clone fast-forwarded and the new copy made. Both are the user's own state, and the next run reuses them. TOOLS.md and the run sheet say so.

D10. The brief's usage has neither --no-rehearsal nor --rehearse-port. Both come from sage-1650 and sage-1652.

Decisions

• rollback has no HEAD check. Its target is an older copy than root's checkout, which already exists. It fetches, copies and rehearses nothing (sequence 7).
• The new args file is written next to the old one (.dev.args.XXXXXX, same mode) before A2, so a full /etc refuses before anything stops. It moves into place after A5. A2f and the rollback remove it; a failed move keeps it and names it.
• Every stack user step gets --copies explicitly, so the copies directory comes from the args file's --home, never from the user's environment.
• The stack's user does the fetch, the fast-forward and the copy, so root never runs git on a tree that user can write.
• A4 is the render, and case2-unit.sh starts the bus, then the Console. The forward path has no systemctl start of its own; only A2f does.

Run environment

Worktree /mnt/storage/scratch/darkwing-r233, TMPDIR=/mnt/storage/scratch/tmp/r233, node from PATH. One whole stack-upgrade.test.mjs run takes about 56 s.

Results

Singles, run 1, 2026-10-21, Sage's slot (sage-1689)

~/darkwing-scratch/r233/singles.sh on darkwing/row233-r1 (base e65a3ab4d, uncommitted tree), 03:48:20Z to 03:59:00Z. These are the test files that read a changed script. stack-instances.test.mjs reads stack-release.sh, so it joined the three named in the plan.

File	Result
stack-upgrade.test.mjs	20 pass, 0 fail
stack-release.test.mjs	194 pass, 1 fail
stack-release-external.test.mjs	25 pass, 0 fail
stack-instances.test.mjs	23 pass, 0 fail

The failure was my test, not the script. "row 233: --backup last takes the newest backup..." compares untouched(e) before and after the run, and untouched includes existsSync(e.log). The test took its before snapshot ahead of rlog(), which writes the log, so that flag went from false to true. Every hash in the snapshot matched. The fix takes the snapshot after rlog(). No other row 233 test calls untouched.

Singles, run 2

~/darkwing-scratch/r233/rerun.sh on aacfa2d7a, Sage's slot (sage-1695), 04:13:05Z to 04:21:28Z: stack-release.test.mjs whole, 195 pass, 0 fail, 0 skipped. The fixed test passed. The other three files did not change after run 1. The gate and the mutants run next, in Sage's slot.

Gate and mutants, 2026-10-21, Sage's slot (sage-1704)

~/darkwing-scratch/r233/gate.sh then mutants.sh, chained behind busy.sh, on d99166ade (code aacfa2d7a; d99166ade changes only this file). Counts are an awk sum over gate-r1.out.

Part	Time (UTC)	Result
cli suite (node --test tests/*.test.mjs)	04:29:01-04:37:27	616 pass, 0 fail, 0 skipped
the 10 scripts/test-*.sh	04:37:27-04:39:38	selftests 429 passed, 0 failed; test-queue's node part 156 pass, 0 fail
gate total		1201 pass, 0 fail, 11 suites, GATE_PASSED
c0 control	04:39:38-04:40:35	exit 0
m1-m8	04:40:35-04:41:25	8 of 8 killed, every one exit 1

Every restore passed its sha256 check, and git diff was clean at d99166ade afterwards. The EXIT trap removed the darkwing-r233_default network.

N1, found in the process sweep after the run: a sleep 60 was left with its cwd in packages/cli. stack-upgrade.test.mjs:312 spawns flock LOCK sleep 60, and its t.after SIGKILLs only the flock, so flock's child outlives the test by up to 60 s. It exited on its own. Row 232's T20 (stack-release.test.mjs:370) spawns its holder detached and kills the group with process.kill(-pid). The fix is the same two lines. That test is one of m1's killers.

Sage ruled it a fix now (sage-1707), and 08db6ac3f makes it, test only. After the gate's d99166ade the only change outside this file is that test:

$ git diff --stat d99166ade 08db6ac3f
 packages/cli/tests/stack-upgrade.test.mjs | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

The gate stands on Sage's ruling. The rerun is the whole stack-upgrade.test.mjs singly plus m1, in Sage's slot, with a sweep for a leftover sleep afterwards.