stack/docs/plans/2026-10-20_installer-rows.md

Briefs: the installer rows I1 and I3 (#432)

Sage, 2026-10-20, on Mos's go mos-1118 (the mapping in sage-1524, accepted). Vikunja P19 #432. Q187 (mos-1108): install and upgrade become product commands before either VM gets the stack. ops-01 proves them on a throwaway PVE VM, and the same commands then install 10.1.1.53 (Case 1, factory profile) and 10.1.1.54 (Case 2, library profile).

The work is in three rows:

• I1 (this file): stack-release.sh and stack-copy.sh learn system units and bundle copies.
• I2: rows 230 and 226, re-briefed in docs/plans/2026-10-20_case2-rows.md (C2-13a and C2-13b). The install command, the unit render, the precheck and the two profile files.
• I3 (this file): the root upgrade driver and the ops-01 host run sheet.

Row 195 shrinks to the factory profile's values and the Case 1 acceptance (docs/plans/2026-10-20_hub-own-uid.md).

The source for most of I1 and I3 is row 195's DESIGN r6, agents/darkwing/work/queue-195/DESIGN.md at b7239bb66, cited below as "r6". r6 designed these pieces for one host under case1-* names. These rows build them once, for any host, under the stack's own names. Where this brief and r6 differ, this brief wins.

Choices Mos accepted (mos-1118). They hold for all three rows:

1. Install starts on a host that ops-01 has prepared with the run sheet. Install's precheck verifies each prepared item and refuses when one is missing. Folding host prep into the command is a post-10-27 row, Vikunja #437.
2. The "next release" for the ops-01 proof is two refactor commits shipped as git bundles. I1 makes stack-copy.sh take a bundle.
3. Decision 96 stays as process: Mos's go per run, and Jason present for --migrate. upgrade refuses a migrating target unless --migrate is given.
4. The instance gets its name through row 190's --instance NAME. .53's profile value is dev, r6's name, which is row 190's default instance (scripts/instance.sh, "other than dev").
5. The Console starts by itself on both hosts (Q189, mos-1122). I2 renders and enables it. Nothing in I1 or I3 starts it by hand.

---

I1 stack-release external units and bundle copies

Gitea #1734. Vikunja P19 #432.

Problem

stack-release.sh calls systemctl --user and journalctl --user throughout, so it can't run a stack whose units are system units owned by root, as both VMs' are. stack-copy.sh makes a copy only from a commit reachable from refs/heads/refactor and refs/remotes/origin/refactor (scripts/stack-copy.sh:36-41), and a VM has no such checkout. Its wrapper verb refuses when no wrapper exists yet (:68-73), so a first install can't make one.

Owner and reviewer

• Owner: Darkwing.
• Reviewer: Filbert (moved from Duckworth, mos-1119 item 4).
• After: row 190, landed as 00d4cc1f6 on 2026-10-20. Build on 00d4cc1f6 on a new branch, then rebase to a new branch name if refactor moves under a path this row owns. Never force-push. Row 200 (Launchpad, activate --sso) also changes scripts/stack-release.sh and is rebasing onto 00d4cc1f6 now. Whichever of the two lands second rebases onto the other.

Files owned

scripts/stack-release.sh, scripts/stack-copy.sh, packages/cli/tests/stack-release-external.test.mjs (new), packages/cli/tests/stack-copy.test.mjs, packages/cli/tests/stack-release.test.mjs where a shared helper moves, and the docs/TOOLS.md and packages/cli/README.md lines for these verbs.

What ships

• --units user|external, default user. Under user, every command line and every byte of output is what it is today, and the existing tests pass unchanged. One units backend holds every systemctl and journalctl call, so the two modes differ in that one place.
• Under external, r6's pieces, as designed there:
  • adopt, activate, rollback and status (r6 :313-316, :459-559).
  • The open-activation marker AWAITING-UNITS and the confirm, revert and gate --since verbs, which refuse without --units external (r6 :443-457).
  • rollback --to ID, which refuses unless ID is from_log previous (r6 :545-555).
  • --plan --expect-from OLD, which refuses pointer-differs (r6 A1, N5).
  • The start floor from the bus unit's ActiveEnterTimestamp, compared in whole seconds, with gate code stale-start (r6 :491-505).
  • The PM check from launches list --json, with no journal read (r6 :507-523).
  • pm-on and activate --pm-after-harness refuse external-pm (r6 :525-526).
• stack-copy.sh check --units external, which reads the system units (r6 :739-741).
• stack-copy.sh copy --bundle FILE SHA. It runs git bundle verify, requires the bundle to carry refs/heads/refactor with SHA reachable from it, and makes the copy from SHA as copy does today. It prints the RELEASE value at SHA and the bundle's sha256. Without --bundle, reachable stays as it is. This is the release identity for a host with no refactor checkout.
• stack-copy.sh wrapper ID --create. It writes the first wrapper only when none exists at the wrapper path, and refuses when one does. Without --create, today's refusal on a missing wrapper stays.

Where the stack user's stack-release.sh runs (darkwing on sage-1558, ruled sage-1561): from the stack user's own clone of the release bundle under its home, as in r6 step 2b. Each upgrade fetches the new bundle into that clone's refs/heads/refactor and refs/remotes/origin/refactor. So check_target's reachability (scripts/stack-release.sh:226, :258), the plan's commit list (:929-938) and status drift (:1220) work unchanged under external, and I1 changes no reachability rule. "A VM has no such checkout" above means no canonical development checkout. The stack user never runs from root's checkout, and nothing adds safe.directory. copy --bundle still ships, for the first copy before the clone exists and as the release identity.

The installer driver that calls these verbs as root is I3, and the install is I2. I1 has no root step and calls no sudo.

Tests

r6 section 7's list for stack-release-external.test.mjs (r6 :742-812), all of it, including the eight sequences on one log. Plus these:

• Under --units user, a fake systemctl that records its argv sees the same calls as before on the existing activate and rollback tests.
• A bundle whose refs/heads/refactor doesn't reach SHA refuses. So do a corrupt bundle and a bundle with no refs/heads/refactor. A good bundle gives a copy byte-identical to copy SHA from a checkout.
• wrapper --create refuses over an existing wrapper, and writes a wrapper equal to wrapper ID's output.

Mutants: r6 section 7's list for these verbs, plus one that drops the bundle reachability check and one that lets --create overwrite.

Gate

Filbert approves the manifest. The cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. Target: in review 10-22, landed 10-23 (sage-1528 line 8).

---

I3 the upgrade command and the host run sheet

Gitea #1735. Vikunja P19 #432.

Problem

Nothing re-pins a stack whose units are root's. r6 designed case1-repin.sh for .53 alone. And nothing tells ops-01 what to put on a fresh host before install.

Owner and reviewer

• Owner: Darkwing. Fallback author: Herb (mos-1118, mos-1119). If I1 isn't approved by 10-23, Herb takes I3 from this brief with no new briefing.
• Reviewer: Filbert (moved from Duckworth, mos-1119 item 4).
• After: I1 lands. It calls I2's render, so its tests use I2's candidate or a stand-in with the same arguments.

Files owned

scripts/stack-upgrade.sh (new), packages/cli/tests/stack-upgrade.test.mjs (new), docs/guides/host-run-sheet.md (new), and the docs/TOOLS.md lines for the new script.

What ships: stack-upgrade.sh

Root runs it from a root-owned checkout of the reviewed commit, never from a copy, because the stack's user can write the copies.

• stack-upgrade.sh upgrade --instance NAME --bundle FILE --sha SHA [--migrate] [--timeout S]
• stack-upgrade.sh rollback --instance NAME --to ID [--restore ACT --events N] [--timeout S]

The steps are r6's activate A1-A5, A2f and R1-R4 (r6 :459-544), and r6's rollback (:545-559), with these changes:

• The unit render is I2's, called with the values from /etc/mosaic-stack/<instance>.args, which I2's install writes. --copy ID is the one value upgrade changes. Upgrade never edits that file except to record the new copy id after a confirmed run.
• Before A1, copy --bundle FILE SHA runs as the stack's user (I1), and the copy id is SHA.
• A rehearsal runs before A2, as the stack's user: stack-release.sh rehearse SHA BUSINESS --instance NAME against the last activation's backup. A failed rehearsal refuses with nothing stopped.
• The root lock (sage-1524 finding, mos-1118). --plan skips lock() (scripts/stack-release.sh:1046, :1058), so two root runs could both pass A1, and one could stop the units in the middle of the other. stack-upgrade.sh takes flock -n on /run/mosaic-stack/upgrade-<instance>.lock before A1 and holds it to the end of the run, rollback included. A second run refuses upgrade-running with exit 3.
• A1 also refuses open-activation when any AWAITING-UNITS file exists or the interrupted set isn't empty, before anything stops.
• A migrating target refuses migrate-required unless --migrate is given (decision 96 stays process).
• Root's step log is /var/log/mosaic-stack/upgrade-<instance>.log, root:root 0640, one line per step (r6 :452-455).
• Exit codes are r6's: 0 ok, 1 failed with the old stack running, 2 rollback-failed with the units stopped, 3 refused with nothing changed.
• duckworth-0055's notes on r6 apply here. Darkwing names how each one lands in the design or BUILD.md.
• Root never parses output the stack's user wrote, beyond exit codes (r6 section 4's rule).

What ships: the host run sheet

docs/guides/host-run-sheet.md is what ops-01 does on a fresh host before install, and once per host for enrollment. Every item names the precheck line that verifies it (I2). Where a precheck line is missing, I3 says so and Gosalyn adds it to I2.

• The stack user and its home (useradd, /srv/mosaic), with no sudo and no supplementary group.
• node pinned under /opt, with the gpg and sha256 checks.
• claude for the factory profile, installed by root with npm install --prefix /opt/mosaic-stack/claude-<pin>, plus managed settings with the autoupdater off. This replaces r6's /srv/mosaic/.local/bin (D :124), per C2-13a's --claude rule.
• bubblewrap, the AppArmor profile, dpkg-statoverride, the no-new-privs test, the pins, hostcheck, the apt hook and the boot unit (r6). These are for any host that runs bwrap, so .54 gets them too.
• The release transport: the bundle file, root's checkout made from it, and the stack user's own clone of the bundle under its home (I1). Upgrade fetches each new bundle into both.
• The machine-grant enrollment step, once per host (Gizmoduck's design r1 f6b6a04b6 section 3, as changed by its review and mos-1120). Mos sent Q-O1 to ops-01 (mos-1121). Until that design is approved, this item names the design and its open question and has no commands.
• The credential slots Jason places, in one table per host, so he can do them in one sitting (mos-1118). Each row gives the host, the absolute path, the file name, the mode, the owner, which profile needs it and which precheck line checks it. Values never appear. The known slots, which the author checks against the code and completes:
  • the model credential, <configDir>/secrets/claude-session.env (Q185 for the factory, ZAI_API_KEY for the library, C2-13a precheck)
  • the library Console's proof file under <home>/.config/mosaic-dev/secrets/ (C2-13a, library only)
  • the library's Gitea token slot (C2-13b)
  • the per-agent store under <configDir>/secrets/agents/<business>/<agent>/ (row 199 D1), which Jason fills only through mosaic credentials issue or, after #436, Approve
  • the installer key under <configDir>/secrets/hosts/<business>/<hostId>/installer (mos-1120), made by enrollment, not placed by hand
• Console start: none by hand. I2 enables the Console unit (Q189).

The run sheet is for ops-01 and Jason, so it uses commands and paths, not design prose.

Tests

stack-upgrade.test.mjs runs the driver as the test's user against a scratch root, with fakes for systemctl, flock's lock dir, I2's render and sudo (the fake execs the real I1 verbs as the test's user, r6's B1 pattern). r6's sequences 1-8 run through the driver, plus:

• a second run while the first holds the lock refuses upgrade-running and changes nothing
• an open AWAITING-UNITS refuses at A1 with no stop
• a migrating target without --migrate refuses migrate-required
• a failed rehearsal refuses with no stop
• a confirmed upgrade records the new copy id in the args file, and a failed one leaves it as it was
• the step log has one line per step, and nothing in it comes from the stack user's output

Mutants: drop the flock, release the lock before rollback ends, skip A1's open-activation check, skip the rehearsal, accept --migrate's absence, and write the args file before confirm.

Out of scope

Running it on a host. The ops-01 PVE proof (install, upgrade to N+1, rollback) is a window with Mos's go. Q187 item 4 covers creating that one throwaway VM, which stays after the proof (mos-1119 item 1). Folding host prep into the command is #437.

Gate

Filbert approves the manifest. The test file, the cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. Target: in review 10-24, landed 10-25 (sage-1528 line 12).

I4 hostcheck: the bwrap profile check, pins and boot unit

Sage, 2026-10-21, split out of I3 (darkwing row 233 G2, sage-1650). Owner Darkwing, reviewer Filbert, gate owner Sage. Darkwing builds it right after I3, and it must land before the ops-01 proof.

Problem

The host run sheet (I3, line 182 above) lists r6's hostcheck among the host items for any host that runs bwrap, and I3's A1 refuses profile-drift unless /usr/local/lib/mosaic-stack/hostcheck.sh --check exits 0. No hostcheck exists on refactor. It's r6 B2 (row 195 DESIGN.md at b7239bb66, :262-303 and :839-860), sized at 150-200 lines of bash and 350-450 of test. That's work separate from the upgrade sequence: the root-side bwrap stop and a network fetch.

What ships

r6 B2 as designed:

• case1-pins.sh holds the pins: the profile sha256, the apparmor-profiles version and the bubblewrap pin.
• hostcheck.sh covers the seven items in three modes:
  • --check exits 0 or 1.
  • --enforce stops bwrap: apparmor_parser -R, then root:root 0700 on /usr/bin/bwrap. It writes /etc/mosaic-stack/DRIFT and a hostcheck.log line, and exits 0.
  • --upstream compares the current apparmor-profiles deb to the pin.
• mosaic-hostcheck.service, a boot oneshot.
• case1-hostcheck.test.mjs uses fakes for every host tool and covers r6's seven case groups and seven mutants.

The I3 run sheet's hostcheck item points at this row. I3's A1 and its fake don't change.

Carried from the I3 review (Filbert 29922 on #1735, sage-1727)

Row 233 lands with three non-blocking notes that must be fixed before ops-01. This row takes them, and that widens its paths to scripts/stack-upgrade.sh and packages/cli/tests/stack-upgrade.test.mjs:

• N1: after A5 confirms, a failed rename of the new args file calls finish 1 args-failed while the new copy is live (stack-upgrade.sh:280-282 at 1a1dc80ca). TOOLS.md and the run sheet define exit 1 as "the old copy runs". Give this case its own exit code, add it to the exit table in TOOLS.md and the run sheet, and add a test that makes the rename fail after A5 and expects the new code.
• N4: run sheet section 3 doesn't put the pinned node's npm on root's PATH and doesn't name the --claude path. The npm symlink node_modules/.bin/claude is what case2-unit.sh:140 refuses. The sheet must name the path the check accepts.
• N5: run sheet sections 8 and 10 get their precheck lines.

Filbert's N2 and N3 (no $REPO check, no signal handling between A2 and A5) are in DEFERRED.md, not here.

Gate

Filbert approves the manifest. The test file, the cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. Target: in review 10-25, landed 10-26, before the ops-01 proof.