stack-upgrade: root's upgrade and rollback driver for root-owned units, with the host run sheet; stack-release --backup last and the external plan refusals (row 233, #1735, installer I3)

This commit is contained in:
jason.woltje committed 2026-10-20 23:00:36 -05:00
1 parent e65a3ab4d2
commit aacfa2d7a4
7 files changed
+1397 -4

No files matched your search

+82 -2
View File
@@ -669,7 +669,7 @@ pinned copy](../packages/cli/README.md#one-pinned-copy-row-85-1579)).
#### Rehearsal on a scratch root (D2, row 163)
```
scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]
scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH|last --port N [--timeout S]
```
`rehearse` starts the target copy's bus host and Console on a copy of a
@@ -680,7 +680,11 @@ runs it against live data. It stops and starts no unit and writes no pointer
or log line.
`--backup` takes an activation id under `<dataRoot>/backups/stack/` or an
absolute path. `--port` is the scratch Console's loopback port, 1024 to
absolute path. `--backup last` (row 233) takes the newest backup the
activation log records and prints `PASS backup-last` naming it. With none in
the log it fails `no-backup`, exits 3 and makes no directory. After the
backup's checks it prints `BACKUP_STORE <activation id> schema <version>`.
`--port` is the scratch Console's loopback port, 1024 to
65535. The run refuses `port-reserved` for 7332, 7333 and the pointer's
`--port` and `--tester-port`, and `port-in-use` for a port something holds;
it never picks another port. `--timeout` (default 60 seconds) bounds the
@@ -779,6 +783,82 @@ instance (the business's `launch.by`), for this business, whose run started at
or after the floor. `status` adds an
`awaiting units:` line for an open activation.
Under `--units external`, `activate --plan` also refuses `open-activation`
while any rollback set holds `AWAITING-UNITS`, and `migrate-required` when
the target migrates the store and the run lacks `--migrate` (row 233). Under
`--units user` the plan's bytes are unchanged.
#### Root's upgrade driver (`scripts/stack-upgrade.sh`, row 233)
```
scripts/stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
[--rehearse-port N] [--timeout S]
scripts/stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
```
Root runs it from a root-owned checkout of the reviewed commit, never from a
copy, because the stack's user can write the copies. Each run needs Mos's go,
and `--migrate` needs Jason present (decision 96). The host must be prepared
first (`docs/guides/host-run-sheet.md`). Only the default instance, `dev`, is
accepted; a named instance's units are a later row.
The unit values come from `/etc/mosaic-stack/dev.args`, which I2's install
writes: one `case2-unit.sh` argument per line, the serve flags after `--`,
owned by root with no group or other write, exactly one `--copy`, and that
copy the one both installed units run. Otherwise the run refuses
`args-unreadable` or `args-differ`. The driver writes the new copy id into
that file only after `confirm` passes.
`upgrade`, in order. Every stack user step runs as
`sudo -u USER -H env PATH=<node dir>:/usr/local/bin:/usr/bin:/bin` from the
user's clone `<home>/src/mosaic-stack`:
1. Takes `flock -n` on `/run/mosaic-stack/upgrade-dev.lock` and holds it to
the end, rollback included. A second run refuses `upgrade-running`, and
a lock it can't open refuses `lock-failed`.
2. Fetches the bundle's `refactor` into the clone and fast-forwards it to
SHA (`fetch-failed`), then `stack-copy.sh copy --bundle FILE SHA`
(`copy-failed`).
3. A1: this checkout's HEAD is SHA (`head-mismatch`), the copy exists
(`copy-missing`), `/usr/local/lib/mosaic-stack/hostcheck.sh --check`
exits 0 (`profile-drift`, row 240), both units name one copy
(`units-unreadable`, `units-disagree`) and it is the args file's
(`args-differ`), and
`activate SHA BUSINESS --units external --plan --expect-from OLD` exits 0
(`plan-refused`, which covers `open-activation`, `migrate-required` and
`pointer-differs`).
4. The rehearsal, `rehearse SHA BUSINESS --backup last --port 7339`
(`--rehearse-port`). Exit 0 goes on. Exit 3 means no backup in the log
and goes on only with `--no-rehearsal`, logging `rehearsal-skipped`;
without the flag it refuses `no-backup`. Exit 0 with `--no-rehearsal`
refuses `rehearsal-required`; any other exit refuses `rehearsal-failed`.
5. A2 stops the Console, then the bus, and waits for both to read inactive
or failed. A3 is `activate --units external`, A4 the render with the new
`--copy` (which also starts the bus, then the Console), A5 `confirm`.
The driver writes the new args file beside the old one before A2
(`args-failed`, exit 3) and moves it into place after A5. A failed move
exits 1 `args-failed`: the new copy is live and confirmed, and root puts
the file in place by hand.
A failure in A2 or a stage that exits 3 starts the old units again and gates
them (A2f). A later failure stops both units, runs `revert`, renders the old
copy and runs `confirm --rollback OLD --cause A3|A4|A5`. `rollback` is the
same from A1 on, with `rollback BUSINESS --to ID`, no fetch, no copy and no
rehearsal; `--restore` and `--events` pass through.
Root reads no output the stack user wrote except exit codes and the
rehearsal's `BACKUP_STORE` line, which it copies into its log after a match
against a fixed pattern (`BACKUP_STORE none` otherwise). The step log
`/var/log/mosaic-stack/upgrade-dev.log`, 0640, gets one
`<time> <step> rc=N argv: ...` line per step and a last
`<time> result exit=N <code> <text>` line.
Exit 0 ok, 1 failed with the old stack running and gated, 2 rollback-failed
with both units stopped, 3 refused with nothing stopped, 4 usage. A refusal
after step 2 leaves the user's clone fast-forwarded and the new copy made. After exit
2 the activation stays open, so the next run's plan refuses until root
finishes it.
## Console service (`scripts/webui-service.sh`)
```bash
+240
View File
@@ -0,0 +1,240 @@
# Host run sheet: preparing a host for the stack
ops-01 runs this on a fresh host before install (I2), and once per host for enrollment. Jason places the
credentials in section 9. Brief: `docs/plans/2026-10-20_installer-rows.md`, I3 (row 233, #1735). The
host design is Darkwing's r6 (`agents/darkwing/work/queue-195/DESIGN.md`, section 2) as the brief
changes it.
The hosts are .53 (profile `factory`) and .54 (profile `library`). Both get every section unless a
section says otherwise. Every command runs as root unless it starts with `sudo -u mosaic`.
Each item names the `scripts/case2-precheck.sh` line that verifies it. Install's precheck runs as the
stack's user with `--business ID --profile factory|library`. Where no precheck line checks an item, the
item says "precheck line missing", and Gosalyn adds the line to I2. Today's precheck (C2-13a) has these
gaps; C2-13b adds the library lines (`library.sync`, persona modes, no Discord binding).
## 1. The stack's user and its home
```sh
useradd --system --user-group --home-dir /srv/mosaic --create-home --shell /usr/sbin/nologin mosaic
chmod 0700 /srv/mosaic
install -d -o root -g root -m 0755 /etc/mosaic-stack
install -d -o root -g root -m 0750 /var/log/mosaic-stack
```
`mosaic` gets no sudo rule and no supplementary group. Check with `sudo -l -U mosaic` (expect "not allowed
to run sudo") and `id -G mosaic` (one number).
Precheck lines:
- `ok HOME is /srv/mosaic`
- `ok sudo -n true fails as mosaic`
- `ok id -G prints one group (<gid>)`
- `/srv/mosaic` at 0700: precheck line missing.
## 2. node, pinned under /opt
The pin is the version and sha256 the window names. `<v>` below is that version, for example `v24.x.y`.
```sh
install -d -o root -g root -m 0755 /etc/mosaic-stack/node-keys /opt/mosaic-stack
cd "$(mktemp -d)"
curl -fsSLO https://nodejs.org/dist/<v>/node-<v>-linux-x64.tar.xz
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt.asc
gpg --no-default-keyring --keyring /etc/mosaic-stack/node-keys/pubring.kbx --verify SHASUMS256.txt.asc SHASUMS256.txt
grep " node-<v>-linux-x64.tar.xz\$" SHASUMS256.txt | sha256sum -c -
echo "<pinned sha256> node-<v>-linux-x64.tar.xz" | sha256sum -c -
tar -xJf node-<v>-linux-x64.tar.xz --no-same-owner
mv node-<v>-linux-x64 /opt/mosaic-stack/node-<v>
chown -R root:root /opt/mosaic-stack/node-<v>
chmod -R go-w /opt/mosaic-stack/node-<v>
```
The keyring holds the Node release keys from the nodejs/release-keys repository, nothing else. Stop on any
failed check; don't unpack. Rollback: `rm -r /opt/mosaic-stack/node-<v>`.
Precheck lines:
- `ok node <v>` (version 24 or later only).
- The `--node` path in the args file must be a root-owned regular file outside the home. `case2-unit.sh`
refuses otherwise at render, which is not a precheck line.
- The installed version equals the pin: precheck line missing.
## 3. claude, factory only (.53)
Root installs it under `/opt`, never under the stack's home (this replaces r6's `/srv/mosaic/.local/bin`).
The pin is 2.1.304 unless the window names a newer one.
```sh
install -d -o root -g root -m 0755 /opt/mosaic-stack/claude-<pin> /etc/claude-code
npm install --prefix /opt/mosaic-stack/claude-<pin> @anthropic-ai/claude-code@<pin>
chown -R root:root /opt/mosaic-stack/claude-<pin>
printf '%s\n' '{"env":{"DISABLE_AUTOUPDATER":"1"}}' > /etc/claude-code/managed-settings.json
chown root:root /etc/claude-code/managed-settings.json
chmod 0644 /etc/claude-code/managed-settings.json
```
Rollback: `rm -r /opt/mosaic-stack/claude-<pin> /etc/claude-code/managed-settings.json`.
The args file's `--claude` names the binary under `/opt/mosaic-stack/claude-<pin>`. `case2-unit.sh`
refuses one that isn't a root-owned regular file outside the home.
Precheck lines:
- `ok claude replaces its prompt with --system-prompt-file (<n> input tokens)`. It runs the claude the bus
unit's `MOSAIC_CLAUDE_BIN` names, so it needs the model credential from section 9.
- Managed settings present with the autoupdater off: precheck line missing.
## 4. bubblewrap, AppArmor and the drift stop
Both hosts. These are r6's items, unchanged.
```sh
apt-get install bubblewrap=0.9.0-1ubuntu0.3
stat -c '%U:%G %a' /usr/bin/bwrap > /etc/mosaic-stack/bwrap.orig # expect root:root 755
cd "$(mktemp -d)"
apt-get download apparmor-profiles=4.0.1really4.0.1-0ubuntu0.24.04.9
dpkg-deb --fsys-tarfile apparmor-profiles_*.deb | tar -x ./usr/share/apparmor/extra-profiles/bwrap-userns-restrict
echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9 usr/share/apparmor/extra-profiles/bwrap-userns-restrict" | sha256sum -c -
install -o root -g root -m 0644 usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/bwrap-userns-restrict
apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict
dpkg-statoverride --update --add root mosaic 0750 /usr/bin/bwrap
```
`aa-status` then lists `bwrap` and `unpriv_bwrap` in enforce mode. Install the one profile file, never the
whole `apparmor-profiles` package, whose 20 other profiles would load at the next boot. Rollback, in
reverse order: `dpkg-statoverride --remove /usr/bin/bwrap`, then `chown` and `chmod` to the values in
`bwrap.orig`; `apparmor_parser -R /etc/apparmor.d/bwrap-userns-restrict`, then `rm` it;
`apt-get remove bubblewrap`.
The no-new-privs test must print `bwrap//&unpriv_bwrap (enforce)`:
```sh
systemd-run --wait --pipe --uid=mosaic --gid=mosaic -p NoNewPrivileges=yes -p ProtectSystem=strict \
-p ProtectHome=yes -p PrivateTmp=yes \
/usr/bin/bwrap --unshare-user --unshare-pid --ro-bind / / cat /proc/self/attr/current
```
The P9 checks: another user's `/usr/bin/bwrap --version` fails with Permission denied, `getent group mosaic`
lists no member, and `sysctl fs.protected_hardlinks` prints 1. `sysctl
kernel.apparmor_restrict_unprivileged_userns` prints 1 too (Ubuntu 24.04's default); leave it, so nothing
else on the host gains user namespaces.
The drift stop is row 240's (I4, #1746): `hostcheck.sh` and its pins in `/usr/local/lib/mosaic-stack/`, the
apt hook `/etc/apt/apt.conf.d/80mosaic-hostcheck`
(`DPkg::Post-Invoke { "/usr/local/lib/mosaic-stack/hostcheck.sh --enforce || true"; };`) and the boot unit
`mosaic-hostcheck.service`. Install them from row 240's run sheet lines once it lands. Until then
`stack-upgrade.sh` refuses `profile-drift` on every run, because it runs `hostcheck.sh --check` and a
missing file refuses.
Precheck lines:
- `ok bwrap --unshare-all runs`
- The AppArmor profile, the statoverride, the no-new-privs result and the P9 checks: precheck line missing.
A line that runs `/usr/local/lib/mosaic-stack/hostcheck.sh --check` would cover them once row 240 lands.
## 5. The port floor
Leave `net.ipv4.ip_unprivileged_port_start` at its default (1024) or higher. The Console's port is below
it and the unit gets `CAP_NET_BIND_SERVICE`, so no other `mosaic` process can bind that port.
Precheck line: `ok net.ipv4.ip_unprivileged_port_start <n> is above the Console's port <port>`.
## 6. The release transport
A release is a git bundle of `refs/heads/refactor` plus the 40-character commit id (I1, mos-1118 choice 2).
Mos's go names both, with the bundle's sha256. `<sha>` below is that id.
First install:
```sh
# Copy the bundle in, then check it before anything reads it.
echo "<bundle sha256> /root/<sha>.bundle" | sha256sum -c -
git bundle verify /root/<sha>.bundle
# Root's checkout. stack-upgrade.sh and case2-unit.sh run from here, never from a copy.
git clone -q -b refactor /root/<sha>.bundle /root/mosaic-stack
git -C /root/mosaic-stack checkout -q --detach <sha>
git -C /root/mosaic-stack rev-parse HEAD # must print <sha>
# The stack's user's copy of the bundle, and its own clone.
install -d -o mosaic -g mosaic -m 0700 /srv/mosaic/incoming /srv/mosaic/src
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
sudo -u mosaic -H git clone -q -b refactor /srv/mosaic/incoming/<sha>.bundle /srv/mosaic/src/mosaic-stack
```
Each upgrade, before `stack-upgrade.sh upgrade`:
```sh
echo "<bundle sha256> /root/<sha>.bundle" | sha256sum -c -
git bundle verify /root/<sha>.bundle
git -C /root/mosaic-stack fetch -q /root/<sha>.bundle refs/heads/refactor:refs/remotes/origin/refactor
git -C /root/mosaic-stack checkout -q --detach <sha>
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
/root/mosaic-stack/scripts/stack-upgrade.sh upgrade --instance dev --bundle /srv/mosaic/incoming/<sha>.bundle --sha <sha>
```
The driver fetches the bundle into the stack's user's clone and makes the copy itself. It refuses
`head-mismatch` when root's checkout isn't at `<sha>`. Usage, exits and the step log are in
`docs/TOOLS.md`, under "Root's upgrade driver". Each run needs Mos's go, and `--migrate` needs Jason present
(decision 96).
When a run doesn't exit 0, send Mos the exit and the last lines of `/var/log/mosaic-stack/upgrade-dev.log`:
- Exit 3: refused before anything stopped. The last line names the code. The stack's user's clone and the
new copy may already hold the new commit; the units still run the old one.
- Exit 1: the old copy runs and passed its gate. Either the driver restarted it after A2 or a refused A3
(A2f), or it rolled back after A3, A4 or A5.
- Exit 2: both units are stopped and the activation stays open, so the next run refuses at its plan until
it is finished with `confirm` or `revert`. One way here: a unit that doesn't stop within `--timeout`
keeps its old start time, A2f's gate fails it `stale-start`, and the driver stops both (duckworth-0055
N1). That is on purpose; a unit that won't stop is a reason to stop.
Precheck lines: none. Root's checkout at `<sha>` and the stack's user's clone: precheck line missing.
After install, `stack-copy.sh check --units external` reads the copy the units run.
## 7. The args file
`/etc/mosaic-stack/dev.args` holds the `case2-unit.sh` arguments, one per line, with the serve flags after a
line holding `--`. I2's install writes it; ops-01 doesn't write it by hand. It must be root's, with no group
or other write (`chmod 0644` or tighter). Before `--` it takes only `--user`, `--home`, `--business`,
`--copy` (exactly once), `--node`, `--profile`, `--claude` and `--pm`. `stack-upgrade.sh` changes only the
`--copy` line, and only after confirm passes.
Precheck line: none. `stack-upgrade.sh` refuses `args-unreadable` on any of these rules.
## 8. Enrollment, once per host
The machine-grant enrollment is Gizmoduck's design r6, commit f9039682e, section 3, steps E1-E4. ops-01's
Q-O1 is answered (mos-1125). The host side is the `tools/host/` files: `mosaic-keys`, `mosaic-keys-write`,
their sudoers line and the sshd drop-in `/etc/ssh/sshd_config.d/<nn>-mosaic-keys.conf`. The verbs that run
it (`hosts.enroll`, the `host-enroll` decision) are rows 234 and 235, not yet on refactor. This item has no
commands until they land.
## 9. Credential slots
Jason places these, all in one sitting per host. No value appears here or in any message. Check a placed
file with `stat -c '%U %a %n' PATH`, never `cat`. `<business>` is the host's business id.
### .53, factory
| Path | File | Mode | Owner | Profile | Precheck line |
|---|---|---|---|---|---|
| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `CLAUDE_CODE_OAUTH_TOKEN=...` (Q185) | 0600 | mosaic | factory | `claude replaces its prompt with --system-prompt-file` runs claude with it. The file's mode and owner: precheck line missing. |
| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
### .54, library
| Path | File | Mode | Owner | Profile | Precheck line |
|---|---|---|---|---|---|
| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `ZAI_API_KEY=...` | 0600 | mosaic | library | `<path> sets ZAI_API_KEY` |
| `/srv/mosaic/.config/mosaic-dev/secrets/` | the Console's proof file, the path the args file's `--proof-file` names | 0600 or 0400 | mosaic | library | none. `case2-unit.sh` refuses at render. Precheck line missing. |
| `/srv/mosaic/.config/mosaic-dev/secrets/<business>/` | `library-gitea` | 0600 | mosaic | library | C2-13b's `library.sync` (not landed) |
| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
The installer key, `<configDir>/secrets/hosts/<business>/<hostId>/installer`, is not a slot anyone places.
Enrollment (section 8) writes it under the enrolling stack's config directory and deletes it when the host
leaves.
## 10. Console start
None by hand. I2's `case2-unit.sh` enables both units, starts the bus, then the Console (Q189, mos-1122).
`stack-upgrade.sh` stops and starts both units itself.