Files
stack/docs/guides/host-run-sheet.md
jason.woltje 6415e81ee7 docs(run-sheet): check config directory ancestors before accounts init (#1873)
mosaic accounts init refuses accounts-unsafe since #1919 when a folder above the
config directory is writable by another user. Step 3 now gives the read-only check.
2026-11-10 23:43:55 -06:00

34 KiB

Host run sheet: preparing a host for the stack

ops-01 runs this on a fresh host before install (I2), and once per host for enrollment. Jason places the credentials in section 9. Brief: docs/plans/2026-10-20_installer-rows.md, I3 (row 233, #1735). The host design is Darkwing's r6 (agents/darkwing/work/queue-195/DESIGN.md, section 2) as the brief changes it.

The hosts are .53 (profile factory) and .54 (profile library). Both get every section unless a section says otherwise. Every command runs as root unless it starts with sudo -u mosaic.

Each item names the scripts/case2-precheck.sh line that verifies it. Install's precheck runs as the stack's user with --business ID --profile factory|library. Where no precheck line checks an item, the item says "precheck line missing", and Gosalyn adds the line to I2. Today's precheck (C2-13a) has these gaps; C2-13b adds the library lines (library.sync, persona modes, no Discord binding).

1. The stack's user and its home

useradd --system --user-group --home-dir /srv/mosaic --create-home --shell /usr/sbin/nologin mosaic
chmod 0700 /srv/mosaic
install -d -o root -g root -m 0755 /etc/mosaic-stack
install -d -o root -g root -m 0750 /var/log/mosaic-stack

mosaic gets no sudo rule and no supplementary group. Check with sudo -l -U mosaic (expect "not allowed to run sudo") and id -G mosaic (one number).

Precheck lines:

  • ok HOME is /srv/mosaic
  • ok sudo -n true fails as mosaic
  • ok id -G prints one group (<gid>)
  • /srv/mosaic at 0700: precheck line missing.

2. node, pinned under /opt

The pin is the version and sha256 the window names. <v> below is that version, for example v24.x.y.

install -d -o root -g root -m 0755 /etc/mosaic-stack/node-keys /opt/mosaic-stack
cd "$(mktemp -d)"
curl -fsSLO https://nodejs.org/dist/<v>/node-<v>-linux-x64.tar.xz
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt.sig
gpg --no-default-keyring --keyring /etc/mosaic-stack/node-keys/pubring.kbx --verify SHASUMS256.txt.sig SHASUMS256.txt
grep " node-<v>-linux-x64.tar.xz\$" SHASUMS256.txt | sha256sum -c -
echo "<pinned sha256>  node-<v>-linux-x64.tar.xz" | sha256sum -c -
tar -xJf node-<v>-linux-x64.tar.xz --no-same-owner
mv node-<v>-linux-x64 /opt/mosaic-stack/node-<v>
chown -R root:root /opt/mosaic-stack/node-<v>
chmod -R go-w /opt/mosaic-stack/node-<v>

The keyring holds the Node release keys from the nodejs/release-keys repository, nothing else. Stop on any failed check; don't unpack. Rollback: rm -r /opt/mosaic-stack/node-<v>.

Precheck lines:

  • ok node <v> (version 24 or later only).
  • The --node path in the args file must be a root-owned regular file outside the home. case2-unit.sh refuses otherwise at render, which is not a precheck line.
  • The installed version equals the pin: precheck line missing.

3. claude, factory only (.53)

Root installs it under /opt, never under the stack's home (this replaces r6's /srv/mosaic/.local/bin). The pin is 2.1.304 unless the window names a newer one.

install -d -o root -g root -m 0755 /opt/mosaic-stack/claude-<pin> /etc/claude-code
PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  npm install --prefix /opt/mosaic-stack/claude-<pin> @anthropic-ai/claude-code@<pin>
chown -R root:root /opt/mosaic-stack/claude-<pin>
/opt/mosaic-stack/claude-<pin>/node_modules/@anthropic-ai/claude-code/bin/claude.exe --version
printf '%s\n' '{"env":{"DISABLE_AUTOUPDATER":"1"}}' > /etc/claude-code/managed-settings.json
chown root:root /etc/claude-code/managed-settings.json
chmod 0644 /etc/claude-code/managed-settings.json

Rollback: rm -r /opt/mosaic-stack/claude-<pin> /etc/claude-code/managed-settings.json.

Root's PATH has no npm, so the install line puts section 2's pinned node first; the package's postinstall also runs node. The --version line must print <pin> (Claude Code). Anything else means the postinstall left its 500-byte stub in bin/claude.exe (no native package for the platform, or scripts were off): stop and roll back.

The args file's --claude is /opt/mosaic-stack/claude-<pin>/node_modules/@anthropic-ai/claude-code/bin/claude.exe, the regular file the postinstall hardlinks or copies the native binary to. Not node_modules/.bin/claude: that is npm's symlink to it, and case2-unit.sh refuses a symlink, as it does anything that isn't a root-owned regular file outside the home. Checked against the 2.1.304 package's install.cjs; a newer pin needs the same check.

Precheck lines:

  • ok claude replaces its prompt with --system-prompt-file (<n> input tokens). It runs the claude the bus unit's MOSAIC_CLAUDE_BIN names, so it needs the model credential from section 9.
  • Managed settings present with the autoupdater off: precheck line missing.

4. bubblewrap, AppArmor and the drift stop

Both hosts. These are r6's items, unchanged.

apt-get install bubblewrap=0.9.0-1ubuntu0.3
stat -c '%U:%G %a' /usr/bin/bwrap > /etc/mosaic-stack/bwrap.orig    # expect root:root 755

cd "$(mktemp -d)"
apt-get download apparmor-profiles=4.0.1really4.0.1-0ubuntu0.24.04.9
dpkg-deb --fsys-tarfile apparmor-profiles_*.deb | tar -x ./usr/share/apparmor/extra-profiles/bwrap-userns-restrict
echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9  usr/share/apparmor/extra-profiles/bwrap-userns-restrict" | sha256sum -c -
install -o root -g root -m 0644 usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/bwrap-userns-restrict
echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9  /etc/apparmor.d/bwrap-userns-restrict" | sha256sum -c -
apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict

dpkg-statoverride --update --add root mosaic 0750 /usr/bin/bwrap

Run as root in a mktemp -d dir, apt-get download prints a line starting W: Download is performed unsandboxed as root, because the _apt user can't write that dir. The line is expected and the download still succeeds. Each sha256sum -c - line must print OK. The second one checks the copy in /etc/apparmor.d, the file that loads. Stop on any other result.

aa-status then lists bwrap and unpriv_bwrap in enforce mode. Install the one profile file, never the whole apparmor-profiles package, whose 20 other profiles would load at the next boot. Rollback, in reverse order: dpkg-statoverride --remove /usr/bin/bwrap, then chown and chmod to the values in bwrap.orig; apparmor_parser -R /etc/apparmor.d/bwrap-userns-restrict, then rm it; apt-get remove bubblewrap.

The no-new-privs test must print bwrap//&unpriv_bwrap (enforce):

systemd-run --wait --pipe --uid=mosaic --gid=mosaic -p NoNewPrivileges=yes -p ProtectSystem=strict \
  -p ProtectHome=yes -p PrivateTmp=yes \
  /usr/bin/bwrap --unshare-user --unshare-pid --ro-bind / / cat /proc/self/attr/current

The P9 checks: another user's /usr/bin/bwrap --version fails with Permission denied, getent group mosaic lists no member, and sysctl fs.protected_hardlinks prints 1. sysctl kernel.apparmor_restrict_unprivileged_userns prints 1 too (Ubuntu 24.04's default); leave it, so nothing else on the host gains user namespaces.

The drift stop (row 240, I4, #1746) goes in last, after the profile and the statoverride. As root, from the root-owned checkout of the reviewed commit:

install -d -o root -g root -m 0755 /usr/local/lib/mosaic-stack
install -o root -g root -m 0755 scripts/case1-hostcheck.sh /usr/local/lib/mosaic-stack/hostcheck.sh
install -o root -g root -m 0644 scripts/case1-pins.sh /usr/local/lib/mosaic-stack/case1-pins.sh
printf '%s\n' 'DPkg::Post-Invoke { "/usr/local/lib/mosaic-stack/hostcheck.sh --enforce || true"; };' \
  > /etc/apt/apt.conf.d/80mosaic-hostcheck
install -o root -g root -m 0644 packages/cli/systemd/mosaic-hostcheck.service /etc/systemd/system/
systemctl daemon-reload && systemctl enable mosaic-hostcheck.service
/usr/local/lib/mosaic-stack/hostcheck.sh --check

--check prints one ok line for each of its seven items and exits 0. Item 7 reads 1023 as the port until I2 installs the Console unit (section 10), then the unit's --port. stack-upgrade.sh runs --check before its plan and refuses profile-drift on any exit but 0, a missing file included. Rollback: systemctl disable mosaic-hostcheck.service, then rm the unit, the hook file and /usr/local/lib/mosaic-stack.

On drift, --enforce (the apt hook after every package run, and the unit at boot) unloads the profile, closes /usr/bin/bwrap to root:root 0700, appends the failed items to /etc/mosaic-stack/DRIFT and logs a line in /var/log/mosaic-stack/hostcheck.log. Every role session then refuses to start. ops-01 reads the log in each STATE report and sends any DRIFT to Mos. Only root removes DRIFT, in a reviewed step with a new pin or a Mos go. After that, chown root:mosaic /usr/bin/bwrap, chmod 0750 /usr/bin/bwrap and apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict put bwrap back, and --check must exit 0.

Before each re-pin window, ops-01 runs /usr/local/lib/mosaic-stack/hostcheck.sh --upstream. It downloads the apparmor-profiles candidate and compares its bwrap-userns-restrict to the pin. A DIFFERS line goes to Mos as a new-pin request; it refuses nothing.

Precheck lines:

  • ok bwrap --unshare-all runs
  • The AppArmor profile, the statoverride, the no-new-privs result and the P9 checks: precheck line missing. /usr/local/lib/mosaic-stack/hostcheck.sh --check covers them (row 240).

5. The port floor

Leave net.ipv4.ip_unprivileged_port_start at its default (1024) or higher. The Console's port is below it and the unit gets CAP_NET_BIND_SERVICE, so no other mosaic process can bind that port.

Precheck line: ok net.ipv4.ip_unprivileged_port_start <n> is above the Console's port <port>.

6. The release transport

A release is a git bundle of refs/heads/refactor plus the 40-character commit id (I1, mos-1118 choice 2). Mos's go names both, with the bundle's sha256. <sha> below is that id.

First install:

# Copy the bundle in, then check it before anything reads it.
echo "<bundle sha256>  /root/<sha>.bundle" | sha256sum -c -
(
  bundle_check="$(mktemp -d)" || exit 1
  trap 'rm -rf "$bundle_check"' EXIT
  git init -q --bare "$bundle_check" &&
    git -C "$bundle_check" bundle verify /root/<sha>.bundle
) || echo "STOP: bundle verify failed; do not continue"

# Root's checkout. stack-upgrade.sh and case2-unit.sh run from here, never from a copy.
git clone -q -b refactor /root/<sha>.bundle /root/mosaic-stack
git -C /root/mosaic-stack checkout -q --detach <sha>
git -C /root/mosaic-stack rev-parse HEAD        # must print <sha>

# The stack's user's copy of the bundle, and its own clone.
install -d -o mosaic -g mosaic -m 0700 /srv/mosaic/incoming /srv/mosaic/src
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
sudo -u mosaic -H git clone -q -b refactor /srv/mosaic/incoming/<sha>.bundle /srv/mosaic/src/mosaic-stack

If the first install refuses, use Mos's retry path (mos-1565, ops-0291). Bootstrap can already have written /srv/mosaic/.config/mosaic-dev/config.json (scripts/case2-install.sh:174, scripts/bootstrap.sh:13, scripts/mosaic-config.mjs:412-424). The config dir can also hold businesses/, the business file, and library persona files and slot dirs (scripts/case2-install.sh:200-211, scripts/case2-persona-import.mjs:300-308). A validation refusal removes only the business file this run created; its directory remains (scripts/case2-install.sh:217-219). The copy may already exist under /srv/mosaic/.local/share/mosaic-stack/copies/<sha> (scripts/case2-install.sh:80, scripts/stack-copy.sh:68).

As root, move these three paths aside with a UTC suffix. Move, never delete, so their owner and mode stay with them (mos-1565). This is the default-instance retry for the paths in this sheet:

(
  cd / || exit 1
  retry_stamp="$(date -u +%Y%m%dT%H%M%SZ)" || exit 1
  for retry_path in /root/mosaic-stack /srv/mosaic/src/mosaic-stack /srv/mosaic/.config/mosaic-dev; do
    if [ -e "$retry_path" ] || [ -L "$retry_path" ]; then
      retry_saved="$retry_path.$retry_stamp"
      if [ -e "$retry_saved" ] || [ -L "$retry_saved" ]; then
        echo "STOP: retry destination already exists: $retry_saved" >&2
        exit 1
      fi
      mv -T -- "$retry_path" "$retry_saved" || exit 1
    fi
  done
) || echo "STOP: could not move the first-install paths aside; do not continue"

Then rerun the first-install block from its first line. Never use the upgrade fetch below on clones whose first install has not finished: it updates refs/remotes/origin/refactor, while copy requires the commit to be reachable from both that ref and refs/heads/refactor (scripts/stack-copy.sh:44-48; mos-1565). Recheck section 9 before starting units. If Jason already filled claude-session.env, the library proof file or library-gitea, have him re-place those files under the new config dir with section 9's owner and mode. Per-agent slots still use mosaic credentials issue or Approve, never a hand copy (section 9; mos-1565). The moved config dir preserves the earlier files.

After a first install, once both units are running (section 10), adopt the installed copy before any activate or upgrade. This step runs only on Mos's go, with Jason or Mos running it (scripts/stack-release.sh:14):

sudo -u mosaic -H env PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  /srv/mosaic/src/mosaic-stack/scripts/stack-release.sh adopt <business> --instance dev --units external

Each upgrade, before stack-upgrade.sh upgrade:

Only the first upgrade after adopt takes --no-rehearsal: append it to the upgrade command below. Adopt writes no backup, so omitting the flag then refuses no-backup. Subsequent upgrades omit it; when a backup exists, supplying it refuses rehearsal-required.

echo "<bundle sha256>  /root/<sha>.bundle" | sha256sum -c -
(
  bundle_check="$(mktemp -d)" || exit 1
  trap 'rm -rf "$bundle_check"' EXIT
  git init -q --bare "$bundle_check" &&
    git -C "$bundle_check" bundle verify /root/<sha>.bundle
) || echo "STOP: bundle verify failed; do not continue"
git -C /root/mosaic-stack fetch -q /root/<sha>.bundle refs/heads/refactor:refs/remotes/origin/refactor
git -C /root/mosaic-stack checkout -q --detach <sha>
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
/root/mosaic-stack/scripts/stack-upgrade.sh upgrade --instance dev --bundle /srv/mosaic/incoming/<sha>.bundle --sha <sha>

The driver fetches the bundle into the stack's user's clone and makes the copy itself. It refuses head-mismatch when root's checkout isn't at <sha>. Usage, exits and the step log are in docs/TOOLS.md, under "Root's upgrade driver". Each run needs Mos's go, and --migrate needs Jason present (decision 96).

When a run doesn't exit 0, send Mos the exit and the last lines of /var/log/mosaic-stack/upgrade-dev.log:

  • Exit 3: refused before anything stopped. The last line names the code. The stack's user's clone and the new copy may already hold the new commit; the units still run the old one.
  • Exit 1: the old copy runs and passed its gate. Either the driver restarted it after A2 or a refused A3 (A2f), or it rolled back after A3, A4 or A5.
  • Exit 2: both units are stopped and the activation stays open, so the next run refuses at its plan until it is finished with confirm or revert. One way here: a unit that doesn't stop within --timeout keeps its old start time, A2f's gate fails it stale-start, and the driver stops both (duckworth-0055 N1). That is on purpose; a unit that won't stop is a reason to stop.
  • Exit 5: args-failed. The new copy is live and passed confirm, but /etc/mosaic-stack/dev.args still names the old copy. The next run refuses args-differ, and a render from that file would put the old copy back. Move the file the result line names into place with mv -f. The two differ only in the --copy value.

To check a backup by hand before rollback or restore, run as the stack's user from its bus/ directory:

sudo -u mosaic -H /bin/sh -c \
  'cd "<dataRoot>/backups/stack/<activation>/bus" && sha256sum -c ../SHA256SUMS'

Expect bus.sqlite: OK; stop on a failed check. The set holds bus/bus.sqlite, and SHA256SUMS names bus.sqlite relative to bus/ (scripts/stack-release.sh:475-482). Running sha256sum -c SHA256SUMS from the backup set's directory looks for bus.sqlite one directory too high (ops-0292). Restore and rehearsal use the same bus/ form (scripts/stack-release.sh:502, scripts/stack-release.sh:2244).

After case2-install.sh succeeds, create the human CLI wrapper before running stack-copy.sh check --units external. Creating this wrapper is a separate live step requiring Mos's go (scripts/stack-copy.sh:3); the install does not create it. Run only on that go: The step runs under sudo's secure_path, so it sets its own PATH and umask.

sudo -u mosaic -H /bin/sh <<'MOSAIC_WRAPPER'
cd / || exit 1
umask 077
PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin; export PATH
mkdir -p -- "$HOME/.npm-global/bin" || exit 1
"$HOME/src/mosaic-stack/scripts/stack-copy.sh" wrapper <sha> --create
MOSAIC_WRAPPER

--create refuses an existing wrapper; stop and review it rather than replacing it. The wrapper must name the same <sha> copy as the units.

Precheck lines: none. Root's checkout at <sha> and the stack's user's clone: precheck line missing. After install, stack-copy.sh check --units external reads the copy the units run.

Before running case2-install.sh, check the exact file you will pass as --business-file. Each non-library projects key must equal the id in <root>/.mosaic/project.json, and that file must exist in the stack's user's clone. Library entries are skipped by validate (packages/organization/src/config-command.mjs:95). Run this read-only check as mosaic with section 2's pinned node:

sudo -u mosaic -H /opt/mosaic-stack/node-<v>/bin/node --input-type=module - '<business-file>' <<'MOSAIC_PROJECT_KEYS'
import { readFileSync } from 'node:fs';
import { join } from 'node:path';
try {
  const business = JSON.parse(readFileSync(process.argv[2], 'utf8'));
  const projects = Object.entries(business.projects);
  if (!projects.length) throw new Error('business file declares no projects');
  for (const [id, entry] of projects) {
    if (entry.kind === 'library') continue;
    const file = join(entry.root, '.mosaic', 'project.json');
    const project = JSON.parse(readFileSync(file, 'utf8'));
    if (project.id !== id)
      throw new Error(`project file ${file} has id ${project.id}, but business ${business.id} declares it as ${id}`);
    console.log(`ok project ${id}: ${file} has id ${project.id}`);
  }
} catch (error) {
  console.error(`STOP: ${error.message}; do not install`);
  process.exitCode = 1;
}
MOSAIC_PROJECT_KEYS

For .53 expect ok project stack: /srv/mosaic/src/mosaic-stack/.mosaic/project.json has id stack (.mosaic/project.json:3, docs/guides/case1-business.mosaic-stack.json:10-11). Any STOP or nonzero exit stops the install. Fix a mismatched business key, not the project id (mos-1563). The install otherwise reaches that refusal at step 7, after making the copy (packages/organization/src/config-command.mjs:103, scripts/case2-install.sh:178, scripts/case2-install.sh:217).

The stack's user's scripts/case2-install.sh (C2-13b, design 10 step 2b) makes the first copy with stack-copy.sh copy <sha>. The copy runs npm ci from package-lock.json, which installs the pinned harness, so the host needs to reach the npm registry during the install. The install runs no release.sh package or activate.

A missing .mosaic/project.json only warns: project <id>: no project file at <path>; it has no project variables. Validate reports that project absent and continues (packages/organization/src/config-command.mjs:97-101). An install can therefore pass step 7 without the project's variables. It captures validate's output and prints business <id> validates (scripts/case2-install.sh:217-221). Treat the missing-file warning as a stop, place the correct project file and repeat the check above before continuing (Duckworth 323 N2; scripts/case2-install.sh:217-221).

7. The args file

/etc/mosaic-stack/dev.args holds the case2-unit.sh arguments, one per line, with the serve flags after a line holding --. Root writes it at the unit render, from the values the go names. The stack's user's install never writes it: that user can't write /etc, and root reads nothing that user wrote. It must be root's, with no group or other write (chmod 0644 or tighter). Before -- it takes only --user, --home, --business, --copy (exactly once), --node, --profile, --claude, --pm and --proxy. stack-upgrade.sh changes only the --copy line, and only after confirm passes.

For a hub installed with --proxy, keep that bare flag on its own line before --; it takes no value. Upgrade and rollback require the proxy, bus and Console units to name the same copy. They stop the Console, then the bus, then the proxy, re-render all three onto the selected copy, and start the proxy, then the bus, then the Console. Recovery after a failed upgrade restores all three to the old copy. A missing or disagreeing proxy unit, or an installed proxy without --proxy in the args file, refuses before anything stops.

A stamped copy, which is every copy stack-copy.sh makes, signs in through SSO accounts in both profiles (M3, #1873; Jason's ruling 2026-11-10). After --, the factory profile takes --sso --send with its private tester socket. The library profile takes a bare --sso, still without --send, so its Console stays read-only. A stamped library copy refuses the proxy-preview trio (--public-host, --auth-header, --allow-user) and the proof pair (--proof-header, --proof-file): proxy-preview admission does not apply to a stamped copy. Only an unstamped library copy, which no host install makes, keeps the old proxy-preview argv and refuses --sso. Before writing anything, case2-unit.sh refuses a stamped copy that lacks --sso or any prerequisite and names each missing step. The migration below lists them in order.

A library host installed before #385 has --send in two places, because the earlier case2-unit.sh required it: a --send line after -- in the args file, and --send in the installed Console unit's ExecStart. stack-upgrade.sh upgrade and rollback check both before anything stops, when the profile is library (no --profile line, or the last one says library):

  • --send after -- in the args file refuses args-unreadable.
  • A --send token after serve in /etc/systemd/system/mosaic-webui.service's ExecStart refuses console-unit-legacy. The current case2-unit.sh refuses that installed unit, so an upgrade or rollback would stop both units and then fail both renders.

Both refusals leave the units running, and the script rewrites neither file. Check the installed unit read-only, as root:

grep -n '^ExecStart=' /etc/systemd/system/mosaic-webui.service | grep -c -- ' --send'
  • 0, and no installed unit carries --send: removing the args line is enough. Open /etc/mosaic-stack/dev.args and delete the one line that reads --send, changing nothing else. Check that stat -c '%U %a' /etc/mosaic-stack/dev.args still gives root and a mode with no group or other write, and that grep -cx -- --send /etc/mosaic-stack/dev.args prints 0. Then rerun the same stack-upgrade.sh command.
  • 1 or more: the host needs a reviewed migration of the unit, the args file and the release pointer together, and none exists yet. Stop. Leave the units running, change neither file and report to the lead with the command's output. Removing only the args line does not help: the run then refuses console-unit-legacy.

The library Console stays read-only after the upgrade (#385). From M3 it signs in through SSO accounts, as the migration below sets up.

Migrating a host to SSO accounts (M3, #1873)

A host re-pinned onto an M3 copy migrates first, in this order. The order is the same for the factory (Case 1 and .53) and library (.54) profiles. Each step is a live step that needs Mos's go. Nothing in the stack picks or invents a subject: the subjects come from console.sso, which a human prepares, and accounts.json comes only from the human verb mosaic accounts init.

  1. Check or create .deployed through the installer. The stack's user's case2-install.sh (section 6) writes <dataRoot>/.deployed naming the instance at its step 2. It never replaces an existing one, and it refuses one that is invalid or names another instance. Check it read-only: stat -c '%U %a' <dataRoot>/.deployed prints mosaic 444.

  2. Prepare SSO. A human adds console.sso to /srv/mosaic/.config/mosaic-dev/config.json, with its providers, their subjects and its client-secret slots, per docs/plans/2026-11-10_accounts-design.md. Jason places the client secrets (section 9).

  3. Run mosaic accounts init --name <name> as mosaic. This is a human verb, run from a human shell and never by an agent. It creates the one operator account from the configured console.sso subjects and refuses if accounts.json exists. It also refuses accounts-unsafe when any folder above the config directory is not root's or mosaic's, or is group or other writable without the sticky bit (#1919). Check first, read-only: every line printed by p=/srv/mosaic/.config/mosaic-dev; while [ "$p" != / ]; do stat -c '%U %a %n' "$p"; p=$(dirname "$p"); done; stat -c '%U %a %n' / names root or mosaic, and its mode has no group or other write bit unless it starts with 1 (sticky). The config directory itself is mosaic's with no group or other write bit. A group-writable .config refuses on purpose; fix the mode, never the check.

  4. Provision the dedicated uid. mosaic must keep section 1's shape: a non-login shell, the only member of its primary group, and owner of its config directory and data root. An accounts instance never runs under a human or development seat uid, and never under the webui-service.sh user units.

  5. Re-pin with --next-args. Leave /etc/mosaic-stack/dev.args as it is. Write the new flags to /etc/mosaic-stack/dev.args.next, root's, mode 0640: the same lines before --, and after -- the serve flags with --sso added (factory: --sso --send). For library, also remove the proxy-preview trio (--public-host, --auth-header, --allow-user) and the proof pair (--proof-header, --proof-file). Then run stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA --next-args.

    A plain re-pin refuses render-refused here, because an installed unit must equal the render but for the copy id. With --next-args, before anything stops, the driver checks the installed copy against dev.args and the target as a first install from dev.args.next. It refuses next-args-differ when the files differ before -- (other than --copy), and next-args-same when they do not differ after it. After the stage it moves the installed unit files into a new 0700 directory /etc/mosaic-stack/dev.replaced.XXXXXX and renders the target as a first install. A failure from there moves them back unchanged and starts the old copy again. After confirm passes, dev.args.next becomes dev.args, and the replaced directory keeps the old unit files and the old dev.args. The result line names the directory. Nothing removes it; root removes it by hand once no rollback needs it.

    To roll back past the migration, put the replaced args back as the next args and roll back with --next-args:

    install -m 0640 -o root -g root /etc/mosaic-stack/dev.replaced.XXXXXX/dev.args /etc/mosaic-stack/dev.args.next
    stack-upgrade.sh rollback --instance dev --to <old copy id> --next-args
    

    For library this is the only rollback to a copy older than M3: case2-unit.sh refuses --sso for an unstamped library copy (#416), so the old copy comes back with its old flags, proxy-preview admission included, as before the migration. A factory rollback may instead keep --sso --send with a plain rollback.

If a step is missing, case2-unit.sh refuses before it writes a unit and prints the missing steps, numbered in this order, for example:

case2-unit: refusing the units for <sha>: a stamped copy's Console starts only with SSO accounts; missing steps, in order:
  1. check or create .deployed: mosaic runs case2-install.sh, whose deployment step writes <dataRoot>/.deployed
  2. prepare SSO: console.sso in mosaic's system config /srv/mosaic/.config/mosaic-dev/config.json, with its client secret slots
  3. run `mosaic accounts init` as mosaic, a human verb: no /srv/mosaic/.config/mosaic-dev/accounts.json
  4. add --sso to the serve flags, then re-pin

Without console.sso, the render does not check accounts.json, because its subjects name providers SSO has not configured yet. The next render after step 2 checks it. A rollback never removes accounts.json, its receipts or .deployed (docs/plans/2026-11-10_accounts-design.md, section 7).

Precheck line: none. stack-upgrade.sh refuses args-unreadable, console-unit-legacy, render-refused, next-args-differ or next-args-same on any of these rules, before it stops a unit.

8. Enrollment, once per host

The machine-grant enrollment is Gizmoduck's design r6, commit f9039682e, section 3, steps E1-E4. ops-01's Q-O1 is answered (mos-1125). The host side is the tools/host/ files: mosaic-keys, mosaic-keys-write, their sudoers line and the sshd drop-in /etc/ssh/sshd_config.d/<nn>-mosaic-keys.conf. The verbs that run it (hosts.enroll, the host-enroll decision) are rows 234 and 235, not yet on refactor. This item has no commands until they land.

Precheck line: none. The enrolled host's key and its mosaic-keys sudoers line: precheck line missing.

9. Credential slots

Jason places these, all in one sitting per host. No value appears here or in any message. Check a placed file with stat -c '%U %a %n' PATH, never cat. <business> is the host's business id.

.53, factory

Path File Mode Owner Profile Precheck line
/srv/mosaic/.config/mosaic-dev/secrets/ claude-session.env, one line CLAUDE_CODE_OAUTH_TOKEN=... (Q185) 0600 mosaic factory claude replaces its prompt with --system-prompt-file runs claude with it. The file's mode and owner: precheck line missing.
/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/ one file per service 0600, dirs 0700 mosaic both none. Filled only by mosaic credentials issue, or Approve after #436. Never by hand.

.54, library

Path File Mode Owner Profile Precheck line
/srv/mosaic/.config/mosaic-dev/secrets/ claude-session.env, one line ZAI_API_KEY=... 0600 mosaic library <path> sets ZAI_API_KEY
/srv/mosaic/.config/mosaic-dev/secrets/ before M3 only, and an unstamped copy only: the Console's proof file, the path the args file's --proof-file names. A stamped copy refuses --proof-file (section 7). The file is one base64url line, at least 32 [A-Za-z0-9_-] characters, with one optional trailing LF (packages/webui/src/serve.mjs:263-264) 0600 or 0400 mosaic library none. Render checks metadata only and never opens the file (scripts/case2-unit.sh:397-408). Bad bytes refuse at Console start with exit 2 (packages/webui/src/serve.mjs:267, packages/webui/src/cli.mjs:138-140). Precheck line missing.
/srv/mosaic/.config/mosaic-dev/secrets/<business>/ library-gitea 0600 mosaic library After a unit start, wait for loopback /healthz with the bounded retry in section 10 before case2-precheck.sh --business <business> --library-status: library-last-error library-credential-missing until it's placed, then library <project> last fetched ...
/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/ one file per service 0600, dirs 0700 mosaic both none. Filled only by mosaic credentials issue, or Approve after #436. Never by hand.

The installer key, <configDir>/secrets/hosts/<business>/<hostId>/installer, is not a slot anyone places. Enrollment (section 8) writes it under the enrolling stack's config directory and deletes it when the host leaves.

9a. Notifier decision, before starting units

Make the explicit per-host decision before section 10: the default below is binding: null, so the bus runs without DMs. A named binding is a later per-host choice that needs Mos's go. The installer does not make this decision or write this file. Replace <dataRoot> with the configured data root and <business> with this host's business id. If the file already exists, stop and review the existing decision; this command refuses to overwrite it.

sudo -u mosaic -H /bin/sh <<'MOSAIC_NOTIFY'
cd / || exit 1
umask 077
notify_root='<dataRoot>/notify'
notify_dir="$notify_root/<business>"
mkdir -p -m 0700 -- "$notify_root" && chmod 0700 -- "$notify_root" || exit 1
mkdir -p -m 0700 -- "$notify_dir" && chmod 0700 -- "$notify_dir" || exit 1
(set -C; printf '%s\n' '{"notifyVersion": 1, "binding": null}' > "$notify_dir/notify.json") || exit 1
chmod 0600 -- "$notify_dir/notify.json"
MOSAIC_NOTIFY

Precheck line: ok notifier config <dataRoot>/notify/<business>/notify.json validates (directory 0700, file 0600).

10. Console start

None by hand. I2's case2-unit.sh enables both units, starts the bus, then the Console (Q189, mos-1122). stack-upgrade.sh stops and starts both units itself.

After a unit start, wait for the Console's loopback /healthz to succeed before running case2-precheck.sh, including --library-status in section 9. Replace <port> with the installed Console unit's --port. This retry is bounded to 30 attempts, each with a 2-second timeout, with 1 second between failed attempts. If it prints STOP, stop here; do not run the precheck or adopt.

(
  attempts=0
  until curl -fsS --max-time 2 -o /dev/null 'http://127.0.0.1:<port>/healthz'; do
    attempts=$((attempts + 1))
    [ "$attempts" -lt 30 ] || { echo "STOP: loopback healthz did not become ready; do not continue" >&2; exit 1; }
    sleep 1
  done
)

Precheck line: none. Both units active after the render: precheck line missing.