Files
stack/docs/guides/host-run-sheet.md
T

16 KiB

Host run sheet: preparing a host for the stack

ops-01 runs this on a fresh host before install (I2), and once per host for enrollment. Jason places the credentials in section 9. Brief: docs/plans/2026-10-20_installer-rows.md, I3 (row 233, #1735). The host design is Darkwing's r6 (agents/darkwing/work/queue-195/DESIGN.md, section 2) as the brief changes it.

The hosts are .53 (profile factory) and .54 (profile library). Both get every section unless a section says otherwise. Every command runs as root unless it starts with sudo -u mosaic.

Each item names the scripts/case2-precheck.sh line that verifies it. Install's precheck runs as the stack's user with --business ID --profile factory|library. Where no precheck line checks an item, the item says "precheck line missing", and Gosalyn adds the line to I2. Today's precheck (C2-13a) has these gaps; C2-13b adds the library lines (library.sync, persona modes, no Discord binding).

1. The stack's user and its home

useradd --system --user-group --home-dir /srv/mosaic --create-home --shell /usr/sbin/nologin mosaic
chmod 0700 /srv/mosaic
install -d -o root -g root -m 0755 /etc/mosaic-stack
install -d -o root -g root -m 0750 /var/log/mosaic-stack

mosaic gets no sudo rule and no supplementary group. Check with sudo -l -U mosaic (expect "not allowed to run sudo") and id -G mosaic (one number).

Precheck lines:

  • ok HOME is /srv/mosaic
  • ok sudo -n true fails as mosaic
  • ok id -G prints one group (<gid>)
  • /srv/mosaic at 0700: precheck line missing.

2. node, pinned under /opt

The pin is the version and sha256 the window names. <v> below is that version, for example v24.x.y.

install -d -o root -g root -m 0755 /etc/mosaic-stack/node-keys /opt/mosaic-stack
cd "$(mktemp -d)"
curl -fsSLO https://nodejs.org/dist/<v>/node-<v>-linux-x64.tar.xz
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt.asc
gpg --no-default-keyring --keyring /etc/mosaic-stack/node-keys/pubring.kbx --verify SHASUMS256.txt.asc SHASUMS256.txt
grep " node-<v>-linux-x64.tar.xz\$" SHASUMS256.txt | sha256sum -c -
echo "<pinned sha256>  node-<v>-linux-x64.tar.xz" | sha256sum -c -
tar -xJf node-<v>-linux-x64.tar.xz --no-same-owner
mv node-<v>-linux-x64 /opt/mosaic-stack/node-<v>
chown -R root:root /opt/mosaic-stack/node-<v>
chmod -R go-w /opt/mosaic-stack/node-<v>

The keyring holds the Node release keys from the nodejs/release-keys repository, nothing else. Stop on any failed check; don't unpack. Rollback: rm -r /opt/mosaic-stack/node-<v>.

Precheck lines:

  • ok node <v> (version 24 or later only).
  • The --node path in the args file must be a root-owned regular file outside the home. case2-unit.sh refuses otherwise at render, which is not a precheck line.
  • The installed version equals the pin: precheck line missing.

3. claude, factory only (.53)

Root installs it under /opt, never under the stack's home (this replaces r6's /srv/mosaic/.local/bin). The pin is 2.1.304 unless the window names a newer one.

install -d -o root -g root -m 0755 /opt/mosaic-stack/claude-<pin> /etc/claude-code
PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  npm install --prefix /opt/mosaic-stack/claude-<pin> @anthropic-ai/claude-code@<pin>
chown -R root:root /opt/mosaic-stack/claude-<pin>
/opt/mosaic-stack/claude-<pin>/node_modules/@anthropic-ai/claude-code/bin/claude.exe --version
printf '%s\n' '{"env":{"DISABLE_AUTOUPDATER":"1"}}' > /etc/claude-code/managed-settings.json
chown root:root /etc/claude-code/managed-settings.json
chmod 0644 /etc/claude-code/managed-settings.json

Rollback: rm -r /opt/mosaic-stack/claude-<pin> /etc/claude-code/managed-settings.json.

Root's PATH has no npm, so the install line puts section 2's pinned node first; the package's postinstall also runs node. The --version line must print <pin> (Claude Code). Anything else means the postinstall left its 500-byte stub in bin/claude.exe (no native package for the platform, or scripts were off): stop and roll back.

The args file's --claude is /opt/mosaic-stack/claude-<pin>/node_modules/@anthropic-ai/claude-code/bin/claude.exe, the regular file the postinstall hardlinks or copies the native binary to. Not node_modules/.bin/claude: that is npm's symlink to it, and case2-unit.sh refuses a symlink, as it does anything that isn't a root-owned regular file outside the home. Checked against the 2.1.304 package's install.cjs; a newer pin needs the same check.

Precheck lines:

  • ok claude replaces its prompt with --system-prompt-file (<n> input tokens). It runs the claude the bus unit's MOSAIC_CLAUDE_BIN names, so it needs the model credential from section 9.
  • Managed settings present with the autoupdater off: precheck line missing.

4. bubblewrap, AppArmor and the drift stop

Both hosts. These are r6's items, unchanged.

apt-get install bubblewrap=0.9.0-1ubuntu0.3
stat -c '%U:%G %a' /usr/bin/bwrap > /etc/mosaic-stack/bwrap.orig    # expect root:root 755

cd "$(mktemp -d)"
apt-get download apparmor-profiles=4.0.1really4.0.1-0ubuntu0.24.04.9
dpkg-deb --fsys-tarfile apparmor-profiles_*.deb | tar -x ./usr/share/apparmor/extra-profiles/bwrap-userns-restrict
echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9  usr/share/apparmor/extra-profiles/bwrap-userns-restrict" | sha256sum -c -
install -o root -g root -m 0644 usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/bwrap-userns-restrict
echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9  /etc/apparmor.d/bwrap-userns-restrict" | sha256sum -c -
apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict

dpkg-statoverride --update --add root mosaic 0750 /usr/bin/bwrap

Run as root in a mktemp -d dir, apt-get download prints a line starting W: Download is performed unsandboxed as root, because the _apt user can't write that dir. The line is expected and the download still succeeds. Each sha256sum -c - line must print OK. The second one checks the copy in /etc/apparmor.d, the file that loads. Stop on any other result.

aa-status then lists bwrap and unpriv_bwrap in enforce mode. Install the one profile file, never the whole apparmor-profiles package, whose 20 other profiles would load at the next boot. Rollback, in reverse order: dpkg-statoverride --remove /usr/bin/bwrap, then chown and chmod to the values in bwrap.orig; apparmor_parser -R /etc/apparmor.d/bwrap-userns-restrict, then rm it; apt-get remove bubblewrap.

The no-new-privs test must print bwrap//&unpriv_bwrap (enforce):

systemd-run --wait --pipe --uid=mosaic --gid=mosaic -p NoNewPrivileges=yes -p ProtectSystem=strict \
  -p ProtectHome=yes -p PrivateTmp=yes \
  /usr/bin/bwrap --unshare-user --unshare-pid --ro-bind / / cat /proc/self/attr/current

The P9 checks: another user's /usr/bin/bwrap --version fails with Permission denied, getent group mosaic lists no member, and sysctl fs.protected_hardlinks prints 1. sysctl kernel.apparmor_restrict_unprivileged_userns prints 1 too (Ubuntu 24.04's default); leave it, so nothing else on the host gains user namespaces.

The drift stop (row 240, I4, #1746) goes in last, after the profile and the statoverride. As root, from the root-owned checkout of the reviewed commit:

install -d -o root -g root -m 0755 /usr/local/lib/mosaic-stack
install -o root -g root -m 0755 scripts/case1-hostcheck.sh /usr/local/lib/mosaic-stack/hostcheck.sh
install -o root -g root -m 0644 scripts/case1-pins.sh /usr/local/lib/mosaic-stack/case1-pins.sh
printf '%s\n' 'DPkg::Post-Invoke { "/usr/local/lib/mosaic-stack/hostcheck.sh --enforce || true"; };' \
  > /etc/apt/apt.conf.d/80mosaic-hostcheck
install -o root -g root -m 0644 packages/cli/systemd/mosaic-hostcheck.service /etc/systemd/system/
systemctl daemon-reload && systemctl enable mosaic-hostcheck.service
/usr/local/lib/mosaic-stack/hostcheck.sh --check

--check prints one ok line for each of its seven items and exits 0. Item 7 reads 1023 as the port until I2 installs the Console unit (section 10), then the unit's --port. stack-upgrade.sh runs --check before its plan and refuses profile-drift on any exit but 0, a missing file included. Rollback: systemctl disable mosaic-hostcheck.service, then rm the unit, the hook file and /usr/local/lib/mosaic-stack.

On drift, --enforce (the apt hook after every package run, and the unit at boot) unloads the profile, closes /usr/bin/bwrap to root:root 0700, appends the failed items to /etc/mosaic-stack/DRIFT and logs a line in /var/log/mosaic-stack/hostcheck.log. Every role session then refuses to start. ops-01 reads the log in each STATE report and sends any DRIFT to Mos. Only root removes DRIFT, in a reviewed step with a new pin or a Mos go. After that, chown root:mosaic /usr/bin/bwrap, chmod 0750 /usr/bin/bwrap and apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict put bwrap back, and --check must exit 0.

Before each re-pin window, ops-01 runs /usr/local/lib/mosaic-stack/hostcheck.sh --upstream. It downloads the apparmor-profiles candidate and compares its bwrap-userns-restrict to the pin. A DIFFERS line goes to Mos as a new-pin request; it refuses nothing.

Precheck lines:

  • ok bwrap --unshare-all runs
  • The AppArmor profile, the statoverride, the no-new-privs result and the P9 checks: precheck line missing. /usr/local/lib/mosaic-stack/hostcheck.sh --check covers them (row 240).

5. The port floor

Leave net.ipv4.ip_unprivileged_port_start at its default (1024) or higher. The Console's port is below it and the unit gets CAP_NET_BIND_SERVICE, so no other mosaic process can bind that port.

Precheck line: ok net.ipv4.ip_unprivileged_port_start <n> is above the Console's port <port>.

6. The release transport

A release is a git bundle of refs/heads/refactor plus the 40-character commit id (I1, mos-1118 choice 2). Mos's go names both, with the bundle's sha256. <sha> below is that id.

First install:

# Copy the bundle in, then check it before anything reads it.
echo "<bundle sha256>  /root/<sha>.bundle" | sha256sum -c -
git bundle verify /root/<sha>.bundle

# Root's checkout. stack-upgrade.sh and case2-unit.sh run from here, never from a copy.
git clone -q -b refactor /root/<sha>.bundle /root/mosaic-stack
git -C /root/mosaic-stack checkout -q --detach <sha>
git -C /root/mosaic-stack rev-parse HEAD        # must print <sha>

# The stack's user's copy of the bundle, and its own clone.
install -d -o mosaic -g mosaic -m 0700 /srv/mosaic/incoming /srv/mosaic/src
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
sudo -u mosaic -H git clone -q -b refactor /srv/mosaic/incoming/<sha>.bundle /srv/mosaic/src/mosaic-stack

Each upgrade, before stack-upgrade.sh upgrade:

echo "<bundle sha256>  /root/<sha>.bundle" | sha256sum -c -
git bundle verify /root/<sha>.bundle
git -C /root/mosaic-stack fetch -q /root/<sha>.bundle refs/heads/refactor:refs/remotes/origin/refactor
git -C /root/mosaic-stack checkout -q --detach <sha>
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
/root/mosaic-stack/scripts/stack-upgrade.sh upgrade --instance dev --bundle /srv/mosaic/incoming/<sha>.bundle --sha <sha>

The driver fetches the bundle into the stack's user's clone and makes the copy itself. It refuses head-mismatch when root's checkout isn't at <sha>. Usage, exits and the step log are in docs/TOOLS.md, under "Root's upgrade driver". Each run needs Mos's go, and --migrate needs Jason present (decision 96).

When a run doesn't exit 0, send Mos the exit and the last lines of /var/log/mosaic-stack/upgrade-dev.log:

  • Exit 3: refused before anything stopped. The last line names the code. The stack's user's clone and the new copy may already hold the new commit; the units still run the old one.
  • Exit 1: the old copy runs and passed its gate. Either the driver restarted it after A2 or a refused A3 (A2f), or it rolled back after A3, A4 or A5.
  • Exit 2: both units are stopped and the activation stays open, so the next run refuses at its plan until it is finished with confirm or revert. One way here: a unit that doesn't stop within --timeout keeps its old start time, A2f's gate fails it stale-start, and the driver stops both (duckworth-0055 N1). That is on purpose; a unit that won't stop is a reason to stop.
  • Exit 5: args-failed. The new copy is live and passed confirm, but /etc/mosaic-stack/dev.args still names the old copy. The next run refuses args-differ, and a render from that file would put the old copy back. Move the file the result line names into place with mv -f. The two differ only in the --copy value.

Precheck lines: none. Root's checkout at <sha> and the stack's user's clone: precheck line missing. After install, stack-copy.sh check --units external reads the copy the units run.

7. The args file

/etc/mosaic-stack/dev.args holds the case2-unit.sh arguments, one per line, with the serve flags after a line holding --. I2's install writes it; ops-01 doesn't write it by hand. It must be root's, with no group or other write (chmod 0644 or tighter). Before -- it takes only --user, --home, --business, --copy (exactly once), --node, --profile, --claude and --pm. stack-upgrade.sh changes only the --copy line, and only after confirm passes.

Precheck line: none. stack-upgrade.sh refuses args-unreadable on any of these rules.

8. Enrollment, once per host

The machine-grant enrollment is Gizmoduck's design r6, commit f9039682e, section 3, steps E1-E4. ops-01's Q-O1 is answered (mos-1125). The host side is the tools/host/ files: mosaic-keys, mosaic-keys-write, their sudoers line and the sshd drop-in /etc/ssh/sshd_config.d/<nn>-mosaic-keys.conf. The verbs that run it (hosts.enroll, the host-enroll decision) are rows 234 and 235, not yet on refactor. This item has no commands until they land.

Precheck line: none. The enrolled host's key and its mosaic-keys sudoers line: precheck line missing.

9. Credential slots

Jason places these, all in one sitting per host. No value appears here or in any message. Check a placed file with stat -c '%U %a %n' PATH, never cat. <business> is the host's business id.

.53, factory

Path File Mode Owner Profile Precheck line
/srv/mosaic/.config/mosaic-dev/secrets/ claude-session.env, one line CLAUDE_CODE_OAUTH_TOKEN=... (Q185) 0600 mosaic factory claude replaces its prompt with --system-prompt-file runs claude with it. The file's mode and owner: precheck line missing.
/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/ one file per service 0600, dirs 0700 mosaic both none. Filled only by mosaic credentials issue, or Approve after #436. Never by hand.

.54, library

Path File Mode Owner Profile Precheck line
/srv/mosaic/.config/mosaic-dev/secrets/ claude-session.env, one line ZAI_API_KEY=... 0600 mosaic library <path> sets ZAI_API_KEY
/srv/mosaic/.config/mosaic-dev/secrets/ the Console's proof file, the path the args file's --proof-file names 0600 or 0400 mosaic library none. case2-unit.sh refuses at render. Precheck line missing.
/srv/mosaic/.config/mosaic-dev/secrets/<business>/ library-gitea 0600 mosaic library C2-13b's library.sync (not landed)
/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/ one file per service 0600, dirs 0700 mosaic both none. Filled only by mosaic credentials issue, or Approve after #436. Never by hand.

The installer key, <configDir>/secrets/hosts/<business>/<hostId>/installer, is not a slot anyone places. Enrollment (section 8) writes it under the enrolling stack's config directory and deletes it when the host leaves.

10. Console start

None by hand. I2's case2-unit.sh enables both units, starts the bus, then the Console (Q189, mos-1122). stack-upgrade.sh stops and starts both units itself.

Precheck line: none. Both units active after the render: precheck line missing.