jason.woltje
6415e81ee7
mosaic accounts init refuses accounts-unsafe since #1919 when a folder above the config directory is writable by another user. Step 3 now gives the read-only check.
585 lines
34 KiB
Markdown
585 lines
34 KiB
Markdown
# Host run sheet: preparing a host for the stack
|
|
|
|
ops-01 runs this on a fresh host before install (I2), and once per host for enrollment. Jason places the
|
|
credentials in section 9. Brief: `docs/plans/2026-10-20_installer-rows.md`, I3 (row 233, #1735). The
|
|
host design is Darkwing's r6 (`agents/darkwing/work/queue-195/DESIGN.md`, section 2) as the brief
|
|
changes it.
|
|
|
|
The hosts are .53 (profile `factory`) and .54 (profile `library`). Both get every section unless a
|
|
section says otherwise. Every command runs as root unless it starts with `sudo -u mosaic`.
|
|
|
|
Each item names the `scripts/case2-precheck.sh` line that verifies it. Install's precheck runs as the
|
|
stack's user with `--business ID --profile factory|library`. Where no precheck line checks an item, the
|
|
item says "precheck line missing", and Gosalyn adds the line to I2. Today's precheck (C2-13a) has these
|
|
gaps; C2-13b adds the library lines (`library.sync`, persona modes, no Discord binding).
|
|
|
|
## 1. The stack's user and its home
|
|
|
|
```sh
|
|
useradd --system --user-group --home-dir /srv/mosaic --create-home --shell /usr/sbin/nologin mosaic
|
|
chmod 0700 /srv/mosaic
|
|
install -d -o root -g root -m 0755 /etc/mosaic-stack
|
|
install -d -o root -g root -m 0750 /var/log/mosaic-stack
|
|
```
|
|
|
|
`mosaic` gets no sudo rule and no supplementary group. Check with `sudo -l -U mosaic` (expect "not allowed
|
|
to run sudo") and `id -G mosaic` (one number).
|
|
|
|
Precheck lines:
|
|
- `ok HOME is /srv/mosaic`
|
|
- `ok sudo -n true fails as mosaic`
|
|
- `ok id -G prints one group (<gid>)`
|
|
- `/srv/mosaic` at 0700: precheck line missing.
|
|
|
|
## 2. node, pinned under /opt
|
|
|
|
The pin is the version and sha256 the window names. `<v>` below is that version, for example `v24.x.y`.
|
|
|
|
```sh
|
|
install -d -o root -g root -m 0755 /etc/mosaic-stack/node-keys /opt/mosaic-stack
|
|
cd "$(mktemp -d)"
|
|
curl -fsSLO https://nodejs.org/dist/<v>/node-<v>-linux-x64.tar.xz
|
|
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt
|
|
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt.sig
|
|
gpg --no-default-keyring --keyring /etc/mosaic-stack/node-keys/pubring.kbx --verify SHASUMS256.txt.sig SHASUMS256.txt
|
|
grep " node-<v>-linux-x64.tar.xz\$" SHASUMS256.txt | sha256sum -c -
|
|
echo "<pinned sha256> node-<v>-linux-x64.tar.xz" | sha256sum -c -
|
|
tar -xJf node-<v>-linux-x64.tar.xz --no-same-owner
|
|
mv node-<v>-linux-x64 /opt/mosaic-stack/node-<v>
|
|
chown -R root:root /opt/mosaic-stack/node-<v>
|
|
chmod -R go-w /opt/mosaic-stack/node-<v>
|
|
```
|
|
|
|
The keyring holds the Node release keys from the nodejs/release-keys repository, nothing else. Stop on any
|
|
failed check; don't unpack. Rollback: `rm -r /opt/mosaic-stack/node-<v>`.
|
|
|
|
Precheck lines:
|
|
- `ok node <v>` (version 24 or later only).
|
|
- The `--node` path in the args file must be a root-owned regular file outside the home. `case2-unit.sh`
|
|
refuses otherwise at render, which is not a precheck line.
|
|
- The installed version equals the pin: precheck line missing.
|
|
|
|
## 3. claude, factory only (.53)
|
|
|
|
Root installs it under `/opt`, never under the stack's home (this replaces r6's `/srv/mosaic/.local/bin`).
|
|
The pin is 2.1.304 unless the window names a newer one.
|
|
|
|
```sh
|
|
install -d -o root -g root -m 0755 /opt/mosaic-stack/claude-<pin> /etc/claude-code
|
|
PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin \
|
|
npm install --prefix /opt/mosaic-stack/claude-<pin> @anthropic-ai/claude-code@<pin>
|
|
chown -R root:root /opt/mosaic-stack/claude-<pin>
|
|
/opt/mosaic-stack/claude-<pin>/node_modules/@anthropic-ai/claude-code/bin/claude.exe --version
|
|
printf '%s\n' '{"env":{"DISABLE_AUTOUPDATER":"1"}}' > /etc/claude-code/managed-settings.json
|
|
chown root:root /etc/claude-code/managed-settings.json
|
|
chmod 0644 /etc/claude-code/managed-settings.json
|
|
```
|
|
|
|
Rollback: `rm -r /opt/mosaic-stack/claude-<pin> /etc/claude-code/managed-settings.json`.
|
|
|
|
Root's PATH has no npm, so the install line puts section 2's pinned node first; the package's postinstall
|
|
also runs `node`. The `--version` line must print `<pin> (Claude Code)`. Anything else means the postinstall
|
|
left its 500-byte stub in `bin/claude.exe` (no native package for the platform, or scripts were off): stop and
|
|
roll back.
|
|
|
|
The args file's `--claude` is
|
|
`/opt/mosaic-stack/claude-<pin>/node_modules/@anthropic-ai/claude-code/bin/claude.exe`, the regular file the
|
|
postinstall hardlinks or copies the native binary to. Not `node_modules/.bin/claude`: that is npm's symlink
|
|
to it, and `case2-unit.sh` refuses a symlink, as it does anything that isn't a root-owned regular file outside
|
|
the home. Checked against the 2.1.304 package's `install.cjs`; a newer pin needs the same check.
|
|
|
|
Precheck lines:
|
|
- `ok claude replaces its prompt with --system-prompt-file (<n> input tokens)`. It runs the claude the bus
|
|
unit's `MOSAIC_CLAUDE_BIN` names, so it needs the model credential from section 9.
|
|
- Managed settings present with the autoupdater off: precheck line missing.
|
|
|
|
## 4. bubblewrap, AppArmor and the drift stop
|
|
|
|
Both hosts. These are r6's items, unchanged.
|
|
|
|
```sh
|
|
apt-get install bubblewrap=0.9.0-1ubuntu0.3
|
|
stat -c '%U:%G %a' /usr/bin/bwrap > /etc/mosaic-stack/bwrap.orig # expect root:root 755
|
|
|
|
cd "$(mktemp -d)"
|
|
apt-get download apparmor-profiles=4.0.1really4.0.1-0ubuntu0.24.04.9
|
|
dpkg-deb --fsys-tarfile apparmor-profiles_*.deb | tar -x ./usr/share/apparmor/extra-profiles/bwrap-userns-restrict
|
|
echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9 usr/share/apparmor/extra-profiles/bwrap-userns-restrict" | sha256sum -c -
|
|
install -o root -g root -m 0644 usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/bwrap-userns-restrict
|
|
echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9 /etc/apparmor.d/bwrap-userns-restrict" | sha256sum -c -
|
|
apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict
|
|
|
|
dpkg-statoverride --update --add root mosaic 0750 /usr/bin/bwrap
|
|
```
|
|
|
|
Run as root in a `mktemp -d` dir, `apt-get download` prints a line starting `W: Download is performed
|
|
unsandboxed as root`, because the `_apt` user can't write that dir. The line is expected and the
|
|
download still succeeds. Each `sha256sum -c -` line must print `OK`. The second one checks the copy in
|
|
`/etc/apparmor.d`, the file that loads. Stop on any other result.
|
|
|
|
`aa-status` then lists `bwrap` and `unpriv_bwrap` in enforce mode. Install the one profile file, never the
|
|
whole `apparmor-profiles` package, whose 20 other profiles would load at the next boot. Rollback, in
|
|
reverse order: `dpkg-statoverride --remove /usr/bin/bwrap`, then `chown` and `chmod` to the values in
|
|
`bwrap.orig`; `apparmor_parser -R /etc/apparmor.d/bwrap-userns-restrict`, then `rm` it;
|
|
`apt-get remove bubblewrap`.
|
|
|
|
The no-new-privs test must print `bwrap//&unpriv_bwrap (enforce)`:
|
|
|
|
```sh
|
|
systemd-run --wait --pipe --uid=mosaic --gid=mosaic -p NoNewPrivileges=yes -p ProtectSystem=strict \
|
|
-p ProtectHome=yes -p PrivateTmp=yes \
|
|
/usr/bin/bwrap --unshare-user --unshare-pid --ro-bind / / cat /proc/self/attr/current
|
|
```
|
|
|
|
The P9 checks: another user's `/usr/bin/bwrap --version` fails with Permission denied, `getent group mosaic`
|
|
lists no member, and `sysctl fs.protected_hardlinks` prints 1. `sysctl
|
|
kernel.apparmor_restrict_unprivileged_userns` prints 1 too (Ubuntu 24.04's default); leave it, so nothing
|
|
else on the host gains user namespaces.
|
|
|
|
The drift stop (row 240, I4, #1746) goes in last, after the profile and the statoverride. As root, from
|
|
the root-owned checkout of the reviewed commit:
|
|
|
|
```sh
|
|
install -d -o root -g root -m 0755 /usr/local/lib/mosaic-stack
|
|
install -o root -g root -m 0755 scripts/case1-hostcheck.sh /usr/local/lib/mosaic-stack/hostcheck.sh
|
|
install -o root -g root -m 0644 scripts/case1-pins.sh /usr/local/lib/mosaic-stack/case1-pins.sh
|
|
printf '%s\n' 'DPkg::Post-Invoke { "/usr/local/lib/mosaic-stack/hostcheck.sh --enforce || true"; };' \
|
|
> /etc/apt/apt.conf.d/80mosaic-hostcheck
|
|
install -o root -g root -m 0644 packages/cli/systemd/mosaic-hostcheck.service /etc/systemd/system/
|
|
systemctl daemon-reload && systemctl enable mosaic-hostcheck.service
|
|
/usr/local/lib/mosaic-stack/hostcheck.sh --check
|
|
```
|
|
|
|
`--check` prints one `ok` line for each of its seven items and exits 0. Item 7 reads 1023 as the port
|
|
until I2 installs the Console unit (section 10), then the unit's `--port`. `stack-upgrade.sh` runs `--check`
|
|
before its plan and refuses `profile-drift` on any exit but 0, a missing file included. Rollback:
|
|
`systemctl disable mosaic-hostcheck.service`, then `rm` the unit, the hook file and
|
|
`/usr/local/lib/mosaic-stack`.
|
|
|
|
On drift, `--enforce` (the apt hook after every package run, and the unit at boot) unloads the profile,
|
|
closes `/usr/bin/bwrap` to `root:root` 0700, appends the failed items to `/etc/mosaic-stack/DRIFT` and logs
|
|
a line in `/var/log/mosaic-stack/hostcheck.log`. Every role session then refuses to start. ops-01 reads the
|
|
log in each STATE report and sends any `DRIFT` to Mos. Only root removes `DRIFT`, in a reviewed step with
|
|
a new pin or a Mos go. After that, `chown root:mosaic /usr/bin/bwrap`, `chmod 0750 /usr/bin/bwrap` and
|
|
`apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict` put bwrap back, and `--check` must exit 0.
|
|
|
|
Before each re-pin window, ops-01 runs `/usr/local/lib/mosaic-stack/hostcheck.sh --upstream`. It
|
|
downloads the `apparmor-profiles` candidate and compares its `bwrap-userns-restrict` to the pin. A
|
|
`DIFFERS` line goes to Mos as a new-pin request; it refuses nothing.
|
|
|
|
Precheck lines:
|
|
- `ok bwrap --unshare-all runs`
|
|
- The AppArmor profile, the statoverride, the no-new-privs result and the P9 checks: precheck line missing.
|
|
`/usr/local/lib/mosaic-stack/hostcheck.sh --check` covers them (row 240).
|
|
|
|
## 5. The port floor
|
|
|
|
Leave `net.ipv4.ip_unprivileged_port_start` at its default (1024) or higher. The Console's port is below
|
|
it and the unit gets `CAP_NET_BIND_SERVICE`, so no other `mosaic` process can bind that port.
|
|
|
|
Precheck line: `ok net.ipv4.ip_unprivileged_port_start <n> is above the Console's port <port>`.
|
|
|
|
## 6. The release transport
|
|
|
|
A release is a git bundle of `refs/heads/refactor` plus the 40-character commit id (I1, mos-1118 choice 2).
|
|
Mos's go names both, with the bundle's sha256. `<sha>` below is that id.
|
|
|
|
First install:
|
|
|
|
```sh
|
|
# Copy the bundle in, then check it before anything reads it.
|
|
echo "<bundle sha256> /root/<sha>.bundle" | sha256sum -c -
|
|
(
|
|
bundle_check="$(mktemp -d)" || exit 1
|
|
trap 'rm -rf "$bundle_check"' EXIT
|
|
git init -q --bare "$bundle_check" &&
|
|
git -C "$bundle_check" bundle verify /root/<sha>.bundle
|
|
) || echo "STOP: bundle verify failed; do not continue"
|
|
|
|
# Root's checkout. stack-upgrade.sh and case2-unit.sh run from here, never from a copy.
|
|
git clone -q -b refactor /root/<sha>.bundle /root/mosaic-stack
|
|
git -C /root/mosaic-stack checkout -q --detach <sha>
|
|
git -C /root/mosaic-stack rev-parse HEAD # must print <sha>
|
|
|
|
# The stack's user's copy of the bundle, and its own clone.
|
|
install -d -o mosaic -g mosaic -m 0700 /srv/mosaic/incoming /srv/mosaic/src
|
|
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
|
|
sudo -u mosaic -H git clone -q -b refactor /srv/mosaic/incoming/<sha>.bundle /srv/mosaic/src/mosaic-stack
|
|
```
|
|
|
|
If the first install refuses, use Mos's retry path (mos-1565, ops-0291). Bootstrap can already have
|
|
written `/srv/mosaic/.config/mosaic-dev/config.json` (`scripts/case2-install.sh:174`,
|
|
`scripts/bootstrap.sh:13`, `scripts/mosaic-config.mjs:412-424`). The config dir can also hold
|
|
`businesses/`, the business file, and library persona files and slot dirs
|
|
(`scripts/case2-install.sh:200-211`, `scripts/case2-persona-import.mjs:300-308`). A validation refusal
|
|
removes only the business file this run created; its directory remains (`scripts/case2-install.sh:217-219`).
|
|
The copy may already exist under `/srv/mosaic/.local/share/mosaic-stack/copies/<sha>`
|
|
(`scripts/case2-install.sh:80`, `scripts/stack-copy.sh:68`).
|
|
|
|
As root, move these three paths aside with a UTC suffix. Move, never delete, so their owner and mode
|
|
stay with them (mos-1565). This is the default-instance retry for the paths in this sheet:
|
|
|
|
```sh
|
|
(
|
|
cd / || exit 1
|
|
retry_stamp="$(date -u +%Y%m%dT%H%M%SZ)" || exit 1
|
|
for retry_path in /root/mosaic-stack /srv/mosaic/src/mosaic-stack /srv/mosaic/.config/mosaic-dev; do
|
|
if [ -e "$retry_path" ] || [ -L "$retry_path" ]; then
|
|
retry_saved="$retry_path.$retry_stamp"
|
|
if [ -e "$retry_saved" ] || [ -L "$retry_saved" ]; then
|
|
echo "STOP: retry destination already exists: $retry_saved" >&2
|
|
exit 1
|
|
fi
|
|
mv -T -- "$retry_path" "$retry_saved" || exit 1
|
|
fi
|
|
done
|
|
) || echo "STOP: could not move the first-install paths aside; do not continue"
|
|
```
|
|
|
|
Then rerun the first-install block from its first line. Never use the upgrade fetch below on clones
|
|
whose first install has not finished: it updates `refs/remotes/origin/refactor`, while copy requires
|
|
the commit to be reachable from both that ref and `refs/heads/refactor` (`scripts/stack-copy.sh:44-48`; mos-1565).
|
|
Recheck section 9 before starting units. If Jason already filled `claude-session.env`, the library
|
|
proof file or `library-gitea`, have him re-place those files under the new config dir with section 9's
|
|
owner and mode. Per-agent slots still use `mosaic credentials issue` or Approve, never a hand copy
|
|
(section 9; mos-1565). The moved config dir preserves the earlier files.
|
|
|
|
After a first install, once both units are running (section 10), adopt the installed copy before
|
|
any activate or upgrade. This step runs only on Mos's go, with Jason or Mos running it
|
|
(`scripts/stack-release.sh:14`):
|
|
|
|
```sh
|
|
sudo -u mosaic -H env PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin \
|
|
/srv/mosaic/src/mosaic-stack/scripts/stack-release.sh adopt <business> --instance dev --units external
|
|
```
|
|
|
|
Each upgrade, before `stack-upgrade.sh upgrade`:
|
|
|
|
Only the first upgrade after adopt takes `--no-rehearsal`: append it to the upgrade command below.
|
|
Adopt writes no backup, so omitting the flag then refuses `no-backup`. Subsequent upgrades omit it;
|
|
when a backup exists, supplying it refuses `rehearsal-required`.
|
|
|
|
```sh
|
|
echo "<bundle sha256> /root/<sha>.bundle" | sha256sum -c -
|
|
(
|
|
bundle_check="$(mktemp -d)" || exit 1
|
|
trap 'rm -rf "$bundle_check"' EXIT
|
|
git init -q --bare "$bundle_check" &&
|
|
git -C "$bundle_check" bundle verify /root/<sha>.bundle
|
|
) || echo "STOP: bundle verify failed; do not continue"
|
|
git -C /root/mosaic-stack fetch -q /root/<sha>.bundle refs/heads/refactor:refs/remotes/origin/refactor
|
|
git -C /root/mosaic-stack checkout -q --detach <sha>
|
|
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
|
|
/root/mosaic-stack/scripts/stack-upgrade.sh upgrade --instance dev --bundle /srv/mosaic/incoming/<sha>.bundle --sha <sha>
|
|
```
|
|
|
|
The driver fetches the bundle into the stack's user's clone and makes the copy itself. It refuses
|
|
`head-mismatch` when root's checkout isn't at `<sha>`. Usage, exits and the step log are in
|
|
`docs/TOOLS.md`, under "Root's upgrade driver". Each run needs Mos's go, and `--migrate` needs Jason present
|
|
(decision 96).
|
|
|
|
When a run doesn't exit 0, send Mos the exit and the last lines of `/var/log/mosaic-stack/upgrade-dev.log`:
|
|
- Exit 3: refused before anything stopped. The last line names the code. The stack's user's clone and the
|
|
new copy may already hold the new commit; the units still run the old one.
|
|
- Exit 1: the old copy runs and passed its gate. Either the driver restarted it after A2 or a refused A3
|
|
(A2f), or it rolled back after A3, A4 or A5.
|
|
- Exit 2: both units are stopped and the activation stays open, so the next run refuses at its plan until
|
|
it is finished with `confirm` or `revert`. One way here: a unit that doesn't stop within `--timeout`
|
|
keeps its old start time, A2f's gate fails it `stale-start`, and the driver stops both (duckworth-0055
|
|
N1). That is on purpose; a unit that won't stop is a reason to stop.
|
|
- Exit 5: `args-failed`. The new copy is live and passed confirm, but `/etc/mosaic-stack/dev.args` still names
|
|
the old copy. The next run refuses `args-differ`, and a render from that file would put the old copy back.
|
|
Move the file the result line names into place with `mv -f`. The two differ only in the `--copy` value.
|
|
|
|
To check a backup by hand before rollback or restore, run as the stack's user from its `bus/` directory:
|
|
|
|
```sh
|
|
sudo -u mosaic -H /bin/sh -c \
|
|
'cd "<dataRoot>/backups/stack/<activation>/bus" && sha256sum -c ../SHA256SUMS'
|
|
```
|
|
|
|
Expect `bus.sqlite: OK`; stop on a failed check. The set holds `bus/bus.sqlite`, and `SHA256SUMS` names
|
|
`bus.sqlite` relative to `bus/` (`scripts/stack-release.sh:475-482`). Running `sha256sum -c SHA256SUMS`
|
|
from the backup set's directory looks for `bus.sqlite` one directory too high (ops-0292). Restore and
|
|
rehearsal use the same `bus/` form
|
|
(`scripts/stack-release.sh:502`, `scripts/stack-release.sh:2244`).
|
|
|
|
After `case2-install.sh` succeeds, create the human CLI wrapper before running
|
|
`stack-copy.sh check --units external`. Creating this wrapper is a separate live
|
|
step requiring Mos's go (`scripts/stack-copy.sh:3`); the install does not create it. Run only on that go:
|
|
The step runs under sudo's `secure_path`, so it sets its own PATH and umask.
|
|
|
|
```sh
|
|
sudo -u mosaic -H /bin/sh <<'MOSAIC_WRAPPER'
|
|
cd / || exit 1
|
|
umask 077
|
|
PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin; export PATH
|
|
mkdir -p -- "$HOME/.npm-global/bin" || exit 1
|
|
"$HOME/src/mosaic-stack/scripts/stack-copy.sh" wrapper <sha> --create
|
|
MOSAIC_WRAPPER
|
|
```
|
|
|
|
`--create` refuses an existing wrapper; stop and review it rather than replacing
|
|
it. The wrapper must name the same `<sha>` copy as the units.
|
|
|
|
Precheck lines: none. Root's checkout at `<sha>` and the stack's user's clone: precheck line missing.
|
|
After install, `stack-copy.sh check --units external` reads the copy the units run.
|
|
|
|
Before running `case2-install.sh`, check the exact file you will pass as `--business-file`. Each
|
|
non-library `projects` key must equal the `id` in `<root>/.mosaic/project.json`, and that file must exist
|
|
in the stack's user's clone. Library entries are skipped by validate (`packages/organization/src/config-command.mjs:95`).
|
|
Run this read-only check as `mosaic` with section 2's pinned node:
|
|
|
|
```sh
|
|
sudo -u mosaic -H /opt/mosaic-stack/node-<v>/bin/node --input-type=module - '<business-file>' <<'MOSAIC_PROJECT_KEYS'
|
|
import { readFileSync } from 'node:fs';
|
|
import { join } from 'node:path';
|
|
try {
|
|
const business = JSON.parse(readFileSync(process.argv[2], 'utf8'));
|
|
const projects = Object.entries(business.projects);
|
|
if (!projects.length) throw new Error('business file declares no projects');
|
|
for (const [id, entry] of projects) {
|
|
if (entry.kind === 'library') continue;
|
|
const file = join(entry.root, '.mosaic', 'project.json');
|
|
const project = JSON.parse(readFileSync(file, 'utf8'));
|
|
if (project.id !== id)
|
|
throw new Error(`project file ${file} has id ${project.id}, but business ${business.id} declares it as ${id}`);
|
|
console.log(`ok project ${id}: ${file} has id ${project.id}`);
|
|
}
|
|
} catch (error) {
|
|
console.error(`STOP: ${error.message}; do not install`);
|
|
process.exitCode = 1;
|
|
}
|
|
MOSAIC_PROJECT_KEYS
|
|
```
|
|
|
|
For .53 expect `ok project stack: /srv/mosaic/src/mosaic-stack/.mosaic/project.json has id stack`
|
|
(`.mosaic/project.json:3`, `docs/guides/case1-business.mosaic-stack.json:10-11`). Any STOP or nonzero exit
|
|
stops the install. Fix a mismatched business key, not the project id (mos-1563). The install otherwise
|
|
reaches that refusal at step 7, after making the copy (`packages/organization/src/config-command.mjs:103`,
|
|
`scripts/case2-install.sh:178`, `scripts/case2-install.sh:217`).
|
|
|
|
The stack's user's `scripts/case2-install.sh` (C2-13b, design 10 step 2b) makes the first copy with
|
|
`stack-copy.sh copy <sha>`. The copy runs `npm ci` from `package-lock.json`, which installs the pinned
|
|
harness, so the host needs to reach the npm registry during the install. The install runs no
|
|
`release.sh package` or `activate`.
|
|
|
|
A missing `.mosaic/project.json` only warns: `project <id>: no project file at <path>; it has no project
|
|
variables`. Validate reports that project `absent` and continues (`packages/organization/src/config-command.mjs:97-101`).
|
|
An install can therefore pass step 7 without the project's variables. It captures validate's output
|
|
and prints `business <id> validates` (`scripts/case2-install.sh:217-221`). Treat the missing-file warning as a stop,
|
|
place the correct project file and repeat the check above before continuing (Duckworth 323 N2;
|
|
`scripts/case2-install.sh:217-221`).
|
|
|
|
## 7. The args file
|
|
|
|
`/etc/mosaic-stack/dev.args` holds the `case2-unit.sh` arguments, one per line, with the serve flags after a
|
|
line holding `--`. Root writes it at the unit render, from the values the go names. The stack's user's
|
|
install never writes it: that user can't write `/etc`, and root reads nothing that user wrote. It must be root's, with no group
|
|
or other write (`chmod 0644` or tighter). Before `--` it takes only `--user`, `--home`, `--business`,
|
|
`--copy` (exactly once), `--node`, `--profile`, `--claude`, `--pm` and `--proxy`. `stack-upgrade.sh` changes only the
|
|
`--copy` line, and only after confirm passes.
|
|
|
|
For a hub installed with `--proxy`, keep that bare flag on its own line before `--`; it takes no value.
|
|
Upgrade and rollback require the proxy, bus and Console units to name the same copy. They stop the Console,
|
|
then the bus, then the proxy, re-render all three onto the selected copy, and start the proxy, then the bus,
|
|
then the Console. Recovery after a failed upgrade restores all three to the old copy. A missing or
|
|
disagreeing proxy unit, or an installed proxy without `--proxy` in the args file, refuses before anything stops.
|
|
|
|
A stamped copy, which is every copy `stack-copy.sh` makes, signs in through SSO accounts in both profiles
|
|
(M3, #1873; Jason's ruling 2026-11-10). After `--`, the `factory` profile takes `--sso --send` with its
|
|
private tester socket. The `library` profile takes a bare `--sso`, still without `--send`, so its Console
|
|
stays read-only. A stamped library copy refuses the proxy-preview trio (`--public-host`, `--auth-header`,
|
|
`--allow-user`) and the proof pair (`--proof-header`, `--proof-file`): proxy-preview admission does not
|
|
apply to a stamped copy. Only an unstamped library copy, which no host install makes, keeps the old
|
|
proxy-preview argv and refuses `--sso`. Before writing anything, `case2-unit.sh` refuses a stamped copy
|
|
that lacks `--sso` or any prerequisite and names each missing step. The migration below lists them in
|
|
order.
|
|
|
|
A library host installed before #385 has `--send` in two places, because the earlier `case2-unit.sh`
|
|
required it: a `--send` line after `--` in the args file, and `--send` in the installed Console unit's
|
|
`ExecStart`. `stack-upgrade.sh upgrade` and `rollback` check both before anything stops, when the
|
|
profile is `library` (no `--profile` line, or the last one says `library`):
|
|
|
|
- `--send` after `--` in the args file refuses `args-unreadable`.
|
|
- A `--send` token after `serve` in `/etc/systemd/system/mosaic-webui.service`'s `ExecStart` refuses
|
|
`console-unit-legacy`. The current `case2-unit.sh` refuses that installed unit, so an upgrade or
|
|
rollback would stop both units and then fail both renders.
|
|
|
|
Both refusals leave the units running, and the script rewrites neither file. Check the installed unit
|
|
read-only, as root:
|
|
|
|
```sh
|
|
grep -n '^ExecStart=' /etc/systemd/system/mosaic-webui.service | grep -c -- ' --send'
|
|
```
|
|
|
|
- `0`, and no installed unit carries `--send`: removing the args line is enough. Open
|
|
`/etc/mosaic-stack/dev.args` and delete the one line that reads `--send`, changing nothing else. Check
|
|
that `stat -c '%U %a' /etc/mosaic-stack/dev.args` still gives `root` and a mode with no group or other
|
|
write, and that `grep -cx -- --send /etc/mosaic-stack/dev.args` prints `0`. Then rerun the same
|
|
`stack-upgrade.sh` command.
|
|
- `1` or more: the host needs a reviewed migration of the unit, the args file and the release pointer
|
|
together, and none exists yet. Stop. Leave the units running, change neither file and report to the
|
|
lead with the command's output. Removing only the args line does not help: the run then refuses
|
|
`console-unit-legacy`.
|
|
|
|
The library Console stays read-only after the upgrade (#385). From M3 it signs in through SSO
|
|
accounts, as the migration below sets up.
|
|
|
|
### Migrating a host to SSO accounts (M3, #1873)
|
|
|
|
A host re-pinned onto an M3 copy migrates first, in this order. The order is the same for the `factory`
|
|
(Case 1 and .53) and `library` (.54) profiles. Each step is a live step that needs Mos's go. Nothing in the
|
|
stack picks or invents a subject: the subjects come from `console.sso`, which a human prepares, and
|
|
`accounts.json` comes only from the human verb `mosaic accounts init`.
|
|
|
|
1. Check or create `.deployed` through the installer. The stack's user's `case2-install.sh` (section 6)
|
|
writes `<dataRoot>/.deployed` naming the instance at its step 2. It never replaces an existing one, and
|
|
it refuses one that is invalid or names another instance. Check it read-only:
|
|
`stat -c '%U %a' <dataRoot>/.deployed` prints `mosaic 444`.
|
|
2. Prepare SSO. A human adds `console.sso` to `/srv/mosaic/.config/mosaic-dev/config.json`, with its
|
|
providers, their subjects and its client-secret slots, per
|
|
`docs/plans/2026-11-10_accounts-design.md`. Jason places the client secrets (section 9).
|
|
3. Run `mosaic accounts init --name <name>` as `mosaic`. This is a human verb, run from a human shell and
|
|
never by an agent. It creates the one operator account from the configured `console.sso` subjects and
|
|
refuses if `accounts.json` exists. It also refuses `accounts-unsafe` when any folder above the config
|
|
directory is not root's or `mosaic`'s, or is group or other writable without the sticky bit (#1919).
|
|
Check first, read-only: every line printed by
|
|
`p=/srv/mosaic/.config/mosaic-dev; while [ "$p" != / ]; do stat -c '%U %a %n' "$p"; p=$(dirname "$p"); done; stat -c '%U %a %n' /`
|
|
names `root` or `mosaic`, and its mode has no group or other write bit unless it starts with 1 (sticky).
|
|
The config directory itself is `mosaic`'s with no group or other write bit. A group-writable `.config`
|
|
refuses on purpose; fix the mode, never the check.
|
|
4. Provision the dedicated uid. `mosaic` must keep section 1's shape: a non-login shell, the only member
|
|
of its primary group, and owner of its config directory and data root. An accounts instance never runs
|
|
under a human or development seat uid, and never under the `webui-service.sh` user units.
|
|
5. Re-pin with `--next-args`. Leave `/etc/mosaic-stack/dev.args` as it is. Write the new flags to
|
|
`/etc/mosaic-stack/dev.args.next`, root's, mode 0640: the same lines before `--`, and after `--` the
|
|
serve flags with `--sso` added (`factory`: `--sso --send`). For `library`, also remove the
|
|
proxy-preview trio (`--public-host`, `--auth-header`, `--allow-user`) and the proof pair
|
|
(`--proof-header`, `--proof-file`). Then run
|
|
`stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA --next-args`.
|
|
|
|
A plain re-pin refuses `render-refused` here, because an installed unit must equal the render but for
|
|
the copy id. With `--next-args`, before anything stops, the driver checks the installed copy against
|
|
`dev.args` and the target as a first install from `dev.args.next`. It refuses `next-args-differ` when
|
|
the files differ before `--` (other than `--copy`), and `next-args-same` when they do not differ after
|
|
it. After the stage it moves the installed unit files into a new 0700 directory
|
|
`/etc/mosaic-stack/dev.replaced.XXXXXX` and renders the target as a first install. A failure from
|
|
there moves them back unchanged and starts the old copy again. After `confirm` passes,
|
|
`dev.args.next` becomes `dev.args`, and the replaced directory keeps the old unit files and the old
|
|
`dev.args`. The result line names the directory. Nothing removes it; root removes it by hand once no
|
|
rollback needs it.
|
|
|
|
To roll back past the migration, put the replaced args back as the next args and roll back with
|
|
`--next-args`:
|
|
|
|
```sh
|
|
install -m 0640 -o root -g root /etc/mosaic-stack/dev.replaced.XXXXXX/dev.args /etc/mosaic-stack/dev.args.next
|
|
stack-upgrade.sh rollback --instance dev --to <old copy id> --next-args
|
|
```
|
|
|
|
For `library` this is the only rollback to a copy older than M3: `case2-unit.sh` refuses `--sso` for
|
|
an unstamped library copy (#416), so the old copy comes back with its old flags, proxy-preview
|
|
admission included, as before the migration. A `factory` rollback may instead keep `--sso --send`
|
|
with a plain `rollback`.
|
|
|
|
If a step is missing, `case2-unit.sh` refuses before it writes a unit and prints the missing steps,
|
|
numbered in this order, for example:
|
|
|
|
```text
|
|
case2-unit: refusing the units for <sha>: a stamped copy's Console starts only with SSO accounts; missing steps, in order:
|
|
1. check or create .deployed: mosaic runs case2-install.sh, whose deployment step writes <dataRoot>/.deployed
|
|
2. prepare SSO: console.sso in mosaic's system config /srv/mosaic/.config/mosaic-dev/config.json, with its client secret slots
|
|
3. run `mosaic accounts init` as mosaic, a human verb: no /srv/mosaic/.config/mosaic-dev/accounts.json
|
|
4. add --sso to the serve flags, then re-pin
|
|
```
|
|
|
|
Without `console.sso`, the render does not check `accounts.json`, because its subjects name providers SSO
|
|
has not configured yet. The next render after step 2 checks it. A rollback never removes accounts.json,
|
|
its receipts or `.deployed` (`docs/plans/2026-11-10_accounts-design.md`, section 7).
|
|
|
|
Precheck line: none. `stack-upgrade.sh` refuses `args-unreadable`, `console-unit-legacy`, `render-refused`,
|
|
`next-args-differ` or `next-args-same` on any of these rules, before it stops a unit.
|
|
|
|
## 8. Enrollment, once per host
|
|
|
|
The machine-grant enrollment is Gizmoduck's design r6, commit f9039682e, section 3, steps E1-E4. ops-01's
|
|
Q-O1 is answered (mos-1125). The host side is the `tools/host/` files: `mosaic-keys`, `mosaic-keys-write`,
|
|
their sudoers line and the sshd drop-in `/etc/ssh/sshd_config.d/<nn>-mosaic-keys.conf`. The verbs that run
|
|
it (`hosts.enroll`, the `host-enroll` decision) are rows 234 and 235, not yet on refactor. This item has no
|
|
commands until they land.
|
|
|
|
Precheck line: none. The enrolled host's key and its `mosaic-keys` sudoers line: precheck line missing.
|
|
|
|
## 9. Credential slots
|
|
|
|
Jason places these, all in one sitting per host. No value appears here or in any message. Check a placed
|
|
file with `stat -c '%U %a %n' PATH`, never `cat`. `<business>` is the host's business id.
|
|
|
|
### .53, factory
|
|
|
|
| Path | File | Mode | Owner | Profile | Precheck line |
|
|
|---|---|---|---|---|---|
|
|
| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `CLAUDE_CODE_OAUTH_TOKEN=...` (Q185) | 0600 | mosaic | factory | `claude replaces its prompt with --system-prompt-file` runs claude with it. The file's mode and owner: precheck line missing. |
|
|
| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
|
|
|
|
### .54, library
|
|
|
|
| Path | File | Mode | Owner | Profile | Precheck line |
|
|
|---|---|---|---|---|---|
|
|
| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `ZAI_API_KEY=...` | 0600 | mosaic | library | `<path> sets ZAI_API_KEY` |
|
|
| `/srv/mosaic/.config/mosaic-dev/secrets/` | before M3 only, and an unstamped copy only: the Console's proof file, the path the args file's `--proof-file` names. A stamped copy refuses `--proof-file` (section 7). The file is one base64url line, at least 32 `[A-Za-z0-9_-]` characters, with one optional trailing LF (`packages/webui/src/serve.mjs:263-264`) | 0600 or 0400 | mosaic | library | none. Render checks metadata only and never opens the file (`scripts/case2-unit.sh:397-408`). Bad bytes refuse at Console start with exit 2 (`packages/webui/src/serve.mjs:267`, `packages/webui/src/cli.mjs:138-140`). Precheck line missing. |
|
|
| `/srv/mosaic/.config/mosaic-dev/secrets/<business>/` | `library-gitea` | 0600 | mosaic | library | After a unit start, wait for loopback `/healthz` with the bounded retry in section 10 before `case2-precheck.sh --business <business> --library-status`: `library-last-error library-credential-missing` until it's placed, then `library <project> last fetched ...` |
|
|
| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
|
|
|
|
The installer key, `<configDir>/secrets/hosts/<business>/<hostId>/installer`, is not a slot anyone places.
|
|
Enrollment (section 8) writes it under the enrolling stack's config directory and deletes it when the host
|
|
leaves.
|
|
|
|
## 9a. Notifier decision, before starting units
|
|
|
|
Make the explicit per-host decision before section 10: the default below is
|
|
`binding: null`, so the bus runs without DMs. A named binding is a later per-host
|
|
choice that needs Mos's go. The installer does not make this decision or write
|
|
this file. Replace `<dataRoot>` with the configured data root and `<business>`
|
|
with this host's business id. If the file already exists, stop and review the
|
|
existing decision; this command refuses to overwrite it.
|
|
|
|
```sh
|
|
sudo -u mosaic -H /bin/sh <<'MOSAIC_NOTIFY'
|
|
cd / || exit 1
|
|
umask 077
|
|
notify_root='<dataRoot>/notify'
|
|
notify_dir="$notify_root/<business>"
|
|
mkdir -p -m 0700 -- "$notify_root" && chmod 0700 -- "$notify_root" || exit 1
|
|
mkdir -p -m 0700 -- "$notify_dir" && chmod 0700 -- "$notify_dir" || exit 1
|
|
(set -C; printf '%s\n' '{"notifyVersion": 1, "binding": null}' > "$notify_dir/notify.json") || exit 1
|
|
chmod 0600 -- "$notify_dir/notify.json"
|
|
MOSAIC_NOTIFY
|
|
```
|
|
|
|
Precheck line: `ok notifier config <dataRoot>/notify/<business>/notify.json validates (directory 0700, file 0600)`.
|
|
|
|
## 10. Console start
|
|
|
|
None by hand. I2's `case2-unit.sh` enables both units, starts the bus, then the Console (Q189, mos-1122).
|
|
`stack-upgrade.sh` stops and starts both units itself.
|
|
|
|
After a unit start, wait for the Console's loopback `/healthz` to succeed before running
|
|
`case2-precheck.sh`, including `--library-status` in section 9. Replace `<port>` with the installed
|
|
Console unit's `--port`. This retry is bounded to 30 attempts, each with a 2-second timeout, with
|
|
1 second between failed attempts. If it prints STOP, stop here; do not run the precheck or adopt.
|
|
|
|
```sh
|
|
(
|
|
attempts=0
|
|
until curl -fsS --max-time 2 -o /dev/null 'http://127.0.0.1:<port>/healthz'; do
|
|
attempts=$((attempts + 1))
|
|
[ "$attempts" -lt 30 ] || { echo "STOP: loopback healthz did not become ready; do not continue" >&2; exit 1; }
|
|
sleep 1
|
|
done
|
|
)
|
|
```
|
|
|
|
Precheck line: none. Both units active after the render: precheck line missing.
|