mosaicstack/stack
25
0
Fork 1
Code Issues 399 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

agents: row 195 Case 1 install design r5 (#1693): Duckworth r4 B1 (one result line per activation, FAILED file and --cause, marker written first, refusals log no activation, real-sequence tests), B2 per mos-1098/mos-1101 (case1-pins.sh and case1-hostcheck.sh, seven-item drift stop with apt hook and boot unit, ops-01 owner, bwrap closed on drift), N1-N4; Console on 127.0.0.1:732 with CAP_NET_BIND_SERVICE, tester 7333 as P10; chains on row 190 r3 and row 230 (C2-13a), node under /opt, four factory profile items for Gosalyn

Browse Source
This commit is contained in:
jason.woltje
2026-10-20 16:46:21 -05:00
parent 0075be3581
commit 0b6d29e909
1 changed files with 404 additions and 118 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agents/darkwing/work/queue-195/DESIGN.md
+404 -118
View File
@@ -1,46 +1,58 @@
# Row 195 design: the Case 1 install on 10.1.1.53 (r4)
# Row 195 design: the Case 1 install on 10.1.1.53 (r5)
Darkwing, 2026-10-20. Brief: `docs/plans/2026-10-20_hub-own-uid.md`, blob 45be7f09afc1 (f24f2cd40,
sage-1439). Gitea #1693. Reviewer Duckworth, gate owner Sage. Chained on row 190's candidate (537a02392
on `darkwing/row190-r2`, round 2, the config canon) and on C2-13's candidate (Gosalyn,
`scripts/case2-install.sh`, `case2-precheck.sh`, `case2-unit.sh`), Q177. C2-13 has no candidate yet. The
interface I read is Gosalyn's design r10 at 4281b0064 on `gosalyn/case2-design` and gosalyn-0046. Sage
(sage-1481) has asked Gosalyn to split out C2-13a in r11: `case2-unit.sh` and `case2-precheck.sh` with
both system units, Webby's r10 render rules (`@PATH@`, `@NODE@`, `@FLAGS@`), the hardening list, and no
`[Install]` on the Console unit until Jason rules. Row 195 chains on C2-13a's candidate once that row
exists. Until r11 is approved I build against 4281b0064 and expect a rebase.
sage-1439). Gitea #1693. Reviewer Duckworth, gate owner Sage. Chained on row 190's candidate (round 3,
df65a3d4c on `darkwing/row190-r3`, rebased on row 192; the config canon) and on row 230's candidate
(C2-13a, #1732, Gosalyn: `scripts/case2-unit.sh` and `case2-precheck.sh`), Q177, sage-1499. Row 230's
brief is `docs/plans/2026-10-20_case2-rows.md` "C2-13a" at 5b8def2f1, from Gosalyn's design r11
(eac92d956 on `gosalyn/case2-design`, approved webby-0038). It has no candidate yet, so I read that
brief as the interface and expect a rebase. Section 2 lists what Case 1 needs from it that the brief
doesn't give.
Changes from r3 (Duckworth's REVIEW-r2.md at 9ba8385a9, duckworth-0050; mos-1096; gosalyn-0046):
Changes from r4 (Duckworth's REVIEW-r3.md at 9f4d5556b on `duckworth/reviews`; mos-1098; mos-1101;
gosalyn-0049):
- B1. Root no longer needs any value that only `mosaic`'s data holds. `confirm` and `revert` take no
activation id: each finds the one open activation itself and refuses unless there is exactly one, and
that one names the copy root passed. Rollback takes `--to ID` from the go, and A1's plan refuses
unless ID is `from_log previous`. A2's failure has its own row, and root keeps its own step log.
Section 4, tests in section 7.
- B2. `confirm` runs `gate` (:522), which polls until `--timeout`. .53 runs the PM. Under
`--units external` the PM check reads `launches list` alone, with this start's time taken from the
bus unit's `ActiveEnterTimestamp`, so it reads no journal. `pm-on` and `--pm-after-harness` refuse
under external. Section 4, tests and mutants in section 7.
- UNITS is settled (gosalyn-0046, design r10): `[email protected]`, then
`mosaic-webui.service`, both from the existing templates. The full `case2-unit.sh` call is in
section 1, and root keeps its arguments in a root-owned file so a re-pin changes only the copy id.
- `DISABLE_AUTOUPDATER` goes in `/etc/claude-code/managed-settings.json`, root-owned. The session
allowlist doesn't pass the variable, and `mosaic`'s `~/.claude` doesn't reach a role session either
(`packages/seat/src/session.mjs:92-93`), so gosalyn-0046 item 3's settings file wouldn't reach one.
The sandbox mounts `/etc` read-only (`packages/seat/src/sandbox.mjs:85`). Section 1.
- N1 and mos-1096 item 2. Ubuntu 24.04 ships `bwrap-userns-restrict`. r4 installs that one file from
the pinned `apparmor-profiles` deb, checked by sha256, in place of r3's own profile. `/usr/bin/bwrap`
gets a `dpkg-statoverride` to `root:mosaic` 0750. Section 1 has both, each with its rollback, and
section 5's P9 and F5 check them.
- P9 is accepted by Mos (mos-1096 item 3). N2: P5 closes after #423 step 4. N3: the marker is
`AWAITING-UNITS`. N4: `--migrate` under external runs only with UNITS stopped, with a test. N5 is moot
now that both units come from C2-13.
- Jason's Q9 (install and upgrade) is open. Mos: carry on, since the re-pin and rollback work is
needed whatever he answers. Section 6.
- B1. An activation now has exactly one result line, written at the end. A5's gate failure and A3's
failure after its capture write no log line. Each writes a `FAILED` file next to `AWAITING-UNITS` and
exits 1, so the activation stays open for R2 and R4, and R4 writes the result. A3 writes
`AWAITING-UNITS` right after `started`. Refusals by `confirm`, `revert` and `gate` log no activation
id, so a refused `confirm` can't close one either. There are real-sequence tests on one log, and
mutants that put back r4's interim line, r4's marker order and a refusal that names the activation.
Section 4 and section 7.
- B2. The bwrap profile has an owner and a stop. `scripts/case1-hostcheck.sh` checks the profile file
against the pin in `scripts/case1-pins.sh`, the two loaded profiles, the live label of a process
bwrap runs, the statoverride and mode, that no other profile names bwrap, two sysctls and the port
floor. It has three callers:
1. A1 of every re-pin refuses `profile-drift`.
2. A root apt hook runs it after every package run, unattended ones included.
3. A boot unit runs it before the bus starts.
On drift, callers 2 and 3 unload the profile and close `/usr/bin/bwrap` to `mosaic`, so every role
session refuses to start. They write `DRIFT` and log it, and only root clears it. ops-01 owns the host
side under a Mos go: it reads the log in each STATE report and, before each re-pin window, compares
the upstream profile to the pin. A changed upstream profile becomes a new pin, which is its own review
round. Section 1 has the details. The facts behind it are in section 0: unattended-upgrades runs daily
on .53 with the security pocket, and the pinned bubblewrap is in that pocket.
- N1 and mos-1098 point 2. P9 has a seventh item: no `mosaic-bwrap` profile is loaded, and no profile
other than Ubuntu's attaches to `/usr/bin/bwrap`. r4 never installed `mosaic-bwrap`. F5 now compares
exact strings. P9 reads `accepted mos-1096` only while all seven items hold.
- N2: a role session gets no capabilities, so a nested bwrap fails. Section 1. N3: A3 refuses "before
it logs `started`". N4: the stale-start rule and its fixtures handle `ActiveEnterTimestamp`'s whole
seconds.
- The port squat (mos-1101 item 5, gosalyn-0049). It applies: role sessions share the host's network
namespace (`packages/seat/src/sandbox.mjs:67` unshares no net), so while the Console is down a
session can bind its loopback port. The Console moves to 127.0.0.1:732 with
`CAP_NET_BIND_SERVICE` on its unit (row 230's line), so no session can take it. The tester port stays
7333, because serve refuses a tester port below 1024. That residue is P10, accepted under Mos's
reading. Case 1 can't meet the preview trio or the proof pair, because .53 has no proxy. Section 2
names the factory serve set for row 230.
- Jason's Q9 (Vikunja #429) is still open. If install and upgrade become product commands, these
steps become that installer's configuration and row 195 rebases on its row (sage-1491). Section 6.
## 0. What I checked on the host
Read only, as `localadmin` over ssh, 2026-10-20 between 20:40Z and 21:00Z (the brief allows this):
Read only, as `localadmin` over ssh, 2026-10-20 between 20:40Z and 21:00Z, and again at 21:27Z for the
last seven rows (the brief allows this):
| Item | 10.1.1.53 today | Needed |
|---|---|---|
@@ -54,11 +66,18 @@ Read only, as `localadmin` over ssh, 2026-10-20 between 20:40Z and 21:00Z (the b
| journald | `SplitMode=uid` (the default) | section 4, B2 |
| `useradd` | `USERGROUPS_ENAB yes` | 2a passes `--user-group` anyway |
| claude | not on localadmin's PATH | the factory's harness; dragon-lin runs 2.1.304 |
| ports 7330-7333 | free (no listener) | Console, board, tester |
| ports 7330-7333 | free (no listener) | tester |
| `localadmin` | NOPASSWD sudo (#423 step 1, applied 20:05:47Z, mos-1066, mos-1077) | the root steps run through it |
| `jarvis` | groups sudo, docker, lxd, adm | #423 step 4 removes all four (section 5, P5) |
| unattended-upgrades | on: `APT::Periodic::Unattended-Upgrade "1"`, `apt-daily-upgrade.timer` enabled. Allowed origins `${distro_id}:${distro_codename}`, `-security`, ESM apps-security and ESM infra-security. No `Package-Blacklist` | B2: packages change daily without a window |
| bubblewrap's pocket | `0.9.0-1ubuntu0.3` is in both `noble-updates` and `noble-security`, so its later security updates install unattended | B2 |
| apparmor | installed `4.0.1really4.0.1-0ubuntu0.24.04.8`. The `.9` candidate is in `noble-updates` only, which unattended-upgrades doesn't take. `aa-status --json` works | B2 |
| profiles that name bwrap | no file under `/etc/apparmor.d` mentions `/usr/bin/bwrap` | N1, P9 item 7 |
| `net.ipv4.ip_unprivileged_port_start` | 1024 | the port squat |
| listeners below 1024 | 22 (sshd) and 53 (the resolver) only; 732 is free | the Console's port (section 1) |
| other listeners | 4003, 6052, 6055, 8123, 8765, 14242, 18555, and on loopback 1933, 5905, 6085, 6379, 6380, 9119, 9225, 18554, 18789, 38147 | none in 7330-7333 |
## 1. The install as inputs to C2-13's scripts
## 1. The install as inputs to the Case 2 scripts
Instance. Row 190's canon with the default name, `dev`, as Case 2 uses it (Gosalyn design 9). `~` is
`mosaic`'s home, `/srv/mosaic`, so:
@@ -70,45 +89,79 @@ Instance. Row 190's canon with the default name, `dev`, as Case 2 uses it (Gosal
| config dir | `/srv/mosaic/.config/mosaic-dev` |
| data root | `/srv/mosaic/.mosaic-dev`, marker `instance.json` `{"instanceVersion":1,"name":"dev"}` |
| business | `mosaic-stack`, file `<configDir>/businesses/mosaic-stack.json` (section 2) |
| profile | `--profile factory` (C2-13) |
| profile | `--profile factory` (row 230, C2-13b) |
| roles | the factory set in section 2 |
| human id | Jason's id, given in the window's go. Not `tester` (`broker.mjs:245`) |
| model credential | `<configDir>/secrets/claude-session.env`, 0600 (row 190's `SESSION_ENV` for the default instance) |
| token slot path | none for this row (section 2) |
| Console, board, tester ports | 7332, 7331, 7333, the default instance's ports |
| harness | claude at `/srv/mosaic/.local/bin/claude`, pinned through `MOSAIC_CLAUDE_BIN` (decision 91) |
| node | `/srv/mosaic/.local/node/bin/node`, a pinned 26.x release |
| PM | on: the bus serve flags carry `--pm` (section 4, B2) |
| Console port | 127.0.0.1:732, below the port floor, bound through the Console unit's `CAP_NET_BIND_SERVICE` (row 230). The port squat below says why |
| tester port | 127.0.0.1:7333 (P10) |
| board | none. UNITS has no board unit, and the Console gets no `--board` |
| harness | claude at `/srv/mosaic/.local/bin/claude`, pinned through `MOSAIC_CLAUDE_BIN` (decision 91), which the bus unit's `@CLAUDE_ENV@` line sets (section 2) |
| node | `/opt/mosaic-stack/node-<version>/bin/node`, a pinned 26.x release, `root:root`. Row 230 refuses a `--node` under the home or not owned by root |
| PM | on: the bus unit's `@PM_FLAGS@` renders ` --pm` (section 2, section 4) |
| source | `/srv/mosaic/src/mosaic-stack` (`mosaic`'s checkout, section 4) and `/root/mosaic-stack` (root's) |
Why `dev` and not a named instance: one instance per OS user, and C2-13 already assumes
Why `dev` and not a named instance: one instance per OS user, and Case 2's rows already assume
`/srv/mosaic/.config/mosaic-dev`. A second instance under `mosaic` on .53 would need a name, and row 190
makes that a flag, not a move.
Why node lives under `/srv/mosaic`: changing `/usr/local/bin/node` would change what Hermes and anything
else on the host runs, and Q181 keeps Hermes running. A per-user node touches no other account.
Why node lives under `/opt/mosaic-stack`: changing `/usr/local/bin/node` would change what Hermes and
anything else on the host runs, and Q181 keeps Hermes running. r4 put node under `/srv/mosaic`, but row
230 requires a root-owned `--node` outside the home. Nothing else on the host puts `/opt/mosaic-stack`
on its PATH.
The units (gosalyn-0046, design r10). Two system units, from the existing templates, in start order:
PATH for `mosaic`'s steps. `sudo -u mosaic -H` resets PATH to sudo's `secure_path`, which finds
`/usr/local/bin/node`, v22. The precheck would then refuse, and `scripts/mosaic` would run v22 (it runs
`exec node` from PATH, `scripts/mosaic:22-40`). So every `mosaic` step in the run sheet and in
`case1-repin.sh` runs as `sudo -u mosaic -H env PATH=<node dir>:/usr/local/bin:/usr/bin:/bin ...`, the
list row 230 renders as the units' `@PATH@`. `case1-repin.sh` builds it from the `--node` in
`case1-unit.args`.
The units (row 230, design r11). Two system units, from the existing templates, in start order:
`[email protected]` from `packages/cli/systemd/mosaic-bus.service.in`, then
`mosaic-webui.service` from `packages/webui/systemd/mosaic-webui.service.in`. Root stops them in the
reverse order. `@REPO@` is the copy dir, and `%h` becomes the literal home. The install call is:
reverse order. `@REPO@` is the copy dir, and `%h` becomes the literal home. Row 230 adds
`ProtectSystem=strict`, `ReadWritePaths=/srv/mosaic`, `ProtectHome=yes` and `PrivateTmp=yes` to both, and
`AmbientCapabilities=CAP_NET_BIND_SERVICE` with `CapabilityBoundingSet=CAP_NET_BIND_SERVICE` to the
Console only. The Console unit has no `[Install]`, so root starts it by hand after a reboot until Jason
rules (mos-1097). The install call is:
```
case2-unit.sh --user mosaic --home /srv/mosaic --business mosaic-stack --copy ID \
--node /srv/mosaic/.local/node/bin/node -- <serve flags>
case2-unit.sh --profile factory --user mosaic --home /srv/mosaic --business mosaic-stack --copy ID \
--node /opt/mosaic-stack/node-<version>/bin/node --pm --claude /srv/mosaic/.local/bin/claude \
-- --port 732 --business mosaic-stack --tester-port 7333
```
The serve flags are `--pm` plus whatever the window's go adds. A re-install must pass the same
`--node` and serve flags, and only the copy id may change (gosalyn-0046 item 2). So 2c writes the
whole argument list, without `--copy`, to `/etc/mosaic-stack/case1-unit.args`, `root:root` 0644, one
argument per line. Every later `case2-unit.sh` call in section 4 reads that file and adds `--copy`.
`mosaic` can't write it. Changing a serve flag is a re-install with its own go, not a re-pin.
`--pm` and `--claude` are the two inputs row 230's brief doesn't have yet (section 2). A re-install
must pass the same `--node`, flags and serve flags, and only the copy id may change (row 230's
re-install check). So 2c writes the whole argument list, without `--copy`, to
`/etc/mosaic-stack/case1-unit.args`, `root:root` 0644, one argument per line. Every later
`case2-unit.sh` call in section 4 reads that file and adds `--copy`. `mosaic` can't write it. Changing
a serve flag is a re-install with its own go, not a re-pin.
Node download (N4). The run sheet pins one release, its tarball name and its sha256. As `mosaic`:
fetch the tarball, `SHASUMS256.txt` and `SHASUMS256.txt.asc`; `gpg --verify` the `.asc` against the
Node release keys listed in the nodejs/release-keys repository, imported into a keyring under
`/srv/mosaic/.local/node-keys`; check that the tarball's line in the verified file is the pinned sha256;
then `sha256sum -c` the tarball and unpack. Any mismatch stops the step.
The port squat (mos-1101 item 5). Role sessions share the host's network namespace:
`packages/seat/src/sandbox.mjs:67` unshares the user, PID and IPC namespaces and not the network. So
while the Console is down, which A2 to A4 of every re-pin and any crash before `Restart=on-failure`
brings it back both cause, a session can bind 127.0.0.1:7332. It would then receive Jason's `ssh -L`
traffic, and it would answer the gate's `/healthz` (`stack-release.sh:485-487`) in the Console's place.
At 732 that can't happen. The floor is 1024 (section 0), a session has no capabilities (below), and
`mosaic`'s own processes outside the sandbox have none either. Only the Console unit holds
`CAP_NET_BIND_SERVICE`. Its children, the human-cli and read scripts it spawns
(`packages/webui/src/bus.mjs:31`, `reads.mjs:79`), inherit the ambient capability. No role session
descends from the Console: sessions are children of the bus unit, which has no capability line. The
gate needs no change, because `console_port` (`stack-release.sh:213-215`) reads the Console's `--port`
from the pointer. Row 230's precheck refuses when the floor is at or below the Console's port, and
`case1-hostcheck.sh` checks it after every package run (B2). The tester port can't go below 1024,
because serve refuses that (`packages/webui/src/cli.mjs:33`), so P10 records it.
Node download (N4). The run sheet pins one release, its tarball name and its sha256. As root, in a
root-owned temp dir: fetch the tarball, `SHASUMS256.txt` and `SHASUMS256.txt.asc`; `gpg --verify` the
`.asc` against the Node release keys listed in the nodejs/release-keys repository, imported into a
keyring under `/etc/mosaic-stack/node-keys`; check that the tarball's line in the verified file is the
pinned sha256; then `sha256sum -c` the tarball and unpack it to `/opt/mosaic-stack/node-<version>`,
`root:root`, no file writable by group or other. Any mismatch stops the step. Rollback: `rm -r` that
dir.
Claude (N5). The run sheet pins the version, 2.1.304 unless the window's plan names a newer one. As
`mosaic`: `npm install --prefix /srv/mosaic/.local @anthropic-ai/claude-code@<pin>`, with `@<pin>`
@@ -147,27 +200,92 @@ order, each with its rollback. The `mosaic` group must exist first (2a).
every program it runs is stacked under `unpriv_bwrap`, which has the same file and network rules
with `audit deny capability`. So a role session gets a user namespace and no capabilities in it.
The sysctl stays at 1, so nothing else on the host gains user namespaces.
One consequence (N2): a bwrap that a session starts inside its own sandbox fails, because it needs
capabilities in the namespace. Claude Code's own sandbox is such a bwrap, and it's off by default,
so a role session doesn't use it. F3 shows a session still answers under the profile.
Rollback: `apparmor_parser -R /etc/apparmor.d/bwrap-userns-restrict`, then `rm` it.
3. `dpkg-statoverride --update --add root mosaic 0750 /usr/bin/bwrap` (Mos). Only root and the
`mosaic` group can read or run bwrap, and package upgrades keep the mode. Rollback:
`dpkg-statoverride --remove /usr/bin/bwrap`, then `chown` and `chmod` to the values in
`bwrap.orig`.
4. The drift stop (B2, below): `case1-hostcheck.sh` and `case1-pins.sh` from `/root/mosaic-stack` to
`/usr/local/lib/mosaic-stack/`, `root:root` 0755 and 0644; the apt hook
`/etc/apt/apt.conf.d/80mosaic-hostcheck`; the boot unit `mosaic-hostcheck.service`, enabled. Then
`case1-hostcheck.sh --check` exits 0. Rollback: disable and remove the unit, `rm` the hook file and
the dir.
Full rollback runs 3, 2, 1. Tests after 2a' and again in section 5: as `mosaic`, `bwrap --unshare-all
--ro-bind / / true` exits 0 (C2-13's precheck line, F4); as `jarvis`, `/usr/bin/bwrap --version`
Full rollback runs 4, 3, 2, 1. Tests after 2a' and again in section 5: as `mosaic`, `bwrap --unshare-all
--ro-bind / / true` exits 0 (row 230's precheck line, F4); as `jarvis`, `/usr/bin/bwrap --version`
fails with "Permission denied" (P9); `getent group mosaic` lists no member and no other account has
gid `mosaic` (P9); `fs.protected_hardlinks` is 1 (P9). A role session running under the profile is F5.
One more test runs in 2a', because I can't settle it from here. Row 230's units keep the templates'
`NoNewPrivileges=yes`, and AppArmor limits profile changes for a task with no-new-privs set. The
`bwrap` attach happens on exec from an unconfined task, and the child's change is a stack
(`bwrap//&unpriv_bwrap`), which only narrows. I expect both to pass under no-new-privs, since bwrap
sets no-new-privs on itself before it runs the child, and Ubuntu's profile works with that. But I
haven't seen it run, and .53 is read-only to me. So as root, after step 4: `systemd-run --wait --pipe
--uid=mosaic --gid=mosaic -p NoNewPrivileges=yes -p ProtectSystem=strict -p ProtectHome=yes -p
PrivateTmp=yes /usr/bin/bwrap --unshare-user --unshare-pid --ro-bind / / cat /proc/self/attr/current`
prints `bwrap//&unpriv_bwrap (enforce)`. If it fails, the window stops at 2a' and rolls back 4 to 1,
before any `mosaic` step. That beats finding it at F5.
The drift stop (B2, Duckworth's REVIEW-r3 B2, mos-1098 point 1). r4 installed a profile file that no
package owns, from a pinned deb, onto a host where unattended-upgrades changes packages every day
(section 0). Three things can move under it: bubblewrap (its security updates install unattended), the
`apparmor` package and its parser, and any new profile a package ships that attaches to
`/usr/bin/bwrap`. Two more host values matter to this design: the user-namespace sysctl and the port
floor. So:
- `scripts/case1-pins.sh` holds the pins, sourced by the scripts below: the `apparmor-profiles` version
the file came from, the file's sha256, and the bubblewrap version 2a' installs. Changing a pin is a
reviewed commit and its own round. The bubblewrap version is an install pin, not a drift item:
a security update to bubblewrap keeps the statoverride, and the profile attaches by path.
- `scripts/case1-hostcheck.sh` checks seven items and prints one line each:
1. `/etc/apparmor.d/bwrap-userns-restrict` has the pinned sha256, and `dpkg -S` finds no package
that owns it.
2. `aa-status --json` lists `bwrap` and `unpriv_bwrap` in enforce mode, and no loaded profile named
`mosaic-bwrap`.
3. No other file under `/etc/apparmor.d` names `bwrap`.
4. The live label: as `mosaic`, `/usr/bin/bwrap --unshare-user --ro-bind / / cat
/proc/self/attr/current` prints exactly `bwrap//&unpriv_bwrap (enforce)`. This catches a profile
that attaches by a glob that item 3's search misses.
5. `dpkg-statoverride --list /usr/bin/bwrap` reads `root mosaic 0750`, and `stat` agrees.
6. `kernel.apparmor_restrict_unprivileged_userns` is 1 and `fs.protected_hardlinks` is 1.
7. `net.ipv4.ip_unprivileged_port_start` is above the Console's `--port` in the installed unit.
`--check` exits 0 when all hold and 1 otherwise. `--enforce` runs the same check and always exits 0.
On a failure, or while `/etc/mosaic-stack/DRIFT` exists, it applies the stop:
`apparmor_parser -R` on the profile if loaded, then `chown root:root` and `chmod 0700` on
`/usr/bin/bwrap`. It writes the failed items to `DRIFT` and appends a line to
`/var/log/mosaic-stack/hostcheck.log`. With bwrap closed to `mosaic`, every role session refuses
to start. That's the fail-closed state, and the stack needs no change for it. The chmod is undone by
the next bubblewrap upgrade, which reapplies the statoverride. But the apt hook runs again right
after that upgrade and finds `DRIFT`, so it closes bwrap again. Only root removes `DRIFT`, in a
reviewed step with a new pin or a Mos go. I don't call `dpkg-statoverride` from inside the hook,
because I haven't confirmed it can take dpkg's lock while apt holds the frontend lock.
- Three callers. A1 of every re-pin runs `--check` and refuses `profile-drift` on exit 1. The apt hook
is `DPkg::Post-Invoke { "/usr/local/lib/mosaic-stack/case1-hostcheck.sh --enforce || true"; };`, so it
runs after every package run, unattended ones included, and can't fail one. The boot unit,
`mosaic-hostcheck.service` (`Type=oneshot`, `After=apparmor.service`,
`Before=mosaic-bus@mosaic-stack.service`, `WantedBy=multi-user.target`), runs `--enforce` before the
bus starts, which covers a kernel or parser change that takes effect at boot.
- Owner. ops-01 owns the host side under a Mos go. It reads `hostcheck.log` in each STATE report and
sends any `DRIFT` to Mos. Before each re-pin window it also runs `--upstream`, which downloads the
current `apparmor-profiles` candidate into a temp dir and compares its `bwrap-userns-restrict` to the
pin. A difference goes to Mos as a new-pin request and refuses nothing. The pins and the scripts are
this row's, and later changes go through review like any other.
Who runs what:
| Step | Runs as | Through |
|---|---|---|
| 2a: `useradd --system --user-group --home-dir /srv/mosaic --shell /usr/sbin/nologin mosaic`, `/srv/mosaic` 0700, `/etc/mosaic-stack` `root:root` 0755, `/var/log/mosaic-stack` `root:root` 0750 | root | ops-01, `ssh localadmin@10.1.1.53`, `sudo` (#423 step 1) |
| 2a': bubblewrap, the AppArmor profile, the statoverride and the managed settings file, each with the test and rollback above | root | ops-01, same path, R27 (mos-1091) |
| 2a': node under `/opt/mosaic-stack`; bubblewrap, the AppArmor profile, the statoverride, the drift stop and the managed settings file, each with the test and rollback above; the no-new-privs test | root | ops-01, same path, R27 (mos-1091) |
| 2a'': a `git bundle` of the reviewed commit copied to .53; root clones it to `/root/mosaic-stack` and checks HEAD is the commit the go names; it installs the bundle as `/srv/mosaic/incoming/<sha>.bundle`, owner `mosaic`, 0600 | root | ops-01, same path |
| 2b: node and claude under `~/.local` (above); `git clone` of the bundle to `/srv/mosaic/src/mosaic-stack`; `scripts/bootstrap.sh`; `stack-copy.sh copy`; `case2-install.sh --profile factory` with the inputs above; `case2-precheck.sh --profile factory` | `mosaic` | `sudo -u mosaic -H` from `localadmin`. `mosaic` gets no key and no login shell |
| 2b: claude under `~/.local` (above); `git clone` of the bundle to `/srv/mosaic/src/mosaic-stack`; `scripts/bootstrap.sh`; `stack-copy.sh copy`; the install with the inputs above (C2-13b's `case2-install.sh --profile factory` if it has landed, else the run sheet's steps) | `mosaic` | `sudo -u mosaic -H env PATH=...` from `localadmin` (PATH above). `mosaic` gets no key and no login shell |
| 2b': place `claude-session.env` | Jason, or who he names (Q173 secrets rule) | section 3 |
| 2c: write `case1-unit.args`; `case2-unit.sh` with those arguments and `--copy <id>`, from `/root/mosaic-stack`; `systemctl enable` the bus unit; the first start of UNITS. The Console unit has no `[Install]` until Jason rules (C2-13a), so it isn't enabled, and after a host reboot root starts it by hand until then. The run sheet says so | root | ops-01, same path |
| 2c: write `case1-unit.args`; `case2-unit.sh` with those arguments and `--copy <id>`, from `/root/mosaic-stack`; then `case2-precheck.sh --profile factory` as `mosaic`, which reads the Console's `--port` from the installed unit for its port-floor line, so it runs after the render; `systemctl enable` the bus unit; the first start of UNITS. The Console unit has no `[Install]` until Jason rules (C2-13a), so it isn't enabled, and after a host reboot root starts it by hand until then. The run sheet says so | root | ops-01, same path |
| 2c': `stack-release.sh adopt mosaic-stack --units external`, which records the running copy as the pointer (`do_adopt`, `scripts/stack-release.sh:856`) | `mosaic` | `sudo -u mosaic -H` |
| every later re-pin or rollback | root, with its `mosaic` steps through `sudo -u mosaic -H` | ops-01 runs `case1-repin.sh` (section 4) |
| acceptance (section 5) | `localadmin`, with `sudo -u mosaic -H` for the in-instance checks | the agent the window names |
@@ -177,7 +295,7 @@ brings a new bundle that `mosaic` fetches into its checkout.
## 2. Where Case 1 differs from Case 2
| Case 2 item (C2-13 brief) | Case 1 | Reason |
| Case 2 item (row 230 brief) | Case 1 | Reason |
|---|---|---|
| business file with the library mapping | business `mosaic-stack` with no `library` key; `--profile factory` refuses a `kind: "library"` project (`profile-library-business`) | the factory has no books repo |
| `bean`, `chet`, persona vars and files, specialists | factory roles from `roles/`, with a chief that has a harness var | Jason names the set in the go. I haven't read his live business file and won't. The acceptance needs only the chief (F3), since a chief with no harness var answers `no-route`. Others are added later by the human with `instances.set`, no new row |
@@ -188,12 +306,42 @@ brings a new bundle that `mosaic` fetches into its checkout.
| precheck: `ZAI_API_KEY` in the key file | doesn't run | no instance has provider `zai` |
| precheck: no Discord binding | kept | the factory's Discord connector stays on dragon-lin. Two connectors on one bot token would split its events |
| precheck: claude marker prompt with `--system-prompt-file` | runs, since the chief's harness is claude-code | |
| precheck: `sudo -n true` fails, `id -G` one entry, run root 0700, config validates, node 24 or later, `bwrap` exits 0, free memory, HOME | kept | node and bwrap are in C2-13 (gosalyn-0034) |
| serve flags | `--pm` | the factory's PM dispatches its work, as on dragon-lin |
| precheck: `sudo -n true` fails, `id -G` one entry, run root 0700, config validates, node 24 or later, `bwrap` exits 0, free memory, HOME | kept | node and bwrap are in row 230 (gosalyn-0034) |
| Console serve flags: `--port` 1..1023, `--business`, the preview trio, `--send`, the proof pair, no `--tester-port` | `--port 732`, `--business mosaic-stack`, `--tester-port 7333`. No preview trio, no `--send`, no proof pair | below |
| bus: `@PM_FLAGS@` and `@CLAUDE_ENV@` render empty | ` --pm`, and `Environment=MOSAIC_CLAUDE_BIN=/srv/mosaic/.local/bin/claude` | below |
All of these are C2-13 inputs. Nothing in C2-13's scripts hardcodes a Case 1 value (gosalyn-0034 item
4). If C2-13's candidate drops any of them, the manifest names the change to `case2-install.sh` or
`case2-precheck.sh`, reviewed by Duckworth with Webby told.
All of these are inputs. Nothing in the Case 2 scripts hardcodes a Case 1 value (gosalyn-0034 item 4).
Row 230's brief says the factory profile takes the library serve set plus an optional `--tester-port`,
and that row 195 names any other difference (gosalyn-0049). .53 can't meet the library set, and the bus
needs two lines the brief renders empty. The factory profile I need from row 230:
1. Serve flags. `--port N` with N in 1..1023 and `--business ID` are required, and `--tester-port N` in
1024..65535, not equal to `--port`, is optional. The preview trio (`--public-host`, `--auth-header`,
`--allow-user`) and the proof pair (`--proof-header`, `--proof-file`) are refused. Both exist for a
proxy in front of the Console: the proxy signs the user in and adds the header, and the proof shows
the request came through it. .53 has no proxy. Jason reaches the Console with `ssh -L` (sage-1451
Q3), so there is no header to trust and no proxy to hold the proof. `--send` is refused too, since
serve takes it only with the trio or `--sso` (`packages/webui/src/cli.mjs:15`). When Q175 SSO reaches
.53, `--sso` with an optional `--send` (row 192) would be the factory's sign-in, and adding it is a
re-install with its own go. Row 230 can accept `--sso [--send]` under factory now or later. Either
way it needs a `console.sso` block in `mosaic`'s config, which this row doesn't write. `--checkout`
and `--board` stay refused, as in the library set.
2. `--pm` (factory only) renders the bus unit's `@PM_FLAGS@` as ` --pm`. The factory's PM dispatches its
work on .53 as on dragon-lin (section 4). With the token empty the bus never launches the PM, and
A5's PM check fails every re-pin.
3. `--claude PATH` (factory only) renders `@CLAUDE_ENV@` as `Environment=MOSAIC_CLAUDE_BIN=PATH`, the
line `scripts/bus-service.sh:79` renders today. PATH must be absolute. Without it the bus runs
`claude` from the unit's PATH, which is the node dir, `/usr/local/bin`, `/usr/bin` and `/bin`, and
holds no claude on .53. The chief's first launch would then fail. The precheck's claude marker
prompt should run the same binary, so it reads the path from the installed bus unit.
4. Row 190's two template tokens, if row 190 lands before row 230. Row 190 adds `@CONFIG_ENV@` to both
templates and `@SESSION_ENV@` to the bus template's `EnvironmentFile=` line. For the default
instance, `@CONFIG_ENV@` is deleted (`scripts/instance.sh`) and `@SESSION_ENV@` is
`<home>/.config/mosaic-dev/secrets/claude-session.env`. Row 230's "every token has one value" list
and its `EnvironmentFile=` check need both. This applies to the library profile too.
I've sent these to Gosalyn. If row 230's candidate lacks any of them, row 195's manifest names the
change to `case2-unit.sh` or `case2-precheck.sh`, reviewed by Duckworth with Webby told.
## 3. Model access
@@ -221,7 +369,7 @@ ops-01 from `/root/mosaic-stack`. Each `mosaic` step is `sudo -u mosaic -H` runn
`stack-release.sh` (`COPY_PATHS`, `scripts/stack-copy.sh:23-30`), so the `mosaic` steps run from that
checkout, as dragon-lin's run from the canonical one.
What root holds, and where each value comes from (B1):
What root holds, and where each value comes from (r4's B1):
| Value | Source |
|---|---|
@@ -235,13 +383,42 @@ What root holds, and where each value comes from (B1):
Root parses nothing `mosaic` wrote and runs nothing under `/srv/mosaic` except through `sudo -u mosaic
-H`. It never needs the activation id: the `mosaic` steps find it themselves.
One result line per activation (B1). The stack log closes an activation on the first line after
`started` that names it, whatever the result (`interrupted`, :146-156). r4 wrote an interim
`... rollback pending` line at A3's and A5's failures. That line closed the activation, so R2 and R4
then refused `open-activation` against an activation the log no longer held open. The rollback could
never run as designed, and nothing tested the steps in sequence. r5 keeps an activation open from A3's
`started` line until its single result line, and A5 or R4 writes that line:
| Activation ends at | Result line | Written by |
|---|---|---|
| A3 fails before its rollback set exists (marker or capture) | `refused code marker-failed` or `refused code capture-failed`, as `capture || refuse` does today (:987). Nothing has changed, so there is nothing to roll back | A3 itself, exit 3, after it removes the marker. Root runs A2f |
| A5 passes | `ok` | A5 |
| A3 (after capture), A4 or A5 fails, and the rollback passes | `failed code <c> rollback ok` for A3 and A4, `gate-failed code <c> rollback ok` for A5 | R4 |
| the rollback fails at R4's gate | `rollback-failed code <c> rollbackCode <r>` | R4 |
| R1, R2 or R3 fails, or the run dies | none. The activation stays open, and the next activate refuses `interrupted-run` (:603-606), which Jason or Mos settles as on dragon-lin | |
A failing `mosaic` step after `started` writes no log line. It writes its code to `FAILED` beside
`AWAITING-UNITS` and exits 1. R4 reads that code for the result line. Root passes R4 `--cause
A3|A4|A5`, the step it saw fail, from its own exit codes. A4 is root's own step and has no `FAILED`
file, so R4 writes `failed code units-start` for it. A cause with no `FAILED` file where one should be
writes code `unrecorded`.
The open activation. A3 writes `$STATE/stack-rollback/<ACT>/AWAITING-UNITS` holding the target copy
(renamed from r3's `PENDING`, which read like the pointer's `pmPending`, `P_PENDING` at :128; N3).
`confirm` and `revert` take the copy root passed, not an activation id. Each lists the
`AWAITING-UNITS` files and the `interrupted` set (:146), and goes on only when there is exactly one
file, its activation is the one `interrupted` names, and the copy in it is the copy root passed.
Anything else refuses `open-activation` with exit 3 and the list it found, and changes nothing.
`confirm` removes the file once it logs a result. So does `confirm --rollback`.
`confirm` without `--rollback` also refuses `activation-failed` when `FAILED` exists, so a second
confirm can't pass an activation that already failed. Whichever `confirm` writes the result line
removes `AWAITING-UNITS` and `FAILED` after it.
Refusals of the three external verbs. `confirm`, `revert` and `gate` act on an open activation by
design, so they skip the `interrupted-run` refusal (:605). Their refusals log `result refused` with no
activation id: they set ACT only after every check has passed. So a refused `confirm`, a wrong copy
say, leaves the open activation open. Today's `refuse` (:62-67) adds the activation id whenever ACT is
set, and the external verbs keep that order. A test and a mutant hold it (section 7).
Root's step log. Every step `case1-repin.sh` runs appends one line to `/var/log/mosaic-stack/case1-repin.log`,
`root:root` 0640: the time from `date -u`, the step, its argv, its exit code. `mosaic` can't write it,
@@ -251,18 +428,25 @@ so it stays readable after a run that dies before `mosaic` logs anything.
| # | Step | Runner | On failure |
|---|---|---|---|
| A1 | `/root/mosaic-stack` HEAD is the commit the go names. ID is 40 lowercase hex and names a copy under `/srv/mosaic/.local/share/mosaic-stack/copies` that holds `scripts/mosaic` (C2-13's copy-id rule). Root reads OLD from the unit files. `stack-release.sh activate ID mosaic-stack --units external --plan` exits 0 | root; the plan as `mosaic` | refuse, exit 3. Nothing has changed |
| A1 | `/root/mosaic-stack` HEAD is the commit the go names. ID is 40 lowercase hex and names a copy under `/srv/mosaic/.local/share/mosaic-stack/copies` that holds `scripts/mosaic` (row 230's copy-id rule). `case1-hostcheck.sh --check` exits 0, else `profile-drift` (B2). Root reads OLD from the unit files. `stack-release.sh activate ID mosaic-stack --units external --plan` exits 0 | root; the plan as `mosaic` | refuse, exit 3. Nothing has changed |
| A2 | `systemctl stop` UNITS in reverse order, then wait until each is inactive, within `--timeout` | root | A2f below |
| A3 | `stack-release.sh activate ID mosaic-stack --units external [--migrate]`. It refuses `units-running` unless each of UNITS is inactive (system-manager reads, which need no privilege) and the store is stopped (`stopped`, :359), before it logs or migrates anything. Then it captures the rollback set (pointer, wrapper, and the sha256 and `ExecStart=` of each installed unit, read only), logs `result started`, takes `backup_store` (:372) into `$DATA/backups/stack/<ACT>`, migrates under `--migrate` (sets TAKEN, as today), installs the wrapper for ID, and writes `AWAITING-UNITS`. It writes no pointer | `mosaic` | it puts the wrapper back, restores TAKEN if it migrated, logs `result failed code <c> rollback pending`, exits 1. Root runs R3-R4. A refusal before `started` exits 3, and root runs A2f |
| A4 | `case2-unit.sh` with `case1-unit.args` and `--copy ID`, `systemctl daemon-reload`, `systemctl start` UNITS in order | root | R1-R4 |
| A5 | `stack-release.sh confirm ID mosaic-stack --units external --timeout S`. This is `gate` (:522), which polls `gate_once` (:471) every second until `--timeout`, with the external checks below. On a pass it writes the pointer, logs `result ok`, removes `AWAITING-UNITS`, exit 0. When the timeout runs out it logs `result gate-failed code <gate> rollback pending`, exit 1 | `mosaic` | R1-R4 |
| A3 | `stack-release.sh activate ID mosaic-stack --units external [--migrate]`. Before it logs `started` (N3), it refuses `units-running` unless each of UNITS is inactive (system-manager reads, which need no privilege) and the store is stopped (`stopped`, :359), and `interrupted-run` while any activation is open. Then, in this order: logs `result started`; writes `AWAITING-UNITS` at once; captures the rollback set (pointer, wrapper, and the sha256 and `ExecStart=` of each installed unit, read only); takes `backup_store` (:372) into `$DATA/backups/stack/<ACT>`; migrates under `--migrate` (sets TAKEN, as today) and records in the rollback set that it did; installs the wrapper for ID. It writes no pointer | `mosaic` | a refusal before `started` exits 3 with nothing changed. A marker or capture failure removes the marker, logs its `refused` line and exits 3. Root runs A2f for both. Any later failure writes `FAILED` with its code, logs nothing, cleans up nothing and exits 1. Root runs R1-R4 with `--cause A3` |
| A4 | `case2-unit.sh` with `case1-unit.args` and `--copy ID`, `systemctl daemon-reload`, `systemctl start` UNITS in order | root | R1-R4 with `--cause A4` |
| A5 | `stack-release.sh confirm ID mosaic-stack --units external --timeout S`. This is `gate` (:522), which polls `gate_once` (:471) every second until `--timeout`, with the external checks below. On a pass it writes the pointer, logs `result ok`, removes `AWAITING-UNITS`, exit 0. When the timeout runs out it writes `FAILED` with the last gate code, logs nothing and exits 1 | `mosaic` | R1-R4 with `--cause A5` |
A2f, a stop that fails or times out. Nothing of `mosaic`'s has changed, and the stack log has no
`started` line for this run. Root runs `systemctl start` UNITS in order, then `stack-release.sh gate
OLD mosaic-stack --units external --timeout S`, a new verb that runs `gate` on OLD and writes no log
line, no pointer and no file. If that passes, root logs `A2 failed, OLD running` in its step log and
exits 1. If it fails, root stops UNITS, logs `A2 failed, OLD gate failed <code>` and exits 2. The stack
log is unchanged either way, so the next run's `interrupted` check finds nothing open.
Why A3 writes the marker first and does no cleanup of its own. With the marker written last, as in r4,
an A3 failure after `started` left an open activation with no marker, and R2 and R4 refused it. With
the marker first, every A3 failure after the marker leaves exactly what R2 needs: the open activation,
its marker and whatever part of the rollback set exists. R2 is then the one cleanup path, for A3, A4
and A5 alike. Two cleanup paths, A3's own and R2, would each need their own tests for each failure point.
A2f, a stop that fails or times out, or an A3 that exits 3. This run has no open activation: A2 logs nothing, and
A3's exit 3 comes either before `started` or after its own `refused` line. Nothing of `mosaic`'s has
changed. Root runs `systemctl start` UNITS in order, then `stack-release.sh gate OLD mosaic-stack
--units external --timeout S`, a new verb that runs `gate` on OLD and writes no log line, no pointer and
no file. If that passes, root logs `A2 failed, OLD running` (or `A3 refused`) in its step log and exits
1. If it fails, root stops UNITS, logs `OLD gate failed <code>` and exits 2. The next run's
`interrupted` check finds nothing open either way.
The external gate. `gate_once` keeps its order and codes, with these changes under `--units external`:
@@ -270,12 +454,18 @@ The external gate. `gate_once` keeps its order and codes, with these changes und
manager.
- This start's time is the bus unit's `ActiveEnterTimestamp`, read with `systemctl show --timestamp=unix
-p ActiveEnterTimestamp`, in place of the `START_MS` that `start_both` (:451) sets on dragon-lin.
`confirm` refuses `stale-start` when that time is before the `started` line of the open activation,
so a unit root never restarted can't pass.
It is whole seconds (`@1792540800`), and the `started` line has milliseconds (N4). So the rule
compares seconds: the start passes when its second is at or after the floor of `started`'s second.
A unit that was active across A3 can't reach this check, because A3 refuses `units-running`, so the
equal second can only be a start after A3. An earlier second fails `gate_once` with the gate code
`stale-start`. It's a gate failure like any other, not a refusal (r4 had it as a refusal, which
would have logged against the open activation): `gate` keeps polling, since `Restart=on-failure`
can bring a fresh start, and at the timeout `FAILED` holds `stale-start`. The PM check's
`startedAt` bound uses the same second.
- `copy-check` runs `stack-copy.sh check --units external`, which reads the system units.
- The PM check is below.
The PM (B2). .53 runs the PM: the factory's PM dispatches its work, as it does on dragon-lin, and the
The PM (r4's B2). .53 runs the PM: the factory's PM dispatches its work, as it does on dragon-lin, and the
serve flags carry `--pm`. The pointer's `pm` comes from the installed bus unit's `ExecStart=`, as the
unit parse at :195 already reads it. Today's `pm_launched` (:505) takes the run id from the
`launched ...` line in `journalctl --user -u BUS_UNIT`. On .53 that read fails twice over: the unit is
@@ -288,7 +478,7 @@ Zero entries, two, a stale one or one for another business fail with `pm-launch-
it saw, as today. A run the PM starts for another role has another instance, so it doesn't count.
Why not the other reads. A regular uid for `mosaic` would put its logs in a per-user journal it can
read, but it changes the account C2-13 creates. Root reading the journal and passing the run id in
read, but it changes the account Case 2's install creates. Root reading the journal and passing the run id in
would make root parse output from `mosaic`'s process, which section 4's rule forbids.
Under `--units external`, `pm-on` (:1137) and `activate --pm-after-harness` refuse `external-pm`. Both
@@ -300,14 +490,19 @@ serve flag in a root-owned unit, so changing it is a re-install with its own go.
| # | Step | Runner | On failure |
|---|---|---|---|
| R1 | `systemctl stop` UNITS, then wait as in A2 | root | rollback-failed |
| R2 | `stack-release.sh revert ID mosaic-stack --units external`. It finds the open activation as above, puts the wrapper back from the rollback set and checks its sha256, and restores TAKEN (`restore_store`, :396) only if this run migrated | `mosaic` | rollback-failed |
| R3 | `case2-unit.sh` with `case1-unit.args` and `--copy OLD`, `systemctl daemon-reload`, `systemctl start` UNITS | root | rollback-failed |
| R4 | `stack-release.sh confirm ID mosaic-stack --units external --rollback OLD --timeout S`. It finds the open activation, checks OLD is its `from`, gates OLD with `gate`, and logs `result gate-failed code <c> rollback ok`, or `result rollback-failed code <c> rollbackCode <r>` | `mosaic` | rollback-failed |
| R2 | `stack-release.sh revert ID mosaic-stack --units external`. It finds the open activation as above, puts the wrapper back from the rollback set and checks its sha256, and restores TAKEN (`restore_store`, :396) only if the rollback set records a migration. Each part is idempotent, so it works for an A3 that stopped partway: no recorded migration means no restore, and copying the saved wrapper over an unchanged one changes nothing. It writes no log line | `mosaic` | rollback-failed, activation left open |
| R3 | `case2-unit.sh` with `case1-unit.args` and `--copy OLD`, `systemctl daemon-reload`, `systemctl start` UNITS | root | rollback-failed, activation left open |
| R4 | `stack-release.sh confirm ID mosaic-stack --units external --rollback OLD --cause A3\|A4\|A5 --timeout S`. It finds the open activation, checks OLD is its `from`, gates OLD with `gate`, and writes the activation's one result line from the cause and the `FAILED` code: `failed code <c> rollback ok` (A3, A4), `gate-failed code <c> rollback ok` (A5), or `rollback-failed code <c> rollbackCode <gate>` when OLD's gate fails | `mosaic` | rollback-failed, logged |
rollback-failed: root stops UNITS and exits 2, as `rollback_failed` (:587) does. When R2 or R4 can't log
it, root's step log has the codes, and the next run's `interrupted` check names the activation that has
`started` and no result. Exit codes match `stack-release.sh`: 0 ok, 1 failed with the old stack running
(rolled back, or A2f), 2 rollback-failed with UNITS stopped, 3 refused with nothing changed.
rollback-failed: root stops UNITS and exits 2, as `rollback_failed` (:587) does. When R4's gate fails,
the log has the `rollback-failed` line and the activation is closed. When R1, R2 or R3 fails, no
`mosaic` step writes a result, so the activation stays open with its marker. Root's step log has the
codes, the next activate refuses `interrupted-run` until Jason or Mos settles it (:605), and status
shows it. I chose that over a log-only verb for root to call after an R2 failure: nothing could gate
the old stack at that point, and a closed activation would let the next activate start against a
wrapper and store in an unknown state. Exit codes match `stack-release.sh`: 0 ok, 1 failed with the old
stack running (rolled back, or A2f), 2 rollback-failed with UNITS stopped, 3 refused with nothing
changed.
### Rollback to the previous copy: `case1-repin.sh rollback --to ID [--restore ACT --events N]`
@@ -342,8 +537,10 @@ re-pin's A3 takes the first backup.
Every `mosaic` line goes to `$DATA/state/stack-activation-log.jsonl`, as today, with `units external`
added. So `from_log previous` reads .53's history the same way it reads dragon-lin's. A3 writes
`started`. A5 or R4 writes the result. If a run dies between them, the log keeps `started` with no
result, the next run's `interrupted` check reports it, and root's step log shows the last step that ran.
`started`. A5 or R4 writes the one result line, or A3 its `refused` line before any change. Refusals of
`confirm`, `revert` and `gate` log no activation id. If a run dies between `started` and the result,
the log keeps `started` with no result, the next run's `interrupted` check reports it, and root's step
log shows the last step that ran.
### Who
@@ -361,10 +558,10 @@ do today, `pm_launched` still reads the user journal, and rollback still takes T
- Row 190's user units under `mosaic`, with root running `loginctl enable-linger mosaic` once. Every
existing re-pin tool would work unchanged and root would be needed only once. But the unit form
would differ from Case 2, and a user manager would run as `mosaic`.
- A sudoers rule that lets `mosaic` start and stop the units. P1 and C2-13's precheck require that
- A sudoers rule that lets `mosaic` start and stop the units. P1 and row 230's precheck require that
`mosaic` has no sudo.
- `confirm` and `revert` taking ACT from root. Root would have to read it from `mosaic`'s output or
log, which the rule above forbids (B1).
log, which the rule above forbids (r4's B1).
## 5. Acceptance checks
@@ -380,10 +577,10 @@ Functional checks. Any failure fails the row:
| Id | Check | How |
|---|---|---|
| F1 | the units run as `mosaic` | `systemctl show <unit> -p User -p MainPID` for each of UNITS; each MainPID's uid in `/proc/<pid>/status` is `mosaic`'s, and each unit is active |
| F2 | the Console listens on 127.0.0.1 on .53 and answers there | `curl -fsS http://127.0.0.1:7332/healthz` on .53 succeeds, and the listener table shows 7332 bound to 127.0.0.1 only. Jason reaches it with `ssh -L 7332:127.0.0.1:7332 localadmin@10.1.1.53`. A LAN listener waits for Q175 SSO (sage-1451 Q3) |
| F2 | the Console listens on 127.0.0.1:732 on .53 and answers there | `curl -fsS http://127.0.0.1:732/healthz` on .53 succeeds, and the listener table shows 732 bound to 127.0.0.1 only, by the Console unit's MainPID. Jason reaches it with `ssh -L 7332:127.0.0.1:732 localadmin@10.1.1.53` and his browser at 127.0.0.1:7332 as today. A LAN listener waits for Q175 SSO (sage-1451 Q3) |
| F3 | one tester message gets a reply | as `mosaic`: `mosaic talk chief <marker> --tester`, then `mosaic follow --tester`, with a 300 s limit. A stored reply to the marked message passes. The inbox alone shows decisions only, so it isn't used (N3) |
| F4 | the precheck passes | `case2-precheck.sh --profile factory` exits 0 as `mosaic` |
| F5 | a role session runs under bwrap with the profile | while F3's reply is pending: a `/usr/bin/bwrap` process owned by `mosaic` has a claude process under it; `/proc/<bwrap pid>/attr/current` reads `bwrap (enforce)` and the claude process's reads `unpriv_bwrap` (stacked) in enforce mode. If `launches list --json` records the sandbox, the build reads that for the first part, since a short session can end before `ps` sees it |
| F5 | a role session runs under bwrap with the profile | while F3's reply is pending: a `/usr/bin/bwrap` process owned by `mosaic` has a claude process under it. `/proc/<bwrap pid>/attr/current` equals `bwrap (enforce)` and the claude process's equals `bwrap//&unpriv_bwrap (enforce)`, each compared as an exact string after the trailing newline is cut (N1). If `launches list --json` records the sandbox, the build reads that for the first part, since a short session can end before `ps` sees it |
| F6 | claude is the pinned version | `/srv/mosaic/.local/bin/claude --version` as `mosaic`, after F3, equals the run sheet's pin |
| F7 | the PM run is live | `launches list --json` as `mosaic` has exactly one live entry for the PM's instance, the check `confirm` runs (section 4) |
@@ -398,20 +595,25 @@ Privilege and host paths. These report and never fail the row:
| P5 | `jarvis`, which runs Hermes, can reach `/srv/mosaic` through sudo, or through the docker or lxd groups, which are root-equivalent | closed when `jarvis` is in none of sudo, docker and lxd; unnamed otherwise. #423 step 4 lands before the install (mos-1091 4c), and its `step4.sh` removes `jarvis` from sudo, docker, lxd and adm, so after it this reads closed (Duckworth N2) | `id jarvis` |
| P6 | other members of sudo, docker, lxd | closed when sudo holds only `localadmin` and docker and lxd are empty. Every other member is its own unnamed line, `jarvis` included (N1) | `getent group` lines |
| P7 | a file under `/etc/sudoers.d` naming `mosaic` | closed when none | the file names |
| P8 | the Console or tester reachable from the LAN | closed when 7331-7333 are bound to 127.0.0.1 only | listener table |
| P9 | a user namespace through `/usr/bin/bwrap` | accepted mos-1096 when all of these hold, unnamed otherwise: `dpkg-statoverride --list /usr/bin/bwrap` reads `root mosaic 0750`; `aa-status` lists `bwrap` and `unpriv_bwrap` in enforce mode; `/etc/apparmor.d/bwrap-userns-restrict` has the pinned sha256; `sudo -u jarvis /usr/bin/bwrap --version` fails with "Permission denied"; `getent group mosaic` lists no member and no other account in `/etc/passwd` has its gid; `fs.protected_hardlinks` is 1 | each item's line, and Mos's reason: "Case 1's agents are the factory's agents. In Q182 and Q183 Jason accepted that the factory reaches root on pve and, through it, every VM. So a user-namespace path to root on .53 gives that class no new reach." With the profile, what's left is bwrap's own setup, run by `mosaic` |
| P8 | the Console or tester reachable from the LAN | closed when 732 and 7333 are bound to 127.0.0.1 only, each by the Console's MainPID, and nothing listens on 7331 | listener table |
| P9 | a user namespace through `/usr/bin/bwrap` | accepted mos-1096 only while all seven hold, unnamed otherwise: (1) `dpkg-statoverride --list /usr/bin/bwrap` reads `root mosaic 0750`; (2) `aa-status --json` lists `bwrap` and `unpriv_bwrap` in enforce mode; (3) `/etc/apparmor.d/bwrap-userns-restrict` has the pinned sha256 and no package owns it; (4) `sudo -u jarvis /usr/bin/bwrap --version` fails with "Permission denied"; (5) `getent group mosaic` lists no member and no other account in `/etc/passwd` has its gid; (6) `fs.protected_hardlinks` is 1; (7) no `mosaic-bwrap` profile is loaded, no other file under `/etc/apparmor.d` names `bwrap`, and the live label of a process bwrap runs is exactly `bwrap//&unpriv_bwrap (enforce)` (N1, mos-1098 point 2). Items 1-3, 6 and 7 are `case1-hostcheck.sh --check`'s, so P9 reuses its lines | each item's line, and Mos's reason: "Case 1's agents are the factory's agents. In Q182 and Q183 Jason accepted that the factory reaches root on pve and, through it, every VM. So a user-namespace path to root on .53 gives that class no new reach." With the profile, what's left is bwrap's own setup, run by `mosaic` |
| P10 | a session binds the tester port, 7333, while the Console is down | accepted mos-1101 while `ip_unprivileged_port_start` is above 732 and 7333 is held by the Console's MainPID at the check. Serve refuses a tester port below 1024 (`cli.mjs:33`), so the Console-port fix can't cover it. A squatter would get the tester's writes and could answer them. The tester's users are the factory's agents, the class Mos's reading covers (mos-1101 item 5). Mos can object, and then this line reads unnamed | the floor, and the owner pid of 7333 |
Tests, in `packages/cli/tests/case1-accept.test.mjs`. The script takes a fake root (`--root DIR`) for
`/proc`, `/etc` and `getent` output, and fake `systemctl`, `curl`, `ss`, `sudo`, `bwrap`, `aa-status`,
`dpkg-statoverride`, `sysctl` and `claude` on PATH. Cases:
- each F check failing gives a non-zero exit with its line;
- each P check in each of its marks, including P1 with output that grants nothing but lacks the refusal
text (unnamed), and P9 unnamed once for each of its six items missing;
text (unnamed), P9 unnamed once for each of its seven items missing, and P10 unnamed with a floor of
700 and with 7333 held by another pid;
- F5 fails on a label of `unpriv_bwrap (enforce)`, `bwrap//&unpriv_bwrap (complain)` and
`bwrap (enforce)` with a trailing space;
- a run with every P check `unnamed` still exits 0;
- the summary counts match the lines.
Mutants: one per F check (the check made to pass on failure), one that maps P5 to `closed`, one that
closes P1 on any output without a rule line, and one that drops P9's jarvis item.
closes P1 on any output without a rule line, one that drops P9's jarvis item, one that drops P9's item
7, and one that makes F5 a substring match.
Nothing in the tests reaches .53 or dragon-lin's live state.
@@ -445,14 +647,40 @@ mos-1096:
whatever he answers. If his answer changes how upgrades reach .53, the change lands in section 4 and
the run sheet as a new round.
mos-1098 and mos-1101 (on Duckworth's REVIEW-r3):
- Duckworth's CHANGES stand. B1 and B2 are fixed in this round, with B1's real-sequence test, and all
three of B2's asks: an owner across upgrades, what fails closed, and the pins as reviewed values.
- r4's PM, statoverride and P9 stand. mos-1098's two questions are answered in section 1: the profile's
owner across upgrades is ops-01 with `case1-hostcheck.sh`, and the stop is bwrap closed to `mosaic`.
Both profiles attach to `/usr/bin/bwrap`: `bwrap` to the binary and `unpriv_bwrap` stacked on what it
runs. P9 item 7 checks that no third one does.
- P9 has seven items and F5 compares exact strings (N1).
- The port squat applies (section 1). The Console takes Case 2's fix, a port below 1024 with
`CAP_NET_BIND_SERVICE` and the floor checked. Only the tester port can't take it, and P10 records it
under Mos's reading that Case 1's sessions are the factory class. I've told Mos, who can object.
- Work on r5 goes ahead while Jason's Q9 (Vikunja #429) is open. If install and upgrade become
product commands, these steps become that installer's configuration and row 195 rebases on its row
(sage-1481, sage-1491).
sage-1499: C2-13a is row 230 (#1732, Gosalyn), and row 195 chains on row 230's candidate.
gosalyn-0049: the factory profile is the library set plus an optional `--tester-port`, and row 195 names
any other difference. Section 2 names four, with the reason .53 can't meet the preview trio, `--send` or
the proof pair. The managed-settings route needs no Case 2 change.
## 7. Manifest (planned)
- `scripts/case1-repin.sh` (new) and `packages/cli/tests/case1-repin.test.mjs` (new). Fake `systemctl`,
`sudo` and `case2-unit.sh` on PATH record their argv, with a scratch home standing in for `/srv/mosaic`,
a scratch dir for `/etc/mosaic-stack` and one for `/var/log/mosaic-stack`. Cases:
- a pass runs A1-A5 in order;
- a failure at A3, A4 or A5 runs R1-R4 and exits 1;
- a failure at A3, A4 or A5 runs R1-R4 with `--cause` set to that step, and exits 1;
- an A3 exit 3 runs A2f, not R1-R4;
- a failure at R2 or R4 exits 2 with UNITS stopped;
- A1 refuses `profile-drift` when a fake `case1-hostcheck.sh --check` exits 1, before any stop;
- every `mosaic` step's argv carries `env PATH=<node dir>:/usr/local/bin:/usr/bin:/bin`, with the node
dir from `case1-unit.args`;
- a HEAD mismatch, an ID that fails the copy-id rule, or unit files that name two copies refuses with
exit 3 before any stop;
- A2f: a fake `systemctl` that never reports inactive. The script starts UNITS, runs `gate OLD`, exits
@@ -477,17 +705,47 @@ mos-1096:
one whose copy isn't the one passed, and with one whose activation isn't in the `interrupted` set;
- `confirm` polls: a fake bus whose status turns good after 3 s passes with `--timeout 10` and fails
with `--timeout 1`;
- `confirm` refuses `stale-start` when `ActiveEnterTimestamp` is older than the `started` line;
- `confirm` on a pass writes the pointer, logs `ok` and removes `AWAITING-UNITS`; on a failure it logs
`gate-failed ... rollback pending` and leaves the pointer;
- `stale-start` (N4): with `started` at 1792540800.900, an `ActiveEnterTimestamp` of `@1792540800`
passes and `@1792540799` fails `gate_once` with `stale-start`. A fake that turns to `@1792540805`
after 2 s passes with `--timeout 10`, which shows it's a gate code that polling can clear, not a
refusal;
- `confirm` on a pass writes the pointer, logs `ok` and removes `AWAITING-UNITS`. On a timeout it
writes `FAILED` with the gate code, leaves the pointer, the log and the marker as they were, and
exits 1;
- `confirm` without `--rollback` refuses `activation-failed` when `FAILED` exists;
- A3's order: a fake that fails the capture leaves the log with `started` and `refused code
capture-failed` for that activation, no marker and no `FAILED`, and exit 3. A fake that fails the
wrapper install after a migration leaves `started` only, the marker, `FAILED`, TAKEN as migrated,
and exit 1;
- refusals by `confirm`, `revert` and `gate` log with no activation id, and the `interrupted` set
is the same before and after each one;
- the PM check passes with one live entry for the PM's instance since the start, and fails
`pm-launch-unconfirmed` with none, with two, with one whose `startedAt` is before the start, with one
not live, and with one for another business. A second live entry for another instance doesn't
count. A fake `journalctl` on PATH records any call, and no external run makes one;
- `pm-on` and `activate --pm-after-harness` refuse `external-pm`;
- `revert` puts the wrapper back, and restores TAKEN only after a migration;
- `confirm --rollback OLD` logs `rollback ok` or `rollback-failed`, and refuses when OLD isn't the
open activation's `from`;
- `confirm --rollback OLD` writes the line for each `--cause` (A3 and A4 `failed`, A5 `gate-failed`,
with the `FAILED` code, `units-start` for A4, `unrecorded` when the file is missing), or
`rollback-failed` when OLD's gate fails. It removes the marker and `FAILED` after the line, and
refuses when OLD isn't the open activation's `from`;
- real sequences on one log (B1). The fake `sudo` execs the real `stack-release.sh` as the test's
user, so each step reads the log and state the last one wrote. Fakes stand in only for
`systemctl`, `mosaic` and `case2-unit.sh`. Each sequence ends with the log holding exactly one
`started` and one result line for its activation, an empty `interrupted` set, no marker or
`FAILED`, and a following `activate --plan` that doesn't refuse `interrupted-run`:
1. activate, confirm times out, revert, `confirm --rollback --cause A5` passes:
`gate-failed code <c> rollback ok`;
2. activate with `--migrate`, the wrapper install fails, revert restores TAKEN, `confirm --rollback
--cause A3`: `failed code <c> rollback ok`;
3. activate, root's A4 fails, revert, `confirm --rollback --cause A4`: `failed code units-start
rollback ok`;
4. activate whose capture fails: `refused code capture-failed` and nothing open, then A2f's
`gate OLD` writes nothing;
5. activate, then `confirm` with the wrong copy is refused, then revert and `confirm --rollback
--cause A5` still close the activation, because the refusal didn't;
6. activate, revert fails (a wrapper whose saved copy is missing): the activation stays open, and
the next `activate --plan` refuses `interrupted-run`;
- `rollback --to` refuses when ID isn't `from_log previous`, and `rollback --restore` finds a backup
that an external activate took;
- `gate` writes nothing: the log, the pointer and the state dir are byte-identical before and after;
@@ -506,14 +764,42 @@ mos-1096:
- `case1-repin.sh` skips R1-R4;
- `case1-repin.sh` exits 0 after A2f;
- `case1-repin.sh` runs `mosaic`'s `stack-release.sh` without `sudo -u mosaic`;
- the default path drops `--user` from one `systemctl` call.
- the default path drops `--user` from one `systemctl` call;
- `confirm`'s timeout writes r4's interim `gate-failed ... rollback pending` line (B1; killed by
sequences 1 and 5);
- A3 writes `AWAITING-UNITS` last, after the wrapper (killed by sequence 2);
- an external verb's refusal logs with the open activation id (ACT set before the checks; killed by
sequence 5 and the refusal test);
- `stale-start` is a refusal again (killed by the polling case);
- `revert` restores TAKEN without a recorded migration (killed by the revert case).
- `scripts/case1-pins.sh` and `scripts/case1-hostcheck.sh` (new),
`packages/cli/systemd/mosaic-hostcheck.service` (new) and `packages/cli/tests/case1-hostcheck.test.mjs`
(new). The script takes `--root DIR` for `/etc`, `/proc/sys` and `/usr/bin/bwrap`'s stat, and fake
`aa-status`, `dpkg`, `dpkg-statoverride`, `apparmor_parser`, `bwrap`, `chown`, `chmod` and `setpriv`
on PATH record their argv. Cases:
- `--check` exits 0 with all seven items and 1 with each one failing alone, its line naming it;
item 3 fails on another profile file naming `bwrap`, item 4 on a label of `unpriv_bwrap (enforce)`
or with a trailing space, item 7 with the floor at 732;
- `--enforce` exits 0 on a pass and on every failure. On a failure it unloads the profile, closes
bwrap to `root:root` 0700, writes `DRIFT` and one log line. With `DRIFT` present and every item
passing, it still closes bwrap;
- `--enforce` never calls `dpkg-statoverride` with `--update`, `--add` or `--remove`;
- a bubblewrap version other than the install pin is a printed line, not a failure;
- `--upstream` against a fake deb whose file differs prints the difference and exits 0;
- the service unit has `Type=oneshot`, `After=apparmor.service`,
`Before=mosaic-bus@mosaic-stack.service` and runs `--enforce`.
Mutants: `--enforce` exits 1 on a drift; `--enforce` ignores `DRIFT` when the items pass; item 1
drops the `dpkg -S` owner check; item 4 matches a substring; item 7 compares with `>=`.
- `scripts/case1-accept.sh` (new) and `packages/cli/tests/case1-accept.test.mjs` (new), section 5.
- C2-13. `--profile`, the node and bwrap precheck lines, the copy-id rule and the two units are in
Gosalyn's design r10 (4281b0064). Any of these that C2-13's candidate lacks becomes a named change
here. I've told Gosalyn that the settings-file route for `DISABLE_AUTOUPDATER` doesn't reach a role
session, and that r4 uses the managed settings file.
- Row 230 (C2-13a). `--profile`, the node, bwrap and port-floor precheck lines, the copy-id rule, the
two units and the Console's capability lines are in its brief at 5b8def2f1. Section 2's four items
are not. Any of them that row 230's candidate lacks becomes a named change here, reviewed by
Duckworth with Webby told. I told Gosalyn earlier that the settings-file route for
`DISABLE_AUTOUPDATER` doesn't reach a role session, and that Case 1 uses the managed settings file.
- Docs: the Case 1 run sheet `docs/guides/case1-install.md` (new). It lists the window's steps in order,
each with who runs it and its go, the pinned node sha256, claude version, bubblewrap version and
profile sha256, the statoverride and the managed settings file, each with its rollback, and the line
that a restore leaves the credential in place.
each with who runs it and its go, the pinned node sha256 and claude version, a pointer to
`case1-pins.sh` for the bubblewrap and profile pins, the statoverride, the drift stop and the managed
settings file, each with its rollback, how ops-01 clears a `DRIFT`, the `ssh -L` line for port 732,
and a note that a restore leaves the credential in place.
- No change to `~/.mosaic-dev`, `~/.config/mosaic-dev`, dragon-lin's units or `~/.mosaic`.