docs(row321): wait for loopback health before precheck (#1817, F11)

agents/gladstone/work/queue-321/BUILD.md

@@ -0,0 +1,171 @@
+# Row 321: F7 wrapper environment and F10 first adoption (#1817)
+
+Owner Gladstone; reviewer Boyd (separate Codex thread); queue/review/landing Sage.
+Authorized by sage-3791. Branch gladstone/row321 starts at fetched origin/refactor
+833ec1cc8d7c11842efb82e57c1a0b256c8d22d7. The row320 branch is unchanged.
+Brief: docs/plans/2026-10-23_host-fixes-1816-brief.md, section
+"Brief: F7, the wrapper step's PATH and umask under sudo", including F10;
+verified blob 7333f94ba9cfd58f87c5a3580d428b84c65845b6 (requested pin 7333f94ba9cf).
+Only the run sheet and this BUILD.md change. No scripts, tests, config, credentials,
+queue state, deployment or live host operation changed. No tests or Node commands run.
+
+## Before and after
+
+F7: sudo secure_path omits the pinned Node directory, so wrapper creation reaches
+instance.sh's bare node and exits127. An inherited umask can leave the bin dir0775.
+The MOSAIC_WRAPPER heredoc now sets umask077 and the sheet's existing pinned Node
+PATH before mkdir. One sentence names secure_path and why the step sets both.
+The existing Mos go remains explicit. No wrapper command is executed here.
+
+F10: first install omits adoption, so the first activate refuses no-pointer.
+The release-transport section now places explicit adoption after both units are
+running (section10) and before any activate/upgrade, run as mosaic on Mos's go by
+Jason or Mos, citing stack-release.sh:14. It uses the configured dev instance and
+external units, with the same pinned Node PATH under sudo.
+Adjacent to the upgrade command, the sheet now says only the first upgrade after
+adopt takes --no-rehearsal: adopt writes no backup; missing flag refuses no-backup;
+later upgrades omit it, because a backup with that flag refuses rehearsal-required.
+Source checks: stack-release.sh pointer_for_business refuses no-pointer and directs
+adopt; stack-upgrade.sh rehearse handles rc3/no-backup and rc0/rehearsal-required.
+This records existing behavior and authority; it adds no requirement or live go.
+
+## Test-reference grep (negative and positive control)
+
+The required negative grep covers both F7 and F10 because neither is read by any
+matched test. The positive control locates the precheck path that the unit/install
+tests actually execute, showing the pathspecs reach their test files.
+
+Negative grep:
+
+```sh
+git grep -n 'host-run-sheet' -- 'packages/*/tests/*' 'scripts/test-*.sh'
+# exit 1
+# no matches
+```
+
+Positive control:
+
+```sh
+git grep -n 'case2-precheck.sh' -- 'packages/*/tests/*' 'scripts/test-*.sh'
+# exit 0
+packages/cli/tests/case1-accept.test.mjs:26:const PRECHECK = `${HOME}/.local/share/mosaic-stack/copies/${COPY}/scripts/case2-precheck.sh`;
+packages/cli/tests/case2-install.test.mjs:2:// scripts/case2-precheck.sh's library checks and --library-status mode. Brief:
+packages/cli/tests/case2-install.test.mjs:23:const PRECHECK = join(REPO, "scripts", "case2-precheck.sh");
+packages/cli/tests/case2-install.test.mjs:271:// ---- case2-precheck.sh, the library profile (D6a) ----
+packages/cli/tests/case2-install.test.mjs:334:// ---- case2-precheck.sh --library-status (D6b) ----
+packages/cli/tests/case2-unit.test.mjs:2:// and scripts/case2-precheck.sh checks the host before they start. Brief: docs/plans/2026-10-20_case2-rows.md
+packages/cli/tests/case2-unit.test.mjs:20:const PRECHECK = join(REPO, "scripts", "case2-precheck.sh");
+packages/cli/tests/case2-unit.test.mjs:588:// ---- case2-precheck.sh ----
+```
+
+## Run-sheet diff against the pinned base (git diff --unified=0)
+
+```diff
+diff --git a/docs/guides/host-run-sheet.md b/docs/guides/host-run-sheet.md
+index d2ba28d7c..96fd4f363 100644
+--- a/docs/guides/host-run-sheet.md
++++ b/docs/guides/host-run-sheet.md
+@@ -209,0 +210,9 @@ sudo -u mosaic -H git clone -q -b refactor /srv/mosaic/incoming/<sha>.bundle /sr
++After a first install, once both units are running (section 10), adopt the installed copy before
++any activate or upgrade. This step runs only on Mos's go, with Jason or Mos running it
++(`scripts/stack-release.sh:14`):
++
++```sh
++sudo -u mosaic -H env PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin \
++  /srv/mosaic/src/mosaic-stack/scripts/stack-release.sh adopt <business> --instance dev --units external
++```
++
+@@ -211,0 +221,4 @@ Each upgrade, before `stack-upgrade.sh upgrade`:
++Only the first upgrade after adopt takes `--no-rehearsal`: append it to the upgrade command below.
++Adopt writes no backup, so omitting the flag then refuses `no-backup`. Subsequent upgrades omit it;
++when a backup exists, supplying it refuses `rehearsal-required`.
++
+@@ -246,0 +260 @@ step requiring Mos's go (`scripts/stack-copy.sh:3`); the install does not create
++The step runs under sudo's `secure_path`, so it sets its own PATH and umask.
+@@ -250,0 +265,2 @@ cd / || exit 1
++umask 077
++PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin; export PATH
+```
+
+## Verification
+
+No matching test reads the changed lines, so the stop condition did not apply.
+No test run or slot is required under sage-3791. Verification is static: the
+required grep and positive control above, exact brief/base pins, scoped diff,
+and git diff --check. Reviewer Boyd and Sage own review/landing; no approval
+or landing is claimed here.
+
+Receipt correction: the first commit embedded ordinary diff context lines that
+contain a single space. git diff --check flagged those as trailing whitespace.
+This follow-up uses the exact zero-context diff instead. The run sheet is
+unchanged, no tests or Node commands run, and no amend or force-push is used.
+
+## F11 addition (sage-3805, mos-1549, ops-0275)
+
+The brief was extended after F7/F10 drafting and re-pinned at revision3733:
+canonical local commit a9bae9af2, same F7 section anchor, full verified blob
+0e684ef2103447dcc2013b6bf81b1c4ca5bbc0bd (requested pin0e684ef21034).
+Read using git show from the canonical checkout; no fetch, rebase or queue write
+is required. This new pin supersedes the earlier brief pin for this follow-up.
+
+Defect: a precheck one second after unit start found no bus.sqlite yet.
+Section10 now requires loopback healthz success after a unit start before any
+case2-precheck.sh, including section9's --library-status. It uses the same
+127.0.0.1:<port>/healthz curl form as scripts/case1-accept.sh:458. The subshell
+tries at most30 times, each curl capped at2 seconds, with1 second between failed
+attempts (at most89 seconds plus scheduling overhead). Exhaustion prints STOP
+and exits only the subshell. The instruction says to stop before precheck/adopt;
+it does not exit the interactive operator shell. The section9 library-gitea row
+now explicitly requires this bounded wait before --library-status. <port> is the
+installed Console unit's --port, not an assumed port or business name.
+No host action, test, Node command or credential access occurred.
+
+Required grep repeated for the widened scope:
+
+```sh
+git grep -n 'host-run-sheet' -- 'packages/*/tests/*' 'scripts/test-*.sh'
+# exit1, no matches
+git grep -n 'case2-precheck.sh' -- 'packages/*/tests/*' 'scripts/test-*.sh'
+# exit0, positive control; output follows
+packages/cli/tests/case1-accept.test.mjs:26:const PRECHECK = `${HOME}/.local/share/mosaic-stack/copies/${COPY}/scripts/case2-precheck.sh`;
+packages/cli/tests/case2-install.test.mjs:2:// scripts/case2-precheck.sh's library checks and --library-status mode. Brief:
+packages/cli/tests/case2-install.test.mjs:23:const PRECHECK = join(REPO, "scripts", "case2-precheck.sh");
+packages/cli/tests/case2-install.test.mjs:271:// ---- case2-precheck.sh, the library profile (D6a) ----
+packages/cli/tests/case2-install.test.mjs:334:// ---- case2-precheck.sh --library-status (D6b) ----
+packages/cli/tests/case2-unit.test.mjs:2:// and scripts/case2-precheck.sh checks the host before they start. Brief: docs/plans/2026-10-20_case2-rows.md
+packages/cli/tests/case2-unit.test.mjs:20:const PRECHECK = join(REPO, "scripts", "case2-precheck.sh");
+packages/cli/tests/case2-unit.test.mjs:588:// ---- case2-precheck.sh ----
+```
+
+No test reads the changed run-sheet lines; the stop condition does not apply.
+No test run is needed under the docs-only brief. Static verification checks both
+locations, retry limit/timeout, the loopback URL, unchanged F7/F10 and scoped diff.
+
+F11 delta against d9400f0d3 (git diff --unified=0):
+
+```diff
+diff --git a/docs/guides/host-run-sheet.md b/docs/guides/host-run-sheet.md
+index 96fd4f363..e9fa87f24 100644
+--- a/docs/guides/host-run-sheet.md
++++ b/docs/guides/host-run-sheet.md
+@@ -322 +322 @@ file with `stat -c '%U %a %n' PATH`, never `cat`. `<business>` is the host's bus
+-| `/srv/mosaic/.config/mosaic-dev/secrets/<business>/` | `library-gitea` | 0600 | mosaic | library | `case2-precheck.sh --business <business> --library-status`, after the bus is up: `library-last-error library-credential-missing` until it's placed, then `library <project> last fetched ...` |
++| `/srv/mosaic/.config/mosaic-dev/secrets/<business>/` | `library-gitea` | 0600 | mosaic | library | After a unit start, wait for loopback `/healthz` with the bounded retry in section 10 before `case2-precheck.sh --business <business> --library-status`: `library-last-error library-credential-missing` until it's placed, then `library <project> last fetched ...` |
+@@ -357,0 +358,16 @@ None by hand. I2's `case2-unit.sh` enables both units, starts the bus, then the
++After a unit start, wait for the Console's loopback `/healthz` to succeed before running
++`case2-precheck.sh`, including `--library-status` in section 9. Replace `<port>` with the installed
++Console unit's `--port`. This retry is bounded to 30 attempts, each with a 2-second timeout, with
++1 second between failed attempts. If it prints STOP, stop here; do not run the precheck or adopt.
++
++```sh
++(
++  attempts=0
++  until curl -fsS --max-time 2 -o /dev/null 'http://127.0.0.1:<port>/healthz'; do
++    attempts=$((attempts + 1))
++    [ "$attempts" -lt 30 ] || { echo "STOP: loopback healthz did not become ready; do not continue" >&2; exit 1; }
++    sleep 1
++  done
++)
++```
++
+```

docs/guides/host-run-sheet.md

@@ -207,8 +207,21 @@ install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha
 sudo -u mosaic -H git clone -q -b refactor /srv/mosaic/incoming/<sha>.bundle /srv/mosaic/src/mosaic-stack
 ```
 
+After a first install, once both units are running (section 10), adopt the installed copy before
+any activate or upgrade. This step runs only on Mos's go, with Jason or Mos running it
+(`scripts/stack-release.sh:14`):
+
+```sh
+sudo -u mosaic -H env PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin \
+  /srv/mosaic/src/mosaic-stack/scripts/stack-release.sh adopt <business> --instance dev --units external
+```
+
 Each upgrade, before `stack-upgrade.sh upgrade`:
 
+Only the first upgrade after adopt takes `--no-rehearsal`: append it to the upgrade command below.
+Adopt writes no backup, so omitting the flag then refuses `no-backup`. Subsequent upgrades omit it;
+when a backup exists, supplying it refuses `rehearsal-required`.
+
 ```sh
 echo "<bundle sha256>  /root/<sha>.bundle" | sha256sum -c -
 (
@@ -244,10 +257,13 @@ When a run doesn't exit 0, send Mos the exit and the last lines of `/var/log/mos
 After `case2-install.sh` succeeds, create the human CLI wrapper before running
 `stack-copy.sh check --units external`. Creating this wrapper is a separate live
 step requiring Mos's go (`scripts/stack-copy.sh:3`); the install does not create it. Run only on that go:
+The step runs under sudo's `secure_path`, so it sets its own PATH and umask.
 
 ```sh
 sudo -u mosaic -H /bin/sh <<'MOSAIC_WRAPPER'
 cd / || exit 1
+umask 077
+PATH=/opt/mosaic-stack/node-<v>/bin:/usr/sbin:/usr/bin:/sbin:/bin; export PATH
 mkdir -p -- "$HOME/.npm-global/bin" || exit 1
 "$HOME/src/mosaic-stack/scripts/stack-copy.sh" wrapper <sha> --create
 MOSAIC_WRAPPER
@@ -303,7 +319,7 @@ file with `stat -c '%U %a %n' PATH`, never `cat`. `<business>` is the host's bus
 |---|---|---|---|---|---|
 | `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `ZAI_API_KEY=...` | 0600 | mosaic | library | `<path> sets ZAI_API_KEY` |
 | `/srv/mosaic/.config/mosaic-dev/secrets/` | the Console's proof file, the path the args file's `--proof-file` names | 0600 or 0400 | mosaic | library | none. `case2-unit.sh` refuses at render. Precheck line missing. |
-| `/srv/mosaic/.config/mosaic-dev/secrets/<business>/` | `library-gitea` | 0600 | mosaic | library | `case2-precheck.sh --business <business> --library-status`, after the bus is up: `library-last-error library-credential-missing` until it's placed, then `library <project> last fetched ...` |
+| `/srv/mosaic/.config/mosaic-dev/secrets/<business>/` | `library-gitea` | 0600 | mosaic | library | After a unit start, wait for loopback `/healthz` with the bounded retry in section 10 before `case2-precheck.sh --business <business> --library-status`: `library-last-error library-credential-missing` until it's placed, then `library <project> last fetched ...` |
 | `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
 
 The installer key, `<configDir>/secrets/hosts/<business>/<hostId>/installer`, is not a slot anyone places.
@@ -339,4 +355,20 @@ Precheck line: `ok      notifier config <dataRoot>/notify/<business>/notify.json
 None by hand. I2's `case2-unit.sh` enables both units, starts the bus, then the Console (Q189, mos-1122).
 `stack-upgrade.sh` stops and starts both units itself.
 
+After a unit start, wait for the Console's loopback `/healthz` to succeed before running
+`case2-precheck.sh`, including `--library-status` in section 9. Replace `<port>` with the installed
+Console unit's `--port`. This retry is bounded to 30 attempts, each with a 2-second timeout, with
+1 second between failed attempts. If it prints STOP, stop here; do not run the precheck or adopt.
+
+```sh
+(
+  attempts=0
+  until curl -fsS --max-time 2 -o /dev/null 'http://127.0.0.1:<port>/healthz'; do
+    attempts=$((attempts + 1))
+    [ "$attempts" -lt 30 ] || { echo "STOP: loopback healthz did not become ready; do not continue" >&2; exit 1; }
+    sleep 1
+  done
+)
+```
+
 Precheck line: none. Both units active after the render: precheck line missing.