Compare commits

...
Author SHA1 Message Date
jason.woltje 9b53620185 sage: 10-24 window draft 0 counts grant look-alike events before --migrate (row 234 A22b, gizmoduck-0107) 2026-10-20 23:29:34 -05:00
jason.woltje 7662b04902 sage: 10-24 window draft 0 adds rows 232 (e65a3ab4d), 241 (5d4a61776) and 242 (5fb736b92, fc1cb856b); 199 and 210 evidence list due in draft 1 2026-10-20 23:24:00 -05:00
jason.woltje 04ce8dfd56 queue: revs 2639-2641 (row 242 Duckworth approve record, landing note and done) 2026-10-20 23:22:41 -05:00
jason.woltje fc1cb856b0 publish: row 242 follow-up docs, the git socket comments name the hold and the late record; BUILD.md residual and :311 cite (Duckworth N1, N2, N4, #1748)
Megavolt's c5a125f4b on megavolt/row242-docs, parent 5fb736b92 (the row 242
landing). Comments and .md only: git-socket.mjs changes comment lines at
:39-44 and :433-435, BUILD.md records N2 and N4. No test pins either path's
hash. Landed without a gate under the doc-only rule.
2026-10-20 23:17:14 -05:00
jason.woltje c0c4907af9 sessions: Sage, rows 241 (5d4a61776) and 242 (5fb736b92) landed, queue revs 2628-2641 2026-10-20 23:14:49 -05:00
jason.woltje 5fb736b92d publish: a retry after publish-timeout pushes once; the bus holds the decision until its push ends and records a late push (row 242, #1748, P19 440)
Megavolt's candidate 635c10924 on megavolt/row242, manifest 1fbfab6c (3 paths:
agents/megavolt/work/queue-242/BUILD.md, packages/bus/src/runtime.mjs,
packages/cli/tests/publish-socket.test.mjs). Duckworth approved (comment 29889, rev 2639).

Mutants M1 and M2, run by Megavolt on Sage's slot: both killed (publish-socket 8/1 each).
Sage's landing gate on 12b8538e0 (96bac588f plus these 3 paths): 15 packages 2826 pass 0 fail (bus 500, cli 594, seat 164), 10 scripts/test-*.sh suites 429 pass 0 fail plus 156 node tests in the queue suite, 03:59Z to 04:12:44Z

The 3 paths are unchanged on refactor since the candidate's base 8fede5fc6. Follow-ups:
Duckworth's doc notes N1, N2 and N4 go in a later doc-only commit; the late-record notice
to the run is Vikunja 440 comment 432. No live change.
2026-10-20 23:13:06 -05:00
jason.woltje 96bac588fb queue: revs 2628-2638 (row 241 approve record, landing note and done; row 232 landing note and done; row 233 brief re-pin and Darkwing claim; row 242 in review round 1, request 29888) 2026-10-20 22:47:48 -05:00
jason.woltje 5d4a617762 bus: row 241 one decision lands once; a broker test with two checkLand tickets on one decision, where the second recordLand refuses decision-consumed and records nothing (row 241, #1747, P19 441)
Megavolt's candidate 3e3ed303e on megavolt/row241, manifest e8266864 (2 paths:
agents/megavolt/work/queue-241/BUILD.md, packages/bus/tests/land.test.mjs). Tests and
packet only, no source change. Duckworth approved (comment 29870, rev 2628).

Sage's slot on bbf3b1ee8 (0444acd08 plus these 2 paths):
- mutants M1 (#spent recheck, broker.mjs:963), M2 (CAS old value, seat working.mjs:391),
  M3 (land-busy, cli git-socket.mjs:351): all killed
- full gate: 15 packages 2797 pass 0 fail (bus 500, cli 565), 10 scripts/test-*.sh suites 429 pass 0 fail plus 156 node tests in the queue suite, 03:21:13Z to 03:34:16Z

Row 232's landing e65a3ab4d sits between that base and this commit. Its 9 paths are
disjoint from these 2 and touch no bus code. No live change.
2026-10-20 22:34:30 -05:00
jason.woltje e65a3ab4d2 stack-release: --units external for root-owned units, with confirm, revert and gate --since; stack-copy copy --bundle and wrapper --create (row 232, #1734, installer I1)
Blobs of Darkwing's round 2 da2da2f6c on origin/darkwing/row232-r2 (base
8fede5fc6), the 9 manifest paths, manifest 7af3c943. Round 2 is a
rebase-only round over row 230 (c8bc5cf4c): packages/cli/README.md is a
clean 3-way merge with 232's hunk unchanged, and the other 8 paths keep the
round 1 hashes (manifest 7a3b4ea1). Filbert approve round 1 (comment 29829)
and round 2 (comment 29865).

Under --units user every command line and every byte of output is the
base's. Under external the script reads root's system units, starts and
stops none, stages a copy behind AWAITING-UNITS, and confirm gates root's
start on the bus unit's ActiveEnterTimestamp and launches list.

Mutants 17 of 17 killed (BUILD.md). Darkwing's landing gate on da2da2f6c,
03:07:15Z to 03:17:48Z: packages/cli 593 pass 0 fail, 10 scripts/test-*.sh
suites 429 pass 0 fail (plus 156 node tests in the queue suite). Refactor
moved to 0444acd08 since, three commits touching no manifest path. No live
change: the stack runs from copy 4429a104f, and --units user stays the
default.
2026-10-20 22:19:47 -05:00
jason.woltje 0444acd084 queue: revs 2618-2627 (row 232 round 2 approve; row 241 in review round 1, request 29868; row 234 and row 242 claimed)
Gizmoduck: row 234 claim (2618). Darkwing: row 232 rebase round 2 onto 8fede5fc6, in-progress and in-review with request comment 29860 (2619-2621). Filbert: row 232 round 2 record, approve, comment 29865 (2622). Megavolt: row 241 in-review round 1, request failed on the token (2623-2624), row 242 claim (2626). Sage: the row 241 request as jarvis and its resolve to comment 29868 (2625, 2627).
2026-10-20 22:06:29 -05:00
jason.woltje 052ba95c7c sessions: Filbert row 232 I1 review round 2, approve (#1734) 2026-10-20 22:01:43 -05:00
jason.woltje a52d64ed41 sage: 10-24 window draft 0 names the landed commits for rows 210 (f5753c0e8), 237 (d7d943199) and the PRD (0adf8ddfe) 2026-10-20 21:52:36 -05:00
jason.woltje 8fede5fc6a sessions: Sage, row 210 landed f5753c0e8, queue revs 2604-2617 2026-10-20 21:51:47 -05:00
jason.woltje c084d8d423 queue: revs 2604-2617 (row 210 done, landed f5753c0e8; rows 241 and 242 added and briefed, 241 claimed by Megavolt; row 234 briefed)
Sage: rows 241 and 242 added and briefed (#1747, #1748), the row 210 round 3 request as jarvis, the row 210 landing note and done move, row 234 queued to briefed. Gizmoduck: row 210 r5 start and in-review round 3 (2607-2611). Duckworth: row 210 round 3 record, approve (2614). Megavolt: row 241 claim (2606).
2026-10-20 21:50:59 -05:00
19 changed files with 3497 additions and 535 deletions
+287
View File
@@ -0,0 +1,287 @@
# Row 232 (I1): stack-release external units and bundle copies
Brief: `docs/plans/2026-10-20_installer-rows.md` section I1, blob `f35e7e57daee` (commit ad6f0be7a). Gitea
#1734, Vikunja P19 #432. Owner Darkwing, reviewer Filbert, gate owner Sage. Design input: row 195 DESIGN r6,
`agents/darkwing/work/queue-195/DESIGN.md` at b7239bb66 (approved in duckworth-0055). Where the brief and
r6 differ, the brief wins. sage-1561 ruled where the stack user's `stack-release.sh` runs: from its own
clone of the release bundle, so `check_target`, the plan's commit list and status drift are unchanged and
nothing adds `safe.directory`.
## Files
Base: cb5fdb064 (row 199's landing, on top of row 200's 9e1085816). Branch `darkwing/row232-r1`. The first
build, 0af0c73c6 on `darkwing/row232`, sat on 00d4cc1f6 (row 190); row 200 landed first, so this one is
rebased onto it (see D9 and D10).
- `scripts/stack-release.sh`: `--units user|external`, the `sc` and `jctl` backend, `confirm`, `revert`,
`gate --since`, `rollback --to`, `--plan --expect-from`, the external gate (start floor, PM check from
`launches list`), `external-pm`, and the `awaiting units:` status line.
- `scripts/stack-copy.sh`: `check --units external`, `copy --bundle FILE SHA`, `wrapper ID --create`.
- `packages/cli/tests/stack-release-external.test.mjs` (new, 25 tests).
- `packages/cli/tests/stack-release-fixture.mjs` (new): the helpers `stack-release.test.mjs` defined, moved
so both files share them. See deviation D1.
- `packages/cli/tests/stack-release.test.mjs`: the helpers leave and one import line comes in. The 74 tests
have the same names and the same bodies as on the base.
- `packages/cli/tests/stack-copy.test.mjs`: three new `row 232:` tests and a wider `node:fs` import.
- `docs/TOOLS.md`: the stack-copy usage block and three paragraphs, the stack-release usage block, and a new
subsection "Root-owned units (`--units external`, row 232)".
- `packages/cli/README.md`: one paragraph.
- This file.
The worktree's `node_modules` symlink is not part of the candidate.
## User mode is unchanged
Every `systemctl` call goes through `sc` and every `journalctl` call through `jctl`. Under `user` they
are `systemctl --user "$@"` and `journalctl --user "$@"`. Under `external`, `sc` allows only `show`,
`is-active` and `list-units`, and `jctl` returns 1, so an external run can't start, stop, install or read
the journal even through a path the tests miss.
Two tests hold the brief's "every command line and every byte of output":
- `user mode: every systemctl call carries --user, and activate, rollback and check make the base's
calls`. A fixture repo gets the base's two scripts from `git show cb5fdb064:`. The same activate,
rollback and `stack-copy.sh check` run from both trees, and the recorded argv (with the fixture root and
port replaced by placeholders) must match line for line. Every call carries `--user`.
- `user mode: a default activate --plan prints the base's bytes`, for four plans.
The base output is produced live from the base blobs, not stored as a fixture file. See deviation D3.
## r6 section 7, test by test
All in `stack-release-external.test.mjs` unless named.
| r6 :742-812 item | Test |
|---|---|
| `units-running` (N4), store, log and TAKEN left | `units-running (N4) refuses while a unit is active...`; also checks that `failed` counts as down and that the Console is checked first |
| activate leaves the pointer, writes `AWAITING-UNITS` and a backup | `activate stages: ...`, which also checks the set's file list, `UNITS.sha256`, no `--user`, no journal call and `status` |
| `open-activation` with zero, two, the wrong copy, one not open | `confirm and revert refuse open-activation ...` |
| `confirm` polls: 3 s fails at `--timeout 1`, passes at `--timeout 10` | `confirm polls: bus status good after 3 s fails ...` and `confirm polls to a pass ...` |
| `stale-start` floor with `started` at .900 | `stale-start: @1792540799 fails ...` and `stale-start is a gate code ...` |
| `confirm` pass writes pointer, ok, removes marker; timeout writes `FAILED` and exits 1 | the two polling tests |
| `activation-failed` when `FAILED` exists | `confirm polls: ... FAILED then refuses activation-failed` |
| lines carry the started op, from, to, activation and backup | `sameAsStarted` in every confirm test, and sequence 7 |
| `gate` (N2) with and without `--since` | `gate (N2, A2f): ...` and the usage test |
| `--plan --expect-from` (N5), activate and `rollback --to` | sequence 7 |
| A3 order: capture-failed, marker-failed, wrapper failure after migrate | `A3 order: ...` and sequence 2 |
| refusals log no activation; `interrupted` unchanged | `open-activation` test and sequence 5 |
| PM check: one, none, two, early, not live, other business, other instance; no journal call | `the external PM check reads launches list ...` (modes '', coder, none, twolive, early, dead, other) and `journalCalls` in every external test |
| `external-pm` | `pm-on and activate --pm-after-harness refuse external-pm` |
| `revert` restores TAKEN only after a migration | `revert puts the wrapper back and leaves the store ...` and sequences 2 and 8 |
| `confirm --rollback` per cause, `unrecorded`, `rollback-failed`, mismatch | sequences 1, 2, 3, 5 and 8, and `confirm --rollback refuses rollback-mismatch, and logs rollback-failed ...` |
| sequences 1-8 | `sequence 1:` to `sequence 8:`, each ending with the `closed` check (one started and one result line, empty `interrupted`, no marker or `FAILED`, a following `--plan` that doesn't refuse `interrupted-run`) |
| `rollback --to` refuses not-previous; `rollback --restore` finds an external backup | sequence 7, and `confirm --rollback after a --restore swap ...` |
| `gate` writes nothing | the `gate (N2, A2f)` test and sequence 4 compare the log, the pointer and the state tree |
| `confirm`, `revert`, `gate`, `--to` refuse without `--units external` | `confirm, revert, gate, --to and --expect-from need --units external ...` |
| default path: every call carries `--user`; default `--plan` bytes | the two user-mode tests above |
The brief's additions, in `stack-copy.test.mjs`:
- `row 232: copy --bundle copies a SHA the bundle's refactor reaches, byte-identical to copy SHA, and
refuses the rest`: a short SHA, a missing file, a symlink, a bundle that fails `git bundle verify`, a
bundle with no `refs/heads/refactor`, a SHA not in the bundle, and a SHA the bundle carries on another
branch that refactor doesn't reach. The bundle copy's tree matches `copy SHA`'s byte for byte, and the
`release:` and `bundle sha256:` lines are checked.
- `row 232: wrapper --create writes the first wrapper only, the same bytes wrapper ID writes, and refuses
over any file`, a symlink included. Without `--create` a missing wrapper still refuses.
- `row 232: check --units external reads the system units, the default reads the user units`.
## Mutants
`mutants.sh` in my scratch (not committed, it edits the worktree), 17 mutants: r6's re-pin list (:817-838)
minus the three `case1-repin.sh` ones, plus the brief's two. Each one applies exact replacements after a
checked backup, runs `bash -n`, runs its named killers, restores the file and checks its sha256. Two
controls run first with no change: the whole external file, and stack-copy's `row 232:` tests.
| Mutant | Killers |
|---|---|
| m01 `confirm` writes the pointer on a gate failure | `confirm polls: bus status` |
| m02 `gate_once` in place of `gate` | `confirm polls to a pass`, `stale-start is a gate code` |
| m03 PM check counts any instance | `external PM check` |
| m04 PM check drops the `startedAt` bound | `external PM check` |
| m05 `confirm` takes the newest `AWAITING-UNITS` | `open-activation` |
| m06 `rollback --to` skips the comparison | `sequence 7:` |
| m07 the default path drops `--user` from one call (`is-active`) | `user mode: every systemctl` |
| m08 the timeout writes r4's interim `rollback pending` line | `confirm polls: bus status`, `sequence 1:`, `sequence 5:` |
| m09 `AWAITING-UNITS` last, after the wrapper | `sequence 2:` |
| m10 a refusal names the open activation | `sequence 5:`, `open-activation` |
| m11 `stale-start` is a refusal | `stale-start is a gate code` |
| m12 `revert` restores without `MIGRATING` | `revert puts the wrapper back` |
| m13 `confirm` logs op `confirm` | `sequence 7:`, `confirm polls to a pass` |
| m14 `MIGRATING` after the migration | `sequence 8:` |
| m15 `gate` takes its floor from the log without `--since` | `gate (N2` |
| m16 `copy --bundle` skips the reachability check | stack-copy `row 232:` |
| m17 `wrapper --create` overwrites | stack-copy `row 232:` |
## Deviations and choices
D1. The shared helpers moved to a new file, `packages/cli/tests/stack-release-fixture.mjs`. The brief's
files list names `stack-release.test.mjs` "where a shared helper moves" but not a destination. Importing
from the other test file would run its 74 tests twice, so the helpers need a module of their own.
D2. The fixture's T17 check (what a run may never remove) now exempts a rollback set's `AWAITING-UNITS` and
`FAILED`. `confirm` removes both when it closes the activation (r6 A5 and R4). User mode writes neither, so
user-mode runs check exactly what they did.
D3. The user-mode argv and `--plan` tests read the base's scripts with `git show cb5fdb064:` rather than
storing their output as a fixture file. A later row that changes user-mode output on purpose moves the
`BASE` constant in the test to its own base, or the test fails. I chose this over a stored fixture because
a stored copy can drift from the base without anyone noticing.
D4. `marker-failed` is reached by making `state/stack-rollback` unwritable. r6 says "an existing `$RS`", but
`next_activation` never picks a taken name, so no test can make the run meet an existing `$RS`. The
exclusive `mkdir` is still what refuses.
D5. The stale-start polling case turns the enter time to `@1792540800`, not r6's `@1792540805`, with the
started line at 1792540800.900. That also checks that the floor is the whole second, so .900 doesn't push it
to 1792540801.
D6. No fake `sudo` or `case2-unit.sh`. r6's sequences ran under them because r6 had `case1-repin.sh` drive
the steps, and that script is I3's. Here the test plays root: it stops and starts the fake system units and
installs with the existing `bus-service.sh` and `webui-service.sh` stand-ins, between the real
`stack-release.sh` calls, on one log.
D7. The fake marker in the `open-activation` test is named `20000101T000000Z-<C1 prefix>` so it sorts
before the real one. A marker that sorted after it would leave m05 (newest marker) alive.
D8. m10 sets `ACT` inside `open_activation`, where r6's wording ("ACT set before the checks") lands in this
code. `ACT` is set only after every check in `confirm`.
D9. Row 200's `--sso` is a usage error under `--units external`, with the proof flags, `--no-proof` and
`--no-send`. It is a Console flag, and root's Console unit carries the Console flags; the script under
external renders no Console line and never calls `console_for_target`, so `sso_checks` doesn't run there
either. The usage text, the TOOLS.md bullet and the external usage test name it.
D10. Row 200's `config_checks` runs under `--units external` too, after `plugin_checks`, in both the
`--plan` and run paths of `activate` and `rollback --to`. It checks the target's loader against the stack
user's `config.json`, which the bus on a VM still reads, so a `console` key the target refuses would break
root's bus start the same way. The cost is one more refusal (`config-console-key`) an external run can
print; I took it over a bus that fails after root's start. The rebase changed no user-mode line: the
conflicts were the `SSO=0` globals line, the `plan` body (row 200's `SSO_PLAN` line sits in the user
branch) and the two `--plan` lines that gained `config_checks`. In the tests, row 200's fixture lines
(the loaders, `sso-config.mjs` and the `--sso` cli.mjs line for C1 and C) moved into
`stack-release-fixture.mjs` with the rest of the copy setup, and the `ALGS` import stays in
`stack-release.test.mjs`, which T57 uses.
## duckworth-0055's notes on r6 (brief line 169)
Duckworth's r6 approval (REVIEW-r5.md at 23f5520b5) carried two notes.
- N1, A2's stop timeout under `gate --since`. A unit that never stopped keeps its old `ActiveEnterTimestamp`,
so `gate --since` fails it with `stale-start`, and the run ends at exit 2 with root stopping the units.
That is intended: a unit that won't stop is a fair reason to stop. In I1 this is the gate code and its
polling (`stale-start: @1792540799 fails ...`, `gate (N2, A2f)`, m11). The run sheet's A2f line and the
re-pin test that points "exits 1 with OLD's gate passing" at an A3 exit 3 belong to I3, which owns
`scripts/stack-upgrade.sh` and `docs/guides/host-run-sheet.md`. I'll name both in the I3 build.
- N2, the claude path under `/opt/mosaic-stack` (gosalyn-0058, sage-1522). I1 doesn't touch it. Under
`--units external` the script checks no claude pin of its own (`no-claude-pin` applies only to user
mode), because root's unit carries it. The `--claude` value, the managed-settings reasoning, F6 and the
run sheet's claude step go to I2 and I3.
## Run environment
Worktree `/mnt/storage/scratch/darkwing-r232`, `TMPDIR=/mnt/storage/scratch/tmp/r232`, node from PATH.
One external file run takes about 94 s.
## Results
Singles on the rebased candidate, before the gate:
| File | Tests | Pass | Fail |
|---|---|---|---|
| `packages/cli/tests/stack-release-external.test.mjs` | 25 | 25 | 0 |
| `packages/cli/tests/stack-copy.test.mjs` | 16 | 16 | 0 |
| `packages/cli/tests/stack-release.test.mjs` (row 200's T57 included) | 192 | 192 | 0 |
| `packages/cli/tests/stack-instances.test.mjs` | 23 | 23 | 0 |
| `packages/webui/tests/pinned-copy.test.mjs` | 10 | 10 | 0 |
| `packages/cli/tests/channel-recovery.test.mjs` (imports `channel-fixture.mjs`, which names the scripts) | 1 | 1 | 0 |
| `packages/cli/tests/host.test.mjs` (reads `packages/cli/README.md`) | 42 | 42 | 0 |
These are every test file that reads `stack-release.sh` or `stack-copy.sh` (a grep of `packages/*/tests` and
`scripts/`). None of them pins a sha256 of a changed file. `packages/cli/tests/host.test.mjs` reads
`packages/cli/README.md` for a unit file name; this row only adds a paragraph there. The grep and the runs
were repeated on the rebased tree on 2026-10-21; the reader set is the same. A dry run of `mutants.sh`
matches each of the 17 replacements once on the rebased scripts.
### Gate, 2026-10-21, Sage's slot (sage-1618)
Candidate a0ffabc3f on `darkwing/row232-r1`, run by `~/darkwing-scratch/r232/gate.sh`: the cli suite, then
every `scripts/test-*.sh`, one after another, stopping at the first failure. busy.sh passed at 01:07:49Z
after Launchpad's single `stack-release.test.mjs` run ended. The gate stopped at the first one, so the
chain waited for that PID to exit and checked again. The EXIT trap removed the compose network; no
container was left.
| Step | Time (UTC) | Result |
|---|---|---|
| `packages/cli`, `node --test tests/*.test.mjs` | 01:07:50 to 01:17:53 | 561 pass, 0 fail, 0 skipped |
| `scripts/test-auth.sh` | 01:17:53 to 01:17:54 | 15 passed, 0 failed |
| `scripts/test-conductor.sh` | 01:17:54 to 01:18:00 | 17 passed, 0 failed |
| `scripts/test-config.sh` | 01:18:00 to 01:18:02 | 31 passed, 0 failed |
| `scripts/test-discord.sh` | 01:18:02 to 01:18:12 | 78 passed, 0 failed |
| `scripts/test-extension-package.sh` | 01:18:12 to 01:18:13 | 18 passed, 0 failed |
| `scripts/test-foundation.sh` | 01:18:13 to 01:19:01 | 44 passed, 0 failed |
| `scripts/test-land-work.sh` | 01:19:01 to 01:19:14 | 86 passed, 0 failed |
| `scripts/test-queue.sh` | 01:19:14 to 01:19:37 | 156 node tests pass, queue suite 28 passed, 0 failed |
| `scripts/test-release.sh` | 01:19:37 to 01:19:51 | 14 passed, 0 failed |
| `scripts/test-task.sh` | 01:19:51 to 01:20:44 | 98 passed, 0 failed |
### Mutants, 2026-10-21, same slot
`~/darkwing-scratch/r232/mutants.sh`, right after the gate. Both controls (the external file whole, and
stack-copy's `^row 232: ` tests) exited 0 on the unmutated tree. Each mutant was applied after a checked
backup and a syntax check, ran its killers from the table above, and was put back with its sha256
checked. The script stops on any restore mismatch, and it ran to its end.
| Mutant | Killer exit | Result |
|---|---|---|
| m01 pointer on gate failure | 1 | killed |
| m02 gate once | 1 | killed |
| m03 PM from any instance | 1 | killed |
| m04 no startedAt check | 1 | killed |
| m05 newest marker | 1 | killed |
| m06 --to not compared | 1 | killed |
| m07 one call without --user | 1 | killed |
| m08 interim pending line | 1 | killed |
| m09 marker written last | 1 | killed |
| m10 refusal names the open activation | 1 | killed |
| m11 stale-start refuses | 1 | killed |
| m12 revert always restores | 1 | killed |
| m13 confirm op | 1 | killed |
| m14 MIGRATING after migrate | 1 | killed |
| m15 gate floor from the log | 1 | killed |
| m16 bundle reachability unchecked | 1 | killed |
| m17 --create overwrites | 1 | killed |
17 of 17 killed, 01:22:16 to 01:24:32Z. I read the logs for m01 and m09, two of the quick kills: each failed on
its named test's assertion (m09 on `AWAITING-UNITS` in sequence 2), not on a load error.
## Round 2: rebase onto 8fede5fc6 (sage-1673)
Rebase only. Before the landing gate, a check of BASE..HEAD on the manifest paths found that row 230
(c8bc5cf4c) had changed one of them after the round 1 base, cb5fdb064. Sage ruled a re-round with a
scoped review, the same shape as row 210 round 3.
Branch `darkwing/row232-r2`: the two round 1 commits, a0ffabc3f and 84f1a91e6, cherry-picked onto
8fede5fc6 with no conflict, then this section. No code or test changed.
- `git merge-tree --write-tree origin/refactor darkwing/row232-r1` (refactor at 8fede5fc6) gives tree
837bde79236c with no conflict. The cherry-picked tip before this section has the same tree.
- `packages/cli/README.md` is a clean 3-way merge. Row 230's hunk is at lines 710-717 (the session credential
paragraph); this row's is a new paragraph at 773. With `git patch-id --stable`, the README diff from r1 to
r2 is 02b679e2, the same as row 230's README diff `cb5fdb064..8fede5fc6`. This row's README diff is
c395ecd2 on both bases (`cb5fdb064..r1` and `8fede5fc6..r2`).
- The seven code, test and doc paths other than the README hash the same as in manifest 7a3b4ea1, which
Filbert approved in round 1: `docs/TOOLS.md`, `packages/cli/tests/stack-copy.test.mjs`,
`packages/cli/tests/stack-release-external.test.mjs`, `packages/cli/tests/stack-release-fixture.mjs`,
`packages/cli/tests/stack-release.test.mjs`, `scripts/stack-copy.sh`, `scripts/stack-release.sh`.
This file also matched until this section was appended. Its other lines are unchanged.
- The one test that reads the README, `host.test.mjs:855`, looks for the unit's `EnvironmentFile=` line.
That line is unchanged context in row 230's hunk.
- No row 230 test reads `stack-release.sh` or `stack-copy.sh`, and no test of this row reads a row 230 file.
Nothing was run for this round. The landing gate runs on the r2 tip on Sage's slot.
Follow-up, doc-only (sage-1673): `scripts/case2-unit.sh:108` and `scripts/case2-precheck.sh:157` cite
`scripts/stack-copy.sh:8` and `:22` for `COPIES` and `copy_root`. With this row those lines are `:10` and
`:26`. Only comments are affected and no test reads them. Row 233 doesn't touch those scripts, so the fix
goes on the #1732 carry list.
+71
View File
@@ -0,0 +1,71 @@
# Row 241: one decision lands once (#1747, P19 441)
Megavolt, 2026-10-21 (from `date -u`). Brief: `docs/plans/2026-10-20_agent-credentials.md`
§ "Brief: row 237 follow-up, one decision lands once, with tests at the broker", c25785d3d. Source:
row 237 residual 3, megavolt-0068, megavolt-0069 and sage-1651. Base: origin/refactor f5753c0e8, row 210's landing.
Reviewer Duckworth, gate owner Sage. Tests only; no source changes.
## Why landing has no row 237 race
Row 237 fixed publish: two tickets on one decision both pushed, because the decision is spent only
after the push, and a push to a remote fork can't be taken back. Landing has the same ticket shape
(`checkLand` opens a ticket, `recordLand` spends the decision), but three guards keep one decision
from landing twice:
1. **land-busy, the outer guard.** `doLand` in `packages/cli/src/git-socket.mjs:351` refuses
`land-busy` while another landing runs. The slot is taken before the first await, and freed only
when the landing ends. There is one git socket per launcher, and one host per dataRoot (host.lock).
So through the socket two landings never overlap, and a later one stops at `checkLand` with
`decision-consumed`. Even past that, applying the same reviewed patch to the new main fails
`base-conflict` in `prepareLand`.
2. **The CAS on main.** `finishLand` (`packages/seat/src/working.mjs:391`) moves main with
`update-ref main <commit> <old>`, so a landing prepared on a tip that has since moved refuses
`base-moved` and writes nothing.
3. **The recordLand recheck.** `finishLand` calls `record()`, which is `recordLand`, under the same
repository lock. `recordLand` runs `#landFacts` again (`packages/bus/src/broker.mjs:963`), which
refuses `decision-consumed` once the decision is spent. When `record()` throws, `finishLand` moves
main back.
Guards 2 and 3 can't be reached end to end, because guard 1 stops a second landing first. So they are
pinned below the socket, and no test bypasses land-busy.
### The unrecorded-commit window
Between the `update-ref` and the end of `record()`, main names the squash, which isn't recorded yet.
That window is under the repository's lock, so no other push, review or landing in this host sees
it. If `record()` refuses, main moves back. If moving back fails, `finishLand` refuses
`rollback-failed` and main keeps naming an unrecorded commit (tested at `working.test.mjs:555`, the
main.lock case). That is existing behaviour, unchanged here.
## The test
`packages/bus/tests/land.test.mjs:312`. Two tickets pass `checkLand` on one decision before either
records. `recordLand` on the first lands. `recordLand` on the second refuses `decision-consumed`
(:319), and leaves one merge `action.allowed`, one RESULT and no `action.refused`. `landRefused`, as
the socket would call it, then files the refusal and ends the ticket.
**The assert that pins the recordLand recheck is :319.** The `#spent` line is shared with
`checkLand`, so dropping it also fails :144 (a `checkLand` after the decision is spent). Only :319
shows the recheck at record time.
Single file on this branch: land.test.mjs 18/18.
Whole suites on this branch before in-review: bus 500/500, seat 164/164, cli 565/565 (02:52 to 03:01Z).
## Mutants
| Mutant | Change | Killed by |
|---|---|---|
| M1 | drop the `#spent` line in `#landFacts` (broker.mjs:963) | land.test.mjs: :312 at :319, and :94 at :144 (16/2) |
| M2 | drop `land.old` from the `update-ref` in `finishLand` (working.mjs:391) | working.test.mjs:555 at :567, the `base-moved` assert (25/1) |
| M3 | drop the `land-busy` throw (git-socket.mjs:351) | git-socket.test.mjs:812 at :825, the `land-busy` assert (23/1) |
Each run was one test file on base f5753c0e8 (row 210's landing, which moved the `#spent` line from
:945 to :963), 2026-10-21 02:51Z. Each assert failed with "Missing expected exception" or "rejection".
`cmp` confirmed each source matched its saved copy after the restore. M2 is killed by the existing
working.test:555, so per the brief working.test.mjs isn't changed. Logs: `/mnt/storage/scratch/megavolt-r241/`
(m1.log to m3.log, with each diff).
## Gate
Per the brief: Duckworth approves the manifest; the bus, seat and cli suites and every
`scripts/test-*.sh` are green on Sage's gate; the three mutants are killed.
+108
View File
@@ -0,0 +1,108 @@
# Row 242 build: a retry after publish-timeout pushes once
Row 242, #1748, P19 440. Brief: docs/plans/2026-10-20_agent-credentials.md, "Brief: row 237
follow-up N1, a retry after publish-timeout pushes once" (9d165be8c). Base: origin/refactor
8fede5fc6 (row 210 landed f5753c0e8).
## The defect
The git socket polls publishResult until publishDeadlineMs, then refuses publish-timeout
(packages/cli/src/git-socket.mjs:433) while the bus process's push (runtime publishPush) can
still be running. The socket then calls publishRefused, which went straight to
broker.publishRefused: it filed the refusal and deleted the ticket. checkPublish no longer saw an
open ticket on the decision, so a retry got a new ticket and pushed a second ref while the first
push was still on its way to the fork.
## Fix chosen: hold the decision until the push ends, then record a push that reached the fork
packages/bus/src/runtime.mjs, `publishRefused` and `settle`:
- If the bus has no push for the ticket, publishRefused goes to the broker as before.
- If the push is still running, the refusal code is held on the entry and the ticket stays open.
checkPublish refuses another publish on the decision with publish-busy (row 237's check),
the same as when the socket is still polling.
- When the push ends, `settle` runs:
- A push that reached the fork is recorded with broker.recordPublish, which spends the
decision. The bus logs `pushed after the git socket refused <code>; recorded`.
- A push that failed is filed as publishRefused with the socket's code.
- If recordPublish refuses with a BusError, that code is filed instead.
- If the push has already ended when the refusal arrives, it's settled at once. The sage-1440 test
relies on this: its push fails fast, so publish-timeout is filed once, immediately, as before.
Why hold rather than kill. Killing the git client doesn't undo a push the fork has already
received: receive-pack can update the ref after the client dies. After a kill the bus would have
to ls-remote the fork to learn what landed, and the result could still race the server. Holding
waits for the push's own outcome. Each git run in publishRemote already has its own limit
(publishRemote defaults to checkout.mjs GIT_LIMITS), so a hold is bounded. Holding also reuses the existing publish-busy
state, and the refusal text is unchanged.
Why record and not only hold. If the hold just filed publish-timeout once the push ended, the
ticket would close and the decision would be free again with a ref on the fork. A retry would then
push a second ref. Recording the late push spends the decision, so the retry gets
decision-consumed.
The publish-timeout refusal text is unchanged (brief condition).
## Test
packages/cli/tests/publish-socket.test.mjs:302, "row 242: a retry after publish-timeout refuses
publish-busy while the bus process's push runs; the late push is recorded and the decision pushes
once".
- publishDeadlineMs 150 and a store token. The fork's pre-receive hook writes `held`, then loops
until the test writes `release`.
- The first publish gives publish-timeout. The test waits for `held`.
- A retry to branch `other` gives publish-busy (:322). Refused events: ["publish-busy"]. No
git.push.remote allowed event.
- The test writes `release` and waits until there is exactly one allowed event, with ref
refs/heads/coder/feat and the work commit.
- The next retry gives decision-consumed (:330).
- Exactly one git-receive-pack request reached the fake Gitea, and the fork has only
refs/heads/coder/feat. Refused events: ["publish-busy", "decision-consumed"]. No TOKEN appears in
events or logs.
Fail fast (sage-1681): the body is wrapped in try/finally, and the finally writes `release`. Under
the first mutant run, the assert at the retry failed and the hook was never released. setup's
teardown waits for the fake Gitea, which waited for the held push, so the file hung until I stopped
it. Now a failed assert ends the hook and the run. Each wait is bounded at 10 s with an assert.fail.
## Mutants
Script: /mnt/storage/scratch/megavolt-r242/mutants.sh. Each mutant: save, one-line sed, run
publish-socket.test.mjs whole, restore, cmp.
| Mutant | Change | Expected kill |
|---|---|---|
| M1 | export `publishRefused: (ticket, args) => broker.publishRefused(ticket, args)` again (removes the fix) | :302 at :322: the retry gets a new ticket and its own push, so publish-timeout, not publish-busy (seen in the stopped 03:06Z run) |
| M2 | `settle` files publishRefused without trying recordPublish (hold only) | :302 at the wait for one allowed event, then :330 |
Results (Sage's slot, sage-1686, 2026-10-21T03:35:00Z to 03:35:15Z): both killed, both restored
(cmp ok).
- M1: publish-socket 8/1, the row 242 test fails at :322 in 0.5 s.
- M2: publish-socket 8/1, the row 242 test fails at :327, the wait for one allowed event, through
the assert.fail inside `until` (:311) after its 10 s bound, in 10.3 s.
## Suites
On Sage's slot (sage-1686), on base 8fede5fc6 plus this row:
- bus `node --test tests/*.test.mjs`: 499/499, 03:35:20Z to 03:35:34Z. This is one fewer than
Sage's 241 gate (500) because the base predates row 241's land.test case.
- cli `node --test tests/*.test.mjs`: 566/566, 03:35:34Z to 03:43:47Z. That's 565 plus the
row 242 test.
Singles before the slot: publish-socket.test.mjs 9/9, bus publish.test.mjs 13/13. The seat suite
and scripts/test-*.sh are left for Sage's landing gate.
## Residuals
- The agent was told publish-timeout, but the push is recorded later and the run isn't
notified. The allowed event and the bus log line are the only trace. A later publish on the
decision gets decision-consumed, which tells the agent the decision was used. Whether the run
should get a RESULT for a late record is a follow-up question for Sage.
- A late push whose recordPublish refuses (Duckworth N2) is filed with that code: the ref stays on
the fork and the decision is not spent, the same residual as an in-band push whose record
refuses.
- The hold lives in the bus process's memory, like the broker's open tickets. A bus restart during
a hold drops both, the same as today for any publish in flight.
@@ -21,11 +21,11 @@ so Mos doesn't check this draft. Draft 1 names NEW and goes to Mos for the P0 ch
- OLD, live: `4429a104f6d65ccf165f6c83b5d1dce3147dfde0`, schema 3i, pointer `pm: true` (record 99a406ed3).
P1 re-reads the activation id and the Console flags from `status`. I don't copy them from the last window.
- NEW: open. It's the row 234 landing, or a later code landing if Mos rules. Candidates in landing order:
210 (r4 due 10-21), 237, 194, 234.
194, 234 (210, 237, 232, 241 and 242 have landed).
- REPO, `P`, `WIN=agents/sage/work/window-repin-<NEW12>/`, `SNAP`: as in `repin-4429a104f.md`, with the
sha checks at P0 and before S1, S2 and any D3.
## What NEW carries over OLD (so far, refactor c8bc5cf4c)
## What NEW carries over OLD (so far, refactor 04ce8dfd5)
`git log --first-parent 4429a104f..HEAD`, path-limited to packages, scripts, roles, contracts, adapters, skills
and the package files:
@@ -39,10 +39,13 @@ and the package files:
| 199 | cb5fdb064 | Credential store per agent. Check: bus start with no store entries for the live agents changes nothing, and the chief's and pm's launches still pass. The P3 rehearsal covers start. |
| N3 | 390ea5261 | README only. |
| 230 | c8bc5cf4c | Case 2 scripts, tools only. Not run live. |
| 210 | open | Host access. The live business has no `hosts`, so nothing is granted. Bus start runs the host-run sweep (M34d). P3 must show it removes nothing. |
| 237 | open | Publish busy check (199 N1). |
| 210 | f5753c0e8 | Host access. The live business has no `hosts`, so nothing is granted. Bus start runs the host-run sweep (M34d). P3 must show it removes nothing. |
| 237 | d7d943199 | Publish busy check (199 N1). |
| 232 | e65a3ab4d | Installer I1: `stack-release.sh --units external` and `stack-copy.sh copy --bundle`, `wrapper --create`. Both are opt-in, and this window uses neither. The default units path must render the same units, so the P2 unit diff covers it with row 190. P0 checks that no window command line carries `--units external`. |
| 241 | 5d4a61776 | Test only (bus land.test.mjs). No live effect. |
| 242 | 5fb736b92 | Bus publish hold: after publish-timeout the bus holds the decision until its push ends, then records a late push. The live business has no publish in flight at W1, and P3 shows none open. Its follow-up fc1cb856b is comments only (git-socket.mjs). |
| 234 | open | Machine grant row a and the 3i to 3j schema step. `--plan` must say `migration: 3i -> 3j`. |
| PRD 1.7+1.8 | open | `docs/prd/mosaic-stack.md` and `packages/tasks/requirements.json` move to version 1.8 (mos-1154, mos-1156, DECISIONS Q192). The live pm cites 1.6 until this window. Check after: the pm's requirements read answers version 1.8. |
| PRD 1.7+1.8 | 0adf8ddfe | `docs/prd/mosaic-stack.md` and `packages/tasks/requirements.json` move to version 1.8 (mos-1154, mos-1156, DECISIONS Q192). The live pm cites 1.6 until this window. Check after: the pm's requirements read answers version 1.8. |
Service script drift OLD to HEAD is 415+/91- over the four scripts, mostly row 190. P5 holds them at NEW.
@@ -54,6 +57,12 @@ Service script drift OLD to HEAD is 415+/91- over the four scripts, mostly row 1
- Rollback after a migration isn't the plain rollback. OLD is 3i and the store is 3j, so going back needs
`--restore <W5 activation>`, and that loses the events written after W5. The row 181 restore guard counts
11 tables, and the untimed ones need the D3 snapshot. Mos decides any restore past W5, as before.
- Grant look-alikes (row 234 A22b, gizmoduck-0107). `3i-3j.sql` adds `events_grant_actor` but doesn't check
rows already in the store, so after the migration only the actor filter in `GRANT_EVIDENCE` (actor_role NULL,
actor_run 'grant') ignores an old session-written row. P3, and again just before `--migrate`: on the
backup copy, count the `action.allowed` events whose body.action or body.operation starts with `grant.`, or
whose body has `hostLineLeft`. Expect 0, since no grant code has run live. Any nonzero count stops the window
until I've read those rows.
- P3 is the rehearsal with `--migrate` on a copy of the live backup. It must show the 3j step and the row
counts before and after.
@@ -71,4 +80,6 @@ history unless the window fails.
`--sso` included, from `status` after that window.
- Q3: who operates. If Darkwing is free, Darkwing operates and I check. Otherwise I operate as in the last
window.
- The P-step evidence list for rows 199 and 210 at bus start, written once both have landed.
- The P-step evidence list for rows 199 and 210 at bus start. Both have landed (cb5fdb064, f5753c0e8), so
it's due in draft 1. I take each log line and exit code from a P3 rehearsal run, not from the code.
- Rows 241 and 242 (bus): P3 shows no publish or land decision open before W5.
+3
View File
@@ -1393,3 +1393,6 @@ are never rewritten or removed; corrections are new entries.
2026-10-21T01:28:05Z | Darkwing (Opus 5.5, 7525c8e4) | row 232 I1 build (sage-1558, sage-1561 ruling a) | rebased onto cb5fdb064 (row 200), gate green and 17/17 mutants killed on Sage's slot; in review round 1, manifest 7a3b4ea1 at 84f1a91e6 on darkwing/row232-r1, comment 29825, revs 2595-2596; Filbert woken
2026-10-21T01:34:13Z | Filbert (Opus 5.5, 048f3c63) | row 232 installer I1 review r1 (#1734, manifest 7a3b4ea133cc) | APPROVE, Gitea 29829, queue rev 2600 (left for Sage's queue-commit); static review, gate and mutants are Darkwing's receipts; 3 non-blocking notes
2026-10-21T01:49:55Z | Sage (Opus 5.5, 1ef1e4f8) | row 237 landed d7d943199 (#1743, gate green, M1 killed); PRD 1.7+1.8 landed 0adf8ddfe (Honker 2884ee496, Duckworth approve bf52628f9, tasks 97/0 bus 478/0 on the tip, DECISIONS Q192); rows 239 (#1745) and 240 (#1746) added; 233 briefed; queue revs 2540-2603 committed 584e2bd82 | landed, push next
2026-10-21T02:51:46Z | Sage (Opus 5.5, 1ef1e4f8) | row 210 landing (r5 0485384e9, Duckworth r3 29851) | landed f5753c0e8, patch-id 723b1c83 = r5 diff, manifest 688d38b9 OK; gate 2796/0 packages, 429/0 scripts; mutants 52/53 (M22b equivalent); #1707 closed (29853), Vikunja 424 comment 429; queue revs 2604-2617 c084d8d42; row 234 briefed; Megavolt has the slot for 241 (sage-1668)
2026-10-21T03:01:43Z | Filbert (Opus 5.5, 048f3c63) | row 232 installer I1 review r2, rebase scope (#1734, manifest 7af3c94308e0, da2da2f6c on 8fede5fc6) | APPROVE, Gitea 29865, queue rev 2622 (left for Sage's queue-commit); README = predicted 3-way blob 89a95b32, 7 blobs same as r1, BUILD.md +31/-0; corrected my r1 reader claim (host.test.mjs:855 reads cli README)
2026-10-21T04:14:49Z | Sage (Opus 5.5, 1ef1e4f8) | lead: rows 241 and 242 landed, slot order | row 241 landed 5d4a61776 (#1747 closed 29883, M1-M3 killed, gate 2797/0 + 429/0); row 242 landed 5fb736b92 (#1748 closed 29894, Duckworth approve 29889, gate 2826/0 + 429/0); late-record notice ruled a follow-up (Vikunja 440 c432); row 233 brief re-pinned rev 2631; carry items on #1744 (29885) and #1734 (29886); queue revs 2628-2638 committed 96bac588f
+72 -4
View File
@@ -408,8 +408,9 @@ exposure of a named Console waits for Jason after Q175 SSO.
```
scripts/stack-copy.sh copy COMMIT [--copies DIR] [--allow-worktree]
scripts/stack-copy.sh wrapper ID [--instance NAME] [--copies DIR] [--bin DIR]
scripts/stack-copy.sh check [--instance NAME] [--copies DIR] [--bin DIR] [--business ID]
scripts/stack-copy.sh copy --bundle FILE SHA [--copies DIR]
scripts/stack-copy.sh wrapper ID [--instance NAME] [--copies DIR] [--bin DIR] [--create]
scripts/stack-copy.sh check [--instance NAME] [--copies DIR] [--bin DIR] [--business ID] [--units user|external]
```
One read-only tree supplies the bus, Console and human CLI (row 85,
@@ -423,6 +424,14 @@ renames beside the target. A published existing copy stays unchanged.
Dependencies use the exact lock and carry Pi's installed lock record.
Failure publishes nothing and removes the temporary tree.
`copy --bundle FILE SHA` (row 232) is for a host with no refactor checkout.
SHA is a full 40-character id. The bundle must pass `git bundle verify`,
carry `refs/heads/refactor`, and reach SHA from it; each miss refuses and
writes nothing. The copy is then made from SHA as `copy` makes it, and the
script prints `release:` with the RELEASE value at SHA and `bundle sha256:`
with the bundle's digest. That pair is the release identity on such a host.
Without `--bundle`, `copy`'s reachability rule is unchanged.
`wrapper` is a live switching step, reserved for Jason or Mos. It accepts
only an existing copy and a regular non-symlink mosaic wrapper with the
Q81 marker on line 2 and its exact lease line to mosaic-old. It preserves
@@ -430,11 +439,16 @@ both lines, adds comments naming the copy root and Q81's mosaic-old lease
probe, and atomically replaces the wrapper's final exec with the
copy's scripts/mosaic. --bin is a fixture override; the default is
~/.npm-global/bin. Neither the Q81 worktree nor mosaic-old is removed.
`wrapper ID --create` (row 232) writes the first wrapper on a new host, the
same file `wrapper` writes. It refuses when anything is at the wrapper path,
a symlink included, and never replaces it. Without `--create`, a missing
wrapper still refuses.
`check` changes nothing: it reads effective systemd WorkingDirectory and
ExecStart (including drop-ins) for the bus and Console, plus the wrapper
exec. It prints all three paths and succeeds only on the same existing
copy; absent/failed reads and a mismatched ExecStart fail. Business defaults
to mosaic-stack. Queue work continues on the canonical scripts/mosaic.
to mosaic-stack. `check --units external` (row 232) reads root's system units
with `systemctl show`, without `--user`. Queue work continues on the canonical scripts/mosaic.
Bus render/install accept --copy ID, --copies DIR, --pm and the unchanged
--claude pin. Without --copy the rendered unit stays the checkout unit.
@@ -455,7 +469,10 @@ scripts/stack-release.sh activate ID BUSINESS [--pm | --pm-after-harness] [--mig
scripts/stack-release.sh pm-on BUSINESS [--timeout S]
scripts/stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan]
scripts/stack-release.sh status BUSINESS
common options: --copies DIR, --bin DIR, --instance NAME
scripts/stack-release.sh confirm ID BUSINESS --units external [--rollback OLD --cause A3|A4|A5] [--timeout S]
scripts/stack-release.sh revert ID BUSINESS --units external
scripts/stack-release.sh gate ID BUSINESS --units external --since EPOCH [--timeout S]
common options: --copies DIR, --bin DIR, --instance NAME, --units user|external
--events N is the total of the per-table counts the post-backup-events refusal prints: events and
every table in COUNTED.
```
@@ -711,6 +728,57 @@ Usage exits 4. Nothing prunes rehearsal directories.
starts no thread ticker, and leaves an earlier host's `sessions.json` and
its runs alone. No unit passes it, and `--pm` with it is a usage error.
#### Root-owned units (`--units external`, row 232)
On a host whose bus and Console are root's system units, the stack user runs
`stack-release.sh` with `--units external` from its own clone of the release
bundle (sage-1561). The script then calls `systemctl show`, `is-active` and
`list-units` without `--user` and reads no journal. It starts, stops and
installs nothing. Root does those steps, and the I3 driver calls the verbs
below in order. Without `--units`, every command line and every byte of
output is what it was before row 232. `confirm`, `revert`, `gate`, `--to`
and `--expect-from` are usage errors without `--units external`.
- `adopt`, `status`, `activate ID` and `rollback --to ID` take the flag.
`rollback` then needs `--to`, and `--to` must be the copy `rollback` alone
would pick from the log, or it refuses `to-not-previous`.
- `--plan --expect-from OLD` refuses `pointer-differs` when the pointer
names a copy other than OLD.
- `pm-on` and `activate --pm-after-harness` refuse `external-pm`; the PM
flag is in root's bus unit. `--pm`, `--tester-port`, the proof flags,
`--no-proof`, `--no-send` and `--sso` are usage errors; root's Console
unit carries the Console flags.
- `activate` and `rollback` refuse `units-running` unless both units read
inactive or failed and the store is closed. They make the rollback set
with an exclusive `mkdir`, write `AWAITING-UNITS` (the target copy) first,
capture the pointer, the wrapper and both units' show output and unit file
sums, back up and migrate or restore the store, and write the wrapper.
They write no pointer and start nothing. The last line says the run is
staged and names `confirm`. A failure after the capture writes `FAILED`
with its code, logs nothing more and exits 1; the activation stays open.
- `confirm ID BUSINESS` needs exactly one `AWAITING-UNITS` for copy ID whose
activation the log holds open, or it refuses `open-activation`. It gates
the start root made and writes the pointer and an `ok` line with the
started line's op, from, to and backup.
`confirm --rollback OLD --cause A3|A4|A5` gates OLD after root put the old
units back and logs `failed` or `gate-failed` with `rollback ok` (exit 1)
or `rollback-failed` (exit 2). Either removes `AWAITING-UNITS` and `FAILED`.
- `revert ID BUSINESS` puts the saved wrapper back and, when the run had
started a migration, restores the backup taken before it. It writes no log
line and can run again.
- `gate ID BUSINESS --since EPOCH` gates a copy with no open activation,
takes no lock and writes nothing: 0 on a pass, 1 otherwise. EPOCH is root's
clock before root started the units.
The gate under `--units external` replaces the journal read. The bus unit's
`ActiveEnterTimestamp`, in whole seconds, must be at or after the started
line's second (`confirm`) or `--since` (`gate`); an older start fails
`stale-start`, so a unit that never stopped can't pass. With the PM on, the
gate reads `launches list --json` for exactly one live launch of the PM's
instance (the business's `launch.by`), for this business, whose run started at
or after the floor. `status` adds an
`awaiting units:` line for an open activation.
## Console service (`scripts/webui-service.sh`)
```bash
+7 -5
View File
@@ -33,7 +33,7 @@ Gaps found while working go to `docs/plans/DEFERRED.md`, not here.
<!-- mosaic-queue:begin -->
Generated from `docs/plans/queue.json` revision 2603 by `queue render`. Do not edit between the markers; change the queue with `scripts/mosaic queue`.
Generated from `docs/plans/queue.json` revision 2641 by `queue render`. Do not edit between the markers; change the queue with `scripts/mosaic queue`.
| # | Piece | Owner | Issues | State | After | Gate | Brief | Note |
|---|---|---|---|---|---|---|---|---|
@@ -246,7 +246,7 @@ Generated from `docs/plans/queue.json` revision 2603 by `queue render`. Do not e
| 207 | 193c: mosaic access user verbs and --level viewer (#1691) | morgana; reviewers duckworth | #1705 | queued | 202 done | Duckworth approves the manifest; cli suite and every scripts/test-*.sh green on Sage's gate; the mutants killed (sage) | `docs/plans/2026-10-20_human-users-build.md` § Brief: 193c, mosaic access user verbs and --level viewer | — |
| 208 | 193d: Console shared list, share controls and viewer read-only (#1691; also after FED R6 when filed) | morgana; reviewers duckworth | #1706 | queued | 192 done, 206 done | Duckworth approves the manifest; webui suite and every scripts/test-*.sh green on Sage's gate; the mutants killed (sage) | `docs/plans/2026-10-20_human-users-build.md` § Brief: 193d, the Console's shared list, share controls and viewer read-only | — |
| 209 | FED R8: the hub window on 10.1.1.54, a recorded acceptance (#414, r4.11 b93f098f0; live window run by Jason and Mos, after the Case 2 install #416, Bean and Chet live and both Hermes gateways stopped) | sage; reviewers duckworth | — | queued | — | Every entry check's output is in the R8 record with who ran it; Jason signs in to the hub and sends one message, Melanie revokes, his next request is refused; Duckworth checks the record (sage) | `docs/plans/2026-10-20_federation-r1-r2-r5.md` § Brief: FED R8, the hub window on 10.1.1.54 | Stays queued until its entry gates near: Case 2 install on .54 (#416, by 2026-10-27), Bean and Chet live, both Hermes gateways on .54 stopped, R7, #412 phase 2, #419 answer. Filed per mos-1073 after duckworth-0034 (Sage) |
| 210 | Agent host access with sudo through the credential store: grant per agent and host, the stack does ssh and sudo, the agent never sees the key (#1707, Vikunja #424, Q183/Q184, due 10-27; chained on row 199's candidate) | gizmoduck; reviewers duckworth | #1707 | in-review, round 2, request posted | — | Duckworth approves DESIGN.md, then the manifest; bus, business, cli suites and every scripts/test-*.sh green on Sage's gate; the mutants killed (sage) | `docs/plans/2026-10-20_host-credentials.md` § Brief: agent host access with sudo through the credential store (#424) | Sage gate 2026-10-21 on 92fac9904 (c8bc5cf4c + 3cf63efb4): mutants 52/53, M22b equivalent (no await broker.mjs:1741 to credentials.mjs:358; coverage moves to row 235, gizmoduck-0088). Gate STOP pkg-bus 497/1: B1 broker.mjs:7 imports business/src/hosts.mjs (not in RUNNER_FILES, nor errors.mjs/util.mjs); project-runtime.test.mjs:12 pin fails; seat sandbox.test.mjs 6/4 vs base 10/0. r4 needed (sage-1617). |
| 210 | Agent host access with sudo through the credential store: grant per agent and host, the stack does ssh and sudo, the agent never sees the key (#1707, Vikunja #424, Q183/Q184, due 10-27; chained on row 199's candidate) | gizmoduck; reviewers duckworth | #1707 | done | — | Duckworth approves DESIGN.md, then the manifest; bus, business, cli suites and every scripts/test-*.sh green on Sage's gate; the mutants killed (sage) | `docs/plans/2026-10-20_host-credentials.md` § Brief: agent host access with sudo through the credential store (#424) | Landed f5753c0e8 on refactor (base 9d165be8c), pushed. Patch-id 723b1c83 matches git diff ad95c888c..0485384e9 on the 23 manifest paths; landed files pass r5 manifest 688d38b9. Duckworth approve r2 rev 2592, r3 29851. Mutants 52/53, M22b equivalent (rev 2584). Gate on ad95c888c+patch: packages 2796/0, scripts 429/0; landed blobs equal the gate tree. |
| 211 | Force-push guard: scripts/git-hooks/pre-push refuses non-fast-forward and delete pushes from every worktree, and scripts/slot-busy.sh exits 1 on BUSY (#1709, Vikunja #426, mos-1078/1083) | rocko; reviewers dewey | #1709 | briefed | — | Dewey approves the manifest; the new or changed scripts/test-*.sh and every other scripts/test-*.sh green on Sage's gate; the mutants killed; hook installed in the canonical checkout with a scripted refused force-push (sage) | `docs/plans/2026-10-20_force-push-guard.md` § Brief: a pre-push hook that refuses force-pushes, and a busy check that fails on BUSY (#426) | — |
| 212 | Case 2 C2-01 packages/library: mapping, scope, reader, resolver, R-rules, renderItems, survey | gosalyn; reviewers webby | #1713 | in-review, round 1, request posted | — | Webby approves the manifest. The library tests and the cli suite are green on Sage's gate, along with every `scripts/test-*.sh`. (sage) | `docs/plans/2026-10-20_case2-rows.md` § C2-01 packages/library: mapping, scope, reader, resolver, R-rules, renderItems, survey | — |
| 213 | Case 2 C2-08 creative roles, persona file and block, M1, M2, M6, M10, prompt passed by file | morgana; reviewers webby | #1714 | in-review, round 2, request posted | — | Webby approves the manifest. The business, bus and harness suites, `scripts/test-config.sh` and every `scripts/test-*.sh` are green on Sage's gate, and the `models` mutant is killed. (sage) | `docs/plans/2026-10-20_case2-rows.md` § C2-08 creative roles, persona file and block, M1, M2, M6, M10, prompt passed by file | — |
@@ -268,15 +268,17 @@ Generated from `docs/plans/queue.json` revision 2603 by `queue render`. Do not e
| 229 | Row B: per-instance pi agent dir for Chet's Codex login, design only, starts after 10-27 (#428) | gosalyn; reviewers webby | #1730 | queued | — | A live check, with Jason's or Mos's go, that two clients on one ChatGPT login don't revoke each other's refresh tokens. No build row starts before it passes. Held until after 10-27. (sage) | `docs/plans/2026-10-20_case2-rows.md` § Brief: row B, a per-instance pi agent dir for Chet's Codex login (#428) | — |
| 230 | Case 2 C2-13a unit render and host precheck (case2-unit.sh, case2-precheck.sh) | gosalyn; reviewers webby | #1732 | done | — | Webby approves the manifest. The test file and the cli suite are green on Sage's gate, along with every scripts/test-*.sh. A run on a host is a separate window with Jason's go. (sage) | `docs/plans/2026-10-20_case2-rows.md` § C2-13a unit render and host precheck | Landed c8bc5cf4c on refactor (base 390ea5261), pushed. Patch-id f4b49eb68997 matches git diff 722136772..96768766c on the 5 manifest paths (manifest fbb0d7e42c67, Webby approve r2 rev 2563). packages/cli/README.md 3-way with row 199, hunks apart. Gosalyn gate on 96768766c: cli 541/0, scripts 429/0, mutants 10/10. Merged tree check: case2-unit 30/0, host 42/0. |
| 231 | Case 2 C2-01b survey escapes, list item lines and attachment embeds (row 212 N1-N3) | gosalyn; reviewers webby | #1733 | briefed | 212 done | Webby approves the manifest. The library tests are green on Sage's gate, along with every scripts/test-*.sh. (sage) | `docs/plans/2026-10-20_case2-rows.md` § C2-01b survey escapes, list item lines and attachment embeds | — |
| 232 | Installer I1: stack-release external units and stack-copy bundle copies (MVP, Q187, Vikunja #432) | darkwing; reviewers filbert | #1734 | in-review, round 1, request posted | — | Filbert approves the manifest. The cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) | `docs/plans/2026-10-20_installer-rows.md` § I1 stack-release external units and bundle copies | — |
| 233 | Installer I3: stack-upgrade.sh root driver and the ops-01 host run sheet (MVP, Q187, Vikunja #432) | darkwing; reviewers filbert | #1735 | briefed | 232 done | Filbert approves the manifest. The test file, the cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) | `docs/plans/2026-10-20_installer-rows.md` § I3 the upgrade command and the host run sheet | — |
| 234 | Machine grant row a: host enrollment, store records and migration 3i-3j (#436a, design r6 f9039682e, lands by 10-25 before the .53/.54 installs) | gizmoduck; reviewers duckworth | #1740 | queued | 210 done | Duckworth approves the manifest. The bus, business and cli suites and every scripts/test-*.sh are green on Sage's gate. The design's section 10 row a tests and mutants pass, the migration test included. (sage) (sage) | `docs/plans/2026-10-20_host-credentials.md` § Machine grant row a: enrollment, store records and migration 3i-3j | — |
| 232 | Installer I1: stack-release external units and stack-copy bundle copies (MVP, Q187, Vikunja #432) | darkwing; reviewers filbert | #1734 | done | — | Filbert approves the manifest. The cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) | `docs/plans/2026-10-20_installer-rows.md` § I1 stack-release external units and bundle copies | Landed e65a3ab4d on refactor (base 0444acd08), pushed. Patch-id 5adfbab9 matches git diff 8fede5fc6..da2da2f6c on the 9 manifest paths; tree entries (modes too) equal da2da2f6c, files pass manifest 7af3c943. Filbert approve r1 29829, r2 29865. Darkwing's landing gate on da2da2f6c: cli 593/0, scripts 429/0 (+156 queue node tests). Mutants 17/17. |
| 233 | Installer I3: stack-upgrade.sh root driver and the ops-01 host run sheet (MVP, Q187, Vikunja #432) | darkwing; reviewers filbert | #1735 | in-progress | 232 done | Filbert approves the manifest. The test file, the cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) | `docs/plans/2026-10-20_installer-rows.md` § I3 the upgrade command and the host run sheet | — |
| 234 | Machine grant row a: host enrollment, store records and migration 3i-3j (#436a, design r6 f9039682e, lands by 10-25 before the .53/.54 installs) | gizmoduck; reviewers duckworth | #1740 | in-progress | 210 done | Duckworth approves the manifest. The bus, business and cli suites and every scripts/test-*.sh are green on Sage's gate. The design's section 10 row a tests and mutants pass, the migration test included. (sage) (sage) | `docs/plans/2026-10-20_host-credentials.md` § Machine grant row a: enrollment, store records and migration 3i-3j | — |
| 235 | Machine grant row b: raise binding, executor, reconcile and revoke (#436, design r6 f9039682e, C1, N1, G27, G28) | gizmoduck; reviewers duckworth | #1741 | queued | 234 done | Duckworth approves the manifest. The bus, business and cli suites and every scripts/test-*.sh are green on Sage's gate. Section 10's row b tests and mutants pass, G27, G28 and the upsert mutant included. (sage) (sage) | `docs/plans/2026-10-20_host-credentials.md` § Machine grant row b: raise binding, executor, reconcile and revoke | — |
| 236 | #436b Gitea grant path build: seat accounts and tokens through an approved grant, schema 3j-3k (design r5 1be535aca, due 10-27 at risk; live waits for Jason's section 8 approval and bootstrap step) | megavolt; reviewers dewey | #1742 | queued | 210 done, 234 done | Dewey approves the manifest. The bus, business and cli suites and every scripts/test-*.sh are green on Sage's gate. Section 9's mutants are killed, N9's restart test included. (sage) (sage) | `docs/plans/2026-10-20_agent-credentials.md` § Brief: #436b Gitea grant path, build (schema 3j-3k) | — |
| 237 | Row 199 follow-up N1: one decision publishes once (publish-busy or #spent recheck) | megavolt; reviewers duckworth | #1743 | done | 199 done | Duckworth approves the manifest. The bus suite and every scripts/test-*.sh are green on Sage's gate, and the mutant is killed. (sage) (sage) | `docs/plans/2026-10-20_agent-credentials.md` § Brief: row 199 follow-up N1, one decision publishes once | Landed d7d943199 on refactor (local, push pending), patch-id c5f4beb4 = e42415935, manifest bb777cb0 blobs checked. Sage gate 01:31-01:44Z on 816d48b03+4 paths: M1 killed (bus 1/13, cli 1/8), every package suite and scripts/test-*.sh green (bus 478, cli 564). |
| 238 | row 200 follow-up N1: the sso-config hint carries --instance NAME (#1744) | launchpad; reviewers webby | #1744 | in-review, round 1, request posted | 200 done | Webby approves the manifest; stack-release.test.mjs whole, cli suite and every scripts/test-*.sh green on Sage's gate; the mutant killed (sage) | `docs/plans/2026-10-20_stack-release-sso.md` § Brief: row 200 follow-up N1, the sso-config hint names the instance | — |
| 239 | Row 194 follow-up: the credential revoke runs under the writer's lock, refusals after a revoke say so, and a stale-revision fire test (duckworth-0074, mos-1158) | honker; reviewers duckworth | #1745 | queued | 194 done | Duckworth approves the manifest. The bus, business, cli and webui suites and every scripts/test-*.sh are green on Sage's gate, and the mutants are killed. (sage) (sage) | `docs/plans/2026-10-20_agent-lifecycle.md` § Brief: row 194 follow-up, the revoke under the writer's lock and a fire test | — |
| 240 | Installer I4: hostcheck, the bwrap profile check, pins and boot unit, from r6 B2 (MVP, Q187, Vikunja #432; split from I3) | darkwing; reviewers filbert | #1746 | queued | 233 done | Filbert approves the manifest. The test file, the cli suite and every scripts/test-*.sh are green on Sage's gate, with the mutants killed. (sage) (sage) | `docs/plans/2026-10-20_installer-rows.md` § I4 hostcheck: the bwrap profile check, pins and boot unit | — |
| 241 | Row 237 follow-up: one decision lands once, a two-ticket broker land test and three mutants on the landing guards (megavolt-0069, sage-1651, P19 441) | megavolt; reviewers duckworth | #1747 | done | 237 done | Duckworth approves the manifest. The bus, seat and cli suites and every scripts/test-*.sh are green on Sage's gate, and the three mutants are killed. (sage) (sage) | `docs/plans/2026-10-20_agent-credentials.md` § Brief: row 237 follow-up, one decision lands once, with tests at the broker | Landed 5d4a61776 on refactor, pushed (base e65a3ab4d). Blobs from 3e3ed303e on 2 paths; patch-id a1649f80 equals git diff f5753c0e8..3e3ed303e on those paths; ls-tree modes equal; manifest e8266864 verifies on the landed files. Sage slot on bbf3b1ee8: M1-M3 killed; gate 15 packages 2797/0, 10 script suites 429/0. Duckworth approve 29870 rev 2628. |
| 242 | Row 237 follow-up N1: a retry after publish-timeout pushes once; hold the decision or kill and wait for the push, a slow-push retry test and a mutant (duckworth-0069, P19 440) | megavolt; reviewers duckworth | #1748 | done | 237 done | Duckworth approves the manifest. The bus and cli suites and every scripts/test-*.sh are green on Sage's gate, and the mutant is killed. (sage) (sage) | `docs/plans/2026-10-20_agent-credentials.md` § Brief: row 237 follow-up N1, a retry after publish-timeout pushes once | Landed 5fb736b92 on refactor, pushed (base 96bac588f). Blobs from 635c10924 on 3 paths; patch-id ceb741ee equals git diff 8fede5fc6..635c10924 on those paths; ls-tree modes equal; manifest 1fbfab6c verifies on the landed files. Megavolt M1/M2 killed on Sage's slot; Sage gate 15 packages 2826/0, 10 script suites 429/0. Duckworth approve 29889 rev 2639. |
<!-- mosaic-queue:end -->
+1262 -26
View File
File diff suppressed because it is too large Load Diff
+35 -2
View File
@@ -257,9 +257,42 @@ export async function startBroker({
if (!(e instanceof BusError)) process.stderr.write(`bus: publish git-failed: ${JSON.stringify({ name: e?.name, code: e?.code })}\n`);
Object.assign(entry, { state: 'refused', code: e instanceof BusError ? e.code : 'git-failed' });
},
);
).then(() => {
if (entry.held) settle(ticket, entry, entry.held);
});
return { state: 'running' };
}
// Row 242 (#1748): the git socket gave up on a publish (publish-timeout) whose push it never
// collected. Its publishRefused waits for the push: until that ends the ticket stays open, so
// checkPublish refuses another publish on the decision with publish-busy (row 237). A push that
// reached the fork is then recorded, which spends the decision; one that didn't is filed with
// the socket's code. Either way the decision can't push a second ref.
function settle(ticket, entry, code) {
remotes.delete(ticket);
try {
if (entry.state === 'pushed') {
try {
broker.recordPublish(ticket, { ref: entry.ref, commit: entry.commit, previous: entry.previous });
process.stderr.write(`bus: publish ${ticket}: pushed after the git socket refused ${code}; recorded\n`);
return;
} catch (e) {
if (!(e instanceof BusError)) throw e;
code = e.code;
}
}
broker.publishRefused(ticket, { code });
} catch (e) {
process.stderr.write(`bus: publish ${ticket}: not settled: ${JSON.stringify({ name: e?.name, code: e?.code })}\n`);
}
}
function publishRefused(ticket, args) {
const entry = remotes.get(ticket);
if (!entry) return broker.publishRefused(ticket, args);
if (!/^[a-z-]{1,64}$/.test(args?.code ?? '')) throw new BusError('invalid-request');
if (entry.state === 'running') entry.held = args.code;
else settle(ticket, entry, args.code);
return args.code;
}
function publishResult(ticket) {
const entry = remotes.get(ticket);
if (!entry) throw new BusError('publish-unknown');
@@ -296,7 +329,7 @@ export async function startBroker({
publishPush,
publishResult,
recordPublish: (ticket, pushed) => broker.recordPublish(ticket, pushed),
publishRefused: (ticket, args) => broker.publishRefused(ticket, args),
publishRefused,
landHeld: (record) => broker.landHeld(record),
// Row 156 (#1651): the launcher's landing pass.
landHolders: (business) => broker.landHolders(business),
+22
View File
@@ -306,6 +306,28 @@ test('a ticket outlives the run that asked, and recordLand checks the task again
assert.throws(() => w.b.landRefused(ticket, { code: 'gate-failed' }), /land-unknown/);
});
// Row 241 (#1747): one decision lands once. Two tickets on one decision can't both reach
// recordLand through the git socket (land-busy, git-socket.mjs), so this pins the broker's own
// guard: recordLand runs #landFacts again, and a decision spent by the first refuses the second.
test('two tickets on one decision: the first recordLand lands, the second refuses decision-consumed and records nothing', (t) => {
const w = setup(t);
const { d } = w.ready();
const first = w.check({ task_ref: TASK, decision: d.id }).ticket;
const second = w.check({ task_ref: TASK, decision: d.id }).ticket;
assert.notEqual(first, second);
assert.deepEqual(w.b.recordLand(first, w.land), { task_ref: TASK, commit: B });
assert.throws(() => w.b.recordLand(second, { ...w.land, commit: C, previous: B }), /decision-consumed/);
assert.equal(w.merges().length, 1);
assert.equal(w.merges()[0].body.commit, B);
assert.equal(w.results().length, 1);
assert.deepEqual(w.events('action.refused'), []);
// The git socket then ends the second ticket with the broker's code, as it does any record refusal.
assert.equal(w.b.landRefused(second, { code: 'decision-consumed', notify: true }), 'decision-consumed');
assert.deepEqual(w.events('action.refused').map((e) => e.body), [{ code: 'decision-consumed' }]);
assert.equal(w.merges().length, 1);
assert.throws(() => w.b.recordLand(second, w.land), /land-unknown/);
});
test('landRefused files the code on the task, tells the role only with notify, and ends the ticket', (t) => {
const w = setup(t);
const { d } = w.ready();
+10
View File
@@ -774,6 +774,16 @@ Each live run requires Mos's go; agents build and test fixtures only, and
so no PM turn may be in flight. Queue writes stay on the canonical
scripts/mosaic.
On a host whose units are root's (row 232), `--units external` makes
stack-release.sh read the system units and start or stop none. `activate`
and `rollback --to` stage the copy and leave the `AWAITING-UNITS` marker;
root installs and starts the units, then `confirm` gates the start and
writes the pointer, `revert` undoes a staged run, and `gate --since` checks
a copy with no open activation. stack-copy.sh adds `copy --bundle FILE SHA`
for a host with no refactor checkout, `wrapper ID --create` for the first
wrapper, and `check --units external`. Details are in
[TOOLS.md, Root-owned units](../../docs/TOOLS.md#root-owned-units---units-external-row-232).
The human commands and `--tester` run only from the tree the bus runs from
(row 130, #1625). The broker checks the path of the calling human-cli.mjs or
tester-cli.mjs against its own. On a host with a pinned copy that tree is
+7 -3
View File
@@ -38,7 +38,10 @@
// one op at a time within 10 s. The poll ends at publishDeadlineMs
// (publish-timeout), past git's own limit on each of the four git
// runs. recordPublish files the push; any refusal after checkPublish
// is filed once, by publishRefused.
// is filed once, by publishRefused. After publish-timeout the bus
// holds the decision until its push ends (a retry gets publish-busy),
// then records a push that reached the fork or files the refusal
// (row 242, #1748).
//
// A refusal is {"ok": false, "error": "<code>"}, and the connection ends.
// The host checks, in order: identity (a role of this business), the request
@@ -427,8 +430,9 @@ export function createGitSocket({ host, business: b, dataRoot, env = process.env
await op({ op: "publishPush", ticket });
const until = Date.now() + publishDeadlineMs;
for (;;) {
// The bus's git may still finish after this; the fork could then hold the commit with
// no record, as when recordPublish refuses after a push.
// The bus's git may still finish after this. The bus holds the decision until it does,
// then records a push that reached the fork (row 242); the fork can still hold the
// commit with no record if recordPublish refuses, as after an in-band push.
if (Date.now() >= until) {
log(`git socket: publish ${task_ref}: publish-timeout; the bus process's push may still reach the fork`);
throw new Refusal("publish-timeout");
+39 -1
View File
@@ -5,7 +5,7 @@
import { test } from "node:test";
import assert from "node:assert/strict";
import { spawnSync } from "node:child_process";
import { chmodSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { chmodSync, existsSync, lstatSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from "node:fs";
import { connect } from "node:net";
import { tmpdir } from "node:os";
import { join } from "node:path";
@@ -298,3 +298,41 @@ test("sage-1440: the publishResult poll is bounded: publish-timeout, filed once,
assert.equal(w.events("action.allowed").filter((e) => e.body.action === "git.push.remote" && e.body.operation !== "decision.raise").length, 0);
assert.ok(w.logs.some((l) => l.includes("publish-timeout")));
});
test("row 242: a retry after publish-timeout refuses publish-busy while the bus process's push runs; the late push is recorded and the decision pushes once", async (t) => {
const w = await setup(t, { publishDeadlineMs: 150 });
w.bus.credentials.issue("demo/coder", "gitea", TOKEN, "2099-01-01");
// The fork holds the push in pre-receive until the test writes `release`; `held` says it's there.
const held = join(w.root, "held");
const release = join(w.root, "release");
writeFileSync(join(w.forkRepo, "hooks", "pre-receive"), `#!/bin/sh\n: > '${held}'\nwhile [ ! -e '${release}' ]; do sleep 0.05; done\ncat > /dev/null\n`, { mode: 0o755 });
const until = async (what, ok) => {
for (const end = Date.now() + 10_000; !ok(); ) {
if (Date.now() > end) assert.fail(`timed out waiting for ${what}`);
await new Promise((r) => setTimeout(r, 20));
}
};
const allowed = () => w.events("action.allowed").filter((e) => e.body.action === "git.push.remote" && e.body.operation !== "decision.raise");
// A failed assert must still let the hook end: teardown waits for the fake Gitea, which waits for the push.
try {
const d = w.approve();
assert.deepEqual(await w.publish(d), { ok: false, error: "publish-timeout" });
await until("the push to reach pre-receive", () => existsSync(held));
// The socket gave up, but the push still runs: the decision is held, and nothing is filed yet.
assert.deepEqual(await w.publish(d, { branch: "other" }), { ok: false, error: "publish-busy" });
assert.deepEqual(w.events("action.refused").map((e) => e.body.code), ["publish-busy"]);
assert.equal(allowed().length, 0);
writeFileSync(release, "");
// The push reached the fork, so it's recorded, which spends the decision.
await until("the late push to be recorded", () => allowed().length === 1);
assert.equal(allowed()[0].body.ref, "refs/heads/coder/feat");
assert.equal(allowed()[0].body.commit, w.work);
assert.deepEqual(await w.publish(d, { branch: "other" }), { ok: false, error: "decision-consumed" });
assert.equal(w.gitea.requests.filter((r) => r.path.endsWith("/git-receive-pack")).length, 1);
assert.equal(git(w.root, `--git-dir=${w.forkRepo}`, "for-each-ref", "--format=%(refname)", "refs/heads/coder/"), "refs/heads/coder/feat");
assert.deepEqual(w.events("action.refused").map((e) => e.body.code), ["publish-busy", "decision-consumed"]);
} finally {
writeFileSync(release, "");
}
assert.equal(JSON.stringify([w.events("action.allowed"), w.events("action.refused"), w.logs]).includes(TOKEN), false);
});
+107 -1
View File
@@ -1,7 +1,8 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import {execFileSync,spawnSync,spawn} from 'node:child_process';
import {symlinkSync,unlinkSync,mkdtempSync,mkdirSync,writeFileSync,readFileSync,cpSync,rmSync,chmodSync,existsSync,statSync} from 'node:fs';
import {symlinkSync,unlinkSync,mkdtempSync,mkdirSync,writeFileSync,readFileSync,cpSync,rmSync,chmodSync,existsSync,statSync,readdirSync,lstatSync,readlinkSync} from 'node:fs';
import {createHash} from 'node:crypto';
import {tmpdir} from 'node:os';
import {join,dirname} from 'node:path';
import {fileURLToPath} from 'node:url';
@@ -227,3 +228,108 @@ test('inherited drop-in unreadable directory or file refuses before writing',t=>
unlinkSync(dropdir);mkdirSync(dropdir);const file=join(dropdir,'broken.conf');symlinkSync(join(s.root,'absent-file'),file);
const broken=s.run('bus-service.sh',['install','--dir',units,'--no-reload']);assert.equal(broken.status,1);assert.ok(broken.stderr.includes(file));assert.equal(readFileSync(target,'utf8'),'previous unit bytes');
});
// Row 232: every file of a copy with its mode and bytes, for the byte-identical comparison.
function tree(dir){
const out=[];
const walk=rel=>{for(const e of readdirSync(join(dir,rel),{withFileTypes:true}).sort((a,b)=>a.name<b.name?-1:1)){
const r=rel?`${rel}/${e.name}`:e.name,st=lstatSync(join(dir,r));
if(e.isDirectory()){out.push(`d ${(st.mode&0o7777).toString(8)} ${r}`);walk(r);}
else if(e.isSymbolicLink())out.push(`l ${r} ${readlinkSync(join(dir,r))}`);
else out.push(`f ${(st.mode&0o7777).toString(8)} ${r} ${createHash('sha256').update(readFileSync(join(dir,r))).digest('hex')}`);
}};
walk('');return out;
}
test('row 232: copy --bundle copies a SHA the bundle\'s refactor reaches, byte-identical to copy SHA, and refuses the rest',t=>{
const s=setup(t),tmp=join(s.root,'t');mkdirSync(tmp);s.env.TMPDIR=tmp;
const good=join(s.root,'good.bundle');git(s.repo,'bundle','create','-q',good,'refactor');
const a=join(s.root,'copies-a'),b=join(s.root,'copies-b');
let r=s.run('stack-copy.sh',['copy',s.id,'--copies',a]);assert.equal(r.status,0,r.stderr);
r=s.run('stack-copy.sh',['copy','--bundle',good,s.id,'--copies',b]);assert.equal(r.status,0,r.stderr);
const sum=createHash('sha256').update(readFileSync(good)).digest('hex'),release=git(s.repo,'show',`${s.id}:RELEASE`);
assert.equal(r.stdout,`copied: ${join(b,s.id)}\nrelease: ${release}\nbundle sha256: ${sum}\n`);
assert.deepEqual(tree(join(b,s.id)),tree(join(a,s.id)));
assert.equal(statSync(join(b,s.id)).mode&0o777,0o555);
const left=()=>readdirSync(tmp).filter(f=>f.startsWith('tmp.'));
assert.deepEqual(left(),[],'the bundle\'s temporary repository is removed');
// A second run finds the copy and still names the release.
r=s.run('stack-copy.sh',['copy','--bundle',good,s.id,'--copies',b]);assert.equal(r.status,0,r.stderr);
assert.equal(r.stdout,`exists: ${join(b,s.id)}\nrelease: ${release}\nbundle sha256: ${sum}\n`);
// A relative bundle path works from the caller's directory.
const c=join(s.root,'copies-c');
r=spawnSync(join(s.repo,'scripts','stack-copy.sh'),['copy','--bundle','good.bundle',s.id,'--copies',c],{cwd:s.root,env:s.env,encoding:'utf8'});
assert.equal(r.status,0,r.stderr);assert.deepEqual(tree(join(c,s.id)),tree(join(a,s.id)));
// A commit on another branch of the bundle: the bundle carries it, refactor does not reach it.
const side=git(s.repo,'commit-tree','-p',s.id,'-m','side',`${s.id}^{tree}`);git(s.repo,'update-ref','refs/heads/side',side);
const both=join(s.root,'both.bundle');git(s.repo,'bundle','create','-q',both,'refactor','side');
const only=join(s.root,'side.bundle');git(s.repo,'bundle','create','-q',only,'side');
const corrupt=join(s.root,'corrupt.bundle'),bytes=readFileSync(good);bytes[bytes.length-30]^=0xff;writeFileSync(corrupt,bytes);
const cut=join(s.root,'cut.bundle');writeFileSync(cut,readFileSync(good).subarray(0,200));
const junk=join(s.root,'junk.bundle');writeFileSync(junk,'# v2 git bundle\nnot a ref line\n\n');
const d=join(s.root,'copies-d');
for(const [label,bundle,id,re] of [
['not reachable',both,side,/not reachable from the bundle's refs\/heads\/refactor/],
['no refactor',only,side,/no refs\/heads\/refactor/],
// verify reads the header and prerequisites; a damaged pack fails the fetch's own check.
['not a bundle',junk,s.id,/git bundle verify failed/],
['corrupt pack',corrupt,s.id,/cannot fetch/],
['truncated pack',cut,s.id,/cannot fetch/],
['absent',join(s.root,'none.bundle'),s.id,/bundle is absent/],
['short id',good,s.id.slice(0,12),/full 40-character commit id/],
]){
r=s.run('stack-copy.sh',['copy','--bundle',bundle,id,'--copies',d]);
assert.equal(r.status,1,label);assert.match(r.stderr,re,label);assert.equal(r.stdout,'',label);
assert.ok(!existsSync(join(d,id)),label);assert.deepEqual(left(),[],label);
if(existsSync(d))assert.deepEqual(readdirSync(d),[],label);
}
const link=join(s.root,'link.bundle');symlinkSync(good,link);
r=s.run('stack-copy.sh',['copy','--bundle',link,s.id,'--copies',d]);assert.equal(r.status,1);assert.match(r.stderr,/symlink/);
for(const args of [['copy','--bundle',good,s.id,'--allow-worktree'],['copy','--bundle',good],['copy','--bundle'],['wrapper','--bundle',good,s.id],['check','--units','system'],['copy',s.id,'--units','external'],['copy',s.id,'--create']])
assert.equal(s.run('stack-copy.sh',args).status,4,args.join(' '));
});
test('row 232: wrapper --create writes the first wrapper only, the same bytes wrapper ID writes, and refuses over any file',t=>{
const s=setup(t);assert.equal(s.run('stack-copy.sh',['copy',s.id,'--copies',s.copies]).status,0);
const wrapper=join(s.bin,'mosaic'),root=join(s.copies,s.id);
let r=s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin]);
assert.equal(r.status,1);assert.match(r.stderr,/wrapper is absent/,'no --create keeps the missing-wrapper refusal');assert.ok(!existsSync(wrapper));
r=s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin,'--create']);assert.equal(r.status,0,r.stderr);
assert.equal(r.stdout,`wrapper: ${root}\n`);
const created=readFileSync(wrapper,'utf8');assert.equal(statSync(wrapper).mode&0o777,0o755);
r=s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin]);assert.equal(r.status,0,r.stderr);
assert.equal(r.stdout,`wrapper: ${root}\n`);assert.equal(readFileSync(wrapper,'utf8'),created,'wrapper ID writes the same bytes');
const foreign='#!/bin/sh\n# not ours\n';
for(const [label,make] of [['own wrapper',()=>{}],['foreign file',()=>writeFileSync(wrapper,foreign)],['symlink',()=>{rmSync(wrapper);symlinkSync(join(s.root,'elsewhere'),wrapper);}]]){
make();const before=existsSync(wrapper)||label==='symlink'?(label==='symlink'?readlinkSync(wrapper):readFileSync(wrapper,'utf8')):null;
r=s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin,'--create']);
assert.equal(r.status,1,label);assert.match(r.stderr,/--create and a wrapper exists/,label);
assert.equal(label==='symlink'?readlinkSync(wrapper):readFileSync(wrapper,'utf8'),before,label);
assert.deepEqual(readdirSync(s.bin).filter(f=>f.startsWith('.')),[],`${label}: no temporary file left`);
}
// A named instance: --create writes mosaic-NAME as wrapper --instance does, once.
assert.equal(s.run('bootstrap.sh',['--instance','alpha']).status,0);
const named=join(s.bin,'mosaic-alpha');
r=s.run('stack-copy.sh',['wrapper',s.id,'--instance','alpha','--copies',s.copies,'--bin',s.bin,'--create']);assert.equal(r.status,0,r.stderr);
const first=readFileSync(named,'utf8');
r=s.run('stack-copy.sh',['wrapper',s.id,'--instance','alpha','--copies',s.copies,'--bin',s.bin]);assert.equal(r.status,0,r.stderr);
assert.equal(readFileSync(named,'utf8'),first);
r=s.run('stack-copy.sh',['wrapper',s.id,'--instance','alpha','--copies',s.copies,'--bin',s.bin,'--create']);
assert.equal(r.status,1);assert.match(r.stderr,/--create and a wrapper exists/);assert.equal(readFileSync(named,'utf8'),first);
});
test('row 232: check --units external reads the system units, the default reads the user units',t=>{
const s=setup(t);assert.equal(s.run('stack-copy.sh',['copy',s.id,'--copies',s.copies]).status,0);
assert.equal(s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin,'--create']).status,0);
const copy=join(s.copies,s.id),argv=join(s.root,'argv.log');
s.fake(`echo "$*" >> '${argv}'; echo 'LoadState=loaded'; echo 'WorkingDirectory=${copy}'; case "$*" in *mosaic-bus*) echo 'ExecStart={ path=${copy}/scripts/mosaic ; argv[]=${copy}/scripts/mosaic bus start mosaic-stack --pm ; }';; *) echo 'ExecStart={ path=/usr/bin/node ; argv[]=/usr/bin/node ${copy}/packages/webui/src/cli.mjs serve ; }';; esac\n`);
const want=`bus: ${copy}\nConsole: ${copy}\nCLI: ${copy}\n`;
let r=s.run('stack-copy.sh',['check','--copies',s.copies,'--bin',s.bin,'--units','external']);assert.equal(r.status,0,r.stderr);assert.equal(r.stdout,want);
assert.deepEqual(readFileSync(argv,'utf8').trimEnd().split('\n'),[
'show [email protected] -p LoadState -p WorkingDirectory -p ExecStart',
'show mosaic-webui.service -p LoadState -p WorkingDirectory -p ExecStart']);
rmSync(argv);
for(const args of [[],['--units','user']]){
r=s.run('stack-copy.sh',['check','--copies',s.copies,'--bin',s.bin,...args]);assert.equal(r.status,0,r.stderr);assert.equal(r.stdout,want);
assert.deepEqual(readFileSync(argv,'utf8').trimEnd().split('\n'),[
'--user show [email protected] -p LoadState -p WorkingDirectory -p ExecStart',
'--user show mosaic-webui.service -p LoadState -p WorkingDirectory -p ExecStart']);
rmSync(argv);
}
});
@@ -0,0 +1,476 @@
// scripts/stack-release.sh --units external (row 232, #1734). Design: r6 section 4 and the section 7
// test list, agents/darkwing/work/queue-195/DESIGN.md; deviations: agents/darkwing/work/queue-232/BUILD.md.
// The stand-ins and the module fixture are stack-release.test.mjs's (stack-release-fixture.mjs).
// Root's part (stop, unit install, start) runs here as the stand-in systemctl and bus-service.sh and
// webui-service.sh installs; no sudo and no real unit.
import test from 'node:test';
import assert from 'node:assert/strict';
import {execFileSync} from 'node:child_process';
import {mkdirSync,writeFileSync,readFileSync,existsSync,readdirSync,rmSync,chmodSync,cpSync} from 'node:fs';
import {DatabaseSync} from 'node:sqlite';
import {join} from 'node:path';
import {B,C,C1,COPIES,ROOT,SBIN,base,fault,lines,logLines,mk,pointer,repo,run,seed,sha,storeVersion,sysctl,tree,units} from './stack-release-fixture.mjs';
// The commit row 232 builds on: the user-mode tests compare the candidate's calls and --plan bytes
// with its scripts. A later row that changes user mode on purpose moves this.
const BASE='cb5fdb064fb4f12edd045374dd463dfdc25c2b42';
const X=(e)=>({...e,flags:[...e.flags,'--units','external']});
const sh=(e,name,...a)=>execFileSync('bash',[join(repo,'scripts',name),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
// Root's steps: R1/A2 stop, A4 install and start. offset is the start's SD_CLOCK_OFFSET.
const down=(e)=>{sysctl(e,'stop',e.con);sysctl(e,'stop',e.bus)};
const up=(e,copy,{pm=false,offset=0}={})=>{
sh(e,'bus-service.sh','install','--copy',copy,'--copies',COPIES,...(pm?['--pm']:[]),'--claude',e.claude);
sh(e,'webui-service.sh','install','--copy',copy,'--copies',COPIES,'--port',e.port,'--business','acme');
execFileSync(join(SBIN,'systemctl'),['--user','start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:String(offset)}});
sysctl(e,'start',e.con);
};
// The stack on B for acme, adopted under --units external, then stopped by root unless live.
async function xseed(t,{pm=false,live=false}={}){
const e=await mk(t);await seed(e,{pm,pointer:'none'});const x=X(e);
const r=await run(x,'adopt','acme');assert.equal(r.code,0,r.err);
if(!live)down(e);
return {e,x};
}
const lastAct=(e)=>logLines(e).filter((o)=>o.result==='started').at(-1).activation;
const rs=(e,a)=>join(e.state,'stack-rollback',a);
const open=(e)=>{const m=new Set();for(const o of logLines(e)){if(!o.activation)continue;if(o.result==='started')m.add(o.activation);else m.delete(o.activation)}return [...m]};
const mine=(e,a)=>logLines(e).filter((o)=>o.activation===a);
const journalCalls=(e)=>lines(join(e.sd,'journal-since.log'));
// A3: stop, stage TO. Returns the activation and the run.
async function stage(e,x,to,...args){
const r=await run(x,'activate',to,'acme',...args);
return {r,a:r.code===0||r.code===1?lastAct(e):null};
}
// The started line's from, to and op on the activation's result line, and its backup the set's.
function sameAsStarted(e,a){
const [s,f,...more]=mine(e,a);assert.equal(more.length,0,JSON.stringify(mine(e,a)));
assert.equal(s.result,'started');
for(const k of ['op','from','to','activation'])assert.equal(f[k],s[k],k);
assert.ok(['activate','rollback'].includes(f.op),f.op);
assert.equal(f.units,'external');
const b=join(rs(e,a),'BACKUP');assert.equal(f.backup,existsSync(b)?readFileSync(b,'utf8').trim():null);
return f;
}
// B1's end state: one started and one result line, nothing open, no marker or FAILED, and a
// following --plan that doesn't refuse interrupted-run.
async function closed(e,x,a){
sameAsStarted(e,a);
assert.deepEqual(open(e),[]);
for(const f of ['AWAITING-UNITS','FAILED'])assert.ok(!existsSync(join(rs(e,a),f)),f);
const s=await run(x,'status','acme');assert.equal(s.code,0,s.err);
assert.ok(!/interrupted run|awaiting units/.test(s.out),s.out);
const p=await run(x,'activate',pointer(e).copy===C1?B:C1,'acme','--plan');
assert.ok(!(p.out+p.err).includes('interrupted-run'),p.out+p.err);
}
const storeSha=(e)=>sha(e.store);
const backupStore=(e,a)=>join(e.data,'backups','stack',a,'bus','bus.sqlite');
test('row 232: confirm, revert, gate, --to and --expect-from need --units external; the user-only flags are usage under it',async(t)=>{
const {e,x}=await xseed(t);
const logs=logLines(e).length;
for(const a of [['confirm',C1,'acme'],['revert',C1,'acme'],['gate',B,'acme','--since','1'],['rollback','acme','--to',B],['activate',C1,'acme','--plan','--expect-from',B]]){
const r=await run(e,...a);assert.equal(r.code,4,a.join(' '));assert.match(r.err,/^usage: stack-release.sh adopt BUSINESS\n/);
}
for(const a of [['activate',C1,'acme','--pm'],['activate',C1,'acme','--tester-port','7444'],['activate',C1,'acme','--no-proof'],['activate',C1,'acme','--no-send'],['activate',C1,'acme','--sso'],['activate',C1,'acme','--proof-header','X','--proof-file','/p'],['rollback','acme'],['gate',B,'acme'],['activate',C1,'acme','--expect-from',B],['confirm',C1,'acme','--rollback',B],['confirm',C1,'acme','--cause','A5']]){
const r=await run(x,...a);assert.equal(r.code,4,a.join(' '));
}
assert.equal(logLines(e).length,logs);
});
test('row 232: pm-on and activate --pm-after-harness refuse external-pm',async(t)=>{
const {e,x}=await xseed(t);
const p=await run(x,'pm-on','acme');assert.equal(p.code,3);
assert.match(p.err,/^stack-release: refusing external-pm: under --units external the PM is a flag in root's bus unit/);
const a=await run(x,'activate',C1,'acme','--pm-after-harness');assert.equal(a.code,3);
assert.match(a.err,/^stack-release: refusing external-pm: /);
const l=logLines(e).slice(-2);assert.deepEqual(l.map((o)=>[o.result,o.code,o.activation]),[['refused','external-pm',null],['refused','external-pm',null]]);
assert.equal(readdirSync(e.state).includes('stack-rollback'),false);
});
test('row 232: units-running (N4) refuses while a unit is active and leaves the store, the log and TAKEN; failed counts as down',async(t)=>{
const {e,x}=await xseed(t,{live:true});
sysctl(e,'stop',e.con);
const before=storeSha(e),logs=logLines(e).length;
const r=await run(x,'activate',C,'acme','--migrate');assert.equal(r.code,3,r.out+r.err);
assert.equal(r.err,`stack-release: refusing units-running: ${e.bus} is active; root stops the units first\n`);
assert.equal(storeSha(e),before);assert.equal(storeVersion(e.store,true),'3g');
const l=logLines(e).slice(logs);assert.deepEqual(l.map((o)=>[o.result,o.code,o.activation,o.units]),[['refused','units-running',null,'external']]);
assert.ok(!existsSync(join(e.data,'backups')));assert.ok(!existsSync(join(e.state,'stack-rollback')));
// The Console active and the bus stopped refuses too.
sysctl(e,'stop',e.bus);sysctl(e,'start',e.con);
const c=await run(x,'activate',C1,'acme');assert.equal(c.code,3);assert.ok(c.err.startsWith(`stack-release: refusing units-running: ${e.con} is active`),c.err);
// A bus unit root's stop left failed is down.
sysctl(e,'stop',e.con);
const s=units(e);s[e.bus].state='failed';writeFileSync(join(e.sd,'units.json'),JSON.stringify(s));
const f=await run(x,'activate',C1,'acme');assert.equal(f.code,0,f.out+f.err);
});
test('row 232: activate stages: the pointer stays, AWAITING-UNITS names TO, a backup of the stopped store, no directory in the set, no --user, no journal',async(t)=>{
const {e,x}=await xseed(t);
const p0=readFileSync(e.pointer,'utf8'),w0=sha(join(e.bin,'mosaic')),argv0=lines(join(e.sd,'argv.log')).length;
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
assert.equal(r.out.split('\n').at(-2),`stack-release: activate staged ${C1} for acme as ${a}; root installs and starts the units, then runs confirm ${C1} acme --units external`);
assert.equal(readFileSync(e.pointer,'utf8'),p0);
assert.equal(readFileSync(join(rs(e,a),'AWAITING-UNITS'),'utf8'),`${C1}\n`);
assert.deepEqual(readdirSync(rs(e,a)).sort(),['AWAITING-UNITS','BACKUP','SHA256SUMS','UNITS.sha256','WRAPPER_MODE','pointer',`show-${e.bus}.txt`,`show-${e.con}.txt`,'steps.log','wrapper'].sort());
assert.equal(readFileSync(join(rs(e,a),'BACKUP'),'utf8'),`${a}\n`);
assert.equal(sha(backupStore(e,a)),storeSha(e));assert.equal(readFileSync(join(e.data,'backups','stack',a,'SCHEMA'),'utf8'),'3g\n');
assert.equal(readFileSync(join(rs(e,a),'UNITS.sha256'),'utf8'),`${sha(join(e.units,e.tmpl))} ${join(e.units,e.tmpl)}\n${sha(join(e.units,e.con))} ${join(e.units,e.con)}\n`);
assert.equal(sha(join(rs(e,a),'wrapper')),w0);assert.notEqual(sha(join(e.bin,'mosaic')),w0);
assert.deepEqual(mine(e,a).map((o)=>o.result),['started']);
assert.deepEqual([units(e)[e.bus].state,units(e)[e.con].state],['inactive','inactive']);
const ext=lines(join(e.sd,'argv.log')).slice(argv0);
assert.ok(ext.length>0);for(const c of ext)assert.match(c,/^(show|is-active|list-units) /,c);
assert.deepEqual(journalCalls(e),[]);
const s=await run(x,'status','acme');assert.equal(s.code,0,s.err);
assert.ok(s.out.includes(`awaiting units: ${a} (copy ${C1})\n`),s.out);assert.ok(s.out.includes(`interrupted run: ${a} `),s.out);
});
test('row 232: A3 order: capture-failed and marker-failed refuse with exit 3 and leave no marker or FAILED',async(t)=>{
const {e,x}=await xseed(t);
fault(e,'no-fragment');
const r=await run(x,'activate',C1,'acme');assert.equal(r.code,3,r.out+r.err);
assert.match(r.err,/^stack-release: refusing capture-failed: could not write the rollback set .*; nothing was changed\n$/);
const a=lastAct(e);
assert.deepEqual(mine(e,a).map((o)=>[o.result,o.code]),[['started',undefined],['refused','capture-failed']]);
for(const f of ['AWAITING-UNITS','FAILED'])assert.ok(!existsSync(join(rs(e,a),f)),f);
assert.deepEqual(open(e),[]);
rmSync(join(e.sd,'faults','no-fragment'));
// The set can't be made: marker-failed, for that activation.
chmodSync(join(e.state,'stack-rollback'),0o555);
const m=await run(x,'activate',C1,'acme');assert.equal(m.code,3,m.out+m.err);
assert.match(m.err,/^stack-release: refusing marker-failed: could not make the rollback set .*; nothing was changed\n$/);
const b=lastAct(e);assert.notEqual(b,a);
assert.deepEqual(mine(e,b).map((o)=>[o.result,o.code]),[['started',undefined],['refused','marker-failed']]);
chmodSync(join(e.state,'stack-rollback'),0o755);
assert.deepEqual(open(e),[]);
});
test('row 232: confirm and revert refuse open-activation with no marker, two, the wrong copy, or one the log holds closed; refusals name no activation',async(t)=>{
const {e,x}=await xseed(t);
const tryBoth=async(id,why)=>{
for(const v of ['confirm','revert']){
const o=open(e),logs=logLines(e).length;
const r=await run(x,v,id,'acme');assert.equal(r.code,3,`${v} ${why}: ${r.out}${r.err}`);
assert.match(r.err,/^stack-release: refusing open-activation: /,why);
assert.deepEqual(logLines(e).slice(logs).map((l)=>[l.result,l.code,l.activation,l.op]),[['refused','open-activation',null,v]],why);
assert.deepEqual(open(e),o,why);
}
};
await tryBoth(C1,'no marker');
// One marker whose activation the log never started. It sorts before any real one, so a confirm
// that took the newest of two markers would take the real one.
const fake=join(e.state,'stack-rollback','20000101T000000Z-'+C1.slice(0,12));
mkdirSync(fake,{recursive:true});writeFileSync(join(fake,'AWAITING-UNITS'),`${C1}\n`);
await tryBoth(C1,'marker not open');
rmSync(fake,{recursive:true});
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
await tryBoth(B,'wrong copy');
mkdirSync(fake);writeFileSync(join(fake,'AWAITING-UNITS'),`${C1}\n`);
await tryBoth(C1,'two markers');
rmSync(fake,{recursive:true});
// gate refuses nothing and writes nothing, open or not.
const logs=readFileSync(e.log,'utf8');
const g=await run(x,'gate',C1,'acme','--since','1','--timeout','1');assert.equal(g.code,1);
assert.equal(readFileSync(e.log,'utf8'),logs);assert.deepEqual(open(e),[a]);
});
test('row 232: confirm polls: bus status good after 3 s fails with --timeout 1, writes FAILED and leaves the rest; FAILED then refuses activation-failed',async(t)=>{
const {e,x}=await xseed(t);
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
up(e,C1);
const p0=readFileSync(e.pointer,'utf8'),l0=readFileSync(e.log,'utf8');
fault(e,`status-until-${C1}`,String(Date.now()+3000));
const c=await run(x,'confirm',C1,'acme','--timeout','1');assert.equal(c.code,1,c.out+c.err);
assert.match(c.err,new RegExp(`^stack-release: activate failed with bus-status; ${a} stays open`));
assert.equal(readFileSync(join(rs(e,a),'FAILED'),'utf8'),'bus-status\n');
assert.equal(readFileSync(e.pointer,'utf8'),p0);assert.equal(readFileSync(e.log,'utf8'),l0);
assert.ok(existsSync(join(rs(e,a),'AWAITING-UNITS')));
const f=await run(x,'confirm',C1,'acme');assert.equal(f.code,3);
assert.match(f.err,new RegExp(`^stack-release: refusing activation-failed: ${a} has a FAILED file`));
assert.equal(logLines(e).at(-1).activation,null);assert.deepEqual(open(e),[a]);
});
test('row 232: confirm polls to a pass with --timeout 10: the pointer, an ok line that reads as the started run, no marker',async(t)=>{
const {e,x}=await xseed(t);
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
up(e,C1);
fault(e,`status-until-${C1}`,String(Date.now()+3000));
const c=await run(x,'confirm',C1,'acme','--timeout','10');assert.equal(c.code,0,c.out+c.err);
assert.equal(c.out.split('\n').at(-2),`stack-release: activate ok: ${C1} is live for acme`);
const p=pointer(e);assert.deepEqual([p.copy,p.activation,p.schema,p.pm,p.console],[C1,a,'3g',false,['--port',e.port,'--business','acme']]);
const f=sameAsStarted(e,a);assert.deepEqual([f.result,f.op,f.from,f.to,f.backup],['ok','activate',B,C1,a]);
assert.ok(!existsSync(join(rs(e,a),'AWAITING-UNITS')));
assert.deepEqual(journalCalls(e),[]);
});
// N4 with the brief's numbers: the started line at 1792540800.900 floors to 1792540800. The host's
// clock runs so boot status is after that second.
const T0=1792540800,offsetTo=(s)=>Math.floor(Date.now()/1000)-s;
function startedAt(e,a,ms){
const l=readFileSync(e.log,'utf8').split('\n').map((x)=>{if(!x)return x;const o=JSON.parse(x);if(o.activation===a&&o.result==='started')o.at=new Date(ms).toISOString();return JSON.stringify(o)});
writeFileSync(e.log,l.join('\n'));
}
test('row 232: stale-start: @1792540799 fails a start floor of 1792540800 with stale-start',async(t)=>{
const {e,x}=await xseed(t);
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
startedAt(e,a,T0*1000+900);up(e,C1,{offset:offsetTo(T0+100)});
fault(e,'enter',JSON.stringify({value:T0-1}));
const c=await run(x,'confirm',C1,'acme','--timeout','1');assert.equal(c.code,1,c.out+c.err);
assert.ok(c.out.includes(`gate failed on ${C1}: stale-start (the bus started at @${T0-1}, before @${T0})`),c.out);
assert.equal(readFileSync(join(rs(e,a),'FAILED'),'utf8'),'stale-start\n');
});
test('row 232: stale-start is a gate code: an enter time that turns to @1792540800 after 2 s passes with --timeout 10',async(t)=>{
const {e,x}=await xseed(t);
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
startedAt(e,a,T0*1000+900);up(e,C1,{offset:offsetTo(T0+100)});
fault(e,'enter',JSON.stringify({value:T0-1,at:Date.now()+2000,then:T0}));
const c=await run(x,'confirm',C1,'acme','--timeout','10');assert.equal(c.code,0,c.out+c.err);
assert.equal(pointer(e).copy,C1);
});
test('row 232: gate (N2, A2f): --since 1792540800 passes @1792540800 and fails @1792540799; it writes nothing',async(t)=>{
const {e,x}=await xseed(t);
up(e,B,{offset:offsetTo(T0+100)});
const snap=()=>[readFileSync(e.log,'utf8'),readFileSync(e.pointer,'utf8'),tree(e.state).join('\n')];
const s0=snap();
fault(e,'enter',JSON.stringify({value:T0}));
const g=await run(x,'gate',B,'acme','--since',String(T0));assert.equal(g.code,0,g.out+g.err);
assert.equal(g.out,`stack-release: gate passed on ${B}\n`);
fault(e,'enter',JSON.stringify({value:T0-1}));
const f=await run(x,'gate',B,'acme','--since',String(T0),'--timeout','1');assert.equal(f.code,1,f.out+f.err);
assert.ok(f.out.includes(`gate failed on ${B}: stale-start (the bus started at @${T0-1}, before @${T0})`),f.out);
assert.deepEqual(snap(),s0);
const u=await run(x,'gate',B,'acme');assert.equal(u.code,4);
assert.deepEqual(journalCalls(e),[]);
});
test('row 232: the external PM check reads launches list: one live run of launch.by since the start passes',async(t)=>{
const {e,x}=await xseed(t,{pm:true});
const cases=[['',0,''],['coder',0,''],['none',1,'no live launch of chief'],['twolive',1,'2 live launches of chief'],['early',1,'started before this start'],['dead',1,'no live launch of chief'],['other',1,'the launch of chief is for other']];
for(const [mode,code,why] of cases){
rmSync(join(e.sd,'launches.json'),{force:true});
if(mode)fault(e,`pm-${B}`,mode);else rmSync(join(e.sd,'faults',`pm-${B}`),{force:true});
const since=Math.floor(Date.now()/1000);
up(e,B,{pm:true});
const g=await run(x,'gate',B,'acme','--since',String(since),'--timeout','1');assert.equal(g.code,code,`${mode}: ${g.out}${g.err}`);
if(why)assert.ok(g.out.includes(`gate failed on ${B}: pm-launch-unconfirmed (`)&&g.out.includes(why),`${mode}: ${g.out}`);
down(e);
}
assert.deepEqual(journalCalls(e),[]);
});
test('row 232: revert puts the wrapper back and leaves the store when nothing migrated',async(t)=>{
const {e,x}=await xseed(t);
const w0=sha(join(e.bin,'mosaic'));
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
// A write to the stopped store after A3: revert has no MIGRATING, so it is not its to undo.
const db=new DatabaseSync(e.store);db.prepare('INSERT INTO events (at) VALUES (?)').run('2001-01-01T00:00:00.000Z');db.close();
const s1=storeSha(e);assert.notEqual(s1,sha(backupStore(e,a)));
const v=await run(x,'revert',C1,'acme');assert.equal(v.code,0,v.out+v.err);
assert.equal(v.out.split('\n').at(-2),`stack-release: revert ok: ${a}'s wrapper is back; root installs the old units next`);
assert.equal(sha(join(e.bin,'mosaic')),w0);assert.equal(storeSha(e),s1);
assert.deepEqual(mine(e,a).map((o)=>o.result),['started']);assert.ok(existsSync(join(rs(e,a),'AWAITING-UNITS')));
});
test('row 232: confirm --rollback refuses rollback-mismatch, and logs rollback-failed with exit 2 when OLD fails its gate',async(t)=>{
const {e,x}=await xseed(t);
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
const v=await run(x,'revert',C1,'acme');assert.equal(v.code,0,v.out+v.err);
const m=await run(x,'confirm',C1,'acme','--rollback',C,'--cause','A5');assert.equal(m.code,3);
assert.equal(m.err,`stack-release: refusing rollback-mismatch: ${C} is not the copy ${a} started from, ${B}\n`);
assert.equal(logLines(e).at(-1).activation,null);assert.deepEqual(open(e),[a]);
fault(e,`healthz-${B}`);up(e,B);
const f=await run(x,'confirm',C1,'acme','--rollback',B,'--cause','A5','--timeout','1');assert.equal(f.code,2,f.out+f.err);
assert.equal(f.err,`stack-release: rollback-failed: the run failed with unrecorded and the rollback with healthz; root stops the units. Rollback set: ${rs(e,a)}, backup ${join(e.data,'backups','stack',a)}\n`);
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback,l.rollbackCode],['rollback-failed','unrecorded','failed','healthz']);
for(const n of ['AWAITING-UNITS','FAILED'])assert.ok(!existsSync(join(rs(e,a),n)),n);
assert.deepEqual(open(e),[]);
});
test('row 232: confirm --rollback after a --restore swap logs rollbackCode not-attempted; rollback --restore finds an external backup',async(t)=>{
const {e,x}=await xseed(t);
const {r,a}=await stage(e,x,C,'--migrate');assert.equal(r.code,0,r.out+r.err);
up(e,C);assert.equal((await run(x,'confirm',C,'acme')).code,0);
down(e);
const b=await run(x,'rollback','acme','--to',B,'--restore',a);assert.equal(b.code,0,b.out+b.err);
const ra=lastAct(e);
assert.ok(b.out.includes(`restore: ${join(e.data,'backups','stack',a)}; `),b.out);
assert.ok(existsSync(join(rs(e,ra),'RESTORED')));assert.equal(readFileSync(join(rs(e,ra),'BACKUP'),'utf8'),`${a}\n`);
assert.equal(storeSha(e),sha(backupStore(e,a)));
const f=await run(x,'confirm',B,'acme','--rollback',C,'--cause','A4');assert.equal(f.code,2,f.out+f.err);
const l=sameAsStarted(e,ra);assert.deepEqual([l.op,l.result,l.code,l.rollbackCode,l.backup],['rollback','rollback-failed','units-start','not-attempted',a]);
});
// --- B1: real sequences on one log ---
test('row 232: sequence 1: activate, confirm times out, revert, confirm --rollback --cause A5: gate-failed rollback ok',async(t)=>{
const {e,x}=await xseed(t);
const w0=sha(join(e.bin,'mosaic')),p0=readFileSync(e.pointer,'utf8');
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
fault(e,`healthz-${C1}`);up(e,C1);
const c=await run(x,'confirm',C1,'acme','--timeout','1');assert.equal(c.code,1,c.out+c.err);
assert.deepEqual(mine(e,a).map((o)=>o.result),['started']);
down(e);
assert.equal((await run(x,'revert',C1,'acme')).code,0);assert.equal(sha(join(e.bin,'mosaic')),w0);
up(e,B);
const k=await run(x,'confirm',C1,'acme','--rollback',B,'--cause','A5');assert.equal(k.code,1,k.out+k.err);
assert.equal(k.err,`stack-release: healthz; rolled back to ${B}, which passed the gate\n`);
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['gate-failed','healthz','ok']);
assert.equal(readFileSync(e.pointer,'utf8'),p0);
await closed(e,x,a);
});
test('row 232: sequence 2: activate --migrate, the wrapper install fails, revert restores TAKEN, confirm --rollback --cause A3',async(t)=>{
const {e,x}=await xseed(t);
const s0=storeSha(e);
chmodSync(e.bin,0o555);
const {r,a}=await stage(e,x,C,'--migrate');chmodSync(e.bin,0o755);
assert.equal(r.code,1,r.out+r.err);assert.match(r.err,new RegExp(`^stack-release: activate failed with wrapper; ${a} stays open`));
assert.deepEqual(mine(e,a).map((o)=>o.result),['started']);
for(const n of ['AWAITING-UNITS','MIGRATING'])assert.ok(existsSync(join(rs(e,a),n)),n);
assert.equal(readFileSync(join(rs(e,a),'FAILED'),'utf8'),'wrapper\n');
assert.equal(storeVersion(e.store,false),'3h');assert.equal(sha(backupStore(e,a)),s0);
const v=await run(x,'revert',C,'acme');assert.equal(v.code,0,v.out+v.err);
assert.equal(v.out.split('\n').at(-2),`stack-release: revert ok: ${a}'s wrapper and store are back; root installs the old units next`);
assert.equal(storeSha(e),s0);
// revert runs again: the store already matches, and nothing moves.
const posts=readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-'));
assert.equal((await run(x,'revert',C,'acme')).code,0);
assert.deepEqual(readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-')),posts);
up(e,B);
const k=await run(x,'confirm',C,'acme','--rollback',B,'--cause','A3');assert.equal(k.code,1,k.out+k.err);
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['failed','wrapper','ok']);
await closed(e,x,a);
});
test('row 232: sequence 3: activate, root\'s A4 fails, revert, confirm --rollback --cause A4: failed code units-start',async(t)=>{
const {e,x}=await xseed(t);
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
assert.equal((await run(x,'revert',C1,'acme')).code,0);
up(e,B);
const k=await run(x,'confirm',C1,'acme','--rollback',B,'--cause','A4');assert.equal(k.code,1,k.out+k.err);
assert.equal(k.err,`stack-release: units-start; rolled back to ${B}, which passed the gate\n`);
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['failed','units-start','ok']);
await closed(e,x,a);
});
test('row 232: sequence 4: a capture failure leaves nothing open, then A2f\'s gate OLD writes nothing',async(t)=>{
const {e,x}=await xseed(t);
fault(e,'no-fragment');
const r=await run(x,'activate',C1,'acme');assert.equal(r.code,3,r.out+r.err);rmSync(join(e.sd,'faults','no-fragment'));
const a=lastAct(e);assert.deepEqual(mine(e,a).map((o)=>[o.result,o.code]),[['started',undefined],['refused','capture-failed']]);
assert.deepEqual(open(e),[]);
const since=Math.floor(Date.now()/1000);up(e,B);
const s0=[readFileSync(e.log,'utf8'),readFileSync(e.pointer,'utf8'),tree(e.state).join('\n')];
const g=await run(x,'gate',B,'acme','--since',String(since));assert.equal(g.code,0,g.out+g.err);
assert.deepEqual([readFileSync(e.log,'utf8'),readFileSync(e.pointer,'utf8'),tree(e.state).join('\n')],s0);
});
test('row 232: sequence 5: a confirm of the wrong copy is refused, and revert and confirm --rollback still close the activation',async(t)=>{
const {e,x}=await xseed(t);
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
up(e,C1);
const w=await run(x,'confirm',B,'acme');assert.equal(w.code,3);assert.match(w.err,/^stack-release: refusing open-activation: /);
assert.equal(logLines(e).at(-1).activation,null);assert.deepEqual(open(e),[a]);
down(e);
assert.equal((await run(x,'revert',C1,'acme')).code,0);
up(e,B);
const k=await run(x,'confirm',C1,'acme','--rollback',B,'--cause','A5');assert.equal(k.code,1,k.out+k.err);
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['gate-failed','unrecorded','ok']);
await closed(e,x,a);
});
test('row 232: sequence 6: a revert that fails leaves the activation open, and the next --plan refuses interrupted-run',async(t)=>{
const {e,x}=await xseed(t);
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
rmSync(join(rs(e,a),'wrapper'));
const v=await run(x,'revert',C1,'acme');assert.equal(v.code,1,v.out+v.err);
assert.equal(v.err,`stack-release: revert failed: the rollback set has no saved wrapper; ${a} stays open. Rollback set: ${rs(e,a)}\n`);
assert.deepEqual(open(e),[a]);assert.ok(existsSync(join(rs(e,a),'AWAITING-UNITS')));
const p=await run(x,'activate',C,'acme','--plan','--migrate');assert.equal(p.code,3,p.out+p.err);
assert.ok(p.out.includes(`refuses interrupted-run: activation ${a} started and has no final line`),p.out);
});
test('row 232: sequence 7: adopt, two confirmed activates, rollback --to the previous copy, confirmed, logs op rollback',async(t)=>{
const {e,x}=await xseed(t);
const one=await stage(e,x,C1);assert.equal(one.r.code,0,one.r.out+one.r.err);
up(e,C1);assert.equal((await run(x,'confirm',C1,'acme')).code,0);down(e);
const two=await stage(e,x,C,'--migrate');assert.equal(two.r.code,0,two.r.out+two.r.err);
up(e,C);assert.equal((await run(x,'confirm',C,'acme')).code,0);down(e);
const l1=sameAsStarted(e,one.a),l2=sameAsStarted(e,two.a);
assert.deepEqual([l1.op,l1.result,l1.from,l1.to,l1.backup],['activate','ok',B,C1,one.a]);
assert.deepEqual([l2.op,l2.result,l2.from,l2.to,l2.backup],['activate','ok',C1,C,two.a]);
// N5: --expect-from names the copy root read from its units.
const d1=await run(x,'activate',C1,'acme','--plan','--expect-from',C1);assert.equal(d1.code,3,d1.out+d1.err);
assert.ok(d1.out.includes(`refuses pointer-differs: the pointer names ${C} and root's units name ${C1}; Jason or Mos settles it`),d1.out);
const d2=await run(x,'rollback','acme','--to',C1,'--plan','--expect-from',B);assert.equal(d2.code,3,d2.out+d2.err);
assert.ok(d2.out.includes(`refuses pointer-differs: the pointer names ${C} and root's units name ${B}`),d2.out);
const ok=await run(x,'rollback','acme','--to',C1,'--restore',two.a,'--plan','--expect-from',C);assert.equal(ok.code,0,ok.out+ok.err);
const logs=logLines(e).length;
const no=await run(x,'rollback','acme','--to',B,'--plan');assert.equal(no.code,3);
assert.equal(no.err,`stack-release: refusing to-not-previous: --to ${B} is not the previous copy; the log names ${C1}\n`);
assert.equal(logLines(e).length,logs);
const nr=await run(x,'rollback','acme','--to',B,'--restore',one.a);assert.equal(nr.code,3);assert.match(nr.err,/^stack-release: refusing to-not-previous: /);
assert.deepEqual(logLines(e).slice(logs).map((o)=>[o.result,o.code,o.activation]),[['refused','to-not-previous',null]]);
const rb=await run(x,'rollback','acme','--to',C1,'--restore',two.a);assert.equal(rb.code,0,rb.out+rb.err);
const ra=lastAct(e);
up(e,C1);
const k=await run(x,'confirm',C1,'acme');assert.equal(k.code,0,k.out+k.err);
assert.equal(k.out.split('\n').at(-2),`stack-release: rollback ok: ${C1} is live for acme`);
const l3=sameAsStarted(e,ra);assert.deepEqual([l3.op,l3.result,l3.from,l3.to,l3.backup],['rollback','ok',C,C1,two.a]);
assert.deepEqual([pointer(e).copy,pointer(e).schema],[C1,'3g']);
const prev=await run(x,'rollback','acme','--to',B,'--plan');assert.match(prev.err,new RegExp(`the log names ${C}\\n$`));
await closed(e,x,ra);
});
test('row 232: sequence 8: a migration that writes and fails: revert restores the backup byte for byte, confirm --rollback --cause A3',async(t)=>{
const {e,x}=await xseed(t);
const s0=storeSha(e);
fault(e,'migrate-fails');
const {r,a}=await stage(e,x,C,'--migrate');assert.equal(r.code,1,r.out+r.err);rmSync(join(e.sd,'faults','migrate-fails'));
assert.equal(readFileSync(join(rs(e,a),'FAILED'),'utf8'),'migrate-failed\n');assert.ok(existsSync(join(rs(e,a),'MIGRATING')));
assert.equal(storeVersion(e.store,false),'3h-partial');
assert.equal((await run(x,'revert',C,'acme')).code,0);
assert.equal(storeSha(e),s0);assert.equal(storeSha(e),sha(backupStore(e,a)));
assert.deepEqual(readdirSync(join(e.data,'bus')),readdirSync(join(e.data,'backups','stack',a,'bus')));
up(e,B);
const k=await run(x,'confirm',C,'acme','--rollback',B,'--cause','A3');assert.equal(k.code,1,k.out+k.err);
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['failed','migrate-failed','ok']);
await closed(e,x,a);
});
// --- User mode against the base ---
// The fixture repo with the base's two scripts: everything else is the same tree and history.
const baseRepo=join(base,'base-repo');
cpSync(repo,baseRepo,{recursive:true});
for(const f of ['scripts/stack-release.sh','scripts/stack-copy.sh'])writeFileSync(join(baseRepo,f),execFileSync('git',['-C',ROOT,'show',`${BASE}:${f}`]),{mode:0o755});
const BASE_SR=join(baseRepo,'scripts','stack-release.sh');
test('row 232: user mode: every systemctl call carries --user, and activate, rollback and check make the base\'s calls',async(t)=>{
const calls=async(sr,dir)=>{
const e=await mk(t);e.sr=sr;await seed(e);
const n=lines(join(e.sd,'argv.log')).length;
const a=await run(e,'activate',C1,'acme');assert.equal(a.code,0,a.out+a.err);
const r=await run(e,'rollback','acme');assert.equal(r.code,0,r.out+r.err);
execFileSync('bash',[join(dir,'scripts','stack-copy.sh'),'check','--copies',COPIES,'--bin',e.bin,'--business','acme'],{env:e.env,stdio:'ignore'});
return lines(join(e.sd,'argv.log')).slice(n).map((l)=>l.split(e.root).join('<root>').split(e.port).join('<port>'));
};
const cand=await calls(undefined,repo),old=await calls(BASE_SR,baseRepo);
assert.ok(cand.length>20);
for(const c of cand)assert.match(c,/^--user /,c);
assert.deepEqual(cand,old);
});
test('row 232: user mode: a default activate --plan prints the base\'s bytes',async(t)=>{
const e=await mk(t);await seed(e);
for(const a of [['activate',C1,'acme','--plan'],['activate',C,'acme','--plan','--migrate'],['activate',C,'acme','--plan'],['rollback','acme','--plan']]){
const cand=await run(e,...a),old=await run({...e,sr:BASE_SR},...a);
assert.deepEqual(cand,old,a.join(' '));
}
assert.ok(!existsSync(join(e.state,'stack-rollback')));
});
@@ -0,0 +1,451 @@
// The stack-release stand-ins and module fixture, shared by stack-release.test.mjs and
// stack-release-external.test.mjs (row 232). Moved unchanged from stack-release.test.mjs, plus
// what the external tests need; each test file runs in its own process, so each gets its own base.
import test, {after} from 'node:test';
import assert from 'node:assert/strict';
import {execFileSync,spawn,spawnSync} from 'node:child_process';
import {mkdtempSync,mkdirSync,writeFileSync,readFileSync,appendFileSync,cpSync,rmSync,chmodSync,existsSync,statSync,lstatSync,symlinkSync,linkSync,readdirSync,realpathSync} from 'node:fs';
import {createServer} from 'node:http';
import {createHash} from 'node:crypto';
import {DatabaseSync} from 'node:sqlite';
import {tmpdir} from 'node:os';
import {join,dirname,basename} from 'node:path';
import {fileURLToPath,pathToFileURL} from 'node:url';
import {once} from 'node:events';
export const ROOT=fileURLToPath(new URL('../../..',import.meta.url)).replace(/\/$/,'');
export const NODE=process.execPath;
export const MARKER='# mosaic-cli wrapper (Q81, agents/sage/work/q81/mosaic-cli.sh)';
export const BUS='[email protected]',CON='mosaic-webui.service';
export const git=(cwd,...args)=>execFileSync('git',['-c','user.name=fixture','-c','user.email=fixture@invalid','-c','commit.gpgsign=false','-c','core.hooksPath=/dev/null','-C',cwd,...args],{encoding:'utf8'}).trim();
export const script=(fn)=>`#!${NODE}\n(${fn})();\n`;
// --- Stand-ins. Each is serialized into its own file, so it uses only process and its own scope. ---
// systemctl --user: unit state in $SD_STATE/units.json, every call in calls.log. A bus start runs
// holder.cjs (the store held open in WAL mode, writer.lock, host.json) and, with --pm in the unit's
// ExecStart, writes the `launched` journal line and the launches entry unless a pm-<copy> fault
// names another case. Faults are files in $SD_STATE/faults named for the copy they hit. Units of a
// named instance (row 190) run on their own data root and host files beside the default's.
export function systemctl(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process'),crypto=process.getBuiltinModule('node:crypto');
const sd=process.env.SD_STATE,data=process.env.SD_DATA,unitDir=path.join(process.env.XDG_CONFIG_HOME,'systemd','user');
const args=process.argv.slice(2).filter((a)=>a!=='--user'),[verb,...rest]=args;
fs.appendFileSync(path.join(sd,'calls.log'),args.join(' ')+'\n');
// Row 232: argv.log keeps each call as given, --user and all, for the user/external argv tests.
fs.appendFileSync(path.join(sd,'argv.log'),process.argv.slice(2).join(' ')+'\n');
const faultFile=(n)=>path.join(sd,'faults',n),fault=(n)=>fs.existsSync(faultFile(n));
const stFile=path.join(sd,'units.json');
const load=()=>{try{return JSON.parse(fs.readFileSync(stFile,'utf8'))}catch{return {}}};
const save=(s)=>{fs.writeFileSync(stFile+'.tmp',JSON.stringify(s));fs.renameSync(stFile+'.tmp',stFile)};
const sleep=(ms)=>Atomics.wait(new Int32Array(new SharedArrayBuffer(4)),0,0,ms);
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
// Row 190: a bus unit is the default's mosaic-bus@B or instance NAME's mosaic-bus-NAME@B; %i is B.
const isBus=(u)=>/^mosaic-bus(-[a-z][a-z0-9]{1,15})?@[a-z0-9-]+\.service$/.test(u),inst=(u)=>u.slice(u.indexOf('@')+1,-'.service'.length);
const unitFile=(u)=>{
const main=isBus(u)?u.replace(/@[^@]*$/,'@.service'):u,file=path.join(unitDir,main);
if(!fs.existsSync(file))return null;
// The drop-in directories in systemd's order: service.d, the name cut after each hyphen, then
// the template's and the unit's own.
const stem=main.slice(0,main.search(/@?\.service$/)),dirs=['service.d'],drops=[];
for(let i=0;i<stem.length;i++)if(stem[i]==='-')dirs.push(stem.slice(0,i+1)+'.service.d');
for(const d of [...dirs,...(isBus(u)?[main+'.d',u+'.d']:[u+'.d'])]){const dir=path.join(unitDir,d);if(fs.existsSync(dir))for(const f of fs.readdirSync(dir).sort())if(f.endsWith('.conf'))drops.push(path.join(dir,f))}
const r={file,wd:'',exec:'',env:[],drops};
for(const f of [file,...drops])for(let line of fs.readFileSync(f,'utf8').split('\n')){
line=line.replace(/%i/g,isBus(u)?inst(u):'').replace(/%h/g,process.env.HOME);
const m=/^(WorkingDirectory|ExecStart|Environment)=(.*)$/.exec(line);if(!m)continue;
if(m[1]==='WorkingDirectory')r.wd=m[2];else if(m[1]==='ExecStart'){if(m[2])r.exec=m[2]}else r.env.push(m[2]);
}
return r;
};
// A named instance's unit sets MOSAIC_CONFIG, and its bus works on that config's data root. A unit
// without it is the default's, on $SD_DATA. Each data root has its own host and launches files.
const dataOf=(f)=>{const c=f&&f.env.find((v)=>v.startsWith('MOSAIC_CONFIG='));return c?JSON.parse(fs.readFileSync(c.slice('MOSAIC_CONFIG='.length),'utf8')).dataRoot:data};
const perData=(d,n)=>path.join(sd,d===data?`${n}.json`:`${n}-${path.basename(d)}.json`);
const snap=(ev,d)=>{
const dir=path.join(d,'bus'),files={};
try{for(const n of fs.readdirSync(dir).sort()){const p=path.join(dir,n),st=fs.statSync(p);if(st.isFile())files[n]=[st.size,st.mtimeMs,crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')]}}catch{}
fs.appendFileSync(path.join(sd,'snap.jsonl'),JSON.stringify({ev,files})+'\n');
};
const pm=(unit,d,id,pid,off)=>{
let mode='';try{mode=fs.readFileSync(faultFile(`pm-${id}`),'utf8').trim()}catch{}
const b=inst(unit),now=Date.now()-off*1000,epoch=Math.floor(now/1000),at=new Date(now).toISOString(),run=`run-${pid}`;
const line=(text)=>fs.appendFileSync(path.join(sd,'journal.jsonl'),JSON.stringify({epoch,unit,text})+'\n');
const ok=()=>line(`launched pm-${b}: run ${run}, claude opus`);
const lf=perData(d,'launches');let list=[];try{list=JSON.parse(fs.readFileSync(lf,'utf8'))}catch{}
// instance is launch.by's, chief in the example business, as the external PM check reads (row 232).
const entry=(o)=>list.push({run,instance:'chief',business:b,live:true,startedAt:at,...o});
line(`bus host up: business ${b}, socket /fixture/bus.sock, notifier off`);
switch(mode){
case 'none':break;
case 'two':ok();line(`launched pm-${b}: run ${run}b, claude opus`);entry({});break;
case 'badword':line(`launched pm-${b}: job ${run}, claude opus`);entry({});break;
case 'notlisted':ok();break;
case 'early':ok();entry({startedAt:new Date(now-3600e3).toISOString()});break;
case 'human':entry({run:`human-${pid}`});break;
case 'other':ok();entry({business:'other'});break;
case 'dead':ok();entry({live:false});break;
// Row 232: two live entries for the PM's instance, and one for it beside a live coder run.
case 'twolive':ok();entry({});entry({run:`${run}b`});break;
case 'coder':ok();entry({});entry({run:`${run}c`,instance:'coder'});break;
default:ok();entry({});
}
fs.writeFileSync(lf,JSON.stringify(list));
};
if(verb==='show'){
// `show [--timestamp=unix] UNIT -p P...` and `show -p P UNIT`, as the external path calls it.
let u='';const props=[];for(let i=0;i<rest.length;i++){if(rest[i]==='-p')props.push(rest[++i]);else if(!rest[i].startsWith('--'))u=rest[i]}
const f=unitFile(u),st=load()[u]||{},id=f?path.basename(f.wd):'';
// ActiveEnterTimestamp (row 232) is the start's second, or what the enter fault says: a JSON
// {value} or {value,at,then}, which answers then from epoch ms at on.
const enter=()=>{
if(!fault('enter'))return st.enter;
const o=JSON.parse(fs.readFileSync(faultFile('enter'),'utf8'));return o.at!==undefined&&Date.now()>=o.at?o.then:o.value;
};
for(const p of props){
if(p==='LoadState')console.log(`LoadState=${f?'loaded':'not-found'}`);
else if(p==='WorkingDirectory')console.log(`WorkingDirectory=${f?f.wd+(!isBus(u)&&fault(`check-${id}`)?'-moved':''):''}`);
else if(p==='ExecStart')console.log(f&&f.exec?`ExecStart={ path=${f.exec.split(' ')[0]} ; argv[]=${f.exec} ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }`:'ExecStart=');
else if(p==='Environment')console.log(`Environment=${f?f.env.join(' '):''}`);
else if(p==='MainPID')console.log(`MainPID=${st.pid||0}`);
else if(p==='DropInPaths')console.log(`DropInPaths=${f?f.drops.join(' '):''}`);
else if(p==='FragmentPath')console.log(`FragmentPath=${f&&!fault('no-fragment')?f.file:''}`);
else if(p==='ActiveEnterTimestamp'){const v=st.state==='active'?enter():undefined;console.log(`ActiveEnterTimestamp=${v===undefined?'':'@'+v}`)}
}
process.exit(0);
}
if(verb==='is-active'){
const q=rest[0]==='--quiet',u=q?rest[1]:rest[0];let s=(load()[u]||{}).state||'inactive';
// bus-stops-after N (T29): the next N bus checks answer the real state, every later one answers
// inactive, and units.json stays as it is, as if the bus stopped between two checks.
if(isBus(u)&&fault('bus-stops-after')){const n=Number(fs.readFileSync(faultFile('bus-stops-after'),'utf8'));if(n>0)fs.writeFileSync(faultFile('bus-stops-after'),String(n-1));else s='inactive'}
if(!q)console.log(s);process.exit(s==='active'?0:3);
}
if(verb==='is-enabled'){console.log('disabled');process.exit(1)}
if(verb==='daemon-reload')process.exit(0);
if(verb==='list-units'){
if(fault('list-fails'))process.exit(1);
const pats=rest.filter((a)=>!a.startsWith('--')).map((p)=>new RegExp('^'+p.replace(/\./g,'\\.').replace(/\*/g,'.*')+'$'));
for(const [u,s] of Object.entries(load()))if(pats.some((re)=>re.test(u)))console.log(`${u} loaded ${s.state} ${s.state==='active'?'running':'dead'} fixture`);
process.exit(0);
}
if(verb==='start'){
const u=rest[0],f=unitFile(u);if(!f){console.error(`Unit ${u} not found.`);process.exit(5)}
const s=load(),id=path.basename(f.wd),d=dataOf(f);
const enter=Math.floor((Date.now()-Number(process.env.SD_CLOCK_OFFSET||'0')*1000)/1000);
if(!isBus(u)){s[u]={state:'active',pid:null,wd:f.wd,enter};save(s);process.exit(0)}
if(fault(`start-dies-${id}`)){s[u]={state:'failed',pid:null,wd:f.wd};save(s);process.exit(0)}
// Row 185 (T48): the copy's start writes one event before the holder opens the store, so the
// store differs from a backup the run took before the start. The close leaves no -wal.
if(fault(`start-writes-row-${id}`)){
const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(d,'bus','bus.sqlite'));
db.prepare('INSERT INTO events (at) VALUES (?)').run('2999-01-01T00:00:00.000Z');db.close();
}
snap('start',d);
const off=process.env.SD_CLOCK_OFFSET||'0';
// The holder must not inherit stack-release.sh's lock (fd 9) or any other fd above 2.
const close='for fd in /proc/self/fd/*; do n=${fd##*/}; [ "$n" -gt 2 ] && eval "exec $n>&-" 2>/dev/null; done; exec "$@"';
const ch=cp.spawn('bash',['-c',close,'holder',process.execPath,path.join(__dirname,'..','holder.cjs'),d,inst(u),sd,off,perData(d,'host')],{detached:true,stdio:'ignore'});
ch.unref();
fs.appendFileSync(path.join(sd,'pids.log'),ch.pid+'\n');
const ready=path.join(sd,`ready-${ch.pid}`),end=Date.now()+10000;
while(!fs.existsSync(ready)){if(Date.now()>end){console.error('fixture holder did not start');process.exit(1)}sleep(20)}
s[u]={state:'active',pid:ch.pid,wd:f.wd,enter};save(s);
if(f.exec.split(' ').slice(1).includes('--pm'))pm(u,d,id,ch.pid,Number(off));
process.exit(0);
}
if(verb==='stop'){
const u=rest[0],s=load(),st=s[u];
if(st&&st.pid){
try{process.kill(st.pid,'SIGTERM')}catch{}
const end=Date.now()+10000;while(alive(st.pid)&&Date.now()<end)sleep(20);
if(alive(st.pid))try{process.kill(st.pid,'SIGKILL')}catch{}
}
if(st){s[u]={...st,state:'inactive',pid:null};save(s)}
if(isBus(u)){
const d=dataOf(unitFile(u));
// One-shot: the stop leaves an empty -wal or -shm beside the store, as an unclean close would.
for(const [name,ext] of [['stop-leaves-wal','-wal'],['stop-leaves-shm','-shm']])if(fault(name)){fs.rmSync(faultFile(name));fs.writeFileSync(path.join(d,'bus','bus.sqlite'+ext),'')}
// One-shot (T45): the stop writes one row after any STAMP into the table the fault names, as a
// write the bus made between rollback's count and its stop would. The close leaves no -wal.
if(fault('stop-writes-row')){
const t=fs.readFileSync(faultFile('stop-writes-row'),'utf8');fs.rmSync(faultFile('stop-writes-row'));
const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(d,'bus','bus.sqlite'));
db.exec(`CREATE TABLE IF NOT EXISTS ${t} (seq INTEGER PRIMARY KEY, at TEXT NOT NULL)`);
db.prepare(`INSERT INTO ${t} (at) VALUES (?)`).run('2999-01-01T00:00:00.000Z');db.close();
}
snap('stop',d);
}
process.exit(0);
}
console.error(`fixture systemctl: unsupported: ${args.join(' ')}`);process.exit(1);
}
// journalctl --user -u UNIT --since @EPOCH -o cat. A bare local time gets exit 1, as M35 needs.
export function journalctl(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
const sd=process.env.SD_STATE,a=process.argv.slice(2);let unit='',since='';
for(let i=0;i<a.length;i++){if(a[i]==='-u')unit=a[++i];else if(a[i]==='--since')since=a[++i]}
fs.appendFileSync(path.join(sd,'journal-since.log'),since+'\n');
if(fs.existsSync(path.join(sd,'faults','journal-fails')))process.exit(1);
let from;
if(/^@\d+$/.test(since))from=Number(since.slice(1));
else{const m=/^(\d{4}-\d\d-\d\d) (\d\d:\d\d:\d\d) UTC$/.exec(since);if(!m)process.exit(1);from=Date.parse(`${m[1]}T${m[2]}Z`)/1000}
let lines=[];try{lines=fs.readFileSync(path.join(sd,'journal.jsonl'),'utf8').split('\n').filter(Boolean).map((l)=>JSON.parse(l))}catch{}
for(const o of lines)if(o.unit===unit&&o.epoch>=from)console.log(o.text);
}
export function systemdAnalyze(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
if(process.argv[2]!=='--user'||process.argv[3]!=='unit-paths')process.exit(1);
console.log(path.join(process.env.XDG_CONFIG_HOME,'systemd','user'));
// The unit-path fault adds its text as one more path, as T28's relative path needs.
const extra=path.join(process.env.SD_STATE,'faults','unit-path');
if(fs.existsSync(extra))console.log(fs.readFileSync(extra,'utf8'));
}
// The bus host: holds the store open in WAL mode until SIGTERM, as the broker does. It writes its
// status to the host file systemctl names for its data root.
export function holder(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
const [data,business,sd,offset,host]=process.argv.slice(2),bus=path.join(data,'bus');
fs.mkdirSync(bus,{recursive:true,mode:0o700});
const db=new DatabaseSync(path.join(bus,'bus.sqlite'));
if(!db.prepare("SELECT count(*) AS n FROM sqlite_master WHERE name='meta'").get().n){
db.exec('PRAGMA journal_mode=WAL');
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
db.prepare("INSERT INTO meta VALUES ('schema_version', '3g')").run();
}
fs.writeFileSync(path.join(bus,'writer.lock'),String(process.pid));
const at=new Date(Date.now()-Number(offset||0)*1000).toISOString();
fs.writeFileSync(host+'.tmp',JSON.stringify({host:{business,pid:process.pid,live:true,startedAt:at,notifier:'ok'},boot:{ok:true,business,at}}));
fs.renameSync(host+'.tmp',host);
process.on('SIGTERM',()=>{db.close();fs.rmSync(path.join(bus,'writer.lock'),{force:true});process.exit(0)});
fs.writeFileSync(path.join(sd,`ready-${process.pid}`),'');
setInterval(()=>{},1<<30);
}
// A copy's scripts/mosaic: bus status and launches list from what the holder and systemctl wrote
// for the data root of the config it runs under (row 190), the default's being $SD_DATA.
export function mosaicStub(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
const id=path.basename(path.resolve(__dirname,'..')),sd=process.env.SD_STATE,a=process.argv.slice(2).join(' ');
const fault=(n)=>fs.existsSync(path.join(sd,'faults',n));
let d=process.env.SD_DATA;try{d=JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8')).dataRoot}catch{}
const perData=(n)=>path.join(sd,d===process.env.SD_DATA?`${n}.json`:`${n}-${path.basename(d)}.json`);
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
if(a==='bus status --json'){
if(fault(`status-${id}`))process.exit(1);
// Row 232: status-until-<copy> fails bus status until the epoch ms it holds.
const until=path.join(sd,'faults',`status-until-${id}`);
if(fs.existsSync(until)&&Date.now()<Number(fs.readFileSync(until,'utf8')))process.exit(1);
let s={host:null,boot:null};
try{s=JSON.parse(fs.readFileSync(perData('host'),'utf8'));s.host.live=alive(s.host.pid)}catch{}
console.log(JSON.stringify(s));
}else if(a==='bus status')console.log(`bus: fixture status for ${id}`);
else if(a==='launches list --json'){
if(fault(`launches-${id}`))process.exit(1);
let l=[];try{l=JSON.parse(fs.readFileSync(perData('launches'),'utf8'))}catch{}
console.log(JSON.stringify(l));
}else{console.error(`fixture mosaic: ${a}`);process.exit(4)}
}
// A copy's migrate-schema.mjs: records the backups it saw, then sets 3h, or 3h-partial and exit 1.
export function migrateStub(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
const data=process.argv[2],sd=process.env.SD_STATE,dir=path.join(data,'backups','stack');
const backups=fs.existsSync(dir)?fs.readdirSync(dir).sort().map((name)=>({name,sums:fs.existsSync(path.join(dir,name,'SHA256SUMS'))})):[];
fs.writeFileSync(path.join(sd,'migrate.json'),JSON.stringify({backups}));
const fail=fs.existsSync(path.join(sd,'faults','migrate-fails'));
const db=new DatabaseSync(path.join(data,'bus','bus.sqlite'));
db.prepare("UPDATE meta SET value=? WHERE key='schema_version'").run(fail?'3h-partial':'3h');db.close();
if(fail)process.exit(1);
console.log('fixture migrate: 3h');
}
// --- The module fixture: one repo, its copies and the stand-ins, shared by every test. ---
export const base=realpathSync(mkdtempSync(join(tmpdir(),'stack-release-')));
after(()=>{execFileSync('chmod',['-R','u+w',base]);rmSync(base,{recursive:true,force:true})});
export const repo=join(base,'repo'),COPIES=join(base,'copies'),SBIN=join(base,'sbin');
mkdirSync(repo);mkdirSync(COPIES);mkdirSync(SBIN);
writeFileSync(join(base,'package.json'),'{"type":"commonjs"}\n');
export const tracked=execFileSync('git',['-C',ROOT,'ls-files','-co','--exclude-standard','--','scripts','packages','adapters','roles','RELEASE','package.json','package-lock.json',':!packages/*/tests',':!scripts/foundation'],{encoding:'utf8'}).trim().split('\n');
for(const f of tracked)if(existsSync(join(ROOT,f)))cpSync(join(ROOT,f),join(repo,f));
export const pkg={name:'copy-fixture',version:'1.0.0'};
writeFileSync(join(repo,'package.json'),JSON.stringify(pkg));
writeFileSync(join(repo,'package-lock.json'),JSON.stringify({name:pkg.name,version:pkg.version,lockfileVersion:3,packages:{'':pkg}}));
git(repo,'init','-q','-b','refactor');git(repo,'config','gc.auto','0');git(repo,'add','.');git(repo,'commit','-qm','fixture');
export const commit=(msg)=>{git(repo,'commit','-q','--allow-empty','-m',msg);return git(repo,'rev-parse','HEAD')};
// refactor: F A B C1 C, then M merging side branch S (from A). U sits on a branch nothing reaches.
export const F=git(repo,'rev-parse','HEAD'),A=commit('row 901: A'),B=commit('row 902: B'),C1=commit('row 903: C1'),C=commit('row 904: C');
git(repo,'checkout','-q','-b','side',A);export const S=commit('row 905: S');
git(repo,'checkout','-q','-b','unreached',A);export const U=commit('row 906: U');
git(repo,'checkout','-q','refactor');git(repo,'merge','-q','--no-ff','-m','merge side','side');
export const M=git(repo,'rev-parse','HEAD');git(repo,'update-ref','refs/remotes/origin/refactor',M);
// `to` false leaves out the `to:` lines (C1); a string puts that value in each one (A). The --plan
// steps line can't be read from either (T27).
export const storeText=(v,froms,to=true)=>`export const SCHEMA_VERSION = '${v}';\nexport const MIGRATIONS = [\n${froms.map((f,i)=>` {\n from: '${f}',\n${to===false?'':` to: '${to===true?froms[i+1]??v:to}',\n`} run() {},\n },\n`).join('')}];\n`;
// Plug-in kinds (row 169): C1 and C know the harness kind, as a copy at or past c216f47db does. The
// rest have it under another name, as ca15d73e9 has no harness kind. Every copy has the real business
// loader and roles, which the tool imports to read the business file.
export const KINDS_NEW=readFileSync(join(repo,'packages','plugins','src','kinds.mjs'),'utf8');
export const KINDS_OLD=KINDS_NEW.replace('\n harness: Object.freeze({','\n "harness-before-c216f47db": Object.freeze({');
assert.notEqual(KINDS_OLD,KINDS_NEW);
// Row 200 (#1698): C1 and C take --sso and have the row 192 loader and checkConsole; C's checker
// refuses RS256, a stand-in for a target whose rules differ from the repository's. B has the loader
// from before row 192, which refuses a `console` key. The other copies have no loader.
const LOADER=readFileSync(join(repo,'scripts','mosaic-config.mjs'),'utf8'),SSO_CONFIG=readFileSync(join(repo,'packages','webui','src','sso-config.mjs'),'utf8');
const LOADER_OLD=LOADER.replace('["configVersion", "environment", "dataRoot", "execution", "console"]','["configVersion", "environment", "dataRoot", "execution"]');
const SSO_CONFIG_C=SSO_CONFIG.replace("'ES256', 'RS256',","'ES256',");
assert.notEqual(LOADER_OLD,LOADER);assert.notEqual(SSO_CONFIG_C,SSO_CONFIG);
const ssoFiles=(id)=>id===B?[['scripts/mosaic-config.mjs',LOADER_OLD]]:[C1,C].includes(id)?[['scripts/mosaic-config.mjs',LOADER],['packages/webui/src/sso-config.mjs',id===C?SSO_CONFIG_C:SSO_CONFIG]]:[];
for(const [id,store] of [[F,'// a copy from before SCHEMA_VERSION\n'],[A,storeText('3g',['3f'],'3x')],[B,storeText('3g',['3f'])],[C1,storeText('3g',['3f'],false)],[C,storeText('3h',['3f','3g'])],[S,storeText('3g',['3f'])],[U,storeText('3g',['3f'])]]){
const d=join(COPIES,id);
for(const [f,text,mode] of [['scripts/mosaic',script(mosaicStub),0o755],['packages/webui/src/cli.mjs',[B,C1,C].includes(id)?`// fixture Console, never run\n// serve accepts '--tester-port'\n// serve accepts '--proof-header' (row 178)\n${[C1,C].includes(id)?"// serve accepts '--sso' (row 192)\n":''}`:'// fixture Console before tester listener, never run\n',0o644],['packages/bus/src/store.mjs',store,0o644],['packages/bus/src/migrate-schema.mjs',`(${migrateStub})();\n`,0o644],['packages/plugins/src/kinds.mjs',[C1,C].includes(id)?KINDS_NEW:KINDS_OLD,0o644],...ssoFiles(id).map(([f,text])=>[f,text,0o644])]){
mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text,{mode});
}
for(const p of ['packages/business/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true});
execFileSync('chmod',['-R','a-w',d]);
}
for(const [name,fn] of [['systemctl',systemctl],['journalctl',journalctl],['systemd-analyze',systemdAnalyze]])writeFileSync(join(SBIN,name),script(fn),{mode:0o755});
writeFileSync(join(base,'holder.cjs'),`(${holder})();\n`);
export const SR=join(repo,'scripts','stack-release.sh');
// --- Per-test roots and helpers ---
export const sha=(f)=>createHash('sha256').update(readFileSync(f)).digest('hex');
export const lines=(f)=>existsSync(f)?readFileSync(f,'utf8').split('\n').filter(Boolean):[];
export const calls=(e)=>lines(join(e.sd,'calls.log'));
export const startsStops=(e,from)=>calls(e).slice(from).filter((c)=>/^(start|stop) /.test(c));
export const logLines=(e)=>lines(e.log).map((l)=>JSON.parse(l));
export const pointer=(e)=>existsSync(e.pointer)?JSON.parse(readFileSync(e.pointer,'utf8')):null;
export const units=(e)=>JSON.parse(readFileSync(join(e.sd,'units.json'),'utf8'));
export const snaps=(e)=>lines(join(e.sd,'snap.jsonl')).map((l)=>JSON.parse(l));
export const fault=(e,name,text='')=>writeFileSync(join(e.sd,'faults',name),text);
export const sysctl=(e,...a)=>execFileSync(join(SBIN,'systemctl'),['--user',...a],{env:e.env,encoding:'utf8'});
export const edit=(f,from,to)=>{const s=readFileSync(f,'utf8');assert.ok(s.includes(from),`${f} has no ${from}`);writeFileSync(f,s.replace(from,to))};
export const posts=(e)=>readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-'));
export const backupSum=(e,act)=>readFileSync(join(e.data,'backups','stack',act,'SHA256SUMS'),'utf8').split(' ')[0];
export const commitLines=(range)=>git(repo,'log','--oneline','--no-merges','--max-count=200',range).split('\n').map((l)=>` ${l}`);
// The units, the wrapper, the pointer and the store, by sha256.
export const fp=(e)=>({bus:sha(join(e.units,e.tmpl)),console:sha(join(e.units,e.con)),wrapper:sha(join(e.bin,e.wrapper)),pointer:existsSync(e.pointer)?sha(e.pointer):null,store:sha(e.store)});
// The schema version: readOnly while the holder has the store open, immutable when it is stopped.
export function storeVersion(file,held){
const db=held?new DatabaseSync(file,{readOnly:true}):new DatabaseSync(pathToFileURL(file).href+'?immutable=1',{readOnly:true});
try{return db.prepare("SELECT value FROM meta WHERE key='schema_version'").get().value}finally{db.close()}
}
// Every file and directory under p with size and mtime: T23 and T16 compare it before and after.
export function tree(p){
const out=[];
const walk=(q)=>{let st;try{st=statSync(q)}catch{return}out.push(st.isDirectory()?`${q}/`:`${q} ${st.size} ${st.mtimeMs}`);if(st.isDirectory())for(const n of readdirSync(q).sort())walk(join(q,n))};
walk(p);return out;
}
// T17: what a run may never remove. Under --units external confirm removes a set's AWAITING-UNITS
// and FAILED when it closes the activation (row 232, r6 A5 and R4); user mode writes neither.
const markers=/\/(AWAITING-UNITS|FAILED) /;
export const kept=(e)=>[...tree(join(e.data,'backups','stack')),...tree(join(e.state,'stack-rollback')).filter((l)=>!markers.test(l)),...posts(e).flatMap((n)=>tree(join(e.data,n))),...tree(COPIES)].map((l)=>l.replace(/ .*/,''));
// The business file (row 169): the committed example made loadable, as acme, with `plugins` when given.
export const EXAMPLE=readFileSync(join(repo,'packages','business','examples','mosaic-stack.example.json'),'utf8');
export function biz(e,plugins){
const b=JSON.parse(EXAMPLE);let n=0;
const walk=(o)=>{for(const [k,v] of Object.entries(o)){if(k==='botId')o[k]=++n;else if(v==='YYYY-MM-DD')o[k]='2027-01-01';else if(v&&typeof v==='object')walk(v)}};
walk(b);b.id='acme';if(plugins!==undefined)b.plugins=plugins;
const f=join(dirname(e.cfg),'businesses','acme.json');mkdirSync(dirname(f),{recursive:true});
writeFileSync(f,JSON.stringify(b,null,2)+'\n',{mode:0o600});chmodSync(f,0o600);
}
export async function mk(t){
const root=realpathSync(mkdtempSync(join(tmpdir(),'sr-')));
const e={root,data:join(root,'data'),bin:join(root,'bin'),sd:join(root,'sd'),units:join(root,'xcfg','systemd','user'),cfg:join(root,'cfg','config.json'),claude:join(root,'claude-bin','claude')};
// The default instance's names; named() sets NAME's (row 190).
Object.assign(e,{flags:[],bus:BUS,tmpl:'[email protected]',con:CON,wrapper:'mosaic'});
Object.assign(e,{state:join(e.data,'state'),pointer:join(e.data,'state','stack-active.json'),log:join(e.data,'state','stack-activation-log.jsonl'),store:join(e.data,'bus','bus.sqlite'),lock:join(e.data,'state','stack-release.lock')});
for(const d of [join(root,'home'),dirname(e.cfg),join(e.data,'bus'),e.bin,dirname(e.claude),join(e.sd,'faults'),e.units,join(root,'xdata')])mkdirSync(d,{recursive:true});
chmodSync(join(e.data,'bus'),0o700);
writeFileSync(e.cfg,JSON.stringify({configVersion:1,environment:'development',dataRoot:e.data,execution:{backend:'docker',provider:'zai',model:'glm-5.3-flash',adapter:'mock'}}));
biz(e);
writeFileSync(join(e.bin,'mosaic'),`#!/usr/bin/env bash\n${MARKER}\n[ "\${1:-}" = __lease-capability ] && exec "${e.bin}/mosaic-old" "$@"\nexec "/old/scripts/mosaic" "$@"\n`,{mode:0o755});
writeFileSync(join(e.bin,'mosaic-old'),'#!/bin/sh\nexit 0\n',{mode:0o755});
writeFileSync(e.claude,'#!/bin/sh\necho "9.9.9 (Claude Code fixture)"\n',{mode:0o755});
e.env={PATH:`${SBIN}:${dirname(NODE)}:/usr/bin:/bin`,HOME:join(root,'home'),XDG_CONFIG_HOME:join(root,'xcfg'),XDG_DATA_HOME:join(root,'xdata'),MOSAIC_CONFIG:e.cfg,SD_STATE:e.sd,SD_DATA:e.data,TMPDIR:tmpdir()};
const server=await healthz(e);
t.after(()=>{
for(const pid of lines(join(e.sd,'pids.log')).map(Number)){
try{if(readFileSync(`/proc/${pid}/cmdline`,'utf8').includes(join(base,'holder.cjs')))process.kill(pid,'SIGKILL')}catch{}
}
for(const s of [server,...(e.servers??[])])s.close();
execFileSync('chmod',['-R','u+w',root]);rmSync(root,{recursive:true,force:true});
});
return e;
}
// The Console's /healthz on e.port: 200 while the stand-in has e.con active and no healthz-<copy>
// fault names its copy.
export async function healthz(e){
const server=createServer((q,r)=>{
let st={};try{st=units(e)[e.con]||{}}catch{}
const ok=q.url==='/healthz'&&st.state==='active'&&!existsSync(join(e.sd,'faults',`healthz-${basename(st.wd||'')}`));
r.writeHead(ok?200:500);r.end();
});
server.listen(0,'127.0.0.1');await once(server,'listening');e.port=String(server.address().port);
return server;
}
// Row 190: stack instance NAME on e's host. It shares e's stand-ins, unit directory and $BIN, and has
// its own config and data root from bootstrap --instance, its own Console port and business acme.
// Its runs pass --instance NAME and no MOSAIC_CONFIG.
export async function named(e,name){
const env={...e.env};delete env.MOSAIC_CONFIG;
execFileSync('bash',[join(repo,'scripts','bootstrap.sh'),'--instance',name],{env,stdio:'ignore'});
const data=join(env.HOME,`.mosaic-${name}`);
const n={...e,env,data,cfg:join(env.HOME,'.config',`mosaic-${name}`,'config.json'),flags:['--instance',name],bus:`mosaic-bus-${name}@acme.service`,tmpl:`mosaic-bus-${name}@.service`,con:`mosaic-webui-${name}.service`,wrapper:`mosaic-${name}`,name};
Object.assign(n,{state:join(data,'state'),pointer:join(data,'state','stack-active.json'),log:join(data,'state','stack-activation-log.jsonl'),store:join(data,'bus','bus.sqlite'),lock:join(data,'state','stack-release.lock')});
mkdirSync(join(data,'bus'),{mode:0o700});
biz(n);
(e.servers??=[]).push(await healthz(n));
return n;
}
export async function run(e,...args){
const before=existsSync(e.data)?kept(e):[];
const ch=spawn('bash',[e.sr??SR,...args,...e.flags,'--copies',COPIES,'--bin',e.bin],{env:e.env,stdio:['ignore','pipe','pipe']});
let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d});
const timer=setTimeout(()=>ch.kill('SIGKILL'),120000);
const [code]=await once(ch,'close');clearTimeout(timer);
const now=new Set(kept(e));
for(const p of before)assert.ok(now.has(p),`T17: ${args.join(' ')} removed ${p}`);
return {code,out,err};
}
// The stack on copy `copy` for business acme, both units active. pointer: adopt (a real adopt and
// its log line), hand (written here, no log) or none. Snapshots start after the seed.
export async function seed(e,{copy=B,pm=false,store='3g',pointer:how='adopt'}={}){
const db=new DatabaseSync(e.store);
db.exec('PRAGMA journal_mode=WAL');
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
db.prepare("INSERT INTO meta VALUES ('schema_version', ?)").run(store);
for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`);
db.close();
const sh=(name,...a)=>execFileSync('bash',[join(repo,'scripts',name),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
sh('bus-service.sh','install',...e.flags,'--copy',copy,'--copies',COPIES,...(pm?['--pm']:[]),'--claude',e.claude);
sh('webui-service.sh','install',...e.flags,'--copy',copy,'--copies',COPIES,'--port',e.port,'--business','acme');
sh('stack-copy.sh','wrapper',copy,...e.flags,'--copies',COPIES,'--bin',e.bin);
// The seeded host started two minutes ago, so its boot status and PM run predate any run's start.
execFileSync(join(SBIN,'systemctl'),['--user','start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:'120'}});
sysctl(e,'start',e.con);
if(how==='adopt'){const r=await run(e,'adopt','acme');assert.equal(r.code,0,r.err)}
else if(how==='hand'){
mkdirSync(e.state,{recursive:true});
writeFileSync(e.pointer,JSON.stringify({copy,business:'acme',pm,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g',activation:'manual',at:'2026-10-18T00:00:00Z'})+'\n');
}
writeFileSync(join(e.sd,'snap.jsonl'),'');
}
export const rolledBack=(code,to=B)=>`stack-release: ${code}; rolled back to ${to}, which passed the gate\n`;
export const planBackup=(e)=>`backup: the run backs up the stopped store to ${e.data}/backups/stack/<activation>/ before any migration`;
export const planNoBackup='backup: none; the run rolls back before the backup';
export const planTail=(e,to,pm=false)=>[`bus: scripts/bus-service.sh install --copy ${to} --copies ${COPIES}${pm?' --pm':''} --claude ${e.claude}`,'proof: off','send: off',`Console: scripts/webui-service.sh install --copy ${to} --copies ${COPIES} --port ${e.port} --business acme`,`CLI: scripts/stack-copy.sh wrapper ${to} --copies ${COPIES} --bin ${e.bin}`];
+1 -416
View File
@@ -13,422 +13,7 @@ import {join,dirname,basename} from 'node:path';
import {fileURLToPath,pathToFileURL} from 'node:url';
import {once} from 'node:events';
import {ALGS} from '../../webui/src/sso-config.mjs';
const ROOT=fileURLToPath(new URL('../../..',import.meta.url)).replace(/\/$/,'');
const NODE=process.execPath;
const MARKER='# mosaic-cli wrapper (Q81, agents/sage/work/q81/mosaic-cli.sh)';
const BUS='[email protected]',CON='mosaic-webui.service';
const git=(cwd,...args)=>execFileSync('git',['-c','user.name=fixture','-c','user.email=fixture@invalid','-c','commit.gpgsign=false','-c','core.hooksPath=/dev/null','-C',cwd,...args],{encoding:'utf8'}).trim();
const script=(fn)=>`#!${NODE}\n(${fn})();\n`;
// --- Stand-ins. Each is serialized into its own file, so it uses only process and its own scope. ---
// systemctl --user: unit state in $SD_STATE/units.json, every call in calls.log. A bus start runs
// holder.cjs (the store held open in WAL mode, writer.lock, host.json) and, with --pm in the unit's
// ExecStart, writes the `launched` journal line and the launches entry unless a pm-<copy> fault
// names another case. Faults are files in $SD_STATE/faults named for the copy they hit. Units of a
// named instance (row 190) run on their own data root and host files beside the default's.
function systemctl(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process'),crypto=process.getBuiltinModule('node:crypto');
const sd=process.env.SD_STATE,data=process.env.SD_DATA,unitDir=path.join(process.env.XDG_CONFIG_HOME,'systemd','user');
const args=process.argv.slice(2).filter((a)=>a!=='--user'),[verb,...rest]=args;
fs.appendFileSync(path.join(sd,'calls.log'),args.join(' ')+'\n');
const faultFile=(n)=>path.join(sd,'faults',n),fault=(n)=>fs.existsSync(faultFile(n));
const stFile=path.join(sd,'units.json');
const load=()=>{try{return JSON.parse(fs.readFileSync(stFile,'utf8'))}catch{return {}}};
const save=(s)=>{fs.writeFileSync(stFile+'.tmp',JSON.stringify(s));fs.renameSync(stFile+'.tmp',stFile)};
const sleep=(ms)=>Atomics.wait(new Int32Array(new SharedArrayBuffer(4)),0,0,ms);
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
// Row 190: a bus unit is the default's mosaic-bus@B or instance NAME's mosaic-bus-NAME@B; %i is B.
const isBus=(u)=>/^mosaic-bus(-[a-z][a-z0-9]{1,15})?@[a-z0-9-]+\.service$/.test(u),inst=(u)=>u.slice(u.indexOf('@')+1,-'.service'.length);
const unitFile=(u)=>{
const main=isBus(u)?u.replace(/@[^@]*$/,'@.service'):u,file=path.join(unitDir,main);
if(!fs.existsSync(file))return null;
// The drop-in directories in systemd's order: service.d, the name cut after each hyphen, then
// the template's and the unit's own.
const stem=main.slice(0,main.search(/@?\.service$/)),dirs=['service.d'],drops=[];
for(let i=0;i<stem.length;i++)if(stem[i]==='-')dirs.push(stem.slice(0,i+1)+'.service.d');
for(const d of [...dirs,...(isBus(u)?[main+'.d',u+'.d']:[u+'.d'])]){const dir=path.join(unitDir,d);if(fs.existsSync(dir))for(const f of fs.readdirSync(dir).sort())if(f.endsWith('.conf'))drops.push(path.join(dir,f))}
const r={wd:'',exec:'',env:[],drops};
for(const f of [file,...drops])for(let line of fs.readFileSync(f,'utf8').split('\n')){
line=line.replace(/%i/g,isBus(u)?inst(u):'').replace(/%h/g,process.env.HOME);
const m=/^(WorkingDirectory|ExecStart|Environment)=(.*)$/.exec(line);if(!m)continue;
if(m[1]==='WorkingDirectory')r.wd=m[2];else if(m[1]==='ExecStart'){if(m[2])r.exec=m[2]}else r.env.push(m[2]);
}
return r;
};
// A named instance's unit sets MOSAIC_CONFIG, and its bus works on that config's data root. A unit
// without it is the default's, on $SD_DATA. Each data root has its own host and launches files.
const dataOf=(f)=>{const c=f&&f.env.find((v)=>v.startsWith('MOSAIC_CONFIG='));return c?JSON.parse(fs.readFileSync(c.slice('MOSAIC_CONFIG='.length),'utf8')).dataRoot:data};
const perData=(d,n)=>path.join(sd,d===data?`${n}.json`:`${n}-${path.basename(d)}.json`);
const snap=(ev,d)=>{
const dir=path.join(d,'bus'),files={};
try{for(const n of fs.readdirSync(dir).sort()){const p=path.join(dir,n),st=fs.statSync(p);if(st.isFile())files[n]=[st.size,st.mtimeMs,crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')]}}catch{}
fs.appendFileSync(path.join(sd,'snap.jsonl'),JSON.stringify({ev,files})+'\n');
};
const pm=(unit,d,id,pid,off)=>{
let mode='';try{mode=fs.readFileSync(faultFile(`pm-${id}`),'utf8').trim()}catch{}
const b=inst(unit),now=Date.now()-off*1000,epoch=Math.floor(now/1000),at=new Date(now).toISOString(),run=`run-${pid}`;
const line=(text)=>fs.appendFileSync(path.join(sd,'journal.jsonl'),JSON.stringify({epoch,unit,text})+'\n');
const ok=()=>line(`launched pm-${b}: run ${run}, claude opus`);
const lf=perData(d,'launches');let list=[];try{list=JSON.parse(fs.readFileSync(lf,'utf8'))}catch{}
const entry=(o)=>list.push({run,business:b,live:true,startedAt:at,...o});
line(`bus host up: business ${b}, socket /fixture/bus.sock, notifier off`);
switch(mode){
case 'none':break;
case 'two':ok();line(`launched pm-${b}: run ${run}b, claude opus`);entry({});break;
case 'badword':line(`launched pm-${b}: job ${run}, claude opus`);entry({});break;
case 'notlisted':ok();break;
case 'early':ok();entry({startedAt:new Date(now-3600e3).toISOString()});break;
case 'human':entry({run:`human-${pid}`});break;
case 'other':ok();entry({business:'other'});break;
case 'dead':ok();entry({live:false});break;
default:ok();entry({});
}
fs.writeFileSync(lf,JSON.stringify(list));
};
if(verb==='show'){
const u=rest[0],props=[];for(let i=1;i<rest.length;i++)if(rest[i]==='-p')props.push(rest[++i]);
const f=unitFile(u),st=load()[u]||{},id=f?path.basename(f.wd):'';
for(const p of props){
if(p==='LoadState')console.log(`LoadState=${f?'loaded':'not-found'}`);
else if(p==='WorkingDirectory')console.log(`WorkingDirectory=${f?f.wd+(!isBus(u)&&fault(`check-${id}`)?'-moved':''):''}`);
else if(p==='ExecStart')console.log(f&&f.exec?`ExecStart={ path=${f.exec.split(' ')[0]} ; argv[]=${f.exec} ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }`:'ExecStart=');
else if(p==='Environment')console.log(`Environment=${f?f.env.join(' '):''}`);
else if(p==='MainPID')console.log(`MainPID=${st.pid||0}`);
else if(p==='DropInPaths')console.log(`DropInPaths=${f?f.drops.join(' '):''}`);
}
process.exit(0);
}
if(verb==='is-active'){
const q=rest[0]==='--quiet',u=q?rest[1]:rest[0];let s=(load()[u]||{}).state||'inactive';
// bus-stops-after N (T29): the next N bus checks answer the real state, every later one answers
// inactive, and units.json stays as it is, as if the bus stopped between two checks.
if(isBus(u)&&fault('bus-stops-after')){const n=Number(fs.readFileSync(faultFile('bus-stops-after'),'utf8'));if(n>0)fs.writeFileSync(faultFile('bus-stops-after'),String(n-1));else s='inactive'}
if(!q)console.log(s);process.exit(s==='active'?0:3);
}
if(verb==='is-enabled'){console.log('disabled');process.exit(1)}
if(verb==='daemon-reload')process.exit(0);
if(verb==='list-units'){
if(fault('list-fails'))process.exit(1);
const pats=rest.filter((a)=>!a.startsWith('--')).map((p)=>new RegExp('^'+p.replace(/\./g,'\\.').replace(/\*/g,'.*')+'$'));
for(const [u,s] of Object.entries(load()))if(pats.some((re)=>re.test(u)))console.log(`${u} loaded ${s.state} ${s.state==='active'?'running':'dead'} fixture`);
process.exit(0);
}
if(verb==='start'){
const u=rest[0],f=unitFile(u);if(!f){console.error(`Unit ${u} not found.`);process.exit(5)}
const s=load(),id=path.basename(f.wd),d=dataOf(f);
if(!isBus(u)){s[u]={state:'active',pid:null,wd:f.wd};save(s);process.exit(0)}
if(fault(`start-dies-${id}`)){s[u]={state:'failed',pid:null,wd:f.wd};save(s);process.exit(0)}
// Row 185 (T48): the copy's start writes one event before the holder opens the store, so the
// store differs from a backup the run took before the start. The close leaves no -wal.
if(fault(`start-writes-row-${id}`)){
const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(d,'bus','bus.sqlite'));
db.prepare('INSERT INTO events (at) VALUES (?)').run('2999-01-01T00:00:00.000Z');db.close();
}
snap('start',d);
const off=process.env.SD_CLOCK_OFFSET||'0';
// The holder must not inherit stack-release.sh's lock (fd 9) or any other fd above 2.
const close='for fd in /proc/self/fd/*; do n=${fd##*/}; [ "$n" -gt 2 ] && eval "exec $n>&-" 2>/dev/null; done; exec "$@"';
const ch=cp.spawn('bash',['-c',close,'holder',process.execPath,path.join(__dirname,'..','holder.cjs'),d,inst(u),sd,off,perData(d,'host')],{detached:true,stdio:'ignore'});
ch.unref();
fs.appendFileSync(path.join(sd,'pids.log'),ch.pid+'\n');
const ready=path.join(sd,`ready-${ch.pid}`),end=Date.now()+10000;
while(!fs.existsSync(ready)){if(Date.now()>end){console.error('fixture holder did not start');process.exit(1)}sleep(20)}
s[u]={state:'active',pid:ch.pid,wd:f.wd};save(s);
if(f.exec.split(' ').slice(1).includes('--pm'))pm(u,d,id,ch.pid,Number(off));
process.exit(0);
}
if(verb==='stop'){
const u=rest[0],s=load(),st=s[u];
if(st&&st.pid){
try{process.kill(st.pid,'SIGTERM')}catch{}
const end=Date.now()+10000;while(alive(st.pid)&&Date.now()<end)sleep(20);
if(alive(st.pid))try{process.kill(st.pid,'SIGKILL')}catch{}
}
if(st){s[u]={...st,state:'inactive',pid:null};save(s)}
if(isBus(u)){
const d=dataOf(unitFile(u));
// One-shot: the stop leaves an empty -wal or -shm beside the store, as an unclean close would.
for(const [name,ext] of [['stop-leaves-wal','-wal'],['stop-leaves-shm','-shm']])if(fault(name)){fs.rmSync(faultFile(name));fs.writeFileSync(path.join(d,'bus','bus.sqlite'+ext),'')}
// One-shot (T45): the stop writes one row after any STAMP into the table the fault names, as a
// write the bus made between rollback's count and its stop would. The close leaves no -wal.
if(fault('stop-writes-row')){
const t=fs.readFileSync(faultFile('stop-writes-row'),'utf8');fs.rmSync(faultFile('stop-writes-row'));
const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(d,'bus','bus.sqlite'));
db.exec(`CREATE TABLE IF NOT EXISTS ${t} (seq INTEGER PRIMARY KEY, at TEXT NOT NULL)`);
db.prepare(`INSERT INTO ${t} (at) VALUES (?)`).run('2999-01-01T00:00:00.000Z');db.close();
}
snap('stop',d);
}
process.exit(0);
}
console.error(`fixture systemctl: unsupported: ${args.join(' ')}`);process.exit(1);
}
// journalctl --user -u UNIT --since @EPOCH -o cat. A bare local time gets exit 1, as M35 needs.
function journalctl(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
const sd=process.env.SD_STATE,a=process.argv.slice(2);let unit='',since='';
for(let i=0;i<a.length;i++){if(a[i]==='-u')unit=a[++i];else if(a[i]==='--since')since=a[++i]}
fs.appendFileSync(path.join(sd,'journal-since.log'),since+'\n');
if(fs.existsSync(path.join(sd,'faults','journal-fails')))process.exit(1);
let from;
if(/^@\d+$/.test(since))from=Number(since.slice(1));
else{const m=/^(\d{4}-\d\d-\d\d) (\d\d:\d\d:\d\d) UTC$/.exec(since);if(!m)process.exit(1);from=Date.parse(`${m[1]}T${m[2]}Z`)/1000}
let lines=[];try{lines=fs.readFileSync(path.join(sd,'journal.jsonl'),'utf8').split('\n').filter(Boolean).map((l)=>JSON.parse(l))}catch{}
for(const o of lines)if(o.unit===unit&&o.epoch>=from)console.log(o.text);
}
function systemdAnalyze(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
if(process.argv[2]!=='--user'||process.argv[3]!=='unit-paths')process.exit(1);
console.log(path.join(process.env.XDG_CONFIG_HOME,'systemd','user'));
// The unit-path fault adds its text as one more path, as T28's relative path needs.
const extra=path.join(process.env.SD_STATE,'faults','unit-path');
if(fs.existsSync(extra))console.log(fs.readFileSync(extra,'utf8'));
}
// The bus host: holds the store open in WAL mode until SIGTERM, as the broker does. It writes its
// status to the host file systemctl names for its data root.
function holder(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
const [data,business,sd,offset,host]=process.argv.slice(2),bus=path.join(data,'bus');
fs.mkdirSync(bus,{recursive:true,mode:0o700});
const db=new DatabaseSync(path.join(bus,'bus.sqlite'));
if(!db.prepare("SELECT count(*) AS n FROM sqlite_master WHERE name='meta'").get().n){
db.exec('PRAGMA journal_mode=WAL');
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
db.prepare("INSERT INTO meta VALUES ('schema_version', '3g')").run();
}
fs.writeFileSync(path.join(bus,'writer.lock'),String(process.pid));
const at=new Date(Date.now()-Number(offset||0)*1000).toISOString();
fs.writeFileSync(host+'.tmp',JSON.stringify({host:{business,pid:process.pid,live:true,startedAt:at,notifier:'ok'},boot:{ok:true,business,at}}));
fs.renameSync(host+'.tmp',host);
process.on('SIGTERM',()=>{db.close();fs.rmSync(path.join(bus,'writer.lock'),{force:true});process.exit(0)});
fs.writeFileSync(path.join(sd,`ready-${process.pid}`),'');
setInterval(()=>{},1<<30);
}
// A copy's scripts/mosaic: bus status and launches list from what the holder and systemctl wrote
// for the data root of the config it runs under (row 190), the default's being $SD_DATA.
function mosaicStub(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
const id=path.basename(path.resolve(__dirname,'..')),sd=process.env.SD_STATE,a=process.argv.slice(2).join(' ');
const fault=(n)=>fs.existsSync(path.join(sd,'faults',n));
let d=process.env.SD_DATA;try{d=JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8')).dataRoot}catch{}
const perData=(n)=>path.join(sd,d===process.env.SD_DATA?`${n}.json`:`${n}-${path.basename(d)}.json`);
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
if(a==='bus status --json'){
if(fault(`status-${id}`))process.exit(1);
let s={host:null,boot:null};
try{s=JSON.parse(fs.readFileSync(perData('host'),'utf8'));s.host.live=alive(s.host.pid)}catch{}
console.log(JSON.stringify(s));
}else if(a==='bus status')console.log(`bus: fixture status for ${id}`);
else if(a==='launches list --json'){
if(fault(`launches-${id}`))process.exit(1);
let l=[];try{l=JSON.parse(fs.readFileSync(perData('launches'),'utf8'))}catch{}
console.log(JSON.stringify(l));
}else{console.error(`fixture mosaic: ${a}`);process.exit(4)}
}
// A copy's migrate-schema.mjs: records the backups it saw, then sets 3h, or 3h-partial and exit 1.
function migrateStub(){
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
const data=process.argv[2],sd=process.env.SD_STATE,dir=path.join(data,'backups','stack');
const backups=fs.existsSync(dir)?fs.readdirSync(dir).sort().map((name)=>({name,sums:fs.existsSync(path.join(dir,name,'SHA256SUMS'))})):[];
fs.writeFileSync(path.join(sd,'migrate.json'),JSON.stringify({backups}));
const fail=fs.existsSync(path.join(sd,'faults','migrate-fails'));
const db=new DatabaseSync(path.join(data,'bus','bus.sqlite'));
db.prepare("UPDATE meta SET value=? WHERE key='schema_version'").run(fail?'3h-partial':'3h');db.close();
if(fail)process.exit(1);
console.log('fixture migrate: 3h');
}
// --- The module fixture: one repo, its copies and the stand-ins, shared by every test. ---
const base=realpathSync(mkdtempSync(join(tmpdir(),'stack-release-')));
after(()=>{execFileSync('chmod',['-R','u+w',base]);rmSync(base,{recursive:true,force:true})});
const repo=join(base,'repo'),COPIES=join(base,'copies'),SBIN=join(base,'sbin');
mkdirSync(repo);mkdirSync(COPIES);mkdirSync(SBIN);
writeFileSync(join(base,'package.json'),'{"type":"commonjs"}\n');
const tracked=execFileSync('git',['-C',ROOT,'ls-files','-co','--exclude-standard','--','scripts','packages','adapters','roles','RELEASE','package.json','package-lock.json',':!packages/*/tests',':!scripts/foundation'],{encoding:'utf8'}).trim().split('\n');
for(const f of tracked)if(existsSync(join(ROOT,f)))cpSync(join(ROOT,f),join(repo,f));
const pkg={name:'copy-fixture',version:'1.0.0'};
writeFileSync(join(repo,'package.json'),JSON.stringify(pkg));
writeFileSync(join(repo,'package-lock.json'),JSON.stringify({name:pkg.name,version:pkg.version,lockfileVersion:3,packages:{'':pkg}}));
git(repo,'init','-q','-b','refactor');git(repo,'config','gc.auto','0');git(repo,'add','.');git(repo,'commit','-qm','fixture');
const commit=(msg)=>{git(repo,'commit','-q','--allow-empty','-m',msg);return git(repo,'rev-parse','HEAD')};
// refactor: F A B C1 C, then M merging side branch S (from A). U sits on a branch nothing reaches.
const F=git(repo,'rev-parse','HEAD'),A=commit('row 901: A'),B=commit('row 902: B'),C1=commit('row 903: C1'),C=commit('row 904: C');
git(repo,'checkout','-q','-b','side',A);const S=commit('row 905: S');
git(repo,'checkout','-q','-b','unreached',A);const U=commit('row 906: U');
git(repo,'checkout','-q','refactor');git(repo,'merge','-q','--no-ff','-m','merge side','side');
const M=git(repo,'rev-parse','HEAD');git(repo,'update-ref','refs/remotes/origin/refactor',M);
// `to` false leaves out the `to:` lines (C1); a string puts that value in each one (A). The --plan
// steps line can't be read from either (T27).
const storeText=(v,froms,to=true)=>`export const SCHEMA_VERSION = '${v}';\nexport const MIGRATIONS = [\n${froms.map((f,i)=>` {\n from: '${f}',\n${to===false?'':` to: '${to===true?froms[i+1]??v:to}',\n`} run() {},\n },\n`).join('')}];\n`;
// Plug-in kinds (row 169): C1 and C know the harness kind, as a copy at or past c216f47db does. The
// rest have it under another name, as ca15d73e9 has no harness kind. Every copy has the real business
// loader and roles, which the tool imports to read the business file.
const KINDS_NEW=readFileSync(join(repo,'packages','plugins','src','kinds.mjs'),'utf8');
const KINDS_OLD=KINDS_NEW.replace('\n harness: Object.freeze({','\n "harness-before-c216f47db": Object.freeze({');
assert.notEqual(KINDS_OLD,KINDS_NEW);
// Row 200 (#1698): C1 and C take --sso and have the row 192 loader and checkConsole; C's checker
// refuses RS256, a stand-in for a target whose rules differ from the repository's. B has the loader
// from before row 192, which refuses a `console` key. The other copies have no loader.
const LOADER=readFileSync(join(repo,'scripts','mosaic-config.mjs'),'utf8'),SSO_CONFIG=readFileSync(join(repo,'packages','webui','src','sso-config.mjs'),'utf8');
const LOADER_OLD=LOADER.replace('["configVersion", "environment", "dataRoot", "execution", "console"]','["configVersion", "environment", "dataRoot", "execution"]');
const SSO_CONFIG_C=SSO_CONFIG.replace("'ES256', 'RS256',","'ES256',");
assert.notEqual(LOADER_OLD,LOADER);assert.notEqual(SSO_CONFIG_C,SSO_CONFIG);
const ssoFiles=(id)=>id===B?[['scripts/mosaic-config.mjs',LOADER_OLD]]:[C1,C].includes(id)?[['scripts/mosaic-config.mjs',LOADER],['packages/webui/src/sso-config.mjs',id===C?SSO_CONFIG_C:SSO_CONFIG]]:[];
for(const [id,store] of [[F,'// a copy from before SCHEMA_VERSION\n'],[A,storeText('3g',['3f'],'3x')],[B,storeText('3g',['3f'])],[C1,storeText('3g',['3f'],false)],[C,storeText('3h',['3f','3g'])],[S,storeText('3g',['3f'])],[U,storeText('3g',['3f'])]]){
const d=join(COPIES,id);
for(const [f,text,mode] of [['scripts/mosaic',script(mosaicStub),0o755],['packages/webui/src/cli.mjs',[B,C1,C].includes(id)?`// fixture Console, never run\n// serve accepts '--tester-port'\n// serve accepts '--proof-header' (row 178)\n${[C1,C].includes(id)?"// serve accepts '--sso' (row 192)\n":''}`:'// fixture Console before tester listener, never run\n',0o644],['packages/bus/src/store.mjs',store,0o644],['packages/bus/src/migrate-schema.mjs',`(${migrateStub})();\n`,0o644],['packages/plugins/src/kinds.mjs',[C1,C].includes(id)?KINDS_NEW:KINDS_OLD,0o644],...ssoFiles(id).map(([f,text])=>[f,text,0o644])]){
mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text,{mode});
}
for(const p of ['packages/business/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true});
execFileSync('chmod',['-R','a-w',d]);
}
for(const [name,fn] of [['systemctl',systemctl],['journalctl',journalctl],['systemd-analyze',systemdAnalyze]])writeFileSync(join(SBIN,name),script(fn),{mode:0o755});
writeFileSync(join(base,'holder.cjs'),`(${holder})();\n`);
const SR=join(repo,'scripts','stack-release.sh');
// --- Per-test roots and helpers ---
const sha=(f)=>createHash('sha256').update(readFileSync(f)).digest('hex');
const lines=(f)=>existsSync(f)?readFileSync(f,'utf8').split('\n').filter(Boolean):[];
const calls=(e)=>lines(join(e.sd,'calls.log'));
const startsStops=(e,from)=>calls(e).slice(from).filter((c)=>/^(start|stop) /.test(c));
const logLines=(e)=>lines(e.log).map((l)=>JSON.parse(l));
const pointer=(e)=>existsSync(e.pointer)?JSON.parse(readFileSync(e.pointer,'utf8')):null;
const units=(e)=>JSON.parse(readFileSync(join(e.sd,'units.json'),'utf8'));
const snaps=(e)=>lines(join(e.sd,'snap.jsonl')).map((l)=>JSON.parse(l));
const fault=(e,name,text='')=>writeFileSync(join(e.sd,'faults',name),text);
const sysctl=(e,...a)=>execFileSync(join(SBIN,'systemctl'),['--user',...a],{env:e.env,encoding:'utf8'});
const edit=(f,from,to)=>{const s=readFileSync(f,'utf8');assert.ok(s.includes(from),`${f} has no ${from}`);writeFileSync(f,s.replace(from,to))};
const posts=(e)=>readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-'));
const backupSum=(e,act)=>readFileSync(join(e.data,'backups','stack',act,'SHA256SUMS'),'utf8').split(' ')[0];
const commitLines=(range)=>git(repo,'log','--oneline','--no-merges','--max-count=200',range).split('\n').map((l)=>` ${l}`);
// The units, the wrapper, the pointer and the store, by sha256.
const fp=(e)=>({bus:sha(join(e.units,e.tmpl)),console:sha(join(e.units,e.con)),wrapper:sha(join(e.bin,e.wrapper)),pointer:existsSync(e.pointer)?sha(e.pointer):null,store:sha(e.store)});
// The schema version: readOnly while the holder has the store open, immutable when it is stopped.
function storeVersion(file,held){
const db=held?new DatabaseSync(file,{readOnly:true}):new DatabaseSync(pathToFileURL(file).href+'?immutable=1',{readOnly:true});
try{return db.prepare("SELECT value FROM meta WHERE key='schema_version'").get().value}finally{db.close()}
}
// Every file and directory under p with size and mtime: T23 and T16 compare it before and after.
function tree(p){
const out=[];
const walk=(q)=>{let st;try{st=statSync(q)}catch{return}out.push(st.isDirectory()?`${q}/`:`${q} ${st.size} ${st.mtimeMs}`);if(st.isDirectory())for(const n of readdirSync(q).sort())walk(join(q,n))};
walk(p);return out;
}
// T17: what a run may never remove.
const kept=(e)=>[...tree(join(e.data,'backups','stack')),...tree(join(e.state,'stack-rollback')),...posts(e).flatMap((n)=>tree(join(e.data,n))),...tree(COPIES)].map((l)=>l.replace(/ .*/,''));
// The business file (row 169): the committed example made loadable, as acme, with `plugins` when given.
const EXAMPLE=readFileSync(join(repo,'packages','business','examples','mosaic-stack.example.json'),'utf8');
function biz(e,plugins){
const b=JSON.parse(EXAMPLE);let n=0;
const walk=(o)=>{for(const [k,v] of Object.entries(o)){if(k==='botId')o[k]=++n;else if(v==='YYYY-MM-DD')o[k]='2027-01-01';else if(v&&typeof v==='object')walk(v)}};
walk(b);b.id='acme';if(plugins!==undefined)b.plugins=plugins;
const f=join(dirname(e.cfg),'businesses','acme.json');mkdirSync(dirname(f),{recursive:true});
writeFileSync(f,JSON.stringify(b,null,2)+'\n',{mode:0o600});chmodSync(f,0o600);
}
async function mk(t){
const root=realpathSync(mkdtempSync(join(tmpdir(),'sr-')));
const e={root,data:join(root,'data'),bin:join(root,'bin'),sd:join(root,'sd'),units:join(root,'xcfg','systemd','user'),cfg:join(root,'cfg','config.json'),claude:join(root,'claude-bin','claude')};
// The default instance's names; named() sets NAME's (row 190).
Object.assign(e,{flags:[],bus:BUS,tmpl:'[email protected]',con:CON,wrapper:'mosaic'});
Object.assign(e,{state:join(e.data,'state'),pointer:join(e.data,'state','stack-active.json'),log:join(e.data,'state','stack-activation-log.jsonl'),store:join(e.data,'bus','bus.sqlite'),lock:join(e.data,'state','stack-release.lock')});
for(const d of [join(root,'home'),dirname(e.cfg),join(e.data,'bus'),e.bin,dirname(e.claude),join(e.sd,'faults'),e.units,join(root,'xdata')])mkdirSync(d,{recursive:true});
chmodSync(join(e.data,'bus'),0o700);
writeFileSync(e.cfg,JSON.stringify({configVersion:1,environment:'development',dataRoot:e.data,execution:{backend:'docker',provider:'zai',model:'glm-5.3-flash',adapter:'mock'}}));
biz(e);
writeFileSync(join(e.bin,'mosaic'),`#!/usr/bin/env bash\n${MARKER}\n[ "\${1:-}" = __lease-capability ] && exec "${e.bin}/mosaic-old" "$@"\nexec "/old/scripts/mosaic" "$@"\n`,{mode:0o755});
writeFileSync(join(e.bin,'mosaic-old'),'#!/bin/sh\nexit 0\n',{mode:0o755});
writeFileSync(e.claude,'#!/bin/sh\necho "9.9.9 (Claude Code fixture)"\n',{mode:0o755});
e.env={PATH:`${SBIN}:${dirname(NODE)}:/usr/bin:/bin`,HOME:join(root,'home'),XDG_CONFIG_HOME:join(root,'xcfg'),XDG_DATA_HOME:join(root,'xdata'),MOSAIC_CONFIG:e.cfg,SD_STATE:e.sd,SD_DATA:e.data,TMPDIR:tmpdir()};
const server=await healthz(e);
t.after(()=>{
for(const pid of lines(join(e.sd,'pids.log')).map(Number)){
try{if(readFileSync(`/proc/${pid}/cmdline`,'utf8').includes(join(base,'holder.cjs')))process.kill(pid,'SIGKILL')}catch{}
}
for(const s of [server,...(e.servers??[])])s.close();
execFileSync('chmod',['-R','u+w',root]);rmSync(root,{recursive:true,force:true});
});
return e;
}
// The Console's /healthz on e.port: 200 while the stand-in has e.con active and no healthz-<copy>
// fault names its copy.
async function healthz(e){
const server=createServer((q,r)=>{
let st={};try{st=units(e)[e.con]||{}}catch{}
const ok=q.url==='/healthz'&&st.state==='active'&&!existsSync(join(e.sd,'faults',`healthz-${basename(st.wd||'')}`));
r.writeHead(ok?200:500);r.end();
});
server.listen(0,'127.0.0.1');await once(server,'listening');e.port=String(server.address().port);
return server;
}
// Row 190: stack instance NAME on e's host. It shares e's stand-ins, unit directory and $BIN, and has
// its own config and data root from bootstrap --instance, its own Console port and business acme.
// Its runs pass --instance NAME and no MOSAIC_CONFIG.
async function named(e,name){
const env={...e.env};delete env.MOSAIC_CONFIG;
execFileSync('bash',[join(repo,'scripts','bootstrap.sh'),'--instance',name],{env,stdio:'ignore'});
const data=join(env.HOME,`.mosaic-${name}`);
const n={...e,env,data,cfg:join(env.HOME,'.config',`mosaic-${name}`,'config.json'),flags:['--instance',name],bus:`mosaic-bus-${name}@acme.service`,tmpl:`mosaic-bus-${name}@.service`,con:`mosaic-webui-${name}.service`,wrapper:`mosaic-${name}`,name};
Object.assign(n,{state:join(data,'state'),pointer:join(data,'state','stack-active.json'),log:join(data,'state','stack-activation-log.jsonl'),store:join(data,'bus','bus.sqlite'),lock:join(data,'state','stack-release.lock')});
mkdirSync(join(data,'bus'),{mode:0o700});
biz(n);
(e.servers??=[]).push(await healthz(n));
return n;
}
async function run(e,...args){
const before=existsSync(e.data)?kept(e):[];
const ch=spawn('bash',[e.sr??SR,...args,...e.flags,'--copies',COPIES,'--bin',e.bin],{env:e.env,stdio:['ignore','pipe','pipe']});
let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d});
const timer=setTimeout(()=>ch.kill('SIGKILL'),120000);
const [code]=await once(ch,'close');clearTimeout(timer);
const now=new Set(kept(e));
for(const p of before)assert.ok(now.has(p),`T17: ${args.join(' ')} removed ${p}`);
return {code,out,err};
}
// The stack on copy `copy` for business acme, both units active. pointer: adopt (a real adopt and
// its log line), hand (written here, no log) or none. Snapshots start after the seed.
async function seed(e,{copy=B,pm=false,store='3g',pointer:how='adopt'}={}){
const db=new DatabaseSync(e.store);
db.exec('PRAGMA journal_mode=WAL');
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
db.prepare("INSERT INTO meta VALUES ('schema_version', ?)").run(store);
for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`);
db.close();
const sh=(name,...a)=>execFileSync('bash',[join(repo,'scripts',name),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
sh('bus-service.sh','install',...e.flags,'--copy',copy,'--copies',COPIES,...(pm?['--pm']:[]),'--claude',e.claude);
sh('webui-service.sh','install',...e.flags,'--copy',copy,'--copies',COPIES,'--port',e.port,'--business','acme');
sh('stack-copy.sh','wrapper',copy,...e.flags,'--copies',COPIES,'--bin',e.bin);
// The seeded host started two minutes ago, so its boot status and PM run predate any run's start.
execFileSync(join(SBIN,'systemctl'),['--user','start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:'120'}});
sysctl(e,'start',e.con);
if(how==='adopt'){const r=await run(e,'adopt','acme');assert.equal(r.code,0,r.err)}
else if(how==='hand'){
mkdirSync(e.state,{recursive:true});
writeFileSync(e.pointer,JSON.stringify({copy,business:'acme',pm,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g',activation:'manual',at:'2026-10-18T00:00:00Z'})+'\n');
}
writeFileSync(join(e.sd,'snap.jsonl'),'');
}
const rolledBack=(code,to=B)=>`stack-release: ${code}; rolled back to ${to}, which passed the gate\n`;
const planBackup=(e)=>`backup: the run backs up the stopped store to ${e.data}/backups/stack/<activation>/ before any migration`;
const planNoBackup='backup: none; the run rolls back before the backup';
const planTail=(e,to,pm=false)=>[`bus: scripts/bus-service.sh install --copy ${to} --copies ${COPIES}${pm?' --pm':''} --claude ${e.claude}`,'proof: off','send: off',`Console: scripts/webui-service.sh install --copy ${to} --copies ${COPIES} --port ${e.port} --business acme`,`CLI: scripts/stack-copy.sh wrapper ${to} --copies ${COPIES} --bin ${e.bin}`];
import {A,B,BUS,C,C1,CON,COPIES,EXAMPLE,F,KINDS_NEW,KINDS_OLD,M,MARKER,NODE,ROOT,S,SBIN,SR,backupSum,base,biz,calls,commit,commitLines,edit,fault,fp,git,healthz,holder,journalctl,kept,lines,logLines,migrateStub,mk,mosaicStub,named,pkg,planBackup,planNoBackup,planTail,pointer,posts,repo,rolledBack,run,script,seed,sha,snaps,startsStops,storeText,storeVersion,sysctl,systemctl,systemdAnalyze,tracked,tree,U,units} from './stack-release-fixture.mjs';
// --- The T table ---
+62 -13
View File
@@ -3,21 +3,25 @@
# copy creates files only; wrapper is a separately Mos-gated live step.
# wrapper and check take --instance NAME (row 190): the units mosaic-bus-NAME@ and mosaic-webui-NAME
# and the wrapper $BIN/mosaic-NAME, which sets NAME's MOSAIC_CONFIG. check needs --business for NAME.
# Row 232: check --units external reads root's system units. copy --bundle makes the copy from a git
# bundle on a host with no checkout, and wrapper --create writes the first wrapper on a new host.
set -euo pipefail
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
COPIES="${XDG_DATA_HOME:-$HOME/.local/share}/mosaic-stack/copies"
BIN="$HOME/.npm-global/bin"
BUSINESS=mosaic-stack BUSINESS_SET=0
ALLOW_WORKTREE=0
ALLOW_WORKTREE=0 UNITS=user BUNDLE="" CREATE=0 CLEAN=""
MARKER='# mosaic-cli wrapper (Q81, agents/sage/work/q81/mosaic-cli.sh)'
usage() { echo 'usage: stack-copy.sh copy COMMIT [--copies DIR] [--allow-worktree] | wrapper ID [--instance NAME] [--copies DIR] [--bin DIR] | check [--instance NAME] [--copies DIR] [--bin DIR] [--business ID]' >&2; exit "${1:-4}"; }
usage() { echo 'usage: stack-copy.sh copy COMMIT [--copies DIR] [--allow-worktree] | copy --bundle FILE SHA [--copies DIR] | wrapper ID [--instance NAME] [--copies DIR] [--bin DIR] [--create] | check [--instance NAME] [--copies DIR] [--bin DIR] [--business ID] [--units user|external]' >&2; exit "${1:-4}"; }
die() { echo "stack-copy: $*" >&2; exit 1; }
plain_path() { case "$1" in /*) ;; *) die "$2 must be an absolute path: $1" ;; esac; case "$1" in *[!A-Za-z0-9/._+-]*) die "$2 contains a character the unit cannot carry: $1" ;; esac; }
parse() { while [ $# -gt 0 ]; do case "$1" in
--copies) [ $# -ge 2 ] || usage; COPIES="$2"; shift 2;;
--bin) [ "$CMD" != copy ] && [ $# -ge 2 ] || usage; BIN="$2";shift 2;;
--business) [ "$CMD" = check ] && [ $# -ge 2 ] || usage; BUSINESS="$2";BUSINESS_SET=1;shift 2;;
--allow-worktree) [ "$CMD" = copy ] || usage; ALLOW_WORKTREE=1;shift;;
--allow-worktree) [ "$CMD" = copy ] && [ -z "$BUNDLE" ] || usage; ALLOW_WORKTREE=1;shift;;
--units) [ "$CMD" = check ] && [ $# -ge 2 ] || usage; case "$2" in user|external) UNITS="$2";; *) usage;; esac; shift 2;;
--create) [ "$CMD" = wrapper ] || usage; CREATE=1;shift;;
*) usage;; esac; done; plain_path "$COPIES" 'copies directory'; }
copy_root() { [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || die '--copy must be a full 40-character commit id'; [ -f "$COPIES/$ID/scripts/mosaic" ] || die "no copy $ID in $COPIES"; echo "$COPIES/$ID"; }
# Closure of scripts/mosaic, bus process/notifier/launcher, Console serve,
@@ -44,16 +48,22 @@ reachable() {
copy_commit() {
[ -d "$REPO/.git" ] || [ "$ALLOW_WORKTREE" = 1 ] || die "refusing: $REPO is not the canonical checkout (its .git is not a directory); run copy from the canonical checkout"
[[ "$ID" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] || die "commit must be a commit id or branch name: $ID"
local full target tmp
local full
full="$(git -C "$REPO" rev-parse --verify --quiet "$ID^{commit}")" || die "not a commit: $ID"
reachable "$full"
copy_tree "$REPO" "$full"
}
# The copy of commit $2 from repository $1. CLEAN names a bundle's temporary repository, which the
# trap removes with the temporary copy.
copy_tree() {
local full="$2" target tmp
target="$COPIES/$full"
[ ! -L "$target" ] || die "refusing: copy target is a symlink: $target"
if [ -e "$target" ]; then echo "exists: $target"; return; fi
mkdir -p "$COPIES"
tmp="$(mktemp -d "$COPIES/.$full.XXXXXX")"
trap "chmod -R u+w '$tmp' 2>/dev/null || true; rm -rf '$tmp'" EXIT
git -C "$REPO" archive --format=tar "$full" -- "${COPY_PATHS[@]}" | tar -xf - -C "$tmp" || die "git archive failed; no copy written"
trap "chmod -R u+w '$tmp' 2>/dev/null || true; rm -rf '$tmp'$CLEAN" EXIT
git -C "$1" archive --format=tar "$full" -- "${COPY_PATHS[@]}" | tar -xf - -C "$tmp" || die "git archive failed; no copy written"
[ -f "$tmp/package.json" ] && [ -f "$tmp/package-lock.json" ] || die 'package.json or package-lock.json missing in temp copy; npm ci not run'
(cd "$tmp" && npm ci --no-audit --no-fund) >&2 || die 'npm ci failed; no copy written'
find "$tmp" -type f -exec chmod a-w {} +
@@ -65,19 +75,53 @@ copy_commit() {
trap - EXIT
echo "copied: $target"
}
# Row 232: the copy of SHA from a git bundle, for a host with no refactor checkout. The bundle must
# verify, carry refs/heads/refactor, and reach SHA from it, as reachable asks of a checkout. Prints
# the RELEASE value at SHA and the bundle's sha256, the release identity on such a host.
copy_bundle() {
[[ "$ID" =~ ^[0-9a-f]{40}$ ]] || die "--bundle needs a full 40-character commit id: $ID"
[ -f "$BUNDLE" ] && [ ! -L "$BUNDLE" ] || die "refusing: bundle is absent, not regular, or a symlink: $BUNDLE"
local g sum release heads
case "$BUNDLE" in /*) ;; *) BUNDLE="$PWD/$BUNDLE";; esac
g="$(mktemp -d)"
CLEAN=" '$g'"
trap "rm -rf '$g'" EXIT
git init -q --bare "$g" || die 'git init failed; no copy written'
git -C "$g" bundle verify -q "$BUNDLE" >/dev/null 2>&1 || die "refusing: git bundle verify failed: $BUNDLE"
heads="$(git -C "$g" bundle list-heads "$BUNDLE" refs/heads/refactor)" || die "refusing: cannot list the bundle's heads: $BUNDLE"
grep -qx '[0-9a-f]\{40\} refs/heads/refactor' <<< "$heads" || die "refusing: the bundle has no refs/heads/refactor: $BUNDLE"
# Every ref, so a SHA the bundle carries on another branch reaches the reachability check.
git -C "$g" fetch -q "$BUNDLE" 'refs/*:refs/*' 2>/dev/null || die "refusing: cannot fetch from $BUNDLE"
git -C "$g" cat-file -e "$ID^{commit}" 2>/dev/null || die "refusing: $ID is not in the bundle"
git -C "$g" merge-base --is-ancestor "$ID" refs/heads/refactor || die "refusing: $ID is not reachable from the bundle's refs/heads/refactor"
release="$(git -C "$g" show "$ID:RELEASE")" || die "no RELEASE at $ID"
sum="$(sha256sum < "$BUNDLE" | cut -d' ' -f1)"
copy_tree "$g" "$ID"
rm -rf "$g"; trap - EXIT
echo "release: $release"
echo "bundle sha256: $sum"
}
# Row 232: --create writes the wrapper only where none is. ln refuses any existing path, a symlink
# included, so this is the one check, and a wrapper that appears mid-run is never replaced.
place() {
if [ "$CREATE" = 1 ]; then ln "$1" "$2" 2>/dev/null || die "refusing: --create and a wrapper exists: $2"; rm -f "$1"
else mv -f "$1" "$2"; fi
}
wrapper() {
[ -z "$MOSAIC_UNIT_SUFFIX" ] || { wrapper_instance; return; }
plain_path "$BIN" 'bin directory'
local root file lease tmp
root="$(copy_root)";file="$BIN/mosaic"
[ -f "$file" ] && [ ! -L "$file" ] || die "refusing: wrapper is absent, not regular, or a symlink: $file"
[ "$(sed -n '2p' "$file")" = "$MARKER" ] || die "refusing: foreign wrapper (missing Q81 marker on line 2): $file"
lease='[ "${1:-}" = __lease-capability ] && exec "'"$BIN"'/mosaic-old" "$@"'
[ "$(grep -Fxc "$lease" "$file" || true)" = 1 ] || die 'refusing: Q81 lease line missing or ambiguous'
if [ "$CREATE" = 0 ]; then
[ -f "$file" ] && [ ! -L "$file" ] || die "refusing: wrapper is absent, not regular, or a symlink: $file"
[ "$(sed -n '2p' "$file")" = "$MARKER" ] || die "refusing: foreign wrapper (missing Q81 marker on line 2): $file"
[ "$(grep -Fxc "$lease" "$file" || true)" = 1 ] || die 'refusing: Q81 lease line missing or ambiguous'
fi
tmp="$(mktemp "$BIN/.mosaic.XXXXXX")"
trap 'rm -f "$tmp"' EXIT
printf '#!/usr/bin/env bash\n%s\n# CLI runs from pinned copy: %s\n# The Q81 lease probe still answers at mosaic-old.\n%s\nexec "%s/scripts/mosaic" "$@"\n' "$MARKER" "$root" "$lease" "$root" > "$tmp"
chmod 0755 "$tmp";mv -f "$tmp" "$file";trap - EXIT
chmod 0755 "$tmp";place "$tmp" "$file";trap - EXIT
echo "wrapper: $root"
}
# Row 190: $BIN/mosaic-NAME. It creates the file or replaces one with its own marker, never another.
@@ -93,12 +137,14 @@ wrapper_instance() {
tmp="$(mktemp "$BIN/.$WRAPPER_NAME.XXXXXX")"
trap 'rm -f "$tmp"' EXIT
printf '#!/usr/bin/env bash\n%s\n# CLI runs from pinned copy: %s\nexport MOSAIC_CONFIG='"'"'%s'"'"'\nexec "%s/scripts/mosaic" "$@"\n' "$INSTANCE_MARKER" "$root" "$MOSAIC_INSTANCE_CONFIG" "$root" > "$tmp"
chmod 0755 "$tmp";mv -f "$tmp" "$file";trap - EXIT
chmod 0755 "$tmp";place "$tmp" "$file";trap - EXIT
echo "wrapper: $root"
}
# --units external reads root's system units, which the stack user can read and not change.
units_show() { if [ "$UNITS" = external ]; then systemctl show "$@"; else systemctl --user show "$@"; fi; }
unit_root() {
local out load root exec_start
out="$(systemctl --user show "$1" -p LoadState -p WorkingDirectory -p ExecStart 2>/dev/null)" || { echo 'absent (systemctl read failed)';return; }
out="$(units_show "$1" -p LoadState -p WorkingDirectory -p ExecStart 2>/dev/null)" || { echo 'absent (systemctl read failed)';return; }
load="$(sed -n 's/^LoadState=//p' <<< "$out")"; root="$(sed -n 's/^WorkingDirectory=//p' <<< "$out")"
exec_start="$(sed -n 's/^ExecStart=//p' <<< "$out")"
if [ "$load" != loaded ] || [ -z "$root" ];then echo absent;return;fi
@@ -133,7 +179,10 @@ CMD="$1";shift
case "$CMD" in check|wrapper) instance_select;; *) [ -z "$INSTANCE_FLAG" ] || usage;; esac
INSTANCE_MARKER="# mosaic-cli wrapper, instance ${MOSAIC_INSTANCE:-dev} (row 190)"
case "$CMD" in
copy|wrapper) [ $# -gt 0 ] || usage;ID="$1";shift;case "$ID" in -*)usage;;esac;parse "$@";if [ "$CMD" = copy ];then copy_commit;else wrapper;fi;;
copy|wrapper) [ $# -gt 0 ] || usage
if [ "$CMD" = copy ] && [ "$1" = --bundle ];then [ $# -ge 3 ] && [ -n "$2" ] || usage;BUNDLE="$2";shift 2;fi
ID="$1";shift;case "$ID" in -*)usage;;esac;parse "$@"
if [ -n "$BUNDLE" ];then copy_bundle;elif [ "$CMD" = copy ];then copy_commit;else wrapper;fi;;
check) parse "$@";[ -z "$MOSAIC_UNIT_SUFFIX" ] || [ "$BUSINESS_SET" = 1 ] || usage;check;;
-h|--help) usage 0;;
*) usage;;esac
+460 -58
View File
@@ -19,6 +19,13 @@
# rehearse (D2, row 163) stops and starts no unit and writes only its own rehearsal directory, but
# it reads the live config and a backup of the live store, so Jason or Mos runs it.
#
# --units external (row 232, #1734; r6 section 4 at agents/darkwing/work/queue-195/DESIGN.md) is
# for a host whose units are root's system units. This script then reads those units and never
# starts or stops one: root stops them, activate or rollback stages the copy (A3), root installs
# and starts them, and confirm gates them and writes the pointer (A5). revert and confirm
# --rollback put the old copy back when a step fails (R2, R4), and gate checks a copy root
# restarted with no activation open (A2f). Without it, everything runs as before.
#
# Most functions below run as `f || handler`, and bash ignores set -e inside such a call, so
# every step that can fail carries its own `|| return 1`.
set -euo pipefail
@@ -32,6 +39,7 @@ BIN="$HOME/.npm-global/bin"
UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
TIMEOUT=60 WANT_PM=0 AFTER_HARNESS=0 MIGRATE=0 PLAN=0 RESTORE="" EVENTS="" TESTER_PORT=""
PROOF_HEADER="" PROOF_FILE="" NO_PROOF=0 NO_SEND=0 SSO=0 R_BACKUP="" R_PORT=""
UNITS=user TO_FLAG="" EXPECT_FROM="" SINCE="" ROLLBACK_OLD="" CAUSE=""
usage() {
cat >&2 <<'USAGE'
@@ -42,11 +50,18 @@ usage: stack-release.sh adopt BUSINESS
stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan]
stack-release.sh status BUSINESS
stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]
common options: --copies DIR, --bin DIR, --instance NAME
stack-release.sh confirm ID BUSINESS --units external [--rollback OLD --cause A3|A4|A5] [--timeout S]
stack-release.sh revert ID BUSINESS --units external
stack-release.sh gate ID BUSINESS --units external --since EPOCH [--timeout S]
common options: --copies DIR, --bin DIR, --instance NAME, --units user|external
--instance NAME (row 190) runs on stack instance NAME: its config, data root, units and the wrapper
mosaic-NAME. Without it, the default instance, as before.
--events N is the total of the per-table counts the post-backup-events refusal prints: events and
every table in COUNTED.
--units external (row 232) reads root's system units and starts or stops none. adopt, activate,
rollback, status, confirm, revert and gate take it; rollback then needs --to ID, the previous copy.
activate and rollback --plan take --expect-from OLD. --pm, --pm-after-harness, --tester-port, the
proof flags, --no-proof, --no-send, --sso and pm-on are not for it.
USAGE
exit "${1:-4}"
}
@@ -58,7 +73,27 @@ now_ms() { date -u +%Y-%m-%dT%H:%M:%S.%3NZ; }
jline() {
node -e 'const a=process.argv.slice(1),o={};for(let i=0;i<a.length;i+=2)o[a[i]]=a[i+1]==="@null"?null:a[i+1];process.stdout.write(JSON.stringify(o)+"\n")' -- "$@"
}
log_line() { jline at "$(now)" op "$OP" from "${FROM:-@null}" to "${TO:-@null}" backup "${BACKUP:-@null}" activation "${ACT:-@null}" "$@" >> "$LOG"; }
log_line() {
[ "$UNITS" = user ] || set -- "$@" units external
jline at "$(now)" op "$OP" from "${FROM:-@null}" to "${TO:-@null}" backup "${BACKUP:-@null}" activation "${ACT:-@null}" "$@" >> "$LOG"
}
# --- The units backend (row 232) ---
# Every systemctl, journalctl and systemd-analyze call goes through these three. user is the
# systemd user manager, as before. external reads root's system units, which needs no privilege,
# and refuses any verb that would change one.
sc() {
if [ "$UNITS" = user ]; then systemctl --user "$@"; return; fi
case "$1" in
show|is-active|list-units) systemctl "$@" ;;
*) echo "stack-release: --units external never runs systemctl $1; root does" >&2; return 1 ;;
esac
}
# The bus unit's journal is a user journal only: under external it lands in the system journal,
# which the stack's user can't read (r6 section 4), so the external PM check reads launches list.
jctl() { [ "$UNITS" = user ] || return 1; journalctl --user "$@"; }
analyze() { [ "$UNITS" = user ] || return 1; systemd-analyze --user "$@"; }
# A refusal before anything stopped: exit 3, and one log line with the code, except for
# run-locked (the lock holder may be writing the log) and in --plan, which writes nothing.
@@ -83,7 +118,7 @@ parse_opts() {
case "$1" in
--copies) [ $# -ge 2 ] || usage; COPIES="$2"; shift 2 ;;
--bin) [ $# -ge 2 ] || usage; BIN="$2"; shift 2 ;;
--timeout) [ "$OP" != status ] && [ "$OP" != adopt ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
--timeout) [ "$OP" != status ] && [ "$OP" != adopt ] && [ "$OP" != revert ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
--plan) [ "$OP" = activate ] || [ "$OP" = rollback ] || usage; PLAN=1; shift ;;
--pm) [ "$OP" = activate ] || usage; WANT_PM=1; shift ;;
--pm-after-harness) [ "$OP" = activate ] || usage; AFTER_HARNESS=1; shift ;;
@@ -98,9 +133,27 @@ parse_opts() {
--backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
--port) [ "$OP" = rehearse ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{1,5}$ ]] && [ "$((10#$2))" -ge 1024 ] && [ "$((10#$2))" -le 65535 ] || usage; R_PORT="$((10#$2))"; shift 2 ;;
--events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
--units) [ "$OP" != rehearse ] && [ $# -ge 2 ] && { [ "$2" = user ] || [ "$2" = external ]; } || usage; UNITS="$2"; shift 2 ;;
--to) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; TO_FLAG="$2"; shift 2 ;;
--expect-from) { [ "$OP" = activate ] || [ "$OP" = rollback ]; } && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; EXPECT_FROM="$2"; shift 2 ;;
--since) [ "$OP" = gate ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]{0,11})$ ]] || usage; SINCE="$2"; shift 2 ;;
--rollback) [ "$OP" = confirm ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; ROLLBACK_OLD="$2"; shift 2 ;;
--cause) [ "$OP" = confirm ] && [ $# -ge 2 ] && [[ "$2" =~ ^A[345]$ ]] || usage; CAUSE="$2"; shift 2 ;;
*) usage ;;
esac
done
# Row 232: confirm, revert, gate, --to and --expect-from are external only, and an external
# rollback names its target. gate needs --since, its start floor (r6 N2), and --expect-from
# goes with --plan. The flags that change a unit's render or the PM are root's under external.
if [ "$UNITS" = user ]; then
[ "$OP" != confirm ] && [ "$OP" != revert ] && [ "$OP" != gate ] && [ -z "$TO_FLAG" ] && [ -z "$EXPECT_FROM" ] || usage
else
[ "$OP" != rollback ] || [ -n "$TO_FLAG" ] || usage
[ "$WANT_PM" = 0 ] && [ -z "$TESTER_PORT" ] && [ -z "$PROOF_HEADER" ] && [ -z "$PROOF_FILE" ] && [ "$NO_PROOF" = 0 ] && [ "$NO_SEND" = 0 ] && [ "$SSO" = 0 ] || usage
fi
[ "$OP" != gate ] || [ -n "$SINCE" ] || usage
[ -z "$EXPECT_FROM" ] || [ "$PLAN" = 1 ] || usage
[ "${ROLLBACK_OLD:+1}" = "${CAUSE:+1}" ] || usage
[ -z "$EVENTS" ] || [ -n "$RESTORE" ] || usage
[ "$OP" != rehearse ] || { [ -n "$R_BACKUP" ] && [ -n "$R_PORT" ]; } || usage
# A bad tester port is a usage error; --plan lists it as tester-port-invalid instead (row 180).
@@ -187,7 +240,7 @@ process.exit(1);' -- "$LOG" "$1" "$P_COPY"
# --- Reading the units ---
show() { systemctl --user show "$1" -p LoadState -p WorkingDirectory -p ExecStart -p Environment -p MainPID -p DropInPaths 2>/dev/null; }
show() { sc show "$1" -p LoadState -p WorkingDirectory -p ExecStart -p Environment -p MainPID -p DropInPaths 2>/dev/null; }
argv_of() { sed -n 's/^ExecStart=.*argv\[\]=\([^;]*\) ;.*/\1/p' <<< "$1"; }
# systemctl show quotes a value that holds a space and escapes with a backslash, and the read -a
# split below can't undo either, so such a line is unreadable rather than read wrong (row 166 item 4).
@@ -286,12 +339,20 @@ check_target() {
# has KIND exits 0 when the copy's KINDS has KIND
# unknown prints the business file's plugins keys the copy's KINDS lacks, from a plain JSON read
# harness prints plugins.harness as JSON, or `none`, from the copy's loader
# launchby prints launch.by, the PM's instance, from a plain JSON read (row 232); it imports
# only business.mjs, for the business file's path
# A failure exits 1 and prints its first line.
PLUGINS_JS='
const {readFileSync}=require("node:fs");const {pathToFileURL}=require("node:url");
const [mode,copy,id,kind]=process.argv.slice(1);
const imp=(p)=>import(pathToFileURL(`${copy}/packages/${p}`).href);
(async()=>{
if(mode==="launchby"){
const {businessFilePath}=await imp("business/src/business.mjs");
const by=JSON.parse(readFileSync(businessFilePath(id),"utf8")).launch?.by;
if(typeof by!=="string"||!by)throw new Error("the business file has no launch.by");
return process.stdout.write(by);
}
const {KINDS}=await imp("plugins/src/kinds.mjs");
if(mode==="has")process.exit(Object.hasOwn(KINDS,kind)?0:1);
const b=await imp("business/src/business.mjs");
@@ -455,13 +516,13 @@ capture() {
# Stops the Console, then the bus, and waits for both and the stopped check, within --timeout.
stop_both() {
systemctl --user stop "$CONSOLE_UNIT" || return 1
systemctl --user stop "$BUS_UNIT" || return 1
sc stop "$CONSOLE_UNIT" || return 1
sc stop "$BUS_UNIT" || return 1
local deadline=$((SECONDS + TIMEOUT)) u s down
while :; do
down=1
for u in "$CONSOLE_UNIT" "$BUS_UNIT"; do
s="$(systemctl --user is-active "$u" 2>/dev/null || true)"
s="$(sc is-active "$u" 2>/dev/null || true)"
case "$s" in inactive|failed) ;; *) down=0 ;; esac
done
if [ "$down" = 1 ] && stopped; then say "stopped both units"; return 0; fi
@@ -472,7 +533,7 @@ stop_both() {
start_both() {
START_EPOCH="$(date +%s)" && START_MS="$(date +%s%3N)" || return 1
systemctl --user start "$BUS_UNIT" && systemctl --user start "$CONSOLE_UNIT"
sc start "$BUS_UNIT" && sc start "$CONSOLE_UNIT"
}
# Step 4: installs copy $1 with pm $2, claude $3 and the Console flags after them. Sets FAIL.
@@ -493,9 +554,19 @@ install_stack() {
gate_once() {
local copy="$COPIES/$1" pm="$2" main out
GATE=units-inactive GATE_WHY=""
systemctl --user is-active --quiet "$BUS_UNIT" && systemctl --user is-active --quiet "$CONSOLE_UNIT" || return 1
sc is-active --quiet "$BUS_UNIT" && sc is-active --quiet "$CONSOLE_UNIT" || return 1
if [ "$UNITS" = external ]; then
# r6 N4: root started the units, so this start's time is the bus unit's, in whole seconds,
# at or after FLOOR. It is a gate code, so polling goes on: Restart=on-failure can bring a
# fresh start.
GATE=stale-start
out="$(sc show --timestamp=unix -p ActiveEnterTimestamp "$BUS_UNIT" 2>/dev/null | sed -n 's/^ActiveEnterTimestamp=@//p')" || return 1
[[ "$out" =~ ^[0-9]{1,12}$ ]] || { GATE_WHY="ActiveEnterTimestamp is unreadable"; return 1; }
[ "$out" -ge "$FLOOR" ] || { GATE_WHY="the bus started at @$out, before @$FLOOR"; return 1; }
START_MS=$((out * 1000))
fi
GATE=bus-status
main="$(systemctl --user show "$BUS_UNIT" -p MainPID | sed -n 's/^MainPID=//p')" || return 1
main="$(sc show "$BUS_UNIT" -p MainPID | sed -n 's/^MainPID=//p')" || return 1
out="$("$copy/scripts/mosaic" bus status --json 2>/dev/null)" || { GATE_WHY="bus status failed"; return 1; }
# Every `bus start` writes boot.at (packages/cli/src/boot-status.mjs), so a value from before
# this start means the host on the socket is not the one this run started.
@@ -508,13 +579,13 @@ process.stdout.write(why);' -- "$out" "$BUSINESS" "$main" "$START_MS" 2>&1)" ||
node -e '
fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>process.exit(r.status===200?0:1),()=>process.exit(1));' -- "$(console_port)" || return 1
GATE=copy-check
out="$(scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>/dev/null)" || return 1
out="$(scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" "${UNITS_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>/dev/null)" || return 1
# check passes when the three agree on one copy; this run needs it to be the target (row 166 item 3).
out="$(sed -n 's/^bus: //p' <<< "$out")"
[ "$out" = "$copy" ] || { GATE_WHY="the bus, Console and CLI agree on $out, not $copy"; return 1; }
if [ "$pm" = 1 ]; then
GATE=pm-launch-unconfirmed
pm_launched "$copy" || return 1
if [ "$UNITS" = user ]; then pm_launched "$copy" || return 1; else pm_external "$1" || return 1; fi
fi
GATE=""
}
@@ -526,7 +597,7 @@ fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(
# because this host's journalctl reads a bare time as local time.
pm_launched() {
local journal run list
journal="$(journalctl --user -u "$BUS_UNIT" --since "@$START_EPOCH" -o cat 2>/dev/null)" || { GATE_WHY="journal read failed"; return 1; }
journal="$(jctl -u "$BUS_UNIT" --since "@$START_EPOCH" -o cat 2>/dev/null)" || { GATE_WHY="journal read failed"; return 1; }
local -a lines=()
mapfile -t lines < <(grep '^launched ' <<< "$journal" || true)
[ "${#lines[@]}" -gt 0 ] || { GATE_WHY="no launched line since the start"; return 1; }
@@ -540,6 +611,20 @@ process.stdout.write(!s?`run ${run} is not in launches list`:s.business!==b?`run
[ -z "$GATE_WHY" ]
}
# r6 section 4: the PM check under --units external, where the bus unit's journal is the system
# journal and unreadable here. It passes when exactly one live launches list entry is for the
# PM's instance (launch.by, packages/cli/src/launcher.mjs:675), and that entry is for BUSINESS and
# started at or after this start. A run the PM starts for another role has another instance.
pm_external() {
local by list
by="$(plugins launchby "$1" 2>&1)" || { GATE_WHY="cannot read launch.by: $by"; return 1; }
list="$("$COPIES/$1/scripts/mosaic" launches list --json 2>/dev/null)" || { GATE_WHY="launches list failed"; return 1; }
GATE_WHY="$(node -e '
const [list,by,b,since]=process.argv.slice(1),l=JSON.parse(list).filter((x)=>x.instance===by&&x.live===true),s=l[0];
process.stdout.write(l.length===0?`no live launch of ${by}`:l.length>1?`${l.length} live launches of ${by}`:s.business!==b?`the launch of ${by} is for ${s.business}`:!(Date.parse(s.startedAt)>=Number(since))?`run ${s.run} started before this start`:"");' -- "$list" "$by" "$BUSINESS" "$START_MS" 2>&1)" || { GATE_WHY="launches list unreadable"; return 1; }
[ -z "$GATE_WHY" ]
}
# Polls every second until every check passes or --timeout runs out.
gate() {
local deadline=$((SECONDS + TIMEOUT))
@@ -573,7 +658,7 @@ put_back() {
done
[ "$(sha256sum < "$BIN/$WRAPPER_NAME" | cut -d' ' -f1)" = "$(sum_of "$RS" wrapper)" ] || return 1
[ "$(stat -c %a "$BIN/$WRAPPER_NAME")" = "$(cat "$RS/WRAPPER_MODE")" ] || return 1
systemctl --user daemon-reload || return 1
sc daemon-reload || return 1
say "put back the unit files and the wrapper from $RS"
}
@@ -608,8 +693,8 @@ after_fail() {
# Stop both units, print both results, log rollback-failed, exit 2. No retry.
rollback_failed() {
trap - INT TERM HUP
systemctl --user stop "$CONSOLE_UNIT" || true
systemctl --user stop "$BUS_UNIT" || true
sc stop "$CONSOLE_UNIT" || true
sc stop "$BUS_UNIT" || true
say "rollback-failed: run $1, rollback $2"
log_line result rollback-failed code "$1" rollback failed rollbackCode "$2"
echo "stack-release: rollback-failed: the run failed with $1 and the rollback with $2; both units are stopped. Rollback set: $RS${BACKUP:+, backup $DATA/backups/stack/$BACKUP}${RESTORED:+, post-attempt store $POST}" >&2
@@ -618,10 +703,15 @@ rollback_failed() {
# --- Refusals (item 3) ---
lock() {
# confirm and revert act on the open activation by design, so they take the lock alone (r6 section 4).
lock_only() {
mkdir -p "$STATE"
exec 9> "$STATE/stack-release.lock"
flock -n 9 || refuse run-locked "another stack-release run holds $STATE/stack-release.lock"
}
lock() {
lock_only
local open
open="$(interrupted | paste -sd' ' -)"
[ -z "$open" ] || refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)"
@@ -629,7 +719,7 @@ lock() {
instance_busy() {
local out unit load state rest
out="$(systemctl --user list-units --all --no-legend --plain "$BUS_GLOB" 2>/dev/null)" || refuse instance-busy "systemctl --user list-units failed, so the bus state is unknown"
out="$(sc list-units --all --no-legend --plain "$BUS_GLOB" 2>/dev/null)" || refuse instance-busy "$SC_NAME list-units failed, so the bus state is unknown"
while read -r unit load state rest; do
[[ "$unit" == $BUS_GLOB ]] || continue
if [ "$unit" = "$BUS_UNIT" ]; then
@@ -645,7 +735,7 @@ instance_busy() {
drop_in_override() {
local paths directory dropdir file content glob
local -a dropdirs
paths="$(systemd-analyze --user unit-paths 2>/dev/null)" && [ -n "$paths" ] || refuse drop-in-override "systemd unit paths are unknown"
paths="$(analyze unit-paths 2>/dev/null)" && [ -n "$paths" ] || refuse drop-in-override "systemd unit paths are unknown"
while IFS= read -r directory; do
[ -n "$directory" ] || continue
case "$directory" in /*) ;; *) refuse drop-in-override "invalid systemd unit path $directory" ;; esac
@@ -679,7 +769,8 @@ flags_drift() {
if [ "$AFTER_HARNESS" = 1 ]; then
[ "$P_PM" = 1 ] || d+=$'\n'" pm: pointer $P_PM, --pm-after-harness needs a pointer with pm 1"
else
[ "$OP" != activate ] || [ "$WANT_PM" = "$P_PM" ] || d+=$'\n'" pm: pointer $P_PM, --pm given $WANT_PM"
# Under --units external the PM is a flag in root's unit, so there is no --pm to compare.
[ "$UNITS" != user ] || [ "$OP" != activate ] || [ "$WANT_PM" = "$P_PM" ] || d+=$'\n'" pm: pointer $P_PM, --pm given $WANT_PM"
fi
[ -z "$d" ] || refuse unit-flags-drift "the units differ from the pointer; settle it first:$d"
}
@@ -1075,11 +1166,12 @@ do_adopt() {
lock
[ ! -f "$POINTER" ] || { read_pointer; FROM="$P_COPY"; refuse already-adopted "$POINTER exists (copy $P_COPY)"; }
local out
out="$(scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1)" || refuse check-failed "stack-copy.sh check did not pass:"$'\n'"$out"
out="$(scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" "${UNITS_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1)" || refuse check-failed "stack-copy.sh check did not pass:"$'\n'"$out"
TO="$(sed -n 's/^bus: //p' <<< "$out")"
TO="${TO#"$COPIES/"}"
read_units || refuse check-failed "cannot read $BUS_UNIT or $CONSOLE_UNIT"
[ -n "$U_CLAUDE" ] || refuse no-claude-pin "$BUS_UNIT has no MOSAIC_CLAUDE_BIN"
# Under --units external the claude pin is root's render, so adopt records what the unit has.
[ "$UNITS" = external ] || [ -n "$U_CLAUDE" ] || refuse no-claude-pin "$BUS_UNIT has no MOSAIC_CLAUDE_BIN"
target_schema "$TO" || refuse copy-schema-unknown "cannot read SCHEMA_VERSION from $COPIES/$TO/packages/bus/src/store.mjs"
next_activation
write_pointer "$TO" "$U_PM" 0 "$U_CLAUDE" "$T_SCHEMA" "${U_CONSOLE[@]}"
@@ -1094,12 +1186,12 @@ plan_store() {
local src="" why=""
S_SCHEMA="" S_FROM="" S_CHANGED=""
if [ ! -e "$STORE" ]; then why="there is no store file"
elif systemctl --user is-active --quiet "$BUS_UNIT"; then
elif sc is-active --quiet "$BUS_UNIT"; then
src="the store the running bus holds"
S_SCHEMA="$(node -e "$STORE_JS" -- heldversion "$STORE" 2>/dev/null)" || S_SCHEMA=""
# A bus that stops during the readOnly open can leave -wal or -shm beside the stopped
# store, so the version read can't be planned on (row 166 item 2).
systemctl --user is-active --quiet "$BUS_UNIT" || S_CHANGED=1
sc is-active --quiet "$BUS_UNIT" || S_CHANGED=1
elif store_unclean; then why="the store was not read: $UNCLEAN is present"
else
src="the stopped store"
@@ -1186,15 +1278,19 @@ plan() {
echo "backup: the run backs up the stopped store to $DATA/backups/stack/<activation>/ before any migration"
fi
plan_commits
[ "$P_PENDING" = 0 ] || echo "pm: the pointer is pending (activate --pm-after-harness); the bus runs without its PM"
[ "$RUN_PENDING" = 0 ] || echo "pm: the bus starts without --pm and the PM stays pending; next: Jason's plugins set of harness, then pm-on"
local flags="${INSTANCE_FLAGS[*]:+${INSTANCE_FLAGS[*]} }"
echo "bus: scripts/bus-service.sh install $flags--copy $TO --copies $COPIES$([ "$RUN_PM" = 0 ] || echo ' --pm') --claude $P_CLAUDE"
[ -z "$DROP_TESTER" ] || echo "drop $DROP_TESTER; the target Console does not support it"
echo "$PROOF_PLAN"
echo "$SEND_PLAN"
[ -z "$SSO_PLAN" ] || echo "$SSO_PLAN"
echo "Console: scripts/webui-service.sh install $flags--copy $TO --copies $COPIES${RUN_CONSOLE[*]:+ ${RUN_CONSOLE[*]}}"
if [ "$UNITS" = external ]; then
echo "units: external; root installs the units for $TO and starts them, then confirm gates them"
else
[ "$P_PENDING" = 0 ] || echo "pm: the pointer is pending (activate --pm-after-harness); the bus runs without its PM"
[ "$RUN_PENDING" = 0 ] || echo "pm: the bus starts without --pm and the PM stays pending; next: Jason's plugins set of harness, then pm-on"
echo "bus: scripts/bus-service.sh install $flags--copy $TO --copies $COPIES$([ "$RUN_PM" = 0 ] || echo ' --pm') --claude $P_CLAUDE"
[ -z "$DROP_TESTER" ] || echo "drop $DROP_TESTER; the target Console does not support it"
echo "$PROOF_PLAN"
echo "$SEND_PLAN"
[ -z "$SSO_PLAN" ] || echo "$SSO_PLAN"
echo "Console: scripts/webui-service.sh install $flags--copy $TO --copies $COPIES${RUN_CONSOLE[*]:+ ${RUN_CONSOLE[*]}}"
fi
echo "CLI: scripts/stack-copy.sh wrapper $TO $flags--copies $COPIES --bin $BIN"
local r
for r in "${PLAN_REFUSE[@]}"; do echo "refuses $r"; done
@@ -1268,6 +1364,16 @@ do_activate() {
pointer_for_business
run_pm
[ "$P_PENDING" = 0 ] || check_refuse pm-pending "the pointer is pending (activate --pm-after-harness); run pm-on or rollback first"
if [ "$UNITS" = external ]; then
ext_pointer_checks
if [ "$PLAN" = 1 ]; then check_target; plugin_checks; config_checks; plan; exit 0; fi
ext_precheck
plugin_checks
config_checks
new_activation
run_external migrate
return
fi
if [ "$PLAN" = 1 ]; then check_target; console_for_target; ports_unclaimed; plugin_checks; config_checks; plan; exit 0; fi
precheck
plugin_checks
@@ -1281,43 +1387,321 @@ do_rollback() {
pointer_for_business
TO="$(from_log previous)" || refuse no-previous "the log names no earlier copy than $P_COPY"
run_pm
if [ "$UNITS" = external ]; then
# r6: root names the target from the go, and it must be the one rollback reads from the log.
[ "$TO_FLAG" = "$TO" ] || refuse to-not-previous "--to $TO_FLAG is not the previous copy; the log names $TO"
ext_pointer_checks
if [ "$PLAN" = 1 ]; then
check_target; plugin_checks; config_checks
[ -z "$RESTORE" ] || { next_activation; restore_checks; }
plan; exit 0
fi
ext_precheck
plugin_checks
config_checks
[ "$P_SCHEMA" = "$T_SCHEMA" ] || [ -n "$RESTORE" ] || refuse schema-differs "the store is at $P_SCHEMA and $TO expects $T_SCHEMA; pass --restore ACTIVATION"
new_activation
[ -z "$RESTORE" ] || restore_checks
run_external restore
return
fi
if [ "$PLAN" = 1 ]; then check_target; console_for_target; ports_unclaimed; plugin_checks; config_checks; plan; exit 0; fi
precheck
plugin_checks
config_checks
[ "$P_SCHEMA" = "$T_SCHEMA" ] || [ -n "$RESTORE" ] || refuse schema-differs "the store is at $P_SCHEMA and $TO expects $T_SCHEMA; pass --restore ACTIVATION"
new_activation
if [ -n "$RESTORE" ]; then
local o="$DATA/backups/stack/$RESTORE" rc=0 c="" text
[ -f "$o/SHA256SUMS" ] && [ "$(cat "$o/DATA_ROOT" 2>/dev/null)" = "$DATA" ] && [ "$(cat "$o/SCHEMA" 2>/dev/null)" = "$T_SCHEMA" ] && [ -f "$o/STAMP" ] \
|| refuse restore-mismatch "$o is not a backup of this root at $T_SCHEMA"
# Before anything stops. A store the running bus holds is read with readOnly, which adds
# no file (the probe); a stopped one falls under the store_unclean rule.
if systemctl --user is-active --quiet "$BUS_UNIT"; then
c="$(node -e "$STORE_JS" -- heldcount "$STORE" "$(cat "$o/STAMP")" "$COUNTED")" || rc=1
# plan_store's recheck (row 166 item 2), before anything stops.
systemctl --user is-active --quiet "$BUS_UNIT" || refuse restore-mismatch "the bus stopped while the events after $(cat "$o/STAMP") were counted, so the store state changed; nothing was stopped"
else
! store_unclean || refuse store-unclean "$UNCLEAN is present; the bus did not close the store cleanly"
c="$(store_read count "$(cat "$o/STAMP")" "$COUNTED")" || rc=1
fi
[ "$rc" = 0 ] || refuse restore-mismatch "cannot count the rows after $(cat "$o/STAMP")"
COUNT="${c%%$'\t'*}" text="${c#*$'\t'}"
echo "restore: $o; the store now in place moves to $POST and is kept; $text were written after the backup"
[ "$COUNT" = 0 ] || [ "$EVENTS" = "$COUNT" ] || refuse post-backup-events "$text were written after the backup; Jason decides, then --events $COUNT"
BACKUP="$RESTORE"
fi
[ -z "$RESTORE" ] || restore_checks
run_steps restore
}
# rollback --restore's checks on the backup and the rows written after it, before anything stops.
# Sets COUNT and BACKUP; POST comes from new_activation.
restore_checks() {
local o="$DATA/backups/stack/$RESTORE" rc=0 c="" text
[ -f "$o/SHA256SUMS" ] && [ "$(cat "$o/DATA_ROOT" 2>/dev/null)" = "$DATA" ] && [ "$(cat "$o/SCHEMA" 2>/dev/null)" = "$T_SCHEMA" ] && [ -f "$o/STAMP" ] \
|| refuse restore-mismatch "$o is not a backup of this root at $T_SCHEMA"
# Before anything stops. A store the running bus holds is read with readOnly, which adds
# no file (the probe); a stopped one falls under the store_unclean rule.
if sc is-active --quiet "$BUS_UNIT"; then
c="$(node -e "$STORE_JS" -- heldcount "$STORE" "$(cat "$o/STAMP")" "$COUNTED")" || rc=1
# plan_store's recheck (row 166 item 2), before anything stops.
sc is-active --quiet "$BUS_UNIT" || refuse restore-mismatch "the bus stopped while the events after $(cat "$o/STAMP") were counted, so the store state changed; nothing was stopped"
else
! store_unclean || refuse store-unclean "$UNCLEAN is present; the bus did not close the store cleanly"
c="$(store_read count "$(cat "$o/STAMP")" "$COUNTED")" || rc=1
fi
[ "$rc" = 0 ] || refuse restore-mismatch "cannot count the rows after $(cat "$o/STAMP")"
COUNT="${c%%$'\t'*}" text="${c#*$'\t'}"
echo "restore: $o; the store now in place moves to $POST and is kept; $text were written after the backup"
[ "$COUNT" = 0 ] || [ "$EVENTS" = "$COUNT" ] || refuse post-backup-events "$text were written after the backup; Jason decides, then --events $COUNT"
BACKUP="$RESTORE"
}
# --- --units external (row 232, r6 section 4) ---
# After the pointer, for activate and rollback, in --plan and in the run. OLD is the copy root read
# from the unit files it installed (r6 A1, N5), so a pointer that names another copy stops the
# re-pin before root stops anything. The PM is a flag in root's unit, so --pm-after-harness has
# nothing to wait on. The run's own lock refuses an open activation; --plan takes no lock, so it
# lists one here.
ext_pointer_checks() {
[ -z "$EXPECT_FROM" ] || [ "$EXPECT_FROM" = "$P_COPY" ] || check_refuse pointer-differs "the pointer names $P_COPY and root's units name $EXPECT_FROM; Jason or Mos settles it"
[ "$AFTER_HARNESS" = 0 ] || check_refuse external-pm "under --units external the PM is a flag in root's bus unit; changing it is a re-install with its own go"
[ "$PLAN" = 1 ] || return 0
local open
open="$(interrupted | paste -sd' ' -)"
[ -z "$open" ] || check_refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)"
}
# A3's checks before `started`. Root has stopped the units and installs them after this run, so
# there is no Console flag, port, drop-in or claude pin of this script's to check.
ext_precheck() {
check_target
instance_busy
read_units || refuse unit-flags-drift "cannot read $BUS_UNIT or $CONSOLE_UNIT"
flags_drift
units_down
}
# r6 A3 (N4): each unit reads inactive, or failed as stop_both accepts, and the store is closed,
# so nothing below runs against a store a live bus holds.
units_down() {
local u s
for u in "$CONSOLE_UNIT" "$BUS_UNIT"; do
s="$(sc is-active "$u" 2>/dev/null || true)"
case "$s" in inactive|failed) ;; *) refuse units-running "$u is ${s:-unknown}; root stops the units first" ;; esac
done
stopped || refuse units-running "the store under $DATA/bus is still open (writer.lock or an open file)"
}
# A3's capture into $RS, which run_external made: the pointer, the wrapper and its mode, and for each
# unit its show output and the sha256 of its unit file. It reads root's units and makes no directory.
capture_units() {
local u frag
[ ! -f "$POINTER" ] || cp -p "$POINTER" "$RS/pointer" || return 1
[ -f "$BIN/$WRAPPER_NAME" ] || { say "capture: no wrapper $BIN/$WRAPPER_NAME"; return 1; }
cp -p "$BIN/$WRAPPER_NAME" "$RS/wrapper" || return 1
stat -c %a "$BIN/$WRAPPER_NAME" > "$RS/WRAPPER_MODE" || return 1
: > "$RS/UNITS.sha256" || return 1
for u in "$BUS_UNIT" "$CONSOLE_UNIT"; do
show "$u" > "$RS/show-$u.txt" || return 1
frag="$(sc show "$u" -p FragmentPath 2>/dev/null | sed -n 's/^FragmentPath=//p')" && [ -n "$frag" ] || { say "capture: no FragmentPath for $u"; return 1; }
sha256sum "$frag" >> "$RS/UNITS.sha256" || return 1
done
sums "$RS" > "$RS/SHA256SUMS" || return 1
say "captured the rollback set: $RS"
}
# A failure after the capture: FAILED holds the code, nothing is logged or cleaned up, exit 1.
# Root runs R1-R4, and revert is the one cleanup path (r6 A3).
ext_fail() {
trap - INT TERM HUP
{ echo "$1" > "$RS/FAILED"; } 2>/dev/null || true
say "failed: $1"
echo "stack-release: $OP failed with $1; $ACT stays open for root's revert and confirm --rollback. Rollback set: $RS" >&2
exit 1
}
# A3 for target $TO. $1 is the store step: migrate or restore. The marker goes in first, so every
# failure after it leaves what revert needs. Writes no pointer and starts nothing.
run_external() {
log_line result started
{ mkdir -p -m 0700 "$STATE/stack-rollback" && mkdir -m 0700 "$RS"; } 2>/dev/null || refuse marker-failed "could not make the rollback set $RS; nothing was changed"
{ echo "$TO" > "$RS/AWAITING-UNITS"; } 2>/dev/null || { rm -f "$RS/AWAITING-UNITS"; refuse marker-failed "could not write $RS/AWAITING-UNITS; nothing was changed"; }
STEPS="$RS/steps.log"
capture_units || { rm -f "$RS/AWAITING-UNITS"; refuse capture-failed "could not write the rollback set $RS; nothing was changed"; }
trap 'ext_fail signal' INT TERM HUP
if [ -e "$STORE" ]; then
if store_unclean; then say "$UNCLEAN is present; the store is left as it is for Jason or Mos"; ext_fail store-unclean; fi
STORE_SCHEMA="$(store_read version)" && [ -n "$STORE_SCHEMA" ] || ext_fail schema-unreadable
say "stopped store schema: $STORE_SCHEMA"
if [ "$1" = migrate ]; then
if [ "$STORE_SCHEMA" != "$T_SCHEMA" ]; then
in_chain "$STORE_SCHEMA" || ext_fail store-newer
[ "$MIGRATE" = 1 ] || ext_fail migration-needs-flag
fi
backup_store || ext_fail backup
echo "$BACKUP" > "$RS/BACKUP" || ext_fail backup
if [ "$STORE_SCHEMA" != "$T_SCHEMA" ]; then
# MIGRATING and TAKEN before the migration, so revert restores a partial one (r6 B2).
: > "$RS/MIGRATING" || ext_fail migrate-failed
TAKEN="$ACT"
node "$COPIES/$TO/packages/bus/src/migrate-schema.mjs" "$DATA" >> "$STEPS" 2>&1 || ext_fail migrate-failed
say "migrated the store to $T_SCHEMA"
fi
else
[ "$STORE_SCHEMA" = "$P_SCHEMA" ] || ext_fail store-differs
if [ -n "$RESTORE" ]; then
local n
n="$(store_read count "$(cat "$DATA/backups/stack/$RESTORE/STAMP")" "$COUNTED")" || ext_fail schema-unreadable
[ "${n%%$'\t'*}" = "$COUNT" ] || ext_fail post-backup-events
echo "$RESTORE" > "$RS/BACKUP" || ext_fail restore
# RESTORED tells confirm --rollback the store was swapped, so it skips OLD's gate
# and logs rollbackCode not-attempted (ruling 4, T14b).
if ! restore_store "$RESTORE"; then
[ -z "$RESTORED" ] || : > "$RS/RESTORED" || true
ext_fail restore
fi
: > "$RS/RESTORED" || ext_fail restore
fi
fi
elif [ -n "$RESTORE" ]; then
ext_fail store-differs
fi
scripts/stack-copy.sh wrapper "$TO" "${INSTANCE_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" >> "$STEPS" 2>&1 || ext_fail wrapper
trap - INT TERM HUP
say "staged $TO"
echo "stack-release: $OP staged $TO for $BUSINESS as $ACT; root installs and starts the units, then runs confirm $TO $BUSINESS --units external"
}
# r6 "The open activation": exactly one AWAITING-UNITS file, its activation the one the log holds
# open, holding the copy root passed. Sets O_ACT. Anything else refuses open-activation, and ACT
# stays unset, so the refused line names no activation and the open one stays open.
open_activation() {
local -a markers open
local m found="" seen=""
mapfile -t markers < <(compgen -G "$STATE/stack-rollback/*/AWAITING-UNITS" || true)
mapfile -t open < <(interrupted)
for m in "${markers[@]}"; do
seen+=" $(basename "$(dirname "$m")") (copy $(cat "$m" 2>/dev/null || echo unreadable))"
done
if [ "${#markers[@]}" = 1 ] && [ "${#open[@]}" = 1 ]; then
m="$(basename "$(dirname "${markers[0]}")")"
[ "$m" != "${open[0]}" ] || [ "$(cat "${markers[0]}" 2>/dev/null)" != "$ID" ] || found="$m"
fi
[ -n "$found" ] || refuse open-activation "need one AWAITING-UNITS for copy $ID whose activation the log holds open; markers:${seen:- none}; open: ${open[*]:-none}"
O_ACT="$found"
}
# The op, from, to and start second of O_ACT's `started` line, one per line.
started_of() {
node -e '
const [file,act]=process.argv.slice(1);
let s;
for(const l of require("fs").readFileSync(file,"utf8").split("\n")){
if(!l.trim())continue;
let o;try{o=JSON.parse(l)}catch{continue}
if(o.activation===act&&o.result==="started")s=o;
}
const at=s&&Math.floor(Date.parse(s.at)/1000);
if(!s||!["activate","rollback"].includes(s.op)||!Number.isFinite(at))process.exit(1);
process.stdout.write([s.op,s.from??"",s.to??"",at].join("\n")+"\n");' -- "$LOG" "$O_ACT"
}
# A5, and R4 with --rollback. The lines it writes carry the open activation's op, from, to and
# activation, and the backup A3 recorded, so from_log reads them as a dragon-lin run's. Every check
# runs before ACT is set.
do_confirm() {
lock_only
open_activation
local s
local -a f
s="$(started_of)" || refuse open-activation "the log's started line for $O_ACT is unreadable"
mapfile -t f <<< "$s"
[ "${f[2]}" = "$ID" ] || refuse open-activation "$O_ACT started for ${f[2]}, not $ID"
if read_pointer; then [ "$P_BUSINESS" = "$BUSINESS" ] || refuse business-mismatch "the pointer is for $P_BUSINESS"; fi
if [ -n "$ROLLBACK_OLD" ]; then
[ "$ROLLBACK_OLD" = "${f[1]}" ] || refuse rollback-mismatch "$ROLLBACK_OLD is not the copy $O_ACT started from, ${f[1]:-none}"
else
[ ! -e "$STATE/stack-rollback/$O_ACT/FAILED" ] || refuse activation-failed "$O_ACT has a FAILED file; root runs revert and confirm --rollback"
target_schema "$ID" || refuse copy-schema-unknown "cannot read SCHEMA_VERSION from $COPIES/$ID/packages/bus/src/store.mjs"
fi
OP="${f[0]}" FROM="${f[1]}" TO="${f[2]}" FLOOR="${f[3]}" ACT="$O_ACT" RS="$STATE/stack-rollback/$O_ACT"
STEPS="$RS/steps.log"
BACKUP="$(cat "$RS/BACKUP" 2>/dev/null || true)"
if [ -z "$ROLLBACK_OLD" ]; then
read_units || ext_fail units-unreadable
P_CONSOLE=("${U_CONSOLE[@]}")
gate "$TO" "$U_PM" || ext_fail "$GATE"
write_pointer "$TO" "$U_PM" 0 "$U_CLAUDE" "$T_SCHEMA" "${U_CONSOLE[@]}"
log_line result ok
rm -f "$RS/AWAITING-UNITS"
echo "stack-release: $OP ok: $TO is live for $BUSINESS"
return
fi
local code rcode=""
if [ "$CAUSE" = A4 ]; then code=units-start
else code="$(head -n 1 "$RS/FAILED" 2>/dev/null || true)"; [ -n "$code" ] || code=unrecorded
fi
if [ -e "$RS/RESTORED" ]; then rcode=not-attempted
elif ! read_units; then rcode=units-unreadable
else
P_CONSOLE=("${U_CONSOLE[@]}")
gate "$ROLLBACK_OLD" "$U_PM" || rcode="$GATE"
fi
if [ -z "$rcode" ]; then
local r=failed
[ "$CAUSE" != A5 ] || r=gate-failed
log_line result "$r" code "$code" rollback ok
rm -f "$RS/AWAITING-UNITS" "$RS/FAILED"
echo "stack-release: $code; rolled back to $ROLLBACK_OLD, which passed the gate" >&2
exit 1
fi
say "rollback-failed: run $code, rollback $rcode"
log_line result rollback-failed code "$code" rollback failed rollbackCode "$rcode"
rm -f "$RS/AWAITING-UNITS" "$RS/FAILED"
echo "stack-release: rollback-failed: the run failed with $code and the rollback with $rcode; root stops the units. Rollback set: $RS${BACKUP:+, backup $DATA/backups/stack/$BACKUP}" >&2
exit 2
}
revert_fail() {
say "revert failed: $1"
echo "stack-release: revert failed: $1; $O_ACT stays open. Rollback set: $RS" >&2
exit 1
}
# The store already holds backup $1 when every file checks against its SHA256SUMS and the listing
# matches, as after a revert that restored it.
store_matches() {
local o="$DATA/backups/stack/$1"
[ -d "$DATA/bus" ] && (cd "$DATA/bus" && sha256sum -c --quiet "$o/SHA256SUMS") >/dev/null 2>&1 &&
[ "$(ls -1A "$DATA/bus")" = "$(ls -1A "$o/bus")" ]
}
# R2: puts the wrapper back and, after a migration, the backup taken before it. Each part can run
# again, so it serves an A3 that stopped partway. A rollback's --restore swap is confirm's to
# report, not revert's to undo. Writes no log line.
do_revert() {
lock_only
open_activation
RS="$STATE/stack-rollback/$O_ACT" STEPS="$STATE/stack-rollback/$O_ACT/steps.log"
[[ "$O_ACT" =~ ^([0-9]{8}T[0-9]{6}Z)-[0-9a-f]{12}(-[1-9])?$ ]] || revert_fail "$O_ACT is not an activation id"
POST="$DATA/bus.post-attempt-${BASH_REMATCH[1]}${BASH_REMATCH[2]}"
[ -f "$RS/wrapper" ] && [ -f "$RS/WRAPPER_MODE" ] || revert_fail "the rollback set has no saved wrapper"
local tmp
tmp="$(mktemp "$BIN/.$WRAPPER_NAME.XXXXXX")" && cp -p "$RS/wrapper" "$tmp" && chmod "$(cat "$RS/WRAPPER_MODE")" "$tmp" && mv -f "$tmp" "$BIN/$WRAPPER_NAME" ||
revert_fail "could not put the wrapper back"
[ "$(sha256sum < "$BIN/$WRAPPER_NAME" | cut -d' ' -f1)" = "$(sum_of "$RS" wrapper)" ] &&
[ "$(stat -c %a "$BIN/$WRAPPER_NAME")" = "$(cat "$RS/WRAPPER_MODE")" ] || revert_fail "the wrapper does not match the rollback set"
say "put back the wrapper from $RS"
if [ -e "$RS/MIGRATING" ]; then
TAKEN="$(cat "$RS/BACKUP" 2>/dev/null || true)"
[ -n "$TAKEN" ] || revert_fail "MIGRATING is set and BACKUP names no backup"
if store_matches "$TAKEN"; then say "the store already matches backup $TAKEN"
else restore_store "$TAKEN" || revert_fail "could not restore backup $TAKEN"
fi
fi
echo "stack-release: revert ok: $O_ACT's wrapper$([ -e "$RS/MIGRATING" ] && echo ' and store are' || echo ' is') back; root installs the old units next"
}
# A2f: gates copy ID with no open activation, on a start at or after --since. Takes no lock and
# writes nothing, so it refuses nothing: 0 on a pass, 1 otherwise.
do_gate() {
FLOOR="$SINCE"
read_units || { echo "stack-release: gate: cannot read $BUS_UNIT or $CONSOLE_UNIT" >&2; exit 1; }
P_CONSOLE=("${U_CONSOLE[@]}")
gate "$ID" "$U_PM" || exit 1
}
# --- pm-on (row 169 item 3) ---
# Stops the bus alone and waits for it and the stopped check, within --timeout. The Console runs on.
stop_bus() {
systemctl --user stop "$BUS_UNIT" || return 1
sc stop "$BUS_UNIT" || return 1
local deadline=$((SECONDS + TIMEOUT)) s
while :; do
s="$(systemctl --user is-active "$BUS_UNIT" 2>/dev/null || true)"
s="$(sc is-active "$BUS_UNIT" 2>/dev/null || true)"
case "$s" in inactive|failed) if stopped; then say "stopped $BUS_UNIT"; return 0; fi ;; esac
[ "$SECONDS" -lt "$deadline" ] || return 1
sleep 1
@@ -1326,7 +1710,7 @@ stop_bus() {
start_bus() {
START_EPOCH="$(date +%s)" && START_MS="$(date +%s%3N)" || return 1
systemctl --user start "$BUS_UNIT"
sc start "$BUS_UNIT"
}
# mos-0848 point 3: put the phase-1 bus unit back (no --pm), restart it and rerun the phase-1 gate.
@@ -1348,8 +1732,8 @@ pm_on_fail() {
exit 1
fi
local pwhy="$GATE_WHY"
systemctl --user stop "$CONSOLE_UNIT" || true
systemctl --user stop "$BUS_UNIT" || true
sc stop "$CONSOLE_UNIT" || true
sc stop "$BUS_UNIT" || true
say "pm-on-down: pm-on $code, phase 1 $pcode"
log_line result pm-on-down code "$code" why "${why:-@null}" phase1 "$pcode" phase1Why "${pwhy:-@null}"
echo "stack-release: pm-on-down: pm-on failed with $code${why:+ ($why)} and the phase-1 bus with $pcode${pwhy:+ ($pwhy)}; both units are stopped. Rollback set: $RS" >&2
@@ -1359,6 +1743,7 @@ pm_on_fail() {
# Phase 2: the bus unit alone, on the pointer's copy with --pm, gated with pm_launched.
do_pm_on() {
lock
[ "$UNITS" = user ] || refuse external-pm "under --units external the PM is a flag in root's bus unit; changing it is a re-install with its own go"
pointer_for_business
[ "$P_PENDING" = 1 ] || refuse not-pending "the pointer is not pending; pm-on follows activate --pm-after-harness"
local out
@@ -1395,8 +1780,16 @@ do_status() {
local open
open="$(interrupted | paste -sd' ' -)"
[ -z "$open" ] || echo "interrupted run: $open (started, no final line; Jason or Mos settles it)"
if [ "$UNITS" = external ]; then
local m
local -a markers
mapfile -t markers < <(compgen -G "$STATE/stack-rollback/*/AWAITING-UNITS" || true)
for m in "${markers[@]}"; do
echo "awaiting units: $(basename "$(dirname "$m")") (copy $(cat "$m" 2>/dev/null || echo unreadable))"
done
fi
echo "check:"
if scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1 | sed 's/^/ /'; then echo " result: pass"; else echo " result: FAIL"; fi
if scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" "${UNITS_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1 | sed 's/^/ /'; then echo " result: pass"; else echo " result: FAIL"; fi
[ -n "$P_COPY" ] || return 0
echo "bus status:"
"$COPIES/$P_COPY/scripts/mosaic" bus status 2>&1 | sed 's/^/ /' || true
@@ -1722,11 +2115,16 @@ ID=""
case "$OP" in
activate|rehearse) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;;
adopt|pm-on|rollback|status) [ $# -ge 1 ] || usage; BUSINESS="$1"; shift ;;
confirm|revert|gate) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;;
-h|--help) usage 0 ;;
*) usage ;;
esac
[[ "$BUSINESS" =~ ^[a-z0-9][a-z0-9-]*$ ]] || usage
parse_opts "$@"
# Row 232: the stack-copy.sh check flags and the systemctl the instance-busy message names.
UNITS_FLAGS=()
SC_NAME="systemctl --user"
if [ "$UNITS" = external ]; then UNITS_FLAGS=(--units external); SC_NAME=systemctl; fi
COPIES_DIR="$COPIES"
instance_select
load_config
@@ -1741,6 +2139,7 @@ STORE_SCHEMA="" START_EPOCH="" START_MS=0 GATE="" GATE_WHY="" S_CHANGED=""
P_COPY="" P_PM=0 P_PENDING=0 P_SCHEMA="" P_CLAUDE="" P_CONSOLE=() RUN_PM=0 RUN_PENDING=0 RUN_CONSOLE=() DROP_TESTER="" PROOF_PLAN="" PROOF_STEP="" SEND_PLAN="" SEND_STEP="" PLAN_REFUSE=()
BASE_CONSOLE=() SERVE_FLAGS=() SSO_PLAN="" SSO_STEP="" SSO_DROP="" SSO_NO_TARGET=""
R="" R_RC=0 R_EARLY=() R_BK="" R_HOST_PID="" R_CONSOLE_PID="" R_STARTED="" LIVE_CFG="" LIVE_BIZ=""
O_ACT="" FLOOR=0
case "$OP" in
adopt) do_adopt ;;
activate) do_activate ;;
@@ -1748,4 +2147,7 @@ case "$OP" in
rollback) do_rollback ;;
status) do_status ;;
rehearse) do_rehearse ;;
confirm) do_confirm ;;
revert) do_revert ;;
gate) do_gate ;;
esac