stack-release: --units external for root-owned units, with confirm, revert and gate --since; stack-copy copy --bundle and wrapper --create (row 232, #1734)
Under --units user every command line and every byte of output is the base's (argv and --plan tests against 00d4cc1f6). Under external the script reads root's system units, starts and stops none, stages a copy behind AWAITING-UNITS, and confirm gates root's start on the bus unit's ActiveEnterTimestamp and launches list. Tests: stack-release-external.test.mjs (25), three stack-copy tests; helpers move to stack-release-fixture.mjs. Docs: TOOLS.md and the cli README. Record: agents/darkwing/work/queue-232/BUILD.md.
This commit is contained in:
@@ -0,0 +1,207 @@
|
||||
# Row 232 (I1): stack-release external units and bundle copies
|
||||
|
||||
Brief: `docs/plans/2026-10-20_installer-rows.md` section I1, blob `f35e7e57daee` (commit ad6f0be7a). Gitea
|
||||
#1734, Vikunja P19 #432. Owner Darkwing, reviewer Filbert, gate owner Sage. Design input: row 195 DESIGN r6,
|
||||
`agents/darkwing/work/queue-195/DESIGN.md` at b7239bb66 (approved in duckworth-0055). Where the brief and
|
||||
r6 differ, the brief wins. sage-1561 ruled where the stack user's `stack-release.sh` runs: from its own
|
||||
clone of the release bundle, so `check_target`, the plan's commit list and status drift are unchanged and
|
||||
nothing adds `safe.directory`.
|
||||
|
||||
## Files
|
||||
|
||||
Base: cb5fdb064 (row 199's landing, on top of row 200's 9e1085816). Branch `darkwing/row232-r1`. The first
|
||||
build, 0af0c73c6 on `darkwing/row232`, sat on 00d4cc1f6 (row 190); row 200 landed first, so this one is
|
||||
rebased onto it (see D9 and D10).
|
||||
|
||||
- `scripts/stack-release.sh`: `--units user|external`, the `sc` and `jctl` backend, `confirm`, `revert`,
|
||||
`gate --since`, `rollback --to`, `--plan --expect-from`, the external gate (start floor, PM check from
|
||||
`launches list`), `external-pm`, and the `awaiting units:` status line.
|
||||
- `scripts/stack-copy.sh`: `check --units external`, `copy --bundle FILE SHA`, `wrapper ID --create`.
|
||||
- `packages/cli/tests/stack-release-external.test.mjs` (new, 25 tests).
|
||||
- `packages/cli/tests/stack-release-fixture.mjs` (new): the helpers `stack-release.test.mjs` defined, moved
|
||||
so both files share them. See deviation D1.
|
||||
- `packages/cli/tests/stack-release.test.mjs`: the helpers leave and one import line comes in. The 74 tests
|
||||
have the same names and the same bodies as on the base.
|
||||
- `packages/cli/tests/stack-copy.test.mjs`: three new `row 232:` tests and a wider `node:fs` import.
|
||||
- `docs/TOOLS.md`: the stack-copy usage block and three paragraphs, the stack-release usage block, and a new
|
||||
subsection "Root-owned units (`--units external`, row 232)".
|
||||
- `packages/cli/README.md`: one paragraph.
|
||||
- This file.
|
||||
|
||||
The worktree's `node_modules` symlink is not part of the candidate.
|
||||
|
||||
## User mode is unchanged
|
||||
|
||||
Every `systemctl` call goes through `sc` and every `journalctl` call through `jctl`. Under `user` they
|
||||
are `systemctl --user "$@"` and `journalctl --user "$@"`. Under `external`, `sc` allows only `show`,
|
||||
`is-active` and `list-units`, and `jctl` returns 1, so an external run can't start, stop, install or read
|
||||
the journal even through a path the tests miss.
|
||||
|
||||
Two tests hold the brief's "every command line and every byte of output":
|
||||
|
||||
- `user mode: every systemctl call carries --user, and activate, rollback and check make the base's
|
||||
calls`. A fixture repo gets the base's two scripts from `git show cb5fdb064:`. The same activate,
|
||||
rollback and `stack-copy.sh check` run from both trees, and the recorded argv (with the fixture root and
|
||||
port replaced by placeholders) must match line for line. Every call carries `--user`.
|
||||
- `user mode: a default activate --plan prints the base's bytes`, for four plans.
|
||||
|
||||
The base output is produced live from the base blobs, not stored as a fixture file. See deviation D3.
|
||||
|
||||
## r6 section 7, test by test
|
||||
|
||||
All in `stack-release-external.test.mjs` unless named.
|
||||
|
||||
| r6 :742-812 item | Test |
|
||||
|---|---|
|
||||
| `units-running` (N4), store, log and TAKEN left | `units-running (N4) refuses while a unit is active...`; also checks that `failed` counts as down and that the Console is checked first |
|
||||
| activate leaves the pointer, writes `AWAITING-UNITS` and a backup | `activate stages: ...`, which also checks the set's file list, `UNITS.sha256`, no `--user`, no journal call and `status` |
|
||||
| `open-activation` with zero, two, the wrong copy, one not open | `confirm and revert refuse open-activation ...` |
|
||||
| `confirm` polls: 3 s fails at `--timeout 1`, passes at `--timeout 10` | `confirm polls: bus status good after 3 s fails ...` and `confirm polls to a pass ...` |
|
||||
| `stale-start` floor with `started` at .900 | `stale-start: @1792540799 fails ...` and `stale-start is a gate code ...` |
|
||||
| `confirm` pass writes pointer, ok, removes marker; timeout writes `FAILED` and exits 1 | the two polling tests |
|
||||
| `activation-failed` when `FAILED` exists | `confirm polls: ... FAILED then refuses activation-failed` |
|
||||
| lines carry the started op, from, to, activation and backup | `sameAsStarted` in every confirm test, and sequence 7 |
|
||||
| `gate` (N2) with and without `--since` | `gate (N2, A2f): ...` and the usage test |
|
||||
| `--plan --expect-from` (N5), activate and `rollback --to` | sequence 7 |
|
||||
| A3 order: capture-failed, marker-failed, wrapper failure after migrate | `A3 order: ...` and sequence 2 |
|
||||
| refusals log no activation; `interrupted` unchanged | `open-activation` test and sequence 5 |
|
||||
| PM check: one, none, two, early, not live, other business, other instance; no journal call | `the external PM check reads launches list ...` (modes '', coder, none, twolive, early, dead, other) and `journalCalls` in every external test |
|
||||
| `external-pm` | `pm-on and activate --pm-after-harness refuse external-pm` |
|
||||
| `revert` restores TAKEN only after a migration | `revert puts the wrapper back and leaves the store ...` and sequences 2 and 8 |
|
||||
| `confirm --rollback` per cause, `unrecorded`, `rollback-failed`, mismatch | sequences 1, 2, 3, 5 and 8, and `confirm --rollback refuses rollback-mismatch, and logs rollback-failed ...` |
|
||||
| sequences 1-8 | `sequence 1:` to `sequence 8:`, each ending with the `closed` check (one started and one result line, empty `interrupted`, no marker or `FAILED`, a following `--plan` that doesn't refuse `interrupted-run`) |
|
||||
| `rollback --to` refuses not-previous; `rollback --restore` finds an external backup | sequence 7, and `confirm --rollback after a --restore swap ...` |
|
||||
| `gate` writes nothing | the `gate (N2, A2f)` test and sequence 4 compare the log, the pointer and the state tree |
|
||||
| `confirm`, `revert`, `gate`, `--to` refuse without `--units external` | `confirm, revert, gate, --to and --expect-from need --units external ...` |
|
||||
| default path: every call carries `--user`; default `--plan` bytes | the two user-mode tests above |
|
||||
|
||||
The brief's additions, in `stack-copy.test.mjs`:
|
||||
|
||||
- `row 232: copy --bundle copies a SHA the bundle's refactor reaches, byte-identical to copy SHA, and
|
||||
refuses the rest`: a short SHA, a missing file, a symlink, a bundle that fails `git bundle verify`, a
|
||||
bundle with no `refs/heads/refactor`, a SHA not in the bundle, and a SHA the bundle carries on another
|
||||
branch that refactor doesn't reach. The bundle copy's tree matches `copy SHA`'s byte for byte, and the
|
||||
`release:` and `bundle sha256:` lines are checked.
|
||||
- `row 232: wrapper --create writes the first wrapper only, the same bytes wrapper ID writes, and refuses
|
||||
over any file`, a symlink included. Without `--create` a missing wrapper still refuses.
|
||||
- `row 232: check --units external reads the system units, the default reads the user units`.
|
||||
|
||||
## Mutants
|
||||
|
||||
`mutants.sh` in my scratch (not committed, it edits the worktree), 17 mutants: r6's re-pin list (:817-838)
|
||||
minus the three `case1-repin.sh` ones, plus the brief's two. Each one applies exact replacements after a
|
||||
checked backup, runs `bash -n`, runs its named killers, restores the file and checks its sha256. Two
|
||||
controls run first with no change: the whole external file, and stack-copy's `row 232:` tests.
|
||||
|
||||
| Mutant | Killers |
|
||||
|---|---|
|
||||
| m01 `confirm` writes the pointer on a gate failure | `confirm polls: bus status` |
|
||||
| m02 `gate_once` in place of `gate` | `confirm polls to a pass`, `stale-start is a gate code` |
|
||||
| m03 PM check counts any instance | `external PM check` |
|
||||
| m04 PM check drops the `startedAt` bound | `external PM check` |
|
||||
| m05 `confirm` takes the newest `AWAITING-UNITS` | `open-activation` |
|
||||
| m06 `rollback --to` skips the comparison | `sequence 7:` |
|
||||
| m07 the default path drops `--user` from one call (`is-active`) | `user mode: every systemctl` |
|
||||
| m08 the timeout writes r4's interim `rollback pending` line | `confirm polls: bus status`, `sequence 1:`, `sequence 5:` |
|
||||
| m09 `AWAITING-UNITS` last, after the wrapper | `sequence 2:` |
|
||||
| m10 a refusal names the open activation | `sequence 5:`, `open-activation` |
|
||||
| m11 `stale-start` is a refusal | `stale-start is a gate code` |
|
||||
| m12 `revert` restores without `MIGRATING` | `revert puts the wrapper back` |
|
||||
| m13 `confirm` logs op `confirm` | `sequence 7:`, `confirm polls to a pass` |
|
||||
| m14 `MIGRATING` after the migration | `sequence 8:` |
|
||||
| m15 `gate` takes its floor from the log without `--since` | `gate (N2` |
|
||||
| m16 `copy --bundle` skips the reachability check | stack-copy `row 232:` |
|
||||
| m17 `wrapper --create` overwrites | stack-copy `row 232:` |
|
||||
|
||||
## Deviations and choices
|
||||
|
||||
D1. The shared helpers moved to a new file, `packages/cli/tests/stack-release-fixture.mjs`. The brief's
|
||||
files list names `stack-release.test.mjs` "where a shared helper moves" but not a destination. Importing
|
||||
from the other test file would run its 74 tests twice, so the helpers need a module of their own.
|
||||
|
||||
D2. The fixture's T17 check (what a run may never remove) now exempts a rollback set's `AWAITING-UNITS` and
|
||||
`FAILED`. `confirm` removes both when it closes the activation (r6 A5 and R4). User mode writes neither, so
|
||||
user-mode runs check exactly what they did.
|
||||
|
||||
D3. The user-mode argv and `--plan` tests read the base's scripts with `git show cb5fdb064:` rather than
|
||||
storing their output as a fixture file. A later row that changes user-mode output on purpose moves the
|
||||
`BASE` constant in the test to its own base, or the test fails. I chose this over a stored fixture because
|
||||
a stored copy can drift from the base without anyone noticing.
|
||||
|
||||
D4. `marker-failed` is reached by making `state/stack-rollback` unwritable. r6 says "an existing `$RS`", but
|
||||
`next_activation` never picks a taken name, so no test can make the run meet an existing `$RS`. The
|
||||
exclusive `mkdir` is still what refuses.
|
||||
|
||||
D5. The stale-start polling case turns the enter time to `@1792540800`, not r6's `@1792540805`, with the
|
||||
started line at 1792540800.900. That also checks that the floor is the whole second, so .900 doesn't push it
|
||||
to 1792540801.
|
||||
|
||||
D6. No fake `sudo` or `case2-unit.sh`. r6's sequences ran under them because r6 had `case1-repin.sh` drive
|
||||
the steps, and that script is I3's. Here the test plays root: it stops and starts the fake system units and
|
||||
installs with the existing `bus-service.sh` and `webui-service.sh` stand-ins, between the real
|
||||
`stack-release.sh` calls, on one log.
|
||||
|
||||
D7. The fake marker in the `open-activation` test is named `20000101T000000Z-<C1 prefix>` so it sorts
|
||||
before the real one. A marker that sorted after it would leave m05 (newest marker) alive.
|
||||
|
||||
D8. m10 sets `ACT` inside `open_activation`, where r6's wording ("ACT set before the checks") lands in this
|
||||
code. `ACT` is set only after every check in `confirm`.
|
||||
|
||||
D9. Row 200's `--sso` is a usage error under `--units external`, with the proof flags, `--no-proof` and
|
||||
`--no-send`. It is a Console flag, and root's Console unit carries the Console flags; the script under
|
||||
external renders no Console line and never calls `console_for_target`, so `sso_checks` doesn't run there
|
||||
either. The usage text, the TOOLS.md bullet and the external usage test name it.
|
||||
|
||||
D10. Row 200's `config_checks` runs under `--units external` too, after `plugin_checks`, in both the
|
||||
`--plan` and run paths of `activate` and `rollback --to`. It checks the target's loader against the stack
|
||||
user's `config.json`, which the bus on a VM still reads, so a `console` key the target refuses would break
|
||||
root's bus start the same way. The cost is one more refusal (`config-console-key`) an external run can
|
||||
print; I took it over a bus that fails after root's start. The rebase changed no user-mode line: the
|
||||
conflicts were the `SSO=0` globals line, the `plan` body (row 200's `SSO_PLAN` line sits in the user
|
||||
branch) and the two `--plan` lines that gained `config_checks`. In the tests, row 200's fixture lines
|
||||
(the loaders, `sso-config.mjs` and the `--sso` cli.mjs line for C1 and C) moved into
|
||||
`stack-release-fixture.mjs` with the rest of the copy setup, and the `ALGS` import stays in
|
||||
`stack-release.test.mjs`, which T57 uses.
|
||||
|
||||
## duckworth-0055's notes on r6 (brief line 169)
|
||||
|
||||
Duckworth's r6 approval (REVIEW-r5.md at 23f5520b5) carried two notes.
|
||||
|
||||
- N1, A2's stop timeout under `gate --since`. A unit that never stopped keeps its old `ActiveEnterTimestamp`,
|
||||
so `gate --since` fails it with `stale-start`, and the run ends at exit 2 with root stopping the units.
|
||||
That is intended: a unit that won't stop is a fair reason to stop. In I1 this is the gate code and its
|
||||
polling (`stale-start: @1792540799 fails ...`, `gate (N2, A2f)`, m11). The run sheet's A2f line and the
|
||||
re-pin test that points "exits 1 with OLD's gate passing" at an A3 exit 3 belong to I3, which owns
|
||||
`scripts/stack-upgrade.sh` and `docs/guides/host-run-sheet.md`. I'll name both in the I3 build.
|
||||
- N2, the claude path under `/opt/mosaic-stack` (gosalyn-0058, sage-1522). I1 doesn't touch it. Under
|
||||
`--units external` the script checks no claude pin of its own (`no-claude-pin` applies only to user
|
||||
mode), because root's unit carries it. The `--claude` value, the managed-settings reasoning, F6 and the
|
||||
run sheet's claude step go to I2 and I3.
|
||||
|
||||
## Run environment
|
||||
|
||||
Worktree `/mnt/storage/scratch/darkwing-r232`, `TMPDIR=/mnt/storage/scratch/tmp/r232`, node from PATH.
|
||||
One external file run takes about 94 s.
|
||||
|
||||
## Results
|
||||
|
||||
Singles on the rebased candidate, before the gate:
|
||||
|
||||
| File | Tests | Pass | Fail |
|
||||
|---|---|---|---|
|
||||
| `packages/cli/tests/stack-release-external.test.mjs` | 25 | 25 | 0 |
|
||||
| `packages/cli/tests/stack-copy.test.mjs` | 16 | 16 | 0 |
|
||||
| `packages/cli/tests/stack-release.test.mjs` (row 200's T57 included) | 192 | 192 | 0 |
|
||||
| `packages/cli/tests/stack-instances.test.mjs` | 23 | 23 | 0 |
|
||||
| `packages/webui/tests/pinned-copy.test.mjs` | 10 | 10 | 0 |
|
||||
| `packages/cli/tests/channel-recovery.test.mjs` (imports `channel-fixture.mjs`, which names the scripts) | 1 | 1 | 0 |
|
||||
| `packages/cli/tests/host.test.mjs` (reads `packages/cli/README.md`) | 42 | 42 | 0 |
|
||||
|
||||
These are every test file that reads `stack-release.sh` or `stack-copy.sh` (a grep of `packages/*/tests` and
|
||||
`scripts/`). None of them pins a sha256 of a changed file. `packages/cli/tests/host.test.mjs` reads
|
||||
`packages/cli/README.md` for a unit file name; this row only adds a paragraph there. The grep and the runs
|
||||
were repeated on the rebased tree on 2026-10-21; the reader set is the same. A dry run of `mutants.sh`
|
||||
matches each of the 17 replacements once on the rebased scripts.
|
||||
|
||||
The gate (the cli suite and every `scripts/test-*.sh`) and the 17 mutants run in Sage's slot. Their results
|
||||
go below when they're in.
|
||||
+72
-4
@@ -408,8 +408,9 @@ exposure of a named Console waits for Jason after Q175 SSO.
|
||||
|
||||
```
|
||||
scripts/stack-copy.sh copy COMMIT [--copies DIR] [--allow-worktree]
|
||||
scripts/stack-copy.sh wrapper ID [--instance NAME] [--copies DIR] [--bin DIR]
|
||||
scripts/stack-copy.sh check [--instance NAME] [--copies DIR] [--bin DIR] [--business ID]
|
||||
scripts/stack-copy.sh copy --bundle FILE SHA [--copies DIR]
|
||||
scripts/stack-copy.sh wrapper ID [--instance NAME] [--copies DIR] [--bin DIR] [--create]
|
||||
scripts/stack-copy.sh check [--instance NAME] [--copies DIR] [--bin DIR] [--business ID] [--units user|external]
|
||||
```
|
||||
|
||||
One read-only tree supplies the bus, Console and human CLI (row 85,
|
||||
@@ -423,6 +424,14 @@ renames beside the target. A published existing copy stays unchanged.
|
||||
Dependencies use the exact lock and carry Pi's installed lock record.
|
||||
Failure publishes nothing and removes the temporary tree.
|
||||
|
||||
`copy --bundle FILE SHA` (row 232) is for a host with no refactor checkout.
|
||||
SHA is a full 40-character id. The bundle must pass `git bundle verify`,
|
||||
carry `refs/heads/refactor`, and reach SHA from it; each miss refuses and
|
||||
writes nothing. The copy is then made from SHA as `copy` makes it, and the
|
||||
script prints `release:` with the RELEASE value at SHA and `bundle sha256:`
|
||||
with the bundle's digest. That pair is the release identity on such a host.
|
||||
Without `--bundle`, `copy`'s reachability rule is unchanged.
|
||||
|
||||
`wrapper` is a live switching step, reserved for Jason or Mos. It accepts
|
||||
only an existing copy and a regular non-symlink mosaic wrapper with the
|
||||
Q81 marker on line 2 and its exact lease line to mosaic-old. It preserves
|
||||
@@ -430,11 +439,16 @@ both lines, adds comments naming the copy root and Q81's mosaic-old lease
|
||||
probe, and atomically replaces the wrapper's final exec with the
|
||||
copy's scripts/mosaic. --bin is a fixture override; the default is
|
||||
~/.npm-global/bin. Neither the Q81 worktree nor mosaic-old is removed.
|
||||
`wrapper ID --create` (row 232) writes the first wrapper on a new host, the
|
||||
same file `wrapper` writes. It refuses when anything is at the wrapper path,
|
||||
a symlink included, and never replaces it. Without `--create`, a missing
|
||||
wrapper still refuses.
|
||||
`check` changes nothing: it reads effective systemd WorkingDirectory and
|
||||
ExecStart (including drop-ins) for the bus and Console, plus the wrapper
|
||||
exec. It prints all three paths and succeeds only on the same existing
|
||||
copy; absent/failed reads and a mismatched ExecStart fail. Business defaults
|
||||
to mosaic-stack. Queue work continues on the canonical scripts/mosaic.
|
||||
to mosaic-stack. `check --units external` (row 232) reads root's system units
|
||||
with `systemctl show`, without `--user`. Queue work continues on the canonical scripts/mosaic.
|
||||
|
||||
Bus render/install accept --copy ID, --copies DIR, --pm and the unchanged
|
||||
--claude pin. Without --copy the rendered unit stays the checkout unit.
|
||||
@@ -455,7 +469,10 @@ scripts/stack-release.sh activate ID BUSINESS [--pm | --pm-after-harness] [--mig
|
||||
scripts/stack-release.sh pm-on BUSINESS [--timeout S]
|
||||
scripts/stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan]
|
||||
scripts/stack-release.sh status BUSINESS
|
||||
common options: --copies DIR, --bin DIR, --instance NAME
|
||||
scripts/stack-release.sh confirm ID BUSINESS --units external [--rollback OLD --cause A3|A4|A5] [--timeout S]
|
||||
scripts/stack-release.sh revert ID BUSINESS --units external
|
||||
scripts/stack-release.sh gate ID BUSINESS --units external --since EPOCH [--timeout S]
|
||||
common options: --copies DIR, --bin DIR, --instance NAME, --units user|external
|
||||
--events N is the total of the per-table counts the post-backup-events refusal prints: events and
|
||||
every table in COUNTED.
|
||||
```
|
||||
@@ -711,6 +728,57 @@ Usage exits 4. Nothing prunes rehearsal directories.
|
||||
starts no thread ticker, and leaves an earlier host's `sessions.json` and
|
||||
its runs alone. No unit passes it, and `--pm` with it is a usage error.
|
||||
|
||||
#### Root-owned units (`--units external`, row 232)
|
||||
|
||||
On a host whose bus and Console are root's system units, the stack user runs
|
||||
`stack-release.sh` with `--units external` from its own clone of the release
|
||||
bundle (sage-1561). The script then calls `systemctl show`, `is-active` and
|
||||
`list-units` without `--user` and reads no journal. It starts, stops and
|
||||
installs nothing. Root does those steps, and the I3 driver calls the verbs
|
||||
below in order. Without `--units`, every command line and every byte of
|
||||
output is what it was before row 232. `confirm`, `revert`, `gate`, `--to`
|
||||
and `--expect-from` are usage errors without `--units external`.
|
||||
|
||||
- `adopt`, `status`, `activate ID` and `rollback --to ID` take the flag.
|
||||
`rollback` then needs `--to`, and `--to` must be the copy `rollback` alone
|
||||
would pick from the log, or it refuses `to-not-previous`.
|
||||
- `--plan --expect-from OLD` refuses `pointer-differs` when the pointer
|
||||
names a copy other than OLD.
|
||||
- `pm-on` and `activate --pm-after-harness` refuse `external-pm`; the PM
|
||||
flag is in root's bus unit. `--pm`, `--tester-port`, the proof flags,
|
||||
`--no-proof`, `--no-send` and `--sso` are usage errors; root's Console
|
||||
unit carries the Console flags.
|
||||
- `activate` and `rollback` refuse `units-running` unless both units read
|
||||
inactive or failed and the store is closed. They make the rollback set
|
||||
with an exclusive `mkdir`, write `AWAITING-UNITS` (the target copy) first,
|
||||
capture the pointer, the wrapper and both units' show output and unit file
|
||||
sums, back up and migrate or restore the store, and write the wrapper.
|
||||
They write no pointer and start nothing. The last line says the run is
|
||||
staged and names `confirm`. A failure after the capture writes `FAILED`
|
||||
with its code, logs nothing more and exits 1; the activation stays open.
|
||||
- `confirm ID BUSINESS` needs exactly one `AWAITING-UNITS` for copy ID whose
|
||||
activation the log holds open, or it refuses `open-activation`. It gates
|
||||
the start root made and writes the pointer and an `ok` line with the
|
||||
started line's op, from, to and backup.
|
||||
`confirm --rollback OLD --cause A3|A4|A5` gates OLD after root put the old
|
||||
units back and logs `failed` or `gate-failed` with `rollback ok` (exit 1)
|
||||
or `rollback-failed` (exit 2). Either removes `AWAITING-UNITS` and `FAILED`.
|
||||
- `revert ID BUSINESS` puts the saved wrapper back and, when the run had
|
||||
started a migration, restores the backup taken before it. It writes no log
|
||||
line and can run again.
|
||||
- `gate ID BUSINESS --since EPOCH` gates a copy with no open activation,
|
||||
takes no lock and writes nothing: 0 on a pass, 1 otherwise. EPOCH is root's
|
||||
clock before root started the units.
|
||||
|
||||
The gate under `--units external` replaces the journal read. The bus unit's
|
||||
`ActiveEnterTimestamp`, in whole seconds, must be at or after the started
|
||||
line's second (`confirm`) or `--since` (`gate`); an older start fails
|
||||
`stale-start`, so a unit that never stopped can't pass. With the PM on, the
|
||||
gate reads `launches list --json` for exactly one live launch of the PM's
|
||||
instance (the business's `launch.by`), for this business, whose run started at
|
||||
or after the floor. `status` adds an
|
||||
`awaiting units:` line for an open activation.
|
||||
|
||||
## Console service (`scripts/webui-service.sh`)
|
||||
|
||||
```bash
|
||||
|
||||
@@ -773,6 +773,16 @@ Each live run requires Mos's go; agents build and test fixtures only, and
|
||||
so no PM turn may be in flight. Queue writes stay on the canonical
|
||||
scripts/mosaic.
|
||||
|
||||
On a host whose units are root's (row 232), `--units external` makes
|
||||
stack-release.sh read the system units and start or stop none. `activate`
|
||||
and `rollback --to` stage the copy and leave the `AWAITING-UNITS` marker;
|
||||
root installs and starts the units, then `confirm` gates the start and
|
||||
writes the pointer, `revert` undoes a staged run, and `gate --since` checks
|
||||
a copy with no open activation. stack-copy.sh adds `copy --bundle FILE SHA`
|
||||
for a host with no refactor checkout, `wrapper ID --create` for the first
|
||||
wrapper, and `check --units external`. Details are in
|
||||
[TOOLS.md, Root-owned units](../../docs/TOOLS.md#root-owned-units---units-external-row-232).
|
||||
|
||||
The human commands and `--tester` run only from the tree the bus runs from
|
||||
(row 130, #1625). The broker checks the path of the calling human-cli.mjs or
|
||||
tester-cli.mjs against its own. On a host with a pinned copy that tree is
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {execFileSync,spawnSync,spawn} from 'node:child_process';
|
||||
import {symlinkSync,unlinkSync,mkdtempSync,mkdirSync,writeFileSync,readFileSync,cpSync,rmSync,chmodSync,existsSync,statSync} from 'node:fs';
|
||||
import {symlinkSync,unlinkSync,mkdtempSync,mkdirSync,writeFileSync,readFileSync,cpSync,rmSync,chmodSync,existsSync,statSync,readdirSync,lstatSync,readlinkSync} from 'node:fs';
|
||||
import {createHash} from 'node:crypto';
|
||||
import {tmpdir} from 'node:os';
|
||||
import {join,dirname} from 'node:path';
|
||||
import {fileURLToPath} from 'node:url';
|
||||
@@ -227,3 +228,108 @@ test('inherited drop-in unreadable directory or file refuses before writing',t=>
|
||||
unlinkSync(dropdir);mkdirSync(dropdir);const file=join(dropdir,'broken.conf');symlinkSync(join(s.root,'absent-file'),file);
|
||||
const broken=s.run('bus-service.sh',['install','--dir',units,'--no-reload']);assert.equal(broken.status,1);assert.ok(broken.stderr.includes(file));assert.equal(readFileSync(target,'utf8'),'previous unit bytes');
|
||||
});
|
||||
// Row 232: every file of a copy with its mode and bytes, for the byte-identical comparison.
|
||||
function tree(dir){
|
||||
const out=[];
|
||||
const walk=rel=>{for(const e of readdirSync(join(dir,rel),{withFileTypes:true}).sort((a,b)=>a.name<b.name?-1:1)){
|
||||
const r=rel?`${rel}/${e.name}`:e.name,st=lstatSync(join(dir,r));
|
||||
if(e.isDirectory()){out.push(`d ${(st.mode&0o7777).toString(8)} ${r}`);walk(r);}
|
||||
else if(e.isSymbolicLink())out.push(`l ${r} ${readlinkSync(join(dir,r))}`);
|
||||
else out.push(`f ${(st.mode&0o7777).toString(8)} ${r} ${createHash('sha256').update(readFileSync(join(dir,r))).digest('hex')}`);
|
||||
}};
|
||||
walk('');return out;
|
||||
}
|
||||
test('row 232: copy --bundle copies a SHA the bundle\'s refactor reaches, byte-identical to copy SHA, and refuses the rest',t=>{
|
||||
const s=setup(t),tmp=join(s.root,'t');mkdirSync(tmp);s.env.TMPDIR=tmp;
|
||||
const good=join(s.root,'good.bundle');git(s.repo,'bundle','create','-q',good,'refactor');
|
||||
const a=join(s.root,'copies-a'),b=join(s.root,'copies-b');
|
||||
let r=s.run('stack-copy.sh',['copy',s.id,'--copies',a]);assert.equal(r.status,0,r.stderr);
|
||||
r=s.run('stack-copy.sh',['copy','--bundle',good,s.id,'--copies',b]);assert.equal(r.status,0,r.stderr);
|
||||
const sum=createHash('sha256').update(readFileSync(good)).digest('hex'),release=git(s.repo,'show',`${s.id}:RELEASE`);
|
||||
assert.equal(r.stdout,`copied: ${join(b,s.id)}\nrelease: ${release}\nbundle sha256: ${sum}\n`);
|
||||
assert.deepEqual(tree(join(b,s.id)),tree(join(a,s.id)));
|
||||
assert.equal(statSync(join(b,s.id)).mode&0o777,0o555);
|
||||
const left=()=>readdirSync(tmp).filter(f=>f.startsWith('tmp.'));
|
||||
assert.deepEqual(left(),[],'the bundle\'s temporary repository is removed');
|
||||
// A second run finds the copy and still names the release.
|
||||
r=s.run('stack-copy.sh',['copy','--bundle',good,s.id,'--copies',b]);assert.equal(r.status,0,r.stderr);
|
||||
assert.equal(r.stdout,`exists: ${join(b,s.id)}\nrelease: ${release}\nbundle sha256: ${sum}\n`);
|
||||
// A relative bundle path works from the caller's directory.
|
||||
const c=join(s.root,'copies-c');
|
||||
r=spawnSync(join(s.repo,'scripts','stack-copy.sh'),['copy','--bundle','good.bundle',s.id,'--copies',c],{cwd:s.root,env:s.env,encoding:'utf8'});
|
||||
assert.equal(r.status,0,r.stderr);assert.deepEqual(tree(join(c,s.id)),tree(join(a,s.id)));
|
||||
// A commit on another branch of the bundle: the bundle carries it, refactor does not reach it.
|
||||
const side=git(s.repo,'commit-tree','-p',s.id,'-m','side',`${s.id}^{tree}`);git(s.repo,'update-ref','refs/heads/side',side);
|
||||
const both=join(s.root,'both.bundle');git(s.repo,'bundle','create','-q',both,'refactor','side');
|
||||
const only=join(s.root,'side.bundle');git(s.repo,'bundle','create','-q',only,'side');
|
||||
const corrupt=join(s.root,'corrupt.bundle'),bytes=readFileSync(good);bytes[bytes.length-30]^=0xff;writeFileSync(corrupt,bytes);
|
||||
const cut=join(s.root,'cut.bundle');writeFileSync(cut,readFileSync(good).subarray(0,200));
|
||||
const junk=join(s.root,'junk.bundle');writeFileSync(junk,'# v2 git bundle\nnot a ref line\n\n');
|
||||
const d=join(s.root,'copies-d');
|
||||
for(const [label,bundle,id,re] of [
|
||||
['not reachable',both,side,/not reachable from the bundle's refs\/heads\/refactor/],
|
||||
['no refactor',only,side,/no refs\/heads\/refactor/],
|
||||
// verify reads the header and prerequisites; a damaged pack fails the fetch's own check.
|
||||
['not a bundle',junk,s.id,/git bundle verify failed/],
|
||||
['corrupt pack',corrupt,s.id,/cannot fetch/],
|
||||
['truncated pack',cut,s.id,/cannot fetch/],
|
||||
['absent',join(s.root,'none.bundle'),s.id,/bundle is absent/],
|
||||
['short id',good,s.id.slice(0,12),/full 40-character commit id/],
|
||||
]){
|
||||
r=s.run('stack-copy.sh',['copy','--bundle',bundle,id,'--copies',d]);
|
||||
assert.equal(r.status,1,label);assert.match(r.stderr,re,label);assert.equal(r.stdout,'',label);
|
||||
assert.ok(!existsSync(join(d,id)),label);assert.deepEqual(left(),[],label);
|
||||
if(existsSync(d))assert.deepEqual(readdirSync(d),[],label);
|
||||
}
|
||||
const link=join(s.root,'link.bundle');symlinkSync(good,link);
|
||||
r=s.run('stack-copy.sh',['copy','--bundle',link,s.id,'--copies',d]);assert.equal(r.status,1);assert.match(r.stderr,/symlink/);
|
||||
for(const args of [['copy','--bundle',good,s.id,'--allow-worktree'],['copy','--bundle',good],['copy','--bundle'],['wrapper','--bundle',good,s.id],['check','--units','system'],['copy',s.id,'--units','external'],['copy',s.id,'--create']])
|
||||
assert.equal(s.run('stack-copy.sh',args).status,4,args.join(' '));
|
||||
});
|
||||
test('row 232: wrapper --create writes the first wrapper only, the same bytes wrapper ID writes, and refuses over any file',t=>{
|
||||
const s=setup(t);assert.equal(s.run('stack-copy.sh',['copy',s.id,'--copies',s.copies]).status,0);
|
||||
const wrapper=join(s.bin,'mosaic'),root=join(s.copies,s.id);
|
||||
let r=s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin]);
|
||||
assert.equal(r.status,1);assert.match(r.stderr,/wrapper is absent/,'no --create keeps the missing-wrapper refusal');assert.ok(!existsSync(wrapper));
|
||||
r=s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin,'--create']);assert.equal(r.status,0,r.stderr);
|
||||
assert.equal(r.stdout,`wrapper: ${root}\n`);
|
||||
const created=readFileSync(wrapper,'utf8');assert.equal(statSync(wrapper).mode&0o777,0o755);
|
||||
r=s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin]);assert.equal(r.status,0,r.stderr);
|
||||
assert.equal(r.stdout,`wrapper: ${root}\n`);assert.equal(readFileSync(wrapper,'utf8'),created,'wrapper ID writes the same bytes');
|
||||
const foreign='#!/bin/sh\n# not ours\n';
|
||||
for(const [label,make] of [['own wrapper',()=>{}],['foreign file',()=>writeFileSync(wrapper,foreign)],['symlink',()=>{rmSync(wrapper);symlinkSync(join(s.root,'elsewhere'),wrapper);}]]){
|
||||
make();const before=existsSync(wrapper)||label==='symlink'?(label==='symlink'?readlinkSync(wrapper):readFileSync(wrapper,'utf8')):null;
|
||||
r=s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin,'--create']);
|
||||
assert.equal(r.status,1,label);assert.match(r.stderr,/--create and a wrapper exists/,label);
|
||||
assert.equal(label==='symlink'?readlinkSync(wrapper):readFileSync(wrapper,'utf8'),before,label);
|
||||
assert.deepEqual(readdirSync(s.bin).filter(f=>f.startsWith('.')),[],`${label}: no temporary file left`);
|
||||
}
|
||||
// A named instance: --create writes mosaic-NAME as wrapper --instance does, once.
|
||||
assert.equal(s.run('bootstrap.sh',['--instance','alpha']).status,0);
|
||||
const named=join(s.bin,'mosaic-alpha');
|
||||
r=s.run('stack-copy.sh',['wrapper',s.id,'--instance','alpha','--copies',s.copies,'--bin',s.bin,'--create']);assert.equal(r.status,0,r.stderr);
|
||||
const first=readFileSync(named,'utf8');
|
||||
r=s.run('stack-copy.sh',['wrapper',s.id,'--instance','alpha','--copies',s.copies,'--bin',s.bin]);assert.equal(r.status,0,r.stderr);
|
||||
assert.equal(readFileSync(named,'utf8'),first);
|
||||
r=s.run('stack-copy.sh',['wrapper',s.id,'--instance','alpha','--copies',s.copies,'--bin',s.bin,'--create']);
|
||||
assert.equal(r.status,1);assert.match(r.stderr,/--create and a wrapper exists/);assert.equal(readFileSync(named,'utf8'),first);
|
||||
});
|
||||
test('row 232: check --units external reads the system units, the default reads the user units',t=>{
|
||||
const s=setup(t);assert.equal(s.run('stack-copy.sh',['copy',s.id,'--copies',s.copies]).status,0);
|
||||
assert.equal(s.run('stack-copy.sh',['wrapper',s.id,'--copies',s.copies,'--bin',s.bin,'--create']).status,0);
|
||||
const copy=join(s.copies,s.id),argv=join(s.root,'argv.log');
|
||||
s.fake(`echo "$*" >> '${argv}'; echo 'LoadState=loaded'; echo 'WorkingDirectory=${copy}'; case "$*" in *mosaic-bus*) echo 'ExecStart={ path=${copy}/scripts/mosaic ; argv[]=${copy}/scripts/mosaic bus start mosaic-stack --pm ; }';; *) echo 'ExecStart={ path=/usr/bin/node ; argv[]=/usr/bin/node ${copy}/packages/webui/src/cli.mjs serve ; }';; esac\n`);
|
||||
const want=`bus: ${copy}\nConsole: ${copy}\nCLI: ${copy}\n`;
|
||||
let r=s.run('stack-copy.sh',['check','--copies',s.copies,'--bin',s.bin,'--units','external']);assert.equal(r.status,0,r.stderr);assert.equal(r.stdout,want);
|
||||
assert.deepEqual(readFileSync(argv,'utf8').trimEnd().split('\n'),[
|
||||
'show [email protected] -p LoadState -p WorkingDirectory -p ExecStart',
|
||||
'show mosaic-webui.service -p LoadState -p WorkingDirectory -p ExecStart']);
|
||||
rmSync(argv);
|
||||
for(const args of [[],['--units','user']]){
|
||||
r=s.run('stack-copy.sh',['check','--copies',s.copies,'--bin',s.bin,...args]);assert.equal(r.status,0,r.stderr);assert.equal(r.stdout,want);
|
||||
assert.deepEqual(readFileSync(argv,'utf8').trimEnd().split('\n'),[
|
||||
'--user show [email protected] -p LoadState -p WorkingDirectory -p ExecStart',
|
||||
'--user show mosaic-webui.service -p LoadState -p WorkingDirectory -p ExecStart']);
|
||||
rmSync(argv);
|
||||
}
|
||||
});
|
||||
|
||||
@@ -0,0 +1,476 @@
|
||||
// scripts/stack-release.sh --units external (row 232, #1734). Design: r6 section 4 and the section 7
|
||||
// test list, agents/darkwing/work/queue-195/DESIGN.md; deviations: agents/darkwing/work/queue-232/BUILD.md.
|
||||
// The stand-ins and the module fixture are stack-release.test.mjs's (stack-release-fixture.mjs).
|
||||
// Root's part (stop, unit install, start) runs here as the stand-in systemctl and bus-service.sh and
|
||||
// webui-service.sh installs; no sudo and no real unit.
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {execFileSync} from 'node:child_process';
|
||||
import {mkdirSync,writeFileSync,readFileSync,existsSync,readdirSync,rmSync,chmodSync,cpSync} from 'node:fs';
|
||||
import {DatabaseSync} from 'node:sqlite';
|
||||
import {join} from 'node:path';
|
||||
import {B,C,C1,COPIES,ROOT,SBIN,base,fault,lines,logLines,mk,pointer,repo,run,seed,sha,storeVersion,sysctl,tree,units} from './stack-release-fixture.mjs';
|
||||
|
||||
// The commit row 232 builds on: the user-mode tests compare the candidate's calls and --plan bytes
|
||||
// with its scripts. A later row that changes user mode on purpose moves this.
|
||||
const BASE='cb5fdb064fb4f12edd045374dd463dfdc25c2b42';
|
||||
|
||||
const X=(e)=>({...e,flags:[...e.flags,'--units','external']});
|
||||
const sh=(e,name,...a)=>execFileSync('bash',[join(repo,'scripts',name),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
|
||||
// Root's steps: R1/A2 stop, A4 install and start. offset is the start's SD_CLOCK_OFFSET.
|
||||
const down=(e)=>{sysctl(e,'stop',e.con);sysctl(e,'stop',e.bus)};
|
||||
const up=(e,copy,{pm=false,offset=0}={})=>{
|
||||
sh(e,'bus-service.sh','install','--copy',copy,'--copies',COPIES,...(pm?['--pm']:[]),'--claude',e.claude);
|
||||
sh(e,'webui-service.sh','install','--copy',copy,'--copies',COPIES,'--port',e.port,'--business','acme');
|
||||
execFileSync(join(SBIN,'systemctl'),['--user','start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:String(offset)}});
|
||||
sysctl(e,'start',e.con);
|
||||
};
|
||||
// The stack on B for acme, adopted under --units external, then stopped by root unless live.
|
||||
async function xseed(t,{pm=false,live=false}={}){
|
||||
const e=await mk(t);await seed(e,{pm,pointer:'none'});const x=X(e);
|
||||
const r=await run(x,'adopt','acme');assert.equal(r.code,0,r.err);
|
||||
if(!live)down(e);
|
||||
return {e,x};
|
||||
}
|
||||
const lastAct=(e)=>logLines(e).filter((o)=>o.result==='started').at(-1).activation;
|
||||
const rs=(e,a)=>join(e.state,'stack-rollback',a);
|
||||
const open=(e)=>{const m=new Set();for(const o of logLines(e)){if(!o.activation)continue;if(o.result==='started')m.add(o.activation);else m.delete(o.activation)}return [...m]};
|
||||
const mine=(e,a)=>logLines(e).filter((o)=>o.activation===a);
|
||||
const journalCalls=(e)=>lines(join(e.sd,'journal-since.log'));
|
||||
// A3: stop, stage TO. Returns the activation and the run.
|
||||
async function stage(e,x,to,...args){
|
||||
const r=await run(x,'activate',to,'acme',...args);
|
||||
return {r,a:r.code===0||r.code===1?lastAct(e):null};
|
||||
}
|
||||
// The started line's from, to and op on the activation's result line, and its backup the set's.
|
||||
function sameAsStarted(e,a){
|
||||
const [s,f,...more]=mine(e,a);assert.equal(more.length,0,JSON.stringify(mine(e,a)));
|
||||
assert.equal(s.result,'started');
|
||||
for(const k of ['op','from','to','activation'])assert.equal(f[k],s[k],k);
|
||||
assert.ok(['activate','rollback'].includes(f.op),f.op);
|
||||
assert.equal(f.units,'external');
|
||||
const b=join(rs(e,a),'BACKUP');assert.equal(f.backup,existsSync(b)?readFileSync(b,'utf8').trim():null);
|
||||
return f;
|
||||
}
|
||||
// B1's end state: one started and one result line, nothing open, no marker or FAILED, and a
|
||||
// following --plan that doesn't refuse interrupted-run.
|
||||
async function closed(e,x,a){
|
||||
sameAsStarted(e,a);
|
||||
assert.deepEqual(open(e),[]);
|
||||
for(const f of ['AWAITING-UNITS','FAILED'])assert.ok(!existsSync(join(rs(e,a),f)),f);
|
||||
const s=await run(x,'status','acme');assert.equal(s.code,0,s.err);
|
||||
assert.ok(!/interrupted run|awaiting units/.test(s.out),s.out);
|
||||
const p=await run(x,'activate',pointer(e).copy===C1?B:C1,'acme','--plan');
|
||||
assert.ok(!(p.out+p.err).includes('interrupted-run'),p.out+p.err);
|
||||
}
|
||||
const storeSha=(e)=>sha(e.store);
|
||||
const backupStore=(e,a)=>join(e.data,'backups','stack',a,'bus','bus.sqlite');
|
||||
|
||||
test('row 232: confirm, revert, gate, --to and --expect-from need --units external; the user-only flags are usage under it',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const logs=logLines(e).length;
|
||||
for(const a of [['confirm',C1,'acme'],['revert',C1,'acme'],['gate',B,'acme','--since','1'],['rollback','acme','--to',B],['activate',C1,'acme','--plan','--expect-from',B]]){
|
||||
const r=await run(e,...a);assert.equal(r.code,4,a.join(' '));assert.match(r.err,/^usage: stack-release.sh adopt BUSINESS\n/);
|
||||
}
|
||||
for(const a of [['activate',C1,'acme','--pm'],['activate',C1,'acme','--tester-port','7444'],['activate',C1,'acme','--no-proof'],['activate',C1,'acme','--no-send'],['activate',C1,'acme','--sso'],['activate',C1,'acme','--proof-header','X','--proof-file','/p'],['rollback','acme'],['gate',B,'acme'],['activate',C1,'acme','--expect-from',B],['confirm',C1,'acme','--rollback',B],['confirm',C1,'acme','--cause','A5']]){
|
||||
const r=await run(x,...a);assert.equal(r.code,4,a.join(' '));
|
||||
}
|
||||
assert.equal(logLines(e).length,logs);
|
||||
});
|
||||
|
||||
test('row 232: pm-on and activate --pm-after-harness refuse external-pm',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const p=await run(x,'pm-on','acme');assert.equal(p.code,3);
|
||||
assert.match(p.err,/^stack-release: refusing external-pm: under --units external the PM is a flag in root's bus unit/);
|
||||
const a=await run(x,'activate',C1,'acme','--pm-after-harness');assert.equal(a.code,3);
|
||||
assert.match(a.err,/^stack-release: refusing external-pm: /);
|
||||
const l=logLines(e).slice(-2);assert.deepEqual(l.map((o)=>[o.result,o.code,o.activation]),[['refused','external-pm',null],['refused','external-pm',null]]);
|
||||
assert.equal(readdirSync(e.state).includes('stack-rollback'),false);
|
||||
});
|
||||
|
||||
test('row 232: units-running (N4) refuses while a unit is active and leaves the store, the log and TAKEN; failed counts as down',async(t)=>{
|
||||
const {e,x}=await xseed(t,{live:true});
|
||||
sysctl(e,'stop',e.con);
|
||||
const before=storeSha(e),logs=logLines(e).length;
|
||||
const r=await run(x,'activate',C,'acme','--migrate');assert.equal(r.code,3,r.out+r.err);
|
||||
assert.equal(r.err,`stack-release: refusing units-running: ${e.bus} is active; root stops the units first\n`);
|
||||
assert.equal(storeSha(e),before);assert.equal(storeVersion(e.store,true),'3g');
|
||||
const l=logLines(e).slice(logs);assert.deepEqual(l.map((o)=>[o.result,o.code,o.activation,o.units]),[['refused','units-running',null,'external']]);
|
||||
assert.ok(!existsSync(join(e.data,'backups')));assert.ok(!existsSync(join(e.state,'stack-rollback')));
|
||||
// The Console active and the bus stopped refuses too.
|
||||
sysctl(e,'stop',e.bus);sysctl(e,'start',e.con);
|
||||
const c=await run(x,'activate',C1,'acme');assert.equal(c.code,3);assert.ok(c.err.startsWith(`stack-release: refusing units-running: ${e.con} is active`),c.err);
|
||||
// A bus unit root's stop left failed is down.
|
||||
sysctl(e,'stop',e.con);
|
||||
const s=units(e);s[e.bus].state='failed';writeFileSync(join(e.sd,'units.json'),JSON.stringify(s));
|
||||
const f=await run(x,'activate',C1,'acme');assert.equal(f.code,0,f.out+f.err);
|
||||
});
|
||||
|
||||
test('row 232: activate stages: the pointer stays, AWAITING-UNITS names TO, a backup of the stopped store, no directory in the set, no --user, no journal',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const p0=readFileSync(e.pointer,'utf8'),w0=sha(join(e.bin,'mosaic')),argv0=lines(join(e.sd,'argv.log')).length;
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
assert.equal(r.out.split('\n').at(-2),`stack-release: activate staged ${C1} for acme as ${a}; root installs and starts the units, then runs confirm ${C1} acme --units external`);
|
||||
assert.equal(readFileSync(e.pointer,'utf8'),p0);
|
||||
assert.equal(readFileSync(join(rs(e,a),'AWAITING-UNITS'),'utf8'),`${C1}\n`);
|
||||
assert.deepEqual(readdirSync(rs(e,a)).sort(),['AWAITING-UNITS','BACKUP','SHA256SUMS','UNITS.sha256','WRAPPER_MODE','pointer',`show-${e.bus}.txt`,`show-${e.con}.txt`,'steps.log','wrapper'].sort());
|
||||
assert.equal(readFileSync(join(rs(e,a),'BACKUP'),'utf8'),`${a}\n`);
|
||||
assert.equal(sha(backupStore(e,a)),storeSha(e));assert.equal(readFileSync(join(e.data,'backups','stack',a,'SCHEMA'),'utf8'),'3g\n');
|
||||
assert.equal(readFileSync(join(rs(e,a),'UNITS.sha256'),'utf8'),`${sha(join(e.units,e.tmpl))} ${join(e.units,e.tmpl)}\n${sha(join(e.units,e.con))} ${join(e.units,e.con)}\n`);
|
||||
assert.equal(sha(join(rs(e,a),'wrapper')),w0);assert.notEqual(sha(join(e.bin,'mosaic')),w0);
|
||||
assert.deepEqual(mine(e,a).map((o)=>o.result),['started']);
|
||||
assert.deepEqual([units(e)[e.bus].state,units(e)[e.con].state],['inactive','inactive']);
|
||||
const ext=lines(join(e.sd,'argv.log')).slice(argv0);
|
||||
assert.ok(ext.length>0);for(const c of ext)assert.match(c,/^(show|is-active|list-units) /,c);
|
||||
assert.deepEqual(journalCalls(e),[]);
|
||||
const s=await run(x,'status','acme');assert.equal(s.code,0,s.err);
|
||||
assert.ok(s.out.includes(`awaiting units: ${a} (copy ${C1})\n`),s.out);assert.ok(s.out.includes(`interrupted run: ${a} `),s.out);
|
||||
});
|
||||
|
||||
test('row 232: A3 order: capture-failed and marker-failed refuse with exit 3 and leave no marker or FAILED',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
fault(e,'no-fragment');
|
||||
const r=await run(x,'activate',C1,'acme');assert.equal(r.code,3,r.out+r.err);
|
||||
assert.match(r.err,/^stack-release: refusing capture-failed: could not write the rollback set .*; nothing was changed\n$/);
|
||||
const a=lastAct(e);
|
||||
assert.deepEqual(mine(e,a).map((o)=>[o.result,o.code]),[['started',undefined],['refused','capture-failed']]);
|
||||
for(const f of ['AWAITING-UNITS','FAILED'])assert.ok(!existsSync(join(rs(e,a),f)),f);
|
||||
assert.deepEqual(open(e),[]);
|
||||
rmSync(join(e.sd,'faults','no-fragment'));
|
||||
// The set can't be made: marker-failed, for that activation.
|
||||
chmodSync(join(e.state,'stack-rollback'),0o555);
|
||||
const m=await run(x,'activate',C1,'acme');assert.equal(m.code,3,m.out+m.err);
|
||||
assert.match(m.err,/^stack-release: refusing marker-failed: could not make the rollback set .*; nothing was changed\n$/);
|
||||
const b=lastAct(e);assert.notEqual(b,a);
|
||||
assert.deepEqual(mine(e,b).map((o)=>[o.result,o.code]),[['started',undefined],['refused','marker-failed']]);
|
||||
chmodSync(join(e.state,'stack-rollback'),0o755);
|
||||
assert.deepEqual(open(e),[]);
|
||||
});
|
||||
|
||||
test('row 232: confirm and revert refuse open-activation with no marker, two, the wrong copy, or one the log holds closed; refusals name no activation',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const tryBoth=async(id,why)=>{
|
||||
for(const v of ['confirm','revert']){
|
||||
const o=open(e),logs=logLines(e).length;
|
||||
const r=await run(x,v,id,'acme');assert.equal(r.code,3,`${v} ${why}: ${r.out}${r.err}`);
|
||||
assert.match(r.err,/^stack-release: refusing open-activation: /,why);
|
||||
assert.deepEqual(logLines(e).slice(logs).map((l)=>[l.result,l.code,l.activation,l.op]),[['refused','open-activation',null,v]],why);
|
||||
assert.deepEqual(open(e),o,why);
|
||||
}
|
||||
};
|
||||
await tryBoth(C1,'no marker');
|
||||
// One marker whose activation the log never started. It sorts before any real one, so a confirm
|
||||
// that took the newest of two markers would take the real one.
|
||||
const fake=join(e.state,'stack-rollback','20000101T000000Z-'+C1.slice(0,12));
|
||||
mkdirSync(fake,{recursive:true});writeFileSync(join(fake,'AWAITING-UNITS'),`${C1}\n`);
|
||||
await tryBoth(C1,'marker not open');
|
||||
rmSync(fake,{recursive:true});
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
await tryBoth(B,'wrong copy');
|
||||
mkdirSync(fake);writeFileSync(join(fake,'AWAITING-UNITS'),`${C1}\n`);
|
||||
await tryBoth(C1,'two markers');
|
||||
rmSync(fake,{recursive:true});
|
||||
// gate refuses nothing and writes nothing, open or not.
|
||||
const logs=readFileSync(e.log,'utf8');
|
||||
const g=await run(x,'gate',C1,'acme','--since','1','--timeout','1');assert.equal(g.code,1);
|
||||
assert.equal(readFileSync(e.log,'utf8'),logs);assert.deepEqual(open(e),[a]);
|
||||
});
|
||||
|
||||
test('row 232: confirm polls: bus status good after 3 s fails with --timeout 1, writes FAILED and leaves the rest; FAILED then refuses activation-failed',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
up(e,C1);
|
||||
const p0=readFileSync(e.pointer,'utf8'),l0=readFileSync(e.log,'utf8');
|
||||
fault(e,`status-until-${C1}`,String(Date.now()+3000));
|
||||
const c=await run(x,'confirm',C1,'acme','--timeout','1');assert.equal(c.code,1,c.out+c.err);
|
||||
assert.match(c.err,new RegExp(`^stack-release: activate failed with bus-status; ${a} stays open`));
|
||||
assert.equal(readFileSync(join(rs(e,a),'FAILED'),'utf8'),'bus-status\n');
|
||||
assert.equal(readFileSync(e.pointer,'utf8'),p0);assert.equal(readFileSync(e.log,'utf8'),l0);
|
||||
assert.ok(existsSync(join(rs(e,a),'AWAITING-UNITS')));
|
||||
const f=await run(x,'confirm',C1,'acme');assert.equal(f.code,3);
|
||||
assert.match(f.err,new RegExp(`^stack-release: refusing activation-failed: ${a} has a FAILED file`));
|
||||
assert.equal(logLines(e).at(-1).activation,null);assert.deepEqual(open(e),[a]);
|
||||
});
|
||||
|
||||
test('row 232: confirm polls to a pass with --timeout 10: the pointer, an ok line that reads as the started run, no marker',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
up(e,C1);
|
||||
fault(e,`status-until-${C1}`,String(Date.now()+3000));
|
||||
const c=await run(x,'confirm',C1,'acme','--timeout','10');assert.equal(c.code,0,c.out+c.err);
|
||||
assert.equal(c.out.split('\n').at(-2),`stack-release: activate ok: ${C1} is live for acme`);
|
||||
const p=pointer(e);assert.deepEqual([p.copy,p.activation,p.schema,p.pm,p.console],[C1,a,'3g',false,['--port',e.port,'--business','acme']]);
|
||||
const f=sameAsStarted(e,a);assert.deepEqual([f.result,f.op,f.from,f.to,f.backup],['ok','activate',B,C1,a]);
|
||||
assert.ok(!existsSync(join(rs(e,a),'AWAITING-UNITS')));
|
||||
assert.deepEqual(journalCalls(e),[]);
|
||||
});
|
||||
|
||||
// N4 with the brief's numbers: the started line at 1792540800.900 floors to 1792540800. The host's
|
||||
// clock runs so boot status is after that second.
|
||||
const T0=1792540800,offsetTo=(s)=>Math.floor(Date.now()/1000)-s;
|
||||
function startedAt(e,a,ms){
|
||||
const l=readFileSync(e.log,'utf8').split('\n').map((x)=>{if(!x)return x;const o=JSON.parse(x);if(o.activation===a&&o.result==='started')o.at=new Date(ms).toISOString();return JSON.stringify(o)});
|
||||
writeFileSync(e.log,l.join('\n'));
|
||||
}
|
||||
|
||||
test('row 232: stale-start: @1792540799 fails a start floor of 1792540800 with stale-start',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
startedAt(e,a,T0*1000+900);up(e,C1,{offset:offsetTo(T0+100)});
|
||||
fault(e,'enter',JSON.stringify({value:T0-1}));
|
||||
const c=await run(x,'confirm',C1,'acme','--timeout','1');assert.equal(c.code,1,c.out+c.err);
|
||||
assert.ok(c.out.includes(`gate failed on ${C1}: stale-start (the bus started at @${T0-1}, before @${T0})`),c.out);
|
||||
assert.equal(readFileSync(join(rs(e,a),'FAILED'),'utf8'),'stale-start\n');
|
||||
});
|
||||
|
||||
test('row 232: stale-start is a gate code: an enter time that turns to @1792540800 after 2 s passes with --timeout 10',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
startedAt(e,a,T0*1000+900);up(e,C1,{offset:offsetTo(T0+100)});
|
||||
fault(e,'enter',JSON.stringify({value:T0-1,at:Date.now()+2000,then:T0}));
|
||||
const c=await run(x,'confirm',C1,'acme','--timeout','10');assert.equal(c.code,0,c.out+c.err);
|
||||
assert.equal(pointer(e).copy,C1);
|
||||
});
|
||||
|
||||
test('row 232: gate (N2, A2f): --since 1792540800 passes @1792540800 and fails @1792540799; it writes nothing',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
up(e,B,{offset:offsetTo(T0+100)});
|
||||
const snap=()=>[readFileSync(e.log,'utf8'),readFileSync(e.pointer,'utf8'),tree(e.state).join('\n')];
|
||||
const s0=snap();
|
||||
fault(e,'enter',JSON.stringify({value:T0}));
|
||||
const g=await run(x,'gate',B,'acme','--since',String(T0));assert.equal(g.code,0,g.out+g.err);
|
||||
assert.equal(g.out,`stack-release: gate passed on ${B}\n`);
|
||||
fault(e,'enter',JSON.stringify({value:T0-1}));
|
||||
const f=await run(x,'gate',B,'acme','--since',String(T0),'--timeout','1');assert.equal(f.code,1,f.out+f.err);
|
||||
assert.ok(f.out.includes(`gate failed on ${B}: stale-start (the bus started at @${T0-1}, before @${T0})`),f.out);
|
||||
assert.deepEqual(snap(),s0);
|
||||
const u=await run(x,'gate',B,'acme');assert.equal(u.code,4);
|
||||
assert.deepEqual(journalCalls(e),[]);
|
||||
});
|
||||
|
||||
test('row 232: the external PM check reads launches list: one live run of launch.by since the start passes',async(t)=>{
|
||||
const {e,x}=await xseed(t,{pm:true});
|
||||
const cases=[['',0,''],['coder',0,''],['none',1,'no live launch of chief'],['twolive',1,'2 live launches of chief'],['early',1,'started before this start'],['dead',1,'no live launch of chief'],['other',1,'the launch of chief is for other']];
|
||||
for(const [mode,code,why] of cases){
|
||||
rmSync(join(e.sd,'launches.json'),{force:true});
|
||||
if(mode)fault(e,`pm-${B}`,mode);else rmSync(join(e.sd,'faults',`pm-${B}`),{force:true});
|
||||
const since=Math.floor(Date.now()/1000);
|
||||
up(e,B,{pm:true});
|
||||
const g=await run(x,'gate',B,'acme','--since',String(since),'--timeout','1');assert.equal(g.code,code,`${mode}: ${g.out}${g.err}`);
|
||||
if(why)assert.ok(g.out.includes(`gate failed on ${B}: pm-launch-unconfirmed (`)&&g.out.includes(why),`${mode}: ${g.out}`);
|
||||
down(e);
|
||||
}
|
||||
assert.deepEqual(journalCalls(e),[]);
|
||||
});
|
||||
|
||||
test('row 232: revert puts the wrapper back and leaves the store when nothing migrated',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const w0=sha(join(e.bin,'mosaic'));
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
// A write to the stopped store after A3: revert has no MIGRATING, so it is not its to undo.
|
||||
const db=new DatabaseSync(e.store);db.prepare('INSERT INTO events (at) VALUES (?)').run('2001-01-01T00:00:00.000Z');db.close();
|
||||
const s1=storeSha(e);assert.notEqual(s1,sha(backupStore(e,a)));
|
||||
const v=await run(x,'revert',C1,'acme');assert.equal(v.code,0,v.out+v.err);
|
||||
assert.equal(v.out.split('\n').at(-2),`stack-release: revert ok: ${a}'s wrapper is back; root installs the old units next`);
|
||||
assert.equal(sha(join(e.bin,'mosaic')),w0);assert.equal(storeSha(e),s1);
|
||||
assert.deepEqual(mine(e,a).map((o)=>o.result),['started']);assert.ok(existsSync(join(rs(e,a),'AWAITING-UNITS')));
|
||||
});
|
||||
|
||||
test('row 232: confirm --rollback refuses rollback-mismatch, and logs rollback-failed with exit 2 when OLD fails its gate',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
const v=await run(x,'revert',C1,'acme');assert.equal(v.code,0,v.out+v.err);
|
||||
const m=await run(x,'confirm',C1,'acme','--rollback',C,'--cause','A5');assert.equal(m.code,3);
|
||||
assert.equal(m.err,`stack-release: refusing rollback-mismatch: ${C} is not the copy ${a} started from, ${B}\n`);
|
||||
assert.equal(logLines(e).at(-1).activation,null);assert.deepEqual(open(e),[a]);
|
||||
fault(e,`healthz-${B}`);up(e,B);
|
||||
const f=await run(x,'confirm',C1,'acme','--rollback',B,'--cause','A5','--timeout','1');assert.equal(f.code,2,f.out+f.err);
|
||||
assert.equal(f.err,`stack-release: rollback-failed: the run failed with unrecorded and the rollback with healthz; root stops the units. Rollback set: ${rs(e,a)}, backup ${join(e.data,'backups','stack',a)}\n`);
|
||||
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback,l.rollbackCode],['rollback-failed','unrecorded','failed','healthz']);
|
||||
for(const n of ['AWAITING-UNITS','FAILED'])assert.ok(!existsSync(join(rs(e,a),n)),n);
|
||||
assert.deepEqual(open(e),[]);
|
||||
});
|
||||
|
||||
test('row 232: confirm --rollback after a --restore swap logs rollbackCode not-attempted; rollback --restore finds an external backup',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const {r,a}=await stage(e,x,C,'--migrate');assert.equal(r.code,0,r.out+r.err);
|
||||
up(e,C);assert.equal((await run(x,'confirm',C,'acme')).code,0);
|
||||
down(e);
|
||||
const b=await run(x,'rollback','acme','--to',B,'--restore',a);assert.equal(b.code,0,b.out+b.err);
|
||||
const ra=lastAct(e);
|
||||
assert.ok(b.out.includes(`restore: ${join(e.data,'backups','stack',a)}; `),b.out);
|
||||
assert.ok(existsSync(join(rs(e,ra),'RESTORED')));assert.equal(readFileSync(join(rs(e,ra),'BACKUP'),'utf8'),`${a}\n`);
|
||||
assert.equal(storeSha(e),sha(backupStore(e,a)));
|
||||
const f=await run(x,'confirm',B,'acme','--rollback',C,'--cause','A4');assert.equal(f.code,2,f.out+f.err);
|
||||
const l=sameAsStarted(e,ra);assert.deepEqual([l.op,l.result,l.code,l.rollbackCode,l.backup],['rollback','rollback-failed','units-start','not-attempted',a]);
|
||||
});
|
||||
|
||||
// --- B1: real sequences on one log ---
|
||||
|
||||
test('row 232: sequence 1: activate, confirm times out, revert, confirm --rollback --cause A5: gate-failed rollback ok',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const w0=sha(join(e.bin,'mosaic')),p0=readFileSync(e.pointer,'utf8');
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
fault(e,`healthz-${C1}`);up(e,C1);
|
||||
const c=await run(x,'confirm',C1,'acme','--timeout','1');assert.equal(c.code,1,c.out+c.err);
|
||||
assert.deepEqual(mine(e,a).map((o)=>o.result),['started']);
|
||||
down(e);
|
||||
assert.equal((await run(x,'revert',C1,'acme')).code,0);assert.equal(sha(join(e.bin,'mosaic')),w0);
|
||||
up(e,B);
|
||||
const k=await run(x,'confirm',C1,'acme','--rollback',B,'--cause','A5');assert.equal(k.code,1,k.out+k.err);
|
||||
assert.equal(k.err,`stack-release: healthz; rolled back to ${B}, which passed the gate\n`);
|
||||
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['gate-failed','healthz','ok']);
|
||||
assert.equal(readFileSync(e.pointer,'utf8'),p0);
|
||||
await closed(e,x,a);
|
||||
});
|
||||
|
||||
test('row 232: sequence 2: activate --migrate, the wrapper install fails, revert restores TAKEN, confirm --rollback --cause A3',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const s0=storeSha(e);
|
||||
chmodSync(e.bin,0o555);
|
||||
const {r,a}=await stage(e,x,C,'--migrate');chmodSync(e.bin,0o755);
|
||||
assert.equal(r.code,1,r.out+r.err);assert.match(r.err,new RegExp(`^stack-release: activate failed with wrapper; ${a} stays open`));
|
||||
assert.deepEqual(mine(e,a).map((o)=>o.result),['started']);
|
||||
for(const n of ['AWAITING-UNITS','MIGRATING'])assert.ok(existsSync(join(rs(e,a),n)),n);
|
||||
assert.equal(readFileSync(join(rs(e,a),'FAILED'),'utf8'),'wrapper\n');
|
||||
assert.equal(storeVersion(e.store,false),'3h');assert.equal(sha(backupStore(e,a)),s0);
|
||||
const v=await run(x,'revert',C,'acme');assert.equal(v.code,0,v.out+v.err);
|
||||
assert.equal(v.out.split('\n').at(-2),`stack-release: revert ok: ${a}'s wrapper and store are back; root installs the old units next`);
|
||||
assert.equal(storeSha(e),s0);
|
||||
// revert runs again: the store already matches, and nothing moves.
|
||||
const posts=readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-'));
|
||||
assert.equal((await run(x,'revert',C,'acme')).code,0);
|
||||
assert.deepEqual(readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-')),posts);
|
||||
up(e,B);
|
||||
const k=await run(x,'confirm',C,'acme','--rollback',B,'--cause','A3');assert.equal(k.code,1,k.out+k.err);
|
||||
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['failed','wrapper','ok']);
|
||||
await closed(e,x,a);
|
||||
});
|
||||
|
||||
test('row 232: sequence 3: activate, root\'s A4 fails, revert, confirm --rollback --cause A4: failed code units-start',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
assert.equal((await run(x,'revert',C1,'acme')).code,0);
|
||||
up(e,B);
|
||||
const k=await run(x,'confirm',C1,'acme','--rollback',B,'--cause','A4');assert.equal(k.code,1,k.out+k.err);
|
||||
assert.equal(k.err,`stack-release: units-start; rolled back to ${B}, which passed the gate\n`);
|
||||
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['failed','units-start','ok']);
|
||||
await closed(e,x,a);
|
||||
});
|
||||
|
||||
test('row 232: sequence 4: a capture failure leaves nothing open, then A2f\'s gate OLD writes nothing',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
fault(e,'no-fragment');
|
||||
const r=await run(x,'activate',C1,'acme');assert.equal(r.code,3,r.out+r.err);rmSync(join(e.sd,'faults','no-fragment'));
|
||||
const a=lastAct(e);assert.deepEqual(mine(e,a).map((o)=>[o.result,o.code]),[['started',undefined],['refused','capture-failed']]);
|
||||
assert.deepEqual(open(e),[]);
|
||||
const since=Math.floor(Date.now()/1000);up(e,B);
|
||||
const s0=[readFileSync(e.log,'utf8'),readFileSync(e.pointer,'utf8'),tree(e.state).join('\n')];
|
||||
const g=await run(x,'gate',B,'acme','--since',String(since));assert.equal(g.code,0,g.out+g.err);
|
||||
assert.deepEqual([readFileSync(e.log,'utf8'),readFileSync(e.pointer,'utf8'),tree(e.state).join('\n')],s0);
|
||||
});
|
||||
|
||||
test('row 232: sequence 5: a confirm of the wrong copy is refused, and revert and confirm --rollback still close the activation',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
up(e,C1);
|
||||
const w=await run(x,'confirm',B,'acme');assert.equal(w.code,3);assert.match(w.err,/^stack-release: refusing open-activation: /);
|
||||
assert.equal(logLines(e).at(-1).activation,null);assert.deepEqual(open(e),[a]);
|
||||
down(e);
|
||||
assert.equal((await run(x,'revert',C1,'acme')).code,0);
|
||||
up(e,B);
|
||||
const k=await run(x,'confirm',C1,'acme','--rollback',B,'--cause','A5');assert.equal(k.code,1,k.out+k.err);
|
||||
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['gate-failed','unrecorded','ok']);
|
||||
await closed(e,x,a);
|
||||
});
|
||||
|
||||
test('row 232: sequence 6: a revert that fails leaves the activation open, and the next --plan refuses interrupted-run',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const {r,a}=await stage(e,x,C1);assert.equal(r.code,0,r.out+r.err);
|
||||
rmSync(join(rs(e,a),'wrapper'));
|
||||
const v=await run(x,'revert',C1,'acme');assert.equal(v.code,1,v.out+v.err);
|
||||
assert.equal(v.err,`stack-release: revert failed: the rollback set has no saved wrapper; ${a} stays open. Rollback set: ${rs(e,a)}\n`);
|
||||
assert.deepEqual(open(e),[a]);assert.ok(existsSync(join(rs(e,a),'AWAITING-UNITS')));
|
||||
const p=await run(x,'activate',C,'acme','--plan','--migrate');assert.equal(p.code,3,p.out+p.err);
|
||||
assert.ok(p.out.includes(`refuses interrupted-run: activation ${a} started and has no final line`),p.out);
|
||||
});
|
||||
|
||||
test('row 232: sequence 7: adopt, two confirmed activates, rollback --to the previous copy, confirmed, logs op rollback',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const one=await stage(e,x,C1);assert.equal(one.r.code,0,one.r.out+one.r.err);
|
||||
up(e,C1);assert.equal((await run(x,'confirm',C1,'acme')).code,0);down(e);
|
||||
const two=await stage(e,x,C,'--migrate');assert.equal(two.r.code,0,two.r.out+two.r.err);
|
||||
up(e,C);assert.equal((await run(x,'confirm',C,'acme')).code,0);down(e);
|
||||
const l1=sameAsStarted(e,one.a),l2=sameAsStarted(e,two.a);
|
||||
assert.deepEqual([l1.op,l1.result,l1.from,l1.to,l1.backup],['activate','ok',B,C1,one.a]);
|
||||
assert.deepEqual([l2.op,l2.result,l2.from,l2.to,l2.backup],['activate','ok',C1,C,two.a]);
|
||||
// N5: --expect-from names the copy root read from its units.
|
||||
const d1=await run(x,'activate',C1,'acme','--plan','--expect-from',C1);assert.equal(d1.code,3,d1.out+d1.err);
|
||||
assert.ok(d1.out.includes(`refuses pointer-differs: the pointer names ${C} and root's units name ${C1}; Jason or Mos settles it`),d1.out);
|
||||
const d2=await run(x,'rollback','acme','--to',C1,'--plan','--expect-from',B);assert.equal(d2.code,3,d2.out+d2.err);
|
||||
assert.ok(d2.out.includes(`refuses pointer-differs: the pointer names ${C} and root's units name ${B}`),d2.out);
|
||||
const ok=await run(x,'rollback','acme','--to',C1,'--restore',two.a,'--plan','--expect-from',C);assert.equal(ok.code,0,ok.out+ok.err);
|
||||
const logs=logLines(e).length;
|
||||
const no=await run(x,'rollback','acme','--to',B,'--plan');assert.equal(no.code,3);
|
||||
assert.equal(no.err,`stack-release: refusing to-not-previous: --to ${B} is not the previous copy; the log names ${C1}\n`);
|
||||
assert.equal(logLines(e).length,logs);
|
||||
const nr=await run(x,'rollback','acme','--to',B,'--restore',one.a);assert.equal(nr.code,3);assert.match(nr.err,/^stack-release: refusing to-not-previous: /);
|
||||
assert.deepEqual(logLines(e).slice(logs).map((o)=>[o.result,o.code,o.activation]),[['refused','to-not-previous',null]]);
|
||||
const rb=await run(x,'rollback','acme','--to',C1,'--restore',two.a);assert.equal(rb.code,0,rb.out+rb.err);
|
||||
const ra=lastAct(e);
|
||||
up(e,C1);
|
||||
const k=await run(x,'confirm',C1,'acme');assert.equal(k.code,0,k.out+k.err);
|
||||
assert.equal(k.out.split('\n').at(-2),`stack-release: rollback ok: ${C1} is live for acme`);
|
||||
const l3=sameAsStarted(e,ra);assert.deepEqual([l3.op,l3.result,l3.from,l3.to,l3.backup],['rollback','ok',C,C1,two.a]);
|
||||
assert.deepEqual([pointer(e).copy,pointer(e).schema],[C1,'3g']);
|
||||
const prev=await run(x,'rollback','acme','--to',B,'--plan');assert.match(prev.err,new RegExp(`the log names ${C}\\n$`));
|
||||
await closed(e,x,ra);
|
||||
});
|
||||
|
||||
test('row 232: sequence 8: a migration that writes and fails: revert restores the backup byte for byte, confirm --rollback --cause A3',async(t)=>{
|
||||
const {e,x}=await xseed(t);
|
||||
const s0=storeSha(e);
|
||||
fault(e,'migrate-fails');
|
||||
const {r,a}=await stage(e,x,C,'--migrate');assert.equal(r.code,1,r.out+r.err);rmSync(join(e.sd,'faults','migrate-fails'));
|
||||
assert.equal(readFileSync(join(rs(e,a),'FAILED'),'utf8'),'migrate-failed\n');assert.ok(existsSync(join(rs(e,a),'MIGRATING')));
|
||||
assert.equal(storeVersion(e.store,false),'3h-partial');
|
||||
assert.equal((await run(x,'revert',C,'acme')).code,0);
|
||||
assert.equal(storeSha(e),s0);assert.equal(storeSha(e),sha(backupStore(e,a)));
|
||||
assert.deepEqual(readdirSync(join(e.data,'bus')),readdirSync(join(e.data,'backups','stack',a,'bus')));
|
||||
up(e,B);
|
||||
const k=await run(x,'confirm',C,'acme','--rollback',B,'--cause','A3');assert.equal(k.code,1,k.out+k.err);
|
||||
const l=sameAsStarted(e,a);assert.deepEqual([l.result,l.code,l.rollback],['failed','migrate-failed','ok']);
|
||||
await closed(e,x,a);
|
||||
});
|
||||
|
||||
// --- User mode against the base ---
|
||||
|
||||
// The fixture repo with the base's two scripts: everything else is the same tree and history.
|
||||
const baseRepo=join(base,'base-repo');
|
||||
cpSync(repo,baseRepo,{recursive:true});
|
||||
for(const f of ['scripts/stack-release.sh','scripts/stack-copy.sh'])writeFileSync(join(baseRepo,f),execFileSync('git',['-C',ROOT,'show',`${BASE}:${f}`]),{mode:0o755});
|
||||
const BASE_SR=join(baseRepo,'scripts','stack-release.sh');
|
||||
|
||||
test('row 232: user mode: every systemctl call carries --user, and activate, rollback and check make the base\'s calls',async(t)=>{
|
||||
const calls=async(sr,dir)=>{
|
||||
const e=await mk(t);e.sr=sr;await seed(e);
|
||||
const n=lines(join(e.sd,'argv.log')).length;
|
||||
const a=await run(e,'activate',C1,'acme');assert.equal(a.code,0,a.out+a.err);
|
||||
const r=await run(e,'rollback','acme');assert.equal(r.code,0,r.out+r.err);
|
||||
execFileSync('bash',[join(dir,'scripts','stack-copy.sh'),'check','--copies',COPIES,'--bin',e.bin,'--business','acme'],{env:e.env,stdio:'ignore'});
|
||||
return lines(join(e.sd,'argv.log')).slice(n).map((l)=>l.split(e.root).join('<root>').split(e.port).join('<port>'));
|
||||
};
|
||||
const cand=await calls(undefined,repo),old=await calls(BASE_SR,baseRepo);
|
||||
assert.ok(cand.length>20);
|
||||
for(const c of cand)assert.match(c,/^--user /,c);
|
||||
assert.deepEqual(cand,old);
|
||||
});
|
||||
|
||||
test('row 232: user mode: a default activate --plan prints the base\'s bytes',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
for(const a of [['activate',C1,'acme','--plan'],['activate',C,'acme','--plan','--migrate'],['activate',C,'acme','--plan'],['rollback','acme','--plan']]){
|
||||
const cand=await run(e,...a),old=await run({...e,sr:BASE_SR},...a);
|
||||
assert.deepEqual(cand,old,a.join(' '));
|
||||
}
|
||||
assert.ok(!existsSync(join(e.state,'stack-rollback')));
|
||||
});
|
||||
@@ -0,0 +1,451 @@
|
||||
// The stack-release stand-ins and module fixture, shared by stack-release.test.mjs and
|
||||
// stack-release-external.test.mjs (row 232). Moved unchanged from stack-release.test.mjs, plus
|
||||
// what the external tests need; each test file runs in its own process, so each gets its own base.
|
||||
import test, {after} from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {execFileSync,spawn,spawnSync} from 'node:child_process';
|
||||
import {mkdtempSync,mkdirSync,writeFileSync,readFileSync,appendFileSync,cpSync,rmSync,chmodSync,existsSync,statSync,lstatSync,symlinkSync,linkSync,readdirSync,realpathSync} from 'node:fs';
|
||||
import {createServer} from 'node:http';
|
||||
import {createHash} from 'node:crypto';
|
||||
import {DatabaseSync} from 'node:sqlite';
|
||||
import {tmpdir} from 'node:os';
|
||||
import {join,dirname,basename} from 'node:path';
|
||||
import {fileURLToPath,pathToFileURL} from 'node:url';
|
||||
import {once} from 'node:events';
|
||||
|
||||
export const ROOT=fileURLToPath(new URL('../../..',import.meta.url)).replace(/\/$/,'');
|
||||
export const NODE=process.execPath;
|
||||
export const MARKER='# mosaic-cli wrapper (Q81, agents/sage/work/q81/mosaic-cli.sh)';
|
||||
export const BUS='[email protected]',CON='mosaic-webui.service';
|
||||
export const git=(cwd,...args)=>execFileSync('git',['-c','user.name=fixture','-c','user.email=fixture@invalid','-c','commit.gpgsign=false','-c','core.hooksPath=/dev/null','-C',cwd,...args],{encoding:'utf8'}).trim();
|
||||
export const script=(fn)=>`#!${NODE}\n(${fn})();\n`;
|
||||
|
||||
// --- Stand-ins. Each is serialized into its own file, so it uses only process and its own scope. ---
|
||||
|
||||
// systemctl --user: unit state in $SD_STATE/units.json, every call in calls.log. A bus start runs
|
||||
// holder.cjs (the store held open in WAL mode, writer.lock, host.json) and, with --pm in the unit's
|
||||
// ExecStart, writes the `launched` journal line and the launches entry unless a pm-<copy> fault
|
||||
// names another case. Faults are files in $SD_STATE/faults named for the copy they hit. Units of a
|
||||
// named instance (row 190) run on their own data root and host files beside the default's.
|
||||
export function systemctl(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process'),crypto=process.getBuiltinModule('node:crypto');
|
||||
const sd=process.env.SD_STATE,data=process.env.SD_DATA,unitDir=path.join(process.env.XDG_CONFIG_HOME,'systemd','user');
|
||||
const args=process.argv.slice(2).filter((a)=>a!=='--user'),[verb,...rest]=args;
|
||||
fs.appendFileSync(path.join(sd,'calls.log'),args.join(' ')+'\n');
|
||||
// Row 232: argv.log keeps each call as given, --user and all, for the user/external argv tests.
|
||||
fs.appendFileSync(path.join(sd,'argv.log'),process.argv.slice(2).join(' ')+'\n');
|
||||
const faultFile=(n)=>path.join(sd,'faults',n),fault=(n)=>fs.existsSync(faultFile(n));
|
||||
const stFile=path.join(sd,'units.json');
|
||||
const load=()=>{try{return JSON.parse(fs.readFileSync(stFile,'utf8'))}catch{return {}}};
|
||||
const save=(s)=>{fs.writeFileSync(stFile+'.tmp',JSON.stringify(s));fs.renameSync(stFile+'.tmp',stFile)};
|
||||
const sleep=(ms)=>Atomics.wait(new Int32Array(new SharedArrayBuffer(4)),0,0,ms);
|
||||
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
|
||||
// Row 190: a bus unit is the default's mosaic-bus@B or instance NAME's mosaic-bus-NAME@B; %i is B.
|
||||
const isBus=(u)=>/^mosaic-bus(-[a-z][a-z0-9]{1,15})?@[a-z0-9-]+\.service$/.test(u),inst=(u)=>u.slice(u.indexOf('@')+1,-'.service'.length);
|
||||
const unitFile=(u)=>{
|
||||
const main=isBus(u)?u.replace(/@[^@]*$/,'@.service'):u,file=path.join(unitDir,main);
|
||||
if(!fs.existsSync(file))return null;
|
||||
// The drop-in directories in systemd's order: service.d, the name cut after each hyphen, then
|
||||
// the template's and the unit's own.
|
||||
const stem=main.slice(0,main.search(/@?\.service$/)),dirs=['service.d'],drops=[];
|
||||
for(let i=0;i<stem.length;i++)if(stem[i]==='-')dirs.push(stem.slice(0,i+1)+'.service.d');
|
||||
for(const d of [...dirs,...(isBus(u)?[main+'.d',u+'.d']:[u+'.d'])]){const dir=path.join(unitDir,d);if(fs.existsSync(dir))for(const f of fs.readdirSync(dir).sort())if(f.endsWith('.conf'))drops.push(path.join(dir,f))}
|
||||
const r={file,wd:'',exec:'',env:[],drops};
|
||||
for(const f of [file,...drops])for(let line of fs.readFileSync(f,'utf8').split('\n')){
|
||||
line=line.replace(/%i/g,isBus(u)?inst(u):'').replace(/%h/g,process.env.HOME);
|
||||
const m=/^(WorkingDirectory|ExecStart|Environment)=(.*)$/.exec(line);if(!m)continue;
|
||||
if(m[1]==='WorkingDirectory')r.wd=m[2];else if(m[1]==='ExecStart'){if(m[2])r.exec=m[2]}else r.env.push(m[2]);
|
||||
}
|
||||
return r;
|
||||
};
|
||||
// A named instance's unit sets MOSAIC_CONFIG, and its bus works on that config's data root. A unit
|
||||
// without it is the default's, on $SD_DATA. Each data root has its own host and launches files.
|
||||
const dataOf=(f)=>{const c=f&&f.env.find((v)=>v.startsWith('MOSAIC_CONFIG='));return c?JSON.parse(fs.readFileSync(c.slice('MOSAIC_CONFIG='.length),'utf8')).dataRoot:data};
|
||||
const perData=(d,n)=>path.join(sd,d===data?`${n}.json`:`${n}-${path.basename(d)}.json`);
|
||||
const snap=(ev,d)=>{
|
||||
const dir=path.join(d,'bus'),files={};
|
||||
try{for(const n of fs.readdirSync(dir).sort()){const p=path.join(dir,n),st=fs.statSync(p);if(st.isFile())files[n]=[st.size,st.mtimeMs,crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')]}}catch{}
|
||||
fs.appendFileSync(path.join(sd,'snap.jsonl'),JSON.stringify({ev,files})+'\n');
|
||||
};
|
||||
const pm=(unit,d,id,pid,off)=>{
|
||||
let mode='';try{mode=fs.readFileSync(faultFile(`pm-${id}`),'utf8').trim()}catch{}
|
||||
const b=inst(unit),now=Date.now()-off*1000,epoch=Math.floor(now/1000),at=new Date(now).toISOString(),run=`run-${pid}`;
|
||||
const line=(text)=>fs.appendFileSync(path.join(sd,'journal.jsonl'),JSON.stringify({epoch,unit,text})+'\n');
|
||||
const ok=()=>line(`launched pm-${b}: run ${run}, claude opus`);
|
||||
const lf=perData(d,'launches');let list=[];try{list=JSON.parse(fs.readFileSync(lf,'utf8'))}catch{}
|
||||
// instance is launch.by's, chief in the example business, as the external PM check reads (row 232).
|
||||
const entry=(o)=>list.push({run,instance:'chief',business:b,live:true,startedAt:at,...o});
|
||||
line(`bus host up: business ${b}, socket /fixture/bus.sock, notifier off`);
|
||||
switch(mode){
|
||||
case 'none':break;
|
||||
case 'two':ok();line(`launched pm-${b}: run ${run}b, claude opus`);entry({});break;
|
||||
case 'badword':line(`launched pm-${b}: job ${run}, claude opus`);entry({});break;
|
||||
case 'notlisted':ok();break;
|
||||
case 'early':ok();entry({startedAt:new Date(now-3600e3).toISOString()});break;
|
||||
case 'human':entry({run:`human-${pid}`});break;
|
||||
case 'other':ok();entry({business:'other'});break;
|
||||
case 'dead':ok();entry({live:false});break;
|
||||
// Row 232: two live entries for the PM's instance, and one for it beside a live coder run.
|
||||
case 'twolive':ok();entry({});entry({run:`${run}b`});break;
|
||||
case 'coder':ok();entry({});entry({run:`${run}c`,instance:'coder'});break;
|
||||
default:ok();entry({});
|
||||
}
|
||||
fs.writeFileSync(lf,JSON.stringify(list));
|
||||
};
|
||||
if(verb==='show'){
|
||||
// `show [--timestamp=unix] UNIT -p P...` and `show -p P UNIT`, as the external path calls it.
|
||||
let u='';const props=[];for(let i=0;i<rest.length;i++){if(rest[i]==='-p')props.push(rest[++i]);else if(!rest[i].startsWith('--'))u=rest[i]}
|
||||
const f=unitFile(u),st=load()[u]||{},id=f?path.basename(f.wd):'';
|
||||
// ActiveEnterTimestamp (row 232) is the start's second, or what the enter fault says: a JSON
|
||||
// {value} or {value,at,then}, which answers then from epoch ms at on.
|
||||
const enter=()=>{
|
||||
if(!fault('enter'))return st.enter;
|
||||
const o=JSON.parse(fs.readFileSync(faultFile('enter'),'utf8'));return o.at!==undefined&&Date.now()>=o.at?o.then:o.value;
|
||||
};
|
||||
for(const p of props){
|
||||
if(p==='LoadState')console.log(`LoadState=${f?'loaded':'not-found'}`);
|
||||
else if(p==='WorkingDirectory')console.log(`WorkingDirectory=${f?f.wd+(!isBus(u)&&fault(`check-${id}`)?'-moved':''):''}`);
|
||||
else if(p==='ExecStart')console.log(f&&f.exec?`ExecStart={ path=${f.exec.split(' ')[0]} ; argv[]=${f.exec} ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }`:'ExecStart=');
|
||||
else if(p==='Environment')console.log(`Environment=${f?f.env.join(' '):''}`);
|
||||
else if(p==='MainPID')console.log(`MainPID=${st.pid||0}`);
|
||||
else if(p==='DropInPaths')console.log(`DropInPaths=${f?f.drops.join(' '):''}`);
|
||||
else if(p==='FragmentPath')console.log(`FragmentPath=${f&&!fault('no-fragment')?f.file:''}`);
|
||||
else if(p==='ActiveEnterTimestamp'){const v=st.state==='active'?enter():undefined;console.log(`ActiveEnterTimestamp=${v===undefined?'':'@'+v}`)}
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
if(verb==='is-active'){
|
||||
const q=rest[0]==='--quiet',u=q?rest[1]:rest[0];let s=(load()[u]||{}).state||'inactive';
|
||||
// bus-stops-after N (T29): the next N bus checks answer the real state, every later one answers
|
||||
// inactive, and units.json stays as it is, as if the bus stopped between two checks.
|
||||
if(isBus(u)&&fault('bus-stops-after')){const n=Number(fs.readFileSync(faultFile('bus-stops-after'),'utf8'));if(n>0)fs.writeFileSync(faultFile('bus-stops-after'),String(n-1));else s='inactive'}
|
||||
if(!q)console.log(s);process.exit(s==='active'?0:3);
|
||||
}
|
||||
if(verb==='is-enabled'){console.log('disabled');process.exit(1)}
|
||||
if(verb==='daemon-reload')process.exit(0);
|
||||
if(verb==='list-units'){
|
||||
if(fault('list-fails'))process.exit(1);
|
||||
const pats=rest.filter((a)=>!a.startsWith('--')).map((p)=>new RegExp('^'+p.replace(/\./g,'\\.').replace(/\*/g,'.*')+'$'));
|
||||
for(const [u,s] of Object.entries(load()))if(pats.some((re)=>re.test(u)))console.log(`${u} loaded ${s.state} ${s.state==='active'?'running':'dead'} fixture`);
|
||||
process.exit(0);
|
||||
}
|
||||
if(verb==='start'){
|
||||
const u=rest[0],f=unitFile(u);if(!f){console.error(`Unit ${u} not found.`);process.exit(5)}
|
||||
const s=load(),id=path.basename(f.wd),d=dataOf(f);
|
||||
const enter=Math.floor((Date.now()-Number(process.env.SD_CLOCK_OFFSET||'0')*1000)/1000);
|
||||
if(!isBus(u)){s[u]={state:'active',pid:null,wd:f.wd,enter};save(s);process.exit(0)}
|
||||
if(fault(`start-dies-${id}`)){s[u]={state:'failed',pid:null,wd:f.wd};save(s);process.exit(0)}
|
||||
// Row 185 (T48): the copy's start writes one event before the holder opens the store, so the
|
||||
// store differs from a backup the run took before the start. The close leaves no -wal.
|
||||
if(fault(`start-writes-row-${id}`)){
|
||||
const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(d,'bus','bus.sqlite'));
|
||||
db.prepare('INSERT INTO events (at) VALUES (?)').run('2999-01-01T00:00:00.000Z');db.close();
|
||||
}
|
||||
snap('start',d);
|
||||
const off=process.env.SD_CLOCK_OFFSET||'0';
|
||||
// The holder must not inherit stack-release.sh's lock (fd 9) or any other fd above 2.
|
||||
const close='for fd in /proc/self/fd/*; do n=${fd##*/}; [ "$n" -gt 2 ] && eval "exec $n>&-" 2>/dev/null; done; exec "$@"';
|
||||
const ch=cp.spawn('bash',['-c',close,'holder',process.execPath,path.join(__dirname,'..','holder.cjs'),d,inst(u),sd,off,perData(d,'host')],{detached:true,stdio:'ignore'});
|
||||
ch.unref();
|
||||
fs.appendFileSync(path.join(sd,'pids.log'),ch.pid+'\n');
|
||||
const ready=path.join(sd,`ready-${ch.pid}`),end=Date.now()+10000;
|
||||
while(!fs.existsSync(ready)){if(Date.now()>end){console.error('fixture holder did not start');process.exit(1)}sleep(20)}
|
||||
s[u]={state:'active',pid:ch.pid,wd:f.wd,enter};save(s);
|
||||
if(f.exec.split(' ').slice(1).includes('--pm'))pm(u,d,id,ch.pid,Number(off));
|
||||
process.exit(0);
|
||||
}
|
||||
if(verb==='stop'){
|
||||
const u=rest[0],s=load(),st=s[u];
|
||||
if(st&&st.pid){
|
||||
try{process.kill(st.pid,'SIGTERM')}catch{}
|
||||
const end=Date.now()+10000;while(alive(st.pid)&&Date.now()<end)sleep(20);
|
||||
if(alive(st.pid))try{process.kill(st.pid,'SIGKILL')}catch{}
|
||||
}
|
||||
if(st){s[u]={...st,state:'inactive',pid:null};save(s)}
|
||||
if(isBus(u)){
|
||||
const d=dataOf(unitFile(u));
|
||||
// One-shot: the stop leaves an empty -wal or -shm beside the store, as an unclean close would.
|
||||
for(const [name,ext] of [['stop-leaves-wal','-wal'],['stop-leaves-shm','-shm']])if(fault(name)){fs.rmSync(faultFile(name));fs.writeFileSync(path.join(d,'bus','bus.sqlite'+ext),'')}
|
||||
// One-shot (T45): the stop writes one row after any STAMP into the table the fault names, as a
|
||||
// write the bus made between rollback's count and its stop would. The close leaves no -wal.
|
||||
if(fault('stop-writes-row')){
|
||||
const t=fs.readFileSync(faultFile('stop-writes-row'),'utf8');fs.rmSync(faultFile('stop-writes-row'));
|
||||
const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(d,'bus','bus.sqlite'));
|
||||
db.exec(`CREATE TABLE IF NOT EXISTS ${t} (seq INTEGER PRIMARY KEY, at TEXT NOT NULL)`);
|
||||
db.prepare(`INSERT INTO ${t} (at) VALUES (?)`).run('2999-01-01T00:00:00.000Z');db.close();
|
||||
}
|
||||
snap('stop',d);
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
console.error(`fixture systemctl: unsupported: ${args.join(' ')}`);process.exit(1);
|
||||
}
|
||||
|
||||
// journalctl --user -u UNIT --since @EPOCH -o cat. A bare local time gets exit 1, as M35 needs.
|
||||
export function journalctl(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
|
||||
const sd=process.env.SD_STATE,a=process.argv.slice(2);let unit='',since='';
|
||||
for(let i=0;i<a.length;i++){if(a[i]==='-u')unit=a[++i];else if(a[i]==='--since')since=a[++i]}
|
||||
fs.appendFileSync(path.join(sd,'journal-since.log'),since+'\n');
|
||||
if(fs.existsSync(path.join(sd,'faults','journal-fails')))process.exit(1);
|
||||
let from;
|
||||
if(/^@\d+$/.test(since))from=Number(since.slice(1));
|
||||
else{const m=/^(\d{4}-\d\d-\d\d) (\d\d:\d\d:\d\d) UTC$/.exec(since);if(!m)process.exit(1);from=Date.parse(`${m[1]}T${m[2]}Z`)/1000}
|
||||
let lines=[];try{lines=fs.readFileSync(path.join(sd,'journal.jsonl'),'utf8').split('\n').filter(Boolean).map((l)=>JSON.parse(l))}catch{}
|
||||
for(const o of lines)if(o.unit===unit&&o.epoch>=from)console.log(o.text);
|
||||
}
|
||||
|
||||
export function systemdAnalyze(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
|
||||
if(process.argv[2]!=='--user'||process.argv[3]!=='unit-paths')process.exit(1);
|
||||
console.log(path.join(process.env.XDG_CONFIG_HOME,'systemd','user'));
|
||||
// The unit-path fault adds its text as one more path, as T28's relative path needs.
|
||||
const extra=path.join(process.env.SD_STATE,'faults','unit-path');
|
||||
if(fs.existsSync(extra))console.log(fs.readFileSync(extra,'utf8'));
|
||||
}
|
||||
|
||||
// The bus host: holds the store open in WAL mode until SIGTERM, as the broker does. It writes its
|
||||
// status to the host file systemctl names for its data root.
|
||||
export function holder(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
|
||||
const [data,business,sd,offset,host]=process.argv.slice(2),bus=path.join(data,'bus');
|
||||
fs.mkdirSync(bus,{recursive:true,mode:0o700});
|
||||
const db=new DatabaseSync(path.join(bus,'bus.sqlite'));
|
||||
if(!db.prepare("SELECT count(*) AS n FROM sqlite_master WHERE name='meta'").get().n){
|
||||
db.exec('PRAGMA journal_mode=WAL');
|
||||
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
|
||||
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
|
||||
db.prepare("INSERT INTO meta VALUES ('schema_version', '3g')").run();
|
||||
}
|
||||
fs.writeFileSync(path.join(bus,'writer.lock'),String(process.pid));
|
||||
const at=new Date(Date.now()-Number(offset||0)*1000).toISOString();
|
||||
fs.writeFileSync(host+'.tmp',JSON.stringify({host:{business,pid:process.pid,live:true,startedAt:at,notifier:'ok'},boot:{ok:true,business,at}}));
|
||||
fs.renameSync(host+'.tmp',host);
|
||||
process.on('SIGTERM',()=>{db.close();fs.rmSync(path.join(bus,'writer.lock'),{force:true});process.exit(0)});
|
||||
fs.writeFileSync(path.join(sd,`ready-${process.pid}`),'');
|
||||
setInterval(()=>{},1<<30);
|
||||
}
|
||||
|
||||
// A copy's scripts/mosaic: bus status and launches list from what the holder and systemctl wrote
|
||||
// for the data root of the config it runs under (row 190), the default's being $SD_DATA.
|
||||
export function mosaicStub(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
|
||||
const id=path.basename(path.resolve(__dirname,'..')),sd=process.env.SD_STATE,a=process.argv.slice(2).join(' ');
|
||||
const fault=(n)=>fs.existsSync(path.join(sd,'faults',n));
|
||||
let d=process.env.SD_DATA;try{d=JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8')).dataRoot}catch{}
|
||||
const perData=(n)=>path.join(sd,d===process.env.SD_DATA?`${n}.json`:`${n}-${path.basename(d)}.json`);
|
||||
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
|
||||
if(a==='bus status --json'){
|
||||
if(fault(`status-${id}`))process.exit(1);
|
||||
// Row 232: status-until-<copy> fails bus status until the epoch ms it holds.
|
||||
const until=path.join(sd,'faults',`status-until-${id}`);
|
||||
if(fs.existsSync(until)&&Date.now()<Number(fs.readFileSync(until,'utf8')))process.exit(1);
|
||||
let s={host:null,boot:null};
|
||||
try{s=JSON.parse(fs.readFileSync(perData('host'),'utf8'));s.host.live=alive(s.host.pid)}catch{}
|
||||
console.log(JSON.stringify(s));
|
||||
}else if(a==='bus status')console.log(`bus: fixture status for ${id}`);
|
||||
else if(a==='launches list --json'){
|
||||
if(fault(`launches-${id}`))process.exit(1);
|
||||
let l=[];try{l=JSON.parse(fs.readFileSync(perData('launches'),'utf8'))}catch{}
|
||||
console.log(JSON.stringify(l));
|
||||
}else{console.error(`fixture mosaic: ${a}`);process.exit(4)}
|
||||
}
|
||||
|
||||
// A copy's migrate-schema.mjs: records the backups it saw, then sets 3h, or 3h-partial and exit 1.
|
||||
export function migrateStub(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
|
||||
const data=process.argv[2],sd=process.env.SD_STATE,dir=path.join(data,'backups','stack');
|
||||
const backups=fs.existsSync(dir)?fs.readdirSync(dir).sort().map((name)=>({name,sums:fs.existsSync(path.join(dir,name,'SHA256SUMS'))})):[];
|
||||
fs.writeFileSync(path.join(sd,'migrate.json'),JSON.stringify({backups}));
|
||||
const fail=fs.existsSync(path.join(sd,'faults','migrate-fails'));
|
||||
const db=new DatabaseSync(path.join(data,'bus','bus.sqlite'));
|
||||
db.prepare("UPDATE meta SET value=? WHERE key='schema_version'").run(fail?'3h-partial':'3h');db.close();
|
||||
if(fail)process.exit(1);
|
||||
console.log('fixture migrate: 3h');
|
||||
}
|
||||
|
||||
// --- The module fixture: one repo, its copies and the stand-ins, shared by every test. ---
|
||||
|
||||
export const base=realpathSync(mkdtempSync(join(tmpdir(),'stack-release-')));
|
||||
after(()=>{execFileSync('chmod',['-R','u+w',base]);rmSync(base,{recursive:true,force:true})});
|
||||
export const repo=join(base,'repo'),COPIES=join(base,'copies'),SBIN=join(base,'sbin');
|
||||
mkdirSync(repo);mkdirSync(COPIES);mkdirSync(SBIN);
|
||||
writeFileSync(join(base,'package.json'),'{"type":"commonjs"}\n');
|
||||
export const tracked=execFileSync('git',['-C',ROOT,'ls-files','-co','--exclude-standard','--','scripts','packages','adapters','roles','RELEASE','package.json','package-lock.json',':!packages/*/tests',':!scripts/foundation'],{encoding:'utf8'}).trim().split('\n');
|
||||
for(const f of tracked)if(existsSync(join(ROOT,f)))cpSync(join(ROOT,f),join(repo,f));
|
||||
export const pkg={name:'copy-fixture',version:'1.0.0'};
|
||||
writeFileSync(join(repo,'package.json'),JSON.stringify(pkg));
|
||||
writeFileSync(join(repo,'package-lock.json'),JSON.stringify({name:pkg.name,version:pkg.version,lockfileVersion:3,packages:{'':pkg}}));
|
||||
git(repo,'init','-q','-b','refactor');git(repo,'config','gc.auto','0');git(repo,'add','.');git(repo,'commit','-qm','fixture');
|
||||
export const commit=(msg)=>{git(repo,'commit','-q','--allow-empty','-m',msg);return git(repo,'rev-parse','HEAD')};
|
||||
// refactor: F A B C1 C, then M merging side branch S (from A). U sits on a branch nothing reaches.
|
||||
export const F=git(repo,'rev-parse','HEAD'),A=commit('row 901: A'),B=commit('row 902: B'),C1=commit('row 903: C1'),C=commit('row 904: C');
|
||||
git(repo,'checkout','-q','-b','side',A);export const S=commit('row 905: S');
|
||||
git(repo,'checkout','-q','-b','unreached',A);export const U=commit('row 906: U');
|
||||
git(repo,'checkout','-q','refactor');git(repo,'merge','-q','--no-ff','-m','merge side','side');
|
||||
export const M=git(repo,'rev-parse','HEAD');git(repo,'update-ref','refs/remotes/origin/refactor',M);
|
||||
// `to` false leaves out the `to:` lines (C1); a string puts that value in each one (A). The --plan
|
||||
// steps line can't be read from either (T27).
|
||||
export const storeText=(v,froms,to=true)=>`export const SCHEMA_VERSION = '${v}';\nexport const MIGRATIONS = [\n${froms.map((f,i)=>` {\n from: '${f}',\n${to===false?'':` to: '${to===true?froms[i+1]??v:to}',\n`} run() {},\n },\n`).join('')}];\n`;
|
||||
// Plug-in kinds (row 169): C1 and C know the harness kind, as a copy at or past c216f47db does. The
|
||||
// rest have it under another name, as ca15d73e9 has no harness kind. Every copy has the real business
|
||||
// loader and roles, which the tool imports to read the business file.
|
||||
export const KINDS_NEW=readFileSync(join(repo,'packages','plugins','src','kinds.mjs'),'utf8');
|
||||
export const KINDS_OLD=KINDS_NEW.replace('\n harness: Object.freeze({','\n "harness-before-c216f47db": Object.freeze({');
|
||||
assert.notEqual(KINDS_OLD,KINDS_NEW);
|
||||
// Row 200 (#1698): C1 and C take --sso and have the row 192 loader and checkConsole; C's checker
|
||||
// refuses RS256, a stand-in for a target whose rules differ from the repository's. B has the loader
|
||||
// from before row 192, which refuses a `console` key. The other copies have no loader.
|
||||
const LOADER=readFileSync(join(repo,'scripts','mosaic-config.mjs'),'utf8'),SSO_CONFIG=readFileSync(join(repo,'packages','webui','src','sso-config.mjs'),'utf8');
|
||||
const LOADER_OLD=LOADER.replace('["configVersion", "environment", "dataRoot", "execution", "console"]','["configVersion", "environment", "dataRoot", "execution"]');
|
||||
const SSO_CONFIG_C=SSO_CONFIG.replace("'ES256', 'RS256',","'ES256',");
|
||||
assert.notEqual(LOADER_OLD,LOADER);assert.notEqual(SSO_CONFIG_C,SSO_CONFIG);
|
||||
const ssoFiles=(id)=>id===B?[['scripts/mosaic-config.mjs',LOADER_OLD]]:[C1,C].includes(id)?[['scripts/mosaic-config.mjs',LOADER],['packages/webui/src/sso-config.mjs',id===C?SSO_CONFIG_C:SSO_CONFIG]]:[];
|
||||
for(const [id,store] of [[F,'// a copy from before SCHEMA_VERSION\n'],[A,storeText('3g',['3f'],'3x')],[B,storeText('3g',['3f'])],[C1,storeText('3g',['3f'],false)],[C,storeText('3h',['3f','3g'])],[S,storeText('3g',['3f'])],[U,storeText('3g',['3f'])]]){
|
||||
const d=join(COPIES,id);
|
||||
for(const [f,text,mode] of [['scripts/mosaic',script(mosaicStub),0o755],['packages/webui/src/cli.mjs',[B,C1,C].includes(id)?`// fixture Console, never run\n// serve accepts '--tester-port'\n// serve accepts '--proof-header' (row 178)\n${[C1,C].includes(id)?"// serve accepts '--sso' (row 192)\n":''}`:'// fixture Console before tester listener, never run\n',0o644],['packages/bus/src/store.mjs',store,0o644],['packages/bus/src/migrate-schema.mjs',`(${migrateStub})();\n`,0o644],['packages/plugins/src/kinds.mjs',[C1,C].includes(id)?KINDS_NEW:KINDS_OLD,0o644],...ssoFiles(id).map(([f,text])=>[f,text,0o644])]){
|
||||
mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text,{mode});
|
||||
}
|
||||
for(const p of ['packages/business/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true});
|
||||
execFileSync('chmod',['-R','a-w',d]);
|
||||
}
|
||||
for(const [name,fn] of [['systemctl',systemctl],['journalctl',journalctl],['systemd-analyze',systemdAnalyze]])writeFileSync(join(SBIN,name),script(fn),{mode:0o755});
|
||||
writeFileSync(join(base,'holder.cjs'),`(${holder})();\n`);
|
||||
export const SR=join(repo,'scripts','stack-release.sh');
|
||||
|
||||
// --- Per-test roots and helpers ---
|
||||
|
||||
export const sha=(f)=>createHash('sha256').update(readFileSync(f)).digest('hex');
|
||||
export const lines=(f)=>existsSync(f)?readFileSync(f,'utf8').split('\n').filter(Boolean):[];
|
||||
export const calls=(e)=>lines(join(e.sd,'calls.log'));
|
||||
export const startsStops=(e,from)=>calls(e).slice(from).filter((c)=>/^(start|stop) /.test(c));
|
||||
export const logLines=(e)=>lines(e.log).map((l)=>JSON.parse(l));
|
||||
export const pointer=(e)=>existsSync(e.pointer)?JSON.parse(readFileSync(e.pointer,'utf8')):null;
|
||||
export const units=(e)=>JSON.parse(readFileSync(join(e.sd,'units.json'),'utf8'));
|
||||
export const snaps=(e)=>lines(join(e.sd,'snap.jsonl')).map((l)=>JSON.parse(l));
|
||||
export const fault=(e,name,text='')=>writeFileSync(join(e.sd,'faults',name),text);
|
||||
export const sysctl=(e,...a)=>execFileSync(join(SBIN,'systemctl'),['--user',...a],{env:e.env,encoding:'utf8'});
|
||||
export const edit=(f,from,to)=>{const s=readFileSync(f,'utf8');assert.ok(s.includes(from),`${f} has no ${from}`);writeFileSync(f,s.replace(from,to))};
|
||||
export const posts=(e)=>readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-'));
|
||||
export const backupSum=(e,act)=>readFileSync(join(e.data,'backups','stack',act,'SHA256SUMS'),'utf8').split(' ')[0];
|
||||
export const commitLines=(range)=>git(repo,'log','--oneline','--no-merges','--max-count=200',range).split('\n').map((l)=>` ${l}`);
|
||||
// The units, the wrapper, the pointer and the store, by sha256.
|
||||
export const fp=(e)=>({bus:sha(join(e.units,e.tmpl)),console:sha(join(e.units,e.con)),wrapper:sha(join(e.bin,e.wrapper)),pointer:existsSync(e.pointer)?sha(e.pointer):null,store:sha(e.store)});
|
||||
// The schema version: readOnly while the holder has the store open, immutable when it is stopped.
|
||||
export function storeVersion(file,held){
|
||||
const db=held?new DatabaseSync(file,{readOnly:true}):new DatabaseSync(pathToFileURL(file).href+'?immutable=1',{readOnly:true});
|
||||
try{return db.prepare("SELECT value FROM meta WHERE key='schema_version'").get().value}finally{db.close()}
|
||||
}
|
||||
// Every file and directory under p with size and mtime: T23 and T16 compare it before and after.
|
||||
export function tree(p){
|
||||
const out=[];
|
||||
const walk=(q)=>{let st;try{st=statSync(q)}catch{return}out.push(st.isDirectory()?`${q}/`:`${q} ${st.size} ${st.mtimeMs}`);if(st.isDirectory())for(const n of readdirSync(q).sort())walk(join(q,n))};
|
||||
walk(p);return out;
|
||||
}
|
||||
// T17: what a run may never remove. Under --units external confirm removes a set's AWAITING-UNITS
|
||||
// and FAILED when it closes the activation (row 232, r6 A5 and R4); user mode writes neither.
|
||||
const markers=/\/(AWAITING-UNITS|FAILED) /;
|
||||
export const kept=(e)=>[...tree(join(e.data,'backups','stack')),...tree(join(e.state,'stack-rollback')).filter((l)=>!markers.test(l)),...posts(e).flatMap((n)=>tree(join(e.data,n))),...tree(COPIES)].map((l)=>l.replace(/ .*/,''));
|
||||
|
||||
// The business file (row 169): the committed example made loadable, as acme, with `plugins` when given.
|
||||
export const EXAMPLE=readFileSync(join(repo,'packages','business','examples','mosaic-stack.example.json'),'utf8');
|
||||
export function biz(e,plugins){
|
||||
const b=JSON.parse(EXAMPLE);let n=0;
|
||||
const walk=(o)=>{for(const [k,v] of Object.entries(o)){if(k==='botId')o[k]=++n;else if(v==='YYYY-MM-DD')o[k]='2027-01-01';else if(v&&typeof v==='object')walk(v)}};
|
||||
walk(b);b.id='acme';if(plugins!==undefined)b.plugins=plugins;
|
||||
const f=join(dirname(e.cfg),'businesses','acme.json');mkdirSync(dirname(f),{recursive:true});
|
||||
writeFileSync(f,JSON.stringify(b,null,2)+'\n',{mode:0o600});chmodSync(f,0o600);
|
||||
}
|
||||
|
||||
export async function mk(t){
|
||||
const root=realpathSync(mkdtempSync(join(tmpdir(),'sr-')));
|
||||
const e={root,data:join(root,'data'),bin:join(root,'bin'),sd:join(root,'sd'),units:join(root,'xcfg','systemd','user'),cfg:join(root,'cfg','config.json'),claude:join(root,'claude-bin','claude')};
|
||||
// The default instance's names; named() sets NAME's (row 190).
|
||||
Object.assign(e,{flags:[],bus:BUS,tmpl:'[email protected]',con:CON,wrapper:'mosaic'});
|
||||
Object.assign(e,{state:join(e.data,'state'),pointer:join(e.data,'state','stack-active.json'),log:join(e.data,'state','stack-activation-log.jsonl'),store:join(e.data,'bus','bus.sqlite'),lock:join(e.data,'state','stack-release.lock')});
|
||||
for(const d of [join(root,'home'),dirname(e.cfg),join(e.data,'bus'),e.bin,dirname(e.claude),join(e.sd,'faults'),e.units,join(root,'xdata')])mkdirSync(d,{recursive:true});
|
||||
chmodSync(join(e.data,'bus'),0o700);
|
||||
writeFileSync(e.cfg,JSON.stringify({configVersion:1,environment:'development',dataRoot:e.data,execution:{backend:'docker',provider:'zai',model:'glm-5.3-flash',adapter:'mock'}}));
|
||||
biz(e);
|
||||
writeFileSync(join(e.bin,'mosaic'),`#!/usr/bin/env bash\n${MARKER}\n[ "\${1:-}" = __lease-capability ] && exec "${e.bin}/mosaic-old" "$@"\nexec "/old/scripts/mosaic" "$@"\n`,{mode:0o755});
|
||||
writeFileSync(join(e.bin,'mosaic-old'),'#!/bin/sh\nexit 0\n',{mode:0o755});
|
||||
writeFileSync(e.claude,'#!/bin/sh\necho "9.9.9 (Claude Code fixture)"\n',{mode:0o755});
|
||||
e.env={PATH:`${SBIN}:${dirname(NODE)}:/usr/bin:/bin`,HOME:join(root,'home'),XDG_CONFIG_HOME:join(root,'xcfg'),XDG_DATA_HOME:join(root,'xdata'),MOSAIC_CONFIG:e.cfg,SD_STATE:e.sd,SD_DATA:e.data,TMPDIR:tmpdir()};
|
||||
const server=await healthz(e);
|
||||
t.after(()=>{
|
||||
for(const pid of lines(join(e.sd,'pids.log')).map(Number)){
|
||||
try{if(readFileSync(`/proc/${pid}/cmdline`,'utf8').includes(join(base,'holder.cjs')))process.kill(pid,'SIGKILL')}catch{}
|
||||
}
|
||||
for(const s of [server,...(e.servers??[])])s.close();
|
||||
execFileSync('chmod',['-R','u+w',root]);rmSync(root,{recursive:true,force:true});
|
||||
});
|
||||
return e;
|
||||
}
|
||||
|
||||
// The Console's /healthz on e.port: 200 while the stand-in has e.con active and no healthz-<copy>
|
||||
// fault names its copy.
|
||||
export async function healthz(e){
|
||||
const server=createServer((q,r)=>{
|
||||
let st={};try{st=units(e)[e.con]||{}}catch{}
|
||||
const ok=q.url==='/healthz'&&st.state==='active'&&!existsSync(join(e.sd,'faults',`healthz-${basename(st.wd||'')}`));
|
||||
r.writeHead(ok?200:500);r.end();
|
||||
});
|
||||
server.listen(0,'127.0.0.1');await once(server,'listening');e.port=String(server.address().port);
|
||||
return server;
|
||||
}
|
||||
|
||||
// Row 190: stack instance NAME on e's host. It shares e's stand-ins, unit directory and $BIN, and has
|
||||
// its own config and data root from bootstrap --instance, its own Console port and business acme.
|
||||
// Its runs pass --instance NAME and no MOSAIC_CONFIG.
|
||||
export async function named(e,name){
|
||||
const env={...e.env};delete env.MOSAIC_CONFIG;
|
||||
execFileSync('bash',[join(repo,'scripts','bootstrap.sh'),'--instance',name],{env,stdio:'ignore'});
|
||||
const data=join(env.HOME,`.mosaic-${name}`);
|
||||
const n={...e,env,data,cfg:join(env.HOME,'.config',`mosaic-${name}`,'config.json'),flags:['--instance',name],bus:`mosaic-bus-${name}@acme.service`,tmpl:`mosaic-bus-${name}@.service`,con:`mosaic-webui-${name}.service`,wrapper:`mosaic-${name}`,name};
|
||||
Object.assign(n,{state:join(data,'state'),pointer:join(data,'state','stack-active.json'),log:join(data,'state','stack-activation-log.jsonl'),store:join(data,'bus','bus.sqlite'),lock:join(data,'state','stack-release.lock')});
|
||||
mkdirSync(join(data,'bus'),{mode:0o700});
|
||||
biz(n);
|
||||
(e.servers??=[]).push(await healthz(n));
|
||||
return n;
|
||||
}
|
||||
|
||||
export async function run(e,...args){
|
||||
const before=existsSync(e.data)?kept(e):[];
|
||||
const ch=spawn('bash',[e.sr??SR,...args,...e.flags,'--copies',COPIES,'--bin',e.bin],{env:e.env,stdio:['ignore','pipe','pipe']});
|
||||
let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d});
|
||||
const timer=setTimeout(()=>ch.kill('SIGKILL'),120000);
|
||||
const [code]=await once(ch,'close');clearTimeout(timer);
|
||||
const now=new Set(kept(e));
|
||||
for(const p of before)assert.ok(now.has(p),`T17: ${args.join(' ')} removed ${p}`);
|
||||
return {code,out,err};
|
||||
}
|
||||
|
||||
// The stack on copy `copy` for business acme, both units active. pointer: adopt (a real adopt and
|
||||
// its log line), hand (written here, no log) or none. Snapshots start after the seed.
|
||||
export async function seed(e,{copy=B,pm=false,store='3g',pointer:how='adopt'}={}){
|
||||
const db=new DatabaseSync(e.store);
|
||||
db.exec('PRAGMA journal_mode=WAL');
|
||||
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
|
||||
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
|
||||
db.prepare("INSERT INTO meta VALUES ('schema_version', ?)").run(store);
|
||||
for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`);
|
||||
db.close();
|
||||
const sh=(name,...a)=>execFileSync('bash',[join(repo,'scripts',name),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
|
||||
sh('bus-service.sh','install',...e.flags,'--copy',copy,'--copies',COPIES,...(pm?['--pm']:[]),'--claude',e.claude);
|
||||
sh('webui-service.sh','install',...e.flags,'--copy',copy,'--copies',COPIES,'--port',e.port,'--business','acme');
|
||||
sh('stack-copy.sh','wrapper',copy,...e.flags,'--copies',COPIES,'--bin',e.bin);
|
||||
// The seeded host started two minutes ago, so its boot status and PM run predate any run's start.
|
||||
execFileSync(join(SBIN,'systemctl'),['--user','start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:'120'}});
|
||||
sysctl(e,'start',e.con);
|
||||
if(how==='adopt'){const r=await run(e,'adopt','acme');assert.equal(r.code,0,r.err)}
|
||||
else if(how==='hand'){
|
||||
mkdirSync(e.state,{recursive:true});
|
||||
writeFileSync(e.pointer,JSON.stringify({copy,business:'acme',pm,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g',activation:'manual',at:'2026-10-18T00:00:00Z'})+'\n');
|
||||
}
|
||||
writeFileSync(join(e.sd,'snap.jsonl'),'');
|
||||
}
|
||||
|
||||
export const rolledBack=(code,to=B)=>`stack-release: ${code}; rolled back to ${to}, which passed the gate\n`;
|
||||
export const planBackup=(e)=>`backup: the run backs up the stopped store to ${e.data}/backups/stack/<activation>/ before any migration`;
|
||||
export const planNoBackup='backup: none; the run rolls back before the backup';
|
||||
export const planTail=(e,to,pm=false)=>[`bus: scripts/bus-service.sh install --copy ${to} --copies ${COPIES}${pm?' --pm':''} --claude ${e.claude}`,'proof: off','send: off',`Console: scripts/webui-service.sh install --copy ${to} --copies ${COPIES} --port ${e.port} --business acme`,`CLI: scripts/stack-copy.sh wrapper ${to} --copies ${COPIES} --bin ${e.bin}`];
|
||||
@@ -13,422 +13,7 @@ import {join,dirname,basename} from 'node:path';
|
||||
import {fileURLToPath,pathToFileURL} from 'node:url';
|
||||
import {once} from 'node:events';
|
||||
import {ALGS} from '../../webui/src/sso-config.mjs';
|
||||
|
||||
const ROOT=fileURLToPath(new URL('../../..',import.meta.url)).replace(/\/$/,'');
|
||||
const NODE=process.execPath;
|
||||
const MARKER='# mosaic-cli wrapper (Q81, agents/sage/work/q81/mosaic-cli.sh)';
|
||||
const BUS='[email protected]',CON='mosaic-webui.service';
|
||||
const git=(cwd,...args)=>execFileSync('git',['-c','user.name=fixture','-c','user.email=fixture@invalid','-c','commit.gpgsign=false','-c','core.hooksPath=/dev/null','-C',cwd,...args],{encoding:'utf8'}).trim();
|
||||
const script=(fn)=>`#!${NODE}\n(${fn})();\n`;
|
||||
|
||||
// --- Stand-ins. Each is serialized into its own file, so it uses only process and its own scope. ---
|
||||
|
||||
// systemctl --user: unit state in $SD_STATE/units.json, every call in calls.log. A bus start runs
|
||||
// holder.cjs (the store held open in WAL mode, writer.lock, host.json) and, with --pm in the unit's
|
||||
// ExecStart, writes the `launched` journal line and the launches entry unless a pm-<copy> fault
|
||||
// names another case. Faults are files in $SD_STATE/faults named for the copy they hit. Units of a
|
||||
// named instance (row 190) run on their own data root and host files beside the default's.
|
||||
function systemctl(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process'),crypto=process.getBuiltinModule('node:crypto');
|
||||
const sd=process.env.SD_STATE,data=process.env.SD_DATA,unitDir=path.join(process.env.XDG_CONFIG_HOME,'systemd','user');
|
||||
const args=process.argv.slice(2).filter((a)=>a!=='--user'),[verb,...rest]=args;
|
||||
fs.appendFileSync(path.join(sd,'calls.log'),args.join(' ')+'\n');
|
||||
const faultFile=(n)=>path.join(sd,'faults',n),fault=(n)=>fs.existsSync(faultFile(n));
|
||||
const stFile=path.join(sd,'units.json');
|
||||
const load=()=>{try{return JSON.parse(fs.readFileSync(stFile,'utf8'))}catch{return {}}};
|
||||
const save=(s)=>{fs.writeFileSync(stFile+'.tmp',JSON.stringify(s));fs.renameSync(stFile+'.tmp',stFile)};
|
||||
const sleep=(ms)=>Atomics.wait(new Int32Array(new SharedArrayBuffer(4)),0,0,ms);
|
||||
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
|
||||
// Row 190: a bus unit is the default's mosaic-bus@B or instance NAME's mosaic-bus-NAME@B; %i is B.
|
||||
const isBus=(u)=>/^mosaic-bus(-[a-z][a-z0-9]{1,15})?@[a-z0-9-]+\.service$/.test(u),inst=(u)=>u.slice(u.indexOf('@')+1,-'.service'.length);
|
||||
const unitFile=(u)=>{
|
||||
const main=isBus(u)?u.replace(/@[^@]*$/,'@.service'):u,file=path.join(unitDir,main);
|
||||
if(!fs.existsSync(file))return null;
|
||||
// The drop-in directories in systemd's order: service.d, the name cut after each hyphen, then
|
||||
// the template's and the unit's own.
|
||||
const stem=main.slice(0,main.search(/@?\.service$/)),dirs=['service.d'],drops=[];
|
||||
for(let i=0;i<stem.length;i++)if(stem[i]==='-')dirs.push(stem.slice(0,i+1)+'.service.d');
|
||||
for(const d of [...dirs,...(isBus(u)?[main+'.d',u+'.d']:[u+'.d'])]){const dir=path.join(unitDir,d);if(fs.existsSync(dir))for(const f of fs.readdirSync(dir).sort())if(f.endsWith('.conf'))drops.push(path.join(dir,f))}
|
||||
const r={wd:'',exec:'',env:[],drops};
|
||||
for(const f of [file,...drops])for(let line of fs.readFileSync(f,'utf8').split('\n')){
|
||||
line=line.replace(/%i/g,isBus(u)?inst(u):'').replace(/%h/g,process.env.HOME);
|
||||
const m=/^(WorkingDirectory|ExecStart|Environment)=(.*)$/.exec(line);if(!m)continue;
|
||||
if(m[1]==='WorkingDirectory')r.wd=m[2];else if(m[1]==='ExecStart'){if(m[2])r.exec=m[2]}else r.env.push(m[2]);
|
||||
}
|
||||
return r;
|
||||
};
|
||||
// A named instance's unit sets MOSAIC_CONFIG, and its bus works on that config's data root. A unit
|
||||
// without it is the default's, on $SD_DATA. Each data root has its own host and launches files.
|
||||
const dataOf=(f)=>{const c=f&&f.env.find((v)=>v.startsWith('MOSAIC_CONFIG='));return c?JSON.parse(fs.readFileSync(c.slice('MOSAIC_CONFIG='.length),'utf8')).dataRoot:data};
|
||||
const perData=(d,n)=>path.join(sd,d===data?`${n}.json`:`${n}-${path.basename(d)}.json`);
|
||||
const snap=(ev,d)=>{
|
||||
const dir=path.join(d,'bus'),files={};
|
||||
try{for(const n of fs.readdirSync(dir).sort()){const p=path.join(dir,n),st=fs.statSync(p);if(st.isFile())files[n]=[st.size,st.mtimeMs,crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')]}}catch{}
|
||||
fs.appendFileSync(path.join(sd,'snap.jsonl'),JSON.stringify({ev,files})+'\n');
|
||||
};
|
||||
const pm=(unit,d,id,pid,off)=>{
|
||||
let mode='';try{mode=fs.readFileSync(faultFile(`pm-${id}`),'utf8').trim()}catch{}
|
||||
const b=inst(unit),now=Date.now()-off*1000,epoch=Math.floor(now/1000),at=new Date(now).toISOString(),run=`run-${pid}`;
|
||||
const line=(text)=>fs.appendFileSync(path.join(sd,'journal.jsonl'),JSON.stringify({epoch,unit,text})+'\n');
|
||||
const ok=()=>line(`launched pm-${b}: run ${run}, claude opus`);
|
||||
const lf=perData(d,'launches');let list=[];try{list=JSON.parse(fs.readFileSync(lf,'utf8'))}catch{}
|
||||
const entry=(o)=>list.push({run,business:b,live:true,startedAt:at,...o});
|
||||
line(`bus host up: business ${b}, socket /fixture/bus.sock, notifier off`);
|
||||
switch(mode){
|
||||
case 'none':break;
|
||||
case 'two':ok();line(`launched pm-${b}: run ${run}b, claude opus`);entry({});break;
|
||||
case 'badword':line(`launched pm-${b}: job ${run}, claude opus`);entry({});break;
|
||||
case 'notlisted':ok();break;
|
||||
case 'early':ok();entry({startedAt:new Date(now-3600e3).toISOString()});break;
|
||||
case 'human':entry({run:`human-${pid}`});break;
|
||||
case 'other':ok();entry({business:'other'});break;
|
||||
case 'dead':ok();entry({live:false});break;
|
||||
default:ok();entry({});
|
||||
}
|
||||
fs.writeFileSync(lf,JSON.stringify(list));
|
||||
};
|
||||
if(verb==='show'){
|
||||
const u=rest[0],props=[];for(let i=1;i<rest.length;i++)if(rest[i]==='-p')props.push(rest[++i]);
|
||||
const f=unitFile(u),st=load()[u]||{},id=f?path.basename(f.wd):'';
|
||||
for(const p of props){
|
||||
if(p==='LoadState')console.log(`LoadState=${f?'loaded':'not-found'}`);
|
||||
else if(p==='WorkingDirectory')console.log(`WorkingDirectory=${f?f.wd+(!isBus(u)&&fault(`check-${id}`)?'-moved':''):''}`);
|
||||
else if(p==='ExecStart')console.log(f&&f.exec?`ExecStart={ path=${f.exec.split(' ')[0]} ; argv[]=${f.exec} ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }`:'ExecStart=');
|
||||
else if(p==='Environment')console.log(`Environment=${f?f.env.join(' '):''}`);
|
||||
else if(p==='MainPID')console.log(`MainPID=${st.pid||0}`);
|
||||
else if(p==='DropInPaths')console.log(`DropInPaths=${f?f.drops.join(' '):''}`);
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
if(verb==='is-active'){
|
||||
const q=rest[0]==='--quiet',u=q?rest[1]:rest[0];let s=(load()[u]||{}).state||'inactive';
|
||||
// bus-stops-after N (T29): the next N bus checks answer the real state, every later one answers
|
||||
// inactive, and units.json stays as it is, as if the bus stopped between two checks.
|
||||
if(isBus(u)&&fault('bus-stops-after')){const n=Number(fs.readFileSync(faultFile('bus-stops-after'),'utf8'));if(n>0)fs.writeFileSync(faultFile('bus-stops-after'),String(n-1));else s='inactive'}
|
||||
if(!q)console.log(s);process.exit(s==='active'?0:3);
|
||||
}
|
||||
if(verb==='is-enabled'){console.log('disabled');process.exit(1)}
|
||||
if(verb==='daemon-reload')process.exit(0);
|
||||
if(verb==='list-units'){
|
||||
if(fault('list-fails'))process.exit(1);
|
||||
const pats=rest.filter((a)=>!a.startsWith('--')).map((p)=>new RegExp('^'+p.replace(/\./g,'\\.').replace(/\*/g,'.*')+'$'));
|
||||
for(const [u,s] of Object.entries(load()))if(pats.some((re)=>re.test(u)))console.log(`${u} loaded ${s.state} ${s.state==='active'?'running':'dead'} fixture`);
|
||||
process.exit(0);
|
||||
}
|
||||
if(verb==='start'){
|
||||
const u=rest[0],f=unitFile(u);if(!f){console.error(`Unit ${u} not found.`);process.exit(5)}
|
||||
const s=load(),id=path.basename(f.wd),d=dataOf(f);
|
||||
if(!isBus(u)){s[u]={state:'active',pid:null,wd:f.wd};save(s);process.exit(0)}
|
||||
if(fault(`start-dies-${id}`)){s[u]={state:'failed',pid:null,wd:f.wd};save(s);process.exit(0)}
|
||||
// Row 185 (T48): the copy's start writes one event before the holder opens the store, so the
|
||||
// store differs from a backup the run took before the start. The close leaves no -wal.
|
||||
if(fault(`start-writes-row-${id}`)){
|
||||
const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(d,'bus','bus.sqlite'));
|
||||
db.prepare('INSERT INTO events (at) VALUES (?)').run('2999-01-01T00:00:00.000Z');db.close();
|
||||
}
|
||||
snap('start',d);
|
||||
const off=process.env.SD_CLOCK_OFFSET||'0';
|
||||
// The holder must not inherit stack-release.sh's lock (fd 9) or any other fd above 2.
|
||||
const close='for fd in /proc/self/fd/*; do n=${fd##*/}; [ "$n" -gt 2 ] && eval "exec $n>&-" 2>/dev/null; done; exec "$@"';
|
||||
const ch=cp.spawn('bash',['-c',close,'holder',process.execPath,path.join(__dirname,'..','holder.cjs'),d,inst(u),sd,off,perData(d,'host')],{detached:true,stdio:'ignore'});
|
||||
ch.unref();
|
||||
fs.appendFileSync(path.join(sd,'pids.log'),ch.pid+'\n');
|
||||
const ready=path.join(sd,`ready-${ch.pid}`),end=Date.now()+10000;
|
||||
while(!fs.existsSync(ready)){if(Date.now()>end){console.error('fixture holder did not start');process.exit(1)}sleep(20)}
|
||||
s[u]={state:'active',pid:ch.pid,wd:f.wd};save(s);
|
||||
if(f.exec.split(' ').slice(1).includes('--pm'))pm(u,d,id,ch.pid,Number(off));
|
||||
process.exit(0);
|
||||
}
|
||||
if(verb==='stop'){
|
||||
const u=rest[0],s=load(),st=s[u];
|
||||
if(st&&st.pid){
|
||||
try{process.kill(st.pid,'SIGTERM')}catch{}
|
||||
const end=Date.now()+10000;while(alive(st.pid)&&Date.now()<end)sleep(20);
|
||||
if(alive(st.pid))try{process.kill(st.pid,'SIGKILL')}catch{}
|
||||
}
|
||||
if(st){s[u]={...st,state:'inactive',pid:null};save(s)}
|
||||
if(isBus(u)){
|
||||
const d=dataOf(unitFile(u));
|
||||
// One-shot: the stop leaves an empty -wal or -shm beside the store, as an unclean close would.
|
||||
for(const [name,ext] of [['stop-leaves-wal','-wal'],['stop-leaves-shm','-shm']])if(fault(name)){fs.rmSync(faultFile(name));fs.writeFileSync(path.join(d,'bus','bus.sqlite'+ext),'')}
|
||||
// One-shot (T45): the stop writes one row after any STAMP into the table the fault names, as a
|
||||
// write the bus made between rollback's count and its stop would. The close leaves no -wal.
|
||||
if(fault('stop-writes-row')){
|
||||
const t=fs.readFileSync(faultFile('stop-writes-row'),'utf8');fs.rmSync(faultFile('stop-writes-row'));
|
||||
const db=new (process.getBuiltinModule('node:sqlite').DatabaseSync)(path.join(d,'bus','bus.sqlite'));
|
||||
db.exec(`CREATE TABLE IF NOT EXISTS ${t} (seq INTEGER PRIMARY KEY, at TEXT NOT NULL)`);
|
||||
db.prepare(`INSERT INTO ${t} (at) VALUES (?)`).run('2999-01-01T00:00:00.000Z');db.close();
|
||||
}
|
||||
snap('stop',d);
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
console.error(`fixture systemctl: unsupported: ${args.join(' ')}`);process.exit(1);
|
||||
}
|
||||
|
||||
// journalctl --user -u UNIT --since @EPOCH -o cat. A bare local time gets exit 1, as M35 needs.
|
||||
function journalctl(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
|
||||
const sd=process.env.SD_STATE,a=process.argv.slice(2);let unit='',since='';
|
||||
for(let i=0;i<a.length;i++){if(a[i]==='-u')unit=a[++i];else if(a[i]==='--since')since=a[++i]}
|
||||
fs.appendFileSync(path.join(sd,'journal-since.log'),since+'\n');
|
||||
if(fs.existsSync(path.join(sd,'faults','journal-fails')))process.exit(1);
|
||||
let from;
|
||||
if(/^@\d+$/.test(since))from=Number(since.slice(1));
|
||||
else{const m=/^(\d{4}-\d\d-\d\d) (\d\d:\d\d:\d\d) UTC$/.exec(since);if(!m)process.exit(1);from=Date.parse(`${m[1]}T${m[2]}Z`)/1000}
|
||||
let lines=[];try{lines=fs.readFileSync(path.join(sd,'journal.jsonl'),'utf8').split('\n').filter(Boolean).map((l)=>JSON.parse(l))}catch{}
|
||||
for(const o of lines)if(o.unit===unit&&o.epoch>=from)console.log(o.text);
|
||||
}
|
||||
|
||||
function systemdAnalyze(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
|
||||
if(process.argv[2]!=='--user'||process.argv[3]!=='unit-paths')process.exit(1);
|
||||
console.log(path.join(process.env.XDG_CONFIG_HOME,'systemd','user'));
|
||||
// The unit-path fault adds its text as one more path, as T28's relative path needs.
|
||||
const extra=path.join(process.env.SD_STATE,'faults','unit-path');
|
||||
if(fs.existsSync(extra))console.log(fs.readFileSync(extra,'utf8'));
|
||||
}
|
||||
|
||||
// The bus host: holds the store open in WAL mode until SIGTERM, as the broker does. It writes its
|
||||
// status to the host file systemctl names for its data root.
|
||||
function holder(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
|
||||
const [data,business,sd,offset,host]=process.argv.slice(2),bus=path.join(data,'bus');
|
||||
fs.mkdirSync(bus,{recursive:true,mode:0o700});
|
||||
const db=new DatabaseSync(path.join(bus,'bus.sqlite'));
|
||||
if(!db.prepare("SELECT count(*) AS n FROM sqlite_master WHERE name='meta'").get().n){
|
||||
db.exec('PRAGMA journal_mode=WAL');
|
||||
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
|
||||
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
|
||||
db.prepare("INSERT INTO meta VALUES ('schema_version', '3g')").run();
|
||||
}
|
||||
fs.writeFileSync(path.join(bus,'writer.lock'),String(process.pid));
|
||||
const at=new Date(Date.now()-Number(offset||0)*1000).toISOString();
|
||||
fs.writeFileSync(host+'.tmp',JSON.stringify({host:{business,pid:process.pid,live:true,startedAt:at,notifier:'ok'},boot:{ok:true,business,at}}));
|
||||
fs.renameSync(host+'.tmp',host);
|
||||
process.on('SIGTERM',()=>{db.close();fs.rmSync(path.join(bus,'writer.lock'),{force:true});process.exit(0)});
|
||||
fs.writeFileSync(path.join(sd,`ready-${process.pid}`),'');
|
||||
setInterval(()=>{},1<<30);
|
||||
}
|
||||
|
||||
// A copy's scripts/mosaic: bus status and launches list from what the holder and systemctl wrote
|
||||
// for the data root of the config it runs under (row 190), the default's being $SD_DATA.
|
||||
function mosaicStub(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
|
||||
const id=path.basename(path.resolve(__dirname,'..')),sd=process.env.SD_STATE,a=process.argv.slice(2).join(' ');
|
||||
const fault=(n)=>fs.existsSync(path.join(sd,'faults',n));
|
||||
let d=process.env.SD_DATA;try{d=JSON.parse(fs.readFileSync(process.env.MOSAIC_CONFIG,'utf8')).dataRoot}catch{}
|
||||
const perData=(n)=>path.join(sd,d===process.env.SD_DATA?`${n}.json`:`${n}-${path.basename(d)}.json`);
|
||||
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
|
||||
if(a==='bus status --json'){
|
||||
if(fault(`status-${id}`))process.exit(1);
|
||||
let s={host:null,boot:null};
|
||||
try{s=JSON.parse(fs.readFileSync(perData('host'),'utf8'));s.host.live=alive(s.host.pid)}catch{}
|
||||
console.log(JSON.stringify(s));
|
||||
}else if(a==='bus status')console.log(`bus: fixture status for ${id}`);
|
||||
else if(a==='launches list --json'){
|
||||
if(fault(`launches-${id}`))process.exit(1);
|
||||
let l=[];try{l=JSON.parse(fs.readFileSync(perData('launches'),'utf8'))}catch{}
|
||||
console.log(JSON.stringify(l));
|
||||
}else{console.error(`fixture mosaic: ${a}`);process.exit(4)}
|
||||
}
|
||||
|
||||
// A copy's migrate-schema.mjs: records the backups it saw, then sets 3h, or 3h-partial and exit 1.
|
||||
function migrateStub(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
|
||||
const data=process.argv[2],sd=process.env.SD_STATE,dir=path.join(data,'backups','stack');
|
||||
const backups=fs.existsSync(dir)?fs.readdirSync(dir).sort().map((name)=>({name,sums:fs.existsSync(path.join(dir,name,'SHA256SUMS'))})):[];
|
||||
fs.writeFileSync(path.join(sd,'migrate.json'),JSON.stringify({backups}));
|
||||
const fail=fs.existsSync(path.join(sd,'faults','migrate-fails'));
|
||||
const db=new DatabaseSync(path.join(data,'bus','bus.sqlite'));
|
||||
db.prepare("UPDATE meta SET value=? WHERE key='schema_version'").run(fail?'3h-partial':'3h');db.close();
|
||||
if(fail)process.exit(1);
|
||||
console.log('fixture migrate: 3h');
|
||||
}
|
||||
|
||||
// --- The module fixture: one repo, its copies and the stand-ins, shared by every test. ---
|
||||
|
||||
const base=realpathSync(mkdtempSync(join(tmpdir(),'stack-release-')));
|
||||
after(()=>{execFileSync('chmod',['-R','u+w',base]);rmSync(base,{recursive:true,force:true})});
|
||||
const repo=join(base,'repo'),COPIES=join(base,'copies'),SBIN=join(base,'sbin');
|
||||
mkdirSync(repo);mkdirSync(COPIES);mkdirSync(SBIN);
|
||||
writeFileSync(join(base,'package.json'),'{"type":"commonjs"}\n');
|
||||
const tracked=execFileSync('git',['-C',ROOT,'ls-files','-co','--exclude-standard','--','scripts','packages','adapters','roles','RELEASE','package.json','package-lock.json',':!packages/*/tests',':!scripts/foundation'],{encoding:'utf8'}).trim().split('\n');
|
||||
for(const f of tracked)if(existsSync(join(ROOT,f)))cpSync(join(ROOT,f),join(repo,f));
|
||||
const pkg={name:'copy-fixture',version:'1.0.0'};
|
||||
writeFileSync(join(repo,'package.json'),JSON.stringify(pkg));
|
||||
writeFileSync(join(repo,'package-lock.json'),JSON.stringify({name:pkg.name,version:pkg.version,lockfileVersion:3,packages:{'':pkg}}));
|
||||
git(repo,'init','-q','-b','refactor');git(repo,'config','gc.auto','0');git(repo,'add','.');git(repo,'commit','-qm','fixture');
|
||||
const commit=(msg)=>{git(repo,'commit','-q','--allow-empty','-m',msg);return git(repo,'rev-parse','HEAD')};
|
||||
// refactor: F A B C1 C, then M merging side branch S (from A). U sits on a branch nothing reaches.
|
||||
const F=git(repo,'rev-parse','HEAD'),A=commit('row 901: A'),B=commit('row 902: B'),C1=commit('row 903: C1'),C=commit('row 904: C');
|
||||
git(repo,'checkout','-q','-b','side',A);const S=commit('row 905: S');
|
||||
git(repo,'checkout','-q','-b','unreached',A);const U=commit('row 906: U');
|
||||
git(repo,'checkout','-q','refactor');git(repo,'merge','-q','--no-ff','-m','merge side','side');
|
||||
const M=git(repo,'rev-parse','HEAD');git(repo,'update-ref','refs/remotes/origin/refactor',M);
|
||||
// `to` false leaves out the `to:` lines (C1); a string puts that value in each one (A). The --plan
|
||||
// steps line can't be read from either (T27).
|
||||
const storeText=(v,froms,to=true)=>`export const SCHEMA_VERSION = '${v}';\nexport const MIGRATIONS = [\n${froms.map((f,i)=>` {\n from: '${f}',\n${to===false?'':` to: '${to===true?froms[i+1]??v:to}',\n`} run() {},\n },\n`).join('')}];\n`;
|
||||
// Plug-in kinds (row 169): C1 and C know the harness kind, as a copy at or past c216f47db does. The
|
||||
// rest have it under another name, as ca15d73e9 has no harness kind. Every copy has the real business
|
||||
// loader and roles, which the tool imports to read the business file.
|
||||
const KINDS_NEW=readFileSync(join(repo,'packages','plugins','src','kinds.mjs'),'utf8');
|
||||
const KINDS_OLD=KINDS_NEW.replace('\n harness: Object.freeze({','\n "harness-before-c216f47db": Object.freeze({');
|
||||
assert.notEqual(KINDS_OLD,KINDS_NEW);
|
||||
// Row 200 (#1698): C1 and C take --sso and have the row 192 loader and checkConsole; C's checker
|
||||
// refuses RS256, a stand-in for a target whose rules differ from the repository's. B has the loader
|
||||
// from before row 192, which refuses a `console` key. The other copies have no loader.
|
||||
const LOADER=readFileSync(join(repo,'scripts','mosaic-config.mjs'),'utf8'),SSO_CONFIG=readFileSync(join(repo,'packages','webui','src','sso-config.mjs'),'utf8');
|
||||
const LOADER_OLD=LOADER.replace('["configVersion", "environment", "dataRoot", "execution", "console"]','["configVersion", "environment", "dataRoot", "execution"]');
|
||||
const SSO_CONFIG_C=SSO_CONFIG.replace("'ES256', 'RS256',","'ES256',");
|
||||
assert.notEqual(LOADER_OLD,LOADER);assert.notEqual(SSO_CONFIG_C,SSO_CONFIG);
|
||||
const ssoFiles=(id)=>id===B?[['scripts/mosaic-config.mjs',LOADER_OLD]]:[C1,C].includes(id)?[['scripts/mosaic-config.mjs',LOADER],['packages/webui/src/sso-config.mjs',id===C?SSO_CONFIG_C:SSO_CONFIG]]:[];
|
||||
for(const [id,store] of [[F,'// a copy from before SCHEMA_VERSION\n'],[A,storeText('3g',['3f'],'3x')],[B,storeText('3g',['3f'])],[C1,storeText('3g',['3f'],false)],[C,storeText('3h',['3f','3g'])],[S,storeText('3g',['3f'])],[U,storeText('3g',['3f'])]]){
|
||||
const d=join(COPIES,id);
|
||||
for(const [f,text,mode] of [['scripts/mosaic',script(mosaicStub),0o755],['packages/webui/src/cli.mjs',[B,C1,C].includes(id)?`// fixture Console, never run\n// serve accepts '--tester-port'\n// serve accepts '--proof-header' (row 178)\n${[C1,C].includes(id)?"// serve accepts '--sso' (row 192)\n":''}`:'// fixture Console before tester listener, never run\n',0o644],['packages/bus/src/store.mjs',store,0o644],['packages/bus/src/migrate-schema.mjs',`(${migrateStub})();\n`,0o644],['packages/plugins/src/kinds.mjs',[C1,C].includes(id)?KINDS_NEW:KINDS_OLD,0o644],...ssoFiles(id).map(([f,text])=>[f,text,0o644])]){
|
||||
mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text,{mode});
|
||||
}
|
||||
for(const p of ['packages/business/src','roles'])cpSync(join(repo,p),join(d,p),{recursive:true});
|
||||
execFileSync('chmod',['-R','a-w',d]);
|
||||
}
|
||||
for(const [name,fn] of [['systemctl',systemctl],['journalctl',journalctl],['systemd-analyze',systemdAnalyze]])writeFileSync(join(SBIN,name),script(fn),{mode:0o755});
|
||||
writeFileSync(join(base,'holder.cjs'),`(${holder})();\n`);
|
||||
const SR=join(repo,'scripts','stack-release.sh');
|
||||
|
||||
// --- Per-test roots and helpers ---
|
||||
|
||||
const sha=(f)=>createHash('sha256').update(readFileSync(f)).digest('hex');
|
||||
const lines=(f)=>existsSync(f)?readFileSync(f,'utf8').split('\n').filter(Boolean):[];
|
||||
const calls=(e)=>lines(join(e.sd,'calls.log'));
|
||||
const startsStops=(e,from)=>calls(e).slice(from).filter((c)=>/^(start|stop) /.test(c));
|
||||
const logLines=(e)=>lines(e.log).map((l)=>JSON.parse(l));
|
||||
const pointer=(e)=>existsSync(e.pointer)?JSON.parse(readFileSync(e.pointer,'utf8')):null;
|
||||
const units=(e)=>JSON.parse(readFileSync(join(e.sd,'units.json'),'utf8'));
|
||||
const snaps=(e)=>lines(join(e.sd,'snap.jsonl')).map((l)=>JSON.parse(l));
|
||||
const fault=(e,name,text='')=>writeFileSync(join(e.sd,'faults',name),text);
|
||||
const sysctl=(e,...a)=>execFileSync(join(SBIN,'systemctl'),['--user',...a],{env:e.env,encoding:'utf8'});
|
||||
const edit=(f,from,to)=>{const s=readFileSync(f,'utf8');assert.ok(s.includes(from),`${f} has no ${from}`);writeFileSync(f,s.replace(from,to))};
|
||||
const posts=(e)=>readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-'));
|
||||
const backupSum=(e,act)=>readFileSync(join(e.data,'backups','stack',act,'SHA256SUMS'),'utf8').split(' ')[0];
|
||||
const commitLines=(range)=>git(repo,'log','--oneline','--no-merges','--max-count=200',range).split('\n').map((l)=>` ${l}`);
|
||||
// The units, the wrapper, the pointer and the store, by sha256.
|
||||
const fp=(e)=>({bus:sha(join(e.units,e.tmpl)),console:sha(join(e.units,e.con)),wrapper:sha(join(e.bin,e.wrapper)),pointer:existsSync(e.pointer)?sha(e.pointer):null,store:sha(e.store)});
|
||||
// The schema version: readOnly while the holder has the store open, immutable when it is stopped.
|
||||
function storeVersion(file,held){
|
||||
const db=held?new DatabaseSync(file,{readOnly:true}):new DatabaseSync(pathToFileURL(file).href+'?immutable=1',{readOnly:true});
|
||||
try{return db.prepare("SELECT value FROM meta WHERE key='schema_version'").get().value}finally{db.close()}
|
||||
}
|
||||
// Every file and directory under p with size and mtime: T23 and T16 compare it before and after.
|
||||
function tree(p){
|
||||
const out=[];
|
||||
const walk=(q)=>{let st;try{st=statSync(q)}catch{return}out.push(st.isDirectory()?`${q}/`:`${q} ${st.size} ${st.mtimeMs}`);if(st.isDirectory())for(const n of readdirSync(q).sort())walk(join(q,n))};
|
||||
walk(p);return out;
|
||||
}
|
||||
// T17: what a run may never remove.
|
||||
const kept=(e)=>[...tree(join(e.data,'backups','stack')),...tree(join(e.state,'stack-rollback')),...posts(e).flatMap((n)=>tree(join(e.data,n))),...tree(COPIES)].map((l)=>l.replace(/ .*/,''));
|
||||
|
||||
// The business file (row 169): the committed example made loadable, as acme, with `plugins` when given.
|
||||
const EXAMPLE=readFileSync(join(repo,'packages','business','examples','mosaic-stack.example.json'),'utf8');
|
||||
function biz(e,plugins){
|
||||
const b=JSON.parse(EXAMPLE);let n=0;
|
||||
const walk=(o)=>{for(const [k,v] of Object.entries(o)){if(k==='botId')o[k]=++n;else if(v==='YYYY-MM-DD')o[k]='2027-01-01';else if(v&&typeof v==='object')walk(v)}};
|
||||
walk(b);b.id='acme';if(plugins!==undefined)b.plugins=plugins;
|
||||
const f=join(dirname(e.cfg),'businesses','acme.json');mkdirSync(dirname(f),{recursive:true});
|
||||
writeFileSync(f,JSON.stringify(b,null,2)+'\n',{mode:0o600});chmodSync(f,0o600);
|
||||
}
|
||||
|
||||
async function mk(t){
|
||||
const root=realpathSync(mkdtempSync(join(tmpdir(),'sr-')));
|
||||
const e={root,data:join(root,'data'),bin:join(root,'bin'),sd:join(root,'sd'),units:join(root,'xcfg','systemd','user'),cfg:join(root,'cfg','config.json'),claude:join(root,'claude-bin','claude')};
|
||||
// The default instance's names; named() sets NAME's (row 190).
|
||||
Object.assign(e,{flags:[],bus:BUS,tmpl:'[email protected]',con:CON,wrapper:'mosaic'});
|
||||
Object.assign(e,{state:join(e.data,'state'),pointer:join(e.data,'state','stack-active.json'),log:join(e.data,'state','stack-activation-log.jsonl'),store:join(e.data,'bus','bus.sqlite'),lock:join(e.data,'state','stack-release.lock')});
|
||||
for(const d of [join(root,'home'),dirname(e.cfg),join(e.data,'bus'),e.bin,dirname(e.claude),join(e.sd,'faults'),e.units,join(root,'xdata')])mkdirSync(d,{recursive:true});
|
||||
chmodSync(join(e.data,'bus'),0o700);
|
||||
writeFileSync(e.cfg,JSON.stringify({configVersion:1,environment:'development',dataRoot:e.data,execution:{backend:'docker',provider:'zai',model:'glm-5.3-flash',adapter:'mock'}}));
|
||||
biz(e);
|
||||
writeFileSync(join(e.bin,'mosaic'),`#!/usr/bin/env bash\n${MARKER}\n[ "\${1:-}" = __lease-capability ] && exec "${e.bin}/mosaic-old" "$@"\nexec "/old/scripts/mosaic" "$@"\n`,{mode:0o755});
|
||||
writeFileSync(join(e.bin,'mosaic-old'),'#!/bin/sh\nexit 0\n',{mode:0o755});
|
||||
writeFileSync(e.claude,'#!/bin/sh\necho "9.9.9 (Claude Code fixture)"\n',{mode:0o755});
|
||||
e.env={PATH:`${SBIN}:${dirname(NODE)}:/usr/bin:/bin`,HOME:join(root,'home'),XDG_CONFIG_HOME:join(root,'xcfg'),XDG_DATA_HOME:join(root,'xdata'),MOSAIC_CONFIG:e.cfg,SD_STATE:e.sd,SD_DATA:e.data,TMPDIR:tmpdir()};
|
||||
const server=await healthz(e);
|
||||
t.after(()=>{
|
||||
for(const pid of lines(join(e.sd,'pids.log')).map(Number)){
|
||||
try{if(readFileSync(`/proc/${pid}/cmdline`,'utf8').includes(join(base,'holder.cjs')))process.kill(pid,'SIGKILL')}catch{}
|
||||
}
|
||||
for(const s of [server,...(e.servers??[])])s.close();
|
||||
execFileSync('chmod',['-R','u+w',root]);rmSync(root,{recursive:true,force:true});
|
||||
});
|
||||
return e;
|
||||
}
|
||||
|
||||
// The Console's /healthz on e.port: 200 while the stand-in has e.con active and no healthz-<copy>
|
||||
// fault names its copy.
|
||||
async function healthz(e){
|
||||
const server=createServer((q,r)=>{
|
||||
let st={};try{st=units(e)[e.con]||{}}catch{}
|
||||
const ok=q.url==='/healthz'&&st.state==='active'&&!existsSync(join(e.sd,'faults',`healthz-${basename(st.wd||'')}`));
|
||||
r.writeHead(ok?200:500);r.end();
|
||||
});
|
||||
server.listen(0,'127.0.0.1');await once(server,'listening');e.port=String(server.address().port);
|
||||
return server;
|
||||
}
|
||||
|
||||
// Row 190: stack instance NAME on e's host. It shares e's stand-ins, unit directory and $BIN, and has
|
||||
// its own config and data root from bootstrap --instance, its own Console port and business acme.
|
||||
// Its runs pass --instance NAME and no MOSAIC_CONFIG.
|
||||
async function named(e,name){
|
||||
const env={...e.env};delete env.MOSAIC_CONFIG;
|
||||
execFileSync('bash',[join(repo,'scripts','bootstrap.sh'),'--instance',name],{env,stdio:'ignore'});
|
||||
const data=join(env.HOME,`.mosaic-${name}`);
|
||||
const n={...e,env,data,cfg:join(env.HOME,'.config',`mosaic-${name}`,'config.json'),flags:['--instance',name],bus:`mosaic-bus-${name}@acme.service`,tmpl:`mosaic-bus-${name}@.service`,con:`mosaic-webui-${name}.service`,wrapper:`mosaic-${name}`,name};
|
||||
Object.assign(n,{state:join(data,'state'),pointer:join(data,'state','stack-active.json'),log:join(data,'state','stack-activation-log.jsonl'),store:join(data,'bus','bus.sqlite'),lock:join(data,'state','stack-release.lock')});
|
||||
mkdirSync(join(data,'bus'),{mode:0o700});
|
||||
biz(n);
|
||||
(e.servers??=[]).push(await healthz(n));
|
||||
return n;
|
||||
}
|
||||
|
||||
async function run(e,...args){
|
||||
const before=existsSync(e.data)?kept(e):[];
|
||||
const ch=spawn('bash',[e.sr??SR,...args,...e.flags,'--copies',COPIES,'--bin',e.bin],{env:e.env,stdio:['ignore','pipe','pipe']});
|
||||
let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d});
|
||||
const timer=setTimeout(()=>ch.kill('SIGKILL'),120000);
|
||||
const [code]=await once(ch,'close');clearTimeout(timer);
|
||||
const now=new Set(kept(e));
|
||||
for(const p of before)assert.ok(now.has(p),`T17: ${args.join(' ')} removed ${p}`);
|
||||
return {code,out,err};
|
||||
}
|
||||
|
||||
// The stack on copy `copy` for business acme, both units active. pointer: adopt (a real adopt and
|
||||
// its log line), hand (written here, no log) or none. Snapshots start after the seed.
|
||||
async function seed(e,{copy=B,pm=false,store='3g',pointer:how='adopt'}={}){
|
||||
const db=new DatabaseSync(e.store);
|
||||
db.exec('PRAGMA journal_mode=WAL');
|
||||
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
|
||||
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
|
||||
db.prepare("INSERT INTO meta VALUES ('schema_version', ?)").run(store);
|
||||
for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`);
|
||||
db.close();
|
||||
const sh=(name,...a)=>execFileSync('bash',[join(repo,'scripts',name),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
|
||||
sh('bus-service.sh','install',...e.flags,'--copy',copy,'--copies',COPIES,...(pm?['--pm']:[]),'--claude',e.claude);
|
||||
sh('webui-service.sh','install',...e.flags,'--copy',copy,'--copies',COPIES,'--port',e.port,'--business','acme');
|
||||
sh('stack-copy.sh','wrapper',copy,...e.flags,'--copies',COPIES,'--bin',e.bin);
|
||||
// The seeded host started two minutes ago, so its boot status and PM run predate any run's start.
|
||||
execFileSync(join(SBIN,'systemctl'),['--user','start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:'120'}});
|
||||
sysctl(e,'start',e.con);
|
||||
if(how==='adopt'){const r=await run(e,'adopt','acme');assert.equal(r.code,0,r.err)}
|
||||
else if(how==='hand'){
|
||||
mkdirSync(e.state,{recursive:true});
|
||||
writeFileSync(e.pointer,JSON.stringify({copy,business:'acme',pm,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g',activation:'manual',at:'2026-10-18T00:00:00Z'})+'\n');
|
||||
}
|
||||
writeFileSync(join(e.sd,'snap.jsonl'),'');
|
||||
}
|
||||
|
||||
const rolledBack=(code,to=B)=>`stack-release: ${code}; rolled back to ${to}, which passed the gate\n`;
|
||||
const planBackup=(e)=>`backup: the run backs up the stopped store to ${e.data}/backups/stack/<activation>/ before any migration`;
|
||||
const planNoBackup='backup: none; the run rolls back before the backup';
|
||||
const planTail=(e,to,pm=false)=>[`bus: scripts/bus-service.sh install --copy ${to} --copies ${COPIES}${pm?' --pm':''} --claude ${e.claude}`,'proof: off','send: off',`Console: scripts/webui-service.sh install --copy ${to} --copies ${COPIES} --port ${e.port} --business acme`,`CLI: scripts/stack-copy.sh wrapper ${to} --copies ${COPIES} --bin ${e.bin}`];
|
||||
import {A,B,BUS,C,C1,CON,COPIES,EXAMPLE,F,KINDS_NEW,KINDS_OLD,M,MARKER,NODE,ROOT,S,SBIN,SR,backupSum,base,biz,calls,commit,commitLines,edit,fault,fp,git,healthz,holder,journalctl,kept,lines,logLines,migrateStub,mk,mosaicStub,named,pkg,planBackup,planNoBackup,planTail,pointer,posts,repo,rolledBack,run,script,seed,sha,snaps,startsStops,storeText,storeVersion,sysctl,systemctl,systemdAnalyze,tracked,tree,U,units} from './stack-release-fixture.mjs';
|
||||
|
||||
// --- The T table ---
|
||||
|
||||
|
||||
+62
-13
@@ -3,21 +3,25 @@
|
||||
# copy creates files only; wrapper is a separately Mos-gated live step.
|
||||
# wrapper and check take --instance NAME (row 190): the units mosaic-bus-NAME@ and mosaic-webui-NAME
|
||||
# and the wrapper $BIN/mosaic-NAME, which sets NAME's MOSAIC_CONFIG. check needs --business for NAME.
|
||||
# Row 232: check --units external reads root's system units. copy --bundle makes the copy from a git
|
||||
# bundle on a host with no checkout, and wrapper --create writes the first wrapper on a new host.
|
||||
set -euo pipefail
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
COPIES="${XDG_DATA_HOME:-$HOME/.local/share}/mosaic-stack/copies"
|
||||
BIN="$HOME/.npm-global/bin"
|
||||
BUSINESS=mosaic-stack BUSINESS_SET=0
|
||||
ALLOW_WORKTREE=0
|
||||
ALLOW_WORKTREE=0 UNITS=user BUNDLE="" CREATE=0 CLEAN=""
|
||||
MARKER='# mosaic-cli wrapper (Q81, agents/sage/work/q81/mosaic-cli.sh)'
|
||||
usage() { echo 'usage: stack-copy.sh copy COMMIT [--copies DIR] [--allow-worktree] | wrapper ID [--instance NAME] [--copies DIR] [--bin DIR] | check [--instance NAME] [--copies DIR] [--bin DIR] [--business ID]' >&2; exit "${1:-4}"; }
|
||||
usage() { echo 'usage: stack-copy.sh copy COMMIT [--copies DIR] [--allow-worktree] | copy --bundle FILE SHA [--copies DIR] | wrapper ID [--instance NAME] [--copies DIR] [--bin DIR] [--create] | check [--instance NAME] [--copies DIR] [--bin DIR] [--business ID] [--units user|external]' >&2; exit "${1:-4}"; }
|
||||
die() { echo "stack-copy: $*" >&2; exit 1; }
|
||||
plain_path() { case "$1" in /*) ;; *) die "$2 must be an absolute path: $1" ;; esac; case "$1" in *[!A-Za-z0-9/._+-]*) die "$2 contains a character the unit cannot carry: $1" ;; esac; }
|
||||
parse() { while [ $# -gt 0 ]; do case "$1" in
|
||||
--copies) [ $# -ge 2 ] || usage; COPIES="$2"; shift 2;;
|
||||
--bin) [ "$CMD" != copy ] && [ $# -ge 2 ] || usage; BIN="$2";shift 2;;
|
||||
--business) [ "$CMD" = check ] && [ $# -ge 2 ] || usage; BUSINESS="$2";BUSINESS_SET=1;shift 2;;
|
||||
--allow-worktree) [ "$CMD" = copy ] || usage; ALLOW_WORKTREE=1;shift;;
|
||||
--allow-worktree) [ "$CMD" = copy ] && [ -z "$BUNDLE" ] || usage; ALLOW_WORKTREE=1;shift;;
|
||||
--units) [ "$CMD" = check ] && [ $# -ge 2 ] || usage; case "$2" in user|external) UNITS="$2";; *) usage;; esac; shift 2;;
|
||||
--create) [ "$CMD" = wrapper ] || usage; CREATE=1;shift;;
|
||||
*) usage;; esac; done; plain_path "$COPIES" 'copies directory'; }
|
||||
copy_root() { [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || die '--copy must be a full 40-character commit id'; [ -f "$COPIES/$ID/scripts/mosaic" ] || die "no copy $ID in $COPIES"; echo "$COPIES/$ID"; }
|
||||
# Closure of scripts/mosaic, bus process/notifier/launcher, Console serve,
|
||||
@@ -44,16 +48,22 @@ reachable() {
|
||||
copy_commit() {
|
||||
[ -d "$REPO/.git" ] || [ "$ALLOW_WORKTREE" = 1 ] || die "refusing: $REPO is not the canonical checkout (its .git is not a directory); run copy from the canonical checkout"
|
||||
[[ "$ID" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] || die "commit must be a commit id or branch name: $ID"
|
||||
local full target tmp
|
||||
local full
|
||||
full="$(git -C "$REPO" rev-parse --verify --quiet "$ID^{commit}")" || die "not a commit: $ID"
|
||||
reachable "$full"
|
||||
copy_tree "$REPO" "$full"
|
||||
}
|
||||
# The copy of commit $2 from repository $1. CLEAN names a bundle's temporary repository, which the
|
||||
# trap removes with the temporary copy.
|
||||
copy_tree() {
|
||||
local full="$2" target tmp
|
||||
target="$COPIES/$full"
|
||||
[ ! -L "$target" ] || die "refusing: copy target is a symlink: $target"
|
||||
if [ -e "$target" ]; then echo "exists: $target"; return; fi
|
||||
mkdir -p "$COPIES"
|
||||
tmp="$(mktemp -d "$COPIES/.$full.XXXXXX")"
|
||||
trap "chmod -R u+w '$tmp' 2>/dev/null || true; rm -rf '$tmp'" EXIT
|
||||
git -C "$REPO" archive --format=tar "$full" -- "${COPY_PATHS[@]}" | tar -xf - -C "$tmp" || die "git archive failed; no copy written"
|
||||
trap "chmod -R u+w '$tmp' 2>/dev/null || true; rm -rf '$tmp'$CLEAN" EXIT
|
||||
git -C "$1" archive --format=tar "$full" -- "${COPY_PATHS[@]}" | tar -xf - -C "$tmp" || die "git archive failed; no copy written"
|
||||
[ -f "$tmp/package.json" ] && [ -f "$tmp/package-lock.json" ] || die 'package.json or package-lock.json missing in temp copy; npm ci not run'
|
||||
(cd "$tmp" && npm ci --no-audit --no-fund) >&2 || die 'npm ci failed; no copy written'
|
||||
find "$tmp" -type f -exec chmod a-w {} +
|
||||
@@ -65,19 +75,53 @@ copy_commit() {
|
||||
trap - EXIT
|
||||
echo "copied: $target"
|
||||
}
|
||||
# Row 232: the copy of SHA from a git bundle, for a host with no refactor checkout. The bundle must
|
||||
# verify, carry refs/heads/refactor, and reach SHA from it, as reachable asks of a checkout. Prints
|
||||
# the RELEASE value at SHA and the bundle's sha256, the release identity on such a host.
|
||||
copy_bundle() {
|
||||
[[ "$ID" =~ ^[0-9a-f]{40}$ ]] || die "--bundle needs a full 40-character commit id: $ID"
|
||||
[ -f "$BUNDLE" ] && [ ! -L "$BUNDLE" ] || die "refusing: bundle is absent, not regular, or a symlink: $BUNDLE"
|
||||
local g sum release heads
|
||||
case "$BUNDLE" in /*) ;; *) BUNDLE="$PWD/$BUNDLE";; esac
|
||||
g="$(mktemp -d)"
|
||||
CLEAN=" '$g'"
|
||||
trap "rm -rf '$g'" EXIT
|
||||
git init -q --bare "$g" || die 'git init failed; no copy written'
|
||||
git -C "$g" bundle verify -q "$BUNDLE" >/dev/null 2>&1 || die "refusing: git bundle verify failed: $BUNDLE"
|
||||
heads="$(git -C "$g" bundle list-heads "$BUNDLE" refs/heads/refactor)" || die "refusing: cannot list the bundle's heads: $BUNDLE"
|
||||
grep -qx '[0-9a-f]\{40\} refs/heads/refactor' <<< "$heads" || die "refusing: the bundle has no refs/heads/refactor: $BUNDLE"
|
||||
# Every ref, so a SHA the bundle carries on another branch reaches the reachability check.
|
||||
git -C "$g" fetch -q "$BUNDLE" 'refs/*:refs/*' 2>/dev/null || die "refusing: cannot fetch from $BUNDLE"
|
||||
git -C "$g" cat-file -e "$ID^{commit}" 2>/dev/null || die "refusing: $ID is not in the bundle"
|
||||
git -C "$g" merge-base --is-ancestor "$ID" refs/heads/refactor || die "refusing: $ID is not reachable from the bundle's refs/heads/refactor"
|
||||
release="$(git -C "$g" show "$ID:RELEASE")" || die "no RELEASE at $ID"
|
||||
sum="$(sha256sum < "$BUNDLE" | cut -d' ' -f1)"
|
||||
copy_tree "$g" "$ID"
|
||||
rm -rf "$g"; trap - EXIT
|
||||
echo "release: $release"
|
||||
echo "bundle sha256: $sum"
|
||||
}
|
||||
# Row 232: --create writes the wrapper only where none is. ln refuses any existing path, a symlink
|
||||
# included, so this is the one check, and a wrapper that appears mid-run is never replaced.
|
||||
place() {
|
||||
if [ "$CREATE" = 1 ]; then ln "$1" "$2" 2>/dev/null || die "refusing: --create and a wrapper exists: $2"; rm -f "$1"
|
||||
else mv -f "$1" "$2"; fi
|
||||
}
|
||||
wrapper() {
|
||||
[ -z "$MOSAIC_UNIT_SUFFIX" ] || { wrapper_instance; return; }
|
||||
plain_path "$BIN" 'bin directory'
|
||||
local root file lease tmp
|
||||
root="$(copy_root)";file="$BIN/mosaic"
|
||||
[ -f "$file" ] && [ ! -L "$file" ] || die "refusing: wrapper is absent, not regular, or a symlink: $file"
|
||||
[ "$(sed -n '2p' "$file")" = "$MARKER" ] || die "refusing: foreign wrapper (missing Q81 marker on line 2): $file"
|
||||
lease='[ "${1:-}" = __lease-capability ] && exec "'"$BIN"'/mosaic-old" "$@"'
|
||||
[ "$(grep -Fxc "$lease" "$file" || true)" = 1 ] || die 'refusing: Q81 lease line missing or ambiguous'
|
||||
if [ "$CREATE" = 0 ]; then
|
||||
[ -f "$file" ] && [ ! -L "$file" ] || die "refusing: wrapper is absent, not regular, or a symlink: $file"
|
||||
[ "$(sed -n '2p' "$file")" = "$MARKER" ] || die "refusing: foreign wrapper (missing Q81 marker on line 2): $file"
|
||||
[ "$(grep -Fxc "$lease" "$file" || true)" = 1 ] || die 'refusing: Q81 lease line missing or ambiguous'
|
||||
fi
|
||||
tmp="$(mktemp "$BIN/.mosaic.XXXXXX")"
|
||||
trap 'rm -f "$tmp"' EXIT
|
||||
printf '#!/usr/bin/env bash\n%s\n# CLI runs from pinned copy: %s\n# The Q81 lease probe still answers at mosaic-old.\n%s\nexec "%s/scripts/mosaic" "$@"\n' "$MARKER" "$root" "$lease" "$root" > "$tmp"
|
||||
chmod 0755 "$tmp";mv -f "$tmp" "$file";trap - EXIT
|
||||
chmod 0755 "$tmp";place "$tmp" "$file";trap - EXIT
|
||||
echo "wrapper: $root"
|
||||
}
|
||||
# Row 190: $BIN/mosaic-NAME. It creates the file or replaces one with its own marker, never another.
|
||||
@@ -93,12 +137,14 @@ wrapper_instance() {
|
||||
tmp="$(mktemp "$BIN/.$WRAPPER_NAME.XXXXXX")"
|
||||
trap 'rm -f "$tmp"' EXIT
|
||||
printf '#!/usr/bin/env bash\n%s\n# CLI runs from pinned copy: %s\nexport MOSAIC_CONFIG='"'"'%s'"'"'\nexec "%s/scripts/mosaic" "$@"\n' "$INSTANCE_MARKER" "$root" "$MOSAIC_INSTANCE_CONFIG" "$root" > "$tmp"
|
||||
chmod 0755 "$tmp";mv -f "$tmp" "$file";trap - EXIT
|
||||
chmod 0755 "$tmp";place "$tmp" "$file";trap - EXIT
|
||||
echo "wrapper: $root"
|
||||
}
|
||||
# --units external reads root's system units, which the stack user can read and not change.
|
||||
units_show() { if [ "$UNITS" = external ]; then systemctl show "$@"; else systemctl --user show "$@"; fi; }
|
||||
unit_root() {
|
||||
local out load root exec_start
|
||||
out="$(systemctl --user show "$1" -p LoadState -p WorkingDirectory -p ExecStart 2>/dev/null)" || { echo 'absent (systemctl read failed)';return; }
|
||||
out="$(units_show "$1" -p LoadState -p WorkingDirectory -p ExecStart 2>/dev/null)" || { echo 'absent (systemctl read failed)';return; }
|
||||
load="$(sed -n 's/^LoadState=//p' <<< "$out")"; root="$(sed -n 's/^WorkingDirectory=//p' <<< "$out")"
|
||||
exec_start="$(sed -n 's/^ExecStart=//p' <<< "$out")"
|
||||
if [ "$load" != loaded ] || [ -z "$root" ];then echo absent;return;fi
|
||||
@@ -133,7 +179,10 @@ CMD="$1";shift
|
||||
case "$CMD" in check|wrapper) instance_select;; *) [ -z "$INSTANCE_FLAG" ] || usage;; esac
|
||||
INSTANCE_MARKER="# mosaic-cli wrapper, instance ${MOSAIC_INSTANCE:-dev} (row 190)"
|
||||
case "$CMD" in
|
||||
copy|wrapper) [ $# -gt 0 ] || usage;ID="$1";shift;case "$ID" in -*)usage;;esac;parse "$@";if [ "$CMD" = copy ];then copy_commit;else wrapper;fi;;
|
||||
copy|wrapper) [ $# -gt 0 ] || usage
|
||||
if [ "$CMD" = copy ] && [ "$1" = --bundle ];then [ $# -ge 3 ] && [ -n "$2" ] || usage;BUNDLE="$2";shift 2;fi
|
||||
ID="$1";shift;case "$ID" in -*)usage;;esac;parse "$@"
|
||||
if [ -n "$BUNDLE" ];then copy_bundle;elif [ "$CMD" = copy ];then copy_commit;else wrapper;fi;;
|
||||
check) parse "$@";[ -z "$MOSAIC_UNIT_SUFFIX" ] || [ "$BUSINESS_SET" = 1 ] || usage;check;;
|
||||
-h|--help) usage 0;;
|
||||
*) usage;;esac
|
||||
|
||||
+460
-58
@@ -19,6 +19,13 @@
|
||||
# rehearse (D2, row 163) stops and starts no unit and writes only its own rehearsal directory, but
|
||||
# it reads the live config and a backup of the live store, so Jason or Mos runs it.
|
||||
#
|
||||
# --units external (row 232, #1734; r6 section 4 at agents/darkwing/work/queue-195/DESIGN.md) is
|
||||
# for a host whose units are root's system units. This script then reads those units and never
|
||||
# starts or stops one: root stops them, activate or rollback stages the copy (A3), root installs
|
||||
# and starts them, and confirm gates them and writes the pointer (A5). revert and confirm
|
||||
# --rollback put the old copy back when a step fails (R2, R4), and gate checks a copy root
|
||||
# restarted with no activation open (A2f). Without it, everything runs as before.
|
||||
#
|
||||
# Most functions below run as `f || handler`, and bash ignores set -e inside such a call, so
|
||||
# every step that can fail carries its own `|| return 1`.
|
||||
set -euo pipefail
|
||||
@@ -32,6 +39,7 @@ BIN="$HOME/.npm-global/bin"
|
||||
UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
|
||||
TIMEOUT=60 WANT_PM=0 AFTER_HARNESS=0 MIGRATE=0 PLAN=0 RESTORE="" EVENTS="" TESTER_PORT=""
|
||||
PROOF_HEADER="" PROOF_FILE="" NO_PROOF=0 NO_SEND=0 SSO=0 R_BACKUP="" R_PORT=""
|
||||
UNITS=user TO_FLAG="" EXPECT_FROM="" SINCE="" ROLLBACK_OLD="" CAUSE=""
|
||||
|
||||
usage() {
|
||||
cat >&2 <<'USAGE'
|
||||
@@ -42,11 +50,18 @@ usage: stack-release.sh adopt BUSINESS
|
||||
stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan]
|
||||
stack-release.sh status BUSINESS
|
||||
stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]
|
||||
common options: --copies DIR, --bin DIR, --instance NAME
|
||||
stack-release.sh confirm ID BUSINESS --units external [--rollback OLD --cause A3|A4|A5] [--timeout S]
|
||||
stack-release.sh revert ID BUSINESS --units external
|
||||
stack-release.sh gate ID BUSINESS --units external --since EPOCH [--timeout S]
|
||||
common options: --copies DIR, --bin DIR, --instance NAME, --units user|external
|
||||
--instance NAME (row 190) runs on stack instance NAME: its config, data root, units and the wrapper
|
||||
mosaic-NAME. Without it, the default instance, as before.
|
||||
--events N is the total of the per-table counts the post-backup-events refusal prints: events and
|
||||
every table in COUNTED.
|
||||
--units external (row 232) reads root's system units and starts or stops none. adopt, activate,
|
||||
rollback, status, confirm, revert and gate take it; rollback then needs --to ID, the previous copy.
|
||||
activate and rollback --plan take --expect-from OLD. --pm, --pm-after-harness, --tester-port, the
|
||||
proof flags, --no-proof, --no-send, --sso and pm-on are not for it.
|
||||
USAGE
|
||||
exit "${1:-4}"
|
||||
}
|
||||
@@ -58,7 +73,27 @@ now_ms() { date -u +%Y-%m-%dT%H:%M:%S.%3NZ; }
|
||||
jline() {
|
||||
node -e 'const a=process.argv.slice(1),o={};for(let i=0;i<a.length;i+=2)o[a[i]]=a[i+1]==="@null"?null:a[i+1];process.stdout.write(JSON.stringify(o)+"\n")' -- "$@"
|
||||
}
|
||||
log_line() { jline at "$(now)" op "$OP" from "${FROM:-@null}" to "${TO:-@null}" backup "${BACKUP:-@null}" activation "${ACT:-@null}" "$@" >> "$LOG"; }
|
||||
log_line() {
|
||||
[ "$UNITS" = user ] || set -- "$@" units external
|
||||
jline at "$(now)" op "$OP" from "${FROM:-@null}" to "${TO:-@null}" backup "${BACKUP:-@null}" activation "${ACT:-@null}" "$@" >> "$LOG"
|
||||
}
|
||||
|
||||
# --- The units backend (row 232) ---
|
||||
|
||||
# Every systemctl, journalctl and systemd-analyze call goes through these three. user is the
|
||||
# systemd user manager, as before. external reads root's system units, which needs no privilege,
|
||||
# and refuses any verb that would change one.
|
||||
sc() {
|
||||
if [ "$UNITS" = user ]; then systemctl --user "$@"; return; fi
|
||||
case "$1" in
|
||||
show|is-active|list-units) systemctl "$@" ;;
|
||||
*) echo "stack-release: --units external never runs systemctl $1; root does" >&2; return 1 ;;
|
||||
esac
|
||||
}
|
||||
# The bus unit's journal is a user journal only: under external it lands in the system journal,
|
||||
# which the stack's user can't read (r6 section 4), so the external PM check reads launches list.
|
||||
jctl() { [ "$UNITS" = user ] || return 1; journalctl --user "$@"; }
|
||||
analyze() { [ "$UNITS" = user ] || return 1; systemd-analyze --user "$@"; }
|
||||
|
||||
# A refusal before anything stopped: exit 3, and one log line with the code, except for
|
||||
# run-locked (the lock holder may be writing the log) and in --plan, which writes nothing.
|
||||
@@ -83,7 +118,7 @@ parse_opts() {
|
||||
case "$1" in
|
||||
--copies) [ $# -ge 2 ] || usage; COPIES="$2"; shift 2 ;;
|
||||
--bin) [ $# -ge 2 ] || usage; BIN="$2"; shift 2 ;;
|
||||
--timeout) [ "$OP" != status ] && [ "$OP" != adopt ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
|
||||
--timeout) [ "$OP" != status ] && [ "$OP" != adopt ] && [ "$OP" != revert ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
|
||||
--plan) [ "$OP" = activate ] || [ "$OP" = rollback ] || usage; PLAN=1; shift ;;
|
||||
--pm) [ "$OP" = activate ] || usage; WANT_PM=1; shift ;;
|
||||
--pm-after-harness) [ "$OP" = activate ] || usage; AFTER_HARNESS=1; shift ;;
|
||||
@@ -98,9 +133,27 @@ parse_opts() {
|
||||
--backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
|
||||
--port) [ "$OP" = rehearse ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{1,5}$ ]] && [ "$((10#$2))" -ge 1024 ] && [ "$((10#$2))" -le 65535 ] || usage; R_PORT="$((10#$2))"; shift 2 ;;
|
||||
--events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
|
||||
--units) [ "$OP" != rehearse ] && [ $# -ge 2 ] && { [ "$2" = user ] || [ "$2" = external ]; } || usage; UNITS="$2"; shift 2 ;;
|
||||
--to) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; TO_FLAG="$2"; shift 2 ;;
|
||||
--expect-from) { [ "$OP" = activate ] || [ "$OP" = rollback ]; } && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; EXPECT_FROM="$2"; shift 2 ;;
|
||||
--since) [ "$OP" = gate ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]{0,11})$ ]] || usage; SINCE="$2"; shift 2 ;;
|
||||
--rollback) [ "$OP" = confirm ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; ROLLBACK_OLD="$2"; shift 2 ;;
|
||||
--cause) [ "$OP" = confirm ] && [ $# -ge 2 ] && [[ "$2" =~ ^A[345]$ ]] || usage; CAUSE="$2"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
# Row 232: confirm, revert, gate, --to and --expect-from are external only, and an external
|
||||
# rollback names its target. gate needs --since, its start floor (r6 N2), and --expect-from
|
||||
# goes with --plan. The flags that change a unit's render or the PM are root's under external.
|
||||
if [ "$UNITS" = user ]; then
|
||||
[ "$OP" != confirm ] && [ "$OP" != revert ] && [ "$OP" != gate ] && [ -z "$TO_FLAG" ] && [ -z "$EXPECT_FROM" ] || usage
|
||||
else
|
||||
[ "$OP" != rollback ] || [ -n "$TO_FLAG" ] || usage
|
||||
[ "$WANT_PM" = 0 ] && [ -z "$TESTER_PORT" ] && [ -z "$PROOF_HEADER" ] && [ -z "$PROOF_FILE" ] && [ "$NO_PROOF" = 0 ] && [ "$NO_SEND" = 0 ] && [ "$SSO" = 0 ] || usage
|
||||
fi
|
||||
[ "$OP" != gate ] || [ -n "$SINCE" ] || usage
|
||||
[ -z "$EXPECT_FROM" ] || [ "$PLAN" = 1 ] || usage
|
||||
[ "${ROLLBACK_OLD:+1}" = "${CAUSE:+1}" ] || usage
|
||||
[ -z "$EVENTS" ] || [ -n "$RESTORE" ] || usage
|
||||
[ "$OP" != rehearse ] || { [ -n "$R_BACKUP" ] && [ -n "$R_PORT" ]; } || usage
|
||||
# A bad tester port is a usage error; --plan lists it as tester-port-invalid instead (row 180).
|
||||
@@ -187,7 +240,7 @@ process.exit(1);' -- "$LOG" "$1" "$P_COPY"
|
||||
|
||||
# --- Reading the units ---
|
||||
|
||||
show() { systemctl --user show "$1" -p LoadState -p WorkingDirectory -p ExecStart -p Environment -p MainPID -p DropInPaths 2>/dev/null; }
|
||||
show() { sc show "$1" -p LoadState -p WorkingDirectory -p ExecStart -p Environment -p MainPID -p DropInPaths 2>/dev/null; }
|
||||
argv_of() { sed -n 's/^ExecStart=.*argv\[\]=\([^;]*\) ;.*/\1/p' <<< "$1"; }
|
||||
# systemctl show quotes a value that holds a space and escapes with a backslash, and the read -a
|
||||
# split below can't undo either, so such a line is unreadable rather than read wrong (row 166 item 4).
|
||||
@@ -286,12 +339,20 @@ check_target() {
|
||||
# has KIND exits 0 when the copy's KINDS has KIND
|
||||
# unknown prints the business file's plugins keys the copy's KINDS lacks, from a plain JSON read
|
||||
# harness prints plugins.harness as JSON, or `none`, from the copy's loader
|
||||
# launchby prints launch.by, the PM's instance, from a plain JSON read (row 232); it imports
|
||||
# only business.mjs, for the business file's path
|
||||
# A failure exits 1 and prints its first line.
|
||||
PLUGINS_JS='
|
||||
const {readFileSync}=require("node:fs");const {pathToFileURL}=require("node:url");
|
||||
const [mode,copy,id,kind]=process.argv.slice(1);
|
||||
const imp=(p)=>import(pathToFileURL(`${copy}/packages/${p}`).href);
|
||||
(async()=>{
|
||||
if(mode==="launchby"){
|
||||
const {businessFilePath}=await imp("business/src/business.mjs");
|
||||
const by=JSON.parse(readFileSync(businessFilePath(id),"utf8")).launch?.by;
|
||||
if(typeof by!=="string"||!by)throw new Error("the business file has no launch.by");
|
||||
return process.stdout.write(by);
|
||||
}
|
||||
const {KINDS}=await imp("plugins/src/kinds.mjs");
|
||||
if(mode==="has")process.exit(Object.hasOwn(KINDS,kind)?0:1);
|
||||
const b=await imp("business/src/business.mjs");
|
||||
@@ -455,13 +516,13 @@ capture() {
|
||||
|
||||
# Stops the Console, then the bus, and waits for both and the stopped check, within --timeout.
|
||||
stop_both() {
|
||||
systemctl --user stop "$CONSOLE_UNIT" || return 1
|
||||
systemctl --user stop "$BUS_UNIT" || return 1
|
||||
sc stop "$CONSOLE_UNIT" || return 1
|
||||
sc stop "$BUS_UNIT" || return 1
|
||||
local deadline=$((SECONDS + TIMEOUT)) u s down
|
||||
while :; do
|
||||
down=1
|
||||
for u in "$CONSOLE_UNIT" "$BUS_UNIT"; do
|
||||
s="$(systemctl --user is-active "$u" 2>/dev/null || true)"
|
||||
s="$(sc is-active "$u" 2>/dev/null || true)"
|
||||
case "$s" in inactive|failed) ;; *) down=0 ;; esac
|
||||
done
|
||||
if [ "$down" = 1 ] && stopped; then say "stopped both units"; return 0; fi
|
||||
@@ -472,7 +533,7 @@ stop_both() {
|
||||
|
||||
start_both() {
|
||||
START_EPOCH="$(date +%s)" && START_MS="$(date +%s%3N)" || return 1
|
||||
systemctl --user start "$BUS_UNIT" && systemctl --user start "$CONSOLE_UNIT"
|
||||
sc start "$BUS_UNIT" && sc start "$CONSOLE_UNIT"
|
||||
}
|
||||
|
||||
# Step 4: installs copy $1 with pm $2, claude $3 and the Console flags after them. Sets FAIL.
|
||||
@@ -493,9 +554,19 @@ install_stack() {
|
||||
gate_once() {
|
||||
local copy="$COPIES/$1" pm="$2" main out
|
||||
GATE=units-inactive GATE_WHY=""
|
||||
systemctl --user is-active --quiet "$BUS_UNIT" && systemctl --user is-active --quiet "$CONSOLE_UNIT" || return 1
|
||||
sc is-active --quiet "$BUS_UNIT" && sc is-active --quiet "$CONSOLE_UNIT" || return 1
|
||||
if [ "$UNITS" = external ]; then
|
||||
# r6 N4: root started the units, so this start's time is the bus unit's, in whole seconds,
|
||||
# at or after FLOOR. It is a gate code, so polling goes on: Restart=on-failure can bring a
|
||||
# fresh start.
|
||||
GATE=stale-start
|
||||
out="$(sc show --timestamp=unix -p ActiveEnterTimestamp "$BUS_UNIT" 2>/dev/null | sed -n 's/^ActiveEnterTimestamp=@//p')" || return 1
|
||||
[[ "$out" =~ ^[0-9]{1,12}$ ]] || { GATE_WHY="ActiveEnterTimestamp is unreadable"; return 1; }
|
||||
[ "$out" -ge "$FLOOR" ] || { GATE_WHY="the bus started at @$out, before @$FLOOR"; return 1; }
|
||||
START_MS=$((out * 1000))
|
||||
fi
|
||||
GATE=bus-status
|
||||
main="$(systemctl --user show "$BUS_UNIT" -p MainPID | sed -n 's/^MainPID=//p')" || return 1
|
||||
main="$(sc show "$BUS_UNIT" -p MainPID | sed -n 's/^MainPID=//p')" || return 1
|
||||
out="$("$copy/scripts/mosaic" bus status --json 2>/dev/null)" || { GATE_WHY="bus status failed"; return 1; }
|
||||
# Every `bus start` writes boot.at (packages/cli/src/boot-status.mjs), so a value from before
|
||||
# this start means the host on the socket is not the one this run started.
|
||||
@@ -508,13 +579,13 @@ process.stdout.write(why);' -- "$out" "$BUSINESS" "$main" "$START_MS" 2>&1)" ||
|
||||
node -e '
|
||||
fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>process.exit(r.status===200?0:1),()=>process.exit(1));' -- "$(console_port)" || return 1
|
||||
GATE=copy-check
|
||||
out="$(scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>/dev/null)" || return 1
|
||||
out="$(scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" "${UNITS_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>/dev/null)" || return 1
|
||||
# check passes when the three agree on one copy; this run needs it to be the target (row 166 item 3).
|
||||
out="$(sed -n 's/^bus: //p' <<< "$out")"
|
||||
[ "$out" = "$copy" ] || { GATE_WHY="the bus, Console and CLI agree on $out, not $copy"; return 1; }
|
||||
if [ "$pm" = 1 ]; then
|
||||
GATE=pm-launch-unconfirmed
|
||||
pm_launched "$copy" || return 1
|
||||
if [ "$UNITS" = user ]; then pm_launched "$copy" || return 1; else pm_external "$1" || return 1; fi
|
||||
fi
|
||||
GATE=""
|
||||
}
|
||||
@@ -526,7 +597,7 @@ fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(
|
||||
# because this host's journalctl reads a bare time as local time.
|
||||
pm_launched() {
|
||||
local journal run list
|
||||
journal="$(journalctl --user -u "$BUS_UNIT" --since "@$START_EPOCH" -o cat 2>/dev/null)" || { GATE_WHY="journal read failed"; return 1; }
|
||||
journal="$(jctl -u "$BUS_UNIT" --since "@$START_EPOCH" -o cat 2>/dev/null)" || { GATE_WHY="journal read failed"; return 1; }
|
||||
local -a lines=()
|
||||
mapfile -t lines < <(grep '^launched ' <<< "$journal" || true)
|
||||
[ "${#lines[@]}" -gt 0 ] || { GATE_WHY="no launched line since the start"; return 1; }
|
||||
@@ -540,6 +611,20 @@ process.stdout.write(!s?`run ${run} is not in launches list`:s.business!==b?`run
|
||||
[ -z "$GATE_WHY" ]
|
||||
}
|
||||
|
||||
# r6 section 4: the PM check under --units external, where the bus unit's journal is the system
|
||||
# journal and unreadable here. It passes when exactly one live launches list entry is for the
|
||||
# PM's instance (launch.by, packages/cli/src/launcher.mjs:675), and that entry is for BUSINESS and
|
||||
# started at or after this start. A run the PM starts for another role has another instance.
|
||||
pm_external() {
|
||||
local by list
|
||||
by="$(plugins launchby "$1" 2>&1)" || { GATE_WHY="cannot read launch.by: $by"; return 1; }
|
||||
list="$("$COPIES/$1/scripts/mosaic" launches list --json 2>/dev/null)" || { GATE_WHY="launches list failed"; return 1; }
|
||||
GATE_WHY="$(node -e '
|
||||
const [list,by,b,since]=process.argv.slice(1),l=JSON.parse(list).filter((x)=>x.instance===by&&x.live===true),s=l[0];
|
||||
process.stdout.write(l.length===0?`no live launch of ${by}`:l.length>1?`${l.length} live launches of ${by}`:s.business!==b?`the launch of ${by} is for ${s.business}`:!(Date.parse(s.startedAt)>=Number(since))?`run ${s.run} started before this start`:"");' -- "$list" "$by" "$BUSINESS" "$START_MS" 2>&1)" || { GATE_WHY="launches list unreadable"; return 1; }
|
||||
[ -z "$GATE_WHY" ]
|
||||
}
|
||||
|
||||
# Polls every second until every check passes or --timeout runs out.
|
||||
gate() {
|
||||
local deadline=$((SECONDS + TIMEOUT))
|
||||
@@ -573,7 +658,7 @@ put_back() {
|
||||
done
|
||||
[ "$(sha256sum < "$BIN/$WRAPPER_NAME" | cut -d' ' -f1)" = "$(sum_of "$RS" wrapper)" ] || return 1
|
||||
[ "$(stat -c %a "$BIN/$WRAPPER_NAME")" = "$(cat "$RS/WRAPPER_MODE")" ] || return 1
|
||||
systemctl --user daemon-reload || return 1
|
||||
sc daemon-reload || return 1
|
||||
say "put back the unit files and the wrapper from $RS"
|
||||
}
|
||||
|
||||
@@ -608,8 +693,8 @@ after_fail() {
|
||||
# Stop both units, print both results, log rollback-failed, exit 2. No retry.
|
||||
rollback_failed() {
|
||||
trap - INT TERM HUP
|
||||
systemctl --user stop "$CONSOLE_UNIT" || true
|
||||
systemctl --user stop "$BUS_UNIT" || true
|
||||
sc stop "$CONSOLE_UNIT" || true
|
||||
sc stop "$BUS_UNIT" || true
|
||||
say "rollback-failed: run $1, rollback $2"
|
||||
log_line result rollback-failed code "$1" rollback failed rollbackCode "$2"
|
||||
echo "stack-release: rollback-failed: the run failed with $1 and the rollback with $2; both units are stopped. Rollback set: $RS${BACKUP:+, backup $DATA/backups/stack/$BACKUP}${RESTORED:+, post-attempt store $POST}" >&2
|
||||
@@ -618,10 +703,15 @@ rollback_failed() {
|
||||
|
||||
# --- Refusals (item 3) ---
|
||||
|
||||
lock() {
|
||||
# confirm and revert act on the open activation by design, so they take the lock alone (r6 section 4).
|
||||
lock_only() {
|
||||
mkdir -p "$STATE"
|
||||
exec 9> "$STATE/stack-release.lock"
|
||||
flock -n 9 || refuse run-locked "another stack-release run holds $STATE/stack-release.lock"
|
||||
}
|
||||
|
||||
lock() {
|
||||
lock_only
|
||||
local open
|
||||
open="$(interrupted | paste -sd' ' -)"
|
||||
[ -z "$open" ] || refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)"
|
||||
@@ -629,7 +719,7 @@ lock() {
|
||||
|
||||
instance_busy() {
|
||||
local out unit load state rest
|
||||
out="$(systemctl --user list-units --all --no-legend --plain "$BUS_GLOB" 2>/dev/null)" || refuse instance-busy "systemctl --user list-units failed, so the bus state is unknown"
|
||||
out="$(sc list-units --all --no-legend --plain "$BUS_GLOB" 2>/dev/null)" || refuse instance-busy "$SC_NAME list-units failed, so the bus state is unknown"
|
||||
while read -r unit load state rest; do
|
||||
[[ "$unit" == $BUS_GLOB ]] || continue
|
||||
if [ "$unit" = "$BUS_UNIT" ]; then
|
||||
@@ -645,7 +735,7 @@ instance_busy() {
|
||||
drop_in_override() {
|
||||
local paths directory dropdir file content glob
|
||||
local -a dropdirs
|
||||
paths="$(systemd-analyze --user unit-paths 2>/dev/null)" && [ -n "$paths" ] || refuse drop-in-override "systemd unit paths are unknown"
|
||||
paths="$(analyze unit-paths 2>/dev/null)" && [ -n "$paths" ] || refuse drop-in-override "systemd unit paths are unknown"
|
||||
while IFS= read -r directory; do
|
||||
[ -n "$directory" ] || continue
|
||||
case "$directory" in /*) ;; *) refuse drop-in-override "invalid systemd unit path $directory" ;; esac
|
||||
@@ -679,7 +769,8 @@ flags_drift() {
|
||||
if [ "$AFTER_HARNESS" = 1 ]; then
|
||||
[ "$P_PM" = 1 ] || d+=$'\n'" pm: pointer $P_PM, --pm-after-harness needs a pointer with pm 1"
|
||||
else
|
||||
[ "$OP" != activate ] || [ "$WANT_PM" = "$P_PM" ] || d+=$'\n'" pm: pointer $P_PM, --pm given $WANT_PM"
|
||||
# Under --units external the PM is a flag in root's unit, so there is no --pm to compare.
|
||||
[ "$UNITS" != user ] || [ "$OP" != activate ] || [ "$WANT_PM" = "$P_PM" ] || d+=$'\n'" pm: pointer $P_PM, --pm given $WANT_PM"
|
||||
fi
|
||||
[ -z "$d" ] || refuse unit-flags-drift "the units differ from the pointer; settle it first:$d"
|
||||
}
|
||||
@@ -1075,11 +1166,12 @@ do_adopt() {
|
||||
lock
|
||||
[ ! -f "$POINTER" ] || { read_pointer; FROM="$P_COPY"; refuse already-adopted "$POINTER exists (copy $P_COPY)"; }
|
||||
local out
|
||||
out="$(scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1)" || refuse check-failed "stack-copy.sh check did not pass:"$'\n'"$out"
|
||||
out="$(scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" "${UNITS_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1)" || refuse check-failed "stack-copy.sh check did not pass:"$'\n'"$out"
|
||||
TO="$(sed -n 's/^bus: //p' <<< "$out")"
|
||||
TO="${TO#"$COPIES/"}"
|
||||
read_units || refuse check-failed "cannot read $BUS_UNIT or $CONSOLE_UNIT"
|
||||
[ -n "$U_CLAUDE" ] || refuse no-claude-pin "$BUS_UNIT has no MOSAIC_CLAUDE_BIN"
|
||||
# Under --units external the claude pin is root's render, so adopt records what the unit has.
|
||||
[ "$UNITS" = external ] || [ -n "$U_CLAUDE" ] || refuse no-claude-pin "$BUS_UNIT has no MOSAIC_CLAUDE_BIN"
|
||||
target_schema "$TO" || refuse copy-schema-unknown "cannot read SCHEMA_VERSION from $COPIES/$TO/packages/bus/src/store.mjs"
|
||||
next_activation
|
||||
write_pointer "$TO" "$U_PM" 0 "$U_CLAUDE" "$T_SCHEMA" "${U_CONSOLE[@]}"
|
||||
@@ -1094,12 +1186,12 @@ plan_store() {
|
||||
local src="" why=""
|
||||
S_SCHEMA="" S_FROM="" S_CHANGED=""
|
||||
if [ ! -e "$STORE" ]; then why="there is no store file"
|
||||
elif systemctl --user is-active --quiet "$BUS_UNIT"; then
|
||||
elif sc is-active --quiet "$BUS_UNIT"; then
|
||||
src="the store the running bus holds"
|
||||
S_SCHEMA="$(node -e "$STORE_JS" -- heldversion "$STORE" 2>/dev/null)" || S_SCHEMA=""
|
||||
# A bus that stops during the readOnly open can leave -wal or -shm beside the stopped
|
||||
# store, so the version read can't be planned on (row 166 item 2).
|
||||
systemctl --user is-active --quiet "$BUS_UNIT" || S_CHANGED=1
|
||||
sc is-active --quiet "$BUS_UNIT" || S_CHANGED=1
|
||||
elif store_unclean; then why="the store was not read: $UNCLEAN is present"
|
||||
else
|
||||
src="the stopped store"
|
||||
@@ -1186,15 +1278,19 @@ plan() {
|
||||
echo "backup: the run backs up the stopped store to $DATA/backups/stack/<activation>/ before any migration"
|
||||
fi
|
||||
plan_commits
|
||||
[ "$P_PENDING" = 0 ] || echo "pm: the pointer is pending (activate --pm-after-harness); the bus runs without its PM"
|
||||
[ "$RUN_PENDING" = 0 ] || echo "pm: the bus starts without --pm and the PM stays pending; next: Jason's plugins set of harness, then pm-on"
|
||||
local flags="${INSTANCE_FLAGS[*]:+${INSTANCE_FLAGS[*]} }"
|
||||
echo "bus: scripts/bus-service.sh install $flags--copy $TO --copies $COPIES$([ "$RUN_PM" = 0 ] || echo ' --pm') --claude $P_CLAUDE"
|
||||
[ -z "$DROP_TESTER" ] || echo "drop $DROP_TESTER; the target Console does not support it"
|
||||
echo "$PROOF_PLAN"
|
||||
echo "$SEND_PLAN"
|
||||
[ -z "$SSO_PLAN" ] || echo "$SSO_PLAN"
|
||||
echo "Console: scripts/webui-service.sh install $flags--copy $TO --copies $COPIES${RUN_CONSOLE[*]:+ ${RUN_CONSOLE[*]}}"
|
||||
if [ "$UNITS" = external ]; then
|
||||
echo "units: external; root installs the units for $TO and starts them, then confirm gates them"
|
||||
else
|
||||
[ "$P_PENDING" = 0 ] || echo "pm: the pointer is pending (activate --pm-after-harness); the bus runs without its PM"
|
||||
[ "$RUN_PENDING" = 0 ] || echo "pm: the bus starts without --pm and the PM stays pending; next: Jason's plugins set of harness, then pm-on"
|
||||
echo "bus: scripts/bus-service.sh install $flags--copy $TO --copies $COPIES$([ "$RUN_PM" = 0 ] || echo ' --pm') --claude $P_CLAUDE"
|
||||
[ -z "$DROP_TESTER" ] || echo "drop $DROP_TESTER; the target Console does not support it"
|
||||
echo "$PROOF_PLAN"
|
||||
echo "$SEND_PLAN"
|
||||
[ -z "$SSO_PLAN" ] || echo "$SSO_PLAN"
|
||||
echo "Console: scripts/webui-service.sh install $flags--copy $TO --copies $COPIES${RUN_CONSOLE[*]:+ ${RUN_CONSOLE[*]}}"
|
||||
fi
|
||||
echo "CLI: scripts/stack-copy.sh wrapper $TO $flags--copies $COPIES --bin $BIN"
|
||||
local r
|
||||
for r in "${PLAN_REFUSE[@]}"; do echo "refuses $r"; done
|
||||
@@ -1268,6 +1364,16 @@ do_activate() {
|
||||
pointer_for_business
|
||||
run_pm
|
||||
[ "$P_PENDING" = 0 ] || check_refuse pm-pending "the pointer is pending (activate --pm-after-harness); run pm-on or rollback first"
|
||||
if [ "$UNITS" = external ]; then
|
||||
ext_pointer_checks
|
||||
if [ "$PLAN" = 1 ]; then check_target; plugin_checks; config_checks; plan; exit 0; fi
|
||||
ext_precheck
|
||||
plugin_checks
|
||||
config_checks
|
||||
new_activation
|
||||
run_external migrate
|
||||
return
|
||||
fi
|
||||
if [ "$PLAN" = 1 ]; then check_target; console_for_target; ports_unclaimed; plugin_checks; config_checks; plan; exit 0; fi
|
||||
precheck
|
||||
plugin_checks
|
||||
@@ -1281,43 +1387,321 @@ do_rollback() {
|
||||
pointer_for_business
|
||||
TO="$(from_log previous)" || refuse no-previous "the log names no earlier copy than $P_COPY"
|
||||
run_pm
|
||||
if [ "$UNITS" = external ]; then
|
||||
# r6: root names the target from the go, and it must be the one rollback reads from the log.
|
||||
[ "$TO_FLAG" = "$TO" ] || refuse to-not-previous "--to $TO_FLAG is not the previous copy; the log names $TO"
|
||||
ext_pointer_checks
|
||||
if [ "$PLAN" = 1 ]; then
|
||||
check_target; plugin_checks; config_checks
|
||||
[ -z "$RESTORE" ] || { next_activation; restore_checks; }
|
||||
plan; exit 0
|
||||
fi
|
||||
ext_precheck
|
||||
plugin_checks
|
||||
config_checks
|
||||
[ "$P_SCHEMA" = "$T_SCHEMA" ] || [ -n "$RESTORE" ] || refuse schema-differs "the store is at $P_SCHEMA and $TO expects $T_SCHEMA; pass --restore ACTIVATION"
|
||||
new_activation
|
||||
[ -z "$RESTORE" ] || restore_checks
|
||||
run_external restore
|
||||
return
|
||||
fi
|
||||
if [ "$PLAN" = 1 ]; then check_target; console_for_target; ports_unclaimed; plugin_checks; config_checks; plan; exit 0; fi
|
||||
precheck
|
||||
plugin_checks
|
||||
config_checks
|
||||
[ "$P_SCHEMA" = "$T_SCHEMA" ] || [ -n "$RESTORE" ] || refuse schema-differs "the store is at $P_SCHEMA and $TO expects $T_SCHEMA; pass --restore ACTIVATION"
|
||||
new_activation
|
||||
if [ -n "$RESTORE" ]; then
|
||||
local o="$DATA/backups/stack/$RESTORE" rc=0 c="" text
|
||||
[ -f "$o/SHA256SUMS" ] && [ "$(cat "$o/DATA_ROOT" 2>/dev/null)" = "$DATA" ] && [ "$(cat "$o/SCHEMA" 2>/dev/null)" = "$T_SCHEMA" ] && [ -f "$o/STAMP" ] \
|
||||
|| refuse restore-mismatch "$o is not a backup of this root at $T_SCHEMA"
|
||||
# Before anything stops. A store the running bus holds is read with readOnly, which adds
|
||||
# no file (the probe); a stopped one falls under the store_unclean rule.
|
||||
if systemctl --user is-active --quiet "$BUS_UNIT"; then
|
||||
c="$(node -e "$STORE_JS" -- heldcount "$STORE" "$(cat "$o/STAMP")" "$COUNTED")" || rc=1
|
||||
# plan_store's recheck (row 166 item 2), before anything stops.
|
||||
systemctl --user is-active --quiet "$BUS_UNIT" || refuse restore-mismatch "the bus stopped while the events after $(cat "$o/STAMP") were counted, so the store state changed; nothing was stopped"
|
||||
else
|
||||
! store_unclean || refuse store-unclean "$UNCLEAN is present; the bus did not close the store cleanly"
|
||||
c="$(store_read count "$(cat "$o/STAMP")" "$COUNTED")" || rc=1
|
||||
fi
|
||||
[ "$rc" = 0 ] || refuse restore-mismatch "cannot count the rows after $(cat "$o/STAMP")"
|
||||
COUNT="${c%%$'\t'*}" text="${c#*$'\t'}"
|
||||
echo "restore: $o; the store now in place moves to $POST and is kept; $text were written after the backup"
|
||||
[ "$COUNT" = 0 ] || [ "$EVENTS" = "$COUNT" ] || refuse post-backup-events "$text were written after the backup; Jason decides, then --events $COUNT"
|
||||
BACKUP="$RESTORE"
|
||||
fi
|
||||
[ -z "$RESTORE" ] || restore_checks
|
||||
run_steps restore
|
||||
}
|
||||
|
||||
# rollback --restore's checks on the backup and the rows written after it, before anything stops.
|
||||
# Sets COUNT and BACKUP; POST comes from new_activation.
|
||||
restore_checks() {
|
||||
local o="$DATA/backups/stack/$RESTORE" rc=0 c="" text
|
||||
[ -f "$o/SHA256SUMS" ] && [ "$(cat "$o/DATA_ROOT" 2>/dev/null)" = "$DATA" ] && [ "$(cat "$o/SCHEMA" 2>/dev/null)" = "$T_SCHEMA" ] && [ -f "$o/STAMP" ] \
|
||||
|| refuse restore-mismatch "$o is not a backup of this root at $T_SCHEMA"
|
||||
# Before anything stops. A store the running bus holds is read with readOnly, which adds
|
||||
# no file (the probe); a stopped one falls under the store_unclean rule.
|
||||
if sc is-active --quiet "$BUS_UNIT"; then
|
||||
c="$(node -e "$STORE_JS" -- heldcount "$STORE" "$(cat "$o/STAMP")" "$COUNTED")" || rc=1
|
||||
# plan_store's recheck (row 166 item 2), before anything stops.
|
||||
sc is-active --quiet "$BUS_UNIT" || refuse restore-mismatch "the bus stopped while the events after $(cat "$o/STAMP") were counted, so the store state changed; nothing was stopped"
|
||||
else
|
||||
! store_unclean || refuse store-unclean "$UNCLEAN is present; the bus did not close the store cleanly"
|
||||
c="$(store_read count "$(cat "$o/STAMP")" "$COUNTED")" || rc=1
|
||||
fi
|
||||
[ "$rc" = 0 ] || refuse restore-mismatch "cannot count the rows after $(cat "$o/STAMP")"
|
||||
COUNT="${c%%$'\t'*}" text="${c#*$'\t'}"
|
||||
echo "restore: $o; the store now in place moves to $POST and is kept; $text were written after the backup"
|
||||
[ "$COUNT" = 0 ] || [ "$EVENTS" = "$COUNT" ] || refuse post-backup-events "$text were written after the backup; Jason decides, then --events $COUNT"
|
||||
BACKUP="$RESTORE"
|
||||
}
|
||||
|
||||
# --- --units external (row 232, r6 section 4) ---
|
||||
|
||||
# After the pointer, for activate and rollback, in --plan and in the run. OLD is the copy root read
|
||||
# from the unit files it installed (r6 A1, N5), so a pointer that names another copy stops the
|
||||
# re-pin before root stops anything. The PM is a flag in root's unit, so --pm-after-harness has
|
||||
# nothing to wait on. The run's own lock refuses an open activation; --plan takes no lock, so it
|
||||
# lists one here.
|
||||
ext_pointer_checks() {
|
||||
[ -z "$EXPECT_FROM" ] || [ "$EXPECT_FROM" = "$P_COPY" ] || check_refuse pointer-differs "the pointer names $P_COPY and root's units name $EXPECT_FROM; Jason or Mos settles it"
|
||||
[ "$AFTER_HARNESS" = 0 ] || check_refuse external-pm "under --units external the PM is a flag in root's bus unit; changing it is a re-install with its own go"
|
||||
[ "$PLAN" = 1 ] || return 0
|
||||
local open
|
||||
open="$(interrupted | paste -sd' ' -)"
|
||||
[ -z "$open" ] || check_refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)"
|
||||
}
|
||||
|
||||
# A3's checks before `started`. Root has stopped the units and installs them after this run, so
|
||||
# there is no Console flag, port, drop-in or claude pin of this script's to check.
|
||||
ext_precheck() {
|
||||
check_target
|
||||
instance_busy
|
||||
read_units || refuse unit-flags-drift "cannot read $BUS_UNIT or $CONSOLE_UNIT"
|
||||
flags_drift
|
||||
units_down
|
||||
}
|
||||
|
||||
# r6 A3 (N4): each unit reads inactive, or failed as stop_both accepts, and the store is closed,
|
||||
# so nothing below runs against a store a live bus holds.
|
||||
units_down() {
|
||||
local u s
|
||||
for u in "$CONSOLE_UNIT" "$BUS_UNIT"; do
|
||||
s="$(sc is-active "$u" 2>/dev/null || true)"
|
||||
case "$s" in inactive|failed) ;; *) refuse units-running "$u is ${s:-unknown}; root stops the units first" ;; esac
|
||||
done
|
||||
stopped || refuse units-running "the store under $DATA/bus is still open (writer.lock or an open file)"
|
||||
}
|
||||
|
||||
# A3's capture into $RS, which run_external made: the pointer, the wrapper and its mode, and for each
|
||||
# unit its show output and the sha256 of its unit file. It reads root's units and makes no directory.
|
||||
capture_units() {
|
||||
local u frag
|
||||
[ ! -f "$POINTER" ] || cp -p "$POINTER" "$RS/pointer" || return 1
|
||||
[ -f "$BIN/$WRAPPER_NAME" ] || { say "capture: no wrapper $BIN/$WRAPPER_NAME"; return 1; }
|
||||
cp -p "$BIN/$WRAPPER_NAME" "$RS/wrapper" || return 1
|
||||
stat -c %a "$BIN/$WRAPPER_NAME" > "$RS/WRAPPER_MODE" || return 1
|
||||
: > "$RS/UNITS.sha256" || return 1
|
||||
for u in "$BUS_UNIT" "$CONSOLE_UNIT"; do
|
||||
show "$u" > "$RS/show-$u.txt" || return 1
|
||||
frag="$(sc show "$u" -p FragmentPath 2>/dev/null | sed -n 's/^FragmentPath=//p')" && [ -n "$frag" ] || { say "capture: no FragmentPath for $u"; return 1; }
|
||||
sha256sum "$frag" >> "$RS/UNITS.sha256" || return 1
|
||||
done
|
||||
sums "$RS" > "$RS/SHA256SUMS" || return 1
|
||||
say "captured the rollback set: $RS"
|
||||
}
|
||||
|
||||
# A failure after the capture: FAILED holds the code, nothing is logged or cleaned up, exit 1.
|
||||
# Root runs R1-R4, and revert is the one cleanup path (r6 A3).
|
||||
ext_fail() {
|
||||
trap - INT TERM HUP
|
||||
{ echo "$1" > "$RS/FAILED"; } 2>/dev/null || true
|
||||
say "failed: $1"
|
||||
echo "stack-release: $OP failed with $1; $ACT stays open for root's revert and confirm --rollback. Rollback set: $RS" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# A3 for target $TO. $1 is the store step: migrate or restore. The marker goes in first, so every
|
||||
# failure after it leaves what revert needs. Writes no pointer and starts nothing.
|
||||
run_external() {
|
||||
log_line result started
|
||||
{ mkdir -p -m 0700 "$STATE/stack-rollback" && mkdir -m 0700 "$RS"; } 2>/dev/null || refuse marker-failed "could not make the rollback set $RS; nothing was changed"
|
||||
{ echo "$TO" > "$RS/AWAITING-UNITS"; } 2>/dev/null || { rm -f "$RS/AWAITING-UNITS"; refuse marker-failed "could not write $RS/AWAITING-UNITS; nothing was changed"; }
|
||||
STEPS="$RS/steps.log"
|
||||
capture_units || { rm -f "$RS/AWAITING-UNITS"; refuse capture-failed "could not write the rollback set $RS; nothing was changed"; }
|
||||
trap 'ext_fail signal' INT TERM HUP
|
||||
if [ -e "$STORE" ]; then
|
||||
if store_unclean; then say "$UNCLEAN is present; the store is left as it is for Jason or Mos"; ext_fail store-unclean; fi
|
||||
STORE_SCHEMA="$(store_read version)" && [ -n "$STORE_SCHEMA" ] || ext_fail schema-unreadable
|
||||
say "stopped store schema: $STORE_SCHEMA"
|
||||
if [ "$1" = migrate ]; then
|
||||
if [ "$STORE_SCHEMA" != "$T_SCHEMA" ]; then
|
||||
in_chain "$STORE_SCHEMA" || ext_fail store-newer
|
||||
[ "$MIGRATE" = 1 ] || ext_fail migration-needs-flag
|
||||
fi
|
||||
backup_store || ext_fail backup
|
||||
echo "$BACKUP" > "$RS/BACKUP" || ext_fail backup
|
||||
if [ "$STORE_SCHEMA" != "$T_SCHEMA" ]; then
|
||||
# MIGRATING and TAKEN before the migration, so revert restores a partial one (r6 B2).
|
||||
: > "$RS/MIGRATING" || ext_fail migrate-failed
|
||||
TAKEN="$ACT"
|
||||
node "$COPIES/$TO/packages/bus/src/migrate-schema.mjs" "$DATA" >> "$STEPS" 2>&1 || ext_fail migrate-failed
|
||||
say "migrated the store to $T_SCHEMA"
|
||||
fi
|
||||
else
|
||||
[ "$STORE_SCHEMA" = "$P_SCHEMA" ] || ext_fail store-differs
|
||||
if [ -n "$RESTORE" ]; then
|
||||
local n
|
||||
n="$(store_read count "$(cat "$DATA/backups/stack/$RESTORE/STAMP")" "$COUNTED")" || ext_fail schema-unreadable
|
||||
[ "${n%%$'\t'*}" = "$COUNT" ] || ext_fail post-backup-events
|
||||
echo "$RESTORE" > "$RS/BACKUP" || ext_fail restore
|
||||
# RESTORED tells confirm --rollback the store was swapped, so it skips OLD's gate
|
||||
# and logs rollbackCode not-attempted (ruling 4, T14b).
|
||||
if ! restore_store "$RESTORE"; then
|
||||
[ -z "$RESTORED" ] || : > "$RS/RESTORED" || true
|
||||
ext_fail restore
|
||||
fi
|
||||
: > "$RS/RESTORED" || ext_fail restore
|
||||
fi
|
||||
fi
|
||||
elif [ -n "$RESTORE" ]; then
|
||||
ext_fail store-differs
|
||||
fi
|
||||
scripts/stack-copy.sh wrapper "$TO" "${INSTANCE_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" >> "$STEPS" 2>&1 || ext_fail wrapper
|
||||
trap - INT TERM HUP
|
||||
say "staged $TO"
|
||||
echo "stack-release: $OP staged $TO for $BUSINESS as $ACT; root installs and starts the units, then runs confirm $TO $BUSINESS --units external"
|
||||
}
|
||||
|
||||
# r6 "The open activation": exactly one AWAITING-UNITS file, its activation the one the log holds
|
||||
# open, holding the copy root passed. Sets O_ACT. Anything else refuses open-activation, and ACT
|
||||
# stays unset, so the refused line names no activation and the open one stays open.
|
||||
open_activation() {
|
||||
local -a markers open
|
||||
local m found="" seen=""
|
||||
mapfile -t markers < <(compgen -G "$STATE/stack-rollback/*/AWAITING-UNITS" || true)
|
||||
mapfile -t open < <(interrupted)
|
||||
for m in "${markers[@]}"; do
|
||||
seen+=" $(basename "$(dirname "$m")") (copy $(cat "$m" 2>/dev/null || echo unreadable))"
|
||||
done
|
||||
if [ "${#markers[@]}" = 1 ] && [ "${#open[@]}" = 1 ]; then
|
||||
m="$(basename "$(dirname "${markers[0]}")")"
|
||||
[ "$m" != "${open[0]}" ] || [ "$(cat "${markers[0]}" 2>/dev/null)" != "$ID" ] || found="$m"
|
||||
fi
|
||||
[ -n "$found" ] || refuse open-activation "need one AWAITING-UNITS for copy $ID whose activation the log holds open; markers:${seen:- none}; open: ${open[*]:-none}"
|
||||
O_ACT="$found"
|
||||
}
|
||||
|
||||
# The op, from, to and start second of O_ACT's `started` line, one per line.
|
||||
started_of() {
|
||||
node -e '
|
||||
const [file,act]=process.argv.slice(1);
|
||||
let s;
|
||||
for(const l of require("fs").readFileSync(file,"utf8").split("\n")){
|
||||
if(!l.trim())continue;
|
||||
let o;try{o=JSON.parse(l)}catch{continue}
|
||||
if(o.activation===act&&o.result==="started")s=o;
|
||||
}
|
||||
const at=s&&Math.floor(Date.parse(s.at)/1000);
|
||||
if(!s||!["activate","rollback"].includes(s.op)||!Number.isFinite(at))process.exit(1);
|
||||
process.stdout.write([s.op,s.from??"",s.to??"",at].join("\n")+"\n");' -- "$LOG" "$O_ACT"
|
||||
}
|
||||
|
||||
# A5, and R4 with --rollback. The lines it writes carry the open activation's op, from, to and
|
||||
# activation, and the backup A3 recorded, so from_log reads them as a dragon-lin run's. Every check
|
||||
# runs before ACT is set.
|
||||
do_confirm() {
|
||||
lock_only
|
||||
open_activation
|
||||
local s
|
||||
local -a f
|
||||
s="$(started_of)" || refuse open-activation "the log's started line for $O_ACT is unreadable"
|
||||
mapfile -t f <<< "$s"
|
||||
[ "${f[2]}" = "$ID" ] || refuse open-activation "$O_ACT started for ${f[2]}, not $ID"
|
||||
if read_pointer; then [ "$P_BUSINESS" = "$BUSINESS" ] || refuse business-mismatch "the pointer is for $P_BUSINESS"; fi
|
||||
if [ -n "$ROLLBACK_OLD" ]; then
|
||||
[ "$ROLLBACK_OLD" = "${f[1]}" ] || refuse rollback-mismatch "$ROLLBACK_OLD is not the copy $O_ACT started from, ${f[1]:-none}"
|
||||
else
|
||||
[ ! -e "$STATE/stack-rollback/$O_ACT/FAILED" ] || refuse activation-failed "$O_ACT has a FAILED file; root runs revert and confirm --rollback"
|
||||
target_schema "$ID" || refuse copy-schema-unknown "cannot read SCHEMA_VERSION from $COPIES/$ID/packages/bus/src/store.mjs"
|
||||
fi
|
||||
OP="${f[0]}" FROM="${f[1]}" TO="${f[2]}" FLOOR="${f[3]}" ACT="$O_ACT" RS="$STATE/stack-rollback/$O_ACT"
|
||||
STEPS="$RS/steps.log"
|
||||
BACKUP="$(cat "$RS/BACKUP" 2>/dev/null || true)"
|
||||
if [ -z "$ROLLBACK_OLD" ]; then
|
||||
read_units || ext_fail units-unreadable
|
||||
P_CONSOLE=("${U_CONSOLE[@]}")
|
||||
gate "$TO" "$U_PM" || ext_fail "$GATE"
|
||||
write_pointer "$TO" "$U_PM" 0 "$U_CLAUDE" "$T_SCHEMA" "${U_CONSOLE[@]}"
|
||||
log_line result ok
|
||||
rm -f "$RS/AWAITING-UNITS"
|
||||
echo "stack-release: $OP ok: $TO is live for $BUSINESS"
|
||||
return
|
||||
fi
|
||||
local code rcode=""
|
||||
if [ "$CAUSE" = A4 ]; then code=units-start
|
||||
else code="$(head -n 1 "$RS/FAILED" 2>/dev/null || true)"; [ -n "$code" ] || code=unrecorded
|
||||
fi
|
||||
if [ -e "$RS/RESTORED" ]; then rcode=not-attempted
|
||||
elif ! read_units; then rcode=units-unreadable
|
||||
else
|
||||
P_CONSOLE=("${U_CONSOLE[@]}")
|
||||
gate "$ROLLBACK_OLD" "$U_PM" || rcode="$GATE"
|
||||
fi
|
||||
if [ -z "$rcode" ]; then
|
||||
local r=failed
|
||||
[ "$CAUSE" != A5 ] || r=gate-failed
|
||||
log_line result "$r" code "$code" rollback ok
|
||||
rm -f "$RS/AWAITING-UNITS" "$RS/FAILED"
|
||||
echo "stack-release: $code; rolled back to $ROLLBACK_OLD, which passed the gate" >&2
|
||||
exit 1
|
||||
fi
|
||||
say "rollback-failed: run $code, rollback $rcode"
|
||||
log_line result rollback-failed code "$code" rollback failed rollbackCode "$rcode"
|
||||
rm -f "$RS/AWAITING-UNITS" "$RS/FAILED"
|
||||
echo "stack-release: rollback-failed: the run failed with $code and the rollback with $rcode; root stops the units. Rollback set: $RS${BACKUP:+, backup $DATA/backups/stack/$BACKUP}" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
revert_fail() {
|
||||
say "revert failed: $1"
|
||||
echo "stack-release: revert failed: $1; $O_ACT stays open. Rollback set: $RS" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
# The store already holds backup $1 when every file checks against its SHA256SUMS and the listing
|
||||
# matches, as after a revert that restored it.
|
||||
store_matches() {
|
||||
local o="$DATA/backups/stack/$1"
|
||||
[ -d "$DATA/bus" ] && (cd "$DATA/bus" && sha256sum -c --quiet "$o/SHA256SUMS") >/dev/null 2>&1 &&
|
||||
[ "$(ls -1A "$DATA/bus")" = "$(ls -1A "$o/bus")" ]
|
||||
}
|
||||
|
||||
# R2: puts the wrapper back and, after a migration, the backup taken before it. Each part can run
|
||||
# again, so it serves an A3 that stopped partway. A rollback's --restore swap is confirm's to
|
||||
# report, not revert's to undo. Writes no log line.
|
||||
do_revert() {
|
||||
lock_only
|
||||
open_activation
|
||||
RS="$STATE/stack-rollback/$O_ACT" STEPS="$STATE/stack-rollback/$O_ACT/steps.log"
|
||||
[[ "$O_ACT" =~ ^([0-9]{8}T[0-9]{6}Z)-[0-9a-f]{12}(-[1-9])?$ ]] || revert_fail "$O_ACT is not an activation id"
|
||||
POST="$DATA/bus.post-attempt-${BASH_REMATCH[1]}${BASH_REMATCH[2]}"
|
||||
[ -f "$RS/wrapper" ] && [ -f "$RS/WRAPPER_MODE" ] || revert_fail "the rollback set has no saved wrapper"
|
||||
local tmp
|
||||
tmp="$(mktemp "$BIN/.$WRAPPER_NAME.XXXXXX")" && cp -p "$RS/wrapper" "$tmp" && chmod "$(cat "$RS/WRAPPER_MODE")" "$tmp" && mv -f "$tmp" "$BIN/$WRAPPER_NAME" ||
|
||||
revert_fail "could not put the wrapper back"
|
||||
[ "$(sha256sum < "$BIN/$WRAPPER_NAME" | cut -d' ' -f1)" = "$(sum_of "$RS" wrapper)" ] &&
|
||||
[ "$(stat -c %a "$BIN/$WRAPPER_NAME")" = "$(cat "$RS/WRAPPER_MODE")" ] || revert_fail "the wrapper does not match the rollback set"
|
||||
say "put back the wrapper from $RS"
|
||||
if [ -e "$RS/MIGRATING" ]; then
|
||||
TAKEN="$(cat "$RS/BACKUP" 2>/dev/null || true)"
|
||||
[ -n "$TAKEN" ] || revert_fail "MIGRATING is set and BACKUP names no backup"
|
||||
if store_matches "$TAKEN"; then say "the store already matches backup $TAKEN"
|
||||
else restore_store "$TAKEN" || revert_fail "could not restore backup $TAKEN"
|
||||
fi
|
||||
fi
|
||||
echo "stack-release: revert ok: $O_ACT's wrapper$([ -e "$RS/MIGRATING" ] && echo ' and store are' || echo ' is') back; root installs the old units next"
|
||||
}
|
||||
|
||||
# A2f: gates copy ID with no open activation, on a start at or after --since. Takes no lock and
|
||||
# writes nothing, so it refuses nothing: 0 on a pass, 1 otherwise.
|
||||
do_gate() {
|
||||
FLOOR="$SINCE"
|
||||
read_units || { echo "stack-release: gate: cannot read $BUS_UNIT or $CONSOLE_UNIT" >&2; exit 1; }
|
||||
P_CONSOLE=("${U_CONSOLE[@]}")
|
||||
gate "$ID" "$U_PM" || exit 1
|
||||
}
|
||||
|
||||
# --- pm-on (row 169 item 3) ---
|
||||
|
||||
# Stops the bus alone and waits for it and the stopped check, within --timeout. The Console runs on.
|
||||
stop_bus() {
|
||||
systemctl --user stop "$BUS_UNIT" || return 1
|
||||
sc stop "$BUS_UNIT" || return 1
|
||||
local deadline=$((SECONDS + TIMEOUT)) s
|
||||
while :; do
|
||||
s="$(systemctl --user is-active "$BUS_UNIT" 2>/dev/null || true)"
|
||||
s="$(sc is-active "$BUS_UNIT" 2>/dev/null || true)"
|
||||
case "$s" in inactive|failed) if stopped; then say "stopped $BUS_UNIT"; return 0; fi ;; esac
|
||||
[ "$SECONDS" -lt "$deadline" ] || return 1
|
||||
sleep 1
|
||||
@@ -1326,7 +1710,7 @@ stop_bus() {
|
||||
|
||||
start_bus() {
|
||||
START_EPOCH="$(date +%s)" && START_MS="$(date +%s%3N)" || return 1
|
||||
systemctl --user start "$BUS_UNIT"
|
||||
sc start "$BUS_UNIT"
|
||||
}
|
||||
|
||||
# mos-0848 point 3: put the phase-1 bus unit back (no --pm), restart it and rerun the phase-1 gate.
|
||||
@@ -1348,8 +1732,8 @@ pm_on_fail() {
|
||||
exit 1
|
||||
fi
|
||||
local pwhy="$GATE_WHY"
|
||||
systemctl --user stop "$CONSOLE_UNIT" || true
|
||||
systemctl --user stop "$BUS_UNIT" || true
|
||||
sc stop "$CONSOLE_UNIT" || true
|
||||
sc stop "$BUS_UNIT" || true
|
||||
say "pm-on-down: pm-on $code, phase 1 $pcode"
|
||||
log_line result pm-on-down code "$code" why "${why:-@null}" phase1 "$pcode" phase1Why "${pwhy:-@null}"
|
||||
echo "stack-release: pm-on-down: pm-on failed with $code${why:+ ($why)} and the phase-1 bus with $pcode${pwhy:+ ($pwhy)}; both units are stopped. Rollback set: $RS" >&2
|
||||
@@ -1359,6 +1743,7 @@ pm_on_fail() {
|
||||
# Phase 2: the bus unit alone, on the pointer's copy with --pm, gated with pm_launched.
|
||||
do_pm_on() {
|
||||
lock
|
||||
[ "$UNITS" = user ] || refuse external-pm "under --units external the PM is a flag in root's bus unit; changing it is a re-install with its own go"
|
||||
pointer_for_business
|
||||
[ "$P_PENDING" = 1 ] || refuse not-pending "the pointer is not pending; pm-on follows activate --pm-after-harness"
|
||||
local out
|
||||
@@ -1395,8 +1780,16 @@ do_status() {
|
||||
local open
|
||||
open="$(interrupted | paste -sd' ' -)"
|
||||
[ -z "$open" ] || echo "interrupted run: $open (started, no final line; Jason or Mos settles it)"
|
||||
if [ "$UNITS" = external ]; then
|
||||
local m
|
||||
local -a markers
|
||||
mapfile -t markers < <(compgen -G "$STATE/stack-rollback/*/AWAITING-UNITS" || true)
|
||||
for m in "${markers[@]}"; do
|
||||
echo "awaiting units: $(basename "$(dirname "$m")") (copy $(cat "$m" 2>/dev/null || echo unreadable))"
|
||||
done
|
||||
fi
|
||||
echo "check:"
|
||||
if scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1 | sed 's/^/ /'; then echo " result: pass"; else echo " result: FAIL"; fi
|
||||
if scripts/stack-copy.sh check "${INSTANCE_FLAGS[@]}" "${UNITS_FLAGS[@]}" --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1 | sed 's/^/ /'; then echo " result: pass"; else echo " result: FAIL"; fi
|
||||
[ -n "$P_COPY" ] || return 0
|
||||
echo "bus status:"
|
||||
"$COPIES/$P_COPY/scripts/mosaic" bus status 2>&1 | sed 's/^/ /' || true
|
||||
@@ -1722,11 +2115,16 @@ ID=""
|
||||
case "$OP" in
|
||||
activate|rehearse) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;;
|
||||
adopt|pm-on|rollback|status) [ $# -ge 1 ] || usage; BUSINESS="$1"; shift ;;
|
||||
confirm|revert|gate) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;;
|
||||
-h|--help) usage 0 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
[[ "$BUSINESS" =~ ^[a-z0-9][a-z0-9-]*$ ]] || usage
|
||||
parse_opts "$@"
|
||||
# Row 232: the stack-copy.sh check flags and the systemctl the instance-busy message names.
|
||||
UNITS_FLAGS=()
|
||||
SC_NAME="systemctl --user"
|
||||
if [ "$UNITS" = external ]; then UNITS_FLAGS=(--units external); SC_NAME=systemctl; fi
|
||||
COPIES_DIR="$COPIES"
|
||||
instance_select
|
||||
load_config
|
||||
@@ -1741,6 +2139,7 @@ STORE_SCHEMA="" START_EPOCH="" START_MS=0 GATE="" GATE_WHY="" S_CHANGED=""
|
||||
P_COPY="" P_PM=0 P_PENDING=0 P_SCHEMA="" P_CLAUDE="" P_CONSOLE=() RUN_PM=0 RUN_PENDING=0 RUN_CONSOLE=() DROP_TESTER="" PROOF_PLAN="" PROOF_STEP="" SEND_PLAN="" SEND_STEP="" PLAN_REFUSE=()
|
||||
BASE_CONSOLE=() SERVE_FLAGS=() SSO_PLAN="" SSO_STEP="" SSO_DROP="" SSO_NO_TARGET=""
|
||||
R="" R_RC=0 R_EARLY=() R_BK="" R_HOST_PID="" R_CONSOLE_PID="" R_STARTED="" LIVE_CFG="" LIVE_BIZ=""
|
||||
O_ACT="" FLOOR=0
|
||||
case "$OP" in
|
||||
adopt) do_adopt ;;
|
||||
activate) do_activate ;;
|
||||
@@ -1748,4 +2147,7 @@ case "$OP" in
|
||||
rollback) do_rollback ;;
|
||||
status) do_status ;;
|
||||
rehearse) do_rehearse ;;
|
||||
confirm) do_confirm ;;
|
||||
revert) do_revert ;;
|
||||
gate) do_gate ;;
|
||||
esac
|
||||
|
||||
Reference in New Issue
Block a user