Lands Darkwing's round 1 candidate, manifest 464ce9eb177a (1a1dc80caon darkwing's branch), approved by Filbert (29922, rev 2659). Seven paths: scripts/stack-upgrade.sh (new), scripts/stack-release.sh, docs/guides/host-run-sheet.md (new), docs/TOOLS.md, packages/cli/tests/stack-upgrade.test.mjs (new), packages/cli/tests/stack-release.test.mjs and agents/darkwing/work/queue-233/BUILD.md. Evidence: Darkwing's gate ond99166ade(basee65a3ab4d) 1201/0, mutants m1-m8 8 of 8 killed. The N1 fix08db6ac3f(flock sleep leak in stack-upgrade.test.mjs) reran c0 20/0 with m1 killed (sage-1708); BUILD.md doesn't record that rerun. Sincee65a3ab4dno commit touches the seven paths. Rows 241 and 242 changed bus and cli files. Only cli tests read the seven paths, so Sage ran the cli suite on5465fd52dplus the seven paths: 617 pass, 0 fail (out233/pkg-cli.log, 05:26:33Z to 05:35:00Z). A combined gate with row 234 failed bus schema.test.mjs:522 on 234's new tables, and Gizmoduck found a second 234 failure (stack-release.test.mjs T46), so Sage stopped it. Row 234 round 3 fixes both and rebuilds on this landing, since its T46 lines go in a file this commit changes. Filbert's N1, N4 and N5 go to row 240 (briefb00d7d94d). N2 and N3 are in DEFERRED.md.
This commit is contained in:
@@ -0,0 +1,249 @@
|
||||
# Row 233 (I3): root's upgrade driver and the host run sheet
|
||||
|
||||
Brief: `docs/plans/2026-10-20_installer-rows.md` section I3, blob `e61291bcf1c6` (commit a0e3d7cfd), re-pinned
|
||||
by Sage (sage-1685, rev 2631) from `f35e7e57daee` (ad6f0be7a). The later blob only appends I4; the I3 text
|
||||
is the same. Gitea #1735, Vikunja P19
|
||||
#432. Owner Darkwing, reviewer Filbert, gate owner Sage. Design input: row 195 DESIGN r6,
|
||||
`agents/darkwing/work/queue-195/DESIGN.md` at b7239bb66, sections 2, 4 and 7. Where the brief and r6
|
||||
differ, the brief wins. Sage's rulings during the build: sage-1650 and sage-1652 (`--backup last`, the
|
||||
`--no-rehearsal` mapping, `--rehearse-port`, hostcheck), and sage-1653 (`from_log newest`, the two external
|
||||
plan refusals, the `BACKUP_STORE` line).
|
||||
|
||||
## Files
|
||||
|
||||
Base: e65a3ab4d, row 232's landing, on `darkwing/row233-r1`. The build started on ea1e85c7b
|
||||
(`darkwing/row233`, row 232's candidate e6ca230d2 and its BUILD.md) before row 232 landed. The three row 232
|
||||
files this row changes are identical on the two bases, so the uncommitted diff moved across unchanged.
|
||||
|
||||
- `scripts/stack-upgrade.sh` (new): `upgrade` and `rollback`, root's side of r6 A1-A5, A2f and R1-R4.
|
||||
- `packages/cli/tests/stack-upgrade.test.mjs` (new, 20 tests).
|
||||
- `docs/guides/host-run-sheet.md` (new).
|
||||
- `docs/TOOLS.md`: a new subsection "Root's upgrade driver (`scripts/stack-upgrade.sh`, row 233)", plus the
|
||||
row 232 lines below.
|
||||
- Row 232 files, changed on Sage's rulings (the brief's files list doesn't name them):
|
||||
- `scripts/stack-release.sh`:
|
||||
- `rehearse --backup last` takes the newest backup the log records (`from_log newest`) and prints
|
||||
`PASS backup-last`. With none it fails `no-backup`, exits 3 and makes no rehearsal directory.
|
||||
A newest backup that isn't an activation id aborts `backup-unreadable`, exit 1.
|
||||
- The rehearsal prints `BACKUP_STORE <activation id> schema <version>` before its migration step.
|
||||
- `--plan` under `--units external` refuses `open-activation` when any rollback set holds
|
||||
`AWAITING-UNITS`, and `migrate-required` for a migrating target without `--migrate`. Root reads
|
||||
only the plan's exit, so these two have to be refusals there. User mode prints the same bytes.
|
||||
- `packages/cli/tests/stack-release.test.mjs`: three new `row 233:` tests for `--backup last`.
|
||||
- `docs/TOOLS.md`: `--backup ACTIVATION|PATH|last` in the rehearse usage, a paragraph on `last`, and a
|
||||
paragraph on the two external plan refusals.
|
||||
- This file.
|
||||
|
||||
## The brief, item by item
|
||||
|
||||
| Brief item | Where |
|
||||
|---|---|
|
||||
| Root runs from a root-owned checkout, never a copy | `head-mismatch` (A1); TOOLS.md; run sheet section 6 |
|
||||
| I2's render with `/etc/mosaic-stack/<instance>.args`, `--copy` the one changed value | `read_args`, `args_with`; the args test |
|
||||
| The args file changes only after a confirmed run | `prepare_args` before A2, `mv` after A5; sequences 1 and 7, m6 |
|
||||
| `copy --bundle` as the stack's user before A1 | the `fetch`, `ff`, `copy` steps; sequence 1's sudo keys |
|
||||
| A rehearsal before A2, a failed one refuses with nothing stopped | `rehearse`; three rehearsal tests, m4 |
|
||||
| The root lock, held to the end, rollback included | `flock -n 9`; the lock test and sequence 3's probe, m1, m2 |
|
||||
| A1 refuses `open-activation` | stack-release.sh's external plan; the AWAITING-UNITS test, m3 |
|
||||
| `migrate-required` without `--migrate` | stack-release.sh's external plan; its test, m5 |
|
||||
| The step log, 0640, one line per step | `logl`, `step`; every test checks lines against `STEP` |
|
||||
| Exit codes 0-3 | `finish`, `refuse`; exit 4 is usage |
|
||||
| duckworth-0055's notes | the next section |
|
||||
| Root parses no stack user output beyond exits | the step log test; the one exception is sage-1653's `BACKUP_STORE` |
|
||||
| The host run sheet | `docs/guides/host-run-sheet.md` |
|
||||
|
||||
## duckworth-0055's notes on r6 (brief line 169)
|
||||
|
||||
- N1, A2's stop timeout under `gate --since`. Row 232's BUILD.md left two pieces to I3. The run sheet's
|
||||
A2f line is in section 6 ("When a run doesn't exit 0", exit 2). The test is `N1: a bus that outlives
|
||||
A2's timeout keeps its old start`: a bus that ignores `stop` keeps its old `ActiveEnterTimestamp`,
|
||||
`wait-down` logs `rc=1 ... state: active`, A2f restarts and gates, the gate fails `stale-start`, and the
|
||||
driver stops both units and exits 2. Sequence 4 is the other half: an A3 exit 3 goes to A2f and exits
|
||||
1 with the old copy's gate passing.
|
||||
- N2, the claude path under `/opt/mosaic-stack`. Run sheet section 3 installs it there as root with
|
||||
managed settings. The args file's `--claude` carries it, and `case2-unit.sh` already refuses a binary
|
||||
that isn't root's or sits under the home. The driver passes the args file through and adds no claude
|
||||
check of its own. F6 is I2's.
|
||||
|
||||
## Sage's rulings during the build
|
||||
|
||||
- sage-1650 and sage-1652. Root always runs the rehearsal and maps its exit:
|
||||
- 0 goes on, or refuses `rehearsal-required` with `--no-rehearsal`.
|
||||
- 3 refuses `no-backup`, or with `--no-rehearsal` logs `rehearsal-skipped` and goes on.
|
||||
- Anything else refuses `rehearsal-failed`.
|
||||
|
||||
`--rehearse-port N`, default 7339. Hostcheck: a missing file or a nonzero `--check` refuses
|
||||
`profile-drift`.
|
||||
- sage-1653. `from_log newest`, and the external plan refusals `open-activation` and `migrate-required`.
|
||||
Residual: the `BACKUP_STORE` id and schema land in root's step log, but nothing ties that backup to the
|
||||
one the activation's own backup step takes later. A stopped backup before the upgrade would close that
|
||||
gap; it's a later row, not this one.
|
||||
|
||||
## Tests
|
||||
|
||||
`stack-upgrade.test.mjs` runs the driver as the test's user against a scratch root, with the test seams
|
||||
`STACK_UPGRADE_*`. The stand-ins are a fake `sudo` (logs its argv, checks the driver's form, then runs the
|
||||
real command as the test's user, r6's B1 pattern), a fake `case2-unit.sh` render that installs and starts
|
||||
the fixture units, a fake `systemctl` for root's system units, and a shell `hostcheck.sh`. The stack's
|
||||
user's steps run the real `stack-release.sh` and `stack-copy.sh` from a clone made from a real bundle. The
|
||||
units, copies and store are `stack-release-fixture.mjs`'s.
|
||||
|
||||
| Test | Covers |
|
||||
|---|---|
|
||||
| sequence 1 | confirm fails healthz, rollback with `--cause A5`, exit 1; every sudo call in the driver's form |
|
||||
| sequence 2 | `activate --migrate` fails at the wrapper, store restored, `--cause A3`, exit 1 |
|
||||
| sequence 3 | render fails, `--cause A4`; a probe run during `render-old` refuses `upgrade-running` |
|
||||
| sequence 4 | stage exits 3, A2f restarts and gates the old units, exit 1 |
|
||||
| N1 | a unit that won't stop: A2f's gate fails `stale-start`, exit 2, units stopped |
|
||||
| sequence 5 | confirm of the wrong copy refused, `--cause A5`, exit 1 |
|
||||
| sequence 6 | revert fails, exit 2; the next run refuses `interrupted-run` at its plan |
|
||||
| sequence 7 | two upgrades and a `rollback --to` with `--restore`; the args file after each confirm; `to-not-previous` |
|
||||
| sequence 8 | a migration that writes and fails, store back byte for byte, exit 1 |
|
||||
| lock | a second run while the lock is held: one log line, no call |
|
||||
| open-activation | any `AWAITING-UNITS` refuses at A1, nothing stopped |
|
||||
| migrate-required | refused without `--migrate`, no rehearsal, nothing stopped |
|
||||
| rehearsal-failed | the rehearsal's argv, exit 3, nothing stopped |
|
||||
| `BACKUP_STORE` | the line after a match; four malformed lines log `BACKUP_STORE none` |
|
||||
| no-backup | the real rehearsal with no backup refuses; `--no-rehearsal` logs the skip and goes on |
|
||||
| backup-last | the real rehearsal names the newest backup; `--no-rehearsal` with one refuses `rehearsal-required` |
|
||||
| args file | group write, other write, two `--copy`, an unknown flag, a symlink, `args-differ` |
|
||||
| hostcheck | nonzero `--check` and a missing file refuse `profile-drift` before the plan |
|
||||
| head-mismatch | checkout not at `--sha`; a commit the clone can't fast-forward to refuses `fetch-failed` |
|
||||
| usage | twelve bad argvs exit 4 with no log and no lock |
|
||||
|
||||
The step log check (`STEP`) runs in every test that reads the log: each line must be a step line or the
|
||||
result line, and none may hold the stack user's output (`stack-release:`, `PASS `, `FAIL `, `REHEARSAL`,
|
||||
the fake rehearsal's text).
|
||||
|
||||
## Mutants
|
||||
|
||||
`~/darkwing-scratch/r233/mutants.sh`, not committed because it edits the worktree. It has the brief's six
|
||||
mutants and two for sage-1650's mapping. The method is row 232's: a checked backup, exact replacements, a
|
||||
syntax check, the named killers, then the file put back with its sha256 checked. One control runs first:
|
||||
the whole `stack-upgrade.test.mjs` with no change. `DRY=1` matched each replacement once on the candidate
|
||||
(2026-10-21).
|
||||
|
||||
| Mutant | File | Killers |
|
||||
|---|---|---|
|
||||
| m1 drop the flock | stack-upgrade.sh | `a second run while the lock is held`, `sequence 3:` |
|
||||
| m2 release the lock when the rollback starts | stack-upgrade.sh | `sequence 3:` |
|
||||
| m3 skip A1's open-activation check | stack-release.sh | `A1 refuses open-activation` |
|
||||
| m4 skip the rehearsal | stack-upgrade.sh | `the rehearsal runs after the plan`, `sequence 1:` |
|
||||
| m5 accept a migrating target without `--migrate` | stack-release.sh | `migrate-required` |
|
||||
| m6 write the args file before confirm | stack-upgrade.sh | `sequence 7:`, `sequence 1:` |
|
||||
| m7 `--no-rehearsal` accepted with a backup | stack-upgrade.sh | `refuses rehearsal-required` |
|
||||
| m8 drop the `rehearsal-skipped` line | stack-upgrade.sh | `a real rehearsal with no backup` |
|
||||
|
||||
## Deviations
|
||||
|
||||
D1. The brief has A1 refuse `open-activation` for an open `AWAITING-UNITS` or a non-empty `interrupted`
|
||||
set. The interrupted case refuses with row 232's existing code, `interrupted-run`; only `AWAITING-UNITS`
|
||||
gives `open-activation`. Both reach root as `plan-refused`, exit 3 (sequence 6, the open-activation test).
|
||||
|
||||
D2. Root copies one line of the stack user's output into its log, `BACKUP_STORE`, on sage-1653. It
|
||||
matches a fixed pattern first, takes the last match, logs `BACKUP_STORE none` otherwise, and decides
|
||||
nothing on it.
|
||||
|
||||
D3. The stack user's steps carry no `--instance`. The driver takes `--instance dev` only, row 190's default
|
||||
instance, which the stack user's commands select with no flag. A named instance's units are a later row.
|
||||
|
||||
D4. Most tests fake the rehearsal in the fake `sudo`: the fixture's stub copies can't pass a real one. Two
|
||||
tests run the real rehearsal (`faults/real-rehearse`): no-backup and backup-last. `stack-release.test.mjs`
|
||||
covers `--backup last` itself.
|
||||
|
||||
D5. Any nonzero `--plan` exit refuses `plan-refused`, exit 3, whatever the code. The stack user's output
|
||||
above it names the refusal; root doesn't parse it.
|
||||
|
||||
D6. `rollback --restore ACT` is accepted without `--events`, as `stack-release.sh rollback` accepts it.
|
||||
The brief's usage pairs them. `--events` without `--restore` is a usage error.
|
||||
|
||||
D7. The refusal of any `STACK_UPGRADE_*` seam when run as root has no test, because the tests don't run
|
||||
as root. It is three lines at the top of the script.
|
||||
|
||||
D8. The fake `sudo` keeps the test's environment, where real `sudo` resets it. The driver passes
|
||||
everything the stack user needs through `env PATH=...` and `-H`, and the fake checks that form.
|
||||
|
||||
D9. Exit 3 means nothing stopped, not nothing changed. A refusal after `fetch` leaves the stack user's
|
||||
clone fast-forwarded and the new copy made. Both are the user's own state, and the next run reuses them.
|
||||
TOOLS.md and the run sheet say so.
|
||||
|
||||
D10. The brief's usage has neither `--no-rehearsal` nor `--rehearse-port`. Both come from sage-1650 and
|
||||
sage-1652.
|
||||
|
||||
## Decisions
|
||||
|
||||
- `rollback` has no HEAD check. Its target is an older copy than root's checkout, which already exists.
|
||||
It fetches, copies and rehearses nothing (sequence 7).
|
||||
- The new args file is written next to the old one (`.dev.args.XXXXXX`, same mode) before A2, so a full
|
||||
`/etc` refuses before anything stops. It moves into place after A5. A2f and the rollback remove it; a failed move keeps it and names it.
|
||||
- Every stack user step gets `--copies` explicitly, so the copies directory comes from the args file's
|
||||
`--home`, never from the user's environment.
|
||||
- The stack's user does the fetch, the fast-forward and the copy, so root never runs git on a tree that
|
||||
user can write.
|
||||
- A4 is the render, and `case2-unit.sh` starts the bus, then the Console. The forward path has no
|
||||
`systemctl start` of its own; only A2f does.
|
||||
|
||||
## Run environment
|
||||
|
||||
Worktree `/mnt/storage/scratch/darkwing-r233`, `TMPDIR=/mnt/storage/scratch/tmp/r233`, node from PATH. One
|
||||
whole `stack-upgrade.test.mjs` run takes about 56 s.
|
||||
|
||||
## Results
|
||||
|
||||
### Singles, run 1, 2026-10-21, Sage's slot (sage-1689)
|
||||
|
||||
`~/darkwing-scratch/r233/singles.sh` on `darkwing/row233-r1` (base e65a3ab4d, uncommitted tree), 03:48:20Z
|
||||
to 03:59:00Z. These are the test files that read a changed script. `stack-instances.test.mjs` reads
|
||||
`stack-release.sh`, so it joined the three named in the plan.
|
||||
|
||||
| File | Result |
|
||||
|---|---|
|
||||
| `stack-upgrade.test.mjs` | 20 pass, 0 fail |
|
||||
| `stack-release.test.mjs` | 194 pass, 1 fail |
|
||||
| `stack-release-external.test.mjs` | 25 pass, 0 fail |
|
||||
| `stack-instances.test.mjs` | 23 pass, 0 fail |
|
||||
|
||||
The failure was my test, not the script. "row 233: --backup last takes the newest backup..." compares
|
||||
`untouched(e)` before and after the run, and `untouched` includes `existsSync(e.log)`. The test took its
|
||||
before snapshot ahead of `rlog()`, which writes the log, so that flag went from false to true. Every hash in
|
||||
the snapshot matched. The fix takes the snapshot after `rlog()`. No other row 233 test calls `untouched`.
|
||||
|
||||
### Singles, run 2
|
||||
|
||||
`~/darkwing-scratch/r233/rerun.sh` on aacfa2d7a, Sage's slot (sage-1695), 04:13:05Z to 04:21:28Z:
|
||||
`stack-release.test.mjs` whole, 195 pass, 0 fail, 0 skipped. The fixed test passed. The other three files
|
||||
did not change after run 1. The gate and the mutants run next, in Sage's slot.
|
||||
|
||||
### Gate and mutants, 2026-10-21, Sage's slot (sage-1704)
|
||||
|
||||
`~/darkwing-scratch/r233/gate.sh` then `mutants.sh`, chained behind `busy.sh`, on d99166ade (code
|
||||
aacfa2d7a; d99166ade changes only this file). Counts are an awk sum over `gate-r1.out`.
|
||||
|
||||
| Part | Time (UTC) | Result |
|
||||
|---|---|---|
|
||||
| cli suite (`node --test tests/*.test.mjs`) | 04:29:01-04:37:27 | 616 pass, 0 fail, 0 skipped |
|
||||
| the 10 `scripts/test-*.sh` | 04:37:27-04:39:38 | selftests 429 passed, 0 failed; `test-queue`'s node part 156 pass, 0 fail |
|
||||
| gate total | | 1201 pass, 0 fail, 11 suites, `GATE_PASSED` |
|
||||
| c0 control | 04:39:38-04:40:35 | exit 0 |
|
||||
| m1-m8 | 04:40:35-04:41:25 | 8 of 8 killed, every one exit 1 |
|
||||
|
||||
Every restore passed its sha256 check, and `git diff` was clean at d99166ade afterwards. The EXIT trap
|
||||
removed the `darkwing-r233_default` network.
|
||||
|
||||
N1, found in the process sweep after the run: a `sleep 60` was left with its cwd in `packages/cli`.
|
||||
`stack-upgrade.test.mjs:312` spawns `flock LOCK sleep 60`, and its `t.after` SIGKILLs only the flock, so
|
||||
flock's child outlives the test by up to 60 s. It exited on its own. Row 232's T20
|
||||
(`stack-release.test.mjs:370`) spawns its holder `detached` and kills the group with `process.kill(-pid)`.
|
||||
The fix is the same two lines. That test is one of m1's killers.
|
||||
|
||||
Sage ruled it a fix now (sage-1707), and 08db6ac3f makes it, test only. After the gate's d99166ade
|
||||
the only change outside this file is that test:
|
||||
|
||||
```
|
||||
$ git diff --stat d99166ade 08db6ac3f
|
||||
packages/cli/tests/stack-upgrade.test.mjs | 4 ++--
|
||||
1 file changed, 2 insertions(+), 2 deletions(-)
|
||||
```
|
||||
|
||||
The gate stands on Sage's ruling. The rerun is the whole `stack-upgrade.test.mjs` singly plus m1, in
|
||||
Sage's slot, with a sweep for a leftover `sleep` afterwards.
|
||||
+82
-2
@@ -669,7 +669,7 @@ pinned copy](../packages/cli/README.md#one-pinned-copy-row-85-1579)).
|
||||
#### Rehearsal on a scratch root (D2, row 163)
|
||||
|
||||
```
|
||||
scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]
|
||||
scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH|last --port N [--timeout S]
|
||||
```
|
||||
|
||||
`rehearse` starts the target copy's bus host and Console on a copy of a
|
||||
@@ -680,7 +680,11 @@ runs it against live data. It stops and starts no unit and writes no pointer
|
||||
or log line.
|
||||
|
||||
`--backup` takes an activation id under `<dataRoot>/backups/stack/` or an
|
||||
absolute path. `--port` is the scratch Console's loopback port, 1024 to
|
||||
absolute path. `--backup last` (row 233) takes the newest backup the
|
||||
activation log records and prints `PASS backup-last` naming it. With none in
|
||||
the log it fails `no-backup`, exits 3 and makes no directory. After the
|
||||
backup's checks it prints `BACKUP_STORE <activation id> schema <version>`.
|
||||
`--port` is the scratch Console's loopback port, 1024 to
|
||||
65535. The run refuses `port-reserved` for 7332, 7333 and the pointer's
|
||||
`--port` and `--tester-port`, and `port-in-use` for a port something holds;
|
||||
it never picks another port. `--timeout` (default 60 seconds) bounds the
|
||||
@@ -779,6 +783,82 @@ instance (the business's `launch.by`), for this business, whose run started at
|
||||
or after the floor. `status` adds an
|
||||
`awaiting units:` line for an open activation.
|
||||
|
||||
Under `--units external`, `activate --plan` also refuses `open-activation`
|
||||
while any rollback set holds `AWAITING-UNITS`, and `migrate-required` when
|
||||
the target migrates the store and the run lacks `--migrate` (row 233). Under
|
||||
`--units user` the plan's bytes are unchanged.
|
||||
|
||||
#### Root's upgrade driver (`scripts/stack-upgrade.sh`, row 233)
|
||||
|
||||
```
|
||||
scripts/stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
|
||||
[--rehearse-port N] [--timeout S]
|
||||
scripts/stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
|
||||
```
|
||||
|
||||
Root runs it from a root-owned checkout of the reviewed commit, never from a
|
||||
copy, because the stack's user can write the copies. Each run needs Mos's go,
|
||||
and `--migrate` needs Jason present (decision 96). The host must be prepared
|
||||
first (`docs/guides/host-run-sheet.md`). Only the default instance, `dev`, is
|
||||
accepted; a named instance's units are a later row.
|
||||
|
||||
The unit values come from `/etc/mosaic-stack/dev.args`, which I2's install
|
||||
writes: one `case2-unit.sh` argument per line, the serve flags after `--`,
|
||||
owned by root with no group or other write, exactly one `--copy`, and that
|
||||
copy the one both installed units run. Otherwise the run refuses
|
||||
`args-unreadable` or `args-differ`. The driver writes the new copy id into
|
||||
that file only after `confirm` passes.
|
||||
|
||||
`upgrade`, in order. Every stack user step runs as
|
||||
`sudo -u USER -H env PATH=<node dir>:/usr/local/bin:/usr/bin:/bin` from the
|
||||
user's clone `<home>/src/mosaic-stack`:
|
||||
|
||||
1. Takes `flock -n` on `/run/mosaic-stack/upgrade-dev.lock` and holds it to
|
||||
the end, rollback included. A second run refuses `upgrade-running`, and
|
||||
a lock it can't open refuses `lock-failed`.
|
||||
2. Fetches the bundle's `refactor` into the clone and fast-forwards it to
|
||||
SHA (`fetch-failed`), then `stack-copy.sh copy --bundle FILE SHA`
|
||||
(`copy-failed`).
|
||||
3. A1: this checkout's HEAD is SHA (`head-mismatch`), the copy exists
|
||||
(`copy-missing`), `/usr/local/lib/mosaic-stack/hostcheck.sh --check`
|
||||
exits 0 (`profile-drift`, row 240), both units name one copy
|
||||
(`units-unreadable`, `units-disagree`) and it is the args file's
|
||||
(`args-differ`), and
|
||||
`activate SHA BUSINESS --units external --plan --expect-from OLD` exits 0
|
||||
(`plan-refused`, which covers `open-activation`, `migrate-required` and
|
||||
`pointer-differs`).
|
||||
4. The rehearsal, `rehearse SHA BUSINESS --backup last --port 7339`
|
||||
(`--rehearse-port`). Exit 0 goes on. Exit 3 means no backup in the log
|
||||
and goes on only with `--no-rehearsal`, logging `rehearsal-skipped`;
|
||||
without the flag it refuses `no-backup`. Exit 0 with `--no-rehearsal`
|
||||
refuses `rehearsal-required`; any other exit refuses `rehearsal-failed`.
|
||||
5. A2 stops the Console, then the bus, and waits for both to read inactive
|
||||
or failed. A3 is `activate --units external`, A4 the render with the new
|
||||
`--copy` (which also starts the bus, then the Console), A5 `confirm`.
|
||||
The driver writes the new args file beside the old one before A2
|
||||
(`args-failed`, exit 3) and moves it into place after A5. A failed move
|
||||
exits 1 `args-failed`: the new copy is live and confirmed, and root puts
|
||||
the file in place by hand.
|
||||
|
||||
A failure in A2 or a stage that exits 3 starts the old units again and gates
|
||||
them (A2f). A later failure stops both units, runs `revert`, renders the old
|
||||
copy and runs `confirm --rollback OLD --cause A3|A4|A5`. `rollback` is the
|
||||
same from A1 on, with `rollback BUSINESS --to ID`, no fetch, no copy and no
|
||||
rehearsal; `--restore` and `--events` pass through.
|
||||
|
||||
Root reads no output the stack user wrote except exit codes and the
|
||||
rehearsal's `BACKUP_STORE` line, which it copies into its log after a match
|
||||
against a fixed pattern (`BACKUP_STORE none` otherwise). The step log
|
||||
`/var/log/mosaic-stack/upgrade-dev.log`, 0640, gets one
|
||||
`<time> <step> rc=N argv: ...` line per step and a last
|
||||
`<time> result exit=N <code> <text>` line.
|
||||
|
||||
Exit 0 ok, 1 failed with the old stack running and gated, 2 rollback-failed
|
||||
with both units stopped, 3 refused with nothing stopped, 4 usage. A refusal
|
||||
after step 2 leaves the user's clone fast-forwarded and the new copy made. After exit
|
||||
2 the activation stays open, so the next run's plan refuses until root
|
||||
finishes it.
|
||||
|
||||
## Console service (`scripts/webui-service.sh`)
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,240 @@
|
||||
# Host run sheet: preparing a host for the stack
|
||||
|
||||
ops-01 runs this on a fresh host before install (I2), and once per host for enrollment. Jason places the
|
||||
credentials in section 9. Brief: `docs/plans/2026-10-20_installer-rows.md`, I3 (row 233, #1735). The
|
||||
host design is Darkwing's r6 (`agents/darkwing/work/queue-195/DESIGN.md`, section 2) as the brief
|
||||
changes it.
|
||||
|
||||
The hosts are .53 (profile `factory`) and .54 (profile `library`). Both get every section unless a
|
||||
section says otherwise. Every command runs as root unless it starts with `sudo -u mosaic`.
|
||||
|
||||
Each item names the `scripts/case2-precheck.sh` line that verifies it. Install's precheck runs as the
|
||||
stack's user with `--business ID --profile factory|library`. Where no precheck line checks an item, the
|
||||
item says "precheck line missing", and Gosalyn adds the line to I2. Today's precheck (C2-13a) has these
|
||||
gaps; C2-13b adds the library lines (`library.sync`, persona modes, no Discord binding).
|
||||
|
||||
## 1. The stack's user and its home
|
||||
|
||||
```sh
|
||||
useradd --system --user-group --home-dir /srv/mosaic --create-home --shell /usr/sbin/nologin mosaic
|
||||
chmod 0700 /srv/mosaic
|
||||
install -d -o root -g root -m 0755 /etc/mosaic-stack
|
||||
install -d -o root -g root -m 0750 /var/log/mosaic-stack
|
||||
```
|
||||
|
||||
`mosaic` gets no sudo rule and no supplementary group. Check with `sudo -l -U mosaic` (expect "not allowed
|
||||
to run sudo") and `id -G mosaic` (one number).
|
||||
|
||||
Precheck lines:
|
||||
- `ok HOME is /srv/mosaic`
|
||||
- `ok sudo -n true fails as mosaic`
|
||||
- `ok id -G prints one group (<gid>)`
|
||||
- `/srv/mosaic` at 0700: precheck line missing.
|
||||
|
||||
## 2. node, pinned under /opt
|
||||
|
||||
The pin is the version and sha256 the window names. `<v>` below is that version, for example `v24.x.y`.
|
||||
|
||||
```sh
|
||||
install -d -o root -g root -m 0755 /etc/mosaic-stack/node-keys /opt/mosaic-stack
|
||||
cd "$(mktemp -d)"
|
||||
curl -fsSLO https://nodejs.org/dist/<v>/node-<v>-linux-x64.tar.xz
|
||||
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt
|
||||
curl -fsSLO https://nodejs.org/dist/<v>/SHASUMS256.txt.asc
|
||||
gpg --no-default-keyring --keyring /etc/mosaic-stack/node-keys/pubring.kbx --verify SHASUMS256.txt.asc SHASUMS256.txt
|
||||
grep " node-<v>-linux-x64.tar.xz\$" SHASUMS256.txt | sha256sum -c -
|
||||
echo "<pinned sha256> node-<v>-linux-x64.tar.xz" | sha256sum -c -
|
||||
tar -xJf node-<v>-linux-x64.tar.xz --no-same-owner
|
||||
mv node-<v>-linux-x64 /opt/mosaic-stack/node-<v>
|
||||
chown -R root:root /opt/mosaic-stack/node-<v>
|
||||
chmod -R go-w /opt/mosaic-stack/node-<v>
|
||||
```
|
||||
|
||||
The keyring holds the Node release keys from the nodejs/release-keys repository, nothing else. Stop on any
|
||||
failed check; don't unpack. Rollback: `rm -r /opt/mosaic-stack/node-<v>`.
|
||||
|
||||
Precheck lines:
|
||||
- `ok node <v>` (version 24 or later only).
|
||||
- The `--node` path in the args file must be a root-owned regular file outside the home. `case2-unit.sh`
|
||||
refuses otherwise at render, which is not a precheck line.
|
||||
- The installed version equals the pin: precheck line missing.
|
||||
|
||||
## 3. claude, factory only (.53)
|
||||
|
||||
Root installs it under `/opt`, never under the stack's home (this replaces r6's `/srv/mosaic/.local/bin`).
|
||||
The pin is 2.1.304 unless the window names a newer one.
|
||||
|
||||
```sh
|
||||
install -d -o root -g root -m 0755 /opt/mosaic-stack/claude-<pin> /etc/claude-code
|
||||
npm install --prefix /opt/mosaic-stack/claude-<pin> @anthropic-ai/claude-code@<pin>
|
||||
chown -R root:root /opt/mosaic-stack/claude-<pin>
|
||||
printf '%s\n' '{"env":{"DISABLE_AUTOUPDATER":"1"}}' > /etc/claude-code/managed-settings.json
|
||||
chown root:root /etc/claude-code/managed-settings.json
|
||||
chmod 0644 /etc/claude-code/managed-settings.json
|
||||
```
|
||||
|
||||
Rollback: `rm -r /opt/mosaic-stack/claude-<pin> /etc/claude-code/managed-settings.json`.
|
||||
|
||||
The args file's `--claude` names the binary under `/opt/mosaic-stack/claude-<pin>`. `case2-unit.sh`
|
||||
refuses one that isn't a root-owned regular file outside the home.
|
||||
|
||||
Precheck lines:
|
||||
- `ok claude replaces its prompt with --system-prompt-file (<n> input tokens)`. It runs the claude the bus
|
||||
unit's `MOSAIC_CLAUDE_BIN` names, so it needs the model credential from section 9.
|
||||
- Managed settings present with the autoupdater off: precheck line missing.
|
||||
|
||||
## 4. bubblewrap, AppArmor and the drift stop
|
||||
|
||||
Both hosts. These are r6's items, unchanged.
|
||||
|
||||
```sh
|
||||
apt-get install bubblewrap=0.9.0-1ubuntu0.3
|
||||
stat -c '%U:%G %a' /usr/bin/bwrap > /etc/mosaic-stack/bwrap.orig # expect root:root 755
|
||||
|
||||
cd "$(mktemp -d)"
|
||||
apt-get download apparmor-profiles=4.0.1really4.0.1-0ubuntu0.24.04.9
|
||||
dpkg-deb --fsys-tarfile apparmor-profiles_*.deb | tar -x ./usr/share/apparmor/extra-profiles/bwrap-userns-restrict
|
||||
echo "11d39094f044f0cda0febb3ad517b830301da6b2ce929664af09ee9e4dd264f9 usr/share/apparmor/extra-profiles/bwrap-userns-restrict" | sha256sum -c -
|
||||
install -o root -g root -m 0644 usr/share/apparmor/extra-profiles/bwrap-userns-restrict /etc/apparmor.d/bwrap-userns-restrict
|
||||
apparmor_parser -r /etc/apparmor.d/bwrap-userns-restrict
|
||||
|
||||
dpkg-statoverride --update --add root mosaic 0750 /usr/bin/bwrap
|
||||
```
|
||||
|
||||
`aa-status` then lists `bwrap` and `unpriv_bwrap` in enforce mode. Install the one profile file, never the
|
||||
whole `apparmor-profiles` package, whose 20 other profiles would load at the next boot. Rollback, in
|
||||
reverse order: `dpkg-statoverride --remove /usr/bin/bwrap`, then `chown` and `chmod` to the values in
|
||||
`bwrap.orig`; `apparmor_parser -R /etc/apparmor.d/bwrap-userns-restrict`, then `rm` it;
|
||||
`apt-get remove bubblewrap`.
|
||||
|
||||
The no-new-privs test must print `bwrap//&unpriv_bwrap (enforce)`:
|
||||
|
||||
```sh
|
||||
systemd-run --wait --pipe --uid=mosaic --gid=mosaic -p NoNewPrivileges=yes -p ProtectSystem=strict \
|
||||
-p ProtectHome=yes -p PrivateTmp=yes \
|
||||
/usr/bin/bwrap --unshare-user --unshare-pid --ro-bind / / cat /proc/self/attr/current
|
||||
```
|
||||
|
||||
The P9 checks: another user's `/usr/bin/bwrap --version` fails with Permission denied, `getent group mosaic`
|
||||
lists no member, and `sysctl fs.protected_hardlinks` prints 1. `sysctl
|
||||
kernel.apparmor_restrict_unprivileged_userns` prints 1 too (Ubuntu 24.04's default); leave it, so nothing
|
||||
else on the host gains user namespaces.
|
||||
|
||||
The drift stop is row 240's (I4, #1746): `hostcheck.sh` and its pins in `/usr/local/lib/mosaic-stack/`, the
|
||||
apt hook `/etc/apt/apt.conf.d/80mosaic-hostcheck`
|
||||
(`DPkg::Post-Invoke { "/usr/local/lib/mosaic-stack/hostcheck.sh --enforce || true"; };`) and the boot unit
|
||||
`mosaic-hostcheck.service`. Install them from row 240's run sheet lines once it lands. Until then
|
||||
`stack-upgrade.sh` refuses `profile-drift` on every run, because it runs `hostcheck.sh --check` and a
|
||||
missing file refuses.
|
||||
|
||||
Precheck lines:
|
||||
- `ok bwrap --unshare-all runs`
|
||||
- The AppArmor profile, the statoverride, the no-new-privs result and the P9 checks: precheck line missing.
|
||||
A line that runs `/usr/local/lib/mosaic-stack/hostcheck.sh --check` would cover them once row 240 lands.
|
||||
|
||||
## 5. The port floor
|
||||
|
||||
Leave `net.ipv4.ip_unprivileged_port_start` at its default (1024) or higher. The Console's port is below
|
||||
it and the unit gets `CAP_NET_BIND_SERVICE`, so no other `mosaic` process can bind that port.
|
||||
|
||||
Precheck line: `ok net.ipv4.ip_unprivileged_port_start <n> is above the Console's port <port>`.
|
||||
|
||||
## 6. The release transport
|
||||
|
||||
A release is a git bundle of `refs/heads/refactor` plus the 40-character commit id (I1, mos-1118 choice 2).
|
||||
Mos's go names both, with the bundle's sha256. `<sha>` below is that id.
|
||||
|
||||
First install:
|
||||
|
||||
```sh
|
||||
# Copy the bundle in, then check it before anything reads it.
|
||||
echo "<bundle sha256> /root/<sha>.bundle" | sha256sum -c -
|
||||
git bundle verify /root/<sha>.bundle
|
||||
|
||||
# Root's checkout. stack-upgrade.sh and case2-unit.sh run from here, never from a copy.
|
||||
git clone -q -b refactor /root/<sha>.bundle /root/mosaic-stack
|
||||
git -C /root/mosaic-stack checkout -q --detach <sha>
|
||||
git -C /root/mosaic-stack rev-parse HEAD # must print <sha>
|
||||
|
||||
# The stack's user's copy of the bundle, and its own clone.
|
||||
install -d -o mosaic -g mosaic -m 0700 /srv/mosaic/incoming /srv/mosaic/src
|
||||
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
|
||||
sudo -u mosaic -H git clone -q -b refactor /srv/mosaic/incoming/<sha>.bundle /srv/mosaic/src/mosaic-stack
|
||||
```
|
||||
|
||||
Each upgrade, before `stack-upgrade.sh upgrade`:
|
||||
|
||||
```sh
|
||||
echo "<bundle sha256> /root/<sha>.bundle" | sha256sum -c -
|
||||
git bundle verify /root/<sha>.bundle
|
||||
git -C /root/mosaic-stack fetch -q /root/<sha>.bundle refs/heads/refactor:refs/remotes/origin/refactor
|
||||
git -C /root/mosaic-stack checkout -q --detach <sha>
|
||||
install -o mosaic -g mosaic -m 0600 /root/<sha>.bundle /srv/mosaic/incoming/<sha>.bundle
|
||||
/root/mosaic-stack/scripts/stack-upgrade.sh upgrade --instance dev --bundle /srv/mosaic/incoming/<sha>.bundle --sha <sha>
|
||||
```
|
||||
|
||||
The driver fetches the bundle into the stack's user's clone and makes the copy itself. It refuses
|
||||
`head-mismatch` when root's checkout isn't at `<sha>`. Usage, exits and the step log are in
|
||||
`docs/TOOLS.md`, under "Root's upgrade driver". Each run needs Mos's go, and `--migrate` needs Jason present
|
||||
(decision 96).
|
||||
|
||||
When a run doesn't exit 0, send Mos the exit and the last lines of `/var/log/mosaic-stack/upgrade-dev.log`:
|
||||
- Exit 3: refused before anything stopped. The last line names the code. The stack's user's clone and the
|
||||
new copy may already hold the new commit; the units still run the old one.
|
||||
- Exit 1: the old copy runs and passed its gate. Either the driver restarted it after A2 or a refused A3
|
||||
(A2f), or it rolled back after A3, A4 or A5.
|
||||
- Exit 2: both units are stopped and the activation stays open, so the next run refuses at its plan until
|
||||
it is finished with `confirm` or `revert`. One way here: a unit that doesn't stop within `--timeout`
|
||||
keeps its old start time, A2f's gate fails it `stale-start`, and the driver stops both (duckworth-0055
|
||||
N1). That is on purpose; a unit that won't stop is a reason to stop.
|
||||
|
||||
Precheck lines: none. Root's checkout at `<sha>` and the stack's user's clone: precheck line missing.
|
||||
After install, `stack-copy.sh check --units external` reads the copy the units run.
|
||||
|
||||
## 7. The args file
|
||||
|
||||
`/etc/mosaic-stack/dev.args` holds the `case2-unit.sh` arguments, one per line, with the serve flags after a
|
||||
line holding `--`. I2's install writes it; ops-01 doesn't write it by hand. It must be root's, with no group
|
||||
or other write (`chmod 0644` or tighter). Before `--` it takes only `--user`, `--home`, `--business`,
|
||||
`--copy` (exactly once), `--node`, `--profile`, `--claude` and `--pm`. `stack-upgrade.sh` changes only the
|
||||
`--copy` line, and only after confirm passes.
|
||||
|
||||
Precheck line: none. `stack-upgrade.sh` refuses `args-unreadable` on any of these rules.
|
||||
|
||||
## 8. Enrollment, once per host
|
||||
|
||||
The machine-grant enrollment is Gizmoduck's design r6, commit f9039682e, section 3, steps E1-E4. ops-01's
|
||||
Q-O1 is answered (mos-1125). The host side is the `tools/host/` files: `mosaic-keys`, `mosaic-keys-write`,
|
||||
their sudoers line and the sshd drop-in `/etc/ssh/sshd_config.d/<nn>-mosaic-keys.conf`. The verbs that run
|
||||
it (`hosts.enroll`, the `host-enroll` decision) are rows 234 and 235, not yet on refactor. This item has no
|
||||
commands until they land.
|
||||
|
||||
## 9. Credential slots
|
||||
|
||||
Jason places these, all in one sitting per host. No value appears here or in any message. Check a placed
|
||||
file with `stat -c '%U %a %n' PATH`, never `cat`. `<business>` is the host's business id.
|
||||
|
||||
### .53, factory
|
||||
|
||||
| Path | File | Mode | Owner | Profile | Precheck line |
|
||||
|---|---|---|---|---|---|
|
||||
| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `CLAUDE_CODE_OAUTH_TOKEN=...` (Q185) | 0600 | mosaic | factory | `claude replaces its prompt with --system-prompt-file` runs claude with it. The file's mode and owner: precheck line missing. |
|
||||
| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
|
||||
|
||||
### .54, library
|
||||
|
||||
| Path | File | Mode | Owner | Profile | Precheck line |
|
||||
|---|---|---|---|---|---|
|
||||
| `/srv/mosaic/.config/mosaic-dev/secrets/` | `claude-session.env`, one line `ZAI_API_KEY=...` | 0600 | mosaic | library | `<path> sets ZAI_API_KEY` |
|
||||
| `/srv/mosaic/.config/mosaic-dev/secrets/` | the Console's proof file, the path the args file's `--proof-file` names | 0600 or 0400 | mosaic | library | none. `case2-unit.sh` refuses at render. Precheck line missing. |
|
||||
| `/srv/mosaic/.config/mosaic-dev/secrets/<business>/` | `library-gitea` | 0600 | mosaic | library | C2-13b's `library.sync` (not landed) |
|
||||
| `/srv/mosaic/.config/mosaic-dev/secrets/agents/<business>/<agent>/` | one file per service | 0600, dirs 0700 | mosaic | both | none. Filled only by `mosaic credentials issue`, or Approve after #436. Never by hand. |
|
||||
|
||||
The installer key, `<configDir>/secrets/hosts/<business>/<hostId>/installer`, is not a slot anyone places.
|
||||
Enrollment (section 8) writes it under the enrolling stack's config directory and deletes it when the host
|
||||
leaves.
|
||||
|
||||
## 10. Console start
|
||||
|
||||
None by hand. I2's `case2-unit.sh` enables both units, starts the bus, then the Console (Q189, mos-1122).
|
||||
`stack-upgrade.sh` stops and starts both units itself.
|
||||
@@ -2062,6 +2062,49 @@ test('R15 a rehearsal stopped by TERM, INT or HUP fails signal and still stops t
|
||||
}
|
||||
});
|
||||
|
||||
// --- Row 233 (#1735): --backup last, the newest backup the log records (sage-1650, sage-1653) ---
|
||||
|
||||
// The log's lines, as the runs write them, with only the fields from_log reads set.
|
||||
function rlog(e,...ls){
|
||||
mkdirSync(e.state,{recursive:true});
|
||||
writeFileSync(e.log,ls.map(([op,to,backup],i)=>JSON.stringify({at:`2026-10-20T00:00:0${i}Z`,op,from:null,to,backup,activation:`20261020T00000${i}Z-${to.slice(0,12)}`,result:'ok'})+'\n').join(''));
|
||||
}
|
||||
|
||||
test('row 233: --backup last takes the newest backup the log records; a later line with backup null does not hide it, and the store line names it',async(t)=>{
|
||||
const e=await rmk(t);
|
||||
// An older backup that isn't on disk: a run that took it would fail backup-missing.
|
||||
rlog(e,['activate',C1,`20261019T000000Z-${C1.slice(0,12)}`],['activate',B,e.act],['rollback',B,null]);
|
||||
const l0=readFileSync(e.log,'utf8'),before=untouched(e);
|
||||
const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','10');
|
||||
assert.equal(r.code,0,r.out+r.err);
|
||||
assert.deepEqual(r.names,['PASS backup-last',...PRE,'PASS migration','PASS gate','PASS no-session',...POST]);
|
||||
assert.ok(r.out.startsWith(`PASS backup-last: the log's newest backup is ${e.act}\n`),r.out);
|
||||
assert.ok(r.out.includes(`\nBACKUP_STORE ${e.act} schema 3g\n`),r.out);
|
||||
assert.equal(readFileSync(e.log,'utf8'),l0);assert.equal(untouched(e),before);
|
||||
// The rehearsal's own steps.log has the PASS and FAIL lines only.
|
||||
assert.deepEqual(lines(join(r.dir,'steps.log')).map((l)=>l.replace(/^\S+ /,'')),r.out.split('\n').filter((l)=>/^(PASS|FAIL) |^REHEARSAL_/.test(l)));
|
||||
});
|
||||
|
||||
test('row 233: --backup last with no backup in the log, or no log, exits 3 with no-backup and makes no rehearsal directory',async(t)=>{
|
||||
const e=await rmk(t);
|
||||
for(const how of ['no log','adopt only']){
|
||||
if(how==='adopt only')rlog(e,['adopt',B,null]);
|
||||
const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,3,how+r.out+r.err);
|
||||
assert.equal(r.out,'FAIL no-backup: the log records no backup\nREHEARSAL_FAIL\n',how);
|
||||
assert.equal(r.dir,null);assert.deepEqual(rDirs(e),[],how);assert.ok(!existsSync(join(e.data,'rehearsals')),how);
|
||||
}
|
||||
});
|
||||
|
||||
test('row 233: --backup last refuses a newest backup that is not an activation id, with exit 1',async(t)=>{
|
||||
const e=await rmk(t);
|
||||
rlog(e,['activate',B,'../../etc']);
|
||||
const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','3');
|
||||
assert.equal(r.code,1,r.out+r.err);
|
||||
assert.equal(r.out,'FAIL backup-unreadable: the log\'s newest backup is not an activation id\nREHEARSAL_FAIL\n');
|
||||
assert.deepEqual(rDirs(e),[]);
|
||||
});
|
||||
|
||||
// --- Row 190 (#1688): stack instances on one host. Design and the T-I table:
|
||||
// agents/darkwing/work/queue-190/DESIGN.md. T-I1, T-I2, T-I5, T-I10 and the loader half of T-I7 are in
|
||||
// stack-instances.test.mjs; these need the units, the stand-ins and stack-release.sh. Every existing
|
||||
|
||||
@@ -0,0 +1,449 @@
|
||||
// scripts/stack-upgrade.sh (row 233, I3, #1735): root's upgrade and rollback of a stack whose units are
|
||||
// root's, driving stack-release.sh --units external (row 232). Design: r6 sections 4 and 7,
|
||||
// agents/darkwing/work/queue-195/DESIGN.md, as the I3 brief changes them; deviations:
|
||||
// agents/darkwing/work/queue-233/BUILD.md. The stand-ins and the module fixture are
|
||||
// stack-release-fixture.mjs's. Root's sudo, case2-unit.sh render and row 240 hostcheck are stand-ins
|
||||
// here; the stack's user's steps are the real stack-release.sh from the user's clone, except the
|
||||
// rehearsal, which the fixture's stub copies can't pass (its own tests are in stack-release.test.mjs).
|
||||
import test from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {execFileSync,spawn} from 'node:child_process';
|
||||
import {mkdirSync,writeFileSync,readFileSync,existsSync,readdirSync,rmSync,chmodSync,copyFileSync,symlinkSync,statSync} from 'node:fs';
|
||||
import {DatabaseSync} from 'node:sqlite';
|
||||
import {join,dirname} from 'node:path';
|
||||
import {once} from 'node:events';
|
||||
import {B,C,C1,U,COPIES,NODE,SBIN,base,fault,git,lines,logLines,mk,pointer,repo,sha,storeVersion,sysctl,units} from './stack-release-fixture.mjs';
|
||||
|
||||
// --- Root's stand-ins. Each is serialized into its own file with K, its baked paths. ---
|
||||
|
||||
// sudo -u USER -H env PATH=P CMD...: the form the driver must use, logged to sudo.log, then CMD with
|
||||
// SBIN before P. A one-shot hook arm/<key>.sh runs first and arm/<key>.id replaces CMD's ID. The
|
||||
// rehearsal prints a fixed BACKUP_STORE line and exits faults/rehearse-rc unless faults/real-rehearse.
|
||||
function fakeSudo(K){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
|
||||
const a=process.argv.slice(2),sd=process.env.SD_STATE,f=(n)=>path.join(sd,'faults',n);
|
||||
if(a[0]!=='-u'||!a[1]||a[2]!=='-H'||a[3]!=='env'||!/^PATH=/.test(a[4]||'')||a.length<6){console.error(`fake sudo: not the driver's form: ${a.join(' ')}`);process.exit(99)}
|
||||
const P=a[4].slice(5),cmd=a.slice(5);
|
||||
let key=path.basename(cmd[0]);
|
||||
if(cmd[0].endsWith('/scripts/stack-release.sh')){
|
||||
key=cmd[1];
|
||||
if(['activate','rollback'].includes(key)&&cmd.includes('--plan'))key+='-plan';
|
||||
if(key==='confirm'&&cmd.includes('--rollback'))key='confirm-rollback';
|
||||
}else if(cmd[0]==='git')key=`git-${cmd[3]}`;
|
||||
fs.appendFileSync(path.join(sd,'sudo.log'),JSON.stringify({user:a[1],path:P,key,cmd})+'\n');
|
||||
const env={...process.env,PATH:`${K.sbin}:${P}`},arm=(n)=>path.join(sd,'arm',`${key}.${n}`);
|
||||
if(fs.existsSync(arm('sh'))){const run=arm('sh')+'.ran';fs.renameSync(arm('sh'),run);cp.spawnSync('bash',[run],{env,stdio:'inherit'})}
|
||||
if(fs.existsSync(arm('id'))){cmd[2]=fs.readFileSync(arm('id'),'utf8').trim();fs.rmSync(arm('id'))}
|
||||
if(key==='rehearse'&&!fs.existsSync(f('real-rehearse'))){
|
||||
console.log(`fixture rehearse: ${cmd.slice(1).join(' ')}`);
|
||||
let line='BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g';try{line=fs.readFileSync(f('rehearse-store-line'),'utf8').replace(/\n$/,'')}catch{}
|
||||
if(line)console.log(line);
|
||||
let rc=0;try{rc=Number(fs.readFileSync(f('rehearse-rc'),'utf8'))}catch{}
|
||||
process.exit(rc);
|
||||
}
|
||||
const r=cp.spawnSync(cmd[0],cmd.slice(1),{env,stdio:'inherit'});process.exit(r.status??1);
|
||||
}
|
||||
|
||||
// case2-unit.sh: logs its argv, then installs the copy's units as the fixture's user units (its
|
||||
// WorkingDirectory under the stack's copies) and starts the bus, then the Console, as case2-unit.sh
|
||||
// does. faults/render-fails-<copy> exits 1 first; faults/render-probe-<copy> (one-shot) first runs
|
||||
// probe.sh, a second driver, and records its exit and stderr.
|
||||
function fakeRender(K){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
|
||||
const a=process.argv.slice(2),sd=process.env.SD_STATE,f=(n)=>path.join(sd,'faults',n);
|
||||
fs.appendFileSync(path.join(sd,'render.log'),JSON.stringify(a)+'\n');
|
||||
const cut=a.indexOf('--'),pre=a.slice(0,cut),serve=a.slice(cut+1),get=(k)=>pre[pre.indexOf(k)+1];
|
||||
const copy=get('--copy'),copies=path.join(get('--home'),'.local','share','mosaic-stack','copies');
|
||||
if(fs.existsSync(f(`render-probe-${copy}`))){
|
||||
fs.rmSync(f(`render-probe-${copy}`));
|
||||
const p=cp.spawnSync('bash',[path.join(sd,'probe.sh')],{encoding:'utf8'});
|
||||
fs.writeFileSync(path.join(sd,'probe.json'),JSON.stringify({code:p.status,err:p.stderr}));
|
||||
}
|
||||
if(fs.existsSync(f(`render-fails-${copy}`))){console.error(`fixture render: ${copy} fails`);process.exit(1)}
|
||||
const sh=(n,...x)=>cp.execFileSync('bash',[path.join(K.repo,'scripts',n),...x],{stdio:['ignore','ignore','inherit']});
|
||||
sh('bus-service.sh','install','--copy',copy,'--copies',copies,...(pre.includes('--pm')?['--pm']:[]),'--claude',get('--claude'));
|
||||
sh('webui-service.sh','install','--copy',copy,'--copies',copies,...serve);
|
||||
for(const u of [`mosaic-bus@${get('--business')}.service`,'mosaic-webui.service'])cp.execFileSync(path.join(K.sbin,'systemctl'),['start',u],{stdio:'inherit'});
|
||||
}
|
||||
|
||||
// Root's systemctl: the fixture's, with systemd's no-op start of an active unit, and a one-shot
|
||||
// faults/stop-noop-<unit> that leaves the unit running, as a stop past its timeout would.
|
||||
function fakeSystemctl(K){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
|
||||
const a=process.argv.slice(2),sd=process.env.SD_STATE;
|
||||
fs.appendFileSync(path.join(sd,'root-systemctl.log'),a.join(' ')+'\n');
|
||||
const noop=path.join(sd,'faults',`stop-noop-${a[1]}`);
|
||||
if(a[0]==='stop'&&fs.existsSync(noop)){fs.rmSync(noop);process.exit(0)}
|
||||
if(a[0]==='start'){let s={};try{s=JSON.parse(fs.readFileSync(path.join(sd,'units.json'),'utf8'))}catch{}if((s[a[1]]||{}).state==='active')process.exit(0)}
|
||||
const r=cp.spawnSync(path.join(K.sbin,'systemctl'),a,{stdio:'inherit'});process.exit(r.status??1);
|
||||
}
|
||||
|
||||
const FAKES=join(base,'upgrade-fakes');mkdirSync(FAKES);
|
||||
const K={sbin:SBIN,repo};
|
||||
for(const [n,fn] of [['sudo',fakeSudo],['render',fakeRender],['systemctl',fakeSystemctl]])writeFileSync(join(FAKES,n),`#!${NODE}\n(${fn})(${JSON.stringify(K)});\n`,{mode:0o755});
|
||||
writeFileSync(join(FAKES,'hostcheck'),'#!/bin/sh\necho "$*" >> "$SD_STATE/hostcheck.log"\nexit "$(cat "$SD_STATE/faults/hostcheck-rc" 2>/dev/null || echo 0)"\n',{mode:0o755});
|
||||
const BUNDLE=join(base,'fixture.bundle');git(repo,'bundle','create','-q',BUNDLE,'refactor');
|
||||
// Root's checkout of a reviewed commit: a clone of the fixture repo, detached at it.
|
||||
const checkouts=new Map();
|
||||
const checkout=(id)=>{
|
||||
if(!checkouts.has(id)){const d=join(base,`root-${id.slice(0,12)}`);git(base,'clone','-q','--no-checkout',repo,d);git(d,'checkout','-q','--detach',id);checkouts.set(id,d)}
|
||||
return checkouts.get(id);
|
||||
};
|
||||
const TO=['--timeout','3'];
|
||||
|
||||
// --- Per-test stack ---
|
||||
|
||||
// The stack's user's home: its copies (the fixture's, through a link), its clone, its bin. Both units
|
||||
// installed on `copy` and active, adopted under --units external, and the I2 args file naming it.
|
||||
async function useed(t,{copy=B}={}){
|
||||
const e=await mk(t),home=e.env.HOME;
|
||||
e.hc=join(home,'.local','share','mosaic-stack','copies');mkdirSync(dirname(e.hc),{recursive:true});symlinkSync(COPIES,e.hc);
|
||||
const bin=join(home,'.npm-global','bin');mkdirSync(bin,{recursive:true});
|
||||
writeFileSync(join(bin,'mosaic'),readFileSync(join(e.bin,'mosaic'),'utf8').replaceAll(`${e.bin}/`,`${bin}/`),{mode:0o755});
|
||||
copyFileSync(join(e.bin,'mosaic-old'),join(bin,'mosaic-old'));chmodSync(join(bin,'mosaic-old'),0o755);
|
||||
e.bin=bin;e.clone=join(home,'src','mosaic-stack');git(base,'clone','-q',repo,e.clone);
|
||||
const db=new DatabaseSync(e.store);
|
||||
db.exec('PRAGMA journal_mode=WAL');
|
||||
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
|
||||
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
|
||||
db.prepare("INSERT INTO meta VALUES ('schema_version', '3g')").run();
|
||||
for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`);
|
||||
db.close();
|
||||
const sh=(n,...a)=>execFileSync('bash',[join(e.clone,'scripts',n),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
|
||||
sh('bus-service.sh','install','--copy',copy,'--copies',e.hc,'--claude',e.claude);
|
||||
sh('webui-service.sh','install','--copy',copy,'--copies',e.hc,'--port',e.port,'--business','acme');
|
||||
sh('stack-copy.sh','wrapper',copy,'--copies',e.hc,'--bin',e.bin);
|
||||
execFileSync(join(SBIN,'systemctl'),['start',e.bus],{env:{...e.env,SD_CLOCK_OFFSET:'120'}});
|
||||
sysctl(e,'start',e.con);
|
||||
sh('stack-release.sh','adopt','acme','--units','external','--copies',e.hc,'--bin',e.bin);
|
||||
e.etc=join(e.root,'etc');mkdirSync(e.etc);e.args=join(e.etc,'dev.args');
|
||||
e.argsText=`--user\nfixture\n--home\n${home}\n--business\nacme\n--copy\n${copy}\n--node\n${NODE}\n--claude\n${e.claude}\n--\n--port\n${e.port}\n--business\nacme\n`;
|
||||
writeFileSync(e.args,e.argsText);chmodSync(e.args,0o640);
|
||||
e.run=join(e.root,'run');e.slog=join(e.root,'log','upgrade-dev.log');e.lockf=join(e.run,'upgrade-dev.lock');
|
||||
mkdirSync(join(e.sd,'arm'));
|
||||
e.denv={...e.env,STACK_UPGRADE_ETC:e.etc,STACK_UPGRADE_RUN:e.run,STACK_UPGRADE_LOG_DIR:dirname(e.slog),STACK_UPGRADE_UNIT_DIR:e.units,STACK_UPGRADE_HOSTCHECK:join(FAKES,'hostcheck'),STACK_UPGRADE_RENDER:join(FAKES,'render'),STACK_UPGRADE_SUDO:join(FAKES,'sudo'),STACK_UPGRADE_SYSTEMCTL:join(FAKES,'systemctl')};
|
||||
writeFileSync(join(e.sd,'probe.sh'),`exec bash ${join(checkout(C1),'scripts','stack-upgrade.sh')} upgrade --instance dev --bundle ${BUNDLE} --sha ${C1} --timeout 3\n`);
|
||||
writeFileSync(join(e.sd,'snap.jsonl'),'');
|
||||
return e;
|
||||
}
|
||||
|
||||
// The driver from root's checkout of `at`.
|
||||
async function drive(e,at,...args){
|
||||
const ch=spawn('bash',[join(checkout(at),'scripts','stack-upgrade.sh'),...args],{env:e.denv,stdio:['ignore','pipe','pipe']});
|
||||
let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d});
|
||||
const timer=setTimeout(()=>ch.kill('SIGKILL'),180000);
|
||||
const [code]=await once(ch,'close');clearTimeout(timer);
|
||||
return {code,out,err,all:out+err};
|
||||
}
|
||||
const upgrade=(e,id,...a)=>drive(e,id,'upgrade','--instance','dev','--bundle',BUNDLE,'--sha',id,...TO,...a);
|
||||
const arm=(e,key,text)=>writeFileSync(join(e.sd,'arm',key),text);
|
||||
|
||||
// The step log: every line `<time> <step> rc=N ...` or the result line, root:root 0640 (here the
|
||||
// test user's), and nothing the stack's user printed. Returns the lines without the time.
|
||||
const STEP=/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ (start|fetch|ff|copy|hostcheck|units|plan|rehearse|rehearse-store|rehearsal-skipped|stop-console|stop-bus|wait-down|stage|render|confirm|args|revert|render-old|confirm-rollback|gate|start-bus|start-console|stop-units) rc=\d+ |^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ result exit=[0-3] [a-z-]+ /;
|
||||
function slog(e){
|
||||
const l=lines(e.slog);
|
||||
for(const x of l){assert.match(x,STEP);assert.ok(!/stack-release:|fixture rehearse|PASS |FAIL |REHEARSAL/.test(x),x)}
|
||||
assert.equal(statSync(e.slog).mode&0o777,0o640);
|
||||
return l.map((x)=>x.slice(21));
|
||||
}
|
||||
const names=(e)=>slog(e).map((x)=>x.split(' ')[0]);
|
||||
const result=(e)=>slog(e).at(-1);
|
||||
const sudoKeys=(e)=>lines(join(e.sd,'sudo.log')).map((l)=>JSON.parse(l).key);
|
||||
const lastAct=(e)=>logLines(e).filter((o)=>o.result==='started').at(-1).activation;
|
||||
const mine=(e,a)=>logLines(e).filter((o)=>o.activation===a);
|
||||
const open=(e)=>{const m=new Set();for(const o of logLines(e)){if(!o.activation)continue;if(o.result==='started')m.add(o.activation);else m.delete(o.activation)}return [...m]};
|
||||
const final=(e,a)=>{const [s,f,...more]=mine(e,a);assert.equal(s.result,'started');assert.equal(more.length,0);return f};
|
||||
const live=(e)=>[units(e)[e.bus].state,units(e)[e.con].state,units(e)[e.bus].wd.split('/').at(-1),units(e)[e.con].wd.split('/').at(-1)];
|
||||
// Nothing the run did reached the args file: the same bytes and mode, and no temporary file left.
|
||||
const argsKept=(e)=>{assert.equal(readFileSync(e.args,'utf8'),e.argsText);assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args'])};
|
||||
const argsCopy=(e)=>{const l=readFileSync(e.args,'utf8').split('\n');return l[l.indexOf('--copy')+1]};
|
||||
const PRE=['start','fetch','ff','copy','hostcheck','units','plan','rehearse','rehearse-store'];
|
||||
const A2=['stop-console','stop-bus','wait-down','wait-down'];
|
||||
const R=[...A2,'revert','render-old','confirm-rollback'];
|
||||
|
||||
// --- r6 sequences 1-8, through the driver ---
|
||||
|
||||
test('row 233: sequence 1: confirm fails healthz, the driver reverts, renders the old copy and confirm --rollback --cause A5 passes: exit 1',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,`healthz-${C1}`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
|
||||
assert.equal(r.err.split('\n').at(-2),`stack-upgrade: failed: A5 failed; rolled back to ${B}, which passed the gate`);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm',...R,'result']);
|
||||
assert.equal(result(e),`result exit=1 failed A5 failed; rolled back to ${B}, which passed the gate`);
|
||||
const a=lastAct(e),l=final(e,a);assert.deepEqual([l.op,l.result,l.code,l.rollback,l.from,l.to],['activate','gate-failed','healthz','ok',B,C1]);
|
||||
assert.deepEqual(open(e),[]);assert.deepEqual(live(e),['active','active',B,B]);
|
||||
argsKept(e);
|
||||
// The stack's user's steps went through sudo in the driver's form, as fixture, the pinned node first.
|
||||
for(const l of lines(join(e.sd,'sudo.log')).map((x)=>JSON.parse(x)))assert.deepEqual([l.user,l.path],['fixture',`${dirname(NODE)}:/usr/local/bin:/usr/bin:/bin`]);
|
||||
assert.deepEqual(sudoKeys(e),['git-fetch','git-merge','stack-copy.sh','activate-plan','rehearse','activate','confirm','revert','confirm-rollback']);
|
||||
});
|
||||
|
||||
test('row 233: sequence 2: activate --migrate fails at the wrapper, revert restores the store, confirm --rollback --cause A3: exit 1',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const s0=sha(e.store);
|
||||
arm(e,'activate.sh',`chmod 0555 ${e.bin}\n`);arm(e,'revert.sh',`chmod 0755 ${e.bin}\n`);
|
||||
const r=await upgrade(e,C,'--migrate');assert.equal(r.code,1,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage',...R,'result']);
|
||||
assert.equal(result(e),`result exit=1 failed A3 failed; rolled back to ${B}, which passed the gate`);
|
||||
const l=final(e,lastAct(e));assert.deepEqual([l.result,l.code,l.rollback],['failed','wrapper','ok']);
|
||||
assert.equal(storeVersion(e.store,true),'3g');assert.deepEqual(live(e),['active','active',B,B]);
|
||||
// The bus holds the store open again; its bytes are the backup's once it stops.
|
||||
sysctl(e,'stop',e.bus);assert.equal(sha(e.store),s0);
|
||||
argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: sequence 3: root\'s render of the new copy fails, the driver rolls back with --cause A4, and holds the lock to the end: exit 1',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,`render-fails-${C1}`);fault(e,`render-probe-${B}`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
|
||||
// The probe ran inside render-old, during the rollback, and its one line is its refusal.
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage','render',...A2,'revert','result','render-old','confirm-rollback','result']);
|
||||
const p=JSON.parse(readFileSync(join(e.sd,'probe.json'),'utf8'));assert.equal(p.code,3,p.err);
|
||||
assert.equal(p.err,`stack-upgrade: upgrade-running: another stack-upgrade.sh run holds ${e.lockf} (nothing was stopped)\n`);
|
||||
assert.match(slog(e).at(-4),/^result exit=3 upgrade-running /);
|
||||
assert.match(slog(e).find((x)=>x.startsWith('render ')),/^render rc=1 /);
|
||||
const l=final(e,lastAct(e));assert.deepEqual([l.result,l.code,l.rollback],['failed','units-start','ok']);
|
||||
assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: sequence 4: a capture failure is exit 3 from the stage, and A2f restarts the old units and gates them: exit 1',async(t)=>{
|
||||
const e=await useed(t);
|
||||
arm(e,'activate.sh',`: > ${join(e.sd,'faults','no-fragment')}\n`);arm(e,'gate.sh',`rm ${join(e.sd,'faults','no-fragment')}\n`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage','start-bus','start-console','gate','result']);
|
||||
assert.match(slog(e).find((x)=>x.startsWith('stage ')),/^stage rc=3 /);
|
||||
assert.equal(result(e),`result exit=1 failed A3 failed; ${B} is running again and passed the gate`);
|
||||
const a=lastAct(e);assert.deepEqual(mine(e,a).map((o)=>[o.result,o.code]),[['started',undefined],['refused','capture-failed']]);
|
||||
assert.deepEqual(open(e),[]);assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: N1: a bus that outlives A2\'s timeout keeps its old start, so A2f\'s gate fails stale-start: exit 2, both units stopped',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,`stop-noop-${e.bus}`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,2,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'start-bus','start-console','gate','stop-units','result']);
|
||||
assert.ok(slog(e).includes(`wait-down rc=1 unit: ${e.bus} state: active`),slog(e).join('\n'));
|
||||
assert.ok(r.out.includes(`gate failed on ${B}: stale-start`),r.out);
|
||||
assert.equal(result(e),`result exit=2 rollback-failed A2 failed and ${B} failed its gate after the restart; both units are stopped`);
|
||||
assert.deepEqual(live(e).slice(0,2),['inactive','inactive']);
|
||||
assert.ok(!sudoKeys(e).includes('activate'));argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: sequence 5: confirm of the wrong copy is refused, and the driver rolls back with --cause A5: exit 1',async(t)=>{
|
||||
const e=await useed(t);
|
||||
arm(e,'confirm.id',B);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
|
||||
assert.ok(r.out.includes('refusing open-activation')||r.err.includes('refusing open-activation'),r.all);
|
||||
assert.match(slog(e).find((x)=>x.startsWith('confirm ')),/^confirm rc=3 /);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm',...R,'result']);
|
||||
const l=final(e,lastAct(e));assert.deepEqual([l.result,l.code,l.rollback],['gate-failed','unrecorded','ok']);
|
||||
assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: sequence 6: a revert that fails stops both units at exit 2, and the next run refuses at its --plan with nothing stopped',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,`render-fails-${C1}`);
|
||||
arm(e,'revert.sh',`for d in ${join(e.state,'stack-rollback')}/*/; do [ -f "$d/AWAITING-UNITS" ] && rm "$d/wrapper"; done\n`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,2,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage','render',...A2,'revert','stop-units','result']);
|
||||
assert.equal(result(e),`result exit=2 rollback-failed A4 failed and revert failed; both units are stopped`);
|
||||
const a=lastAct(e);assert.deepEqual(open(e),[a]);assert.deepEqual(live(e).slice(0,2),['inactive','inactive']);
|
||||
argsKept(e);
|
||||
const n=slog(e).length,sc=lines(join(e.sd,'root-systemctl.log')).length;
|
||||
rmSync(join(e.sd,'faults',`render-fails-${C1}`));
|
||||
const again=await upgrade(e,C1);assert.equal(again.code,3,again.all);
|
||||
assert.ok(again.out.includes(`refuses interrupted-run: activation ${a} started and has no final line`),again.out);
|
||||
assert.deepEqual(names(e).slice(n),PRE.slice(0,7).concat('result'));
|
||||
assert.match(result(e),/^result exit=3 plan-refused /);
|
||||
assert.equal(lines(join(e.sd,'root-systemctl.log')).length,sc);argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: sequence 7: two upgrades and a rollback --to the previous copy with --restore, the args file following each confirm',async(t)=>{
|
||||
const e=await useed(t);
|
||||
// The args file at confirm still names the old copy; the new one moves in after it.
|
||||
arm(e,'confirm.sh',`cp ${e.args} ${join(e.sd,'args-at-confirm')}\n`);
|
||||
const one=await upgrade(e,C1);assert.equal(one.code,0,one.all);
|
||||
assert.equal(one.out.split('\n').at(-2),`stack-upgrade: upgrade ok: ${C1} is live for acme`);
|
||||
assert.equal(readFileSync(join(e.sd,'args-at-confirm'),'utf8'),e.argsText);
|
||||
assert.equal(readFileSync(e.args,'utf8'),e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`));
|
||||
assert.equal(statSync(e.args).mode&0o777,0o640);assert.deepEqual(readdirSync(e.etc),['dev.args']);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm','args','result']);
|
||||
assert.ok(slog(e).includes(`args rc=0 copy: ${C1}`));
|
||||
const a1=lastAct(e);
|
||||
const two=await upgrade(e,C,'--migrate');assert.equal(two.code,0,two.all);
|
||||
const a2=lastAct(e);assert.equal(argsCopy(e),C);
|
||||
const l1=final(e,a1),l2=final(e,a2);
|
||||
assert.deepEqual([l1.op,l1.result,l1.from,l1.to],['activate','ok',B,C1]);
|
||||
assert.deepEqual([l2.op,l2.result,l2.from,l2.to,l2.backup],['activate','ok',C1,C,a2]);
|
||||
assert.equal(storeVersion(e.store,true),'3h');
|
||||
const n=slog(e).length;
|
||||
const rb=await drive(e,C,'rollback','--instance','dev','--to',C1,'--restore',a2,...TO);assert.equal(rb.code,0,rb.all);
|
||||
assert.equal(rb.out.split('\n').at(-2),`stack-upgrade: rollback ok: ${C1} is live for acme`);
|
||||
// A rollback fetches, copies and rehearses nothing.
|
||||
assert.deepEqual(names(e).slice(n),['start','hostcheck','units','plan',...A2,'stage','render','confirm','args','result']);
|
||||
const ra=lastAct(e),l3=final(e,ra);assert.deepEqual([l3.op,l3.result,l3.from,l3.to,l3.backup],['rollback','ok',C,C1,a2]);
|
||||
assert.deepEqual([pointer(e).copy,pointer(e).schema],[C1,'3g']);assert.equal(storeVersion(e.store,true),'3g');
|
||||
assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
|
||||
const m=slog(e).length,sc=lines(join(e.sd,'root-systemctl.log')).length;
|
||||
const no=await drive(e,C,'rollback','--instance','dev','--to',B,...TO);assert.equal(no.code,3,no.all);
|
||||
assert.ok(no.all.includes(`refusing to-not-previous: --to ${B} is not the previous copy; the log names ${C}`),no.all);
|
||||
assert.deepEqual(names(e).slice(m),['start','hostcheck','units','plan','result']);
|
||||
assert.equal(lines(join(e.sd,'root-systemctl.log')).length,sc);assert.equal(argsCopy(e),C1);
|
||||
});
|
||||
|
||||
test('row 233: sequence 8: a migration that writes and fails: the driver reverts to the backup byte for byte, --cause A3: exit 1',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const s0=sha(e.store);
|
||||
fault(e,'migrate-fails');
|
||||
const r=await upgrade(e,C,'--migrate');assert.equal(r.code,1,r.all);
|
||||
assert.match(slog(e).find((x)=>x.startsWith('stage ')),/^stage rc=1 /);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage',...R,'result']);
|
||||
const a=lastAct(e),l=final(e,a);assert.deepEqual([l.result,l.code,l.rollback],['failed','migrate-failed','ok']);
|
||||
sysctl(e,'stop',e.bus);assert.equal(sha(e.store),s0);assert.equal(storeVersion(e.store,false),'3g');
|
||||
argsKept(e);
|
||||
});
|
||||
|
||||
// --- The lock, A1's refusals and the rehearsal ---
|
||||
|
||||
test('row 233: a second run while the lock is held refuses upgrade-running with one log line and no call',async(t)=>{
|
||||
const e=await useed(t);
|
||||
mkdirSync(e.run,{recursive:true});
|
||||
const holder=spawn('flock',[e.lockf,'sleep','60'],{detached:true,stdio:'ignore'});
|
||||
t.after(()=>{try{process.kill(-holder.pid,'SIGKILL')}catch{}});
|
||||
for(let i=0;i<100&&execFileSync('bash',['-c','flock -n "$1" true && echo free || echo held','_',e.lockf],{encoding:'utf8'}).trim()==='free';i++)await new Promise((r)=>setTimeout(r,50));
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
|
||||
assert.equal(r.err,`stack-upgrade: upgrade-running: another stack-upgrade.sh run holds ${e.lockf} (nothing was stopped)\n`);
|
||||
assert.deepEqual(names(e),['result']);
|
||||
assert.ok(!existsSync(join(e.sd,'sudo.log')));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
|
||||
assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: A1 refuses open-activation on any AWAITING-UNITS file, through the stack\'s user\'s --plan, with nothing stopped',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const fake=join(e.state,'stack-rollback',`20000101T000000Z-${C1.slice(0,12)}`);mkdirSync(fake,{recursive:true});writeFileSync(join(fake,'AWAITING-UNITS'),`${C1}\n`);
|
||||
const logs=logLines(e).length;
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
|
||||
assert.ok(r.out.includes('refuses open-activation'),r.out);
|
||||
assert.deepEqual(names(e),[...PRE.slice(0,7),'result']);assert.match(result(e),/^result exit=3 plan-refused /);
|
||||
assert.equal(logLines(e).length,logs);assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
|
||||
assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: a target that migrates refuses migrate-required without --migrate, with nothing stopped',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const r=await upgrade(e,C);assert.equal(r.code,3,r.all);
|
||||
assert.ok(r.out.includes('refuses migrate-required'),r.out);
|
||||
assert.deepEqual(names(e),[...PRE.slice(0,7),'result']);assert.match(result(e),/^result exit=3 plan-refused /);
|
||||
assert.ok(!sudoKeys(e).includes('rehearse'));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
|
||||
assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: the rehearsal runs after the plan and before A2; a failed one refuses rehearsal-failed with nothing stopped',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,'rehearse-rc','1');
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
|
||||
assert.ok(r.out.includes('fixture rehearse: '),r.out);
|
||||
assert.deepEqual(names(e),[...PRE,'result']);
|
||||
assert.equal(result(e),'result exit=3 rehearsal-failed the rehearsal exited 1 (nothing was stopped)');
|
||||
const rh=lines(join(e.sd,'sudo.log')).map((l)=>JSON.parse(l)).find((l)=>l.key==='rehearse');
|
||||
assert.deepEqual(rh.cmd.slice(1),['rehearse',C1,'acme','--copies',e.hc,'--backup','last','--port','7339','--timeout','3']);
|
||||
assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: the rehearsal\'s BACKUP_STORE line goes to the step log after a match, a malformed one as none, the rest of its output never',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,'rehearse-rc','1');
|
||||
const r=await upgrade(e,C1,'--rehearse-port','7449');assert.equal(r.code,3,r.all);
|
||||
assert.ok(slog(e).includes('rehearse-store rc=0 BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g'),slog(e).join('\n'));
|
||||
assert.ok(slog(e).find((x)=>x.startsWith('rehearse ')).includes(' --port 7449 '));
|
||||
for(const bad of ['BACKUP_STORE ../../etc schema 3g','BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g; rm -rf /','stack-release: BACKUP_STORE 20261020T000000Z-0123456789ab schema 3g','']){
|
||||
fault(e,'rehearse-store-line',bad);
|
||||
const n=slog(e).length;
|
||||
const b=await upgrade(e,C1);assert.equal(b.code,3,b.all);
|
||||
assert.deepEqual(slog(e).slice(n).filter((x)=>x.startsWith('rehearse-store ')),['rehearse-store rc=0 BACKUP_STORE none'],bad);
|
||||
}
|
||||
});
|
||||
|
||||
test('row 233: a real rehearsal with no backup in the log refuses no-backup, and goes on only with --no-rehearsal, which logs the skip',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,'real-rehearse');
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
|
||||
assert.ok(r.out.includes('FAIL no-backup: the log records no backup\nREHEARSAL_FAIL\n'),r.out);
|
||||
assert.equal(result(e),'result exit=3 no-backup the log records no backup to rehearse against; only a go that names --no-rehearsal goes on (nothing was stopped)');
|
||||
assert.ok(slog(e).includes('rehearse-store rc=0 BACKUP_STORE none'));
|
||||
assert.ok(!existsSync(join(e.data,'rehearsals')));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));
|
||||
const n=slog(e).length;
|
||||
const s=await upgrade(e,C1,'--no-rehearsal');assert.equal(s.code,0,s.all);
|
||||
assert.deepEqual(names(e).slice(n),[...PRE,'rehearsal-skipped',...A2,'stage','render','confirm','args','result']);
|
||||
assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
|
||||
});
|
||||
|
||||
test('row 233: a real rehearsal after an upgrade names the log\'s newest backup; --no-rehearsal with a backup refuses rehearsal-required',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const one=await upgrade(e,C1);assert.equal(one.code,0,one.all);
|
||||
const a1=lastAct(e);
|
||||
fault(e,'real-rehearse');
|
||||
const n=slog(e).length;
|
||||
// The stub copies fail the rehearsal right after it names the backup (copy-without-no-launch).
|
||||
const r=await upgrade(e,C,'--migrate');assert.equal(r.code,3,r.all);
|
||||
assert.ok(r.out.includes(`PASS backup-last: the log's newest backup is ${a1}\n`),r.out);
|
||||
assert.deepEqual(names(e).slice(n),[...PRE,'result']);assert.match(result(e),/^result exit=3 rehearsal-failed the rehearsal exited 1 /);
|
||||
rmSync(join(e.sd,'faults','real-rehearse'));
|
||||
const m=slog(e).length;
|
||||
const q=await upgrade(e,C,'--migrate','--no-rehearsal');assert.equal(q.code,3,q.all);
|
||||
assert.deepEqual(names(e).slice(m),[...PRE,'result']);
|
||||
assert.equal(result(e),'result exit=3 rehearsal-required the log records a backup and the rehearsal ran; --no-rehearsal is only for a stack with none (nothing was stopped)');
|
||||
assert.equal(argsCopy(e),C1);assert.deepEqual(live(e),['active','active',C1,C1]);
|
||||
});
|
||||
|
||||
// --- Before A1 and the args file ---
|
||||
|
||||
test('row 233: the args file must be the running user\'s, with no group or other write, one --copy, and the copy the units run',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const tryArgs=async(text,mode,code,why)=>{
|
||||
writeFileSync(e.args,text);chmodSync(e.args,mode);
|
||||
const n=existsSync(e.slog)?slog(e).length:0;
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,`${why}: ${r.all}`);
|
||||
assert.match(slog(e).at(-1),new RegExp(`^result exit=3 ${code} `),why);
|
||||
assert.ok(!slog(e).slice(n).some((x)=>/^(stop|stage|render|confirm)/.test(x)),why);
|
||||
};
|
||||
await tryArgs(e.argsText,0o660,'args-unreadable','group write');
|
||||
await tryArgs(e.argsText,0o642,'args-unreadable','other write');
|
||||
await tryArgs(e.argsText.replace('--\n',`--copy\n${C1}\n--\n`),0o640,'args-unreadable','two --copy');
|
||||
await tryArgs(e.argsText.replace('--\n','--tester\n--\n'),0o640,'args-unreadable','an unknown flag');
|
||||
await tryArgs(e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`),0o640,'args-differ','the units run B');
|
||||
rmSync(e.args);symlinkSync(join(e.sd,'probe.sh'),e.args);
|
||||
const l=await upgrade(e,C1);assert.equal(l.code,3);assert.match(result(e),/^result exit=3 args-unreadable /);
|
||||
assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));assert.deepEqual(live(e),['active','active',B,B]);
|
||||
});
|
||||
|
||||
test('row 233: hostcheck (row 240): a nonzero --check or a missing file refuses profile-drift before the plan',async(t)=>{
|
||||
const e=await useed(t);
|
||||
fault(e,'hostcheck-rc','1');
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);
|
||||
assert.match(result(e),/^result exit=3 profile-drift /);
|
||||
assert.deepEqual(lines(join(e.sd,'hostcheck.log')),['--check']);
|
||||
assert.deepEqual(sudoKeys(e),['git-fetch','git-merge','stack-copy.sh']);
|
||||
e.denv.STACK_UPGRADE_HOSTCHECK=join(e.root,'no-hostcheck');
|
||||
const m=await upgrade(e,C1);assert.equal(m.code,3,m.all);
|
||||
assert.equal(result(e),`result exit=3 profile-drift no executable ${join(e.root,'no-hostcheck')} (row 240) (nothing was stopped)`);
|
||||
assert.ok(!sudoKeys(e).includes('activate-plan'));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: the checkout must be at --sha (head-mismatch), and a commit the clone can\'t fast-forward to refuses fetch-failed',async(t)=>{
|
||||
const e=await useed(t);
|
||||
const r=await drive(e,C1,'upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C,'--migrate',...TO);assert.equal(r.code,3,r.all);
|
||||
assert.equal(result(e),`result exit=3 head-mismatch this checkout, ${checkout(C1)}, is at ${C1}, not ${C} (nothing was stopped)`);
|
||||
const u=await drive(e,U,'upgrade','--instance','dev','--bundle',BUNDLE,'--sha',U,...TO);assert.equal(u.code,3,u.all);
|
||||
assert.match(result(e),/^result exit=3 fetch-failed .* fast-forward /);
|
||||
assert.ok(!sudoKeys(e).includes('activate-plan'));assert.ok(!existsSync(join(e.sd,'root-systemctl.log')));argsKept(e);
|
||||
});
|
||||
|
||||
test('row 233: usage exits 4 before any log or lock',async(t)=>{
|
||||
const e=await useed(t);
|
||||
for(const a of [[],['status'],['upgrade','--instance','dev','--bundle',BUNDLE],['upgrade','--instance','other','--bundle',BUNDLE,'--sha',C1],['upgrade','--instance','dev','--bundle','rel/x','--sha',C1],['upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C1.slice(0,12)],['upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C1,'--to',B],['upgrade','--instance','dev','--bundle',BUNDLE,'--sha',C1,'--rehearse-port','80'],['rollback','--instance','dev','--to',B,'--migrate'],['rollback','--instance','dev','--to',B,'--events','3'],['rollback','--instance','dev'],['rollback','--instance','dev','--to',B,'--restore','x']]){
|
||||
const r=await drive(e,C1,...a);assert.equal(r.code,4,a.join(' '));assert.match(r.err,/^usage: stack-upgrade.sh upgrade /,a.join(' '));
|
||||
}
|
||||
assert.ok(!existsSync(dirname(e.slog)));assert.ok(!existsSync(e.run));
|
||||
});
|
||||
@@ -130,7 +130,7 @@ parse_opts() {
|
||||
--sso) [ "$OP" = activate ] || usage; SSO=1; shift ;;
|
||||
--migrate) [ "$OP" = activate ] || usage; MIGRATE=1; shift ;;
|
||||
--restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || usage; RESTORE="$2"; shift 2 ;;
|
||||
--backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
|
||||
--backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [ "$2" = last ] || [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
|
||||
--port) [ "$OP" = rehearse ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{1,5}$ ]] && [ "$((10#$2))" -ge 1024 ] && [ "$((10#$2))" -le 65535 ] || usage; R_PORT="$((10#$2))"; shift 2 ;;
|
||||
--events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
|
||||
--units) [ "$OP" != rehearse ] && [ $# -ge 2 ] && { [ "$2" = user ] || [ "$2" = external ]; } || usage; UNITS="$2"; shift 2 ;;
|
||||
@@ -220,7 +220,8 @@ for(const k of open.keys())process.stdout.write(k+"\n");' -- "$LOG"
|
||||
|
||||
# Reads the log's ok lines, newest first. `previous` prints rollback's target: the first `to`
|
||||
# that is not the pointer's copy (release.sh's rule). `backup` prints the backup field of the
|
||||
# line that made the pointer's copy live, the backup a rollback would restore. `arrival` prints the
|
||||
# line that made the pointer's copy live, the backup a rollback would restore. `newest` prints the
|
||||
# backup field of the newest line that has one, rehearse's --backup last (row 233). `arrival` prints the
|
||||
# activation and `from` of the line that moved the stack onto the pointer's copy from another one,
|
||||
# past any same-copy re-pin after it (row 200, sage-1389).
|
||||
from_log() {
|
||||
@@ -233,6 +234,7 @@ for(const l of lines.reverse()){
|
||||
if(o.result!=="ok"||!["adopt","activate","rollback"].includes(o.op)||!o.to)continue;
|
||||
if(mode==="previous"&&o.to!==current){process.stdout.write(o.to+"\n");process.exit(0)}
|
||||
if(mode==="backup"&&o.to===current){if(o.backup)process.stdout.write(o.backup+"\n");process.exit(o.backup?0:1)}
|
||||
if(mode==="newest"&&o.backup){process.stdout.write(o.backup+"\n");process.exit(0)}
|
||||
if(mode==="arrival"&&o.to===current&&o.from!==current){process.stdout.write(`${o.activation??""} ${o.from??""}\n`);process.exit(0)}
|
||||
}
|
||||
process.exit(1);' -- "$LOG" "$1" "$P_COPY"
|
||||
@@ -1262,6 +1264,8 @@ plan() {
|
||||
echo "needs: --migrate$([ "$MIGRATE" = 1 ] && echo ' (given)' || echo ' (not given; the run would roll back with migration-needs-flag)')"
|
||||
echo "needs: Jason present for this run (decision 96 condition 2)"
|
||||
[ "$MIGRATE" = 1 ] || early=1
|
||||
# Row 233: root reads only the plan's exit, so under external a missing --migrate refuses here.
|
||||
[ "$UNITS" = user ] || [ "$MIGRATE" = 1 ] || PLAN_REFUSE+=("migrate-required: the store is $S_SCHEMA and $TO migrates it to $T_SCHEMA; the go names --migrate")
|
||||
elif [ "$OP" = activate ]; then
|
||||
echo "migration: none possible; $S_SCHEMA is not in $T_SCHEMA's migration chain, so the run would roll back with store-newer"
|
||||
early=1
|
||||
@@ -1452,6 +1456,13 @@ ext_pointer_checks() {
|
||||
local open
|
||||
open="$(interrupted | paste -sd' ' -)"
|
||||
[ -z "$open" ] || check_refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)"
|
||||
# Row 233: root can't read the stack's state, so the plan lists any rollback set still waiting
|
||||
# for units, whatever the log says.
|
||||
local m sets=""
|
||||
local -a markers
|
||||
mapfile -t markers < <(compgen -G "$STATE/stack-rollback/*/AWAITING-UNITS" || true)
|
||||
for m in "${markers[@]}"; do sets+=" $(basename "$(dirname "$m")")"; done
|
||||
[ -z "$sets" ] || check_refuse open-activation "rollback set(s)$sets hold AWAITING-UNITS; root finishes them with confirm or revert first"
|
||||
}
|
||||
|
||||
# A3's checks before `started`. Root has stopped the units and installs them after this run, so
|
||||
@@ -1973,6 +1984,9 @@ r_env() { env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json"
|
||||
r_migration() {
|
||||
local copy="$COPIES/$TO" old="$COPIES/$P_COPY" v out rc
|
||||
v="$(node -e "$STORE_JS" -- version "$R/data/bus/bus.sqlite" 2>&1)" || r_abort store-unreadable "$v"
|
||||
# Row 233 (sage-1653): which store the rehearsal ran against, for root's step log. Output only:
|
||||
# the rehearsal's own steps.log has the backup's path on its backup line.
|
||||
echo "BACKUP_STORE $R_BACKUP schema $v"
|
||||
if [ "$v" = "$T_SCHEMA" ]; then r_pass migration "not needed: the backup is already $v"; return 0; fi
|
||||
in_chain "$v" || r_abort store-newer "the backup is $v, which $TO's migration chain does not start from"
|
||||
(umask 077; mkdir "$R/refuse" "$R/refuse/bus" && cp "$R_BK"/bus/* "$R/refuse/bus/") || r_abort scratch-build "cannot copy the backup to $R/refuse"
|
||||
@@ -2043,6 +2057,13 @@ do_rehearse() {
|
||||
trap 'r_fail signal "the rehearsal was interrupted"; r_end' INT TERM HUP
|
||||
read_pointer || r_abort no-pointer "no $POINTER; the pointer's copy is the old copy the checks use"
|
||||
[ "$P_BUSINESS" = "$BUSINESS" ] || r_abort business-mismatch "the pointer is for $P_BUSINESS"
|
||||
# Row 233 (sage-1650, sage-1653): last is the newest backup the log records. With none, exit 3
|
||||
# and no directory, the one rehearse refusal stack-upgrade.sh reads as no-backup.
|
||||
if [ "$R_BACKUP" = last ]; then
|
||||
R_BACKUP="$(from_log newest)" || { r_fail no-backup "the log records no backup"; R_RC=3; r_end; }
|
||||
[[ "$R_BACKUP" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || r_abort backup-unreadable "the log's newest backup is not an activation id"
|
||||
r_pass backup-last "the log's newest backup is $R_BACKUP"
|
||||
fi
|
||||
copy_exists "$TO" 2>/dev/null || r_abort copy-missing "no copy $TO in $COPIES"
|
||||
copy_ok "$TO" 2>/dev/null || r_abort copy-unreachable "$TO is not reachable from refactor and origin/refactor"
|
||||
target_schema "$TO" || r_abort copy-schema-unknown "cannot read SCHEMA_VERSION and its chain from $COPIES/$TO/packages/bus/src/store.mjs"
|
||||
|
||||
Executable
+347
@@ -0,0 +1,347 @@
|
||||
#!/usr/bin/env bash
|
||||
# Row 233 (I3, #1735): root's upgrade and rollback for a stack whose units are root's system units
|
||||
# (stack-release.sh --units external, row 232). r6's activate A1-A5, A2f, R1-R4 and rollback
|
||||
# (agents/darkwing/work/queue-195/DESIGN.md), as the I3 brief changes them
|
||||
# (docs/plans/2026-10-20_installer-rows.md). Deviations and decisions:
|
||||
# agents/darkwing/work/queue-233/BUILD.md.
|
||||
#
|
||||
# stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
|
||||
# [--rehearse-port N] [--timeout S]
|
||||
# stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
|
||||
#
|
||||
# Root runs it from a root-owned checkout of the reviewed commit, never from a copy, because the
|
||||
# stack's user can write the copies. The unit values come from /etc/mosaic-stack/<instance>.args,
|
||||
# which I2's install writes: one case2-unit.sh argument per line, the serve flags after `--`. Only
|
||||
# --copy changes, and the file changes only after confirm passes. The stack's user runs its steps
|
||||
# from its own clone, <home>/src/mosaic-stack, through sudo. Root reads none of that user's output
|
||||
# but exit codes and the rehearsal's BACKUP_STORE line (sage-1653), which it copies into its step
|
||||
# log after a match against a fixed pattern and decides nothing on.
|
||||
#
|
||||
# --no-rehearsal is for a stack whose log records no backup. The rehearsal runs either way: exit 3
|
||||
# from it (no-backup) goes on only with the flag, and exit 0 with the flag refuses
|
||||
# rehearsal-required (sage-1650, sage-1652).
|
||||
#
|
||||
# The lock /run/mosaic-stack/upgrade-<instance>.lock is held from before A1 to the end of the run,
|
||||
# rollback included; every child runs with the lock's descriptor closed. The step log
|
||||
# /var/log/mosaic-stack/upgrade-<instance>.log gets one line per step, `<time> <step> rc=N argv:
|
||||
# <argv>`, and a last `<time> result exit=N <code> <text>` line.
|
||||
#
|
||||
# Exit 0 ok, 1 failed with the old stack running, 2 rollback-failed with the units stopped,
|
||||
# 3 refused with nothing stopped, 4 usage.
|
||||
#
|
||||
# Test seams, for packages/cli/tests/stack-upgrade.test.mjs only. Run as root, it refuses any:
|
||||
# STACK_UPGRADE_ETC default /etc/mosaic-stack
|
||||
# STACK_UPGRADE_RUN default /run/mosaic-stack
|
||||
# STACK_UPGRADE_LOG_DIR default /var/log/mosaic-stack
|
||||
# STACK_UPGRADE_UNIT_DIR default /etc/systemd/system
|
||||
# STACK_UPGRADE_HOSTCHECK default /usr/local/lib/mosaic-stack/hostcheck.sh (row 240)
|
||||
# STACK_UPGRADE_RENDER default scripts/case2-unit.sh in this checkout
|
||||
# STACK_UPGRADE_SUDO default sudo
|
||||
# STACK_UPGRADE_SYSTEMCTL default systemctl
|
||||
set -uo pipefail
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
|
||||
usage() {
|
||||
cat >&2 <<'USAGE'
|
||||
usage: stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal] [--rehearse-port N] [--timeout S]
|
||||
stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
|
||||
USAGE
|
||||
exit 4
|
||||
}
|
||||
|
||||
if [ "$(id -u)" = 0 ]; then
|
||||
for v in ${!STACK_UPGRADE_@}; do echo "stack-upgrade: refusing: $v is a test seam, and root takes none" >&2; exit 3; done
|
||||
fi
|
||||
ETC="${STACK_UPGRADE_ETC:-/etc/mosaic-stack}"
|
||||
RUN_DIR="${STACK_UPGRADE_RUN:-/run/mosaic-stack}"
|
||||
LOG_DIR="${STACK_UPGRADE_LOG_DIR:-/var/log/mosaic-stack}"
|
||||
UNIT_DIR="${STACK_UPGRADE_UNIT_DIR:-/etc/systemd/system}"
|
||||
HOSTCHECK="${STACK_UPGRADE_HOSTCHECK:-/usr/local/lib/mosaic-stack/hostcheck.sh}"
|
||||
RENDER="${STACK_UPGRADE_RENDER:-$REPO/scripts/case2-unit.sh}"
|
||||
SUDO="${STACK_UPGRADE_SUDO:-sudo}"
|
||||
SYSTEMCTL="${STACK_UPGRADE_SYSTEMCTL:-systemctl}"
|
||||
ID_RE='^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$'
|
||||
|
||||
ARGV=("$@")
|
||||
[ $# -gt 0 ] || usage
|
||||
OP="$1"; shift
|
||||
case "$OP" in upgrade|rollback) ;; *) usage ;; esac
|
||||
INSTANCE="" BUNDLE="" SHA="" MIGRATE=0 NO_REHEARSAL=0 R_PORT=7339 TIMEOUT=60 TO="" RESTORE="" EVENTS=""
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--instance) [ $# -ge 2 ] || usage; INSTANCE="$2"; shift 2 ;;
|
||||
--bundle) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^/[A-Za-z0-9/._+-]+$ ]] || usage; BUNDLE="$2"; shift 2 ;;
|
||||
--sha) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; SHA="$2"; shift 2 ;;
|
||||
--migrate) [ "$OP" = upgrade ] || usage; MIGRATE=1; shift ;;
|
||||
--no-rehearsal) [ "$OP" = upgrade ] || usage; NO_REHEARSAL=1; shift ;;
|
||||
--rehearse-port) [ "$OP" = upgrade ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{3,4}$ ]] && [ "$2" -le 65535 ] || usage; R_PORT="$2"; shift 2 ;;
|
||||
--timeout) [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
|
||||
--to) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9a-f]{40}$ ]] || usage; TO="$2"; shift 2 ;;
|
||||
--restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ $ID_RE ]] || usage; RESTORE="$2"; shift 2 ;;
|
||||
--events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
# The default instance only, whose name is dev (row 190); a named instance's units are a later row.
|
||||
[ "$INSTANCE" = dev ] || usage
|
||||
if [ "$OP" = upgrade ]; then [ -n "$BUNDLE" ] && [ -n "$SHA" ] || usage; else [ -n "$TO" ] || usage; fi
|
||||
[ -z "$EVENTS" ] || [ -n "$RESTORE" ] || usage
|
||||
|
||||
# --- The step log and the lock ---
|
||||
|
||||
umask 027
|
||||
LOG="$LOG_DIR/upgrade-$INSTANCE.log"
|
||||
LOCK="$RUN_DIR/upgrade-$INSTANCE.lock"
|
||||
{ mkdir -p "$LOG_DIR" "$RUN_DIR" && touch "$LOG" && chmod 0640 "$LOG"; } 2>/dev/null ||
|
||||
{ echo "stack-upgrade: refusing log-failed: cannot write $LOG or make $RUN_DIR; nothing was changed" >&2; exit 3; }
|
||||
logl() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*" >> "$LOG" || true; }
|
||||
# The run's last line, in the log and on the terminal, then the exit.
|
||||
finish() {
|
||||
local rc="$1" code="$2"; shift 2
|
||||
logl "result exit=$rc $code $*"
|
||||
if [ "$rc" = 0 ]; then echo "stack-upgrade: $*"; else echo "stack-upgrade: $code: $*" >&2; fi
|
||||
exit "$rc"
|
||||
}
|
||||
refuse() { local code="$1"; shift; finish 3 "$code" "$* (nothing was stopped)"; }
|
||||
# One step: $1 its name, the rest the command, run with the lock's descriptor closed. One log line
|
||||
# with its exit and argv; the command's output goes to the terminal only.
|
||||
step() {
|
||||
local name="$1" rc=0; shift
|
||||
"$@" 9>&- || rc=$?
|
||||
logl "$name rc=$rc argv:$(printf ' %q' "$@")"
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
exec 9>> "$LOCK" || refuse lock-failed "cannot open $LOCK"
|
||||
flock -n 9 || refuse upgrade-running "another stack-upgrade.sh run holds $LOCK"
|
||||
logl "start rc=0 argv:$(printf ' %q' "$0" "${ARGV[@]}")"
|
||||
|
||||
# --- The args file and the stack's user ---
|
||||
|
||||
# I2's file: one case2-unit.sh argument per line, owned by root with no group or other write. Sets
|
||||
# ARGS (the lines), S_USER, S_HOME, BUSINESS, NODE and ARGS_COPY, and the paths derived from them.
|
||||
read_args() {
|
||||
local f="$ETC/$INSTANCE.args" st i copies=0 v
|
||||
[ -f "$f" ] && [ ! -L "$f" ] || refuse args-unreadable "no regular file $f"
|
||||
st="$(stat -c '%u %a' "$f")" || refuse args-unreadable "cannot stat $f"
|
||||
[ "${st% *}" = "$(id -u)" ] && (( (8#${st#* } & 8#022) == 0 )) || refuse args-unreadable "$f must be $(id -un)'s with no group or other write (it is uid ${st% *}, mode ${st#* })"
|
||||
mapfile -t ARGS < "$f" || refuse args-unreadable "cannot read $f"
|
||||
S_USER="" S_HOME="" BUSINESS="" NODE="" ARGS_COPY=""
|
||||
for ((i = 0; i < ${#ARGS[@]}; i++)); do
|
||||
case "${ARGS[i]}" in
|
||||
--) break ;;
|
||||
--pm) ;;
|
||||
--user|--home|--business|--copy|--node|--profile|--claude)
|
||||
v="${ARGS[i + 1]:-}"
|
||||
case "${ARGS[i]}" in
|
||||
--user) S_USER="$v" ;; --home) S_HOME="$v" ;; --business) BUSINESS="$v" ;; --node) NODE="$v" ;;
|
||||
--copy) ARGS_COPY="$v"; copies=$((copies + 1)) ;;
|
||||
esac
|
||||
i=$((i + 1)) ;;
|
||||
*) refuse args-unreadable "$f has ${ARGS[i]} before --, which case2-unit.sh does not take" ;;
|
||||
esac
|
||||
done
|
||||
[ "$copies" = 1 ] || refuse args-unreadable "$f needs exactly one --copy before --, not $copies"
|
||||
[[ "$S_USER" =~ ^[a-z_][a-z0-9_-]*$ ]] && [[ "$S_HOME" =~ ^/[A-Za-z0-9/._+-]+$ ]] && [[ "$BUSINESS" =~ ^[a-z0-9][a-z0-9-]*$ ]] &&
|
||||
[[ "$NODE" =~ ^/[A-Za-z0-9/._+-]+$ ]] && [[ "$ARGS_COPY" =~ ^[0-9a-f]{40}$ ]] ||
|
||||
refuse args-unreadable "$f lacks a plain --user, --home, --business, --node or --copy"
|
||||
COPIES_DIR="$S_HOME/.local/share/mosaic-stack/copies"
|
||||
CLONE="$S_HOME/src/mosaic-stack"
|
||||
SR="$CLONE/scripts/stack-release.sh"
|
||||
BUS_UNIT="mosaic-bus@$BUSINESS.service" CON_UNIT=mosaic-webui.service
|
||||
# The stack's user's steps: its own HOME and default config, the pinned node first on PATH.
|
||||
AS_USER=("$SUDO" -u "$S_USER" -H env "PATH=$(dirname "$NODE"):/usr/local/bin:/usr/bin:/bin")
|
||||
}
|
||||
|
||||
# RENDER_ARGS: the args file's lines with copy $1 in place of --copy's value.
|
||||
args_with() {
|
||||
local i past=""
|
||||
RENDER_ARGS=()
|
||||
for ((i = 0; i < ${#ARGS[@]}; i++)); do
|
||||
RENDER_ARGS+=("${ARGS[i]}")
|
||||
[ "${ARGS[i]}" != -- ] || past=1
|
||||
if [ -z "$past" ] && [ "${ARGS[i]}" = --copy ]; then RENDER_ARGS+=("$1"); i=$((i + 1)); fi
|
||||
done
|
||||
}
|
||||
|
||||
# The new args file, written beside the old one before anything stops, moved into place only after
|
||||
# confirm passes. $1 is the copy.
|
||||
prepare_args() {
|
||||
args_with "$1"
|
||||
NEW_ARGS="$(mktemp "$ETC/.$INSTANCE.args.XXXXXX")" &&
|
||||
printf '%s\n' "${RENDER_ARGS[@]}" > "$NEW_ARGS" && chmod --reference="$ETC/$INSTANCE.args" "$NEW_ARGS" ||
|
||||
{ rm -f "${NEW_ARGS:-}"; refuse args-failed "cannot write the new args file in $ETC"; }
|
||||
}
|
||||
drop_args() { rm -f "${NEW_ARGS:-}"; }
|
||||
|
||||
# --- A1 ---
|
||||
|
||||
# OLD: the copy both installed units run, from their WorkingDirectory under the stack's copies.
|
||||
read_old() {
|
||||
local f ids=() id n
|
||||
for f in "$UNIT_DIR/[email protected]" "$UNIT_DIR/$CON_UNIT"; do
|
||||
[ -f "$f" ] && [ ! -L "$f" ] || refuse units-unreadable "no regular file $f"
|
||||
n="$(grep -c '^WorkingDirectory=' "$f")"
|
||||
id="$(awk -v c="WorkingDirectory=$COPIES_DIR/" 'index($0, c) == 1 { print substr($0, length(c) + 1) }' "$f")"
|
||||
[ "$n" = 1 ] && [[ "$id" =~ ^[0-9a-f]{40}$ ]] || refuse units-unreadable "$f names no copy under $COPIES_DIR"
|
||||
ids+=("$id")
|
||||
done
|
||||
[ "${ids[0]}" = "${ids[1]}" ] || refuse units-disagree "$UNIT_DIR's units name ${ids[0]} and ${ids[1]}"
|
||||
OLD="${ids[0]}"
|
||||
logl "units rc=0 old: $OLD"
|
||||
[ "$ARGS_COPY" = "$OLD" ] || refuse args-differ "$ETC/$INSTANCE.args names $ARGS_COPY and the units name $OLD"
|
||||
}
|
||||
|
||||
# r6 A1 and row 240: the host's profile still matches, on every host.
|
||||
hostcheck() {
|
||||
[ -f "$HOSTCHECK" ] && [ -x "$HOSTCHECK" ] || refuse profile-drift "no executable $HOSTCHECK (row 240)"
|
||||
step hostcheck "$HOSTCHECK" --check || refuse profile-drift "$HOSTCHECK --check failed"
|
||||
}
|
||||
|
||||
# The stack's user's --plan for the run, $@ its op and arguments; any refusal stops here.
|
||||
plan() {
|
||||
local rc=0
|
||||
step plan "${AS_USER[@]}" "$SR" "$@" --units external --copies "$COPIES_DIR" --plan --expect-from "$OLD" || rc=$?
|
||||
[ "$rc" = 0 ] || refuse plan-refused "the stack's user's $1 --plan exited $rc; its output above names the refusal"
|
||||
}
|
||||
|
||||
# --- The units ---
|
||||
|
||||
stop_wait() {
|
||||
local u s deadline=$((SECONDS + TIMEOUT)) rc=0
|
||||
step stop-console "$SYSTEMCTL" stop "$CON_UNIT" || rc=1
|
||||
step stop-bus "$SYSTEMCTL" stop "$BUS_UNIT" || rc=1
|
||||
for u in "$CON_UNIT" "$BUS_UNIT"; do
|
||||
while :; do
|
||||
s="$("$SYSTEMCTL" is-active "$u" 9>&- 2>/dev/null)"
|
||||
case "$s" in inactive|failed) break ;; esac
|
||||
[ "$SECONDS" -lt "$deadline" ] || { rc=1; break; }
|
||||
sleep 1
|
||||
done
|
||||
case "$s" in inactive|failed) logl "wait-down rc=0 unit: $u state: $s" ;; *) logl "wait-down rc=1 unit: $u state: ${s:-unknown}" ;; esac
|
||||
done
|
||||
return "$rc"
|
||||
}
|
||||
stop_units() { "$SYSTEMCTL" stop "$CON_UNIT" 9>&- || true; "$SYSTEMCTL" stop "$BUS_UNIT" 9>&- || true; logl "stop-units rc=0 units: $CON_UNIT $BUS_UNIT"; }
|
||||
|
||||
# A2f: the old units back on the copy they never left, gated on a start at or after this second.
|
||||
# An A2 that timed out leaves a unit on its old start, which the gate refuses as stale-start.
|
||||
a2f() {
|
||||
local cause="$1" since
|
||||
drop_args
|
||||
since="$(date -u +%s)"
|
||||
step start-bus "$SYSTEMCTL" start "$BUS_UNIT"
|
||||
step start-console "$SYSTEMCTL" start "$CON_UNIT"
|
||||
if step gate "${AS_USER[@]}" "$SR" gate "$OLD" "$BUSINESS" --units external --copies "$COPIES_DIR" --since "$since" --timeout "$TIMEOUT"; then
|
||||
finish 1 failed "$cause failed; $OLD is running again and passed the gate"
|
||||
fi
|
||||
stop_units
|
||||
finish 2 rollback-failed "$cause failed and $OLD failed its gate after the restart; both units are stopped"
|
||||
}
|
||||
|
||||
# R1-R4: back to OLD after the stack's user staged $1. $2 is the cause: A3, A4 or A5.
|
||||
rollback_to_old() {
|
||||
local id="$1" cause="$2"
|
||||
drop_args
|
||||
stop_wait || { stop_units; finish 2 rollback-failed "$cause failed and the units did not stop for the rollback; both units are stopped"; }
|
||||
step revert "${AS_USER[@]}" "$SR" revert "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and revert failed; both units are stopped"; }
|
||||
args_with "$OLD"
|
||||
step render-old "$RENDER" "${RENDER_ARGS[@]}" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the render of $OLD failed; both units are stopped"; }
|
||||
# confirm --rollback exits 1 when OLD passed its gate and 2 when it failed; anything else is a
|
||||
# refusal, with OLD's gate never run.
|
||||
local rc=0
|
||||
step confirm-rollback "${AS_USER[@]}" "$SR" confirm "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" --rollback "$OLD" --cause "$cause" --timeout "$TIMEOUT" || rc=$?
|
||||
[ "$rc" != 1 ] || finish 1 failed "$cause failed; rolled back to $OLD, which passed the gate"
|
||||
stop_units
|
||||
finish 2 rollback-failed "$cause failed and the rollback to $OLD did not pass (confirm --rollback exited $rc); both units are stopped"
|
||||
}
|
||||
|
||||
# A2 to A5 for copy $1, once A1 and any rehearsal passed. $@ after it is the stack's user's op.
|
||||
run_steps() {
|
||||
local id="$1" rc=0; shift
|
||||
prepare_args "$id"
|
||||
# A2
|
||||
stop_wait || a2f A2
|
||||
# A3
|
||||
step stage "${AS_USER[@]}" "$SR" "$@" --units external --copies "$COPIES_DIR" || rc=$?
|
||||
case "$rc" in
|
||||
0) ;;
|
||||
3) a2f A3 ;;
|
||||
*) rollback_to_old "$id" A3 ;;
|
||||
esac
|
||||
# A4
|
||||
args_with "$id"
|
||||
step render "$RENDER" "${RENDER_ARGS[@]}" || rollback_to_old "$id" A4
|
||||
# A5
|
||||
step confirm "${AS_USER[@]}" "$SR" confirm "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" --timeout "$TIMEOUT" ||
|
||||
rollback_to_old "$id" A5
|
||||
if mv -f "$NEW_ARGS" "$ETC/$INSTANCE.args"; then logl "args rc=0 copy: $id"
|
||||
else logl "args rc=1 copy: $id"; finish 1 args-failed "$id is live and confirmed, but $ETC/$INSTANCE.args still names $OLD; put $NEW_ARGS in its place"
|
||||
fi
|
||||
}
|
||||
|
||||
do_upgrade() {
|
||||
read_args
|
||||
# Before A1: the bundle into the stack's user's clone, then the copy from it (I1).
|
||||
step fetch "${AS_USER[@]}" git -C "$CLONE" fetch -q "$BUNDLE" refs/heads/refactor:refs/remotes/origin/refactor ||
|
||||
refuse fetch-failed "the stack's user could not fetch $BUNDLE into $CLONE"
|
||||
step ff "${AS_USER[@]}" git -C "$CLONE" merge -q --ff-only "$SHA" ||
|
||||
refuse fetch-failed "the stack's user could not fast-forward $CLONE's refactor to $SHA"
|
||||
step copy "${AS_USER[@]}" "$CLONE/scripts/stack-copy.sh" copy --bundle "$BUNDLE" "$SHA" --copies "$COPIES_DIR" ||
|
||||
refuse copy-failed "the stack's user could not make copy $SHA from $BUNDLE"
|
||||
# A1
|
||||
local head
|
||||
head="$(git -C "$REPO" rev-parse HEAD 2>/dev/null)"
|
||||
[ "$head" = "$SHA" ] || refuse head-mismatch "this checkout, $REPO, is at ${head:-unknown}, not $SHA"
|
||||
[ -x "$COPIES_DIR/$SHA/scripts/mosaic" ] || refuse copy-missing "no $COPIES_DIR/$SHA/scripts/mosaic"
|
||||
hostcheck
|
||||
read_old
|
||||
local -a migrate=()
|
||||
[ "$MIGRATE" = 0 ] || migrate=(--migrate)
|
||||
plan activate "$SHA" "$BUSINESS" "${migrate[@]}"
|
||||
rehearse
|
||||
run_steps "$SHA" activate "$SHA" "$BUSINESS" "${migrate[@]}"
|
||||
finish 0 ok "upgrade ok: $SHA is live for $BUSINESS"
|
||||
}
|
||||
|
||||
# The rehearsal against the log's newest backup, before anything stops. Its output goes to the
|
||||
# terminal; root keeps only the BACKUP_STORE line, so the go and the log show which store it used.
|
||||
rehearse() {
|
||||
local cap rc bs
|
||||
cap="$(mktemp "$RUN_DIR/rehearse-$INSTANCE.XXXXXX")" || refuse rehearsal-failed "cannot make a capture file in $RUN_DIR"
|
||||
local -a cmd=("${AS_USER[@]}" "$SR" rehearse "$SHA" "$BUSINESS" --copies "$COPIES_DIR" --backup last --port "$R_PORT" --timeout "$TIMEOUT")
|
||||
"${cmd[@]}" 9>&- 2>&1 | tee "$cap" 9>&-
|
||||
rc="${PIPESTATUS[0]}"
|
||||
logl "rehearse rc=$rc argv:$(printf ' %q' "${cmd[@]}")"
|
||||
bs="$(grep -E '^BACKUP_STORE [0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])? schema [0-9a-z]{1,16}$' "$cap" | tail -n 1)"
|
||||
rm -f "$cap"
|
||||
logl "rehearse-store rc=0 ${bs:-BACKUP_STORE none}"
|
||||
case "$rc" in
|
||||
0) [ "$NO_REHEARSAL" = 0 ] || refuse rehearsal-required "the log records a backup and the rehearsal ran; --no-rehearsal is only for a stack with none" ;;
|
||||
3) [ "$NO_REHEARSAL" = 1 ] || refuse no-backup "the log records no backup to rehearse against; only a go that names --no-rehearsal goes on"
|
||||
logl "rehearsal-skipped rc=0 the log records no backup and the go names --no-rehearsal" ;;
|
||||
*) refuse rehearsal-failed "the rehearsal exited $rc" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# r6 rollback: to the log's previous copy, which the stack's user checks against --to. No HEAD
|
||||
# check: the target is an older copy than this checkout, and no rehearsal.
|
||||
do_rollback() {
|
||||
read_args
|
||||
[ -x "$COPIES_DIR/$TO/scripts/mosaic" ] || refuse copy-missing "no $COPIES_DIR/$TO/scripts/mosaic"
|
||||
hostcheck
|
||||
read_old
|
||||
local -a restore=()
|
||||
[ -z "$RESTORE" ] || restore=(--restore "$RESTORE")
|
||||
[ -z "$EVENTS" ] || restore+=(--events "$EVENTS")
|
||||
plan rollback "$BUSINESS" --to "$TO" "${restore[@]}"
|
||||
run_steps "$TO" rollback "$BUSINESS" --to "$TO" "${restore[@]}"
|
||||
finish 0 ok "rollback ok: $TO is live for $BUSINESS"
|
||||
}
|
||||
|
||||
case "$OP" in
|
||||
upgrade) do_upgrade ;;
|
||||
rollback) do_rollback ;;
|
||||
esac
|
||||
Reference in New Issue
Block a user