salvage 2026-11-06: uncommitted work from scratch/sage-r233/wt233 (disk cleanup #824)

This commit is contained in:
mos committed 2026-11-06 10:06:14 -06:00
1 parent 5465fd52d5
commit 0955a1c76b
3 files changed
+148 -4

No files matched your search

+82 -2
View File
@@ -669,7 +669,7 @@ pinned copy](../packages/cli/README.md#one-pinned-copy-row-85-1579)).
#### Rehearsal on a scratch root (D2, row 163)
```
scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH --port N [--timeout S]
scripts/stack-release.sh rehearse ID BUSINESS --backup ACTIVATION|PATH|last --port N [--timeout S]
```
`rehearse` starts the target copy's bus host and Console on a copy of a
@@ -680,7 +680,11 @@ runs it against live data. It stops and starts no unit and writes no pointer
or log line.
`--backup` takes an activation id under `<dataRoot>/backups/stack/` or an
absolute path. `--port` is the scratch Console's loopback port, 1024 to
absolute path. `--backup last` (row 233) takes the newest backup the
activation log records and prints `PASS backup-last` naming it. With none in
the log it fails `no-backup`, exits 3 and makes no directory. After the
backup's checks it prints `BACKUP_STORE <activation id> schema <version>`.
`--port` is the scratch Console's loopback port, 1024 to
65535. The run refuses `port-reserved` for 7332, 7333 and the pointer's
`--port` and `--tester-port`, and `port-in-use` for a port something holds;
it never picks another port. `--timeout` (default 60 seconds) bounds the
@@ -779,6 +783,82 @@ instance (the business's `launch.by`), for this business, whose run started at
or after the floor. `status` adds an
`awaiting units:` line for an open activation.
Under `--units external`, `activate --plan` also refuses `open-activation`
while any rollback set holds `AWAITING-UNITS`, and `migrate-required` when
the target migrates the store and the run lacks `--migrate` (row 233). Under
`--units user` the plan's bytes are unchanged.
#### Root's upgrade driver (`scripts/stack-upgrade.sh`, row 233)
```
scripts/stack-upgrade.sh upgrade --instance dev --bundle FILE --sha SHA [--migrate] [--no-rehearsal]
[--rehearse-port N] [--timeout S]
scripts/stack-upgrade.sh rollback --instance dev --to ID [--restore ACT [--events N]] [--timeout S]
```
Root runs it from a root-owned checkout of the reviewed commit, never from a
copy, because the stack's user can write the copies. Each run needs Mos's go,
and `--migrate` needs Jason present (decision 96). The host must be prepared
first (`docs/guides/host-run-sheet.md`). Only the default instance, `dev`, is
accepted; a named instance's units are a later row.
The unit values come from `/etc/mosaic-stack/dev.args`, which I2's install
writes: one `case2-unit.sh` argument per line, the serve flags after `--`,
owned by root with no group or other write, exactly one `--copy`, and that
copy the one both installed units run. Otherwise the run refuses
`args-unreadable` or `args-differ`. The driver writes the new copy id into
that file only after `confirm` passes.
`upgrade`, in order. Every stack user step runs as
`sudo -u USER -H env PATH=<node dir>:/usr/local/bin:/usr/bin:/bin` from the
user's clone `<home>/src/mosaic-stack`:
1. Takes `flock -n` on `/run/mosaic-stack/upgrade-dev.lock` and holds it to
the end, rollback included. A second run refuses `upgrade-running`, and
a lock it can't open refuses `lock-failed`.
2. Fetches the bundle's `refactor` into the clone and fast-forwards it to
SHA (`fetch-failed`), then `stack-copy.sh copy --bundle FILE SHA`
(`copy-failed`).
3. A1: this checkout's HEAD is SHA (`head-mismatch`), the copy exists
(`copy-missing`), `/usr/local/lib/mosaic-stack/hostcheck.sh --check`
exits 0 (`profile-drift`, row 240), both units name one copy
(`units-unreadable`, `units-disagree`) and it is the args file's
(`args-differ`), and
`activate SHA BUSINESS --units external --plan --expect-from OLD` exits 0
(`plan-refused`, which covers `open-activation`, `migrate-required` and
`pointer-differs`).
4. The rehearsal, `rehearse SHA BUSINESS --backup last --port 7339`
(`--rehearse-port`). Exit 0 goes on. Exit 3 means no backup in the log
and goes on only with `--no-rehearsal`, logging `rehearsal-skipped`;
without the flag it refuses `no-backup`. Exit 0 with `--no-rehearsal`
refuses `rehearsal-required`; any other exit refuses `rehearsal-failed`.
5. A2 stops the Console, then the bus, and waits for both to read inactive
or failed. A3 is `activate --units external`, A4 the render with the new
`--copy` (which also starts the bus, then the Console), A5 `confirm`.
The driver writes the new args file beside the old one before A2
(`args-failed`, exit 3) and moves it into place after A5. A failed move
exits 1 `args-failed`: the new copy is live and confirmed, and root puts
the file in place by hand.
A failure in A2 or a stage that exits 3 starts the old units again and gates
them (A2f). A later failure stops both units, runs `revert`, renders the old
copy and runs `confirm --rollback OLD --cause A3|A4|A5`. `rollback` is the
same from A1 on, with `rollback BUSINESS --to ID`, no fetch, no copy and no
rehearsal; `--restore` and `--events` pass through.
Root reads no output the stack user wrote except exit codes and the
rehearsal's `BACKUP_STORE` line, which it copies into its log after a match
against a fixed pattern (`BACKUP_STORE none` otherwise). The step log
`/var/log/mosaic-stack/upgrade-dev.log`, 0640, gets one
`<time> <step> rc=N argv: ...` line per step and a last
`<time> result exit=N <code> <text>` line.
Exit 0 ok, 1 failed with the old stack running and gated, 2 rollback-failed
with both units stopped, 3 refused with nothing stopped, 4 usage. A refusal
after step 2 leaves the user's clone fast-forwarded and the new copy made. After exit
2 the activation stays open, so the next run's plan refuses until root
finishes it.
## Console service (`scripts/webui-service.sh`)
```bash
+43
View File
@@ -2062,6 +2062,49 @@ test('R15 a rehearsal stopped by TERM, INT or HUP fails signal and still stops t
}
});
// --- Row 233 (#1735): --backup last, the newest backup the log records (sage-1650, sage-1653) ---
// The log's lines, as the runs write them, with only the fields from_log reads set.
function rlog(e,...ls){
mkdirSync(e.state,{recursive:true});
writeFileSync(e.log,ls.map(([op,to,backup],i)=>JSON.stringify({at:`2026-10-20T00:00:0${i}Z`,op,from:null,to,backup,activation:`20261020T00000${i}Z-${to.slice(0,12)}`,result:'ok'})+'\n').join(''));
}
test('row 233: --backup last takes the newest backup the log records; a later line with backup null does not hide it, and the store line names it',async(t)=>{
const e=await rmk(t);
// An older backup that isn't on disk: a run that took it would fail backup-missing.
rlog(e,['activate',C1,`20261019T000000Z-${C1.slice(0,12)}`],['activate',B,e.act],['rollback',B,null]);
const l0=readFileSync(e.log,'utf8'),before=untouched(e);
const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','10');
assert.equal(r.code,0,r.out+r.err);
assert.deepEqual(r.names,['PASS backup-last',...PRE,'PASS migration','PASS gate','PASS no-session',...POST]);
assert.ok(r.out.startsWith(`PASS backup-last: the log's newest backup is ${e.act}\n`),r.out);
assert.ok(r.out.includes(`\nBACKUP_STORE ${e.act} schema 3g\n`),r.out);
assert.equal(readFileSync(e.log,'utf8'),l0);assert.equal(untouched(e),before);
// The rehearsal's own steps.log has the PASS and FAIL lines only.
assert.deepEqual(lines(join(r.dir,'steps.log')).map((l)=>l.replace(/^\S+ /,'')),r.out.split('\n').filter((l)=>/^(PASS|FAIL) |^REHEARSAL_/.test(l)));
});
test('row 233: --backup last with no backup in the log, or no log, exits 3 with no-backup and makes no rehearsal directory',async(t)=>{
const e=await rmk(t);
for(const how of ['no log','adopt only']){
if(how==='adopt only')rlog(e,['adopt',B,null]);
const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','3');
assert.equal(r.code,3,how+r.out+r.err);
assert.equal(r.out,'FAIL no-backup: the log records no backup\nREHEARSAL_FAIL\n',how);
assert.equal(r.dir,null);assert.deepEqual(rDirs(e),[],how);assert.ok(!existsSync(join(e.data,'rehearsals')),how);
}
});
test('row 233: --backup last refuses a newest backup that is not an activation id, with exit 1',async(t)=>{
const e=await rmk(t);
rlog(e,['activate',B,'../../etc']);
const r=await rehearse(e,B,'--backup','last','--port',await freePort(),'--timeout','3');
assert.equal(r.code,1,r.out+r.err);
assert.equal(r.out,'FAIL backup-unreadable: the log\'s newest backup is not an activation id\nREHEARSAL_FAIL\n');
assert.deepEqual(rDirs(e),[]);
});
// --- Row 190 (#1688): stack instances on one host. Design and the T-I table:
// agents/darkwing/work/queue-190/DESIGN.md. T-I1, T-I2, T-I5, T-I10 and the loader half of T-I7 are in
// stack-instances.test.mjs; these need the units, the stand-ins and stack-release.sh. Every existing
+23 -2
View File
@@ -130,7 +130,7 @@ parse_opts() {
--sso) [ "$OP" = activate ] || usage; SSO=1; shift ;;
--migrate) [ "$OP" = activate ] || usage; MIGRATE=1; shift ;;
--restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || usage; RESTORE="$2"; shift 2 ;;
--backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
--backup) [ "$OP" = rehearse ] && [ $# -ge 2 ] && { [ "$2" = last ] || [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || [[ "$2" =~ ^/[A-Za-z0-9/._+-]*$ ]]; } || usage; R_BACKUP="$2"; shift 2 ;;
--port) [ "$OP" = rehearse ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{1,5}$ ]] && [ "$((10#$2))" -ge 1024 ] && [ "$((10#$2))" -le 65535 ] || usage; R_PORT="$((10#$2))"; shift 2 ;;
--events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
--units) [ "$OP" != rehearse ] && [ $# -ge 2 ] && { [ "$2" = user ] || [ "$2" = external ]; } || usage; UNITS="$2"; shift 2 ;;
@@ -220,7 +220,8 @@ for(const k of open.keys())process.stdout.write(k+"\n");' -- "$LOG"
# Reads the log's ok lines, newest first. `previous` prints rollback's target: the first `to`
# that is not the pointer's copy (release.sh's rule). `backup` prints the backup field of the
# line that made the pointer's copy live, the backup a rollback would restore. `arrival` prints the
# line that made the pointer's copy live, the backup a rollback would restore. `newest` prints the
# backup field of the newest line that has one, rehearse's --backup last (row 233). `arrival` prints the
# activation and `from` of the line that moved the stack onto the pointer's copy from another one,
# past any same-copy re-pin after it (row 200, sage-1389).
from_log() {
@@ -233,6 +234,7 @@ for(const l of lines.reverse()){
if(o.result!=="ok"||!["adopt","activate","rollback"].includes(o.op)||!o.to)continue;
if(mode==="previous"&&o.to!==current){process.stdout.write(o.to+"\n");process.exit(0)}
if(mode==="backup"&&o.to===current){if(o.backup)process.stdout.write(o.backup+"\n");process.exit(o.backup?0:1)}
if(mode==="newest"&&o.backup){process.stdout.write(o.backup+"\n");process.exit(0)}
if(mode==="arrival"&&o.to===current&&o.from!==current){process.stdout.write(`${o.activation??""} ${o.from??""}\n`);process.exit(0)}
}
process.exit(1);' -- "$LOG" "$1" "$P_COPY"
@@ -1262,6 +1264,8 @@ plan() {
echo "needs: --migrate$([ "$MIGRATE" = 1 ] && echo ' (given)' || echo ' (not given; the run would roll back with migration-needs-flag)')"
echo "needs: Jason present for this run (decision 96 condition 2)"
[ "$MIGRATE" = 1 ] || early=1
# Row 233: root reads only the plan's exit, so under external a missing --migrate refuses here.
[ "$UNITS" = user ] || [ "$MIGRATE" = 1 ] || PLAN_REFUSE+=("migrate-required: the store is $S_SCHEMA and $TO migrates it to $T_SCHEMA; the go names --migrate")
elif [ "$OP" = activate ]; then
echo "migration: none possible; $S_SCHEMA is not in $T_SCHEMA's migration chain, so the run would roll back with store-newer"
early=1
@@ -1452,6 +1456,13 @@ ext_pointer_checks() {
local open
open="$(interrupted | paste -sd' ' -)"
[ -z "$open" ] || check_refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)"
# Row 233: root can't read the stack's state, so the plan lists any rollback set still waiting
# for units, whatever the log says.
local m sets=""
local -a markers
mapfile -t markers < <(compgen -G "$STATE/stack-rollback/*/AWAITING-UNITS" || true)
for m in "${markers[@]}"; do sets+=" $(basename "$(dirname "$m")")"; done
[ -z "$sets" ] || check_refuse open-activation "rollback set(s)$sets hold AWAITING-UNITS; root finishes them with confirm or revert first"
}
# A3's checks before `started`. Root has stopped the units and installs them after this run, so
@@ -1973,6 +1984,9 @@ r_env() { env -i HOME="$HOME" PATH="$PATH" MOSAIC_CONFIG="$R/config/config.json"
r_migration() {
local copy="$COPIES/$TO" old="$COPIES/$P_COPY" v out rc
v="$(node -e "$STORE_JS" -- version "$R/data/bus/bus.sqlite" 2>&1)" || r_abort store-unreadable "$v"
# Row 233 (sage-1653): which store the rehearsal ran against, for root's step log. Output only:
# the rehearsal's own steps.log has the backup's path on its backup line.
echo "BACKUP_STORE $R_BACKUP schema $v"
if [ "$v" = "$T_SCHEMA" ]; then r_pass migration "not needed: the backup is already $v"; return 0; fi
in_chain "$v" || r_abort store-newer "the backup is $v, which $TO's migration chain does not start from"
(umask 077; mkdir "$R/refuse" "$R/refuse/bus" && cp "$R_BK"/bus/* "$R/refuse/bus/") || r_abort scratch-build "cannot copy the backup to $R/refuse"
@@ -2043,6 +2057,13 @@ do_rehearse() {
trap 'r_fail signal "the rehearsal was interrupted"; r_end' INT TERM HUP
read_pointer || r_abort no-pointer "no $POINTER; the pointer's copy is the old copy the checks use"
[ "$P_BUSINESS" = "$BUSINESS" ] || r_abort business-mismatch "the pointer is for $P_BUSINESS"
# Row 233 (sage-1650, sage-1653): last is the newest backup the log records. With none, exit 3
# and no directory, the one rehearse refusal stack-upgrade.sh reads as no-backup.
if [ "$R_BACKUP" = last ]; then
R_BACKUP="$(from_log newest)" || { r_fail no-backup "the log records no backup"; R_RC=3; r_end; }
[[ "$R_BACKUP" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}(-[1-9])?$ ]] || r_abort backup-unreadable "the log's newest backup is not an activation id"
r_pass backup-last "the log's newest backup is $R_BACKUP"
fi
copy_exists "$TO" 2>/dev/null || r_abort copy-missing "no copy $TO in $COPIES"
copy_ok "$TO" 2>/dev/null || r_abort copy-unreachable "$TO is not reachable from refactor and origin/refactor"
target_schema "$TO" || r_abort copy-schema-unknown "cannot read SCHEMA_VERSION and its chain from $COPIES/$TO/packages/bus/src/store.mjs"