scripts: health-gated stack activate and rollback, stack-release.sh (row 162, D1, #1659)
New scripts/stack-release.sh with four verbs. adopt records the running copy, the unit flags and the store schema in state/stack-active.json; it stops nothing. activate ID BUSINESS [--pm] [--migrate] stops the units, checks the stopped store, backs it up, migrates when the target's schema differs, installs and starts the target, then gates on both units active, a live bus host whose pid is the bus unit's MainPID with boot ok, Console /healthz, stack-copy.sh check and one launched journal line. A failed gate rolls back inside the same run and keeps the attempted store as bus.post-attempt-<stamp>. rollback [--restore ACT [--events N]] goes back to the log's previous copy and counts the events written after the backup. status prints the pointer, the log tail and the copy check. --plan prints what a run would do and writes nothing; it reads the store schema from the store the running bus holds, or from a stopped store. Exit codes: refusal 3, gate failure with rollback 1, rollback failed 2, usage 4. Nothing is pruned. stack-copy.sh gets a reachable function split from copy_commit and a guard so its helpers can be sourced. TOOLS.md "Pinned stack update" and the cli README pinned-stack paragraph describe the verbs. Owner Darkwing, brief docs/plans/2026-10-18_stack-release.md (D1). Reviewer Filbert: r1 approve (comment 28997) on 091693c5, r2 confirm-only approve (29004) on candidate manifest c213a801...bd15c after row 125 moved the base under TOOLS.md and the cli README. Sage gate on23622980f: 25 of 25 steps rc=0 on c96a95320 (15 packages + 10 scripts), node pass 2353 fail 0, selftests 422 passed 0 failed, 2026-10-18T20:32:13Z to 20:38:55Z, network sage-r162_default removed, 0 containers. Nothing live. The8e7ea89a1window (repin-8e7ea89a1.md) runs these verbs on Mos's go with Jason present. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Claude Opus 5.5
+59
-33
@@ -342,43 +342,69 @@ reports checkout or copy identity and refactor drift, and writes nothing.
|
||||
|
||||
### Pinned stack update
|
||||
|
||||
```
|
||||
scripts/stack-release.sh adopt BUSINESS
|
||||
scripts/stack-release.sh activate ID BUSINESS [--pm] [--migrate] [--timeout S] [--plan]
|
||||
scripts/stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan]
|
||||
scripts/stack-release.sh status BUSINESS
|
||||
common options: --copies DIR, --bin DIR
|
||||
```
|
||||
|
||||
This is an operator plan, not an action agents run. Both reviews and the
|
||||
landing precede it. Mos grants each live step first. A bus restart ends
|
||||
running sessions; do this only when no PM turn is in flight.
|
||||
landing come before it. Mos grants each live run first. A bus restart ends
|
||||
running sessions, so run it only when no PM turn is in flight. `--plan` and
|
||||
`status` change nothing and need no go.
|
||||
|
||||
1. Make the pushed target copy with `stack-copy.sh copy COMMIT` and note
|
||||
its full id. Before the first update, preserve the current Q81 CLI wrapper
|
||||
(bytes and mode), bus and Console unit files and drop-ins, and the effective
|
||||
WorkingDirectory/ExecStart output from each status command. Record the bus's
|
||||
--claude pin and the Console's preview/send flags. Keep the bus pm.conf and
|
||||
the existing Console copy or checkout: the first rollback needs them.
|
||||
On later updates, also record the prior shared copy's full id.
|
||||
2. With Mos's go, stop the Console, then stop the bus.
|
||||
3. Mos removes overriding bus drop-ins identified by status. The first
|
||||
update removes `~/.config/systemd/user/[email protected]/pm.conf`.
|
||||
Reload systemd after removal; keep unrelated environment drop-ins.
|
||||
4. With Mos's go, install the bus with `bus-service.sh install --copy ID --pm`
|
||||
and its existing --claude pin, then the Console with
|
||||
`webui-service.sh install --copy ID` and its existing preview/send flags.
|
||||
5. With Mos's go, run `stack-copy.sh wrapper ID` to switch the human CLI.
|
||||
6. Mos starts the bus first, then the Console. Read `stack-copy.sh check`,
|
||||
both service status commands, Console `/healthz` and the canonical
|
||||
`scripts/mosaic launches list`; the latter must show the PM.
|
||||
7. Jason's plain `mosaic talk pm "..."` must print sent and a reply before
|
||||
#225 closes. Keep the receipts for the three paths and reply.
|
||||
The script keeps `<dataRoot>/state/stack-active.json` (the live copy, business,
|
||||
PM flag, claude pin, Console flags, schema and activation) and appends every
|
||||
run to `<dataRoot>/state/stack-activation-log.jsonl`: a `started` line, then
|
||||
one final line with `result` `ok`, `refused`, `gate-failed` or
|
||||
`rollback-failed`. A refusal exits 3, a gate failure that rolled back exits 1,
|
||||
a failed rollback exits 2, usage exits 4. Only one run at a time holds
|
||||
`state/stack-release.lock` (`run-locked`).
|
||||
|
||||
The first rollback restores the pre-update mixed setup, which has no shared
|
||||
copy id. Each live step needs Mos's go: stop the Console then the bus;
|
||||
reinstall the checkout bus unit without --copy or --pm, retaining its saved
|
||||
--claude pin; restore the saved pm.conf and other bus drop-ins; restore the
|
||||
saved Console unit and drop-ins pointing to its original checkout or
|
||||
mosaic-console copy; restore the saved Q81 wrapper bytes and mode; reload
|
||||
systemd; start the bus then the Console. Check their effective paths against
|
||||
the saved status output, Console /healthz and the PM launch. The shared-copy
|
||||
check is expected to refuse this restored mixed setup; #225 remains open.
|
||||
1. Once per host, `adopt BUSINESS` records the running stack as the pointer:
|
||||
the copy `stack-copy.sh check` reports, the bus unit's `--pm` and
|
||||
`MOSAIC_CLAUDE_BIN`, the Console's flags without `--checkout`, and the
|
||||
copy's schema version. It refuses when the check fails, the bus unit has no
|
||||
`MOSAIC_CLAUDE_BIN` or a pointer exists.
|
||||
2. Make the pushed target copy with `stack-copy.sh copy COMMIT` and note its
|
||||
full id.
|
||||
3. Run `activate ID BUSINESS --plan` and take the output to Mos. It names the
|
||||
store's schema and the target's, any migration with `--migrate` and the
|
||||
decision 96 condition 2 line (Jason present), the commits the run adds,
|
||||
and the three install commands. It refuses `copies-unrelated` when neither
|
||||
copy is an ancestor of the other.
|
||||
4. Mos removes any bus drop-in that overrides ExecStart or WorkingDirectory;
|
||||
`activate` refuses `drop-in-override` and names the file.
|
||||
5. With Mos's go, `activate ID BUSINESS`, plus `--migrate` when the plan
|
||||
says so. Before it stops anything, it refuses a missing, unpushed or
|
||||
unreadable copy, a busy `mosaic-bus@` instance, and unit flags that differ
|
||||
from the pointer. Then it saves the unit files, drop-ins and wrapper under
|
||||
`state/stack-rollback/<activation>/`, stops the Console then the bus, and
|
||||
reads the stopped store. A `-wal` or `-shm` beside it refuses
|
||||
`store-unclean` and leaves the store alone. A migration takes an
|
||||
owner-only backup under `<dataRoot>/backups/stack/<activation>/` first.
|
||||
It installs the bus, the Console and the wrapper from the pointer's flags,
|
||||
starts the bus then the Console, and runs the gate: both units active, a
|
||||
live bus host for this start, Console `/healthz`, `stack-copy.sh check`
|
||||
and, with `--pm`, exactly one `launched` journal line for a live run of
|
||||
this start. A pass writes the pointer.
|
||||
6. A gate failure puts back the saved units and wrapper, restores the backup
|
||||
when a migration ran, keeps the store it replaced as
|
||||
`bus.post-attempt-<stamp>`, starts the old copy and gates it. If that gate
|
||||
fails too, both units stay stopped, the log says `rollback-failed`, and
|
||||
Jason or Mos settles it from the rollback set the message names.
|
||||
7. `rollback BUSINESS` returns to the copy before the pointer's, taken from
|
||||
the log. When the schemas differ it needs `--restore ACTIVATION`; events
|
||||
written after that backup need `--events N` with the count it prints, and
|
||||
Jason decides first.
|
||||
8. Jason's plain `mosaic talk pm "..."` must print sent and a reply. Keep the
|
||||
`status` output with the receipts.
|
||||
|
||||
Later rollback repeats steps 2-6 with the prior shared copy's full id, each
|
||||
live step again approved by Mos. Nothing prunes copies or removes mosaic-old.
|
||||
A `started` line with no final line is an interrupted run. `status` shows
|
||||
it, and `adopt`, `activate` and `rollback` refuse until Jason or Mos settles
|
||||
it. Nothing prunes copies, rollback sets, backups or post-attempt stores.
|
||||
|
||||
After a switch, the human commands and `--tester` work only through `mosaic`
|
||||
on PATH; the checkout's `scripts/mosaic` exits 3 with `human-required` or
|
||||
|
||||
+10
-4
@@ -645,10 +645,16 @@ script only warns about dependency version schemes. The copy runs the
|
||||
already published bundle and does not build it. No allowScripts change is
|
||||
needed for these paths; adding a runtime loader requires a fresh audit.
|
||||
|
||||
The one update and rollback plan is [TOOLS.md, Pinned stack update](../../docs/TOOLS.md#pinned-stack-update).
|
||||
Each live switch/unit change/stop/start requires Mos's go; agents build
|
||||
and test fixtures only. Restarting the bus ends its sessions, so no PM
|
||||
turn may be in flight. Queue writes stay on the canonical scripts/mosaic.
|
||||
`scripts/stack-release.sh` runs the update and the rollback (row 162, D1):
|
||||
`adopt` records the running stack once, `activate ID` switches the bus,
|
||||
Console and wrapper to a copy and gates it, `rollback` returns to the
|
||||
previous copy from the log, and `status` reads. A failed gate rolls back to
|
||||
the old copy. The pointer and the log live under `<dataRoot>/state/`. The
|
||||
steps are in [TOOLS.md, Pinned stack update](../../docs/TOOLS.md#pinned-stack-update).
|
||||
Each live run requires Mos's go; agents build and test fixtures only, and
|
||||
`--plan` and `status` change nothing. Restarting the bus ends its sessions,
|
||||
so no PM turn may be in flight. Queue writes stay on the canonical
|
||||
scripts/mosaic.
|
||||
|
||||
The human commands and `--tester` run only from the tree the bus runs from
|
||||
(row 130, #1625). The broker checks the path of the calling human-cli.mjs or
|
||||
|
||||
@@ -0,0 +1,732 @@
|
||||
// scripts/stack-release.sh (D1, #1659). Design, the T table and the mutants:
|
||||
// agents/darkwing/work/queue-162/BUILD.md. Every run uses a temp data root, temp units and stand-in
|
||||
// systemctl, journalctl and systemd-analyze on PATH; nothing here reads ~/.mosaic-dev or a live unit.
|
||||
import test, {after} from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {execFileSync,spawn,spawnSync} from 'node:child_process';
|
||||
import {mkdtempSync,mkdirSync,writeFileSync,readFileSync,appendFileSync,cpSync,rmSync,chmodSync,existsSync,statSync,readdirSync,realpathSync} from 'node:fs';
|
||||
import {createServer} from 'node:http';
|
||||
import {createHash} from 'node:crypto';
|
||||
import {DatabaseSync} from 'node:sqlite';
|
||||
import {tmpdir} from 'node:os';
|
||||
import {join,dirname,basename} from 'node:path';
|
||||
import {fileURLToPath,pathToFileURL} from 'node:url';
|
||||
import {once} from 'node:events';
|
||||
|
||||
const ROOT=fileURLToPath(new URL('../../..',import.meta.url)).replace(/\/$/,'');
|
||||
const NODE=process.execPath;
|
||||
const MARKER='# mosaic-cli wrapper (Q81, agents/sage/work/q81/mosaic-cli.sh)';
|
||||
const BUS='[email protected]',CON='mosaic-webui.service';
|
||||
const git=(cwd,...args)=>execFileSync('git',['-c','user.name=fixture','-c','user.email=fixture@invalid','-c','commit.gpgsign=false','-c','core.hooksPath=/dev/null','-C',cwd,...args],{encoding:'utf8'}).trim();
|
||||
const script=(fn)=>`#!${NODE}\n(${fn})();\n`;
|
||||
|
||||
// --- Stand-ins. Each is serialized into its own file, so it uses only process and its own scope. ---
|
||||
|
||||
// systemctl --user: unit state in $SD_STATE/units.json, every call in calls.log. A bus start runs
|
||||
// holder.cjs (the store held open in WAL mode, writer.lock, host.json) and, with --pm in the unit's
|
||||
// ExecStart, writes the `launched` journal line and the launches entry unless a pm-<copy> fault
|
||||
// names another case. Faults are files in $SD_STATE/faults named for the copy they hit.
|
||||
function systemctl(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process'),crypto=process.getBuiltinModule('node:crypto');
|
||||
const sd=process.env.SD_STATE,data=process.env.SD_DATA,unitDir=path.join(process.env.XDG_CONFIG_HOME,'systemd','user');
|
||||
const args=process.argv.slice(2).filter((a)=>a!=='--user'),[verb,...rest]=args;
|
||||
fs.appendFileSync(path.join(sd,'calls.log'),args.join(' ')+'\n');
|
||||
const faultFile=(n)=>path.join(sd,'faults',n),fault=(n)=>fs.existsSync(faultFile(n));
|
||||
const stFile=path.join(sd,'units.json');
|
||||
const load=()=>{try{return JSON.parse(fs.readFileSync(stFile,'utf8'))}catch{return {}}};
|
||||
const save=(s)=>{fs.writeFileSync(stFile+'.tmp',JSON.stringify(s));fs.renameSync(stFile+'.tmp',stFile)};
|
||||
const sleep=(ms)=>Atomics.wait(new Int32Array(new SharedArrayBuffer(4)),0,0,ms);
|
||||
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
|
||||
const isBus=(u)=>/^mosaic-bus@[a-z0-9-]+\.service$/.test(u),inst=(u)=>u.slice('mosaic-bus@'.length,-'.service'.length);
|
||||
const unitFile=(u)=>{
|
||||
const main=isBus(u)?'[email protected]':u,file=path.join(unitDir,main);
|
||||
if(!fs.existsSync(file))return null;
|
||||
const drops=[];
|
||||
for(const d of isBus(u)?[main+'.d',u+'.d']:[u+'.d']){const dir=path.join(unitDir,d);if(fs.existsSync(dir))for(const f of fs.readdirSync(dir).sort())if(f.endsWith('.conf'))drops.push(path.join(dir,f))}
|
||||
const r={wd:'',exec:'',env:[],drops};
|
||||
for(const f of [file,...drops])for(let line of fs.readFileSync(f,'utf8').split('\n')){
|
||||
line=line.replace(/%i/g,isBus(u)?inst(u):'').replace(/%h/g,process.env.HOME);
|
||||
const m=/^(WorkingDirectory|ExecStart|Environment)=(.*)$/.exec(line);if(!m)continue;
|
||||
if(m[1]==='WorkingDirectory')r.wd=m[2];else if(m[1]==='ExecStart'){if(m[2])r.exec=m[2]}else r.env.push(m[2]);
|
||||
}
|
||||
return r;
|
||||
};
|
||||
const snap=(ev)=>{
|
||||
const dir=path.join(data,'bus'),files={};
|
||||
try{for(const n of fs.readdirSync(dir).sort()){const p=path.join(dir,n),st=fs.statSync(p);if(st.isFile())files[n]=[st.size,st.mtimeMs,crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')]}}catch{}
|
||||
fs.appendFileSync(path.join(sd,'snap.jsonl'),JSON.stringify({ev,files})+'\n');
|
||||
};
|
||||
const pm=(b,id,pid,off)=>{
|
||||
let mode='';try{mode=fs.readFileSync(faultFile(`pm-${id}`),'utf8').trim()}catch{}
|
||||
const now=Date.now()-off*1000,epoch=Math.floor(now/1000),at=new Date(now).toISOString(),run=`run-${pid}`,unit=`mosaic-bus@${b}.service`;
|
||||
const line=(text)=>fs.appendFileSync(path.join(sd,'journal.jsonl'),JSON.stringify({epoch,unit,text})+'\n');
|
||||
const ok=()=>line(`launched pm-${b}: run ${run}, claude opus`);
|
||||
const lf=path.join(sd,'launches.json');let list=[];try{list=JSON.parse(fs.readFileSync(lf,'utf8'))}catch{}
|
||||
const entry=(o)=>list.push({run,business:b,live:true,startedAt:at,...o});
|
||||
line(`bus host up: business ${b}, socket /fixture/bus.sock, notifier off`);
|
||||
switch(mode){
|
||||
case 'none':break;
|
||||
case 'two':ok();line(`launched pm-${b}: run ${run}b, claude opus`);entry({});break;
|
||||
case 'badword':line(`launched pm-${b}: job ${run}, claude opus`);entry({});break;
|
||||
case 'notlisted':ok();break;
|
||||
case 'early':ok();entry({startedAt:new Date(now-3600e3).toISOString()});break;
|
||||
case 'human':entry({run:`human-${pid}`});break;
|
||||
case 'other':ok();entry({business:'other'});break;
|
||||
case 'dead':ok();entry({live:false});break;
|
||||
default:ok();entry({});
|
||||
}
|
||||
fs.writeFileSync(lf,JSON.stringify(list));
|
||||
};
|
||||
if(verb==='show'){
|
||||
const u=rest[0],props=[];for(let i=1;i<rest.length;i++)if(rest[i]==='-p')props.push(rest[++i]);
|
||||
const f=unitFile(u),st=load()[u]||{},id=f?path.basename(f.wd):'';
|
||||
for(const p of props){
|
||||
if(p==='LoadState')console.log(`LoadState=${f?'loaded':'not-found'}`);
|
||||
else if(p==='WorkingDirectory')console.log(`WorkingDirectory=${f?f.wd+(!isBus(u)&&fault(`check-${id}`)?'-moved':''):''}`);
|
||||
else if(p==='ExecStart')console.log(f&&f.exec?`ExecStart={ path=${f.exec.split(' ')[0]} ; argv[]=${f.exec} ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }`:'ExecStart=');
|
||||
else if(p==='Environment')console.log(`Environment=${f?f.env.join(' '):''}`);
|
||||
else if(p==='MainPID')console.log(`MainPID=${st.pid||0}`);
|
||||
else if(p==='DropInPaths')console.log(`DropInPaths=${f?f.drops.join(' '):''}`);
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
if(verb==='is-active'){const q=rest[0]==='--quiet',u=q?rest[1]:rest[0],s=(load()[u]||{}).state||'inactive';if(!q)console.log(s);process.exit(s==='active'?0:3)}
|
||||
if(verb==='is-enabled'){console.log('disabled');process.exit(1)}
|
||||
if(verb==='daemon-reload')process.exit(0);
|
||||
if(verb==='list-units'){
|
||||
if(fault('list-fails'))process.exit(1);
|
||||
const pats=rest.filter((a)=>!a.startsWith('--')).map((p)=>new RegExp('^'+p.replace(/\./g,'\\.').replace(/\*/g,'.*')+'$'));
|
||||
for(const [u,s] of Object.entries(load()))if(pats.some((re)=>re.test(u)))console.log(`${u} loaded ${s.state} ${s.state==='active'?'running':'dead'} fixture`);
|
||||
process.exit(0);
|
||||
}
|
||||
if(verb==='start'){
|
||||
const u=rest[0],f=unitFile(u);if(!f){console.error(`Unit ${u} not found.`);process.exit(5)}
|
||||
const s=load(),id=path.basename(f.wd);
|
||||
if(!isBus(u)){s[u]={state:'active',pid:null,wd:f.wd};save(s);process.exit(0)}
|
||||
if(fault(`start-dies-${id}`)){s[u]={state:'failed',pid:null,wd:f.wd};save(s);process.exit(0)}
|
||||
snap('start');
|
||||
const off=process.env.SD_CLOCK_OFFSET||'0';
|
||||
// The holder must not inherit stack-release.sh's lock (fd 9) or any other fd above 2.
|
||||
const close='for fd in /proc/self/fd/*; do n=${fd##*/}; [ "$n" -gt 2 ] && eval "exec $n>&-" 2>/dev/null; done; exec "$@"';
|
||||
const ch=cp.spawn('bash',['-c',close,'holder',process.execPath,path.join(__dirname,'..','holder.cjs'),data,inst(u),sd,off],{detached:true,stdio:'ignore'});
|
||||
ch.unref();
|
||||
fs.appendFileSync(path.join(sd,'pids.log'),ch.pid+'\n');
|
||||
const ready=path.join(sd,`ready-${ch.pid}`),end=Date.now()+10000;
|
||||
while(!fs.existsSync(ready)){if(Date.now()>end){console.error('fixture holder did not start');process.exit(1)}sleep(20)}
|
||||
s[u]={state:'active',pid:ch.pid,wd:f.wd};save(s);
|
||||
if(f.exec.split(' ').slice(1).includes('--pm'))pm(inst(u),id,ch.pid,Number(off));
|
||||
process.exit(0);
|
||||
}
|
||||
if(verb==='stop'){
|
||||
const u=rest[0],s=load(),st=s[u];
|
||||
if(st&&st.pid){
|
||||
try{process.kill(st.pid,'SIGTERM')}catch{}
|
||||
const end=Date.now()+10000;while(alive(st.pid)&&Date.now()<end)sleep(20);
|
||||
if(alive(st.pid))try{process.kill(st.pid,'SIGKILL')}catch{}
|
||||
}
|
||||
if(st){s[u]={...st,state:'inactive',pid:null};save(s)}
|
||||
if(isBus(u)){
|
||||
// One-shot: the stop leaves an empty -wal or -shm beside the store, as an unclean close would.
|
||||
for(const [name,ext] of [['stop-leaves-wal','-wal'],['stop-leaves-shm','-shm']])if(fault(name)){fs.rmSync(faultFile(name));fs.writeFileSync(path.join(data,'bus','bus.sqlite'+ext),'')}
|
||||
snap('stop');
|
||||
}
|
||||
process.exit(0);
|
||||
}
|
||||
console.error(`fixture systemctl: unsupported: ${args.join(' ')}`);process.exit(1);
|
||||
}
|
||||
|
||||
// journalctl --user -u UNIT --since @EPOCH -o cat. A bare local time gets exit 1, as M35 needs.
|
||||
function journalctl(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
|
||||
const sd=process.env.SD_STATE,a=process.argv.slice(2);let unit='',since='';
|
||||
for(let i=0;i<a.length;i++){if(a[i]==='-u')unit=a[++i];else if(a[i]==='--since')since=a[++i]}
|
||||
fs.appendFileSync(path.join(sd,'journal-since.log'),since+'\n');
|
||||
if(fs.existsSync(path.join(sd,'faults','journal-fails')))process.exit(1);
|
||||
let from;
|
||||
if(/^@\d+$/.test(since))from=Number(since.slice(1));
|
||||
else{const m=/^(\d{4}-\d\d-\d\d) (\d\d:\d\d:\d\d) UTC$/.exec(since);if(!m)process.exit(1);from=Date.parse(`${m[1]}T${m[2]}Z`)/1000}
|
||||
let lines=[];try{lines=fs.readFileSync(path.join(sd,'journal.jsonl'),'utf8').split('\n').filter(Boolean).map((l)=>JSON.parse(l))}catch{}
|
||||
for(const o of lines)if(o.unit===unit&&o.epoch>=from)console.log(o.text);
|
||||
}
|
||||
|
||||
function systemdAnalyze(){
|
||||
const path=process.getBuiltinModule('node:path');
|
||||
if(process.argv[2]!=='--user'||process.argv[3]!=='unit-paths')process.exit(1);
|
||||
console.log(path.join(process.env.XDG_CONFIG_HOME,'systemd','user'));
|
||||
}
|
||||
|
||||
// The bus host: holds the store open in WAL mode until SIGTERM, as the broker does.
|
||||
function holder(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
|
||||
const [data,business,sd,offset]=process.argv.slice(2),bus=path.join(data,'bus');
|
||||
fs.mkdirSync(bus,{recursive:true,mode:0o700});
|
||||
const db=new DatabaseSync(path.join(bus,'bus.sqlite'));
|
||||
if(!db.prepare("SELECT count(*) AS n FROM sqlite_master WHERE name='meta'").get().n){
|
||||
db.exec('PRAGMA journal_mode=WAL');
|
||||
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
|
||||
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
|
||||
db.prepare("INSERT INTO meta VALUES ('schema_version', '3g')").run();
|
||||
}
|
||||
fs.writeFileSync(path.join(bus,'writer.lock'),String(process.pid));
|
||||
const at=new Date(Date.now()-Number(offset||0)*1000).toISOString(),host=path.join(sd,'host.json');
|
||||
fs.writeFileSync(host+'.tmp',JSON.stringify({host:{business,pid:process.pid,live:true,startedAt:at,notifier:'ok'},boot:{ok:true,business,at}}));
|
||||
fs.renameSync(host+'.tmp',host);
|
||||
process.on('SIGTERM',()=>{db.close();fs.rmSync(path.join(bus,'writer.lock'),{force:true});process.exit(0)});
|
||||
fs.writeFileSync(path.join(sd,`ready-${process.pid}`),'');
|
||||
setInterval(()=>{},1<<30);
|
||||
}
|
||||
|
||||
// A copy's scripts/mosaic: bus status and launches list from what the holder and systemctl wrote.
|
||||
function mosaicStub(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path');
|
||||
const id=path.basename(path.resolve(__dirname,'..')),sd=process.env.SD_STATE,a=process.argv.slice(2).join(' ');
|
||||
const fault=(n)=>fs.existsSync(path.join(sd,'faults',n));
|
||||
const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}};
|
||||
if(a==='bus status --json'){
|
||||
if(fault(`status-${id}`))process.exit(1);
|
||||
let s={host:null,boot:null};
|
||||
try{s=JSON.parse(fs.readFileSync(path.join(sd,'host.json'),'utf8'));s.host.live=alive(s.host.pid)}catch{}
|
||||
console.log(JSON.stringify(s));
|
||||
}else if(a==='bus status')console.log(`bus: fixture status for ${id}`);
|
||||
else if(a==='launches list --json'){
|
||||
if(fault(`launches-${id}`))process.exit(1);
|
||||
let l=[];try{l=JSON.parse(fs.readFileSync(path.join(sd,'launches.json'),'utf8'))}catch{}
|
||||
console.log(JSON.stringify(l));
|
||||
}else{console.error(`fixture mosaic: ${a}`);process.exit(4)}
|
||||
}
|
||||
|
||||
// A copy's migrate-schema.mjs: records the backups it saw, then sets 3h, or 3h-partial and exit 1.
|
||||
function migrateStub(){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite');
|
||||
const data=process.argv[2],sd=process.env.SD_STATE,dir=path.join(data,'backups','stack');
|
||||
const backups=fs.existsSync(dir)?fs.readdirSync(dir).sort().map((name)=>({name,sums:fs.existsSync(path.join(dir,name,'SHA256SUMS'))})):[];
|
||||
fs.writeFileSync(path.join(sd,'migrate.json'),JSON.stringify({backups}));
|
||||
const fail=fs.existsSync(path.join(sd,'faults','migrate-fails'));
|
||||
const db=new DatabaseSync(path.join(data,'bus','bus.sqlite'));
|
||||
db.prepare("UPDATE meta SET value=? WHERE key='schema_version'").run(fail?'3h-partial':'3h');db.close();
|
||||
if(fail)process.exit(1);
|
||||
console.log('fixture migrate: 3h');
|
||||
}
|
||||
|
||||
// --- The module fixture: one repo, its copies and the stand-ins, shared by every test. ---
|
||||
|
||||
const base=realpathSync(mkdtempSync(join(tmpdir(),'stack-release-')));
|
||||
after(()=>{execFileSync('chmod',['-R','u+w',base]);rmSync(base,{recursive:true,force:true})});
|
||||
const repo=join(base,'repo'),COPIES=join(base,'copies'),SBIN=join(base,'sbin');
|
||||
mkdirSync(repo);mkdirSync(COPIES);mkdirSync(SBIN);
|
||||
writeFileSync(join(base,'package.json'),'{"type":"commonjs"}\n');
|
||||
const tracked=execFileSync('git',['-C',ROOT,'ls-files','-co','--exclude-standard','--','scripts','packages','adapters','roles','RELEASE','package.json','package-lock.json',':!packages/*/tests',':!scripts/foundation'],{encoding:'utf8'}).trim().split('\n');
|
||||
for(const f of tracked)if(existsSync(join(ROOT,f)))cpSync(join(ROOT,f),join(repo,f));
|
||||
const pkg={name:'copy-fixture',version:'1.0.0'};
|
||||
writeFileSync(join(repo,'package.json'),JSON.stringify(pkg));
|
||||
writeFileSync(join(repo,'package-lock.json'),JSON.stringify({name:pkg.name,version:pkg.version,lockfileVersion:3,packages:{'':pkg}}));
|
||||
git(repo,'init','-q','-b','refactor');git(repo,'config','gc.auto','0');git(repo,'add','.');git(repo,'commit','-qm','fixture');
|
||||
const commit=(msg)=>{git(repo,'commit','-q','--allow-empty','-m',msg);return git(repo,'rev-parse','HEAD')};
|
||||
// refactor: F A B C1 C, then M merging side branch S (from A). U sits on a branch nothing reaches.
|
||||
const F=git(repo,'rev-parse','HEAD'),A=commit('row 901: A'),B=commit('row 902: B'),C1=commit('row 903: C1'),C=commit('row 904: C');
|
||||
git(repo,'checkout','-q','-b','side',A);const S=commit('row 905: S');
|
||||
git(repo,'checkout','-q','-b','unreached',A);const U=commit('row 906: U');
|
||||
git(repo,'checkout','-q','refactor');git(repo,'merge','-q','--no-ff','-m','merge side','side');
|
||||
const M=git(repo,'rev-parse','HEAD');git(repo,'update-ref','refs/remotes/origin/refactor',M);
|
||||
// `to` false leaves out the `to:` lines (C1); a string puts that value in each one (A). The --plan
|
||||
// steps line can't be read from either (T27).
|
||||
const storeText=(v,froms,to=true)=>`export const SCHEMA_VERSION = '${v}';\nexport const MIGRATIONS = [\n${froms.map((f,i)=>` {\n from: '${f}',\n${to===false?'':` to: '${to===true?froms[i+1]??v:to}',\n`} run() {},\n },\n`).join('')}];\n`;
|
||||
for(const [id,store] of [[F,'// a copy from before SCHEMA_VERSION\n'],[A,storeText('3g',['3f'],'3x')],[B,storeText('3g',['3f'])],[C1,storeText('3g',['3f'],false)],[C,storeText('3h',['3f','3g'])],[S,storeText('3g',['3f'])],[U,storeText('3g',['3f'])]]){
|
||||
const d=join(COPIES,id);
|
||||
for(const [f,text,mode] of [['scripts/mosaic',script(mosaicStub),0o755],['packages/webui/src/cli.mjs','// fixture Console, never run\n',0o644],['packages/bus/src/store.mjs',store,0o644],['packages/bus/src/migrate-schema.mjs',`(${migrateStub})();\n`,0o644]]){
|
||||
mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text,{mode});
|
||||
}
|
||||
execFileSync('chmod',['-R','a-w',d]);
|
||||
}
|
||||
for(const [name,fn] of [['systemctl',systemctl],['journalctl',journalctl],['systemd-analyze',systemdAnalyze]])writeFileSync(join(SBIN,name),script(fn),{mode:0o755});
|
||||
writeFileSync(join(base,'holder.cjs'),`(${holder})();\n`);
|
||||
const SR=join(repo,'scripts','stack-release.sh');
|
||||
|
||||
// --- Per-test roots and helpers ---
|
||||
|
||||
const sha=(f)=>createHash('sha256').update(readFileSync(f)).digest('hex');
|
||||
const lines=(f)=>existsSync(f)?readFileSync(f,'utf8').split('\n').filter(Boolean):[];
|
||||
const calls=(e)=>lines(join(e.sd,'calls.log'));
|
||||
const startsStops=(e,from)=>calls(e).slice(from).filter((c)=>/^(start|stop) /.test(c));
|
||||
const logLines=(e)=>lines(e.log).map((l)=>JSON.parse(l));
|
||||
const pointer=(e)=>existsSync(e.pointer)?JSON.parse(readFileSync(e.pointer,'utf8')):null;
|
||||
const units=(e)=>JSON.parse(readFileSync(join(e.sd,'units.json'),'utf8'));
|
||||
const snaps=(e)=>lines(join(e.sd,'snap.jsonl')).map((l)=>JSON.parse(l));
|
||||
const fault=(e,name,text='')=>writeFileSync(join(e.sd,'faults',name),text);
|
||||
const sysctl=(e,...a)=>execFileSync(join(SBIN,'systemctl'),['--user',...a],{env:e.env,encoding:'utf8'});
|
||||
const edit=(f,from,to)=>{const s=readFileSync(f,'utf8');assert.ok(s.includes(from),`${f} has no ${from}`);writeFileSync(f,s.replace(from,to))};
|
||||
const posts=(e)=>readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-'));
|
||||
const backupSum=(e,act)=>readFileSync(join(e.data,'backups','stack',act,'SHA256SUMS'),'utf8').split(' ')[0];
|
||||
const commitLines=(range)=>git(repo,'log','--oneline','--no-merges','--max-count=200',range).split('\n').map((l)=>` ${l}`);
|
||||
// The units, the wrapper, the pointer and the store, by sha256.
|
||||
const fp=(e)=>({bus:sha(join(e.units,'[email protected]')),console:sha(join(e.units,CON)),wrapper:sha(join(e.bin,'mosaic')),pointer:existsSync(e.pointer)?sha(e.pointer):null,store:sha(e.store)});
|
||||
// The schema version: readOnly while the holder has the store open, immutable when it is stopped.
|
||||
function storeVersion(file,held){
|
||||
const db=held?new DatabaseSync(file,{readOnly:true}):new DatabaseSync(pathToFileURL(file).href+'?immutable=1',{readOnly:true});
|
||||
try{return db.prepare("SELECT value FROM meta WHERE key='schema_version'").get().value}finally{db.close()}
|
||||
}
|
||||
// Every file and directory under p with size and mtime: T23 and T16 compare it before and after.
|
||||
function tree(p){
|
||||
const out=[];
|
||||
const walk=(q)=>{let st;try{st=statSync(q)}catch{return}out.push(st.isDirectory()?`${q}/`:`${q} ${st.size} ${st.mtimeMs}`);if(st.isDirectory())for(const n of readdirSync(q).sort())walk(join(q,n))};
|
||||
walk(p);return out;
|
||||
}
|
||||
// T17: what a run may never remove.
|
||||
const kept=(e)=>[...tree(join(e.data,'backups','stack')),...tree(join(e.state,'stack-rollback')),...posts(e).flatMap((n)=>tree(join(e.data,n))),...tree(COPIES)].map((l)=>l.replace(/ .*/,''));
|
||||
|
||||
async function mk(t){
|
||||
const root=realpathSync(mkdtempSync(join(tmpdir(),'sr-')));
|
||||
const e={root,data:join(root,'data'),bin:join(root,'bin'),sd:join(root,'sd'),units:join(root,'xcfg','systemd','user'),cfg:join(root,'cfg','config.json'),claude:join(root,'claude-bin','claude')};
|
||||
Object.assign(e,{state:join(e.data,'state'),pointer:join(e.data,'state','stack-active.json'),log:join(e.data,'state','stack-activation-log.jsonl'),store:join(e.data,'bus','bus.sqlite'),lock:join(e.data,'state','stack-release.lock')});
|
||||
for(const d of [join(root,'home'),dirname(e.cfg),join(e.data,'bus'),e.bin,dirname(e.claude),join(e.sd,'faults'),e.units,join(root,'xdata')])mkdirSync(d,{recursive:true});
|
||||
chmodSync(join(e.data,'bus'),0o700);
|
||||
writeFileSync(e.cfg,JSON.stringify({configVersion:1,environment:'development',dataRoot:e.data,execution:{backend:'docker',provider:'zai',model:'glm-5.3-flash',adapter:'mock'}}));
|
||||
writeFileSync(join(e.bin,'mosaic'),`#!/usr/bin/env bash\n${MARKER}\n[ "\${1:-}" = __lease-capability ] && exec "${e.bin}/mosaic-old" "$@"\nexec "/old/scripts/mosaic" "$@"\n`,{mode:0o755});
|
||||
writeFileSync(join(e.bin,'mosaic-old'),'#!/bin/sh\nexit 0\n',{mode:0o755});
|
||||
writeFileSync(e.claude,'#!/bin/sh\necho "9.9.9 (Claude Code fixture)"\n',{mode:0o755});
|
||||
e.env={PATH:`${SBIN}:${dirname(NODE)}:/usr/bin:/bin`,HOME:join(root,'home'),XDG_CONFIG_HOME:join(root,'xcfg'),XDG_DATA_HOME:join(root,'xdata'),MOSAIC_CONFIG:e.cfg,SD_STATE:e.sd,SD_DATA:e.data,TMPDIR:tmpdir()};
|
||||
// The Console's /healthz: 200 while the stand-in has it active and no healthz-<copy> fault names its copy.
|
||||
const server=createServer((q,r)=>{
|
||||
let st={};try{st=units(e)[CON]||{}}catch{}
|
||||
const ok=q.url==='/healthz'&&st.state==='active'&&!existsSync(join(e.sd,'faults',`healthz-${basename(st.wd||'')}`));
|
||||
r.writeHead(ok?200:500);r.end();
|
||||
});
|
||||
server.listen(0,'127.0.0.1');await once(server,'listening');e.port=String(server.address().port);
|
||||
t.after(()=>{
|
||||
for(const pid of lines(join(e.sd,'pids.log')).map(Number)){
|
||||
try{if(readFileSync(`/proc/${pid}/cmdline`,'utf8').includes(join(base,'holder.cjs')))process.kill(pid,'SIGKILL')}catch{}
|
||||
}
|
||||
server.close();execFileSync('chmod',['-R','u+w',root]);rmSync(root,{recursive:true,force:true});
|
||||
});
|
||||
return e;
|
||||
}
|
||||
|
||||
async function run(e,...args){
|
||||
const before=existsSync(e.data)?kept(e):[];
|
||||
const ch=spawn('bash',[SR,...args,'--copies',COPIES,'--bin',e.bin],{env:e.env,stdio:['ignore','pipe','pipe']});
|
||||
let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d});
|
||||
const timer=setTimeout(()=>ch.kill('SIGKILL'),120000);
|
||||
const [code]=await once(ch,'close');clearTimeout(timer);
|
||||
const now=new Set(kept(e));
|
||||
for(const p of before)assert.ok(now.has(p),`T17: ${args.join(' ')} removed ${p}`);
|
||||
return {code,out,err};
|
||||
}
|
||||
|
||||
// The stack on copy `copy` for business acme, both units active. pointer: adopt (a real adopt and
|
||||
// its log line), hand (written here, no log) or none. Snapshots start after the seed.
|
||||
async function seed(e,{copy=B,pm=false,store='3g',pointer:how='adopt'}={}){
|
||||
const db=new DatabaseSync(e.store);
|
||||
db.exec('PRAGMA journal_mode=WAL');
|
||||
db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)');
|
||||
db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)');
|
||||
db.prepare("INSERT INTO meta VALUES ('schema_version', ?)").run(store);
|
||||
for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`);
|
||||
db.close();
|
||||
const sh=(name,...a)=>execFileSync('bash',[join(repo,'scripts',name),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']});
|
||||
sh('bus-service.sh','install','--copy',copy,'--copies',COPIES,...(pm?['--pm']:[]),'--claude',e.claude);
|
||||
sh('webui-service.sh','install','--copy',copy,'--copies',COPIES,'--port',e.port,'--business','acme');
|
||||
sh('stack-copy.sh','wrapper',copy,'--copies',COPIES,'--bin',e.bin);
|
||||
// The seeded host started two minutes ago, so its boot status and PM run predate any run's start.
|
||||
execFileSync(join(SBIN,'systemctl'),['--user','start',BUS],{env:{...e.env,SD_CLOCK_OFFSET:'120'}});
|
||||
sysctl(e,'start',CON);
|
||||
if(how==='adopt'){const r=await run(e,'adopt','acme');assert.equal(r.code,0,r.err)}
|
||||
else if(how==='hand'){
|
||||
mkdirSync(e.state,{recursive:true});
|
||||
writeFileSync(e.pointer,JSON.stringify({copy,business:'acme',pm,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g',activation:'manual',at:'2026-10-18T00:00:00Z'})+'\n');
|
||||
}
|
||||
writeFileSync(join(e.sd,'snap.jsonl'),'');
|
||||
}
|
||||
|
||||
const rolledBack=(code,to=B)=>`stack-release: ${code}; rolled back to ${to}, which passed the gate\n`;
|
||||
const planTail=(e,to,pm=false)=>[`bus: scripts/bus-service.sh install --copy ${to} --copies ${COPIES}${pm?' --pm':''} --claude ${e.claude}`,`Console: scripts/webui-service.sh install --copy ${to} --copies ${COPIES} --port ${e.port} --business acme`,`CLI: scripts/stack-copy.sh wrapper ${to} --copies ${COPIES} --bin ${e.bin}`];
|
||||
|
||||
// --- The T table ---
|
||||
|
||||
test('T1 adopt writes the pointer from the units once; a failing check refuses check-failed',async(t)=>{
|
||||
const e=await mk(t);await seed(e,{pointer:'none'});
|
||||
const r=await run(e,'adopt','acme');assert.equal(r.code,0,r.err);
|
||||
assert.equal(r.out,`adopted ${B} for acme (pm 0, schema 3g)\n`);
|
||||
const {at,activation,...p}=pointer(e);
|
||||
assert.deepEqual(p,{copy:B,business:'acme',pm:false,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g'});
|
||||
assert.match(activation,new RegExp(`^\\d{8}T\\d{6}Z-${B.slice(0,12)}$`));assert.match(at,/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$/);
|
||||
assert.deepEqual(logLines(e).map((l)=>[l.op,l.from,l.to,l.result]),[['adopt',null,B,'ok']]);
|
||||
const again=await run(e,'adopt','acme');assert.equal(again.code,3);
|
||||
assert.equal(again.err,`stack-release: refusing already-adopted: ${e.pointer} exists (copy ${B})\n`);
|
||||
assert.equal(logLines(e).at(-1).code,'already-adopted');
|
||||
const f=await mk(t);await seed(f,{pointer:'none'});fault(f,`check-${B}`);
|
||||
const bad=await run(f,'adopt','acme');assert.equal(bad.code,3);
|
||||
assert.match(bad.err,/^stack-release: refusing check-failed: stack-copy.sh check did not pass:\nbus: /);
|
||||
assert.equal(pointer(f),null);
|
||||
});
|
||||
|
||||
test('T2 activate installs exactly the pointer\'s pm, claude and Console flags',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
const r=await run(e,'activate',C1,'acme');assert.equal(r.code,0,r.err+r.out);
|
||||
assert.ok(r.out.endsWith(`stack-release: activate ok: ${C1} is live for acme\n`),r.out);
|
||||
const bus=readFileSync(join(e.units,'[email protected]'),'utf8').split('\n'),con=readFileSync(join(e.units,CON),'utf8').split('\n');
|
||||
assert.ok(bus.includes(`ExecStart=${COPIES}/${C1}/scripts/mosaic bus start %i`),bus.join('\n'));
|
||||
assert.ok(bus.includes(`Environment=MOSAIC_CLAUDE_BIN=${e.claude}`));
|
||||
const exec=con.find((l)=>l.startsWith('ExecStart='));
|
||||
assert.ok(exec.endsWith(` ${COPIES}/${C1}/packages/webui/src/cli.mjs serve --checkout ${repo} --port ${e.port} --business acme`),exec);
|
||||
assert.equal(readFileSync(join(e.bin,'mosaic'),'utf8').trimEnd().split('\n').at(-1),`exec "${COPIES}/${C1}/scripts/mosaic" "$@"`);
|
||||
const {at,activation,...p}=pointer(e);
|
||||
assert.deepEqual(p,{copy:C1,business:'acme',pm:false,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g'});
|
||||
});
|
||||
|
||||
test('T3 a hand-edited unit refuses unit-flags-drift, names the difference and changes nothing',async(t)=>{
|
||||
const cases=[
|
||||
['pm',(e)=>edit(join(e.units,'[email protected]'),' bus start %i\n',' bus start %i --pm\n'),()=>' pm: pointer 0, unit 1'],
|
||||
['claude',(e)=>{const d=join(e.units,`${BUS}.d`);mkdirSync(d);writeFileSync(join(d,'claude.conf'),'[Service]\nEnvironment=MOSAIC_CLAUDE_BIN=/opt/other/claude\n')},(e)=>` claude: pointer ${e.claude}, unit /opt/other/claude`],
|
||||
['console',(e)=>edit(join(e.units,CON),'--business acme\n','--business acme --board http://127.0.0.1:9\n'),(e)=>` console: pointer --port ${e.port} --business acme, unit --port ${e.port} --business acme --board http://127.0.0.1:9`],
|
||||
];
|
||||
for(const [name,change,line] of cases)await t.test(name,async(t)=>{
|
||||
const e=await mk(t);await seed(e);change(e);
|
||||
const before=fp(e),n=calls(e).length,logs=logLines(e).length;
|
||||
const r=await run(e,'activate',C1,'acme');assert.equal(r.code,3);
|
||||
assert.equal(r.err,`stack-release: refusing unit-flags-drift: the units differ from the pointer; settle it first:\n${line(e)}\n`);
|
||||
assert.deepEqual(fp(e),before);assert.deepEqual(startsStops(e,n),[]);
|
||||
assert.equal(logLines(e).length,logs+1);assert.equal(logLines(e).at(-1).code,'unit-flags-drift');
|
||||
});
|
||||
});
|
||||
|
||||
test('T4 each item 3 refusal changes nothing and logs one line with its code',async(t)=>{
|
||||
const setUnit=(e,u,state)=>{const s=units(e);s[u]={...(s[u]||{pid:null,wd:''}),state};writeFileSync(join(e.sd,'units.json'),JSON.stringify(s))};
|
||||
const cases=[
|
||||
['copy-missing',()=>M,()=>`no copy ${M} in ${COPIES}`],
|
||||
['copy-unreachable',()=>U,()=>`${U} is not reachable from refactor and origin/refactor`],
|
||||
['copy-schema-unknown',()=>F,()=>`cannot read SCHEMA_VERSION and its chain from ${COPIES}/${F}/packages/bus/src/store.mjs`],
|
||||
['already-active',()=>B,()=>`${B} is the pointer's copy`],
|
||||
['instance-busy',(e)=>{setUnit(e,'[email protected]','active');return C1},()=>'[email protected] is active; the copy and wrapper switch reach it too','another business active'],
|
||||
['instance-busy',(e)=>{setUnit(e,BUS,'activating');return C1},()=>`${BUS} is activating`,'this business activating'],
|
||||
['drop-in-override',(e)=>{const d=join(e.units,`${BUS}.d`);mkdirSync(d);writeFileSync(join(d,'override.conf'),'[Service]\nWorkingDirectory=/elsewhere\n');return C1},(e)=>`${join(e.units,`${BUS}.d`,'override.conf')} overrides ExecStart or WorkingDirectory; the operator removes it as in TOOLS.md step 4`],
|
||||
['no-pointer',()=>C1,(e)=>`no ${e.pointer}; run adopt first`,'',{pointer:'none'}],
|
||||
];
|
||||
for(const [code,setup,msg,label,opts] of cases)await t.test(label?`${code} (${label})`:code,async(t)=>{
|
||||
const e=await mk(t);await seed(e,opts);const id=setup(e);
|
||||
const before=fp(e),n=calls(e).length,logs=logLines(e).length;
|
||||
const r=await run(e,'activate',id,'acme');assert.equal(r.code,3,r.err);
|
||||
assert.equal(r.err,`stack-release: refusing ${code}: ${msg(e)}\n`);
|
||||
assert.deepEqual(fp(e),before);assert.deepEqual(startsStops(e,n),[]);
|
||||
assert.equal(logLines(e).length,logs+1);assert.deepEqual([logLines(e).at(-1).result,logLines(e).at(-1).code],['refused',code]);
|
||||
});
|
||||
});
|
||||
|
||||
test('T5 an activate with no migration stops the Console then the bus, installs, starts the bus then the Console',async(t)=>{
|
||||
const e=await mk(t);await seed(e);const n=calls(e).length;
|
||||
const r=await run(e,'activate',C1,'acme');assert.equal(r.code,0,r.err);
|
||||
assert.deepEqual(startsStops(e,n),[`stop ${CON}`,`stop ${BUS}`,`start ${BUS}`,`start ${CON}`]);
|
||||
const c=calls(e).slice(n),reload=c.indexOf('daemon-reload');
|
||||
assert.ok(reload>c.indexOf(`stop ${BUS}`)&&reload<c.indexOf(`start ${BUS}`),'daemon-reload between the stops and the starts');
|
||||
assert.equal(pointer(e).copy,C1);assert.equal(units(e)[BUS].wd,join(COPIES,C1));
|
||||
assert.ok(!existsSync(join(e.data,'backups','stack')));
|
||||
const l=logLines(e).slice(-2);assert.deepEqual(l.map((x)=>[x.op,x.from,x.to,x.result]),[['activate',B,C1,'started'],['activate',B,C1,'ok']]);
|
||||
});
|
||||
|
||||
test('T6 a migration without --migrate rolls back with migration-needs-flag',async(t)=>{
|
||||
const e=await mk(t);await seed(e);const before=fp(e);
|
||||
const r=await run(e,'activate',C,'acme','--timeout','3');assert.equal(r.code,1,r.out);
|
||||
assert.equal(r.err,rolledBack('migration-needs-flag'));
|
||||
assert.deepEqual(fp(e),before);
|
||||
assert.deepEqual([units(e)[BUS].state,units(e)[BUS].wd,units(e)[CON].state],['active',join(COPIES,B),'active']);
|
||||
const l=logLines(e).at(-1);assert.deepEqual([l.result,l.code,l.rollback],['gate-failed','migration-needs-flag','ok']);
|
||||
assert.equal(storeVersion(e.store,true),'3g');assert.ok(!existsSync(join(e.data,'backups','stack')));
|
||||
});
|
||||
|
||||
test('T7 --migrate takes the backup before the migration runs',async(t)=>{
|
||||
const e=await mk(t);await seed(e);const before=fp(e);
|
||||
const r=await run(e,'activate',C,'acme','--migrate');assert.equal(r.code,0,r.err+r.out);
|
||||
const act=pointer(e).activation;
|
||||
assert.deepEqual(JSON.parse(readFileSync(join(e.sd,'migrate.json'),'utf8')),{backups:[{name:act,sums:true}]});
|
||||
assert.equal(backupSum(e,act),before.store);
|
||||
assert.deepEqual([pointer(e).copy,pointer(e).schema],[C,'3h']);assert.equal(storeVersion(e.store,true),'3h');
|
||||
assert.equal(logLines(e).at(-1).backup,act);
|
||||
});
|
||||
|
||||
test('T8 a failed migration restores the backup and keeps the store it replaced',async(t)=>{
|
||||
const e=await mk(t);await seed(e);const before=fp(e);fault(e,'migrate-fails');
|
||||
const r=await run(e,'activate',C,'acme','--migrate','--timeout','3');assert.equal(r.code,1,r.out);
|
||||
assert.equal(r.err,rolledBack('migrate-failed'));
|
||||
assert.equal(posts(e).length,1);assert.equal(storeVersion(join(e.data,posts(e)[0],'bus.sqlite'),false),'3h-partial');
|
||||
const act=logLines(e).at(-1).activation;
|
||||
assert.equal(sha(e.store),backupSum(e,act));assert.equal(sha(e.store),before.store);
|
||||
assert.deepEqual(fp(e),before);assert.equal(units(e)[BUS].wd,join(COPIES,B));
|
||||
});
|
||||
|
||||
test('T9 a store outside the target\'s chain rolls back with store-newer',async(t)=>{
|
||||
const e=await mk(t);await seed(e,{store:'3z'});const before=fp(e);
|
||||
const r=await run(e,'activate',C,'acme','--migrate','--timeout','3');assert.equal(r.code,1,r.out);
|
||||
assert.equal(r.err,rolledBack('store-newer'));
|
||||
assert.deepEqual(fp(e),before);assert.equal(storeVersion(e.store,true),'3z');assert.ok(!existsSync(join(e.data,'backups','stack')));
|
||||
});
|
||||
|
||||
test('T10 each gate check failing alone rolls back and puts every file back',async(t)=>{
|
||||
const cases=[['units-inactive',`start-dies-${C1}`],['bus-status',`status-${C1}`],['healthz',`healthz-${C1}`],['copy-check',`check-${C1}`],['pm-launch-unconfirmed',`pm-${C1}`,'none']];
|
||||
for(const [code,name,text] of cases)await t.test(code,async(t)=>{
|
||||
const pm=code==='pm-launch-unconfirmed',e=await mk(t);await seed(e,{pm});fault(e,name,text);
|
||||
const before=fp(e);
|
||||
const r=await run(e,'activate',C1,'acme',...(pm?['--pm']:[]),'--timeout','3');assert.equal(r.code,1,r.out+r.err);
|
||||
assert.equal(r.err,rolledBack(code));assert.ok(r.out.includes(`stack-release: gate failed on ${C1}: ${code}`),r.out);
|
||||
assert.deepEqual(fp(e),before);
|
||||
assert.deepEqual([units(e)[BUS].state,units(e)[BUS].wd],['active',join(COPIES,B)]);
|
||||
const l=logLines(e).at(-1);assert.deepEqual([l.result,l.code,l.rollback],['gate-failed',code,'ok']);
|
||||
});
|
||||
});
|
||||
|
||||
test('T10b the PM check needs exactly one launched line for a live run of this start',async(t)=>{
|
||||
const cases=[
|
||||
['no launched line','none',/\(no launched line since the start\)/],
|
||||
['two lines','two',/\(2 launched lines since the start\)/],
|
||||
['a changed word','badword',/\(the launched line does not parse: launched pm-acme: job run-\d+, claude opus\)/],
|
||||
['run not in launches list','notlisted',/\(run run-\d+ is not in launches list\)/],
|
||||
['run started before the start','early',/\(run run-\d+ started before this start\)/],
|
||||
['a human launch with no journal line','human',/\(no launched line since the start\)/],
|
||||
['run for another business','other',/\(run run-\d+ is for other\)/],
|
||||
['run not live','dead',/\(run run-\d+ is not live\)/],
|
||||
];
|
||||
for(const [name,mode,why] of cases)await t.test(name,async(t)=>{
|
||||
const e=await mk(t);await seed(e,{pm:true});fault(e,`pm-${C1}`,mode);
|
||||
const r=await run(e,'activate',C1,'acme','--pm','--timeout','3');assert.equal(r.code,1,r.out+r.err);
|
||||
assert.equal(r.err,rolledBack('pm-launch-unconfirmed'));
|
||||
const gate=r.out.split('\n').find((l)=>l.startsWith(`stack-release: gate failed on ${C1}: pm-launch-unconfirmed`));
|
||||
assert.ok(gate,r.out);assert.match(gate,why);
|
||||
});
|
||||
await t.test('one well-formed line and a live run pass',async(t)=>{
|
||||
const e=await mk(t);await seed(e,{pm:true});
|
||||
const r=await run(e,'activate',C1,'acme','--pm');assert.equal(r.code,0,r.out+r.err);
|
||||
assert.ok(readFileSync(join(e.units,'[email protected]'),'utf8').includes(`ExecStart=${COPIES}/${C1}/scripts/mosaic bus start %i --pm\n`));
|
||||
assert.match(lines(join(e.sd,'journal-since.log')).at(-1),/^@\d+$/);
|
||||
assert.equal(pointer(e).pm,true);
|
||||
});
|
||||
});
|
||||
|
||||
test('T11 a rollback whose gate fails too: rollback-failed, exit 2, both units stopped, no retry',async(t)=>{
|
||||
const e=await mk(t);await seed(e);fault(e,`healthz-${C1}`);fault(e,`healthz-${B}`);const n=calls(e).length;
|
||||
const r=await run(e,'activate',C1,'acme','--timeout','3');assert.equal(r.code,2,r.out+r.err);
|
||||
const l=logLines(e).at(-1),rs=join(e.state,'stack-rollback',l.activation);
|
||||
assert.equal(r.err,`stack-release: rollback-failed: the run failed with healthz and the rollback with healthz; both units are stopped. Rollback set: ${rs}\n`);
|
||||
assert.deepEqual([l.result,l.code,l.rollback,l.rollbackCode],['rollback-failed','healthz','failed','healthz']);
|
||||
assert.deepEqual([units(e)[BUS].state,units(e)[CON].state],['inactive','inactive']);
|
||||
const ss=startsStops(e,n);
|
||||
assert.deepEqual(ss.filter((c)=>c.startsWith('start ')),[`start ${BUS}`,`start ${CON}`,`start ${BUS}`,`start ${CON}`]);
|
||||
assert.deepEqual(ss.slice(-2),[`stop ${CON}`,`stop ${BUS}`]);
|
||||
assert.equal(pointer(e).copy,B);
|
||||
});
|
||||
|
||||
test('T12 rollback takes the previous copy from the log',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
assert.equal((await run(e,'activate',C1,'acme')).code,0);
|
||||
const r=await run(e,'rollback','acme');assert.equal(r.code,0,r.out+r.err);
|
||||
assert.ok(r.out.endsWith(`stack-release: rollback ok: ${B} is live for acme\n`),r.out);
|
||||
assert.equal(pointer(e).copy,B);
|
||||
const l=logLines(e).at(-1);assert.deepEqual([l.op,l.from,l.to,l.result],['rollback',C1,B,'ok']);
|
||||
});
|
||||
|
||||
test('T13 a rollback across a migration needs --restore, then restores and goes live',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
assert.equal((await run(e,'activate',C,'acme','--migrate')).code,0);const act=pointer(e).activation,storeC=sha(e.store);
|
||||
const no=await run(e,'rollback','acme');assert.equal(no.code,3);
|
||||
assert.equal(no.err,`stack-release: refusing schema-differs: the store is at 3h and ${B} expects 3g; pass --restore ACTIVATION\n`);
|
||||
const r=await run(e,'rollback','acme','--restore',act);assert.equal(r.code,0,r.out+r.err);
|
||||
const o=join(e.data,'backups','stack',act);
|
||||
assert.match(r.out,new RegExp(`^restore: ${o}; the store now in place moves to ${e.data}/bus\\.post-attempt-\\d{8}T\\d{6}Z and is kept; 0 event\\(s\\) were written after the backup\\n`));
|
||||
assert.deepEqual([pointer(e).copy,pointer(e).schema],[B,'3g']);assert.equal(storeVersion(e.store,true),'3g');
|
||||
assert.equal(sha(e.store),backupSum(e,act));
|
||||
assert.equal(posts(e).length,1);assert.equal(sha(join(e.data,posts(e)[0],'bus.sqlite')),storeC);
|
||||
assert.equal(logLines(e).at(-1).backup,act);
|
||||
});
|
||||
|
||||
test('T14 events written after the backup need --events N with the right N',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
assert.equal((await run(e,'activate',C,'acme','--migrate')).code,0);const act=pointer(e).activation;
|
||||
const db=new DatabaseSync(e.store);for(let i=0;i<2;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run('2999-01-01T00:00:00.000Z');db.close();
|
||||
const want='stack-release: refusing post-backup-events: 2 event(s) were written after the backup; Jason decides, then --events 2\n';
|
||||
for(const extra of [[],['--events','1']]){
|
||||
const before=fp(e),r=await run(e,'rollback','acme','--restore',act,...extra);
|
||||
assert.equal(r.code,3,extra.join(' '));assert.equal(r.err,want);assert.deepEqual(fp(e),before);
|
||||
}
|
||||
const r=await run(e,'rollback','acme','--restore',act,'--events','2');assert.equal(r.code,0,r.out+r.err);
|
||||
assert.match(r.out,/; 2 event\(s\) were written after the backup\n/);
|
||||
assert.deepEqual([pointer(e).copy,pointer(e).schema],[B,'3g']);
|
||||
});
|
||||
|
||||
test('T14b a rollback --restore whose gate fails stops both units and skips item 6',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
assert.equal((await run(e,'activate',C,'acme','--migrate')).code,0);const act=pointer(e).activation,storeC=sha(e.store),before=fp(e);
|
||||
fault(e,`healthz-${B}`);const n=calls(e).length;
|
||||
const r=await run(e,'rollback','acme','--restore',act,'--timeout','3');assert.equal(r.code,2,r.out+r.err);
|
||||
const l=logLines(e).at(-1),rs=join(e.state,'stack-rollback',l.activation),post=join(e.data,posts(e)[0]);
|
||||
assert.equal(r.err,`stack-release: rollback-failed: the run failed with healthz and the rollback with not-attempted; both units are stopped. Rollback set: ${rs}, backup ${join(e.data,'backups','stack',act)}, post-attempt store ${post}\n`);
|
||||
assert.deepEqual([l.result,l.code,l.rollbackCode],['rollback-failed','healthz','not-attempted']);
|
||||
assert.deepEqual([units(e)[BUS].state,units(e)[CON].state],['inactive','inactive']);
|
||||
assert.equal(posts(e).length,1);assert.equal(sha(join(post,'bus.sqlite')),storeC);assert.equal(sha(e.store),backupSum(e,act));
|
||||
assert.deepEqual(startsStops(e,n).filter((c)=>c.startsWith('start ')),[`start ${BUS}`,`start ${CON}`]);
|
||||
assert.equal(pointer(e).copy,C);assert.equal(sha(e.pointer),before.pointer);
|
||||
});
|
||||
|
||||
test('T14c a rollback without --restore whose gate fails runs item 6',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
assert.equal((await run(e,'activate',C1,'acme')).code,0);const before=fp(e);
|
||||
fault(e,`healthz-${B}`);
|
||||
const r=await run(e,'rollback','acme','--timeout','3');assert.equal(r.code,1,r.out+r.err);
|
||||
assert.equal(r.err,rolledBack('healthz',C1));
|
||||
const l=logLines(e).at(-1);assert.deepEqual([l.op,l.result,l.code,l.rollback],['rollback','gate-failed','healthz','ok']);
|
||||
assert.deepEqual(fp(e),before);assert.deepEqual([units(e)[BUS].state,units(e)[BUS].wd],['active',join(COPIES,C1)]);
|
||||
assert.deepEqual(readdirSync(join(e.data,'bus')).filter((n)=>n.startsWith('bus.sqlite')).sort(),['bus.sqlite','bus.sqlite-shm','bus.sqlite-wal']);
|
||||
});
|
||||
|
||||
test('T15 --plan names the migration and the restore and changes nothing',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
const quiet=async(...args)=>{
|
||||
const before=fp(e),n=calls(e).length,logs=logLines(e).length,r=await run(e,...args);
|
||||
assert.equal(r.code,0,r.err);assert.equal(r.err,'');
|
||||
assert.deepEqual(fp(e),before);assert.deepEqual(startsStops(e,n),[]);assert.equal(logLines(e).length,logs);
|
||||
return r.out;
|
||||
};
|
||||
assert.ok((await quiet('activate',C,'acme','--plan')).includes('needs: --migrate (not given; the run would roll back with migration-needs-flag)\n'));
|
||||
assert.equal((await run(e,'activate',C,'acme','--migrate')).code,0);const act=pointer(e).activation;
|
||||
assert.equal(await quiet('rollback','acme','--plan'),[
|
||||
'plan: rollback for acme',`from: ${C}`,`to: ${B}`,
|
||||
'store schema: 3h (read from the store the running bus holds)','target schema: 3g',
|
||||
`restore: the store must go back to 3g; needs --restore ${act}`,
|
||||
'commits: the run takes away 2 (newest first, at most 200):',...commitLines(`${B}..${C}`),
|
||||
...planTail(e,B),''].join('\n'));
|
||||
});
|
||||
|
||||
test('T26 --plan lists the commits an activate adds, after the migration lines',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
const head=(mig)=>['plan: activate for acme',`from: ${B}`,`to: ${C}`,'store schema: 3g (read from the store the running bus holds)','target schema: 3h',
|
||||
'migration: 3g to 3h, after a backup of the stopped store','migration steps: 3g to 3h',`needs: --migrate (${mig})`,'needs: Jason present for this run (decision 96 condition 2)'];
|
||||
const adds=commitLines(`${B}..${C}`);
|
||||
assert.equal(adds.length,2);assert.match(adds[0],/^ {2}[0-9a-f]{7,} row 904: C$/);assert.match(adds[1],/^ {2}[0-9a-f]{7,} row 903: C1$/);
|
||||
for(const [flags,mig] of [[[],'not given; the run would roll back with migration-needs-flag'],[['--migrate'],'given']]){
|
||||
const r=await run(e,'activate',C,'acme','--plan',...flags);assert.equal(r.code,0,r.err);
|
||||
assert.equal(r.out,[...head(mig),'commits: the run adds 2 (newest first, at most 200):',...adds,...planTail(e,C),''].join('\n'));
|
||||
}
|
||||
const same=await run(e,'activate',C1,'acme','--plan');assert.equal(same.code,0,same.err);
|
||||
assert.ok(same.out.includes('target schema: 3g\nmigration: none; the store is at the target\'s version\ncommits: the run adds 1 (newest first, at most 200):\n'),same.out);
|
||||
});
|
||||
|
||||
test('T27 --plan names each migration step, or says the steps cannot be read',async(t)=>{
|
||||
const e=await mk(t);await seed(e,{store:'3f',pointer:'hand'});
|
||||
const two=await run(e,'activate',C,'acme','--plan');assert.equal(two.code,0,two.err);
|
||||
assert.ok(two.out.includes('\nmigration: 3f to 3h, after a backup of the stopped store\nmigration steps: 3f to 3g, 3g to 3h\nneeds: --migrate'),two.out);
|
||||
const none=await run(e,'activate',C1,'acme','--plan');assert.equal(none.code,0,none.err);
|
||||
assert.ok(none.out.includes(`\nmigration: 3f to 3g, after a backup of the stopped store\nmigration steps: cannot be read from ${COPIES}/${C1}/packages/bus/src/store.mjs\nneeds: --migrate`),none.out);
|
||||
const short=await run(e,'activate',A,'acme','--plan');assert.equal(short.code,0,short.err);
|
||||
assert.ok(short.out.includes(`\nmigration: 3f to 3g, after a backup of the stopped store\nmigration steps: cannot be read from ${COPIES}/${A}/packages/bus/src/store.mjs\nneeds: --migrate`),short.out);
|
||||
});
|
||||
|
||||
test('T25 --plan refuses copies-unrelated when neither copy is an ancestor of the other',async(t)=>{
|
||||
const e=await mk(t);await seed(e,{pointer:'hand'});
|
||||
const r=await run(e,'activate',S,'acme','--plan');assert.equal(r.code,3);
|
||||
assert.equal(r.err,`stack-release: refusing copies-unrelated: neither ${B} nor ${S} is an ancestor of the other, so the plan cannot list the commits\n`);
|
||||
assert.equal(r.out,'');assert.ok(!existsSync(e.log));assert.ok(!existsSync(e.lock));
|
||||
});
|
||||
|
||||
test('T23 --plan reads the store without changing a file, stopped or held',async(t)=>{
|
||||
const want=['target schema: 3h','migration: 3g to 3h, after a backup of the stopped store','migration steps: 3g to 3h','needs: --migrate (not given; the run would roll back with migration-needs-flag)','needs: Jason present for this run (decision 96 condition 2)'];
|
||||
for(const [name,stop,src] of [['stopped store',true,'read from the stopped store'],['held store',false,'read from the store the running bus holds']])await t.test(name,async(t)=>{
|
||||
const e=await mk(t);await seed(e,{pointer:'hand'});if(stop)sysctl(e,'stop',BUS);
|
||||
const before=tree(e.data),units0=[sha(join(e.units,'[email protected]')),sha(join(e.units,CON)),sha(join(e.bin,'mosaic'))],store0=sha(e.store);
|
||||
const r=await run(e,'activate',C,'acme','--plan');assert.equal(r.code,0,r.err);
|
||||
const out=r.out.split('\n');
|
||||
assert.ok(out.includes(`store schema: 3g (${src})`),r.out);
|
||||
assert.deepEqual(out.slice(out.indexOf('target schema: 3h'),out.indexOf('target schema: 3h')+5),want);
|
||||
assert.deepEqual(tree(e.data),before);assert.equal(sha(e.store),store0);
|
||||
assert.deepEqual([sha(join(e.units,'[email protected]')),sha(join(e.units,CON)),sha(join(e.bin,'mosaic'))],units0);
|
||||
assert.ok(!existsSync(e.log));assert.ok(!existsSync(e.lock));
|
||||
if(stop)assert.deepEqual(readdirSync(join(e.data,'bus')),['bus.sqlite']);
|
||||
});
|
||||
});
|
||||
|
||||
test('T24 business-mismatch and capture-failed refuse with exit 3; a short id is a usage error',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
let n=calls(e).length,logs=logLines(e).length;
|
||||
const r=await run(e,'activate',C1,'other');assert.equal(r.code,3);
|
||||
assert.equal(r.err,'stack-release: refusing business-mismatch: the pointer is for acme\n');
|
||||
assert.equal(logLines(e).length,logs+1);assert.equal(logLines(e).at(-1).code,'business-mismatch');assert.deepEqual(startsStops(e,n),[]);
|
||||
const sets=join(e.state,'stack-rollback');mkdirSync(sets);chmodSync(sets,0o555);
|
||||
n=calls(e).length;logs=logLines(e).length;const before=fp(e);
|
||||
const c=await run(e,'activate',C1,'acme');assert.equal(c.code,3);
|
||||
assert.match(c.err,new RegExp(`^stack-release: refusing capture-failed: could not write the rollback set ${sets}/\\d{8}T\\d{6}Z-${C1.slice(0,12)}; nothing was stopped\\n$`));
|
||||
const l=logLines(e).slice(logs);
|
||||
assert.deepEqual(l.map((x)=>[x.result,x.code]),[['started',undefined],['refused','capture-failed']]);assert.equal(l[0].activation,l[1].activation);
|
||||
assert.deepEqual(startsStops(e,n),[]);assert.deepEqual(fp(e),before);
|
||||
const s=await run(e,'status','acme');assert.equal(s.code,0,s.err);assert.ok(!s.out.includes('interrupted run'),s.out);
|
||||
logs=logLines(e).length;
|
||||
const u=await run(e,'activate','abc123','acme');assert.equal(u.code,4);assert.match(u.err,/^usage: stack-release.sh adopt BUSINESS\n/);
|
||||
assert.equal(logLines(e).length,logs);
|
||||
});
|
||||
|
||||
test('T16 status prints the pointer, the log, the check, bus status and drift, and writes nothing',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
const before=tree(e.data),fp0=fp(e);
|
||||
const r=await run(e,'status','acme');assert.equal(r.code,0,r.err);
|
||||
const copy=join(COPIES,B);
|
||||
assert.equal(r.out,[
|
||||
`pointer: ${e.pointer}`,...readFileSync(e.pointer,'utf8').trimEnd().split('\n').map((l)=>` ${l}`),
|
||||
'log (last five lines):',...lines(e.log).map((l)=>` ${l}`),
|
||||
'check:',` bus: ${copy}`,` Console: ${copy}`,` CLI: ${copy}`,' result: pass',
|
||||
'bus status:',` bus: fixture status for ${B}`,
|
||||
'unit flags: match the pointer',
|
||||
'drift: 4 commit(s) on refactor since the pointer\'s copy',''].join('\n'));
|
||||
assert.deepEqual(tree(e.data),before);assert.deepEqual(fp(e),fp0);
|
||||
});
|
||||
|
||||
test('T19 a started line with no final line: status shows it and every changing verb refuses',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
const act=`20261018T000000Z-${C1.slice(0,12)}`;
|
||||
appendFileSync(e.log,JSON.stringify({at:'2026-10-18T00:00:00Z',op:'activate',from:B,to:C1,backup:null,activation:act,result:'started'})+'\n');
|
||||
const s=await run(e,'status','acme');assert.equal(s.code,0);
|
||||
assert.ok(s.out.includes(`interrupted run: ${act} (started, no final line; Jason or Mos settles it)\n`),s.out);
|
||||
for(const args of [['adopt','acme'],['activate',C1,'acme'],['rollback','acme']]){
|
||||
const before=fp(e),n=calls(e).length,r=await run(e,...args);
|
||||
assert.equal(r.code,3,args.join(' '));
|
||||
assert.equal(r.err,`stack-release: refusing interrupted-run: activation ${act} started and has no final line; Jason or Mos settles it (status shows it)\n`);
|
||||
assert.deepEqual(fp(e),before);assert.deepEqual(startsStops(e,n),[]);
|
||||
}
|
||||
});
|
||||
|
||||
test('T20 a held run lock refuses run-locked without a log line; status and --plan still run',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
const h=spawn('flock',[e.lock,'sleep','30'],{detached:true,stdio:'ignore'});
|
||||
t.after(()=>{try{process.kill(-h.pid,'SIGKILL')}catch{}});
|
||||
const end=Date.now()+10000;
|
||||
while(spawnSync('flock',['-n',e.lock,'true']).status!==1){assert.ok(Date.now()<end,'the lock holder did not take the lock');await new Promise((r)=>setTimeout(r,50))}
|
||||
for(const args of [['adopt','acme'],['activate',C1,'acme'],['rollback','acme']]){
|
||||
const before=fp(e),logs=logLines(e).length,r=await run(e,...args);
|
||||
assert.equal(r.code,3,args.join(' '));
|
||||
assert.equal(r.err,`stack-release: refusing run-locked: another stack-release run holds ${e.lock}\n`);
|
||||
assert.equal(logLines(e).length,logs);assert.deepEqual(fp(e),before);
|
||||
}
|
||||
assert.equal((await run(e,'status','acme')).code,0);
|
||||
const p=await run(e,'activate',C1,'acme','--plan');assert.equal(p.code,0,p.err);
|
||||
});
|
||||
|
||||
test('T21 a bus unit with no MOSAIC_CLAUDE_BIN: adopt and activate refuse no-claude-pin',async(t)=>{
|
||||
const drop=(e)=>edit(join(e.units,'[email protected]'),`Environment=MOSAIC_CLAUDE_BIN=${e.claude}\n`,'');
|
||||
const want=`stack-release: refusing no-claude-pin: ${BUS} has no MOSAIC_CLAUDE_BIN\n`;
|
||||
const a=await mk(t);await seed(a,{pointer:'none'});drop(a);
|
||||
const r=await run(a,'adopt','acme');assert.equal(r.code,3);assert.equal(r.err,want);assert.equal(pointer(a),null);
|
||||
const b=await mk(t);await seed(b);drop(b);const before=fp(b),n=calls(b).length;
|
||||
const s=await run(b,'activate',C1,'acme');assert.equal(s.code,3);assert.equal(s.err,want);
|
||||
assert.deepEqual(fp(b),before);assert.deepEqual(startsStops(b,n),[]);
|
||||
});
|
||||
|
||||
test('T22 the stopped store read: a -wal or -shm refuses store-unclean and is left alone; a clean read adds no file',async(t)=>{
|
||||
for(const ext of ['-wal','-shm'])await t.test(`bus.sqlite${ext}`,async(t)=>{
|
||||
const e=await mk(t);await seed(e);fault(e,`stop-leaves${ext}`);
|
||||
const r=await run(e,'activate',C1,'acme','--timeout','3');assert.equal(r.code,1,r.out+r.err);
|
||||
assert.equal(r.err,rolledBack('store-unclean'));
|
||||
assert.ok(r.out.includes(`stack-release: ${e.store}${ext} is present; the store is left as it is for Jason or Mos\n`),r.out);
|
||||
const s=snaps(e),first=s[0],start=s.find((x)=>x.ev==='start');
|
||||
assert.equal(first.ev,'stop');assert.ok(`bus.sqlite${ext}` in first.files,JSON.stringify(first));
|
||||
assert.deepEqual(start.files,first.files);
|
||||
});
|
||||
await t.test('clean',async(t)=>{
|
||||
const e=await mk(t);await seed(e);
|
||||
const r=await run(e,'activate',C1,'acme');assert.equal(r.code,0,r.out+r.err);
|
||||
const s=snaps(e),stop=s.find((x)=>x.ev==='stop'),start=s.find((x)=>x.ev==='start');
|
||||
assert.deepEqual(Object.keys(stop.files),['bus.sqlite']);assert.deepEqual(start.files,stop.files);
|
||||
});
|
||||
});
|
||||
+12
-5
@@ -31,15 +31,20 @@ COPY_PATHS=(
|
||||
# Whole directory: every in-repo plug-in module ships with the copy (row 131).
|
||||
packages/plugins
|
||||
)
|
||||
# stack-release.sh sources this file and calls reachable before an activate (D1).
|
||||
reachable() {
|
||||
local ref
|
||||
for ref in refs/heads/refactor refs/remotes/origin/refactor; do
|
||||
git -C "$REPO" rev-parse --verify --quiet "$ref^{commit}" >/dev/null || die "no $ref in $REPO"
|
||||
git -C "$REPO" merge-base --is-ancestor "$1" "$ref" || die "refusing: $1 is not reachable from $ref"
|
||||
done
|
||||
}
|
||||
copy_commit() {
|
||||
[ -d "$REPO/.git" ] || [ "$ALLOW_WORKTREE" = 1 ] || die "refusing: $REPO is not the canonical checkout (its .git is not a directory); run copy from the canonical checkout"
|
||||
[[ "$ID" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] || die "commit must be a commit id or branch name: $ID"
|
||||
local full ref target tmp
|
||||
local full target tmp
|
||||
full="$(git -C "$REPO" rev-parse --verify --quiet "$ID^{commit}")" || die "not a commit: $ID"
|
||||
for ref in refs/heads/refactor refs/remotes/origin/refactor; do
|
||||
git -C "$REPO" rev-parse --verify --quiet "$ref^{commit}" >/dev/null || die "no $ref in $REPO"
|
||||
git -C "$REPO" merge-base --is-ancestor "$full" "$ref" || die "refusing: $full is not reachable from $ref"
|
||||
done
|
||||
reachable "$full"
|
||||
target="$COPIES/$full"
|
||||
[ ! -L "$target" ] || die "refusing: copy target is a symlink: $target"
|
||||
if [ -e "$target" ]; then echo "exists: $target"; return; fi
|
||||
@@ -96,6 +101,8 @@ check() {
|
||||
printf 'bus: %s\nConsole: %s\nCLI: %s\n' "$bus" "$console" "$cli"
|
||||
[[ "$bus" == "$COPIES/"* ]] && [[ "${bus#"$COPIES/"}" =~ ^[0-9a-f]{40}$ ]] && [ -f "$bus/scripts/mosaic" ] && [ "$bus" = "$console" ] && [ "$bus" = "$cli" ]
|
||||
}
|
||||
# Sourced (stack-release.sh): define the functions, run nothing.
|
||||
[ "${BASH_SOURCE[0]}" = "$0" ] || return 0
|
||||
[ $# -gt 0 ] || usage
|
||||
CMD="$1";shift
|
||||
case "$CMD" in
|
||||
|
||||
Executable
+788
@@ -0,0 +1,788 @@
|
||||
#!/usr/bin/env bash
|
||||
# Stack activate and rollback, health-gated (D1, #1659; docs/plans/2026-10-18_stack-release.md).
|
||||
# Moves the bus, the Console and the human CLI from the pointer's copy to another copy together,
|
||||
# behind a health gate, and puts the old stack back inside the same run when the gate fails.
|
||||
# Design and tests: agents/darkwing/work/queue-162/BUILD.md.
|
||||
#
|
||||
# Who runs it. Decision 96 (Mos, mos-0716) lets this script stop and start the live units under
|
||||
# four conditions:
|
||||
# 1. every adopt, activate or rollback runs on Mos's go, and Jason or Mos runs it;
|
||||
# 2. a run that migrates the store (activate --migrate) happens with Jason present;
|
||||
# 3. a rollback --restore with events written after the backup waits for Jason, which is why it
|
||||
# needs --events N;
|
||||
# 4. this script never prunes backups, rollback sets, post-attempt directories or copies.
|
||||
# Decisions 94 and 102 still decide who starts the Console; webui-service.sh keeps having no start
|
||||
# or stop verb. status and --plan only read, and every seat may run them.
|
||||
#
|
||||
# Most functions below run as `f || handler`, and bash ignores set -e inside such a call, so
|
||||
# every step that can fail carries its own `|| return 1`.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
REPO="$(pwd)"
|
||||
source scripts/common.sh
|
||||
|
||||
COPIES="${XDG_DATA_HOME:-$HOME/.local/share}/mosaic-stack/copies"
|
||||
BIN="$HOME/.npm-global/bin"
|
||||
UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user"
|
||||
CONSOLE_UNIT=mosaic-webui.service
|
||||
TIMEOUT=60 WANT_PM=0 MIGRATE=0 PLAN=0 RESTORE="" EVENTS=""
|
||||
|
||||
usage() {
|
||||
cat >&2 <<'USAGE'
|
||||
usage: stack-release.sh adopt BUSINESS
|
||||
stack-release.sh activate ID BUSINESS [--pm] [--migrate] [--timeout S] [--plan]
|
||||
stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan]
|
||||
stack-release.sh status BUSINESS
|
||||
common options: --copies DIR, --bin DIR
|
||||
USAGE
|
||||
exit "${1:-4}"
|
||||
}
|
||||
say() { echo "stack-release: $*"; [ -z "${STEPS:-}" ] || echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $*" >> "$STEPS" || true; }
|
||||
now() { date -u +%Y-%m-%dT%H:%M:%SZ; }
|
||||
now_ms() { date -u +%Y-%m-%dT%H:%M:%S.%3NZ; }
|
||||
|
||||
# One JSON object from KEY VALUE pairs; the value @null is JSON null.
|
||||
jline() {
|
||||
node -e 'const a=process.argv.slice(1),o={};for(let i=0;i<a.length;i+=2)o[a[i]]=a[i+1]==="@null"?null:a[i+1];process.stdout.write(JSON.stringify(o)+"\n")' -- "$@"
|
||||
}
|
||||
log_line() { jline at "$(now)" op "$OP" from "${FROM:-@null}" to "${TO:-@null}" backup "${BACKUP:-@null}" activation "${ACT:-@null}" "$@" >> "$LOG"; }
|
||||
|
||||
# A refusal before anything stopped: exit 3, and one log line with the code, except for
|
||||
# run-locked (the lock holder may be writing the log) and in --plan, which writes nothing.
|
||||
refuse() {
|
||||
local code="$1"; shift
|
||||
echo "stack-release: refusing $code: $*" >&2
|
||||
[ "$code" = run-locked ] || [ "$PLAN" = 1 ] || log_line result refused code "$code"
|
||||
exit 3
|
||||
}
|
||||
|
||||
parse_opts() {
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--copies) [ $# -ge 2 ] || usage; COPIES="$2"; shift 2 ;;
|
||||
--bin) [ $# -ge 2 ] || usage; BIN="$2"; shift 2 ;;
|
||||
--timeout) [ "$OP" != status ] && [ "$OP" != adopt ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;;
|
||||
--plan) [ "$OP" = activate ] || [ "$OP" = rollback ] || usage; PLAN=1; shift ;;
|
||||
--pm) [ "$OP" = activate ] || usage; WANT_PM=1; shift ;;
|
||||
--migrate) [ "$OP" = activate ] || usage; MIGRATE=1; shift ;;
|
||||
--restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$ ]] || usage; RESTORE="$2"; shift 2 ;;
|
||||
--events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
[ -z "$EVENTS" ] || [ -n "$RESTORE" ] || usage
|
||||
local d
|
||||
for d in "$COPIES" "$BIN"; do
|
||||
case "$d" in /*) ;; *) usage ;; esac
|
||||
case "$d" in *[!A-Za-z0-9/._+-]*) usage ;; esac
|
||||
done
|
||||
}
|
||||
|
||||
# --- The pointer and the log (item 1) ---
|
||||
|
||||
# Sets P_COPY P_BUSINESS P_PM P_CLAUDE P_SCHEMA P_ACT and the array P_CONSOLE; returns 1 with no
|
||||
# pointer. A pointer that is there but malformed stops the script: Jason or Mos settles it.
|
||||
read_pointer() {
|
||||
[ -f "$POINTER" ] || return 1
|
||||
local fields
|
||||
fields="$(node -e '
|
||||
const p=JSON.parse(require("fs").readFileSync(process.argv[1],"utf8"));
|
||||
const ok=/^[0-9a-f]{40}$/.test(p.copy)&&typeof p.business==="string"&&typeof p.pm==="boolean"&&typeof p.claude==="string"&&typeof p.schema==="string"&&typeof p.activation==="string"&&Array.isArray(p.console)&&p.console.every((t)=>typeof t==="string"&&t.length>0&&!/[\n\r]/.test(t));
|
||||
if(!ok)throw new Error("pointer fields are missing or malformed");
|
||||
process.stdout.write([p.copy,p.business,p.pm?1:0,p.claude,p.schema,p.activation,...p.console].join("\n")+"\n");' -- "$POINTER")" || { echo "stack-release: $POINTER is unreadable; Jason or Mos settles it" >&2; exit 1; }
|
||||
local -a f
|
||||
mapfile -t f <<< "$fields"
|
||||
P_COPY="${f[0]}" P_BUSINESS="${f[1]}" P_PM="${f[2]}" P_CLAUDE="${f[3]}" P_SCHEMA="${f[4]}" P_ACT="${f[5]}"
|
||||
P_CONSOLE=("${f[@]:6}")
|
||||
}
|
||||
|
||||
# Writes the pointer through a temp file in state/ and a rename. Args: copy pm claude schema console...
|
||||
write_pointer() {
|
||||
local tmp
|
||||
tmp="$(mktemp "$STATE/.stack-active.XXXXXX")"
|
||||
node -e '
|
||||
const [copy,business,pm,claude,schema,activation,at,...console]=process.argv.slice(1);
|
||||
process.stdout.write(JSON.stringify({copy,business,pm:pm==="1",claude,console,schema,activation,at})+"\n");' -- \
|
||||
"$1" "$BUSINESS" "$2" "$3" "$4" "$ACT" "$(now)" "${@:5}" > "$tmp"
|
||||
chmod 0644 "$tmp"
|
||||
mv -f "$tmp" "$POINTER"
|
||||
}
|
||||
|
||||
# Prints the activation id of each `started` line with no final line after it.
|
||||
interrupted() {
|
||||
[ -f "$LOG" ] || return 0
|
||||
node -e '
|
||||
const open=new Map();
|
||||
for(const l of require("fs").readFileSync(process.argv[1],"utf8").split("\n")){
|
||||
if(!l.trim())continue;
|
||||
let o;try{o=JSON.parse(l)}catch{continue}
|
||||
if(!o.activation)continue;
|
||||
if(o.result==="started")open.set(o.activation,o);else open.delete(o.activation);
|
||||
}
|
||||
for(const k of open.keys())process.stdout.write(k+"\n");' -- "$LOG"
|
||||
}
|
||||
|
||||
# Reads the log's ok lines, newest first. `previous` prints rollback's target: the first `to`
|
||||
# that is not the pointer's copy (release.sh's rule). `backup` prints the backup field of the
|
||||
# line that made the pointer's copy live, the backup a rollback would restore.
|
||||
from_log() {
|
||||
[ -f "$LOG" ] || return 1
|
||||
node -e '
|
||||
const [file,mode,current]=process.argv.slice(1);
|
||||
const lines=require("fs").readFileSync(file,"utf8").split("\n").filter((l)=>l.trim());
|
||||
for(const l of lines.reverse()){
|
||||
let o;try{o=JSON.parse(l)}catch{continue}
|
||||
if(o.result!=="ok"||!["adopt","activate","rollback"].includes(o.op)||!o.to)continue;
|
||||
if(mode==="previous"&&o.to!==current){process.stdout.write(o.to+"\n");process.exit(0)}
|
||||
if(mode==="backup"&&o.to===current){if(o.backup)process.stdout.write(o.backup+"\n");process.exit(o.backup?0:1)}
|
||||
}
|
||||
process.exit(1);' -- "$LOG" "$1" "$P_COPY"
|
||||
}
|
||||
|
||||
# --- Reading the units ---
|
||||
|
||||
show() { systemctl --user show "$1" -p LoadState -p WorkingDirectory -p ExecStart -p Environment -p MainPID -p DropInPaths 2>/dev/null; }
|
||||
argv_of() { sed -n 's/^ExecStart=.*argv\[\]=\([^;]*\) ;.*/\1/p' <<< "$1"; }
|
||||
|
||||
# Sets U_PM U_CLAUDE U_CONSOLE (array) from the effective units; returns 1 if a unit can't be read.
|
||||
read_units() {
|
||||
local bus console tok
|
||||
bus="$(show "$BUS_UNIT")" && console="$(show "$CONSOLE_UNIT")" || return 1
|
||||
[ "$(sed -n 's/^LoadState=//p' <<< "$bus")" = loaded ] && [ "$(sed -n 's/^LoadState=//p' <<< "$console")" = loaded ] || return 1
|
||||
local -a argv
|
||||
read -r -a argv <<< "$(argv_of "$bus")"
|
||||
U_PM=0
|
||||
for tok in "${argv[@]:1}"; do [ "$tok" != --pm ] || U_PM=1; done
|
||||
U_CLAUDE=""
|
||||
read -r -a argv <<< "$(sed -n 's/^Environment=//p' <<< "$bus")"
|
||||
for tok in "${argv[@]}"; do case "$tok" in MOSAIC_CLAUDE_BIN=*) U_CLAUDE="${tok#MOSAIC_CLAUDE_BIN=}" ;; esac; done
|
||||
read -r -a argv <<< "$(argv_of "$console")"
|
||||
U_CONSOLE=()
|
||||
local i serve=0
|
||||
for ((i = 0; i < ${#argv[@]}; i++)); do
|
||||
if [ "$serve" = 0 ]; then [ "${argv[i]}" != serve ] || serve=1; continue; fi
|
||||
if [ "${argv[i]}" = --checkout ]; then i=$((i + 1)); continue; fi
|
||||
U_CONSOLE+=("${argv[i]}")
|
||||
done
|
||||
[ "$serve" = 1 ] || return 1
|
||||
}
|
||||
|
||||
console_port() {
|
||||
local i
|
||||
for ((i = 0; i < ${#P_CONSOLE[@]}; i++)); do
|
||||
[ "${P_CONSOLE[i]}" != --port ] || { echo "${P_CONSOLE[i + 1]:-7332}"; return; }
|
||||
done
|
||||
echo 7332
|
||||
}
|
||||
|
||||
# --- Copies ---
|
||||
|
||||
# stack-copy.sh's own helpers, in a subshell so its die, parse and globals stay out of this script.
|
||||
copy_ok() ( source "$REPO/scripts/stack-copy.sh"; ID="$1"; COPIES="$COPIES_DIR"; copy_root >/dev/null && reachable "$ID" )
|
||||
copy_exists() ( source "$REPO/scripts/stack-copy.sh"; ID="$1"; COPIES="$COPIES_DIR"; copy_root >/dev/null )
|
||||
|
||||
# Prints the copy's SCHEMA_VERSION, then each `from:` of its migration chain, one per line. Text
|
||||
# only: the copy's code is never imported or run for this.
|
||||
copy_schema() {
|
||||
local f="$COPIES/$1/packages/bus/src/store.mjs" v
|
||||
[ -f "$f" ] || return 1
|
||||
v="$(grep -E "^export const SCHEMA_VERSION = '[A-Za-z0-9]+';\$" "$f")" || return 1
|
||||
[ "$(wc -l <<< "$v")" = 1 ] || return 1
|
||||
v="${v#*\'}"; echo "${v%%\'*}"
|
||||
{ grep -E "^[[:space:]]*from: '[A-Za-z0-9]+',\$" "$f" || true; } | sed "s/.*from: '\\([A-Za-z0-9]*\\)',\$/\\1/"
|
||||
}
|
||||
|
||||
# Sets T_SCHEMA and T_CHAIN (array) for copy $1; returns 1 when either read fails.
|
||||
target_schema() {
|
||||
local out
|
||||
out="$(copy_schema "$1")" || return 1
|
||||
local -a l
|
||||
mapfile -t l <<< "$out"
|
||||
T_SCHEMA="${l[0]}"
|
||||
T_CHAIN=("${l[@]:1}")
|
||||
}
|
||||
|
||||
in_chain() { local v; for v in "${T_CHAIN[@]}"; do [ "$v" != "$1" ] || return 0; done; return 1; }
|
||||
|
||||
# The item 3 checks on the target copy, in order: copy, schema, already-active.
|
||||
check_target() {
|
||||
copy_exists "$TO" 2>/dev/null || refuse copy-missing "no copy $TO in $COPIES"
|
||||
copy_ok "$TO" 2>/dev/null || refuse copy-unreachable "$TO is not reachable from refactor and origin/refactor"
|
||||
target_schema "$TO" || refuse copy-schema-unknown "cannot read SCHEMA_VERSION and its chain from $COPIES/$TO/packages/bus/src/store.mjs"
|
||||
[ "$TO" != "$P_COPY" ] || refuse already-active "$TO is the pointer's copy"
|
||||
}
|
||||
|
||||
# --- The store ---
|
||||
|
||||
STORE_JS='
|
||||
const {DatabaseSync}=require("node:sqlite");const {pathToFileURL}=require("node:url");
|
||||
const [mode,file,stamp]=process.argv.slice(1);
|
||||
// immutable=1 on a stopped store with no -wal or -shm: it creates no file and leaves the main
|
||||
// file mtime and sha256 as they were (the row 162 probe). A plain readOnly open of a stopped
|
||||
// WAL store creates -shm and -wal, so it is used only on a store the running bus holds.
|
||||
const db=mode.startsWith("held")?new DatabaseSync(file,{readOnly:true}):new DatabaseSync(pathToFileURL(file).href+"?immutable=1",{readOnly:true});
|
||||
try{
|
||||
if(mode.endsWith("version"))process.stdout.write(db.prepare("SELECT value FROM meta WHERE key=?").get("schema_version").value+"\n");
|
||||
else process.stdout.write(db.prepare("SELECT count(*) AS n FROM events WHERE at > ?").get(stamp).n+"\n");
|
||||
}finally{db.close()}'
|
||||
|
||||
# Sets UNCLEAN and returns 0 when a -wal or -shm sits beside the store. An immutable read would
|
||||
# skip that WAL and could give a stale answer, so the caller refuses store-unclean and leaves the
|
||||
# store as it is. Called directly, never inside $(...), so UNCLEAN reaches the caller.
|
||||
store_unclean() {
|
||||
local f
|
||||
for f in "$STORE-wal" "$STORE-shm"; do
|
||||
if [ -e "$f" ] || [ -L "$f" ]; then UNCLEAN="$f"; return 0; fi
|
||||
done
|
||||
return 1
|
||||
}
|
||||
# A read of the stopped store: version, or count STAMP. Call store_unclean first.
|
||||
store_read() { node -e "$STORE_JS" -- "$1" "$STORE" "${2:-}"; }
|
||||
|
||||
# No writer.lock, and no process with a file open under <dataRoot>/bus/ (stopped-check.sh's
|
||||
# test). One find over the /proc fd links, not a readlink per link: this host has about 16,000.
|
||||
# find's status is ignored, since other users' fd dirs are unreadable; so were they in the loop.
|
||||
stopped() {
|
||||
[ ! -e "$DATA/bus/writer.lock" ] || return 1
|
||||
local pat
|
||||
pat="$(sed 's/[][*?\\]/\\&/g' <<< "$DATA/bus")/*"
|
||||
[ -z "$(find /proc/[0-9]*/fd -mindepth 1 -maxdepth 1 -lname "$pat" -print -quit 2>/dev/null || true)" ]
|
||||
}
|
||||
|
||||
sums() { (cd "$1" && find . -type f ! -name SHA256SUMS ! -name steps.log -print0 | sort -z | xargs -0 -r sha256sum); }
|
||||
sum_of() { awk -v f="./$2" '$2 == f { print $1 }' "$1/SHA256SUMS"; }
|
||||
|
||||
# The backup of the stopped store (backup.sh's steps). Only bus.sqlite: a -wal or -shm was
|
||||
# already refused as store-unclean. Sets TAKEN once the backup is whole.
|
||||
backup_store() {
|
||||
local o="$DATA/backups/stack/$ACT"
|
||||
(
|
||||
umask 077
|
||||
mkdir -p "$DATA/backups/stack" &&
|
||||
mkdir -m 700 "$o" "$o/bus" &&
|
||||
(cd "$DATA/bus" && sha256sum bus.sqlite) > "$o/SOURCE.sha256" &&
|
||||
cp -p "$STORE" "$o/bus/" &&
|
||||
(cd "$o/bus" && sha256sum bus.sqlite) > "$o/SHA256SUMS" &&
|
||||
cmp -s "$o/SOURCE.sha256" "$o/SHA256SUMS" &&
|
||||
stat -c '%n %a %U %s %y' "$STORE" > "$o/MODES.txt" &&
|
||||
ls -1A "$DATA/bus" > "$o/LISTING.txt" &&
|
||||
echo "$DATA" > "$o/DATA_ROOT" &&
|
||||
now_ms > "$o/STAMP" &&
|
||||
echo "$STORE_SCHEMA" > "$o/SCHEMA" &&
|
||||
echo "$RS" > "$o/ROLLBACK_SET"
|
||||
) || return 1
|
||||
stopped || return 1
|
||||
TAKEN="$ACT" BACKUP="$ACT"
|
||||
say "backup: $o"
|
||||
}
|
||||
|
||||
# Puts backup $1 back as restore.sh does: the current bus/ moves to $POST, nothing is deleted.
|
||||
# Sets RESTORED once the old bus/ has moved, so the caller knows the store was swapped.
|
||||
restore_store() {
|
||||
local o="$DATA/backups/stack/$1"
|
||||
[ "$(cat "$o/DATA_ROOT" 2>/dev/null)" = "$DATA" ] || return 1
|
||||
stopped || return 1
|
||||
(cd "$o/bus" && sha256sum -c --quiet ../SHA256SUMS) || return 1
|
||||
[ ! -e "$POST" ] || return 1
|
||||
mv "$DATA/bus" "$POST" || return 1
|
||||
RESTORED=1
|
||||
(umask 077; mkdir -m 700 "$DATA/bus") || return 1
|
||||
cp -p "$o"/bus/* "$DATA/bus/" || return 1
|
||||
(cd "$DATA/bus" && sha256sum -c --quiet "$o/SHA256SUMS") || return 1
|
||||
[ "$(ls -1A "$DATA/bus")" = "$(ls -1A "$o/bus")" ] || return 1
|
||||
say "restored $o; the store from before the restore is at $POST"
|
||||
}
|
||||
|
||||
# --- The run (item 4) ---
|
||||
|
||||
# Step 1: the rollback set. A failure here has changed nothing but the set itself.
|
||||
capture() {
|
||||
mkdir -p -m 0700 "$STATE/stack-rollback" 2>/dev/null || return 1
|
||||
mkdir -m 0700 "$RS" 2>/dev/null || return 1
|
||||
STEPS="$RS/steps.log"
|
||||
local u d
|
||||
for u in [email protected] "$CONSOLE_UNIT"; do
|
||||
[ -f "$UNIT_DIR/$u" ] || { say "capture: no $UNIT_DIR/$u"; return 1; }
|
||||
cp -p "$UNIT_DIR/$u" "$RS/$u" || return 1
|
||||
done
|
||||
for d in [email protected] "mosaic-bus@$BUSINESS.service.d" "$CONSOLE_UNIT.d"; do
|
||||
[ ! -d "$UNIT_DIR/$d" ] || cp -pr "$UNIT_DIR/$d" "$RS/$d" || return 1
|
||||
done
|
||||
[ -f "$BIN/mosaic" ] || { say "capture: no wrapper $BIN/mosaic"; return 1; }
|
||||
cp -p "$BIN/mosaic" "$RS/wrapper" || return 1
|
||||
stat -c %a "$BIN/mosaic" > "$RS/WRAPPER_MODE" || return 1
|
||||
for u in "$BUS_UNIT" "$CONSOLE_UNIT"; do show "$u" > "$RS/show-$u.txt" || return 1; done
|
||||
sums "$RS" > "$RS/SHA256SUMS" || return 1
|
||||
say "captured the rollback set: $RS"
|
||||
}
|
||||
|
||||
# Stops the Console, then the bus, and waits for both and the stopped check, within --timeout.
|
||||
stop_both() {
|
||||
systemctl --user stop "$CONSOLE_UNIT" || return 1
|
||||
systemctl --user stop "$BUS_UNIT" || return 1
|
||||
local deadline=$((SECONDS + TIMEOUT)) u s down
|
||||
while :; do
|
||||
down=1
|
||||
for u in "$CONSOLE_UNIT" "$BUS_UNIT"; do
|
||||
s="$(systemctl --user is-active "$u" 2>/dev/null || true)"
|
||||
case "$s" in inactive|failed) ;; *) down=0 ;; esac
|
||||
done
|
||||
if [ "$down" = 1 ] && stopped; then say "stopped both units"; return 0; fi
|
||||
[ "$SECONDS" -lt "$deadline" ] || return 1
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
start_both() {
|
||||
START_EPOCH="$(date +%s)" && START_MS="$(date +%s%3N)" || return 1
|
||||
systemctl --user start "$BUS_UNIT" && systemctl --user start "$CONSOLE_UNIT"
|
||||
}
|
||||
|
||||
# Step 4: installs copy $1 with pm $2, claude $3 and the Console flags after them. Sets FAIL.
|
||||
install_stack() {
|
||||
local id="$1" pm="$2" claude="$3"; shift 3
|
||||
local -a pmflag=()
|
||||
[ "$pm" = 0 ] || pmflag=(--pm)
|
||||
scripts/bus-service.sh install --copy "$id" --copies "$COPIES" "${pmflag[@]}" --claude "$claude" >> "$STEPS" 2>&1 || { FAIL=install-bus; return 1; }
|
||||
scripts/webui-service.sh install --copy "$id" --copies "$COPIES" "$@" >> "$STEPS" 2>&1 || { FAIL=install-console; return 1; }
|
||||
scripts/stack-copy.sh wrapper "$id" --copies "$COPIES" --bin "$BIN" >> "$STEPS" 2>&1 || { FAIL=wrapper; return 1; }
|
||||
say "installed $id"
|
||||
}
|
||||
|
||||
# --- The health gate (item 5) ---
|
||||
|
||||
# One pass of every check against copy $1 with pm $2. Sets GATE to the first failing code and
|
||||
# GATE_WHY to what it saw.
|
||||
gate_once() {
|
||||
local copy="$COPIES/$1" pm="$2" main out
|
||||
GATE=units-inactive GATE_WHY=""
|
||||
systemctl --user is-active --quiet "$BUS_UNIT" && systemctl --user is-active --quiet "$CONSOLE_UNIT" || return 1
|
||||
GATE=bus-status
|
||||
main="$(systemctl --user show "$BUS_UNIT" -p MainPID | sed -n 's/^MainPID=//p')" || return 1
|
||||
out="$("$copy/scripts/mosaic" bus status --json 2>/dev/null)" || { GATE_WHY="bus status failed"; return 1; }
|
||||
# Every `bus start` writes boot.at (packages/cli/src/boot-status.mjs), so a value from before
|
||||
# this start means the host on the socket is not the one this run started.
|
||||
GATE_WHY="$(node -e '
|
||||
const [b,main,since]=process.argv.slice(2),s=JSON.parse(process.argv[1]);
|
||||
const why=!s.host?"no host":s.host.business!==b?`host for ${s.host.business}`:s.host.live!==true?"host not live":String(s.host.pid)!==main?`host pid ${s.host.pid} is not MainPID ${main}`:!s.boot||s.boot.ok!==true?"boot status not ok":s.boot.business!==b?`boot status for ${s.boot.business}`:!(Date.parse(s.boot.at)>=Number(since))?"boot status is from before this start":"";
|
||||
process.stdout.write(why);' -- "$out" "$BUSINESS" "$main" "$START_MS" 2>&1)" || { GATE_WHY="bus status unreadable"; return 1; }
|
||||
[ -z "$GATE_WHY" ] || return 1
|
||||
GATE=healthz
|
||||
node -e '
|
||||
fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>process.exit(r.status===200?0:1),()=>process.exit(1));' -- "$(console_port)" || return 1
|
||||
GATE=copy-check
|
||||
scripts/stack-copy.sh check --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" > /dev/null 2>&1 || return 1
|
||||
if [ "$pm" = 1 ]; then
|
||||
GATE=pm-launch-unconfirmed
|
||||
pm_launched "$copy" || return 1
|
||||
fi
|
||||
GATE=""
|
||||
}
|
||||
|
||||
# Ruling 2 (sage-0736, sage-0738). The PM's run is the one named on the single
|
||||
# `launched <instance>: run <run>, <harness> <model>` line that `bus start --pm` prints
|
||||
# (packages/cli/src/cli.mjs) to the unit's journal since this start. The gate depends on that
|
||||
# wording: if it changes, this check refuses rather than passes. --since takes @<epoch seconds>,
|
||||
# because this host's journalctl reads a bare time as local time.
|
||||
pm_launched() {
|
||||
local journal run list
|
||||
journal="$(journalctl --user -u "$BUS_UNIT" --since "@$START_EPOCH" -o cat 2>/dev/null)" || { GATE_WHY="journal read failed"; return 1; }
|
||||
local -a lines=()
|
||||
mapfile -t lines < <(grep '^launched ' <<< "$journal" || true)
|
||||
[ "${#lines[@]}" -gt 0 ] || { GATE_WHY="no launched line since the start"; return 1; }
|
||||
[ "${#lines[@]}" = 1 ] || { GATE_WHY="${#lines[@]} launched lines since the start"; return 1; }
|
||||
[[ "${lines[0]}" =~ ^launched\ ([^[:space:]]+):\ run\ ([^[:space:]]+),\ ]] || { GATE_WHY="the launched line does not parse: ${lines[0]}"; return 1; }
|
||||
run="${BASH_REMATCH[2]}"
|
||||
list="$("$1/scripts/mosaic" launches list --json 2>/dev/null)" || { GATE_WHY="launches list failed"; return 1; }
|
||||
GATE_WHY="$(node -e '
|
||||
const [list,run,b,since]=process.argv.slice(1),s=JSON.parse(list).find((x)=>x.run===run);
|
||||
process.stdout.write(!s?`run ${run} is not in launches list`:s.business!==b?`run ${run} is for ${s.business}`:s.live!==true?`run ${run} is not live`:!(Date.parse(s.startedAt)>=Number(since))?`run ${run} started before this start`:"");' -- "$list" "$run" "$BUSINESS" "$START_MS" 2>&1)" || { GATE_WHY="launches list unreadable"; return 1; }
|
||||
[ -z "$GATE_WHY" ]
|
||||
}
|
||||
|
||||
# Polls every second until every check passes or --timeout runs out.
|
||||
gate() {
|
||||
local deadline=$((SECONDS + TIMEOUT))
|
||||
while :; do
|
||||
if gate_once "$1" "$2"; then say "gate passed on $1"; return 0; fi
|
||||
[ "$SECONDS" -lt "$deadline" ] || { say "gate failed on $1: $GATE${GATE_WHY:+ ($GATE_WHY)}"; return 1; }
|
||||
sleep 1
|
||||
done
|
||||
}
|
||||
|
||||
# --- Rollback inside the run (item 6) ---
|
||||
|
||||
# Puts back the captured unit files, drop-ins and wrapper, checked against SHA256SUMS.
|
||||
put_back() {
|
||||
local u f rel tmp
|
||||
for u in [email protected] "$CONSOLE_UNIT"; do
|
||||
tmp="$(mktemp "$UNIT_DIR/.$u.XXXXXX")" && cp -p "$RS/$u" "$tmp" && mv -f "$tmp" "$UNIT_DIR/$u" || return 1
|
||||
done
|
||||
for u in [email protected] "mosaic-bus@$BUSINESS.service.d" "$CONSOLE_UNIT.d"; do
|
||||
[ -d "$RS/$u" ] || continue
|
||||
while IFS= read -r -d '' f; do
|
||||
rel="${f#"$RS/$u/"}"
|
||||
! cmp -s "$f" "$UNIT_DIR/$u/$rel" 2>/dev/null || continue
|
||||
mkdir -p "$(dirname "$UNIT_DIR/$u/$rel")" || return 1
|
||||
tmp="$(mktemp "$UNIT_DIR/$u/.restore.XXXXXX")" && cp -p "$f" "$tmp" && mv -f "$tmp" "$UNIT_DIR/$u/$rel" || return 1
|
||||
done < <(find "$RS/$u" -type f -print0)
|
||||
done
|
||||
tmp="$(mktemp "$BIN/.mosaic.XXXXXX")" && cp -p "$RS/wrapper" "$tmp" && chmod "$(cat "$RS/WRAPPER_MODE")" "$tmp" && mv -f "$tmp" "$BIN/mosaic" || return 1
|
||||
for u in [email protected] "$CONSOLE_UNIT"; do
|
||||
[ "$(sha256sum < "$UNIT_DIR/$u" | cut -d' ' -f1)" = "$(sum_of "$RS" "$u")" ] || return 1
|
||||
done
|
||||
[ "$(sha256sum < "$BIN/mosaic" | cut -d' ' -f1)" = "$(sum_of "$RS" wrapper)" ] || return 1
|
||||
[ "$(stat -c %a "$BIN/mosaic")" = "$(cat "$RS/WRAPPER_MODE")" ] || return 1
|
||||
systemctl --user daemon-reload || return 1
|
||||
say "put back the unit files and the wrapper from $RS"
|
||||
}
|
||||
|
||||
# A failure after step 2: put the pointer's stack back and gate it. $1 is the run's code. Only a
|
||||
# backup this run took (TAKEN) is restored; a rollback's --restore backup never is.
|
||||
rollback_in_run() {
|
||||
trap - INT TERM HUP
|
||||
local code="$1" rcode=""
|
||||
say "rolling back: $code"
|
||||
if ! stop_both; then rcode=stop-timeout
|
||||
elif ! put_back; then rcode=put-back
|
||||
elif [ -n "$TAKEN" ] && ! restore_store "$TAKEN"; then rcode=restore
|
||||
elif ! start_both; then rcode=start
|
||||
elif ! gate "$P_COPY" "$P_PM"; then rcode="$GATE"
|
||||
fi
|
||||
if [ -z "$rcode" ]; then
|
||||
log_line result gate-failed code "$code" rollback ok
|
||||
echo "stack-release: $code; rolled back to $P_COPY, which passed the gate" >&2
|
||||
exit 1
|
||||
fi
|
||||
rollback_failed "$code" "$rcode"
|
||||
}
|
||||
|
||||
# Ruling 4 (T14b): once rollback --restore has swapped the store, any later failure skips item 6,
|
||||
# since putting the newer store back would be a second swap. Before the swap, item 6 runs.
|
||||
after_fail() {
|
||||
[ -z "$RESTORED" ] || rollback_failed "$1" not-attempted
|
||||
rollback_in_run "$1"
|
||||
}
|
||||
|
||||
# Stop both units, print both results, log rollback-failed, exit 2. No retry.
|
||||
rollback_failed() {
|
||||
trap - INT TERM HUP
|
||||
systemctl --user stop "$CONSOLE_UNIT" || true
|
||||
systemctl --user stop "$BUS_UNIT" || true
|
||||
say "rollback-failed: run $1, rollback $2"
|
||||
log_line result rollback-failed code "$1" rollback failed rollbackCode "$2"
|
||||
echo "stack-release: rollback-failed: the run failed with $1 and the rollback with $2; both units are stopped. Rollback set: $RS${BACKUP:+, backup $DATA/backups/stack/$BACKUP}${RESTORED:+, post-attempt store $POST}" >&2
|
||||
exit 2
|
||||
}
|
||||
|
||||
# --- Refusals (item 3) ---
|
||||
|
||||
lock() {
|
||||
mkdir -p "$STATE"
|
||||
exec 9> "$STATE/stack-release.lock"
|
||||
flock -n 9 || refuse run-locked "another stack-release run holds $STATE/stack-release.lock"
|
||||
local open
|
||||
open="$(interrupted | paste -sd' ' -)"
|
||||
[ -z "$open" ] || refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)"
|
||||
}
|
||||
|
||||
instance_busy() {
|
||||
local out unit load state rest
|
||||
out="$(systemctl --user list-units --all --no-legend --plain 'mosaic-bus@*.service' 2>/dev/null)" || refuse instance-busy "systemctl --user list-units failed, so the bus state is unknown"
|
||||
while read -r unit load state rest; do
|
||||
[[ "$unit" == mosaic-bus@*.service ]] || continue
|
||||
if [ "$unit" = "$BUS_UNIT" ]; then
|
||||
case "$state" in activating|deactivating|reloading) refuse instance-busy "$unit is $state" ;; esac
|
||||
else
|
||||
case "$state" in inactive|failed) ;; *) refuse instance-busy "$unit is $state; the copy and wrapper switch reach it too" ;; esac
|
||||
fi
|
||||
done <<< "$out"
|
||||
}
|
||||
|
||||
# install_guard's drop-in scan (bus-service.sh), done before anything stops.
|
||||
drop_in_override() {
|
||||
local paths directory dropdir file
|
||||
paths="$(systemd-analyze --user unit-paths 2>/dev/null)" && [ -n "$paths" ] || refuse drop-in-override "systemd unit paths are unknown"
|
||||
while IFS= read -r directory; do
|
||||
[ -n "$directory" ] && [ -d "$directory" ] || continue
|
||||
for dropdir in "$directory"/mosaic-bus@*.service.d "$directory"/mosaic-.service.d "$directory"/service.d; do
|
||||
[ -d "$dropdir" ] || continue
|
||||
for file in "$dropdir"/*.conf; do
|
||||
[ -e "$file" ] || continue
|
||||
if grep -Eq '^[[:space:]]*(ExecStart|WorkingDirectory)[[:space:]]*=' "$file" 2>/dev/null; then
|
||||
refuse drop-in-override "$file overrides ExecStart or WorkingDirectory; the operator removes it as in TOOLS.md step 4"
|
||||
fi
|
||||
done
|
||||
done
|
||||
done <<< "$paths"
|
||||
}
|
||||
|
||||
# unit-flags-drift: the effective units against the pointer, and --pm against the pointer's pm.
|
||||
flags_drift() {
|
||||
local d=""
|
||||
[ "$U_PM" = "$P_PM" ] || d+=$'\n'" pm: pointer $P_PM, unit $U_PM"
|
||||
[ "$U_CLAUDE" = "$P_CLAUDE" ] || d+=$'\n'" claude: pointer $P_CLAUDE, unit $U_CLAUDE"
|
||||
[ "${P_CONSOLE[*]}" = "${U_CONSOLE[*]}" ] || d+=$'\n'" console: pointer ${P_CONSOLE[*]}, unit ${U_CONSOLE[*]}"
|
||||
[ "$OP" != activate ] || [ "$WANT_PM" = "$P_PM" ] || d+=$'\n'" pm: pointer $P_PM, --pm given $WANT_PM"
|
||||
[ -z "$d" ] || refuse unit-flags-drift "the units differ from the pointer; settle it first:$d"
|
||||
}
|
||||
|
||||
# The item 3 checks after the pointer, shared by activate and rollback, for target $TO.
|
||||
precheck() {
|
||||
check_target
|
||||
instance_busy
|
||||
drop_in_override
|
||||
read_units || refuse unit-flags-drift "cannot read $BUS_UNIT or $CONSOLE_UNIT"
|
||||
[ -n "$U_CLAUDE" ] || refuse no-claude-pin "$BUS_UNIT has no MOSAIC_CLAUDE_BIN"
|
||||
flags_drift
|
||||
}
|
||||
|
||||
pointer_for_business() {
|
||||
read_pointer || refuse no-pointer "no $POINTER; run adopt first"
|
||||
FROM="$P_COPY"
|
||||
[ "$P_BUSINESS" = "$BUSINESS" ] || refuse business-mismatch "the pointer is for $P_BUSINESS"
|
||||
}
|
||||
|
||||
new_activation() {
|
||||
ACT="$(date -u +%Y%m%dT%H%M%SZ)-${TO:0:12}"
|
||||
RS="$STATE/stack-rollback/$ACT"
|
||||
POST="$DATA/bus.post-attempt-$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
}
|
||||
|
||||
# --- Verbs ---
|
||||
|
||||
do_adopt() {
|
||||
lock
|
||||
[ ! -f "$POINTER" ] || { read_pointer; FROM="$P_COPY"; refuse already-adopted "$POINTER exists (copy $P_COPY)"; }
|
||||
local out
|
||||
out="$(scripts/stack-copy.sh check --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1)" || refuse check-failed "stack-copy.sh check did not pass:"$'\n'"$out"
|
||||
TO="$(sed -n 's/^bus: //p' <<< "$out")"
|
||||
TO="${TO#"$COPIES/"}"
|
||||
read_units || refuse check-failed "cannot read $BUS_UNIT or $CONSOLE_UNIT"
|
||||
[ -n "$U_CLAUDE" ] || refuse no-claude-pin "$BUS_UNIT has no MOSAIC_CLAUDE_BIN"
|
||||
target_schema "$TO" || refuse copy-schema-unknown "cannot read SCHEMA_VERSION from $COPIES/$TO/packages/bus/src/store.mjs"
|
||||
ACT="$(date -u +%Y%m%dT%H%M%SZ)-${TO:0:12}"
|
||||
write_pointer "$TO" "$U_PM" "$U_CLAUDE" "$T_SCHEMA" "${U_CONSOLE[@]}"
|
||||
log_line result ok
|
||||
echo "adopted $TO for $BUSINESS (pm $U_PM, schema $T_SCHEMA)"
|
||||
}
|
||||
|
||||
# The store's version for --plan, from the store itself: readOnly while the bus holds it (adds no
|
||||
# file), immutable when it is stopped and clean. Sets S_SCHEMA and S_FROM. Falls back to the
|
||||
# pointer's schema, and says so, when there is no store file or it can't be read.
|
||||
plan_store() {
|
||||
local src="" why=""
|
||||
S_SCHEMA="" S_FROM=""
|
||||
if [ ! -e "$STORE" ]; then why="there is no store file"
|
||||
elif systemctl --user is-active --quiet "$BUS_UNIT"; then
|
||||
src="the store the running bus holds"
|
||||
S_SCHEMA="$(node -e "$STORE_JS" -- heldversion "$STORE" 2>/dev/null)" || S_SCHEMA=""
|
||||
elif store_unclean; then why="the store was not read: $UNCLEAN is present"
|
||||
else
|
||||
src="the stopped store"
|
||||
S_SCHEMA="$(store_read version 2>/dev/null)" || S_SCHEMA=""
|
||||
fi
|
||||
if [ -n "$S_SCHEMA" ]; then S_FROM="read from $src"
|
||||
else S_SCHEMA="$P_SCHEMA" S_FROM="from the pointer; ${why:-$src could not be read}"
|
||||
fi
|
||||
}
|
||||
|
||||
# sage-0774 and sage-0778: the commits the run adds (activate) or takes away (rollback), so the
|
||||
# rows they carry can be named. Copies where neither is an ancestor of the other refuse the plan
|
||||
# before it prints anything. Sets RANGE and VERB.
|
||||
plan_range() {
|
||||
if git -C "$REPO" merge-base --is-ancestor "$P_COPY" "$TO" 2>/dev/null; then RANGE="$P_COPY..$TO" VERB=adds
|
||||
elif git -C "$REPO" merge-base --is-ancestor "$TO" "$P_COPY" 2>/dev/null; then RANGE="$TO..$P_COPY" VERB="takes away"
|
||||
else refuse copies-unrelated "neither $P_COPY nor $TO is an ancestor of the other, so the plan cannot list the commits"
|
||||
fi
|
||||
}
|
||||
plan_commits() {
|
||||
echo "commits: the run $VERB $(git -C "$REPO" rev-list --count --no-merges "$RANGE") (newest first, at most 200):"
|
||||
git -C "$REPO" log --oneline --no-merges --max-count=200 "$RANGE" | sed 's/^/ /'
|
||||
}
|
||||
|
||||
# Each step from $S_SCHEMA to $T_SCHEMA, from the target store.mjs's `from:` and `to:` pairs. Text
|
||||
# only, like copy_schema. Prints "cannot be read from FILE" when the lines don't pair up or the
|
||||
# steps don't reach $T_SCHEMA.
|
||||
plan_steps() {
|
||||
local f="$COPIES/$TO/packages/bus/src/store.mjs" v="$S_SCHEMA" out="" i
|
||||
local -a l
|
||||
local -A next=()
|
||||
mapfile -t l < <(grep -E "^[[:space:]]*(from|to): '[A-Za-z0-9]+',\$" "$f" | sed -E "s/^[[:space:]]*(from|to): '([A-Za-z0-9]*)',\$/\\1 \\2/")
|
||||
(( ${#l[@]} % 2 == 0 )) || { echo "cannot be read from $f"; return; }
|
||||
for ((i = 0; i < ${#l[@]}; i += 2)); do
|
||||
[[ "${l[i]}" == "from "* && "${l[i+1]}" == "to "* ]] || { echo "cannot be read from $f"; return; }
|
||||
next["${l[i]#from }"]="${l[i+1]#to }"
|
||||
done
|
||||
for ((i = 0; i < ${#next[@]}; i++)); do
|
||||
[ "$v" != "$T_SCHEMA" ] && [ -n "${next[$v]:-}" ] || break
|
||||
out+="${out:+, }$v to ${next[$v]}"; v="${next[$v]}"
|
||||
done
|
||||
if [ "$v" = "$T_SCHEMA" ] && [ -n "$out" ]; then echo "$out"; else echo "cannot be read from $f"; fi
|
||||
}
|
||||
|
||||
# Item 8 and sage-0745: what the run would do, with any migration named plainly. Reads the
|
||||
# pointer, the log, the target's store.mjs and the store's version. Writes nothing.
|
||||
plan() {
|
||||
plan_range
|
||||
plan_store
|
||||
echo "plan: $OP for $BUSINESS"
|
||||
echo "from: $P_COPY"
|
||||
echo "to: $TO"
|
||||
echo "store schema: $S_SCHEMA ($S_FROM)"
|
||||
echo "target schema: $T_SCHEMA"
|
||||
[ "$S_SCHEMA" = "$P_SCHEMA" ] || echo "note: the pointer says $P_SCHEMA; the run reads the stopped store and goes by that"
|
||||
if [ "$S_SCHEMA" = "$T_SCHEMA" ]; then
|
||||
echo "migration: none; the store is at the target's version"
|
||||
elif [ "$OP" = activate ] && in_chain "$S_SCHEMA"; then
|
||||
echo "migration: $S_SCHEMA to $T_SCHEMA, after a backup of the stopped store"
|
||||
echo "migration steps: $(plan_steps)"
|
||||
echo "needs: --migrate$([ "$MIGRATE" = 1 ] && echo ' (given)' || echo ' (not given; the run would roll back with migration-needs-flag)')"
|
||||
echo "needs: Jason present for this run (decision 96 condition 2)"
|
||||
elif [ "$OP" = activate ]; then
|
||||
echo "migration: none possible; $S_SCHEMA is not in $T_SCHEMA's migration chain, so the run would roll back with store-newer"
|
||||
else
|
||||
echo "restore: the store must go back to $T_SCHEMA; needs --restore $(from_log backup || echo ACTIVATION)"
|
||||
fi
|
||||
plan_commits
|
||||
echo "bus: scripts/bus-service.sh install --copy $TO --copies $COPIES$([ "$P_PM" = 0 ] || echo ' --pm') --claude $P_CLAUDE"
|
||||
echo "Console: scripts/webui-service.sh install --copy $TO --copies $COPIES${P_CONSOLE[*]:+ ${P_CONSOLE[*]}}"
|
||||
echo "CLI: scripts/stack-copy.sh wrapper $TO --copies $COPIES --bin $BIN"
|
||||
}
|
||||
|
||||
# Steps 4.1 to 4.5 and the gate for target $TO. $1 is the store step: migrate or restore.
|
||||
run_steps() {
|
||||
log_line result started
|
||||
capture || refuse capture-failed "could not write the rollback set $RS; nothing was stopped"
|
||||
trap 'after_fail signal' INT TERM HUP
|
||||
stop_both || rollback_in_run stop-timeout
|
||||
if [ -e "$STORE" ]; then
|
||||
if store_unclean; then say "$UNCLEAN is present; the store is left as it is for Jason or Mos"; rollback_in_run store-unclean; fi
|
||||
STORE_SCHEMA="$(store_read version)" && [ -n "$STORE_SCHEMA" ] || rollback_in_run schema-unreadable
|
||||
say "stopped store schema: $STORE_SCHEMA"
|
||||
if [ "$1" = migrate ]; then
|
||||
if [ "$STORE_SCHEMA" != "$T_SCHEMA" ]; then
|
||||
in_chain "$STORE_SCHEMA" || rollback_in_run store-newer
|
||||
[ "$MIGRATE" = 1 ] || rollback_in_run migration-needs-flag
|
||||
backup_store || rollback_in_run backup
|
||||
node "$COPIES/$TO/packages/bus/src/migrate-schema.mjs" "$DATA" >> "$STEPS" 2>&1 || rollback_in_run migrate-failed
|
||||
say "migrated the store to $T_SCHEMA"
|
||||
fi
|
||||
else
|
||||
[ "$STORE_SCHEMA" = "$P_SCHEMA" ] || rollback_in_run store-differs
|
||||
if [ -n "$RESTORE" ]; then
|
||||
local n
|
||||
n="$(store_read count "$(cat "$DATA/backups/stack/$RESTORE/STAMP")")" || rollback_in_run schema-unreadable
|
||||
[ "$n" = "$COUNT" ] || rollback_in_run post-backup-events
|
||||
restore_store "$RESTORE" || after_fail restore
|
||||
fi
|
||||
fi
|
||||
elif [ -n "$RESTORE" ]; then
|
||||
rollback_in_run store-differs
|
||||
fi
|
||||
install_stack "$TO" "$P_PM" "$P_CLAUDE" "${P_CONSOLE[@]}" || after_fail "$FAIL"
|
||||
start_both || after_fail start
|
||||
gate "$TO" "$P_PM" || after_fail "$GATE"
|
||||
trap - INT TERM HUP
|
||||
write_pointer "$TO" "$P_PM" "$P_CLAUDE" "$T_SCHEMA" "${P_CONSOLE[@]}"
|
||||
log_line result ok
|
||||
echo "stack-release: $OP ok: $TO is live for $BUSINESS"
|
||||
}
|
||||
|
||||
do_activate() {
|
||||
TO="$ID"
|
||||
[ "$PLAN" = 1 ] || lock
|
||||
pointer_for_business
|
||||
if [ "$PLAN" = 1 ]; then check_target; plan; exit 0; fi
|
||||
precheck
|
||||
new_activation
|
||||
run_steps migrate
|
||||
}
|
||||
|
||||
do_rollback() {
|
||||
[ "$PLAN" = 1 ] || lock
|
||||
pointer_for_business
|
||||
TO="$(from_log previous)" || refuse no-previous "the log names no earlier copy than $P_COPY"
|
||||
if [ "$PLAN" = 1 ]; then check_target; plan; exit 0; fi
|
||||
precheck
|
||||
[ "$P_SCHEMA" = "$T_SCHEMA" ] || [ -n "$RESTORE" ] || refuse schema-differs "the store is at $P_SCHEMA and $TO expects $T_SCHEMA; pass --restore ACTIVATION"
|
||||
new_activation
|
||||
if [ -n "$RESTORE" ]; then
|
||||
local o="$DATA/backups/stack/$RESTORE" rc=0
|
||||
[ -f "$o/SHA256SUMS" ] && [ "$(cat "$o/DATA_ROOT" 2>/dev/null)" = "$DATA" ] && [ "$(cat "$o/SCHEMA" 2>/dev/null)" = "$T_SCHEMA" ] && [ -f "$o/STAMP" ] \
|
||||
|| refuse restore-mismatch "$o is not a backup of this root at $T_SCHEMA"
|
||||
# Before anything stops. A store the running bus holds is read with readOnly, which adds
|
||||
# no file (the probe); a stopped one falls under the store_unclean rule.
|
||||
if systemctl --user is-active --quiet "$BUS_UNIT"; then
|
||||
COUNT="$(node -e "$STORE_JS" -- heldcount "$STORE" "$(cat "$o/STAMP")")" || rc=1
|
||||
else
|
||||
! store_unclean || refuse store-unclean "$UNCLEAN is present; the bus did not close the store cleanly"
|
||||
COUNT="$(store_read count "$(cat "$o/STAMP")")" || rc=1
|
||||
fi
|
||||
[ "$rc" = 0 ] || refuse restore-mismatch "cannot count the events after $(cat "$o/STAMP")"
|
||||
echo "restore: $o; the store now in place moves to $POST and is kept; $COUNT event(s) were written after the backup"
|
||||
[ "$COUNT" = 0 ] || [ "$EVENTS" = "$COUNT" ] || refuse post-backup-events "$COUNT event(s) were written after the backup; Jason decides, then --events $COUNT"
|
||||
BACKUP="$RESTORE"
|
||||
fi
|
||||
run_steps restore
|
||||
}
|
||||
|
||||
do_status() {
|
||||
echo "pointer: $POINTER"
|
||||
if read_pointer; then sed 's/^/ /' "$POINTER"; else echo " none"; fi
|
||||
echo "log (last five lines):"
|
||||
if [ -f "$LOG" ]; then tail -n 5 "$LOG" | sed 's/^/ /'; else echo " none"; fi
|
||||
local open
|
||||
open="$(interrupted | paste -sd' ' -)"
|
||||
[ -z "$open" ] || echo "interrupted run: $open (started, no final line; Jason or Mos settles it)"
|
||||
echo "check:"
|
||||
if scripts/stack-copy.sh check --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1 | sed 's/^/ /'; then echo " result: pass"; else echo " result: FAIL"; fi
|
||||
[ -n "$P_COPY" ] || return 0
|
||||
echo "bus status:"
|
||||
"$COPIES/$P_COPY/scripts/mosaic" bus status 2>&1 | sed 's/^/ /' || true
|
||||
if read_units; then
|
||||
local d=""
|
||||
[ "$U_PM" = "$P_PM" ] || d+=" pm"
|
||||
[ "$U_CLAUDE" = "$P_CLAUDE" ] || d+=" claude"
|
||||
[ "${P_CONSOLE[*]}" = "${U_CONSOLE[*]}" ] || d+=" console"
|
||||
echo "unit flags: ${d:+differ from the pointer:}${d:-match the pointer}"
|
||||
else
|
||||
echo "unit flags: cannot read the units"
|
||||
fi
|
||||
if git -C "$REPO" merge-base --is-ancestor "$P_COPY" refs/heads/refactor 2>/dev/null; then
|
||||
echo "drift: $(git -C "$REPO" rev-list --count "$P_COPY..refs/heads/refactor") commit(s) on refactor since the pointer's copy"
|
||||
else
|
||||
echo "drift: the pointer's copy is not an ancestor of refactor"
|
||||
fi
|
||||
}
|
||||
|
||||
# --- Dispatch ---
|
||||
|
||||
[ $# -gt 0 ] || usage
|
||||
OP="$1"; shift
|
||||
ID=""
|
||||
case "$OP" in
|
||||
activate) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;;
|
||||
adopt|rollback|status) [ $# -ge 1 ] || usage; BUSINESS="$1"; shift ;;
|
||||
-h|--help) usage 0 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
[[ "$BUSINESS" =~ ^[a-z0-9][a-z0-9-]*$ ]] || usage
|
||||
parse_opts "$@"
|
||||
COPIES_DIR="$COPIES"
|
||||
load_config
|
||||
DATA="$MOSAIC_DATA_ROOT"
|
||||
STATE="$DATA/state"
|
||||
POINTER="$STATE/stack-active.json"
|
||||
LOG="$STATE/stack-activation-log.jsonl"
|
||||
STORE="$DATA/bus/bus.sqlite"
|
||||
BUS_UNIT="mosaic-bus@$BUSINESS.service"
|
||||
FROM="" TO="" ACT="" RS="" POST="" BACKUP="" TAKEN="" RESTORED="" STEPS="" COUNT="" UNCLEAN="" FAIL=""
|
||||
STORE_SCHEMA="" START_EPOCH="" START_MS=0 GATE="" GATE_WHY=""
|
||||
P_COPY="" P_PM=0 P_SCHEMA="" P_CLAUDE="" P_CONSOLE=()
|
||||
case "$OP" in
|
||||
adopt) do_adopt ;;
|
||||
activate) do_activate ;;
|
||||
rollback) do_rollback ;;
|
||||
status) do_status ;;
|
||||
esac
|
||||
Reference in New Issue
Block a user