diff --git a/docs/TOOLS.md b/docs/TOOLS.md index 509fa5509..02e2f992f 100644 --- a/docs/TOOLS.md +++ b/docs/TOOLS.md @@ -342,43 +342,69 @@ reports checkout or copy identity and refactor drift, and writes nothing. ### Pinned stack update +``` +scripts/stack-release.sh adopt BUSINESS +scripts/stack-release.sh activate ID BUSINESS [--pm] [--migrate] [--timeout S] [--plan] +scripts/stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan] +scripts/stack-release.sh status BUSINESS +common options: --copies DIR, --bin DIR +``` + This is an operator plan, not an action agents run. Both reviews and the -landing precede it. Mos grants each live step first. A bus restart ends -running sessions; do this only when no PM turn is in flight. - -1. Make the pushed target copy with `stack-copy.sh copy COMMIT` and note - its full id. Before the first update, preserve the current Q81 CLI wrapper - (bytes and mode), bus and Console unit files and drop-ins, and the effective - WorkingDirectory/ExecStart output from each status command. Record the bus's - --claude pin and the Console's preview/send flags. Keep the bus pm.conf and - the existing Console copy or checkout: the first rollback needs them. - On later updates, also record the prior shared copy's full id. -2. With Mos's go, stop the Console, then stop the bus. -3. Mos removes overriding bus drop-ins identified by status. The first - update removes `~/.config/systemd/user/mosaic-bus@mosaic-stack.service.d/pm.conf`. - Reload systemd after removal; keep unrelated environment drop-ins. -4. With Mos's go, install the bus with `bus-service.sh install --copy ID --pm` - and its existing --claude pin, then the Console with - `webui-service.sh install --copy ID` and its existing preview/send flags. -5. With Mos's go, run `stack-copy.sh wrapper ID` to switch the human CLI. -6. Mos starts the bus first, then the Console. Read `stack-copy.sh check`, - both service status commands, Console `/healthz` and the canonical - `scripts/mosaic launches list`; the latter must show the PM. -7. Jason's plain `mosaic talk pm "..."` must print sent and a reply before - #225 closes. Keep the receipts for the three paths and reply. - -The first rollback restores the pre-update mixed setup, which has no shared -copy id. Each live step needs Mos's go: stop the Console then the bus; -reinstall the checkout bus unit without --copy or --pm, retaining its saved ---claude pin; restore the saved pm.conf and other bus drop-ins; restore the -saved Console unit and drop-ins pointing to its original checkout or -mosaic-console copy; restore the saved Q81 wrapper bytes and mode; reload -systemd; start the bus then the Console. Check their effective paths against -the saved status output, Console /healthz and the PM launch. The shared-copy -check is expected to refuse this restored mixed setup; #225 remains open. - -Later rollback repeats steps 2-6 with the prior shared copy's full id, each -live step again approved by Mos. Nothing prunes copies or removes mosaic-old. +landing come before it. Mos grants each live run first. A bus restart ends +running sessions, so run it only when no PM turn is in flight. `--plan` and +`status` change nothing and need no go. + +The script keeps `/state/stack-active.json` (the live copy, business, +PM flag, claude pin, Console flags, schema and activation) and appends every +run to `/state/stack-activation-log.jsonl`: a `started` line, then +one final line with `result` `ok`, `refused`, `gate-failed` or +`rollback-failed`. A refusal exits 3, a gate failure that rolled back exits 1, +a failed rollback exits 2, usage exits 4. Only one run at a time holds +`state/stack-release.lock` (`run-locked`). + +1. Once per host, `adopt BUSINESS` records the running stack as the pointer: + the copy `stack-copy.sh check` reports, the bus unit's `--pm` and + `MOSAIC_CLAUDE_BIN`, the Console's flags without `--checkout`, and the + copy's schema version. It refuses when the check fails, the bus unit has no + `MOSAIC_CLAUDE_BIN` or a pointer exists. +2. Make the pushed target copy with `stack-copy.sh copy COMMIT` and note its + full id. +3. Run `activate ID BUSINESS --plan` and take the output to Mos. It names the + store's schema and the target's, any migration with `--migrate` and the + decision 96 condition 2 line (Jason present), the commits the run adds, + and the three install commands. It refuses `copies-unrelated` when neither + copy is an ancestor of the other. +4. Mos removes any bus drop-in that overrides ExecStart or WorkingDirectory; + `activate` refuses `drop-in-override` and names the file. +5. With Mos's go, `activate ID BUSINESS`, plus `--migrate` when the plan + says so. Before it stops anything, it refuses a missing, unpushed or + unreadable copy, a busy `mosaic-bus@` instance, and unit flags that differ + from the pointer. Then it saves the unit files, drop-ins and wrapper under + `state/stack-rollback//`, stops the Console then the bus, and + reads the stopped store. A `-wal` or `-shm` beside it refuses + `store-unclean` and leaves the store alone. A migration takes an + owner-only backup under `/backups/stack//` first. + It installs the bus, the Console and the wrapper from the pointer's flags, + starts the bus then the Console, and runs the gate: both units active, a + live bus host for this start, Console `/healthz`, `stack-copy.sh check` + and, with `--pm`, exactly one `launched` journal line for a live run of + this start. A pass writes the pointer. +6. A gate failure puts back the saved units and wrapper, restores the backup + when a migration ran, keeps the store it replaced as + `bus.post-attempt-`, starts the old copy and gates it. If that gate + fails too, both units stay stopped, the log says `rollback-failed`, and + Jason or Mos settles it from the rollback set the message names. +7. `rollback BUSINESS` returns to the copy before the pointer's, taken from + the log. When the schemas differ it needs `--restore ACTIVATION`; events + written after that backup need `--events N` with the count it prints, and + Jason decides first. +8. Jason's plain `mosaic talk pm "..."` must print sent and a reply. Keep the + `status` output with the receipts. + +A `started` line with no final line is an interrupted run. `status` shows +it, and `adopt`, `activate` and `rollback` refuse until Jason or Mos settles +it. Nothing prunes copies, rollback sets, backups or post-attempt stores. After a switch, the human commands and `--tester` work only through `mosaic` on PATH; the checkout's `scripts/mosaic` exits 3 with `human-required` or diff --git a/packages/cli/README.md b/packages/cli/README.md index 7802d72f0..d28333a7c 100644 --- a/packages/cli/README.md +++ b/packages/cli/README.md @@ -645,10 +645,16 @@ script only warns about dependency version schemes. The copy runs the already published bundle and does not build it. No allowScripts change is needed for these paths; adding a runtime loader requires a fresh audit. -The one update and rollback plan is [TOOLS.md, Pinned stack update](../../docs/TOOLS.md#pinned-stack-update). -Each live switch/unit change/stop/start requires Mos's go; agents build -and test fixtures only. Restarting the bus ends its sessions, so no PM -turn may be in flight. Queue writes stay on the canonical scripts/mosaic. +`scripts/stack-release.sh` runs the update and the rollback (row 162, D1): +`adopt` records the running stack once, `activate ID` switches the bus, +Console and wrapper to a copy and gates it, `rollback` returns to the +previous copy from the log, and `status` reads. A failed gate rolls back to +the old copy. The pointer and the log live under `/state/`. The +steps are in [TOOLS.md, Pinned stack update](../../docs/TOOLS.md#pinned-stack-update). +Each live run requires Mos's go; agents build and test fixtures only, and +`--plan` and `status` change nothing. Restarting the bus ends its sessions, +so no PM turn may be in flight. Queue writes stay on the canonical +scripts/mosaic. The human commands and `--tester` run only from the tree the bus runs from (row 130, #1625). The broker checks the path of the calling human-cli.mjs or diff --git a/packages/cli/tests/stack-release.test.mjs b/packages/cli/tests/stack-release.test.mjs new file mode 100644 index 000000000..e5610bb0d --- /dev/null +++ b/packages/cli/tests/stack-release.test.mjs @@ -0,0 +1,732 @@ +// scripts/stack-release.sh (D1, #1659). Design, the T table and the mutants: +// agents/darkwing/work/queue-162/BUILD.md. Every run uses a temp data root, temp units and stand-in +// systemctl, journalctl and systemd-analyze on PATH; nothing here reads ~/.mosaic-dev or a live unit. +import test, {after} from 'node:test'; +import assert from 'node:assert/strict'; +import {execFileSync,spawn,spawnSync} from 'node:child_process'; +import {mkdtempSync,mkdirSync,writeFileSync,readFileSync,appendFileSync,cpSync,rmSync,chmodSync,existsSync,statSync,readdirSync,realpathSync} from 'node:fs'; +import {createServer} from 'node:http'; +import {createHash} from 'node:crypto'; +import {DatabaseSync} from 'node:sqlite'; +import {tmpdir} from 'node:os'; +import {join,dirname,basename} from 'node:path'; +import {fileURLToPath,pathToFileURL} from 'node:url'; +import {once} from 'node:events'; + +const ROOT=fileURLToPath(new URL('../../..',import.meta.url)).replace(/\/$/,''); +const NODE=process.execPath; +const MARKER='# mosaic-cli wrapper (Q81, agents/sage/work/q81/mosaic-cli.sh)'; +const BUS='mosaic-bus@acme.service',CON='mosaic-webui.service'; +const git=(cwd,...args)=>execFileSync('git',['-c','user.name=fixture','-c','user.email=fixture@invalid','-c','commit.gpgsign=false','-c','core.hooksPath=/dev/null','-C',cwd,...args],{encoding:'utf8'}).trim(); +const script=(fn)=>`#!${NODE}\n(${fn})();\n`; + +// --- Stand-ins. Each is serialized into its own file, so it uses only process and its own scope. --- + +// systemctl --user: unit state in $SD_STATE/units.json, every call in calls.log. A bus start runs +// holder.cjs (the store held open in WAL mode, writer.lock, host.json) and, with --pm in the unit's +// ExecStart, writes the `launched` journal line and the launches entry unless a pm- fault +// names another case. Faults are files in $SD_STATE/faults named for the copy they hit. +function systemctl(){ + const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process'),crypto=process.getBuiltinModule('node:crypto'); + const sd=process.env.SD_STATE,data=process.env.SD_DATA,unitDir=path.join(process.env.XDG_CONFIG_HOME,'systemd','user'); + const args=process.argv.slice(2).filter((a)=>a!=='--user'),[verb,...rest]=args; + fs.appendFileSync(path.join(sd,'calls.log'),args.join(' ')+'\n'); + const faultFile=(n)=>path.join(sd,'faults',n),fault=(n)=>fs.existsSync(faultFile(n)); + const stFile=path.join(sd,'units.json'); + const load=()=>{try{return JSON.parse(fs.readFileSync(stFile,'utf8'))}catch{return {}}}; + const save=(s)=>{fs.writeFileSync(stFile+'.tmp',JSON.stringify(s));fs.renameSync(stFile+'.tmp',stFile)}; + const sleep=(ms)=>Atomics.wait(new Int32Array(new SharedArrayBuffer(4)),0,0,ms); + const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}}; + const isBus=(u)=>/^mosaic-bus@[a-z0-9-]+\.service$/.test(u),inst=(u)=>u.slice('mosaic-bus@'.length,-'.service'.length); + const unitFile=(u)=>{ + const main=isBus(u)?'mosaic-bus@.service':u,file=path.join(unitDir,main); + if(!fs.existsSync(file))return null; + const drops=[]; + for(const d of isBus(u)?[main+'.d',u+'.d']:[u+'.d']){const dir=path.join(unitDir,d);if(fs.existsSync(dir))for(const f of fs.readdirSync(dir).sort())if(f.endsWith('.conf'))drops.push(path.join(dir,f))} + const r={wd:'',exec:'',env:[],drops}; + for(const f of [file,...drops])for(let line of fs.readFileSync(f,'utf8').split('\n')){ + line=line.replace(/%i/g,isBus(u)?inst(u):'').replace(/%h/g,process.env.HOME); + const m=/^(WorkingDirectory|ExecStart|Environment)=(.*)$/.exec(line);if(!m)continue; + if(m[1]==='WorkingDirectory')r.wd=m[2];else if(m[1]==='ExecStart'){if(m[2])r.exec=m[2]}else r.env.push(m[2]); + } + return r; + }; + const snap=(ev)=>{ + const dir=path.join(data,'bus'),files={}; + try{for(const n of fs.readdirSync(dir).sort()){const p=path.join(dir,n),st=fs.statSync(p);if(st.isFile())files[n]=[st.size,st.mtimeMs,crypto.createHash('sha256').update(fs.readFileSync(p)).digest('hex')]}}catch{} + fs.appendFileSync(path.join(sd,'snap.jsonl'),JSON.stringify({ev,files})+'\n'); + }; + const pm=(b,id,pid,off)=>{ + let mode='';try{mode=fs.readFileSync(faultFile(`pm-${id}`),'utf8').trim()}catch{} + const now=Date.now()-off*1000,epoch=Math.floor(now/1000),at=new Date(now).toISOString(),run=`run-${pid}`,unit=`mosaic-bus@${b}.service`; + const line=(text)=>fs.appendFileSync(path.join(sd,'journal.jsonl'),JSON.stringify({epoch,unit,text})+'\n'); + const ok=()=>line(`launched pm-${b}: run ${run}, claude opus`); + const lf=path.join(sd,'launches.json');let list=[];try{list=JSON.parse(fs.readFileSync(lf,'utf8'))}catch{} + const entry=(o)=>list.push({run,business:b,live:true,startedAt:at,...o}); + line(`bus host up: business ${b}, socket /fixture/bus.sock, notifier off`); + switch(mode){ + case 'none':break; + case 'two':ok();line(`launched pm-${b}: run ${run}b, claude opus`);entry({});break; + case 'badword':line(`launched pm-${b}: job ${run}, claude opus`);entry({});break; + case 'notlisted':ok();break; + case 'early':ok();entry({startedAt:new Date(now-3600e3).toISOString()});break; + case 'human':entry({run:`human-${pid}`});break; + case 'other':ok();entry({business:'other'});break; + case 'dead':ok();entry({live:false});break; + default:ok();entry({}); + } + fs.writeFileSync(lf,JSON.stringify(list)); + }; + if(verb==='show'){ + const u=rest[0],props=[];for(let i=1;i!a.startsWith('--')).map((p)=>new RegExp('^'+p.replace(/\./g,'\\.').replace(/\*/g,'.*')+'$')); + for(const [u,s] of Object.entries(load()))if(pats.some((re)=>re.test(u)))console.log(`${u} loaded ${s.state} ${s.state==='active'?'running':'dead'} fixture`); + process.exit(0); + } + if(verb==='start'){ + const u=rest[0],f=unitFile(u);if(!f){console.error(`Unit ${u} not found.`);process.exit(5)} + const s=load(),id=path.basename(f.wd); + if(!isBus(u)){s[u]={state:'active',pid:null,wd:f.wd};save(s);process.exit(0)} + if(fault(`start-dies-${id}`)){s[u]={state:'failed',pid:null,wd:f.wd};save(s);process.exit(0)} + snap('start'); + const off=process.env.SD_CLOCK_OFFSET||'0'; + // The holder must not inherit stack-release.sh's lock (fd 9) or any other fd above 2. + const close='for fd in /proc/self/fd/*; do n=${fd##*/}; [ "$n" -gt 2 ] && eval "exec $n>&-" 2>/dev/null; done; exec "$@"'; + const ch=cp.spawn('bash',['-c',close,'holder',process.execPath,path.join(__dirname,'..','holder.cjs'),data,inst(u),sd,off],{detached:true,stdio:'ignore'}); + ch.unref(); + fs.appendFileSync(path.join(sd,'pids.log'),ch.pid+'\n'); + const ready=path.join(sd,`ready-${ch.pid}`),end=Date.now()+10000; + while(!fs.existsSync(ready)){if(Date.now()>end){console.error('fixture holder did not start');process.exit(1)}sleep(20)} + s[u]={state:'active',pid:ch.pid,wd:f.wd};save(s); + if(f.exec.split(' ').slice(1).includes('--pm'))pm(inst(u),id,ch.pid,Number(off)); + process.exit(0); + } + if(verb==='stop'){ + const u=rest[0],s=load(),st=s[u]; + if(st&&st.pid){ + try{process.kill(st.pid,'SIGTERM')}catch{} + const end=Date.now()+10000;while(alive(st.pid)&&Date.now()JSON.parse(l))}catch{} + for(const o of lines)if(o.unit===unit&&o.epoch>=from)console.log(o.text); +} + +function systemdAnalyze(){ + const path=process.getBuiltinModule('node:path'); + if(process.argv[2]!=='--user'||process.argv[3]!=='unit-paths')process.exit(1); + console.log(path.join(process.env.XDG_CONFIG_HOME,'systemd','user')); +} + +// The bus host: holds the store open in WAL mode until SIGTERM, as the broker does. +function holder(){ + const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite'); + const [data,business,sd,offset]=process.argv.slice(2),bus=path.join(data,'bus'); + fs.mkdirSync(bus,{recursive:true,mode:0o700}); + const db=new DatabaseSync(path.join(bus,'bus.sqlite')); + if(!db.prepare("SELECT count(*) AS n FROM sqlite_master WHERE name='meta'").get().n){ + db.exec('PRAGMA journal_mode=WAL'); + db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)'); + db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)'); + db.prepare("INSERT INTO meta VALUES ('schema_version', '3g')").run(); + } + fs.writeFileSync(path.join(bus,'writer.lock'),String(process.pid)); + const at=new Date(Date.now()-Number(offset||0)*1000).toISOString(),host=path.join(sd,'host.json'); + fs.writeFileSync(host+'.tmp',JSON.stringify({host:{business,pid:process.pid,live:true,startedAt:at,notifier:'ok'},boot:{ok:true,business,at}})); + fs.renameSync(host+'.tmp',host); + process.on('SIGTERM',()=>{db.close();fs.rmSync(path.join(bus,'writer.lock'),{force:true});process.exit(0)}); + fs.writeFileSync(path.join(sd,`ready-${process.pid}`),''); + setInterval(()=>{},1<<30); +} + +// A copy's scripts/mosaic: bus status and launches list from what the holder and systemctl wrote. +function mosaicStub(){ + const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'); + const id=path.basename(path.resolve(__dirname,'..')),sd=process.env.SD_STATE,a=process.argv.slice(2).join(' '); + const fault=(n)=>fs.existsSync(path.join(sd,'faults',n)); + const alive=(pid)=>{try{const s=fs.readFileSync(`/proc/${pid}/stat`,'utf8');return s[s.lastIndexOf(')')+2]!=='Z'}catch{return false}}; + if(a==='bus status --json'){ + if(fault(`status-${id}`))process.exit(1); + let s={host:null,boot:null}; + try{s=JSON.parse(fs.readFileSync(path.join(sd,'host.json'),'utf8'));s.host.live=alive(s.host.pid)}catch{} + console.log(JSON.stringify(s)); + }else if(a==='bus status')console.log(`bus: fixture status for ${id}`); + else if(a==='launches list --json'){ + if(fault(`launches-${id}`))process.exit(1); + let l=[];try{l=JSON.parse(fs.readFileSync(path.join(sd,'launches.json'),'utf8'))}catch{} + console.log(JSON.stringify(l)); + }else{console.error(`fixture mosaic: ${a}`);process.exit(4)} +} + +// A copy's migrate-schema.mjs: records the backups it saw, then sets 3h, or 3h-partial and exit 1. +function migrateStub(){ + const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),{DatabaseSync}=process.getBuiltinModule('node:sqlite'); + const data=process.argv[2],sd=process.env.SD_STATE,dir=path.join(data,'backups','stack'); + const backups=fs.existsSync(dir)?fs.readdirSync(dir).sort().map((name)=>({name,sums:fs.existsSync(path.join(dir,name,'SHA256SUMS'))})):[]; + fs.writeFileSync(path.join(sd,'migrate.json'),JSON.stringify({backups})); + const fail=fs.existsSync(path.join(sd,'faults','migrate-fails')); + const db=new DatabaseSync(path.join(data,'bus','bus.sqlite')); + db.prepare("UPDATE meta SET value=? WHERE key='schema_version'").run(fail?'3h-partial':'3h');db.close(); + if(fail)process.exit(1); + console.log('fixture migrate: 3h'); +} + +// --- The module fixture: one repo, its copies and the stand-ins, shared by every test. --- + +const base=realpathSync(mkdtempSync(join(tmpdir(),'stack-release-'))); +after(()=>{execFileSync('chmod',['-R','u+w',base]);rmSync(base,{recursive:true,force:true})}); +const repo=join(base,'repo'),COPIES=join(base,'copies'),SBIN=join(base,'sbin'); +mkdirSync(repo);mkdirSync(COPIES);mkdirSync(SBIN); +writeFileSync(join(base,'package.json'),'{"type":"commonjs"}\n'); +const tracked=execFileSync('git',['-C',ROOT,'ls-files','-co','--exclude-standard','--','scripts','packages','adapters','roles','RELEASE','package.json','package-lock.json',':!packages/*/tests',':!scripts/foundation'],{encoding:'utf8'}).trim().split('\n'); +for(const f of tracked)if(existsSync(join(ROOT,f)))cpSync(join(ROOT,f),join(repo,f)); +const pkg={name:'copy-fixture',version:'1.0.0'}; +writeFileSync(join(repo,'package.json'),JSON.stringify(pkg)); +writeFileSync(join(repo,'package-lock.json'),JSON.stringify({name:pkg.name,version:pkg.version,lockfileVersion:3,packages:{'':pkg}})); +git(repo,'init','-q','-b','refactor');git(repo,'config','gc.auto','0');git(repo,'add','.');git(repo,'commit','-qm','fixture'); +const commit=(msg)=>{git(repo,'commit','-q','--allow-empty','-m',msg);return git(repo,'rev-parse','HEAD')}; +// refactor: F A B C1 C, then M merging side branch S (from A). U sits on a branch nothing reaches. +const F=git(repo,'rev-parse','HEAD'),A=commit('row 901: A'),B=commit('row 902: B'),C1=commit('row 903: C1'),C=commit('row 904: C'); +git(repo,'checkout','-q','-b','side',A);const S=commit('row 905: S'); +git(repo,'checkout','-q','-b','unreached',A);const U=commit('row 906: U'); +git(repo,'checkout','-q','refactor');git(repo,'merge','-q','--no-ff','-m','merge side','side'); +const M=git(repo,'rev-parse','HEAD');git(repo,'update-ref','refs/remotes/origin/refactor',M); +// `to` false leaves out the `to:` lines (C1); a string puts that value in each one (A). The --plan +// steps line can't be read from either (T27). +const storeText=(v,froms,to=true)=>`export const SCHEMA_VERSION = '${v}';\nexport const MIGRATIONS = [\n${froms.map((f,i)=>` {\n from: '${f}',\n${to===false?'':` to: '${to===true?froms[i+1]??v:to}',\n`} run() {},\n },\n`).join('')}];\n`; +for(const [id,store] of [[F,'// a copy from before SCHEMA_VERSION\n'],[A,storeText('3g',['3f'],'3x')],[B,storeText('3g',['3f'])],[C1,storeText('3g',['3f'],false)],[C,storeText('3h',['3f','3g'])],[S,storeText('3g',['3f'])],[U,storeText('3g',['3f'])]]){ + const d=join(COPIES,id); + for(const [f,text,mode] of [['scripts/mosaic',script(mosaicStub),0o755],['packages/webui/src/cli.mjs','// fixture Console, never run\n',0o644],['packages/bus/src/store.mjs',store,0o644],['packages/bus/src/migrate-schema.mjs',`(${migrateStub})();\n`,0o644]]){ + mkdirSync(dirname(join(d,f)),{recursive:true});writeFileSync(join(d,f),text,{mode}); + } + execFileSync('chmod',['-R','a-w',d]); +} +for(const [name,fn] of [['systemctl',systemctl],['journalctl',journalctl],['systemd-analyze',systemdAnalyze]])writeFileSync(join(SBIN,name),script(fn),{mode:0o755}); +writeFileSync(join(base,'holder.cjs'),`(${holder})();\n`); +const SR=join(repo,'scripts','stack-release.sh'); + +// --- Per-test roots and helpers --- + +const sha=(f)=>createHash('sha256').update(readFileSync(f)).digest('hex'); +const lines=(f)=>existsSync(f)?readFileSync(f,'utf8').split('\n').filter(Boolean):[]; +const calls=(e)=>lines(join(e.sd,'calls.log')); +const startsStops=(e,from)=>calls(e).slice(from).filter((c)=>/^(start|stop) /.test(c)); +const logLines=(e)=>lines(e.log).map((l)=>JSON.parse(l)); +const pointer=(e)=>existsSync(e.pointer)?JSON.parse(readFileSync(e.pointer,'utf8')):null; +const units=(e)=>JSON.parse(readFileSync(join(e.sd,'units.json'),'utf8')); +const snaps=(e)=>lines(join(e.sd,'snap.jsonl')).map((l)=>JSON.parse(l)); +const fault=(e,name,text='')=>writeFileSync(join(e.sd,'faults',name),text); +const sysctl=(e,...a)=>execFileSync(join(SBIN,'systemctl'),['--user',...a],{env:e.env,encoding:'utf8'}); +const edit=(f,from,to)=>{const s=readFileSync(f,'utf8');assert.ok(s.includes(from),`${f} has no ${from}`);writeFileSync(f,s.replace(from,to))}; +const posts=(e)=>readdirSync(e.data).filter((n)=>n.startsWith('bus.post-attempt-')); +const backupSum=(e,act)=>readFileSync(join(e.data,'backups','stack',act,'SHA256SUMS'),'utf8').split(' ')[0]; +const commitLines=(range)=>git(repo,'log','--oneline','--no-merges','--max-count=200',range).split('\n').map((l)=>` ${l}`); +// The units, the wrapper, the pointer and the store, by sha256. +const fp=(e)=>({bus:sha(join(e.units,'mosaic-bus@.service')),console:sha(join(e.units,CON)),wrapper:sha(join(e.bin,'mosaic')),pointer:existsSync(e.pointer)?sha(e.pointer):null,store:sha(e.store)}); +// The schema version: readOnly while the holder has the store open, immutable when it is stopped. +function storeVersion(file,held){ + const db=held?new DatabaseSync(file,{readOnly:true}):new DatabaseSync(pathToFileURL(file).href+'?immutable=1',{readOnly:true}); + try{return db.prepare("SELECT value FROM meta WHERE key='schema_version'").get().value}finally{db.close()} +} +// Every file and directory under p with size and mtime: T23 and T16 compare it before and after. +function tree(p){ + const out=[]; + const walk=(q)=>{let st;try{st=statSync(q)}catch{return}out.push(st.isDirectory()?`${q}/`:`${q} ${st.size} ${st.mtimeMs}`);if(st.isDirectory())for(const n of readdirSync(q).sort())walk(join(q,n))}; + walk(p);return out; +} +// T17: what a run may never remove. +const kept=(e)=>[...tree(join(e.data,'backups','stack')),...tree(join(e.state,'stack-rollback')),...posts(e).flatMap((n)=>tree(join(e.data,n))),...tree(COPIES)].map((l)=>l.replace(/ .*/,'')); + +async function mk(t){ + const root=realpathSync(mkdtempSync(join(tmpdir(),'sr-'))); + const e={root,data:join(root,'data'),bin:join(root,'bin'),sd:join(root,'sd'),units:join(root,'xcfg','systemd','user'),cfg:join(root,'cfg','config.json'),claude:join(root,'claude-bin','claude')}; + Object.assign(e,{state:join(e.data,'state'),pointer:join(e.data,'state','stack-active.json'),log:join(e.data,'state','stack-activation-log.jsonl'),store:join(e.data,'bus','bus.sqlite'),lock:join(e.data,'state','stack-release.lock')}); + for(const d of [join(root,'home'),dirname(e.cfg),join(e.data,'bus'),e.bin,dirname(e.claude),join(e.sd,'faults'),e.units,join(root,'xdata')])mkdirSync(d,{recursive:true}); + chmodSync(join(e.data,'bus'),0o700); + writeFileSync(e.cfg,JSON.stringify({configVersion:1,environment:'development',dataRoot:e.data,execution:{backend:'docker',provider:'zai',model:'glm-5.3-flash',adapter:'mock'}})); + writeFileSync(join(e.bin,'mosaic'),`#!/usr/bin/env bash\n${MARKER}\n[ "\${1:-}" = __lease-capability ] && exec "${e.bin}/mosaic-old" "$@"\nexec "/old/scripts/mosaic" "$@"\n`,{mode:0o755}); + writeFileSync(join(e.bin,'mosaic-old'),'#!/bin/sh\nexit 0\n',{mode:0o755}); + writeFileSync(e.claude,'#!/bin/sh\necho "9.9.9 (Claude Code fixture)"\n',{mode:0o755}); + e.env={PATH:`${SBIN}:${dirname(NODE)}:/usr/bin:/bin`,HOME:join(root,'home'),XDG_CONFIG_HOME:join(root,'xcfg'),XDG_DATA_HOME:join(root,'xdata'),MOSAIC_CONFIG:e.cfg,SD_STATE:e.sd,SD_DATA:e.data,TMPDIR:tmpdir()}; + // The Console's /healthz: 200 while the stand-in has it active and no healthz- fault names its copy. + const server=createServer((q,r)=>{ + let st={};try{st=units(e)[CON]||{}}catch{} + const ok=q.url==='/healthz'&&st.state==='active'&&!existsSync(join(e.sd,'faults',`healthz-${basename(st.wd||'')}`)); + r.writeHead(ok?200:500);r.end(); + }); + server.listen(0,'127.0.0.1');await once(server,'listening');e.port=String(server.address().port); + t.after(()=>{ + for(const pid of lines(join(e.sd,'pids.log')).map(Number)){ + try{if(readFileSync(`/proc/${pid}/cmdline`,'utf8').includes(join(base,'holder.cjs')))process.kill(pid,'SIGKILL')}catch{} + } + server.close();execFileSync('chmod',['-R','u+w',root]);rmSync(root,{recursive:true,force:true}); + }); + return e; +} + +async function run(e,...args){ + const before=existsSync(e.data)?kept(e):[]; + const ch=spawn('bash',[SR,...args,'--copies',COPIES,'--bin',e.bin],{env:e.env,stdio:['ignore','pipe','pipe']}); + let out='',err='';ch.stdout.on('data',(d)=>{out+=d});ch.stderr.on('data',(d)=>{err+=d}); + const timer=setTimeout(()=>ch.kill('SIGKILL'),120000); + const [code]=await once(ch,'close');clearTimeout(timer); + const now=new Set(kept(e)); + for(const p of before)assert.ok(now.has(p),`T17: ${args.join(' ')} removed ${p}`); + return {code,out,err}; +} + +// The stack on copy `copy` for business acme, both units active. pointer: adopt (a real adopt and +// its log line), hand (written here, no log) or none. Snapshots start after the seed. +async function seed(e,{copy=B,pm=false,store='3g',pointer:how='adopt'}={}){ + const db=new DatabaseSync(e.store); + db.exec('PRAGMA journal_mode=WAL'); + db.exec('CREATE TABLE meta (key TEXT PRIMARY KEY, value TEXT NOT NULL)'); + db.exec('CREATE TABLE events (id INTEGER PRIMARY KEY, at TEXT NOT NULL)'); + db.prepare("INSERT INTO meta VALUES ('schema_version', ?)").run(store); + for(let i=0;i<3;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run(`2000-01-01T00:00:0${i}.000Z`); + db.close(); + const sh=(name,...a)=>execFileSync('bash',[join(repo,'scripts',name),...a],{env:e.env,encoding:'utf8',stdio:['ignore','pipe','pipe']}); + sh('bus-service.sh','install','--copy',copy,'--copies',COPIES,...(pm?['--pm']:[]),'--claude',e.claude); + sh('webui-service.sh','install','--copy',copy,'--copies',COPIES,'--port',e.port,'--business','acme'); + sh('stack-copy.sh','wrapper',copy,'--copies',COPIES,'--bin',e.bin); + // The seeded host started two minutes ago, so its boot status and PM run predate any run's start. + execFileSync(join(SBIN,'systemctl'),['--user','start',BUS],{env:{...e.env,SD_CLOCK_OFFSET:'120'}}); + sysctl(e,'start',CON); + if(how==='adopt'){const r=await run(e,'adopt','acme');assert.equal(r.code,0,r.err)} + else if(how==='hand'){ + mkdirSync(e.state,{recursive:true}); + writeFileSync(e.pointer,JSON.stringify({copy,business:'acme',pm,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g',activation:'manual',at:'2026-10-18T00:00:00Z'})+'\n'); + } + writeFileSync(join(e.sd,'snap.jsonl'),''); +} + +const rolledBack=(code,to=B)=>`stack-release: ${code}; rolled back to ${to}, which passed the gate\n`; +const planTail=(e,to,pm=false)=>[`bus: scripts/bus-service.sh install --copy ${to} --copies ${COPIES}${pm?' --pm':''} --claude ${e.claude}`,`Console: scripts/webui-service.sh install --copy ${to} --copies ${COPIES} --port ${e.port} --business acme`,`CLI: scripts/stack-copy.sh wrapper ${to} --copies ${COPIES} --bin ${e.bin}`]; + +// --- The T table --- + +test('T1 adopt writes the pointer from the units once; a failing check refuses check-failed',async(t)=>{ + const e=await mk(t);await seed(e,{pointer:'none'}); + const r=await run(e,'adopt','acme');assert.equal(r.code,0,r.err); + assert.equal(r.out,`adopted ${B} for acme (pm 0, schema 3g)\n`); + const {at,activation,...p}=pointer(e); + assert.deepEqual(p,{copy:B,business:'acme',pm:false,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g'}); + assert.match(activation,new RegExp(`^\\d{8}T\\d{6}Z-${B.slice(0,12)}$`));assert.match(at,/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ$/); + assert.deepEqual(logLines(e).map((l)=>[l.op,l.from,l.to,l.result]),[['adopt',null,B,'ok']]); + const again=await run(e,'adopt','acme');assert.equal(again.code,3); + assert.equal(again.err,`stack-release: refusing already-adopted: ${e.pointer} exists (copy ${B})\n`); + assert.equal(logLines(e).at(-1).code,'already-adopted'); + const f=await mk(t);await seed(f,{pointer:'none'});fault(f,`check-${B}`); + const bad=await run(f,'adopt','acme');assert.equal(bad.code,3); + assert.match(bad.err,/^stack-release: refusing check-failed: stack-copy.sh check did not pass:\nbus: /); + assert.equal(pointer(f),null); +}); + +test('T2 activate installs exactly the pointer\'s pm, claude and Console flags',async(t)=>{ + const e=await mk(t);await seed(e); + const r=await run(e,'activate',C1,'acme');assert.equal(r.code,0,r.err+r.out); + assert.ok(r.out.endsWith(`stack-release: activate ok: ${C1} is live for acme\n`),r.out); + const bus=readFileSync(join(e.units,'mosaic-bus@.service'),'utf8').split('\n'),con=readFileSync(join(e.units,CON),'utf8').split('\n'); + assert.ok(bus.includes(`ExecStart=${COPIES}/${C1}/scripts/mosaic bus start %i`),bus.join('\n')); + assert.ok(bus.includes(`Environment=MOSAIC_CLAUDE_BIN=${e.claude}`)); + const exec=con.find((l)=>l.startsWith('ExecStart=')); + assert.ok(exec.endsWith(` ${COPIES}/${C1}/packages/webui/src/cli.mjs serve --checkout ${repo} --port ${e.port} --business acme`),exec); + assert.equal(readFileSync(join(e.bin,'mosaic'),'utf8').trimEnd().split('\n').at(-1),`exec "${COPIES}/${C1}/scripts/mosaic" "$@"`); + const {at,activation,...p}=pointer(e); + assert.deepEqual(p,{copy:C1,business:'acme',pm:false,claude:e.claude,console:['--port',e.port,'--business','acme'],schema:'3g'}); +}); + +test('T3 a hand-edited unit refuses unit-flags-drift, names the difference and changes nothing',async(t)=>{ + const cases=[ + ['pm',(e)=>edit(join(e.units,'mosaic-bus@.service'),' bus start %i\n',' bus start %i --pm\n'),()=>' pm: pointer 0, unit 1'], + ['claude',(e)=>{const d=join(e.units,`${BUS}.d`);mkdirSync(d);writeFileSync(join(d,'claude.conf'),'[Service]\nEnvironment=MOSAIC_CLAUDE_BIN=/opt/other/claude\n')},(e)=>` claude: pointer ${e.claude}, unit /opt/other/claude`], + ['console',(e)=>edit(join(e.units,CON),'--business acme\n','--business acme --board http://127.0.0.1:9\n'),(e)=>` console: pointer --port ${e.port} --business acme, unit --port ${e.port} --business acme --board http://127.0.0.1:9`], + ]; + for(const [name,change,line] of cases)await t.test(name,async(t)=>{ + const e=await mk(t);await seed(e);change(e); + const before=fp(e),n=calls(e).length,logs=logLines(e).length; + const r=await run(e,'activate',C1,'acme');assert.equal(r.code,3); + assert.equal(r.err,`stack-release: refusing unit-flags-drift: the units differ from the pointer; settle it first:\n${line(e)}\n`); + assert.deepEqual(fp(e),before);assert.deepEqual(startsStops(e,n),[]); + assert.equal(logLines(e).length,logs+1);assert.equal(logLines(e).at(-1).code,'unit-flags-drift'); + }); +}); + +test('T4 each item 3 refusal changes nothing and logs one line with its code',async(t)=>{ + const setUnit=(e,u,state)=>{const s=units(e);s[u]={...(s[u]||{pid:null,wd:''}),state};writeFileSync(join(e.sd,'units.json'),JSON.stringify(s))}; + const cases=[ + ['copy-missing',()=>M,()=>`no copy ${M} in ${COPIES}`], + ['copy-unreachable',()=>U,()=>`${U} is not reachable from refactor and origin/refactor`], + ['copy-schema-unknown',()=>F,()=>`cannot read SCHEMA_VERSION and its chain from ${COPIES}/${F}/packages/bus/src/store.mjs`], + ['already-active',()=>B,()=>`${B} is the pointer's copy`], + ['instance-busy',(e)=>{setUnit(e,'mosaic-bus@other.service','active');return C1},()=>'mosaic-bus@other.service is active; the copy and wrapper switch reach it too','another business active'], + ['instance-busy',(e)=>{setUnit(e,BUS,'activating');return C1},()=>`${BUS} is activating`,'this business activating'], + ['drop-in-override',(e)=>{const d=join(e.units,`${BUS}.d`);mkdirSync(d);writeFileSync(join(d,'override.conf'),'[Service]\nWorkingDirectory=/elsewhere\n');return C1},(e)=>`${join(e.units,`${BUS}.d`,'override.conf')} overrides ExecStart or WorkingDirectory; the operator removes it as in TOOLS.md step 4`], + ['no-pointer',()=>C1,(e)=>`no ${e.pointer}; run adopt first`,'',{pointer:'none'}], + ]; + for(const [code,setup,msg,label,opts] of cases)await t.test(label?`${code} (${label})`:code,async(t)=>{ + const e=await mk(t);await seed(e,opts);const id=setup(e); + const before=fp(e),n=calls(e).length,logs=logLines(e).length; + const r=await run(e,'activate',id,'acme');assert.equal(r.code,3,r.err); + assert.equal(r.err,`stack-release: refusing ${code}: ${msg(e)}\n`); + assert.deepEqual(fp(e),before);assert.deepEqual(startsStops(e,n),[]); + assert.equal(logLines(e).length,logs+1);assert.deepEqual([logLines(e).at(-1).result,logLines(e).at(-1).code],['refused',code]); + }); +}); + +test('T5 an activate with no migration stops the Console then the bus, installs, starts the bus then the Console',async(t)=>{ + const e=await mk(t);await seed(e);const n=calls(e).length; + const r=await run(e,'activate',C1,'acme');assert.equal(r.code,0,r.err); + assert.deepEqual(startsStops(e,n),[`stop ${CON}`,`stop ${BUS}`,`start ${BUS}`,`start ${CON}`]); + const c=calls(e).slice(n),reload=c.indexOf('daemon-reload'); + assert.ok(reload>c.indexOf(`stop ${BUS}`)&&reload[x.op,x.from,x.to,x.result]),[['activate',B,C1,'started'],['activate',B,C1,'ok']]); +}); + +test('T6 a migration without --migrate rolls back with migration-needs-flag',async(t)=>{ + const e=await mk(t);await seed(e);const before=fp(e); + const r=await run(e,'activate',C,'acme','--timeout','3');assert.equal(r.code,1,r.out); + assert.equal(r.err,rolledBack('migration-needs-flag')); + assert.deepEqual(fp(e),before); + assert.deepEqual([units(e)[BUS].state,units(e)[BUS].wd,units(e)[CON].state],['active',join(COPIES,B),'active']); + const l=logLines(e).at(-1);assert.deepEqual([l.result,l.code,l.rollback],['gate-failed','migration-needs-flag','ok']); + assert.equal(storeVersion(e.store,true),'3g');assert.ok(!existsSync(join(e.data,'backups','stack'))); +}); + +test('T7 --migrate takes the backup before the migration runs',async(t)=>{ + const e=await mk(t);await seed(e);const before=fp(e); + const r=await run(e,'activate',C,'acme','--migrate');assert.equal(r.code,0,r.err+r.out); + const act=pointer(e).activation; + assert.deepEqual(JSON.parse(readFileSync(join(e.sd,'migrate.json'),'utf8')),{backups:[{name:act,sums:true}]}); + assert.equal(backupSum(e,act),before.store); + assert.deepEqual([pointer(e).copy,pointer(e).schema],[C,'3h']);assert.equal(storeVersion(e.store,true),'3h'); + assert.equal(logLines(e).at(-1).backup,act); +}); + +test('T8 a failed migration restores the backup and keeps the store it replaced',async(t)=>{ + const e=await mk(t);await seed(e);const before=fp(e);fault(e,'migrate-fails'); + const r=await run(e,'activate',C,'acme','--migrate','--timeout','3');assert.equal(r.code,1,r.out); + assert.equal(r.err,rolledBack('migrate-failed')); + assert.equal(posts(e).length,1);assert.equal(storeVersion(join(e.data,posts(e)[0],'bus.sqlite'),false),'3h-partial'); + const act=logLines(e).at(-1).activation; + assert.equal(sha(e.store),backupSum(e,act));assert.equal(sha(e.store),before.store); + assert.deepEqual(fp(e),before);assert.equal(units(e)[BUS].wd,join(COPIES,B)); +}); + +test('T9 a store outside the target\'s chain rolls back with store-newer',async(t)=>{ + const e=await mk(t);await seed(e,{store:'3z'});const before=fp(e); + const r=await run(e,'activate',C,'acme','--migrate','--timeout','3');assert.equal(r.code,1,r.out); + assert.equal(r.err,rolledBack('store-newer')); + assert.deepEqual(fp(e),before);assert.equal(storeVersion(e.store,true),'3z');assert.ok(!existsSync(join(e.data,'backups','stack'))); +}); + +test('T10 each gate check failing alone rolls back and puts every file back',async(t)=>{ + const cases=[['units-inactive',`start-dies-${C1}`],['bus-status',`status-${C1}`],['healthz',`healthz-${C1}`],['copy-check',`check-${C1}`],['pm-launch-unconfirmed',`pm-${C1}`,'none']]; + for(const [code,name,text] of cases)await t.test(code,async(t)=>{ + const pm=code==='pm-launch-unconfirmed',e=await mk(t);await seed(e,{pm});fault(e,name,text); + const before=fp(e); + const r=await run(e,'activate',C1,'acme',...(pm?['--pm']:[]),'--timeout','3');assert.equal(r.code,1,r.out+r.err); + assert.equal(r.err,rolledBack(code));assert.ok(r.out.includes(`stack-release: gate failed on ${C1}: ${code}`),r.out); + assert.deepEqual(fp(e),before); + assert.deepEqual([units(e)[BUS].state,units(e)[BUS].wd],['active',join(COPIES,B)]); + const l=logLines(e).at(-1);assert.deepEqual([l.result,l.code,l.rollback],['gate-failed',code,'ok']); + }); +}); + +test('T10b the PM check needs exactly one launched line for a live run of this start',async(t)=>{ + const cases=[ + ['no launched line','none',/\(no launched line since the start\)/], + ['two lines','two',/\(2 launched lines since the start\)/], + ['a changed word','badword',/\(the launched line does not parse: launched pm-acme: job run-\d+, claude opus\)/], + ['run not in launches list','notlisted',/\(run run-\d+ is not in launches list\)/], + ['run started before the start','early',/\(run run-\d+ started before this start\)/], + ['a human launch with no journal line','human',/\(no launched line since the start\)/], + ['run for another business','other',/\(run run-\d+ is for other\)/], + ['run not live','dead',/\(run run-\d+ is not live\)/], + ]; + for(const [name,mode,why] of cases)await t.test(name,async(t)=>{ + const e=await mk(t);await seed(e,{pm:true});fault(e,`pm-${C1}`,mode); + const r=await run(e,'activate',C1,'acme','--pm','--timeout','3');assert.equal(r.code,1,r.out+r.err); + assert.equal(r.err,rolledBack('pm-launch-unconfirmed')); + const gate=r.out.split('\n').find((l)=>l.startsWith(`stack-release: gate failed on ${C1}: pm-launch-unconfirmed`)); + assert.ok(gate,r.out);assert.match(gate,why); + }); + await t.test('one well-formed line and a live run pass',async(t)=>{ + const e=await mk(t);await seed(e,{pm:true}); + const r=await run(e,'activate',C1,'acme','--pm');assert.equal(r.code,0,r.out+r.err); + assert.ok(readFileSync(join(e.units,'mosaic-bus@.service'),'utf8').includes(`ExecStart=${COPIES}/${C1}/scripts/mosaic bus start %i --pm\n`)); + assert.match(lines(join(e.sd,'journal-since.log')).at(-1),/^@\d+$/); + assert.equal(pointer(e).pm,true); + }); +}); + +test('T11 a rollback whose gate fails too: rollback-failed, exit 2, both units stopped, no retry',async(t)=>{ + const e=await mk(t);await seed(e);fault(e,`healthz-${C1}`);fault(e,`healthz-${B}`);const n=calls(e).length; + const r=await run(e,'activate',C1,'acme','--timeout','3');assert.equal(r.code,2,r.out+r.err); + const l=logLines(e).at(-1),rs=join(e.state,'stack-rollback',l.activation); + assert.equal(r.err,`stack-release: rollback-failed: the run failed with healthz and the rollback with healthz; both units are stopped. Rollback set: ${rs}\n`); + assert.deepEqual([l.result,l.code,l.rollback,l.rollbackCode],['rollback-failed','healthz','failed','healthz']); + assert.deepEqual([units(e)[BUS].state,units(e)[CON].state],['inactive','inactive']); + const ss=startsStops(e,n); + assert.deepEqual(ss.filter((c)=>c.startsWith('start ')),[`start ${BUS}`,`start ${CON}`,`start ${BUS}`,`start ${CON}`]); + assert.deepEqual(ss.slice(-2),[`stop ${CON}`,`stop ${BUS}`]); + assert.equal(pointer(e).copy,B); +}); + +test('T12 rollback takes the previous copy from the log',async(t)=>{ + const e=await mk(t);await seed(e); + assert.equal((await run(e,'activate',C1,'acme')).code,0); + const r=await run(e,'rollback','acme');assert.equal(r.code,0,r.out+r.err); + assert.ok(r.out.endsWith(`stack-release: rollback ok: ${B} is live for acme\n`),r.out); + assert.equal(pointer(e).copy,B); + const l=logLines(e).at(-1);assert.deepEqual([l.op,l.from,l.to,l.result],['rollback',C1,B,'ok']); +}); + +test('T13 a rollback across a migration needs --restore, then restores and goes live',async(t)=>{ + const e=await mk(t);await seed(e); + assert.equal((await run(e,'activate',C,'acme','--migrate')).code,0);const act=pointer(e).activation,storeC=sha(e.store); + const no=await run(e,'rollback','acme');assert.equal(no.code,3); + assert.equal(no.err,`stack-release: refusing schema-differs: the store is at 3h and ${B} expects 3g; pass --restore ACTIVATION\n`); + const r=await run(e,'rollback','acme','--restore',act);assert.equal(r.code,0,r.out+r.err); + const o=join(e.data,'backups','stack',act); + assert.match(r.out,new RegExp(`^restore: ${o}; the store now in place moves to ${e.data}/bus\\.post-attempt-\\d{8}T\\d{6}Z and is kept; 0 event\\(s\\) were written after the backup\\n`)); + assert.deepEqual([pointer(e).copy,pointer(e).schema],[B,'3g']);assert.equal(storeVersion(e.store,true),'3g'); + assert.equal(sha(e.store),backupSum(e,act)); + assert.equal(posts(e).length,1);assert.equal(sha(join(e.data,posts(e)[0],'bus.sqlite')),storeC); + assert.equal(logLines(e).at(-1).backup,act); +}); + +test('T14 events written after the backup need --events N with the right N',async(t)=>{ + const e=await mk(t);await seed(e); + assert.equal((await run(e,'activate',C,'acme','--migrate')).code,0);const act=pointer(e).activation; + const db=new DatabaseSync(e.store);for(let i=0;i<2;i++)db.prepare('INSERT INTO events (at) VALUES (?)').run('2999-01-01T00:00:00.000Z');db.close(); + const want='stack-release: refusing post-backup-events: 2 event(s) were written after the backup; Jason decides, then --events 2\n'; + for(const extra of [[],['--events','1']]){ + const before=fp(e),r=await run(e,'rollback','acme','--restore',act,...extra); + assert.equal(r.code,3,extra.join(' '));assert.equal(r.err,want);assert.deepEqual(fp(e),before); + } + const r=await run(e,'rollback','acme','--restore',act,'--events','2');assert.equal(r.code,0,r.out+r.err); + assert.match(r.out,/; 2 event\(s\) were written after the backup\n/); + assert.deepEqual([pointer(e).copy,pointer(e).schema],[B,'3g']); +}); + +test('T14b a rollback --restore whose gate fails stops both units and skips item 6',async(t)=>{ + const e=await mk(t);await seed(e); + assert.equal((await run(e,'activate',C,'acme','--migrate')).code,0);const act=pointer(e).activation,storeC=sha(e.store),before=fp(e); + fault(e,`healthz-${B}`);const n=calls(e).length; + const r=await run(e,'rollback','acme','--restore',act,'--timeout','3');assert.equal(r.code,2,r.out+r.err); + const l=logLines(e).at(-1),rs=join(e.state,'stack-rollback',l.activation),post=join(e.data,posts(e)[0]); + assert.equal(r.err,`stack-release: rollback-failed: the run failed with healthz and the rollback with not-attempted; both units are stopped. Rollback set: ${rs}, backup ${join(e.data,'backups','stack',act)}, post-attempt store ${post}\n`); + assert.deepEqual([l.result,l.code,l.rollbackCode],['rollback-failed','healthz','not-attempted']); + assert.deepEqual([units(e)[BUS].state,units(e)[CON].state],['inactive','inactive']); + assert.equal(posts(e).length,1);assert.equal(sha(join(post,'bus.sqlite')),storeC);assert.equal(sha(e.store),backupSum(e,act)); + assert.deepEqual(startsStops(e,n).filter((c)=>c.startsWith('start ')),[`start ${BUS}`,`start ${CON}`]); + assert.equal(pointer(e).copy,C);assert.equal(sha(e.pointer),before.pointer); +}); + +test('T14c a rollback without --restore whose gate fails runs item 6',async(t)=>{ + const e=await mk(t);await seed(e); + assert.equal((await run(e,'activate',C1,'acme')).code,0);const before=fp(e); + fault(e,`healthz-${B}`); + const r=await run(e,'rollback','acme','--timeout','3');assert.equal(r.code,1,r.out+r.err); + assert.equal(r.err,rolledBack('healthz',C1)); + const l=logLines(e).at(-1);assert.deepEqual([l.op,l.result,l.code,l.rollback],['rollback','gate-failed','healthz','ok']); + assert.deepEqual(fp(e),before);assert.deepEqual([units(e)[BUS].state,units(e)[BUS].wd],['active',join(COPIES,C1)]); + assert.deepEqual(readdirSync(join(e.data,'bus')).filter((n)=>n.startsWith('bus.sqlite')).sort(),['bus.sqlite','bus.sqlite-shm','bus.sqlite-wal']); +}); + +test('T15 --plan names the migration and the restore and changes nothing',async(t)=>{ + const e=await mk(t);await seed(e); + const quiet=async(...args)=>{ + const before=fp(e),n=calls(e).length,logs=logLines(e).length,r=await run(e,...args); + assert.equal(r.code,0,r.err);assert.equal(r.err,''); + assert.deepEqual(fp(e),before);assert.deepEqual(startsStops(e,n),[]);assert.equal(logLines(e).length,logs); + return r.out; + }; + assert.ok((await quiet('activate',C,'acme','--plan')).includes('needs: --migrate (not given; the run would roll back with migration-needs-flag)\n')); + assert.equal((await run(e,'activate',C,'acme','--migrate')).code,0);const act=pointer(e).activation; + assert.equal(await quiet('rollback','acme','--plan'),[ + 'plan: rollback for acme',`from: ${C}`,`to: ${B}`, + 'store schema: 3h (read from the store the running bus holds)','target schema: 3g', + `restore: the store must go back to 3g; needs --restore ${act}`, + 'commits: the run takes away 2 (newest first, at most 200):',...commitLines(`${B}..${C}`), + ...planTail(e,B),''].join('\n')); +}); + +test('T26 --plan lists the commits an activate adds, after the migration lines',async(t)=>{ + const e=await mk(t);await seed(e); + const head=(mig)=>['plan: activate for acme',`from: ${B}`,`to: ${C}`,'store schema: 3g (read from the store the running bus holds)','target schema: 3h', + 'migration: 3g to 3h, after a backup of the stopped store','migration steps: 3g to 3h',`needs: --migrate (${mig})`,'needs: Jason present for this run (decision 96 condition 2)']; + const adds=commitLines(`${B}..${C}`); + assert.equal(adds.length,2);assert.match(adds[0],/^ {2}[0-9a-f]{7,} row 904: C$/);assert.match(adds[1],/^ {2}[0-9a-f]{7,} row 903: C1$/); + for(const [flags,mig] of [[[],'not given; the run would roll back with migration-needs-flag'],[['--migrate'],'given']]){ + const r=await run(e,'activate',C,'acme','--plan',...flags);assert.equal(r.code,0,r.err); + assert.equal(r.out,[...head(mig),'commits: the run adds 2 (newest first, at most 200):',...adds,...planTail(e,C),''].join('\n')); + } + const same=await run(e,'activate',C1,'acme','--plan');assert.equal(same.code,0,same.err); + assert.ok(same.out.includes('target schema: 3g\nmigration: none; the store is at the target\'s version\ncommits: the run adds 1 (newest first, at most 200):\n'),same.out); +}); + +test('T27 --plan names each migration step, or says the steps cannot be read',async(t)=>{ + const e=await mk(t);await seed(e,{store:'3f',pointer:'hand'}); + const two=await run(e,'activate',C,'acme','--plan');assert.equal(two.code,0,two.err); + assert.ok(two.out.includes('\nmigration: 3f to 3h, after a backup of the stopped store\nmigration steps: 3f to 3g, 3g to 3h\nneeds: --migrate'),two.out); + const none=await run(e,'activate',C1,'acme','--plan');assert.equal(none.code,0,none.err); + assert.ok(none.out.includes(`\nmigration: 3f to 3g, after a backup of the stopped store\nmigration steps: cannot be read from ${COPIES}/${C1}/packages/bus/src/store.mjs\nneeds: --migrate`),none.out); + const short=await run(e,'activate',A,'acme','--plan');assert.equal(short.code,0,short.err); + assert.ok(short.out.includes(`\nmigration: 3f to 3g, after a backup of the stopped store\nmigration steps: cannot be read from ${COPIES}/${A}/packages/bus/src/store.mjs\nneeds: --migrate`),short.out); +}); + +test('T25 --plan refuses copies-unrelated when neither copy is an ancestor of the other',async(t)=>{ + const e=await mk(t);await seed(e,{pointer:'hand'}); + const r=await run(e,'activate',S,'acme','--plan');assert.equal(r.code,3); + assert.equal(r.err,`stack-release: refusing copies-unrelated: neither ${B} nor ${S} is an ancestor of the other, so the plan cannot list the commits\n`); + assert.equal(r.out,'');assert.ok(!existsSync(e.log));assert.ok(!existsSync(e.lock)); +}); + +test('T23 --plan reads the store without changing a file, stopped or held',async(t)=>{ + const want=['target schema: 3h','migration: 3g to 3h, after a backup of the stopped store','migration steps: 3g to 3h','needs: --migrate (not given; the run would roll back with migration-needs-flag)','needs: Jason present for this run (decision 96 condition 2)']; + for(const [name,stop,src] of [['stopped store',true,'read from the stopped store'],['held store',false,'read from the store the running bus holds']])await t.test(name,async(t)=>{ + const e=await mk(t);await seed(e,{pointer:'hand'});if(stop)sysctl(e,'stop',BUS); + const before=tree(e.data),units0=[sha(join(e.units,'mosaic-bus@.service')),sha(join(e.units,CON)),sha(join(e.bin,'mosaic'))],store0=sha(e.store); + const r=await run(e,'activate',C,'acme','--plan');assert.equal(r.code,0,r.err); + const out=r.out.split('\n'); + assert.ok(out.includes(`store schema: 3g (${src})`),r.out); + assert.deepEqual(out.slice(out.indexOf('target schema: 3h'),out.indexOf('target schema: 3h')+5),want); + assert.deepEqual(tree(e.data),before);assert.equal(sha(e.store),store0); + assert.deepEqual([sha(join(e.units,'mosaic-bus@.service')),sha(join(e.units,CON)),sha(join(e.bin,'mosaic'))],units0); + assert.ok(!existsSync(e.log));assert.ok(!existsSync(e.lock)); + if(stop)assert.deepEqual(readdirSync(join(e.data,'bus')),['bus.sqlite']); + }); +}); + +test('T24 business-mismatch and capture-failed refuse with exit 3; a short id is a usage error',async(t)=>{ + const e=await mk(t);await seed(e); + let n=calls(e).length,logs=logLines(e).length; + const r=await run(e,'activate',C1,'other');assert.equal(r.code,3); + assert.equal(r.err,'stack-release: refusing business-mismatch: the pointer is for acme\n'); + assert.equal(logLines(e).length,logs+1);assert.equal(logLines(e).at(-1).code,'business-mismatch');assert.deepEqual(startsStops(e,n),[]); + const sets=join(e.state,'stack-rollback');mkdirSync(sets);chmodSync(sets,0o555); + n=calls(e).length;logs=logLines(e).length;const before=fp(e); + const c=await run(e,'activate',C1,'acme');assert.equal(c.code,3); + assert.match(c.err,new RegExp(`^stack-release: refusing capture-failed: could not write the rollback set ${sets}/\\d{8}T\\d{6}Z-${C1.slice(0,12)}; nothing was stopped\\n$`)); + const l=logLines(e).slice(logs); + assert.deepEqual(l.map((x)=>[x.result,x.code]),[['started',undefined],['refused','capture-failed']]);assert.equal(l[0].activation,l[1].activation); + assert.deepEqual(startsStops(e,n),[]);assert.deepEqual(fp(e),before); + const s=await run(e,'status','acme');assert.equal(s.code,0,s.err);assert.ok(!s.out.includes('interrupted run'),s.out); + logs=logLines(e).length; + const u=await run(e,'activate','abc123','acme');assert.equal(u.code,4);assert.match(u.err,/^usage: stack-release.sh adopt BUSINESS\n/); + assert.equal(logLines(e).length,logs); +}); + +test('T16 status prints the pointer, the log, the check, bus status and drift, and writes nothing',async(t)=>{ + const e=await mk(t);await seed(e); + const before=tree(e.data),fp0=fp(e); + const r=await run(e,'status','acme');assert.equal(r.code,0,r.err); + const copy=join(COPIES,B); + assert.equal(r.out,[ + `pointer: ${e.pointer}`,...readFileSync(e.pointer,'utf8').trimEnd().split('\n').map((l)=>` ${l}`), + 'log (last five lines):',...lines(e.log).map((l)=>` ${l}`), + 'check:',` bus: ${copy}`,` Console: ${copy}`,` CLI: ${copy}`,' result: pass', + 'bus status:',` bus: fixture status for ${B}`, + 'unit flags: match the pointer', + 'drift: 4 commit(s) on refactor since the pointer\'s copy',''].join('\n')); + assert.deepEqual(tree(e.data),before);assert.deepEqual(fp(e),fp0); +}); + +test('T19 a started line with no final line: status shows it and every changing verb refuses',async(t)=>{ + const e=await mk(t);await seed(e); + const act=`20261018T000000Z-${C1.slice(0,12)}`; + appendFileSync(e.log,JSON.stringify({at:'2026-10-18T00:00:00Z',op:'activate',from:B,to:C1,backup:null,activation:act,result:'started'})+'\n'); + const s=await run(e,'status','acme');assert.equal(s.code,0); + assert.ok(s.out.includes(`interrupted run: ${act} (started, no final line; Jason or Mos settles it)\n`),s.out); + for(const args of [['adopt','acme'],['activate',C1,'acme'],['rollback','acme']]){ + const before=fp(e),n=calls(e).length,r=await run(e,...args); + assert.equal(r.code,3,args.join(' ')); + assert.equal(r.err,`stack-release: refusing interrupted-run: activation ${act} started and has no final line; Jason or Mos settles it (status shows it)\n`); + assert.deepEqual(fp(e),before);assert.deepEqual(startsStops(e,n),[]); + } +}); + +test('T20 a held run lock refuses run-locked without a log line; status and --plan still run',async(t)=>{ + const e=await mk(t);await seed(e); + const h=spawn('flock',[e.lock,'sleep','30'],{detached:true,stdio:'ignore'}); + t.after(()=>{try{process.kill(-h.pid,'SIGKILL')}catch{}}); + const end=Date.now()+10000; + while(spawnSync('flock',['-n',e.lock,'true']).status!==1){assert.ok(Date.now()setTimeout(r,50))} + for(const args of [['adopt','acme'],['activate',C1,'acme'],['rollback','acme']]){ + const before=fp(e),logs=logLines(e).length,r=await run(e,...args); + assert.equal(r.code,3,args.join(' ')); + assert.equal(r.err,`stack-release: refusing run-locked: another stack-release run holds ${e.lock}\n`); + assert.equal(logLines(e).length,logs);assert.deepEqual(fp(e),before); + } + assert.equal((await run(e,'status','acme')).code,0); + const p=await run(e,'activate',C1,'acme','--plan');assert.equal(p.code,0,p.err); +}); + +test('T21 a bus unit with no MOSAIC_CLAUDE_BIN: adopt and activate refuse no-claude-pin',async(t)=>{ + const drop=(e)=>edit(join(e.units,'mosaic-bus@.service'),`Environment=MOSAIC_CLAUDE_BIN=${e.claude}\n`,''); + const want=`stack-release: refusing no-claude-pin: ${BUS} has no MOSAIC_CLAUDE_BIN\n`; + const a=await mk(t);await seed(a,{pointer:'none'});drop(a); + const r=await run(a,'adopt','acme');assert.equal(r.code,3);assert.equal(r.err,want);assert.equal(pointer(a),null); + const b=await mk(t);await seed(b);drop(b);const before=fp(b),n=calls(b).length; + const s=await run(b,'activate',C1,'acme');assert.equal(s.code,3);assert.equal(s.err,want); + assert.deepEqual(fp(b),before);assert.deepEqual(startsStops(b,n),[]); +}); + +test('T22 the stopped store read: a -wal or -shm refuses store-unclean and is left alone; a clean read adds no file',async(t)=>{ + for(const ext of ['-wal','-shm'])await t.test(`bus.sqlite${ext}`,async(t)=>{ + const e=await mk(t);await seed(e);fault(e,`stop-leaves${ext}`); + const r=await run(e,'activate',C1,'acme','--timeout','3');assert.equal(r.code,1,r.out+r.err); + assert.equal(r.err,rolledBack('store-unclean')); + assert.ok(r.out.includes(`stack-release: ${e.store}${ext} is present; the store is left as it is for Jason or Mos\n`),r.out); + const s=snaps(e),first=s[0],start=s.find((x)=>x.ev==='start'); + assert.equal(first.ev,'stop');assert.ok(`bus.sqlite${ext}` in first.files,JSON.stringify(first)); + assert.deepEqual(start.files,first.files); + }); + await t.test('clean',async(t)=>{ + const e=await mk(t);await seed(e); + const r=await run(e,'activate',C1,'acme');assert.equal(r.code,0,r.out+r.err); + const s=snaps(e),stop=s.find((x)=>x.ev==='stop'),start=s.find((x)=>x.ev==='start'); + assert.deepEqual(Object.keys(stop.files),['bus.sqlite']);assert.deepEqual(start.files,stop.files); + }); +}); diff --git a/scripts/stack-copy.sh b/scripts/stack-copy.sh index 087d7699f..24bbb02a6 100755 --- a/scripts/stack-copy.sh +++ b/scripts/stack-copy.sh @@ -31,15 +31,20 @@ COPY_PATHS=( # Whole directory: every in-repo plug-in module ships with the copy (row 131). packages/plugins ) +# stack-release.sh sources this file and calls reachable before an activate (D1). +reachable() { + local ref + for ref in refs/heads/refactor refs/remotes/origin/refactor; do + git -C "$REPO" rev-parse --verify --quiet "$ref^{commit}" >/dev/null || die "no $ref in $REPO" + git -C "$REPO" merge-base --is-ancestor "$1" "$ref" || die "refusing: $1 is not reachable from $ref" + done +} copy_commit() { [ -d "$REPO/.git" ] || [ "$ALLOW_WORKTREE" = 1 ] || die "refusing: $REPO is not the canonical checkout (its .git is not a directory); run copy from the canonical checkout" [[ "$ID" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] || die "commit must be a commit id or branch name: $ID" - local full ref target tmp + local full target tmp full="$(git -C "$REPO" rev-parse --verify --quiet "$ID^{commit}")" || die "not a commit: $ID" - for ref in refs/heads/refactor refs/remotes/origin/refactor; do - git -C "$REPO" rev-parse --verify --quiet "$ref^{commit}" >/dev/null || die "no $ref in $REPO" - git -C "$REPO" merge-base --is-ancestor "$full" "$ref" || die "refusing: $full is not reachable from $ref" - done + reachable "$full" target="$COPIES/$full" [ ! -L "$target" ] || die "refusing: copy target is a symlink: $target" if [ -e "$target" ]; then echo "exists: $target"; return; fi @@ -96,6 +101,8 @@ check() { printf 'bus: %s\nConsole: %s\nCLI: %s\n' "$bus" "$console" "$cli" [[ "$bus" == "$COPIES/"* ]] && [[ "${bus#"$COPIES/"}" =~ ^[0-9a-f]{40}$ ]] && [ -f "$bus/scripts/mosaic" ] && [ "$bus" = "$console" ] && [ "$bus" = "$cli" ] } +# Sourced (stack-release.sh): define the functions, run nothing. +[ "${BASH_SOURCE[0]}" = "$0" ] || return 0 [ $# -gt 0 ] || usage CMD="$1";shift case "$CMD" in diff --git a/scripts/stack-release.sh b/scripts/stack-release.sh new file mode 100755 index 000000000..4b16e848a --- /dev/null +++ b/scripts/stack-release.sh @@ -0,0 +1,788 @@ +#!/usr/bin/env bash +# Stack activate and rollback, health-gated (D1, #1659; docs/plans/2026-10-18_stack-release.md). +# Moves the bus, the Console and the human CLI from the pointer's copy to another copy together, +# behind a health gate, and puts the old stack back inside the same run when the gate fails. +# Design and tests: agents/darkwing/work/queue-162/BUILD.md. +# +# Who runs it. Decision 96 (Mos, mos-0716) lets this script stop and start the live units under +# four conditions: +# 1. every adopt, activate or rollback runs on Mos's go, and Jason or Mos runs it; +# 2. a run that migrates the store (activate --migrate) happens with Jason present; +# 3. a rollback --restore with events written after the backup waits for Jason, which is why it +# needs --events N; +# 4. this script never prunes backups, rollback sets, post-attempt directories or copies. +# Decisions 94 and 102 still decide who starts the Console; webui-service.sh keeps having no start +# or stop verb. status and --plan only read, and every seat may run them. +# +# Most functions below run as `f || handler`, and bash ignores set -e inside such a call, so +# every step that can fail carries its own `|| return 1`. +set -euo pipefail +cd "$(dirname "$0")/.." +REPO="$(pwd)" +source scripts/common.sh + +COPIES="${XDG_DATA_HOME:-$HOME/.local/share}/mosaic-stack/copies" +BIN="$HOME/.npm-global/bin" +UNIT_DIR="${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user" +CONSOLE_UNIT=mosaic-webui.service +TIMEOUT=60 WANT_PM=0 MIGRATE=0 PLAN=0 RESTORE="" EVENTS="" + +usage() { + cat >&2 <<'USAGE' +usage: stack-release.sh adopt BUSINESS + stack-release.sh activate ID BUSINESS [--pm] [--migrate] [--timeout S] [--plan] + stack-release.sh rollback BUSINESS [--restore ACTIVATION [--events N]] [--timeout S] [--plan] + stack-release.sh status BUSINESS +common options: --copies DIR, --bin DIR +USAGE + exit "${1:-4}" +} +say() { echo "stack-release: $*"; [ -z "${STEPS:-}" ] || echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) $*" >> "$STEPS" || true; } +now() { date -u +%Y-%m-%dT%H:%M:%SZ; } +now_ms() { date -u +%Y-%m-%dT%H:%M:%S.%3NZ; } + +# One JSON object from KEY VALUE pairs; the value @null is JSON null. +jline() { + node -e 'const a=process.argv.slice(1),o={};for(let i=0;i> "$LOG"; } + +# A refusal before anything stopped: exit 3, and one log line with the code, except for +# run-locked (the lock holder may be writing the log) and in --plan, which writes nothing. +refuse() { + local code="$1"; shift + echo "stack-release: refusing $code: $*" >&2 + [ "$code" = run-locked ] || [ "$PLAN" = 1 ] || log_line result refused code "$code" + exit 3 +} + +parse_opts() { + while [ $# -gt 0 ]; do + case "$1" in + --copies) [ $# -ge 2 ] || usage; COPIES="$2"; shift 2 ;; + --bin) [ $# -ge 2 ] || usage; BIN="$2"; shift 2 ;; + --timeout) [ "$OP" != status ] && [ "$OP" != adopt ] && [ $# -ge 2 ] && [[ "$2" =~ ^[1-9][0-9]{0,4}$ ]] || usage; TIMEOUT="$2"; shift 2 ;; + --plan) [ "$OP" = activate ] || [ "$OP" = rollback ] || usage; PLAN=1; shift ;; + --pm) [ "$OP" = activate ] || usage; WANT_PM=1; shift ;; + --migrate) [ "$OP" = activate ] || usage; MIGRATE=1; shift ;; + --restore) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^[0-9]{8}T[0-9]{6}Z-[0-9a-f]{12}$ ]] || usage; RESTORE="$2"; shift 2 ;; + --events) [ "$OP" = rollback ] && [ $# -ge 2 ] && [[ "$2" =~ ^(0|[1-9][0-9]*)$ ]] || usage; EVENTS="$2"; shift 2 ;; + *) usage ;; + esac + done + [ -z "$EVENTS" ] || [ -n "$RESTORE" ] || usage + local d + for d in "$COPIES" "$BIN"; do + case "$d" in /*) ;; *) usage ;; esac + case "$d" in *[!A-Za-z0-9/._+-]*) usage ;; esac + done +} + +# --- The pointer and the log (item 1) --- + +# Sets P_COPY P_BUSINESS P_PM P_CLAUDE P_SCHEMA P_ACT and the array P_CONSOLE; returns 1 with no +# pointer. A pointer that is there but malformed stops the script: Jason or Mos settles it. +read_pointer() { + [ -f "$POINTER" ] || return 1 + local fields + fields="$(node -e ' +const p=JSON.parse(require("fs").readFileSync(process.argv[1],"utf8")); +const ok=/^[0-9a-f]{40}$/.test(p.copy)&&typeof p.business==="string"&&typeof p.pm==="boolean"&&typeof p.claude==="string"&&typeof p.schema==="string"&&typeof p.activation==="string"&&Array.isArray(p.console)&&p.console.every((t)=>typeof t==="string"&&t.length>0&&!/[\n\r]/.test(t)); +if(!ok)throw new Error("pointer fields are missing or malformed"); +process.stdout.write([p.copy,p.business,p.pm?1:0,p.claude,p.schema,p.activation,...p.console].join("\n")+"\n");' -- "$POINTER")" || { echo "stack-release: $POINTER is unreadable; Jason or Mos settles it" >&2; exit 1; } + local -a f + mapfile -t f <<< "$fields" + P_COPY="${f[0]}" P_BUSINESS="${f[1]}" P_PM="${f[2]}" P_CLAUDE="${f[3]}" P_SCHEMA="${f[4]}" P_ACT="${f[5]}" + P_CONSOLE=("${f[@]:6}") +} + +# Writes the pointer through a temp file in state/ and a rename. Args: copy pm claude schema console... +write_pointer() { + local tmp + tmp="$(mktemp "$STATE/.stack-active.XXXXXX")" + node -e ' +const [copy,business,pm,claude,schema,activation,at,...console]=process.argv.slice(1); +process.stdout.write(JSON.stringify({copy,business,pm:pm==="1",claude,console,schema,activation,at})+"\n");' -- \ + "$1" "$BUSINESS" "$2" "$3" "$4" "$ACT" "$(now)" "${@:5}" > "$tmp" + chmod 0644 "$tmp" + mv -f "$tmp" "$POINTER" +} + +# Prints the activation id of each `started` line with no final line after it. +interrupted() { + [ -f "$LOG" ] || return 0 + node -e ' +const open=new Map(); +for(const l of require("fs").readFileSync(process.argv[1],"utf8").split("\n")){ + if(!l.trim())continue; + let o;try{o=JSON.parse(l)}catch{continue} + if(!o.activation)continue; + if(o.result==="started")open.set(o.activation,o);else open.delete(o.activation); +} +for(const k of open.keys())process.stdout.write(k+"\n");' -- "$LOG" +} + +# Reads the log's ok lines, newest first. `previous` prints rollback's target: the first `to` +# that is not the pointer's copy (release.sh's rule). `backup` prints the backup field of the +# line that made the pointer's copy live, the backup a rollback would restore. +from_log() { + [ -f "$LOG" ] || return 1 + node -e ' +const [file,mode,current]=process.argv.slice(1); +const lines=require("fs").readFileSync(file,"utf8").split("\n").filter((l)=>l.trim()); +for(const l of lines.reverse()){ + let o;try{o=JSON.parse(l)}catch{continue} + if(o.result!=="ok"||!["adopt","activate","rollback"].includes(o.op)||!o.to)continue; + if(mode==="previous"&&o.to!==current){process.stdout.write(o.to+"\n");process.exit(0)} + if(mode==="backup"&&o.to===current){if(o.backup)process.stdout.write(o.backup+"\n");process.exit(o.backup?0:1)} +} +process.exit(1);' -- "$LOG" "$1" "$P_COPY" +} + +# --- Reading the units --- + +show() { systemctl --user show "$1" -p LoadState -p WorkingDirectory -p ExecStart -p Environment -p MainPID -p DropInPaths 2>/dev/null; } +argv_of() { sed -n 's/^ExecStart=.*argv\[\]=\([^;]*\) ;.*/\1/p' <<< "$1"; } + +# Sets U_PM U_CLAUDE U_CONSOLE (array) from the effective units; returns 1 if a unit can't be read. +read_units() { + local bus console tok + bus="$(show "$BUS_UNIT")" && console="$(show "$CONSOLE_UNIT")" || return 1 + [ "$(sed -n 's/^LoadState=//p' <<< "$bus")" = loaded ] && [ "$(sed -n 's/^LoadState=//p' <<< "$console")" = loaded ] || return 1 + local -a argv + read -r -a argv <<< "$(argv_of "$bus")" + U_PM=0 + for tok in "${argv[@]:1}"; do [ "$tok" != --pm ] || U_PM=1; done + U_CLAUDE="" + read -r -a argv <<< "$(sed -n 's/^Environment=//p' <<< "$bus")" + for tok in "${argv[@]}"; do case "$tok" in MOSAIC_CLAUDE_BIN=*) U_CLAUDE="${tok#MOSAIC_CLAUDE_BIN=}" ;; esac; done + read -r -a argv <<< "$(argv_of "$console")" + U_CONSOLE=() + local i serve=0 + for ((i = 0; i < ${#argv[@]}; i++)); do + if [ "$serve" = 0 ]; then [ "${argv[i]}" != serve ] || serve=1; continue; fi + if [ "${argv[i]}" = --checkout ]; then i=$((i + 1)); continue; fi + U_CONSOLE+=("${argv[i]}") + done + [ "$serve" = 1 ] || return 1 +} + +console_port() { + local i + for ((i = 0; i < ${#P_CONSOLE[@]}; i++)); do + [ "${P_CONSOLE[i]}" != --port ] || { echo "${P_CONSOLE[i + 1]:-7332}"; return; } + done + echo 7332 +} + +# --- Copies --- + +# stack-copy.sh's own helpers, in a subshell so its die, parse and globals stay out of this script. +copy_ok() ( source "$REPO/scripts/stack-copy.sh"; ID="$1"; COPIES="$COPIES_DIR"; copy_root >/dev/null && reachable "$ID" ) +copy_exists() ( source "$REPO/scripts/stack-copy.sh"; ID="$1"; COPIES="$COPIES_DIR"; copy_root >/dev/null ) + +# Prints the copy's SCHEMA_VERSION, then each `from:` of its migration chain, one per line. Text +# only: the copy's code is never imported or run for this. +copy_schema() { + local f="$COPIES/$1/packages/bus/src/store.mjs" v + [ -f "$f" ] || return 1 + v="$(grep -E "^export const SCHEMA_VERSION = '[A-Za-z0-9]+';\$" "$f")" || return 1 + [ "$(wc -l <<< "$v")" = 1 ] || return 1 + v="${v#*\'}"; echo "${v%%\'*}" + { grep -E "^[[:space:]]*from: '[A-Za-z0-9]+',\$" "$f" || true; } | sed "s/.*from: '\\([A-Za-z0-9]*\\)',\$/\\1/" +} + +# Sets T_SCHEMA and T_CHAIN (array) for copy $1; returns 1 when either read fails. +target_schema() { + local out + out="$(copy_schema "$1")" || return 1 + local -a l + mapfile -t l <<< "$out" + T_SCHEMA="${l[0]}" + T_CHAIN=("${l[@]:1}") +} + +in_chain() { local v; for v in "${T_CHAIN[@]}"; do [ "$v" != "$1" ] || return 0; done; return 1; } + +# The item 3 checks on the target copy, in order: copy, schema, already-active. +check_target() { + copy_exists "$TO" 2>/dev/null || refuse copy-missing "no copy $TO in $COPIES" + copy_ok "$TO" 2>/dev/null || refuse copy-unreachable "$TO is not reachable from refactor and origin/refactor" + target_schema "$TO" || refuse copy-schema-unknown "cannot read SCHEMA_VERSION and its chain from $COPIES/$TO/packages/bus/src/store.mjs" + [ "$TO" != "$P_COPY" ] || refuse already-active "$TO is the pointer's copy" +} + +# --- The store --- + +STORE_JS=' +const {DatabaseSync}=require("node:sqlite");const {pathToFileURL}=require("node:url"); +const [mode,file,stamp]=process.argv.slice(1); +// immutable=1 on a stopped store with no -wal or -shm: it creates no file and leaves the main +// file mtime and sha256 as they were (the row 162 probe). A plain readOnly open of a stopped +// WAL store creates -shm and -wal, so it is used only on a store the running bus holds. +const db=mode.startsWith("held")?new DatabaseSync(file,{readOnly:true}):new DatabaseSync(pathToFileURL(file).href+"?immutable=1",{readOnly:true}); +try{ + if(mode.endsWith("version"))process.stdout.write(db.prepare("SELECT value FROM meta WHERE key=?").get("schema_version").value+"\n"); + else process.stdout.write(db.prepare("SELECT count(*) AS n FROM events WHERE at > ?").get(stamp).n+"\n"); +}finally{db.close()}' + +# Sets UNCLEAN and returns 0 when a -wal or -shm sits beside the store. An immutable read would +# skip that WAL and could give a stale answer, so the caller refuses store-unclean and leaves the +# store as it is. Called directly, never inside $(...), so UNCLEAN reaches the caller. +store_unclean() { + local f + for f in "$STORE-wal" "$STORE-shm"; do + if [ -e "$f" ] || [ -L "$f" ]; then UNCLEAN="$f"; return 0; fi + done + return 1 +} +# A read of the stopped store: version, or count STAMP. Call store_unclean first. +store_read() { node -e "$STORE_JS" -- "$1" "$STORE" "${2:-}"; } + +# No writer.lock, and no process with a file open under /bus/ (stopped-check.sh's +# test). One find over the /proc fd links, not a readlink per link: this host has about 16,000. +# find's status is ignored, since other users' fd dirs are unreadable; so were they in the loop. +stopped() { + [ ! -e "$DATA/bus/writer.lock" ] || return 1 + local pat + pat="$(sed 's/[][*?\\]/\\&/g' <<< "$DATA/bus")/*" + [ -z "$(find /proc/[0-9]*/fd -mindepth 1 -maxdepth 1 -lname "$pat" -print -quit 2>/dev/null || true)" ] +} + +sums() { (cd "$1" && find . -type f ! -name SHA256SUMS ! -name steps.log -print0 | sort -z | xargs -0 -r sha256sum); } +sum_of() { awk -v f="./$2" '$2 == f { print $1 }' "$1/SHA256SUMS"; } + +# The backup of the stopped store (backup.sh's steps). Only bus.sqlite: a -wal or -shm was +# already refused as store-unclean. Sets TAKEN once the backup is whole. +backup_store() { + local o="$DATA/backups/stack/$ACT" + ( + umask 077 + mkdir -p "$DATA/backups/stack" && + mkdir -m 700 "$o" "$o/bus" && + (cd "$DATA/bus" && sha256sum bus.sqlite) > "$o/SOURCE.sha256" && + cp -p "$STORE" "$o/bus/" && + (cd "$o/bus" && sha256sum bus.sqlite) > "$o/SHA256SUMS" && + cmp -s "$o/SOURCE.sha256" "$o/SHA256SUMS" && + stat -c '%n %a %U %s %y' "$STORE" > "$o/MODES.txt" && + ls -1A "$DATA/bus" > "$o/LISTING.txt" && + echo "$DATA" > "$o/DATA_ROOT" && + now_ms > "$o/STAMP" && + echo "$STORE_SCHEMA" > "$o/SCHEMA" && + echo "$RS" > "$o/ROLLBACK_SET" + ) || return 1 + stopped || return 1 + TAKEN="$ACT" BACKUP="$ACT" + say "backup: $o" +} + +# Puts backup $1 back as restore.sh does: the current bus/ moves to $POST, nothing is deleted. +# Sets RESTORED once the old bus/ has moved, so the caller knows the store was swapped. +restore_store() { + local o="$DATA/backups/stack/$1" + [ "$(cat "$o/DATA_ROOT" 2>/dev/null)" = "$DATA" ] || return 1 + stopped || return 1 + (cd "$o/bus" && sha256sum -c --quiet ../SHA256SUMS) || return 1 + [ ! -e "$POST" ] || return 1 + mv "$DATA/bus" "$POST" || return 1 + RESTORED=1 + (umask 077; mkdir -m 700 "$DATA/bus") || return 1 + cp -p "$o"/bus/* "$DATA/bus/" || return 1 + (cd "$DATA/bus" && sha256sum -c --quiet "$o/SHA256SUMS") || return 1 + [ "$(ls -1A "$DATA/bus")" = "$(ls -1A "$o/bus")" ] || return 1 + say "restored $o; the store from before the restore is at $POST" +} + +# --- The run (item 4) --- + +# Step 1: the rollback set. A failure here has changed nothing but the set itself. +capture() { + mkdir -p -m 0700 "$STATE/stack-rollback" 2>/dev/null || return 1 + mkdir -m 0700 "$RS" 2>/dev/null || return 1 + STEPS="$RS/steps.log" + local u d + for u in mosaic-bus@.service "$CONSOLE_UNIT"; do + [ -f "$UNIT_DIR/$u" ] || { say "capture: no $UNIT_DIR/$u"; return 1; } + cp -p "$UNIT_DIR/$u" "$RS/$u" || return 1 + done + for d in mosaic-bus@.service.d "mosaic-bus@$BUSINESS.service.d" "$CONSOLE_UNIT.d"; do + [ ! -d "$UNIT_DIR/$d" ] || cp -pr "$UNIT_DIR/$d" "$RS/$d" || return 1 + done + [ -f "$BIN/mosaic" ] || { say "capture: no wrapper $BIN/mosaic"; return 1; } + cp -p "$BIN/mosaic" "$RS/wrapper" || return 1 + stat -c %a "$BIN/mosaic" > "$RS/WRAPPER_MODE" || return 1 + for u in "$BUS_UNIT" "$CONSOLE_UNIT"; do show "$u" > "$RS/show-$u.txt" || return 1; done + sums "$RS" > "$RS/SHA256SUMS" || return 1 + say "captured the rollback set: $RS" +} + +# Stops the Console, then the bus, and waits for both and the stopped check, within --timeout. +stop_both() { + systemctl --user stop "$CONSOLE_UNIT" || return 1 + systemctl --user stop "$BUS_UNIT" || return 1 + local deadline=$((SECONDS + TIMEOUT)) u s down + while :; do + down=1 + for u in "$CONSOLE_UNIT" "$BUS_UNIT"; do + s="$(systemctl --user is-active "$u" 2>/dev/null || true)" + case "$s" in inactive|failed) ;; *) down=0 ;; esac + done + if [ "$down" = 1 ] && stopped; then say "stopped both units"; return 0; fi + [ "$SECONDS" -lt "$deadline" ] || return 1 + sleep 1 + done +} + +start_both() { + START_EPOCH="$(date +%s)" && START_MS="$(date +%s%3N)" || return 1 + systemctl --user start "$BUS_UNIT" && systemctl --user start "$CONSOLE_UNIT" +} + +# Step 4: installs copy $1 with pm $2, claude $3 and the Console flags after them. Sets FAIL. +install_stack() { + local id="$1" pm="$2" claude="$3"; shift 3 + local -a pmflag=() + [ "$pm" = 0 ] || pmflag=(--pm) + scripts/bus-service.sh install --copy "$id" --copies "$COPIES" "${pmflag[@]}" --claude "$claude" >> "$STEPS" 2>&1 || { FAIL=install-bus; return 1; } + scripts/webui-service.sh install --copy "$id" --copies "$COPIES" "$@" >> "$STEPS" 2>&1 || { FAIL=install-console; return 1; } + scripts/stack-copy.sh wrapper "$id" --copies "$COPIES" --bin "$BIN" >> "$STEPS" 2>&1 || { FAIL=wrapper; return 1; } + say "installed $id" +} + +# --- The health gate (item 5) --- + +# One pass of every check against copy $1 with pm $2. Sets GATE to the first failing code and +# GATE_WHY to what it saw. +gate_once() { + local copy="$COPIES/$1" pm="$2" main out + GATE=units-inactive GATE_WHY="" + systemctl --user is-active --quiet "$BUS_UNIT" && systemctl --user is-active --quiet "$CONSOLE_UNIT" || return 1 + GATE=bus-status + main="$(systemctl --user show "$BUS_UNIT" -p MainPID | sed -n 's/^MainPID=//p')" || return 1 + out="$("$copy/scripts/mosaic" bus status --json 2>/dev/null)" || { GATE_WHY="bus status failed"; return 1; } + # Every `bus start` writes boot.at (packages/cli/src/boot-status.mjs), so a value from before + # this start means the host on the socket is not the one this run started. + GATE_WHY="$(node -e ' +const [b,main,since]=process.argv.slice(2),s=JSON.parse(process.argv[1]); +const why=!s.host?"no host":s.host.business!==b?`host for ${s.host.business}`:s.host.live!==true?"host not live":String(s.host.pid)!==main?`host pid ${s.host.pid} is not MainPID ${main}`:!s.boot||s.boot.ok!==true?"boot status not ok":s.boot.business!==b?`boot status for ${s.boot.business}`:!(Date.parse(s.boot.at)>=Number(since))?"boot status is from before this start":""; +process.stdout.write(why);' -- "$out" "$BUSINESS" "$main" "$START_MS" 2>&1)" || { GATE_WHY="bus status unreadable"; return 1; } + [ -z "$GATE_WHY" ] || return 1 + GATE=healthz + node -e ' +fetch(`http://127.0.0.1:${process.argv[1]}/healthz`,{signal:AbortSignal.timeout(5000)}).then((r)=>process.exit(r.status===200?0:1),()=>process.exit(1));' -- "$(console_port)" || return 1 + GATE=copy-check + scripts/stack-copy.sh check --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" > /dev/null 2>&1 || return 1 + if [ "$pm" = 1 ]; then + GATE=pm-launch-unconfirmed + pm_launched "$copy" || return 1 + fi + GATE="" +} + +# Ruling 2 (sage-0736, sage-0738). The PM's run is the one named on the single +# `launched : run , ` line that `bus start --pm` prints +# (packages/cli/src/cli.mjs) to the unit's journal since this start. The gate depends on that +# wording: if it changes, this check refuses rather than passes. --since takes @, +# because this host's journalctl reads a bare time as local time. +pm_launched() { + local journal run list + journal="$(journalctl --user -u "$BUS_UNIT" --since "@$START_EPOCH" -o cat 2>/dev/null)" || { GATE_WHY="journal read failed"; return 1; } + local -a lines=() + mapfile -t lines < <(grep '^launched ' <<< "$journal" || true) + [ "${#lines[@]}" -gt 0 ] || { GATE_WHY="no launched line since the start"; return 1; } + [ "${#lines[@]}" = 1 ] || { GATE_WHY="${#lines[@]} launched lines since the start"; return 1; } + [[ "${lines[0]}" =~ ^launched\ ([^[:space:]]+):\ run\ ([^[:space:]]+),\ ]] || { GATE_WHY="the launched line does not parse: ${lines[0]}"; return 1; } + run="${BASH_REMATCH[2]}" + list="$("$1/scripts/mosaic" launches list --json 2>/dev/null)" || { GATE_WHY="launches list failed"; return 1; } + GATE_WHY="$(node -e ' +const [list,run,b,since]=process.argv.slice(1),s=JSON.parse(list).find((x)=>x.run===run); +process.stdout.write(!s?`run ${run} is not in launches list`:s.business!==b?`run ${run} is for ${s.business}`:s.live!==true?`run ${run} is not live`:!(Date.parse(s.startedAt)>=Number(since))?`run ${run} started before this start`:"");' -- "$list" "$run" "$BUSINESS" "$START_MS" 2>&1)" || { GATE_WHY="launches list unreadable"; return 1; } + [ -z "$GATE_WHY" ] +} + +# Polls every second until every check passes or --timeout runs out. +gate() { + local deadline=$((SECONDS + TIMEOUT)) + while :; do + if gate_once "$1" "$2"; then say "gate passed on $1"; return 0; fi + [ "$SECONDS" -lt "$deadline" ] || { say "gate failed on $1: $GATE${GATE_WHY:+ ($GATE_WHY)}"; return 1; } + sleep 1 + done +} + +# --- Rollback inside the run (item 6) --- + +# Puts back the captured unit files, drop-ins and wrapper, checked against SHA256SUMS. +put_back() { + local u f rel tmp + for u in mosaic-bus@.service "$CONSOLE_UNIT"; do + tmp="$(mktemp "$UNIT_DIR/.$u.XXXXXX")" && cp -p "$RS/$u" "$tmp" && mv -f "$tmp" "$UNIT_DIR/$u" || return 1 + done + for u in mosaic-bus@.service.d "mosaic-bus@$BUSINESS.service.d" "$CONSOLE_UNIT.d"; do + [ -d "$RS/$u" ] || continue + while IFS= read -r -d '' f; do + rel="${f#"$RS/$u/"}" + ! cmp -s "$f" "$UNIT_DIR/$u/$rel" 2>/dev/null || continue + mkdir -p "$(dirname "$UNIT_DIR/$u/$rel")" || return 1 + tmp="$(mktemp "$UNIT_DIR/$u/.restore.XXXXXX")" && cp -p "$f" "$tmp" && mv -f "$tmp" "$UNIT_DIR/$u/$rel" || return 1 + done < <(find "$RS/$u" -type f -print0) + done + tmp="$(mktemp "$BIN/.mosaic.XXXXXX")" && cp -p "$RS/wrapper" "$tmp" && chmod "$(cat "$RS/WRAPPER_MODE")" "$tmp" && mv -f "$tmp" "$BIN/mosaic" || return 1 + for u in mosaic-bus@.service "$CONSOLE_UNIT"; do + [ "$(sha256sum < "$UNIT_DIR/$u" | cut -d' ' -f1)" = "$(sum_of "$RS" "$u")" ] || return 1 + done + [ "$(sha256sum < "$BIN/mosaic" | cut -d' ' -f1)" = "$(sum_of "$RS" wrapper)" ] || return 1 + [ "$(stat -c %a "$BIN/mosaic")" = "$(cat "$RS/WRAPPER_MODE")" ] || return 1 + systemctl --user daemon-reload || return 1 + say "put back the unit files and the wrapper from $RS" +} + +# A failure after step 2: put the pointer's stack back and gate it. $1 is the run's code. Only a +# backup this run took (TAKEN) is restored; a rollback's --restore backup never is. +rollback_in_run() { + trap - INT TERM HUP + local code="$1" rcode="" + say "rolling back: $code" + if ! stop_both; then rcode=stop-timeout + elif ! put_back; then rcode=put-back + elif [ -n "$TAKEN" ] && ! restore_store "$TAKEN"; then rcode=restore + elif ! start_both; then rcode=start + elif ! gate "$P_COPY" "$P_PM"; then rcode="$GATE" + fi + if [ -z "$rcode" ]; then + log_line result gate-failed code "$code" rollback ok + echo "stack-release: $code; rolled back to $P_COPY, which passed the gate" >&2 + exit 1 + fi + rollback_failed "$code" "$rcode" +} + +# Ruling 4 (T14b): once rollback --restore has swapped the store, any later failure skips item 6, +# since putting the newer store back would be a second swap. Before the swap, item 6 runs. +after_fail() { + [ -z "$RESTORED" ] || rollback_failed "$1" not-attempted + rollback_in_run "$1" +} + +# Stop both units, print both results, log rollback-failed, exit 2. No retry. +rollback_failed() { + trap - INT TERM HUP + systemctl --user stop "$CONSOLE_UNIT" || true + systemctl --user stop "$BUS_UNIT" || true + say "rollback-failed: run $1, rollback $2" + log_line result rollback-failed code "$1" rollback failed rollbackCode "$2" + echo "stack-release: rollback-failed: the run failed with $1 and the rollback with $2; both units are stopped. Rollback set: $RS${BACKUP:+, backup $DATA/backups/stack/$BACKUP}${RESTORED:+, post-attempt store $POST}" >&2 + exit 2 +} + +# --- Refusals (item 3) --- + +lock() { + mkdir -p "$STATE" + exec 9> "$STATE/stack-release.lock" + flock -n 9 || refuse run-locked "another stack-release run holds $STATE/stack-release.lock" + local open + open="$(interrupted | paste -sd' ' -)" + [ -z "$open" ] || refuse interrupted-run "activation $open started and has no final line; Jason or Mos settles it (status shows it)" +} + +instance_busy() { + local out unit load state rest + out="$(systemctl --user list-units --all --no-legend --plain 'mosaic-bus@*.service' 2>/dev/null)" || refuse instance-busy "systemctl --user list-units failed, so the bus state is unknown" + while read -r unit load state rest; do + [[ "$unit" == mosaic-bus@*.service ]] || continue + if [ "$unit" = "$BUS_UNIT" ]; then + case "$state" in activating|deactivating|reloading) refuse instance-busy "$unit is $state" ;; esac + else + case "$state" in inactive|failed) ;; *) refuse instance-busy "$unit is $state; the copy and wrapper switch reach it too" ;; esac + fi + done <<< "$out" +} + +# install_guard's drop-in scan (bus-service.sh), done before anything stops. +drop_in_override() { + local paths directory dropdir file + paths="$(systemd-analyze --user unit-paths 2>/dev/null)" && [ -n "$paths" ] || refuse drop-in-override "systemd unit paths are unknown" + while IFS= read -r directory; do + [ -n "$directory" ] && [ -d "$directory" ] || continue + for dropdir in "$directory"/mosaic-bus@*.service.d "$directory"/mosaic-.service.d "$directory"/service.d; do + [ -d "$dropdir" ] || continue + for file in "$dropdir"/*.conf; do + [ -e "$file" ] || continue + if grep -Eq '^[[:space:]]*(ExecStart|WorkingDirectory)[[:space:]]*=' "$file" 2>/dev/null; then + refuse drop-in-override "$file overrides ExecStart or WorkingDirectory; the operator removes it as in TOOLS.md step 4" + fi + done + done + done <<< "$paths" +} + +# unit-flags-drift: the effective units against the pointer, and --pm against the pointer's pm. +flags_drift() { + local d="" + [ "$U_PM" = "$P_PM" ] || d+=$'\n'" pm: pointer $P_PM, unit $U_PM" + [ "$U_CLAUDE" = "$P_CLAUDE" ] || d+=$'\n'" claude: pointer $P_CLAUDE, unit $U_CLAUDE" + [ "${P_CONSOLE[*]}" = "${U_CONSOLE[*]}" ] || d+=$'\n'" console: pointer ${P_CONSOLE[*]}, unit ${U_CONSOLE[*]}" + [ "$OP" != activate ] || [ "$WANT_PM" = "$P_PM" ] || d+=$'\n'" pm: pointer $P_PM, --pm given $WANT_PM" + [ -z "$d" ] || refuse unit-flags-drift "the units differ from the pointer; settle it first:$d" +} + +# The item 3 checks after the pointer, shared by activate and rollback, for target $TO. +precheck() { + check_target + instance_busy + drop_in_override + read_units || refuse unit-flags-drift "cannot read $BUS_UNIT or $CONSOLE_UNIT" + [ -n "$U_CLAUDE" ] || refuse no-claude-pin "$BUS_UNIT has no MOSAIC_CLAUDE_BIN" + flags_drift +} + +pointer_for_business() { + read_pointer || refuse no-pointer "no $POINTER; run adopt first" + FROM="$P_COPY" + [ "$P_BUSINESS" = "$BUSINESS" ] || refuse business-mismatch "the pointer is for $P_BUSINESS" +} + +new_activation() { + ACT="$(date -u +%Y%m%dT%H%M%SZ)-${TO:0:12}" + RS="$STATE/stack-rollback/$ACT" + POST="$DATA/bus.post-attempt-$(date -u +%Y%m%dT%H%M%SZ)" +} + +# --- Verbs --- + +do_adopt() { + lock + [ ! -f "$POINTER" ] || { read_pointer; FROM="$P_COPY"; refuse already-adopted "$POINTER exists (copy $P_COPY)"; } + local out + out="$(scripts/stack-copy.sh check --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1)" || refuse check-failed "stack-copy.sh check did not pass:"$'\n'"$out" + TO="$(sed -n 's/^bus: //p' <<< "$out")" + TO="${TO#"$COPIES/"}" + read_units || refuse check-failed "cannot read $BUS_UNIT or $CONSOLE_UNIT" + [ -n "$U_CLAUDE" ] || refuse no-claude-pin "$BUS_UNIT has no MOSAIC_CLAUDE_BIN" + target_schema "$TO" || refuse copy-schema-unknown "cannot read SCHEMA_VERSION from $COPIES/$TO/packages/bus/src/store.mjs" + ACT="$(date -u +%Y%m%dT%H%M%SZ)-${TO:0:12}" + write_pointer "$TO" "$U_PM" "$U_CLAUDE" "$T_SCHEMA" "${U_CONSOLE[@]}" + log_line result ok + echo "adopted $TO for $BUSINESS (pm $U_PM, schema $T_SCHEMA)" +} + +# The store's version for --plan, from the store itself: readOnly while the bus holds it (adds no +# file), immutable when it is stopped and clean. Sets S_SCHEMA and S_FROM. Falls back to the +# pointer's schema, and says so, when there is no store file or it can't be read. +plan_store() { + local src="" why="" + S_SCHEMA="" S_FROM="" + if [ ! -e "$STORE" ]; then why="there is no store file" + elif systemctl --user is-active --quiet "$BUS_UNIT"; then + src="the store the running bus holds" + S_SCHEMA="$(node -e "$STORE_JS" -- heldversion "$STORE" 2>/dev/null)" || S_SCHEMA="" + elif store_unclean; then why="the store was not read: $UNCLEAN is present" + else + src="the stopped store" + S_SCHEMA="$(store_read version 2>/dev/null)" || S_SCHEMA="" + fi + if [ -n "$S_SCHEMA" ]; then S_FROM="read from $src" + else S_SCHEMA="$P_SCHEMA" S_FROM="from the pointer; ${why:-$src could not be read}" + fi +} + +# sage-0774 and sage-0778: the commits the run adds (activate) or takes away (rollback), so the +# rows they carry can be named. Copies where neither is an ancestor of the other refuse the plan +# before it prints anything. Sets RANGE and VERB. +plan_range() { + if git -C "$REPO" merge-base --is-ancestor "$P_COPY" "$TO" 2>/dev/null; then RANGE="$P_COPY..$TO" VERB=adds + elif git -C "$REPO" merge-base --is-ancestor "$TO" "$P_COPY" 2>/dev/null; then RANGE="$TO..$P_COPY" VERB="takes away" + else refuse copies-unrelated "neither $P_COPY nor $TO is an ancestor of the other, so the plan cannot list the commits" + fi +} +plan_commits() { + echo "commits: the run $VERB $(git -C "$REPO" rev-list --count --no-merges "$RANGE") (newest first, at most 200):" + git -C "$REPO" log --oneline --no-merges --max-count=200 "$RANGE" | sed 's/^/ /' +} + +# Each step from $S_SCHEMA to $T_SCHEMA, from the target store.mjs's `from:` and `to:` pairs. Text +# only, like copy_schema. Prints "cannot be read from FILE" when the lines don't pair up or the +# steps don't reach $T_SCHEMA. +plan_steps() { + local f="$COPIES/$TO/packages/bus/src/store.mjs" v="$S_SCHEMA" out="" i + local -a l + local -A next=() + mapfile -t l < <(grep -E "^[[:space:]]*(from|to): '[A-Za-z0-9]+',\$" "$f" | sed -E "s/^[[:space:]]*(from|to): '([A-Za-z0-9]*)',\$/\\1 \\2/") + (( ${#l[@]} % 2 == 0 )) || { echo "cannot be read from $f"; return; } + for ((i = 0; i < ${#l[@]}; i += 2)); do + [[ "${l[i]}" == "from "* && "${l[i+1]}" == "to "* ]] || { echo "cannot be read from $f"; return; } + next["${l[i]#from }"]="${l[i+1]#to }" + done + for ((i = 0; i < ${#next[@]}; i++)); do + [ "$v" != "$T_SCHEMA" ] && [ -n "${next[$v]:-}" ] || break + out+="${out:+, }$v to ${next[$v]}"; v="${next[$v]}" + done + if [ "$v" = "$T_SCHEMA" ] && [ -n "$out" ]; then echo "$out"; else echo "cannot be read from $f"; fi +} + +# Item 8 and sage-0745: what the run would do, with any migration named plainly. Reads the +# pointer, the log, the target's store.mjs and the store's version. Writes nothing. +plan() { + plan_range + plan_store + echo "plan: $OP for $BUSINESS" + echo "from: $P_COPY" + echo "to: $TO" + echo "store schema: $S_SCHEMA ($S_FROM)" + echo "target schema: $T_SCHEMA" + [ "$S_SCHEMA" = "$P_SCHEMA" ] || echo "note: the pointer says $P_SCHEMA; the run reads the stopped store and goes by that" + if [ "$S_SCHEMA" = "$T_SCHEMA" ]; then + echo "migration: none; the store is at the target's version" + elif [ "$OP" = activate ] && in_chain "$S_SCHEMA"; then + echo "migration: $S_SCHEMA to $T_SCHEMA, after a backup of the stopped store" + echo "migration steps: $(plan_steps)" + echo "needs: --migrate$([ "$MIGRATE" = 1 ] && echo ' (given)' || echo ' (not given; the run would roll back with migration-needs-flag)')" + echo "needs: Jason present for this run (decision 96 condition 2)" + elif [ "$OP" = activate ]; then + echo "migration: none possible; $S_SCHEMA is not in $T_SCHEMA's migration chain, so the run would roll back with store-newer" + else + echo "restore: the store must go back to $T_SCHEMA; needs --restore $(from_log backup || echo ACTIVATION)" + fi + plan_commits + echo "bus: scripts/bus-service.sh install --copy $TO --copies $COPIES$([ "$P_PM" = 0 ] || echo ' --pm') --claude $P_CLAUDE" + echo "Console: scripts/webui-service.sh install --copy $TO --copies $COPIES${P_CONSOLE[*]:+ ${P_CONSOLE[*]}}" + echo "CLI: scripts/stack-copy.sh wrapper $TO --copies $COPIES --bin $BIN" +} + +# Steps 4.1 to 4.5 and the gate for target $TO. $1 is the store step: migrate or restore. +run_steps() { + log_line result started + capture || refuse capture-failed "could not write the rollback set $RS; nothing was stopped" + trap 'after_fail signal' INT TERM HUP + stop_both || rollback_in_run stop-timeout + if [ -e "$STORE" ]; then + if store_unclean; then say "$UNCLEAN is present; the store is left as it is for Jason or Mos"; rollback_in_run store-unclean; fi + STORE_SCHEMA="$(store_read version)" && [ -n "$STORE_SCHEMA" ] || rollback_in_run schema-unreadable + say "stopped store schema: $STORE_SCHEMA" + if [ "$1" = migrate ]; then + if [ "$STORE_SCHEMA" != "$T_SCHEMA" ]; then + in_chain "$STORE_SCHEMA" || rollback_in_run store-newer + [ "$MIGRATE" = 1 ] || rollback_in_run migration-needs-flag + backup_store || rollback_in_run backup + node "$COPIES/$TO/packages/bus/src/migrate-schema.mjs" "$DATA" >> "$STEPS" 2>&1 || rollback_in_run migrate-failed + say "migrated the store to $T_SCHEMA" + fi + else + [ "$STORE_SCHEMA" = "$P_SCHEMA" ] || rollback_in_run store-differs + if [ -n "$RESTORE" ]; then + local n + n="$(store_read count "$(cat "$DATA/backups/stack/$RESTORE/STAMP")")" || rollback_in_run schema-unreadable + [ "$n" = "$COUNT" ] || rollback_in_run post-backup-events + restore_store "$RESTORE" || after_fail restore + fi + fi + elif [ -n "$RESTORE" ]; then + rollback_in_run store-differs + fi + install_stack "$TO" "$P_PM" "$P_CLAUDE" "${P_CONSOLE[@]}" || after_fail "$FAIL" + start_both || after_fail start + gate "$TO" "$P_PM" || after_fail "$GATE" + trap - INT TERM HUP + write_pointer "$TO" "$P_PM" "$P_CLAUDE" "$T_SCHEMA" "${P_CONSOLE[@]}" + log_line result ok + echo "stack-release: $OP ok: $TO is live for $BUSINESS" +} + +do_activate() { + TO="$ID" + [ "$PLAN" = 1 ] || lock + pointer_for_business + if [ "$PLAN" = 1 ]; then check_target; plan; exit 0; fi + precheck + new_activation + run_steps migrate +} + +do_rollback() { + [ "$PLAN" = 1 ] || lock + pointer_for_business + TO="$(from_log previous)" || refuse no-previous "the log names no earlier copy than $P_COPY" + if [ "$PLAN" = 1 ]; then check_target; plan; exit 0; fi + precheck + [ "$P_SCHEMA" = "$T_SCHEMA" ] || [ -n "$RESTORE" ] || refuse schema-differs "the store is at $P_SCHEMA and $TO expects $T_SCHEMA; pass --restore ACTIVATION" + new_activation + if [ -n "$RESTORE" ]; then + local o="$DATA/backups/stack/$RESTORE" rc=0 + [ -f "$o/SHA256SUMS" ] && [ "$(cat "$o/DATA_ROOT" 2>/dev/null)" = "$DATA" ] && [ "$(cat "$o/SCHEMA" 2>/dev/null)" = "$T_SCHEMA" ] && [ -f "$o/STAMP" ] \ + || refuse restore-mismatch "$o is not a backup of this root at $T_SCHEMA" + # Before anything stops. A store the running bus holds is read with readOnly, which adds + # no file (the probe); a stopped one falls under the store_unclean rule. + if systemctl --user is-active --quiet "$BUS_UNIT"; then + COUNT="$(node -e "$STORE_JS" -- heldcount "$STORE" "$(cat "$o/STAMP")")" || rc=1 + else + ! store_unclean || refuse store-unclean "$UNCLEAN is present; the bus did not close the store cleanly" + COUNT="$(store_read count "$(cat "$o/STAMP")")" || rc=1 + fi + [ "$rc" = 0 ] || refuse restore-mismatch "cannot count the events after $(cat "$o/STAMP")" + echo "restore: $o; the store now in place moves to $POST and is kept; $COUNT event(s) were written after the backup" + [ "$COUNT" = 0 ] || [ "$EVENTS" = "$COUNT" ] || refuse post-backup-events "$COUNT event(s) were written after the backup; Jason decides, then --events $COUNT" + BACKUP="$RESTORE" + fi + run_steps restore +} + +do_status() { + echo "pointer: $POINTER" + if read_pointer; then sed 's/^/ /' "$POINTER"; else echo " none"; fi + echo "log (last five lines):" + if [ -f "$LOG" ]; then tail -n 5 "$LOG" | sed 's/^/ /'; else echo " none"; fi + local open + open="$(interrupted | paste -sd' ' -)" + [ -z "$open" ] || echo "interrupted run: $open (started, no final line; Jason or Mos settles it)" + echo "check:" + if scripts/stack-copy.sh check --copies "$COPIES" --bin "$BIN" --business "$BUSINESS" 2>&1 | sed 's/^/ /'; then echo " result: pass"; else echo " result: FAIL"; fi + [ -n "$P_COPY" ] || return 0 + echo "bus status:" + "$COPIES/$P_COPY/scripts/mosaic" bus status 2>&1 | sed 's/^/ /' || true + if read_units; then + local d="" + [ "$U_PM" = "$P_PM" ] || d+=" pm" + [ "$U_CLAUDE" = "$P_CLAUDE" ] || d+=" claude" + [ "${P_CONSOLE[*]}" = "${U_CONSOLE[*]}" ] || d+=" console" + echo "unit flags: ${d:+differ from the pointer:}${d:-match the pointer}" + else + echo "unit flags: cannot read the units" + fi + if git -C "$REPO" merge-base --is-ancestor "$P_COPY" refs/heads/refactor 2>/dev/null; then + echo "drift: $(git -C "$REPO" rev-list --count "$P_COPY..refs/heads/refactor") commit(s) on refactor since the pointer's copy" + else + echo "drift: the pointer's copy is not an ancestor of refactor" + fi +} + +# --- Dispatch --- + +[ $# -gt 0 ] || usage +OP="$1"; shift +ID="" +case "$OP" in + activate) [ $# -ge 2 ] || usage; ID="$1"; BUSINESS="$2"; shift 2; [[ "$ID" =~ ^[0-9a-f]{40}$ ]] || usage ;; + adopt|rollback|status) [ $# -ge 1 ] || usage; BUSINESS="$1"; shift ;; + -h|--help) usage 0 ;; + *) usage ;; +esac +[[ "$BUSINESS" =~ ^[a-z0-9][a-z0-9-]*$ ]] || usage +parse_opts "$@" +COPIES_DIR="$COPIES" +load_config +DATA="$MOSAIC_DATA_ROOT" +STATE="$DATA/state" +POINTER="$STATE/stack-active.json" +LOG="$STATE/stack-activation-log.jsonl" +STORE="$DATA/bus/bus.sqlite" +BUS_UNIT="mosaic-bus@$BUSINESS.service" +FROM="" TO="" ACT="" RS="" POST="" BACKUP="" TAKEN="" RESTORED="" STEPS="" COUNT="" UNCLEAN="" FAIL="" +STORE_SCHEMA="" START_EPOCH="" START_MS=0 GATE="" GATE_WHY="" +P_COPY="" P_PM=0 P_SCHEMA="" P_CLAUDE="" P_CONSOLE=() +case "$OP" in + adopt) do_adopt ;; + activate) do_activate ;; + rollback) do_rollback ;; + status) do_status ;; +esac