fix(cli): upgrade and roll back Case 2 proxy hubs (#772)
This commit is contained in:
jason.woltje committed
1 parent
c098fa022c
commit
793f3b157d
5 files changed
+179
-19
No files matched your search
@@ -6016,3 +6016,4 @@ Owner directed static-only fallback after actual nativepriority drift: no new pr
|
||||
|
||||
2026-11-04T16:35:38Z | GPT-6.1 Sol implementation sub-agent, t3/bootstrap-hardening-fix-802-1 | #802 bootstrap hardening fix round 1 before/after | Before: fast-forwarded the assigned worktree to reviewed 52b283c8a; C1 foreign working-repo retry adoption/internal object links, C2 stale records and intent commits, C3 broker serving before the activate receipt. After: a first-attempt foreign repository gets a durable refused working receipt; recovery requires this plan's ownership claim, created exclusively inside the init directory before rename, as well as the existing ref checks. Every working-repository Git child checks for internal links and alternates; bootstrap also rechecks between its recovery reads. The existing kill-and-adopt recovery remains green. Organization holdAsync keeps the existing on-disk lock through asynchronous effects and releases it on success or failure; records initialization, intent publication/add/commit and intent rollback stay protected, while the bootstrap's exact-add mapping keeps its own transaction. The complete marker is verified and the activate receipt commits before the broker reload; runtime no longer overrides pending admission, and receipt I/O failure stays fenced until a fault-free retry. RED on 52b283c8a with final tests overlaid: 79 existing tests pass and all 16 new regressions fail as expected across request-bootstrap, project-bootstrap and working (95 total, zero skips); all five supplied reproducers were ported with corrected assertions. Final gates, all zero failures/skips/cancellations: focused 103/103 (baseline 90), bus 1499/1499 (1488), business 307/307 (305), project+seat 274/274 (271), launcher+stack-release including external-release cases 286/286 (286). Gate 5 uses node --test packages/cli/tests/launcher.test.mjs packages/cli/tests/stack-release*.test.mjs because these are CLI test files, not separate packages. Corrections: an initial gate-5 command named nonexistent package directories and ran no tests; an intermediate bus run failed the newly added per-child C1 regression, also reproduced on the base and fixed before the final green rerun. An earlier candidate gate-5 run was superseded and only its own descendant processes stopped; the final run completed. No preexisting test failure was found. Evidence: /mnt/storage/scratch/tmp/qRkZh1/ (before-1..4.log, before-5-cli.log, after-1..5.log, red-final-snapshot.log, verified-files.sha256). Provider interruption recovered from complete logs and matching source/test hashes. The queue refuses linked worktrees, so the explicit controller assignment was followed without changing the canonical queue. One local commit only; no push, live-instance action, role/contract/migration edit or residual C1/C2/C3 finding.
|
||||
2026-11-04T04:09:35Z | Codex implementation sub-agent, t3/lifecycle-prd-approval-activation | #797 / Gitea #1839 lifecycle Task 4 core | Before: parent 6dd4f92b0 has intake/bootstrap and schema 3p, with no exact product PRD activation authority. After: bounded committed candidate reads (64 requirements, 32 KiB PRD, 16 KiB requirement text, 256 KiB aggregate), strict records, server-derived blob IDs, selected-path HEAD equality and current owner spec; PM-only record.write submission raises the existing nonblocking prd.approve action. Autonomous routing requires a distinct authorized technical arbiter instance and definition; unusable/self routing records setup waits. Guided approval uses the same action without a routing citation, while retaining its spec/mode binding. Host-only catalog activation rereads candidate/spec/mode/current role authority and the exact approving resolution, independent of run liveness; snapshot, activation event, continuation and receipt share one SQLite transaction, with deterministic idempotent replay. Broker.requirementsFor keeps the agreed authenticated consumer signature; no tasks are manufactured. Mode/spec changes invalidate admissions and revoke snapshots; owner-only prd.revoke.console is revision/op checked and replayable, and request.draft.console reads only the bound committed candidate. Runtime recovers from journal/records alone and drains queued lifecycle work on close. New append-only migration 3p to 3q widens request event kinds, preserving rows, sequences and all guards; current version/fixture pins updated, table/guard counts stay 30/114. RED on 6dd4f92b0 with the final new tests: 0/27 pass, 27 expected missing-feature failures (baseline-new-tests-red.log). Corrections: the first fixture tried to initialize an already bootstrapped records repo, fixed before feature RED; journal composition initially attempted nested transactions, now composed inside one host transaction; new automatic closure continuation needed shutdown draining; one revoke assertion needed awaited rejection. First full bus run 1496/1498: the old two-verb whitelist pin was updated to the authorized lifecycle verbs, and unchanged #656 FIFO child measured niceness 16 rather than 19. Both files pass 38/38 on exact parent 6dd4f92b0 and on this branch; the FIFO failure did not reproduce in isolation. Final GREEN: new suites 27/27, requested kept-green trio 21/21, bus 1498/1498, project 50/50, business 305/305, all zero skips; import-closure checks 3/3, git diff --check clean. Full logs under /mnt/storage/scratch/tmp/wZ4am1 (focused-final.log, kept-green-final.log, bus-final.log, project.log, business.log; earlier runs retained). Boundaries: Console HTTP/browser files, revoke route and request-panel UI deferred until Task 3 integration; tracker/task consumer wiring belongs to Task 5; built-in legacy binding is deliberately unset here. The Stack PRD amendment remains proposed. Queue command refused this linked worktree, so queue movement stays with the canonical coordinator. One local commit authorized, no push, merge, deployment, live instance changes, role/contract edits or secret access.
|
||||
2026-11-04T17:07:37Z | Codex implementation sub-agent, t3/placeholder | #772 / row 334 F2 Case 2 proxy upgrade and rollback | Before: c098fa022 rejects the bare --proxy args line; read_old and stop/recovery cover only bus and Console, so a proxy hub cannot upgrade or roll back. After: accept only the bare flag, require all selected installed units to name one copy, refuse missing/linked/malformed/disagreeing proxy units and an installed proxy without its flag before stopping anything; stop Console, bus, proxy, wait for each, and preserve --proxy through both new and old renders. A2f starts proxy first and refuses a failed proxy restart with all three stopped; all rollback-failed cleanup includes the proxy. Header exit-code meanings and the ordinary two-unit path remain unchanged. Section 7 documents the flag and lifecycle. Nine added tests cover upgrade, explicit rollback, failed-upgrade restoration of all three unit files, A2f recovery/start failure, final stopped cleanup, no-proxy compatibility, malformed args, installed-unit refusals, and the real factory renderer re-pin/rollback. RED: focused #772 3 pass/5 fail/0 skip on the base, all five failures at args-unreadable --proxy. Corrections: first GREEN candidate 7/1/0 exposed the render stand-in using a different checkout path; a --checkout override was unsupported (6/3/0), so the stand-in now runs from the same fixture clone as setup; final focused 9/0/0. Required final gates (pass/fail/skip): node --test packages/cli/tests/stack-upgrade.test.mjs 34/0/0; node --test packages/cli/tests/case1-accept.test.mjs 53/0/0; node --test packages/cli/tests/case2-install.test.mjs 28/0/0; node --test packages/cli/tests/case2-unit.test.mjs 88/0/0; bash scripts/test-case2.sh 21/0/0. The four matching test files were found with git grep. Initial case2-install 6/22/0 lacked node_modules/yaml; initial scratch acceptance 13/8/0 lacked the installed dependency tree and passed after lockfile-only npm ci --ignore-scripts with distinct empty user/global configs and a scratch cache. The first npm setup attempt refused duplicate /dev/null config paths before installation. Every test invocation set TMPDIR with mktemp -d /mnt/storage/scratch/tmp/XXXXXX. bash -n and git diff --check passed. Logs: /mnt/storage/scratch/tmp/JBjOT3/ (RED/intermediate/focused and all required gate logs, npm setup, verified-source.sha256). Queue next refused the linked worktree; canonical queue movement remains with the coordinator. Fixture systemctl/sudo and synthetic configs only; no live host/unit, real credentials, ~/.config/mosaic-*, ~/.mosaic or /srv/mosaic access. One local commit authorized; no push, merge or deployment. No remaining code limitation; live .53 acceptance is outside this assignment.
|
||||
@@ -2004,3 +2004,4 @@ are never rewritten or removed; corrections are new entries.
|
||||
2026-11-04 | Claude Sonnet 5.5 implementation sub-agent, t3/harden-project-bootstrap-executor | #802 bootstrap executor hardening | N1 working path confined before and inside the helper, I1 guarded() makes the freshness check and every protected write one synchronous run under the organization lock, I2 complete marker published before activation and readiness needs marker plus committed receipt at boot, reload and admission. Brief trio 90/90, bus 1488/1488, business 305/305, project+seat 271/271, launcher+stack-release 261/261, zero skips; no schema change; local commit only, no push.
|
||||
2026-11-04 | GPT-6.1 Sol implementation sub-agent, t3/bootstrap-hardening-fix-802-1 | #802 bootstrap hardening fix round 1 | C1 permanent foreign-repo refusal, exclusive ownership claim and per-child internal-link/alternates checks; C2 async-held organization lock through records init, intent commit and rollback; C3 committed activate receipt before broker reload, fenced receipt failure and successful retry. All 16 added regressions fail on 52b283c8a; final focused 103/103, bus 1499/1499, business 307/307, project+seat 274/274, launcher+release including external cases 286/286, zero skips. Logs /mnt/storage/scratch/tmp/qRkZh1/; one local commit, no push, no role/contract/migration/live changes.
|
||||
2026-11-04 | Codex implementation sub-agent, t3/lifecycle-prd-approval-activation | #797 / Gitea #1839 lifecycle Task 4 core | Exact committed product PRD approval/activation, authenticated requirementsFor, owner revoke/mode/draft broker controls, atomic append-only snapshots/receipts and shutdown-safe recovery; migration 3p to 3q. New 27/27, kept-green 21/21, bus 1498/1498, project 50/50, business 305/305, zero skips; logs /mnt/storage/scratch/tmp/wZ4am1. Console HTTP/browser slice deferred for Task 3 integration. Local commit only, no push.
|
||||
2026-11-04 | Codex implementation sub-agent, t3/placeholder | #772 / row 334 F2 Case 2 proxy upgrade and rollback | Proxy args, three-unit copy checks and ordered stop/re-render/start/recovery; run-sheet section 7 updated. Focused 9/9; required upgrade 34/34, case1-accept 53/53, case2-install 28/28, case2-unit 88/88, scratch acceptance 21/21, zero skips; lockfile dependencies restored after initial setup failures. Evidence /mnt/storage/scratch/tmp/JBjOT3/. Linked-worktree queue refusal left to coordinator; local commit only, no push or live/secret access.
|
||||
@@ -377,9 +377,15 @@ place the correct project file and repeat the check above before continuing (Duc
|
||||
line holding `--`. Root writes it at the unit render, from the values the go names. The stack's user's
|
||||
install never writes it: that user can't write `/etc`, and root reads nothing that user wrote. It must be root's, with no group
|
||||
or other write (`chmod 0644` or tighter). Before `--` it takes only `--user`, `--home`, `--business`,
|
||||
`--copy` (exactly once), `--node`, `--profile`, `--claude` and `--pm`. `stack-upgrade.sh` changes only the
|
||||
`--copy` (exactly once), `--node`, `--profile`, `--claude`, `--pm` and `--proxy`. `stack-upgrade.sh` changes only the
|
||||
`--copy` line, and only after confirm passes.
|
||||
|
||||
For a hub installed with `--proxy`, keep that bare flag on its own line before `--`; it takes no value.
|
||||
Upgrade and rollback require the proxy, bus and Console units to name the same copy. They stop the Console,
|
||||
then the bus, then the proxy, re-render all three onto the selected copy, and start the proxy, then the bus,
|
||||
then the Console. Recovery after a failed upgrade restores all three to the old copy. A missing or
|
||||
disagreeing proxy unit, or an installed proxy without `--proxy` in the args file, refuses before anything stops.
|
||||
|
||||
For the `library` profile, the serve argv after `--` keeps the proxy-preview trio and required proof pair
|
||||
but omits `--send`. That Console is read-only until Case 2 gains real OIDC in #412; `case2-unit.sh`
|
||||
explicitly refuses `--send`. Proxy preview and proof-protected reads remain available. The `factory`
|
||||
|
||||
@@ -46,8 +46,8 @@ function fakeSudo(K){
|
||||
}
|
||||
|
||||
// case2-unit.sh: logs its argv, then installs the copy's units as the fixture's user units (its
|
||||
// WorkingDirectory under the stack's copies) and starts the bus, then the Console, as case2-unit.sh
|
||||
// does. faults/render-fails-<copy> exits 1 first; faults/render-probe-<copy> (one-shot) first runs
|
||||
// WorkingDirectory under the stack's copies) and starts the optional proxy, bus, then Console,
|
||||
// as case2-unit.sh does. faults/render-fails-<copy> exits 1 first; faults/render-probe-<copy> (one-shot) first runs
|
||||
// probe.sh, a second driver, and records its exit and stderr.
|
||||
function fakeRender(K){
|
||||
const fs=process.getBuiltinModule('node:fs'),path=process.getBuiltinModule('node:path'),cp=process.getBuiltinModule('node:child_process');
|
||||
@@ -61,10 +61,15 @@ function fakeRender(K){
|
||||
fs.writeFileSync(path.join(sd,'probe.json'),JSON.stringify({code:p.status,err:p.stderr}));
|
||||
}
|
||||
if(fs.existsSync(f(`render-fails-${copy}`))){console.error(`fixture render: ${copy} fails`);process.exit(1)}
|
||||
const sh=(n,...x)=>cp.execFileSync('bash',[path.join(K.repo,'scripts',n),...x],{stdio:['ignore','ignore','inherit']});
|
||||
const sh=(n,...x)=>cp.execFileSync('bash',[path.join(get('--home'),'src','mosaic-stack','scripts',n),...x],{stdio:['ignore','ignore','inherit']});
|
||||
sh('bus-service.sh','install','--copy',copy,'--copies',copies,...(pre.includes('--pm')?['--pm']:[]),'--claude',get('--claude'));
|
||||
sh('webui-service.sh','install','--copy',copy,'--copies',copies,...serve);
|
||||
for(const u of [`mosaic-bus@${get('--business')}.service`,'mosaic-webui.service'])cp.execFileSync(path.join(K.sbin,'systemctl'),['start',u],{stdio:'inherit'});
|
||||
const names=[`mosaic-bus@${get('--business')}.service`,'mosaic-webui.service'];
|
||||
if(pre.includes('--proxy')){
|
||||
fs.writeFileSync(path.join(process.env.XDG_CONFIG_HOME,'systemd','user','mosaic-proxy.service'),`[Service]\nWorkingDirectory=${copies}/${copy}\nExecStart=${get('--node')} ${copies}/${copy}/packages/model-proxy/src/run.mjs\n`);
|
||||
names.unshift('mosaic-proxy.service');
|
||||
}
|
||||
for(const u of names)cp.execFileSync(path.join(K.sbin,'systemctl'),['start',u],{stdio:'inherit'});
|
||||
}
|
||||
|
||||
// Root's systemctl: the fixture's, with systemd's no-op start of an active unit, and a one-shot
|
||||
@@ -75,6 +80,7 @@ function fakeSystemctl(K){
|
||||
fs.appendFileSync(path.join(sd,'root-systemctl.log'),a.join(' ')+'\n');
|
||||
const noop=path.join(sd,'faults',`stop-noop-${a[1]}`);
|
||||
if(a[0]==='stop'&&fs.existsSync(noop)){fs.rmSync(noop);process.exit(0)}
|
||||
if(a[0]==='start'&&fs.existsSync(path.join(sd,'faults',`root-start-fails-${a[1]}`)))process.exit(1);
|
||||
if(a[0]==='start'){let s={};try{s=JSON.parse(fs.readFileSync(path.join(sd,'units.json'),'utf8'))}catch{}if((s[a[1]]||{}).state==='active')process.exit(0)}
|
||||
const r=cp.spawnSync(path.join(K.sbin,'systemctl'),a,{stdio:'inherit'});process.exit(r.status??1);
|
||||
}
|
||||
@@ -141,7 +147,7 @@ const arm=(e,key,text)=>writeFileSync(join(e.sd,'arm',key),text);
|
||||
|
||||
// The step log: every line `<time> <step> rc=N ...` or the result line, root:root 0640 (here the
|
||||
// test user's), and nothing the stack's user printed. Returns the lines without the time.
|
||||
const STEP=/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ (start|fetch|ff|copy|hostcheck|units|plan|rehearse|rehearse-store|rehearsal-skipped|stop-console|stop-bus|wait-down|stage|render|confirm|args|revert|render-old|confirm-rollback|gate|start-bus|start-console|stop-units) rc=\d+ |^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ result exit=[0-35] [a-z-]+ /;
|
||||
const STEP=/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ (start|fetch|ff|copy|hostcheck|units|plan|rehearse|rehearse-store|rehearsal-skipped|stop-console|stop-bus|stop-proxy|wait-down|stage|render|confirm|args|revert|render-old|confirm-rollback|gate|start-proxy|start-bus|start-console|stop-units) rc=\d+ |^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\dZ result exit=[0-35] [a-z-]+ /;
|
||||
function slog(e){
|
||||
const l=lines(e.slog);
|
||||
for(const x of l){assert.match(x,STEP);assert.ok(!/stack-release:|fixture rehearse|PASS |FAIL |REHEARSAL/.test(x),x)}
|
||||
@@ -163,6 +169,112 @@ const PRE=['start','fetch','ff','copy','hostcheck','units','plan','rehearse','re
|
||||
const A2=['stop-console','stop-bus','wait-down','wait-down'];
|
||||
const R=[...A2,'revert','render-old','confirm-rollback'];
|
||||
|
||||
// #772: the proxy is another selected unit, including A2f and failed-upgrade recovery.
|
||||
// The real renderer's installed-unit checks are exercised separately below; this fixture keeps
|
||||
// the existing high-port health server and user-unit stand-ins for stack-release's real gate.
|
||||
const PROXY='mosaic-proxy.service';
|
||||
const P2=['stop-console','stop-bus','stop-proxy','wait-down','wait-down','wait-down'];
|
||||
async function proxyHub(t){
|
||||
const e=await useed(t);
|
||||
e.argsText=e.argsText.replace('--\n','--profile\nfactory\n--proxy\n--\n');writeFileSync(e.args,e.argsText);
|
||||
writeFileSync(join(e.units,PROXY),`[Service]\nWorkingDirectory=${e.hc}/${B}\nExecStart=${NODE} ${e.hc}/${B}/packages/model-proxy/src/run.mjs\n`);
|
||||
sysctl(e,'start',PROXY);
|
||||
return e;
|
||||
}
|
||||
const transitions=(e,from=0)=>lines(join(e.sd,'calls.log')).slice(from).filter((x)=>/^(stop|start) /.test(x));
|
||||
const proxyLive=(e,copy)=>{
|
||||
assert.deepEqual(live(e),['active','active',copy,copy]);
|
||||
assert.deepEqual([units(e)[PROXY].state,units(e)[PROXY].wd],['active',`${e.hc}/${copy}`]);
|
||||
assert.match(readFileSync(join(e.units,PROXY),'utf8'),new RegExp(`^WorkingDirectory=${e.hc}/${copy}$`,'m'));
|
||||
};
|
||||
const P_CYCLE=['stop mosaic-webui.service','stop [email protected]','stop mosaic-proxy.service','start mosaic-proxy.service','start [email protected]','start mosaic-webui.service'];
|
||||
|
||||
test('#772: a --proxy hub stops Console, bus, proxy, upgrades all three and starts proxy first',async(t)=>{
|
||||
const e=await proxyHub(t),n=lines(join(e.sd,'calls.log')).length;
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,0,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...P2,'stage','render','confirm','args','result']);
|
||||
assert.deepEqual(transitions(e,n),P_CYCLE);proxyLive(e,C1);
|
||||
assert.equal(readFileSync(e.args,'utf8'),e.argsText.replace(`--copy\n${B}\n`,`--copy\n${C1}\n`));
|
||||
assert.ok(JSON.parse(lines(join(e.sd,'render.log'))[0]).includes('--proxy'));
|
||||
const m=lines(join(e.sd,'calls.log')).length;
|
||||
const rb=await drive(e,C1,'rollback','--instance','dev','--to',B,...TO);assert.equal(rb.code,0,rb.all);
|
||||
assert.deepEqual(transitions(e,m),P_CYCLE);proxyLive(e,B);argsKept(e);
|
||||
});
|
||||
|
||||
test('#772: a failed upgrade restores and restarts all three units on the old copy',async(t)=>{
|
||||
const e=await proxyHub(t),n=lines(join(e.sd,'calls.log')).length;
|
||||
const before=Object.fromEntries([e.tmpl,e.con,PROXY].map((u)=>[u,readFileSync(join(e.units,u),'utf8')]));
|
||||
fault(e,`healthz-${C1}`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...P2,'stage','render','confirm',...P2,'revert','render-old','confirm-rollback','result']);
|
||||
assert.deepEqual(transitions(e,n),[...P_CYCLE,...P_CYCLE]);proxyLive(e,B);argsKept(e);
|
||||
for(const [u,text] of Object.entries(before))assert.equal(readFileSync(join(e.units,u),'utf8'),text,u);
|
||||
assert.equal(final(e,lastAct(e)).rollback,'ok');
|
||||
});
|
||||
|
||||
test('#772: A2f restarts the proxy before the old bus and Console after a refused stage',async(t)=>{
|
||||
const e=await proxyHub(t),n=lines(join(e.sd,'calls.log')).length;
|
||||
arm(e,'activate.sh',`: > ${join(e.sd,'faults','no-fragment')}\n`);arm(e,'gate.sh',`rm ${join(e.sd,'faults','no-fragment')}\n`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,1,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...P2,'stage','start-proxy','start-bus','start-console','gate','result']);
|
||||
assert.deepEqual(transitions(e,n),P_CYCLE);proxyLive(e,B);argsKept(e);
|
||||
});
|
||||
|
||||
test('#772: failed rollback leaves all three selected units stopped',async(t)=>{
|
||||
const e=await proxyHub(t);
|
||||
fault(e,`render-fails-${C1}`);fault(e,`render-fails-${B}`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,2,r.all);
|
||||
assert.deepEqual([units(e)[e.con].state,units(e)[e.bus].state,units(e)[PROXY].state],['inactive','inactive','inactive']);
|
||||
assert.deepEqual(transitions(e).slice(-3),P_CYCLE.slice(0,3));argsKept(e);
|
||||
});
|
||||
|
||||
test('#772: a failed proxy restart in A2f stops all three and never starts its dependents',async(t)=>{
|
||||
const e=await proxyHub(t),n=lines(join(e.sd,'calls.log')).length;
|
||||
arm(e,'activate.sh',`: > ${join(e.sd,'faults','no-fragment')}\n`);
|
||||
fault(e,`root-start-fails-${PROXY}`);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,2,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...P2,'stage','start-proxy','stop-units','result']);
|
||||
assert.deepEqual(transitions(e,n),[...P_CYCLE.slice(0,3),...P_CYCLE.slice(0,3)]);
|
||||
assert.deepEqual([units(e)[e.con].state,units(e)[e.bus].state,units(e)[PROXY].state],['inactive','inactive','inactive']);argsKept(e);
|
||||
});
|
||||
|
||||
test('#772: a hub without --proxy keeps its two-unit upgrade and never creates or calls a proxy unit',async(t)=>{
|
||||
const e=await useed(t),n=lines(join(e.sd,'calls.log')).length;
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,0,r.all);
|
||||
assert.deepEqual(names(e),[...PRE,...A2,'stage','render','confirm','args','result']);
|
||||
assert.deepEqual(transitions(e,n),['stop mosaic-webui.service','stop [email protected]','start [email protected]','start mosaic-webui.service']);
|
||||
assert.ok(!existsSync(join(e.units,PROXY)));assert.ok(!lines(join(e.sd,'calls.log')).some((x)=>x.includes(PROXY)));
|
||||
assert.deepEqual(live(e),['active','active',C1,C1]);
|
||||
});
|
||||
|
||||
test('#772: malformed proxy args refuse before any unit or user command, for upgrade and rollback',async(t)=>{
|
||||
const e=await proxyHub(t),n=lines(join(e.sd,'calls.log')).length;
|
||||
for(const bad of ['--proxy=true','--proxy true','--proxy\ntrue'])for(const op of ['upgrade','rollback']){
|
||||
const text=e.argsText.replace('--proxy\n',bad+'\n');writeFileSync(e.args,text);
|
||||
const r=op==='upgrade'?await upgrade(e,C1):await drive(e,B,'rollback','--instance','dev','--to',B,...TO);
|
||||
assert.equal(r.code,3,r.all);assert.match(result(e),/^result exit=3 args-unreadable /);
|
||||
assert.equal(readFileSync(e.args,'utf8'),text);assert.deepEqual(readdirSync(e.etc),['dev.args']);
|
||||
}
|
||||
assert.deepEqual(transitions(e,n),[]);assert.ok(!existsSync(join(e.sd,'sudo.log')));assert.ok(!existsSync(join(e.sd,'render.log')));proxyLive(e,B);
|
||||
});
|
||||
|
||||
test('#772: an absent, linked, malformed or disagreeing proxy unit refuses before stopping anything',async(t)=>{
|
||||
const e=await proxyHub(t),file=join(e.units,PROXY),text=readFileSync(file,'utf8'),n=lines(join(e.sd,'calls.log')).length;
|
||||
for(const [why,put,code] of [
|
||||
['absent',()=>{},'units-unreadable'],
|
||||
['linked',()=>symlinkSync(join(e.units,e.con),file),'units-unreadable'],
|
||||
['malformed',()=>writeFileSync(file,'WorkingDirectory=/foreign/copy\n'),'units-unreadable'],
|
||||
['disagreeing',()=>writeFileSync(file,text.replaceAll(B,C1)),'units-disagree'],
|
||||
]){
|
||||
rmSync(file,{force:true});put();
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,`${why}: ${r.all}`);assert.match(result(e),new RegExp(`^result exit=3 ${code} `),why);argsKept(e);
|
||||
}
|
||||
writeFileSync(file,text);
|
||||
e.argsText=e.argsText.replace('--proxy\n','');writeFileSync(e.args,e.argsText);
|
||||
const r=await upgrade(e,C1);assert.equal(r.code,3,r.all);assert.match(result(e),/^result exit=3 units-unreadable /);argsKept(e);
|
||||
assert.deepEqual(transitions(e,n),[]);proxyLive(e,B);
|
||||
});
|
||||
|
||||
// --- r6 sequences 1-8, through the driver ---
|
||||
|
||||
test('row 233: sequence 1: confirm fails healthz, the driver reverts, renders the old copy and confirm --rollback --cause A5 passes: exit 1',async(t)=>{
|
||||
@@ -517,6 +629,25 @@ function case2Host(e,dir){
|
||||
const use=(text)=>{writeFileSync(e.args,text);chmodSync(e.args,0o640);e.argsText=text;e.denv.STACK_UPGRADE_UNIT_DIR=dir;delete e.denv.STACK_UPGRADE_RENDER;Object.assign(e.denv,env)};
|
||||
return {home,dir,args,render,legacy,bytes,syslog,use,con:join(dir,'mosaic-webui.service')};
|
||||
}
|
||||
|
||||
test('#772: the real Case 2 renderer re-pins and rolls back all three factory units with the saved --proxy args',async(t)=>{
|
||||
const e=await useed(t),h=case2Host(e,join(e.root,'case2','units')),script=join(checkout(C1),'scripts','case2-unit.sh');
|
||||
const args=(copy)=>['--user','fixture','--home',h.home,'--business','acme','--copy',copy,'--node',NODE,'--profile','factory','--proxy','--','--port','443','--business','acme'];
|
||||
const first=h.render(script,args(B));assert.equal(first.code,0,first.out+first.err);
|
||||
const before=h.bytes();
|
||||
const next=h.render(script,args(C1));assert.equal(next.code,0,next.out+next.err);
|
||||
for(const [u,text] of Object.entries(h.bytes()))assert.ok(text.includes(`WorkingDirectory=${e.hc}/${C1}\n`),u);
|
||||
assert.ok(h.bytes()[PROXY].includes(`ExecStart=${NODE} ${e.hc}/${C1}/packages/model-proxy/src/run.mjs\n`));
|
||||
const changed=h.bytes(),n=h.syslog().length;
|
||||
const missing=h.render(script,args(C1).filter((x)=>x!=='--proxy'));assert.equal(missing.code,1,missing.out+missing.err);
|
||||
assert.deepEqual(h.bytes(),changed);assert.equal(h.syslog().length,n);
|
||||
const back=h.render(script,args(B));assert.equal(back.code,0,back.out+back.err);assert.deepEqual(h.bytes(),before);
|
||||
assert.deepEqual(h.syslog().filter((x)=>x.startsWith('start ')),[
|
||||
'start mosaic-proxy.service','start [email protected]','start mosaic-webui.service',
|
||||
'start mosaic-proxy.service','start [email protected]','start mosaic-webui.service',
|
||||
'start mosaic-proxy.service','start [email protected]','start mosaic-webui.service',
|
||||
]);
|
||||
});
|
||||
const legacyUnit=(f)=>`${f} runs serve with --send, so it was rendered before #385, and stack-upgrade.sh cannot migrate it to the read-only library Console: the units are left running; stop and report (docs/guides/host-run-sheet.md section 7) (nothing was stopped)`;
|
||||
test('#385 R3-F1: an installed pre-#385 library Console unit with --send refuses console-unit-legacy before anything stops, for upgrade and rollback',async(t)=>{
|
||||
const e=await useed(t),h=case2Host(e,join(e.root,'case2','units'));
|
||||
|
||||
+34
-13
@@ -120,7 +120,7 @@ logl "start rc=0 argv:$(printf ' %q' "$0" "${ARGV[@]}")"
|
||||
# --- The args file and the stack's user ---
|
||||
|
||||
# I2's file: one case2-unit.sh argument per line, owned by root with no group or other write. Sets
|
||||
# ARGS (the lines), S_USER, S_HOME, BUSINESS, NODE, ARGS_COPY and ARGS_PROFILE, and the paths derived
|
||||
# ARGS (the lines), S_USER, S_HOME, BUSINESS, NODE, ARGS_COPY, ARGS_PROFILE and ARGS_PROXY, and the paths derived
|
||||
# from them.
|
||||
# #385 R2-F1: base case2-unit.sh required --send for the library profile (its default, and the last
|
||||
# --profile wins, as there), and the current one refuses it. A2 would stop both units, and A4 and
|
||||
@@ -132,11 +132,12 @@ read_args() {
|
||||
st="$(stat -c '%u %a' "$f")" || refuse args-unreadable "cannot stat $f"
|
||||
[ "${st% *}" = "$(id -u)" ] && (( (8#${st#* } & 8#022) == 0 )) || refuse args-unreadable "$f must be $(id -un)'s with no group or other write (it is uid ${st% *}, mode ${st#* })"
|
||||
mapfile -t ARGS < "$f" || refuse args-unreadable "cannot read $f"
|
||||
S_USER="" S_HOME="" BUSINESS="" NODE="" ARGS_COPY=""
|
||||
S_USER="" S_HOME="" BUSINESS="" NODE="" ARGS_COPY="" ARGS_PROXY=0
|
||||
for ((i = 0; i < ${#ARGS[@]}; i++)); do
|
||||
case "${ARGS[i]}" in
|
||||
--) break ;;
|
||||
--pm) ;;
|
||||
--proxy) ARGS_PROXY=1 ;;
|
||||
--user|--home|--business|--copy|--node|--profile|--claude)
|
||||
v="${ARGS[i + 1]:-}"
|
||||
case "${ARGS[i]}" in
|
||||
@@ -162,7 +163,10 @@ read_args() {
|
||||
COPIES_DIR="$S_HOME/.local/share/mosaic-stack/copies"
|
||||
CLONE="$S_HOME/src/mosaic-stack"
|
||||
SR="$CLONE/scripts/stack-release.sh"
|
||||
BUS_UNIT="mosaic-bus@$BUSINESS.service" CON_UNIT=mosaic-webui.service
|
||||
BUS_UNIT="mosaic-bus@$BUSINESS.service" CON_UNIT=mosaic-webui.service PROXY_UNIT=mosaic-proxy.service
|
||||
STOP_UNITS=("$CON_UNIT" "$BUS_UNIT")
|
||||
STOPPED_UNITS="both units"
|
||||
if [ "$ARGS_PROXY" = 1 ]; then STOP_UNITS+=("$PROXY_UNIT"); STOPPED_UNITS="all three units"; fi
|
||||
# The stack's user's steps: its own HOME and default config, the pinned node first on PATH.
|
||||
AS_USER=("$SUDO" -u "$S_USER" -H env "PATH=$(dirname "$NODE"):/usr/local/bin:/usr/bin:/bin")
|
||||
}
|
||||
@@ -214,17 +218,25 @@ drop_args() { rm -f "${NEW_ARGS:-}"; }
|
||||
|
||||
# --- A1 ---
|
||||
|
||||
# OLD: the copy both installed units run, from their WorkingDirectory under the stack's copies.
|
||||
# OLD: the copy all selected installed units run, from their WorkingDirectory under the stack's copies.
|
||||
read_old() {
|
||||
local f ids=() id n
|
||||
for f in "$UNIT_DIR/[email protected]" "$UNIT_DIR/$CON_UNIT"; do
|
||||
local files=("$UNIT_DIR/[email protected]" "$UNIT_DIR/$CON_UNIT")
|
||||
if [ "$ARGS_PROXY" = 1 ]; then files+=("$UNIT_DIR/$PROXY_UNIT")
|
||||
else
|
||||
[ ! -e "$UNIT_DIR/$PROXY_UNIT" ] && [ ! -L "$UNIT_DIR/$PROXY_UNIT" ] ||
|
||||
refuse units-unreadable "installed $UNIT_DIR/$PROXY_UNIT requires --proxy"
|
||||
fi
|
||||
for f in "${files[@]}"; do
|
||||
[ -f "$f" ] && [ ! -L "$f" ] || refuse units-unreadable "no regular file $f"
|
||||
n="$(grep -c '^WorkingDirectory=' "$f")"
|
||||
id="$(awk -v c="WorkingDirectory=$COPIES_DIR/" 'index($0, c) == 1 { print substr($0, length(c) + 1) }' "$f")"
|
||||
[ "$n" = 1 ] && [[ "$id" =~ ^[0-9a-f]{40}$ ]] || refuse units-unreadable "$f names no copy under $COPIES_DIR"
|
||||
ids+=("$id")
|
||||
done
|
||||
[ "${ids[0]}" = "${ids[1]}" ] || refuse units-disagree "$UNIT_DIR's units name ${ids[0]} and ${ids[1]}"
|
||||
for id in "${ids[@]}"; do
|
||||
[ "${ids[0]}" = "$id" ] || refuse units-disagree "$UNIT_DIR's units name ${ids[0]} and $id"
|
||||
done
|
||||
OLD="${ids[0]}"
|
||||
logl "units rc=0 old: $OLD"
|
||||
[ "$ARGS_COPY" = "$OLD" ] || refuse args-differ "$ETC/$INSTANCE.args names $ARGS_COPY and the units name $OLD"
|
||||
@@ -249,7 +261,8 @@ stop_wait() {
|
||||
local u s deadline=$((SECONDS + TIMEOUT)) rc=0
|
||||
step stop-console "$SYSTEMCTL" stop "$CON_UNIT" || rc=1
|
||||
step stop-bus "$SYSTEMCTL" stop "$BUS_UNIT" || rc=1
|
||||
for u in "$CON_UNIT" "$BUS_UNIT"; do
|
||||
if [ "$ARGS_PROXY" = 1 ]; then step stop-proxy "$SYSTEMCTL" stop "$PROXY_UNIT" || rc=1; fi
|
||||
for u in "${STOP_UNITS[@]}"; do
|
||||
while :; do
|
||||
s="$("$SYSTEMCTL" is-active "$u" 9>&- 2>/dev/null)"
|
||||
case "$s" in inactive|failed) break ;; esac
|
||||
@@ -260,7 +273,11 @@ stop_wait() {
|
||||
done
|
||||
return "$rc"
|
||||
}
|
||||
stop_units() { "$SYSTEMCTL" stop "$CON_UNIT" 9>&- || true; "$SYSTEMCTL" stop "$BUS_UNIT" 9>&- || true; logl "stop-units rc=0 units: $CON_UNIT $BUS_UNIT"; }
|
||||
stop_units() {
|
||||
local u
|
||||
for u in "${STOP_UNITS[@]}"; do "$SYSTEMCTL" stop "$u" 9>&- || true; done
|
||||
logl "stop-units rc=0 units: ${STOP_UNITS[*]}"
|
||||
}
|
||||
|
||||
# A2f: the old units back on the copy they never left, gated on a start at or after this second.
|
||||
# An A2 that timed out leaves a unit on its old start, which the gate refuses as stale-start.
|
||||
@@ -268,32 +285,36 @@ a2f() {
|
||||
local cause="$1" since
|
||||
drop_args
|
||||
since="$(date -u +%s)"
|
||||
if [ "$ARGS_PROXY" = 1 ]; then
|
||||
step start-proxy "$SYSTEMCTL" start "$PROXY_UNIT" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the proxy did not start for $OLD; $STOPPED_UNITS are stopped"; }
|
||||
fi
|
||||
step start-bus "$SYSTEMCTL" start "$BUS_UNIT"
|
||||
step start-console "$SYSTEMCTL" start "$CON_UNIT"
|
||||
if step gate "${AS_USER[@]}" "$SR" gate "$OLD" "$BUSINESS" --units external --copies "$COPIES_DIR" --since "$since" --timeout "$TIMEOUT"; then
|
||||
finish 1 failed "$cause failed; $OLD is running again and passed the gate"
|
||||
fi
|
||||
stop_units
|
||||
finish 2 rollback-failed "$cause failed and $OLD failed its gate after the restart; both units are stopped"
|
||||
finish 2 rollback-failed "$cause failed and $OLD failed its gate after the restart; $STOPPED_UNITS are stopped"
|
||||
}
|
||||
|
||||
# R1-R4: back to OLD after the stack's user staged $1. $2 is the cause: A3, A4 or A5.
|
||||
rollback_to_old() {
|
||||
local id="$1" cause="$2"
|
||||
drop_args
|
||||
stop_wait || { stop_units; finish 2 rollback-failed "$cause failed and the units did not stop for the rollback; both units are stopped"; }
|
||||
stop_wait || { stop_units; finish 2 rollback-failed "$cause failed and the units did not stop for the rollback; $STOPPED_UNITS are stopped"; }
|
||||
step revert "${AS_USER[@]}" "$SR" revert "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and revert failed; both units are stopped"; }
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and revert failed; $STOPPED_UNITS are stopped"; }
|
||||
args_with "$OLD"
|
||||
step render-old "$RENDER" "${RENDER_ARGS[@]}" ||
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the render of $OLD failed; both units are stopped"; }
|
||||
{ stop_units; finish 2 rollback-failed "$cause failed and the render of $OLD failed; $STOPPED_UNITS are stopped"; }
|
||||
# confirm --rollback exits 1 when OLD passed its gate and 2 when it failed; anything else is a
|
||||
# refusal, with OLD's gate never run.
|
||||
local rc=0
|
||||
step confirm-rollback "${AS_USER[@]}" "$SR" confirm "$id" "$BUSINESS" --units external --copies "$COPIES_DIR" --rollback "$OLD" --cause "$cause" --timeout "$TIMEOUT" || rc=$?
|
||||
[ "$rc" != 1 ] || finish 1 failed "$cause failed; rolled back to $OLD, which passed the gate"
|
||||
stop_units
|
||||
finish 2 rollback-failed "$cause failed and the rollback to $OLD did not pass (confirm --rollback exited $rc); both units are stopped"
|
||||
finish 2 rollback-failed "$cause failed and the rollback to $OLD did not pass (confirm --rollback exited $rc); $STOPPED_UNITS are stopped"
|
||||
}
|
||||
|
||||
# A2 to A5 for copy $1, once A1 and any rehearsal passed. $@ after it is the stack's user's op.
|
||||
|
||||
Reference in new issue
Block a user