test: clean proof temp roots and catch preparation signals (#1695)
This commit is contained in:
jason.woltje committed
1 parent
8ae9d8acd9
commit
b8461266cd
2 files changed
+165
-14
No files matched your search
@@ -0,0 +1,126 @@
|
||||
# Two-lane proof command and schedule
|
||||
|
||||
This evidence stays outside r5's 32 manifest paths. No proof or image build has
|
||||
run. The driver and example config are committed on herb/row197-proof, rooted
|
||||
at 9aa6d52d9. Sage runs the committed copy from their own checkout.
|
||||
|
||||
Driver: `agents/herb/work/queue-197/proof/run-proof.mjs`.
|
||||
The commit message to Sage carries its exact SHA-256. Check it with
|
||||
`git show <proof-commit>:agents/herb/work/queue-197/proof/run-proof.mjs | sha256sum`.
|
||||
|
||||
After Sage fills config.json from config.example.json and grants the whole block:
|
||||
|
||||
```sh
|
||||
MOSAIC_PROOF_GRANT=ACTUAL_SAGE_GRANT_ID node /SAGE_PROOF_CHECKOUT/agents/herb/work/queue-197/proof/run-proof.mjs /SAGE_OWNED_CONFIG/config.json
|
||||
```
|
||||
|
||||
The example config pins the launcher and candidate A to 197 r5:
|
||||
9aa6d52d9431efd215aa31d748d7e273eb321474. The launcher is a separate clean
|
||||
checkout owned by Sage, so later row 215 work cannot change it. Fill candidateB with
|
||||
Sage's approved combined 212+197 commit, imageReceipt with the granted pinned
|
||||
image-provenance receipt, and runRoot with a fresh private scratch path. Both
|
||||
candidates must carry identical lane tooling and image inputs and use the same
|
||||
pinned image ID. The driver refuses missing parameters, equal candidate SHAs,
|
||||
dirty launcher, changed tooling, mismatched image inputs or image ID.
|
||||
|
||||
D5 order is exactly two solo gates, then three pairs:
|
||||
|
||||
| Wave | Candidate A | Candidate B |
|
||||
| --- | --- | --- |
|
||||
| 1 | solo, lane a | idle |
|
||||
| 2 | idle | solo, lane b |
|
||||
| 3 | paired, lane a | paired, lane b |
|
||||
| 4 | paired, lane b | paired, lane a |
|
||||
| 5 | paired, lane a | paired, lane b |
|
||||
|
||||
Every invocation gets its own clean clone, output, short temp root and random
|
||||
Compose project. The wrapper discovers all package and shell steps, with two
|
||||
Node test-file workers per lane. The summary requires three overlaps, the lane
|
||||
swap, successful clean receipts and zero warm image builds. Raw logs and per-step
|
||||
receipts remain available. No failed gate or series is retried automatically.
|
||||
|
||||
A live monitor stops both gates on the first failure line, records lane, step,
|
||||
source file when the runner emits it, test name and preserved log, then enriches
|
||||
that report after shutdown. If the runner gives no source path, the report says
|
||||
unresolved and points to the log rather than guessing. It stops earlier than the
|
||||
two-failure limit. TERM/INT/HUP stop owned supervisors; cleanup waits for them.
|
||||
|
||||
Container and network names come from Compose's random recorded project. The
|
||||
project and service configuration have no fixed global names. Exact-label Docker
|
||||
event streams record create/destroy IDs and names, including --rm containers.
|
||||
An exit trap inventories names/labels, removes only exact-project objects with
|
||||
the approved cleanup helper, verifies emptiness and stops its own event streams.
|
||||
The driver's Docker inspection never reads Config.Env or credential contents, and never builds, pulls,
|
||||
retags or prunes. Missing resource-name evidence, an ended event stream, unknown
|
||||
ownership, leftover objects or quarantined lanes fail the proof. A SIGKILL cannot
|
||||
run an exit trap; durable wrapper receipts then require granted recovery.
|
||||
|
||||
Run this as its own whole-slot phase. A still-running legacy landing.mjs gate or
|
||||
gate.sh is LEGACY BUSY, including a parent process. The driver does not suppress
|
||||
that refusal. The ordinary landing gate must exit before this proof phase starts.
|
||||
|
||||
Planning runtime from sage-1837 is 12-15 minutes per wave, so reserve 60-75 minutes
|
||||
for all five waves, plus any separately granted image preparation. This is an
|
||||
estimate, not a measurement. Measured proof runtime is unavailable until this
|
||||
first run. Historical row 189 full gate was 659 seconds; the r4 focused 19-file
|
||||
run was 761.33 seconds and is not a full solo baseline. The resulting summary
|
||||
records actual per-gate times, pair makespans, slowdowns and total wall time.
|
||||
|
||||
No image provenance receipt or combined integration SHA is available to Herb yet.
|
||||
The driver remains unexecuted. Sage will read the exact hashed driver and request
|
||||
Webby's static review before the landing/proof grant.
|
||||
|
||||
Static node --check passed after sage-1839 single cleared. This is syntax-only;
|
||||
the driver has not run a proof or Docker command. Its runtime behavior awaits
|
||||
Sage and Webby review and the separately granted proof block.
|
||||
|
||||
## Driver review fixes, sage-1857
|
||||
|
||||
Webby's first review was CHANGES, B1, at bba870ee9. The revised driver takes
|
||||
tempRoot only from immutable start records for the plan's checkouts. After
|
||||
every owned gate exits, Docker cleanup and the final status/process checks
|
||||
run. The driver requires the exact short ga-/gb- scratch spelling, a matching
|
||||
lane, an owned real directory and no live owner, child group or process with
|
||||
the run paths in its argv before removal. It also requires a clean terminal
|
||||
receipt. A successful gate's root is removed recursively. A failed gate's
|
||||
root stays for diagnosis; an unverified cleanup also stays and fails the
|
||||
proof. docker-summary.json records removed, kept and error entries. Any
|
||||
cleanup error makes the driver fail. No unrelated scratch directory is read.
|
||||
|
||||
Signal handlers are installed before configuration and preparation. Clone
|
||||
preparation is inside the try/finally, and yields between synchronous clones
|
||||
so a pending signal stops it before another invocation. A preparation signal
|
||||
writes failure evidence and reaches the cleanup path. Partial clones remain
|
||||
under runRoot as permitted by Sage; they are never confused with lane roots.
|
||||
SIGKILL cannot be trapped and still needs granted recovery.
|
||||
|
||||
At 2026-10-21T08:26:41Z, Herb removed four earlier roots recorded in their
|
||||
own packet: ga-VPXhG7, gb-XeTAUS, ga-35Zo9Y and ga-R5IOT3. Each was an owned
|
||||
real directory with no process argv/cwd reference before removal. Names
|
||||
ga-AMtfI3 and ga-Vd9Ef2 were visible in the scratch directory listing but had
|
||||
no ownership evidence in Herb's packet; neither was opened or changed.
|
||||
|
||||
## N4 read-only trace, pre-existing and out of scope
|
||||
|
||||
Sage ruled in sage-1859 that N4 does not block 197. Legacy suites already
|
||||
reach Compose's read-only host auth mount, and the lane adds no exposure.
|
||||
Sage owns a DEFERRED entry for fixture PI_AUTH_FILE coverage. No fix is made
|
||||
in 197, and this procedure does not claim that the full gate is secret-free.
|
||||
|
||||
At 9aa6d52d9:
|
||||
|
||||
- gate-lane.mjs:109-111 keeps HOME real but neither keeps nor sets
|
||||
PI_AUTH_FILE. compose.yaml:43 therefore selects its host auth-file default.
|
||||
- script-task is discovered by gate-lane.mjs:36. test-task.sh has no
|
||||
PI_AUTH_FILE assignment. Calls at :149-150 (mock), :444, :460, :480, :483
|
||||
and :520 (live) reach run-task. mosaic-task.mjs:321 copies the environment
|
||||
and :395-400 invokes compose run mosaic-agent without changing that path.
|
||||
- script-release calls release activate at test-release.sh:69,85,93.
|
||||
release.sh:41-54 health_check calls run-task with no PI_AUTH_FILE override,
|
||||
then reaches the same Compose call. These tests use scratch MOSAIC_CONFIG;
|
||||
that does not override the separate auth-file bind mount.
|
||||
- test-auth.sh:48-92 sets explicit fixtures, but calls auth.sh instead of
|
||||
Compose. Package runs/task-cli.test.mjs exercises list/show, not run.
|
||||
|
||||
This was a static source trace only. No auth file was opened, no credential
|
||||
value was read, and no Docker command, image build or proof was executed.
|
||||
@@ -1,9 +1,12 @@
|
||||
// Evidence driver only. Runs no test without Sage's whole-slot grant.
|
||||
import { readFileSync, writeFileSync, appendFileSync, mkdirSync, existsSync, realpathSync, openSync, closeSync, symlinkSync, readdirSync, statSync, readSync } from 'node:fs';
|
||||
import { readFileSync, writeFileSync, appendFileSync, mkdirSync, existsSync, realpathSync, openSync, closeSync, symlinkSync, readdirSync, statSync, lstatSync, readSync, rmSync } from 'node:fs';
|
||||
import { resolve, join, dirname, relative, isAbsolute } from 'node:path';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
import { execFileSync, spawn } from 'node:child_process';
|
||||
const proofStart=performance.now();
|
||||
let interrupted=false,abortOnSignal=()=>{};
|
||||
const signal=()=>{interrupted=true;abortOnSignal();};
|
||||
for(const name of ['SIGINT','SIGTERM','SIGHUP'])process.on(name,signal);
|
||||
const fail=m=>{throw Error(m);};
|
||||
const within=(root,path)=>{const p=relative(root,path);return p===''||!p.startsWith('..')&&!isAbsolute(p);};
|
||||
const grant=process.env.MOSAIC_PROOF_GRANT;
|
||||
@@ -20,7 +23,7 @@ const root=resolve(config.runRoot);
|
||||
if(existsSync(root)||!within('/mnt/storage/scratch',realpathSync(dirname(root))))fail('fresh scratch runRoot required');
|
||||
const {validateTooling}=await import(pathToFileURL(join(launcher,'scripts/testing/gate-lane.mjs')));
|
||||
const {verifyImage}=await import(pathToFileURL(join(launcher,'scripts/testing/lane-image.mjs')));
|
||||
const {runRecords,status}=await import(pathToFileURL(join(launcher,'scripts/testing/lane-state.mjs')));
|
||||
const {runRecords,status,processInventory,sameProcess}=await import(pathToFileURL(join(launcher,'scripts/testing/lane-state.mjs')));
|
||||
const {summarizePairs}=await import(pathToFileURL(join(launcher,'scripts/testing/lane-evidence.mjs')));
|
||||
const {cleanupProject}=await import(pathToFileURL(join(launcher,'scripts/testing/lane-docker.mjs')));
|
||||
const docker=execFileSync('which',['docker'],{encoding:'utf8'}).trim();
|
||||
@@ -36,15 +39,7 @@ const waves=[
|
||||
];
|
||||
const plan=waves.map(w=>w.map(([label,lane,candidate])=>({label,lane,candidate,dir:join(root,label),checkout:join(root,label,'wt'),out:join(root,label,'out')})));
|
||||
put('plan.json',{grant,...config,plan,image,started:new Date().toISOString()});
|
||||
// Prepare every clean candidate before the first test. Each invocation has its own clone.
|
||||
for(const run of plan.flat()){
|
||||
mkdirSync(run.dir,{mode:0o700});
|
||||
execFileSync('git',['clone','--quiet','--no-hardlinks','--local',repo,run.checkout],{stdio:'pipe'});
|
||||
execFileSync('git',['-C',run.checkout,'checkout','--quiet','--detach',run.candidate],{stdio:'pipe'});
|
||||
symlinkSync('/mnt/storage/src/mosaic-stack/node_modules',join(run.checkout,'node_modules'));
|
||||
validateTooling(run.checkout);verifyImage(image,run.checkout);
|
||||
}
|
||||
const active=new Set(),watchers=new Map(),objects=new Map();let interrupted=false,firstFailure=null;
|
||||
const active=new Set(),watchers=new Map(),objects=new Map();let firstFailure=null;
|
||||
const eventError=[];
|
||||
const knownRuns=new Map();let abortTimer;
|
||||
const ownedRuns=()=>runRecords(state).flatMap(r=>r.records.filter(e=>e.kind==='start'&&plan.flat().some(p=>p.checkout===e.checkout)));
|
||||
@@ -99,6 +94,7 @@ function abortActive(){
|
||||
for(const job of active)job.child.kill('SIGTERM');
|
||||
if(!abortTimer){abortTimer=setTimeout(()=>{for(const job of active)job.child.kill('SIGKILL');},30000);abortTimer.unref();}
|
||||
}
|
||||
abortOnSignal=abortActive;
|
||||
function newFailure(job){
|
||||
if(!existsSync(job.run.out))return null;job.logs??=new Map();
|
||||
for(const name of readdirSync(job.run.out).filter(n=>n.endsWith('.log'))){
|
||||
@@ -137,12 +133,21 @@ async function stop(){
|
||||
const timer=setTimeout(()=>{for(const j of jobs)j.child.kill('SIGKILL');},30000);timer.unref();
|
||||
await Promise.all(jobs.map(j=>j.done));clearTimeout(timer);
|
||||
}
|
||||
const signal=()=>{interrupted=true;abortActive();};
|
||||
for(const name of ['SIGINT','SIGTERM','SIGHUP'])process.on(name,signal);
|
||||
const measured=[],start=proofStart;
|
||||
const monitorTimer=setInterval(monitor,100);
|
||||
let summary;
|
||||
try{
|
||||
// Preparation is inside the cleanup path. Yield between synchronous clones
|
||||
// so a pending signal stops preparation before another invocation is made.
|
||||
for(const run of plan.flat()){
|
||||
await new Promise(resolve=>setImmediate(resolve));
|
||||
if(interrupted)fail('proof interrupted during preparation');
|
||||
mkdirSync(run.dir,{mode:0o700});
|
||||
execFileSync('git',['clone','--quiet','--no-hardlinks','--local',repo,run.checkout],{stdio:'pipe'});
|
||||
execFileSync('git',['-C',run.checkout,'checkout','--quiet','--detach',run.candidate],{stdio:'pipe'});
|
||||
symlinkSync('/mnt/storage/src/mosaic-stack/node_modules',join(run.checkout,'node_modules'));
|
||||
validateTooling(run.checkout);verifyImage(image,run.checkout);
|
||||
}
|
||||
for(const wave of plan){
|
||||
if(interrupted)fail('proof interrupted');
|
||||
const observed=status(state);
|
||||
@@ -166,7 +171,27 @@ try{
|
||||
for(const watcher of watchers.values()){watcher.stopping=true;watcher.child.kill('SIGTERM');}
|
||||
await Promise.all([...watchers.values()].map(w=>w.done));
|
||||
let finalStatus;try{finalStatus=status(state);if(finalStatus.legacy.length||finalStatus.a.state!=='CLEAR'||finalStatus.b.state!=='CLEAR')cleanupErrors.push({message:'lane status not clear',status:finalStatus});}catch(error){cleanupErrors.push({message:error.message});}
|
||||
put('docker-summary.json',{objects:[...objects.values()],eventError,cleanupErrors,finalStatus});
|
||||
// Start receipts establish which short roots belong to this proof. Never
|
||||
// discover roots by scanning scratch or follow a substituted root symlink.
|
||||
const tempRoots=[];let procs;
|
||||
try{procs=processInventory();}catch(error){cleanupErrors.push({message:'process inventory failed: '+error.message});}
|
||||
for(const run of knownRuns.values()){
|
||||
const record={gate:plan.flat().find(p=>p.checkout===run.checkout)?.label,lane:run.lane,tempRoot:run.tempRoot};
|
||||
try{
|
||||
if(!/^\/mnt\/storage\/scratch\/g[ab]-[A-Za-z0-9]{6}$/.test(run.tempRoot??'')||!run.tempRoot.startsWith('/mnt/storage/scratch/g'+run.lane+'-'))fail('invalid recorded temp root');
|
||||
const st=lstatSync(run.tempRoot);
|
||||
if(!st.isDirectory()||st.isSymbolicLink()||st.uid!==process.getuid()||realpathSync(run.tempRoot)!==run.tempRoot)fail('temp root is not an owned real directory');
|
||||
const receipt=runRecords(state).find(r=>r.records.some(e=>e.kind==='start'&&e.checkout===run.checkout));
|
||||
const groups=new Set(receipt?.records.filter(e=>e.kind==='child').map(e=>e.group));
|
||||
if(!procs||sameProcess(run.owner)||procs.some(p=>p.state!=='Z'&&p.pid!==process.pid&&(groups.has(p.group)||p.argv.some(a=>a.includes(run.tempRoot)||a.includes(run.checkout)))))fail('run process still present or inventory unavailable; temp root retained');
|
||||
const terminal=receipt?.records.filter(e=>e.kind==='terminal').at(-1);
|
||||
if(!terminal?.clean){record.action='kept';record.reason='run cleanup unverified';cleanupErrors.push({...record});tempRoots.push(record);continue;}
|
||||
if(terminal.rc!==0||firstFailure?.gate===record.gate){record.action='kept';record.reason='failed gate diagnosis';}
|
||||
else{rmSync(run.tempRoot,{recursive:true});record.action='removed';}
|
||||
}catch(error){record.action='error';record.message=error.message;cleanupErrors.push({...record});}
|
||||
tempRoots.push(record);
|
||||
}
|
||||
put('docker-summary.json',{objects:[...objects.values()],eventError,cleanupErrors,finalStatus,tempRoots});
|
||||
if(cleanupErrors.length||eventError.length||[...objects.values()].some(o=>!o.name))process.exitCode=1;
|
||||
if(summary&&!process.exitCode){
|
||||
for(const run of knownRuns.values())for(const kind of ['container','network'])if(![...objects.values()].some(o=>o.project===run.project&&o.kind===kind&&o.created)){process.exitCode=1;process.stderr.write('missing Docker creation evidence: '+run.project+' '+kind+'\n');}
|
||||
|
||||
Reference in new issue
Block a user