mosaicstack/stack
25
0
Fork 1
Code Issues 400 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

fix(proof): match network names and preflight Docker watchers

Browse Source
This commit is contained in:
Herb
2026-10-21 10:17:16 -05:00
parent b8461266cd
commit a5b6367503
3 changed files with 252 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agents/herb/work/queue-197/proof/BUILD.md
+117 -2
View File
@@ -46,8 +46,9 @@ unresolved and points to the log rather than guessing. It stops earlier than the
two-failure limit. TERM/INT/HUP stop owned supervisors; cleanup waits for them.
Container and network names come from Compose's random recorded project. The
project and service configuration have no fixed global names. Exact-label Docker
event streams record create/destroy IDs and names, including --rm containers.
project and service configuration have no fixed global names. Container event
streams filter the exact project label; network streams match the exact project_
name prefix. Both record create/destroy IDs and names, including --rm containers.
An exit trap inventories names/labels, removes only exact-project objects with
the approved cleanup helper, verifies emptiness and stops its own event streams.
The driver's Docker inspection never reads Config.Env or credential contents, and never builds, pulls,
@@ -124,3 +125,117 @@ At 9aa6d52d9:
This was a static source trace only. No auth file was opened, no credential
value was read, and no Docker command, image build or proof was executed.
## Proof4 watcher change (sage-2029, mos-1174)
Proof3's eight gates passed and cleaned up, but Docker29 network events do
not carry Compose project labels, so the driver's combined label filter
missed their creation. Only this proof packet changes; none of the37
c45ddeb6b candidate paths changes.
Container events retain the label filter. Network events use a type filter
and a local exact project_ name-prefix match, accepting default and other
owned network names. The existing recorded-name project guard stays.
The same exported watcher/parser powers unit tests, the proof and a daemon
preflight before wave1. Importing the module does not execute a proof.
Direct execution still installs signal handlers before preparation.
The preflight creates one named and labelled mg197 probe network and one
create-only container from the pinned imageId, --pull never, network none,
no mounts or start. Exact-project label-checked cleanup removes both,
including on failure. All four create/destroy observations must arrive
within5000ms. Docker commands share that deadline; cleanup has a separate
3000ms cap. Watcher shutdown is bounded. Failure prevents all gate waves.
Probe metadata and outcomes go into docker-summary.json and summary.json.
No Config.Env, credential file contents or raw foreign events are saved.
Tests cover network events with name/type only, all project_ network names,
foreign names/labels, recorded-name ownership, fragmented streams, early
exit, missing create/destroy observations, deadline and cleanup failures.
Sage-2034 accepted this scope. Grant sage-2035 supplied the runtime evidence below.
## Granted verification, sage-2035
On 2026-10-21, the whole events.test.mjs passed 8/0, with zero skipped or
cancelled. node --check passed on run-proof.mjs and events.test.mjs. One
exported probeDockerWatchers call used the supplied proof3 image ID and
cleanupProject from the unchanged r8 launcher. No container was started;
no pull, build, lane or wave ran. Both filtered post-probe Docker listings
were empty, and the owned /proc cwd/project scan was empty. The slot was
returned at 2026-10-21T15:16:52Z.
Driver SHA-256:
`998f41948355ea86169c1711324a3c9e6cf4bdbe5157dcae9e78e414c1c16342`.
Sage updates their proof-phase.sh digest pin to this value.
The complete standalone probe output was:
```json
{
"result": {
"project": "mg197-a-724f1800af2c",
"imageId": "sha256:1b83521d092a3f005ea2192e9e6268099e6e0cdab85d41a5d107b4fb824131a6",
"timeoutMs": 5000,
"names": {
"container": "mg197-a-724f1800af2c-watcher-probe",
"network": "mg197-a-724f1800af2c_default"
},
"events": [
{
"project": "mg197-a-724f1800af2c",
"kind": "network",
"id": "930b1c80596fc93b4dc104ab1e278e5b1e1b2bc6373a5cb9cba7bba46f230ef8",
"name": "mg197-a-724f1800af2c_default",
"action": "create",
"observedMs": 38.03536499999999
},
{
"project": "mg197-a-724f1800af2c",
"kind": "container",
"id": "8fcdf9d06c437e162698095b2234402ce75fd9744669ced4017f823b5a99fc83",
"name": "mg197-a-724f1800af2c-watcher-probe",
"action": "create",
"observedMs": 123.93732700000001
},
{
"project": "mg197-a-724f1800af2c",
"kind": "container",
"id": "8fcdf9d06c437e162698095b2234402ce75fd9744669ced4017f823b5a99fc83",
"name": "mg197-a-724f1800af2c-watcher-probe",
"action": "destroy",
"observedMs": 297.033782
},
{
"project": "mg197-a-724f1800af2c",
"kind": "network",
"id": "930b1c80596fc93b4dc104ab1e278e5b1e1b2bc6373a5cb9cba7bba46f230ef8",
"name": "mg197-a-724f1800af2c_default",
"action": "destroy",
"observedMs": 297.07703999999995
}
],
"errors": [],
"cleanup": {
"clean": true,
"removed": [
{
"kind": "container",
"id": "8fcdf9d06c43"
},
{
"kind": "network",
"id": "930b1c80596f"
}
]
},
"ok": true,
"missing": [],
"wallMs": 308.567205
},
"inventories": {
"containers": "",
"networks": ""
}
}
```
agents/herb/work/queue-197/proof/events.test.mjs
+60
View File
@@ -0,0 +1,60 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import { EventEmitter } from 'node:events';
import { PassThrough } from 'node:stream';
import { dockerEvent, watchDockerProject, probeDockerWatchers } from './run-proof.mjs';
const project='mg197-a-0123456789ab', imageId='sha256:'+'a'.repeat(64);
const ids={container:'b'.repeat(64),network:'c'.repeat(64)};
const event=(kind,action,name=kind==='network'?project+'_default':project+'-probe')=>({Type:kind,Action:action,Actor:{ID:ids[kind],Attributes:{name,type:'bridge',...(kind==='container'?{'com.docker.compose.project':project}:{})}}});
function streams(){
const children=new Map(),calls=[];
const spawnImpl=(binary,args)=>{const kind=args.includes('type=network')?'network':'container';calls.push(args);const child=new EventEmitter();child.stdout=new PassThrough();child.stderr=new PassThrough();child.kill=()=>{queueMicrotask(()=>child.emit('close',0));return true;};children.set(kind,child);return child;};
const emit=(kind,action,name)=>children.get(kind).stdout.write(JSON.stringify(event(kind,action,name))+'\n');
return {children,calls,spawnImpl,emit};
}
test('network create/destroy with only name and type match every exact project_ network',()=>{
for(const name of [project+'_default',project+'_other_network'])for(const action of ['create','destroy']){
const value=dockerEvent(project,'network',event('network',action,name));
assert.deepEqual(value,{project,kind:'network',id:ids.network,name,action});
}
for(const name of ['foreign_default',project+'x_default',project+'-other',project])assert.equal(dockerEvent(project,'network',event('network','create',name)),null);
assert.equal(dockerEvent(project,'network',event('network','connect')),null);
});
test('container labels and recorded name ownership remain strict',()=>{
assert.equal(dockerEvent(project,'container',event('container','create')).id,ids.container);
const foreign=event('container','create');foreign.Actor.Attributes['com.docker.compose.project']='other';assert.equal(dockerEvent(project,'container',foreign),null);
assert.throws(()=>dockerEvent(project,'container',event('container','create','foreign-container')),/name outside/);
const invalid=event('network','create');invalid.Actor.ID='not-an-id';assert.throws(()=>dockerEvent(project,'network',invalid),/ownership metadata/);
});
test('two real watcher shapes decode fragmented lines and network stream has no label filter',async()=>{
const s=streams(),seen=[],errors=[];const watcher=watchDockerProject(project,{docker:'/fixture/docker',spawnImpl:s.spawnImpl,onEvent:e=>seen.push(e),onError:e=>errors.push(e)});
const text=JSON.stringify(event('network','create'))+'\n';s.children.get('network').stdout.write(text.slice(0,17));s.children.get('network').stdout.write(text.slice(17));s.emit('network','destroy');s.emit('container','create');s.emit('container','destroy');
assert.equal(seen.length,4);assert.ok(s.calls.find(a=>a.includes('type=container')).includes('label=com.docker.compose.project='+project));assert.ok(!s.calls.find(a=>a.includes('type=network')).some(a=>a.startsWith('label=')));
await watcher.stop();assert.deepEqual(errors,[]);
});
test('an event watcher ending early reports an error',async()=>{
const s=streams(),errors=[];const watcher=watchDockerProject(project,{docker:'/fixture/docker',spawnImpl:s.spawnImpl,onEvent:()=>{},onError:e=>errors.push(e)});s.children.get('network').emit('close',1);assert.match(errors[0],/ended early/);await watcher.stop();
});
function probeFixture({missing=null,commandFails=false,cleanupFails=false}={}){
const s=streams(),calls=[];let cleaned=false;
const invoke=async args=>{calls.push(args);const kind=args[0];if(commandFails)throw Error('fixture daemon error');if(missing!==kind+'.create')s.emit(kind,'create');return ids[kind]+'\n';};
const cleanupProject=run=>{assert.equal(run.project,project);cleaned=true;if(cleanupFails)throw Error('fixture cleanup refusal');for(const kind of ['container','network'])if(missing!==kind+'.destroy')s.emit(kind,'destroy');return {clean:true,removed:Object.entries(ids).map(([kind,id])=>({kind,id}))};};
return {options:{docker:'/fixture/docker',imageId,project,spawnImpl:s.spawnImpl,invoke,cleanupProject,timeoutMs:80},calls,get cleaned(){return cleaned;}};
}
test('preflight uses the same watchers, creates only pinned no-pull objects, removes both and sees all four events',async()=>{
const f=probeFixture(),result=await probeDockerWatchers(f.options);assert.equal(result.ok,true);assert.equal(f.cleaned,true);assert.deepEqual(result.missing,[]);assert.equal(result.events.length,4);
const create=f.calls.find(a=>a[0]==='container');assert.equal(create[1],'create');assert.equal(create[create.indexOf('--pull')+1],'never');assert.ok(create.includes(imageId));assert.ok(create.includes('--network')&&create.includes('none'));assert.ok(f.calls.every(a=>!a.includes('start')&&!a.includes('build')));
});
test('missing either create or destroy fails within the bounded wait and still cleans up',async()=>{
for(const missing of ['container.create','container.destroy','network.create','network.destroy']){
const f=probeFixture({missing}),start=performance.now(),result=await probeDockerWatchers(f.options);assert.equal(result.ok,false);assert.ok(result.missing.includes(missing));assert.ok(performance.now()-start<1500);assert.equal(f.cleaned,true);
}
});
test('probe command and cleanup failures refuse and retain their own result',async()=>{
for(const options of [{commandFails:true},{cleanupFails:true}]){const f=probeFixture(options),result=await probeDockerWatchers(f.options);assert.equal(result.ok,false);assert.ok(result.errors.length);assert.equal(f.cleaned,true);}
});
test('late observations cannot turn a missed deadline into success',async()=>{
const f=probeFixture(),invoke=f.options.invoke;f.options.invoke=async args=>{await new Promise(r=>setTimeout(r,100));return invoke(args);};
const result=await probeDockerWatchers(f.options);assert.equal(result.ok,false);assert.ok(result.missing.includes('network.create'));assert.equal(f.cleaned,true);
});
agents/herb/work/queue-197/proof/run-proof.mjs
+75 -12
View File
@@ -2,7 +2,68 @@
import { readFileSync, writeFileSync, appendFileSync, mkdirSync, existsSync, realpathSync, openSync, closeSync, symlinkSync, readdirSync, statSync, lstatSync, readSync, rmSync } from 'node:fs';
import { resolve, join, dirname, relative, isAbsolute } from 'node:path';
import { pathToFileURL } from 'node:url';
import { execFileSync, spawn } from 'node:child_process';
import { execFileSync, execFile, spawn } from 'node:child_process';
import { randomBytes } from 'node:crypto';
const projectPattern=/^mg197-[ab]-[a-f0-9]{12}$/;
export function dockerEvent(project,kind,event){
if(!projectPattern.test(project)||!['container','network'].includes(kind))throw Error('invalid Docker ownership metadata');
if(event.Type!==kind||!['create','destroy'].includes(event.Action))return null;
const attrs=event.Actor?.Attributes??{},name=typeof attrs.name==='string'?attrs.name.replace(/^\//,''):null;
if(kind==='network'){
if(!name?.startsWith(project+'_'))return null;
if(attrs['com.docker.compose.project']&&attrs['com.docker.compose.project']!==project)throw Error('foreign project event');
}else if(attrs['com.docker.compose.project']!==project)return null;
if(!name?.startsWith(project))throw Error('Docker name outside its recorded project');
const id=event.Actor?.ID;if(typeof id!=='string'||!/^[a-f0-9]{12,64}$/.test(id))throw Error('invalid Docker ownership metadata');
return {project,kind,id,name,action:event.Action};
}
export function watchDockerProject(project,{docker,onEvent,onError,spawnImpl=spawn}){
if(!projectPattern.test(project))throw Error('invalid Docker ownership project');
const streams=[];
for(const kind of ['container','network']){
// Network events on Docker29 have name/type but no Compose labels. Filter
// their exact project_ name prefix locally, never by a missing label.
const args=['events','--since',String(Math.floor(Date.now()/1000)-5),'--filter','type='+kind];
if(kind==='container')args.push('--filter','label=com.docker.compose.project='+project);
args.push('--filter','event=create','--filter','event=destroy','--format','{{json .}}');
const child=spawnImpl(docker,args,{stdio:['ignore','pipe','pipe']}),stream={child,stopping:false,buffer:'',closed:false};streams.push(stream);
child.stdout.on('data',bytes=>{stream.buffer+=bytes;let end;while((end=stream.buffer.indexOf('\n'))>=0){const line=stream.buffer.slice(0,end);stream.buffer=stream.buffer.slice(end+1);if(!line)continue;try{const event=dockerEvent(project,kind,JSON.parse(line));if(event)onEvent(event);}catch(error){onError(error.message);}}});
child.stderr.resume();stream.done=new Promise(resolve=>{child.once('error',error=>{stream.closed=true;onError(error.message);resolve();});child.once('close',code=>{stream.closed=true;if(!stream.stopping)onError('Docker '+kind+' event stream ended early: '+code);resolve();});});
}
return {async stop(){
for(const stream of streams){stream.stopping=true;if(!stream.closed)stream.child.kill('SIGTERM');}
let timer;await Promise.race([Promise.all(streams.map(s=>s.done)),new Promise(resolve=>{timer=setTimeout(resolve,1000);})]);clearTimeout(timer);
for(const stream of streams)if(!stream.closed){stream.child.kill('SIGKILL');onError('Docker event watcher did not stop promptly');}
}};
}
export async function probeDockerWatchers({docker,imageId,cleanupProject,project='mg197-a-'+randomBytes(6).toString('hex'),timeoutMs=5000,spawnImpl=spawn,invoke}){
if(!projectPattern.test(project)||!/^sha256:[a-f0-9]{64}$/.test(imageId??''))throw Error('invalid watcher probe ownership or image');
const start=performance.now(),deadline=start+timeoutMs,events=[],errors=[],names={container:project+'-watcher-probe',network:project+'_default'};
const result={project,imageId,timeoutMs,names,events,errors,cleanup:null,ok:false};
const watcher=watchDockerProject(project,{docker,spawnImpl,onEvent:e=>events.push({...e,observedMs:performance.now()-start}),onError:e=>errors.push(e)});
const remaining=()=>{const ms=Math.ceil(deadline-performance.now());if(ms<=0)throw Error('Docker watcher probe deadline');if(errors.length)throw Error('Docker watcher probe stream failed');return ms;};
const call=invoke??((args,timeout)=>new Promise((resolve,reject)=>execFile(docker,args,{encoding:'utf8',timeout,killSignal:'SIGKILL',maxBuffer:65536},(error,stdout)=>error?reject(Error('Docker probe '+args.slice(0,2).join(' ')+' failed: '+error.code)):resolve(stdout))));
try{
for(const args of [
['network','create','--label','com.docker.compose.project='+project,names.network],
['container','create','--pull','never','--name',names.container,'--label','com.docker.compose.project='+project,'--network','none',imageId,'/bin/true'],
]){const id=(await call(args,remaining())).trim();if(!/^[a-f0-9]{12,64}$/.test(id))throw Error('invalid Docker probe create ID');}
}catch(error){errors.push(error.message);}
finally{
// Label-checked cleanup also covers an uncertain create result. Each CLI
// call shares a separate bounded cleanup budget and never reads Config.Env.
const cleanupDeadline=performance.now()+3000;
try{result.cleanup=cleanupProject({project},args=>{const timeout=Math.ceil(cleanupDeadline-performance.now());if(timeout<=0)throw Error('probe cleanup deadline');return execFileSync(docker,args,{encoding:'utf8',stdio:['ignore','pipe','pipe'],timeout,killSignal:'SIGKILL',maxBuffer:65536});});if(!result.cleanup?.clean)throw Error('probe cleanup not confirmed');}
catch(error){errors.push('probe cleanup: '+error.message);}
}
const missing=()=>['container.create','container.destroy','network.create','network.destroy'].filter(key=>!events.some(e=>e.kind+'.'+e.action===key&&e.observedMs<=timeoutMs));
while(!errors.length&&missing().length&&performance.now()<deadline)await new Promise(resolve=>setTimeout(resolve,10));
result.missing=missing();if(result.missing.length)errors.push('missing probe events: '+result.missing.join(', '));
await watcher.stop();result.wallMs=performance.now()-start;result.ok=!errors.length&&result.cleanup?.clean===true;
return result;
}
export async function runProof(){
const proofStart=performance.now();
let interrupted=false,abortOnSignal=()=>{};
const signal=()=>{interrupted=true;abortOnSignal();};
@@ -52,12 +113,8 @@ function logObject(project,kind,id,name,action){
function ensureWatchers(final=false){
if(final||[...active].some(j=>!knownRuns.has(j.run.checkout)))for(const run of ownedRuns())knownRuns.set(run.checkout,run);
for(const run of knownRuns.values())if(!watchers.has(run.project)){
// Replay the few seconds before the immutable start was noticed; the random
// exact project label excludes all foreign resources. No raw events are saved.
const child=spawn(docker,['events','--since',String(Math.floor(Date.now()/1000)-5),'--filter','label=com.docker.compose.project='+run.project,'--filter','type=container','--filter','type=network','--filter','event=create','--filter','event=destroy','--format','{{json .}}'],{stdio:['ignore','pipe','pipe']});
const watcher={child,stopping:false,buffer:''};watchers.set(run.project,watcher);
child.stdout.on('data',bytes=>{watcher.buffer+=bytes;let end;while((end=watcher.buffer.indexOf('\n'))>=0){const line=watcher.buffer.slice(0,end);watcher.buffer=watcher.buffer.slice(end+1);if(!line)continue;try{const event=JSON.parse(line),kind=event.Type,action=event.Action;const attrs=event.Actor?.Attributes??{};if(attrs['com.docker.compose.project']&&attrs['com.docker.compose.project']!==run.project)throw Error('foreign project event');if(['container','network'].includes(kind)&&['create','destroy'].includes(action))logObject(run.project,kind,event.Actor.ID,attrs.name,action);}catch(error){eventError.push(error.message);}}});
child.stderr.resume();watcher.done=new Promise(resolve=>{child.once('error',error=>{eventError.push(error.message);resolve();});child.once('close',code=>{if(!watcher.stopping)eventError.push('Docker event stream ended early: '+code);resolve();});});
const watcher=watchDockerProject(run.project,{docker,onEvent:event=>logObject(event.project,event.kind,event.id,event.name,event.action),onError:message=>eventError.push(message)});
watchers.set(run.project,watcher);
}
}
function inventory(project){
@@ -135,7 +192,7 @@ async function stop(){
}
const measured=[],start=proofStart;
const monitorTimer=setInterval(monitor,100);
let summary;
let summary,watcherProbe=null;
try{
// Preparation is inside the cleanup path. Yield between synchronous clones
// so a pending signal stops preparation before another invocation is made.
@@ -148,6 +205,10 @@ try{
symlinkSync('/mnt/storage/src/mosaic-stack/node_modules',join(run.checkout,'node_modules'));
validateTooling(run.checkout);verifyImage(image,run.checkout);
}
const probeProject='mg197-a-'+randomBytes(6).toString('hex');
put('watcher-probe-plan.json',{project:probeProject,container:probeProject+'-watcher-probe',network:probeProject+'_default',imageId:image.imageId});
watcherProbe=await probeDockerWatchers({docker,imageId:image.imageId,cleanupProject,project:probeProject});
if(!watcherProbe.ok)fail('Docker watcher preflight failed: '+JSON.stringify(watcherProbe));
for(const wave of plan){
if(interrupted)fail('proof interrupted');
const observed=status(state);
@@ -161,15 +222,14 @@ try{
const end=status(state);
if(end.legacy.length||end.a.state!=='CLEAR'||end.b.state!=='CLEAR')fail('unclean wave status');
}
const solos=measured.slice(0,2).flat(),pairs=measured.slice(2);summary={grant,solos,pairs,summary:summarizePairs(solos,pairs),wallMs:performance.now()-start,finished:new Date().toISOString()};
const solos=measured.slice(0,2).flat(),pairs=measured.slice(2);summary={grant,watcherProbe,solos,pairs,summary:summarizePairs(solos,pairs),wallMs:performance.now()-start,finished:new Date().toISOString()};
}catch(error){
await stop();put('failed.json',{grant,message:error.message,firstFailure,measured,at:new Date().toISOString()});process.stderr.write(error.message+'\n');process.exitCode=1;
}finally{
clearInterval(monitorTimer);clearTimeout(abortTimer);await stop();const cleanupErrors=[];
try{ensureWatchers(true);}catch(error){cleanupErrors.push({message:error.message});}
for(const run of knownRuns.values())try{inventory(run.project);cleanupProject({project:run.project},docker);if(inventory(run.project).length)fail('Docker resources remain');}catch(error){cleanupErrors.push({project:run.project,message:error.message});}
for(const watcher of watchers.values()){watcher.stopping=true;watcher.child.kill('SIGTERM');}
await Promise.all([...watchers.values()].map(w=>w.done));
await Promise.all([...watchers.values()].map(w=>w.stop()));
let finalStatus;try{finalStatus=status(state);if(finalStatus.legacy.length||finalStatus.a.state!=='CLEAR'||finalStatus.b.state!=='CLEAR')cleanupErrors.push({message:'lane status not clear',status:finalStatus});}catch(error){cleanupErrors.push({message:error.message});}
// Start receipts establish which short roots belong to this proof. Never
// discover roots by scanning scratch or follow a substituted root symlink.
@@ -191,7 +251,7 @@ try{
}catch(error){record.action='error';record.message=error.message;cleanupErrors.push({...record});}
tempRoots.push(record);
}
put('docker-summary.json',{objects:[...objects.values()],eventError,cleanupErrors,finalStatus,tempRoots});
put('docker-summary.json',{watcherProbe,objects:[...objects.values()],eventError,cleanupErrors,finalStatus,tempRoots});
if(cleanupErrors.length||eventError.length||[...objects.values()].some(o=>!o.name))process.exitCode=1;
if(summary&&!process.exitCode){
for(const run of knownRuns.values())for(const kind of ['container','network'])if(![...objects.values()].some(o=>o.project===run.project&&o.kind===kind&&o.created)){process.exitCode=1;process.stderr.write('missing Docker creation evidence: '+run.project+' '+kind+'\n');}
@@ -199,3 +259,6 @@ try{
}
for(const name of ['SIGINT','SIGTERM','SIGHUP'])process.off(name,signal);
}
}
if(process.argv[1]&&import.meta.url===pathToFileURL(process.argv[1]).href)await runProof();