stack/agents/fethry/work/queue-328/BUILD.md

# Row 328 — T9 second-start library sync race (#1826, Vikunja 622)

Fethry14f80e3c; branch fethry/row328; new worktree
/home/jwoltje/.t3/worktrees/mosaic-stack/fethry-row328 on base2ce476d54.
Brief: docs/plans/2026-10-24_t9-library-sync-race.md, section Brief.
Authority: sage4190 and Mos1594. Review Boyd first, Dewey landing; Sage owns slot/gate/landing.
Status: UNTESTED. No CLEAR yet; Gyro then Della have the next slots. No tests run.
Local root npm ci --ignore-scripts --no-audit --no-fund completed, no shared dependency links.

## Root cause and bounded fix

Row266 bus999/1 exposed T9's revoked second start racing a later timer pass. Waiting for the
first held log only orders the initial pass: sync.mjs clears running in finally before the
runtime writes the held log. A later50ms timer can acquire running before the manual verb.
The verb then gets library-sync-busy instead of the asserted library-fetch-no-grant.

Only packages/bus/tests/library-credential.test.mjs T9 changes. Give its start closure an
intervalMs parameter default50. Leave the first start as start(); pass2147483647 to the second.
This is Node TIMEOUT_MAX, within the runtime positive safe integer validator. Do not exceed it:
Node clamps overflow to1ms. The startup tick still runs and the existing held-log wait orders it;
no periodic tick can overlap the second verb within this test's60s lifetime. close still clears
its unref'd interval. A comment names mos-1594 and why the interval differs.

No assertion changes, other-test changes, product code, timer hooks or row278 work. If more is
needed, stop for Sage/Mos. Existing approved test numstat will be recorded after implementation.

## Plan and acceptance (sage4190 / pinned brief)

- [x] Read brief and relevant runtime/sync/T9 paths; create separate worktree; queue next names328.
- [x] Owner start from canonical, op fethry-0037, rev3992, no timeout.
- [x] Implement only the start parameter/second interval and short comment; assert the diff has
      no changed assertion text and no production path. Write the stress/M1 runner below.
- [ ] Commit/push marked Untested, per Sage's explicit instruction overriding the default test-
      before-commit rule for this gated assignment. Send tip, manifest and concrete run plan.
- [ ] After Sage CLEAR: RED M1-50,50 T9-only runs, each fresh mktemp-d TMPDIR under scratch.
      If no race failure, run M1-1 for50 and retain both. Neither failing means stop/report.
- [ ] Restore exact fixed bytes/SHA after each mutant; git diff --stat against build commit
      must be empty. Run the same50 fixed T9 attempts, all must pass; whole touched file too.
- [ ] Record all commands/counts/load averages and TAPs, cleanup own process groups, return slot.
      Sage's landing gate: bus suite and every scripts/test-*.sh. Boyd then Dewey review.

## Stress runner contract

stress.py --clear-key KEY --output ABS_NEW_DIR --runs 50
runs M1-50 first; M1-1 only if needed; then fixed50; then the whole touched file.
Each child is timeout -k10 120 node --test --test-reporter=tap --test-concurrency=1
--test-timeout=90000 [--test-name-pattern '^C2-03b T9 runtime:'] TEST_PATH.
Each child gets its own process group and unique mktemp-d TMPDIR. Live-provider env is unset.
Every TAP and per-run load average/exit/counts is retained. Only an assertion of actual
library-sync-busy versus expected library-fetch-no-grant counts as the required race failure;
other failures stop for Sage. Fixed runs require exactly one passing T9, no failures/cancels;
whole file requires all six pass. Mutants restore in finally and verify byte hash against HEAD.
The runner does not weaken assertions, manufacture load, change production, or retry away a red.

## Prepared candidate (UNTESTED, pending CLEAR)

Approved pre-existing test: packages/bus/tests/library-credential.test.mjs numstat +4/-3.
Exact diff is limited to default50 parameter, shorthand intervalMs, second-start2147483647
and one Mos1594 comment. Python static comparison verifies the complete expected transformation
and unchanged assertion lines. No other package path changes. stress.py parses with Python AST;
no test/harness execution, Node syntax run, suite, stress attempt or mutant has run.
Manifest.r1 freezes the single test file; the runner/docs are review-packet support.

Run plan after CLEAR (about3-5min for100-150 T9 runs plus whole six-test file):
  mkdir only the desired parent output directory, then from this worktree:
  python3 agents/fethry/work/queue-328/stress.py --clear-key SAGE_CLEAR_KEY \
    --runs 50 --output /mnt/storage/scratch/tmp/fethry-328-EVIDENCE_UNIQUE
The runner creates that new output dir itself. It does NOT stop at M1's first expected race: all50
are retained, then optional50 M1-1 if M1-50 all green, then50 fixed and whole touched file.
Unexpected red, neither mutant failing, or any fixed red stops and reports; no additional fix
is authorized. Subsequent bus/scripts scope belongs to Sage's landing gate unless CLEAR expands it.
After execution copy/record evidence, counts/load, restoration git diff --stat and own-process sweep,
commit/push evidence, return slot, then first-review handoff Boyd. No review/green claim yet.

## sage4194 attempt — STOPPED, excluded (2026-10-24 18:17:24Z)

Exact build36668b698. Root npm ci already complete before slot, real local dependencies.
Preflight git diff --stat HEAD and git status --porcelain both empty. Parent shell first
reported nice0. Actual launcher: nice -n19 python3 -B -c '[assert os.getpriority(self)==19;
print proof; exec python3 -B stress.py]' with --clear-key sage-4194 --runs50 and output
/mnt/storage/scratch/tmp/fethry-328-sage4194-20261024T181643Z. No bytecode created.

Console proves absolute nice19 at launcher entry. Subsequent ps showed runner542244 nice-4;
independent os.getpriority(pid) and /proc/pid/stat both read-4. Thus priority changed after
initial proof. Cause unknown; external priority management is a hypothesis only.
Stopped own runner SIGINT per sage4194/mos1592, no rerun.17 completed M1-50 runs GREEN,
zero race failures; attempt18 interrupted. No fallback, fixed50 or whole file started.
Not acceptance evidence. Start load averages(1/5/15min):3.043457/3.404785/5.511719;
end:3.304688/3.436523/5.487305. Each completed run retained its own load/TMPDIR/TAP.

Finally restored exact test bytes/SHA, empty git diff --stat HEAD, empty porcelain, and
no own node/Chromium remaining. Scan saw only inspection Python, which exited. Slot
returned fethry0039 while Sage rules; no test/runner modifications. Raw and normalized
receipts retained in evidence/aborted-sage4194 with provenance and explicit EXCLUDED.md.

## sage4196 priority ruling — preparation only

Sage reports active ananicy-cpp (15s checks/apply_nice); no specific Python rule proven.
Priority drift alone is no longer a stop condition and has no bearing on test validity.
Fresh CLEAR required after Della. Begin again at attempt1, initial absolute nice19, Python-B.

Only stress.py instrumentation added: a read-only100ms /proc sampler records runner and
timeout/Node descendant pid, comm, nice and elapsed sample time for every attempt, retaining
changes. Existing wait, timeout, verdict, phase order, mutation and restoration paths unchanged.
No renice loop or host/rule change. Test/manifest remain identical to36668b698. Static AST
parse only; no runner or tests executed. Stop remains fixed red, dirty tree or other unexpected
event. Final clean checks include full git status --porcelain as well as diff --stat.

## sage4205 fresh CLEAR / launch corrections

Sage4199 was withdrawn by4201 before any runner started: zero attempts, nothing to exclude.
Sage4205 restores CLEAR after row224 controls. Start with os.setpriority(PRIO_PROCESS,0,19)
before exec (absolute, independent of inherited niceness); record the observed result.
The requested short shell mktemp template is /mnt/storage/scratch/tmp/gXXX,29bytes and
alphanumeric basename. Prepared that literal change in stress.py before execution.
Also made both mutant-restore and final clean checks record full git diff --stat HEAD and
git status --porcelain, refusing any dirty result before the fixed loop. Existing tests,
assertions, verdict rules and phase order remain unchanged. Harness-only preflight commit;
package test remains byte-identical to d7cef2ffb / original36668b698. No tests yet.

## sage4205 result — required stop, no M1 reproduction

Source08055f419 (harness/doc-only follow-up to d7cef2ffb; package test unchanged).
Preflight clean. Command from /home/jwoltje/.t3/worktrees/mosaic-stack/fethry-row328:
  python3 -B -c 'os.setpriority(os.PRIO_PROCESS,0,19); assert getpriority(self)==19;
    exec python3 -B stress.py' agents/fethry/work/queue-328/stress.py \
    --clear-key sage-4205 --runs 50 \
    --output /mnt/storage/scratch/tmp/fethry-328-sage4205-20261024T183200Z
Exact emitted child commands are in every run's summary record:
  timeout -k10 120 node --test --test-reporter=tap --test-concurrency=1 \
    --test-timeout=90000 --test-name-pattern '^C2-03b T9 runtime:' \
    packages/bus/tests/library-credential.test.mjs
Every attempt gets a distinct shell mktemp-d /mnt/storage/scratch/tmp/gXXX TMPDIR.
All100 actual paths verified alphanumeric basename,29bytes, no underscore.

Counts (not including the17 aborted sage4194 runs or withdrawn sage4199/zero runs):
M1 second interval50ms:50 attempts,50 pass,0 fail,0 skip,0 cancel;56.394s total.
M1 fallback second interval1ms:50 attempts,50 pass,0 fail,0 skip,0 cancel;59.799s total.
Race failures:0 in each phase. Thus Mos1594 cannot be satisfied on this attempt.
The runner deliberately stopped/exited1 with Neither M1 interval failed before any fixed
loop or whole-file run. Fixed50:NOT RUN. Whole file:NOT RUN. No acceptance/green claim.

Host load averages1/5/15min at run start:6.323730/4.219727/4.300781;
M1-50 first/last:6.618164/4.315918/4.331543 →5.056152/4.287109/4.323242;
M1-1 first/last:5.056152/4.287109/4.323242 →4.894043/4.374023/4.351074;
run end:4.894043/4.374023/4.351074. Per-attempt exact values retained in summary.json.

Absolute priority19 was set and verified before exec. Read-only samples observe runner19
then-4; M1-50 Node samples19,-4,16; fallback Node-4,16 and runner-4. These include
pid/comm/nice/elapsed changes for runner, timeout, Node and descendants per attempt.
Drift only recorded per sage4196. No host rules or process priorities changed after launch.

Both mutant phases restore SHA ba485ecf337b4845a1c3e8ba2f4e2f0ee221d981b8d74080a3b0bcbef55d6605.
After each restore and at final exit: git diff --stat HEAD empty, git status --porcelain empty.
All test process groups cleaned (no extra signals needed); own cwd sweep no node/Chromium/
runner remaining. Slot RETURNED fethry0043 at18:34:26Z; no rerun or further test/source change.
Raw100 TAPs/console/summary retained, normalized copies and raw SHA provenance under
evidence/sage4205-no-reproduction. Original summary status running is an exception-path
artifact, clarified by OUTCOME.md; actual process exited1 at the required boundary.
Await Sage/Mos ruling. Row remains in-progress; do not move to in-review without acceptance.

## mos1599 option A/B — concurrent plan prepared, NOT RUN

Authority sage4211: prepare now, tests wait for fresh CLEAR after Della224. The prior serial
50+50 non-reproduction stays in the record; it is not counted toward this new400+400 plan.
New support runner stress-concurrent.py imports the existing read-only priority sampler and
constants from stress.py (python3-B avoids bytecode). No package/test/product change.

Command after CLEAR from the row328 worktree:
  python3 -B agents/fethry/work/queue-328/stress-concurrent.py --clear-key SAGE_CLEAR_KEY \
    --output /mnt/storage/scratch/tmp/fethry-328-CONCURRENT_UNIQUE

The runner sets its priority absolutely19; every child also calls os.setpriority to19 before
exec of timeout/Node. Observed Node nice values/drift are sampled per attempt and do not
change the verdict. Each test has timeout-k10 120, per-test timeout90000, own process group,
TAP/run JSON/load/nice record and fresh shell mktemp-d /mnt/storage/scratch/tmp/gXXX.
8 persistent workers begin together behind a barrier, each loops50 times:400 M1-50 attempts.
One shared test mutation is installed before any worker; no worker changes the source.
All workers finish before exact restoration, full diff/stat+porcelain clean check, then the
same8 workers x50 fixed attempts. No1ms phase in this amended plan. Whole file follows.

Data-root isolation is supplied by the unchanged fixture: library/tests/helpers.mjs:37-39
repo(t) creates a new library-* under node:os.tmpdir(), hence each invocation's unique TMPDIR;
T9:266 sets dataRoot=join(f.root,'data'). Therefore no two workers share a data root. Each run
records its own TMPDIR and derived dataRootPattern; that is an honest pattern, not a claimed
observed exact path. No credential or data-root content is read by the support runner.

A: M1 race fails>=1, fix400/400; then whole file and review.
B automatically if M1 never fails (mos1599, no another ask): record400/400 non-reproduction,
then fix400 and whole file. Construction citation: runtime.mjs:597 writes library-sync-held
only after sync returns; sync.mjs:203 finally clears running before that. T9's startup wait
(original:315) sees the line after startup released the mirror. TIMEOUT_MAX removes the other
periodic caller during the verb. This proof is saved in the B result, not invented as a pass.
Any fixed red, unknown error, malformed TAP or dirty restore stops all own active worker
process groups and reports; no hidden retries or weaker assertions. Expected M1 refusal is
counted only for ERR_ASSERTION actual library-sync-busy vs expected library-fetch-no-grant.
All801 receipts (400 M1+400 fixed+1 whole) retained on success. Cleanup/restoration occurs
in finally, including interruption. No tests executed; Python AST and static source review only.

## sage4214 / mos1599 accepted concurrent run — GREEN

Executed exact script commit06c57a7db on clean worktree; package test remains original
approved36668b698 bytes. Command:
  python3 -B agents/fethry/work/queue-328/stress-concurrent.py --clear-key sage-4214 \
    --output /mnt/storage/scratch/tmp/fethry-328-sage4214-20261024T184538Z
Eight workers,50 attempts each, fresh shell mktemp-d gXXX for every test. All801 TMPDIRs
distinct,29bytes, alphanumeric basenames. Existing repo(t) and T9 join(root,data) ensure
per-invocation data roots. Each timeout/Node child sets absolute nice19 before exec.
Actual child commands and TMPDIR/load/nice/exit/counts pinned per record.

M1-50:400 attempts,374 PASS,26 expected assertion failures actual library-sync-busy versus
expected library-fetch-no-grant. No other failure, skip or cancellation. Every worker
completed50. This reproduces the old race under concurrent load.
Mutant restored exact fixed SHA ba485ecf337b4845a1c3e8ba2f4e2f0ee221d981b8d74080a3b0bcbef55d6605.
Before fixed phase: full git diff --stat HEAD EMPTY; git status --porcelain EMPTY.
Fixed:400/400 PASS,0 fail,0 skip/cancel; same eight workers x50, no rerun.
Whole packages/bus/tests/library-credential.test.mjs:6/6 PASS,0 fail,0 skip/cancel.
Overall runner exit0/statusgreen. Mos1599 outcome A; construction alternative B not needed.
Earlier serial non-reproduction and aborted17 remain separate and uncounted.

Host load averages1/5/15min: start4.833496/5.014160/4.664063;
M1 end14.994141/8.808594/6.086914; fixed end24.249023/14.094727/8.335449;
whole-file/end25.190430/14.458496/8.484375. Exact per-attempt loads retained.
Runner initially19, sampled19 then-4 in M1; fixed/whole runner-4. Actual Node nice values
16 and19 in BOTH mutant and fixed phases and whole file. Drift recorded only, as ruled.
All process groups exited cleanly, no extra kill signals needed. Own cwd node/Chromium/
runner sweep EMPTY. Final exact hash restored; full diff/stat and porcelain EMPTY before
receipts. No __pycache__, no product code or assertion changes.

Evidence/sage4214-concurrent contains all801 normalized TAPs, full801 run metadata/nice
records in run-records.jsonl, original summary/console display copies, process sweep and
raw-provenance digests. Raw files remain at the scratch path. Only trailing whitespace
is stripped from display TAP/console; metadata JSON is losslessly condensed into JSONL.
Manifest.r1 digest1e0104683f49dcc96cb87975393665aa4ee30114055b4bbb45ad5e9ba9611733,
one package test path. Approved old-test numstat STILL+4/-3; no assertion/product change.
Ready for Boyd first review then Dewey, after owner in-review move. Sage posts missing-
credential request and runs full bus plus every scripts/test-*.sh landing gate.

## Owner review move and first review

Canonical fethry-328-inreview first found Sage's review-outcome lock; waited30s and retried
same op. Retry succeeded rev3997: row328 in-review round1 #1826, manifest1e010468.
Request outcome rev3998 failed solely because MOSAIC_GITEA_CREDENTIAL_FILE is unset.
No alternate credential used or credential file read. Sage posts the arranged request.
Boyd0039 APPROVE/no findings on e43b40afc, independently inspecting all801 TAPs/run records,
manifest, exact+4/-3 and unchanged assertion/product bytes. Verdict ready in his REVIEW-r1.md;
comment/record await Sage request route. Dewey landing review and Sage bus/12scripts gate remain.
Slot returned in0047 (0046 T3 metadata timeout superseded), no tests or processes running.