docs: record processing gate failures and timer fixture correction

This commit is contained in:
rocko committed 2026-10-21 21:24:19 -05:00
1 parent 4db8b57785
commit 0f8235277e
1 file changed
+394
@@ -0,0 +1,394 @@
# Row 191 stopRun inspection round
Design: ../DESIGN-r2.md, approved by Dewey in 29473; scope sage-2244;
per-run contract ../STOP-RUN-CONTRACT.md, honker-0079/0081. Not review-ready:
full package/shell/Docker gate and acceptance/guard mutants remain pending.
Current commit 7aa18b0f6e1a09188fb602853ebb02815b642ac9, new branch
rocko/row191-stoprun-dcd8-20261022, parent dcd8d4dda. Worktree
/mnt/storage/scratch/rocko-r191-r3. Earlier inspection f7bf7dfec is preserved.
All 79 manifest paths have identical blobs at f7bf7dfec and 7aa18b0f6;
rebase-product-diff.stat is empty, rebase-product-check.txt names the check.
The manifest SHA-256 stays
9971ac5fe56d0d36eb914ecaf2c282e14ab3cd29860fd27b268ec0baadaf0e52.
Rocko author and committer identity were selected explicitly. No attribution
lines, force-push or branch deletion. Sage publishes through jarvis.
## Changes Dewey must review: two focused-run fixes
1. packages/bus/src/broker.mjs, requestHost: the new processingWork guard was
initially inserted into the inner refusal logger's try block. That let the
original request reach validation, then prevented its refusal being logged
while halted, returning storage-unavailable. Move it to the main admission
try, before keys/credentials/authority/effects. The refusal logger retains
its existing unconditional audit behavior. The new host-admission test now
expects processing-halted and proves neither executor method was called.
Initial evidence: ../logs/stoprun-focused-packages_bus_tests_processing.test.mjs.log.
2. packages/bus/tests/processing.test.mjs, real socket await test: row 196's
asynchronous kernel-peer subprocess now precedes dispatch. The old 5 ms
server deadline could fire during that subprocess, before the processing
await branch extends it. This test already stubs authenticateHuman; give it
a deterministic peerReader too, with read and close, and assert exactly two
proof reads for halt/resume. Keep the 5 ms ordinary timeout and 30 ms halt
delay. Production peer discovery, proof/transport and timeout behavior are
unchanged. The test still fails if the async branch timeout is removed.
The first fixture retry omitted close(); its teardown failed and left its
own server running. Only that verified test group 2831504 was killed. Both
unsuccessful logs remain. Corrected evidence:
../logs/stoprun-focused-bus-processing-fixed-2.log (14/14).
These are diagnosed code/fixture mistakes, not declared flakes. No failed log
was removed and no existing security assertion was weakened.
## Focused evidence, sage-2250
Ran singly on 41b0cf755 because none of row 254's seven paths overlaps ours.
Sage-2265 carries this evidence to dcd8d4dda after the empty 79-path diff.
Scripted busy check exited 0 before the first file. Every test runner used its
own process group, 180 s timeout, scratch TMPDIR
/mnt/storage/scratch/rocko-r191-tmp and DOCKER_HOST=unix:///nonexistent.sock.
There was no Docker, browser, package suite, shell suite or mutant run.
| File | Final passed | Failed | Skipped |
|---|---:|---:|---:|
| processing-control/tests/stop-run.test.mjs | 12 | 0 | 0 |
| processing-control/tests/runtimes.test.mjs | 10 | 0 | 0 |
| processing-control/tests/coordinator.test.mjs | 3 | 0 | 0 |
| processing-control/tests/containers.test.mjs | 6 | 0 | 0 |
| processing-control/tests/effects.test.mjs | 5 | 0 | 0 |
| processing-control/tests/task.test.mjs | 7 | 0 | 0 |
| bus/tests/processing.test.mjs | 14 | 0 | 0 |
| bus/tests/processing-runtime.test.mjs | 2 | 0 | 0 |
| bus/tests/project-runtime.test.mjs | 3 | 0 | 0 |
| Script sum | 62 | 0 | 0 |
All final runners exited 0. The batch stopped at the initial bus failure,
then that file was diagnosed/rerun before the final two files. Counts are from
focused-run-table-final.json and include only each final successful file log.
Earlier bus failures are reported above, not silently included as green.
Approximate run span 23:59Z–00:00:41Z; start second was not recorded.
Cwd-scoped node/chromium sweep was empty at 00:00:41Z; slot returned to Sage.
## stopRun implementation
The export resolves host-owned run identity even without a bound bus claim.
It checks boot/start/group identity immediately before each signal, preserves
verified member anchors, waits 5 s before KILL and observes for 2 s. Signals
are acknowledgements only, never invented. Root/business/run cache joins the
same Promise. Atomic hard-link publication plus durable exclusive claim
coordinates other host processes. First owner/reason/deadline/result persists;
completed retries never re-escalate or overwrite that result. Halt links its
generation to the first fire receipt without relabelling the runtime's stop
cause. A later verified exit is separate immutable observation evidence.
The seam writes no bus events. Honker maps killed/alive to lifecycle events.
No global unit, live config, binding, token, slot file or live store touched.
No schema migration. First activation still requires stopping old processing
before trusted initialization and aligning the copy/image/read-only mount in
Mos's window. Existing result.json evidence remains write-once.
## Integration and outstanding work
The 12 latest-base conflict resolutions are listed in STATUS.md. New landed
host execution, publish and library sync paths are now wrapped in processing
scopes; host credential cancellation composes with its existing revocation
signal. This changes only files already owned by 191. Those wider integrations
still need the full owning-file/package gate and mutant evidence.
Fifteen proposed per-run mutants preflight exact once in stoprun-mutants.json;
none has been applied or run. The broader kill-switch acceptance/guard table
and executable mutant runner are still being completed. No kill count or
review-ready claim is made. Ask Sage for the later CLEAR when the packet is
complete, as sage-2259/2262 requires.
### Gate preparation after the focused slot
The published product implementation is unchanged. Three uncommitted test-file
strengthenings are saved in post-inspection-tests.patch: effects.test.mjs adds a
4-second bounded abort expectation for the watcher mutant; Discord's early
start fence is also checked before a malformed journal can be read; WebUI's
body limit now uses oversized valid JSON, and a direct-handler test isolates
its own origin/site guards from serve's outer fence. These edits are unrun and
are not included in the published 9971ac5f manifest. They must pass before a
new candidate is cut. Thus 62/62 is prior focused evidence, not a claim for
these changed test bytes.
The complete proposed table has 70 exact-once sites in all-mutants.json,
including expected killing test names. Static preflight is clean; no mutant
has been applied. run-mutants.py validates preflight, uses new process groups,
tracks only its own verified descendant identities, kills those groups on
timeout, checks named TAP failures, and restores all source bytes in finally.
A timeout or wrong-test failure is not counted as a valid kill. The table and
runner are prepared evidence, not a kill result. The requested nine-package,
ten-shell, seventy-mutant gate is in gate-plan.json; Sage CLEAR is pending.
## Published test-strengthening commit and full gate (sage-2269/2271)
81889b58d57889b12a584aa225e1ac26e90e5700 is one fast-forward commit on
7aa18b0f6, same branch rocko/row191-stoprun-dcd8-20261022. It commits the
three test changes saved in post-inspection-tests.patch; that patch remains
historical evidence, not an uncommitted candidate. Product implementation is
unchanged. Earlier manifest 9971ac5f belongs to 7aa18b0f6; candidate.txt now
names the manifest for 81889b58d. Sage published this tip before granting CLEAR.
The full gate checks HEAD equality and clean product bytes before its first
file. Changed test files run singly, then nine packages, all ten shell suites,
then seventy mutants. Each step gates on its exit status and retains its full
log. run-gate.sh uses pipefail and an EXIT trap for the label-verified Compose
network rocko-r191-gate_default, project rocko-r191-gate. TMPDIR is
/mnt/storage/scratch/tmp/rocko-r191-gate. Docker is available only for test-task
and test-release; other steps use unix:///nonexistent.sock. Live provider calls
are disabled. No daemon, socket, volume or live configuration changes.
Gate start: 2026-10-22T00:18:00Z. Results remain pending until gate-results.json
and mutant-results.json provide actual exit statuses and counts.
### First full-gate stop and fixture correction (sage-2274)
The 81889b58d run stopped at its first failure, 00:18:00Z–00:19:21Z.
27 files completed: 268 passed, 1 failed, 0 skipped. Packages, shells and
mutants did not start. Full logs/results are in gate-attempt-81889/.
The tester refusal test's fixture requested tester:true over TCP, which row
196 now refuses before route handling. This is a stale fixture, not a flake.
The correction uses the existing startTesterSocket helper, a private fixture
root, socketPath requests, TESTER_ORIGIN and no human identity header. Every
assertion remains; the request now reaches the intended processing refusal
rather than failing listener setup or the outer identity fence. A grep of all
changed test files found no other tester:true TCP setup. The cwd node/chromium
sweep returned []; no Docker test workload ran. Commit c78f48946 is a single
fast-forward test-only fix on 81889b58d, authorized by sage-2274. The complete
gate restarts from its first single file only after Sage publishes this tip.
Sage-2275 published c78f48946 and granted full restart. Second attempt begins
2026-10-22T00:21:03Z; any further failure stops before additional edits.
Candidate manifest b49fd493dc75ef2484ffc255213dc8eeaaaa348dee416c8d26d1c2bab8e20067.
### Second attempt stopped; no further changes (sage-2275)
Run 00:21:03Z–00:23:50Z on published c78f48946: all 27 changed singles
269/269, processing-control 61/61, bus 646/646. The CLI package emitted 26
failed subtests before its verified own runner group 3119310 was stopped.
Its result is interrupted, not a complete package count. Six gate-lane image
checks refuse because scripts/testing/lane-image.mjs pins the old four COPY
lines and 191 adds processing-control to Containerfile. Twenty launch-account,
launch-limit-checks, launch-limits and launch-routes cases hit
processing-uninitialized on fixture roots with no explicit control init.
No edits, retries, later packages, shells, Docker test workloads or mutants
followed this second failure. Parent runner exited; cwd node/chromium sweep
returned [] at 00:23:50Z. Slot returned to Sage at 00:24:05Z with the exact
failures and a request for scope on image-manifest and fixture initialization.
This is an integration failure, not a declared flake or a passing gate.
## Image-input manifest and launcher fixture scope (sage-2278, sage-2282)
Commit eb3abba12 is one fast-forward commit on c78f48946. This is an image-input
manifest change: scripts/testing/lane-image.mjs now pins the exact Containerfile
COPY of packages/processing-control/src/state.mjs and entrypoint.mjs and hashes
both named source paths, without a wildcard. Its only two changed lines add
those inputs. Without them, a lane could reuse a stale image after a reader or
entrypoint change. The image test fixtures copy both files and each has its
own receipt-invalidating test. Mutant KS56 drops state.mjs from the hash list;
its named killer is the state.mjs receipt-invalidating test. Static preflight
is 71 sites / 0 errors. No new run result is claimed before CLEAR.
The launch-account, launch-limit-checks, launch-limits and launch-routes fixtures
now explicitly call initializeControl(f.dataRoot) before constructing their
launchers. Existing assertion lines are byte-identical. A new launch-limit-checks
test leaves initialization off and asserts processing-uninitialized, no
mayLaunch/authorizeLaunch, and no registered session. A live root gets control
initialized by the bus broker. The re-pin order is bus start before any seat
launch; a launcher started before that refuses. This supplements the existing
window precondition to stop old processing before the first trusted init.
Sage-2282 supersedes the test-only image ruling: lane-image.mjs is a product
blob, so mos-1200 carries no earlier results. The complete plan restarts from
the first single file, all nine packages, all ten shells and all 71 mutants
only after publication and Sage CLEAR. Earlier attempts remain in their
separate gate-attempt directories; neither is a passing full gate.
Sage-2283 published eb3abba12. Current 85-path candidate manifest is
01a2b33651414c9b0cb094d2108f9e39c48a41b79974dcc16bdec374f8371091.
The gate prints HEAD and this digest, verifies every manifest blob before its
first test, and carries no old results. Awaiting CLEAR after Gander 253.
## Full restart on eb3abba12 stopped at Discord (sage-2286)
Verified HEAD and all 85 manifest blobs before tests. Run span:
2026-10-22T00:34:50Z–00:47:33Z. No old result carried.
| Step | Passed | Failed | Skipped | Exit |
|---|---:|---:|---:|---:|
| 32 changed test files singly | 304 | 0 | 0 | 0 |
| processing-control | 61 | 0 | 0 | 0 |
| bus | 646 | 0 | 0 | 0 |
| cli | 761 | 0 | 0 | 0 |
| discord | 190 | 2 | 0 | 1 |
Singles sum independently confirmed by awk over all 32 TAP logs. Discord's
binding.test.mjs:201 STOP check and recover.test.mjs:145 ready recovery check
both get processing-uninitialized before the expected behavior. Those older
fixture roots do not explicitly initialize control. The gate stopped at this
step; no subsequent package, shell suite, Docker test workload or mutant ran.
Docker availability was checked read-only (29.7.2); no daemon/volume changes.
The EXIT trap completed. Cwd node/chromium sweep returned [] at 00:47:49Z,
source tree remains clean, and the slot was handed back at 00:47:58Z.
No test or product edits or retries followed this failure; Sage has the table
and the fixture-initialization recommendation. This is not a green full gate.
## Discord fixture init and all-package static sweep (sage-2294)
Commit e6800380dc0765ce3f1477044e2ce4f773ebb43c is test-only and fast-forwards eb3abba12.
Only binding.test.mjs and recover.test.mjs change: one initializeControl import
and one initializeControl(dep.dataRoot) call in each affected CLI fixture.
Every original assertion line is retained. Product blobs are unchanged.
FIXTURE-SWEEP.md and fixture-sweep.json enumerate 338 source call-site hits,
with the two fixes and the already-initialized chains, plus the screens for
rootless optional contexts and read-only/usage CLI calls. The generator reads
repository source only and invokes no fixture. Deliberate missing/deleted/
corrupt-state refusal tests remain negative; no shared helper masks them.
Current manifest be36af8f0855f57336e6a33283aa0a8e94297f338b1f8180936cbccf1119c60a, 87 paths. Sage-2294 carries only unchanged
processing-control 61/61, bus 646/646, CLI 761/761 and unchanged singles from
gate-attempt-eb3/. The resume must run both touched files wholly, whole Discord,
then the never-run WebUI/tasks/seat/project/library packages, every shell suite
and all 71 mutants. No result is claimed for the new test bytes before CLEAR.
The live initialization order remains bus start before any seat or Discord
start; no fixture change initializes or repairs a live root.
## Resume on e6800380d stopped at WebUI (sage-2311)
Start 2026-10-22T01:27:40Z; failed-step exit 01:28:45Z. Startup verified HEAD,
manifest be36af8f and all 87 blobs, and read-only Docker availability (29.7.2).
| New step | Passed | Failed | Skipped | Exit |
|---|---:|---:|---:|---:|
| binding.test.mjs whole | 15 | 0 | 0 | 0 |
| recover.test.mjs whole | 5 | 0 | 0 | 0 |
| Discord whole | 192 | 0 | 0 | 0 |
| WebUI whole | 314 | 4 | 1 | 1 |
Carried unchanged under sage-2294: 32 singles 304/304, processing-control 61/61,
bus 646/646 and CLI 761/761, from gate-attempt-eb3/. These are explicitly carried
results, not fresh runs on e6800380d.
Four WebUI failures are retained in gate-logs/package-webui.log:
- board-held.test.mjs:91: processing's config/state GETs omit redirect:manual.
- board-held.test.mjs:194 and send-browser.test.mjs:58: processing's independent
hash/focus/interval refresh continues to read after the shell signs out.
- shell.test.mjs:104: the added control bar makes the short desktop page 1095px
high at a 1000px viewport. Its new height is not accounted for by the existing
frame/sidebar/footer layout.
Static diagnosis: processing.js ignores the existing body.signed-out state and
mosaic-signed-out event that app.js and bus.js already share. Its GETs also lack
manual redirect handling. processing.css adds an external bar while shell.css's
frame minimum and sidebar heights still subtract only bar-h and foot-h. These
are product integration defects; no assertions were weakened and no fix was
made after the failed step. Sage received the table and recommendations.
One TAP skip: the editor rebuild requires exactly esbuild 0.28.2; installed
shared node_modules has 0.28.1. Full skip line is in tap-skips.json. No shell
suite ran, so there are no ^skip shell lines (skip-lines.txt remains empty).
No tasks/seat/project/library suite, shell Docker workload or mutant ran after
WebUI failed. Parent runner exited, EXIT cleanup completed, cwd node/chromium
sweep returned [] at 01:29:00Z, source tree is clean, and slot returned to Sage.
No retries or source edits were made. This is not a passing gate.
## Console shared sign-out and bounded desktop layout (sage-2317)
Commit 8e1f55c1db6a096e227b5114cd13a7142493b112, fast-forward on e6800380d, changes only the
processing assets and their owning two tests. Current 87-path manifest:
d3b014e6a0e9e547a29ae780eb27b1c0e57cc7b0e73dc49954abbff623929f41.
All three processing fetch sites use redirect:manual. responseBody dispatches
the existing mosaic-signed-out event for an opaque redirect, or its signIn
variant for exactly 401/sign-in required. No independent sign-out state is
added: refresh/action guards read body.signed-out. The existing event cancels
the interval, invalidates pending reads, clears stale state and disables
writes. In-flight answers cannot restore state after sign-out. Initial work
waits for DOMContentLoaded because this deferred asset precedes app.js in the
HTML and needs its shared sign-out listener registered first. app.js/bus.js
and their proof baselines remain untouched.
The CSS keeps the control bar in normal document flow. On desktop, the frame
flexes into the remaining viewport rather than imposing another full-height
minimum below the bar. Fixed navigation retains the shell's command/footer
bounds and existing responsive column widths. Mobile retains natural flow.
No short-page, sidebar or footer assertion is changed; rendering remains to
be verified on the next authorized gate, not claimed from static checks.
The VM fixture models the existing body class, document events and DOM ready;
its original assertion lines are unchanged. Six new browser cases cover config,
state and POST opaque redirects and sign-in-required replies, manual redirect
requests, interval cancellation and no subsequent fetch from forced refreshes
or clicks. KS57 removes manual redirect from the config GET; KS58 removes the
refresh signed-out guard. Both name the config-redirect browser case as their
killer. Static preflight: 73 exact-once sites / 0 errors. Syntax checks pass;
no owning test or mutant has run on this commit.
A product blob changes, so sage-2317/mos-1200 carries nothing. The next CLEAR
runs all changed files singly, all nine package suites, all ten shell suites
with Docker available and all 73 mutants. The editor rebuild's esbuild0.28.1
vs pinned0.28.2 skip remains host drift and is reported separately. Earlier
failed attempts remain immutable in gate-attempt-* directories.
## Rebase onto row 202 (sage-2329)
New branch rocko/row191-stoprun-ddfb-20261022, tip eafcbd3e7d6ef6e259fb59d87a7b0857b42337aa,
base ddfb29af7. The prior published branch is preserved; no force-push, delete
or old-name reuse. Six Rocko commits rebased, with one manual conflict in
scripts/mosaic's human CLI case: prepend halt|resume|processing and retain
row 202's appended access. packages/cli/src/cli.mjs auto-merges retain the
access import/branch and all three processing commands. CLI README retains
both the Access and Instance processing switch sections.
The full old/new tip diff has 15 paths (base plans/queue/logs, row 202's new
access files and its three shared CLI paths), 2327 insertions/32 deletions.
Within row 191's owned paths only CLI README, cli.mjs and scripts/mosaic differ,
with the row 202 additions. rebase-ddfb-log.txt, rebase-ddfb-tip-diff.stat,
rebase-ddfb-owned-diff.stat and rebase-ddfb-range-diff.txt retain exact evidence.
Manifest fd0f2740b1cccadaaad8c613640fdddb0497cdab33ec2c0fd11debf3cae6c4ee, 87 candidate paths. The package command lists were
regenerated from this base; CLI includes access.test.mjs. All 73 mutant sites
must preflight again. No tests ran, no result carries, and the complete gate
waits for publication and Sage CLEAR after Megavolt's debug run.
Sage-2333 published eafcbd3e7. Their per-file patch-id comparison matches
the prior six-commit range except the exactly ruled scripts/mosaic dispatch
line; binary/secrets/attribution checks pass. Gate remains unrun pending CLEAR.
## eafcbd3e7 full gate stopped on timer fixture attribution (sage-2339)
Run 2026-10-22T02:18:04Z–02:19:25Z. HEAD, 87 blobs and fd0f2740 manifest
verified. Short mktemp TMPDIR /mnt/storage/scratch/tmp/gxbC; named Compose
project rocko-r191-gate. Nothing carried. 32 changed files completed:
313 passed, 6 failed, 0 skipped. All six new processing-browser sign-out
cases stop at their common precondition, processingTimers length2 vs1.
The original two browser cases pass. Their recorder attributes every 5000ms
interval to processing, but app.js:307 already arms its own freshness timer.
The intended processing interval is at processing.js:87. These new cases have
not yet exercised sign-out or killed KS57/KS58, so no behavior proof is claimed.
The gate stopped at this first red step. No package, shell, Docker workload
or mutant ran; the esbuild rebuild was not reached. Runner exited, EXIT cleanup
completed and cwd node/chromium sweep returned [] at 02:19:34Z. No source edit
or retry followed the failure. Sage received the exact failure and a proposed
test-only caller-attribution filter with the assertions unchanged. This is a
fixture error, not a passing gate or a declared flake.
## Timer recorder attribution correction (sage-2345)
Commit 4db8b57785895bc62f5f015468a69515da7fc77f, fast-forward on eafcbd3e7, is a one-line
test-only correction in processing-browser.test.mjs. Cause: app.js:307's
freshness interval is also 5000ms, and the recorder incorrectly counted it as
a processing interval. The recorder now requires the processing.js caller
stack. Every assertion remains byte-identical, including exactly one timer
before sign-out and zero after it. A second 5000ms interval from processing.js
would still be counted and fail the exactly-one assertion. No product byte
changes, and no test or mutant ran for this correction.
Manifest ffb46fbdb875f97ff2eb7a7bed8bf6d5c4df53b8a63c738ffcde5fc5f6b7f6ad, 87 paths. Full restart after publication and CLEAR,
with nothing carried. Sage-2345 counts the timer-fixture red as the first in
this 191 window; any second red stops and is escalated to Mos. Previous logs
are retained in gate-attempt-eaf/.