diff --git a/agents/gizmoduck/work/queue-292/BUILD.md b/agents/gizmoduck/work/queue-292/BUILD.md index 0a3138f15..501c43669 100644 --- a/agents/gizmoduck/work/queue-292/BUILD.md +++ b/agents/gizmoduck/work/queue-292/BUILD.md @@ -164,35 +164,60 @@ the round key. ## Mutants agents/gizmoduck/work/queue-292/mutants.py. Each anchor appears exactly once, and each mutated -script passes `bash -n` (checked with python3 and bash only, no node). The kill lines below come -from reading the tests. None has run. +script passes `bash -n` (checked with python3 and bash only, no node). None has run. M1-M5 are +Gander's edits (changes.json) on the resolved function. M6-M9 test the resolution. M1 restores the +console-only skip, the brief's required mutant. -| id | edit | should die in | -|---|---|---| -| M1 | restore the console-only skip (brief's required mutant) | config-any-key non-console refusal; the pre-270 test; row 203 federation | -| M2 | ignore a failed target validate | config-any-key non-console refusal (exit 0, not 3) | -| M3 | no-loader target passes any later key | config-any-key missing-loader | -| M4 | malformed JSON passes as absent | config-any-key malformed JSON | -| M5 | skip the loader on an oldest-key config | config-any-key accepting loaders (no witness line) | -| M6 | probe drops the federation preference | missing-loader federation case (gets the later-key text) | -| M7 | probe drops the console preference | missing-loader console case | -| M8 | final refusal treats only console as 203's | row 203 federation test; missing-loader federation case | -| M9 | no-loader branch treats only console as 203's | missing-loader federation case | +### Kill rules (sage-3152) -M1-M5 are Gander's edits (changes.json) on the resolved function. M6-M9 test the resolution. +Each mutant runs only its killing test: `node --test --test-name-pattern '' +tests/stack-release.test.mjs`. The patterns are the test names, anchored with ^ and $, and all +four are top-level tests. A mutant counts as killed only when all three of these hold: -## Run plan (on Sage's CLEAR only, C2) +1. The run ends with `ℹ fail 1`, and the failing test is the named one. The spec reporter lists + it twice, inline and in the failing-tests summary. +2. The first error's stack names every listed stack-release.test.mjs line. +3. The log holds the listed text. The texts carry no quote characters, which node's inspect may + escape. -Gate per the brief. stack-release.test.mjs whole, the cli suite whole and the ten -`scripts/test-*.sh`, then M1-M9, each on stack-release.test.mjs whole. Every mutant gets -`git checkout`, sha256 equal to the candidate and a clean tree before the next. TMPDIR from -`mktemp -d /mnt/storage/scratch/tmp/gXXX`, pipefail, tee, `timeout -k 10` everywhere. +Anything else is a STOP that goes to Sage before the next mutant: a fail in another test or at +another line, a missing text, zero tests matched, a timeout or a crash before the assertion. A +fixture or module-load error is the second-fence trap from 264a M2. A mutant whose targeted run +passes gets one run of stack-release.test.mjs whole before it counts as a survivor. -Limits from the run 5 receipts (gizmo-271-out/r4, 01:49Z 10-23): stack-release.test.mjs whole took -682 s (duration_ms 681648), the cli suite took 624 s, and the longest shell suite took 44 s. At 3x: -2100 s for stack-release.test.mjs, alone or as a mutant run, 2100 s for the cli suite and 900 s per -shell suite. Expect about 12 + 11 + 3 minutes for the base and about 9 x 12 minutes for the mutants, -so roughly 2.2 hours in all. Sage may prefer a shorter mutant run; this packet doesn't assume one. +| id | edit | test (pattern) | stack lines | text | why that assertion | +|---|---|---|---|---|---| +| M1 | console-only skip restored | non-console target refusal | :517 (refused), :2619 | `actual: 0` | no console key, so the check returns 0 and activate B runs: exit 0, not 3 | +| M2 | failed validate ignored (`\|\| true`) | non-console target refusal | :517, :2619 | `actual: 0` | the stand-in's exit 2 is lost, so activate B runs | +| M3 | no-loader target passes any later key | missing-loader copies | :2670 | `actual: 0` | compactionReadBytes, the first case, passes with status 0 | +| M4 | malformed JSON treated as absent | missing config passes but malformed JSON | :2684 | `actual: 0` | key 2 falls through to the witness loader, which exits 0 | +| M5 | oldest-key config skips the loader | accepting loaders run validate | :2654 | `actual: []` | the witness file is never written | +| M6 | probe drops the federation preference | missing-loader copies | :2677 | `to validate configuration key: ` | federation after compactionStreams: the probe picks compactionStreams | +| M7 | probe drops the console preference | missing-loader copies | :2677 | `to validate configuration key: ` | the same for console, the first case of that loop | +| M8 | final refusal treats only console as 203's | missing-loader copies | :2677 | `is refused by old` | federation gets Gander's text, not KEY_MSG | +| M9 | no-loader branch treats only console as 203's | missing-loader copies | :2677 | `to validate configuration key: ` | federation's reason gains the later-key wording | + +M1 would also fail the pre-270 test and the row 203 federation test, and M8 the row 203 federation +test. The targeted runs don't select those, so they aren't counted. + +## Run plan (on Sage's CLEAR only, after Duckworth approves this round) + +1. stack-release.test.mjs whole, limit 2100 s. +2. The cli suite whole, `node --test tests/*.test.mjs` in packages/cli, limit 2100 s. +3. The ten `scripts/test-*.sh`, limit 900 s each. +4. Only on a green base: M1-M9 under the kill rules above. Targeted runs get 600 s and a + whole-file rerun of a survivor gets 2100 s. After each mutant: `git checkout -- + scripts/stack-release.sh`, sha256 equal to the candidate's (8f535d53...) and a clean tree. + +Every stage gets its own short TMPDIR, `mktemp -d /mnt/storage/scratch/tmp/gXXX`, directly under +/mnt/storage/scratch/tmp (sage-3152, 574 run 1's socket-path fault). Every mutant run gets one too. +All runs use pipefail, tee and `timeout -k 10`. The busy check passes before each start. A first red +in stages 1-3 stops the run. + +The limits come from the run 5 receipts (gizmo-271-out/r4, 01:49Z 10-23). stack-release.test.mjs +whole took 682 s (duration_ms 681648), the cli suite 624 s and the longest shell suite 44 s. Each +limit is at least 3x. Expected time: about 26 minutes for the base, plus about a minute per targeted +mutant (fixture setup and one test, not measured yet), plus 12 minutes per whole-file rerun. C2 (duckworth-0229, sage-3146 item 4): every B, C1 and C release now validates its copied real loader. Any red in an old cli test is a suspect from this change. It stops the run and goes to Sage diff --git a/agents/gizmoduck/work/queue-292/mutants.py b/agents/gizmoduck/work/queue-292/mutants.py index 76a5d37fd..53bcbd0fd 100644 --- a/agents/gizmoduck/work/queue-292/mutants.py +++ b/agents/gizmoduck/work/queue-292/mutants.py @@ -3,8 +3,7 @@ # the same edits on the resolved config_checks; M1 restores the console-only skip (the brief's # required mutant). M6-M9 are the resolution's: row 203's console and federation text. `apply ` # edits the worktree and prints the touched path; each edit asserts its anchor appears exactly once. -# `check` prints each anchor's count and kill. K names the test that should fail (checked by reading, -# not run). A restore is `git checkout -- `, then sha256 against the blob in BUILD.md. +# `check` prints each anchor's count and kill. K is read from the tests, not run. A restore is `git checkout -- `, then sha256 against the blob in BUILD.md. import sys WT = "/mnt/storage/scratch/gizmo-292/" S = "scripts/stack-release.sh" @@ -25,23 +24,27 @@ M = { "M9": (S, '[ "$extra" = \'"federation"\' ]; then out="it has no scripts/mosaic-config.mjs"', '[ "$extra" = \'"federation-off"\' ]; then out="it has no scripts/mosaic-config.mjs"'), } +# K: (--test-name-pattern, the test.mjs lines the failing stack must show, the text the failure +# must show). Kill rule (sage-3152): the named test fails, its first error's stack names every +# listed line, and the log holds the text (no quote characters, which node's inspect may escape). A fail anywhere else, or a crash first, is a STOP to Sage. +# A survivor of its targeted run gets one whole-file run before it counts as a survivor. K = { - "M1": "config-any-key: a non-console target refusal stops activate and rollback before any stop; also the pre-270 test and row 203's federation test", - "M2": "config-any-key: a non-console target refusal stops activate and rollback before any stop (exit 0, not 3)", - "M3": "config-any-key: missing-loader copies accept only the four oldest keys (compactionReadBytes passes)", - "M4": "config-any-key: missing config passes but malformed JSON refuses without calling the loader", - "M5": "config-any-key: accepting loaders run validate with the selected config (no witness line on an oldest-key config)", - "M6": "config-any-key: missing-loader copies ... (federation after compactionStreams gets the later-key text)", - "M7": "config-any-key: missing-loader copies ... (console after compactionStreams gets the later-key text)", - "M8": "row 203: a config whose federation key the target loader refuses ... (Gander's text, not KEY_MSG); also the missing-loader federation case", - "M9": "config-any-key: missing-loader copies ... (federation gets 'to validate configuration key')", + "M1": ('^config-any-key: a non-console target refusal stops activate and rollback before any stop$', [517, 2619], "actual: 0"), + "M2": ('^config-any-key: a non-console target refusal stops activate and rollback before any stop$', [517, 2619], "actual: 0"), + "M3": ('^config-any-key: missing-loader copies accept only the four oldest keys$', [2670], "actual: 0"), + "M4": ('^config-any-key: missing config passes but malformed JSON refuses without calling the loader$', [2684], "actual: 0"), + "M5": ('^config-any-key: accepting loaders run validate with the selected config on activate and rollback$', [2654], "actual: []"), + "M6": ('^config-any-key: missing-loader copies accept only the four oldest keys$', [2677], "to validate configuration key: "), + "M7": ('^config-any-key: missing-loader copies accept only the four oldest keys$', [2677], "to validate configuration key: "), + "M8": ('^config-any-key: missing-loader copies accept only the four oldest keys$', [2677], "is refused by old"), + "M9": ('^config-any-key: missing-loader copies accept only the four oldest keys$', [2677], "to validate configuration key: "), } cmd = sys.argv[1] if cmd == "list": print(" ".join(M)); sys.exit(0) if cmd == "check": for k, (f, a, b) in M.items(): - print(k, f, open(WT + f).read().count(a), "kill:", K[k]) + print(k, f, open(WT + f).read().count(a), "kill:", *K[k]) sys.exit(0) f, a, b = M[sys.argv[2]] s = open(WT + f).read()