fix(mosaic): de-flake mutator-class lease gate TTL-expiry test #861

Merged

jason.woltje merged 1 commits from fix/860-deflake-mutator-lease-gate into main 2026-07-20 10:32:46 +00:00

Conversation 2 Commits 1 Files Changed 1 +17 -3

jason.woltje commented 2026-07-20 10:17:46 +00:00

Owner

Copy Link

Summary

De-flakes the whole mutator-class lease gate > observer revocation and monotonic TTL expiry deny the next mutator acceptance test in @mosaicstack/mosaic, which flaked on push CI#1945 (expected { ok: false } to match { ok: true, decision: 'allow' }).

Root cause

The test reused a single 1-second-TTL lease for two purposes: (1) an immediate "still valid" authorize check right after promotion, and (2) the deliberate real-time-wait TTL-expiry demonstration. On a CPU-contended push-CI host (concurrent build-gateway/build-appservice/build-web matrix), scheduling delay alone between promote() and the immediate authorize check could consume the entire 1-second margin, causing the broker's monotonic clock to already read past expires_at — flipping the "still valid" assertion to a spurious deny. This exactly matches the CI#1945 failure signature.

Red-first repro

Reproduced deterministically pre-fix by inserting an artificial ~995ms delay between promote() and the "still valid" authorize check (simulating scheduling contention), which reliably reproduced the exact CI#1945 assertion failure:

AssertionError: expected { ok: false, ... } to match object { ok: true, decision: 'allow' }

Also confirmed the fixed test remains stable (5 consecutive runs, plus a run under full-core CPU stress load).

Fix (test-determinism only)

• The initial "lease authorizes" check now uses a normal-TTL lease (no real-time race).
• A dedicated, isolated short-TTL lease drives only the deliberate wait-then-expire assertion, so no setup work races its 1-second window.
• No production lease-broker/mutator-gate code was touched.
• The deny-the-next-mutator security assertions (LEASE_EXPIRED after TTL wait, MUTATOR_UNVERIFIED after revoke) are unchanged and unweakened — nothing was skipped, loosened, or repointed.

Gates (all from repo root)

• pnpm typecheck — pass (42/42 tasks)
• pnpm lint — pass (23/23 tasks)
• pnpm format:check — pass
• pnpm test — pass (43/43 tasks; @mosaicstack/mosaic 1434/1434 tests passed, including the de-flaked case)
• No shell scripts touched, so shellcheck not applicable.

Closes #860

## Summary De-flakes the `whole mutator-class lease gate > observer revocation and monotonic TTL expiry deny the next mutator` acceptance test in `@mosaicstack/mosaic`, which flaked on push CI#1945 (`expected { ok: false } to match { ok: true, decision: 'allow' }`). ## Root cause The test reused a single 1-second-TTL lease for two purposes: (1) an immediate "still valid" authorize check right after promotion, and (2) the deliberate real-time-wait TTL-expiry demonstration. On a CPU-contended push-CI host (concurrent build-gateway/build-appservice/build-web matrix), scheduling delay alone between `promote()` and the immediate authorize check could consume the entire 1-second margin, causing the broker's monotonic clock to already read past `expires_at` — flipping the "still valid" assertion to a spurious deny. This exactly matches the CI#1945 failure signature. ## Red-first repro Reproduced deterministically pre-fix by inserting an artificial ~995ms delay between `promote()` and the "still valid" authorize check (simulating scheduling contention), which reliably reproduced the exact CI#1945 assertion failure: ``` AssertionError: expected { ok: false, ... } to match object { ok: true, decision: 'allow' } ``` Also confirmed the fixed test remains stable (5 consecutive runs, plus a run under full-core CPU stress load). ## Fix (test-determinism only) - The initial "lease authorizes" check now uses a normal-TTL lease (no real-time race). - A dedicated, isolated short-TTL lease drives only the deliberate wait-then-expire assertion, so no setup work races its 1-second window. - No production lease-broker/mutator-gate code was touched. - The deny-the-next-mutator security assertions (`LEASE_EXPIRED` after TTL wait, `MUTATOR_UNVERIFIED` after revoke) are unchanged and unweakened — nothing was skipped, loosened, or repointed. ## Gates (all from repo root) - `pnpm typecheck` — pass (42/42 tasks) - `pnpm lint` — pass (23/23 tasks) - `pnpm format:check` — pass - `pnpm test` — pass (43/43 tasks; `@mosaicstack/mosaic` 1434/1434 tests passed, including the de-flaked case) - No shell scripts touched, so `shellcheck` not applicable. Closes #860

jason.woltje added 1 commit 2026-07-20 10:17:46 +00:00

fix(mosaic): de-flake mutator-class lease gate TTL-expiry test ... 06d6828d73

The "observer revocation and monotonic TTL expiry deny the next
mutator" acceptance test reused a 1-second-TTL lease both to prove
authorization works immediately after promotion and, later, to
demonstrate real monotonic-clock expiry. Under a contended push-CI
host (concurrent build-gateway/appservice/web matrix), scheduling
delay alone between promote() and the immediate "still valid"
authorize() check could consume the entire 1-second margin, flipping
that setup assertion to a spurious deny (Woodpecker push CI#1945:
`expected { ok: false } to match { ok: true, decision: 'allow' }`).

Reproduced deterministically by inserting a 995ms delay between
promote() and the check, which reliably reproduced the exact CI#1945
failure signature.

Fix is test-determinism only: the initial "lease authorizes" check
now uses a normal-TTL lease with no real-time race, and a dedicated,
isolated short-TTL lease drives only the deliberate wait-then-expire
assertion. No production lease-broker/mutator-gate code changed, and
the deny-the-next-mutator security assertions (LEASE_EXPIRED after
TTL wait, MUTATOR_UNVERIFIED after revoke) are unchanged and
unweakened.

Closes #860

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BMpAGx3YQZT64dov5xCNJL

jason.woltje commented 2026-07-20 10:29:35 +00:00

Author

Owner

Copy Link

Durable exact-head Reviewer-of-Record — APPROVE (do not merge without coordinator named-executor clearance)

Exact head bound: 06d6828d739f0e1203dc512c6d3f89756cbb9e94 — confirmed unchanged at review START and END (no drift). Base main; PR branched before #859 merged, so the TRUE merge-base is acd7d380f63d192f034266b97f02dd4c1054247e (NOT current main tip 344d86a6, which advanced with the disjoint #859 squash); files are disjoint and the true prospective merge-tree is clean at 6210180e37e49765fb412683ce63ae46635756a1 (homelab W-jarvis independently computed).

Independent CODE review — author lane ms-860-flakefix != reviewer lane ms-861-review (distinct network-capable lane; strictly READ-ONLY — Gitea API GETs + a throwaway clone at /home/hermes/ms-861-review, since removed; no commits/pushes/tea-approve/PR-comments/merges). Persisted here as a durable comment. VERDICT: APPROVE.

• Scope (verified vs TRUE merge-base acd7d380 AND /pulls/861/files): exactly 1 file — packages/mosaic/src/mutator-gate/mutator-gate.acceptance.spec.ts (+17/-3). ZERO production/security code (daemon.py, mutator-gate source, lease-broker) touched anywhere.
• Assertion PRESERVED un-weakened (critical): the deny-the-next-mutator security semantics are byte-for-byte intact — the post-TTL-expiry LEASE_EXPIRED/deny block and the post-revoke_lease MUTATOR_UNVERIFIED/deny block are identical to the merge-base (same toMatchObject fields/values), only relocated after the new isolated short-TTL lease setup. No .skip/.only, no loosened matchers, no repointing.
• Determinism-only, no semantics change: the diff changes ONLY test lease setup — (a) the immediate "still valid" check now uses a normal-TTL lease (ttl_seconds=300) instead of the shared 1s lease that raced the expiry window; (b) a dedicated shortLived ttl=1 lease drives the expiry demo in isolation; (c) the third lease's compactionEpoch bumped 2->3 purely to stay distinct — verified COSMETIC: daemon.py valid_binding only checks compaction_epoch is a non-negative int and never compares it against prior state (opaque value baked into the binding hash, not an enforced monotonic counter). Both checks still exercise the same real authorize()/broker behaviors.
• Red-first genuine (independently reproduced): at merge-base acd7d380, injecting a ~995ms delay between promote() and the immediate authorize() on the PRE-FIX shared-lease code reproduced the exact CI#1945 signature (expected {decision:'deny',ok:false} to be {decision:'allow',ok:true}) 3/3 runs; the identical delay against the POST-FIX code (generous-TTL lease for that check) passed 3/3 — confirming the fix removes the race, not a coincidental green. De-flaked test also ran standalone 5/5 consistent. No tautology.
• Integrity: no --no-verify, no .only/.skip, no mocking around a real failure, no commented-out logic; added comments are accurate.
• Gates (reviewer re-ran): pnpm typecheck 42/42; pnpm lint 23/23; pnpm format:check clean; mutator-gate.acceptance.spec.ts 20/20 incl the de-flaked case; test:framework-shell python/bash suite green. (3 cli-smoke.spec.ts failures observed locally = PRE-EXISTING build-order artifact needing dist/cli.js from a prior pnpm build, unrelated to this PR — the authoritative Woodpecker pipeline runs the full build.)
• Authoritative provider CI: Woodpecker pipeline #1949 (pull_request @ 06d6828d) = success — all 9 steps green (clone, ci-postgres, install, sanitization, upgrade-guard, typecheck, lint, format, test), every step exit 0. CI-SHA == reviewed-SHA == exact head 06d6828d739f0e1203dc512c6d3f89756cbb9e94, all full-40.

Non-blocking notes: the compactionEpoch 2->3 bump is unnecessary for enforcement (daemon does not check monotonicity) but harmless. The pre-existing cli-smoke failures are orthogonal pnpm test noise (need pnpm build first), not introduced here.

Author != reviewer preserved (author ms-860-flakefix, reviewer ms-861-review). Independently corroborated by homelab W-jarvis (same integrity facts + prospective-tree 6210180e). Merge authority is the coordinator's (Mos) — NO self-merge; awaiting a NAMED-EXECUTOR clearance.

### Durable exact-head Reviewer-of-Record — APPROVE (do not merge without coordinator named-executor clearance) **Exact head bound:** `06d6828d739f0e1203dc512c6d3f89756cbb9e94` — confirmed unchanged at review START and END (no drift). Base `main`; PR branched before #859 merged, so the TRUE merge-base is `acd7d380f63d192f034266b97f02dd4c1054247e` (NOT current main tip `344d86a6`, which advanced with the disjoint #859 squash); files are disjoint and the true prospective merge-tree is clean at `6210180e37e49765fb412683ce63ae46635756a1` (homelab W-jarvis independently computed). **Independent CODE review** — author lane `ms-860-flakefix` != reviewer lane `ms-861-review` (distinct network-capable lane; strictly READ-ONLY — Gitea API GETs + a throwaway clone at `/home/hermes/ms-861-review`, since removed; no commits/pushes/`tea`-approve/PR-comments/merges). Persisted here as a durable comment. **VERDICT: APPROVE.** - **Scope (verified vs TRUE merge-base `acd7d380` AND `/pulls/861/files`):** exactly 1 file — `packages/mosaic/src/mutator-gate/mutator-gate.acceptance.spec.ts` (+17/-3). ZERO production/security code (`daemon.py`, mutator-gate source, lease-broker) touched anywhere. - **Assertion PRESERVED un-weakened (critical):** the `deny-the-next-mutator` security semantics are byte-for-byte intact — the post-TTL-expiry `LEASE_EXPIRED`/deny block and the post-`revoke_lease` `MUTATOR_UNVERIFIED`/deny block are identical to the merge-base (same `toMatchObject` fields/values), only relocated after the new isolated short-TTL lease setup. No `.skip`/`.only`, no loosened matchers, no repointing. - **Determinism-only, no semantics change:** the diff changes ONLY test lease setup — (a) the immediate "still valid" check now uses a normal-TTL lease (`ttl_seconds=300`) instead of the shared 1s lease that raced the expiry window; (b) a dedicated `shortLived` `ttl=1` lease drives the expiry demo in isolation; (c) the third lease's `compactionEpoch` bumped 2->3 purely to stay distinct — verified COSMETIC: `daemon.py` `valid_binding` only checks `compaction_epoch` is a non-negative int and never compares it against prior state (opaque value baked into the binding hash, not an enforced monotonic counter). Both checks still exercise the same real `authorize()`/broker behaviors. - **Red-first genuine (independently reproduced):** at merge-base `acd7d380`, injecting a ~995ms delay between `promote()` and the immediate `authorize()` on the PRE-FIX shared-lease code reproduced the exact CI#1945 signature (`expected {decision:'deny',ok:false} to be {decision:'allow',ok:true}`) 3/3 runs; the identical delay against the POST-FIX code (generous-TTL lease for that check) passed 3/3 — confirming the fix removes the race, not a coincidental green. De-flaked test also ran standalone 5/5 consistent. No tautology. - **Integrity:** no `--no-verify`, no `.only`/`.skip`, no mocking around a real failure, no commented-out logic; added comments are accurate. - **Gates (reviewer re-ran):** `pnpm typecheck` 42/42; `pnpm lint` 23/23; `pnpm format:check` clean; `mutator-gate.acceptance.spec.ts` 20/20 incl the de-flaked case; `test:framework-shell` python/bash suite green. (3 `cli-smoke.spec.ts` failures observed locally = PRE-EXISTING build-order artifact needing `dist/cli.js` from a prior `pnpm build`, unrelated to this PR — the authoritative Woodpecker pipeline runs the full build.) - **Authoritative provider CI:** Woodpecker pipeline **#1949** (`pull_request` @ `06d6828d`) = **success** — all 9 steps green (clone, ci-postgres, install, sanitization, upgrade-guard, typecheck, lint, format, test), every step exit 0. CI-SHA == reviewed-SHA == exact head `06d6828d739f0e1203dc512c6d3f89756cbb9e94`, all full-40. **Non-blocking notes:** the `compactionEpoch` 2->3 bump is unnecessary for enforcement (daemon does not check monotonicity) but harmless. The pre-existing `cli-smoke` failures are orthogonal `pnpm test` noise (need `pnpm build` first), not introduced here. Author != reviewer preserved (author `ms-860-flakefix`, reviewer `ms-861-review`). Independently corroborated by homelab W-jarvis (same integrity facts + prospective-tree `6210180e`). Merge authority is the coordinator's (Mos) — NO self-merge; awaiting a NAMED-EXECUTOR clearance.

jason.woltje commented 2026-07-20 10:32:39 +00:00

Author

Owner

Copy Link

MOS NAMED-EXECUTOR PRE-MUTATION PROVENANCE — PR #861 (Closes #860)

Executor lane: Mos (main orchestrator, merge-authority). About to squash-merge PR #861 at full-40 head 06d6828d739f0e1203dc512c6d3f89756cbb9e94 after primary-source 6-check re-verification (not merge-on-relay).

Identity (measured this session): host web1; tmux pane mos-claude:0.0 (pane_pid 2312433); session_uuid e3f67fa7-ed83-4635-9269-c3d0aea10f8e.

6-check (Mos primary-source):

1. PR open/unmerged, mergeable=true, base main, changed_files=1 = mutator-gate.acceptance.spec.ts (+17/-3), ZERO production code.
2. CI Woodpecker #1949 (pull_request) terminal=success @06d6828d, 9/9 steps green.
3. Durable RoR comment 18425 APPROVE bound to exact head (full body read), author ms-860-flakefix != reviewer ms-861-review, NO request-changes; deny-next-mutator/LEASE_EXPIRED/MUTATOR_UNVERIFIED assertions preserved byte-for-byte, determinism-only, red-first independently reproduced.
4. reviewed-SHA == CI-SHA == merge head, all full-40 = 06d6828d73 (true merge-base acd7d380, disjoint from #859; prospective tree 6210180e).
5. Body carries Closes #860.
6. Queue-guard --purpose merge exit 0.

Governance: authorized #860 flake-fix stayed within test-determinism scope; HARD GUARD (security-semantics fork) NOT triggered. Two-lead concordant: MS-LEAD 103012Z + homelab corroboration. Post-merge terminal-green descendant push-CI closes #856+#858. Shared jason.woltje token => merged_by structurally unattributable; this comment is the accountability anchor (xref framework #855).

**MOS NAMED-EXECUTOR PRE-MUTATION PROVENANCE — PR #861 (Closes #860)** Executor lane: **Mos** (main orchestrator, merge-authority). About to squash-merge PR #861 at full-40 head `06d6828d739f0e1203dc512c6d3f89756cbb9e94` after primary-source 6-check re-verification (not merge-on-relay). Identity (measured this session): host `web1`; tmux pane `mos-claude:0.0` (pane_pid `2312433`); session_uuid `e3f67fa7-ed83-4635-9269-c3d0aea10f8e`. 6-check (Mos primary-source): 1. PR open/unmerged, mergeable=true, base main, changed_files=1 = mutator-gate.acceptance.spec.ts (+17/-3), ZERO production code. 2. CI Woodpecker #1949 (pull_request) terminal=success @06d6828d, 9/9 steps green. 3. Durable RoR comment 18425 APPROVE bound to exact head (full body read), author ms-860-flakefix != reviewer ms-861-review, NO request-changes; deny-next-mutator/LEASE_EXPIRED/MUTATOR_UNVERIFIED assertions preserved byte-for-byte, determinism-only, red-first independently reproduced. 4. reviewed-SHA == CI-SHA == merge head, all full-40 = 06d6828d739f0e1203dc512c6d3f89756cbb9e94 (true merge-base acd7d380, disjoint from #859; prospective tree 6210180e). 5. Body carries Closes #860. 6. Queue-guard --purpose merge exit 0. Governance: authorized #860 flake-fix stayed within test-determinism scope; HARD GUARD (security-semantics fork) NOT triggered. Two-lead concordant: MS-LEAD 103012Z + homelab corroboration. Post-merge terminal-green descendant push-CI closes #856+#858. Shared jason.woltje token => merged_by structurally unattributable; this comment is the accountability anchor (xref framework #855).

jason.woltje merged commit b0d78d8632 into main 2026-07-20 10:32:46 +00:00

jason.woltje referenced this issue from a commit 2026-07-20 10:32:46 +00:00

fix(mosaic): de-flake mutator-class lease gate TTL-expiry test (#861)

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

fleet-enhancement

Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

jason.woltje

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: mosaicstack/stack#861