fix(wake): #908 unify observed_seq on a single store-side allocator (dissolve detector-private-counter seam) #915

Merged
Mos merged 2 commits from feat/wake-908-allocator into main 2026-07-26 06:02:03 +00:00
Owner

#908 — unify observed_seq on a single store-side allocator

observed_seq had two allocators: the detector's PRIVATE observed_seq_counter
(detector.sh _next_observed_seq, handed to store.sh enqueue --seq) AND the store
cursor (used by the reconciler at store+offset). That one private-counter seam is the
shared root of three independently-observed defects:

  1. #908 original — burn-before-enqueue. _next_observed_seq bumped the persistent
    counter BEFORE store.sh enqueue; an enqueue failure burned a seq that never reached
    the store → future consume --upto N hard-rejects on the interior-gap check →
    PERMANENT inbox wedge.
  2. W5 aliasing (reconcile.sh dual-allocator hazard, harness R9). Detector private
    counter vs store-cursor+offset diverge: the reconciler enumerating alpha→seq1,
    beta→seq2 while the detector's private counter is still 0 means the detector's next
    delta re-allocates seq1, ALIASING a distinct obligation → a consumer acking CONSUMED 1
    silently drops one. (Was mitigated only by the reconciler refusing to enumerate when a
    detector co-feeds — fail-closed.)
  3. LIVE pilot finding — migration-restart silent swallow. §5 migration seeds the STORE
    cursors but not the detector's private counter. After migration the detector restarts
    reading the counter at 0 → _next_observed_seq returns 1,2,3… which are ≤ the
    already-consumed store seqs → store.sh enqueue --seq hits the seq<=consumed
    idempotent-ignore (exit 0, no-op) WHILE the detector advances its per-watch hash → the
    delta is PERMANENTLY, SILENTLY SWALLOWED. The killer.

The fix — single store-side allocator (dissolves all three structurally)

  • store.sh cmd_enqueue: --seq is no longer required. Under an exclusive per-namespace
    enqueue lock (flock) the store ATOMICALLY reads the observed_seq cursor, computes
    next = observed_seq+1, writes the pending record + observed.set + the cursor as ONE
    transaction, and prints next. The cursor bump is written LAST and only if the durable
    pending/observed.set writes succeed (arrow #1 dissolved). Two concurrent enqueues take the
    lock and get distinct seqs (aliasing race dissolved).
  • Anti-swallow FAIL-LOUD: an allocated next is always > consumed (observed_seq ≥
    consumed by the contiguous-prefix invariant). The old seq<=consumed → silent no-op is
    now UNREACHABLE for an allocation; if the store ever computes next<=consumed it FAILS
    LOUD. A LEGACY explicit --seq (retained only for tests/tools that must place a specific
    seq, e.g. build a gap) that is <=consumed is REFUSED loudly — never a silent swallow
    (arrow #3 structurally impossible).
  • detector.sh: _next_observed_seq and observed_seq_counter DELETED. Enqueues WITHOUT
    --seq and captures the store-returned seq. The per-watch hash still advances only after a
    durable enqueue. cursors now reports the store's observed_seq (single SoT).
  • reconcile.sh: dual-allocator hazard GONE. Enumeration allocates via the same
    store.sh enqueue (no independent offset). The fail-closed "refuse to enumerate when a
    detector co-feeds" guard is RETIRED exactly to the extent the single allocator makes safe;
    --allow-enumerate / WAKE_RECONCILE_ALLOW_ENUMERATE remain accepted as deprecated no-ops.
    The G3 accounting invariants (0-UNACCOUNTED, R2 vacuous-pass prevention) are unchanged.
  • §5 migration: with no detector-private counter, seeding the store cursors (which
    migration already does) is now SUFFICIENT — the seam disappears; a post-migration first
    allocation is always > consumed. Any legacy on-disk observed_seq_counter is vestigial.

Red-first evidence (mutation → RED → real change → GREEN)

  • MIGRATION-RESTART killer (detector D8) — seed store consumed=observed=5, restart,
    feed one delta. GREEN: store allocates 6 (>5), depth 1, deliverable, CONSUMED 6 valid.
    RED (private-counter+silent-swallow mutation): delta swallowed — observed stuck at 5,
    depth 0.
  • Burn-before-enqueue (store T9) — force the durable pending write to fail. GREEN:
    enqueue exits non-zero, observed_seq unchanged, CONSUMED still valid, next alloc resumes
    cleanly. RED (premature cursor commit): seq burned (observed advances to 2 on the failed
    write).
  • Co-feed no-alias (reconcile R9) — detector + reconciler both feed one store. GREEN:
    every observed_seq distinct + gapless (1 2 3), CONSUMED 3 valid. RED (dual-allocator
    mutation): aliasing (1 2 2).
  • Concurrency (store T10) — two concurrent enqueues. GREEN: distinct seqs {1,2}, depth 2.
    RED (flock disabled): both allocate 1 (aliasing / lost write). SKIPs without flock.

Invariants preserved

W2 triple-ratified contiguous-gapless-prefix CONSUMED contract, §2.4 cursor semantics
(monotonic authoritative observed_seq; consumed advances only over the gapless prefix; no
regress) — unchanged and green. All existing W2/W4/W5 assertions still pass (store-ack now 11,
detector 8, reconcile 8), adapted only to the no---seq signature and the store-as-SoT
cursors read — no invariant weakened. #869 fail-closed posture kept: all callers updated
atomically; no silent fallthrough (a latent exec 8>… 2>/dev/null that would have
permanently silenced stderr was also fixed).

Gates

shellcheck 0.11 clean on all changed shell files; verify-sanitized PASS; full
pnpm run test:framework-shell rc=0 both normally AND with openssl masked from PATH (HMAC
groups SKIP); firewall clean.

Manifest bumped 0.6.0 → 0.6.1.

Closes #908
Part of #892

## #908 — unify `observed_seq` on a single store-side allocator `observed_seq` had **two allocators**: the detector's PRIVATE `observed_seq_counter` (`detector.sh _next_observed_seq`, handed to `store.sh enqueue --seq`) AND the store cursor (used by the reconciler at store+offset). That one private-counter seam is the shared root of **three independently-observed defects**: 1. **#908 original — burn-before-enqueue.** `_next_observed_seq` bumped the persistent counter BEFORE `store.sh enqueue`; an enqueue failure burned a seq that never reached the store → future `consume --upto N` hard-rejects on the interior-gap check → PERMANENT inbox wedge. 2. **W5 aliasing (reconcile.sh dual-allocator hazard, harness R9).** Detector private counter vs store-cursor+offset diverge: the reconciler enumerating alpha→seq1, beta→seq2 while the detector's private counter is still 0 means the detector's next delta re-allocates seq1, ALIASING a distinct obligation → a consumer acking CONSUMED 1 silently drops one. (Was mitigated only by the reconciler refusing to enumerate when a detector co-feeds — fail-closed.) 3. **LIVE pilot finding — migration-restart silent swallow.** §5 migration seeds the STORE cursors but not the detector's private counter. After migration the detector restarts reading the counter at 0 → `_next_observed_seq` returns 1,2,3… which are ≤ the already-consumed store seqs → `store.sh enqueue --seq` hits the `seq<=consumed` idempotent-ignore (exit 0, no-op) WHILE the detector advances its per-watch hash → the delta is PERMANENTLY, SILENTLY SWALLOWED. The killer. ### The fix — single store-side allocator (dissolves all three structurally) - **store.sh `cmd_enqueue`**: `--seq` is no longer required. Under an exclusive per-namespace enqueue lock (flock) the store ATOMICALLY reads the `observed_seq` cursor, computes `next = observed_seq+1`, writes the pending record + `observed.set` + the cursor as ONE transaction, and prints `next`. The cursor bump is written LAST and only if the durable pending/observed.set writes succeed (arrow #1 dissolved). Two concurrent enqueues take the lock and get distinct seqs (aliasing race dissolved). - **Anti-swallow FAIL-LOUD**: an allocated `next` is always `> consumed` (observed_seq ≥ consumed by the contiguous-prefix invariant). The old `seq<=consumed → silent no-op` is now UNREACHABLE for an allocation; if the store ever computes `next<=consumed` it FAILS LOUD. A LEGACY explicit `--seq` (retained only for tests/tools that must place a specific seq, e.g. build a gap) that is `<=consumed` is REFUSED loudly — never a silent swallow (arrow #3 structurally impossible). - **detector.sh**: `_next_observed_seq` and `observed_seq_counter` DELETED. Enqueues WITHOUT `--seq` and captures the store-returned seq. The per-watch hash still advances only after a durable enqueue. `cursors` now reports the store's `observed_seq` (single SoT). - **reconcile.sh**: dual-allocator hazard GONE. Enumeration allocates via the same `store.sh enqueue` (no independent offset). The fail-closed "refuse to enumerate when a detector co-feeds" guard is RETIRED exactly to the extent the single allocator makes safe; `--allow-enumerate` / `WAKE_RECONCILE_ALLOW_ENUMERATE` remain accepted as deprecated no-ops. The G3 accounting invariants (0-UNACCOUNTED, R2 vacuous-pass prevention) are unchanged. - **§5 migration**: with no detector-private counter, seeding the store cursors (which migration already does) is now SUFFICIENT — the seam disappears; a post-migration first allocation is always `> consumed`. Any legacy on-disk `observed_seq_counter` is vestigial. ### Red-first evidence (mutation → RED → real change → GREEN) - **MIGRATION-RESTART killer (detector D8)** — seed store `consumed=observed=5`, restart, feed one delta. GREEN: store allocates 6 (>5), depth 1, deliverable, CONSUMED 6 valid. RED (private-counter+silent-swallow mutation): delta swallowed — observed stuck at 5, depth 0. - **Burn-before-enqueue (store T9)** — force the durable pending write to fail. GREEN: enqueue exits non-zero, `observed_seq` unchanged, CONSUMED still valid, next alloc resumes cleanly. RED (premature cursor commit): seq burned (observed advances to 2 on the failed write). - **Co-feed no-alias (reconcile R9)** — detector + reconciler both feed one store. GREEN: every `observed_seq` distinct + gapless (`1 2 3`), CONSUMED 3 valid. RED (dual-allocator mutation): aliasing (`1 2 2`). - **Concurrency (store T10)** — two concurrent enqueues. GREEN: distinct seqs {1,2}, depth 2. RED (flock disabled): both allocate 1 (aliasing / lost write). SKIPs without flock. ### Invariants preserved W2 triple-ratified contiguous-gapless-prefix CONSUMED contract, §2.4 cursor semantics (monotonic authoritative observed_seq; consumed advances only over the gapless prefix; no regress) — unchanged and green. All existing W2/W4/W5 assertions still pass (store-ack now 11, detector 8, reconcile 8), adapted only to the no-`--seq` signature and the store-as-SoT `cursors` read — no invariant weakened. #869 fail-closed posture kept: all callers updated atomically; no silent fallthrough (a latent `exec 8>… 2>/dev/null` that would have permanently silenced stderr was also fixed). ### Gates shellcheck 0.11 clean on all changed shell files; verify-sanitized PASS; full `pnpm run test:framework-shell` rc=0 both normally AND with openssl masked from PATH (HMAC groups SKIP); firewall clean. Manifest bumped 0.6.0 → 0.6.1. Closes #908 Part of #892
jason.woltje added 1 commit 2026-07-26 05:10:03 +00:00
store.sh enqueue becomes the SOLE allocator of observed_seq: under an exclusive
enqueue lock it reads its own cursor, sets next=observed_seq+1, writes the
pending record + observed.set + the cursor as ONE transaction (cursor committed
IFF the durable write succeeds), and prints the allocated seq. The detector's
private observed_seq_counter and its --seq hand-off are deleted; the reconciler
enumerates via the same store allocator and its dual-allocator fail-closed guard
is retired. This dissolves the three defects rooted in the private-counter seam:
burn-before-enqueue, W5 co-feed aliasing, and the migration-restart silent-swallow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb
jason.woltje added 1 commit 2026-07-26 05:49:21 +00:00
T9 (burn-before-enqueue, #908 arrow #1) forced the durable pending-write
failure via `chmod 500 "$STATE_DIR"`. That works only as a non-root user:
DAC permission checks (including a read-only directory) are bypassed
entirely by root, and Woodpecker CI containers run as root — so the write
silently succeeded, observed_seq advanced, and all three "failed-write"
assertions went spuriously green. The chmod also blocked the cursor's
(observed_seq) own mktemp in the same directory, so even under a non-root
run T9 could never have caught a premature cursor commit — narrower
coverage than its comment claimed. The product code (store.sh) was already
correct: the durable pending write is textually first and the cursor write
is gated on its success (verified independently, unchanged here).

Fix: bind-mount a throwaway file over $STATE_DIR/pending.jsonl (the exact
_atomic_write rename TARGET for the durable write) so the atomic write's
`mv -f tmp target` hits EBUSY — the kernel refuses to rename any file onto
an ACTIVE MOUNT POINT. That is a structural VFS constraint, not a DAC
check: no uid, root included, can bypass it short of an explicit umount.
observed_seq is left an ordinary writable file, untouched by the mount, so
a regression that commits the cursor before/independent of the durable
write is now genuinely caught. The mount + forced enqueue run inside a
private `unshare --mount --user --map-root-user` namespace so the same
mechanism applies whether the harness runs unprivileged (dev) or as root
(CI); the bind mount is private to that namespace and vanishes the instant
it exits, so no manual cleanup step is needed afterward.

sweep: grep chmod/chattr across all packages/mosaic/framework/tools/wake/test-*.sh
-> T9's `chmod 500 "$STATE_DIR"` (fixed here) was the ONLY permission-based
fault-injection use. Every other chmod hit is `chmod +x` making a mock/
fixture script executable (benign setup, not failure injection, unaffected
by root): test-wake-reconcile.sh:68, test-wake-detector.sh:110,
test-wake-digest-hmac.sh:164, test-wake-beacon.sh:70/79/88/103/311,
test-wake-store-ack.sh:246 (mock curl/wget/nc/ssh binaries).

Verified: enqueue under the injection exits non-zero as the current
non-root user, with the process reporting uid 0 INSIDE the namespace
(genuine kernel-mapped root) and still failing EBUSY — a direct proof the
mechanism is root-proof, not just an argument. fakeroot was tried per
request but only fakes stat/chown (no real mount capability), so it is not
a meaningful vehicle for this injection and was not used as the proof.
Coverage-RED reproduction: temporarily moved the observed_seq cursor write
in store.sh to before the durable pending write (premature commit) -> T9
went RED (observed_seq wrongly advanced to 2, post-failure allocation
wrongly resumed at 3); reverted -> GREEN again, diff-clean against origin.
Full `pnpm run test:framework-shell` rc=0 (all harnesses, T9 included,
11/11 wake store/ack groups green); re-run with openssl masked from PATH
(command -v openssl confirmed empty) rc=0, wake store/ack harness
unaffected (an unrelated harness, test-wake-digest-hmac.sh, gracefully
SKIPs without openssl -- pre-existing, orthogonal, HMAC is W3/A5 not W2).
shellcheck 0.11 clean (one deliberate SC2016 on nested-shell variable
expansion, disabled inline per this repo's existing convention).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb
Author
Owner

Record of Review — PR #915 (issue #908): single store-side observed_seq allocator — FINAL, CI-GREEN

VERDICT: APPROVE — GO. REVIEWED-HEAD (full-40): eed04d55a5ecff93aeab1e53a9cb5a7b778d3200. CI: TERMINAL-GREEN at this exact head.

Review basis (product APPROVE + cumulative delta-re-review)

  • Product review: independent subagent aab36467 (opus, ≠ builder mosaic-coder) at 2ce332db6cabac979094cdbfea9436f874468841APPROVE, all 8 criteria PASS, verified against runtime; criteria 2/3/5/6 independently reproduced RED→GREEN: atomic allocate+enqueue under flock (cursor-write LAST, gated on the durable writes → arrow-#1 no burn); anti-swallow FAIL-LOUD (allocated next always >consumed; ≤consumed refused loud → arrow-#3 migration-restart swallow structurally impossible); detector private counter DELETED (store is the single SoT); concurrency flock (distinct seqs); the self-reported stderr-suppression #869-risk verified fixed (fail-loud diagnostics survive acquire/release); reconciler dual-allocator gate removed with G3 accounting intact; W2 contiguous-gapless-prefix CONSUMED + §2.4 preserved (assertions adapted, none deleted). 4 red-first: migration-KILLER D8, burn-before-enqueue T9, co-feed-no-alias R9, concurrency T10.
  • Cumulative delta-re-review (MS-LEAD, independent diff): the only change since the reviewed head is the T9 fault-injection fix (eed04d55) — test-only (test-wake-store-ack.sh); store.sh/manifest.txt byte-identical → the aab36467 product APPROVE carries. The T9 fix replaces a root-bypassable chmod 500 injection (which no-opd under root-in-CI, the CI failure) with a privilege-invariant EBUSY-on-active-bind-mount mechanism (rename onto an active mountpoint fails for any uid) run inside unshare --mount --user --map-root-user — proven under a genuine kernel-mapped uid=0, and now CI-terminal-green confirms it survives the Woodpecker container. It also fixes T9s coverage (chmod previously blocked the cursor mktemp too); coverage-RED reproduced (premature cursor commit → RED). Sweep recorded in the commit: T9 was the only permission-based fault-injection (other chmods are benign +x).

Non-blocking follow-up

obs#2 (pre-existing ungated final observed_seq cursor write; defense-in-depth, practically-unreachable) → filed as issue #917.

6-check — GO

  1. open/not-merged/mergeable base main ✓ · 2. CI TERMINAL-GREEN @ full-40 eed04d55a5ecff93aeab1e53a9cb5a7b778d3200 ✓ · 3. reviewer(aab36467)≠builder + MS-LEAD cumulative delta-re-review, durable RoR ✓ · 4. reviewed==CI==merge-head all eed04d55 ✓ · 5. body Closes #908 + Part of #892 ✓ · 6. queue-guard at merge.
    GO for id-11 + squash-merge at FULL-40 eed04d55a5ecff93aeab1e53a9cb5a7b778d3200. Closes #908 (dissolves the detector-private-counter seam: all 3 arrows incl the live pilot migration-restart swallow). Note: #916 also bumps manifest 0.6.1 → union-rebase whichever merges 2nd. Non-executor.
# Record of Review — PR #915 (issue #908): single store-side observed_seq allocator — FINAL, CI-GREEN **VERDICT: APPROVE — GO.** **REVIEWED-HEAD (full-40):** `eed04d55a5ecff93aeab1e53a9cb5a7b778d3200`. **CI: TERMINAL-GREEN** at this exact head. ## Review basis (product APPROVE + cumulative delta-re-review) - **Product review:** independent subagent `aab36467` (opus, ≠ builder `mosaic-coder`) at `2ce332db6cabac979094cdbfea9436f874468841` → **APPROVE, all 8 criteria PASS**, verified against runtime; criteria 2/3/5/6 independently reproduced RED→GREEN: atomic allocate+enqueue under flock (cursor-write LAST, gated on the durable writes → arrow-#1 no burn); anti-swallow FAIL-LOUD (allocated next always >consumed; ≤consumed refused loud → arrow-#3 migration-restart swallow structurally impossible); detector private counter DELETED (store is the single SoT); concurrency flock (distinct seqs); the self-reported stderr-suppression #869-risk verified fixed (fail-loud diagnostics survive acquire/release); reconciler dual-allocator gate removed with G3 accounting intact; W2 contiguous-gapless-prefix CONSUMED + §2.4 preserved (assertions adapted, none deleted). 4 red-first: migration-KILLER D8, burn-before-enqueue T9, co-feed-no-alias R9, concurrency T10. - **Cumulative delta-re-review (MS-LEAD, independent diff):** the only change since the reviewed head is the T9 fault-injection fix (`eed04d55`) — **test-only** (`test-wake-store-ack.sh`); `store.sh`/`manifest.txt` byte-identical → the `aab36467` product APPROVE **carries**. The T9 fix replaces a root-bypassable `chmod 500` injection (which no-opd under root-in-CI, the CI failure) with a **privilege-invariant EBUSY-on-active-bind-mount** mechanism (rename onto an active mountpoint fails for any uid) run inside `unshare --mount --user --map-root-user` — proven under a genuine kernel-mapped uid=0, and now **CI-terminal-green confirms it survives the Woodpecker container**. It also fixes T9s coverage (chmod previously blocked the cursor mktemp too); coverage-RED reproduced (premature cursor commit → RED). Sweep recorded in the commit: T9 was the only permission-based fault-injection (other chmods are benign `+x`). ## Non-blocking follow-up obs#2 (pre-existing ungated final observed_seq cursor write; defense-in-depth, practically-unreachable) → filed as issue **#917**. ## 6-check — GO 1. open/not-merged/mergeable base main ✓ · 2. **CI TERMINAL-GREEN @ full-40 `eed04d55a5ecff93aeab1e53a9cb5a7b778d3200`** ✓ · 3. reviewer(`aab36467`)≠builder + MS-LEAD cumulative delta-re-review, durable RoR ✓ · 4. reviewed==CI==merge-head all `eed04d55` ✓ · 5. body `Closes #908` + `Part of #892` ✓ · 6. queue-guard at merge. **GO for id-11 + squash-merge at FULL-40 `eed04d55a5ecff93aeab1e53a9cb5a7b778d3200`. Closes #908 (dissolves the detector-private-counter seam: all 3 arrows incl the live pilot migration-restart swallow). Note: #916 also bumps manifest 0.6.1 → union-rebase whichever merges 2nd. Non-executor.**
Mos approved these changes 2026-07-26 06:02:02 +00:00
Mos left a comment
First-time contributor

GO — Gate-16 id-11 stamp, pinned to exact FULL head eed04d55a5.
Basis: verbatim FINAL RoR (comment 19039) — product review aab36467 APPROVE (8/8; criteria 2/3/5/6 reproduced RED→GREEN incl the migration-restart killer) carrying through the cumulative test-only delta (T9's privilege-invariant EBUSY-mountpoint injection, proven under kernel-mapped uid=0 and now CI-confirmed in the Woodpecker container; chmod-sibling sweep bound recorded). With this merge #908 CLOSES: store.sh enqueue is the SOLE atomic allocator (cursor-write-last; allocation ≤ consumed is a loud impossibility), the detector-private counter is DELETED, and the dual-allocator seam that produced all three evidence arrows — the burn wedge, the W5 aliasing repro, and the live dragon-lin migration-restart silent swallow — is structurally dissolved. Defense-in-depth residual tracked #917. Part of #892. Named executor: Mos (id-11), squash pinned.

GO — Gate-16 id-11 stamp, pinned to exact FULL head eed04d55a5ecff93aeab1e53a9cb5a7b778d3200. Basis: verbatim FINAL RoR (comment 19039) — product review aab36467 APPROVE (8/8; criteria 2/3/5/6 reproduced RED→GREEN incl the migration-restart killer) carrying through the cumulative test-only delta (T9's privilege-invariant EBUSY-mountpoint injection, proven under kernel-mapped uid=0 and now CI-confirmed in the Woodpecker container; chmod-sibling sweep bound recorded). With this merge #908 CLOSES: store.sh enqueue is the SOLE atomic allocator (cursor-write-last; allocation ≤ consumed is a loud impossibility), the detector-private counter is DELETED, and the dual-allocator seam that produced all three evidence arrows — the burn wedge, the W5 aliasing repro, and the live dragon-lin migration-restart silent swallow — is structurally dissolved. Defense-in-depth residual tracked #917. Part of #892. Named executor: Mos (id-11), squash pinned.
Mos merged commit e2ec927b1c into main 2026-07-26 06:02:03 +00:00
Mos deleted branch feat/wake-908-allocator 2026-07-26 06:02:04 +00:00
Sign in to join this conversation.
No Reviewers
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#915