fix(wake): #908 unify observed_seq on a single store-side allocator (dissolve detector-private-counter seam) #915
Reference in New Issue
Block a user
Delete Branch "feat/wake-908-allocator"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
#908 — unify
observed_seqon a single store-side allocatorobserved_seqhad two allocators: the detector's PRIVATEobserved_seq_counter(
detector.sh _next_observed_seq, handed tostore.sh enqueue --seq) AND the storecursor (used by the reconciler at store+offset). That one private-counter seam is the
shared root of three independently-observed defects:
_next_observed_seqbumped the persistentcounter BEFORE
store.sh enqueue; an enqueue failure burned a seq that never reachedthe store → future
consume --upto Nhard-rejects on the interior-gap check →PERMANENT inbox wedge.
counter vs store-cursor+offset diverge: the reconciler enumerating alpha→seq1,
beta→seq2 while the detector's private counter is still 0 means the detector's next
delta re-allocates seq1, ALIASING a distinct obligation → a consumer acking CONSUMED 1
silently drops one. (Was mitigated only by the reconciler refusing to enumerate when a
detector co-feeds — fail-closed.)
cursors but not the detector's private counter. After migration the detector restarts
reading the counter at 0 →
_next_observed_seqreturns 1,2,3… which are ≤ thealready-consumed store seqs →
store.sh enqueue --seqhits theseq<=consumedidempotent-ignore (exit 0, no-op) WHILE the detector advances its per-watch hash → the
delta is PERMANENTLY, SILENTLY SWALLOWED. The killer.
The fix — single store-side allocator (dissolves all three structurally)
cmd_enqueue:--seqis no longer required. Under an exclusive per-namespaceenqueue lock (flock) the store ATOMICALLY reads the
observed_seqcursor, computesnext = observed_seq+1, writes the pending record +observed.set+ the cursor as ONEtransaction, and prints
next. The cursor bump is written LAST and only if the durablepending/observed.set writes succeed (arrow #1 dissolved). Two concurrent enqueues take the
lock and get distinct seqs (aliasing race dissolved).
nextis always> consumed(observed_seq ≥consumed by the contiguous-prefix invariant). The old
seq<=consumed → silent no-opisnow UNREACHABLE for an allocation; if the store ever computes
next<=consumedit FAILSLOUD. A LEGACY explicit
--seq(retained only for tests/tools that must place a specificseq, e.g. build a gap) that is
<=consumedis REFUSED loudly — never a silent swallow(arrow #3 structurally impossible).
_next_observed_seqandobserved_seq_counterDELETED. Enqueues WITHOUT--seqand captures the store-returned seq. The per-watch hash still advances only after adurable enqueue.
cursorsnow reports the store'sobserved_seq(single SoT).store.sh enqueue(no independent offset). The fail-closed "refuse to enumerate when adetector co-feeds" guard is RETIRED exactly to the extent the single allocator makes safe;
--allow-enumerate/WAKE_RECONCILE_ALLOW_ENUMERATEremain accepted as deprecated no-ops.The G3 accounting invariants (0-UNACCOUNTED, R2 vacuous-pass prevention) are unchanged.
migration already does) is now SUFFICIENT — the seam disappears; a post-migration first
allocation is always
> consumed. Any legacy on-diskobserved_seq_counteris vestigial.Red-first evidence (mutation → RED → real change → GREEN)
consumed=observed=5, restart,feed one delta. GREEN: store allocates 6 (>5), depth 1, deliverable, CONSUMED 6 valid.
RED (private-counter+silent-swallow mutation): delta swallowed — observed stuck at 5,
depth 0.
enqueue exits non-zero,
observed_sequnchanged, CONSUMED still valid, next alloc resumescleanly. RED (premature cursor commit): seq burned (observed advances to 2 on the failed
write).
every
observed_seqdistinct + gapless (1 2 3), CONSUMED 3 valid. RED (dual-allocatormutation): aliasing (
1 2 2).RED (flock disabled): both allocate 1 (aliasing / lost write). SKIPs without flock.
Invariants preserved
W2 triple-ratified contiguous-gapless-prefix CONSUMED contract, §2.4 cursor semantics
(monotonic authoritative observed_seq; consumed advances only over the gapless prefix; no
regress) — unchanged and green. All existing W2/W4/W5 assertions still pass (store-ack now 11,
detector 8, reconcile 8), adapted only to the no-
--seqsignature and the store-as-SoTcursorsread — no invariant weakened. #869 fail-closed posture kept: all callers updatedatomically; no silent fallthrough (a latent
exec 8>… 2>/dev/nullthat would havepermanently silenced stderr was also fixed).
Gates
shellcheck 0.11 clean on all changed shell files; verify-sanitized PASS; full
pnpm run test:framework-shellrc=0 both normally AND with openssl masked from PATH (HMACgroups SKIP); firewall clean.
Manifest bumped 0.6.0 → 0.6.1.
Closes #908
Part of #892
Record of Review — PR #915 (issue #908): single store-side observed_seq allocator — FINAL, CI-GREEN
VERDICT: APPROVE — GO. REVIEWED-HEAD (full-40):
eed04d55a5ecff93aeab1e53a9cb5a7b778d3200. CI: TERMINAL-GREEN at this exact head.Review basis (product APPROVE + cumulative delta-re-review)
aab36467(opus, ≠ buildermosaic-coder) at2ce332db6cabac979094cdbfea9436f874468841→ APPROVE, all 8 criteria PASS, verified against runtime; criteria 2/3/5/6 independently reproduced RED→GREEN: atomic allocate+enqueue under flock (cursor-write LAST, gated on the durable writes → arrow-#1 no burn); anti-swallow FAIL-LOUD (allocated next always >consumed; ≤consumed refused loud → arrow-#3 migration-restart swallow structurally impossible); detector private counter DELETED (store is the single SoT); concurrency flock (distinct seqs); the self-reported stderr-suppression #869-risk verified fixed (fail-loud diagnostics survive acquire/release); reconciler dual-allocator gate removed with G3 accounting intact; W2 contiguous-gapless-prefix CONSUMED + §2.4 preserved (assertions adapted, none deleted). 4 red-first: migration-KILLER D8, burn-before-enqueue T9, co-feed-no-alias R9, concurrency T10.eed04d55) — test-only (test-wake-store-ack.sh);store.sh/manifest.txtbyte-identical → theaab36467product APPROVE carries. The T9 fix replaces a root-bypassablechmod 500injection (which no-opd under root-in-CI, the CI failure) with a privilege-invariant EBUSY-on-active-bind-mount mechanism (rename onto an active mountpoint fails for any uid) run insideunshare --mount --user --map-root-user— proven under a genuine kernel-mapped uid=0, and now CI-terminal-green confirms it survives the Woodpecker container. It also fixes T9s coverage (chmod previously blocked the cursor mktemp too); coverage-RED reproduced (premature cursor commit → RED). Sweep recorded in the commit: T9 was the only permission-based fault-injection (other chmods are benign+x).Non-blocking follow-up
obs#2 (pre-existing ungated final observed_seq cursor write; defense-in-depth, practically-unreachable) → filed as issue #917.
6-check — GO
eed04d55a5ecff93aeab1e53a9cb5a7b778d3200✓ · 3. reviewer(aab36467)≠builder + MS-LEAD cumulative delta-re-review, durable RoR ✓ · 4. reviewed==CI==merge-head alleed04d55✓ · 5. bodyCloses #908+Part of #892✓ · 6. queue-guard at merge.GO for id-11 + squash-merge at FULL-40
eed04d55a5ecff93aeab1e53a9cb5a7b778d3200. Closes #908 (dissolves the detector-private-counter seam: all 3 arrows incl the live pilot migration-restart swallow). Note: #916 also bumps manifest 0.6.1 → union-rebase whichever merges 2nd. Non-executor.GO — Gate-16 id-11 stamp, pinned to exact FULL head
eed04d55a5.Basis: verbatim FINAL RoR (comment 19039) — product review aab36467 APPROVE (8/8; criteria 2/3/5/6 reproduced RED→GREEN incl the migration-restart killer) carrying through the cumulative test-only delta (T9's privilege-invariant EBUSY-mountpoint injection, proven under kernel-mapped uid=0 and now CI-confirmed in the Woodpecker container; chmod-sibling sweep bound recorded). With this merge #908 CLOSES: store.sh enqueue is the SOLE atomic allocator (cursor-write-last; allocation ≤ consumed is a loud impossibility), the detector-private counter is DELETED, and the dual-allocator seam that produced all three evidence arrows — the burn wedge, the W5 aliasing repro, and the live dragon-lin migration-restart silent swallow — is structurally dissolved. Defense-in-depth residual tracked #917. Part of #892. Named executor: Mos (id-11), squash pinned.