fix(wake): #917 gate the final observed_seq cursor write + observed.set/cursor consistency (defense-in-depth)

Defense-in-depth hardening of store.sh cmd_enqueue surfaced by the #915 review
(obs#2): the final observed_seq cursor _atomic_write was the ONE durable write not
wrapped in a failure check, and was cross-file non-atomic with the observed.set
write immediately before it. Practically unreachable today, but a cursor-write
failure after a successful observed.set write could (a) return spurious success
while the allocation stayed uncommitted, and (b) leave observed.set advanced while
the cursor lagged (cross-file inconsistent).

Fix (additive; preserves #908 exactly):
- GATE the final cursor write like the pending/observed.set writes (#908): a
  cursor-write failure is now FAIL-LOUD (non-zero + diagnostic), never swallowed.
- On cursor-write failure, ROLL observed.set BACK to its pre-write snapshot, so
  observed.set and the cursor either BOTH advance or NEITHER does — never a
  stranded observed.set entry above the cursor.

#908 invariants intact: single store-side allocator, atomic allocate+enqueue under
flock, arrow-1 no-burn (pending write still FIRST, still aborts before any cursor
advance), anti-swallow <=consumed fail-loud, W2 contiguous-prefix CONSUMED. The
cursor stays the sole COMMIT point, so a pending-ahead state is exactly the one
#908 already tolerates on its observed.set-failure path. On-disk format UNCHANGED
(read-compatible).

Test: new T11 in test-wake-store-ack.sh forces the final cursor _atomic_write to
fail (bind-mount EBUSY over observed_seq inside an unshare mount+user namespace,
seeded so the cursor READ still succeeds) after pending+observed.set succeed, and
asserts fail-loud + observed.set/cursor consistency + intact consumed prefix. RED
against pre-#917 code; all store-ack/race/reconcile/detector groups stay GREEN.
manifest bumped 0.6.8 -> 0.6.9.

Closes #917
Part of #892

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0158NZqN2n2ymKFeJAZ4GUCb

packages/mosaic/framework/tools/wake/test-wake-store-ack.sh

@@ -472,6 +472,88 @@ else
   ok
 fi
 
+echo "== T11: final observed_seq cursor write FAILURE — fail loud + observed.set/cursor stay CONSISTENT (#917 def-in-depth) =="
+(
+  WAKE_STATE_HOME="$(fresh_state t11)"
+  export WAKE_STATE_HOME
+  unset WAKE_AGENT
+  STATE_DIR="$WAKE_STATE_HOME/default"
+  # Baseline: one clean allocation. observed_seq cursor=1, observed.set={1}, the
+  # cursor FILE now exists (so it can be bind-mounted over below).
+  s1="$("$STORE" enqueue --class actionable --locators '{"n":1}')"
+  [ "$s1" = "1" ] || fail_msg "T11: baseline allocation should be 1, got '$s1'"
+  obs_before="$("$STORE" cursors | sed -n 's/^observed_seq=//p')"
+  [ "$obs_before" = "1" ] || fail_msg "T11: observed_seq should be 1 before the forced failure, got '$obs_before'"
+
+  # Force ONLY the FINAL observed_seq cursor _atomic_write to fail, AFTER the
+  # pending + observed.set writes have already succeeded. Same privilege-invariant
+  # trigger as T9 (bind-mount a throwaway file OVER _atomic_write's rename TARGET
+  # so `mv -f tmp target` hits EBUSY — the kernel refuses to rename onto an ACTIVE
+  # MOUNT POINT; a structural VFS rule root does NOT bypass), but the mount is over
+  # $STATE_DIR/observed_seq (the CURSOR) instead of pending.jsonl. pending.jsonl and
+  # observed.set stay ordinary writable files, so their writes SUCCEED and only the
+  # cursor commit fails — exactly the #917 window (cursor-write failure after a good
+  # observed.set write). The mount lives inside a private mount+user namespace
+  # (unshare --mount --user --map-root-user) so it is identical unprivileged (dev)
+  # or as root (CI) and vanishes the instant the namespace exits — no cleanup.
+  errfile="$TMP_ROOT/t11.err"
+  : >"$errfile"
+  rc=0
+  if ! command -v unshare >/dev/null 2>&1; then
+    fail_msg "T11: 'unshare' unavailable — cannot construct the privilege-invariant fault injection in this sandbox"
+    rc=99
+  else
+    ro_src="$TMP_ROOT/t11-ro-src"
+    mkdir -p "$ro_src"
+    # Seed the throwaway file with the CURRENT cursor value so the allocator's
+    # READ of observed_seq still returns 1 (an empty file would read as 0 and
+    # mis-allocate seq=1, never exercising the cursor-WRITE failure). Only the
+    # final `mv -f tmp observed_seq` commit is blocked by the mountpoint.
+    printf '%s' "$obs_before" >"$ro_src/observed_seq"
+    # shellcheck disable=SC2016 # deliberate: $1..$4 expand in the NESTED bash
+    # (inside the unshare'd namespace), not this outer single-quoted one.
+    unshare --mount --user --map-root-user bash -c '
+      set -u
+      ro_src="$1"
+      state_dir="$2"
+      store="$3"
+      errfile="$4"
+      if ! mount --bind "$ro_src/observed_seq" "$state_dir/observed_seq" 2>/dev/null; then
+        exit 98
+      fi
+      mount -o remount,ro,bind "$state_dir/observed_seq" 2>/dev/null || true
+      "$store" enqueue --class actionable --locators "{\"n\":2}" >/dev/null 2>"$errfile"
+    ' _ "$ro_src" "$STATE_DIR" "$STORE" "$errfile"
+    rc=$?
+    if [ "$rc" -eq 98 ]; then
+      fail_msg "T11: could not bind-mount over observed_seq (mount unavailable in this sandbox) — fault injection not constructed"
+    fi
+  fi
+
+  # (1) FAIL LOUD: the cursor-write failure must EXIT NON-ZERO, never a silent
+  # success (RED against pre-#917 code, whose ungated final cursor write swallows
+  # the _atomic_write failure and returns 0).
+  [ "$rc" -ne 0 ] || fail_msg "T11: an enqueue whose FINAL cursor write fails must EXIT NON-ZERO (fail loud), got rc=$rc"
+  # The loud diagnostic names the cursor write (not a generic error).
+  grep -qi 'cursor' "$errfile" || fail_msg "T11: the failure diagnostic must name the observed_seq cursor write [$(cat "$errfile")]"
+
+  # (2) The cursor did NOT advance — the allocation is NOT committed.
+  obs_after="$("$STORE" cursors | sed -n 's/^observed_seq=//p')"
+  [ "$obs_after" = "1" ] || fail_msg "T11: a failed cursor write must NOT leave the cursor advanced, got '$obs_after'"
+
+  # (3) THE #917 CONSISTENCY ASSERTION: observed.set and the cursor must NEVER be
+  # left cross-file inconsistent. observed.set must carry NO seq greater than the
+  # cursor — either the observed.set advance was rolled back (neither advances) or
+  # both advanced together; there is no state where observed.set is ahead of the
+  # cursor. RED against pre-#917 code: observed.set={1,2} while the cursor lags at
+  # 1, so seq 2 is stranded above the cursor.
+  stranded="$(awk -v c="$obs_after" 'NF && $1+0 > c' "$STATE_DIR/observed.set" 2>/dev/null || true)"
+  [ -z "$stranded" ] || fail_msg "T11: observed.set left CROSS-FILE INCONSISTENT with the cursor — seq(s) above cursor $obs_after: [$stranded]"
+
+  # (4) The consumed prefix is intact — no interior gap corrupts future consume.
+  "$STORE" consume --upto 1 >/dev/null 2>&1 || fail_msg "T11: CONSUMED 1 must remain valid after the failed cursor write (no gap)"
+) && ok
+
 echo
 if [ -s "$FAILFILE" ]; then
   echo "wake store/ack harness: FAILED ($(grep -c . "$FAILFILE") assertion(s))" >&2