mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

wake: digest ack watermark passes quarantined entries — consumer records deliveries that never happened #946

New Issue
Closed
opened 2026-07-30 13:11:21 +00:00 by Ghost · 3 comments
Ghost commented 2026-07-30 13:11:21 +00:00
Copy Link
Copy Source

Discovered by: mos-dt (consumer-side, from their own lane state, 2026-07-30). Severity: HIGH, raised further by MOS Ruling 4. Tracked in fleet records as fw-wake-digest-ack-watermark-passes-quarantine. Scheduled as item (2) of the wake-lane schedule: authored by pepper as its own PR immediately after #945 (the #944 predicate fix) merges.

Defect

The digest correctly QUARANTINES an entry that fails the §2.1 hard-locator gate — it is excluded from the rendered digest, dead-lettered, and alarmed. But the copy-run ack line still advances --upto past the quarantined entry's seq. The consumer then acks a watermark covering an entry that was never disclosed to it.

Two independent consequences:

  1. Step-over: the held entry is silently skipped — the consumer never learns it existed (digest shows ACTIONABLE (none) while an actionable claim is held).
  2. False witness (the worse half, MOS Ruling 4): the watermark write puts the held entry into the consumer's consumed-hash record. Live evidence: the mos-dt lane's consumed-hashes.jsonl holds observed_hash=9d0f639f…, observed_seq=63 for the heartbeat-planning source — an assertion of a delivery that never occurred, indistinguishable from a true record. The consumer's own state becomes a false witness.

This is a permanent surface, not a consequence of one bad predicate: any future quarantine cause (not just the #944 gate defect) reproduces it.

Live evidence

  • seq 63 (dragon-lin, 2026-07-30): dead-lettered at render, then acked past 254s later by a copy-run --upto 66 at consumed_seq=61. Consumer state now asserts consumption of a claim never disclosed.
  • Derived audit check (mos-dt): upto − consumed_seq MUST EQUAL entries disclosed; retrospective run flags seqs 12/15/21/66.

Ruled fix — option (c), settled beyond argument

Per MOS ruling: disclose AND clamp.

  • Disclose quarantined entries in the digest (the consumer must see that something is held, not (none)).
  • Clamp the ack watermark to the highest disclosed seq — the watermark must never advance past a quarantined entry via the ordinary path.
  • A force/override flag is the ONLY way to advance past a quarantined entry.

(Option (a), clamp-only, is the floor but rejected as insufficient: it leaves the consumer staring at ACTIONABLE (none) while an actionable claim is stranded. The false-witness evidence settles (c): a clamp alone would have prevented the corrupting write.)

Acceptance criteria (on record, pre-registered by ruling)

  1. Force flag is the ONLY way past (mos-dt gate): no ordinary code path may advance the watermark over a quarantined entry — "a flag that can be bypassed by the ordinary code path is decoration." Reviewer will hunt for ordinary-path bypasses.
  2. Repair or disposition corrupted records (MOS Ruling 4): the PR must either repair already-corrupted consumed-hash records or explicitly disposition them in the PR body. At least one known-false row exists in the field right now (the 9d0f639f…@63 row on the mos-dt lane) and must not survive silently into a lane declared fixed. Fixing the writer does not unwrite the record.
  3. Quarantined-entry disclosure appears in the rendered digest (no silent (none) while entries are held).
  4. Clamp asserted mechanically in tests: an ordinary ack against a stream containing a quarantined entry must not advance past it; with the force flag it must; the force path must be loud.

Ordering

Blocked on #945 (the #944 predicate fix) merging first — the two defects share the quarantine surface and the test harness touched by #945's Q11/fixture migration.

**Discovered by:** mos-dt (consumer-side, from their own lane state, 2026-07-30). **Severity: HIGH**, raised further by MOS Ruling 4. Tracked in fleet records as `fw-wake-digest-ack-watermark-passes-quarantine`. Scheduled as item (2) of the wake-lane schedule: authored by pepper as its own PR immediately after #945 (the #944 predicate fix) merges. ## Defect The digest correctly QUARANTINES an entry that fails the §2.1 hard-locator gate — it is excluded from the rendered digest, dead-lettered, and alarmed. But the copy-run ack line still advances `--upto` past the quarantined entry's seq. The consumer then acks a watermark covering an entry that was never disclosed to it. Two independent consequences: 1. **Step-over:** the held entry is silently skipped — the consumer never learns it existed (digest shows `ACTIONABLE (none)` while an actionable claim is held). 2. **False witness (the worse half, MOS Ruling 4):** the watermark write puts the held entry into the consumer's consumed-hash record. Live evidence: the mos-dt lane's `consumed-hashes.jsonl` holds `observed_hash=9d0f639f…, observed_seq=63` for the heartbeat-planning source — an assertion of a delivery that never occurred, indistinguishable from a true record. The consumer's own state becomes a false witness. This is a **permanent surface, not a consequence of one bad predicate**: any future quarantine cause (not just the #944 gate defect) reproduces it. ## Live evidence - seq 63 (dragon-lin, 2026-07-30): dead-lettered at render, then acked past 254s later by a copy-run `--upto 66` at `consumed_seq=61`. Consumer state now asserts consumption of a claim never disclosed. - Derived audit check (mos-dt): `upto − consumed_seq MUST EQUAL entries disclosed`; retrospective run flags seqs 12/15/21/66. ## Ruled fix — option (c), settled beyond argument Per MOS ruling: **disclose AND clamp**. - Disclose quarantined entries in the digest (the consumer must see that something is held, not `(none)`). - Clamp the ack watermark to the highest **disclosed** seq — the watermark must never advance past a quarantined entry via the ordinary path. - A force/override flag is the ONLY way to advance past a quarantined entry. (Option (a), clamp-only, is the floor but rejected as insufficient: it leaves the consumer staring at `ACTIONABLE (none)` while an actionable claim is stranded. The false-witness evidence settles (c): a clamp alone would have prevented the corrupting write.) ## Acceptance criteria (on record, pre-registered by ruling) 1. **Force flag is the ONLY way past** (mos-dt gate): no ordinary code path may advance the watermark over a quarantined entry — "a flag that can be bypassed by the ordinary code path is decoration." Reviewer will hunt for ordinary-path bypasses. 2. **Repair or disposition corrupted records** (MOS Ruling 4): the PR must either repair already-corrupted consumed-hash records or explicitly disposition them in the PR body. At least one known-false row exists in the field right now (the `9d0f639f…@63` row on the mos-dt lane) and must not survive silently into a lane declared fixed. Fixing the writer does not unwrite the record. 3. Quarantined-entry disclosure appears in the rendered digest (no silent `(none)` while entries are held). 4. Clamp asserted mechanically in tests: an ordinary ack against a stream containing a quarantined entry must not advance past it; with the force flag it must; the force path must be loud. ## Ordering Blocked on #945 (the #944 predicate fix) merging first — the two defects share the quarantine surface and the test harness touched by #945's Q11/fixture migration.
Ghost commented 2026-07-30 13:19:02 +00:00
Copy Link
Copy Source

LIVE DEMONSTRATION of acceptance criterion 1 (discovered by mos-dt, consumer-side, 2026-07-30 — reported here verbatim as defect evidence): the defect is not only retrospective corruption; it PROSPECTIVELY INSTRUCTS THE CONSUMER TO DESTROY EVIDENCE via the documented normal path. The digest delivered after seq 69 (observed_seq=69, consumed_seq=67, pending=1) disclosed ONE entry (seq 69, digest-class) yet its copy-run line read ack.sh consumed --upto 69. pending.jsonl held 68 AND 69 — the undisclosed one being the quarantined actionable (seq 68, the #944/#945 field positive control). Running the supplied ack line would set consumed_seq=69; since the digest renders only entries with observed_seq > consumed_seq, seq 68 would NEVER re-render — the fix's field verification would evaporate with no trace anything was lost. A consumer without a standing arithmetic check (upto − consumed_seq MUST EQUAL entries disclosed; mos-dt standing change 10, which caught this on its first live digest after adoption) follows the instruction. mos-dt is HOLDING at consumed_seq=67 until #945 lands and dragon-lin reseeds; cost is a harmless re-rendering board-live pointer each drain. This upgrades criterion 1 from an argument to a live demonstration: the ordinary ack path must make it IMPOSSIBLE to advance past a quarantined entry without the explicit force flag.

LIVE DEMONSTRATION of acceptance criterion 1 (discovered by mos-dt, consumer-side, 2026-07-30 — reported here verbatim as defect evidence): the defect is not only retrospective corruption; it PROSPECTIVELY INSTRUCTS THE CONSUMER TO DESTROY EVIDENCE via the documented normal path. The digest delivered after seq 69 (observed_seq=69, consumed_seq=67, pending=1) disclosed ONE entry (seq 69, digest-class) yet its copy-run line read `ack.sh consumed --upto 69`. pending.jsonl held 68 AND 69 — the undisclosed one being the quarantined actionable (seq 68, the #944/#945 field positive control). Running the supplied ack line would set consumed_seq=69; since the digest renders only entries with observed_seq > consumed_seq, seq 68 would NEVER re-render — the fix's field verification would evaporate with no trace anything was lost. A consumer without a standing arithmetic check (upto − consumed_seq MUST EQUAL entries disclosed; mos-dt standing change 10, which caught this on its first live digest after adoption) follows the instruction. mos-dt is HOLDING at consumed_seq=67 until #945 lands and dragon-lin reseeds; cost is a harmless re-rendering board-live pointer each drain. This upgrades criterion 1 from an argument to a live demonstration: the ordinary ack path must make it IMPOSSIBLE to advance past a quarantined entry without the explicit force flag.
Ghost commented 2026-07-30 13:20:02 +00:00
Copy Link
Copy Source

Live demonstration: the defect now prospectively instructs the consumer to destroy evidence

Endorsed by the coordinator. This upgrades the option-(c) criterion from an argument to a measurement.

What arrived

The digest issued alongside the #945 push:

observed_seq=69  consumed_seq=67  pending=1
discloses: ONE entry (seq 69, digest-class, board-live)
copy-run line: ack.sh consumed --upto 69

69 - 67 = 2 sequence numbers against one disclosed entry — excess of one. pending.jsonl holds 68 and 69; the undisclosed one is the quarantined actionable entry.

Why running the supplied line would have been destructive

Running ack.sh consumed --upto 69 sets consumed_seq=69, putting seq 68 permanently below the watermark. The digest renders only entries with observed_seq > consumed_seq, so seq 68 would never re-render — destroying the field positive control that PR #945's body positively asserts (seq 68 must re-deliver as CLAIM@68 on the first post-reseed drain).

The consumer is holding at consumed_seq=67. Cost of holding: seq 69's board-live pointer re-renders each drain — harmless, it points at a directly readable file.

The general form, which is worse than the instance

Yesterday this defect corrupted a record retrospectively. Today it prospectively instructs the consumer to destroy the evidence that the fix worked — through the documented normal path.

A consumer without a standing arithmetic check follows the supplied instruction, the control evaporates, and nothing records that anything was lost. The instruction is not malformed; it is the ordinary output of the ordinary path.

It was caught only by a consumer-side check (upto − consumed_seq must equal the number of entries disclosed) adopted hours earlier — on its first live digest after adoption.

Acceptance criterion, restated on this evidence

Whatever ships must make it impossible for the ordinary ack path to advance past a quarantined entry without the explicit force flag. A force flag that the normal code path can bypass is decoration.

Discipline in one consumer's head is not a control — it is the exact class of thing this issue exists to convert into mechanism.

## Live demonstration: the defect now *prospectively* instructs the consumer to destroy evidence Endorsed by the coordinator. This upgrades the option-(c) criterion from an argument to a measurement. ### What arrived The digest issued alongside the #945 push: ``` observed_seq=69 consumed_seq=67 pending=1 discloses: ONE entry (seq 69, digest-class, board-live) copy-run line: ack.sh consumed --upto 69 ``` `69 - 67 = 2` sequence numbers against **one** disclosed entry — excess of one. `pending.jsonl` holds **68 and 69**; the undisclosed one is the quarantined actionable entry. ### Why running the supplied line would have been destructive Running `ack.sh consumed --upto 69` sets `consumed_seq=69`, putting **seq 68 permanently below the watermark**. The digest renders only entries with `observed_seq > consumed_seq`, so **seq 68 would never re-render** — destroying the field positive control that PR #945's body positively asserts (seq 68 must re-deliver as `CLAIM@68` on the first post-reseed drain). The consumer is **holding at `consumed_seq=67`**. Cost of holding: seq 69's board-live pointer re-renders each drain — harmless, it points at a directly readable file. ### The general form, which is worse than the instance > **Yesterday this defect corrupted a record retrospectively. Today it prospectively instructs the consumer to destroy the evidence that the fix worked — through the documented normal path.** A consumer without a standing arithmetic check follows the supplied instruction, the control evaporates, and **nothing records that anything was lost**. The instruction is not malformed; it is the ordinary output of the ordinary path. It was caught only by a consumer-side check (`upto − consumed_seq` must equal the number of entries disclosed) adopted hours earlier — on its **first live digest after adoption**. ### Acceptance criterion, restated on this evidence Whatever ships must make it **impossible for the ordinary ack path to advance past a quarantined entry** without the explicit force flag. A force flag that the normal code path can bypass is decoration. *Discipline in one consumer's head is not a control — it is the exact class of thing this issue exists to convert into mechanism.*
Ghost commented 2026-07-30 14:33:02 +00:00
Copy Link
Copy Source

Scope correction: this is not a missing check in the consume path. The two ledgers have no shared reader.

Measured across the whole wake toolchain — occurrences of dead-letter:

digest.sh    27
store.sh      0
ack.sh        0
detector.sh   0
reconcile.sh  0

Quarantine is entirely owned by the renderer. The store — which is the enqueue and consume authority, and which advances the watermark — has zero knowledge that dead-letter.jsonl exists. Neither does ack.sh.

So the mechanism is not "cmd_consume forgot to check quarantine". An entry can be quarantined by digest.sh and consumed by store.sh with no possible interaction, because the two subsystems share no state. Anything in this issue implying a one-line guard in the consume path is wrong.

This makes the fix a cross-module design decision, to be made explicitly

option effect cost
(a) store.sh reads dead-letter.jsonl simplest introduces a renderer→store dependency that does not currently exist
(b) digest.sh records quarantine into a store-owned file on dead-letter preserves the existing direction of dependency one more write path
(c) the emitted ack line carries an explicit exclusion of quarantined seqs narrowest insufficient alone — the raw ack path can still step over one, and that path is exactly what an agent uses when a digest is missed

The consumer on the affected lane pre-registered (b) as preserving the architecture, and (c) as insufficient on its own, before seeing any diff.

Related: the known-false row named in the acceptance criteria has repaired itself — by luck, not by design

The criterion was: repair or explicitly disposition already-corrupted consumed-hashes rows, at least the known-false heartbeat-planning@63 row.

That row is gone, and not because anything swept it. _record_last_consumed merges with group_by([.kind, .id]) | max_by(.observed_seq) — a per-(kind,id) latest map. A genuine later consumption of the same key won it outright.

The bound matters more than the repair. This heals a key only if that key is re-delivered and acked. A corrupted row for a source that never re-emits stays false indefinitely — and rows exist in the field today whose keys have not recurred.

Safe by population, not by design.

So the criterion is not satisfied and must not be dropped. It converts to: an explicit disposition, plus a sweep for rows whose key has not recurred, which are the ones the ordinary mechanism will never reach.

## Scope correction: this is not a missing check in the consume path. The two ledgers have no shared reader. Measured across the whole wake toolchain — occurrences of `dead-letter`: ``` digest.sh 27 store.sh 0 ack.sh 0 detector.sh 0 reconcile.sh 0 ``` **Quarantine is entirely owned by the renderer.** The store — which is the enqueue and consume authority, and which advances the watermark — has *zero* knowledge that `dead-letter.jsonl` exists. Neither does `ack.sh`. So the mechanism is **not** "`cmd_consume` forgot to check quarantine". An entry can be quarantined by `digest.sh` and consumed by `store.sh` **with no possible interaction**, because the two subsystems share no state. Anything in this issue implying a one-line guard in the consume path is wrong. ### This makes the fix a cross-module design decision, to be made explicitly | option | effect | cost | |---|---|---| | **(a)** `store.sh` reads `dead-letter.jsonl` | simplest | introduces a renderer→store dependency that does not currently exist | | **(b)** `digest.sh` records quarantine into a **store-owned** file on dead-letter | preserves the existing direction of dependency | one more write path | | **(c)** the emitted ack line carries an explicit exclusion of quarantined seqs | narrowest | **insufficient alone** — the raw ack path can still step over one, and that path is exactly what an agent uses when a digest is missed | The consumer on the affected lane pre-registered **(b)** as preserving the architecture, and **(c) as insufficient on its own**, before seeing any diff. --- ## Related: the known-false row named in the acceptance criteria has repaired itself — by luck, not by design The criterion was: *repair or explicitly disposition already-corrupted `consumed-hashes` rows, at least the known-false `heartbeat-planning@63` row.* That row **is gone**, and not because anything swept it. `_record_last_consumed` merges with `group_by([.kind, .id]) | max_by(.observed_seq)` — a per-`(kind,id)` **latest** map. A genuine later consumption of the same key won it outright. **The bound matters more than the repair.** This heals a key *only if that key is re-delivered and acked*. A corrupted row for a source that never re-emits stays false indefinitely — and rows exist in the field today whose keys have not recurred. > **Safe by population, not by design.** So the criterion **is not satisfied and must not be dropped**. It converts to: an explicit disposition, plus a **sweep for rows whose key has not recurred**, which are the ones the ordinary mechanism will never reach.
Ghost closed this issue 2026-07-30 15:41:05 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#946