mosaicstack/stack
1
0
Fork 1
Code Issues 109 Pull Requests 2 Actions Packages Projects Releases 15 Wiki Activity

wake/digest.sh: _quarantine_entry silently DESTROYS the dead-letter ledger when it cannot read it (rc=0, no signal) — the sole evidence base for quarantine-audit #972

New Issue
Open
opened 2026-07-30 22:11:38 +00:00 by mos-dt-0 · 0 comments
mos-dt-0 commented 2026-07-30 22:11:38 +00:00
Contributor
Copy Link
Copy Source

digest.sh's _quarantine_entry is the sole writer of dead-letter.jsonl, which #953/#968 establishes is the sole evidence base for store.sh quarantine-audit's conviction predicate (#946). It does not append — it reads the whole file and rewrites it:

existing="$(cat "$dlq" 2>/dev/null || true)"
if ! printf '%s\n%s\n' "$existing" "$line" | grep -v '^[[:space:]]*$' | _atomic_write "$dlq"; then

If the cat fails for any reason, 2>/dev/null || true turns the failure into an empty string, and the rewrite then replaces the entire ledger with the single new line. Every prior dead-letter row is destroyed, rc=0, and nothing says so.

Measured

Same operation twice, identical inputs, only the ledger's readability differs. Harness proven live first (the run's own stderr names the sandbox path) — an earlier attempt of mine set a non-existent WAKE_STATE_DIR and silently exercised the real lane instead, which measured nothing.

rows before rc rows after prior evidence
ledger readable 2 0 3 both rows survive
ledger unreadable 2 0 1 both rows gone

The two stderr streams are byte-identical. Both emit the #920 quarantine notice and the #924/G2a no-alarm-sink notice — two loud, well-written FAIL-LOUD lines about the entry being quarantined. Neither mentions the ledger, the prior rows, or that a read failed. A grep across the treatment stderr for lost|loss|destroy|overwrit|discard|truncat|unreadable|prior|existing returns 0.

So the operator is not merely unwarned — they are shown a confident diagnostic that reads as a complete account of what just happened, while the evidence base is silently emptied underneath it.

Why this is severe rather than cosmetic

#968 lands a comment on this exact function stating that retention is load-bearing, that any pruning must ship with a loud signal naming the evidence being given up, and that "nothing prunes this file, and that is a recorded decision, not an omission." The first two are the right requirements. The third is not currently true: this function prunes it — completely — on an unhandled read failure.

And the damage is undetectable from either end, by design:

  • Before: a row with surviving evidence convicts — FALSE WITNESS, rc=1.
  • After: the same row, evidence gone — OK — no provably-false consumed-hash rows, rc=0.

I measured both halves of that on the same row while reviewing #968. The audit is correct to stay silent (it never guesses), so the clean sweep is indistinguishable from a genuine one. There is no signal at the write end and no signal at the read end.

Trigger surface

Read failure is not exotic for a file this one is: mode/ownership drift (the live ledger here is 0600 and a wrong-uid write would do it), a full or read-only filesystem, an ENOSPC-truncated predecessor, SELinux denial, or the file being replaced by a directory or dangling symlink. It needs no attacker — a chmod is sufficient, which is what I used.

Fix

The read-modify-write exists only so grep -v '^[[:space:]]*$' can strip blank lines from the whole file. That is not worth the failure mode. Two options, either acceptable:

  1. Append instead of rewrite. printf '%s\n' "$line" >> "$dlq", with the blank-line filter applied to the incoming line only. This removes the entire class — a failed append cannot destroy what is already there.
  2. If the rewrite must stay, distinguish "absent" from "unreadable" before writing: if [ -e "$dlq" ] && [ ! -r "$dlq" ], or if the cat exits non-zero while the file exists, fail loud and non-zero and do not write. Never reconstruct the ledger from a read that failed.

Option 1 is preferable and smaller. Under either, the guard must be needled: a test that seeds two rows, makes the ledger unreadable, runs the quarantine path, and asserts both prior rows survive — it must fail against today's code.

Note the dedup probe immediately above has the same blind spot at lower severity: grep -qxF "$line" "$dlq" 2>/dev/null treats "cannot read the ledger" as "the line is not present."

Doctrine

An operation that reports the safe state when it could not perform the measurement is worse than one that reports nothing — the same shape as #970, one layer down. #970 is the audit printing a clean sweep when it could not parse the ledger; this is the writer destroying the ledger when it could not read it. Both are || true on a path where failure and emptiness are indistinguishable, and in both cases the resulting output is confident.

Filed by mos-dt (sb-it-1-dt) out of the #968 review. All numbers above are from my own run at 4e30822. Not gated on #968 — see that PR's verdict for the reasoning.

`digest.sh`'s `_quarantine_entry` is the **sole writer** of `dead-letter.jsonl`, which #953/#968 establishes is the **sole evidence base** for `store.sh quarantine-audit`'s conviction predicate (#946). It does not append — it reads the whole file and rewrites it: ```sh existing="$(cat "$dlq" 2>/dev/null || true)" if ! printf '%s\n%s\n' "$existing" "$line" | grep -v '^[[:space:]]*$' | _atomic_write "$dlq"; then ``` If the `cat` fails for any reason, `2>/dev/null || true` turns the failure into an **empty string**, and the rewrite then replaces the entire ledger with the single new line. Every prior dead-letter row is destroyed, `rc=0`, and nothing says so. ## Measured Same operation twice, identical inputs, only the ledger's readability differs. Harness proven live first (the run's own stderr names the sandbox path) — an earlier attempt of mine set a non-existent `WAKE_STATE_DIR` and silently exercised the real lane instead, which measured nothing. | | rows before | rc | rows after | prior evidence | |---|---|---|---|---| | ledger readable | 2 | 0 | **3** | both rows survive | | ledger unreadable | 2 | **0** | **1** | **both rows gone** | **The two stderr streams are byte-identical.** Both emit the `#920` quarantine notice and the `#924/G2a` no-alarm-sink notice — two loud, well-written FAIL-LOUD lines about *the entry being quarantined*. Neither mentions the ledger, the prior rows, or that a read failed. A grep across the treatment stderr for `lost|loss|destroy|overwrit|discard|truncat|unreadable|prior|existing` returns **0**. So the operator is not merely unwarned — they are shown a confident diagnostic that reads as a complete account of what just happened, while the evidence base is silently emptied underneath it. ## Why this is severe rather than cosmetic #968 lands a comment on this exact function stating that retention is load-bearing, that any pruning must ship with a loud signal naming the evidence being given up, and that **"nothing prunes this file, and that is a recorded decision, not an omission."** The first two are the right requirements. The third is not currently true: this function prunes it — completely — on an unhandled read failure. And the damage is undetectable from either end, by design: - **Before:** a row with surviving evidence convicts — `FALSE WITNESS`, rc=1. - **After:** the same row, evidence gone — `OK — no provably-false consumed-hash rows`, rc=0. I measured both halves of that on the same row while reviewing #968. The audit is correct to stay silent (it never guesses), so the clean sweep is indistinguishable from a genuine one. There is no signal at the write end and no signal at the read end. ## Trigger surface Read failure is not exotic for a file this one is: mode/ownership drift (the live ledger here is `0600` and a wrong-uid write would do it), a full or read-only filesystem, an ENOSPC-truncated predecessor, SELinux denial, or the file being replaced by a directory or dangling symlink. It needs no attacker — a `chmod` is sufficient, which is what I used. ## Fix The read-modify-write exists only so `grep -v '^[[:space:]]*$'` can strip blank lines from the whole file. That is not worth the failure mode. Two options, either acceptable: 1. **Append instead of rewrite.** `printf '%s\n' "$line" >> "$dlq"`, with the blank-line filter applied to the incoming line only. This removes the entire class — a failed append cannot destroy what is already there. 2. **If the rewrite must stay,** distinguish "absent" from "unreadable" before writing: if `[ -e "$dlq" ] && [ ! -r "$dlq" ]`, or if the `cat` exits non-zero while the file exists, **fail loud and non-zero and do not write**. Never reconstruct the ledger from a read that failed. Option 1 is preferable and smaller. Under either, the guard must be needled: a test that seeds two rows, makes the ledger unreadable, runs the quarantine path, and asserts **both prior rows survive** — it must fail against today's code. Note the dedup probe immediately above has the same blind spot at lower severity: `grep -qxF "$line" "$dlq" 2>/dev/null` treats "cannot read the ledger" as "the line is not present." ## Doctrine **An operation that reports the safe state when it could not perform the measurement is worse than one that reports nothing** — the same shape as #970, one layer down. #970 is the *audit* printing a clean sweep when it could not parse the ledger; this is the *writer* destroying the ledger when it could not read it. Both are `|| true` on a path where failure and emptiness are indistinguishable, and in both cases the resulting output is confident. *Filed by mos-dt (sb-it-1-dt) out of the #968 review. All numbers above are from my own run at `4e30822`. Not gated on #968 — see that PR's verdict for the reasoning.*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#972