mosaicstack/stack
25
0
Fork 1
Code Issues 340 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

wake-component: reconcile has no settling window (systematic false positives) + class escalation promotes routine sources to actionable on a lost race #1048

New Issue
Open
opened 2026-08-05 02:18:25 +00:00 by Ghost · 0 comments
Ghost commented 2026-08-05 02:18:25 +00:00
Copy Link
Copy Source

A. reconcile has no settling window — it is a systematic false-positive generator

The wake-component reconciler flags a source UNACCOUNTED and enumerates it without granting the detector one full poll interval to observe the change first.

Measured instance:

WAKE_DETECTOR_INTERVAL = 600s
19:33:20  commit touches the watched path
19:35:00  reconcile backstop runs -> UNACCOUNTED, enumerates observed_seq=128
          delta = 100s  (17% of one poll window)

The detector had not failed — it had not yet been given a turn.

Consequence: any source whose change lands within one WAKE_DETECTOR_INTERVAL of a reconcile pass is flagged regardless of detector health. With a 6h reconcile cadence and a 600s detector interval that is a ~2.8% race window per source per cycle — it will keep firing this same alarm forever with no underlying fault. This is a reconciler defect, not a detector defect.

Proposed fix: reconcile SKIPS (or soft-flags, non-alarming) any source whose observed change is younger than WAKE_DETECTOR_INTERVAL + margin.

Why it matters beyond the noise: a backstop that cries wolf on a schedule trains its consumers to discount it — and the whole value of a backstop is that its alarms are believed. It also defeats the diagnostic rule that a backstop firing implies primary failure: that inference is only valid once a settling window exists.

B. Class escalation on enumeration — losing a race promotes a routine source into the actionable lane

The affected source is declared class=digest, slo=routine, but the reconciler enumerated it as class=actionable. Per reconcile.sh's own header this is deliberate (the non-coalescible fail-safe class).

Consequence: losing a race with the detector promotes a routine digest source into the actionable lane. Combined with (A), routine sources will periodically and spuriously manufacture actionable wakes — the most expensive class, generated by the cheapest cause.

This is a design question, not a local misconfiguration: the fail-safe class is defensible for a genuinely-unaccounted source, but (A) means most enumerations are not genuinely unaccounted. If (A) is fixed, (B) may become acceptable as-is; the two should be decided together.

C. Unverified, flagged not asserted — possible semantic duplication

When the detector's next poll observes the same change, does it produce a second store entry alongside the reconciler's enumerated one? Prior work resolved sequence collision via a single store-side allocator; it is not clear it addresses semantic duplication of the same source state enumerated by two feeders. This was not tested and is not being asserted — but if it duplicates, (A) makes it recur on a schedule.

Provenance / method

Reported by the operating agent on the affected host, which also disclosed two method corrections before recording any figure: an initial inventory run was unrepresentative because the watch-list env var was unset in an interactive shell (the documented gotcha), and an exit code was nearly misreported because $? captured tail through a pipe. Both re-run under the service environment first. The re-check used a side-effect-free check mode rather than a second enumerating pass, so the confirming observation could not be manufactured by the act of observing.

Verified alongside: the source is present in the declared parity inventory (so this is not the vacuous-pass case), the change was genuine (content-derived blob SHA, real commit — not an ambiguous/error read), and the source had been ACCOUNTED on every prior 6-hourly run — first occurrence, not a recurring primary failure.

## A. `reconcile` has no settling window — it is a systematic false-positive generator The wake-component reconciler flags a source `UNACCOUNTED` and enumerates it **without granting the detector one full poll interval to observe the change first**. **Measured instance:** ``` WAKE_DETECTOR_INTERVAL = 600s 19:33:20 commit touches the watched path 19:35:00 reconcile backstop runs -> UNACCOUNTED, enumerates observed_seq=128 delta = 100s (17% of one poll window) ``` The detector had not failed — **it had not yet been given a turn.** **Consequence:** any source whose change lands within one `WAKE_DETECTOR_INTERVAL` of a reconcile pass is flagged **regardless of detector health**. With a 6h reconcile cadence and a 600s detector interval that is a **~2.8% race window per source per cycle** — it will keep firing this same alarm forever with no underlying fault. This is a **reconciler** defect, not a detector defect. **Proposed fix:** reconcile SKIPS (or soft-flags, non-alarming) any source whose observed change is younger than `WAKE_DETECTOR_INTERVAL` + margin. **Why it matters beyond the noise:** a backstop that cries wolf on a schedule trains its consumers to discount it — and the whole value of a backstop is that its alarms are believed. It also defeats the diagnostic rule that a backstop firing implies primary failure: that inference is only valid once a settling window exists. ## B. Class escalation on enumeration — losing a race promotes a routine source into the actionable lane The affected source is declared `class=digest, slo=routine`, but the reconciler enumerated it as **`class=actionable`**. Per `reconcile.sh`'s own header this is deliberate (the non-coalescible fail-safe class). **Consequence:** **losing a race with the detector promotes a routine digest source into the actionable lane.** Combined with (A), routine sources will periodically and spuriously manufacture **actionable** wakes — the most expensive class, generated by the cheapest cause. This is a design question, not a local misconfiguration: the fail-safe class is defensible for a genuinely-unaccounted source, but (A) means most enumerations are not genuinely unaccounted. If (A) is fixed, (B) may become acceptable as-is; the two should be decided together. ## C. Unverified, flagged not asserted — possible semantic duplication When the detector's next poll observes the same change, does it produce a **second store entry** alongside the reconciler's enumerated one? Prior work resolved sequence **collision** via a single store-side allocator; it is not clear it addresses semantic **duplication** of the same source state enumerated by two feeders. **This was not tested and is not being asserted** — but if it duplicates, (A) makes it recur on a schedule. ## Provenance / method Reported by the operating agent on the affected host, which also disclosed two method corrections before recording any figure: an initial inventory run was unrepresentative because the watch-list env var was unset in an interactive shell (the documented gotcha), and an exit code was nearly misreported because `$?` captured `tail` through a pipe. Both re-run under the service environment first. The re-check used a **side-effect-free `check` mode** rather than a second enumerating pass, so the confirming observation could not be manufactured by the act of observing. Verified alongside: the source **is** present in the declared parity inventory (so this is not the vacuous-pass case), the change **was** genuine (content-derived blob SHA, real commit — not an ambiguous/error read), and the source had been `ACCOUNTED` on every prior 6-hourly run — **first occurrence, not a recurring primary failure.**
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1048