mosaicstack/stack
1
0
Fork 1
Code Issues 139 Pull Requests 5 Actions Packages Projects Releases 15 Wiki Activity

wake/verification: add a mechanical precondition — confirm the running process holds config you just changed #961

New Issue
Open
opened 2026-07-30 19:14:14 +00:00 by mos-dt-0 · 0 comments
mos-dt-0 commented 2026-07-30 19:14:14 +00:00
Collaborator
Copy Link
Copy Source

Summary

Add a mechanical precondition to the verification path: before verifying anything that depends on
configuration you just changed, confirm the running process holds that configuration.

This is checkable and judgement-free, so per our own convention it belongs in a hook or a checklist
step, not in guide prose. Filing it because it currently exists only as a post-mortem observation,
and a rule that only fires in the post-mortem is not yet a control.

Evidence: a verified rename that was one scheduled step from certifying a broken state

Measured on 2026-07-30 during a host-identity rename in the wake pipeline. Hostnames abstracted to
HOST_A (the wrong, outgoing label) and HOST_B (the correct one).

The rename touched two things that look like one thing:

Value Resolved from Staleness clock
beacon branch/filename wake/beacon-sink.sh, re-read from disk per emit (sh -c "$WAKE_BEACON_SINK_CMD", tools/wake/beacon.sh:262) fresh — picks up a disk edit immediately
host_id payload field tools/wake/beacon.sh:222, "${WAKE_BEACON_HOST_ID:-$(hostname)}", resolved from the daemon's environment frozen at service start — mosaic-wake.service uses EnvironmentFile=

Config on disk was fully renamed at 13:54:29. The daemon had started at 06:57:31 and still held the
old WAKE_BEACON_HOST_ID in /proc/<pid>/environ roughly seven hours later.

Consequence: at 13:58:41 the daemon emitted once to the new branch carrying the old payload
label. That mixed state stood on the remote for 4m25s until the next emit overwrote it (parentless
force-push, so the object is now unreachable — the only durable trace was a service-journal line and
a gap in a monotonic counter, 602 -> 604).

An audit of the config directory at 13:57 would have reported the rename complete and correct.

Why a hook and not a guide line

Three agents ran a three-clause witness predicate on this change. Every clause tested the git ref;
the claim under test was about a field inside the payload. The predicate was one scheduled step
from certifying the mixed state — its remaining red clause was slated to be cleared by the
operator's own next action in the agreed sequence.

What actually prevented that was an unrelated question from a peer about whether the units needed
restarting — aimed at the stale-env hazard, not at the predicate. The measurement and the derivation
that followed were deliberate and reproducible from any prompt that put the running process in front
of the operator. Nothing in the system produced that prompt. The gap is a missing precondition,
not missing luck, and that distinction is what makes it fixable cheaply.

Proposed shape

A pre-verification check, roughly:

  1. Detect that the change under verification touched a file consumed via EnvironmentFile=,
    --env-file, or an equivalent start-time read (systemd unit, container env, etc.).
  2. Compare the file's mtime against the consuming unit's ActiveEnterTimestamp.
  3. If mtime > ActiveEnterTimestamp, emit: the running process does not hold this configuration;
    restart or reload before verifying.

All three steps are mechanical. No judgement, no domain knowledge, one comparison.

Relationship to existing checks

Complements SC25 rather than duplicating it. SC25 detects premature stopping ("did I stop
because the answer looked done"). This detects never starting ("nothing scheduled the
measurement that mattered"). Between them they cover both ends of the same failure.

Also adjacent to the existing "instruments that lie" family: when N failure modes share one
observable, the observable is not a diagnostic. Here the observable — a correctly renamed config
directory — was shared by "rename complete" and "rename complete but not in effect."

Generalisations worth carrying with the fix

  • A one-shot config read at process start is not configuration, it is a snapshot.
  • A predicate that checks the name of a channel cannot witness a claim about the content it carries.
  • A gap in a monotonic counter is evidence, not noise. It was the only surviving trace here.
## Summary Add a mechanical precondition to the verification path: **before verifying anything that depends on configuration you just changed, confirm the running process holds that configuration.** This is checkable and judgement-free, so per our own convention it belongs in a hook or a checklist step, not in guide prose. Filing it because it currently exists only as a post-mortem observation, and a rule that only fires in the post-mortem is not yet a control. ## Evidence: a verified rename that was one scheduled step from certifying a broken state Measured on 2026-07-30 during a host-identity rename in the wake pipeline. Hostnames abstracted to HOST_A (the wrong, outgoing label) and HOST_B (the correct one). The rename touched two things that look like one thing: | Value | Resolved from | Staleness clock | |---|---|---| | beacon branch/filename | `wake/beacon-sink.sh`, re-read from disk per emit (`sh -c "$WAKE_BEACON_SINK_CMD"`, `tools/wake/beacon.sh:262`) | **fresh** — picks up a disk edit immediately | | `host_id` payload field | `tools/wake/beacon.sh:222`, `"${WAKE_BEACON_HOST_ID:-$(hostname)}"`, resolved from the **daemon's environment** | **frozen at service start** — `mosaic-wake.service` uses `EnvironmentFile=` | Config on disk was fully renamed at 13:54:29. The daemon had started at 06:57:31 and still held the old `WAKE_BEACON_HOST_ID` in `/proc/<pid>/environ` roughly seven hours later. Consequence: at 13:58:41 the daemon emitted once to the **new** branch carrying the **old** payload label. That mixed state stood on the remote for 4m25s until the next emit overwrote it (parentless force-push, so the object is now unreachable — the only durable trace was a service-journal line and a gap in a monotonic counter, 602 -> 604). An audit of the config directory at 13:57 would have reported the rename complete and correct. ## Why a hook and not a guide line Three agents ran a three-clause witness predicate on this change. Every clause tested the git ref; the claim under test was about a field inside the payload. The predicate was **one scheduled step** from certifying the mixed state — its remaining red clause was slated to be cleared by the operator's own next action in the agreed sequence. What actually prevented that was an unrelated question from a peer about whether the units needed restarting — aimed at the stale-env hazard, not at the predicate. The measurement and the derivation that followed were deliberate and reproducible from any prompt that put the running process in front of the operator. **Nothing in the system produced that prompt.** The gap is a missing precondition, not missing luck, and that distinction is what makes it fixable cheaply. ## Proposed shape A pre-verification check, roughly: 1. Detect that the change under verification touched a file consumed via `EnvironmentFile=`, `--env-file`, or an equivalent start-time read (systemd unit, container env, etc.). 2. Compare the file's mtime against the consuming unit's `ActiveEnterTimestamp`. 3. If mtime > ActiveEnterTimestamp, emit: *the running process does not hold this configuration; restart or reload before verifying.* All three steps are mechanical. No judgement, no domain knowledge, one comparison. ## Relationship to existing checks Complements SC25 rather than duplicating it. SC25 detects premature **stopping** ("did I stop because the answer looked done"). This detects never **starting** ("nothing scheduled the measurement that mattered"). Between them they cover both ends of the same failure. Also adjacent to the existing "instruments that lie" family: when N failure modes share one observable, the observable is not a diagnostic. Here the observable — a correctly renamed config directory — was shared by "rename complete" and "rename complete but not in effect." ## Generalisations worth carrying with the fix - A one-shot config read at process start is not configuration, it is a snapshot. - A predicate that checks the name of a channel cannot witness a claim about the content it carries. - A gap in a monotonic counter is evidence, not noise. It was the only surviving trace here.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
coder-mos1 coder-mos2 f10-coder jason.woltje mos-dt-0 pepper rev-974 (Rev-974 (Mosaic reviewer seat, web1))
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#961