mosaicstack/stack
16
0
Fork 1
Code Issues 330 Pull Requests 12 Actions Packages Projects Releases 15 Wiki Activity

Keep-mode upgrade verify-net reverts live operator state: 142 files restored over running fleet writes (TOCTOU; follow-on to #791) #1205

New Issue
Open
opened 2026-08-13 15:46:38 +00:00 by Ghost · 0 comments
Ghost commented 2026-08-13 15:46:38 +00:00
Copy Link
Copy Source

Summary

install.sh's keep-mode verify-net "restored" 142 operator files from the pre-update snapshot
during a routine framework refresh — including live fleet/run/*.hb.native heartbeat files that
running fleet seats had legitimately rewritten while the install was in progress.

The net reported them as "unexpectedly changed … restored from the pre-update snapshot" and reverted
live runtime state to stale snapshot content.

Follow-on to the closed #791. #791 made upgrades stop destroying operator config; this is the
inverse failure — the safety net now reverts operator state that a running process is actively
and correctly writing.

Why it happens

The net snapshots [operator] paths before the update and compares after. It cannot distinguish:

  • a file the installer wrongly touched (what the net exists to catch), from
  • a file a third party — a live daemon, a running seat — wrote during the window (a TOCTOU race).

framework-manifest.txt lists fleet/run/** under [operator], and fleet/run/ is precisely the
directory that live seats write to continuously. Any upgrade on a host with an active fleet will
race it.

Impact here

Contained, but only by luck of the workload: heartbeats are rewritten on a short interval, so all
were 0m fresh within a minute and the 40 fleet-socket / 88 default-socket sessions were unharmed.

A path under [operator] with a slower write cadence — a roster, a backlog file, a state
document written once per task — would have been silently rolled back to a stale version with no
error, and the loss would surface much later as an inexplicable state regression.

Proposed acceptance criteria

  • Distinguish volatile runtime state from durable operator config in the manifest. Paths like
    fleet/run/** are runtime, not config, and must be excluded from the verify-net (they
    may still be preserved, but never restored over).
  • Where the net does act, decide by who wrote it, not that it changed: compare against the
    installer's own intended writes rather than against any post-hoc difference.
  • Never restore a file whose mtime is after the installer finished touching that path.
  • Report restorations at a visible level with the full list, not as a count buried in success
    output — a rc=0 install that reverted 142 files should not read as an uneventful success.
  • Regression test: mutate an [operator] path from a concurrent process mid-install; assert
    the mutation survives.
## Summary `install.sh`'s keep-mode verify-net "restored" **142 operator files** from the pre-update snapshot during a routine framework refresh — including live `fleet/run/*.hb.native` heartbeat files that **running fleet seats had legitimately rewritten while the install was in progress**. The net reported them as "unexpectedly changed … restored from the pre-update snapshot" and reverted live runtime state to stale snapshot content. Follow-on to the closed #791. #791 made upgrades stop *destroying* operator config; this is the inverse failure — the safety net now *reverts* operator state that a running process is actively and correctly writing. ## Why it happens The net snapshots `[operator]` paths before the update and compares after. It cannot distinguish: - a file the installer wrongly touched (what the net exists to catch), from - a file a **third party** — a live daemon, a running seat — wrote during the window (a TOCTOU race). `framework-manifest.txt` lists `fleet/run/**` under `[operator]`, and `fleet/run/` is precisely the directory that live seats write to continuously. Any upgrade on a host with an active fleet will race it. ## Impact here Contained, but only by luck of the workload: heartbeats are rewritten on a short interval, so all were 0m fresh within a minute and the 40 fleet-socket / 88 default-socket sessions were unharmed. A path under `[operator]` with a **slower** write cadence — a roster, a backlog file, a state document written once per task — would have been silently rolled back to a stale version with no error, and the loss would surface much later as an inexplicable state regression. ## Proposed acceptance criteria - [ ] Distinguish volatile runtime state from durable operator config in the manifest. Paths like `fleet/run/**` are runtime, not config, and must be **excluded from the verify-net** (they may still be preserved, but never restored over). - [ ] Where the net does act, decide by *who wrote it*, not *that it changed*: compare against the installer's own intended writes rather than against any post-hoc difference. - [ ] Never restore a file whose mtime is **after** the installer finished touching that path. - [ ] Report restorations at a visible level with the full list, not as a count buried in success output — a `rc=0` install that reverted 142 files should not read as an uneventful success. - [ ] Regression test: mutate an `[operator]` path from a concurrent process mid-install; assert the mutation survives.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) pepper rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1205