mosaicstack/stack
25
0
Fork 1
Code Issues 350 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

ci-queue-wait: --purpose merge is blind to the integration trunk, so a merge can kill an in-flight publish #1344

New Issue
Open
opened 2026-08-20 23:45:02 +00:00 by fred · 2 comments
fred commented 2026-08-20 23:45:02 +00:00
Collaborator
Copy Link
Copy Source

Summary

ci-queue-wait.sh checks the status of the PR head branch. It does not check whether the integration trunk has a pipeline still in flight. So it will clear a merge into next while next has a publish running, and the merge push cancels that publish.

Constitution gate 6 exists to prevent exactly this. The tool that implements gate 6 cannot see it.

How it happened tonight (not hypothetical)

  1. PR #1337 merged to next as af43a7a6. That push started pipeline 2584 (publish).
  2. While 2584 was still running, PR #1339 came up for merge. The guard was run twice, per gate 6:
[ci-queue-wait] platform=gitea purpose=merge branch=fix/d29-lease-revoke-noop sha=dc6b593c...
[ci-queue-wait] state=terminal-success purpose=merge branch=fix/d29-lease-revoke-noop

rc=0, both times. The guard was telling the truth: the head branch was genuinely green (CI 2585, all 9 steps).

  1. #1339 merged as 6306914965. That push started pipeline 2586 on next.
  2. Woodpecker cancels older running pipelines on the same branch. 2584 went to killed.

What was lost

Step State at kill
clone success
install success
verify success (20.2 min)
build success
publish-next-npm success, exit 0
build-gateway killed at 12.4 min

The npm publish had already completed, so #1337's artifact landed. Only the gateway build died, and 2586 rebuilds it for the superseding head. Net loss this time: none. That is luck about step ordering, not a property of the guard.

Had the kill landed a minute earlier, publish-next-npm would have died mid-publish instead.

Why the guard cannot catch it as written

--purpose merge resolves the branch under test from the PR head. A merge changes the trunk, so the state that matters is the trunk's, and nothing queries it. The guard is scoped to the thing that is safe and blind to the thing that is being mutated.

Suggested fix

For --purpose merge, check both: the head branch is terminal-success and the base branch has no pipeline in running/pending. The base ref is already available — pr-merge.sh prints it in its own preflight (base=next).

Scope note

Filed against the framework tool, not against either PR. Both merges were correct and both had terminal-green CI on their own heads.

Found by @fred, by causing it.

## Summary `ci-queue-wait.sh` checks the status of the **PR head branch**. It does not check whether the **integration trunk** has a pipeline still in flight. So it will clear a merge into `next` while `next` has a publish running, and the merge push cancels that publish. Constitution gate 6 exists to prevent exactly this. The tool that implements gate 6 cannot see it. ## How it happened tonight (not hypothetical) 1. PR #1337 merged to `next` as `af43a7a6`. That push started pipeline **2584** (`publish`). 2. While 2584 was still running, PR #1339 came up for merge. The guard was run twice, per gate 6: ``` [ci-queue-wait] platform=gitea purpose=merge branch=fix/d29-lease-revoke-noop sha=dc6b593c... [ci-queue-wait] state=terminal-success purpose=merge branch=fix/d29-lease-revoke-noop ``` `rc=0`, both times. The guard was telling the truth: the head branch was genuinely green (CI 2585, all 9 steps). 3. #1339 merged as `6306914965`. That push started pipeline **2586** on `next`. 4. Woodpecker cancels older running pipelines on the same branch. **2584 went to `killed`.** ## What was lost | Step | State at kill | |---|---| | clone | success | | install | success | | verify | success (20.2 min) | | build | success | | `publish-next-npm` | **success, exit 0** | | `build-gateway` | **killed at 12.4 min** | The npm publish had already completed, so #1337's artifact landed. Only the gateway build died, and 2586 rebuilds it for the superseding head. **Net loss this time: none.** That is luck about step ordering, not a property of the guard. Had the kill landed a minute earlier, `publish-next-npm` would have died mid-publish instead. ## Why the guard cannot catch it as written `--purpose merge` resolves the branch under test from the PR head. A merge changes the trunk, so the state that matters is the trunk's, and nothing queries it. The guard is scoped to the thing that is safe and blind to the thing that is being mutated. ## Suggested fix For `--purpose merge`, check **both**: the head branch is terminal-success **and** the base branch has no pipeline in `running`/`pending`. The base ref is already available — `pr-merge.sh` prints it in its own preflight (`base=next`). ## Scope note Filed against the framework tool, not against either PR. Both merges were correct and both had terminal-green CI on their own heads. Found by @fred, by causing it.
fred commented 2026-08-21 00:50:45 +00:00
Author
Collaborator
Copy Link
Copy Source

Reproduced live during a real merge, 2026-08-20 (fred)

Not a reading of the code this time. Observed while squash-merging #1297 to next.

What I ran, and what the wrapper's own guard ran:

# my pre-merge guard, run by hand from ~/src/mosaic-stack
ci-queue-wait.sh --purpose merge -B next
rc=0  [ci-queue-wait] platform=gitea purpose=merge branch=next sha=6306914965c0c1515651a912cfd48b9e3ee17dc1
      [ci-queue-wait] state=terminal-success purpose=merge branch=next

# the guard pr-merge.sh runs internally, same merge, seconds later
pr-merge.sh -n 1297 -m squash --expect-head 04a87262c8221e0b8cd70f5543c5badd656049d2
rc=0  [ci-queue-wait] ... branch=fix/1292-lease-broker-activation
      PR #1297 merged successfully

branch=fix/1292-lease-broker-activation is the PR head branch. Nothing is ever merged into it.
The guard that is supposed to protect the trunk from an in-flight publish asserted against the
source branch instead, and returned green because that branch's own pipeline was green.

Why this is worse than the issue title suggests. The guard did not fail, warn, or say
CANNOT_ASSERT. It printed a normal green line naming the wrong branch, one line above
merged successfully. An operator reading the merge output sees a queue guard that passed. The
only reason I caught it is that I had run the correct guard by hand a minute earlier and had the
trunk sha in front of me to compare against.

The hand-run guard is not a workaround. It asserts the trunk at time T; pr-merge.sh then
merges at time T+n having asserted nothing about the trunk. The window is exactly the one this
issue describes.

Second observation from the same merge, worth recording so nobody chases it as a provider
outage: run from a shallow shim clone whose only branch is master, the same guard returned

rc=75 [ci-queue-wait] platform=gitea purpose=merge branch=master sha=04a87262...
      curl: (22) The requested URL returned error: 404
      CANNOT_ASSERT reason=status-provider-unreachable

That is a correct fail-closed on a branch the remote does not have, reported as
status-provider-unreachable. The reason string blames the provider for a caller-side condition.
Separate defect from this one; noting it here because both were seen in the same five minutes and
they look alike from the output alone.

## Reproduced live during a real merge, 2026-08-20 (fred) Not a reading of the code this time. Observed while squash-merging #1297 to `next`. **What I ran, and what the wrapper's own guard ran:** ``` # my pre-merge guard, run by hand from ~/src/mosaic-stack ci-queue-wait.sh --purpose merge -B next rc=0 [ci-queue-wait] platform=gitea purpose=merge branch=next sha=6306914965c0c1515651a912cfd48b9e3ee17dc1 [ci-queue-wait] state=terminal-success purpose=merge branch=next # the guard pr-merge.sh runs internally, same merge, seconds later pr-merge.sh -n 1297 -m squash --expect-head 04a87262c8221e0b8cd70f5543c5badd656049d2 rc=0 [ci-queue-wait] ... branch=fix/1292-lease-broker-activation PR #1297 merged successfully ``` `branch=fix/1292-lease-broker-activation` is the PR head branch. Nothing is ever merged into it. The guard that is supposed to protect the trunk from an in-flight publish asserted against the source branch instead, and returned green because that branch's own pipeline was green. **Why this is worse than the issue title suggests.** The guard did not fail, warn, or say `CANNOT_ASSERT`. It printed a normal green line naming the wrong branch, one line above `merged successfully`. An operator reading the merge output sees a queue guard that passed. The only reason I caught it is that I had run the correct guard by hand a minute earlier and had the trunk sha in front of me to compare against. **The hand-run guard is not a workaround.** It asserts the trunk at time T; `pr-merge.sh` then merges at time T+n having asserted nothing about the trunk. The window is exactly the one this issue describes. **Second observation from the same merge, worth recording so nobody chases it as a provider outage:** run from a shallow shim clone whose only branch is `master`, the same guard returned ``` rc=75 [ci-queue-wait] platform=gitea purpose=merge branch=master sha=04a87262... curl: (22) The requested URL returned error: 404 CANNOT_ASSERT reason=status-provider-unreachable ``` That is a correct fail-closed on a branch the remote does not have, reported as `status-provider-unreachable`. The reason string blames the provider for a caller-side condition. Separate defect from this one; noting it here because both were seen in the same five minutes and they look alike from the output alone.
ops-deploy-01 commented 2026-08-24 22:52:03 +00:00
Member
Copy Link
Copy Source

Another measured instance of the declaration-vs-tooling gap in this family (fred asked for it to be recorded here): pr-create.sh defaults its PR base to the repo's default_branch (main), ignoring .mosaic/repo.json's integration_trunk declaration.

Measured 2026-08-24 on mosaicstack/stack while opening PR #1403:

  • Repo declares in .mosaic/repo.json (origin/next): "integration_trunk": "next" and notes "Never branch work off main".
  • pr-create.sh -t … -b … (no explicit -B) created the PR against base main, which on this repo is the PRODUCTION release branch — a merge there would have shipped an unreviewed fix straight to release.
  • Caught only by readback of the created PR (base.ref == main); retargeted via API PATCH to next before review.

The wrapper has all the information it needs at creation time — repo.json sits in the tree it runs from — but never reads it. Same shape as this issue's merge-guard blindness: the declaration exists, the tooling doesn't consume it.

Suggested fix when this family gets worked: wrappers that take a base/trunk decision (pr-create default base, ci-queue-wait trunk pipeline check) should read integration_trunk from .mosaic/repo.json (falling back to default_branch only when no declaration exists), or hard-fail when the declaration and their default disagree.

(Note: issue-comment.sh failed with "Configured Gitea URL not found for comment read-back verification" — this comment was posted via the REST API fallback, verified by readback.)

Another measured instance of the declaration-vs-tooling gap in this family (fred asked for it to be recorded here): **pr-create.sh defaults its PR base to the repo's default_branch (main), ignoring .mosaic/repo.json's integration_trunk declaration.** Measured 2026-08-24 on mosaicstack/stack while opening PR #1403: - Repo declares in .mosaic/repo.json (origin/next): `"integration_trunk": "next"` and notes "Never branch work off main". - `pr-create.sh -t … -b …` (no explicit -B) created the PR against **base main**, which on this repo is the PRODUCTION release branch — a merge there would have shipped an unreviewed fix straight to release. - Caught only by readback of the created PR (`base.ref == main`); retargeted via API PATCH to next before review. The wrapper has all the information it needs at creation time — repo.json sits in the tree it runs from — but never reads it. Same shape as this issue's merge-guard blindness: the declaration exists, the tooling doesn't consume it. Suggested fix when this family gets worked: wrappers that take a base/trunk decision (pr-create default base, ci-queue-wait trunk pipeline check) should read integration_trunk from .mosaic/repo.json (falling back to default_branch only when no declaration exists), or hard-fail when the declaration and their default disagree. *(Note: issue-comment.sh failed with "Configured Gitea URL not found for comment read-back verification" — this comment was posted via the REST API fallback, verified by readback.)*
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#1344