mosaicstack/stack
1
0
Fork 0
Code Issues 85 Pull Requests Actions Packages Projects Releases 15 Wiki Activity

wake: seq-integrity fault-injections skip in non-privileged CI — need mount-free privilege-invariant injection #934

New Issue
Closed
opened 2026-07-26 14:52:52 +00:00 by jason.woltje · 0 comments
jason.woltje commented 2026-07-26 14:52:52 +00:00
Owner
Copy Link

Problem

The wake allocator's seq-integrity fault-injection tests (test-wake-store-ack.sh) build their fault injection on unshare --mount --user --map-root-user + a bind-mount to force EBUSY on the target write. The real Woodpecker CI runner is non-privileged and denies mount-in-userns:

unshare: can't mount none on / (flags:0x44000): Permission denied

So these injections cannot set up in the real CI runner and are guarded to SKIP:

  • T9 (#915, arrow-1 no-burn: cursor must not advance before the durable write) — guards + skips.
  • T11 (#917, cursor-write gate: final observed_seq write fail-loud + observed.set rollback) — now guarded to skip too (#933).

They run + assert fully only in a privileged/local env (docker run --privileged, dev boxes). In the actual unprivileged runner they are no-ops.

Why it matters

This is the skipped-trust-layer condition — the same class #912 cured for the digest/HMAC suite (making it run in real CI). A test that silently skips its own assertion in CI is unenforced-assertion rot wearing a new coat: the allocator's most safety-critical invariants (no cursor burn before durable write; both-advance-or-neither on the cursor gate) are not exercised by the pipeline that gates merges.

Direction (per Mos ruling)

Replace the mount-based injection with a mount-free, privilege-invariant mechanism that runs unprivileged in the real runner. Candidates:

  1. rlimit-based — ulimit -f 0 (RLIMIT_FSIZE=0) makes the durable write fail (SIGXFSZ/EFBIG) for any uid with no mounts and no userns. Scope it to the injected write only (subshell), assert the fail-loud diagnostic + rollback fire.
  2. Harness-controlled injection seam — a test-only hook the store honors (e.g. an env-gated forced-failure point on the final cursor _atomic_write) so the harness can trigger the failure path deterministically without OS capabilities.

Non-goal: granting the runner mount/CAP_SYS_ADMIN — weakening CI isolation to enable a test inverts the priority.

Acceptance

  • The seq-integrity injections (T9 + T11 equivalents) demonstrably RUN and ASSERT in the real non-privileged Woodpecker runner (verify by the PR's own pipeline exercising the failure path — not a --privileged local run, which masks it).
  • No product-code behavior change to store.sh on-disk format or semantics; test-mechanism + any minimal env-gated seam only.
  • Full test:framework-shell green in a non-privileged node:24-alpine container.

Context

Env-divergence doctrine sub-class 5b (privileged-vs-unprivileged container; docker-reproduce must match the runner's PRIVILEGE LEVEL, not just its image). Surfaced by #933 (#917) CI-red; T11 unguarded-fail vs T9 guarded-skip.

## Problem The wake allocator's seq-integrity fault-injection tests (`test-wake-store-ack.sh`) build their fault injection on `unshare --mount --user --map-root-user` + a bind-mount to force EBUSY on the target write. The **real Woodpecker CI runner is non-privileged and denies mount-in-userns**: ``` unshare: can't mount none on / (flags:0x44000): Permission denied ``` So these injections **cannot set up in the real CI runner** and are guarded to SKIP: - **T9** (#915, arrow-1 no-burn: cursor must not advance before the durable write) — guards + skips. - **T11** (#917, cursor-write gate: final observed_seq write fail-loud + observed.set rollback) — now guarded to skip too (#933). They run + assert fully **only** in a privileged/local env (`docker run --privileged`, dev boxes). In the actual unprivileged runner they are **no-ops**. ## Why it matters This is the **skipped-trust-layer** condition — the same class #912 cured for the digest/HMAC suite (making it run in *real* CI). A test that silently skips its own assertion in CI is unenforced-assertion rot wearing a new coat: the allocator's most safety-critical invariants (no cursor burn before durable write; both-advance-or-neither on the cursor gate) are **not exercised by the pipeline that gates merges**. ## Direction (per Mos ruling) Replace the mount-based injection with a **mount-free, privilege-invariant** mechanism that runs **unprivileged** in the real runner. Candidates: 1. **rlimit-based** — `ulimit -f 0` (RLIMIT_FSIZE=0) makes the durable write fail (SIGXFSZ/EFBIG) for any uid with **no mounts and no userns**. Scope it to the injected write only (subshell), assert the fail-loud diagnostic + rollback fire. 2. **Harness-controlled injection seam** — a test-only hook the store honors (e.g. an env-gated forced-failure point on the final cursor `_atomic_write`) so the harness can trigger the failure path deterministically without OS capabilities. **Non-goal:** granting the runner mount/`CAP_SYS_ADMIN` — weakening CI isolation to enable a test inverts the priority. ## Acceptance - The seq-integrity injections (T9 + T11 equivalents) **demonstrably RUN and ASSERT** in the real non-privileged Woodpecker runner (verify by the PR's own pipeline exercising the failure path — not a `--privileged` local run, which masks it). - No product-code behavior change to `store.sh` on-disk format or semantics; test-mechanism + any minimal env-gated seam only. - Full `test:framework-shell` green in a non-privileged `node:24-alpine` container. ## Context Env-divergence doctrine sub-class 5b (privileged-vs-unprivileged container; docker-reproduce must match the runner's PRIVILEGE LEVEL, not just its image). Surfaced by #933 (#917) CI-red; T11 unguarded-fail vs T9 guarded-skip.
Mos closed this issue 2026-07-26 16:34:59 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
jason.woltje
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#934
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?