mosaicstack/stack
25
0
Fork 1
Code Issues 344 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

CI: exercise both wake HMAC legs (W3 digest + W7 beacon) — add openssl, flip SKIP→hard-require in CI #912

New Issue
Closed
opened 2026-07-26 03:27:19 +00:00 by jason.woltje · 2 comments
jason.woltje commented 2026-07-26 03:27:19 +00:00
Owner
Copy Link
Copy Source

Problem. The wake component trust layer is currently never exercised in CI. The CI image has no openssl, so both HMAC harnesses SKIP their signing/verification groups: the W3 digest HMAC (§2.5 non-circular signer) and the W7 beacon HMAC-verify (B12, monitor-side spoof rejection). Signing/verification can regress with fully green CI — the same unenforced-assertions-rot class closed for digest-class tests in #901.

Surfaced by W7 (#911): B12 hard-failed in CI only because it lacked the openssl-absent SKIP-guard W3 uses; the immediate #911 fix restores that skip (consistent, narrow) but leaves BOTH legs unexercised in CI — hence this tracked follow-up.

Scope.

  1. Add openssl to the CI image (or a dedicated hmac-leg CI step).
  2. Flip both SKIP-guards (W3 digest harness + W7 test-wake-beacon.sh B12) to hard-require in CI via env detection, KEEPING the skip for openssl-less local DEV envs.
  3. Red-first proof both legs run: a targeted mutation to each sign/verify path must turn CI RED.

Priority: normal; pre-vector preferred — the §4 vector G6 evidence must come from CI-exercised HMAC, not skipped legs.

Refs: #892 (wake EPIC) · #901 (assertion-enforcement precedent) · #911 (where the skip-gap surfaced).

**Problem.** The wake component trust layer is currently never exercised in CI. The CI image has no openssl, so both HMAC harnesses SKIP their signing/verification groups: the W3 **digest** HMAC (§2.5 non-circular signer) and the W7 **beacon** HMAC-verify (B12, monitor-side spoof rejection). Signing/verification can regress with fully green CI — the same unenforced-assertions-rot class closed for digest-class tests in #901. Surfaced by W7 (#911): B12 hard-failed in CI only because it lacked the openssl-absent SKIP-guard W3 uses; the immediate #911 fix restores that skip (consistent, narrow) but leaves BOTH legs unexercised in CI — hence this tracked follow-up. **Scope.** 1. Add openssl to the CI image (or a dedicated hmac-leg CI step). 2. Flip both SKIP-guards (W3 digest harness + W7 test-wake-beacon.sh B12) to **hard-require in CI** via env detection, KEEPING the skip for openssl-less local DEV envs. 3. **Red-first proof both legs run**: a targeted mutation to each sign/verify path must turn CI RED. **Priority:** normal; **pre-vector preferred** — the §4 vector G6 evidence must come from CI-exercised HMAC, not skipped legs. **Refs:** #892 (wake EPIC) · #901 (assertion-enforcement precedent) · #911 (where the skip-gap surfaced).
jason.woltje commented 2026-07-26 05:35:59 +00:00
Author
Owner
Copy Link
Copy Source

SCOPE GROWTH (dragon-lin pilot finding, via #916 / #914). This issue is now MORE load-bearing than originally scoped. The #914 digest work rescoped a file-level openssl guard and revealed that the ENTIRE digest-hmac harness has NEVER actually run in CI — the whole file was skipped whenever openssl is absent (which is always, in CI). So it is not just the two HMAC legs (W3 digest sign + W7 beacon) that are unexercised in CI: the NON-HMAC digest tests D1-D4 (cumulative-state delta, hard-locator enforcement, secret redaction/scrub) plus the new D5/D6 (ack namespace, digest-class locator) also never run in CI, and they FAIL under the CI environment when unmasked. #916 reverts the unmask (restores the whole-file skip) so the renderer contract fix can land without curing the CI-env problems in the wrong PR; the CI-clean work lands HERE. GROWN SCOPE: (1) add openssl to CI (or a dedicated hmac-leg step); (2) make the WHOLE digest-hmac suite D1-D6 + H1-H2 run AND pass under the CI env (root + openssl-absent-then-present) — flip the whole-file skip to hard-require-in-CI keeping dev-skip; (3) red-first proof each runs. D1-D6 CI failure-mode evidence (captured from the #916 revert builder) will be attached below before the #916 revert merges. This is pre-vector: the §4 vector G6 evidence must come from a digest/HMAC suite that actually runs in CI, which today it does not at all.

**SCOPE GROWTH (dragon-lin pilot finding, via #916 / #914).** This issue is now MORE load-bearing than originally scoped. The #914 digest work rescoped a file-level openssl guard and revealed that the ENTIRE digest-hmac harness has NEVER actually run in CI — the whole file was skipped whenever openssl is absent (which is always, in CI). So it is not just the two HMAC legs (W3 digest sign + W7 beacon) that are unexercised in CI: the NON-HMAC digest tests D1-D4 (cumulative-state delta, hard-locator enforcement, secret redaction/scrub) plus the new D5/D6 (ack namespace, digest-class locator) also never run in CI, and they FAIL under the CI environment when unmasked. #916 reverts the unmask (restores the whole-file skip) so the renderer contract fix can land without curing the CI-env problems in the wrong PR; the CI-clean work lands HERE. GROWN SCOPE: (1) add openssl to CI (or a dedicated hmac-leg step); (2) make the WHOLE digest-hmac suite D1-D6 + H1-H2 run AND pass under the CI env (root + openssl-absent-then-present) — flip the whole-file skip to hard-require-in-CI keeping dev-skip; (3) red-first proof each runs. D1-D6 CI failure-mode evidence (captured from the #916 revert builder) will be attached below before the #916 revert merges. This is pre-vector: the §4 vector G6 evidence must come from a digest/HMAC suite that actually runs in CI, which today it does not at all.
jason.woltje commented 2026-07-26 05:43:48 +00:00
Author
Owner
Copy Link
Copy Source

D1-D6 CI-failure-mode evidence (preserved here before #916 reverts the unmask). When #914 (PR #916) temporarily scoped the digest-hmac harness's openssl guard to only H1/H2 (so D1-D6 run without openssl), Woodpecker CI pipeline 2056 (@ 2fb15b3a) ran the whole suite and FAILED with 9 assertions across FOUR groups — pre-existing D1/D4 (which had never run in CI) and the new D5/D6:

FAIL: D1: OLDER change (issue #11) dropped — digest is a delta, not cumulative state
FAIL: D1: newer change (issue #22) missing
FAIL: D4: secret redaction marker absent — canary may not have been scrubbed
FAIL: D4: legitimate 40-hex SHA locator was wrongly scrubbed
FAIL: D5: embedded ack line has no explicit WAKE_AGENT=someagent prefix — env-less copy-run silently resolves to 'default'
FAIL: D5: env-less copy-run of the embedded ack line exited non-zero (rc=1)
FAIL: D5: env-less copy-run did NOT advance the someagent namespace's consumed_seq (got '0')
FAIL: D6: no ORIENTATION line found for seq 1
FAIL: D6: digest-class ORIENTATION pointer does not carry a usable locator []
wake digest/hmac harness: FAILED (9 assertion(s))

KEY DIAGNOSTIC — the failure does NOT reproduce in a masked sandbox. The #916-revert builder re-ran the exact PR-state (guard scoped to H1/H2) with openssl masked out of PATH in the build sandbox: all 6 groups (D1-D6, H1/H2 individually skipped) PASSED, harness rc=0. So the Woodpecker CI failure is NOT simply "openssl absent" — a masked non-CI env passes D1-D6. The divergence is Woodpecker-runner-SPECIFIC: candidate causes are root-vs-user (CI runs as root — cf. the #915 T9 chmod/root finding, OB bee9d29b), a jq/bash/coreutils VERSION difference, LOCALE (D4's redaction/scrub + D1's delta ordering are string/regex-sensitive), or container-FS specifics. The diverse failure signature (delta ordering, secret-scrub, ack-namespace, locator) points at a systemic env/tooling difference rather than 4 independent bugs.

Implication for #912 scope: #912 is now 'make the digest suite actually run+pass in the real Woodpecker CI env', which requires (a) reproducing the CI runner env (root + its exact tool versions/locale — a masked local run is INSUFFICIENT, as shown) and (b) fixing whatever diverges, THEN flipping the whole-file skip to hard-require-in-CI. The masked-suite acceptance gate that catches openssl gaps does NOT catch this class. This is pre-vector: §4 G6 evidence must come from a digest/HMAC suite that runs in the actual CI runner.

**D1-D6 CI-failure-mode evidence (preserved here before #916 reverts the unmask).** When #914 (PR #916) temporarily scoped the digest-hmac harness's openssl guard to only H1/H2 (so D1-D6 run without openssl), Woodpecker CI pipeline 2056 (@ 2fb15b3a) ran the whole suite and FAILED with 9 assertions across FOUR groups — pre-existing D1/D4 (which had never run in CI) and the new D5/D6: ``` FAIL: D1: OLDER change (issue #11) dropped — digest is a delta, not cumulative state FAIL: D1: newer change (issue #22) missing FAIL: D4: secret redaction marker absent — canary may not have been scrubbed FAIL: D4: legitimate 40-hex SHA locator was wrongly scrubbed FAIL: D5: embedded ack line has no explicit WAKE_AGENT=someagent prefix — env-less copy-run silently resolves to 'default' FAIL: D5: env-less copy-run of the embedded ack line exited non-zero (rc=1) FAIL: D5: env-less copy-run did NOT advance the someagent namespace's consumed_seq (got '0') FAIL: D6: no ORIENTATION line found for seq 1 FAIL: D6: digest-class ORIENTATION pointer does not carry a usable locator [] wake digest/hmac harness: FAILED (9 assertion(s)) ``` **KEY DIAGNOSTIC — the failure does NOT reproduce in a masked sandbox.** The #916-revert builder re-ran the exact PR-state (guard scoped to H1/H2) with openssl masked out of PATH in the build sandbox: all 6 groups (D1-D6, H1/H2 individually skipped) PASSED, harness rc=0. So the Woodpecker CI failure is NOT simply "openssl absent" — a masked non-CI env passes D1-D6. The divergence is Woodpecker-runner-SPECIFIC: candidate causes are root-vs-user (CI runs as root — cf. the #915 T9 chmod/root finding, OB bee9d29b), a jq/bash/coreutils VERSION difference, LOCALE (D4's redaction/scrub + D1's delta ordering are string/regex-sensitive), or container-FS specifics. The diverse failure signature (delta ordering, secret-scrub, ack-namespace, locator) points at a systemic env/tooling difference rather than 4 independent bugs. **Implication for #912 scope:** #912 is now 'make the digest suite actually run+pass in the real Woodpecker CI env', which requires (a) reproducing the CI runner env (root + its exact tool versions/locale — a masked local run is INSUFFICIENT, as shown) and (b) fixing whatever diverges, THEN flipping the whole-file skip to hard-require-in-CI. The masked-suite acceptance gate that catches openssl gaps does NOT catch this class. This is pre-vector: §4 G6 evidence must come from a digest/HMAC suite that runs in the actual CI runner.
Ghost closed this issue 2026-07-26 08:11:30 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) darkwing (Mosaic fleet seat darkwing) dewey (Mosaic fleet seat dewey) fargo filbert (Mosaic fleet seat filbert) fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) rocko (Mosaic fleet seat rocko) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#912