fix(git): #1007 suite hermeticity — pin repo-local mosaic.gitIdentity in five test suites #1024

Open

mos-dt-0 wants to merge 2 commits from fix/1007-suite-hermeticity into main

Author SHA1 Message Date

Author	SHA1	Message	Date
Jason Woltje	2fa6bcd576	fix(git): #1007 — test-issue-comment-readback is a FIFTH affected suite (second census correction) ci/woodpecker/pr/ci Pipeline was successful Details My previous commit said four. It is five. `test-issue-comment-readback.sh` has the same defect and is fixed the same way, and I had already looked straight at it and filed it as an unrelated silent failure. Correcting that here rather than folding it in quietly. WHY IT WAS MISSED — the general lesson, not the excuse. `run_comment()` sends the wrapper's stdout AND stderr to `$OUTPUT_FILE`, and the `EXIT` trap deletes `$WORK_DIR`. The suite therefore exits 1 with ZERO bytes on stdout and stderr, and the one line that says what went wrong — Error: Gitea authenticated-identity read failed with HTTP 401 — lives only inside a directory that no longer exists when anyone looks. Every oracle I had swept the family with greps for a SYMPTOM in surviving output, so against this suite all of them returned "nothing found", which I read as "clean" in the first sweep and as "unrelated pre-existing failure" in the second. A suite that discards or deletes its own evidence converts a post-hoc assay into a non-measurement, and I wrote that sentence into the previous commit while it was already false about a file in the same directory. HOW IT WAS ACTUALLY FOUND. Intercept the identity read at its SOURCE instead of grepping for its consequence: a PATH shim over `git` that logs every `mosaic.gitIdentity` read — args, rc, and resolved value — to a file OUTSIDE any suite's work dir, then execs the real git. Deletion-proof by construction, and it measures the defect's cause rather than one of its symptoms. Sweeping all 16 suites with it under an ordinary invocation: resolves a REAL identity (`mos-dt-0`) before the fix: test-issue-comment-readback 1 read rc=1 (RED on every seat) test-pr-review-repo-host-override 6 reads rc=0 test-ci-queue-wait-branch-absent 3 reads rc=0 the four fixed in the previous commit now read empty; the rest never read at all. The latter two are NOT affected and are deliberately left alone: under a seat replica (identity set, no per-slot token) neither reaches `get_gitea_token`'s fail-loud branch, and under a canary HOME neither carries the canary credential into any surviving artifact. They read the identity and never enter a credential path. That residual is structural and belongs to the wrapper half of #1007 — scoping the read with `git -C "$repo"` removes it for everyone at once. An earlier version of that sweep reported the four fixed suites as still resolving a real identity. That was my grep, not the suites: `value=\[..\]` is satisfied by `value=[] args=[…]`, because `.` runs past the empty pair and matches the closing bracket of the NEXT one. `value=\[[^]]` is the correct test. Recorded because the wrong pattern failed in the direction that would have sent me re-fixing four already-correct files. VERIFICATION of this suite, four HOME arms, all rc=0 with zero non-empty identity reads and the pass line on stdout: real HOME, seat replica, canary HOME, and an empty HOME with no identity at all. Full 16-suite sweep after the change: every suite rc=0. CONSEQUENCE FOR THE FINDING LIST IN THE PREVIOUS COMMIT: item 2 there — the "silently red, unrelated to #1007" suite — is withdrawn. It was #1007 all along. Item 1 (`pr-metadata.sh:89-92`, the anonymous fallback that reports an HTTP 200 carrying valid JSON as "unknown API error") stands and is still unfixed here. Refs #1007	2026-07-31 07:27:10 -05:00
Jason Woltje	1afe2b36dc	fix(git): #1007 suite hermeticity — pin repo-local mosaic.gitIdentity in four test suites CENSUS CORRECTION: FOUR suites, not the three my own #1007 audit named. The fourth (test-pr-metadata-gitea.sh) was outside the candidate set that audit worked from and was found only by sweeping the discriminator across all 16 tools/git/test-*.sh suites. Recording that as a correction to my finding, not as part of the original claim. THE DEFECT. get_gitea_token() (detect-platform.sh:502-599) resolves a per-agent identity at STEP 0, from `git config --get mosaic.gitIdentity`, BEFORE both the Mosaic credential loader (step 1) and the GITEA_TOKEN env check (step 2). On a provisioned agent seat that value is set GLOBALLY in ~/.gitconfig and is inherited by any freshly-`git init`ed repo, so step 0 reads a REAL per-slot token out of $HOME and returns it without ever consulting the suite's own MOSAIC_CREDENTIALS_FILE / GITEA_TOKEN fixtures. The suites were running against production credentials, and the fixture credential each one carefully constructs was inert. THE FIX: an empty repo-local `mosaic.gitIdentity`. An empty local value shadows the global one and reads back empty at rc=0, so step 0 declines. The env route does NOT work: detect-platform.sh reads "${MOSAIC_GIT_IDENTITY:-}", and `:-` treats set-but-empty identically to unset. OPERATIVE vs CONTAINMENT — the two mechanisms are not interchangeable and the comment in each suite says so. The pin is operative: it prevents the resolution. The sandboxed HOME each suite now also gets is containment: it bounds a failure the pin should already have prevented. Conflating them is how this class stays invisible, because a decoy HOME REMOVES the trigger (~/.gitconfig is where the global identity lives), so any suite audited under one reads clean however vulnerable it is. To MEASURE, replicate a seat: a decoy HOME whose .gitconfig sets mosaic.gitIdentity with no per-slot token, so step 0 reaches its fail-loud branch. That note is in each file for the next auditor. SECOND, INDEPENDENT DEFECT in test-pr-metadata-gitea.sh. Applying the pin alone turned that suite RED — and a control at baseline `826a8b3` under a plain HOME reproduced the same failure, so it is pre-existing, not introduced. Its `GITEA_TOKEN="stub-token"` / `GITEA_URL="https://git.example.test"` pair can never satisfy step 2, because step 2 accepts GITEA_TOKEN only when GITEA_URL matches the remote host and this repo's origin is git.uscllc.com. The suite had therefore only ever passed by resolving a REAL credential — step 0 on a seat, or step 1 from the operator's own credentials.json. A MOSAIC_CREDENTIALS_FILE fixture is added rather than leaning on the sandboxed HOME making step 1 find nothing: a test that passes because production configuration is ABSENT fails the moment it is present. Shipping the pin without this would have moved the failure rather than removed it. NO CI ARM. .woodpecker/ci.yml does not run these suites; packages/mosaic/ package.json:28 (test:framework-shell) runs an ENUMERATED list that excludes all four. They run only by hand — i.e. exclusively on a provisioned seat, the one environment where the defect is live. "Passes in CI, fails on a seat" does not apply here; there is no CI observation at all. VERIFICATION (seat replica = decoy HOME with mosaic.gitIdentity set, no per-slot token; canary = same plus a marked non-credential at both per-slot paths; plain = empty HOME; real = ordinary invocation): - bash -n clean on all four. - Sweep of all 16 suites at baseline `826a8b3` under the seat replica: test-gitea-login-resolution rc=1 REACHES-STEP0; test-issue-create- interactive-auth rc=1 REACHES-STEP0; test-pr-merge-gitea-empty-uid rc=1 REACHES-STEP0; test-pr-metadata-gitea rc=1 REACHES-STEP0. - Same sweep after: every row rc=0 with step0 absent. - test-gitea-token-identity flags REACHES-STEP0 in BOTH arms and is NOT a defect: it runs under `env -i HOME="$FAKE_HOME"` (line 77) and its hit is its own deliberate assert_failloud fixtures (lines 158-171). The fail-loud grep matches the intended behaviour as well as the defect, so it needs the second discriminator; recorded here so the next sweep does not re-file it. - Durable-argv assay (a PATH shim that tees argv out of each suite's own mock curl, because test-pr-merge-gitea-empty-uid truncates its log between phases and its EXIT trap removes the sandbox — a post-hoc read of that suite is a non-measurement, and "no trace" there is not a clearance): test-pr-merge-gitea-empty-uid before: canary token in argv, fixture never used. after: fixture token in argv, canary absent. 5 curl calls both arms. test-pr-metadata-gitea before: canary in argv. after: both calls carry the fixture token against git.uscllc.com. - test-pr-metadata-gitea across seat/canary/plain HOMEs after the fix: rc=0, rc=0, rc=0. - All four under the real HOME: rc=0. No regression to ordinary invocation. The comment block is duplicated across the four files rather than pointing at a shared note. Deliberate, and matching the merged #1006 precedent (test-pr-review-gitea-comment.sh:87-95): the reader who needs it is auditing one file. TWO FINDINGS DELIBERATELY NOT FIXED HERE (out of this branch's scope, to be filed): 1. pr-metadata.sh:89-92 — the anonymous curl fallback does not check ^2, so an HTTP 200 carrying valid JSON is reported as "unknown API error" at rc=1. 2. test-issue-comment-readback.sh exits 1 with ZERO bytes on stdout AND stderr, dying at its first seed_state python3 heredoc. Reproduces at baseline `826a8b3` under both a seat replica and the real HOME. Silently red at main for everyone; unrelated to #1007. Refs #1007	2026-07-31 07:13:13 -05:00

Jason Woltje

2fa6bcd576

fix(git): #1007 — test-issue-comment-readback is a FIFTH affected suite (second census correction)

ci/woodpecker/pr/ci Pipeline was successful

Details

My previous commit said four. It is five. `test-issue-comment-readback.sh` has
the same defect and is fixed the same way, and I had already looked straight at
it and filed it as an *unrelated* silent failure. Correcting that here rather
than folding it in quietly.

WHY IT WAS MISSED — the general lesson, not the excuse. `run_comment()` sends
the wrapper's stdout AND stderr to `$OUTPUT_FILE`, and the `EXIT` trap deletes
`$WORK_DIR`. The suite therefore exits 1 with ZERO bytes on stdout and stderr,
and the one line that says what went wrong —

    Error: Gitea authenticated-identity read failed with HTTP 401

— lives only inside a directory that no longer exists when anyone looks. Every
oracle I had swept the family with greps for a SYMPTOM in surviving output, so
against this suite all of them returned "nothing found", which I read as "clean"
in the first sweep and as "unrelated pre-existing failure" in the second. A
suite that discards or deletes its own evidence converts a post-hoc assay into a
non-measurement, and I wrote that sentence into the previous commit while it was
already false about a file in the same directory.

HOW IT WAS ACTUALLY FOUND. Intercept the identity read at its SOURCE instead of
grepping for its consequence: a PATH shim over `git` that logs every
`mosaic.gitIdentity` read — args, rc, and resolved value — to a file OUTSIDE any
suite's work dir, then execs the real git. Deletion-proof by construction, and
it measures the defect's cause rather than one of its symptoms. Sweeping all 16
suites with it under an ordinary invocation:

  resolves a REAL identity (`mos-dt-0`) before the fix:
    test-issue-comment-readback          1 read   rc=1 (RED on every seat)
    test-pr-review-repo-host-override    6 reads  rc=0
    test-ci-queue-wait-branch-absent     3 reads  rc=0
  the four fixed in the previous commit now read empty; the rest never read at all.

The latter two are NOT affected and are deliberately left alone: under a seat
replica (identity set, no per-slot token) neither reaches `get_gitea_token`'s
fail-loud branch, and under a canary HOME neither carries the canary credential
into any surviving artifact. They read the identity and never enter a credential
path. That residual is structural and belongs to the wrapper half of #1007 —
scoping the read with `git -C "$repo"` removes it for everyone at once.

An earlier version of that sweep reported the four fixed suites as still
resolving a real identity. That was my grep, not the suites: `value=\[..*\]` is
satisfied by `value=[] args=[…]`, because `.*` runs past the empty pair and
matches the closing bracket of the NEXT one. `value=\[[^]]` is the correct test.
Recorded because the wrong pattern failed in the direction that would have sent
me re-fixing four already-correct files.

VERIFICATION of this suite, four HOME arms, all rc=0 with zero non-empty
identity reads and the pass line on stdout: real HOME, seat replica, canary
HOME, and an empty HOME with no identity at all. Full 16-suite sweep after the
change: every suite rc=0.

CONSEQUENCE FOR THE FINDING LIST IN THE PREVIOUS COMMIT: item 2 there — the
"silently red, unrelated to #1007" suite — is withdrawn. It was #1007 all along.
Item 1 (`pr-metadata.sh:89-92`, the anonymous fallback that reports an HTTP 200
carrying valid JSON as "unknown API error") stands and is still unfixed here.

Refs #1007

2026-07-31 07:27:10 -05:00

Jason Woltje

1afe2b36dc

fix(git): #1007 suite hermeticity — pin repo-local mosaic.gitIdentity in four test suites

CENSUS CORRECTION: FOUR suites, not the three my own #1007 audit named. The
fourth (test-pr-metadata-gitea.sh) was outside the candidate set that audit
worked from and was found only by sweeping the discriminator across all 16
tools/git/test-*.sh suites. Recording that as a correction to my finding, not
as part of the original claim.

THE DEFECT. get_gitea_token() (detect-platform.sh:502-599) resolves a per-agent
identity at STEP 0, from `git config --get mosaic.gitIdentity`, BEFORE both the
Mosaic credential loader (step 1) and the GITEA_TOKEN env check (step 2). On a
provisioned agent seat that value is set GLOBALLY in ~/.gitconfig and is
inherited by any freshly-`git init`ed repo, so step 0 reads a REAL per-slot
token out of $HOME and returns it without ever consulting the suite's own
MOSAIC_CREDENTIALS_FILE / GITEA_TOKEN fixtures. The suites were running against
production credentials, and the fixture credential each one carefully
constructs was inert.

THE FIX: an empty repo-local `mosaic.gitIdentity`. An empty local value shadows
the global one and reads back empty at rc=0, so step 0 declines. The env route
does NOT work: detect-platform.sh reads "${MOSAIC_GIT_IDENTITY:-}", and `:-`
treats set-but-empty identically to unset.

OPERATIVE vs CONTAINMENT — the two mechanisms are not interchangeable and the
comment in each suite says so. The pin is operative: it prevents the resolution.
The sandboxed HOME each suite now also gets is containment: it bounds a failure
the pin should already have prevented. Conflating them is how this class stays
invisible, because a decoy HOME REMOVES the trigger (~/.gitconfig is where the
global identity lives), so any suite audited under one reads clean however
vulnerable it is. To MEASURE, replicate a seat: a decoy HOME whose .gitconfig
sets mosaic.gitIdentity with no per-slot token, so step 0 reaches its fail-loud
branch. That note is in each file for the next auditor.

SECOND, INDEPENDENT DEFECT in test-pr-metadata-gitea.sh. Applying the pin alone
turned that suite RED — and a control at baseline 826a8b3 under a plain HOME
reproduced the same failure, so it is pre-existing, not introduced. Its
`GITEA_TOKEN="stub-token"` / `GITEA_URL="https://git.example.test"` pair can
never satisfy step 2, because step 2 accepts GITEA_TOKEN only when GITEA_URL
matches the remote host and this repo's origin is git.uscllc.com. The suite had
therefore only ever passed by resolving a REAL credential — step 0 on a seat, or
step 1 from the operator's own credentials.json. A MOSAIC_CREDENTIALS_FILE
fixture is added rather than leaning on the sandboxed HOME making step 1 find
nothing: a test that passes because production configuration is ABSENT fails the
moment it is present. Shipping the pin without this would have moved the failure
rather than removed it.

NO CI ARM. .woodpecker/ci.yml does not run these suites; packages/mosaic/
package.json:28 (test:framework-shell) runs an ENUMERATED list that excludes all
four. They run only by hand — i.e. exclusively on a provisioned seat, the one
environment where the defect is live. "Passes in CI, fails on a seat" does not
apply here; there is no CI observation at all.

VERIFICATION (seat replica = decoy HOME with mosaic.gitIdentity set, no per-slot
token; canary = same plus a marked non-credential at both per-slot paths; plain
= empty HOME; real = ordinary invocation):
  - bash -n clean on all four.
  - Sweep of all 16 suites at baseline 826a8b3 under the seat replica:
    test-gitea-login-resolution rc=1 REACHES-STEP0; test-issue-create-
    interactive-auth rc=1 REACHES-STEP0; test-pr-merge-gitea-empty-uid rc=1
    REACHES-STEP0; test-pr-metadata-gitea rc=1 REACHES-STEP0.
  - Same sweep after: every row rc=0 with step0 absent.
  - test-gitea-token-identity flags REACHES-STEP0 in BOTH arms and is NOT a
    defect: it runs under `env -i HOME="$FAKE_HOME"` (line 77) and its hit is
    its own deliberate assert_failloud fixtures (lines 158-171). The fail-loud
    grep matches the intended behaviour as well as the defect, so it needs the
    second discriminator; recorded here so the next sweep does not re-file it.
  - Durable-argv assay (a PATH shim that tees argv out of each suite's own mock
    curl, because test-pr-merge-gitea-empty-uid truncates its log between phases
    and its EXIT trap removes the sandbox — a post-hoc read of that suite is a
    non-measurement, and "no trace" there is not a clearance):
      test-pr-merge-gitea-empty-uid  before: canary token in argv, fixture never
        used. after: fixture token in argv, canary absent. 5 curl calls both arms.
      test-pr-metadata-gitea         before: canary in argv. after: both calls
        carry the fixture token against git.uscllc.com.
  - test-pr-metadata-gitea across seat/canary/plain HOMEs after the fix: rc=0,
    rc=0, rc=0.
  - All four under the real HOME: rc=0. No regression to ordinary invocation.

The comment block is duplicated across the four files rather than pointing at a
shared note. Deliberate, and matching the merged #1006 precedent
(test-pr-review-gitea-comment.sh:87-95): the reader who needs it is auditing one
file.

TWO FINDINGS DELIBERATELY NOT FIXED HERE (out of this branch's scope, to be
filed):
  1. pr-metadata.sh:89-92 — the anonymous curl fallback does not check ^2, so an
     HTTP 200 carrying valid JSON is reported as "unknown API error" at rc=1.
  2. test-issue-comment-readback.sh exits 1 with ZERO bytes on stdout AND
     stderr, dying at its first seed_state python3 heredoc. Reproduces at
     baseline 826a8b3 under both a seat replica and the real HOME. Silently red
     at main for everyone; unrelated to #1007.

Refs #1007

2026-07-31 07:13:13 -05:00