Both defects found in review by rev-code-01 on #1311.
F3 — the JSONL record interpolated every field with a bare %s. An identity comes
from git config or the environment and a cwd is whatever directory git ran in, so
either can contain a quote or a backslash. One such refusal turned the day's spool
into unparseable JSONL, and the operator would only discover it while reading the
record that explains an outage. Fields are now JSON-escaped.
F2 — the diagnostic printed "record: <spool>/<date>.jsonl" unconditionally, but
the record is only written inside the branch where mkdir -p succeeded. When the
spool cannot be created the helper named a file that does not exist, on exactly
the hosts where the escalation was lost. It now reports the real path or says
NOT WRITTEN.
Also: prettier on README.md, which was the format-step failure on pipeline 2508.
It reflowed only the two tables this branch added.
Tests: cases 14 and 15 cover both. Verified discriminating — against the previous
helper with these same tests, case 14 fails with the unparseable record printed
and case 15 fails on both assertions; against this one both pass.
The first draft of case 14 used `ls "$spool"/*.jsonl | head -1`, which under
`set -o pipefail` exits 2 on a missed glob and killed the suite with zero output
— the same silent-nonzero failure rev-code-01 hit from a partial tools/ extraction
and the reason this file exists. Replaced with a glob loop and a comment.
Two changes to one rule: a credential is resolved from exactly one place,
and an identity that cannot be resolved is refused rather than substituted.
FAIL CLOSED. Both readers ended in an unconditional fall-through to the
shared Gitea account whenever an identity did not resolve. Every seat in a
fleet therefore pushed, opened PRs and filed reviews under one account, and
a record made that way cannot be traced to the agent that made it
afterwards. The fallback now applies only where there is no attribution to
lose: a host with no fleet. Where seats exist, an unresolvable request emits
nothing, exits nonzero, explains itself on stderr, and — in the git helper —
appends a record naming the identity, host, reason and cwd, and no token
value, to ${MOSAIC_CREDENTIAL_SPOOL:-~/.local/state/mosaic-credential-escalations}.
A host runs a fleet when <brain>/fleet/agents exists, which is the signal
packages/mosaic/src/fleet/brain-home.ts already uses to decide a brain is
active, resolved the same way (MOSAIC_BRAIN_HOME, else ~/.mosaic). This is
what keeps the change a no-op for an operator who has not provisioned
per-slot tokens: no fleet directory, shared account, unchanged. It is also
why there is no environment variable to restore the old behavior — one would
reintroduce the substitution being removed.
STORE SELECTION. Both readers hardcoded ~/.config/mosaic/secrets/gitea-tokens,
so a seat's own secrets/ slot was invisible to the framework: a seat could
hold a valid credential and still be served the shared account. The store is
now chosen by what the identity is. An identity with a directory under
<brain>/fleet/agents/ is a seat and is read only from
<brain>/fleet/agents/<id>/secrets/; any other identity is a service identity
and is read from the framework store. There is no precedence between them
and no fallback from one to the other, so a seat with an empty slot is
refused even when a same-named token sits in the framework store. Two copies
of one credential are drift rather than redundancy, and drift surfaces as
the stale copy returning 401, which reads as a revoked token and sends
whoever debugs it somewhere else.
detect-platform.sh is in scope alongside git-credential-mosaic because they
are the two readers of these tokens. Patching only the git helper would make
"one credential, one location" true for push and fetch and false for
pr-create.sh, issue-create.sh and pr-review.sh, which is the harder failure
to notice.
TESTS. The three assertions that pinned the shared-account fall-through are
now fail-closed assertions, and a refusal is checked four independent ways:
nonzero exit, empty stdout, a stderr diagnostic naming identity and host,
and no shared token value anywhere in the output. The exit code alone would
pass against a helper that emitted the credential and then failed. Added:
seat-slot resolution, the no-cross-store-fallback case with a control
proving the framework-store file it declines to read is readable, no-identity
on a fleet host, the fleet gate firing on the default ~/.mosaic and not only
on an injected MOSAIC_BRAIN_HOME, and a cross-host leak check. Both suites
were run against the pre-change code as a control and fail there on exactly
the shared-token emission.
shellcheck is not installed on the authoring host, so the rewritten helper
is unlinted locally and CI is the first lint of it.
issue-comment.sh and pr-review.sh verify a durable write by pinning the
provider-returned object URL's origin and full path. The origin included the
SCHEME verbatim. On a Gitea whose ROOT_URL is configured `http://` while every
client reaches it over `https://`, the provider returns `http://` object URLs,
so the comparison rejects the provider's own truthful answer about a write that
LANDED. The failure is deterministic, not intermittent: every comment, every
time, on such a deployment.
The scheme was never what the check defends. The forgeries it exists to catch —
look-alike host, decoy path prefix, wrong owner/repo/kind/number — all vary the
HOST or the PATH. Both stay strict. `http` and `https` now collapse to one
scheme class; any other scheme (file:, ftp:, javascript:) stays distinguishing,
and an EXPLICIT non-default port still distinguishes, because a different port
is a different service on the same host.
Consequences of the bug, both observed:
- The wrapper reports failure on a comment that is durably on the issue/PR, and
attributes it to #865 ("no durable comment created"). The write landed; the
citation is wrong. Reproduced here: the harness's persisted state contains the
record while the wrapper exits 1.
- pr-review.sh's comment path is worse. On a host where no seat can create a
review OBJECT, comment-form is the only gate-16 review record obtainable, and
this check refuses all of it.
Test gap this closes: every URL fixture in both harnesses was `https://`, and
every negative case varied only host or path. The one axis that fails in
production had zero coverage — the fixtures encoded the assumption that breaks.
Added, in both suites:
- scheme-downgrade (http vs https, otherwise correct) — must be ACCEPTED. Fails
against the unmodified wrappers, passes against the fixed ones; verified in
both directions, and the negative control's captured output is the #865
misattribution above.
- explicit non-default port (`:8443`) — must stay REJECTED.
- non-web scheme (`ftp://`) — must stay REJECTED.
Also fixes test-issue-comment-readback.sh hermeticity (#1007), without which the
suite cannot run on any seat that has a per-agent Gitea token: detect-platform's
step-0 identity lookup reads ~/.config/mosaic/gitea-tokens/<identity>, outside
both XDG_CONFIG_HOME and MOSAIC_CREDENTIALS_FILE, so the suite resolved a
PRODUCTION credential and died at HTTP 401 before case 1. Same two-part fix
already merged for test-pr-review-gitea-comment.sh in #1006: a sandboxed HOME
plus an empty REPO-LOCAL mosaic.gitIdentity to shadow the global. Note the
env-var route does NOT work — detect-platform.sh reads `${MOSAIC_GIT_IDENTITY:-}`
and `:-` treats set-but-empty identically to unset.
The owner-side half of #991 (setting the deployment's Gitea ROOT_URL to https)
is not in scope here and is not made unnecessary by this change; this makes the
wrappers correct against a deployment that returns either scheme.
My previous commit said four. It is five. `test-issue-comment-readback.sh` has
the same defect and is fixed the same way, and I had already looked straight at
it and filed it as an *unrelated* silent failure. Correcting that here rather
than folding it in quietly.
WHY IT WAS MISSED — the general lesson, not the excuse. `run_comment()` sends
the wrapper's stdout AND stderr to `$OUTPUT_FILE`, and the `EXIT` trap deletes
`$WORK_DIR`. The suite therefore exits 1 with ZERO bytes on stdout and stderr,
and the one line that says what went wrong —
Error: Gitea authenticated-identity read failed with HTTP 401
— lives only inside a directory that no longer exists when anyone looks. Every
oracle I had swept the family with greps for a SYMPTOM in surviving output, so
against this suite all of them returned "nothing found", which I read as "clean"
in the first sweep and as "unrelated pre-existing failure" in the second. A
suite that discards or deletes its own evidence converts a post-hoc assay into a
non-measurement, and I wrote that sentence into the previous commit while it was
already false about a file in the same directory.
HOW IT WAS ACTUALLY FOUND. Intercept the identity read at its SOURCE instead of
grepping for its consequence: a PATH shim over `git` that logs every
`mosaic.gitIdentity` read — args, rc, and resolved value — to a file OUTSIDE any
suite's work dir, then execs the real git. Deletion-proof by construction, and
it measures the defect's cause rather than one of its symptoms. Sweeping all 16
suites with it under an ordinary invocation:
resolves a REAL identity (`mos-dt-0`) before the fix:
test-issue-comment-readback 1 read rc=1 (RED on every seat)
test-pr-review-repo-host-override 6 reads rc=0
test-ci-queue-wait-branch-absent 3 reads rc=0
the four fixed in the previous commit now read empty; the rest never read at all.
The latter two are NOT affected and are deliberately left alone: under a seat
replica (identity set, no per-slot token) neither reaches `get_gitea_token`'s
fail-loud branch, and under a canary HOME neither carries the canary credential
into any surviving artifact. They read the identity and never enter a credential
path. That residual is structural and belongs to the wrapper half of #1007 —
scoping the read with `git -C "$repo"` removes it for everyone at once.
An earlier version of that sweep reported the four fixed suites as still
resolving a real identity. That was my grep, not the suites: `value=\[..*\]` is
satisfied by `value=[] args=[…]`, because `.*` runs past the empty pair and
matches the closing bracket of the NEXT one. `value=\[[^]]` is the correct test.
Recorded because the wrong pattern failed in the direction that would have sent
me re-fixing four already-correct files.
VERIFICATION of this suite, four HOME arms, all rc=0 with zero non-empty
identity reads and the pass line on stdout: real HOME, seat replica, canary
HOME, and an empty HOME with no identity at all. Full 16-suite sweep after the
change: every suite rc=0.
CONSEQUENCE FOR THE FINDING LIST IN THE PREVIOUS COMMIT: item 2 there — the
"silently red, unrelated to #1007" suite — is withdrawn. It was #1007 all along.
Item 1 (`pr-metadata.sh:89-92`, the anonymous fallback that reports an HTTP 200
carrying valid JSON as "unknown API error") stands and is still unfixed here.
Refs #1007
CENSUS CORRECTION: FOUR suites, not the three my own #1007 audit named. The
fourth (test-pr-metadata-gitea.sh) was outside the candidate set that audit
worked from and was found only by sweeping the discriminator across all 16
tools/git/test-*.sh suites. Recording that as a correction to my finding, not
as part of the original claim.
THE DEFECT. get_gitea_token() (detect-platform.sh:502-599) resolves a per-agent
identity at STEP 0, from `git config --get mosaic.gitIdentity`, BEFORE both the
Mosaic credential loader (step 1) and the GITEA_TOKEN env check (step 2). On a
provisioned agent seat that value is set GLOBALLY in ~/.gitconfig and is
inherited by any freshly-`git init`ed repo, so step 0 reads a REAL per-slot
token out of $HOME and returns it without ever consulting the suite's own
MOSAIC_CREDENTIALS_FILE / GITEA_TOKEN fixtures. The suites were running against
production credentials, and the fixture credential each one carefully
constructs was inert.
THE FIX: an empty repo-local `mosaic.gitIdentity`. An empty local value shadows
the global one and reads back empty at rc=0, so step 0 declines. The env route
does NOT work: detect-platform.sh reads "${MOSAIC_GIT_IDENTITY:-}", and `:-`
treats set-but-empty identically to unset.
OPERATIVE vs CONTAINMENT — the two mechanisms are not interchangeable and the
comment in each suite says so. The pin is operative: it prevents the resolution.
The sandboxed HOME each suite now also gets is containment: it bounds a failure
the pin should already have prevented. Conflating them is how this class stays
invisible, because a decoy HOME REMOVES the trigger (~/.gitconfig is where the
global identity lives), so any suite audited under one reads clean however
vulnerable it is. To MEASURE, replicate a seat: a decoy HOME whose .gitconfig
sets mosaic.gitIdentity with no per-slot token, so step 0 reaches its fail-loud
branch. That note is in each file for the next auditor.
SECOND, INDEPENDENT DEFECT in test-pr-metadata-gitea.sh. Applying the pin alone
turned that suite RED — and a control at baseline 826a8b3 under a plain HOME
reproduced the same failure, so it is pre-existing, not introduced. Its
`GITEA_TOKEN="stub-token"` / `GITEA_URL="https://git.example.test"` pair can
never satisfy step 2, because step 2 accepts GITEA_TOKEN only when GITEA_URL
matches the remote host and this repo's origin is git.uscllc.com. The suite had
therefore only ever passed by resolving a REAL credential — step 0 on a seat, or
step 1 from the operator's own credentials.json. A MOSAIC_CREDENTIALS_FILE
fixture is added rather than leaning on the sandboxed HOME making step 1 find
nothing: a test that passes because production configuration is ABSENT fails the
moment it is present. Shipping the pin without this would have moved the failure
rather than removed it.
NO CI ARM. .woodpecker/ci.yml does not run these suites; packages/mosaic/
package.json:28 (test:framework-shell) runs an ENUMERATED list that excludes all
four. They run only by hand — i.e. exclusively on a provisioned seat, the one
environment where the defect is live. "Passes in CI, fails on a seat" does not
apply here; there is no CI observation at all.
VERIFICATION (seat replica = decoy HOME with mosaic.gitIdentity set, no per-slot
token; canary = same plus a marked non-credential at both per-slot paths; plain
= empty HOME; real = ordinary invocation):
- bash -n clean on all four.
- Sweep of all 16 suites at baseline 826a8b3 under the seat replica:
test-gitea-login-resolution rc=1 REACHES-STEP0; test-issue-create-
interactive-auth rc=1 REACHES-STEP0; test-pr-merge-gitea-empty-uid rc=1
REACHES-STEP0; test-pr-metadata-gitea rc=1 REACHES-STEP0.
- Same sweep after: every row rc=0 with step0 absent.
- test-gitea-token-identity flags REACHES-STEP0 in BOTH arms and is NOT a
defect: it runs under `env -i HOME="$FAKE_HOME"` (line 77) and its hit is
its own deliberate assert_failloud fixtures (lines 158-171). The fail-loud
grep matches the intended behaviour as well as the defect, so it needs the
second discriminator; recorded here so the next sweep does not re-file it.
- Durable-argv assay (a PATH shim that tees argv out of each suite's own mock
curl, because test-pr-merge-gitea-empty-uid truncates its log between phases
and its EXIT trap removes the sandbox — a post-hoc read of that suite is a
non-measurement, and "no trace" there is not a clearance):
test-pr-merge-gitea-empty-uid before: canary token in argv, fixture never
used. after: fixture token in argv, canary absent. 5 curl calls both arms.
test-pr-metadata-gitea before: canary in argv. after: both calls
carry the fixture token against git.uscllc.com.
- test-pr-metadata-gitea across seat/canary/plain HOMEs after the fix: rc=0,
rc=0, rc=0.
- All four under the real HOME: rc=0. No regression to ordinary invocation.
The comment block is duplicated across the four files rather than pointing at a
shared note. Deliberate, and matching the merged #1006 precedent
(test-pr-review-gitea-comment.sh:87-95): the reader who needs it is auditing one
file.
TWO FINDINGS DELIBERATELY NOT FIXED HERE (out of this branch's scope, to be
filed):
1. pr-metadata.sh:89-92 — the anonymous curl fallback does not check ^2, so an
HTTP 200 carrying valid JSON is reported as "unknown API error" at rc=1.
2. test-issue-comment-readback.sh exits 1 with ZERO bytes on stdout AND
stderr, dying at its first seed_state python3 heredoc. Reproduces at
baseline 826a8b3 under both a seat replica and the real HOME. Silently red
at main for everyone; unrelated to #1007.
Refs #1007
ops-01
fred
jason.woltjeandMos