fix(git): PyYAML-absent fallback parity for quoted-scalar breaks and tag/anchor properties (#865 round 11)
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

Two residual over-rejections in the PyYAML-absent fallback of
get_gitea_token_for_login, both fail-closed today but diverging from
real PyYAML 6.0.3 for valid inputs. Neither is a fail-open; the fix
loosens ONLY to exact PyYAML parity and keeps failing closed elsewhere.

Blocker A - embedded printable line break inside a quoted scalar.
The fallback split the raw document with str.splitlines(), which breaks
at NEL(U+0085), LS(U+2028) and PS(U+2029) -- all PRINTABLE to PyYAML's
Reader. Inside a flow (quoted) scalar PyYAML does not break at these: it
line-folds a double/single-quoted scalar (NEL/LF/CR -> one space; LS/PS
verbatim), resolving one token, while splitlines() cut mid-quote and
failed the whole document closed. Replace splitlines() with
_split_logical_lines, a quote-aware splitter that reproduces PyYAML's
flow folding (scan_flow_scalar_spaces/breaks) and raises on a col-0
doc marker (---/...) in a folded continuation. The same code points
UNQUOTED or in a comment still make PyYAML raise, so those stay closed.

Blocker B - leading non-specific tag / anchor property. The recognizer
blanket-rejected any scalar starting with '!' or '&'. But '! ' (bang +
space) and '&name ' are transparent node properties: PyYAML strips them
and applies normal implicit typing, so '! x'/'&a x' resolve the string
'x'. Add _strip_properties, consuming <=1 non-specific tag and <=1 anchor
(either order) and recursing on the remainder. '!x' (tag handle),
'!foo x', non-string nodes ('! 123'), duplicate properties and explicit
tags ('!!str x') still fail closed.

Fallback now matches real PyYAML 6.0.3 (oracle) on a two-direction
differential fuzz of the full Unicode line-break set x {double, single,
plain, comment, token-position} and the full leading tag/anchor space:
833 cases, 0 fail-opens, 0 present-mismatches, 52 endorsed fail-closed
over-rejects (blank-line-in-quote \n folds; explicit !!str/verbose tags;
property+quoted; anchor-without-space). Wrapper scripts pr-review.sh and
issue-comment.sh unchanged (0-line diff). Tests: additions-only sections
23 (quoted-scalar breaks) and 24 (tag/anchor properties).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Hermes Agent
2026-07-22 02:24:33 -05:00
parent 0905cdc292
commit 2d821324fe
2 changed files with 310 additions and 10 deletions

View File

@@ -922,7 +922,9 @@ elif pos == "token":
elif pos == "nelterm":
# NEL (U+0085) used as the line terminator throughout: PyYAML treats it as a
# line break (printable, NOT a ReaderError) and resolves the token; the
# fallback's splitlines() splits on NEL identically -> parity, token resolves.
# fallback's _split_logical_lines splits on NEL identically OUTSIDE a quote
# -> parity, token resolves. (Round 23 covers NEL/LS/PS INSIDE a quote, where
# PyYAML folds rather than breaks and a naive splitlines() would over-split.)
doc = ("logins:" + ch + " - name: primary" + ch
+ " url: https://git.example" + ch + " token: TOK_NEL" + ch)
else:
@@ -1007,4 +1009,116 @@ write_fixture 'logins:
'
assert_both_fail_closed "trailing colon (map indicator) token fails closed" primary git.example
# 23. EMBEDDED LINE-BREAK INSIDE A QUOTED SCALAR (#865 round-11 blocker A): the
# round-10 fallback split the raw document with str.splitlines(), which breaks
# at NEL(U+0085), LS(U+2028) and PS(U+2029) -- code points that are PRINTABLE
# to PyYAML's Reader. Inside a flow (quoted) scalar PyYAML does NOT break at
# these: it LINE-FOLDS a double/single-quoted scalar (NEL/LF/CR -> a single
# space; LS/PS -> the char verbatim), so it resolves ONE token, while
# splitlines() cut the value mid-quote and failed the whole document closed
# (over-rejection). The fallback now uses _split_logical_lines, which
# reproduces PyYAML's flow-folding. Verified empirically vs real PyYAML 6.0.3.
# write_break_fixture emits a chosen break code point in a selectable context.
write_break_fixture() {
# $1 = hex code point of the break; $2 = context: dq|sq|plain|comment|dq2
CP_HEX="$1" Q_STYLE="$2" python3 - "$FIXTURE_XDG/tea/config.yml" <<'PY'
import sys, os
cp = int(os.environ["CP_HEX"], 16)
q = os.environ["Q_STYLE"]
ch = chr(cp)
head = "logins:\n - name: primary\n url: https://git.example\n token: "
if q == "dq":
doc = head + '"tok' + ch + 'en"'
elif q == "sq":
doc = head + "'tok" + ch + "en'"
elif q == "plain":
doc = head + "tok" + ch + "en"
elif q == "dq2":
# blank line inside a quoted scalar: PyYAML folds a two-break run to a literal
# newline, which the recognizer's key regex cannot carry -> endorsed
# fail-closed over-reject (see assert_fallback_fails_closed below).
doc = head + '"tok' + ch + ch + 'en"'
elif q == "comment":
doc = ("logins:\n - name: primary\n url: https://git.example\n"
" token: TOK_PLAIN\n# c" + ch + "x")
else:
raise SystemExit("bad q")
with open(sys.argv[1], "w", encoding="utf-8") as f:
f.write(doc + "\n")
PY
}
# NEL folds to a single space inside double- AND single-quoted scalars: the token
# resolves identically in both paths (fallback no longer over-splits).
write_break_fixture 0x85 dq
assert_token "NEL inside double-quote folds to space, token resolves" "tok en" primary git.example
write_break_fixture 0x85 sq
assert_token "NEL inside single-quote folds to space, token resolves" "tok en" primary git.example
# LS(U+2028)/PS(U+2029) are preserved VERBATIM by PyYAML's flow fold (they are
# not \n-class breaks); the fallback must surface them byte-for-byte.
write_break_fixture 0x2028 dq
assert_token "LS inside double-quote is verbatim" "$(printf 'tok\342\200\250en')" primary git.example
write_break_fixture 0x2028 sq
assert_token "LS inside single-quote is verbatim" "$(printf 'tok\342\200\250en')" primary git.example
write_break_fixture 0x2029 dq
assert_token "PS inside double-quote is verbatim" "$(printf 'tok\342\200\251en')" primary git.example
# Direction-sensitivity: the SAME code points UNQUOTED (a plain scalar) or in a
# COMMENT make PyYAML raise a scanner error, so both paths must fail closed. The
# fold rule applies ONLY inside a quoted scalar.
write_break_fixture 0x85 plain
assert_token "NEL in an unquoted plain scalar fails closed" "" primary git.example
write_break_fixture 0x2028 plain
assert_token "LS in an unquoted plain scalar fails closed" "" primary git.example
write_break_fixture 0x85 comment
assert_token "NEL in a comment fails closed" "" primary git.example
# Endorsed fail-closed over-reject: a blank line inside a quoted scalar folds to a
# literal newline that the recognizer cannot carry -- PyYAML resolves a
# (newline-bearing) token, the fallback fails closed (strictly safer).
write_break_fixture 0x85 dq2
assert_fallback_fails_closed "blank-line-in-quote (NEL run) fails closed" primary git.example
# 24. LEADING NON-SPECIFIC TAG / ANCHOR PROPERTY (#865 round-11 blocker B): the
# round-10 recognizer blanket-rejected any scalar beginning with '!' or '&'.
# But a NON-SPECIFIC tag '! ' (bang + SPACE) and an anchor '&name ' are
# transparent node properties: PyYAML strips them and applies normal implicit
# typing to the node, so '! x'/'&a x' resolve the STRING 'x'. The fallback now
# consumes <=1 non-specific tag and <=1 anchor (either order) and recurses on
# the remainder. Verified empirically vs real PyYAML 6.0.3.
for pair in '! x=x' '&a x=x' '! x y=x y' '! "q"=q' "! 'q'=q" '! &b x=x' '&b ! x=x'; do
tv="${pair%%=*}"; want="${pair#*=}"
write_fixture "logins:
- name: primary
url: https://git.example
token: ${tv}
"
assert_token "property token '$tv' resolves node string" "$want" primary git.example
done
# Fail-OPEN direction stays shut. '!x' (bang + NON-space) is a tag HANDLE ->
# ConstructorError; '!foo x'/'* a'/'! !x' raise; a property over a NON-string node
# ('! 123'/'! true'/'! null') types non-str -> no usable token. All fail closed in
# BOTH paths.
for tv in '!x' '!foo x' '* a' '! !x' '! 123' '! true' '! null' '&a &b x'; do
write_fixture "logins:
- name: primary
url: https://git.example
token: ${tv}
"
assert_token "non-resolving property token '$tv' fails closed" "" primary git.example
done
# Endorsed fail-closed over-reject: an EXPLICIT tag ('!!str x', verbose
# '!<tag:yaml.org,2002:str> x') forces a string PyYAML resolves, but the fallback
# recognizes only the transparent non-specific tag and fails closed (safer).
for tv in '!!str x' '!<tag:yaml.org,2002:str> x'; do
write_fixture "logins:
- name: primary
url: https://git.example
token: ${tv}
"
assert_fallback_fails_closed "explicit-tag token '$tv' fails closed" primary git.example
done
echo "Gitea login resolution regression harness passed"