fix(git): PyYAML-absent fallback parity for quoted-scalar breaks and tag/anchor properties (#865 round 11)
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
Two residual over-rejections in the PyYAML-absent fallback of
get_gitea_token_for_login, both fail-closed today but diverging from
real PyYAML 6.0.3 for valid inputs. Neither is a fail-open; the fix
loosens ONLY to exact PyYAML parity and keeps failing closed elsewhere.
Blocker A - embedded printable line break inside a quoted scalar.
The fallback split the raw document with str.splitlines(), which breaks
at NEL(U+0085), LS(U+2028) and PS(U+2029) -- all PRINTABLE to PyYAML's
Reader. Inside a flow (quoted) scalar PyYAML does not break at these: it
line-folds a double/single-quoted scalar (NEL/LF/CR -> one space; LS/PS
verbatim), resolving one token, while splitlines() cut mid-quote and
failed the whole document closed. Replace splitlines() with
_split_logical_lines, a quote-aware splitter that reproduces PyYAML's
flow folding (scan_flow_scalar_spaces/breaks) and raises on a col-0
doc marker (---/...) in a folded continuation. The same code points
UNQUOTED or in a comment still make PyYAML raise, so those stay closed.
Blocker B - leading non-specific tag / anchor property. The recognizer
blanket-rejected any scalar starting with '!' or '&'. But '! ' (bang +
space) and '&name ' are transparent node properties: PyYAML strips them
and applies normal implicit typing, so '! x'/'&a x' resolve the string
'x'. Add _strip_properties, consuming <=1 non-specific tag and <=1 anchor
(either order) and recursing on the remainder. '!x' (tag handle),
'!foo x', non-string nodes ('! 123'), duplicate properties and explicit
tags ('!!str x') still fail closed.
Fallback now matches real PyYAML 6.0.3 (oracle) on a two-direction
differential fuzz of the full Unicode line-break set x {double, single,
plain, comment, token-position} and the full leading tag/anchor space:
833 cases, 0 fail-opens, 0 present-mismatches, 52 endorsed fail-closed
over-rejects (blank-line-in-quote \n folds; explicit !!str/verbose tags;
property+quoted; anchor-without-space). Wrapper scripts pr-review.sh and
issue-comment.sh unchanged (0-line diff). Tests: additions-only sections
23 (quoted-scalar breaks) and 24 (tag/anchor properties).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -922,7 +922,9 @@ elif pos == "token":
|
||||
elif pos == "nelterm":
|
||||
# NEL (U+0085) used as the line terminator throughout: PyYAML treats it as a
|
||||
# line break (printable, NOT a ReaderError) and resolves the token; the
|
||||
# fallback's splitlines() splits on NEL identically -> parity, token resolves.
|
||||
# fallback's _split_logical_lines splits on NEL identically OUTSIDE a quote
|
||||
# -> parity, token resolves. (Round 23 covers NEL/LS/PS INSIDE a quote, where
|
||||
# PyYAML folds rather than breaks and a naive splitlines() would over-split.)
|
||||
doc = ("logins:" + ch + " - name: primary" + ch
|
||||
+ " url: https://git.example" + ch + " token: TOK_NEL" + ch)
|
||||
else:
|
||||
@@ -1007,4 +1009,116 @@ write_fixture 'logins:
|
||||
'
|
||||
assert_both_fail_closed "trailing colon (map indicator) token fails closed" primary git.example
|
||||
|
||||
# 23. EMBEDDED LINE-BREAK INSIDE A QUOTED SCALAR (#865 round-11 blocker A): the
|
||||
# round-10 fallback split the raw document with str.splitlines(), which breaks
|
||||
# at NEL(U+0085), LS(U+2028) and PS(U+2029) -- code points that are PRINTABLE
|
||||
# to PyYAML's Reader. Inside a flow (quoted) scalar PyYAML does NOT break at
|
||||
# these: it LINE-FOLDS a double/single-quoted scalar (NEL/LF/CR -> a single
|
||||
# space; LS/PS -> the char verbatim), so it resolves ONE token, while
|
||||
# splitlines() cut the value mid-quote and failed the whole document closed
|
||||
# (over-rejection). The fallback now uses _split_logical_lines, which
|
||||
# reproduces PyYAML's flow-folding. Verified empirically vs real PyYAML 6.0.3.
|
||||
# write_break_fixture emits a chosen break code point in a selectable context.
|
||||
write_break_fixture() {
|
||||
# $1 = hex code point of the break; $2 = context: dq|sq|plain|comment|dq2
|
||||
CP_HEX="$1" Q_STYLE="$2" python3 - "$FIXTURE_XDG/tea/config.yml" <<'PY'
|
||||
import sys, os
|
||||
cp = int(os.environ["CP_HEX"], 16)
|
||||
q = os.environ["Q_STYLE"]
|
||||
ch = chr(cp)
|
||||
head = "logins:\n - name: primary\n url: https://git.example\n token: "
|
||||
if q == "dq":
|
||||
doc = head + '"tok' + ch + 'en"'
|
||||
elif q == "sq":
|
||||
doc = head + "'tok" + ch + "en'"
|
||||
elif q == "plain":
|
||||
doc = head + "tok" + ch + "en"
|
||||
elif q == "dq2":
|
||||
# blank line inside a quoted scalar: PyYAML folds a two-break run to a literal
|
||||
# newline, which the recognizer's key regex cannot carry -> endorsed
|
||||
# fail-closed over-reject (see assert_fallback_fails_closed below).
|
||||
doc = head + '"tok' + ch + ch + 'en"'
|
||||
elif q == "comment":
|
||||
doc = ("logins:\n - name: primary\n url: https://git.example\n"
|
||||
" token: TOK_PLAIN\n# c" + ch + "x")
|
||||
else:
|
||||
raise SystemExit("bad q")
|
||||
with open(sys.argv[1], "w", encoding="utf-8") as f:
|
||||
f.write(doc + "\n")
|
||||
PY
|
||||
}
|
||||
|
||||
# NEL folds to a single space inside double- AND single-quoted scalars: the token
|
||||
# resolves identically in both paths (fallback no longer over-splits).
|
||||
write_break_fixture 0x85 dq
|
||||
assert_token "NEL inside double-quote folds to space, token resolves" "tok en" primary git.example
|
||||
write_break_fixture 0x85 sq
|
||||
assert_token "NEL inside single-quote folds to space, token resolves" "tok en" primary git.example
|
||||
# LS(U+2028)/PS(U+2029) are preserved VERBATIM by PyYAML's flow fold (they are
|
||||
# not \n-class breaks); the fallback must surface them byte-for-byte.
|
||||
write_break_fixture 0x2028 dq
|
||||
assert_token "LS inside double-quote is verbatim" "$(printf 'tok\342\200\250en')" primary git.example
|
||||
write_break_fixture 0x2028 sq
|
||||
assert_token "LS inside single-quote is verbatim" "$(printf 'tok\342\200\250en')" primary git.example
|
||||
write_break_fixture 0x2029 dq
|
||||
assert_token "PS inside double-quote is verbatim" "$(printf 'tok\342\200\251en')" primary git.example
|
||||
|
||||
# Direction-sensitivity: the SAME code points UNQUOTED (a plain scalar) or in a
|
||||
# COMMENT make PyYAML raise a scanner error, so both paths must fail closed. The
|
||||
# fold rule applies ONLY inside a quoted scalar.
|
||||
write_break_fixture 0x85 plain
|
||||
assert_token "NEL in an unquoted plain scalar fails closed" "" primary git.example
|
||||
write_break_fixture 0x2028 plain
|
||||
assert_token "LS in an unquoted plain scalar fails closed" "" primary git.example
|
||||
write_break_fixture 0x85 comment
|
||||
assert_token "NEL in a comment fails closed" "" primary git.example
|
||||
|
||||
# Endorsed fail-closed over-reject: a blank line inside a quoted scalar folds to a
|
||||
# literal newline that the recognizer cannot carry -- PyYAML resolves a
|
||||
# (newline-bearing) token, the fallback fails closed (strictly safer).
|
||||
write_break_fixture 0x85 dq2
|
||||
assert_fallback_fails_closed "blank-line-in-quote (NEL run) fails closed" primary git.example
|
||||
|
||||
# 24. LEADING NON-SPECIFIC TAG / ANCHOR PROPERTY (#865 round-11 blocker B): the
|
||||
# round-10 recognizer blanket-rejected any scalar beginning with '!' or '&'.
|
||||
# But a NON-SPECIFIC tag '! ' (bang + SPACE) and an anchor '&name ' are
|
||||
# transparent node properties: PyYAML strips them and applies normal implicit
|
||||
# typing to the node, so '! x'/'&a x' resolve the STRING 'x'. The fallback now
|
||||
# consumes <=1 non-specific tag and <=1 anchor (either order) and recurses on
|
||||
# the remainder. Verified empirically vs real PyYAML 6.0.3.
|
||||
for pair in '! x=x' '&a x=x' '! x y=x y' '! "q"=q' "! 'q'=q" '! &b x=x' '&b ! x=x'; do
|
||||
tv="${pair%%=*}"; want="${pair#*=}"
|
||||
write_fixture "logins:
|
||||
- name: primary
|
||||
url: https://git.example
|
||||
token: ${tv}
|
||||
"
|
||||
assert_token "property token '$tv' resolves node string" "$want" primary git.example
|
||||
done
|
||||
|
||||
# Fail-OPEN direction stays shut. '!x' (bang + NON-space) is a tag HANDLE ->
|
||||
# ConstructorError; '!foo x'/'* a'/'! !x' raise; a property over a NON-string node
|
||||
# ('! 123'/'! true'/'! null') types non-str -> no usable token. All fail closed in
|
||||
# BOTH paths.
|
||||
for tv in '!x' '!foo x' '* a' '! !x' '! 123' '! true' '! null' '&a &b x'; do
|
||||
write_fixture "logins:
|
||||
- name: primary
|
||||
url: https://git.example
|
||||
token: ${tv}
|
||||
"
|
||||
assert_token "non-resolving property token '$tv' fails closed" "" primary git.example
|
||||
done
|
||||
|
||||
# Endorsed fail-closed over-reject: an EXPLICIT tag ('!!str x', verbose
|
||||
# '!<tag:yaml.org,2002:str> x') forces a string PyYAML resolves, but the fallback
|
||||
# recognizes only the transparent non-specific tag and fails closed (safer).
|
||||
for tv in '!!str x' '!<tag:yaml.org,2002:str> x'; do
|
||||
write_fixture "logins:
|
||||
- name: primary
|
||||
url: https://git.example
|
||||
token: ${tv}
|
||||
"
|
||||
assert_fallback_fails_closed "explicit-tag token '$tv' fails closed" primary git.example
|
||||
done
|
||||
|
||||
echo "Gitea login resolution regression harness passed"
|
||||
|
||||
Reference in New Issue
Block a user