fix(git): match PyYAML Reader.NON_PRINTABLE and block-context plain rules in tea token fallback (#865)
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

Round-10 auditor blockers in the PyYAML-absent fallback of
get_gitea_token_for_login (detect-platform.sh):

Blocker 1 (fail-open): the round-9 control-char guard used a hand-rolled
C0/DEL subset that missed code points PyYAML's Reader also rejects -- the
C1 block (U+0080-0084, U+0086-009F), the surrogate range, and the BMP
noncharacters U+FFFE/U+FFFF -- so the fallback still emitted a token from
documents PyYAML rejects whole. _FORBIDDEN_CONTROL now uses PyYAML 6.0.3's
EXACT Reader.NON_PRINTABLE character class (negated), verified 0-mismatch
against real PyYAML across the full BMP + astral range.

Blocker 2 (over-reject): the recognizer blanket-rejected any plain scalar
containing a flow indicator, but in BLOCK context (where a tea config
value always sits) PyYAML treats ',[]{}' as ordinary content and treats
'!&*|>#%@`' and quotes as significant only at the first non-space char.
The blanket scan dropped tokens PyYAML emits verbatim (internal '!', ','
'[' ']' '{' '}' '#'-not-space-preceded, ':'-not-space-followed). Removed
it; the leading-char guard and the ' #' / ': ' / trailing-':' guards keep
the fail-open direction shut.

Tests: additions-only sections 21 (printable-boundary fail-close for C1 +
noncharacters; NEL/U+00A0/astral still resolve) and 22 (internal
indicators resolve; leading indicator / ': ' inline map / trailing ':'
fail closed). Verified vs real PyYAML 6.0.3 both directions (present as
oracle, absent via PYTHONPATH shadow): wide 602 cases 0 fail-open/0
present-mismatch, dense 712 cases 0 fail-open/0 over-reject/0 mismatch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Hermes Agent
2026-07-22 01:19:27 -05:00
parent acf7955f87
commit 0905cdc292
2 changed files with 161 additions and 22 deletions

View File

@@ -810,19 +810,27 @@ def _constructible(text):
# PyYAML's Reader.check_printable scans the ENTIRE raw input stream (not just
# scalar contents, and NOT scoped by quoting) for bytes outside its printable
# set and raises ReaderError -- a whole-document reject -- the instant one is
# found, no matter where it sits (an unrelated field, a comment, inside or
# outside quotes). Empirically verified against real PyYAML 6.0.3: EVERY C0
# control byte below plus DEL (0x7F) rejects in plain scalars, inside
# double-quoted scalars, and inside single-quoted scalars alike; only
# TAB(0x09), LF(0x0A), and CR(0x0D) are accepted among the C0 range (LF/CR are
# line separators, handled elsewhere; TAB has its own narrower, position-aware
# guard above since PyYAML accepts it in some contexts). This set therefore
# excludes 0x09/0x0A/0x0D and matches the forbidden C0 set {0x00-0x08, 0x0B,
# 0x0C, 0x0E-0x1F} plus DEL (0x7F).
# scalar contents, and NOT scoped by quoting) for code points outside its
# printable set and raises ReaderError -- a whole-document reject -- the instant
# one is found, no matter where it sits (an unrelated field, a comment, inside or
# outside quotes). To guarantee parity WITHOUT a hand-rolled subset (which missed
# the C1 block and BMP noncharacters), this is PyYAML 6.0.3's EXACT Reader
# printable definition, negated verbatim:
# Reader.NON_PRINTABLE =
# re.compile('[^\x09\x0A\x0D\x20-\x7E\x85\xA0-퟿-<2D>'
# '\U00010000-\U0010FFFF]')
# i.e. PRINTABLE = {TAB(0x09), LF(0x0A), CR(0x0D), 0x20-0x7E, NEL(0x85),
# 0xA0-0xD7FF, 0xE000-0xFFFD, 0x10000-0x10FFFF}; EVERYTHING else (all other C0
# controls, DEL 0x7F, the entire C1 block 0x80-0x84/0x86-0x9F, the surrogate
# range 0xD800-0xDFFF, and the BMP noncharacters 0xFFFE/0xFFFF) is non-printable
# -> whole-document fail closed. Verified empirically against real PyYAML 6.0.3
# in both directions: the C1 bytes and 0xFFFE/0xFFFF reject; NEL(0x85), 0xA0, and
# astral code points (e.g. U+1F600) are accepted and resolve the token. The prior
# C0/DEL/tab behavior is a strict subset of this. (A surrogate code point cannot
# occur in valid UTF-8, so it also trips the UTF-8 decode on read and fails
# closed; it is included here for exact definitional parity.)
_FORBIDDEN_CONTROL = re.compile(
"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]"
"[^\x09\x0a\x0d\x20-\x7e\x85\xa0-퟿-<2D>\U00010000-\U0010ffff]"
)
# Sentinel: "outside the supported subset -> fail closed". Distinct from a
@@ -903,8 +911,21 @@ def _scalar(raw):
# remain valid plain scalars and fall through. Fail closed on the bare
# indicator so the fallback never emits a token PyYAML would refuse.
return _FAIL
if any(ch in _FLOW for ch in value):
return _FAIL
# NO blanket internal-indicator reject here. In BLOCK context (where a tea
# config value always sits) PyYAML treats ',[]{}' as ordinary plain-scalar
# content -- they are flow indicators ONLY inside a flow collection -- and
# treats '!&*|>#%@`' plus quotes as significant ONLY at the FIRST non-space
# character of a node (a leading one starts a tag/anchor/alias/block-scalar/
# comment/directive/reserved/quote), which the leading-char guard above
# (value[0] ...) already rejects. Verified empirically against real PyYAML
# 6.0.3: an INTERNAL '!' ',' '[' ']' '{' '}' '&' '*' '#'(not space-preceded)
# ':'(not space-followed) all resolve as a plain-string token. The only
# internal positions PyYAML treats as significant in block context are ' #'
# (whitespace-preceded '#' -> comment; stripped by the loop above) and ': '
# or a trailing ':' (-> mapping; rejected by the ": "/endswith(":") guard
# below). A blanket any(ch in _FLOW ...) scan over-rejected those internal
# indicators, dropping a token PyYAML emits verbatim; removing it restores
# parity while the position-specific guards keep the fail-open direction shut.
if "\t" in value:
# A tab anywhere in a plain scalar (leading, trailing, or embedded) is a
# PyYAML scanner error on the whole document -- it accepts tabs ONLY
@@ -1182,14 +1203,18 @@ def _token_via_lines():
raw_text = handle.read()
# Whole-document, position-independent reject: PyYAML's Reader rejects the
# ENTIRE document (ReaderError) if a forbidden C0/DEL control byte appears
# ANYWHERE in the raw stream -- in a plain scalar, inside single- or
# double-quoted scalars, in a comment, or in a field this recognizer never
# even looks at. Checking the raw text (before splitlines(), which itself
# splits on some of these same control bytes -- e.g. \x0b, \x0c, \x1c-\x1e
# -- and would otherwise obscure them) mirrors that exactly: fail closed
# for the whole document rather than risk emitting a token PyYAML would
# have refused.
# ENTIRE document (ReaderError) if a NON-printable code point (per its exact
# printable definition -- see _FORBIDDEN_CONTROL: C0 controls, DEL, the C1
# block, surrogates, and 0xFFFE/0xFFFF) appears ANYWHERE in the raw stream --
# in a plain scalar, inside single- or double-quoted scalars, in a comment,
# or in a field this recognizer never even looks at. Checking the raw text
# (before splitlines(), which itself splits on some of these same code points
# -- e.g. \x0b, \x0c, \x1c-\x1e, and NEL \x85 -- and would otherwise obscure
# them) mirrors that exactly: fail closed for the whole document rather than
# risk emitting a token PyYAML would have refused. (NEL 0x85 is PRINTABLE to
# PyYAML's Reader, so it is NOT in _FORBIDDEN_CONTROL; where PyYAML treats it
# as a line break the fallback's splitlines() agrees, and where PyYAML then
# errors the recognizer's structural guards fail closed identically.)
if _FORBIDDEN_CONTROL.search(raw_text):
return None