fix(git): match PyYAML Reader.NON_PRINTABLE and block-context plain rules in tea token fallback (#865)
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
Round-10 auditor blockers in the PyYAML-absent fallback of
get_gitea_token_for_login (detect-platform.sh):
Blocker 1 (fail-open): the round-9 control-char guard used a hand-rolled
C0/DEL subset that missed code points PyYAML's Reader also rejects -- the
C1 block (U+0080-0084, U+0086-009F), the surrogate range, and the BMP
noncharacters U+FFFE/U+FFFF -- so the fallback still emitted a token from
documents PyYAML rejects whole. _FORBIDDEN_CONTROL now uses PyYAML 6.0.3's
EXACT Reader.NON_PRINTABLE character class (negated), verified 0-mismatch
against real PyYAML across the full BMP + astral range.
Blocker 2 (over-reject): the recognizer blanket-rejected any plain scalar
containing a flow indicator, but in BLOCK context (where a tea config
value always sits) PyYAML treats ',[]{}' as ordinary content and treats
'!&*|>#%@`' and quotes as significant only at the first non-space char.
The blanket scan dropped tokens PyYAML emits verbatim (internal '!', ','
'[' ']' '{' '}' '#'-not-space-preceded, ':'-not-space-followed). Removed
it; the leading-char guard and the ' #' / ': ' / trailing-':' guards keep
the fail-open direction shut.
Tests: additions-only sections 21 (printable-boundary fail-close for C1 +
noncharacters; NEL/U+00A0/astral still resolve) and 22 (internal
indicators resolve; leading indicator / ': ' inline map / trailing ':'
fail closed). Verified vs real PyYAML 6.0.3 both directions (present as
oracle, absent via PYTHONPATH shadow): wide 602 cases 0 fail-open/0
present-mismatch, dense 712 cases 0 fail-open/0 over-reject/0 mismatch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -810,19 +810,27 @@ def _constructible(text):
|
||||
|
||||
|
||||
# PyYAML's Reader.check_printable scans the ENTIRE raw input stream (not just
|
||||
# scalar contents, and NOT scoped by quoting) for bytes outside its printable
|
||||
# set and raises ReaderError -- a whole-document reject -- the instant one is
|
||||
# found, no matter where it sits (an unrelated field, a comment, inside or
|
||||
# outside quotes). Empirically verified against real PyYAML 6.0.3: EVERY C0
|
||||
# control byte below plus DEL (0x7F) rejects in plain scalars, inside
|
||||
# double-quoted scalars, and inside single-quoted scalars alike; only
|
||||
# TAB(0x09), LF(0x0A), and CR(0x0D) are accepted among the C0 range (LF/CR are
|
||||
# line separators, handled elsewhere; TAB has its own narrower, position-aware
|
||||
# guard above since PyYAML accepts it in some contexts). This set therefore
|
||||
# excludes 0x09/0x0A/0x0D and matches the forbidden C0 set {0x00-0x08, 0x0B,
|
||||
# 0x0C, 0x0E-0x1F} plus DEL (0x7F).
|
||||
# scalar contents, and NOT scoped by quoting) for code points outside its
|
||||
# printable set and raises ReaderError -- a whole-document reject -- the instant
|
||||
# one is found, no matter where it sits (an unrelated field, a comment, inside or
|
||||
# outside quotes). To guarantee parity WITHOUT a hand-rolled subset (which missed
|
||||
# the C1 block and BMP noncharacters), this is PyYAML 6.0.3's EXACT Reader
|
||||
# printable definition, negated verbatim:
|
||||
# Reader.NON_PRINTABLE =
|
||||
# re.compile('[^\x09\x0A\x0D\x20-\x7E\x85\xA0--<2D>'
|
||||
# '\U00010000-\U0010FFFF]')
|
||||
# i.e. PRINTABLE = {TAB(0x09), LF(0x0A), CR(0x0D), 0x20-0x7E, NEL(0x85),
|
||||
# 0xA0-0xD7FF, 0xE000-0xFFFD, 0x10000-0x10FFFF}; EVERYTHING else (all other C0
|
||||
# controls, DEL 0x7F, the entire C1 block 0x80-0x84/0x86-0x9F, the surrogate
|
||||
# range 0xD800-0xDFFF, and the BMP noncharacters 0xFFFE/0xFFFF) is non-printable
|
||||
# -> whole-document fail closed. Verified empirically against real PyYAML 6.0.3
|
||||
# in both directions: the C1 bytes and 0xFFFE/0xFFFF reject; NEL(0x85), 0xA0, and
|
||||
# astral code points (e.g. U+1F600) are accepted and resolve the token. The prior
|
||||
# C0/DEL/tab behavior is a strict subset of this. (A surrogate code point cannot
|
||||
# occur in valid UTF-8, so it also trips the UTF-8 decode on read and fails
|
||||
# closed; it is included here for exact definitional parity.)
|
||||
_FORBIDDEN_CONTROL = re.compile(
|
||||
"[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]"
|
||||
"[^\x09\x0a\x0d\x20-\x7e\x85\xa0--<2D>\U00010000-\U0010ffff]"
|
||||
)
|
||||
|
||||
# Sentinel: "outside the supported subset -> fail closed". Distinct from a
|
||||
@@ -903,8 +911,21 @@ def _scalar(raw):
|
||||
# remain valid plain scalars and fall through. Fail closed on the bare
|
||||
# indicator so the fallback never emits a token PyYAML would refuse.
|
||||
return _FAIL
|
||||
if any(ch in _FLOW for ch in value):
|
||||
return _FAIL
|
||||
# NO blanket internal-indicator reject here. In BLOCK context (where a tea
|
||||
# config value always sits) PyYAML treats ',[]{}' as ordinary plain-scalar
|
||||
# content -- they are flow indicators ONLY inside a flow collection -- and
|
||||
# treats '!&*|>#%@`' plus quotes as significant ONLY at the FIRST non-space
|
||||
# character of a node (a leading one starts a tag/anchor/alias/block-scalar/
|
||||
# comment/directive/reserved/quote), which the leading-char guard above
|
||||
# (value[0] ...) already rejects. Verified empirically against real PyYAML
|
||||
# 6.0.3: an INTERNAL '!' ',' '[' ']' '{' '}' '&' '*' '#'(not space-preceded)
|
||||
# ':'(not space-followed) all resolve as a plain-string token. The only
|
||||
# internal positions PyYAML treats as significant in block context are ' #'
|
||||
# (whitespace-preceded '#' -> comment; stripped by the loop above) and ': '
|
||||
# or a trailing ':' (-> mapping; rejected by the ": "/endswith(":") guard
|
||||
# below). A blanket any(ch in _FLOW ...) scan over-rejected those internal
|
||||
# indicators, dropping a token PyYAML emits verbatim; removing it restores
|
||||
# parity while the position-specific guards keep the fail-open direction shut.
|
||||
if "\t" in value:
|
||||
# A tab anywhere in a plain scalar (leading, trailing, or embedded) is a
|
||||
# PyYAML scanner error on the whole document -- it accepts tabs ONLY
|
||||
@@ -1182,14 +1203,18 @@ def _token_via_lines():
|
||||
raw_text = handle.read()
|
||||
|
||||
# Whole-document, position-independent reject: PyYAML's Reader rejects the
|
||||
# ENTIRE document (ReaderError) if a forbidden C0/DEL control byte appears
|
||||
# ANYWHERE in the raw stream -- in a plain scalar, inside single- or
|
||||
# double-quoted scalars, in a comment, or in a field this recognizer never
|
||||
# even looks at. Checking the raw text (before splitlines(), which itself
|
||||
# splits on some of these same control bytes -- e.g. \x0b, \x0c, \x1c-\x1e
|
||||
# -- and would otherwise obscure them) mirrors that exactly: fail closed
|
||||
# for the whole document rather than risk emitting a token PyYAML would
|
||||
# have refused.
|
||||
# ENTIRE document (ReaderError) if a NON-printable code point (per its exact
|
||||
# printable definition -- see _FORBIDDEN_CONTROL: C0 controls, DEL, the C1
|
||||
# block, surrogates, and 0xFFFE/0xFFFF) appears ANYWHERE in the raw stream --
|
||||
# in a plain scalar, inside single- or double-quoted scalars, in a comment,
|
||||
# or in a field this recognizer never even looks at. Checking the raw text
|
||||
# (before splitlines(), which itself splits on some of these same code points
|
||||
# -- e.g. \x0b, \x0c, \x1c-\x1e, and NEL \x85 -- and would otherwise obscure
|
||||
# them) mirrors that exactly: fail closed for the whole document rather than
|
||||
# risk emitting a token PyYAML would have refused. (NEL 0x85 is PRINTABLE to
|
||||
# PyYAML's Reader, so it is NOT in _FORBIDDEN_CONTROL; where PyYAML treats it
|
||||
# as a line break the fallback's splitlines() agrees, and where PyYAML then
|
||||
# errors the recognizer's structural guards fail closed identically.)
|
||||
if _FORBIDDEN_CONTROL.search(raw_text):
|
||||
return None
|
||||
|
||||
|
||||
Reference in New Issue
Block a user