Hermes Agent
b1254f52f3
ci/woodpecker/pr/ci Pipeline was canceled
Round-five review found command substitution executing inside the very quoted spans the skeleton was discarding as prose: echo "$(curl -d@b .../issues/1/comments)" msg="$(curl -d@b .../issues/1/comments)" The unquoted and process-substitution forms already blocked, so the same call was refused or allowed depending on a quote character. That makes it a classification defect rather than another spelling, and it is the nineteenth write to reach execution through this file by the same route: the client was ABSENT from the skeleton, so the guard allowed. The reviewer's judgement, which I asked for and accept: this is fitting to the test set. Answering "code or data" from shell text with sed and awk is not a hard problem, it is the wrong problem. It was also not portable. CI has been red at `sanitization` since round four, and the log says why: under the image's busybox awk the octal escape in the quote-stripping regex does not bite, every quoted span survives into the skeleton, and the guard began refusing ordinary prose. Five allow-direction fixtures failed in CI that pass under GNU awk. A control that reverses its verdict with the awk on the host is not a control. So the client detection is gone — the skeleton, the invoker list, the prefix list, the option-value skipping, all of it. What remains asks two questions of the text: is this a write, and does it name an endpoint a wrapper owns. It cannot fail open by hiding the caller because it never looks for one, and it now catches clients it was never taught: `python -c ... requests.post(...)` and `wget --post-data` are both fixtures. The cost is stated in the file and pinned in both directions: QUOTING one of these calls on a Bash command line is refused as well. Ten fixtures that used to assert "discussing a call is not making one" now assert the opposite, and the boundary that stops this becoming block-everything is asserted just as hard — a quoted READ, an endpoint named without a body flag, a quoted write to an UNWRAPPED endpoint, and the wrapper's own body flag all still pass. The 18-command ordinary-work sweep blocks none. The rule an agent can hold without a parser: do not put a raw write to a wrapped forge endpoint on a Bash command line, not even inside quotes. Write the example with a file-writing tool. 60/60 fixtures, verified inside the CI image (busybox) as well as locally. Gates: sanitization, resident budget, test enumeration, tools-index (self-test 4/4, git suite 100%), issue-close, prettier.
169 lines
16 KiB
Bash
Executable File
169 lines
16 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# test-wrapper-guard.sh — hermetic behavioural regression for wrapper-guard.sh.
|
|
#
|
|
# Resolves no credentials, touches no network, and creates no repository: the
|
|
# guard reads a hook payload on stdin and answers with an exit code, so the whole
|
|
# contract is testable from fixtures.
|
|
#
|
|
# The fixtures are written to a temp file rather than passed inline, and this is
|
|
# not stylistic. The guard inspects the literal text of the Bash command it is
|
|
# handed. A test that embeds `git clone ... $HOME` inside its own command line
|
|
# trips the guard on the harness instead of on the fixture — which is exactly
|
|
# what happened the first time this was checked by hand. Substring matching over
|
|
# whole command text is the guard's deliberate fail-closed posture; a test that
|
|
# does not account for it silently measures the wrong thing.
|
|
#
|
|
# Exit: 0 = every fixture behaved as specified · 1 = at least one did not
|
|
|
|
set -uo pipefail
|
|
|
|
HERE="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
|
GUARD="${1:-$HERE/wrapper-guard.sh}"
|
|
[ -x "$GUARD" ] || { printf 'test-wrapper-guard: not executable: %s\n' "$GUARD" >&2; exit 2; }
|
|
|
|
TMP="$(mktemp -d)"; trap 'rm -rf "$TMP"' EXIT
|
|
FIXTURES="$TMP/fixtures.tsv"
|
|
|
|
# Each line: <expected-exit> TAB <hook payload> TAB <what it proves>
|
|
# 0 = allowed, 2 = blocked.
|
|
{
|
|
printf '2\t{"tool_input":{"command":"git clone https://example.invalid/x ~/wt"}}\tcheckout into $HOME is refused\n'
|
|
printf '2\t{"tool_input":{"command":"git worktree add ~/wt topic"}}\tworktree into $HOME is refused\n'
|
|
printf '0\t{"tool_input":{"command":"git clone https://example.invalid/x /src/wt"}}\tcheckout onto a work filesystem is fine\n'
|
|
printf '0\t{"tool_input":{"command":"curl -s -X GET https://git.example.invalid/api/v1/repos/a/b/pulls/1"}}\treads are never blocked\n'
|
|
printf '2\t{"tool_input":{"command":"curl -X POST -d @b https://git.example.invalid/api/v1/repos/a/b/pulls/1/reviews"}}\treview write has a wrapper\n'
|
|
printf '2\t{"tool_input":{"command":"curl -X POST -d @b https://git.example.invalid/api/v1/repos/a/b/pulls/1/merge"}}\tmerge write has a wrapper\n'
|
|
printf '2\t{"tool_input":{"command":"curl -X POST -d @b https://api.github.com/repos/a/b/issues"}}\tGitHub host is covered too\n'
|
|
printf '0\t{"tool_input":{"command":"curl -X POST -d @b https://git.example.invalid/api/v1/repos/a/b/releases"}}\tan endpoint with no wrapper passes\n'
|
|
printf '2\t{"tool_input":{"command":"curl -X POST -d {\\"event\\":\\"APPROVE\\"} https://example.invalid/x"}}\tthe APPROVE token is caught anywhere\n'
|
|
printf '0\t{"tool_input":{"command":"ls -la /src"}}\tordinary commands are untouched\n'
|
|
printf '0\t{"tool_input":{"command":"MOSAIC_WRAPPER_OVERRIDE=1 curl -X POST -d @b https://git.example.invalid/api/v1/repos/a/b/pulls"}}\tbreak-glass works\n'
|
|
printf '0\t{"tool_input":{}}\tan empty payload does not block the session\n'
|
|
# --- bypasses an independent reviewer demonstrated against the first version.
|
|
# Each of these returned 0 (allowed) and each is a real write. They are pinned
|
|
# as fixtures rather than fixed-and-forgotten because the class is recurring:
|
|
# the guard reads text, so every spelling it does not know is a hole.
|
|
printf '2\t{"tool_input":{"command":"curl -d@b https://git.example.invalid/api/v1/repos/a/b/pulls/1/reviews"}}\t-d@body with no space is still a body\n'
|
|
printf '2\t{"tool_input":{"command":"curl --request=POST -d@b https://git.example.invalid/api/v1/repos/a/b/pulls/1/reviews"}}\t--request=POST equals-form is still a method\n'
|
|
printf '2\t{"tool_input":{"command":"p=/api/v1/repo; q=s/a/b/pulls/1/reviews; curl -d@b https://git.example.invalid${p}${q}"}}\ta path split across variables is still that path\n'
|
|
printf '2\t{"tool_input":{"command":"curl --data-binary @b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments"}}\t--data-binary is a body\n'
|
|
printf '2\t{"tool_input":{"command":"curl -F f=@b https://git.example.invalid/api/v1/repos/a/b/issues"}}\t-F multipart is a body\n'
|
|
# Reads must survive every one of those broadenings, or the guard gets disabled.
|
|
printf '0\t{"tool_input":{"command":"curl -s https://git.example.invalid/api/v1/repos/a/b/pulls/1/reviews"}}\tno body and no verb is a read\n'
|
|
printf '0\t{"tool_input":{"command":"grep -rn /pulls/ src/ | head -20"}}\ta path fragment in a grep is not an API call\n'
|
|
printf '0\t{"tool_input":{"command":"curl -X POST -d @b https://registry.example.invalid/v2/x/manifests/latest"}}\tan unwrapped API is not this guard'"'"'s business\n'
|
|
# --- round two of the same review. Splitting the ENDPOINT TOKEN defeats any
|
|
# amount of fragment matching, because the endpoint does not exist until the
|
|
# shell expands it. The guard now refuses to clear a write whose URL it cannot
|
|
# read, rather than pretending it read one.
|
|
printf '2\t{"tool_input":{"command":"a=/api/v1/repos/a/b/iss; b=ues/1/comments; curl -d@body https://git.example.invalid${a}${b}"}}\tan endpoint token split across variables is unreadable, not absent\n'
|
|
printf '2\t{"tool_input":{"command":"a=/api/v1/repos/a/b/pu; b=lls/1/reviews; curl -d@body https://git.example.invalid${a}${b}"}}\tsame split, review endpoint\n'
|
|
printf '0\t{"tool_input":{"command":"curl -X POST -d @payload https://hooks.example.invalid/services/${WEBHOOK_ID}"}}\tan opaque URL that is not forge-shaped stays allowed\n'
|
|
# --- round six changed the contract in this direction, and these fixtures are
|
|
# where it shows. They used to assert that discussing a call is not making one.
|
|
# Five rounds proved there is no textual way to tell a quoted example from a
|
|
# quoted command, so the guard stopped trying: it judges the payload, and a
|
|
# payload inside quotes is still a payload. Quoting one of these on a Bash
|
|
# command line is now refused, and the way to write the example is a
|
|
# file-writing tool. This is the deliberate cost of the mechanism change.
|
|
printf '2\t{"tool_input":{"command":"grep -R \\"curl -d https://git.example.invalid/api/v1/repos/a/b/issues\\" docs/"}}\tquoting a wrapped write is refused even in a grep\n'
|
|
printf '2\t{"tool_input":{"command":"echo \\"curl -d https://git.example.invalid/api/v1/repos/a/b/pulls\\" > note.txt"}}\t...and when written into a file\n'
|
|
printf '2\t{"tool_input":{"command":"python3 -c '"'"'print(\\"curl -d https://git.example.invalid/api/v1/repos/a/b/issues\\")'"'"'"}}\t...and when printed from another language\n'
|
|
# The boundary that keeps this from being "block everything": what is refused
|
|
# is a WRITE to a WRAPPED endpoint. Mentioning either alone still passes, and
|
|
# these are asserted as hard as the blocks above.
|
|
printf '0\t{"tool_input":{"command":"grep -R \\"curl -s https://git.example.invalid/api/v1/repos/a/b/issues/1/comments\\" docs/"}}\tquoting a READ example is untouched\n'
|
|
printf '0\t{"tool_input":{"command":"echo \\"the wrapped endpoint is https://git.example.invalid/api/v1/repos/a/b/issues/1/comments\\" >> notes.md"}}\tnaming the endpoint without a body flag is untouched\n'
|
|
printf '0\t{"tool_input":{"command":"grep -R \\"curl -d@b https://git.example.invalid/api/v1/repos/a/b/releases\\" docs/"}}\tquoting a write to an UNWRAPPED endpoint is untouched\n'
|
|
printf '0\t{"tool_input":{"command":"issue-comment.sh --repo a/b --issue 1 --body @msg.md"}}\tthe wrapper itself carries a body flag and must never trip its own guard\n'
|
|
# Command position must still catch the real thing behind operators and env.
|
|
printf '2\t{"tool_input":{"command":"cd /tmp && GITEA_TOKEN=$T curl -d@b https://git.example.invalid/api/v1/repos/a/b/pulls/1/merge"}}\ta real call behind && and an assignment is still a call\n'
|
|
# --- the case the AUTHOR hit twice while chasing the above: sending a message
|
|
# that QUOTED one of these fixtures. Under the old contract that was a defect
|
|
# to be parsed away; under this one it is the documented cost, and the message
|
|
# gets composed with a file-writing tool instead.
|
|
printf '2\t{"tool_input":{"command":"send.sh -m \\"repro was: cd /tmp && curl -d@b https://git.example.invalid/api/v1/repos/a/b/pulls/1/merge\\""}}\tquoting the repro in a message is refused too\n'
|
|
printf '2\t{"tool_input":{"command":"cat >> notes.md <<EOF\\nwe ran: curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues\\nEOF"}}\ta heredoc body carrying the payload is refused with it\n'
|
|
# ...but quotes stop being data the moment something executes them.
|
|
printf '2\t{"tool_input":{"command":"bash -c \\"curl -d@b https://git.example.invalid/api/v1/repos/a/b/pulls/1/merge\\""}}\tbash -c makes the quoted text code again\n'
|
|
# --- round three. Each of these four is a real write that a bare-name match
|
|
# for the client could not see, because an ordinary word sat in front of it.
|
|
# They are kept as fixtures after the mechanism change even though the guard no
|
|
# longer looks for a client at all: they are the evidence for WHY it stopped,
|
|
# and a future re-narrowing that reintroduced position would fail here first.
|
|
printf '2\t{"tool_input":{"command":"env GITEA_TOKEN=$T curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments"}}\tenv VAR=... in front of the client is still the client\n'
|
|
printf '2\t{"tool_input":{"command":"command curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments"}}\tcommand in front of the client is still the client\n'
|
|
printf '2\t{"tool_input":{"command":"timeout 10 curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments"}}\ttimeout N in front of the client is still the client\n'
|
|
printf '2\t{"tool_input":{"command":"/usr/bin/curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments"}}\tan absolute path to the client is still the client\n'
|
|
printf '2\t{"tool_input":{"command":"echo timeout 10 curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments >> notes.md"}}\tnaming the call after echo carries the payload, so it is refused\n'
|
|
# A shell standing between quoted data and execution makes that data code,
|
|
# and the pipe is the form agents actually use. Filing it as data allowed the
|
|
# call to vanish from the skeleton while still running.
|
|
printf '2\t{"tool_input":{"command":"printf '"'"'%%s\\\\n'"'"' '"'"'curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments'"'"' | sh"}}\tquoted code piped to a shell is code\n'
|
|
printf '2\t{"tool_input":{"command":"cat <<EOF | sh\\ncurl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments\\nEOF"}}\ta heredoc piped to a shell is code\n'
|
|
printf '2\t{"tool_input":{"command":"sh -s <<EOF\\ncurl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments\\nEOF"}}\tsh -s reads its script from the heredoc\n'
|
|
# ...and the questions that used to follow — is the pipe target a shell, does a
|
|
# shell on one line execute a string on another — no longer have to be answered
|
|
# at all. Both of these carry the payload, both are refused, and neither
|
|
# outcome depends on parsing what the pipe or the other line does.
|
|
printf '2\t{"tool_input":{"command":"grep -R \\"curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments\\" docs/ | wc -l"}}\tpiping the payload to wc is refused without asking what wc is\n'
|
|
printf '2\t{"tool_input":{"command":"docker run --rm alpine sh -c '"'"'echo hi'"'"'\\necho \\"example: curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments\\" >> notes.md"}}\tan unrelated shell on another line no longer changes the answer either way\n'
|
|
# --- round four. The guard was still reading the command as typed rather than
|
|
# as the shell will run it: a backslash before a newline is removed before
|
|
# anything else happens, so the endpoint token can be split across the join.
|
|
printf '2\t{"tool_input":{"command":"curl -d@b https://git.example.invalid/api/v1/repos/a/b/iss\\\\\\nues/1/comments"}}\ta line continuation inside the endpoint token is still that endpoint\n'
|
|
printf '2\t{"tool_input":{"command":"curl -d@b https://git.example.invalid/api/v1/repos/a/b/pu\\\\\\nlls/1/reviews"}}\tsame join, review endpoint\n'
|
|
printf '2\t{"tool_input":{"command":"cat >> notes.md <<EOF\\nwe ran: curl -d@b https://git.example.invalid/api/v1/repos/a/b/iss\\\\\\nues/1/comments\\nEOF"}}\tthe join still runs first, and the joined payload is refused in a document too\n'
|
|
# Transparent prefixes take option VALUES, and the value was a word the list
|
|
# did not know — so the client went missing again behind an ordinary `sudo -u`.
|
|
printf '2\t{"tool_input":{"command":"sudo -u root curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments"}}\tan option value after a prefix does not hide the client\n'
|
|
printf '2\t{"tool_input":{"command":"timeout --signal TERM 10 curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments"}}\tan option pair plus a duration does not hide the client\n'
|
|
printf '2\t{"tool_input":{"command":"xargs echo curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments"}}\tthe payload behind xargs echo is refused rather than adjudicated\n'
|
|
printf '0\t{"tool_input":{"command":"sudo apt-get install curl"}}\tinstalling the client is not calling it\n'
|
|
# Execution through another command needed its own case under the old design.
|
|
printf '2\t{"tool_input":{"command":"find . -maxdepth 0 -exec curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments ;"}}\tfind -exec runs the client\n'
|
|
# --- round five, and the finding that ended the parser. Command substitution
|
|
# inside a double-quoted span EXECUTES, while the skeleton was discarding that
|
|
# span as inert prose. The unquoted and process-substitution forms already
|
|
# blocked, which is what made it a classification defect rather than a spelling
|
|
# one: the same call was refused or allowed depending on a quote character.
|
|
printf '2\t{"tool_input":{"command":"echo \\"$(curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments)\\""}}\tcommand substitution inside double quotes executes\n'
|
|
printf '2\t{"tool_input":{"command":"echo \\"`curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments`\\""}}\tso does the backtick form\n'
|
|
printf '2\t{"tool_input":{"command":"msg=\\"$(curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments)\\""}}\tand an assignment RHS is not data either\n'
|
|
printf '2\t{"tool_input":{"command":"echo $(curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments)"}}\tthe unquoted form, which blocked before and must keep blocking\n'
|
|
printf '2\t{"tool_input":{"command":"cat <(curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments)"}}\tprocess substitution, same\n'
|
|
printf '2\t{"tool_input":{"command":"bash --command \\"curl -d@b https://git.example.invalid/api/v1/repos/a/b/issues/1/comments\\""}}\tthe long-option spelling of bash -c needs no entry in any list now\n'
|
|
# A client the guard was never taught is the point of dropping client
|
|
# detection: neither of these names curl at all.
|
|
printf '2\t{"tool_input":{"command":"python3 -c '"'"'import requests; requests.post(\\"https://git.example.invalid/api/v1/repos/a/b/issues/1/comments\\", json={})'"'"'"}}\ta library call is a write with no flag and no curl\n'
|
|
printf '2\t{"tool_input":{"command":"wget --post-data=x https://git.example.invalid/api/v1/repos/a/b/issues/1/comments"}}\twget spells its body differently and is still a write\n'
|
|
} > "$FIXTURES"
|
|
|
|
fail=0 n=0
|
|
while IFS=$'\t' read -r want payload why; do
|
|
[ -n "${want:-}" ] || continue
|
|
n=$((n + 1))
|
|
printf '%s' "$payload" | "$GUARD" >/dev/null 2>&1
|
|
got=$?
|
|
if [ "$got" = "$want" ]; then
|
|
printf 'ok %s\n' "$why"
|
|
else
|
|
printf 'FAIL %s (want exit %s, got %s)\n' "$why" "$want" "$got"
|
|
fail=1
|
|
fi
|
|
done < "$FIXTURES"
|
|
|
|
printf '\n'
|
|
if [ "$fail" -eq 0 ]; then
|
|
printf 'wrapper-guard: %d/%d fixtures behaved as specified.\n' "$n" "$n"
|
|
else
|
|
cat <<'EOF'
|
|
wrapper-guard drifted from its contract.
|
|
|
|
A guard that blocks too much gets routed around, and a guard that blocks too
|
|
little is decoration. Both directions are failures here, which is why the
|
|
allowed cases are asserted as hard as the blocked ones.
|
|
EOF
|
|
fi
|
|
exit "$fail"
|