Hermes Agent
8b7ac5b51e
ci/woodpecker/pr/ci Pipeline was canceled
An independent reviewer broke all three new controls before they shipped. Every finding is reproduced as a fixture or a repro, because the class is recurring rather than incidental: each hole was a case where the answer was "allow" because something was ABSENT rather than because it was CHECKED. 1. wrapper-guard read only the spellings it knew. `curl -d@body` (no space), `--request=POST` (equals form), and a URL path assembled from shell variables each carried a real provider write straight through. Write detection now covers every body and method form curl accepts, and the endpoint match no longer anchors on a literal host path that a variable can dissolve. 2. wrapper-guard blocked only when the wrapper FILE existed. A host with a broken or partial install therefore permitted exactly the raw writes the guard exists to stop. Blocking is now on the endpoint; a missing wrapper changes the remedy text, not the verdict — a broken install is not permission to bypass gate 7. 3. mosaic-worktree read a worktree's safety from two questions, and a clean, fully-pushed tree holding a gitignored `local.secret` answered both with zero. `git worktree remove` then deleted the one copy in existence. A file is gitignored precisely so nothing else holds it, so ignored-but-not- disposable files are now a third evidence question. Build junk (node_modules, .venv, dist, caches, *.pyc) stays disposable, so the common case still reads SAFE. 4. check-tools-index counted a documented tool as discoverable at mode 0644. Every caller tests `[ -x ]`, so a non-executable tool is a missing tool; it now fails the gate with its own message. Local gates green: sanitization, resident budget, test enumeration, tools-index (4/4 self-test, 100% on the enforced git suite), and wrapper-guard 20/20.
294 lines
12 KiB
Bash
Executable File
294 lines
12 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# check-tools-index.sh — assert every shipped tool is discoverable from the
|
|
# resident documentation an agent actually has in context.
|
|
#
|
|
# WHY THIS GATE EXISTS
|
|
# --------------------
|
|
# The framework ships 26 git wrappers. Before this gate, 20 of them were named
|
|
# in neither `defaults/TOOLS.md` nor `guides/TOOLS-REFERENCE.md`. One of the
|
|
# undocumented ones was `pr-review.sh` — the wrapper that carries the
|
|
# APPROVED/APPROVE provider-dialect split.
|
|
#
|
|
# The observable consequence, on a live fleet host: an agent needing to place a
|
|
# review verdict reached for raw `curl`, sent GitHub's `APPROVE` to a Gitea
|
|
# host, and got HTTP 200 with the review silently filed PENDING — three times,
|
|
# because nothing about the failure pointed at the wrapper that already handled
|
|
# it correctly. The agent was not ignoring Constitution gate 7. It was obeying
|
|
# an index that said the tool did not exist.
|
|
#
|
|
# That is not a discipline problem and no amount of prose fixes it. A wrapper
|
|
# that is not in the resident index is, from inside a session, indistinguishable
|
|
# from a wrapper that was never written. So the invariant is mechanical:
|
|
#
|
|
# shipping a tool and documenting it are the same commit, or CI fails.
|
|
#
|
|
# WHAT IT CHECKS
|
|
# --------------
|
|
# forward every non-excluded tool in an ENFORCED suite is named in at least
|
|
# one index document (missing tool -> undiscoverable -> FAIL)
|
|
# reverse every `<name>.sh` an index document attributes to an enforced
|
|
# suite exists on disk (stale reference -> agent runs a ghost -> FAIL)
|
|
#
|
|
# Suites outside the enforced set are reported with a coverage percentage but do
|
|
# not fail the build, so the ratchet can be tightened one suite per PR instead of
|
|
# landing as one unreviewable sweep. `--strict` fails on those too.
|
|
#
|
|
# WHY THE ENFORCED LIST LIVES HERE AND NOT IN A MARKER INSIDE THE DOC
|
|
# -------------------------------------------------------------------
|
|
# `TOOLS.md` is operator-owned (see framework-manifest.txt). A marker inside it
|
|
# would let an operator silence this gate by editing their own copy — the gate
|
|
# would then be strongest exactly where it is least needed and absent where it
|
|
# is needed most. The list is framework-owned and changes only through a
|
|
# reviewed PR.
|
|
#
|
|
# Usage:
|
|
# check-tools-index.sh [--tools-dir DIR] [--doc FILE]... [--strict] [--self-test]
|
|
#
|
|
# Exit: 0 = every enforced suite fully discoverable · 1 = drift · 2 = bad usage
|
|
|
|
set -euo pipefail
|
|
|
|
# Suites whose coverage is a HARD requirement. Add a suite here only together
|
|
# with the doc changes that make it pass.
|
|
#
|
|
# `git` is first because it is the suite Constitution gates 6-8 make mandatory:
|
|
# an undiscoverable git wrapper converts a hard gate into a coin flip.
|
|
ENFORCED_SUITES=(git)
|
|
|
|
# Files that are not agent-callable tools and must not be required in an index.
|
|
EXCLUDE_GLOBS=(
|
|
'test-*' # hermetic regression scripts, invoked by CI not by agents
|
|
'_*' # private helpers (_lib, _scripts internals)
|
|
'*.bak' # editor/installer debris
|
|
'*.pre-*' # pre-change backups (e.g. ci-queue-wait.sh.pre-404fix-bak)
|
|
'README.md'
|
|
)
|
|
|
|
STRICT=0
|
|
SELF_TEST=0
|
|
TOOLS_DIR=""
|
|
DOCS=()
|
|
|
|
die() { printf 'check-tools-index: %s\n' "$*" >&2; exit 2; }
|
|
|
|
while [ $# -gt 0 ]; do
|
|
case "$1" in
|
|
--tools-dir) TOOLS_DIR="${2:-}"; shift 2 ;;
|
|
--doc) DOCS+=("${2:-}"); shift 2 ;;
|
|
--strict) STRICT=1; shift ;;
|
|
--self-test) SELF_TEST=1; shift ;;
|
|
-h|--help) sed -n '2,48p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
|
*) die "unknown argument: $1" ;;
|
|
esac
|
|
done
|
|
|
|
# ---- location resolution ---------------------------------------------------
|
|
# Runs from two places with different layouts, and must not silently check the
|
|
# wrong tree: a CI checkout (repo-relative) and an installed host ($MOSAIC_HOME).
|
|
resolve_locations() {
|
|
local here framework
|
|
here="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
|
# .../framework/tools/quality/scripts -> .../framework
|
|
framework="$(cd -- "$here/../../.." && pwd)"
|
|
|
|
if [ -z "$TOOLS_DIR" ]; then
|
|
if [ -d "$framework/tools" ]; then
|
|
TOOLS_DIR="$framework/tools"
|
|
else
|
|
TOOLS_DIR="${MOSAIC_HOME:-$HOME/.config/mosaic}/tools"
|
|
fi
|
|
fi
|
|
|
|
if [ ${#DOCS[@]} -eq 0 ]; then
|
|
# The two layouts are mutually exclusive on purpose. Unioning them would let
|
|
# a well-maintained operator TOOLS.md on the developer's own machine mask a
|
|
# gap in the shipped defaults — the check would pass locally and the defect
|
|
# would still install on every other host. Repo layout wins when present.
|
|
if [ -f "$framework/defaults/TOOLS.md" ]; then
|
|
DOCS+=("$framework/defaults/TOOLS.md")
|
|
[ -f "$framework/guides/TOOLS-REFERENCE.md" ] && DOCS+=("$framework/guides/TOOLS-REFERENCE.md")
|
|
else
|
|
local mosaic_home="${MOSAIC_HOME:-$HOME/.config/mosaic}"
|
|
[ -f "$mosaic_home/TOOLS.md" ] && DOCS+=("$mosaic_home/TOOLS.md")
|
|
[ -f "$mosaic_home/guides/TOOLS-REFERENCE.md" ] && DOCS+=("$mosaic_home/guides/TOOLS-REFERENCE.md")
|
|
fi
|
|
fi
|
|
|
|
[ -d "$TOOLS_DIR" ] || die "tools dir not found: $TOOLS_DIR"
|
|
[ ${#DOCS[@]} -gt 0 ] || die "no index documents found (pass --doc FILE)"
|
|
}
|
|
|
|
is_excluded() {
|
|
local name="$1" glob
|
|
for glob in "${EXCLUDE_GLOBS[@]}"; do
|
|
# shellcheck disable=SC2254 # glob is intentionally a pattern
|
|
case "$name" in $glob) return 0 ;; esac
|
|
done
|
|
return 1
|
|
}
|
|
|
|
# A tool counts as documented when its basename appears anywhere in the corpus.
|
|
# Deliberately permissive about *form* (table cell, code fence, prose) and strict
|
|
# about *presence*: the gate's job is "an agent can find it", not house style.
|
|
documented() { grep -qF -- "$1" "$CORPUS"; }
|
|
|
|
# ---- the check -------------------------------------------------------------
|
|
run_check() {
|
|
local rc=0 suite dir tool base enforced
|
|
|
|
CORPUS="$(mktemp)"; trap 'rm -f "$CORPUS"' RETURN
|
|
cat "${DOCS[@]}" > "$CORPUS"
|
|
|
|
printf 'tools: %s\n' "$TOOLS_DIR"
|
|
for d in "${DOCS[@]}"; do printf 'index: %s\n' "$d"; done
|
|
printf '\n'
|
|
|
|
for dir in "$TOOLS_DIR"/*/; do
|
|
[ -d "$dir" ] || continue
|
|
suite="$(basename -- "$dir")"
|
|
case " ${ENFORCED_SUITES[*]} " in *" $suite "*) enforced=1 ;; *) enforced=0 ;; esac
|
|
[ "$STRICT" -eq 1 ] && enforced=1
|
|
case "$suite" in _*) continue ;; esac
|
|
|
|
local total=0 found=0
|
|
local -a suite_missing=() suite_noexec=()
|
|
for tool in "$dir"*.sh; do
|
|
[ -e "$tool" ] || continue
|
|
base="$(basename -- "$tool")"
|
|
is_excluded "$base" && continue
|
|
total=$((total + 1))
|
|
if documented "$base"; then
|
|
found=$((found + 1))
|
|
# Documented AND present is not enough. The index presents these as
|
|
# commands to run, and every caller — the wrapper guard included —
|
|
# decides "is this tool here?" with `[ -x ]`. A 0644 wrapper is
|
|
# documented, present, and dead: it reads as absent to every check that
|
|
# matters while scoring 100% here. That is a false green, which is worse
|
|
# than a red, so it fails rather than warns.
|
|
[ -x "$tool" ] || suite_noexec+=("$base")
|
|
else
|
|
suite_missing+=("$base")
|
|
fi
|
|
done
|
|
[ "$total" -eq 0 ] && continue
|
|
|
|
local pct=$(( found * 100 / total ))
|
|
if [ "$enforced" -eq 1 ] && [ ${#suite_noexec[@]} -gt 0 ]; then
|
|
printf 'FAIL %-12s %3d%% (%d/%d) documented but not executable: %s\n' \
|
|
"$suite" "$pct" "$found" "$total" "${suite_noexec[*]}"
|
|
rc=1
|
|
fi
|
|
if [ "$enforced" -eq 1 ] && [ ${#suite_missing[@]} -gt 0 ]; then
|
|
printf 'FAIL %-12s %3d%% (%d/%d) undocumented: %s\n' \
|
|
"$suite" "$pct" "$found" "$total" "${suite_missing[*]}"
|
|
rc=1
|
|
elif [ "$enforced" -eq 1 ] && [ ${#suite_noexec[@]} -eq 0 ]; then
|
|
printf 'ok %-12s %3d%% (%d/%d) [enforced]\n' "$suite" "$pct" "$found" "$total"
|
|
else
|
|
printf 'info %-12s %3d%% (%d/%d) not yet enforced\n' "$suite" "$pct" "$found" "$total"
|
|
fi
|
|
|
|
# Reverse: an index that names a tool this suite does not have sends agents
|
|
# after something that cannot run. Only checked for enforced suites, where
|
|
# the naming is unambiguous enough to attribute.
|
|
if [ "$enforced" -eq 1 ]; then
|
|
local -a stale=()
|
|
local ref
|
|
while read -r ref; do
|
|
[ -n "$ref" ] || continue
|
|
is_excluded "$ref" && continue
|
|
[ -e "$dir$ref" ] || stale+=("$ref")
|
|
done < <(grep -oE "$suite/[a-z0-9][a-z0-9._-]*\.sh" "$CORPUS" \
|
|
| sed "s|^$suite/||" | sort -u)
|
|
if [ ${#stale[@]} -gt 0 ]; then
|
|
printf 'FAIL %-12s stale index references (no such file): %s\n' \
|
|
"$suite" "${stale[*]}"
|
|
rc=1
|
|
fi
|
|
fi
|
|
done
|
|
|
|
printf '\n'
|
|
if [ "$rc" -ne 0 ]; then
|
|
cat <<EOF
|
|
Undocumented tools are undiscoverable. An agent cannot obey a hard gate that
|
|
tells it to use a wrapper it has no way to learn exists — it will reach for raw
|
|
curl/gh/tea instead, and the wrapper's provider-dialect handling will be lost.
|
|
|
|
Fix by naming each tool above in one of the index documents listed at the top,
|
|
in the same commit that ships it.
|
|
EOF
|
|
else
|
|
printf 'every enforced suite is fully discoverable.\n'
|
|
fi
|
|
return "$rc"
|
|
}
|
|
|
|
# ---- self-test -------------------------------------------------------------
|
|
# Proves the gate can actually fail. A checker that only ever passes is
|
|
# indistinguishable from one that is not running, which is the failure mode this
|
|
# whole file exists to prevent — so it must demonstrate a red on demand.
|
|
self_test() {
|
|
local tmp rc
|
|
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' RETURN
|
|
mkdir -p "$tmp/tools/git"
|
|
printf '#!/bin/sh\n' > "$tmp/tools/git/documented-tool.sh"
|
|
printf '#!/bin/sh\n' > "$tmp/tools/git/test-ignored.sh"
|
|
chmod +x "$tmp/tools/git/documented-tool.sh" "$tmp/tools/git/test-ignored.sh"
|
|
|
|
# run_check reads the TOOLS_DIR / DOCS globals; an array cannot ride in a
|
|
# command-prefix assignment, so point the globals at the fixture directly.
|
|
TOOLS_DIR="$tmp/tools"
|
|
DOCS=("$tmp/doc.md")
|
|
|
|
# Case 1: fully documented -> pass.
|
|
printf 'see tools/git/documented-tool.sh for details\n' > "$tmp/doc.md"
|
|
if run_check >/dev/null; then
|
|
printf 'self-test 1/4 ok (complete index passes)\n'
|
|
else
|
|
printf 'self-test 1/4 FAIL (complete index should pass)\n'; return 1
|
|
fi
|
|
|
|
# Case 2: an undocumented tool -> fail.
|
|
printf '#!/bin/sh\n' > "$tmp/tools/git/undocumented-tool.sh"
|
|
rc=0; run_check >/dev/null || rc=$?
|
|
if [ "$rc" -eq 1 ]; then
|
|
printf 'self-test 2/4 ok (undocumented tool fails the gate)\n'
|
|
else
|
|
printf 'self-test 2/4 FAIL (undocumented tool should fail, got rc=%s)\n' "$rc"; return 1
|
|
fi
|
|
|
|
# Case 3: a stale index reference -> fail.
|
|
rm "$tmp/tools/git/undocumented-tool.sh"
|
|
printf 'also tools/git/deleted-tool.sh\n' >> "$tmp/doc.md"
|
|
rc=0; run_check >/dev/null || rc=$?
|
|
if [ "$rc" -eq 1 ]; then
|
|
printf 'self-test 3/4 ok (stale index reference fails the gate)\n'
|
|
else
|
|
printf 'self-test 3/4 FAIL (stale reference should fail, got rc=%s)\n' "$rc"; return 1
|
|
fi
|
|
|
|
# Case 4: documented, present, and NOT executable -> fail. Found by an
|
|
# independent reviewer: a 0644 wrapper scored 100% here while reading as
|
|
# absent to every `[ -x ]` in the fleet, including the wrapper guard's.
|
|
sed -i '/deleted-tool/d' "$tmp/doc.md"
|
|
printf '#!/bin/sh\n' > "$tmp/tools/git/noexec-tool.sh"
|
|
chmod 0644 "$tmp/tools/git/noexec-tool.sh"
|
|
printf 'and tools/git/noexec-tool.sh\n' >> "$tmp/doc.md"
|
|
rc=0; run_check >/dev/null || rc=$?
|
|
if [ "$rc" -eq 1 ]; then
|
|
printf 'self-test 4/4 ok (documented but non-executable tool fails the gate)\n'
|
|
else
|
|
printf 'self-test 4/4 FAIL (non-executable tool should fail, got rc=%s)\n' "$rc"; return 1
|
|
fi
|
|
|
|
printf '\nself-test passed: the gate demonstrably reds on every drift direction.\n'
|
|
}
|
|
|
|
if [ "$SELF_TEST" -eq 1 ]; then
|
|
self_test
|
|
else
|
|
resolve_locations
|
|
run_check
|
|
fi
|