framework: make tool discoverability, workspace placement and model tiering mechanical
An undocumented tool is, from inside an agent session, indistinguishable from a tool that was never written. The framework shipped 26 git wrappers and named 6 of them in its resident index docs — 23% discoverability, with pr-review.sh among the missing. The observable consequence was an agent obeying Constitution gate 7 as best it could see it, reaching for raw curl, sending GitHub's APPROVE to a Gitea host, and getting HTTP 200 with the review silently filed PENDING. Three times. That is not a discipline failure and no amount of prose fixes it. Four changes, each converting a rule that decayed into a mechanism that cannot: - check-tools-index.sh (new, CI-blocking): every tool in an enforced suite must be named in a resident index doc, and every tool an index names must exist. The git suite is enforced now; other suites report coverage without failing, so the ratchet tightens one reviewed PR at a time instead of landing as one sweep. The enforced list is framework-owned rather than a marker inside operator-owned TOOLS.md — a doc marker would let an operator silence the gate on exactly the host where it matters most. Carries --self-test, because a checker that only ever passes is indistinguishable from one that is not running. - TOOLS-REFERENCE.md: complete 28-entry git index, plus the APPROVED/APPROVE dialect note that explains why pr-review.sh is not a formality. - mosaic-worktree.sh + wrapper-guard.sh (upstreamed): the rule "big work goes on a work filesystem" already existed in prose, and 255 GB accumulated in $HOME across 842 directories anyway, under five simultaneous placement conventions on one host. The helper therefore exposes no placement decision — given a branch name, every path is derived from `git worktree list --porcelain`. Worktrees rather than clones because enumerability is the only thing that makes reclaim safe, and reclaim is by evidence (clean tree + no unpushed commits), never by size or age. The guard blocks three mechanically-detectable mistakes and nothing else: a checkout into $HOME, a raw provider-API write to an endpoint that has a wrapper, and the literal APPROVE event. Reads pass untouched. - STANDARDS.md: model tiering as a standard, named by capability class so it survives a model generation. Start cheapest, escalate on evidence, benchmark before demoting a task class, and keep the class->model binding in operator config with the DB-backed config service as the end state. Registering the guard in runtime/claude/settings.json is the point of upstreaming it: ~/.claude/settings.json is a framework-managed copy, so a hand-added hook there is destroyed by the next upgrade. In the template it survives, and it reaches every host instead of one.
This commit is contained in:
Hermes Agent
@@ -0,0 +1,266 @@
|
||||
#!/usr/bin/env bash
|
||||
# check-tools-index.sh — assert every shipped tool is discoverable from the
|
||||
# resident documentation an agent actually has in context.
|
||||
#
|
||||
# WHY THIS GATE EXISTS
|
||||
# --------------------
|
||||
# The framework ships 26 git wrappers. Before this gate, 20 of them were named
|
||||
# in neither `defaults/TOOLS.md` nor `guides/TOOLS-REFERENCE.md`. One of the
|
||||
# undocumented ones was `pr-review.sh` — the wrapper that carries the
|
||||
# APPROVED/APPROVE provider-dialect split.
|
||||
#
|
||||
# The observable consequence, on a live fleet host: an agent needing to place a
|
||||
# review verdict reached for raw `curl`, sent GitHub's `APPROVE` to a Gitea
|
||||
# host, and got HTTP 200 with the review silently filed PENDING — three times,
|
||||
# because nothing about the failure pointed at the wrapper that already handled
|
||||
# it correctly. The agent was not ignoring Constitution gate 7. It was obeying
|
||||
# an index that said the tool did not exist.
|
||||
#
|
||||
# That is not a discipline problem and no amount of prose fixes it. A wrapper
|
||||
# that is not in the resident index is, from inside a session, indistinguishable
|
||||
# from a wrapper that was never written. So the invariant is mechanical:
|
||||
#
|
||||
# shipping a tool and documenting it are the same commit, or CI fails.
|
||||
#
|
||||
# WHAT IT CHECKS
|
||||
# --------------
|
||||
# forward every non-excluded tool in an ENFORCED suite is named in at least
|
||||
# one index document (missing tool -> undiscoverable -> FAIL)
|
||||
# reverse every `<name>.sh` an index document attributes to an enforced
|
||||
# suite exists on disk (stale reference -> agent runs a ghost -> FAIL)
|
||||
#
|
||||
# Suites outside the enforced set are reported with a coverage percentage but do
|
||||
# not fail the build, so the ratchet can be tightened one suite per PR instead of
|
||||
# landing as one unreviewable sweep. `--strict` fails on those too.
|
||||
#
|
||||
# WHY THE ENFORCED LIST LIVES HERE AND NOT IN A MARKER INSIDE THE DOC
|
||||
# -------------------------------------------------------------------
|
||||
# `TOOLS.md` is operator-owned (see framework-manifest.txt). A marker inside it
|
||||
# would let an operator silence this gate by editing their own copy — the gate
|
||||
# would then be strongest exactly where it is least needed and absent where it
|
||||
# is needed most. The list is framework-owned and changes only through a
|
||||
# reviewed PR.
|
||||
#
|
||||
# Usage:
|
||||
# check-tools-index.sh [--tools-dir DIR] [--doc FILE]... [--strict] [--self-test]
|
||||
#
|
||||
# Exit: 0 = every enforced suite fully discoverable · 1 = drift · 2 = bad usage
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Suites whose coverage is a HARD requirement. Add a suite here only together
|
||||
# with the doc changes that make it pass.
|
||||
#
|
||||
# `git` is first because it is the suite Constitution gates 6-8 make mandatory:
|
||||
# an undiscoverable git wrapper converts a hard gate into a coin flip.
|
||||
ENFORCED_SUITES=(git)
|
||||
|
||||
# Files that are not agent-callable tools and must not be required in an index.
|
||||
EXCLUDE_GLOBS=(
|
||||
'test-*' # hermetic regression scripts, invoked by CI not by agents
|
||||
'_*' # private helpers (_lib, _scripts internals)
|
||||
'*.bak' # editor/installer debris
|
||||
'*.pre-*' # pre-change backups (e.g. ci-queue-wait.sh.pre-404fix-bak)
|
||||
'README.md'
|
||||
)
|
||||
|
||||
STRICT=0
|
||||
SELF_TEST=0
|
||||
TOOLS_DIR=""
|
||||
DOCS=()
|
||||
|
||||
die() { printf 'check-tools-index: %s\n' "$*" >&2; exit 2; }
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--tools-dir) TOOLS_DIR="${2:-}"; shift 2 ;;
|
||||
--doc) DOCS+=("${2:-}"); shift 2 ;;
|
||||
--strict) STRICT=1; shift ;;
|
||||
--self-test) SELF_TEST=1; shift ;;
|
||||
-h|--help) sed -n '2,48p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;;
|
||||
*) die "unknown argument: $1" ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ---- location resolution ---------------------------------------------------
|
||||
# Runs from two places with different layouts, and must not silently check the
|
||||
# wrong tree: a CI checkout (repo-relative) and an installed host ($MOSAIC_HOME).
|
||||
resolve_locations() {
|
||||
local here framework
|
||||
here="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# .../framework/tools/quality/scripts -> .../framework
|
||||
framework="$(cd -- "$here/../../.." && pwd)"
|
||||
|
||||
if [ -z "$TOOLS_DIR" ]; then
|
||||
if [ -d "$framework/tools" ]; then
|
||||
TOOLS_DIR="$framework/tools"
|
||||
else
|
||||
TOOLS_DIR="${MOSAIC_HOME:-$HOME/.config/mosaic}/tools"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [ ${#DOCS[@]} -eq 0 ]; then
|
||||
# The two layouts are mutually exclusive on purpose. Unioning them would let
|
||||
# a well-maintained operator TOOLS.md on the developer's own machine mask a
|
||||
# gap in the shipped defaults — the check would pass locally and the defect
|
||||
# would still install on every other host. Repo layout wins when present.
|
||||
if [ -f "$framework/defaults/TOOLS.md" ]; then
|
||||
DOCS+=("$framework/defaults/TOOLS.md")
|
||||
[ -f "$framework/guides/TOOLS-REFERENCE.md" ] && DOCS+=("$framework/guides/TOOLS-REFERENCE.md")
|
||||
else
|
||||
local mosaic_home="${MOSAIC_HOME:-$HOME/.config/mosaic}"
|
||||
[ -f "$mosaic_home/TOOLS.md" ] && DOCS+=("$mosaic_home/TOOLS.md")
|
||||
[ -f "$mosaic_home/guides/TOOLS-REFERENCE.md" ] && DOCS+=("$mosaic_home/guides/TOOLS-REFERENCE.md")
|
||||
fi
|
||||
fi
|
||||
|
||||
[ -d "$TOOLS_DIR" ] || die "tools dir not found: $TOOLS_DIR"
|
||||
[ ${#DOCS[@]} -gt 0 ] || die "no index documents found (pass --doc FILE)"
|
||||
}
|
||||
|
||||
is_excluded() {
|
||||
local name="$1" glob
|
||||
for glob in "${EXCLUDE_GLOBS[@]}"; do
|
||||
# shellcheck disable=SC2254 # glob is intentionally a pattern
|
||||
case "$name" in $glob) return 0 ;; esac
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# A tool counts as documented when its basename appears anywhere in the corpus.
|
||||
# Deliberately permissive about *form* (table cell, code fence, prose) and strict
|
||||
# about *presence*: the gate's job is "an agent can find it", not house style.
|
||||
documented() { grep -qF -- "$1" "$CORPUS"; }
|
||||
|
||||
# ---- the check -------------------------------------------------------------
|
||||
run_check() {
|
||||
local rc=0 suite dir tool base enforced
|
||||
|
||||
CORPUS="$(mktemp)"; trap 'rm -f "$CORPUS"' RETURN
|
||||
cat "${DOCS[@]}" > "$CORPUS"
|
||||
|
||||
printf 'tools: %s\n' "$TOOLS_DIR"
|
||||
for d in "${DOCS[@]}"; do printf 'index: %s\n' "$d"; done
|
||||
printf '\n'
|
||||
|
||||
for dir in "$TOOLS_DIR"/*/; do
|
||||
[ -d "$dir" ] || continue
|
||||
suite="$(basename -- "$dir")"
|
||||
case " ${ENFORCED_SUITES[*]} " in *" $suite "*) enforced=1 ;; *) enforced=0 ;; esac
|
||||
[ "$STRICT" -eq 1 ] && enforced=1
|
||||
case "$suite" in _*) continue ;; esac
|
||||
|
||||
local total=0 found=0
|
||||
local -a suite_missing=()
|
||||
for tool in "$dir"*.sh; do
|
||||
[ -e "$tool" ] || continue
|
||||
base="$(basename -- "$tool")"
|
||||
is_excluded "$base" && continue
|
||||
total=$((total + 1))
|
||||
if documented "$base"; then
|
||||
found=$((found + 1))
|
||||
else
|
||||
suite_missing+=("$base")
|
||||
fi
|
||||
done
|
||||
[ "$total" -eq 0 ] && continue
|
||||
|
||||
local pct=$(( found * 100 / total ))
|
||||
if [ "$enforced" -eq 1 ] && [ ${#suite_missing[@]} -gt 0 ]; then
|
||||
printf 'FAIL %-12s %3d%% (%d/%d) undocumented: %s\n' \
|
||||
"$suite" "$pct" "$found" "$total" "${suite_missing[*]}"
|
||||
rc=1
|
||||
elif [ "$enforced" -eq 1 ]; then
|
||||
printf 'ok %-12s %3d%% (%d/%d) [enforced]\n' "$suite" "$pct" "$found" "$total"
|
||||
else
|
||||
printf 'info %-12s %3d%% (%d/%d) not yet enforced\n' "$suite" "$pct" "$found" "$total"
|
||||
fi
|
||||
|
||||
# Reverse: an index that names a tool this suite does not have sends agents
|
||||
# after something that cannot run. Only checked for enforced suites, where
|
||||
# the naming is unambiguous enough to attribute.
|
||||
if [ "$enforced" -eq 1 ]; then
|
||||
local -a stale=()
|
||||
local ref
|
||||
while read -r ref; do
|
||||
[ -n "$ref" ] || continue
|
||||
is_excluded "$ref" && continue
|
||||
[ -e "$dir$ref" ] || stale+=("$ref")
|
||||
done < <(grep -oE "$suite/[a-z0-9][a-z0-9._-]*\.sh" "$CORPUS" \
|
||||
| sed "s|^$suite/||" | sort -u)
|
||||
if [ ${#stale[@]} -gt 0 ]; then
|
||||
printf 'FAIL %-12s stale index references (no such file): %s\n' \
|
||||
"$suite" "${stale[*]}"
|
||||
rc=1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
printf '\n'
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
cat <<EOF
|
||||
Undocumented tools are undiscoverable. An agent cannot obey a hard gate that
|
||||
tells it to use a wrapper it has no way to learn exists — it will reach for raw
|
||||
curl/gh/tea instead, and the wrapper's provider-dialect handling will be lost.
|
||||
|
||||
Fix by naming each tool above in one of the index documents listed at the top,
|
||||
in the same commit that ships it.
|
||||
EOF
|
||||
else
|
||||
printf 'every enforced suite is fully discoverable.\n'
|
||||
fi
|
||||
return "$rc"
|
||||
}
|
||||
|
||||
# ---- self-test -------------------------------------------------------------
|
||||
# Proves the gate can actually fail. A checker that only ever passes is
|
||||
# indistinguishable from one that is not running, which is the failure mode this
|
||||
# whole file exists to prevent — so it must demonstrate a red on demand.
|
||||
self_test() {
|
||||
local tmp rc
|
||||
tmp="$(mktemp -d)"; trap 'rm -rf "$tmp"' RETURN
|
||||
mkdir -p "$tmp/tools/git"
|
||||
printf '#!/bin/sh\n' > "$tmp/tools/git/documented-tool.sh"
|
||||
printf '#!/bin/sh\n' > "$tmp/tools/git/test-ignored.sh"
|
||||
|
||||
# run_check reads the TOOLS_DIR / DOCS globals; an array cannot ride in a
|
||||
# command-prefix assignment, so point the globals at the fixture directly.
|
||||
TOOLS_DIR="$tmp/tools"
|
||||
DOCS=("$tmp/doc.md")
|
||||
|
||||
# Case 1: fully documented -> pass.
|
||||
printf 'see tools/git/documented-tool.sh for details\n' > "$tmp/doc.md"
|
||||
if run_check >/dev/null; then
|
||||
printf 'self-test 1/3 ok (complete index passes)\n'
|
||||
else
|
||||
printf 'self-test 1/3 FAIL (complete index should pass)\n'; return 1
|
||||
fi
|
||||
|
||||
# Case 2: an undocumented tool -> fail.
|
||||
printf '#!/bin/sh\n' > "$tmp/tools/git/undocumented-tool.sh"
|
||||
rc=0; run_check >/dev/null || rc=$?
|
||||
if [ "$rc" -eq 1 ]; then
|
||||
printf 'self-test 2/3 ok (undocumented tool fails the gate)\n'
|
||||
else
|
||||
printf 'self-test 2/3 FAIL (undocumented tool should fail, got rc=%s)\n' "$rc"; return 1
|
||||
fi
|
||||
|
||||
# Case 3: a stale index reference -> fail.
|
||||
rm "$tmp/tools/git/undocumented-tool.sh"
|
||||
printf 'also tools/git/deleted-tool.sh\n' >> "$tmp/doc.md"
|
||||
rc=0; run_check >/dev/null || rc=$?
|
||||
if [ "$rc" -eq 1 ]; then
|
||||
printf 'self-test 3/3 ok (stale index reference fails the gate)\n'
|
||||
else
|
||||
printf 'self-test 3/3 FAIL (stale reference should fail, got rc=%s)\n' "$rc"; return 1
|
||||
fi
|
||||
|
||||
printf '\nself-test passed: the gate demonstrably reds on both drift directions.\n'
|
||||
}
|
||||
|
||||
if [ "$SELF_TEST" -eq 1 ]; then
|
||||
self_test
|
||||
else
|
||||
resolve_locations
|
||||
run_check
|
||||
fi
|
||||
Reference in New Issue
Block a user