Pipeline 2468/2556 failed the CI 'sanitization' step; the log (2556, step
60166) shows verify-sanitized.sh PASSED both times — the failure was the
test-enumeration guard bundled in that step: the new
tools/fleet/test-agent-session-broker-preflight.sh existed on disk but was
not enumerated on any CI surface. Wired as the last link of
test:framework-shell (guard: population 62, enumerated 47, excluded 16).
Freshened onto current next, which also grew a required generated-env key
(MOSAIC_GIT_IDENTITY, = agent name): the suite's hermetic env.generated
fixture now declares it, keeping the suite CI-fit against the current
start-agent-session.sh contract rather than the one it was written for.
rev-security-03 REQUEST_CHANGES (review note on the brain), all four
findings verified by measurement before fixing:
F1 BLOCKER — brokerSocketPresent used access(path, S_IFSOCK).
S_IFSOCK (0xC000) is a file-type constant, not an access() mode (0-7):
on node 24.18.0 the call throws ERR_OUT_OF_RANGE, so the probe could
NEVER return true and the swallow-catch made every un-seamed call report
the broker absent — on this host (roster v1) 'mosaic fleet start' would
have refused broker-absent forever. Now stat().isSocket(), matching the
bash side's [ -S ]. New spec exercises the REAL probe against a REAL
unix socket (true), a regular file (false), and an absent path (false) —
no seam, so no seam can hide this again.
F2 — placeUnitFile's single catch conflated destination-absent with
unlink-FAILED; on unlink failure it copied through the still-live
symlink (copy-through overwrite measured by the reviewer). Now: ENOENT
is the only swallowed lstat outcome; unlink failure aborts with a named
UnitPlacementError BEFORE any copy. New spec proves the residue target's
bytes survive an unlink failure and the destination link is untouched.
F3 — observeBroker defaulted unit/socket to false when seams unset, and
production injects none: plan/status/doctor reported a healthy broker as
absent. Seams still take precedence; with no seam the real stat()
probes run. The v2 start path (and apply-with-running) now re-check the
socket after enable+start with a NAMED lifecycle-precondition-failed
refusal — previously only the v1 path had that protection, and the
refusal was masked into the generic recoverable result. Acceptance
fixtures gain hermetic brokerSocketEnv paths (the old fixture asserted
the lying default).
F4 (note) — unlink race stays inside the user-owned dir; no action.
Local gates on this tree: vitest 1566/1566, typecheck, lint, prettier
3.8.1, verify-sanitized all pass. CI 2468's sanitization failure did not
reproduce locally on the identical tree; watching the fresh pipeline.
Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.
Activation lands in the control plane, not the launcher:
- fleet install places ALL FOUR units through placeUnitFile — a placement
helper that unlinks any by-path-enable symlink at the destination
BEFORE copying (Node copyFile follows the link and overwrites the SEED
template; measured on a throwaway systemd user instance 2026-08-17,
with both cp and fs.copyFile), removes a stale wants-symlink pointing
outside the active dir (readlink — readFile returns the target's
content, not the link path), then copies and daemon-reloads. The same
measurement showed systemctl enable <name> does NOT rewrite an existing
by-path wants-symlink — reconciliation must be explicit. Idempotent:
second install on by-path residue converges to the identical state.
Until now the copy block named three units and omitted the broker, and
the residue set / copy set were disjoint only by accident (fomo-lin
survived copy-through because its one symlink was the one unit not
copied); adding the broker made them intersect on first run. See the
SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
lifecycle effect, then RE-CHECK the socket (not unit state) and exit
nonzero with a named code if it did not appear. Re-probed on every
invocation — a RemainAfterExit=yes dead-looking-active unit can never
make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
first-class member; the socket is the signal (enabled-but-dead units
report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
effect (moved ahead of the ownership probe): absent -> exit 75
(EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
pane. The agent@ unit is Type=oneshot with no Restart=, so the message
survives instead of looping. The preflight detects and refuses; it
never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
'mosaic fleet install (it reconciles either enable convention)' —
written from the measurement; teaching a manual systemctl line could
leave a host with competing wants-symlinks.
Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.
Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.
Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.
Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.
Per ops-ci-01 (PR #1324 comments 23413/23422) and fred's one-at-a-time
authorization: one serialized run of #1324's tree on the pinned image
(ci-base:lock-9cb7ffcd8828, carried since the cb9a0d1 absorb). This commit
changes no files — the tree is identical to 9af456c; it exists only to mint a
fresh pipeline head instead of restarting 2545.
Per ops-ci-01's procedure (comments 23386/23394, brain D27): pipeline config
travels with the commit under test, so the controlled rerun requires the pinned
anchor in the PR head itself. This merge absorbs everything next took since the
original merge parent (1bdeed62): the ci-base pin (cb9a0d1, lock-9cb7ffcd8828),
the legacy-credential removal (#1325), and the ci-queue-wait statuses:null fix
(#1129 — the branch from the divergence analysis, now landed).
One conflict, same file as round 1: test:framework-shell union — our 54-entry
chain plus next's new test-ci-queue-wait-no-status.sh entry at its position
(55 entries, every target existence-verified). Enumeration guard green:
population 61, enumerated 46, excluded (signed) 16.
Pipeline 2534 (018d96a), two test-step failures plus the format failure
root-caused by rev-code-01 (id 202):
1. pipefail-early-exit.test.mjs: main's window wrote the test but never
wired scripts/*.test.mjs into its own CI, so its tools/install.sh pipes
were never executed against it; next's test wiring runs them and flags
two load-bearing pipes. Both rewritten pipefail-safe via process
substitution (newest_matching_file ls|head, check_fleet_transport
sed|head|tr|awk). Test now 7/7 locally.
2. verify-release.test.mjs: merge kept main's 9-command ci.yml sanitization
but next's 4-command canonical stage. Canonical updated to mirror ci.yml
exactly (9=9, order verified). upgrade-guard checked equal (4=4).
3. format: docs/reports/quality/1099-pipefail-sweep.md (from main f158be8)
prettier-formatted under pinned 3.8.1.
Enumeration guard re-verified: population 60, enumerated 45, excluded 16.
Review stack#1324 id 201 (rev-code-01): the union resolution chained main's #1073
fleet test while keeping next's signed exclusion of it (#1271 burn-down: the test
asserts missing-binary behavior but CI provides pi on PATH — fails by design).
check-test-enumeration CONTRADICTORY EXCLUSION, CI pipeline 2532 terminal failure.
Fix per verified review recommendation: remove the chain entry (55 -> 54 parts),
keeping next's exclusion and #1271 state. Guard now passes:
population 60, enumerated 45, excluded (signed) 16.
The repo format:check glob covers ts/js alongside md; four non-markdown
scripts inside the folded tree were flagged after the md pass. Same pinned
prettier 3.8.1, same markup-only class (verified: node --check still passes
on the js files).
Phase D3 of plan 2026-08-19 (decision S21): skills-local is the local test
bed; promote individually as each proves out. ms-unslop is the only one of
the seven local skills with working enforcement evidence — a checker
(tools/unslop-hook/unslop-check.js), a machine-source list (lists.json), a
19-test suite, a measured corpus, and a regression fixture.
The checker itself stays fleet-local for now (it binds to a specific
harness extension surface); this promotes the skill document only.
Gates verified on the moved file: sanitization denylist clean, prettier
3.8.1 clean (6730 chars was fred's measure at assignment; 7249 as shipped
today — both pass).
Phase D2 of plan 2026-08-19: single package, single install, single command.
The framework installer already treats skills/** as a shipped, manifest-owned
framework subtree, so the folded skills now install into
$MOSAIC_HOME/skills with the rest of the framework — no second repository,
no separate sync step:
- mosaic-sync-skills (bash + powershell): the fetch machinery is gone (clone,
pull, dirty-state migration, rsync from sources/agent-skills). The script
now only links installed skills into runtime homes. --link-only is a compat
no-op; --no-link exits having nothing to do.
- catalog.ts: the sources/agent-skills fallback is dead and removed.
- install.sh, launch.ts, defaults/README.md, README.md, skills/README.md:
references to the second repo rewritten to describe the shipped path.
Verified: clean install into a fresh MOSAIC_HOME produces 102 skills with no
sources/ directory; the linker then links the selected skills into the four
runtime homes with no git involvement.
963 markdown files reformatted with the repository's pinned prettier so
pnpm format:check covers the folded tree like every other repo file.
The formatter's embedded-language pass also normalized code fences
(TS semicolons, closed HTML tags in examples, lowercased CSS hex colors,
one renumbered list that skipped an index). Alphanumeric token deltas vs
the fold commit were audited file-by-file; all are formatter-equivalent
markup normalizations plus the four sanitized skills.
Four folded skills carried operator identity tokens that the sanitization
gate (verify-sanitized.sh) forbids in the public framework package:
- kickstart: template path pointed at a private brain checkout; now uses the
framework-shipped $MOSAIC_HOME/templates/docs/TASKS.md.template
- mosaic-deploy: dropped one estate-specific stack-name row from the example
table
- mosaic-portainer, mosaic-woodpecker: credentials now name the framework
credentials store (load_credentials <service>) instead of a private
checkout path
Estate-specific values can live in a skills-local override, which the linker
applies with precedence over canonical skills.
Fold the agent-skills repository into the monorepo as the shipped canonical
skills package (plan 2026-08-19 phase D1, decision S19: single package, single
install, single command).
History is preserved by rewriting each commit's paths from skills/ to
packages/mosaic/framework/skills/ (git fast-export/import) and merging the
rewritten history with --allow-unrelated-histories, so the original commits
with their authors, dates, and messages remain reachable. Blob content is
untouched by the rewrite; tree fidelity was verified blob-sha-for-blob-sha.
This change must be merged with a real merge commit (not squash) or the
history link is destroyed.
rev-code-01's non-blocking nit on #1313 round 2. The no-linking-step paragraph
pointed at the bridge explanation as 'described below'; it is above. Now names the
section, which survives further reordering better than a direction word does.
Text-only. Verified with the repo's PINNED prettier (3.8.1 via pnpm-lock.yaml) and
the sanitization gate, both clean.
All four findings reproduced before fixing. rev-code-01 was right on each.
B2 (blocker, mine). SEAT-IDENTITY provisioning step 4 said to symlink the
framework store entry to the seat slot, while the same file says those bridges
must not be recreated. The same bridge, told both ways, in one document. I
rewrote the resolution and token-location sections when the deploy made them
stale and did not carry the change into the numbered steps. Step 4 is gone and
the file now says explicitly that no provisioning step links the store to the
slot, so the omission cannot read as an oversight.
S1 (mine). The guide claimed the helper "attempts a fleet notification" on
refusal. The shipped helper does no such thing — its only reference to
notification is a comment saying an alert built on the record is best-effort, and
there is no send or wake call anywhere in the file. Now: it writes a durable
record, the record is what exists, and nobody should wait for a notification that
nothing sends. A guide that promises an alert is worse than one that promises
nothing.
S2. Estate-local content removed from files that ship to every estate: the
~/.mosaic/fleet/bin script paths (dead paths elsewhere) and the 2026-08-18 dates,
which dated a specific host's migration rather than describing behavior. The
bridge-removal passage now states the ORDERING that matters — remove bridges only
after a seat-aware helper can reach the slot, never before — which is the part
that transfers.
B1. prettier reformatted all three files. Reproduced the pipeline 2515 failure
locally before and confirmed clean after; the other three guides prettier flags
are untouched by this branch (0 changes vs origin/next) and are pre-existing.
Sanitization gate re-run and passing.
Verified for the record, since I could not verify my own work: rev-code-01
confirmed the no-fallback claim TRUE against helper content on origin/next, and
judged the evidence rules actionable on the grounds that each names an executable
replacement.
Pipeline 2514 failed the sanitization gate on 'Jason mints the token into the
seat slot'. The denylist is jarvis|jason|woltje|... and a shipped framework file
must not carry operator identity. My mistake: I generalized the estate paths and
seat names when promoting this guide and did not check the operator name.
Now reads 'the estate operator', with the accompanying rule that an agent does
not ask another agent to mint one either.
Verified by running tools/quality/scripts/verify-sanitized.sh locally rather than
guessing at the pattern: gate passes.
The #1280 prohibition carried an explanation of how the tools misattribute and
why the failure is invisible from inside them. A reader who is not going to use
the tool cannot act on any of it. Same for rule 2's closing clause about what
reviews commonly miss. Both cut to the decree and the corrective action.
Rules 1 and 3-12 keep their trailing sentences: those are corrective actions or
the detail that makes the case recognizable, not justification.
Three guides that existed only as one host's working copy, promoted to framework
templates so every estate gets them. A working copy under ~/.mosaic binds one
host; only a template here binds all of them.
SEAT-IDENTITY.md (new) documents how a seat's git credential is actually
resolved after #1311: identity from MOSAIC_GIT_IDENTITY, then
mosaic.gitIdentity, then the stdin username; host mapped to a store prefix; then
ONE of two stores chosen by whether the seat directory exists, with no
precedence and no fallback between them. A seat with a directory and an empty
slot fails closed rather than reaching the service store, and that is the point.
It also corrects how to find the helper. credential.helper commonly names an
absolute path, so `command -v git-credential-mosaic` answers a different question
than the one git asks, and the two stop agreeing the moment the PATH copy is
removed. Git also tries EVERY configured helper in order, so a fail-closed helper
in front silently hands the request to whatever is configured behind it. The
guide says to read the whole list.
FLEET-COMMS.md (new) documents agent-send.sh: the class table, the addressing
preamble, and the exit codes — including that rc=2 means the text reached the
pane as an unsubmitted draft, so retrying double-sends it. Confirm with
capture-pane instead. It also says to measure the fleet rather than trust
roster.yaml, which on a live host was simultaneously naming a socket that did not
exist, listing seats that were not running, and omitting seats that were.
CODE-REVIEW.md gains an Evidence Discipline section: a green is not a result
until you have shown it could go red, measurement and explanation are separate
sentences, verify by content on the ref that ships rather than by ancestry of a
local sha, and confidence is part of a finding. Plus four shell-measurement rules
earned on #1311, each of which produced a wrong conclusion first — `cmd | tail;
echo rc=$?` reports tail's status, a missed glob under pipefail exits 2 and kills
the run under set -e, nonzero-with-no-output is an environment question before it
is a code question, and `git -C` in a non-repo directory answers from the
enclosing repo.
The estate-specific repository exception that lived in the working copy is not
carried here. The template says an estate may document one, scoped to a named
repository and never precedent for a second.
Both new guides are added to the two routing tables that agents read.
Both defects found in review by rev-code-01 on #1311.
F3 — the JSONL record interpolated every field with a bare %s. An identity comes
from git config or the environment and a cwd is whatever directory git ran in, so
either can contain a quote or a backslash. One such refusal turned the day's spool
into unparseable JSONL, and the operator would only discover it while reading the
record that explains an outage. Fields are now JSON-escaped.
F2 — the diagnostic printed "record: <spool>/<date>.jsonl" unconditionally, but
the record is only written inside the branch where mkdir -p succeeded. When the
spool cannot be created the helper named a file that does not exist, on exactly
the hosts where the escalation was lost. It now reports the real path or says
NOT WRITTEN.
Also: prettier on README.md, which was the format-step failure on pipeline 2508.
It reflowed only the two tables this branch added.
Tests: cases 14 and 15 cover both. Verified discriminating — against the previous
helper with these same tests, case 14 fails with the unparseable record printed
and case 15 fails on both assertions; against this one both pass.
The first draft of case 14 used `ls "$spool"/*.jsonl | head -1`, which under
`set -o pipefail` exits 2 on a missed glob and killed the suite with zero output
— the same silent-nonzero failure rev-code-01 hit from a partial tools/ extraction
and the reason this file exists. Replaced with a glob loop and a comment.
Two changes to one rule: a credential is resolved from exactly one place,
and an identity that cannot be resolved is refused rather than substituted.
FAIL CLOSED. Both readers ended in an unconditional fall-through to the
shared Gitea account whenever an identity did not resolve. Every seat in a
fleet therefore pushed, opened PRs and filed reviews under one account, and
a record made that way cannot be traced to the agent that made it
afterwards. The fallback now applies only where there is no attribution to
lose: a host with no fleet. Where seats exist, an unresolvable request emits
nothing, exits nonzero, explains itself on stderr, and — in the git helper —
appends a record naming the identity, host, reason and cwd, and no token
value, to ${MOSAIC_CREDENTIAL_SPOOL:-~/.local/state/mosaic-credential-escalations}.
A host runs a fleet when <brain>/fleet/agents exists, which is the signal
packages/mosaic/src/fleet/brain-home.ts already uses to decide a brain is
active, resolved the same way (MOSAIC_BRAIN_HOME, else ~/.mosaic). This is
what keeps the change a no-op for an operator who has not provisioned
per-slot tokens: no fleet directory, shared account, unchanged. It is also
why there is no environment variable to restore the old behavior — one would
reintroduce the substitution being removed.
STORE SELECTION. Both readers hardcoded ~/.config/mosaic/secrets/gitea-tokens,
so a seat's own secrets/ slot was invisible to the framework: a seat could
hold a valid credential and still be served the shared account. The store is
now chosen by what the identity is. An identity with a directory under
<brain>/fleet/agents/ is a seat and is read only from
<brain>/fleet/agents/<id>/secrets/; any other identity is a service identity
and is read from the framework store. There is no precedence between them
and no fallback from one to the other, so a seat with an empty slot is
refused even when a same-named token sits in the framework store. Two copies
of one credential are drift rather than redundancy, and drift surfaces as
the stale copy returning 401, which reads as a revoked token and sends
whoever debugs it somewhere else.
detect-platform.sh is in scope alongside git-credential-mosaic because they
are the two readers of these tokens. Patching only the git helper would make
"one credential, one location" true for push and fetch and false for
pr-create.sh, issue-create.sh and pr-review.sh, which is the harder failure
to notice.
TESTS. The three assertions that pinned the shared-account fall-through are
now fail-closed assertions, and a refusal is checked four independent ways:
nonzero exit, empty stdout, a stderr diagnostic naming identity and host,
and no shared token value anywhere in the output. The exit code alone would
pass against a helper that emitted the credential and then failed. Added:
seat-slot resolution, the no-cross-store-fallback case with a control
proving the framework-store file it declines to read is readable, no-identity
on a fleet host, the fleet gate firing on the default ~/.mosaic and not only
on an injected MOSAIC_BRAIN_HOME, and a cross-host leak check. Both suites
were run against the pre-change code as a control and fail there on exactly
the shared-token emission.
shellcheck is not installed on the authoring host, so the rewritten helper
is unlinted locally and CI is the first lint of it.
Squash-merged by topher (jarvis principal) via break-glass API path (wrapper main-only gap, documented). Gates: CI 2492 green at head e19013ed, review 181 APPROVED (fred) at pinned head. Diagnostic merge: the push pipeline now exercises kaniko with the REGISTRY_* secrets - valid creds yield the first fully green gated publish; invalid yield an explicit 401.
Co-authored-by: Jarvis <[email protected]>
Squash-merged by topher (jarvis principal) via break-glass API path (wrapper main-only gap, documented in #1275 log). Gates: CI 2487 green at head 81f500bd, review 180 APPROVED (fred) at pinned head. Unblocks gated publish: verify's no-DB path now skips cleanly.
Co-authored-by: Jarvis <[email protected]>
Squash-merged by topher (jarvis principal) via break-glass: pr-merge.sh hard-codes main-only merge targets and cannot express this repo's next trunk. Gates: CI 2476 green at head 46784c8d, review 177 APPROVED (fred) at pinned head. First gated publish: every publish step now depends on verify-release at the exact commit.
Squash-merged by topher (jarvis principal) via break-glass: pr-merge.sh hard-codes main-only merge targets and cannot express this repo's next trunk. Gates: CI 2475 green at head 758659dd, review 176 APPROVED (fred) at pinned head.
The first PR for this change was filed under the retired mos-dt-0 principal
(pr-create.sh has no --login flag and find_tea_login_for_host returns the first
host match) and was closed and refiled as #1270. That left in-tree references
pointing at a closed duplicate PR rather than at the burn-down issue, which is
the wrong target for them anyway: the open design question belongs on #1271.
The `test` step has failed on every `next` pipeline since #1017 on exactly one
assertion, and it is the same one on unrelated PRs:
FAIL: host provides 'pi' in the system path; missing-binary cases are not
measurable here (framework/tools/fleet/test-start-agent-session.sh:103)
Measured 2026-08-16 across pipelines 2444 (#1256), 2438 (#1240) and 2441
(#1017-quality): exactly one FAIL line in each full log, identical, this line.
Control `zzz-not-present-zzz` -> 0 on all three.
Cause. #1241 (5c35a250) added the guard: the suite shims fake mosaic/pi/npm into
$FAKE_BIN, but the constructed PANE_PATH always ends in the real system path, so
on a host that installs those binaries the missing-binary cases cannot be
measured and a green run would mean nothing. The guard says so instead of
passing. Its own pipeline 2430 was green only because the suite was CI-excluded
at the time, so the guard had never run in CI. #1017 (c56483eb) then enumerated
it and dropped the exclusion. The CI image installs
@earendil-works/[email protected].1 on purpose, so the precondition is
unsatisfiable there. Both commits are mine.
The guard is correct and is not being softened. A check that cannot measure its
property and reports success is the failure mode this repo has been cataloguing
all week; the error was wiring the suite into an image that violates its
precondition, so the wiring is what gets reverted.
Second effect, which is the reason this cost a day rather than an hour:
test:framework-shell is one && chain and this sat at position 44 of 48, so
glpi/test-list-http-status.sh, orchestrator/test-board-roll.sh,
woodpecker/test-ci-wait-exit-matrix.sh and _scripts/test-fleet-transport-check.sh
have not run at all since the merge. The pipeline reported one failure, never
"one failure plus four unrun". All four are green when run directly on
sb-it-1-dt, so the mask hid nothing broken -- but that is a local result on one
host, not a CI-image result.
Verification, with controls:
- enumeration guard OK (population 52, enumerated 36, signed-excluded 16).
- control A, exclusion line removed while unwired -> FAIL UNENUMERATED.
- control B, exclusion line kept while rewired -> FAIL CONTRADICTORY EXCLUSION.
The gate discriminates in both directions, so its OK is load-bearing.
- the four formerly-masked suites: rc=0 each, run directly.
- the full chain cannot be run to completion on sb-it-1-dt: it stops earlier, at
the lease-broker Invariant R test, because this host carries the quarantined
operator-global pi 0.84.2 against a measured 0.84.1. That is host-specific and
out of scope here -- CI pins 0.84.1, and the single FAIL line in those three
pipelines proves positions 1-43 passed there.
Burn-down is to control the tail of PANE_PATH inside the test, not to remove pi
from the image. Recorded in the exclusion reason and in #1269.
fargo
fredandgate-merge-01
ops-ci-01
ops-01
fargo
jason.woltje
jarvisandfargo