Pipeline 2562: the shell chain now reaches the preflight suite and dies
at 'mktemp: : Invalid argument' — the CI image's mktemp dialect rejects
'mktemp -u /tmp/...XXXXXX.sock' (GNU accepts it locally, which is why
the suite passed on dev hosts only). The path is now composed
(/tmp/mosaic-preflight-$RANDOM-$$.sock): short enough for the 108-byte
AF_UNIX bind limit (a repo-deep path fails with 'path too long',
measured), unique per run, no pre-existing file, cleaned by the existing
trap. Verified locally: suite passes, no /tmp residue.
Pipeline 2561 failed 5 tests that pass on a dev host, two环境 classes:
1. Un-seamed success-path start/apply tests answered the HOST's broker
state: this dev machine has a live lease-broker socket at the default
XDG path, so the real probe (correctly, since #1297 F3) returned true
locally; CI has no broker, so the new start re-check refused
broker-absent. Fixed by injecting a deterministic present-broker seam
in exactly the tests whose property is ordering/targeting (reconciler
start-ordering, two apply-running tests, and the CLI command spec's
shared helper). Refusal and real-probe tests keep explicit false
seams / explicit socket paths.
2. The placeUnitFile unlink-failure spec relies on EACCES from a 0500
directory; CI steps run as root (apk add succeeds unprivileged-free)
and CAP_DAC_OVERRIDE makes unlink succeed, so the abort path cannot
be triggered there. Now it.skip under uid 0 with the reason stated;
exercised on every non-root dev host.
The remaining unseamed apply tests use the stopped-agent path (no broker
interaction) and passed CI unchanged — verified against the 2561 log,
not assumed.
Pipeline 2468/2556 failed the CI 'sanitization' step; the log (2556, step
60166) shows verify-sanitized.sh PASSED both times — the failure was the
test-enumeration guard bundled in that step: the new
tools/fleet/test-agent-session-broker-preflight.sh existed on disk but was
not enumerated on any CI surface. Wired as the last link of
test:framework-shell (guard: population 62, enumerated 47, excluded 16).
Freshened onto current next, which also grew a required generated-env key
(MOSAIC_GIT_IDENTITY, = agent name): the suite's hermetic env.generated
fixture now declares it, keeping the suite CI-fit against the current
start-agent-session.sh contract rather than the one it was written for.
rev-security-03 REQUEST_CHANGES (review note on the brain), all four
findings verified by measurement before fixing:
F1 BLOCKER — brokerSocketPresent used access(path, S_IFSOCK).
S_IFSOCK (0xC000) is a file-type constant, not an access() mode (0-7):
on node 24.18.0 the call throws ERR_OUT_OF_RANGE, so the probe could
NEVER return true and the swallow-catch made every un-seamed call report
the broker absent — on this host (roster v1) 'mosaic fleet start' would
have refused broker-absent forever. Now stat().isSocket(), matching the
bash side's [ -S ]. New spec exercises the REAL probe against a REAL
unix socket (true), a regular file (false), and an absent path (false) —
no seam, so no seam can hide this again.
F2 — placeUnitFile's single catch conflated destination-absent with
unlink-FAILED; on unlink failure it copied through the still-live
symlink (copy-through overwrite measured by the reviewer). Now: ENOENT
is the only swallowed lstat outcome; unlink failure aborts with a named
UnitPlacementError BEFORE any copy. New spec proves the residue target's
bytes survive an unlink failure and the destination link is untouched.
F3 — observeBroker defaulted unit/socket to false when seams unset, and
production injects none: plan/status/doctor reported a healthy broker as
absent. Seams still take precedence; with no seam the real stat()
probes run. The v2 start path (and apply-with-running) now re-check the
socket after enable+start with a NAMED lifecycle-precondition-failed
refusal — previously only the v1 path had that protection, and the
refusal was masked into the generic recoverable result. Acceptance
fixtures gain hermetic brokerSocketEnv paths (the old fixture asserted
the lying default).
F4 (note) — unlink race stays inside the user-owned dir; no action.
Local gates on this tree: vitest 1566/1566, typecheck, lint, prettier
3.8.1, verify-sanitized all pass. CI 2468's sanitization failure did not
reproduce locally on the identical tree; watching the fresh pipeline.
Wall 6: no documented path ever enabled or started the shipped
mosaic-lease-broker.service — every gated runtime died ~4s in at lease
registration while fleet start reported rc0, and a broker not in the
reconciler plan could not be reported as drifted.
Activation lands in the control plane, not the launcher:
- fleet install places ALL FOUR units through placeUnitFile — a placement
helper that unlinks any by-path-enable symlink at the destination
BEFORE copying (Node copyFile follows the link and overwrites the SEED
template; measured on a throwaway systemd user instance 2026-08-17,
with both cp and fs.copyFile), removes a stale wants-symlink pointing
outside the active dir (readlink — readFile returns the target's
content, not the link path), then copies and daemon-reloads. The same
measurement showed systemctl enable <name> does NOT rewrite an existing
by-path wants-symlink — reconciliation must be explicit. Idempotent:
second install on by-path residue converges to the identical state.
Until now the copy block named three units and omitted the broker, and
the residue set / copy set were disjoint only by accident (fomo-lin
survived copy-through because its one symlink was the one unit not
copied); adding the broker made them intersect on first run. See the
SET-INDEPENDENCE note on the helper before adding a fifth unit.
- enableFleetUnits enables the broker first, alongside the holder.
- fleet start / reconciler start the broker BEFORE any holder/agent
lifecycle effect, then RE-CHECK the socket (not unit state) and exit
nonzero with a named code if it did not appear. Re-probed on every
invocation — a RemainAfterExit=yes dead-looking-active unit can never
make retry look like repair (the sticky-retry check).
- The reconciler plan carries broker {unitInstalled, socketPresent} as a
first-class member; the socket is the signal (enabled-but-dead units
report socketPresent=false).
- start-agent-session.sh preflights the broker socket BEFORE any tmux
effect (moved ahead of the ownership probe): absent -> exit 75
(EX_TEMPFAIL), named refusal with socket path and remedy, no doomed
pane. The agent@ unit is Type=oneshot with no Restart=, so the message
survives instead of looping. The preflight detects and refuses; it
never starts the broker.
- mosaic doctor's lease check names one convention-neutral remedy:
'mosaic fleet install (it reconciles either enable convention)' —
written from the measurement; teaching a manual systemctl line could
leave a host with competing wants-symlinks.
Tests: fleet-place-unit.spec.ts (8: clean-host negative control,
by-path residue -> seed bytes AND mtime unchanged [the finding-2 check],
wants-residue cleared, idempotence single + double-install convergence);
fleet.spec.ts broker-first enable ordering, refused start emits no
holder/agent calls, second-start re-probe; reconciler broker plan member
(enabled-but-dead shape) + broker-before-agent ordering in both command
and apply paths; test-agent-session-broker-preflight.sh (CI-fit: fake
tmux, real unix socket at a short /tmp path — AF_UNIX caps at 108 bytes,
hermetic env; absent -> exit 75 + no tmux session, live socket passes,
explicit env wins, --stop not fenced). 1563/1563 vitest, lint, root
build 25/25, root typecheck 45/45.
Sabotage controls: placement unlink removed -> exactly the seed-integrity
test reddens (1/8); socket re-check disabled -> exactly the two preflight
specs redden; shell preflight removed -> the bash suite reddens (6 FAIL
assertions, rc=1). All restored byte-identically (sha256-verified), all
green again.
Test 6 (greenfield 1124, seat alive 2min + second fleet start) runs on
sandbox after daphne's baseline, coordinated with fred.
Note: the preflight uses exit 75 measured against the unit's Restart=
policy (oneshot, none) — no restart loop.
The repo format:check glob covers ts/js alongside md; four non-markdown
scripts inside the folded tree were flagged after the md pass. Same pinned
prettier 3.8.1, same markup-only class (verified: node --check still passes
on the js files).
Phase D3 of plan 2026-08-19 (decision S21): skills-local is the local test
bed; promote individually as each proves out. ms-unslop is the only one of
the seven local skills with working enforcement evidence — a checker
(tools/unslop-hook/unslop-check.js), a machine-source list (lists.json), a
19-test suite, a measured corpus, and a regression fixture.
The checker itself stays fleet-local for now (it binds to a specific
harness extension surface); this promotes the skill document only.
Gates verified on the moved file: sanitization denylist clean, prettier
3.8.1 clean (6730 chars was fred's measure at assignment; 7249 as shipped
today — both pass).
Phase D2 of plan 2026-08-19: single package, single install, single command.
The framework installer already treats skills/** as a shipped, manifest-owned
framework subtree, so the folded skills now install into
$MOSAIC_HOME/skills with the rest of the framework — no second repository,
no separate sync step:
- mosaic-sync-skills (bash + powershell): the fetch machinery is gone (clone,
pull, dirty-state migration, rsync from sources/agent-skills). The script
now only links installed skills into runtime homes. --link-only is a compat
no-op; --no-link exits having nothing to do.
- catalog.ts: the sources/agent-skills fallback is dead and removed.
- install.sh, launch.ts, defaults/README.md, README.md, skills/README.md:
references to the second repo rewritten to describe the shipped path.
Verified: clean install into a fresh MOSAIC_HOME produces 102 skills with no
sources/ directory; the linker then links the selected skills into the four
runtime homes with no git involvement.
963 markdown files reformatted with the repository's pinned prettier so
pnpm format:check covers the folded tree like every other repo file.
The formatter's embedded-language pass also normalized code fences
(TS semicolons, closed HTML tags in examples, lowercased CSS hex colors,
one renumbered list that skipped an index). Alphanumeric token deltas vs
the fold commit were audited file-by-file; all are formatter-equivalent
markup normalizations plus the four sanitized skills.
Four folded skills carried operator identity tokens that the sanitization
gate (verify-sanitized.sh) forbids in the public framework package:
- kickstart: template path pointed at a private brain checkout; now uses the
framework-shipped $MOSAIC_HOME/templates/docs/TASKS.md.template
- mosaic-deploy: dropped one estate-specific stack-name row from the example
table
- mosaic-portainer, mosaic-woodpecker: credentials now name the framework
credentials store (load_credentials <service>) instead of a private
checkout path
Estate-specific values can live in a skills-local override, which the linker
applies with precedence over canonical skills.
Fold the agent-skills repository into the monorepo as the shipped canonical
skills package (plan 2026-08-19 phase D1, decision S19: single package, single
install, single command).
History is preserved by rewriting each commit's paths from skills/ to
packages/mosaic/framework/skills/ (git fast-export/import) and merging the
rewritten history with --allow-unrelated-histories, so the original commits
with their authors, dates, and messages remain reachable. Blob content is
untouched by the rewrite; tree fidelity was verified blob-sha-for-blob-sha.
This change must be merged with a real merge commit (not squash) or the
history link is destroyed.
fargo