Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 fargo
|
9d3e22b1c1 |
@@ -1,186 +0,0 @@
|
||||
# Quality-Rails Probe Inventory — RI-3-001
|
||||
|
||||
- **Task:** RI-3-001 (SDLC-D-037 first half; PRD § Release Integrity Workstream, RI-N4)
|
||||
- **Date:** 2026-08-18
|
||||
- **Base:** `origin/next` @ `8199261c` (branch `docs/ri-050-qr-probe-inventory`)
|
||||
- **Follow-up:** RI-3-002 consumes the dispositions here when building the single TS evaluator.
|
||||
|
||||
## 0. Scope and method
|
||||
|
||||
Every mechanism in this repository that verifies a quality, integrity, safety, or release
|
||||
property — TypeScript checks, shell probes, pipeline steps, git hooks, and installer-side
|
||||
assertions — gets one row. Each row's "what it actually verifies" was written from the
|
||||
probe's **code**, not its name or docs. Framework tool unit/regression suites (git wrappers,
|
||||
wake, tmux, orchestrator, …) are treated as one enforcement surface (`test:framework-shell`)
|
||||
because they test tool behavior rather than repo quality; their wiring integrity is itself
|
||||
guarded by `check-test-enumeration.sh`, and the quality-relevant members are rowed
|
||||
individually.
|
||||
|
||||
**Kinds:** `ts` (TypeScript/Node check), `shell` (bash/python probe), `pipeline-step`
|
||||
(exists only inside a Woodpecker pipeline).
|
||||
|
||||
**Enforcement points:** `local` (operator-invoked), `pre-commit`, `pre-push`,
|
||||
`CI ci.yml#<step>`, `publish.yml#<step>` (CI on push to main/next), `turbo <task>`,
|
||||
`agent-runtime` (framework hooks on an agent host), `installer` (host install path),
|
||||
`unwired`.
|
||||
|
||||
**Dispositions** (recommendations for RI-3-002): `preserve` (keep as-is; already the
|
||||
canonical or a correct guard-of-the-guard), `strengthen` (keep, but a concrete gap must
|
||||
close — usually absorption into the TS evaluator), `strengthen (review)` (viable retirement
|
||||
candidate once the evaluator absorbs it; do not retire yet). Note: RI-N4 requires that
|
||||
effective shell probes be **absorbed before** their independent paths retire — no row here
|
||||
is marked `retire` because no absorption exists yet.
|
||||
|
||||
## 1. Inventory
|
||||
|
||||
### 1.1 Repo-level gate tasks (pnpm / turbo)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------------------- | ------------------------------------------------------------------------------------ | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | ------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `pnpm preflight` (checkout preflight) | `scripts/preflight.mjs` | ts | Six gate binaries (eslint, husky, prettier, tsc, turbo, vitest) exist and are executable in `node_modules/.bin` (exit 42 if not); no stale `.mosaic-test-work/web-build.lock` (exit 43); `apps/web/.next` is a real directory (not a symlink), every entry owned by the current uid, and its `.mosaic-source-hash` fingerprint + `.mosaic-symlink-manifest` hash match the certified build written by `scripts/build-web.mjs` | `pre-push`; inside `pnpm typecheck` (→ `CI ci.yml#typecheck`, verify-release `typecheck` stage) | QC-1 Checkout integrity | preserve | Blocks a poisoned/stale generated `.next` from faking a green typecheck (the five-month-stale-`.next` class); trust chain is self-contained per-checkout. |
|
||||
| `pnpm typecheck` | root `package.json` → `turbo run typecheck` | ts | Per-package `tsc --noEmit` (all 20 packages); turbo `typecheck` depends on `^build`, so package builds must succeed first; prefixed by checkout preflight | `CI ci.yml#typecheck`; `pre-push`; verify-release `typecheck` stage; `turbo typecheck` | QC-2 Workspace typecheck | preserve | The single workspace-wide type gate; CI and hooks invoke the same task, no divergent checklist. |
|
||||
| `pnpm lint` | root `package.json` → `turbo run lint` | ts | Per-package `eslint src` under root `eslint.config.mjs` (ignores `dist`, `.next`, `framework/**`, etc.) | `CI ci.yml#lint`; `pre-push`; verify-release `lint` stage; `turbo lint` | QC-3 Workspace lint | preserve | Same-task invocation from every surface; no second lint definition. |
|
||||
| `pnpm format:check` | root `package.json` → `prettier --check` | ts | Prettier parse/format equality over `**/*.{ts,tsx,js,jsx,json,md}` minus `.prettierignore` (generated trees, `docs/scratchpads/`, venvs, …) | `CI ci.yml#format`; `pre-push`; verify-release `format` stage | QC-4 Format check | preserve | Single formatter, single ignore list, enforced identically everywhere. |
|
||||
| `pnpm test` | root `package.json` `test` = `test:checkout` && `turbo run test` && `test:installer` | ts | (a) `node --test scripts/*.test.mjs` — checkout-tool units; (b) per-package `vitest run` (mosaic appends the 47-command `test:framework-shell` chain); (c) `tools/install-next-lane.test.sh`; turbo `test` declares DB env vars and depends on `^build` | `CI ci.yml#test` (with `DATABASE_URL` + `db:migrate` first); verify-release `test` stage; `turbo test` | QC-5 Test suite execution | preserve | One composed test command; the chain property (any link red ⇒ step red) is the gate. |
|
||||
| `pnpm build` | root `package.json` → `turbo run build` | ts | Per-package build (`tsc`/Next) with `^build` dependency and `dist/**` outputs | `publish.yml#build`; verify-release `build` stage; `turbo build` | QC-6 Workspace build | preserve | Publish artifacts derive from the same build task CI verifies. |
|
||||
|
||||
### 1.2 Framework quality shell probes (`packages/mosaic/framework/tools/quality/`)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------------------------- | ----------------------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
|
||||
| Sanitization gate | `scripts/verify-sanitized.sh` | shell | Built-in self-test first (planted identity/structural/YAML+service fixtures; exit 2 if the regexes or extension coverage break), then: (1) identity denylist grep (`jarvis\|jason\|woltje\|brain.woltje.com\|/home/jwoltje\|\bPDA\b`) over all shipped text files **including** `examples/`; (2) structural grep for private `$HOME/src` defaults in shipped scripts **excluding** `examples/`. Any hit ⇒ exit 1 | `CI ci.yml#sanitization`; verify-release `sanitization` stage | QC-7 Framework sanitization | preserve | Labeled one-time regression guard with a self-test that prevents silent no-op; correctly scoped (identity vs structural) and documented as not a general PII detector. |
|
||||
| Resident-context budget | `scripts/check-resident-budget.sh` (+ `--self-test`) | shell | Self-test of the comparator, then `wc -l` vs per-file ceilings (CONSTITUTION 120, AGENTS 120, each RUNTIME.md 90); missing file ⇒ fail; over ceiling ⇒ exit 1 | `CI ci.yml#sanitization` (both modes); verify-release `sanitization` stage | QC-8 Resident-context budget | preserve | Caps the container (lines), never the wording — the deliberate anti-drift design (DESIGN §7); CI-enforceable half only, by design. |
|
||||
| Test-membership enumeration guard (#1017) | `scripts/check-test-enumeration.sh` + `test-enumeration-exclusions.txt` | shell | Parses surface S1 (`packages/mosaic` `test:framework-shell` via JSON+shlex) and S2 (every `framework/tools/\*.sh | .py`token in`ci.yml`, comment lines stripped); population = `_test_.sh`under`framework/tools`; FAILS on: suite-shaped file on disk neither enumerated nor signed-excluded; surface naming a path missing on disk (both directions); exclusion without reason / stale / outside population / contradicting enumeration. Proves **naming, not reachability** (stated in-file) | `CI ci.yml#sanitization` (direct line); link [0] of `test:framework-shell` (thus `CI ci.yml#test`); verify-release `sanitization` stage | QC-9 Test-membership enumeration | preserve | Makes silent under-run impossible; invoked from both surfaces it audits so severing the chain cannot silence it. |
|
||||
| Enumeration-guard needles | `scripts/test-check-test-enumeration.sh` | shell | Needle/control fixtures driven through `--root`: every promised failure mode must trip the guard **on its own words**, plus controls that must pass (null-case defense); covers commented-out ci.yml lines (F1) and line-range parsing (n2b) | `test:framework-shell` → `CI ci.yml#test`; verify-release `test` stage | QC-9 Test-membership enumeration | preserve | Guard-of-the-guard with both polarities; same canonical check by design. |
|
||||
| Upgrade manifest guard (#791 HARD GATE) | `scripts/test-upgrade-manifest-guard.sh` | shell | Keep-mode `install.sh` upgrade against seeded throwaway `MOSAIC_HOME`: every operator sentinel — including an **unanticipated** one — survives byte-identical with unchanged mtime; framework files still update; retired framework files pruned; matrix run with rsync present AND absent (keep path must be rsync-independent); fail-closed matrix (empty/operator-only/malformed/missing manifest aborts loudly, operator files untouched); operator secret never appears in installer output | `CI ci.yml#upgrade-guard`; verify-release `upgrade-guard` stage | QC-10 Upgrade/install safety | preserve | The operator-data hard gate for the `mosaic update` path; negative controls are load-bearing and documented. |
|
||||
| Upgrade rollback gate (#791 B1) | `scripts/test-upgrade-rollback.sh` | shell | Mid-sync failure (PATH-shadowing `cp` shim) must trigger snapshot restore: restore message fires, corrupted file restored, target byte-identical to pre-upgrade; control installer with `set -E` stripped must NOT roll back (proves errtrace is load-bearing); plus signal/exit-guard controls | `CI ci.yml#upgrade-guard`; verify-release `upgrade-guard` stage | QC-10 Upgrade/install safety | preserve | Proves the rollback trap actually fires; the `-E`-stripped control keeps Part A honest. |
|
||||
| Durable-snapshot gate (#791 PR2) | `scripts/test-upgrade-durable-snapshot.sh` | shell | Pre-update snapshot taken before any mutation (0700/0600 perms, secret never logged, retention-pruned); post-sync verify net restores operator files a manifest bug lets the sync touch; CWE-59 symlink-leaf guard proven with a portable cp shim in both polarities (write-through-link must not happen); v1→v2 migration semantics (intended `bin/` removal not healed) | `CI ci.yml#upgrade-guard`; verify-release `upgrade-guard` stage | QC-10 Upgrade/install safety | preserve | Covers tampering and leak vectors the manifest guard cannot see; the shim rationale (busybox vs GNU cp) is documented in-file. |
|
||||
| Install migration matrix (v2→v3) | `scripts/test-install-migration.sh` | shell | Fixture matrix running the real installer with `MOSAIC_SYNC_ONLY=1`: fresh install seeds + stamps version 3; legacy user-edited AGENTS overwritten with `.pre-constitution.bak` preserved (and idempotent); tuned STANDARDS overwritten; operator files (SOUL, credentials) preserved. Mirrors the TS suite `packages/mosaic/src/config/file-adapter.test.ts` — both installers must behave identically | `CI ci.yml#upgrade-guard`; verify-release `upgrade-guard` stage | QC-10 Upgrade/install safety | preserve | Pins the shell/TS installer parity contract; removal would orphan that parity requirement. |
|
||||
| Enforcement verification probe (bash) | `scripts/verify.sh` | shell | Attempts **real commits** in the target repo: planted type error must produce a commit blocked with `error`; planted `any` must trip `no-explicit-any`; planted lint error must trip `prettier`; gitleaks binary must exist (3a) and detect a planted AWS key via `gitleaks git --pre-commit --staged --redact` (3b). Verdicts are output-grep matches on hook stderr | `local` via installed `mosaic-quality-verify` on scaffolded target projects; **not run in this repo's CI** | QC-20 Downstream enforcement verification | strengthen (review) | Mechanism is genuinely behavioral (stronger than file presence) but verdict logic is grep-on-output and it is unwired here; absorb as the evaluator's enforcement-probe check (the RI-N4 evaluator invokes it or reimplements it) before retiring the shell path. |
|
||||
| Enforcement verification probe (PowerShell) | `scripts/verify.ps1` | shell | Windows port of `verify.sh`: same planted-commit tests with `$output -match` matching; no gitleaks self-test parity beyond the same checks | `local` (Windows operator); no Windows CI runner exists | QC-20 Downstream enforcement verification | strengthen (review) | A hand-maintained twin of `verify.sh` with no CI coverage — exactly the drift shape the single evaluator removes; retire after the TS evaluator owns the probe. |
|
||||
| Quality template installer (bash) | `scripts/install.sh` | shell | Copies template files (`.husky/pre-commit` incl. mandatory gitleaks, `.lintstagedrc.js`, `.eslintrc.js`, `tsconfig.json`, `.woodpecker.yml`, `.gitleaks.toml`) into a target project; **warns** (does not verify) about `package.json` snippet merge; no post-condition check | `local` / via `mosaic-quality-apply` | QC-21 Downstream rails scaffolding | strengthen (review) | Duplicates the TS `quality-rails init` scaffolder for a different template set; converging on one scaffolder (with post-scaffold verification) is prerequisite to retiring this path. |
|
||||
| Quality template installer (PowerShell) | `scripts/install.ps1` | shell | Windows twin of the template copy above | `local` (Windows operator) | QC-21 Downstream rails scaffolding | strengthen (review) | Same twin-drift risk as `verify.ps1`; no runner exercises it. |
|
||||
| `mosaic-quality-verify` adapter | `framework/tools/_scripts/mosaic-quality-verify` | shell | Thin adapter: validates target dir exists, asserts `verify.sh` present+executable, `cd` target, exec it. No verdict logic of its own | `local` (installed framework bin) | QC-20 Downstream enforcement verification | preserve | Already the thin-adapter shape RI-N4 prescribes for shell surfaces. |
|
||||
| `mosaic-quality-apply` adapter | `framework/tools/_scripts/mosaic-quality-apply` | shell | Thin adapter: arg validation then exec of quality `install.sh --template … --target …` | `local` (installed framework bin) | QC-21 Downstream rails scaffolding | preserve | Thin adapter, no separate verdict; disposition follows its target script's convergence. |
|
||||
| Roster schema regression | `scripts/test-roster-schema.py` | shell | jsonschema `Draft202012Validator` over `fleet/roster.schema.json` with valid/invalid connector-kind fixtures (tmux/discord/matrix conditional fields) | **unwired** — not on S1 or S2, not signed-excluded; also outside the enumeration guard's `*.sh` population, so the guard cannot see it | QC-5 Test suite execution | strengthen (review) | A real regression suite that currently runs nowhere; wire it into a CI surface or sign an exclusion — leaving it invisible re-arms the exact gap #1017 closed. |
|
||||
| Framework shell chain (S1) | `packages/mosaic/package.json` `test:framework-shell` | shell | 47-command `&&` chain: enumeration guard + needles, 14 lease-broker/mutator-gate python unitests, `check-runtime-launches.py`, and ~30 framework-tool shell suites (git wrappers, wake, woodpecker, tmux, glpi, orchestrator, `_scripts`). Quality-relevant members rowed separately below | `turbo test` → `CI ci.yml#test`; verify-release `test` stage | QC-5 Test suite execution | preserve | The chain is the execution surface the enumeration guard audits; known residuals: a failing link stops later suites (measured in #1270 — suites after position 44 had not run), and the guard proves naming, not reachability. |
|
||||
|
||||
### 1.3 Framework runtime hooks and their harnesses (agent-host enforcement)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------------------- | ----------------------------------------------------------------------------------------- | ----- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------- | --------------------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| QA edit hook seam | `framework/tools/qa/qa-hook-stdin.sh` (+ `qa-hook-handler.sh`) | shell | PostToolUse stdin hook: extracts edited file from the tool JSON (jq or grep fallback), skips non-JS/TS, then the deps-preflight gate — exits 1 with the legible sentinel `deps not installed — run pnpm install` when `node_modules/.bin` is missing/empty (the #856 false-red class); the downstream handler only files QA remediation **report templates** (no verification logic) | `agent-runtime` (framework `runtime/claude/settings.json` PostToolUse); never CI | QC-16 Agent-runtime edit-time checks | strengthen (review) | The sentinel gate is real enforcement; the handler's report-filing adds no verdict and its name promises more than the code does — evaluator absorption should keep the sentinel, drop the report theater. |
|
||||
| Typecheck-on-edit hook | `framework/tools/qa/typecheck-hook.sh` | shell | PostToolUse: for edited `.ts/.tsx`, finds nearest `tsconfig.json` and runs `tsc --noEmit`, surfacing errors nonzero to the agent immediately | `agent-runtime` (framework `runtime/claude/settings.json` PostToolUse) | QC-16 Agent-runtime edit-time checks | strengthen (review) | Edit-time duplicate of QC-2 with independent invocation logic; keep behavior, converge invocation through the evaluator adapter. |
|
||||
| Deps-preflight harness | `framework/tools/qa/test-deps-preflight.sh` | shell | Five assertions against the seam incl. a documented RED control (raw `not found`), sentinel behavior for missing and empty `.bin`, and no-false-positive once populated | `test:framework-shell` → `CI ci.yml#test` | QC-16 Agent-runtime edit-time checks | preserve | Guard-of-the-check with a red control; keeps the sentinel from regressing. |
|
||||
| Prompt-helper RCE regression | `framework/tools/_scripts/test-mosaic-init-rce.sh` | shell | Sources the prompt helpers and proves a literal `$(touch /tmp/pwned)` answer round-trips verbatim and never executes (no `/tmp/pwned` created) | `test:framework-shell` → `CI ci.yml#test` | QC-5 Test suite execution | preserve | Cheap, load-bearing security regression on the installer's input path. |
|
||||
| Install-ordering harness (#869 C2) | `framework/tools/_scripts/test-install-ordering-guard.sh` | shell | Drives `mosaic-link-runtime-assets` with a fake `mosaic` on PATH: probe ok ⇒ settings copied + exit 0; probe fail ⇒ exit 1 with degraded outcome but all other runtime files still copied; `--allow-inactive-enforcement` forwarded; no-mosaic-on-PATH ⇒ python3 fallback strips enforcement hooks and exits 1; fallback + flag ⇒ wires as-is, exit 0 | `test:framework-shell` → `CI ci.yml#test` | QC-17 Lease-enforcement wiring safety | preserve | Exercises the shell wiring seam independently of the TS guard's own spec suite (complementary coverage, by design). |
|
||||
| Fleet-transport harness (#1240) | `framework/tools/_scripts/test-fleet-transport-check.sh` | shell | Extracts the shipped `check_fleet_transport`/`fleet_declared_transport` functions **from the shipped scripts** (fails loud if extraction yields nothing) and drives both implementations (mosaic-doctor + `tools/install.sh`) from one case table | `test:framework-shell` → `CI ci.yml#test` | QC-18 Operator-host drift audit | preserve | The anti-drift harness for the one rule shipped twice; extraction-from-source keeps it from testing a stale copy. |
|
||||
| Terminal-green contract (RM-61/#1000) | `framework/tools/woodpecker/test-terminal-green-contract.sh` + `verify-terminal-green.py` | shell | Red-first fixtures: pipeline JSON variants (service failure, step failure, cancelled, etc.) must produce the correct terminal-green verdict; controls must pass | `test:framework-shell` → `CI ci.yml#test` | QC-5 Test suite execution | preserve | Keeps the CI-wait wrapper's green-detection honest; a false green here would poison every merge gate that trusts `pr-ci-wait.sh`. |
|
||||
| Lease-gate launch invariant | `framework/tools/lease-broker/check-runtime-launches.py` | shell | Scans production roots (`packages/`, `apps/`, `plugins/`, `tools/`) across sh/py/ts/yaml suffixes for Claude/Pi process launches **outside** the lease gate; allowlist-based; fails CI on violation | `test:framework-shell` → `CI ci.yml#test` | QC-15 Lease-gate architecture invariant | preserve | The only architectural "no ungated launches" rail; grep+allowlist is the right cost/benefit for this invariant. |
|
||||
|
||||
### 1.4 TypeScript quality logic (`@mosaicstack/quality-rails` + mosaic CLI)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ---------------------------------------- | ---------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | ------------------------------------- | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `quality-rails check` | `packages/quality-rails/src/cli.ts` (`mosaic quality-rails check --project`) | ts | **Expected-file presence only**: loops `expectedFilesForKind` (node: `.eslintrc`, `biome.json`, `.githooks/pre-commit`, `PR-CHECKLIST.md`; python: `pyproject.toml`+hooks+checklist; rust: `rustfmt.toml`+…) and exits 1 listing missing paths. Does not execute any linter, formatter, hook, or scanner | `local` (operator CLI); **no CI wiring in this repo** | QC-19 Downstream rails presence check | strengthen | This is the RI-N4 evaluator seed. Today presence ≠ parity (explicitly called out by RI-N4): it must grow typed verdicts (`passed/failed/blocked/error/not-applicable`), check versioning/subject/reason, digested definitions, and absorb the effective shell probes (QC-20 first). |
|
||||
| `quality-rails doctor` | `packages/quality-rails/src/cli.ts` | ts | Same presence data as `check`, printed with ok/missing lines; **cannot fail** (no nonzero exit on missing files) | `local` (operator CLI) | QC-19 Downstream rails presence check | strengthen | A doctor that cannot fail is advisory; fold into `check` (or return typed states) when the evaluator lands. |
|
||||
| `quality-rails init` | `packages/quality-rails/src/cli.ts` + `scaffolder.ts`/`templates.ts` | ts | Scaffolds rails files per detected kind/profile (linters/formatters lists are advisory strings; hooks flag always true); writes files, prints follow-ups — no post-condition verification | `local` (operator CLI) | QC-21 Downstream rails scaffolding | strengthen (review) | Second scaffolding path alongside quality `install.sh` (§1.2); converge on one with post-scaffold verification before retiring either. |
|
||||
| Lease activation probe (#869 C1, hidden) | `packages/mosaic/src/commands/lease-activation-probe.ts` | ts | Real capability probe, not file presence: resolves the installed mosaic CLI and requires it to advertise the exact `{name, version}` activation contract; all deps injectable; registered as hidden CLI command and consumed by C2/C5 | `local` (hidden CLI + consumed by C2/C5); spec-tested via `lease-activation-probe.spec.ts` in `turbo test` | QC-17 Lease-enforcement wiring safety | preserve | The versioned-contract probe is precisely the fail-closed capability check RI-N2 generalizes; already typed and injectable. |
|
||||
| Install-ordering guard (#869 C2, hidden) | `packages/mosaic/src/commands/install-ordering-guard.ts` | ts | Decides whether enforcement hook entries are written into the `~/.claude/settings.json` the framework reseed ships: not activatable ⇒ strip hooks + nonzero loud outcome (default); explicit per-invocation `--allow-inactive-enforcement` opt-out wires-with-warning. Never touches the runtime gate's own fail-closed behavior | `installer` (framework reseed via `mosaic-link-runtime-assets`); spec + shell harness coverage in `turbo test` | QC-17 Lease-enforcement wiring safety | preserve | Correct default-deny with an explicit, non-env opt-out; test-locked from both the TS and shell sides. |
|
||||
| Lease doctor check (#869 C5) | `packages/mosaic/src/commands/lease-doctor-check.ts` | ts | Combines hook-wiring detection in `~/.claude/settings.json` with C1 activatable and C3 broker-supervisor health: wired ∧ (¬activatable ∨ ¬healthy) ⇒ loud `[ERROR]` that forces `mosaic doctor` exit 1 regardless of the bash audit's own exit | `local` (inside `mosaic doctor`); spec coverage in `turbo test` | QC-17 Lease-enforcement wiring safety | preserve | Closes the "bricked host looks green" hole; cannot be masked by the bash script — that composition is the point. |
|
||||
| `mosaic doctor` (framework drift audit) | `packages/mosaic/src/commands/launch.ts` (`doctor`) + `framework/tools/_scripts/mosaic-doctor` | shell+ts | Bash audit of the installed framework home: ~40 expected files/dirs present; runtime files are copies (not symlinks) matching source (`cmp`) or composed runtime-contract markers; hard-gates block present in AGENTS.md; sequential-thinking MCP configured; fleet transport binary present per roster (warn); legacy symlink trees gone; skills synced — **warn-based, exit 1 only with `--fail-on-warn`**, plus C5's forced error | `local` (operator audit) | QC-18 Operator-host drift audit | preserve | Host-state audit CI cannot see (user files by design, DESIGN §7); advisory exit is the documented contract — do not silently change it. |
|
||||
| `mosaic gateway doctor` | `packages/mosaic/src/commands/gateway-doctor.ts` | ts | Probes per-service health (PostgreSQL, Valkey, pgvector) via `@mosaicstack/storage`, reports tier and JSON; exit 1 only when at least one **required** service fails (yellow stays 0) | `local` (operator) | QC-18 Operator-host drift audit | preserve | Service health with correct red/yellow exit semantics; JSON mode exists for scripting. |
|
||||
| `mosaic gateway verify` | `packages/mosaic/src/commands/gateway/verify.ts` | ts | Post-install liveness: daemon meta via HTTP with retries, admin token on file, bootstrap endpoint reachable; aggregated pass/fail | `local`; consumed by `tools/e2e-install-test.sh` | QC-18 Operator-host drift audit | preserve | The first-run proof the installer E2E relies on; retry-aware so startup races don't false-red. |
|
||||
| `mosaic fleet doctor` | `packages/mosaic/src/commands/fleet-reconciler-command.ts` | ts | Classifies local roster-owned drift (no mutation) from the parsed v2 roster | `local` (operator) | QC-18 Operator-host drift audit | preserve | Dry-run classification is the correct non-mutating audit shape. |
|
||||
|
||||
### 1.5 Git hooks (developer machine)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------- | --------------------------------------------------------- | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | --------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------- |
|
||||
| Pre-commit staged hygiene | `.husky/pre-commit` → `npx lint-staged` (`.lintstagedrc`) | shell | On staged files only: `prettier --write` + `eslint --fix` for ts/tsx/js/jsx; `prettier --write` for json/md/yaml/yml. **Mutating** (fixes and re-stages); commit blocks only if a fixer itself fails | `pre-commit` (every local commit; hooks activated by `install-hooks.mjs` via `core.hooksPath .husky/_`) | QC-13 Staged-change hygiene | preserve | Correct scoped fast gate; note it auto-fixes rather than rejects (deliberate). Gap: no secret scan here — see §3. |
|
||||
| Pre-push gate | `.husky/pre-push` | shell | `pnpm preflight && pnpm typecheck && pnpm lint && pnpm format:check` (no test run — documented in AGENTS.md) | `pre-push` | QC-14 Pre-push gate | preserve | Composes QC-1..4 exactly as specified in AGENTS.md; tests intentionally left to CI. |
|
||||
| Hook installer | `scripts/install-hooks.mjs` (`pnpm prepare`) | ts | Stages husky hooks into a scratch repo first, asserts husky produced its `h` shim, quarantines incomplete previous sets, verifies idempotence via full directory snapshot comparison, then sets `core.hooksPath`; skips cleanly with `HUSKY=0` or no git | `installer` (runs on `pnpm install`) | QC-13 Staged-change hygiene | preserve | Self-verifying wiring for the hook gates — a corrupted half-install cannot silently disable them. |
|
||||
|
||||
### 1.6 CI pipeline steps (`.woodpecker/`)
|
||||
|
||||
Step-to-probe mapping for container steps: `ci.yml#sanitization` = QC-7+QC-8+QC-9 (rows §1.2, plus `apk add bash` env prep); `ci.yml#upgrade-guard` = QC-10 (rows §1.2, plus `apk add rsync`); `ci.yml#typecheck`/`#lint`/`#format`/`#test` = QC-2/3/4/5 (rows §1.1). Rows below are mechanisms that exist only in a pipeline.
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| -------------------------------------- | -------------------------------------------------------------------------------------- | ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------- | ----------------------------------- | --------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------- |
|
||||
| Frozen install | `ci.yml#install` | pipeline-step | `pnpm install --frozen-lockfile --prefer-offline` against the baked ci-base store — lockfile supply integrity; a drifted lockfile fails the build before any gate runs | `CI ci.yml#install` | QC-1 Checkout integrity | preserve | Lockfile-pinned dep resolution is the supply-chain floor under every later gate. |
|
||||
| Test-step readiness prelude | `ci.yml#test` prologue | pipeline-step | Installs pinned `@earendil-works/[email protected]` (Invariant R suite requires the real binary) + openssl; waits up to 60×1s on `pg_isready` for the `ci-postgres` service and fails fast if it never comes up; runs `db:migrate` before tests | `CI ci.yml#test` | QC-5 Test suite execution | preserve | Fail-fast environment preconditions — a missing service produces a legible failure, not a wall of red tests. |
|
||||
| Publish verify step (pending RI-1-001) | `publish.yml#verify` (branch `feat/ri-050-publish-gate` @ `46784c8d`, not yet on next) | pipeline-step | (a) Commit identity: fails closed if `CI_COMMIT_SHA` empty, `git rev-parse HEAD` empty, or the two differ; (b) runs the canonical `pnpm verify:release`. **Every publish effect depends on this step; it carries no path filter** | `publish.yml#verify` | QC-11 Terminal release verification | preserve | The RI-N1 exact-commit binding; until it merges, publish steps on next depend on `build` only (see §3 gap 1). |
|
||||
| Publish error classification | `publish.yml#publish-npm` | pipeline-step | Publishes `@mosaicstack/*` (minus web) and classifies outcome: success, or the **only tolerated failure** = already-published (EPUBLISHCONFLICT / "cannot publish over" / "previously published"); explicit fatal on npm `E404/E401/ENEEDAUTH/ECONNREFUSED/ETIMEDOUT/ENOTFOUND` and on any unrecognized failure (replacing the old ` | | echo` that hid a registry 404) | `publish.yml#publish-npm` (main/tags, path-filtered on `packages/**`) | QC-12 Publish-effect integrity | preserve | Converts silent publish fall-on-floor into loud failure; allowlist-of-one error tolerance is the right shape. |
|
||||
| Next-lane publish assertions | `publish.yml#publish-next-npm` | pipeline-step | Guards: branch must be `next`, `CI_PIPELINE_NUMBER` required; registry dist-tags JSON must be usable; walks all manifests, strictly parses stable semver, rewrites `X.Y.(Z+1)-next.<N>`; publishes with `--tag next` (never latest); post-publish asserts `npm view @mosaicstack/mosaic@next` resolves to the exact expected version | `publish.yml#publish-next-npm` (push/manual on next) | QC-12 Publish-effect integrity | preserve | Durable prerelease lane with end-to-end resolution proof — the published artifact is verified, not assumed. |
|
||||
| Image destination policy | `publish.yml#build-gateway` / `#build-appservice` / `#build-web` | pipeline-step | Kaniko builds with destination policy: `next` ⇒ sha-tag only (fatal if a tag event sneaks in); `main` ⇒ sha + `latest`; tag events ⇒ sha + `<tag>`; anything else fatal. Path filters only skip **effects**, never the verify step | `publish.yml#build-*` | QC-12 Publish-effect integrity | preserve | Fail-closed tagging matrix; the exclude-list default-safe design keeps stale images impossible. |
|
||||
|
||||
Adjacent pipeline surface (not a probe): `.woodpecker/ci-image.yml` rebuilds the ci-base image on `pnpm-lock.yaml`/`Dockerfile.ci` change with an immutable `lock-<hash>` tag; pipelines consume `:latest`. Recorded for completeness — no code-quality property is checked.
|
||||
|
||||
### 1.7 Root installer tooling (`tools/`)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| --------------------------- | --------------------------------------------------------- | ----- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ------------------------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Next-lane installer test | `tools/install-next-lane.test.sh` (`pnpm test:installer`) | shell | Drives `tools/install.sh --next` with faked `node`/`npm` binaries (no network): Node 20 must be rejected; installs must pin **exact** versions (mutable `@next` forbidden); fast path must not unexpectedly fall back to source; gateway-install failure takes the documented fallback | `turbo`-external tail of `pnpm test` → `CI ci.yml#test` | QC-5 Test suite execution | preserve | Hermetic (shimmed) regression net for the installer lane; runs as part of the standard test command. |
|
||||
| Clean-container install E2E | `tools/e2e-install-test.sh` | shell | Full first-run flow in a node:22-alpine container: `install.sh --yes` → `mosaic wizard` (non-interactive) → `mosaic gateway install` → `mosaic gateway verify` exit check (with EXPECTED-SKIP if the installed CLI predates `gateway verify`); skips gracefully without Docker | `local` (manual; requires Docker); **not wired in CI** | QC-5 Test suite execution | strengthen (review) | The only end-to-end proof of the install→verify path; currently operator-initiated only — wire into a periodic/manual CI lane or sign its exclusion explicitly. |
|
||||
| Host installer advisories | `tools/install.sh` (`--check`; `check_fleet_transport`) | shell | `--check` = version comparison only, no install; `check_fleet_transport` warns (non-blocking, by design — tmux is the fleet's dependency, not mosaic's) when the roster-declared transport binary is absent, naming exactly what it blocks; PATH-persistence warnings | `installer` (operator-run) | QC-18 Operator-host drift audit | preserve | Advisory-by-design warnings; the parallel doctor check is drift-tested by §1.3's harness. |
|
||||
|
||||
### 1.8 Pending workstream additions (branch `feat/ri-050-publish-gate` @ `46784c8d`)
|
||||
|
||||
| check | location | kind | what it actually verifies | enforcement point | canonical check | disposition | rationale |
|
||||
| ------------------------------- | ---------------------------------------------------- | ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------- | ----------------------------------- | ----------- | ---------------------------------------------------------------------------------------------- |
|
||||
| Canonical terminal verification | `scripts/verify-release.mjs` (`pnpm verify:release`) | ts | One command replaying the full mandatory set as stages — sanitization, upgrade-guard, typecheck (incl. preflight), lint, format, test, build — mirroring `ci.yml` step-for-step; fail-fast on first failing command; requires `bash`+`rsync` on PATH; `--stage <name>` for wiring smoke-tests only | `publish.yml#verify` (pending); `local` (`pnpm verify:release`) | QC-11 Terminal release verification | preserve | The RI-N1 canonical command — CI and publication share one semantic checklist by construction. |
|
||||
| Verify-parity contract test | `scripts/verify-release.test.mjs` | ts | Parses the real `ci.yml`/`publish.yml`: stage table must match ci.yml step-for-step; every publish-effect step (name `publish*` or image-pushing) must transitively depend on `verify`; commit-identity assertion must be present; `verify` must carry no path filter | `test:checkout` → `CI ci.yml#test` (once merged) | QC-11 Terminal release verification | preserve | Guard-of-the-guard at checkout time — the two surfaces cannot drift apart silently. |
|
||||
|
||||
## 2. Canonical check set
|
||||
|
||||
The deduplicated checks every row above maps onto. IDs are stable for RI-3-002 to consume.
|
||||
|
||||
- **QC-1 Checkout integrity.** Owns: the checkout can run its gates — frozen-lockfile dependency resolution, required gate binaries present, no stale build lock, and the `apps/web/.next` generated-state trust chain (real directory, uid ownership, certified source fingerprint, certified symlink manifest). Implemented by `scripts/preflight.mjs` + frozen install steps.
|
||||
- **QC-2 Workspace typecheck.** Owns workspace-wide TypeScript soundness: per-package `tsc --noEmit` over built dependencies (`turbo typecheck`). The single definition invoked by CI, pre-push, and terminal verification.
|
||||
- **QC-3 Workspace lint.** Owns static-analysis policy: per-package ESLint under the root config. One config, one task, every surface.
|
||||
- **QC-4 Format check.** Owns formatting uniformity: Prettier check with the repo ignore list. (The pre-commit variant additionally fixes; the verdict form is this check.)
|
||||
- **QC-5 Test suite execution.** Owns execution of all test surfaces: checkout script units (`node --test`), per-package Vitest suites (including the framework shell chain and its python unitests), the installer-lane shim test, and — once wired — `test-roster-schema.py` and container E2E. Also owns guards-of-the-gate that live inside the chain (terminal-green contract, RCE regression).
|
||||
- **QC-6 Workspace build.** Owns artifact buildability: `turbo build` producing the artifacts publication consumes.
|
||||
- **QC-7 Framework sanitization.** Owns the open-source guarantee for the shipped framework package: no operator-identity tokens anywhere (examples included), no private `$HOME` defaults in shipped scripts, with a self-test that keeps the regexes honest.
|
||||
- **QC-8 Resident-context budget.** Owns the line-count ceilings on framework files injected into every agent's context (Constitution, dispatcher, RUNTIME.md slices) — the CI-enforceable half of the resident-prompt budget.
|
||||
- **QC-9 Test-membership enumeration.** Owns the property that no test suite can silently fall out of CI: disk population vs parsed enumeration surfaces, both-directions staleness, and signed exclusions with reasons. Includes its needle/control harness.
|
||||
- **QC-10 Upgrade/install safety.** Owns the #791 family: operator-path byte-identity across keep-mode upgrades (manifest guard), mid-failure rollback (errtrace-proven), durable pre-update snapshot + verify net + CWE-59 leaf guard, and the v2→v3 migration matrix with shell/TS parity.
|
||||
- **QC-11 Terminal release verification.** Owns the RI-N1 exact-commit binding: commit-identity assertion plus one canonical command (`pnpm verify:release`) replaying the complete mandatory set, with every publish effect depending on it; plus the checkout-time parity/DAG contract test that keeps pipeline and command in sync.
|
||||
- **QC-12 Publish-effect integrity.** Owns publication correctness: npm publish error classification (only already-published tolerated), next-lane versioning and post-publish resolution proof, and image destination/tag policy.
|
||||
- **QC-13 Staged-change hygiene.** Owns commit-time hygiene on staged files (prettier/eslint fix-and-restage) and the self-verifying hook wiring that guarantees the gates are actually installed.
|
||||
- **QC-14 Pre-push gate.** Owns the local push composition: preflight + typecheck + lint + format:check (tests deliberately deferred to CI).
|
||||
- **QC-15 Lease-gate architecture invariant.** Owns "no ungated runtime launches in production code": the scan + allowlist over `packages/`, `apps/`, `plugins/`, `tools/`.
|
||||
- **QC-16 Agent-runtime edit-time checks.** Owns edit-time feedback on agent hosts: the deps-preflight legibility sentinel and typecheck-on-edit, plus their regression harnesses.
|
||||
- **QC-17 Lease-enforcement wiring safety.** Owns the #869 C1/C2/C5 trio: activation capability probe (versioned contract), enforcement-hook wiring gate (default-deny with explicit opt-out), and the doctor check that surfaces a bricked host — with their shell/TS harnesses.
|
||||
- **QC-18 Operator-host drift audit.** Owns host-state health CI cannot see: `mosaic doctor` drift audit (+ fleet transport, both implementations), `fleet doctor` roster classification, `gateway doctor`/`gateway verify` service health, and installer advisories. Advisory exits are part of the contract.
|
||||
- **QC-19 Downstream rails presence check.** Owns "does a scaffolded project still carry its rails files" — today the TS `quality-rails check/doctor` presence loop; per RI-N4 this is the seed that must become the typed evaluator (presence alone is explicitly not parity).
|
||||
- **QC-20 Downstream enforcement verification.** Owns "do the rails actually block" on scaffolded projects: the behavioral planted-commit probe (type error, `any`, lint, gitleaks secret) currently in `verify.sh`/`verify.ps1` behind the `mosaic-quality-verify` adapter.
|
||||
- **QC-21 Downstream rails scaffolding.** Owns putting rails files into a target project: the shell template installer (+ PowerShell twin) and the TS `quality-rails init` scaffolder — currently two paths that must converge.
|
||||
|
||||
## 3. Coverage gaps
|
||||
|
||||
Enforced nowhere but implied, or named in docs/tooling but not wired:
|
||||
|
||||
1. **Publication not yet bound to verification on `next`.** At this base (`8199261c`), `publish.yml` publish steps depend on `build` only; the `verify` step and `scripts/verify-release.mjs` exist on `feat/ri-050-publish-gate` (`46784c8d`) but are not merged. Until RI-1-001 lands, AC-RI-1's negative control cannot hold on the real pipeline.
|
||||
2. **Playwright E2E unwired.** `apps/web` ships `test:e2e` (`playwright test`) with real suites (`admin/auth/chat/navigation.spec.ts`); neither `pnpm test` nor any CI step invokes it. The web UI's user flows are verified only when an operator runs them manually.
|
||||
3. **No secret scanning on this repo.** The framework's own template pre-commit makes gitleaks **required**, and `verify.sh` proves detection with a planted key — but this repository's `.husky/pre-commit` (lint-staged only) and CI run no secret scan. The repo ships the control it does not use.
|
||||
4. **No dependency audit.** The quality `.woodpecker.yml` templates and `docs/CI-SETUP.md` specify `npm audit --audit-level=high` as a pipeline stage; nothing equivalent runs for this repo.
|
||||
5. **No coverage thresholds.** Templates enforce 80% Jest coverage thresholds; this repo's Vitest configs collect coverage with no thresholds — coverage is measured nowhere and enforced nowhere.
|
||||
6. **`test-roster-schema.py` invisible.** A real jsonschema regression suite wired to no surface and invisible to the enumeration guard (its population is `*.sh`; the suite is `.py`). Either enumerate it or sign an exclusion — silence here is the #1017 defect shape.
|
||||
7. **Presence-checker expectations ≠ this repo.** `quality-rails check` expects `.eslintrc`, `biome.json`, `.githooks/pre-commit`, `PR-CHECKLIST.md` for node projects — none describe this monorepo (husky, flat eslint config, no biome, no PR-CHECKLIST.md). The evaluator's check set must be per-subject (versioned, digested), not one global file list.
|
||||
8. **Chain-ordering residual (documented).** `test:framework-shell` is one `&&` chain: a failing link skips every later suite while the step still fails (measured in #1270 — four suites after position 44 had not run since a prior merge). The enumeration guard proves naming, not reachability; both residuals are in-file documented but structurally unfixed.
|
||||
9. **Signed-exclusion burndown open.** 16 signed exclusions remain in `test-enumeration-exclusions.txt`; several are "unmeasured in CI image" or blocked on missing CI tooling (tmux, setsid) — tracked under #1017/#1271. Each is an enforcement promise deferred, not delivered.
|
||||
10. **Windows twins unexercised.** `verify.ps1`, `install.ps1`, `mosaic-doctor.ps1` have no runner anywhere (no Windows CI); behavioral drift from their bash twins is undetectable by construction.
|
||||
11. **QA hook name vs behavior.** `qa-hook-handler.sh` files remediation report templates but performs no verification; the seam's actual gate value is only the deps-preflight sentinel. Anything relying on "QA automation hook" as a check is relying on report-filing.
|
||||
12. **Two test paths, one gated.** CI runs tests against ci-postgres (`DATABASE_URL` set); the local PGlite path is the documented default (AGENTS.md) until KBN-101-02/101-05. Only the CI path is enforced by pipeline.
|
||||
|
||||
## 4. Disposition summary
|
||||
|
||||
| disposition | rows | checks |
|
||||
| ------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| preserve | 43 | Every canonical owner (QC-1..QC-18) plus correct guards-of-the-guard and thin adapters: all of §1.1, the CI-invoked framework probes and adapters in §1.2, all of §1.3, the C1/C2/C5 trio and doctors in §1.4, all of §1.5, all pipeline-only steps in §1.6, §1.7 rows 1 and 3, and §1.8. |
|
||||
| strengthen | 2 | `quality-rails check` and `quality-rails doctor` (QC-19) — the RI-N4 evaluator seed: typed verdicts, versioned/digested check definitions, per-subject check sets. |
|
||||
| strengthen (review) | 9 | `verify.sh` + `verify.ps1` (QC-20), quality `install.sh`/`install.ps1` + `quality-rails init` (QC-21 — scaffold-path convergence), `test-roster-schema.py` (QC-5 — wire or sign), `qa-hook-stdin.sh` seam + `typecheck-hook.sh` (QC-16), `tools/e2e-install-test.sh` (QC-5 — CI lane). |
|
||||
| retire | 0 | None meet the bar: RI-N4 requires effective shell probes be **absorbed before** their paths retire, and no absorption exists yet. The `strengthen (review)` rows are the retirement candidates for RI-3-002 once the evaluator owns their behavior. |
|
||||
|
||||
Row total: 54. Canonical checks: 21 (QC-1..QC-21).
|
||||
@@ -233,8 +233,36 @@ assert_owned_tmux_server() {
|
||||
fail "tmux server ownership or environment validation failed"
|
||||
}
|
||||
|
||||
# Validate exact server ownership before querying, cleaning, or creating any
|
||||
# managed session. An unmanaged or contaminated named socket is never repaired.
|
||||
# Lease-broker socket preflight (#1292). The gated runtime (`mosaic yolo …` →
|
||||
# launch-runtime.py) registers with the broker or dies ~4 seconds in, with the
|
||||
# diagnostic invisible because tmux destroys the dead pane. This check runs
|
||||
# BEFORE any tmux effect — including the ownership probe below — so a host
|
||||
# without a broker produces a named, surviving refusal instead of a doomed
|
||||
# pane. Exit 75 (EX_TEMPFAIL), distinct from 64 (bad projection) and 69 (host
|
||||
# not ready for other reasons); the agent@ unit is Type=oneshot with no
|
||||
# Restart=, so the failed unit keeps its message instead of looping. Socket
|
||||
# resolution matches launch.ts's defaultLeaseBrokerSocket precedence exactly.
|
||||
# This preflight DETECTS and REFUSES — it never starts the broker (activation
|
||||
# belongs to the fleet control plane; a component that both detects and fixes
|
||||
# cannot be used to measure whether the fix worked).
|
||||
broker_socket_path() {
|
||||
if [ -n "${MOSAIC_LEASE_BROKER_SOCKET:-}" ]; then
|
||||
printf '%s\n' "$MOSAIC_LEASE_BROKER_SOCKET"
|
||||
return 0
|
||||
fi
|
||||
local runtime_dir="${XDG_RUNTIME_DIR:-/run/user/$(id -u)}"
|
||||
printf '%s\n' "${runtime_dir}/mosaic-lease/broker.sock"
|
||||
}
|
||||
|
||||
if [ "$MODE" = "launch" ]; then
|
||||
_broker_socket=$(broker_socket_path)
|
||||
if [ ! -S "$_broker_socket" ]; then
|
||||
echo "[fleet] FAIL_LAUNCH broker-absent: lease broker socket ${_broker_socket} missing; runtime launch denied (#1292)." >&2
|
||||
echo "[fleet] remedy: systemctl --user enable --now mosaic-lease-broker.service (or reinstall via: mosaic fleet install)" >&2
|
||||
exit 75
|
||||
fi
|
||||
fi
|
||||
|
||||
assert_owned_tmux_server
|
||||
|
||||
if [ "$MODE" = interaction ]; then
|
||||
|
||||
+216
@@ -0,0 +1,216 @@
|
||||
#!/usr/bin/env bash
|
||||
# CI-fit regression suite for the #1292 lease-broker socket preflight in
|
||||
# start-agent-session.sh.
|
||||
#
|
||||
# WHY THIS SUITE IS CI-FIT WHERE test-start-agent-session.sh IS NOT (#1017/#1270
|
||||
# context): that older suite's precondition is "the host does not have the pi
|
||||
# binary", which a CI image that ships pi violates — its guard correctly
|
||||
# refuses to report a pass there, so it is excluded from the chain. THIS suite
|
||||
# controls its own preconditions instead of inheriting them from the host: a
|
||||
# fake tmux on PATH, a fake mosaic on PATH, a real unix socket created in a
|
||||
# tmpdir, a hermetic env (env -i, fake HOME, GIT_CONFIG_GLOBAL severed). It
|
||||
# never depends on what the host has installed, so a green here means the same
|
||||
# thing on every host. Anyone adding cases: keep that property — no case may
|
||||
# depend on host state.
|
||||
#
|
||||
# The failure this suite is written down to catch (#1292): a seat launched on a
|
||||
# host with no lease broker dies ~4 seconds in at registration, with the
|
||||
# diagnostic invisible because tmux destroys the dead pane. The preflight runs
|
||||
# BEFORE any tmux effect and refuses with a NAMED code (exit 75, EX_TEMPFAIL)
|
||||
# so the message survives. The agent@ unit is Type=oneshot with no Restart=,
|
||||
# so a failed unit keeps its output instead of looping.
|
||||
#
|
||||
# Cases:
|
||||
# 1. absent socket -> exit 75, message names broker-absent + socket path +
|
||||
# remedy, and NO tmux session was ever created (the doomed-pane half).
|
||||
# 2. present socket (real unix socket in tmpdir) -> proceeds PAST the
|
||||
# preflight (the suite then stops at the next precondition, proving the
|
||||
# preflight was not the refusal).
|
||||
# 3. explicit MOSAIC_LEASE_BROKER_SOCKET wins over XDG_RUNTIME_DIR default.
|
||||
# 4. --stop mode does NOT require the broker (teardown must not be fenced on
|
||||
# a component whose absence is exactly what teardown may follow).
|
||||
#
|
||||
# Sabotage control, run by the developer (not in-suite): remove the preflight
|
||||
# block from start-agent-session.sh, re-run — case 1 fails (a tmux session is
|
||||
# created / exit is not 75), cases 2-4 still pass; restore byte-identically.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/agent-session-broker-preflight}"
|
||||
FAKE_HOME="$WORK_DIR/home"
|
||||
BIN_DIR="$WORK_DIR/bin"
|
||||
ENV_DIR="$WORK_DIR/env"
|
||||
SOCK_DIR="$WORK_DIR/sockets"
|
||||
LOG_FILE="$WORK_DIR/tmux-calls.log"
|
||||
|
||||
rm -rf "$WORK_DIR"
|
||||
# The script asserts a managed directory tree under MOSAIC_HOME: mosaic/,
|
||||
# mosaic/fleet/, mosaic/fleet/agents/ — private (0700/0750-style) modes, no
|
||||
# symlinks — plus a per-agent env projection. Build the full tree the launcher
|
||||
# expects so the suite reaches the BROKER preflight rather than dying at
|
||||
# environment validation.
|
||||
mkdir -p "$FAKE_HOME/.config/mosaic/fleet/agents" "$BIN_DIR" "$SOCK_DIR"
|
||||
chmod 700 "$FAKE_HOME/.config/mosaic" "$FAKE_HOME/.config/mosaic/fleet/agents"
|
||||
chmod 750 "$FAKE_HOME/.config/mosaic/fleet"
|
||||
cat > "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated" <<'ENVEOF'
|
||||
MOSAIC_AGENT_NAME=preflight-test
|
||||
MOSAIC_AGENT_CLASS=worker
|
||||
MOSAIC_AGENT_RUNTIME=pi
|
||||
MOSAIC_AGENT_MODEL=
|
||||
MOSAIC_AGENT_REASONING=
|
||||
MOSAIC_AGENT_TOOL_POLICY=code
|
||||
MOSAIC_AGENT_WORKDIR=/tmp
|
||||
MOSAIC_TMUX_SOCKET=mosaic-fleet
|
||||
ENVEOF
|
||||
chmod 600 "$FAKE_HOME/.config/mosaic/fleet/agents/preflight-test.env.generated"
|
||||
|
||||
# ─── Fake tmux: records every invocation; new-session marks the marker. ────
|
||||
: > "$LOG_FILE"
|
||||
cat > "$BIN_DIR/tmux" <<SH
|
||||
#!/usr/bin/env bash
|
||||
printf 'tmux %s\n' "\$*" >> "$LOG_FILE"
|
||||
if [[ "\$*" == *new-session* ]]; then
|
||||
echo "TMUX-NEW-SESSION-INVOKED" >> "$LOG_FILE"
|
||||
fi
|
||||
exit 0
|
||||
SH
|
||||
chmod +x "$BIN_DIR/tmux"
|
||||
|
||||
# ─── Fake mosaic/pi binaries so the script proceeds past its own lookups. ───
|
||||
for bin in mosaic pi claude; do
|
||||
printf '#!/usr/bin/env bash\nexit 0\n' > "$BIN_DIR/$bin"
|
||||
chmod +x "$BIN_DIR/$bin"
|
||||
done
|
||||
|
||||
# ─── Minimal launch environment the script expects. ────────────────────────
|
||||
# (Enough for the preflight to be reached; later stages will still fail in
|
||||
# case 2 — that is expected and asserted.)
|
||||
run_session_script() {
|
||||
local mode="$1"; shift
|
||||
(
|
||||
cd "$WORK_DIR"
|
||||
env -i HOME="$FAKE_HOME" PATH="$BIN_DIR:/usr/bin:/bin" \
|
||||
GIT_CONFIG_GLOBAL=/dev/null GIT_CONFIG_SYSTEM=/dev/null \
|
||||
MOSAIC_HOME="$FAKE_HOME/.config/mosaic" \
|
||||
AGENT_NAME=preflight-test \
|
||||
"$@" \
|
||||
bash "$SCRIPT_DIR/start-agent-session.sh" $mode preflight-test
|
||||
)
|
||||
}
|
||||
|
||||
fail=0
|
||||
assert() {
|
||||
local desc="$1" expected="$2" actual="$3"
|
||||
if [[ "$expected" != "$actual" ]]; then
|
||||
echo "FAIL: $desc — expected '$expected', got '$actual'" >&2
|
||||
fail=1
|
||||
fi
|
||||
}
|
||||
assert_contains() {
|
||||
local desc="$1" haystack="$2" needle="$3"
|
||||
[[ "$haystack" == *"$needle"* ]] || { echo "FAIL: $desc — missing '$needle' in: $haystack" >&2; fail=1; }
|
||||
}
|
||||
assert_not_contains() {
|
||||
local desc="$1" haystack="$2" needle="$3"
|
||||
if [[ "$haystack" == *"$needle"* ]]; then
|
||||
echo "FAIL: $desc — must not contain '$needle'" >&2
|
||||
fail=1
|
||||
fi
|
||||
return 0
|
||||
}
|
||||
|
||||
# ─── 1. Absent socket → named refusal, NO tmux session. ────────────────────
|
||||
: > "$LOG_FILE"
|
||||
stderr_file="$WORK_DIR/stderr-1.tmp"
|
||||
set +e
|
||||
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent.sock" 2>"$stderr_file")
|
||||
rc=$?
|
||||
set -e
|
||||
assert "absent socket exit code" "75" "$rc"
|
||||
err=$(cat "$stderr_file")
|
||||
assert_contains "absent socket names the failure" "$err" "FAIL_LAUNCH broker-absent"
|
||||
assert_contains "absent socket names the socket path" "$err" "$SOCK_DIR/absent.sock"
|
||||
assert_contains "absent socket names a remedy" "$err" "mosaic fleet install"
|
||||
log1=$(cat "$LOG_FILE")
|
||||
assert_not_contains "absent socket must not create a tmux session" "$log1" "TMUX-NEW-SESSION-INVOKED"
|
||||
|
||||
# ─── 2. Present socket → passes the preflight. ─────────────────────────────
|
||||
# Expected: ownership/env checks AFTER the preflight may refuse (fixture is
|
||||
# minimal by design); the assertion is only that the refusal is NOT
|
||||
# broker-absent and the exit is NOT 75.
|
||||
# Create a REAL unix socket: a detached python holder binds it and stays alive
|
||||
# for the duration (bash cannot create sockets; a foreground python would
|
||||
# close the socket on exit and -S on a closed-but-unlinked path fails). Written
|
||||
# as a script file + setsid nohup so no job-control/heredoc interaction with
|
||||
# set -e can silently kill the suite.
|
||||
# AF_UNIX binds cap at 108 path bytes; the suite's workdir exceeds that, so
|
||||
# the live socket lives at a SHORT path under /tmp (unique per run, cleaned
|
||||
# with the suite). The preflight takes its socket path explicitly, so this
|
||||
# stays fully controlled.
|
||||
LIVE_SOCK=$(mktemp -u /tmp/mosaic-preflight-XXXXXX.sock)
|
||||
trap 'rm -f "$LIVE_SOCK"' EXIT
|
||||
rm -f "$SOCK_DIR/live.sock" "$LIVE_SOCK"
|
||||
cat > "$SOCK_DIR/holder.py" <<'PY'
|
||||
import socket, sys, time
|
||||
path = sys.argv[1]
|
||||
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
s.bind(path)
|
||||
s.listen(1)
|
||||
time.sleep(120)
|
||||
PY
|
||||
python3 "$SOCK_DIR/holder.py" "$LIVE_SOCK" >/dev/null 2>"$SOCK_DIR/holder.err" &
|
||||
HOLDER_PID=$!
|
||||
# Wait for the socket object to exist (bind is near-instant, but do not race it).
|
||||
for _ in $(seq 1 50); do
|
||||
[ -S "$LIVE_SOCK" ] && break
|
||||
sleep 0.1
|
||||
done
|
||||
if [ ! -S "$LIVE_SOCK" ]; then
|
||||
echo "FAIL: could not create live socket fixture (holder pid $HOLDER_PID)" >&2
|
||||
ps -p "$HOLDER_PID" -o pid,stat,cmd --no-headers >&2 || echo "(holder exited)" >&2
|
||||
cat "$SOCK_DIR/holder.err" >&2 || true
|
||||
exit 1
|
||||
fi
|
||||
: > "$LOG_FILE"
|
||||
set +e
|
||||
out=$(run_session_script "" MOSAIC_LEASE_BROKER_SOCKET="$LIVE_SOCK" 2>"$WORK_DIR/stderr-2.tmp")
|
||||
rc=$?
|
||||
set -e
|
||||
# The preflight PASSED if the failure (whatever later stage refused) is NOT
|
||||
# the broker refusal, and tmux was reached or a later precondition named
|
||||
# something else.
|
||||
err2=$(cat "$WORK_DIR/stderr-2.tmp")
|
||||
assert_not_contains "live socket must not refuse broker-absent" "$err2" "broker-absent"
|
||||
if [[ "$rc" == "75" ]]; then
|
||||
echo "FAIL: live socket — preflight still refused (exit 75) with a live socket" >&2
|
||||
fail=1
|
||||
fi
|
||||
|
||||
# ─── 3. Explicit socket env wins over XDG default. ─────────────────────────
|
||||
set +e
|
||||
out=$(run_session_script "" XDG_RUNTIME_DIR="$SOCK_DIR/no-runtime-here" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent2.sock" 2>"$WORK_DIR/stderr-3.tmp")
|
||||
rc=$?
|
||||
set -e
|
||||
assert "explicit env wins (exit 75)" "75" "$rc"
|
||||
assert_contains "explicit env path named" "$(cat "$WORK_DIR/stderr-3.tmp")" "$SOCK_DIR/absent2.sock"
|
||||
|
||||
# ─── 4. --stop is not fenced on the broker. ────────────────────────────────
|
||||
: > "$LOG_FILE"
|
||||
set +e
|
||||
out=$(run_session_script "--stop" MOSAIC_LEASE_BROKER_SOCKET="$SOCK_DIR/absent3.sock" 2>"$WORK_DIR/stderr-4.tmp")
|
||||
rc=$?
|
||||
set -e
|
||||
err4=$(cat "$WORK_DIR/stderr-4.tmp")
|
||||
assert_not_contains "--stop must not refuse broker-absent" "$err4" "broker-absent"
|
||||
if [[ "$rc" == "75" ]]; then
|
||||
echo "FAIL: --stop — exit 75 means teardown was fenced on the broker" >&2
|
||||
fail=1
|
||||
fi
|
||||
|
||||
kill "$HOLDER_PID" 2>/dev/null || true
|
||||
|
||||
if [[ "$fail" -eq 0 ]]; then
|
||||
echo "start-agent-session lease-broker preflight regression passed"
|
||||
fi
|
||||
exit "$fail"
|
||||
@@ -0,0 +1,177 @@
|
||||
import { lstat, mkdir, mkdtemp, readFile, rm, symlink, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { afterEach, describe, expect, it } from 'vitest';
|
||||
|
||||
import { placeUnitFile, resolveLeaseBrokerSocketForPreflight } from './fleet.js';
|
||||
|
||||
/**
|
||||
* Unit-placement regression harness for #1292.
|
||||
*
|
||||
* The two measured defects this suite pins:
|
||||
* 1. `systemctl enable <name>` does NOT rewrite an existing by-path
|
||||
* wants-symlink — so placement must remove stale residue explicitly, and
|
||||
* acceptance asserts on the RESULTING SYMLINK TARGET, never on the enable
|
||||
* call's argument (asserting the call cannot see where the link ended up).
|
||||
* 2. Node's copyFile FOLLOWS a by-path symlink at the destination and
|
||||
* overwrites the SEED template. Acceptance asserts on the SEED's bytes
|
||||
* AND mtime — unchanged — which is the only check that can redden for
|
||||
* finding 2. The symlink-target assertion catches finding 1; these are
|
||||
* different defects with different failure modes.
|
||||
*
|
||||
* Fixtures are entirely inside tmpdirs (source template, active systemd dir,
|
||||
* wants dir) — no real host paths are touched by this suite.
|
||||
*/
|
||||
|
||||
describe('placeUnitFile (#1292 unit placement)', () => {
|
||||
const cleanup: string[] = [];
|
||||
afterEach(async () => {
|
||||
while (cleanup.length > 0) {
|
||||
await rm(cleanup.pop()!, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
async function fixture() {
|
||||
const root = await mkdtemp(join(tmpdir(), 'place-unit-'));
|
||||
cleanup.push(root);
|
||||
const seedDir = join(root, 'seed');
|
||||
const activeDir = join(root, 'active');
|
||||
await mkdir(seedDir, { recursive: true });
|
||||
await mkdir(activeDir, { recursive: true });
|
||||
const seedTemplate = join(seedDir, 'unit-under-test.service');
|
||||
await writeFile(
|
||||
seedTemplate,
|
||||
'[Unit]\nDescription=seed template\n[Service]\nType=oneshot\nExecStart=/bin/true\n[Install]\nWantedBy=default.target\n',
|
||||
);
|
||||
const activeSource = join(root, 'active-source.service');
|
||||
await writeFile(
|
||||
activeSource,
|
||||
'[Unit]\nDescription=active copy v2\n[Service]\nType=oneshot\nExecStart=/bin/true\n[Install]\nWantedBy=default.target\n',
|
||||
);
|
||||
return { root, seedDir, activeDir, seedTemplate, activeSource };
|
||||
}
|
||||
|
||||
it('places a regular file on a clean host (negative control: no residue anywhere)', async () => {
|
||||
const f = await fixture();
|
||||
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||
expect(result.unlinkedDestinationSymlink).toBe(false);
|
||||
expect(result.removedStaleWantsSymlink).toBe(false);
|
||||
const info = await lstat(join(f.activeDir, 'unit-under-test.service'));
|
||||
expect(info.isSymbolicLink()).toBe(false);
|
||||
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toContain(
|
||||
'active copy v2',
|
||||
);
|
||||
// Seed untouched by construction — but assert it, so the clean-host case
|
||||
// cannot silently regress into seed-mutation.
|
||||
expect(await readFile(f.seedTemplate, 'utf8')).toContain('seed template');
|
||||
});
|
||||
|
||||
it('by-path residue: unlinks destination symlink, places the file, seed bytes AND mtime unchanged (finding 2)', async () => {
|
||||
const f = await fixture();
|
||||
const seedBefore = await readFile(f.seedTemplate, 'utf8');
|
||||
const mtimeBefore = (await lstat(f.seedTemplate)).mtimeMs;
|
||||
// The fomo-lin convention: by-path enable left a symlink AT the unit name
|
||||
// pointing at the seed template, plus a wants-symlink doing the same.
|
||||
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
|
||||
const wantsDir = join(f.activeDir, 'default.target.wants');
|
||||
await mkdir(wantsDir, { recursive: true });
|
||||
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
|
||||
|
||||
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||
expect(result.unlinkedDestinationSymlink).toBe(true);
|
||||
expect(result.removedStaleWantsSymlink).toBe(true);
|
||||
|
||||
// FINDING 2's check: the seed is byte-identical and its mtime did not move.
|
||||
expect(await readFile(f.seedTemplate, 'utf8')).toBe(seedBefore);
|
||||
expect((await lstat(f.seedTemplate)).mtimeMs).toBe(mtimeBefore);
|
||||
|
||||
// The destination is now a regular file carrying the ACTIVE content.
|
||||
const destInfo = await lstat(join(f.activeDir, 'unit-under-test.service'));
|
||||
expect(destInfo.isSymbolicLink()).toBe(false);
|
||||
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toContain(
|
||||
'active copy v2',
|
||||
);
|
||||
});
|
||||
|
||||
it('by-path residue: no wants-symlink remains pointing at the seed (finding 1 residue cleared)', async () => {
|
||||
const f = await fixture();
|
||||
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
|
||||
const wantsDir = join(f.activeDir, 'default.target.wants');
|
||||
await mkdir(wantsDir, { recursive: true });
|
||||
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
|
||||
|
||||
await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||
|
||||
// After placement the stale wants link is GONE (enable-by-name recreates
|
||||
// it correctly). A link still present must not point at the seed.
|
||||
try {
|
||||
const link = await lstat(join(wantsDir, 'unit-under-test.service'));
|
||||
if (link.isSymbolicLink()) {
|
||||
const target = await readFile(join(wantsDir, 'unit-under-test.service'), 'utf8').catch(
|
||||
async () => '',
|
||||
);
|
||||
expect(target).not.toContain('seed template');
|
||||
}
|
||||
} catch {
|
||||
// absent wants link — the expected post-placement state
|
||||
}
|
||||
});
|
||||
|
||||
it('idempotence: second placement on a reconciled host is a no-op producing the identical final state', async () => {
|
||||
const f = await fixture();
|
||||
// Reconciled starting state: regular file at the name, wants link to the active copy.
|
||||
await writeFile(
|
||||
join(f.activeDir, 'unit-under-test.service'),
|
||||
await readFile(f.activeSource, 'utf8'),
|
||||
);
|
||||
const wantsDir = join(f.activeDir, 'default.target.wants');
|
||||
await mkdir(wantsDir, { recursive: true });
|
||||
await symlink(
|
||||
join(f.activeDir, 'unit-under-test.service'),
|
||||
join(wantsDir, 'unit-under-test.service'),
|
||||
);
|
||||
const before = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
|
||||
|
||||
const result = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||
// No destructive step fired: no unlink, no wants removal.
|
||||
expect(result.unlinkedDestinationSymlink).toBe(false);
|
||||
expect(result.removedStaleWantsSymlink).toBe(false);
|
||||
// Identical final state.
|
||||
expect(await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8')).toBe(before);
|
||||
const link = await lstat(join(wantsDir, 'unit-under-test.service'));
|
||||
expect(link.isSymbolicLink()).toBe(true);
|
||||
});
|
||||
|
||||
it('double install on by-path residue converges to the identical reconciled state', async () => {
|
||||
const f = await fixture();
|
||||
await symlink(f.seedTemplate, join(f.activeDir, 'unit-under-test.service'));
|
||||
const wantsDir = join(f.activeDir, 'default.target.wants');
|
||||
await mkdir(wantsDir, { recursive: true });
|
||||
await symlink(f.seedTemplate, join(wantsDir, 'unit-under-test.service'));
|
||||
|
||||
await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||
const first = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
|
||||
const secondRun = await placeUnitFile(f.activeSource, f.activeDir, 'unit-under-test.service');
|
||||
const second = await readFile(join(f.activeDir, 'unit-under-test.service'), 'utf8');
|
||||
expect(secondRun.unlinkedDestinationSymlink).toBe(false);
|
||||
expect(second).toBe(first);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveLeaseBrokerSocketForPreflight (#1292 preflight resolution)', () => {
|
||||
it('explicit MOSAIC_LEASE_BROKER_SOCKET wins', () => {
|
||||
expect(
|
||||
resolveLeaseBrokerSocketForPreflight({ MOSAIC_LEASE_BROKER_SOCKET: '/custom/sock' }, 1000),
|
||||
).toBe('/custom/sock');
|
||||
});
|
||||
it('XDG_RUNTIME_DIR next', () => {
|
||||
expect(resolveLeaseBrokerSocketForPreflight({ XDG_RUNTIME_DIR: '/run/user/1001' }, 1000)).toBe(
|
||||
'/run/user/1001/mosaic-lease/broker.sock',
|
||||
);
|
||||
});
|
||||
it('falls back to /run/user/<uid>', () => {
|
||||
expect(resolveLeaseBrokerSocketForPreflight({}, 1002)).toBe(
|
||||
'/run/user/1002/mosaic-lease/broker.sock',
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -835,13 +835,25 @@ describe('fleet command construction', () => {
|
||||
};
|
||||
const program = new Command();
|
||||
program.exitOverride();
|
||||
registerFleetCommand(program, { runner, mosaicHome: home });
|
||||
// #1292: inject a present broker socket so the preflight passes and this
|
||||
// spec keeps testing its ORIGINAL property (holder-before-agent ordering).
|
||||
// The preflight's own refusal behavior has dedicated specs below.
|
||||
registerFleetCommand(program, {
|
||||
runner,
|
||||
mosaicHome: home,
|
||||
checkBrokerSocket: async () => true,
|
||||
});
|
||||
|
||||
try {
|
||||
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
|
||||
await program.parseAsync(['node', 'mosaic', 'fleet', 'stop']);
|
||||
|
||||
expect(calls).toEqual([
|
||||
// #1292: fleet start enables + starts the broker FIRST (enable is
|
||||
// idempotent; the unit exists after install), re-checking the socket
|
||||
// before any holder/agent lifecycle effect.
|
||||
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
|
||||
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
|
||||
['systemctl', '--user', 'start', 'mosaic-tmux-holder.service'],
|
||||
['systemctl', '--user', 'start', '[email protected]'],
|
||||
['systemctl', '--user', 'stop', '[email protected]'],
|
||||
@@ -852,6 +864,92 @@ describe('fleet command construction', () => {
|
||||
}
|
||||
});
|
||||
|
||||
it('fleet start refuses with a named error when the broker socket does not appear (#1292)', async () => {
|
||||
const home = await tempDir();
|
||||
const rosterPath = join(home, 'fleet', 'roster.yaml');
|
||||
await mkdir(join(home, 'fleet'), { recursive: true });
|
||||
await writeFile(
|
||||
rosterPath,
|
||||
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
|
||||
'\n',
|
||||
),
|
||||
);
|
||||
const calls: string[][] = [];
|
||||
const runner: CommandRunner = async (command, args) => {
|
||||
calls.push([command, ...args]);
|
||||
return { stdout: '', stderr: '', exitCode: 0 };
|
||||
};
|
||||
const program = new Command();
|
||||
program.exitOverride();
|
||||
const errors: string[] = [];
|
||||
const origError = console.error;
|
||||
console.error = (...args: unknown[]) => {
|
||||
errors.push(args.join(' '));
|
||||
};
|
||||
registerFleetCommand(program, {
|
||||
runner,
|
||||
mosaicHome: home,
|
||||
checkBrokerSocket: async () => false,
|
||||
});
|
||||
try {
|
||||
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
|
||||
// Refused: no holder/agent starts were issued after the broker attempt.
|
||||
expect(calls).toEqual([
|
||||
['systemctl', '--user', 'enable', 'mosaic-lease-broker.service'],
|
||||
['systemctl', '--user', 'start', 'mosaic-lease-broker.service'],
|
||||
]);
|
||||
expect(errors.join('\n')).toContain('broker-absent');
|
||||
expect(errors.join('\n')).toContain('mosaic fleet install');
|
||||
} finally {
|
||||
console.error = origError;
|
||||
await rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('fleet start re-probes the broker on the SECOND invocation — no ActiveState trust (#1292 sticky half)', async () => {
|
||||
const home = await tempDir();
|
||||
const rosterPath = join(home, 'fleet', 'roster.yaml');
|
||||
await mkdir(join(home, 'fleet'), { recursive: true });
|
||||
await writeFile(
|
||||
rosterPath,
|
||||
['version: 1', 'transport: tmux', 'agents:', ' - name: coder0', ' runtime: codex'].join(
|
||||
'\n',
|
||||
),
|
||||
);
|
||||
const calls: string[][] = [];
|
||||
const runner: CommandRunner = async (command, args) => {
|
||||
calls.push([command, ...args]);
|
||||
return { stdout: '', stderr: '', exitCode: 0 };
|
||||
};
|
||||
const program = new Command();
|
||||
program.exitOverride();
|
||||
// Broker socket NEVER appears — the second start must refuse exactly like
|
||||
// the first; RemainAfterExit-style stale unit state changes nothing
|
||||
// because the check is the socket, not systemctl.
|
||||
registerFleetCommand(program, {
|
||||
runner,
|
||||
mosaicHome: home,
|
||||
checkBrokerSocket: async () => false,
|
||||
});
|
||||
const errors: string[] = [];
|
||||
const origError = console.error;
|
||||
console.error = (...args: unknown[]) => {
|
||||
errors.push(args.join(' '));
|
||||
};
|
||||
try {
|
||||
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
|
||||
await program.parseAsync(['node', 'mosaic', 'fleet', 'start']);
|
||||
// Two invocations, each refusing after its own broker attempt:
|
||||
expect(
|
||||
calls.filter((c) => c.join(' ') === 'systemctl --user start [email protected]'),
|
||||
).toHaveLength(0);
|
||||
expect(errors.filter((e) => e.includes('broker-absent')).length).toBeGreaterThanOrEqual(2);
|
||||
} finally {
|
||||
console.error = origError;
|
||||
await rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('waits for an in-flight restart to clear before relaunching (re-entry guard)', async () => {
|
||||
const home = await tempDir();
|
||||
const rosterPath = join(home, 'fleet', 'roster.yaml');
|
||||
@@ -2065,8 +2163,19 @@ describe('fleet install — auto-enable units for boot-survival', () => {
|
||||
|
||||
await enableFleetUnits(runner, minimalRoster, {});
|
||||
|
||||
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-lease-broker.service']);
|
||||
expect(calls).toContainEqual(['systemctl', '--user', 'enable', 'mosaic-tmux-holder.service']);
|
||||
expect(calls).toContainEqual(['systemctl', '--user', 'enable', '[email protected]']);
|
||||
// The broker must be enabled BEFORE the holder and agents: a start of any
|
||||
// gated runtime without the broker is exactly the #1292 4-second death.
|
||||
const brokerIndex = calls.findIndex(
|
||||
(c) => c.join(' ') === 'systemctl --user enable mosaic-lease-broker.service',
|
||||
);
|
||||
const holderIndex = calls.findIndex(
|
||||
(c) => c.join(' ') === 'systemctl --user enable mosaic-tmux-holder.service',
|
||||
);
|
||||
expect(brokerIndex).toBeGreaterThanOrEqual(0);
|
||||
expect(brokerIndex).toBeLessThan(holderIndex);
|
||||
});
|
||||
|
||||
it('install still succeeds when systemctl enable returns non-zero (non-fatal)', async () => {
|
||||
|
||||
@@ -3,9 +3,11 @@ import {
|
||||
access,
|
||||
chmod,
|
||||
copyFile,
|
||||
lstat,
|
||||
mkdir,
|
||||
open,
|
||||
readFile,
|
||||
readlink,
|
||||
stat,
|
||||
unlink,
|
||||
writeFile,
|
||||
@@ -89,6 +91,8 @@ export type SleepFn = (ms: number) => Promise<void>;
|
||||
|
||||
export interface FleetCommandDeps {
|
||||
runner?: CommandRunner;
|
||||
/** Test seam for the #1292 fleet-start broker preflight (socket presence). */
|
||||
checkBrokerSocket?: (path: string) => Promise<boolean> | boolean;
|
||||
/** Injectable interactive runner for commands needing inherited TTY (e.g., `tmux attach`). */
|
||||
interactiveRunner?: InteractiveRunner;
|
||||
/**
|
||||
@@ -797,6 +801,96 @@ export function buildSystemdEnableCommand(unit: string): string[] {
|
||||
return ['systemctl', '--user', 'enable', unit];
|
||||
}
|
||||
|
||||
/**
|
||||
* Place a unit file into the ACTIVE systemd user directory, never through a
|
||||
* symlink (#1292, measured 2026-08-17).
|
||||
*
|
||||
* ⚠ SET-INDEPENDENCE (fomo-lin, 2026-08-17): the set of unit names carrying
|
||||
* by-path residue and the set of unit names this install copies are
|
||||
* INDEPENDENT. Until 0.0.50 they were disjoint only by accident of which
|
||||
* units the install happened to name — fomo-lin survived copy-through solely
|
||||
* because its one by-path symlink (the broker) was the one unit the install
|
||||
* did NOT copy. Adding the broker to the copy set made the intersection
|
||||
* non-empty on the first run. Whoever adds a fifth unit to the placement
|
||||
* list inherits this helper and its unlink step; do not place units with a
|
||||
* bare copyFile.
|
||||
*
|
||||
* A host provisioned by the enable-by-path convention carries a symlink AT
|
||||
* the unit-name path in ~/.config/systemd/user/ pointing at the shipped
|
||||
* template under ~/.config/mosaic/systemd/user/. Node's copyFile FOLLOWS
|
||||
* that link and overwrites the SEED template instead of placing the active
|
||||
* unit (verified with fs.copyFile on a throwaway systemd user instance) —
|
||||
* silent, rc=0, and it mutates the directory every later reseed reads from.
|
||||
* The same measurement showed `systemctl enable <name>` does NOT rewrite an
|
||||
* existing by-path wants-symlink, so reconciliation must be explicit.
|
||||
*
|
||||
* Placement therefore: if the destination is a symlink, unlink it first
|
||||
* (unlink → copy — copy-then-unlink would mutate the seed and then destroy
|
||||
* the evidence that it did); then copy. Also removes a stale
|
||||
* `default.target.wants/<name>` symlink that points outside the active
|
||||
* directory (readlink — NOT readFile, which follows the link and returns the
|
||||
* target's CONTENT), so the subsequent enable-by-name recreates it against
|
||||
* the active copy. Idempotent: on a clean or already-reconciled destination
|
||||
* every step is a no-op (the copy rewrites identical bytes).
|
||||
*
|
||||
* Returns what was done, for assertions and install reporting.
|
||||
*/
|
||||
export interface PlaceUnitResult {
|
||||
readonly unit: string;
|
||||
readonly destination: string;
|
||||
/** A symlink at the unit-name path was unlinked (by-path residue). */
|
||||
readonly unlinkedDestinationSymlink: boolean;
|
||||
/** A stale wants-symlink pointing outside the active dir was removed. */
|
||||
readonly removedStaleWantsSymlink: boolean;
|
||||
}
|
||||
|
||||
export async function placeUnitFile(
|
||||
source: string,
|
||||
systemdUserDir: string,
|
||||
unit: string,
|
||||
): Promise<PlaceUnitResult> {
|
||||
const destination = join(systemdUserDir, unit);
|
||||
let unlinkedDestinationSymlink = false;
|
||||
try {
|
||||
const destInfo = await lstat(destination);
|
||||
if (destInfo.isSymbolicLink()) {
|
||||
await unlink(destination);
|
||||
unlinkedDestinationSymlink = true;
|
||||
}
|
||||
} catch {
|
||||
// absent destination — nothing to unlink
|
||||
}
|
||||
await copyFile(source, destination);
|
||||
|
||||
let removedStaleWantsSymlink = false;
|
||||
const wantsLink = join(systemdUserDir, 'default.target.wants', unit);
|
||||
try {
|
||||
const wantsInfo = await lstat(wantsLink);
|
||||
if (wantsInfo.isSymbolicLink()) {
|
||||
// readlink — NOT readFile: readFile FOLLOWS the link and returns the
|
||||
// target file's CONTENT, which is not the question being asked.
|
||||
let target: string | undefined;
|
||||
try {
|
||||
target = await readlink(wantsLink);
|
||||
} catch {
|
||||
target = undefined;
|
||||
}
|
||||
// Normalize (systemctl writes absolute targets; a relative one resolves
|
||||
// against the wants dir). A wants-symlink pointing anywhere other than
|
||||
// the active copy (the by-path convention points at the seed template)
|
||||
// survives enable-by-name unchanged — remove it so enable recreates it.
|
||||
if (target !== undefined && resolve(dirname(wantsLink), target) !== destination) {
|
||||
await unlink(wantsLink);
|
||||
removedStaleWantsSymlink = true;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// absent wants link — nothing to reconcile
|
||||
}
|
||||
|
||||
return { unit, destination, unlinkedDestinationSymlink, removedStaleWantsSymlink };
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the systemctl --user disable command for a given unit.
|
||||
* Used by `fleet remove` so a removed agent's enabled unit cannot resurrect on
|
||||
@@ -831,6 +925,22 @@ export async function enableFleetUnits(
|
||||
let succeeded = 0;
|
||||
let failed = 0;
|
||||
|
||||
// The lease broker ships with the fleet and every gated runtime needs it
|
||||
// (#1292): seats die at lease registration without it, and no documented
|
||||
// path ever enabled it. Enabled first — alongside the holder — and the
|
||||
// unit must have been placed by installFleet's placeUnitFile step.
|
||||
const brokerResult = await runner(
|
||||
...splitCommand(buildSystemdEnableCommand('mosaic-lease-broker.service')),
|
||||
);
|
||||
if (brokerResult.exitCode === 0) {
|
||||
succeeded++;
|
||||
} else {
|
||||
failed++;
|
||||
process.stderr.write(
|
||||
`Warning: could not enable mosaic-lease-broker.service: ${brokerResult.stderr || brokerResult.stdout || 'non-zero exit'}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
const holderResult = await runner(
|
||||
...splitCommand(buildSystemdEnableCommand('mosaic-tmux-holder.service')),
|
||||
);
|
||||
@@ -1527,7 +1637,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
||||
.description('Install local fleet tools and user systemd units')
|
||||
.option('--no-enable', 'Skip enabling units for boot-survival')
|
||||
.action(async (opts: { enable?: boolean }) => {
|
||||
await installFleet(cmd, frameworkRoot);
|
||||
await installFleet(cmd, frameworkRoot, runner);
|
||||
// Unit enablement needs agent names only, so it reads either version.
|
||||
const roster = await loadRosterReadModel(cmd);
|
||||
await enableFleetUnits(runner, roster, opts);
|
||||
@@ -1538,7 +1648,7 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
||||
.description('Install local fleet tools and user systemd units')
|
||||
.option('--no-enable', 'Skip enabling units for boot-survival')
|
||||
.action(async (opts: { enable?: boolean }) => {
|
||||
await installFleet(cmd, frameworkRoot);
|
||||
await installFleet(cmd, frameworkRoot, runner);
|
||||
// Unit enablement needs agent names only, so it reads either version.
|
||||
const roster = await loadRosterReadModel(cmd);
|
||||
await enableFleetUnits(runner, roster, opts);
|
||||
@@ -1591,6 +1701,37 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (action === 'start') {
|
||||
// Broker preflight (#1292), re-probed on EVERY invocation: a
|
||||
// gated runtime started without a live lease broker dies ~4s in
|
||||
// while the unit reports active (RemainAfterExit) — enabling +
|
||||
// starting here and then RE-CHECKING the socket refuses loudly
|
||||
// instead of reporting rc0 over a doomed start. This is the
|
||||
// second-start check as much as the first: it never trusts unit
|
||||
// ActiveState.
|
||||
await runChecked(runner, [
|
||||
'systemctl',
|
||||
'--user',
|
||||
'enable',
|
||||
'mosaic-lease-broker.service',
|
||||
]);
|
||||
await runChecked(runner, [
|
||||
'systemctl',
|
||||
'--user',
|
||||
'start',
|
||||
'mosaic-lease-broker.service',
|
||||
]);
|
||||
if (!(await brokerSocketPresent(deps))) {
|
||||
console.error(
|
||||
'[fleet] broker-absent: lease broker socket did not appear after enable+start (#1292).',
|
||||
);
|
||||
console.error(
|
||||
'[fleet] remedy: mosaic fleet install (it reconciles either enable convention)',
|
||||
);
|
||||
process.exitCode = 1;
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (action === 'restart') {
|
||||
// Serialize the holder+agents teardown/relaunch behind the restart lock
|
||||
// so a re-entrant restart waits for clean shutdown before relaunching,
|
||||
@@ -2349,7 +2490,11 @@ export function registerFleetAgentCommands(
|
||||
});
|
||||
}
|
||||
|
||||
async function installFleet(cmd: Command, frameworkRoot: string): Promise<void> {
|
||||
async function installFleet(
|
||||
cmd: Command,
|
||||
frameworkRoot: string,
|
||||
runner: CommandRunner,
|
||||
): Promise<void> {
|
||||
const activePaths = resolveFleetPaths(cmd.opts<{ mosaicHome: string }>().mosaicHome);
|
||||
assertDefaultMosaicHomeForSystemd(activePaths.mosaicHome);
|
||||
// Read model first: every file this function places is roster-independent, and
|
||||
@@ -2401,18 +2546,40 @@ async function installFleet(cmd: Command, frameworkRoot: string): Promise<void>
|
||||
for (const toolPath of executableToolPaths) {
|
||||
await chmod(toolPath, 0o755);
|
||||
}
|
||||
await copyFile(
|
||||
join(frameworkRoot, 'systemd', 'user', 'mosaic-tmux-holder.service'),
|
||||
join(activePaths.systemdUserDir, 'mosaic-tmux-holder.service'),
|
||||
// Unit placement (#1292): every unit goes through placeUnitFile — never a
|
||||
// bare copyFile — so a by-path-enable symlink at the destination is
|
||||
// unlinked rather than written through (copy-through would silently
|
||||
// overwrite the SEED template, measured 2026-08-17). The lease broker unit
|
||||
// is placed here too: previously the install named three units and omitted
|
||||
// the broker entirely, which is why no documented path ever enabled it.
|
||||
const placedUnits = await Promise.all(
|
||||
[
|
||||
'mosaic-tmux-holder.service',
|
||||
'[email protected]',
|
||||
'[email protected]',
|
||||
'mosaic-lease-broker.service',
|
||||
].map((unit) =>
|
||||
placeUnitFile(join(frameworkRoot, 'systemd', 'user', unit), activePaths.systemdUserDir, unit),
|
||||
),
|
||||
);
|
||||
await copyFile(
|
||||
join(frameworkRoot, 'systemd', 'user', '[email protected]'),
|
||||
join(activePaths.systemdUserDir, '[email protected]'),
|
||||
);
|
||||
await copyFile(
|
||||
join(frameworkRoot, 'systemd', 'user', '[email protected]'),
|
||||
join(activePaths.systemdUserDir, '[email protected]'),
|
||||
const reconciled = placedUnits.filter(
|
||||
(result) => result.unlinkedDestinationSymlink || result.removedStaleWantsSymlink,
|
||||
);
|
||||
if (reconciled.length > 0) {
|
||||
console.log(
|
||||
`Reconciled ${reconciled.length} unit placement(s) from by-path enable residue: ${reconciled.map((r) => r.unit).join(', ')}`,
|
||||
);
|
||||
}
|
||||
// systemd will not see a replaced unit file without a reload; do it once
|
||||
// after all placements, before any enable call below. runCommand never
|
||||
// rejects (it resolves exitCode 127 on spawn error), so a plain await with
|
||||
// an exitCode check matches the rest of this file's systemctl handling.
|
||||
const reloadResult = await runner(...splitCommand(['systemctl', '--user', 'daemon-reload']));
|
||||
if (reloadResult.exitCode !== 0) {
|
||||
process.stderr.write(
|
||||
`Warning: systemctl --user daemon-reload after unit placement failed (non-systemd host?): ${reloadResult.stderr || reloadResult.stdout || 'non-zero exit'}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
// On roster v2 the reconciler owns the generated env: `apply` writes it and
|
||||
// `regen` rebuilds it, both from projectRosterV2AgentGeneratedEnv. Writing it
|
||||
@@ -2609,6 +2776,40 @@ function splitCommand(command: string[]): [string, string[]] {
|
||||
return [bin, args];
|
||||
}
|
||||
|
||||
/**
|
||||
* Lease-broker socket presence for the fleet-start preflight (#1292).
|
||||
* Resolution precedence matches launch.ts's defaultLeaseBrokerSocket and
|
||||
* start-agent-session.sh's broker_socket_path: explicit
|
||||
* MOSAIC_LEASE_BROKER_SOCKET, else $XDG_RUNTIME_DIR/mosaic-lease/broker.sock,
|
||||
* else /run/user/<uid>/mosaic-lease/broker.sock. Pure filesystem check — this
|
||||
* deliberately does NOT consult systemd state: a unit can be active
|
||||
* (RemainAfterExit) with no live socket, and the socket is the thing the
|
||||
* gated runtime connects to. Injectable via deps for tests.
|
||||
*/
|
||||
export function resolveLeaseBrokerSocketForPreflight(
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
uid: number = typeof process.getuid === 'function' ? process.getuid() : 0,
|
||||
): string {
|
||||
if (env['MOSAIC_LEASE_BROKER_SOCKET']) return env['MOSAIC_LEASE_BROKER_SOCKET'];
|
||||
const runtimeDir = env['XDG_RUNTIME_DIR'] ?? `/run/user/${uid}`;
|
||||
return join(runtimeDir, 'mosaic-lease', 'broker.sock');
|
||||
}
|
||||
|
||||
async function brokerSocketPresent(
|
||||
deps: FleetCommandDeps,
|
||||
env: NodeJS.ProcessEnv = process.env,
|
||||
): Promise<boolean> {
|
||||
const check = deps.checkBrokerSocket;
|
||||
if (check) return check(resolveLeaseBrokerSocketForPreflight(env));
|
||||
try {
|
||||
const socketPath = resolveLeaseBrokerSocketForPreflight(env);
|
||||
await access(socketPath, constants.S_IFSOCK);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** All supported fleet profile names. */
|
||||
export type FleetProfile =
|
||||
| 'general'
|
||||
|
||||
@@ -205,6 +205,12 @@ export async function runLeaseEnforcementDoctorCheck(
|
||||
message:
|
||||
`Lease-enforcement hooks (${matchedMarkers.join(', ')}) are wired in ~/.claude/settings.json, but ${reasons.join(' and ')}. ` +
|
||||
'Every gated tool call will fail closed and BRICK this agent (see #869). ' +
|
||||
'Remediate by activating the lease-broker supervisor (systemd unit + socket) or by removing the enforcement hooks from ~/.claude/settings.json.',
|
||||
// #1292: one remedy, correct under BOTH enable conventions (by-path on
|
||||
// the seed template, and copy-then-enable in the active dir). Written
|
||||
// from the 2026-08-17 symlink measurement: `systemctl enable` by name
|
||||
// does NOT rewrite an existing by-path wants-symlink, so teaching a
|
||||
// manual systemctl line here could leave a host with two competing
|
||||
// wants links. fleet install reconciles either shape.
|
||||
'Remedy: run `mosaic fleet install` (it reconciles either enable convention), or remove the enforcement hooks from ~/.claude/settings.json.',
|
||||
};
|
||||
}
|
||||
|
||||
@@ -459,6 +459,7 @@ describe('FCM-M3-002 reconciler lifecycle acceptance', (): void => {
|
||||
plan: {
|
||||
generation: 7,
|
||||
holder: 'owned',
|
||||
broker: { unitInstalled: false, socketPresent: false },
|
||||
agents: [
|
||||
{
|
||||
name: 'coder0',
|
||||
|
||||
@@ -92,6 +92,112 @@ async function run(command: FleetReconcileCommand, overrides: Partial<FleetRecon
|
||||
}
|
||||
|
||||
describe('fleet roster-owned reconciler', (): void => {
|
||||
// ── #1292: broker as first-class plan member + broker-first start ordering ──
|
||||
|
||||
it('reports broker unit and socket state in the plan (socket is the signal, not unit state)', async (): Promise<void> => {
|
||||
const result = await run('status', {
|
||||
statPath: async () => true,
|
||||
checkBrokerSocket: async () => true,
|
||||
});
|
||||
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: true });
|
||||
});
|
||||
|
||||
it('reports a dead broker as socketPresent=false even when the unit is installed (enabled-but-dead is the #1292 shape)', async (): Promise<void> => {
|
||||
const result = await run('status', {
|
||||
statPath: async () => true,
|
||||
checkBrokerSocket: async () => false,
|
||||
});
|
||||
expect(result.plan.broker).toEqual({ unitInstalled: true, socketPresent: false });
|
||||
});
|
||||
|
||||
it('reports broker-absent when neither seam is present (defaults false, never guesses healthy)', async (): Promise<void> => {
|
||||
const result = await run('status');
|
||||
expect(result.plan.broker).toEqual({ unitInstalled: false, socketPresent: false });
|
||||
});
|
||||
|
||||
it('command start enables and starts the broker BEFORE the holder and any agent unit', async (): Promise<void> => {
|
||||
const calls: string[][] = [];
|
||||
const result = await run('start', {
|
||||
runner: async (command, args) => {
|
||||
calls.push([command, ...args]);
|
||||
if (command === 'tmux' && args.includes('list-sessions')) {
|
||||
return { stdout: '_holder\ncoder0\n', stderr: '', exitCode: 0 };
|
||||
}
|
||||
if (command === 'tmux' && args.includes('show-environment')) {
|
||||
return {
|
||||
stdout:
|
||||
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
|
||||
stderr: '',
|
||||
exitCode: 0,
|
||||
};
|
||||
}
|
||||
return { stdout: '', stderr: '', exitCode: 0 };
|
||||
},
|
||||
});
|
||||
expect(result.lifecycle).toBe('complete');
|
||||
const brokerEnable = calls.findIndex(
|
||||
(c) => c.join(' ') === 'systemctl --user enable mosaic-lease-broker.service',
|
||||
);
|
||||
const brokerStart = calls.findIndex(
|
||||
(c) => c.join(' ') === 'systemctl --user start mosaic-lease-broker.service',
|
||||
);
|
||||
const holderStart = calls.findIndex(
|
||||
(c) => c.join(' ') === 'systemctl --user start mosaic-tmux-holder.service',
|
||||
);
|
||||
const agentStart = calls.findIndex(
|
||||
(c) => c.join(' ') === 'systemctl --user start [email protected]',
|
||||
);
|
||||
expect(brokerEnable).toBeGreaterThanOrEqual(0);
|
||||
expect(brokerStart).toBeGreaterThan(brokerEnable);
|
||||
// Holder start may be absent (holder 'owned' in this fixture); if present it must follow the broker.
|
||||
if (holderStart >= 0) expect(holderStart).toBeGreaterThan(brokerStart);
|
||||
expect(agentStart).toBeGreaterThan(brokerStart);
|
||||
});
|
||||
|
||||
it('apply with a running desired agent also enables and starts the broker first', async (): Promise<void> => {
|
||||
const calls: string[][] = [];
|
||||
const runningRoster: FleetRosterV2 = {
|
||||
...roster,
|
||||
agents: roster.agents.map((agent) =>
|
||||
agent.name === 'coder0'
|
||||
? { ...agent, lifecycle: { enabled: true, desiredState: 'running' as const } }
|
||||
: agent,
|
||||
),
|
||||
};
|
||||
const result = await executeFleetReconcile({
|
||||
roster: runningRoster,
|
||||
command: 'apply',
|
||||
expectedGeneration: 7,
|
||||
deps: deps({
|
||||
readRoster: async () => runningRoster,
|
||||
runner: async (command, args) => {
|
||||
calls.push([command, ...args]);
|
||||
if (command === 'tmux' && args.includes('list-sessions')) {
|
||||
return { stdout: '_holder\n', stderr: '', exitCode: 0 };
|
||||
}
|
||||
if (command === 'tmux' && args.includes('show-environment')) {
|
||||
return {
|
||||
stdout:
|
||||
'HOME=/home/mosaic\nMOSAIC_FLEET_OWNER=11111111-1111-4111-8111-111111111111\nMOSAIC_TMUX_HOLDER=_holder\nMOSAIC_TMUX_SOCKET=mosaic-fleet\nPATH=/usr/bin:/bin\nPWD=/home/mosaic\n',
|
||||
stderr: '',
|
||||
exitCode: 0,
|
||||
};
|
||||
}
|
||||
return { stdout: '', stderr: '', exitCode: 0 };
|
||||
},
|
||||
}),
|
||||
});
|
||||
expect(result.applied).toBe(true);
|
||||
const brokerStart = calls.findIndex(
|
||||
(c) => c.join(' ') === 'systemctl --user start mosaic-lease-broker.service',
|
||||
);
|
||||
const agentStart = calls.findIndex(
|
||||
(c) => c.join(' ') === 'systemctl --user start [email protected]',
|
||||
);
|
||||
expect(brokerStart).toBeGreaterThanOrEqual(0);
|
||||
expect(agentStart).toBeGreaterThan(brokerStart);
|
||||
});
|
||||
|
||||
it('fails closed on a symlinked fleet ancestor without touching its target', async (): Promise<void> => {
|
||||
const home = await lockHome();
|
||||
const fleet = join(home, 'fleet');
|
||||
|
||||
@@ -43,6 +43,10 @@ export interface FleetReconcileDeps {
|
||||
readonly overrideDir?: string;
|
||||
readonly homeDirectory?: string;
|
||||
readonly readHolderIdentity?: () => Promise<string>;
|
||||
/** Test/observation seams for the lease-broker plan member (#1292). */
|
||||
readonly statPath?: (path: string) => Promise<boolean> | boolean;
|
||||
readonly checkBrokerSocket?: (path: string) => Promise<boolean> | boolean;
|
||||
readonly brokerSocketEnv?: NodeJS.ProcessEnv;
|
||||
readonly validateRoster?: (roster: FleetRosterV2) => Promise<void>;
|
||||
readonly prepareProjections?: (roster: FleetRosterV2) => Promise<readonly unknown[]>;
|
||||
readonly applyProjection?: (prepared: unknown) => Promise<unknown>;
|
||||
@@ -74,6 +78,17 @@ export interface FleetReconcileObservedAgent {
|
||||
export interface FleetReconcilePlan {
|
||||
readonly generation: number;
|
||||
readonly holder: 'owned' | 'missing' | 'ownership-mismatch';
|
||||
/**
|
||||
* Lease broker observation (#1292): every gated runtime registers with the
|
||||
* broker or dies ~4s in — a broker not in the plan cannot be reported as
|
||||
* drifted, which made "broker died an hour ago" and "broker fine"
|
||||
* produce identical output. `unitInstalled` = unit file present in the
|
||||
* active dir; `socketPresent` = live broker at the resolved socket path.
|
||||
*/
|
||||
readonly broker: {
|
||||
readonly unitInstalled: boolean;
|
||||
readonly socketPresent: boolean;
|
||||
};
|
||||
readonly agents: readonly FleetReconcileObservedAgent[];
|
||||
readonly unmanagedSessions: readonly string[];
|
||||
}
|
||||
@@ -314,6 +329,39 @@ function isObservational(command: FleetReconcileCommand): boolean {
|
||||
return command === 'plan' || command === 'status' || command === 'verify' || command === 'doctor';
|
||||
}
|
||||
|
||||
/**
|
||||
* Observe the lease broker for the plan (#1292). Unit presence via systemctl
|
||||
* is-system-running is NOT the signal — a unit can be enabled-but-dead. The
|
||||
* authoritative signal is the socket the gated runtimes connect to, matching
|
||||
* broker-supervisor.ts's `checkBrokerSupervisorHealth` (healthy ===
|
||||
* socketPresent). Injectable so tests drive every branch without a broker.
|
||||
*/
|
||||
async function observeBroker(deps: FleetReconcileDeps): Promise<FleetReconcilePlan['broker']> {
|
||||
const homeDirectory = deps.homeDirectory ?? homedir();
|
||||
const env = (deps.brokerSocketEnv ?? process.env) as NodeJS.ProcessEnv;
|
||||
const uid = typeof process.getuid === 'function' ? process.getuid() : 0;
|
||||
const runtimeDir = env['XDG_RUNTIME_DIR'] ?? `/run/user/${uid}`;
|
||||
const socketPath =
|
||||
env['MOSAIC_LEASE_BROKER_SOCKET'] ?? join(runtimeDir, 'mosaic-lease', 'broker.sock');
|
||||
const configHome = env['XDG_CONFIG_HOME'] ?? join(homeDirectory, '.config');
|
||||
const unitPath = join(configHome, 'systemd', 'user', 'mosaic-lease-broker.service');
|
||||
const statPath = deps.statPath;
|
||||
const checkBrokerSocket = deps.checkBrokerSocket;
|
||||
let unitInstalled = false;
|
||||
let socketPresent = false;
|
||||
try {
|
||||
unitInstalled = statPath ? await statPath(unitPath) : false;
|
||||
} catch {
|
||||
unitInstalled = false;
|
||||
}
|
||||
try {
|
||||
socketPresent = checkBrokerSocket ? await checkBrokerSocket(socketPath) : false;
|
||||
} catch {
|
||||
socketPresent = false;
|
||||
}
|
||||
return { unitInstalled, socketPresent };
|
||||
}
|
||||
|
||||
async function observeFleet(
|
||||
roster: FleetRosterV2,
|
||||
deps: FleetReconcileDeps,
|
||||
@@ -324,10 +372,12 @@ async function observeFleet(
|
||||
'-F',
|
||||
'#{session_name}',
|
||||
]);
|
||||
const broker = await observeBroker(deps);
|
||||
if (sessionsResult.exitCode !== 0) {
|
||||
return {
|
||||
generation: roster.generation,
|
||||
holder: 'missing',
|
||||
broker,
|
||||
agents: await observeAgents(roster, deps, new Set<string>()),
|
||||
unmanagedSessions: [],
|
||||
};
|
||||
@@ -350,6 +400,7 @@ async function observeFleet(
|
||||
return {
|
||||
generation: roster.generation,
|
||||
holder,
|
||||
broker,
|
||||
agents: await observeAgents(roster, deps, sessions),
|
||||
unmanagedSessions: Object.freeze(unmanagedSessions.sort()),
|
||||
};
|
||||
@@ -517,6 +568,24 @@ async function executeExplicitLifecycle(
|
||||
}
|
||||
}
|
||||
try {
|
||||
// Broker FIRST (#1292): a gated runtime started without a running lease
|
||||
// broker dies ~4 seconds in at registration — enable the unit (install
|
||||
// places it) and start it before any holder/agent lifecycle effect. The
|
||||
// socket re-check after start is the same probe observeBroker uses, so a
|
||||
// unit that starts but never produces a socket is caught here, not four
|
||||
// seconds later inside a doomed seat.
|
||||
if (request.command === 'start') {
|
||||
await runChecked(request.deps, 'systemctl', [
|
||||
'--user',
|
||||
'enable',
|
||||
'mosaic-lease-broker.service',
|
||||
]);
|
||||
await runChecked(request.deps, 'systemctl', [
|
||||
'--user',
|
||||
'start',
|
||||
'mosaic-lease-broker.service',
|
||||
]);
|
||||
}
|
||||
if (request.command === 'start' && plan.holder === 'missing') {
|
||||
await runChecked(request.deps, 'systemctl', [
|
||||
'--user',
|
||||
@@ -562,6 +631,12 @@ async function applyDesiredLifecycle(
|
||||
(agent: FleetRosterV2Agent): boolean =>
|
||||
agent.lifecycle.enabled && agent.lifecycle.desiredState === 'running',
|
||||
);
|
||||
// Broker before any running agent, same ordering and reason as the
|
||||
// command-driven path above (#1292).
|
||||
if (needsRunningAgent) {
|
||||
await runChecked(deps, 'systemctl', ['--user', 'enable', 'mosaic-lease-broker.service']);
|
||||
await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-lease-broker.service']);
|
||||
}
|
||||
if (needsRunningAgent && plan.holder === 'missing') {
|
||||
await runChecked(deps, 'systemctl', ['--user', 'start', 'mosaic-tmux-holder.service']);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user