Compare commits

..

2 Commits

Author SHA1 Message Date
Hermes Agent
d12c5f7808 test(mosaic): make #791 PR2 Part 7 symlink control busybox-portable (#791 PR2)
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
CI #1881 failed at head 8bee1b65: the durable-snapshot gate was 40/41, the sole
failure being the Part 7 NEGATIVE control ("secret leaked through the symlink").
Root cause is a test-harness portability gap, not a code defect: node:24-alpine
runs busybox cp as root, and busybox cp REPLACES a symlinked destination instead
of following it, whereas GNU/BSD cp — what a real operator runs `mosaic update`
under — follows the link and leaks. So under the CI harness the CWE-59 leak vector
the control asserts simply cannot occur, and the negative control can't reproduce.

Fix is test-only; install.sh (independently approved at 8bee1b65) and the real
security assertions are untouched. make_symlink_leaf_shim now emulates GNU cp's
follow-through-dest-symlink behavior portably: when the destination is a symlink it
writes the source bytes through the link via redirection (which follows symlinks on
every coreutils, busybox included); otherwise it delegates to the host cp unchanged.
Both the shipped-case and the control run through this single shim, so the ONLY
difference between them remains the SYMLINK-LEAF-GUARD — the control stays
load-bearing and non-tautological, now on busybox too. With the guard present the
symlinked leaf is dropped before this cp runs, so the shipped path is unchanged.

Verified in the exact CI image (node:24-alpine, busybox, root, apk add bash rsync):
durable-snapshot 41/41, manifest-guard 193, rollback 28. GNU host 41/41, shellcheck
clean. Sole tracked delta vs 8bee1b65 = this test file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 19:24:49 -05:00
Hermes Agent
8bee1b65ea feat(mosaic): durable pre-update snapshot + verify net + restore CLI (#791 PR2)
Some checks failed
ci/woodpecker/pr/ci Pipeline failed
Third, independent recovery layer for keep-mode upgrades (Part of #791). PR1's
manifest keeps the sync out of operator paths and the ephemeral /tmp snapshot
rolls back a mid-write crash; this adds the case neither covers — a "successful"
upgrade that a manifest/logic bug silently let touch an operator file.

- install.sh make_durable_snapshot(): before any mutation, copy the operator-owned
  surface that exists to $XDG_STATE_HOME/mosaic/backups/pre-update-<UTC-ts>/
  (0700 dirs / 0600 files, umask 077 scoped + restored). Fail-open; retention
  MOSAIC_BACKUP_RETENTION (default 5).
- install.sh verify_operator_surface(): post-sync, restore any operator file the
  upgrade diverged/removed and warn loudly (a divergence = manifest bug). Honors
  intentional migration removals; refuses to restore through a symlink (CWE-59);
  guards mkdir under set -e.
- mosaic restore (TS): --list (default dry-run), --from <ts> confirmation-gated
  restore. Strict timestamp selector (CWE-22), O_NOFOLLOW 0600 writes (CWE-59),
  containment + managed-dir checks via secure-file.ts. Counts/paths only — never
  emits file contents (secrev).
- Gates: typecheck/lint/format ✓; mosaic vitest 1252; restore.spec 30;
  bash gates durable-snapshot 41 / manifest-guard 193 / rollback 28 / migration 21.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 19:04:14 -05:00
10 changed files with 37 additions and 1994 deletions

View File

@@ -98,39 +98,6 @@ Expected results:
that means the unit ran, not that an agent pane is live. Treat tmux
`has-session`, `list-panes`, process tree, and logs as the liveness evidence.
## Recovery — rebuild generated env projections
Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
deterministic projection of `roster.yaml` (the SSOT) that the launcher
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
wipes or diverges those projections, rebuild them from the roster with
`mosaic fleet regen` — do NOT restart the affected unit first.
```bash
mosaic fleet regen # dry-run (default): show create/rebuild plan per agent
mosaic fleet regen --json # same plan, machine-readable
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated on disk
```
`regen` is projection-only and **never restarts an agent** — it has no path to
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
only (never the projected `KEY=value` body). After `--write`, verify each unit
resolves the intended values before restarting one unit at a time. The unit sets
no `EnvironmentFile=``start-agent-session.sh` sources `.env.generated` itself —
so verify the generated file directly and the launcher path, not a nonexistent
`EnvironmentFile` property:
```bash
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
systemctl --user cat mosaic-agent@<name> | grep ExecStart
systemctl --user restart mosaic-agent@<name>
```
Full recovery runbook and the three-layer #791 protection model (manifest
ownership → pre-update snapshot/restore → regen): see
[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md).
## Release Preflight
Run this checklist before cutting or dogfooding a fleet release:

View File

@@ -1,147 +0,0 @@
# Upgrade Safety & Recovery
How Mosaic protects operator-owned configuration under `~/.config/mosaic` across
framework upgrades, and how to recover if a projection is ever lost.
A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`,
`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident
this hardening addresses: an upgrade that silently overwrites or deletes a file
the operator owns — credentials, personas, a roster, or a generated agent env —
with no snapshot to fall back to.
Protection is layered. Each layer is independent; a later layer catches what an
earlier one misses.
## Layer 1 — Manifest-owned sync (prevention)
The single source of truth for ownership is
[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt).
Both the bash installer and the TypeScript sync path resolve every path against
this one file (parity is enforced by test), so they can never drift.
- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a
`[framework]` glob matches and no `[operator]` carve-out overrides it.
- **Unknown paths default to operator** (fail-safe): a file the manifest never
anticipated is treated as operator-owned and is never pruned.
- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that
only ever iterates framework globs — operator and unknown paths are
structurally unreachable by the prune.
Result: a correct upgrade cannot touch operator config at all.
## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)
Before **any** mutation, the installer snapshots the operator-owned surface that
exists into:
```
${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/
```
- `0700` directories / `0600` files (`umask 077`, scoped and restored),
outside `~/.config/mosaic` and outside any repo.
- **Fail-open**: a snapshot failure warns but never aborts the upgrade it
protects.
- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5).
After the sync, a **verify net** compares each snapshot file against its target
and restores (with a loud warning) any operator file the upgrade diverged or
removed — a divergence means a manifest bug slipped through Layer 1.
Inspect and restore snapshots with the CLI:
```bash
mosaic restore --list # dry-run: enumerate snapshots by timestamp
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
mosaic restore --from <ts> --dry-run # preview a specific restore without writing
```
`mosaic restore` reports **counts and relative paths only** — it never emits file
contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores
are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf
atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails
closed rather than following out of the managed tree).
## Layer 3 — Regeneration from roster SSOT (recovery)
Some operator files are **derived** and do not need a byte-for-byte snapshot to
recover — they can be rebuilt from their source of truth. The fleet's per-agent
generated env projections are the prime case:
- `~/.config/mosaic/fleet/agents/<name>.env.generated` is a deterministic
projection of `~/.config/mosaic/fleet/roster.yaml`.
- The launcher (`start-agent-session.sh`, invoked by
`mosaic-agent@<name>.service`) sources that generated projection to establish
each agent's identity, runtime, model, and working directory. If it is missing
or wrong, the agent cannot launch with its intended identity.
`mosaic fleet regen` rebuilds those projections from the roster SSOT:
```bash
mosaic fleet regen # dry-run (default): show what would be rebuilt
mosaic fleet regen --json # same, machine-readable
mosaic fleet regen --write # rebuild the projections on disk
```
- **Dry-run by default.** Nothing is written until you pass `--write`.
- **Deterministic and idempotent** — the projection is a pure function of the
roster, so repeated `--write` runs produce byte-identical files.
- **Projection-only. It never restarts an agent.** Recovery order forbids
restart-before-verify; `regen` has no path to systemd lifecycle at all.
- **It rebuilds only `<name>.env.generated`** — it never writes, relocates, or
deletes the operator-owned `.env` / `.env.local` surface.
- It **validates the roster the same way `reconcile` does** (persona resolution
and protected-class tool-policy match), so a hand-edited or corrupt roster is
rejected rather than projected, and a `--write` takes the shared reconcile
lock so it cannot race a concurrent reconcile.
- Output is **paths and counts only** — the rendered `KEY=value` body is never
echoed.
`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a
recovered projection matches what a normal reconcile would have written.
## Recovery runbook — wiped `fleet/agents/*.env.generated`
If an upgrade (or a manual mistake) has left an agent without its generated
projection, **do not restart the unit first** — a launch against a missing
projection fails closed, and any stale state must be corrected before restart,
not after.
1. **Prefer a snapshot restore if one exists** (byte-exact operator state):
```bash
mosaic restore --list
mosaic restore --from <UTC-timestamp>
```
2. **Otherwise regenerate the derived projections from the roster SSOT:**
```bash
mosaic fleet regen # confirm the plan (create vs rebuild per agent)
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated
```
3. **Verify each unit will resolve the intended runtime/workdir _before_ any
restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal
environment and `start-agent-session.sh` sources `.env.generated` itself, so
verify the generated file directly and confirm the launcher path:
```bash
# Confirm fleet/agents/<name>.env.generated exists and carries the intended
# MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
# Confirm the unit launches the session script that reads it.
systemctl --user cat mosaic-agent@<name> | grep ExecStart
```
4. **Only then restart, one unit at a time:**
```bash
systemctl --user restart mosaic-agent@<name>
```
## See also
- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md)
- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md)
- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)

View File

@@ -296,234 +296,3 @@ vitest **1252** · restore.spec **30** · durable-snapshot **41** · manifest-gu
migration 21. shellcheck clean on all new lines; new test markers mirror the existing `# VERIFY-NET`
anchor convention. NOTE: codex self-review does NOT satisfy the independent-review gate — an independent
(author≠reviewer) review + durable Gitea Reviewer-of-Record comment is still required before MS-LEAD merges.
## Session 4 (2026-07-16) — PR2 MERGED, PR3 built (fleet regen — recovery layer)
PR2 (#811) squash-merged → main `31607a4a`; issue #791 stays open (final PR of the 3-PR DAG). Independent
exact-head RoR at `d12c5f78` APPROVE (Gitea cmt 17904); #1882 green; busybox-portable Part 7 control fix
verified in-Alpine. PR3 UNBLOCKED.
PR3 branch: `feat/791-pr3-fleet-regen` off `origin/main` 31607a4. Same discipline: tests-first red-first,
independent review + durable Gitea RoR BEFORE MS-LEAD runs the queue guard/merge. PR body `Part of #791`.
### PR3 scope (ratified §4/§7 of design doc) — `mosaic fleet regen`
Projection-only recovery command: rebuilds each `fleet/agents/<name>.env.generated` from `roster.yaml`
(SSOT). Dry-run default; `--write` applies; `--json` machine output. Structural guarantee: NO code path to
systemd lifecycle — **never restarts an agent**. Single-SSOT: reuses `projectRosterV2AgentGeneratedEnv`
(extracted, shared with the reconciler apply path) so regen and reconcile cannot drift. Secrev: paths +
counts only, never the rendered KEY=value body.
New files: `commands/fleet-regen-command.ts` (+ `.spec.ts`), guide `docs/guides/upgrade-safety-and-recovery.md`
(three-layer model: PR1 manifest ownership → PR2 snapshot/restore → PR3 regen; do-NOT-restart-before-verify
runbook), regen reference added to `docs/guides/fleet-local-canary.md`. Wired in `commands/fleet.ts`.
### Independent review (3 reviewers: subagent code-reviewer + codex code-review + codex security) → 4 fixes, red-first
- **A · BLOCKER (codex) — regen mutated/deleted legacy operator env.** `applyPreparedAgentEnvironmentProjection`
also writes `.env.local`/`.env.quarantine` and unlinks legacy `.env`. Violated projection-only contract.
**Fix:** NEW generated-only boundary primitives `prepareGeneratedAgentEnvironmentProjection` +
`applyPreparedGeneratedAgentEnvironmentProjection` (write ONLY `<name>.env.generated`). regen now has no
code path that touches `.env`/`.env.local`/`.env.quarantine`. **Test:** projection-only leaves legacy `.env`
verbatim, no local/quarantine fabricated.
- **B · should-fix (codex + subagent + security) — partial write on mid-loop failure.** Interleaved
prepare/apply left earlier agents written when a later agent failed prepare. **Fix:** PREPARE ALL agents
before writing ANY (mirrors reconciler `defaultPrepareProjections`). **Test:** 2nd agent's projection
pre-seeded 0644 → prepare rejects → coder0 NOT written, exit 1.
- **C · subagent — semantic-validation bypass.** Default readRoster skipped `validateRosterV2Semantics`, so
a tampered protected-class `tool_policy` would be silently projected. **Fix:** default readRoster now runs
`validateRosterV2Semantics` (persona resolution + protected-class match), rolesDir/overrideDir defaults
mirroring the reconciler. **Test:** merge-gate agent w/ tool_policy=code → fails closed, no write.
- **D · MEDIUM (codex security, CWE-362) — concurrent-reconcile race.** regen `--write` wrote without the
reconcile lock. **Fix:** `--write` acquires `acquirePrivateReconcileLock(mosaicHome)` for the whole
read-prepare-apply sequence, released in `finally`; dry-run stays lock-free. **Test:** pre-held lock →
regen fails closed, no write.
**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1265**
(regen spec 13, incl. 4 new red-first regressions). NOTE: codex self-review does NOT satisfy the
independent-review gate — an independent (author≠reviewer) review + durable Gitea RoR is still required
before MS-LEAD merges. STOP at PR-open for MS-LEAD's exact-head review; do NOT self-merge.
## Session 5 — PR3 review round 2 (finding L + M1/M2/M3), red-first fixes
Second review pass on the lock-cleanup plumbing surfaced one round-1 residual (L) and three round-2
findings (M1 blocker, M2/M3 should-fix). All fixed red-first (RED proven per-finding, then GREEN).
- **L · should-fix (codex r1) — mutation-lock release swallowed unlink failures.** regen's
`acquirePrivateRosterMutationLock` release copied CRUD's `unlink().catch(()=>{})`, hiding a stale
`roster.yaml.mutation.lock`. **Fix:** its release PROPAGATES the unlink fault (finding-J stale-lock
warning then fires for this lock too). **Test:** acquire real lock, `rm` it, assert `release()` rejects.
- **M1 · BLOCKER (codex r2) — replacement-lock race.** The propagating release from L did an
UNCONDITIONAL `unlink(lockPath)` without proving ownership. If the lock is cleared + re-created by
another writer mid-op, regen deletes the STRANGER's live lock → a third writer enters → mutual
exclusion defeated. **Fix (reuse, not reimplement):** generalized the reconciler's ownership-proving
lock body into shared `acquirePrivateManagedRosterLock(mosaicHome, lockLeaf, busyMessage, openLock)`;
`acquirePrivateReconcileLock` delegates to it (behavior-identical: same leaf/codes/messages), and a NEW
hardened `acquirePrivateRosterMutationLock` (now in fleet-reconciler.ts, leaf `roster.yaml.mutation.lock`)
records dev/ino + ownership token and RE-PROVES ownership (`assertLockOwnership`) before unlinking —
fails closed as `lock-cleanup-failed` if replaced. Removed the crud-based export; reverted
`acquireMutationLock` (fleet-agent-crud.ts) to its original inline empty-file/swallowing-release form
(CRUD behavior intentionally unchanged). Compatibility: CRUD empty-file `wx` and regen tokened `wx`
contend on the same path but never co-own (wx winner owns; loser → concurrent-mutation), so the token
is only ever read back by the same regen invocation. **Test:** acquire, `rm`+recreate lock (new inode),
assert `release()` rejects AND the replacement survives (not unlinked).
- **M2 · should-fix (codex r2) — acquire-unwind fault dropped.** The acquire-failure catch discarded
`releaseFleetLocks`' return (a possible fault on the already-held first lock). **Fix:** capture and
augment — `const releaseFault = await releaseFleetLocks(releases); throw augmentWithLockCleanupFault(error, releaseFault);`
(symmetric to finding J). **Test:** mutation lock acquires w/ faulting release + reconcile acquire
throws → thrown error mentions stale/lock, nothing written.
- **M3 · should-fix (codex r2 + subagent REQUEST-CHANGES) — cleanup warning named only reconcile lock.**
Finding L made the mutation-lock release fault reachable, so the `cleanup` marker can originate from
EITHER lock. **Fix:** `formatFleetRegenReport`'s WARNING now names BOTH `roster.yaml.mutation.lock` and
`roster.yaml.reconcile.lock`, matching `augmentWithLockCleanupFault`. **Test:** fault the mutation-lock
release specifically → report names both lock files.
**Refactor note (no cycle):** neither fleet-reconciler nor fleet-agent-crud imports the other; regen
imports lock acquirers from fleet-reconciler and the projection mapping from fleet-reconciler. The two
reconcile-lock reviewers reconciled: independent reviewer validated acquire-time empty-file compatibility
(preserved), codex flagged RELEASE-time replacement race (closed by ownership proof) — non-contradictory.
**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1275**
(regen spec 23, incl. 7 red-first lock regressions E/F/G/K/L/M1/M2/M3). RED proven per-finding by
temporary revert before re-applying each fix. Independent (author≠reviewer) review of M1/M2/M3 + codex
code/security re-run in flight. STOP at PR-open for MS-LEAD's exact-head review + durable Gitea RoR; do
NOT self-merge; #791 umbrella stays OPEN; PR body `Part of #791`.
### Round 3 review (after M1/M2/M3) — independent review PASS + codex residual-TOCTOU disposition
Three reviewers on the post-M1/M2/M3 head:
- **Independent (subagent, author≠reviewer) — PASS.** Verified M1/M2/M3 all correctly fixed; "never
restarts" is STRUCTURAL (runner never referenced in executable code); no secrets; no deadlock (only
regen holds both locks); tests meaningful (assert inode preservation + exact lock-file names). Raised:
- **should-fix #1 (fixed, red-first):** generalizing the lock helper left `assertSafeLockLeafIfPresent`/
`assertLockOwnership` hardcoding "reconciliation lock" in thrown messages → a MUTATION-lock fault
misreported as the reconcile lock, undercutting M3's accurate-diagnosis goal. **Fix:** thread
`lockLabel = fleet/<leaf>` through both helpers + the generic lock-io messages, so every fault names
the actual lock file. Red-first: strengthened the M1 test to assert `/roster\.yaml\.mutation\.lock/`
(RED: got "reconciliation lock"; GREEN after). Also resolves nit #3 (generic-message drift).
- **nit #2 (fixed):** `FleetRegenResult.cleanup` JSDoc still said "the shared reconcile lock"; now names
both locks (regen holds both).
- **nit #4 (fixed):** removed the redundant duplicate `assertLockOwnership` call before unlink
(pre-existing in merged main; harmless but dead — dropped since the fn was already being touched).
- **Codex security — clean (risk: none).** Validates roster semantics, constrains env values, no shell
eval, no secret output, generated-only writes, serialized against both locks.
- **Codex code — request-changes, 1 "blocker": residual check-then-unlink TOCTOU.** Between the final
`assertLockOwnership` and the path-based `unlink`, an external actor could vacate our inode and a new
writer grab the path, so the unlink deletes the stranger's lock. **Disposition: documented known
limitation, NOT fixed in PR3.** Rationale: (1) byte-identical to the MERGED, shipped reconcile-lock
release on origin/main (fleet-reconciler.ts L654-659) — not introduced here; (2) UNREACHABLE within the
`wx` writer protocol — no Mosaic writer removes a lock it doesn't own (wx fails EEXIST while our inode
exists), so only external interference can vacate our inode in the sub-instruction window; (3) the
ownership guard DOES close the reachable case (stale-lock reaper/operator cleared our lock + another
writer took it BEFORE release began → fail closed, don't delete stranger's lock); (4) the true atomic
fix — fd-held advisory lock (flock/lockf) adopted by ALL fleet writers (CRUD + reconcile + regen) — is
a cross-cutting mechanism change touching merged CRUD + reconciler, out of scope for a projection-only
recovery PR. Documented honestly in the acquirer doc + M1 test comment. **The binding independent
review did NOT treat this as a blocker.** Recommendation to MS-LEAD: proceed to PR-open + spin a
SEPARATE follow-up issue for the fd-advisory-lock migration; MS-LEAD adjudicates scope at exact-head
review (merge authority).
**Gates after round-3 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
**1275** (regen spec 23). Fresh codex code re-run in flight to confirm no NEW issues from the label fix.
---
## Session 6 — Round 4/5 convergence (stranded-lock robustness)
**Two independent reviewers converged on the SAME should-fix** on the init-failure cleanup path,
strengthening confidence it was real:
- **Codex code-review-5 — 0 blockers, 1 should-fix.** "Stat failure after lock creation strands the new
lock." When `handle.stat()` ITSELF fails right after the `wx` create (transient EIO/EBADF), `created`
is `undefined`, so `removeOwnedLockLeafBestEffort` had `if (!created) return;` → no cleanup → the
just-created `roster.yaml.mutation.lock`/`reconcile.lock` is stranded, permanently blocking future
regen + CRUD. (Notably NO blocker, and the TOCTOU is no longer flagged in code-review as of r5.)
- **Independent delta reviewer (author≠reviewer, pr-review-toolkit) — no blockers, same should-fix.**
Independently flagged the identical `!created` gap; validated FIX 1 (label threading — no call site
missed, codes unchanged, no test depended on old text) and FIX 2 (dev/ino-guarded cleanup, best-effort,
happy-path release reuses captured dev/ino) as correct. Suggested an unconditional best-effort unlink
in the `!created` branch; I took the **safer** variant below.
- **Codex security-review-5 — 0 crit / 0 high / 1 medium.** The single medium is the SAME residual
check-then-unlink TOCTOU already dispositioned in round 3 (its own remediation = "migrate every writer
to an fd-held advisory lock" = the follow-up issue). No new security finding. No secrets.
**Fix (red-first, safer than an unconditional unlink):** thread the persisted random `token` into
`removeOwnedLockLeafBestEffort`. Two independent ownership proofs now: primary dev/ino (unchanged), and a
**fallback** when the post-create stat failed — read the leaf and unlink ONLY if its content equals our
`randomUUID()` token. Only OUR lock carries that token, so a CRUD (empty) or differently-tokened
replacement is never deleted. `tokenPersisted` guards passing the token (only after `writeFile` lands).
Doubly-degenerate case (stat fails AND token write never landed) leaves the lock in place rather than
risk deleting a stranger's file — requires two independent fs faults on a just-created fd; documented.
- **Red-first proof:** new test `does not strand the lock file when the post-create stat itself fails`
injects a real `wx` create + a Proxy handle whose `stat()` rejects (writeFile/close succeed), asserts
`exists(lockPath) === false`. RED before fix (`expected true to be false` — lock stranded); GREEN after.
- **Also fixed (delta nit #3):** `fleet-regen-command.ts` `acquireRosterMutationLock` JSDoc said "CRUD's
private lock"; the default is the reconciler's hardened ownership-proving acquirer for the same
`fleet/roster.yaml.mutation.lock` path. Corrected.
- **PR-description note (delta nit #2):** FIX 1 also collapsed a pre-existing duplicate back-to-back
`assertLockOwnership` call in the release closure (identical args, no intervening logic) into one — a
no-op simplification of merged code, not a behavior change. Called out so a future reader doesn't
wonder if the duplicate had a purpose.
**Gates after round-4 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
**1277** (regen spec now 25: +1 stat-failure stranded-lock regression). Residual TOCTOU still deferred to
the fd-advisory-lock follow-up issue; MS-LEAD adjudicates scope at exact-head review (merge authority).
---
## Session 6 — Round 6 (persona-root wiring)
**Codex code-review-6 — 0 blockers, 1 should-fix (NEW, distinct from the lock work).** "Forward
configured persona directories to regen." `registerFleetRegenCommand` was registered at
`fleet.ts:2069` with only `{ runner, mosaicHome }`, discarding `deps.reconcileDeps.rolesDir` /
`overrideDir`. The regen command ALREADY has those seams (validates roster semantics via
`validateRosterV2Semantics({ rolesDir, overrideDir })`, defaulting to `<mosaicHome>/fleet/roles{,.local}`),
but the top-level wiring never forwarded the configured roots. **Impact:** in a deployment with custom
persona roots, `fleet reconcile` (which honors the overrides) would ACCEPT a roster while `fleet regen`
REJECTS the same roster (persona resolution against the wrong default dir) — blocking the recovery
command and violating the documented "resolves personas the SAME way reconcile does" contract.
**Fix (red-first):** forward `rolesDir`/`overrideDir` from `deps.reconcileDeps` into
`registerFleetRegenCommand` at `fleet.ts:2069`. Red-first test `forwards configured persona roots
(rolesDir/overrideDir) from reconcileDeps into regen`: seeds personas ONLY under a custom root, leaves
the default `<home>/fleet/roles` empty, registers with `reconcileDeps: { rolesDir, overrideDir }`, and
requires `fleet regen` to SUCCEED. RED before fix (`expected 1 not to be 1` — regen validated against the
empty default and exited 1); GREEN after.
**Codex security-review-6 — 0 crit / 0 high / 1 medium.** Same residual check-then-unlink TOCTOU, now
noted at BOTH the release closure and the init-cleanup path; remediation = fd-held advisory lock across
all writers = the SAME deferred follow-up item. No new security finding, no secrets.
**Independent confirmation review of the token-fallback fix (Session 6/round 4) — PASS, no findings.**
All 7 verification points confirmed; reviewer mechanically reverted `removeOwnedLockLeafBestEffort` to
the pre-fix `if (!created) return;` and re-ran the new test → RED (`expected true to be false`),
confirming the test genuinely pins the fix; restored after. No lint/type issues; doc-comment accurate.
**Gates after round-6 fix (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
**1278** (regen spec now 26: +1 persona-root wiring regression).
---
## Session 6 — Round 7 convergence (review CLOSED for PR-open)
- **Codex code-review-7 — 0 blockers, 1 should-fix = the residual TOCTOU** (previously a "blocker" in r3,
dropped in r4/r5, now re-surfaced as a should-fix). **Codex security-review-7 — 0 crit / 0 high /
1 medium = the SAME residual TOCTOU.** Codex has CONVERGED: the only remaining finding across both
streams is that one race, whose own remediation is "fd-held advisory lock shared by all fleet writers"
= the deferred follow-up. No new distinct finding; the wiring fix introduced nothing.
- **Independent confirmation review of the persona-root wiring fix — PASS, no findings.** Reviewer
mechanically reverted the two forwarded lines → RED (`Roster v2 agent "coder0" class "code" does not
resolve to a readable persona` → exit 1), restored → GREEN (26 regen + 204 fleet tests). Confirmed the
optional-chaining fallback preserves default-deployment behavior and no type/lint issue.
**Review disposition for PR-open:** ALL actionable findings fixed red-first across rounds 36 (label
threading, stranded-lock on init failure, stat-failure strand, persona-root wiring). The residual
check-then-unlink TOCTOU is the ONLY open item and is DEFERRED to a follow-up issue (fd-advisory-lock
migration across CRUD + reconcile + regen) — byte-identical to merged origin/main's reconcile-lock
release, unreachable within the `wx` writer protocol (no Mosaic writer removes a lock it doesn't own;
only external `rm`/a stale-lock reaper can vacate the inode mid-release), and its true fix is a
cross-cutting mechanism change out of scope for a projection-only recovery PR. Two independent human-agent
reviews (author≠reviewer) treated it as non-blocking. MS-LEAD adjudicates scope at exact-head review
(merge authority); recommendation = proceed to PR-open + spin the follow-up issue.
**Final gates (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1278**
(regen spec 26). No secret values in any snapshot/projection/report output (counts + paths only). Regen
NEVER issues a lifecycle/restart call (load-bearing recordingRunner gate). STOP at PR-open for MS-LEAD's
exact-head review + durable Reviewer-of-Record before any merge; do NOT self-merge.

View File

@@ -1,885 +0,0 @@
import { chmod, mkdir, mkdtemp, open, readFile, rm, stat, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { Command } from 'commander';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { registerFleetCommand, type CommandResult, type FleetCommandDeps } from './fleet.js';
import { executeFleetRegen, formatFleetRegenReport } from './fleet-regen-command.js';
import {
acquirePrivateRosterMutationLock,
projectRosterV2AgentGeneratedEnv,
} from '../fleet/fleet-reconciler.js';
import { applyPreparedGeneratedAgentEnvironmentProjection } from '../fleet/generated-env-boundary.js';
import { parseRosterV2 } from '../fleet/roster-v2.js';
// A two-agent roster-v2 SSOT. `mosaic fleet regen` must rebuild each agent's
// `fleet/agents/<name>.env.generated` projection from exactly this source and
// nothing else — deterministically, without ever touching agent lifecycle.
const rosterYaml = `
version: 2
generation: 7
transport: tmux
tmux:
socket_name: mosaic-fleet
holder_session: _holder
defaults:
working_directory: /srv/mosaic
runtime: pi
runtimes:
pi:
reset_command: /new
agents:
- name: coder0
alias: Coder 0
class: code
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: high
tool_policy: code
working_directory: /srv/mosaic
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
- name: coder1
alias: Coder 1
class: code
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: medium
tool_policy: code
working_directory: /srv/other
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
`;
let cleanup: string | undefined;
afterEach(async (): Promise<void> => {
vi.restoreAllMocks();
process.exitCode = undefined;
if (cleanup) await rm(cleanup, { recursive: true, force: true });
cleanup = undefined;
});
async function fleetHome(withRoster = rosterYaml): Promise<string> {
cleanup = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-command-'));
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
await mkdir(join(cleanup, directory), { recursive: true, mode: 0o700 });
await chmod(join(cleanup, directory), 0o700);
}
await chmod(cleanup, 0o700);
await writeFile(join(cleanup, 'fleet', 'roster.yaml'), withRoster, { mode: 0o600 });
// Semantic roster validation (matching `fleet reconcile`) resolves each agent
// class to a persona; seed the classes the fixtures reference.
for (const klass of ['code', 'merge-gate']) {
await writeFile(
join(cleanup, 'fleet', 'roles', `${klass}.md`),
`\`class: ${klass}\`\n\n# ${klass} persona\n`,
{ mode: 0o600 },
);
}
return cleanup;
}
/**
* A runner spy that records EVERY invocation. `regen` is projection-only and
* must never issue a lifecycle/restart call, so a non-empty call log is a
* hard failure — this is the load-bearing "never restarts" gate.
*/
function recordingRunner(
calls: string[][],
): (command: string, args: string[]) => Promise<CommandResult> {
return async (command: string, args: string[]): Promise<CommandResult> => {
calls.push([command, ...args]);
return { stdout: '', stderr: '', exitCode: 0 };
};
}
function program(mosaicHome: string, runner: FleetCommandDeps['runner']): Command {
const result = new Command();
result.exitOverride();
registerFleetCommand(result, { mosaicHome, runner });
return result;
}
function capture(): string[] {
const lines: string[] = [];
vi.spyOn(console, 'log').mockImplementation((value: string): void => {
lines.push(value);
});
return lines;
}
async function exists(path: string): Promise<boolean> {
try {
await stat(path);
return true;
} catch {
return false;
}
}
describe('projectRosterV2AgentGeneratedEnv', (): void => {
it('maps a roster-v2 agent to exactly the eight generated projection keys', (): void => {
const roster = parseRosterV2(rosterYaml, 'yaml');
const agent = roster.agents.find((candidate) => candidate.name === 'coder0');
expect(agent).toBeDefined();
const values = projectRosterV2AgentGeneratedEnv(roster, agent!);
expect(values).toEqual({
MOSAIC_AGENT_NAME: 'coder0',
MOSAIC_AGENT_CLASS: 'code',
MOSAIC_AGENT_RUNTIME: 'pi',
MOSAIC_AGENT_MODEL: 'gpt-5.6-sol',
MOSAIC_AGENT_REASONING: 'high',
MOSAIC_AGENT_TOOL_POLICY: 'code',
MOSAIC_AGENT_WORKDIR: '/srv/mosaic',
MOSAIC_TMUX_SOCKET: 'mosaic-fleet',
});
});
});
describe('mosaic fleet regen', (): void => {
it('is dry-run by default: reports the plan and writes nothing', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const lines = capture();
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--json',
]);
const result = JSON.parse(lines.pop() ?? '{}');
expect(result).toMatchObject({
mode: 'dry-run',
generation: 7,
agentCount: 2,
written: 0,
});
expect(result.agents.map((agent: { name: string }) => agent.name)).toEqual([
'coder0',
'coder1',
]);
expect(
result.agents.every((agent: { disposition: string }) => agent.disposition === 'create'),
).toBe(true);
// Nothing on disk.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
// No lifecycle/restart call — ever.
expect(calls).toEqual([]);
});
it('forwards configured persona roots (rolesDir/overrideDir) from reconcileDeps into regen', async (): Promise<void> => {
// Codex r6: regen must resolve personas the SAME way reconcile does. If the
// top-level registration drops reconcileDeps.rolesDir/overrideDir, a deployment
// with custom persona roots gets reconcile ACCEPTING a roster while regen
// REJECTS it (validating against the wrong default `<home>/fleet/roles`),
// blocking the recovery command. Pin the wiring: seed personas ONLY under the
// custom roots, leave the default roles dir empty, require regen to succeed.
const home = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-wiring-'));
cleanup = home;
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
await mkdir(join(home, directory), { recursive: true, mode: 0o700 });
await chmod(join(home, directory), 0o700);
}
await chmod(home, 0o700);
await writeFile(join(home, 'fleet', 'roster.yaml'), rosterYaml, { mode: 0o600 });
// Personas live ONLY under the configured roots — the default `fleet/roles`
// stays empty, so broken wiring fails persona resolution.
const customRoles = join(home, 'custom-roles');
const customOverride = join(home, 'custom-roles.local');
await mkdir(customRoles, { recursive: true, mode: 0o700 });
await mkdir(customOverride, { recursive: true, mode: 0o700 });
for (const klass of ['code', 'merge-gate']) {
await writeFile(join(customRoles, `${klass}.md`), `\`class: ${klass}\`\n\n# ${klass}\n`, {
mode: 0o600,
});
}
const command = new Command();
command.exitOverride();
registerFleetCommand(command, {
mosaicHome: home,
runner: recordingRunner([]),
reconcileDeps: { rolesDir: customRoles, overrideDir: customOverride },
});
const lines = capture();
await command.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
// Regen validated against the CONFIGURED roots → success. Broken wiring
// resolves against the empty default and fails (exitCode 1, no JSON result).
expect(process.exitCode).not.toBe(1);
const result = JSON.parse(lines.pop() ?? '{}');
expect(result).toMatchObject({ mode: 'dry-run', agentCount: 2 });
});
it('--write rebuilds each generated projection from the roster SSOT', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const lines = capture();
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
const result = JSON.parse(lines.pop() ?? '{}');
expect(result).toMatchObject({ mode: 'write', written: 2, agentCount: 2 });
const coder0 = await readFile(join(home, 'fleet', 'agents', 'coder0.env.generated'), 'utf8');
expect(coder0).toContain('MOSAIC_AGENT_NAME=coder0');
expect(coder0).toContain('MOSAIC_AGENT_WORKDIR=/srv/mosaic');
expect(coder0).toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet');
const coder1 = await readFile(join(home, 'fleet', 'agents', 'coder1.env.generated'), 'utf8');
expect(coder1).toContain('MOSAIC_AGENT_NAME=coder1');
expect(coder1).toContain('MOSAIC_AGENT_WORKDIR=/srv/other');
// No lifecycle/restart call — ever.
expect(calls).toEqual([]);
});
it('is deterministic and idempotent across repeated --write runs', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const path = join(home, 'fleet', 'agents', 'coder0.env.generated');
const cli = program(home, recordingRunner(calls));
capture();
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
const first = await readFile(path, 'utf8');
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
const second = await readFile(path, 'utf8');
expect(second).toBe(first);
expect(calls).toEqual([]);
});
it('reports rebuild disposition once the generated projection already exists', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const cli = program(home, recordingRunner(calls));
const lines = capture();
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
lines.length = 0;
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
const result = JSON.parse(lines.pop() ?? '{}');
expect(
result.agents.every((agent: { disposition: string }) => agent.disposition === 'rebuild'),
).toBe(true);
expect(result.written).toBe(0);
expect(calls).toEqual([]);
});
it('never issues a lifecycle/restart call in either mode', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const cli = program(home, recordingRunner(calls));
capture();
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
// Structural guarantee: regen has no path to systemctl/tmux at all.
expect(calls).toEqual([]);
const systemctlCalls = calls.filter(([command]) => command === 'systemctl');
expect(systemctlCalls).toEqual([]);
});
it('human-readable --write output prints the do-not-restart recovery runbook', async (): Promise<void> => {
const home = await fleetHome();
const lines = capture();
await program(home, recordingRunner([])).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
]);
const output = lines.join('\n');
expect(output).toMatch(/do not restart|before.*restart/i);
expect(output).toMatch(/env\.generated/);
// The unit has no EnvironmentFile= directive, so the runbook must NOT tell the
// operator to verify one — verify the launcher/generated file instead.
expect(output).not.toContain('EnvironmentFile');
expect(output).toMatch(/systemctl --user cat mosaic-agent@<name>/);
expect(output).toMatch(/restart/i);
});
it('emits paths and counts only — never the projected env body (secrev)', async (): Promise<void> => {
const home = await fleetHome();
const lines = capture();
await program(home, recordingRunner([])).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
]);
const output = lines.join('\n');
// Relative paths + counts are fine; the rendered projection body (KEY=value
// lines) must never be echoed to stdout.
expect(output).toContain('coder0.env.generated');
expect(output).not.toMatch(/^MOSAIC_AGENT_\w+=/m);
expect(output).not.toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet');
});
it('is projection-only: leaves legacy .env untouched and writes no .env.local/.env.quarantine', async (): Promise<void> => {
// Recovery contract: regen rebuilds ONLY <name>.env.generated. It must never
// relocate, quarantine, or unlink the operator-owned legacy .env surface — the
// full reconciler apply path does, so regen must NOT use it.
const home = await fleetHome();
const legacyPath = join(home, 'fleet', 'agents', 'coder0.env');
await writeFile(legacyPath, 'MOSAIC_RUNTIME_BIN=/usr/bin/pi\n', { mode: 0o600 });
capture();
await program(home, recordingRunner([])).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
// Generated projection rebuilt...
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
// ...but the operator's legacy .env is preserved verbatim, and no local/quarantine
// files were fabricated from it.
expect(await readFile(legacyPath, 'utf8')).toBe('MOSAIC_RUNTIME_BIN=/usr/bin/pi\n');
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.local'))).toBe(false);
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.quarantine'))).toBe(false);
});
it('prepares every agent before writing any: a later prepare failure leaves nothing written', async (): Promise<void> => {
// Fail-closed across agents. Pre-seed coder1's projection with world-readable
// perms so its prepare rejects; coder0 (valid) must NOT be written because
// preparation is fully completed before the first apply.
const home = await fleetHome();
const coder1Path = join(home, 'fleet', 'agents', 'coder1.env.generated');
await writeFile(coder1Path, 'MOSAIC_AGENT_NAME=coder1\n', { mode: 0o644 });
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
// coder0 is valid but must remain unwritten — no partial rebuild.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('fails closed on a semantically invalid roster (protected-class tool-policy mismatch)', async (): Promise<void> => {
// A hand-edited roster giving a protected class a weaker tool_policy is rejected
// by reconcile/plan/verify; regen must enforce the SAME gate, not silently
// project the downgraded policy into <name>.env.generated.
const home = await fleetHome(`
version: 2
generation: 3
transport: tmux
tmux:
socket_name: mosaic-fleet
holder_session: _holder
defaults:
working_directory: /srv/mosaic
runtime: pi
runtimes:
pi:
reset_command: /new
agents:
- name: gate0
alias: Gate 0
class: merge-gate
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: high
tool_policy: code
working_directory: /srv/mosaic
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
`);
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
expect(await exists(join(home, 'fleet', 'agents', 'gate0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('refuses to write while a concurrent reconcile holds the mutation lock', async (): Promise<void> => {
// regen --write mutates the same projections as reconcile; it must take the
// shared reconcile lock so a concurrent reconcile cannot race a stale write.
const home = await fleetHome();
await writeFile(join(home, 'fleet', 'roster.yaml.reconcile.lock'), 'held\n', { mode: 0o600 });
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('fails closed (non-zero, no writes) on an invalid roster', async (): Promise<void> => {
// roster-v2 requires >= 1 agent; a malformed roster must abort regen without
// writing any projection and without touching lifecycle.
const home = await fleetHome('version: 2\ngeneration: 1\n');
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('rejects a non-canonical --roster path (canonical roster only)', async (): Promise<void> => {
// `fleet` exposes a global `--roster`; reconcile rejects a non-canonical value
// rather than silently reading the canonical roster. regen must enforce the
// SAME guard so `--roster /elsewhere regen --write` cannot mislead the operator.
const home = await fleetHome();
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'--roster',
join(home, 'other', 'roster.yaml'),
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
expect(errors.join('')).toMatch(/canonical roster/i);
// No projection written against a misdirected roster path.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('reports an incomplete rebuild when a later projection write fails mid-loop', async (): Promise<void> => {
// Fault injected on the SECOND apply: coder0 is already replaced on disk, so a
// bare throw would hide the partial state. regen must surface which agents were
// rebuilt and which failed, with a verify-before-restart recovery instruction.
const home = await fleetHome();
const realApply = applyPreparedGeneratedAgentEnvironmentProjection;
const result = await executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
applyProjection: async (prepared): Promise<string> => {
if (prepared.generatedPath.endsWith('coder1.env.generated')) {
throw new Error('simulated ENOSPC on coder1');
}
return realApply(prepared);
},
},
{ write: true },
);
// coder0 written before the fault; coder1 not.
expect(result.written).toBe(1);
expect(result.incomplete).toMatchObject({
code: 'projection-apply-failed',
failedAgent: 'coder1',
writtenAgents: ['coder0'],
});
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
// The human report names the failed agent and directs a verify-before-restart
// recovery — not a bare success runbook.
const report = formatFleetRegenReport(result).join('\n');
expect(report).toMatch(/incomplete/i);
expect(report).toContain('coder1');
expect(report).toMatch(/verify|re-run|retry/i);
// Structural: still no lifecycle call anywhere on the failure path.
expect(report).not.toMatch(/systemctl --user restart mosaic-agent@<name>\n.*\n/);
});
it('refuses to write while agent CRUD holds the roster mutation lock', async (): Promise<void> => {
// regen --write overwrites the SAME generated projections that `fleet agent
// create/update/delete` writes under roster.yaml.mutation.lock. If regen only
// took the reconcile lock it could read a stale roster and overwrite a just-
// committed projection. It must also take the mutation lock and fail closed.
const home = await fleetHome();
await writeFile(join(home, 'fleet', 'roster.yaml.mutation.lock'), 'crud\n', { mode: 0o600 });
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
// No projection written against a roster a concurrent CRUD is mutating.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('surfaces BOTH the roster failure and a stale-lock warning on a double fault', async (): Promise<void> => {
// Worst case with nothing written: run() fails (invalid roster, fail-closed) AND
// a lock release faults. The operator must be told the lock may be stale, not
// just the roster error — otherwise the next regen/reconcile is silently blocked.
const home = await fleetHome('version: 2\ngeneration: 1\n');
const throwingRelease =
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated lock unlink fault');
};
await expect(
executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
acquireReconcileLock: throwingRelease,
acquireRosterMutationLock: throwingRelease,
},
{ write: true },
),
).rejects.toThrow(/stale|inspect|lock/i);
// Nothing written on the fail-closed path.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
});
it('propagates a roster-mutation-lock release failure instead of silently keeping a stale lock', async (): Promise<void> => {
// Finding L: the real `acquirePrivateRosterMutationLock` release must SURFACE an
// unlink failure, not swallow it. If it swallowed (like the CRUD-internal
// acquireMutationLock does), a stale roster.yaml.mutation.lock left after a
// successful regen would block later regen/agent CRUD while the command reported
// success — and the finding-J stale-lock warning would never fire for this lock.
const home = await fleetHome();
const release = await acquirePrivateRosterMutationLock(home)();
// Simulate the lock vanishing (or being unremovable) before release runs: the
// underlying unlink then faults. A swallowing release would resolve and hide it.
await rm(join(home, 'fleet', 'roster.yaml.mutation.lock'));
await expect(release()).rejects.toThrow();
});
it('preserves a complete rebuild result and flags lock-cleanup when release faults', async (): Promise<void> => {
// A lock-release fault after a successful write must NOT discard the fact that
// projections are now live — the operator needs to know the write happened and
// that the shared lock may be stale, not just a bare "regen failed".
const home = await fleetHome();
const result = await executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
acquireReconcileLock:
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated lock unlink fault');
},
},
{ write: true },
);
expect(result.written).toBe(2);
expect(result.incomplete).toBeUndefined();
expect(result.cleanup).toMatchObject({
code: 'lock-cleanup-failed',
action: 'inspect-lock-before-retry',
});
// Projections are genuinely on disk despite the release fault.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(true);
const report = formatFleetRegenReport(result).join('\n');
expect(report).toMatch(/lock/i);
expect(report).toMatch(/stale|inspect|clear/i);
});
it('preserves a partial rebuild AND flags lock-cleanup when both apply and release fault', async (): Promise<void> => {
// Worst case: a mid-loop apply fault leaves a partial rebuild, then the release
// also faults. Both recovery states must survive so the operator sees exactly
// which projections are live and that the lock may be stale.
const home = await fleetHome();
const realApply = applyPreparedGeneratedAgentEnvironmentProjection;
const result = await executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
applyProjection: async (prepared): Promise<string> => {
if (prepared.generatedPath.endsWith('coder1.env.generated')) {
throw new Error('simulated ENOSPC on coder1');
}
return realApply(prepared);
},
acquireReconcileLock:
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated lock unlink fault');
},
},
{ write: true },
);
expect(result.written).toBe(1);
expect(result.incomplete).toMatchObject({
code: 'projection-apply-failed',
failedAgent: 'coder1',
writtenAgents: ['coder0'],
});
expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' });
const report = formatFleetRegenReport(result).join('\n');
expect(report).toMatch(/incomplete/i);
expect(report).toMatch(/lock/i);
});
it('fails closed without deleting a REPLACEMENT roster-mutation lock it no longer owns', async (): Promise<void> => {
// Finding M1 (replacement-lock race): after this invocation created its lock, the
// lock is cleared and re-created by ANOTHER writer (new inode + foreign token)
// BEFORE release runs. The release must PROVE ownership (device/inode + token) and
// refuse to unlink the stranger's live lock — otherwise it deletes it, a third
// writer enters concurrently, and mutual exclusion over the projections is broken.
// This pins the REACHABLE window (replaced-before-release); the residual sub-window
// between the final check and the path `unlink` is unreachable within the `wx`
// writer protocol and matches the merged reconcile lock — see the acquirer doc.
const home = await fleetHome();
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
const release = await acquirePrivateRosterMutationLock(home)();
// A different writer replaces the lock: same path, NEW inode, foreign content.
await rm(lockPath);
await writeFile(lockPath, 'a-different-writer\n', { mode: 0o600 });
const replacement = await stat(lockPath);
// Release must reject (it no longer owns this file) AND the diagnosis must name
// the MUTATION lock specifically — a fault on this shared helper must not be
// mislabeled as the reconcile lock (that would defeat the M3 accurate-diagnosis
// goal, since regen serializes on both) ...
await expect(release()).rejects.toThrow(/roster\.yaml\.mutation\.lock/);
// ... and MUST NOT have unlinked the stranger's live lock.
expect(await exists(lockPath)).toBe(true);
expect((await stat(lockPath)).ino).toBe(replacement.ino);
});
it('does not strand the lock file when initialization fails after creating it', async (): Promise<void> => {
// Codex r4: if writeFile/stat/close/ownership-check throws AFTER the `wx` create
// succeeds, the just-created lock must NOT be left behind — a stranded lock would
// permanently block future regen AND agent CRUD (both contend on this path). The
// cleanup is dev/ino-guarded (never deletes a replacement) and best-effort (never
// masks the initialization error).
const home = await fleetHome();
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
// Inject an opener that performs the REAL `wx` create (so the lock file really
// lands on disk) but whose handle faults on the token write — a post-create
// initialization failure.
const faultingOpen = (async (
path: Parameters<typeof open>[0],
flags: Parameters<typeof open>[1],
mode: Parameters<typeof open>[2],
) => {
const handle = await open(path, flags, mode);
return new Proxy(handle, {
get(target, prop, receiver): unknown {
if (prop === 'writeFile') {
return async (): Promise<never> => {
throw new Error('simulated ENOSPC on token write');
};
}
const value = Reflect.get(target, prop, receiver);
return typeof value === 'function' ? value.bind(target) : value;
},
});
}) as typeof open;
await expect(acquirePrivateRosterMutationLock(home, faultingOpen)()).rejects.toThrow(
/mutation\.lock/,
);
// The lock we created must have been cleaned up — not stranded on disk.
expect(await exists(lockPath)).toBe(false);
});
it('does not strand the lock file when the post-create stat itself fails', async (): Promise<void> => {
// Codex r5: the narrower sub-path where `handle.stat()` ITSELF rejects right after
// the `wx` create succeeds (transient EIO/EBADF). device/inode identity is then
// unavailable, so the init-failure cleanup cannot prove ownership by dev/ino — yet
// the lock must STILL not be stranded, because a stranded lock permanently blocks
// future regen AND agent CRUD. The token we persisted before the failure uniquely
// identifies OUR lock, so cleanup falls back to matching it.
const home = await fleetHome();
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
// Real `wx` create (lock really lands on disk); the token write SUCCEEDS so our
// random token is persisted, then `stat()` faults — a post-create init failure.
const faultingStatOpen = (async (
path: Parameters<typeof open>[0],
flags: Parameters<typeof open>[1],
mode: Parameters<typeof open>[2],
) => {
const handle = await open(path, flags, mode);
return new Proxy(handle, {
get(target, prop, receiver): unknown {
if (prop === 'stat') {
return async (): Promise<never> => {
throw new Error('simulated EIO on post-create stat');
};
}
const value = Reflect.get(target, prop, receiver);
return typeof value === 'function' ? value.bind(target) : value;
},
});
}) as typeof open;
await expect(acquirePrivateRosterMutationLock(home, faultingStatOpen)()).rejects.toThrow(
/mutation\.lock/,
);
// Even without dev/ino, the token-based fallback must have removed our lock.
expect(await exists(lockPath)).toBe(false);
});
it('surfaces a lock-cleanup fault when unwinding after a later lock acquire fails', async (): Promise<void> => {
// Finding M2: the FIRST lock is taken, the SECOND lock's acquire throws, and
// unwinding the first lock's release ALSO faults. The acquire-unwind path must
// surface both — the already-taken lock may now be stale and silently block the
// next regen/reconcile — not just re-throw the acquire error and drop the fault.
const home = await fleetHome();
await expect(
executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
// Mutation lock (acquired first) succeeds, but its RELEASE faults on unwind.
acquireRosterMutationLock:
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated mutation-lock unlink fault');
},
// Reconcile lock (acquired second) ACQUIRE fails, triggering the unwind.
acquireReconcileLock: () => async (): Promise<() => Promise<void>> => {
throw new Error('simulated reconcile acquire failure');
},
},
{ write: true },
),
).rejects.toThrow(/stale|inspect|lock/i);
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
});
it('names BOTH fleet locks in the stale-lock warning (cleanup can come from either)', async (): Promise<void> => {
// Finding M3: finding L made the roster MUTATION lock's release fault reachable, so
// the cleanup marker can now originate from either lock. The human report must not
// claim only the reconcile lock is stale — it must name both candidate lock files
// so the operator inspects the right one.
const home = await fleetHome();
const result = await executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
// Fault the MUTATION lock's release specifically (reconcile lock is real).
acquireRosterMutationLock:
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated mutation-lock unlink fault');
},
},
{ write: true },
);
expect(result.written).toBe(2);
expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' });
const report = formatFleetRegenReport(result).join('\n');
expect(report).toContain('roster.yaml.mutation.lock');
expect(report).toContain('roster.yaml.reconcile.lock');
});
});

View File

@@ -1,441 +0,0 @@
import { readFile, stat } from 'node:fs/promises';
import { homedir } from 'node:os';
import { join, relative, resolve } from 'node:path';
import type { Command } from 'commander';
import type { CommandRunner } from './fleet.js';
import {
applyPreparedGeneratedAgentEnvironmentProjection,
prepareGeneratedAgentEnvironmentProjection,
type PreparedGeneratedAgentEnvironmentProjection,
} from '../fleet/generated-env-boundary.js';
import {
acquirePrivateReconcileLock,
acquirePrivateRosterMutationLock,
projectRosterV2AgentGeneratedEnv,
} from '../fleet/fleet-reconciler.js';
import {
parseRosterV2,
validateRosterV2Semantics,
type FleetRosterV2,
} from '../fleet/roster-v2.js';
/**
* `mosaic fleet regen` — recovery-framed regeneration of the roster-derived
* agent env projections (`fleet/agents/<name>.env.generated`) from the roster
* SSOT. It is the recovery layer of #791: when a wiped/diverged operator surface
* has cost an agent its generated projection, regen rebuilds it deterministically
* from `roster.yaml` so the launcher can source the intended identity again.
*
* It is intentionally a THIN projection-only wrapper over the same mapping the
* reconciler apply path uses ({@link projectRosterV2AgentGeneratedEnv}), and it
* has NO code path to systemd lifecycle: regen never starts, stops, or restarts
* an agent. Recovery order forbids restart-before-verify, so the operator must
* verify each rebuilt projection resolves before restarting anything.
*/
export interface FleetRegenAgentPlan {
readonly name: string;
/** Generated-projection path, relative to the Mosaic home (never absolute in output). */
readonly path: string;
/** `create` when the projection is currently absent, `rebuild` when it already exists. */
readonly disposition: 'create' | 'rebuild';
}
/**
* Present ONLY when a `--write` projection apply failed partway through the
* fleet. Every prior agent was already validated and prepared, so `writtenAgents`
* are byte-complete on disk; `failedAgent` is where the write faulted. The
* command surfaces this instead of a bare throw so the operator can finish the
* rebuild and verify every projection before restarting any unit.
*/
export interface FleetRegenIncomplete {
readonly code: 'projection-apply-failed';
/** The agent whose generated-projection write faulted. */
readonly failedAgent: string;
/** Agents whose projections were fully written before the fault (in apply order). */
readonly writtenAgents: readonly string[];
}
export interface FleetRegenResult {
readonly mode: 'dry-run' | 'write';
/** Roster path, relative to the Mosaic home. */
readonly rosterPath: string;
readonly generation: number;
readonly agentCount: number;
/** Count of projections written to disk (always 0 in dry-run). */
readonly written: number;
readonly agents: readonly FleetRegenAgentPlan[];
/** Set ONLY when a `--write` apply faulted mid-fleet — a partial rebuild. */
readonly incomplete?: FleetRegenIncomplete;
/**
* Set ONLY when a shared fleet lock could not be released after a write completed
* (or partially completed) — EITHER `roster.yaml.mutation.lock` or
* `roster.yaml.reconcile.lock`, since regen holds both. The projections in this
* result are real; the lock file may be stale and must be inspected before the
* next mutation. Independent of {@link incomplete} — both can be present at once.
*/
readonly cleanup?: {
readonly code: 'lock-cleanup-failed';
readonly action: 'inspect-lock-before-retry';
};
}
export interface FleetRegenDeps {
/**
* Present ONLY so an accidental future lifecycle wiring is caught by the
* "never restarts" test — regen is contractually forbidden to invoke it.
*/
readonly runner: CommandRunner;
readonly mosaicHome?: string;
/** Persona roots for semantic roster validation (defaults mirror the reconciler). */
readonly rolesDir?: string;
readonly overrideDir?: string;
/** Test seam: canonical roster reader. Defaults to parse + semantic validation. */
readonly readRoster?: (rosterPath: string) => Promise<FleetRosterV2>;
/**
* Test seam: generated-only projection validator. Defaults to the audited
* boundary helper that validates ONLY `<name>.env.generated`.
*/
readonly prepareProjection?: typeof prepareGeneratedAgentEnvironmentProjection;
/** Test seam: generated-only projection writer. Defaults to the audited boundary helper. */
readonly applyProjection?: typeof applyPreparedGeneratedAgentEnvironmentProjection;
/**
* Test seam: acquire the shared reconcile lock before a `--write`. Defaults to
* the reconciler's private lock so regen and reconcile are mutually exclusive
* and a concurrent reconcile cannot race a stale projection write.
*/
readonly acquireReconcileLock?: (mosaicHome: string) => () => Promise<() => Promise<void>>;
/**
* Test seam: acquire the shared roster mutation lock before a `--write`.
* Defaults to the reconciler's hardened, ownership-proving acquirer for the
* same `fleet/roster.yaml.mutation.lock` path that CRUD contends on, so regen
* serializes against agent create/update/delete — both rewrite the same
* generated projections, so without this a concurrent CRUD could race a stale
* projection write.
*/
readonly acquireRosterMutationLock?: (mosaicHome: string) => () => Promise<() => Promise<void>>;
/** Test seam: existence probe for create/rebuild disposition. */
readonly fileExists?: (path: string) => Promise<boolean>;
}
interface FleetRegenOptions {
readonly write?: boolean;
}
function defaultMosaicHome(deps: FleetRegenDeps): string {
return deps.mosaicHome ?? process.env['MOSAIC_HOME'] ?? join(homedir(), '.config', 'mosaic');
}
async function defaultFileExists(path: string): Promise<boolean> {
try {
await stat(path);
return true;
} catch {
return false;
}
}
/**
* Projection-only regeneration core. Deterministic and side-effect-free in
* dry-run; in `--write` it rebuilds each generated projection through the
* audited generated-only boundary writer and NOTHING else — it never writes
* `.env.local`/`.env.quarantine`, never unlinks the legacy `.env`, and never
* touches `deps.runner`.
*
* Fail-closed by construction: the roster is validated (structural + semantic)
* and EVERY agent's projection is prepared before ANY is written, so a single
* invalid agent (or a semantically rejected roster) leaves the whole surface
* untouched. In `--write` mode the read-prepare-apply sequence runs under the
* shared reconcile lock so a concurrent reconcile cannot race a stale write.
*/
export async function executeFleetRegen(
deps: FleetRegenDeps,
options: FleetRegenOptions,
): Promise<FleetRegenResult> {
const mosaicHome = defaultMosaicHome(deps);
const agentEnvDir = join(mosaicHome, 'fleet', 'agents');
const rosterPath = join(mosaicHome, 'fleet', 'roster.yaml');
const readRoster = deps.readRoster ?? defaultReadRoster(deps, mosaicHome);
const prepare = deps.prepareProjection ?? prepareGeneratedAgentEnvironmentProjection;
const apply = deps.applyProjection ?? applyPreparedGeneratedAgentEnvironmentProjection;
const fileExists = deps.fileExists ?? defaultFileExists;
const acquireReconcileLock = deps.acquireReconcileLock ?? acquirePrivateReconcileLock;
const acquireRosterMutationLock =
deps.acquireRosterMutationLock ?? acquirePrivateRosterMutationLock;
const write = options.write === true;
const run = async (): Promise<FleetRegenResult> => {
const roster = await readRoster(rosterPath);
// Prepare (validate) every agent BEFORE any write, so a later failure never
// leaves an earlier projection partially rebuilt.
const prepared: (PreparedGeneratedAgentEnvironmentProjection & {
readonly name: string;
readonly disposition: 'create' | 'rebuild';
})[] = [];
for (const agent of roster.agents) {
const projection = await prepare({
mosaicHome,
agentEnvDir,
agentName: agent.name,
generated: projectRosterV2AgentGeneratedEnv(roster, agent),
});
const disposition = (await fileExists(projection.generatedPath)) ? 'rebuild' : 'create';
prepared.push({ ...projection, name: agent.name, disposition });
}
// Every projection is already validated (prepared); an apply here can only
// fault on the underlying I/O (ENOSPC, EIO, a race outside the lock). If a
// later write faults, earlier agents are already replaced on disk, so we
// stop and report the partial state rather than throwing a bare error that
// hides which projections are now live.
let written = 0;
let incomplete: FleetRegenIncomplete | undefined;
if (write) {
for (const projection of prepared) {
try {
await apply(projection);
} catch {
incomplete = {
code: 'projection-apply-failed',
failedAgent: projection.name,
writtenAgents: prepared.slice(0, written).map((entry) => entry.name),
};
break;
}
written += 1;
}
}
return {
mode: write ? 'write' : 'dry-run',
rosterPath: relative(mosaicHome, rosterPath),
generation: roster.generation,
agentCount: roster.agents.length,
written,
agents: prepared.map((projection) => ({
name: projection.name,
path: relative(mosaicHome, projection.generatedPath),
disposition: projection.disposition,
})),
...(incomplete ? { incomplete } : {}),
};
};
// Dry-run is preview-only and never mutates, so it stays lock-free (usable even
// while a reconcile or CRUD holds a lock). A `--write` must serialize against
// BOTH roster writers of the same generated projections: agent CRUD (roster
// mutation lock) and reconcile (reconcile lock). Acquire in a fixed order — the
// locks are non-blocking (they throw on contention rather than wait), so no
// deadlock is possible — and fail closed if either is already held.
if (!write) return run();
const releases: (() => Promise<void>)[] = [];
try {
releases.push(await acquireRosterMutationLock(mosaicHome)());
releases.push(await acquireReconcileLock(mosaicHome)());
} catch (error: unknown) {
// A later acquire failed; unwind any lock already taken (reverse order). If
// that unwind ALSO faults, surface both — the already-taken lock may now be
// stale and silently block the next regen/reconcile — rather than dropping it.
const releaseFault = await releaseFleetLocks(releases);
throw augmentWithLockCleanupFault(error, releaseFault);
}
let result: FleetRegenResult | undefined;
let primaryError: unknown;
try {
result = await run();
} catch (error: unknown) {
primaryError = error;
}
// Release both locks (reverse acquire order), continuing past a fault so neither
// leaks. A release fault must never discard a completed (or partial) rebuild —
// the projections are already on disk — and must not be silently hidden even
// when the run itself failed with nothing written.
const releaseFault = await releaseFleetLocks(releases);
if (result) {
return releaseFault === undefined
? result
: {
...result,
cleanup: { code: 'lock-cleanup-failed', action: 'inspect-lock-before-retry' },
};
}
// result is undefined ⇒ run() threw ⇒ primaryError is set.
throw augmentWithLockCleanupFault(primaryError, releaseFault);
}
/**
* Releases held locks in reverse acquire order, continuing past a fault so a
* single failed release never leaks the others. Returns the first fault seen (or
* undefined when every release succeeded).
*/
async function releaseFleetLocks(releases: readonly (() => Promise<void>)[]): Promise<unknown> {
let firstFault: unknown;
for (const release of [...releases].reverse()) {
try {
await release();
} catch (error: unknown) {
if (firstFault === undefined) firstFault = error;
}
}
return firstFault;
}
/**
* When a run failed with nothing written AND lock cleanup also faulted, surface
* both: the operator needs to know the lock may be stale (else the next
* regen/reconcile is silently blocked), not just the original run error.
*/
function augmentWithLockCleanupFault(primaryError: unknown, releaseFault: unknown): unknown {
if (releaseFault === undefined) return primaryError;
const base = primaryError instanceof Error ? primaryError.message : String(primaryError);
return new Error(
`${base} (additionally, a fleet lock could not be released — ` +
'fleet/roster.yaml.mutation.lock or roster.yaml.reconcile.lock may be stale; ' +
'inspect and clear it before retry)',
);
}
function defaultReadRoster(
deps: FleetRegenDeps,
mosaicHome: string,
): (rosterPath: string) => Promise<FleetRosterV2> {
return async (rosterPath: string): Promise<FleetRosterV2> => {
const roster = parseRosterV2(await readFile(rosterPath, 'utf8'), 'yaml');
// Enforce the SAME semantic gate as reconcile/plan/verify (persona resolution
// + protected-class tool-policy match) so regen cannot project a roster the
// rest of the fleet surface would reject.
await validateRosterV2Semantics(roster, {
rolesDir: deps.rolesDir ?? join(mosaicHome, 'fleet', 'roles'),
overrideDir: deps.overrideDir ?? join(mosaicHome, 'fleet', 'roles.local'),
});
return roster;
};
}
/**
* Human-readable report — paths and counts ONLY. The rendered projection body
* (KEY=value lines) is never echoed (secrev). In `--write` mode it prints the
* recovery runbook: verify each projection resolves BEFORE any restart.
*/
export function formatFleetRegenReport(result: FleetRegenResult): string[] {
const lines: string[] = [];
const header =
result.mode === 'dry-run'
? 'mosaic fleet regen — dry run (no changes written)'
: 'mosaic fleet regen — wrote roster-derived projections';
lines.push(header);
lines.push(
`roster: ${result.rosterPath} (generation ${result.generation}, ${result.agentCount} agent(s))`,
);
for (const agent of result.agents) {
lines.push(` ${agent.name} ${agent.path} [${agent.disposition}]`);
}
if (result.mode === 'dry-run') {
lines.push(
`Plan: would write ${result.agentCount} generated projection(s). ` +
`Re-run with --write to apply. regen never restarts agents.`,
);
return lines;
}
if (result.incomplete) {
const { failedAgent, writtenAgents } = result.incomplete;
lines.push(
`INCOMPLETE: rebuild stopped at agent ${failedAgent} — its projection write faulted.`,
);
lines.push(
`Rebuilt before the fault: ${writtenAgents.length} projection(s)` +
(writtenAgents.length > 0 ? ` (${writtenAgents.join(', ')})` : '') +
`; ${result.agentCount - writtenAgents.length} not written.`,
);
lines.push(
'Recover: re-run `mosaic fleet regen --write` to finish, then VERIFY every ' +
"agent's fleet/agents/<name>.env.generated resolves BEFORE restarting any " +
'unit. regen never restarts agents.',
);
} else {
lines.push(`Wrote ${result.written} generated projection(s).`);
lines.push('Next steps — DO NOT restart agents before verifying:');
lines.push(
" 1. Confirm each agent's fleet/agents/<name>.env.generated exists and carries " +
'the intended MOSAIC_AGENT_* values.',
);
// The unit sets NO EnvironmentFile= — it launches from a minimal env and the
// launcher (start-agent-session.sh) sources .env.generated itself. Verify the
// launcher path, not a nonexistent EnvironmentFile property.
lines.push(
' 2. Confirm the unit launches from it: `systemctl --user cat mosaic-agent@<name>` ' +
'shows ExecStart running start-agent-session.sh, which reads .env.generated.',
);
lines.push(
' 3. Only then restart one unit at a time: systemctl --user restart mosaic-agent@<name>.',
);
}
if (result.cleanup) {
lines.push(
'WARNING: a fleet lock could not be released — ' +
'fleet/roster.yaml.mutation.lock or fleet/roster.yaml.reconcile.lock may be ' +
'stale; inspect and clear it before the next reconcile or regen.',
);
}
return lines;
}
function resolveRegenMosaicHome(fleetCommand: Command, deps: FleetRegenDeps): string {
const options = fleetCommand.optsWithGlobals<{ mosaicHome?: string }>();
return options.mosaicHome ?? defaultMosaicHome(deps);
}
/**
* regen reads the roster SSOT at `<mosaicHome>/fleet/roster.yaml` and ignores the
* `fleet --roster <path>` global. If an operator passes a non-canonical `--roster`
* they must NOT be silently served the canonical file — mirror reconcile's guard
* and reject, so `mosaic fleet --roster /elsewhere regen --write` fails closed.
*/
function assertRegenCanonicalRoster(fleetCommand: Command, mosaicHome: string): void {
const options = fleetCommand.optsWithGlobals<{ roster?: string }>();
const canonical = join(mosaicHome, 'fleet', 'roster.yaml');
if (options.roster !== undefined && resolve(options.roster) !== resolve(canonical)) {
throw new Error(
'Roster-v2 regeneration requires the canonical roster path (fleet/roster.yaml).',
);
}
}
/** Registers the recovery-framed `mosaic fleet regen` command. */
export function registerFleetRegenCommand(fleetCommand: Command, deps: FleetRegenDeps): void {
fleetCommand
.command('regen')
.description(
'Regenerate roster-derived agent env projections (dry-run default, --write to apply; never restarts)',
)
.option('--write', 'Write the regenerated projections to disk (default: dry-run)')
.option('--json', 'Emit the plan/result as JSON')
.action(async (opts: { write?: boolean; json?: boolean }): Promise<void> => {
const mosaicHome = resolveRegenMosaicHome(fleetCommand, deps);
try {
assertRegenCanonicalRoster(fleetCommand, mosaicHome);
const result = await executeFleetRegen(
{ ...deps, mosaicHome },
{ write: opts.write === true },
);
if (opts.json === true) {
console.log(JSON.stringify(result));
} else {
for (const line of formatFleetRegenReport(result)) console.log(line);
}
// A partial rebuild or a stale-lock cleanup state is a non-zero outcome:
// the operator must finish/verify (and clear the lock) before restarting.
if (result.incomplete || result.cleanup) process.exitCode = 1;
} catch (error: unknown) {
process.exitCode = 1;
const message = error instanceof Error ? error.message : String(error);
process.stderr.write(`mosaic fleet regen failed: ${message}\n`);
}
});
}

View File

@@ -97,7 +97,6 @@ describe('registerFleetCommand', () => {
'provision',
'ps',
'reconcile',
'regen',
'remove',
'restart',
'start',

View File

@@ -44,7 +44,6 @@ import {
registerFleetReconcilerCommands,
type FleetReconcilerCommandDeps,
} from './fleet-reconciler-command.js';
import { registerFleetRegenCommand } from './fleet-regen-command.js';
import { resolveCommsBlock } from '../fleet/comms-onboarding.js';
import {
applyPreparedAgentEnvironmentProjection,
@@ -2064,18 +2063,6 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
reconcileDeps: deps.reconcileDeps,
});
// Recovery (#791 PR3): rebuild roster-derived env projections from the roster
// SSOT. Projection-only and preview-first — never issues a lifecycle restart.
registerFleetRegenCommand(cmd, {
runner,
mosaicHome: deps.mosaicHome,
// Resolve personas the SAME way reconcile does: forward any configured roots
// so a custom-persona-root deployment cannot have reconcile accept a roster
// that regen then rejects against the default `<mosaicHome>/fleet/roles`.
rolesDir: deps.reconcileDeps?.rolesDir,
overrideDir: deps.reconcileDeps?.overrideDir,
});
return cmd;
}

View File

@@ -381,22 +381,6 @@ function isMissingFile(error: unknown): boolean {
return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT';
}
/**
* Non-blocking acquire of the roster mutation lock: an exclusive `wx` create that
* throws `concurrent-mutation` (never waits) if the lock is already held. The
* on-disk format is an empty private file. CRUD releases in a plain `finally` with
* no result-preservation logic, so a release fault here must not override the
* mutation result — the release swallows unlink failures by design.
*
* The recovery-framed `fleet regen` contends on this exact same lock file, but it
* does so through the hardened, ownership-proving acquirer in the reconciler
* (`acquirePrivateRosterMutationLock`), not this one: whoever wins the `wx` create
* owns the file (the loser always gets `concurrent-mutation`), so the reconciler's
* ownership token is only ever written and read back by the same regen invocation,
* never by this empty-file writer. The two acquirers stay mutually compatible at
* the `wx`-contention layer while regen additionally proves ownership before it
* unlinks — a guarantee CRUD does not need because it releases unconditionally.
*/
async function acquireMutationLock(lockPath: string): Promise<() => Promise<void>> {
try {
const handle = await open(lockPath, 'wx', 0o600);

View File

@@ -584,29 +584,6 @@ function defaultValidateRoster(
};
}
/**
* The single roster-v2 → generated-projection mapping. Both the reconciler's
* apply path ({@link defaultPrepareProjections}) and the recovery-framed
* `mosaic fleet regen` command derive their generated env from THIS one
* function, so the two paths can never drift (the #791 single-SSOT invariant).
* Pure: no IO, no clock — a deterministic function of the roster alone.
*/
export function projectRosterV2AgentGeneratedEnv(
roster: FleetRosterV2,
agent: FleetRosterV2Agent,
): Readonly<Record<string, string>> {
return {
MOSAIC_AGENT_NAME: agent.name,
MOSAIC_AGENT_CLASS: agent.className,
MOSAIC_AGENT_RUNTIME: agent.runtime,
MOSAIC_AGENT_MODEL: agent.model,
MOSAIC_AGENT_REASONING: agent.reasoning,
MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy,
MOSAIC_AGENT_WORKDIR: agent.workingDirectory,
MOSAIC_TMUX_SOCKET: roster.tmux.socketName,
};
}
function defaultPrepareProjections(
request: FleetReconcileRequest,
): (roster: FleetRosterV2) => Promise<readonly PreparedAgentEnvironmentProjection[]> {
@@ -619,7 +596,16 @@ function defaultPrepareProjections(
mosaicHome,
agentEnvDir: join(mosaicHome, 'fleet', 'agents'),
agentName: agent.name,
generated: projectRosterV2AgentGeneratedEnv(roster, agent),
generated: {
MOSAIC_AGENT_NAME: agent.name,
MOSAIC_AGENT_CLASS: agent.className,
MOSAIC_AGENT_RUNTIME: agent.runtime,
MOSAIC_AGENT_MODEL: agent.model,
MOSAIC_AGENT_REASONING: agent.reasoning,
MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy,
MOSAIC_AGENT_WORKDIR: agent.workingDirectory,
MOSAIC_TMUX_SOCKET: roster.tmux.socketName,
},
}),
),
);
@@ -634,193 +620,62 @@ function mosaicHomeFor(deps: FleetReconcileDeps): string {
return deps.mosaicHome ?? join(homedir(), '.config', 'mosaic');
}
/** Acquires the private reconcile lock only after proving the canonical managed path. */
/** Acquires a private lock only after proving the canonical managed path. */
export function acquirePrivateReconcileLock(
mosaicHome: string,
openLock: typeof open = open,
): () => Promise<() => Promise<void>> {
return acquirePrivateManagedRosterLock(
mosaicHome,
'roster.yaml.reconcile.lock',
'Another roster reconciliation is in progress.',
openLock,
);
}
/**
* Acquires the private roster MUTATION lock (`roster.yaml.mutation.lock`) with the
* exact same ownership-proof discipline as the reconcile lock. Exposed so the
* recovery-framed `mosaic fleet regen` can serialize its projection rewrites
* against agent CRUD — which holds this very lock while it rewrites the roster and
* the same derived projections — reusing this hardened acquirer rather than
* reimplementing it.
*
* The release PROVES ownership (device/inode + token) before unlinking, so a lock
* that was already cleared and replaced by the time release runs is never deleted
* out from under the new owner (it fails closed as `lock-cleanup-failed` instead),
* and that cleanup fault is propagated to the caller rather than silently dropped.
* CRUD contends on the identical path with a plain empty-file `wx` create; the two
* never co-own it (whoever wins `wx` owns it; the loser gets `concurrent-mutation`),
* so the token this writes is only ever read back by the same regen invocation that
* wrote it.
*
* SCOPE of the guarantee (matches the merged reconcile lock exactly — see
* {@link acquirePrivateManagedRosterLock}): the ownership proof closes the
* *reachable* window — a stale-lock reaper or operator cleared our lock and another
* writer took it BEFORE our release began — not the residual sub-instruction window
* between the final check and the path-based `unlink`. Closing that residual fully
* requires an fd-held advisory lock adopted by every fleet writer (CRUD, reconcile,
* regen), which is a cross-cutting mechanism change out of scope for this
* projection-only recovery command; it is unreachable within the `wx` writer
* protocol regardless (no Mosaic writer removes a lock it does not own, so only
* external interference can vacate our inode mid-release).
*/
export function acquirePrivateRosterMutationLock(
mosaicHome: string,
openLock: typeof open = open,
): () => Promise<() => Promise<void>> {
return acquirePrivateManagedRosterLock(
mosaicHome,
'roster.yaml.mutation.lock',
'Another roster mutation is in progress.',
openLock,
);
}
/**
* Shared body for the ownership-proving managed roster locks. Acquires `lockLeaf`
* under `<mosaicHome>/fleet` with a private `wx` create, writes an ownership token,
* and returns a release that re-proves device/inode + token before unlinking so it
* can never remove a lock it no longer owns. Non-blocking: throws `concurrent-mutation`
* (with `busyMessage`) on contention rather than waiting.
*/
function acquirePrivateManagedRosterLock(
mosaicHome: string,
lockLeaf: string,
busyMessage: string,
openLock: typeof open,
): () => Promise<() => Promise<void>> {
const fleetDir = join(mosaicHome, 'fleet');
const lockPath = join(fleetDir, lockLeaf);
// The message-producing helpers below serve BOTH managed locks, so every fault
// names the actual lock file (`fleet/<leaf>`) — an operator needs to know WHICH
// lock is stale, not a hardcoded "reconciliation lock".
const lockLabel = `fleet/${lockLeaf}`;
const lockPath = join(fleetDir, 'roster.yaml.reconcile.lock');
return async (): Promise<() => Promise<void>> => {
await assertPrivateManagedDirectory(mosaicHome);
await assertPrivateManagedDirectory(fleetDir);
await assertSafeLockLeafIfPresent(lockPath, lockLabel);
await assertSafeLockLeafIfPresent(lockPath);
let handle: FileHandle;
try {
handle = await openLock(lockPath, 'wx', 0o600);
} catch (error: unknown) {
if (isCode(error, 'EEXIST')) {
await assertSafeLockLeafIfPresent(lockPath, lockLabel);
throw new FleetReconcileError('concurrent-mutation', busyMessage);
await assertSafeLockLeafIfPresent(lockPath);
throw new FleetReconcileError(
'concurrent-mutation',
'Another roster reconciliation is in progress.',
);
}
throw new FleetReconcileError('lock-io-failed', `The ${lockLabel} lock cannot be created.`);
throw new FleetReconcileError('lock-io-failed', 'The reconciliation lock cannot be created.');
}
// Identity of the lock WE exclusively created (the `wx` create guaranteed it is
// ours). Captured up front so both the release closure and the init-failure
// cleanup below can prove ownership by device/inode before touching the file.
const created = await handle.stat().catch((): undefined => undefined);
const token = randomUUID();
let tokenPersisted = false;
try {
await handle.writeFile(`${token}\n`, 'utf8');
tokenPersisted = true;
const opened = await handle.stat();
await handle.close();
if (!created)
throw new FleetReconcileError(
'lock-io-failed',
`The ${lockLabel} lock cannot be initialized.`,
);
await assertLockOwnership(
lockPath,
created.dev,
created.ino,
token,
'unsafe-lock',
lockLabel,
);
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'unsafe-lock');
return async (): Promise<void> => {
try {
await assertLockOwnership(
lockPath,
created.dev,
created.ino,
token,
'lock-cleanup-failed',
lockLabel,
);
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed');
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed');
await unlink(lockPath);
} catch (error: unknown) {
if (error instanceof FleetReconcileError) throw error;
throw new FleetReconcileError('lock-cleanup-failed', `The ${lockLabel} cleanup failed.`);
throw new FleetReconcileError(
'lock-cleanup-failed',
'The reconciliation lock cleanup failed.',
);
}
};
} catch (error: unknown) {
await handle.close().catch((): void => {});
// Best-effort: remove the lock THIS invocation created so a transient init
// fault does not strand a lock that would block future regen and agent CRUD.
// dev/ino-guarded so it can never delete a replacement; swallowed so cleanup
// failure never masks the initialization error being surfaced. The token is
// passed only once persisted, so the fallback path (when the post-create stat
// failed and dev/ino is unavailable) can still prove ownership by content.
await removeOwnedLockLeafBestEffort(lockPath, created, tokenPersisted ? token : undefined);
if (error instanceof FleetReconcileError) throw error;
throw new FleetReconcileError(
'lock-io-failed',
`The ${lockLabel} lock cannot be initialized.`,
'The reconciliation lock cannot be initialized.',
);
}
};
}
/**
* Best-effort removal of a managed lock leaf THIS process created, used only on the
* initialization-failure path. Two independent ownership proofs, so a lock that was
* already replaced is never deleted:
*
* - primary: the captured device/inode of the file we exclusively `wx`-created; or
* - fallback: when the post-create stat itself failed (dev/ino unavailable), the
* random `ownershipToken` we persisted — only OUR lock carries it, so a CRUD
* (empty) or a differently-tokened replacement is never removed.
*
* Every fault is swallowed because the caller is already surfacing the initialization
* error and a leftover lock is recoverable via the documented runbook. If BOTH proofs
* are unavailable (stat failed AND the token write never landed) the lock is left in
* place rather than risk deleting another writer's file — a doubly-degenerate case
* requiring two independent fs faults on a just-created fd.
*/
async function removeOwnedLockLeafBestEffort(
lockPath: string,
created: { dev: number; ino: number } | undefined,
ownershipToken: string | undefined,
): Promise<void> {
try {
const current = await lstat(lockPath);
if (!current.isFile() || current.isSymbolicLink()) return;
if (created) {
if (current.dev === created.dev && current.ino === created.ino) {
await unlink(lockPath);
}
return;
}
if (ownershipToken !== undefined) {
const contents = await readFile(lockPath, 'utf8');
if (contents.trim() === ownershipToken) {
await unlink(lockPath);
}
}
} catch {
// Swallowed: leftover lock is recoverable; do not mask the init error.
}
}
async function assertPrivateManagedDirectory(path: string): Promise<void> {
try {
const metadata = await lstat(path);
@@ -836,16 +691,16 @@ async function assertPrivateManagedDirectory(path: string): Promise<void> {
}
}
async function assertSafeLockLeafIfPresent(lockPath: string, lockLabel: string): Promise<void> {
async function assertSafeLockLeafIfPresent(lockPath: string): Promise<void> {
try {
const metadata = await lstat(lockPath);
if (!metadata.isFile() || metadata.isSymbolicLink() || (metadata.mode & 0o077) !== 0) {
throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unsafe.`);
throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unsafe.');
}
} catch (error: unknown) {
if (isCode(error, 'ENOENT')) return;
if (error instanceof FleetReconcileError) throw error;
throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unavailable.`);
throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unavailable.');
}
}
@@ -855,7 +710,6 @@ async function assertLockOwnership(
inode: number,
token: string,
failureCode: 'unsafe-lock' | 'lock-cleanup-failed',
lockLabel: string,
): Promise<void> {
try {
const metadata = await lstat(lockPath);
@@ -866,21 +720,24 @@ async function assertLockOwnership(
metadata.dev !== device ||
metadata.ino !== inode
) {
throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`);
throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.');
}
const handle = await open(lockPath, constants.O_RDONLY | constants.O_NOFOLLOW);
try {
const opened = await handle.stat();
const contents = await handle.readFile({ encoding: 'utf8' });
if (opened.dev !== device || opened.ino !== inode || contents !== `${token}\n`) {
throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`);
throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.');
}
} finally {
await handle.close();
}
} catch (error: unknown) {
if (error instanceof FleetReconcileError) throw error;
throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership cannot be proven.`);
throw new FleetReconcileError(
failureCode,
'The reconciliation lock ownership cannot be proven.',
);
}
}

View File

@@ -204,53 +204,6 @@ export async function prepareAgentEnvironmentProjection(
};
}
/** A generated-only projection validated without touching any legacy/local/quarantine file. */
export interface PreparedGeneratedAgentEnvironmentProjection {
readonly mosaicHome: string;
readonly agentEnvDir: string;
readonly generatedPath: string;
readonly generated: string;
}
/**
* Validates ONLY the roster-derived generated projection for a recovery rebuild.
* Unlike {@link prepareAgentEnvironmentProjection}, this never reads, classifies,
* relocates, or quarantines the legacy `.env` / `.env.local` operator surface — it
* exists so `mosaic fleet regen` has no code path that can mutate anything except
* `<name>.env.generated`. The existing generated file, if present, must already be
* a private regular file.
*/
export async function prepareGeneratedAgentEnvironmentProjection(
options: AgentEnvironmentProjectionOptions,
): Promise<PreparedGeneratedAgentEnvironmentProjection> {
if (!AGENT_NAME.test(options.agentName)) {
throw new AgentEnvBoundaryError('unsafe-agent-name', 'MOSAIC_AGENT_NAME', options.agentName);
}
await validatePrivateProjectionDirectory(options.mosaicHome, options.agentEnvDir);
const generatedPath = join(options.agentEnvDir, `${options.agentName}.env.generated`);
const generated = renderGeneratedAgentEnvironment(options.generated);
await assertPrivateRegularFileIfPresent(generatedPath);
return {
mosaicHome: options.mosaicHome,
agentEnvDir: options.agentEnvDir,
generatedPath,
generated,
};
}
/**
* Applies a generated-only projection: writes ONLY `<name>.env.generated` atomically
* and touches nothing else. It never writes `.env.local`/`.env.quarantine` and never
* unlinks the legacy `.env` — the projection-only recovery guarantee is structural.
*/
export async function applyPreparedGeneratedAgentEnvironmentProjection(
prepared: PreparedGeneratedAgentEnvironmentProjection,
): Promise<string> {
await ensurePrivateProjectionDirectory(prepared.mosaicHome, prepared.agentEnvDir);
await writePrivateAtomically(prepared.generatedPath, prepared.generated);
return prepared.generatedPath;
}
/**
* Validates only the exact generated projection eligible for a roster delete.
* Local overrides, legacy input, and quarantine records are operator-retained and