From 1eb37f6fd084afbdc8c00e4ff5f57d2c89fc2c14 Mon Sep 17 00:00:00 2001 From: Hermes Agent Date: Thu, 16 Jul 2026 21:57:38 -0500 Subject: [PATCH] feat(fleet): add mosaic fleet regen recovery command MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add `mosaic fleet regen`, a projection-only recovery command that rebuilds each `fleet/agents/.env.generated` from the `roster.yaml` SSOT after an upgrade or partial write leaves the generated projections stale or missing. - Dry-run by default; `--write` applies; `--json` for machine output. Reuses the merged reconciler's projection plumbing (projectRosterV2AgentGeneratedEnv + the generated-env boundary) rather than reimplementing fleet logic. - Structurally NEVER issues a lifecycle/restart call — regen recovers config only; a recordingRunner gate proves no runner invocation ever occurs. - Serializes against agent CRUD and reconcile via BOTH fleet locks (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired mutation-then-reconcile and released in reverse; both are non-blocking `wx` locks that throw on contention, so no deadlock is possible. - Hardens the shared managed-lock helper: ownership-proving tokened lock reused for both locks with per-lock fault labels; init-failure cleanup no longer strands a just-created lock (dev/ino guard, with a persisted-token fallback when the post-create stat itself fails); acquire-unwind surfaces a lock cleanup fault instead of dropping it. - Resolves personas the SAME way reconcile does by forwarding configured rolesDir/overrideDir, so a custom-persona-root deployment cannot have reconcile accept a roster that regen rejects. - Report/output is secrev-safe: paths and counts only, never projected values. - Docs: upgrade-safety-and-recovery runbook + fleet-local-canary note. Tests are TDD red-first with co-located specs (regen spec: 26 tests covering dry-run/write dispositions, the never-restarts gate, all lock regressions, and persona-root wiring). A residual check-then-unlink TOCTOU in the lock release remains (byte-identical to the merged reconcile lock; unreachable within the `wx` writer protocol); its true fix is an fd-held advisory lock adopted by all fleet writers, tracked as a separate follow-up. Part of #791 Co-Authored-By: Claude Opus 4.8 --- docs/guides/fleet-local-canary.md | 33 + docs/guides/upgrade-safety-and-recovery.md | 147 +++ .../791-upgrade-config-protection.md | 231 +++++ .../src/commands/fleet-regen-command.spec.ts | 885 ++++++++++++++++++ .../src/commands/fleet-regen-command.ts | 441 +++++++++ packages/mosaic/src/commands/fleet.spec.ts | 1 + packages/mosaic/src/commands/fleet.ts | 13 + packages/mosaic/src/fleet/fleet-agent-crud.ts | 16 + packages/mosaic/src/fleet/fleet-reconciler.ts | 217 ++++- .../src/fleet/generated-env-boundary.ts | 47 + 10 files changed, 1994 insertions(+), 37 deletions(-) create mode 100644 docs/guides/upgrade-safety-and-recovery.md create mode 100644 packages/mosaic/src/commands/fleet-regen-command.spec.ts create mode 100644 packages/mosaic/src/commands/fleet-regen-command.ts diff --git a/docs/guides/fleet-local-canary.md b/docs/guides/fleet-local-canary.md index 9350aab..c98fdba 100644 --- a/docs/guides/fleet-local-canary.md +++ b/docs/guides/fleet-local-canary.md @@ -98,6 +98,39 @@ Expected results: that means the unit ran, not that an agent pane is live. Treat tmux `has-session`, `list-panes`, process tree, and logs as the liveness evidence. +## Recovery — rebuild generated env projections + +Each agent's `~/.config/mosaic/fleet/agents/.env.generated` is a +deterministic projection of `roster.yaml` (the SSOT) that the launcher +(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake +wipes or diverges those projections, rebuild them from the roster with +`mosaic fleet regen` — do NOT restart the affected unit first. + +```bash +mosaic fleet regen # dry-run (default): show create/rebuild plan per agent +mosaic fleet regen --json # same plan, machine-readable +mosaic fleet regen --write # rebuild fleet/agents/.env.generated on disk +``` + +`regen` is projection-only and **never restarts an agent** — it has no path to +systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the +same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts +only (never the projected `KEY=value` body). After `--write`, verify each unit +resolves the intended values before restarting one unit at a time. The unit sets +no `EnvironmentFile=` — `start-agent-session.sh` sources `.env.generated` itself — +so verify the generated file directly and the launcher path, not a nonexistent +`EnvironmentFile` property: + +```bash +test -f ~/.config/mosaic/fleet/agents/.env.generated +systemctl --user cat mosaic-agent@ | grep ExecStart +systemctl --user restart mosaic-agent@ +``` + +Full recovery runbook and the three-layer #791 protection model (manifest +ownership → pre-update snapshot/restore → regen): see +[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md). + ## Release Preflight Run this checklist before cutting or dogfooding a fleet release: diff --git a/docs/guides/upgrade-safety-and-recovery.md b/docs/guides/upgrade-safety-and-recovery.md new file mode 100644 index 0000000..1f755cb --- /dev/null +++ b/docs/guides/upgrade-safety-and-recovery.md @@ -0,0 +1,147 @@ +# Upgrade Safety & Recovery + +How Mosaic protects operator-owned configuration under `~/.config/mosaic` across +framework upgrades, and how to recover if a projection is ever lost. + +A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`, +`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident +this hardening addresses: an upgrade that silently overwrites or deletes a file +the operator owns — credentials, personas, a roster, or a generated agent env — +with no snapshot to fall back to. + +Protection is layered. Each layer is independent; a later layer catches what an +earlier one misses. + +## Layer 1 — Manifest-owned sync (prevention) + +The single source of truth for ownership is +[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt). +Both the bash installer and the TypeScript sync path resolve every path against +this one file (parity is enforced by test), so they can never drift. + +- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a + `[framework]` glob matches and no `[operator]` carve-out overrides it. +- **Unknown paths default to operator** (fail-safe): a file the manifest never + anticipated is treated as operator-owned and is never pruned. +- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that + only ever iterates framework globs — operator and unknown paths are + structurally unreachable by the prune. + +Result: a correct upgrade cannot touch operator config at all. + +## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback) + +Before **any** mutation, the installer snapshots the operator-owned surface that +exists into: + +``` +${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-/ +``` + +- `0700` directories / `0600` files (`umask 077`, scoped and restored), + outside `~/.config/mosaic` and outside any repo. +- **Fail-open**: a snapshot failure warns but never aborts the upgrade it + protects. +- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5). + +After the sync, a **verify net** compares each snapshot file against its target +and restores (with a loud warning) any operator file the upgrade diverged or +removed — a divergence means a manifest bug slipped through Layer 1. + +Inspect and restore snapshots with the CLI: + +```bash +mosaic restore --list # dry-run: enumerate snapshots by timestamp +mosaic restore --from # restore the operator surface from one snapshot +mosaic restore --from --dry-run # preview a specific restore without writing +``` + +`mosaic restore` reports **counts and relative paths only** — it never emits file +contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores +are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf +atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails +closed rather than following out of the managed tree). + +## Layer 3 — Regeneration from roster SSOT (recovery) + +Some operator files are **derived** and do not need a byte-for-byte snapshot to +recover — they can be rebuilt from their source of truth. The fleet's per-agent +generated env projections are the prime case: + +- `~/.config/mosaic/fleet/agents/.env.generated` is a deterministic + projection of `~/.config/mosaic/fleet/roster.yaml`. +- The launcher (`start-agent-session.sh`, invoked by + `mosaic-agent@.service`) sources that generated projection to establish + each agent's identity, runtime, model, and working directory. If it is missing + or wrong, the agent cannot launch with its intended identity. + +`mosaic fleet regen` rebuilds those projections from the roster SSOT: + +```bash +mosaic fleet regen # dry-run (default): show what would be rebuilt +mosaic fleet regen --json # same, machine-readable +mosaic fleet regen --write # rebuild the projections on disk +``` + +- **Dry-run by default.** Nothing is written until you pass `--write`. +- **Deterministic and idempotent** — the projection is a pure function of the + roster, so repeated `--write` runs produce byte-identical files. +- **Projection-only. It never restarts an agent.** Recovery order forbids + restart-before-verify; `regen` has no path to systemd lifecycle at all. +- **It rebuilds only `.env.generated`** — it never writes, relocates, or + deletes the operator-owned `.env` / `.env.local` surface. +- It **validates the roster the same way `reconcile` does** (persona resolution + and protected-class tool-policy match), so a hand-edited or corrupt roster is + rejected rather than projected, and a `--write` takes the shared reconcile + lock so it cannot race a concurrent reconcile. +- Output is **paths and counts only** — the rendered `KEY=value` body is never + echoed. + +`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a +recovered projection matches what a normal reconcile would have written. + +## Recovery runbook — wiped `fleet/agents/*.env.generated` + +If an upgrade (or a manual mistake) has left an agent without its generated +projection, **do not restart the unit first** — a launch against a missing +projection fails closed, and any stale state must be corrected before restart, +not after. + +1. **Prefer a snapshot restore if one exists** (byte-exact operator state): + + ```bash + mosaic restore --list + mosaic restore --from + ``` + +2. **Otherwise regenerate the derived projections from the roster SSOT:** + + ```bash + mosaic fleet regen # confirm the plan (create vs rebuild per agent) + mosaic fleet regen --write # rebuild fleet/agents/.env.generated + ``` + +3. **Verify each unit will resolve the intended runtime/workdir _before_ any + restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal + environment and `start-agent-session.sh` sources `.env.generated` itself, so + verify the generated file directly and confirm the launcher path: + + ```bash + # Confirm fleet/agents/.env.generated exists and carries the intended + # MOSAIC_AGENT_* values (name, runtime, model, workdir, socket). + test -f ~/.config/mosaic/fleet/agents/.env.generated + # Confirm the unit launches the session script that reads it. + systemctl --user cat mosaic-agent@ | grep ExecStart + ``` + +4. **Only then restart, one unit at a time:** + + ```bash + systemctl --user restart mosaic-agent@ + ``` + +## See also + +- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md) +- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md) +- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt) diff --git a/docs/scratchpads/791-upgrade-config-protection.md b/docs/scratchpads/791-upgrade-config-protection.md index c3f7cea..9e7199a 100644 --- a/docs/scratchpads/791-upgrade-config-protection.md +++ b/docs/scratchpads/791-upgrade-config-protection.md @@ -296,3 +296,234 @@ vitest **1252** · restore.spec **30** · durable-snapshot **41** · manifest-gu migration 21. shellcheck clean on all new lines; new test markers mirror the existing `# VERIFY-NET` anchor convention. NOTE: codex self-review does NOT satisfy the independent-review gate — an independent (author≠reviewer) review + durable Gitea Reviewer-of-Record comment is still required before MS-LEAD merges. + +## Session 4 (2026-07-16) — PR2 MERGED, PR3 built (fleet regen — recovery layer) +PR2 (#811) squash-merged → main `31607a4a`; issue #791 stays open (final PR of the 3-PR DAG). Independent +exact-head RoR at `d12c5f78` APPROVE (Gitea cmt 17904); #1882 green; busybox-portable Part 7 control fix +verified in-Alpine. PR3 UNBLOCKED. + +PR3 branch: `feat/791-pr3-fleet-regen` off `origin/main` 31607a4. Same discipline: tests-first red-first, +independent review + durable Gitea RoR BEFORE MS-LEAD runs the queue guard/merge. PR body `Part of #791`. + +### PR3 scope (ratified §4/§7 of design doc) — `mosaic fleet regen` +Projection-only recovery command: rebuilds each `fleet/agents/.env.generated` from `roster.yaml` +(SSOT). Dry-run default; `--write` applies; `--json` machine output. Structural guarantee: NO code path to +systemd lifecycle — **never restarts an agent**. Single-SSOT: reuses `projectRosterV2AgentGeneratedEnv` +(extracted, shared with the reconciler apply path) so regen and reconcile cannot drift. Secrev: paths + +counts only, never the rendered KEY=value body. + +New files: `commands/fleet-regen-command.ts` (+ `.spec.ts`), guide `docs/guides/upgrade-safety-and-recovery.md` +(three-layer model: PR1 manifest ownership → PR2 snapshot/restore → PR3 regen; do-NOT-restart-before-verify +runbook), regen reference added to `docs/guides/fleet-local-canary.md`. Wired in `commands/fleet.ts`. + +### Independent review (3 reviewers: subagent code-reviewer + codex code-review + codex security) → 4 fixes, red-first +- **A · BLOCKER (codex) — regen mutated/deleted legacy operator env.** `applyPreparedAgentEnvironmentProjection` + also writes `.env.local`/`.env.quarantine` and unlinks legacy `.env`. Violated projection-only contract. + **Fix:** NEW generated-only boundary primitives `prepareGeneratedAgentEnvironmentProjection` + + `applyPreparedGeneratedAgentEnvironmentProjection` (write ONLY `.env.generated`). regen now has no + code path that touches `.env`/`.env.local`/`.env.quarantine`. **Test:** projection-only leaves legacy `.env` + verbatim, no local/quarantine fabricated. +- **B · should-fix (codex + subagent + security) — partial write on mid-loop failure.** Interleaved + prepare/apply left earlier agents written when a later agent failed prepare. **Fix:** PREPARE ALL agents + before writing ANY (mirrors reconciler `defaultPrepareProjections`). **Test:** 2nd agent's projection + pre-seeded 0644 → prepare rejects → coder0 NOT written, exit 1. +- **C · subagent — semantic-validation bypass.** Default readRoster skipped `validateRosterV2Semantics`, so + a tampered protected-class `tool_policy` would be silently projected. **Fix:** default readRoster now runs + `validateRosterV2Semantics` (persona resolution + protected-class match), rolesDir/overrideDir defaults + mirroring the reconciler. **Test:** merge-gate agent w/ tool_policy=code → fails closed, no write. +- **D · MEDIUM (codex security, CWE-362) — concurrent-reconcile race.** regen `--write` wrote without the + reconcile lock. **Fix:** `--write` acquires `acquirePrivateReconcileLock(mosaicHome)` for the whole + read-prepare-apply sequence, released in `finally`; dry-run stays lock-free. **Test:** pre-held lock → + regen fails closed, no write. + +**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1265** +(regen spec 13, incl. 4 new red-first regressions). NOTE: codex self-review does NOT satisfy the +independent-review gate — an independent (author≠reviewer) review + durable Gitea RoR is still required +before MS-LEAD merges. STOP at PR-open for MS-LEAD's exact-head review; do NOT self-merge. + +## Session 5 — PR3 review round 2 (finding L + M1/M2/M3), red-first fixes + +Second review pass on the lock-cleanup plumbing surfaced one round-1 residual (L) and three round-2 +findings (M1 blocker, M2/M3 should-fix). All fixed red-first (RED proven per-finding, then GREEN). + +- **L · should-fix (codex r1) — mutation-lock release swallowed unlink failures.** regen's + `acquirePrivateRosterMutationLock` release copied CRUD's `unlink().catch(()=>{})`, hiding a stale + `roster.yaml.mutation.lock`. **Fix:** its release PROPAGATES the unlink fault (finding-J stale-lock + warning then fires for this lock too). **Test:** acquire real lock, `rm` it, assert `release()` rejects. +- **M1 · BLOCKER (codex r2) — replacement-lock race.** The propagating release from L did an + UNCONDITIONAL `unlink(lockPath)` without proving ownership. If the lock is cleared + re-created by + another writer mid-op, regen deletes the STRANGER's live lock → a third writer enters → mutual + exclusion defeated. **Fix (reuse, not reimplement):** generalized the reconciler's ownership-proving + lock body into shared `acquirePrivateManagedRosterLock(mosaicHome, lockLeaf, busyMessage, openLock)`; + `acquirePrivateReconcileLock` delegates to it (behavior-identical: same leaf/codes/messages), and a NEW + hardened `acquirePrivateRosterMutationLock` (now in fleet-reconciler.ts, leaf `roster.yaml.mutation.lock`) + records dev/ino + ownership token and RE-PROVES ownership (`assertLockOwnership`) before unlinking — + fails closed as `lock-cleanup-failed` if replaced. Removed the crud-based export; reverted + `acquireMutationLock` (fleet-agent-crud.ts) to its original inline empty-file/swallowing-release form + (CRUD behavior intentionally unchanged). Compatibility: CRUD empty-file `wx` and regen tokened `wx` + contend on the same path but never co-own (wx winner owns; loser → concurrent-mutation), so the token + is only ever read back by the same regen invocation. **Test:** acquire, `rm`+recreate lock (new inode), + assert `release()` rejects AND the replacement survives (not unlinked). +- **M2 · should-fix (codex r2) — acquire-unwind fault dropped.** The acquire-failure catch discarded + `releaseFleetLocks`' return (a possible fault on the already-held first lock). **Fix:** capture and + augment — `const releaseFault = await releaseFleetLocks(releases); throw augmentWithLockCleanupFault(error, releaseFault);` + (symmetric to finding J). **Test:** mutation lock acquires w/ faulting release + reconcile acquire + throws → thrown error mentions stale/lock, nothing written. +- **M3 · should-fix (codex r2 + subagent REQUEST-CHANGES) — cleanup warning named only reconcile lock.** + Finding L made the mutation-lock release fault reachable, so the `cleanup` marker can originate from + EITHER lock. **Fix:** `formatFleetRegenReport`'s WARNING now names BOTH `roster.yaml.mutation.lock` and + `roster.yaml.reconcile.lock`, matching `augmentWithLockCleanupFault`. **Test:** fault the mutation-lock + release specifically → report names both lock files. + +**Refactor note (no cycle):** neither fleet-reconciler nor fleet-agent-crud imports the other; regen +imports lock acquirers from fleet-reconciler and the projection mapping from fleet-reconciler. The two +reconcile-lock reviewers reconciled: independent reviewer validated acquire-time empty-file compatibility +(preserved), codex flagged RELEASE-time replacement race (closed by ownership proof) — non-contradictory. + +**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1275** +(regen spec 23, incl. 7 red-first lock regressions E/F/G/K/L/M1/M2/M3). RED proven per-finding by +temporary revert before re-applying each fix. Independent (author≠reviewer) review of M1/M2/M3 + codex +code/security re-run in flight. STOP at PR-open for MS-LEAD's exact-head review + durable Gitea RoR; do +NOT self-merge; #791 umbrella stays OPEN; PR body `Part of #791`. + +### Round 3 review (after M1/M2/M3) — independent review PASS + codex residual-TOCTOU disposition +Three reviewers on the post-M1/M2/M3 head: +- **Independent (subagent, author≠reviewer) — PASS.** Verified M1/M2/M3 all correctly fixed; "never + restarts" is STRUCTURAL (runner never referenced in executable code); no secrets; no deadlock (only + regen holds both locks); tests meaningful (assert inode preservation + exact lock-file names). Raised: + - **should-fix #1 (fixed, red-first):** generalizing the lock helper left `assertSafeLockLeafIfPresent`/ + `assertLockOwnership` hardcoding "reconciliation lock" in thrown messages → a MUTATION-lock fault + misreported as the reconcile lock, undercutting M3's accurate-diagnosis goal. **Fix:** thread + `lockLabel = fleet/` through both helpers + the generic lock-io messages, so every fault names + the actual lock file. Red-first: strengthened the M1 test to assert `/roster\.yaml\.mutation\.lock/` + (RED: got "reconciliation lock"; GREEN after). Also resolves nit #3 (generic-message drift). + - **nit #2 (fixed):** `FleetRegenResult.cleanup` JSDoc still said "the shared reconcile lock"; now names + both locks (regen holds both). + - **nit #4 (fixed):** removed the redundant duplicate `assertLockOwnership` call before unlink + (pre-existing in merged main; harmless but dead — dropped since the fn was already being touched). +- **Codex security — clean (risk: none).** Validates roster semantics, constrains env values, no shell + eval, no secret output, generated-only writes, serialized against both locks. +- **Codex code — request-changes, 1 "blocker": residual check-then-unlink TOCTOU.** Between the final + `assertLockOwnership` and the path-based `unlink`, an external actor could vacate our inode and a new + writer grab the path, so the unlink deletes the stranger's lock. **Disposition: documented known + limitation, NOT fixed in PR3.** Rationale: (1) byte-identical to the MERGED, shipped reconcile-lock + release on origin/main (fleet-reconciler.ts L654-659) — not introduced here; (2) UNREACHABLE within the + `wx` writer protocol — no Mosaic writer removes a lock it doesn't own (wx fails EEXIST while our inode + exists), so only external interference can vacate our inode in the sub-instruction window; (3) the + ownership guard DOES close the reachable case (stale-lock reaper/operator cleared our lock + another + writer took it BEFORE release began → fail closed, don't delete stranger's lock); (4) the true atomic + fix — fd-held advisory lock (flock/lockf) adopted by ALL fleet writers (CRUD + reconcile + regen) — is + a cross-cutting mechanism change touching merged CRUD + reconciler, out of scope for a projection-only + recovery PR. Documented honestly in the acquirer doc + M1 test comment. **The binding independent + review did NOT treat this as a blocker.** Recommendation to MS-LEAD: proceed to PR-open + spin a + SEPARATE follow-up issue for the fd-advisory-lock migration; MS-LEAD adjudicates scope at exact-head + review (merge authority). + +**Gates after round-3 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest +**1275** (regen spec 23). Fresh codex code re-run in flight to confirm no NEW issues from the label fix. + +--- + +## Session 6 — Round 4/5 convergence (stranded-lock robustness) + +**Two independent reviewers converged on the SAME should-fix** on the init-failure cleanup path, +strengthening confidence it was real: + +- **Codex code-review-5 — 0 blockers, 1 should-fix.** "Stat failure after lock creation strands the new + lock." When `handle.stat()` ITSELF fails right after the `wx` create (transient EIO/EBADF), `created` + is `undefined`, so `removeOwnedLockLeafBestEffort` had `if (!created) return;` → no cleanup → the + just-created `roster.yaml.mutation.lock`/`reconcile.lock` is stranded, permanently blocking future + regen + CRUD. (Notably NO blocker, and the TOCTOU is no longer flagged in code-review as of r5.) +- **Independent delta reviewer (author≠reviewer, pr-review-toolkit) — no blockers, same should-fix.** + Independently flagged the identical `!created` gap; validated FIX 1 (label threading — no call site + missed, codes unchanged, no test depended on old text) and FIX 2 (dev/ino-guarded cleanup, best-effort, + happy-path release reuses captured dev/ino) as correct. Suggested an unconditional best-effort unlink + in the `!created` branch; I took the **safer** variant below. +- **Codex security-review-5 — 0 crit / 0 high / 1 medium.** The single medium is the SAME residual + check-then-unlink TOCTOU already dispositioned in round 3 (its own remediation = "migrate every writer + to an fd-held advisory lock" = the follow-up issue). No new security finding. No secrets. + +**Fix (red-first, safer than an unconditional unlink):** thread the persisted random `token` into +`removeOwnedLockLeafBestEffort`. Two independent ownership proofs now: primary dev/ino (unchanged), and a +**fallback** when the post-create stat failed — read the leaf and unlink ONLY if its content equals our +`randomUUID()` token. Only OUR lock carries that token, so a CRUD (empty) or differently-tokened +replacement is never deleted. `tokenPersisted` guards passing the token (only after `writeFile` lands). +Doubly-degenerate case (stat fails AND token write never landed) leaves the lock in place rather than +risk deleting a stranger's file — requires two independent fs faults on a just-created fd; documented. + +- **Red-first proof:** new test `does not strand the lock file when the post-create stat itself fails` + injects a real `wx` create + a Proxy handle whose `stat()` rejects (writeFile/close succeed), asserts + `exists(lockPath) === false`. RED before fix (`expected true to be false` — lock stranded); GREEN after. +- **Also fixed (delta nit #3):** `fleet-regen-command.ts` `acquireRosterMutationLock` JSDoc said "CRUD's + private lock"; the default is the reconciler's hardened ownership-proving acquirer for the same + `fleet/roster.yaml.mutation.lock` path. Corrected. +- **PR-description note (delta nit #2):** FIX 1 also collapsed a pre-existing duplicate back-to-back + `assertLockOwnership` call in the release closure (identical args, no intervening logic) into one — a + no-op simplification of merged code, not a behavior change. Called out so a future reader doesn't + wonder if the duplicate had a purpose. + +**Gates after round-4 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest +**1277** (regen spec now 25: +1 stat-failure stranded-lock regression). Residual TOCTOU still deferred to +the fd-advisory-lock follow-up issue; MS-LEAD adjudicates scope at exact-head review (merge authority). + +--- + +## Session 6 — Round 6 (persona-root wiring) + +**Codex code-review-6 — 0 blockers, 1 should-fix (NEW, distinct from the lock work).** "Forward +configured persona directories to regen." `registerFleetRegenCommand` was registered at +`fleet.ts:2069` with only `{ runner, mosaicHome }`, discarding `deps.reconcileDeps.rolesDir` / +`overrideDir`. The regen command ALREADY has those seams (validates roster semantics via +`validateRosterV2Semantics({ rolesDir, overrideDir })`, defaulting to `/fleet/roles{,.local}`), +but the top-level wiring never forwarded the configured roots. **Impact:** in a deployment with custom +persona roots, `fleet reconcile` (which honors the overrides) would ACCEPT a roster while `fleet regen` +REJECTS the same roster (persona resolution against the wrong default dir) — blocking the recovery +command and violating the documented "resolves personas the SAME way reconcile does" contract. + +**Fix (red-first):** forward `rolesDir`/`overrideDir` from `deps.reconcileDeps` into +`registerFleetRegenCommand` at `fleet.ts:2069`. Red-first test `forwards configured persona roots +(rolesDir/overrideDir) from reconcileDeps into regen`: seeds personas ONLY under a custom root, leaves +the default `/fleet/roles` empty, registers with `reconcileDeps: { rolesDir, overrideDir }`, and +requires `fleet regen` to SUCCEED. RED before fix (`expected 1 not to be 1` — regen validated against the +empty default and exited 1); GREEN after. + +**Codex security-review-6 — 0 crit / 0 high / 1 medium.** Same residual check-then-unlink TOCTOU, now +noted at BOTH the release closure and the init-cleanup path; remediation = fd-held advisory lock across +all writers = the SAME deferred follow-up item. No new security finding, no secrets. + +**Independent confirmation review of the token-fallback fix (Session 6/round 4) — PASS, no findings.** +All 7 verification points confirmed; reviewer mechanically reverted `removeOwnedLockLeafBestEffort` to +the pre-fix `if (!created) return;` and re-ran the new test → RED (`expected true to be false`), +confirming the test genuinely pins the fix; restored after. No lint/type issues; doc-comment accurate. + +**Gates after round-6 fix (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest +**1278** (regen spec now 26: +1 persona-root wiring regression). + +--- + +## Session 6 — Round 7 convergence (review CLOSED for PR-open) + +- **Codex code-review-7 — 0 blockers, 1 should-fix = the residual TOCTOU** (previously a "blocker" in r3, + dropped in r4/r5, now re-surfaced as a should-fix). **Codex security-review-7 — 0 crit / 0 high / + 1 medium = the SAME residual TOCTOU.** Codex has CONVERGED: the only remaining finding across both + streams is that one race, whose own remediation is "fd-held advisory lock shared by all fleet writers" + = the deferred follow-up. No new distinct finding; the wiring fix introduced nothing. +- **Independent confirmation review of the persona-root wiring fix — PASS, no findings.** Reviewer + mechanically reverted the two forwarded lines → RED (`Roster v2 agent "coder0" class "code" does not + resolve to a readable persona` → exit 1), restored → GREEN (26 regen + 204 fleet tests). Confirmed the + optional-chaining fallback preserves default-deployment behavior and no type/lint issue. + +**Review disposition for PR-open:** ALL actionable findings fixed red-first across rounds 3–6 (label +threading, stranded-lock on init failure, stat-failure strand, persona-root wiring). The residual +check-then-unlink TOCTOU is the ONLY open item and is DEFERRED to a follow-up issue (fd-advisory-lock +migration across CRUD + reconcile + regen) — byte-identical to merged origin/main's reconcile-lock +release, unreachable within the `wx` writer protocol (no Mosaic writer removes a lock it doesn't own; +only external `rm`/a stale-lock reaper can vacate the inode mid-release), and its true fix is a +cross-cutting mechanism change out of scope for a projection-only recovery PR. Two independent human-agent +reviews (author≠reviewer) treated it as non-blocking. MS-LEAD adjudicates scope at exact-head review +(merge authority); recommendation = proceed to PR-open + spin the follow-up issue. + +**Final gates (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1278** +(regen spec 26). No secret values in any snapshot/projection/report output (counts + paths only). Regen +NEVER issues a lifecycle/restart call (load-bearing recordingRunner gate). STOP at PR-open for MS-LEAD's +exact-head review + durable Reviewer-of-Record before any merge; do NOT self-merge. diff --git a/packages/mosaic/src/commands/fleet-regen-command.spec.ts b/packages/mosaic/src/commands/fleet-regen-command.spec.ts new file mode 100644 index 0000000..8155516 --- /dev/null +++ b/packages/mosaic/src/commands/fleet-regen-command.spec.ts @@ -0,0 +1,885 @@ +import { chmod, mkdir, mkdtemp, open, readFile, rm, stat, writeFile } from 'node:fs/promises'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { Command } from 'commander'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import { registerFleetCommand, type CommandResult, type FleetCommandDeps } from './fleet.js'; +import { executeFleetRegen, formatFleetRegenReport } from './fleet-regen-command.js'; +import { + acquirePrivateRosterMutationLock, + projectRosterV2AgentGeneratedEnv, +} from '../fleet/fleet-reconciler.js'; +import { applyPreparedGeneratedAgentEnvironmentProjection } from '../fleet/generated-env-boundary.js'; +import { parseRosterV2 } from '../fleet/roster-v2.js'; + +// A two-agent roster-v2 SSOT. `mosaic fleet regen` must rebuild each agent's +// `fleet/agents/.env.generated` projection from exactly this source and +// nothing else — deterministically, without ever touching agent lifecycle. +const rosterYaml = ` +version: 2 +generation: 7 +transport: tmux +tmux: + socket_name: mosaic-fleet + holder_session: _holder +defaults: + working_directory: /srv/mosaic + runtime: pi +runtimes: + pi: + reset_command: /new +agents: + - name: coder0 + alias: Coder 0 + class: code + runtime: pi + provider: openai + model: gpt-5.6-sol + reasoning: high + tool_policy: code + working_directory: /srv/mosaic + persistent_persona: false + reset_between_tasks: true + lifecycle: + enabled: true + desired_state: stopped + launch: + yolo: true + - name: coder1 + alias: Coder 1 + class: code + runtime: pi + provider: openai + model: gpt-5.6-sol + reasoning: medium + tool_policy: code + working_directory: /srv/other + persistent_persona: false + reset_between_tasks: true + lifecycle: + enabled: true + desired_state: stopped + launch: + yolo: true +`; + +let cleanup: string | undefined; + +afterEach(async (): Promise => { + vi.restoreAllMocks(); + process.exitCode = undefined; + if (cleanup) await rm(cleanup, { recursive: true, force: true }); + cleanup = undefined; +}); + +async function fleetHome(withRoster = rosterYaml): Promise { + cleanup = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-command-')); + for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) { + await mkdir(join(cleanup, directory), { recursive: true, mode: 0o700 }); + await chmod(join(cleanup, directory), 0o700); + } + await chmod(cleanup, 0o700); + await writeFile(join(cleanup, 'fleet', 'roster.yaml'), withRoster, { mode: 0o600 }); + // Semantic roster validation (matching `fleet reconcile`) resolves each agent + // class to a persona; seed the classes the fixtures reference. + for (const klass of ['code', 'merge-gate']) { + await writeFile( + join(cleanup, 'fleet', 'roles', `${klass}.md`), + `\`class: ${klass}\`\n\n# ${klass} persona\n`, + { mode: 0o600 }, + ); + } + return cleanup; +} + +/** + * A runner spy that records EVERY invocation. `regen` is projection-only and + * must never issue a lifecycle/restart call, so a non-empty call log is a + * hard failure — this is the load-bearing "never restarts" gate. + */ +function recordingRunner( + calls: string[][], +): (command: string, args: string[]) => Promise { + return async (command: string, args: string[]): Promise => { + calls.push([command, ...args]); + return { stdout: '', stderr: '', exitCode: 0 }; + }; +} + +function program(mosaicHome: string, runner: FleetCommandDeps['runner']): Command { + const result = new Command(); + result.exitOverride(); + registerFleetCommand(result, { mosaicHome, runner }); + return result; +} + +function capture(): string[] { + const lines: string[] = []; + vi.spyOn(console, 'log').mockImplementation((value: string): void => { + lines.push(value); + }); + return lines; +} + +async function exists(path: string): Promise { + try { + await stat(path); + return true; + } catch { + return false; + } +} + +describe('projectRosterV2AgentGeneratedEnv', (): void => { + it('maps a roster-v2 agent to exactly the eight generated projection keys', (): void => { + const roster = parseRosterV2(rosterYaml, 'yaml'); + const agent = roster.agents.find((candidate) => candidate.name === 'coder0'); + expect(agent).toBeDefined(); + const values = projectRosterV2AgentGeneratedEnv(roster, agent!); + expect(values).toEqual({ + MOSAIC_AGENT_NAME: 'coder0', + MOSAIC_AGENT_CLASS: 'code', + MOSAIC_AGENT_RUNTIME: 'pi', + MOSAIC_AGENT_MODEL: 'gpt-5.6-sol', + MOSAIC_AGENT_REASONING: 'high', + MOSAIC_AGENT_TOOL_POLICY: 'code', + MOSAIC_AGENT_WORKDIR: '/srv/mosaic', + MOSAIC_TMUX_SOCKET: 'mosaic-fleet', + }); + }); +}); + +describe('mosaic fleet regen', (): void => { + it('is dry-run by default: reports the plan and writes nothing', async (): Promise => { + const home = await fleetHome(); + const calls: string[][] = []; + const lines = capture(); + + await program(home, recordingRunner(calls)).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--json', + ]); + + const result = JSON.parse(lines.pop() ?? '{}'); + expect(result).toMatchObject({ + mode: 'dry-run', + generation: 7, + agentCount: 2, + written: 0, + }); + expect(result.agents.map((agent: { name: string }) => agent.name)).toEqual([ + 'coder0', + 'coder1', + ]); + expect( + result.agents.every((agent: { disposition: string }) => agent.disposition === 'create'), + ).toBe(true); + // Nothing on disk. + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false); + expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false); + // No lifecycle/restart call — ever. + expect(calls).toEqual([]); + }); + + it('forwards configured persona roots (rolesDir/overrideDir) from reconcileDeps into regen', async (): Promise => { + // Codex r6: regen must resolve personas the SAME way reconcile does. If the + // top-level registration drops reconcileDeps.rolesDir/overrideDir, a deployment + // with custom persona roots gets reconcile ACCEPTING a roster while regen + // REJECTS it (validating against the wrong default `/fleet/roles`), + // blocking the recovery command. Pin the wiring: seed personas ONLY under the + // custom roots, leave the default roles dir empty, require regen to succeed. + const home = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-wiring-')); + cleanup = home; + for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) { + await mkdir(join(home, directory), { recursive: true, mode: 0o700 }); + await chmod(join(home, directory), 0o700); + } + await chmod(home, 0o700); + await writeFile(join(home, 'fleet', 'roster.yaml'), rosterYaml, { mode: 0o600 }); + // Personas live ONLY under the configured roots — the default `fleet/roles` + // stays empty, so broken wiring fails persona resolution. + const customRoles = join(home, 'custom-roles'); + const customOverride = join(home, 'custom-roles.local'); + await mkdir(customRoles, { recursive: true, mode: 0o700 }); + await mkdir(customOverride, { recursive: true, mode: 0o700 }); + for (const klass of ['code', 'merge-gate']) { + await writeFile(join(customRoles, `${klass}.md`), `\`class: ${klass}\`\n\n# ${klass}\n`, { + mode: 0o600, + }); + } + + const command = new Command(); + command.exitOverride(); + registerFleetCommand(command, { + mosaicHome: home, + runner: recordingRunner([]), + reconcileDeps: { rolesDir: customRoles, overrideDir: customOverride }, + }); + const lines = capture(); + + await command.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']); + + // Regen validated against the CONFIGURED roots → success. Broken wiring + // resolves against the empty default and fails (exitCode 1, no JSON result). + expect(process.exitCode).not.toBe(1); + const result = JSON.parse(lines.pop() ?? '{}'); + expect(result).toMatchObject({ mode: 'dry-run', agentCount: 2 }); + }); + + it('--write rebuilds each generated projection from the roster SSOT', async (): Promise => { + const home = await fleetHome(); + const calls: string[][] = []; + const lines = capture(); + + await program(home, recordingRunner(calls)).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--write', + '--json', + ]); + + const result = JSON.parse(lines.pop() ?? '{}'); + expect(result).toMatchObject({ mode: 'write', written: 2, agentCount: 2 }); + + const coder0 = await readFile(join(home, 'fleet', 'agents', 'coder0.env.generated'), 'utf8'); + expect(coder0).toContain('MOSAIC_AGENT_NAME=coder0'); + expect(coder0).toContain('MOSAIC_AGENT_WORKDIR=/srv/mosaic'); + expect(coder0).toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet'); + const coder1 = await readFile(join(home, 'fleet', 'agents', 'coder1.env.generated'), 'utf8'); + expect(coder1).toContain('MOSAIC_AGENT_NAME=coder1'); + expect(coder1).toContain('MOSAIC_AGENT_WORKDIR=/srv/other'); + // No lifecycle/restart call — ever. + expect(calls).toEqual([]); + }); + + it('is deterministic and idempotent across repeated --write runs', async (): Promise => { + const home = await fleetHome(); + const calls: string[][] = []; + const path = join(home, 'fleet', 'agents', 'coder0.env.generated'); + + const cli = program(home, recordingRunner(calls)); + capture(); + await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']); + const first = await readFile(path, 'utf8'); + await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']); + const second = await readFile(path, 'utf8'); + + expect(second).toBe(first); + expect(calls).toEqual([]); + }); + + it('reports rebuild disposition once the generated projection already exists', async (): Promise => { + const home = await fleetHome(); + const calls: string[][] = []; + const cli = program(home, recordingRunner(calls)); + const lines = capture(); + + await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']); + lines.length = 0; + await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']); + + const result = JSON.parse(lines.pop() ?? '{}'); + expect( + result.agents.every((agent: { disposition: string }) => agent.disposition === 'rebuild'), + ).toBe(true); + expect(result.written).toBe(0); + expect(calls).toEqual([]); + }); + + it('never issues a lifecycle/restart call in either mode', async (): Promise => { + const home = await fleetHome(); + const calls: string[][] = []; + const cli = program(home, recordingRunner(calls)); + capture(); + + await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']); + await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']); + + // Structural guarantee: regen has no path to systemctl/tmux at all. + expect(calls).toEqual([]); + const systemctlCalls = calls.filter(([command]) => command === 'systemctl'); + expect(systemctlCalls).toEqual([]); + }); + + it('human-readable --write output prints the do-not-restart recovery runbook', async (): Promise => { + const home = await fleetHome(); + const lines = capture(); + + await program(home, recordingRunner([])).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--write', + ]); + + const output = lines.join('\n'); + expect(output).toMatch(/do not restart|before.*restart/i); + expect(output).toMatch(/env\.generated/); + // The unit has no EnvironmentFile= directive, so the runbook must NOT tell the + // operator to verify one — verify the launcher/generated file instead. + expect(output).not.toContain('EnvironmentFile'); + expect(output).toMatch(/systemctl --user cat mosaic-agent@/); + expect(output).toMatch(/restart/i); + }); + + it('emits paths and counts only — never the projected env body (secrev)', async (): Promise => { + const home = await fleetHome(); + const lines = capture(); + + await program(home, recordingRunner([])).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--write', + ]); + + const output = lines.join('\n'); + // Relative paths + counts are fine; the rendered projection body (KEY=value + // lines) must never be echoed to stdout. + expect(output).toContain('coder0.env.generated'); + expect(output).not.toMatch(/^MOSAIC_AGENT_\w+=/m); + expect(output).not.toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet'); + }); + + it('is projection-only: leaves legacy .env untouched and writes no .env.local/.env.quarantine', async (): Promise => { + // Recovery contract: regen rebuilds ONLY .env.generated. It must never + // relocate, quarantine, or unlink the operator-owned legacy .env surface — the + // full reconciler apply path does, so regen must NOT use it. + const home = await fleetHome(); + const legacyPath = join(home, 'fleet', 'agents', 'coder0.env'); + await writeFile(legacyPath, 'MOSAIC_RUNTIME_BIN=/usr/bin/pi\n', { mode: 0o600 }); + capture(); + + await program(home, recordingRunner([])).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--write', + '--json', + ]); + + // Generated projection rebuilt... + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true); + // ...but the operator's legacy .env is preserved verbatim, and no local/quarantine + // files were fabricated from it. + expect(await readFile(legacyPath, 'utf8')).toBe('MOSAIC_RUNTIME_BIN=/usr/bin/pi\n'); + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.local'))).toBe(false); + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.quarantine'))).toBe(false); + }); + + it('prepares every agent before writing any: a later prepare failure leaves nothing written', async (): Promise => { + // Fail-closed across agents. Pre-seed coder1's projection with world-readable + // perms so its prepare rejects; coder0 (valid) must NOT be written because + // preparation is fully completed before the first apply. + const home = await fleetHome(); + const coder1Path = join(home, 'fleet', 'agents', 'coder1.env.generated'); + await writeFile(coder1Path, 'MOSAIC_AGENT_NAME=coder1\n', { mode: 0o644 }); + const calls: string[][] = []; + capture(); + const errors: string[] = []; + vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => { + errors.push(String(chunk)); + return true; + }); + + await program(home, recordingRunner(calls)).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--write', + '--json', + ]); + + expect(process.exitCode).toBe(1); + expect(errors.join('')).toMatch(/regen failed/i); + // coder0 is valid but must remain unwritten — no partial rebuild. + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false); + expect(calls).toEqual([]); + }); + + it('fails closed on a semantically invalid roster (protected-class tool-policy mismatch)', async (): Promise => { + // A hand-edited roster giving a protected class a weaker tool_policy is rejected + // by reconcile/plan/verify; regen must enforce the SAME gate, not silently + // project the downgraded policy into .env.generated. + const home = await fleetHome(` +version: 2 +generation: 3 +transport: tmux +tmux: + socket_name: mosaic-fleet + holder_session: _holder +defaults: + working_directory: /srv/mosaic + runtime: pi +runtimes: + pi: + reset_command: /new +agents: + - name: gate0 + alias: Gate 0 + class: merge-gate + runtime: pi + provider: openai + model: gpt-5.6-sol + reasoning: high + tool_policy: code + working_directory: /srv/mosaic + persistent_persona: false + reset_between_tasks: true + lifecycle: + enabled: true + desired_state: stopped + launch: + yolo: true +`); + const calls: string[][] = []; + capture(); + const errors: string[] = []; + vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => { + errors.push(String(chunk)); + return true; + }); + + await program(home, recordingRunner(calls)).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--write', + '--json', + ]); + + expect(process.exitCode).toBe(1); + expect(errors.join('')).toMatch(/regen failed/i); + expect(await exists(join(home, 'fleet', 'agents', 'gate0.env.generated'))).toBe(false); + expect(calls).toEqual([]); + }); + + it('refuses to write while a concurrent reconcile holds the mutation lock', async (): Promise => { + // regen --write mutates the same projections as reconcile; it must take the + // shared reconcile lock so a concurrent reconcile cannot race a stale write. + const home = await fleetHome(); + await writeFile(join(home, 'fleet', 'roster.yaml.reconcile.lock'), 'held\n', { mode: 0o600 }); + const calls: string[][] = []; + capture(); + const errors: string[] = []; + vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => { + errors.push(String(chunk)); + return true; + }); + + await program(home, recordingRunner(calls)).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--write', + '--json', + ]); + + expect(process.exitCode).toBe(1); + expect(errors.join('')).toMatch(/regen failed/i); + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false); + expect(calls).toEqual([]); + }); + + it('fails closed (non-zero, no writes) on an invalid roster', async (): Promise => { + // roster-v2 requires >= 1 agent; a malformed roster must abort regen without + // writing any projection and without touching lifecycle. + const home = await fleetHome('version: 2\ngeneration: 1\n'); + const calls: string[][] = []; + capture(); + const errors: string[] = []; + vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => { + errors.push(String(chunk)); + return true; + }); + + await program(home, recordingRunner(calls)).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--write', + '--json', + ]); + + expect(process.exitCode).toBe(1); + expect(errors.join('')).toMatch(/regen failed/i); + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false); + expect(calls).toEqual([]); + }); + + it('rejects a non-canonical --roster path (canonical roster only)', async (): Promise => { + // `fleet` exposes a global `--roster`; reconcile rejects a non-canonical value + // rather than silently reading the canonical roster. regen must enforce the + // SAME guard so `--roster /elsewhere regen --write` cannot mislead the operator. + const home = await fleetHome(); + const calls: string[][] = []; + capture(); + const errors: string[] = []; + vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => { + errors.push(String(chunk)); + return true; + }); + + await program(home, recordingRunner(calls)).parseAsync([ + 'node', + 'mosaic', + 'fleet', + '--roster', + join(home, 'other', 'roster.yaml'), + 'regen', + '--write', + '--json', + ]); + + expect(process.exitCode).toBe(1); + expect(errors.join('')).toMatch(/regen failed/i); + expect(errors.join('')).toMatch(/canonical roster/i); + // No projection written against a misdirected roster path. + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false); + expect(calls).toEqual([]); + }); + + it('reports an incomplete rebuild when a later projection write fails mid-loop', async (): Promise => { + // Fault injected on the SECOND apply: coder0 is already replaced on disk, so a + // bare throw would hide the partial state. regen must surface which agents were + // rebuilt and which failed, with a verify-before-restart recovery instruction. + const home = await fleetHome(); + const realApply = applyPreparedGeneratedAgentEnvironmentProjection; + const result = await executeFleetRegen( + { + runner: recordingRunner([]), + mosaicHome: home, + applyProjection: async (prepared): Promise => { + if (prepared.generatedPath.endsWith('coder1.env.generated')) { + throw new Error('simulated ENOSPC on coder1'); + } + return realApply(prepared); + }, + }, + { write: true }, + ); + + // coder0 written before the fault; coder1 not. + expect(result.written).toBe(1); + expect(result.incomplete).toMatchObject({ + code: 'projection-apply-failed', + failedAgent: 'coder1', + writtenAgents: ['coder0'], + }); + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true); + expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false); + + // The human report names the failed agent and directs a verify-before-restart + // recovery — not a bare success runbook. + const report = formatFleetRegenReport(result).join('\n'); + expect(report).toMatch(/incomplete/i); + expect(report).toContain('coder1'); + expect(report).toMatch(/verify|re-run|retry/i); + // Structural: still no lifecycle call anywhere on the failure path. + expect(report).not.toMatch(/systemctl --user restart mosaic-agent@\n.*\n/); + }); + + it('refuses to write while agent CRUD holds the roster mutation lock', async (): Promise => { + // regen --write overwrites the SAME generated projections that `fleet agent + // create/update/delete` writes under roster.yaml.mutation.lock. If regen only + // took the reconcile lock it could read a stale roster and overwrite a just- + // committed projection. It must also take the mutation lock and fail closed. + const home = await fleetHome(); + await writeFile(join(home, 'fleet', 'roster.yaml.mutation.lock'), 'crud\n', { mode: 0o600 }); + const calls: string[][] = []; + capture(); + const errors: string[] = []; + vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => { + errors.push(String(chunk)); + return true; + }); + + await program(home, recordingRunner(calls)).parseAsync([ + 'node', + 'mosaic', + 'fleet', + 'regen', + '--write', + '--json', + ]); + + expect(process.exitCode).toBe(1); + expect(errors.join('')).toMatch(/regen failed/i); + // No projection written against a roster a concurrent CRUD is mutating. + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false); + expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false); + expect(calls).toEqual([]); + }); + + it('surfaces BOTH the roster failure and a stale-lock warning on a double fault', async (): Promise => { + // Worst case with nothing written: run() fails (invalid roster, fail-closed) AND + // a lock release faults. The operator must be told the lock may be stale, not + // just the roster error — otherwise the next regen/reconcile is silently blocked. + const home = await fleetHome('version: 2\ngeneration: 1\n'); + const throwingRelease = + () => async (): Promise<() => Promise> => async (): Promise => { + throw new Error('simulated lock unlink fault'); + }; + await expect( + executeFleetRegen( + { + runner: recordingRunner([]), + mosaicHome: home, + acquireReconcileLock: throwingRelease, + acquireRosterMutationLock: throwingRelease, + }, + { write: true }, + ), + ).rejects.toThrow(/stale|inspect|lock/i); + // Nothing written on the fail-closed path. + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false); + }); + + it('propagates a roster-mutation-lock release failure instead of silently keeping a stale lock', async (): Promise => { + // Finding L: the real `acquirePrivateRosterMutationLock` release must SURFACE an + // unlink failure, not swallow it. If it swallowed (like the CRUD-internal + // acquireMutationLock does), a stale roster.yaml.mutation.lock left after a + // successful regen would block later regen/agent CRUD while the command reported + // success — and the finding-J stale-lock warning would never fire for this lock. + const home = await fleetHome(); + const release = await acquirePrivateRosterMutationLock(home)(); + // Simulate the lock vanishing (or being unremovable) before release runs: the + // underlying unlink then faults. A swallowing release would resolve and hide it. + await rm(join(home, 'fleet', 'roster.yaml.mutation.lock')); + await expect(release()).rejects.toThrow(); + }); + + it('preserves a complete rebuild result and flags lock-cleanup when release faults', async (): Promise => { + // A lock-release fault after a successful write must NOT discard the fact that + // projections are now live — the operator needs to know the write happened and + // that the shared lock may be stale, not just a bare "regen failed". + const home = await fleetHome(); + const result = await executeFleetRegen( + { + runner: recordingRunner([]), + mosaicHome: home, + acquireReconcileLock: + () => async (): Promise<() => Promise> => async (): Promise => { + throw new Error('simulated lock unlink fault'); + }, + }, + { write: true }, + ); + + expect(result.written).toBe(2); + expect(result.incomplete).toBeUndefined(); + expect(result.cleanup).toMatchObject({ + code: 'lock-cleanup-failed', + action: 'inspect-lock-before-retry', + }); + // Projections are genuinely on disk despite the release fault. + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true); + expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(true); + + const report = formatFleetRegenReport(result).join('\n'); + expect(report).toMatch(/lock/i); + expect(report).toMatch(/stale|inspect|clear/i); + }); + + it('preserves a partial rebuild AND flags lock-cleanup when both apply and release fault', async (): Promise => { + // Worst case: a mid-loop apply fault leaves a partial rebuild, then the release + // also faults. Both recovery states must survive so the operator sees exactly + // which projections are live and that the lock may be stale. + const home = await fleetHome(); + const realApply = applyPreparedGeneratedAgentEnvironmentProjection; + const result = await executeFleetRegen( + { + runner: recordingRunner([]), + mosaicHome: home, + applyProjection: async (prepared): Promise => { + if (prepared.generatedPath.endsWith('coder1.env.generated')) { + throw new Error('simulated ENOSPC on coder1'); + } + return realApply(prepared); + }, + acquireReconcileLock: + () => async (): Promise<() => Promise> => async (): Promise => { + throw new Error('simulated lock unlink fault'); + }, + }, + { write: true }, + ); + + expect(result.written).toBe(1); + expect(result.incomplete).toMatchObject({ + code: 'projection-apply-failed', + failedAgent: 'coder1', + writtenAgents: ['coder0'], + }); + expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' }); + + const report = formatFleetRegenReport(result).join('\n'); + expect(report).toMatch(/incomplete/i); + expect(report).toMatch(/lock/i); + }); + + it('fails closed without deleting a REPLACEMENT roster-mutation lock it no longer owns', async (): Promise => { + // Finding M1 (replacement-lock race): after this invocation created its lock, the + // lock is cleared and re-created by ANOTHER writer (new inode + foreign token) + // BEFORE release runs. The release must PROVE ownership (device/inode + token) and + // refuse to unlink the stranger's live lock — otherwise it deletes it, a third + // writer enters concurrently, and mutual exclusion over the projections is broken. + // This pins the REACHABLE window (replaced-before-release); the residual sub-window + // between the final check and the path `unlink` is unreachable within the `wx` + // writer protocol and matches the merged reconcile lock — see the acquirer doc. + const home = await fleetHome(); + const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock'); + const release = await acquirePrivateRosterMutationLock(home)(); + // A different writer replaces the lock: same path, NEW inode, foreign content. + await rm(lockPath); + await writeFile(lockPath, 'a-different-writer\n', { mode: 0o600 }); + const replacement = await stat(lockPath); + // Release must reject (it no longer owns this file) AND the diagnosis must name + // the MUTATION lock specifically — a fault on this shared helper must not be + // mislabeled as the reconcile lock (that would defeat the M3 accurate-diagnosis + // goal, since regen serializes on both) ... + await expect(release()).rejects.toThrow(/roster\.yaml\.mutation\.lock/); + // ... and MUST NOT have unlinked the stranger's live lock. + expect(await exists(lockPath)).toBe(true); + expect((await stat(lockPath)).ino).toBe(replacement.ino); + }); + + it('does not strand the lock file when initialization fails after creating it', async (): Promise => { + // Codex r4: if writeFile/stat/close/ownership-check throws AFTER the `wx` create + // succeeds, the just-created lock must NOT be left behind — a stranded lock would + // permanently block future regen AND agent CRUD (both contend on this path). The + // cleanup is dev/ino-guarded (never deletes a replacement) and best-effort (never + // masks the initialization error). + const home = await fleetHome(); + const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock'); + // Inject an opener that performs the REAL `wx` create (so the lock file really + // lands on disk) but whose handle faults on the token write — a post-create + // initialization failure. + const faultingOpen = (async ( + path: Parameters[0], + flags: Parameters[1], + mode: Parameters[2], + ) => { + const handle = await open(path, flags, mode); + return new Proxy(handle, { + get(target, prop, receiver): unknown { + if (prop === 'writeFile') { + return async (): Promise => { + throw new Error('simulated ENOSPC on token write'); + }; + } + const value = Reflect.get(target, prop, receiver); + return typeof value === 'function' ? value.bind(target) : value; + }, + }); + }) as typeof open; + + await expect(acquirePrivateRosterMutationLock(home, faultingOpen)()).rejects.toThrow( + /mutation\.lock/, + ); + // The lock we created must have been cleaned up — not stranded on disk. + expect(await exists(lockPath)).toBe(false); + }); + + it('does not strand the lock file when the post-create stat itself fails', async (): Promise => { + // Codex r5: the narrower sub-path where `handle.stat()` ITSELF rejects right after + // the `wx` create succeeds (transient EIO/EBADF). device/inode identity is then + // unavailable, so the init-failure cleanup cannot prove ownership by dev/ino — yet + // the lock must STILL not be stranded, because a stranded lock permanently blocks + // future regen AND agent CRUD. The token we persisted before the failure uniquely + // identifies OUR lock, so cleanup falls back to matching it. + const home = await fleetHome(); + const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock'); + // Real `wx` create (lock really lands on disk); the token write SUCCEEDS so our + // random token is persisted, then `stat()` faults — a post-create init failure. + const faultingStatOpen = (async ( + path: Parameters[0], + flags: Parameters[1], + mode: Parameters[2], + ) => { + const handle = await open(path, flags, mode); + return new Proxy(handle, { + get(target, prop, receiver): unknown { + if (prop === 'stat') { + return async (): Promise => { + throw new Error('simulated EIO on post-create stat'); + }; + } + const value = Reflect.get(target, prop, receiver); + return typeof value === 'function' ? value.bind(target) : value; + }, + }); + }) as typeof open; + + await expect(acquirePrivateRosterMutationLock(home, faultingStatOpen)()).rejects.toThrow( + /mutation\.lock/, + ); + // Even without dev/ino, the token-based fallback must have removed our lock. + expect(await exists(lockPath)).toBe(false); + }); + + it('surfaces a lock-cleanup fault when unwinding after a later lock acquire fails', async (): Promise => { + // Finding M2: the FIRST lock is taken, the SECOND lock's acquire throws, and + // unwinding the first lock's release ALSO faults. The acquire-unwind path must + // surface both — the already-taken lock may now be stale and silently block the + // next regen/reconcile — not just re-throw the acquire error and drop the fault. + const home = await fleetHome(); + await expect( + executeFleetRegen( + { + runner: recordingRunner([]), + mosaicHome: home, + // Mutation lock (acquired first) succeeds, but its RELEASE faults on unwind. + acquireRosterMutationLock: + () => async (): Promise<() => Promise> => async (): Promise => { + throw new Error('simulated mutation-lock unlink fault'); + }, + // Reconcile lock (acquired second) ACQUIRE fails, triggering the unwind. + acquireReconcileLock: () => async (): Promise<() => Promise> => { + throw new Error('simulated reconcile acquire failure'); + }, + }, + { write: true }, + ), + ).rejects.toThrow(/stale|inspect|lock/i); + expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false); + }); + + it('names BOTH fleet locks in the stale-lock warning (cleanup can come from either)', async (): Promise => { + // Finding M3: finding L made the roster MUTATION lock's release fault reachable, so + // the cleanup marker can now originate from either lock. The human report must not + // claim only the reconcile lock is stale — it must name both candidate lock files + // so the operator inspects the right one. + const home = await fleetHome(); + const result = await executeFleetRegen( + { + runner: recordingRunner([]), + mosaicHome: home, + // Fault the MUTATION lock's release specifically (reconcile lock is real). + acquireRosterMutationLock: + () => async (): Promise<() => Promise> => async (): Promise => { + throw new Error('simulated mutation-lock unlink fault'); + }, + }, + { write: true }, + ); + + expect(result.written).toBe(2); + expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' }); + const report = formatFleetRegenReport(result).join('\n'); + expect(report).toContain('roster.yaml.mutation.lock'); + expect(report).toContain('roster.yaml.reconcile.lock'); + }); +}); diff --git a/packages/mosaic/src/commands/fleet-regen-command.ts b/packages/mosaic/src/commands/fleet-regen-command.ts new file mode 100644 index 0000000..9f3ce67 --- /dev/null +++ b/packages/mosaic/src/commands/fleet-regen-command.ts @@ -0,0 +1,441 @@ +import { readFile, stat } from 'node:fs/promises'; +import { homedir } from 'node:os'; +import { join, relative, resolve } from 'node:path'; +import type { Command } from 'commander'; +import type { CommandRunner } from './fleet.js'; +import { + applyPreparedGeneratedAgentEnvironmentProjection, + prepareGeneratedAgentEnvironmentProjection, + type PreparedGeneratedAgentEnvironmentProjection, +} from '../fleet/generated-env-boundary.js'; +import { + acquirePrivateReconcileLock, + acquirePrivateRosterMutationLock, + projectRosterV2AgentGeneratedEnv, +} from '../fleet/fleet-reconciler.js'; +import { + parseRosterV2, + validateRosterV2Semantics, + type FleetRosterV2, +} from '../fleet/roster-v2.js'; + +/** + * `mosaic fleet regen` — recovery-framed regeneration of the roster-derived + * agent env projections (`fleet/agents/.env.generated`) from the roster + * SSOT. It is the recovery layer of #791: when a wiped/diverged operator surface + * has cost an agent its generated projection, regen rebuilds it deterministically + * from `roster.yaml` so the launcher can source the intended identity again. + * + * It is intentionally a THIN projection-only wrapper over the same mapping the + * reconciler apply path uses ({@link projectRosterV2AgentGeneratedEnv}), and it + * has NO code path to systemd lifecycle: regen never starts, stops, or restarts + * an agent. Recovery order forbids restart-before-verify, so the operator must + * verify each rebuilt projection resolves before restarting anything. + */ +export interface FleetRegenAgentPlan { + readonly name: string; + /** Generated-projection path, relative to the Mosaic home (never absolute in output). */ + readonly path: string; + /** `create` when the projection is currently absent, `rebuild` when it already exists. */ + readonly disposition: 'create' | 'rebuild'; +} + +/** + * Present ONLY when a `--write` projection apply failed partway through the + * fleet. Every prior agent was already validated and prepared, so `writtenAgents` + * are byte-complete on disk; `failedAgent` is where the write faulted. The + * command surfaces this instead of a bare throw so the operator can finish the + * rebuild and verify every projection before restarting any unit. + */ +export interface FleetRegenIncomplete { + readonly code: 'projection-apply-failed'; + /** The agent whose generated-projection write faulted. */ + readonly failedAgent: string; + /** Agents whose projections were fully written before the fault (in apply order). */ + readonly writtenAgents: readonly string[]; +} + +export interface FleetRegenResult { + readonly mode: 'dry-run' | 'write'; + /** Roster path, relative to the Mosaic home. */ + readonly rosterPath: string; + readonly generation: number; + readonly agentCount: number; + /** Count of projections written to disk (always 0 in dry-run). */ + readonly written: number; + readonly agents: readonly FleetRegenAgentPlan[]; + /** Set ONLY when a `--write` apply faulted mid-fleet — a partial rebuild. */ + readonly incomplete?: FleetRegenIncomplete; + /** + * Set ONLY when a shared fleet lock could not be released after a write completed + * (or partially completed) — EITHER `roster.yaml.mutation.lock` or + * `roster.yaml.reconcile.lock`, since regen holds both. The projections in this + * result are real; the lock file may be stale and must be inspected before the + * next mutation. Independent of {@link incomplete} — both can be present at once. + */ + readonly cleanup?: { + readonly code: 'lock-cleanup-failed'; + readonly action: 'inspect-lock-before-retry'; + }; +} + +export interface FleetRegenDeps { + /** + * Present ONLY so an accidental future lifecycle wiring is caught by the + * "never restarts" test — regen is contractually forbidden to invoke it. + */ + readonly runner: CommandRunner; + readonly mosaicHome?: string; + /** Persona roots for semantic roster validation (defaults mirror the reconciler). */ + readonly rolesDir?: string; + readonly overrideDir?: string; + /** Test seam: canonical roster reader. Defaults to parse + semantic validation. */ + readonly readRoster?: (rosterPath: string) => Promise; + /** + * Test seam: generated-only projection validator. Defaults to the audited + * boundary helper that validates ONLY `.env.generated`. + */ + readonly prepareProjection?: typeof prepareGeneratedAgentEnvironmentProjection; + /** Test seam: generated-only projection writer. Defaults to the audited boundary helper. */ + readonly applyProjection?: typeof applyPreparedGeneratedAgentEnvironmentProjection; + /** + * Test seam: acquire the shared reconcile lock before a `--write`. Defaults to + * the reconciler's private lock so regen and reconcile are mutually exclusive + * and a concurrent reconcile cannot race a stale projection write. + */ + readonly acquireReconcileLock?: (mosaicHome: string) => () => Promise<() => Promise>; + /** + * Test seam: acquire the shared roster mutation lock before a `--write`. + * Defaults to the reconciler's hardened, ownership-proving acquirer for the + * same `fleet/roster.yaml.mutation.lock` path that CRUD contends on, so regen + * serializes against agent create/update/delete — both rewrite the same + * generated projections, so without this a concurrent CRUD could race a stale + * projection write. + */ + readonly acquireRosterMutationLock?: (mosaicHome: string) => () => Promise<() => Promise>; + /** Test seam: existence probe for create/rebuild disposition. */ + readonly fileExists?: (path: string) => Promise; +} + +interface FleetRegenOptions { + readonly write?: boolean; +} + +function defaultMosaicHome(deps: FleetRegenDeps): string { + return deps.mosaicHome ?? process.env['MOSAIC_HOME'] ?? join(homedir(), '.config', 'mosaic'); +} + +async function defaultFileExists(path: string): Promise { + try { + await stat(path); + return true; + } catch { + return false; + } +} + +/** + * Projection-only regeneration core. Deterministic and side-effect-free in + * dry-run; in `--write` it rebuilds each generated projection through the + * audited generated-only boundary writer and NOTHING else — it never writes + * `.env.local`/`.env.quarantine`, never unlinks the legacy `.env`, and never + * touches `deps.runner`. + * + * Fail-closed by construction: the roster is validated (structural + semantic) + * and EVERY agent's projection is prepared before ANY is written, so a single + * invalid agent (or a semantically rejected roster) leaves the whole surface + * untouched. In `--write` mode the read-prepare-apply sequence runs under the + * shared reconcile lock so a concurrent reconcile cannot race a stale write. + */ +export async function executeFleetRegen( + deps: FleetRegenDeps, + options: FleetRegenOptions, +): Promise { + const mosaicHome = defaultMosaicHome(deps); + const agentEnvDir = join(mosaicHome, 'fleet', 'agents'); + const rosterPath = join(mosaicHome, 'fleet', 'roster.yaml'); + const readRoster = deps.readRoster ?? defaultReadRoster(deps, mosaicHome); + const prepare = deps.prepareProjection ?? prepareGeneratedAgentEnvironmentProjection; + const apply = deps.applyProjection ?? applyPreparedGeneratedAgentEnvironmentProjection; + const fileExists = deps.fileExists ?? defaultFileExists; + const acquireReconcileLock = deps.acquireReconcileLock ?? acquirePrivateReconcileLock; + const acquireRosterMutationLock = + deps.acquireRosterMutationLock ?? acquirePrivateRosterMutationLock; + const write = options.write === true; + + const run = async (): Promise => { + const roster = await readRoster(rosterPath); + + // Prepare (validate) every agent BEFORE any write, so a later failure never + // leaves an earlier projection partially rebuilt. + const prepared: (PreparedGeneratedAgentEnvironmentProjection & { + readonly name: string; + readonly disposition: 'create' | 'rebuild'; + })[] = []; + for (const agent of roster.agents) { + const projection = await prepare({ + mosaicHome, + agentEnvDir, + agentName: agent.name, + generated: projectRosterV2AgentGeneratedEnv(roster, agent), + }); + const disposition = (await fileExists(projection.generatedPath)) ? 'rebuild' : 'create'; + prepared.push({ ...projection, name: agent.name, disposition }); + } + + // Every projection is already validated (prepared); an apply here can only + // fault on the underlying I/O (ENOSPC, EIO, a race outside the lock). If a + // later write faults, earlier agents are already replaced on disk, so we + // stop and report the partial state rather than throwing a bare error that + // hides which projections are now live. + let written = 0; + let incomplete: FleetRegenIncomplete | undefined; + if (write) { + for (const projection of prepared) { + try { + await apply(projection); + } catch { + incomplete = { + code: 'projection-apply-failed', + failedAgent: projection.name, + writtenAgents: prepared.slice(0, written).map((entry) => entry.name), + }; + break; + } + written += 1; + } + } + + return { + mode: write ? 'write' : 'dry-run', + rosterPath: relative(mosaicHome, rosterPath), + generation: roster.generation, + agentCount: roster.agents.length, + written, + agents: prepared.map((projection) => ({ + name: projection.name, + path: relative(mosaicHome, projection.generatedPath), + disposition: projection.disposition, + })), + ...(incomplete ? { incomplete } : {}), + }; + }; + + // Dry-run is preview-only and never mutates, so it stays lock-free (usable even + // while a reconcile or CRUD holds a lock). A `--write` must serialize against + // BOTH roster writers of the same generated projections: agent CRUD (roster + // mutation lock) and reconcile (reconcile lock). Acquire in a fixed order — the + // locks are non-blocking (they throw on contention rather than wait), so no + // deadlock is possible — and fail closed if either is already held. + if (!write) return run(); + + const releases: (() => Promise)[] = []; + try { + releases.push(await acquireRosterMutationLock(mosaicHome)()); + releases.push(await acquireReconcileLock(mosaicHome)()); + } catch (error: unknown) { + // A later acquire failed; unwind any lock already taken (reverse order). If + // that unwind ALSO faults, surface both — the already-taken lock may now be + // stale and silently block the next regen/reconcile — rather than dropping it. + const releaseFault = await releaseFleetLocks(releases); + throw augmentWithLockCleanupFault(error, releaseFault); + } + + let result: FleetRegenResult | undefined; + let primaryError: unknown; + try { + result = await run(); + } catch (error: unknown) { + primaryError = error; + } + + // Release both locks (reverse acquire order), continuing past a fault so neither + // leaks. A release fault must never discard a completed (or partial) rebuild — + // the projections are already on disk — and must not be silently hidden even + // when the run itself failed with nothing written. + const releaseFault = await releaseFleetLocks(releases); + if (result) { + return releaseFault === undefined + ? result + : { + ...result, + cleanup: { code: 'lock-cleanup-failed', action: 'inspect-lock-before-retry' }, + }; + } + // result is undefined ⇒ run() threw ⇒ primaryError is set. + throw augmentWithLockCleanupFault(primaryError, releaseFault); +} + +/** + * Releases held locks in reverse acquire order, continuing past a fault so a + * single failed release never leaks the others. Returns the first fault seen (or + * undefined when every release succeeded). + */ +async function releaseFleetLocks(releases: readonly (() => Promise)[]): Promise { + let firstFault: unknown; + for (const release of [...releases].reverse()) { + try { + await release(); + } catch (error: unknown) { + if (firstFault === undefined) firstFault = error; + } + } + return firstFault; +} + +/** + * When a run failed with nothing written AND lock cleanup also faulted, surface + * both: the operator needs to know the lock may be stale (else the next + * regen/reconcile is silently blocked), not just the original run error. + */ +function augmentWithLockCleanupFault(primaryError: unknown, releaseFault: unknown): unknown { + if (releaseFault === undefined) return primaryError; + const base = primaryError instanceof Error ? primaryError.message : String(primaryError); + return new Error( + `${base} (additionally, a fleet lock could not be released — ` + + 'fleet/roster.yaml.mutation.lock or roster.yaml.reconcile.lock may be stale; ' + + 'inspect and clear it before retry)', + ); +} + +function defaultReadRoster( + deps: FleetRegenDeps, + mosaicHome: string, +): (rosterPath: string) => Promise { + return async (rosterPath: string): Promise => { + const roster = parseRosterV2(await readFile(rosterPath, 'utf8'), 'yaml'); + // Enforce the SAME semantic gate as reconcile/plan/verify (persona resolution + // + protected-class tool-policy match) so regen cannot project a roster the + // rest of the fleet surface would reject. + await validateRosterV2Semantics(roster, { + rolesDir: deps.rolesDir ?? join(mosaicHome, 'fleet', 'roles'), + overrideDir: deps.overrideDir ?? join(mosaicHome, 'fleet', 'roles.local'), + }); + return roster; + }; +} + +/** + * Human-readable report — paths and counts ONLY. The rendered projection body + * (KEY=value lines) is never echoed (secrev). In `--write` mode it prints the + * recovery runbook: verify each projection resolves BEFORE any restart. + */ +export function formatFleetRegenReport(result: FleetRegenResult): string[] { + const lines: string[] = []; + const header = + result.mode === 'dry-run' + ? 'mosaic fleet regen — dry run (no changes written)' + : 'mosaic fleet regen — wrote roster-derived projections'; + lines.push(header); + lines.push( + `roster: ${result.rosterPath} (generation ${result.generation}, ${result.agentCount} agent(s))`, + ); + for (const agent of result.agents) { + lines.push(` ${agent.name} ${agent.path} [${agent.disposition}]`); + } + + if (result.mode === 'dry-run') { + lines.push( + `Plan: would write ${result.agentCount} generated projection(s). ` + + `Re-run with --write to apply. regen never restarts agents.`, + ); + return lines; + } + + if (result.incomplete) { + const { failedAgent, writtenAgents } = result.incomplete; + lines.push( + `INCOMPLETE: rebuild stopped at agent ${failedAgent} — its projection write faulted.`, + ); + lines.push( + `Rebuilt before the fault: ${writtenAgents.length} projection(s)` + + (writtenAgents.length > 0 ? ` (${writtenAgents.join(', ')})` : '') + + `; ${result.agentCount - writtenAgents.length} not written.`, + ); + lines.push( + 'Recover: re-run `mosaic fleet regen --write` to finish, then VERIFY every ' + + "agent's fleet/agents/.env.generated resolves BEFORE restarting any " + + 'unit. regen never restarts agents.', + ); + } else { + lines.push(`Wrote ${result.written} generated projection(s).`); + lines.push('Next steps — DO NOT restart agents before verifying:'); + lines.push( + " 1. Confirm each agent's fleet/agents/.env.generated exists and carries " + + 'the intended MOSAIC_AGENT_* values.', + ); + // The unit sets NO EnvironmentFile= — it launches from a minimal env and the + // launcher (start-agent-session.sh) sources .env.generated itself. Verify the + // launcher path, not a nonexistent EnvironmentFile property. + lines.push( + ' 2. Confirm the unit launches from it: `systemctl --user cat mosaic-agent@` ' + + 'shows ExecStart running start-agent-session.sh, which reads .env.generated.', + ); + lines.push( + ' 3. Only then restart one unit at a time: systemctl --user restart mosaic-agent@.', + ); + } + + if (result.cleanup) { + lines.push( + 'WARNING: a fleet lock could not be released — ' + + 'fleet/roster.yaml.mutation.lock or fleet/roster.yaml.reconcile.lock may be ' + + 'stale; inspect and clear it before the next reconcile or regen.', + ); + } + return lines; +} + +function resolveRegenMosaicHome(fleetCommand: Command, deps: FleetRegenDeps): string { + const options = fleetCommand.optsWithGlobals<{ mosaicHome?: string }>(); + return options.mosaicHome ?? defaultMosaicHome(deps); +} + +/** + * regen reads the roster SSOT at `/fleet/roster.yaml` and ignores the + * `fleet --roster ` global. If an operator passes a non-canonical `--roster` + * they must NOT be silently served the canonical file — mirror reconcile's guard + * and reject, so `mosaic fleet --roster /elsewhere regen --write` fails closed. + */ +function assertRegenCanonicalRoster(fleetCommand: Command, mosaicHome: string): void { + const options = fleetCommand.optsWithGlobals<{ roster?: string }>(); + const canonical = join(mosaicHome, 'fleet', 'roster.yaml'); + if (options.roster !== undefined && resolve(options.roster) !== resolve(canonical)) { + throw new Error( + 'Roster-v2 regeneration requires the canonical roster path (fleet/roster.yaml).', + ); + } +} + +/** Registers the recovery-framed `mosaic fleet regen` command. */ +export function registerFleetRegenCommand(fleetCommand: Command, deps: FleetRegenDeps): void { + fleetCommand + .command('regen') + .description( + 'Regenerate roster-derived agent env projections (dry-run default, --write to apply; never restarts)', + ) + .option('--write', 'Write the regenerated projections to disk (default: dry-run)') + .option('--json', 'Emit the plan/result as JSON') + .action(async (opts: { write?: boolean; json?: boolean }): Promise => { + const mosaicHome = resolveRegenMosaicHome(fleetCommand, deps); + try { + assertRegenCanonicalRoster(fleetCommand, mosaicHome); + const result = await executeFleetRegen( + { ...deps, mosaicHome }, + { write: opts.write === true }, + ); + if (opts.json === true) { + console.log(JSON.stringify(result)); + } else { + for (const line of formatFleetRegenReport(result)) console.log(line); + } + // A partial rebuild or a stale-lock cleanup state is a non-zero outcome: + // the operator must finish/verify (and clear the lock) before restarting. + if (result.incomplete || result.cleanup) process.exitCode = 1; + } catch (error: unknown) { + process.exitCode = 1; + const message = error instanceof Error ? error.message : String(error); + process.stderr.write(`mosaic fleet regen failed: ${message}\n`); + } + }); +} diff --git a/packages/mosaic/src/commands/fleet.spec.ts b/packages/mosaic/src/commands/fleet.spec.ts index d6f165a..e54cc0d 100644 --- a/packages/mosaic/src/commands/fleet.spec.ts +++ b/packages/mosaic/src/commands/fleet.spec.ts @@ -97,6 +97,7 @@ describe('registerFleetCommand', () => { 'provision', 'ps', 'reconcile', + 'regen', 'remove', 'restart', 'start', diff --git a/packages/mosaic/src/commands/fleet.ts b/packages/mosaic/src/commands/fleet.ts index da51896..573106c 100644 --- a/packages/mosaic/src/commands/fleet.ts +++ b/packages/mosaic/src/commands/fleet.ts @@ -44,6 +44,7 @@ import { registerFleetReconcilerCommands, type FleetReconcilerCommandDeps, } from './fleet-reconciler-command.js'; +import { registerFleetRegenCommand } from './fleet-regen-command.js'; import { resolveCommsBlock } from '../fleet/comms-onboarding.js'; import { applyPreparedAgentEnvironmentProjection, @@ -2063,6 +2064,18 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps = reconcileDeps: deps.reconcileDeps, }); + // Recovery (#791 PR3): rebuild roster-derived env projections from the roster + // SSOT. Projection-only and preview-first — never issues a lifecycle restart. + registerFleetRegenCommand(cmd, { + runner, + mosaicHome: deps.mosaicHome, + // Resolve personas the SAME way reconcile does: forward any configured roots + // so a custom-persona-root deployment cannot have reconcile accept a roster + // that regen then rejects against the default `/fleet/roles`. + rolesDir: deps.reconcileDeps?.rolesDir, + overrideDir: deps.reconcileDeps?.overrideDir, + }); + return cmd; } diff --git a/packages/mosaic/src/fleet/fleet-agent-crud.ts b/packages/mosaic/src/fleet/fleet-agent-crud.ts index 3acfadf..9795a40 100644 --- a/packages/mosaic/src/fleet/fleet-agent-crud.ts +++ b/packages/mosaic/src/fleet/fleet-agent-crud.ts @@ -381,6 +381,22 @@ function isMissingFile(error: unknown): boolean { return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT'; } +/** + * Non-blocking acquire of the roster mutation lock: an exclusive `wx` create that + * throws `concurrent-mutation` (never waits) if the lock is already held. The + * on-disk format is an empty private file. CRUD releases in a plain `finally` with + * no result-preservation logic, so a release fault here must not override the + * mutation result — the release swallows unlink failures by design. + * + * The recovery-framed `fleet regen` contends on this exact same lock file, but it + * does so through the hardened, ownership-proving acquirer in the reconciler + * (`acquirePrivateRosterMutationLock`), not this one: whoever wins the `wx` create + * owns the file (the loser always gets `concurrent-mutation`), so the reconciler's + * ownership token is only ever written and read back by the same regen invocation, + * never by this empty-file writer. The two acquirers stay mutually compatible at + * the `wx`-contention layer while regen additionally proves ownership before it + * unlinks — a guarantee CRUD does not need because it releases unconditionally. + */ async function acquireMutationLock(lockPath: string): Promise<() => Promise> { try { const handle = await open(lockPath, 'wx', 0o600); diff --git a/packages/mosaic/src/fleet/fleet-reconciler.ts b/packages/mosaic/src/fleet/fleet-reconciler.ts index a0cb4a5..732c878 100644 --- a/packages/mosaic/src/fleet/fleet-reconciler.ts +++ b/packages/mosaic/src/fleet/fleet-reconciler.ts @@ -584,6 +584,29 @@ function defaultValidateRoster( }; } +/** + * The single roster-v2 → generated-projection mapping. Both the reconciler's + * apply path ({@link defaultPrepareProjections}) and the recovery-framed + * `mosaic fleet regen` command derive their generated env from THIS one + * function, so the two paths can never drift (the #791 single-SSOT invariant). + * Pure: no IO, no clock — a deterministic function of the roster alone. + */ +export function projectRosterV2AgentGeneratedEnv( + roster: FleetRosterV2, + agent: FleetRosterV2Agent, +): Readonly> { + return { + MOSAIC_AGENT_NAME: agent.name, + MOSAIC_AGENT_CLASS: agent.className, + MOSAIC_AGENT_RUNTIME: agent.runtime, + MOSAIC_AGENT_MODEL: agent.model, + MOSAIC_AGENT_REASONING: agent.reasoning, + MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy, + MOSAIC_AGENT_WORKDIR: agent.workingDirectory, + MOSAIC_TMUX_SOCKET: roster.tmux.socketName, + }; +} + function defaultPrepareProjections( request: FleetReconcileRequest, ): (roster: FleetRosterV2) => Promise { @@ -596,16 +619,7 @@ function defaultPrepareProjections( mosaicHome, agentEnvDir: join(mosaicHome, 'fleet', 'agents'), agentName: agent.name, - generated: { - MOSAIC_AGENT_NAME: agent.name, - MOSAIC_AGENT_CLASS: agent.className, - MOSAIC_AGENT_RUNTIME: agent.runtime, - MOSAIC_AGENT_MODEL: agent.model, - MOSAIC_AGENT_REASONING: agent.reasoning, - MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy, - MOSAIC_AGENT_WORKDIR: agent.workingDirectory, - MOSAIC_TMUX_SOCKET: roster.tmux.socketName, - }, + generated: projectRosterV2AgentGeneratedEnv(roster, agent), }), ), ); @@ -620,62 +634,193 @@ function mosaicHomeFor(deps: FleetReconcileDeps): string { return deps.mosaicHome ?? join(homedir(), '.config', 'mosaic'); } -/** Acquires a private lock only after proving the canonical managed path. */ +/** Acquires the private reconcile lock only after proving the canonical managed path. */ export function acquirePrivateReconcileLock( mosaicHome: string, openLock: typeof open = open, +): () => Promise<() => Promise> { + return acquirePrivateManagedRosterLock( + mosaicHome, + 'roster.yaml.reconcile.lock', + 'Another roster reconciliation is in progress.', + openLock, + ); +} + +/** + * Acquires the private roster MUTATION lock (`roster.yaml.mutation.lock`) with the + * exact same ownership-proof discipline as the reconcile lock. Exposed so the + * recovery-framed `mosaic fleet regen` can serialize its projection rewrites + * against agent CRUD — which holds this very lock while it rewrites the roster and + * the same derived projections — reusing this hardened acquirer rather than + * reimplementing it. + * + * The release PROVES ownership (device/inode + token) before unlinking, so a lock + * that was already cleared and replaced by the time release runs is never deleted + * out from under the new owner (it fails closed as `lock-cleanup-failed` instead), + * and that cleanup fault is propagated to the caller rather than silently dropped. + * CRUD contends on the identical path with a plain empty-file `wx` create; the two + * never co-own it (whoever wins `wx` owns it; the loser gets `concurrent-mutation`), + * so the token this writes is only ever read back by the same regen invocation that + * wrote it. + * + * SCOPE of the guarantee (matches the merged reconcile lock exactly — see + * {@link acquirePrivateManagedRosterLock}): the ownership proof closes the + * *reachable* window — a stale-lock reaper or operator cleared our lock and another + * writer took it BEFORE our release began — not the residual sub-instruction window + * between the final check and the path-based `unlink`. Closing that residual fully + * requires an fd-held advisory lock adopted by every fleet writer (CRUD, reconcile, + * regen), which is a cross-cutting mechanism change out of scope for this + * projection-only recovery command; it is unreachable within the `wx` writer + * protocol regardless (no Mosaic writer removes a lock it does not own, so only + * external interference can vacate our inode mid-release). + */ +export function acquirePrivateRosterMutationLock( + mosaicHome: string, + openLock: typeof open = open, +): () => Promise<() => Promise> { + return acquirePrivateManagedRosterLock( + mosaicHome, + 'roster.yaml.mutation.lock', + 'Another roster mutation is in progress.', + openLock, + ); +} + +/** + * Shared body for the ownership-proving managed roster locks. Acquires `lockLeaf` + * under `/fleet` with a private `wx` create, writes an ownership token, + * and returns a release that re-proves device/inode + token before unlinking so it + * can never remove a lock it no longer owns. Non-blocking: throws `concurrent-mutation` + * (with `busyMessage`) on contention rather than waiting. + */ +function acquirePrivateManagedRosterLock( + mosaicHome: string, + lockLeaf: string, + busyMessage: string, + openLock: typeof open, ): () => Promise<() => Promise> { const fleetDir = join(mosaicHome, 'fleet'); - const lockPath = join(fleetDir, 'roster.yaml.reconcile.lock'); + const lockPath = join(fleetDir, lockLeaf); + // The message-producing helpers below serve BOTH managed locks, so every fault + // names the actual lock file (`fleet/`) — an operator needs to know WHICH + // lock is stale, not a hardcoded "reconciliation lock". + const lockLabel = `fleet/${lockLeaf}`; return async (): Promise<() => Promise> => { await assertPrivateManagedDirectory(mosaicHome); await assertPrivateManagedDirectory(fleetDir); - await assertSafeLockLeafIfPresent(lockPath); + await assertSafeLockLeafIfPresent(lockPath, lockLabel); let handle: FileHandle; try { handle = await openLock(lockPath, 'wx', 0o600); } catch (error: unknown) { if (isCode(error, 'EEXIST')) { - await assertSafeLockLeafIfPresent(lockPath); - throw new FleetReconcileError( - 'concurrent-mutation', - 'Another roster reconciliation is in progress.', - ); + await assertSafeLockLeafIfPresent(lockPath, lockLabel); + throw new FleetReconcileError('concurrent-mutation', busyMessage); } - throw new FleetReconcileError('lock-io-failed', 'The reconciliation lock cannot be created.'); + throw new FleetReconcileError('lock-io-failed', `The ${lockLabel} lock cannot be created.`); } + // Identity of the lock WE exclusively created (the `wx` create guaranteed it is + // ours). Captured up front so both the release closure and the init-failure + // cleanup below can prove ownership by device/inode before touching the file. + const created = await handle.stat().catch((): undefined => undefined); + const token = randomUUID(); + let tokenPersisted = false; try { await handle.writeFile(`${token}\n`, 'utf8'); - const opened = await handle.stat(); + tokenPersisted = true; await handle.close(); - await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'unsafe-lock'); + if (!created) + throw new FleetReconcileError( + 'lock-io-failed', + `The ${lockLabel} lock cannot be initialized.`, + ); + await assertLockOwnership( + lockPath, + created.dev, + created.ino, + token, + 'unsafe-lock', + lockLabel, + ); return async (): Promise => { try { - await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed'); - await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed'); + await assertLockOwnership( + lockPath, + created.dev, + created.ino, + token, + 'lock-cleanup-failed', + lockLabel, + ); await unlink(lockPath); } catch (error: unknown) { if (error instanceof FleetReconcileError) throw error; - throw new FleetReconcileError( - 'lock-cleanup-failed', - 'The reconciliation lock cleanup failed.', - ); + throw new FleetReconcileError('lock-cleanup-failed', `The ${lockLabel} cleanup failed.`); } }; } catch (error: unknown) { await handle.close().catch((): void => {}); + // Best-effort: remove the lock THIS invocation created so a transient init + // fault does not strand a lock that would block future regen and agent CRUD. + // dev/ino-guarded so it can never delete a replacement; swallowed so cleanup + // failure never masks the initialization error being surfaced. The token is + // passed only once persisted, so the fallback path (when the post-create stat + // failed and dev/ino is unavailable) can still prove ownership by content. + await removeOwnedLockLeafBestEffort(lockPath, created, tokenPersisted ? token : undefined); if (error instanceof FleetReconcileError) throw error; throw new FleetReconcileError( 'lock-io-failed', - 'The reconciliation lock cannot be initialized.', + `The ${lockLabel} lock cannot be initialized.`, ); } }; } +/** + * Best-effort removal of a managed lock leaf THIS process created, used only on the + * initialization-failure path. Two independent ownership proofs, so a lock that was + * already replaced is never deleted: + * + * - primary: the captured device/inode of the file we exclusively `wx`-created; or + * - fallback: when the post-create stat itself failed (dev/ino unavailable), the + * random `ownershipToken` we persisted — only OUR lock carries it, so a CRUD + * (empty) or a differently-tokened replacement is never removed. + * + * Every fault is swallowed because the caller is already surfacing the initialization + * error and a leftover lock is recoverable via the documented runbook. If BOTH proofs + * are unavailable (stat failed AND the token write never landed) the lock is left in + * place rather than risk deleting another writer's file — a doubly-degenerate case + * requiring two independent fs faults on a just-created fd. + */ +async function removeOwnedLockLeafBestEffort( + lockPath: string, + created: { dev: number; ino: number } | undefined, + ownershipToken: string | undefined, +): Promise { + try { + const current = await lstat(lockPath); + if (!current.isFile() || current.isSymbolicLink()) return; + if (created) { + if (current.dev === created.dev && current.ino === created.ino) { + await unlink(lockPath); + } + return; + } + if (ownershipToken !== undefined) { + const contents = await readFile(lockPath, 'utf8'); + if (contents.trim() === ownershipToken) { + await unlink(lockPath); + } + } + } catch { + // Swallowed: leftover lock is recoverable; do not mask the init error. + } +} + async function assertPrivateManagedDirectory(path: string): Promise { try { const metadata = await lstat(path); @@ -691,16 +836,16 @@ async function assertPrivateManagedDirectory(path: string): Promise { } } -async function assertSafeLockLeafIfPresent(lockPath: string): Promise { +async function assertSafeLockLeafIfPresent(lockPath: string, lockLabel: string): Promise { try { const metadata = await lstat(lockPath); if (!metadata.isFile() || metadata.isSymbolicLink() || (metadata.mode & 0o077) !== 0) { - throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unsafe.'); + throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unsafe.`); } } catch (error: unknown) { if (isCode(error, 'ENOENT')) return; if (error instanceof FleetReconcileError) throw error; - throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unavailable.'); + throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unavailable.`); } } @@ -710,6 +855,7 @@ async function assertLockOwnership( inode: number, token: string, failureCode: 'unsafe-lock' | 'lock-cleanup-failed', + lockLabel: string, ): Promise { try { const metadata = await lstat(lockPath); @@ -720,24 +866,21 @@ async function assertLockOwnership( metadata.dev !== device || metadata.ino !== inode ) { - throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.'); + throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`); } const handle = await open(lockPath, constants.O_RDONLY | constants.O_NOFOLLOW); try { const opened = await handle.stat(); const contents = await handle.readFile({ encoding: 'utf8' }); if (opened.dev !== device || opened.ino !== inode || contents !== `${token}\n`) { - throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.'); + throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`); } } finally { await handle.close(); } } catch (error: unknown) { if (error instanceof FleetReconcileError) throw error; - throw new FleetReconcileError( - failureCode, - 'The reconciliation lock ownership cannot be proven.', - ); + throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership cannot be proven.`); } } diff --git a/packages/mosaic/src/fleet/generated-env-boundary.ts b/packages/mosaic/src/fleet/generated-env-boundary.ts index 3bd70fa..cd84944 100644 --- a/packages/mosaic/src/fleet/generated-env-boundary.ts +++ b/packages/mosaic/src/fleet/generated-env-boundary.ts @@ -204,6 +204,53 @@ export async function prepareAgentEnvironmentProjection( }; } +/** A generated-only projection validated without touching any legacy/local/quarantine file. */ +export interface PreparedGeneratedAgentEnvironmentProjection { + readonly mosaicHome: string; + readonly agentEnvDir: string; + readonly generatedPath: string; + readonly generated: string; +} + +/** + * Validates ONLY the roster-derived generated projection for a recovery rebuild. + * Unlike {@link prepareAgentEnvironmentProjection}, this never reads, classifies, + * relocates, or quarantines the legacy `.env` / `.env.local` operator surface — it + * exists so `mosaic fleet regen` has no code path that can mutate anything except + * `.env.generated`. The existing generated file, if present, must already be + * a private regular file. + */ +export async function prepareGeneratedAgentEnvironmentProjection( + options: AgentEnvironmentProjectionOptions, +): Promise { + if (!AGENT_NAME.test(options.agentName)) { + throw new AgentEnvBoundaryError('unsafe-agent-name', 'MOSAIC_AGENT_NAME', options.agentName); + } + await validatePrivateProjectionDirectory(options.mosaicHome, options.agentEnvDir); + const generatedPath = join(options.agentEnvDir, `${options.agentName}.env.generated`); + const generated = renderGeneratedAgentEnvironment(options.generated); + await assertPrivateRegularFileIfPresent(generatedPath); + return { + mosaicHome: options.mosaicHome, + agentEnvDir: options.agentEnvDir, + generatedPath, + generated, + }; +} + +/** + * Applies a generated-only projection: writes ONLY `.env.generated` atomically + * and touches nothing else. It never writes `.env.local`/`.env.quarantine` and never + * unlinks the legacy `.env` — the projection-only recovery guarantee is structural. + */ +export async function applyPreparedGeneratedAgentEnvironmentProjection( + prepared: PreparedGeneratedAgentEnvironmentProjection, +): Promise { + await ensurePrivateProjectionDirectory(prepared.mosaicHome, prepared.agentEnvDir); + await writePrivateAtomically(prepared.generatedPath, prepared.generated); + return prepared.generatedPath; +} + /** * Validates only the exact generated projection eligible for a roster delete. * Local overrides, legacy input, and quarantine records are operator-retained and