Compare commits

...

2 Commits

Author SHA1 Message Date
Hermes Agent
1eb37f6fd0 feat(fleet): add mosaic fleet regen recovery command
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful
Add `mosaic fleet regen`, a projection-only recovery command that rebuilds
each `fleet/agents/<name>.env.generated` from the `roster.yaml` SSOT after an
upgrade or partial write leaves the generated projections stale or missing.

- Dry-run by default; `--write` applies; `--json` for machine output. Reuses the
  merged reconciler's projection plumbing (projectRosterV2AgentGeneratedEnv +
  the generated-env boundary) rather than reimplementing fleet logic.
- Structurally NEVER issues a lifecycle/restart call — regen recovers config
  only; a recordingRunner gate proves no runner invocation ever occurs.
- Serializes against agent CRUD and reconcile via BOTH fleet locks
  (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired
  mutation-then-reconcile and released in reverse; both are non-blocking `wx`
  locks that throw on contention, so no deadlock is possible.
- Hardens the shared managed-lock helper: ownership-proving tokened lock reused
  for both locks with per-lock fault labels; init-failure cleanup no longer
  strands a just-created lock (dev/ino guard, with a persisted-token fallback
  when the post-create stat itself fails); acquire-unwind surfaces a lock
  cleanup fault instead of dropping it.
- Resolves personas the SAME way reconcile does by forwarding configured
  rolesDir/overrideDir, so a custom-persona-root deployment cannot have
  reconcile accept a roster that regen rejects.
- Report/output is secrev-safe: paths and counts only, never projected values.
- Docs: upgrade-safety-and-recovery runbook + fleet-local-canary note.

Tests are TDD red-first with co-located specs (regen spec: 26 tests covering
dry-run/write dispositions, the never-restarts gate, all lock regressions, and
persona-root wiring).

A residual check-then-unlink TOCTOU in the lock release remains (byte-identical
to the merged reconcile lock; unreachable within the `wx` writer protocol); its
true fix is an fd-held advisory lock adopted by all fleet writers, tracked as a
separate follow-up.

Part of #791

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 21:57:38 -05:00
31607a4af6 feat(mosaic): durable pre-update snapshot + verify net + restore CLI (#791 PR2) (#811)
All checks were successful
ci/woodpecker/push/publish Pipeline was successful
ci/woodpecker/push/ci Pipeline was successful
2026-07-17 00:43:18 +00:00
17 changed files with 3418 additions and 41 deletions

View File

@@ -48,15 +48,19 @@ steps:
# keep mode is a single cp-based path that must not depend on rsync), and that a # keep mode is a single cp-based path that must not depend on rsync), and that a
# corrupt/empty/missing manifest aborts fail-closed leaving operator files # corrupt/empty/missing manifest aborts fail-closed leaving operator files
# untouched (B2/B3). The rollback gate proves a mid-sync failure is rolled back # untouched (B2/B3). The rollback gate proves a mid-sync failure is rolled back
# from the pre-update snapshot (B1). The migration matrix pins the v2→v3 # from the pre-update snapshot (B1). The durable-snapshot gate (#791 PR2) proves
# contract-file semantics. Pure bash, no node_modules — runs early alongside # the retained, operator-scoped pre-update backup is taken before any mutation
# sanitization. # (0700/0600, secret never logged, retention-pruned) and that the post-sync
# verify net restores any operator file a manifest bug lets the sync touch. The
# migration matrix pins the v2→v3 contract-file semantics. Pure bash, no
# node_modules — runs early alongside sanitization.
upgrade-guard: upgrade-guard:
image: *node_image image: *node_image
commands: commands:
- apk add --no-cache bash rsync - apk add --no-cache bash rsync
- bash packages/mosaic/framework/tools/quality/scripts/test-upgrade-manifest-guard.sh - bash packages/mosaic/framework/tools/quality/scripts/test-upgrade-manifest-guard.sh
- bash packages/mosaic/framework/tools/quality/scripts/test-upgrade-rollback.sh - bash packages/mosaic/framework/tools/quality/scripts/test-upgrade-rollback.sh
- bash packages/mosaic/framework/tools/quality/scripts/test-upgrade-durable-snapshot.sh
- bash packages/mosaic/framework/tools/quality/scripts/test-install-migration.sh - bash packages/mosaic/framework/tools/quality/scripts/test-install-migration.sh
typecheck: typecheck:

View File

@@ -98,6 +98,39 @@ Expected results:
that means the unit ran, not that an agent pane is live. Treat tmux that means the unit ran, not that an agent pane is live. Treat tmux
`has-session`, `list-panes`, process tree, and logs as the liveness evidence. `has-session`, `list-panes`, process tree, and logs as the liveness evidence.
## Recovery — rebuild generated env projections
Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
deterministic projection of `roster.yaml` (the SSOT) that the launcher
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
wipes or diverges those projections, rebuild them from the roster with
`mosaic fleet regen` — do NOT restart the affected unit first.
```bash
mosaic fleet regen # dry-run (default): show create/rebuild plan per agent
mosaic fleet regen --json # same plan, machine-readable
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated on disk
```
`regen` is projection-only and **never restarts an agent** — it has no path to
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
only (never the projected `KEY=value` body). After `--write`, verify each unit
resolves the intended values before restarting one unit at a time. The unit sets
no `EnvironmentFile=``start-agent-session.sh` sources `.env.generated` itself —
so verify the generated file directly and the launcher path, not a nonexistent
`EnvironmentFile` property:
```bash
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
systemctl --user cat mosaic-agent@<name> | grep ExecStart
systemctl --user restart mosaic-agent@<name>
```
Full recovery runbook and the three-layer #791 protection model (manifest
ownership → pre-update snapshot/restore → regen): see
[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md).
## Release Preflight ## Release Preflight
Run this checklist before cutting or dogfooding a fleet release: Run this checklist before cutting or dogfooding a fleet release:

View File

@@ -0,0 +1,147 @@
# Upgrade Safety & Recovery
How Mosaic protects operator-owned configuration under `~/.config/mosaic` across
framework upgrades, and how to recover if a projection is ever lost.
A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`,
`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident
this hardening addresses: an upgrade that silently overwrites or deletes a file
the operator owns — credentials, personas, a roster, or a generated agent env —
with no snapshot to fall back to.
Protection is layered. Each layer is independent; a later layer catches what an
earlier one misses.
## Layer 1 — Manifest-owned sync (prevention)
The single source of truth for ownership is
[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt).
Both the bash installer and the TypeScript sync path resolve every path against
this one file (parity is enforced by test), so they can never drift.
- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a
`[framework]` glob matches and no `[operator]` carve-out overrides it.
- **Unknown paths default to operator** (fail-safe): a file the manifest never
anticipated is treated as operator-owned and is never pruned.
- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that
only ever iterates framework globs — operator and unknown paths are
structurally unreachable by the prune.
Result: a correct upgrade cannot touch operator config at all.
## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)
Before **any** mutation, the installer snapshots the operator-owned surface that
exists into:
```
${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/
```
- `0700` directories / `0600` files (`umask 077`, scoped and restored),
outside `~/.config/mosaic` and outside any repo.
- **Fail-open**: a snapshot failure warns but never aborts the upgrade it
protects.
- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5).
After the sync, a **verify net** compares each snapshot file against its target
and restores (with a loud warning) any operator file the upgrade diverged or
removed — a divergence means a manifest bug slipped through Layer 1.
Inspect and restore snapshots with the CLI:
```bash
mosaic restore --list # dry-run: enumerate snapshots by timestamp
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
mosaic restore --from <ts> --dry-run # preview a specific restore without writing
```
`mosaic restore` reports **counts and relative paths only** — it never emits file
contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores
are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf
atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails
closed rather than following out of the managed tree).
## Layer 3 — Regeneration from roster SSOT (recovery)
Some operator files are **derived** and do not need a byte-for-byte snapshot to
recover — they can be rebuilt from their source of truth. The fleet's per-agent
generated env projections are the prime case:
- `~/.config/mosaic/fleet/agents/<name>.env.generated` is a deterministic
projection of `~/.config/mosaic/fleet/roster.yaml`.
- The launcher (`start-agent-session.sh`, invoked by
`mosaic-agent@<name>.service`) sources that generated projection to establish
each agent's identity, runtime, model, and working directory. If it is missing
or wrong, the agent cannot launch with its intended identity.
`mosaic fleet regen` rebuilds those projections from the roster SSOT:
```bash
mosaic fleet regen # dry-run (default): show what would be rebuilt
mosaic fleet regen --json # same, machine-readable
mosaic fleet regen --write # rebuild the projections on disk
```
- **Dry-run by default.** Nothing is written until you pass `--write`.
- **Deterministic and idempotent** — the projection is a pure function of the
roster, so repeated `--write` runs produce byte-identical files.
- **Projection-only. It never restarts an agent.** Recovery order forbids
restart-before-verify; `regen` has no path to systemd lifecycle at all.
- **It rebuilds only `<name>.env.generated`** — it never writes, relocates, or
deletes the operator-owned `.env` / `.env.local` surface.
- It **validates the roster the same way `reconcile` does** (persona resolution
and protected-class tool-policy match), so a hand-edited or corrupt roster is
rejected rather than projected, and a `--write` takes the shared reconcile
lock so it cannot race a concurrent reconcile.
- Output is **paths and counts only** — the rendered `KEY=value` body is never
echoed.
`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a
recovered projection matches what a normal reconcile would have written.
## Recovery runbook — wiped `fleet/agents/*.env.generated`
If an upgrade (or a manual mistake) has left an agent without its generated
projection, **do not restart the unit first** — a launch against a missing
projection fails closed, and any stale state must be corrected before restart,
not after.
1. **Prefer a snapshot restore if one exists** (byte-exact operator state):
```bash
mosaic restore --list
mosaic restore --from <UTC-timestamp>
```
2. **Otherwise regenerate the derived projections from the roster SSOT:**
```bash
mosaic fleet regen # confirm the plan (create vs rebuild per agent)
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated
```
3. **Verify each unit will resolve the intended runtime/workdir _before_ any
restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal
environment and `start-agent-session.sh` sources `.env.generated` itself, so
verify the generated file directly and confirm the launcher path:
```bash
# Confirm fleet/agents/<name>.env.generated exists and carries the intended
# MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
# Confirm the unit launches the session script that reads it.
systemctl --user cat mosaic-agent@<name> | grep ExecStart
```
4. **Only then restart, one unit at a time:**
```bash
systemctl --user restart mosaic-agent@<name>
```
## See also
- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md)
- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md)
- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)

View File

@@ -214,3 +214,316 @@ Re-ran codex again; it found two more rollback-path gaps `set -E` cannot catch.
- Gates: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1094 · HARD GATE 193/193 · - Gates: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1094 · HARD GATE 193/193 ·
rollback 28/28 (was 14, +14 for D1/D2 with controls) · migration 21/21. shellcheck clean on new lines. rollback 28/28 (was 14, +14 for D1/D2 with controls) · migration 21/21. shellcheck clean on new lines.
No --no-verify. Committing FORWARD (no rebase of 34e55d4a/0a5e703a). No --no-verify. Committing FORWARD (no rebase of 34e55d4a/0a5e703a).
## Session 3 (2026-07-16) — PR1 MERGED, starting PR2 (durable snapshot + restore + secrev)
PR1 (#802) squash-merged → main `32a0ffba`; issue #791 stays open (3-PR DAG umbrella). Independent Opus
adversarial/security review APPROVED at head `af627e75` (Gitea RoR cmt 17892); lead ran rollback 28/28 +
HARD GATE 193/193 green; CI #1877 green. PR2 UNBLOCKED.
PR2 branch: `feat/791-pr2-snapshot-restore` off `origin/main` 32a0ffba. Same treatment applies:
tests-first red-first, independent review + durable Gitea Reviewer-of-Record comment BEFORE MS-LEAD runs
the queue guard/merge. Report PR2 number + exact head when ready. PR body: `Part of #791` (NOT Fixes).
### PR2 scope (ratified §3/§5 of design doc, Mos-approved — do NOT re-litigate)
- **(a) Durable pre-update snapshot** to `${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-ts>/`
— OUTSIDE ~/.config/mosaic and any repo. Perms dir 0700 / files 0600 (umask 077 + explicit chmod).
Scope = operator-owned surface that EXISTS (operatorReserved paths), not the framework tree. Taken
BEFORE any mutation. Retention N=5 (`MOSAIC_BACKUP_RETENTION`), prune older.
- **Post-sync verify + selective restore**: diff operator surface vs snapshot; (b) should never touch
operator paths, so ANY diff = manifest bug → restore affected paths + warn loudly. (a) catches a (b) miss.
- **`mosaic restore`** (TS CLI): `--list` (default, dry-run) enumerates snapshots by ts; `--from <ts>`
restores over operator surface, confirmation-gated. Counts/paths only.
- **Secret-safety (secrev)**: snapshot/restore NEVER emit file contents; only paths/counts. Tests assert
0700/0600 AND that a secret value seeded in tools/_lib/credentials.json never appears in any output.
### PR2 implementation status (2026-07-16, ready-for-review)
All three tasks implemented, red-first proven, unit-green:
- **Task #10 — durable snapshot (install.sh)**: `backup_root()`/`enumerate_operator_files()`/
`prune_durable_snapshots()`/`make_durable_snapshot()` wired into keep-mode main() after `manifest_load`,
before any mutation. umask 077 + explicit chmod 700/600. UTC ts, collision suffix. FAIL-OPEN (a backup
failure never aborts the upgrade it protects). Retention `MOSAIC_BACKUP_RETENTION` (default 5), in-place
`sort -r -o` prune (no `mv` — stays inside the rsync-absent coreutils whitelist).
- **Task #11 — post-sync verify net (install.sh)**: `verify_operator_surface()` runs after sync (trap
disarmed), `cmp -s` each snapshot file vs target; restores any diverged/missing operator file + warns
loudly (a divergence = manifest bug). VERIFY-NET wired before `cleanup_snapshot`.
- **Task #12`mosaic restore` (TS)**: `src/commands/restore.ts` + co-located spec (19 tests).
`--list` default (dry-run enumerate), `--from <ts>` confirmation-gated restore, `--dry-run`, `--yes`/
`MOSAIC_ASSUME_YES`. Injectable `confirm` for testability (proceed/decline/env-bypass covered). Restored
files forced 0600. Registered in `cli.ts`. Path convention mirrors install.sh `backup_root()`.
- **CI**: `.woodpecker/ci.yml` upgrade-guard runs the new `test-upgrade-durable-snapshot.sh` gate.
- **Gates green**: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1241 (+5) ·
durable-snapshot 26/26 · manifest-guard 193/193 · rollback 28/28 · migration 21/21.
Est. new-code coverage ≈93% (only the interactive readline default + process.exit-on-error uncovered).
- Regression fixed: PR2's `date`/`sort`/`mv` broke the rsync-absent manifest-guard PATH whitelist →
made date/sort fail-open, replaced `mv` with in-place `sort -o`, added `date sort` to the test whitelist
+ isolated `XDG_STATE_HOME`. All 193 manifest-guard assertions green under restricted PATH.
- Codex code-review + security-review (secrev) run on the uncommitted diff before commit.
### PR2 review round 1 — findings + remediations (2026-07-16, pre-PR)
Codex code-review returned **request-changes** (1 blocker + 3 should-fix); Codex security-review returned
**high** (1 high + 1 medium). Deduped to 5 distinct defects, ALL legitimate, ALL fixed FORWARD, each with
a red-first regression test whose control neuters exactly the guard under test:
- **A · BLOCKER — verify net undid the legacy bin/ migration (install.sh).** On a pre-v2 install `bin/**`
is operator-classified, so the durable snapshot captured it; `run_migrations()` deletes bin/ on purpose,
but `verify_operator_surface()` then saw it "missing" and healed it back — the migration would be silently
undone forever once the version stamps. **Fix:** `MIGRATION_REMOVED_PATHS[]` recorded by run_migrations
(`bin`,`rails`) + `is_migration_removed()` skip in the verify loop (`# MIGRATION-SKIP-GUARD`).
**Test:** Part 6 — v1 fixture with bin/; shipped keeps it removed + stamps v3; control (guard stripped)
wrongly restores bin/tool.sh.
- **B · HIGH (CWE-59) — restore/verify wrote secrets THROUGH a symlink (install.sh + restore.ts).** An
attacker swapping an operator path (e.g. tools/_lib/credentials.json) for a symlink after the snapshot
would make `cp`/`copyFileSync` write the snapshot's secret out through the link. **Fix (bash):** refuse a
symlinked ancestor (`has_symlinked_parent`), drop a symlinked leaf before restore
(`# SYMLINK-LEAF-GUARD`). **Fix (TS):** reuse audited `secure-file.ts``assertCanonicalContainment`
+ `ensureManagedDirectory` on every dst, open the leaf `O_NOFOLLOW|O_CREAT|O_TRUNC` 0600 (ELOOP =
fail-closed). **Tests:** Part 7 (shipped leaves external exfil target untouched, restores a real 0600
file; control leaks the secret through the link) + restore.spec symlinked-leaf/ancestor cases (red-first).
- **C · MEDIUM/should-fix (CWE-22) — `--from` traversal escaped the backup root (restore.ts).**
`join(root, from)` accepted `../poison`. **Fix:** validate the selector against
`^\d{8}T\d{6}Z(?:-\d+)?$`, build exactly `join(root,'pre-update-'+ts)`, `lstat` (reject symlinked snap
dir). **Test:** restore.spec `it.each` of 6 malformed selectors + `--from ../poison` fail-closed (red-first).
- **D · should-fix — verify `mkdir -p` unguarded under set -e (install.sh).** A parent replaced by a
regular file aborted the installer before the recovery pointer printed. **Fix:** guard `mkdir -p`, warn
+ `continue` on failure (keeps healing remaining files).
- **E · should-fix — snapshot `umask 077` leaked process-global (install.sh).** Later sync copies/dirs
inherited 0600/0700. **Fix:** save `old_umask`, restore on EVERY return path (`# UMASK-RESTORE-NORMAL`).
**Test:** Part 8 — synced framework file is 0644 while the secret backup stays 0600; control (restore
stripped) makes the synced file 0600.
**Full gate suite re-run after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic
vitest **1252** · restore.spec **30** · durable-snapshot **41** · manifest-guard 193 · rollback 28 ·
migration 21. shellcheck clean on all new lines; new test markers mirror the existing `# VERIFY-NET`
anchor convention. NOTE: codex self-review does NOT satisfy the independent-review gate — an independent
(author≠reviewer) review + durable Gitea Reviewer-of-Record comment is still required before MS-LEAD merges.
## Session 4 (2026-07-16) — PR2 MERGED, PR3 built (fleet regen — recovery layer)
PR2 (#811) squash-merged → main `31607a4a`; issue #791 stays open (final PR of the 3-PR DAG). Independent
exact-head RoR at `d12c5f78` APPROVE (Gitea cmt 17904); #1882 green; busybox-portable Part 7 control fix
verified in-Alpine. PR3 UNBLOCKED.
PR3 branch: `feat/791-pr3-fleet-regen` off `origin/main` 31607a4. Same discipline: tests-first red-first,
independent review + durable Gitea RoR BEFORE MS-LEAD runs the queue guard/merge. PR body `Part of #791`.
### PR3 scope (ratified §4/§7 of design doc) — `mosaic fleet regen`
Projection-only recovery command: rebuilds each `fleet/agents/<name>.env.generated` from `roster.yaml`
(SSOT). Dry-run default; `--write` applies; `--json` machine output. Structural guarantee: NO code path to
systemd lifecycle — **never restarts an agent**. Single-SSOT: reuses `projectRosterV2AgentGeneratedEnv`
(extracted, shared with the reconciler apply path) so regen and reconcile cannot drift. Secrev: paths +
counts only, never the rendered KEY=value body.
New files: `commands/fleet-regen-command.ts` (+ `.spec.ts`), guide `docs/guides/upgrade-safety-and-recovery.md`
(three-layer model: PR1 manifest ownership → PR2 snapshot/restore → PR3 regen; do-NOT-restart-before-verify
runbook), regen reference added to `docs/guides/fleet-local-canary.md`. Wired in `commands/fleet.ts`.
### Independent review (3 reviewers: subagent code-reviewer + codex code-review + codex security) → 4 fixes, red-first
- **A · BLOCKER (codex) — regen mutated/deleted legacy operator env.** `applyPreparedAgentEnvironmentProjection`
also writes `.env.local`/`.env.quarantine` and unlinks legacy `.env`. Violated projection-only contract.
**Fix:** NEW generated-only boundary primitives `prepareGeneratedAgentEnvironmentProjection` +
`applyPreparedGeneratedAgentEnvironmentProjection` (write ONLY `<name>.env.generated`). regen now has no
code path that touches `.env`/`.env.local`/`.env.quarantine`. **Test:** projection-only leaves legacy `.env`
verbatim, no local/quarantine fabricated.
- **B · should-fix (codex + subagent + security) — partial write on mid-loop failure.** Interleaved
prepare/apply left earlier agents written when a later agent failed prepare. **Fix:** PREPARE ALL agents
before writing ANY (mirrors reconciler `defaultPrepareProjections`). **Test:** 2nd agent's projection
pre-seeded 0644 → prepare rejects → coder0 NOT written, exit 1.
- **C · subagent — semantic-validation bypass.** Default readRoster skipped `validateRosterV2Semantics`, so
a tampered protected-class `tool_policy` would be silently projected. **Fix:** default readRoster now runs
`validateRosterV2Semantics` (persona resolution + protected-class match), rolesDir/overrideDir defaults
mirroring the reconciler. **Test:** merge-gate agent w/ tool_policy=code → fails closed, no write.
- **D · MEDIUM (codex security, CWE-362) — concurrent-reconcile race.** regen `--write` wrote without the
reconcile lock. **Fix:** `--write` acquires `acquirePrivateReconcileLock(mosaicHome)` for the whole
read-prepare-apply sequence, released in `finally`; dry-run stays lock-free. **Test:** pre-held lock →
regen fails closed, no write.
**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1265**
(regen spec 13, incl. 4 new red-first regressions). NOTE: codex self-review does NOT satisfy the
independent-review gate — an independent (author≠reviewer) review + durable Gitea RoR is still required
before MS-LEAD merges. STOP at PR-open for MS-LEAD's exact-head review; do NOT self-merge.
## Session 5 — PR3 review round 2 (finding L + M1/M2/M3), red-first fixes
Second review pass on the lock-cleanup plumbing surfaced one round-1 residual (L) and three round-2
findings (M1 blocker, M2/M3 should-fix). All fixed red-first (RED proven per-finding, then GREEN).
- **L · should-fix (codex r1) — mutation-lock release swallowed unlink failures.** regen's
`acquirePrivateRosterMutationLock` release copied CRUD's `unlink().catch(()=>{})`, hiding a stale
`roster.yaml.mutation.lock`. **Fix:** its release PROPAGATES the unlink fault (finding-J stale-lock
warning then fires for this lock too). **Test:** acquire real lock, `rm` it, assert `release()` rejects.
- **M1 · BLOCKER (codex r2) — replacement-lock race.** The propagating release from L did an
UNCONDITIONAL `unlink(lockPath)` without proving ownership. If the lock is cleared + re-created by
another writer mid-op, regen deletes the STRANGER's live lock → a third writer enters → mutual
exclusion defeated. **Fix (reuse, not reimplement):** generalized the reconciler's ownership-proving
lock body into shared `acquirePrivateManagedRosterLock(mosaicHome, lockLeaf, busyMessage, openLock)`;
`acquirePrivateReconcileLock` delegates to it (behavior-identical: same leaf/codes/messages), and a NEW
hardened `acquirePrivateRosterMutationLock` (now in fleet-reconciler.ts, leaf `roster.yaml.mutation.lock`)
records dev/ino + ownership token and RE-PROVES ownership (`assertLockOwnership`) before unlinking —
fails closed as `lock-cleanup-failed` if replaced. Removed the crud-based export; reverted
`acquireMutationLock` (fleet-agent-crud.ts) to its original inline empty-file/swallowing-release form
(CRUD behavior intentionally unchanged). Compatibility: CRUD empty-file `wx` and regen tokened `wx`
contend on the same path but never co-own (wx winner owns; loser → concurrent-mutation), so the token
is only ever read back by the same regen invocation. **Test:** acquire, `rm`+recreate lock (new inode),
assert `release()` rejects AND the replacement survives (not unlinked).
- **M2 · should-fix (codex r2) — acquire-unwind fault dropped.** The acquire-failure catch discarded
`releaseFleetLocks`' return (a possible fault on the already-held first lock). **Fix:** capture and
augment — `const releaseFault = await releaseFleetLocks(releases); throw augmentWithLockCleanupFault(error, releaseFault);`
(symmetric to finding J). **Test:** mutation lock acquires w/ faulting release + reconcile acquire
throws → thrown error mentions stale/lock, nothing written.
- **M3 · should-fix (codex r2 + subagent REQUEST-CHANGES) — cleanup warning named only reconcile lock.**
Finding L made the mutation-lock release fault reachable, so the `cleanup` marker can originate from
EITHER lock. **Fix:** `formatFleetRegenReport`'s WARNING now names BOTH `roster.yaml.mutation.lock` and
`roster.yaml.reconcile.lock`, matching `augmentWithLockCleanupFault`. **Test:** fault the mutation-lock
release specifically → report names both lock files.
**Refactor note (no cycle):** neither fleet-reconciler nor fleet-agent-crud imports the other; regen
imports lock acquirers from fleet-reconciler and the projection mapping from fleet-reconciler. The two
reconcile-lock reviewers reconciled: independent reviewer validated acquire-time empty-file compatibility
(preserved), codex flagged RELEASE-time replacement race (closed by ownership proof) — non-contradictory.
**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1275**
(regen spec 23, incl. 7 red-first lock regressions E/F/G/K/L/M1/M2/M3). RED proven per-finding by
temporary revert before re-applying each fix. Independent (author≠reviewer) review of M1/M2/M3 + codex
code/security re-run in flight. STOP at PR-open for MS-LEAD's exact-head review + durable Gitea RoR; do
NOT self-merge; #791 umbrella stays OPEN; PR body `Part of #791`.
### Round 3 review (after M1/M2/M3) — independent review PASS + codex residual-TOCTOU disposition
Three reviewers on the post-M1/M2/M3 head:
- **Independent (subagent, author≠reviewer) — PASS.** Verified M1/M2/M3 all correctly fixed; "never
restarts" is STRUCTURAL (runner never referenced in executable code); no secrets; no deadlock (only
regen holds both locks); tests meaningful (assert inode preservation + exact lock-file names). Raised:
- **should-fix #1 (fixed, red-first):** generalizing the lock helper left `assertSafeLockLeafIfPresent`/
`assertLockOwnership` hardcoding "reconciliation lock" in thrown messages → a MUTATION-lock fault
misreported as the reconcile lock, undercutting M3's accurate-diagnosis goal. **Fix:** thread
`lockLabel = fleet/<leaf>` through both helpers + the generic lock-io messages, so every fault names
the actual lock file. Red-first: strengthened the M1 test to assert `/roster\.yaml\.mutation\.lock/`
(RED: got "reconciliation lock"; GREEN after). Also resolves nit #3 (generic-message drift).
- **nit #2 (fixed):** `FleetRegenResult.cleanup` JSDoc still said "the shared reconcile lock"; now names
both locks (regen holds both).
- **nit #4 (fixed):** removed the redundant duplicate `assertLockOwnership` call before unlink
(pre-existing in merged main; harmless but dead — dropped since the fn was already being touched).
- **Codex security — clean (risk: none).** Validates roster semantics, constrains env values, no shell
eval, no secret output, generated-only writes, serialized against both locks.
- **Codex code — request-changes, 1 "blocker": residual check-then-unlink TOCTOU.** Between the final
`assertLockOwnership` and the path-based `unlink`, an external actor could vacate our inode and a new
writer grab the path, so the unlink deletes the stranger's lock. **Disposition: documented known
limitation, NOT fixed in PR3.** Rationale: (1) byte-identical to the MERGED, shipped reconcile-lock
release on origin/main (fleet-reconciler.ts L654-659) — not introduced here; (2) UNREACHABLE within the
`wx` writer protocol — no Mosaic writer removes a lock it doesn't own (wx fails EEXIST while our inode
exists), so only external interference can vacate our inode in the sub-instruction window; (3) the
ownership guard DOES close the reachable case (stale-lock reaper/operator cleared our lock + another
writer took it BEFORE release began → fail closed, don't delete stranger's lock); (4) the true atomic
fix — fd-held advisory lock (flock/lockf) adopted by ALL fleet writers (CRUD + reconcile + regen) — is
a cross-cutting mechanism change touching merged CRUD + reconciler, out of scope for a projection-only
recovery PR. Documented honestly in the acquirer doc + M1 test comment. **The binding independent
review did NOT treat this as a blocker.** Recommendation to MS-LEAD: proceed to PR-open + spin a
SEPARATE follow-up issue for the fd-advisory-lock migration; MS-LEAD adjudicates scope at exact-head
review (merge authority).
**Gates after round-3 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
**1275** (regen spec 23). Fresh codex code re-run in flight to confirm no NEW issues from the label fix.
---
## Session 6 — Round 4/5 convergence (stranded-lock robustness)
**Two independent reviewers converged on the SAME should-fix** on the init-failure cleanup path,
strengthening confidence it was real:
- **Codex code-review-5 — 0 blockers, 1 should-fix.** "Stat failure after lock creation strands the new
lock." When `handle.stat()` ITSELF fails right after the `wx` create (transient EIO/EBADF), `created`
is `undefined`, so `removeOwnedLockLeafBestEffort` had `if (!created) return;` → no cleanup → the
just-created `roster.yaml.mutation.lock`/`reconcile.lock` is stranded, permanently blocking future
regen + CRUD. (Notably NO blocker, and the TOCTOU is no longer flagged in code-review as of r5.)
- **Independent delta reviewer (author≠reviewer, pr-review-toolkit) — no blockers, same should-fix.**
Independently flagged the identical `!created` gap; validated FIX 1 (label threading — no call site
missed, codes unchanged, no test depended on old text) and FIX 2 (dev/ino-guarded cleanup, best-effort,
happy-path release reuses captured dev/ino) as correct. Suggested an unconditional best-effort unlink
in the `!created` branch; I took the **safer** variant below.
- **Codex security-review-5 — 0 crit / 0 high / 1 medium.** The single medium is the SAME residual
check-then-unlink TOCTOU already dispositioned in round 3 (its own remediation = "migrate every writer
to an fd-held advisory lock" = the follow-up issue). No new security finding. No secrets.
**Fix (red-first, safer than an unconditional unlink):** thread the persisted random `token` into
`removeOwnedLockLeafBestEffort`. Two independent ownership proofs now: primary dev/ino (unchanged), and a
**fallback** when the post-create stat failed — read the leaf and unlink ONLY if its content equals our
`randomUUID()` token. Only OUR lock carries that token, so a CRUD (empty) or differently-tokened
replacement is never deleted. `tokenPersisted` guards passing the token (only after `writeFile` lands).
Doubly-degenerate case (stat fails AND token write never landed) leaves the lock in place rather than
risk deleting a stranger's file — requires two independent fs faults on a just-created fd; documented.
- **Red-first proof:** new test `does not strand the lock file when the post-create stat itself fails`
injects a real `wx` create + a Proxy handle whose `stat()` rejects (writeFile/close succeed), asserts
`exists(lockPath) === false`. RED before fix (`expected true to be false` — lock stranded); GREEN after.
- **Also fixed (delta nit #3):** `fleet-regen-command.ts` `acquireRosterMutationLock` JSDoc said "CRUD's
private lock"; the default is the reconciler's hardened ownership-proving acquirer for the same
`fleet/roster.yaml.mutation.lock` path. Corrected.
- **PR-description note (delta nit #2):** FIX 1 also collapsed a pre-existing duplicate back-to-back
`assertLockOwnership` call in the release closure (identical args, no intervening logic) into one — a
no-op simplification of merged code, not a behavior change. Called out so a future reader doesn't
wonder if the duplicate had a purpose.
**Gates after round-4 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
**1277** (regen spec now 25: +1 stat-failure stranded-lock regression). Residual TOCTOU still deferred to
the fd-advisory-lock follow-up issue; MS-LEAD adjudicates scope at exact-head review (merge authority).
---
## Session 6 — Round 6 (persona-root wiring)
**Codex code-review-6 — 0 blockers, 1 should-fix (NEW, distinct from the lock work).** "Forward
configured persona directories to regen." `registerFleetRegenCommand` was registered at
`fleet.ts:2069` with only `{ runner, mosaicHome }`, discarding `deps.reconcileDeps.rolesDir` /
`overrideDir`. The regen command ALREADY has those seams (validates roster semantics via
`validateRosterV2Semantics({ rolesDir, overrideDir })`, defaulting to `<mosaicHome>/fleet/roles{,.local}`),
but the top-level wiring never forwarded the configured roots. **Impact:** in a deployment with custom
persona roots, `fleet reconcile` (which honors the overrides) would ACCEPT a roster while `fleet regen`
REJECTS the same roster (persona resolution against the wrong default dir) — blocking the recovery
command and violating the documented "resolves personas the SAME way reconcile does" contract.
**Fix (red-first):** forward `rolesDir`/`overrideDir` from `deps.reconcileDeps` into
`registerFleetRegenCommand` at `fleet.ts:2069`. Red-first test `forwards configured persona roots
(rolesDir/overrideDir) from reconcileDeps into regen`: seeds personas ONLY under a custom root, leaves
the default `<home>/fleet/roles` empty, registers with `reconcileDeps: { rolesDir, overrideDir }`, and
requires `fleet regen` to SUCCEED. RED before fix (`expected 1 not to be 1` — regen validated against the
empty default and exited 1); GREEN after.
**Codex security-review-6 — 0 crit / 0 high / 1 medium.** Same residual check-then-unlink TOCTOU, now
noted at BOTH the release closure and the init-cleanup path; remediation = fd-held advisory lock across
all writers = the SAME deferred follow-up item. No new security finding, no secrets.
**Independent confirmation review of the token-fallback fix (Session 6/round 4) — PASS, no findings.**
All 7 verification points confirmed; reviewer mechanically reverted `removeOwnedLockLeafBestEffort` to
the pre-fix `if (!created) return;` and re-ran the new test → RED (`expected true to be false`),
confirming the test genuinely pins the fix; restored after. No lint/type issues; doc-comment accurate.
**Gates after round-6 fix (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
**1278** (regen spec now 26: +1 persona-root wiring regression).
---
## Session 6 — Round 7 convergence (review CLOSED for PR-open)
- **Codex code-review-7 — 0 blockers, 1 should-fix = the residual TOCTOU** (previously a "blocker" in r3,
dropped in r4/r5, now re-surfaced as a should-fix). **Codex security-review-7 — 0 crit / 0 high /
1 medium = the SAME residual TOCTOU.** Codex has CONVERGED: the only remaining finding across both
streams is that one race, whose own remediation is "fd-held advisory lock shared by all fleet writers"
= the deferred follow-up. No new distinct finding; the wiring fix introduced nothing.
- **Independent confirmation review of the persona-root wiring fix — PASS, no findings.** Reviewer
mechanically reverted the two forwarded lines → RED (`Roster v2 agent "coder0" class "code" does not
resolve to a readable persona` → exit 1), restored → GREEN (26 regen + 204 fleet tests). Confirmed the
optional-chaining fallback preserves default-deployment behavior and no type/lint issue.
**Review disposition for PR-open:** ALL actionable findings fixed red-first across rounds 36 (label
threading, stranded-lock on init failure, stat-failure strand, persona-root wiring). The residual
check-then-unlink TOCTOU is the ONLY open item and is DEFERRED to a follow-up issue (fd-advisory-lock
migration across CRUD + reconcile + regen) — byte-identical to merged origin/main's reconcile-lock
release, unreachable within the `wx` writer protocol (no Mosaic writer removes a lock it doesn't own;
only external `rm`/a stale-lock reaper can vacate the inode mid-release), and its true fix is a
cross-cutting mechanism change out of scope for a projection-only recovery PR. Two independent human-agent
reviews (author≠reviewer) treated it as non-blocking. MS-LEAD adjudicates scope at exact-head review
(merge authority); recommendation = proceed to PR-open + spin the follow-up issue.
**Final gates (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1278**
(regen spec 26). No secret values in any snapshot/projection/report output (counts + paths only). Regen
NEVER issues a lifecycle/restart call (load-bearing recordingRunner gate). STOP at PR-open for MS-LEAD's
exact-head review + durable Reviewer-of-Record before any merge; do NOT self-merge.

View File

@@ -109,6 +109,225 @@ restore_snapshot() {
} }
cleanup_snapshot() { [[ -n "$SNAPSHOT_DIR" && -d "$SNAPSHOT_DIR" ]] && rm -rf "$SNAPSHOT_DIR"; SNAPSHOT_DIR=""; } cleanup_snapshot() { [[ -n "$SNAPSHOT_DIR" && -d "$SNAPSHOT_DIR" ]] && rm -rf "$SNAPSHOT_DIR"; SNAPSHOT_DIR=""; }
# ─── durable operator-config snapshot (#791 PR2) ─────────────────────────────
# A SECOND, independent safety layer, distinct from SNAPSHOT_DIR above:
# • SNAPSHOT_DIR is ephemeral (/tmp, deleted on success) and mirrors the WHOLE
# target for CRASH rollback if the sync aborts mid-write.
# • DURABLE_SNAPSHOT_DIR is RETAINED, holds only the operator-owned surface, and
# lives OUTSIDE the framework tree and any repo. It exists for the failure the
# crash-rollback cannot see: a sync that finishes "successfully" yet a
# manifest/logic bug let it modify an operator file. verify_operator_surface()
# (post-sync) heals from it; `mosaic restore` recovers from it days later.
# Path convention is mirrored in packages/mosaic/src/commands/restore.ts — keep
# the two in sync (there is no shared code across the bash/TS boundary).
DURABLE_SNAPSHOT_DIR=""
backup_root() { printf '%s/mosaic/backups' "${XDG_STATE_HOME:-$HOME/.local/state}"; }
# Relative paths that a migration INTENTIONALLY removes from the target (e.g. the
# legacy bin/ tree). Such a path is operator-classified by the manifest (unknown⇒
# operator), so the durable snapshot captures it — but its post-migration absence
# is correct, NOT a manifest bug. run_migrations() records each removal here so
# verify_operator_surface() does not "heal" it back and silently undo the
# migration (which would then be skipped forever once the version is stamped).
MIGRATION_REMOVED_PATHS=()
# True (0) if $1 (a path relative to TARGET_DIR) equals or lives under a path a
# migration deliberately removed this run.
is_migration_removed() {
local rel="$1" removed
for removed in ${MIGRATION_REMOVED_PATHS[@]+"${MIGRATION_REMOVED_PATHS[@]}"}; do
[[ -n "$removed" ]] || continue
[[ "$rel" == "$removed" || "$rel" == "$removed"/* ]] && return 0
done
return 1
}
# True (0) if any parent directory of $1 (relative to TARGET_DIR) is a symlink.
# Restoring THROUGH a symlinked ancestor would let cp write snapshot contents —
# possibly secrets — outside the target (CWE-59), so the verify net refuses it.
has_symlinked_parent() {
local rel="$1" dir p seg
dir="$(dirname "$rel")"
[[ "$dir" == "." ]] && return 1
p="$TARGET_DIR"
local IFS='/'
for seg in $dir; do
[[ -n "$seg" ]] || continue
p="$p/$seg"
[[ -L "$p" ]] && return 0
done
return 1
}
# Emit (NUL-delimited, into file $1) the operator-owned relative paths that exist
# under TARGET_DIR, classified via the shared manifest (deny-wins; unknown⇒
# operator). Returns non-zero if the filesystem walk itself failed — we must
# NEVER snapshot from a truncated scan (a `< <(find …)` process substitution
# would hide that error; capture-then-check does not — cf. #791 blocker-D1).
enumerate_operator_files() {
local out="$1" scan abs rel
scan="$(mktemp)"
if ! find "$TARGET_DIR" -type f -print0 > "$scan"; then
rm -f "$scan"
return 1 # OP-SCAN-GUARD
fi
: > "$out"
while IFS= read -r -d '' abs; do
rel="${abs#"$TARGET_DIR"/}"
# Not operator config: version marker and any VCS metadata.
case "$rel" in .framework-version|.git|.git/*) continue ;; esac
manifest_is_framework "$rel" || printf '%s\0' "$rel" >> "$out"
done < "$scan"
rm -f "$scan"
}
# Retain only the newest MOSAIC_BACKUP_RETENTION (default 5) snapshots. The
# pre-update-<UTC-ts> names sort lexicographically = chronologically, so a
# reverse sort is newest-first. Pruning failures are non-fatal (they only leave
# extra old backups); the enclosing find's status is still honored, not swallowed.
prune_durable_snapshots() {
local root keep list d i=0
root="$(backup_root)"
keep="${MOSAIC_BACKUP_RETENTION:-5}"
[[ "$keep" =~ ^[0-9]+$ ]] && (( keep >= 1 )) || keep=5
list="$(mktemp)"
if ! find "$root" -maxdepth 1 -type d -name 'pre-update-*' > "$list"; then
rm -f "$list"; return 0
fi
# Newest-first ordering needs `sort` (`-o` writes back in place — no `mv`
# dependency); if it is somehow unavailable, leave the backups untouched rather
# than risk pruning in an undefined order.
if ! LC_ALL=C sort -r -o "$list" "$list" 2>/dev/null; then
rm -f "$list"; return 0
fi
while IFS= read -r d; do
[[ -n "$d" ]] || continue
i=$((i + 1))
(( i > keep )) && rm -rf "$d"
done < "$list"
rm -f "$list"
}
# Take the durable pre-update snapshot BEFORE any mutation. Fail-OPEN: the durable
# snapshot is a recovery bonus on top of the manifest (which already keeps the
# sync out of operator paths) and the crash-rollback — so an un-writable backup
# location warns and continues rather than blocking the upgrade. Everything it
# creates is private (umask 077 + explicit 0700 dirs / 0600 files): the snapshot
# mirrors operator config, which may hold secrets, and must never be world-readable.
make_durable_snapshot() {
is_existing_install || return 0
local root ts dir list rel src dst count=0 old_umask
root="$(backup_root)"
# Fail-open if we cannot even stamp a timestamp: the durable snapshot is a
# recovery bonus and must never be the thing that aborts an upgrade.
ts="$(date -u +%Y%m%dT%H%M%SZ 2>/dev/null || true)"
if [[ -z "$ts" ]]; then
warn "Durable snapshot skipped: no UTC timestamp available (upgrade continues)."
return 0
fi
# umask 077 makes every dir/file the snapshot creates private from birth (it
# mirrors operator config, which may hold secrets). It is PROCESS-global, so we
# save and restore it around exactly this block — otherwise every later sync
# copy and new framework dir would inherit 0600/0700 instead of 0644/0755.
old_umask="$(umask)"
umask 077
if ! mkdir -p "$root"; then
umask "$old_umask"
warn "Durable snapshot skipped: cannot create backup dir $root (upgrade continues; operator files remain manifest-protected)."
return 0
fi
chmod 700 "$root" 2>/dev/null || true
dir="$root/pre-update-$ts"
if [[ -e "$dir" ]]; then # same-second re-run: disambiguate
local n=1; while [[ -e "$dir-$n" ]]; do n=$((n + 1)); done; dir="$dir-$n"
fi
if ! mkdir -p "$dir"; then
umask "$old_umask"
warn "Durable snapshot skipped: cannot create $dir (upgrade continues)."
return 0
fi
chmod 700 "$dir"
list="$(mktemp)"
if ! enumerate_operator_files "$list"; then
umask "$old_umask"
warn "Durable snapshot skipped: could not enumerate operator files (upgrade continues)."
rm -f "$list"; rmdir "$dir" 2>/dev/null || true
return 0
fi
while IFS= read -r -d '' rel; do
src="$TARGET_DIR/$rel"; dst="$dir/$rel"
[[ -f "$src" ]] || continue
mkdir -p "$(dirname "$dst")"
if ! cp "$src" "$dst"; then
warn "Durable snapshot: could not copy operator file '$rel' (skipped)."
continue
fi
chmod 600 "$dst" 2>/dev/null || true
count=$((count + 1))
done < "$list"
rm -f "$list"
# Tighten every dir the copy created (mkdir -p honors umask, but be explicit).
find "$dir" -type d -exec chmod 700 {} + 2>/dev/null || true
umask "$old_umask" # UMASK-RESTORE-NORMAL — restore before the upgrade proper resumes (see above)
DURABLE_SNAPSHOT_DIR="$dir"
ok "Durable pre-update snapshot: $count operator file(s) saved to $dir (recover with: mosaic restore --list)"
prune_durable_snapshots
}
# Post-sync safety net: a keep-mode upgrade must NEVER modify an operator file.
# Compare every file in the durable snapshot to its current target counterpart;
# any that changed (or vanished) was touched by a framework bug — restore it from
# the snapshot and warn loudly. This does NOT abort: the framework itself synced
# correctly; we only heal the operator collateral. Runs after the restore trap is
# disarmed so its corrective copies can't spuriously trip a full rollback, and
# every step is guarded so `set -e` cannot exit silently mid-heal (cf. blocker-D2).
verify_operator_surface() {
[[ -n "$DURABLE_SNAPSHOT_DIR" && -d "$DURABLE_SNAPSHOT_DIR" ]] || return 0
local scan snap rel cur healed=0
scan="$(mktemp)"
if ! find "$DURABLE_SNAPSHOT_DIR" -type f -print0 > "$scan"; then
rm -f "$scan"
warn "Post-upgrade verify skipped: could not enumerate the pre-update snapshot at $DURABLE_SNAPSHOT_DIR."
return 0
fi
while IFS= read -r -d '' snap; do
rel="${snap#"$DURABLE_SNAPSHOT_DIR"/}"
cur="$TARGET_DIR/$rel"
# A migration may legitimately delete an operator-classified path (e.g. legacy
# bin/). Its absence is intended — do not heal it back, or the migration is
# silently undone and never re-runs once the version is stamped (#791 PR2).
is_migration_removed "$rel" && continue # MIGRATION-SKIP-GUARD
if [[ ! -e "$cur" ]] || ! cmp -s "$snap" "$cur"; then
# Never restore THROUGH a symlink: an operator path swapped for a link would
# otherwise let cp write snapshot contents (possibly secrets) outside the
# target (CWE-59). Refuse a symlinked parent; drop a symlinked leaf and write
# a real file in its place.
if has_symlinked_parent "$rel"; then
warn "Operator path '$rel' has a symlinked parent under $TARGET_DIR; refusing to restore through it (possible tampering) — recover it manually from $DURABLE_SNAPSHOT_DIR."
continue
fi
[[ -L "$cur" ]] && rm -f "$cur" # SYMLINK-LEAF-GUARD
# Guard mkdir too: under set -e (trap already disarmed) a bare failure would
# exit the whole installer before the recovery pointer below is emitted.
if ! mkdir -p "$(dirname "$cur")"; then
warn "Operator file '$rel' was modified by the upgrade but could NOT be auto-restored (parent dir unavailable) — recover it manually from $DURABLE_SNAPSHOT_DIR."
continue
fi
if cp "$snap" "$cur"; then
chmod 600 "$cur" 2>/dev/null || true
warn "Operator file was modified by the upgrade and has been restored from the pre-update snapshot: $rel"
healed=$((healed + 1))
else
warn "Operator file '$rel' was modified by the upgrade but could NOT be auto-restored — recover it manually from $DURABLE_SNAPSHOT_DIR."
fi
fi
done < "$scan"
rm -f "$scan"
if (( healed > 0 )); then
warn "$healed operator file(s) were unexpectedly changed by this upgrade and were restored from the pre-update snapshot. A keep-mode upgrade must never modify operator files — this indicates a framework manifest bug; please report it (#791)."
fi
}
# Reconcile contract files after sync: framework-owned overwrite (backup-once), # Reconcile contract files after sync: framework-owned overwrite (backup-once),
# user-seeded seed-if-absent. # user-seeded seed-if-absent.
reconcile_framework_files() { reconcile_framework_files() {
@@ -326,6 +545,10 @@ run_migrations() {
# Remove bin/ directory — all executables now live in the npm CLI. # Remove bin/ directory — all executables now live in the npm CLI.
# Scripts that were in bin/ are now in tools/_scripts/. # Scripts that were in bin/ are now in tools/_scripts/.
if [[ "$from_version" -lt 2 ]]; then if [[ "$from_version" -lt 2 ]]; then
# bin/ and the rails symlink are operator-classified by the manifest (unknown⇒
# operator) and thus captured in the durable snapshot; record them as
# intentional removals so the post-sync verify net does not restore them.
MIGRATION_REMOVED_PATHS+=("bin" "rails")
if [[ -d "$TARGET_DIR/bin" ]]; then if [[ -d "$TARGET_DIR/bin" ]]; then
ok "Removing legacy bin/ directory (executables now in npm CLI)" ok "Removing legacy bin/ directory (executables now in npm CLI)"
rm -rf "$TARGET_DIR/bin" rm -rf "$TARGET_DIR/bin"
@@ -383,6 +606,9 @@ fi
# not for a validation failure that has touched nothing yet (#791 blocker-1). # not for a validation failure that has touched nothing yet (#791 blocker-1).
if [[ "$INSTALL_MODE" == "keep" ]]; then if [[ "$INSTALL_MODE" == "keep" ]]; then
manifest_load manifest_load
# Durable, operator-scoped backup taken BEFORE any mutation (#791 PR2). Kept
# outside the framework tree; recovered later via `mosaic restore`. Fail-open.
make_durable_snapshot
fi fi
# Snapshot before any destructive file operation; restore on interrupt/failure. # Snapshot before any destructive file operation; restore on interrupt/failure.
@@ -421,6 +647,10 @@ run_migrations
# File-system phase complete and consistent — clear the restore trap. # File-system phase complete and consistent — clear the restore trap.
trap - ERR INT TERM trap - ERR INT TERM
# Post-sync safety net: heal any operator file a manifest bug let the sync touch,
# using the durable pre-update snapshot (#791 PR2). Runs with the trap disarmed so
# a corrective copy can't spuriously trigger a full rollback.
verify_operator_surface # VERIFY-NET (#791 PR2)
cleanup_snapshot cleanup_snapshot
# Testability / minimal-install hook: stop after the file-system phase, before any # Testability / minimal-install hook: stop after the file-system phase, before any

View File

@@ -0,0 +1,313 @@
#!/usr/bin/env bash
# test-upgrade-durable-snapshot.sh — the #791 PR2 regression gate.
#
# PR1 gave keep-mode upgrades two protections: the manifest (a keep-sync only
# ever writes framework-owned paths — operator config is structurally untouched)
# and an EPHEMERAL /tmp snapshot that rolls the whole target back if the sync
# CRASHES mid-write. PR2 adds a third, independent layer for the case neither
# covers: a "successful" upgrade that a manifest/logic bug silently let touch an
# operator file. That layer is a DURABLE, operator-scoped pre-update snapshot:
#
# Part 1 (scope): before any mutation, the installer copies exactly the
# operator-owned files that exist into a retained backup
# under $XDG_STATE_HOME/mosaic/backups/pre-update-<ts>/ —
# framework files are NOT captured.
# Part 2 (perms): the backup root, snapshot dir and every nested dir are
# 0700; every backed-up file is 0600 (never world-readable,
# even though operator config may hold secrets).
# Part 3 (no leak): a secret seeded into credentials.json is copied into the
# snapshot (proving coverage) but its value never appears
# on stdout/stderr — the snapshot reports counts/paths only.
# Part 4 (retention): only the newest MOSAIC_BACKUP_RETENTION snapshots survive;
# older ones are pruned.
# Part 5 (verify net): if the upgrade DID modify an operator file (injected here
# with a cp shim that scribbles on SOUL.md while a framework
# file is copied), the post-sync verify restores that file
# from the durable snapshot and warns loudly. The control —
# the same installer with the verify call stripped — leaves
# the corruption in place, proving the net is load-bearing.
#
# Usage: bash test-upgrade-durable-snapshot.sh
set -uo pipefail
FW="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" # packages/mosaic/framework
INSTALL="$FW/install.sh"
ORIG_PATH="$PATH"
FRAMEWORK_VERSION="$(grep -m1 '^FRAMEWORK_VERSION=' "$INSTALL" | cut -d= -f2)"
# Control installers must live INSIDE $FW: install.sh derives SOURCE_DIR from its
# own path and sources tools/_lib/manifest.sh relative to it, so a copy anywhere
# else aborts before the sync. Each control is a shipped installer with one guard
# line stripped (keyed off a `# <MARKER>` anchor), proving that guard load-bearing.
# All controls share the .install-*.tmp.sh glob so one trap sweeps them on exit.
VERIFYCTRL="$FW/.install-verifynet-control.tmp.sh"
rm -f "$FW"/.install-*.tmp.sh
trap 'rm -f "$FW"/.install-*.tmp.sh' EXIT
# mk_control <marker-regex> <name> — echo a control installer path ($FW-local) that
# is $INSTALL with every line matching /<marker-regex>/ deleted.
mk_control() {
local path="$FW/.install-$2.tmp.sh"
sed "/$1/d" "$INSTALL" > "$path"
printf '%s' "$path"
}
pass=0; fail=0
chk() { if eval "$2"; then echo "$1"; pass=$((pass + 1)); else echo "$1"; fail=$((fail + 1)); fi; }
SECRET='SUPER-SECRET-TOKEN-do-not-log-pr2'
SOUL_ORIG='# persona'
# A framework file the sync copies (source ships it; the seeded target omits it,
# so the bytes differ and cp is attempted). The Part-5 shim keys off this path.
POISON_REL='guides/E2E-DELIVERY.md'
# Seed a recognized keep-mode install holding four operator-owned files across
# the identity file, an operator subtree, memory, and the credentials carve-out.
seed_home() {
local H="$1"
mkdir -p "$H/agents" "$H/tools/_lib" "$H/memory"
printf '%s\n' "$SOUL_ORIG" > "$H/SOUL.md" # recognized install → keep mode
printf 'MODEL=opus\n' > "$H/agents/coder0.conf"
printf '# operator memory\n' > "$H/memory/note.md"
printf 'TOKEN=%s\n' "$SECRET" > "$H/tools/_lib/credentials.json"
echo 3 > "$H/.framework-version"
# Deliberately NO guides/E2E-DELIVERY.md so the sync copies it (framework file,
# bytes differ) — that copy is where the Part-5 corruption shim fires.
}
# A pre-v2 (legacy) keep-mode install: SOUL.md marks it recognized, and a bin/
# tree with NO .framework-version makes installed_framework_version() report 1, so
# the v1→v2 migration (which deletes bin/) runs. bin/ is unknown⇒operator, so the
# durable snapshot captures it — the verify net must NOT heal the intended removal.
seed_home_v1() {
local H="$1"
mkdir -p "$H/agents" "$H/tools/_lib" "$H/memory" "$H/bin"
printf '%s\n' "$SOUL_ORIG" > "$H/SOUL.md"
printf 'TOKEN=%s\n' "$SECRET" > "$H/tools/_lib/credentials.json"
printf '#!/bin/sh\necho legacy\n' > "$H/bin/tool.sh"; chmod +x "$H/bin/tool.sh"
# Deliberately NO .framework-version and NO guides/E2E-DELIVERY.md (see seed_home).
}
# A cp shim that, while the framework POISON file is being copied during sync,
# swaps the operator credentials file for a symlink pointing at an attacker-
# readable file OUTSIDE the target — simulating post-snapshot tampering (CWE-59).
# The durable snapshot already holds the real credentials (it is taken before any
# sync), so the verify net must restore a REAL file in place WITHOUT following the
# link (which would write the snapshot's secret out through it). $EXFIL_TARGET is
# expanded at shim-write time from the caller's environment.
#
# PORTABILITY (why this shim, not the real `cp`): the CWE-59 leak this exercises is
# `cp` writing THROUGH a symlinked destination. GNU/BSD cp — what a real operator
# runs `mosaic update` under — follows the dest symlink and leaks. busybox cp (the
# Alpine CI image) REPLACES a symlinked dest instead of following it, so under the
# CI harness the leak vector simply does not exist and the negative control could
# never reproduce it. This shim therefore emulates the real-target GNU cp behavior
# PORTABLY: when the destination is a symlink it writes the source bytes through the
# link via redirection (which follows symlinks on every coreutils, busybox included);
# otherwise it delegates to the host's real cp unchanged. Both the shipped-case and
# the negative control run through this identical shim, so the ONLY difference
# between them remains the SYMLINK-LEAF-GUARD — the control stays load-bearing and
# non-tautological. It does NOT touch install.sh (approved) or the real assertions:
# with the guard present the symlinked leaf is dropped BEFORE this cp runs, so the
# dest is a real file and the delegate path is taken exactly as on a GNU host.
make_symlink_leaf_shim() {
local dir="$1" home="$2"
cat > "$dir/cp" <<SHIM
#!/usr/bin/env bash
dest="\${@: -1}"
src="\${@:(-2):1}"
case "\$dest" in
*/$POISON_REL)
rm -f "$home/tools/_lib/credentials.json"
ln -s "$EXFIL_TARGET" "$home/tools/_lib/credentials.json"
;;
esac
# Coreutils-agnostic emulation of GNU cp's follow-through-dest-symlink behavior.
if [[ -L "\$dest" && -f "\$src" ]]; then
cat "\$src" > "\$dest"
exit \$?
fi
exec env PATH="$ORIG_PATH" cp "\$@"
SHIM
chmod +x "$dir/cp"
}
# A cp shim that, while the framework POISON file is being copied during sync,
# also appends garbage to the operator SOUL.md — simulating a manifest bug that
# writes outside the framework lane. The framework copy itself still succeeds
# (real cp runs), so the sync completes 0 and the post-sync verify is what must
# catch and undo the operator-file damage. The snapshot's own cp only ever
# targets operator files (never guides/…), so it is never corrupted by this shim.
make_corrupt_shim() {
local dir="$1" home="$2"
cat > "$dir/cp" <<SHIM
#!/usr/bin/env bash
dest="\${@: -1}"
case "\$dest" in
*/$POISON_REL) printf 'CORRUPTION-mid-sync\n' >> "$home/SOUL.md" 2>/dev/null || true ;;
esac
exec env PATH="$ORIG_PATH" cp "\$@"
SHIM
chmod +x "$dir/cp"
}
# Run one keep-mode, sync-only upgrade with $XDG_STATE_HOME redirected to a
# throwaway dir (so the real ~/.local/state is never touched). Optional args:
# $2 shim-maker (default none), $3 MOSAIC_BACKUP_RETENTION (default unset).
# Echoes: "<exit>\t<out>\t<state-dir>\t<home>".
# $4 seeder (default seed_home) — swap in seed_home_v1 for the migration case.
run_snap() {
local installer="$1" shim_maker="${2:-}" retention="${3:-}" seeder="${4:-seed_home}" H STATE OUT SHIM rc pathpre
H=$(mktemp -d); STATE=$(mktemp -d); OUT=$(mktemp); pathpre="$ORIG_PATH"
"$seeder" "$H"
if [[ -n "$shim_maker" ]]; then
SHIM=$(mktemp -d); "$shim_maker" "$SHIM" "$H"; pathpre="$SHIM:$ORIG_PATH"
fi
set +e
env PATH="$pathpre" XDG_STATE_HOME="$STATE" \
${retention:+MOSAIC_BACKUP_RETENTION="$retention"} \
MOSAIC_HOME="$H" MOSAIC_INSTALL_MODE=keep MOSAIC_SYNC_ONLY=1 \
bash "$installer" >"$OUT" 2>&1
rc=$?
set -e 2>/dev/null || true
[[ -n "$shim_maker" ]] && rm -rf "$SHIM"
printf '%s\t%s\t%s\t%s\n' "$rc" "$OUT" "$STATE" "$H"
}
# Resolve the single pre-update-* snapshot dir under a state dir (newest if many).
snap_dir() {
find "$1/mosaic/backups" -maxdepth 1 -type d -name 'pre-update-*' 2>/dev/null \
| LC_ALL=C sort -r | head -1
}
echo "── Part 1/2/3: durable snapshot scope, perms, no-leak ──────────────────"
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$INSTALL")
SNAP="$(snap_dir "$STATE")"
chk "upgrade succeeds" "[ '$rc' -eq 0 ]"
chk "exactly one pre-update snapshot created" "[ \$(find '$STATE/mosaic/backups' -maxdepth 1 -type d -name 'pre-update-*' | wc -l) -eq 1 ]"
chk "snapshot: SOUL.md captured" "[ -f '$SNAP/SOUL.md' ]"
chk "snapshot: operator subtree captured" "[ -f '$SNAP/agents/coder0.conf' ]"
chk "snapshot: memory captured" "[ -f '$SNAP/memory/note.md' ]"
chk "snapshot: credentials carve-out captured" "[ -f '$SNAP/tools/_lib/credentials.json' ]"
chk "snapshot: SOUL.md bytes preserved" "[ \"\$(cat '$SNAP/SOUL.md')\" = '$SOUL_ORIG' ]"
chk "snapshot: framework file NOT captured" "[ ! -e '$SNAP/CONSTITUTION.md' ] && [ ! -e '$SNAP/$POISON_REL' ]"
# Part 2 — permissions (0700 dirs, 0600 files); never world-readable.
chk "perms: backup root is 0700" "[ \$(stat -c '%a' '$STATE/mosaic/backups') -eq 700 ]"
chk "perms: snapshot dir is 0700" "[ \$(stat -c '%a' '$SNAP') -eq 700 ]"
chk "perms: nested dir is 0700" "[ \$(stat -c '%a' '$SNAP/agents') -eq 700 ]"
chk "perms: credentials backup is 0600" "[ \$(stat -c '%a' '$SNAP/tools/_lib/credentials.json') -eq 600 ]"
chk "perms: SOUL.md backup is 0600" "[ \$(stat -c '%a' '$SNAP/SOUL.md') -eq 600 ]"
# Part 3 — the secret is backed up but never emitted to stdout/stderr.
chk "no-leak: secret IS in the backup file" "grep -q '$SECRET' '$SNAP/tools/_lib/credentials.json'"
chk "no-leak: secret NOT on stdout/stderr" "! grep -q '$SECRET' '$OUT'"
rm -rf "$STATE" "$H"; rm -f "$OUT"
echo "── Part 4: retention prune (MOSAIC_BACKUP_RETENTION) ───────────────────"
# Pre-seed four dated snapshots, then take one real snapshot with retention=2:
# only the two newest (the fresh real one + the newest pre-seeded) must survive.
IFS=$'\t' read -r rc OUT STATE H < <(
H=$(mktemp -d); STATE=$(mktemp -d); OUT=$(mktemp)
seed_home "$H"
mkdir -p "$STATE/mosaic/backups"
for ts in 20200101T000000Z 20210101T000000Z 20220101T000000Z 20230101T000000Z; do
mkdir -p "$STATE/mosaic/backups/pre-update-$ts"
done
set +e
env PATH="$ORIG_PATH" XDG_STATE_HOME="$STATE" MOSAIC_BACKUP_RETENTION=2 \
MOSAIC_HOME="$H" MOSAIC_INSTALL_MODE=keep MOSAIC_SYNC_ONLY=1 \
bash "$INSTALL" >"$OUT" 2>&1
rc=$?
set -e 2>/dev/null || true
printf '%s\t%s\t%s\t%s\n' "$rc" "$OUT" "$STATE" "$H"
)
chk "retention: upgrade succeeds" "[ '$rc' -eq 0 ]"
chk "retention: pruned to exactly 2 snapshots" "[ \$(find '$STATE/mosaic/backups' -maxdepth 1 -type d -name 'pre-update-*' | wc -l) -eq 2 ]"
chk "retention: newest pre-seeded survives" "[ -d '$STATE/mosaic/backups/pre-update-20230101T000000Z' ]"
chk "retention: oldest pre-seeded pruned" "[ ! -d '$STATE/mosaic/backups/pre-update-20200101T000000Z' ]"
rm -rf "$STATE" "$H"; rm -f "$OUT"
echo "── Part 5: post-sync verify restores an operator file (+ control) ──────"
# Shipped installer: the cp shim corrupts SOUL.md mid-sync; verify must restore it.
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$INSTALL" make_corrupt_shim)
chk "verify: upgrade still succeeds" "[ '$rc' -eq 0 ]"
chk "verify: SOUL.md restored to original" "[ \"\$(cat '$H/SOUL.md')\" = '$SOUL_ORIG' ]"
chk "verify: no corruption remains in SOUL.md" "! grep -q 'CORRUPTION-mid-sync' '$H/SOUL.md'"
chk "verify: loud restore warning emitted" "grep -qi 'restored from the pre-update snapshot' '$OUT'"
chk "verify: secret still not leaked" "! grep -q '$SECRET' '$OUT'"
rm -rf "$STATE" "$H"; rm -f "$OUT"
# Control: strip the verify call → the corruption must SURVIVE (net is load-bearing).
sed '/# VERIFY-NET/d' "$INSTALL" > "$VERIFYCTRL"
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$VERIFYCTRL" make_corrupt_shim)
chk "control: SOUL.md corruption survives" "grep -q 'CORRUPTION-mid-sync' '$H/SOUL.md'"
chk "control: no restore warning emitted" "! grep -qi 'restored from the pre-update snapshot' '$OUT'"
rm -rf "$STATE" "$H"; rm -f "$OUT"
echo "── Part 6: verify net honors an intentional migration removal (+ control) ─"
# BLOCKER regression: on a pre-v2 install, bin/ is operator-classified so the durable
# snapshot captures it — but the v1→v2 migration deletes bin/ ON PURPOSE. The verify
# net must SKIP that removal (is_migration_removed), or it heals bin/ back and the
# migration is silently undone forever once the version is stamped.
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$INSTALL" "" "" seed_home_v1)
chk "migration: upgrade succeeds" "[ '$rc' -eq 0 ]"
chk "migration: legacy bin/ stays removed" "[ ! -e '$H/bin' ]"
chk "migration: operator SOUL.md untouched" "[ \"\$(cat '$H/SOUL.md')\" = '$SOUL_ORIG' ]"
chk "migration: version stamped to $FRAMEWORK_VERSION" "[ \"\$(cat '$H/.framework-version')\" = '$FRAMEWORK_VERSION' ]"
rm -rf "$STATE" "$H"; rm -f "$OUT"
# Control: strip the MIGRATION-SKIP-GUARD → the verify net restores bin/ from the
# snapshot, silently undoing the migration (proves the guard is load-bearing).
MIGCTRL="$(mk_control 'MIGRATION-SKIP-GUARD' migration-control)"
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$MIGCTRL" "" "" seed_home_v1)
chk "control: bin/ wrongly restored by verify" "[ -e '$H/bin/tool.sh' ]"
rm -rf "$STATE" "$H"; rm -f "$OUT"
echo "── Part 7: verify net never restores a secret through a symlink (+ control) ─"
# HIGH (CWE-59) regression: an attacker who swaps an operator file for a symlink
# AFTER the durable snapshot must not cause the verify net's restore to write the
# snapshot's secret out THROUGH that link. The shipped net drops a symlinked leaf and
# writes a real file in its place, leaving the external target untouched.
EXFIL_DIR=$(mktemp -d); EXFIL_TARGET="$EXFIL_DIR/stolen"
printf 'ATTACKER-PLACEHOLDER\n' > "$EXFIL_TARGET"
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$INSTALL" make_symlink_leaf_shim)
chk "symlink-leaf: upgrade succeeds" "[ '$rc' -eq 0 ]"
chk "symlink-leaf: secret NOT written through link" "! grep -q '$SECRET' '$EXFIL_TARGET'"
chk "symlink-leaf: credentials.json is a real file" "[ -f '$H/tools/_lib/credentials.json' ] && [ ! -L '$H/tools/_lib/credentials.json' ]"
chk "symlink-leaf: credentials.json restored intact" "grep -q '$SECRET' '$H/tools/_lib/credentials.json'"
chk "symlink-leaf: secret not leaked to stdout/stderr" "! grep -q '$SECRET' '$OUT'"
rm -rf "$STATE" "$H" "$EXFIL_DIR"; rm -f "$OUT"
# Control: strip the SYMLINK-LEAF-GUARD → cp follows the swapped-in link and writes
# the snapshot secret out through it (proves the guard is load-bearing).
EXFIL_DIR=$(mktemp -d); EXFIL_TARGET="$EXFIL_DIR/stolen"
printf 'ATTACKER-PLACEHOLDER\n' > "$EXFIL_TARGET"
LEAFCTRL="$(mk_control 'SYMLINK-LEAF-GUARD' symlinkleaf-control)"
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$LEAFCTRL" make_symlink_leaf_shim)
chk "control: secret leaked through the symlink" "grep -q '$SECRET' '$EXFIL_TARGET'"
rm -rf "$STATE" "$H" "$EXFIL_DIR"; rm -f "$OUT"
echo "── Part 8: snapshot umask 077 does not leak into synced files (+ control) ──"
# SHOULD-FIX regression: umask 077 is process-global. Scoped to the snapshot it keeps
# backups 0600; leaked past it, every later cp/mkdir inherits 0600/0700. A freshly-
# synced framework file must be 0644 (per the ambient 022 umask) while the backup of
# a secret stays 0600.
IFS=$'\t' read -r rc OUT STATE H < <(umask 022; run_snap "$INSTALL")
SNAP="$(snap_dir "$STATE")"
chk "umask: upgrade succeeds" "[ '$rc' -eq 0 ]"
chk "umask: synced framework file is 0644" "[ \$(stat -c '%a' '$H/$POISON_REL') -eq 644 ]"
chk "umask: backup of a secret stays 0600" "[ \$(stat -c '%a' '$SNAP/tools/_lib/credentials.json') -eq 600 ]"
rm -rf "$STATE" "$H"; rm -f "$OUT"
# Control: strip the UMASK-RESTORE-NORMAL line → umask 077 leaks past the snapshot,
# so the newly-synced framework file is created 0600 (proves the restore matters).
UMASKCTRL="$(mk_control 'UMASK-RESTORE-NORMAL' umask-control)"
IFS=$'\t' read -r rc OUT STATE H < <(umask 022; run_snap "$UMASKCTRL")
chk "control: leaked umask makes synced file 0600" "[ \$(stat -c '%a' '$H/$POISON_REL') -eq 600 ]"
rm -rf "$STATE" "$H"; rm -f "$OUT"
echo ""
echo "RESULT: $pass passed, $fail failed"
[ "$fail" -eq 0 ]

View File

@@ -29,6 +29,13 @@ INSTALL="$FW/install.sh"
pass=0; fail=0 pass=0; fail=0
chk() { if eval "$2"; then echo "$1"; pass=$((pass + 1)); else echo "$1"; fail=$((fail + 1)); fi; } chk() { if eval "$2"; then echo "$1"; pass=$((pass + 1)); else echo "$1"; fail=$((fail + 1)); fi; }
# Redirect the #791 PR2 durable pre-update snapshot ($XDG_STATE_HOME/mosaic/backups)
# into a throwaway so a keep-mode upgrade under test never writes into the real
# ~/.local/state. This test asserts operator-surface fidelity, not backup content.
export XDG_STATE_HOME
XDG_STATE_HOME="$(mktemp -d)"
trap 'rm -rf "$XDG_STATE_HOME"' EXIT
SECRET='SUPER-SECRET-TOKEN-do-not-log-3f9a' SECRET='SUPER-SECRET-TOKEN-do-not-log-3f9a'
# Seed a throwaway MOSAIC_HOME with an operator sentinel per ownership class. # Seed a throwaway MOSAIC_HOME with an operator sentinel per ownership class.
@@ -206,7 +213,7 @@ fi
# does not silently depend on rsync being installed. (Provide the coreutils the # does not silently depend on rsync being installed. (Provide the coreutils the
# installer needs on the stripped PATH.) # installer needs on the stripped PATH.)
FBIN=$(mktemp -d) FBIN=$(mktemp -d)
for t in bash cp find mktemp rm mkdir chmod cmp sed grep cat dirname basename stat sha256sum awk tr; do for t in bash cp find mktemp rm mkdir chmod cmp sed grep cat dirname basename stat sha256sum awk tr date sort; do
p=$(command -v "$t" 2>/dev/null) && ln -s "$p" "$FBIN/$t" p=$(command -v "$t" 2>/dev/null) && ln -s "$p" "$FBIN/$t"
done done
run_matrix "rsync-absent" env "PATH=$FBIN" run_matrix "rsync-absent" env "PATH=$FBIN"

View File

@@ -17,6 +17,7 @@ import { registerConfigCommand } from './commands/config.js';
import { registerFleetCommand } from './commands/fleet.js'; import { registerFleetCommand } from './commands/fleet.js';
import { registerMissionCommand } from './commands/mission.js'; import { registerMissionCommand } from './commands/mission.js';
import { registerUninstallCommand } from './commands/uninstall.js'; import { registerUninstallCommand } from './commands/uninstall.js';
import { registerRestoreCommand } from './commands/restore.js';
// prdy is registered via launch.ts // prdy is registered via launch.ts
import { registerLaunchCommands } from './commands/launch.js'; import { registerLaunchCommands } from './commands/launch.js';
import { registerAuthCommand } from './commands/auth.js'; import { registerAuthCommand } from './commands/auth.js';
@@ -406,6 +407,10 @@ registerStorageCommand(program);
registerUninstallCommand(program); registerUninstallCommand(program);
// ─── restore ─────────────────────────────────────────────────────────────────
registerRestoreCommand(program);
// ─── telemetry ─────────────────────────────────────────────────────────────── // ─── telemetry ───────────────────────────────────────────────────────────────
registerTelemetryCommand(program); registerTelemetryCommand(program);

View File

@@ -0,0 +1,885 @@
import { chmod, mkdir, mkdtemp, open, readFile, rm, stat, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { Command } from 'commander';
import { afterEach, describe, expect, it, vi } from 'vitest';
import { registerFleetCommand, type CommandResult, type FleetCommandDeps } from './fleet.js';
import { executeFleetRegen, formatFleetRegenReport } from './fleet-regen-command.js';
import {
acquirePrivateRosterMutationLock,
projectRosterV2AgentGeneratedEnv,
} from '../fleet/fleet-reconciler.js';
import { applyPreparedGeneratedAgentEnvironmentProjection } from '../fleet/generated-env-boundary.js';
import { parseRosterV2 } from '../fleet/roster-v2.js';
// A two-agent roster-v2 SSOT. `mosaic fleet regen` must rebuild each agent's
// `fleet/agents/<name>.env.generated` projection from exactly this source and
// nothing else — deterministically, without ever touching agent lifecycle.
const rosterYaml = `
version: 2
generation: 7
transport: tmux
tmux:
socket_name: mosaic-fleet
holder_session: _holder
defaults:
working_directory: /srv/mosaic
runtime: pi
runtimes:
pi:
reset_command: /new
agents:
- name: coder0
alias: Coder 0
class: code
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: high
tool_policy: code
working_directory: /srv/mosaic
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
- name: coder1
alias: Coder 1
class: code
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: medium
tool_policy: code
working_directory: /srv/other
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
`;
let cleanup: string | undefined;
afterEach(async (): Promise<void> => {
vi.restoreAllMocks();
process.exitCode = undefined;
if (cleanup) await rm(cleanup, { recursive: true, force: true });
cleanup = undefined;
});
async function fleetHome(withRoster = rosterYaml): Promise<string> {
cleanup = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-command-'));
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
await mkdir(join(cleanup, directory), { recursive: true, mode: 0o700 });
await chmod(join(cleanup, directory), 0o700);
}
await chmod(cleanup, 0o700);
await writeFile(join(cleanup, 'fleet', 'roster.yaml'), withRoster, { mode: 0o600 });
// Semantic roster validation (matching `fleet reconcile`) resolves each agent
// class to a persona; seed the classes the fixtures reference.
for (const klass of ['code', 'merge-gate']) {
await writeFile(
join(cleanup, 'fleet', 'roles', `${klass}.md`),
`\`class: ${klass}\`\n\n# ${klass} persona\n`,
{ mode: 0o600 },
);
}
return cleanup;
}
/**
* A runner spy that records EVERY invocation. `regen` is projection-only and
* must never issue a lifecycle/restart call, so a non-empty call log is a
* hard failure — this is the load-bearing "never restarts" gate.
*/
function recordingRunner(
calls: string[][],
): (command: string, args: string[]) => Promise<CommandResult> {
return async (command: string, args: string[]): Promise<CommandResult> => {
calls.push([command, ...args]);
return { stdout: '', stderr: '', exitCode: 0 };
};
}
function program(mosaicHome: string, runner: FleetCommandDeps['runner']): Command {
const result = new Command();
result.exitOverride();
registerFleetCommand(result, { mosaicHome, runner });
return result;
}
function capture(): string[] {
const lines: string[] = [];
vi.spyOn(console, 'log').mockImplementation((value: string): void => {
lines.push(value);
});
return lines;
}
async function exists(path: string): Promise<boolean> {
try {
await stat(path);
return true;
} catch {
return false;
}
}
describe('projectRosterV2AgentGeneratedEnv', (): void => {
it('maps a roster-v2 agent to exactly the eight generated projection keys', (): void => {
const roster = parseRosterV2(rosterYaml, 'yaml');
const agent = roster.agents.find((candidate) => candidate.name === 'coder0');
expect(agent).toBeDefined();
const values = projectRosterV2AgentGeneratedEnv(roster, agent!);
expect(values).toEqual({
MOSAIC_AGENT_NAME: 'coder0',
MOSAIC_AGENT_CLASS: 'code',
MOSAIC_AGENT_RUNTIME: 'pi',
MOSAIC_AGENT_MODEL: 'gpt-5.6-sol',
MOSAIC_AGENT_REASONING: 'high',
MOSAIC_AGENT_TOOL_POLICY: 'code',
MOSAIC_AGENT_WORKDIR: '/srv/mosaic',
MOSAIC_TMUX_SOCKET: 'mosaic-fleet',
});
});
});
describe('mosaic fleet regen', (): void => {
it('is dry-run by default: reports the plan and writes nothing', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const lines = capture();
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--json',
]);
const result = JSON.parse(lines.pop() ?? '{}');
expect(result).toMatchObject({
mode: 'dry-run',
generation: 7,
agentCount: 2,
written: 0,
});
expect(result.agents.map((agent: { name: string }) => agent.name)).toEqual([
'coder0',
'coder1',
]);
expect(
result.agents.every((agent: { disposition: string }) => agent.disposition === 'create'),
).toBe(true);
// Nothing on disk.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
// No lifecycle/restart call — ever.
expect(calls).toEqual([]);
});
it('forwards configured persona roots (rolesDir/overrideDir) from reconcileDeps into regen', async (): Promise<void> => {
// Codex r6: regen must resolve personas the SAME way reconcile does. If the
// top-level registration drops reconcileDeps.rolesDir/overrideDir, a deployment
// with custom persona roots gets reconcile ACCEPTING a roster while regen
// REJECTS it (validating against the wrong default `<home>/fleet/roles`),
// blocking the recovery command. Pin the wiring: seed personas ONLY under the
// custom roots, leave the default roles dir empty, require regen to succeed.
const home = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-wiring-'));
cleanup = home;
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
await mkdir(join(home, directory), { recursive: true, mode: 0o700 });
await chmod(join(home, directory), 0o700);
}
await chmod(home, 0o700);
await writeFile(join(home, 'fleet', 'roster.yaml'), rosterYaml, { mode: 0o600 });
// Personas live ONLY under the configured roots — the default `fleet/roles`
// stays empty, so broken wiring fails persona resolution.
const customRoles = join(home, 'custom-roles');
const customOverride = join(home, 'custom-roles.local');
await mkdir(customRoles, { recursive: true, mode: 0o700 });
await mkdir(customOverride, { recursive: true, mode: 0o700 });
for (const klass of ['code', 'merge-gate']) {
await writeFile(join(customRoles, `${klass}.md`), `\`class: ${klass}\`\n\n# ${klass}\n`, {
mode: 0o600,
});
}
const command = new Command();
command.exitOverride();
registerFleetCommand(command, {
mosaicHome: home,
runner: recordingRunner([]),
reconcileDeps: { rolesDir: customRoles, overrideDir: customOverride },
});
const lines = capture();
await command.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
// Regen validated against the CONFIGURED roots → success. Broken wiring
// resolves against the empty default and fails (exitCode 1, no JSON result).
expect(process.exitCode).not.toBe(1);
const result = JSON.parse(lines.pop() ?? '{}');
expect(result).toMatchObject({ mode: 'dry-run', agentCount: 2 });
});
it('--write rebuilds each generated projection from the roster SSOT', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const lines = capture();
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
const result = JSON.parse(lines.pop() ?? '{}');
expect(result).toMatchObject({ mode: 'write', written: 2, agentCount: 2 });
const coder0 = await readFile(join(home, 'fleet', 'agents', 'coder0.env.generated'), 'utf8');
expect(coder0).toContain('MOSAIC_AGENT_NAME=coder0');
expect(coder0).toContain('MOSAIC_AGENT_WORKDIR=/srv/mosaic');
expect(coder0).toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet');
const coder1 = await readFile(join(home, 'fleet', 'agents', 'coder1.env.generated'), 'utf8');
expect(coder1).toContain('MOSAIC_AGENT_NAME=coder1');
expect(coder1).toContain('MOSAIC_AGENT_WORKDIR=/srv/other');
// No lifecycle/restart call — ever.
expect(calls).toEqual([]);
});
it('is deterministic and idempotent across repeated --write runs', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const path = join(home, 'fleet', 'agents', 'coder0.env.generated');
const cli = program(home, recordingRunner(calls));
capture();
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
const first = await readFile(path, 'utf8');
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
const second = await readFile(path, 'utf8');
expect(second).toBe(first);
expect(calls).toEqual([]);
});
it('reports rebuild disposition once the generated projection already exists', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const cli = program(home, recordingRunner(calls));
const lines = capture();
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
lines.length = 0;
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
const result = JSON.parse(lines.pop() ?? '{}');
expect(
result.agents.every((agent: { disposition: string }) => agent.disposition === 'rebuild'),
).toBe(true);
expect(result.written).toBe(0);
expect(calls).toEqual([]);
});
it('never issues a lifecycle/restart call in either mode', async (): Promise<void> => {
const home = await fleetHome();
const calls: string[][] = [];
const cli = program(home, recordingRunner(calls));
capture();
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
// Structural guarantee: regen has no path to systemctl/tmux at all.
expect(calls).toEqual([]);
const systemctlCalls = calls.filter(([command]) => command === 'systemctl');
expect(systemctlCalls).toEqual([]);
});
it('human-readable --write output prints the do-not-restart recovery runbook', async (): Promise<void> => {
const home = await fleetHome();
const lines = capture();
await program(home, recordingRunner([])).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
]);
const output = lines.join('\n');
expect(output).toMatch(/do not restart|before.*restart/i);
expect(output).toMatch(/env\.generated/);
// The unit has no EnvironmentFile= directive, so the runbook must NOT tell the
// operator to verify one — verify the launcher/generated file instead.
expect(output).not.toContain('EnvironmentFile');
expect(output).toMatch(/systemctl --user cat mosaic-agent@<name>/);
expect(output).toMatch(/restart/i);
});
it('emits paths and counts only — never the projected env body (secrev)', async (): Promise<void> => {
const home = await fleetHome();
const lines = capture();
await program(home, recordingRunner([])).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
]);
const output = lines.join('\n');
// Relative paths + counts are fine; the rendered projection body (KEY=value
// lines) must never be echoed to stdout.
expect(output).toContain('coder0.env.generated');
expect(output).not.toMatch(/^MOSAIC_AGENT_\w+=/m);
expect(output).not.toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet');
});
it('is projection-only: leaves legacy .env untouched and writes no .env.local/.env.quarantine', async (): Promise<void> => {
// Recovery contract: regen rebuilds ONLY <name>.env.generated. It must never
// relocate, quarantine, or unlink the operator-owned legacy .env surface — the
// full reconciler apply path does, so regen must NOT use it.
const home = await fleetHome();
const legacyPath = join(home, 'fleet', 'agents', 'coder0.env');
await writeFile(legacyPath, 'MOSAIC_RUNTIME_BIN=/usr/bin/pi\n', { mode: 0o600 });
capture();
await program(home, recordingRunner([])).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
// Generated projection rebuilt...
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
// ...but the operator's legacy .env is preserved verbatim, and no local/quarantine
// files were fabricated from it.
expect(await readFile(legacyPath, 'utf8')).toBe('MOSAIC_RUNTIME_BIN=/usr/bin/pi\n');
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.local'))).toBe(false);
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.quarantine'))).toBe(false);
});
it('prepares every agent before writing any: a later prepare failure leaves nothing written', async (): Promise<void> => {
// Fail-closed across agents. Pre-seed coder1's projection with world-readable
// perms so its prepare rejects; coder0 (valid) must NOT be written because
// preparation is fully completed before the first apply.
const home = await fleetHome();
const coder1Path = join(home, 'fleet', 'agents', 'coder1.env.generated');
await writeFile(coder1Path, 'MOSAIC_AGENT_NAME=coder1\n', { mode: 0o644 });
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
// coder0 is valid but must remain unwritten — no partial rebuild.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('fails closed on a semantically invalid roster (protected-class tool-policy mismatch)', async (): Promise<void> => {
// A hand-edited roster giving a protected class a weaker tool_policy is rejected
// by reconcile/plan/verify; regen must enforce the SAME gate, not silently
// project the downgraded policy into <name>.env.generated.
const home = await fleetHome(`
version: 2
generation: 3
transport: tmux
tmux:
socket_name: mosaic-fleet
holder_session: _holder
defaults:
working_directory: /srv/mosaic
runtime: pi
runtimes:
pi:
reset_command: /new
agents:
- name: gate0
alias: Gate 0
class: merge-gate
runtime: pi
provider: openai
model: gpt-5.6-sol
reasoning: high
tool_policy: code
working_directory: /srv/mosaic
persistent_persona: false
reset_between_tasks: true
lifecycle:
enabled: true
desired_state: stopped
launch:
yolo: true
`);
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
expect(await exists(join(home, 'fleet', 'agents', 'gate0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('refuses to write while a concurrent reconcile holds the mutation lock', async (): Promise<void> => {
// regen --write mutates the same projections as reconcile; it must take the
// shared reconcile lock so a concurrent reconcile cannot race a stale write.
const home = await fleetHome();
await writeFile(join(home, 'fleet', 'roster.yaml.reconcile.lock'), 'held\n', { mode: 0o600 });
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('fails closed (non-zero, no writes) on an invalid roster', async (): Promise<void> => {
// roster-v2 requires >= 1 agent; a malformed roster must abort regen without
// writing any projection and without touching lifecycle.
const home = await fleetHome('version: 2\ngeneration: 1\n');
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('rejects a non-canonical --roster path (canonical roster only)', async (): Promise<void> => {
// `fleet` exposes a global `--roster`; reconcile rejects a non-canonical value
// rather than silently reading the canonical roster. regen must enforce the
// SAME guard so `--roster /elsewhere regen --write` cannot mislead the operator.
const home = await fleetHome();
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'--roster',
join(home, 'other', 'roster.yaml'),
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
expect(errors.join('')).toMatch(/canonical roster/i);
// No projection written against a misdirected roster path.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('reports an incomplete rebuild when a later projection write fails mid-loop', async (): Promise<void> => {
// Fault injected on the SECOND apply: coder0 is already replaced on disk, so a
// bare throw would hide the partial state. regen must surface which agents were
// rebuilt and which failed, with a verify-before-restart recovery instruction.
const home = await fleetHome();
const realApply = applyPreparedGeneratedAgentEnvironmentProjection;
const result = await executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
applyProjection: async (prepared): Promise<string> => {
if (prepared.generatedPath.endsWith('coder1.env.generated')) {
throw new Error('simulated ENOSPC on coder1');
}
return realApply(prepared);
},
},
{ write: true },
);
// coder0 written before the fault; coder1 not.
expect(result.written).toBe(1);
expect(result.incomplete).toMatchObject({
code: 'projection-apply-failed',
failedAgent: 'coder1',
writtenAgents: ['coder0'],
});
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
// The human report names the failed agent and directs a verify-before-restart
// recovery — not a bare success runbook.
const report = formatFleetRegenReport(result).join('\n');
expect(report).toMatch(/incomplete/i);
expect(report).toContain('coder1');
expect(report).toMatch(/verify|re-run|retry/i);
// Structural: still no lifecycle call anywhere on the failure path.
expect(report).not.toMatch(/systemctl --user restart mosaic-agent@<name>\n.*\n/);
});
it('refuses to write while agent CRUD holds the roster mutation lock', async (): Promise<void> => {
// regen --write overwrites the SAME generated projections that `fleet agent
// create/update/delete` writes under roster.yaml.mutation.lock. If regen only
// took the reconcile lock it could read a stale roster and overwrite a just-
// committed projection. It must also take the mutation lock and fail closed.
const home = await fleetHome();
await writeFile(join(home, 'fleet', 'roster.yaml.mutation.lock'), 'crud\n', { mode: 0o600 });
const calls: string[][] = [];
capture();
const errors: string[] = [];
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
errors.push(String(chunk));
return true;
});
await program(home, recordingRunner(calls)).parseAsync([
'node',
'mosaic',
'fleet',
'regen',
'--write',
'--json',
]);
expect(process.exitCode).toBe(1);
expect(errors.join('')).toMatch(/regen failed/i);
// No projection written against a roster a concurrent CRUD is mutating.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
expect(calls).toEqual([]);
});
it('surfaces BOTH the roster failure and a stale-lock warning on a double fault', async (): Promise<void> => {
// Worst case with nothing written: run() fails (invalid roster, fail-closed) AND
// a lock release faults. The operator must be told the lock may be stale, not
// just the roster error — otherwise the next regen/reconcile is silently blocked.
const home = await fleetHome('version: 2\ngeneration: 1\n');
const throwingRelease =
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated lock unlink fault');
};
await expect(
executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
acquireReconcileLock: throwingRelease,
acquireRosterMutationLock: throwingRelease,
},
{ write: true },
),
).rejects.toThrow(/stale|inspect|lock/i);
// Nothing written on the fail-closed path.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
});
it('propagates a roster-mutation-lock release failure instead of silently keeping a stale lock', async (): Promise<void> => {
// Finding L: the real `acquirePrivateRosterMutationLock` release must SURFACE an
// unlink failure, not swallow it. If it swallowed (like the CRUD-internal
// acquireMutationLock does), a stale roster.yaml.mutation.lock left after a
// successful regen would block later regen/agent CRUD while the command reported
// success — and the finding-J stale-lock warning would never fire for this lock.
const home = await fleetHome();
const release = await acquirePrivateRosterMutationLock(home)();
// Simulate the lock vanishing (or being unremovable) before release runs: the
// underlying unlink then faults. A swallowing release would resolve and hide it.
await rm(join(home, 'fleet', 'roster.yaml.mutation.lock'));
await expect(release()).rejects.toThrow();
});
it('preserves a complete rebuild result and flags lock-cleanup when release faults', async (): Promise<void> => {
// A lock-release fault after a successful write must NOT discard the fact that
// projections are now live — the operator needs to know the write happened and
// that the shared lock may be stale, not just a bare "regen failed".
const home = await fleetHome();
const result = await executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
acquireReconcileLock:
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated lock unlink fault');
},
},
{ write: true },
);
expect(result.written).toBe(2);
expect(result.incomplete).toBeUndefined();
expect(result.cleanup).toMatchObject({
code: 'lock-cleanup-failed',
action: 'inspect-lock-before-retry',
});
// Projections are genuinely on disk despite the release fault.
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(true);
const report = formatFleetRegenReport(result).join('\n');
expect(report).toMatch(/lock/i);
expect(report).toMatch(/stale|inspect|clear/i);
});
it('preserves a partial rebuild AND flags lock-cleanup when both apply and release fault', async (): Promise<void> => {
// Worst case: a mid-loop apply fault leaves a partial rebuild, then the release
// also faults. Both recovery states must survive so the operator sees exactly
// which projections are live and that the lock may be stale.
const home = await fleetHome();
const realApply = applyPreparedGeneratedAgentEnvironmentProjection;
const result = await executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
applyProjection: async (prepared): Promise<string> => {
if (prepared.generatedPath.endsWith('coder1.env.generated')) {
throw new Error('simulated ENOSPC on coder1');
}
return realApply(prepared);
},
acquireReconcileLock:
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated lock unlink fault');
},
},
{ write: true },
);
expect(result.written).toBe(1);
expect(result.incomplete).toMatchObject({
code: 'projection-apply-failed',
failedAgent: 'coder1',
writtenAgents: ['coder0'],
});
expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' });
const report = formatFleetRegenReport(result).join('\n');
expect(report).toMatch(/incomplete/i);
expect(report).toMatch(/lock/i);
});
it('fails closed without deleting a REPLACEMENT roster-mutation lock it no longer owns', async (): Promise<void> => {
// Finding M1 (replacement-lock race): after this invocation created its lock, the
// lock is cleared and re-created by ANOTHER writer (new inode + foreign token)
// BEFORE release runs. The release must PROVE ownership (device/inode + token) and
// refuse to unlink the stranger's live lock — otherwise it deletes it, a third
// writer enters concurrently, and mutual exclusion over the projections is broken.
// This pins the REACHABLE window (replaced-before-release); the residual sub-window
// between the final check and the path `unlink` is unreachable within the `wx`
// writer protocol and matches the merged reconcile lock — see the acquirer doc.
const home = await fleetHome();
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
const release = await acquirePrivateRosterMutationLock(home)();
// A different writer replaces the lock: same path, NEW inode, foreign content.
await rm(lockPath);
await writeFile(lockPath, 'a-different-writer\n', { mode: 0o600 });
const replacement = await stat(lockPath);
// Release must reject (it no longer owns this file) AND the diagnosis must name
// the MUTATION lock specifically — a fault on this shared helper must not be
// mislabeled as the reconcile lock (that would defeat the M3 accurate-diagnosis
// goal, since regen serializes on both) ...
await expect(release()).rejects.toThrow(/roster\.yaml\.mutation\.lock/);
// ... and MUST NOT have unlinked the stranger's live lock.
expect(await exists(lockPath)).toBe(true);
expect((await stat(lockPath)).ino).toBe(replacement.ino);
});
it('does not strand the lock file when initialization fails after creating it', async (): Promise<void> => {
// Codex r4: if writeFile/stat/close/ownership-check throws AFTER the `wx` create
// succeeds, the just-created lock must NOT be left behind — a stranded lock would
// permanently block future regen AND agent CRUD (both contend on this path). The
// cleanup is dev/ino-guarded (never deletes a replacement) and best-effort (never
// masks the initialization error).
const home = await fleetHome();
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
// Inject an opener that performs the REAL `wx` create (so the lock file really
// lands on disk) but whose handle faults on the token write — a post-create
// initialization failure.
const faultingOpen = (async (
path: Parameters<typeof open>[0],
flags: Parameters<typeof open>[1],
mode: Parameters<typeof open>[2],
) => {
const handle = await open(path, flags, mode);
return new Proxy(handle, {
get(target, prop, receiver): unknown {
if (prop === 'writeFile') {
return async (): Promise<never> => {
throw new Error('simulated ENOSPC on token write');
};
}
const value = Reflect.get(target, prop, receiver);
return typeof value === 'function' ? value.bind(target) : value;
},
});
}) as typeof open;
await expect(acquirePrivateRosterMutationLock(home, faultingOpen)()).rejects.toThrow(
/mutation\.lock/,
);
// The lock we created must have been cleaned up — not stranded on disk.
expect(await exists(lockPath)).toBe(false);
});
it('does not strand the lock file when the post-create stat itself fails', async (): Promise<void> => {
// Codex r5: the narrower sub-path where `handle.stat()` ITSELF rejects right after
// the `wx` create succeeds (transient EIO/EBADF). device/inode identity is then
// unavailable, so the init-failure cleanup cannot prove ownership by dev/ino — yet
// the lock must STILL not be stranded, because a stranded lock permanently blocks
// future regen AND agent CRUD. The token we persisted before the failure uniquely
// identifies OUR lock, so cleanup falls back to matching it.
const home = await fleetHome();
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
// Real `wx` create (lock really lands on disk); the token write SUCCEEDS so our
// random token is persisted, then `stat()` faults — a post-create init failure.
const faultingStatOpen = (async (
path: Parameters<typeof open>[0],
flags: Parameters<typeof open>[1],
mode: Parameters<typeof open>[2],
) => {
const handle = await open(path, flags, mode);
return new Proxy(handle, {
get(target, prop, receiver): unknown {
if (prop === 'stat') {
return async (): Promise<never> => {
throw new Error('simulated EIO on post-create stat');
};
}
const value = Reflect.get(target, prop, receiver);
return typeof value === 'function' ? value.bind(target) : value;
},
});
}) as typeof open;
await expect(acquirePrivateRosterMutationLock(home, faultingStatOpen)()).rejects.toThrow(
/mutation\.lock/,
);
// Even without dev/ino, the token-based fallback must have removed our lock.
expect(await exists(lockPath)).toBe(false);
});
it('surfaces a lock-cleanup fault when unwinding after a later lock acquire fails', async (): Promise<void> => {
// Finding M2: the FIRST lock is taken, the SECOND lock's acquire throws, and
// unwinding the first lock's release ALSO faults. The acquire-unwind path must
// surface both — the already-taken lock may now be stale and silently block the
// next regen/reconcile — not just re-throw the acquire error and drop the fault.
const home = await fleetHome();
await expect(
executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
// Mutation lock (acquired first) succeeds, but its RELEASE faults on unwind.
acquireRosterMutationLock:
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated mutation-lock unlink fault');
},
// Reconcile lock (acquired second) ACQUIRE fails, triggering the unwind.
acquireReconcileLock: () => async (): Promise<() => Promise<void>> => {
throw new Error('simulated reconcile acquire failure');
},
},
{ write: true },
),
).rejects.toThrow(/stale|inspect|lock/i);
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
});
it('names BOTH fleet locks in the stale-lock warning (cleanup can come from either)', async (): Promise<void> => {
// Finding M3: finding L made the roster MUTATION lock's release fault reachable, so
// the cleanup marker can now originate from either lock. The human report must not
// claim only the reconcile lock is stale — it must name both candidate lock files
// so the operator inspects the right one.
const home = await fleetHome();
const result = await executeFleetRegen(
{
runner: recordingRunner([]),
mosaicHome: home,
// Fault the MUTATION lock's release specifically (reconcile lock is real).
acquireRosterMutationLock:
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
throw new Error('simulated mutation-lock unlink fault');
},
},
{ write: true },
);
expect(result.written).toBe(2);
expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' });
const report = formatFleetRegenReport(result).join('\n');
expect(report).toContain('roster.yaml.mutation.lock');
expect(report).toContain('roster.yaml.reconcile.lock');
});
});

View File

@@ -0,0 +1,441 @@
import { readFile, stat } from 'node:fs/promises';
import { homedir } from 'node:os';
import { join, relative, resolve } from 'node:path';
import type { Command } from 'commander';
import type { CommandRunner } from './fleet.js';
import {
applyPreparedGeneratedAgentEnvironmentProjection,
prepareGeneratedAgentEnvironmentProjection,
type PreparedGeneratedAgentEnvironmentProjection,
} from '../fleet/generated-env-boundary.js';
import {
acquirePrivateReconcileLock,
acquirePrivateRosterMutationLock,
projectRosterV2AgentGeneratedEnv,
} from '../fleet/fleet-reconciler.js';
import {
parseRosterV2,
validateRosterV2Semantics,
type FleetRosterV2,
} from '../fleet/roster-v2.js';
/**
* `mosaic fleet regen` — recovery-framed regeneration of the roster-derived
* agent env projections (`fleet/agents/<name>.env.generated`) from the roster
* SSOT. It is the recovery layer of #791: when a wiped/diverged operator surface
* has cost an agent its generated projection, regen rebuilds it deterministically
* from `roster.yaml` so the launcher can source the intended identity again.
*
* It is intentionally a THIN projection-only wrapper over the same mapping the
* reconciler apply path uses ({@link projectRosterV2AgentGeneratedEnv}), and it
* has NO code path to systemd lifecycle: regen never starts, stops, or restarts
* an agent. Recovery order forbids restart-before-verify, so the operator must
* verify each rebuilt projection resolves before restarting anything.
*/
export interface FleetRegenAgentPlan {
readonly name: string;
/** Generated-projection path, relative to the Mosaic home (never absolute in output). */
readonly path: string;
/** `create` when the projection is currently absent, `rebuild` when it already exists. */
readonly disposition: 'create' | 'rebuild';
}
/**
* Present ONLY when a `--write` projection apply failed partway through the
* fleet. Every prior agent was already validated and prepared, so `writtenAgents`
* are byte-complete on disk; `failedAgent` is where the write faulted. The
* command surfaces this instead of a bare throw so the operator can finish the
* rebuild and verify every projection before restarting any unit.
*/
export interface FleetRegenIncomplete {
readonly code: 'projection-apply-failed';
/** The agent whose generated-projection write faulted. */
readonly failedAgent: string;
/** Agents whose projections were fully written before the fault (in apply order). */
readonly writtenAgents: readonly string[];
}
export interface FleetRegenResult {
readonly mode: 'dry-run' | 'write';
/** Roster path, relative to the Mosaic home. */
readonly rosterPath: string;
readonly generation: number;
readonly agentCount: number;
/** Count of projections written to disk (always 0 in dry-run). */
readonly written: number;
readonly agents: readonly FleetRegenAgentPlan[];
/** Set ONLY when a `--write` apply faulted mid-fleet — a partial rebuild. */
readonly incomplete?: FleetRegenIncomplete;
/**
* Set ONLY when a shared fleet lock could not be released after a write completed
* (or partially completed) — EITHER `roster.yaml.mutation.lock` or
* `roster.yaml.reconcile.lock`, since regen holds both. The projections in this
* result are real; the lock file may be stale and must be inspected before the
* next mutation. Independent of {@link incomplete} — both can be present at once.
*/
readonly cleanup?: {
readonly code: 'lock-cleanup-failed';
readonly action: 'inspect-lock-before-retry';
};
}
export interface FleetRegenDeps {
/**
* Present ONLY so an accidental future lifecycle wiring is caught by the
* "never restarts" test — regen is contractually forbidden to invoke it.
*/
readonly runner: CommandRunner;
readonly mosaicHome?: string;
/** Persona roots for semantic roster validation (defaults mirror the reconciler). */
readonly rolesDir?: string;
readonly overrideDir?: string;
/** Test seam: canonical roster reader. Defaults to parse + semantic validation. */
readonly readRoster?: (rosterPath: string) => Promise<FleetRosterV2>;
/**
* Test seam: generated-only projection validator. Defaults to the audited
* boundary helper that validates ONLY `<name>.env.generated`.
*/
readonly prepareProjection?: typeof prepareGeneratedAgentEnvironmentProjection;
/** Test seam: generated-only projection writer. Defaults to the audited boundary helper. */
readonly applyProjection?: typeof applyPreparedGeneratedAgentEnvironmentProjection;
/**
* Test seam: acquire the shared reconcile lock before a `--write`. Defaults to
* the reconciler's private lock so regen and reconcile are mutually exclusive
* and a concurrent reconcile cannot race a stale projection write.
*/
readonly acquireReconcileLock?: (mosaicHome: string) => () => Promise<() => Promise<void>>;
/**
* Test seam: acquire the shared roster mutation lock before a `--write`.
* Defaults to the reconciler's hardened, ownership-proving acquirer for the
* same `fleet/roster.yaml.mutation.lock` path that CRUD contends on, so regen
* serializes against agent create/update/delete — both rewrite the same
* generated projections, so without this a concurrent CRUD could race a stale
* projection write.
*/
readonly acquireRosterMutationLock?: (mosaicHome: string) => () => Promise<() => Promise<void>>;
/** Test seam: existence probe for create/rebuild disposition. */
readonly fileExists?: (path: string) => Promise<boolean>;
}
interface FleetRegenOptions {
readonly write?: boolean;
}
function defaultMosaicHome(deps: FleetRegenDeps): string {
return deps.mosaicHome ?? process.env['MOSAIC_HOME'] ?? join(homedir(), '.config', 'mosaic');
}
async function defaultFileExists(path: string): Promise<boolean> {
try {
await stat(path);
return true;
} catch {
return false;
}
}
/**
* Projection-only regeneration core. Deterministic and side-effect-free in
* dry-run; in `--write` it rebuilds each generated projection through the
* audited generated-only boundary writer and NOTHING else — it never writes
* `.env.local`/`.env.quarantine`, never unlinks the legacy `.env`, and never
* touches `deps.runner`.
*
* Fail-closed by construction: the roster is validated (structural + semantic)
* and EVERY agent's projection is prepared before ANY is written, so a single
* invalid agent (or a semantically rejected roster) leaves the whole surface
* untouched. In `--write` mode the read-prepare-apply sequence runs under the
* shared reconcile lock so a concurrent reconcile cannot race a stale write.
*/
export async function executeFleetRegen(
deps: FleetRegenDeps,
options: FleetRegenOptions,
): Promise<FleetRegenResult> {
const mosaicHome = defaultMosaicHome(deps);
const agentEnvDir = join(mosaicHome, 'fleet', 'agents');
const rosterPath = join(mosaicHome, 'fleet', 'roster.yaml');
const readRoster = deps.readRoster ?? defaultReadRoster(deps, mosaicHome);
const prepare = deps.prepareProjection ?? prepareGeneratedAgentEnvironmentProjection;
const apply = deps.applyProjection ?? applyPreparedGeneratedAgentEnvironmentProjection;
const fileExists = deps.fileExists ?? defaultFileExists;
const acquireReconcileLock = deps.acquireReconcileLock ?? acquirePrivateReconcileLock;
const acquireRosterMutationLock =
deps.acquireRosterMutationLock ?? acquirePrivateRosterMutationLock;
const write = options.write === true;
const run = async (): Promise<FleetRegenResult> => {
const roster = await readRoster(rosterPath);
// Prepare (validate) every agent BEFORE any write, so a later failure never
// leaves an earlier projection partially rebuilt.
const prepared: (PreparedGeneratedAgentEnvironmentProjection & {
readonly name: string;
readonly disposition: 'create' | 'rebuild';
})[] = [];
for (const agent of roster.agents) {
const projection = await prepare({
mosaicHome,
agentEnvDir,
agentName: agent.name,
generated: projectRosterV2AgentGeneratedEnv(roster, agent),
});
const disposition = (await fileExists(projection.generatedPath)) ? 'rebuild' : 'create';
prepared.push({ ...projection, name: agent.name, disposition });
}
// Every projection is already validated (prepared); an apply here can only
// fault on the underlying I/O (ENOSPC, EIO, a race outside the lock). If a
// later write faults, earlier agents are already replaced on disk, so we
// stop and report the partial state rather than throwing a bare error that
// hides which projections are now live.
let written = 0;
let incomplete: FleetRegenIncomplete | undefined;
if (write) {
for (const projection of prepared) {
try {
await apply(projection);
} catch {
incomplete = {
code: 'projection-apply-failed',
failedAgent: projection.name,
writtenAgents: prepared.slice(0, written).map((entry) => entry.name),
};
break;
}
written += 1;
}
}
return {
mode: write ? 'write' : 'dry-run',
rosterPath: relative(mosaicHome, rosterPath),
generation: roster.generation,
agentCount: roster.agents.length,
written,
agents: prepared.map((projection) => ({
name: projection.name,
path: relative(mosaicHome, projection.generatedPath),
disposition: projection.disposition,
})),
...(incomplete ? { incomplete } : {}),
};
};
// Dry-run is preview-only and never mutates, so it stays lock-free (usable even
// while a reconcile or CRUD holds a lock). A `--write` must serialize against
// BOTH roster writers of the same generated projections: agent CRUD (roster
// mutation lock) and reconcile (reconcile lock). Acquire in a fixed order — the
// locks are non-blocking (they throw on contention rather than wait), so no
// deadlock is possible — and fail closed if either is already held.
if (!write) return run();
const releases: (() => Promise<void>)[] = [];
try {
releases.push(await acquireRosterMutationLock(mosaicHome)());
releases.push(await acquireReconcileLock(mosaicHome)());
} catch (error: unknown) {
// A later acquire failed; unwind any lock already taken (reverse order). If
// that unwind ALSO faults, surface both — the already-taken lock may now be
// stale and silently block the next regen/reconcile — rather than dropping it.
const releaseFault = await releaseFleetLocks(releases);
throw augmentWithLockCleanupFault(error, releaseFault);
}
let result: FleetRegenResult | undefined;
let primaryError: unknown;
try {
result = await run();
} catch (error: unknown) {
primaryError = error;
}
// Release both locks (reverse acquire order), continuing past a fault so neither
// leaks. A release fault must never discard a completed (or partial) rebuild —
// the projections are already on disk — and must not be silently hidden even
// when the run itself failed with nothing written.
const releaseFault = await releaseFleetLocks(releases);
if (result) {
return releaseFault === undefined
? result
: {
...result,
cleanup: { code: 'lock-cleanup-failed', action: 'inspect-lock-before-retry' },
};
}
// result is undefined ⇒ run() threw ⇒ primaryError is set.
throw augmentWithLockCleanupFault(primaryError, releaseFault);
}
/**
* Releases held locks in reverse acquire order, continuing past a fault so a
* single failed release never leaks the others. Returns the first fault seen (or
* undefined when every release succeeded).
*/
async function releaseFleetLocks(releases: readonly (() => Promise<void>)[]): Promise<unknown> {
let firstFault: unknown;
for (const release of [...releases].reverse()) {
try {
await release();
} catch (error: unknown) {
if (firstFault === undefined) firstFault = error;
}
}
return firstFault;
}
/**
* When a run failed with nothing written AND lock cleanup also faulted, surface
* both: the operator needs to know the lock may be stale (else the next
* regen/reconcile is silently blocked), not just the original run error.
*/
function augmentWithLockCleanupFault(primaryError: unknown, releaseFault: unknown): unknown {
if (releaseFault === undefined) return primaryError;
const base = primaryError instanceof Error ? primaryError.message : String(primaryError);
return new Error(
`${base} (additionally, a fleet lock could not be released — ` +
'fleet/roster.yaml.mutation.lock or roster.yaml.reconcile.lock may be stale; ' +
'inspect and clear it before retry)',
);
}
function defaultReadRoster(
deps: FleetRegenDeps,
mosaicHome: string,
): (rosterPath: string) => Promise<FleetRosterV2> {
return async (rosterPath: string): Promise<FleetRosterV2> => {
const roster = parseRosterV2(await readFile(rosterPath, 'utf8'), 'yaml');
// Enforce the SAME semantic gate as reconcile/plan/verify (persona resolution
// + protected-class tool-policy match) so regen cannot project a roster the
// rest of the fleet surface would reject.
await validateRosterV2Semantics(roster, {
rolesDir: deps.rolesDir ?? join(mosaicHome, 'fleet', 'roles'),
overrideDir: deps.overrideDir ?? join(mosaicHome, 'fleet', 'roles.local'),
});
return roster;
};
}
/**
* Human-readable report — paths and counts ONLY. The rendered projection body
* (KEY=value lines) is never echoed (secrev). In `--write` mode it prints the
* recovery runbook: verify each projection resolves BEFORE any restart.
*/
export function formatFleetRegenReport(result: FleetRegenResult): string[] {
const lines: string[] = [];
const header =
result.mode === 'dry-run'
? 'mosaic fleet regen — dry run (no changes written)'
: 'mosaic fleet regen — wrote roster-derived projections';
lines.push(header);
lines.push(
`roster: ${result.rosterPath} (generation ${result.generation}, ${result.agentCount} agent(s))`,
);
for (const agent of result.agents) {
lines.push(` ${agent.name} ${agent.path} [${agent.disposition}]`);
}
if (result.mode === 'dry-run') {
lines.push(
`Plan: would write ${result.agentCount} generated projection(s). ` +
`Re-run with --write to apply. regen never restarts agents.`,
);
return lines;
}
if (result.incomplete) {
const { failedAgent, writtenAgents } = result.incomplete;
lines.push(
`INCOMPLETE: rebuild stopped at agent ${failedAgent} — its projection write faulted.`,
);
lines.push(
`Rebuilt before the fault: ${writtenAgents.length} projection(s)` +
(writtenAgents.length > 0 ? ` (${writtenAgents.join(', ')})` : '') +
`; ${result.agentCount - writtenAgents.length} not written.`,
);
lines.push(
'Recover: re-run `mosaic fleet regen --write` to finish, then VERIFY every ' +
"agent's fleet/agents/<name>.env.generated resolves BEFORE restarting any " +
'unit. regen never restarts agents.',
);
} else {
lines.push(`Wrote ${result.written} generated projection(s).`);
lines.push('Next steps — DO NOT restart agents before verifying:');
lines.push(
" 1. Confirm each agent's fleet/agents/<name>.env.generated exists and carries " +
'the intended MOSAIC_AGENT_* values.',
);
// The unit sets NO EnvironmentFile= — it launches from a minimal env and the
// launcher (start-agent-session.sh) sources .env.generated itself. Verify the
// launcher path, not a nonexistent EnvironmentFile property.
lines.push(
' 2. Confirm the unit launches from it: `systemctl --user cat mosaic-agent@<name>` ' +
'shows ExecStart running start-agent-session.sh, which reads .env.generated.',
);
lines.push(
' 3. Only then restart one unit at a time: systemctl --user restart mosaic-agent@<name>.',
);
}
if (result.cleanup) {
lines.push(
'WARNING: a fleet lock could not be released — ' +
'fleet/roster.yaml.mutation.lock or fleet/roster.yaml.reconcile.lock may be ' +
'stale; inspect and clear it before the next reconcile or regen.',
);
}
return lines;
}
function resolveRegenMosaicHome(fleetCommand: Command, deps: FleetRegenDeps): string {
const options = fleetCommand.optsWithGlobals<{ mosaicHome?: string }>();
return options.mosaicHome ?? defaultMosaicHome(deps);
}
/**
* regen reads the roster SSOT at `<mosaicHome>/fleet/roster.yaml` and ignores the
* `fleet --roster <path>` global. If an operator passes a non-canonical `--roster`
* they must NOT be silently served the canonical file — mirror reconcile's guard
* and reject, so `mosaic fleet --roster /elsewhere regen --write` fails closed.
*/
function assertRegenCanonicalRoster(fleetCommand: Command, mosaicHome: string): void {
const options = fleetCommand.optsWithGlobals<{ roster?: string }>();
const canonical = join(mosaicHome, 'fleet', 'roster.yaml');
if (options.roster !== undefined && resolve(options.roster) !== resolve(canonical)) {
throw new Error(
'Roster-v2 regeneration requires the canonical roster path (fleet/roster.yaml).',
);
}
}
/** Registers the recovery-framed `mosaic fleet regen` command. */
export function registerFleetRegenCommand(fleetCommand: Command, deps: FleetRegenDeps): void {
fleetCommand
.command('regen')
.description(
'Regenerate roster-derived agent env projections (dry-run default, --write to apply; never restarts)',
)
.option('--write', 'Write the regenerated projections to disk (default: dry-run)')
.option('--json', 'Emit the plan/result as JSON')
.action(async (opts: { write?: boolean; json?: boolean }): Promise<void> => {
const mosaicHome = resolveRegenMosaicHome(fleetCommand, deps);
try {
assertRegenCanonicalRoster(fleetCommand, mosaicHome);
const result = await executeFleetRegen(
{ ...deps, mosaicHome },
{ write: opts.write === true },
);
if (opts.json === true) {
console.log(JSON.stringify(result));
} else {
for (const line of formatFleetRegenReport(result)) console.log(line);
}
// A partial rebuild or a stale-lock cleanup state is a non-zero outcome:
// the operator must finish/verify (and clear the lock) before restarting.
if (result.incomplete || result.cleanup) process.exitCode = 1;
} catch (error: unknown) {
process.exitCode = 1;
const message = error instanceof Error ? error.message : String(error);
process.stderr.write(`mosaic fleet regen failed: ${message}\n`);
}
});
}

View File

@@ -97,6 +97,7 @@ describe('registerFleetCommand', () => {
'provision', 'provision',
'ps', 'ps',
'reconcile', 'reconcile',
'regen',
'remove', 'remove',
'restart', 'restart',
'start', 'start',

View File

@@ -44,6 +44,7 @@ import {
registerFleetReconcilerCommands, registerFleetReconcilerCommands,
type FleetReconcilerCommandDeps, type FleetReconcilerCommandDeps,
} from './fleet-reconciler-command.js'; } from './fleet-reconciler-command.js';
import { registerFleetRegenCommand } from './fleet-regen-command.js';
import { resolveCommsBlock } from '../fleet/comms-onboarding.js'; import { resolveCommsBlock } from '../fleet/comms-onboarding.js';
import { import {
applyPreparedAgentEnvironmentProjection, applyPreparedAgentEnvironmentProjection,
@@ -2063,6 +2064,18 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
reconcileDeps: deps.reconcileDeps, reconcileDeps: deps.reconcileDeps,
}); });
// Recovery (#791 PR3): rebuild roster-derived env projections from the roster
// SSOT. Projection-only and preview-first — never issues a lifecycle restart.
registerFleetRegenCommand(cmd, {
runner,
mosaicHome: deps.mosaicHome,
// Resolve personas the SAME way reconcile does: forward any configured roots
// so a custom-persona-root deployment cannot have reconcile accept a roster
// that regen then rejects against the default `<mosaicHome>/fleet/roles`.
rolesDir: deps.reconcileDeps?.rolesDir,
overrideDir: deps.reconcileDeps?.overrideDir,
});
return cmd; return cmd;
} }

View File

@@ -0,0 +1,466 @@
/**
* Tests for `mosaic restore` (#791 PR2 Task 12).
*
* The durable pre-update snapshot (install.sh: make_durable_snapshot) writes the
* operator-owned surface to $XDG_STATE_HOME/mosaic/backups/pre-update-<ts>/ with
* 0700 dirs / 0600 files. `mosaic restore` is the recovery counterpart:
* • --list (default) enumerate snapshots by timestamp — dry-run, never mutates.
* • --from <ts> restore that snapshot over MOSAIC_HOME, confirmation-gated.
* It reports counts and relative paths ONLY — a snapshot may contain secrets
* (credentials.json), so no file content is ever printed (secrev invariant).
*/
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
import {
mkdtempSync,
rmSync,
mkdirSync,
writeFileSync,
readFileSync,
statSync,
lstatSync,
symlinkSync,
} from 'node:fs';
import { join } from 'node:path';
import { tmpdir, homedir } from 'node:os';
import { Command } from 'commander';
import {
resolveBackupRoot,
listSnapshots,
resolveSnapshotDir,
planRestore,
applyRestore,
runRestore,
registerRestoreCommand,
} from './restore.js';
const SECRET = 'SUPER-SECRET-TOKEN-do-not-log-restore';
function seedSnapshot(root: string, ts: string, files: Record<string, string>): string {
const dir = join(root, `pre-update-${ts}`);
for (const [rel, content] of Object.entries(files)) {
const abs = join(dir, rel);
mkdirSync(join(abs, '..'), { recursive: true });
writeFileSync(abs, content);
}
return dir;
}
describe('mosaic restore (#791 PR2)', () => {
let tmp: string;
let backups: string;
beforeEach(() => {
tmp = mkdtempSync(join(tmpdir(), 'mosaic-restore-'));
backups = join(tmp, 'state', 'mosaic', 'backups');
mkdirSync(backups, { recursive: true });
vi.clearAllMocks();
});
afterEach(() => {
rmSync(tmp, { recursive: true, force: true });
});
describe('resolveBackupRoot', () => {
it('honors XDG_STATE_HOME', () => {
expect(resolveBackupRoot({ XDG_STATE_HOME: '/x/state' })).toBe('/x/state/mosaic/backups');
});
it('falls back to ~/.local/state', () => {
expect(resolveBackupRoot({})).toBe(join(homedir(), '.local', 'state', 'mosaic', 'backups'));
});
});
describe('listSnapshots', () => {
it('returns [] when the backup root does not exist', () => {
expect(listSnapshots(join(tmp, 'nope'))).toEqual([]);
});
it('lists pre-update snapshots newest-first with file counts, ignoring other dirs', () => {
seedSnapshot(backups, '20240101T000000Z', { 'SOUL.md': 'a' });
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'b', 'agents/x.conf': 'c' });
mkdirSync(join(backups, 'unrelated-dir'), { recursive: true });
const snaps = listSnapshots(backups);
expect(snaps.map((s) => s.timestamp)).toEqual(['20260101T000000Z', '20240101T000000Z']);
expect(snaps[0]!.fileCount).toBe(2);
expect(snaps[1]!.fileCount).toBe(1);
});
});
describe('resolveSnapshotDir', () => {
it('resolves by bare timestamp and by full pre-update-<ts> name', () => {
const dir = seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'a' });
expect(resolveSnapshotDir(backups, '20260101T000000Z')).toBe(dir);
expect(resolveSnapshotDir(backups, 'pre-update-20260101T000000Z')).toBe(dir);
});
it('returns undefined for an unknown timestamp', () => {
expect(resolveSnapshotDir(backups, '19990101T000000Z')).toBeUndefined();
});
});
describe('planRestore', () => {
it('walks nested dirs and returns every relative file path', () => {
const dir = seedSnapshot(backups, '20260101T000000Z', {
'SOUL.md': 'a',
'agents/x.conf': 'b',
'tools/_lib/credentials.json': 'c',
});
expect(planRestore(dir).sort()).toEqual(
['SOUL.md', 'agents/x.conf', 'tools/_lib/credentials.json'].sort(),
);
});
});
describe('applyRestore', () => {
it('restores byte-exact content, creates parent dirs, and sets 0600', () => {
const dir = seedSnapshot(backups, '20260101T000000Z', {
'SOUL.md': 'original-soul',
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
});
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
// A diverged operator file that restore must overwrite.
writeFileSync(join(home, 'SOUL.md'), 'CORRUPTED');
const n = applyRestore(dir, home, planRestore(dir));
expect(n).toBe(2);
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('original-soul');
expect(readFileSync(join(home, 'tools/_lib/credentials.json'), 'utf8')).toBe(
`TOKEN=${SECRET}\n`,
);
expect(statSync(join(home, 'SOUL.md')).mode & 0o777).toBe(0o600);
expect(statSync(join(home, 'tools/_lib/credentials.json')).mode & 0o777).toBe(0o600);
});
});
describe('runRestore', () => {
it('--list prints timestamps and counts, mutating nothing', async () => {
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'a', 'agents/x.conf': 'b' });
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
const code = await runRestore({
list: true,
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
});
expect(code).toBe(0);
const out = log.mock.calls.flat().join('\n');
expect(out).toContain('20260101T000000Z');
expect(out).toMatch(/2\b/); // the file count is surfaced
log.mockRestore();
});
it('--from restores the snapshot over MOSAIC_HOME byte-exact (yes bypasses prompt)', async () => {
seedSnapshot(backups, '20260101T000000Z', {
'SOUL.md': 'restored-soul',
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
});
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
writeFileSync(join(home, 'SOUL.md'), 'STALE');
const code = await runRestore({
from: '20260101T000000Z',
yes: true,
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
});
expect(code).toBe(0);
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('restored-soul');
expect(readFileSync(join(home, 'tools/_lib/credentials.json'), 'utf8')).toBe(
`TOKEN=${SECRET}\n`,
);
});
it('--from with an unknown timestamp fails without mutating', async () => {
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
writeFileSync(join(home, 'SOUL.md'), 'KEEP');
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
const code = await runRestore({
from: '19990101T000000Z',
yes: true,
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
});
expect(code).toBe(1);
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('KEEP');
err.mockRestore();
});
it('--dry-run with --from reports the plan but mutates nothing', async () => {
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'snap' });
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
writeFileSync(join(home, 'SOUL.md'), 'UNCHANGED');
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
const code = await runRestore({
from: '20260101T000000Z',
dryRun: true,
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
});
expect(code).toBe(0);
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('UNCHANGED');
log.mockRestore();
});
it('--from prompts and applies the restore when the operator confirms', async () => {
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'confirmed-soul' });
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
writeFileSync(join(home, 'SOUL.md'), 'STALE');
vi.spyOn(console, 'log').mockImplementation(() => {});
const confirm = vi.fn().mockResolvedValue(true);
const code = await runRestore({
from: '20260101T000000Z',
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
confirm,
});
expect(code).toBe(0);
expect(confirm).toHaveBeenCalledOnce();
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('confirmed-soul');
});
it('--from aborts without mutating when the operator declines', async () => {
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'snap' });
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
writeFileSync(join(home, 'SOUL.md'), 'KEEP');
vi.spyOn(console, 'log').mockImplementation(() => {});
const confirm = vi.fn().mockResolvedValue(false);
const code = await runRestore({
from: '20260101T000000Z',
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
confirm,
});
expect(code).toBe(0);
expect(confirm).toHaveBeenCalledOnce();
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('KEEP');
});
it('MOSAIC_ASSUME_YES=1 bypasses the confirmation prompt', async () => {
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'env-yes' });
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
writeFileSync(join(home, 'SOUL.md'), 'STALE');
vi.spyOn(console, 'log').mockImplementation(() => {});
const confirm = vi.fn().mockResolvedValue(false);
const code = await runRestore({
from: '20260101T000000Z',
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state'), MOSAIC_ASSUME_YES: '1' },
confirm,
});
expect(code).toBe(0);
expect(confirm).not.toHaveBeenCalled();
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('env-yes');
});
it('--list reports gracefully when no snapshots exist', async () => {
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
const code = await runRestore({
list: true,
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'empty-state') },
});
expect(code).toBe(0);
expect(log.mock.calls.flat().join('\n')).toMatch(/No pre-update snapshots/);
});
it('never prints a secret value found inside a backed-up file', async () => {
seedSnapshot(backups, '20260101T000000Z', {
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
});
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
await runRestore({
list: true,
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
});
await runRestore({
from: '20260101T000000Z',
yes: true,
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
});
const all = [...log.mock.calls, ...err.mock.calls].flat().join('\n');
expect(all).not.toContain(SECRET);
log.mockRestore();
err.mockRestore();
});
});
// Regression coverage for the codex code+security review of PR2 (#791):
// CWE-22 traversal via --from, and CWE-59 symlink write-through in applyRestore.
describe('security hardening', () => {
it.each([
'../../etc',
'pre-update-/../../tmp/poison',
'pre-update-../evil',
'20260101T000000Z/../../../tmp',
'not-a-timestamp',
'2026-01-01',
])('resolveSnapshotDir rejects traversal / malformed selector %j', (bad) => {
// Even if a matching directory exists on disk, a non-timestamp selector
// must not resolve — the only accepted shape is <8>T<6>Z[-n].
expect(resolveSnapshotDir(backups, bad)).toBeUndefined();
});
it('runRestore --from a traversal selector fails closed without copying', async () => {
// Plant a real dir one level ABOVE the backup root. The naive resolver
// `join(root, from)` with `from='../poison'` would reach it (backups is
// .../mosaic/backups, so `../poison` == .../mosaic/poison) and import it.
const outside = join(tmp, 'state', 'mosaic', 'poison');
mkdirSync(outside, { recursive: true });
writeFileSync(join(outside, 'x'), 'attacker');
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
const code = await runRestore({
from: '../poison',
yes: true,
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
});
expect(code).toBe(1);
expect(statSync(home).isDirectory()).toBe(true);
// Nothing from `outside` was imported.
expect(() => statSync(join(home, 'x'))).toThrow();
err.mockRestore();
});
it('applyRestore refuses to write a secret through a symlinked leaf (CWE-59)', () => {
const dir = seedSnapshot(backups, '20260101T000000Z', {
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
});
const home = join(tmp, 'home');
mkdirSync(join(home, 'tools', '_lib'), { recursive: true });
// Attacker points the operator credentials file at a file they can read.
const exfil = join(tmp, 'exfil-target');
writeFileSync(exfil, 'original-attacker-content');
symlinkSync(exfil, join(home, 'tools', '_lib', 'credentials.json'));
expect(() => applyRestore(dir, home, planRestore(dir))).toThrow();
// The secret was NOT written through the link into the attacker's file.
expect(readFileSync(exfil, 'utf8')).toBe('original-attacker-content');
});
it('applyRestore refuses to write through a symlinked ancestor (CWE-59)', () => {
const dir = seedSnapshot(backups, '20260101T000000Z', {
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
});
const home = join(tmp, 'home');
mkdirSync(join(home, 'tools'), { recursive: true });
// Attacker replaces the `tools/_lib` ancestor with a symlink out of the root.
const exfilDir = join(tmp, 'exfil-dir');
mkdirSync(exfilDir, { recursive: true });
symlinkSync(exfilDir, join(home, 'tools', '_lib'));
expect(() => applyRestore(dir, home, planRestore(dir))).toThrow();
// Nothing was written into the attacker-controlled directory.
expect(() => statSync(join(exfilDir, 'credentials.json'))).toThrow();
});
it('runRestore surfaces a symlink violation as exit 1 without leaking the secret', async () => {
seedSnapshot(backups, '20260101T000000Z', {
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
});
const home = join(tmp, 'home');
mkdirSync(join(home, 'tools', '_lib'), { recursive: true });
const exfil = join(tmp, 'exfil-target');
writeFileSync(exfil, 'attacker');
symlinkSync(exfil, join(home, 'tools', '_lib', 'credentials.json'));
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
const code = await runRestore({
from: '20260101T000000Z',
yes: true,
mosaicHome: home,
env: { XDG_STATE_HOME: join(tmp, 'state') },
});
expect(code).toBe(1);
expect(readFileSync(exfil, 'utf8')).toBe('attacker');
const all = [...log.mock.calls, ...err.mock.calls].flat().join('\n');
expect(all).not.toContain(SECRET);
log.mockRestore();
err.mockRestore();
});
it('applyRestore replaces a diverged regular file in place with 0600 (not a symlink)', () => {
const dir = seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'restored' });
const home = join(tmp, 'home');
mkdirSync(home, { recursive: true });
writeFileSync(join(home, 'SOUL.md'), 'stale');
const n = applyRestore(dir, home, planRestore(dir));
expect(n).toBe(1);
expect(lstatSync(join(home, 'SOUL.md')).isSymbolicLink()).toBe(false);
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('restored');
expect(statSync(join(home, 'SOUL.md')).mode & 0o777).toBe(0o600);
});
});
describe('registerRestoreCommand', () => {
it('registers `restore` with the expected flags', () => {
const program = new Command();
program.exitOverride();
registerRestoreCommand(program);
const cmd = program.commands.find((c) => c.name() === 'restore');
expect(cmd).toBeDefined();
const longs = cmd!.options.map((o) => o.long);
expect(longs).toEqual(
expect.arrayContaining(['--list', '--from', '--dry-run', '--yes', '--mosaic-home']),
);
});
it('runs the list action end-to-end via the parsed command', async () => {
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'a' });
const prevXdg = process.env['XDG_STATE_HOME'];
process.env['XDG_STATE_HOME'] = join(tmp, 'state');
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
try {
const program = new Command();
program.exitOverride();
registerRestoreCommand(program);
await program.parseAsync(['restore', '--list', '--mosaic-home', join(tmp, 'home')], {
from: 'user',
});
expect(log.mock.calls.flat().join('\n')).toContain('20260101T000000Z');
} finally {
log.mockRestore();
if (prevXdg === undefined) delete process.env['XDG_STATE_HOME'];
else process.env['XDG_STATE_HOME'] = prevXdg;
}
});
});
});

View File

@@ -0,0 +1,313 @@
/**
* restore.ts — top-level `mosaic restore` command (#791 PR2 Task 12)
*
* Recovery counterpart to the durable pre-update snapshot taken by install.sh
* (make_durable_snapshot). Before a keep-mode upgrade mutates anything, the
* installer copies the operator-owned surface to
* $XDG_STATE_HOME/mosaic/backups/pre-update-<UTC-ts>/ (0700 dirs / 0600 files)
* This command lets the operator inspect and roll back to those snapshots:
*
* mosaic restore # == --list: enumerate snapshots (dry-run)
* mosaic restore --list
* mosaic restore --from <ts> # restore that snapshot over MOSAIC_HOME
* mosaic restore --from <ts> --dry-run
*
* SECREV INVARIANT: a snapshot may contain secrets (e.g. tools/_lib/credentials.json).
* This command reports counts and RELATIVE PATHS only — it never reads a backed-up
* file into any logged string. Restored files are written back 0600 (owner-only),
* matching the snapshot's own private posture. The path convention here mirrors
* install.sh `backup_root()`; keep the two in sync (no shared code across the boundary).
*/
import {
existsSync,
readdirSync,
statSync,
lstatSync,
readFileSync,
openSync,
writeSync,
fchmodSync,
closeSync,
constants,
} from 'node:fs';
import { createInterface } from 'node:readline';
import { homedir } from 'node:os';
import { join, dirname, relative } from 'node:path';
import type { Command } from 'commander';
import { DEFAULT_MOSAIC_HOME } from '../constants.js';
import { assertCanonicalContainment, ensureManagedDirectory } from '../fleet/secure-file.js';
// ─── types ───────────────────────────────────────────────────────────────────
export interface SnapshotInfo {
/** The UTC stamp after the `pre-update-` prefix, e.g. "20260716T232225Z". */
readonly timestamp: string;
/** Absolute path to the snapshot directory. */
readonly dir: string;
/** Number of files captured in the snapshot. */
readonly fileCount: number;
}
export interface RestoreOptions {
list?: boolean;
from?: string;
dryRun?: boolean;
yes?: boolean;
mosaicHome: string;
/** Environment source (injectable for tests); defaults to process.env. */
env?: NodeJS.ProcessEnv;
/**
* Confirmation gate (injectable for tests); defaults to an interactive
* readline prompt. Returns true to proceed with the overwrite.
*/
confirm?: (question: string) => Promise<boolean>;
}
const SNAPSHOT_PREFIX = 'pre-update-';
/**
* The exact shape install.sh `make_durable_snapshot()` stamps: `<8>T<6>Z` UTC,
* with an optional `-<n>` same-second collision suffix. `--from` is matched
* against this — nothing containing a path separator or `..` can pass, so a
* selector can never escape the backup root (CWE-22).
*/
const SNAPSHOT_TS_RE = /^\d{8}T\d{6}Z(?:-\d+)?$/;
// ─── pure helpers ─────────────────────────────────────────────────────────────
/** Resolve the durable-snapshot root, mirroring install.sh `backup_root()`. */
export function resolveBackupRoot(env: NodeJS.ProcessEnv = process.env): string {
const stateHome = env['XDG_STATE_HOME'] || join(homedir(), '.local', 'state');
return join(stateHome, 'mosaic', 'backups');
}
/** Recursively collect every file under `dir` as a path relative to `dir`. */
export function planRestore(dir: string): string[] {
const out: string[] = [];
const walk = (cur: string): void => {
for (const entry of readdirSync(cur, { withFileTypes: true })) {
const abs = join(cur, entry.name);
if (entry.isDirectory()) {
walk(abs);
} else if (entry.isFile()) {
out.push(relative(dir, abs));
}
}
};
if (existsSync(dir)) walk(dir);
return out;
}
/** Enumerate snapshots newest-first (the `pre-update-<ts>` names sort chronologically). */
export function listSnapshots(root: string): SnapshotInfo[] {
if (!existsSync(root)) return [];
let entries: string[];
try {
entries = readdirSync(root);
} catch {
return [];
}
return entries
.filter((name) => name.startsWith(SNAPSHOT_PREFIX))
.map((name) => join(root, name))
.filter((dir) => {
try {
return statSync(dir).isDirectory();
} catch {
return false;
}
})
.sort()
.reverse()
.map((dir) => ({
timestamp: dir.split('/').at(-1)!.slice(SNAPSHOT_PREFIX.length),
dir,
fileCount: planRestore(dir).length,
}));
}
/**
* Resolve a snapshot dir from a `--from` selector. Accepts ONLY a strict
* generated identifier — a bare `<ts>` or the full `pre-update-<ts>` name — and
* builds exactly `join(root, 'pre-update-' + ts)`. A selector containing `/`,
* `..`, or anything but the timestamp shape is rejected (returns undefined), so
* `--from` can never traverse outside the backup root (CWE-22). The resolved dir
* must be a real, non-symlink directory (lstat, not stat), so a symlinked
* snapshot entry can't redirect the restore either.
*/
export function resolveSnapshotDir(root: string, from: string): string | undefined {
const ts = from.startsWith(SNAPSHOT_PREFIX) ? from.slice(SNAPSHOT_PREFIX.length) : from;
if (!SNAPSHOT_TS_RE.test(ts)) return undefined;
const dir = join(root, `${SNAPSHOT_PREFIX}${ts}`);
try {
if (lstatSync(dir).isDirectory()) return dir;
} catch {
/* absent or inaccessible */
}
return undefined;
}
/**
* Copy each `relPaths` entry from the snapshot back into `mosaicHome`, forcing
* 0600 on the restored file (owner-only — the operator surface may hold secrets).
* Returns the number of files restored. Never reads a file's content into a
* logged string.
*
* SYMLINK-SAFE (CWE-59): a snapshot may hold secrets, so we must never let a
* tampered destination redirect the write. Every destination path is contained
* within `mosaicHome` (assertCanonicalContainment) and every ancestor is proven
* to be a real, non-symlink directory (ensureManagedDirectory) before we write.
* The leaf itself is opened O_NOFOLLOW, so if it was swapped for a symlink the
* open fails closed (ELOOP) rather than writing the secret through the link.
*/
export function applyRestore(
snapDir: string,
mosaicHome: string,
relPaths: readonly string[],
): number {
let restored = 0;
for (const rel of relPaths) {
const src = join(snapDir, rel);
const dst = join(mosaicHome, rel);
// Fail closed if the target path escapes the managed root or any ancestor is
// a symlink; create missing ancestors as private (0700) real directories.
assertCanonicalContainment(mosaicHome, dst);
ensureManagedDirectory(mosaicHome, dirname(dst));
// O_NOFOLLOW: refuse to follow a symlink at the leaf (secret exfil guard).
// O_CREAT|O_TRUNC: create a fresh 0600 file, or overwrite a diverged real one.
const fd = openSync(
dst,
constants.O_WRONLY | constants.O_CREAT | constants.O_TRUNC | constants.O_NOFOLLOW,
0o600,
);
try {
fchmodSync(fd, 0o600); // enforce 0600 even when the file pre-existed
writeSync(fd, readFileSync(src));
} finally {
closeSync(fd);
}
restored += 1;
}
return restored;
}
// ─── orchestration ────────────────────────────────────────────────────────────
async function promptConfirm(question: string): Promise<boolean> {
const rl = createInterface({ input: process.stdin, output: process.stdout });
try {
return await new Promise<boolean>((resolve) => {
rl.question(`${question} [y/N] `, (ans) => resolve(ans.trim().toLowerCase() === 'y'));
});
} finally {
rl.close();
}
}
/**
* Run `mosaic restore`. Returns a process exit code (0 ok, 1 error) rather than
* calling process.exit, so it stays unit-testable.
*/
export async function runRestore(opts: RestoreOptions): Promise<number> {
const env = opts.env ?? process.env;
const root = resolveBackupRoot(env);
// Default action (and explicit --list): enumerate, never mutate.
if (opts.list || !opts.from) {
const snaps = listSnapshots(root);
if (snaps.length === 0) {
console.log(`No pre-update snapshots found under ${root}.`);
return 0;
}
console.log(`Pre-update snapshots under ${root} (newest first):\n`);
for (const s of snaps) {
console.log(` ${s.timestamp}${s.fileCount} file(s)`);
}
console.log(`\nRestore one with: mosaic restore --from <timestamp>`);
return 0;
}
// --from <ts>: restore over the operator surface.
const snapDir = resolveSnapshotDir(root, opts.from);
if (!snapDir) {
console.error(`No snapshot matching '${opts.from}' under ${root}.`);
console.error(`Run 'mosaic restore --list' to see available timestamps.`);
return 1;
}
const relPaths = planRestore(snapDir);
const ts = snapDir.split('/').at(-1)!.slice(SNAPSHOT_PREFIX.length);
if (opts.dryRun) {
console.log(
`[dry-run] Would restore ${relPaths.length} file(s) from snapshot ${ts} into ${opts.mosaicHome}:`,
);
for (const rel of relPaths) console.log(` ${rel}`);
console.log('[dry-run] No changes made.');
return 0;
}
const assumeYes = opts.yes || env['MOSAIC_ASSUME_YES'] === '1';
if (!assumeYes) {
console.log(
`About to restore ${relPaths.length} operator file(s) from snapshot ${ts} into ${opts.mosaicHome}.`,
);
console.log('This OVERWRITES those files with their pre-update contents.');
const ok = await (opts.confirm ?? promptConfirm)('Proceed?');
if (!ok) {
console.log('Restore cancelled. No changes made.');
return 0;
}
}
let n: number;
try {
n = applyRestore(snapDir, opts.mosaicHome, relPaths);
} catch (err) {
// A containment/symlink violation is a fail-closed security stop, not a
// routine error — surface it without leaking file contents and abort.
console.error(
`Restore aborted: a destination path under ${opts.mosaicHome} is unsafe to write ` +
`(symlink or escapes the managed root). No files were restored. (${(err as Error).message})`,
);
return 1;
}
console.log(`Restored ${n} operator file(s) from snapshot ${ts} into ${opts.mosaicHome}.`);
return 0;
}
// ─── commander registration ───────────────────────────────────────────────────
export function registerRestoreCommand(program: Command): void {
program
.command('restore')
.description('List or restore durable pre-update snapshots of your operator config (#791)')
.option('--list', 'List available snapshots by timestamp (default action)')
.option('--from <timestamp>', 'Restore the snapshot with this timestamp over MOSAIC_HOME')
.option('--dry-run', 'With --from: show what would be restored without changing anything')
.option('--yes, -y', 'Skip the confirmation prompt (also: MOSAIC_ASSUME_YES=1)')
.option(
'--mosaic-home <path>',
'Override MOSAIC_HOME directory',
process.env['MOSAIC_HOME'] ?? DEFAULT_MOSAIC_HOME,
)
.action(
async (opts: {
list?: boolean;
from?: string;
dryRun?: boolean;
yes?: boolean;
mosaicHome: string;
}) => {
const code = await runRestore({
list: opts.list,
from: opts.from,
dryRun: opts.dryRun,
yes: opts.yes,
mosaicHome: opts.mosaicHome,
});
if (code !== 0) process.exit(code);
},
);
}

View File

@@ -381,6 +381,22 @@ function isMissingFile(error: unknown): boolean {
return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT'; return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT';
} }
/**
* Non-blocking acquire of the roster mutation lock: an exclusive `wx` create that
* throws `concurrent-mutation` (never waits) if the lock is already held. The
* on-disk format is an empty private file. CRUD releases in a plain `finally` with
* no result-preservation logic, so a release fault here must not override the
* mutation result — the release swallows unlink failures by design.
*
* The recovery-framed `fleet regen` contends on this exact same lock file, but it
* does so through the hardened, ownership-proving acquirer in the reconciler
* (`acquirePrivateRosterMutationLock`), not this one: whoever wins the `wx` create
* owns the file (the loser always gets `concurrent-mutation`), so the reconciler's
* ownership token is only ever written and read back by the same regen invocation,
* never by this empty-file writer. The two acquirers stay mutually compatible at
* the `wx`-contention layer while regen additionally proves ownership before it
* unlinks — a guarantee CRUD does not need because it releases unconditionally.
*/
async function acquireMutationLock(lockPath: string): Promise<() => Promise<void>> { async function acquireMutationLock(lockPath: string): Promise<() => Promise<void>> {
try { try {
const handle = await open(lockPath, 'wx', 0o600); const handle = await open(lockPath, 'wx', 0o600);

View File

@@ -584,6 +584,29 @@ function defaultValidateRoster(
}; };
} }
/**
* The single roster-v2 → generated-projection mapping. Both the reconciler's
* apply path ({@link defaultPrepareProjections}) and the recovery-framed
* `mosaic fleet regen` command derive their generated env from THIS one
* function, so the two paths can never drift (the #791 single-SSOT invariant).
* Pure: no IO, no clock — a deterministic function of the roster alone.
*/
export function projectRosterV2AgentGeneratedEnv(
roster: FleetRosterV2,
agent: FleetRosterV2Agent,
): Readonly<Record<string, string>> {
return {
MOSAIC_AGENT_NAME: agent.name,
MOSAIC_AGENT_CLASS: agent.className,
MOSAIC_AGENT_RUNTIME: agent.runtime,
MOSAIC_AGENT_MODEL: agent.model,
MOSAIC_AGENT_REASONING: agent.reasoning,
MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy,
MOSAIC_AGENT_WORKDIR: agent.workingDirectory,
MOSAIC_TMUX_SOCKET: roster.tmux.socketName,
};
}
function defaultPrepareProjections( function defaultPrepareProjections(
request: FleetReconcileRequest, request: FleetReconcileRequest,
): (roster: FleetRosterV2) => Promise<readonly PreparedAgentEnvironmentProjection[]> { ): (roster: FleetRosterV2) => Promise<readonly PreparedAgentEnvironmentProjection[]> {
@@ -596,16 +619,7 @@ function defaultPrepareProjections(
mosaicHome, mosaicHome,
agentEnvDir: join(mosaicHome, 'fleet', 'agents'), agentEnvDir: join(mosaicHome, 'fleet', 'agents'),
agentName: agent.name, agentName: agent.name,
generated: { generated: projectRosterV2AgentGeneratedEnv(roster, agent),
MOSAIC_AGENT_NAME: agent.name,
MOSAIC_AGENT_CLASS: agent.className,
MOSAIC_AGENT_RUNTIME: agent.runtime,
MOSAIC_AGENT_MODEL: agent.model,
MOSAIC_AGENT_REASONING: agent.reasoning,
MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy,
MOSAIC_AGENT_WORKDIR: agent.workingDirectory,
MOSAIC_TMUX_SOCKET: roster.tmux.socketName,
},
}), }),
), ),
); );
@@ -620,62 +634,193 @@ function mosaicHomeFor(deps: FleetReconcileDeps): string {
return deps.mosaicHome ?? join(homedir(), '.config', 'mosaic'); return deps.mosaicHome ?? join(homedir(), '.config', 'mosaic');
} }
/** Acquires a private lock only after proving the canonical managed path. */ /** Acquires the private reconcile lock only after proving the canonical managed path. */
export function acquirePrivateReconcileLock( export function acquirePrivateReconcileLock(
mosaicHome: string, mosaicHome: string,
openLock: typeof open = open, openLock: typeof open = open,
): () => Promise<() => Promise<void>> {
return acquirePrivateManagedRosterLock(
mosaicHome,
'roster.yaml.reconcile.lock',
'Another roster reconciliation is in progress.',
openLock,
);
}
/**
* Acquires the private roster MUTATION lock (`roster.yaml.mutation.lock`) with the
* exact same ownership-proof discipline as the reconcile lock. Exposed so the
* recovery-framed `mosaic fleet regen` can serialize its projection rewrites
* against agent CRUD — which holds this very lock while it rewrites the roster and
* the same derived projections — reusing this hardened acquirer rather than
* reimplementing it.
*
* The release PROVES ownership (device/inode + token) before unlinking, so a lock
* that was already cleared and replaced by the time release runs is never deleted
* out from under the new owner (it fails closed as `lock-cleanup-failed` instead),
* and that cleanup fault is propagated to the caller rather than silently dropped.
* CRUD contends on the identical path with a plain empty-file `wx` create; the two
* never co-own it (whoever wins `wx` owns it; the loser gets `concurrent-mutation`),
* so the token this writes is only ever read back by the same regen invocation that
* wrote it.
*
* SCOPE of the guarantee (matches the merged reconcile lock exactly — see
* {@link acquirePrivateManagedRosterLock}): the ownership proof closes the
* *reachable* window — a stale-lock reaper or operator cleared our lock and another
* writer took it BEFORE our release began — not the residual sub-instruction window
* between the final check and the path-based `unlink`. Closing that residual fully
* requires an fd-held advisory lock adopted by every fleet writer (CRUD, reconcile,
* regen), which is a cross-cutting mechanism change out of scope for this
* projection-only recovery command; it is unreachable within the `wx` writer
* protocol regardless (no Mosaic writer removes a lock it does not own, so only
* external interference can vacate our inode mid-release).
*/
export function acquirePrivateRosterMutationLock(
mosaicHome: string,
openLock: typeof open = open,
): () => Promise<() => Promise<void>> {
return acquirePrivateManagedRosterLock(
mosaicHome,
'roster.yaml.mutation.lock',
'Another roster mutation is in progress.',
openLock,
);
}
/**
* Shared body for the ownership-proving managed roster locks. Acquires `lockLeaf`
* under `<mosaicHome>/fleet` with a private `wx` create, writes an ownership token,
* and returns a release that re-proves device/inode + token before unlinking so it
* can never remove a lock it no longer owns. Non-blocking: throws `concurrent-mutation`
* (with `busyMessage`) on contention rather than waiting.
*/
function acquirePrivateManagedRosterLock(
mosaicHome: string,
lockLeaf: string,
busyMessage: string,
openLock: typeof open,
): () => Promise<() => Promise<void>> { ): () => Promise<() => Promise<void>> {
const fleetDir = join(mosaicHome, 'fleet'); const fleetDir = join(mosaicHome, 'fleet');
const lockPath = join(fleetDir, 'roster.yaml.reconcile.lock'); const lockPath = join(fleetDir, lockLeaf);
// The message-producing helpers below serve BOTH managed locks, so every fault
// names the actual lock file (`fleet/<leaf>`) — an operator needs to know WHICH
// lock is stale, not a hardcoded "reconciliation lock".
const lockLabel = `fleet/${lockLeaf}`;
return async (): Promise<() => Promise<void>> => { return async (): Promise<() => Promise<void>> => {
await assertPrivateManagedDirectory(mosaicHome); await assertPrivateManagedDirectory(mosaicHome);
await assertPrivateManagedDirectory(fleetDir); await assertPrivateManagedDirectory(fleetDir);
await assertSafeLockLeafIfPresent(lockPath); await assertSafeLockLeafIfPresent(lockPath, lockLabel);
let handle: FileHandle; let handle: FileHandle;
try { try {
handle = await openLock(lockPath, 'wx', 0o600); handle = await openLock(lockPath, 'wx', 0o600);
} catch (error: unknown) { } catch (error: unknown) {
if (isCode(error, 'EEXIST')) { if (isCode(error, 'EEXIST')) {
await assertSafeLockLeafIfPresent(lockPath); await assertSafeLockLeafIfPresent(lockPath, lockLabel);
throw new FleetReconcileError( throw new FleetReconcileError('concurrent-mutation', busyMessage);
'concurrent-mutation',
'Another roster reconciliation is in progress.',
);
} }
throw new FleetReconcileError('lock-io-failed', 'The reconciliation lock cannot be created.'); throw new FleetReconcileError('lock-io-failed', `The ${lockLabel} lock cannot be created.`);
} }
// Identity of the lock WE exclusively created (the `wx` create guaranteed it is
// ours). Captured up front so both the release closure and the init-failure
// cleanup below can prove ownership by device/inode before touching the file.
const created = await handle.stat().catch((): undefined => undefined);
const token = randomUUID(); const token = randomUUID();
let tokenPersisted = false;
try { try {
await handle.writeFile(`${token}\n`, 'utf8'); await handle.writeFile(`${token}\n`, 'utf8');
const opened = await handle.stat(); tokenPersisted = true;
await handle.close(); await handle.close();
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'unsafe-lock'); if (!created)
throw new FleetReconcileError(
'lock-io-failed',
`The ${lockLabel} lock cannot be initialized.`,
);
await assertLockOwnership(
lockPath,
created.dev,
created.ino,
token,
'unsafe-lock',
lockLabel,
);
return async (): Promise<void> => { return async (): Promise<void> => {
try { try {
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed'); await assertLockOwnership(
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed'); lockPath,
created.dev,
created.ino,
token,
'lock-cleanup-failed',
lockLabel,
);
await unlink(lockPath); await unlink(lockPath);
} catch (error: unknown) { } catch (error: unknown) {
if (error instanceof FleetReconcileError) throw error; if (error instanceof FleetReconcileError) throw error;
throw new FleetReconcileError( throw new FleetReconcileError('lock-cleanup-failed', `The ${lockLabel} cleanup failed.`);
'lock-cleanup-failed',
'The reconciliation lock cleanup failed.',
);
} }
}; };
} catch (error: unknown) { } catch (error: unknown) {
await handle.close().catch((): void => {}); await handle.close().catch((): void => {});
// Best-effort: remove the lock THIS invocation created so a transient init
// fault does not strand a lock that would block future regen and agent CRUD.
// dev/ino-guarded so it can never delete a replacement; swallowed so cleanup
// failure never masks the initialization error being surfaced. The token is
// passed only once persisted, so the fallback path (when the post-create stat
// failed and dev/ino is unavailable) can still prove ownership by content.
await removeOwnedLockLeafBestEffort(lockPath, created, tokenPersisted ? token : undefined);
if (error instanceof FleetReconcileError) throw error; if (error instanceof FleetReconcileError) throw error;
throw new FleetReconcileError( throw new FleetReconcileError(
'lock-io-failed', 'lock-io-failed',
'The reconciliation lock cannot be initialized.', `The ${lockLabel} lock cannot be initialized.`,
); );
} }
}; };
} }
/**
* Best-effort removal of a managed lock leaf THIS process created, used only on the
* initialization-failure path. Two independent ownership proofs, so a lock that was
* already replaced is never deleted:
*
* - primary: the captured device/inode of the file we exclusively `wx`-created; or
* - fallback: when the post-create stat itself failed (dev/ino unavailable), the
* random `ownershipToken` we persisted — only OUR lock carries it, so a CRUD
* (empty) or a differently-tokened replacement is never removed.
*
* Every fault is swallowed because the caller is already surfacing the initialization
* error and a leftover lock is recoverable via the documented runbook. If BOTH proofs
* are unavailable (stat failed AND the token write never landed) the lock is left in
* place rather than risk deleting another writer's file — a doubly-degenerate case
* requiring two independent fs faults on a just-created fd.
*/
async function removeOwnedLockLeafBestEffort(
lockPath: string,
created: { dev: number; ino: number } | undefined,
ownershipToken: string | undefined,
): Promise<void> {
try {
const current = await lstat(lockPath);
if (!current.isFile() || current.isSymbolicLink()) return;
if (created) {
if (current.dev === created.dev && current.ino === created.ino) {
await unlink(lockPath);
}
return;
}
if (ownershipToken !== undefined) {
const contents = await readFile(lockPath, 'utf8');
if (contents.trim() === ownershipToken) {
await unlink(lockPath);
}
}
} catch {
// Swallowed: leftover lock is recoverable; do not mask the init error.
}
}
async function assertPrivateManagedDirectory(path: string): Promise<void> { async function assertPrivateManagedDirectory(path: string): Promise<void> {
try { try {
const metadata = await lstat(path); const metadata = await lstat(path);
@@ -691,16 +836,16 @@ async function assertPrivateManagedDirectory(path: string): Promise<void> {
} }
} }
async function assertSafeLockLeafIfPresent(lockPath: string): Promise<void> { async function assertSafeLockLeafIfPresent(lockPath: string, lockLabel: string): Promise<void> {
try { try {
const metadata = await lstat(lockPath); const metadata = await lstat(lockPath);
if (!metadata.isFile() || metadata.isSymbolicLink() || (metadata.mode & 0o077) !== 0) { if (!metadata.isFile() || metadata.isSymbolicLink() || (metadata.mode & 0o077) !== 0) {
throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unsafe.'); throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unsafe.`);
} }
} catch (error: unknown) { } catch (error: unknown) {
if (isCode(error, 'ENOENT')) return; if (isCode(error, 'ENOENT')) return;
if (error instanceof FleetReconcileError) throw error; if (error instanceof FleetReconcileError) throw error;
throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unavailable.'); throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unavailable.`);
} }
} }
@@ -710,6 +855,7 @@ async function assertLockOwnership(
inode: number, inode: number,
token: string, token: string,
failureCode: 'unsafe-lock' | 'lock-cleanup-failed', failureCode: 'unsafe-lock' | 'lock-cleanup-failed',
lockLabel: string,
): Promise<void> { ): Promise<void> {
try { try {
const metadata = await lstat(lockPath); const metadata = await lstat(lockPath);
@@ -720,24 +866,21 @@ async function assertLockOwnership(
metadata.dev !== device || metadata.dev !== device ||
metadata.ino !== inode metadata.ino !== inode
) { ) {
throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.'); throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`);
} }
const handle = await open(lockPath, constants.O_RDONLY | constants.O_NOFOLLOW); const handle = await open(lockPath, constants.O_RDONLY | constants.O_NOFOLLOW);
try { try {
const opened = await handle.stat(); const opened = await handle.stat();
const contents = await handle.readFile({ encoding: 'utf8' }); const contents = await handle.readFile({ encoding: 'utf8' });
if (opened.dev !== device || opened.ino !== inode || contents !== `${token}\n`) { if (opened.dev !== device || opened.ino !== inode || contents !== `${token}\n`) {
throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.'); throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`);
} }
} finally { } finally {
await handle.close(); await handle.close();
} }
} catch (error: unknown) { } catch (error: unknown) {
if (error instanceof FleetReconcileError) throw error; if (error instanceof FleetReconcileError) throw error;
throw new FleetReconcileError( throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership cannot be proven.`);
failureCode,
'The reconciliation lock ownership cannot be proven.',
);
} }
} }

View File

@@ -204,6 +204,53 @@ export async function prepareAgentEnvironmentProjection(
}; };
} }
/** A generated-only projection validated without touching any legacy/local/quarantine file. */
export interface PreparedGeneratedAgentEnvironmentProjection {
readonly mosaicHome: string;
readonly agentEnvDir: string;
readonly generatedPath: string;
readonly generated: string;
}
/**
* Validates ONLY the roster-derived generated projection for a recovery rebuild.
* Unlike {@link prepareAgentEnvironmentProjection}, this never reads, classifies,
* relocates, or quarantines the legacy `.env` / `.env.local` operator surface — it
* exists so `mosaic fleet regen` has no code path that can mutate anything except
* `<name>.env.generated`. The existing generated file, if present, must already be
* a private regular file.
*/
export async function prepareGeneratedAgentEnvironmentProjection(
options: AgentEnvironmentProjectionOptions,
): Promise<PreparedGeneratedAgentEnvironmentProjection> {
if (!AGENT_NAME.test(options.agentName)) {
throw new AgentEnvBoundaryError('unsafe-agent-name', 'MOSAIC_AGENT_NAME', options.agentName);
}
await validatePrivateProjectionDirectory(options.mosaicHome, options.agentEnvDir);
const generatedPath = join(options.agentEnvDir, `${options.agentName}.env.generated`);
const generated = renderGeneratedAgentEnvironment(options.generated);
await assertPrivateRegularFileIfPresent(generatedPath);
return {
mosaicHome: options.mosaicHome,
agentEnvDir: options.agentEnvDir,
generatedPath,
generated,
};
}
/**
* Applies a generated-only projection: writes ONLY `<name>.env.generated` atomically
* and touches nothing else. It never writes `.env.local`/`.env.quarantine` and never
* unlinks the legacy `.env` — the projection-only recovery guarantee is structural.
*/
export async function applyPreparedGeneratedAgentEnvironmentProjection(
prepared: PreparedGeneratedAgentEnvironmentProjection,
): Promise<string> {
await ensurePrivateProjectionDirectory(prepared.mosaicHome, prepared.agentEnvDir);
await writePrivateAtomically(prepared.generatedPath, prepared.generated);
return prepared.generatedPath;
}
/** /**
* Validates only the exact generated projection eligible for a roster delete. * Validates only the exact generated projection eligible for a roster delete.
* Local overrides, legacy input, and quarantine records are operator-retained and * Local overrides, legacy input, and quarantine records are operator-retained and