Compare commits
2 Commits
feat/791-p
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 9ddc6fbda8 | |||
| 31607a4af6 |
@@ -48,15 +48,19 @@ steps:
|
||||
# keep mode is a single cp-based path that must not depend on rsync), and that a
|
||||
# corrupt/empty/missing manifest aborts fail-closed leaving operator files
|
||||
# untouched (B2/B3). The rollback gate proves a mid-sync failure is rolled back
|
||||
# from the pre-update snapshot (B1). The migration matrix pins the v2→v3
|
||||
# contract-file semantics. Pure bash, no node_modules — runs early alongside
|
||||
# sanitization.
|
||||
# from the pre-update snapshot (B1). The durable-snapshot gate (#791 PR2) proves
|
||||
# the retained, operator-scoped pre-update backup is taken before any mutation
|
||||
# (0700/0600, secret never logged, retention-pruned) and that the post-sync
|
||||
# verify net restores any operator file a manifest bug lets the sync touch. The
|
||||
# migration matrix pins the v2→v3 contract-file semantics. Pure bash, no
|
||||
# node_modules — runs early alongside sanitization.
|
||||
upgrade-guard:
|
||||
image: *node_image
|
||||
commands:
|
||||
- apk add --no-cache bash rsync
|
||||
- bash packages/mosaic/framework/tools/quality/scripts/test-upgrade-manifest-guard.sh
|
||||
- bash packages/mosaic/framework/tools/quality/scripts/test-upgrade-rollback.sh
|
||||
- bash packages/mosaic/framework/tools/quality/scripts/test-upgrade-durable-snapshot.sh
|
||||
- bash packages/mosaic/framework/tools/quality/scripts/test-install-migration.sh
|
||||
|
||||
typecheck:
|
||||
|
||||
@@ -98,6 +98,39 @@ Expected results:
|
||||
that means the unit ran, not that an agent pane is live. Treat tmux
|
||||
`has-session`, `list-panes`, process tree, and logs as the liveness evidence.
|
||||
|
||||
## Recovery — rebuild generated env projections
|
||||
|
||||
Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
|
||||
deterministic projection of `roster.yaml` (the SSOT) that the launcher
|
||||
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
|
||||
wipes or diverges those projections, rebuild them from the roster with
|
||||
`mosaic fleet regen` — do NOT restart the affected unit first.
|
||||
|
||||
```bash
|
||||
mosaic fleet regen # dry-run (default): show create/rebuild plan per agent
|
||||
mosaic fleet regen --json # same plan, machine-readable
|
||||
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated on disk
|
||||
```
|
||||
|
||||
`regen` is projection-only and **never restarts an agent** — it has no path to
|
||||
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
|
||||
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
|
||||
only (never the projected `KEY=value` body). After `--write`, verify each unit
|
||||
resolves the intended values before restarting one unit at a time. The unit sets
|
||||
no `EnvironmentFile=` — `start-agent-session.sh` sources `.env.generated` itself —
|
||||
so verify the generated file directly and the launcher path, not a nonexistent
|
||||
`EnvironmentFile` property:
|
||||
|
||||
```bash
|
||||
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
|
||||
systemctl --user cat mosaic-agent@<name> | grep ExecStart
|
||||
systemctl --user restart mosaic-agent@<name>
|
||||
```
|
||||
|
||||
Full recovery runbook and the three-layer #791 protection model (manifest
|
||||
ownership → pre-update snapshot/restore → regen): see
|
||||
[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md).
|
||||
|
||||
## Release Preflight
|
||||
|
||||
Run this checklist before cutting or dogfooding a fleet release:
|
||||
|
||||
147
docs/guides/upgrade-safety-and-recovery.md
Normal file
147
docs/guides/upgrade-safety-and-recovery.md
Normal file
@@ -0,0 +1,147 @@
|
||||
# Upgrade Safety & Recovery
|
||||
|
||||
How Mosaic protects operator-owned configuration under `~/.config/mosaic` across
|
||||
framework upgrades, and how to recover if a projection is ever lost.
|
||||
|
||||
A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`,
|
||||
`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident
|
||||
this hardening addresses: an upgrade that silently overwrites or deletes a file
|
||||
the operator owns — credentials, personas, a roster, or a generated agent env —
|
||||
with no snapshot to fall back to.
|
||||
|
||||
Protection is layered. Each layer is independent; a later layer catches what an
|
||||
earlier one misses.
|
||||
|
||||
## Layer 1 — Manifest-owned sync (prevention)
|
||||
|
||||
The single source of truth for ownership is
|
||||
[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt).
|
||||
Both the bash installer and the TypeScript sync path resolve every path against
|
||||
this one file (parity is enforced by test), so they can never drift.
|
||||
|
||||
- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a
|
||||
`[framework]` glob matches and no `[operator]` carve-out overrides it.
|
||||
- **Unknown paths default to operator** (fail-safe): a file the manifest never
|
||||
anticipated is treated as operator-owned and is never pruned.
|
||||
- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that
|
||||
only ever iterates framework globs — operator and unknown paths are
|
||||
structurally unreachable by the prune.
|
||||
|
||||
Result: a correct upgrade cannot touch operator config at all.
|
||||
|
||||
## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)
|
||||
|
||||
Before **any** mutation, the installer snapshots the operator-owned surface that
|
||||
exists into:
|
||||
|
||||
```
|
||||
${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/
|
||||
```
|
||||
|
||||
- `0700` directories / `0600` files (`umask 077`, scoped and restored),
|
||||
outside `~/.config/mosaic` and outside any repo.
|
||||
- **Fail-open**: a snapshot failure warns but never aborts the upgrade it
|
||||
protects.
|
||||
- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5).
|
||||
|
||||
After the sync, a **verify net** compares each snapshot file against its target
|
||||
and restores (with a loud warning) any operator file the upgrade diverged or
|
||||
removed — a divergence means a manifest bug slipped through Layer 1.
|
||||
|
||||
Inspect and restore snapshots with the CLI:
|
||||
|
||||
```bash
|
||||
mosaic restore --list # dry-run: enumerate snapshots by timestamp
|
||||
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
|
||||
mosaic restore --from <ts> --dry-run # preview a specific restore without writing
|
||||
```
|
||||
|
||||
`mosaic restore` reports **counts and relative paths only** — it never emits file
|
||||
contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores
|
||||
are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf
|
||||
atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails
|
||||
closed rather than following out of the managed tree).
|
||||
|
||||
## Layer 3 — Regeneration from roster SSOT (recovery)
|
||||
|
||||
Some operator files are **derived** and do not need a byte-for-byte snapshot to
|
||||
recover — they can be rebuilt from their source of truth. The fleet's per-agent
|
||||
generated env projections are the prime case:
|
||||
|
||||
- `~/.config/mosaic/fleet/agents/<name>.env.generated` is a deterministic
|
||||
projection of `~/.config/mosaic/fleet/roster.yaml`.
|
||||
- The launcher (`start-agent-session.sh`, invoked by
|
||||
`mosaic-agent@<name>.service`) sources that generated projection to establish
|
||||
each agent's identity, runtime, model, and working directory. If it is missing
|
||||
or wrong, the agent cannot launch with its intended identity.
|
||||
|
||||
`mosaic fleet regen` rebuilds those projections from the roster SSOT:
|
||||
|
||||
```bash
|
||||
mosaic fleet regen # dry-run (default): show what would be rebuilt
|
||||
mosaic fleet regen --json # same, machine-readable
|
||||
mosaic fleet regen --write # rebuild the projections on disk
|
||||
```
|
||||
|
||||
- **Dry-run by default.** Nothing is written until you pass `--write`.
|
||||
- **Deterministic and idempotent** — the projection is a pure function of the
|
||||
roster, so repeated `--write` runs produce byte-identical files.
|
||||
- **Projection-only. It never restarts an agent.** Recovery order forbids
|
||||
restart-before-verify; `regen` has no path to systemd lifecycle at all.
|
||||
- **It rebuilds only `<name>.env.generated`** — it never writes, relocates, or
|
||||
deletes the operator-owned `.env` / `.env.local` surface.
|
||||
- It **validates the roster the same way `reconcile` does** (persona resolution
|
||||
and protected-class tool-policy match), so a hand-edited or corrupt roster is
|
||||
rejected rather than projected, and a `--write` takes the shared reconcile
|
||||
lock so it cannot race a concurrent reconcile.
|
||||
- Output is **paths and counts only** — the rendered `KEY=value` body is never
|
||||
echoed.
|
||||
|
||||
`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a
|
||||
recovered projection matches what a normal reconcile would have written.
|
||||
|
||||
## Recovery runbook — wiped `fleet/agents/*.env.generated`
|
||||
|
||||
If an upgrade (or a manual mistake) has left an agent without its generated
|
||||
projection, **do not restart the unit first** — a launch against a missing
|
||||
projection fails closed, and any stale state must be corrected before restart,
|
||||
not after.
|
||||
|
||||
1. **Prefer a snapshot restore if one exists** (byte-exact operator state):
|
||||
|
||||
```bash
|
||||
mosaic restore --list
|
||||
mosaic restore --from <UTC-timestamp>
|
||||
```
|
||||
|
||||
2. **Otherwise regenerate the derived projections from the roster SSOT:**
|
||||
|
||||
```bash
|
||||
mosaic fleet regen # confirm the plan (create vs rebuild per agent)
|
||||
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated
|
||||
```
|
||||
|
||||
3. **Verify each unit will resolve the intended runtime/workdir _before_ any
|
||||
restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal
|
||||
environment and `start-agent-session.sh` sources `.env.generated` itself, so
|
||||
verify the generated file directly and confirm the launcher path:
|
||||
|
||||
```bash
|
||||
# Confirm fleet/agents/<name>.env.generated exists and carries the intended
|
||||
# MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
|
||||
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
|
||||
# Confirm the unit launches the session script that reads it.
|
||||
systemctl --user cat mosaic-agent@<name> | grep ExecStart
|
||||
```
|
||||
|
||||
4. **Only then restart, one unit at a time:**
|
||||
|
||||
```bash
|
||||
systemctl --user restart mosaic-agent@<name>
|
||||
```
|
||||
|
||||
## See also
|
||||
|
||||
- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md)
|
||||
- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md)
|
||||
- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)
|
||||
@@ -214,3 +214,316 @@ Re-ran codex again; it found two more rollback-path gaps `set -E` cannot catch.
|
||||
- Gates: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1094 · HARD GATE 193/193 ·
|
||||
rollback 28/28 (was 14, +14 for D1/D2 with controls) · migration 21/21. shellcheck clean on new lines.
|
||||
No --no-verify. Committing FORWARD (no rebase of 34e55d4a/0a5e703a).
|
||||
|
||||
## Session 3 (2026-07-16) — PR1 MERGED, starting PR2 (durable snapshot + restore + secrev)
|
||||
PR1 (#802) squash-merged → main `32a0ffba`; issue #791 stays open (3-PR DAG umbrella). Independent Opus
|
||||
adversarial/security review APPROVED at head `af627e75` (Gitea RoR cmt 17892); lead ran rollback 28/28 +
|
||||
HARD GATE 193/193 green; CI #1877 green. PR2 UNBLOCKED.
|
||||
|
||||
PR2 branch: `feat/791-pr2-snapshot-restore` off `origin/main` 32a0ffba. Same treatment applies:
|
||||
tests-first red-first, independent review + durable Gitea Reviewer-of-Record comment BEFORE MS-LEAD runs
|
||||
the queue guard/merge. Report PR2 number + exact head when ready. PR body: `Part of #791` (NOT Fixes).
|
||||
|
||||
### PR2 scope (ratified §3/§5 of design doc, Mos-approved — do NOT re-litigate)
|
||||
- **(a) Durable pre-update snapshot** to `${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-ts>/`
|
||||
— OUTSIDE ~/.config/mosaic and any repo. Perms dir 0700 / files 0600 (umask 077 + explicit chmod).
|
||||
Scope = operator-owned surface that EXISTS (operatorReserved paths), not the framework tree. Taken
|
||||
BEFORE any mutation. Retention N=5 (`MOSAIC_BACKUP_RETENTION`), prune older.
|
||||
- **Post-sync verify + selective restore**: diff operator surface vs snapshot; (b) should never touch
|
||||
operator paths, so ANY diff = manifest bug → restore affected paths + warn loudly. (a) catches a (b) miss.
|
||||
- **`mosaic restore`** (TS CLI): `--list` (default, dry-run) enumerates snapshots by ts; `--from <ts>`
|
||||
restores over operator surface, confirmation-gated. Counts/paths only.
|
||||
- **Secret-safety (secrev)**: snapshot/restore NEVER emit file contents; only paths/counts. Tests assert
|
||||
0700/0600 AND that a secret value seeded in tools/_lib/credentials.json never appears in any output.
|
||||
|
||||
### PR2 implementation status (2026-07-16, ready-for-review)
|
||||
All three tasks implemented, red-first proven, unit-green:
|
||||
- **Task #10 — durable snapshot (install.sh)**: `backup_root()`/`enumerate_operator_files()`/
|
||||
`prune_durable_snapshots()`/`make_durable_snapshot()` wired into keep-mode main() after `manifest_load`,
|
||||
before any mutation. umask 077 + explicit chmod 700/600. UTC ts, collision suffix. FAIL-OPEN (a backup
|
||||
failure never aborts the upgrade it protects). Retention `MOSAIC_BACKUP_RETENTION` (default 5), in-place
|
||||
`sort -r -o` prune (no `mv` — stays inside the rsync-absent coreutils whitelist).
|
||||
- **Task #11 — post-sync verify net (install.sh)**: `verify_operator_surface()` runs after sync (trap
|
||||
disarmed), `cmp -s` each snapshot file vs target; restores any diverged/missing operator file + warns
|
||||
loudly (a divergence = manifest bug). VERIFY-NET wired before `cleanup_snapshot`.
|
||||
- **Task #12 — `mosaic restore` (TS)**: `src/commands/restore.ts` + co-located spec (19 tests).
|
||||
`--list` default (dry-run enumerate), `--from <ts>` confirmation-gated restore, `--dry-run`, `--yes`/
|
||||
`MOSAIC_ASSUME_YES`. Injectable `confirm` for testability (proceed/decline/env-bypass covered). Restored
|
||||
files forced 0600. Registered in `cli.ts`. Path convention mirrors install.sh `backup_root()`.
|
||||
- **CI**: `.woodpecker/ci.yml` upgrade-guard runs the new `test-upgrade-durable-snapshot.sh` gate.
|
||||
- **Gates green**: typecheck ✓ lint ✓ format:check ✓ · full mosaic vitest 1241 (+5) ·
|
||||
durable-snapshot 26/26 · manifest-guard 193/193 · rollback 28/28 · migration 21/21.
|
||||
Est. new-code coverage ≈93% (only the interactive readline default + process.exit-on-error uncovered).
|
||||
- Regression fixed: PR2's `date`/`sort`/`mv` broke the rsync-absent manifest-guard PATH whitelist →
|
||||
made date/sort fail-open, replaced `mv` with in-place `sort -o`, added `date sort` to the test whitelist
|
||||
+ isolated `XDG_STATE_HOME`. All 193 manifest-guard assertions green under restricted PATH.
|
||||
- Codex code-review + security-review (secrev) run on the uncommitted diff before commit.
|
||||
|
||||
### PR2 review round 1 — findings + remediations (2026-07-16, pre-PR)
|
||||
Codex code-review returned **request-changes** (1 blocker + 3 should-fix); Codex security-review returned
|
||||
**high** (1 high + 1 medium). Deduped to 5 distinct defects, ALL legitimate, ALL fixed FORWARD, each with
|
||||
a red-first regression test whose control neuters exactly the guard under test:
|
||||
|
||||
- **A · BLOCKER — verify net undid the legacy bin/ migration (install.sh).** On a pre-v2 install `bin/**`
|
||||
is operator-classified, so the durable snapshot captured it; `run_migrations()` deletes bin/ on purpose,
|
||||
but `verify_operator_surface()` then saw it "missing" and healed it back — the migration would be silently
|
||||
undone forever once the version stamps. **Fix:** `MIGRATION_REMOVED_PATHS[]` recorded by run_migrations
|
||||
(`bin`,`rails`) + `is_migration_removed()` skip in the verify loop (`# MIGRATION-SKIP-GUARD`).
|
||||
**Test:** Part 6 — v1 fixture with bin/; shipped keeps it removed + stamps v3; control (guard stripped)
|
||||
wrongly restores bin/tool.sh.
|
||||
- **B · HIGH (CWE-59) — restore/verify wrote secrets THROUGH a symlink (install.sh + restore.ts).** An
|
||||
attacker swapping an operator path (e.g. tools/_lib/credentials.json) for a symlink after the snapshot
|
||||
would make `cp`/`copyFileSync` write the snapshot's secret out through the link. **Fix (bash):** refuse a
|
||||
symlinked ancestor (`has_symlinked_parent`), drop a symlinked leaf before restore
|
||||
(`# SYMLINK-LEAF-GUARD`). **Fix (TS):** reuse audited `secure-file.ts` — `assertCanonicalContainment`
|
||||
+ `ensureManagedDirectory` on every dst, open the leaf `O_NOFOLLOW|O_CREAT|O_TRUNC` 0600 (ELOOP =
|
||||
fail-closed). **Tests:** Part 7 (shipped leaves external exfil target untouched, restores a real 0600
|
||||
file; control leaks the secret through the link) + restore.spec symlinked-leaf/ancestor cases (red-first).
|
||||
- **C · MEDIUM/should-fix (CWE-22) — `--from` traversal escaped the backup root (restore.ts).**
|
||||
`join(root, from)` accepted `../poison`. **Fix:** validate the selector against
|
||||
`^\d{8}T\d{6}Z(?:-\d+)?$`, build exactly `join(root,'pre-update-'+ts)`, `lstat` (reject symlinked snap
|
||||
dir). **Test:** restore.spec `it.each` of 6 malformed selectors + `--from ../poison` fail-closed (red-first).
|
||||
- **D · should-fix — verify `mkdir -p` unguarded under set -e (install.sh).** A parent replaced by a
|
||||
regular file aborted the installer before the recovery pointer printed. **Fix:** guard `mkdir -p`, warn
|
||||
+ `continue` on failure (keeps healing remaining files).
|
||||
- **E · should-fix — snapshot `umask 077` leaked process-global (install.sh).** Later sync copies/dirs
|
||||
inherited 0600/0700. **Fix:** save `old_umask`, restore on EVERY return path (`# UMASK-RESTORE-NORMAL`).
|
||||
**Test:** Part 8 — synced framework file is 0644 while the secret backup stays 0600; control (restore
|
||||
stripped) makes the synced file 0600.
|
||||
|
||||
**Full gate suite re-run after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic
|
||||
vitest **1252** · restore.spec **30** · durable-snapshot **41** · manifest-guard 193 · rollback 28 ·
|
||||
migration 21. shellcheck clean on all new lines; new test markers mirror the existing `# VERIFY-NET`
|
||||
anchor convention. NOTE: codex self-review does NOT satisfy the independent-review gate — an independent
|
||||
(author≠reviewer) review + durable Gitea Reviewer-of-Record comment is still required before MS-LEAD merges.
|
||||
|
||||
## Session 4 (2026-07-16) — PR2 MERGED, PR3 built (fleet regen — recovery layer)
|
||||
PR2 (#811) squash-merged → main `31607a4a`; issue #791 stays open (final PR of the 3-PR DAG). Independent
|
||||
exact-head RoR at `d12c5f78` APPROVE (Gitea cmt 17904); #1882 green; busybox-portable Part 7 control fix
|
||||
verified in-Alpine. PR3 UNBLOCKED.
|
||||
|
||||
PR3 branch: `feat/791-pr3-fleet-regen` off `origin/main` 31607a4. Same discipline: tests-first red-first,
|
||||
independent review + durable Gitea RoR BEFORE MS-LEAD runs the queue guard/merge. PR body `Part of #791`.
|
||||
|
||||
### PR3 scope (ratified §4/§7 of design doc) — `mosaic fleet regen`
|
||||
Projection-only recovery command: rebuilds each `fleet/agents/<name>.env.generated` from `roster.yaml`
|
||||
(SSOT). Dry-run default; `--write` applies; `--json` machine output. Structural guarantee: NO code path to
|
||||
systemd lifecycle — **never restarts an agent**. Single-SSOT: reuses `projectRosterV2AgentGeneratedEnv`
|
||||
(extracted, shared with the reconciler apply path) so regen and reconcile cannot drift. Secrev: paths +
|
||||
counts only, never the rendered KEY=value body.
|
||||
|
||||
New files: `commands/fleet-regen-command.ts` (+ `.spec.ts`), guide `docs/guides/upgrade-safety-and-recovery.md`
|
||||
(three-layer model: PR1 manifest ownership → PR2 snapshot/restore → PR3 regen; do-NOT-restart-before-verify
|
||||
runbook), regen reference added to `docs/guides/fleet-local-canary.md`. Wired in `commands/fleet.ts`.
|
||||
|
||||
### Independent review (3 reviewers: subagent code-reviewer + codex code-review + codex security) → 4 fixes, red-first
|
||||
- **A · BLOCKER (codex) — regen mutated/deleted legacy operator env.** `applyPreparedAgentEnvironmentProjection`
|
||||
also writes `.env.local`/`.env.quarantine` and unlinks legacy `.env`. Violated projection-only contract.
|
||||
**Fix:** NEW generated-only boundary primitives `prepareGeneratedAgentEnvironmentProjection` +
|
||||
`applyPreparedGeneratedAgentEnvironmentProjection` (write ONLY `<name>.env.generated`). regen now has no
|
||||
code path that touches `.env`/`.env.local`/`.env.quarantine`. **Test:** projection-only leaves legacy `.env`
|
||||
verbatim, no local/quarantine fabricated.
|
||||
- **B · should-fix (codex + subagent + security) — partial write on mid-loop failure.** Interleaved
|
||||
prepare/apply left earlier agents written when a later agent failed prepare. **Fix:** PREPARE ALL agents
|
||||
before writing ANY (mirrors reconciler `defaultPrepareProjections`). **Test:** 2nd agent's projection
|
||||
pre-seeded 0644 → prepare rejects → coder0 NOT written, exit 1.
|
||||
- **C · subagent — semantic-validation bypass.** Default readRoster skipped `validateRosterV2Semantics`, so
|
||||
a tampered protected-class `tool_policy` would be silently projected. **Fix:** default readRoster now runs
|
||||
`validateRosterV2Semantics` (persona resolution + protected-class match), rolesDir/overrideDir defaults
|
||||
mirroring the reconciler. **Test:** merge-gate agent w/ tool_policy=code → fails closed, no write.
|
||||
- **D · MEDIUM (codex security, CWE-362) — concurrent-reconcile race.** regen `--write` wrote without the
|
||||
reconcile lock. **Fix:** `--write` acquires `acquirePrivateReconcileLock(mosaicHome)` for the whole
|
||||
read-prepare-apply sequence, released in `finally`; dry-run stays lock-free. **Test:** pre-held lock →
|
||||
regen fails closed, no write.
|
||||
|
||||
**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1265**
|
||||
(regen spec 13, incl. 4 new red-first regressions). NOTE: codex self-review does NOT satisfy the
|
||||
independent-review gate — an independent (author≠reviewer) review + durable Gitea RoR is still required
|
||||
before MS-LEAD merges. STOP at PR-open for MS-LEAD's exact-head review; do NOT self-merge.
|
||||
|
||||
## Session 5 — PR3 review round 2 (finding L + M1/M2/M3), red-first fixes
|
||||
|
||||
Second review pass on the lock-cleanup plumbing surfaced one round-1 residual (L) and three round-2
|
||||
findings (M1 blocker, M2/M3 should-fix). All fixed red-first (RED proven per-finding, then GREEN).
|
||||
|
||||
- **L · should-fix (codex r1) — mutation-lock release swallowed unlink failures.** regen's
|
||||
`acquirePrivateRosterMutationLock` release copied CRUD's `unlink().catch(()=>{})`, hiding a stale
|
||||
`roster.yaml.mutation.lock`. **Fix:** its release PROPAGATES the unlink fault (finding-J stale-lock
|
||||
warning then fires for this lock too). **Test:** acquire real lock, `rm` it, assert `release()` rejects.
|
||||
- **M1 · BLOCKER (codex r2) — replacement-lock race.** The propagating release from L did an
|
||||
UNCONDITIONAL `unlink(lockPath)` without proving ownership. If the lock is cleared + re-created by
|
||||
another writer mid-op, regen deletes the STRANGER's live lock → a third writer enters → mutual
|
||||
exclusion defeated. **Fix (reuse, not reimplement):** generalized the reconciler's ownership-proving
|
||||
lock body into shared `acquirePrivateManagedRosterLock(mosaicHome, lockLeaf, busyMessage, openLock)`;
|
||||
`acquirePrivateReconcileLock` delegates to it (behavior-identical: same leaf/codes/messages), and a NEW
|
||||
hardened `acquirePrivateRosterMutationLock` (now in fleet-reconciler.ts, leaf `roster.yaml.mutation.lock`)
|
||||
records dev/ino + ownership token and RE-PROVES ownership (`assertLockOwnership`) before unlinking —
|
||||
fails closed as `lock-cleanup-failed` if replaced. Removed the crud-based export; reverted
|
||||
`acquireMutationLock` (fleet-agent-crud.ts) to its original inline empty-file/swallowing-release form
|
||||
(CRUD behavior intentionally unchanged). Compatibility: CRUD empty-file `wx` and regen tokened `wx`
|
||||
contend on the same path but never co-own (wx winner owns; loser → concurrent-mutation), so the token
|
||||
is only ever read back by the same regen invocation. **Test:** acquire, `rm`+recreate lock (new inode),
|
||||
assert `release()` rejects AND the replacement survives (not unlinked).
|
||||
- **M2 · should-fix (codex r2) — acquire-unwind fault dropped.** The acquire-failure catch discarded
|
||||
`releaseFleetLocks`' return (a possible fault on the already-held first lock). **Fix:** capture and
|
||||
augment — `const releaseFault = await releaseFleetLocks(releases); throw augmentWithLockCleanupFault(error, releaseFault);`
|
||||
(symmetric to finding J). **Test:** mutation lock acquires w/ faulting release + reconcile acquire
|
||||
throws → thrown error mentions stale/lock, nothing written.
|
||||
- **M3 · should-fix (codex r2 + subagent REQUEST-CHANGES) — cleanup warning named only reconcile lock.**
|
||||
Finding L made the mutation-lock release fault reachable, so the `cleanup` marker can originate from
|
||||
EITHER lock. **Fix:** `formatFleetRegenReport`'s WARNING now names BOTH `roster.yaml.mutation.lock` and
|
||||
`roster.yaml.reconcile.lock`, matching `augmentWithLockCleanupFault`. **Test:** fault the mutation-lock
|
||||
release specifically → report names both lock files.
|
||||
|
||||
**Refactor note (no cycle):** neither fleet-reconciler nor fleet-agent-crud imports the other; regen
|
||||
imports lock acquirers from fleet-reconciler and the projection mapping from fleet-reconciler. The two
|
||||
reconcile-lock reviewers reconciled: independent reviewer validated acquire-time empty-file compatibility
|
||||
(preserved), codex flagged RELEASE-time replacement race (closed by ownership proof) — non-contradictory.
|
||||
|
||||
**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1275**
|
||||
(regen spec 23, incl. 7 red-first lock regressions E/F/G/K/L/M1/M2/M3). RED proven per-finding by
|
||||
temporary revert before re-applying each fix. Independent (author≠reviewer) review of M1/M2/M3 + codex
|
||||
code/security re-run in flight. STOP at PR-open for MS-LEAD's exact-head review + durable Gitea RoR; do
|
||||
NOT self-merge; #791 umbrella stays OPEN; PR body `Part of #791`.
|
||||
|
||||
### Round 3 review (after M1/M2/M3) — independent review PASS + codex residual-TOCTOU disposition
|
||||
Three reviewers on the post-M1/M2/M3 head:
|
||||
- **Independent (subagent, author≠reviewer) — PASS.** Verified M1/M2/M3 all correctly fixed; "never
|
||||
restarts" is STRUCTURAL (runner never referenced in executable code); no secrets; no deadlock (only
|
||||
regen holds both locks); tests meaningful (assert inode preservation + exact lock-file names). Raised:
|
||||
- **should-fix #1 (fixed, red-first):** generalizing the lock helper left `assertSafeLockLeafIfPresent`/
|
||||
`assertLockOwnership` hardcoding "reconciliation lock" in thrown messages → a MUTATION-lock fault
|
||||
misreported as the reconcile lock, undercutting M3's accurate-diagnosis goal. **Fix:** thread
|
||||
`lockLabel = fleet/<leaf>` through both helpers + the generic lock-io messages, so every fault names
|
||||
the actual lock file. Red-first: strengthened the M1 test to assert `/roster\.yaml\.mutation\.lock/`
|
||||
(RED: got "reconciliation lock"; GREEN after). Also resolves nit #3 (generic-message drift).
|
||||
- **nit #2 (fixed):** `FleetRegenResult.cleanup` JSDoc still said "the shared reconcile lock"; now names
|
||||
both locks (regen holds both).
|
||||
- **nit #4 (fixed):** removed the redundant duplicate `assertLockOwnership` call before unlink
|
||||
(pre-existing in merged main; harmless but dead — dropped since the fn was already being touched).
|
||||
- **Codex security — clean (risk: none).** Validates roster semantics, constrains env values, no shell
|
||||
eval, no secret output, generated-only writes, serialized against both locks.
|
||||
- **Codex code — request-changes, 1 "blocker": residual check-then-unlink TOCTOU.** Between the final
|
||||
`assertLockOwnership` and the path-based `unlink`, an external actor could vacate our inode and a new
|
||||
writer grab the path, so the unlink deletes the stranger's lock. **Disposition: documented known
|
||||
limitation, NOT fixed in PR3.** Rationale: (1) byte-identical to the MERGED, shipped reconcile-lock
|
||||
release on origin/main (fleet-reconciler.ts L654-659) — not introduced here; (2) UNREACHABLE within the
|
||||
`wx` writer protocol — no Mosaic writer removes a lock it doesn't own (wx fails EEXIST while our inode
|
||||
exists), so only external interference can vacate our inode in the sub-instruction window; (3) the
|
||||
ownership guard DOES close the reachable case (stale-lock reaper/operator cleared our lock + another
|
||||
writer took it BEFORE release began → fail closed, don't delete stranger's lock); (4) the true atomic
|
||||
fix — fd-held advisory lock (flock/lockf) adopted by ALL fleet writers (CRUD + reconcile + regen) — is
|
||||
a cross-cutting mechanism change touching merged CRUD + reconciler, out of scope for a projection-only
|
||||
recovery PR. Documented honestly in the acquirer doc + M1 test comment. **The binding independent
|
||||
review did NOT treat this as a blocker.** Recommendation to MS-LEAD: proceed to PR-open + spin a
|
||||
SEPARATE follow-up issue for the fd-advisory-lock migration; MS-LEAD adjudicates scope at exact-head
|
||||
review (merge authority).
|
||||
|
||||
**Gates after round-3 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
|
||||
**1275** (regen spec 23). Fresh codex code re-run in flight to confirm no NEW issues from the label fix.
|
||||
|
||||
---
|
||||
|
||||
## Session 6 — Round 4/5 convergence (stranded-lock robustness)
|
||||
|
||||
**Two independent reviewers converged on the SAME should-fix** on the init-failure cleanup path,
|
||||
strengthening confidence it was real:
|
||||
|
||||
- **Codex code-review-5 — 0 blockers, 1 should-fix.** "Stat failure after lock creation strands the new
|
||||
lock." When `handle.stat()` ITSELF fails right after the `wx` create (transient EIO/EBADF), `created`
|
||||
is `undefined`, so `removeOwnedLockLeafBestEffort` had `if (!created) return;` → no cleanup → the
|
||||
just-created `roster.yaml.mutation.lock`/`reconcile.lock` is stranded, permanently blocking future
|
||||
regen + CRUD. (Notably NO blocker, and the TOCTOU is no longer flagged in code-review as of r5.)
|
||||
- **Independent delta reviewer (author≠reviewer, pr-review-toolkit) — no blockers, same should-fix.**
|
||||
Independently flagged the identical `!created` gap; validated FIX 1 (label threading — no call site
|
||||
missed, codes unchanged, no test depended on old text) and FIX 2 (dev/ino-guarded cleanup, best-effort,
|
||||
happy-path release reuses captured dev/ino) as correct. Suggested an unconditional best-effort unlink
|
||||
in the `!created` branch; I took the **safer** variant below.
|
||||
- **Codex security-review-5 — 0 crit / 0 high / 1 medium.** The single medium is the SAME residual
|
||||
check-then-unlink TOCTOU already dispositioned in round 3 (its own remediation = "migrate every writer
|
||||
to an fd-held advisory lock" = the follow-up issue). No new security finding. No secrets.
|
||||
|
||||
**Fix (red-first, safer than an unconditional unlink):** thread the persisted random `token` into
|
||||
`removeOwnedLockLeafBestEffort`. Two independent ownership proofs now: primary dev/ino (unchanged), and a
|
||||
**fallback** when the post-create stat failed — read the leaf and unlink ONLY if its content equals our
|
||||
`randomUUID()` token. Only OUR lock carries that token, so a CRUD (empty) or differently-tokened
|
||||
replacement is never deleted. `tokenPersisted` guards passing the token (only after `writeFile` lands).
|
||||
Doubly-degenerate case (stat fails AND token write never landed) leaves the lock in place rather than
|
||||
risk deleting a stranger's file — requires two independent fs faults on a just-created fd; documented.
|
||||
|
||||
- **Red-first proof:** new test `does not strand the lock file when the post-create stat itself fails`
|
||||
injects a real `wx` create + a Proxy handle whose `stat()` rejects (writeFile/close succeed), asserts
|
||||
`exists(lockPath) === false`. RED before fix (`expected true to be false` — lock stranded); GREEN after.
|
||||
- **Also fixed (delta nit #3):** `fleet-regen-command.ts` `acquireRosterMutationLock` JSDoc said "CRUD's
|
||||
private lock"; the default is the reconciler's hardened ownership-proving acquirer for the same
|
||||
`fleet/roster.yaml.mutation.lock` path. Corrected.
|
||||
- **PR-description note (delta nit #2):** FIX 1 also collapsed a pre-existing duplicate back-to-back
|
||||
`assertLockOwnership` call in the release closure (identical args, no intervening logic) into one — a
|
||||
no-op simplification of merged code, not a behavior change. Called out so a future reader doesn't
|
||||
wonder if the duplicate had a purpose.
|
||||
|
||||
**Gates after round-4 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
|
||||
**1277** (regen spec now 25: +1 stat-failure stranded-lock regression). Residual TOCTOU still deferred to
|
||||
the fd-advisory-lock follow-up issue; MS-LEAD adjudicates scope at exact-head review (merge authority).
|
||||
|
||||
---
|
||||
|
||||
## Session 6 — Round 6 (persona-root wiring)
|
||||
|
||||
**Codex code-review-6 — 0 blockers, 1 should-fix (NEW, distinct from the lock work).** "Forward
|
||||
configured persona directories to regen." `registerFleetRegenCommand` was registered at
|
||||
`fleet.ts:2069` with only `{ runner, mosaicHome }`, discarding `deps.reconcileDeps.rolesDir` /
|
||||
`overrideDir`. The regen command ALREADY has those seams (validates roster semantics via
|
||||
`validateRosterV2Semantics({ rolesDir, overrideDir })`, defaulting to `<mosaicHome>/fleet/roles{,.local}`),
|
||||
but the top-level wiring never forwarded the configured roots. **Impact:** in a deployment with custom
|
||||
persona roots, `fleet reconcile` (which honors the overrides) would ACCEPT a roster while `fleet regen`
|
||||
REJECTS the same roster (persona resolution against the wrong default dir) — blocking the recovery
|
||||
command and violating the documented "resolves personas the SAME way reconcile does" contract.
|
||||
|
||||
**Fix (red-first):** forward `rolesDir`/`overrideDir` from `deps.reconcileDeps` into
|
||||
`registerFleetRegenCommand` at `fleet.ts:2069`. Red-first test `forwards configured persona roots
|
||||
(rolesDir/overrideDir) from reconcileDeps into regen`: seeds personas ONLY under a custom root, leaves
|
||||
the default `<home>/fleet/roles` empty, registers with `reconcileDeps: { rolesDir, overrideDir }`, and
|
||||
requires `fleet regen` to SUCCEED. RED before fix (`expected 1 not to be 1` — regen validated against the
|
||||
empty default and exited 1); GREEN after.
|
||||
|
||||
**Codex security-review-6 — 0 crit / 0 high / 1 medium.** Same residual check-then-unlink TOCTOU, now
|
||||
noted at BOTH the release closure and the init-cleanup path; remediation = fd-held advisory lock across
|
||||
all writers = the SAME deferred follow-up item. No new security finding, no secrets.
|
||||
|
||||
**Independent confirmation review of the token-fallback fix (Session 6/round 4) — PASS, no findings.**
|
||||
All 7 verification points confirmed; reviewer mechanically reverted `removeOwnedLockLeafBestEffort` to
|
||||
the pre-fix `if (!created) return;` and re-ran the new test → RED (`expected true to be false`),
|
||||
confirming the test genuinely pins the fix; restored after. No lint/type issues; doc-comment accurate.
|
||||
|
||||
**Gates after round-6 fix (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
|
||||
**1278** (regen spec now 26: +1 persona-root wiring regression).
|
||||
|
||||
---
|
||||
|
||||
## Session 6 — Round 7 convergence (review CLOSED for PR-open)
|
||||
|
||||
- **Codex code-review-7 — 0 blockers, 1 should-fix = the residual TOCTOU** (previously a "blocker" in r3,
|
||||
dropped in r4/r5, now re-surfaced as a should-fix). **Codex security-review-7 — 0 crit / 0 high /
|
||||
1 medium = the SAME residual TOCTOU.** Codex has CONVERGED: the only remaining finding across both
|
||||
streams is that one race, whose own remediation is "fd-held advisory lock shared by all fleet writers"
|
||||
= the deferred follow-up. No new distinct finding; the wiring fix introduced nothing.
|
||||
- **Independent confirmation review of the persona-root wiring fix — PASS, no findings.** Reviewer
|
||||
mechanically reverted the two forwarded lines → RED (`Roster v2 agent "coder0" class "code" does not
|
||||
resolve to a readable persona` → exit 1), restored → GREEN (26 regen + 204 fleet tests). Confirmed the
|
||||
optional-chaining fallback preserves default-deployment behavior and no type/lint issue.
|
||||
|
||||
**Review disposition for PR-open:** ALL actionable findings fixed red-first across rounds 3–6 (label
|
||||
threading, stranded-lock on init failure, stat-failure strand, persona-root wiring). The residual
|
||||
check-then-unlink TOCTOU is the ONLY open item and is DEFERRED to a follow-up issue (fd-advisory-lock
|
||||
migration across CRUD + reconcile + regen) — byte-identical to merged origin/main's reconcile-lock
|
||||
release, unreachable within the `wx` writer protocol (no Mosaic writer removes a lock it doesn't own;
|
||||
only external `rm`/a stale-lock reaper can vacate the inode mid-release), and its true fix is a
|
||||
cross-cutting mechanism change out of scope for a projection-only recovery PR. Two independent human-agent
|
||||
reviews (author≠reviewer) treated it as non-blocking. MS-LEAD adjudicates scope at exact-head review
|
||||
(merge authority); recommendation = proceed to PR-open + spin the follow-up issue.
|
||||
|
||||
**Final gates (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1278**
|
||||
(regen spec 26). No secret values in any snapshot/projection/report output (counts + paths only). Regen
|
||||
NEVER issues a lifecycle/restart call (load-bearing recordingRunner gate). STOP at PR-open for MS-LEAD's
|
||||
exact-head review + durable Reviewer-of-Record before any merge; do NOT self-merge.
|
||||
|
||||
@@ -109,6 +109,225 @@ restore_snapshot() {
|
||||
}
|
||||
cleanup_snapshot() { [[ -n "$SNAPSHOT_DIR" && -d "$SNAPSHOT_DIR" ]] && rm -rf "$SNAPSHOT_DIR"; SNAPSHOT_DIR=""; }
|
||||
|
||||
# ─── durable operator-config snapshot (#791 PR2) ─────────────────────────────
|
||||
# A SECOND, independent safety layer, distinct from SNAPSHOT_DIR above:
|
||||
# • SNAPSHOT_DIR is ephemeral (/tmp, deleted on success) and mirrors the WHOLE
|
||||
# target for CRASH rollback if the sync aborts mid-write.
|
||||
# • DURABLE_SNAPSHOT_DIR is RETAINED, holds only the operator-owned surface, and
|
||||
# lives OUTSIDE the framework tree and any repo. It exists for the failure the
|
||||
# crash-rollback cannot see: a sync that finishes "successfully" yet a
|
||||
# manifest/logic bug let it modify an operator file. verify_operator_surface()
|
||||
# (post-sync) heals from it; `mosaic restore` recovers from it days later.
|
||||
# Path convention is mirrored in packages/mosaic/src/commands/restore.ts — keep
|
||||
# the two in sync (there is no shared code across the bash/TS boundary).
|
||||
DURABLE_SNAPSHOT_DIR=""
|
||||
backup_root() { printf '%s/mosaic/backups' "${XDG_STATE_HOME:-$HOME/.local/state}"; }
|
||||
|
||||
# Relative paths that a migration INTENTIONALLY removes from the target (e.g. the
|
||||
# legacy bin/ tree). Such a path is operator-classified by the manifest (unknown⇒
|
||||
# operator), so the durable snapshot captures it — but its post-migration absence
|
||||
# is correct, NOT a manifest bug. run_migrations() records each removal here so
|
||||
# verify_operator_surface() does not "heal" it back and silently undo the
|
||||
# migration (which would then be skipped forever once the version is stamped).
|
||||
MIGRATION_REMOVED_PATHS=()
|
||||
|
||||
# True (0) if $1 (a path relative to TARGET_DIR) equals or lives under a path a
|
||||
# migration deliberately removed this run.
|
||||
is_migration_removed() {
|
||||
local rel="$1" removed
|
||||
for removed in ${MIGRATION_REMOVED_PATHS[@]+"${MIGRATION_REMOVED_PATHS[@]}"}; do
|
||||
[[ -n "$removed" ]] || continue
|
||||
[[ "$rel" == "$removed" || "$rel" == "$removed"/* ]] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# True (0) if any parent directory of $1 (relative to TARGET_DIR) is a symlink.
|
||||
# Restoring THROUGH a symlinked ancestor would let cp write snapshot contents —
|
||||
# possibly secrets — outside the target (CWE-59), so the verify net refuses it.
|
||||
has_symlinked_parent() {
|
||||
local rel="$1" dir p seg
|
||||
dir="$(dirname "$rel")"
|
||||
[[ "$dir" == "." ]] && return 1
|
||||
p="$TARGET_DIR"
|
||||
local IFS='/'
|
||||
for seg in $dir; do
|
||||
[[ -n "$seg" ]] || continue
|
||||
p="$p/$seg"
|
||||
[[ -L "$p" ]] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Emit (NUL-delimited, into file $1) the operator-owned relative paths that exist
|
||||
# under TARGET_DIR, classified via the shared manifest (deny-wins; unknown⇒
|
||||
# operator). Returns non-zero if the filesystem walk itself failed — we must
|
||||
# NEVER snapshot from a truncated scan (a `< <(find …)` process substitution
|
||||
# would hide that error; capture-then-check does not — cf. #791 blocker-D1).
|
||||
enumerate_operator_files() {
|
||||
local out="$1" scan abs rel
|
||||
scan="$(mktemp)"
|
||||
if ! find "$TARGET_DIR" -type f -print0 > "$scan"; then
|
||||
rm -f "$scan"
|
||||
return 1 # OP-SCAN-GUARD
|
||||
fi
|
||||
: > "$out"
|
||||
while IFS= read -r -d '' abs; do
|
||||
rel="${abs#"$TARGET_DIR"/}"
|
||||
# Not operator config: version marker and any VCS metadata.
|
||||
case "$rel" in .framework-version|.git|.git/*) continue ;; esac
|
||||
manifest_is_framework "$rel" || printf '%s\0' "$rel" >> "$out"
|
||||
done < "$scan"
|
||||
rm -f "$scan"
|
||||
}
|
||||
|
||||
# Retain only the newest MOSAIC_BACKUP_RETENTION (default 5) snapshots. The
|
||||
# pre-update-<UTC-ts> names sort lexicographically = chronologically, so a
|
||||
# reverse sort is newest-first. Pruning failures are non-fatal (they only leave
|
||||
# extra old backups); the enclosing find's status is still honored, not swallowed.
|
||||
prune_durable_snapshots() {
|
||||
local root keep list d i=0
|
||||
root="$(backup_root)"
|
||||
keep="${MOSAIC_BACKUP_RETENTION:-5}"
|
||||
[[ "$keep" =~ ^[0-9]+$ ]] && (( keep >= 1 )) || keep=5
|
||||
list="$(mktemp)"
|
||||
if ! find "$root" -maxdepth 1 -type d -name 'pre-update-*' > "$list"; then
|
||||
rm -f "$list"; return 0
|
||||
fi
|
||||
# Newest-first ordering needs `sort` (`-o` writes back in place — no `mv`
|
||||
# dependency); if it is somehow unavailable, leave the backups untouched rather
|
||||
# than risk pruning in an undefined order.
|
||||
if ! LC_ALL=C sort -r -o "$list" "$list" 2>/dev/null; then
|
||||
rm -f "$list"; return 0
|
||||
fi
|
||||
while IFS= read -r d; do
|
||||
[[ -n "$d" ]] || continue
|
||||
i=$((i + 1))
|
||||
(( i > keep )) && rm -rf "$d"
|
||||
done < "$list"
|
||||
rm -f "$list"
|
||||
}
|
||||
|
||||
# Take the durable pre-update snapshot BEFORE any mutation. Fail-OPEN: the durable
|
||||
# snapshot is a recovery bonus on top of the manifest (which already keeps the
|
||||
# sync out of operator paths) and the crash-rollback — so an un-writable backup
|
||||
# location warns and continues rather than blocking the upgrade. Everything it
|
||||
# creates is private (umask 077 + explicit 0700 dirs / 0600 files): the snapshot
|
||||
# mirrors operator config, which may hold secrets, and must never be world-readable.
|
||||
make_durable_snapshot() {
|
||||
is_existing_install || return 0
|
||||
local root ts dir list rel src dst count=0 old_umask
|
||||
root="$(backup_root)"
|
||||
# Fail-open if we cannot even stamp a timestamp: the durable snapshot is a
|
||||
# recovery bonus and must never be the thing that aborts an upgrade.
|
||||
ts="$(date -u +%Y%m%dT%H%M%SZ 2>/dev/null || true)"
|
||||
if [[ -z "$ts" ]]; then
|
||||
warn "Durable snapshot skipped: no UTC timestamp available (upgrade continues)."
|
||||
return 0
|
||||
fi
|
||||
# umask 077 makes every dir/file the snapshot creates private from birth (it
|
||||
# mirrors operator config, which may hold secrets). It is PROCESS-global, so we
|
||||
# save and restore it around exactly this block — otherwise every later sync
|
||||
# copy and new framework dir would inherit 0600/0700 instead of 0644/0755.
|
||||
old_umask="$(umask)"
|
||||
umask 077
|
||||
if ! mkdir -p "$root"; then
|
||||
umask "$old_umask"
|
||||
warn "Durable snapshot skipped: cannot create backup dir $root (upgrade continues; operator files remain manifest-protected)."
|
||||
return 0
|
||||
fi
|
||||
chmod 700 "$root" 2>/dev/null || true
|
||||
dir="$root/pre-update-$ts"
|
||||
if [[ -e "$dir" ]]; then # same-second re-run: disambiguate
|
||||
local n=1; while [[ -e "$dir-$n" ]]; do n=$((n + 1)); done; dir="$dir-$n"
|
||||
fi
|
||||
if ! mkdir -p "$dir"; then
|
||||
umask "$old_umask"
|
||||
warn "Durable snapshot skipped: cannot create $dir (upgrade continues)."
|
||||
return 0
|
||||
fi
|
||||
chmod 700 "$dir"
|
||||
list="$(mktemp)"
|
||||
if ! enumerate_operator_files "$list"; then
|
||||
umask "$old_umask"
|
||||
warn "Durable snapshot skipped: could not enumerate operator files (upgrade continues)."
|
||||
rm -f "$list"; rmdir "$dir" 2>/dev/null || true
|
||||
return 0
|
||||
fi
|
||||
while IFS= read -r -d '' rel; do
|
||||
src="$TARGET_DIR/$rel"; dst="$dir/$rel"
|
||||
[[ -f "$src" ]] || continue
|
||||
mkdir -p "$(dirname "$dst")"
|
||||
if ! cp "$src" "$dst"; then
|
||||
warn "Durable snapshot: could not copy operator file '$rel' (skipped)."
|
||||
continue
|
||||
fi
|
||||
chmod 600 "$dst" 2>/dev/null || true
|
||||
count=$((count + 1))
|
||||
done < "$list"
|
||||
rm -f "$list"
|
||||
# Tighten every dir the copy created (mkdir -p honors umask, but be explicit).
|
||||
find "$dir" -type d -exec chmod 700 {} + 2>/dev/null || true
|
||||
umask "$old_umask" # UMASK-RESTORE-NORMAL — restore before the upgrade proper resumes (see above)
|
||||
DURABLE_SNAPSHOT_DIR="$dir"
|
||||
ok "Durable pre-update snapshot: $count operator file(s) saved to $dir (recover with: mosaic restore --list)"
|
||||
prune_durable_snapshots
|
||||
}
|
||||
|
||||
# Post-sync safety net: a keep-mode upgrade must NEVER modify an operator file.
|
||||
# Compare every file in the durable snapshot to its current target counterpart;
|
||||
# any that changed (or vanished) was touched by a framework bug — restore it from
|
||||
# the snapshot and warn loudly. This does NOT abort: the framework itself synced
|
||||
# correctly; we only heal the operator collateral. Runs after the restore trap is
|
||||
# disarmed so its corrective copies can't spuriously trip a full rollback, and
|
||||
# every step is guarded so `set -e` cannot exit silently mid-heal (cf. blocker-D2).
|
||||
verify_operator_surface() {
|
||||
[[ -n "$DURABLE_SNAPSHOT_DIR" && -d "$DURABLE_SNAPSHOT_DIR" ]] || return 0
|
||||
local scan snap rel cur healed=0
|
||||
scan="$(mktemp)"
|
||||
if ! find "$DURABLE_SNAPSHOT_DIR" -type f -print0 > "$scan"; then
|
||||
rm -f "$scan"
|
||||
warn "Post-upgrade verify skipped: could not enumerate the pre-update snapshot at $DURABLE_SNAPSHOT_DIR."
|
||||
return 0
|
||||
fi
|
||||
while IFS= read -r -d '' snap; do
|
||||
rel="${snap#"$DURABLE_SNAPSHOT_DIR"/}"
|
||||
cur="$TARGET_DIR/$rel"
|
||||
# A migration may legitimately delete an operator-classified path (e.g. legacy
|
||||
# bin/). Its absence is intended — do not heal it back, or the migration is
|
||||
# silently undone and never re-runs once the version is stamped (#791 PR2).
|
||||
is_migration_removed "$rel" && continue # MIGRATION-SKIP-GUARD
|
||||
if [[ ! -e "$cur" ]] || ! cmp -s "$snap" "$cur"; then
|
||||
# Never restore THROUGH a symlink: an operator path swapped for a link would
|
||||
# otherwise let cp write snapshot contents (possibly secrets) outside the
|
||||
# target (CWE-59). Refuse a symlinked parent; drop a symlinked leaf and write
|
||||
# a real file in its place.
|
||||
if has_symlinked_parent "$rel"; then
|
||||
warn "Operator path '$rel' has a symlinked parent under $TARGET_DIR; refusing to restore through it (possible tampering) — recover it manually from $DURABLE_SNAPSHOT_DIR."
|
||||
continue
|
||||
fi
|
||||
[[ -L "$cur" ]] && rm -f "$cur" # SYMLINK-LEAF-GUARD
|
||||
# Guard mkdir too: under set -e (trap already disarmed) a bare failure would
|
||||
# exit the whole installer before the recovery pointer below is emitted.
|
||||
if ! mkdir -p "$(dirname "$cur")"; then
|
||||
warn "Operator file '$rel' was modified by the upgrade but could NOT be auto-restored (parent dir unavailable) — recover it manually from $DURABLE_SNAPSHOT_DIR."
|
||||
continue
|
||||
fi
|
||||
if cp "$snap" "$cur"; then
|
||||
chmod 600 "$cur" 2>/dev/null || true
|
||||
warn "Operator file was modified by the upgrade and has been restored from the pre-update snapshot: $rel"
|
||||
healed=$((healed + 1))
|
||||
else
|
||||
warn "Operator file '$rel' was modified by the upgrade but could NOT be auto-restored — recover it manually from $DURABLE_SNAPSHOT_DIR."
|
||||
fi
|
||||
fi
|
||||
done < "$scan"
|
||||
rm -f "$scan"
|
||||
if (( healed > 0 )); then
|
||||
warn "$healed operator file(s) were unexpectedly changed by this upgrade and were restored from the pre-update snapshot. A keep-mode upgrade must never modify operator files — this indicates a framework manifest bug; please report it (#791)."
|
||||
fi
|
||||
}
|
||||
|
||||
# Reconcile contract files after sync: framework-owned overwrite (backup-once),
|
||||
# user-seeded seed-if-absent.
|
||||
reconcile_framework_files() {
|
||||
@@ -326,6 +545,10 @@ run_migrations() {
|
||||
# Remove bin/ directory — all executables now live in the npm CLI.
|
||||
# Scripts that were in bin/ are now in tools/_scripts/.
|
||||
if [[ "$from_version" -lt 2 ]]; then
|
||||
# bin/ and the rails symlink are operator-classified by the manifest (unknown⇒
|
||||
# operator) and thus captured in the durable snapshot; record them as
|
||||
# intentional removals so the post-sync verify net does not restore them.
|
||||
MIGRATION_REMOVED_PATHS+=("bin" "rails")
|
||||
if [[ -d "$TARGET_DIR/bin" ]]; then
|
||||
ok "Removing legacy bin/ directory (executables now in npm CLI)"
|
||||
rm -rf "$TARGET_DIR/bin"
|
||||
@@ -383,6 +606,9 @@ fi
|
||||
# not for a validation failure that has touched nothing yet (#791 blocker-1).
|
||||
if [[ "$INSTALL_MODE" == "keep" ]]; then
|
||||
manifest_load
|
||||
# Durable, operator-scoped backup taken BEFORE any mutation (#791 PR2). Kept
|
||||
# outside the framework tree; recovered later via `mosaic restore`. Fail-open.
|
||||
make_durable_snapshot
|
||||
fi
|
||||
|
||||
# Snapshot before any destructive file operation; restore on interrupt/failure.
|
||||
@@ -421,6 +647,10 @@ run_migrations
|
||||
|
||||
# File-system phase complete and consistent — clear the restore trap.
|
||||
trap - ERR INT TERM
|
||||
# Post-sync safety net: heal any operator file a manifest bug let the sync touch,
|
||||
# using the durable pre-update snapshot (#791 PR2). Runs with the trap disarmed so
|
||||
# a corrective copy can't spuriously trigger a full rollback.
|
||||
verify_operator_surface # VERIFY-NET (#791 PR2)
|
||||
cleanup_snapshot
|
||||
|
||||
# Testability / minimal-install hook: stop after the file-system phase, before any
|
||||
|
||||
@@ -0,0 +1,313 @@
|
||||
#!/usr/bin/env bash
|
||||
# test-upgrade-durable-snapshot.sh — the #791 PR2 regression gate.
|
||||
#
|
||||
# PR1 gave keep-mode upgrades two protections: the manifest (a keep-sync only
|
||||
# ever writes framework-owned paths — operator config is structurally untouched)
|
||||
# and an EPHEMERAL /tmp snapshot that rolls the whole target back if the sync
|
||||
# CRASHES mid-write. PR2 adds a third, independent layer for the case neither
|
||||
# covers: a "successful" upgrade that a manifest/logic bug silently let touch an
|
||||
# operator file. That layer is a DURABLE, operator-scoped pre-update snapshot:
|
||||
#
|
||||
# Part 1 (scope): before any mutation, the installer copies exactly the
|
||||
# operator-owned files that exist into a retained backup
|
||||
# under $XDG_STATE_HOME/mosaic/backups/pre-update-<ts>/ —
|
||||
# framework files are NOT captured.
|
||||
# Part 2 (perms): the backup root, snapshot dir and every nested dir are
|
||||
# 0700; every backed-up file is 0600 (never world-readable,
|
||||
# even though operator config may hold secrets).
|
||||
# Part 3 (no leak): a secret seeded into credentials.json is copied into the
|
||||
# snapshot (proving coverage) but its value never appears
|
||||
# on stdout/stderr — the snapshot reports counts/paths only.
|
||||
# Part 4 (retention): only the newest MOSAIC_BACKUP_RETENTION snapshots survive;
|
||||
# older ones are pruned.
|
||||
# Part 5 (verify net): if the upgrade DID modify an operator file (injected here
|
||||
# with a cp shim that scribbles on SOUL.md while a framework
|
||||
# file is copied), the post-sync verify restores that file
|
||||
# from the durable snapshot and warns loudly. The control —
|
||||
# the same installer with the verify call stripped — leaves
|
||||
# the corruption in place, proving the net is load-bearing.
|
||||
#
|
||||
# Usage: bash test-upgrade-durable-snapshot.sh
|
||||
set -uo pipefail
|
||||
|
||||
FW="$(cd "$(dirname "${BASH_SOURCE[0]}")/../../.." && pwd)" # packages/mosaic/framework
|
||||
INSTALL="$FW/install.sh"
|
||||
ORIG_PATH="$PATH"
|
||||
FRAMEWORK_VERSION="$(grep -m1 '^FRAMEWORK_VERSION=' "$INSTALL" | cut -d= -f2)"
|
||||
|
||||
# Control installers must live INSIDE $FW: install.sh derives SOURCE_DIR from its
|
||||
# own path and sources tools/_lib/manifest.sh relative to it, so a copy anywhere
|
||||
# else aborts before the sync. Each control is a shipped installer with one guard
|
||||
# line stripped (keyed off a `# <MARKER>` anchor), proving that guard load-bearing.
|
||||
# All controls share the .install-*.tmp.sh glob so one trap sweeps them on exit.
|
||||
VERIFYCTRL="$FW/.install-verifynet-control.tmp.sh"
|
||||
rm -f "$FW"/.install-*.tmp.sh
|
||||
trap 'rm -f "$FW"/.install-*.tmp.sh' EXIT
|
||||
|
||||
# mk_control <marker-regex> <name> — echo a control installer path ($FW-local) that
|
||||
# is $INSTALL with every line matching /<marker-regex>/ deleted.
|
||||
mk_control() {
|
||||
local path="$FW/.install-$2.tmp.sh"
|
||||
sed "/$1/d" "$INSTALL" > "$path"
|
||||
printf '%s' "$path"
|
||||
}
|
||||
|
||||
pass=0; fail=0
|
||||
chk() { if eval "$2"; then echo " ✓ $1"; pass=$((pass + 1)); else echo " ✗ $1"; fail=$((fail + 1)); fi; }
|
||||
|
||||
SECRET='SUPER-SECRET-TOKEN-do-not-log-pr2'
|
||||
SOUL_ORIG='# persona'
|
||||
# A framework file the sync copies (source ships it; the seeded target omits it,
|
||||
# so the bytes differ and cp is attempted). The Part-5 shim keys off this path.
|
||||
POISON_REL='guides/E2E-DELIVERY.md'
|
||||
|
||||
# Seed a recognized keep-mode install holding four operator-owned files across
|
||||
# the identity file, an operator subtree, memory, and the credentials carve-out.
|
||||
seed_home() {
|
||||
local H="$1"
|
||||
mkdir -p "$H/agents" "$H/tools/_lib" "$H/memory"
|
||||
printf '%s\n' "$SOUL_ORIG" > "$H/SOUL.md" # recognized install → keep mode
|
||||
printf 'MODEL=opus\n' > "$H/agents/coder0.conf"
|
||||
printf '# operator memory\n' > "$H/memory/note.md"
|
||||
printf 'TOKEN=%s\n' "$SECRET" > "$H/tools/_lib/credentials.json"
|
||||
echo 3 > "$H/.framework-version"
|
||||
# Deliberately NO guides/E2E-DELIVERY.md so the sync copies it (framework file,
|
||||
# bytes differ) — that copy is where the Part-5 corruption shim fires.
|
||||
}
|
||||
|
||||
# A pre-v2 (legacy) keep-mode install: SOUL.md marks it recognized, and a bin/
|
||||
# tree with NO .framework-version makes installed_framework_version() report 1, so
|
||||
# the v1→v2 migration (which deletes bin/) runs. bin/ is unknown⇒operator, so the
|
||||
# durable snapshot captures it — the verify net must NOT heal the intended removal.
|
||||
seed_home_v1() {
|
||||
local H="$1"
|
||||
mkdir -p "$H/agents" "$H/tools/_lib" "$H/memory" "$H/bin"
|
||||
printf '%s\n' "$SOUL_ORIG" > "$H/SOUL.md"
|
||||
printf 'TOKEN=%s\n' "$SECRET" > "$H/tools/_lib/credentials.json"
|
||||
printf '#!/bin/sh\necho legacy\n' > "$H/bin/tool.sh"; chmod +x "$H/bin/tool.sh"
|
||||
# Deliberately NO .framework-version and NO guides/E2E-DELIVERY.md (see seed_home).
|
||||
}
|
||||
|
||||
# A cp shim that, while the framework POISON file is being copied during sync,
|
||||
# swaps the operator credentials file for a symlink pointing at an attacker-
|
||||
# readable file OUTSIDE the target — simulating post-snapshot tampering (CWE-59).
|
||||
# The durable snapshot already holds the real credentials (it is taken before any
|
||||
# sync), so the verify net must restore a REAL file in place WITHOUT following the
|
||||
# link (which would write the snapshot's secret out through it). $EXFIL_TARGET is
|
||||
# expanded at shim-write time from the caller's environment.
|
||||
#
|
||||
# PORTABILITY (why this shim, not the real `cp`): the CWE-59 leak this exercises is
|
||||
# `cp` writing THROUGH a symlinked destination. GNU/BSD cp — what a real operator
|
||||
# runs `mosaic update` under — follows the dest symlink and leaks. busybox cp (the
|
||||
# Alpine CI image) REPLACES a symlinked dest instead of following it, so under the
|
||||
# CI harness the leak vector simply does not exist and the negative control could
|
||||
# never reproduce it. This shim therefore emulates the real-target GNU cp behavior
|
||||
# PORTABLY: when the destination is a symlink it writes the source bytes through the
|
||||
# link via redirection (which follows symlinks on every coreutils, busybox included);
|
||||
# otherwise it delegates to the host's real cp unchanged. Both the shipped-case and
|
||||
# the negative control run through this identical shim, so the ONLY difference
|
||||
# between them remains the SYMLINK-LEAF-GUARD — the control stays load-bearing and
|
||||
# non-tautological. It does NOT touch install.sh (approved) or the real assertions:
|
||||
# with the guard present the symlinked leaf is dropped BEFORE this cp runs, so the
|
||||
# dest is a real file and the delegate path is taken exactly as on a GNU host.
|
||||
make_symlink_leaf_shim() {
|
||||
local dir="$1" home="$2"
|
||||
cat > "$dir/cp" <<SHIM
|
||||
#!/usr/bin/env bash
|
||||
dest="\${@: -1}"
|
||||
src="\${@:(-2):1}"
|
||||
case "\$dest" in
|
||||
*/$POISON_REL)
|
||||
rm -f "$home/tools/_lib/credentials.json"
|
||||
ln -s "$EXFIL_TARGET" "$home/tools/_lib/credentials.json"
|
||||
;;
|
||||
esac
|
||||
# Coreutils-agnostic emulation of GNU cp's follow-through-dest-symlink behavior.
|
||||
if [[ -L "\$dest" && -f "\$src" ]]; then
|
||||
cat "\$src" > "\$dest"
|
||||
exit \$?
|
||||
fi
|
||||
exec env PATH="$ORIG_PATH" cp "\$@"
|
||||
SHIM
|
||||
chmod +x "$dir/cp"
|
||||
}
|
||||
|
||||
# A cp shim that, while the framework POISON file is being copied during sync,
|
||||
# also appends garbage to the operator SOUL.md — simulating a manifest bug that
|
||||
# writes outside the framework lane. The framework copy itself still succeeds
|
||||
# (real cp runs), so the sync completes 0 and the post-sync verify is what must
|
||||
# catch and undo the operator-file damage. The snapshot's own cp only ever
|
||||
# targets operator files (never guides/…), so it is never corrupted by this shim.
|
||||
make_corrupt_shim() {
|
||||
local dir="$1" home="$2"
|
||||
cat > "$dir/cp" <<SHIM
|
||||
#!/usr/bin/env bash
|
||||
dest="\${@: -1}"
|
||||
case "\$dest" in
|
||||
*/$POISON_REL) printf 'CORRUPTION-mid-sync\n' >> "$home/SOUL.md" 2>/dev/null || true ;;
|
||||
esac
|
||||
exec env PATH="$ORIG_PATH" cp "\$@"
|
||||
SHIM
|
||||
chmod +x "$dir/cp"
|
||||
}
|
||||
|
||||
# Run one keep-mode, sync-only upgrade with $XDG_STATE_HOME redirected to a
|
||||
# throwaway dir (so the real ~/.local/state is never touched). Optional args:
|
||||
# $2 shim-maker (default none), $3 MOSAIC_BACKUP_RETENTION (default unset).
|
||||
# Echoes: "<exit>\t<out>\t<state-dir>\t<home>".
|
||||
# $4 seeder (default seed_home) — swap in seed_home_v1 for the migration case.
|
||||
run_snap() {
|
||||
local installer="$1" shim_maker="${2:-}" retention="${3:-}" seeder="${4:-seed_home}" H STATE OUT SHIM rc pathpre
|
||||
H=$(mktemp -d); STATE=$(mktemp -d); OUT=$(mktemp); pathpre="$ORIG_PATH"
|
||||
"$seeder" "$H"
|
||||
if [[ -n "$shim_maker" ]]; then
|
||||
SHIM=$(mktemp -d); "$shim_maker" "$SHIM" "$H"; pathpre="$SHIM:$ORIG_PATH"
|
||||
fi
|
||||
set +e
|
||||
env PATH="$pathpre" XDG_STATE_HOME="$STATE" \
|
||||
${retention:+MOSAIC_BACKUP_RETENTION="$retention"} \
|
||||
MOSAIC_HOME="$H" MOSAIC_INSTALL_MODE=keep MOSAIC_SYNC_ONLY=1 \
|
||||
bash "$installer" >"$OUT" 2>&1
|
||||
rc=$?
|
||||
set -e 2>/dev/null || true
|
||||
[[ -n "$shim_maker" ]] && rm -rf "$SHIM"
|
||||
printf '%s\t%s\t%s\t%s\n' "$rc" "$OUT" "$STATE" "$H"
|
||||
}
|
||||
|
||||
# Resolve the single pre-update-* snapshot dir under a state dir (newest if many).
|
||||
snap_dir() {
|
||||
find "$1/mosaic/backups" -maxdepth 1 -type d -name 'pre-update-*' 2>/dev/null \
|
||||
| LC_ALL=C sort -r | head -1
|
||||
}
|
||||
|
||||
echo "── Part 1/2/3: durable snapshot scope, perms, no-leak ──────────────────"
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$INSTALL")
|
||||
SNAP="$(snap_dir "$STATE")"
|
||||
|
||||
chk "upgrade succeeds" "[ '$rc' -eq 0 ]"
|
||||
chk "exactly one pre-update snapshot created" "[ \$(find '$STATE/mosaic/backups' -maxdepth 1 -type d -name 'pre-update-*' | wc -l) -eq 1 ]"
|
||||
chk "snapshot: SOUL.md captured" "[ -f '$SNAP/SOUL.md' ]"
|
||||
chk "snapshot: operator subtree captured" "[ -f '$SNAP/agents/coder0.conf' ]"
|
||||
chk "snapshot: memory captured" "[ -f '$SNAP/memory/note.md' ]"
|
||||
chk "snapshot: credentials carve-out captured" "[ -f '$SNAP/tools/_lib/credentials.json' ]"
|
||||
chk "snapshot: SOUL.md bytes preserved" "[ \"\$(cat '$SNAP/SOUL.md')\" = '$SOUL_ORIG' ]"
|
||||
chk "snapshot: framework file NOT captured" "[ ! -e '$SNAP/CONSTITUTION.md' ] && [ ! -e '$SNAP/$POISON_REL' ]"
|
||||
|
||||
# Part 2 — permissions (0700 dirs, 0600 files); never world-readable.
|
||||
chk "perms: backup root is 0700" "[ \$(stat -c '%a' '$STATE/mosaic/backups') -eq 700 ]"
|
||||
chk "perms: snapshot dir is 0700" "[ \$(stat -c '%a' '$SNAP') -eq 700 ]"
|
||||
chk "perms: nested dir is 0700" "[ \$(stat -c '%a' '$SNAP/agents') -eq 700 ]"
|
||||
chk "perms: credentials backup is 0600" "[ \$(stat -c '%a' '$SNAP/tools/_lib/credentials.json') -eq 600 ]"
|
||||
chk "perms: SOUL.md backup is 0600" "[ \$(stat -c '%a' '$SNAP/SOUL.md') -eq 600 ]"
|
||||
|
||||
# Part 3 — the secret is backed up but never emitted to stdout/stderr.
|
||||
chk "no-leak: secret IS in the backup file" "grep -q '$SECRET' '$SNAP/tools/_lib/credentials.json'"
|
||||
chk "no-leak: secret NOT on stdout/stderr" "! grep -q '$SECRET' '$OUT'"
|
||||
rm -rf "$STATE" "$H"; rm -f "$OUT"
|
||||
|
||||
echo "── Part 4: retention prune (MOSAIC_BACKUP_RETENTION) ───────────────────"
|
||||
# Pre-seed four dated snapshots, then take one real snapshot with retention=2:
|
||||
# only the two newest (the fresh real one + the newest pre-seeded) must survive.
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(
|
||||
H=$(mktemp -d); STATE=$(mktemp -d); OUT=$(mktemp)
|
||||
seed_home "$H"
|
||||
mkdir -p "$STATE/mosaic/backups"
|
||||
for ts in 20200101T000000Z 20210101T000000Z 20220101T000000Z 20230101T000000Z; do
|
||||
mkdir -p "$STATE/mosaic/backups/pre-update-$ts"
|
||||
done
|
||||
set +e
|
||||
env PATH="$ORIG_PATH" XDG_STATE_HOME="$STATE" MOSAIC_BACKUP_RETENTION=2 \
|
||||
MOSAIC_HOME="$H" MOSAIC_INSTALL_MODE=keep MOSAIC_SYNC_ONLY=1 \
|
||||
bash "$INSTALL" >"$OUT" 2>&1
|
||||
rc=$?
|
||||
set -e 2>/dev/null || true
|
||||
printf '%s\t%s\t%s\t%s\n' "$rc" "$OUT" "$STATE" "$H"
|
||||
)
|
||||
chk "retention: upgrade succeeds" "[ '$rc' -eq 0 ]"
|
||||
chk "retention: pruned to exactly 2 snapshots" "[ \$(find '$STATE/mosaic/backups' -maxdepth 1 -type d -name 'pre-update-*' | wc -l) -eq 2 ]"
|
||||
chk "retention: newest pre-seeded survives" "[ -d '$STATE/mosaic/backups/pre-update-20230101T000000Z' ]"
|
||||
chk "retention: oldest pre-seeded pruned" "[ ! -d '$STATE/mosaic/backups/pre-update-20200101T000000Z' ]"
|
||||
rm -rf "$STATE" "$H"; rm -f "$OUT"
|
||||
|
||||
echo "── Part 5: post-sync verify restores an operator file (+ control) ──────"
|
||||
# Shipped installer: the cp shim corrupts SOUL.md mid-sync; verify must restore it.
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$INSTALL" make_corrupt_shim)
|
||||
chk "verify: upgrade still succeeds" "[ '$rc' -eq 0 ]"
|
||||
chk "verify: SOUL.md restored to original" "[ \"\$(cat '$H/SOUL.md')\" = '$SOUL_ORIG' ]"
|
||||
chk "verify: no corruption remains in SOUL.md" "! grep -q 'CORRUPTION-mid-sync' '$H/SOUL.md'"
|
||||
chk "verify: loud restore warning emitted" "grep -qi 'restored from the pre-update snapshot' '$OUT'"
|
||||
chk "verify: secret still not leaked" "! grep -q '$SECRET' '$OUT'"
|
||||
rm -rf "$STATE" "$H"; rm -f "$OUT"
|
||||
|
||||
# Control: strip the verify call → the corruption must SURVIVE (net is load-bearing).
|
||||
sed '/# VERIFY-NET/d' "$INSTALL" > "$VERIFYCTRL"
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$VERIFYCTRL" make_corrupt_shim)
|
||||
chk "control: SOUL.md corruption survives" "grep -q 'CORRUPTION-mid-sync' '$H/SOUL.md'"
|
||||
chk "control: no restore warning emitted" "! grep -qi 'restored from the pre-update snapshot' '$OUT'"
|
||||
rm -rf "$STATE" "$H"; rm -f "$OUT"
|
||||
|
||||
echo "── Part 6: verify net honors an intentional migration removal (+ control) ─"
|
||||
# BLOCKER regression: on a pre-v2 install, bin/ is operator-classified so the durable
|
||||
# snapshot captures it — but the v1→v2 migration deletes bin/ ON PURPOSE. The verify
|
||||
# net must SKIP that removal (is_migration_removed), or it heals bin/ back and the
|
||||
# migration is silently undone forever once the version is stamped.
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$INSTALL" "" "" seed_home_v1)
|
||||
chk "migration: upgrade succeeds" "[ '$rc' -eq 0 ]"
|
||||
chk "migration: legacy bin/ stays removed" "[ ! -e '$H/bin' ]"
|
||||
chk "migration: operator SOUL.md untouched" "[ \"\$(cat '$H/SOUL.md')\" = '$SOUL_ORIG' ]"
|
||||
chk "migration: version stamped to $FRAMEWORK_VERSION" "[ \"\$(cat '$H/.framework-version')\" = '$FRAMEWORK_VERSION' ]"
|
||||
rm -rf "$STATE" "$H"; rm -f "$OUT"
|
||||
|
||||
# Control: strip the MIGRATION-SKIP-GUARD → the verify net restores bin/ from the
|
||||
# snapshot, silently undoing the migration (proves the guard is load-bearing).
|
||||
MIGCTRL="$(mk_control 'MIGRATION-SKIP-GUARD' migration-control)"
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$MIGCTRL" "" "" seed_home_v1)
|
||||
chk "control: bin/ wrongly restored by verify" "[ -e '$H/bin/tool.sh' ]"
|
||||
rm -rf "$STATE" "$H"; rm -f "$OUT"
|
||||
|
||||
echo "── Part 7: verify net never restores a secret through a symlink (+ control) ─"
|
||||
# HIGH (CWE-59) regression: an attacker who swaps an operator file for a symlink
|
||||
# AFTER the durable snapshot must not cause the verify net's restore to write the
|
||||
# snapshot's secret out THROUGH that link. The shipped net drops a symlinked leaf and
|
||||
# writes a real file in its place, leaving the external target untouched.
|
||||
EXFIL_DIR=$(mktemp -d); EXFIL_TARGET="$EXFIL_DIR/stolen"
|
||||
printf 'ATTACKER-PLACEHOLDER\n' > "$EXFIL_TARGET"
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$INSTALL" make_symlink_leaf_shim)
|
||||
chk "symlink-leaf: upgrade succeeds" "[ '$rc' -eq 0 ]"
|
||||
chk "symlink-leaf: secret NOT written through link" "! grep -q '$SECRET' '$EXFIL_TARGET'"
|
||||
chk "symlink-leaf: credentials.json is a real file" "[ -f '$H/tools/_lib/credentials.json' ] && [ ! -L '$H/tools/_lib/credentials.json' ]"
|
||||
chk "symlink-leaf: credentials.json restored intact" "grep -q '$SECRET' '$H/tools/_lib/credentials.json'"
|
||||
chk "symlink-leaf: secret not leaked to stdout/stderr" "! grep -q '$SECRET' '$OUT'"
|
||||
rm -rf "$STATE" "$H" "$EXFIL_DIR"; rm -f "$OUT"
|
||||
|
||||
# Control: strip the SYMLINK-LEAF-GUARD → cp follows the swapped-in link and writes
|
||||
# the snapshot secret out through it (proves the guard is load-bearing).
|
||||
EXFIL_DIR=$(mktemp -d); EXFIL_TARGET="$EXFIL_DIR/stolen"
|
||||
printf 'ATTACKER-PLACEHOLDER\n' > "$EXFIL_TARGET"
|
||||
LEAFCTRL="$(mk_control 'SYMLINK-LEAF-GUARD' symlinkleaf-control)"
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(run_snap "$LEAFCTRL" make_symlink_leaf_shim)
|
||||
chk "control: secret leaked through the symlink" "grep -q '$SECRET' '$EXFIL_TARGET'"
|
||||
rm -rf "$STATE" "$H" "$EXFIL_DIR"; rm -f "$OUT"
|
||||
|
||||
echo "── Part 8: snapshot umask 077 does not leak into synced files (+ control) ──"
|
||||
# SHOULD-FIX regression: umask 077 is process-global. Scoped to the snapshot it keeps
|
||||
# backups 0600; leaked past it, every later cp/mkdir inherits 0600/0700. A freshly-
|
||||
# synced framework file must be 0644 (per the ambient 022 umask) while the backup of
|
||||
# a secret stays 0600.
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(umask 022; run_snap "$INSTALL")
|
||||
SNAP="$(snap_dir "$STATE")"
|
||||
chk "umask: upgrade succeeds" "[ '$rc' -eq 0 ]"
|
||||
chk "umask: synced framework file is 0644" "[ \$(stat -c '%a' '$H/$POISON_REL') -eq 644 ]"
|
||||
chk "umask: backup of a secret stays 0600" "[ \$(stat -c '%a' '$SNAP/tools/_lib/credentials.json') -eq 600 ]"
|
||||
rm -rf "$STATE" "$H"; rm -f "$OUT"
|
||||
|
||||
# Control: strip the UMASK-RESTORE-NORMAL line → umask 077 leaks past the snapshot,
|
||||
# so the newly-synced framework file is created 0600 (proves the restore matters).
|
||||
UMASKCTRL="$(mk_control 'UMASK-RESTORE-NORMAL' umask-control)"
|
||||
IFS=$'\t' read -r rc OUT STATE H < <(umask 022; run_snap "$UMASKCTRL")
|
||||
chk "control: leaked umask makes synced file 0600" "[ \$(stat -c '%a' '$H/$POISON_REL') -eq 600 ]"
|
||||
rm -rf "$STATE" "$H"; rm -f "$OUT"
|
||||
|
||||
echo ""
|
||||
echo "RESULT: $pass passed, $fail failed"
|
||||
[ "$fail" -eq 0 ]
|
||||
@@ -29,6 +29,13 @@ INSTALL="$FW/install.sh"
|
||||
pass=0; fail=0
|
||||
chk() { if eval "$2"; then echo " ✓ $1"; pass=$((pass + 1)); else echo " ✗ $1"; fail=$((fail + 1)); fi; }
|
||||
|
||||
# Redirect the #791 PR2 durable pre-update snapshot ($XDG_STATE_HOME/mosaic/backups)
|
||||
# into a throwaway so a keep-mode upgrade under test never writes into the real
|
||||
# ~/.local/state. This test asserts operator-surface fidelity, not backup content.
|
||||
export XDG_STATE_HOME
|
||||
XDG_STATE_HOME="$(mktemp -d)"
|
||||
trap 'rm -rf "$XDG_STATE_HOME"' EXIT
|
||||
|
||||
SECRET='SUPER-SECRET-TOKEN-do-not-log-3f9a'
|
||||
|
||||
# Seed a throwaway MOSAIC_HOME with an operator sentinel per ownership class.
|
||||
@@ -206,7 +213,7 @@ fi
|
||||
# does not silently depend on rsync being installed. (Provide the coreutils the
|
||||
# installer needs on the stripped PATH.)
|
||||
FBIN=$(mktemp -d)
|
||||
for t in bash cp find mktemp rm mkdir chmod cmp sed grep cat dirname basename stat sha256sum awk tr; do
|
||||
for t in bash cp find mktemp rm mkdir chmod cmp sed grep cat dirname basename stat sha256sum awk tr date sort; do
|
||||
p=$(command -v "$t" 2>/dev/null) && ln -s "$p" "$FBIN/$t"
|
||||
done
|
||||
run_matrix "rsync-absent" env "PATH=$FBIN"
|
||||
|
||||
@@ -17,6 +17,7 @@ import { registerConfigCommand } from './commands/config.js';
|
||||
import { registerFleetCommand } from './commands/fleet.js';
|
||||
import { registerMissionCommand } from './commands/mission.js';
|
||||
import { registerUninstallCommand } from './commands/uninstall.js';
|
||||
import { registerRestoreCommand } from './commands/restore.js';
|
||||
// prdy is registered via launch.ts
|
||||
import { registerLaunchCommands } from './commands/launch.js';
|
||||
import { registerAuthCommand } from './commands/auth.js';
|
||||
@@ -406,6 +407,10 @@ registerStorageCommand(program);
|
||||
|
||||
registerUninstallCommand(program);
|
||||
|
||||
// ─── restore ─────────────────────────────────────────────────────────────────
|
||||
|
||||
registerRestoreCommand(program);
|
||||
|
||||
// ─── telemetry ───────────────────────────────────────────────────────────────
|
||||
|
||||
registerTelemetryCommand(program);
|
||||
|
||||
885
packages/mosaic/src/commands/fleet-regen-command.spec.ts
Normal file
885
packages/mosaic/src/commands/fleet-regen-command.spec.ts
Normal file
@@ -0,0 +1,885 @@
|
||||
import { chmod, mkdir, mkdtemp, open, readFile, rm, stat, writeFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { Command } from 'commander';
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||
import { registerFleetCommand, type CommandResult, type FleetCommandDeps } from './fleet.js';
|
||||
import { executeFleetRegen, formatFleetRegenReport } from './fleet-regen-command.js';
|
||||
import {
|
||||
acquirePrivateRosterMutationLock,
|
||||
projectRosterV2AgentGeneratedEnv,
|
||||
} from '../fleet/fleet-reconciler.js';
|
||||
import { applyPreparedGeneratedAgentEnvironmentProjection } from '../fleet/generated-env-boundary.js';
|
||||
import { parseRosterV2 } from '../fleet/roster-v2.js';
|
||||
|
||||
// A two-agent roster-v2 SSOT. `mosaic fleet regen` must rebuild each agent's
|
||||
// `fleet/agents/<name>.env.generated` projection from exactly this source and
|
||||
// nothing else — deterministically, without ever touching agent lifecycle.
|
||||
const rosterYaml = `
|
||||
version: 2
|
||||
generation: 7
|
||||
transport: tmux
|
||||
tmux:
|
||||
socket_name: mosaic-fleet
|
||||
holder_session: _holder
|
||||
defaults:
|
||||
working_directory: /srv/mosaic
|
||||
runtime: pi
|
||||
runtimes:
|
||||
pi:
|
||||
reset_command: /new
|
||||
agents:
|
||||
- name: coder0
|
||||
alias: Coder 0
|
||||
class: code
|
||||
runtime: pi
|
||||
provider: openai
|
||||
model: gpt-5.6-sol
|
||||
reasoning: high
|
||||
tool_policy: code
|
||||
working_directory: /srv/mosaic
|
||||
persistent_persona: false
|
||||
reset_between_tasks: true
|
||||
lifecycle:
|
||||
enabled: true
|
||||
desired_state: stopped
|
||||
launch:
|
||||
yolo: true
|
||||
- name: coder1
|
||||
alias: Coder 1
|
||||
class: code
|
||||
runtime: pi
|
||||
provider: openai
|
||||
model: gpt-5.6-sol
|
||||
reasoning: medium
|
||||
tool_policy: code
|
||||
working_directory: /srv/other
|
||||
persistent_persona: false
|
||||
reset_between_tasks: true
|
||||
lifecycle:
|
||||
enabled: true
|
||||
desired_state: stopped
|
||||
launch:
|
||||
yolo: true
|
||||
`;
|
||||
|
||||
let cleanup: string | undefined;
|
||||
|
||||
afterEach(async (): Promise<void> => {
|
||||
vi.restoreAllMocks();
|
||||
process.exitCode = undefined;
|
||||
if (cleanup) await rm(cleanup, { recursive: true, force: true });
|
||||
cleanup = undefined;
|
||||
});
|
||||
|
||||
async function fleetHome(withRoster = rosterYaml): Promise<string> {
|
||||
cleanup = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-command-'));
|
||||
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
|
||||
await mkdir(join(cleanup, directory), { recursive: true, mode: 0o700 });
|
||||
await chmod(join(cleanup, directory), 0o700);
|
||||
}
|
||||
await chmod(cleanup, 0o700);
|
||||
await writeFile(join(cleanup, 'fleet', 'roster.yaml'), withRoster, { mode: 0o600 });
|
||||
// Semantic roster validation (matching `fleet reconcile`) resolves each agent
|
||||
// class to a persona; seed the classes the fixtures reference.
|
||||
for (const klass of ['code', 'merge-gate']) {
|
||||
await writeFile(
|
||||
join(cleanup, 'fleet', 'roles', `${klass}.md`),
|
||||
`\`class: ${klass}\`\n\n# ${klass} persona\n`,
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
}
|
||||
return cleanup;
|
||||
}
|
||||
|
||||
/**
|
||||
* A runner spy that records EVERY invocation. `regen` is projection-only and
|
||||
* must never issue a lifecycle/restart call, so a non-empty call log is a
|
||||
* hard failure — this is the load-bearing "never restarts" gate.
|
||||
*/
|
||||
function recordingRunner(
|
||||
calls: string[][],
|
||||
): (command: string, args: string[]) => Promise<CommandResult> {
|
||||
return async (command: string, args: string[]): Promise<CommandResult> => {
|
||||
calls.push([command, ...args]);
|
||||
return { stdout: '', stderr: '', exitCode: 0 };
|
||||
};
|
||||
}
|
||||
|
||||
function program(mosaicHome: string, runner: FleetCommandDeps['runner']): Command {
|
||||
const result = new Command();
|
||||
result.exitOverride();
|
||||
registerFleetCommand(result, { mosaicHome, runner });
|
||||
return result;
|
||||
}
|
||||
|
||||
function capture(): string[] {
|
||||
const lines: string[] = [];
|
||||
vi.spyOn(console, 'log').mockImplementation((value: string): void => {
|
||||
lines.push(value);
|
||||
});
|
||||
return lines;
|
||||
}
|
||||
|
||||
async function exists(path: string): Promise<boolean> {
|
||||
try {
|
||||
await stat(path);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
describe('projectRosterV2AgentGeneratedEnv', (): void => {
|
||||
it('maps a roster-v2 agent to exactly the eight generated projection keys', (): void => {
|
||||
const roster = parseRosterV2(rosterYaml, 'yaml');
|
||||
const agent = roster.agents.find((candidate) => candidate.name === 'coder0');
|
||||
expect(agent).toBeDefined();
|
||||
const values = projectRosterV2AgentGeneratedEnv(roster, agent!);
|
||||
expect(values).toEqual({
|
||||
MOSAIC_AGENT_NAME: 'coder0',
|
||||
MOSAIC_AGENT_CLASS: 'code',
|
||||
MOSAIC_AGENT_RUNTIME: 'pi',
|
||||
MOSAIC_AGENT_MODEL: 'gpt-5.6-sol',
|
||||
MOSAIC_AGENT_REASONING: 'high',
|
||||
MOSAIC_AGENT_TOOL_POLICY: 'code',
|
||||
MOSAIC_AGENT_WORKDIR: '/srv/mosaic',
|
||||
MOSAIC_TMUX_SOCKET: 'mosaic-fleet',
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe('mosaic fleet regen', (): void => {
|
||||
it('is dry-run by default: reports the plan and writes nothing', async (): Promise<void> => {
|
||||
const home = await fleetHome();
|
||||
const calls: string[][] = [];
|
||||
const lines = capture();
|
||||
|
||||
await program(home, recordingRunner(calls)).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--json',
|
||||
]);
|
||||
|
||||
const result = JSON.parse(lines.pop() ?? '{}');
|
||||
expect(result).toMatchObject({
|
||||
mode: 'dry-run',
|
||||
generation: 7,
|
||||
agentCount: 2,
|
||||
written: 0,
|
||||
});
|
||||
expect(result.agents.map((agent: { name: string }) => agent.name)).toEqual([
|
||||
'coder0',
|
||||
'coder1',
|
||||
]);
|
||||
expect(
|
||||
result.agents.every((agent: { disposition: string }) => agent.disposition === 'create'),
|
||||
).toBe(true);
|
||||
// Nothing on disk.
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
|
||||
// No lifecycle/restart call — ever.
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('forwards configured persona roots (rolesDir/overrideDir) from reconcileDeps into regen', async (): Promise<void> => {
|
||||
// Codex r6: regen must resolve personas the SAME way reconcile does. If the
|
||||
// top-level registration drops reconcileDeps.rolesDir/overrideDir, a deployment
|
||||
// with custom persona roots gets reconcile ACCEPTING a roster while regen
|
||||
// REJECTS it (validating against the wrong default `<home>/fleet/roles`),
|
||||
// blocking the recovery command. Pin the wiring: seed personas ONLY under the
|
||||
// custom roots, leave the default roles dir empty, require regen to succeed.
|
||||
const home = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-wiring-'));
|
||||
cleanup = home;
|
||||
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
|
||||
await mkdir(join(home, directory), { recursive: true, mode: 0o700 });
|
||||
await chmod(join(home, directory), 0o700);
|
||||
}
|
||||
await chmod(home, 0o700);
|
||||
await writeFile(join(home, 'fleet', 'roster.yaml'), rosterYaml, { mode: 0o600 });
|
||||
// Personas live ONLY under the configured roots — the default `fleet/roles`
|
||||
// stays empty, so broken wiring fails persona resolution.
|
||||
const customRoles = join(home, 'custom-roles');
|
||||
const customOverride = join(home, 'custom-roles.local');
|
||||
await mkdir(customRoles, { recursive: true, mode: 0o700 });
|
||||
await mkdir(customOverride, { recursive: true, mode: 0o700 });
|
||||
for (const klass of ['code', 'merge-gate']) {
|
||||
await writeFile(join(customRoles, `${klass}.md`), `\`class: ${klass}\`\n\n# ${klass}\n`, {
|
||||
mode: 0o600,
|
||||
});
|
||||
}
|
||||
|
||||
const command = new Command();
|
||||
command.exitOverride();
|
||||
registerFleetCommand(command, {
|
||||
mosaicHome: home,
|
||||
runner: recordingRunner([]),
|
||||
reconcileDeps: { rolesDir: customRoles, overrideDir: customOverride },
|
||||
});
|
||||
const lines = capture();
|
||||
|
||||
await command.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
|
||||
|
||||
// Regen validated against the CONFIGURED roots → success. Broken wiring
|
||||
// resolves against the empty default and fails (exitCode 1, no JSON result).
|
||||
expect(process.exitCode).not.toBe(1);
|
||||
const result = JSON.parse(lines.pop() ?? '{}');
|
||||
expect(result).toMatchObject({ mode: 'dry-run', agentCount: 2 });
|
||||
});
|
||||
|
||||
it('--write rebuilds each generated projection from the roster SSOT', async (): Promise<void> => {
|
||||
const home = await fleetHome();
|
||||
const calls: string[][] = [];
|
||||
const lines = capture();
|
||||
|
||||
await program(home, recordingRunner(calls)).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--write',
|
||||
'--json',
|
||||
]);
|
||||
|
||||
const result = JSON.parse(lines.pop() ?? '{}');
|
||||
expect(result).toMatchObject({ mode: 'write', written: 2, agentCount: 2 });
|
||||
|
||||
const coder0 = await readFile(join(home, 'fleet', 'agents', 'coder0.env.generated'), 'utf8');
|
||||
expect(coder0).toContain('MOSAIC_AGENT_NAME=coder0');
|
||||
expect(coder0).toContain('MOSAIC_AGENT_WORKDIR=/srv/mosaic');
|
||||
expect(coder0).toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet');
|
||||
const coder1 = await readFile(join(home, 'fleet', 'agents', 'coder1.env.generated'), 'utf8');
|
||||
expect(coder1).toContain('MOSAIC_AGENT_NAME=coder1');
|
||||
expect(coder1).toContain('MOSAIC_AGENT_WORKDIR=/srv/other');
|
||||
// No lifecycle/restart call — ever.
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('is deterministic and idempotent across repeated --write runs', async (): Promise<void> => {
|
||||
const home = await fleetHome();
|
||||
const calls: string[][] = [];
|
||||
const path = join(home, 'fleet', 'agents', 'coder0.env.generated');
|
||||
|
||||
const cli = program(home, recordingRunner(calls));
|
||||
capture();
|
||||
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
|
||||
const first = await readFile(path, 'utf8');
|
||||
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
|
||||
const second = await readFile(path, 'utf8');
|
||||
|
||||
expect(second).toBe(first);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('reports rebuild disposition once the generated projection already exists', async (): Promise<void> => {
|
||||
const home = await fleetHome();
|
||||
const calls: string[][] = [];
|
||||
const cli = program(home, recordingRunner(calls));
|
||||
const lines = capture();
|
||||
|
||||
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
|
||||
lines.length = 0;
|
||||
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
|
||||
|
||||
const result = JSON.parse(lines.pop() ?? '{}');
|
||||
expect(
|
||||
result.agents.every((agent: { disposition: string }) => agent.disposition === 'rebuild'),
|
||||
).toBe(true);
|
||||
expect(result.written).toBe(0);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('never issues a lifecycle/restart call in either mode', async (): Promise<void> => {
|
||||
const home = await fleetHome();
|
||||
const calls: string[][] = [];
|
||||
const cli = program(home, recordingRunner(calls));
|
||||
capture();
|
||||
|
||||
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
|
||||
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
|
||||
|
||||
// Structural guarantee: regen has no path to systemctl/tmux at all.
|
||||
expect(calls).toEqual([]);
|
||||
const systemctlCalls = calls.filter(([command]) => command === 'systemctl');
|
||||
expect(systemctlCalls).toEqual([]);
|
||||
});
|
||||
|
||||
it('human-readable --write output prints the do-not-restart recovery runbook', async (): Promise<void> => {
|
||||
const home = await fleetHome();
|
||||
const lines = capture();
|
||||
|
||||
await program(home, recordingRunner([])).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--write',
|
||||
]);
|
||||
|
||||
const output = lines.join('\n');
|
||||
expect(output).toMatch(/do not restart|before.*restart/i);
|
||||
expect(output).toMatch(/env\.generated/);
|
||||
// The unit has no EnvironmentFile= directive, so the runbook must NOT tell the
|
||||
// operator to verify one — verify the launcher/generated file instead.
|
||||
expect(output).not.toContain('EnvironmentFile');
|
||||
expect(output).toMatch(/systemctl --user cat mosaic-agent@<name>/);
|
||||
expect(output).toMatch(/restart/i);
|
||||
});
|
||||
|
||||
it('emits paths and counts only — never the projected env body (secrev)', async (): Promise<void> => {
|
||||
const home = await fleetHome();
|
||||
const lines = capture();
|
||||
|
||||
await program(home, recordingRunner([])).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--write',
|
||||
]);
|
||||
|
||||
const output = lines.join('\n');
|
||||
// Relative paths + counts are fine; the rendered projection body (KEY=value
|
||||
// lines) must never be echoed to stdout.
|
||||
expect(output).toContain('coder0.env.generated');
|
||||
expect(output).not.toMatch(/^MOSAIC_AGENT_\w+=/m);
|
||||
expect(output).not.toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet');
|
||||
});
|
||||
|
||||
it('is projection-only: leaves legacy .env untouched and writes no .env.local/.env.quarantine', async (): Promise<void> => {
|
||||
// Recovery contract: regen rebuilds ONLY <name>.env.generated. It must never
|
||||
// relocate, quarantine, or unlink the operator-owned legacy .env surface — the
|
||||
// full reconciler apply path does, so regen must NOT use it.
|
||||
const home = await fleetHome();
|
||||
const legacyPath = join(home, 'fleet', 'agents', 'coder0.env');
|
||||
await writeFile(legacyPath, 'MOSAIC_RUNTIME_BIN=/usr/bin/pi\n', { mode: 0o600 });
|
||||
capture();
|
||||
|
||||
await program(home, recordingRunner([])).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--write',
|
||||
'--json',
|
||||
]);
|
||||
|
||||
// Generated projection rebuilt...
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
|
||||
// ...but the operator's legacy .env is preserved verbatim, and no local/quarantine
|
||||
// files were fabricated from it.
|
||||
expect(await readFile(legacyPath, 'utf8')).toBe('MOSAIC_RUNTIME_BIN=/usr/bin/pi\n');
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.local'))).toBe(false);
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.quarantine'))).toBe(false);
|
||||
});
|
||||
|
||||
it('prepares every agent before writing any: a later prepare failure leaves nothing written', async (): Promise<void> => {
|
||||
// Fail-closed across agents. Pre-seed coder1's projection with world-readable
|
||||
// perms so its prepare rejects; coder0 (valid) must NOT be written because
|
||||
// preparation is fully completed before the first apply.
|
||||
const home = await fleetHome();
|
||||
const coder1Path = join(home, 'fleet', 'agents', 'coder1.env.generated');
|
||||
await writeFile(coder1Path, 'MOSAIC_AGENT_NAME=coder1\n', { mode: 0o644 });
|
||||
const calls: string[][] = [];
|
||||
capture();
|
||||
const errors: string[] = [];
|
||||
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||
errors.push(String(chunk));
|
||||
return true;
|
||||
});
|
||||
|
||||
await program(home, recordingRunner(calls)).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--write',
|
||||
'--json',
|
||||
]);
|
||||
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(errors.join('')).toMatch(/regen failed/i);
|
||||
// coder0 is valid but must remain unwritten — no partial rebuild.
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('fails closed on a semantically invalid roster (protected-class tool-policy mismatch)', async (): Promise<void> => {
|
||||
// A hand-edited roster giving a protected class a weaker tool_policy is rejected
|
||||
// by reconcile/plan/verify; regen must enforce the SAME gate, not silently
|
||||
// project the downgraded policy into <name>.env.generated.
|
||||
const home = await fleetHome(`
|
||||
version: 2
|
||||
generation: 3
|
||||
transport: tmux
|
||||
tmux:
|
||||
socket_name: mosaic-fleet
|
||||
holder_session: _holder
|
||||
defaults:
|
||||
working_directory: /srv/mosaic
|
||||
runtime: pi
|
||||
runtimes:
|
||||
pi:
|
||||
reset_command: /new
|
||||
agents:
|
||||
- name: gate0
|
||||
alias: Gate 0
|
||||
class: merge-gate
|
||||
runtime: pi
|
||||
provider: openai
|
||||
model: gpt-5.6-sol
|
||||
reasoning: high
|
||||
tool_policy: code
|
||||
working_directory: /srv/mosaic
|
||||
persistent_persona: false
|
||||
reset_between_tasks: true
|
||||
lifecycle:
|
||||
enabled: true
|
||||
desired_state: stopped
|
||||
launch:
|
||||
yolo: true
|
||||
`);
|
||||
const calls: string[][] = [];
|
||||
capture();
|
||||
const errors: string[] = [];
|
||||
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||
errors.push(String(chunk));
|
||||
return true;
|
||||
});
|
||||
|
||||
await program(home, recordingRunner(calls)).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--write',
|
||||
'--json',
|
||||
]);
|
||||
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(errors.join('')).toMatch(/regen failed/i);
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'gate0.env.generated'))).toBe(false);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('refuses to write while a concurrent reconcile holds the mutation lock', async (): Promise<void> => {
|
||||
// regen --write mutates the same projections as reconcile; it must take the
|
||||
// shared reconcile lock so a concurrent reconcile cannot race a stale write.
|
||||
const home = await fleetHome();
|
||||
await writeFile(join(home, 'fleet', 'roster.yaml.reconcile.lock'), 'held\n', { mode: 0o600 });
|
||||
const calls: string[][] = [];
|
||||
capture();
|
||||
const errors: string[] = [];
|
||||
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||
errors.push(String(chunk));
|
||||
return true;
|
||||
});
|
||||
|
||||
await program(home, recordingRunner(calls)).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--write',
|
||||
'--json',
|
||||
]);
|
||||
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(errors.join('')).toMatch(/regen failed/i);
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('fails closed (non-zero, no writes) on an invalid roster', async (): Promise<void> => {
|
||||
// roster-v2 requires >= 1 agent; a malformed roster must abort regen without
|
||||
// writing any projection and without touching lifecycle.
|
||||
const home = await fleetHome('version: 2\ngeneration: 1\n');
|
||||
const calls: string[][] = [];
|
||||
capture();
|
||||
const errors: string[] = [];
|
||||
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||
errors.push(String(chunk));
|
||||
return true;
|
||||
});
|
||||
|
||||
await program(home, recordingRunner(calls)).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--write',
|
||||
'--json',
|
||||
]);
|
||||
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(errors.join('')).toMatch(/regen failed/i);
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('rejects a non-canonical --roster path (canonical roster only)', async (): Promise<void> => {
|
||||
// `fleet` exposes a global `--roster`; reconcile rejects a non-canonical value
|
||||
// rather than silently reading the canonical roster. regen must enforce the
|
||||
// SAME guard so `--roster /elsewhere regen --write` cannot mislead the operator.
|
||||
const home = await fleetHome();
|
||||
const calls: string[][] = [];
|
||||
capture();
|
||||
const errors: string[] = [];
|
||||
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||
errors.push(String(chunk));
|
||||
return true;
|
||||
});
|
||||
|
||||
await program(home, recordingRunner(calls)).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'--roster',
|
||||
join(home, 'other', 'roster.yaml'),
|
||||
'regen',
|
||||
'--write',
|
||||
'--json',
|
||||
]);
|
||||
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(errors.join('')).toMatch(/regen failed/i);
|
||||
expect(errors.join('')).toMatch(/canonical roster/i);
|
||||
// No projection written against a misdirected roster path.
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('reports an incomplete rebuild when a later projection write fails mid-loop', async (): Promise<void> => {
|
||||
// Fault injected on the SECOND apply: coder0 is already replaced on disk, so a
|
||||
// bare throw would hide the partial state. regen must surface which agents were
|
||||
// rebuilt and which failed, with a verify-before-restart recovery instruction.
|
||||
const home = await fleetHome();
|
||||
const realApply = applyPreparedGeneratedAgentEnvironmentProjection;
|
||||
const result = await executeFleetRegen(
|
||||
{
|
||||
runner: recordingRunner([]),
|
||||
mosaicHome: home,
|
||||
applyProjection: async (prepared): Promise<string> => {
|
||||
if (prepared.generatedPath.endsWith('coder1.env.generated')) {
|
||||
throw new Error('simulated ENOSPC on coder1');
|
||||
}
|
||||
return realApply(prepared);
|
||||
},
|
||||
},
|
||||
{ write: true },
|
||||
);
|
||||
|
||||
// coder0 written before the fault; coder1 not.
|
||||
expect(result.written).toBe(1);
|
||||
expect(result.incomplete).toMatchObject({
|
||||
code: 'projection-apply-failed',
|
||||
failedAgent: 'coder1',
|
||||
writtenAgents: ['coder0'],
|
||||
});
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
|
||||
|
||||
// The human report names the failed agent and directs a verify-before-restart
|
||||
// recovery — not a bare success runbook.
|
||||
const report = formatFleetRegenReport(result).join('\n');
|
||||
expect(report).toMatch(/incomplete/i);
|
||||
expect(report).toContain('coder1');
|
||||
expect(report).toMatch(/verify|re-run|retry/i);
|
||||
// Structural: still no lifecycle call anywhere on the failure path.
|
||||
expect(report).not.toMatch(/systemctl --user restart mosaic-agent@<name>\n.*\n/);
|
||||
});
|
||||
|
||||
it('refuses to write while agent CRUD holds the roster mutation lock', async (): Promise<void> => {
|
||||
// regen --write overwrites the SAME generated projections that `fleet agent
|
||||
// create/update/delete` writes under roster.yaml.mutation.lock. If regen only
|
||||
// took the reconcile lock it could read a stale roster and overwrite a just-
|
||||
// committed projection. It must also take the mutation lock and fail closed.
|
||||
const home = await fleetHome();
|
||||
await writeFile(join(home, 'fleet', 'roster.yaml.mutation.lock'), 'crud\n', { mode: 0o600 });
|
||||
const calls: string[][] = [];
|
||||
capture();
|
||||
const errors: string[] = [];
|
||||
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||
errors.push(String(chunk));
|
||||
return true;
|
||||
});
|
||||
|
||||
await program(home, recordingRunner(calls)).parseAsync([
|
||||
'node',
|
||||
'mosaic',
|
||||
'fleet',
|
||||
'regen',
|
||||
'--write',
|
||||
'--json',
|
||||
]);
|
||||
|
||||
expect(process.exitCode).toBe(1);
|
||||
expect(errors.join('')).toMatch(/regen failed/i);
|
||||
// No projection written against a roster a concurrent CRUD is mutating.
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
|
||||
expect(calls).toEqual([]);
|
||||
});
|
||||
|
||||
it('surfaces BOTH the roster failure and a stale-lock warning on a double fault', async (): Promise<void> => {
|
||||
// Worst case with nothing written: run() fails (invalid roster, fail-closed) AND
|
||||
// a lock release faults. The operator must be told the lock may be stale, not
|
||||
// just the roster error — otherwise the next regen/reconcile is silently blocked.
|
||||
const home = await fleetHome('version: 2\ngeneration: 1\n');
|
||||
const throwingRelease =
|
||||
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||
throw new Error('simulated lock unlink fault');
|
||||
};
|
||||
await expect(
|
||||
executeFleetRegen(
|
||||
{
|
||||
runner: recordingRunner([]),
|
||||
mosaicHome: home,
|
||||
acquireReconcileLock: throwingRelease,
|
||||
acquireRosterMutationLock: throwingRelease,
|
||||
},
|
||||
{ write: true },
|
||||
),
|
||||
).rejects.toThrow(/stale|inspect|lock/i);
|
||||
// Nothing written on the fail-closed path.
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||
});
|
||||
|
||||
it('propagates a roster-mutation-lock release failure instead of silently keeping a stale lock', async (): Promise<void> => {
|
||||
// Finding L: the real `acquirePrivateRosterMutationLock` release must SURFACE an
|
||||
// unlink failure, not swallow it. If it swallowed (like the CRUD-internal
|
||||
// acquireMutationLock does), a stale roster.yaml.mutation.lock left after a
|
||||
// successful regen would block later regen/agent CRUD while the command reported
|
||||
// success — and the finding-J stale-lock warning would never fire for this lock.
|
||||
const home = await fleetHome();
|
||||
const release = await acquirePrivateRosterMutationLock(home)();
|
||||
// Simulate the lock vanishing (or being unremovable) before release runs: the
|
||||
// underlying unlink then faults. A swallowing release would resolve and hide it.
|
||||
await rm(join(home, 'fleet', 'roster.yaml.mutation.lock'));
|
||||
await expect(release()).rejects.toThrow();
|
||||
});
|
||||
|
||||
it('preserves a complete rebuild result and flags lock-cleanup when release faults', async (): Promise<void> => {
|
||||
// A lock-release fault after a successful write must NOT discard the fact that
|
||||
// projections are now live — the operator needs to know the write happened and
|
||||
// that the shared lock may be stale, not just a bare "regen failed".
|
||||
const home = await fleetHome();
|
||||
const result = await executeFleetRegen(
|
||||
{
|
||||
runner: recordingRunner([]),
|
||||
mosaicHome: home,
|
||||
acquireReconcileLock:
|
||||
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||
throw new Error('simulated lock unlink fault');
|
||||
},
|
||||
},
|
||||
{ write: true },
|
||||
);
|
||||
|
||||
expect(result.written).toBe(2);
|
||||
expect(result.incomplete).toBeUndefined();
|
||||
expect(result.cleanup).toMatchObject({
|
||||
code: 'lock-cleanup-failed',
|
||||
action: 'inspect-lock-before-retry',
|
||||
});
|
||||
// Projections are genuinely on disk despite the release fault.
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(true);
|
||||
|
||||
const report = formatFleetRegenReport(result).join('\n');
|
||||
expect(report).toMatch(/lock/i);
|
||||
expect(report).toMatch(/stale|inspect|clear/i);
|
||||
});
|
||||
|
||||
it('preserves a partial rebuild AND flags lock-cleanup when both apply and release fault', async (): Promise<void> => {
|
||||
// Worst case: a mid-loop apply fault leaves a partial rebuild, then the release
|
||||
// also faults. Both recovery states must survive so the operator sees exactly
|
||||
// which projections are live and that the lock may be stale.
|
||||
const home = await fleetHome();
|
||||
const realApply = applyPreparedGeneratedAgentEnvironmentProjection;
|
||||
const result = await executeFleetRegen(
|
||||
{
|
||||
runner: recordingRunner([]),
|
||||
mosaicHome: home,
|
||||
applyProjection: async (prepared): Promise<string> => {
|
||||
if (prepared.generatedPath.endsWith('coder1.env.generated')) {
|
||||
throw new Error('simulated ENOSPC on coder1');
|
||||
}
|
||||
return realApply(prepared);
|
||||
},
|
||||
acquireReconcileLock:
|
||||
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||
throw new Error('simulated lock unlink fault');
|
||||
},
|
||||
},
|
||||
{ write: true },
|
||||
);
|
||||
|
||||
expect(result.written).toBe(1);
|
||||
expect(result.incomplete).toMatchObject({
|
||||
code: 'projection-apply-failed',
|
||||
failedAgent: 'coder1',
|
||||
writtenAgents: ['coder0'],
|
||||
});
|
||||
expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' });
|
||||
|
||||
const report = formatFleetRegenReport(result).join('\n');
|
||||
expect(report).toMatch(/incomplete/i);
|
||||
expect(report).toMatch(/lock/i);
|
||||
});
|
||||
|
||||
it('fails closed without deleting a REPLACEMENT roster-mutation lock it no longer owns', async (): Promise<void> => {
|
||||
// Finding M1 (replacement-lock race): after this invocation created its lock, the
|
||||
// lock is cleared and re-created by ANOTHER writer (new inode + foreign token)
|
||||
// BEFORE release runs. The release must PROVE ownership (device/inode + token) and
|
||||
// refuse to unlink the stranger's live lock — otherwise it deletes it, a third
|
||||
// writer enters concurrently, and mutual exclusion over the projections is broken.
|
||||
// This pins the REACHABLE window (replaced-before-release); the residual sub-window
|
||||
// between the final check and the path `unlink` is unreachable within the `wx`
|
||||
// writer protocol and matches the merged reconcile lock — see the acquirer doc.
|
||||
const home = await fleetHome();
|
||||
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
|
||||
const release = await acquirePrivateRosterMutationLock(home)();
|
||||
// A different writer replaces the lock: same path, NEW inode, foreign content.
|
||||
await rm(lockPath);
|
||||
await writeFile(lockPath, 'a-different-writer\n', { mode: 0o600 });
|
||||
const replacement = await stat(lockPath);
|
||||
// Release must reject (it no longer owns this file) AND the diagnosis must name
|
||||
// the MUTATION lock specifically — a fault on this shared helper must not be
|
||||
// mislabeled as the reconcile lock (that would defeat the M3 accurate-diagnosis
|
||||
// goal, since regen serializes on both) ...
|
||||
await expect(release()).rejects.toThrow(/roster\.yaml\.mutation\.lock/);
|
||||
// ... and MUST NOT have unlinked the stranger's live lock.
|
||||
expect(await exists(lockPath)).toBe(true);
|
||||
expect((await stat(lockPath)).ino).toBe(replacement.ino);
|
||||
});
|
||||
|
||||
it('does not strand the lock file when initialization fails after creating it', async (): Promise<void> => {
|
||||
// Codex r4: if writeFile/stat/close/ownership-check throws AFTER the `wx` create
|
||||
// succeeds, the just-created lock must NOT be left behind — a stranded lock would
|
||||
// permanently block future regen AND agent CRUD (both contend on this path). The
|
||||
// cleanup is dev/ino-guarded (never deletes a replacement) and best-effort (never
|
||||
// masks the initialization error).
|
||||
const home = await fleetHome();
|
||||
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
|
||||
// Inject an opener that performs the REAL `wx` create (so the lock file really
|
||||
// lands on disk) but whose handle faults on the token write — a post-create
|
||||
// initialization failure.
|
||||
const faultingOpen = (async (
|
||||
path: Parameters<typeof open>[0],
|
||||
flags: Parameters<typeof open>[1],
|
||||
mode: Parameters<typeof open>[2],
|
||||
) => {
|
||||
const handle = await open(path, flags, mode);
|
||||
return new Proxy(handle, {
|
||||
get(target, prop, receiver): unknown {
|
||||
if (prop === 'writeFile') {
|
||||
return async (): Promise<never> => {
|
||||
throw new Error('simulated ENOSPC on token write');
|
||||
};
|
||||
}
|
||||
const value = Reflect.get(target, prop, receiver);
|
||||
return typeof value === 'function' ? value.bind(target) : value;
|
||||
},
|
||||
});
|
||||
}) as typeof open;
|
||||
|
||||
await expect(acquirePrivateRosterMutationLock(home, faultingOpen)()).rejects.toThrow(
|
||||
/mutation\.lock/,
|
||||
);
|
||||
// The lock we created must have been cleaned up — not stranded on disk.
|
||||
expect(await exists(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
it('does not strand the lock file when the post-create stat itself fails', async (): Promise<void> => {
|
||||
// Codex r5: the narrower sub-path where `handle.stat()` ITSELF rejects right after
|
||||
// the `wx` create succeeds (transient EIO/EBADF). device/inode identity is then
|
||||
// unavailable, so the init-failure cleanup cannot prove ownership by dev/ino — yet
|
||||
// the lock must STILL not be stranded, because a stranded lock permanently blocks
|
||||
// future regen AND agent CRUD. The token we persisted before the failure uniquely
|
||||
// identifies OUR lock, so cleanup falls back to matching it.
|
||||
const home = await fleetHome();
|
||||
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
|
||||
// Real `wx` create (lock really lands on disk); the token write SUCCEEDS so our
|
||||
// random token is persisted, then `stat()` faults — a post-create init failure.
|
||||
const faultingStatOpen = (async (
|
||||
path: Parameters<typeof open>[0],
|
||||
flags: Parameters<typeof open>[1],
|
||||
mode: Parameters<typeof open>[2],
|
||||
) => {
|
||||
const handle = await open(path, flags, mode);
|
||||
return new Proxy(handle, {
|
||||
get(target, prop, receiver): unknown {
|
||||
if (prop === 'stat') {
|
||||
return async (): Promise<never> => {
|
||||
throw new Error('simulated EIO on post-create stat');
|
||||
};
|
||||
}
|
||||
const value = Reflect.get(target, prop, receiver);
|
||||
return typeof value === 'function' ? value.bind(target) : value;
|
||||
},
|
||||
});
|
||||
}) as typeof open;
|
||||
|
||||
await expect(acquirePrivateRosterMutationLock(home, faultingStatOpen)()).rejects.toThrow(
|
||||
/mutation\.lock/,
|
||||
);
|
||||
// Even without dev/ino, the token-based fallback must have removed our lock.
|
||||
expect(await exists(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
it('surfaces a lock-cleanup fault when unwinding after a later lock acquire fails', async (): Promise<void> => {
|
||||
// Finding M2: the FIRST lock is taken, the SECOND lock's acquire throws, and
|
||||
// unwinding the first lock's release ALSO faults. The acquire-unwind path must
|
||||
// surface both — the already-taken lock may now be stale and silently block the
|
||||
// next regen/reconcile — not just re-throw the acquire error and drop the fault.
|
||||
const home = await fleetHome();
|
||||
await expect(
|
||||
executeFleetRegen(
|
||||
{
|
||||
runner: recordingRunner([]),
|
||||
mosaicHome: home,
|
||||
// Mutation lock (acquired first) succeeds, but its RELEASE faults on unwind.
|
||||
acquireRosterMutationLock:
|
||||
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||
throw new Error('simulated mutation-lock unlink fault');
|
||||
},
|
||||
// Reconcile lock (acquired second) ACQUIRE fails, triggering the unwind.
|
||||
acquireReconcileLock: () => async (): Promise<() => Promise<void>> => {
|
||||
throw new Error('simulated reconcile acquire failure');
|
||||
},
|
||||
},
|
||||
{ write: true },
|
||||
),
|
||||
).rejects.toThrow(/stale|inspect|lock/i);
|
||||
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||
});
|
||||
|
||||
it('names BOTH fleet locks in the stale-lock warning (cleanup can come from either)', async (): Promise<void> => {
|
||||
// Finding M3: finding L made the roster MUTATION lock's release fault reachable, so
|
||||
// the cleanup marker can now originate from either lock. The human report must not
|
||||
// claim only the reconcile lock is stale — it must name both candidate lock files
|
||||
// so the operator inspects the right one.
|
||||
const home = await fleetHome();
|
||||
const result = await executeFleetRegen(
|
||||
{
|
||||
runner: recordingRunner([]),
|
||||
mosaicHome: home,
|
||||
// Fault the MUTATION lock's release specifically (reconcile lock is real).
|
||||
acquireRosterMutationLock:
|
||||
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||
throw new Error('simulated mutation-lock unlink fault');
|
||||
},
|
||||
},
|
||||
{ write: true },
|
||||
);
|
||||
|
||||
expect(result.written).toBe(2);
|
||||
expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' });
|
||||
const report = formatFleetRegenReport(result).join('\n');
|
||||
expect(report).toContain('roster.yaml.mutation.lock');
|
||||
expect(report).toContain('roster.yaml.reconcile.lock');
|
||||
});
|
||||
});
|
||||
441
packages/mosaic/src/commands/fleet-regen-command.ts
Normal file
441
packages/mosaic/src/commands/fleet-regen-command.ts
Normal file
@@ -0,0 +1,441 @@
|
||||
import { readFile, stat } from 'node:fs/promises';
|
||||
import { homedir } from 'node:os';
|
||||
import { join, relative, resolve } from 'node:path';
|
||||
import type { Command } from 'commander';
|
||||
import type { CommandRunner } from './fleet.js';
|
||||
import {
|
||||
applyPreparedGeneratedAgentEnvironmentProjection,
|
||||
prepareGeneratedAgentEnvironmentProjection,
|
||||
type PreparedGeneratedAgentEnvironmentProjection,
|
||||
} from '../fleet/generated-env-boundary.js';
|
||||
import {
|
||||
acquirePrivateReconcileLock,
|
||||
acquirePrivateRosterMutationLock,
|
||||
projectRosterV2AgentGeneratedEnv,
|
||||
} from '../fleet/fleet-reconciler.js';
|
||||
import {
|
||||
parseRosterV2,
|
||||
validateRosterV2Semantics,
|
||||
type FleetRosterV2,
|
||||
} from '../fleet/roster-v2.js';
|
||||
|
||||
/**
|
||||
* `mosaic fleet regen` — recovery-framed regeneration of the roster-derived
|
||||
* agent env projections (`fleet/agents/<name>.env.generated`) from the roster
|
||||
* SSOT. It is the recovery layer of #791: when a wiped/diverged operator surface
|
||||
* has cost an agent its generated projection, regen rebuilds it deterministically
|
||||
* from `roster.yaml` so the launcher can source the intended identity again.
|
||||
*
|
||||
* It is intentionally a THIN projection-only wrapper over the same mapping the
|
||||
* reconciler apply path uses ({@link projectRosterV2AgentGeneratedEnv}), and it
|
||||
* has NO code path to systemd lifecycle: regen never starts, stops, or restarts
|
||||
* an agent. Recovery order forbids restart-before-verify, so the operator must
|
||||
* verify each rebuilt projection resolves before restarting anything.
|
||||
*/
|
||||
export interface FleetRegenAgentPlan {
|
||||
readonly name: string;
|
||||
/** Generated-projection path, relative to the Mosaic home (never absolute in output). */
|
||||
readonly path: string;
|
||||
/** `create` when the projection is currently absent, `rebuild` when it already exists. */
|
||||
readonly disposition: 'create' | 'rebuild';
|
||||
}
|
||||
|
||||
/**
|
||||
* Present ONLY when a `--write` projection apply failed partway through the
|
||||
* fleet. Every prior agent was already validated and prepared, so `writtenAgents`
|
||||
* are byte-complete on disk; `failedAgent` is where the write faulted. The
|
||||
* command surfaces this instead of a bare throw so the operator can finish the
|
||||
* rebuild and verify every projection before restarting any unit.
|
||||
*/
|
||||
export interface FleetRegenIncomplete {
|
||||
readonly code: 'projection-apply-failed';
|
||||
/** The agent whose generated-projection write faulted. */
|
||||
readonly failedAgent: string;
|
||||
/** Agents whose projections were fully written before the fault (in apply order). */
|
||||
readonly writtenAgents: readonly string[];
|
||||
}
|
||||
|
||||
export interface FleetRegenResult {
|
||||
readonly mode: 'dry-run' | 'write';
|
||||
/** Roster path, relative to the Mosaic home. */
|
||||
readonly rosterPath: string;
|
||||
readonly generation: number;
|
||||
readonly agentCount: number;
|
||||
/** Count of projections written to disk (always 0 in dry-run). */
|
||||
readonly written: number;
|
||||
readonly agents: readonly FleetRegenAgentPlan[];
|
||||
/** Set ONLY when a `--write` apply faulted mid-fleet — a partial rebuild. */
|
||||
readonly incomplete?: FleetRegenIncomplete;
|
||||
/**
|
||||
* Set ONLY when a shared fleet lock could not be released after a write completed
|
||||
* (or partially completed) — EITHER `roster.yaml.mutation.lock` or
|
||||
* `roster.yaml.reconcile.lock`, since regen holds both. The projections in this
|
||||
* result are real; the lock file may be stale and must be inspected before the
|
||||
* next mutation. Independent of {@link incomplete} — both can be present at once.
|
||||
*/
|
||||
readonly cleanup?: {
|
||||
readonly code: 'lock-cleanup-failed';
|
||||
readonly action: 'inspect-lock-before-retry';
|
||||
};
|
||||
}
|
||||
|
||||
export interface FleetRegenDeps {
|
||||
/**
|
||||
* Present ONLY so an accidental future lifecycle wiring is caught by the
|
||||
* "never restarts" test — regen is contractually forbidden to invoke it.
|
||||
*/
|
||||
readonly runner: CommandRunner;
|
||||
readonly mosaicHome?: string;
|
||||
/** Persona roots for semantic roster validation (defaults mirror the reconciler). */
|
||||
readonly rolesDir?: string;
|
||||
readonly overrideDir?: string;
|
||||
/** Test seam: canonical roster reader. Defaults to parse + semantic validation. */
|
||||
readonly readRoster?: (rosterPath: string) => Promise<FleetRosterV2>;
|
||||
/**
|
||||
* Test seam: generated-only projection validator. Defaults to the audited
|
||||
* boundary helper that validates ONLY `<name>.env.generated`.
|
||||
*/
|
||||
readonly prepareProjection?: typeof prepareGeneratedAgentEnvironmentProjection;
|
||||
/** Test seam: generated-only projection writer. Defaults to the audited boundary helper. */
|
||||
readonly applyProjection?: typeof applyPreparedGeneratedAgentEnvironmentProjection;
|
||||
/**
|
||||
* Test seam: acquire the shared reconcile lock before a `--write`. Defaults to
|
||||
* the reconciler's private lock so regen and reconcile are mutually exclusive
|
||||
* and a concurrent reconcile cannot race a stale projection write.
|
||||
*/
|
||||
readonly acquireReconcileLock?: (mosaicHome: string) => () => Promise<() => Promise<void>>;
|
||||
/**
|
||||
* Test seam: acquire the shared roster mutation lock before a `--write`.
|
||||
* Defaults to the reconciler's hardened, ownership-proving acquirer for the
|
||||
* same `fleet/roster.yaml.mutation.lock` path that CRUD contends on, so regen
|
||||
* serializes against agent create/update/delete — both rewrite the same
|
||||
* generated projections, so without this a concurrent CRUD could race a stale
|
||||
* projection write.
|
||||
*/
|
||||
readonly acquireRosterMutationLock?: (mosaicHome: string) => () => Promise<() => Promise<void>>;
|
||||
/** Test seam: existence probe for create/rebuild disposition. */
|
||||
readonly fileExists?: (path: string) => Promise<boolean>;
|
||||
}
|
||||
|
||||
interface FleetRegenOptions {
|
||||
readonly write?: boolean;
|
||||
}
|
||||
|
||||
function defaultMosaicHome(deps: FleetRegenDeps): string {
|
||||
return deps.mosaicHome ?? process.env['MOSAIC_HOME'] ?? join(homedir(), '.config', 'mosaic');
|
||||
}
|
||||
|
||||
async function defaultFileExists(path: string): Promise<boolean> {
|
||||
try {
|
||||
await stat(path);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Projection-only regeneration core. Deterministic and side-effect-free in
|
||||
* dry-run; in `--write` it rebuilds each generated projection through the
|
||||
* audited generated-only boundary writer and NOTHING else — it never writes
|
||||
* `.env.local`/`.env.quarantine`, never unlinks the legacy `.env`, and never
|
||||
* touches `deps.runner`.
|
||||
*
|
||||
* Fail-closed by construction: the roster is validated (structural + semantic)
|
||||
* and EVERY agent's projection is prepared before ANY is written, so a single
|
||||
* invalid agent (or a semantically rejected roster) leaves the whole surface
|
||||
* untouched. In `--write` mode the read-prepare-apply sequence runs under the
|
||||
* shared reconcile lock so a concurrent reconcile cannot race a stale write.
|
||||
*/
|
||||
export async function executeFleetRegen(
|
||||
deps: FleetRegenDeps,
|
||||
options: FleetRegenOptions,
|
||||
): Promise<FleetRegenResult> {
|
||||
const mosaicHome = defaultMosaicHome(deps);
|
||||
const agentEnvDir = join(mosaicHome, 'fleet', 'agents');
|
||||
const rosterPath = join(mosaicHome, 'fleet', 'roster.yaml');
|
||||
const readRoster = deps.readRoster ?? defaultReadRoster(deps, mosaicHome);
|
||||
const prepare = deps.prepareProjection ?? prepareGeneratedAgentEnvironmentProjection;
|
||||
const apply = deps.applyProjection ?? applyPreparedGeneratedAgentEnvironmentProjection;
|
||||
const fileExists = deps.fileExists ?? defaultFileExists;
|
||||
const acquireReconcileLock = deps.acquireReconcileLock ?? acquirePrivateReconcileLock;
|
||||
const acquireRosterMutationLock =
|
||||
deps.acquireRosterMutationLock ?? acquirePrivateRosterMutationLock;
|
||||
const write = options.write === true;
|
||||
|
||||
const run = async (): Promise<FleetRegenResult> => {
|
||||
const roster = await readRoster(rosterPath);
|
||||
|
||||
// Prepare (validate) every agent BEFORE any write, so a later failure never
|
||||
// leaves an earlier projection partially rebuilt.
|
||||
const prepared: (PreparedGeneratedAgentEnvironmentProjection & {
|
||||
readonly name: string;
|
||||
readonly disposition: 'create' | 'rebuild';
|
||||
})[] = [];
|
||||
for (const agent of roster.agents) {
|
||||
const projection = await prepare({
|
||||
mosaicHome,
|
||||
agentEnvDir,
|
||||
agentName: agent.name,
|
||||
generated: projectRosterV2AgentGeneratedEnv(roster, agent),
|
||||
});
|
||||
const disposition = (await fileExists(projection.generatedPath)) ? 'rebuild' : 'create';
|
||||
prepared.push({ ...projection, name: agent.name, disposition });
|
||||
}
|
||||
|
||||
// Every projection is already validated (prepared); an apply here can only
|
||||
// fault on the underlying I/O (ENOSPC, EIO, a race outside the lock). If a
|
||||
// later write faults, earlier agents are already replaced on disk, so we
|
||||
// stop and report the partial state rather than throwing a bare error that
|
||||
// hides which projections are now live.
|
||||
let written = 0;
|
||||
let incomplete: FleetRegenIncomplete | undefined;
|
||||
if (write) {
|
||||
for (const projection of prepared) {
|
||||
try {
|
||||
await apply(projection);
|
||||
} catch {
|
||||
incomplete = {
|
||||
code: 'projection-apply-failed',
|
||||
failedAgent: projection.name,
|
||||
writtenAgents: prepared.slice(0, written).map((entry) => entry.name),
|
||||
};
|
||||
break;
|
||||
}
|
||||
written += 1;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
mode: write ? 'write' : 'dry-run',
|
||||
rosterPath: relative(mosaicHome, rosterPath),
|
||||
generation: roster.generation,
|
||||
agentCount: roster.agents.length,
|
||||
written,
|
||||
agents: prepared.map((projection) => ({
|
||||
name: projection.name,
|
||||
path: relative(mosaicHome, projection.generatedPath),
|
||||
disposition: projection.disposition,
|
||||
})),
|
||||
...(incomplete ? { incomplete } : {}),
|
||||
};
|
||||
};
|
||||
|
||||
// Dry-run is preview-only and never mutates, so it stays lock-free (usable even
|
||||
// while a reconcile or CRUD holds a lock). A `--write` must serialize against
|
||||
// BOTH roster writers of the same generated projections: agent CRUD (roster
|
||||
// mutation lock) and reconcile (reconcile lock). Acquire in a fixed order — the
|
||||
// locks are non-blocking (they throw on contention rather than wait), so no
|
||||
// deadlock is possible — and fail closed if either is already held.
|
||||
if (!write) return run();
|
||||
|
||||
const releases: (() => Promise<void>)[] = [];
|
||||
try {
|
||||
releases.push(await acquireRosterMutationLock(mosaicHome)());
|
||||
releases.push(await acquireReconcileLock(mosaicHome)());
|
||||
} catch (error: unknown) {
|
||||
// A later acquire failed; unwind any lock already taken (reverse order). If
|
||||
// that unwind ALSO faults, surface both — the already-taken lock may now be
|
||||
// stale and silently block the next regen/reconcile — rather than dropping it.
|
||||
const releaseFault = await releaseFleetLocks(releases);
|
||||
throw augmentWithLockCleanupFault(error, releaseFault);
|
||||
}
|
||||
|
||||
let result: FleetRegenResult | undefined;
|
||||
let primaryError: unknown;
|
||||
try {
|
||||
result = await run();
|
||||
} catch (error: unknown) {
|
||||
primaryError = error;
|
||||
}
|
||||
|
||||
// Release both locks (reverse acquire order), continuing past a fault so neither
|
||||
// leaks. A release fault must never discard a completed (or partial) rebuild —
|
||||
// the projections are already on disk — and must not be silently hidden even
|
||||
// when the run itself failed with nothing written.
|
||||
const releaseFault = await releaseFleetLocks(releases);
|
||||
if (result) {
|
||||
return releaseFault === undefined
|
||||
? result
|
||||
: {
|
||||
...result,
|
||||
cleanup: { code: 'lock-cleanup-failed', action: 'inspect-lock-before-retry' },
|
||||
};
|
||||
}
|
||||
// result is undefined ⇒ run() threw ⇒ primaryError is set.
|
||||
throw augmentWithLockCleanupFault(primaryError, releaseFault);
|
||||
}
|
||||
|
||||
/**
|
||||
* Releases held locks in reverse acquire order, continuing past a fault so a
|
||||
* single failed release never leaks the others. Returns the first fault seen (or
|
||||
* undefined when every release succeeded).
|
||||
*/
|
||||
async function releaseFleetLocks(releases: readonly (() => Promise<void>)[]): Promise<unknown> {
|
||||
let firstFault: unknown;
|
||||
for (const release of [...releases].reverse()) {
|
||||
try {
|
||||
await release();
|
||||
} catch (error: unknown) {
|
||||
if (firstFault === undefined) firstFault = error;
|
||||
}
|
||||
}
|
||||
return firstFault;
|
||||
}
|
||||
|
||||
/**
|
||||
* When a run failed with nothing written AND lock cleanup also faulted, surface
|
||||
* both: the operator needs to know the lock may be stale (else the next
|
||||
* regen/reconcile is silently blocked), not just the original run error.
|
||||
*/
|
||||
function augmentWithLockCleanupFault(primaryError: unknown, releaseFault: unknown): unknown {
|
||||
if (releaseFault === undefined) return primaryError;
|
||||
const base = primaryError instanceof Error ? primaryError.message : String(primaryError);
|
||||
return new Error(
|
||||
`${base} (additionally, a fleet lock could not be released — ` +
|
||||
'fleet/roster.yaml.mutation.lock or roster.yaml.reconcile.lock may be stale; ' +
|
||||
'inspect and clear it before retry)',
|
||||
);
|
||||
}
|
||||
|
||||
function defaultReadRoster(
|
||||
deps: FleetRegenDeps,
|
||||
mosaicHome: string,
|
||||
): (rosterPath: string) => Promise<FleetRosterV2> {
|
||||
return async (rosterPath: string): Promise<FleetRosterV2> => {
|
||||
const roster = parseRosterV2(await readFile(rosterPath, 'utf8'), 'yaml');
|
||||
// Enforce the SAME semantic gate as reconcile/plan/verify (persona resolution
|
||||
// + protected-class tool-policy match) so regen cannot project a roster the
|
||||
// rest of the fleet surface would reject.
|
||||
await validateRosterV2Semantics(roster, {
|
||||
rolesDir: deps.rolesDir ?? join(mosaicHome, 'fleet', 'roles'),
|
||||
overrideDir: deps.overrideDir ?? join(mosaicHome, 'fleet', 'roles.local'),
|
||||
});
|
||||
return roster;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Human-readable report — paths and counts ONLY. The rendered projection body
|
||||
* (KEY=value lines) is never echoed (secrev). In `--write` mode it prints the
|
||||
* recovery runbook: verify each projection resolves BEFORE any restart.
|
||||
*/
|
||||
export function formatFleetRegenReport(result: FleetRegenResult): string[] {
|
||||
const lines: string[] = [];
|
||||
const header =
|
||||
result.mode === 'dry-run'
|
||||
? 'mosaic fleet regen — dry run (no changes written)'
|
||||
: 'mosaic fleet regen — wrote roster-derived projections';
|
||||
lines.push(header);
|
||||
lines.push(
|
||||
`roster: ${result.rosterPath} (generation ${result.generation}, ${result.agentCount} agent(s))`,
|
||||
);
|
||||
for (const agent of result.agents) {
|
||||
lines.push(` ${agent.name} ${agent.path} [${agent.disposition}]`);
|
||||
}
|
||||
|
||||
if (result.mode === 'dry-run') {
|
||||
lines.push(
|
||||
`Plan: would write ${result.agentCount} generated projection(s). ` +
|
||||
`Re-run with --write to apply. regen never restarts agents.`,
|
||||
);
|
||||
return lines;
|
||||
}
|
||||
|
||||
if (result.incomplete) {
|
||||
const { failedAgent, writtenAgents } = result.incomplete;
|
||||
lines.push(
|
||||
`INCOMPLETE: rebuild stopped at agent ${failedAgent} — its projection write faulted.`,
|
||||
);
|
||||
lines.push(
|
||||
`Rebuilt before the fault: ${writtenAgents.length} projection(s)` +
|
||||
(writtenAgents.length > 0 ? ` (${writtenAgents.join(', ')})` : '') +
|
||||
`; ${result.agentCount - writtenAgents.length} not written.`,
|
||||
);
|
||||
lines.push(
|
||||
'Recover: re-run `mosaic fleet regen --write` to finish, then VERIFY every ' +
|
||||
"agent's fleet/agents/<name>.env.generated resolves BEFORE restarting any " +
|
||||
'unit. regen never restarts agents.',
|
||||
);
|
||||
} else {
|
||||
lines.push(`Wrote ${result.written} generated projection(s).`);
|
||||
lines.push('Next steps — DO NOT restart agents before verifying:');
|
||||
lines.push(
|
||||
" 1. Confirm each agent's fleet/agents/<name>.env.generated exists and carries " +
|
||||
'the intended MOSAIC_AGENT_* values.',
|
||||
);
|
||||
// The unit sets NO EnvironmentFile= — it launches from a minimal env and the
|
||||
// launcher (start-agent-session.sh) sources .env.generated itself. Verify the
|
||||
// launcher path, not a nonexistent EnvironmentFile property.
|
||||
lines.push(
|
||||
' 2. Confirm the unit launches from it: `systemctl --user cat mosaic-agent@<name>` ' +
|
||||
'shows ExecStart running start-agent-session.sh, which reads .env.generated.',
|
||||
);
|
||||
lines.push(
|
||||
' 3. Only then restart one unit at a time: systemctl --user restart mosaic-agent@<name>.',
|
||||
);
|
||||
}
|
||||
|
||||
if (result.cleanup) {
|
||||
lines.push(
|
||||
'WARNING: a fleet lock could not be released — ' +
|
||||
'fleet/roster.yaml.mutation.lock or fleet/roster.yaml.reconcile.lock may be ' +
|
||||
'stale; inspect and clear it before the next reconcile or regen.',
|
||||
);
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
function resolveRegenMosaicHome(fleetCommand: Command, deps: FleetRegenDeps): string {
|
||||
const options = fleetCommand.optsWithGlobals<{ mosaicHome?: string }>();
|
||||
return options.mosaicHome ?? defaultMosaicHome(deps);
|
||||
}
|
||||
|
||||
/**
|
||||
* regen reads the roster SSOT at `<mosaicHome>/fleet/roster.yaml` and ignores the
|
||||
* `fleet --roster <path>` global. If an operator passes a non-canonical `--roster`
|
||||
* they must NOT be silently served the canonical file — mirror reconcile's guard
|
||||
* and reject, so `mosaic fleet --roster /elsewhere regen --write` fails closed.
|
||||
*/
|
||||
function assertRegenCanonicalRoster(fleetCommand: Command, mosaicHome: string): void {
|
||||
const options = fleetCommand.optsWithGlobals<{ roster?: string }>();
|
||||
const canonical = join(mosaicHome, 'fleet', 'roster.yaml');
|
||||
if (options.roster !== undefined && resolve(options.roster) !== resolve(canonical)) {
|
||||
throw new Error(
|
||||
'Roster-v2 regeneration requires the canonical roster path (fleet/roster.yaml).',
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** Registers the recovery-framed `mosaic fleet regen` command. */
|
||||
export function registerFleetRegenCommand(fleetCommand: Command, deps: FleetRegenDeps): void {
|
||||
fleetCommand
|
||||
.command('regen')
|
||||
.description(
|
||||
'Regenerate roster-derived agent env projections (dry-run default, --write to apply; never restarts)',
|
||||
)
|
||||
.option('--write', 'Write the regenerated projections to disk (default: dry-run)')
|
||||
.option('--json', 'Emit the plan/result as JSON')
|
||||
.action(async (opts: { write?: boolean; json?: boolean }): Promise<void> => {
|
||||
const mosaicHome = resolveRegenMosaicHome(fleetCommand, deps);
|
||||
try {
|
||||
assertRegenCanonicalRoster(fleetCommand, mosaicHome);
|
||||
const result = await executeFleetRegen(
|
||||
{ ...deps, mosaicHome },
|
||||
{ write: opts.write === true },
|
||||
);
|
||||
if (opts.json === true) {
|
||||
console.log(JSON.stringify(result));
|
||||
} else {
|
||||
for (const line of formatFleetRegenReport(result)) console.log(line);
|
||||
}
|
||||
// A partial rebuild or a stale-lock cleanup state is a non-zero outcome:
|
||||
// the operator must finish/verify (and clear the lock) before restarting.
|
||||
if (result.incomplete || result.cleanup) process.exitCode = 1;
|
||||
} catch (error: unknown) {
|
||||
process.exitCode = 1;
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
process.stderr.write(`mosaic fleet regen failed: ${message}\n`);
|
||||
}
|
||||
});
|
||||
}
|
||||
@@ -97,6 +97,7 @@ describe('registerFleetCommand', () => {
|
||||
'provision',
|
||||
'ps',
|
||||
'reconcile',
|
||||
'regen',
|
||||
'remove',
|
||||
'restart',
|
||||
'start',
|
||||
|
||||
@@ -44,6 +44,7 @@ import {
|
||||
registerFleetReconcilerCommands,
|
||||
type FleetReconcilerCommandDeps,
|
||||
} from './fleet-reconciler-command.js';
|
||||
import { registerFleetRegenCommand } from './fleet-regen-command.js';
|
||||
import { resolveCommsBlock } from '../fleet/comms-onboarding.js';
|
||||
import {
|
||||
applyPreparedAgentEnvironmentProjection,
|
||||
@@ -2063,6 +2064,18 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
||||
reconcileDeps: deps.reconcileDeps,
|
||||
});
|
||||
|
||||
// Recovery (#791 PR3): rebuild roster-derived env projections from the roster
|
||||
// SSOT. Projection-only and preview-first — never issues a lifecycle restart.
|
||||
registerFleetRegenCommand(cmd, {
|
||||
runner,
|
||||
mosaicHome: deps.mosaicHome,
|
||||
// Resolve personas the SAME way reconcile does: forward any configured roots
|
||||
// so a custom-persona-root deployment cannot have reconcile accept a roster
|
||||
// that regen then rejects against the default `<mosaicHome>/fleet/roles`.
|
||||
rolesDir: deps.reconcileDeps?.rolesDir,
|
||||
overrideDir: deps.reconcileDeps?.overrideDir,
|
||||
});
|
||||
|
||||
return cmd;
|
||||
}
|
||||
|
||||
|
||||
466
packages/mosaic/src/commands/restore.spec.ts
Normal file
466
packages/mosaic/src/commands/restore.spec.ts
Normal file
@@ -0,0 +1,466 @@
|
||||
/**
|
||||
* Tests for `mosaic restore` (#791 PR2 Task 12).
|
||||
*
|
||||
* The durable pre-update snapshot (install.sh: make_durable_snapshot) writes the
|
||||
* operator-owned surface to $XDG_STATE_HOME/mosaic/backups/pre-update-<ts>/ with
|
||||
* 0700 dirs / 0600 files. `mosaic restore` is the recovery counterpart:
|
||||
* • --list (default) enumerate snapshots by timestamp — dry-run, never mutates.
|
||||
* • --from <ts> restore that snapshot over MOSAIC_HOME, confirmation-gated.
|
||||
* It reports counts and relative paths ONLY — a snapshot may contain secrets
|
||||
* (credentials.json), so no file content is ever printed (secrev invariant).
|
||||
*/
|
||||
|
||||
import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest';
|
||||
import {
|
||||
mkdtempSync,
|
||||
rmSync,
|
||||
mkdirSync,
|
||||
writeFileSync,
|
||||
readFileSync,
|
||||
statSync,
|
||||
lstatSync,
|
||||
symlinkSync,
|
||||
} from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { tmpdir, homedir } from 'node:os';
|
||||
import { Command } from 'commander';
|
||||
import {
|
||||
resolveBackupRoot,
|
||||
listSnapshots,
|
||||
resolveSnapshotDir,
|
||||
planRestore,
|
||||
applyRestore,
|
||||
runRestore,
|
||||
registerRestoreCommand,
|
||||
} from './restore.js';
|
||||
|
||||
const SECRET = 'SUPER-SECRET-TOKEN-do-not-log-restore';
|
||||
|
||||
function seedSnapshot(root: string, ts: string, files: Record<string, string>): string {
|
||||
const dir = join(root, `pre-update-${ts}`);
|
||||
for (const [rel, content] of Object.entries(files)) {
|
||||
const abs = join(dir, rel);
|
||||
mkdirSync(join(abs, '..'), { recursive: true });
|
||||
writeFileSync(abs, content);
|
||||
}
|
||||
return dir;
|
||||
}
|
||||
|
||||
describe('mosaic restore (#791 PR2)', () => {
|
||||
let tmp: string;
|
||||
let backups: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmp = mkdtempSync(join(tmpdir(), 'mosaic-restore-'));
|
||||
backups = join(tmp, 'state', 'mosaic', 'backups');
|
||||
mkdirSync(backups, { recursive: true });
|
||||
vi.clearAllMocks();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe('resolveBackupRoot', () => {
|
||||
it('honors XDG_STATE_HOME', () => {
|
||||
expect(resolveBackupRoot({ XDG_STATE_HOME: '/x/state' })).toBe('/x/state/mosaic/backups');
|
||||
});
|
||||
it('falls back to ~/.local/state', () => {
|
||||
expect(resolveBackupRoot({})).toBe(join(homedir(), '.local', 'state', 'mosaic', 'backups'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('listSnapshots', () => {
|
||||
it('returns [] when the backup root does not exist', () => {
|
||||
expect(listSnapshots(join(tmp, 'nope'))).toEqual([]);
|
||||
});
|
||||
|
||||
it('lists pre-update snapshots newest-first with file counts, ignoring other dirs', () => {
|
||||
seedSnapshot(backups, '20240101T000000Z', { 'SOUL.md': 'a' });
|
||||
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'b', 'agents/x.conf': 'c' });
|
||||
mkdirSync(join(backups, 'unrelated-dir'), { recursive: true });
|
||||
|
||||
const snaps = listSnapshots(backups);
|
||||
expect(snaps.map((s) => s.timestamp)).toEqual(['20260101T000000Z', '20240101T000000Z']);
|
||||
expect(snaps[0]!.fileCount).toBe(2);
|
||||
expect(snaps[1]!.fileCount).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveSnapshotDir', () => {
|
||||
it('resolves by bare timestamp and by full pre-update-<ts> name', () => {
|
||||
const dir = seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'a' });
|
||||
expect(resolveSnapshotDir(backups, '20260101T000000Z')).toBe(dir);
|
||||
expect(resolveSnapshotDir(backups, 'pre-update-20260101T000000Z')).toBe(dir);
|
||||
});
|
||||
it('returns undefined for an unknown timestamp', () => {
|
||||
expect(resolveSnapshotDir(backups, '19990101T000000Z')).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe('planRestore', () => {
|
||||
it('walks nested dirs and returns every relative file path', () => {
|
||||
const dir = seedSnapshot(backups, '20260101T000000Z', {
|
||||
'SOUL.md': 'a',
|
||||
'agents/x.conf': 'b',
|
||||
'tools/_lib/credentials.json': 'c',
|
||||
});
|
||||
expect(planRestore(dir).sort()).toEqual(
|
||||
['SOUL.md', 'agents/x.conf', 'tools/_lib/credentials.json'].sort(),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('applyRestore', () => {
|
||||
it('restores byte-exact content, creates parent dirs, and sets 0600', () => {
|
||||
const dir = seedSnapshot(backups, '20260101T000000Z', {
|
||||
'SOUL.md': 'original-soul',
|
||||
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
|
||||
});
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
// A diverged operator file that restore must overwrite.
|
||||
writeFileSync(join(home, 'SOUL.md'), 'CORRUPTED');
|
||||
|
||||
const n = applyRestore(dir, home, planRestore(dir));
|
||||
expect(n).toBe(2);
|
||||
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('original-soul');
|
||||
expect(readFileSync(join(home, 'tools/_lib/credentials.json'), 'utf8')).toBe(
|
||||
`TOKEN=${SECRET}\n`,
|
||||
);
|
||||
expect(statSync(join(home, 'SOUL.md')).mode & 0o777).toBe(0o600);
|
||||
expect(statSync(join(home, 'tools/_lib/credentials.json')).mode & 0o777).toBe(0o600);
|
||||
});
|
||||
});
|
||||
|
||||
describe('runRestore', () => {
|
||||
it('--list prints timestamps and counts, mutating nothing', async () => {
|
||||
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'a', 'agents/x.conf': 'b' });
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
|
||||
const code = await runRestore({
|
||||
list: true,
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
});
|
||||
|
||||
expect(code).toBe(0);
|
||||
const out = log.mock.calls.flat().join('\n');
|
||||
expect(out).toContain('20260101T000000Z');
|
||||
expect(out).toMatch(/2\b/); // the file count is surfaced
|
||||
log.mockRestore();
|
||||
});
|
||||
|
||||
it('--from restores the snapshot over MOSAIC_HOME byte-exact (yes bypasses prompt)', async () => {
|
||||
seedSnapshot(backups, '20260101T000000Z', {
|
||||
'SOUL.md': 'restored-soul',
|
||||
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
|
||||
});
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
writeFileSync(join(home, 'SOUL.md'), 'STALE');
|
||||
|
||||
const code = await runRestore({
|
||||
from: '20260101T000000Z',
|
||||
yes: true,
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
});
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('restored-soul');
|
||||
expect(readFileSync(join(home, 'tools/_lib/credentials.json'), 'utf8')).toBe(
|
||||
`TOKEN=${SECRET}\n`,
|
||||
);
|
||||
});
|
||||
|
||||
it('--from with an unknown timestamp fails without mutating', async () => {
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
writeFileSync(join(home, 'SOUL.md'), 'KEEP');
|
||||
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
|
||||
|
||||
const code = await runRestore({
|
||||
from: '19990101T000000Z',
|
||||
yes: true,
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
});
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('KEEP');
|
||||
err.mockRestore();
|
||||
});
|
||||
|
||||
it('--dry-run with --from reports the plan but mutates nothing', async () => {
|
||||
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'snap' });
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
writeFileSync(join(home, 'SOUL.md'), 'UNCHANGED');
|
||||
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
|
||||
const code = await runRestore({
|
||||
from: '20260101T000000Z',
|
||||
dryRun: true,
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
});
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('UNCHANGED');
|
||||
log.mockRestore();
|
||||
});
|
||||
|
||||
it('--from prompts and applies the restore when the operator confirms', async () => {
|
||||
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'confirmed-soul' });
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
writeFileSync(join(home, 'SOUL.md'), 'STALE');
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
const confirm = vi.fn().mockResolvedValue(true);
|
||||
|
||||
const code = await runRestore({
|
||||
from: '20260101T000000Z',
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
confirm,
|
||||
});
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(confirm).toHaveBeenCalledOnce();
|
||||
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('confirmed-soul');
|
||||
});
|
||||
|
||||
it('--from aborts without mutating when the operator declines', async () => {
|
||||
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'snap' });
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
writeFileSync(join(home, 'SOUL.md'), 'KEEP');
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
const confirm = vi.fn().mockResolvedValue(false);
|
||||
|
||||
const code = await runRestore({
|
||||
from: '20260101T000000Z',
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
confirm,
|
||||
});
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(confirm).toHaveBeenCalledOnce();
|
||||
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('KEEP');
|
||||
});
|
||||
|
||||
it('MOSAIC_ASSUME_YES=1 bypasses the confirmation prompt', async () => {
|
||||
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'env-yes' });
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
writeFileSync(join(home, 'SOUL.md'), 'STALE');
|
||||
vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
const confirm = vi.fn().mockResolvedValue(false);
|
||||
|
||||
const code = await runRestore({
|
||||
from: '20260101T000000Z',
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state'), MOSAIC_ASSUME_YES: '1' },
|
||||
confirm,
|
||||
});
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(confirm).not.toHaveBeenCalled();
|
||||
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('env-yes');
|
||||
});
|
||||
|
||||
it('--list reports gracefully when no snapshots exist', async () => {
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
|
||||
const code = await runRestore({
|
||||
list: true,
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'empty-state') },
|
||||
});
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(log.mock.calls.flat().join('\n')).toMatch(/No pre-update snapshots/);
|
||||
});
|
||||
|
||||
it('never prints a secret value found inside a backed-up file', async () => {
|
||||
seedSnapshot(backups, '20260101T000000Z', {
|
||||
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
|
||||
});
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
|
||||
|
||||
await runRestore({
|
||||
list: true,
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
});
|
||||
await runRestore({
|
||||
from: '20260101T000000Z',
|
||||
yes: true,
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
});
|
||||
|
||||
const all = [...log.mock.calls, ...err.mock.calls].flat().join('\n');
|
||||
expect(all).not.toContain(SECRET);
|
||||
log.mockRestore();
|
||||
err.mockRestore();
|
||||
});
|
||||
});
|
||||
|
||||
// Regression coverage for the codex code+security review of PR2 (#791):
|
||||
// CWE-22 traversal via --from, and CWE-59 symlink write-through in applyRestore.
|
||||
describe('security hardening', () => {
|
||||
it.each([
|
||||
'../../etc',
|
||||
'pre-update-/../../tmp/poison',
|
||||
'pre-update-../evil',
|
||||
'20260101T000000Z/../../../tmp',
|
||||
'not-a-timestamp',
|
||||
'2026-01-01',
|
||||
])('resolveSnapshotDir rejects traversal / malformed selector %j', (bad) => {
|
||||
// Even if a matching directory exists on disk, a non-timestamp selector
|
||||
// must not resolve — the only accepted shape is <8>T<6>Z[-n].
|
||||
expect(resolveSnapshotDir(backups, bad)).toBeUndefined();
|
||||
});
|
||||
|
||||
it('runRestore --from a traversal selector fails closed without copying', async () => {
|
||||
// Plant a real dir one level ABOVE the backup root. The naive resolver
|
||||
// `join(root, from)` with `from='../poison'` would reach it (backups is
|
||||
// .../mosaic/backups, so `../poison` == .../mosaic/poison) and import it.
|
||||
const outside = join(tmp, 'state', 'mosaic', 'poison');
|
||||
mkdirSync(outside, { recursive: true });
|
||||
writeFileSync(join(outside, 'x'), 'attacker');
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
|
||||
|
||||
const code = await runRestore({
|
||||
from: '../poison',
|
||||
yes: true,
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
});
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(statSync(home).isDirectory()).toBe(true);
|
||||
// Nothing from `outside` was imported.
|
||||
expect(() => statSync(join(home, 'x'))).toThrow();
|
||||
err.mockRestore();
|
||||
});
|
||||
|
||||
it('applyRestore refuses to write a secret through a symlinked leaf (CWE-59)', () => {
|
||||
const dir = seedSnapshot(backups, '20260101T000000Z', {
|
||||
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
|
||||
});
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(join(home, 'tools', '_lib'), { recursive: true });
|
||||
// Attacker points the operator credentials file at a file they can read.
|
||||
const exfil = join(tmp, 'exfil-target');
|
||||
writeFileSync(exfil, 'original-attacker-content');
|
||||
symlinkSync(exfil, join(home, 'tools', '_lib', 'credentials.json'));
|
||||
|
||||
expect(() => applyRestore(dir, home, planRestore(dir))).toThrow();
|
||||
// The secret was NOT written through the link into the attacker's file.
|
||||
expect(readFileSync(exfil, 'utf8')).toBe('original-attacker-content');
|
||||
});
|
||||
|
||||
it('applyRestore refuses to write through a symlinked ancestor (CWE-59)', () => {
|
||||
const dir = seedSnapshot(backups, '20260101T000000Z', {
|
||||
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
|
||||
});
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(join(home, 'tools'), { recursive: true });
|
||||
// Attacker replaces the `tools/_lib` ancestor with a symlink out of the root.
|
||||
const exfilDir = join(tmp, 'exfil-dir');
|
||||
mkdirSync(exfilDir, { recursive: true });
|
||||
symlinkSync(exfilDir, join(home, 'tools', '_lib'));
|
||||
|
||||
expect(() => applyRestore(dir, home, planRestore(dir))).toThrow();
|
||||
// Nothing was written into the attacker-controlled directory.
|
||||
expect(() => statSync(join(exfilDir, 'credentials.json'))).toThrow();
|
||||
});
|
||||
|
||||
it('runRestore surfaces a symlink violation as exit 1 without leaking the secret', async () => {
|
||||
seedSnapshot(backups, '20260101T000000Z', {
|
||||
'tools/_lib/credentials.json': `TOKEN=${SECRET}\n`,
|
||||
});
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(join(home, 'tools', '_lib'), { recursive: true });
|
||||
const exfil = join(tmp, 'exfil-target');
|
||||
writeFileSync(exfil, 'attacker');
|
||||
symlinkSync(exfil, join(home, 'tools', '_lib', 'credentials.json'));
|
||||
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
const err = vi.spyOn(console, 'error').mockImplementation(() => {});
|
||||
|
||||
const code = await runRestore({
|
||||
from: '20260101T000000Z',
|
||||
yes: true,
|
||||
mosaicHome: home,
|
||||
env: { XDG_STATE_HOME: join(tmp, 'state') },
|
||||
});
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(readFileSync(exfil, 'utf8')).toBe('attacker');
|
||||
const all = [...log.mock.calls, ...err.mock.calls].flat().join('\n');
|
||||
expect(all).not.toContain(SECRET);
|
||||
log.mockRestore();
|
||||
err.mockRestore();
|
||||
});
|
||||
|
||||
it('applyRestore replaces a diverged regular file in place with 0600 (not a symlink)', () => {
|
||||
const dir = seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'restored' });
|
||||
const home = join(tmp, 'home');
|
||||
mkdirSync(home, { recursive: true });
|
||||
writeFileSync(join(home, 'SOUL.md'), 'stale');
|
||||
|
||||
const n = applyRestore(dir, home, planRestore(dir));
|
||||
expect(n).toBe(1);
|
||||
expect(lstatSync(join(home, 'SOUL.md')).isSymbolicLink()).toBe(false);
|
||||
expect(readFileSync(join(home, 'SOUL.md'), 'utf8')).toBe('restored');
|
||||
expect(statSync(join(home, 'SOUL.md')).mode & 0o777).toBe(0o600);
|
||||
});
|
||||
});
|
||||
|
||||
describe('registerRestoreCommand', () => {
|
||||
it('registers `restore` with the expected flags', () => {
|
||||
const program = new Command();
|
||||
program.exitOverride();
|
||||
registerRestoreCommand(program);
|
||||
const cmd = program.commands.find((c) => c.name() === 'restore');
|
||||
expect(cmd).toBeDefined();
|
||||
const longs = cmd!.options.map((o) => o.long);
|
||||
expect(longs).toEqual(
|
||||
expect.arrayContaining(['--list', '--from', '--dry-run', '--yes', '--mosaic-home']),
|
||||
);
|
||||
});
|
||||
|
||||
it('runs the list action end-to-end via the parsed command', async () => {
|
||||
seedSnapshot(backups, '20260101T000000Z', { 'SOUL.md': 'a' });
|
||||
const prevXdg = process.env['XDG_STATE_HOME'];
|
||||
process.env['XDG_STATE_HOME'] = join(tmp, 'state');
|
||||
const log = vi.spyOn(console, 'log').mockImplementation(() => {});
|
||||
try {
|
||||
const program = new Command();
|
||||
program.exitOverride();
|
||||
registerRestoreCommand(program);
|
||||
await program.parseAsync(['restore', '--list', '--mosaic-home', join(tmp, 'home')], {
|
||||
from: 'user',
|
||||
});
|
||||
expect(log.mock.calls.flat().join('\n')).toContain('20260101T000000Z');
|
||||
} finally {
|
||||
log.mockRestore();
|
||||
if (prevXdg === undefined) delete process.env['XDG_STATE_HOME'];
|
||||
else process.env['XDG_STATE_HOME'] = prevXdg;
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
313
packages/mosaic/src/commands/restore.ts
Normal file
313
packages/mosaic/src/commands/restore.ts
Normal file
@@ -0,0 +1,313 @@
|
||||
/**
|
||||
* restore.ts — top-level `mosaic restore` command (#791 PR2 Task 12)
|
||||
*
|
||||
* Recovery counterpart to the durable pre-update snapshot taken by install.sh
|
||||
* (make_durable_snapshot). Before a keep-mode upgrade mutates anything, the
|
||||
* installer copies the operator-owned surface to
|
||||
* $XDG_STATE_HOME/mosaic/backups/pre-update-<UTC-ts>/ (0700 dirs / 0600 files)
|
||||
* This command lets the operator inspect and roll back to those snapshots:
|
||||
*
|
||||
* mosaic restore # == --list: enumerate snapshots (dry-run)
|
||||
* mosaic restore --list
|
||||
* mosaic restore --from <ts> # restore that snapshot over MOSAIC_HOME
|
||||
* mosaic restore --from <ts> --dry-run
|
||||
*
|
||||
* SECREV INVARIANT: a snapshot may contain secrets (e.g. tools/_lib/credentials.json).
|
||||
* This command reports counts and RELATIVE PATHS only — it never reads a backed-up
|
||||
* file into any logged string. Restored files are written back 0600 (owner-only),
|
||||
* matching the snapshot's own private posture. The path convention here mirrors
|
||||
* install.sh `backup_root()`; keep the two in sync (no shared code across the boundary).
|
||||
*/
|
||||
|
||||
import {
|
||||
existsSync,
|
||||
readdirSync,
|
||||
statSync,
|
||||
lstatSync,
|
||||
readFileSync,
|
||||
openSync,
|
||||
writeSync,
|
||||
fchmodSync,
|
||||
closeSync,
|
||||
constants,
|
||||
} from 'node:fs';
|
||||
import { createInterface } from 'node:readline';
|
||||
import { homedir } from 'node:os';
|
||||
import { join, dirname, relative } from 'node:path';
|
||||
import type { Command } from 'commander';
|
||||
import { DEFAULT_MOSAIC_HOME } from '../constants.js';
|
||||
import { assertCanonicalContainment, ensureManagedDirectory } from '../fleet/secure-file.js';
|
||||
|
||||
// ─── types ───────────────────────────────────────────────────────────────────
|
||||
|
||||
export interface SnapshotInfo {
|
||||
/** The UTC stamp after the `pre-update-` prefix, e.g. "20260716T232225Z". */
|
||||
readonly timestamp: string;
|
||||
/** Absolute path to the snapshot directory. */
|
||||
readonly dir: string;
|
||||
/** Number of files captured in the snapshot. */
|
||||
readonly fileCount: number;
|
||||
}
|
||||
|
||||
export interface RestoreOptions {
|
||||
list?: boolean;
|
||||
from?: string;
|
||||
dryRun?: boolean;
|
||||
yes?: boolean;
|
||||
mosaicHome: string;
|
||||
/** Environment source (injectable for tests); defaults to process.env. */
|
||||
env?: NodeJS.ProcessEnv;
|
||||
/**
|
||||
* Confirmation gate (injectable for tests); defaults to an interactive
|
||||
* readline prompt. Returns true to proceed with the overwrite.
|
||||
*/
|
||||
confirm?: (question: string) => Promise<boolean>;
|
||||
}
|
||||
|
||||
const SNAPSHOT_PREFIX = 'pre-update-';
|
||||
|
||||
/**
|
||||
* The exact shape install.sh `make_durable_snapshot()` stamps: `<8>T<6>Z` UTC,
|
||||
* with an optional `-<n>` same-second collision suffix. `--from` is matched
|
||||
* against this — nothing containing a path separator or `..` can pass, so a
|
||||
* selector can never escape the backup root (CWE-22).
|
||||
*/
|
||||
const SNAPSHOT_TS_RE = /^\d{8}T\d{6}Z(?:-\d+)?$/;
|
||||
|
||||
// ─── pure helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
/** Resolve the durable-snapshot root, mirroring install.sh `backup_root()`. */
|
||||
export function resolveBackupRoot(env: NodeJS.ProcessEnv = process.env): string {
|
||||
const stateHome = env['XDG_STATE_HOME'] || join(homedir(), '.local', 'state');
|
||||
return join(stateHome, 'mosaic', 'backups');
|
||||
}
|
||||
|
||||
/** Recursively collect every file under `dir` as a path relative to `dir`. */
|
||||
export function planRestore(dir: string): string[] {
|
||||
const out: string[] = [];
|
||||
const walk = (cur: string): void => {
|
||||
for (const entry of readdirSync(cur, { withFileTypes: true })) {
|
||||
const abs = join(cur, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
walk(abs);
|
||||
} else if (entry.isFile()) {
|
||||
out.push(relative(dir, abs));
|
||||
}
|
||||
}
|
||||
};
|
||||
if (existsSync(dir)) walk(dir);
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Enumerate snapshots newest-first (the `pre-update-<ts>` names sort chronologically). */
|
||||
export function listSnapshots(root: string): SnapshotInfo[] {
|
||||
if (!existsSync(root)) return [];
|
||||
let entries: string[];
|
||||
try {
|
||||
entries = readdirSync(root);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
return entries
|
||||
.filter((name) => name.startsWith(SNAPSHOT_PREFIX))
|
||||
.map((name) => join(root, name))
|
||||
.filter((dir) => {
|
||||
try {
|
||||
return statSync(dir).isDirectory();
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
})
|
||||
.sort()
|
||||
.reverse()
|
||||
.map((dir) => ({
|
||||
timestamp: dir.split('/').at(-1)!.slice(SNAPSHOT_PREFIX.length),
|
||||
dir,
|
||||
fileCount: planRestore(dir).length,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve a snapshot dir from a `--from` selector. Accepts ONLY a strict
|
||||
* generated identifier — a bare `<ts>` or the full `pre-update-<ts>` name — and
|
||||
* builds exactly `join(root, 'pre-update-' + ts)`. A selector containing `/`,
|
||||
* `..`, or anything but the timestamp shape is rejected (returns undefined), so
|
||||
* `--from` can never traverse outside the backup root (CWE-22). The resolved dir
|
||||
* must be a real, non-symlink directory (lstat, not stat), so a symlinked
|
||||
* snapshot entry can't redirect the restore either.
|
||||
*/
|
||||
export function resolveSnapshotDir(root: string, from: string): string | undefined {
|
||||
const ts = from.startsWith(SNAPSHOT_PREFIX) ? from.slice(SNAPSHOT_PREFIX.length) : from;
|
||||
if (!SNAPSHOT_TS_RE.test(ts)) return undefined;
|
||||
const dir = join(root, `${SNAPSHOT_PREFIX}${ts}`);
|
||||
try {
|
||||
if (lstatSync(dir).isDirectory()) return dir;
|
||||
} catch {
|
||||
/* absent or inaccessible */
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Copy each `relPaths` entry from the snapshot back into `mosaicHome`, forcing
|
||||
* 0600 on the restored file (owner-only — the operator surface may hold secrets).
|
||||
* Returns the number of files restored. Never reads a file's content into a
|
||||
* logged string.
|
||||
*
|
||||
* SYMLINK-SAFE (CWE-59): a snapshot may hold secrets, so we must never let a
|
||||
* tampered destination redirect the write. Every destination path is contained
|
||||
* within `mosaicHome` (assertCanonicalContainment) and every ancestor is proven
|
||||
* to be a real, non-symlink directory (ensureManagedDirectory) before we write.
|
||||
* The leaf itself is opened O_NOFOLLOW, so if it was swapped for a symlink the
|
||||
* open fails closed (ELOOP) rather than writing the secret through the link.
|
||||
*/
|
||||
export function applyRestore(
|
||||
snapDir: string,
|
||||
mosaicHome: string,
|
||||
relPaths: readonly string[],
|
||||
): number {
|
||||
let restored = 0;
|
||||
for (const rel of relPaths) {
|
||||
const src = join(snapDir, rel);
|
||||
const dst = join(mosaicHome, rel);
|
||||
// Fail closed if the target path escapes the managed root or any ancestor is
|
||||
// a symlink; create missing ancestors as private (0700) real directories.
|
||||
assertCanonicalContainment(mosaicHome, dst);
|
||||
ensureManagedDirectory(mosaicHome, dirname(dst));
|
||||
// O_NOFOLLOW: refuse to follow a symlink at the leaf (secret exfil guard).
|
||||
// O_CREAT|O_TRUNC: create a fresh 0600 file, or overwrite a diverged real one.
|
||||
const fd = openSync(
|
||||
dst,
|
||||
constants.O_WRONLY | constants.O_CREAT | constants.O_TRUNC | constants.O_NOFOLLOW,
|
||||
0o600,
|
||||
);
|
||||
try {
|
||||
fchmodSync(fd, 0o600); // enforce 0600 even when the file pre-existed
|
||||
writeSync(fd, readFileSync(src));
|
||||
} finally {
|
||||
closeSync(fd);
|
||||
}
|
||||
restored += 1;
|
||||
}
|
||||
return restored;
|
||||
}
|
||||
|
||||
// ─── orchestration ────────────────────────────────────────────────────────────
|
||||
|
||||
async function promptConfirm(question: string): Promise<boolean> {
|
||||
const rl = createInterface({ input: process.stdin, output: process.stdout });
|
||||
try {
|
||||
return await new Promise<boolean>((resolve) => {
|
||||
rl.question(`${question} [y/N] `, (ans) => resolve(ans.trim().toLowerCase() === 'y'));
|
||||
});
|
||||
} finally {
|
||||
rl.close();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run `mosaic restore`. Returns a process exit code (0 ok, 1 error) rather than
|
||||
* calling process.exit, so it stays unit-testable.
|
||||
*/
|
||||
export async function runRestore(opts: RestoreOptions): Promise<number> {
|
||||
const env = opts.env ?? process.env;
|
||||
const root = resolveBackupRoot(env);
|
||||
|
||||
// Default action (and explicit --list): enumerate, never mutate.
|
||||
if (opts.list || !opts.from) {
|
||||
const snaps = listSnapshots(root);
|
||||
if (snaps.length === 0) {
|
||||
console.log(`No pre-update snapshots found under ${root}.`);
|
||||
return 0;
|
||||
}
|
||||
console.log(`Pre-update snapshots under ${root} (newest first):\n`);
|
||||
for (const s of snaps) {
|
||||
console.log(` ${s.timestamp} — ${s.fileCount} file(s)`);
|
||||
}
|
||||
console.log(`\nRestore one with: mosaic restore --from <timestamp>`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
// --from <ts>: restore over the operator surface.
|
||||
const snapDir = resolveSnapshotDir(root, opts.from);
|
||||
if (!snapDir) {
|
||||
console.error(`No snapshot matching '${opts.from}' under ${root}.`);
|
||||
console.error(`Run 'mosaic restore --list' to see available timestamps.`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
const relPaths = planRestore(snapDir);
|
||||
const ts = snapDir.split('/').at(-1)!.slice(SNAPSHOT_PREFIX.length);
|
||||
|
||||
if (opts.dryRun) {
|
||||
console.log(
|
||||
`[dry-run] Would restore ${relPaths.length} file(s) from snapshot ${ts} into ${opts.mosaicHome}:`,
|
||||
);
|
||||
for (const rel of relPaths) console.log(` ${rel}`);
|
||||
console.log('[dry-run] No changes made.');
|
||||
return 0;
|
||||
}
|
||||
|
||||
const assumeYes = opts.yes || env['MOSAIC_ASSUME_YES'] === '1';
|
||||
if (!assumeYes) {
|
||||
console.log(
|
||||
`About to restore ${relPaths.length} operator file(s) from snapshot ${ts} into ${opts.mosaicHome}.`,
|
||||
);
|
||||
console.log('This OVERWRITES those files with their pre-update contents.');
|
||||
const ok = await (opts.confirm ?? promptConfirm)('Proceed?');
|
||||
if (!ok) {
|
||||
console.log('Restore cancelled. No changes made.');
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
let n: number;
|
||||
try {
|
||||
n = applyRestore(snapDir, opts.mosaicHome, relPaths);
|
||||
} catch (err) {
|
||||
// A containment/symlink violation is a fail-closed security stop, not a
|
||||
// routine error — surface it without leaking file contents and abort.
|
||||
console.error(
|
||||
`Restore aborted: a destination path under ${opts.mosaicHome} is unsafe to write ` +
|
||||
`(symlink or escapes the managed root). No files were restored. (${(err as Error).message})`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
console.log(`Restored ${n} operator file(s) from snapshot ${ts} into ${opts.mosaicHome}.`);
|
||||
return 0;
|
||||
}
|
||||
|
||||
// ─── commander registration ───────────────────────────────────────────────────
|
||||
|
||||
export function registerRestoreCommand(program: Command): void {
|
||||
program
|
||||
.command('restore')
|
||||
.description('List or restore durable pre-update snapshots of your operator config (#791)')
|
||||
.option('--list', 'List available snapshots by timestamp (default action)')
|
||||
.option('--from <timestamp>', 'Restore the snapshot with this timestamp over MOSAIC_HOME')
|
||||
.option('--dry-run', 'With --from: show what would be restored without changing anything')
|
||||
.option('--yes, -y', 'Skip the confirmation prompt (also: MOSAIC_ASSUME_YES=1)')
|
||||
.option(
|
||||
'--mosaic-home <path>',
|
||||
'Override MOSAIC_HOME directory',
|
||||
process.env['MOSAIC_HOME'] ?? DEFAULT_MOSAIC_HOME,
|
||||
)
|
||||
.action(
|
||||
async (opts: {
|
||||
list?: boolean;
|
||||
from?: string;
|
||||
dryRun?: boolean;
|
||||
yes?: boolean;
|
||||
mosaicHome: string;
|
||||
}) => {
|
||||
const code = await runRestore({
|
||||
list: opts.list,
|
||||
from: opts.from,
|
||||
dryRun: opts.dryRun,
|
||||
yes: opts.yes,
|
||||
mosaicHome: opts.mosaicHome,
|
||||
});
|
||||
if (code !== 0) process.exit(code);
|
||||
},
|
||||
);
|
||||
}
|
||||
@@ -381,6 +381,22 @@ function isMissingFile(error: unknown): boolean {
|
||||
return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT';
|
||||
}
|
||||
|
||||
/**
|
||||
* Non-blocking acquire of the roster mutation lock: an exclusive `wx` create that
|
||||
* throws `concurrent-mutation` (never waits) if the lock is already held. The
|
||||
* on-disk format is an empty private file. CRUD releases in a plain `finally` with
|
||||
* no result-preservation logic, so a release fault here must not override the
|
||||
* mutation result — the release swallows unlink failures by design.
|
||||
*
|
||||
* The recovery-framed `fleet regen` contends on this exact same lock file, but it
|
||||
* does so through the hardened, ownership-proving acquirer in the reconciler
|
||||
* (`acquirePrivateRosterMutationLock`), not this one: whoever wins the `wx` create
|
||||
* owns the file (the loser always gets `concurrent-mutation`), so the reconciler's
|
||||
* ownership token is only ever written and read back by the same regen invocation,
|
||||
* never by this empty-file writer. The two acquirers stay mutually compatible at
|
||||
* the `wx`-contention layer while regen additionally proves ownership before it
|
||||
* unlinks — a guarantee CRUD does not need because it releases unconditionally.
|
||||
*/
|
||||
async function acquireMutationLock(lockPath: string): Promise<() => Promise<void>> {
|
||||
try {
|
||||
const handle = await open(lockPath, 'wx', 0o600);
|
||||
|
||||
@@ -584,6 +584,29 @@ function defaultValidateRoster(
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The single roster-v2 → generated-projection mapping. Both the reconciler's
|
||||
* apply path ({@link defaultPrepareProjections}) and the recovery-framed
|
||||
* `mosaic fleet regen` command derive their generated env from THIS one
|
||||
* function, so the two paths can never drift (the #791 single-SSOT invariant).
|
||||
* Pure: no IO, no clock — a deterministic function of the roster alone.
|
||||
*/
|
||||
export function projectRosterV2AgentGeneratedEnv(
|
||||
roster: FleetRosterV2,
|
||||
agent: FleetRosterV2Agent,
|
||||
): Readonly<Record<string, string>> {
|
||||
return {
|
||||
MOSAIC_AGENT_NAME: agent.name,
|
||||
MOSAIC_AGENT_CLASS: agent.className,
|
||||
MOSAIC_AGENT_RUNTIME: agent.runtime,
|
||||
MOSAIC_AGENT_MODEL: agent.model,
|
||||
MOSAIC_AGENT_REASONING: agent.reasoning,
|
||||
MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy,
|
||||
MOSAIC_AGENT_WORKDIR: agent.workingDirectory,
|
||||
MOSAIC_TMUX_SOCKET: roster.tmux.socketName,
|
||||
};
|
||||
}
|
||||
|
||||
function defaultPrepareProjections(
|
||||
request: FleetReconcileRequest,
|
||||
): (roster: FleetRosterV2) => Promise<readonly PreparedAgentEnvironmentProjection[]> {
|
||||
@@ -596,16 +619,7 @@ function defaultPrepareProjections(
|
||||
mosaicHome,
|
||||
agentEnvDir: join(mosaicHome, 'fleet', 'agents'),
|
||||
agentName: agent.name,
|
||||
generated: {
|
||||
MOSAIC_AGENT_NAME: agent.name,
|
||||
MOSAIC_AGENT_CLASS: agent.className,
|
||||
MOSAIC_AGENT_RUNTIME: agent.runtime,
|
||||
MOSAIC_AGENT_MODEL: agent.model,
|
||||
MOSAIC_AGENT_REASONING: agent.reasoning,
|
||||
MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy,
|
||||
MOSAIC_AGENT_WORKDIR: agent.workingDirectory,
|
||||
MOSAIC_TMUX_SOCKET: roster.tmux.socketName,
|
||||
},
|
||||
generated: projectRosterV2AgentGeneratedEnv(roster, agent),
|
||||
}),
|
||||
),
|
||||
);
|
||||
@@ -620,62 +634,193 @@ function mosaicHomeFor(deps: FleetReconcileDeps): string {
|
||||
return deps.mosaicHome ?? join(homedir(), '.config', 'mosaic');
|
||||
}
|
||||
|
||||
/** Acquires a private lock only after proving the canonical managed path. */
|
||||
/** Acquires the private reconcile lock only after proving the canonical managed path. */
|
||||
export function acquirePrivateReconcileLock(
|
||||
mosaicHome: string,
|
||||
openLock: typeof open = open,
|
||||
): () => Promise<() => Promise<void>> {
|
||||
return acquirePrivateManagedRosterLock(
|
||||
mosaicHome,
|
||||
'roster.yaml.reconcile.lock',
|
||||
'Another roster reconciliation is in progress.',
|
||||
openLock,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Acquires the private roster MUTATION lock (`roster.yaml.mutation.lock`) with the
|
||||
* exact same ownership-proof discipline as the reconcile lock. Exposed so the
|
||||
* recovery-framed `mosaic fleet regen` can serialize its projection rewrites
|
||||
* against agent CRUD — which holds this very lock while it rewrites the roster and
|
||||
* the same derived projections — reusing this hardened acquirer rather than
|
||||
* reimplementing it.
|
||||
*
|
||||
* The release PROVES ownership (device/inode + token) before unlinking, so a lock
|
||||
* that was already cleared and replaced by the time release runs is never deleted
|
||||
* out from under the new owner (it fails closed as `lock-cleanup-failed` instead),
|
||||
* and that cleanup fault is propagated to the caller rather than silently dropped.
|
||||
* CRUD contends on the identical path with a plain empty-file `wx` create; the two
|
||||
* never co-own it (whoever wins `wx` owns it; the loser gets `concurrent-mutation`),
|
||||
* so the token this writes is only ever read back by the same regen invocation that
|
||||
* wrote it.
|
||||
*
|
||||
* SCOPE of the guarantee (matches the merged reconcile lock exactly — see
|
||||
* {@link acquirePrivateManagedRosterLock}): the ownership proof closes the
|
||||
* *reachable* window — a stale-lock reaper or operator cleared our lock and another
|
||||
* writer took it BEFORE our release began — not the residual sub-instruction window
|
||||
* between the final check and the path-based `unlink`. Closing that residual fully
|
||||
* requires an fd-held advisory lock adopted by every fleet writer (CRUD, reconcile,
|
||||
* regen), which is a cross-cutting mechanism change out of scope for this
|
||||
* projection-only recovery command; it is unreachable within the `wx` writer
|
||||
* protocol regardless (no Mosaic writer removes a lock it does not own, so only
|
||||
* external interference can vacate our inode mid-release).
|
||||
*/
|
||||
export function acquirePrivateRosterMutationLock(
|
||||
mosaicHome: string,
|
||||
openLock: typeof open = open,
|
||||
): () => Promise<() => Promise<void>> {
|
||||
return acquirePrivateManagedRosterLock(
|
||||
mosaicHome,
|
||||
'roster.yaml.mutation.lock',
|
||||
'Another roster mutation is in progress.',
|
||||
openLock,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Shared body for the ownership-proving managed roster locks. Acquires `lockLeaf`
|
||||
* under `<mosaicHome>/fleet` with a private `wx` create, writes an ownership token,
|
||||
* and returns a release that re-proves device/inode + token before unlinking so it
|
||||
* can never remove a lock it no longer owns. Non-blocking: throws `concurrent-mutation`
|
||||
* (with `busyMessage`) on contention rather than waiting.
|
||||
*/
|
||||
function acquirePrivateManagedRosterLock(
|
||||
mosaicHome: string,
|
||||
lockLeaf: string,
|
||||
busyMessage: string,
|
||||
openLock: typeof open,
|
||||
): () => Promise<() => Promise<void>> {
|
||||
const fleetDir = join(mosaicHome, 'fleet');
|
||||
const lockPath = join(fleetDir, 'roster.yaml.reconcile.lock');
|
||||
const lockPath = join(fleetDir, lockLeaf);
|
||||
// The message-producing helpers below serve BOTH managed locks, so every fault
|
||||
// names the actual lock file (`fleet/<leaf>`) — an operator needs to know WHICH
|
||||
// lock is stale, not a hardcoded "reconciliation lock".
|
||||
const lockLabel = `fleet/${lockLeaf}`;
|
||||
return async (): Promise<() => Promise<void>> => {
|
||||
await assertPrivateManagedDirectory(mosaicHome);
|
||||
await assertPrivateManagedDirectory(fleetDir);
|
||||
await assertSafeLockLeafIfPresent(lockPath);
|
||||
await assertSafeLockLeafIfPresent(lockPath, lockLabel);
|
||||
|
||||
let handle: FileHandle;
|
||||
try {
|
||||
handle = await openLock(lockPath, 'wx', 0o600);
|
||||
} catch (error: unknown) {
|
||||
if (isCode(error, 'EEXIST')) {
|
||||
await assertSafeLockLeafIfPresent(lockPath);
|
||||
throw new FleetReconcileError(
|
||||
'concurrent-mutation',
|
||||
'Another roster reconciliation is in progress.',
|
||||
);
|
||||
await assertSafeLockLeafIfPresent(lockPath, lockLabel);
|
||||
throw new FleetReconcileError('concurrent-mutation', busyMessage);
|
||||
}
|
||||
throw new FleetReconcileError('lock-io-failed', 'The reconciliation lock cannot be created.');
|
||||
throw new FleetReconcileError('lock-io-failed', `The ${lockLabel} lock cannot be created.`);
|
||||
}
|
||||
|
||||
// Identity of the lock WE exclusively created (the `wx` create guaranteed it is
|
||||
// ours). Captured up front so both the release closure and the init-failure
|
||||
// cleanup below can prove ownership by device/inode before touching the file.
|
||||
const created = await handle.stat().catch((): undefined => undefined);
|
||||
|
||||
const token = randomUUID();
|
||||
let tokenPersisted = false;
|
||||
try {
|
||||
await handle.writeFile(`${token}\n`, 'utf8');
|
||||
const opened = await handle.stat();
|
||||
tokenPersisted = true;
|
||||
await handle.close();
|
||||
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'unsafe-lock');
|
||||
if (!created)
|
||||
throw new FleetReconcileError(
|
||||
'lock-io-failed',
|
||||
`The ${lockLabel} lock cannot be initialized.`,
|
||||
);
|
||||
await assertLockOwnership(
|
||||
lockPath,
|
||||
created.dev,
|
||||
created.ino,
|
||||
token,
|
||||
'unsafe-lock',
|
||||
lockLabel,
|
||||
);
|
||||
return async (): Promise<void> => {
|
||||
try {
|
||||
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed');
|
||||
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed');
|
||||
await assertLockOwnership(
|
||||
lockPath,
|
||||
created.dev,
|
||||
created.ino,
|
||||
token,
|
||||
'lock-cleanup-failed',
|
||||
lockLabel,
|
||||
);
|
||||
await unlink(lockPath);
|
||||
} catch (error: unknown) {
|
||||
if (error instanceof FleetReconcileError) throw error;
|
||||
throw new FleetReconcileError(
|
||||
'lock-cleanup-failed',
|
||||
'The reconciliation lock cleanup failed.',
|
||||
);
|
||||
throw new FleetReconcileError('lock-cleanup-failed', `The ${lockLabel} cleanup failed.`);
|
||||
}
|
||||
};
|
||||
} catch (error: unknown) {
|
||||
await handle.close().catch((): void => {});
|
||||
// Best-effort: remove the lock THIS invocation created so a transient init
|
||||
// fault does not strand a lock that would block future regen and agent CRUD.
|
||||
// dev/ino-guarded so it can never delete a replacement; swallowed so cleanup
|
||||
// failure never masks the initialization error being surfaced. The token is
|
||||
// passed only once persisted, so the fallback path (when the post-create stat
|
||||
// failed and dev/ino is unavailable) can still prove ownership by content.
|
||||
await removeOwnedLockLeafBestEffort(lockPath, created, tokenPersisted ? token : undefined);
|
||||
if (error instanceof FleetReconcileError) throw error;
|
||||
throw new FleetReconcileError(
|
||||
'lock-io-failed',
|
||||
'The reconciliation lock cannot be initialized.',
|
||||
`The ${lockLabel} lock cannot be initialized.`,
|
||||
);
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Best-effort removal of a managed lock leaf THIS process created, used only on the
|
||||
* initialization-failure path. Two independent ownership proofs, so a lock that was
|
||||
* already replaced is never deleted:
|
||||
*
|
||||
* - primary: the captured device/inode of the file we exclusively `wx`-created; or
|
||||
* - fallback: when the post-create stat itself failed (dev/ino unavailable), the
|
||||
* random `ownershipToken` we persisted — only OUR lock carries it, so a CRUD
|
||||
* (empty) or a differently-tokened replacement is never removed.
|
||||
*
|
||||
* Every fault is swallowed because the caller is already surfacing the initialization
|
||||
* error and a leftover lock is recoverable via the documented runbook. If BOTH proofs
|
||||
* are unavailable (stat failed AND the token write never landed) the lock is left in
|
||||
* place rather than risk deleting another writer's file — a doubly-degenerate case
|
||||
* requiring two independent fs faults on a just-created fd.
|
||||
*/
|
||||
async function removeOwnedLockLeafBestEffort(
|
||||
lockPath: string,
|
||||
created: { dev: number; ino: number } | undefined,
|
||||
ownershipToken: string | undefined,
|
||||
): Promise<void> {
|
||||
try {
|
||||
const current = await lstat(lockPath);
|
||||
if (!current.isFile() || current.isSymbolicLink()) return;
|
||||
if (created) {
|
||||
if (current.dev === created.dev && current.ino === created.ino) {
|
||||
await unlink(lockPath);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (ownershipToken !== undefined) {
|
||||
const contents = await readFile(lockPath, 'utf8');
|
||||
if (contents.trim() === ownershipToken) {
|
||||
await unlink(lockPath);
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Swallowed: leftover lock is recoverable; do not mask the init error.
|
||||
}
|
||||
}
|
||||
|
||||
async function assertPrivateManagedDirectory(path: string): Promise<void> {
|
||||
try {
|
||||
const metadata = await lstat(path);
|
||||
@@ -691,16 +836,16 @@ async function assertPrivateManagedDirectory(path: string): Promise<void> {
|
||||
}
|
||||
}
|
||||
|
||||
async function assertSafeLockLeafIfPresent(lockPath: string): Promise<void> {
|
||||
async function assertSafeLockLeafIfPresent(lockPath: string, lockLabel: string): Promise<void> {
|
||||
try {
|
||||
const metadata = await lstat(lockPath);
|
||||
if (!metadata.isFile() || metadata.isSymbolicLink() || (metadata.mode & 0o077) !== 0) {
|
||||
throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unsafe.');
|
||||
throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unsafe.`);
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
if (isCode(error, 'ENOENT')) return;
|
||||
if (error instanceof FleetReconcileError) throw error;
|
||||
throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unavailable.');
|
||||
throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unavailable.`);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -710,6 +855,7 @@ async function assertLockOwnership(
|
||||
inode: number,
|
||||
token: string,
|
||||
failureCode: 'unsafe-lock' | 'lock-cleanup-failed',
|
||||
lockLabel: string,
|
||||
): Promise<void> {
|
||||
try {
|
||||
const metadata = await lstat(lockPath);
|
||||
@@ -720,24 +866,21 @@ async function assertLockOwnership(
|
||||
metadata.dev !== device ||
|
||||
metadata.ino !== inode
|
||||
) {
|
||||
throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.');
|
||||
throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`);
|
||||
}
|
||||
const handle = await open(lockPath, constants.O_RDONLY | constants.O_NOFOLLOW);
|
||||
try {
|
||||
const opened = await handle.stat();
|
||||
const contents = await handle.readFile({ encoding: 'utf8' });
|
||||
if (opened.dev !== device || opened.ino !== inode || contents !== `${token}\n`) {
|
||||
throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.');
|
||||
throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`);
|
||||
}
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
if (error instanceof FleetReconcileError) throw error;
|
||||
throw new FleetReconcileError(
|
||||
failureCode,
|
||||
'The reconciliation lock ownership cannot be proven.',
|
||||
);
|
||||
throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership cannot be proven.`);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -204,6 +204,53 @@ export async function prepareAgentEnvironmentProjection(
|
||||
};
|
||||
}
|
||||
|
||||
/** A generated-only projection validated without touching any legacy/local/quarantine file. */
|
||||
export interface PreparedGeneratedAgentEnvironmentProjection {
|
||||
readonly mosaicHome: string;
|
||||
readonly agentEnvDir: string;
|
||||
readonly generatedPath: string;
|
||||
readonly generated: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates ONLY the roster-derived generated projection for a recovery rebuild.
|
||||
* Unlike {@link prepareAgentEnvironmentProjection}, this never reads, classifies,
|
||||
* relocates, or quarantines the legacy `.env` / `.env.local` operator surface — it
|
||||
* exists so `mosaic fleet regen` has no code path that can mutate anything except
|
||||
* `<name>.env.generated`. The existing generated file, if present, must already be
|
||||
* a private regular file.
|
||||
*/
|
||||
export async function prepareGeneratedAgentEnvironmentProjection(
|
||||
options: AgentEnvironmentProjectionOptions,
|
||||
): Promise<PreparedGeneratedAgentEnvironmentProjection> {
|
||||
if (!AGENT_NAME.test(options.agentName)) {
|
||||
throw new AgentEnvBoundaryError('unsafe-agent-name', 'MOSAIC_AGENT_NAME', options.agentName);
|
||||
}
|
||||
await validatePrivateProjectionDirectory(options.mosaicHome, options.agentEnvDir);
|
||||
const generatedPath = join(options.agentEnvDir, `${options.agentName}.env.generated`);
|
||||
const generated = renderGeneratedAgentEnvironment(options.generated);
|
||||
await assertPrivateRegularFileIfPresent(generatedPath);
|
||||
return {
|
||||
mosaicHome: options.mosaicHome,
|
||||
agentEnvDir: options.agentEnvDir,
|
||||
generatedPath,
|
||||
generated,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Applies a generated-only projection: writes ONLY `<name>.env.generated` atomically
|
||||
* and touches nothing else. It never writes `.env.local`/`.env.quarantine` and never
|
||||
* unlinks the legacy `.env` — the projection-only recovery guarantee is structural.
|
||||
*/
|
||||
export async function applyPreparedGeneratedAgentEnvironmentProjection(
|
||||
prepared: PreparedGeneratedAgentEnvironmentProjection,
|
||||
): Promise<string> {
|
||||
await ensurePrivateProjectionDirectory(prepared.mosaicHome, prepared.agentEnvDir);
|
||||
await writePrivateAtomically(prepared.generatedPath, prepared.generated);
|
||||
return prepared.generatedPath;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validates only the exact generated projection eligible for a roster delete.
|
||||
* Local overrides, legacy input, and quarantine records are operator-retained and
|
||||
|
||||
Reference in New Issue
Block a user