Compare commits
8 Commits
feat/791-p
...
fix/795-co
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
027a99b558 | ||
|
|
0e50c694cb | ||
|
|
080a6b4663 | ||
|
|
13c50c5fa9 | ||
| fe7a468c9d | |||
| cabf02e7b9 | |||
| 9ddc6fbda8 | |||
| 31607a4af6 |
@@ -22,10 +22,10 @@
|
|||||||
FROM node:24-alpine
|
FROM node:24-alpine
|
||||||
|
|
||||||
# Native toolchain required to compile node-gyp deps on musl, plus the
|
# Native toolchain required to compile node-gyp deps on musl, plus the
|
||||||
# postgresql-client used by the test step's pg_isready readiness probe. `bash`
|
# postgresql-client used by the test step's pg_isready readiness probe. `bash`,
|
||||||
# is baked here too — the sanitization step in ci.yml otherwise does a per-run
|
# `git`, and `jq` are baked here too — framework shell tests and the shipped
|
||||||
# `apk add bash`.
|
# Codex review wrappers require them without per-run installation in ci.yml.
|
||||||
RUN apk add --no-cache python3 make g++ postgresql-client bash
|
RUN apk add --no-cache python3 make g++ postgresql-client bash git jq
|
||||||
|
|
||||||
# Pin pnpm to the repo's packageManager version via corepack.
|
# Pin pnpm to the repo's packageManager version via corepack.
|
||||||
RUN corepack enable && corepack prepare pnpm@10.6.2 --activate
|
RUN corepack enable && corepack prepare pnpm@10.6.2 --activate
|
||||||
|
|||||||
@@ -98,6 +98,39 @@ Expected results:
|
|||||||
that means the unit ran, not that an agent pane is live. Treat tmux
|
that means the unit ran, not that an agent pane is live. Treat tmux
|
||||||
`has-session`, `list-panes`, process tree, and logs as the liveness evidence.
|
`has-session`, `list-panes`, process tree, and logs as the liveness evidence.
|
||||||
|
|
||||||
|
## Recovery — rebuild generated env projections
|
||||||
|
|
||||||
|
Each agent's `~/.config/mosaic/fleet/agents/<name>.env.generated` is a
|
||||||
|
deterministic projection of `roster.yaml` (the SSOT) that the launcher
|
||||||
|
(`start-agent-session.sh`) sources at start. If an upgrade or a manual mistake
|
||||||
|
wipes or diverges those projections, rebuild them from the roster with
|
||||||
|
`mosaic fleet regen` — do NOT restart the affected unit first.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mosaic fleet regen # dry-run (default): show create/rebuild plan per agent
|
||||||
|
mosaic fleet regen --json # same plan, machine-readable
|
||||||
|
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated on disk
|
||||||
|
```
|
||||||
|
|
||||||
|
`regen` is projection-only and **never restarts an agent** — it has no path to
|
||||||
|
systemd lifecycle. It is dry-run by default, deterministic/idempotent, uses the
|
||||||
|
same roster→env mapping as `mosaic fleet reconcile`, and emits paths and counts
|
||||||
|
only (never the projected `KEY=value` body). After `--write`, verify each unit
|
||||||
|
resolves the intended values before restarting one unit at a time. The unit sets
|
||||||
|
no `EnvironmentFile=` — `start-agent-session.sh` sources `.env.generated` itself —
|
||||||
|
so verify the generated file directly and the launcher path, not a nonexistent
|
||||||
|
`EnvironmentFile` property:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
|
||||||
|
systemctl --user cat mosaic-agent@<name> | grep ExecStart
|
||||||
|
systemctl --user restart mosaic-agent@<name>
|
||||||
|
```
|
||||||
|
|
||||||
|
Full recovery runbook and the three-layer #791 protection model (manifest
|
||||||
|
ownership → pre-update snapshot/restore → regen): see
|
||||||
|
[Upgrade Safety & Recovery](./upgrade-safety-and-recovery.md).
|
||||||
|
|
||||||
## Release Preflight
|
## Release Preflight
|
||||||
|
|
||||||
Run this checklist before cutting or dogfooding a fleet release:
|
Run this checklist before cutting or dogfooding a fleet release:
|
||||||
|
|||||||
147
docs/guides/upgrade-safety-and-recovery.md
Normal file
147
docs/guides/upgrade-safety-and-recovery.md
Normal file
@@ -0,0 +1,147 @@
|
|||||||
|
# Upgrade Safety & Recovery
|
||||||
|
|
||||||
|
How Mosaic protects operator-owned configuration under `~/.config/mosaic` across
|
||||||
|
framework upgrades, and how to recover if a projection is ever lost.
|
||||||
|
|
||||||
|
A framework upgrade runs `install.sh` in keep-mode (`MOSAIC_INSTALL_MODE=keep`,
|
||||||
|
`MOSAIC_SYNC_ONLY=1`) to refresh framework-owned files in place. The incident
|
||||||
|
this hardening addresses: an upgrade that silently overwrites or deletes a file
|
||||||
|
the operator owns — credentials, personas, a roster, or a generated agent env —
|
||||||
|
with no snapshot to fall back to.
|
||||||
|
|
||||||
|
Protection is layered. Each layer is independent; a later layer catches what an
|
||||||
|
earlier one misses.
|
||||||
|
|
||||||
|
## Layer 1 — Manifest-owned sync (prevention)
|
||||||
|
|
||||||
|
The single source of truth for ownership is
|
||||||
|
[`framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt).
|
||||||
|
Both the bash installer and the TypeScript sync path resolve every path against
|
||||||
|
this one file (parity is enforced by test), so they can never drift.
|
||||||
|
|
||||||
|
- Ownership is **allow-list, deny-wins**: a path is framework-owned only if a
|
||||||
|
`[framework]` glob matches and no `[operator]` carve-out overrides it.
|
||||||
|
- **Unknown paths default to operator** (fail-safe): a file the manifest never
|
||||||
|
anticipated is treated as operator-owned and is never pruned.
|
||||||
|
- Keep-mode does a non-deleting copy plus an explicit, manifest-scoped prune that
|
||||||
|
only ever iterates framework globs — operator and unknown paths are
|
||||||
|
structurally unreachable by the prune.
|
||||||
|
|
||||||
|
Result: a correct upgrade cannot touch operator config at all.
|
||||||
|
|
||||||
|
## Layer 2 — Durable pre-update snapshot + verify net (safety + rollback)
|
||||||
|
|
||||||
|
Before **any** mutation, the installer snapshots the operator-owned surface that
|
||||||
|
exists into:
|
||||||
|
|
||||||
|
```
|
||||||
|
${XDG_STATE_HOME:-~/.local/state}/mosaic/backups/pre-update-<UTC-timestamp>/
|
||||||
|
```
|
||||||
|
|
||||||
|
- `0700` directories / `0600` files (`umask 077`, scoped and restored),
|
||||||
|
outside `~/.config/mosaic` and outside any repo.
|
||||||
|
- **Fail-open**: a snapshot failure warns but never aborts the upgrade it
|
||||||
|
protects.
|
||||||
|
- Retention is `MOSAIC_BACKUP_RETENTION` snapshots (default 5).
|
||||||
|
|
||||||
|
After the sync, a **verify net** compares each snapshot file against its target
|
||||||
|
and restores (with a loud warning) any operator file the upgrade diverged or
|
||||||
|
removed — a divergence means a manifest bug slipped through Layer 1.
|
||||||
|
|
||||||
|
Inspect and restore snapshots with the CLI:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mosaic restore --list # dry-run: enumerate snapshots by timestamp
|
||||||
|
mosaic restore --from <UTC-timestamp> # restore the operator surface from one snapshot
|
||||||
|
mosaic restore --from <ts> --dry-run # preview a specific restore without writing
|
||||||
|
```
|
||||||
|
|
||||||
|
`mosaic restore` reports **counts and relative paths only** — it never emits file
|
||||||
|
contents, so a secret in `tools/_lib/credentials.json` is never echoed. Restores
|
||||||
|
are confirmation-gated (`--yes` or `MOSAIC_ASSUME_YES`) and write each leaf
|
||||||
|
atomically with `O_NOFOLLOW` (a symlink swapped in after the snapshot fails
|
||||||
|
closed rather than following out of the managed tree).
|
||||||
|
|
||||||
|
## Layer 3 — Regeneration from roster SSOT (recovery)
|
||||||
|
|
||||||
|
Some operator files are **derived** and do not need a byte-for-byte snapshot to
|
||||||
|
recover — they can be rebuilt from their source of truth. The fleet's per-agent
|
||||||
|
generated env projections are the prime case:
|
||||||
|
|
||||||
|
- `~/.config/mosaic/fleet/agents/<name>.env.generated` is a deterministic
|
||||||
|
projection of `~/.config/mosaic/fleet/roster.yaml`.
|
||||||
|
- The launcher (`start-agent-session.sh`, invoked by
|
||||||
|
`mosaic-agent@<name>.service`) sources that generated projection to establish
|
||||||
|
each agent's identity, runtime, model, and working directory. If it is missing
|
||||||
|
or wrong, the agent cannot launch with its intended identity.
|
||||||
|
|
||||||
|
`mosaic fleet regen` rebuilds those projections from the roster SSOT:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mosaic fleet regen # dry-run (default): show what would be rebuilt
|
||||||
|
mosaic fleet regen --json # same, machine-readable
|
||||||
|
mosaic fleet regen --write # rebuild the projections on disk
|
||||||
|
```
|
||||||
|
|
||||||
|
- **Dry-run by default.** Nothing is written until you pass `--write`.
|
||||||
|
- **Deterministic and idempotent** — the projection is a pure function of the
|
||||||
|
roster, so repeated `--write` runs produce byte-identical files.
|
||||||
|
- **Projection-only. It never restarts an agent.** Recovery order forbids
|
||||||
|
restart-before-verify; `regen` has no path to systemd lifecycle at all.
|
||||||
|
- **It rebuilds only `<name>.env.generated`** — it never writes, relocates, or
|
||||||
|
deletes the operator-owned `.env` / `.env.local` surface.
|
||||||
|
- It **validates the roster the same way `reconcile` does** (persona resolution
|
||||||
|
and protected-class tool-policy match), so a hand-edited or corrupt roster is
|
||||||
|
rejected rather than projected, and a `--write` takes the shared reconcile
|
||||||
|
lock so it cannot race a concurrent reconcile.
|
||||||
|
- Output is **paths and counts only** — the rendered `KEY=value` body is never
|
||||||
|
echoed.
|
||||||
|
|
||||||
|
`regen` uses the exact same roster→env mapping as `mosaic fleet reconcile`, so a
|
||||||
|
recovered projection matches what a normal reconcile would have written.
|
||||||
|
|
||||||
|
## Recovery runbook — wiped `fleet/agents/*.env.generated`
|
||||||
|
|
||||||
|
If an upgrade (or a manual mistake) has left an agent without its generated
|
||||||
|
projection, **do not restart the unit first** — a launch against a missing
|
||||||
|
projection fails closed, and any stale state must be corrected before restart,
|
||||||
|
not after.
|
||||||
|
|
||||||
|
1. **Prefer a snapshot restore if one exists** (byte-exact operator state):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mosaic restore --list
|
||||||
|
mosaic restore --from <UTC-timestamp>
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Otherwise regenerate the derived projections from the roster SSOT:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mosaic fleet regen # confirm the plan (create vs rebuild per agent)
|
||||||
|
mosaic fleet regen --write # rebuild fleet/agents/<name>.env.generated
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Verify each unit will resolve the intended runtime/workdir _before_ any
|
||||||
|
restart.** The unit sets **no** `EnvironmentFile=` — it launches from a minimal
|
||||||
|
environment and `start-agent-session.sh` sources `.env.generated` itself, so
|
||||||
|
verify the generated file directly and confirm the launcher path:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Confirm fleet/agents/<name>.env.generated exists and carries the intended
|
||||||
|
# MOSAIC_AGENT_* values (name, runtime, model, workdir, socket).
|
||||||
|
test -f ~/.config/mosaic/fleet/agents/<name>.env.generated
|
||||||
|
# Confirm the unit launches the session script that reads it.
|
||||||
|
systemctl --user cat mosaic-agent@<name> | grep ExecStart
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Only then restart, one unit at a time:**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl --user restart mosaic-agent@<name>
|
||||||
|
```
|
||||||
|
|
||||||
|
## See also
|
||||||
|
|
||||||
|
- Design: [`docs/design/791-upgrade-config-protection.md`](../design/791-upgrade-config-protection.md)
|
||||||
|
- Fleet operations: [`docs/guides/fleet-local-canary.md`](./fleet-local-canary.md)
|
||||||
|
- Ownership SSOT: [`packages/mosaic/framework/framework-manifest.txt`](../../packages/mosaic/framework/framework-manifest.txt)
|
||||||
@@ -296,3 +296,234 @@ vitest **1252** · restore.spec **30** · durable-snapshot **41** · manifest-gu
|
|||||||
migration 21. shellcheck clean on all new lines; new test markers mirror the existing `# VERIFY-NET`
|
migration 21. shellcheck clean on all new lines; new test markers mirror the existing `# VERIFY-NET`
|
||||||
anchor convention. NOTE: codex self-review does NOT satisfy the independent-review gate — an independent
|
anchor convention. NOTE: codex self-review does NOT satisfy the independent-review gate — an independent
|
||||||
(author≠reviewer) review + durable Gitea Reviewer-of-Record comment is still required before MS-LEAD merges.
|
(author≠reviewer) review + durable Gitea Reviewer-of-Record comment is still required before MS-LEAD merges.
|
||||||
|
|
||||||
|
## Session 4 (2026-07-16) — PR2 MERGED, PR3 built (fleet regen — recovery layer)
|
||||||
|
PR2 (#811) squash-merged → main `31607a4a`; issue #791 stays open (final PR of the 3-PR DAG). Independent
|
||||||
|
exact-head RoR at `d12c5f78` APPROVE (Gitea cmt 17904); #1882 green; busybox-portable Part 7 control fix
|
||||||
|
verified in-Alpine. PR3 UNBLOCKED.
|
||||||
|
|
||||||
|
PR3 branch: `feat/791-pr3-fleet-regen` off `origin/main` 31607a4. Same discipline: tests-first red-first,
|
||||||
|
independent review + durable Gitea RoR BEFORE MS-LEAD runs the queue guard/merge. PR body `Part of #791`.
|
||||||
|
|
||||||
|
### PR3 scope (ratified §4/§7 of design doc) — `mosaic fleet regen`
|
||||||
|
Projection-only recovery command: rebuilds each `fleet/agents/<name>.env.generated` from `roster.yaml`
|
||||||
|
(SSOT). Dry-run default; `--write` applies; `--json` machine output. Structural guarantee: NO code path to
|
||||||
|
systemd lifecycle — **never restarts an agent**. Single-SSOT: reuses `projectRosterV2AgentGeneratedEnv`
|
||||||
|
(extracted, shared with the reconciler apply path) so regen and reconcile cannot drift. Secrev: paths +
|
||||||
|
counts only, never the rendered KEY=value body.
|
||||||
|
|
||||||
|
New files: `commands/fleet-regen-command.ts` (+ `.spec.ts`), guide `docs/guides/upgrade-safety-and-recovery.md`
|
||||||
|
(three-layer model: PR1 manifest ownership → PR2 snapshot/restore → PR3 regen; do-NOT-restart-before-verify
|
||||||
|
runbook), regen reference added to `docs/guides/fleet-local-canary.md`. Wired in `commands/fleet.ts`.
|
||||||
|
|
||||||
|
### Independent review (3 reviewers: subagent code-reviewer + codex code-review + codex security) → 4 fixes, red-first
|
||||||
|
- **A · BLOCKER (codex) — regen mutated/deleted legacy operator env.** `applyPreparedAgentEnvironmentProjection`
|
||||||
|
also writes `.env.local`/`.env.quarantine` and unlinks legacy `.env`. Violated projection-only contract.
|
||||||
|
**Fix:** NEW generated-only boundary primitives `prepareGeneratedAgentEnvironmentProjection` +
|
||||||
|
`applyPreparedGeneratedAgentEnvironmentProjection` (write ONLY `<name>.env.generated`). regen now has no
|
||||||
|
code path that touches `.env`/`.env.local`/`.env.quarantine`. **Test:** projection-only leaves legacy `.env`
|
||||||
|
verbatim, no local/quarantine fabricated.
|
||||||
|
- **B · should-fix (codex + subagent + security) — partial write on mid-loop failure.** Interleaved
|
||||||
|
prepare/apply left earlier agents written when a later agent failed prepare. **Fix:** PREPARE ALL agents
|
||||||
|
before writing ANY (mirrors reconciler `defaultPrepareProjections`). **Test:** 2nd agent's projection
|
||||||
|
pre-seeded 0644 → prepare rejects → coder0 NOT written, exit 1.
|
||||||
|
- **C · subagent — semantic-validation bypass.** Default readRoster skipped `validateRosterV2Semantics`, so
|
||||||
|
a tampered protected-class `tool_policy` would be silently projected. **Fix:** default readRoster now runs
|
||||||
|
`validateRosterV2Semantics` (persona resolution + protected-class match), rolesDir/overrideDir defaults
|
||||||
|
mirroring the reconciler. **Test:** merge-gate agent w/ tool_policy=code → fails closed, no write.
|
||||||
|
- **D · MEDIUM (codex security, CWE-362) — concurrent-reconcile race.** regen `--write` wrote without the
|
||||||
|
reconcile lock. **Fix:** `--write` acquires `acquirePrivateReconcileLock(mosaicHome)` for the whole
|
||||||
|
read-prepare-apply sequence, released in `finally`; dry-run stays lock-free. **Test:** pre-held lock →
|
||||||
|
regen fails closed, no write.
|
||||||
|
|
||||||
|
**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1265**
|
||||||
|
(regen spec 13, incl. 4 new red-first regressions). NOTE: codex self-review does NOT satisfy the
|
||||||
|
independent-review gate — an independent (author≠reviewer) review + durable Gitea RoR is still required
|
||||||
|
before MS-LEAD merges. STOP at PR-open for MS-LEAD's exact-head review; do NOT self-merge.
|
||||||
|
|
||||||
|
## Session 5 — PR3 review round 2 (finding L + M1/M2/M3), red-first fixes
|
||||||
|
|
||||||
|
Second review pass on the lock-cleanup plumbing surfaced one round-1 residual (L) and three round-2
|
||||||
|
findings (M1 blocker, M2/M3 should-fix). All fixed red-first (RED proven per-finding, then GREEN).
|
||||||
|
|
||||||
|
- **L · should-fix (codex r1) — mutation-lock release swallowed unlink failures.** regen's
|
||||||
|
`acquirePrivateRosterMutationLock` release copied CRUD's `unlink().catch(()=>{})`, hiding a stale
|
||||||
|
`roster.yaml.mutation.lock`. **Fix:** its release PROPAGATES the unlink fault (finding-J stale-lock
|
||||||
|
warning then fires for this lock too). **Test:** acquire real lock, `rm` it, assert `release()` rejects.
|
||||||
|
- **M1 · BLOCKER (codex r2) — replacement-lock race.** The propagating release from L did an
|
||||||
|
UNCONDITIONAL `unlink(lockPath)` without proving ownership. If the lock is cleared + re-created by
|
||||||
|
another writer mid-op, regen deletes the STRANGER's live lock → a third writer enters → mutual
|
||||||
|
exclusion defeated. **Fix (reuse, not reimplement):** generalized the reconciler's ownership-proving
|
||||||
|
lock body into shared `acquirePrivateManagedRosterLock(mosaicHome, lockLeaf, busyMessage, openLock)`;
|
||||||
|
`acquirePrivateReconcileLock` delegates to it (behavior-identical: same leaf/codes/messages), and a NEW
|
||||||
|
hardened `acquirePrivateRosterMutationLock` (now in fleet-reconciler.ts, leaf `roster.yaml.mutation.lock`)
|
||||||
|
records dev/ino + ownership token and RE-PROVES ownership (`assertLockOwnership`) before unlinking —
|
||||||
|
fails closed as `lock-cleanup-failed` if replaced. Removed the crud-based export; reverted
|
||||||
|
`acquireMutationLock` (fleet-agent-crud.ts) to its original inline empty-file/swallowing-release form
|
||||||
|
(CRUD behavior intentionally unchanged). Compatibility: CRUD empty-file `wx` and regen tokened `wx`
|
||||||
|
contend on the same path but never co-own (wx winner owns; loser → concurrent-mutation), so the token
|
||||||
|
is only ever read back by the same regen invocation. **Test:** acquire, `rm`+recreate lock (new inode),
|
||||||
|
assert `release()` rejects AND the replacement survives (not unlinked).
|
||||||
|
- **M2 · should-fix (codex r2) — acquire-unwind fault dropped.** The acquire-failure catch discarded
|
||||||
|
`releaseFleetLocks`' return (a possible fault on the already-held first lock). **Fix:** capture and
|
||||||
|
augment — `const releaseFault = await releaseFleetLocks(releases); throw augmentWithLockCleanupFault(error, releaseFault);`
|
||||||
|
(symmetric to finding J). **Test:** mutation lock acquires w/ faulting release + reconcile acquire
|
||||||
|
throws → thrown error mentions stale/lock, nothing written.
|
||||||
|
- **M3 · should-fix (codex r2 + subagent REQUEST-CHANGES) — cleanup warning named only reconcile lock.**
|
||||||
|
Finding L made the mutation-lock release fault reachable, so the `cleanup` marker can originate from
|
||||||
|
EITHER lock. **Fix:** `formatFleetRegenReport`'s WARNING now names BOTH `roster.yaml.mutation.lock` and
|
||||||
|
`roster.yaml.reconcile.lock`, matching `augmentWithLockCleanupFault`. **Test:** fault the mutation-lock
|
||||||
|
release specifically → report names both lock files.
|
||||||
|
|
||||||
|
**Refactor note (no cycle):** neither fleet-reconciler nor fleet-agent-crud imports the other; regen
|
||||||
|
imports lock acquirers from fleet-reconciler and the projection mapping from fleet-reconciler. The two
|
||||||
|
reconcile-lock reviewers reconciled: independent reviewer validated acquire-time empty-file compatibility
|
||||||
|
(preserved), codex flagged RELEASE-time replacement race (closed by ownership proof) — non-contradictory.
|
||||||
|
|
||||||
|
**Gate suite after fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1275**
|
||||||
|
(regen spec 23, incl. 7 red-first lock regressions E/F/G/K/L/M1/M2/M3). RED proven per-finding by
|
||||||
|
temporary revert before re-applying each fix. Independent (author≠reviewer) review of M1/M2/M3 + codex
|
||||||
|
code/security re-run in flight. STOP at PR-open for MS-LEAD's exact-head review + durable Gitea RoR; do
|
||||||
|
NOT self-merge; #791 umbrella stays OPEN; PR body `Part of #791`.
|
||||||
|
|
||||||
|
### Round 3 review (after M1/M2/M3) — independent review PASS + codex residual-TOCTOU disposition
|
||||||
|
Three reviewers on the post-M1/M2/M3 head:
|
||||||
|
- **Independent (subagent, author≠reviewer) — PASS.** Verified M1/M2/M3 all correctly fixed; "never
|
||||||
|
restarts" is STRUCTURAL (runner never referenced in executable code); no secrets; no deadlock (only
|
||||||
|
regen holds both locks); tests meaningful (assert inode preservation + exact lock-file names). Raised:
|
||||||
|
- **should-fix #1 (fixed, red-first):** generalizing the lock helper left `assertSafeLockLeafIfPresent`/
|
||||||
|
`assertLockOwnership` hardcoding "reconciliation lock" in thrown messages → a MUTATION-lock fault
|
||||||
|
misreported as the reconcile lock, undercutting M3's accurate-diagnosis goal. **Fix:** thread
|
||||||
|
`lockLabel = fleet/<leaf>` through both helpers + the generic lock-io messages, so every fault names
|
||||||
|
the actual lock file. Red-first: strengthened the M1 test to assert `/roster\.yaml\.mutation\.lock/`
|
||||||
|
(RED: got "reconciliation lock"; GREEN after). Also resolves nit #3 (generic-message drift).
|
||||||
|
- **nit #2 (fixed):** `FleetRegenResult.cleanup` JSDoc still said "the shared reconcile lock"; now names
|
||||||
|
both locks (regen holds both).
|
||||||
|
- **nit #4 (fixed):** removed the redundant duplicate `assertLockOwnership` call before unlink
|
||||||
|
(pre-existing in merged main; harmless but dead — dropped since the fn was already being touched).
|
||||||
|
- **Codex security — clean (risk: none).** Validates roster semantics, constrains env values, no shell
|
||||||
|
eval, no secret output, generated-only writes, serialized against both locks.
|
||||||
|
- **Codex code — request-changes, 1 "blocker": residual check-then-unlink TOCTOU.** Between the final
|
||||||
|
`assertLockOwnership` and the path-based `unlink`, an external actor could vacate our inode and a new
|
||||||
|
writer grab the path, so the unlink deletes the stranger's lock. **Disposition: documented known
|
||||||
|
limitation, NOT fixed in PR3.** Rationale: (1) byte-identical to the MERGED, shipped reconcile-lock
|
||||||
|
release on origin/main (fleet-reconciler.ts L654-659) — not introduced here; (2) UNREACHABLE within the
|
||||||
|
`wx` writer protocol — no Mosaic writer removes a lock it doesn't own (wx fails EEXIST while our inode
|
||||||
|
exists), so only external interference can vacate our inode in the sub-instruction window; (3) the
|
||||||
|
ownership guard DOES close the reachable case (stale-lock reaper/operator cleared our lock + another
|
||||||
|
writer took it BEFORE release began → fail closed, don't delete stranger's lock); (4) the true atomic
|
||||||
|
fix — fd-held advisory lock (flock/lockf) adopted by ALL fleet writers (CRUD + reconcile + regen) — is
|
||||||
|
a cross-cutting mechanism change touching merged CRUD + reconciler, out of scope for a projection-only
|
||||||
|
recovery PR. Documented honestly in the acquirer doc + M1 test comment. **The binding independent
|
||||||
|
review did NOT treat this as a blocker.** Recommendation to MS-LEAD: proceed to PR-open + spin a
|
||||||
|
SEPARATE follow-up issue for the fd-advisory-lock migration; MS-LEAD adjudicates scope at exact-head
|
||||||
|
review (merge authority).
|
||||||
|
|
||||||
|
**Gates after round-3 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
|
||||||
|
**1275** (regen spec 23). Fresh codex code re-run in flight to confirm no NEW issues from the label fix.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session 6 — Round 4/5 convergence (stranded-lock robustness)
|
||||||
|
|
||||||
|
**Two independent reviewers converged on the SAME should-fix** on the init-failure cleanup path,
|
||||||
|
strengthening confidence it was real:
|
||||||
|
|
||||||
|
- **Codex code-review-5 — 0 blockers, 1 should-fix.** "Stat failure after lock creation strands the new
|
||||||
|
lock." When `handle.stat()` ITSELF fails right after the `wx` create (transient EIO/EBADF), `created`
|
||||||
|
is `undefined`, so `removeOwnedLockLeafBestEffort` had `if (!created) return;` → no cleanup → the
|
||||||
|
just-created `roster.yaml.mutation.lock`/`reconcile.lock` is stranded, permanently blocking future
|
||||||
|
regen + CRUD. (Notably NO blocker, and the TOCTOU is no longer flagged in code-review as of r5.)
|
||||||
|
- **Independent delta reviewer (author≠reviewer, pr-review-toolkit) — no blockers, same should-fix.**
|
||||||
|
Independently flagged the identical `!created` gap; validated FIX 1 (label threading — no call site
|
||||||
|
missed, codes unchanged, no test depended on old text) and FIX 2 (dev/ino-guarded cleanup, best-effort,
|
||||||
|
happy-path release reuses captured dev/ino) as correct. Suggested an unconditional best-effort unlink
|
||||||
|
in the `!created` branch; I took the **safer** variant below.
|
||||||
|
- **Codex security-review-5 — 0 crit / 0 high / 1 medium.** The single medium is the SAME residual
|
||||||
|
check-then-unlink TOCTOU already dispositioned in round 3 (its own remediation = "migrate every writer
|
||||||
|
to an fd-held advisory lock" = the follow-up issue). No new security finding. No secrets.
|
||||||
|
|
||||||
|
**Fix (red-first, safer than an unconditional unlink):** thread the persisted random `token` into
|
||||||
|
`removeOwnedLockLeafBestEffort`. Two independent ownership proofs now: primary dev/ino (unchanged), and a
|
||||||
|
**fallback** when the post-create stat failed — read the leaf and unlink ONLY if its content equals our
|
||||||
|
`randomUUID()` token. Only OUR lock carries that token, so a CRUD (empty) or differently-tokened
|
||||||
|
replacement is never deleted. `tokenPersisted` guards passing the token (only after `writeFile` lands).
|
||||||
|
Doubly-degenerate case (stat fails AND token write never landed) leaves the lock in place rather than
|
||||||
|
risk deleting a stranger's file — requires two independent fs faults on a just-created fd; documented.
|
||||||
|
|
||||||
|
- **Red-first proof:** new test `does not strand the lock file when the post-create stat itself fails`
|
||||||
|
injects a real `wx` create + a Proxy handle whose `stat()` rejects (writeFile/close succeed), asserts
|
||||||
|
`exists(lockPath) === false`. RED before fix (`expected true to be false` — lock stranded); GREEN after.
|
||||||
|
- **Also fixed (delta nit #3):** `fleet-regen-command.ts` `acquireRosterMutationLock` JSDoc said "CRUD's
|
||||||
|
private lock"; the default is the reconciler's hardened ownership-proving acquirer for the same
|
||||||
|
`fleet/roster.yaml.mutation.lock` path. Corrected.
|
||||||
|
- **PR-description note (delta nit #2):** FIX 1 also collapsed a pre-existing duplicate back-to-back
|
||||||
|
`assertLockOwnership` call in the release closure (identical args, no intervening logic) into one — a
|
||||||
|
no-op simplification of merged code, not a behavior change. Called out so a future reader doesn't
|
||||||
|
wonder if the duplicate had a purpose.
|
||||||
|
|
||||||
|
**Gates after round-4 fixes (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
|
||||||
|
**1277** (regen spec now 25: +1 stat-failure stranded-lock regression). Residual TOCTOU still deferred to
|
||||||
|
the fd-advisory-lock follow-up issue; MS-LEAD adjudicates scope at exact-head review (merge authority).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session 6 — Round 6 (persona-root wiring)
|
||||||
|
|
||||||
|
**Codex code-review-6 — 0 blockers, 1 should-fix (NEW, distinct from the lock work).** "Forward
|
||||||
|
configured persona directories to regen." `registerFleetRegenCommand` was registered at
|
||||||
|
`fleet.ts:2069` with only `{ runner, mosaicHome }`, discarding `deps.reconcileDeps.rolesDir` /
|
||||||
|
`overrideDir`. The regen command ALREADY has those seams (validates roster semantics via
|
||||||
|
`validateRosterV2Semantics({ rolesDir, overrideDir })`, defaulting to `<mosaicHome>/fleet/roles{,.local}`),
|
||||||
|
but the top-level wiring never forwarded the configured roots. **Impact:** in a deployment with custom
|
||||||
|
persona roots, `fleet reconcile` (which honors the overrides) would ACCEPT a roster while `fleet regen`
|
||||||
|
REJECTS the same roster (persona resolution against the wrong default dir) — blocking the recovery
|
||||||
|
command and violating the documented "resolves personas the SAME way reconcile does" contract.
|
||||||
|
|
||||||
|
**Fix (red-first):** forward `rolesDir`/`overrideDir` from `deps.reconcileDeps` into
|
||||||
|
`registerFleetRegenCommand` at `fleet.ts:2069`. Red-first test `forwards configured persona roots
|
||||||
|
(rolesDir/overrideDir) from reconcileDeps into regen`: seeds personas ONLY under a custom root, leaves
|
||||||
|
the default `<home>/fleet/roles` empty, registers with `reconcileDeps: { rolesDir, overrideDir }`, and
|
||||||
|
requires `fleet regen` to SUCCEED. RED before fix (`expected 1 not to be 1` — regen validated against the
|
||||||
|
empty default and exited 1); GREEN after.
|
||||||
|
|
||||||
|
**Codex security-review-6 — 0 crit / 0 high / 1 medium.** Same residual check-then-unlink TOCTOU, now
|
||||||
|
noted at BOTH the release closure and the init-cleanup path; remediation = fd-held advisory lock across
|
||||||
|
all writers = the SAME deferred follow-up item. No new security finding, no secrets.
|
||||||
|
|
||||||
|
**Independent confirmation review of the token-fallback fix (Session 6/round 4) — PASS, no findings.**
|
||||||
|
All 7 verification points confirmed; reviewer mechanically reverted `removeOwnedLockLeafBestEffort` to
|
||||||
|
the pre-fix `if (!created) return;` and re-ran the new test → RED (`expected true to be false`),
|
||||||
|
confirming the test genuinely pins the fix; restored after. No lint/type issues; doc-comment accurate.
|
||||||
|
|
||||||
|
**Gates after round-6 fix (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest
|
||||||
|
**1278** (regen spec now 26: +1 persona-root wiring regression).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session 6 — Round 7 convergence (review CLOSED for PR-open)
|
||||||
|
|
||||||
|
- **Codex code-review-7 — 0 blockers, 1 should-fix = the residual TOCTOU** (previously a "blocker" in r3,
|
||||||
|
dropped in r4/r5, now re-surfaced as a should-fix). **Codex security-review-7 — 0 crit / 0 high /
|
||||||
|
1 medium = the SAME residual TOCTOU.** Codex has CONVERGED: the only remaining finding across both
|
||||||
|
streams is that one race, whose own remediation is "fd-held advisory lock shared by all fleet writers"
|
||||||
|
= the deferred follow-up. No new distinct finding; the wiring fix introduced nothing.
|
||||||
|
- **Independent confirmation review of the persona-root wiring fix — PASS, no findings.** Reviewer
|
||||||
|
mechanically reverted the two forwarded lines → RED (`Roster v2 agent "coder0" class "code" does not
|
||||||
|
resolve to a readable persona` → exit 1), restored → GREEN (26 regen + 204 fleet tests). Confirmed the
|
||||||
|
optional-chaining fallback preserves default-deployment behavior and no type/lint issue.
|
||||||
|
|
||||||
|
**Review disposition for PR-open:** ALL actionable findings fixed red-first across rounds 3–6 (label
|
||||||
|
threading, stranded-lock on init failure, stat-failure strand, persona-root wiring). The residual
|
||||||
|
check-then-unlink TOCTOU is the ONLY open item and is DEFERRED to a follow-up issue (fd-advisory-lock
|
||||||
|
migration across CRUD + reconcile + regen) — byte-identical to merged origin/main's reconcile-lock
|
||||||
|
release, unreachable within the `wx` writer protocol (no Mosaic writer removes a lock it doesn't own;
|
||||||
|
only external `rm`/a stale-lock reaper can vacate the inode mid-release), and its true fix is a
|
||||||
|
cross-cutting mechanism change out of scope for a projection-only recovery PR. Two independent human-agent
|
||||||
|
reviews (author≠reviewer) treated it as non-blocking. MS-LEAD adjudicates scope at exact-head review
|
||||||
|
(merge authority); recommendation = proceed to PR-open + spin the follow-up issue.
|
||||||
|
|
||||||
|
**Final gates (all green):** typecheck ✓ · lint ✓ · format:check ✓ · full mosaic vitest **1278**
|
||||||
|
(regen spec 26). No secret values in any snapshot/projection/report output (counts + paths only). Regen
|
||||||
|
NEVER issues a lifecycle/restart call (load-bearing recordingRunner gate). STOP at PR-open for MS-LEAD's
|
||||||
|
exact-head review + durable Reviewer-of-Record before any merge; do NOT self-merge.
|
||||||
|
|||||||
@@ -70,6 +70,8 @@ Security vulnerability review focusing on:
|
|||||||
~/.config/mosaic/tools/codex/codex-security-review.sh -n 42
|
~/.config/mosaic/tools/codex/codex-security-review.sh -n 42
|
||||||
```
|
```
|
||||||
|
|
||||||
|
PR mode resolves the provider's PR diff rather than relying on the caller's checked-out branch. On Gitea, it fetches the base and `refs/pull/<number>/head` refs and diffs those explicit refs. If the refs cannot be fetched or the resulting diff is empty, the command exits nonzero before Codex runs or a review is posted.
|
||||||
|
|
||||||
### Review Against Base Branch
|
### Review Against Base Branch
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -253,7 +255,7 @@ Run the script from inside a git repository.
|
|||||||
|
|
||||||
### "No changes found to review"
|
### "No changes found to review"
|
||||||
|
|
||||||
The specified mode (--uncommitted, --base, etc.) found no changes to review.
|
The specified non-PR mode (`--uncommitted`, `--base`, etc.) found no changes to review. PR mode instead fails closed with an actionable error when it cannot construct a non-empty provider diff; verify the PR number, remote, provider login, and ref access before retrying.
|
||||||
|
|
||||||
### "Codex produced no output"
|
### "Codex produced no output"
|
||||||
|
|
||||||
|
|||||||
@@ -44,38 +44,47 @@ build_diff_context() {
|
|||||||
diff_text=$(git show "$value" 2>/dev/null)
|
diff_text=$(git show "$value" 2>/dev/null)
|
||||||
;;
|
;;
|
||||||
pr)
|
pr)
|
||||||
# For PRs, we need to fetch the PR diff
|
# Provider detection writes its result to stdout; suppress it so it cannot
|
||||||
detect_platform
|
# be mistaken for diff content when this function is used in a substitution.
|
||||||
|
detect_platform >/dev/null
|
||||||
if [[ "$PLATFORM" == "github" ]]; then
|
if [[ "$PLATFORM" == "github" ]]; then
|
||||||
diff_text=$(gh pr diff "$value" 2>/dev/null)
|
diff_text=$(gh pr diff "$value" 2>/dev/null) || {
|
||||||
|
echo "Error: Failed to fetch the diff for PR #${value}." >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
elif [[ "$PLATFORM" == "gitea" ]]; then
|
elif [[ "$PLATFORM" == "gitea" ]]; then
|
||||||
# tea doesn't have a direct pr diff command, use git
|
local pr_base base_ref pr_head_ref
|
||||||
local pr_base
|
pr_base=$(tea pr list --fields index,base --output simple 2>/dev/null | awk -v pr="$value" '$1 == pr { print $2; exit }')
|
||||||
pr_base=$(tea pr list --fields index,base --output simple 2>/dev/null | grep "^${value}" | awk '{print $2}')
|
if [[ -z "$pr_base" ]]; then
|
||||||
if [[ -n "$pr_base" ]]; then
|
echo "Error: Could not resolve the base branch for Gitea PR #${value}." >&2
|
||||||
diff_text=$(git diff "${pr_base}...HEAD" 2>/dev/null)
|
return 1
|
||||||
else
|
|
||||||
# Fallback: fetch PR info via API
|
|
||||||
local repo_info
|
|
||||||
repo_info=$(get_repo_info)
|
|
||||||
local remote_url
|
|
||||||
remote_url=$(git remote get-url origin 2>/dev/null)
|
|
||||||
local host
|
|
||||||
host=$(echo "$remote_url" | sed -E 's|.*://([^/]+).*|\1|; s|.*@([^:]+).*|\1|')
|
|
||||||
diff_text=$(curl -s "https://${host}/api/v1/repos/${repo_info}/pulls/${value}" \
|
|
||||||
-H "Authorization: token $(tea login list --output simple 2>/dev/null | head -1 | awk '{print $2}')" \
|
|
||||||
2>/dev/null | jq -r '.diff_url // empty')
|
|
||||||
if [[ -n "$diff_text" && "$diff_text" != "null" ]]; then
|
|
||||||
diff_text=$(curl -s "$diff_text" 2>/dev/null)
|
|
||||||
else
|
|
||||||
diff_text=$(git diff "main...HEAD" 2>/dev/null)
|
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
base_ref="refs/remotes/origin/${pr_base}"
|
||||||
|
pr_head_ref="refs/remotes/origin/pr/${value}/head"
|
||||||
|
if ! git fetch --quiet origin \
|
||||||
|
"+refs/heads/${pr_base}:${base_ref}" \
|
||||||
|
"+refs/pull/${value}/head:${pr_head_ref}"; then
|
||||||
|
echo "Error: Failed to fetch the base and head refs for Gitea PR #${value}." >&2
|
||||||
|
return 1
|
||||||
fi
|
fi
|
||||||
|
diff_text=$(git diff "${base_ref}...${pr_head_ref}") || {
|
||||||
|
echo "Error: Failed to diff the fetched refs for Gitea PR #${value}." >&2
|
||||||
|
return 1
|
||||||
|
}
|
||||||
|
else
|
||||||
|
echo "Error: Unsupported git platform while resolving PR #${value}." >&2
|
||||||
|
return 1
|
||||||
fi
|
fi
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
echo "$diff_text"
|
if [[ "$mode" == "pr" && -z "${diff_text//[[:space:]]/}" ]]; then
|
||||||
|
echo "Error: Unable to construct a non-empty diff for PR #${value}; verify the PR refs and provider access." >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
printf '%s\n' "$diff_text"
|
||||||
}
|
}
|
||||||
|
|
||||||
# Format JSON findings as markdown for PR comments
|
# Format JSON findings as markdown for PR comments
|
||||||
|
|||||||
158
packages/mosaic/framework/tools/codex/test-pr-diff-context.sh
Normal file
158
packages/mosaic/framework/tools/codex/test-pr-diff-context.sh
Normal file
@@ -0,0 +1,158 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Hermetic regression coverage for Gitea PR diff construction and fail-closed reviews.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
TMP_DIR=$(mktemp -d)
|
||||||
|
trap 'rm -rf "$TMP_DIR"' EXIT
|
||||||
|
|
||||||
|
fail() {
|
||||||
|
echo "not ok - $*" >&2
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
assert_contains() {
|
||||||
|
local haystack="$1" needle="$2"
|
||||||
|
if [[ "$haystack" != *"$needle"* ]]; then
|
||||||
|
printf 'actual output:\n%s\n' "$haystack" >&2
|
||||||
|
fail "expected output to contain: $needle"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Prevent CI-provided repository context from leaking into the fixture repositories.
|
||||||
|
unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_OBJECT_DIRECTORY \
|
||||||
|
GIT_ALTERNATE_OBJECT_DIRECTORIES GIT_COMMON_DIR
|
||||||
|
export GIT_AUTHOR_NAME="Codex Fixture"
|
||||||
|
export GIT_AUTHOR_EMAIL="codex-fixture@example.test"
|
||||||
|
export GIT_COMMITTER_NAME="$GIT_AUTHOR_NAME"
|
||||||
|
export GIT_COMMITTER_EMAIL="$GIT_AUTHOR_EMAIL"
|
||||||
|
export GITEA_LOGIN="fixture"
|
||||||
|
export GITEA_TOKEN="fixture-token"
|
||||||
|
export GITEA_URL="file://$TMP_DIR"
|
||||||
|
|
||||||
|
create_pr_fixture() {
|
||||||
|
local fixture_root="$1" head_mode="$2"
|
||||||
|
local origin="$fixture_root/origin.git"
|
||||||
|
local seed="$fixture_root/seed"
|
||||||
|
local work="$fixture_root/work"
|
||||||
|
local base_sha head_sha
|
||||||
|
|
||||||
|
mkdir -p "$fixture_root"
|
||||||
|
git init --quiet --bare "$origin"
|
||||||
|
git init --quiet --initial-branch=release/next "$seed"
|
||||||
|
printf 'base\n' > "$seed/pr-change.ts"
|
||||||
|
git -C "$seed" add pr-change.ts
|
||||||
|
git -C "$seed" commit --quiet -m "fixture base"
|
||||||
|
base_sha=$(git -C "$seed" rev-parse HEAD)
|
||||||
|
git -C "$seed" remote add origin "$origin"
|
||||||
|
git -C "$seed" push --quiet origin release/next
|
||||||
|
git --git-dir="$origin" symbolic-ref HEAD refs/heads/release/next
|
||||||
|
|
||||||
|
if [[ "$head_mode" == "changed" ]]; then
|
||||||
|
git -C "$seed" switch --quiet -c feature/pr-795
|
||||||
|
printf 'actual-pr-change\n' > "$seed/pr-change.ts"
|
||||||
|
git -C "$seed" commit --quiet -am "fixture PR head"
|
||||||
|
head_sha=$(git -C "$seed" rev-parse HEAD)
|
||||||
|
git -C "$seed" push --quiet origin HEAD:refs/pull/795/head
|
||||||
|
else
|
||||||
|
head_sha="$base_sha"
|
||||||
|
git --git-dir="$origin" update-ref refs/pull/795/head "$head_sha"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Gitea's provider-owned PR head ref now exists in the local bare origin.
|
||||||
|
git clone --quiet "$origin" "$work"
|
||||||
|
printf '%s\n' "$work"
|
||||||
|
}
|
||||||
|
|
||||||
|
FAKE_BIN="$TMP_DIR/bin"
|
||||||
|
mkdir -p "$FAKE_BIN"
|
||||||
|
cat > "$FAKE_BIN/tea" <<'STUB'
|
||||||
|
#!/bin/bash
|
||||||
|
if [[ "$*" == "pr list --fields index,base --output simple" ]]; then
|
||||||
|
printf '795 release/next\n'
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
exit 1
|
||||||
|
STUB
|
||||||
|
cat > "$FAKE_BIN/codex" <<'STUB'
|
||||||
|
#!/bin/bash
|
||||||
|
printf 'CODEX %s\n' "$*" >> "$CODEX_LOG"
|
||||||
|
exit 99
|
||||||
|
STUB
|
||||||
|
chmod +x "$FAKE_BIN/tea" "$FAKE_BIN/codex"
|
||||||
|
export PATH="$FAKE_BIN:$PATH"
|
||||||
|
|
||||||
|
# The valid fixture is a fresh clone on the non-main base. The PR head exists only
|
||||||
|
# at refs/pull/795/head, so local HEAD cannot accidentally satisfy the assertion.
|
||||||
|
if [[ "${1:-all}" != "fail-closed" ]]; then
|
||||||
|
VALID_WORK=$(create_pr_fixture "$TMP_DIR/valid" changed)
|
||||||
|
(
|
||||||
|
cd "$VALID_WORK"
|
||||||
|
# shellcheck source=common.sh
|
||||||
|
source "$SCRIPT_DIR/common.sh"
|
||||||
|
diff_context=$(build_diff_context pr 795)
|
||||||
|
assert_contains "$diff_context" "actual-pr-change"
|
||||||
|
|
||||||
|
base_sha=$(git rev-parse refs/remotes/origin/release/next)
|
||||||
|
head_sha=$(git rev-parse refs/remotes/origin/pr/795/head)
|
||||||
|
local_sha=$(git rev-parse HEAD)
|
||||||
|
[[ "$local_sha" == "$base_sha" ]] || fail "fixture clone is not on the PR base"
|
||||||
|
[[ "$head_sha" != "$base_sha" ]] || fail "fixture PR head does not differ from its base"
|
||||||
|
git show-ref --verify --quiet refs/remotes/origin/pr/795/head || \
|
||||||
|
fail "fetched PR head ref is missing"
|
||||||
|
[[ "$(git diff --name-only "${base_sha}...${head_sha}")" == "pr-change.ts" ]] || \
|
||||||
|
fail "explicit PR refs do not contain the fixture change"
|
||||||
|
if git show-ref --verify --quiet refs/heads/main || \
|
||||||
|
git show-ref --verify --quiet refs/remotes/origin/main; then
|
||||||
|
fail "fixture unexpectedly contains a main ref"
|
||||||
|
fi
|
||||||
|
)
|
||||||
|
echo "ok - Gitea PR mode fetches and diffs explicit non-main base and PR head refs"
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Build an empty PR entirely inside another local repository. Both review wrappers
|
||||||
|
# must emit the PR-numbered error before Codex or the stubbed post path can execute.
|
||||||
|
if [[ "${1:-all}" != "pr-head" ]]; then
|
||||||
|
EMPTY_WORK=$(create_pr_fixture "$TMP_DIR/empty" empty)
|
||||||
|
SANDBOX="$TMP_DIR/sandbox"
|
||||||
|
mkdir -p "$SANDBOX/tools/codex/schemas" "$SANDBOX/tools/git"
|
||||||
|
cp "$SCRIPT_DIR/common.sh" \
|
||||||
|
"$SCRIPT_DIR/codex-code-review.sh" \
|
||||||
|
"$SCRIPT_DIR/codex-security-review.sh" \
|
||||||
|
"$SANDBOX/tools/codex/"
|
||||||
|
cp "$SCRIPT_DIR/schemas/code-review-schema.json" \
|
||||||
|
"$SCRIPT_DIR/schemas/security-review-schema.json" \
|
||||||
|
"$SANDBOX/tools/codex/schemas/"
|
||||||
|
cp "$SCRIPT_DIR/../git/detect-platform.sh" "$SANDBOX/tools/git/"
|
||||||
|
|
||||||
|
cat > "$SANDBOX/tools/git/pr-review.sh" <<'STUB'
|
||||||
|
#!/bin/bash
|
||||||
|
printf 'POST %s\n' "$*" >> "$POST_LOG"
|
||||||
|
STUB
|
||||||
|
chmod +x "$SANDBOX/tools/git/pr-review.sh"
|
||||||
|
|
||||||
|
POST_LOG="$TMP_DIR/post.log"
|
||||||
|
CODEX_LOG="$TMP_DIR/codex.log"
|
||||||
|
export POST_LOG CODEX_LOG
|
||||||
|
|
||||||
|
for review_kind in code security; do
|
||||||
|
: > "$POST_LOG"
|
||||||
|
: > "$CODEX_LOG"
|
||||||
|
review_script="$SANDBOX/tools/codex/codex-${review_kind}-review.sh"
|
||||||
|
set +e
|
||||||
|
(
|
||||||
|
cd "$EMPTY_WORK"
|
||||||
|
"$review_script" -n 795
|
||||||
|
) >"$TMP_DIR/${review_kind}.stdout" 2>"$TMP_DIR/${review_kind}.stderr"
|
||||||
|
review_status=$?
|
||||||
|
set -e
|
||||||
|
|
||||||
|
stderr_text=$(cat "$TMP_DIR/${review_kind}.stderr")
|
||||||
|
[[ "$review_status" -ne 0 ]] || fail "${review_kind} review returned success for an empty PR diff"
|
||||||
|
[[ ! -s "$CODEX_LOG" ]] || fail "Codex ran for an empty ${review_kind} PR diff"
|
||||||
|
[[ ! -s "$POST_LOG" ]] || fail "${review_kind} review auto-post ran for an empty PR diff"
|
||||||
|
assert_contains "$stderr_text" "Error:"
|
||||||
|
assert_contains "$stderr_text" "PR #795"
|
||||||
|
echo "ok - empty ${review_kind} PR diff fails closed before Codex and auto-post"
|
||||||
|
done
|
||||||
|
fi
|
||||||
@@ -24,7 +24,8 @@
|
|||||||
"build": "tsc",
|
"build": "tsc",
|
||||||
"lint": "eslint src",
|
"lint": "eslint src",
|
||||||
"typecheck": "tsc --noEmit",
|
"typecheck": "tsc --noEmit",
|
||||||
"test": "vitest run --passWithNoTests"
|
"test": "vitest run --passWithNoTests && pnpm run test:framework-shell",
|
||||||
|
"test:framework-shell": "bash framework/tools/codex/test-pr-diff-context.sh"
|
||||||
},
|
},
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@mosaicstack/brain": "workspace:*",
|
"@mosaicstack/brain": "workspace:*",
|
||||||
|
|||||||
885
packages/mosaic/src/commands/fleet-regen-command.spec.ts
Normal file
885
packages/mosaic/src/commands/fleet-regen-command.spec.ts
Normal file
@@ -0,0 +1,885 @@
|
|||||||
|
import { chmod, mkdir, mkdtemp, open, readFile, rm, stat, writeFile } from 'node:fs/promises';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { join } from 'node:path';
|
||||||
|
import { Command } from 'commander';
|
||||||
|
import { afterEach, describe, expect, it, vi } from 'vitest';
|
||||||
|
import { registerFleetCommand, type CommandResult, type FleetCommandDeps } from './fleet.js';
|
||||||
|
import { executeFleetRegen, formatFleetRegenReport } from './fleet-regen-command.js';
|
||||||
|
import {
|
||||||
|
acquirePrivateRosterMutationLock,
|
||||||
|
projectRosterV2AgentGeneratedEnv,
|
||||||
|
} from '../fleet/fleet-reconciler.js';
|
||||||
|
import { applyPreparedGeneratedAgentEnvironmentProjection } from '../fleet/generated-env-boundary.js';
|
||||||
|
import { parseRosterV2 } from '../fleet/roster-v2.js';
|
||||||
|
|
||||||
|
// A two-agent roster-v2 SSOT. `mosaic fleet regen` must rebuild each agent's
|
||||||
|
// `fleet/agents/<name>.env.generated` projection from exactly this source and
|
||||||
|
// nothing else — deterministically, without ever touching agent lifecycle.
|
||||||
|
const rosterYaml = `
|
||||||
|
version: 2
|
||||||
|
generation: 7
|
||||||
|
transport: tmux
|
||||||
|
tmux:
|
||||||
|
socket_name: mosaic-fleet
|
||||||
|
holder_session: _holder
|
||||||
|
defaults:
|
||||||
|
working_directory: /srv/mosaic
|
||||||
|
runtime: pi
|
||||||
|
runtimes:
|
||||||
|
pi:
|
||||||
|
reset_command: /new
|
||||||
|
agents:
|
||||||
|
- name: coder0
|
||||||
|
alias: Coder 0
|
||||||
|
class: code
|
||||||
|
runtime: pi
|
||||||
|
provider: openai
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
reasoning: high
|
||||||
|
tool_policy: code
|
||||||
|
working_directory: /srv/mosaic
|
||||||
|
persistent_persona: false
|
||||||
|
reset_between_tasks: true
|
||||||
|
lifecycle:
|
||||||
|
enabled: true
|
||||||
|
desired_state: stopped
|
||||||
|
launch:
|
||||||
|
yolo: true
|
||||||
|
- name: coder1
|
||||||
|
alias: Coder 1
|
||||||
|
class: code
|
||||||
|
runtime: pi
|
||||||
|
provider: openai
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
reasoning: medium
|
||||||
|
tool_policy: code
|
||||||
|
working_directory: /srv/other
|
||||||
|
persistent_persona: false
|
||||||
|
reset_between_tasks: true
|
||||||
|
lifecycle:
|
||||||
|
enabled: true
|
||||||
|
desired_state: stopped
|
||||||
|
launch:
|
||||||
|
yolo: true
|
||||||
|
`;
|
||||||
|
|
||||||
|
let cleanup: string | undefined;
|
||||||
|
|
||||||
|
afterEach(async (): Promise<void> => {
|
||||||
|
vi.restoreAllMocks();
|
||||||
|
process.exitCode = undefined;
|
||||||
|
if (cleanup) await rm(cleanup, { recursive: true, force: true });
|
||||||
|
cleanup = undefined;
|
||||||
|
});
|
||||||
|
|
||||||
|
async function fleetHome(withRoster = rosterYaml): Promise<string> {
|
||||||
|
cleanup = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-command-'));
|
||||||
|
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
|
||||||
|
await mkdir(join(cleanup, directory), { recursive: true, mode: 0o700 });
|
||||||
|
await chmod(join(cleanup, directory), 0o700);
|
||||||
|
}
|
||||||
|
await chmod(cleanup, 0o700);
|
||||||
|
await writeFile(join(cleanup, 'fleet', 'roster.yaml'), withRoster, { mode: 0o600 });
|
||||||
|
// Semantic roster validation (matching `fleet reconcile`) resolves each agent
|
||||||
|
// class to a persona; seed the classes the fixtures reference.
|
||||||
|
for (const klass of ['code', 'merge-gate']) {
|
||||||
|
await writeFile(
|
||||||
|
join(cleanup, 'fleet', 'roles', `${klass}.md`),
|
||||||
|
`\`class: ${klass}\`\n\n# ${klass} persona\n`,
|
||||||
|
{ mode: 0o600 },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return cleanup;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A runner spy that records EVERY invocation. `regen` is projection-only and
|
||||||
|
* must never issue a lifecycle/restart call, so a non-empty call log is a
|
||||||
|
* hard failure — this is the load-bearing "never restarts" gate.
|
||||||
|
*/
|
||||||
|
function recordingRunner(
|
||||||
|
calls: string[][],
|
||||||
|
): (command: string, args: string[]) => Promise<CommandResult> {
|
||||||
|
return async (command: string, args: string[]): Promise<CommandResult> => {
|
||||||
|
calls.push([command, ...args]);
|
||||||
|
return { stdout: '', stderr: '', exitCode: 0 };
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function program(mosaicHome: string, runner: FleetCommandDeps['runner']): Command {
|
||||||
|
const result = new Command();
|
||||||
|
result.exitOverride();
|
||||||
|
registerFleetCommand(result, { mosaicHome, runner });
|
||||||
|
return result;
|
||||||
|
}
|
||||||
|
|
||||||
|
function capture(): string[] {
|
||||||
|
const lines: string[] = [];
|
||||||
|
vi.spyOn(console, 'log').mockImplementation((value: string): void => {
|
||||||
|
lines.push(value);
|
||||||
|
});
|
||||||
|
return lines;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function exists(path: string): Promise<boolean> {
|
||||||
|
try {
|
||||||
|
await stat(path);
|
||||||
|
return true;
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('projectRosterV2AgentGeneratedEnv', (): void => {
|
||||||
|
it('maps a roster-v2 agent to exactly the eight generated projection keys', (): void => {
|
||||||
|
const roster = parseRosterV2(rosterYaml, 'yaml');
|
||||||
|
const agent = roster.agents.find((candidate) => candidate.name === 'coder0');
|
||||||
|
expect(agent).toBeDefined();
|
||||||
|
const values = projectRosterV2AgentGeneratedEnv(roster, agent!);
|
||||||
|
expect(values).toEqual({
|
||||||
|
MOSAIC_AGENT_NAME: 'coder0',
|
||||||
|
MOSAIC_AGENT_CLASS: 'code',
|
||||||
|
MOSAIC_AGENT_RUNTIME: 'pi',
|
||||||
|
MOSAIC_AGENT_MODEL: 'gpt-5.6-sol',
|
||||||
|
MOSAIC_AGENT_REASONING: 'high',
|
||||||
|
MOSAIC_AGENT_TOOL_POLICY: 'code',
|
||||||
|
MOSAIC_AGENT_WORKDIR: '/srv/mosaic',
|
||||||
|
MOSAIC_TMUX_SOCKET: 'mosaic-fleet',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('mosaic fleet regen', (): void => {
|
||||||
|
it('is dry-run by default: reports the plan and writes nothing', async (): Promise<void> => {
|
||||||
|
const home = await fleetHome();
|
||||||
|
const calls: string[][] = [];
|
||||||
|
const lines = capture();
|
||||||
|
|
||||||
|
await program(home, recordingRunner(calls)).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--json',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const result = JSON.parse(lines.pop() ?? '{}');
|
||||||
|
expect(result).toMatchObject({
|
||||||
|
mode: 'dry-run',
|
||||||
|
generation: 7,
|
||||||
|
agentCount: 2,
|
||||||
|
written: 0,
|
||||||
|
});
|
||||||
|
expect(result.agents.map((agent: { name: string }) => agent.name)).toEqual([
|
||||||
|
'coder0',
|
||||||
|
'coder1',
|
||||||
|
]);
|
||||||
|
expect(
|
||||||
|
result.agents.every((agent: { disposition: string }) => agent.disposition === 'create'),
|
||||||
|
).toBe(true);
|
||||||
|
// Nothing on disk.
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
|
||||||
|
// No lifecycle/restart call — ever.
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('forwards configured persona roots (rolesDir/overrideDir) from reconcileDeps into regen', async (): Promise<void> => {
|
||||||
|
// Codex r6: regen must resolve personas the SAME way reconcile does. If the
|
||||||
|
// top-level registration drops reconcileDeps.rolesDir/overrideDir, a deployment
|
||||||
|
// with custom persona roots gets reconcile ACCEPTING a roster while regen
|
||||||
|
// REJECTS it (validating against the wrong default `<home>/fleet/roles`),
|
||||||
|
// blocking the recovery command. Pin the wiring: seed personas ONLY under the
|
||||||
|
// custom roots, leave the default roles dir empty, require regen to succeed.
|
||||||
|
const home = await mkdtemp(join(tmpdir(), 'mosaic-fleet-regen-wiring-'));
|
||||||
|
cleanup = home;
|
||||||
|
for (const directory of ['fleet', 'fleet/agents', 'fleet/roles']) {
|
||||||
|
await mkdir(join(home, directory), { recursive: true, mode: 0o700 });
|
||||||
|
await chmod(join(home, directory), 0o700);
|
||||||
|
}
|
||||||
|
await chmod(home, 0o700);
|
||||||
|
await writeFile(join(home, 'fleet', 'roster.yaml'), rosterYaml, { mode: 0o600 });
|
||||||
|
// Personas live ONLY under the configured roots — the default `fleet/roles`
|
||||||
|
// stays empty, so broken wiring fails persona resolution.
|
||||||
|
const customRoles = join(home, 'custom-roles');
|
||||||
|
const customOverride = join(home, 'custom-roles.local');
|
||||||
|
await mkdir(customRoles, { recursive: true, mode: 0o700 });
|
||||||
|
await mkdir(customOverride, { recursive: true, mode: 0o700 });
|
||||||
|
for (const klass of ['code', 'merge-gate']) {
|
||||||
|
await writeFile(join(customRoles, `${klass}.md`), `\`class: ${klass}\`\n\n# ${klass}\n`, {
|
||||||
|
mode: 0o600,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
const command = new Command();
|
||||||
|
command.exitOverride();
|
||||||
|
registerFleetCommand(command, {
|
||||||
|
mosaicHome: home,
|
||||||
|
runner: recordingRunner([]),
|
||||||
|
reconcileDeps: { rolesDir: customRoles, overrideDir: customOverride },
|
||||||
|
});
|
||||||
|
const lines = capture();
|
||||||
|
|
||||||
|
await command.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
|
||||||
|
|
||||||
|
// Regen validated against the CONFIGURED roots → success. Broken wiring
|
||||||
|
// resolves against the empty default and fails (exitCode 1, no JSON result).
|
||||||
|
expect(process.exitCode).not.toBe(1);
|
||||||
|
const result = JSON.parse(lines.pop() ?? '{}');
|
||||||
|
expect(result).toMatchObject({ mode: 'dry-run', agentCount: 2 });
|
||||||
|
});
|
||||||
|
|
||||||
|
it('--write rebuilds each generated projection from the roster SSOT', async (): Promise<void> => {
|
||||||
|
const home = await fleetHome();
|
||||||
|
const calls: string[][] = [];
|
||||||
|
const lines = capture();
|
||||||
|
|
||||||
|
await program(home, recordingRunner(calls)).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
'--json',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const result = JSON.parse(lines.pop() ?? '{}');
|
||||||
|
expect(result).toMatchObject({ mode: 'write', written: 2, agentCount: 2 });
|
||||||
|
|
||||||
|
const coder0 = await readFile(join(home, 'fleet', 'agents', 'coder0.env.generated'), 'utf8');
|
||||||
|
expect(coder0).toContain('MOSAIC_AGENT_NAME=coder0');
|
||||||
|
expect(coder0).toContain('MOSAIC_AGENT_WORKDIR=/srv/mosaic');
|
||||||
|
expect(coder0).toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet');
|
||||||
|
const coder1 = await readFile(join(home, 'fleet', 'agents', 'coder1.env.generated'), 'utf8');
|
||||||
|
expect(coder1).toContain('MOSAIC_AGENT_NAME=coder1');
|
||||||
|
expect(coder1).toContain('MOSAIC_AGENT_WORKDIR=/srv/other');
|
||||||
|
// No lifecycle/restart call — ever.
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('is deterministic and idempotent across repeated --write runs', async (): Promise<void> => {
|
||||||
|
const home = await fleetHome();
|
||||||
|
const calls: string[][] = [];
|
||||||
|
const path = join(home, 'fleet', 'agents', 'coder0.env.generated');
|
||||||
|
|
||||||
|
const cli = program(home, recordingRunner(calls));
|
||||||
|
capture();
|
||||||
|
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
|
||||||
|
const first = await readFile(path, 'utf8');
|
||||||
|
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
|
||||||
|
const second = await readFile(path, 'utf8');
|
||||||
|
|
||||||
|
expect(second).toBe(first);
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reports rebuild disposition once the generated projection already exists', async (): Promise<void> => {
|
||||||
|
const home = await fleetHome();
|
||||||
|
const calls: string[][] = [];
|
||||||
|
const cli = program(home, recordingRunner(calls));
|
||||||
|
const lines = capture();
|
||||||
|
|
||||||
|
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
|
||||||
|
lines.length = 0;
|
||||||
|
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
|
||||||
|
|
||||||
|
const result = JSON.parse(lines.pop() ?? '{}');
|
||||||
|
expect(
|
||||||
|
result.agents.every((agent: { disposition: string }) => agent.disposition === 'rebuild'),
|
||||||
|
).toBe(true);
|
||||||
|
expect(result.written).toBe(0);
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('never issues a lifecycle/restart call in either mode', async (): Promise<void> => {
|
||||||
|
const home = await fleetHome();
|
||||||
|
const calls: string[][] = [];
|
||||||
|
const cli = program(home, recordingRunner(calls));
|
||||||
|
capture();
|
||||||
|
|
||||||
|
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--json']);
|
||||||
|
await cli.parseAsync(['node', 'mosaic', 'fleet', 'regen', '--write', '--json']);
|
||||||
|
|
||||||
|
// Structural guarantee: regen has no path to systemctl/tmux at all.
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
const systemctlCalls = calls.filter(([command]) => command === 'systemctl');
|
||||||
|
expect(systemctlCalls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('human-readable --write output prints the do-not-restart recovery runbook', async (): Promise<void> => {
|
||||||
|
const home = await fleetHome();
|
||||||
|
const lines = capture();
|
||||||
|
|
||||||
|
await program(home, recordingRunner([])).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const output = lines.join('\n');
|
||||||
|
expect(output).toMatch(/do not restart|before.*restart/i);
|
||||||
|
expect(output).toMatch(/env\.generated/);
|
||||||
|
// The unit has no EnvironmentFile= directive, so the runbook must NOT tell the
|
||||||
|
// operator to verify one — verify the launcher/generated file instead.
|
||||||
|
expect(output).not.toContain('EnvironmentFile');
|
||||||
|
expect(output).toMatch(/systemctl --user cat mosaic-agent@<name>/);
|
||||||
|
expect(output).toMatch(/restart/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('emits paths and counts only — never the projected env body (secrev)', async (): Promise<void> => {
|
||||||
|
const home = await fleetHome();
|
||||||
|
const lines = capture();
|
||||||
|
|
||||||
|
await program(home, recordingRunner([])).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
]);
|
||||||
|
|
||||||
|
const output = lines.join('\n');
|
||||||
|
// Relative paths + counts are fine; the rendered projection body (KEY=value
|
||||||
|
// lines) must never be echoed to stdout.
|
||||||
|
expect(output).toContain('coder0.env.generated');
|
||||||
|
expect(output).not.toMatch(/^MOSAIC_AGENT_\w+=/m);
|
||||||
|
expect(output).not.toContain('MOSAIC_TMUX_SOCKET=mosaic-fleet');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('is projection-only: leaves legacy .env untouched and writes no .env.local/.env.quarantine', async (): Promise<void> => {
|
||||||
|
// Recovery contract: regen rebuilds ONLY <name>.env.generated. It must never
|
||||||
|
// relocate, quarantine, or unlink the operator-owned legacy .env surface — the
|
||||||
|
// full reconciler apply path does, so regen must NOT use it.
|
||||||
|
const home = await fleetHome();
|
||||||
|
const legacyPath = join(home, 'fleet', 'agents', 'coder0.env');
|
||||||
|
await writeFile(legacyPath, 'MOSAIC_RUNTIME_BIN=/usr/bin/pi\n', { mode: 0o600 });
|
||||||
|
capture();
|
||||||
|
|
||||||
|
await program(home, recordingRunner([])).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
'--json',
|
||||||
|
]);
|
||||||
|
|
||||||
|
// Generated projection rebuilt...
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
|
||||||
|
// ...but the operator's legacy .env is preserved verbatim, and no local/quarantine
|
||||||
|
// files were fabricated from it.
|
||||||
|
expect(await readFile(legacyPath, 'utf8')).toBe('MOSAIC_RUNTIME_BIN=/usr/bin/pi\n');
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.local'))).toBe(false);
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.quarantine'))).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('prepares every agent before writing any: a later prepare failure leaves nothing written', async (): Promise<void> => {
|
||||||
|
// Fail-closed across agents. Pre-seed coder1's projection with world-readable
|
||||||
|
// perms so its prepare rejects; coder0 (valid) must NOT be written because
|
||||||
|
// preparation is fully completed before the first apply.
|
||||||
|
const home = await fleetHome();
|
||||||
|
const coder1Path = join(home, 'fleet', 'agents', 'coder1.env.generated');
|
||||||
|
await writeFile(coder1Path, 'MOSAIC_AGENT_NAME=coder1\n', { mode: 0o644 });
|
||||||
|
const calls: string[][] = [];
|
||||||
|
capture();
|
||||||
|
const errors: string[] = [];
|
||||||
|
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||||
|
errors.push(String(chunk));
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
|
||||||
|
await program(home, recordingRunner(calls)).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
'--json',
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(process.exitCode).toBe(1);
|
||||||
|
expect(errors.join('')).toMatch(/regen failed/i);
|
||||||
|
// coder0 is valid but must remain unwritten — no partial rebuild.
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('fails closed on a semantically invalid roster (protected-class tool-policy mismatch)', async (): Promise<void> => {
|
||||||
|
// A hand-edited roster giving a protected class a weaker tool_policy is rejected
|
||||||
|
// by reconcile/plan/verify; regen must enforce the SAME gate, not silently
|
||||||
|
// project the downgraded policy into <name>.env.generated.
|
||||||
|
const home = await fleetHome(`
|
||||||
|
version: 2
|
||||||
|
generation: 3
|
||||||
|
transport: tmux
|
||||||
|
tmux:
|
||||||
|
socket_name: mosaic-fleet
|
||||||
|
holder_session: _holder
|
||||||
|
defaults:
|
||||||
|
working_directory: /srv/mosaic
|
||||||
|
runtime: pi
|
||||||
|
runtimes:
|
||||||
|
pi:
|
||||||
|
reset_command: /new
|
||||||
|
agents:
|
||||||
|
- name: gate0
|
||||||
|
alias: Gate 0
|
||||||
|
class: merge-gate
|
||||||
|
runtime: pi
|
||||||
|
provider: openai
|
||||||
|
model: gpt-5.6-sol
|
||||||
|
reasoning: high
|
||||||
|
tool_policy: code
|
||||||
|
working_directory: /srv/mosaic
|
||||||
|
persistent_persona: false
|
||||||
|
reset_between_tasks: true
|
||||||
|
lifecycle:
|
||||||
|
enabled: true
|
||||||
|
desired_state: stopped
|
||||||
|
launch:
|
||||||
|
yolo: true
|
||||||
|
`);
|
||||||
|
const calls: string[][] = [];
|
||||||
|
capture();
|
||||||
|
const errors: string[] = [];
|
||||||
|
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||||
|
errors.push(String(chunk));
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
|
||||||
|
await program(home, recordingRunner(calls)).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
'--json',
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(process.exitCode).toBe(1);
|
||||||
|
expect(errors.join('')).toMatch(/regen failed/i);
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'gate0.env.generated'))).toBe(false);
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('refuses to write while a concurrent reconcile holds the mutation lock', async (): Promise<void> => {
|
||||||
|
// regen --write mutates the same projections as reconcile; it must take the
|
||||||
|
// shared reconcile lock so a concurrent reconcile cannot race a stale write.
|
||||||
|
const home = await fleetHome();
|
||||||
|
await writeFile(join(home, 'fleet', 'roster.yaml.reconcile.lock'), 'held\n', { mode: 0o600 });
|
||||||
|
const calls: string[][] = [];
|
||||||
|
capture();
|
||||||
|
const errors: string[] = [];
|
||||||
|
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||||
|
errors.push(String(chunk));
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
|
||||||
|
await program(home, recordingRunner(calls)).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
'--json',
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(process.exitCode).toBe(1);
|
||||||
|
expect(errors.join('')).toMatch(/regen failed/i);
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('fails closed (non-zero, no writes) on an invalid roster', async (): Promise<void> => {
|
||||||
|
// roster-v2 requires >= 1 agent; a malformed roster must abort regen without
|
||||||
|
// writing any projection and without touching lifecycle.
|
||||||
|
const home = await fleetHome('version: 2\ngeneration: 1\n');
|
||||||
|
const calls: string[][] = [];
|
||||||
|
capture();
|
||||||
|
const errors: string[] = [];
|
||||||
|
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||||
|
errors.push(String(chunk));
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
|
||||||
|
await program(home, recordingRunner(calls)).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
'--json',
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(process.exitCode).toBe(1);
|
||||||
|
expect(errors.join('')).toMatch(/regen failed/i);
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('rejects a non-canonical --roster path (canonical roster only)', async (): Promise<void> => {
|
||||||
|
// `fleet` exposes a global `--roster`; reconcile rejects a non-canonical value
|
||||||
|
// rather than silently reading the canonical roster. regen must enforce the
|
||||||
|
// SAME guard so `--roster /elsewhere regen --write` cannot mislead the operator.
|
||||||
|
const home = await fleetHome();
|
||||||
|
const calls: string[][] = [];
|
||||||
|
capture();
|
||||||
|
const errors: string[] = [];
|
||||||
|
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||||
|
errors.push(String(chunk));
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
|
||||||
|
await program(home, recordingRunner(calls)).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'--roster',
|
||||||
|
join(home, 'other', 'roster.yaml'),
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
'--json',
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(process.exitCode).toBe(1);
|
||||||
|
expect(errors.join('')).toMatch(/regen failed/i);
|
||||||
|
expect(errors.join('')).toMatch(/canonical roster/i);
|
||||||
|
// No projection written against a misdirected roster path.
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reports an incomplete rebuild when a later projection write fails mid-loop', async (): Promise<void> => {
|
||||||
|
// Fault injected on the SECOND apply: coder0 is already replaced on disk, so a
|
||||||
|
// bare throw would hide the partial state. regen must surface which agents were
|
||||||
|
// rebuilt and which failed, with a verify-before-restart recovery instruction.
|
||||||
|
const home = await fleetHome();
|
||||||
|
const realApply = applyPreparedGeneratedAgentEnvironmentProjection;
|
||||||
|
const result = await executeFleetRegen(
|
||||||
|
{
|
||||||
|
runner: recordingRunner([]),
|
||||||
|
mosaicHome: home,
|
||||||
|
applyProjection: async (prepared): Promise<string> => {
|
||||||
|
if (prepared.generatedPath.endsWith('coder1.env.generated')) {
|
||||||
|
throw new Error('simulated ENOSPC on coder1');
|
||||||
|
}
|
||||||
|
return realApply(prepared);
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{ write: true },
|
||||||
|
);
|
||||||
|
|
||||||
|
// coder0 written before the fault; coder1 not.
|
||||||
|
expect(result.written).toBe(1);
|
||||||
|
expect(result.incomplete).toMatchObject({
|
||||||
|
code: 'projection-apply-failed',
|
||||||
|
failedAgent: 'coder1',
|
||||||
|
writtenAgents: ['coder0'],
|
||||||
|
});
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
|
||||||
|
|
||||||
|
// The human report names the failed agent and directs a verify-before-restart
|
||||||
|
// recovery — not a bare success runbook.
|
||||||
|
const report = formatFleetRegenReport(result).join('\n');
|
||||||
|
expect(report).toMatch(/incomplete/i);
|
||||||
|
expect(report).toContain('coder1');
|
||||||
|
expect(report).toMatch(/verify|re-run|retry/i);
|
||||||
|
// Structural: still no lifecycle call anywhere on the failure path.
|
||||||
|
expect(report).not.toMatch(/systemctl --user restart mosaic-agent@<name>\n.*\n/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('refuses to write while agent CRUD holds the roster mutation lock', async (): Promise<void> => {
|
||||||
|
// regen --write overwrites the SAME generated projections that `fleet agent
|
||||||
|
// create/update/delete` writes under roster.yaml.mutation.lock. If regen only
|
||||||
|
// took the reconcile lock it could read a stale roster and overwrite a just-
|
||||||
|
// committed projection. It must also take the mutation lock and fail closed.
|
||||||
|
const home = await fleetHome();
|
||||||
|
await writeFile(join(home, 'fleet', 'roster.yaml.mutation.lock'), 'crud\n', { mode: 0o600 });
|
||||||
|
const calls: string[][] = [];
|
||||||
|
capture();
|
||||||
|
const errors: string[] = [];
|
||||||
|
vi.spyOn(process.stderr, 'write').mockImplementation((chunk: string | Uint8Array): boolean => {
|
||||||
|
errors.push(String(chunk));
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
|
||||||
|
await program(home, recordingRunner(calls)).parseAsync([
|
||||||
|
'node',
|
||||||
|
'mosaic',
|
||||||
|
'fleet',
|
||||||
|
'regen',
|
||||||
|
'--write',
|
||||||
|
'--json',
|
||||||
|
]);
|
||||||
|
|
||||||
|
expect(process.exitCode).toBe(1);
|
||||||
|
expect(errors.join('')).toMatch(/regen failed/i);
|
||||||
|
// No projection written against a roster a concurrent CRUD is mutating.
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(false);
|
||||||
|
expect(calls).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('surfaces BOTH the roster failure and a stale-lock warning on a double fault', async (): Promise<void> => {
|
||||||
|
// Worst case with nothing written: run() fails (invalid roster, fail-closed) AND
|
||||||
|
// a lock release faults. The operator must be told the lock may be stale, not
|
||||||
|
// just the roster error — otherwise the next regen/reconcile is silently blocked.
|
||||||
|
const home = await fleetHome('version: 2\ngeneration: 1\n');
|
||||||
|
const throwingRelease =
|
||||||
|
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||||
|
throw new Error('simulated lock unlink fault');
|
||||||
|
};
|
||||||
|
await expect(
|
||||||
|
executeFleetRegen(
|
||||||
|
{
|
||||||
|
runner: recordingRunner([]),
|
||||||
|
mosaicHome: home,
|
||||||
|
acquireReconcileLock: throwingRelease,
|
||||||
|
acquireRosterMutationLock: throwingRelease,
|
||||||
|
},
|
||||||
|
{ write: true },
|
||||||
|
),
|
||||||
|
).rejects.toThrow(/stale|inspect|lock/i);
|
||||||
|
// Nothing written on the fail-closed path.
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('propagates a roster-mutation-lock release failure instead of silently keeping a stale lock', async (): Promise<void> => {
|
||||||
|
// Finding L: the real `acquirePrivateRosterMutationLock` release must SURFACE an
|
||||||
|
// unlink failure, not swallow it. If it swallowed (like the CRUD-internal
|
||||||
|
// acquireMutationLock does), a stale roster.yaml.mutation.lock left after a
|
||||||
|
// successful regen would block later regen/agent CRUD while the command reported
|
||||||
|
// success — and the finding-J stale-lock warning would never fire for this lock.
|
||||||
|
const home = await fleetHome();
|
||||||
|
const release = await acquirePrivateRosterMutationLock(home)();
|
||||||
|
// Simulate the lock vanishing (or being unremovable) before release runs: the
|
||||||
|
// underlying unlink then faults. A swallowing release would resolve and hide it.
|
||||||
|
await rm(join(home, 'fleet', 'roster.yaml.mutation.lock'));
|
||||||
|
await expect(release()).rejects.toThrow();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('preserves a complete rebuild result and flags lock-cleanup when release faults', async (): Promise<void> => {
|
||||||
|
// A lock-release fault after a successful write must NOT discard the fact that
|
||||||
|
// projections are now live — the operator needs to know the write happened and
|
||||||
|
// that the shared lock may be stale, not just a bare "regen failed".
|
||||||
|
const home = await fleetHome();
|
||||||
|
const result = await executeFleetRegen(
|
||||||
|
{
|
||||||
|
runner: recordingRunner([]),
|
||||||
|
mosaicHome: home,
|
||||||
|
acquireReconcileLock:
|
||||||
|
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||||
|
throw new Error('simulated lock unlink fault');
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{ write: true },
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result.written).toBe(2);
|
||||||
|
expect(result.incomplete).toBeUndefined();
|
||||||
|
expect(result.cleanup).toMatchObject({
|
||||||
|
code: 'lock-cleanup-failed',
|
||||||
|
action: 'inspect-lock-before-retry',
|
||||||
|
});
|
||||||
|
// Projections are genuinely on disk despite the release fault.
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(true);
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder1.env.generated'))).toBe(true);
|
||||||
|
|
||||||
|
const report = formatFleetRegenReport(result).join('\n');
|
||||||
|
expect(report).toMatch(/lock/i);
|
||||||
|
expect(report).toMatch(/stale|inspect|clear/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('preserves a partial rebuild AND flags lock-cleanup when both apply and release fault', async (): Promise<void> => {
|
||||||
|
// Worst case: a mid-loop apply fault leaves a partial rebuild, then the release
|
||||||
|
// also faults. Both recovery states must survive so the operator sees exactly
|
||||||
|
// which projections are live and that the lock may be stale.
|
||||||
|
const home = await fleetHome();
|
||||||
|
const realApply = applyPreparedGeneratedAgentEnvironmentProjection;
|
||||||
|
const result = await executeFleetRegen(
|
||||||
|
{
|
||||||
|
runner: recordingRunner([]),
|
||||||
|
mosaicHome: home,
|
||||||
|
applyProjection: async (prepared): Promise<string> => {
|
||||||
|
if (prepared.generatedPath.endsWith('coder1.env.generated')) {
|
||||||
|
throw new Error('simulated ENOSPC on coder1');
|
||||||
|
}
|
||||||
|
return realApply(prepared);
|
||||||
|
},
|
||||||
|
acquireReconcileLock:
|
||||||
|
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||||
|
throw new Error('simulated lock unlink fault');
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{ write: true },
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result.written).toBe(1);
|
||||||
|
expect(result.incomplete).toMatchObject({
|
||||||
|
code: 'projection-apply-failed',
|
||||||
|
failedAgent: 'coder1',
|
||||||
|
writtenAgents: ['coder0'],
|
||||||
|
});
|
||||||
|
expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' });
|
||||||
|
|
||||||
|
const report = formatFleetRegenReport(result).join('\n');
|
||||||
|
expect(report).toMatch(/incomplete/i);
|
||||||
|
expect(report).toMatch(/lock/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('fails closed without deleting a REPLACEMENT roster-mutation lock it no longer owns', async (): Promise<void> => {
|
||||||
|
// Finding M1 (replacement-lock race): after this invocation created its lock, the
|
||||||
|
// lock is cleared and re-created by ANOTHER writer (new inode + foreign token)
|
||||||
|
// BEFORE release runs. The release must PROVE ownership (device/inode + token) and
|
||||||
|
// refuse to unlink the stranger's live lock — otherwise it deletes it, a third
|
||||||
|
// writer enters concurrently, and mutual exclusion over the projections is broken.
|
||||||
|
// This pins the REACHABLE window (replaced-before-release); the residual sub-window
|
||||||
|
// between the final check and the path `unlink` is unreachable within the `wx`
|
||||||
|
// writer protocol and matches the merged reconcile lock — see the acquirer doc.
|
||||||
|
const home = await fleetHome();
|
||||||
|
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
|
||||||
|
const release = await acquirePrivateRosterMutationLock(home)();
|
||||||
|
// A different writer replaces the lock: same path, NEW inode, foreign content.
|
||||||
|
await rm(lockPath);
|
||||||
|
await writeFile(lockPath, 'a-different-writer\n', { mode: 0o600 });
|
||||||
|
const replacement = await stat(lockPath);
|
||||||
|
// Release must reject (it no longer owns this file) AND the diagnosis must name
|
||||||
|
// the MUTATION lock specifically — a fault on this shared helper must not be
|
||||||
|
// mislabeled as the reconcile lock (that would defeat the M3 accurate-diagnosis
|
||||||
|
// goal, since regen serializes on both) ...
|
||||||
|
await expect(release()).rejects.toThrow(/roster\.yaml\.mutation\.lock/);
|
||||||
|
// ... and MUST NOT have unlinked the stranger's live lock.
|
||||||
|
expect(await exists(lockPath)).toBe(true);
|
||||||
|
expect((await stat(lockPath)).ino).toBe(replacement.ino);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('does not strand the lock file when initialization fails after creating it', async (): Promise<void> => {
|
||||||
|
// Codex r4: if writeFile/stat/close/ownership-check throws AFTER the `wx` create
|
||||||
|
// succeeds, the just-created lock must NOT be left behind — a stranded lock would
|
||||||
|
// permanently block future regen AND agent CRUD (both contend on this path). The
|
||||||
|
// cleanup is dev/ino-guarded (never deletes a replacement) and best-effort (never
|
||||||
|
// masks the initialization error).
|
||||||
|
const home = await fleetHome();
|
||||||
|
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
|
||||||
|
// Inject an opener that performs the REAL `wx` create (so the lock file really
|
||||||
|
// lands on disk) but whose handle faults on the token write — a post-create
|
||||||
|
// initialization failure.
|
||||||
|
const faultingOpen = (async (
|
||||||
|
path: Parameters<typeof open>[0],
|
||||||
|
flags: Parameters<typeof open>[1],
|
||||||
|
mode: Parameters<typeof open>[2],
|
||||||
|
) => {
|
||||||
|
const handle = await open(path, flags, mode);
|
||||||
|
return new Proxy(handle, {
|
||||||
|
get(target, prop, receiver): unknown {
|
||||||
|
if (prop === 'writeFile') {
|
||||||
|
return async (): Promise<never> => {
|
||||||
|
throw new Error('simulated ENOSPC on token write');
|
||||||
|
};
|
||||||
|
}
|
||||||
|
const value = Reflect.get(target, prop, receiver);
|
||||||
|
return typeof value === 'function' ? value.bind(target) : value;
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}) as typeof open;
|
||||||
|
|
||||||
|
await expect(acquirePrivateRosterMutationLock(home, faultingOpen)()).rejects.toThrow(
|
||||||
|
/mutation\.lock/,
|
||||||
|
);
|
||||||
|
// The lock we created must have been cleaned up — not stranded on disk.
|
||||||
|
expect(await exists(lockPath)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('does not strand the lock file when the post-create stat itself fails', async (): Promise<void> => {
|
||||||
|
// Codex r5: the narrower sub-path where `handle.stat()` ITSELF rejects right after
|
||||||
|
// the `wx` create succeeds (transient EIO/EBADF). device/inode identity is then
|
||||||
|
// unavailable, so the init-failure cleanup cannot prove ownership by dev/ino — yet
|
||||||
|
// the lock must STILL not be stranded, because a stranded lock permanently blocks
|
||||||
|
// future regen AND agent CRUD. The token we persisted before the failure uniquely
|
||||||
|
// identifies OUR lock, so cleanup falls back to matching it.
|
||||||
|
const home = await fleetHome();
|
||||||
|
const lockPath = join(home, 'fleet', 'roster.yaml.mutation.lock');
|
||||||
|
// Real `wx` create (lock really lands on disk); the token write SUCCEEDS so our
|
||||||
|
// random token is persisted, then `stat()` faults — a post-create init failure.
|
||||||
|
const faultingStatOpen = (async (
|
||||||
|
path: Parameters<typeof open>[0],
|
||||||
|
flags: Parameters<typeof open>[1],
|
||||||
|
mode: Parameters<typeof open>[2],
|
||||||
|
) => {
|
||||||
|
const handle = await open(path, flags, mode);
|
||||||
|
return new Proxy(handle, {
|
||||||
|
get(target, prop, receiver): unknown {
|
||||||
|
if (prop === 'stat') {
|
||||||
|
return async (): Promise<never> => {
|
||||||
|
throw new Error('simulated EIO on post-create stat');
|
||||||
|
};
|
||||||
|
}
|
||||||
|
const value = Reflect.get(target, prop, receiver);
|
||||||
|
return typeof value === 'function' ? value.bind(target) : value;
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}) as typeof open;
|
||||||
|
|
||||||
|
await expect(acquirePrivateRosterMutationLock(home, faultingStatOpen)()).rejects.toThrow(
|
||||||
|
/mutation\.lock/,
|
||||||
|
);
|
||||||
|
// Even without dev/ino, the token-based fallback must have removed our lock.
|
||||||
|
expect(await exists(lockPath)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('surfaces a lock-cleanup fault when unwinding after a later lock acquire fails', async (): Promise<void> => {
|
||||||
|
// Finding M2: the FIRST lock is taken, the SECOND lock's acquire throws, and
|
||||||
|
// unwinding the first lock's release ALSO faults. The acquire-unwind path must
|
||||||
|
// surface both — the already-taken lock may now be stale and silently block the
|
||||||
|
// next regen/reconcile — not just re-throw the acquire error and drop the fault.
|
||||||
|
const home = await fleetHome();
|
||||||
|
await expect(
|
||||||
|
executeFleetRegen(
|
||||||
|
{
|
||||||
|
runner: recordingRunner([]),
|
||||||
|
mosaicHome: home,
|
||||||
|
// Mutation lock (acquired first) succeeds, but its RELEASE faults on unwind.
|
||||||
|
acquireRosterMutationLock:
|
||||||
|
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||||
|
throw new Error('simulated mutation-lock unlink fault');
|
||||||
|
},
|
||||||
|
// Reconcile lock (acquired second) ACQUIRE fails, triggering the unwind.
|
||||||
|
acquireReconcileLock: () => async (): Promise<() => Promise<void>> => {
|
||||||
|
throw new Error('simulated reconcile acquire failure');
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{ write: true },
|
||||||
|
),
|
||||||
|
).rejects.toThrow(/stale|inspect|lock/i);
|
||||||
|
expect(await exists(join(home, 'fleet', 'agents', 'coder0.env.generated'))).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('names BOTH fleet locks in the stale-lock warning (cleanup can come from either)', async (): Promise<void> => {
|
||||||
|
// Finding M3: finding L made the roster MUTATION lock's release fault reachable, so
|
||||||
|
// the cleanup marker can now originate from either lock. The human report must not
|
||||||
|
// claim only the reconcile lock is stale — it must name both candidate lock files
|
||||||
|
// so the operator inspects the right one.
|
||||||
|
const home = await fleetHome();
|
||||||
|
const result = await executeFleetRegen(
|
||||||
|
{
|
||||||
|
runner: recordingRunner([]),
|
||||||
|
mosaicHome: home,
|
||||||
|
// Fault the MUTATION lock's release specifically (reconcile lock is real).
|
||||||
|
acquireRosterMutationLock:
|
||||||
|
() => async (): Promise<() => Promise<void>> => async (): Promise<void> => {
|
||||||
|
throw new Error('simulated mutation-lock unlink fault');
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{ write: true },
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(result.written).toBe(2);
|
||||||
|
expect(result.cleanup).toMatchObject({ code: 'lock-cleanup-failed' });
|
||||||
|
const report = formatFleetRegenReport(result).join('\n');
|
||||||
|
expect(report).toContain('roster.yaml.mutation.lock');
|
||||||
|
expect(report).toContain('roster.yaml.reconcile.lock');
|
||||||
|
});
|
||||||
|
});
|
||||||
441
packages/mosaic/src/commands/fleet-regen-command.ts
Normal file
441
packages/mosaic/src/commands/fleet-regen-command.ts
Normal file
@@ -0,0 +1,441 @@
|
|||||||
|
import { readFile, stat } from 'node:fs/promises';
|
||||||
|
import { homedir } from 'node:os';
|
||||||
|
import { join, relative, resolve } from 'node:path';
|
||||||
|
import type { Command } from 'commander';
|
||||||
|
import type { CommandRunner } from './fleet.js';
|
||||||
|
import {
|
||||||
|
applyPreparedGeneratedAgentEnvironmentProjection,
|
||||||
|
prepareGeneratedAgentEnvironmentProjection,
|
||||||
|
type PreparedGeneratedAgentEnvironmentProjection,
|
||||||
|
} from '../fleet/generated-env-boundary.js';
|
||||||
|
import {
|
||||||
|
acquirePrivateReconcileLock,
|
||||||
|
acquirePrivateRosterMutationLock,
|
||||||
|
projectRosterV2AgentGeneratedEnv,
|
||||||
|
} from '../fleet/fleet-reconciler.js';
|
||||||
|
import {
|
||||||
|
parseRosterV2,
|
||||||
|
validateRosterV2Semantics,
|
||||||
|
type FleetRosterV2,
|
||||||
|
} from '../fleet/roster-v2.js';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `mosaic fleet regen` — recovery-framed regeneration of the roster-derived
|
||||||
|
* agent env projections (`fleet/agents/<name>.env.generated`) from the roster
|
||||||
|
* SSOT. It is the recovery layer of #791: when a wiped/diverged operator surface
|
||||||
|
* has cost an agent its generated projection, regen rebuilds it deterministically
|
||||||
|
* from `roster.yaml` so the launcher can source the intended identity again.
|
||||||
|
*
|
||||||
|
* It is intentionally a THIN projection-only wrapper over the same mapping the
|
||||||
|
* reconciler apply path uses ({@link projectRosterV2AgentGeneratedEnv}), and it
|
||||||
|
* has NO code path to systemd lifecycle: regen never starts, stops, or restarts
|
||||||
|
* an agent. Recovery order forbids restart-before-verify, so the operator must
|
||||||
|
* verify each rebuilt projection resolves before restarting anything.
|
||||||
|
*/
|
||||||
|
export interface FleetRegenAgentPlan {
|
||||||
|
readonly name: string;
|
||||||
|
/** Generated-projection path, relative to the Mosaic home (never absolute in output). */
|
||||||
|
readonly path: string;
|
||||||
|
/** `create` when the projection is currently absent, `rebuild` when it already exists. */
|
||||||
|
readonly disposition: 'create' | 'rebuild';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Present ONLY when a `--write` projection apply failed partway through the
|
||||||
|
* fleet. Every prior agent was already validated and prepared, so `writtenAgents`
|
||||||
|
* are byte-complete on disk; `failedAgent` is where the write faulted. The
|
||||||
|
* command surfaces this instead of a bare throw so the operator can finish the
|
||||||
|
* rebuild and verify every projection before restarting any unit.
|
||||||
|
*/
|
||||||
|
export interface FleetRegenIncomplete {
|
||||||
|
readonly code: 'projection-apply-failed';
|
||||||
|
/** The agent whose generated-projection write faulted. */
|
||||||
|
readonly failedAgent: string;
|
||||||
|
/** Agents whose projections were fully written before the fault (in apply order). */
|
||||||
|
readonly writtenAgents: readonly string[];
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FleetRegenResult {
|
||||||
|
readonly mode: 'dry-run' | 'write';
|
||||||
|
/** Roster path, relative to the Mosaic home. */
|
||||||
|
readonly rosterPath: string;
|
||||||
|
readonly generation: number;
|
||||||
|
readonly agentCount: number;
|
||||||
|
/** Count of projections written to disk (always 0 in dry-run). */
|
||||||
|
readonly written: number;
|
||||||
|
readonly agents: readonly FleetRegenAgentPlan[];
|
||||||
|
/** Set ONLY when a `--write` apply faulted mid-fleet — a partial rebuild. */
|
||||||
|
readonly incomplete?: FleetRegenIncomplete;
|
||||||
|
/**
|
||||||
|
* Set ONLY when a shared fleet lock could not be released after a write completed
|
||||||
|
* (or partially completed) — EITHER `roster.yaml.mutation.lock` or
|
||||||
|
* `roster.yaml.reconcile.lock`, since regen holds both. The projections in this
|
||||||
|
* result are real; the lock file may be stale and must be inspected before the
|
||||||
|
* next mutation. Independent of {@link incomplete} — both can be present at once.
|
||||||
|
*/
|
||||||
|
readonly cleanup?: {
|
||||||
|
readonly code: 'lock-cleanup-failed';
|
||||||
|
readonly action: 'inspect-lock-before-retry';
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface FleetRegenDeps {
|
||||||
|
/**
|
||||||
|
* Present ONLY so an accidental future lifecycle wiring is caught by the
|
||||||
|
* "never restarts" test — regen is contractually forbidden to invoke it.
|
||||||
|
*/
|
||||||
|
readonly runner: CommandRunner;
|
||||||
|
readonly mosaicHome?: string;
|
||||||
|
/** Persona roots for semantic roster validation (defaults mirror the reconciler). */
|
||||||
|
readonly rolesDir?: string;
|
||||||
|
readonly overrideDir?: string;
|
||||||
|
/** Test seam: canonical roster reader. Defaults to parse + semantic validation. */
|
||||||
|
readonly readRoster?: (rosterPath: string) => Promise<FleetRosterV2>;
|
||||||
|
/**
|
||||||
|
* Test seam: generated-only projection validator. Defaults to the audited
|
||||||
|
* boundary helper that validates ONLY `<name>.env.generated`.
|
||||||
|
*/
|
||||||
|
readonly prepareProjection?: typeof prepareGeneratedAgentEnvironmentProjection;
|
||||||
|
/** Test seam: generated-only projection writer. Defaults to the audited boundary helper. */
|
||||||
|
readonly applyProjection?: typeof applyPreparedGeneratedAgentEnvironmentProjection;
|
||||||
|
/**
|
||||||
|
* Test seam: acquire the shared reconcile lock before a `--write`. Defaults to
|
||||||
|
* the reconciler's private lock so regen and reconcile are mutually exclusive
|
||||||
|
* and a concurrent reconcile cannot race a stale projection write.
|
||||||
|
*/
|
||||||
|
readonly acquireReconcileLock?: (mosaicHome: string) => () => Promise<() => Promise<void>>;
|
||||||
|
/**
|
||||||
|
* Test seam: acquire the shared roster mutation lock before a `--write`.
|
||||||
|
* Defaults to the reconciler's hardened, ownership-proving acquirer for the
|
||||||
|
* same `fleet/roster.yaml.mutation.lock` path that CRUD contends on, so regen
|
||||||
|
* serializes against agent create/update/delete — both rewrite the same
|
||||||
|
* generated projections, so without this a concurrent CRUD could race a stale
|
||||||
|
* projection write.
|
||||||
|
*/
|
||||||
|
readonly acquireRosterMutationLock?: (mosaicHome: string) => () => Promise<() => Promise<void>>;
|
||||||
|
/** Test seam: existence probe for create/rebuild disposition. */
|
||||||
|
readonly fileExists?: (path: string) => Promise<boolean>;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface FleetRegenOptions {
|
||||||
|
readonly write?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
function defaultMosaicHome(deps: FleetRegenDeps): string {
|
||||||
|
return deps.mosaicHome ?? process.env['MOSAIC_HOME'] ?? join(homedir(), '.config', 'mosaic');
|
||||||
|
}
|
||||||
|
|
||||||
|
async function defaultFileExists(path: string): Promise<boolean> {
|
||||||
|
try {
|
||||||
|
await stat(path);
|
||||||
|
return true;
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Projection-only regeneration core. Deterministic and side-effect-free in
|
||||||
|
* dry-run; in `--write` it rebuilds each generated projection through the
|
||||||
|
* audited generated-only boundary writer and NOTHING else — it never writes
|
||||||
|
* `.env.local`/`.env.quarantine`, never unlinks the legacy `.env`, and never
|
||||||
|
* touches `deps.runner`.
|
||||||
|
*
|
||||||
|
* Fail-closed by construction: the roster is validated (structural + semantic)
|
||||||
|
* and EVERY agent's projection is prepared before ANY is written, so a single
|
||||||
|
* invalid agent (or a semantically rejected roster) leaves the whole surface
|
||||||
|
* untouched. In `--write` mode the read-prepare-apply sequence runs under the
|
||||||
|
* shared reconcile lock so a concurrent reconcile cannot race a stale write.
|
||||||
|
*/
|
||||||
|
export async function executeFleetRegen(
|
||||||
|
deps: FleetRegenDeps,
|
||||||
|
options: FleetRegenOptions,
|
||||||
|
): Promise<FleetRegenResult> {
|
||||||
|
const mosaicHome = defaultMosaicHome(deps);
|
||||||
|
const agentEnvDir = join(mosaicHome, 'fleet', 'agents');
|
||||||
|
const rosterPath = join(mosaicHome, 'fleet', 'roster.yaml');
|
||||||
|
const readRoster = deps.readRoster ?? defaultReadRoster(deps, mosaicHome);
|
||||||
|
const prepare = deps.prepareProjection ?? prepareGeneratedAgentEnvironmentProjection;
|
||||||
|
const apply = deps.applyProjection ?? applyPreparedGeneratedAgentEnvironmentProjection;
|
||||||
|
const fileExists = deps.fileExists ?? defaultFileExists;
|
||||||
|
const acquireReconcileLock = deps.acquireReconcileLock ?? acquirePrivateReconcileLock;
|
||||||
|
const acquireRosterMutationLock =
|
||||||
|
deps.acquireRosterMutationLock ?? acquirePrivateRosterMutationLock;
|
||||||
|
const write = options.write === true;
|
||||||
|
|
||||||
|
const run = async (): Promise<FleetRegenResult> => {
|
||||||
|
const roster = await readRoster(rosterPath);
|
||||||
|
|
||||||
|
// Prepare (validate) every agent BEFORE any write, so a later failure never
|
||||||
|
// leaves an earlier projection partially rebuilt.
|
||||||
|
const prepared: (PreparedGeneratedAgentEnvironmentProjection & {
|
||||||
|
readonly name: string;
|
||||||
|
readonly disposition: 'create' | 'rebuild';
|
||||||
|
})[] = [];
|
||||||
|
for (const agent of roster.agents) {
|
||||||
|
const projection = await prepare({
|
||||||
|
mosaicHome,
|
||||||
|
agentEnvDir,
|
||||||
|
agentName: agent.name,
|
||||||
|
generated: projectRosterV2AgentGeneratedEnv(roster, agent),
|
||||||
|
});
|
||||||
|
const disposition = (await fileExists(projection.generatedPath)) ? 'rebuild' : 'create';
|
||||||
|
prepared.push({ ...projection, name: agent.name, disposition });
|
||||||
|
}
|
||||||
|
|
||||||
|
// Every projection is already validated (prepared); an apply here can only
|
||||||
|
// fault on the underlying I/O (ENOSPC, EIO, a race outside the lock). If a
|
||||||
|
// later write faults, earlier agents are already replaced on disk, so we
|
||||||
|
// stop and report the partial state rather than throwing a bare error that
|
||||||
|
// hides which projections are now live.
|
||||||
|
let written = 0;
|
||||||
|
let incomplete: FleetRegenIncomplete | undefined;
|
||||||
|
if (write) {
|
||||||
|
for (const projection of prepared) {
|
||||||
|
try {
|
||||||
|
await apply(projection);
|
||||||
|
} catch {
|
||||||
|
incomplete = {
|
||||||
|
code: 'projection-apply-failed',
|
||||||
|
failedAgent: projection.name,
|
||||||
|
writtenAgents: prepared.slice(0, written).map((entry) => entry.name),
|
||||||
|
};
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
written += 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
mode: write ? 'write' : 'dry-run',
|
||||||
|
rosterPath: relative(mosaicHome, rosterPath),
|
||||||
|
generation: roster.generation,
|
||||||
|
agentCount: roster.agents.length,
|
||||||
|
written,
|
||||||
|
agents: prepared.map((projection) => ({
|
||||||
|
name: projection.name,
|
||||||
|
path: relative(mosaicHome, projection.generatedPath),
|
||||||
|
disposition: projection.disposition,
|
||||||
|
})),
|
||||||
|
...(incomplete ? { incomplete } : {}),
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
|
// Dry-run is preview-only and never mutates, so it stays lock-free (usable even
|
||||||
|
// while a reconcile or CRUD holds a lock). A `--write` must serialize against
|
||||||
|
// BOTH roster writers of the same generated projections: agent CRUD (roster
|
||||||
|
// mutation lock) and reconcile (reconcile lock). Acquire in a fixed order — the
|
||||||
|
// locks are non-blocking (they throw on contention rather than wait), so no
|
||||||
|
// deadlock is possible — and fail closed if either is already held.
|
||||||
|
if (!write) return run();
|
||||||
|
|
||||||
|
const releases: (() => Promise<void>)[] = [];
|
||||||
|
try {
|
||||||
|
releases.push(await acquireRosterMutationLock(mosaicHome)());
|
||||||
|
releases.push(await acquireReconcileLock(mosaicHome)());
|
||||||
|
} catch (error: unknown) {
|
||||||
|
// A later acquire failed; unwind any lock already taken (reverse order). If
|
||||||
|
// that unwind ALSO faults, surface both — the already-taken lock may now be
|
||||||
|
// stale and silently block the next regen/reconcile — rather than dropping it.
|
||||||
|
const releaseFault = await releaseFleetLocks(releases);
|
||||||
|
throw augmentWithLockCleanupFault(error, releaseFault);
|
||||||
|
}
|
||||||
|
|
||||||
|
let result: FleetRegenResult | undefined;
|
||||||
|
let primaryError: unknown;
|
||||||
|
try {
|
||||||
|
result = await run();
|
||||||
|
} catch (error: unknown) {
|
||||||
|
primaryError = error;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Release both locks (reverse acquire order), continuing past a fault so neither
|
||||||
|
// leaks. A release fault must never discard a completed (or partial) rebuild —
|
||||||
|
// the projections are already on disk — and must not be silently hidden even
|
||||||
|
// when the run itself failed with nothing written.
|
||||||
|
const releaseFault = await releaseFleetLocks(releases);
|
||||||
|
if (result) {
|
||||||
|
return releaseFault === undefined
|
||||||
|
? result
|
||||||
|
: {
|
||||||
|
...result,
|
||||||
|
cleanup: { code: 'lock-cleanup-failed', action: 'inspect-lock-before-retry' },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
// result is undefined ⇒ run() threw ⇒ primaryError is set.
|
||||||
|
throw augmentWithLockCleanupFault(primaryError, releaseFault);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Releases held locks in reverse acquire order, continuing past a fault so a
|
||||||
|
* single failed release never leaks the others. Returns the first fault seen (or
|
||||||
|
* undefined when every release succeeded).
|
||||||
|
*/
|
||||||
|
async function releaseFleetLocks(releases: readonly (() => Promise<void>)[]): Promise<unknown> {
|
||||||
|
let firstFault: unknown;
|
||||||
|
for (const release of [...releases].reverse()) {
|
||||||
|
try {
|
||||||
|
await release();
|
||||||
|
} catch (error: unknown) {
|
||||||
|
if (firstFault === undefined) firstFault = error;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return firstFault;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* When a run failed with nothing written AND lock cleanup also faulted, surface
|
||||||
|
* both: the operator needs to know the lock may be stale (else the next
|
||||||
|
* regen/reconcile is silently blocked), not just the original run error.
|
||||||
|
*/
|
||||||
|
function augmentWithLockCleanupFault(primaryError: unknown, releaseFault: unknown): unknown {
|
||||||
|
if (releaseFault === undefined) return primaryError;
|
||||||
|
const base = primaryError instanceof Error ? primaryError.message : String(primaryError);
|
||||||
|
return new Error(
|
||||||
|
`${base} (additionally, a fleet lock could not be released — ` +
|
||||||
|
'fleet/roster.yaml.mutation.lock or roster.yaml.reconcile.lock may be stale; ' +
|
||||||
|
'inspect and clear it before retry)',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function defaultReadRoster(
|
||||||
|
deps: FleetRegenDeps,
|
||||||
|
mosaicHome: string,
|
||||||
|
): (rosterPath: string) => Promise<FleetRosterV2> {
|
||||||
|
return async (rosterPath: string): Promise<FleetRosterV2> => {
|
||||||
|
const roster = parseRosterV2(await readFile(rosterPath, 'utf8'), 'yaml');
|
||||||
|
// Enforce the SAME semantic gate as reconcile/plan/verify (persona resolution
|
||||||
|
// + protected-class tool-policy match) so regen cannot project a roster the
|
||||||
|
// rest of the fleet surface would reject.
|
||||||
|
await validateRosterV2Semantics(roster, {
|
||||||
|
rolesDir: deps.rolesDir ?? join(mosaicHome, 'fleet', 'roles'),
|
||||||
|
overrideDir: deps.overrideDir ?? join(mosaicHome, 'fleet', 'roles.local'),
|
||||||
|
});
|
||||||
|
return roster;
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Human-readable report — paths and counts ONLY. The rendered projection body
|
||||||
|
* (KEY=value lines) is never echoed (secrev). In `--write` mode it prints the
|
||||||
|
* recovery runbook: verify each projection resolves BEFORE any restart.
|
||||||
|
*/
|
||||||
|
export function formatFleetRegenReport(result: FleetRegenResult): string[] {
|
||||||
|
const lines: string[] = [];
|
||||||
|
const header =
|
||||||
|
result.mode === 'dry-run'
|
||||||
|
? 'mosaic fleet regen — dry run (no changes written)'
|
||||||
|
: 'mosaic fleet regen — wrote roster-derived projections';
|
||||||
|
lines.push(header);
|
||||||
|
lines.push(
|
||||||
|
`roster: ${result.rosterPath} (generation ${result.generation}, ${result.agentCount} agent(s))`,
|
||||||
|
);
|
||||||
|
for (const agent of result.agents) {
|
||||||
|
lines.push(` ${agent.name} ${agent.path} [${agent.disposition}]`);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (result.mode === 'dry-run') {
|
||||||
|
lines.push(
|
||||||
|
`Plan: would write ${result.agentCount} generated projection(s). ` +
|
||||||
|
`Re-run with --write to apply. regen never restarts agents.`,
|
||||||
|
);
|
||||||
|
return lines;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (result.incomplete) {
|
||||||
|
const { failedAgent, writtenAgents } = result.incomplete;
|
||||||
|
lines.push(
|
||||||
|
`INCOMPLETE: rebuild stopped at agent ${failedAgent} — its projection write faulted.`,
|
||||||
|
);
|
||||||
|
lines.push(
|
||||||
|
`Rebuilt before the fault: ${writtenAgents.length} projection(s)` +
|
||||||
|
(writtenAgents.length > 0 ? ` (${writtenAgents.join(', ')})` : '') +
|
||||||
|
`; ${result.agentCount - writtenAgents.length} not written.`,
|
||||||
|
);
|
||||||
|
lines.push(
|
||||||
|
'Recover: re-run `mosaic fleet regen --write` to finish, then VERIFY every ' +
|
||||||
|
"agent's fleet/agents/<name>.env.generated resolves BEFORE restarting any " +
|
||||||
|
'unit. regen never restarts agents.',
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
lines.push(`Wrote ${result.written} generated projection(s).`);
|
||||||
|
lines.push('Next steps — DO NOT restart agents before verifying:');
|
||||||
|
lines.push(
|
||||||
|
" 1. Confirm each agent's fleet/agents/<name>.env.generated exists and carries " +
|
||||||
|
'the intended MOSAIC_AGENT_* values.',
|
||||||
|
);
|
||||||
|
// The unit sets NO EnvironmentFile= — it launches from a minimal env and the
|
||||||
|
// launcher (start-agent-session.sh) sources .env.generated itself. Verify the
|
||||||
|
// launcher path, not a nonexistent EnvironmentFile property.
|
||||||
|
lines.push(
|
||||||
|
' 2. Confirm the unit launches from it: `systemctl --user cat mosaic-agent@<name>` ' +
|
||||||
|
'shows ExecStart running start-agent-session.sh, which reads .env.generated.',
|
||||||
|
);
|
||||||
|
lines.push(
|
||||||
|
' 3. Only then restart one unit at a time: systemctl --user restart mosaic-agent@<name>.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (result.cleanup) {
|
||||||
|
lines.push(
|
||||||
|
'WARNING: a fleet lock could not be released — ' +
|
||||||
|
'fleet/roster.yaml.mutation.lock or fleet/roster.yaml.reconcile.lock may be ' +
|
||||||
|
'stale; inspect and clear it before the next reconcile or regen.',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return lines;
|
||||||
|
}
|
||||||
|
|
||||||
|
function resolveRegenMosaicHome(fleetCommand: Command, deps: FleetRegenDeps): string {
|
||||||
|
const options = fleetCommand.optsWithGlobals<{ mosaicHome?: string }>();
|
||||||
|
return options.mosaicHome ?? defaultMosaicHome(deps);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* regen reads the roster SSOT at `<mosaicHome>/fleet/roster.yaml` and ignores the
|
||||||
|
* `fleet --roster <path>` global. If an operator passes a non-canonical `--roster`
|
||||||
|
* they must NOT be silently served the canonical file — mirror reconcile's guard
|
||||||
|
* and reject, so `mosaic fleet --roster /elsewhere regen --write` fails closed.
|
||||||
|
*/
|
||||||
|
function assertRegenCanonicalRoster(fleetCommand: Command, mosaicHome: string): void {
|
||||||
|
const options = fleetCommand.optsWithGlobals<{ roster?: string }>();
|
||||||
|
const canonical = join(mosaicHome, 'fleet', 'roster.yaml');
|
||||||
|
if (options.roster !== undefined && resolve(options.roster) !== resolve(canonical)) {
|
||||||
|
throw new Error(
|
||||||
|
'Roster-v2 regeneration requires the canonical roster path (fleet/roster.yaml).',
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Registers the recovery-framed `mosaic fleet regen` command. */
|
||||||
|
export function registerFleetRegenCommand(fleetCommand: Command, deps: FleetRegenDeps): void {
|
||||||
|
fleetCommand
|
||||||
|
.command('regen')
|
||||||
|
.description(
|
||||||
|
'Regenerate roster-derived agent env projections (dry-run default, --write to apply; never restarts)',
|
||||||
|
)
|
||||||
|
.option('--write', 'Write the regenerated projections to disk (default: dry-run)')
|
||||||
|
.option('--json', 'Emit the plan/result as JSON')
|
||||||
|
.action(async (opts: { write?: boolean; json?: boolean }): Promise<void> => {
|
||||||
|
const mosaicHome = resolveRegenMosaicHome(fleetCommand, deps);
|
||||||
|
try {
|
||||||
|
assertRegenCanonicalRoster(fleetCommand, mosaicHome);
|
||||||
|
const result = await executeFleetRegen(
|
||||||
|
{ ...deps, mosaicHome },
|
||||||
|
{ write: opts.write === true },
|
||||||
|
);
|
||||||
|
if (opts.json === true) {
|
||||||
|
console.log(JSON.stringify(result));
|
||||||
|
} else {
|
||||||
|
for (const line of formatFleetRegenReport(result)) console.log(line);
|
||||||
|
}
|
||||||
|
// A partial rebuild or a stale-lock cleanup state is a non-zero outcome:
|
||||||
|
// the operator must finish/verify (and clear the lock) before restarting.
|
||||||
|
if (result.incomplete || result.cleanup) process.exitCode = 1;
|
||||||
|
} catch (error: unknown) {
|
||||||
|
process.exitCode = 1;
|
||||||
|
const message = error instanceof Error ? error.message : String(error);
|
||||||
|
process.stderr.write(`mosaic fleet regen failed: ${message}\n`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
@@ -97,6 +97,7 @@ describe('registerFleetCommand', () => {
|
|||||||
'provision',
|
'provision',
|
||||||
'ps',
|
'ps',
|
||||||
'reconcile',
|
'reconcile',
|
||||||
|
'regen',
|
||||||
'remove',
|
'remove',
|
||||||
'restart',
|
'restart',
|
||||||
'start',
|
'start',
|
||||||
|
|||||||
@@ -44,6 +44,7 @@ import {
|
|||||||
registerFleetReconcilerCommands,
|
registerFleetReconcilerCommands,
|
||||||
type FleetReconcilerCommandDeps,
|
type FleetReconcilerCommandDeps,
|
||||||
} from './fleet-reconciler-command.js';
|
} from './fleet-reconciler-command.js';
|
||||||
|
import { registerFleetRegenCommand } from './fleet-regen-command.js';
|
||||||
import { resolveCommsBlock } from '../fleet/comms-onboarding.js';
|
import { resolveCommsBlock } from '../fleet/comms-onboarding.js';
|
||||||
import {
|
import {
|
||||||
applyPreparedAgentEnvironmentProjection,
|
applyPreparedAgentEnvironmentProjection,
|
||||||
@@ -2063,6 +2064,18 @@ export function registerFleetCommand(program: Command, deps: FleetCommandDeps =
|
|||||||
reconcileDeps: deps.reconcileDeps,
|
reconcileDeps: deps.reconcileDeps,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Recovery (#791 PR3): rebuild roster-derived env projections from the roster
|
||||||
|
// SSOT. Projection-only and preview-first — never issues a lifecycle restart.
|
||||||
|
registerFleetRegenCommand(cmd, {
|
||||||
|
runner,
|
||||||
|
mosaicHome: deps.mosaicHome,
|
||||||
|
// Resolve personas the SAME way reconcile does: forward any configured roots
|
||||||
|
// so a custom-persona-root deployment cannot have reconcile accept a roster
|
||||||
|
// that regen then rejects against the default `<mosaicHome>/fleet/roles`.
|
||||||
|
rolesDir: deps.reconcileDeps?.rolesDir,
|
||||||
|
overrideDir: deps.reconcileDeps?.overrideDir,
|
||||||
|
});
|
||||||
|
|
||||||
return cmd;
|
return cmd;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -381,6 +381,22 @@ function isMissingFile(error: unknown): boolean {
|
|||||||
return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT';
|
return typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT';
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Non-blocking acquire of the roster mutation lock: an exclusive `wx` create that
|
||||||
|
* throws `concurrent-mutation` (never waits) if the lock is already held. The
|
||||||
|
* on-disk format is an empty private file. CRUD releases in a plain `finally` with
|
||||||
|
* no result-preservation logic, so a release fault here must not override the
|
||||||
|
* mutation result — the release swallows unlink failures by design.
|
||||||
|
*
|
||||||
|
* The recovery-framed `fleet regen` contends on this exact same lock file, but it
|
||||||
|
* does so through the hardened, ownership-proving acquirer in the reconciler
|
||||||
|
* (`acquirePrivateRosterMutationLock`), not this one: whoever wins the `wx` create
|
||||||
|
* owns the file (the loser always gets `concurrent-mutation`), so the reconciler's
|
||||||
|
* ownership token is only ever written and read back by the same regen invocation,
|
||||||
|
* never by this empty-file writer. The two acquirers stay mutually compatible at
|
||||||
|
* the `wx`-contention layer while regen additionally proves ownership before it
|
||||||
|
* unlinks — a guarantee CRUD does not need because it releases unconditionally.
|
||||||
|
*/
|
||||||
async function acquireMutationLock(lockPath: string): Promise<() => Promise<void>> {
|
async function acquireMutationLock(lockPath: string): Promise<() => Promise<void>> {
|
||||||
try {
|
try {
|
||||||
const handle = await open(lockPath, 'wx', 0o600);
|
const handle = await open(lockPath, 'wx', 0o600);
|
||||||
|
|||||||
@@ -584,6 +584,29 @@ function defaultValidateRoster(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The single roster-v2 → generated-projection mapping. Both the reconciler's
|
||||||
|
* apply path ({@link defaultPrepareProjections}) and the recovery-framed
|
||||||
|
* `mosaic fleet regen` command derive their generated env from THIS one
|
||||||
|
* function, so the two paths can never drift (the #791 single-SSOT invariant).
|
||||||
|
* Pure: no IO, no clock — a deterministic function of the roster alone.
|
||||||
|
*/
|
||||||
|
export function projectRosterV2AgentGeneratedEnv(
|
||||||
|
roster: FleetRosterV2,
|
||||||
|
agent: FleetRosterV2Agent,
|
||||||
|
): Readonly<Record<string, string>> {
|
||||||
|
return {
|
||||||
|
MOSAIC_AGENT_NAME: agent.name,
|
||||||
|
MOSAIC_AGENT_CLASS: agent.className,
|
||||||
|
MOSAIC_AGENT_RUNTIME: agent.runtime,
|
||||||
|
MOSAIC_AGENT_MODEL: agent.model,
|
||||||
|
MOSAIC_AGENT_REASONING: agent.reasoning,
|
||||||
|
MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy,
|
||||||
|
MOSAIC_AGENT_WORKDIR: agent.workingDirectory,
|
||||||
|
MOSAIC_TMUX_SOCKET: roster.tmux.socketName,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
function defaultPrepareProjections(
|
function defaultPrepareProjections(
|
||||||
request: FleetReconcileRequest,
|
request: FleetReconcileRequest,
|
||||||
): (roster: FleetRosterV2) => Promise<readonly PreparedAgentEnvironmentProjection[]> {
|
): (roster: FleetRosterV2) => Promise<readonly PreparedAgentEnvironmentProjection[]> {
|
||||||
@@ -596,16 +619,7 @@ function defaultPrepareProjections(
|
|||||||
mosaicHome,
|
mosaicHome,
|
||||||
agentEnvDir: join(mosaicHome, 'fleet', 'agents'),
|
agentEnvDir: join(mosaicHome, 'fleet', 'agents'),
|
||||||
agentName: agent.name,
|
agentName: agent.name,
|
||||||
generated: {
|
generated: projectRosterV2AgentGeneratedEnv(roster, agent),
|
||||||
MOSAIC_AGENT_NAME: agent.name,
|
|
||||||
MOSAIC_AGENT_CLASS: agent.className,
|
|
||||||
MOSAIC_AGENT_RUNTIME: agent.runtime,
|
|
||||||
MOSAIC_AGENT_MODEL: agent.model,
|
|
||||||
MOSAIC_AGENT_REASONING: agent.reasoning,
|
|
||||||
MOSAIC_AGENT_TOOL_POLICY: agent.toolPolicy,
|
|
||||||
MOSAIC_AGENT_WORKDIR: agent.workingDirectory,
|
|
||||||
MOSAIC_TMUX_SOCKET: roster.tmux.socketName,
|
|
||||||
},
|
|
||||||
}),
|
}),
|
||||||
),
|
),
|
||||||
);
|
);
|
||||||
@@ -620,62 +634,193 @@ function mosaicHomeFor(deps: FleetReconcileDeps): string {
|
|||||||
return deps.mosaicHome ?? join(homedir(), '.config', 'mosaic');
|
return deps.mosaicHome ?? join(homedir(), '.config', 'mosaic');
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Acquires a private lock only after proving the canonical managed path. */
|
/** Acquires the private reconcile lock only after proving the canonical managed path. */
|
||||||
export function acquirePrivateReconcileLock(
|
export function acquirePrivateReconcileLock(
|
||||||
mosaicHome: string,
|
mosaicHome: string,
|
||||||
openLock: typeof open = open,
|
openLock: typeof open = open,
|
||||||
|
): () => Promise<() => Promise<void>> {
|
||||||
|
return acquirePrivateManagedRosterLock(
|
||||||
|
mosaicHome,
|
||||||
|
'roster.yaml.reconcile.lock',
|
||||||
|
'Another roster reconciliation is in progress.',
|
||||||
|
openLock,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Acquires the private roster MUTATION lock (`roster.yaml.mutation.lock`) with the
|
||||||
|
* exact same ownership-proof discipline as the reconcile lock. Exposed so the
|
||||||
|
* recovery-framed `mosaic fleet regen` can serialize its projection rewrites
|
||||||
|
* against agent CRUD — which holds this very lock while it rewrites the roster and
|
||||||
|
* the same derived projections — reusing this hardened acquirer rather than
|
||||||
|
* reimplementing it.
|
||||||
|
*
|
||||||
|
* The release PROVES ownership (device/inode + token) before unlinking, so a lock
|
||||||
|
* that was already cleared and replaced by the time release runs is never deleted
|
||||||
|
* out from under the new owner (it fails closed as `lock-cleanup-failed` instead),
|
||||||
|
* and that cleanup fault is propagated to the caller rather than silently dropped.
|
||||||
|
* CRUD contends on the identical path with a plain empty-file `wx` create; the two
|
||||||
|
* never co-own it (whoever wins `wx` owns it; the loser gets `concurrent-mutation`),
|
||||||
|
* so the token this writes is only ever read back by the same regen invocation that
|
||||||
|
* wrote it.
|
||||||
|
*
|
||||||
|
* SCOPE of the guarantee (matches the merged reconcile lock exactly — see
|
||||||
|
* {@link acquirePrivateManagedRosterLock}): the ownership proof closes the
|
||||||
|
* *reachable* window — a stale-lock reaper or operator cleared our lock and another
|
||||||
|
* writer took it BEFORE our release began — not the residual sub-instruction window
|
||||||
|
* between the final check and the path-based `unlink`. Closing that residual fully
|
||||||
|
* requires an fd-held advisory lock adopted by every fleet writer (CRUD, reconcile,
|
||||||
|
* regen), which is a cross-cutting mechanism change out of scope for this
|
||||||
|
* projection-only recovery command; it is unreachable within the `wx` writer
|
||||||
|
* protocol regardless (no Mosaic writer removes a lock it does not own, so only
|
||||||
|
* external interference can vacate our inode mid-release).
|
||||||
|
*/
|
||||||
|
export function acquirePrivateRosterMutationLock(
|
||||||
|
mosaicHome: string,
|
||||||
|
openLock: typeof open = open,
|
||||||
|
): () => Promise<() => Promise<void>> {
|
||||||
|
return acquirePrivateManagedRosterLock(
|
||||||
|
mosaicHome,
|
||||||
|
'roster.yaml.mutation.lock',
|
||||||
|
'Another roster mutation is in progress.',
|
||||||
|
openLock,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Shared body for the ownership-proving managed roster locks. Acquires `lockLeaf`
|
||||||
|
* under `<mosaicHome>/fleet` with a private `wx` create, writes an ownership token,
|
||||||
|
* and returns a release that re-proves device/inode + token before unlinking so it
|
||||||
|
* can never remove a lock it no longer owns. Non-blocking: throws `concurrent-mutation`
|
||||||
|
* (with `busyMessage`) on contention rather than waiting.
|
||||||
|
*/
|
||||||
|
function acquirePrivateManagedRosterLock(
|
||||||
|
mosaicHome: string,
|
||||||
|
lockLeaf: string,
|
||||||
|
busyMessage: string,
|
||||||
|
openLock: typeof open,
|
||||||
): () => Promise<() => Promise<void>> {
|
): () => Promise<() => Promise<void>> {
|
||||||
const fleetDir = join(mosaicHome, 'fleet');
|
const fleetDir = join(mosaicHome, 'fleet');
|
||||||
const lockPath = join(fleetDir, 'roster.yaml.reconcile.lock');
|
const lockPath = join(fleetDir, lockLeaf);
|
||||||
|
// The message-producing helpers below serve BOTH managed locks, so every fault
|
||||||
|
// names the actual lock file (`fleet/<leaf>`) — an operator needs to know WHICH
|
||||||
|
// lock is stale, not a hardcoded "reconciliation lock".
|
||||||
|
const lockLabel = `fleet/${lockLeaf}`;
|
||||||
return async (): Promise<() => Promise<void>> => {
|
return async (): Promise<() => Promise<void>> => {
|
||||||
await assertPrivateManagedDirectory(mosaicHome);
|
await assertPrivateManagedDirectory(mosaicHome);
|
||||||
await assertPrivateManagedDirectory(fleetDir);
|
await assertPrivateManagedDirectory(fleetDir);
|
||||||
await assertSafeLockLeafIfPresent(lockPath);
|
await assertSafeLockLeafIfPresent(lockPath, lockLabel);
|
||||||
|
|
||||||
let handle: FileHandle;
|
let handle: FileHandle;
|
||||||
try {
|
try {
|
||||||
handle = await openLock(lockPath, 'wx', 0o600);
|
handle = await openLock(lockPath, 'wx', 0o600);
|
||||||
} catch (error: unknown) {
|
} catch (error: unknown) {
|
||||||
if (isCode(error, 'EEXIST')) {
|
if (isCode(error, 'EEXIST')) {
|
||||||
await assertSafeLockLeafIfPresent(lockPath);
|
await assertSafeLockLeafIfPresent(lockPath, lockLabel);
|
||||||
throw new FleetReconcileError(
|
throw new FleetReconcileError('concurrent-mutation', busyMessage);
|
||||||
'concurrent-mutation',
|
|
||||||
'Another roster reconciliation is in progress.',
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
throw new FleetReconcileError('lock-io-failed', 'The reconciliation lock cannot be created.');
|
throw new FleetReconcileError('lock-io-failed', `The ${lockLabel} lock cannot be created.`);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Identity of the lock WE exclusively created (the `wx` create guaranteed it is
|
||||||
|
// ours). Captured up front so both the release closure and the init-failure
|
||||||
|
// cleanup below can prove ownership by device/inode before touching the file.
|
||||||
|
const created = await handle.stat().catch((): undefined => undefined);
|
||||||
|
|
||||||
const token = randomUUID();
|
const token = randomUUID();
|
||||||
|
let tokenPersisted = false;
|
||||||
try {
|
try {
|
||||||
await handle.writeFile(`${token}\n`, 'utf8');
|
await handle.writeFile(`${token}\n`, 'utf8');
|
||||||
const opened = await handle.stat();
|
tokenPersisted = true;
|
||||||
await handle.close();
|
await handle.close();
|
||||||
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'unsafe-lock');
|
if (!created)
|
||||||
|
throw new FleetReconcileError(
|
||||||
|
'lock-io-failed',
|
||||||
|
`The ${lockLabel} lock cannot be initialized.`,
|
||||||
|
);
|
||||||
|
await assertLockOwnership(
|
||||||
|
lockPath,
|
||||||
|
created.dev,
|
||||||
|
created.ino,
|
||||||
|
token,
|
||||||
|
'unsafe-lock',
|
||||||
|
lockLabel,
|
||||||
|
);
|
||||||
return async (): Promise<void> => {
|
return async (): Promise<void> => {
|
||||||
try {
|
try {
|
||||||
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed');
|
await assertLockOwnership(
|
||||||
await assertLockOwnership(lockPath, opened.dev, opened.ino, token, 'lock-cleanup-failed');
|
lockPath,
|
||||||
|
created.dev,
|
||||||
|
created.ino,
|
||||||
|
token,
|
||||||
|
'lock-cleanup-failed',
|
||||||
|
lockLabel,
|
||||||
|
);
|
||||||
await unlink(lockPath);
|
await unlink(lockPath);
|
||||||
} catch (error: unknown) {
|
} catch (error: unknown) {
|
||||||
if (error instanceof FleetReconcileError) throw error;
|
if (error instanceof FleetReconcileError) throw error;
|
||||||
throw new FleetReconcileError(
|
throw new FleetReconcileError('lock-cleanup-failed', `The ${lockLabel} cleanup failed.`);
|
||||||
'lock-cleanup-failed',
|
|
||||||
'The reconciliation lock cleanup failed.',
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
} catch (error: unknown) {
|
} catch (error: unknown) {
|
||||||
await handle.close().catch((): void => {});
|
await handle.close().catch((): void => {});
|
||||||
|
// Best-effort: remove the lock THIS invocation created so a transient init
|
||||||
|
// fault does not strand a lock that would block future regen and agent CRUD.
|
||||||
|
// dev/ino-guarded so it can never delete a replacement; swallowed so cleanup
|
||||||
|
// failure never masks the initialization error being surfaced. The token is
|
||||||
|
// passed only once persisted, so the fallback path (when the post-create stat
|
||||||
|
// failed and dev/ino is unavailable) can still prove ownership by content.
|
||||||
|
await removeOwnedLockLeafBestEffort(lockPath, created, tokenPersisted ? token : undefined);
|
||||||
if (error instanceof FleetReconcileError) throw error;
|
if (error instanceof FleetReconcileError) throw error;
|
||||||
throw new FleetReconcileError(
|
throw new FleetReconcileError(
|
||||||
'lock-io-failed',
|
'lock-io-failed',
|
||||||
'The reconciliation lock cannot be initialized.',
|
`The ${lockLabel} lock cannot be initialized.`,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Best-effort removal of a managed lock leaf THIS process created, used only on the
|
||||||
|
* initialization-failure path. Two independent ownership proofs, so a lock that was
|
||||||
|
* already replaced is never deleted:
|
||||||
|
*
|
||||||
|
* - primary: the captured device/inode of the file we exclusively `wx`-created; or
|
||||||
|
* - fallback: when the post-create stat itself failed (dev/ino unavailable), the
|
||||||
|
* random `ownershipToken` we persisted — only OUR lock carries it, so a CRUD
|
||||||
|
* (empty) or a differently-tokened replacement is never removed.
|
||||||
|
*
|
||||||
|
* Every fault is swallowed because the caller is already surfacing the initialization
|
||||||
|
* error and a leftover lock is recoverable via the documented runbook. If BOTH proofs
|
||||||
|
* are unavailable (stat failed AND the token write never landed) the lock is left in
|
||||||
|
* place rather than risk deleting another writer's file — a doubly-degenerate case
|
||||||
|
* requiring two independent fs faults on a just-created fd.
|
||||||
|
*/
|
||||||
|
async function removeOwnedLockLeafBestEffort(
|
||||||
|
lockPath: string,
|
||||||
|
created: { dev: number; ino: number } | undefined,
|
||||||
|
ownershipToken: string | undefined,
|
||||||
|
): Promise<void> {
|
||||||
|
try {
|
||||||
|
const current = await lstat(lockPath);
|
||||||
|
if (!current.isFile() || current.isSymbolicLink()) return;
|
||||||
|
if (created) {
|
||||||
|
if (current.dev === created.dev && current.ino === created.ino) {
|
||||||
|
await unlink(lockPath);
|
||||||
|
}
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
if (ownershipToken !== undefined) {
|
||||||
|
const contents = await readFile(lockPath, 'utf8');
|
||||||
|
if (contents.trim() === ownershipToken) {
|
||||||
|
await unlink(lockPath);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// Swallowed: leftover lock is recoverable; do not mask the init error.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
async function assertPrivateManagedDirectory(path: string): Promise<void> {
|
async function assertPrivateManagedDirectory(path: string): Promise<void> {
|
||||||
try {
|
try {
|
||||||
const metadata = await lstat(path);
|
const metadata = await lstat(path);
|
||||||
@@ -691,16 +836,16 @@ async function assertPrivateManagedDirectory(path: string): Promise<void> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
async function assertSafeLockLeafIfPresent(lockPath: string): Promise<void> {
|
async function assertSafeLockLeafIfPresent(lockPath: string, lockLabel: string): Promise<void> {
|
||||||
try {
|
try {
|
||||||
const metadata = await lstat(lockPath);
|
const metadata = await lstat(lockPath);
|
||||||
if (!metadata.isFile() || metadata.isSymbolicLink() || (metadata.mode & 0o077) !== 0) {
|
if (!metadata.isFile() || metadata.isSymbolicLink() || (metadata.mode & 0o077) !== 0) {
|
||||||
throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unsafe.');
|
throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unsafe.`);
|
||||||
}
|
}
|
||||||
} catch (error: unknown) {
|
} catch (error: unknown) {
|
||||||
if (isCode(error, 'ENOENT')) return;
|
if (isCode(error, 'ENOENT')) return;
|
||||||
if (error instanceof FleetReconcileError) throw error;
|
if (error instanceof FleetReconcileError) throw error;
|
||||||
throw new FleetReconcileError('unsafe-lock', 'The reconciliation lock path is unavailable.');
|
throw new FleetReconcileError('unsafe-lock', `The ${lockLabel} lock path is unavailable.`);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -710,6 +855,7 @@ async function assertLockOwnership(
|
|||||||
inode: number,
|
inode: number,
|
||||||
token: string,
|
token: string,
|
||||||
failureCode: 'unsafe-lock' | 'lock-cleanup-failed',
|
failureCode: 'unsafe-lock' | 'lock-cleanup-failed',
|
||||||
|
lockLabel: string,
|
||||||
): Promise<void> {
|
): Promise<void> {
|
||||||
try {
|
try {
|
||||||
const metadata = await lstat(lockPath);
|
const metadata = await lstat(lockPath);
|
||||||
@@ -720,24 +866,21 @@ async function assertLockOwnership(
|
|||||||
metadata.dev !== device ||
|
metadata.dev !== device ||
|
||||||
metadata.ino !== inode
|
metadata.ino !== inode
|
||||||
) {
|
) {
|
||||||
throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.');
|
throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`);
|
||||||
}
|
}
|
||||||
const handle = await open(lockPath, constants.O_RDONLY | constants.O_NOFOLLOW);
|
const handle = await open(lockPath, constants.O_RDONLY | constants.O_NOFOLLOW);
|
||||||
try {
|
try {
|
||||||
const opened = await handle.stat();
|
const opened = await handle.stat();
|
||||||
const contents = await handle.readFile({ encoding: 'utf8' });
|
const contents = await handle.readFile({ encoding: 'utf8' });
|
||||||
if (opened.dev !== device || opened.ino !== inode || contents !== `${token}\n`) {
|
if (opened.dev !== device || opened.ino !== inode || contents !== `${token}\n`) {
|
||||||
throw new FleetReconcileError(failureCode, 'The reconciliation lock ownership changed.');
|
throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership changed.`);
|
||||||
}
|
}
|
||||||
} finally {
|
} finally {
|
||||||
await handle.close();
|
await handle.close();
|
||||||
}
|
}
|
||||||
} catch (error: unknown) {
|
} catch (error: unknown) {
|
||||||
if (error instanceof FleetReconcileError) throw error;
|
if (error instanceof FleetReconcileError) throw error;
|
||||||
throw new FleetReconcileError(
|
throw new FleetReconcileError(failureCode, `The ${lockLabel} ownership cannot be proven.`);
|
||||||
failureCode,
|
|
||||||
'The reconciliation lock ownership cannot be proven.',
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -204,6 +204,53 @@ export async function prepareAgentEnvironmentProjection(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** A generated-only projection validated without touching any legacy/local/quarantine file. */
|
||||||
|
export interface PreparedGeneratedAgentEnvironmentProjection {
|
||||||
|
readonly mosaicHome: string;
|
||||||
|
readonly agentEnvDir: string;
|
||||||
|
readonly generatedPath: string;
|
||||||
|
readonly generated: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validates ONLY the roster-derived generated projection for a recovery rebuild.
|
||||||
|
* Unlike {@link prepareAgentEnvironmentProjection}, this never reads, classifies,
|
||||||
|
* relocates, or quarantines the legacy `.env` / `.env.local` operator surface — it
|
||||||
|
* exists so `mosaic fleet regen` has no code path that can mutate anything except
|
||||||
|
* `<name>.env.generated`. The existing generated file, if present, must already be
|
||||||
|
* a private regular file.
|
||||||
|
*/
|
||||||
|
export async function prepareGeneratedAgentEnvironmentProjection(
|
||||||
|
options: AgentEnvironmentProjectionOptions,
|
||||||
|
): Promise<PreparedGeneratedAgentEnvironmentProjection> {
|
||||||
|
if (!AGENT_NAME.test(options.agentName)) {
|
||||||
|
throw new AgentEnvBoundaryError('unsafe-agent-name', 'MOSAIC_AGENT_NAME', options.agentName);
|
||||||
|
}
|
||||||
|
await validatePrivateProjectionDirectory(options.mosaicHome, options.agentEnvDir);
|
||||||
|
const generatedPath = join(options.agentEnvDir, `${options.agentName}.env.generated`);
|
||||||
|
const generated = renderGeneratedAgentEnvironment(options.generated);
|
||||||
|
await assertPrivateRegularFileIfPresent(generatedPath);
|
||||||
|
return {
|
||||||
|
mosaicHome: options.mosaicHome,
|
||||||
|
agentEnvDir: options.agentEnvDir,
|
||||||
|
generatedPath,
|
||||||
|
generated,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Applies a generated-only projection: writes ONLY `<name>.env.generated` atomically
|
||||||
|
* and touches nothing else. It never writes `.env.local`/`.env.quarantine` and never
|
||||||
|
* unlinks the legacy `.env` — the projection-only recovery guarantee is structural.
|
||||||
|
*/
|
||||||
|
export async function applyPreparedGeneratedAgentEnvironmentProjection(
|
||||||
|
prepared: PreparedGeneratedAgentEnvironmentProjection,
|
||||||
|
): Promise<string> {
|
||||||
|
await ensurePrivateProjectionDirectory(prepared.mosaicHome, prepared.agentEnvDir);
|
||||||
|
await writePrivateAtomically(prepared.generatedPath, prepared.generated);
|
||||||
|
return prepared.generatedPath;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Validates only the exact generated projection eligible for a roster delete.
|
* Validates only the exact generated projection eligible for a roster delete.
|
||||||
* Local overrides, legacy input, and quarantine records are operator-retained and
|
* Local overrides, legacy input, and quarantine records are operator-retained and
|
||||||
|
|||||||
Reference in New Issue
Block a user