feat(fleet): add mosaic fleet regen recovery command
All checks were successful
ci/woodpecker/pr/ci Pipeline was successful

Add `mosaic fleet regen`, a projection-only recovery command that rebuilds
each `fleet/agents/<name>.env.generated` from the `roster.yaml` SSOT after an
upgrade or partial write leaves the generated projections stale or missing.

- Dry-run by default; `--write` applies; `--json` for machine output. Reuses the
  merged reconciler's projection plumbing (projectRosterV2AgentGeneratedEnv +
  the generated-env boundary) rather than reimplementing fleet logic.
- Structurally NEVER issues a lifecycle/restart call — regen recovers config
  only; a recordingRunner gate proves no runner invocation ever occurs.
- Serializes against agent CRUD and reconcile via BOTH fleet locks
  (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired
  mutation-then-reconcile and released in reverse; both are non-blocking `wx`
  locks that throw on contention, so no deadlock is possible.
- Hardens the shared managed-lock helper: ownership-proving tokened lock reused
  for both locks with per-lock fault labels; init-failure cleanup no longer
  strands a just-created lock (dev/ino guard, with a persisted-token fallback
  when the post-create stat itself fails); acquire-unwind surfaces a lock
  cleanup fault instead of dropping it.
- Resolves personas the SAME way reconcile does by forwarding configured
  rolesDir/overrideDir, so a custom-persona-root deployment cannot have
  reconcile accept a roster that regen rejects.
- Report/output is secrev-safe: paths and counts only, never projected values.
- Docs: upgrade-safety-and-recovery runbook + fleet-local-canary note.

Tests are TDD red-first with co-located specs (regen spec: 26 tests covering
dry-run/write dispositions, the never-restarts gate, all lock regressions, and
persona-root wiring).

A residual check-then-unlink TOCTOU in the lock release remains (byte-identical
to the merged reconcile lock; unreachable within the `wx` writer protocol); its
true fix is an fd-held advisory lock adopted by all fleet writers, tracked as a
separate follow-up.

Part of #791

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Hermes Agent
2026-07-16 21:57:38 -05:00
parent 31607a4af6
commit 1eb37f6fd0
10 changed files with 1994 additions and 37 deletions

View File

@@ -204,6 +204,53 @@ export async function prepareAgentEnvironmentProjection(
};
}
/** A generated-only projection validated without touching any legacy/local/quarantine file. */
export interface PreparedGeneratedAgentEnvironmentProjection {
readonly mosaicHome: string;
readonly agentEnvDir: string;
readonly generatedPath: string;
readonly generated: string;
}
/**
* Validates ONLY the roster-derived generated projection for a recovery rebuild.
* Unlike {@link prepareAgentEnvironmentProjection}, this never reads, classifies,
* relocates, or quarantines the legacy `.env` / `.env.local` operator surface — it
* exists so `mosaic fleet regen` has no code path that can mutate anything except
* `<name>.env.generated`. The existing generated file, if present, must already be
* a private regular file.
*/
export async function prepareGeneratedAgentEnvironmentProjection(
options: AgentEnvironmentProjectionOptions,
): Promise<PreparedGeneratedAgentEnvironmentProjection> {
if (!AGENT_NAME.test(options.agentName)) {
throw new AgentEnvBoundaryError('unsafe-agent-name', 'MOSAIC_AGENT_NAME', options.agentName);
}
await validatePrivateProjectionDirectory(options.mosaicHome, options.agentEnvDir);
const generatedPath = join(options.agentEnvDir, `${options.agentName}.env.generated`);
const generated = renderGeneratedAgentEnvironment(options.generated);
await assertPrivateRegularFileIfPresent(generatedPath);
return {
mosaicHome: options.mosaicHome,
agentEnvDir: options.agentEnvDir,
generatedPath,
generated,
};
}
/**
* Applies a generated-only projection: writes ONLY `<name>.env.generated` atomically
* and touches nothing else. It never writes `.env.local`/`.env.quarantine` and never
* unlinks the legacy `.env` — the projection-only recovery guarantee is structural.
*/
export async function applyPreparedGeneratedAgentEnvironmentProjection(
prepared: PreparedGeneratedAgentEnvironmentProjection,
): Promise<string> {
await ensurePrivateProjectionDirectory(prepared.mosaicHome, prepared.agentEnvDir);
await writePrivateAtomically(prepared.generatedPath, prepared.generated);
return prepared.generatedPath;
}
/**
* Validates only the exact generated projection eligible for a roster delete.
* Local overrides, legacy input, and quarantine records are operator-retained and