mosaicstack/stack
21
0
Fork 1
Code Issues 336 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

feat(fleet): mosaic fleet regen — regenerate roster-derived projections (PR3 of #791) #813

Merged
jason.woltje merged 1 commits from feat/791-pr3-fleet-regen into main 2026-07-17 03:17:56 +00:00
Conversation 1 Commits 1 Files Changed 10
+1994 -37
jason.woltje commented 2026-07-17 02:59:47 +00:00
Owner
Copy Link
Copy Source

PR3 of #791 — mosaic fleet regen (regeneration-from-SSOT)

Part of #791. Third and final PR of the ratified 3-PR DAG (b strict ownership separation #802 ✅ merged · a transactional pre-update snapshot #811 ✅ merged · d regeneration-from-SSOT — this PR). The umbrella #791 stays open until MS-LEAD closes it.

What this adds

mosaic fleet regen — a projection-only recovery command that rebuilds each fleet/agents/<name>.env.generated from the roster.yaml SSOT after an upgrade or partial write leaves the generated projections stale or missing.

  • Dry-run by default; --write applies; --json for machine output.
  • Never restarts agents — regen recovers config only. It is structurally incapable of a lifecycle call (the CommandRunner is declared but never referenced on any executable path), and a recordingRunner gate asserts zero runner invocations across every test.
  • Reuses the merged reconciler plumbing (projectRosterV2AgentGeneratedEnv + the generated-env boundary) — it does not reimplement fleet logic.
  • Secrev-safe output — report/JSON emit paths and counts only, never a projected KEY=value.
  • Same semantic gate as reconcile — validates roster persona resolution + tool-policy the same way reconcile/plan/verify do, and now forwards configured rolesDir/overrideDir so a custom-persona-root deployment cannot have reconcile accept a roster that regen rejects.

Concurrency / lock hardening

Regen serializes against agent CRUD and reconcile via both fleet locks (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired mutation→reconcile and released in reverse. Both are non-blocking wx locks that throw on contention → no deadlock. The shared managed-lock helper was hardened:

  • One ownership-proving tokened lock helper reused for both locks, with per-lock fault labels (a mutation-lock fault no longer mislabels itself as the reconcile lock).
  • Init-failure cleanup no longer strands a just-created lock: dev/ino-guarded removal, plus a persisted-token fallback for the narrow case where the post-create stat itself fails (dev/ino unavailable) — it removes only a leaf whose content matches our random UUID, so it can never delete a CRUD (empty) or differently-tokened replacement.
  • Acquire-unwind surfaces a lock-cleanup fault instead of dropping it.

Tests (TDD, red-first, co-located)

fleet-regen-command.spec.ts — 26 tests: dry-run/write dispositions, the load-bearing never-restarts gate, all lock regressions (ownership-proof-before-unlink, acquire-unwind fault, dual-lock stale warning, stranded-lock on init failure, stat-failure strand), and persona-root wiring. Every finding fixed in review was proven red-first.

Known limitation — deferred to a follow-up

A residual check-then-unlink TOCTOU remains in the lock release (between the final ownership proof and the path unlink). It is byte-identical to the already-merged reconcile-lock release on main, unreachable within the wx writer protocol (no Mosaic writer removes a lock it doesn't own; only an external rm/stale-lock reaper can vacate the inode mid-release), and its true fix is a cross-cutting mechanism change — an fd-held advisory lock adopted by all fleet writers (CRUD + reconcile + regen) — out of scope for a projection-only recovery PR. Tracked as a separate follow-up issue. Two independent reviews (author≠reviewer) treated it as non-blocking.

Review evidence

  • Gates green: typecheck · lint · format:check · full mosaic vitest 1278.
  • Independent reviews (author≠reviewer): core lock hardening PASS; token-fallback delta PASS; persona-root wiring PASS — each confirmed by revert-to-RED.
  • Codex code + security review converged to the single deferred TOCTOU; no secrets, no injection, generated-only writes.

Base: 31607a4 (fresh main). No self-merge — holding at PR-open for MS-LEAD's exact-head review + durable Reviewer-of-Record before merge.

## PR3 of #791 — `mosaic fleet regen` (regeneration-from-SSOT) Part of #791. Third and final PR of the ratified 3-PR DAG (b strict ownership separation #802 ✅ merged · a transactional pre-update snapshot #811 ✅ merged · **d regeneration-from-SSOT — this PR**). The umbrella #791 stays open until MS-LEAD closes it. ### What this adds `mosaic fleet regen` — a **projection-only recovery command** that rebuilds each `fleet/agents/<name>.env.generated` from the `roster.yaml` SSOT after an upgrade or partial write leaves the generated projections stale or missing. - **Dry-run by default**; `--write` applies; `--json` for machine output. - **Never restarts agents** — regen recovers *config only*. It is structurally incapable of a lifecycle call (the `CommandRunner` is declared but never referenced on any executable path), and a `recordingRunner` gate asserts *zero* runner invocations across every test. - **Reuses the merged reconciler plumbing** (`projectRosterV2AgentGeneratedEnv` + the generated-env boundary) — it does not reimplement fleet logic. - **Secrev-safe output** — report/JSON emit **paths and counts only**, never a projected `KEY=value`. - **Same semantic gate as reconcile** — validates roster persona resolution + tool-policy the same way `reconcile`/`plan`/`verify` do, and now forwards configured `rolesDir`/`overrideDir` so a custom-persona-root deployment cannot have reconcile accept a roster that regen rejects. ### Concurrency / lock hardening Regen serializes against agent CRUD and reconcile via **both** fleet locks (`roster.yaml.mutation.lock` + `roster.yaml.reconcile.lock`), acquired mutation→reconcile and released in reverse. Both are non-blocking `wx` locks that throw on contention → no deadlock. The shared managed-lock helper was hardened: - One ownership-proving **tokened** lock helper reused for both locks, with **per-lock fault labels** (a mutation-lock fault no longer mislabels itself as the reconcile lock). - **Init-failure cleanup no longer strands a just-created lock**: dev/ino-guarded removal, plus a **persisted-token fallback** for the narrow case where the post-create `stat` itself fails (dev/ino unavailable) — it removes only a leaf whose content matches our random UUID, so it can never delete a CRUD (empty) or differently-tokened replacement. - **Acquire-unwind surfaces a lock-cleanup fault** instead of dropping it. ### Tests (TDD, red-first, co-located) `fleet-regen-command.spec.ts` — **26 tests**: dry-run/write dispositions, the load-bearing never-restarts gate, all lock regressions (ownership-proof-before-unlink, acquire-unwind fault, dual-lock stale warning, stranded-lock on init failure, stat-failure strand), and persona-root wiring. Every finding fixed in review was proven red-first. ### Known limitation — deferred to a follow-up A residual check-then-unlink **TOCTOU** remains in the lock release (between the final ownership proof and the path `unlink`). It is **byte-identical to the already-merged reconcile-lock release on `main`**, unreachable within the `wx` writer protocol (no Mosaic writer removes a lock it doesn't own; only an external `rm`/stale-lock reaper can vacate the inode mid-release), and its true fix is a cross-cutting mechanism change — an **fd-held advisory lock adopted by all fleet writers (CRUD + reconcile + regen)** — out of scope for a projection-only recovery PR. Tracked as a separate follow-up issue. Two independent reviews (author≠reviewer) treated it as non-blocking. ### Review evidence - Gates green: `typecheck` · `lint` · `format:check` · full mosaic vitest **1278**. - Independent reviews (author≠reviewer): core lock hardening PASS; token-fallback delta PASS; persona-root wiring PASS — each confirmed by revert-to-RED. - Codex code + security review converged to the single deferred TOCTOU; no secrets, no injection, generated-only writes. **Base:** `31607a4` (fresh `main`). **No self-merge** — holding at PR-open for MS-LEAD's exact-head review + durable Reviewer-of-Record before merge.
jason.woltje added 1 commit 2026-07-17 02:59:48 +00:00
feat(fleet): add mosaic fleet regen recovery command
ci/woodpecker/pr/ci Pipeline was successful
Details
1eb37f6fd0 
Add `mosaic fleet regen`, a projection-only recovery command that rebuilds
each `fleet/agents/<name>.env.generated` from the `roster.yaml` SSOT after an
upgrade or partial write leaves the generated projections stale or missing.

- Dry-run by default; `--write` applies; `--json` for machine output. Reuses the
  merged reconciler's projection plumbing (projectRosterV2AgentGeneratedEnv +
  the generated-env boundary) rather than reimplementing fleet logic.
- Structurally NEVER issues a lifecycle/restart call — regen recovers config
  only; a recordingRunner gate proves no runner invocation ever occurs.
- Serializes against agent CRUD and reconcile via BOTH fleet locks
  (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired
  mutation-then-reconcile and released in reverse; both are non-blocking `wx`
  locks that throw on contention, so no deadlock is possible.
- Hardens the shared managed-lock helper: ownership-proving tokened lock reused
  for both locks with per-lock fault labels; init-failure cleanup no longer
  strands a just-created lock (dev/ino guard, with a persisted-token fallback
  when the post-create stat itself fails); acquire-unwind surfaces a lock
  cleanup fault instead of dropping it.
- Resolves personas the SAME way reconcile does by forwarding configured
  rolesDir/overrideDir, so a custom-persona-root deployment cannot have
  reconcile accept a roster that regen rejects.
- Report/output is secrev-safe: paths and counts only, never projected values.
- Docs: upgrade-safety-and-recovery runbook + fleet-local-canary note.

Tests are TDD red-first with co-located specs (regen spec: 26 tests covering
dry-run/write dispositions, the never-restarts gate, all lock regressions, and
persona-root wiring).

A residual check-then-unlink TOCTOU in the lock release remains (byte-identical
to the merged reconcile lock; unreachable within the `wx` writer protocol); its
true fix is an fd-held advisory lock adopted by all fleet writers, tracked as a
separate follow-up.

Part of #791

Co-Authored-By: Claude Opus 4.8 <[email protected]>
jason.woltje commented 2026-07-17 03:17:28 +00:00
Author
Owner
Copy Link
Copy Source

Reviewer-of-Record — exact head 1eb37f6f — VERDICT: APPROVE

Independent exact-head adversarial/security review (FRESH standalone, not carried-forward).

  • Author lane: ms-791 (Claude/Opus) · Reviewer lane: independent Opus adversarial/security reviewer (dispatched by MS Team Lead). author≠reviewer = PROCESS/LANE-attested (single Gitea account jason.woltje; separation by lane, not by provider account).
  • Head: 1eb37f6fd084afbdc8c00e4ff5f57d2c89fc2c14 · tree 614169817658f7f280e22054efdd4d8812ad12ff · base 31607a4 (merge-base == current main).
  • CI: authoritative Gitea combined commit-status at this exact head = success (Woodpecker #1884).
  • Scope: 10 files (+1994/−37), clean — no stray secret/snapshot/session.lock/.env.local.

Findings — all 8 adversarial areas PASS (zero blockers):

  1. Projection-only proven structural — deps.runner never invoked; recordingRunner gate asserts calls==[] && systemctl==[]; sole side-effect = one atomic write.
  2. Lock correctness — dual non-blocking wx, deadlock-impossible (only regen holds both, fixed order); init-failure cleanup dev/ino+token-guarded (no stranded lock on writeFile- or stat-fault); per-lock fault labels correct (no mislabel).
  3. CWE-367 release TOCTOU → SOUND-TO-DEFER (#814): reachable "replaced-before-release" window is CLOSED & pinned by test; residual check→unlink window is identical to the already-merged reconcile-lock release and unreachable within the wx writer protocol (only external rm/stale-reaper vacates the inode). Appropriate to defer for a projection-only PR.
  4. Secrev — result/JSON/report emit paths+counts+timestamps only, never a projected value; test asserts secrets absent from output.
  5. Generated-env boundary — AGENT_NAME validated before path build; writes ONLY <name>.env.generated; legacy .env/.env.local/.env.quarantine untouched.
  6. Persona-root wiring — regen forwards rolesDir/overrideDir + same validateRosterV2Semantics gate → regen and reconcile cannot disagree on persona acceptance.
  7. --write safety — atomic temp(0o600,wx)+rename; fail-closed (all validated before any write; mid-loop fault → incomplete with exact lists).
  8. Test integrity — assertions concrete, non-tautological, load-bearing.

INFO (non-blocking): (a) refactor makes the merged reconcile-lock release functionally equivalent but not byte-identical to main (removed one redundant ownership check; closed no window) — minor correction to the "byte-identical" wording; (b) no explicit test asserts both locks unlinked after a successful --write (covered indirectly). Neither blocks.

Gate status → mergeable: exact-head authoritative-green CI ✓ · fresh independent exact-head RoR APPROVE ✓ · scope clean ✓ · based on current main ✓ · CWE-367 residual deferred to #814 ✓. Durable RoR persisted to provider before queue guard.

## Reviewer-of-Record — exact head `1eb37f6f` — VERDICT: APPROVE **Independent exact-head adversarial/security review** (FRESH standalone, not carried-forward). - **Author lane:** ms-791 (Claude/Opus) · **Reviewer lane:** independent Opus adversarial/security reviewer (dispatched by MS Team Lead). author≠reviewer = PROCESS/LANE-attested (single Gitea account `jason.woltje`; separation by lane, not by provider account). - **Head:** `1eb37f6fd084afbdc8c00e4ff5f57d2c89fc2c14` · **tree** `614169817658f7f280e22054efdd4d8812ad12ff` · **base** `31607a4` (merge-base == current main). - **CI:** authoritative Gitea combined commit-status at this exact head = **success** (Woodpecker #1884). - **Scope:** 10 files (+1994/−37), clean — no stray secret/snapshot/session.lock/.env.local. **Findings — all 8 adversarial areas PASS (zero blockers):** 1. Projection-only proven structural — `deps.runner` never invoked; `recordingRunner` gate asserts `calls==[]` && `systemctl==[]`; sole side-effect = one atomic write. 2. Lock correctness — dual non-blocking `wx`, deadlock-impossible (only regen holds both, fixed order); init-failure cleanup dev/ino+token-guarded (no stranded lock on writeFile- or stat-fault); per-lock fault labels correct (no mislabel). 3. **CWE-367 release TOCTOU → SOUND-TO-DEFER (#814):** reachable "replaced-before-release" window is CLOSED & pinned by test; residual check→unlink window is identical to the already-merged reconcile-lock release and unreachable within the `wx` writer protocol (only external `rm`/stale-reaper vacates the inode). Appropriate to defer for a projection-only PR. 4. Secrev — result/JSON/report emit paths+counts+timestamps only, never a projected value; test asserts secrets absent from output. 5. Generated-env boundary — AGENT_NAME validated before path build; writes ONLY `<name>.env.generated`; legacy `.env`/`.env.local`/`.env.quarantine` untouched. 6. Persona-root wiring — regen forwards rolesDir/overrideDir + same `validateRosterV2Semantics` gate → regen and reconcile cannot disagree on persona acceptance. 7. `--write` safety — atomic temp(`0o600`,`wx`)+rename; fail-closed (all validated before any write; mid-loop fault → `incomplete` with exact lists). 8. Test integrity — assertions concrete, non-tautological, load-bearing. **INFO (non-blocking):** (a) refactor makes the merged reconcile-lock release *functionally* equivalent but not *byte-identical* to main (removed one redundant ownership check; closed no window) — minor correction to the "byte-identical" wording; (b) no explicit test asserts both locks unlinked after a successful `--write` (covered indirectly). Neither blocks. **Gate status → mergeable:** exact-head authoritative-green CI ✓ · fresh independent exact-head RoR APPROVE ✓ · scope clean ✓ · based on current main ✓ · CWE-367 residual deferred to #814 ✓. Durable RoR persisted to provider before queue guard.
jason.woltje merged commit 9ddc6fbda8 into main 2026-07-17 03:17:56 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
bug
Defect in framework tooling or delivered code
 fleet-enhancement
Fleet-wide reliability/speed/cost improvements surfaced by the nightly enhance pass
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
code-be-01 (Mosaic fleet seat code-be-01) code-be-02 (Mosaic fleet seat code-be-02) code-dogfood-01 (Mosaic fleet seat code-dogfood-01) code-infra-01 (Mosaic fleet seat code-infra-01) fargo fred gate-merge-01 (Mosaic fleet seat gate-merge-01) happy jason.woltje (Jason Woltje) marcie merge-gate ops-01 (Mosaic fleet seat ops-01) ops-02 (Mosaic fleet seat ops-02) ops-03 (Mosaic fleet seat ops-03) ops-ci-01 (Mosaic fleet seat ops-ci-01) ops-deploy-01 (Mosaic fleet seat ops-deploy-01) orch-01 (Mosaic fleet seat orch-01) pepper resume rev-code-01 rev-code-02 rev-security-01 rev-security-02 rev-security-03 (Mosaic fleet seat rev-security-03) sanity scooby (Scooby) scrappy shaggy tiny topher (Mosaic fleet seat topher) velma veronica (Mosaic fleet seat veronica) vision woodpecker
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: mosaicstack/stack#813