feat(fleet): add mosaic fleet regen recovery command

Add `mosaic fleet regen`, a projection-only recovery command that rebuilds
each `fleet/agents/<name>.env.generated` from the `roster.yaml` SSOT after an
upgrade or partial write leaves the generated projections stale or missing.

- Dry-run by default; `--write` applies; `--json` for machine output. Reuses the
  merged reconciler's projection plumbing (projectRosterV2AgentGeneratedEnv +
  the generated-env boundary) rather than reimplementing fleet logic.
- Structurally NEVER issues a lifecycle/restart call — regen recovers config
  only; a recordingRunner gate proves no runner invocation ever occurs.
- Serializes against agent CRUD and reconcile via BOTH fleet locks
  (roster.yaml.mutation.lock + roster.yaml.reconcile.lock), acquired
  mutation-then-reconcile and released in reverse; both are non-blocking `wx`
  locks that throw on contention, so no deadlock is possible.
- Hardens the shared managed-lock helper: ownership-proving tokened lock reused
  for both locks with per-lock fault labels; init-failure cleanup no longer
  strands a just-created lock (dev/ino guard, with a persisted-token fallback
  when the post-create stat itself fails); acquire-unwind surfaces a lock
  cleanup fault instead of dropping it.
- Resolves personas the SAME way reconcile does by forwarding configured
  rolesDir/overrideDir, so a custom-persona-root deployment cannot have
  reconcile accept a roster that regen rejects.
- Report/output is secrev-safe: paths and counts only, never projected values.
- Docs: upgrade-safety-and-recovery runbook + fleet-local-canary note.

Tests are TDD red-first with co-located specs (regen spec: 26 tests covering
dry-run/write dispositions, the never-restarts gate, all lock regressions, and
persona-root wiring).

A residual check-then-unlink TOCTOU in the lock release remains (byte-identical
to the merged reconcile lock; unreachable within the `wx` writer protocol); its
true fix is an fd-held advisory lock adopted by all fleet writers, tracked as a
separate follow-up.

Part of #791

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

docs/scratchpads/812-pr-review-comment.md

@@ -1,49 +0,0 @@
-# Issue #812 — durable Gitea PR review comments
-
-- **Lane:** ms-812
-- **Branch:** `fix/812-pr-review-comment`
-- **Issue:** mosaicstack/stack#812
-- **Budget:** 15K working estimate; single focused shell-wrapper/test/docs change.
-
-## Objective
-
-Make the Gitea `comment` action in `packages/mosaic/framework/tools/git/pr-review.sh` use tea v0.11.1's supported `tea comment` command and report success only after provider read-back verifies the created comment against the intended repository, PR, and exact body.
-
-## Plan
-
-1. Add and commit a failing shell regression harness before production changes.
-2. Verify RED against the nonexistent `tea pr comment` fallback false-positive.
-3. Implement the minimal supported write plus ID-based provider read-back.
-4. Document that wrapper write output is not durable provenance until read-back succeeds.
-5. Run focused regression tests, touched-package tests, and repository quality gates.
-6. Run independent review, remediate findings, queue-guard, push, and open a PR. Stop at PR-open for coordinator review/merge.
-
-## Progress checkpoints
-
-- [x] RED regression committed and reported to mosaic-100 (`3585cdfa`)
-- [x] Initial minimal fix implemented (`c3a83833`)
-- [x] Focused and package tests green after dependency build
-- [ ] Independent review passed
-- [ ] PR opened and reported with exact head SHA
-
-## Tests run
-
-- RED: `bash packages/mosaic/framework/tools/git/test-pr-review-gitea-comment.sh` failed against the old path because `tea pr comment` fell back and the wrapper returned success.
-- GREEN: all `packages/mosaic/framework/tools/git/test-*.sh` harnesses passed.
-- `pnpm build` passed (23/23 tasks).
-- `pnpm --filter @mosaicstack/mosaic test` passed (67 files, 1278 tests).
-- Package and root typecheck/lint/format gates passed.
-- Codex security review found no issues.
-- Codex code review requested changes: tea v0.11.1 has no supported `tea api`; replace read-back with the existing host-scoped authenticated curl/API path when work resumes.
-
-## Risks / blockers
-
-- **Coordination hold:** Mos directed #812 to remain parked until in-flight #789 reaches terminal state because #789 actively uses `pr-review.sh`. Branch must be pushed but no PR opened.
-- Initial implementation's `tea api` read-back is not compatible with tea v0.11.1 and must be remediated before PR.
-- Tea output shape must provide a created comment ID in JSON; failure to parse must fail closed.
-- Read-back must not accept a pre-existing comment with the same body.
-- Existing approve/request-changes behavior must remain unchanged and covered.
-
-## Final verification evidence
-
-Parked pending #789 terminal. No PR opened.

packages/mosaic/framework/tools/git/README.md

@@ -1,9 +0,0 @@
-# Git provider wrappers
-
-These scripts provide host-aware GitHub and Gitea issue, pull-request, milestone, and CI operations.
-
-## Durable review provenance
-
-A successful provider write command—or a wrapper message based only on that command's exit code—is **not** durable review provenance. Review comments count as durable provenance only after the wrapper reads the created provider record back and verifies that it belongs to the intended repository and pull request and contains the exact submitted body (or verifies the provider-returned record ID).
-
-`pr-review.sh` therefore fails closed when a Gitea comment cannot be written, its created comment ID cannot be identified, or provider read-back does not match. It reports comment success only after that read-back verification passes.

packages/mosaic/framework/tools/git/pr-review.sh

@@ -101,58 +101,8 @@ elif [[ "$PLATFORM" == "gitea" ]]; then
                 echo "Error: Comment required"
                 exit 1
             fi
-
+            tea pr comment "$PR_NUMBER" "$COMMENT" $(get_gitea_repo_args)
-            repo=$(get_repo_slug)
+            echo "Added comment to Gitea PR #$PR_NUMBER"
-            host=$(get_remote_host)
-            login=$(get_gitea_login_for_host "$host")
-
-            comment_json=$(tea comment "$PR_NUMBER" "$COMMENT" --login "$login" --repo "$repo" --output json)
-            comment_id=$(printf '%s' "$comment_json" | python3 -c '
-import json
-import sys
-
-try:
-    comment = json.load(sys.stdin)
-    if isinstance(comment, list) and len(comment) == 1:
-        comment = comment[0]
-    comment_id = comment.get("id") if isinstance(comment, dict) else None
-    if not isinstance(comment_id, int) or comment_id <= 0:
-        raise ValueError("missing positive comment id")
-except (json.JSONDecodeError, ValueError) as error:
-    print(f"Error: could not identify created Gitea comment: {error}", file=sys.stderr)
-    raise SystemExit(1)
-print(comment_id)
-')
-
-            readback_json=$(tea api --login "$login" "/repos/$repo/issues/comments/$comment_id")
-            EXPECTED_COMMENT_ID="$comment_id" EXPECTED_COMMENT_BODY="$COMMENT" EXPECTED_REPO="$repo" EXPECTED_PR_NUMBER="$PR_NUMBER" \
-                python3 -c '
-import json
-import os
-import sys
-from urllib.parse import urlparse
-
-try:
-    comment = json.load(sys.stdin)
-    expected_id = int(os.environ["EXPECTED_COMMENT_ID"])
-    expected_body = os.environ["EXPECTED_COMMENT_BODY"]
-    expected_repo = os.environ["EXPECTED_REPO"]
-    expected_pr = os.environ["EXPECTED_PR_NUMBER"]
-    issue_url = comment.get("issue_url", "") if isinstance(comment, dict) else ""
-    issue_path = urlparse(issue_url).path.rstrip("/")
-    expected_suffix = f"/repos/{expected_repo}/issues/{expected_pr}"
-    if comment.get("id") != expected_id:
-        raise ValueError("comment id mismatch")
-    if comment.get("body") != expected_body:
-        raise ValueError("comment body mismatch")
-    if not issue_path.endswith(expected_suffix):
-        raise ValueError("repository or PR mismatch")
-except (json.JSONDecodeError, KeyError, TypeError, ValueError) as error:
-    print(f"Error: Gitea comment persistence verification failed: {error}", file=sys.stderr)
-    raise SystemExit(1)
-' <<< "$readback_json"
-
-            echo "Added and verified comment on Gitea PR #$PR_NUMBER (comment ID $comment_id)"
             ;;
         *)
             echo "Error: Unknown action: $ACTION"

packages/mosaic/framework/tools/git/test-pr-review-gitea-comment.sh

@@ -1,144 +0,0 @@
-#!/usr/bin/env bash
-# Regression harness for durable Gitea PR review comments (#812).
-
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-WORK_DIR="${MOSAIC_TEST_WORK_DIR:-$PWD/.mosaic-test-work/pr-review-gitea-comment}"
-REPO_DIR="$WORK_DIR/repo"
-BIN_DIR="$WORK_DIR/bin"
-LOG_FILE="$WORK_DIR/tea.log"
-OUTPUT_FILE="$WORK_DIR/output.log"
-
-cleanup() {
-    rm -rf "$WORK_DIR"
-}
-trap cleanup EXIT
-
-mkdir -p "$REPO_DIR" "$BIN_DIR"
-git -C "$REPO_DIR" init -q
-git -C "$REPO_DIR" remote add origin https://git.mosaicstack.dev/mosaicstack/stack.git
-
-cat > "$BIN_DIR/tea" <<'SH'
-#!/usr/bin/env bash
-set -euo pipefail
-
-printf '%s\n' "$*" >> "$PR_REVIEW_TEST_LOG"
-
-if [[ "$*" == "login list --output json" ]]; then
-    printf '%s\n' '[{"name":"mosaicstack","url":"https://git.mosaicstack.dev"}]'
-    exit 0
-fi
-
-case "${PR_REVIEW_TEST_MODE:-}" in
-    approve)
-        [[ "$*" == "pr approve 123 --repo mosaicstack/stack --login mosaicstack" ]] || exit 90
-        ;;
-    request-changes)
-        [[ "$*" == "pr reject 123 --repo mosaicstack/stack --login mosaicstack --comment changes-required" ]] || exit 91
-        ;;
-    legacy-fallback)
-        if [[ "$*" == pr\ comment* ]]; then
-            # tea v0.11.1 treats the nonexistent subcommand as `tea pr list` and exits 0.
-            printf '%s\n' 'INDEX TITLE STATE'
-            exit 0
-        fi
-        if [[ "$*" == comment* ]]; then
-            printf '%s\n' 'supported comment write deliberately unavailable' >&2
-            exit 23
-        fi
-        exit 92
-        ;;
-    comment-success|readback-failure)
-        if [[ "$*" == "comment 123 durable-body --login mosaicstack --repo mosaicstack/stack --output json" ]]; then
-            printf '%s\n' '{"id":456,"body":"durable-body"}'
-            exit 0
-        fi
-        if [[ "$*" == "api --login mosaicstack /repos/mosaicstack/stack/issues/comments/456" ]]; then
-            if [[ "$PR_REVIEW_TEST_MODE" == "readback-failure" ]]; then
-                printf '%s\n' '{"id":456,"body":"different-body","issue_url":"https://git.mosaicstack.dev/api/v1/repos/mosaicstack/stack/issues/123"}'
-            else
-                printf '%s\n' '{"id":456,"body":"durable-body","issue_url":"https://git.mosaicstack.dev/api/v1/repos/mosaicstack/stack/issues/123"}'
-            fi
-            exit 0
-        fi
-        if [[ "$*" == pr\ comment* ]]; then
-            # Reproduce the old false-positive fallback if the wrapper regresses.
-            printf '%s\n' 'INDEX TITLE STATE'
-            exit 0
-        fi
-        exit 93
-        ;;
-    write-failure)
-        if [[ "$*" == "comment 123 durable-body --login mosaicstack --repo mosaicstack/stack --output json" ]]; then
-            printf '%s\n' 'provider rejected comment' >&2
-            exit 24
-        fi
-        exit 94
-        ;;
-    *)
-        exit 95
-        ;;
-esac
-SH
-chmod +x "$BIN_DIR/tea"
-
-run_review() {
-    local mode="$1" action="$2" comment="${3:-}"
-    : > "$LOG_FILE"
-    : > "$OUTPUT_FILE"
-    (
-        cd "$REPO_DIR"
-        PATH="$BIN_DIR:$PATH" \
-        PR_REVIEW_TEST_LOG="$LOG_FILE" \
-        PR_REVIEW_TEST_MODE="$mode" \
-        "$SCRIPT_DIR/pr-review.sh" -n 123 -a "$action" ${comment:+-c "$comment"}
-    ) > "$OUTPUT_FILE" 2>&1
-}
-
-run_review approve approve
-grep -q '^pr approve 123 --repo mosaicstack/stack --login mosaicstack$' "$LOG_FILE"
-grep -q 'Approved Gitea PR #123' "$OUTPUT_FILE"
-
-run_review request-changes request-changes changes-required
-grep -q '^pr reject 123 --repo mosaicstack/stack --login mosaicstack --comment changes-required$' "$LOG_FILE"
-grep -q 'Requested changes on Gitea PR #123' "$OUTPUT_FILE"
-
-if run_review legacy-fallback comment durable-body; then
-    echo "The old nonexistent tea pr comment fallback returned success" >&2
-    cat "$OUTPUT_FILE" >&2
-    exit 1
-fi
-if grep -q '^pr comment ' "$LOG_FILE"; then
-    echo "Wrapper invoked unsupported tea pr comment" >&2
-    exit 1
-fi
-if grep -q 'Added comment to Gitea PR' "$OUTPUT_FILE"; then
-    echo "Wrapper reported success without durable persistence" >&2
-    exit 1
-fi
-
-run_review comment-success comment durable-body
-grep -q '^comment 123 durable-body --login mosaicstack --repo mosaicstack/stack --output json$' "$LOG_FILE"
-grep -q '^api --login mosaicstack /repos/mosaicstack/stack/issues/comments/456$' "$LOG_FILE"
-grep -q 'Added and verified comment on Gitea PR #123' "$OUTPUT_FILE"
-
-if run_review write-failure comment durable-body; then
-    echo "Expected provider write failure to return nonzero" >&2
-    exit 1
-fi
-if grep -q 'Added and verified comment' "$OUTPUT_FILE"; then
-    echo "Write failure reported durable success" >&2
-    exit 1
-fi
-
-if run_review readback-failure comment durable-body; then
-    echo "Expected mismatched provider read-back to return nonzero" >&2
-    exit 1
-fi
-if grep -q 'Added and verified comment' "$OUTPUT_FILE"; then
-    echo "Read-back mismatch reported durable success" >&2
-    exit 1
-fi
-
-echo "pr-review.sh durable Gitea comment regression passed"