Compare commits

..
Author SHA1 Message Date
fred 5c35a250de test(fleet): name what the pane-boundary case's binary check rides on (#1241)
ci/woodpecker/pr/ci Pipeline was successful
Review finding from scooby. This case does not use run_start, so
install_pane_binaries' symlinks land under a home its launcher never consults
(HOME is the trusted parent here). It resolves mosaic and pi through
MOSAIC_RUNTIME_BIN=$FAKE_BIN instead. Valid path, valid green — and a trap for
anyone who later drops that env var believing the symlinks cover it, which
would break the #1241 binary check rather than exercise it.

Comment only; no behavior change. Harness rc=0.

Refs #1241.
2026-08-16 00:21:13 -05:00
fred 10a1f82031 test(fleet): cover the pane-pid-unresolved branch this PR shipped (#1241)
ci/woodpecker/pr/ci Pipeline was canceled
Review finding from scooby: this PR added a failure branch the harness
structurally could not reach. The fake tmux answered `has-session` only for
`=_holder:0.0`, so every non-holder agent landed in the session-is-gone branch
no matter what — the `elif` (tmux still reports the session, no pane PID after
the retries) had zero coverage and no way to get any.

That is the same shape as the bug this PR exists to fix, one layer down: a code
path shipped green where the gate that should measure it cannot. Less severe,
because the branch fails closed at exit 69 rather than reporting success — but
"the harness can't reach it" is the sentence that precedes the next silent
regression, so it gets closed here rather than filed.

`MOSAIC_TEST_HELD_SESSIONS` lets a case name targets the shim should also
answer for. It answers them only AFTER `new-session`, and that detail is the
whole trick: the launcher asks `has-session` about the same name twice — once
at line 255 where a yes means "already running, exit 0", and once at 417 where
a yes means "the session survived". A shim answering yes to both short-circuits
at the first and never reaches the branch under test. It would have looked like
coverage while measuring the idempotency path.

Both failure modes were measured, not reasoned about:
- toggle absent (the old shim): `code=pane-did-not-survive` — the case lands on
  the wrong branch, which is exactly the unreachability being reported.
- toggle answering unconditionally: launcher exits 0 via the idempotency
  short-circuit — "launcher reported success over a session with no resolvable
  pane PID".
- toggle gated on new-session: `code=pane-pid-unresolved`, exit 69.

The case also asserts the diagnostic is not `pane-did-not-survive` and does not
mention the heartbeat, so the two pane faults cannot collapse into one message.

Gates: bash -n · launcher harness rc=0 · test-fleet-units.sh (real tmux) rc=0 ·
fleet specs 342 passed.

Refs #1241.
2026-08-16 00:19:25 -05:00
fred 61a907a12f fix(fleet): fail the agent launcher when the pane cannot survive (#1241)
ci/woodpecker/pr/ci Pipeline was successful
`mosaic fleet start` returned 0 over three dead panes. The launcher knew,
and said the wrong thing at the wrong severity to the wrong layer.

The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared
environment. When that binary is absent the pane dies in under a second,
tmux destroys the session, and the diagnostic goes with it. The launcher
then found no PANE_PID, printed a WARNING about the *heartbeat sidecar*,
and exited 0 — so systemd logged "Finished ... successfully" and
`fleet start` reported success. `fleet ps` was the only component telling
the truth.

Two changes, both in start-agent-session.sh:

1. Before any effect, resolve `mosaic` and the roster's runtime against
   PANE_PATH — the pane's own view of the path, not the launcher's.
   `mosaic yolo <runtime>` calls checkRuntime(runtime) and looks for a
   binary named exactly like the runtime, so this asks the same question
   the pane will ask a moment later, while an operator can still see the
   answer. Absent binary -> exit 69, code=missing-binary, no session
   created.

2. Replace the dead-pane WARNING+exit-0. An absent session one second
   after new-session is a runtime that died on startup, not a heartbeat
   problem -> exit 69, code=pane-did-not-survive, with the command to run
   by hand to see why. A present session with no pane PID after five
   attempts -> code=pane-pid-unresolved. Neither branch kills the
   session; destroying a possibly-live pane on a guess is worse than
   leaving it for inspection.

Exit 69 (EX_UNAVAILABLE) is deliberate: the 64s already in this file mean
the projection was bad, and here the data is fine and the host is not
ready. Callers separate the cases by `code=`, the same way fail_env's
codes share 64.

This propagates for free. `fleet start` calls runChecked() for the holder
and each agent, and runChecked throws on non-zero, so layers 4 and 5 stop
lying without a TypeScript change. Two adjacent defects are left for a
follow-up issue rather than widened into this diff: the per-agent loop
aborts on the first failure instead of attempting all and reporting an
aggregate, and runChecked's bare throw surfaces the launcher's message
under a Node unhandled-rejection stack trace because program.parse() is
synchronous.

Tests:

- test-start-agent-session.sh gains three cases: `mosaic` absent from the
  pane path, the runtime absent from the pane path, and a pane that does
  not survive. Each was verified individually red against the unmodified
  origin/next launcher.
- The two cases asserting a valid launch now supply a pane PID. Until now
  the suite's one success path was itself a dead pane the launcher
  reported as fine.
- The harness fakes `npm` so PANE_PATH stops depending on whatever the
  host has installed, and fails loudly if the host provides `mosaic` or
  `pi` in the system path, where the missing-binary cases would not be
  measurable at all.
- test-fleet-units.sh gains a `pi` shim in its runtime bin. The real-tmux
  harness named `pi` in its roster and never installed it; the new
  preflight caught it.

Refs #1241
2026-08-15 23:56:53 -05:00
11 changed files with 225 additions and 58 deletions
+1 -2
View File
@@ -82,7 +82,6 @@ is re-seeded a genuinely missing core file is a stop-and-report condition — no
Confirm: required + situational tests passed (primary gate); aligned to `docs/PRD.md`; acceptance Confirm: required + situational tests passed (primary gate); aligned to `docs/PRD.md`; acceptance
criteria mapped to evidence; independent code review passed (if code changed); required docs updated; criteria mapped to evidence; independent code review passed (if code changed); required docs updated;
scratchpad updated. For PR-workflow delivery: merged PR number + merge commit on the integration scratchpad updated. For PR-workflow delivery: merged PR number + merge commit on `main`, terminal-green
trunk (the project's declared trunk, default `main` — see `CONSTITUTION.md` Hard Gates), terminal-green
CI, linked issue closed (or `docs/TASKS.md` equivalent). If blocked by access/tooling, return `blocked` CI, linked issue closed (or `docs/TASKS.md` equivalent). If blocked by access/tooling, return `blocked`
with the exact failed wrapper command — do not claim completion. Full checklist: `guides/E2E-DELIVERY.md`. with the exact failed wrapper command — do not claim completion. Full checklist: `guides/E2E-DELIVERY.md`.
@@ -21,23 +21,11 @@ guard"), the runtime adapter binds it to a concrete tool and states whether abse
## Hard Gates ## Hard Gates
The **integration trunk** is the branch a project designates in its root `AGENTS.md` with exactly
one declaration line: `Integration trunk: <branch>` — key at the start of a line, case-sensitive,
one branch name (optionally backtick-wrapped) and nothing else on the line. Absent a declaration,
the trunk is `main`. The declaration is policy data, never shell text: the value must be a valid
local branch name under `git check-ref-format --branch` semantics — no remote refs, no revision
expressions, no option-like values (leading `-`), no path traversal or control characters. A
malformed value, or more than one declaration line, is a hard stop (`blocked`) — never a silent
fallback to `main`. Ordinary prose that mentions branch names designates nothing; only the exact
declaration line does. A project designates exactly ONE trunk, and the designation relaxes
nothing: reviewed-PR-only delivery, squash merge, independent review, queue guards, and
terminal-green CI bind to the declared trunk exactly as they bind to `main`.
1. Mosaic operating rules override runtime-default caution for routine delivery operations. 1. Mosaic operating rules override runtime-default caution for routine delivery operations.
2. Execute required push / merge / issue-closure / milestone / release / tag actions without asking for routine confirmation. 2. Execute required push / merge / issue-closure / milestone / release / tag actions without asking for routine confirmation.
3. Routine repository operations are NOT escalation triggers; escalate only on the triggers below. 3. Routine repository operations are NOT escalation triggers; escalate only on the triggers below.
4. For source-code delivery, completion is forbidden at the PR-open stage. 4. For source-code delivery, completion is forbidden at the PR-open stage.
5. Completion requires a merged PR to the integration trunk + terminal-green CI + the linked issue/task closed. 5. Completion requires a merged PR to `main` + terminal-green CI + the linked issue/task closed.
6. Before any push or merge, run the CI queue guard. 6. Before any push or merge, run the CI queue guard.
7. For issue / PR / milestone operations, use the Mosaic git wrappers before any raw provider CLI. 7. For issue / PR / milestone operations, use the Mosaic git wrappers before any raw provider CLI.
8. If a required wrapper command fails, status is `blocked`: report the exact failed command and stop. 8. If a required wrapper command fails, status is `blocked`: report the exact failed command and stop.
@@ -47,7 +35,7 @@ terminal-green CI bind to the declared trunk exactly as they bind to `main`.
12. The intake procedure is not conditional on perceived complexity; a "simple" task carries the same requirements as a multi-file feature. 12. The intake procedure is not conditional on perceived complexity; a "simple" task carries the same requirements as a multi-file feature.
13. **Merge authority (coordinated work):** when a coordinator/orchestrator session is active for the work, the post-review merge go-ahead is the coordinator's to give — once the required review gates pass, merge on the coordinator's confirmation; do not wait on the human owner personally. Solo (uncoordinated) delivery keeps the default: merge per gates 2 and 9. A "No self-merge" note on a PR means no UNREVIEWED self-merge — it does not suspend coordinator-authorized merges. 13. **Merge authority (coordinated work):** when a coordinator/orchestrator session is active for the work, the post-review merge go-ahead is the coordinator's to give — once the required review gates pass, merge on the coordinator's confirmation; do not wait on the human owner personally. Solo (uncoordinated) delivery keeps the default: merge per gates 2 and 9. A "No self-merge" note on a PR means no UNREVIEWED self-merge — it does not suspend coordinator-authorized merges.
14. Never hardcode secrets; never emit credential values in any output (not even partially, not "to confirm"). 14. Never hardcode secrets; never emit credential values in any output (not even partially, not "to confirm").
15. Trunk-based git only: branch from the integration trunk, merge via a reviewed PR (squash), never push directly to the trunk. 15. Trunk-based git only: branch from `main`, merge via a reviewed PR (squash), never push directly to `main`.
16. If you modify source code, an independent review (author ≠ reviewer) must pass before completion. 16. If you modify source code, an independent review (author ≠ reviewer) must pass before completion.
## Integrity (quality gates are never bypassed) ## Integrity (quality gates are never bypassed)
+10 -11
View File
@@ -12,7 +12,7 @@ This guide covers how to bootstrap a project so AI agents (Claude, Codex, etc.)
4. Issue tracking is consistent across projects 4. Issue tracking is consistent across projects
5. Documentation standards and API contracts are enforced from day one 5. Documentation standards and API contracts are enforced from day one
6. PRD requirements are established before coding begins 6. PRD requirements are established before coding begins
7. Branching/merging is consistent: branch -> integration trunk (default `main`) via PR with squash-only merges 7. Branching/merging is consistent: `branch -> main` via PR with squash-only merges
8. Steered-autonomy execution is enabled so agents can run end-to-end with escalation-only human intervention 8. Steered-autonomy execution is enabled so agents can run end-to-end with escalation-only human intervention
## Agent Host Prerequisites ## Agent Host Prerequisites
@@ -206,7 +206,7 @@ Every runtime context file should contain:
6. **Issue tracking** — Issue and commit conventions 6. **Issue tracking** — Issue and commit conventions
7. **Code review** — Required review process 7. **Code review** — Required review process
8. **Runtime notes** — Runtime-specific behavior references 8. **Runtime notes** — Runtime-specific behavior references
9. **Branch and merge policy** — Trunk workflow (branch -> integration trunk via PR, squash-only) 9. **Branch and merge policy** — Trunk workflow (`branch -> main` via PR, squash-only)
10. **Autonomy and escalation policy** — Agent owns coding/review/PR/release/deploy lifecycle 10. **Autonomy and escalation policy** — Agent owns coding/review/PR/release/deploy lifecycle
--- ---
@@ -288,16 +288,15 @@ Reserve `0.1.0` for the MVP release milestone.
--- ---
## Step 5b: Configure Trunk Branch Protection (Hard Rule) ## Step 5b: Configure Main Branch Protection (Hard Rule)
Apply equivalent settings in Gitea, GitHub, or GitLab, targeting the project's integration trunk Apply equivalent settings in Gitea, GitHub, or GitLab:
(the branch its root `AGENTS.md` declares; default `main` — see `CONSTITUTION.md` Hard Gates):
1. Protect the integration trunk from direct pushes. 1. Protect `main` from direct pushes.
2. Require pull requests to merge into the integration trunk. 2. Require pull requests to merge into `main`.
3. Require required CI/status checks to pass before merge. 3. Require required CI/status checks to pass before merge.
4. Require code review approval before merge. 4. Require code review approval before merge.
5. Allow **squash merge only** for PRs into the integration trunk (disable merge commits and rebase merges for it). 5. Allow **squash merge only** for PRs into `main` (disable merge commits and rebase merges for `main`).
This enforces one merge strategy across human and agent workflows. This enforces one merge strategy across human and agent workflows.
@@ -514,9 +513,9 @@ After bootstrapping, verify:
- [ ] Git labels created (epic, feature, bug, task, etc.) - [ ] Git labels created (epic, feature, bug, task, etc.)
- [ ] Initial pre-MVP milestone created (0.0.1) - [ ] Initial pre-MVP milestone created (0.0.1)
- [ ] MVP milestone reserved for release (0.1.0) - [ ] MVP milestone reserved for release (0.1.0)
- [ ] The integration trunk is protected from direct pushes - [ ] `main` is protected from direct pushes
- [ ] PRs into the integration trunk are required - [ ] PRs into `main` are required
- [ ] Merge method for the integration trunk is squash-only - [ ] Merge method for `main` is squash-only
- [ ] Quality gates run successfully - [ ] Quality gates run successfully
- [ ] `.env.example` exists (if project uses env vars) - [ ] `.env.example` exists (if project uses env vars)
- [ ] CI/CD pipeline configured (if using Woodpecker/GitHub Actions) - [ ] CI/CD pipeline configured (if using Woodpecker/GitHub Actions)
@@ -4,11 +4,6 @@
## Overview ## Overview
> **Integration trunk:** the YAML examples in this guide use the default integration trunk `main`
> in branch conditions and version rules. A project that declares a different trunk in its root
> `AGENTS.md` (see `CONSTITUTION.md` Hard Gates) substitutes its declared trunk wherever `main`
> appears as the trunk branch.
This guide covers the canonical CI/CD pattern used across projects. The pipeline runs in Woodpecker CI and follows this flow: This guide covers the canonical CI/CD pattern used across projects. The pipeline runs in Woodpecker CI and follows this flow:
``` ```
@@ -870,7 +865,7 @@ steps:
```yaml ```yaml
image: git.example.com/org/service@${IMAGE_DIGEST} image: git.example.com/org/service@${IMAGE_DIGEST}
``` ```
7. **Test on a short-lived non-trunk branch first** — open a PR and verify quality gates before merging to the integration trunk 7. **Test on a short-lived non-main branch first** — open a PR and verify quality gates before merging to `main`
8. **Verify images appear** in Gitea Packages tab after successful pipeline 8. **Verify images appear** in Gitea Packages tab after successful pipeline
## Terminal-Green Full-Step Contract ## Terminal-Green Full-Step Contract
@@ -911,7 +906,7 @@ For source-code delivery, completion is not allowed at "PR opened" stage.
Required sequence: Required sequence:
1. Merge PR to the integration trunk (squash) via Mosaic wrapper. 1. Merge PR to `main` (squash) via Mosaic wrapper.
2. Monitor CI to terminal status: 2. Monitor CI to terminal status:
```bash ```bash
~/.config/mosaic/tools/git/pr-ci-wait.sh -n <PR_NUMBER> ~/.config/mosaic/tools/git/pr-ci-wait.sh -n <PR_NUMBER>
@@ -1117,5 +1112,5 @@ If a project currently uses Verdaccio (e.g., U-Connect at `npm.uscllc.net`), fol
### Pipeline runs Docker builds on pull requests ### Pipeline runs Docker builds on pull requests
- Verify `when` clause on Docker build steps restricts to the integration trunk (`branch: [main]` by default) - Verify `when` clause on Docker build steps restricts to `branch: [main]`
- Pull requests should only run quality gates, not build/push images - Pull requests should only run quality gates, not build/push images
@@ -10,10 +10,9 @@ If implementation diverges from `docs/PRD.md` or `docs/PRD.json` without PRD upd
Merge strategy enforcement (HARD RULE): Merge strategy enforcement (HARD RULE):
- The integration trunk is the branch the project's root `AGENTS.md` declares (default: `main`) — see `CONSTITUTION.md` Hard Gates. - PR target for delivery is `main`.
- PR target for delivery is the integration trunk. - Direct pushes to `main` are prohibited.
- Direct pushes to the integration trunk are prohibited. - Merge to `main` MUST be squash-only.
- Merge to the integration trunk MUST be squash-only.
- Use `~/.config/mosaic/tools/git/pr-merge.sh -n {PR_NUMBER} -m squash --expect-head {approved_full_sha}` (or PowerShell equivalent). - Use `~/.config/mosaic/tools/git/pr-merge.sh -n {PR_NUMBER} -m squash --expect-head {approved_full_sha}` (or PowerShell equivalent).
## Review Checklist ## Review Checklist
@@ -115,8 +114,8 @@ Use `~/.config/mosaic/templates/docs/DOCUMENTATION-CHECKLIST.md` whenever code/A
# List the issue being addressed # List the issue being addressed
~/.config/mosaic/tools/git/issue-list.sh -i {issue-number} ~/.config/mosaic/tools/git/issue-list.sh -i {issue-number}
# View the changes (diff against the integration trunk; default: main) # View the changes
git diff {integration_trunk}...HEAD git diff main...HEAD
``` ```
### Providing Feedback ### Providing Feedback
@@ -152,4 +151,4 @@ This pattern appears in 3 places. A shared helper would reduce duplication.
2. If changes requested, assign back to author 2. If changes requested, assign back to author
3. If approved, note approval in issue comments 3. If approved, note approval in issue comments
4. For merges, ensure CI passes first 4. For merges, ensure CI passes first
5. Merge PR to the integration trunk with squash strategy only 5. Merge PR to `main` with squash strategy only
@@ -78,7 +78,7 @@ For implementation work, you MUST run this cycle in order:
7. `commit` - commit only when the logical unit passes tests and review. 7. `commit` - commit only when the logical unit passes tests and review.
8. `pre-push queue guard` - before pushing, wait for running/queued project pipelines to clear: `~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose push`. 8. `pre-push queue guard` - before pushing, wait for running/queued project pipelines to clear: `~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose push`.
9. `push` - push immediately after queue guard passes. 9. `push` - push immediately after queue guard passes.
10. `PR integration` - if external git provider is available, create/update PR to the integration trunk (the project's declared trunk, default `main`) and merge with required strategy via Mosaic wrappers. 10. `PR integration` - if external git provider is available, create/update PR to `main` and merge with required strategy via Mosaic wrappers.
11. `pre-merge queue guard` - before merging PR, wait for running/queued project pipelines on the exact PR head to clear: `~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose merge -B <PR_HEAD_BRANCH> -R <PR_HEAD_OWNER/REPO> --sha <PR_HEAD_FULL_SHA>`. 11. `pre-merge queue guard` - before merging PR, wait for running/queued project pipelines on the exact PR head to clear: `~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose merge -B <PR_HEAD_BRANCH> -R <PR_HEAD_OWNER/REPO> --sha <PR_HEAD_FULL_SHA>`.
12. `CI/pipeline verification` - wait for terminal CI status and require green before completion (`~/.config/mosaic/tools/git/pr-ci-wait.sh` for PR-based workflow). 12. `CI/pipeline verification` - wait for terminal CI status and require green before completion (`~/.config/mosaic/tools/git/pr-ci-wait.sh` for PR-based workflow).
13. `issue closure` - close linked external issue (or close internal `docs/TASKS.md` task ref when provider is unavailable). 13. `issue closure` - close linked external issue (or close internal `docs/TASKS.md` task ref when provider is unavailable).
@@ -199,7 +199,7 @@ Before running this checklist, pause and self-interrogate: did I fulfill the use
10. No unresolved blocker hidden. 10. No unresolved blocker hidden.
11. If deployment is in scope, deployment target, release version, and post-deploy verification evidence are documented. 11. If deployment is in scope, deployment target, release version, and post-deploy verification evidence are documented.
12. `docs/TASKS.md` status and issue/internal references are updated to match delivered work. 12. `docs/TASKS.md` status and issue/internal references are updated to match delivered work.
13. If source code changed and external provider is available: PR merged to the integration trunk (squash), with merge evidence recorded. 13. If source code changed and external provider is available: PR merged to `main` (squash), with merge evidence recorded.
14. CI/pipeline status is terminal green for the merged PR/head commit. 14. CI/pipeline status is terminal green for the merged PR/head commit.
15. Linked external issue is closed (or internal task ref is closed when no provider exists). 15. Linked external issue is closed (or internal task ref is closed when no provider exists).
16. If any of items 13-15 fail due access/tooling, report `blocked` with exact failed wrapper command and do not claim completion. 16. If any of items 13-15 fail due access/tooling, report `blocked` with exact failed wrapper command and do not claim completion.
@@ -253,7 +253,7 @@ status → mission → run → repeat
- [ ] All milestone tasks in TASKS.md are `done` - [ ] All milestone tasks in TASKS.md are `done`
- [ ] CI/pipeline green - [ ] CI/pipeline green
- [ ] PR merged to the integration trunk - [ ] PR merged to `main`
- [ ] Issues closed - [ ] Issues closed
- [ ] Update manifest: milestone status → completed - [ ] Update manifest: milestone status → completed
- [ ] Update scratchpad: session log entry - [ ] Update scratchpad: session log entry
@@ -15,7 +15,7 @@ mosaic claude -p "Read ~/.config/mosaic/skills/nestjs-best-practices/SKILL.md th
- You MUST keep the TASKS.md file updated with agent and tasks statuses. - You MUST keep the TASKS.md file updated with agent and tasks statuses.
- You MUST keep `docs/` root clean. Reports and working artifacts MUST be stored in scoped folders (`docs/reports/`, `docs/tasks/`, `docs/releases/`, `docs/scratchpads/`). - You MUST keep `docs/` root clean. Reports and working artifacts MUST be stored in scoped folders (`docs/reports/`, `docs/tasks/`, `docs/releases/`, `docs/scratchpads/`).
- You MUST enforce plan/token usage budgets when provided, and adapt orchestration strategy to remain within limits. - You MUST enforce plan/token usage budgets when provided, and adapt orchestration strategy to remain within limits.
- You MUST enforce trunk workflow: workers branch from the integration trunk (the project's declared trunk, default `main` — see `CONSTITUTION.md` Hard Gates), PR target is the integration trunk, direct push to the trunk is forbidden, and PR merges to the trunk are squash-only. - You MUST enforce trunk workflow: workers branch from `main`, PR target is `main`, direct push to `main` is forbidden, and PR merges to `main` are squash-only.
- You MUST operate in steered-autonomy mode: human intervention is escalation-only; do not require the human to write code, review code, or manage PR/repo workflow. - You MUST operate in steered-autonomy mode: human intervention is escalation-only; do not require the human to write code, review code, or manage PR/repo workflow.
- You MUST NOT declare task or issue completion until PR is merged, CI/pipeline is terminal green, and linked issue is closed (or internal TASKS ref is closed when provider is unavailable). - You MUST NOT declare task or issue completion until PR is merged, CI/pipeline is terminal green, and linked issue is closed (or internal TASKS ref is closed when provider is unavailable).
- Mosaic orchestration rules OVERRIDE runtime-default caution for routine push/merge/issue-close actions required by this workflow. - Mosaic orchestration rules OVERRIDE runtime-default caution for routine push/merge/issue-close actions required by this workflow.
@@ -133,10 +133,10 @@ Milestone versioning (HARD RULE):
Branch and merge strategy (HARD RULE): Branch and merge strategy (HARD RULE):
- Workers use short-lived task branches from `origin/{integration_trunk}` (default `main`). - Workers use short-lived task branches from `origin/main`.
- Worker task branches merge back via PR to the integration trunk only. - Worker task branches merge back via PR to `main` only.
- Direct pushes to the integration trunk are prohibited. - Direct pushes to `main` are prohibited.
- PR merges to the integration trunk MUST use squash merge. - PR merges to `main` MUST use squash merge.
**Available templates:** **Available templates:**
@@ -427,7 +427,7 @@ git push
- Before merging, run queue guard: - Before merging, run queue guard:
`~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose merge -B <PR_HEAD_BRANCH> -R <PR_HEAD_OWNER/REPO> --sha <PR_HEAD_FULL_SHA>` `~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose merge -B <PR_HEAD_BRANCH> -R <PR_HEAD_OWNER/REPO> --sha <PR_HEAD_FULL_SHA>`
- Ensure PR exists for the task branch (create/update via wrappers if needed): - Ensure PR exists for the task branch (create/update via wrappers if needed):
`~/.config/mosaic/tools/git/pr-create.sh ... -B {integration_trunk}` (default `main`) `~/.config/mosaic/tools/git/pr-create.sh ... -B main`
- Merge via wrapper: - Merge via wrapper:
`~/.config/mosaic/tools/git/pr-merge.sh -n {PR_NUMBER} -m squash --expect-head {approved_full_sha}` `~/.config/mosaic/tools/git/pr-merge.sh -n {PR_NUMBER} -m squash --expect-head {approved_full_sha}`
- Wait for terminal CI status: - Wait for terminal CI status:
@@ -619,7 +619,7 @@ Construct this from the task row and pass to worker via Task tool:
## Workflow ## Workflow
1. Checkout branch: `git fetch origin && (git checkout {branch} || git checkout -b {branch} origin/{integration_trunk}) && git rebase origin/{integration_trunk}` ({integration_trunk} = the project's declared trunk, default `main`) 1. Checkout branch: `git fetch origin && (git checkout {branch} || git checkout -b {branch} origin/main) && git rebase origin/main`
2. Read `docs/PRD.md` or `docs/PRD.json` and align implementation with PRD requirements 2. Read `docs/PRD.md` or `docs/PRD.json` and align implementation with PRD requirements
3. Read the finding details from the report 3. Read the finding details from the report
4. Implement the fix following existing code patterns 4. Implement the fix following existing code patterns
@@ -637,7 +637,7 @@ Do NOT leave lint warnings or errors for someone else to clean up. 6. Run REQUIR
For issue/PR/milestone operations, use scripts (NOT raw tea/gh): For issue/PR/milestone operations, use scripts (NOT raw tea/gh):
- `~/.config/mosaic/tools/git/issue-view.sh -i {N}` - `~/.config/mosaic/tools/git/issue-view.sh -i {N}`
- `~/.config/mosaic/tools/git/pr-create.sh -t "Title" -b "Desc" -B {integration_trunk}` - `~/.config/mosaic/tools/git/pr-create.sh -t "Title" -b "Desc" -B main`
- Push: `~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose push -B {task_branch}` - Push: `~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose push -B {task_branch}`
- Merge: `~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose merge -B {pr_head_branch} -R {pr_head_owner/repo} --sha {pr_head_full_sha}` - Merge: `~/.config/mosaic/tools/git/ci-queue-wait.sh --purpose merge -B {pr_head_branch} -R {pr_head_owner/repo} --sha {pr_head_full_sha}`
- `~/.config/mosaic/tools/git/pr-merge.sh -n {PR_NUMBER} -m squash --expect-head {approved_full_sha}` - `~/.config/mosaic/tools/git/pr-merge.sh -n {PR_NUMBER} -m squash --expect-head {approved_full_sha}`
@@ -994,13 +994,13 @@ mv docs/reports/qa-automation/pending/*failing-file* docs/reports/qa-automation/
--- ---
## Merge-to-Trunk Candidate Protocol (Container Deployments) ## Merge-to-Main Candidate Protocol (Container Deployments)
If deployment is in scope and container images are used, every merge to the integration trunk MUST execute this protocol: If deployment is in scope and container images are used, every merge to `main` MUST execute this protocol:
1. Build and push immutable candidate image tags: 1. Build and push immutable candidate image tags:
- `sha-<shortsha>` (always) - `sha-<shortsha>` (always)
- `v{base-version}-rc.{build}` (for integration-trunk merges) - `v{base-version}-rc.{build}` (for `main` merges)
- `testing` mutable pointer to the same digest - `testing` mutable pointer to the same digest
2. Resolve and record the image digest for each service. 2. Resolve and record the image digest for each service.
3. Deploy by digest to testing environment (never deploy by mutable tag alone). 3. Deploy by digest to testing environment (never deploy by mutable tag alone).
@@ -128,6 +128,14 @@ EOF
sleep 30 sleep 30
EOF EOF
chmod 700 "$AGENT_BIN/mosaic" chmod 700 "$AGENT_BIN/mosaic"
# The launcher resolves the roster's runtime against PANE_PATH before it
# spawns anything (#1241), so the runtime this projection names has to be
# present here even though the fake `mosaic` above never execs it.
cat > "$AGENT_BIN/pi" <<'EOF'
#!/bin/sh
sleep 30
EOF
chmod 700 "$AGENT_BIN/pi"
server_environment_before=$(tmux -L "$TEST_SOCKET" show-environment -g | sort) server_environment_before=$(tmux -L "$TEST_SOCKET" show-environment -g | sort)
server_sessions_before=$(tmux -L "$TEST_SOCKET" list-sessions | sort) server_sessions_before=$(tmux -L "$TEST_SOCKET" list-sessions | sort)
if /usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin MOSAIC_HOME="$AGENT_HOME" \ if /usr/bin/env -i HOME="$HOLDER_HOME" PATH=/usr/bin:/bin MOSAIC_HOME="$AGENT_HOME" \
@@ -286,6 +286,36 @@ _build_runtime_bin_prefix() {
MOSAIC_RUNTIME_BIN_PREFIX=$(_build_runtime_bin_prefix) MOSAIC_RUNTIME_BIN_PREFIX=$(_build_runtime_bin_prefix)
PANE_PATH=${MOSAIC_RUNTIME_BIN_PREFIX:+${MOSAIC_RUNTIME_BIN_PREFIX}:}/usr/local/bin:/usr/bin:/bin PANE_PATH=${MOSAIC_RUNTIME_BIN_PREFIX:+${MOSAIC_RUNTIME_BIN_PREFIX}:}/usr/local/bin:/usr/bin:/bin
# #1241. The pane runs `mosaic yolo <runtime>` under PANE_PATH with a cleared
# environment. A binary missing from *that* path is a pane that dies in under a
# second, inside a session nobody is attached to, with its diagnostic scrolled
# into a pane tmux then destroys. Resolve both here, before any effect, where
# the failure is still attributable to the thing that caused it.
#
# `mosaic yolo <runtime>` runs checkRuntime(runtime) and the binary it looks for
# is named exactly like the runtime, so resolving the runtime name is the same
# question the pane will ask a moment later — asked while an operator can still
# see the answer.
_resolve_in_pane_path() {
PATH="$PANE_PATH" command -v -- "$1" 2>/dev/null
}
# Exit 69 (EX_UNAVAILABLE): the seat cannot be provided. Distinguished from the
# 64 (EX_USAGE) rejections above, which mean the projection itself was bad —
# here the data is fine and the host is not ready. Callers tell the individual
# cases apart by `code=`, the same way fail_env's many codes share exit 64.
fail_launch() {
local code="$1"
shift
echo "ERROR: agent launch aborted: code=${code} agent=${AGENT_NAME} $*" >&2
exit 69
}
for required_binary in mosaic "$MOSAIC_AGENT_RUNTIME"; do
_resolve_in_pane_path "$required_binary" >/dev/null ||
fail_launch missing-binary "'${required_binary}' is not on the pane PATH (${PANE_PATH})"
done
_ensure_claude_workdir_trusted() { _ensure_claude_workdir_trusted() {
local workdir="$1" local workdir="$1"
local resolved local resolved
@@ -384,6 +414,19 @@ if [ -n "$PANE_PID" ]; then
_start_heartbeat_sidecar "$AGENT_NAME" "$PANE_PID" \ _start_heartbeat_sidecar "$AGENT_NAME" "$PANE_PID" \
"$MOSAIC_HEARTBEAT_RUN_DIR" "$MOSAIC_HEARTBEAT_INTERVAL" || \ "$MOSAIC_HEARTBEAT_RUN_DIR" "$MOSAIC_HEARTBEAT_INTERVAL" || \
echo "WARNING: heartbeat sidecar could not be started for $AGENT_NAME" >&2 echo "WARNING: heartbeat sidecar could not be started for $AGENT_NAME" >&2
elif _tmux has-session -t "=${AGENT_NAME}:0.0" 2>/dev/null; then
# #1241. Session present, no pane PID after a second of retries. Whatever this
# is, it is not a seat an operator can use, so it is not a success either.
fail_launch pane-pid-unresolved \
"tmux reports the session but no pane PID after 5 attempts"
else else
echo "WARNING: could not resolve pane PID for $AGENT_NAME — heartbeat sidecar not started" >&2 # #1241. This branch used to print a WARNING about the heartbeat sidecar and
# exit 0. It is not a heartbeat problem: tmux destroys a session when its pane
# command exits, so an absent session one second after new-session means the
# runtime died on startup. Reporting it as success is what let `fleet start`
# return 0 over three dead panes — the launcher knew, and said the wrong thing
# at the wrong severity to the wrong layer.
fail_launch pane-did-not-survive \
"the pane exited immediately and tmux destroyed the session;" \
"run 'mosaic yolo ${MOSAIC_AGENT_RUNTIME}' in ${MOSAIC_AGENT_WORKDIR} to see why"
fi fi
@@ -23,8 +23,26 @@ index=0
if [ "${args[0]:-}" = -L ]; then index=2; fi if [ "${args[0]:-}" = -L ]; then index=2; fi
case "${args[$index]:-}" in case "${args[$index]:-}" in
has-session) has-session)
# The holder always answers. MOSAIC_TEST_HELD_SESSIONS lets a case add
# other targets that should answer too — without it there is no way to
# model "tmux still reports the session" for a non-holder agent, and the
# launcher's pane-pid-unresolved branch is unreachable from this harness.
#
# A listed target answers only AFTER new-session, because the launcher asks
# this question twice about the same name: once before launching, where a
# yes means "already running, nothing to do, exit 0", and once after, where
# a yes means "the session survived". A shim that answered yes to both
# would short-circuit at the first and never reach the branch under test —
# it would look like coverage and measure the idempotency path instead.
for argument in "${args[@]}"; do for argument in "${args[@]}"; do
[ "$argument" = '=_holder:0.0' ] && exit 0 [ "$argument" = '=_holder:0.0' ] && exit 0
case " ${MOSAIC_TEST_HELD_SESSIONS:-} " in
*" $argument "*)
if tr '\0' '\n' < "${MOSAIC_TEST_TMUX_CALLS:?}" | grep -qxF new-session; then
exit 0
fi
;;
esac
done done
exit 1 exit 1
;; ;;
@@ -62,6 +80,30 @@ env -0 > "${MOSAIC_HOME:?}/fleet/pane-environment"
SHIM SHIM
chmod +x "$FAKE_BIN/mosaic" chmod +x "$FAKE_BIN/mosaic"
# The runtime the rosters below name. The launcher resolves it against PANE_PATH
# before spawning (#1241), so it has to exist somewhere the pane would find it —
# not merely on the launcher's own PATH.
printf '#!/usr/bin/env bash\nexit 0\n' > "$FAKE_BIN/pi"
chmod +x "$FAKE_BIN/pi"
# PANE_PATH is derived partly from `npm config get prefix`. Left to the real npm
# it would splice whatever the host has installed into the path under test, and
# the missing-binary cases below would pass or fail by accident of the machine.
cat > "$FAKE_BIN/npm" <<'SHIM'
#!/usr/bin/env bash
printf '%s\n' "${MOSAIC_TEST_NPM_PREFIX:-/nonexistent}"
SHIM
chmod +x "$FAKE_BIN/npm"
# PANE_PATH always ends in the system path. A host that installs these there can
# not measure the missing-binary cases at all, and a green run would mean
# nothing — so say so instead of passing.
for host_binary in mosaic pi; do
if PATH=/usr/local/bin:/usr/bin:/bin command -v "$host_binary" >/dev/null 2>&1; then
fail "host provides '$host_binary' in the system path; missing-binary cases are not measurable here"
fi
done
write_generated() { write_generated() {
local home="$1" local home="$1"
local agent="$2" local agent="$2"
@@ -81,6 +123,19 @@ MOSAIC_TMUX_SOCKET=mosaic-test
EOF EOF
chmod 600 "$home/fleet/agents/$agent.env.generated" chmod 600 "$home/fleet/agents/$agent.env.generated"
mkdir -p "$home/work" mkdir -p "$home/work"
install_pane_binaries "$home"
}
# `$PANE_HOME/.npm-global/bin` is one of the prefixes the launcher folds into
# PANE_PATH, so this is the pane's own view of "installed", distinct from the
# launcher's PATH. Tests that need a binary *absent* remove it from here.
install_pane_binaries() {
local pane_home="$1"
mkdir -p "$pane_home/.npm-global/bin"
local binary
for binary in mosaic pi; do
ln -sf "$FAKE_BIN/$binary" "$pane_home/.npm-global/bin/$binary"
done
} }
run_start() { run_start() {
@@ -88,6 +143,7 @@ run_start() {
local agent="$2" local agent="$2"
HOME="$home" PATH="$FAKE_BIN:$PATH" MOSAIC_TEST_TMUX_CALLS="$TMUX_CALLS" \ HOME="$home" PATH="$FAKE_BIN:$PATH" MOSAIC_TEST_TMUX_CALLS="$TMUX_CALLS" \
MOSAIC_TEST_PANE_PID="${MOSAIC_TEST_PANE_PID:-}" \ MOSAIC_TEST_PANE_PID="${MOSAIC_TEST_PANE_PID:-}" \
MOSAIC_TEST_HELD_SESSIONS="${MOSAIC_TEST_HELD_SESSIONS:-}" \
MOSAIC_TEST_HOME="$home" \ MOSAIC_TEST_HOME="$home" \
MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \ MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \
MOSAIC_HOME="$home" "$START" "$agent" MOSAIC_HOME="$home" "$START" "$agent"
@@ -98,7 +154,10 @@ run_start() {
HOME_VALID="$ROOT/valid" HOME_VALID="$ROOT/valid"
AGENT_VALID="coder0" AGENT_VALID="coder0"
write_generated "$HOME_VALID" "$AGENT_VALID" write_generated "$HOME_VALID" "$AGENT_VALID"
run_start "$HOME_VALID" "$AGENT_VALID" # A live pane PID is part of what "valid launch" means. Until #1241 this case
# ran with none, so the suite's one success path was itself a dead pane the
# launcher reported as fine.
MOSAIC_TEST_PANE_PID=$$ run_start "$HOME_VALID" "$AGENT_VALID"
valid_args=$(tr '\0' '\n' < "$TMUX_CALLS") valid_args=$(tr '\0' '\n' < "$TMUX_CALLS")
echo "$valid_args" | grep -qF new-session || fail "valid generated projection did not reach tmux" echo "$valid_args" | grep -qF new-session || fail "valid generated projection did not reach tmux"
echo "$valid_args" | grep -qF 'mosaic' || fail "fixed mosaic launcher command missing" echo "$valid_args" | grep -qF 'mosaic' || fail "fixed mosaic launcher command missing"
@@ -245,6 +304,13 @@ PANE_BASH_ENV="$ROOT/pane-boundary.bash-env"
printf 'MOSAIC_RUNTIME_BIN=%s\n' "$FAKE_BIN" > \ printf 'MOSAIC_RUNTIME_BIN=%s\n' "$FAKE_BIN" > \
"$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local" "$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local"
chmod 600 "$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local" chmod 600 "$HOME_PANE_BOUNDARY/fleet/agents/coder-pane-boundary.env.local"
# This case does not go through run_start, so its pane binaries come from
# MOSAIC_RUNTIME_BIN=$FAKE_BIN in the env.local written above — not from the
# symlinks install_pane_binaries planted under the generated home, which this
# launcher never consults because HOME here is the trusted parent. That is a
# legitimate resolution path, but it means dropping MOSAIC_RUNTIME_BIN from
# this case on the belief that the symlinks cover it would break the #1241
# binary check rather than exercise it.
LD_PRELOAD='/not/loaded/by-clean-bootstrap.so' \ LD_PRELOAD='/not/loaded/by-clean-bootstrap.so' \
BASH_ENV="$PANE_BASH_ENV" \ BASH_ENV="$PANE_BASH_ENV" \
MOSAIC_UNTRUSTED_SENTINEL='must-not-reach-pane' \ MOSAIC_UNTRUSTED_SENTINEL='must-not-reach-pane' \
@@ -258,6 +324,7 @@ PATH="$PANE_STALE_PATH" \
"MOSAIC_TEST_HOME=$PANE_TRUSTED_HOME" \ "MOSAIC_TEST_HOME=$PANE_TRUSTED_HOME" \
MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \ MOSAIC_TEST_FLEET_OWNER=123e4567-e89b-12d3-a456-426614174000 \
MOSAIC_TEST_EXECUTE_PANE=1 \ MOSAIC_TEST_EXECUTE_PANE=1 \
"MOSAIC_TEST_PANE_PID=$$" \
"$START" coder-pane-boundary "$START" coder-pane-boundary
pane_args=$(tr '\0' '\n' < "$TMUX_CALLS") pane_args=$(tr '\0' '\n' < "$TMUX_CALLS")
echo "$pane_args" | grep -qxF "HOME=$PANE_TRUSTED_HOME" || \ echo "$pane_args" | grep -qxF "HOME=$PANE_TRUSTED_HOME" || \
@@ -392,6 +459,75 @@ echo "$interaction_policy_args" | grep -qF 'new-session' && \
echo "$output" | grep -qF 'operator interaction service requires runtime pi' || \ echo "$output" | grep -qF 'operator interaction service requires runtime pi' || \
fail "interaction pinned-policy check did not follow strict parsing" fail "interaction pinned-policy check did not follow strict parsing"
# #1241. The pane runs `mosaic yolo <runtime>` against PANE_PATH. A binary
# missing from that path is a launch failure, and it has to be named before the
# session is created — after it, the diagnostic dies with the pane.
assert_missing_pane_binary_rejected() {
local binary="$1"
local home="$ROOT/missing-$binary"
local agent="coder-missing-$binary"
write_generated "$home" "$agent"
rm -f "$home/.npm-global/bin/$binary"
: > "$TMUX_CALLS"
local output
if output=$(MOSAIC_TEST_PANE_PID=$$ run_start "$home" "$agent" 2>&1); then
fail "launch succeeded with '$binary' absent from the pane PATH"
fi
echo "$output" | grep -qF 'code=missing-binary' || fail "missing '$binary' diagnostic missing"
echo "$output" | grep -qF "'$binary'" || fail "missing-binary diagnostic did not name $binary"
if tr '\0' '\n' < "$TMUX_CALLS" | grep -qF new-session; then
fail "launcher created a session it knew would die ($binary absent)"
fi
}
assert_missing_pane_binary_rejected mosaic
assert_missing_pane_binary_rejected pi
# #1241. tmux destroys a session when its pane command exits, so no pane PID a
# second after new-session means the runtime died on startup. This used to be a
# WARNING about the heartbeat sidecar followed by exit 0 — three layers above it
# then reported a fleet that was not running.
: > "$TMUX_CALLS"
HOME_DEAD_PANE="$ROOT/dead-pane"
write_generated "$HOME_DEAD_PANE" "coder-dead-pane"
if output=$(MOSAIC_TEST_PANE_PID='' run_start "$HOME_DEAD_PANE" coder-dead-pane 2>&1); then
fail "launcher reported success over a pane that did not survive"
fi
echo "$output" | grep -qF 'code=pane-did-not-survive' || fail "dead-pane diagnostic missing"
if echo "$output" | grep -qiF 'heartbeat'; then
fail "dead pane is still being reported as a heartbeat-sidecar problem"
fi
tr '\0' '\n' < "$TMUX_CALLS" | grep -qF new-session || \
fail "dead-pane case did not reach the launch it is measuring"
# #1241, the other way a pane fails. Above, tmux destroyed the session and
# has-session said so. Here the session is still there and no PID comes back
# after the retries — a different fault (the pane is alive but unusable, or
# tmux is answering inconsistently) that an operator has to be told apart from
# a runtime that died on startup.
#
# This case exists because the branch that handles it shipped with nothing able
# to reach it: the shim answered has-session only for the holder, so every
# non-holder agent landed in the session-is-gone branch no matter what. A
# defensive branch nothing exercises is the same shape as the bug this whole
# change is about, one layer down.
: > "$TMUX_CALLS"
HOME_NO_PID="$ROOT/pane-no-pid"
write_generated "$HOME_NO_PID" "coder-no-pid"
if output=$(MOSAIC_TEST_PANE_PID='' MOSAIC_TEST_HELD_SESSIONS='=coder-no-pid:0.0' \
run_start "$HOME_NO_PID" coder-no-pid 2>&1); then
fail "launcher reported success over a session with no resolvable pane PID"
fi
echo "$output" | grep -qF 'code=pane-pid-unresolved' || \
fail "session-present/no-PID was not reported as pane-pid-unresolved: $output"
if echo "$output" | grep -qF 'code=pane-did-not-survive'; then
fail "a session tmux still reports was diagnosed as a destroyed session"
fi
if echo "$output" | grep -qiF 'heartbeat'; then
fail "an unresolvable pane PID is still being reported as a heartbeat-sidecar problem"
fi
# Exact stop derives the socket exclusively from the validated generated # Exact stop derives the socket exclusively from the validated generated
# projection and ignores an ambient socket supplied by the caller. # projection and ignores an ambient socket supplied by the caller.
: > "$TMUX_CALLS" : > "$TMUX_CALLS"