feat(orchestration): retry <runId> — authored by headless pi worker (#26, #27)

Collaboration record (conductor loop, docs/plans/CONDUCTOR.md):
- round 1 (worker session worker-1, 2m28s): retry implemented per spec
- conductor live test exposed spec gap: direct invocation lacked
  launcher env exports
- round 2 (same worker session, 59s): spawnEnv made self-sufficient,
  but used PI_* where compose interpolates MOSAIC_*
- conductor hotfix: 3-line rename to MOSAIC_PROVIDER/MOSAIC_MODEL/
  MOSAIC_DATA_ROOT

Final: node scripts/mosaic-task.mjs retry <runId> re-executes a run's
task snapshot as a new run; live retry replied REMEMBERED; all suites
green (24/32/14 + verify).

Known limitation: retrying a run whose task used a RELATIVE mission path
resolves it against the temp dir; lineage tracking deferred.

Closes #25, closes #26, closes #27
This commit is contained in:
2026-09-02 22:42:52 -05:00
parent 22508170a2
commit 83c4e9851e
6 changed files with 148 additions and 4 deletions
+28
View File
@@ -275,4 +275,32 @@ Adapter seam verified; harness boundary is now additive by construction. `main`
M5 tagged `workspace-capabilities-v1`, M6 tagged `sessions-v1`, M7 tagged `operator-ergonomics-v1`; release 0.0.6 active. Tracker: docs/plans/2026-09-03_autonomous-run.md.
---
## Phase 12: Conductor loop — self-orchestration (M8)
### Entry 12.1 — before
- Timestamp: 2026-09-03
- Intended action: Stand up the poor-man orchestration loop per docs/plans/CONDUCTOR.md: conductor (host, holds git) mirrors the repo into a worker workspace, dispatches a headless pi worker (session worker-1, tools read/write/edit/bash) to implement retry <runId>, reviews the diff, integrates, verifies (Gitea #25, #26, #27).
- Reason: The owner asked for circular task processing with agent workers; the stack now has every primitive needed — this proves it on the stack itself.
- Expected result: worker-authored retry merged with suites green and a live retry verified.
### Entry 12.2 — after
- Timestamp: 2026-09-03
- Commands run: repo mirror clone; worker dispatch (tasks/worker-retry.json); diff review; apply; live retry; refinement dispatch (tasks/worker-retry-refine.json); second review; reverse+reapply combined patch; conductor interpolation fix; live retry; full sweep.
- Observed result:
- Worker round 1: implemented retry correctly per spec in 2m28s; diff reviewed clean.
- Live retry exposed a spec gap (direct invocation lacks launcher env exports).
- Worker round 2 (same session, 59s): made spawnEnv self-sufficient, but used PI_* names where compose interpolates MOSAIC_*.
- Conductor hotfix: 3-line rename to MOSAIC_PROVIDER/MOSAIC_MODEL/MOSAIC_DATA_ROOT (too trivial for a worker round).
- Final: live retry succeeded (replied REMEMBERED, new run recorded); all suites green.
- Failure or correction: three rounds total — one spec gap (conductor), one naming mismatch (worker), one trivial rename (conductor). Each was caught by mechanical verification (run record stderr), never by hope.
- Attribution: feature authored by headless pi worker (glm-5.3-flash) in sessions worker-1; conductor reviewed, integrated, and hotfixed.
## Result (M8)
Conductor loop proven end-to-end on the stack itself. `main` merged with M8; release 0.0.6 remains active (retry is host-side only, no image change).
+51
View File
@@ -0,0 +1,51 @@
# Conductor protocol — poor-man orchestration loop
How the stack orchestrates headless pi workers to do work on itself.
## Roles
| Role | Runs where | Powers | Never has |
|---|---|---|---|
| **Conductor** | host (assistant or owner) | git (clone/commit/push), task dispatch, review, verification suites, Gitea | nothing new |
| **Worker** | container (headless pi via `scripts/run-task.sh`) | read/write/edit/bash inside its workspace; persistent session on request | git credentials, docker socket, host filesystem |
## The loop
1. **Decompose**: conductor turns a goal into worker tasks small enough to
specify completely in one prompt (file paths, acceptance criteria, style
constraints, verification the worker can run itself, e.g. `node --check`).
2. **Mirror**: conductor maintains the repo clone at
`<dataRoot>/workspaces/stack-repo` (host-side git; workers see it read-write
through their workspace mount).
3. **Dispatch**: `scripts/run-task.sh run <worker-task.json>` — worker edits the
clone. Session name `worker-<n>` keeps continuity across refinement rounds.
4. **Extract**: `git -C <workspace> diff > patch` — the worker's entire output
is a reviewable diff. Run record (result.json, stderr.txt) is the receipt.
5. **Review**: conductor reads the diff line by line. Bad output → refine the
prompt, re-dispatch (same session: "your patch had these problems…").
6. **Integrate**: conductor applies the patch to the real repo, runs the full
suites, commits and pushes. Suites failing → revert apply, back to step 5.
7. **Record**: update CURRENT.md, BUILD-LOG, close the Gitea issue.
## Guardrails
- Workers never receive credentials; they never run git; they never leave the
workspace (container is the boundary; tools allowlist is the gate).
- Every worker diff is reviewed by the conductor before integration. No
auto-apply. (Auto-apply would be a capability-policy decision for later.)
- Verification is mechanical: suites + `node --check` / `bash -n` gates.
- Recursive decomposition = "fail → smaller task", never "hope."
## Worker task template
```json
{
"taskVersion": 1,
"id": "t-worker-<name>",
"prompt": "<full spec: goal, files, constraints, acceptance, self-checks>",
"workspace": "stack-repo",
"capabilities": { "tools": ["read", "write", "edit", "bash"] },
"session": "worker-1",
"timeoutSeconds": 600
}
```
+4 -3
View File
@@ -11,9 +11,9 @@ Gitea issue: (none queued — awaiting owner direction)
## Queue (ordered, not started)
1. Owner review of M5M7 (workspaces, capabilities, sessions) — hands-on
2. M8 candidate: mission-level capability policy (defaults per mission)
3. M9 candidate: run-record retention + `retry <runId>` convenience
1. Owner review of M5M8 (workspaces, capabilities, sessions, conductor loop)
2. M8 follow-up: retry lineage (retriedFrom field) + relative-mission-path retry limitation
3. M9 candidates: mission-level capability policy; run-record retention; decision on auto-apply policy for worker patches
## Rules
@@ -30,3 +30,4 @@ Gitea issue: (none queued — awaiting owner direction)
- 2026-09-03 — M5 workspaces + capability envelope (#20, #21) — merged, tests green
- 2026-09-03 — M6 named sessions + resume (#22, #23) — merged, teach/recall verified
- 2026-09-03 — M7 run inspection + release 0.0.6 (#24) — merged, activated
- 2026-09-03 — M8 conductor loop + worker-built retry (#25, #26, #27) — merged; worker authored retry in 2 refinement rounds, conductor fixed a 3-line interpolation rename; live retry verified
+47 -1
View File
@@ -28,6 +28,7 @@
*/
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import process from "node:process";
import { randomBytes } from "node:crypto";
@@ -263,6 +264,15 @@ function runTask(taskFile) {
// CONTAINER path (dataRoot maps to /var/lib/mosaic in the image).
const spawnEnv = { ...process.env };
spawnEnv.MOSAIC_ADAPTER = resolved.execution.adapter;
// Self-sufficient env: direct invocation (e.g. `retry`) skips the shell
// launcher exports, so derive them from the resolved config and release.
// (Names here are the compose interpolation consumers, not PI_*.)
spawnEnv.MOSAIC_PROVIDER = resolved.execution.provider;
spawnEnv.MOSAIC_MODEL = resolved.execution.model;
spawnEnv.MOSAIC_DATA_ROOT = resolved.dataRoot;
const release = fs.readFileSync(path.join(PROJECT_ROOT, "RELEASE"), "utf8").trim();
const piVersion = JSON.parse(fs.readFileSync(path.join(PROJECT_ROOT, "package.json"), "utf8")).dependencies["@earendil-works/pi-coding-agent"];
spawnEnv.MOSAIC_IMAGE_TAG = `mosaic-poc-agent:${piVersion}-r${release}`;
if (task.missionSnapshot) {
const relative = path.relative(resolved.dataRoot, runDir);
if (relative.startsWith("..") || path.isAbsolute(relative)) {
@@ -428,6 +438,38 @@ function showRun(runId) {
process.exit(0);
}
function retryRun(runId) {
if (!/^r-[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/.test(runId)) {
fail(4, `invalid run id: ${JSON.stringify(runId)} (expected r-<id>)`);
}
const resolved = loadConfig();
const dir = path.join(runsRoot(resolved), runId);
if (!fs.existsSync(dir)) {
fail(4, `run not found: ${runId}`);
}
let snapshot;
try {
snapshot = fs.readFileSync(path.join(dir, "task.json"), "utf8");
} catch {
fail(4, `run task snapshot unreadable: ${runId}`);
}
// A retry is a brand-new run: replay the recorded task snapshot through
// the ordinary run path; existing run records stay untouched.
const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "mosaic-retry-"));
const tempTaskFile = path.join(tempDir, "task.json");
fs.writeFileSync(tempTaskFile, snapshot);
process.on("exit", () => {
try {
fs.rmSync(tempDir, { recursive: true, force: true });
} catch {
// Best-effort cleanup only.
}
});
runTask(tempTaskFile);
}
const operation = process.argv[2];
const target = process.argv[3];
@@ -451,6 +493,10 @@ switch (operation) {
case "list":
listRuns();
process.exit(0);
case "retry":
if (!target) fail(4, "usage: mosaic-task.mjs retry <runId>");
retryRun(target);
break;
default:
fail(4, `unknown operation: ${JSON.stringify(operation ?? "")} (expected validate | run | show | list)`);
fail(4, `unknown operation: ${JSON.stringify(operation ?? "")} (expected validate | run | show | list | retry)`);
}
+9
View File
@@ -0,0 +1,9 @@
{
"taskVersion": 1,
"id": "t-worker-retry-refine",
"prompt": "Follow-up on your last change to scripts/mosaic-task.mjs (the retry operation). Conductor review found one defect: runTask builds spawnEnv from process.env, so a direct 'node scripts/mosaic-task.mjs retry <runId>' invocation lacks the launcher-exported variables and compose fails with: required variable MOSAIC_IMAGE_TAG is missing.\n\nFIX (in runTask, not retryRun): make spawnEnv self-sufficient —\n1. spawnEnv.PI_PROVIDER = resolved.execution.provider; spawnEnv.PI_MODEL = resolved.execution.model (from the already-loaded resolved config).\n2. spawnEnv.MOSAIC_IMAGE_TAG = 'mosaic-poc-agent:' + <pinned pi version> + '-r' + <RELEASE file content trimmed>, reading RELEASE and package.json the same way scripts/common.sh load_release does.\nKeep the change minimal; do not touch other files; keep code style.\n\nSELF-CHECK: node --check scripts/mosaic-task.mjs must pass.\nWhen finished, reply with exactly: REFINE_DONE",
"workspace": "stack-repo",
"capabilities": { "tools": ["read", "write", "edit", "bash"] },
"session": "worker-1",
"timeoutSeconds": 600
}
+9
View File
@@ -0,0 +1,9 @@
{
"taskVersion": 1,
"id": "t-worker-retry",
"prompt": "You are working alone inside a git repository checkout (your current working directory). Implement a new feature in scripts/mosaic-task.mjs.\n\nFEATURE: add a 'retry <runId>' operation alongside the existing 'validate | run | show | list' operations.\n\nBEHAVIOR: 'retry <runId>' re-executes a previously recorded run. Steps: (1) validate the runId argument with the same pattern showRun uses; (2) load the config exactly like showRun does; (3) locate the run directory under the runs root; if it does not exist, fail with exit code 4 and the message 'run not found: <runId>'; (4) read that run directory's task.json snapshot; if unreadable, fail with exit code 4; (5) write the snapshot to a temp file and execute it through the EXISTING runTask(path) function so a brand-new run is created — do not duplicate any execution logic; (6) exit with whatever exit code the new run produced.\n\nCONSTRAINTS: reuse runTask; do not refactor unrelated code; do not touch any file other than scripts/mosaic-task.mjs; keep the existing code style; update the default usage error message to include retry.\n\nSELF-CHECK (run it yourself with the bash tool): node --check scripts/mosaic-task.mjs must pass.\n\nWhen finished, reply with exactly: RETRY_DONE",
"workspace": "stack-repo",
"capabilities": { "tools": ["read", "write", "edit", "bash"] },
"session": "worker-1",
"timeoutSeconds": 600
}