Merge M8: conductor loop - self-orchestration

Closes #25, closes #26, closes #27

BUILD-LOG.md

@@ -221,4 +221,86 @@ Release model and safe updates verified by drills. `main` merged with M3 and tag
 
 Adapter seam verified; harness boundary is now additive by construction. `main` merged with M4 and tagged `adapter-seam-v1`.
 
+---
+
+## Phase 9: Workspaces + capability envelope (M5)
+
+### Entry 9.1 — before
+
+- Timestamp: 2026-09-03
+- Intended action: Optional task workspace (absent / ":run" ephemeral / named persistent) and capabilities.tools allowlist (pi built-ins); runner plumbing via MOSAIC_WORKSPACE/MOSAIC_TOOLS; pi adapter maps to cwd + --tools; mock adapter logs delivered MOSAIC_* vars for deterministic assertions (Gitea #20, #21).
+- Reason: Agents that only answer text cannot do work; the workspace+tools pair is the smallest real capability step, bounded by the container.
+- Expected result: plumbing asserted via run-record stderr; live pi writes a host-visible file.
+
+### Entry 9.2 — after
+
+- Timestamp: 2026-09-03
+- Commands run: build; mock plumbing run; validate negatives; full task suite; live workspace demo.
+- Observed result: task suite 32/32; MOSAIC_WORKSPACE/MOSAIC_TOOLS asserted in run record; dataRoot/workspaces/<name> created host-side; live pi used bash to write proof.txt into the demo workspace (host-visible).
+- Failure or correction: (1) batch edit dropped SUPPORTED_TOOLS const (runtime ReferenceError, exit 1 instead of 2) — restored; (2) mock env dump used `export` which this dash prints as `export K='v'` — switched to `env`; (3) selftest fed a mismatching mock response to the plain-task case — test bug, fixed.
+
+## Phase 10: Named sessions (M6)
+
+### Entry 10.1 — before
+
+- Timestamp: 2026-09-03
+- Intended action: Optional task.session name -> persistent session dir dataRoot/sessions/<name> via pi --session-dir; resume most recent with -c when present; isolation per name; teach/recall demo fixtures (Gitea #22, #23).
+- Reason: L1 persistence is the prerequisite for any multi-step agent work.
+- Expected result: session dir populated after first run; second run recalls taught context.
+
+### Entry 10.2 — after
+
+- Timestamp: 2026-09-03
+- Commands run: build; mock plumbing run; live teach/recall E2E; task suite.
+- Observed result: task suite 32/32; teach run replied REMEMBERED and session JSONL persisted host-side; recall run resumed (-c) and answered exactly 'mosaico'; single continued session file (no duplicate sessions).
+- Failure or correction: none. Design note: ephemeral (--no-session) remains the default when no session is declared.
+
+## Phase 11: Operator ergonomics (M7)
+
+### Entry 11.1 — before
+
+- Timestamp: 2026-09-03
+- Intended action: mosaic-task.mjs show <runId> (full record + snapshots + artifacts, traversal-safe), list with workspace/session columns, RELEASE -> 0.0.6, package + health-gated activate, docs (Gitea #24).
+- Reason: Run records are only as valuable as they are inspectable; release activation closes the loop on container-content changes.
+- Expected result: show works for real/missing/traversal ids; suites green; 0.0.6 active.
+
+### Entry 11.2 — after
+
+- Timestamp: 2026-09-03
+- Commands run: build; show on real/missing/traversal ids; full sweep; release package + activate.
+- Observed result: config 24/24, task 32/32, release 14/14, verify PASS; 0.0.6 packaged and activated via exact-marker health gate.
+- Failure or correction: showRun initially rejected valid run ids (lowercase-only regex vs uppercase timestamp) and crashed on missing ids (uncaught readdir) — both fixed and covered.
+
+## Autonomous run result
+
+M5 tagged `workspace-capabilities-v1`, M6 tagged `sessions-v1`, M7 tagged `operator-ergonomics-v1`; release 0.0.6 active. Tracker: docs/plans/2026-09-03_autonomous-run.md.
+
+---
+
+## Phase 12: Conductor loop — self-orchestration (M8)
+
+### Entry 12.1 — before
+
+- Timestamp: 2026-09-03
+- Intended action: Stand up the poor-man orchestration loop per docs/plans/CONDUCTOR.md: conductor (host, holds git) mirrors the repo into a worker workspace, dispatches a headless pi worker (session worker-1, tools read/write/edit/bash) to implement retry <runId>, reviews the diff, integrates, verifies (Gitea #25, #26, #27).
+- Reason: The owner asked for circular task processing with agent workers; the stack now has every primitive needed — this proves it on the stack itself.
+- Expected result: worker-authored retry merged with suites green and a live retry verified.
+
+### Entry 12.2 — after
+
+- Timestamp: 2026-09-03
+- Commands run: repo mirror clone; worker dispatch (tasks/worker-retry.json); diff review; apply; live retry; refinement dispatch (tasks/worker-retry-refine.json); second review; reverse+reapply combined patch; conductor interpolation fix; live retry; full sweep.
+- Observed result:
+  - Worker round 1: implemented retry correctly per spec in 2m28s; diff reviewed clean.
+  - Live retry exposed a spec gap (direct invocation lacks launcher env exports).
+  - Worker round 2 (same session, 59s): made spawnEnv self-sufficient, but used PI_* names where compose interpolates MOSAIC_*.
+  - Conductor hotfix: 3-line rename to MOSAIC_PROVIDER/MOSAIC_MODEL/MOSAIC_DATA_ROOT (too trivial for a worker round).
+  - Final: live retry succeeded (replied REMEMBERED, new run recorded); all suites green.
+- Failure or correction: three rounds total — one spec gap (conductor), one naming mismatch (worker), one trivial rename (conductor). Each was caught by mechanical verification (run record stderr), never by hope.
+- Attribution: feature authored by headless pi worker (glm-5.3-flash) in sessions worker-1; conductor reviewed, integrated, and hotfixed.
+
+## Result (M8)
+
+Conductor loop proven end-to-end on the stack itself. `main` merged with M8; release 0.0.6 remains active (retry is host-side only, no image change).
+
 

README.md

@@ -119,6 +119,31 @@ adapters/<name>/adapter.sh   env in: MOSAIC_SYSTEM_PROMPT_FILE, MOSAIC_REQUEST,
 
 See `adapters/README.md` for the full contract.
 
+## Workspaces, capabilities, sessions (M5/M6)
+
+Optional task fields extend what an agent can do — all defaulting to the previous behavior:
+
+```json
+{
+  "workspace": "demo",                          // ":run" ephemeral, or persistent dataRoot/workspaces/<name>
+  "capabilities": { "tools": ["bash", "read"] }, // pi tool allowlist; absent = no tools
+  "session": "demo"                             // persistent session at dataRoot/sessions/<name>
+}
+```
+
+- The adapter runs inside the workspace; files it writes are host-visible (`dataRoot/workspaces/<name>`).
+- Sessions persist via pi's documented `--session-dir`; a follow-up run in the same session resumes the conversation (`-c`) and can recall prior context. Distinct names never share state. Ephemeral (`--no-session`) remains the default when no session is declared.
+- Selection authority: config for adapter/provider/model; the task file for workspace/capabilities/session.
+
+Inspect anything:
+
+```bash
+node scripts/mosaic-task.mjs list                # runs with task/workspace/session columns
+node scripts/mosaic-task.mjs show <runId>        # full record + snapshots + artifacts
+```
+
+Demo fixtures: `tasks/workspace-demo.json`, `tasks/session-demo-1.json` + `tasks/session-demo-2.json`.
+
 See `docs/plans/2026-09-02_atomic-mosaic-foundation.md` for the full plan.
 
 Inside the container:

RELEASE

@@ -1 +1 @@
-0.0.5
+0.0.6

docs/plans/2026-09-03_autonomous-run.md

@@ -13,8 +13,8 @@ Advance the Mosaic Stack rebuild several verified layers in one batch, focused o
 |---|---|---|
 | M5 | Task workspaces + capability envelope (tools allowlist) | DONE |
 | M6 | Named sessions — persistence and resume (L1) | DONE |
-| M7 | Operator ergonomics: run inspection commands | DONE (partial by design) |
-| — | Release 0.0.6 packaged + health-gated activated | DONE |
+| M7 | Operator ergonomics: run inspection commands | DONE |
+| — | Releases 0.0.5+0.0.6 packaged; 0.0.6 health-gated activated | DONE |
 
 Explicitly deferred (do not mistake for forgotten):
 - Claude/Codex/OpenCode adapters (owner: focus on Pi for now)
@@ -60,14 +60,17 @@ Explicitly deferred (do not mistake for forgotten):
 6. Inspect any run: `node scripts/mosaic-task.mjs show <runId>`
 7. Gitea: milestones M5/M6/M7 closed; issues referenced by merge commits
 
-## Results (to be filled at end of run)
+## Results
 
-- M5: pending
-- M6: pending
-- M7: pending
-- Release activation: pending
-- Final suite counts: pending
-- Corrections encountered: pending
+- M5 merged on `main` (merge commit `ddb1554`), tagged `workspace-capabilities-v1`
+- M6 merged on `main` (merge commit `4e2a413`), tagged `sessions-v1`
+- M7 merged on `main`, tagged `operator-ergonomics-v1`
+- Release 0.0.6 packaged, health-gated activated, full sweep green
+- Suites at end of run: config 24/24, task 32/32, release 14/14, verify PASS
+- Build log: Phases 9 (M5), 10 (M6), 11 (M7) appended with corrections
+- Corrections encountered: dropped constant from a failed atomic edit batch (SUPPORTED_TOOLS); dash `export` output format vs `env`; three selftest authoring defects; showRun id-regex case sensitivity + missing-run crash. All fixed and covered by tests.
+- Commits pushed incrementally; nothing left uncommitted
+- Live proof: workspace file host-visible; session teach/recall ('mosaico') verified
 
 ## Next steps after this run (not started)
 

docs/plans/CONDUCTOR.md

@@ -0,0 +1,51 @@
+# Conductor protocol — poor-man orchestration loop
+
+How the stack orchestrates headless pi workers to do work on itself.
+
+## Roles
+
+| Role | Runs where | Powers | Never has |
+|---|---|---|---|
+| **Conductor** | host (assistant or owner) | git (clone/commit/push), task dispatch, review, verification suites, Gitea | nothing new |
+| **Worker** | container (headless pi via `scripts/run-task.sh`) | read/write/edit/bash inside its workspace; persistent session on request | git credentials, docker socket, host filesystem |
+
+## The loop
+
+1. **Decompose**: conductor turns a goal into worker tasks small enough to
+   specify completely in one prompt (file paths, acceptance criteria, style
+   constraints, verification the worker can run itself, e.g. `node --check`).
+2. **Mirror**: conductor maintains the repo clone at
+   `<dataRoot>/workspaces/stack-repo` (host-side git; workers see it read-write
+   through their workspace mount).
+3. **Dispatch**: `scripts/run-task.sh run <worker-task.json>` — worker edits the
+   clone. Session name `worker-<n>` keeps continuity across refinement rounds.
+4. **Extract**: `git -C <workspace> diff > patch` — the worker's entire output
+   is a reviewable diff. Run record (result.json, stderr.txt) is the receipt.
+5. **Review**: conductor reads the diff line by line. Bad output → refine the
+   prompt, re-dispatch (same session: "your patch had these problems…").
+6. **Integrate**: conductor applies the patch to the real repo, runs the full
+   suites, commits and pushes. Suites failing → revert apply, back to step 5.
+7. **Record**: update CURRENT.md, BUILD-LOG, close the Gitea issue.
+
+## Guardrails
+
+- Workers never receive credentials; they never run git; they never leave the
+  workspace (container is the boundary; tools allowlist is the gate).
+- Every worker diff is reviewed by the conductor before integration. No
+  auto-apply. (Auto-apply would be a capability-policy decision for later.)
+- Verification is mechanical: suites + `node --check` / `bash -n` gates.
+- Recursive decomposition = "fail → smaller task", never "hope."
+
+## Worker task template
+
+```json
+{
+  "taskVersion": 1,
+  "id": "t-worker-<name>",
+  "prompt": "<full spec: goal, files, constraints, acceptance, self-checks>",
+  "workspace": "stack-repo",
+  "capabilities": { "tools": ["read", "write", "edit", "bash"] },
+  "session": "worker-1",
+  "timeoutSeconds": 600
+}
+```

docs/plans/CURRENT.md

@@ -0,0 +1,33 @@
+# CURRENT — single source of "what happens next"
+
+This file always names exactly one next action. Any "continue" / "next" /
+"proceed" message means: execute the action below, fully (implement → test →
+verify against its acceptance criteria → commit → push → close the issue →
+update this file to the next action). No ambiguity, no re-planning.
+
+## Next action
+
+Gitea issue: (none queued — awaiting owner direction)
+
+## Queue (ordered, not started)
+
+1. Owner review of M5–M8 (workspaces, capabilities, sessions, conductor loop)
+2. M8 follow-up: retry lineage (retriedFrom field) + relative-mission-path retry limitation
+3. M9 candidates: mission-level capability policy; run-record retention; decision on auto-apply policy for worker patches
+
+## Rules
+
+- One action in flight. Update this file at the END of every action.
+- Blocked? Move the item to "Blocked" below with the reason and stop.
+- Completed actions move to the log at the bottom (date + issue + result).
+
+## Blocked
+
+(none)
+
+## Completed log
+
+- 2026-09-03 — M5 workspaces + capability envelope (#20, #21) — merged, tests green
+- 2026-09-03 — M6 named sessions + resume (#22, #23) — merged, teach/recall verified
+- 2026-09-03 — M7 run inspection + release 0.0.6 (#24) — merged, activated
+- 2026-09-03 — M8 conductor loop + worker-built retry (#25, #26, #27) — merged; worker authored retry in 2 refinement rounds, conductor fixed a 3-line interpolation rename; live retry verified

scripts/mosaic-task.mjs

@@ -28,6 +28,7 @@
  */
 
 import fs from "node:fs";
+import os from "node:os";
 import path from "node:path";
 import process from "node:process";
 import { randomBytes } from "node:crypto";
@@ -263,6 +264,15 @@ function runTask(taskFile) {
   // CONTAINER path (dataRoot maps to /var/lib/mosaic in the image).
   const spawnEnv = { ...process.env };
   spawnEnv.MOSAIC_ADAPTER = resolved.execution.adapter;
+  // Self-sufficient env: direct invocation (e.g. `retry`) skips the shell
+  // launcher exports, so derive them from the resolved config and release.
+  // (Names here are the compose interpolation consumers, not PI_*.)
+  spawnEnv.MOSAIC_PROVIDER = resolved.execution.provider;
+  spawnEnv.MOSAIC_MODEL = resolved.execution.model;
+  spawnEnv.MOSAIC_DATA_ROOT = resolved.dataRoot;
+  const release = fs.readFileSync(path.join(PROJECT_ROOT, "RELEASE"), "utf8").trim();
+  const piVersion = JSON.parse(fs.readFileSync(path.join(PROJECT_ROOT, "package.json"), "utf8")).dependencies["@earendil-works/pi-coding-agent"];
+  spawnEnv.MOSAIC_IMAGE_TAG = `mosaic-poc-agent:${piVersion}-r${release}`;
   if (task.missionSnapshot) {
     const relative = path.relative(resolved.dataRoot, runDir);
     if (relative.startsWith("..") || path.isAbsolute(relative)) {
@@ -365,17 +375,101 @@ function listRuns() {
   for (const runId of entries) {
     let status = "unknown";
     let taskId = "-";
+    let workspace = "-";
+    let session = "-";
     try {
       const result = JSON.parse(fs.readFileSync(path.join(root, runId, "result.json"), "utf8"));
       status = result.status;
       taskId = result.taskId;
+      workspace = result.workspace ?? "-";
+      session = result.session ?? "-";
     } catch {
       // Incomplete run record; report as unknown.
     }
-    process.stdout.write(`${runId}  ${status.padEnd(9)}  ${taskId}\n`);
+    process.stdout.write(`${runId}  ${status.padEnd(9)}  task=${taskId.padEnd(18)} ws=${String(workspace).padEnd(10)} session=${session}\n`);
   }
 }
 
+function showRun(runId) {
+  if (!/^r-[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/.test(runId)) {
+    fail(4, `invalid run id: ${JSON.stringify(runId)} (expected r-<id>)`);
+  }
+  const resolved = loadConfig();
+  const dir = path.join(runsRoot(resolved), runId);
+  if (!fs.existsSync(dir)) {
+    fail(4, `run not found: ${runId} (under ${runsRoot(resolved)})`);
+  }
+
+  const read = (name) => {
+    try {
+      return JSON.parse(fs.readFileSync(path.join(dir, name), "utf8"));
+    } catch {
+      return null;
+    }
+  };
+  const result = read("result.json");
+  const task = read("task.json");
+  const mission = read("mission.json");
+
+  process.stdout.write(`run:       ${runId}\n`);
+  if (result) {
+    process.stdout.write(
+      [
+        `status:    ${result.status}${result.reason ? ` (${result.reason})` : ""}`,
+        `task:      ${result.taskId}`,
+        result.missionId ? `mission:   ${result.missionId}` : null,
+        result.workspace ? `workspace: ${result.workspace}` : null,
+        result.session ? `session:   ${result.session}` : null,
+        result.tools ? `tools:     ${result.tools.join(", ")}` : null,
+        `adapter:   ${"(see config)"} provider=${result.provider} model=${result.model}`,
+        `request:   ${JSON.stringify(result.request)}`,
+        `response:  ${JSON.stringify(result.response)}`,
+        result.expectedExact !== null && result.expectedExact !== undefined ? `expected:  ${JSON.stringify(result.expectedExact)}` : null,
+        `timing:    ${result.startedAt} -> ${result.finishedAt} (${result.durationMs} ms)`,
+        `exit:      ${result.exitCode}${result.signal ? ` signal=${result.signal}` : ""}`,
+      ].filter((line) => line !== null).join("\n") + "\n",
+    );
+  } else {
+    process.stdout.write("result.json: (missing or unreadable)\n");
+  }
+  if (task) process.stdout.write(`task snapshot: ${"task.json"} present\n`);
+  if (mission) process.stdout.write(`mission snapshot: ${mission.id} - ${mission.objective}\n`);
+  process.stdout.write(`artifacts: ${fs.readdirSync(dir).map((f) => `${f}`).join(", ")}\n`);
+  process.exit(0);
+}
+
+function retryRun(runId) {
+  if (!/^r-[A-Za-z0-9][A-Za-z0-9._-]{0,63}$/.test(runId)) {
+    fail(4, `invalid run id: ${JSON.stringify(runId)} (expected r-<id>)`);
+  }
+  const resolved = loadConfig();
+  const dir = path.join(runsRoot(resolved), runId);
+  if (!fs.existsSync(dir)) {
+    fail(4, `run not found: ${runId}`);
+  }
+
+  let snapshot;
+  try {
+    snapshot = fs.readFileSync(path.join(dir, "task.json"), "utf8");
+  } catch {
+    fail(4, `run task snapshot unreadable: ${runId}`);
+  }
+
+  // A retry is a brand-new run: replay the recorded task snapshot through
+  // the ordinary run path; existing run records stay untouched.
+  const tempDir = fs.mkdtempSync(path.join(os.tmpdir(), "mosaic-retry-"));
+  const tempTaskFile = path.join(tempDir, "task.json");
+  fs.writeFileSync(tempTaskFile, snapshot);
+  process.on("exit", () => {
+    try {
+      fs.rmSync(tempDir, { recursive: true, force: true });
+    } catch {
+      // Best-effort cleanup only.
+    }
+  });
+  runTask(tempTaskFile);
+}
+
 const operation = process.argv[2];
 const target = process.argv[3];
 
@@ -392,9 +486,17 @@ switch (operation) {
     if (!target) fail(4, "usage: mosaic-task.mjs run <taskFile>");
     runTask(path.resolve(target));
     break;
+  case "show":
+    if (!target) fail(4, "usage: mosaic-task.mjs show <runId>");
+    showRun(target);
+    break;
   case "list":
     listRuns();
     process.exit(0);
+  case "retry":
+    if (!target) fail(4, "usage: mosaic-task.mjs retry <runId>");
+    retryRun(target);
+    break;
   default:
-    fail(4, `unknown operation: ${JSON.stringify(operation ?? "")} (expected validate | run | list)`);
+    fail(4, `unknown operation: ${JSON.stringify(operation ?? "")} (expected validate | run | show | list | retry)`);
 }

tasks/worker-retry-refine.json

@@ -0,0 +1,9 @@
+{
+  "taskVersion": 1,
+  "id": "t-worker-retry-refine",
+  "prompt": "Follow-up on your last change to scripts/mosaic-task.mjs (the retry operation). Conductor review found one defect: runTask builds spawnEnv from process.env, so a direct 'node scripts/mosaic-task.mjs retry <runId>' invocation lacks the launcher-exported variables and compose fails with: required variable MOSAIC_IMAGE_TAG is missing.\n\nFIX (in runTask, not retryRun): make spawnEnv self-sufficient —\n1. spawnEnv.PI_PROVIDER = resolved.execution.provider; spawnEnv.PI_MODEL = resolved.execution.model (from the already-loaded resolved config).\n2. spawnEnv.MOSAIC_IMAGE_TAG = 'mosaic-poc-agent:' + <pinned pi version> + '-r' + <RELEASE file content trimmed>, reading RELEASE and package.json the same way scripts/common.sh load_release does.\nKeep the change minimal; do not touch other files; keep code style.\n\nSELF-CHECK: node --check scripts/mosaic-task.mjs must pass.\nWhen finished, reply with exactly: REFINE_DONE",
+  "workspace": "stack-repo",
+  "capabilities": { "tools": ["read", "write", "edit", "bash"] },
+  "session": "worker-1",
+  "timeoutSeconds": 600
+}

tasks/worker-retry.json

@@ -0,0 +1,9 @@
+{
+  "taskVersion": 1,
+  "id": "t-worker-retry",
+  "prompt": "You are working alone inside a git repository checkout (your current working directory). Implement a new feature in scripts/mosaic-task.mjs.\n\nFEATURE: add a 'retry <runId>' operation alongside the existing 'validate | run | show | list' operations.\n\nBEHAVIOR: 'retry <runId>' re-executes a previously recorded run. Steps: (1) validate the runId argument with the same pattern showRun uses; (2) load the config exactly like showRun does; (3) locate the run directory under the runs root; if it does not exist, fail with exit code 4 and the message 'run not found: <runId>'; (4) read that run directory's task.json snapshot; if unreadable, fail with exit code 4; (5) write the snapshot to a temp file and execute it through the EXISTING runTask(path) function so a brand-new run is created — do not duplicate any execution logic; (6) exit with whatever exit code the new run produced.\n\nCONSTRAINTS: reuse runTask; do not refactor unrelated code; do not touch any file other than scripts/mosaic-task.mjs; keep the existing code style; update the default usage error message to include retry.\n\nSELF-CHECK (run it yourself with the bash tool): node --check scripts/mosaic-task.mjs must pass.\n\nWhen finished, reply with exactly: RETRY_DONE",
+  "workspace": "stack-repo",
+  "capabilities": { "tools": ["read", "write", "edit", "bash"] },
+  "session": "worker-1",
+  "timeoutSeconds": 600
+}