mosaicstack/stack
21
0
Fork 1
Code Issues 338 Pull Requests 13 Actions Packages Projects Releases 15 Wiki Activity

docs: concept annexation, provider/reference docs, ACT-1 groundwork

Browse Source 
Mosaic concepts pages now own the adapted content; source/license
metadata under docs/reference/concepts. Adds ACT-1 agent-context
planning capture, pinned concept test package + preparation utility,
foundation observation notes (durability, evidence, federation,
onboarding, workflow), and the #1495 consolidation assessment.
TOOLS.md updated for the host-dev launcher.
This commit is contained in:
jason.woltje
2026-09-07 14:07:05 -05:00
parent 7c580a5625
commit 193479b52d
119 changed files with 21185 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/providers/comfy.md
+393
View File
@@ -0,0 +1,393 @@
---
summary: "ComfyUI workflow image, video, and music generation setup in OpenClaw"
title: "ComfyUI"
read_when:
- You want to use local ComfyUI workflows with OpenClaw
- You want to use Comfy Cloud with image, video, or music workflows
- You need the comfy plugin config keys
---
Install the official `comfy` plugin for workflow-driven ComfyUI runs:
```bash
openclaw plugins install @openclaw/comfy-provider
openclaw gateway restart
```
The plugin is entirely workflow-driven: OpenClaw does not map generic `size`,
`aspectRatio`, `resolution`, `durationSeconds`, or TTS-style controls onto
your graph.
| Property | Detail |
| ------------ | ------------------------------------------------------------------------------------------ |
| Provider | `comfy` |
| Model | `comfy/workflow` |
| Shared tools | `image_generate`, `video_generate`, `music_generate` |
| Auth | Optional `headers` for local HTTP auth; `COMFY_API_KEY` or `COMFY_CLOUD_API_KEY` for cloud |
| API | ComfyUI `/prompt` / `/history` / `/view`; Comfy Cloud `/api/*` |
## What it supports
- Image generation and editing from a workflow JSON (edit takes 1 uploaded reference image)
- Video generation from a workflow JSON, text-to-video or image-to-video (1 reference image)
- Music/audio generation through the shared `music_generate` tool, with an optional 1 reference image
- Output download from a configured node, or from all matching output nodes when none is configured
## Getting started
Choose between running ComfyUI on your own machine or using Comfy Cloud.
<Tabs>
<Tab title="Local">
**Best for:** running your own ComfyUI instance on your machine or LAN.
<Steps>
<Step title="Start ComfyUI locally">
Make sure your local ComfyUI instance is running (defaults to `http://127.0.0.1:8188`).
</Step>
<Step title="Prepare your workflow JSON">
Export or create a ComfyUI workflow JSON file. Note the node IDs for the prompt input node and the output node you want OpenClaw to read from.
</Step>
<Step title="Configure the provider">
Set `mode: "local"` and point at your workflow file. Minimal image example:
```json5
{
plugins: {
entries: {
comfy: {
config: {
mode: "local",
baseUrl: "http://127.0.0.1:8188",
image: {
workflowPath: "./workflows/flux-api.json",
promptNodeId: "6",
outputNodeId: "9",
},
},
},
},
},
}
```
</Step>
<Step title="Set the default model">
Point OpenClaw at the `comfy/workflow` model for the capability you configured:
```json5
{
agents: {
defaults: {
mediaModels: {
image: {
primary: "comfy/workflow",
},
},
},
},
}
```
</Step>
<Step title="Verify">
```bash
openclaw models list --provider comfy
```
</Step>
</Steps>
</Tab>
<Tab title="Comfy Cloud">
**Best for:** running workflows on Comfy Cloud without managing local GPU resources.
<Steps>
<Step title="Get an API key">
Sign up at [comfy.org](https://comfy.org) and generate an API key from your account dashboard.
</Step>
<Step title="Set the API key">
Provide your key through any of these methods:
```bash
# Onboarding flag
openclaw onboard --comfy-api-key "your-key"
# Environment variable (preferred for daemons)
export COMFY_API_KEY="your-key"
# Alternative environment variable
export COMFY_CLOUD_API_KEY="your-key"
# Or inline in config
openclaw config set plugins.entries.comfy.config.apiKey "your-key"
```
</Step>
<Step title="Prepare your workflow JSON">
Export or create a ComfyUI workflow JSON file. Note the node IDs for the prompt input node and the output node.
</Step>
<Step title="Configure the provider">
Set `mode: "cloud"` and point at your workflow file:
```json5
{
plugins: {
entries: {
comfy: {
config: {
mode: "cloud",
image: {
workflowPath: "./workflows/flux-api.json",
promptNodeId: "6",
outputNodeId: "9",
},
},
},
},
},
}
```
<Tip>
Cloud mode defaults `baseUrl` to `https://cloud.comfy.org`. Set `baseUrl` only for a custom cloud endpoint.
</Tip>
</Step>
<Step title="Set the default model">
```json5
{
agents: {
defaults: {
mediaModels: {
image: {
primary: "comfy/workflow",
},
},
},
},
}
```
</Step>
<Step title="Verify">
```bash
openclaw models list --provider comfy
```
</Step>
</Steps>
</Tab>
</Tabs>
## Configuration
Comfy supports shared top-level connection settings plus per-capability workflow sections (`image`, `video`, `music`):
```json5
{
plugins: {
entries: {
comfy: {
config: {
mode: "local",
baseUrl: "http://127.0.0.1:8188",
image: {
workflowPath: "./workflows/flux-api.json",
promptNodeId: "6",
outputNodeId: "9",
},
video: {
workflowPath: "./workflows/video-api.json",
promptNodeId: "12",
outputNodeId: "21",
},
music: {
workflowPath: "./workflows/music-api.json",
promptNodeId: "3",
outputNodeId: "18",
},
},
},
},
},
}
```
### Shared keys
| Key | Type | Description |
| --------------------- | ---------------------- | ------------------------------------------------------------------------------------- |
| `mode` | `"local"` or `"cloud"` | Connection mode. Defaults to `"local"`. |
| `baseUrl` | string | Defaults to `http://127.0.0.1:8188` for local or `https://cloud.comfy.org` for cloud. |
| `apiKey` | string or SecretRef | Optional cloud key, alternative to `COMFY_API_KEY` / `COMFY_CLOUD_API_KEY` env vars. |
| `allowPrivateNetwork` | boolean | Allow a private/LAN `baseUrl` in cloud mode or a local private-DNS FQDN. |
| `headers` | object | Extra request headers; each value accepts a string or SecretRef. |
Use `headers.Authorization` for a ComfyUI instance behind HTTP authentication.
Prefer a [secret reference](/gateway/config-secrets-env#secrets) for credentials.
Headers apply to uploads, workflow submissions, polling, and downloads in both
modes. They override default headers case-insensitively, except `Content-Type`
on image uploads: the runtime sets the multipart boundary. An unavailable
header SecretRef fails before any request is sent. Reflected header values are
redacted from response errors.
<Note>
In `local` mode, loopback/private IP literals and single-label service names such as `http://comfyui:8188` work without `allowPrivateNetwork`. Public-looking private-DNS FQDNs such as `https://comfy.local.example.com` require `allowPrivateNetwork: true`. Private-origin trust stays scoped to the configured scheme, hostname, and port; local redirects cannot leave the configured hostname, while cloud redirects to public CDNs are checked with the default SSRF policy.
</Note>
### Per-capability keys
These keys apply inside the `image`, `video`, or `music` sections:
| Key | Required | Default | Description |
| ---------------------------- | -------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `workflow` or `workflowPath` | Yes | -- | Inline workflow JSON, or path to the ComfyUI workflow JSON file. |
| `promptNodeId` | Yes | -- | Node ID that receives the text prompt. |
| `promptInputName` | No | `"text"` | Input name on the prompt node. |
| `seedNodeId` | No | -- | Node ID whose input receives a fresh random seed on every submission. Omit to reuse whatever seed is baked into the workflow file on every run. |
| `seedInputName` | No | `"seed"` | Input name on the seed node. |
| `outputNodeId` | No | -- | Node ID to read output from. If omitted, all matching output nodes are used. |
| `pollIntervalMs` | No | `1500` | Polling interval in milliseconds for job completion. |
| `timeoutMs` | No | `300000` | Timeout in milliseconds for the workflow run. |
The `image` and `video` sections also support a reference-image input node:
| Key | Required | Default | Description |
| --------------------- | ------------------------------------ | --------- | --------------------------------------------------- |
| `inputImageNodeId` | Yes (when passing a reference image) | -- | Node ID that receives the uploaded reference image. |
| `inputImageInputName` | No | `"image"` | Input name on the image node. |
`apiKey` accepts either a literal string or a [secret reference](/gateway/config-secrets-env#secrets) object.
## Workflow details
<AccordionGroup>
<Accordion title="Image workflows">
Set the default image model to `comfy/workflow`:
```json5
{
agents: {
defaults: {
mediaModels: {
image: {
primary: "comfy/workflow",
},
},
},
},
}
```
**Reference-image editing example:**
To enable image editing with an uploaded reference image, add `inputImageNodeId` to your image config:
```json5
{
plugins: {
entries: {
comfy: {
config: {
image: {
workflowPath: "./workflows/edit-api.json",
promptNodeId: "6",
inputImageNodeId: "7",
inputImageInputName: "image",
outputNodeId: "9",
},
},
},
},
},
}
```
</Accordion>
<Accordion title="Video workflows">
Set the default video model to `comfy/workflow`:
```json5
{
agents: {
defaults: {
mediaModels: {
video: {
primary: "comfy/workflow",
},
},
},
},
}
```
Comfy video workflows support text-to-video and image-to-video through the configured graph.
<Note>
OpenClaw does not pass input videos into Comfy workflows. Only text prompts and single reference images are supported as inputs.
</Note>
</Accordion>
<Accordion title="Music workflows">
The bundled plugin registers a music-generation provider for workflow-defined audio or music outputs, surfaced through the shared `music_generate` tool. It accepts an optional reference image (up to 1):
```text
/tool music_generate prompt="Warm ambient synth loop with soft tape texture"
```
Use the `music` config section to point at your audio workflow JSON and output node.
</Accordion>
<Accordion title="Backward compatibility">
Existing top-level image config (without the nested `image` section) still works:
```json5
{
plugins: {
entries: {
comfy: {
config: {
workflowPath: "./workflows/flux-api.json",
promptNodeId: "6",
outputNodeId: "9",
},
},
},
},
}
```
OpenClaw treats that legacy shape as the image workflow config. You do not need to migrate immediately, but the nested `image` / `video` / `music` sections are recommended for new setups. If you only use image generation, the legacy flat config and the new nested `image` section are functionally equivalent.
</Accordion>
<Accordion title="Live tests">
Opt-in live coverage exists for the bundled plugin:
```bash
OPENCLAW_LIVE_TEST=1 COMFY_LIVE_TEST=1 pnpm test:live -- extensions/comfy/comfy.live.test.ts
```
The live test skips individual image, video, or music cases unless the matching Comfy workflow section is configured.
</Accordion>
</AccordionGroup>
## Related
<CardGroup cols={2}>
<Card title="Image Generation" href="/tools/image-generation" icon="image">
Image generation tool configuration and usage.
</Card>
<Card title="Video Generation" href="/tools/video-generation" icon="video">
Video generation tool configuration and usage.
</Card>
<Card title="Music Generation" href="/tools/music-generation" icon="music">
Music and audio generation tool setup.
</Card>
<Card title="Provider Directory" href="/providers/index" icon="layers">
Overview of all providers and model refs.
</Card>
<Card title="Configuration reference" href="/gateway/config-agents#agent-defaults" icon="gear">
Full config reference including agent defaults.
</Card>
</CardGroup>