jason.woltje
193479b52d
Mosaic concepts pages now own the adapted content; source/license metadata under docs/reference/concepts. Adds ACT-1 agent-context planning capture, pinned concept test package + preparation utility, foundation observation notes (durability, evidence, federation, onboarding, workflow), and the #1495 consolidation assessment. TOOLS.md updated for the host-dev launcher.
9.1 KiB
9.1 KiB
summary, read_when, title
| summary | read_when | title | |||
|---|---|---|---|---|---|
| Run OpenClaw through llmman (OpenAI-compatible local server) |
|
llmman |
llmman pulls GGUF/safetensors models from OCI registries and serves them behind Ollama-, OpenAI-, and Anthropic-compatible APIs. It uses llama-server for GGUF models and vllm or mlx_lm.server for safetensors models. OpenClaw talks to it through the generic openai-completions adapter.
| Property | Value |
|---|---|
| Provider id | llmman (custom; configure under models.providers.llmman) |
| Plugin | none — not a bundled OpenClaw provider plugin |
| Auth env var | none required; any value works, llmman serve has no auth |
| API | OpenAI-compatible (openai-completions) |
| Default base URL | http://127.0.0.1:17434/v1 |
Getting started
```bash LLMMAN_CONTEXT_LENGTH=65536 llmman serve gemma4 ````llmman serve` listens on `127.0.0.1:17434` by default. Set `LLMMAN_HOST` before startup to override the bind address; there are no `--host`/`--port` flags. GPU acceleration (CUDA, ROCm, Vulkan, or Metal) is auto-detected; set `LLMMAN_LLM_LIBRARY` to override it because there is no `--device` flag. The model argument is optional — omit it to start the server and load models on the first request that names them instead.
The example fixes the server context at 65,536 tokens and uses the same value in OpenClaw below. If you change `LLMMAN_CONTEXT_LENGTH`, keep the OpenClaw model's `contextWindow` at or below that value.
`llmman serve` has no dedicated `/health` route at the top level; use `/v1/models` or `/api/version` for a readiness probe.
Full config example
Gemma 4 on a local llmman server:
{
agents: {
defaults: {
model: { primary: "llmman/gemma4" },
models: {
"llmman/gemma4": {
alias: "Gemma 4 (llmman)",
},
},
},
},
models: {
mode: "merge",
providers: {
llmman: {
baseUrl: "http://127.0.0.1:17434/v1",
apiKey: "llmman-local",
api: "openai-completions",
models: [
{
id: "gemma4",
name: "Gemma 4 (llmman)",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 65536,
maxTokens: 4096,
},
],
},
},
},
}
On-demand startup
OpenClaw can start llmman itself only when an llmman/... model is selected. Add localService to the same provider entry:
{
models: {
providers: {
llmman: {
baseUrl: "http://127.0.0.1:17434/v1",
apiKey: "llmman-local",
api: "openai-completions",
timeoutSeconds: 300,
localService: {
command: "/opt/homebrew/bin/llmman",
args: ["serve", "gemma4"],
env: { LLMMAN_CONTEXT_LENGTH: "65536" },
healthUrl: "http://127.0.0.1:17434/v1/models",
readyTimeoutMs: 180000,
idleStopMs: 0,
},
models: [
{
id: "gemma4",
name: "Gemma 4 (llmman)",
reasoning: false,
input: ["text"],
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
contextWindow: 65536,
maxTokens: 4096,
},
],
},
},
},
}
command must be an absolute path. Run which llmman on the Gateway host and use that path. Full field reference: Local model services.
Advanced configuration
`llmman` resolves and loads the requested model, rewrites its id for the selected backend, and adds generation defaults such as `repeat_penalty`. It forwards message content and tool schemas without normalizing them, so compatibility for those fields depends on the selected backend and model.<Warning>
If OpenClaw runs fail with:
```text
messages[1].content: invalid type: sequence, expected a string
```
set `compat.requiresStringContent: true` in the model entry. OpenClaw then flattens pure text content parts into plain strings before sending the request.
</Warning>
```json5
compat: {
supportsTools: false
}
```
That reduces prompt pressure on stricter local backends. If tiny direct requests still work but normal OpenClaw agent turns keep crashing inside `llama-server`, treat it as an upstream model/server limitation rather than an OpenClaw transport issue.
```bash
curl http://127.0.0.1:17434/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"gemma4","messages":[{"role":"user","content":"What is 2 + 2?"}],"stream":false}'
```
```bash
openclaw infer model run \
--model llmman/gemma4 \
--prompt "What is 2 + 2? Reply with one short sentence." \
--json
```
If the first command works but the second fails, see Troubleshooting below.