jason.woltje
193479b52d
Mosaic concepts pages now own the adapted content; source/license metadata under docs/reference/concepts. Adds ACT-1 agent-context planning capture, pinned concept test package + preparation utility, foundation observation notes (durability, evidence, federation, onboarding, workflow), and the #1495 consolidation assessment. TOOLS.md updated for the host-dev launcher.
235 lines
9.1 KiB
Markdown
235 lines
9.1 KiB
Markdown
---
|
|
summary: "Run OpenClaw through llmman (OpenAI-compatible local server)"
|
|
read_when:
|
|
- You want to run OpenClaw against a local llmman server
|
|
- You are serving Gemma or another model through llmman
|
|
- You need the exact OpenClaw compat flags for llmman
|
|
title: "llmman"
|
|
---
|
|
|
|
[llmman](https://github.com/llmmanorg/llmman) pulls GGUF/safetensors models from OCI registries and serves them behind Ollama-, OpenAI-, and Anthropic-compatible APIs. It uses `llama-server` for GGUF models and `vllm` or `mlx_lm.server` for safetensors models. OpenClaw talks to it through the generic `openai-completions` adapter.
|
|
|
|
| Property | Value |
|
|
| ---------------- | ------------------------------------------------------------ |
|
|
| Provider id | `llmman` (custom; configure under `models.providers.llmman`) |
|
|
| Plugin | none — not a bundled OpenClaw provider plugin |
|
|
| Auth env var | none required; any value works, `llmman serve` has no auth |
|
|
| API | OpenAI-compatible (`openai-completions`) |
|
|
| Default base URL | `http://127.0.0.1:17434/v1` |
|
|
|
|
<Note>
|
|
`llmman` is a custom self-hosted OpenAI-compatible backend, not a dedicated OpenClaw provider plugin: you configure it under `models.providers.llmman` instead of picking an onboarding auth choice. For a bundled plugin with auto-discovery, see [SGLang](/providers/sglang) or [vLLM](/providers/vllm).
|
|
</Note>
|
|
|
|
<Info>
|
|
Version scope: this page is verified against [llmman b315](https://github.com/llmmanorg/llmman/releases/tag/b315), commit [`0e7a3ed`](https://github.com/llmmanorg/llmman/commit/0e7a3ed815d49a74d7aad1b1c70b5eb6c3013b18).
|
|
</Info>
|
|
|
|
## Getting started
|
|
|
|
<Steps>
|
|
<Step title="Start llmman with a model">
|
|
```bash
|
|
LLMMAN_CONTEXT_LENGTH=65536 llmman serve gemma4
|
|
```
|
|
|
|
`llmman serve` listens on `127.0.0.1:17434` by default. Set `LLMMAN_HOST` before startup to override the bind address; there are no `--host`/`--port` flags. GPU acceleration (CUDA, ROCm, Vulkan, or Metal) is auto-detected; set `LLMMAN_LLM_LIBRARY` to override it because there is no `--device` flag. The model argument is optional — omit it to start the server and load models on the first request that names them instead.
|
|
|
|
The example fixes the server context at 65,536 tokens and uses the same value in OpenClaw below. If you change `LLMMAN_CONTEXT_LENGTH`, keep the OpenClaw model's `contextWindow` at or below that value.
|
|
|
|
</Step>
|
|
<Step title="Verify the server is reachable">
|
|
```bash
|
|
curl http://127.0.0.1:17434/v1/models
|
|
curl http://127.0.0.1:17434/api/version
|
|
```
|
|
|
|
`llmman serve` has no dedicated `/health` route at the top level; use `/v1/models` or `/api/version` for a readiness probe.
|
|
|
|
</Step>
|
|
<Step title="Add an OpenClaw provider entry">
|
|
Add an explicit provider entry and point your default model at it. See the config example below.
|
|
</Step>
|
|
</Steps>
|
|
|
|
## Full config example
|
|
|
|
Gemma 4 on a local `llmman` server:
|
|
|
|
```json5
|
|
{
|
|
agents: {
|
|
defaults: {
|
|
model: { primary: "llmman/gemma4" },
|
|
models: {
|
|
"llmman/gemma4": {
|
|
alias: "Gemma 4 (llmman)",
|
|
},
|
|
},
|
|
},
|
|
},
|
|
models: {
|
|
mode: "merge",
|
|
providers: {
|
|
llmman: {
|
|
baseUrl: "http://127.0.0.1:17434/v1",
|
|
apiKey: "llmman-local",
|
|
api: "openai-completions",
|
|
models: [
|
|
{
|
|
id: "gemma4",
|
|
name: "Gemma 4 (llmman)",
|
|
reasoning: false,
|
|
input: ["text"],
|
|
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
|
contextWindow: 65536,
|
|
maxTokens: 4096,
|
|
},
|
|
],
|
|
},
|
|
},
|
|
},
|
|
}
|
|
```
|
|
|
|
## On-demand startup
|
|
|
|
OpenClaw can start `llmman` itself only when an `llmman/...` model is selected. Add `localService` to the same provider entry:
|
|
|
|
```json5
|
|
{
|
|
models: {
|
|
providers: {
|
|
llmman: {
|
|
baseUrl: "http://127.0.0.1:17434/v1",
|
|
apiKey: "llmman-local",
|
|
api: "openai-completions",
|
|
timeoutSeconds: 300,
|
|
localService: {
|
|
command: "/opt/homebrew/bin/llmman",
|
|
args: ["serve", "gemma4"],
|
|
env: { LLMMAN_CONTEXT_LENGTH: "65536" },
|
|
healthUrl: "http://127.0.0.1:17434/v1/models",
|
|
readyTimeoutMs: 180000,
|
|
idleStopMs: 0,
|
|
},
|
|
models: [
|
|
{
|
|
id: "gemma4",
|
|
name: "Gemma 4 (llmman)",
|
|
reasoning: false,
|
|
input: ["text"],
|
|
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
|
contextWindow: 65536,
|
|
maxTokens: 4096,
|
|
},
|
|
],
|
|
},
|
|
},
|
|
},
|
|
}
|
|
```
|
|
|
|
`command` must be an absolute path. Run `which llmman` on the Gateway host and use that path. Full field reference: [Local model services](/gateway/local-model-services).
|
|
|
|
## Advanced configuration
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="Why requiresStringContent might matter">
|
|
`llmman` resolves and loads the requested model, rewrites its id for the selected backend, and adds generation defaults such as `repeat_penalty`. It forwards message content and tool schemas without normalizing them, so compatibility for those fields depends on the selected backend and model.
|
|
|
|
<Warning>
|
|
If OpenClaw runs fail with:
|
|
|
|
```text
|
|
messages[1].content: invalid type: sequence, expected a string
|
|
```
|
|
|
|
set `compat.requiresStringContent: true` in the model entry. OpenClaw then flattens pure text content parts into plain strings before sending the request.
|
|
</Warning>
|
|
|
|
</Accordion>
|
|
|
|
<Accordion title="Tool-schema caveat">
|
|
If a model accepts small direct `/v1/chat/completions` requests but fails on full OpenClaw agent-runtime turns, try disabling the tool schema surface first:
|
|
|
|
```json5
|
|
compat: {
|
|
supportsTools: false
|
|
}
|
|
```
|
|
|
|
That reduces prompt pressure on stricter local backends. If tiny direct requests still work but normal OpenClaw agent turns keep crashing inside `llama-server`, treat it as an upstream model/server limitation rather than an OpenClaw transport issue.
|
|
|
|
</Accordion>
|
|
|
|
<Accordion title="Manual smoke test">
|
|
Test both layers once configured:
|
|
|
|
```bash
|
|
curl http://127.0.0.1:17434/v1/chat/completions \
|
|
-H 'content-type: application/json' \
|
|
-d '{"model":"gemma4","messages":[{"role":"user","content":"What is 2 + 2?"}],"stream":false}'
|
|
```
|
|
|
|
```bash
|
|
openclaw infer model run \
|
|
--model llmman/gemma4 \
|
|
--prompt "What is 2 + 2? Reply with one short sentence." \
|
|
--json
|
|
```
|
|
|
|
If the first command works but the second fails, see Troubleshooting below.
|
|
|
|
</Accordion>
|
|
|
|
<Accordion title="Proxy-style behavior">
|
|
Because `llmman` uses the generic `openai-completions` adapter (not `openai-responses`), native-OpenAI-only request shaping never applies: no `service_tier`, no Responses `store`, no prompt-cache hints, and no OpenAI reasoning-compat payload shaping get sent.
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
## Troubleshooting
|
|
|
|
<AccordionGroup>
|
|
<Accordion title="curl /v1/models fails">
|
|
`llmman serve` is not running or is not reachable at the configured address. The default is `127.0.0.1:17434`; if you set `LLMMAN_HOST`, update the OpenClaw `baseUrl` and `healthUrl` to match.
|
|
</Accordion>
|
|
|
|
<Accordion title="messages[].content expected a string">
|
|
Set `compat.requiresStringContent: true` in the model entry (see above).
|
|
</Accordion>
|
|
|
|
<Accordion title="Direct /v1/chat/completions calls pass but openclaw infer model run fails">
|
|
Both probes are tool-free, so `compat.supportsTools` cannot change this failure. Check the configured base URL and model id, inspect the `llmman`/backend logs, and compare the two request payloads and responses.
|
|
</Accordion>
|
|
|
|
<Accordion title="Model run passes but a normal agent turn fails">
|
|
The agent turn includes a larger prompt and may include tool schemas. Try `compat.supportsTools: false` to isolate tool-schema pressure (see the tool-schema caveat above).
|
|
</Accordion>
|
|
|
|
<Accordion title="llama-server still crashes on larger agent turns">
|
|
If schema errors are gone but the spawned `llama-server` still crashes on larger agent turns, treat it as an upstream `llama.cpp` or model limitation. Reduce prompt pressure or switch backend/model.
|
|
</Accordion>
|
|
</AccordionGroup>
|
|
|
|
<Tip>
|
|
For general help, see [Troubleshooting](/help/troubleshooting) and [FAQ](/help/faq).
|
|
</Tip>
|
|
|
|
## Related
|
|
|
|
<CardGroup cols={2}>
|
|
<Card title="Local models" href="/gateway/local-models" icon="server">
|
|
Running OpenClaw against local model servers.
|
|
</Card>
|
|
<Card title="Local model services" href="/gateway/local-model-services" icon="play">
|
|
Starting local model servers on demand for configured providers.
|
|
</Card>
|
|
<Card title="Gateway troubleshooting" href="/gateway/troubleshooting#local-openai-compatible-backend-passes-direct-probes-but-agent-runs-fail" icon="wrench">
|
|
Debugging local OpenAI-compatible backends that pass probes but fail agent runs.
|
|
</Card>
|
|
<Card title="Model selection" href="/concepts/model-providers" icon="layers">
|
|
Overview of all providers, model refs, and failover behavior.
|
|
</Card>
|
|
</CardGroup>
|