docs: concept annexation, provider/reference docs, ACT-1 groundwork
Mosaic concepts pages now own the adapted content; source/license metadata under docs/reference/concepts. Adds ACT-1 agent-context planning capture, pinned concept test package + preparation utility, foundation observation notes (durability, evidence, federation, onboarding, workflow), and the #1495 consolidation assessment. TOOLS.md updated for the host-dev launcher.
This commit is contained in:
@@ -0,0 +1,234 @@
|
||||
---
|
||||
summary: "Run OpenClaw through llmman (OpenAI-compatible local server)"
|
||||
read_when:
|
||||
- You want to run OpenClaw against a local llmman server
|
||||
- You are serving Gemma or another model through llmman
|
||||
- You need the exact OpenClaw compat flags for llmman
|
||||
title: "llmman"
|
||||
---
|
||||
|
||||
[llmman](https://github.com/llmmanorg/llmman) pulls GGUF/safetensors models from OCI registries and serves them behind Ollama-, OpenAI-, and Anthropic-compatible APIs. It uses `llama-server` for GGUF models and `vllm` or `mlx_lm.server` for safetensors models. OpenClaw talks to it through the generic `openai-completions` adapter.
|
||||
|
||||
| Property | Value |
|
||||
| ---------------- | ------------------------------------------------------------ |
|
||||
| Provider id | `llmman` (custom; configure under `models.providers.llmman`) |
|
||||
| Plugin | none — not a bundled OpenClaw provider plugin |
|
||||
| Auth env var | none required; any value works, `llmman serve` has no auth |
|
||||
| API | OpenAI-compatible (`openai-completions`) |
|
||||
| Default base URL | `http://127.0.0.1:17434/v1` |
|
||||
|
||||
<Note>
|
||||
`llmman` is a custom self-hosted OpenAI-compatible backend, not a dedicated OpenClaw provider plugin: you configure it under `models.providers.llmman` instead of picking an onboarding auth choice. For a bundled plugin with auto-discovery, see [SGLang](/providers/sglang) or [vLLM](/providers/vllm).
|
||||
</Note>
|
||||
|
||||
<Info>
|
||||
Version scope: this page is verified against [llmman b315](https://github.com/llmmanorg/llmman/releases/tag/b315), commit [`0e7a3ed`](https://github.com/llmmanorg/llmman/commit/0e7a3ed815d49a74d7aad1b1c70b5eb6c3013b18).
|
||||
</Info>
|
||||
|
||||
## Getting started
|
||||
|
||||
<Steps>
|
||||
<Step title="Start llmman with a model">
|
||||
```bash
|
||||
LLMMAN_CONTEXT_LENGTH=65536 llmman serve gemma4
|
||||
```
|
||||
|
||||
`llmman serve` listens on `127.0.0.1:17434` by default. Set `LLMMAN_HOST` before startup to override the bind address; there are no `--host`/`--port` flags. GPU acceleration (CUDA, ROCm, Vulkan, or Metal) is auto-detected; set `LLMMAN_LLM_LIBRARY` to override it because there is no `--device` flag. The model argument is optional — omit it to start the server and load models on the first request that names them instead.
|
||||
|
||||
The example fixes the server context at 65,536 tokens and uses the same value in OpenClaw below. If you change `LLMMAN_CONTEXT_LENGTH`, keep the OpenClaw model's `contextWindow` at or below that value.
|
||||
|
||||
</Step>
|
||||
<Step title="Verify the server is reachable">
|
||||
```bash
|
||||
curl http://127.0.0.1:17434/v1/models
|
||||
curl http://127.0.0.1:17434/api/version
|
||||
```
|
||||
|
||||
`llmman serve` has no dedicated `/health` route at the top level; use `/v1/models` or `/api/version` for a readiness probe.
|
||||
|
||||
</Step>
|
||||
<Step title="Add an OpenClaw provider entry">
|
||||
Add an explicit provider entry and point your default model at it. See the config example below.
|
||||
</Step>
|
||||
</Steps>
|
||||
|
||||
## Full config example
|
||||
|
||||
Gemma 4 on a local `llmman` server:
|
||||
|
||||
```json5
|
||||
{
|
||||
agents: {
|
||||
defaults: {
|
||||
model: { primary: "llmman/gemma4" },
|
||||
models: {
|
||||
"llmman/gemma4": {
|
||||
alias: "Gemma 4 (llmman)",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
models: {
|
||||
mode: "merge",
|
||||
providers: {
|
||||
llmman: {
|
||||
baseUrl: "http://127.0.0.1:17434/v1",
|
||||
apiKey: "llmman-local",
|
||||
api: "openai-completions",
|
||||
models: [
|
||||
{
|
||||
id: "gemma4",
|
||||
name: "Gemma 4 (llmman)",
|
||||
reasoning: false,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 65536,
|
||||
maxTokens: 4096,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## On-demand startup
|
||||
|
||||
OpenClaw can start `llmman` itself only when an `llmman/...` model is selected. Add `localService` to the same provider entry:
|
||||
|
||||
```json5
|
||||
{
|
||||
models: {
|
||||
providers: {
|
||||
llmman: {
|
||||
baseUrl: "http://127.0.0.1:17434/v1",
|
||||
apiKey: "llmman-local",
|
||||
api: "openai-completions",
|
||||
timeoutSeconds: 300,
|
||||
localService: {
|
||||
command: "/opt/homebrew/bin/llmman",
|
||||
args: ["serve", "gemma4"],
|
||||
env: { LLMMAN_CONTEXT_LENGTH: "65536" },
|
||||
healthUrl: "http://127.0.0.1:17434/v1/models",
|
||||
readyTimeoutMs: 180000,
|
||||
idleStopMs: 0,
|
||||
},
|
||||
models: [
|
||||
{
|
||||
id: "gemma4",
|
||||
name: "Gemma 4 (llmman)",
|
||||
reasoning: false,
|
||||
input: ["text"],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 65536,
|
||||
maxTokens: 4096,
|
||||
},
|
||||
],
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
`command` must be an absolute path. Run `which llmman` on the Gateway host and use that path. Full field reference: [Local model services](/gateway/local-model-services).
|
||||
|
||||
## Advanced configuration
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="Why requiresStringContent might matter">
|
||||
`llmman` resolves and loads the requested model, rewrites its id for the selected backend, and adds generation defaults such as `repeat_penalty`. It forwards message content and tool schemas without normalizing them, so compatibility for those fields depends on the selected backend and model.
|
||||
|
||||
<Warning>
|
||||
If OpenClaw runs fail with:
|
||||
|
||||
```text
|
||||
messages[1].content: invalid type: sequence, expected a string
|
||||
```
|
||||
|
||||
set `compat.requiresStringContent: true` in the model entry. OpenClaw then flattens pure text content parts into plain strings before sending the request.
|
||||
</Warning>
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Tool-schema caveat">
|
||||
If a model accepts small direct `/v1/chat/completions` requests but fails on full OpenClaw agent-runtime turns, try disabling the tool schema surface first:
|
||||
|
||||
```json5
|
||||
compat: {
|
||||
supportsTools: false
|
||||
}
|
||||
```
|
||||
|
||||
That reduces prompt pressure on stricter local backends. If tiny direct requests still work but normal OpenClaw agent turns keep crashing inside `llama-server`, treat it as an upstream model/server limitation rather than an OpenClaw transport issue.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Manual smoke test">
|
||||
Test both layers once configured:
|
||||
|
||||
```bash
|
||||
curl http://127.0.0.1:17434/v1/chat/completions \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{"model":"gemma4","messages":[{"role":"user","content":"What is 2 + 2?"}],"stream":false}'
|
||||
```
|
||||
|
||||
```bash
|
||||
openclaw infer model run \
|
||||
--model llmman/gemma4 \
|
||||
--prompt "What is 2 + 2? Reply with one short sentence." \
|
||||
--json
|
||||
```
|
||||
|
||||
If the first command works but the second fails, see Troubleshooting below.
|
||||
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Proxy-style behavior">
|
||||
Because `llmman` uses the generic `openai-completions` adapter (not `openai-responses`), native-OpenAI-only request shaping never applies: no `service_tier`, no Responses `store`, no prompt-cache hints, and no OpenAI reasoning-compat payload shaping get sent.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
<AccordionGroup>
|
||||
<Accordion title="curl /v1/models fails">
|
||||
`llmman serve` is not running or is not reachable at the configured address. The default is `127.0.0.1:17434`; if you set `LLMMAN_HOST`, update the OpenClaw `baseUrl` and `healthUrl` to match.
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="messages[].content expected a string">
|
||||
Set `compat.requiresStringContent: true` in the model entry (see above).
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Direct /v1/chat/completions calls pass but openclaw infer model run fails">
|
||||
Both probes are tool-free, so `compat.supportsTools` cannot change this failure. Check the configured base URL and model id, inspect the `llmman`/backend logs, and compare the two request payloads and responses.
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="Model run passes but a normal agent turn fails">
|
||||
The agent turn includes a larger prompt and may include tool schemas. Try `compat.supportsTools: false` to isolate tool-schema pressure (see the tool-schema caveat above).
|
||||
</Accordion>
|
||||
|
||||
<Accordion title="llama-server still crashes on larger agent turns">
|
||||
If schema errors are gone but the spawned `llama-server` still crashes on larger agent turns, treat it as an upstream `llama.cpp` or model limitation. Reduce prompt pressure or switch backend/model.
|
||||
</Accordion>
|
||||
</AccordionGroup>
|
||||
|
||||
<Tip>
|
||||
For general help, see [Troubleshooting](/help/troubleshooting) and [FAQ](/help/faq).
|
||||
</Tip>
|
||||
|
||||
## Related
|
||||
|
||||
<CardGroup cols={2}>
|
||||
<Card title="Local models" href="/gateway/local-models" icon="server">
|
||||
Running OpenClaw against local model servers.
|
||||
</Card>
|
||||
<Card title="Local model services" href="/gateway/local-model-services" icon="play">
|
||||
Starting local model servers on demand for configured providers.
|
||||
</Card>
|
||||
<Card title="Gateway troubleshooting" href="/gateway/troubleshooting#local-openai-compatible-backend-passes-direct-probes-but-agent-runs-fail" icon="wrench">
|
||||
Debugging local OpenAI-compatible backends that pass probes but fail agent runs.
|
||||
</Card>
|
||||
<Card title="Model selection" href="/concepts/model-providers" icon="layers">
|
||||
Overview of all providers, model refs, and failover behavior.
|
||||
</Card>
|
||||
</CardGroup>
|
||||
Reference in New Issue
Block a user