Use vLLM open models with crystl

This tutorial connects crystl on your Mac to vLLM on a Linux GPU machine. vLLM is built for high-throughput serving and exposes an OpenAI-compatible API that Codex can use.

The model server receives every prompt, file excerpt, and tool result the agent sends. Keep it on infrastructure you trust.

1. Prepare the model server

Follow vLLM’s installation guide for your hardware. For a current NVIDIA setup, vLLM recommends an isolated environment managed by uv:

uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto

The exact wheel and driver requirements differ for NVIDIA, AMD, CPU, and other accelerators. Finish vLLM’s hardware-specific checks before continuing.

2. Serve a tool-capable model

This example uses Qwen2.5-Coder 7B. vLLM’s tool-calling guide specifies the Hermes parser for Qwen2.5 models:

vllm serve Qwen/Qwen2.5-Coder-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --api-key '<choose-a-long-random-key>'

The first launch downloads the model from Hugging Face. Substitute another model only after checking vLLM’s tool-calling flags for that model; a text-only chat model is not enough for a coding agent.

Binding to 0.0.0.0 makes the server reachable from the network. Limit port 8000 to your LAN or Tailscale network and keep the API key enabled. Use 127.0.0.1 instead when crystl and vLLM run on the same machine.

3. Check vLLM directly

From your Mac, replace the host and key, then run:

curl http://<server-address>:8000/v1/models \
  -H 'Authorization: Bearer <the-same-key>'

The response should list Qwen/Qwen2.5-Coder-7B-Instruct. If this request cannot connect, fix the server address, bind setting, or firewall before opening crystl.

4. Add vLLM to crystl

Open Settings → agents → providers & keys, scroll to the open-model connections, and add:

Field	Value
Label	`vllm-box`
Kind	OpenAI-compatible
Base URL	`http://<server-address>:8000/v1`
As seen from remote	`http://localhost:8000/v1` if agents also run on that server
Key	the value passed to `--api-key`
Context window	the value passed to `--max-model-len`
Enabled	on

Unlike an Anthropic-compatible connection, this base URL includes /v1.

The context window field is optional, and it is worth filling in. crystl uses it to warn you when an endpoint is too small for the work you are about to start, which is the most common cause of an agent producing broken tool calls or stopping partway. Record the context length your server is actually configured for.

5. Start a Codex shard

crystl shard create --gem myapp --agent codex --local vllm-box \
  --model Qwen/Qwen2.5-Coder-7B-Instruct \
  --prompt "inspect this project and suggest the best first task"

crystl shard create is a Guild control command. You can create the same shard from the desktop app without Guild.

crystl supplies the connection URL and key, adds Codex’s local-model flags, and preserves the model id. Give the shard a small task that reads a file and runs a command to confirm that tool calls work, not just chat.

Troubleshooting

The model will not start: the selected weights or requested context do not fit the available GPU memory. Lower --max-model-len, choose a smaller model, or follow vLLM’s tensor-parallel guidance for multiple GPUs.

The API returns 401: the crystl connection key and vLLM’s --api-key value do not match.

Chat works but tools do not: confirm the model supports tool calling and that the server has both --enable-auto-tool-choice and the parser required for that model.

Codex says model not found: copy the exact id returned by /v1/models. If you use vLLM’s --served-model-name, pass that name instead of the Hugging Face id.