# Use vLLM open models with crystl

> Install vLLM on a GPU server, serve a tool-capable model, connect it to crystl, and start a Codex shard.

This tutorial connects crystl on your Mac to vLLM on a Linux GPU machine. vLLM is built for high-throughput serving and exposes an OpenAI-compatible API that Codex can use.

The model server receives every prompt, file excerpt, and tool result the agent sends. Keep it on infrastructure you trust.

## 1. Prepare the model server

Follow [vLLM's installation guide](https://docs.vllm.ai/en/latest/getting_started/installation/) for your hardware. For a current NVIDIA setup, vLLM recommends an isolated environment managed by `uv`:

```bash
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
```

The exact wheel and driver requirements differ for NVIDIA, AMD, CPU, and other accelerators. Finish vLLM's hardware-specific checks before continuing.

## 2. Serve a tool-capable model

This example uses Qwen2.5-Coder 7B. vLLM's tool-calling guide specifies the Hermes parser for Qwen2.5 models:

```bash
vllm serve Qwen/Qwen2.5-Coder-7B-Instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --api-key '<choose-a-long-random-key>'
```

The first launch downloads the model from Hugging Face. Substitute another model only after checking vLLM's [tool-calling flags](https://docs.vllm.ai/en/latest/features/tool_calling/) for that model; a text-only chat model is not enough for a coding agent.

Binding to `0.0.0.0` makes the server reachable from the network. Limit port `8000` to your LAN or Tailscale network and keep the API key enabled. Use `127.0.0.1` instead when crystl and vLLM run on the same machine.

## 3. Check vLLM directly

From your Mac, replace the host and key, then run:

```bash
curl http://<server-address>:8000/v1/models \
  -H 'Authorization: Bearer <the-same-key>'
```

The response should list `Qwen/Qwen2.5-Coder-7B-Instruct`. If this request cannot connect, fix the server address, bind setting, or firewall before opening crystl.

## 4. Add vLLM to crystl

Open **Settings → agents → providers & keys**, scroll to the open-model connections, and add:

| Field | Value |
|-------|-------|
| Label | `vllm-box` |
| Kind | OpenAI-compatible |
| Base URL | `http://<server-address>:8000/v1` |
| As seen from remote | `http://localhost:8000/v1` if agents also run on that server |
| Key | the value passed to `--api-key` |
| Context window | the value passed to `--max-model-len` |
| Enabled | on |

Unlike an Anthropic-compatible connection, this base URL includes `/v1`.

The context window field is optional, and it is worth filling in. crystl uses it to warn you when an endpoint is too small for the work you are about to start, which is the most common cause of an agent producing broken tool calls or stopping partway. Record the context length your server is actually configured for.

## 5. Start a Codex shard

```bash
crystl shard create --gem myapp --agent codex --local vllm-box \
  --model Qwen/Qwen2.5-Coder-7B-Instruct \
  --prompt "inspect this project and suggest the best first task"
```

`crystl shard create` is a Guild control command. You can create the same shard from the desktop app without Guild.

crystl supplies the connection URL and key, adds Codex's local-model flags, and preserves the model id. Give the shard a small task that reads a file and runs a command to confirm that tool calls work, not just chat.

## Troubleshooting

**The model will not start:** the selected weights or requested context do not fit the available GPU memory. Lower `--max-model-len`, choose a smaller model, or follow vLLM's tensor-parallel guidance for multiple GPUs.

**The API returns 401:** the crystl connection key and vLLM's `--api-key` value do not match.

**Chat works but tools do not:** confirm the model supports tool calling and that the server has both `--enable-auto-tool-choice` and the parser required for that model.

**Codex says model not found:** copy the exact id returned by `/v1/models`. If you use vLLM's `--served-model-name`, pass that name instead of the Hugging Face id.

## Other setup tutorials

- [Ollama](/docs/open-models-ollama/): a local runtime you drive from the command line.
- [LM Studio](/docs/open-models-lm-studio/): a graphical local runtime with an Anthropic-compatible server.
- [llama.cpp](/docs/open-models-llama-cpp/): a smaller single-binary server with direct control over quantization.
- [z.ai](/docs/open-models-zai/): hosted GLM models, no local GPU.

## Related

- [Open Models](/docs/open-models/) for orchestration, keys, and remote-vantage URLs.
- [vLLM's Codex integration](https://docs.vllm.ai/en/stable/serving/integrations/codex/) for the underlying agent requirements.

---
Source: https://crystl.dev/docs/open-models-vllm/