# Use llama.cpp open models with crystl

> Install llama.cpp, download and serve a GGUF coding model, connect it to crystl, and start a Codex shard.

This tutorial runs a GGUF coding model with llama.cpp and connects it to a Codex shard. llama.cpp is a good fit when you want a small native server, direct control over quantization, or broad CPU and GPU support.

## 1. Install llama.cpp

On macOS or Linux with Homebrew:

```bash
brew install llama.cpp
```

The project also publishes binaries and build instructions in its [installation guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md). Metal support is enabled by default on macOS builds.

## 2. Start a coding model

This example downloads a Qwen2.5-Coder 7B GGUF from the llama.cpp organization's Hugging Face account and starts the OpenAI-compatible server:

```bash
llama-server \
  -hf ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF \
  --ctx-size 32768 \
  --jinja \
  --host 127.0.0.1 \
  --port 8080
```

The first run downloads the model. `--jinja` enables its tool-aware chat template; without structured tool calls, Codex can chat but cannot reliably operate on a project. The Q8 model is roughly 8 GB before context and runtime overhead. Choose a different GGUF quantization or model if it does not fit your machine.

To serve a GGUF file you already have, replace `-hf …` with `--model /absolute/path/to/model.gguf`.

## 3. Check llama.cpp directly

In another terminal:

```bash
curl http://localhost:8080/v1/models
```

Copy the model's exact `id` from the response. You will pass that value to Codex.

## 4. Add llama.cpp to crystl

Open **Settings → agents → providers & keys**, scroll to the open-model connections, and add:

| Field | Value |
|-------|-------|
| Label | `llama-cpp` |
| Kind | OpenAI-compatible |
| Base URL | `http://localhost:8080/v1` |
| As seen from remote | leave empty |
| Key | leave empty |
| Context window | the value passed to `-c` / `--ctx-size` |
| Enabled | on |

Save the connection, then run `crystl status`. It should report `llama-cpp` as reachable.

The context window field is optional, and it is worth filling in. crystl uses it to warn you when an endpoint is too small for the work you are about to start, which is the most common cause of an agent producing broken tool calls or stopping partway. Record the context length your server is actually configured for.

## 5. Start a Codex shard

Replace `<model-id>` with the exact value returned by `/v1/models`:

```bash
crystl shard create --gem myapp --agent codex --local llama-cpp \
  --model '<model-id>' \
  --prompt "inspect this project and suggest the best first task"
```

`crystl shard create` is a Guild control command. You can create the same shard from the desktop app without Guild.

Give the shard a small task that reads a file and runs a command. A plain text response proves generation works; a completed tool-using task proves the agent path works.

## Run llama.cpp on another machine

Change the server bind address to `--host 0.0.0.0`, then use the machine's private LAN or Tailscale address in crystl. Protect the port with a firewall and use llama.cpp's API-key option if more than your own machine can reach it. If an SSH shard runs on the model machine, set **As seen from remote** to `http://localhost:8080/v1`.

## Troubleshooting

**The server exits while loading:** the model or context is too large. Choose a smaller quantization, lower `--ctx-size`, or reduce GPU offload.

**Codex says model not found:** use the exact id from `curl http://localhost:8080/v1/models`, not the GGUF filename you expected.

**The agent prints tool syntax instead of using tools:** keep `--jinja` enabled and choose a model whose chat template supports tool use.

**The connection works locally but not from your Mac:** bind to a reachable private interface, check the firewall, and test the `/v1/models` URL from the Mac before retrying crystl.

## Other setup tutorials

- [Ollama](/docs/open-models-ollama/): a local runtime you drive from the command line.
- [LM Studio](/docs/open-models-lm-studio/): a graphical local runtime with an Anthropic-compatible server.
- [vLLM](/docs/open-models-vllm/): a higher-throughput GPU server for Linux.
- [z.ai](/docs/open-models-zai/): hosted GLM models, no local GPU.

## Related

- [Open Models](/docs/open-models/) for orchestration, exact model ids, and privacy boundaries.
- [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) for every server and security option.

---
Source: https://crystl.dev/docs/open-models-llama-cpp/