Use llama.cpp open models with crystl

This tutorial runs a GGUF coding model with llama.cpp and connects it to a Codex shard. llama.cpp is a good fit when you want a small native server, direct control over quantization, or broad CPU and GPU support.

1. Install llama.cpp

On macOS or Linux with Homebrew:

brew install llama.cpp

The project also publishes binaries and build instructions in its installation guide. Metal support is enabled by default on macOS builds.

2. Start a coding model

This example downloads a Qwen2.5-Coder 7B GGUF from the llama.cpp organization’s Hugging Face account and starts the OpenAI-compatible server:

llama-server \
  -hf ggml-org/Qwen2.5-Coder-7B-Q8_0-GGUF \
  --ctx-size 32768 \
  --jinja \
  --host 127.0.0.1 \
  --port 8080

The first run downloads the model. --jinja enables its tool-aware chat template; without structured tool calls, Codex can chat but cannot reliably operate on a project. The Q8 model is roughly 8 GB before context and runtime overhead. Choose a different GGUF quantization or model if it does not fit your machine.

To serve a GGUF file you already have, replace -hf … with --model /absolute/path/to/model.gguf.

3. Check llama.cpp directly

In another terminal:

curl http://localhost:8080/v1/models

Copy the model’s exact id from the response. You will pass that value to Codex.

4. Add llama.cpp to crystl

Open Settings → agents → providers & keys, scroll to the open-model connections, and add:

Field	Value
Label	`llama-cpp`
Kind	OpenAI-compatible
Base URL	`http://localhost:8080/v1`
As seen from remote	leave empty
Key	leave empty
Context window	the value passed to `-c` / `--ctx-size`
Enabled	on

Save the connection, then run crystl status. It should report llama-cpp as reachable.

The context window field is optional, and it is worth filling in. crystl uses it to warn you when an endpoint is too small for the work you are about to start, which is the most common cause of an agent producing broken tool calls or stopping partway. Record the context length your server is actually configured for.

5. Start a Codex shard

Replace <model-id> with the exact value returned by /v1/models:

crystl shard create --gem myapp --agent codex --local llama-cpp \
  --model '<model-id>' \
  --prompt "inspect this project and suggest the best first task"

crystl shard create is a Guild control command. You can create the same shard from the desktop app without Guild.

Give the shard a small task that reads a file and runs a command. A plain text response proves generation works; a completed tool-using task proves the agent path works.

Run llama.cpp on another machine

Change the server bind address to --host 0.0.0.0, then use the machine’s private LAN or Tailscale address in crystl. Protect the port with a firewall and use llama.cpp’s API-key option if more than your own machine can reach it. If an SSH shard runs on the model machine, set As seen from remote to http://localhost:8080/v1.

Troubleshooting

The server exits while loading: the model or context is too large. Choose a smaller quantization, lower --ctx-size, or reduce GPU offload.

Codex says model not found: use the exact id from curl http://localhost:8080/v1/models, not the GGUF filename you expected.

The agent prints tool syntax instead of using tools: keep --jinja enabled and choose a model whose chat template supports tool use.

The connection works locally but not from your Mac: bind to a reachable private interface, check the firewall, and test the /v1/models URL from the Mac before retrying crystl.