Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français
Lire en français

llama.cpp: an LLM inference server on CPU and GPU

17 min read
Improve this page

You may already use Ollama to run an LLM locally. Under the hood, it is llama.cpp that loads the model and generates the tokens. This guide shows how to use llama.cpp directly: why go down to that level, how to launch llama-server (the OpenAI-compatible HTTP server), how to force JSON outputs and how to measure performance with llama-bench. Target audience: a developer or administrator comfortable with Docker who wants fine-grained control over local inference without depending on a SaaS service.

What you will learn

  • Understand what llama.cpp brings compared with Ollama and when to choose it.
  • Launch an OpenAI-compatible inference server with Docker Compose.
  • Query the REST API and force a valid JSON answer with a schema.
  • Measure the CPU throughput (tokens/second) with the official llama-bench tool.
  • Know how to enable a GPU (CUDA, Vulkan, Metal) and harden the network exposure.

Prerequisites

This guide can be reproduced entirely on a CPU machine, with no graphics card. You need:

  • Docker and the Docker Compose plugin installed and working.
  • About 2 GB of disk space for the test model and 4 GB of free RAM.
  • Some knowledge of the command line and of HTTP requests (curl).

If the notions of inference or quantization do not mean much to you yet, first read the guides Understanding LLM inference (in French) and Understanding quantization (in French): they set up the vocabulary used here. To place llama.cpp among the other engines, see the comparison of inference backends (in French).

llama.cpp, the engine behind Ollama

llama.cpp is an open source project (MIT license) written in C/C++ that runs language models with one precise goal: run everywhere, from a Raspberry Pi to a multi-GPU server. It is the inference building block that mainstream tools such as Ollama or LM Studio embed without saying so. Learning to use it directly means understanding what really happens when your local model answers.

The project ships several complementary binaries. llama-server exposes an HTTP server with an OpenAI-compatible API: it is the one you will put in production. llama-cli offers an interactive terminal mode, handy for a quick test. llama-bench measures the inference throughput in a reproducible way. llama-quantize converts a model to a lighter format, the subject of a dedicated guide.

llama.cpp reads only the GGUF format (GPT-Generated Unified Format), a single container that embeds the model weights, the vocabulary and the metadata. A GGUF model is already quantized: its weights are compressed to 4, 5 or 8 bits instead of the original 16 bits, which divides the size by three to four and makes CPU execution realistic.

Why go down to the llama.cpp level

Ollama remains the simplest choice to get started. But three needs justify using llama.cpp directly rather than the abstraction layer.

The first is control over parameters. Ollama hides many settings behind its default values. With llama-server, you explicitly drive the context size, the number of threads, the batch and the layers offloaded to the GPU. The second is hardware portability: llama.cpp compiles for CUDA (NVIDIA), Vulkan (Intel/AMD iGPUs and mainstream cards), Metal (Apple Silicon) or ROCm (AMD), where Ollama follows a subset. The third is measurement: llama-bench gives comparable figures that you can feed into an architecture decision.

Launching llama-server with Docker

The llama.cpp team publishes official images on the ghcr.io/ggml-org/llama.cpp registry. The :server image contains the ready-to-use llama-server binary. This approach avoids any compilation and guarantees a reproducible environment.

  1. Create the working folder and download a GGUF model.

    The test model is Qwen2.5-1.5B-Instruct, a compact Alibaba model that answers correctly in French. It is quantized in Q4_K_M (a little under 5 bits per weight on average), about 940 MB, small enough to run comfortably on CPU.

    Terminal window
    mkdir -p llama-cpp/models && cd llama-cpp
    curl -L --fail -o models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
    https://huggingface.co/bartowski/Qwen2.5-1.5B-Instruct-GGUF/resolve/main/Qwen2.5-1.5B-Instruct-Q4_K_M.gguf

    The bartowski repository is public: no Hugging Face account is required. Check that the file weighs around 940 MB with ls -lh models/.

  2. Describe the stack in a compose.yml file.

    This file mounts the models/ folder read-only in the container and passes llama-server's options after the command keyword.

    services:
    llama-server:
    image: ghcr.io/ggml-org/llama.cpp:server@sha256:5b22b94a6a736a40bfbff13e8de2caf3f5dca5d24feba30fff38fa5d509a6c41
    container_name: llama-server
    restart: unless-stopped
    ports:
    - "8080:8080"
    volumes:
    - ./models:/models:ro
    command:
    - -m
    - /models/qwen2.5-1.5b-instruct-q4_k_m.gguf
    - --host
    - "0.0.0.0"
    - --port
    - "8080"
    - --ctx-size
    - "4096"
    - --threads
    - "8"
    - --n-gpu-layers
    - "0"
    - --alias
    - qwen2.5-1.5b
    healthcheck:
    test: ["CMD-SHELL", "curl -fsS http://localhost:8080/health || exit 1"]
    interval: 10s
    timeout: 5s
    retries: 6
    start_period: 30s

    The options deserve a word. --ctx-size 4096 sets the context window to 4096 tokens. --threads 8 matches the computation to the number of physical cores of the machine, with no need to count Hyper-Threading's logical threads. --n-gpu-layers 0 forces pure CPU; we will change it to enable a GPU. --alias gives the model a short name reused in API calls.

  3. Start the server and wait for the healthy state.

    Terminal window
    docker compose up -d
    docker compose ps

    The healthcheck queries the /health endpoint every 10 seconds. The container goes from health: starting to healthy once the model is loaded in memory, which takes a few seconds for a model of this size.

  4. Check that the model is served.

    Terminal window
    curl -fsS http://localhost:8080/health

    The expected answer is exactly {"status":"ok"}. If you get it, the inference server is operational.

Querying the OpenAI-compatible API

llama-server exposes a REST API aligned with the OpenAI format. In practice, any client or library that knows how to talk to the OpenAI API works unchanged: you only need to point the base URL to your server. This is what makes llama.cpp interchangeable with other backends.

The /v1/models endpoint lists the loaded models. It is the fastest test after the healthcheck.

Terminal window
curl -fsS http://localhost:8080/v1/models

The central endpoint is /v1/chat/completions. It takes a list of messages with their roles (system, user, assistant) and returns the generated answer.

Terminal window
curl -fsS http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2.5-1.5b",
"messages": [
{"role": "system", "content": "Tu es un assistant DevOps qui répond en français en une phrase."},
{"role": "user", "content": "Quelle commande systemd affiche les services actifs ?"}
],
"temperature": 0.2,
"max_tokens": 80
}'

The answer is a JSON object whose choices[0].message.content field contains the generated text, here systemctl list-units --active --type=service. The usage block details the tokens consumed and the timings block gives the request's actual throughput (predicted_per_second), information that the public OpenAI API does not provide.

The temperature parameter controls randomness: a low value (0.1 to 0.3) makes answers deterministic and factual, useful for extraction or code. max_tokens caps the length of the answer and protects against runaway generations.

Forcing a structured JSON answer

By default, an LLM returns free text, hard for a program to use. llama-server can constrain the output to follow a JSON schema: this is structured output. The server converts the schema into a grammar and prevents the model from producing a single token that would violate the structure. The answer is therefore always valid JSON.

The schema is declared through the response_format field. The example below extracts entities from a sentence, an ideal use case for a small model.

Terminal window
curl -fsS http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2.5-1.5b",
"messages": [
{"role": "user", "content": "Extrais la distribution Linux et son gestionnaire de paquets : Sur Debian 12, on installe les paquets avec apt."}
],
"temperature": 0.1,
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "extraction",
"strict": true,
"schema": {
"type": "object",
"properties": {
"distribution": { "type": "string" },
"version": { "type": "string" },
"gestionnaire_paquets": { "type": "string" }
},
"required": ["distribution", "version", "gestionnaire_paquets"]
}
}
},
"max_tokens": 120
}'

The content field of the answer then contains exactly {"distribution": "Debian", "version": "12", "gestionnaire_paquets": "apt"}. You can pass it straight to json.loads() with no cleanup. The strict: true option forbids any extra field not declared in the schema.

Measuring performance with llama-bench

Saying "it is fast" is not enough for an architecture decision. llama-bench measures throughput in a reproducible way, repeating each test and giving a standard deviation. The binary lives in the :full image, distinct from the :server image.

The tool distinguishes two phases. Prompt processing (denoted pp) is the speed at which the model ingests the input context. Text generation (denoted tg) is the speed at which it produces the answer tokens, the figure the user feels.

Terminal window
docker run --rm \
-v "$PWD/models:/models:ro" \
--entrypoint /app/llama-bench \
ghcr.io/ggml-org/llama.cpp:full@sha256:838e2f49cbd96e70c8996cae3b46ca2e551abd26e41b655c9e1463682a2486ee \
-m /models/qwen2.5-1.5b-instruct-q4_k_m.gguf \
-p 128,512 -n 64,128 -t 8 -ngl 0 -r 3

The -p and -n options list the prompt and generation sizes to test, -t sets the threads, -ngl the GPU layers (0 here) and -r 3 repeats each measurement three times. On an i7-12650H in pure CPU, 8 threads, the result obtained is as follows.

TestMeaningMeasured throughput
pp128Ingesting a 128-token prompt160.97 ± 0.84 t/s
pp512Ingesting a 512-token prompt164.32 ± 1.43 t/s
tg64Generating 64 tokens28.57 ± 0.08 t/s
tg128Generating 128 tokens28.42 ± 0.18 t/s

The reading is instructive. Prompt processing is five to six times faster than generation: ingesting a context parallelizes well, producing tokens one by one much less so. A generation around 28 tokens/second remains comfortable for an assistant read by a human, but shows the limit of the CPU as soon as you aim for several simultaneous requests. That is precisely the tipping point toward a GPU or toward a throughput-oriented backend such as vLLM.

Using a GPU: CUDA, Vulkan, Metal

The CPU is enough for a small model and single-user use. Beyond that, a GPU changes the scale. llama.cpp offloads all or part of the model's layers to the GPU through the --n-gpu-layers option: the value -1 offloads all layers, an intermediate number splits the load between GPU and CPU when the VRAM is insufficient.

The choice of image depends on your hardware. Each target has its dedicated build because the compute libraries differ.

On an NVIDIA GPU, use the :server-cuda image and expose the card to the container. The NVIDIA Container Toolkit must be installed on the host.

image: ghcr.io/ggml-org/llama.cpp:server-cuda@sha256:72ca3a0323ec891af9332afc8d501396557a45eca2b1b3712828d5cf57e36743
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]

Then pass --n-gpu-layers -1 in the command section to offload the whole model to the GPU.

Security

llama-server ships no authentication by default. The --host 0.0.0.0 option used above makes the server listen on all interfaces: convenient for Docker, but to be never exposed as is on the Internet or on an uncontrolled network.

Three measures are enough for a healthy deployment. First, restrict the listening: when running locally outside a container, prefer --host 127.0.0.1 to accept only connections from the machine. Next, require a key: the --api-key YOUR_KEY option imposes an Authorization: Bearer YOUR_KEY header on every request. This key is a secret: pass it through an environment variable, never in clear text in a versioned compose.yml. Finally, for real exposure, place the server behind a reverse proxy that terminates TLS and applies rate limiting.

Troubleshooting

The errors met in practice come down to a few recurring causes. The table below links the observed symptom to its cause and to the fix.

SymptomLikely causeSolution
port is already allocated at startupPort 8080 is taken by another serviceChange the host mapping: "8888:8080" in compose.yml
Container in a Restarting loopWrong model path in commandCheck that the .gguf is in models/ and that the -m path is exact
failed to load model in the logsTruncated or corrupted GGUF fileDownload again; compare the expected size with ls -lh
Container killed during loadingInsufficient RAM (OOM)Choose a smaller quantization (Q4 instead of Q8) or a lighter model
Very slow generationToo many or too few threadsMatch --threads to the number of physical cores
Truncated JSON answer (finish_reason: length)Model looping, max_tokens reachedSimplify the schema, lower the temperature, or take a larger model

To inspect what the server is doing, docker compose logs -f llama-server continuously displays the model loading, the detected chat template and each request processed.

Key takeaways

  • llama.cpp is the inference engine that tools such as Ollama embed; using it directly gives fine-grained control.
  • llama-server exposes an OpenAI-compatible API: any OpenAI client works by changing the base URL.
  • The GGUF format embeds a model that is already quantized, which makes CPU inference realistic.
  • Structured output guarantees valid JSON; a small model can still botch the content, however.
  • llama-bench measures a reproducible throughput: tell pp (ingestion) and tg (generation) apart.
  • Without an API key or a reverse proxy, llama-server must never be exposed outside the machine.

Knowledge check

Check that you have the essentials of this guide. The questions cover only what was just explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions
6 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer