You know why to quantize a model. This guide shows how to do it, and above all what it really changes, backed by measurements. You will quantize a model end to end in GGUF for local inference, serve AWQ, GPTQ and FP8 variants on a GPU, and read the real trade-off between size, speed and quality. All the figures in this guide were taken in the lab, on CPU for the GGUF part, on an H100 GPU for the server part. Target audience: a developer or platform engineer comfortable with Docker and the command line.
What you will learn
- Convert a Hugging Face model to GGUF and then quantize it yourself.
- Measure the size / throughput / perplexity trade-off of a quantization.
- Serve a model already quantized in AWQ, GPTQ or FP8 with vLLM.
- Read the real VRAM and throughput gain of each format on a GPU.
- Choose the right format according to your hardware and your priority.
Prerequisites
This guide assumes you already have the vocabulary of quantization: precision, bits per parameter, formats. If these notions are fuzzy, first read Understanding LLM quantization (in French). You also need:
- Docker and the Compose plugin.
- Python 3.12 and
gitfor the GGUF conversion part. - An NVIDIA GPU for the server part (otherwise, this part remains instructive as reading).
The GGUF part can be reproduced entirely on a CPU machine. The GPU part was carried out on an H100, but the principle holds for any recent GPU. The server measurements were taken in May 2026 on an H100, vLLM version not recorded; the image in this guide is pinned to v0.33.0, so rerun the bench on your own version and GPU.
The GGUF pipeline: convert and quantize yourself
The GGUF format is that of the llama.cpp and Ollama ecosystem. Quantizing it yourself takes two steps: convert the Hugging Face model to an unquantized GGUF (16 bits), then quantize that GGUF to a compact format. The demonstration model is Qwen2.5-1.5B-Instruct, small enough for the whole pipeline to fit in a few minutes.
-
Fetch the Hugging Face model.
Download the model files (
.safetensorsweights, tokenizer, configs) into amodels/Qwen2.5-1.5B-Instruct/folder. TheQwen/Qwen2.5-1.5B-Instructrepository is public, no account is required. Check thatmodel.safetensorsweighs about 3.1 GB, a truncated file will make the conversion fail. -
Convert to 16-bit GGUF.
The conversion uses llama.cpp's
convert_hf_to_gguf.pyscript. This script needs Python dependencies (torch, transformers) and its internal module, so clone the repository and work in a virtual environment.Terminal window git clone --depth 1 https://github.com/ggml-org/llama.cppcd llama.cpppython3 -m venv .venv./.venv/bin/pip install --only-binary=:all: \-r requirements/requirements-convert_hf_to_gguf.txt./.venv/bin/python convert_hf_to_gguf.py ../models/Qwen2.5-1.5B-Instruct \--outtype f16 --outfile ../models/qwen2.5-1.5b-f16.ggufThe resulting
qwen2.5-1.5b-f16.gguffile weighs about 2.9 GiB. It is the unquantized base, which will also serve as the quality reference. -
Quantize to Q4_K_M, Q5_K_M and Q8_0.
The
llama-quantizetool turns the 16-bit GGUF into compact variants. It is included in the official Docker imagellama.cpp:full(digest pinned in the command below).Terminal window for Q in Q4_K_M Q5_K_M Q8_0; dodocker run --rm -v "$PWD/models:/models" \--entrypoint /app/llama-quantize \ghcr.io/ggml-org/llama.cpp:full@sha256:838e2f49cbd96e70c8996cae3b46ca2e551abd26e41b655c9e1463682a2486ee \/models/qwen2.5-1.5b-f16.gguf /models/qwen2.5-1.5b-$Q.gguf $QdoneThe output is three files: the Q8_0 (8 bits), the Q5_K_M and the Q4_K_M. Compared with the size of the F16 in the table below, their sizes give about 5.8 bits per weight for Q5_K_M and 5.1 bits for Q4_K_M on this model.
Measuring the size / speed / quality trade-off
Quantizing without measuring makes no sense. Three llama.cpp tools are enough: llama-bench for throughput, llama-perplexity for quality, and a plain ls -lh for the size.
Perplexity deserves a word. It is a measure of a language model's quality: it assesses how "surprised" the model is by a reference text. The lower it is, the better. Quantizing slightly degrades the model, and therefore raises the perplexity a little; the stake is to know by how much.
Here are the measurements taken on Qwen2.5-1.5B, CPU, with a sample of the wikitext corpus.
| Format | Size | Generation | Perplexity | Quality gap |
|---|---|---|---|---|
| F16 | 2.88 GiB | 10.5 tok/s | 9.44 | reference |
| Q8_0 | 1.53 GiB | 16.9 tok/s | 9.46 | +0.2% |
| Q5_K_M | 1.04 GiB | 24.2 tok/s | 9.62 | +1.9% |
| Q4_K_M | 935 MiB | 30.8 tok/s | 10.01 | +6.0% |
Three lessons stand out. The size drops sharply: Q4_K_M divides the model by 3.2. Generation speeds up as you quantize, from 10.5 to 30.8 tokens/second, that is three times faster, because fewer bits means less data to move in memory. And the quality degrades little: Q8_0 is imperceptible, Q5_K_M very discreet, Q4_K_M visible but moderate.
Serving a quantized model on a GPU
On a GPU, you rarely quantize yourself: you serve an already quantized model, published on Hugging Face. Three formats dominate, AWQ and GPTQ (4 bits), FP8 (8 bits), and vLLM (in French) loads them all. The demonstration model switches to Qwen2.5-14B-Instruct, a size where quantization is really worthwhile.
The AWQ and GPTQ variants are separate Hugging Face repositories. vLLM automatically detects the quantization from the model's configuration.
docker run -d --name vllm-quant --runtime nvidia --gpus all --ipc host \ -p 8000:8000 -v hf-cache:/root/.cache/huggingface \ vllm/vllm-openai:v0.33.0@sha256:d825baf9ce4b1b6bfc5ecc7b79880c7e6748c65a72d697ce3d47607aafa84c6e \ --model Qwen/Qwen2.5-14B-Instruct-AWQ \ --max-model-len 8192 --gpu-memory-utilization 0.90Replace the -AWQ suffix with -GPTQ-Int4 to serve the GPTQ variant.
FP8 needs no dedicated repository: vLLM can quantize the model on the fly at load time, from the 16-bit weights, through the --quantization fp8 option.
docker run -d --name vllm-quant --runtime nvidia --gpus all --ipc host \ -p 8000:8000 -v hf-cache:/root/.cache/huggingface \ vllm/vllm-openai:v0.33.0@sha256:d825baf9ce4b1b6bfc5ecc7b79880c7e6748c65a72d697ce3d47607aafa84c6e \ --model Qwen/Qwen2.5-14B-Instruct --quantization fp8 \ --max-model-len 8192 --gpu-memory-utilization 0.90To find the memory actually used by the weights, read the logs at startup: vLLM prints a Model loading took X GiB line. It is this figure, not the total VRAM, which includes the cache, that measures the gain from quantization.
The gain measured on a GPU
The four variants were served on the same H100, with the same model Qwen2.5-14B and the same bench (200 requests, ceiling throughput). These figures were measured in May 2026 on an H100, vLLM version not recorded: the guide's image is pinned to v0.33.0, and your numbers will depend on your version and GPU.
| Variant | Weights VRAM | Total throughput | Latency (TPOT) |
|---|---|---|---|
| bf16 (reference) | 27.6 GiB | 8,354 tok/s | 74.6 ms |
| AWQ 4 bits | 9.5 GiB | 9,474 tok/s | 70.1 ms |
| GPTQ-Int4 4 bits | 9.4 GiB | 9,604 tok/s | 69.4 ms |
| FP8 8 bits | 16.1 GiB | 12,417 tok/s | 55.8 ms |
The result runs against a received idea: all the quantized variants are faster than bf16, not just lighter. The 4-bit formats divide the weights VRAM by 2.9 and gain ~13 to 15% of throughput. FP8 goes much further: +49% throughput compared with bf16.
Which format to choose
No format is universally superior. The right choice depends on your main constraint.
| Your situation | Recommended format |
|---|---|
| Local CPU inference, single-user use | GGUF Q4_K_M, the most compact of the three measured, fast generation |
| Local inference, quality comes first | GGUF Q5_K_M or Q8_0 |
| GPU, fitting a large model on little VRAM | AWQ or GPTQ, compression by ~3 |
| Recent GPU (FP8 Tensor Cores), throughput first | FP8, best throughput in our measurement |
| Reference quality, enough VRAM | bf16, no quantization |
The practical rule: start with a 4-bit quantization (Q4_K_M locally, AWQ/GPTQ on GPU). It is a good starting point. You go back to a higher precision only if you measure a troublesome degradation on your use case.
Troubleshooting
Quantization incidents come down to a few known causes.
| Symptom | Likely cause | Solution |
|---|---|---|
model.safetensors rejected at conversion | File truncated during download | Download again; check the size with ls -lh |
ModuleNotFoundError when launching convert_hf_to_gguf.py | Script launched outside the cloned repository | Work from the cloned llama.cpp repository, not an isolated script |
CUDA out of memory when serving the quantized model | --max-model-len too high | Reduce the context length, or lower --gpu-memory-utilization |
| FP8 does not speed things up | GPU without FP8 Tensor Cores | Use AWQ/GPTQ on a GPU older than the Hopper/Ada architecture |
| Visibly degraded quality | Quantization too aggressive, especially on a small model | Move up to Q5/Q8, or try a larger model and repeat the measurement |
Key takeaways
- The GGUF pipeline takes two steps: convert to 16 bits, then quantize to Q4/Q5/Q8.
- Quantizing reduces the size and speeds up generation; it is not only a trade-off, it is often a double gain.
- The quality loss measured stays modest down to 4 bits on a small model (+6% perplexity in Q4_K_M); on another model, repeat the measurement before choosing.
- On GPU, AWQ and GPTQ compress the most (÷2.9 VRAM); FP8 favors throughput.
- FP8 on H100 gains +49% throughput thanks to the native FP8 Tensor Cores, a result tied to the hardware.
- Choosing a format means choosing your priority constraint: VRAM, throughput or quality.
pip installruns code: auditing your sources is part of the pipeline.
Knowledge check
Check that you have the essentials of this guide. The questions cover only what was just explained here.
Knowledge check
Check what you have learned with this interactive quiz
How it works
- The timer starts when you click Start
- Multiple choice, true/false and short answer questions
- You can move between questions
- Detailed results are shown at the end
Starts the quiz and the timer
Review
(0/0)Score by domain
A domain groups several guides and gets several questions, so its score is more reliable than a per-topic score, which often rests on a single question.
Summary
Strengths
No topic passed at 80% on at least two questions yet.
Weak areas
No topic below the pass mark.
Recommended next
Hints before you try again?
A new full quiz with random questions
Work again only on the questions you missed