Skip to content
Français
Account: not signed in, sync available My learning
Select theme
Français
Lire en français

Quantizing an LLM in practice: GGUF, AWQ, GPTQ, FP8

12 min read
Improve this page

You know why to quantize a model. This guide shows how to do it, and above all what it really changes, backed by measurements. You will quantize a model end to end in GGUF for local inference, serve AWQ, GPTQ and FP8 variants on a GPU, and read the real trade-off between size, speed and quality. All the figures in this guide were taken in the lab, on CPU for the GGUF part, on an H100 GPU for the server part. Target audience: a developer or platform engineer comfortable with Docker and the command line.

What you will learn

  • Convert a Hugging Face model to GGUF and then quantize it yourself.
  • Measure the size / throughput / perplexity trade-off of a quantization.
  • Serve a model already quantized in AWQ, GPTQ or FP8 with vLLM.
  • Read the real VRAM and throughput gain of each format on a GPU.
  • Choose the right format according to your hardware and your priority.

Prerequisites

This guide assumes you already have the vocabulary of quantization: precision, bits per parameter, formats. If these notions are fuzzy, first read Understanding LLM quantization (in French). You also need:

  • Docker and the Compose plugin.
  • Python 3.12 and git for the GGUF conversion part.
  • An NVIDIA GPU for the server part (otherwise, this part remains instructive as reading).

The GGUF part can be reproduced entirely on a CPU machine. The GPU part was carried out on an H100, but the principle holds for any recent GPU. The server measurements were taken in May 2026 on an H100, vLLM version not recorded; the image in this guide is pinned to v0.33.0, so rerun the bench on your own version and GPU.

The GGUF pipeline: convert and quantize yourself

The GGUF format is that of the llama.cpp and Ollama ecosystem. Quantizing it yourself takes two steps: convert the Hugging Face model to an unquantized GGUF (16 bits), then quantize that GGUF to a compact format. The demonstration model is Qwen2.5-1.5B-Instruct, small enough for the whole pipeline to fit in a few minutes.

  1. Fetch the Hugging Face model.

    Download the model files (.safetensors weights, tokenizer, configs) into a models/Qwen2.5-1.5B-Instruct/ folder. The Qwen/Qwen2.5-1.5B-Instruct repository is public, no account is required. Check that model.safetensors weighs about 3.1 GB, a truncated file will make the conversion fail.

  2. Convert to 16-bit GGUF.

    The conversion uses llama.cpp's convert_hf_to_gguf.py script. This script needs Python dependencies (torch, transformers) and its internal module, so clone the repository and work in a virtual environment.

    Terminal window
    git clone --depth 1 https://github.com/ggml-org/llama.cpp
    cd llama.cpp
    python3 -m venv .venv
    ./.venv/bin/pip install --only-binary=:all: \
    -r requirements/requirements-convert_hf_to_gguf.txt
    ./.venv/bin/python convert_hf_to_gguf.py ../models/Qwen2.5-1.5B-Instruct \
    --outtype f16 --outfile ../models/qwen2.5-1.5b-f16.gguf

    The resulting qwen2.5-1.5b-f16.gguf file weighs about 2.9 GiB. It is the unquantized base, which will also serve as the quality reference.

  3. Quantize to Q4_K_M, Q5_K_M and Q8_0.

    The llama-quantize tool turns the 16-bit GGUF into compact variants. It is included in the official Docker image llama.cpp:full (digest pinned in the command below).

    Terminal window
    for Q in Q4_K_M Q5_K_M Q8_0; do
    docker run --rm -v "$PWD/models:/models" \
    --entrypoint /app/llama-quantize \
    ghcr.io/ggml-org/llama.cpp:full@sha256:838e2f49cbd96e70c8996cae3b46ca2e551abd26e41b655c9e1463682a2486ee \
    /models/qwen2.5-1.5b-f16.gguf /models/qwen2.5-1.5b-$Q.gguf $Q
    done

    The output is three files: the Q8_0 (8 bits), the Q5_K_M and the Q4_K_M. Compared with the size of the F16 in the table below, their sizes give about 5.8 bits per weight for Q5_K_M and 5.1 bits for Q4_K_M on this model.

Measuring the size / speed / quality trade-off

Quantizing without measuring makes no sense. Three llama.cpp tools are enough: llama-bench for throughput, llama-perplexity for quality, and a plain ls -lh for the size.

Perplexity deserves a word. It is a measure of a language model's quality: it assesses how "surprised" the model is by a reference text. The lower it is, the better. Quantizing slightly degrades the model, and therefore raises the perplexity a little; the stake is to know by how much.

Here are the measurements taken on Qwen2.5-1.5B, CPU, with a sample of the wikitext corpus.

FormatSizeGenerationPerplexityQuality gap
F162.88 GiB10.5 tok/s9.44reference
Q8_01.53 GiB16.9 tok/s9.46+0.2%
Q5_K_M1.04 GiB24.2 tok/s9.62+1.9%
Q4_K_M935 MiB30.8 tok/s10.01+6.0%

Three lessons stand out. The size drops sharply: Q4_K_M divides the model by 3.2. Generation speeds up as you quantize, from 10.5 to 30.8 tokens/second, that is three times faster, because fewer bits means less data to move in memory. And the quality degrades little: Q8_0 is imperceptible, Q5_K_M very discreet, Q4_K_M visible but moderate.

Serving a quantized model on a GPU

On a GPU, you rarely quantize yourself: you serve an already quantized model, published on Hugging Face. Three formats dominate, AWQ and GPTQ (4 bits), FP8 (8 bits), and vLLM (in French) loads them all. The demonstration model switches to Qwen2.5-14B-Instruct, a size where quantization is really worthwhile.

The AWQ and GPTQ variants are separate Hugging Face repositories. vLLM automatically detects the quantization from the model's configuration.

Terminal window
docker run -d --name vllm-quant --runtime nvidia --gpus all --ipc host \
-p 8000:8000 -v hf-cache:/root/.cache/huggingface \
vllm/vllm-openai:v0.33.0@sha256:d825baf9ce4b1b6bfc5ecc7b79880c7e6748c65a72d697ce3d47607aafa84c6e \
--model Qwen/Qwen2.5-14B-Instruct-AWQ \
--max-model-len 8192 --gpu-memory-utilization 0.90

Replace the -AWQ suffix with -GPTQ-Int4 to serve the GPTQ variant.

To find the memory actually used by the weights, read the logs at startup: vLLM prints a Model loading took X GiB line. It is this figure, not the total VRAM, which includes the cache, that measures the gain from quantization.

The gain measured on a GPU

The four variants were served on the same H100, with the same model Qwen2.5-14B and the same bench (200 requests, ceiling throughput). These figures were measured in May 2026 on an H100, vLLM version not recorded: the guide's image is pinned to v0.33.0, and your numbers will depend on your version and GPU.

VariantWeights VRAMTotal throughputLatency (TPOT)
bf16 (reference)27.6 GiB8,354 tok/s74.6 ms
AWQ 4 bits9.5 GiB9,474 tok/s70.1 ms
GPTQ-Int4 4 bits9.4 GiB9,604 tok/s69.4 ms
FP8 8 bits16.1 GiB12,417 tok/s55.8 ms

The result runs against a received idea: all the quantized variants are faster than bf16, not just lighter. The 4-bit formats divide the weights VRAM by 2.9 and gain ~13 to 15% of throughput. FP8 goes much further: +49% throughput compared with bf16.

Which format to choose

No format is universally superior. The right choice depends on your main constraint.

Your situationRecommended format
Local CPU inference, single-user useGGUF Q4_K_M, the most compact of the three measured, fast generation
Local inference, quality comes firstGGUF Q5_K_M or Q8_0
GPU, fitting a large model on little VRAMAWQ or GPTQ, compression by ~3
Recent GPU (FP8 Tensor Cores), throughput firstFP8, best throughput in our measurement
Reference quality, enough VRAMbf16, no quantization

The practical rule: start with a 4-bit quantization (Q4_K_M locally, AWQ/GPTQ on GPU). It is a good starting point. You go back to a higher precision only if you measure a troublesome degradation on your use case.

Troubleshooting

Quantization incidents come down to a few known causes.

SymptomLikely causeSolution
model.safetensors rejected at conversionFile truncated during downloadDownload again; check the size with ls -lh
ModuleNotFoundError when launching convert_hf_to_gguf.pyScript launched outside the cloned repositoryWork from the cloned llama.cpp repository, not an isolated script
CUDA out of memory when serving the quantized model--max-model-len too highReduce the context length, or lower --gpu-memory-utilization
FP8 does not speed things upGPU without FP8 Tensor CoresUse AWQ/GPTQ on a GPU older than the Hopper/Ada architecture
Visibly degraded qualityQuantization too aggressive, especially on a small modelMove up to Q5/Q8, or try a larger model and repeat the measurement

Key takeaways

  • The GGUF pipeline takes two steps: convert to 16 bits, then quantize to Q4/Q5/Q8.
  • Quantizing reduces the size and speeds up generation; it is not only a trade-off, it is often a double gain.
  • The quality loss measured stays modest down to 4 bits on a small model (+6% perplexity in Q4_K_M); on another model, repeat the measurement before choosing.
  • On GPU, AWQ and GPTQ compress the most (÷2.9 VRAM); FP8 favors throughput.
  • FP8 on H100 gains +49% throughput thanks to the native FP8 Tensor Cores, a result tied to the hardware.
  • Choosing a format means choosing your priority constraint: VRAM, throughput or quality.
  • pip install runs code: auditing your sources is part of the pipeline.

Knowledge check

Check that you have the essentials of this guide. The questions cover only what was just explained here.

Knowledge check

Check what you have learned with this interactive quiz

6 questions
6 min
70% to pass

How it works

  • The timer starts when you click Start
  • Multiple choice, true/false and short answer questions
  • You can move between questions
  • Detailed results are shown at the end

Starts the quiz and the timer