How to Use llama-quantize for GGUF Models

Sun, 12 Apr 2026 09:42:36 +0800

llama-quantize is the quantization tool in llama.cpp. It is used to convert high-precision GGUF models into smaller quantized versions.

Its most common use is turning formats such as F32, BF16, or FP16 into versions like Q4_K_M, Q5_K_M, or Q8_0 that are easier to run locally. After quantization, models usually become much smaller and often faster at inference, but some quality loss is expected.

Basic workflow

A typical workflow is to prepare the original model, convert it to GGUF, and then run quantization.

# install Python dependencies
python3 -m pip install -r requirements.txt

# convert the model to ggml FP16 format
python3 convert_hf_to_gguf.py ./models/mymodel/

# quantize the model to 4-bits (using Q4_K_M method)
./llama-quantize ./models/mymodel/ggml-model-f16.gguf ./models/mymodel/ggml-model-Q4_K_M.gguf Q4_K_M

After that, you can run the quantized model with llama-cli:

1
2

# start inference on a gguf model
./llama-cli -m ./models/mymodel/ggml-model-Q4_K_M.gguf -cnv -p "You are a helpful assistant"

Common options

--allow-requantize: allows requantizing an already quantized model, usually not ideal for quality
--leave-output-tensor: keeps the output layer unquantized, increasing size but sometimes helping quality
--pure: disables mixed quantization and uses a more uniform quant type
--imatrix: uses an importance matrix to improve quantization quality
--keep-split: keeps the original shard layout instead of producing one merged file

If you just want a practical starting point, this is often enough:

`1`	`./llama-quantize ./models/mymodel/ggml-model-f16.gguf ./models/mymodel/ggml-model-Q4_K_M.gguf Q4_K_M`

How to choose a quant

You can think of quant levels as a tradeoff between size, speed, and quality:

Q8_0: larger, but usually safer for quality
Q6_K / Q5_K_M: common balanced choices
Q4_K_M: a very common default with a good size-quality balance
Q3 / Q2: useful when hardware is very limited, but quality loss is more visible

The practical goal is usually not to pick the biggest quant you can fit, but the one that runs reliably on your hardware while keeping acceptable quality.

Practical takeaway

start with Q4_K_M or Q5_K_M
move up to Q6_K or Q8_0 if quality matters more
move down to Q3 or Q2 if memory is tight
compare versions with the same prompt set

In short, llama-quantize is useful because it makes GGUF models easier to run on local hardware, not just because it makes files smaller.

Model Conversion on KnightLi Blog

How to Use llama-quantize for GGUF Models

Basic workflow

Common options

How to choose a quant

Practical takeaway