LLM Quantization Explained: How to Choose FP16, Q8, Q5, Q4, or Q2

The core goal of quantization is simple: trade a small amount of precision for a smaller model size, lower VRAM usage, and faster inference.
For local deployment, picking the right quantization format is often more important than chasing a larger parameter count.

What Is Quantization

Quantization means compressing model parameters from higher-precision formats (such as FP16) into lower-bit formats (such as Q8 and Q4).

A simple analogy:

Original model: like a high-quality photo, clear but large.
Quantized model: like a compressed photo, slightly less detail but lighter and faster.

Common Quantization Formats

Quantization	Precision / Bit Width	Size	Quality Loss	Recommended Use
FP16	16-bit float	Largest	Almost none	Research, evaluation, max quality
Q8_0	8-bit integer	Larger	Almost none	High-end PCs, quality + performance
Q5_K_M	5-bit mixed	Medium	Slight	Daily driver, balanced choice
Q4_K_M	4-bit mixed	Smaller	Acceptable	General default, strong value
Q3_K_M	3-bit mixed	Very small	Noticeable	Low-spec devices, run-first
Q2_K	2-bit mixed	Smallest	Significant	Extreme resource limits, fallback

Quantization Naming Rules

Take gemma-4:4b-q4_k_m as an example:

gemma-4:4b: model name and parameter scale.
q4: 4-bit quantization.
k: K-quants (an improved quantization method).
m: medium level (common options also include s/small and l/large).

Quick Selection by VRAM

RAM / VRAM	Recommended Quantization
4 GB	Q3_K_M / Q2_K
8 GB	Q4_K_M
16 GB	Q5_K_M / Q8_0
32 GB+	FP16 / Q8_0

Start with a version that runs stably on your machine, then move up in precision step by step instead of jumping straight to the biggest model.

Practical Tips

Start with Q4_K_M by default and test real tasks first.
If response quality is not enough, move up to Q5_K_M or Q8_0.
If VRAM or speed is the main bottleneck, move down to Q3_K_M.
Use the same test set every time you switch quantization formats.

Quantize a GGUF with llama-quantize

Start from an FP16, BF16, or F32 GGUF whenever possible. Do not quantize an already heavily quantized file again; --allow-requantize can cause additional quality loss.

1
2
3
4


./build/bin/llama-quantize \
  model-bf16.gguf \
  model-Q4_K_M.gguf \
  Q4_K_M

On Windows, the executable is commonly located here:

1
2
3
4


.\build\bin\Release\llama-quantize.exe `
  .\model-bf16.gguf `
  .\model-Q4_K_M.gguf `
  Q4_K_M

Record the output size and hash:

1
2


ls -lh model-bf16.gguf model-Q4_K_M.gguf
sha256sum model-Q4_K_M.gguf

Why Bit Width Alone Is Not a VRAM Estimate

Besides weights, inference needs room for the KV cache, compute buffers, multimodal projectors, the desktop, and backend overhead. A model whose weights are close to 8 GB is therefore not guaranteed to fit on an 8 GB GPU. Leave roughly 10% to 20% headroom and confirm GPU offload in the backend logs.

Run a Comparable Acceptance Test

Test both variants with the same model family, backend, context, and prompt:

1
2


llama-cli -m model-Q4_K_M.gguf -c 4096 -n 128 \
  -p "Summarize the trade-offs of model quantization."

Item	Record
Model	Full filename and hash
Backend	CUDA, ROCm, Metal, Vulkan, or CPU
Context	`-c` and actual prompt tokens
Memory	Peak RAM/VRAM and partial offload
Speed	Prompt processing and generation token/s
Quality	Errors, formatting problems, and omissions on fixed questions

Code, math, long-context tasks, and structured output are more sensitive to quantization. Do not call a format “nearly lossless” after one chat prompt.

Conclusion

Quality first: FP16 or Q8_0.
Balance first: Q5_K_M.
General default: Q4_K_M.
Low-spec fallback: Q3_K_M or Q2_K.

The key is not “bigger is always better”, but “the most stable and usable result under your hardware limits.”

Context Length Changes the Choice

Quantization is only one part of memory use. A model that fits at a short context can still run out of VRAM when its KV cache grows, so test the context length and batch size you actually need before deciding that a higher-precision file is practical.