Q4_K_M vs Q6_K vs FP8: the quality you lose and what fits your GPU

Open a model repo today and you'll see a menu: BF16, FP8, NVFP4, then a separate GGUF repo with Q4_K_M, Q5_K_M, Q6_K, Q8_0 and IQ-something. The question everyone asks is the same: how much quality do I lose, and which one fits my GPU?

This guide covers the formats and the published numbers. For the memory math itself, see how much VRAM you need.

What quantization does

A model is billions of numbers (weights). Quantization stores them in fewer bits.

llama.cpp's quantize docs put it plainly: it "shrinks the model's size and can speed up inference", but "may introduce some accuracy loss". That loss is usually measured with perplexity (lower is better) or KL divergence.

Most formats group weights into blocks that share a scale factor. That's why "4-bit" files are usually a bit more than 4 bits per weight.

Q4_K_M vs Q6_K: the published numbers

When llama.cpp added its "k-quants" in PR #1684 (merged June 5, 2023), it published perplexity on wikitext for the original LLaMA 7B and 13B:

Percentages are our arithmetic from the PR's perplexity figures.

The PR's author summed it up: the 6-bit result "is within 0.1% or better from the original fp16 model". Quality falls off slowly down to 4-bit, then quickly below it.

Two caveats:

• These are 2023 models and a 2023 test. Newer models and newer quant methods (importance matrices, "IQ" types) can behave differently.
• Perplexity is an average. Code, math and long tool-calling chains can be more sensitive than prose. Test the quant on your own tasks.

File sizes on a current model

llama.cpp's current README lists these for Llama 3.1 8B (checked Oct 22, 2026):

Which file fits your GPU

The weights are only part of what you need. Context (the KV cache) and runtime overhead come on top, and our VRAM guide shows how to budget for them.

A simple way to pick:

1. Find the largest quant whose file fits in your VRAM (or a Mac's unified memory) with room left for context.
2. Prefer Q6_K or Q5_K_M if they fit. Per the numbers above, they're very close to FP16.
3. Drop to Q4_K_M if they don't. It's the usual sweet spot between size and quality.
4. Go below 4-bit only to fit a bigger model. In the 2023 PR's table, LLaMA 13B at Q3_K_M (5.88G, perplexity 5.4498) beat 7B at Q6_K (5.15G, 5.9110). Test it on your tasks.
5. Still doesn't fit? Partial offloading to system RAM works, but it's slower. See running a model bigger than your RAM.

If you're on a Mac, unified memory changes the math. To get a GGUF running, start with Ollama for beginners.

The formats, one by one

FP16 and BF16 (16-bit)

Most models are released in one of these. Both use 16 bits per weight.

• FP16 is IEEE half precision: 1 sign, 5 exponent and 10 mantissa bits.
• BF16 keeps FP32's 8 exponent bits and stores 7 mantissa bits. Google's docs say the dynamic range of bfloat16 and float32 are equivalent while using half the memory.

These are "full quality" for inference. Treat them as the baseline, not something you run on a laptop.

FP8 (8-bit float)

FP8 has two encodings, defined in the FP8 formats paper: E4M3 (4 exponent, 3 mantissa bits, max value 448) and E5M2 (5 exponent, 2 mantissa bits). The paper recommends E4M3 for weights and activations.

It halves memory versus 16-bit. NVIDIA says FP8 is supported on its Hopper and Blackwell GPUs. DeepSeek's V4.1 Flash, for example, is tagged FP8 on Hugging Face. See our DeepSeek V4.1 Flash piece.

MXFP8 and MXFP4 (microscaling)

The OCP Microscaling (MX) spec (v1.0, Sept 2023) groups 32 values into a block. All 32 share one 8-bit scale (E8M0, a power of two).

• MXFP8: FP8 elements (E4M3 or E5M2) plus the shared scale.
• MXFP4: 4-bit (E2M1) elements plus the shared scale. GGUF supports MXFP4 too, per Hugging Face's GGUF docs.

MiniMax M3.1 Flash is a real-world mix. Its config file lists mxfp8 with blocks of 32, and NVFP4 for the mixture-of-experts layers (checked Oct 22, 2026). More in our MiniMax M3.1 Flash piece.

NVFP4 (NVIDIA 4-bit float)

NVIDIA's NVFP4 (introduced June 24, 2025, for Blackwell GPUs):

• uses 4-bit E2M1 values, with an FP8 (E4M3) scale per 16-value block and an FP32 scale per tensor
• works out to about 4.5 bits per value
• is about 3.5× smaller than FP16, per NVIDIA

NVIDIA reported 1% or less accuracy loss versus FP8 on DeepSeek-R1-0528 in that June 2025 post. For Nemotron 3.5 Super, NVIDIA says its NVFP4 checkpoint keeps 99.62% median accuracy versus BF16. See Nemotron 3.5 explained.

INT4 (AWQ, GPTQ)

INT4 stores weights as 4-bit integers, usually in groups of 128 with a scale per group, while activations stay at 16-bit ("W4A16").

The AWQ paper shows how small the gap can be on big models. On Llama-2-70B, WikiText-2 perplexity went from 3.32 (FP16) to 3.41 with AWQ INT4. You'll see AWQ and GPTQ repos used with vLLM and similar servers.

GGUF (the local format)

GGUF isn't a number format. It's a file format. Per Hugging Face's docs, it packs the tensors plus metadata, and it's used by llama.cpp, LM Studio, GPT4All and Ollama. Inside, each tensor can use a different quant type:

• K-quants (Q2_K to Q6_K): blocks with their own scales. Q4_K is 4.5 bits per weight and Q6_K is 6.5625, per the docs.
• The _S / _M / _L suffix is the mix. In the original PR, Q4_K_M used Q6_K for half of two sensitive tensor types, and Q4_K for the rest.
• IQ types (IQ4_XS, IQ3_S, IQ2_XXS…) use an importance matrix to decide what to protect. They're useful below 4 bits.
• Q8_0, Q4_0 and friends are the older "legacy" types, per the docs. Q8_0 is still the common 8-bit option, at 8.50 bits per weight in the table above.

Cheat sheet

Bottom line: 6-bit is close to free, and 4-bit is a small, measurable hit. Below 4-bit, only go there to fit a bigger model, and test it on your real prompts first.