87 GGUF quants from 7 uploaders, measured by KL divergence against the BF16 reference using ~250,000 tokens of coding, chat, tool calling, science, non-Latin scripts, and long documents. Full methodology.

• unsloth/Qwen3.6-27B-GGUF (21 quants)

• lmstudio-community/Qwen3.6-27B-GGUF (3 quants)

• Jackrong/Qwen3.6-27B-GGUF (11 quants)

• bartowski/Qwen_Qwen3.6-27B-GGUF (26 quants)

• mradermacher/Qwen3.6-27B-i1-GGUF (23 quants)

• ubergarm/Qwen3.6-27B-GGUF (2 quants, ik_llama.cpp)

• ggml-org/Qwen3.6-27B-GGUF (1 quant)

Results

Pareto frontier

The best quant at each size. If it’s not in this table, a smaller file with lower KL exists.

Findings

• This is the most competitive uploader field so far. bartowski and mradermacher tie at 12 frontier positions each, unsloth has 10. No single uploader dominates.

• ubergarm’s ik_llama.cpp quants earn both their frontier spots: IQ5_KS (19.9 GB, KL 0.128) and IQ4_KS (15.8 GB, KL 0.209). These special, ik_llama.cpp-only data types show their worth in this dense model.

• Jackrong has 0 frontier appearances out of 11 quants. All are dominated by bartowski or unsloth at similar sizes. lmstudio-community also has 0.

• Q8_0 has KL 0.075, similar to the Qwen 3.6 35B A3B (0.069) and better than the Qwen 3.5 27B (0.120).

• Even at IQ2_XXS (8.4 GB), the model retains 77.5% top-1.

Per-category performance

Long documents dominate the quality loss: UD-Q8_K_XL scores KL 0.001 on coding but 0.373 on long documents. Tool calling is the second worst category throughout.

Methodology

• Inference: TextGen + patched llama.cpp (logprob extraction from prompt)

• Reference: BF16 GGUF by unsloth

• Dataset: ~250,000 tokens across 6 categories (coding, general chat, tool calling, science, non-Latin scripts, long documents)

• Input format: full OpenAI-compatible messages rendered through the model’s Jinja2 chat template

• Metric: KL divergence, computed token-by-token between reference and quantized top-40 log-probability distributions

Full methodology

Please do not share the plots and tables in this post publicly. Instead, share the post URL. These measurements are expensive to produce and more subscribers means more models benchmarked.

KV cache quantization stores the key-value cache in lower precision to save memory. q8_0 halves the cache memory, q4_0 quarters it. The common wisdom is that q8_0 is “practically lossless.”

Each model was tested using the BF16 GGUF from Unsloth, loaded three times with f16, q8_0, and q4_0 cache on the same machine. The only variable changing between runs is cache precision. These measurements include the recently added TurboQuant-inspired attention rotation that llama.cpp applies automatically. Full methodology.

Results

Findings

Gemma is sensitive to KV cache quantization, Qwen is not

“q8_0 is practically lossless” is wrong for Gemma.

Gemma 31B at q8_0 cache has KL 0.108. Gemma 26B A4B at q8_0 is 0.377. Qwen handles it much better: both Qwen models stay below KL 0.04 at q8_0, and even q4_0 cache (KL 0.087-0.117) is usable.

MoE amplifies the problem for Gemma

The Gemma 4 26B A4B is the most quantization-sensitive model tested so far, both for weights and for KV cache. Its q8_0 cache KL (0.377) is 3.5x worse than the dense Gemma 31B (0.108), and q4_0 reaches KL 1.088 with 68.0% top-1. Qwen’s MoE shows no such amplification (0.024 dense, 0.039 MoE).

How cache quant compares to weight quant

Cache quantization and weight quantization are independent sources of quality loss. If you run a Q4_K_M with q8_0 cache, both penalties stack. The table below maps each cache result to the weight quant that causes equivalent damage.

Full weight quant benchmarks: Gemma 4 31B, Gemma 4 26B A4B, Qwen 3.6 27B, Qwen 3.6 35B A3B.

Per-category: Gemma degrades everywhere, Qwen only on long docs

Gemma degrades uniformly: even its best category at q8_0 (science, KL 0.214) is worse than Qwen’s worst (long docs, KL 0.142). Qwen concentrates nearly all damage in long documents (KL 0.581 at q4_0) and tool calling (0.086), with other categories staying near zero.

Per-category performance

Gemma 4 31B (Dense)

Gemma 4 26B A4B (MoE)

Qwen 3.6 27B (Dense)

Qwen 3.6 35B A3B (MoE)

Methodology

• Inference: TextGen + patched llama.cpp (logprob extraction from prompt)

• Reference: BF16 GGUF loaded with f16 KV cache (the llama.cpp default)

• Test: Same BF16 GGUF loaded with q8_0 or q4_0 KV cache

• Dataset: ~250,000 tokens across 6 categories (coding, general chat, tool calling, science, non-Latin scripts, long documents)

• Metric: KL divergence, computed token-by-token between f16-cache and quantized-cache top-40 log-probability distributions

Full methodology

64 GGUF quants from 6 uploaders, measured by KL divergence against the BF16 reference using ~250,000 tokens of coding, chat, tool calling, science, non-Latin scripts, and long documents. Full methodology.

• unsloth/Qwen3.6-35B-A3B-GGUF (22 quants)

• bartowski/Qwen_Qwen3.6-35B-A3B-GGUF (27 quants)

• lmstudio-community/Qwen3.6-35B-A3B-GGUF (3 quants)

• ggml-org/Qwen3.6-35B-A3B-GGUF (1 quant)

• mudler/Qwen3.6-35B-A3B-APEX-GGUF (7 quants)

• AesSedai/Qwen3.6-35B-A3B-GGUF (4 quants)

Results

Pareto frontier

The best quant at each size. If it’s not in this table, a smaller file with lower KL exists.

Findings

• Qwen 3.6 35B A3B quantizes significantly better than its predecessor Qwen 3.5 35B A3B. Q8_0 has KL 0.069, compared to 0.121 for Qwen 3.5 35B A3B.

• UD-Q8_K_XL (38.4 GB, KL 0.073) is not on the frontier. ggml-org’s Q8_0 (36.9 GB, KL 0.069) is smaller and has lower KL.

• unsloth dominates: 14 of 26 frontier positions. AesSedai earns 4 by filling size gaps between unsloth’s UD quants.

• lmstudio-community and mudler (APEX) never appear on the frontier.

• Tool calling is the worst category at Q8_0 (KL 0.177), worse than long documents (0.121). This is unusual compared to other Qwen models.

• Even at UD-IQ1_M (10.0 GB), the model retains 86.9% top-1.

Per-category performance

Tool calling dominates the quality loss at Q8_0 (KL 0.177), followed by long documents (0.121). Coding and science stay at or below 0.010. This pattern is different from Qwen 3.5 models where long documents were always the worst category.

Methodology

• Inference: TextGen + patched llama.cpp (logprob extraction from prompt)

• Reference: BF16 GGUF by unsloth

• Dataset: ~250,000 tokens across 6 categories (coding, general chat, tool calling, science, non-Latin scripts, long documents)

• Input format: full OpenAI-compatible messages rendered through the model’s Jinja2 chat template

• Metric: KL divergence, computed token-by-token between reference and quantized top-40 log-probability distributions

Full methodology

Please do not share the plots and tables in this post publicly. Instead, share the post URL. These measurements are expensive to produce and more subscribers means more models benchmarked.

72 GGUF quants from 4 uploaders, measured by KL divergence against the BF16 reference using ~250,000 tokens of coding, chat, tool calling, science, non-Latin scripts, and long documents. Full methodology.

• unsloth/Qwen3.5-9B-GGUF (21 quants)

• bartowski/Qwen_Qwen3.5-9B-GGUF (24 quants)

• lmstudio-community/Qwen3.5-9B-GGUF (3 quants)

• mradermacher/Qwen3.5-9B-i1-GGUF (24 quants)

Results

Pareto frontier

The best quant at each size. If it’s not in this table, a smaller file with lower KL exists.

Findings

• Qwen 3.5 9B quantizes better than its larger siblings. Q8_0 has KL 0.066, compared to 0.120 for Qwen 3.5 27B and 0.121 for Qwen 3.5 35B A3B.

• No single uploader dominates. bartowski wins the mid-range (5-8 GB), mradermacher owns below 4 GB, and unsloth’s UD quants fill the gaps.

• lmstudio-community never appears on the frontier. Avoid them.

• Even at 3.4 GB (IQ2_M), the model retains 79.3% top-1. Below that, quality drops fast: IQ1_S at 2.4 GB is barely functional at 53.7% top-1.

Per-category performance

Long documents dominate the quality loss: UD-Q8_K_XL scores KL 0.001 on coding but 0.288 on long documents. Q8_0 is even worse at 0.367 for long documents. All other categories stay at or below 0.028 at Q8_0.

Methodology

• Inference: TextGen + patched llama.cpp (logprob extraction from prompt)

• Reference: BF16 GGUF by unsloth

• Dataset: ~250,000 tokens across 6 categories (coding, general chat, tool calling, science, non-Latin scripts, long documents)

• Input format: full OpenAI-compatible messages rendered through the model’s Jinja2 chat template

• Metric: KL divergence, computed token-by-token between reference and quantized top-40 log-probability distributions

Full methodology

Please do not share the plots and tables in this post publicly. Instead, share the post URL. These measurements are expensive to produce and more subscribers means more models benchmarked.

90 GGUF quants from 8 uploaders, measured by KL divergence against the BF16 reference using ~250,000 tokens of coding, chat, tool calling, science, non-Latin scripts, and long documents. Full methodology.

• unsloth/Qwen3.5-35B-A3B-GGUF (23 quants)

• bartowski/Qwen_Qwen3.5-35B-A3B-GGUF (27 quants)

• lmstudio-community/Qwen3.5-35B-A3B-GGUF (3 quants)

• ggml-org/Qwen3.5-35B-A3B-GGUF (1 quant)

• mradermacher/Qwen3.5-35B-A3B-i1-GGUF (23 quants)

• mudler/Qwen3.5-35B-A3B-APEX-GGUF (7 quants)

• AesSedai/Qwen3.5-35B-A3B-GGUF (4 quants)

• ubergarm/Qwen3.5-35B-A3B-GGUF (2 quants, ik_llama.cpp)

Results

Pareto frontier

The best quant at each size. If it’s not in this table, a smaller file with lower KL exists.

Findings

• This is a Mixture of Experts model with 3B active parameters out of 35B total. Unlike the Gemma 4 26B A4B, it quantizes well. Q8_0 has KL 0.121 and 95.4% top-1, close to the dense Qwen 3.5 27B (0.120, 97.0%). The Gemma 4 26B A4B has 0.544 and 77.9% at Q8_0.

• Q8_0 is not on the Pareto frontier. UD-Q6_K_XL (32.1 GB, KL 0.112) is smaller and better.

• unsloth dominates: 18 of 23 frontier points. The pattern is similar to the Gemma 4 26B A4B, but the quality loss is much smaller.

• 8 uploaders tested. AesSedai, mudler (APEX), ubergarm, ggml-org, and lmstudio-community never appear on the frontier.

• Even at IQ2_XXS (9.9 GB), top-1 is still 83.6%.

Per-category performance

Long documents degrade the fastest (KL 0.336 at UD-Q8_K_XL). Tool calling is the second worst (KL 0.152). Coding and science stay below 0.01 until Q4 levels.

Methodology

• Inference: TextGen + patched llama.cpp (logprob extraction from prompt)

• Reference: BF16 GGUF by unsloth

• Dataset: ~250,000 tokens across 6 categories (coding, general chat, tool calling, science, non-Latin scripts, long documents)

• Input format: full OpenAI-compatible messages rendered through the model’s Jinja2 chat template

• Metric: KL divergence, computed token-by-token between reference and quantized top-40 log-probability distributions

Full methodology

75 GGUF quants from 5 uploaders, measured by KL divergence against the BF16 reference using ~250,000 tokens of coding, chat, tool calling, science, non-Latin scripts, and long documents. Full methodology.

• unsloth/Qwen3.5-27B-GGUF (21 quants)

• bartowski/Qwen_Qwen3.5-27B-GGUF (26 quants)

• lmstudio-community/Qwen3.5-27B-GGUF (3 quants)

• mradermacher/Qwen3.5-27B-i1-GGUF (23 quants)

• ubergarm/Qwen3.5-27B-GGUF (2 quants, ik_llama.cpp)

Results

Pareto frontier

The best quant at each size. If it’s not in this table, a smaller file with lower KL exists.

Findings

• Qwen 3.5 27B quantizes exceptionally well. Q8_0 has KL 0.120 and 97.0% top-1. Even IQ2_M at 9.5 GB retains 84.5% top-1. For comparison, the dense Gemma 4 31B at Q8_0 has KL 0.163 and 92.0% top-1.

• The Pareto frontier is split across unsloth, bartowski, mradermacher, and ubergarm. No single uploader dominates. This is the most competitive uploader field so far.

• Differences between uploaders at the same quant type are small. At Q5_K_M: unsloth 0.215, bartowski 0.219, mradermacher 0.227. All within 6%.

• lmstudio-community never appears on the frontier. Their Q4_K_M has KL 0.329 vs bartowski’s 0.260 at the same quant.

• ubergarm’s IQ5_KS (ik_llama.cpp quant) earns a spot on the frontier at 19.9 GB with KL 0.172.

Per-category performance

Long documents account for nearly all the KL divergence. At Q8_0, coding and science have KL of 0.001 and 0.003. Long documents: 0.713. The gap grows at lower quants but the pattern is the same: this model holds up across all categories except long documents.

Methodology

• Inference: TextGen + patched llama.cpp (logprob extraction from prompt)

• Reference: BF16 GGUF by unsloth

• Dataset: ~250,000 tokens across 6 categories (coding, general chat, tool calling, science, non-Latin scripts, long documents)

• Input format: full OpenAI-compatible messages rendered through the model’s Jinja2 chat template

• Metric: KL divergence, computed token-by-token between reference and quantized top-40 log-probability distributions

Full methodology

Please do not share the plots and tables in this post publicly. Instead, share the post URL. These measurements are expensive to produce and more subscribers means more models benchmarked.

62 GGUF quants from 5 uploaders, measured by KL divergence against the BF16 reference using ~250,000 tokens of coding, chat, tool calling, science, non-Latin scripts, and long documents. Full methodology.

• unsloth/gemma-4-E4B-it-GGUF (20 quants)

• bartowski/google_gemma-4-E4B-it-GGUF (22 quants)

• lmstudio-community/gemma-4-E4B-it-GGUF (3 quants)

• ggml-org/gemma-4-E4B-it-GGUF (2 quants)

• mradermacher/gemma-4-E4B-it-i1-GGUF (15 quants)

Results

Pareto frontier

The best quant at each size. If it’s not in this table, a smaller file with lower KL exists.

Findings

• Gemma 4 E4B is a small Mixture of Experts model. It quantizes extremely well. UD-Q8_K_XL has KL 0.003 and 98.7% top-1, unsloth Q8_0 has KL 0.003 and 98.6%. This is the best quantization tolerance of any model tested so far.

• Q8_0 differs across uploaders. unsloth’s Q8_0 (8.2 GB, KL 0.003) is larger but measurably better than bartowski/ggml-org (8.0 GB, KL 0.006).

• unsloth dominates the frontier: 13 of 17 points.

• lmstudio-community never appears on the frontier. lmstudio-community and ggml-org Q4_K_M score KL 0.207/0.208 vs bartowski’s 0.072. Avoid them.

• The entire model fits in 3.5-8.6 GB. Even UD-IQ2_M at 3.5 GB retains 78.8% top-1.

Per-category performance

At Q8_0, all categories are essentially 0 (KL 0.001-0.006). The per-category spread only becomes visible below Q5. Long documents degrade fastest.

Methodology

• Inference: TextGen + patched llama.cpp (logprob extraction from prompt)

• Reference: BF16 GGUF by unsloth

• Dataset: ~250,000 tokens across 6 categories (coding, general chat, tool calling, science, non-Latin scripts, long documents)

• Input format: full OpenAI-compatible messages rendered through the model’s Jinja2 chat template

• Metric: KL divergence, computed token-by-token between reference and quantized top-40 log-probability distributions

Full methodology

Please do not share the plots and tables in this post publicly. Instead, share the post URL. These measurements are expensive to produce and more subscribers means more models benchmarked.

80 GGUF quants from 6 uploaders, measured by KL divergence against the BF16 reference using ~250,000 tokens of coding, chat, tool calling, science, non-Latin scripts, and long documents. Full methodology.

• unsloth/gemma-4-26B-A4B-it-GGUF (21 quants)

• bartowski/google_gemma-4-26B-A4B-it-GGUF (26 quants)

• lmstudio-community/gemma-4-26B-A4B-it-GGUF (3 quants)

• ggml-org/gemma-4-26B-A4B-it-GGUF (2 quants)

• mradermacher/gemma-4-26B-A4B-it-i1-GGUF (21 quants)

• mudler/gemma-4-26B-A4B-it-APEX-GGUF (7 quants)

Results

Pareto frontier

The best quant at each size. If it’s not in this table, a smaller file with lower KL exists.

Findings

• This is a Mixture of Experts model with 4B active parameters out of 26B total. It quantizes much worse than the dense 31B. Q8_0 already shows KL 0.544 and 77.9% top-1, compared to 0.163 and 92.0% for the dense Gemma 4 31B. That’s 3.3x worse KL at the same quant level.

• The Pareto frontier is dominated by unsloth UD quants. Only bartowski Q6_K_L and two IQ2_XXS entries (bartowski, mradermacher) at the very bottom break the streak. ggml-org, lmstudio-community, and mudler never appear on the frontier (except Q8_0 where everyone is the same).

• ggml-org and lmstudio-community Q4_K_M score KL 2.126/2.116 vs bartowski’s 1.093 at the same quant type. Avoid them.

• mudler’s APEX quants never appear on the frontier.

• unsloth’s MXFP4_MOE (16.6 GB, KL 0.963) is beaten by their own UD-Q4_K_XL (17.1 GB, KL 0.747). The MoE-specific quant format doesn’t help.

Per-category performance

Long documents hit KL 1.055 at Q8_0. For comparison, the dense 31B has 0.466 at Q8_0. All categories degrade roughly in proportion, with science and tool calling the most resilient throughout.

Methodology

• Inference: TextGen + patched llama.cpp (logprob extraction from prompt)

• Reference: BF16 GGUF by unsloth

• Dataset: ~250,000 tokens across 6 categories (coding, general chat, tool calling, science, non-Latin scripts, long documents)

• Input format: full OpenAI-compatible messages rendered through the model’s Jinja2 chat template

• Metric: KL divergence, computed token-by-token between reference and quantized top-40 log-probability distributions

Full methodology

Please do not share the plots and tables in this post publicly. Instead, share the post URL. These measurements are expensive to produce and more subscribers means more models benchmarked.

KL divergence measures how different the quantized model’s token probability distribution is from the original model’s. A KL of 0 means the quant is identical to the original. Higher values mean more information is lost.

Unlike perplexity, KL divergence directly compares two models against each other. It answers the question: “how much does this quant change the model’s output?”

Why not perplexity?

Perplexity measures how well a model predicts text. It doesn’t directly answer “how much did quantization change this model?” Two models can have similar perplexity but very different token distributions. KL divergence captures exactly what was lost.

Why are these numbers higher than usual?

Most KL divergence benchmarks use Wikipedia with ~2048 token context. It’s a lot easier to score well there. This benchmark uses real-world inputs up to ~30k tokens across 6 task categories, which is much harder on quantized models. If you’ve seen KL values of 0.01-0.03 for Q4_K_M elsewhere, that’s why.

KL divergence is dataset-dependent, same as perplexity. The difference is that KL divergence compares two models against each other, while perplexity only measures one model against the evaluation text.

Inference

All measurements are collected using TextGen’s OpenAI-compatible API.

llama.cpp doesn’t natively support extracting logprobs from the prompt, so I patched it. The patch is included in the llama.cpp binary shipped with TextGen. Source: commit on my llama.cpp fork.

Reference model

The BF16 GGUF from unsloth is used as the reference for each model. BF16 is lossless relative to the original weights.

Dataset

~250,000 tokens across 6 categories:

• Coding: programming tasks and code generation

• General chat: typical assistant conversations

• Tool calling: samples with tool definitions and tool_calls, rendered through the model’s chat template

• Science: technical and scientific questions

• Non-Latin scripts: Chinese, Japanese, Korean, Arabic, Cyrillic, and others

• Long documents: extended context inputs up to ~30k tokens

Every sample is a full OpenAI-compatible messages array, not raw text. The samples go through the model’s Jinja2 chat template, so the benchmark captures how quantization affects real usage including special tokens, system prompts, and tool formatting.

Measurement process

KL divergence is computed on prompt tokens, not on generated text. The model processes the prompt and returns log-probabilities for each token position, which are then compared against the reference.

1. The Jinja2 chat template is taken from the official model release (not from the GGUF metadata). The same template is used for every quant of that model.

2. Each sample’s messages are converted to a prompt string using the /v1/internal/chat-prompt endpoint.

3. Each prompt is sent to /v1/completions with echo: true and logprobs: 40, which returns the top-40 token log-probabilities for every token in the prompt.

4. For each token position, the KL is computed over the union of tokens that appear in either model’s top-40 list. For tokens missing from one list, a floor logprob is used (lowest observed logprob in that distribution minus 2).

5. The per-token KL values are averaged across all tokens and all prompts to produce the final number.

Why top-40?

Top-40 covers virtually all the probability mass in practice. Tokens outside the top 40 contribute negligible probability mass, so the approximation error is negligible.

Top-1 agreement

In addition to KL divergence, each benchmark reports top-1 agreement: the percentage of tokens where the quantized model’s most likely token matches the reference model’s most likely token. A top-1 agreement of 85% means that for 85 out of every 100 tokens, the quant would pick the same token as the original under greedy decoding.

53 GGUF quants from 4 uploaders, measured by KL divergence against the BF16 reference using ~250,000 tokens of coding, chat, tool calling, science, non-Latin scripts, and long documents.

• unsloth/gemma-4-31B-it-GGUF (21 quants)

• bartowski/google_gemma-4-31B-it-GGUF (27 quants)

• lmstudio-community/gemma-4-31B-it-GGUF (3 quants)

• ggml-org/gemma-4-31B-it-GGUF (2 quants)

Results

Pareto frontier

The best quant at each size. If it’s not in this table, a smaller file with lower KL exists.

Findings

• Q8_0 is identical across all four uploaders.

• ggml-org and lmstudio-community quants never appear on the Pareto frontier (except Q8_0 where everyone is the same). Avoid them.

• 8 out of 9 unsloth UD quants are on the Pareto frontier. The exception is UD-IQ3_XXS, dominated by another UD quant (UD-Q2_K_XL) at the same size with lower KL.

• The Pareto frontier is split between unsloth and bartowski. Neither dominates the other across all sizes.

Per-category performance

Long documents and non-Latin scripts degrade the fastest at every quant level. Even at Q8_0, long documents show a KL of 0.466 and non-Latin scripts 0.222. Science and tool calling are the most resilient (0.069 and 0.078 at Q8_0). This pattern holds across all quant sizes.

Methodology

• Inference: TextGen + patched llama.cpp (logprob extraction from prompt)

• Reference: BF16 GGUF by unsloth

• Dataset: ~250,000 tokens across 6 categories (coding, general chat, tool calling, science, non-Latin scripts, long documents)

• Input format: full OpenAI-compatible messages rendered through the model’s Jinja2 chat template

• Metric: KL divergence, computed token-by-token between reference and quantized top-40 log-probability distributions

Full methodology