KL divergence is nothing close to a replacement for benchmarks. As flawed as benchmarks are, KL divergence is a barely useful signal. The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
You're very right about KL divergence. I spent a couple days playing with the Gemma 4 models. That's 10 separate models (varying weights, MoE, QAT or not, etc...) with identical tokenizers. I treated 31B at BF16 as the gold standard, feeding Wikipedia snippets, and anthropomorphizing a bit:
Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"
Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)
Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)
Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)
For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.
> The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.
The talk around KL divergence is oversold. People talk about it as if it’s not a benchmark, but at its core it is in the same neighborhood! You get a different KL divergence number on different “calibration datasets”, so in other words it’s data-dependent. It is NOT a universal guarantee about the fundamental divergence of a model.
When I tested various eval benchmarks on Qwen3.5/3.6 27B with Unsloth's quants, the scores usually dropped 0-5% between UD-Q6 and UD-Q3 depending on the eval.
Unsloth publishes KL divergence numbers which measures how much the quantised probability distribution changes vs unquantised: https://unsloth.ai/docs/models/qwen3.8#quantization-analysis
It's a bit bare at the moment, I assume they are going to add further detail later (eg comparison to other quants), similar to their other releases.
KL divergence is nothing close to a replacement for benchmarks. As flawed as benchmarks are, KL divergence is a barely useful signal. The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
Actually we do publish non KLD benchmarks - top-1% is better - for NVFP4 for eg we did MMLU Pro, GPQA, AIME 2025: https://unsloth.ai/docs/models/qwen3.6#nvfp4-benchmarks
Sometimes they're just slow and expensive, so we we KLD as a proxy measure and it's very high correlation (95%+)
You're very right about KL divergence. I spent a couple days playing with the Gemma 4 models. That's 10 separate models (varying weights, MoE, QAT or not, etc...) with identical tokenizers. I treated 31B at BF16 as the gold standard, feeding Wikipedia snippets, and anthropomorphizing a bit:
Gemma 4 31B: "Um, if I really said all of that, I guess I'd say this next"
Gemma 4 26B: "Dude, I would've said completely different stuff" (large divergence)
Gemma 4 12B: "Umm, there's zero chance I would've said some of this" (INFINITE divergence)
Gemma 4 E4B and E2B: "Derp derp, I'm happy to say almost anything" (lowest divergence)
For models which are chat trained, they simply would not recite Wikipedia, so the divergence is almost meaningless. I thought about capturing a realistic coding session and trying to use that as the corpus, but you need to preserve the turn-based tokens and such, so I moved on to other things.
> The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.
The talk around KL divergence is oversold. People talk about it as if it’s not a benchmark, but at its core it is in the same neighborhood! You get a different KL divergence number on different “calibration datasets”, so in other words it’s data-dependent. It is NOT a universal guarantee about the fundamental divergence of a model.
That's not a replacement for benchmarks
When I tested various eval benchmarks on Qwen3.5/3.6 27B with Unsloth's quants, the scores usually dropped 0-5% between UD-Q6 and UD-Q3 depending on the eval.