The future is already here :)
These 3bit ISTA quants of qwen are supposed to be 95% as accurate as the full model and runs moothly on laptops:
https://huggingface.co/ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF
There are multiple independent tests of this on YouTube.
They aren't "Just as accurate". What you see with these quants is the providers only ever show performance on already saturated benchmarks. You'll see labs put out 1-bit quants Deepseek v4.1 claiming only 1% degradation because they're only showing the benchmarks which tell that story. Then you look at the performance on those same benchmarks for significantly weaker models like Qwen3.8-27B even and it's virtually the same because that benchmark is saturated.
I like these quants as much as the next guy when they're all I can run but I have both R9700 with 32GB VRAM and a 5060 Ti with only 16GB VRAM and so I run both Q6 Unlsoth quants and ISTA IQ3_XXS quants and yeah there is an absolute world of difference in realworld performance.
IMHO I think it's pretty scummy behavior from the quant providers to frame these quants this way. Just be honest about where the degradation is really occurring on the non-saturated benchmarks.
This is a really good point that goes the other way as well. Some use cases are great for smaller and cheaper models, but it's not obviously when that's the case so I can bank the savings.
All I know is that Strata and these models are the most I have ever gotten out of my machine. I get 70 t/s on a standard gaming rig 9070 XT / 5900X / 64 GB DDR4 @3400.
I don't know, it hallucinates quite badly - confuses js with python.
Apparently the ud-q4_k_xl version of qwen 3.8 27b is quite better than the q3 and not much larger. At least per r/localllama. (I'm also trying to find a good local model).