Looks impressive, and this size fits achievable home hardware.
That said, if someone would kindly quantise this down for the 64GB paupers, that would be appreciated. (I know there’s likely degradation, but some people reported good results with a 2 bit version of Qwen 3.5 122B, and this is starting from a higher point. Would be interesting to try, at least.)
Edit: someone in the process of doing so: https://huggingface.co/vcruz305/Laguna-S-2.1-GGUF
They have also published smaller 33B model called Laguna XS 2.1, its Q4 gguf is 20GB.
https://huggingface.co/poolside/Laguna-XS-2.1-GGUF/tree/main
Thanks for flagging. From the few benchmarks I can find, it looks there or thereabouts with Qwen 3.6-35B-A3B, or maybe a touch below. I'm interested to compare a model that is a big jump larger with pretty impressive benchmarks, but more heavily quantized to fit.
> if someone would kindly quantise this down for the 64GB paupers
The Q4_K_M is 75GB. I'm exactly at 64GB, and I wouldn't quantize it further. Instead, do partial weight residency and stream the rest from SSD.
The tool I've been using, llm-compressor, can quant models that do not fit in memory (use the sequential pipeline)
https://github.com/vllm-project/llm-compressor
my setup to help you on your way: https://github.com/verdverm/quantr
Though it seems these will not be needed as Poolside has published quants & dflash with their models.