Huh what? Qwen3.5-35B-A3B runs just fine with maximum context, on an RTX SUPER 12 GB, with offloading of some expert layers to DDR4-3200.
Same story on an RTX 4060 Ti 16 GB. MTP is a serious boost to tg.
Downloading the model is a simple hf command that HuggingFace's web UI even gives you.
llama.cpp is trivial to use, and so is llama-swap, if you want to use other models too.
If you don't know what arguments to run it with, you download ggrun and use that.
Local LLMs are incredibly capable and don't need expensive hardware. A $500 GPU will do. Or even cheaper.
This is all trivial.
Full model or a 4-bit quant? I have a 5090 and I'm not sure whether I should use a quant that fits within the VRAM or a much bigger version where I'd have to offload a lot to 64GB RAM and a beefy CPU (but still a CPU)
I personally run the Q6 quant on my RX 9070 XT (16GB VRAM). On r/LocalLlama there was a post recently as well, which talked about the degradation of different quants (for the 27B version)[0]
[0]: https://www.reddit.com/r/LocalLLaMA/comments/1vef79c/quantiz...
RTX Super 12GB costs $700. An openrouter account costs nothing.
A lot of people already have 12GB+ GPUs lying around for playing games, doing video editing, etc. I would not get a GPU or mac just to run LLMs personally, but if one wants to get such a device for other tasks too, it may make sense to eg choose a slightly higher (v)RAM variant if they want to run some bigger models. Then what you pay for the local llms is just the difference.