I was immediately hoping for some local-AI setups that do real work, on limited resources. Unfortunately it looks like this is still waiting on more people to share.
I was immediately hoping for some local-AI setups that do real work, on limited resources. Unfortunately it looks like this is still waiting on more people to share.
Not a SWE, but I do Freetoken (backend) + unsloth (front) to run qwen 3.6 35B A3B NVFP4 (~20GB) in 8GB VRAM + ~20GB RAM. Getting around 50 t/s on a 3070. I don't think I can even share through this site.
If not coding what do you use it for? Also what kind of performance do you tend to get out of it? I've been thinking about a similar setup but am skeptical it's worth the effort.
It's still typically for code, like python data stuff, bash scripts, general web search, small javascript stuff for my website. By saying not SWE, I mean I don't really see much benefit from "agentic" stuff, although I've tried. 50 t/s means 40~60 seconds for a typical thinking response. I used to run gemma 4 e4b-it-qat fully in GPU (~150 t/s), but the quality improvement moving to a much larger MoE model was 100% worth the switch. Especially because I had a lot of idle ram (from the before times :( ). There is some tradeoff in terms of context length, but I just use it as a chat. Also it's fairly trivial to setup as long as the GPU is supported.