In the open weights scene, it is always chasing closely behind. Many leading edge big open models are trained in-part with distillation of large commercial hosted models.
At this point though gaming GPUs are not exactly affordable. You need a 32GB GPU to be able to run something genuinely useful for coding, and that's thousands of dollars. Still, happy to see open models make lots of progress. I just hope hardware prices come back to earth.
Realistically I think 16GB is the lower limit and you can find those cheap enough still. Then you run llama.cpp to bond your cards and it's off to the races.
I use Qwen 3.8 27b, yeah. I don't really shop around for models a whole lot besides generational jumps, and it doesn't feel like there will be anything too worth switching to until Qwen 4 (hoping they release a similarly sized model, I'm not actually certain they have said they will?)
Scrappy builds aren't "affordable" but are more so than running Nvidia + optimized builds. You could put together a machine for $5000 or less that competes with frontier models from less than a year ago, albeit a little slower.
And cards like the $1600 (sheesh) R9700 have actually gotten more capable through community efforts at optimization, since their launch.
I think if the bubble can ever deflate/pop, and GPUs and RAM back off a bit, we'll see the dam burst for local AI.
Right now it's only accessible to people who can justify the costs.
If you had been paying any attention at all over the last year, there were 32GB AMD cards at like $1200-1400 and Intel at a little over $1000. I actually hadn't yet noticed they've gone up in price and had last checked only a couple months ago. But even at like $2000, that's still comparable to the nominal price of a mid 90s consumer PC for actual magic.
Can confirm. Our household is involved in raising two natural intelligences. They are quite expensive to maintain. Chances they will offer positive economic return are unclear. The value proposition on comptuer-based intelligence, by-token or through hardware ownership, seems much clearer.
I have to admit I am a sort of enthusiast for the natural intelligences, though.
Likewise, I have four of those. Occasionally there are installation issues and some of the training feels like there is resistance but the final results are definitely worth it, far more satisfying than any piece of technology.
You have a 25K rig.[1] What can a gaming rig do by comparison? One agentic project at a time while yours does twenty? Okay, in that case regular people can do regular serial work (unlike the most agentic setups or whatever is going on).
That rig is running single GPU most of the time and like that it is good for 95% of what I'm doing. If I need to evaluate a large model or if the smaller ones get stuck on a problem (which does happen) then I run GLM 5.3 3.86 bpw on all of the GPUs. It's a nice space heater in that configuration, so I see it as co-generation ;)
In the open weights scene, it is always chasing closely behind. Many leading edge big open models are trained in-part with distillation of large commercial hosted models.
It already has. If you have a gaming GPU you can run AI locally.
At this point though gaming GPUs are not exactly affordable. You need a 32GB GPU to be able to run something genuinely useful for coding, and that's thousands of dollars. Still, happy to see open models make lots of progress. I just hope hardware prices come back to earth.
24 GB 7900 XTX user here. You can do a ton with less than 32 GB. It definitely drops off quick, but 32 GB is not the floor for useful AI.
Realistically I think 16GB is the lower limit and you can find those cheap enough still. Then you run llama.cpp to bond your cards and it's off to the races.
24 GB 3090 user here. What is the best model for coding you have used? Qwen3.8 27B?
I use Qwen 3.8 27b, yeah. I don't really shop around for models a whole lot besides generational jumps, and it doesn't feel like there will be anything too worth switching to until Qwen 4 (hoping they release a similarly sized model, I'm not actually certain they have said they will?)
If you haven't found it yet: ninfer-3090 is worth checking out if you use just one GPU.
Scrappy builds aren't "affordable" but are more so than running Nvidia + optimized builds. You could put together a machine for $5000 or less that competes with frontier models from less than a year ago, albeit a little slower.
And cards like the $1600 (sheesh) R9700 have actually gotten more capable through community efforts at optimization, since their launch.
I think if the bubble can ever deflate/pop, and GPUs and RAM back off a bit, we'll see the dam burst for local AI.
Right now it's only accessible to people who can justify the costs.
If you had been paying any attention at all over the last year, there were 32GB AMD cards at like $1200-1400 and Intel at a little over $1000. I actually hadn't yet noticed they've gone up in price and had last checked only a couple months ago. But even at like $2000, that's still comparable to the nominal price of a mid 90s consumer PC for actual magic.
> for actual magic
Precisely. It is cheap at almost any price.
Can confirm. Our household is involved in raising two natural intelligences. They are quite expensive to maintain. Chances they will offer positive economic return are unclear. The value proposition on comptuer-based intelligence, by-token or through hardware ownership, seems much clearer.
I have to admit I am a sort of enthusiast for the natural intelligences, though.
Likewise, I have four of those. Occasionally there are installation issues and some of the training feels like there is resistance but the final results are definitely worth it, far more satisfying than any piece of technology.
You have a 25K rig.[1] What can a gaming rig do by comparison? One agentic project at a time while yours does twenty? Okay, in that case regular people can do regular serial work (unlike the most agentic setups or whatever is going on).
[1] https://news.ycombinator.com/item?id=50238816
That rig is running single GPU most of the time and like that it is good for 95% of what I'm doing. If I need to evaluate a large model or if the smaller ones get stuck on a problem (which does happen) then I run GLM 5.3 3.86 bpw on all of the GPUs. It's a nice space heater in that configuration, so I see it as co-generation ;)
yeah i trained a 1000 param LLM in my browser literally yesterday. it's game over for Anthropic /s.