Serving barely useful GLM 5.2 costs what? $15k? Actually useful is like $50k? You’ll never recoup the cost unless you ‘locally’ means ‘inference provider is not the model provider’?
Serving barely useful GLM 5.2 costs what? $15k? Actually useful is like $50k? You’ll never recoup the cost unless you ‘locally’ means ‘inference provider is not the model provider’?
The high costs are necessary for high speed.
When a low speed of the order of one token per second is accepted, any open weights LLM can be run on an ordinary PC (with the weights read from SSDs) and the cost becomes negligible.
Such a low speed would be annoying for a chat, but I do not believe that it is "barely useful" for a coding assistant. There are plenty of tasks for which it is fine to get results some hours later or even overnight, and batching multiple tasks can complete them in about the same time as a single task.
I don't know. Even the frontier models do dumb things sometimes. Being able to iterate (and iterate quickly) is really important. If you get 1 try a day, you're probably back to it being better to just code by hand. Also, you're going to get absolutely outpaced by anyone who uses AI that goes faster.
So maybe for a hobby project this is fine, but for something you have to take to market and compete with... I think it'd be a really rough sell.
EDIT: also, just to be clear: if there was a practical path to using local AI, I'd take it in a heartbeat. I hope it gets to the point that it's better to use local than paying someone $200/mo. But right now, that $200/mo is the clear best option. I get making compromises for ideology but the compromises are too big for me right now.
Not "local" in the literal sense, but I set it up to serve at half quant for $23/hr and full quant for $35/hr.
You don't need to have it always on? This is a far cry from "$200/month," but I do not think it's $50k for "useful." Do you see it differently?
This is probably the dumbest possible way to do it. Just buy tokens through open router and you could run it all month 24/7 at 100tps for practically nothing. There are tons of ways to pay for things without giving your personal information.
A decent deal but Flash is quite dumb and you still have to pay for output tokens
Yes they mean open weight models offered by various providers
$15k or $50k is pretty cheap all things considered (a year ago it would have been more expensive, one person can spend that in a month or two)
I bought my spark and the models have already improved in that time (qwen3.6, speculative decoding 2x tgen, diffusion gemma 4x tgen) and I expect this to improve. Look out another 2-3 years, local is going to be very competitive.
You can recoup the costs quicker if you resell access to your local LLM on a reselling service.
Cheaper to just buy T-bills when I saw the numbers last time
glm-5.2 is available for $20/month on ollama.com and is IMHO more functional than the $200/month claude max subscription. you can even use the same claude harness [0]. You get about 20x more token usage at 10x less the price.
[0] https://ollama.com/library/glm-5.2