I'll take the opposite here. If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy.
I think it's worth acknowledging that the power of LLMs at this point is not really so much in the smarts, but in the coordination and the surrounding harness tech. "Written english" turning into sequences of commands[0]. The whole agentic "stuff" in general. Tools + coordination is the superpower. The reasoning... it doesn't have to be _that_ good for the rest of the stuff to work. On good codebases and infra, at least.
And I say this as someone who really would rather most of this stuff disappear!
[0]: programming is obviously text to commands, but there's a loooooooot of futziness that LLM reasoning has let us remove in some flows
> If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy.
You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.
DGX Spark is a biiiiit costly but neat to hear!
Do you know what kinda throughput you’re getting on that kinda setup?
(I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)
It is costly, especially right now. I don’t think you can make a case for it on cost savings!
The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth.
I think these numbers may improve because the model is pretty new and optimizations aren’t done.