Tokens per second? That only really matters for things like chat and sessions where you are monitoring a session as it runs which is going to be an increasingly small percentage of usage. That's bottlenecked by human attention. Agents in the background, long running tasks, unattended tasks, all that stuff is what is even capable of driving significant usage because you can just increase consumption massively by parallelizing without being the bottleneck. I wouldn't be shocked if already the vast majority of token usage was from unattended sessions. TPS is not that important there
It is impossible to make sophisticated software products at a high level with a low TPS. Try making a modern emulator or an award winning game. Both require a lot of iteration and not everything works well with multi agent.
DeepSeek is serving more TPS than anyone, about as much as OpenRouter does in a week in a day, at 38$/H800/Ascend so about 900k/~90% profit over 3 years (or ~80s% profit including training expenses).
Meanwhile, Oracle is handing Tencent 100.000 GPUs at 1.60$ hour, dangerously close to breakeven...
Tokens per second? That only really matters for things like chat and sessions where you are monitoring a session as it runs which is going to be an increasingly small percentage of usage. That's bottlenecked by human attention. Agents in the background, long running tasks, unattended tasks, all that stuff is what is even capable of driving significant usage because you can just increase consumption massively by parallelizing without being the bottleneck. I wouldn't be shocked if already the vast majority of token usage was from unattended sessions. TPS is not that important there
It is impossible to make sophisticated software products at a high level with a low TPS. Try making a modern emulator or an award winning game. Both require a lot of iteration and not everything works well with multi agent.
DeepSeek is serving more TPS than anyone, about as much as OpenRouter does in a week in a day, at 38$/H800/Ascend so about 900k/~90% profit over 3 years (or ~80s% profit including training expenses).
Meanwhile, Oracle is handing Tencent 100.000 GPUs at 1.60$ hour, dangerously close to breakeven...
What does this tell you?