If you used the bare API pricing, 1M tokens @ 30% input/70% output/50% cached, you'd pay $0.05805. Even with four discounted sparks, how much are you paying for the same tokens/distribution?
If you used the bare API pricing, 1M tokens @ 30% input/70% output/50% cached, you'd pay $0.05805. Even with four discounted sparks, how much are you paying for the same tokens/distribution?
There's soooo much by way of experiments, explorations, tinkering, and even projects that you can't possibly pursue through a some SaaS API.
The more reasonable comparison is against rented GPU's, while looking at tradeoffs in latency and upload/download/storage/instance management overhead.
Buying hardware for local models is meeting a wholly different need than buying tokens through OpenRouter or whatever.
It cuts both ways. A GPU in your basement is a depreciating asset with fixed computing power and consumes electricity. Switching model providers is trivial.
> A GPU in your basement is a depreciating asset
All decades prior and up to about a year ago, I would have agreed with you. My Framework Desktop, however has appreciated in value by 75% since I bought it. Will it stay there for a long time? Probably not. But it shows that there are no hard and fast rules about things anymore.
I just bought a Framework Desktop. Would have been nice to get it at the introductory price, or perhaps the new 192gb model refresh they’re now teasing, but I settled and got a 64 gb model. At the time, the 128’s price had already risen again, but the 64’s price was still at a lower price.
64 can still easily do a Qwen 4.8 model, so I’m relatively happy with my purchase… plus, it’s price change has caused it to quickly appreciate in value… so I could sell it if my situation ever turned dire lol
At the current point in time I'd argue it's more about opportunity cost/value.
If I'm a professional photographer chasing the best possible end product, I'm not buying cameras because they're economical. I'm buying the best camera I can get my hands on to get the best product I can produce within reason under the understanding that it doesn't have to equate to the best economic decision to be the _right_ decision.
If you're in a position to be able to take advantage of the local inference - it's a no brainer. If you're not sure how that would be done, then it's not a good move.
For me it's entirely because I have a bunch of projects with my own personal data that would be tough to do with openrouter/claude or any other cloud.
For example, I have a small posix-shell-based LLM harness that can SSH into my NAS and run organization tasks using the local DS4Flash that I have right now. It's already been a massive help for me to keep me organized, and that's just 2x DGX Spark's worth of compute.
I'm not trying to say there is no use case. I just want to know the cost. Is it less than the API cost? Is it the same? Is it more? I'm looking for hard numbers. If the cost is the same or more, then the decision for local isn't to save money
If your usage wouldn't change with local inference and you don't have security/privacy concerns then at the currently heavily subsidized pricing, sure.. not economical.
But things change real fast when you're no longer bound by costs/apis/rate limits. All of a sudden it's not about "how can I do this right and efficiently" and more about "I can poke at and test _all the things_ that might make this better".
I think most people who can't see this value in the local inference approach are likely still copy/pasting from their web LLM ui's or don't even come close to subscription quotas. Meanwhile, 1b tokens a day is a light day for me with 3 $200/m subscriptions + some level of sub at basically every frontier level provider. Had I been less frugal and ponied up for the hardware before things got crazy I wouldn't need 80% of that - just the frontier models for the most complex tasks, the open weight models would handle the rest easily _and_ I'd get to do a lot more exploratory work without concern about quotas.
And here I am, feeling a bit guilty for using between 2 and 5M tokens... since 1 August!
Employer just sent an email that.. things are changing when it comes to token spend...
What did I do with these?
Setup record/replay for our product using qemu, several variatons thereof including experiments on target hardware. Fixed a tricky bug in qemu that I sadly can't upstream..
Experimented with rr on WSL2 and our target arch. Failed experiment.
Setup mutation testing PoC.
Optimized pipelines
etc. etc. Just contung code its soo much more than I would normally produce, but its also 95% experiments that are still not productized, and much of it never will be.
What do you do with all those tokens?