You mean serving inference? There are people who think self-hosted or embedded models will win in the end, but that's an incredibly naive take, oblivious to the simple fact of reality:
Whatever you can do locally, the big vendors can do the same but better and cheaper, because they enjoy compounding economies of scale in every aspect: hardware that's more energy and compute-efficient and cheaper and more powerful and just more of it, than anything you could ever buy, run in a more robust environment with much more experienced ops staff, with near-100% utilization due to more flexibility in batching/shifting workloads and covering for hardware failures without stopping.
And that's only when considering the vendors running exactly the same thing you are, which they always can - and they already have a strict advantage there. But on top of that, they can afford to innovate themselves, and stay ahead of you at every step.
There is no way in which cloud inference isn't a better deal than local inference, excepting applications that are constrained by literal speed of light.
The absolute value of those numbers matters a lot. The cloud providers could be 100 times cheaper than running locally, but if it still costs say, 10 cents a day to run locally, you’re not going to care about this difference very much. And what you keep in privacy out-weighs the trivial savings afforded by the cloud provider.
I never said local models will disappear. There will be equilibrium. But excluding special applications where communicating with external servers is not an option, cloud is always going to be able to provide better inference for lower costs. That's structural.
> The cloud providers could be 100 times cheaper than running locally, but if it still costs say, 10 cents a day to run locally, you’re not going to care about this difference very much
For ad-hoc use, maybe not - but anyone running a business that's some form of pushing input through LLM to get output, will see costs proportional to use and error rate inversely proportional to quality, and they'll not be looking at it as "$0.1 isn't much", but "cloud lets me reduce costs 100x", and translate that to some mix of more volume, higher quality, and broader reach.
> And what you keep in privacy out-weighs the trivial savings afforded by the cloud provider.
That's even more niche than running LLMs on Martian robots. Most real privacy concerns are solved with contracts and audits. Individual ad-hoc use may lean more heavily towards local processing, but that's still a rounding error in overall use.
There is a coherent argument that once LLMs reach the top of their S curve, the gap between small/medium local models and large cloud hosted ones converges.
Especially if GPU performance increases or market oversupply mean you can get good performance for a couple thousand dollars.
I’m not sure about the nature or timeframe for an S curve in LLMs but I don’t think it’s unreasonable to think about one, nor to entertain the hosting consequences of a progression on one.
I don't know the argument so I won't insist on the point, but I fail to see how it is relevant. Even if all proprietary LLMs disappeared today, efficiencies of scale alone mean the big cloud vendors can take the same open-weight LLMs you use locally, and sell inference with them for less money, and much more reliably, than you can afford yourself.