This is not about information but about capital. Even if we had free access to the weights of the best models in the world: who would be able to run them?
This is not about information but about capital. Even if we had free access to the weights of the best models in the world: who would be able to run them?
Technology is deflationary. I am holding in my hand a device that would have been a supercomputer 30 years ago. It costed me a couple of hundreds of dollars.
These models and the hardware they are running on will get even more efficient. We are nowhere near the physical limits of what we can achieve.
> Technology is deflationary.
Not anymore! Well, if you're like Elon and already taking down the bottle of Cuatro Comas from the high shelf, the economies of scale will continue to work in your favor.
But one of the really neat things about AI is that there is no limit in sight to the scaling incentive. More compute will always get you more: more training, more inference, more parameters, more capacity to build more and better models, more spare capacity to run the slop your models have already built to generate the slop that will succeed it. Back in the dot-com days, or even the "big data" days, you wanted to scale up rapidly but there was a limit: there were only so many customers and they could only produce so much data you could only ingest so fast. In the late 90s, one of the world's most trafficked sites, ftp.cdrom.com, ran on a (single!) dual-processor Pentium Pro system. That was just serving files, and there was certainly room for more CPU oomph to provide more sophisticated services to a huge customer base. But once those customers were served, more compute, storage, and network capacity didn't buy you enough to justify the capex. That is emphatically not the case with AI, and so the incentives for the AI companies are to buy as much compute as they possibly can. What this means in practicing is pre-purchasing capacity at the semiconductor fabs to manufacture chips exclusively for you, and there's only so much of that capacity in the world. Trillion-dollar companies can easily outbid the entire consumer market, and so the incentives for the fabs are now to sell to AI companies at the expense of the consumer market. That's why you're seeing memory prices go through the roof. Modularized RAM for end-user PC builds will soon go the way of the CRT: it will cease to exist as a market product, it won't be manufactured anywhere by anyone. GPUs, CPUs, and storage will soon follow. The only devices end users will be permitted to purchase are all-in-one integrated devices, with CPU, RAM, GPU, storage, and networking either integrated in-chip or soldered on, and they will have just enough capacity to connect to the cloud services the user wants most to use. Most likely, you will be permitted a subscription to such a device, with automatic hardware upgrades at periodic intervals supplied by the manufacturer. If your subscription lapses the device bricks itself. Almost certainly, the OS will be locked down, with no end-user option to install a different one or even run unapproved software.
If reasonably powerful computer hardware for end users exists in this future, it will be available from a single company: Apple. Only they have the leverage to prevent ~100% of manufacturing capacity from going to high-roller, big-tech firms.
> Trillion-dollar companies can easily outbid the entire consumer market
I don't think this is true. I think prices are rising at the consumer and prosumer level because that's what's required for the mass market to collectively outbid the handful of trillion-dollar companies, at least for the limited share of production they can sustainably demand. This process can continue pretty much indefinitely.
> But one of the really neat things about AI is that there is no limit in sight to the scaling incentive.
How you can be so confident? I can imagine there is some limit and with each scaling iteration gain you achieved will decrease so that further iterations would be more and more look pointless
I'm sure a limit will come around eventually. But plans are afoot to build city-sized data centers, and even then that's not enough to sate the AI superscalers' ambitions, hence Elon's talk about putting data centers in space. This is a level of compute scaling unheard of in our lifetime, and we're still a long, long way off from AGI. So while the juice may theoretically not be worth the squeeze at some point, with the current capacity we have there is no end within sight to the incentive to build more. It will take a number of years at least, and who knows how much environmental/economic destruction, before the dropoff in return on capex begins in earnest.
Well it would be anyone that has access to a datacenter to run them. Which is a ton of companies. And those companies will rent out access to those models. And if they do something stupid to screw over consumers, well the whole point is that there would be a bunch of companies that you could use instead.