> Someone can always optimize a model down to running on a typical 32 GB M6 or M5 Mac. It doesn't need to be the best.
Two observations - one, there is some set of tasks where intelligence is already saturated, and sure, that threshold is rising by the month. So, sure, for some tasks you don’t need the best model. But there are other tasks that need models smarter than what we have now - like “make a cure for <disease>” that will yield billions of dollars of value, but likely require quintillions of tokens and higher IQ/token than anything we have now.
As long as we have non-saturated economically valuable tasks there will be incentive to train new and smarter models, and the frontier will always be on racks of GPUs not local distilled versions.
Two, it’s not clear that you can distill all the way down to 32GB even for the current frontier models. IIRC Karpathy has speculated that you can get an AGI into GPT-3 size, if you drop all the memorized facts and keep the reasoning core. But this is highly speculative! The best 32GB models right now are substantially dumber than even the large open-weight models, let alone the frontier closed models.
Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?
>Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?
Someone much smarter than me would figure this out.
One option would be an alternative bare metal OS which could allocate ram more carefully.
I'm sure with billions upon billions in VC funding a solution could be found.
Someone much smarter than me would figure this out.
Imagine if we had some kind of really intelligent AI model that we could pose problems like this to. It's obviously not the kind of thing an open weight model running on your local machine could do, but if we had some kind of really large, closed source model running in the cloud that could solve really difficult problems like this, I wonder if there'd be a market for it...
One way would be to generate a new model for every request. Currently we create an entire universe for each step when training if we predetermined what needed to be created it could be much smaller.
I stumbled upon this neural net compressor by Bellard that beats gzip on compression benchmarks
https://bellard.org/nncp/
> like “make a cure for <disease>” that will yield billions of dollars of value
I think it's important to qualify that while I will very much appreciate such an innovation and I think it'll provide significant value to society... I remain doubtful of the ability for any company to actually capture such an astronomical yield. Either there's enough of an investment required by the company to help get that to market that the yield will be significantly lower or everyone and their brother will just launch competing products two days later. And, as a society, I don't think it's logical for someone writing "Hey Alex, cure cancer" with billions of dollars so I doubt there's going to be much appetite for creating such a moat via legislation.
This gets to the same point of "If ChatGPT can make you a successful million dollar business in a few prompts why is OpenAI sharing it" paradox. It very well may be that AI tools can accomplish the labor that used to be required to create such a business, and I'm sure a few people will land a jackpot and get rich in the transitional phase but if it's so cheap to build it why would someone pay you money for yours instead of just running the same prompting themselves? For most people that's because they lack the knowledge that they can but you'll still have a significant number of instant impersonators.
> This gets to the same point of "If ChatGPT can make you a successful million dollar business in a few prompts why is OpenAI sharing it" paradox
I agree fully, to be clear o was using that prompt as a shorthand, I don’t think _you_ can type that prompt and get $1B back.
It can’t do this now, obviously. You need a lot of scaffolding and domain expertise to get the most out of these models. (Eg Harvey.)
But eventually, the labs can just run the business themselves.
I’ve long held that as we see models become more capable, they will be less available to the public. It seems pretty obvious that vertical integration on private models (perhaps a per-industry fine-tune, eg a legal model) licensed to subsidiaries will let the labs capture more value.
Why would you publish your model, enable competitors, and enable distillation of your hard-won intelligence? Only antitrust law can prevent this outcome IMO, the business pressure is extremely strong.