That argument swings both ways. There's no real reason to bet on OAI or Anthropic. Not with the rise of local models and the increasingly desperate attempts at regulatory capture.
That argument swings both ways. There's no real reason to bet on OAI or Anthropic. Not with the rise of local models and the increasingly desperate attempts at regulatory capture.
That's what I'm saying. Either one could be Facebook. Or it could be Friendster. The rise of local models raises the even more daunting possibility that nobody becomes Facebook, but you have to imagine somebody will win the consumer market for people who just don't want to bother figuring out a local setup. That market is orders of magnitude less lucrative than the enterprise market, but a company that dominated it would still be one of the hundred or so largest in the world.
Someone can always optimize a model down to running on a typical 32 GB M6 or M5 Mac.
It doesn't need to be the best. It just needs to be good enough. I'm not smart enough, but if I was I'd be working on a locally ran interface engine. That's where you charge say 100$ a year. Said engine would allow you to use a large open source model locally to the point where it's good enough to rival Haiku.
If I had to guess someone is working on this
> Someone can always optimize a model down to running on a typical 32 GB M6 or M5 Mac. It doesn't need to be the best.
Two observations - one, there is some set of tasks where intelligence is already saturated, and sure, that threshold is rising by the month. So, sure, for some tasks you don’t need the best model. But there are other tasks that need models smarter than what we have now - like “make a cure for <disease>” that will yield billions of dollars of value, but likely require quintillions of tokens and higher IQ/token than anything we have now.
As long as we have non-saturated economically valuable tasks there will be incentive to train new and smarter models, and the frontier will always be on racks of GPUs not local distilled versions.
Two, it’s not clear that you can distill all the way down to 32GB even for the current frontier models. IIRC Karpathy has speculated that you can get an AGI into GPT-3 size, if you drop all the memorized facts and keep the reasoning core. But this is highly speculative! The best 32GB models right now are substantially dumber than even the large open-weight models, let alone the frontier closed models.
Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?
>Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?
Someone much smarter than me would figure this out.
One option would be an alternative bare metal OS which could allocate ram more carefully.
I'm sure with billions upon billions in VC funding a solution could be found.
Someone much smarter than me would figure this out.
Imagine if we had some kind of really intelligent AI model that we could pose problems like this to. It's obviously not the kind of thing an open weight model running on your local machine could do, but if we had some kind of really large, closed source model running in the cloud that could solve really difficult problems like this, I wonder if there'd be a market for it...
One way would be to generate a new model for every request. Currently we create an entire universe for each step when training if we predetermined what needed to be created it could be much smaller.
I stumbled upon this neural net compressor by Bellard that beats gzip on compression benchmarks
https://bellard.org/nncp/
> like “make a cure for <disease>” that will yield billions of dollars of value
I think it's important to qualify that while I will very much appreciate such an innovation and I think it'll provide significant value to society... I remain doubtful of the ability for any company to actually capture such an astronomical yield. Either there's enough of an investment required by the company to help get that to market that the yield will be significantly lower or everyone and their brother will just launch competing products two days later. And, as a society, I don't think it's logical for someone writing "Hey Alex, cure cancer" with billions of dollars so I doubt there's going to be much appetite for creating such a moat via legislation.
This gets to the same point of "If ChatGPT can make you a successful million dollar business in a few prompts why is OpenAI sharing it" paradox. It very well may be that AI tools can accomplish the labor that used to be required to create such a business, and I'm sure a few people will land a jackpot and get rich in the transitional phase but if it's so cheap to build it why would someone pay you money for yours instead of just running the same prompting themselves? For most people that's because they lack the knowledge that they can but you'll still have a significant number of instant impersonators.
> This gets to the same point of "If ChatGPT can make you a successful million dollar business in a few prompts why is OpenAI sharing it" paradox
I agree fully, to be clear o was using that prompt as a shorthand, I don’t think _you_ can type that prompt and get $1B back.
It can’t do this now, obviously. You need a lot of scaffolding and domain expertise to get the most out of these models. (Eg Harvey.)
But eventually, the labs can just run the business themselves.
I’ve long held that as we see models become more capable, they will be less available to the public. It seems pretty obvious that vertical integration on private models (perhaps a per-industry fine-tune, eg a legal model) licensed to subsidiaries will let the labs capture more value.
Why would you publish your model, enable competitors, and enable distillation of your hard-won intelligence? Only antitrust law can prevent this outcome IMO, the business pressure is extremely strong.
Great comment, I'm working on my own Hermes/Mac Mini & the amount of sec ops/setting up its arms is a good challenge as a non-technical.
Building the iPhone-platform equivalent of this would be interesting.
The thing is, would the top AI minds rather charge end users 40 to 300$ a month for AI usage of sell a 100$ a year license to run it locally.
I'm far from an expert, but I don't think it's impossible. Future models won't give out vacation ideas, they'll focus on one or two things. This might allow us to get the size down.
Would AI companies rather have $100/yr of license revenue or $480/yr of token revenue at 50% margin? Obviously the latter, by more than 2x? And even larger dollar profit from the higher sub levels you mention.
The constraints will work against them long term. All this is doing is forcing people to look at how to run more with less. These companies are bloated beyond reality and have no incentive to optimize anything. They're starting out with access to everything and still producing very little tangible value comparative to the inputs. OAI and Anthropic represent the pure gluttony of the US tech bro market and it's not going to be the long term winner. It can't be lest the masses are cool at crowning these narcissists as kings.
I think the hardware makers are going to have the last laugh, but not sure if that would be NVIDIA
Rise of local models? what rise? I run qwen3.8 flash next on my 200GB/s 128gb Strix Halo and even that is barely acceptable, big models require so much more. Nobody's running a local model on their power sipping phone. It's just a nerd's wet dream.
That's true today, but what will it look like in 20 years? Over time, algorithms and hardware always gets faster and more efficient. Like if I wanted to run a fluid dynamics simulation, what would have required a supercomputer 20 years ago can now run on a single desktop computer.
256x speedup on FAT32 just earlier today; I expect we'll have a nice bump in performance from domain cross-pollination for a good while now.
barely acceptable for all use cases? like a speed issue?
Regardless, looking back even a year at SOTA local models that fit on a non-absurd system creates a crazy trendline toward the future. And I think model routing is the gap filler that can make/break now vs. Oct 2027
I suspect Anthropic is of this general thinking as well; they've clearly switched from a thoughtful progression toward a natural IPO to "omg sprint for it". Along with the regulatory capture attempts, it's hard to imagine what is breathing down their neck so hard outside of this.
Totally agree local inference of any major scale is a nerd's wet dream short of many stacked improvements in efficiency. But again, auto-routing smooths over the rough edges if you are ok with a little bit of API spend when needed.