I read a comment on here a few months back I wanna restate. Basically, there is a good chance that Apple is betting that the LLMs in the future will be so efficient that those that consumers will use everyday will be easily computed by the iPhone or even bigger ones on Macs. Honestly makes the most sense that we are heading that way in a few years latest.
The innovation curve doesn’t always reward the innovators. It pays to let others test the waters before you.
Especially if you’re Apple and your expertise and reputation is in making privacy conscious devices and cloud services that are exceptionally user friendly.
Then you go to the AI labs and say, “We see you’ve spent a trillion dollars on that model… we’ll write you a check today to let us run it how we want.”
Skips the ugly horse race and massive risk while focusing on what they do better than anyone.
And they’re the only ones who have cultivated the trust with normal people to be able to say, “We now have the only AI service that can safely reference your personal info without giving it all to strangers.”
They could possibly end up the second biggest winners in this insane hype cycle… without having had to compete on making a sota model.
What hardware advances would we need to see for that to happen? It feels like everything in that arena has kind of plateaued.
A lot of this will come from co-optimizing hardware and low level machine code for this specific use case… something apple is coincidently very good at.
Apple has worked very hard to make unified memory a feasible approach, and the benefits of that are pretty clear in apple silicon- that efficiency not only results in power and therefore thermal gains, but also in a significantly faster full loop per process: or a faster time to token. This is why even their single core mobile chips in the budget line Neo out perform PC processors with several times more threads and RAM[1]. Turns out, unified memory lets you have a whole lot more control over things like RAM bussing and core use for specific workflows.
Speculatively, a unified memory approach could also allow you to more easily integrate things like ReRAM to solve the current memory swapping bottleneck.
Let’s say a friend of mine works hardware at apple and works on exactly this… on device processing is the future I’m betting on.
[1] https://youtu.be/x26A28DoT-w?t=605
> This is why even their single core mobile chips in the budget line Neo
That’s a six core processor. It’s an A18 Pro in the Neo, same chip as on the Iphone 16 Pro
The models themselves have far from plateaued. Maybe someone finds a way to get a really capable model down to, say, 12GB of ram. Then we'd be in business.
Agreed. We've just seen DeepSeek post-train their ~300 billion parameter flash model to outperform their 1.6 trillion parameter pro model, in the space of a few months. There would seem to still be quite a few opportunities on the table to bring big model smarts down to the smaller models
I think a major factor is memory bandwidth. Apple has raised it steadily for each M series generation, and that hasn't plateaued.
Nvidia leads in bandwidth and specialized architecture, but local inference takes off when it's usably fast at much lower cost and power consumption.
It could be on software side too. OpenAI has certainly not plateaued.
I could definitely image Apple embedding a kind of LLM-optimized FPGA: slow to load (update) an LLM, but blazing fast at computing tokens.
Who needs memory when your model is set in silicon ?
You don't an FPGA if you're taping out your own chips. But that is just a MMA accelerator with decent memory bandwidth. No secret sauce here.
I am talking about reconfigurable gates to implement an LLM in silicon, i.e. an FPGA...