I had a thought a while back: sell large local models burned onto fused compute / ROM chips. Like cartridges for old game consoles. Slot (or probably plug into USB-C) and go.
It’s an ASIC with the model wired into it so it’s very low power and fast.
I’d buy these. Say $100 for a frontier class model. Maybe more.
Taalas is developing this, but not for Frontier class models. I hope that if we can least get the easy 80% of work done on that sort of hardware, we can greatly reduce the demand for GPUs, HBM and energy to some extent.
There is an amount of brute forcing that becomes possible at those speeds that I think could even take us beyond 80%. If we could have Qwen3.6-27B running at 15k t/s, run 100 attempts concurrently, select top-K solutions and synthesize a final result from them.
There was a paper a while back that showed top-K selection like that with tiny models was able to reliably solve some 1M-step Tower of Hanoi when no frontier model could. Very big level up in capability just from horizontally scaling compute.
100 dumb folks don't make an Einstein
But some (Meta, Anthropic) suggested that optimizing and extending the "<think>" process can produce extra value. (I do not know if that requires an improved underlying architecture - frontier models architectures are sometimes not public.)
You pull out Einstein when you need a breakthrough.
Taalas[0] seems to be what you're talking about.
0. https://taalas.com/
I love this for the popular sci-fi trope too, where you see some ship engineer swap one glowing crystal "compute core" for another.
We could have the photonic AI model ASICs for real!
Or Galatea’s personality chip in Bicentennial Man :)
I believes the weights are burned as ROM microcode, but for an effective inference speedup, you do want to burn the architecture (matmuls, activation functions, MoE gates, etc) as well which will differ from model to model.
It's not as simple as a weight swap between identical architectures.
The speed gains are also from not having to route the weights through wiring like with ROM cartridges.
Sounds like something DARPA would be working on right now.
Interestingly, you could easily run them from the said old consoles! You'd just need a bit of console code to interface (text input/output) with your fully independent LLM subsystem. Imagine Claude for the NES without Internet?
This would be very compelling. Can anyone share more details on how it would work? Only issue is that you are stuck at a certain point in time but that’s not a huge deal. Even just a good 27b model would be useful.
Talaas have done this with a llama 3 model. Runs at like, 16k/tokens a second oror something obscene. Very little power draw too.
Doesn’t need hbm or lots of memory, because the hardware can just forward the data straight to the next layer and you don’t need to round trip through memory.
They claim to be working on an approach to make the underlying hardware a bit more reusable between models.
Yeah, if you have a fixed llm topology, you can just effectively burns 2 top layers of the chip as Rom (model weights) - which has a per area density even better than dram - so it’s just attention and kv streaming that is hbm to sram transfer.
Most big model weights will not fit a single reticle sized chip - so you’d have prob 30 different chips to split the model .
And you’d need super fast chip to chip comms for the all-reduce and similar.
So scaling to 1T models is hard - and a long lead time - but can be very power efficient.
There a lot of ways this could work.
1) the hardest, custom silicon + MCU to manage the USB interface
2) not as hard, shared memory, NPU + MCU to manage inference and USB interface
Theoretically you could do 2 with the right MCU, NPU, and memory combo. You'd stream/DMA the weights from memory into the NPU and then read the results with the MCU. From a user's perspective, it might take the form of an openAI API compatible endpoint that enumerates when they plug the USB device in. There would likely be some host-side software to ease the pain of trying to use a USB device as an HTTP API.
Kinda already exists.
demo https://chatjimmy.ai/
https://news.ycombinator.com/item?id=47103661
Hallucinates on the first question I ask, as 90% of these models that try to take shortcuts.
You’re expecting the wrong thing. The demo demonstrates the insane inference rate of dedicated hardware. Iirc it’s llama 3 or something. Not a very good model by today’s standards. But it runs at 16k tokens per second, an order of magnitude above the competition.
Imagine what’s possible if you had GLM-5.2 turned into a hardware chip like this.
Your statement reminds me of Avenger's scene where Tony choosing Friday among other AI's catridges to use.
That sounds good and practical to happen!
> I’d buy these. Say $100 for a frontier class model. Maybe more.
Sure you would. Running frontier class models on current hardware costs in the order of tens of thousands of dollars. It is more likely that these custom ASICs will be priced competitively with that, and not with Super Mario Bros.
Oh, and energy consumption will be in the same order.
https://chatjimmy.ai
You need terabytes of memory to run a frontier class model
I wonder, if you can run at 8k or 15k t/s, you could in theory run 10 or 20 agents (or more) at the same time and generate hundreds of versions, then just analyze them. Think thinking mode x1000 at least... Would be interesting to see how good it would be
How very Cyberdyne.
[dead]
Wow, I'd really love it if that were the case. I'm already pretty satisfied with just GPT 5.6 as it is.