This would be very compelling. Can anyone share more details on how it would work? Only issue is that you are stuck at a certain point in time but that’s not a huge deal. Even just a good 27b model would be useful.
This would be very compelling. Can anyone share more details on how it would work? Only issue is that you are stuck at a certain point in time but that’s not a huge deal. Even just a good 27b model would be useful.
Talaas have done this with a llama 3 model. Runs at like, 16k/tokens a second oror something obscene. Very little power draw too.
Doesn’t need hbm or lots of memory, because the hardware can just forward the data straight to the next layer and you don’t need to round trip through memory.
They claim to be working on an approach to make the underlying hardware a bit more reusable between models.
Yeah, if you have a fixed llm topology, you can just effectively burns 2 top layers of the chip as Rom (model weights) - which has a per area density even better than dram - so it’s just attention and kv streaming that is hbm to sram transfer.
Most big model weights will not fit a single reticle sized chip - so you’d have prob 30 different chips to split the model .
And you’d need super fast chip to chip comms for the all-reduce and similar.
So scaling to 1T models is hard - and a long lead time - but can be very power efficient.
There a lot of ways this could work.
1) the hardest, custom silicon + MCU to manage the USB interface
2) not as hard, shared memory, NPU + MCU to manage inference and USB interface
Theoretically you could do 2 with the right MCU, NPU, and memory combo. You'd stream/DMA the weights from memory into the NPU and then read the results with the MCU. From a user's perspective, it might take the form of an openAI API compatible endpoint that enumerates when they plug the USB device in. There would likely be some host-side software to ease the pain of trying to use a USB device as an HTTP API.