It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
It is my understanding that just baking the model itself into silicon only gives moderate gains because memory bandwidth remains a bottleneck.
Did you try using the the talaas chat? Something stupid like 18k tokens/second.
Think it's called Askjimmy or similar.
Oh boy, thanks for sharing this, truly mind blowing. It was chatjimmy.ai
The big benefit is ROM cells require fewer components than DRAM. So the chips would be tiny, dense, cheap and consume far less power.
I thought DRAM was pretty dense already. Is mask ROM that much denser?
Yes, each rom bit can be a transistor or even a diode with a decoder circuit. Simplest Dram cell is capacitor+transistor - and you need a clock, refresh circuit etc.
Someday, I imagine model weights could even be encoded as analog resistors (memristors or similar) for even greater density
Hm. I wonder how many relays I'd need to make a physical MNIST classifier. That'd be dope