The 6.5 transistors per coefficient ratio is fascinating. At 3-bit quantization you're already losing a lot of model quality, so the real question is whether the latency gains from running directly on silicon make up for the accuracy loss.

For inference-heavy edge deployments (think always-on voice assistants or real-time video processing), this could be huge even with degraded accuracy. You don't need GPT-4 quality for most embedded use cases. But for anything that needs to be updated or fine-tuned, you're stuck with a new chip fab cycle, which kind of defeats the purpose of using neural nets in the first place.