I'm somewhat doubtful that we will be seeing something as large as Kimi K3 in silicon any time soon.
This tech can definitely scale up from the current 8B prototype, but - at least as far as my limited understanding of the tech involved goes - you cannot just ASIC a trillion weights model due to physical size constraints.
___
Specification HC1
Model Llama 3.1 8B (hardwired)
Process TSMC 6nm
Die size 815mm²
___
So the current prototype already pushes the limits of what we can fit on a single die, and that is already likely going to limit your yield.
This is an architectural limitation that may be overcome by how you bake the MoE (mixture-of-experts) onto silicon.
If you could manage a per-die expert somehow and keep the expert routing gate relatively fast (through an interposer interconnect or doing wafer-scale Cerebras type shit) you don't need to keep the whole thing on the same die. Small dies with one expert per die on an interposer, and a very tiny router might be sufficient.
Kimi K3 is huge, though. Deepseek V4 Flash is a much more moderate model (284B total), and it works extremely well. Models of that size, and smaller, are just going to keep getting better and better. Presumably there's a threshold below which models are not generally useful or competitive, but if models-on-silicon can scale up to just 256B, that would be really remarkable.
My 2 cents to for your point:
- Deepseek V4 Flash is impressively capable. Sonnet still beats it out by a thin margin, but the real kicker is that a typical session with Sonnet at current API costs is ~$2. The same session with Deepseek is 2 cents (ha). Its even allowed me to consider offering free-with-limits API usage on my own app. - Taalas (or competitors) have a lot going for them. If anything I feel like they need to join hands with these smaller model makers and converge in 2028