> At 20 billion parameters per chip, you’d need just 50 accelerators to support a trillion-parameter model
I don't see any evidence that this is possible. From my understanding, the whole model needs to be on a single chip. Which rules out any popular frontier models with several trillions of parameters. Even smaller sub-frontier models have hundreds of millions of parameters, so these would be ruled out as well.
The methods for splitting weights across multiple chips are well established. Groq/Cerebras can't hold a model on one chip either.
Umm I have an extra 35, do you have layer 6?
I think it's enough that a single layer fits on each chip if you can daisy-chain them with good interconnects.
They pipeline-parallelize across multiple chips. DeepSeek v4 Pro will be 30 chips.