SOTA models are reportedly MoE, not dense.