This is where MoEs shine though. You don't need all experts in memory at once. Diffusion inference doesn't have sparse inference.