MoE doesn't work that way. It's not about increasing performance of a task, but speeding it up. To get a faster model, you need to make it smaller. If you make it smaller, the less it can remember. MoE tries to get faster responses without reducing the capabilities by running a smaller part of the model, and constantly swapping the parts used, so that the whole model gets to influence the output collectively, but each token is created by a smaller set of weights.

Anyone else feels like they can tell when the underlying model is MoE? Dense models are more concise in their answers. MoE seems to stall for time, writing vaugely until a right set of weights gets picked and commits to an answer. MoE also backtracks a whole lot more.