MOEs are built by training a second "router" model to identify which parts matter inside the dense model.

Think of MOE as ignoring noise, rather than a more efficient encoding of the data we teach it which ones can be ignored. Turning down the noise actually sharpens the results sometimes.

Modern LLM's are wildly inefficient.