Seems like what Apple's going for with afm3. Their latest model that will be embedded in macOS 27 is a quantized dense 20B that only select between 1 to 4B at inference, based on the prompt, not token by token. If only they could make a 100B or 400B dense that selects ~5 to 15B...
I don't understand, if they are only using a subset of the tokens then it's a sparse model. What do you mean by dense?
Could it be some sort of permanently routed MoE where they detect and switch for the whole prompt instead of token by token?