> If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens

You can!

> AFM 3 Core Advanced makes routing decisions per prompt. A lightweight, dense block selects a fixed set of experts during initial processing, periodically reselecting them during generation.

https://machinelearning.apple.com/research/introducing-third...