Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.
Presumably it’s MoE and only needs to read a small fraction of the weights per token. Bonus points if you can get decent speculative decoding without becoming ALU-limited.
Speculative decoding is not really worthwhile for sparsely-loaded models. You end up paying in both memory bandwith and compute (loading experts based on wrongly-predicted tokens) which leaves you worse off overall. It becomes viable (even for sparse MoE) once you're batching so widely that you end up having to load most of your total weights anyway.
> Speculative decoding is not really worthwhile for sparsely-loaded models.
If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens, without too much quality loss.
Another fun idea might be to try to build a model where the router chooses the expert 1-3 tokens in advance.
> If wonder if you can train a model to optimize this, by trying to make the expert selection sticky across a few tokens
You can!
> AFM 3 Core Advanced makes routing decisions per prompt. A lightweight, dense block selects a fixed set of experts during initial processing, periodically reselecting them during generation.
https://machinelearning.apple.com/research/introducing-third...