This might be silly, but … since the assistant models are so much smaller than the full models. What if we just use those smaller models?
Any idea how much worse they will be ? Or is the issue that their error will really diverge as you accept more of their tokens?
I think they'll be extremely worse on their own
Predicting "America" in "The United States of ..." Is a different task from predicting the whole sentence.
So the small model is laying the blocks, and the bigger model would be cementing them in place or kicking them down. The bigger model's course correction is what keeps the smaller models predictions relatively on track
I assume these are just output layers that are trained on the hidden state from the larger model - that's how MTP works. It's not a separate drafting model.