Keep in mind that the model is thinking in a token space, itself a compressive representation of language.
(Note: there's still a huge grammar penalty, so, ugh do think small.)
Keep in mind that the model is thinking in a token space, itself a compressive representation of language.
(Note: there's still a huge grammar penalty, so, ugh do think small.)
The real breakthrough is going to be thinking in latent space.
Arguably this is already happening: the whole state of the model gets fed through from token to token, and even just shoving a bunch of dashes in between the input tokens and the model's output can improve performance (thinking tokens from the model help a little bit more, but the difference is not as large as you mught expect).
It selects tokens but they expand to embedding vectors which are huge, also in memory and attention requirements, I think?