What next token is “optimal” is fuzzy and subjective. All transformer based models have a “temperature” setting whose sole purpose is to randomly make choices other than the most likely next token. This is crucial to good output, but you wouldn’t call those choices “non-optimal” even if they are less likely. In any text generation task there are constant opportunities to make a choice from equivalent options.