> You’re essentially forcing the model to express the actual answer it wants to express in a constrained language.
You surely aren't implying that the model is sentient or has any "desire" to give an answer, right?
And how is that different from prompting in general? Isn't using english already a constraint? And isn't that what it is designed for, to work with prompts that provide limits in which to determine the output text? Like there is no "real" answer that you supress by changing your prompt.
So I don't think its a plausible explanation to say this happens because we are "making" the model return its answerr in a "constrained language" at all.
> You surely aren't implying that the model is sentient or has any "desire" to give an answer, right?
The model is a probabilistic machine that was trained to generate completions and then fine tuned to generate chat style interactions. There is an output, given the prompt and weights, that is most likely under the model. That’s what one could call the model’s “desired” answer if you want to anthropomorphize. When you constrain which tokens can be sampled at a given timestep you by definition diverge from that