I'm sorry, I'm going to sound dumb for a second.
I know that you can use LLMs to write ascii numbers that represent a "confidence" as in TFA. This is fraught and strange and hard to trust.
But can't you also use the next token likelihood itself, so long as you rearchitect to use the input classes as the only possible output tokens? Something like stripping away speech and only working with the input labels as vocabulary...
I guess I thought that's what decision-like models were doing to output confidence scoring. (or something like that)
That is the opposite of dumb, you can use the next token likelihood to macgyver together a jev in a pinch exactly that way, you just would also want to take some steps to scramble up the choices to smooth over bias and tokenization artifacts. llms have favorite letters for example. Qwen likes A and D.