You certainly can't ask an LLM for its confidence level because it will generate that answer as it sees fit.

My understanding was that models can be post-trained to assess their own confidence on short prompts within a level of accuracy (there was an article about this a few weeks ago that I can't find), but can't use it to reason, and that research has shown that internally they effectively have a measurable sense of truth but can't surface it:

https://arxiv.org/abs/2410.02707

I mangled what I was trying to say but the point I guess is, it is effectively in there, and researchers can see it and therefore score it, but it is not something the LLM can use.

The problem seems to me (layman's understanding at best) that the LLM is inherently confident within the words of its answer, because of what the model is trained to do and how it is trained. So you can never get an accurate "I don't know this" from a model while it is answering; it is bullshitting.

(One of the things that is most interesting to me at the moment is asking a small local LLM what it knows about a topic and then testing it. It can have no idea of what it wasn't trained on, but any question about what it knows is seemingly going to trigger it to work through all the summary word associations it did find in training, so it can sort of summarise its knowledge that way, with some likelihood of success, without any sense of introspection)