While I agree the autoregressive training is poor at calibrating a model, it's clear that LLMs encode a kind of world model to be able to reason, do mathematics, programming, and logic. They are certainly imperfect, but the emergent internal representations go beyond mere "what text comes next" even though LLMs typically simply use tokens.

By analogy to your 'Measuring confidence' section: a function estimator (neural nets can be good at this) and a bunch of past observations could be pretty calibrated at making weather predictions. An LLM has a wild mix of past observations and random collated opinions from previous text (it might mention months, recency of rain etc.. etc...) so lands somewhere between 'text completion' and 'estimator' with a model of unknown fidelity. It is not 'merely' generating a plausible sentence.

I would never use an LLM for accurate forecasting, and certainly agree they are not calibrated, but I would not dismiss them out of hand.