What is the difference between reasoning and predicting what reasoning would look like?

The same as the difference between a thing and a statistical model of a thing.

They don't always match up.

In theory, theory and practice should be the same; in practice, they are not.

Human reasoning also does not match up with reality. Isn't it statistical modeling all the way down?

That statement demands an explanation.

As far as I'm aware, when reasoning does not meet reality, it is called a delusion, and I'm fairly certain that majority of humans are not delusional.

I wish that were true, but in an election year, I'm constantly reminded that it isn't.

One has a human behind it and one has a matrix latent space. Usually human understanding is more "real"

Stallman vehemently states llms are bullshit generators with no true understanding.

But sometimes you can approximate your problem and unfold an llm interlocator to producing the strings you need for your task