>without an actual understanding of what they're doing.

At what point do you start to question your assumptions that are causing you so much cognitive dissonance?

But to answer your question: to predict the next token really well you just have to model the world. Think of it like this, a simple statistical model might say "when token A is seen respond with token B". The next step will add conditions, "...respond with token B unless X has been seen, then respond with Y". Add a few billion more of these contexual clauses and you have a sequence of logical rules that indirectly model the relevant processes in the world.