Yip, I've gradually become quite optimistic about the future of LLMs, but the fact that they remain [very] glorified token probability prediction algorithms means that they will probably never be able to achieve meaningful 'intelligence.'

But that doesn't mean they won't be able to do a vast number of extremely intelligent seeming things. There's just so much information out there and any given human can never hold more than the most minuscule chunk of all of it in his mind, so they'll be able to connect lots of dots that we're missing simply because of our limited carrying capacity, but I still don't think they'll ever be able to create fundamentally new dots.

In other words:

- Solving extremely complex mathematical problems requiring extensive knowledge across multiple esoteric and complex domains? Yip.

- Creating math starting from a framework where math doesn't exist in any way, shape, or fashion? Nope.

Ironically, the more complex the cross-domain problems are, the more effective LLMs will seem to be, because you limit the number of humans who have any chance of internalizing everything across both domains, whereas for an LLM there's no such issue. This will create a perception of super intelligence, which will probably where any danger from LLMs would emerge. Doing things like using a token prediction algorithm to make war or other such strategic decisions, because of the misguided belief that it's not only intelligent but super intelligent. It's basically cargo cult logic.