Why are you surprised by LLMs doing irrational or weird things?

All machine learning models start off in a random state. As they progress through their training, their input/output pairs tend to mimic what they've been trained to mimic.

LLMs have been doing a great job mimicking our human flaws from the beginning because we train them on a ton of human generated data. Other weird behavior can be easily attributed to simple fact that they're initialized at a random state.

Being able to work on and prove non-trivial theorems is a better indication of AGI, IMO.