I think you're making the wrong takeaway here. LLMs aren't bad because of sycophancy tendencies. Any prediction at all is bad when it isn't based on real data, whether or not it ends up being correct, whether or not it is made by an LLM, a human, or some other thing capable of spitting out a number.

Analogize to basketball here. How confident are you that you can hit a free throw? If you ask someone who has played a thousand hours a year for the past ten years, has recorded their own statistics, and has averaged 70% success rate oer those past ten years, and they tell you they're 70% confident they can hit a free throw, that's a good prediction. If you ask me and I give you any prediction at all, I'm full of shit. I haven't played basketball or taken a free throw in 30 years. I can pull a number out of my ass, I can reason through it, I can do a Fermi, I can assume I'll be the average of all people who have ever tried and do my best to collect whatever data might exist on brand new players with no experience trying to take a free throw for the first time. I can narrow that to people my height, my age. Now we're getting better, but it's still not as good as having a real record of my actual past performance and basing a prediction off of that.

This is just science. Hierarchy of evidence. Epistemology. Sometimes answering a question requires collecting data that doesn't exist yet and waiting a while, in the meantime admitting you don't have an answer. It was already sorely lacking in the world, but in this age of oracles, it's only getting worse.