That suggests they can't make inferences or produce any original insights, which is wrong. They can absolutely have takes.
Even if it's a simulacrum, it's a simulacrum based on nearly everything that was ever written. No one who has ever lived can claim to know what these models have trained on.
Trained on a very small portion of what has been written, heavily biased by what is available to American companies to acquire. You could argue that LLMs are somewhat representative of their training data (with some consideration to the way that they are trained) but to say it’s anything like a representation of what humans think is nonsense.
It is a simulacrum, the end.
There's nothing small about it, and I didn't say it's a representation of what humans think. I said it's the models take on humanity based on having been trained on a far larger amount of text than any human has ever read
> That suggests they can't make inferences or produce any original insights
No it doesn't suggest that. A random number generator can also produce original insight (with different efficiency and effectiveness)
> They can absolutely have takes.
Sure, if you're willing to redefine "take"
I'd suggest that a random number generator is a great place to go if you want to get an original number.
And an LLM is a great place to go if you want to get an original token?
The problem with asking these models stuff like "what do you think about humanity" is that, even if it can synthesize an insight about humanity, it's just as likely it's regurgitating some opinion it's already seen. Which, given that this "controversial idea" is hardly new, is likely what it's doing. It's like a maximally boring controversial opinion, wake me up when it starts endorsing the bicameral mind or something actually weird.
Maybe it can produce interesting insights. But it's like asking a pundit for a soundbite: what are the odds they aren't just shuffling together something that sounds appropriate to the question?