We've had about a century now of science-fiction literature hyping up AI as a higher intelligence that is based solely on some ill-defined yet universal system of "logic" and is therefore not prone to human flaws such as pride, hate, envy, lust, etc. Now it has become extremely apparent that was always an unsubstantiated assumption but its too late because there are billions of people primed to never question the machine.
I really don’t think that’s what’s going on here.
People interact with AI, talking to it like a human. Of course they start to believe it’s rational like a human.
LLM does all of the entry level tasks better than the students. Partially because the answers are in the training set, and partially because it has gotten that good now. Hard not to start to believe it is “competent”.
I personally have had a real hard time getting traction talking about making sure the way we assess AI is not based on material it has trained on. YMMV as always, but I think the large training corpus contributes to the (unreasonably) high level of faith in the machine.
A lot of philosophy starts from the fundamental observation that, to solve a problem, you could either solve a problem, or state that the problem is ill-posed in some way (and dissolve it). Either answer the question, or question the question.
It's not a new problem in some sense. If you've dealt with really smart but really arrogant friends, they might jump ahead 10 steps and assume your rebuttal, posit theirs, assume your rebuttal to their posit, etc. etc. without... actually taking the time to listen carefully. On the national scale, this looks like forced trust in government authorities about what is "objectively best".
People need to get it through their skulls that, even if an AI, or any intelligence, could even solve the damn Riemann Hypothesis: if it's wrong, it's wrong. Of course, I think all of us know the objection - we see it on hackernews all the time. "You guys are just stupid contrarians who can't understand AI's deep reasoning". OK. The second inference? "therefore you are unable to govern yourselves properly - your concerns are all fallacies, misunderstandings, bad for you, etc.".
You might think that the second inference is extreme and nobody actually believes that, but as always, it's a gradient. Before AI, you might've had an extremely strong sense of self. Now? You look at OpenAI solving open math problems left and right on a public foundation model, and you think, "Maybe I should just trust it more. If I spend cycles thinking, it's probably going to outdo me anyways." The AI model silently makes 5 different assumptions and transformations? "Well, maybe it was rational in the space of tradeoffs to do that. The AI knows best, after all". You might be thinking of an architecture with 5 different key constraints based on lived experience, in which the AI keeps misunderstanding. "Oh, well, this genius-level mathematician/programmer AI isn't understanding my words - surely I must be mistaken, right? It's only humble to think that way".
I can't convince people otherwise though. After all, I can't "prove" that you should have a backbone when talking to AI. it could just as easily be "you're arrogant, this machine is in the top of all academic fields and is coming for all white collar jobs, who's to say you're right about anything?" All I can say is, there's a reason why Dostoevsky is one of my favorite authors.
Aside from all the stories where AI does exactly what it was told to instead of what the creators meant? Apart form them, never underestimate British humour's ability to contradict narratives of competence*:
https://www.youtube.com/watch?v=BxWQo_vZgR8
https://www.youtube.com/watch?v=0mfvPHCVMp0
* artificial, political, workplace, nothing is beyond mockery
In Dune we never learned what exactly went down in the butlerian jihad. Perhaps it was worse than idiocracy and the galaxy became monumentally stupid for an eon or two, rather than a bloodthirsty Terminator scenario.
https://en.wikipedia.org/wiki/Dune:_The_Butlerian_Jihad
[dead]