No modern LLM can tell how many eyes the magic card Pit Imp has. They all say 2. This is across all reasoning levels and paid Gemini, Claude, and GPT (Sol)
Drawing a line red to split up the image then has them answer correctly.
Their failure modes are highly correlated.
Yes - those failures (like strawberry) are. And those failures are very rare, which is why you had to reach for the Pit Imp MTG card, which I had to Google to understand your point.
But the failures are universal and correlated
Not really.
Cohen, Hamri, Geva & Globerson, "LM vs LM: Detecting Factual Errors via Cross Examination": Cross-examination "detects over 70% of the incorrect claims while maintaining a high precision of >80%".
[dead]