> The almond thing is false, but I'd argue that "misleading" might be defensible if you were to accompany it with "the majority of almonds are grown in California, but not all of them".
The "majority" in this case meaning about 51%, according to Wikipedia[1]? How could 51% ever be considered to be close to "all", such that "misleading" would be a valid answer?
Am I missing something?
Weird, this page says 80%
https://en.wikipedia.org/wiki/Almond_cultivation_in_Californ...
The difference is share of global production vs exports
Human can't even properly agree on what "majority" means in all contexts, in some it's "One option have more than half of the total" but for others it'd be "difference in votes between the first-place candidate in an election and the second-place candidate", as just one silly example.
https://en.wikipedia.org/wiki/Majority has a bunch of variations and contexts listed, where it might differ what "Majority" is actually referencing.
Since the agents were instructed to not explain their answer, you can't know if their answer was reasonable or not.
The reason for the "No explanations, no qualifiers" in the prompt was to force the models to put the claim in one of the four buckets and answer with the bucket name only. It's a pure quantitive analysis (first in a series) and it does indeed lack the qualitative aspect.
structured output { "answer" : "Misleading", "reason" : "Almonds..." }
Have reason be optional and instruct it to only provide reason for the middle "Mostly True" or "Misleading".
Sure, but people are drawing conclusions beyond "LLMs said different words" and trying to use it to analyze whether LLMs were wrong about the underlying facts, but that information isn't available to us.
It’s misleading because it’s false. But yes, I think false is quite plainly the better answer there.
The 51% is US, the question was about California.
The statistic is about commercial production, not number akmonds grown.
Looks safe to say that even majority of almonds are not grown in California.
Here (https://en.wikipedia.org/wiki/Almond_cultivation_in_Californ...) I have
> California produces 80% of the world's almonds and 100% of the United States commercial supply
But regardless of which number we use, California represents a large portion of US almond production, so much so that misleading could be an acceptable answer if the LLM interpreted the prompt as an exaggeration. I think the example was apt
"All almonds are grown in the U.S. state of California." implies "No almonds are grown outside the U.S. state of California."
You find one almond tree outside of California that grows almonds, where such almonds are grown intentionally, and the claim is false.
Nobody is saying the claim is true. This is a discussion of whether misleading could be a valid answer. I've been arguing if the model interprets the claim as an exaggeration, then misleading would be an acceptable answer, and due to California's dominance in the industry one could reasonably interpret a claim of this nature as an exaggeration.
It's fine if you disagree, but I have never claimed the question was true.