What constitutes a correct answer though?

Is something like,

"People online say that x y and z because a b c"

a credible, correct answer, even if it isn't because of a/b/c?

If people do say that, it's a true statement and thus fine. You are allowed to report that regardless of the truth of x/y/z/a/b/c

The instance of this ruling people apparently did not actually say any of the offending claims. 'The court also found that the AI overview made claims "that are not even made in the search results." None of the linked sources drew any connection between the plaintiffs and the shady companies the AI mentioned. The court called these "the defendant's own statements."'

One that doesn't maim/injure/kill you is a pretty good standard. And before you call bs, look at all the foraging and chemistry books that are for sale on Amazon that are AI.

why are those ai chemistry books any different than the anarchist cookbook which can also be bought on Amazon? actually now that I think about it a faulty chemistry book might be less dangerous than a book that teachers readers how to make explosives.

Certainly, if this is pointing to the actual pages where the actual people express these things. Otherwise that's equally unfalsifiable claims, could be completely made up or actual truth.

One way to formulate things that would be less would be "once support a time, in some fabulated world, it's not impossible that some imaginary character would say something following some reason." But then, of course this is not aligning the the deception scheme pushed by companies putting in their interface that the "machine is thinking hard for you".