Exactly, it's like saying Wikipedia shouldn't be liable and immediately shut down for any wrong information shown on it.

Wikipedia doesn't have to shut down, but they have to remove the libel.

The problem for LLMs is that they do not learn, and can't be prevented to produce that libel ever again. If Google finds a way to make that happen, no court would stop them from offering an LLM.