In some cases like the Air Canada one where the courts made them uphold a deal offered by their chatbot it'll be "accurate" information whether the company wants it to be or not!

Not not everything an LLM tells you is going to be worth going to court over if it's wrong though.