It will make mistakes if you tell it to, so I assume it will make no mistakes if you tell it not to.

When this news came out I amused myself by asking Claude to prove that 0.999... != 1. First it did so for the hyperreals. To do it for the reals I had to tell it it was allowed to make mistakes, although it didn't end up interestingly wrong - just very fuzzy and vague.

Actually it's not true for the hyperreals either, though this is a common misconception. It's extremely easy for me to believe that an llm would produce a "proof" and that people would fall for it.

Fable, prove that for every positive integer n, repeatedly dividing by 2 if even or multiplying by 3 and adding 1 if odd will always eventually reduce the sequence to 1.