They won’t make punctuation mistakes though.

Wouldn't they do exactly that if they were trained on enough text with punctuation mistakes?

No because of post training