That's kind of crazy. Why doesn't Microsoft revoke such certs such that you can't sign new software with it?

Because it's mostly just performative.