I think people who care about correctness and also read a lot automatically see the difference and it seems (and is) technically incorrect to use a hyphen where an em dash belongs. That’s really it. Kind of like you wouldn’t just leave out the apostrophes in your writing even though in most contexts they are not strictly necessary for comprehension.