That is the drawback - sometimes important calls come from someone not in my contacts (like an emergency where someone borrowed a phone, or a contractor trying to call you.) Still, the beauty of voice mail is that, if the caller is really trying to reach you, they will leave a message. Some random number with no message is almost assuredly spam. That model wouldn't work with email (you would need an equivalent real-time notification of an email going to a spam folder, then the user would have to decide to send to spam or not send anything.)

Anyone without caller id is also suspicious. Emails have a sender, but it is also about as reliable as a caller id (i.e. not very) when it comes to identity.