Believe it or don't, most mailservers actually are set up this way, but it's completely invisible to the user: https://en.wikipedia.org/wiki/Greylisting_(email)
Greylisting is not perfect, not even close, but it allows servers to reject huge amounts of garbage mail while allowing new mail from unexpected parties without user intervention.
It is similar but not the same thing. It does not have anything to do with the email being legit (non spam) or not. It only catches emails that do not do what they supposed to do when they are deferred for a little bit. They are supposed to try again later. (A super nice feature of email which protects you from losing all incoming emails when your server is down for a short time). The reasoning here is hopefully a spam script or software will not implement the standard properly or will skip apparently broken destinations.
What parent wants is also possible and I remember people doing it as far back as the early 2000s. You would basically manually whitelist an email or have them bypass the list (once or always) with something like a key word or token sort of like a captcha which can be shared off band.
It's definitely possible, a few companies do it, and let me tell you: People hate it.
I've been doing it for almost a decade. Only one person has complained. I'm surprised by the hoops people are willing to go through to contact me.
https://blog.nawaz.org/posts/2018/Sep/solving-my-email-probl...
I'm glad it works for you, but the overwhelming majority of people are not you, and they hate it.
Citation needed.