Keep in mind that some AirPods are now certified hearing aids.

So before calling someone rude for talking to you with their AirPods on, make sure they don't need them on to hear you in the first place.

I myself always put at least one down when talking to someone, but I've learned to make the distinction between someone that is actually involved in the discussion, and someone that's still listening to what's coming from their connected device.