Yes but what prejudices we accept and which we do not is arbitrary. You have no choice over your height. Taller people can have more promotions and more dating opportunities, but we don't have affirmative action for short people, and we don't treat women who say they don't date short guys like we treat women who say they won't date black guys. There's always going to be discrimination, what form of discrimination is acceptable or unacceptable is still arbitrary.
First, let's not talk about dating and work in the same argument - they are fundamentally different in many important ways, and it's not conductive to the conversation.
Second, at least call a spade a spade - according to you, when people say meritocracy they actually mean "meritocracy with handicaps for non-whites and non-males". Let's not call that "meritocracy", okay?