For one, because it's an indicator of health and healthy development, which are in turn related to other desirable features. Analogous things can be said for women.
You also can't ignore assortive mating. We're not all in the same mating pool, as it were. Women prefer men taller than they are, but that doesn't mean that a woman of 1.25m will pair with a man of 2m. The typical height difference between husband and wife is about 10-15cm, regardless of the absolute height of either spouse.
I've seen plenty of evidence that goes against this. From what I have seen, short women will absolutely prefer a man who is taller than other men, regardless of their own height, so something does not add up. In fact, I've seen it often be the other way around; a man will sometimes prefer a woman who is not a midget compared to himself, purely for practical reasons (unless of course he falls in love).
I will go as far as to say that assortative mating is likely a side-effect of people ending up in the same or similar socioeconomic circles, but that is just me reaching. In reality I would guess that it's a bit of that, and being used to people from a certain "class".