The vast majority of the human population is lactose intolerant, both historically and today. Genetically intolerant populations in South and Central Asia have microbiotic help with their dairy-heavy diets, but for people who didn't spend thousands of years developing a culture around it, dairy is just a quick road to an upset stomach and/or food poisoning.
That makes some sense. Given the historic sometime scarcity of food and pressure of starvation, and the widespread availability of milk, I would think people would adapt to it.
I guess that lactose-intolerent people today would drink milk rather than starve - do they get zero nutrients from it? - and that evolution would select for those who could survive that way.