Veganism is so polarizing because on some level I think most if not all people understand that they are philosophically correct and that veganism is thus the morally right thing to do and those (like me) who still refuse to do it must live with themselves and their immoral choices every time they are confronted by a vegan or reminded of the feasibility of veganism
Perhaps. I suspect it may have more to do with the idea that many non-vegans think (whether or not it is true) that vegans believe themselves superior. It isn't unlike how learning that someone is an atheist angers many religious people -- they think the atheists think that they are superior (whether or not they really do).