So either they care about the safety of women and children, or they want to have some brown skinned and "weird" people to heave their hate on. I know which possibility I think is most likely...

That's a false dichotomy.