"Brown people existing doesn't hurt you"
Yes, existing doesn't hurt. But when you import mass amounts of people who don't talk your countries language, have no intention of learning, and have no intention of getting a job, and on top of it are intensely religious supporting a religion which is antithetical to your values, then it DOES hurt.
Western Europe has simply allowed too many Muslim immigrants in then they could ever successfully integrated. Now Europe is full of ghettos of Muslim immigrants who don't get jobs, don't learn the languages, and sap resources which the countries cannot afford.
Note: this isn't about being racist or not, it's just common sense. There is a limit of how many immigrants a country can take, financially and culturally.
So is the religion the biggest problem or just a symptom?
I'd agree immigration must have limits and don't recall any prominent voices calling for unlimited migration or even limitless compassion.
Being a union, one can hope the load of migrants fleeing wars and violence would be spread among all the states, in proportion to their ability. Economic migration is a more complex matter since everyone benefits from it up to some threshold. Where it crosses into an undeniable problem is subjective. If it negatively affects me personally then it's bad, especially if I can overlook all the benefits I accrued (like lower prices of commodities and services).