People drive to work anyway. A vacation or two every year is probably not even a double digit percent of a person’s total fossil fuel usage, and gives them a lot of happiness and reason to work and do things that are good for society.
Also: https://en.m.wikipedia.org/wiki/Fuel_economy_in_aircraft
It is pollution and it harms people. So in order to do things that are good we must harm others first (or after)?
Why is there a segment of the population that wants to live in poverty and squalor?
How much pollution is okay? Why not argue for efficiency standards rather than bans?
Everything could be said to “harm people”. Banning travel could make some people depressed and who knows what that could lead to? Or it might lead to a less connected world and less familiarity with people in other places, and maybe makes wars more likely?