>s that many who do use their cars the most often have their neighborhoods protected from the noise they contribute to everywhere else.

This, so much this. All the noise producing infrastructure in cities is dumped in the highly dense poor areas, and the rich people living in the quiet suburbs in single family home who need to drive in front of your home, are protected by this externality.