Black swans are actually fairly well-populated birds that are easy to spot. So as a metaphor it works well for a common thing that smug people of limited knowledge try to claim is rare and surprising.