10,000 people is as many people as some entire towns, I don't think society would hold together very long if it were true.
100,000 supposes that there are... hmm... about eighty thousand non-evil people in the world, and (odds are) exactly none of them are Marshallese and about 2 are Samoan, to give a sense of how silly this is.
I think you are assuming there are a unique 10,000 people for each gifted individual. I think it safe to say there is overlap and that those 10,000 people are each going after several gifted people.
You also appear to be assuming there are no mid range people who everyone ignores, that is going to be the majority of the population.
But those people don’t have to be evil all the time.
Society holds well if someone steals a candy from the shop.
Those people can become shitty and despicable for that one person they envy and can be great for their friends and family all the other time.
It doesn't . But usually that third of the population is busy going for each others throat and ignoring the "fools" while then taking the coins that mysteriously spawn near those as psychopath price mobey.
Oh, so for every 2 good people there's actually 1 bad person. That's a very different ratio.