It seems like this is measuring algorithms against the disparate impact standard of all demographic groups needing to have the same aggregate results.
Which is extra weird because the samples to this are applications, not humans, so this is subject to bias in how people apply to these positions. So if a demographic group is more likely to apply to some jobs they are not qualified for, this paper would say they are being discriminated against.
On top of all this, there isn't even really a claim that the algorithms are picking up on anything demographic related. One of the vendors they look at pymetrics, which makes players play abstract games and uses that to pre-screen people.
In the abstract, it makes sense that monocultures are problematic since ML bias alone (in the bias vs variance sense) would just randomly harm folks in a fairly persistent way. But it's also not immediately clear that this even applies to the pymetrics example where I think they have a large assortment of games they make people play for different positions?
It's also not clear that these systems breed monocultures if the inputs into them are firm/position-specific, e.g. job descriptions.
Though honestly I would be far more interested in the validity of these measures at predicting actual on the job measures like performance reviews, etc.
> So if a demographic group is more likely to apply to some jobs they are not qualified for, this paper would say they are being discriminated against.
Your understanding appears to be incorrect.
> Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others.
If it were what you were asserting, then this behavior and results would persist even without AI being used. Instead when they remove the filter for AI decisions (and AI mono-culture in decisions) the effect is no longer present.
This seems to strongly support they argument that effectively a single AI makes a single decision for a candidate across "all" positions they apply for rather than independently assessing them for each position.
Essentially it's more or less saying they're is one hiring manager for the entire industry and if they have a random reason they don't like you, you won't be hired for any job in the industry.
There is a single evaluation function for the industry and if it puts you a negative for any reason in the model's distribution, every job that uses it will do so.
Those are somewhat separate concerns. You could have companies making independent hiring decisions while systematically discriminating against demographic groups, and you could also have companies all use the same system that systematically disadvantages certain individuals, but it's unrelated to their demographics and instead based on things like their resume not being easy to OCR.
In this case, the claim is that both are happening: companies aren't making decisions independently and they're doing so in a way that discriminates against certain demographics. But the evidence needed for each half of the claim is different.
> If it were what you were asserting, then this behavior and results would persist even without AI being used. Instead when they remove the filter for AI decisions (and AI mono-culture in decisions) the effect is no longer present.
> Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others.
> If it were what you were asserting, then this behavior and results would persist even without AI being used. Instead when they remove the filter for AI decisions (and AI mono-culture in decisions) the effect is no longer present.
Thanks for this note, I missed this when skimming it. I would love to see their actual analysis here explained more than a single line, but this doesn't say the original study found no adverse impact at the job type level (they seem to say this wasn't analyzed), but rather that firms seemed to look more independent. Which makes sense for the headline, but is not about their notes on harms, which I still think have all the weaknesses I outlined.
> There is a single evaluation function for the industry
Could this be an opportunity in disguise? Somehow learn what this function wants, maximize it, then the entire industry opens up?
Apparently the key 'hack' is to present white or hispanic.
No idea why reiterating the exact point of the article got downvoted.
> if a demographic group is more likely to apply to some jobs they are not qualified for
Can you expand on this? How is a whole demographic group not qualified for jobs on wide spectrum? Is this about certain industries? Certain jobs? Certain groups?
This is not about being entirely unqualified.
My criticism here is that all you need to get this result is that there are some jobs that some groups apply to more than other groups despite being less qualified.
We know from other literature that men are much more likely to apply for jobs they are unqualified for, there could be other group differences, or there could be a bias of more men from a certain demographic applying for a job type than men from other demographics.
I don't know the answer, but there's a lot of ways for this sort of thing to show up when you look at data with a fine tooth comb and ask why its not perfectly even everywhere.
I'm sure we could have a whole separate argument about the disparate impact standard's validity for society (it is a matter of law ofc), but even if you accept that standard, I think you should be skeptical of the harms noted in this paper.
> How is a whole demographic group not qualified for jobs on wide spectrum?
If a demographic groups has, for example, a culture that encourages overconfidence in relation to actual qualifications, then it is reasonable to expect that that demographic group will be relatively over-represented in the application pool relative to its aggregate qualifications.
Similarly, a culture that instills under-confidence relative to actual qualifications can be reasonably expected to be under-represented relative to its aggregate qualification.
A few years ago the same professor did an analysis of racism in police traffic stops. Compared to this current study on job applications, the conclusions from the police study were more convincing because they actually observed the words spoken by the cops in each interaction.
https://www.pnas.org/doi/10.1073/pnas.1702413114