I'm not sure this is very easy to show this is a breach of non-discrimination requirements, like under Council Directive 2000/78/EC for employment.
Due to acting like an irrational gambling machine, I agree it can have unwanted indirect discrimination effect in general. But it will probably not differentiate "on the grounds of religion or belief, disability, age or sexual orientation". It is possible, but that would take a lot of work for the lawyers to prove to the court.
I believe the more interesting part is that the EU AI Act (still not in force in this regard until 2 December 2027). This will be clearly a high-risk AI system: "AI systems intended to be used for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates".
Which does not mean prohibited, but it could later turn out that LLMs will be excluded from being used in high-risk AI use cases (falling under article 6 with no exemptions).
Considering that none of the standards are published yet, I have absolultely no idea how they will ensure compliance with the following parts of Article 10 when using LLMs for such tasks: "(f) examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights or lead to discrimination prohibited under Union law, especially where data outputs influence inputs for future operations; (g) appropriate measures to detect, prevent and mitigate possible biases identified according to point (f)"
I don't think that's technically possible to do so with LLMs in general at the moment, even with the full cooperation of the model providers. Maybe you can do some meaningful audits for smaller models. But the EU AI Act may end up excluding all the generic "using-LLM-but-not-entirely-sure-why" vibe coded approaches from high-risk use cases (in Annex III). Which would make sense.
EU AI Act got hijacked by huge corpo with last minute changed with moved it from "could probably work" to "catastrophe".
Even at 2 December 2027 it might be intentionally not enforced at all due to that for a while, through I think the goal is currently to amend it until then.
> that LLMs will be excluded from being used in high-risk AI use cases
no, it won't I can guarantee you this. At best they will get additional restrictions over time, as things go wrong. Anyone who could make this happen has way too much interest to not make it happen. (Most/All? EU country legal systems are overloaded to a point of not working correctly anymore, and have been before AI generated law suites and other AI nonsense started. I won't go into detail but many believe AI assistance (for certain tasks, always with a human doing any final decisions) is the only way to get out of this mess).
> standards are published yet
or exist,
like seriously this isn't a case of there being non public WIP standards which will pin all the nitty bitty details down, but cases of state agencies (and in last instance judges) having to decide if a specific standard (or implementation) is sufficient or not.
but also to some degree it shouldn't be tightly coupled to tech standards as there are often many ways to implement the things the law requires and accepting only one is undesirable (and likely wouldn't legally hold up). But having tech standards which are a "guaranteed to be enough if you comply with" (but not the only valid way) would have been preferable, bringing us to the next point
> have absolutely no idea how they will ensure compliance
nor do they know, the original non big corpo hijacked version had exceptions for most companies affected now. So it would only have affected a handful of huge companies, which have many of the things required already in place, in some form or another. Most likely this would have played out as this companies presenting how their measurements are "sufficient" and the agencies then evaluating it and potentially requiring some changes, going back and force over a longer duration leading to documented cases of rough technical standards about "what is sufficient" they then can pass to other organizations in the future. But now the law affects not just a handful of companies but like thousands, if not tens of thousands. Many not stuffed in a way where such a process could work, or even do the necessary documentation to show "compliance"...
So from a practicability POV, if enforced starting 2027, it currently excludes close to _any_ (meaningful) use of AI, down to a trivial linear regression or similar. Including any "old school ML/AI" any Bank uses for risk assessment.
Banking stopping running in December and there not being any (meaningfull) AI startups or adoption at all is not something anyone (in power in any state organ) wants to see, so guess how much it will be enforced ;)
And as mentioned the chance of AI as technology being excluded "in general" is close to none. Maybe specific usages could be excluded (and/or are already excluded) but thats it.
Oh and as a bonus a malicious reading of f+g remove any proper privacy protections for any AI usage in high risk context, where it is often most relevant... (a more sane reading allow it, with ... tricks).