A better way to reformulate this problem is for the LLM to be tasked with making a _comparative_ judgement between two CVs. This should prove much more reliable, especially if you give it a third “too close to call” option. You can also ask for clear justifications of preference.
That's a good idea.
The only drawback I see is that you should compare every pair of CVs for best results, and that grows quadraticly with number of CVs. Of course you can settle for fewer comparisons and not perfect results. But then I'm not sure if you can hit a good ratio of quality and token spend.
Could probably do an elo system and sample pairs. E.g.
1. Set the elo of all CVs to 1000 elo
2. Randomly pair up CVs and compare. Winners gain elo, losers lose elo.
3. Repeat #2 for a few iterations, then remove bottom X% of CVs.
4. Repeat 2-3 until the amount of remaining CVs is small enough to do an exhaustive comparison.
I don't have a mathematical proof, but I suspect that this is a decent cost-effective approximation of comparing every pair (depending on the parameters)
> you should compare every pair of CVs for best results
Or compare each one to a reference set? Take 5 resumes of existing employees, rank all candidates against that set, maybe you get some useful level prediction into the bargain
I'd just do a quick filter, probably deterministic, then perform a deeper comparison on the selected few.