It seems like the design is flawed, probably because the scoring structure and conditions are wrong. And originally, due to the nature of LLMs, even if the input is unstructured, when you design something like a RAG system, you usually need to create a verifiable evidence table. Even with that, the scores are still probabilistic by nature, but at least they stay within an error distribution that I can verify. But it doesn't seem like there's any such evaluation criteria here.

Typically, retrieval should be tied to evaluation metrics, evidence should be linked to scores, and you also need to account for parsing errors.

But personally, I'm weak to these kinds of ATS systems (ugly appearance, non-native English speaker, didn't go to a good university), so if this kind of filtering existed, I probably would have never had a job in my entire life. Come to think of it, even now I don't have a proper job—I just bid on projects at the lowest price and implement them. So maybe it doesn't really matter whether such a system exists or not