I’m not sure what the “biggest” problem is, but I do think diversity is vastly underappreciated compared to relevance.
You can have maximally relevant search results that are horrible. Because most users (and LLMs) want to understand the range of options, not just one type of relevant option.
Search for “shoes” and only see athletic shoes is a bad experience. You’ll sell more shoes, and keep the user engaged, if you show a diverse range of shoes.
I liked how Karpathy explained part of this problem as "silent collapse" in his recent Dwarkesh podcast. Meaning the models tend to fall into a local minima situation of using a few output wording templates for a large number of similar questions, and this lack of entropy diversity it becomes a tough hard to detect problem when doing distillation or synthetic data generation in general. These algorithms as nice python functions are also useful repurposed for labeling parts of ontology and topic clusters etc [1]. Will definitely star and keep an eye on the repo !
[1] https://jina.ai/news/submodular-optimization-for-text-select...
Nice, I actually read that Jina article when it was published, but forgot they use facility location as well! The saturated coverage algorithm looks pretty interesting, I'll have a look at how feasible it would be to add that to Pyversity.