Usually what I need a LLM to do is find me a elegant agorithm for a problem I've encountered where I know there's an elegant algorithm but I've got no idea what it's called or how to google search for it.
Usually what I need a LLM to do is find me a elegant agorithm for a problem I've encountered where I know there's an elegant algorithm but I've got no idea what it's called or how to google search for it.
It makes sense but if the goal is to replace software engineers as claimed, then these benchmarks aren't going to achieve that.
Companies are still stuck in this mindset conflating software engineering with puzzle-solving. This is evident from their job interviews and also these LLM benchmarks.