It makes sense but if the goal is to replace software engineers as claimed, then these benchmarks aren't going to achieve that.

Companies are still stuck in this mindset conflating software engineering with puzzle-solving. This is evident from their job interviews and also these LLM benchmarks.