For coding, I like the Aider polyglot benchmark, since it covers multiple programming languages.
Gemini 2.5 Pro got 72.9%
o3 high gets 81.3%, o4-mini high gets 68.9%
For coding, I like the Aider polyglot benchmark, since it covers multiple programming languages.
Gemini 2.5 Pro got 72.9%
o3 high gets 81.3%, o4-mini high gets 68.9%
Isn't it easy to train on the specific Exercism exercises that this benchmark uses?
where do you find those o3 high numbers? https://aider.chat/docs/leaderboards/ currently has gemini 2.5 pro as the leader at, as you say, 72.9%.
It's in the OpenAI article post (OP) i.e. OpenAI ran Aider themselves.
Update: the leaderboard has o3 high + 4o tops of the charts now with 82.7%. This is a) amazing b) 20x more expensive than Gemini.
It was a good benchmark until it entered the training set.
thanks