I would guess only on the nitpicked benchmarks?
The pelicans place it nicely between Sol and Luna in terms of both pricing and results.
I would guess only on the nitpicked benchmarks?
The pelicans place it nicely between Sol and Luna in terms of both pricing and results.
pelican is also nitpicked (and probably the most useless)
Cherry-picked is the term, not nitpicked.
Yes, it may be dumb, but it isn't useless because it is both a) is a benchmark and b) is not picked by openai
IMO it's useless because we don't even know what a pelican on a bike is supposed to look like, It's very impressive that that models can make these directly using SVG, but we are way past the "wow" stage and test that can't be gamed and are easy to prove how good the model is in that specific arena are hard to find.
The only thing that pelican have is "models where not trained on this" maybe that true months (or years ago) now it's on every HN post and these AI labs are monitoring HN