Wow. If those charts are true, this effectively matches GPT-6-Astra at a fraction of the cost, and is neck-and-neck with 6.1-Sol.
Wow. If those charts are true, this effectively matches GPT-6-Astra at a fraction of the cost, and is neck-and-neck with 6.1-Sol.
Looks great on the first chart, but terrible on the Terminal Bench 4.0 where it needs xhigh to come close to 6.1 Sol, and costs more for a slightly worse result.
I would guess only on the nitpicked benchmarks?
The pelicans place it nicely between Sol and Luna in terms of both pricing and results.
pelican is also nitpicked (and probably the most useless)
Cherry-picked is the term, not nitpicked.
Yes, it may be dumb, but it isn't useless because it is both a) is a benchmark and b) is not picked by openai
IMO it's useless because we don't even know what a pelican on a bike is supposed to look like, It's very impressive that that models can make these directly using SVG, but we are way past the "wow" stage and test that can't be gamed and are easy to prove how good the model is in that specific arena are hard to find.
The only thing that pelican have is "models where not trained on this" maybe that true months (or years ago) now it's on every HN post and these AI labs are monitoring HN
Not necessarily, they may have selected benchmarks where Terra performs relatively well compared to those models. Pretty much every model release these days just cherry-picks & highlights the benchmarks where their latest model performs best. We'll have to wait for the complete benchmark suite to have a better picture.