Simon, at this point I really wonder if teams aren’t gaming this. You should pick a random animal doing a random thing every time.
Simon, at this point I really wonder if teams aren’t gaming this. You should pick a random animal doing a random thing every time.
They're still not yet at the point where pelicanmaxxing is the best way to win this benchmark. Earlier models sucked because their SVG skills sucked. Newer models are likely better because more/better SVG models are being added to their training data.
We should just consider the pelican bench as saturated and mostly meaningless.
But the general improvements are obvious. Get them to draw something very different (e.g. a wifi rotary phone with a peeled banana handset and a coiled cable, or a pink tennis ball with strawberry seeds and a reset button) and you can see that improvements are not narrowly tailored.