I generated pelicans riding bicycles on both thinking level low and thinking level high:
https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304...
The high one is notably better - the bicycle frame is the correct shape, unlike thinking level low.
For comparison, here's Opus 4.7: https://gist.github.com/simonw/afcb19addf3f38eb1996e1ebe749c...
I don't see how a frame without a headtube can be "the correct shape".
> the bicycle frame is the correct shape
No, the handlebar is wrong. The handle bar is rotating the frame instead of rotating the front wheel. The handle bar should be mounted on the same line as the front wheel is.
Hopefully 4.9 will read my comments :)
Maybe the pelican is just riding a road bike/gravel bike
Could be an extremely high angle stem that just happens to match the downtube angle.
I bet someone shares this link every time you post about bicycles, but since I didn't see anyone share it yet in this thread, I'll take the opportunity to do so:
https://www.gianlucagimini.it/portfolio-item/velocipedia/
Turns out even humans can be pretty bad at drawing bicycles :)
> The most unintelligible drawing has also the most unintelligible handwriting. It was made by a doctor.
Haha
On a new model release, you can guarantee two things are in the replies to Simon. One is your link, the other is "surely the models are being trained on this now"
But if you need to draw a bicycle, you wouldn’t pick a random person in the street. You would hire an artist and you’d be guaranteed to have at least a believable one if not a perfect rendering.
No guarantees is why LLM is akin to gambling. Every new context is essentially picking someone out of the crowd.
Glad to see that the "high thinking" level adds a helmet. Always a smart choice.
Here's pelicans in all of the thinking levels - low, medium, high, xhigh, max
https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
I like the way the max pelican has a stern look on his face
Is the output on the max level meant to be missing?
I just fixed that (force refresh). It hit my default 8,000 output token limit, it worked when I bumped that up.
For max I used 25 input, 17,167 output which cost me 43 cents! https://www.llm-prices.com/#it=25&ot=17167&ic=5&oc=25&sel=cl...
You've peed in the pool Simon, this has to be a part of the internal evals by now! You got to try something new - maybe a panda in a canoe?
If these were in the internal evals then the output would be much better. The 4.8 pelicans are pretty meh
Click the link
I really like that thinking level high gave the pelican a helmet.
I actually like the 4.7 the most, interestingly enough. Not like you can "objectively" weight artistic output like this.
Hey simonw I love your test, do you think using thinking level "max" makes sense for this test? I would love to see the results about it.
I don't think the API supports "max" as an option, that might just be a Claude Code harness thing.
UPDATE: My mistake, the API does support max. I added a max one at the bottom of this page (cost 43 cents): https://tools.simonwillison.net/markdown-svg-renderer#url=ht...
I find the most miraculous thing about 4.7 to be that the pelican is facing left, wonder why the right facing everything is so ubiquitous in these images.
This happened to me in elementary school. We were doing fingerpaintings using plasticine. After all the bikes were hung on the wall, mine was racing the other way... Somehow it really stuck with me.
What do you think it means?
It's facing left but looking right...
Profound political commentary?
[dead]
Simon, is your pelican test really captures differences among models or should you at least try like 10 times or something to average the random effects
I've been meaning to do a "run 3 times and pick the best" version for quite a while, I should really pull the trigger on that one. Currently it's one-shot only.
Best-of-3 would be cheating, ruin the test, middle of 3 makes more sense
Why would you need the 3rd run if you pick the "one in the middle"?
Middle as in not the best, and not the worst. As opposed to the second generated in sequence.
But not the best/not the worst is somewhat subjective.. so not sure how well that would work.
The vast majority (if not all) of these make it impossible to turn, among other fun things. Only out of curiosity, have you tried prompting further with how a bike must operate to see if it does the right thing?
thanks for always providing this very much on time. I'm wondering what the next, harder challenge could be? Maybe some animated svg?
For comparison, what's GPT-5.5 producing today?
The reasoning xhigh one is pretty solid: https://simonwillison.net/2026/Apr/23/gpt-5-5/#and-some-peli...
That little red hat on hard mode is sending me. 4.8 has whimsy
Is the "opossum riding an e-scooter" benchmark in the works for Opus 4.8? ;)
Good call, it's cute: https://gist.github.com/simonw/68560eddb0b268a8417f80ceb7304... - but nothing like GLM-5.1: shttps://static.simonwillison.net/static/2026/glm-possum-esco...
[flagged]
Eventually the frontier model folks are going to pick up on your pelican on a bike test and bake-in flawless results for that particular request.
Am I allowed to say that pelican's little helmet is adorable? I can't provide a strong computational proof, or even a shred of anecdata...
...but that pelican's little helmet is adorable.
4.7 reigns supreme IMO.