Here are 18 pelicans - six each for Luna, Terra and Sol at the six different reasoning effort levels (plus the price to generate each one): https://static.simonwillison.net/static/2026/gpt-5.6-pelican...
Or if you want to see some in 3D, OpenAI featured a pelican riding a tricycle, bicycle, pony and another pelican in their livestream this morning: https://www.youtube.com/live/Wq45rvPGNHs?t=1070s
Time to dump this test. Probably not a coincidence every version has the same rolling green hills, gradient blue sky, sun in the corner, etc.
On the one hand: yes, pelicans on bikes are definitely in the training set at this point.
On the other hand: the test is clearly not saturated, given that you can see a clear difference in output at the various reasoning levels / model versions.
I sort of agree, but within the same model I expect the reasoning effort to be reflected in the quality of output and that's basically how it played out. When you're comparing different models, then it's just who benchmaxxed the best and there's not a lot of value there.
I partially agree, but in this case it kinda illustrates that it may not be worth using Terra on any reasoning level below high; those are some awful penguins on bikes.
I don't know. If they were training on this, I feel like they would be able to get the shape of a bike frame right; it's a pretty simple polygon, and a lot of the bike frames that are getting generated would be impossible to steer.
I mean, humans can't draw bikes!
https://themagnet.substack.com/p/why-is-it-so-hard-to-draw-a...
Goodhart's law.
I think the 'pelican test' is becoming useless. It's been around long enough that now I'm sure good examples are in the training data, and hell they might even do some hand tuning to make it do a decent job since they know people will ask about it.
But either way, with no real way to visualize the result of the text it starts with - it will always be stabbing in the dark. It can't understand conceptually what any of it should look like and then refine the SVG to improve it gradually. It just throws darts at a wall and hopes it comes out alright.
I think it's still useful in a "hello world" sort of way. It means you actually tried the new model.
Honestly that's the main value I get from it myself - making a pelican means I have to figure out API keys and how to talk to the provider, or how to run it locally for the local models.
Pelicans, maybe, but the point is to measure how good the "internal visualization" abilities are. Throw curveballs, like a unicorn with a duck bill serving coffee at a basketball court. An elephant playing a piano while its trunk swings a baseball bat at a tiny alien spaceship buzzing its head.
Have them use tikz instead of svg, or have it write code that moves the cursor and draws the thing in paint.
Compositionality and visualization are generally much, much better at each new generation / release cycle.
It's fascinating how well models have internalized visualizing things without actually having joint embeddings / broad multimodality.
What's strange with this is the prompt "Photorealistic photograph of a pelican riding a bicycle down a coastal boardwalk, wings gripping the handlebars, webbed feet on the pedals, large orange bill, detailed feather texture, golden hour lighting, shallow depth of field, shot on a DSLR with 85mm lens, natural motion blur on the wheels" produced, well, exactly what I asked it for. I wonder if I tell it then to make it SVG ...
https://chatgpt.com/share/6a5009de-fff8-83ea-98ff-0da17d1d04...
Cool. I still find these a useful visualization of some the qualities of llms. Even if they did train for [animal] on [vehicle] svg, it's still nice to see at a glance how the different models and reasoning levels perform. Lunar misses part of the frame, except on max reasoning. While most of the others have a mostly correct bike at all reasoning levels.
I once used something like karpathy's auto-scientist to mutate the prompts and rank them with a vison model. Some of the winners where pretty neat. I think they have a lot more style than the gpt-5.6 ones. https://xcancel.com/xundecidability/status/20449185674144196...
people are saying this is benchmark is saturated but all of these have occlusion issues, even sol max.
A skilled human artist wouldn't have both legs in front of the bike, or a single straight line representing both leg's crank arms.
Yeah it makes no sense at all to dismiss the test, when even the very best examples are noticeably below what a skilled teenager could produce.
Dead internet theory? Semi-random parroting by real people? Or something else.
Is the direction of the pelicans encoded in your prompt? Curious why they are all left to right with the exception of terra xhigh.
History moves left to right.
Cultural bias.
At what stages will models start to internally reflect the drawn SVG and automatically fix their own mistakes?
I assume multimodal models can do it already do it today if constantly asked "make it better"
I haven't tried this in a few months, but last time I tried a loop that rendered the pelican and asked for improvements the results were actually quite disappointing. Be interesting to try that again against GPT-5.6 at Claude Fable 5 though.
The quality of sol on effort=none makes me think this test is saturated or they are benchmarkmaxxing this exercise.
I'm waiting for the day that the "generate a Pelican" test comes back with a SVG-art like illustration of a Pelican equipment case, like a model 1620 or similar.
https://www.google.com/search?client=firefox-b-d&q=pelican+1...
They said in the AI community, a pelican riding a bicycle is a good test to measure effectiveness of the model, wondering if they were referring to you, or is it really a standard in the AI community ?
Also would be good to have a tool where users can select models and instantly see each model's generated pelicans. That will make it easy to compare the output of different models.
Simon did start the pelicans on bicycles as an SVG, but I think it's more of a fun goofy thing to see how the model performs at. I don't think it has a direct correlation to a model's performance though.
Ok, I'll never use max effort again on OAI models..
Is that... an x-rated, censored pelican?
Surely "how to draw a SVG pelican on a bike" has made it into the training data by now ...
If that was the case in a non-trivial way you'd see mode collapse, but you don't, they come out differently.
It's because all the frequent comment that this pelican is in the trainings set now also got into the trainings set and models adapt. /joking (I hope)
Surely this comment is literally on every new model release post.
It's part of the pelican tradition at this point.
I eagerly await the models replying with that: "I'd be happy to create a pelican riding a bicycle, but just a note that this might already be in my training data. Simon."
Someone told me that a model said "oh, the classic" when they asked for this recently. I think that was Opus 4.8.
And yet... only one is any good.
something is wrong with Terra model series, most pelicans, except Max, looks bad
Seems to match the pareto frontier on Artificial Analysis as well. Terra is nowhere on it.
Thank you Simon! Luna is surprisingly decent across all reasoning levels.
I think all of Luna's are bad. The only decent one is sol @ xhigh. Even sol @ max is weird. Sol @ high and @ medium are ok, and every other single one across every model is bad.
Strong disagree, but to each their own. For sol I really like how only medium uses the wings on the handlebars to ride the bike. For all the other sols the pelican evolved a new set of arms separate from the wings.
somehow Terra really struggles here even compare to Luna.
Apparently plus users do not have access to Sol, so I'm really worried about the ugly Terra Pelican.
max effort sol clearly over-engineered
gpt-5.6-sol Max pelican didn’t skip neck day
gpt-5.6-sol x XHIGH is my favourite
AI really sucks at bicycles...
LLMs really suck*
I like Terra High the best. That pelican is utterly yoked.
[dead]
[flagged]
You can't post like this to HN, regardless of how you feel about someone else's posts.
Doing it repeatedly crosses into harassment, and you've done it more than 3 times now - e.g.
https://news.ycombinator.com/item?id=48504823
https://news.ycombinator.com/item?id=48504654
So please especially stop doing that. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
Those 2 comments are from a month ago, on the same day, and my point was he was spamming those threads with basically the same comment, and he links often to his site and his posts are all about his software. Both of my replies you link have positive counts (with sure had plenty of downvotes as he had positive upvotes), and even now after your link probably added more downvotes to mine. And other commenters in those threads agree with me.
HN search shows 21 comments in the past month from simonw+pelican: https://hn.algolia.com/?dateRange=pastMonth&page=0&prefix=tr...
You can choose to look the other way but 3 comments out of 21, where 2 were kind of the same one on the same day, and 27 days ago... But you say "repeatedly crosses into harassment" as if I'm a troll chasing him around.
I was caring about the quality of this site with my 18 year old account... Unbelievable.
But as you bless him, I give up. It's your site. Quite a bummer after I defended you many times over the years, though. I remember in the beginning your interventions were not generally welcome.
Edit:
His comment is now 12th in this thread and there are comments saying his pelican SVG test is pointless. At least there's some sanity with other users. Or are they harassing him, too?
Sadly his other comment on Hy3 thread is still the top comment. And of course it also has a link to his site. https://news.ycombinator.com/item?id=48848950
this looks like the same shit from 4 years ago. give it up.
The first time I did this was actually less than two years ago - in October 2024 - and it's fun seeing how much better they've got since then: https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
Absolutely, the change in quality over time is a great yardstick.
Nice to see you did the quality level comparisons and did three passes.
I've been using that technique myself on my image gen reviews[1] and it also works well in presentations and for personal study.
[1] https://generative-ai.review/2025/12/beast-mode-activated-op...