I don't think it's a bad way to benchmark new models, I just find it concerning that the author implies that "pelican on a bicycle" has been exhausted. At the risk of making overly broad, unfalsifiable claims I think multi-year exposure to AI content has dramatically raised our expectations for speed and volume but lowered them for quality. We see a very janky pelican and declare the problem solved.
I don't think he's claiming it's been exhausted. It's just that things have progressed to a point where people are arguing over the finer points of which pelican looks better -- which is often a matter of taste, and an indication that we've hit the knee in benchmark where models are no longer failing in obviously awful ways.
I haven't really seen evidence that any ai can reliably draw a pelican riding a bicycle. Not if you look at the image long enough to take it in. Even the best ones have something wrong with them. Not a matter of taste but a matter of having both legs peddling on the viewer's side of the bicycle or having two beaks.
I'm actually beginning to wonder if some people who ignore these things have a different, somewhat lesser ability to percieve image details than I do.
I mean I guess its fine to go on to another test despite never actually passing the pelican bike test, but there's a sense that we have to use another test because AI is now good at pelicans on bikes, which is just not true.
AI has deeply changed the way I think, feel and act around a computer. In the same way that dialing into the internet changed things for me. Since using ChatGPT the first time until now I have never cared once to look at these pelicans on bikes people seem to get hung up about. It could never have been a thing and nothing would change. See the forest through the trees.
What you’re saying is that you’re not interested in benchmarks. But then you go beyond that, and say this particular benchmark test is inconsequential. It’s like saying that if you didn’t exist, nothing would change. And yet, that would be kind of insensitive and a rude thing to say, wouldn’t it?
> I haven't really seen evidence that any ai can reliably draw a pelican riding a bicycle
Please remember, we've started from there :
https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/
When it started, it was clear what LLM would stand out, its style, etc. Nowadays, the pelicans look similar, the difference is in details and sometimes hard to catch. Sure, the task is not completed perfectly, but that's not the point. It was supposed to be a benchmark to quickly benchmark a LLM against others.
When is it ever hard to catch?
Sure, the task is not completed perfectly, but that's not the point.
Isn't it?
If the computer can't do it better than a human being, then what's the point?
Being wrong at scale is not better than being right.
Many humans would struggle with this even with very good tooling (ie not writing raw svg and using illustrator). I struggle to draw a bicycle accurately. But yes, I suspect it will be diminishing returns and I doubt it will ever be perfect due to the average nature of AI but I’d like to be wrong.
>If the computer can't do it better than a human being, then what's the point?
Because the benchmark wasn't testing "can an LLM draw a pelican like a human". The original article was testing the relative capabilities between LLMs. Now that LLMs can all draw pelicans all similarly, the test is less interesting as a comparative benchmark.
Trillions of dollars spent. Trillions of gigawatts consumed. And people still celebrate "Yay! We're less wrong than the other guys!"
This is what the tech industry has become?
Less of a failure is still failure.
I think it's touching the limit of what one can reasonably expect any intelligent thing to produce with the only direction being "produce an svg of a pelican riding a bicycle".
When you aren't sure if an LLM can write an svg well, or that it will be able to form a pelican shape, or animate a bicycle, it's a good test. After that, it's all judgement: how detailed should the pelican be? pelicans are the wrong shape for a bicycle by default, so how much can I change its physiology to match using a bicycle before it isn't a pelican? Do I care about how well the client is able to render complex geometry?
It's not that there isn't room to do better, or that it doesn't tell you anything at all, but rather we've reached a point where what it tells us isn't very clear anymore.
> I think it's touching the limit of what one can reasonably expect any intelligent thing to produce with the only direction being "produce an svg of a pelican riding a bicycle".
Oh come now. I am extremely confident that if I hired a professional artist to draw a picture of a pelican riding a bicycle, I would get something inarguably much better than what today's best coding LLMs can produce.
At that rate you could use an image model, which was designed for the task. Thats the absurdity of this test. It’s often a text only generation model that has never seen a pelican, coerced into creating an xml graphical representation that a human might recognize. That it does anything passable is already astounding.
I agree, it's hard! That's why it's still a good benchmark.
It probably has a few million inputs on how a pelican and bicycle looks, not to mention the amount of data on how to create SVG’s. Ask it to create a relaxing spa website and it will, even though it has never seen a spa.
Could you share any examples that come close to those limits? I haven’t seen any that don’t have obvious flaws in proportions, layering, composition, color palette, visual clutter, or stylistic consistency.
> touching the limit of what one can reasonably expect
If the expectation is that AI is going to replace "knowledge workers" then the limit would be a darn perfect drawing. We are nowhere close to that.
And Elon is already propagating the age of abundance where money won't exist anymore, right before calling the interviewing journalist dishonest and deservedly losing public trust. Smh my head.
Also, one of the advantages of the pelican test is that you can evaluate it all at once. There’s no reason the two-dimensional depiction can’t be made more challenging. Yes, at some point the pelicans might approach the subjectivity of a fine art painting, but we haven’t even seen a depiction that’s competent by the standards of a high school art class. That’s not to say the elementary school–level SVGs aren’t amazing - rather, I agree with your point.
Agreed. It's far from solved. Modern LLMs still generate pelican bike SVGs with obvious errors:
* some omitted the bottom of the diamond which connects from the pedals to the rear wheel
* some added an extra connection from the pedals to the front wheel, making it impossible to steer
* none could align the head tube with the fork
* none added a correct offset to the fork
* none could generate the chain properly in a way that attaches to the two sprockets correctly
I mean just look at these:
* Grok 4.5: https://s3.eu-west-1.amazonaws.com/images.dylancastillo.co/p...
* GPT 5.6 Terra: https://s3.eu-west-1.amazonaws.com/images.dylancastillo.co/p...
* Sonnet 5: https://s3.eu-west-1.amazonaws.com/images.dylancastillo.co/p...
from https://dylancastillo.co/posts/pelicanmaxxing.html
Can someone explain what the pelican on a bicycle tests exactly? And why is it so important? I've never understood how it could translate to a useful task in real life.
On the most basic level, its asking the ai to generate valid svg code for a picture of a pelican riding a bicycle, as a way of checking its intelligence. Popularized (invented?) by simonw, its been used as part of his reviews of new models as they come out since oct 2025.
It used to be a very difficult task for models, see [2,3,4]
it cuts across several tasks that AI used to be very bad at, but now has improved quite a bit. Namely, spatial reasoning (because it has to manually place the points of the svg such that they make sense and form what it says it forms. This used to not work very well, with random shapes floating around that it would mark things like "eyebrows" but were nowhere near the "eyes", etc.
It also tests the model's world knowledge (what do pelicans look like? sure they have wings, feet, beaks etc, but what shape are they? how to get proportions roughly right? this isn't a given from text data about the bird. This goes doubly for a bike, which is a quite complex shape that most humans fail to draw correctly[1] (many draw the frame or chain connecting in impossible ways that would not ever function mechanically)
Before it was pelican on a bicycle there were people having it do horses/unicorns making the rounds - gpt4.0 or whatever would often make hideous abominations of legs and mouths
[1] https://www.gianlucagimini.it/portfolio-item/velocipedia/
[2] https://static.simonwillison.net/static/2026/mistral-small-4...
[3] https://static.simonwillison.net/static/2025/codex-hacking-m...
[4] https://static.simonwillison.net/static/2025/gemini-2.5-flas...
If your school mascot is a pelican and you need to make a flyer for the upcoming bike safety event, then this precise thing becomes useful. Most things that are useful in the real world seem not useful out of context.
Sure, but then you would just use an image generation or multimodal model to generate that image. I don't think you'd want a weird looking svg.
Exactly. I would want a good looking svg. That's the evaluation.
Yeah, pelican on a bicycle tells you how well the model can extrapolate outside of existing data, rather than just interpolate between it.
100%. Why waste the tokens to render Lord of the Rings when the pelican test still clearly benchmarks so well.
Humans are drawing pelicans riding bicycles now. Just google it and you will find 5 or 10 of them in the first few results. Including a t-shirt design.
So it's a pretty much pointless test now.
Yet no LLM can actually do it.
It's quite surprising actually.