> the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model
You would not expect that to happen if the models trained on the unrecognizable mess, right?
> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts
And the labs clearly did focus on improving image rendering.
> they have a uniform style
SVG output from LLMs always looks like that. It looked that way from the beginning; no LLM ever produced a watercolor when asked for SVG output. They all render the prompted element centered in the picture. They all tend to draw things going from left to right, and so on.
I’m not suggesting Simon’s pelicans in the dataset are having a meaningful impact. I’m expecting that a company like ScaleAI has a product along the lines of “benchmax dataset: SimonW’s Pelican on Bikes test” which is a private curated series of well-drawn SVGs of animals riding vehicles for training and RL.
If such a product existed I'm reasonably confident someone would have tipped me off by now, NDAs be damned.