the nature of the test was to see if the models can effectively compose an image of a novel concept outside the training set. If they are trained on it, it ceases to be an interesting test to some extent.
I would urge you to re-read the blog post you are commenting on. It pretty clearly explains how it is an interesting test independently of "see[ing] if the models can effectively compose an image of a novel concept outside the training set".
it's still interesting because there's no pelican-on-bike model, and if you're training a model well enough, then it should be obvious when a model has reached "AGI" or whatever.
Yes and that would improve its ability to draw SVGs of pelicans on bikes, no?
Would it? Tongue in cheek.
and that is bad because ?
the nature of the test was to see if the models can effectively compose an image of a novel concept outside the training set. If they are trained on it, it ceases to be an interesting test to some extent.
I would urge you to re-read the blog post you are commenting on. It pretty clearly explains how it is an interesting test independently of "see[ing] if the models can effectively compose an image of a novel concept outside the training set".
it's still interesting because there's no pelican-on-bike model, and if you're training a model well enough, then it should be obvious when a model has reached "AGI" or whatever.