There have been papers about model collapse, but the underlying assumption is that you constantly train on only the outputs of the previous model. Later research has shown that as long as you retain some "real" data, training on largely synthetic data is ok.

And in the case the previous poster describes, the other model doesn't generate datasets, it generates environments which the next generation interact with to learn from.

> I've read multiple times that this approach is harmful in training.

There's a lot of nuance here. Note that I said "prepare" datasets and not just "generate" datasets.

First, the "model collapse" paper(s) were highly misunderstood and the "media" / content creators ran with it because negativity sells. In that initial paper the authors took things to the extreme, and presented as a given what happens in the literal worse case scenario. They used small models, they generated data w/ those models and indiscriminately trained on that data. It obviously led to model collapse. But that's not what you do in the real world.

The way you do this in the real world is different. For pre-training data you can do things to improve the quality of your inputs:

First, you can use the models to curate your datasets. And this is something that everyone has done since the days of "chug common crawl into the model and see what comes out". It turns out that quality of the data is very important and common crawl is really bad. So we've seen attempts at curating that data. The better the filtering models, the better the initial pre-training data.

Then you can have data augmentation, where you take some piece of content, and generate augmentations for it. Current models are good enough that you can take a piece of "authoritative" text (say a book on writing style) and a bunch of articles, and "improve" them. Or take a piece of content and "translate" it into simple / advanced explanations. Or take a piece of code and "explain" what it does, based on a paragraph from an authoritative book. And so on.

Then, for the mid-training / post-training with RL:

You need to find both good scenarios (i.e. problems) for your model to solve and a good verification schema. Like they say in the quote above, those problems need to be complicated enough for each new model. Here you can again use old models to prepare datasets for the new models.

One simple approach is to take a codebase, have your current model identify a set of features. Then instruct the model to remove code relating to feature "a" but keep its tests. Then verify that every other feature works in the code, bar the one you removed. Then, during RL, you train your new model on that task (you present it as a "prompt" / "situation") and you score the model based on the new feature passing the original tests.

Then there are more advanced ways of using prev gen models for "open ended" problems. You can't really apply RL if the task is not easily verifiable (like above, with tests). But you can use something like RLAIF (reinforcement learning w/ AI feedback) where you grade responses with the previous gen models. Now, in general this is lower quality / lower signal than RLVR (verifiable rewards) but you can still do smart things. Instead of rating an answer good / bad, or ask it one-shot what answer is better, you can use a method based on rubrics. You can first ask the preparing model to select tasks, and a list of rubrics on how that task should be scored (like they generally do on open ended exam questions). Then while doing RL you grade each response by asking the prev gen model to generate said rubrics. Does the answer touch on subject a / b / c? Does the answer mention x y z? Is this mathematically sound? And so on. You still get better results than nothing, even if the task is "open ended". And, again, as models improve so does your pipeline.

There was one highly discussed paper. And about a month after publication much much stronger models were released. The models got better because they used such synthetic data.