So most training data would be grey and a little bit coloured? Ok, that sounds plausible. But then maybe they tried and the current models get it already right 99.99% of the time, so observing any improvement is very hard.
They have a lot of data in the form: user input, LLM output.
Then the model learns what the previous LLM models produced, with all their flaws. The core LLM premise is that it learns from all available human text.
This hasn't been the full story for years now. All SOTA models are strongly post-trained with reinforcement learning to improve performance on specific problems and interaction patterns.
The vast majority of this training data is generated synthetically.
I think OpenAI and Anthropic probably have a lot of that lying around by now.
So most training data would be grey and a little bit coloured? Ok, that sounds plausible. But then maybe they tried and the current models get it already right 99.99% of the time, so observing any improvement is very hard.
They have a lot of data in the form: user input, LLM output. Then the model learns what the previous LLM models produced, with all their flaws. The core LLM premise is that it learns from all available human text.
This hasn't been the full story for years now. All SOTA models are strongly post-trained with reinforcement learning to improve performance on specific problems and interaction patterns.
The vast majority of this training data is generated synthetically.