What proof of a ceiling are you talking about? Wouldn’t proving this require a good definition for intelligence, which I don’t think there is consensus on?
What proof of a ceiling are you talking about? Wouldn’t proving this require a good definition for intelligence, which I don’t think there is consensus on?
- https://en.wikipedia.org/wiki/Model_collapse
- https://en.wikipedia.org/wiki/Catastrophic_interference
- https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)
- https://en.wikipedia.org/wiki/Entropy_(information_theory)
As for intelligence, the only way we have that is by allowing the model to fill the blanks which have to come from the training data. The models cannot have true intelligence for as long as they are linear models, what we see with reasoning is "boxed" intelligence where the models are effectively "modifying" themselves by feeding it's own reasoning data back into input deriving most plasible output given known information. However, the model is not able to retain what it has learned therefore that intelligence is gone the moment the session is 'full'. You can go pretty far by continiously distilling discovered information, but again all that has to come from the original training data and models own outputs, which it has to take for granted as the 'intelligence' gained is lost creating what we see is the maximum possible benchmark performance and why smaller models are not able to score as high while theoretically having the same capabilities. We can see this with larger models where they can solve tasks much faster than smaller ones as it does not require to generate the solution due to the fact that the solution is already in the training data as 'baked' intelligence and it doesn't have to 'create' it during reasoning.
So, when you said “proven” I thought you might have meant like, a mathematical proof. It appears that this is not what you meant. (Right?) (If it is what you meant, I was asking for like, the particular theorem.)
Also, ok, when I said “intelligence”, it was because you were already talking about how “smart” the model could be. So, I thought you were already on board with using the word “intelligence” to refer to the phenomenon where these kinds of models produce outputs that satisfy the kinds of tasks they are pointed at.
None of those links give an argument that the current 1GB models are the best they can be.
My understanding is that so far when training a model by distillation (using the logits of the teacher model), one can achieve better outcomes than one could if training the 1GB model from scratch on the same training data as the large model, and that so far, better models as the teacher model have yielded better results for the student model.