IMHO training weights has peaked and now it is time for a training paradigm for prompts and code. We don't have the gradient descent here - but I think it can be more sample efficient because causal theories can be better than just correlations.
I am working on a unified theory in https://zby.github.io/commonplace/ - it is all agent edited so it might be hard to read, but hopefully we are catching most logical errors. Some day the llm prose will improve.
I have even a preliminary theory on what is needed for the positive feedback loop: https://zby.github.io/commonplace/articles/reflective-self-i... - (this is not stable yet - but I think you can give it to your agent to read :).
Is there any reason to think that training weights has peaked rather than is accelerating? It feels like now they are increasingly able to pick some low hanging fruit by using the models in order to improve themselves and test optimizations.
to be frank - mostly because they are now good enough to unlock the other learning paradigms
What date was the peak? If it is today it’s not something you can know so I assume you think the peak was many months ago.
I think it is now or close to now - and not because there is no more to be gained - but rather because it is now clear that you can gain more with the right agent environment.
But benchmark saturation is also something to account for.