And yet, we're clearly way into the period of diminishing returns.

I'm sure there's still some improvements that can be made to the current LLMs, but most of those improvements are not in making the models actually better at getting the things they generate right.

If we want more significant improvements in what generative AI can do, we're going to need new breakthroughs in theory or technique, and that's not going to come by simply iterating on the transformers paper or throwing more compute at it. Breakthroughs, almost by definition, aren't predictable, either in when or whether they will come.