current way the models works is that they don't have memory, it's included in training (or has to be provided as context).

So to keep up with times the models have to be constantly trained.

One thing though is that right now it's not just incremental training, the whole thing gets updated - multiple parameters and how the model is trained is different.

This might not be the case in the future where the training could become more efficient and switch to incremental updates where you don't have to re-feed all the training data but only the new things.

I am simplifying here for brevity, but I think the gist is still there.

Updating the internal knowledge is not the primary motivator here, as you can easily, and more reliably (less hallucination), get that information at inference stage (through web search tool).

They're training new models because the (software) technology keeps improving, (proprietary) data sets keep improving (through a lot of manual labelling but also synthetic data generation), and in general researchers have better understanding of what's important when it comes to LLMs.

Sure the training can be made efficient, but how much better can these LLMs get in functionality?