I wonder how (and if) continuous learning models will achieve stability.
They are unpredictable enough without learning, this is cool but I wonder how useful it will be in the long run
I wonder how (and if) continuous learning models will achieve stability.
They are unpredictable enough without learning, this is cool but I wonder how useful it will be in the long run
There are two questions about that stability I have.
One, things like catastrophic forgetting and falling into incoherence.
Two, less likely but far more worrying, falling into unwanted attractor states. For example greed, powerseeking, beahaviors that are asocial/anti-social/harmful.
Aren't such attractors also problems during training? Presumably alignment constraints would need to apply to continuous learning as well.
I mean yes, but that's far more difficult than one would expect in a layered system. You may be able to keep a concept aligned, but can you keep the meta concept aligned? Continuous learning means there is continous opportunity for a more powerful system of misalignment to form itself and take control of your alignment. Even worse is hidden layers of this misalignment that can hide itself by not using an interpretable language.
I do it all the time. Am I stable? Depends who you talk too.
It will eventually be super useful, and so disruptive that it will make today's LLMs look like nothing particularly special IMHO.
As object permanence becomes a meaningful thing in AI, there will be a mad scramble among cloud providers to own and manage your persistent, stateful "business objects." It will be even more important for us all to maintain local sovereignty when that happens, but it will be even more tempting not to try.
Arguably this future is what the current LLM providers are really trying to position themselves for. Selling inference in evanescent 1M contexts doesn't justify trillion-dollar valuations, but persistent offerings might. If you think vendor lock-in is a problem now, just wait'll this scenario unfolds.
Serving requests where every user has their own set of self-updating weights will absolutely murder whatever minimal margin the AI providers have today
Who says the weights have to be duplicated in their entirety? Even that will likely be worth it.
Imagine an OpenAI owning the ERP and CRM databases and workflows of a big chunk of the Fortune 500. Their typical customer's employee headcount might be 10% of what it once was, and OpenAI might capture 25% of the resulting savings. The contracts are signed in the same hemoglobin-based ink that Larry Ellison uses.