Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure
Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure
It doesn't show any indications of solving catastrophic forgetting.
In fact it says the opposite - that there is pruning.
Our brain also has some capacity limit, and maybe degraded memory performance over time, but in either case it's a graceful degradation - you may forget fine details of things that happened a long time ago etc, but you don't forget how to ride a bike just because it's been a while.
Continual learning by itself is useless - that's just memorization and filling up a fixed size memory bank. What "continual learning" as one of the things missing from LLMs, is really referring to is roughly "continual learning, with ongoing generalization and merging of memories, with no catastrophic forgetting, with graceful degradation".
>roughly "continual learning, with ongoing generalization and merging of memories, with no catastrophic forgetting, with graceful degradation".
Sounds a bit like real-time "distillation" to me.
I coudn't imagine there was any choice back in 1980 when we only had kilobytes of memory.