It’s more like a nightly training, isn’t it? IIUC the human brain learns from its experiences while it’s asleep, so it might be kind of like taking things out of context windows and fine tuning on them every night.

Correct and working on it. You can take the approach of mixed experts and train the network in chunks that share known interfaces over which they communicate results. These chunks can be trained on their own, but you cannot have a set training set here.

Then if you go further and alter the architecture by introducing clean category theory morphisms and build from there you can have a dynamic network - but you will still have to retrain this network every time you change the structure.

You can spin this further and know the need for a real-world training set and a loss function that will have to competete against other networks. In the end a human brain is already best at this and embodied in the real world.

What i want to add here is that our neurons not take in weights - they also fire depending on whether one input comes after another or before and differs down to the nanoseconds here - unmatched in IT and ofc heaps more efficient.

I still would say its possible though and currently work on 4D lifeforms built on dynamic compute graphs that can do this in a set virtual environment.

So this is pretty awesome stuff, but its a long fetch from anything we do right now.