If a new physics break through gets published today, no existing model will be able to fully integrate it - beyond a context window. If I put the paper in my session and it isnt in yours the model knows nothing. It wont retain it past that session.
Models are trained, they do not learn.
I think GP is using a different level of abstraction from yours in their metaphor.
You are saying:
Pre-Training == Everything you store in your memory throughout your life. Model weights == The lessons you learned
Context == whatever you're currently thinking about
One inference run == one thought
They are saying:
Pre-Training == building the DNA template of human brain through millions of years evolution. Model weights == Human DNA
Context == Everything you store in your memory throughout your life, plus whatever you're currently thinking about
One inference run == One human life. One instance == one human
Applying their metaphor, your sentence becomes:
> If a new physics breakthrough gets published today, no existing DNA structure will be able to fully integrate it - beyond an individual person. If I put the paper in my mind by learning it, and it isn't in yours, the DNA of human species stores nothing. It won't retain it past my lifetime.
> The human species is trained (through evolution), it doesn't learn.