Agreed. This isn't being oversold.
And actually, the fact that the brain is actually two "developmental" systems combined into one seems like big news to me. We might not know its other uses, but probably for cognitive science and even AI research this might be super useful. In AI, maybe two models should be developed and trained in conjunction that talk to each other, each with different functions and roles, because brains work better this way.
And from what I read, evolution did this type of combined evolution somewhat often. The parts of the ear evolved from different "Gill slits" from when we were fish, meaning the ear isn't evolved from one group of cells in the embryo, but from multiple groups of cells that evolved into one organ system, the ear. Yeah, the middle ear evolved from a different "gill slit" from the outer and inner ear.
"The brain does it" is not in itself a reason to do the same in AI.
Sometimes, what the brain does is genuinely a good solution to a given task - one that's good regardless of whether your neural network is of artificial or biological variety. But sometimes, what the brain does is an evolutionary kludge, or a hack that works around one of the "being made of flesh" issues - of which there are a great many.
We have known examples of both - and a few known features that might go either way.
The brain does have some known useful features that we are yet to plunder - usually because we know they're there somewhere but not how they work. We don't know how the brain stabilizes online learning, for example. Or what low k-complexity priors and data augmentation processes does it use to enable its sample efficiency.
But a two-system split? Useless by itself. Splitting a network in two is easy - but if we don't know what that split does, what it buys us, what useful bias does it impart? We're just adding complexity. See: the investigation into HRMs, and how the "hierarchical" part proved to be a lot less meaningful than anticipated.
Eh, think you might not have read my comment well enough, because actually I partially agree with you in that two models may not be the best way to train an LLM. Yeah, I did say "In AI, maybe two models should be developed and trained in conjunction," meaning this is just some random idea based on the two system idea that might be useful.
Was just implying we don't know all the useful ideas (not just in tech but in medicine, genetics, evolutionary bioligy, mathematics...) that might be developed because we now know that the brain is two separate systems. It's all good though.
> "The brain does it" is not in itself a reason to do the same in AI.
What are the Tensors in LLMs modeled after?
That's the thing: we ripped the entire idea of learning connectionist networks straight out of the brain - but our implementation of that idea is very different.
The asynchronous nature of BNNs? The brain does that because its neurons are slow and synchronization is unnatural. The predictive Hebbian learning hybrid that the brain uses? Loses to backprop if you have what it takes to implement backprop.
There are a lot of mediocre or unapplicable ideas in the brain for every good one. Figuring out how the brain does the things it does is hard enough - but making use of that is even harder. It's why ANNs and BNNs diverge so much in form even as they converge in function.
> And actually, the fact that the brain is actually two "developmental" systems combined into one seems like big news to me.
It was known and discussed in evolutionary-developmental neuroscience, but not much outside that.
> maybe two models should be developed and trained in conjunction that talk to each other, each with different functions and roles, because brains work better this way.
I mean, is that not essentially what a Mixture of Experts model is?
Ah, didn't know the term for this, but vaguely remembered articles multiple models being trained together. Good to know:)