~85% accuracy on MNIST. Sigh.
How does it do on CIFAR-10, or even better, ImageNet?
Interesting research, not sure it's a backprop alternative.
===
EDIT: accuracy on MNIST is not ~90%. It's ~85%.
~85% accuracy on MNIST. Sigh.
How does it do on CIFAR-10, or even better, ImageNet?
Interesting research, not sure it's a backprop alternative.
===
EDIT: accuracy on MNIST is not ~90%. It's ~85%.
It might be beneficial while not being optimal on its own.
The obvious example is if it has different behaviour around local minima, it could be an altenate pathway out.
I have often wondered if doing training with radically different aproaches for the first few iterarions would avoid any method specific artifacts before the weights had time to denoise.
Yes, it's interesting research, I agree and even wrote so in my post :-)
But the submission title calls it a "backprop alternative." It is not, at least not yet.
You can probably distribute training more easily too
They state replacing backprop is not their goal.
Their goal is to understand how distributed systems which cannot do backprop (the brain) can still do learning.
"Backprop alternative" was the OP's headline.
I think my initial assessment is correct: interesting research, but not an alternative, at least not yet.
Ah. I missed that. Yeah. "Backprop alternative" is the wrong framing.
Personally I think the dirty secret of the brain is that a lot of things are hard coded. And many things that we need to learn are also hard coded except that some parameters need to be tuned.
If we puke, the brain will not do general aversive learning, it will learn to avoid specifically the last thing eaten, because it instinctively knows about food poisoning.
Imprinting is absolutely fascinating. Some newborn animals will run a very simple pattern detector like looking for a red dot or something and use that to bootstrap their conception of their parent.
For fully general learning I have a hunch that it can be done using local history plus a semi-global reward scalar (global neurotransmittor levels).
regardless if the intelligence in the brain is hardcoded or not, to the extent it is, this information must have been compressed in the genome, which runs counter to almost all observations: a child doesn't remember the experience of their ancestors, for example. The only sense in which we do carry mental state without relearning is emotions, instincts, reflexes (some neuronal pathways that connect the eye to the middle ear), hormonal driven behavior (fear adrenalin).
For another, there are about 200k promotor regions (including non-coding) in the human genome.
A promotor region might have say 6 to 15 bits of information.
Can you compress 2025 or even 2024 era LLM intelligence into 3 megabit = ~400 kB ? I think not. I think a lot of compression is still possible, but 400 kB?
So I think we can box up the idea of "dirty secrets of the braing: not learning but hard coding". There is a lot of hard coding in biology, but brains are evolved specifically to enable learning within the individual lifetime instead of only learning by natural selection.
I also don't buy the following argument:
> If we puke, the brain will not do general aversive learning, it will learn to avoid specifically the last thing eaten, because it instinctively knows about food poisoning.
Each time it happens that I end up puking, I do feel aversion and try to avoid puking at all, sometimes I succeed but sometimes is just puke. There must be fundamental puke reflexes (which one fails to avoid) and avertable puke reflexes.
You are thinking on the wrong level. Of course we don't have an encyclopedic knowledge of the world encoded into our genome. It's like we have certain structures of the world hard coded, and they may use different learning algorithms.
We instinctively know that there are other intelligent beings, and we have the the machinery to model them. We are born with the capacity for language. It must still be learned, the specific words aren't hardcoded, but the concept of language is. We are born with the capacity to store and replay memories. The brain knows some aspects of how the world is supposed to look like visually, and if it doesn't it will try to correct that. People who used optics to see the world upside down have found that after a brief time their brain learned to flip the world right side up.
> Can you compress 2025 or even 2024 era LLM intelligence into 3 megabit = ~400 kB ? I think not. I think a lot of compression is still possible, but 400 kB?
There are a few extra levels of interpretation (like protein synthesis) that are more like a transpiler than compression (imo), over a 4-base language that is read in a sliding window and is affected by surrounding conditions, so the same "token" sequence may produce different things depending on external factors. Some biologists I used to collaborate with talked about 7 layers to this process, I have only described one level here
From an information theory perspective it does not matter how many levels of interpretation are in between, that hardcoded information must pass the genome. for example we can discuss "what about protein synthesis", for example binding affinities, folding helpers etc. they in turn were encoded genetically as well.
> There are a few extra levels of interpretation (like protein synthesis) that are more like a transpiler than compression (imo), over a 4-base language that is read in a sliding window and is affected by surrounding conditions, so the same "token" sequence may produce different things depending on external factors. Some biologists I used to collaborate with talked about 7 layers to this process, I have only described one level here
so the same "token" sequence may produce different things depending on external factors.
yes, non-hereditary learning depends on external factors, thank you for paraphrasing me while shifting attention.
the multi-scale nature (transpilers etc.) doesn't change the theorems in probability and information theory which seriously constrain the maximum amount of information a message can store.
The whole point of a brain is that it is an organ dedicated to storing, retrieving and timely utilising information one can't afford to store in a genome.
I'm not convinced information theory is the right avenue for such non-deterministic, environmentally affected, open ended systems. Protein synthesis is defined by far more than the DNA, which is always being processed, most of which is junk and being recycled. The layers in between are more than interpretation of the prior layers, the data grows at each level to incorporate more sources under your information theoretic formulation.
I was never paraphrasing you, but thank you for attempting rhetorical antics?
Their image classification benchmarks include both: https://pub.sakana.ai/pc-alm/assets/figures/benchmark_accura...
~74% on CIFAR-10. Still a far cry from backprop.
I didn't see ImageNet. TinyImageNet is something else.
There might be only one paper that's trained on full Imagenet using methods like these.
Training a Predictive Coding Network on ImageNet using Equilibrium Propagation Tugdual Kerjan, Rasmus Høier, Benjamin Scellier https://arxiv.org/abs/2606.03584
It's quite an undertaking.
It's really sad that they are only a few years away from making backpropagation completely obsolete.