I'm trying to keep an open mind and understand what the author is trying to say because he is credentialed.
His main point is that discoveries involve
1. Variation,
2. Evaluation, and
3. Selective retention.
He makes a jump saying AI is only capable of 1) and humans are capable of 1) 2) and 3). I don't know what makes humans special enough that they can do 2) and 3)?
In fact, the more you think of this it is kind of strange - in science humans can only do "evaluation" because they have access to the real world. They can evaluate a new drug because they can do it on people so it is not some inherent limitation of AI but rather access to physical realm.
Finally I want to ask a specific thing: how do you mathematically falsify what this person is saying? How can you formally prove that - no AI can not "evaluate"? I ask because I make AI evaluate a lot of people's claims and it works for me.
He's saying that pre-training an LLM alone can't do it, but if you run an LLM in a loop with tools (like any coding agent) then it can. Also, the technique his group came up with should be used more:
> This is the weakness of deep learning that is alleviated with a new algorithm that my group presented in Nature a couple of years ago. Our “continual backpropagation” made one small change: every so often a less-used neuron would be re-initialized to small random weights. This allows the variation to continue and plasticity to be retained.
Here's the paper: https://www.nature.com/articles/s41586-024-07711-7
It has a fair number of citations, but I haven't looked into how much it's used.
Sorry this makes no sense — humans also use tools to evaluate their discoveries.
Kant said something like this: knowledge can’t be obtained by pure thinking, it needs interaction with the world.
This is obvious to me so why is the author making a claim that LLMs can make knowledge without access to environment but purely through thinking in aether
He actually says the areas in which AI has had the novel successes are those which can be evaluated (like coding or Go). Not that it can’t happen at all.
That’s my point, he says ai does well where evaluation is neurosymbolically closed.
But so do humans? How do humans make discoveries without having formal ways to evaluate? In my pharma drug example, humans could evaluate only because they had access to the physical realm.
I can’t think of an example of humans evaluating a discovery in a way that LLMs can’t. can you?
I don't think there is any "humans are metaphysically superior to LLMs" subtext to this talk, it's just a technical/educational observation.
Access to some forms of evaluation and selective retention is inherent to humans and it's not inherent to LLMS. But it can be somehow bolted on and that's when they work best. It makes sense that more focus on those principles can yield better AI. I think the retention part is the real limitation of LLMs, because it's limited to stuffing things in context window.