I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do. They figure “it’s science, of course there’s math and theory behind it.”
AI research is almost as purely empirical as the gradient descent loops its practitioners use to optimize their models. “Why” anything at all works is barely an afterthought.
> I’m shocked how many otherwise well-informed people don’t understand or agree with this very fundamental fact of just how little we actually understand about why LLMs work as well as they do.
Well-informed people understand that LLMs work as well as they do for the same reasons as horoscopes, fortune-telling and homeopathy.
They work great for banging out POC apps or writing boilerplate code. This is a real use.
Homeopathy triggers the placebo effect. That is a real use.
If horoscopes and homeopathy can generate tests, then call me a Pisces and pass me the singular molecule of deadly nightshade toxin.
Be careful what you wish for. This weekend in tearing out hundreds of crappy tests that got vibed into a codebase.
...well, you could train an LLM on whatever horoscopes and then ask it :-?
> why LLMs work as well as they do.
That's a very different claim from being "poorly understood" though. The emergent properties of any system with billions of parameters is hard to understand completely, that's the fault of data science more than computer science or even mathematics.
I think ”poorly understood” is accurate. Understanding has levels. How brains think is also poorly understood.
I disagree, because you can represent the constituent parts of any AI model as code and data. We can reliably build AI with this knowledge, but not brains.
Understanding does have layers, and that's why "poorly understood" is a meaningless goalpost. A book can be well understood without researching the gematria behind character's the names when you write them in reverse. An LLM can be well-understood even if you don't comprehensively test each quantization for miraculous unexpected behavior at the FFN level.
A book can be poorly understood if you know it's made out of paper and ink, but can't read. The contents would be recognizable but are meaningless symbols and numbers. You might note that some patterns of symbols repeat, but be completely clueless why or what it represents.
Notably, you could still print them all day.
Being able to put a PC together doesn't mean you understand computers. If I ask you to make me a computer that runs a wristwatch, you'll likely be lost unless you have very specialized understanding.
I think we actually all know what "poorly understood" means. There's no need to play tedious semantic games.
The semantic game occurred when someone was equivocating "LLMs are poorly understood" with "brains are poorly understood", which are worlds apart in the extent to which they are "not understood".
“equivocating” is not the word you meant to use here. Probably “equating” I suspect. But yes, I stand behind my claim and I deny that it’s a semantic game. The two areas are not identical in human understanding but they’re on the same order.
If I ask you what makes a good fantasy novel and you explain to me the English alphabet and grammar, you actually haven’t explained what I asked.
It’s likely that at some point we will also be able to represent a scan of the human brain digitally as code and data. Assuming we don’t separately make a huge number of leaps in neuroscience which are far from assured, it will be very likely that we can get the appearance of an operational human brain simulated digitally well before we gain much if any understanding of why it does what it does.
All that is to say, being able to build something is not not not the same thing as understanding it.
Your DNA is merely data, and humans can reliably make more of it too.
If you are serious, comparing how well we understand the brain vs how well we understand LLMs, .. it's not a stretch simplifying that to "we don't understand brains, we do understand LLMs".
Because the extent to which we don't understand the brain, is quite overpowering.
Some people forget that when they say "but it's not different from what a human does" ...
I am absolutely serious. I agree with you that the extent to which we don’t understand brains is overpowering. And I would stand by the proposition that “we don’t understand LLMs in almost exactly the way we don’t understand brains”.
We know lots about human development and genetics and biology and evolution and neuroscience and the physics of how brains are connected and send signals and how generally they are put together and have names for their parts and all that, but we’re clueless when it comes to “the hard question” of how qualia and consciousness emerges from that.
The scenario with the spooky simulation of thinking that emerges from LLMs is in the same category, with different details. Lots of knowledge about the substrate of the phenomenon, little to none about the much bigger question of how we get the appearance of cognition from these trained artifacts.
Clearly we understand extremely well how LLMs are created mechanically. We invented them and are currently putting massive amounts of work into studying and improving them. But that work is perforce largely empirical; figuring out the why once again eludes us. It just goes to show how mysterious the underlying phenomenon of cognition is.
I'm not following you here, you seem to be conflating LLMs ability of language use with the brain's ability of thinking and cognition?
Are you saying that thinking and cognition requires language use? Cause I think not.
Or are you saying that language use is sufficient for cognition and thinking? Cause I'm also not convinced of that.
What I am convinced of, is that a machine capable of language use is capable of tricking people into believing there's a "there", there. In pretty much the same way as the famous supra-normal stimuli experiment made baby seagulls believe that a stick with a red dot was their parent. It's exploiting our instincts.
> as purely empirical as the gradient descent loops
Are you suggesting that gradient descent is an empirically found and not understood technique? It was originally proposed by Cauchy in 1847, its properties are very well understood.
You might be referring to properties of the domains its being applied to.
I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
> I’m not saying gradient descent was empirically discovered, I’m saying that its use in machine learning is itself a form of empiricism. A repeated observe/adjust-based-on-data cycle
The data is the input, the output is to generally find the lowest amount of a loss function. It’s a greedy approach because brute forcing is inefficient.
It’s no more empirical than a greedy algorithm for scheduling.
> It’s no more empirical than a greedy algorithm for scheduling.
Right, GP is drawing a distinction between search, ie mechanical exploration of a space, with understanding, ie having a map of the territory such that you don’t need trial and error.
> its use in machine learning (or elsewhere, I suppose) is itself a form of empiricism in that what it is is essentially a repeated observe/measure error/adjust cycle.
"Empiricism" implies that the technique is based on observable, but not mathematically proven foundations. If a problem space is convex, gradient descent is guaranteed to converge to a global optimal solution, regardless of whether you know the exact formulation of the space.
Applying it when you don't understand if a space is convex is another question, but that's not a fault of gradient descent.